Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-35964

Assertion `ist_seqno >= cc_seqno' failed in galera_vote_rejoin_dml

Details

    • Bug
    • Status: Open (View Workflow)
    • Critical
    • Resolution: Unresolved
    • 11.4
    • 10.5, 11.4
    • Galera
    • None

    Description

      Assertion `ist_seqno >= cc_seqno' failed in galera_vote_rejoin_dml:

      galera.galera_vote_rejoin_dml		 [ fail ]
      	Test ended at 2025-01-29 00:58:22
       
      WSREP did not transition to state READY
       
       - saving '/home/panda/maria-11.4/build/mysql-test/var/log/galera.galera_vote_rejoin_dml/' to '/home/panda/maria-11.4/build/mysql-test/var/log/galera.galera_vote_rejoin_dml/'
      ***Warnings generated in error logs during shutdown after running tests: galera.galera_vote_rejoin_dml
       
      2025-01-29  0:54:17 2 [ERROR] WSREP: Receiving IST failed, node restart required: IST receiver reported failure: 'IST didn't contain all write sets, expected last: 3 last received: -1' (71)
      mariadbd: /home/panda/galera-es/galera/src/replicator_str.cpp:1086: void galera::ReplicatorSMM::request_state_transfer(void*, int, const wsrep_uuid_t&, wsrep_seqno_t, const void*, ssize_t): Assertion `ist_seqno >= cc_seqno' failed.
      Attempting backtrace. You can use the following information to find out

      Fragment from node_3 log:

      2025-01-29  0:54:17 0 [Note] WSREP: SST received: 1b7e0386-ddd3-11ef-85a3-369b4e14843d:3
      2025-01-29  0:54:17 0 [Note] WSREP: SST succeeded for position 1b7e0386-ddd3-11ef-85a3-369b4e14843d:3
      2025-01-29  0:54:17 0 [Note] WSREP: Joiner monitor thread ended with total time 27 sec
      2025-01-29  0:54:17 2 [Note] WSREP: Installed new state from SST: 1b7e0386-ddd3-11ef-85a3-369b4e14843d:3
      2025-01-29  0:54:17 0 [Note] /home/panda/maria-11.4/build/sql/mariadbd: ready for connections.
      Version: '11.4.5-3-MariaDB-debug-log'  socket: '/home/panda/maria-11.4/build/mysql-test/var/tmp/mysqld.3.sock'  port: 19002  Source distribution
      2025-01-29  0:54:17 2 [Note] WSREP: Cert. index preload up to 3
      2025-01-29  0:54:17 0 [Note] WSREP: ####### IST applying starts with 4
      2025-01-29  0:54:17 2 [ERROR] WSREP: Receiving IST failed, node restart required: IST receiver reported failure: 'IST didn't contain all write sets, expected last: 3 last received: -1' (71)
      	 at /home/panda/galera-es/galera/src/replicator_smm.hpp:pop_front():339. Null event.
      2025-01-29  0:54:17 2 [Note] WSREP: Closing send monitor...
      2025-01-29  0:54:17 2 [Note] WSREP: Closed send monitor.
      2025-01-29  0:54:17 2 [Note] WSREP: gcomm: terminating thread
      2025-01-29  0:54:17 2 [Note] WSREP: gcomm: joining thread
      2025-01-29  0:54:17 2 [Note] WSREP: gcomm: closing backend
      2025-01-29  0:54:17 7 [Note] WSREP: assigned new next trx id: 12
      2025-01-29  0:54:17 7 [Note] WSREP: wsrep_after_statement for 7 client_state exec  client_mode local trans_state executing
      2025-01-29  0:54:17 7 [Note] WSREP: assigned new next trx id: 13
      2025-01-29  0:54:17 2 [Note] WSREP: view(view_id(NON_PRIM,1b7796c3-8b7f,4) memb {
      	1ce68133-983c,0
      } joined {
      } left {
      } partitioned {
      	1b7796c3-8b7f,0
      	1c87e7b9-bdcf,0
      	1d305b6a-af01,0
      })
      2025-01-29  0:54:17 2 [Note] WSREP: PC protocol downgrade 1 -> 0
      2025-01-29  0:54:17 2 [Note] WSREP: view((empty))
      2025-01-29  0:54:17 2 [Note] WSREP: gcomm: closed
      2025-01-29  0:54:17 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
      2025-01-29  0:54:17 0 [Note] WSREP: Flow-control interval: [16, 16]
      2025-01-29  0:54:17 0 [Note] WSREP: Received NON-PRIMARY.
      2025-01-29  0:54:17 0 [Note] WSREP: Shifting JOINER -> OPEN (TO: 3)
      2025-01-29  0:54:17 0 [Note] WSREP: New SELF-LEAVE.
      2025-01-29  0:54:17 0 [Note] WSREP: Flow-control interval: [0, 0]
      2025-01-29  0:54:17 0 [Note] WSREP: Received SELF-LEAVE. Closing connection.
      2025-01-29  0:54:17 0 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 3)
      2025-01-29  0:54:17 0 [Note] WSREP: RECV thread exiting 0: Success
      2025-01-29  0:54:17 2 [Note] WSREP: recv_thread() joined.
      2025-01-29  0:54:17 2 [Note] WSREP: Closing send queue.
      2025-01-29  0:54:17 2 [Note] WSREP: Closing receive queue.
      mariadbd: /home/panda/galera-es/galera/src/replicator_str.cpp:1086: void galera::ReplicatorSMM::request_state_transfer(void*, int, const wsrep_uuid_t&, wsrep_seqno_t, const void*, ssize_t): Assertion `ist_seqno >= cc_seqno' failed.
      250129	0:54:17 [ERROR] mysqld got signal 6 ;
      Sorry, we probably made a mistake, and this is a bug.
       
      Your assistance in bug reporting will enable us to fix this for the next release.
      To report this bug, see https://mariadb.com/kb/en/reporting-bugs
       
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed,
      something is definitely wrong and this may fail.
       
      Server version: 11.4.5-3-MariaDB-debug-log source revision: 1f1a1f8e3326b05bdeb292c04371e05714cdc860
      key_buffer_size=1048576
      read_buffer_size=131072
      max_used_connections=1
      max_threads=153
      thread_count=3
      It is possible that mysqld could use up to
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 64027 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      WSREP: Suppressing further logging
      WSREP: Shutting down network communications
       
      Thread pointer: 0x7f5d3c000dc8
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x7f5d642af000 thread_stack 0x49000
      2025-01-29  0:54:17 7 [Note] WSREP: wsrep_after_statement for 7 client_state exec  client_mode local trans_state aborted
      2025-01-29  0:54:17 7 [Note] WSREP: wsrep_thd_binlog_reset
      2025-01-29  0:54:17 7 [Note] WSREP: wsrep_after_statement for 7 client_state exec  client_mode local trans_state aborted
      2025-01-29  0:54:17 7 [Note] WSREP: wsrep_thd_binlog_reset
      2025-01-29  0:54:17 7 [Note] WSREP: assigned new next trx id: 14
      2025-01-29  0:54:17 0 [Note] /home/panda/maria-11.4/build/sql/mariadbd (initiated by: unknown): Normal shutdown
      2025-01-29  0:54:17 0 [Note] WSREP: Shutdown replication
      2025-01-29  0:54:17 0 [Note] WSREP: Disconnect provider
      2025-01-29  0:54:17 0 [Note] WSREP: server panda state change: joined -> disconnecting
      2025-01-29  0:54:17 0 [Note] WSREP: Server status change joined -> disconnecting
      2025-01-29  0:54:17 0 [Note] WSREP: Setting wsrep_ready to 0
      2025-01-29  0:54:17 0 [Note] WSREP: wsrep_notify_status server not yet ready : wsrep_ready=0 status 8
      mysys/stacktrace.c:215(my_print_stacktrace)[0x55f685287065]
      sql/signal_handler.cc:247(handle_fatal_signal)[0x55f68489cb30]
      libc_sigaction.c:0(__restore_rt)[0x7f5d6ab02520]
      nptl/pthread_kill.c:44(__pthread_kill_implementation)[0x7f5d6ab569fc]
      posix/raise.c:27(__GI_raise)[0x7f5d6ab02476]
      stdlib/abort.c:81(__GI_abort)[0x7f5d6aae87f3]
      intl/loadmsgcat.c:1177(_nl_load_domain)[0x7f5d6aae871b]
      /lib/x86_64-linux-gnu/libc.so.6(+0x39e96)[0x7f5d6aaf9e96]
      src/replicator_str.cpp:1087(galera::ReplicatorSMM::request_state_transfer(void*, int, wsrep_uuid const&, long, void const*, long))[0x7f5d65097eb2]
      src/replicator_smm.cpp:2720(galera::ReplicatorSMM::process_st_required(void*, int, wsrep_view_info const*))[0x7f5d650702ac]
      src/replicator_smm.cpp:3013(galera::ReplicatorSMM::process_prim_conf_change(void*, gcs_act_cchange const&, int, void*))[0x7f5d650718d1]
      src/replicator_smm.cpp:2505(galera::ReplicatorSMM::process_conf_change(void*, gcs_act_cchange const&, gcs_action const&))[0x7f5d6506e7f6]
      src/gcs_action_source.cpp:128(galera::GcsActionSource::dispatch(void*, gcs_action const&, bool&))[0x7f5d650c6b41]
      src/gcs_action_source.cpp:192(galera::GcsActionSource::process(void*, bool&))[0x7f5d650c721d]
      src/replicator_smm.cpp:404(galera::ReplicatorSMM::async_recv(void*))[0x7f5d650610ba]
      src/wsrep_provider.cpp:292(galera_recv)[0x7f5d65037c86]
      src/wsrep_provider_v26.cpp:866(wsrep::wsrep_provider_v26::run_applier(wsrep::high_priority_service*))[0x55f6853a900c]
      sql/wsrep_thd.cc:57(wsrep_replication_process(THD*, void*))[0x55f684ccdbab]
      sql/wsrep_mysqld.cc:4200(start_wsrep_THD(void*))[0x55f684ca3afd]
      perfschema/pfs.cc:2203(pfs_spawn_thread)[0x55f684bedda0]
      nptl/pthread_create.c:442(start_thread)[0x7f5d6ab54ac3]
      x86_64/clone3.S:83(__clone3)[0x7f5d6abe6850]

      Attachments

        1. mysqld.1.err
          59 kB
        2. mysqld.2.err
          53 kB
        3. mysqld.3.err
          36 kB
        4. mysqld.4.err
          43 kB
        5. stdout
          0.9 kB

        Activity

          There are no comments yet on this issue.

          People

            Yurchenko Alexey
            sysprg Julius Goryavsky
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.