Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-32276

wsrep::client_state::bf_abort(wsrep::seqno): Assertion `mode_ == m_local || transaction_.is_streaming()' failed

    XMLWordPrintable

Details

    • Bug
    • Status: Open (View Workflow)
    • Critical
    • Resolution: Unresolved
    • 10.10.2
    • 10.11, 11.0, 11.1
    • Galera SST
    • None
    • mariadb galera cluster 6 nodes.

    Description

      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed, 
      something is definitely wrong and this may fail.
       
      Server version: 10.10.2-MariaDB
      key_buffer_size=134217728
      read_buffer_size=131072
      max_used_connections=167
      max_threads=2002
      thread_count=172
      It is possible that mysqld could use up to 
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 4540309 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x7f232c0009b8
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x7f23b4092ca0 thread_stack 0x49000
      /usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x55f6ad4bfdfe]
      /usr/sbin/mariadbd(handle_fatal_signal+0x307)[0x55f6acf25cc7]
      sigaction.c:0(__restore_rt)[0x7f23bacdb5d0]
      :0(__GI_raise)[0x7f23ba128207]
      :0(__GI_abort)[0x7f23ba1298f8]
      :0(__assert_fail_base)[0x7f23ba121026]
      :0(__GI___assert_fail)[0x7f23ba1210d2]
      /usr/sbin/mariadbd(_Z14wsrep_bf_abortP3THDS0_+0x622)[0x55f6ad1ce852]
      /usr/sbin/mariadbd(wsrep_thd_bf_abort+0x19)[0x55f6ad1d5259]
      /usr/sbin/mariadbd(+0xd13e29)[0x55f6ad1f6e29]
      /usr/sbin/mariadbd(+0x6a6ff1)[0x55f6acb89ff1]
      /usr/sbin/mariadbd(+0xd615e2)[0x55f6ad2445e2]
      /usr/sbin/mariadbd(+0xde3a2f)[0x55f6ad2c6a2f]
      /usr/sbin/mariadbd(+0xde606d)[0x55f6ad2c906d]
      wsrep/client_state.hpp:679(wsrep_bf_abort(THD*, THD*))[0x55f6ad20293e]
      sql/service_wsrep.cc:228(wsrep_thd_bf_abort)[0x55f6acf3349f]
      handler/ha_innodb.cc:18635(wsrep_notify_status(wsrep::server_state::state, wsrep::view const*))[0x55f6ad058125]
      /usr/sbin/mariadbd(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x116)[0x55f6ad058626]
      /usr/sbin/mariadbd(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x2d6)[0x55f6ad04cc06]
      lock/lock0lock.cc:1004(??)[0x55f6ad044607]
      lock/lock0lock.cc:1772(wsrep_notify_status(wsrep::server_state::state, wsrep::view const*))[0x55f6ad1cd7fd]
      row/row0mysql.cc:682(void std::__introsort_loop<unsigned char**, long>(unsigned char**, unsigned char**, long))[0x55f6ad1b2ed0]
      row/row0mysql.cc:1324(void std::__introsort_loop<unsigned char**, long>(unsigned char**, unsigned char**, long))[0x55f6ad1b2fc3]
      handler/ha_innodb.cc:7843(wsrep_notify_status(wsrep::server_state::state, wsrep::view const*))[0x55f6ad5620ff]
      sql/handler.cc:7561(handler::ha_write_row(unsigned char const*))[0x55f6ad562e25]
      sql/log_event_server.cc:7701(Rows_log_event::write_row(rpl_group_info*, bool))[0x55f6ad576758]
      src/trx_handle.cpp:392(galera::TrxHandleSlave::apply(void*, wsrep_cb_status (*)(void*, wsrep_ws_handle const*, unsigned int, wsrep_buf const*, wsrep_trx_meta const*, bool*), wsrep_trx_meta const&, bool&))[0x7f23b6d645f0]
      src/replicator_smm.cpp:516(galera::ReplicatorSMM::apply_trx(void*, galera::TrxHandleSlave&))[0x7f23b6d6e250]
      src/replicator_smm.cpp:2156(galera::ReplicatorSMM::process_trx(void*, boost::shared_ptr<galera::TrxHandleSlave> const&))[0x7f23b6d747b1]
      src/gcs_action_source.cpp:63(galera::GcsActionSource::process_writeset(void*, gcs_action const&, bool&))[0x7f23b6da0f10]
      src/gcs_action_source.cpp:110(galera::GcsActionSource::dispatch(void*, gcs_action const&, bool&))[0x7f23b6da1d32]
      src/gcs_action_source.cpp:29(~Release)[0x7f23b6da25b4]
      src/replicator_smm.cpp:402(galera::ReplicatorSMM::async_recv(void*))[0x7f23b6d7567c]
      src/wsrep_provider.cpp:291(galera_recv)[0x7f23b6d56408]
      /usr/sbin/mariadbd(_ZN5wsrep18wsrep_provider_v2611run_applierEPNS_21high_priority_serviceE+0xe)[0x55f6ad576e8e]
      /usr/sbin/mariadbd(+0xcec322)[0x55f6ad1cf322]
      /usr/sbin/mariadbd(_Z15start_wsrep_THDPv+0x2c8)[0x55f6ad1bf7c8]
      /usr/sbin/mariadbd(+0xc6b97c)[0x55f6ad14e97c]
      pthread_create.c:0(start_thread)[0x7f23bacd3dd5]
      ??:0(__clone)[0x7f23ba1efead]
       
      Trying to get some variables.
      Some pointers may be invalid and cause the dump to abort.
      Query (0x7f239f7f4ce3): insert into t_zxg_group_stock (group_stock_id, group_id, sort_no,
          stock_code, un_convertable_source, collect_date,collect_price,create_by, update_by)
          values
            
          (null,
          7523495,
          1,
          '600400.sh|4353',
          'zz',
          '2023-09-17',
          100.00,
          '72100008089',
          '72100008089')
           , 
          (null,
          7523495,
          2,
          '600300.sh|4353',
          'zz',
          '2023-09-17',
          100.00,
          '72100008089',
          '72100008089')
           , 
          (null,
          7523495,
          3,
          '600301.sh|4353',
          'zz',
          '2023-09-17',
          100.00,
          '72100008089',
          '72100008089')
       
      Connection ID (thread ID): 6
      Status: NOT_KILLED
       
      Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off
       
      The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
      information that should help you find out what is causing the crash.
      Writing a core file...
      Working directory at /rdc/data/mysql
      Resource Limits:
      Limit                     Soft Limit           Hard Limit           Units     
      Max cpu time              unlimited            unlimited            seconds   
      Max file size             unlimited            unlimited            bytes     
      Max data size             unlimited            unlimited            bytes     
      Max stack size            8388608              unlimited            bytes     
      Max core file size        0                    unlimited            bytes     
      Max resident set          unlimited            unlimited            bytes     
      Max processes             96340                96340                processes 
      Max open files            32768                32768                files     
      Max locked memory         65536                65536                bytes     
      Max address space         unlimited            unlimited            bytes     
      Max file locks            unlimited            unlimited            locks     
      Max pending signals       96340                96340                signals   
      Max msgqueue size         819200               819200               bytes     
      Max nice priority         0                    0                    
      Max realtime priority     0                    0                    
      Max realtime timeout      unlimited            unlimited            us        
      Core pattern: core
      

      Kernel version: Linux version 3.10.0-957.el7.x86_64 (mockbuild@x86-040.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-36) (GCC) ) #1 SMP Thu Oct 4 20:48:51 UTC 2018
      

      2023-09-17 16:44:42 0 [Note] WSREP: Loading provider /usr/lib64/galera-4/libgalera_smm.so initial position: 171a890e-7d0a-11ed-8263-6722050e329f:13215589
      2023-09-17 16:44:42 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/galera-4/libgalera_smm.so'
      2023-09-17 16:44:42 0 [Note] WSREP: wsrep_load(): Galera 26.4.13(rfe497ae) by Codership Oy <info@codership.com> loaded successfully.
      2023-09-17 16:44:42 0 [Note] WSREP: Initializing allowlist service v1
      2023-09-17 16:44:42 0 [Note] WSREP: CRC-32C: using 64-bit x86 acceleration.
      2023-09-17 16:44:42 0 [Note] WSREP: Found saved state: 171a890e-7d0a-11ed-8263-6722050e329f:-1, safe_to_bootstrap: 0
      2023-09-17 16:44:42 0 [Note] WSREP: GCache DEBUG: opened preamble:
      Version: 2
      UUID: 171a890e-7d0a-11ed-8263-6722050e329f
      Seqno: -1 - -1
      Offset: -1
      Synced: 0
      2023-09-17 16:44:42 0 [Note] WSREP: Recovering GCache ring buffer: version: 2, UUID: 171a890e-7d0a-11ed-8263-6722050e329f, offset: -1
      2023-09-17 16:44:42 0 [Note] WSREP: GCache::RingBuffer initial scan...  0.0% (        0/314572824 bytes) complete.
      2023-09-17 16:44:42 0 [Note] WSREP: GCache::RingBuffer initial scan...100.0% (314572824/314572824 bytes) complete.
      2023-09-17 16:44:42 0 [Note] WSREP: Recovering GCache ring buffer: found gapless sequence 13018942-13215593
      2023-09-17 16:44:42 0 [Note] WSREP: GCache::RingBuffer unused buffers scan...  0.0% (        0/314303296 bytes) complete.
      2023-09-17 16:44:42 0 [Note] WSREP: Recovering GCache ring buffer: found 0/196652 locked buffers
      2023-09-17 16:44:42 0 [Note] WSREP: Recovering GCache ring buffer: free space: 269504/314572800
      2023-09-17 16:44:42 0 [Note] WSREP: GCache::RingBuffer unused buffers scan...100.0% (314303296/314303296 bytes) complete.
      2023-09-17 16:44:42 0 [Note] WSREP: Passing config to GCS: base_dir = /rdc/data/mysql/; base_host = 15.24.131.139; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /rdc/data/mysql/; gcache.keep_pages_size = 0; gcache.keep_plaintext_size = 300M; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 300M; gcache.recover = yes; gcache.size = 300M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.fc_single_primary = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment =
      2023-09-17 16:44:42 0 [Note] WSREP: Service thread queue flushed.
      2023-09-17 16:44:42 0 [Note] WSREP: ####### Assign initial position for certification: 171a890e-7d0a-11ed-8263-6722050e329f:13215589, protocol version: -1
      2023-09-17 16:44:42 0 [Note] WSREP: Start replication
      2023-09-17 16:44:42 0 [Note] WSREP: Connecting with bootstrap option: 0
      2023-09-17 16:44:42 0 [Note] WSREP: Setting GCS initial position to 171a890e-7d0a-11ed-8263-6722050e329f:13215589
      2023-09-17 16:44:42 0 [Note] WSREP: protonet asio version 0
      2023-09-17 16:44:42 0 [Note] WSREP: Using CRC-32C for message checksums.
      2023-09-17 16:44:42 0 [Note] WSREP: backend: asio
      2023-09-17 16:44:42 0 [Note] WSREP: gcomm thread scheduling priority set to other:0 
      2023-09-17 16:44:42 0 [Note] WSREP: restore pc from disk successfully
      2023-09-17 16:44:42 0 [Note] WSREP: GMCast version 0
      2023-09-17 16:44:42 0 [Note] WSREP: (c1807c32-9da7, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
      2023-09-17 16:44:42 0 [Note] WSREP: (c1807c32-9da7, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
      2023-09-17 16:44:42 0 [Note] WSREP: EVS version 1
      2023-09-17 16:44:42 0 [Note] WSREP: gcomm: connecting to group 'rdc-cloud', peer '15.24.131.139:,15.24.131.140:,15.24.131.141:,10.35.163.64:,10.35.163.65:,10.35.163.66:'
      2023-09-17 16:44:42 0 [Note] WSREP: (c1807c32-9da7, 'tcp://0.0.0.0:4567') Found matching local endpoint for a connection, blacklisting address tcp://15.24.131.139:4567
      2023-09-17 16:44:42 0 [Note] WSREP: (c1807c32-9da7, 'tcp://0.0.0.0:4567') connection established to c7052230-8912 tcp://15.24.131.140:4567
      2023-09-17 16:44:42 0 [Note] WSREP: (c1807c32-9da7, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: 
      2023-09-17 16:44:42 0 [Note] WSREP: (c1807c32-9da7, 'tcp://0.0.0.0:4567') connection established to b5e2166d-bb21 tcp://15.24.131.141:4567
      2023-09-17 16:44:42 0 [Note] WSREP: (c1807c32-9da7, 'tcp://0.0.0.0:4567') connection established to 36974392-87e4 tcp://10.35.163.64:4567
      2023-09-17 16:44:42 0 [Note] WSREP: (c1807c32-9da7, 'tcp://0.0.0.0:4567') connection established to 1c6062d8-a53e tcp://10.35.163.65:4567
      2023-09-17 16:44:42 0 [Note] WSREP: (c1807c32-9da7, 'tcp://0.0.0.0:4567') connection established to 7e9b63be-a85e tcp://10.35.163.66:4567
      2023-09-17 16:44:42 0 [Note] WSREP: gcomm: connected
      2023-09-17 16:44:42 0 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
      2023-09-17 16:44:42 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
      2023-09-17 16:44:42 0 [Note] WSREP: Opened channel 'rdc-cloud'
      2023-09-17 16:44:42 1 [Note] WSREP: Starting rollbacker thread 1
      2023-09-17 16:44:42 2 [Note] WSREP: Starting applier thread 2
      2023-09-17 16:44:43 0 [Note] WSREP: EVS version upgrade 0 -> 1
      2023-09-17 16:44:43 0 [Note] WSREP: declaring 1c6062d8-a53e at tcp://10.35.163.65:4567 stable
      2023-09-17 16:44:43 0 [Note] WSREP: declaring 36974392-87e4 at tcp://10.35.163.64:4567 stable
      2023-09-17 16:44:43 0 [Note] WSREP: declaring 7e9b63be-a85e at tcp://10.35.163.66:4567 stable
      2023-09-17 16:44:43 0 [Note] WSREP: declaring b5e2166d-bb21 at tcp://15.24.131.141:4567 stable
      2023-09-17 16:44:43 0 [Note] WSREP: declaring c7052230-8912 at tcp://15.24.131.140:4567 stable
      2023-09-17 16:44:43 0 [Note] WSREP: PC protocol upgrade 0 -> 1
      2023-09-17 16:44:43 0 [Note] WSREP: Node 1c6062d8-a53e state prim
      2023-09-17 16:44:43 0 [Note] WSREP: view(view_id(PRIM,1c6062d8-a53e,27) memb {
      	1c6062d8-a53e,0
      	36974392-87e4,0
      	7e9b63be-a85e,0
      	b5e2166d-bb21,0
      	c1807c32-9da7,0
      	c7052230-8912,0
      } joined {
      } left {
      } partitioned {
      })
      2023-09-17 16:44:43 0 [Note] WSREP: save pc into disk
      2023-09-17 16:44:43 0 [Note] WSREP: clear restored view
      2023-09-17 16:44:43 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 4, memb_num = 6
      2023-09-17 16:44:43 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
      2023-09-17 16:44:43 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 72aa79fc-5536-11ee-bbe8-eea3d97d73a1
      2023-09-17 16:44:43 0 [Warning] WSREP: Action message in non-primary configuration from member 3
      2023-09-17 16:44:43 0 [Warning] WSREP: Action message in non-primary configuration from member 5
      2023-09-17 16:44:43 0 [Warning] WSREP: Action message in non-primary configuration from member 3
      2023-09-17 16:44:43 0 [Note] WSREP: STATE EXCHANGE: got state msg: 72aa79fc-5536-11ee-bbe8-eea3d97d73a1 from 0 (node-65)
      2023-09-17 16:44:43 0 [Note] WSREP: STATE EXCHANGE: got state msg: 72aa79fc-5536-11ee-bbe8-eea3d97d73a1 from 1 (node-64)
      2023-09-17 16:44:43 0 [Note] WSREP: STATE EXCHANGE: got state msg: 72aa79fc-5536-11ee-bbe8-eea3d97d73a1 from 2 (node-66)
      2023-09-17 16:44:43 0 [Note] WSREP: STATE EXCHANGE: got state msg: 72aa79fc-5536-11ee-bbe8-eea3d97d73a1 from 3 (node-141)
      2023-09-17 16:44:43 0 [Note] WSREP: STATE EXCHANGE: got state msg: 72aa79fc-5536-11ee-bbe8-eea3d97d73a1 from 4 (node-139)
      2023-09-17 16:44:43 0 [Note] WSREP: STATE EXCHANGE: got state msg: 72aa79fc-5536-11ee-bbe8-eea3d97d73a1 from 5 (node-14/usr/lib64/galera-4/libgalera_smm.so0)
      2023-09-17 16:44:43 0 [Note] WSREP: Quorum results:
      	version    = 6,
      	component  = PRIMARY,
      	conf_id    = 21,
      	members    = 5/6 (joined/total),
      	act_id     = 13217576,
      	last_appl. = 13217425,
      	protocols  = 2/10/4 (gcs/repl/appl),
      	vote policy= 0,
      	group UUID = 171a890e-7d0a-11ed-8263-6722050e329f
      2023-09-17 16:44:43 0 [Note] WSREP: Flow-control interval: [39, 39]
      2023-09-17 16:44:43 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 13217577)
      2023-09-17 16:44:43 2 [Note] WSREP: ####### processing CC 13217577, local, ordered
      2023-09-17 16:44:43 2 [Note] WSREP: Process first view: 171a890e-7d0a-11ed-8263-6722050e329f my uuid: c1807c32-8da4-11ed-9da7-6af9dc6e176c
      2023-09-17 16:44:43 2 [Note] WSREP: Server node-139 connected to cluster at position 171a890e-7d0a-11ed-8263-6722050e329f:13217577 with ID c1807c32-8da4-11ed-9da7-6af9dc6e176c
      2023-09-17 16:44:43 2 [Note] WSREP: Server status change disconnected -> connected
      2023-09-17 16:44:43 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      2023-09-17 16:44:43 2 [Note] WSREP: ####### My UUID: c1807c32-8da4-11ed-9da7-6af9dc6e176c
      2023-09-17 16:44:43 2 [Note] WSREP: Cert index reset to 00000000-0000-0000-0000-000000000000:-1 (proto: 10), state transfer needed: yes
      2023-09-17 16:44:43 0 [Note] WSREP: Service thread queue flushed.
      2023-09-17 16:44:43 2 [Note] WSREP: ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: -1
      2023-09-17 16:44:43 2 [Note] WSREP: State transfer required: 
      	Group state: 171a890e-7d0a-11ed-8263-6722050e329f:13217577
      	Local state: 171a890e-7d0a-11ed-8263-6722050e329f:13215589
      2023-09-17 16:44:43 2 [Note] WSREP: Server status change connected -> joiner
      2023-09-17 16:44:43 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      2023-09-17 16:44:43 0 [Note] WSREP: Joiner monitor thread started to monitor
      2023-09-17 16:44:43 0 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '15.24.131.139' --datadir '/rdc/data/mysql/' --parent 6373 --progress 0 --mysqld-args --wsrep_start_position=171a890e-7d0a-11ed-8263-6722050e329f:13215589'
      WSREP_SST: [INFO] rsync SST started on joiner (20230917 16:44:43.625)
      2023-09-17 16:44:43 2 [Note] WSREP: ####### IST uuid:171a890e-7d0a-11ed-8263-6722050e329f f: 13215590, l: 13217577, STRv: 3
      2023-09-17 16:44:43 2 [Note] WSREP: IST receiver addr using tcp://15.24.131.139:4568
      2023-09-17 16:44:43 2 [Note] WSREP: Prepared IST receiver for 13215590-13217577, listening at: tcp://15.24.131.139:4568
      2023-09-17 16:44:43 0 [Note] WSREP: Member 4.0 (node-139) requested state transfer from '*any*'. Selected 5.0 (node-14/usr/lib64/galera-4/libgalera_smm.so0)(SYNCED) as donor.
      2023-09-17 16:44:43 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 13217828)
      2023-09-17 16:44:43 2 [Note] WSREP: Requesting state transfer: success, donor: 5
      2023-09-17 16:44:44 0 [Note] WSREP: 5.0 (node-14/usr/lib64/galera-4/libgalera_smm.so0): State transfer to 4.0 (node-139) complete.
      2023-09-17 16:44:44 0 [Note] WSREP: Member 5.0 (node-14/usr/lib64/galera-4/libgalera_smm.so0) synced with group.
      WSREP_SST: [INFO] Galera co-ords from recovery: 171a890e-7d0a-11ed-8263-6722050e329f:13215589 0 (20230917 16:44:44.852)
      WSREP_SST: [INFO] rsync IST completed on joiner (20230917 16:44:44.853)
      WSREP_SST: [INFO] Joiner cleanup: rsync PID=6515, stunnel PID=0 (20230917 16:44:44.856)
      WSREP_SST: [INFO] Joiner cleanup done. (20230917 16:44:45.391)
      2023-09-17 16:44:45 3 [Note] WSREP: SST received
      2023-09-17 16:44:45 3 [Note] WSREP: Server status change joiner -> initializing
      2023-09-17 16:44:45 3 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      2023-09-17 16:44:45 0 [Note] InnoDB: Compressed tables use zlib 1.2.7
      2023-09-17 16:44:45 0 [Note] InnoDB: Number of transaction pools: 1
      2023-09-17 16:44:45 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
      2023-09-17 16:44:45 0 [Note] InnoDB: Using Linux native AIO
      2023-09-17 16:44:45 0 [Note] InnoDB: Initializing buffer pool, total size = 128.000MiB, chunk size = 2.000MiB
      2023-09-17 16:44:45 0 [Note] InnoDB: Completed initialization of buffer pool
      2023-09-17 16:44:45 0 [Note] InnoDB: File system buffers for log disabled (block size=512 bytes)
      2023-09-17 16:44:45 0 [Note] InnoDB: 128 rollback segments are active.
      2023-09-17 16:44:45 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...
      2023-09-17 16:44:45 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.
      2023-09-17 16:44:45 0 [Note] InnoDB: log sequence number 886430924614; transaction id 27550200
      2023-09-17 16:44:45 0 [Note] InnoDB: Loading buffer pool(s) from /rdc/data/mysql/ib_buffer_pool
      2023-09-17 16:44:45 0 [Note] Plugin 'FEEDBACK' is disabled.
      2023-09-17 16:44:45 0 [Note] Server socket created on IP: '0.0.0.0'.
      2023-09-17 16:44:45 0 [Note] InnoDB: Buffer pool(s) load completed at 230917 16:44:45
      2023-09-17 16:44:45 0 [Note] WSREP: wsrep_init_schema_and_SR (nil)
      2023-09-17 16:44:45 0 [Note] WSREP: Server initialized
      2023-09-17 16:44:45 0 [Note] WSREP: Server status change initializing -> initialized
      2023-09-17 16:44:45 0 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      2023-09-17 16:44:45 3 [Note] WSREP: Recovered position from storage: 171a890e-7d0a-11ed-8263-6722050e329f:13215589
      2023-09-17 16:44:45 7 [Note] WSREP: Starting applier thread 7
      2023-09-17 16:44:45 6 [Note] WSREP: Starting applier thread 6
      2023-09-17 16:44:45 8 [Note] WSREP: Starting applier thread 8
      2023-09-17 16:44:45 3 [Note] WSREP: Recovered view from SST:
        id: 171a890e-7d0a-11ed-8263-6722050e329f:6338746
        status: primary
        protocol_version: 4
        capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
        final: no
        own_index: -1
        members(6):
      	0: 1c6062d8-8da6-11ed-a53e-c6e69dac5e95, node-65
      	1: 36974392-8db9-11ed-87e4-9f6b63483dcb, node-64
      	2: 7e9b63be-8da6-11ed-a85e-96bb23734d31, node-66
      	3: b5e2166d-8da4-11ed-bb21-17668aac5515, node-141
      	4: c1807c32-8da4-11ed-9da6-6af9dc6e176c, node-139
      	5: c7052230-8da4-11ed-8912-f2d54f607f5b, node-14/usr/lib64/galera-4/libg
       
      2023-09-17 16:44:45 3 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      2023-09-17 16:44:45 9 [Note] WSREP: Recovered cluster id 171a890e-7d0a-11ed-8263-6722050e329f
      2023-09-17 16:44:45 3 [Note] WSREP: SST received: 171a890e-7d0a-11ed-8263-6722050e329f:13215589
      2023-09-17 16:44:45 3 [Note] WSREP: SST succeeded for position 171a890e-7d0a-11ed-8263-6722050e329f:13215589
      2023-09-17 16:44:45 2 [Note] WSREP: Installed new state from SST: 171a890e-7d0a-11ed-8263-6722050e329f:13215589
      2023-09-17 16:44:45 0 [Note] WSREP: Joiner monitor thread ended with total time 2 sec
      2023-09-17 16:44:45 0 [Note] /usr/sbin/mariadbd: ready for connections.
      Version: '10.10.2-MariaDB'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MariaDB Server
      2023-09-17 16:44:45 2 [Note] WSREP: Receiving IST: 1988 writesets, seqnos 13215590-13217577
      2023-09-17 16:44:45 0 [Note] WSREP: ####### IST applying starts with 13215590
      2023-09-17 16:44:45 0 [Note] WSREP: ####### IST current seqno initialized to 13215590
      2023-09-17 16:44:45 0 [Note] WSREP: Receiving IST...  0.0% (   0/1988 events) complete.
      2023-09-17 16:44:45 0 [Note] WSREP: Service thread queue flushed.
      2023-09-17 16:44:45 0 [Note] WSREP: ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:13215589, protocol version: 5
      2023-09-17 16:44:45 0 [Note] WSREP: REPL Protocols: 10 (5)
      2023-09-17 16:44:45 0 [Note] WSREP: ####### Adjusting cert position: 13215589 -> 13215772
      2023-09-17 16:44:45 0 [Note] WSREP: Service thread queue flushed.
      2023-09-17 16:44:45 0 [Note] WSREP: Lowest cert index boundary for CC from ist: 13215772
      2023-09-17 16:44:45 0 [Note] WSREP: Min available from gcache for CC from ist: 13020521
      2023-09-17 16:44:45 6 [Note] WSREP: ================================================
      View:
        id: 171a890e-7d0a-11ed-8263-6722050e329f:13215772
        status: primary
        protocol_version: 4
        capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
        final: no
        own_index: -1
        members(5):
      	0: 1c6062d8-8da6-11ed-a53e-c6e69dac5e95, node-65
      	1: 36974392-8db9-11ed-87e4-9f6b63483dcb, node-64
      	2: 7e9b63be-8da6-11ed-a85e-96bb23734d31, node-66
      	3: b5e2166d-8da4-11ed-bb21-17668aac5515, node-141
      	4: c7052230-8da4-11ed-8912-f2d54f607f5b, node-14/usr/lib64/galera-4/libg
      =================================================
      2023-09-17 16:44:45 6 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      2023-09-17 16:44:45 0 [Note] WSREP: IST preload starting at 13217426
      2023-09-17 16:44:45 0 [Note] WSREP: REPL Protocols: 10 (5)
      2023-09-17 16:44:45 0 [Note] WSREP: ####### Adjusting cert position: 13217576 -> 13217577
      2023-09-17 16:44:45 0 [Note] WSREP: Service thread queue flushed.
      2023-09-17 16:44:45 0 [Note] WSREP: Lowest cert index boundary for CC from ist: 13217426
      2023-09-17 16:44:45 0 [Note] WSREP: Min available from gcache for CC from ist: 13021525
      2023-09-17 16:44:45 0 [Note] WSREP: Receiving IST...100.0% (1988/1988 events) complete.
      2023-09-17 16:44:45 8 [Note] WSREP: ================================================
      View:
        id: 171a890e-7d0a-11ed-8263-6722050e329f:13217577
        status: primary
        protocol_version: 4
        capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
        final: no
        own_index: 4
        members(6):
      	0: 1c6062d8-8da6-11ed-a53e-c6e69dac5e95, node-65
      	1: 36974392-8db9-11ed-87e4-9f6b63483dcb, node-64
      	2: 7e9b63be-8da6-11ed-a85e-96bb23734d31, node-66
      	3: b5e2166d-8da4-11ed-bb21-17668aac5515, node-141
      	4: c1807c32-8da4-11ed-9da7-6af9dc6e176c, node-139
      	5: c7052230-8da4-11ed-8912-f2d54f607f5b, node-14/usr/lib64/galera-4/libg
      =================================================
      2023-09-17 16:44:45 8 [Note] WSREP: Server status change initialized -> joined
      2023-09-17 16:44:45 8 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      2023-09-17 16:44:45 8 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      2023-09-17 16:44:45 2 [Note] WSREP: Draining apply monitors after IST up to 13217577
      2023-09-17 16:44:45 2 [Note] WSREP: IST received: 171a890e-7d0a-11ed-8263-6722050e329f:13217577
      2023-09-17 16:44:45 2 [Note] WSREP: Lowest cert index boundary for CC from sst: 13217426
      2023-09-17 16:44:45 2 [Note] WSREP: Min available from gcache for CC from sst: 13021525
      2023-09-17 16:44:45 0 [Note] WSREP: 4.0 (node-139): State transfer from 5.0 (node-14/usr/lib64/galera-4/libgalera_smm.so0) complete.
      2023-09-17 16:44:45 0 [Note] WSREP: Shifting JOINER -> JOINED (TO: 13220600)
      2023-09-17 16:44:45 0 [Note] WSREP: Processing event queue:...  0.0% (   0/2836 events) complete.
      2023-09-17 16:44:45 0 [Note] WSREP: (c1807c32-9da7, 'tcp://0.0.0.0:4567') turning message relay requesting off
      2023-09-17 16:44:46 0 [Note] WSREP: Member 4.0 (node-139) synced with group.
      2023-09-17 16:44:46 0 [Note] WSREP: Processing event queue:...100.0% (2969/2969 events) complete.
      2023-09-17 16:44:46 0 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 13220730)
      2023-09-17 16:44:46 2 [Note] WSREP: Server node-139 synced with group
      2023-09-17 16:44:46 2 [Note] WSREP: Server status change joined -> synced
      2023-09-17 16:44:46 2 [Note] WSREP: Synchronized with group, ready for connections
      2023-09-17 16:44:46 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      2023-09-17 16:45:06 26 [Warning] IP address '15.24.131.140' could not be resolved: Name or service not known
      2023-09-17 16:59:08 877 [Warning] Aborted connection 877 to db: 'rdc-cloud' user: 'rdccloud' host: '15.24.131.140' (Got an error reading communication packets)
      mariadbd: /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.10.2/wsrep-lib/include/wsrep/client_state.hpp:668: int wsrep::client_state::bf_abort(wsrep::seqno): Assertion `mode_ == m_local || transaction_.is_streaming()' failed.
      230917 17:00:34 [ERROR] mysqld got signal 6 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.

      Attachments

        Issue Links

          Activity

            People

              seppo Seppo Jaakola
              jixian8784 wangpengyu
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.