Details
-
Bug
-
Status: Closed (View Workflow)
-
Critical
-
Resolution: Incomplete
-
10.10.2
-
None
-
mariadb galera cluster 6 nodes.
Description
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
Server version: 10.10.2-MariaDB
|
key_buffer_size=134217728
|
read_buffer_size=131072
|
max_used_connections=167
|
max_threads=2002
|
thread_count=172
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 4540309 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
Thread pointer: 0x7f232c0009b8
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x7f23b4092ca0 thread_stack 0x49000
|
/usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x55f6ad4bfdfe]
|
/usr/sbin/mariadbd(handle_fatal_signal+0x307)[0x55f6acf25cc7]
|
sigaction.c:0(__restore_rt)[0x7f23bacdb5d0]
|
:0(__GI_raise)[0x7f23ba128207]
|
:0(__GI_abort)[0x7f23ba1298f8]
|
:0(__assert_fail_base)[0x7f23ba121026]
|
:0(__GI___assert_fail)[0x7f23ba1210d2]
|
/usr/sbin/mariadbd(_Z14wsrep_bf_abortP3THDS0_+0x622)[0x55f6ad1ce852]
|
/usr/sbin/mariadbd(wsrep_thd_bf_abort+0x19)[0x55f6ad1d5259]
|
/usr/sbin/mariadbd(+0xd13e29)[0x55f6ad1f6e29]
|
/usr/sbin/mariadbd(+0x6a6ff1)[0x55f6acb89ff1]
|
/usr/sbin/mariadbd(+0xd615e2)[0x55f6ad2445e2]
|
/usr/sbin/mariadbd(+0xde3a2f)[0x55f6ad2c6a2f]
|
/usr/sbin/mariadbd(+0xde606d)[0x55f6ad2c906d]
|
wsrep/client_state.hpp:679(wsrep_bf_abort(THD*, THD*))[0x55f6ad20293e]
|
sql/service_wsrep.cc:228(wsrep_thd_bf_abort)[0x55f6acf3349f]
|
handler/ha_innodb.cc:18635(wsrep_notify_status(wsrep::server_state::state, wsrep::view const*))[0x55f6ad058125]
|
/usr/sbin/mariadbd(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x116)[0x55f6ad058626]
|
/usr/sbin/mariadbd(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x2d6)[0x55f6ad04cc06]
|
lock/lock0lock.cc:1004(??)[0x55f6ad044607]
|
lock/lock0lock.cc:1772(wsrep_notify_status(wsrep::server_state::state, wsrep::view const*))[0x55f6ad1cd7fd]
|
row/row0mysql.cc:682(void std::__introsort_loop<unsigned char**, long>(unsigned char**, unsigned char**, long))[0x55f6ad1b2ed0]
|
row/row0mysql.cc:1324(void std::__introsort_loop<unsigned char**, long>(unsigned char**, unsigned char**, long))[0x55f6ad1b2fc3]
|
handler/ha_innodb.cc:7843(wsrep_notify_status(wsrep::server_state::state, wsrep::view const*))[0x55f6ad5620ff]
|
sql/handler.cc:7561(handler::ha_write_row(unsigned char const*))[0x55f6ad562e25]
|
sql/log_event_server.cc:7701(Rows_log_event::write_row(rpl_group_info*, bool))[0x55f6ad576758]
|
src/trx_handle.cpp:392(galera::TrxHandleSlave::apply(void*, wsrep_cb_status (*)(void*, wsrep_ws_handle const*, unsigned int, wsrep_buf const*, wsrep_trx_meta const*, bool*), wsrep_trx_meta const&, bool&))[0x7f23b6d645f0]
|
src/replicator_smm.cpp:516(galera::ReplicatorSMM::apply_trx(void*, galera::TrxHandleSlave&))[0x7f23b6d6e250]
|
src/replicator_smm.cpp:2156(galera::ReplicatorSMM::process_trx(void*, boost::shared_ptr<galera::TrxHandleSlave> const&))[0x7f23b6d747b1]
|
src/gcs_action_source.cpp:63(galera::GcsActionSource::process_writeset(void*, gcs_action const&, bool&))[0x7f23b6da0f10]
|
src/gcs_action_source.cpp:110(galera::GcsActionSource::dispatch(void*, gcs_action const&, bool&))[0x7f23b6da1d32]
|
src/gcs_action_source.cpp:29(~Release)[0x7f23b6da25b4]
|
src/replicator_smm.cpp:402(galera::ReplicatorSMM::async_recv(void*))[0x7f23b6d7567c]
|
src/wsrep_provider.cpp:291(galera_recv)[0x7f23b6d56408]
|
/usr/sbin/mariadbd(_ZN5wsrep18wsrep_provider_v2611run_applierEPNS_21high_priority_serviceE+0xe)[0x55f6ad576e8e]
|
/usr/sbin/mariadbd(+0xcec322)[0x55f6ad1cf322]
|
/usr/sbin/mariadbd(_Z15start_wsrep_THDPv+0x2c8)[0x55f6ad1bf7c8]
|
/usr/sbin/mariadbd(+0xc6b97c)[0x55f6ad14e97c]
|
pthread_create.c:0(start_thread)[0x7f23bacd3dd5]
|
??:0(__clone)[0x7f23ba1efead]
|
|
Trying to get some variables.
|
Some pointers may be invalid and cause the dump to abort.
|
Query (0x7f239f7f4ce3): insert into t_zxg_group_stock (group_stock_id, group_id, sort_no,
|
stock_code, un_convertable_source, collect_date,collect_price,create_by, update_by)
|
values
|
|
(null,
|
7523495,
|
1,
|
'600400.sh|4353',
|
'zz',
|
'2023-09-17',
|
100.00,
|
'72100008089',
|
'72100008089')
|
,
|
(null,
|
7523495,
|
2,
|
'600300.sh|4353',
|
'zz',
|
'2023-09-17',
|
100.00,
|
'72100008089',
|
'72100008089')
|
,
|
(null,
|
7523495,
|
3,
|
'600301.sh|4353',
|
'zz',
|
'2023-09-17',
|
100.00,
|
'72100008089',
|
'72100008089')
|
|
Connection ID (thread ID): 6
|
Status: NOT_KILLED
|
|
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off
|
|
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
|
information that should help you find out what is causing the crash.
|
Writing a core file...
|
Working directory at /rdc/data/mysql
|
Resource Limits:
|
Limit Soft Limit Hard Limit Units
|
Max cpu time unlimited unlimited seconds
|
Max file size unlimited unlimited bytes
|
Max data size unlimited unlimited bytes
|
Max stack size 8388608 unlimited bytes
|
Max core file size 0 unlimited bytes
|
Max resident set unlimited unlimited bytes
|
Max processes 96340 96340 processes
|
Max open files 32768 32768 files
|
Max locked memory 65536 65536 bytes
|
Max address space unlimited unlimited bytes
|
Max file locks unlimited unlimited locks
|
Max pending signals 96340 96340 signals
|
Max msgqueue size 819200 819200 bytes
|
Max nice priority 0 0
|
Max realtime priority 0 0
|
Max realtime timeout unlimited unlimited us
|
Core pattern: core
|
Kernel version: Linux version 3.10.0-957.el7.x86_64 (mockbuild@x86-040.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-36) (GCC) ) #1 SMP Thu Oct 4 20:48:51 UTC 2018
|
2023-09-17 16:44:42 0 [Note] WSREP: Loading provider /usr/lib64/galera-4/libgalera_smm.so initial position: 171a890e-7d0a-11ed-8263-6722050e329f:13215589
|
2023-09-17 16:44:42 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/galera-4/libgalera_smm.so'
|
2023-09-17 16:44:42 0 [Note] WSREP: wsrep_load(): Galera 26.4.13(rfe497ae) by Codership Oy <info@codership.com> loaded successfully.
|
2023-09-17 16:44:42 0 [Note] WSREP: Initializing allowlist service v1
|
2023-09-17 16:44:42 0 [Note] WSREP: CRC-32C: using 64-bit x86 acceleration.
|
2023-09-17 16:44:42 0 [Note] WSREP: Found saved state: 171a890e-7d0a-11ed-8263-6722050e329f:-1, safe_to_bootstrap: 0
|
2023-09-17 16:44:42 0 [Note] WSREP: GCache DEBUG: opened preamble:
|
Version: 2
|
UUID: 171a890e-7d0a-11ed-8263-6722050e329f
|
Seqno: -1 - -1
|
Offset: -1
|
Synced: 0
|
2023-09-17 16:44:42 0 [Note] WSREP: Recovering GCache ring buffer: version: 2, UUID: 171a890e-7d0a-11ed-8263-6722050e329f, offset: -1
|
2023-09-17 16:44:42 0 [Note] WSREP: GCache::RingBuffer initial scan... 0.0% ( 0/314572824 bytes) complete.
|
2023-09-17 16:44:42 0 [Note] WSREP: GCache::RingBuffer initial scan...100.0% (314572824/314572824 bytes) complete.
|
2023-09-17 16:44:42 0 [Note] WSREP: Recovering GCache ring buffer: found gapless sequence 13018942-13215593
|
2023-09-17 16:44:42 0 [Note] WSREP: GCache::RingBuffer unused buffers scan... 0.0% ( 0/314303296 bytes) complete.
|
2023-09-17 16:44:42 0 [Note] WSREP: Recovering GCache ring buffer: found 0/196652 locked buffers
|
2023-09-17 16:44:42 0 [Note] WSREP: Recovering GCache ring buffer: free space: 269504/314572800
|
2023-09-17 16:44:42 0 [Note] WSREP: GCache::RingBuffer unused buffers scan...100.0% (314303296/314303296 bytes) complete.
|
2023-09-17 16:44:42 0 [Note] WSREP: Passing config to GCS: base_dir = /rdc/data/mysql/; base_host = 15.24.131.139; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /rdc/data/mysql/; gcache.keep_pages_size = 0; gcache.keep_plaintext_size = 300M; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 300M; gcache.recover = yes; gcache.size = 300M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.fc_single_primary = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment =
|
2023-09-17 16:44:42 0 [Note] WSREP: Service thread queue flushed.
|
2023-09-17 16:44:42 0 [Note] WSREP: ####### Assign initial position for certification: 171a890e-7d0a-11ed-8263-6722050e329f:13215589, protocol version: -1
|
2023-09-17 16:44:42 0 [Note] WSREP: Start replication
|
2023-09-17 16:44:42 0 [Note] WSREP: Connecting with bootstrap option: 0
|
2023-09-17 16:44:42 0 [Note] WSREP: Setting GCS initial position to 171a890e-7d0a-11ed-8263-6722050e329f:13215589
|
2023-09-17 16:44:42 0 [Note] WSREP: protonet asio version 0
|
2023-09-17 16:44:42 0 [Note] WSREP: Using CRC-32C for message checksums.
|
2023-09-17 16:44:42 0 [Note] WSREP: backend: asio
|
2023-09-17 16:44:42 0 [Note] WSREP: gcomm thread scheduling priority set to other:0
|
2023-09-17 16:44:42 0 [Note] WSREP: restore pc from disk successfully
|
2023-09-17 16:44:42 0 [Note] WSREP: GMCast version 0
|
2023-09-17 16:44:42 0 [Note] WSREP: (c1807c32-9da7, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
|
2023-09-17 16:44:42 0 [Note] WSREP: (c1807c32-9da7, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
|
2023-09-17 16:44:42 0 [Note] WSREP: EVS version 1
|
2023-09-17 16:44:42 0 [Note] WSREP: gcomm: connecting to group 'rdc-cloud', peer '15.24.131.139:,15.24.131.140:,15.24.131.141:,10.35.163.64:,10.35.163.65:,10.35.163.66:'
|
2023-09-17 16:44:42 0 [Note] WSREP: (c1807c32-9da7, 'tcp://0.0.0.0:4567') Found matching local endpoint for a connection, blacklisting address tcp://15.24.131.139:4567
|
2023-09-17 16:44:42 0 [Note] WSREP: (c1807c32-9da7, 'tcp://0.0.0.0:4567') connection established to c7052230-8912 tcp://15.24.131.140:4567
|
2023-09-17 16:44:42 0 [Note] WSREP: (c1807c32-9da7, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
|
2023-09-17 16:44:42 0 [Note] WSREP: (c1807c32-9da7, 'tcp://0.0.0.0:4567') connection established to b5e2166d-bb21 tcp://15.24.131.141:4567
|
2023-09-17 16:44:42 0 [Note] WSREP: (c1807c32-9da7, 'tcp://0.0.0.0:4567') connection established to 36974392-87e4 tcp://10.35.163.64:4567
|
2023-09-17 16:44:42 0 [Note] WSREP: (c1807c32-9da7, 'tcp://0.0.0.0:4567') connection established to 1c6062d8-a53e tcp://10.35.163.65:4567
|
2023-09-17 16:44:42 0 [Note] WSREP: (c1807c32-9da7, 'tcp://0.0.0.0:4567') connection established to 7e9b63be-a85e tcp://10.35.163.66:4567
|
2023-09-17 16:44:42 0 [Note] WSREP: gcomm: connected
|
2023-09-17 16:44:42 0 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
|
2023-09-17 16:44:42 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
|
2023-09-17 16:44:42 0 [Note] WSREP: Opened channel 'rdc-cloud'
|
2023-09-17 16:44:42 1 [Note] WSREP: Starting rollbacker thread 1
|
2023-09-17 16:44:42 2 [Note] WSREP: Starting applier thread 2
|
2023-09-17 16:44:43 0 [Note] WSREP: EVS version upgrade 0 -> 1
|
2023-09-17 16:44:43 0 [Note] WSREP: declaring 1c6062d8-a53e at tcp://10.35.163.65:4567 stable
|
2023-09-17 16:44:43 0 [Note] WSREP: declaring 36974392-87e4 at tcp://10.35.163.64:4567 stable
|
2023-09-17 16:44:43 0 [Note] WSREP: declaring 7e9b63be-a85e at tcp://10.35.163.66:4567 stable
|
2023-09-17 16:44:43 0 [Note] WSREP: declaring b5e2166d-bb21 at tcp://15.24.131.141:4567 stable
|
2023-09-17 16:44:43 0 [Note] WSREP: declaring c7052230-8912 at tcp://15.24.131.140:4567 stable
|
2023-09-17 16:44:43 0 [Note] WSREP: PC protocol upgrade 0 -> 1
|
2023-09-17 16:44:43 0 [Note] WSREP: Node 1c6062d8-a53e state prim
|
2023-09-17 16:44:43 0 [Note] WSREP: view(view_id(PRIM,1c6062d8-a53e,27) memb {
|
1c6062d8-a53e,0
|
36974392-87e4,0
|
7e9b63be-a85e,0
|
b5e2166d-bb21,0
|
c1807c32-9da7,0
|
c7052230-8912,0
|
} joined {
|
} left {
|
} partitioned {
|
})
|
2023-09-17 16:44:43 0 [Note] WSREP: save pc into disk
|
2023-09-17 16:44:43 0 [Note] WSREP: clear restored view
|
2023-09-17 16:44:43 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 4, memb_num = 6
|
2023-09-17 16:44:43 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
|
2023-09-17 16:44:43 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 72aa79fc-5536-11ee-bbe8-eea3d97d73a1
|
2023-09-17 16:44:43 0 [Warning] WSREP: Action message in non-primary configuration from member 3
|
2023-09-17 16:44:43 0 [Warning] WSREP: Action message in non-primary configuration from member 5
|
2023-09-17 16:44:43 0 [Warning] WSREP: Action message in non-primary configuration from member 3
|
2023-09-17 16:44:43 0 [Note] WSREP: STATE EXCHANGE: got state msg: 72aa79fc-5536-11ee-bbe8-eea3d97d73a1 from 0 (node-65)
|
2023-09-17 16:44:43 0 [Note] WSREP: STATE EXCHANGE: got state msg: 72aa79fc-5536-11ee-bbe8-eea3d97d73a1 from 1 (node-64)
|
2023-09-17 16:44:43 0 [Note] WSREP: STATE EXCHANGE: got state msg: 72aa79fc-5536-11ee-bbe8-eea3d97d73a1 from 2 (node-66)
|
2023-09-17 16:44:43 0 [Note] WSREP: STATE EXCHANGE: got state msg: 72aa79fc-5536-11ee-bbe8-eea3d97d73a1 from 3 (node-141)
|
2023-09-17 16:44:43 0 [Note] WSREP: STATE EXCHANGE: got state msg: 72aa79fc-5536-11ee-bbe8-eea3d97d73a1 from 4 (node-139)
|
2023-09-17 16:44:43 0 [Note] WSREP: STATE EXCHANGE: got state msg: 72aa79fc-5536-11ee-bbe8-eea3d97d73a1 from 5 (node-14/usr/lib64/galera-4/libgalera_smm.so0)
|
2023-09-17 16:44:43 0 [Note] WSREP: Quorum results:
|
version = 6,
|
component = PRIMARY,
|
conf_id = 21,
|
members = 5/6 (joined/total),
|
act_id = 13217576,
|
last_appl. = 13217425,
|
protocols = 2/10/4 (gcs/repl/appl),
|
vote policy= 0,
|
group UUID = 171a890e-7d0a-11ed-8263-6722050e329f
|
2023-09-17 16:44:43 0 [Note] WSREP: Flow-control interval: [39, 39]
|
2023-09-17 16:44:43 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 13217577)
|
2023-09-17 16:44:43 2 [Note] WSREP: ####### processing CC 13217577, local, ordered
|
2023-09-17 16:44:43 2 [Note] WSREP: Process first view: 171a890e-7d0a-11ed-8263-6722050e329f my uuid: c1807c32-8da4-11ed-9da7-6af9dc6e176c
|
2023-09-17 16:44:43 2 [Note] WSREP: Server node-139 connected to cluster at position 171a890e-7d0a-11ed-8263-6722050e329f:13217577 with ID c1807c32-8da4-11ed-9da7-6af9dc6e176c
|
2023-09-17 16:44:43 2 [Note] WSREP: Server status change disconnected -> connected
|
2023-09-17 16:44:43 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
|
2023-09-17 16:44:43 2 [Note] WSREP: ####### My UUID: c1807c32-8da4-11ed-9da7-6af9dc6e176c
|
2023-09-17 16:44:43 2 [Note] WSREP: Cert index reset to 00000000-0000-0000-0000-000000000000:-1 (proto: 10), state transfer needed: yes
|
2023-09-17 16:44:43 0 [Note] WSREP: Service thread queue flushed.
|
2023-09-17 16:44:43 2 [Note] WSREP: ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: -1
|
2023-09-17 16:44:43 2 [Note] WSREP: State transfer required:
|
Group state: 171a890e-7d0a-11ed-8263-6722050e329f:13217577
|
Local state: 171a890e-7d0a-11ed-8263-6722050e329f:13215589
|
2023-09-17 16:44:43 2 [Note] WSREP: Server status change connected -> joiner
|
2023-09-17 16:44:43 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
|
2023-09-17 16:44:43 0 [Note] WSREP: Joiner monitor thread started to monitor
|
2023-09-17 16:44:43 0 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '15.24.131.139' --datadir '/rdc/data/mysql/' --parent 6373 --progress 0 --mysqld-args --wsrep_start_position=171a890e-7d0a-11ed-8263-6722050e329f:13215589'
|
WSREP_SST: [INFO] rsync SST started on joiner (20230917 16:44:43.625)
|
2023-09-17 16:44:43 2 [Note] WSREP: ####### IST uuid:171a890e-7d0a-11ed-8263-6722050e329f f: 13215590, l: 13217577, STRv: 3
|
2023-09-17 16:44:43 2 [Note] WSREP: IST receiver addr using tcp://15.24.131.139:4568
|
2023-09-17 16:44:43 2 [Note] WSREP: Prepared IST receiver for 13215590-13217577, listening at: tcp://15.24.131.139:4568
|
2023-09-17 16:44:43 0 [Note] WSREP: Member 4.0 (node-139) requested state transfer from '*any*'. Selected 5.0 (node-14/usr/lib64/galera-4/libgalera_smm.so0)(SYNCED) as donor.
|
2023-09-17 16:44:43 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 13217828)
|
2023-09-17 16:44:43 2 [Note] WSREP: Requesting state transfer: success, donor: 5
|
2023-09-17 16:44:44 0 [Note] WSREP: 5.0 (node-14/usr/lib64/galera-4/libgalera_smm.so0): State transfer to 4.0 (node-139) complete.
|
2023-09-17 16:44:44 0 [Note] WSREP: Member 5.0 (node-14/usr/lib64/galera-4/libgalera_smm.so0) synced with group.
|
WSREP_SST: [INFO] Galera co-ords from recovery: 171a890e-7d0a-11ed-8263-6722050e329f:13215589 0 (20230917 16:44:44.852)
|
WSREP_SST: [INFO] rsync IST completed on joiner (20230917 16:44:44.853)
|
WSREP_SST: [INFO] Joiner cleanup: rsync PID=6515, stunnel PID=0 (20230917 16:44:44.856)
|
WSREP_SST: [INFO] Joiner cleanup done. (20230917 16:44:45.391)
|
2023-09-17 16:44:45 3 [Note] WSREP: SST received
|
2023-09-17 16:44:45 3 [Note] WSREP: Server status change joiner -> initializing
|
2023-09-17 16:44:45 3 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
|
2023-09-17 16:44:45 0 [Note] InnoDB: Compressed tables use zlib 1.2.7
|
2023-09-17 16:44:45 0 [Note] InnoDB: Number of transaction pools: 1
|
2023-09-17 16:44:45 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
|
2023-09-17 16:44:45 0 [Note] InnoDB: Using Linux native AIO
|
2023-09-17 16:44:45 0 [Note] InnoDB: Initializing buffer pool, total size = 128.000MiB, chunk size = 2.000MiB
|
2023-09-17 16:44:45 0 [Note] InnoDB: Completed initialization of buffer pool
|
2023-09-17 16:44:45 0 [Note] InnoDB: File system buffers for log disabled (block size=512 bytes)
|
2023-09-17 16:44:45 0 [Note] InnoDB: 128 rollback segments are active.
|
2023-09-17 16:44:45 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...
|
2023-09-17 16:44:45 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.
|
2023-09-17 16:44:45 0 [Note] InnoDB: log sequence number 886430924614; transaction id 27550200
|
2023-09-17 16:44:45 0 [Note] InnoDB: Loading buffer pool(s) from /rdc/data/mysql/ib_buffer_pool
|
2023-09-17 16:44:45 0 [Note] Plugin 'FEEDBACK' is disabled.
|
2023-09-17 16:44:45 0 [Note] Server socket created on IP: '0.0.0.0'.
|
2023-09-17 16:44:45 0 [Note] InnoDB: Buffer pool(s) load completed at 230917 16:44:45
|
2023-09-17 16:44:45 0 [Note] WSREP: wsrep_init_schema_and_SR (nil)
|
2023-09-17 16:44:45 0 [Note] WSREP: Server initialized
|
2023-09-17 16:44:45 0 [Note] WSREP: Server status change initializing -> initialized
|
2023-09-17 16:44:45 0 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
|
2023-09-17 16:44:45 3 [Note] WSREP: Recovered position from storage: 171a890e-7d0a-11ed-8263-6722050e329f:13215589
|
2023-09-17 16:44:45 7 [Note] WSREP: Starting applier thread 7
|
2023-09-17 16:44:45 6 [Note] WSREP: Starting applier thread 6
|
2023-09-17 16:44:45 8 [Note] WSREP: Starting applier thread 8
|
2023-09-17 16:44:45 3 [Note] WSREP: Recovered view from SST:
|
id: 171a890e-7d0a-11ed-8263-6722050e329f:6338746
|
status: primary
|
protocol_version: 4
|
capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
|
final: no
|
own_index: -1
|
members(6):
|
0: 1c6062d8-8da6-11ed-a53e-c6e69dac5e95, node-65
|
1: 36974392-8db9-11ed-87e4-9f6b63483dcb, node-64
|
2: 7e9b63be-8da6-11ed-a85e-96bb23734d31, node-66
|
3: b5e2166d-8da4-11ed-bb21-17668aac5515, node-141
|
4: c1807c32-8da4-11ed-9da6-6af9dc6e176c, node-139
|
5: c7052230-8da4-11ed-8912-f2d54f607f5b, node-14/usr/lib64/galera-4/libg
|
|
2023-09-17 16:44:45 3 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
|
2023-09-17 16:44:45 9 [Note] WSREP: Recovered cluster id 171a890e-7d0a-11ed-8263-6722050e329f
|
2023-09-17 16:44:45 3 [Note] WSREP: SST received: 171a890e-7d0a-11ed-8263-6722050e329f:13215589
|
2023-09-17 16:44:45 3 [Note] WSREP: SST succeeded for position 171a890e-7d0a-11ed-8263-6722050e329f:13215589
|
2023-09-17 16:44:45 2 [Note] WSREP: Installed new state from SST: 171a890e-7d0a-11ed-8263-6722050e329f:13215589
|
2023-09-17 16:44:45 0 [Note] WSREP: Joiner monitor thread ended with total time 2 sec
|
2023-09-17 16:44:45 0 [Note] /usr/sbin/mariadbd: ready for connections.
|
Version: '10.10.2-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server
|
2023-09-17 16:44:45 2 [Note] WSREP: Receiving IST: 1988 writesets, seqnos 13215590-13217577
|
2023-09-17 16:44:45 0 [Note] WSREP: ####### IST applying starts with 13215590
|
2023-09-17 16:44:45 0 [Note] WSREP: ####### IST current seqno initialized to 13215590
|
2023-09-17 16:44:45 0 [Note] WSREP: Receiving IST... 0.0% ( 0/1988 events) complete.
|
2023-09-17 16:44:45 0 [Note] WSREP: Service thread queue flushed.
|
2023-09-17 16:44:45 0 [Note] WSREP: ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:13215589, protocol version: 5
|
2023-09-17 16:44:45 0 [Note] WSREP: REPL Protocols: 10 (5)
|
2023-09-17 16:44:45 0 [Note] WSREP: ####### Adjusting cert position: 13215589 -> 13215772
|
2023-09-17 16:44:45 0 [Note] WSREP: Service thread queue flushed.
|
2023-09-17 16:44:45 0 [Note] WSREP: Lowest cert index boundary for CC from ist: 13215772
|
2023-09-17 16:44:45 0 [Note] WSREP: Min available from gcache for CC from ist: 13020521
|
2023-09-17 16:44:45 6 [Note] WSREP: ================================================
|
View:
|
id: 171a890e-7d0a-11ed-8263-6722050e329f:13215772
|
status: primary
|
protocol_version: 4
|
capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
|
final: no
|
own_index: -1
|
members(5):
|
0: 1c6062d8-8da6-11ed-a53e-c6e69dac5e95, node-65
|
1: 36974392-8db9-11ed-87e4-9f6b63483dcb, node-64
|
2: 7e9b63be-8da6-11ed-a85e-96bb23734d31, node-66
|
3: b5e2166d-8da4-11ed-bb21-17668aac5515, node-141
|
4: c7052230-8da4-11ed-8912-f2d54f607f5b, node-14/usr/lib64/galera-4/libg
|
=================================================
|
2023-09-17 16:44:45 6 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
|
2023-09-17 16:44:45 0 [Note] WSREP: IST preload starting at 13217426
|
2023-09-17 16:44:45 0 [Note] WSREP: REPL Protocols: 10 (5)
|
2023-09-17 16:44:45 0 [Note] WSREP: ####### Adjusting cert position: 13217576 -> 13217577
|
2023-09-17 16:44:45 0 [Note] WSREP: Service thread queue flushed.
|
2023-09-17 16:44:45 0 [Note] WSREP: Lowest cert index boundary for CC from ist: 13217426
|
2023-09-17 16:44:45 0 [Note] WSREP: Min available from gcache for CC from ist: 13021525
|
2023-09-17 16:44:45 0 [Note] WSREP: Receiving IST...100.0% (1988/1988 events) complete.
|
2023-09-17 16:44:45 8 [Note] WSREP: ================================================
|
View:
|
id: 171a890e-7d0a-11ed-8263-6722050e329f:13217577
|
status: primary
|
protocol_version: 4
|
capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
|
final: no
|
own_index: 4
|
members(6):
|
0: 1c6062d8-8da6-11ed-a53e-c6e69dac5e95, node-65
|
1: 36974392-8db9-11ed-87e4-9f6b63483dcb, node-64
|
2: 7e9b63be-8da6-11ed-a85e-96bb23734d31, node-66
|
3: b5e2166d-8da4-11ed-bb21-17668aac5515, node-141
|
4: c1807c32-8da4-11ed-9da7-6af9dc6e176c, node-139
|
5: c7052230-8da4-11ed-8912-f2d54f607f5b, node-14/usr/lib64/galera-4/libg
|
=================================================
|
2023-09-17 16:44:45 8 [Note] WSREP: Server status change initialized -> joined
|
2023-09-17 16:44:45 8 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
|
2023-09-17 16:44:45 8 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
|
2023-09-17 16:44:45 2 [Note] WSREP: Draining apply monitors after IST up to 13217577
|
2023-09-17 16:44:45 2 [Note] WSREP: IST received: 171a890e-7d0a-11ed-8263-6722050e329f:13217577
|
2023-09-17 16:44:45 2 [Note] WSREP: Lowest cert index boundary for CC from sst: 13217426
|
2023-09-17 16:44:45 2 [Note] WSREP: Min available from gcache for CC from sst: 13021525
|
2023-09-17 16:44:45 0 [Note] WSREP: 4.0 (node-139): State transfer from 5.0 (node-14/usr/lib64/galera-4/libgalera_smm.so0) complete.
|
2023-09-17 16:44:45 0 [Note] WSREP: Shifting JOINER -> JOINED (TO: 13220600)
|
2023-09-17 16:44:45 0 [Note] WSREP: Processing event queue:... 0.0% ( 0/2836 events) complete.
|
2023-09-17 16:44:45 0 [Note] WSREP: (c1807c32-9da7, 'tcp://0.0.0.0:4567') turning message relay requesting off
|
2023-09-17 16:44:46 0 [Note] WSREP: Member 4.0 (node-139) synced with group.
|
2023-09-17 16:44:46 0 [Note] WSREP: Processing event queue:...100.0% (2969/2969 events) complete.
|
2023-09-17 16:44:46 0 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 13220730)
|
2023-09-17 16:44:46 2 [Note] WSREP: Server node-139 synced with group
|
2023-09-17 16:44:46 2 [Note] WSREP: Server status change joined -> synced
|
2023-09-17 16:44:46 2 [Note] WSREP: Synchronized with group, ready for connections
|
2023-09-17 16:44:46 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
|
2023-09-17 16:45:06 26 [Warning] IP address '15.24.131.140' could not be resolved: Name or service not known
|
2023-09-17 16:59:08 877 [Warning] Aborted connection 877 to db: 'rdc-cloud' user: 'rdccloud' host: '15.24.131.140' (Got an error reading communication packets)
|
mariadbd: /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.10.2/wsrep-lib/include/wsrep/client_state.hpp:668: int wsrep::client_state::bf_abort(wsrep::seqno): Assertion `mode_ == m_local || transaction_.is_streaming()' failed.
|
230917 17:00:34 [ERROR] mysqld got signal 6 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
Attachments
Issue Links
- relates to
-
MDEV-30217 Assertion `mode_ == m_local || transaction_.is_streaming()' failed in int wsrep::client_state::bf_abort(wsrep::seqno)
-
- Closed
-
jixian8784 Can you try with more recent version of MariaDB server and Galera library. If problem is reproducible we would need more information about how to reproduce. For example you could provide full error logs from all nodes (you could even use --wsrep-debug=1), node configuration files, output of show processlist and some instructions of workload used.