[MDEV-26818] Random Crashes on one Galera node Created: 2021-10-13  Updated: 2023-04-11

Status: Open
Project: MariaDB Server
Component/s: Galera
Affects Version/s: 10.6.4
Fix Version/s: 10.6

Type: Bug Priority: Major
Reporter: Matt Assignee: Julius Goryavsky
Resolution: Unresolved Votes: 6
Labels: crash, galera
Environment:

Ubuntu 20 LTS
Maria 10.6.4
Galera 4

Issue Links:
Problem/Incident

 Description   

Randomly we're seeing crashes on one galera/mariadb node out of 3.

It's not a full crash but any clients connected fail until the node is full downed.

The process seems to fail, but on restart, it then fails to start normally.

The crash:

Oct 13 11:52:51 db02 mariadbd[14877]: mariadbd: /home/buildbot/buildbot/build/mariadb-10.6.4/wsrep-lib/include/wsrep/client_state.hpp:668: int wsrep::client_state::bf_abort(wsrep::seqno): Assertion `mode_ == m_local || transaction_.is_streaming()' failed.
 
Oct 13 11:52:51 db02 mariadbd[14877]: 211013 11:52:51 [ERROR] mysqld got signal 6 ;
 
Oct 13 11:52:51 db02 mariadbd[14877]: This could be because you hit a bug. It is also possible that this binary
 
Oct 13 11:52:51 db02 mariadbd[14877]: or one of the libraries it was linked against is corrupt, improperly built,
 
Oct 13 11:52:51 db02 mariadbd[14877]: or misconfigured. This error can also be caused by malfunctioning hardware.
 
Oct 13 11:52:51 db02 mariadbd[14877]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
Oct 13 11:52:51 db02 mariadbd[14877]: We will try our best to scrape up some info that will hopefully help
 
Oct 13 11:52:51 db02 mariadbd[14877]: diagnose the problem, but since we have already crashed,
 
Oct 13 11:52:51 db02 mariadbd[14877]: something is definitely wrong and this may fail.
 
Oct 13 11:52:51 db02 mariadbd[14877]: Server version: 10.6.4-MariaDB-1:10.6.4+maria~focal
 
Oct 13 11:52:51 db02 mariadbd[14877]: key_buffer_size=134217728
 
Oct 13 11:52:52 db02 mariadbd[14877]: read_buffer_size=131072
 
Oct 13 11:52:52 db02 mariadbd[14877]: max_used_connections=30
 
Oct 13 11:52:52 db02 mariadbd[14877]: max_threads=153
 
Oct 13 11:52:52 db02 mariadbd[14877]: thread_count=39
 
Oct 13 11:52:52 db02 mariadbd[14877]: It is possible that mysqld could use up to
 
Oct 13 11:52:52 db02 mariadbd[14877]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467956 K  bytes of memory
 
Oct 13 11:52:52 db02 mariadbd[14877]: Hope that's ok; if not, decrease some variables in the equation.
 
Oct 13 11:52:52 db02 mariadbd[14877]: Thread pointer: 0x7f4120000c58
 
Oct 13 11:52:52 db02 mariadbd[14877]: Attempting backtrace. You can use the following information to find out
 
Oct 13 11:52:52 db02 mariadbd[14877]: where mysqld died. If you see no messages after this, something went
 
Oct 13 11:52:52 db02 mariadbd[14877]: terribly wrong...
 
Oct 13 11:52:52 db02 mariadbd[14877]: stack_bottom = 0x7f4174bc1dc8 thread_stack 0x49000
 
Oct 13 11:52:52 db02 mariadbd[14877]: Printing to addr2line failed
 
Oct 13 11:52:52 db02 mariadbd[14877]: /usr/sbin/mariadbd(my_print_stacktrace+0x32)[0x558120e74f12]
 
Oct 13 11:52:52 db02 mariadbd[14877]: /usr/sbin/mariadbd(handle_fatal_signal+0x485)[0x55812092f2b5]
 
Oct 13 11:52:52 db02 mariadbd[14877]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0)[0x7f41a66ab3c0]
 
Oct 13 11:52:52 db02 mariadbd[14877]: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x7f41a61b118b]
 
Oct 13 11:52:52 db02 mariadbd[14877]: /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b)[0x7f41a6190859]
 
Oct 13 11:52:52 db02 mariadbd[14877]: /lib/x86_64-linux-gnu/libc.so.6(+0x25729)[0x7f41a6190729]
 
Oct 13 11:52:52 db02 mariadbd[14877]: /lib/x86_64-linux-gnu/libc.so.6(+0x36f36)[0x7f41a61a1f36]
 
Oct 13 11:52:52 db02 mariadbd[14877]: /usr/sbin/mariadbd(_Z14wsrep_bf_abortPK3THDPS_+0x626)[0x558120beef56]
 
Oct 13 11:52:52 db02 mariadbd[14877]: /usr/sbin/mariadbd(wsrep_thd_bf_abort+0x1d)[0x558120bf692d]
 
Oct 13 11:52:52 db02 mariadbd[14877]: /usr/sbin/mariadbd(+0xc9c346)[0x558120c17346]
 
Oct 13 11:52:52 db02 mariadbd[14877]: /usr/sbin/mariadbd(+0x633cf9)[0x5581205aecf9]
 
Oct 13 11:52:52 db02 mariadbd[14877]: /usr/sbin/mariadbd(+0x63466a)[0x5581205af66a]
 
Oct 13 11:52:52 db02 mariadbd[14877]: /usr/sbin/mariadbd(+0xd51c08)[0x558120cccc08]
 
Oct 13 11:52:52 db02 mariadbd[14877]: /usr/sbin/mariadbd(+0xd54aa1)[0x558120ccfaa1]
 
Oct 13 11:52:52 db02 mariadbd[14877]: /usr/sbin/mariadbd(+0xca863a)[0x558120c2363a]
 
Oct 13 11:52:52 db02 mariadbd[14877]: /usr/sbin/mariadbd(_ZN7handler12ha_write_rowEPKh+0x190)[0x55812093e350]
 
Oct 13 11:52:52 db02 mariadbd[14877]: /usr/sbin/mariadbd(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x184)[0x558120a57e94]
 
Oct 13 11:52:52 db02 mariadbd[14877]: /usr/sbin/mariadbd(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x111)[0x558120a58511]
 
Oct 13 11:52:52 db02 mariadbd[14877]: /usr/sbin/mariadbd(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x31f)[0x558120a4c36f]
 
Oct 13 11:52:52 db02 mariadbd[14877]: /usr/sbin/mariadbd(_Z18wsrep_apply_eventsP3THDP14Relay_log_infoPKvm+0x1e9)[0x558120bedd79]
 
Oct 13 11:52:52 db02 mariadbd[14877]: /usr/sbin/mariadbd(_ZN21Wsrep_applier_service15apply_write_setERKN5wsrep7ws_metaERKNS0_12const_bufferERNS0_14mutable_bufferE+0xab)[0x558120bd59fb]
 
Oct 13 11:52:52 db02 mariadbd[14877]: /usr/sbin/mariadbd(+0xf7c100)[0x558120ef7100]
 
Oct 13 11:52:52 db02 mariadbd[14877]: /usr/sbin/mariadbd(_ZN5wsrep12server_state8on_applyERNS_21high_priority_serviceERKNS_9ws_handleERKNS_7ws_metaERKNS_12const_bufferE+0xc1)[0x558120ef7fc1]
 
Oct 13 11:52:52 db02 mariadbd[14877]: /usr/sbin/mariadbd(+0xf8e36c)[0x558120f0936c]
 
Oct 13 11:52:52 db02 mariadbd[14877]: /usr/lib/galera/libgalera_smm.so(+0x639eb)[0x7f41a5bd89eb]
 
Oct 13 11:52:52 db02 mariadbd[14877]: /usr/lib/galera/libgalera_smm.so(+0x7582e)[0x7f41a5bea82e]
 
Oct 13 11:52:52 db02 mariadbd[14877]: /usr/lib/galera/libgalera_smm.so(+0x78408)[0x7f41a5bed408]
 
Oct 13 11:52:52 db02 mariadbd[14877]: /usr/lib/galera/libgalera_smm.so(+0xabd92)[0x7f41a5c20d92]
 
Oct 13 11:52:52 db02 mariadbd[14877]: /usr/lib/galera/libgalera_smm.so(+0xabf1b)[0x7f41a5c20f1b]
 
Oct 13 11:52:52 db02 mariadbd[14877]: /usr/lib/galera/libgalera_smm.so(+0xac112)[0x7f41a5c21112]
 
Oct 13 11:52:52 db02 mariadbd[14877]: /usr/lib/galera/libgalera_smm.so(+0x78e00)[0x7f41a5bede00]
 
Oct 13 11:52:52 db02 mariadbd[14877]: /usr/lib/galera/libgalera_smm.so(+0x4efe1)[0x7f41a5bc3fe1]
 
Oct 13 11:52:52 db02 mariadbd[14877]: /usr/sbin/mariadbd(_ZN5wsrep18wsrep_provider_v2611run_applierEPNS_21high_priority_serviceE+0x12)[0x558120f09982]
 
Oct 13 11:52:52 db02 mariadbd[14877]: /usr/sbin/mariadbd(+0xc74c47)[0x558120befc47]
 
Oct 13 11:52:52 db02 mariadbd[14877]: /usr/sbin/mariadbd(_Z15start_wsrep_THDPv+0x263)[0x558120be03c3]
 
Oct 13 11:52:52 db02 mariadbd[14877]: /usr/sbin/mariadbd(+0xbf517d)[0x558120b7017d]
 
Oct 13 11:52:52 db02 mariadbd[14877]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609)[0x7f41a669f609]
 
Oct 13 11:52:52 db02 mariadbd[14877]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x43)[0x7f41a628d293]
 
Oct 13 11:52:52 db02 mariadbd[14877]: Trying to get some variables.
 
Oct 13 11:52:52 db02 mariadbd[14877]: Some pointers may be invalid and cause the dump to abort.
 
Oct 13 11:52:52 db02 mariadbd[14877]: Query (0x7f419eeff5d9): INSERT INTO `jobs` (`queue`, `status`, `job`, `params`, `priority`, `run_at`, `created_at`) VALUES ('normal', 'pending', 'Task::NotifyJob', '---\n:model: Ticket\n:id: 138898\n', 10, '2021-10-13 10:52:51', '2021-10-13 10:52:51')
 
Oct 13 11:52:52 db02 mariadbd[14877]: Connection ID (thread ID): 11
 
Oct 13 11:52:52 db02 mariadbd[14877]: Status: NOT_KILLED
 
Oct 13 11:52:52 db02 mariadbd[14877]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off
 
Oct 13 11:52:52 db02 mariadbd[14877]: The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
 
Oct 13 11:52:52 db02 mariadbd[14877]: information that should help you find out what is causing the crash.
 
Oct 13 11:52:52 db02 mariadbd[14877]: Writing a core file...
 
Oct 13 11:52:52 db02 mariadbd[14877]: Working directory at /var/lib/mysql
 
Oct 13 11:52:52 db02 mariadbd[14877]: Resource Limits:
 
Oct 13 11:52:52 db02 mariadbd[14877]: Limit                     Soft Limit           Hard Limit           Units
 
Oct 13 11:52:52 db02 mariadbd[14877]: Max cpu time              unlimited            unlimited            seconds
 
Oct 13 11:52:52 db02 mariadbd[14877]: Max file size             unlimited            unlimited            bytes
 
Oct 13 11:52:52 db02 mariadbd[14877]: Max data size             unlimited            unlimited            bytes
 
Oct 13 11:52:52 db02 mariadbd[14877]: Max stack size            8388608              unlimited            bytes
 
Oct 13 11:52:52 db02 mariadbd[14877]: Max core file size        0                    unlimited            bytes
 
Oct 13 11:52:52 db02 mariadbd[14877]: Max resident set          unlimited            unlimited            bytes
 
Oct 13 11:52:52 db02 mariadbd[14877]: Max processes             3311                 3311                 processes
 
Oct 13 11:52:52 db02 mariadbd[14877]: Max open files            32768                32768                files
 
Oct 13 11:52:52 db02 mariadbd[14877]: Max locked memory         65536                65536                bytes
 
Oct 13 11:52:52 db02 mariadbd[14877]: Max address space         unlimited            unlimited            bytes
 
Oct 13 11:52:52 db02 mariadbd[14877]: Max file locks            unlimited            unlimited            locks
 
Oct 13 11:52:52 db02 mariadbd[14877]: Max pending signals       3311                 3311                 signals
 
Oct 13 11:52:52 db02 mariadbd[14877]: Max msgqueue size         819200               819200               bytes
 
Oct 13 11:52:52 db02 mariadbd[14877]: Max nice priority         0                    0
 
Oct 13 11:52:52 db02 mariadbd[14877]: Max realtime priority     0                    0
 
Oct 13 11:52:52 db02 mariadbd[14877]: Max realtime timeout      unlimited            unlimited            us
 
Oct 13 11:52:52 db02 mariadbd[14877]: Core pattern: |/usr/share/apport/apport %p %s %c %d %P %E
 
Oct 13 11:52:53 db02 systemd[1]: mariadb.service: Main process exited, code=dumped, status=6/ABRT
 
Oct 13 11:52:53 db02 systemd[1]: mariadb.service: Failed with result 'core-dump'.

The restart:

Oct 13 11:52:58 db02 systemd[1]: mariadb.service: Scheduled restart job, restart counter is at 3.
 
Oct 13 11:53:04 db02 mariadbd[21181]: 2021-10-13 11:53:04 0 [Note] /usr/sbin/mariadbd (server 10.6.4-MariaDB-1:10.6.4+maria~focal) starting as process 21181 ...
 
Oct 13 11:53:04 db02 mariadbd[21181]: 2021-10-13 11:53:04 0 [Note] WSREP: Loading provider /usr/lib/galera/libgalera_smm.so initial position: 66ee349a-0be0-11e9-beca-de1b1793d905:494356723
 
Oct 13 11:53:04 db02 mariadbd[21181]: 2021-10-13 11:53:04 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
 
Oct 13 11:53:04 db02 mariadbd[21181]: 2021-10-13 11:53:04 0 [Note] WSREP: wsrep_load(): Galera 26.4.9(r819f29cb) by Codership Oy <info@codership.com> loaded successfully.
 
Oct 13 11:53:04 db02 mariadbd[21181]: 2021-10-13 11:53:04 0 [Note] WSREP: CRC-32C: using "slicing-by-8" algorithm.
 
Oct 13 11:53:04 db02 mariadbd[21181]: 2021-10-13 11:53:04 0 [Note] WSREP: Found saved state: 66ee349a-0be0-11e9-beca-de1b1793d905:-1, safe_to_bootstrap: 0
 
Oct 13 11:53:04 db02 mariadbd[21181]: 2021-10-13 11:53:04 0 [Note] WSREP: GCache DEBUG: opened preamble:
 
Oct 13 11:53:04 db02 mariadbd[21181]: Version: 2
 
Oct 13 11:53:04 db02 mariadbd[21181]: UUID: 66ee349a-0be0-11e9-beca-de1b1793d905
 
Oct 13 11:53:04 db02 mariadbd[21181]: Seqno: -1 - -1
 
Oct 13 11:53:04 db02 mariadbd[21181]: Offset: -1
 
Oct 13 11:53:04 db02 mariadbd[21181]: Synced: 0
 
Oct 13 11:53:04 db02 mariadbd[21181]: 2021-10-13 11:53:04 0 [Note] WSREP: Recovering GCache ring buffer: version: 2, UUID: 66ee349a-0be0-11e9-beca-de1b1793d905, offset: -1
 
Oct 13 11:53:04 db02 mariadbd[21181]: 2021-10-13 11:53:04 0 [Note] WSREP: GCache::RingBuffer initial scan...  0.0% (        0/134217752 bytes) complete.
 
Oct 13 11:53:04 db02 mariadbd[21181]: 2021-10-13 11:53:04 0 [Note] WSREP: GCache::RingBuffer initial scan...100.0% (134217752/134217752 bytes) complete.
 
Oct 13 11:53:04 db02 mariadbd[21181]: 2021-10-13 11:53:04 0 [Note] WSREP: Recovering GCache ring buffer: found gapless sequence 494268637-494356730
 
Oct 13 11:53:04 db02 mariadbd[21181]: 2021-10-13 11:53:04 0 [Note] WSREP: GCache::RingBuffer unused buffers scan...  0.0% (        0/134212640 bytes) complete.
 
Oct 13 11:53:04 db02 mariadbd[21181]: 2021-10-13 11:53:04 0 [Note] WSREP: GCache::RingBuffer unused buffers scan...100.0% (134212640/134212640 bytes) complete.
 
Oct 13 11:53:04 db02 mariadbd[21181]: 2021-10-13 11:53:04 0 [Note] WSREP: GCache DEBUG: RingBuffer::recover(): found 5/88099 locked buffers
 
Oct 13 11:53:04 db02 mariadbd[21181]: 2021-10-13 11:53:04 0 [Note] WSREP: GCache DEBUG: RingBuffer::recover(): free space: 5408/134217728
 
Oct 13 11:53:04 db02 mariadbd[21181]: 2021-10-13 11:53:04 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 172.20.100.122; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum
 
Oct 13 11:53:04 db02 mariadbd[21181]: 2021-10-13 11:53:04 0 [Note] WSREP: Service thread queue flushed.
 
Oct 13 11:53:04 db02 mariadbd[21181]: 2021-10-13 11:53:04 0 [Note] WSREP: ####### Assign initial position for certification: 66ee349a-0be0-11e9-beca-de1b1793d905:494356723, protocol version: -1
 
Oct 13 11:53:04 db02 mariadbd[21181]: 2021-10-13 11:53:04 0 [Note] WSREP: Start replication
 
Oct 13 11:53:04 db02 mariadbd[21181]: 2021-10-13 11:53:04 0 [Note] WSREP: Connecting with bootstrap option: 0
 
Oct 13 11:53:04 db02 mariadbd[21181]: 2021-10-13 11:53:04 0 [Note] WSREP: Setting GCS initial position to 66ee349a-0be0-11e9-beca-de1b1793d905:494356723
 
Oct 13 11:53:04 db02 mariadbd[21181]: 2021-10-13 11:53:04 0 [Note] WSREP: protonet asio version 0
 
Oct 13 11:53:04 db02 mariadbd[21181]: 2021-10-13 11:53:04 0 [Note] WSREP: Using CRC-32C for message checksums.
 
Oct 13 11:53:04 db02 mariadbd[21181]: 2021-10-13 11:53:04 0 [Note] WSREP: backend: asio
 
Oct 13 11:53:04 db02 mariadbd[21181]: 2021-10-13 11:53:04 0 [Note] WSREP: gcomm thread scheduling priority set to other:0
 
Oct 13 11:53:04 db02 mariadbd[21181]: 2021-10-13 11:53:04 0 [Note] WSREP: restore pc from disk successfully
 
Oct 13 11:53:04 db02 mariadbd[21181]: 2021-10-13 11:53:04 0 [Note] WSREP: GMCast version 0
 
Oct 13 11:53:04 db02 mariadbd[21181]: 2021-10-13 11:53:04 0 [Note] WSREP: (92d6077e-9592, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
 
Oct 13 11:53:04 db02 mariadbd[21181]: 2021-10-13 11:53:04 0 [Note] WSREP: (92d6077e-9592, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
 
Oct 13 11:53:04 db02 mariadbd[21181]: 2021-10-13 11:53:04 0 [Note] WSREP: EVS version 1
 
Oct 13 11:53:04 db02 mariadbd[21181]: 2021-10-13 11:53:04 0 [Note] WSREP: gcomm: connecting to group 'mariadb-galera-cluster', peer '172.20.100.121:,172.20.100.122:,172.20.100.123:'
 
Oct 13 11:53:04 db02 mariadbd[21181]: 2021-10-13 11:53:04 0 [Note] WSREP: (92d6077e-9592, 'tcp://0.0.0.0:4567') Found matching local endpoint for a connection, blacklisting address tcp://172.20.100.122:4567
 
Oct 13 11:53:04 db02 mariadbd[21181]: 2021-10-13 11:53:04 0 [Note] WSREP: (92d6077e-9592, 'tcp://0.0.0.0:4567') connection established to a0fe6424-8595 tcp://172.20.100.121:4567
 
Oct 13 11:53:04 db02 mariadbd[21181]: 2021-10-13 11:53:04 0 [Note] WSREP: (92d6077e-9592, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
 
Oct 13 11:53:04 db02 mariadbd[21181]: 2021-10-13 11:53:04 0 [Note] WSREP: (92d6077e-9592, 'tcp://0.0.0.0:4567') connection established to 8d09d855-a8c7 tcp://172.20.100.123:4567
 
Oct 13 11:53:05 db02 mariadbd[21181]: 2021-10-13 11:53:05 0 [Note] WSREP: EVS version upgrade 0 -> 1
 
Oct 13 11:53:05 db02 mariadbd[21181]: 2021-10-13 11:53:05 0 [Note] WSREP: declaring 8d09d855-a8c7 at tcp://172.20.100.123:4567 stable
 
Oct 13 11:53:05 db02 mariadbd[21181]: 2021-10-13 11:53:05 0 [Note] WSREP: declaring a0fe6424-8595 at tcp://172.20.100.121:4567 stable
 
Oct 13 11:53:05 db02 mariadbd[21181]: 2021-10-13 11:53:05 0 [Note] WSREP: PC protocol upgrade 0 -> 1
 
Oct 13 11:53:05 db02 mariadbd[21181]: 2021-10-13 11:53:05 0 [Note] WSREP: Node 8d09d855-a8c7 state prim
 
Oct 13 11:53:05 db02 mariadbd[21181]: 2021-10-13 11:53:05 0 [Note] WSREP: view(view_id(PRIM,8d09d855-a8c7,272) memb {
 
Oct 13 11:53:05 db02 mariadbd[21181]: #0118d09d855-a8c7,0
 
Oct 13 11:53:05 db02 mariadbd[21181]: #01192d6077e-9592,0
 
Oct 13 11:53:05 db02 mariadbd[21181]: #011a0fe6424-8595,0
 
Oct 13 11:53:05 db02 mariadbd[21181]: } joined {
 
Oct 13 11:53:05 db02 mariadbd[21181]: } left {
 
Oct 13 11:53:05 db02 mariadbd[21181]: } partitioned {
 
Oct 13 11:53:05 db02 mariadbd[21181]: })
 
Oct 13 11:53:05 db02 mariadbd[21181]: 2021-10-13 11:53:05 0 [Note] WSREP: save pc into disk
 
Oct 13 11:53:05 db02 mariadbd[21181]: 2021-10-13 11:53:05 0 [Note] WSREP: clear restored view
 
Oct 13 11:53:05 db02 mariadbd[21181]: 2021-10-13 11:53:05 0 [Note] WSREP: gcomm: connected
 
Oct 13 11:53:05 db02 mariadbd[21181]: 2021-10-13 11:53:05 0 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
 
Oct 13 11:53:05 db02 mariadbd[21181]: 2021-10-13 11:53:05 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
 
Oct 13 11:53:05 db02 mariadbd[21181]: 2021-10-13 11:53:05 0 [Note] WSREP: Opened channel 'mariadb-galera-cluster'
 
Oct 13 11:53:05 db02 mariadbd[21181]: 2021-10-13 11:53:05 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 3
 
Oct 13 11:53:05 db02 mariadbd[21181]: 2021-10-13 11:53:05 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
 
Oct 13 11:53:05 db02 mariadbd[21181]: 2021-10-13 11:53:05 0 [Note] WSREP: STATE EXCHANGE: sent state msg: be6755ce-2c13-11ec-8844-326009979834
 
Oct 13 11:53:05 db02 mariadbd[21181]: 2021-10-13 11:53:05 0 [Note] WSREP: STATE EXCHANGE: got state msg: be6755ce-2c13-11ec-8844-326009979834 from 0 (db03)
 
Oct 13 11:53:05 db02 mariadbd[21181]: 2021-10-13 11:53:05 0 [Note] WSREP: STATE EXCHANGE: got state msg: be6755ce-2c13-11ec-8844-326009979834 from 2 (db01)
 
Oct 13 11:53:05 db02 mariadbd[21181]: 2021-10-13 11:53:05 1 [Note] WSREP: Starting rollbacker thread 1
 
Oct 13 11:53:05 db02 mariadbd[21181]: 2021-10-13 11:53:05 2 [Note] WSREP: Starting applier thread 2
 
Oct 13 11:53:05 db02 mariadbd[21181]: 2021-10-13 11:53:05 0 [Note] WSREP: STATE EXCHANGE: got state msg: be6755ce-2c13-11ec-8844-326009979834 from 1 (db02)
 
Oct 13 11:53:05 db02 mariadbd[21181]: 2021-10-13 11:53:05 0 [Note] WSREP: Quorum results:
 
Oct 13 11:53:05 db02 mariadbd[21181]: #011version    = 6,
 
Oct 13 11:53:05 db02 mariadbd[21181]: #011component  = PRIMARY,
 
Oct 13 11:53:05 db02 mariadbd[21181]: #011conf_id    = 206,
 
Oct 13 11:53:05 db02 mariadbd[21181]: #011members    = 2/3 (joined/total),
 
Oct 13 11:53:05 db02 mariadbd[21181]: #011act_id     = 494356758,
 
Oct 13 11:53:05 db02 mariadbd[21181]: #011last_appl. = 494356740,
 
Oct 13 11:53:05 db02 mariadbd[21181]: #011protocols  = 2/10/4 (gcs/repl/appl),
 
Oct 13 11:53:05 db02 mariadbd[21181]: #011vote policy= 0,
 
Oct 13 11:53:05 db02 mariadbd[21181]: #011group UUID = 66ee349a-0be0-11e9-beca-de1b1793d905
 
Oct 13 11:53:05 db02 mariadbd[21181]: 2021-10-13 11:53:05 0 [Note] WSREP: Flow-control interval: [28, 28]
 
Oct 13 11:53:05 db02 mariadbd[21181]: 2021-10-13 11:53:05 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 494356759)
 
Oct 13 11:53:05 db02 mariadbd[21181]: 2021-10-13 11:53:05 2 [Note] WSREP: ####### processing CC 494356759, local, ordered
 
Oct 13 11:53:05 db02 mariadbd[21181]: 2021-10-13 11:53:05 2 [Note] WSREP: Process first view: 66ee349a-0be0-11e9-beca-de1b1793d905 my uuid: 92d6077e-2aae-11ec-9592-f6f1e5ba1e01
 
Oct 13 11:53:05 db02 mariadbd[21181]: 2021-10-13 11:53:05 2 [Note] WSREP: Server db02 connected to cluster at position 66ee349a-0be0-11e9-beca-de1b1793d905:494356759 with ID 92d6077e-2aae-11ec-9592-f6f1e5ba1e01
 
Oct 13 11:53:05 db02 mariadbd[21181]: 2021-10-13 11:53:05 2 [Note] WSREP: Server status change disconnected -> connected
 
Oct 13 11:53:05 db02 mariadbd[21181]: 2021-10-13 11:53:05 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
Oct 13 11:53:05 db02 mariadbd[21181]: 2021-10-13 11:53:05 2 [Note] WSREP: ####### My UUID: 92d6077e-2aae-11ec-9592-f6f1e5ba1e01
 
Oct 13 11:53:05 db02 mariadbd[21181]: 2021-10-13 11:53:05 2 [Note] WSREP: Cert index reset to 00000000-0000-0000-0000-000000000000:-1 (proto: 10), state transfer needed: yes
 
Oct 13 11:53:05 db02 mariadbd[21181]: 2021-10-13 11:53:05 0 [Note] WSREP: Service thread queue flushed.
 
Oct 13 11:53:05 db02 mariadbd[21181]: 2021-10-13 11:53:05 2 [Note] WSREP: ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: -1
 
Oct 13 11:53:05 db02 mariadbd[21181]: 2021-10-13 11:53:05 2 [Note] WSREP: State transfer required:
 
Oct 13 11:53:05 db02 mariadbd[21181]: #011Group state: 66ee349a-0be0-11e9-beca-de1b1793d905:494356759
 
Oct 13 11:53:05 db02 mariadbd[21181]: #011Local state: 66ee349a-0be0-11e9-beca-de1b1793d905:494356723
 
Oct 13 11:53:05 db02 mariadbd[21181]: 2021-10-13 11:53:05 2 [Note] WSREP: Server status change connected -> joiner
 
Oct 13 11:53:05 db02 mariadbd[21181]: 2021-10-13 11:53:05 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
Oct 13 11:53:05 db02 mariadbd[21181]: 2021-10-13 11:53:05 0 [Note] WSREP: Joiner monitor thread started to monitor
 
Oct 13 11:53:05 db02 mariadbd[21181]: 2021-10-13 11:53:05 0 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '172.20.100.122' --datadir '/var/lib/mysql/' --parent '21181' --mysqld-args --wsrep_start_position=66ee349a-0be0-11e9-beca-de1b1793d905:494356723'
 
Oct 13 11:53:06 db02 mariadbd[21181]: 2021-10-13 11:53:06 2 [Note] WSREP: ####### IST uuid:66ee349a-0be0-11e9-beca-de1b1793d905 f: 494356724, l: 494356759, STRv: 3
 
Oct 13 11:53:06 db02 mariadbd[21181]: 2021-10-13 11:53:06 2 [Note] WSREP: IST receiver addr using tcp://172.20.100.122:4568
 
Oct 13 11:53:06 db02 mariadbd[21181]: 2021-10-13 11:53:06 2 [Note] WSREP: Prepared IST receiver for 494356724-494356759, listening at: tcp://172.20.100.122:4568
 
Oct 13 11:53:06 db02 mariadbd[21181]: 2021-10-13 11:53:06 0 [Note] WSREP: Member 1.0 (db02) requested state transfer from '*any*'. Selected 2.0 (db01)(SYNCED) as donor.
 
Oct 13 11:53:06 db02 mariadbd[21181]: 2021-10-13 11:53:06 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 494356759)
 
Oct 13 11:53:06 db02 mariadbd[21181]: 2021-10-13 11:53:06 2 [Note] WSREP: Requesting state transfer: success, donor: 2
 
Oct 13 11:53:06 db02 mariadbd[21181]: 2021-10-13 11:53:06 0 [Note] WSREP: 2.0 (db01): State transfer to 1.0 (db02) complete.
 
Oct 13 11:53:06 db02 mariadbd[21181]: 2021-10-13 11:53:06 0 [Note] WSREP: Member 2.0 (db01) synced with group.
 
Oct 13 11:53:07 db02 mariadbd[21191]: WSREP_SST: [INFO] Joiner cleanup: rsync PID=21289, stunnel PID=0 (20211013 11:53:07.133)
 
Oct 13 11:53:07 db02 mariadbd[21191]: WSREP_SST: [INFO] Joiner cleanup done. (20211013 11:53:07.694)
 
Oct 13 11:53:07 db02 mariadbd[21181]: 2021-10-13 11:53:07 3 [Note] WSREP: SST received
 
Oct 13 11:53:07 db02 mariadbd[21181]: 2021-10-13 11:53:07 3 [Note] WSREP: Server status change joiner -> initializing
 
Oct 13 11:53:07 db02 mariadbd[21181]: 2021-10-13 11:53:07 3 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
Oct 13 11:53:07 db02 mariadbd[21181]: 2021-10-13 11:53:07 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
 
Oct 13 11:53:07 db02 mariadbd[21181]: 2021-10-13 11:53:07 0 [Note] InnoDB: Number of pools: 1
 
Oct 13 11:53:07 db02 mariadbd[21181]: 2021-10-13 11:53:07 0 [Note] InnoDB: Using generic crc32 instructions
 
Oct 13 11:53:07 db02 mariadbd[21181]: 2021-10-13 11:53:07 0 [Note] mariadbd: O_TMPFILE is not supported on /tmp (disabling future attempts)
 
Oct 13 11:53:07 db02 mariadbd[21181]: 2021-10-13 11:53:07 0 [Note] InnoDB: Using Linux native AIO
 
Oct 13 11:53:07 db02 mariadbd[21181]: 2021-10-13 11:53:07 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
 
Oct 13 11:53:07 db02 mariadbd[21181]: 2021-10-13 11:53:07 0 [Note] InnoDB: Completed initialization of buffer pool
 
Oct 13 11:53:07 db02 mariadbd[21181]: 2021-10-13 11:53:07 0 [Note] InnoDB: 1 transaction(s) which must be rolled back or cleaned up in total 12 row operations to undo
 
Oct 13 11:53:07 db02 mariadbd[21181]: 2021-10-13 11:53:07 0 [Note] InnoDB: Trx id counter is 1268078797
 
Oct 13 11:53:07 db02 mariadbd[21181]: 2021-10-13 11:53:07 0 [Note] InnoDB: 128 rollback segments are active.
 
Oct 13 11:53:08 db02 mariadbd[21181]: 2021-10-13 11:53:08 0 [Note] InnoDB: Creating shared tablespace for temporary tables
 
Oct 13 11:53:08 db02 mariadbd[21181]: 2021-10-13 11:53:08 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
 
Oct 13 11:53:08 db02 mariadbd[21181]: 2021-10-13 11:53:08 0 [Note] InnoDB: Starting in background the rollback of recovered transactions
 
Oct 13 11:53:08 db02 mariadbd[21181]: 2021-10-13 11:53:08 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
 
Oct 13 11:53:08 db02 mariadbd[21181]: 2021-10-13 11:53:08 0 [Note] InnoDB: 10.6.4 started; log sequence number 566275668195; transaction id 1268078798
 
Oct 13 11:53:08 db02 mariadbd[21181]: 2021-10-13 11:53:08 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
 
Oct 13 11:53:08 db02 mariadbd[21181]: 2021-10-13 11:53:08 0 [Note] InnoDB: Cannot open '/var/lib/mysql/ib_buffer_pool' for reading: No such file or directory
 
Oct 13 11:53:08 db02 mariadbd[21181]: 2021-10-13 11:53:08 0 [Note] Plugin 'FEEDBACK' is disabled.
 
Oct 13 11:53:08 db02 mariadbd[21181]: 2021-10-13 11:53:08 0 [Warning] You need to use --log-bin to make --expire-logs-days or --binlog-expire-logs-seconds work.
 
Oct 13 11:53:08 db02 mariadbd[21181]: 2021-10-13 11:53:08 0 [Note] Server socket created on IP: '0.0.0.0'.
 
Oct 13 11:53:08 db02 mariadbd[21181]: 2021-10-13 11:53:08 0 [Warning] 'user' entry 'cerb@*' ignored in --skip-name-resolve mode.
 
Oct 13 11:53:08 db02 mariadbd[21181]: 2021-10-13 11:53:08 0 [Warning] 'db' entry 'cerb cerb@*' ignored in --skip-name-resolve mode.
 
Oct 13 11:53:08 db02 mariadbd[21181]: 2021-10-13 11:53:08 0 [Note] WSREP: wsrep_init_schema_and_SR 0x0
 
Oct 13 11:53:08 db02 mariadbd[21181]: 2021-10-13 11:53:08 0 [Note] WSREP: Server initialized
 
Oct 13 11:53:08 db02 mariadbd[21181]: 2021-10-13 11:53:08 0 [Note] WSREP: Server status change initializing -> initialized
 
Oct 13 11:53:08 db02 mariadbd[21181]: 2021-10-13 11:53:08 0 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
Oct 13 11:53:08 db02 mariadbd[21181]: 2021-10-13 11:53:08 3 [Note] WSREP: Server status change initialized -> joined
 
Oct 13 11:53:08 db02 mariadbd[21181]: 2021-10-13 11:53:08 3 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
Oct 13 11:53:08 db02 mariadbd[21181]: 2021-10-13 11:53:08 3 [Note] WSREP: Recovered position from storage: 66ee349a-0be0-11e9-beca-de1b1793d905:494356723
 
Oct 13 11:53:08 db02 mariadbd[21181]: 2021-10-13 11:53:08 9 [Note] WSREP: Starting applier thread 9
 
Oct 13 11:53:08 db02 mariadbd[21181]: 2021-10-13 11:53:08 5 [Note] WSREP: Starting applier thread 5
 
Oct 13 11:53:08 db02 mariadbd[21181]: 2021-10-13 11:53:08 8 [Note] WSREP: Starting applier thread 8
 
Oct 13 11:53:08 db02 mariadbd[21181]: 2021-10-13 11:53:08 7 [Note] WSREP: Starting applier thread 7
 
Oct 13 11:53:08 db02 mariadbd[21181]: 2021-10-13 11:53:08 10 [Note] WSREP: Starting applier thread 10
 
Oct 13 11:53:08 db02 mariadbd[21181]: 2021-10-13 11:53:08 6 [Note] WSREP: Starting applier thread 6
 
Oct 13 11:53:08 db02 mariadbd[21181]: 2021-10-13 11:53:08 12 [Note] WSREP: Starting applier thread 12
 
Oct 13 11:53:08 db02 mariadbd[21181]: 2021-10-13 11:53:08 0 [Note] /usr/sbin/mariadbd: ready for connections.
 
Oct 13 11:53:08 db02 mariadbd[21181]: Version: '10.6.4-MariaDB-1:10.6.4+maria~focal'  socket: '/run/mysqld/mysqld.sock'  port: 3307  mariadb.org binary distribution
 
Oct 13 11:53:08 db02 mariadbd[21181]: 2021-10-13 11:53:08 3 [Note] WSREP: Recovered view from SST:
 
Oct 13 11:53:08 db02 mariadbd[21181]:   id: 66ee349a-0be0-11e9-beca-de1b1793d905:493933115
 
Oct 13 11:53:08 db02 mariadbd[21181]:   status: primary
 
Oct 13 11:53:08 db02 mariadbd[21181]:   protocol_version: 4
 
Oct 13 11:53:08 db02 mariadbd[21181]:   capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
 
Oct 13 11:53:08 db02 mariadbd[21181]:   final: no
 
Oct 13 11:53:08 db02 mariadbd[21181]:   own_index: -1
 
Oct 13 11:53:08 db02 mariadbd[21181]:   members(3):
 
Oct 13 11:53:08 db02 mariadbd[21181]: #0110: 8d09d855-2aa9-11ec-a8c7-6707eaa8ebe2, db03
 
Oct 13 11:53:08 db02 mariadbd[21181]: #0111: 92d6077e-2aae-11ec-9591-f6f1e5ba1e01, db02
 
Oct 13 11:53:08 db02 mariadbd[21181]: #0112: a0fe6424-2ab2-11ec-8595-9f12a64f3c1f, db01
 
Oct 13 11:53:08 db02 mariadbd[21181]: 2021-10-13 11:53:08 3 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
Oct 13 11:53:08 db02 mariadbd[21181]: 2021-10-13 11:53:08 14 [Note] WSREP: Recovered cluster id 66ee349a-0be0-11e9-beca-de1b1793d905
 
Oct 13 11:53:08 db02 mariadbd[21181]: 2021-10-13 11:53:08 3 [Note] WSREP: SST received: 66ee349a-0be0-11e9-beca-de1b1793d905:494356723
 
Oct 13 11:53:08 db02 mariadbd[21181]: 2021-10-13 11:53:08 3 [Note] WSREP: SST succeeded for position 66ee349a-0be0-11e9-beca-de1b1793d905:494356723
 
Oct 13 11:53:08 db02 mariadbd[21181]: 2021-10-13 11:53:08 0 [Note] WSREP: Joiner monitor thread ended with total time 3 sec
 
Oct 13 11:53:08 db02 mariadbd[21181]: 2021-10-13 11:53:08 2 [Note] WSREP: Installed new state from SST: 66ee349a-0be0-11e9-beca-de1b1793d905:494356723
 
Oct 13 11:53:08 db02 mariadbd[21181]: 2021-10-13 11:53:08 2 [Note] WSREP: Receiving IST: 36 writesets, seqnos 494356724-494356759
 
Oct 13 11:53:08 db02 mariadbd[21181]: 2021-10-13 11:53:08 0 [Note] WSREP: ####### IST applying starts with 494356724
 
Oct 13 11:53:08 db02 mariadbd[21181]: 2021-10-13 11:53:08 0 [Note] WSREP: ####### IST current seqno initialized to 494356724
 
Oct 13 11:53:08 db02 mariadbd[21181]: 2021-10-13 11:53:08 0 [Note] WSREP: Receiving IST...  0.0% ( 0/36 events) complete.
 
Oct 13 11:53:08 db02 mariadbd[21181]: 2021-10-13 11:53:08 0 [Note] WSREP: Service thread queue flushed.
 
Oct 13 11:53:08 db02 mariadbd[21181]: 2021-10-13 11:53:08 0 [Note] WSREP: ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:494356723, protocol version: 5
 
Oct 13 11:53:08 db02 mariadbd[21181]: 211013 11:53:08 [ERROR] mysqld got signal 11 ;
 
Oct 13 11:53:08 db02 mariadbd[21181]: This could be because you hit a bug. It is also possible that this binary
 
Oct 13 11:53:08 db02 mariadbd[21181]: or one of the libraries it was linked against is corrupt, improperly built,
 
Oct 13 11:53:08 db02 mariadbd[21181]: or misconfigured. This error can also be caused by malfunctioning hardware.
 
Oct 13 11:53:08 db02 mariadbd[21181]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
Oct 13 11:53:08 db02 mariadbd[21181]: We will try our best to scrape up some info that will hopefully help
 
Oct 13 11:53:08 db02 mariadbd[21181]: diagnose the problem, but since we have already crashed,
 
Oct 13 11:53:08 db02 mariadbd[21181]: something is definitely wrong and this may fail.
 
Oct 13 11:53:08 db02 mariadbd[21181]: Server version: 10.6.4-MariaDB-1:10.6.4+maria~focal
 
Oct 13 11:53:08 db02 mariadbd[21181]: key_buffer_size=134217728
 
Oct 13 11:53:08 db02 mariadbd[21181]: read_buffer_size=131072
 
Oct 13 11:53:08 db02 mariadbd[21181]: max_used_connections=0
 
Oct 13 11:53:08 db02 mariadbd[21181]: max_threads=153
 
Oct 13 11:53:08 db02 mariadbd[21181]: thread_count=9
 
Oct 13 11:53:08 db02 mariadbd[21181]: It is possible that mysqld could use up to
 
Oct 13 11:53:08 db02 mariadbd[21181]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467956 K  bytes of memory
 
Oct 13 11:53:08 db02 mariadbd[21181]: Hope that's ok; if not, decrease some variables in the equation.
 
Oct 13 11:53:08 db02 mariadbd[21181]: Thread pointer: 0x7f6508000c58
 
Oct 13 11:53:08 db02 mariadbd[21181]: Attempting backtrace. You can use the following information to find out
 
Oct 13 11:53:08 db02 mariadbd[21181]: where mysqld died. If you see no messages after this, something went
 
Oct 13 11:53:08 db02 mariadbd[21181]: terribly wrong...
 
Oct 13 11:53:08 db02 mariadbd[21181]: stack_bottom = 0x7f658c161dc8 thread_stack 0x49000
 
Oct 13 11:53:08 db02 mariadbd[21181]: 2021-10-13 11:53:08 0 [Note] WSREP: (92d6077e-9592, 'tcp://0.0.0.0:4567') turning message relay requesting off
 
Oct 13 11:53:09 db02 mariadbd[21181]: /usr/sbin/mariadbd(my_print_stacktrace+0x32)[0x55f2cb285f12]
 
Oct 13 11:53:09 db02 mariadbd[21181]: /usr/sbin/mariadbd(handle_fatal_signal+0x485)[0x55f2cad402b5]
 
Oct 13 11:59:50 db02 mariadbd[21181]: 2021-10-13 11:59:50 0 [Note] /usr/sbin/mariadbd (initiated by: unknown): Normal shutdown
 
Oct 13 11:59:50 db02 mariadbd[21181]: 2021-10-13 11:59:50 0 [Note] WSREP: Shutdown replication
 
Oct 13 11:59:50 db02 mariadbd[21181]: 2021-10-13 11:59:50 0 [Note] WSREP: Server status change joined -> disconnecting
 
Oct 13 11:59:50 db02 mariadbd[21181]: 2021-10-13 11:59:50 0 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
Oct 13 11:59:50 db02 mariadbd[21181]: 2021-10-13 11:59:50 0 [Note] WSREP: Closing send monitor...
 
Oct 13 11:59:50 db02 mariadbd[21181]: 2021-10-13 11:59:50 0 [Note] WSREP: Closed send monitor.
 
Oct 13 11:59:50 db02 mariadbd[21181]: 2021-10-13 11:59:50 0 [Note] WSREP: gcomm: terminating thread
 
Oct 13 11:59:50 db02 mariadbd[21181]: 2021-10-13 11:59:50 0 [Note] WSREP: gcomm: joining thread
 
Oct 13 11:59:50 db02 mariadbd[21181]: 2021-10-13 11:59:50 0 [Note] WSREP: gcomm: closing backend
 
Oct 13 11:59:50 db02 mariadbd[21181]: 2021-10-13 11:59:50 0 [Note] WSREP: view(view_id(NON_PRIM,8d09d855-a8c7,272) memb {
 
Oct 13 11:59:50 db02 mariadbd[21181]: #01192d6077e-9592,0
 
Oct 13 11:59:50 db02 mariadbd[21181]: } joined {
 
Oct 13 11:59:50 db02 mariadbd[21181]: } left {
 
Oct 13 11:59:50 db02 mariadbd[21181]: } partitioned {
 
Oct 13 11:59:50 db02 mariadbd[21181]: #0118d09d855-a8c7,0
 
Oct 13 11:59:50 db02 mariadbd[21181]: #011a0fe6424-8595,0
 
Oct 13 11:59:50 db02 mariadbd[21181]: })
 
Oct 13 11:59:50 db02 mariadbd[21181]: 2021-10-13 11:59:50 0 [Note] WSREP: PC protocol downgrade 1 -> 0
 
Oct 13 11:59:50 db02 mariadbd[21181]: 2021-10-13 11:59:50 0 [Note] WSREP: view((empty))
 
Oct 13 11:59:50 db02 mariadbd[21181]: 2021-10-13 11:59:50 0 [Note] WSREP: gcomm: closed
 
Oct 13 11:59:50 db02 mariadbd[21181]: 2021-10-13 11:59:50 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
 
Oct 13 11:59:50 db02 mariadbd[21181]: 2021-10-13 11:59:50 0 [Note] WSREP: Flow-control interval: [16, 16]
 
Oct 13 11:59:50 db02 mariadbd[21181]: 2021-10-13 11:59:50 0 [Note] WSREP: Received NON-PRIMARY.
 
Oct 13 11:59:50 db02 mariadbd[21181]: 2021-10-13 11:59:50 0 [Note] WSREP: Shifting JOINER -> OPEN (TO: 494357677)
 
Oct 13 11:59:50 db02 mariadbd[21181]: 2021-10-13 11:59:50 0 [Note] WSREP: New SELF-LEAVE.
 
Oct 13 11:59:50 db02 mariadbd[21181]: 2021-10-13 11:59:50 0 [Note] WSREP: Flow-control interval: [0, 0]
 
Oct 13 11:59:50 db02 mariadbd[21181]: 2021-10-13 11:59:50 0 [Note] WSREP: Received SELF-LEAVE. Closing connection.
 
Oct 13 11:59:50 db02 mariadbd[21181]: 2021-10-13 11:59:50 0 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 494357677)
 
Oct 13 11:59:50 db02 mariadbd[21181]: 2021-10-13 11:59:50 0 [Note] WSREP: RECV thread exiting 0: Success
 
Oct 13 11:59:50 db02 mariadbd[21181]: 2021-10-13 11:59:50 0 [Note] WSREP: recv_thread() joined.
 
Oct 13 11:59:50 db02 mariadbd[21181]: 2021-10-13 11:59:50 0 [Note] WSREP: Closing replication queue.
 
Oct 13 11:59:50 db02 mariadbd[21181]: 2021-10-13 11:59:50 0 [Note] WSREP: Closing slave action queue.

At a bit of a loss as to what could be the issue.

 Comments   
Comment by Thomas Lutz [ 2021-10-22 ]

Maybe the same issue that is https://jira.mariadb.org/browse/MDEV-25368 ?

Comment by Rick Tuk [ 2021-10-26 ]

We are experiencing the same issue, so far the queries that are logged in this crash report are very simple UPDATE `table` SET `field` = 'x' WHERE `id` = 'y';
When restarting the node it tries to reapply the query based on IST and fails again. SST is the only way to resolve the issue.
On one occasion we have seen that after the crash MariaDB does come back online with a local state of "inconsistent" rather than just crashing repeatedly.

@tholu the issue you linked seems to be different, for us the cluster does not freeze, it is just one node in the cluster that fully crashes, the rest of the cluster keeps working as expected

Comment by Mark Reibert [ 2021-11-09 ]

I am seeing this same thing with MariaDB 10.4.21 on Ubuntu 20.04, and when I write the "same thing" I mean the same thing. First the Assertion `mode_ == m_local || transaction_.is_streaming()' failed resulting in a SIGABRT, then a SIGSEGV upon restart.

Here is the SIGABRT

Nov 09 18:15:08 mysqld[3119964]: mysqld: /home/buildbot/buildbot/build/mariadb-10.4.21/wsrep-lib/include/wsrep/client_state.hpp:668: int wsrep::client_state::bf_abort(wsrep::seqno): Assertion `mode_ == m_local || transaction_.is_streaming()' failed.
 
Nov 09 18:15:08 mysqld[3119964]: 211109 18:15:08 [ERROR] mysqld got signal 6 ;
 
Nov 09 18:15:08 mysqld[3119964]: This could be because you hit a bug. It is also possible that this binary
 
Nov 09 18:15:08 mysqld[3119964]: or one of the libraries it was linked against is corrupt, improperly built,
 
Nov 09 18:15:08 mysqld[3119964]: or misconfigured. This error can also be caused by malfunctioning hardware.
 
Nov 09 18:15:08 mysqld[3119964]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
Nov 09 18:15:08 mysqld[3119964]: We will try our best to scrape up some info that will hopefully help
 
Nov 09 18:15:08 mysqld[3119964]: diagnose the problem, but since we have already crashed,
 
Nov 09 18:15:08 mysqld[3119964]: something is definitely wrong and this may fail.
 
Nov 09 18:15:08 mysqld[3119964]: Server version: 10.4.21-MariaDB-1:10.4.21+maria~focal-log
 
Nov 09 18:15:08 mysqld[3119964]: key_buffer_size=134217728
 
Nov 09 18:15:08 mysqld[3119964]: read_buffer_size=131072
 
Nov 09 18:15:08 mysqld[3119964]: max_used_connections=216
 
Nov 09 18:15:08 mysqld[3119964]: max_threads=403
 
Nov 09 18:15:08 mysqld[3119964]: thread_count=286
 
Nov 09 18:15:08 mysqld[3119964]: It is possible that mysqld could use up to
 
Nov 09 18:15:08 mysqld[3119964]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6795373 K  bytes of memory
 
Nov 09 18:15:08 mysqld[3119964]: Hope that's ok; if not, decrease some variables in the equation.
 
Nov 09 18:15:08 mysqld[3119964]: Thread pointer: 0x0
 
Nov 09 18:15:08 mysqld[3119964]: Attempting backtrace. You can use the following information to find out
 
Nov 09 18:15:08 mysqld[3119964]: where mysqld died. If you see no messages after this, something went
 
Nov 09 18:15:08 mysqld[3119964]: terribly wrong...
 
Nov 09 18:15:08 mysqld[3119964]: stack_bottom = 0x0 thread_stack 0x49000

And the SIGSEGV on restart:

Nov 09 18:23:20 mysqld[1046371]: 2021-11-09 18:23:20 0 [Note] WSREP: Receiving IST...  0.0% (   0/2844 events) complete.
 
Nov 09 18:23:20 mysqld[1046371]: 2021-11-09 18:23:20 0 [Note] WSREP: IST preload starting at 4608191245
 
Nov 09 18:23:20 mysqld[1046371]: 2021-11-09 18:23:20 0 [Note] WSREP: Service thread queue flushed.
 
Nov 09 18:23:20 mysqld[1046371]: 2021-11-09 18:23:20 0 [Note] WSREP: ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:4608191244, protocol version: 5
 
Nov 09 18:23:20 mysqld[1046371]: 211109 18:23:20 [ERROR] mysqld got signal 11 ;
 
Nov 09 18:23:20 mysqld[1046371]: This could be because you hit a bug. It is also possible that this binary
 
Nov 09 18:23:20 mysqld[1046371]: or one of the libraries it was linked against is corrupt, improperly built,
 
Nov 09 18:23:20 mysqld[1046371]: or misconfigured. This error can also be caused by malfunctioning hardware.
 
Nov 09 18:23:20 mysqld[1046371]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
Nov 09 18:23:20 mysqld[1046371]: We will try our best to scrape up some info that will hopefully help
 
Nov 09 18:23:20 mysqld[1046371]: diagnose the problem, but since we have already crashed,
 
Nov 09 18:23:20 mysqld[1046371]: something is definitely wrong and this may fail.
 
Nov 09 18:23:20 mysqld[1046371]: Server version: 10.4.21-MariaDB-1:10.4.21+maria~focal-log
 
Nov 09 18:23:20 mysqld[1046371]: key_buffer_size=134217728
 
Nov 09 18:23:20 mysqld[1046371]: read_buffer_size=131072
 
Nov 09 18:23:20 mysqld[1046371]: max_used_connections=0
 
Nov 09 18:23:20 mysqld[1046371]: max_threads=403
 
Nov 09 18:23:20 mysqld[1046371]: thread_count=70
 
Nov 09 18:23:20 mysqld[1046371]: It is possible that mysqld could use up to
 
Nov 09 18:23:20 mysqld[1046371]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6795373 K  bytes of memory
 
Nov 09 18:23:20 mysqld[1046371]: Hope that's ok; if not, decrease some variables in the equation.
 
Nov 09 18:23:20 mysqld[1046371]: Thread pointer: 0x7dd0a0000c48
 
Nov 09 18:23:20 mysqld[1046371]: Attempting backtrace. You can use the following information to find out
 
Nov 09 18:23:20 mysqld[1046371]: where mysqld died. If you see no messages after this, something went
 
Nov 09 18:23:20 mysqld[1046371]: terribly wrong...
 
Nov 09 18:23:20 mysqld[1046371]: stack_bottom = 0x7eed14454e08 thread_stack 0x49000
 
Nov 09 18:23:20 systemd[1]: Started MariaDB 10.4.21 database server.
 
Nov 09 18:23:20 /etc/mysql/debian-start[1047246]: Upgrading MySQL tables if necessary.
 
Nov 09 18:23:21 mysqld[1046371]: 2021-11-09 18:23:21 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT1.52806S), skipping check
 
Nov 09 18:23:21 /etc/mysql/debian-start[1047249]: /usr/bin/mysql_upgrade: the '--basedir' option is always ignored
 
Nov 09 18:23:21 /etc/mysql/debian-start[1047249]: Looking for 'mysql' as: /usr/bin/mysql
 
Nov 09 18:23:21 /etc/mysql/debian-start[1047249]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
 
Nov 09 18:23:21 /etc/mysql/debian-start[1047249]: This installation of MariaDB is already upgraded to 10.4.21-MariaDB, use --force if you still need to run mysql_upgrade
 
Nov 09 18:23:21 mysqld[1047288]: ERROR 1047 (08S01) at line 1: WSREP has not yet prepared node for application use
 
Nov 09 18:23:22 mysqld[1046371]: /usr/sbin/mysqld(my_print_stacktrace+0x32)[0x5584c19bce72]
 
Nov 09 18:23:22 mysqld[1046371]: /usr/sbin/mysqld(handle_fatal_signal+0x55d)[0x5584c146fbbd]
 
Nov 09 18:23:24 mysqld[1046371]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0)[0x7f0e2269a3c0]
 
Nov 09 18:23:26 mysqld[1046371]: /usr/sbin/mysqld(thd_get_thread_id+0x4)[0x5584c11fa994]
 
Nov 09 18:23:27 mysqld[1046371]: /usr/sbin/mysqld(+0xab7d36)[0x5584c1639d36]
 
Nov 09 18:23:27 mysqld[1046371]: /usr/sbin/mysqld(+0xb082d4)[0x5584c168a2d4]
 
Nov 09 18:23:27 mysqld[1046371]: /usr/sbin/mysqld(+0xb0e3b2)[0x5584c16903b2]
 
Nov 09 18:23:27 mysqld[1046371]: /usr/sbin/mysqld(+0xb9a251)[0x5584c171c251]
 
Nov 09 18:23:27 mysqld[1046371]: /usr/sbin/mysqld(+0xac1979)[0x5584c1643979]
 
Nov 09 18:23:27 mysqld[1046371]: /usr/sbin/mysqld(_ZN7handler10ha_rnd_posEPhS0_+0x50)[0x5584c1476800]
 
Nov 09 18:23:27 mysqld[1046371]: /usr/sbin/mysqld(+0x813890)[0x5584c1395890]
 
Nov 09 18:23:27 mysqld[1046371]: /usr/sbin/mysqld(_ZN14Rows_log_event8find_rowEP14rpl_group_info+0x451)[0x5584c158bd51]
 
Nov 09 18:23:27 mysqld[1046371]: /usr/sbin/mysqld(_ZN21Delete_rows_log_event11do_exec_rowEP14rpl_group_info+0x92)[0x5584c158c012]
 
Nov 09 18:23:27 mysqld[1046371]: /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x23f)[0x5584c157f55f]
 
Nov 09 18:23:27 mysqld[1046371]: /usr/sbin/mysqld(_Z18wsrep_apply_eventsP3THDP14Relay_log_infoPKvm+0x1b9)[0x5584c13e9879]
 
Nov 09 18:23:27 mysqld[1046371]: /usr/sbin/mysqld(_ZN21Wsrep_applier_service15apply_write_setERKN5wsrep7ws_metaERKNS0_12const_bufferERNS0_14mutable_bufferE+0xa3)[0x5584c13d1483]
 
Nov 09 18:23:27 mysqld[1046371]: /usr/sbin/mysqld(+0xea8980)[0x5584c1a2a980]
 
Nov 09 18:23:27 mysqld[1046371]: /usr/sbin/mysqld(_ZN5wsrep12server_state8on_applyERNS_21high_priority_serviceERKNS_9ws_handleERKNS_7ws_metaERKNS_12const_bufferE+0xc1)[0x5584c1a2b841]
 
Nov 09 18:23:27 mysqld[1046371]: /usr/sbin/mysqld(+0xebabec)[0x5584c1a3cbec]
 
Nov 09 18:23:28 mysqld[1046371]: /usr/lib/libgalera_smm.so(+0x639eb)[0x7f0e2140e9eb]
 
Nov 09 18:23:29 mysqld[1046371]: /usr/lib/libgalera_smm.so(+0x7582e)[0x7f0e2142082e]
 
Nov 09 18:23:29 mysqld[1046371]: src/trx_handle.cpp:391(galera::TrxHandleSlave::apply(void*, wsrep_cb_status (*)(void*, wsrep_ws_handle const*, unsigned int, wsrep_buf const*, wsrep_trx_meta const*, bool*), wsrep_trx_meta const&, bool&))[0x7f0e21437dc4]
 
Nov 09 18:23:29 mysqld[1046371]: src/replicator_smm.cpp:504(galera::ReplicatorSMM::apply_trx(void*, galera::TrxHandleSlave&))[0x7f0e21438462]
 
Nov 09 18:23:29 mysqld[1046371]: src/replicator_str.cpp:1131(galera::ReplicatorSMM::process_IST_writeset(void*, boost::shared_ptr<galera::TrxHandleSlave> const&))[0x7f0e21423ea3]
 
Nov 09 18:23:29 mysqld[1046371]: src/replicator_smm.cpp:396(galera::ReplicatorSMM::async_recv(void*))[0x7f0e213f9fe1]
 
Nov 09 18:23:32 mysqld[1046371]: /usr/sbin/mysqld(_ZN5wsrep18wsrep_provider_v2611run_applierEPNS_21high_priority_serviceE+0x12)[0x5584c1a3d202]
 
Nov 09 18:23:32 mysqld[1046371]: /usr/sbin/mysqld(+0x869693)[0x5584c13eb693]
 
Nov 09 18:23:32 mysqld[1046371]: /usr/sbin/mysqld(_Z15start_wsrep_THDPv+0x2ec)[0x5584c13dc48c]
 
Nov 09 18:23:33 mysqld[1046371]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609)[0x7f0e2268e609]
 
Nov 09 18:23:36 mysqld[1046371]: 2021-11-09 18:23:36 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT1.57696S), skipping check
 
Nov 09 18:23:36 mysqld[1046371]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x43)[0x7f0e2227c293]
 
Nov 09 18:23:36 mysqld[1046371]: Trying to get some variables.
 
Nov 09 18:23:36 mysqld[1046371]: Some pointers may be invalid and cause the dump to abort.
 
Nov 09 18:23:36 mysqld[1046371]: Query (0x7ef005bbb433): delete from problem where r_clock<>0 and r_clock<1636395301
 
Nov 09 18:23:36 mysqld[1046371]: Connection ID (thread ID): 31
 
Nov 09 18:23:36 mysqld[1046371]: Status: NOT_KILLED
 
Nov 09 18:23:36 mysqld[1046371]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on
 
Nov 09 18:23:36 mysqld[1046371]: The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
 
Nov 09 18:23:36 mysqld[1046371]: information that should help you find out what is causing the crash.
 
Nov 09 18:23:36 mysqld[1046371]: Writing a core file...
 
Nov 09 18:23:36 mysqld[1046371]: Working directory at /var/lib/mysql
 
Nov 09 18:23:36 mysqld[1046371]: Resource Limits:
 
Nov 09 18:23:36 mysqld[1046371]: Limit                     Soft Limit           Hard Limit           Units
 
Nov 09 18:23:36 mysqld[1046371]: Max cpu time              unlimited            unlimited            seconds
 
Nov 09 18:23:36 mysqld[1046371]: Max file size             unlimited            unlimited            bytes
 
Nov 09 18:23:36 mysqld[1046371]: Max data size             unlimited            unlimited            bytes
 
Nov 09 18:23:36 mysqld[1046371]: Max stack size            8388608              unlimited            bytes
 
Nov 09 18:23:36 mysqld[1046371]: Max core file size        0                    unlimited            bytes
 
Nov 09 18:23:36 mysqld[1046371]: Max resident set          unlimited            unlimited            bytes
 
Nov 09 18:23:36 mysqld[1046371]: Max processes             6186824              6186824              processes
 
Nov 09 18:23:36 mysqld[1046371]: Max open files            65536                65536                files
 
Nov 09 18:23:36 mysqld[1046371]: Max locked memory         65536                65536                bytes
 
Nov 09 18:23:36 mysqld[1046371]: Max address space         unlimited            unlimited            bytes
 
Nov 09 18:23:36 mysqld[1046371]: Max file locks            unlimited            unlimited            locks
 
Nov 09 18:23:36 mysqld[1046371]: Max pending signals       6186824              6186824              signals
 
Nov 09 18:23:36 mysqld[1046371]: Max msgqueue size         819200               819200               bytes
 
Nov 09 18:23:36 mysqld[1046371]: Max nice priority         0                    0
 
Nov 09 18:23:36 mysqld[1046371]: Max realtime priority     0                    0
 
Nov 09 18:23:36 mysqld[1046371]: Max realtime timeout      unlimited            unlimited            us
 
Nov 09 18:23:36 mysqld[1046371]: Core pattern: /var/crash/%e.%p
 
Nov 09 18:23:43 systemd[1]: mariadb.service: Main process exited, code=killed, status=11/SEGV

Comment by Mark Reibert [ 2021-11-09 ]

A couple of notes:

  1. I first started seeing this problem when I upgraded to 10.4.19. I was running that for a short time because my servers were Ubuntu 16.04, and 10.4.19 is the latest available for Ubuntu 16.04.
  2. In an attempt to get past 10.4.19 (where the problem first appeared) I upgraded my servers to Ubuntu 20.04 specifically so I could get the latest MariaDB 10.4 (which was 10.4.21 at the time). Unfortunately, the problem still occurs with 10.4.21.
  3. This looks similar to MDEV-26099, which is written against MariaDB 10.5.
  4. This could also be related to MDEV-25718, which is another transaction.is_streaming() assertion failure. That one is a little different because it does not have the mode_ == m_local OR condition in the assertion, but perhaps the transaction.is_streaming() is the culprit here.

So with MDEV-25718 and MDEV-26099, is it possible this issue has therefore been fixed in 10.4.22 (as it was in 10.5.13)? One can hope, because this is a serious issue. In my particular situation I have not been able to have a fully operational cluster since upgrading to MariaDB 10.4.19. In fact, I have been largely running with only one of my three nodes online because any time I try to bring up another node it just fails.

Comment by Mark Reibert [ 2021-11-10 ]

It looks like this could also be MDEV-25551, which seems to have been swept under the rug as the result of a table with no PK. That's … unfortunately because nothing in SQL requires a table to have a PK!

Note also MDEV-26298, which does seem to have a fix in place for 10.4.22.

So again, perhaps this is fixed in 10.4.22 by way of one of the other myriad Jira issues circling around this issue!

Comment by Jan Lindström (Inactive) [ 2021-12-02 ]

Can you share table structure from jobs or any other table in stack trace?

Comment by Mark Reibert [ 2021-12-02 ]

Unfortunately I cannot. My situation is a 9.5TB Zabbix database with 147 tables, and to be honest the queries Zabbix issues are now always … shall we say "ideal" for such a large database. That aside, I do not know how to correlate the crash with a particular query (if that is even possible).

What I can tell you is whereas the crashes were epidemic with MariaDB 10.4.19 and 10.4.21, I have not experienced a single incident since upgrading to 10.4.22.

Comment by Mark Reibert [ 2021-12-03 ]

It seems I jinxed 10.4.22. Not more than an hour or two after I typed that previous comment claiming "all is fine" I had a node SIGABRT with 10.4.22. Doh!

Here is the full log from the crash:

Dec 02 21:15:02 mysqld[1566419]: mysqld: /home/buildbot/buildbot/build/mariadb-10.4.22/wsrep-lib/include/wsrep/client_state.hpp:668: int wsrep::client_state::bf_abort(wsrep::seqno): Assertion `mode_ == m_local || transaction_.is_streaming()' failed.
 
Dec 02 21:15:02 mysqld[1566419]: 211202 21:15:02 [ERROR] mysqld got signal 6 ;
 
Dec 02 21:15:02 mysqld[1566419]: This could be because you hit a bug. It is also possible that this binary
 
Dec 02 21:15:02 mysqld[1566419]: or one of the libraries it was linked against is corrupt, improperly built,
 
Dec 02 21:15:02 mysqld[1566419]: or misconfigured. This error can also be caused by malfunctioning hardware.
 
Dec 02 21:15:02 mysqld[1566419]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
Dec 02 21:15:02 mysqld[1566419]: We will try our best to scrape up some info that will hopefully help
 
Dec 02 21:15:02 mysqld[1566419]: diagnose the problem, but since we have already crashed,
 
Dec 02 21:15:02 mysqld[1566419]: something is definitely wrong and this may fail.
 
Dec 02 21:15:02 mysqld[1566419]: Server version: 10.4.22-MariaDB-1:10.4.22+maria~focal-log
 
Dec 02 21:15:02 mysqld[1566419]: key_buffer_size=134217728
 
Dec 02 21:15:02 mysqld[1566419]: read_buffer_size=131072
 
Dec 02 21:15:02 mysqld[1566419]: max_used_connections=3
 
Dec 02 21:15:02 mysqld[1566419]: max_threads=403
 
Dec 02 21:15:02 mysqld[1566419]: thread_count=73
 
Dec 02 21:15:02 mysqld[1566419]: It is possible that mysqld could use up to
 
Dec 02 21:15:02 mysqld[1566419]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6795376 K  bytes of memory
 
Dec 02 21:15:02 mysqld[1566419]: Hope that's ok; if not, decrease some variables in the equation.
 
Dec 02 21:15:02 mysqld[1566419]: Thread pointer: 0x7e020c000c48
 
Dec 02 21:15:02 mysqld[1566419]: Attempting backtrace. You can use the following information to find out
 
Dec 02 21:15:02 mysqld[1566419]: where mysqld died. If you see no messages after this, something went
 
Dec 02 21:15:02 mysqld[1566419]: terribly wrong...
 
Dec 02 21:15:02 mysqld[1566419]: stack_bottom = 0x7e031c473e08 thread_stack 0x49000
 
Dec 02 21:15:09 mysqld[1566419]: 2021-12-02 21:15:09 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection to peer 1e77c150-8d79 with addr tcp://192.168.77.160:4567 timed out, no messages seen in PT3S, socket stats: rtt: 1257 rttvar: 1806 rto: 204000 lost: 0 last_data_recv: 3252 cwnd: 10 last_queued_since: 25298782 last_delivered_since: 6616014371 send_queue_length: 0 send_queue_bytes: 0 segment: 0 messages: 0
 
Dec 02 21:15:09 mysqld[1566419]: 2021-12-02 21:15:09 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://192.168.77.160:4567
 
Dec 02 21:15:09 mysqld[1566419]: 2021-12-02 21:15:09 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT6.87437S), skipping check
 
Dec 02 21:15:09 mysqld[1566419]: 2021-12-02 21:15:09 0 [Note] WSREP: evs::proto(2f8c7fd2-91e6, OPERATIONAL, view_id(REG,1e77c150-8d79,104)) suspecting node: 1e77c150-8d79
 
Dec 02 21:15:09 mysqld[1566419]: 2021-12-02 21:15:09 0 [Note] WSREP: evs::proto(2f8c7fd2-91e6, OPERATIONAL, view_id(REG,1e77c150-8d79,104)) suspected node without join message, declaring inactive
 
Dec 02 21:15:10 mysqld[1566419]: 2021-12-02 21:15:10 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') reconnecting to 1e77c150-8d79 (tcp://192.168.77.160:4567), attempt 0
 
Dec 02 21:15:10 mysqld[1566419]: 2021-12-02 21:15:10 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 1e77c150-8d79 tcp://192.168.77.160:4567
 
Dec 02 21:15:10 mysqld[1566419]: 2021-12-02 21:15:10 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') reconnecting to 58fa90ba-871c (tcp://192.168.77.162:4567), attempt 0
 
Dec 02 21:15:10 mysqld[1566419]: 2021-12-02 21:15:10 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 58fa90ba-871c tcp://192.168.77.162:4567
 
Dec 02 21:15:11 mysqld[1566419]: 2021-12-02 21:15:11 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') reconnecting to 1e77c150-8d79 (tcp://192.168.77.160:4567), attempt 0
 
Dec 02 21:15:11 mysqld[1566419]: 2021-12-02 21:15:11 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 1e77c150-8d79 tcp://192.168.77.160:4567
 
Dec 02 21:15:11 mysqld[1566419]: 2021-12-02 21:15:11 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') reconnecting to 58fa90ba-871c (tcp://192.168.77.162:4567), attempt 0
 
Dec 02 21:15:11 mysqld[1566419]: 2021-12-02 21:15:11 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 58fa90ba-871c tcp://192.168.77.162:4567
 
Dec 02 21:15:14 mysqld[1566419]: 2021-12-02 21:15:14 0 [Note] WSREP: evs::proto(2f8c7fd2-91e6, GATHER, view_id(REG,1e77c150-8d79,104)) suspecting node: 58fa90ba-871c
 
Dec 02 21:15:14 mysqld[1566419]: 2021-12-02 21:15:14 0 [Note] WSREP: evs::proto(2f8c7fd2-91e6, GATHER, view_id(REG,1e77c150-8d79,104)) suspected node without join message, declaring inactive
 
Dec 02 21:15:14 mysqld[1566419]: 2021-12-02 21:15:14 0 [Note] WSREP: view(view_id(NON_PRIM,1e77c150-8d79,104) memb {
 
Dec 02 21:15:14 mysqld[1566419]:         2f8c7fd2-91e6,0
 
Dec 02 21:15:14 mysqld[1566419]: } joined {
 
Dec 02 21:15:14 mysqld[1566419]: } left {
 
Dec 02 21:15:14 mysqld[1566419]: } partitioned {
 
Dec 02 21:15:14 mysqld[1566419]:         1e77c150-8d79,0
 
Dec 02 21:15:14 mysqld[1566419]:         58fa90ba-871c,0
 
Dec 02 21:15:14 mysqld[1566419]: })
 
Dec 02 21:15:14 mysqld[1566419]: 2021-12-02 21:15:14 0 [Note] WSREP: view(view_id(NON_PRIM,2f8c7fd2-91e6,105) memb {
 
Dec 02 21:15:14 mysqld[1566419]:         2f8c7fd2-91e6,0
 
Dec 02 21:15:14 mysqld[1566419]: } joined {
 
Dec 02 21:15:14 mysqld[1566419]: } left {
 
Dec 02 21:15:14 mysqld[1566419]: } partitioned {
 
Dec 02 21:15:14 mysqld[1566419]:         1e77c150-8d79,0
 
Dec 02 21:15:14 mysqld[1566419]:         58fa90ba-871c,0
 
Dec 02 21:15:14 mysqld[1566419]: })
 
Dec 02 21:15:14 mysqld[1566419]: 2021-12-02 21:15:14 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
 
Dec 02 21:15:14 mysqld[1566419]: 2021-12-02 21:15:14 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
 
Dec 02 21:15:14 mysqld[1566419]: 2021-12-02 21:15:14 0 [Note] WSREP: Received NON-PRIMARY.
 
Dec 02 21:15:14 mysqld[1566419]: 2021-12-02 21:15:14 0 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 4857087606)
 
Dec 02 21:15:14 mysqld[1566419]: 2021-12-02 21:15:14 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
 
Dec 02 21:15:14 mysqld[1566419]: 2021-12-02 21:15:14 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
 
Dec 02 21:15:14 mysqld[1566419]: 2021-12-02 21:15:14 0 [Note] WSREP: Received NON-PRIMARY.
 
Dec 02 21:15:15 mysqld[1566419]: 2021-12-02 21:15:15 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') turning message relay requesting off
 
Dec 02 21:15:15 mysqld[1566419]: 2021-12-02 21:15:15 0 [Note] WSREP: declaring 1e77c150-8d79 at tcp://192.168.77.160:4567 stable
 
Dec 02 21:15:15 mysqld[1566419]: 2021-12-02 21:15:15 0 [Note] WSREP: declaring 58fa90ba-871c at tcp://192.168.77.162:4567 stable
 
Dec 02 21:15:15 mysqld[1566419]: 2021-12-02 21:15:15 0 [Note] WSREP: Node 1e77c150-8d79 state prim
 
Dec 02 21:15:15 mysqld[1566419]: 2021-12-02 21:15:15 0 [Note] WSREP: view(view_id(PRIM,1e77c150-8d79,106) memb {
 
Dec 02 21:15:15 mysqld[1566419]:         1e77c150-8d79,0
 
Dec 02 21:15:15 mysqld[1566419]:         2f8c7fd2-91e6,0
 
Dec 02 21:15:15 mysqld[1566419]:         58fa90ba-871c,0
 
Dec 02 21:15:15 mysqld[1566419]: } joined {
 
Dec 02 21:15:15 mysqld[1566419]: } left {
 
Dec 02 21:15:15 mysqld[1566419]: } partitioned {
 
Dec 02 21:15:15 mysqld[1566419]: })
 
Dec 02 21:15:15 mysqld[1566419]: 2021-12-02 21:15:15 0 [Note] WSREP: save pc into disk
 
Dec 02 21:15:15 mysqld[1566419]: 2021-12-02 21:15:15 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 3
 
Dec 02 21:15:15 mysqld[1566419]: 2021-12-02 21:15:15 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
 
Dec 02 21:15:15 mysqld[1566419]: 2021-12-02 21:15:15 0 [Note] WSREP: STATE EXCHANGE: sent state msg: f1ed7f1a-53b4-11ec-a50b-9e152798ae9a
 
Dec 02 21:15:15 mysqld[1566419]: 2021-12-02 21:15:15 0 [Note] WSREP: STATE EXCHANGE: got state msg: f1ed7f1a-53b4-11ec-a50b-9e152798ae9a from 0 (zabbix-db04)
 
Dec 02 21:15:15 mysqld[1566419]: 2021-12-02 21:15:15 0 [Note] WSREP: STATE EXCHANGE: got state msg: f1ed7f1a-53b4-11ec-a50b-9e152798ae9a from 1 (zabbix-db05)
 
Dec 02 21:15:15 mysqld[1566419]: 2021-12-02 21:15:15 0 [Note] WSREP: STATE EXCHANGE: got state msg: f1ed7f1a-53b4-11ec-a50b-9e152798ae9a from 2 (zabbix-db06)
 
Dec 02 21:15:15 mysqld[1566419]: 2021-12-02 21:15:15 0 [Note] WSREP: Quorum results:
 
Dec 02 21:15:15 mysqld[1566419]:         version    = 6,
 
Dec 02 21:15:15 mysqld[1566419]:         component  = PRIMARY,
 
Dec 02 21:15:15 mysqld[1566419]:         conf_id    = 39,
 
Dec 02 21:15:15 mysqld[1566419]:         members    = 2/3 (joined/total),
 
Dec 02 21:15:15 mysqld[1566419]:         act_id     = 4857088711,
 
Dec 02 21:15:15 mysqld[1566419]:         last_appl. = 4857088690,
 
Dec 02 21:15:15 mysqld[1566419]:         protocols  = 2/10/4 (gcs/repl/appl),
 
Dec 02 21:15:15 mysqld[1566419]:         vote policy= 0,
 
Dec 02 21:15:15 mysqld[1566419]:         group UUID = 810a5d0c-231c-11e9-b756-dbb7899d7dd0
 
Dec 02 21:15:15 mysqld[1566419]: 2021-12-02 21:15:15 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
 
Dec 02 21:15:15 mysqld[1566419]: 2021-12-02 21:15:15 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 4857088712)
 
Dec 02 21:15:18 mysqld[1566419]: /usr/sbin/mysqld(my_print_stacktrace+0x32)[0x564297590d12]
 
Dec 02 21:15:18 mysqld[1566419]: /usr/sbin/mysqld(handle_fatal_signal+0x55d)[0x564297042a5d]
 
Dec 02 21:15:23 mysqld[1566419]: 2021-12-02 21:15:23 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection to peer 1e77c150-8d79 with addr tcp://192.168.77.160:4567 timed out, no messages seen in PT3S, socket stats: rtt: 2620 rttvar: 3448 rto: 204000 lost: 0 last_data_recv: 2696 cwnd: 10 last_queued_since: 559365 last_delivered_since: 5870988113 send_queue_length: 0 send_queue_bytes: 0 segment: 0 messages: 0
 
Dec 02 21:15:23 mysqld[1566419]: 2021-12-02 21:15:23 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://192.168.77.160:4567
 
Dec 02 21:15:23 mysqld[1566419]: 2021-12-02 21:15:23 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT5.93833S), skipping check
 
Dec 02 21:15:23 mysqld[1566419]: 2021-12-02 21:15:23 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 1e77c150-8d79 tcp://192.168.77.160:4567
 
Dec 02 21:15:23 mysqld[1566419]: 2021-12-02 21:15:23 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 58fa90ba-871c tcp://192.168.77.162:4567
 
Dec 02 21:15:24 mysqld[1566419]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0)[0x7f348779e3c0]
 
Dec 02 21:15:24 mysqld[1566419]: 2021-12-02 21:15:24 0 [Note] WSREP: evs::proto(2f8c7fd2-91e6, OPERATIONAL, view_id(REG,1e77c150-8d79,106)) suspecting node: 1e77c150-8d79
 
Dec 02 21:15:24 mysqld[1566419]: 2021-12-02 21:15:24 0 [Note] WSREP: evs::proto(2f8c7fd2-91e6, OPERATIONAL, view_id(REG,1e77c150-8d79,106)) suspected node without join message, declaring inactive
 
Dec 02 21:15:24 mysqld[1566419]: 2021-12-02 21:15:24 0 [Note] WSREP: view(view_id(NON_PRIM,1e77c150-8d79,106) memb {
 
Dec 02 21:15:24 mysqld[1566419]:         2f8c7fd2-91e6,0
 
Dec 02 21:15:24 mysqld[1566419]: } joined {
 
Dec 02 21:15:24 mysqld[1566419]: } left {
 
Dec 02 21:15:24 mysqld[1566419]: } partitioned {
 
Dec 02 21:15:24 mysqld[1566419]:         1e77c150-8d79,0
 
Dec 02 21:15:24 mysqld[1566419]:         58fa90ba-871c,0
 
Dec 02 21:15:24 mysqld[1566419]: })
 
Dec 02 21:15:24 mysqld[1566419]: 2021-12-02 21:15:24 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
 
Dec 02 21:15:24 mysqld[1566419]: 2021-12-02 21:15:24 0 [Note] WSREP: view(view_id(NON_PRIM,2f8c7fd2-91e6,107) memb {
 
Dec 02 21:15:24 mysqld[1566419]:         2f8c7fd2-91e6,0
 
Dec 02 21:15:24 mysqld[1566419]: } joined {
 
Dec 02 21:15:24 mysqld[1566419]: } left {
 
Dec 02 21:15:24 mysqld[1566419]: } partitioned {
 
Dec 02 21:15:24 mysqld[1566419]:         1e77c150-8d79,0
 
Dec 02 21:15:24 mysqld[1566419]:         58fa90ba-871c,0
 
Dec 02 21:15:24 mysqld[1566419]: })
 
Dec 02 21:15:24 mysqld[1566419]: 2021-12-02 21:15:24 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
 
Dec 02 21:15:24 mysqld[1566419]: 2021-12-02 21:15:24 0 [Note] WSREP: Received NON-PRIMARY.
 
Dec 02 21:15:24 mysqld[1566419]: 2021-12-02 21:15:24 0 [Note] WSREP: Shifting PRIMARY -> OPEN (TO: 4857089108)
 
Dec 02 21:15:24 mysqld[1566419]: 2021-12-02 21:15:24 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
 
Dec 02 21:15:24 mysqld[1566419]: 2021-12-02 21:15:24 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
 
Dec 02 21:15:24 mysqld[1566419]: 2021-12-02 21:15:24 0 [Note] WSREP: Received NON-PRIMARY.
 
Dec 02 21:15:25 mysqld[1566419]: 2021-12-02 21:15:25 0 [Note] WSREP: declaring 1e77c150-8d79 at tcp://192.168.77.160:4567 stable
 
Dec 02 21:15:25 mysqld[1566419]: 2021-12-02 21:15:25 0 [Note] WSREP: declaring 58fa90ba-871c at tcp://192.168.77.162:4567 stable
 
Dec 02 21:15:25 mysqld[1566419]: 2021-12-02 21:15:25 0 [Note] WSREP: Node 1e77c150-8d79 state prim
 
Dec 02 21:15:25 mysqld[1566419]: 2021-12-02 21:15:25 0 [Note] WSREP: view(view_id(PRIM,1e77c150-8d79,108) memb {
 
Dec 02 21:15:25 mysqld[1566419]:         1e77c150-8d79,0
 
Dec 02 21:15:25 mysqld[1566419]:         2f8c7fd2-91e6,0
 
Dec 02 21:15:25 mysqld[1566419]:         58fa90ba-871c,0
 
Dec 02 21:15:25 mysqld[1566419]: } joined {
 
Dec 02 21:15:25 mysqld[1566419]: } left {
 
Dec 02 21:15:25 mysqld[1566419]: } partitioned {
 
Dec 02 21:15:25 mysqld[1566419]: })
 
Dec 02 21:15:25 mysqld[1566419]: 2021-12-02 21:15:25 0 [Note] WSREP: save pc into disk
 
Dec 02 21:15:25 mysqld[1566419]: 2021-12-02 21:15:25 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 3
 
Dec 02 21:15:25 mysqld[1566419]: 2021-12-02 21:15:25 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
 
Dec 02 21:15:25 mysqld[1566419]: 2021-12-02 21:15:25 0 [Note] WSREP: STATE EXCHANGE: sent state msg: f79778e9-53b4-11ec-9167-0b066174fefa
 
Dec 02 21:15:25 mysqld[1566419]: 2021-12-02 21:15:25 0 [Note] WSREP: STATE EXCHANGE: got state msg: f79778e9-53b4-11ec-9167-0b066174fefa from 0 (zabbix-db04)
 
Dec 02 21:15:25 mysqld[1566419]: 2021-12-02 21:15:25 0 [Note] WSREP: STATE EXCHANGE: got state msg: f79778e9-53b4-11ec-9167-0b066174fefa from 1 (zabbix-db05)
 
Dec 02 21:15:25 mysqld[1566419]: 2021-12-02 21:15:25 0 [Note] WSREP: STATE EXCHANGE: got state msg: f79778e9-53b4-11ec-9167-0b066174fefa from 2 (zabbix-db06)
 
Dec 02 21:15:25 mysqld[1566419]: 2021-12-02 21:15:25 0 [Note] WSREP: Quorum results:
 
Dec 02 21:15:25 mysqld[1566419]:         version    = 6,
 
Dec 02 21:15:25 mysqld[1566419]:         component  = PRIMARY,
 
Dec 02 21:15:25 mysqld[1566419]:         conf_id    = 40,
 
Dec 02 21:15:25 mysqld[1566419]:         members    = 2/3 (joined/total),
 
Dec 02 21:15:25 mysqld[1566419]:         act_id     = 4857089126,
 
Dec 02 21:15:25 mysqld[1566419]:         last_appl. = 4857088690,
 
Dec 02 21:15:25 mysqld[1566419]:         protocols  = 2/10/4 (gcs/repl/appl),
 
Dec 02 21:15:25 mysqld[1566419]:         vote policy= 0,
 
Dec 02 21:15:25 mysqld[1566419]:         group UUID = 810a5d0c-231c-11e9-b756-dbb7899d7dd0
 
Dec 02 21:15:25 mysqld[1566419]: 2021-12-02 21:15:25 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
 
Dec 02 21:15:25 mysqld[1566419]: 2021-12-02 21:15:25 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 4857089127)
 
Dec 02 21:15:27 mysqld[1566419]: 2021-12-02 21:15:27 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') turning message relay requesting off
 
Dec 02 21:15:38 mysqld[1566419]: 2021-12-02 21:15:38 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection to peer 1e77c150-8d79 with addr tcp://192.168.77.160:4567 timed out, no messages seen in PT3S, socket stats: rtt: 1508 rttvar: 2174 rto: 204000 lost: 0 last_data_recv: 3732 cwnd: 10 last_queued_since: 412691 last_delivered_since: 6254106919 send_queue_length: 1 send_queue_bytes: 80 segment: 0 messages: 1
 
Dec 02 21:15:38 mysqld[1566419]: 2021-12-02 21:15:38 0 [Note] WSREP: Deferred close timer started for socket with remote endpoint: tcp://192.168.77.160:51202
 
Dec 02 21:15:38 mysqld[1566419]: 2021-12-02 21:15:38 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://192.168.77.160:4567
 
Dec 02 21:15:38 mysqld[1566419]: 2021-12-02 21:15:38 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT6.52657S), skipping check
 
Dec 02 21:15:38 mysqld[1566419]: 2021-12-02 21:15:38 0 [Note] WSREP: Deferred close timer handle_wait Operation aborted. for 0x7f13a003fb10
 
Dec 02 21:15:38 mysqld[1566419]: 2021-12-02 21:15:38 0 [Note] WSREP: Deferred close timer destruct
 
Dec 02 21:15:39 mysqld[1566419]: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x7f34872a418b]
 
Dec 02 21:15:39 mysqld[1566419]: 2021-12-02 21:15:39 0 [Note] WSREP: evs::proto(2f8c7fd2-91e6, OPERATIONAL, view_id(REG,1e77c150-8d79,108)) suspecting node: 1e77c150-8d79
 
Dec 02 21:15:39 mysqld[1566419]: 2021-12-02 21:15:39 0 [Note] WSREP: evs::proto(2f8c7fd2-91e6, OPERATIONAL, view_id(REG,1e77c150-8d79,108)) suspected node without join message, declaring inactive
 
Dec 02 21:15:39 mysqld[1566419]: /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b)[0x7f3487283859]
 
Dec 02 21:15:39 mysqld[1566419]: 2021-12-02 21:15:39 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') reconnecting to 1e77c150-8d79 (tcp://192.168.77.160:4567), attempt 0
 
Dec 02 21:15:39 mysqld[1566419]: 2021-12-02 21:15:39 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 1e77c150-8d79 tcp://192.168.77.160:4567
 
Dec 02 21:15:40 mysqld[1566419]: 2021-12-02 21:15:40 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') reconnecting to 58fa90ba-871c (tcp://192.168.77.162:4567), attempt 0
 
Dec 02 21:15:40 mysqld[1566419]: /lib/x86_64-linux-gnu/libc.so.6(+0x25729)[0x7f3487283729]
 
Dec 02 21:15:40 mysqld[1566419]: 2021-12-02 21:15:40 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 58fa90ba-871c tcp://192.168.77.162:4567
 
Dec 02 21:15:40 mysqld[1566419]: /lib/x86_64-linux-gnu/libc.so.6(+0x36f36)[0x7f3487294f36]
 
Dec 02 21:15:41 mysqld[1566419]: 2021-12-02 21:15:41 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') reconnecting to 1e77c150-8d79 (tcp://192.168.77.160:4567), attempt 0
 
Dec 02 21:15:41 mysqld[1566419]: 2021-12-02 21:15:41 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 1e77c150-8d79 tcp://192.168.77.160:4567
 
Dec 02 21:15:41 mysqld[1566419]: 2021-12-02 21:15:41 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') reconnecting to 58fa90ba-871c (tcp://192.168.77.162:4567), attempt 0
 
Dec 02 21:15:41 mysqld[1566419]: 2021-12-02 21:15:41 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 58fa90ba-871c tcp://192.168.77.162:4567
 
Dec 02 21:15:41 mysqld[1566419]: 2021-12-02 21:15:41 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 1e77c150-8d79 tcp://192.168.77.160:4567
 
Dec 02 21:15:43 mysqld[1566419]: 2021-12-02 21:15:43 0 [Note] WSREP: evs::proto(2f8c7fd2-91e6, GATHER, view_id(REG,1e77c150-8d79,108)) suspecting node: 58fa90ba-871c
 
Dec 02 21:15:43 mysqld[1566419]: 2021-12-02 21:15:43 0 [Note] WSREP: evs::proto(2f8c7fd2-91e6, GATHER, view_id(REG,1e77c150-8d79,108)) suspected node without join message, declaring inactive
 
Dec 02 21:15:44 mysqld[1566419]: 2021-12-02 21:15:44 0 [Note] WSREP: view(view_id(NON_PRIM,1e77c150-8d79,108) memb {
 
Dec 02 21:15:44 mysqld[1566419]:         2f8c7fd2-91e6,0
 
Dec 02 21:15:44 mysqld[1566419]: } joined {
 
Dec 02 21:15:44 mysqld[1566419]: } left {
 
Dec 02 21:15:44 mysqld[1566419]: } partitioned {
 
Dec 02 21:15:44 mysqld[1566419]:         1e77c150-8d79,0
 
Dec 02 21:15:44 mysqld[1566419]:         58fa90ba-871c,0
 
Dec 02 21:15:44 mysqld[1566419]: })
 
Dec 02 21:15:44 mysqld[1566419]: 2021-12-02 21:15:44 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
 
Dec 02 21:15:44 mysqld[1566419]: 2021-12-02 21:15:44 0 [Note] WSREP: view(view_id(NON_PRIM,2f8c7fd2-91e6,109) memb {
 
Dec 02 21:15:44 mysqld[1566419]:         2f8c7fd2-91e6,0
 
Dec 02 21:15:44 mysqld[1566419]: } joined {
 
Dec 02 21:15:44 mysqld[1566419]: } left {
 
Dec 02 21:15:44 mysqld[1566419]: } partitioned {
 
Dec 02 21:15:44 mysqld[1566419]:         1e77c150-8d79,0
 
Dec 02 21:15:44 mysqld[1566419]:         58fa90ba-871c,0
 
Dec 02 21:15:44 mysqld[1566419]: })
 
Dec 02 21:15:44 mysqld[1566419]: 2021-12-02 21:15:44 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
 
Dec 02 21:15:44 mysqld[1566419]: 2021-12-02 21:15:44 0 [Note] WSREP: Received NON-PRIMARY.
 
Dec 02 21:15:44 mysqld[1566419]: 2021-12-02 21:15:44 0 [Note] WSREP: Shifting PRIMARY -> OPEN (TO: 4857090486)
 
Dec 02 21:15:44 mysqld[1566419]: 2021-12-02 21:15:44 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
 
Dec 02 21:15:44 mysqld[1566419]: 2021-12-02 21:15:44 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
 
Dec 02 21:15:44 mysqld[1566419]: 2021-12-02 21:15:44 0 [Note] WSREP: Received NON-PRIMARY.
 
Dec 02 21:15:45 mysqld[1566419]: 2021-12-02 21:15:45 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') turning message relay requesting off
 
Dec 02 21:15:46 mysqld[1566419]: 2021-12-02 21:15:46 0 [Note] WSREP: declaring 1e77c150-8d79 at tcp://192.168.77.160:4567 stable
 
Dec 02 21:15:46 mysqld[1566419]: 2021-12-02 21:15:46 0 [Note] WSREP: declaring 58fa90ba-871c at tcp://192.168.77.162:4567 stable
 
Dec 02 21:15:46 mysqld[1566419]: 2021-12-02 21:15:46 0 [Note] WSREP: Node 1e77c150-8d79 state prim
 
Dec 02 21:15:46 mysqld[1566419]: 2021-12-02 21:15:46 0 [Note] WSREP: view(view_id(PRIM,1e77c150-8d79,110) memb {
 
Dec 02 21:15:46 mysqld[1566419]:         1e77c150-8d79,0
 
Dec 02 21:15:46 mysqld[1566419]:         2f8c7fd2-91e6,0
 
Dec 02 21:15:46 mysqld[1566419]:         58fa90ba-871c,0
 
Dec 02 21:15:46 mysqld[1566419]: } joined {
 
Dec 02 21:15:46 mysqld[1566419]: } left {
 
Dec 02 21:15:46 mysqld[1566419]: } partitioned {
 
Dec 02 21:15:46 mysqld[1566419]: })
 
Dec 02 21:15:46 mysqld[1566419]: 2021-12-02 21:15:46 0 [Note] WSREP: save pc into disk
 
Dec 02 21:15:46 mysqld[1566419]: 2021-12-02 21:15:46 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 3
 
Dec 02 21:15:46 mysqld[1566419]: 2021-12-02 21:15:46 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
 
Dec 02 21:15:46 mysqld[1566419]: 2021-12-02 21:15:46 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 03d428c1-53b5-11ec-be9c-4bd7ca174788
 
Dec 02 21:15:46 mysqld[1566419]: 2021-12-02 21:15:46 0 [Note] WSREP: STATE EXCHANGE: got state msg: 03d428c1-53b5-11ec-be9c-4bd7ca174788 from 0 (zabbix-db04)
 
Dec 02 21:15:46 mysqld[1566419]: 2021-12-02 21:15:46 0 [Note] WSREP: STATE EXCHANGE: got state msg: 03d428c1-53b5-11ec-be9c-4bd7ca174788 from 1 (zabbix-db05)
 
Dec 02 21:15:46 mysqld[1566419]: 2021-12-02 21:15:46 0 [Note] WSREP: STATE EXCHANGE: got state msg: 03d428c1-53b5-11ec-be9c-4bd7ca174788 from 2 (zabbix-db06)
 
Dec 02 21:15:46 mysqld[1566419]: 2021-12-02 21:15:46 0 [Note] WSREP: Quorum results:
 
Dec 02 21:15:46 mysqld[1566419]:         version    = 6,
 
Dec 02 21:15:46 mysqld[1566419]:         component  = PRIMARY,
 
Dec 02 21:15:46 mysqld[1566419]:         conf_id    = 42,
 
Dec 02 21:15:46 mysqld[1566419]:         members    = 2/3 (joined/total),
 
Dec 02 21:15:46 mysqld[1566419]:         act_id     = 4857091473,
 
Dec 02 21:15:46 mysqld[1566419]:         last_appl. = 4857091469,
 
Dec 02 21:15:46 mysqld[1566419]:         protocols  = 2/10/4 (gcs/repl/appl),
 
Dec 02 21:15:46 mysqld[1566419]:         vote policy= 0,
 
Dec 02 21:15:46 mysqld[1566419]:         group UUID = 810a5d0c-231c-11e9-b756-dbb7899d7dd0
 
Dec 02 21:15:46 mysqld[1566419]: 2021-12-02 21:15:46 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
 
Dec 02 21:15:46 mysqld[1566419]: 2021-12-02 21:15:46 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 4857091474)
 
Dec 02 21:15:52 mysqld[1566419]: 2021-12-02 21:15:52 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection to peer 1e77c150-8d79 with addr tcp://192.168.77.160:4567 timed out, no messages seen in PT3S, socket stats: rtt: 1209 rttvar: 1476 rto: 204000 lost: 0 last_data_recv: 2284 cwnd: 10 last_queued_since: 363245 last_delivered_since: 5600232142 send_queue_length: 6 send_queue_bytes: 480 segment: 0 messages: 6
 
Dec 02 21:15:52 mysqld[1566419]: 2021-12-02 21:15:52 0 [Note] WSREP: Deferred close timer started for socket with remote endpoint: tcp://192.168.77.160:51208
 
Dec 02 21:15:52 mysqld[1566419]: 2021-12-02 21:15:52 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://192.168.77.160:4567
 
Dec 02 21:15:52 mysqld[1566419]: 2021-12-02 21:15:52 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT5.99973S), skipping check
 
Dec 02 21:15:52 mysqld[1566419]: 2021-12-02 21:15:52 0 [Note] WSREP: Deferred close timer handle_wait Operation aborted. for 0x56429cac2270
 
Dec 02 21:15:52 mysqld[1566419]: 2021-12-02 21:15:52 0 [Note] WSREP: Deferred close timer destruct
 
Dec 02 21:15:52 mysqld[1566419]: 2021-12-02 21:15:52 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 1e77c150-8d79 tcp://192.168.77.160:4567
 
Dec 02 21:15:52 mysqld[1566419]: 2021-12-02 21:15:52 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 58fa90ba-871c tcp://192.168.77.162:4567
 
Dec 02 21:15:52 mysqld[1566419]: /usr/sbin/mysqld(_Z14wsrep_bf_abortP3THDS0_+0x6e9)[0x564296fbd7f9]
 
Dec 02 21:15:53 mysqld[1566419]: 2021-12-02 21:15:53 0 [Note] WSREP: evs::proto(2f8c7fd2-91e6, OPERATIONAL, view_id(REG,1e77c150-8d79,110)) suspecting node: 1e77c150-8d79
 
Dec 02 21:15:53 mysqld[1566419]: 2021-12-02 21:15:53 0 [Note] WSREP: evs::proto(2f8c7fd2-91e6, OPERATIONAL, view_id(REG,1e77c150-8d79,110)) suspected node without join message, declaring inactive
 
Dec 02 21:15:53 mysqld[1566419]: 2021-12-02 21:15:53 0 [Note] WSREP: view(view_id(NON_PRIM,1e77c150-8d79,110) memb {
 
Dec 02 21:15:53 mysqld[1566419]:         2f8c7fd2-91e6,0
 
Dec 02 21:15:53 mysqld[1566419]: } joined {
 
Dec 02 21:15:53 mysqld[1566419]: } left {
 
Dec 02 21:15:53 mysqld[1566419]: } partitioned {
 
Dec 02 21:15:53 mysqld[1566419]:         1e77c150-8d79,0
 
Dec 02 21:15:53 mysqld[1566419]:         58fa90ba-871c,0
 
Dec 02 21:15:53 mysqld[1566419]: })
 
Dec 02 21:15:53 mysqld[1566419]: 2021-12-02 21:15:53 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
 
Dec 02 21:15:53 mysqld[1566419]: 2021-12-02 21:15:53 0 [Note] WSREP: view(view_id(NON_PRIM,2f8c7fd2-91e6,111) memb {
 
Dec 02 21:15:53 mysqld[1566419]:         2f8c7fd2-91e6,0
 
Dec 02 21:15:53 mysqld[1566419]: } joined {
 
Dec 02 21:15:53 mysqld[1566419]: } left {
 
Dec 02 21:15:53 mysqld[1566419]: } partitioned {
 
Dec 02 21:15:53 mysqld[1566419]:         1e77c150-8d79,0
 
Dec 02 21:15:53 mysqld[1566419]:         58fa90ba-871c,0
 
Dec 02 21:15:53 mysqld[1566419]: })
 
Dec 02 21:15:53 mysqld[1566419]: 2021-12-02 21:15:53 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
 
Dec 02 21:15:53 mysqld[1566419]: 2021-12-02 21:15:53 0 [Note] WSREP: Received NON-PRIMARY.
 
Dec 02 21:15:53 mysqld[1566419]: 2021-12-02 21:15:53 0 [Note] WSREP: Shifting PRIMARY -> OPEN (TO: 4857091694)
 
Dec 02 21:15:53 mysqld[1566419]: 2021-12-02 21:15:53 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
 
Dec 02 21:15:53 mysqld[1566419]: 2021-12-02 21:15:53 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
 
Dec 02 21:15:53 mysqld[1566419]: 2021-12-02 21:15:53 0 [Note] WSREP: Received NON-PRIMARY.
 
Dec 02 21:15:53 mysqld[1566419]: /usr/sbin/mysqld(wsrep_thd_bf_abort+0x1d)[0x564296fc56bd]
 
Dec 02 21:15:53 mysqld[1566419]: /usr/sbin/mysqld(+0xaab1b6)[0x5642971fb1b6]
 
Dec 02 21:15:54 mysqld[1566419]: /usr/sbin/mysqld(+0xb0cd84)[0x56429725cd84]
 
Dec 02 21:15:54 mysqld[1566419]: /usr/sbin/mysqld(+0xb12e62)[0x564297262e62]
 
Dec 02 21:15:55 mysqld[1566419]: 2021-12-02 21:15:55 0 [Note] WSREP: declaring 1e77c150-8d79 at tcp://192.168.77.160:4567 stable
 
Dec 02 21:15:55 mysqld[1566419]: 2021-12-02 21:15:55 0 [Note] WSREP: declaring 58fa90ba-871c at tcp://192.168.77.162:4567 stable
 
Dec 02 21:15:55 mysqld[1566419]: 2021-12-02 21:15:55 0 [Note] WSREP: Node 1e77c150-8d79 state prim
 
Dec 02 21:15:55 mysqld[1566419]: 2021-12-02 21:15:55 0 [Note] WSREP: view(view_id(PRIM,1e77c150-8d79,112) memb {
 
Dec 02 21:15:55 mysqld[1566419]:         1e77c150-8d79,0
 
Dec 02 21:15:55 mysqld[1566419]:         2f8c7fd2-91e6,0
 
Dec 02 21:15:55 mysqld[1566419]:         58fa90ba-871c,0
 
Dec 02 21:15:55 mysqld[1566419]: } joined {
 
Dec 02 21:15:55 mysqld[1566419]: } left {
 
Dec 02 21:15:55 mysqld[1566419]: } partitioned {
 
Dec 02 21:15:55 mysqld[1566419]: })
 
Dec 02 21:15:55 mysqld[1566419]: 2021-12-02 21:15:55 0 [Note] WSREP: save pc into disk
 
Dec 02 21:15:55 mysqld[1566419]: 2021-12-02 21:15:55 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 3
 
Dec 02 21:15:55 mysqld[1566419]: 2021-12-02 21:15:55 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
 
Dec 02 21:15:55 mysqld[1566419]: 2021-12-02 21:15:55 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 0931be05-53b5-11ec-bba0-7f8149696efd
 
Dec 02 21:15:55 mysqld[1566419]: 2021-12-02 21:15:55 0 [Note] WSREP: STATE EXCHANGE: got state msg: 0931be05-53b5-11ec-bba0-7f8149696efd from 0 (zabbix-db04)
 
Dec 02 21:15:55 mysqld[1566419]: 2021-12-02 21:15:55 0 [Note] WSREP: STATE EXCHANGE: got state msg: 0931be05-53b5-11ec-bba0-7f8149696efd from 1 (zabbix-db05)
 
Dec 02 21:15:55 mysqld[1566419]: 2021-12-02 21:15:55 0 [Note] WSREP: STATE EXCHANGE: got state msg: 0931be05-53b5-11ec-bba0-7f8149696efd from 2 (zabbix-db06)
 
Dec 02 21:15:55 mysqld[1566419]: 2021-12-02 21:15:55 0 [Note] WSREP: Quorum results:
 
Dec 02 21:15:55 mysqld[1566419]:         version    = 6,
 
Dec 02 21:15:55 mysqld[1566419]:         component  = PRIMARY,
 
Dec 02 21:15:55 mysqld[1566419]:         conf_id    = 43,
 
Dec 02 21:15:55 mysqld[1566419]:         members    = 2/3 (joined/total),
 
Dec 02 21:15:55 mysqld[1566419]:         act_id     = 4857091708,
 
Dec 02 21:15:55 mysqld[1566419]:         last_appl. = 4857091469,
 
Dec 02 21:15:55 mysqld[1566419]:         protocols  = 2/10/4 (gcs/repl/appl),
 
Dec 02 21:15:55 mysqld[1566419]:         vote policy= 0,
 
Dec 02 21:15:55 mysqld[1566419]:         group UUID = 810a5d0c-231c-11e9-b756-dbb7899d7dd0
 
Dec 02 21:15:55 mysqld[1566419]: 2021-12-02 21:15:55 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
 
Dec 02 21:15:55 mysqld[1566419]: 2021-12-02 21:15:55 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 4857091709)
 
Dec 02 21:15:55 mysqld[1566419]: /usr/sbin/mysqld(+0xb9fdd1)[0x5642972efdd1]
 
Dec 02 21:15:55 mysqld[1566419]: /usr/sbin/mysqld(+0xac65b1)[0x5642972165b1]
 
Dec 02 21:15:56 mysqld[1566419]: 2021-12-02 21:15:56 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') turning message relay requesting off
 
Dec 02 21:15:56 mysqld[1566419]: /usr/sbin/mysqld(_ZN7handler10ha_rnd_posEPhS0_+0x50)[0x564297049690]
 
Dec 02 21:15:56 mysqld[1566419]: /usr/sbin/mysqld(+0x818150)[0x564296f68150]
 
Dec 02 21:15:57 mysqld[1566419]: /usr/sbin/mysqld(_ZN14Rows_log_event8find_rowEP14rpl_group_info+0x451)[0x56429715ef91]
 
Dec 02 21:15:57 mysqld[1566419]: /usr/sbin/mysqld(_ZN21Delete_rows_log_event11do_exec_rowEP14rpl_group_info+0x92)[0x56429715f252]
 
Dec 02 21:15:58 mysqld[1566419]: /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x23f)[0x56429715275f]
 
Dec 02 21:15:58 mysqld[1566419]: /usr/sbin/mysqld(_Z18wsrep_apply_eventsP3THDP14Relay_log_infoPKvm+0x1b9)[0x564296fbc979]
 
Dec 02 21:15:59 mysqld[1566419]: /usr/sbin/mysqld(_ZN21Wsrep_applier_service15apply_write_setERKN5wsrep7ws_metaERKNS0_12const_bufferERNS0_14mutable_bufferE+0xa3)[0x564296fa3fc3]
 
Dec 02 21:15:59 mysqld[1566419]: /usr/sbin/mysqld(+0xeaf580)[0x5642975ff580]
 
Dec 02 21:16:00 mysqld[1566419]: /usr/sbin/mysqld(_ZN5wsrep12server_state8on_applyERNS_21high_priority_serviceERKNS_9ws_handleERKNS_7ws_metaERKNS_12const_bufferE+0xc1)[0x564297600441]
 
Dec 02 21:16:00 mysqld[1566419]: /usr/sbin/mysqld(+0xec187c)[0x56429761187c]
 
Dec 02 21:16:05 mysqld[1566419]: 2021-12-02 21:16:05 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection to peer 1e77c150-8d79 with addr tcp://192.168.77.160:4567 timed out, no messages seen in PT3S, socket stats: rtt: 3289 rttvar: 6042 rto: 204000 lost: 0 last_data_recv: 2212 cwnd: 10 last_queued_since: 458625 last_delivered_since: 5346986204 send_queue_length: 0 send_queue_bytes: 0 segment: 0 messages: 0
 
Dec 02 21:16:05 mysqld[1566419]: 2021-12-02 21:16:05 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://192.168.77.160:4567
 
Dec 02 21:16:05 mysqld[1566419]: 2021-12-02 21:16:05 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT5.4279S), skipping check
 
Dec 02 21:16:05 mysqld[1566419]: 2021-12-02 21:16:05 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 58fa90ba-871c tcp://192.168.77.162:4567
 
Dec 02 21:16:05 mysqld[1566419]: 2021-12-02 21:16:05 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 1e77c150-8d79 tcp://192.168.77.160:4567
 
Dec 02 21:16:05 mysqld[1566419]: 2021-12-02 21:16:05 0 [Note] WSREP: view(view_id(NON_PRIM,1e77c150-8d79,112) memb {
 
Dec 02 21:16:05 mysqld[1566419]:         2f8c7fd2-91e6,0
 
Dec 02 21:16:05 mysqld[1566419]: } joined {
 
Dec 02 21:16:05 mysqld[1566419]: } left {
 
Dec 02 21:16:05 mysqld[1566419]: } partitioned {
 
Dec 02 21:16:05 mysqld[1566419]:         1e77c150-8d79,0
 
Dec 02 21:16:05 mysqld[1566419]:         58fa90ba-871c,0
 
Dec 02 21:16:05 mysqld[1566419]: })
 
Dec 02 21:16:05 mysqld[1566419]: 2021-12-02 21:16:05 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
 
Dec 02 21:16:05 mysqld[1566419]: 2021-12-02 21:16:05 0 [Note] WSREP: view(view_id(NON_PRIM,2f8c7fd2-91e6,113) memb {
 
Dec 02 21:16:05 mysqld[1566419]:         2f8c7fd2-91e6,0
 
Dec 02 21:16:05 mysqld[1566419]: } joined {
 
Dec 02 21:16:05 mysqld[1566419]: } left {
 
Dec 02 21:16:05 mysqld[1566419]: } partitioned {
 
Dec 02 21:16:05 mysqld[1566419]:         1e77c150-8d79,0
 
Dec 02 21:16:05 mysqld[1566419]:         58fa90ba-871c,0
 
Dec 02 21:16:05 mysqld[1566419]: })
 
Dec 02 21:16:05 mysqld[1566419]: 2021-12-02 21:16:05 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
 
Dec 02 21:16:05 mysqld[1566419]: 2021-12-02 21:16:05 0 [Note] WSREP: Received NON-PRIMARY.
 
Dec 02 21:16:05 mysqld[1566419]: 2021-12-02 21:16:05 0 [Note] WSREP: Shifting PRIMARY -> OPEN (TO: 4857092761)
 
Dec 02 21:16:05 mysqld[1566419]: 2021-12-02 21:16:05 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
 
Dec 02 21:16:05 mysqld[1566419]: 2021-12-02 21:16:05 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
 
Dec 02 21:16:05 mysqld[1566419]: 2021-12-02 21:16:05 0 [Note] WSREP: Received NON-PRIMARY.
 
Dec 02 21:16:05 mysqld[1566419]: /usr/lib/libgalera_smm.so(+0x639eb)[0x7f34865129eb]
 
Dec 02 21:16:06 mysqld[1566419]: /usr/lib/libgalera_smm.so(+0x7582e)[0x7f348652482e]
 
Dec 02 21:16:06 mysqld[1566419]: /usr/lib/libgalera_smm.so(+0x78408)[0x7f3486527408]
 
Dec 02 21:16:07 mysqld[1566419]: /usr/lib/libgalera_smm.so(+0xabd92)[0x7f348655ad92]
 
Dec 02 21:16:07 mysqld[1566419]: 2021-12-02 21:16:07 0 [Note] WSREP: declaring 1e77c150-8d79 at tcp://192.168.77.160:4567 stable
 
Dec 02 21:16:07 mysqld[1566419]: 2021-12-02 21:16:07 0 [Note] WSREP: declaring 58fa90ba-871c at tcp://192.168.77.162:4567 stable
 
Dec 02 21:16:07 mysqld[1566419]: 2021-12-02 21:16:07 0 [Note] WSREP: Node 1e77c150-8d79 state prim
 
Dec 02 21:16:07 mysqld[1566419]: 2021-12-02 21:16:07 0 [Note] WSREP: view(view_id(PRIM,1e77c150-8d79,114) memb {
 
Dec 02 21:16:07 mysqld[1566419]:         1e77c150-8d79,0
 
Dec 02 21:16:07 mysqld[1566419]:         2f8c7fd2-91e6,0
 
Dec 02 21:16:07 mysqld[1566419]:         58fa90ba-871c,0
 
Dec 02 21:16:07 mysqld[1566419]: } joined {
 
Dec 02 21:16:07 mysqld[1566419]: } left {
 
Dec 02 21:16:07 mysqld[1566419]: } partitioned {
 
Dec 02 21:16:07 mysqld[1566419]: })
 
Dec 02 21:16:07 mysqld[1566419]: 2021-12-02 21:16:07 0 [Note] WSREP: save pc into disk
 
Dec 02 21:16:07 mysqld[1566419]: 2021-12-02 21:16:07 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 3
 
Dec 02 21:16:07 mysqld[1566419]: 2021-12-02 21:16:07 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
 
Dec 02 21:16:07 mysqld[1566419]: 2021-12-02 21:16:07 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 10a57d2b-53b5-11ec-b6d6-efe5f64759fc
 
Dec 02 21:16:07 mysqld[1566419]: 2021-12-02 21:16:07 0 [Note] WSREP: STATE EXCHANGE: got state msg: 10a57d2b-53b5-11ec-b6d6-efe5f64759fc from 0 (zabbix-db04)
 
Dec 02 21:16:07 mysqld[1566419]: 2021-12-02 21:16:07 0 [Note] WSREP: STATE EXCHANGE: got state msg: 10a57d2b-53b5-11ec-b6d6-efe5f64759fc from 1 (zabbix-db05)
 
Dec 02 21:16:07 mysqld[1566419]: 2021-12-02 21:16:07 0 [Note] WSREP: STATE EXCHANGE: got state msg: 10a57d2b-53b5-11ec-b6d6-efe5f64759fc from 2 (zabbix-db06)
 
Dec 02 21:16:07 mysqld[1566419]: 2021-12-02 21:16:07 0 [Note] WSREP: Quorum results:
 
Dec 02 21:16:07 mysqld[1566419]:         version    = 6,
 
Dec 02 21:16:07 mysqld[1566419]:         component  = PRIMARY,
 
Dec 02 21:16:07 mysqld[1566419]:         conf_id    = 44,
 
Dec 02 21:16:07 mysqld[1566419]:         members    = 2/3 (joined/total),
 
Dec 02 21:16:07 mysqld[1566419]:         act_id     = 4857092776,
 
Dec 02 21:16:07 mysqld[1566419]:         last_appl. = 4857091469,
 
Dec 02 21:16:07 mysqld[1566419]:         protocols  = 2/10/4 (gcs/repl/appl),
 
Dec 02 21:16:07 mysqld[1566419]:         vote policy= 0,
 
Dec 02 21:16:07 mysqld[1566419]:         group UUID = 810a5d0c-231c-11e9-b756-dbb7899d7dd0
 
Dec 02 21:16:07 mysqld[1566419]: 2021-12-02 21:16:07 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
 
Dec 02 21:16:07 mysqld[1566419]: 2021-12-02 21:16:07 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 4857092777)
 
Dec 02 21:16:07 mysqld[1566419]: /usr/lib/libgalera_smm.so(+0xabf1b)[0x7f348655af1b]
 
Dec 02 21:16:08 mysqld[1566419]: /usr/lib/libgalera_smm.so(+0xac112)[0x7f348655b112]
 
Dec 02 21:16:08 mysqld[1566419]: 2021-12-02 21:16:08 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') turning message relay requesting off
 
Dec 02 21:16:08 mysqld[1566419]: /usr/lib/libgalera_smm.so(+0x78e00)[0x7f3486527e00]
 
Dec 02 21:16:09 mysqld[1566419]: /usr/lib/libgalera_smm.so(+0x4efe1)[0x7f34864fdfe1]
 
Dec 02 21:16:19 mysqld[1566419]: 2021-12-02 21:16:19 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection to peer 1e77c150-8d79 with addr tcp://192.168.77.160:4567 timed out, no messages seen in PT3S, socket stats: rtt: 5633 rttvar: 10029 rto: 208000 lost: 0 last_data_recv: 2420 cwnd: 10 last_queued_since: 854370 last_delivered_since: 5590235758 send_queue_length: 0 send_queue_bytes: 0 segment: 0 messages: 0
 
Dec 02 21:16:19 mysqld[1566419]: 2021-12-02 21:16:19 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://192.168.77.160:4567
 
Dec 02 21:16:19 mysqld[1566419]: 2021-12-02 21:16:19 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT5.63521S), skipping check
 
Dec 02 21:16:19 mysqld[1566419]: 2021-12-02 21:16:19 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 1e77c150-8d79 tcp://192.168.77.160:4567
 
Dec 02 21:16:19 mysqld[1566419]: 2021-12-02 21:16:19 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 58fa90ba-871c tcp://192.168.77.162:4567
 
Dec 02 21:16:20 mysqld[1566419]: 2021-12-02 21:16:20 0 [Note] WSREP: evs::proto(2f8c7fd2-91e6, OPERATIONAL, view_id(REG,1e77c150-8d79,114)) suspecting node: 1e77c150-8d79
 
Dec 02 21:16:20 mysqld[1566419]: 2021-12-02 21:16:20 0 [Note] WSREP: evs::proto(2f8c7fd2-91e6, OPERATIONAL, view_id(REG,1e77c150-8d79,114)) suspected node without join message, declaring inactive
 
Dec 02 21:16:20 mysqld[1566419]: 2021-12-02 21:16:20 0 [Note] WSREP: view(view_id(NON_PRIM,1e77c150-8d79,114) memb {
 
Dec 02 21:16:20 mysqld[1566419]:         2f8c7fd2-91e6,0
 
Dec 02 21:16:20 mysqld[1566419]: } joined {
 
Dec 02 21:16:20 mysqld[1566419]: } left {
 
Dec 02 21:16:20 mysqld[1566419]: } partitioned {
 
Dec 02 21:16:20 mysqld[1566419]:         1e77c150-8d79,0
 
Dec 02 21:16:20 mysqld[1566419]:         58fa90ba-871c,0
 
Dec 02 21:16:20 mysqld[1566419]: })
 
Dec 02 21:16:20 mysqld[1566419]: 2021-12-02 21:16:20 0 [Note] WSREP: view(view_id(NON_PRIM,2f8c7fd2-91e6,115) memb {
 
Dec 02 21:16:20 mysqld[1566419]:         2f8c7fd2-91e6,0
 
Dec 02 21:16:20 mysqld[1566419]: } joined {
 
Dec 02 21:16:20 mysqld[1566419]: } left {
 
Dec 02 21:16:20 mysqld[1566419]: } partitioned {
 
Dec 02 21:16:20 mysqld[1566419]:         1e77c150-8d79,0
 
Dec 02 21:16:20 mysqld[1566419]:         58fa90ba-871c,0
 
Dec 02 21:16:20 mysqld[1566419]: })
 
Dec 02 21:16:20 mysqld[1566419]: 2021-12-02 21:16:20 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
 
Dec 02 21:16:20 mysqld[1566419]: 2021-12-02 21:16:20 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
 
Dec 02 21:16:20 mysqld[1566419]: 2021-12-02 21:16:20 0 [Note] WSREP: Received NON-PRIMARY.
 
Dec 02 21:16:20 mysqld[1566419]: 2021-12-02 21:16:20 0 [Note] WSREP: Shifting PRIMARY -> OPEN (TO: 4857094033)
 
Dec 02 21:16:20 mysqld[1566419]: 2021-12-02 21:16:20 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
 
Dec 02 21:16:20 mysqld[1566419]: 2021-12-02 21:16:20 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
 
Dec 02 21:16:20 mysqld[1566419]: 2021-12-02 21:16:20 0 [Note] WSREP: Received NON-PRIMARY.
 
Dec 02 21:16:21 mysqld[1566419]: 2021-12-02 21:16:21 0 [Note] WSREP: declaring 1e77c150-8d79 at tcp://192.168.77.160:4567 stable
 
Dec 02 21:16:21 mysqld[1566419]: 2021-12-02 21:16:21 0 [Note] WSREP: declaring 58fa90ba-871c at tcp://192.168.77.162:4567 stable
 
Dec 02 21:16:21 mysqld[1566419]: 2021-12-02 21:16:21 0 [Note] WSREP: Node 1e77c150-8d79 state prim
 
Dec 02 21:16:21 mysqld[1566419]: 2021-12-02 21:16:21 0 [Note] WSREP: view(view_id(PRIM,1e77c150-8d79,116) memb {
 
Dec 02 21:16:21 mysqld[1566419]:         1e77c150-8d79,0
 
Dec 02 21:16:21 mysqld[1566419]:         2f8c7fd2-91e6,0
 
Dec 02 21:16:21 mysqld[1566419]:         58fa90ba-871c,0
 
Dec 02 21:16:21 mysqld[1566419]: } joined {
 
Dec 02 21:16:21 mysqld[1566419]: } left {
 
Dec 02 21:16:21 mysqld[1566419]: } partitioned {
 
Dec 02 21:16:21 mysqld[1566419]: })
 
Dec 02 21:16:21 mysqld[1566419]: 2021-12-02 21:16:21 0 [Note] WSREP: save pc into disk
 
Dec 02 21:16:21 mysqld[1566419]: 2021-12-02 21:16:21 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 3
 
Dec 02 21:16:21 mysqld[1566419]: 2021-12-02 21:16:21 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
 
Dec 02 21:16:21 mysqld[1566419]: 2021-12-02 21:16:21 0 [Warning] WSREP: Last Applied Action message in non-primary configuration from member 0
 
Dec 02 21:16:21 mysqld[1566419]: 2021-12-02 21:16:21 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 18fe4842-53b5-11ec-bd11-4b161e80910e
 
Dec 02 21:16:21 mysqld[1566419]: 2021-12-02 21:16:21 0 [Note] WSREP: STATE EXCHANGE: got state msg: 18fe4842-53b5-11ec-bd11-4b161e80910e from 2 (zabbix-db06)
 
Dec 02 21:16:21 mysqld[1566419]: 2021-12-02 21:16:21 0 [Note] WSREP: STATE EXCHANGE: got state msg: 18fe4842-53b5-11ec-bd11-4b161e80910e from 0 (zabbix-db04)
 
Dec 02 21:16:21 mysqld[1566419]: 2021-12-02 21:16:21 0 [Note] WSREP: STATE EXCHANGE: got state msg: 18fe4842-53b5-11ec-bd11-4b161e80910e from 1 (zabbix-db05)
 
Dec 02 21:16:21 mysqld[1566419]: 2021-12-02 21:16:21 0 [Note] WSREP: Quorum results:
 
Dec 02 21:16:21 mysqld[1566419]:         version    = 6,
 
Dec 02 21:16:21 mysqld[1566419]:         component  = PRIMARY,
 
Dec 02 21:16:21 mysqld[1566419]:         conf_id    = 45,
 
Dec 02 21:16:21 mysqld[1566419]:         members    = 2/3 (joined/total),
 
Dec 02 21:16:21 mysqld[1566419]:         act_id     = 4857094048,
 
Dec 02 21:16:21 mysqld[1566419]:         last_appl. = 4857091469,
 
Dec 02 21:16:21 mysqld[1566419]:         protocols  = 2/10/4 (gcs/repl/appl),
 
Dec 02 21:16:21 mysqld[1566419]:         vote policy= 0,
 
Dec 02 21:16:21 mysqld[1566419]:         group UUID = 810a5d0c-231c-11e9-b756-dbb7899d7dd0
 
Dec 02 21:16:21 mysqld[1566419]: 2021-12-02 21:16:21 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
 
Dec 02 21:16:21 mysqld[1566419]: 2021-12-02 21:16:21 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 4857094049)
 
Dec 02 21:16:23 mysqld[1566419]: 2021-12-02 21:16:23 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') turning message relay requesting off
 
Dec 02 21:16:28 mysqld[1566419]: /usr/sbin/mysqld(_ZN5wsrep18wsrep_provider_v2611run_applierEPNS_21high_priority_serviceE+0x12)[0x564297611e92]
 
Dec 02 21:16:28 mysqld[1566419]: /usr/sbin/mysqld(+0x86e7c3)[0x564296fbe7c3]
 
Dec 02 21:16:28 mysqld[1566419]: /usr/sbin/mysqld(_Z15start_wsrep_THDPv+0x2ec)[0x564296faef1c]
 
Dec 02 21:16:34 mysqld[1566419]: 2021-12-02 21:16:34 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection to peer 1e77c150-8d79 with addr tcp://192.168.77.160:4567 timed out, no messages seen in PT3S, socket stats: rtt: 2200 rttvar: 2183 rto: 204000 lost: 0 last_data_recv: 3048 cwnd: 10 last_queued_since: 420943 last_delivered_since: 6114827970 send_queue_length: 4 send_queue_bytes: 320 segment: 0 messages: 4
 
Dec 02 21:16:34 mysqld[1566419]: 2021-12-02 21:16:34 0 [Note] WSREP: Deferred close timer started for socket with remote endpoint: tcp://192.168.77.160:51218
 
Dec 02 21:16:34 mysqld[1566419]: 2021-12-02 21:16:34 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://192.168.77.160:4567
 
Dec 02 21:16:34 mysqld[1566419]: 2021-12-02 21:16:34 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT6.26427S), skipping check
 
Dec 02 21:16:34 mysqld[1566419]: 2021-12-02 21:16:34 0 [Note] WSREP: Deferred close timer handle_wait Operation aborted. for 0x56429a295c70
 
Dec 02 21:16:34 mysqld[1566419]: 2021-12-02 21:16:34 0 [Note] WSREP: Deferred close timer destruct
 
Dec 02 21:16:34 mysqld[1566419]: 2021-12-02 21:16:34 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 58fa90ba-871c tcp://192.168.77.162:4567
 
Dec 02 21:16:34 mysqld[1566419]: 2021-12-02 21:16:34 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 1e77c150-8d79 tcp://192.168.77.160:4567
 
Dec 02 21:16:34 mysqld[1566419]: 2021-12-02 21:16:34 0 [Note] WSREP: view(view_id(NON_PRIM,1e77c150-8d79,116) memb {
 
Dec 02 21:16:34 mysqld[1566419]:         2f8c7fd2-91e6,0
 
Dec 02 21:16:34 mysqld[1566419]: } joined {
 
Dec 02 21:16:34 mysqld[1566419]: } left {
 
Dec 02 21:16:34 mysqld[1566419]: } partitioned {
 
Dec 02 21:16:34 mysqld[1566419]:         1e77c150-8d79,0
 
Dec 02 21:16:34 mysqld[1566419]:         58fa90ba-871c,0
 
Dec 02 21:16:34 mysqld[1566419]: })
 
Dec 02 21:16:34 mysqld[1566419]: 2021-12-02 21:16:34 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
 
Dec 02 21:16:34 mysqld[1566419]: 2021-12-02 21:16:34 0 [Note] WSREP: view(view_id(NON_PRIM,2f8c7fd2-91e6,117) memb {
 
Dec 02 21:16:34 mysqld[1566419]:         2f8c7fd2-91e6,0
 
Dec 02 21:16:34 mysqld[1566419]: } joined {
 
Dec 02 21:16:34 mysqld[1566419]: } left {
 
Dec 02 21:16:34 mysqld[1566419]: } partitioned {
 
Dec 02 21:16:34 mysqld[1566419]:         1e77c150-8d79,0
 
Dec 02 21:16:34 mysqld[1566419]:         58fa90ba-871c,0
 
Dec 02 21:16:34 mysqld[1566419]: })
 
Dec 02 21:16:34 mysqld[1566419]: 2021-12-02 21:16:34 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
 
Dec 02 21:16:34 mysqld[1566419]: 2021-12-02 21:16:34 0 [Note] WSREP: Received NON-PRIMARY.
 
Dec 02 21:16:34 mysqld[1566419]: 2021-12-02 21:16:34 0 [Note] WSREP: Shifting PRIMARY -> OPEN (TO: 4857095264)
 
Dec 02 21:16:34 mysqld[1566419]: 2021-12-02 21:16:34 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
 
Dec 02 21:16:34 mysqld[1566419]: 2021-12-02 21:16:34 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
 
Dec 02 21:16:34 mysqld[1566419]: 2021-12-02 21:16:34 0 [Note] WSREP: Received NON-PRIMARY.
 
Dec 02 21:16:34 mysqld[1566419]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609)[0x7f3487792609]
 
Dec 02 21:16:35 mysqld[1566419]: 2021-12-02 21:16:35 0 [Note] WSREP: declaring 1e77c150-8d79 at tcp://192.168.77.160:4567 stable
 
Dec 02 21:16:35 mysqld[1566419]: 2021-12-02 21:16:35 0 [Note] WSREP: declaring 58fa90ba-871c at tcp://192.168.77.162:4567 stable
 
Dec 02 21:16:36 mysqld[1566419]: 2021-12-02 21:16:36 0 [Note] WSREP: Node 1e77c150-8d79 state prim
 
Dec 02 21:16:36 mysqld[1566419]: 2021-12-02 21:16:36 0 [Note] WSREP: view(view_id(PRIM,1e77c150-8d79,118) memb {
 
Dec 02 21:16:36 mysqld[1566419]:         1e77c150-8d79,0
 
Dec 02 21:16:36 mysqld[1566419]:         2f8c7fd2-91e6,0
 
Dec 02 21:16:36 mysqld[1566419]:         58fa90ba-871c,0
 
Dec 02 21:16:36 mysqld[1566419]: } joined {
 
Dec 02 21:16:36 mysqld[1566419]: } left {
 
Dec 02 21:16:36 mysqld[1566419]: } partitioned {
 
Dec 02 21:16:36 mysqld[1566419]: })
 
Dec 02 21:16:36 mysqld[1566419]: 2021-12-02 21:16:36 0 [Note] WSREP: save pc into disk
 
Dec 02 21:16:36 mysqld[1566419]: 2021-12-02 21:16:36 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 3
 
Dec 02 21:16:36 mysqld[1566419]: 2021-12-02 21:16:36 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
 
Dec 02 21:16:36 mysqld[1566419]: 2021-12-02 21:16:36 0 [Warning] WSREP: Action message in non-primary configuration from member 2
 
Dec 02 21:16:36 mysqld[1566419]: 2021-12-02 21:16:36 0 [Warning] WSREP: Action message in non-primary configuration from member 2
 
Dec 02 21:16:36 mysqld[1566419]: 2021-12-02 21:16:36 0 [Warning] WSREP: Action message in non-primary configuration from member 2
 
Dec 02 21:16:36 mysqld[1566419]: 2021-12-02 21:16:36 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 21efa719-53b5-11ec-b8ad-2b1fb78722e9
 
Dec 02 21:16:36 mysqld[1566419]: 2021-12-02 21:16:36 0 [Note] WSREP: STATE EXCHANGE: got state msg: 21efa719-53b5-11ec-b8ad-2b1fb78722e9 from 0 (zabbix-db04)
 
Dec 02 21:16:36 mysqld[1566419]: 2021-12-02 21:16:36 0 [Note] WSREP: STATE EXCHANGE: got state msg: 21efa719-53b5-11ec-b8ad-2b1fb78722e9 from 1 (zabbix-db05)
 
Dec 02 21:16:36 mysqld[1566419]: 2021-12-02 21:16:36 0 [Note] WSREP: STATE EXCHANGE: got state msg: 21efa719-53b5-11ec-b8ad-2b1fb78722e9 from 2 (zabbix-db06)
 
Dec 02 21:16:36 mysqld[1566419]: 2021-12-02 21:16:36 0 [Note] WSREP: Quorum results:
 
Dec 02 21:16:36 mysqld[1566419]:         version    = 6,
 
Dec 02 21:16:36 mysqld[1566419]:         component  = PRIMARY,
 
Dec 02 21:16:36 mysqld[1566419]:         conf_id    = 46,
 
Dec 02 21:16:36 mysqld[1566419]:         members    = 2/3 (joined/total),
 
Dec 02 21:16:36 mysqld[1566419]:         act_id     = 4857095281,
 
Dec 02 21:16:36 mysqld[1566419]:         last_appl. = 4857091469,
 
Dec 02 21:16:36 mysqld[1566419]:         protocols  = 2/10/4 (gcs/repl/appl),
 
Dec 02 21:16:36 mysqld[1566419]:         vote policy= 0,
 
Dec 02 21:16:36 mysqld[1566419]:         group UUID = 810a5d0c-231c-11e9-b756-dbb7899d7dd0
 
Dec 02 21:16:36 mysqld[1566419]: 2021-12-02 21:16:36 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
 
Dec 02 21:16:36 mysqld[1566419]: 2021-12-02 21:16:36 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 4857095282)
 
Dec 02 21:16:37 mysqld[1566419]: 2021-12-02 21:16:37 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') turning message relay requesting off
 
Dec 02 21:16:48 mysqld[1566419]: 2021-12-02 21:16:48 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection to peer 1e77c150-8d79 with addr tcp://192.168.77.160:4567 timed out, no messages seen in PT3S, socket stats: rtt: 2007 rttvar: 2859 rto: 204000 lost: 0 last_data_recv: 2264 cwnd: 10 last_queued_since: 422869 last_delivered_since: 5570929836 send_queue_length: 4 send_queue_bytes: 320 segment: 0 messages: 4
 
Dec 02 21:16:48 mysqld[1566419]: 2021-12-02 21:16:48 0 [Note] WSREP: Deferred close timer started for socket with remote endpoint: tcp://192.168.77.160:51222
 
Dec 02 21:16:48 mysqld[1566419]: 2021-12-02 21:16:48 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://192.168.77.160:4567
 
Dec 02 21:16:48 mysqld[1566419]: 2021-12-02 21:16:48 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT5.97614S), skipping check
 
Dec 02 21:16:48 mysqld[1566419]: 2021-12-02 21:16:48 0 [Note] WSREP: Deferred close timer handle_wait Operation aborted. for 0x7f13a0044ea0
 
Dec 02 21:16:48 mysqld[1566419]: 2021-12-02 21:16:48 0 [Note] WSREP: Deferred close timer destruct
 
Dec 02 21:16:48 mysqld[1566419]: 2021-12-02 21:16:48 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 58fa90ba-871c tcp://192.168.77.162:4567
 
Dec 02 21:16:48 mysqld[1566419]: 2021-12-02 21:16:48 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 1e77c150-8d79 tcp://192.168.77.160:4567
 
Dec 02 21:16:48 mysqld[1566419]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x43)[0x7f3487380293]
 
Dec 02 21:16:48 mysqld[1566419]: Trying to get some variables.
 
Dec 02 21:16:48 mysqld[1566419]: Some pointers may be invalid and cause the dump to abort.
 
Dec 02 21:16:48 mysqld[1566419]: Query (0x7f16e8a3da93): delete from problem where r_clock<>0 and r_clock<1638393301
 
Dec 02 21:16:48 mysqld[1566419]: Connection ID (thread ID): 66
 
Dec 02 21:16:48 mysqld[1566419]: Status: NOT_KILLED
 
Dec 02 21:16:48 mysqld[1566419]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on
 
Dec 02 21:16:48 mysqld[1566419]: The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
 
Dec 02 21:16:48 mysqld[1566419]: information that should help you find out what is causing the crash.
 
Dec 02 21:16:48 mysqld[1566419]: Writing a core file...
 
Dec 02 21:16:48 mysqld[1566419]: Working directory at /var/lib/mysql
 
Dec 02 21:16:48 mysqld[1566419]: Resource Limits:
 
Dec 02 21:16:48 mysqld[1566419]: Limit                     Soft Limit           Hard Limit           Units
 
Dec 02 21:16:48 mysqld[1566419]: Max cpu time              unlimited            unlimited            seconds
 
Dec 02 21:16:48 mysqld[1566419]: Max file size             unlimited            unlimited            bytes
 
Dec 02 21:16:48 mysqld[1566419]: Max data size             unlimited            unlimited            bytes
 
Dec 02 21:16:48 mysqld[1566419]: Max stack size            8388608              unlimited            bytes
 
Dec 02 21:16:48 mysqld[1566419]: Max core file size        0                    unlimited            bytes
 
Dec 02 21:16:48 mysqld[1566419]: Max resident set          unlimited            unlimited            bytes
 
Dec 02 21:16:48 mysqld[1566419]: Max processes             5931800              5931800              processes
 
Dec 02 21:16:48 mysqld[1566419]: Max open files            65536                65536                files
 
Dec 02 21:16:48 mysqld[1566419]: Max locked memory         65536                65536                bytes
 
Dec 02 21:16:48 mysqld[1566419]: Max address space         unlimited            unlimited            bytes
 
Dec 02 21:16:48 mysqld[1566419]: Max file locks            unlimited            unlimited            locks
 
Dec 02 21:16:48 mysqld[1566419]: Max pending signals       5931800              5931800              signals
 
Dec 02 21:16:48 mysqld[1566419]: Max msgqueue size         819200               819200               bytes
 
Dec 02 21:16:48 mysqld[1566419]: Max nice priority         0                    0
 
Dec 02 21:16:48 mysqld[1566419]: Max realtime priority     0                    0
 
Dec 02 21:16:48 mysqld[1566419]: Max realtime timeout      unlimited            unlimited            us
 
Dec 02 21:16:48 mysqld[1566419]: Core pattern: /var/crash/%e.%p
 
Dec 02 21:17:13 systemd[1]: mariadb.service: Main process exited, code=killed, status=6/ABRT
 
Dec 02 21:17:13 systemd[1]: mariadb.service: Failed with result 'signal'.

Comment by Mark Reibert [ 2021-12-03 ]

And here is what happened (another SIGABRT) when I simply attempted to restart MariaDB:

Dec 02 22:28:24 systemd[1]: Starting MariaDB 10.4.22 database server...
 
Dec 02 22:29:52 mysqld[2095405]: WSREP: Recovered position 810a5d0c-231c-11e9-b756-dbb7899d7dd0:4857087468
 
Dec 02 22:29:52 mysqld[2095962]: 2021-12-02 22:29:52 0 [Note] /usr/sbin/mysqld (mysqld 10.4.22-MariaDB-1:10.4.22+maria~focal-log) starting as process 2095962 ...
 
Dec 02 22:29:52 mysqld[2095962]: 2021-12-02 22:29:52 0 [Note] WSREP: Loading provider /usr/lib/libgalera_smm.so initial position: 810a5d0c-231c-11e9-b756-dbb7899d7dd0:4857087468
 
Dec 02 22:29:52 mysqld[2095962]: 2021-12-02 22:29:52 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/libgalera_smm.so'
 
Dec 02 22:29:52 mysqld[2095962]: 2021-12-02 22:29:52 0 [Note] WSREP: wsrep_load(): Galera 26.4.9(r819f29cb) by Codership Oy <info@codership.com> loaded successfully.
 
Dec 02 22:29:52 mysqld[2095962]: 2021-12-02 22:29:52 0 [Note] WSREP: CRC-32C: using 64-bit x86 acceleration.
 
Dec 02 22:29:52 mysqld[2095962]: 2021-12-02 22:29:52 0 [Note] WSREP: Found saved state: 810a5d0c-231c-11e9-b756-dbb7899d7dd0:-1, safe_to_bootstrap: 0
 
Dec 02 22:29:52 mysqld[2095962]: 2021-12-02 22:29:52 0 [Note] WSREP: GCache DEBUG: opened preamble:
 
Dec 02 22:29:52 mysqld[2095962]: Version: 2
 
Dec 02 22:29:52 mysqld[2095962]: UUID: 810a5d0c-231c-11e9-b756-dbb7899d7dd0
 
Dec 02 22:29:52 mysqld[2095962]: Seqno: -1 - -1
 
Dec 02 22:29:52 mysqld[2095962]: Offset: -1
 
Dec 02 22:29:52 mysqld[2095962]: Synced: 0
 
Dec 02 22:29:52 mysqld[2095962]: 2021-12-02 22:29:52 0 [Note] WSREP: Recovering GCache ring buffer: version: 2, UUID: 810a5d0c-231c-11e9-b756-dbb7899d7dd0, offset: -1
 
Dec 02 22:29:52 mysqld[2095962]: 2021-12-02 22:29:52 0 [Note] WSREP: GCache::RingBuffer initial scan...  0.0% (           0/140928614424 bytes) complete.
 
Dec 02 22:30:02 mysqld[2095962]: 2021-12-02 22:30:02 0 [Note] WSREP: GCache::RingBuffer initial scan... 79.1% (111525388192/140928614424 bytes) complete.
 
Dec 02 22:30:05 mysqld[2095962]: 2021-12-02 22:30:05 0 [Note] WSREP: GCache::RingBuffer initial scan...100.0% (140928614424/140928614424 bytes) complete.
 
Dec 02 22:30:05 mysqld[2095962]: 2021-12-02 22:30:05 0 [Note] WSREP: Recovering GCache ring buffer: found gapless sequence 4846387206-4857087532
 
Dec 02 22:30:05 mysqld[2095962]: 2021-12-02 22:30:05 0 [Note] WSREP: GCache::RingBuffer unused buffers scan...  0.0% (           0/140821082440 bytes) complete.
 
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: GCache::RingBuffer unused buffers scan...100.0% (140821082440/140821082440 bytes) complete.
 
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: GCache DEBUG: RingBuffer::recover(): found 7/10700334 locked buffers
 
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: GCache DEBUG: RingBuffer::recover(): free space: 107589352/140928614400
 
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 192.168.77.161; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 134400M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 2000000; gcs.fc_master_slave = yes; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.
 
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: Service thread queue flushed.
 
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: ####### Assign initial position for certification: 810a5d0c-231c-11e9-b756-dbb7899d7dd0:4857087468, protocol version: -1
 
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: Start replication
 
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: Connecting with bootstrap option: 0
 
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: Setting GCS initial position to 810a5d0c-231c-11e9-b756-dbb7899d7dd0:4857087468
 
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: protonet asio version 0
 
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: Using CRC-32C for message checksums.
 
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: backend: asio
 
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: gcomm thread scheduling priority set to other:0
 
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: restore pc from disk successfully
 
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: GMCast version 0
 
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: (2f8c7fd2-91e7, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
 
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: (2f8c7fd2-91e7, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
 
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: EVS version 1
 
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: gcomm: connecting to group 'galera_cluster', peer '192.168.77.160:,192.168.77.161:,192.168.77.162:'
 
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: (2f8c7fd2-91e7, 'tcp://0.0.0.0:4567') connection established to 1e77c150-8d79 tcp://192.168.77.160:4567
 
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: (2f8c7fd2-91e7, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
 
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: (2f8c7fd2-91e7, 'tcp://0.0.0.0:4567') connection established to 58fa90ba-871c tcp://192.168.77.162:4567
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: gcomm: connected
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: Opened channel 'galera_cluster'
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 1 [Note] WSREP: Starting rollbacker thread 1
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 2 [Note] WSREP: Starting applier thread 2
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: EVS version upgrade 0 -> 1
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: declaring 1e77c150-8d79 at tcp://192.168.77.160:4567 stable
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: declaring 58fa90ba-871c at tcp://192.168.77.162:4567 stable
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: PC protocol upgrade 0 -> 1
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: Node 1e77c150-8d79 state prim
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: view(view_id(PRIM,1e77c150-8d79,120) memb {
 
Dec 02 22:30:08 mysqld[2095962]:         1e77c150-8d79,0
 
Dec 02 22:30:08 mysqld[2095962]:         2f8c7fd2-91e7,0
 
Dec 02 22:30:08 mysqld[2095962]:         58fa90ba-871c,0
 
Dec 02 22:30:08 mysqld[2095962]: } joined {
 
Dec 02 22:30:08 mysqld[2095962]: } left {
 
Dec 02 22:30:08 mysqld[2095962]: } partitioned {
 
Dec 02 22:30:08 mysqld[2095962]: })
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: save pc into disk
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: discarding pending addr without UUID: tcp://192.168.77.161:4567
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: discarding pending addr proto entry 0x55c63ede73c0
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: clear restored view
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 3
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 677c3d90-53bf-11ec-99c7-8a27c0e85aa9
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: STATE EXCHANGE: got state msg: 677c3d90-53bf-11ec-99c7-8a27c0e85aa9 from 0 (zabbix-db04)
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: STATE EXCHANGE: got state msg: 677c3d90-53bf-11ec-99c7-8a27c0e85aa9 from 2 (zabbix-db06)
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: STATE EXCHANGE: got state msg: 677c3d90-53bf-11ec-99c7-8a27c0e85aa9 from 1 (zabbix-db05)
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: Quorum results:
 
Dec 02 22:30:08 mysqld[2095962]:         version    = 6,
 
Dec 02 22:30:08 mysqld[2095962]:         component  = PRIMARY,
 
Dec 02 22:30:08 mysqld[2095962]:         conf_id    = 48,
 
Dec 02 22:30:08 mysqld[2095962]:         members    = 2/3 (joined/total),
 
Dec 02 22:30:08 mysqld[2095962]:         act_id     = 4857698808,
 
Dec 02 22:30:08 mysqld[2095962]:         last_appl. = 4857698803,
 
Dec 02 22:30:08 mysqld[2095962]:         protocols  = 2/10/4 (gcs/repl/appl),
 
Dec 02 22:30:08 mysqld[2095962]:         vote policy= 0,
 
Dec 02 22:30:08 mysqld[2095962]:         group UUID = 810a5d0c-231c-11e9-b756-dbb7899d7dd0
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 4857698809)
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 2 [Note] WSREP: ####### processing CC 4857698809, local, ordered
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 2 [Note] WSREP: Process first view: 810a5d0c-231c-11e9-b756-dbb7899d7dd0 my uuid: 2f8c7fd2-5334-11ec-91e7-46483b39372d
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 2 [Note] WSREP: Server zabbix-db05 connected to cluster at position 810a5d0c-231c-11e9-b756-dbb7899d7dd0:4857698809 with ID 2f8c7fd2-5334-11ec-91e7-46483b39372d
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 2 [Note] WSREP: Server status change disconnected -> connected
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 2 [Note] WSREP: ####### My UUID: 2f8c7fd2-5334-11ec-91e7-46483b39372d
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 2 [Note] WSREP: Cert index reset to 00000000-0000-0000-0000-000000000000:-1 (proto: 10), state transfer needed: yes
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: Service thread queue flushed.
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 2 [Note] WSREP: ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: -1
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 2 [Note] WSREP: State transfer required:
 
Dec 02 22:30:08 mysqld[2095962]:         Group state: 810a5d0c-231c-11e9-b756-dbb7899d7dd0:4857698809
 
Dec 02 22:30:08 mysqld[2095962]:         Local state: 810a5d0c-231c-11e9-b756-dbb7899d7dd0:4857087468
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 2 [Note] WSREP: Server status change connected -> joiner
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: Joiner monitor thread started to monitor
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'joiner' --address '192.168.77.161' --datadir '/var/lib/mysql/' --parent '2095962' --mysqld-args --wsrep_start_position=810a5d0c-231c-11e9-b756-dbb7899d7dd0:4857087468'
 
Dec 02 22:30:08 mysqld[2096028]: WSREP_SST: [INFO] SSL configuration: CA='', CERT='', KEY='', MODE='DISABLED', encrypt='0' (20211202 22:30:08.213)
 
Dec 02 22:30:08 mysqld[2096028]: WSREP_SST: [INFO] Logging all stderr of SST/mariabackup to syslog (20211202 22:30:08.284)
 
Dec 02 22:30:08 -wsrep-sst-joiner[2096168]: Using socat as streamer
 
Dec 02 22:30:09 mysqld[2095962]: 2021-12-02 22:30:09 2 [Note] WSREP: ####### IST uuid:810a5d0c-231c-11e9-b756-dbb7899d7dd0 f: 4857087469, l: 4857698809, STRv: 3
 
Dec 02 22:30:09 mysqld[2095962]: 2021-12-02 22:30:09 2 [Note] WSREP: IST receiver addr using tcp://192.168.77.161:4568
 
Dec 02 22:30:09 mysqld[2095962]: 2021-12-02 22:30:09 2 [Note] WSREP: Prepared IST receiver for 4857087469-4857698809, listening at: tcp://192.168.77.161:4568
 
Dec 02 22:30:09 mysqld[2095962]: 2021-12-02 22:30:09 0 [Note] WSREP: Member 1.0 (zabbix-db05) requested state transfer from 'zabbix-db04'. Selected 0.0 (zabbix-db04)(SYNCED) as donor.
 
Dec 02 22:30:09 mysqld[2095962]: 2021-12-02 22:30:09 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 4857698990)
 
Dec 02 22:30:09 mysqld[2095962]: 2021-12-02 22:30:09 2 [Note] WSREP: Requesting state transfer: success, donor: 0
 
Dec 02 22:30:09 mysqld[2095962]: 2021-12-02 22:30:09 0 [Note] WSREP: 0.0 (zabbix-db04): State transfer to 1.0 (zabbix-db05) complete.
 
Dec 02 22:30:09 mysqld[2095962]: 2021-12-02 22:30:09 0 [Note] WSREP: Member 0.0 (zabbix-db04) synced with group.
 
Dec 02 22:30:09 -wsrep-sst-joiner[2096217]: Removing the sst_in_progress file
 
Dec 02 22:30:09 mysqld[2095962]: 2021-12-02 22:30:09 3 [Note] WSREP: SST received
 
Dec 02 22:30:09 mysqld[2095962]: 2021-12-02 22:30:09 3 [Note] WSREP: Server status change joiner -> initializing
 
Dec 02 22:30:09 mysqld[2095962]: 2021-12-02 22:30:09 3 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
Dec 02 22:30:09 mysqld[2095962]: 2021-12-02 22:30:09 0 [Note] InnoDB: Using Linux native AIO
 
Dec 02 22:30:09 mysqld[2095962]: 2021-12-02 22:30:09 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
 
Dec 02 22:30:09 mysqld[2095962]: 2021-12-02 22:30:09 0 [Note] InnoDB: Uses event mutexes
 
Dec 02 22:30:09 mysqld[2095962]: 2021-12-02 22:30:09 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
 
Dec 02 22:30:09 mysqld[2095962]: 2021-12-02 22:30:09 0 [Note] InnoDB: Number of pools: 1
 
Dec 02 22:30:09 mysqld[2095962]: 2021-12-02 22:30:09 0 [Note] InnoDB: Using SSE2 crc32 instructions
 
Dec 02 22:30:09 mysqld[2095962]: 2021-12-02 22:30:09 0 [Note] InnoDB: Initializing buffer pool, total size = 1087G, instances = 1, chunk size = 128M
 
Dec 02 22:30:10 mysqld[2095962]: 2021-12-02 22:30:10 0 [Note] WSREP: (2f8c7fd2-91e7, 'tcp://0.0.0.0:4567') turning message relay requesting off
 
Dec 02 22:30:49 mysqld[2095962]: 2021-12-02 22:30:49 0 [Note] InnoDB: Completed initialization of buffer pool
 
Dec 02 22:30:49 mysqld[2095962]: 2021-12-02 22:30:49 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
 
Dec 02 22:30:49 mysqld[2095962]: 2021-12-02 22:30:49 0 [Note] InnoDB: Setting log file ./ib_logfile101 size to 6442450944 bytes
 
Dec 02 22:30:50 mysqld[2095962]: 2021-12-02 22:30:50 0 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0
 
Dec 02 22:30:50 mysqld[2095962]: 2021-12-02 22:30:50 0 [Note] InnoDB: New log files created, LSN=192780380073929
 
Dec 02 22:30:50 mysqld[2095962]: 2021-12-02 22:30:50 0 [Note] InnoDB: 47 transaction(s) which must be rolled back or cleaned up in total 20286 row operations to undo
 
Dec 02 22:30:50 mysqld[2095962]: 2021-12-02 22:30:50 0 [Note] InnoDB: Trx id counter is 9733845087
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] InnoDB: Starting in background the rollback of recovered transactions
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] InnoDB: Creating shared tablespace for temporary tables
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] InnoDB: Waiting for purge to start
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] InnoDB: Rolled back recovered transaction 9733845042
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] InnoDB: Rolled back recovered transaction 9733845058
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] InnoDB: Rolled back recovered transaction 9733845034
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] InnoDB: 10.4.22 started; log sequence number 192780380073996; transaction id 9733845091
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] Plugin 'FEEDBACK' is disabled.
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] Server socket created on IP: '0.0.0.0'.
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] WSREP: wsrep_init_schema_and_SR (nil)
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] WSREP: Server initialized
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] WSREP: Server status change initializing -> initialized
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 3 [Note] WSREP: Server status change initialized -> joined
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 3 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 3 [Note] WSREP: Recovered position from storage: 810a5d0c-231c-11e9-b756-dbb7899d7dd0:4857087468
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 3 [Note] WSREP: Recovered view from SST:
 
Dec 02 22:30:59 mysqld[2095962]:   id: 810a5d0c-231c-11e9-b756-dbb7899d7dd0:4850155776
 
Dec 02 22:30:59 mysqld[2095962]:   status: primary
 
Dec 02 22:30:59 mysqld[2095962]:   protocol_version: 4
 
Dec 02 22:30:59 mysqld[2095962]:   capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
 
Dec 02 22:30:59 mysqld[2095962]:   final: no
 
Dec 02 22:30:59 mysqld[2095962]:   own_index: -1
 
Dec 02 22:30:59 mysqld[2095962]:   members(3):
 
Dec 02 22:30:59 mysqld[2095962]:         0: 1e77c150-4403-11ec-8d79-83292a6d14d2, zabbix-db04
 
Dec 02 22:30:59 mysqld[2095962]:         1: 2f8c7fd2-5334-11ec-91e6-46483b39372d, zabbix-db05
 
Dec 02 22:30:59 mysqld[2095962]:         2: 58fa90ba-483b-11ec-871c-6b4c25d26355, zabbix-db06
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 3 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 14 [Note] WSREP: Starting applier thread 14
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 15 [Note] WSREP: Recovered cluster id 810a5d0c-231c-11e9-b756-dbb7899d7dd0
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 3 [Note] WSREP: SST received: 810a5d0c-231c-11e9-b756-dbb7899d7dd0:4857087468
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 3 [Note] WSREP: SST succeeded for position 810a5d0c-231c-11e9-b756-dbb7899d7dd0:4857087468
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] WSREP: Joiner monitor thread ended with total time 51 sec
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 2 [Note] WSREP: Installed new state from SST: 810a5d0c-231c-11e9-b756-dbb7899d7dd0:4857087468
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 2 [Note] WSREP: Receiving IST: 611341 writesets, seqnos 4857087469-4857698809
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] WSREP: ####### IST applying starts with 4857087469
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 31 [Note] WSREP: Starting applier thread 31
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 32 [Note] WSREP: Starting applier thread 32
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 42 [Note] WSREP: Starting applier thread 42
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 46 [Note] WSREP: Starting applier thread 46
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 21 [Note] WSREP: Starting applier thread 21
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 33 [Note] WSREP: Starting applier thread 33
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 64 [Note] WSREP: Starting applier thread 64
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] WSREP: ####### IST current seqno initialized to 4857087469
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 45 [Note] WSREP: Starting applier thread 45
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 48 [Note] WSREP: Starting applier thread 48
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 72 [Note] WSREP: Starting applier thread 72
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 53 [Note] WSREP: Starting applier thread 53
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 29 [Note] WSREP: Starting applier thread 29
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 47 [Note] WSREP: Starting applier thread 47
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 61 [Note] WSREP: Starting applier thread 61
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 30 [Note] WSREP: Starting applier thread 30
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 25 [Note] WSREP: Starting applier thread 25
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 65 [Note] WSREP: Starting applier thread 65
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 49 [Note] WSREP: Starting applier thread 49
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 19 [Note] WSREP: Starting applier thread 19
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 73 [Note] WSREP: Starting applier thread 73
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 70 [Note] WSREP: Starting applier thread 70
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 16 [Note] WSREP: Starting applier thread 16
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 28 [Note] WSREP: Starting applier thread 28
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 10 [Note] WSREP: Starting applier thread 10
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 52 [Note] WSREP: Starting applier thread 52
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 12 [Note] WSREP: Starting applier thread 12
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 23 [Note] WSREP: Starting applier thread 23
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 54 [Note] WSREP: Starting applier thread 54
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 17 [Note] WSREP: Starting applier thread 17
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 55 [Note] WSREP: Starting applier thread 55
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 35 [Note] WSREP: Starting applier thread 35
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 56 [Note] WSREP: Starting applier thread 56
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 24 [Note] WSREP: Starting applier thread 24
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 57 [Note] WSREP: Starting applier thread 57
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 34 [Note] WSREP: Starting applier thread 34
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 58 [Note] WSREP: Starting applier thread 58
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 36 [Note] WSREP: Starting applier thread 36
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 18 [Note] WSREP: Starting applier thread 18
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 59 [Note] WSREP: Starting applier thread 59
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 20 [Note] WSREP: Starting applier thread 20
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 60 [Note] WSREP: Starting applier thread 60
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 37 [Note] WSREP: Starting applier thread 37
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 38 [Note] WSREP: Starting applier thread 38
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 13 [Note] WSREP: Starting applier thread 13
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 39 [Note] WSREP: Starting applier thread 39
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 62 [Note] WSREP: Starting applier thread 62
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 63 [Note] WSREP: Starting applier thread 63
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 40 [Note] WSREP: Starting applier thread 40
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 22 [Note] WSREP: Starting applier thread 22
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 11 [Note] WSREP: Starting applier thread 11
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 66 [Note] WSREP: Starting applier thread 66
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] WSREP: Receiving IST...  0.0% (     0/611341 events) complete.
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 67 [Note] WSREP: Starting applier thread 67
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 68 [Note] WSREP: Starting applier thread 68
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 26 [Note] WSREP: Starting applier thread 26
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 41 [Note] WSREP: Starting applier thread 41
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 69 [Note] WSREP: Starting applier thread 69
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 50 [Note] WSREP: Starting applier thread 50
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 27 [Note] WSREP: Starting applier thread 27
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 43 [Note] WSREP: Starting applier thread 43
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 51 [Note] WSREP: Starting applier thread 51
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 71 [Note] WSREP: Starting applier thread 71
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 44 [Note] WSREP: Starting applier thread 44
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] WSREP: Service thread queue flushed.
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] WSREP: ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:4857087468, protocol version: 5
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] Reading of all Master_info entries succeeded
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] Added new Master_info '' to hash table
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] /usr/sbin/mysqld: ready for connections.
 
Dec 02 22:30:59 mysqld[2095962]: Version: '10.4.22-MariaDB-1:10.4.22+maria~focal-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 21 [Warning] WSREP: no THD for trx: 9733845039
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 14 [Warning] WSREP: no THD for trx: 9733845033
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 42 [Warning] WSREP: no THD for trx: 9733845037
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 69 [Warning] WSREP: no THD for trx: 9733845076
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 66 [Warning] WSREP: no THD for trx: 9733845073
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 50 [Warning] WSREP: no THD for trx: 9733845077
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 44 [Warning] WSREP: no THD for trx: 9733845080
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 63 [Warning] WSREP: no THD for trx: 9733845069
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 41 [Warning] WSREP: no THD for trx: 9733845075
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 34 [Warning] WSREP: no THD for trx: 9733845065
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 67 [Warning] WSREP: no THD for trx: 9733845072
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 48 [Warning] WSREP: no THD for trx: 9733845043
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 64 [Warning] WSREP: no THD for trx: 9733845041
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 10 [Warning] WSREP: no THD for trx: 9733845056
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 28 [Warning] WSREP: no THD for trx: 9733845055
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 47 [Warning] WSREP: no THD for trx: 9733845047
 
Dec 02 22:30:59 systemd[1]: Started MariaDB 10.4.22 database server.
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 40 [Warning] WSREP: no THD for trx: 9733845070
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 39 [Warning] WSREP: no THD for trx: 9733845067
 
Dec 02 22:30:59 /etc/mysql/debian-start[2096564]: Upgrading MySQL tables if necessary.
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 62 [Warning] WSREP: no THD for trx: 9733845068
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 49 [Warning] WSREP: no THD for trx: 9733845052
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 31 [Warning] WSREP: no THD for trx: 9733845035
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 17 [Warning] WSREP: no THD for trx: 9733845060
 
Dec 02 22:30:59 /etc/mysql/debian-start[2096567]: /usr/bin/mysql_upgrade: the '--basedir' option is always ignored
 
Dec 02 22:30:59 /etc/mysql/debian-start[2096567]: Looking for 'mysql' as: /usr/bin/mysql
 
Dec 02 22:30:59 /etc/mysql/debian-start[2096567]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
 
Dec 02 22:30:59 /etc/mysql/debian-start[2096567]: This installation of MariaDB is already upgraded to 10.4.22-MariaDB, use --force if you still need to run mysql_upgrade
 
Dec 02 22:30:59 /etc/mysql/debian-start[2096575]: Checking for insecure root accounts.
 
Dec 02 22:30:59 mysqld[2096578]: ERROR 1047 (08S01) at line 1: WSREP has not yet prepared node for application use
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 29 [Warning] WSREP: no THD for trx: 9733845045
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 30 [Warning] WSREP: no THD for trx: 9733845049
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] InnoDB: Rolled back recovered transaction 9733845066
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] InnoDB: Rolled back recovered transaction 9733845050
 
Dec 02 22:31:00 mysqld[2095962]: 2021-12-02 22:31:00 0 [Note] InnoDB: Rolled back recovered transaction 9733845078
 
Dec 02 22:31:00 mysqld[2095962]: 2021-12-02 22:31:00 0 [Note] InnoDB: Rolled back recovered transaction 9733845046
 
Dec 02 22:31:00 mysqld[2095962]: 2021-12-02 22:31:00 0 [Note] InnoDB: Rolled back recovered transaction 9733845062
 
Dec 02 22:31:00 mysqld[2095962]: 2021-12-02 22:31:00 0 [Note] InnoDB: Rolled back recovered transaction 9733845070
 
Dec 02 22:31:00 mysqld[2095962]: 2021-12-02 22:31:00 0 [Note] InnoDB: Rolled back recovered transaction 9733845038
 
Dec 02 22:31:00 mysqld[2095962]: 2021-12-02 22:31:00 0 [Note] InnoDB: Rolled back recovered transaction 9733845054
 
Dec 02 22:31:00 mysqld[2095962]: 2021-12-02 22:31:00 0 [Note] InnoDB: Rolled back recovered transaction 9733845040
 
Dec 02 22:31:00 mysqld[2095962]: 2021-12-02 22:31:00 0 [Note] InnoDB: Rolled back recovered transaction 9733845072
 
Dec 02 22:31:00 mysqld[2095962]: 2021-12-02 22:31:00 0 [Note] InnoDB: Rolled back recovered transaction 9733845056
 
Dec 02 22:31:00 mysqld[2095962]: 2021-12-02 22:31:00 0 [Note] InnoDB: Rolled back recovered transaction 9733845080
 
Dec 02 22:31:00 mysqld[2095962]: 2021-12-02 22:31:00 0 [Note] InnoDB: Rolled back recovered transaction 9733845048
 
Dec 02 22:31:00 mysqld[2095962]: 2021-12-02 22:31:00 0 [Note] InnoDB: Rolled back recovered transaction 9733845064
 
Dec 02 22:31:00 mysqld[2095962]: 2021-12-02 22:31:00 0 [Note] InnoDB: Rolled back recovered transaction 9733845060
 
Dec 02 22:31:00 mysqld[2095962]: 2021-12-02 22:31:00 0 [Note] InnoDB: Rolled back recovered transaction 9733845076
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845044
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845068
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845036
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845052
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845035
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845067
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845051
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845043
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845075
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845059
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845079
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845047
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845063
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845055
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845071
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845039
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845073
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845041
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845057
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845049
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845065
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845033
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845053
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845037
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845069
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845061
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845045
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845077
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rollback of non-prepared transactions completed
 
Dec 02 22:31:01 mysqld[2095962]: mysqld: /home/buildbot/buildbot/build/mariadb-10.4.22/wsrep-lib/include/wsrep/client_state.hpp:668: int wsrep::client_state::bf_abort(wsrep::seqno): Assertion `mode_ == m_local || transaction_.is_streaming()' failed.
 
Dec 02 22:31:01 mysqld[2095962]: 211202 22:31:01 [ERROR] mysqld got signal 6 ;
 
Dec 02 22:31:01 mysqld[2095962]: This could be because you hit a bug. It is also possible that this binary
 
Dec 02 22:31:01 mysqld[2095962]: or one of the libraries it was linked against is corrupt, improperly built,
 
Dec 02 22:31:01 mysqld[2095962]: or misconfigured. This error can also be caused by malfunctioning hardware.
 
Dec 02 22:31:01 mysqld[2095962]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
Dec 02 22:31:01 mysqld[2095962]: We will try our best to scrape up some info that will hopefully help
 
Dec 02 22:31:01 mysqld[2095962]: diagnose the problem, but since we have already crashed,
 
Dec 02 22:31:01 mysqld[2095962]: something is definitely wrong and this may fail.
 
Dec 02 22:31:01 mysqld[2095962]: Server version: 10.4.22-MariaDB-1:10.4.22+maria~focal-log
 
Dec 02 22:31:01 mysqld[2095962]: key_buffer_size=134217728
 
Dec 02 22:31:01 mysqld[2095962]: read_buffer_size=131072
 
Dec 02 22:31:01 mysqld[2095962]: max_used_connections=1
 
Dec 02 22:31:01 mysqld[2095962]: max_threads=403
 
Dec 02 22:31:01 mysqld[2095962]: thread_count=71
 
Dec 02 22:31:01 mysqld[2095962]: It is possible that mysqld could use up to
 
Dec 02 22:31:01 mysqld[2095962]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6795376 K  bytes of memory
 
Dec 02 22:31:01 mysqld[2095962]: Hope that's ok; if not, decrease some variables in the equation.
 
Dec 02 22:31:01 mysqld[2095962]: Thread pointer: 0x7e01d8000c48
 
Dec 02 22:31:01 mysqld[2095962]: Attempting backtrace. You can use the following information to find out
 
Dec 02 22:31:01 mysqld[2095962]: where mysqld died. If you see no messages after this, something went
 
Dec 02 22:31:01 mysqld[2095962]: terribly wrong...
 
Dec 02 22:31:01 mysqld[2095962]: stack_bottom = 0x7f12b01dce08 thread_stack 0x49000
 
Dec 02 22:31:04 mysqld[2095962]: /usr/sbin/mysqld(my_print_stacktrace+0x32)[0x55c6369f7d12]
 
Dec 02 22:31:04 mysqld[2095962]: /usr/sbin/mysqld(handle_fatal_signal+0x55d)[0x55c6364a9a5d]
 
Dec 02 22:31:05 mysqld[2095962]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0)[0x7f338c51f3c0]
 
Dec 02 22:31:09 mysqld[2095962]: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x7f338c02518b]
 
Dec 02 22:31:09 mysqld[2095962]: /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b)[0x7f338c004859]
 
Dec 02 22:31:09 mysqld[2095962]: /lib/x86_64-linux-gnu/libc.so.6(+0x25729)[0x7f338c004729]
 
Dec 02 22:31:09 mysqld[2095962]: /lib/x86_64-linux-gnu/libc.so.6(+0x36f36)[0x7f338c015f36]
 
Dec 02 22:31:10 mysqld[2095962]: /usr/sbin/mysqld(_Z14wsrep_bf_abortP3THDS0_+0x6e9)[0x55c6364247f9]
 
Dec 02 22:31:11 mysqld[2095962]: /usr/sbin/mysqld(wsrep_thd_bf_abort+0x1d)[0x55c63642c6bd]
 
Dec 02 22:31:11 mysqld[2095962]: /usr/sbin/mysqld(+0xaab1b6)[0x55c6366621b6]
 
Dec 02 22:31:11 mysqld[2095962]: /usr/sbin/mysqld(+0xb0cd84)[0x55c6366c3d84]
 
Dec 02 22:31:11 mysqld[2095962]: /usr/sbin/mysqld(+0xb12e62)[0x55c6366c9e62]
 
Dec 02 22:31:11 mysqld[2095962]: /usr/sbin/mysqld(+0xb9fdd1)[0x55c636756dd1]
 
Dec 02 22:31:11 mysqld[2095962]: /usr/sbin/mysqld(+0xac65b1)[0x55c63667d5b1]
 
Dec 02 22:31:11 mysqld[2095962]: /usr/sbin/mysqld(_ZN7handler10ha_rnd_posEPhS0_+0x50)[0x55c6364b0690]
 
Dec 02 22:31:11 mysqld[2095962]: /usr/sbin/mysqld(+0x818150)[0x55c6363cf150]
 
Dec 02 22:31:11 mysqld[2095962]: /usr/sbin/mysqld(_ZN14Rows_log_event8find_rowEP14rpl_group_info+0x451)[0x55c6365c5f91]
 
Dec 02 22:31:11 mysqld[2095962]: /usr/sbin/mysqld(_ZN21Delete_rows_log_event11do_exec_rowEP14rpl_group_info+0x92)[0x55c6365c6252]
 
Dec 02 22:31:11 mysqld[2095962]: /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x23f)[0x55c6365b975f]
 
Dec 02 22:31:11 mysqld[2095962]: /usr/sbin/mysqld(_Z18wsrep_apply_eventsP3THDP14Relay_log_infoPKvm+0x1b9)[0x55c636423979]
 
Dec 02 22:31:11 mysqld[2095962]: /usr/sbin/mysqld(_ZN21Wsrep_applier_service15apply_write_setERKN5wsrep7ws_metaERKNS0_12const_bufferERNS0_14mutable_bufferE+0xa3)[0x55c63640afc3]
 
Dec 02 22:31:11 mysqld[2095962]: /usr/sbin/mysqld(+0xeaf580)[0x55c636a66580]
 
Dec 02 22:31:11 mysqld[2095962]: /usr/sbin/mysqld(_ZN5wsrep12server_state8on_applyERNS_21high_priority_serviceERKNS_9ws_handleERKNS_7ws_metaERKNS_12const_bufferE+0xc1)[0x55c636a67441]
 
Dec 02 22:31:11 mysqld[2095962]: /usr/sbin/mysqld(+0xec187c)[0x55c636a7887c]
 
Dec 02 22:31:12 mysqld[2095962]: /usr/lib/libgalera_smm.so(+0x639eb)[0x7f338b2939eb]
 
Dec 02 22:31:13 mysqld[2095962]: /usr/lib/libgalera_smm.so(+0x7582e)[0x7f338b2a582e]
 
Dec 02 22:31:13 mysqld[2095962]: src/trx_handle.cpp:391(galera::TrxHandleSlave::apply(void*, wsrep_cb_status (*)(void*, wsrep_ws_handle const*, unsigned int, wsrep_buf const*, wsrep_trx_meta const*, bool*), wsrep_trx_meta const&, bool&))[0x7f338b2bcdc4]
 
Dec 02 22:31:13 mysqld[2095962]: src/replicator_smm.cpp:504(galera::ReplicatorSMM::apply_trx(void*, galera::TrxHandleSlave&))[0x7f338b2bd462]
 
Dec 02 22:31:14 mysqld[2095962]: src/replicator_str.cpp:1131(galera::ReplicatorSMM::process_IST_writeset(void*, boost::shared_ptr<galera::TrxHandleSlave> const&))[0x7f338b2a8ea3]
 
Dec 02 22:31:14 mysqld[2095962]: src/replicator_smm.cpp:396(galera::ReplicatorSMM::async_recv(void*))[0x7f338b27efe1]
 
Dec 02 22:31:15 mysqld[2095962]: /usr/sbin/mysqld(_ZN5wsrep18wsrep_provider_v2611run_applierEPNS_21high_priority_serviceE+0x12)[0x55c636a78e92]
 
Dec 02 22:31:15 mysqld[2095962]: /usr/sbin/mysqld(+0x86e7c3)[0x55c6364257c3]
 
Dec 02 22:31:16 mysqld[2095962]: /usr/sbin/mysqld(_Z15start_wsrep_THDPv+0x2ec)[0x55c636415f1c]
 
Dec 02 22:31:17 mysqld[2095962]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609)[0x7f338c513609]
 
Dec 02 22:31:20 mysqld[2095962]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x43)[0x7f338c101293]
 
Dec 02 22:31:20 mysqld[2095962]: Trying to get some variables.
 
Dec 02 22:31:20 mysqld[2095962]: Some pointers may be invalid and cause the dump to abort.
 
Dec 02 22:31:20 mysqld[2095962]: Query (0x7f15f1893f5b): delete from problem where r_clock<>0 and r_clock<1638393301
 
Dec 02 22:31:20 mysqld[2095962]: Connection ID (thread ID): 14
 
Dec 02 22:31:20 mysqld[2095962]: Status: NOT_KILLED
 
Dec 02 22:31:20 mysqld[2095962]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on
 
Dec 02 22:31:20 mysqld[2095962]: The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
 
Dec 02 22:31:20 mysqld[2095962]: information that should help you find out what is causing the crash.
 
Dec 02 22:31:20 mysqld[2095962]: Writing a core file...
 
Dec 02 22:31:20 mysqld[2095962]: Working directory at /var/lib/mysql
 
Dec 02 22:31:20 mysqld[2095962]: Resource Limits:
 
Dec 02 22:31:20 mysqld[2095962]: Limit                     Soft Limit           Hard Limit           Units
 
Dec 02 22:31:20 mysqld[2095962]: Max cpu time              unlimited            unlimited            seconds
 
Dec 02 22:31:20 mysqld[2095962]: Max file size             unlimited            unlimited            bytes
 
Dec 02 22:31:20 mysqld[2095962]: Max data size             unlimited            unlimited            bytes
 
Dec 02 22:31:20 mysqld[2095962]: Max stack size            8388608              unlimited            bytes
 
Dec 02 22:31:20 mysqld[2095962]: Max core file size        0                    unlimited            bytes
 
Dec 02 22:31:20 mysqld[2095962]: Max resident set          unlimited            unlimited            bytes
 
Dec 02 22:31:20 mysqld[2095962]: Max processes             5931800              5931800              processes
 
Dec 02 22:31:20 mysqld[2095962]: Max open files            65536                65536                files
 
Dec 02 22:31:20 mysqld[2095962]: Max locked memory         65536                65536                bytes
 
Dec 02 22:31:20 mysqld[2095962]: Max address space         unlimited            unlimited            bytes
 
Dec 02 22:31:20 mysqld[2095962]: Max file locks            unlimited            unlimited            locks
 
Dec 02 22:31:20 mysqld[2095962]: Max pending signals       5931800              5931800              signals
 
Dec 02 22:31:20 mysqld[2095962]: Max msgqueue size         819200               819200               bytes
 
Dec 02 22:31:20 mysqld[2095962]: Max nice priority         0                    0
 
Dec 02 22:31:20 mysqld[2095962]: Max realtime priority     0                    0
 
Dec 02 22:31:20 mysqld[2095962]: Max realtime timeout      unlimited            unlimited            us
 
Dec 02 22:31:20 mysqld[2095962]: Core pattern: /var/crash/%e.%p
 
Dec 02 22:31:28 systemd[1]: mariadb.service: Main process exited, code=killed, status=6/ABRT
 
Dec 02 22:31:28 systemd[1]: mariadb.service: Failed with result 'signal'.

Comment by Mark Reibert [ 2021-12-06 ]

Yet another crash.

Comment by Elena Stepanova [ 2022-01-06 ]

mreibert

That aside, I do not know how to correlate the crash with a particular query (if that is even possible).

You can see the query right in the crash reports in your log snippets (not always, but in most of them).
In the original one it is

INSERT INTO `jobs` (`queue`, `status`, `job`, `params`, `priority`, `run_at`, `created_at`) VALUES ('normal', 'pending', 'Task::NotifyJob', '---\n:model: Ticket\n:id: 138898\n', 10, '2021-10-13 10:52:51', '2021-10-13 10:52:51')

So the table in question is `jobs`.

In the later ones, it is

delete from problem where r_clock<>0 and r_clock<1636395301

– remarkably, the exact same query 3 times, so it's hardly random – so the table is `problem`.

Comment by Mark Reibert [ 2022-01-07 ]

elenst — Thank you for looking into this. Note this issue was originally created by mattm, and hence the reference to the jobs table is his.

The problem table, however, is mine. Here is the structure of that table:

MariaDB [zabbix]> show create table problem;
 
| Table   | Create Table
 
| problem | CREATE TABLE `problem` (
 
  `eventid` bigint(20) unsigned NOT NULL,
 
  `source` int(11) NOT NULL DEFAULT 0,
 
  `object` int(11) NOT NULL DEFAULT 0,
 
  `objectid` bigint(20) unsigned NOT NULL DEFAULT 0,
 
  `clock` int(11) NOT NULL DEFAULT 0,
 
  `ns` int(11) NOT NULL DEFAULT 0,
 
  `r_eventid` bigint(20) unsigned DEFAULT NULL,
 
  `r_clock` int(11) NOT NULL DEFAULT 0,
 
  `r_ns` int(11) NOT NULL DEFAULT 0,
 
  `correlationid` bigint(20) unsigned DEFAULT NULL,
 
  `userid` bigint(20) unsigned DEFAULT NULL,
 
  `name` varchar(2048) NOT NULL DEFAULT '',
 
  `acknowledged` int(11) NOT NULL DEFAULT 0,
 
  `severity` int(11) NOT NULL DEFAULT 0,
 
  PRIMARY KEY (`eventid`),
 
  KEY `problem_1` (`source`,`object`,`objectid`),
 
  KEY `problem_2` (`r_clock`),
 
  KEY `problem_3` (`r_eventid`),
 
  CONSTRAINT `c_problem_1` FOREIGN KEY (`eventid`) REFERENCES `events` (`eventid`) ON DELETE CASCADE,
 
  CONSTRAINT `c_problem_2` FOREIGN KEY (`r_eventid`) REFERENCES `events` (`eventid`) ON DELETE CASCADE
 
) ENGINE=InnoDB DEFAULT CHARSET=latin1 |

What this tells us I am unsure. However, the table does have FK constraints and that has been shown to be problematic if cert.optimistic_pa = yes (which it was for me at the time I experienced these crashes). See MDEV-26298 for a discussion of that issue and MDEV-26803 for (what is hoped to be) a fix.

I should note I have not experienced this problem since setting cert.optimistic_pa = no.

Comment by Matt [ 2022-08-19 ]

This has come back to bite us again it seems and we're now running 10.7.

Aug 19 17:34:05 db02 mariadbd[959]: mariadbd: /home/buildbot/buildbot/build/mariadb-10.7.1/wsrep-lib/include/wsrep/client_state.hpp:668: int wsrep::client_state::bf_abort(wsrep::seqno): Assertion `mode_ == m_local || transaction_.is_streaming()' failed.
 
Aug 19 17:34:05 db02 mariadbd[959]: 220819 17:34:05 [ERROR] mysqld got signal 6 ;
 
Aug 19 17:34:05 db02 mariadbd[959]: This could be because you hit a bug. It is also possible that this binary
 
Aug 19 17:34:05 db02 mariadbd[959]: or one of the libraries it was linked against is corrupt, improperly built,
 
Aug 19 17:34:05 db02 mariadbd[959]: or misconfigured. This error can also be caused by malfunctioning hardware.
 
Aug 19 17:34:05 db02 mariadbd[959]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
Aug 19 17:34:05 db02 mariadbd[959]: We will try our best to scrape up some info that will hopefully help
 
Aug 19 17:34:05 db02 mariadbd[959]: diagnose the problem, but since we have already crashed,
 
Aug 19 17:34:05 db02 mariadbd[959]: something is definitely wrong and this may fail.
 
Aug 19 17:34:05 db02 mariadbd[959]: Server version: 10.7.1-MariaDB-1:10.7.1+maria~focal
 
Aug 19 17:34:05 db02 mariadbd[959]: key_buffer_size=134217728
 
Aug 19 17:34:05 db02 mariadbd[959]: read_buffer_size=131072
 
Aug 19 17:34:05 db02 mariadbd[959]: max_used_connections=18
 
Aug 19 17:34:05 db02 mariadbd[959]: max_threads=153
 
Aug 19 17:34:05 db02 mariadbd[959]: thread_count=27
 
Aug 19 17:34:05 db02 mariadbd[959]: It is possible that mysqld could use up to
 
Aug 19 17:34:05 db02 mariadbd[959]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467994 K  bytes of memory
 
Aug 19 17:34:05 db02 mariadbd[959]: Hope that's ok; if not, decrease some variables in the equation.
 
Aug 19 17:34:05 db02 mariadbd[959]: Thread pointer: 0x7f3d40000c58
 
Aug 19 17:34:05 db02 mariadbd[959]: Attempting backtrace. You can use the following information to find out
 
Aug 19 17:34:05 db02 mariadbd[959]: where mysqld died. If you see no messages after this, something went
 
Aug 19 17:34:05 db02 mariadbd[959]: terribly wrong...
 
Aug 19 17:34:05 db02 mariadbd[959]: stack_bottom = 0x7f3e8c100dc8 thread_stack 0x49000
 
Aug 19 17:34:05 db02 mariadbd[959]: Printing to addr2line failed
 
Aug 19 17:34:05 db02 mariadbd[959]: /usr/sbin/mariadbd(my_print_stacktrace+0x32)[0x5632afd237d2]
 
Aug 19 17:34:05 db02 mariadbd[959]: /usr/sbin/mariadbd(handle_fatal_signal+0x485)[0x5632af7f5495]
 
Aug 19 17:34:05 db02 mariadbd[959]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420)[0x7f3e9ff6d420]
 
Aug 19 17:34:05 db02 mariadbd[959]: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x7f3e9fa7100b]
 
Aug 19 17:34:05 db02 mariadbd[959]: /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b)[0x7f3e9fa50859]
 
Aug 19 17:34:05 db02 mariadbd[959]: /lib/x86_64-linux-gnu/libc.so.6(+0x22729)[0x7f3e9fa50729]
 
Aug 19 17:34:05 db02 mariadbd[959]: /lib/x86_64-linux-gnu/libc.so.6(+0x33fd6)[0x7f3e9fa61fd6]
 
Aug 19 17:34:05 db02 mariadbd[959]: /usr/sbin/mariadbd(_Z14wsrep_bf_abortP3THDS0_+0x5e7)[0x5632afa968f7]
 
Aug 19 17:34:05 db02 mariadbd[959]: /usr/sbin/mariadbd(wsrep_thd_bf_abort+0x1d)[0x5632afa9e09d]
 
Aug 19 17:34:05 db02 mariadbd[959]: /usr/sbin/mariadbd(+0xcd17e2)[0x5632afabe7e2]
 
Aug 19 17:34:05 db02 mariadbd[959]: /usr/sbin/mariadbd(+0x646d85)[0x5632af433d85]
 
Aug 19 17:34:05 db02 mariadbd[959]: /usr/sbin/mariadbd(+0x6476f1)[0x5632af4346f1]
 
Aug 19 17:34:05 db02 mariadbd[959]: /usr/sbin/mariadbd(+0xd780b4)[0x5632afb650b4]
 
Aug 19 17:34:05 db02 mariadbd[959]: /usr/sbin/mariadbd(+0xdabc53)[0x5632afb98c53]
 
Aug 19 17:34:05 db02 mariadbd[959]: /usr/sbin/mariadbd(+0xdafb05)[0x5632afb9cb05]
 
Aug 19 17:34:05 db02 mariadbd[959]: /usr/sbin/mariadbd(+0xdb163e)[0x5632afb9e63e]
 
Aug 19 17:34:05 db02 mariadbd[959]: /usr/sbin/mariadbd(+0xd8e382)[0x5632afb7b382]
 
Aug 19 17:34:05 db02 mariadbd[959]: /usr/sbin/mariadbd(+0xcd3ac7)[0x5632afac0ac7]
 
Aug 19 17:34:05 db02 mariadbd[959]: /usr/sbin/mariadbd(_ZN7handler13ha_delete_rowEPKh+0x32f)[0x5632af8050ef]
 
Aug 19 17:34:05 db02 mariadbd[959]: /usr/sbin/mariadbd(_ZN21Delete_rows_log_event11do_exec_rowEP14rpl_group_info+0x209)[0x5632af937099]
 
Aug 19 17:34:05 db02 mariadbd[959]: /usr/sbin/mariadbd(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x31f)[0x5632af92a23f]
 
Aug 19 17:34:05 db02 mariadbd[959]: /usr/sbin/mariadbd(_Z18wsrep_apply_eventsP3THDP14Relay_log_infoPKvm+0x1e9)[0x5632afa95759]
 
Aug 19 17:34:05 db02 mariadbd[959]: /usr/sbin/mariadbd(_ZN21Wsrep_applier_service15apply_write_setERKN5wsrep7ws_metaERKNS0_12const_bufferERNS0_14mutable_bufferE+0xab)[0x5632afa7c84b]
 
Aug 19 17:34:05 db02 mariadbd[959]: /usr/sbin/mariadbd(+0xfc69f0)[0x5632afdb39f0]
 
Aug 19 17:34:05 db02 mariadbd[959]: /usr/sbin/mariadbd(_ZN5wsrep12server_state8on_applyERNS_21high_priority_serviceERKNS_9ws_handleERKNS_7ws_metaERKNS_12const_bufferE+0xc1)[0x5632afdb48b1]
 
Aug 19 17:34:05 db02 mariadbd[959]: /usr/sbin/mariadbd(+0xfd8cec)[0x5632afdc5cec]
 
Aug 19 17:34:05 db02 mariadbd[959]: /usr/lib/galera/libgalera_smm.so(+0x639eb)[0x7f3e9f4799eb]
 
Aug 19 17:34:05 db02 mariadbd[959]: /usr/lib/galera/libgalera_smm.so(+0x7582e)[0x7f3e9f48b82e]
 
Aug 19 17:34:05 db02 mariadbd[959]: /usr/lib/galera/libgalera_smm.so(+0x78408)[0x7f3e9f48e408]
 
Aug 19 17:34:05 db02 mariadbd[959]: /usr/lib/galera/libgalera_smm.so(+0xabd92)[0x7f3e9f4c1d92]
 
Aug 19 17:34:05 db02 mariadbd[959]: /usr/lib/galera/libgalera_smm.so(+0xabf1b)[0x7f3e9f4c1f1b]
 
Aug 19 17:34:05 db02 mariadbd[959]: /usr/lib/galera/libgalera_smm.so(+0xac112)[0x7f3e9f4c2112]
 
Aug 19 17:34:05 db02 mariadbd[959]: /usr/lib/galera/libgalera_smm.so(+0x78e00)[0x7f3e9f48ee00]
 
Aug 19 17:34:05 db02 mariadbd[959]: /usr/lib/galera/libgalera_smm.so(+0x4efe1)[0x7f3e9f464fe1]
 
Aug 19 17:34:05 db02 mariadbd[959]: /usr/sbin/mariadbd(_ZN5wsrep18wsrep_provider_v2611run_applierEPNS_21high_priority_serviceE+0x12)[0x5632afdc6302]
 
Aug 19 17:34:05 db02 mariadbd[959]: /usr/sbin/mariadbd(+0xcaa607)[0x5632afa97607]
 
Aug 19 17:34:05 db02 mariadbd[959]: /usr/sbin/mariadbd(_Z15start_wsrep_THDPv+0x263)[0x5632afa875e3]
 
Aug 19 17:34:05 db02 mariadbd[959]: /usr/sbin/mariadbd(+0xc29eed)[0x5632afa16eed]
 
Aug 19 17:34:05 db02 mariadbd[959]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609)[0x7f3e9ff61609]
 
Aug 19 17:34:06 db02 mariadbd[959]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x43)[0x7f3e9fb4d133]
 
Aug 19 17:34:06 db02 mariadbd[959]: Trying to get some variables.
 
Aug 19 17:34:06 db02 mariadbd[959]: Some pointers may be invalid and cause the dump to abort.
 
Aug 19 17:34:06 db02 mariadbd[959]: Query (0x7f3e93e0ea3b): delete from `email_log` where `created_at` < ?
 
Aug 19 17:34:06 db02 mariadbd[959]: Connection ID (thread ID): 8
 
Aug 19 17:34:06 db02 mariadbd[959]: Status: NOT_KILLED
 
Aug 19 17:34:06 db02 mariadbd[959]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off
 
Aug 19 17:34:06 db02 mariadbd[959]: The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
 
Aug 19 17:34:06 db02 mariadbd[959]: information that should help you find out what is causing the crash.
 
Aug 19 17:34:06 db02 mariadbd[959]: Writing a core file...
 
Aug 19 17:34:06 db02 mariadbd[959]: Working directory at /var/lib/mysql
 
Aug 19 17:34:06 db02 mariadbd[959]: Resource Limits:
 
Aug 19 17:34:06 db02 mariadbd[959]: Limit                     Soft Limit           Hard Limit           Units
 
Aug 19 17:34:06 db02 mariadbd[959]: Max cpu time              unlimited            unlimited            seconds
 
Aug 19 17:34:06 db02 mariadbd[959]: Max file size             unlimited            unlimited            bytes
 
Aug 19 17:34:06 db02 mariadbd[959]: Max data size             unlimited            unlimited            bytes
 
Aug 19 17:34:06 db02 mariadbd[959]: Max stack size            8388608              unlimited            bytes
 
Aug 19 17:34:06 db02 mariadbd[959]: Max core file size        0                    unlimited            bytes
 
Aug 19 17:34:06 db02 mariadbd[959]: Max resident set          unlimited            unlimited            bytes
 
Aug 19 17:34:06 db02 mariadbd[959]: Max processes             3293                 3293                 processes
 
Aug 19 17:34:06 db02 mariadbd[959]: Max open files            32768                32768                files
 
Aug 19 17:34:06 db02 mariadbd[959]: Max locked memory         65536                65536                bytes
 
Aug 19 17:34:06 db02 mariadbd[959]: Max address space         unlimited            unlimited            bytes
 
Aug 19 17:34:06 db02 mariadbd[959]: Max file locks            unlimited            unlimited            locks
 
Aug 19 17:34:06 db02 mariadbd[959]: Max pending signals       3293                 3293                 signals
 
Aug 19 17:34:06 db02 mariadbd[959]: Max msgqueue size         819200               819200               bytes
 
Aug 19 17:34:06 db02 mariadbd[959]: Max nice priority         0                    0
 
Aug 19 17:34:06 db02 mariadbd[959]: Max realtime priority     0                    0
 
Aug 19 17:34:06 db02 mariadbd[959]: Max realtime timeout      unlimited            unlimited            us
 
Aug 19 17:34:06 db02 mariadbd[959]: Core pattern: |/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E

I wonder if `cert.optimistic_pa = no` would be viable for us although not sure of the implications of changing from the default (YES) to no.

Comment by Mark Reibert [ 2022-08-19 ]

Hello mattm — What I can tell you is that since setting cert.optimistic_pa = no I have not experienced any crashes, not have I noticed any ill side effects. So from my perspective it was a "safe" change.

That being written, the underlying problem is discussed—and in theory fixed—in MDEV-26803. According to that you need 10.7.2 to get the fix, and your logs above indicate you are running 10.7.1. So you may wish to upgrade to 10.7.2 if you do not want to set cert.optimistic_pa = no.

Just my $0.02 worth, as always with a street value not to exceed $0.01.

Comment by Matt [ 2022-08-19 ]

Thanks for the info.

10.7.2 isn't on the repo yet so will wait for that to come out. Last version was 10.7.1 in November it seems.

Wonder if 10.8 is worth checking out, although MDEV-26803 doesn't seem covered by it so maybe it isn't affected.

Comment by Stephan Vos [ 2022-09-27 ]

This very much seems related to this:
https://rewardgateway.engineering/2021/12/11/a-galera-replication-race-condition/
https://jira.percona.com/browse/PXC-2959

Comment by Luke Cousins [ 2022-10-04 ]

This might be related to MDEV-29493, which we've been experiencing on 10.8.4 and 10.8.5.

Generated at Thu Feb 08 09:48:10 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.