[MDEV-26298] Galera cluster slave abort with Assertion `mode_ == m_local || transaction_.is_streaming()' failed despite primary added for all tables Created: 2021-08-03  Updated: 2023-05-02  Resolved: 2022-02-24

Status: Closed
Project: MariaDB Server
Component/s: Galera
Affects Version/s: 10.4.20, 10.4.21
Fix Version/s: 10.4.23

Type: Bug Priority: Major
Reporter: jordane fillatre Assignee: Seppo Jaakola
Resolution: Fixed Votes: 3
Labels: replication
Environment:

Centos8

Attachments: File mariadb1-10.4.20-core.232870.gz     File mariadb1_mysql_error.log     Zip Archive rd-devops-sql2.zip     File reibert_mariadb.service-2.log     File reibert_mariadb.service.log     Text File reibert_mysqld-abort-stacktrace-2.txt     Text File reibert_mysqld-abort-stacktrace.txt    
Issue Links:
Relates
relates to MDEV-25551 Assertion `mode_ == m_local || transa... Closed

 Description   

Hi, Despite have add Primary Key to all tables, we still encounter the `Assertion `mode_ == m_local || transaction_.is_streaming()' failed.` Galera failure.
We operate 3 Mariadb Server 10.4.20 nodes (galera-4-26.4.6-1.el8.x86_64). Our aplications reach cluster using maxscale 2.2.9 configured with a read write splite router.

It seems to be related to a specfic request, ans some times both slaves fails and produce a core:

mysqld: 
/home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.4.20/wsrep-lib/include/wsrep/client_state.hpp:668: int wsrep::client_state::bf_abort(wsrep::seqno): Assertion `mode_ == m_local || transaction_.is_streaming()' failed.
 
210802 11:55:00 [ERROR] mysqld got signal 6 ;

Please found in attachment:

  • the Master general query log rd-devops-sql2.log
  • A Slave core
  • A Slave mysql error log

 Comments   
Comment by Seppo Jaakola [ 2021-09-15 ]

jfillatre does you database schema contain a table with foreign key constraint defined over a character (e.g. varchar) column, and the table is using some multi-byte character set (e.g. UTF8) ?

Comment by Brendon Abbott [ 2021-09-22 ]

Hello, sorry for jumping on this ticket. I have recently opened an Enterprise ticket, in which we got this exact same error having recently moved to the 10.4 series.

This was whilst running a stored procedure, that is being used to clear down a few tables. Up until now, I haven't been able to reproduce - however I have just been able to. I am now going to see if I can reproduce again, as I changed 2 variables since my last attempt

  • using MySQL Workbench from a remote machine
  • Using a different user, with less privileges - due to it being a remote machine.

Let me know if you want me to check anything in particular out.

Comment by jordane fillatre [ 2021-09-24 ]

Something like this :

MariaDB [INFORMATION_SCHEMA]> select TABLE_SCHEMA,TABLE_NAME,TABLE_COLLATION from TABLES where TABLE_NAME = 'CLIENT' \G
 
*************************** 1. row ***************************
 
   TABLE_SCHEMA: keycloak
 
     TABLE_NAME: CLIENT
 
TABLE_COLLATION: utf8_general_ci
 
1 row in set (0.001 sec)
 
 
 
MariaDB [INFORMATION_SCHEMA]> SELECT CONSTRAINT_SCHEMA, TABLE_NAME, CONSTRAINT_NAME, REFERENCED_TABLE_NAME, REFERENCED_COLUMN_NAME FROM information_schema.key_column_usage where CONSTRAINT_NAME = 'FK_P56CTINXXB9GSK57FO49F9TAC' \G
 
*************************** 1. row ***************************
 
     CONSTRAINT_SCHEMA: keycloak
 
            TABLE_NAME: CLIENT
 
       CONSTRAINT_NAME: FK_P56CTINXXB9GSK57FO49F9TAC
 
 REFERENCED_TABLE_NAME: REALM
 
REFERENCED_COLUMN_NAME: ID
 
1 row in set (0.001 sec)
 
 
 
MariaDB [INFORMATION_SCHEMA]> SHOW COLUMNS FROM CLIENT FROM keycloak where Field = 'ID' \G
 
*************************** 1. row ***************************
 
  Field: ID
 
   Type: varchar(36)
 
   Null: NO
 
    Key: PRI
 
Default: NULL
 
  Extra: 
1 row in set (0.001 sec)

Comment by Jan Lindström (Inactive) [ 2021-09-30 ]

Fixed on 47ba55230

Comment by Brendon Abbott [ 2021-10-11 ]

After testing a recent build, I have reported my issue as a new ticket MDEV-26803

Comment by Jan Lindström (Inactive) [ 2021-12-02 ]

Issue is not fixed.

Comment by Mark Reibert [ 2021-12-02 ]

For what it is worth, on a large (9.5TB) Zabbix database I was experiencing this problem in epidemic proportions with MariaDB 10.4.19 and 10.4.21, but have not observed a single incident since upgrading to 10.4.22.

Comment by Mark Reibert [ 2021-12-03 ]

It seems I jinxed 10.4.22. Not more than an hour or two after I typed that previous comment claiming "all is fine" I had a node SIGABRT with 10.4.22. Doh!

Here is the full log from the crash:

Dec 02 21:15:02 mysqld[1566419]: mysqld: /home/buildbot/buildbot/build/mariadb-10.4.22/wsrep-lib/include/wsrep/client_state.hpp:668: int wsrep::client_state::bf_abort(wsrep::seqno): Assertion `mode_ == m_local || transaction_.is_streaming()' failed.
 
Dec 02 21:15:02 mysqld[1566419]: 211202 21:15:02 [ERROR] mysqld got signal 6 ;
 
Dec 02 21:15:02 mysqld[1566419]: This could be because you hit a bug. It is also possible that this binary
 
Dec 02 21:15:02 mysqld[1566419]: or one of the libraries it was linked against is corrupt, improperly built,
 
Dec 02 21:15:02 mysqld[1566419]: or misconfigured. This error can also be caused by malfunctioning hardware.
 
Dec 02 21:15:02 mysqld[1566419]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
Dec 02 21:15:02 mysqld[1566419]: We will try our best to scrape up some info that will hopefully help
 
Dec 02 21:15:02 mysqld[1566419]: diagnose the problem, but since we have already crashed,
 
Dec 02 21:15:02 mysqld[1566419]: something is definitely wrong and this may fail.
 
Dec 02 21:15:02 mysqld[1566419]: Server version: 10.4.22-MariaDB-1:10.4.22+maria~focal-log
 
Dec 02 21:15:02 mysqld[1566419]: key_buffer_size=134217728
 
Dec 02 21:15:02 mysqld[1566419]: read_buffer_size=131072
 
Dec 02 21:15:02 mysqld[1566419]: max_used_connections=3
 
Dec 02 21:15:02 mysqld[1566419]: max_threads=403
 
Dec 02 21:15:02 mysqld[1566419]: thread_count=73
 
Dec 02 21:15:02 mysqld[1566419]: It is possible that mysqld could use up to
 
Dec 02 21:15:02 mysqld[1566419]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6795376 K  bytes of memory
 
Dec 02 21:15:02 mysqld[1566419]: Hope that's ok; if not, decrease some variables in the equation.
 
Dec 02 21:15:02 mysqld[1566419]: Thread pointer: 0x7e020c000c48
 
Dec 02 21:15:02 mysqld[1566419]: Attempting backtrace. You can use the following information to find out
 
Dec 02 21:15:02 mysqld[1566419]: where mysqld died. If you see no messages after this, something went
 
Dec 02 21:15:02 mysqld[1566419]: terribly wrong...
 
Dec 02 21:15:02 mysqld[1566419]: stack_bottom = 0x7e031c473e08 thread_stack 0x49000
 
Dec 02 21:15:09 mysqld[1566419]: 2021-12-02 21:15:09 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection to peer 1e77c150-8d79 with addr tcp://192.168.77.160:4567 timed out, no messages seen in PT3S, socket stats: rtt: 1257 rttvar: 1806 rto: 204000 lost: 0 last_data_recv: 3252 cwnd: 10 last_queued_since: 25298782 last_delivered_since: 6616014371 send_queue_length: 0 send_queue_bytes: 0 segment: 0 messages: 0
 
Dec 02 21:15:09 mysqld[1566419]: 2021-12-02 21:15:09 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://192.168.77.160:4567
 
Dec 02 21:15:09 mysqld[1566419]: 2021-12-02 21:15:09 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT6.87437S), skipping check
 
Dec 02 21:15:09 mysqld[1566419]: 2021-12-02 21:15:09 0 [Note] WSREP: evs::proto(2f8c7fd2-91e6, OPERATIONAL, view_id(REG,1e77c150-8d79,104)) suspecting node: 1e77c150-8d79
 
Dec 02 21:15:09 mysqld[1566419]: 2021-12-02 21:15:09 0 [Note] WSREP: evs::proto(2f8c7fd2-91e6, OPERATIONAL, view_id(REG,1e77c150-8d79,104)) suspected node without join message, declaring inactive
 
Dec 02 21:15:10 mysqld[1566419]: 2021-12-02 21:15:10 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') reconnecting to 1e77c150-8d79 (tcp://192.168.77.160:4567), attempt 0
 
Dec 02 21:15:10 mysqld[1566419]: 2021-12-02 21:15:10 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 1e77c150-8d79 tcp://192.168.77.160:4567
 
Dec 02 21:15:10 mysqld[1566419]: 2021-12-02 21:15:10 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') reconnecting to 58fa90ba-871c (tcp://192.168.77.162:4567), attempt 0
 
Dec 02 21:15:10 mysqld[1566419]: 2021-12-02 21:15:10 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 58fa90ba-871c tcp://192.168.77.162:4567
 
Dec 02 21:15:11 mysqld[1566419]: 2021-12-02 21:15:11 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') reconnecting to 1e77c150-8d79 (tcp://192.168.77.160:4567), attempt 0
 
Dec 02 21:15:11 mysqld[1566419]: 2021-12-02 21:15:11 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 1e77c150-8d79 tcp://192.168.77.160:4567
 
Dec 02 21:15:11 mysqld[1566419]: 2021-12-02 21:15:11 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') reconnecting to 58fa90ba-871c (tcp://192.168.77.162:4567), attempt 0
 
Dec 02 21:15:11 mysqld[1566419]: 2021-12-02 21:15:11 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 58fa90ba-871c tcp://192.168.77.162:4567
 
Dec 02 21:15:14 mysqld[1566419]: 2021-12-02 21:15:14 0 [Note] WSREP: evs::proto(2f8c7fd2-91e6, GATHER, view_id(REG,1e77c150-8d79,104)) suspecting node: 58fa90ba-871c
 
Dec 02 21:15:14 mysqld[1566419]: 2021-12-02 21:15:14 0 [Note] WSREP: evs::proto(2f8c7fd2-91e6, GATHER, view_id(REG,1e77c150-8d79,104)) suspected node without join message, declaring inactive
 
Dec 02 21:15:14 mysqld[1566419]: 2021-12-02 21:15:14 0 [Note] WSREP: view(view_id(NON_PRIM,1e77c150-8d79,104) memb {
 
Dec 02 21:15:14 mysqld[1566419]:         2f8c7fd2-91e6,0
 
Dec 02 21:15:14 mysqld[1566419]: } joined {
 
Dec 02 21:15:14 mysqld[1566419]: } left {
 
Dec 02 21:15:14 mysqld[1566419]: } partitioned {
 
Dec 02 21:15:14 mysqld[1566419]:         1e77c150-8d79,0
 
Dec 02 21:15:14 mysqld[1566419]:         58fa90ba-871c,0
 
Dec 02 21:15:14 mysqld[1566419]: })
 
Dec 02 21:15:14 mysqld[1566419]: 2021-12-02 21:15:14 0 [Note] WSREP: view(view_id(NON_PRIM,2f8c7fd2-91e6,105) memb {
 
Dec 02 21:15:14 mysqld[1566419]:         2f8c7fd2-91e6,0
 
Dec 02 21:15:14 mysqld[1566419]: } joined {
 
Dec 02 21:15:14 mysqld[1566419]: } left {
 
Dec 02 21:15:14 mysqld[1566419]: } partitioned {
 
Dec 02 21:15:14 mysqld[1566419]:         1e77c150-8d79,0
 
Dec 02 21:15:14 mysqld[1566419]:         58fa90ba-871c,0
 
Dec 02 21:15:14 mysqld[1566419]: })
 
Dec 02 21:15:14 mysqld[1566419]: 2021-12-02 21:15:14 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
 
Dec 02 21:15:14 mysqld[1566419]: 2021-12-02 21:15:14 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
 
Dec 02 21:15:14 mysqld[1566419]: 2021-12-02 21:15:14 0 [Note] WSREP: Received NON-PRIMARY.
 
Dec 02 21:15:14 mysqld[1566419]: 2021-12-02 21:15:14 0 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 4857087606)
 
Dec 02 21:15:14 mysqld[1566419]: 2021-12-02 21:15:14 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
 
Dec 02 21:15:14 mysqld[1566419]: 2021-12-02 21:15:14 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
 
Dec 02 21:15:14 mysqld[1566419]: 2021-12-02 21:15:14 0 [Note] WSREP: Received NON-PRIMARY.
 
Dec 02 21:15:15 mysqld[1566419]: 2021-12-02 21:15:15 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') turning message relay requesting off
 
Dec 02 21:15:15 mysqld[1566419]: 2021-12-02 21:15:15 0 [Note] WSREP: declaring 1e77c150-8d79 at tcp://192.168.77.160:4567 stable
 
Dec 02 21:15:15 mysqld[1566419]: 2021-12-02 21:15:15 0 [Note] WSREP: declaring 58fa90ba-871c at tcp://192.168.77.162:4567 stable
 
Dec 02 21:15:15 mysqld[1566419]: 2021-12-02 21:15:15 0 [Note] WSREP: Node 1e77c150-8d79 state prim
 
Dec 02 21:15:15 mysqld[1566419]: 2021-12-02 21:15:15 0 [Note] WSREP: view(view_id(PRIM,1e77c150-8d79,106) memb {
 
Dec 02 21:15:15 mysqld[1566419]:         1e77c150-8d79,0
 
Dec 02 21:15:15 mysqld[1566419]:         2f8c7fd2-91e6,0
 
Dec 02 21:15:15 mysqld[1566419]:         58fa90ba-871c,0
 
Dec 02 21:15:15 mysqld[1566419]: } joined {
 
Dec 02 21:15:15 mysqld[1566419]: } left {
 
Dec 02 21:15:15 mysqld[1566419]: } partitioned {
 
Dec 02 21:15:15 mysqld[1566419]: })
 
Dec 02 21:15:15 mysqld[1566419]: 2021-12-02 21:15:15 0 [Note] WSREP: save pc into disk
 
Dec 02 21:15:15 mysqld[1566419]: 2021-12-02 21:15:15 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 3
 
Dec 02 21:15:15 mysqld[1566419]: 2021-12-02 21:15:15 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
 
Dec 02 21:15:15 mysqld[1566419]: 2021-12-02 21:15:15 0 [Note] WSREP: STATE EXCHANGE: sent state msg: f1ed7f1a-53b4-11ec-a50b-9e152798ae9a
 
Dec 02 21:15:15 mysqld[1566419]: 2021-12-02 21:15:15 0 [Note] WSREP: STATE EXCHANGE: got state msg: f1ed7f1a-53b4-11ec-a50b-9e152798ae9a from 0 (zabbix-db04)
 
Dec 02 21:15:15 mysqld[1566419]: 2021-12-02 21:15:15 0 [Note] WSREP: STATE EXCHANGE: got state msg: f1ed7f1a-53b4-11ec-a50b-9e152798ae9a from 1 (zabbix-db05)
 
Dec 02 21:15:15 mysqld[1566419]: 2021-12-02 21:15:15 0 [Note] WSREP: STATE EXCHANGE: got state msg: f1ed7f1a-53b4-11ec-a50b-9e152798ae9a from 2 (zabbix-db06)
 
Dec 02 21:15:15 mysqld[1566419]: 2021-12-02 21:15:15 0 [Note] WSREP: Quorum results:
 
Dec 02 21:15:15 mysqld[1566419]:         version    = 6,
 
Dec 02 21:15:15 mysqld[1566419]:         component  = PRIMARY,
 
Dec 02 21:15:15 mysqld[1566419]:         conf_id    = 39,
 
Dec 02 21:15:15 mysqld[1566419]:         members    = 2/3 (joined/total),
 
Dec 02 21:15:15 mysqld[1566419]:         act_id     = 4857088711,
 
Dec 02 21:15:15 mysqld[1566419]:         last_appl. = 4857088690,
 
Dec 02 21:15:15 mysqld[1566419]:         protocols  = 2/10/4 (gcs/repl/appl),
 
Dec 02 21:15:15 mysqld[1566419]:         vote policy= 0,
 
Dec 02 21:15:15 mysqld[1566419]:         group UUID = 810a5d0c-231c-11e9-b756-dbb7899d7dd0
 
Dec 02 21:15:15 mysqld[1566419]: 2021-12-02 21:15:15 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
 
Dec 02 21:15:15 mysqld[1566419]: 2021-12-02 21:15:15 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 4857088712)
 
Dec 02 21:15:18 mysqld[1566419]: /usr/sbin/mysqld(my_print_stacktrace+0x32)[0x564297590d12]
 
Dec 02 21:15:18 mysqld[1566419]: /usr/sbin/mysqld(handle_fatal_signal+0x55d)[0x564297042a5d]
 
Dec 02 21:15:23 mysqld[1566419]: 2021-12-02 21:15:23 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection to peer 1e77c150-8d79 with addr tcp://192.168.77.160:4567 timed out, no messages seen in PT3S, socket stats: rtt: 2620 rttvar: 3448 rto: 204000 lost: 0 last_data_recv: 2696 cwnd: 10 last_queued_since: 559365 last_delivered_since: 5870988113 send_queue_length: 0 send_queue_bytes: 0 segment: 0 messages: 0
 
Dec 02 21:15:23 mysqld[1566419]: 2021-12-02 21:15:23 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://192.168.77.160:4567
 
Dec 02 21:15:23 mysqld[1566419]: 2021-12-02 21:15:23 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT5.93833S), skipping check
 
Dec 02 21:15:23 mysqld[1566419]: 2021-12-02 21:15:23 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 1e77c150-8d79 tcp://192.168.77.160:4567
 
Dec 02 21:15:23 mysqld[1566419]: 2021-12-02 21:15:23 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 58fa90ba-871c tcp://192.168.77.162:4567
 
Dec 02 21:15:24 mysqld[1566419]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0)[0x7f348779e3c0]
 
Dec 02 21:15:24 mysqld[1566419]: 2021-12-02 21:15:24 0 [Note] WSREP: evs::proto(2f8c7fd2-91e6, OPERATIONAL, view_id(REG,1e77c150-8d79,106)) suspecting node: 1e77c150-8d79
 
Dec 02 21:15:24 mysqld[1566419]: 2021-12-02 21:15:24 0 [Note] WSREP: evs::proto(2f8c7fd2-91e6, OPERATIONAL, view_id(REG,1e77c150-8d79,106)) suspected node without join message, declaring inactive
 
Dec 02 21:15:24 mysqld[1566419]: 2021-12-02 21:15:24 0 [Note] WSREP: view(view_id(NON_PRIM,1e77c150-8d79,106) memb {
 
Dec 02 21:15:24 mysqld[1566419]:         2f8c7fd2-91e6,0
 
Dec 02 21:15:24 mysqld[1566419]: } joined {
 
Dec 02 21:15:24 mysqld[1566419]: } left {
 
Dec 02 21:15:24 mysqld[1566419]: } partitioned {
 
Dec 02 21:15:24 mysqld[1566419]:         1e77c150-8d79,0
 
Dec 02 21:15:24 mysqld[1566419]:         58fa90ba-871c,0
 
Dec 02 21:15:24 mysqld[1566419]: })
 
Dec 02 21:15:24 mysqld[1566419]: 2021-12-02 21:15:24 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
 
Dec 02 21:15:24 mysqld[1566419]: 2021-12-02 21:15:24 0 [Note] WSREP: view(view_id(NON_PRIM,2f8c7fd2-91e6,107) memb {
 
Dec 02 21:15:24 mysqld[1566419]:         2f8c7fd2-91e6,0
 
Dec 02 21:15:24 mysqld[1566419]: } joined {
 
Dec 02 21:15:24 mysqld[1566419]: } left {
 
Dec 02 21:15:24 mysqld[1566419]: } partitioned {
 
Dec 02 21:15:24 mysqld[1566419]:         1e77c150-8d79,0
 
Dec 02 21:15:24 mysqld[1566419]:         58fa90ba-871c,0
 
Dec 02 21:15:24 mysqld[1566419]: })
 
Dec 02 21:15:24 mysqld[1566419]: 2021-12-02 21:15:24 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
 
Dec 02 21:15:24 mysqld[1566419]: 2021-12-02 21:15:24 0 [Note] WSREP: Received NON-PRIMARY.
 
Dec 02 21:15:24 mysqld[1566419]: 2021-12-02 21:15:24 0 [Note] WSREP: Shifting PRIMARY -> OPEN (TO: 4857089108)
 
Dec 02 21:15:24 mysqld[1566419]: 2021-12-02 21:15:24 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
 
Dec 02 21:15:24 mysqld[1566419]: 2021-12-02 21:15:24 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
 
Dec 02 21:15:24 mysqld[1566419]: 2021-12-02 21:15:24 0 [Note] WSREP: Received NON-PRIMARY.
 
Dec 02 21:15:25 mysqld[1566419]: 2021-12-02 21:15:25 0 [Note] WSREP: declaring 1e77c150-8d79 at tcp://192.168.77.160:4567 stable
 
Dec 02 21:15:25 mysqld[1566419]: 2021-12-02 21:15:25 0 [Note] WSREP: declaring 58fa90ba-871c at tcp://192.168.77.162:4567 stable
 
Dec 02 21:15:25 mysqld[1566419]: 2021-12-02 21:15:25 0 [Note] WSREP: Node 1e77c150-8d79 state prim
 
Dec 02 21:15:25 mysqld[1566419]: 2021-12-02 21:15:25 0 [Note] WSREP: view(view_id(PRIM,1e77c150-8d79,108) memb {
 
Dec 02 21:15:25 mysqld[1566419]:         1e77c150-8d79,0
 
Dec 02 21:15:25 mysqld[1566419]:         2f8c7fd2-91e6,0
 
Dec 02 21:15:25 mysqld[1566419]:         58fa90ba-871c,0
 
Dec 02 21:15:25 mysqld[1566419]: } joined {
 
Dec 02 21:15:25 mysqld[1566419]: } left {
 
Dec 02 21:15:25 mysqld[1566419]: } partitioned {
 
Dec 02 21:15:25 mysqld[1566419]: })
 
Dec 02 21:15:25 mysqld[1566419]: 2021-12-02 21:15:25 0 [Note] WSREP: save pc into disk
 
Dec 02 21:15:25 mysqld[1566419]: 2021-12-02 21:15:25 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 3
 
Dec 02 21:15:25 mysqld[1566419]: 2021-12-02 21:15:25 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
 
Dec 02 21:15:25 mysqld[1566419]: 2021-12-02 21:15:25 0 [Note] WSREP: STATE EXCHANGE: sent state msg: f79778e9-53b4-11ec-9167-0b066174fefa
 
Dec 02 21:15:25 mysqld[1566419]: 2021-12-02 21:15:25 0 [Note] WSREP: STATE EXCHANGE: got state msg: f79778e9-53b4-11ec-9167-0b066174fefa from 0 (zabbix-db04)
 
Dec 02 21:15:25 mysqld[1566419]: 2021-12-02 21:15:25 0 [Note] WSREP: STATE EXCHANGE: got state msg: f79778e9-53b4-11ec-9167-0b066174fefa from 1 (zabbix-db05)
 
Dec 02 21:15:25 mysqld[1566419]: 2021-12-02 21:15:25 0 [Note] WSREP: STATE EXCHANGE: got state msg: f79778e9-53b4-11ec-9167-0b066174fefa from 2 (zabbix-db06)
 
Dec 02 21:15:25 mysqld[1566419]: 2021-12-02 21:15:25 0 [Note] WSREP: Quorum results:
 
Dec 02 21:15:25 mysqld[1566419]:         version    = 6,
 
Dec 02 21:15:25 mysqld[1566419]:         component  = PRIMARY,
 
Dec 02 21:15:25 mysqld[1566419]:         conf_id    = 40,
 
Dec 02 21:15:25 mysqld[1566419]:         members    = 2/3 (joined/total),
 
Dec 02 21:15:25 mysqld[1566419]:         act_id     = 4857089126,
 
Dec 02 21:15:25 mysqld[1566419]:         last_appl. = 4857088690,
 
Dec 02 21:15:25 mysqld[1566419]:         protocols  = 2/10/4 (gcs/repl/appl),
 
Dec 02 21:15:25 mysqld[1566419]:         vote policy= 0,
 
Dec 02 21:15:25 mysqld[1566419]:         group UUID = 810a5d0c-231c-11e9-b756-dbb7899d7dd0
 
Dec 02 21:15:25 mysqld[1566419]: 2021-12-02 21:15:25 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
 
Dec 02 21:15:25 mysqld[1566419]: 2021-12-02 21:15:25 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 4857089127)
 
Dec 02 21:15:27 mysqld[1566419]: 2021-12-02 21:15:27 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') turning message relay requesting off
 
Dec 02 21:15:38 mysqld[1566419]: 2021-12-02 21:15:38 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection to peer 1e77c150-8d79 with addr tcp://192.168.77.160:4567 timed out, no messages seen in PT3S, socket stats: rtt: 1508 rttvar: 2174 rto: 204000 lost: 0 last_data_recv: 3732 cwnd: 10 last_queued_since: 412691 last_delivered_since: 6254106919 send_queue_length: 1 send_queue_bytes: 80 segment: 0 messages: 1
 
Dec 02 21:15:38 mysqld[1566419]: 2021-12-02 21:15:38 0 [Note] WSREP: Deferred close timer started for socket with remote endpoint: tcp://192.168.77.160:51202
 
Dec 02 21:15:38 mysqld[1566419]: 2021-12-02 21:15:38 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://192.168.77.160:4567
 
Dec 02 21:15:38 mysqld[1566419]: 2021-12-02 21:15:38 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT6.52657S), skipping check
 
Dec 02 21:15:38 mysqld[1566419]: 2021-12-02 21:15:38 0 [Note] WSREP: Deferred close timer handle_wait Operation aborted. for 0x7f13a003fb10
 
Dec 02 21:15:38 mysqld[1566419]: 2021-12-02 21:15:38 0 [Note] WSREP: Deferred close timer destruct
 
Dec 02 21:15:39 mysqld[1566419]: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x7f34872a418b]
 
Dec 02 21:15:39 mysqld[1566419]: 2021-12-02 21:15:39 0 [Note] WSREP: evs::proto(2f8c7fd2-91e6, OPERATIONAL, view_id(REG,1e77c150-8d79,108)) suspecting node: 1e77c150-8d79
 
Dec 02 21:15:39 mysqld[1566419]: 2021-12-02 21:15:39 0 [Note] WSREP: evs::proto(2f8c7fd2-91e6, OPERATIONAL, view_id(REG,1e77c150-8d79,108)) suspected node without join message, declaring inactive
 
Dec 02 21:15:39 mysqld[1566419]: /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b)[0x7f3487283859]
 
Dec 02 21:15:39 mysqld[1566419]: 2021-12-02 21:15:39 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') reconnecting to 1e77c150-8d79 (tcp://192.168.77.160:4567), attempt 0
 
Dec 02 21:15:39 mysqld[1566419]: 2021-12-02 21:15:39 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 1e77c150-8d79 tcp://192.168.77.160:4567
 
Dec 02 21:15:40 mysqld[1566419]: 2021-12-02 21:15:40 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') reconnecting to 58fa90ba-871c (tcp://192.168.77.162:4567), attempt 0
 
Dec 02 21:15:40 mysqld[1566419]: /lib/x86_64-linux-gnu/libc.so.6(+0x25729)[0x7f3487283729]
 
Dec 02 21:15:40 mysqld[1566419]: 2021-12-02 21:15:40 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 58fa90ba-871c tcp://192.168.77.162:4567
 
Dec 02 21:15:40 mysqld[1566419]: /lib/x86_64-linux-gnu/libc.so.6(+0x36f36)[0x7f3487294f36]
 
Dec 02 21:15:41 mysqld[1566419]: 2021-12-02 21:15:41 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') reconnecting to 1e77c150-8d79 (tcp://192.168.77.160:4567), attempt 0
 
Dec 02 21:15:41 mysqld[1566419]: 2021-12-02 21:15:41 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 1e77c150-8d79 tcp://192.168.77.160:4567
 
Dec 02 21:15:41 mysqld[1566419]: 2021-12-02 21:15:41 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') reconnecting to 58fa90ba-871c (tcp://192.168.77.162:4567), attempt 0
 
Dec 02 21:15:41 mysqld[1566419]: 2021-12-02 21:15:41 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 58fa90ba-871c tcp://192.168.77.162:4567
 
Dec 02 21:15:41 mysqld[1566419]: 2021-12-02 21:15:41 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 1e77c150-8d79 tcp://192.168.77.160:4567
 
Dec 02 21:15:43 mysqld[1566419]: 2021-12-02 21:15:43 0 [Note] WSREP: evs::proto(2f8c7fd2-91e6, GATHER, view_id(REG,1e77c150-8d79,108)) suspecting node: 58fa90ba-871c
 
Dec 02 21:15:43 mysqld[1566419]: 2021-12-02 21:15:43 0 [Note] WSREP: evs::proto(2f8c7fd2-91e6, GATHER, view_id(REG,1e77c150-8d79,108)) suspected node without join message, declaring inactive
 
Dec 02 21:15:44 mysqld[1566419]: 2021-12-02 21:15:44 0 [Note] WSREP: view(view_id(NON_PRIM,1e77c150-8d79,108) memb {
 
Dec 02 21:15:44 mysqld[1566419]:         2f8c7fd2-91e6,0
 
Dec 02 21:15:44 mysqld[1566419]: } joined {
 
Dec 02 21:15:44 mysqld[1566419]: } left {
 
Dec 02 21:15:44 mysqld[1566419]: } partitioned {
 
Dec 02 21:15:44 mysqld[1566419]:         1e77c150-8d79,0
 
Dec 02 21:15:44 mysqld[1566419]:         58fa90ba-871c,0
 
Dec 02 21:15:44 mysqld[1566419]: })
 
Dec 02 21:15:44 mysqld[1566419]: 2021-12-02 21:15:44 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
 
Dec 02 21:15:44 mysqld[1566419]: 2021-12-02 21:15:44 0 [Note] WSREP: view(view_id(NON_PRIM,2f8c7fd2-91e6,109) memb {
 
Dec 02 21:15:44 mysqld[1566419]:         2f8c7fd2-91e6,0
 
Dec 02 21:15:44 mysqld[1566419]: } joined {
 
Dec 02 21:15:44 mysqld[1566419]: } left {
 
Dec 02 21:15:44 mysqld[1566419]: } partitioned {
 
Dec 02 21:15:44 mysqld[1566419]:         1e77c150-8d79,0
 
Dec 02 21:15:44 mysqld[1566419]:         58fa90ba-871c,0
 
Dec 02 21:15:44 mysqld[1566419]: })
 
Dec 02 21:15:44 mysqld[1566419]: 2021-12-02 21:15:44 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
 
Dec 02 21:15:44 mysqld[1566419]: 2021-12-02 21:15:44 0 [Note] WSREP: Received NON-PRIMARY.
 
Dec 02 21:15:44 mysqld[1566419]: 2021-12-02 21:15:44 0 [Note] WSREP: Shifting PRIMARY -> OPEN (TO: 4857090486)
 
Dec 02 21:15:44 mysqld[1566419]: 2021-12-02 21:15:44 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
 
Dec 02 21:15:44 mysqld[1566419]: 2021-12-02 21:15:44 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
 
Dec 02 21:15:44 mysqld[1566419]: 2021-12-02 21:15:44 0 [Note] WSREP: Received NON-PRIMARY.
 
Dec 02 21:15:45 mysqld[1566419]: 2021-12-02 21:15:45 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') turning message relay requesting off
 
Dec 02 21:15:46 mysqld[1566419]: 2021-12-02 21:15:46 0 [Note] WSREP: declaring 1e77c150-8d79 at tcp://192.168.77.160:4567 stable
 
Dec 02 21:15:46 mysqld[1566419]: 2021-12-02 21:15:46 0 [Note] WSREP: declaring 58fa90ba-871c at tcp://192.168.77.162:4567 stable
 
Dec 02 21:15:46 mysqld[1566419]: 2021-12-02 21:15:46 0 [Note] WSREP: Node 1e77c150-8d79 state prim
 
Dec 02 21:15:46 mysqld[1566419]: 2021-12-02 21:15:46 0 [Note] WSREP: view(view_id(PRIM,1e77c150-8d79,110) memb {
 
Dec 02 21:15:46 mysqld[1566419]:         1e77c150-8d79,0
 
Dec 02 21:15:46 mysqld[1566419]:         2f8c7fd2-91e6,0
 
Dec 02 21:15:46 mysqld[1566419]:         58fa90ba-871c,0
 
Dec 02 21:15:46 mysqld[1566419]: } joined {
 
Dec 02 21:15:46 mysqld[1566419]: } left {
 
Dec 02 21:15:46 mysqld[1566419]: } partitioned {
 
Dec 02 21:15:46 mysqld[1566419]: })
 
Dec 02 21:15:46 mysqld[1566419]: 2021-12-02 21:15:46 0 [Note] WSREP: save pc into disk
 
Dec 02 21:15:46 mysqld[1566419]: 2021-12-02 21:15:46 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 3
 
Dec 02 21:15:46 mysqld[1566419]: 2021-12-02 21:15:46 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
 
Dec 02 21:15:46 mysqld[1566419]: 2021-12-02 21:15:46 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 03d428c1-53b5-11ec-be9c-4bd7ca174788
 
Dec 02 21:15:46 mysqld[1566419]: 2021-12-02 21:15:46 0 [Note] WSREP: STATE EXCHANGE: got state msg: 03d428c1-53b5-11ec-be9c-4bd7ca174788 from 0 (zabbix-db04)
 
Dec 02 21:15:46 mysqld[1566419]: 2021-12-02 21:15:46 0 [Note] WSREP: STATE EXCHANGE: got state msg: 03d428c1-53b5-11ec-be9c-4bd7ca174788 from 1 (zabbix-db05)
 
Dec 02 21:15:46 mysqld[1566419]: 2021-12-02 21:15:46 0 [Note] WSREP: STATE EXCHANGE: got state msg: 03d428c1-53b5-11ec-be9c-4bd7ca174788 from 2 (zabbix-db06)
 
Dec 02 21:15:46 mysqld[1566419]: 2021-12-02 21:15:46 0 [Note] WSREP: Quorum results:
 
Dec 02 21:15:46 mysqld[1566419]:         version    = 6,
 
Dec 02 21:15:46 mysqld[1566419]:         component  = PRIMARY,
 
Dec 02 21:15:46 mysqld[1566419]:         conf_id    = 42,
 
Dec 02 21:15:46 mysqld[1566419]:         members    = 2/3 (joined/total),
 
Dec 02 21:15:46 mysqld[1566419]:         act_id     = 4857091473,
 
Dec 02 21:15:46 mysqld[1566419]:         last_appl. = 4857091469,
 
Dec 02 21:15:46 mysqld[1566419]:         protocols  = 2/10/4 (gcs/repl/appl),
 
Dec 02 21:15:46 mysqld[1566419]:         vote policy= 0,
 
Dec 02 21:15:46 mysqld[1566419]:         group UUID = 810a5d0c-231c-11e9-b756-dbb7899d7dd0
 
Dec 02 21:15:46 mysqld[1566419]: 2021-12-02 21:15:46 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
 
Dec 02 21:15:46 mysqld[1566419]: 2021-12-02 21:15:46 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 4857091474)
 
Dec 02 21:15:52 mysqld[1566419]: 2021-12-02 21:15:52 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection to peer 1e77c150-8d79 with addr tcp://192.168.77.160:4567 timed out, no messages seen in PT3S, socket stats: rtt: 1209 rttvar: 1476 rto: 204000 lost: 0 last_data_recv: 2284 cwnd: 10 last_queued_since: 363245 last_delivered_since: 5600232142 send_queue_length: 6 send_queue_bytes: 480 segment: 0 messages: 6
 
Dec 02 21:15:52 mysqld[1566419]: 2021-12-02 21:15:52 0 [Note] WSREP: Deferred close timer started for socket with remote endpoint: tcp://192.168.77.160:51208
 
Dec 02 21:15:52 mysqld[1566419]: 2021-12-02 21:15:52 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://192.168.77.160:4567
 
Dec 02 21:15:52 mysqld[1566419]: 2021-12-02 21:15:52 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT5.99973S), skipping check
 
Dec 02 21:15:52 mysqld[1566419]: 2021-12-02 21:15:52 0 [Note] WSREP: Deferred close timer handle_wait Operation aborted. for 0x56429cac2270
 
Dec 02 21:15:52 mysqld[1566419]: 2021-12-02 21:15:52 0 [Note] WSREP: Deferred close timer destruct
 
Dec 02 21:15:52 mysqld[1566419]: 2021-12-02 21:15:52 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 1e77c150-8d79 tcp://192.168.77.160:4567
 
Dec 02 21:15:52 mysqld[1566419]: 2021-12-02 21:15:52 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 58fa90ba-871c tcp://192.168.77.162:4567
 
Dec 02 21:15:52 mysqld[1566419]: /usr/sbin/mysqld(_Z14wsrep_bf_abortP3THDS0_+0x6e9)[0x564296fbd7f9]
 
Dec 02 21:15:53 mysqld[1566419]: 2021-12-02 21:15:53 0 [Note] WSREP: evs::proto(2f8c7fd2-91e6, OPERATIONAL, view_id(REG,1e77c150-8d79,110)) suspecting node: 1e77c150-8d79
 
Dec 02 21:15:53 mysqld[1566419]: 2021-12-02 21:15:53 0 [Note] WSREP: evs::proto(2f8c7fd2-91e6, OPERATIONAL, view_id(REG,1e77c150-8d79,110)) suspected node without join message, declaring inactive
 
Dec 02 21:15:53 mysqld[1566419]: 2021-12-02 21:15:53 0 [Note] WSREP: view(view_id(NON_PRIM,1e77c150-8d79,110) memb {
 
Dec 02 21:15:53 mysqld[1566419]:         2f8c7fd2-91e6,0
 
Dec 02 21:15:53 mysqld[1566419]: } joined {
 
Dec 02 21:15:53 mysqld[1566419]: } left {
 
Dec 02 21:15:53 mysqld[1566419]: } partitioned {
 
Dec 02 21:15:53 mysqld[1566419]:         1e77c150-8d79,0
 
Dec 02 21:15:53 mysqld[1566419]:         58fa90ba-871c,0
 
Dec 02 21:15:53 mysqld[1566419]: })
 
Dec 02 21:15:53 mysqld[1566419]: 2021-12-02 21:15:53 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
 
Dec 02 21:15:53 mysqld[1566419]: 2021-12-02 21:15:53 0 [Note] WSREP: view(view_id(NON_PRIM,2f8c7fd2-91e6,111) memb {
 
Dec 02 21:15:53 mysqld[1566419]:         2f8c7fd2-91e6,0
 
Dec 02 21:15:53 mysqld[1566419]: } joined {
 
Dec 02 21:15:53 mysqld[1566419]: } left {
 
Dec 02 21:15:53 mysqld[1566419]: } partitioned {
 
Dec 02 21:15:53 mysqld[1566419]:         1e77c150-8d79,0
 
Dec 02 21:15:53 mysqld[1566419]:         58fa90ba-871c,0
 
Dec 02 21:15:53 mysqld[1566419]: })
 
Dec 02 21:15:53 mysqld[1566419]: 2021-12-02 21:15:53 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
 
Dec 02 21:15:53 mysqld[1566419]: 2021-12-02 21:15:53 0 [Note] WSREP: Received NON-PRIMARY.
 
Dec 02 21:15:53 mysqld[1566419]: 2021-12-02 21:15:53 0 [Note] WSREP: Shifting PRIMARY -> OPEN (TO: 4857091694)
 
Dec 02 21:15:53 mysqld[1566419]: 2021-12-02 21:15:53 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
 
Dec 02 21:15:53 mysqld[1566419]: 2021-12-02 21:15:53 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
 
Dec 02 21:15:53 mysqld[1566419]: 2021-12-02 21:15:53 0 [Note] WSREP: Received NON-PRIMARY.
 
Dec 02 21:15:53 mysqld[1566419]: /usr/sbin/mysqld(wsrep_thd_bf_abort+0x1d)[0x564296fc56bd]
 
Dec 02 21:15:53 mysqld[1566419]: /usr/sbin/mysqld(+0xaab1b6)[0x5642971fb1b6]
 
Dec 02 21:15:54 mysqld[1566419]: /usr/sbin/mysqld(+0xb0cd84)[0x56429725cd84]
 
Dec 02 21:15:54 mysqld[1566419]: /usr/sbin/mysqld(+0xb12e62)[0x564297262e62]
 
Dec 02 21:15:55 mysqld[1566419]: 2021-12-02 21:15:55 0 [Note] WSREP: declaring 1e77c150-8d79 at tcp://192.168.77.160:4567 stable
 
Dec 02 21:15:55 mysqld[1566419]: 2021-12-02 21:15:55 0 [Note] WSREP: declaring 58fa90ba-871c at tcp://192.168.77.162:4567 stable
 
Dec 02 21:15:55 mysqld[1566419]: 2021-12-02 21:15:55 0 [Note] WSREP: Node 1e77c150-8d79 state prim
 
Dec 02 21:15:55 mysqld[1566419]: 2021-12-02 21:15:55 0 [Note] WSREP: view(view_id(PRIM,1e77c150-8d79,112) memb {
 
Dec 02 21:15:55 mysqld[1566419]:         1e77c150-8d79,0
 
Dec 02 21:15:55 mysqld[1566419]:         2f8c7fd2-91e6,0
 
Dec 02 21:15:55 mysqld[1566419]:         58fa90ba-871c,0
 
Dec 02 21:15:55 mysqld[1566419]: } joined {
 
Dec 02 21:15:55 mysqld[1566419]: } left {
 
Dec 02 21:15:55 mysqld[1566419]: } partitioned {
 
Dec 02 21:15:55 mysqld[1566419]: })
 
Dec 02 21:15:55 mysqld[1566419]: 2021-12-02 21:15:55 0 [Note] WSREP: save pc into disk
 
Dec 02 21:15:55 mysqld[1566419]: 2021-12-02 21:15:55 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 3
 
Dec 02 21:15:55 mysqld[1566419]: 2021-12-02 21:15:55 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
 
Dec 02 21:15:55 mysqld[1566419]: 2021-12-02 21:15:55 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 0931be05-53b5-11ec-bba0-7f8149696efd
 
Dec 02 21:15:55 mysqld[1566419]: 2021-12-02 21:15:55 0 [Note] WSREP: STATE EXCHANGE: got state msg: 0931be05-53b5-11ec-bba0-7f8149696efd from 0 (zabbix-db04)
 
Dec 02 21:15:55 mysqld[1566419]: 2021-12-02 21:15:55 0 [Note] WSREP: STATE EXCHANGE: got state msg: 0931be05-53b5-11ec-bba0-7f8149696efd from 1 (zabbix-db05)
 
Dec 02 21:15:55 mysqld[1566419]: 2021-12-02 21:15:55 0 [Note] WSREP: STATE EXCHANGE: got state msg: 0931be05-53b5-11ec-bba0-7f8149696efd from 2 (zabbix-db06)
 
Dec 02 21:15:55 mysqld[1566419]: 2021-12-02 21:15:55 0 [Note] WSREP: Quorum results:
 
Dec 02 21:15:55 mysqld[1566419]:         version    = 6,
 
Dec 02 21:15:55 mysqld[1566419]:         component  = PRIMARY,
 
Dec 02 21:15:55 mysqld[1566419]:         conf_id    = 43,
 
Dec 02 21:15:55 mysqld[1566419]:         members    = 2/3 (joined/total),
 
Dec 02 21:15:55 mysqld[1566419]:         act_id     = 4857091708,
 
Dec 02 21:15:55 mysqld[1566419]:         last_appl. = 4857091469,
 
Dec 02 21:15:55 mysqld[1566419]:         protocols  = 2/10/4 (gcs/repl/appl),
 
Dec 02 21:15:55 mysqld[1566419]:         vote policy= 0,
 
Dec 02 21:15:55 mysqld[1566419]:         group UUID = 810a5d0c-231c-11e9-b756-dbb7899d7dd0
 
Dec 02 21:15:55 mysqld[1566419]: 2021-12-02 21:15:55 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
 
Dec 02 21:15:55 mysqld[1566419]: 2021-12-02 21:15:55 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 4857091709)
 
Dec 02 21:15:55 mysqld[1566419]: /usr/sbin/mysqld(+0xb9fdd1)[0x5642972efdd1]
 
Dec 02 21:15:55 mysqld[1566419]: /usr/sbin/mysqld(+0xac65b1)[0x5642972165b1]
 
Dec 02 21:15:56 mysqld[1566419]: 2021-12-02 21:15:56 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') turning message relay requesting off
 
Dec 02 21:15:56 mysqld[1566419]: /usr/sbin/mysqld(_ZN7handler10ha_rnd_posEPhS0_+0x50)[0x564297049690]
 
Dec 02 21:15:56 mysqld[1566419]: /usr/sbin/mysqld(+0x818150)[0x564296f68150]
 
Dec 02 21:15:57 mysqld[1566419]: /usr/sbin/mysqld(_ZN14Rows_log_event8find_rowEP14rpl_group_info+0x451)[0x56429715ef91]
 
Dec 02 21:15:57 mysqld[1566419]: /usr/sbin/mysqld(_ZN21Delete_rows_log_event11do_exec_rowEP14rpl_group_info+0x92)[0x56429715f252]
 
Dec 02 21:15:58 mysqld[1566419]: /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x23f)[0x56429715275f]
 
Dec 02 21:15:58 mysqld[1566419]: /usr/sbin/mysqld(_Z18wsrep_apply_eventsP3THDP14Relay_log_infoPKvm+0x1b9)[0x564296fbc979]
 
Dec 02 21:15:59 mysqld[1566419]: /usr/sbin/mysqld(_ZN21Wsrep_applier_service15apply_write_setERKN5wsrep7ws_metaERKNS0_12const_bufferERNS0_14mutable_bufferE+0xa3)[0x564296fa3fc3]
 
Dec 02 21:15:59 mysqld[1566419]: /usr/sbin/mysqld(+0xeaf580)[0x5642975ff580]
 
Dec 02 21:16:00 mysqld[1566419]: /usr/sbin/mysqld(_ZN5wsrep12server_state8on_applyERNS_21high_priority_serviceERKNS_9ws_handleERKNS_7ws_metaERKNS_12const_bufferE+0xc1)[0x564297600441]
 
Dec 02 21:16:00 mysqld[1566419]: /usr/sbin/mysqld(+0xec187c)[0x56429761187c]
 
Dec 02 21:16:05 mysqld[1566419]: 2021-12-02 21:16:05 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection to peer 1e77c150-8d79 with addr tcp://192.168.77.160:4567 timed out, no messages seen in PT3S, socket stats: rtt: 3289 rttvar: 6042 rto: 204000 lost: 0 last_data_recv: 2212 cwnd: 10 last_queued_since: 458625 last_delivered_since: 5346986204 send_queue_length: 0 send_queue_bytes: 0 segment: 0 messages: 0
 
Dec 02 21:16:05 mysqld[1566419]: 2021-12-02 21:16:05 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://192.168.77.160:4567
 
Dec 02 21:16:05 mysqld[1566419]: 2021-12-02 21:16:05 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT5.4279S), skipping check
 
Dec 02 21:16:05 mysqld[1566419]: 2021-12-02 21:16:05 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 58fa90ba-871c tcp://192.168.77.162:4567
 
Dec 02 21:16:05 mysqld[1566419]: 2021-12-02 21:16:05 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 1e77c150-8d79 tcp://192.168.77.160:4567
 
Dec 02 21:16:05 mysqld[1566419]: 2021-12-02 21:16:05 0 [Note] WSREP: view(view_id(NON_PRIM,1e77c150-8d79,112) memb {
 
Dec 02 21:16:05 mysqld[1566419]:         2f8c7fd2-91e6,0
 
Dec 02 21:16:05 mysqld[1566419]: } joined {
 
Dec 02 21:16:05 mysqld[1566419]: } left {
 
Dec 02 21:16:05 mysqld[1566419]: } partitioned {
 
Dec 02 21:16:05 mysqld[1566419]:         1e77c150-8d79,0
 
Dec 02 21:16:05 mysqld[1566419]:         58fa90ba-871c,0
 
Dec 02 21:16:05 mysqld[1566419]: })
 
Dec 02 21:16:05 mysqld[1566419]: 2021-12-02 21:16:05 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
 
Dec 02 21:16:05 mysqld[1566419]: 2021-12-02 21:16:05 0 [Note] WSREP: view(view_id(NON_PRIM,2f8c7fd2-91e6,113) memb {
 
Dec 02 21:16:05 mysqld[1566419]:         2f8c7fd2-91e6,0
 
Dec 02 21:16:05 mysqld[1566419]: } joined {
 
Dec 02 21:16:05 mysqld[1566419]: } left {
 
Dec 02 21:16:05 mysqld[1566419]: } partitioned {
 
Dec 02 21:16:05 mysqld[1566419]:         1e77c150-8d79,0
 
Dec 02 21:16:05 mysqld[1566419]:         58fa90ba-871c,0
 
Dec 02 21:16:05 mysqld[1566419]: })
 
Dec 02 21:16:05 mysqld[1566419]: 2021-12-02 21:16:05 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
 
Dec 02 21:16:05 mysqld[1566419]: 2021-12-02 21:16:05 0 [Note] WSREP: Received NON-PRIMARY.
 
Dec 02 21:16:05 mysqld[1566419]: 2021-12-02 21:16:05 0 [Note] WSREP: Shifting PRIMARY -> OPEN (TO: 4857092761)
 
Dec 02 21:16:05 mysqld[1566419]: 2021-12-02 21:16:05 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
 
Dec 02 21:16:05 mysqld[1566419]: 2021-12-02 21:16:05 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
 
Dec 02 21:16:05 mysqld[1566419]: 2021-12-02 21:16:05 0 [Note] WSREP: Received NON-PRIMARY.
 
Dec 02 21:16:05 mysqld[1566419]: /usr/lib/libgalera_smm.so(+0x639eb)[0x7f34865129eb]
 
Dec 02 21:16:06 mysqld[1566419]: /usr/lib/libgalera_smm.so(+0x7582e)[0x7f348652482e]
 
Dec 02 21:16:06 mysqld[1566419]: /usr/lib/libgalera_smm.so(+0x78408)[0x7f3486527408]
 
Dec 02 21:16:07 mysqld[1566419]: /usr/lib/libgalera_smm.so(+0xabd92)[0x7f348655ad92]
 
Dec 02 21:16:07 mysqld[1566419]: 2021-12-02 21:16:07 0 [Note] WSREP: declaring 1e77c150-8d79 at tcp://192.168.77.160:4567 stable
 
Dec 02 21:16:07 mysqld[1566419]: 2021-12-02 21:16:07 0 [Note] WSREP: declaring 58fa90ba-871c at tcp://192.168.77.162:4567 stable
 
Dec 02 21:16:07 mysqld[1566419]: 2021-12-02 21:16:07 0 [Note] WSREP: Node 1e77c150-8d79 state prim
 
Dec 02 21:16:07 mysqld[1566419]: 2021-12-02 21:16:07 0 [Note] WSREP: view(view_id(PRIM,1e77c150-8d79,114) memb {
 
Dec 02 21:16:07 mysqld[1566419]:         1e77c150-8d79,0
 
Dec 02 21:16:07 mysqld[1566419]:         2f8c7fd2-91e6,0
 
Dec 02 21:16:07 mysqld[1566419]:         58fa90ba-871c,0
 
Dec 02 21:16:07 mysqld[1566419]: } joined {
 
Dec 02 21:16:07 mysqld[1566419]: } left {
 
Dec 02 21:16:07 mysqld[1566419]: } partitioned {
 
Dec 02 21:16:07 mysqld[1566419]: })
 
Dec 02 21:16:07 mysqld[1566419]: 2021-12-02 21:16:07 0 [Note] WSREP: save pc into disk
 
Dec 02 21:16:07 mysqld[1566419]: 2021-12-02 21:16:07 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 3
 
Dec 02 21:16:07 mysqld[1566419]: 2021-12-02 21:16:07 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
 
Dec 02 21:16:07 mysqld[1566419]: 2021-12-02 21:16:07 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 10a57d2b-53b5-11ec-b6d6-efe5f64759fc
 
Dec 02 21:16:07 mysqld[1566419]: 2021-12-02 21:16:07 0 [Note] WSREP: STATE EXCHANGE: got state msg: 10a57d2b-53b5-11ec-b6d6-efe5f64759fc from 0 (zabbix-db04)
 
Dec 02 21:16:07 mysqld[1566419]: 2021-12-02 21:16:07 0 [Note] WSREP: STATE EXCHANGE: got state msg: 10a57d2b-53b5-11ec-b6d6-efe5f64759fc from 1 (zabbix-db05)
 
Dec 02 21:16:07 mysqld[1566419]: 2021-12-02 21:16:07 0 [Note] WSREP: STATE EXCHANGE: got state msg: 10a57d2b-53b5-11ec-b6d6-efe5f64759fc from 2 (zabbix-db06)
 
Dec 02 21:16:07 mysqld[1566419]: 2021-12-02 21:16:07 0 [Note] WSREP: Quorum results:
 
Dec 02 21:16:07 mysqld[1566419]:         version    = 6,
 
Dec 02 21:16:07 mysqld[1566419]:         component  = PRIMARY,
 
Dec 02 21:16:07 mysqld[1566419]:         conf_id    = 44,
 
Dec 02 21:16:07 mysqld[1566419]:         members    = 2/3 (joined/total),
 
Dec 02 21:16:07 mysqld[1566419]:         act_id     = 4857092776,
 
Dec 02 21:16:07 mysqld[1566419]:         last_appl. = 4857091469,
 
Dec 02 21:16:07 mysqld[1566419]:         protocols  = 2/10/4 (gcs/repl/appl),
 
Dec 02 21:16:07 mysqld[1566419]:         vote policy= 0,
 
Dec 02 21:16:07 mysqld[1566419]:         group UUID = 810a5d0c-231c-11e9-b756-dbb7899d7dd0
 
Dec 02 21:16:07 mysqld[1566419]: 2021-12-02 21:16:07 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
 
Dec 02 21:16:07 mysqld[1566419]: 2021-12-02 21:16:07 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 4857092777)
 
Dec 02 21:16:07 mysqld[1566419]: /usr/lib/libgalera_smm.so(+0xabf1b)[0x7f348655af1b]
 
Dec 02 21:16:08 mysqld[1566419]: /usr/lib/libgalera_smm.so(+0xac112)[0x7f348655b112]
 
Dec 02 21:16:08 mysqld[1566419]: 2021-12-02 21:16:08 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') turning message relay requesting off
 
Dec 02 21:16:08 mysqld[1566419]: /usr/lib/libgalera_smm.so(+0x78e00)[0x7f3486527e00]
 
Dec 02 21:16:09 mysqld[1566419]: /usr/lib/libgalera_smm.so(+0x4efe1)[0x7f34864fdfe1]
 
Dec 02 21:16:19 mysqld[1566419]: 2021-12-02 21:16:19 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection to peer 1e77c150-8d79 with addr tcp://192.168.77.160:4567 timed out, no messages seen in PT3S, socket stats: rtt: 5633 rttvar: 10029 rto: 208000 lost: 0 last_data_recv: 2420 cwnd: 10 last_queued_since: 854370 last_delivered_since: 5590235758 send_queue_length: 0 send_queue_bytes: 0 segment: 0 messages: 0
 
Dec 02 21:16:19 mysqld[1566419]: 2021-12-02 21:16:19 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://192.168.77.160:4567
 
Dec 02 21:16:19 mysqld[1566419]: 2021-12-02 21:16:19 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT5.63521S), skipping check
 
Dec 02 21:16:19 mysqld[1566419]: 2021-12-02 21:16:19 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 1e77c150-8d79 tcp://192.168.77.160:4567
 
Dec 02 21:16:19 mysqld[1566419]: 2021-12-02 21:16:19 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 58fa90ba-871c tcp://192.168.77.162:4567
 
Dec 02 21:16:20 mysqld[1566419]: 2021-12-02 21:16:20 0 [Note] WSREP: evs::proto(2f8c7fd2-91e6, OPERATIONAL, view_id(REG,1e77c150-8d79,114)) suspecting node: 1e77c150-8d79
 
Dec 02 21:16:20 mysqld[1566419]: 2021-12-02 21:16:20 0 [Note] WSREP: evs::proto(2f8c7fd2-91e6, OPERATIONAL, view_id(REG,1e77c150-8d79,114)) suspected node without join message, declaring inactive
 
Dec 02 21:16:20 mysqld[1566419]: 2021-12-02 21:16:20 0 [Note] WSREP: view(view_id(NON_PRIM,1e77c150-8d79,114) memb {
 
Dec 02 21:16:20 mysqld[1566419]:         2f8c7fd2-91e6,0
 
Dec 02 21:16:20 mysqld[1566419]: } joined {
 
Dec 02 21:16:20 mysqld[1566419]: } left {
 
Dec 02 21:16:20 mysqld[1566419]: } partitioned {
 
Dec 02 21:16:20 mysqld[1566419]:         1e77c150-8d79,0
 
Dec 02 21:16:20 mysqld[1566419]:         58fa90ba-871c,0
 
Dec 02 21:16:20 mysqld[1566419]: })
 
Dec 02 21:16:20 mysqld[1566419]: 2021-12-02 21:16:20 0 [Note] WSREP: view(view_id(NON_PRIM,2f8c7fd2-91e6,115) memb {
 
Dec 02 21:16:20 mysqld[1566419]:         2f8c7fd2-91e6,0
 
Dec 02 21:16:20 mysqld[1566419]: } joined {
 
Dec 02 21:16:20 mysqld[1566419]: } left {
 
Dec 02 21:16:20 mysqld[1566419]: } partitioned {
 
Dec 02 21:16:20 mysqld[1566419]:         1e77c150-8d79,0
 
Dec 02 21:16:20 mysqld[1566419]:         58fa90ba-871c,0
 
Dec 02 21:16:20 mysqld[1566419]: })
 
Dec 02 21:16:20 mysqld[1566419]: 2021-12-02 21:16:20 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
 
Dec 02 21:16:20 mysqld[1566419]: 2021-12-02 21:16:20 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
 
Dec 02 21:16:20 mysqld[1566419]: 2021-12-02 21:16:20 0 [Note] WSREP: Received NON-PRIMARY.
 
Dec 02 21:16:20 mysqld[1566419]: 2021-12-02 21:16:20 0 [Note] WSREP: Shifting PRIMARY -> OPEN (TO: 4857094033)
 
Dec 02 21:16:20 mysqld[1566419]: 2021-12-02 21:16:20 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
 
Dec 02 21:16:20 mysqld[1566419]: 2021-12-02 21:16:20 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
 
Dec 02 21:16:20 mysqld[1566419]: 2021-12-02 21:16:20 0 [Note] WSREP: Received NON-PRIMARY.
 
Dec 02 21:16:21 mysqld[1566419]: 2021-12-02 21:16:21 0 [Note] WSREP: declaring 1e77c150-8d79 at tcp://192.168.77.160:4567 stable
 
Dec 02 21:16:21 mysqld[1566419]: 2021-12-02 21:16:21 0 [Note] WSREP: declaring 58fa90ba-871c at tcp://192.168.77.162:4567 stable
 
Dec 02 21:16:21 mysqld[1566419]: 2021-12-02 21:16:21 0 [Note] WSREP: Node 1e77c150-8d79 state prim
 
Dec 02 21:16:21 mysqld[1566419]: 2021-12-02 21:16:21 0 [Note] WSREP: view(view_id(PRIM,1e77c150-8d79,116) memb {
 
Dec 02 21:16:21 mysqld[1566419]:         1e77c150-8d79,0
 
Dec 02 21:16:21 mysqld[1566419]:         2f8c7fd2-91e6,0
 
Dec 02 21:16:21 mysqld[1566419]:         58fa90ba-871c,0
 
Dec 02 21:16:21 mysqld[1566419]: } joined {
 
Dec 02 21:16:21 mysqld[1566419]: } left {
 
Dec 02 21:16:21 mysqld[1566419]: } partitioned {
 
Dec 02 21:16:21 mysqld[1566419]: })
 
Dec 02 21:16:21 mysqld[1566419]: 2021-12-02 21:16:21 0 [Note] WSREP: save pc into disk
 
Dec 02 21:16:21 mysqld[1566419]: 2021-12-02 21:16:21 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 3
 
Dec 02 21:16:21 mysqld[1566419]: 2021-12-02 21:16:21 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
 
Dec 02 21:16:21 mysqld[1566419]: 2021-12-02 21:16:21 0 [Warning] WSREP: Last Applied Action message in non-primary configuration from member 0
 
Dec 02 21:16:21 mysqld[1566419]: 2021-12-02 21:16:21 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 18fe4842-53b5-11ec-bd11-4b161e80910e
 
Dec 02 21:16:21 mysqld[1566419]: 2021-12-02 21:16:21 0 [Note] WSREP: STATE EXCHANGE: got state msg: 18fe4842-53b5-11ec-bd11-4b161e80910e from 2 (zabbix-db06)
 
Dec 02 21:16:21 mysqld[1566419]: 2021-12-02 21:16:21 0 [Note] WSREP: STATE EXCHANGE: got state msg: 18fe4842-53b5-11ec-bd11-4b161e80910e from 0 (zabbix-db04)
 
Dec 02 21:16:21 mysqld[1566419]: 2021-12-02 21:16:21 0 [Note] WSREP: STATE EXCHANGE: got state msg: 18fe4842-53b5-11ec-bd11-4b161e80910e from 1 (zabbix-db05)
 
Dec 02 21:16:21 mysqld[1566419]: 2021-12-02 21:16:21 0 [Note] WSREP: Quorum results:
 
Dec 02 21:16:21 mysqld[1566419]:         version    = 6,
 
Dec 02 21:16:21 mysqld[1566419]:         component  = PRIMARY,
 
Dec 02 21:16:21 mysqld[1566419]:         conf_id    = 45,
 
Dec 02 21:16:21 mysqld[1566419]:         members    = 2/3 (joined/total),
 
Dec 02 21:16:21 mysqld[1566419]:         act_id     = 4857094048,
 
Dec 02 21:16:21 mysqld[1566419]:         last_appl. = 4857091469,
 
Dec 02 21:16:21 mysqld[1566419]:         protocols  = 2/10/4 (gcs/repl/appl),
 
Dec 02 21:16:21 mysqld[1566419]:         vote policy= 0,
 
Dec 02 21:16:21 mysqld[1566419]:         group UUID = 810a5d0c-231c-11e9-b756-dbb7899d7dd0
 
Dec 02 21:16:21 mysqld[1566419]: 2021-12-02 21:16:21 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
 
Dec 02 21:16:21 mysqld[1566419]: 2021-12-02 21:16:21 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 4857094049)
 
Dec 02 21:16:23 mysqld[1566419]: 2021-12-02 21:16:23 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') turning message relay requesting off
 
Dec 02 21:16:28 mysqld[1566419]: /usr/sbin/mysqld(_ZN5wsrep18wsrep_provider_v2611run_applierEPNS_21high_priority_serviceE+0x12)[0x564297611e92]
 
Dec 02 21:16:28 mysqld[1566419]: /usr/sbin/mysqld(+0x86e7c3)[0x564296fbe7c3]
 
Dec 02 21:16:28 mysqld[1566419]: /usr/sbin/mysqld(_Z15start_wsrep_THDPv+0x2ec)[0x564296faef1c]
 
Dec 02 21:16:34 mysqld[1566419]: 2021-12-02 21:16:34 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection to peer 1e77c150-8d79 with addr tcp://192.168.77.160:4567 timed out, no messages seen in PT3S, socket stats: rtt: 2200 rttvar: 2183 rto: 204000 lost: 0 last_data_recv: 3048 cwnd: 10 last_queued_since: 420943 last_delivered_since: 6114827970 send_queue_length: 4 send_queue_bytes: 320 segment: 0 messages: 4
 
Dec 02 21:16:34 mysqld[1566419]: 2021-12-02 21:16:34 0 [Note] WSREP: Deferred close timer started for socket with remote endpoint: tcp://192.168.77.160:51218
 
Dec 02 21:16:34 mysqld[1566419]: 2021-12-02 21:16:34 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://192.168.77.160:4567
 
Dec 02 21:16:34 mysqld[1566419]: 2021-12-02 21:16:34 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT6.26427S), skipping check
 
Dec 02 21:16:34 mysqld[1566419]: 2021-12-02 21:16:34 0 [Note] WSREP: Deferred close timer handle_wait Operation aborted. for 0x56429a295c70
 
Dec 02 21:16:34 mysqld[1566419]: 2021-12-02 21:16:34 0 [Note] WSREP: Deferred close timer destruct
 
Dec 02 21:16:34 mysqld[1566419]: 2021-12-02 21:16:34 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 58fa90ba-871c tcp://192.168.77.162:4567
 
Dec 02 21:16:34 mysqld[1566419]: 2021-12-02 21:16:34 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 1e77c150-8d79 tcp://192.168.77.160:4567
 
Dec 02 21:16:34 mysqld[1566419]: 2021-12-02 21:16:34 0 [Note] WSREP: view(view_id(NON_PRIM,1e77c150-8d79,116) memb {
 
Dec 02 21:16:34 mysqld[1566419]:         2f8c7fd2-91e6,0
 
Dec 02 21:16:34 mysqld[1566419]: } joined {
 
Dec 02 21:16:34 mysqld[1566419]: } left {
 
Dec 02 21:16:34 mysqld[1566419]: } partitioned {
 
Dec 02 21:16:34 mysqld[1566419]:         1e77c150-8d79,0
 
Dec 02 21:16:34 mysqld[1566419]:         58fa90ba-871c,0
 
Dec 02 21:16:34 mysqld[1566419]: })
 
Dec 02 21:16:34 mysqld[1566419]: 2021-12-02 21:16:34 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
 
Dec 02 21:16:34 mysqld[1566419]: 2021-12-02 21:16:34 0 [Note] WSREP: view(view_id(NON_PRIM,2f8c7fd2-91e6,117) memb {
 
Dec 02 21:16:34 mysqld[1566419]:         2f8c7fd2-91e6,0
 
Dec 02 21:16:34 mysqld[1566419]: } joined {
 
Dec 02 21:16:34 mysqld[1566419]: } left {
 
Dec 02 21:16:34 mysqld[1566419]: } partitioned {
 
Dec 02 21:16:34 mysqld[1566419]:         1e77c150-8d79,0
 
Dec 02 21:16:34 mysqld[1566419]:         58fa90ba-871c,0
 
Dec 02 21:16:34 mysqld[1566419]: })
 
Dec 02 21:16:34 mysqld[1566419]: 2021-12-02 21:16:34 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
 
Dec 02 21:16:34 mysqld[1566419]: 2021-12-02 21:16:34 0 [Note] WSREP: Received NON-PRIMARY.
 
Dec 02 21:16:34 mysqld[1566419]: 2021-12-02 21:16:34 0 [Note] WSREP: Shifting PRIMARY -> OPEN (TO: 4857095264)
 
Dec 02 21:16:34 mysqld[1566419]: 2021-12-02 21:16:34 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
 
Dec 02 21:16:34 mysqld[1566419]: 2021-12-02 21:16:34 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
 
Dec 02 21:16:34 mysqld[1566419]: 2021-12-02 21:16:34 0 [Note] WSREP: Received NON-PRIMARY.
 
Dec 02 21:16:34 mysqld[1566419]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609)[0x7f3487792609]
 
Dec 02 21:16:35 mysqld[1566419]: 2021-12-02 21:16:35 0 [Note] WSREP: declaring 1e77c150-8d79 at tcp://192.168.77.160:4567 stable
 
Dec 02 21:16:35 mysqld[1566419]: 2021-12-02 21:16:35 0 [Note] WSREP: declaring 58fa90ba-871c at tcp://192.168.77.162:4567 stable
 
Dec 02 21:16:36 mysqld[1566419]: 2021-12-02 21:16:36 0 [Note] WSREP: Node 1e77c150-8d79 state prim
 
Dec 02 21:16:36 mysqld[1566419]: 2021-12-02 21:16:36 0 [Note] WSREP: view(view_id(PRIM,1e77c150-8d79,118) memb {
 
Dec 02 21:16:36 mysqld[1566419]:         1e77c150-8d79,0
 
Dec 02 21:16:36 mysqld[1566419]:         2f8c7fd2-91e6,0
 
Dec 02 21:16:36 mysqld[1566419]:         58fa90ba-871c,0
 
Dec 02 21:16:36 mysqld[1566419]: } joined {
 
Dec 02 21:16:36 mysqld[1566419]: } left {
 
Dec 02 21:16:36 mysqld[1566419]: } partitioned {
 
Dec 02 21:16:36 mysqld[1566419]: })
 
Dec 02 21:16:36 mysqld[1566419]: 2021-12-02 21:16:36 0 [Note] WSREP: save pc into disk
 
Dec 02 21:16:36 mysqld[1566419]: 2021-12-02 21:16:36 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 3
 
Dec 02 21:16:36 mysqld[1566419]: 2021-12-02 21:16:36 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
 
Dec 02 21:16:36 mysqld[1566419]: 2021-12-02 21:16:36 0 [Warning] WSREP: Action message in non-primary configuration from member 2
 
Dec 02 21:16:36 mysqld[1566419]: 2021-12-02 21:16:36 0 [Warning] WSREP: Action message in non-primary configuration from member 2
 
Dec 02 21:16:36 mysqld[1566419]: 2021-12-02 21:16:36 0 [Warning] WSREP: Action message in non-primary configuration from member 2
 
Dec 02 21:16:36 mysqld[1566419]: 2021-12-02 21:16:36 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 21efa719-53b5-11ec-b8ad-2b1fb78722e9
 
Dec 02 21:16:36 mysqld[1566419]: 2021-12-02 21:16:36 0 [Note] WSREP: STATE EXCHANGE: got state msg: 21efa719-53b5-11ec-b8ad-2b1fb78722e9 from 0 (zabbix-db04)
 
Dec 02 21:16:36 mysqld[1566419]: 2021-12-02 21:16:36 0 [Note] WSREP: STATE EXCHANGE: got state msg: 21efa719-53b5-11ec-b8ad-2b1fb78722e9 from 1 (zabbix-db05)
 
Dec 02 21:16:36 mysqld[1566419]: 2021-12-02 21:16:36 0 [Note] WSREP: STATE EXCHANGE: got state msg: 21efa719-53b5-11ec-b8ad-2b1fb78722e9 from 2 (zabbix-db06)
 
Dec 02 21:16:36 mysqld[1566419]: 2021-12-02 21:16:36 0 [Note] WSREP: Quorum results:
 
Dec 02 21:16:36 mysqld[1566419]:         version    = 6,
 
Dec 02 21:16:36 mysqld[1566419]:         component  = PRIMARY,
 
Dec 02 21:16:36 mysqld[1566419]:         conf_id    = 46,
 
Dec 02 21:16:36 mysqld[1566419]:         members    = 2/3 (joined/total),
 
Dec 02 21:16:36 mysqld[1566419]:         act_id     = 4857095281,
 
Dec 02 21:16:36 mysqld[1566419]:         last_appl. = 4857091469,
 
Dec 02 21:16:36 mysqld[1566419]:         protocols  = 2/10/4 (gcs/repl/appl),
 
Dec 02 21:16:36 mysqld[1566419]:         vote policy= 0,
 
Dec 02 21:16:36 mysqld[1566419]:         group UUID = 810a5d0c-231c-11e9-b756-dbb7899d7dd0
 
Dec 02 21:16:36 mysqld[1566419]: 2021-12-02 21:16:36 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
 
Dec 02 21:16:36 mysqld[1566419]: 2021-12-02 21:16:36 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 4857095282)
 
Dec 02 21:16:37 mysqld[1566419]: 2021-12-02 21:16:37 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') turning message relay requesting off
 
Dec 02 21:16:48 mysqld[1566419]: 2021-12-02 21:16:48 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection to peer 1e77c150-8d79 with addr tcp://192.168.77.160:4567 timed out, no messages seen in PT3S, socket stats: rtt: 2007 rttvar: 2859 rto: 204000 lost: 0 last_data_recv: 2264 cwnd: 10 last_queued_since: 422869 last_delivered_since: 5570929836 send_queue_length: 4 send_queue_bytes: 320 segment: 0 messages: 4
 
Dec 02 21:16:48 mysqld[1566419]: 2021-12-02 21:16:48 0 [Note] WSREP: Deferred close timer started for socket with remote endpoint: tcp://192.168.77.160:51222
 
Dec 02 21:16:48 mysqld[1566419]: 2021-12-02 21:16:48 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://192.168.77.160:4567
 
Dec 02 21:16:48 mysqld[1566419]: 2021-12-02 21:16:48 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT5.97614S), skipping check
 
Dec 02 21:16:48 mysqld[1566419]: 2021-12-02 21:16:48 0 [Note] WSREP: Deferred close timer handle_wait Operation aborted. for 0x7f13a0044ea0
 
Dec 02 21:16:48 mysqld[1566419]: 2021-12-02 21:16:48 0 [Note] WSREP: Deferred close timer destruct
 
Dec 02 21:16:48 mysqld[1566419]: 2021-12-02 21:16:48 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 58fa90ba-871c tcp://192.168.77.162:4567
 
Dec 02 21:16:48 mysqld[1566419]: 2021-12-02 21:16:48 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 1e77c150-8d79 tcp://192.168.77.160:4567
 
Dec 02 21:16:48 mysqld[1566419]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x43)[0x7f3487380293]
 
Dec 02 21:16:48 mysqld[1566419]: Trying to get some variables.
 
Dec 02 21:16:48 mysqld[1566419]: Some pointers may be invalid and cause the dump to abort.
 
Dec 02 21:16:48 mysqld[1566419]: Query (0x7f16e8a3da93): delete from problem where r_clock<>0 and r_clock<1638393301
 
Dec 02 21:16:48 mysqld[1566419]: Connection ID (thread ID): 66
 
Dec 02 21:16:48 mysqld[1566419]: Status: NOT_KILLED
 
Dec 02 21:16:48 mysqld[1566419]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on
 
Dec 02 21:16:48 mysqld[1566419]: The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
 
Dec 02 21:16:48 mysqld[1566419]: information that should help you find out what is causing the crash.
 
Dec 02 21:16:48 mysqld[1566419]: Writing a core file...
 
Dec 02 21:16:48 mysqld[1566419]: Working directory at /var/lib/mysql
 
Dec 02 21:16:48 mysqld[1566419]: Resource Limits:
 
Dec 02 21:16:48 mysqld[1566419]: Limit                     Soft Limit           Hard Limit           Units
 
Dec 02 21:16:48 mysqld[1566419]: Max cpu time              unlimited            unlimited            seconds
 
Dec 02 21:16:48 mysqld[1566419]: Max file size             unlimited            unlimited            bytes
 
Dec 02 21:16:48 mysqld[1566419]: Max data size             unlimited            unlimited            bytes
 
Dec 02 21:16:48 mysqld[1566419]: Max stack size            8388608              unlimited            bytes
 
Dec 02 21:16:48 mysqld[1566419]: Max core file size        0                    unlimited            bytes
 
Dec 02 21:16:48 mysqld[1566419]: Max resident set          unlimited            unlimited            bytes
 
Dec 02 21:16:48 mysqld[1566419]: Max processes             5931800              5931800              processes
 
Dec 02 21:16:48 mysqld[1566419]: Max open files            65536                65536                files
 
Dec 02 21:16:48 mysqld[1566419]: Max locked memory         65536                65536                bytes
 
Dec 02 21:16:48 mysqld[1566419]: Max address space         unlimited            unlimited            bytes
 
Dec 02 21:16:48 mysqld[1566419]: Max file locks            unlimited            unlimited            locks
 
Dec 02 21:16:48 mysqld[1566419]: Max pending signals       5931800              5931800              signals
 
Dec 02 21:16:48 mysqld[1566419]: Max msgqueue size         819200               819200               bytes
 
Dec 02 21:16:48 mysqld[1566419]: Max nice priority         0                    0
 
Dec 02 21:16:48 mysqld[1566419]: Max realtime priority     0                    0
 
Dec 02 21:16:48 mysqld[1566419]: Max realtime timeout      unlimited            unlimited            us
 
Dec 02 21:16:48 mysqld[1566419]: Core pattern: /var/crash/%e.%p
 
Dec 02 21:17:13 systemd[1]: mariadb.service: Main process exited, code=killed, status=6/ABRT
 
Dec 02 21:17:13 systemd[1]: mariadb.service: Failed with result 'signal'.

Comment by Mark Reibert [ 2021-12-03 ]

And here is what happened (another SIGABRT) when I simply attempted to restart MariaDB:

Dec 02 22:28:24 systemd[1]: Starting MariaDB 10.4.22 database server...
 
Dec 02 22:29:52 mysqld[2095405]: WSREP: Recovered position 810a5d0c-231c-11e9-b756-dbb7899d7dd0:4857087468
 
Dec 02 22:29:52 mysqld[2095962]: 2021-12-02 22:29:52 0 [Note] /usr/sbin/mysqld (mysqld 10.4.22-MariaDB-1:10.4.22+maria~focal-log) starting as process 2095962 ...
 
Dec 02 22:29:52 mysqld[2095962]: 2021-12-02 22:29:52 0 [Note] WSREP: Loading provider /usr/lib/libgalera_smm.so initial position: 810a5d0c-231c-11e9-b756-dbb7899d7dd0:4857087468
 
Dec 02 22:29:52 mysqld[2095962]: 2021-12-02 22:29:52 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/libgalera_smm.so'
 
Dec 02 22:29:52 mysqld[2095962]: 2021-12-02 22:29:52 0 [Note] WSREP: wsrep_load(): Galera 26.4.9(r819f29cb) by Codership Oy <info@codership.com> loaded successfully.
 
Dec 02 22:29:52 mysqld[2095962]: 2021-12-02 22:29:52 0 [Note] WSREP: CRC-32C: using 64-bit x86 acceleration.
 
Dec 02 22:29:52 mysqld[2095962]: 2021-12-02 22:29:52 0 [Note] WSREP: Found saved state: 810a5d0c-231c-11e9-b756-dbb7899d7dd0:-1, safe_to_bootstrap: 0
 
Dec 02 22:29:52 mysqld[2095962]: 2021-12-02 22:29:52 0 [Note] WSREP: GCache DEBUG: opened preamble:
 
Dec 02 22:29:52 mysqld[2095962]: Version: 2
 
Dec 02 22:29:52 mysqld[2095962]: UUID: 810a5d0c-231c-11e9-b756-dbb7899d7dd0
 
Dec 02 22:29:52 mysqld[2095962]: Seqno: -1 - -1
 
Dec 02 22:29:52 mysqld[2095962]: Offset: -1
 
Dec 02 22:29:52 mysqld[2095962]: Synced: 0
 
Dec 02 22:29:52 mysqld[2095962]: 2021-12-02 22:29:52 0 [Note] WSREP: Recovering GCache ring buffer: version: 2, UUID: 810a5d0c-231c-11e9-b756-dbb7899d7dd0, offset: -1
 
Dec 02 22:29:52 mysqld[2095962]: 2021-12-02 22:29:52 0 [Note] WSREP: GCache::RingBuffer initial scan...  0.0% (           0/140928614424 bytes) complete.
 
Dec 02 22:30:02 mysqld[2095962]: 2021-12-02 22:30:02 0 [Note] WSREP: GCache::RingBuffer initial scan... 79.1% (111525388192/140928614424 bytes) complete.
 
Dec 02 22:30:05 mysqld[2095962]: 2021-12-02 22:30:05 0 [Note] WSREP: GCache::RingBuffer initial scan...100.0% (140928614424/140928614424 bytes) complete.
 
Dec 02 22:30:05 mysqld[2095962]: 2021-12-02 22:30:05 0 [Note] WSREP: Recovering GCache ring buffer: found gapless sequence 4846387206-4857087532
 
Dec 02 22:30:05 mysqld[2095962]: 2021-12-02 22:30:05 0 [Note] WSREP: GCache::RingBuffer unused buffers scan...  0.0% (           0/140821082440 bytes) complete.
 
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: GCache::RingBuffer unused buffers scan...100.0% (140821082440/140821082440 bytes) complete.
 
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: GCache DEBUG: RingBuffer::recover(): found 7/10700334 locked buffers
 
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: GCache DEBUG: RingBuffer::recover(): free space: 107589352/140928614400
 
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 192.168.77.161; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 134400M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 2000000; gcs.fc_master_slave = yes; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.
 
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: Service thread queue flushed.
 
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: ####### Assign initial position for certification: 810a5d0c-231c-11e9-b756-dbb7899d7dd0:4857087468, protocol version: -1
 
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: Start replication
 
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: Connecting with bootstrap option: 0
 
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: Setting GCS initial position to 810a5d0c-231c-11e9-b756-dbb7899d7dd0:4857087468
 
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: protonet asio version 0
 
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: Using CRC-32C for message checksums.
 
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: backend: asio
 
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: gcomm thread scheduling priority set to other:0
 
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: restore pc from disk successfully
 
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: GMCast version 0
 
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: (2f8c7fd2-91e7, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
 
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: (2f8c7fd2-91e7, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
 
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: EVS version 1
 
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: gcomm: connecting to group 'galera_cluster', peer '192.168.77.160:,192.168.77.161:,192.168.77.162:'
 
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: (2f8c7fd2-91e7, 'tcp://0.0.0.0:4567') connection established to 1e77c150-8d79 tcp://192.168.77.160:4567
 
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: (2f8c7fd2-91e7, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
 
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: (2f8c7fd2-91e7, 'tcp://0.0.0.0:4567') connection established to 58fa90ba-871c tcp://192.168.77.162:4567
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: gcomm: connected
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: Opened channel 'galera_cluster'
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 1 [Note] WSREP: Starting rollbacker thread 1
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 2 [Note] WSREP: Starting applier thread 2
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: EVS version upgrade 0 -> 1
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: declaring 1e77c150-8d79 at tcp://192.168.77.160:4567 stable
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: declaring 58fa90ba-871c at tcp://192.168.77.162:4567 stable
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: PC protocol upgrade 0 -> 1
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: Node 1e77c150-8d79 state prim
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: view(view_id(PRIM,1e77c150-8d79,120) memb {
 
Dec 02 22:30:08 mysqld[2095962]:         1e77c150-8d79,0
 
Dec 02 22:30:08 mysqld[2095962]:         2f8c7fd2-91e7,0
 
Dec 02 22:30:08 mysqld[2095962]:         58fa90ba-871c,0
 
Dec 02 22:30:08 mysqld[2095962]: } joined {
 
Dec 02 22:30:08 mysqld[2095962]: } left {
 
Dec 02 22:30:08 mysqld[2095962]: } partitioned {
 
Dec 02 22:30:08 mysqld[2095962]: })
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: save pc into disk
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: discarding pending addr without UUID: tcp://192.168.77.161:4567
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: discarding pending addr proto entry 0x55c63ede73c0
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: clear restored view
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 3
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 677c3d90-53bf-11ec-99c7-8a27c0e85aa9
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: STATE EXCHANGE: got state msg: 677c3d90-53bf-11ec-99c7-8a27c0e85aa9 from 0 (zabbix-db04)
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: STATE EXCHANGE: got state msg: 677c3d90-53bf-11ec-99c7-8a27c0e85aa9 from 2 (zabbix-db06)
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: STATE EXCHANGE: got state msg: 677c3d90-53bf-11ec-99c7-8a27c0e85aa9 from 1 (zabbix-db05)
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: Quorum results:
 
Dec 02 22:30:08 mysqld[2095962]:         version    = 6,
 
Dec 02 22:30:08 mysqld[2095962]:         component  = PRIMARY,
 
Dec 02 22:30:08 mysqld[2095962]:         conf_id    = 48,
 
Dec 02 22:30:08 mysqld[2095962]:         members    = 2/3 (joined/total),
 
Dec 02 22:30:08 mysqld[2095962]:         act_id     = 4857698808,
 
Dec 02 22:30:08 mysqld[2095962]:         last_appl. = 4857698803,
 
Dec 02 22:30:08 mysqld[2095962]:         protocols  = 2/10/4 (gcs/repl/appl),
 
Dec 02 22:30:08 mysqld[2095962]:         vote policy= 0,
 
Dec 02 22:30:08 mysqld[2095962]:         group UUID = 810a5d0c-231c-11e9-b756-dbb7899d7dd0
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 4857698809)
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 2 [Note] WSREP: ####### processing CC 4857698809, local, ordered
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 2 [Note] WSREP: Process first view: 810a5d0c-231c-11e9-b756-dbb7899d7dd0 my uuid: 2f8c7fd2-5334-11ec-91e7-46483b39372d
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 2 [Note] WSREP: Server zabbix-db05 connected to cluster at position 810a5d0c-231c-11e9-b756-dbb7899d7dd0:4857698809 with ID 2f8c7fd2-5334-11ec-91e7-46483b39372d
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 2 [Note] WSREP: Server status change disconnected -> connected
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 2 [Note] WSREP: ####### My UUID: 2f8c7fd2-5334-11ec-91e7-46483b39372d
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 2 [Note] WSREP: Cert index reset to 00000000-0000-0000-0000-000000000000:-1 (proto: 10), state transfer needed: yes
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: Service thread queue flushed.
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 2 [Note] WSREP: ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: -1
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 2 [Note] WSREP: State transfer required:
 
Dec 02 22:30:08 mysqld[2095962]:         Group state: 810a5d0c-231c-11e9-b756-dbb7899d7dd0:4857698809
 
Dec 02 22:30:08 mysqld[2095962]:         Local state: 810a5d0c-231c-11e9-b756-dbb7899d7dd0:4857087468
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 2 [Note] WSREP: Server status change connected -> joiner
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: Joiner monitor thread started to monitor
 
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'joiner' --address '192.168.77.161' --datadir '/var/lib/mysql/' --parent '2095962' --mysqld-args --wsrep_start_position=810a5d0c-231c-11e9-b756-dbb7899d7dd0:4857087468'
 
Dec 02 22:30:08 mysqld[2096028]: WSREP_SST: [INFO] SSL configuration: CA='', CERT='', KEY='', MODE='DISABLED', encrypt='0' (20211202 22:30:08.213)
 
Dec 02 22:30:08 mysqld[2096028]: WSREP_SST: [INFO] Logging all stderr of SST/mariabackup to syslog (20211202 22:30:08.284)
 
Dec 02 22:30:08 -wsrep-sst-joiner[2096168]: Using socat as streamer
 
Dec 02 22:30:09 mysqld[2095962]: 2021-12-02 22:30:09 2 [Note] WSREP: ####### IST uuid:810a5d0c-231c-11e9-b756-dbb7899d7dd0 f: 4857087469, l: 4857698809, STRv: 3
 
Dec 02 22:30:09 mysqld[2095962]: 2021-12-02 22:30:09 2 [Note] WSREP: IST receiver addr using tcp://192.168.77.161:4568
 
Dec 02 22:30:09 mysqld[2095962]: 2021-12-02 22:30:09 2 [Note] WSREP: Prepared IST receiver for 4857087469-4857698809, listening at: tcp://192.168.77.161:4568
 
Dec 02 22:30:09 mysqld[2095962]: 2021-12-02 22:30:09 0 [Note] WSREP: Member 1.0 (zabbix-db05) requested state transfer from 'zabbix-db04'. Selected 0.0 (zabbix-db04)(SYNCED) as donor.
 
Dec 02 22:30:09 mysqld[2095962]: 2021-12-02 22:30:09 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 4857698990)
 
Dec 02 22:30:09 mysqld[2095962]: 2021-12-02 22:30:09 2 [Note] WSREP: Requesting state transfer: success, donor: 0
 
Dec 02 22:30:09 mysqld[2095962]: 2021-12-02 22:30:09 0 [Note] WSREP: 0.0 (zabbix-db04): State transfer to 1.0 (zabbix-db05) complete.
 
Dec 02 22:30:09 mysqld[2095962]: 2021-12-02 22:30:09 0 [Note] WSREP: Member 0.0 (zabbix-db04) synced with group.
 
Dec 02 22:30:09 -wsrep-sst-joiner[2096217]: Removing the sst_in_progress file
 
Dec 02 22:30:09 mysqld[2095962]: 2021-12-02 22:30:09 3 [Note] WSREP: SST received
 
Dec 02 22:30:09 mysqld[2095962]: 2021-12-02 22:30:09 3 [Note] WSREP: Server status change joiner -> initializing
 
Dec 02 22:30:09 mysqld[2095962]: 2021-12-02 22:30:09 3 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
Dec 02 22:30:09 mysqld[2095962]: 2021-12-02 22:30:09 0 [Note] InnoDB: Using Linux native AIO
 
Dec 02 22:30:09 mysqld[2095962]: 2021-12-02 22:30:09 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
 
Dec 02 22:30:09 mysqld[2095962]: 2021-12-02 22:30:09 0 [Note] InnoDB: Uses event mutexes
 
Dec 02 22:30:09 mysqld[2095962]: 2021-12-02 22:30:09 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
 
Dec 02 22:30:09 mysqld[2095962]: 2021-12-02 22:30:09 0 [Note] InnoDB: Number of pools: 1
 
Dec 02 22:30:09 mysqld[2095962]: 2021-12-02 22:30:09 0 [Note] InnoDB: Using SSE2 crc32 instructions
 
Dec 02 22:30:09 mysqld[2095962]: 2021-12-02 22:30:09 0 [Note] InnoDB: Initializing buffer pool, total size = 1087G, instances = 1, chunk size = 128M
 
Dec 02 22:30:10 mysqld[2095962]: 2021-12-02 22:30:10 0 [Note] WSREP: (2f8c7fd2-91e7, 'tcp://0.0.0.0:4567') turning message relay requesting off
 
Dec 02 22:30:49 mysqld[2095962]: 2021-12-02 22:30:49 0 [Note] InnoDB: Completed initialization of buffer pool
 
Dec 02 22:30:49 mysqld[2095962]: 2021-12-02 22:30:49 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
 
Dec 02 22:30:49 mysqld[2095962]: 2021-12-02 22:30:49 0 [Note] InnoDB: Setting log file ./ib_logfile101 size to 6442450944 bytes
 
Dec 02 22:30:50 mysqld[2095962]: 2021-12-02 22:30:50 0 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0
 
Dec 02 22:30:50 mysqld[2095962]: 2021-12-02 22:30:50 0 [Note] InnoDB: New log files created, LSN=192780380073929
 
Dec 02 22:30:50 mysqld[2095962]: 2021-12-02 22:30:50 0 [Note] InnoDB: 47 transaction(s) which must be rolled back or cleaned up in total 20286 row operations to undo
 
Dec 02 22:30:50 mysqld[2095962]: 2021-12-02 22:30:50 0 [Note] InnoDB: Trx id counter is 9733845087
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] InnoDB: Starting in background the rollback of recovered transactions
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] InnoDB: Creating shared tablespace for temporary tables
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] InnoDB: Waiting for purge to start
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] InnoDB: Rolled back recovered transaction 9733845042
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] InnoDB: Rolled back recovered transaction 9733845058
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] InnoDB: Rolled back recovered transaction 9733845034
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] InnoDB: 10.4.22 started; log sequence number 192780380073996; transaction id 9733845091
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] Plugin 'FEEDBACK' is disabled.
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] Server socket created on IP: '0.0.0.0'.
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] WSREP: wsrep_init_schema_and_SR (nil)
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] WSREP: Server initialized
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] WSREP: Server status change initializing -> initialized
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 3 [Note] WSREP: Server status change initialized -> joined
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 3 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 3 [Note] WSREP: Recovered position from storage: 810a5d0c-231c-11e9-b756-dbb7899d7dd0:4857087468
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 3 [Note] WSREP: Recovered view from SST:
 
Dec 02 22:30:59 mysqld[2095962]:   id: 810a5d0c-231c-11e9-b756-dbb7899d7dd0:4850155776
 
Dec 02 22:30:59 mysqld[2095962]:   status: primary
 
Dec 02 22:30:59 mysqld[2095962]:   protocol_version: 4
 
Dec 02 22:30:59 mysqld[2095962]:   capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
 
Dec 02 22:30:59 mysqld[2095962]:   final: no
 
Dec 02 22:30:59 mysqld[2095962]:   own_index: -1
 
Dec 02 22:30:59 mysqld[2095962]:   members(3):
 
Dec 02 22:30:59 mysqld[2095962]:         0: 1e77c150-4403-11ec-8d79-83292a6d14d2, zabbix-db04
 
Dec 02 22:30:59 mysqld[2095962]:         1: 2f8c7fd2-5334-11ec-91e6-46483b39372d, zabbix-db05
 
Dec 02 22:30:59 mysqld[2095962]:         2: 58fa90ba-483b-11ec-871c-6b4c25d26355, zabbix-db06
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 3 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 14 [Note] WSREP: Starting applier thread 14
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 15 [Note] WSREP: Recovered cluster id 810a5d0c-231c-11e9-b756-dbb7899d7dd0
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 3 [Note] WSREP: SST received: 810a5d0c-231c-11e9-b756-dbb7899d7dd0:4857087468
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 3 [Note] WSREP: SST succeeded for position 810a5d0c-231c-11e9-b756-dbb7899d7dd0:4857087468
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] WSREP: Joiner monitor thread ended with total time 51 sec
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 2 [Note] WSREP: Installed new state from SST: 810a5d0c-231c-11e9-b756-dbb7899d7dd0:4857087468
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 2 [Note] WSREP: Receiving IST: 611341 writesets, seqnos 4857087469-4857698809
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] WSREP: ####### IST applying starts with 4857087469
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 31 [Note] WSREP: Starting applier thread 31
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 32 [Note] WSREP: Starting applier thread 32
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 42 [Note] WSREP: Starting applier thread 42
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 46 [Note] WSREP: Starting applier thread 46
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 21 [Note] WSREP: Starting applier thread 21
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 33 [Note] WSREP: Starting applier thread 33
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 64 [Note] WSREP: Starting applier thread 64
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] WSREP: ####### IST current seqno initialized to 4857087469
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 45 [Note] WSREP: Starting applier thread 45
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 48 [Note] WSREP: Starting applier thread 48
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 72 [Note] WSREP: Starting applier thread 72
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 53 [Note] WSREP: Starting applier thread 53
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 29 [Note] WSREP: Starting applier thread 29
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 47 [Note] WSREP: Starting applier thread 47
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 61 [Note] WSREP: Starting applier thread 61
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 30 [Note] WSREP: Starting applier thread 30
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 25 [Note] WSREP: Starting applier thread 25
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 65 [Note] WSREP: Starting applier thread 65
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 49 [Note] WSREP: Starting applier thread 49
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 19 [Note] WSREP: Starting applier thread 19
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 73 [Note] WSREP: Starting applier thread 73
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 70 [Note] WSREP: Starting applier thread 70
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 16 [Note] WSREP: Starting applier thread 16
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 28 [Note] WSREP: Starting applier thread 28
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 10 [Note] WSREP: Starting applier thread 10
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 52 [Note] WSREP: Starting applier thread 52
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 12 [Note] WSREP: Starting applier thread 12
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 23 [Note] WSREP: Starting applier thread 23
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 54 [Note] WSREP: Starting applier thread 54
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 17 [Note] WSREP: Starting applier thread 17
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 55 [Note] WSREP: Starting applier thread 55
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 35 [Note] WSREP: Starting applier thread 35
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 56 [Note] WSREP: Starting applier thread 56
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 24 [Note] WSREP: Starting applier thread 24
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 57 [Note] WSREP: Starting applier thread 57
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 34 [Note] WSREP: Starting applier thread 34
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 58 [Note] WSREP: Starting applier thread 58
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 36 [Note] WSREP: Starting applier thread 36
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 18 [Note] WSREP: Starting applier thread 18
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 59 [Note] WSREP: Starting applier thread 59
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 20 [Note] WSREP: Starting applier thread 20
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 60 [Note] WSREP: Starting applier thread 60
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 37 [Note] WSREP: Starting applier thread 37
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 38 [Note] WSREP: Starting applier thread 38
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 13 [Note] WSREP: Starting applier thread 13
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 39 [Note] WSREP: Starting applier thread 39
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 62 [Note] WSREP: Starting applier thread 62
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 63 [Note] WSREP: Starting applier thread 63
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 40 [Note] WSREP: Starting applier thread 40
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 22 [Note] WSREP: Starting applier thread 22
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 11 [Note] WSREP: Starting applier thread 11
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 66 [Note] WSREP: Starting applier thread 66
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] WSREP: Receiving IST...  0.0% (     0/611341 events) complete.
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 67 [Note] WSREP: Starting applier thread 67
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 68 [Note] WSREP: Starting applier thread 68
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 26 [Note] WSREP: Starting applier thread 26
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 41 [Note] WSREP: Starting applier thread 41
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 69 [Note] WSREP: Starting applier thread 69
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 50 [Note] WSREP: Starting applier thread 50
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 27 [Note] WSREP: Starting applier thread 27
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 43 [Note] WSREP: Starting applier thread 43
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 51 [Note] WSREP: Starting applier thread 51
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 71 [Note] WSREP: Starting applier thread 71
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 44 [Note] WSREP: Starting applier thread 44
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] WSREP: Service thread queue flushed.
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] WSREP: ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:4857087468, protocol version: 5
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] Reading of all Master_info entries succeeded
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] Added new Master_info '' to hash table
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] /usr/sbin/mysqld: ready for connections.
 
Dec 02 22:30:59 mysqld[2095962]: Version: '10.4.22-MariaDB-1:10.4.22+maria~focal-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 21 [Warning] WSREP: no THD for trx: 9733845039
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 14 [Warning] WSREP: no THD for trx: 9733845033
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 42 [Warning] WSREP: no THD for trx: 9733845037
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 69 [Warning] WSREP: no THD for trx: 9733845076
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 66 [Warning] WSREP: no THD for trx: 9733845073
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 50 [Warning] WSREP: no THD for trx: 9733845077
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 44 [Warning] WSREP: no THD for trx: 9733845080
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 63 [Warning] WSREP: no THD for trx: 9733845069
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 41 [Warning] WSREP: no THD for trx: 9733845075
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 34 [Warning] WSREP: no THD for trx: 9733845065
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 67 [Warning] WSREP: no THD for trx: 9733845072
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 48 [Warning] WSREP: no THD for trx: 9733845043
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 64 [Warning] WSREP: no THD for trx: 9733845041
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 10 [Warning] WSREP: no THD for trx: 9733845056
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 28 [Warning] WSREP: no THD for trx: 9733845055
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 47 [Warning] WSREP: no THD for trx: 9733845047
 
Dec 02 22:30:59 systemd[1]: Started MariaDB 10.4.22 database server.
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 40 [Warning] WSREP: no THD for trx: 9733845070
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 39 [Warning] WSREP: no THD for trx: 9733845067
 
Dec 02 22:30:59 /etc/mysql/debian-start[2096564]: Upgrading MySQL tables if necessary.
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 62 [Warning] WSREP: no THD for trx: 9733845068
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 49 [Warning] WSREP: no THD for trx: 9733845052
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 31 [Warning] WSREP: no THD for trx: 9733845035
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 17 [Warning] WSREP: no THD for trx: 9733845060
 
Dec 02 22:30:59 /etc/mysql/debian-start[2096567]: /usr/bin/mysql_upgrade: the '--basedir' option is always ignored
 
Dec 02 22:30:59 /etc/mysql/debian-start[2096567]: Looking for 'mysql' as: /usr/bin/mysql
 
Dec 02 22:30:59 /etc/mysql/debian-start[2096567]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
 
Dec 02 22:30:59 /etc/mysql/debian-start[2096567]: This installation of MariaDB is already upgraded to 10.4.22-MariaDB, use --force if you still need to run mysql_upgrade
 
Dec 02 22:30:59 /etc/mysql/debian-start[2096575]: Checking for insecure root accounts.
 
Dec 02 22:30:59 mysqld[2096578]: ERROR 1047 (08S01) at line 1: WSREP has not yet prepared node for application use
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 29 [Warning] WSREP: no THD for trx: 9733845045
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 30 [Warning] WSREP: no THD for trx: 9733845049
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] InnoDB: Rolled back recovered transaction 9733845066
 
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] InnoDB: Rolled back recovered transaction 9733845050
 
Dec 02 22:31:00 mysqld[2095962]: 2021-12-02 22:31:00 0 [Note] InnoDB: Rolled back recovered transaction 9733845078
 
Dec 02 22:31:00 mysqld[2095962]: 2021-12-02 22:31:00 0 [Note] InnoDB: Rolled back recovered transaction 9733845046
 
Dec 02 22:31:00 mysqld[2095962]: 2021-12-02 22:31:00 0 [Note] InnoDB: Rolled back recovered transaction 9733845062
 
Dec 02 22:31:00 mysqld[2095962]: 2021-12-02 22:31:00 0 [Note] InnoDB: Rolled back recovered transaction 9733845070
 
Dec 02 22:31:00 mysqld[2095962]: 2021-12-02 22:31:00 0 [Note] InnoDB: Rolled back recovered transaction 9733845038
 
Dec 02 22:31:00 mysqld[2095962]: 2021-12-02 22:31:00 0 [Note] InnoDB: Rolled back recovered transaction 9733845054
 
Dec 02 22:31:00 mysqld[2095962]: 2021-12-02 22:31:00 0 [Note] InnoDB: Rolled back recovered transaction 9733845040
 
Dec 02 22:31:00 mysqld[2095962]: 2021-12-02 22:31:00 0 [Note] InnoDB: Rolled back recovered transaction 9733845072
 
Dec 02 22:31:00 mysqld[2095962]: 2021-12-02 22:31:00 0 [Note] InnoDB: Rolled back recovered transaction 9733845056
 
Dec 02 22:31:00 mysqld[2095962]: 2021-12-02 22:31:00 0 [Note] InnoDB: Rolled back recovered transaction 9733845080
 
Dec 02 22:31:00 mysqld[2095962]: 2021-12-02 22:31:00 0 [Note] InnoDB: Rolled back recovered transaction 9733845048
 
Dec 02 22:31:00 mysqld[2095962]: 2021-12-02 22:31:00 0 [Note] InnoDB: Rolled back recovered transaction 9733845064
 
Dec 02 22:31:00 mysqld[2095962]: 2021-12-02 22:31:00 0 [Note] InnoDB: Rolled back recovered transaction 9733845060
 
Dec 02 22:31:00 mysqld[2095962]: 2021-12-02 22:31:00 0 [Note] InnoDB: Rolled back recovered transaction 9733845076
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845044
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845068
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845036
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845052
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845035
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845067
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845051
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845043
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845075
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845059
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845079
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845047
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845063
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845055
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845071
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845039
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845073
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845041
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845057
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845049
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845065
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845033
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845053
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845037
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845069
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845061
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845045
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845077
 
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rollback of non-prepared transactions completed
 
Dec 02 22:31:01 mysqld[2095962]: mysqld: /home/buildbot/buildbot/build/mariadb-10.4.22/wsrep-lib/include/wsrep/client_state.hpp:668: int wsrep::client_state::bf_abort(wsrep::seqno): Assertion `mode_ == m_local || transaction_.is_streaming()' failed.
 
Dec 02 22:31:01 mysqld[2095962]: 211202 22:31:01 [ERROR] mysqld got signal 6 ;
 
Dec 02 22:31:01 mysqld[2095962]: This could be because you hit a bug. It is also possible that this binary
 
Dec 02 22:31:01 mysqld[2095962]: or one of the libraries it was linked against is corrupt, improperly built,
 
Dec 02 22:31:01 mysqld[2095962]: or misconfigured. This error can also be caused by malfunctioning hardware.
 
Dec 02 22:31:01 mysqld[2095962]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
Dec 02 22:31:01 mysqld[2095962]: We will try our best to scrape up some info that will hopefully help
 
Dec 02 22:31:01 mysqld[2095962]: diagnose the problem, but since we have already crashed,
 
Dec 02 22:31:01 mysqld[2095962]: something is definitely wrong and this may fail.
 
Dec 02 22:31:01 mysqld[2095962]: Server version: 10.4.22-MariaDB-1:10.4.22+maria~focal-log
 
Dec 02 22:31:01 mysqld[2095962]: key_buffer_size=134217728
 
Dec 02 22:31:01 mysqld[2095962]: read_buffer_size=131072
 
Dec 02 22:31:01 mysqld[2095962]: max_used_connections=1
 
Dec 02 22:31:01 mysqld[2095962]: max_threads=403
 
Dec 02 22:31:01 mysqld[2095962]: thread_count=71
 
Dec 02 22:31:01 mysqld[2095962]: It is possible that mysqld could use up to
 
Dec 02 22:31:01 mysqld[2095962]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6795376 K  bytes of memory
 
Dec 02 22:31:01 mysqld[2095962]: Hope that's ok; if not, decrease some variables in the equation.
 
Dec 02 22:31:01 mysqld[2095962]: Thread pointer: 0x7e01d8000c48
 
Dec 02 22:31:01 mysqld[2095962]: Attempting backtrace. You can use the following information to find out
 
Dec 02 22:31:01 mysqld[2095962]: where mysqld died. If you see no messages after this, something went
 
Dec 02 22:31:01 mysqld[2095962]: terribly wrong...
 
Dec 02 22:31:01 mysqld[2095962]: stack_bottom = 0x7f12b01dce08 thread_stack 0x49000
 
Dec 02 22:31:04 mysqld[2095962]: /usr/sbin/mysqld(my_print_stacktrace+0x32)[0x55c6369f7d12]
 
Dec 02 22:31:04 mysqld[2095962]: /usr/sbin/mysqld(handle_fatal_signal+0x55d)[0x55c6364a9a5d]
 
Dec 02 22:31:05 mysqld[2095962]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0)[0x7f338c51f3c0]
 
Dec 02 22:31:09 mysqld[2095962]: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x7f338c02518b]
 
Dec 02 22:31:09 mysqld[2095962]: /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b)[0x7f338c004859]
 
Dec 02 22:31:09 mysqld[2095962]: /lib/x86_64-linux-gnu/libc.so.6(+0x25729)[0x7f338c004729]
 
Dec 02 22:31:09 mysqld[2095962]: /lib/x86_64-linux-gnu/libc.so.6(+0x36f36)[0x7f338c015f36]
 
Dec 02 22:31:10 mysqld[2095962]: /usr/sbin/mysqld(_Z14wsrep_bf_abortP3THDS0_+0x6e9)[0x55c6364247f9]
 
Dec 02 22:31:11 mysqld[2095962]: /usr/sbin/mysqld(wsrep_thd_bf_abort+0x1d)[0x55c63642c6bd]
 
Dec 02 22:31:11 mysqld[2095962]: /usr/sbin/mysqld(+0xaab1b6)[0x55c6366621b6]
 
Dec 02 22:31:11 mysqld[2095962]: /usr/sbin/mysqld(+0xb0cd84)[0x55c6366c3d84]
 
Dec 02 22:31:11 mysqld[2095962]: /usr/sbin/mysqld(+0xb12e62)[0x55c6366c9e62]
 
Dec 02 22:31:11 mysqld[2095962]: /usr/sbin/mysqld(+0xb9fdd1)[0x55c636756dd1]
 
Dec 02 22:31:11 mysqld[2095962]: /usr/sbin/mysqld(+0xac65b1)[0x55c63667d5b1]
 
Dec 02 22:31:11 mysqld[2095962]: /usr/sbin/mysqld(_ZN7handler10ha_rnd_posEPhS0_+0x50)[0x55c6364b0690]
 
Dec 02 22:31:11 mysqld[2095962]: /usr/sbin/mysqld(+0x818150)[0x55c6363cf150]
 
Dec 02 22:31:11 mysqld[2095962]: /usr/sbin/mysqld(_ZN14Rows_log_event8find_rowEP14rpl_group_info+0x451)[0x55c6365c5f91]
 
Dec 02 22:31:11 mysqld[2095962]: /usr/sbin/mysqld(_ZN21Delete_rows_log_event11do_exec_rowEP14rpl_group_info+0x92)[0x55c6365c6252]
 
Dec 02 22:31:11 mysqld[2095962]: /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x23f)[0x55c6365b975f]
 
Dec 02 22:31:11 mysqld[2095962]: /usr/sbin/mysqld(_Z18wsrep_apply_eventsP3THDP14Relay_log_infoPKvm+0x1b9)[0x55c636423979]
 
Dec 02 22:31:11 mysqld[2095962]: /usr/sbin/mysqld(_ZN21Wsrep_applier_service15apply_write_setERKN5wsrep7ws_metaERKNS0_12const_bufferERNS0_14mutable_bufferE+0xa3)[0x55c63640afc3]
 
Dec 02 22:31:11 mysqld[2095962]: /usr/sbin/mysqld(+0xeaf580)[0x55c636a66580]
 
Dec 02 22:31:11 mysqld[2095962]: /usr/sbin/mysqld(_ZN5wsrep12server_state8on_applyERNS_21high_priority_serviceERKNS_9ws_handleERKNS_7ws_metaERKNS_12const_bufferE+0xc1)[0x55c636a67441]
 
Dec 02 22:31:11 mysqld[2095962]: /usr/sbin/mysqld(+0xec187c)[0x55c636a7887c]
 
Dec 02 22:31:12 mysqld[2095962]: /usr/lib/libgalera_smm.so(+0x639eb)[0x7f338b2939eb]
 
Dec 02 22:31:13 mysqld[2095962]: /usr/lib/libgalera_smm.so(+0x7582e)[0x7f338b2a582e]
 
Dec 02 22:31:13 mysqld[2095962]: src/trx_handle.cpp:391(galera::TrxHandleSlave::apply(void*, wsrep_cb_status (*)(void*, wsrep_ws_handle const*, unsigned int, wsrep_buf const*, wsrep_trx_meta const*, bool*), wsrep_trx_meta const&, bool&))[0x7f338b2bcdc4]
 
Dec 02 22:31:13 mysqld[2095962]: src/replicator_smm.cpp:504(galera::ReplicatorSMM::apply_trx(void*, galera::TrxHandleSlave&))[0x7f338b2bd462]
 
Dec 02 22:31:14 mysqld[2095962]: src/replicator_str.cpp:1131(galera::ReplicatorSMM::process_IST_writeset(void*, boost::shared_ptr<galera::TrxHandleSlave> const&))[0x7f338b2a8ea3]
 
Dec 02 22:31:14 mysqld[2095962]: src/replicator_smm.cpp:396(galera::ReplicatorSMM::async_recv(void*))[0x7f338b27efe1]
 
Dec 02 22:31:15 mysqld[2095962]: /usr/sbin/mysqld(_ZN5wsrep18wsrep_provider_v2611run_applierEPNS_21high_priority_serviceE+0x12)[0x55c636a78e92]
 
Dec 02 22:31:15 mysqld[2095962]: /usr/sbin/mysqld(+0x86e7c3)[0x55c6364257c3]
 
Dec 02 22:31:16 mysqld[2095962]: /usr/sbin/mysqld(_Z15start_wsrep_THDPv+0x2ec)[0x55c636415f1c]
 
Dec 02 22:31:17 mysqld[2095962]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609)[0x7f338c513609]
 
Dec 02 22:31:20 mysqld[2095962]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x43)[0x7f338c101293]
 
Dec 02 22:31:20 mysqld[2095962]: Trying to get some variables.
 
Dec 02 22:31:20 mysqld[2095962]: Some pointers may be invalid and cause the dump to abort.
 
Dec 02 22:31:20 mysqld[2095962]: Query (0x7f15f1893f5b): delete from problem where r_clock<>0 and r_clock<1638393301
 
Dec 02 22:31:20 mysqld[2095962]: Connection ID (thread ID): 14
 
Dec 02 22:31:20 mysqld[2095962]: Status: NOT_KILLED
 
Dec 02 22:31:20 mysqld[2095962]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on
 
Dec 02 22:31:20 mysqld[2095962]: The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
 
Dec 02 22:31:20 mysqld[2095962]: information that should help you find out what is causing the crash.
 
Dec 02 22:31:20 mysqld[2095962]: Writing a core file...
 
Dec 02 22:31:20 mysqld[2095962]: Working directory at /var/lib/mysql
 
Dec 02 22:31:20 mysqld[2095962]: Resource Limits:
 
Dec 02 22:31:20 mysqld[2095962]: Limit                     Soft Limit           Hard Limit           Units
 
Dec 02 22:31:20 mysqld[2095962]: Max cpu time              unlimited            unlimited            seconds
 
Dec 02 22:31:20 mysqld[2095962]: Max file size             unlimited            unlimited            bytes
 
Dec 02 22:31:20 mysqld[2095962]: Max data size             unlimited            unlimited            bytes
 
Dec 02 22:31:20 mysqld[2095962]: Max stack size            8388608              unlimited            bytes
 
Dec 02 22:31:20 mysqld[2095962]: Max core file size        0                    unlimited            bytes
 
Dec 02 22:31:20 mysqld[2095962]: Max resident set          unlimited            unlimited            bytes
 
Dec 02 22:31:20 mysqld[2095962]: Max processes             5931800              5931800              processes
 
Dec 02 22:31:20 mysqld[2095962]: Max open files            65536                65536                files
 
Dec 02 22:31:20 mysqld[2095962]: Max locked memory         65536                65536                bytes
 
Dec 02 22:31:20 mysqld[2095962]: Max address space         unlimited            unlimited            bytes
 
Dec 02 22:31:20 mysqld[2095962]: Max file locks            unlimited            unlimited            locks
 
Dec 02 22:31:20 mysqld[2095962]: Max pending signals       5931800              5931800              signals
 
Dec 02 22:31:20 mysqld[2095962]: Max msgqueue size         819200               819200               bytes
 
Dec 02 22:31:20 mysqld[2095962]: Max nice priority         0                    0
 
Dec 02 22:31:20 mysqld[2095962]: Max realtime priority     0                    0
 
Dec 02 22:31:20 mysqld[2095962]: Max realtime timeout      unlimited            unlimited            us
 
Dec 02 22:31:20 mysqld[2095962]: Core pattern: /var/crash/%e.%p
 
Dec 02 22:31:28 systemd[1]: mariadb.service: Main process exited, code=killed, status=6/ABRT
 
Dec 02 22:31:28 systemd[1]: mariadb.service: Failed with result 'signal'.

Comment by Mark Reibert [ 2021-12-06 ]

And again this happened on 5 Dec. I run a three-node Galera cluster with all connections routed to one node (the primary). Early on 5 Dec both "backup" nodes at exactly the same time (to within the one-second resolution in the logs) aborted with:

Dec 05 01:15:15 mysqld[687955]: mysqld: /home/buildbot/buildbot/build/mariadb-10.4.22/wsrep-lib/include/wsrep/client_state.hpp:668: int wsrep::client_state::bf_abort(wsrep::seqno): Assertion `mode_ == m_local || transaction_.is_streaming()' failed.
 
Dec 05 01:15:15 mysqld[687955]: 211205  1:15:15 [ERROR] mysqld got signal 6 ;

Of course, with both backup nodes failing the primary node loses quorum, leaving me dead in the water with no functioning database.

I cannot stress enough how critical an issue this is. My Galera cluster is a mission-critical, production system and for the past month+ (since upgrading to 10.4.19, then 10.4.21, and now with 10.4.22) the cluster is dramatically unstable. Every day is another opportunity for yet another crash.

Whatever I can do to help debug I will do. Do you want logs? I will provide them. Do you need extra or different logging? I will reconfigure my nodes to provide that. Just let me know! Thanks.

Comment by Jan Lindström (Inactive) [ 2021-12-07 ]

mreibert Do you get core dump? If you get can you open it on debugger by using MariaDB debug symbols package and run command thread apply all bt and provide output from that. Do you know what kind of query causes crash? If you do please provide affected table structure. Do you have tables with foreign keys using charset utf8 ?

Comment by Mark Reibert [ 2021-12-07 ]

jplindst — Unfortunately cores are disabled on my DB nodes; I will change that so next time it crashes I get the core.

Regarding the offending query, that too I do not have. This database backs a large and very busy Zabbix installation and the Zabbix servers are responsible for the queries. Is there some way to log and correlate those queries to the crash?

Regarding the charset, the Zabbix database on this cluster is latin1.

Comment by Mark Reibert [ 2021-12-07 ]

jplindst — I should mention there is also a database backing Grafana on this cluster, and that database is utf8mb4. However, whereas there are indexes on varchar columns I do not see a single FK constraint in this database.

Comment by Mark Reibert [ 2021-12-09 ]

jplindst — I received another SIGABRT and captured the core dump. The stack trace you requested is attached in reibert_mysqld-abort-stacktrace.txt and the journal log is attached in reibert_mariadb.service.log.

Here is the scenario that lead to the assertion failure. The node (one of three in the cluster) was being brought online via a SST using mariabackup. The SST transferred normally (you'll see that took 6.5 hours; yes that is normal for this 9.5TB database). After the transfer the mariabackup "prepare" (apply) and "move" phases completed normally. (These took nearly an hour; again that is normal for this large database.) The server then comes online and begins the delta IST, which itself can be large since nearly 8 hours have passed since the SST began. Not long into that the mode_ == m_local || transaction_.is_streaming() assertion fails with the resulting SIGABRT.

Interestingly, near the end of the log we see a delete from events ... query; seeing a query in the log is actually not common. Could this be the query that triggered the assertion failure? That seems at least plausible since the stack trace indicates the assertion failure occurred when deleting a row in a writeset.

Please do not hesitate to ask if there is any other information I can provide.

Comment by Jan Lindström (Inactive) [ 2021-12-10 ]

Thank you. Your stack trace contains row_upd_check_references_constraint so there is foreign keys also in this case.

Comment by Mark Reibert [ 2021-12-10 ]

There are two databases on this cluster, one for Zabbix and one for Grafana. The Zabbix DB does have FKs, but that DB is entirely latin1. The Grafana DB is utf8mb4, but has no FKs.

The load on this cluster is dominated by the Zabbix DB, both in size and query frequency/volume. I suspect the problem is related to that, so I am not surprised there is a FK reference in the stack trace.

Comment by Jan Lindström (Inactive) [ 2021-12-13 ]

mreibert Can you try with provider configuration: cert.optimistic_pa=false ?

Comment by Mark Reibert [ 2021-12-13 ]

jplindst — Yes, I will try that.

In the mean time, I had another SIGABRT. The stack trace is attached in reibert_mysqld-abort-stacktrace-2.txt and the journal log (from server startup to stop) is attached in reibert_mariadb.service-2.log.

The scenario for this crash is a little different. This time I had only two nodes up and running. (The node that crashed several days ago is still down.) Nothing "special" was going on with the nodes; they had both been up for several days. So this was not an IST-triggered event but rather just a "normal" writeset application. The stack trace indicates there was a record lock conflict on a delete row. In that vein, I should note I have wsrep_slave_threads = 64 on these nodes.

Comment by Mark Reibert [ 2021-12-13 ]

On a potentially related topic, should I set wsrep_certification_rules = optimized ? From the documentation it seems that would have an effect only for a cluster that has multiple nodes taking writes (which is not my case; all writes are directed to one node), however Percona sets that by default on their distribution since it is "safer". (Percona also sets cert.optimistic_pa = no by default.)

Comment by Jan Lindström (Inactive) [ 2021-12-14 ]

mreibert I would not set wsrep_certification_rules = optimized in your case.

Comment by Mark Reibert [ 2021-12-15 ]

jplindst — thanks.

In other news, I now have cert.optimistic_pa = no on all (three) cluster nodes. With that setting I was able to SST the two down nodes back into the cluster with no errors or issues. Whether or not that is an actual fix, however, only time will tell. (Recall from my last core dump I had two nodes up for three days before one experienced the assertion failure "out of the blue".)

Keeping my fingers crossed …

Comment by Seppo Jaakola [ 2021-12-20 ]

mreibert your first stacktrace attached (reibert_mysqld-abort-stacktrace.txt) has one replication applier thread (Thread 73) in the middle of processing of Foreign key constraint's cascade operation (row_ins_foreign_check_on_constraint). This suggest that this crash relates to issue MDEV-26803, where the reason for the crashes was that cascading executions did not reliably record all rows manipulated through cascade operation in the replication write set. And this has resulted in unsafe parallel applying in replica nodes.
The second stacktrace has also ongoing foreign key checking, but in different stages.

Comment by Mark Reibert [ 2021-12-23 ]

seppo — Thank you for the update and reference to MDEV-26803. I see in that issue setting cert.optimistic_pa = no is a suggested workaround, as has been discussed in comments here. With the fix for MDEV-26803 in 10.4.23, would it be considered "safe" to return to cert.optimistic_pa = yes? To ask the question a little differently, is there a downside to staying with cert.optimistic_pa = no? After 1–2 months of very frequent crashes with 10.4.19–22, I have been rock solid for the past week after setting cert.optimistic_pa = no. So i am (I think understandably) hesitant to turn that back on … ever.

Comment by Seppo Jaakola [ 2021-12-23 ]

mreibert "cert.optimistic_pa = no" configuration is a viable workaround for this and also as permanent configuration, if replication performance remains in acceptable level.
The "cert.optimistic_pa = no" configuration has global effect on all transactions, and will limit the concurrency of replication applying in replica nodes. Whereas, the fix for MDEV-26803, will monitor the transaction's foreign key constraint processing in the first node, and only when cascading delete operation really takes place, then that specific transaction only is marked as not safe for parallel applying.

Your stack trace contains at least one ongoing FK cascade delete operation, and this makes it plausible candidate for MDEV-26803 scenario.

Comment by Mark Reibert [ 2022-01-03 ]

seppo — thank you for the feedback.

At present my replication performance is holding fine with cert.optimistic_pa = no and I haven't experienced a single SIGABRT/SIGSEGV failure since making that change. So I will leave that disabled in my configuration. But if I do at some point start seeing replication fall behind I will consider re-enabling cert.optimistic_pa (after upgrading to a version that contains MDEV-26803, of course).

Comment by COUNOTTE CEDRIC [ 2023-05-02 ]

@Mark Reibert, we are experiencing the same issue from time to time and would like to set cert.optimistic_pa = no, however I'm not sure how to proceed safely applying this as a permanent configuration ?

Do I need to apply the setting on all nodes or can one node be set for testing?

Is adding such a line in /etc/mysql/mariadb.conf.d/50-server.cnf ok:

wsrep_provider_options = "cert.optimistic_pa = no"

Or should I copy/paste all existing values from an already running node, which looks like below, with a lot of options set:

wsrep_provider_options  base_dir = /var/lib/mysql/; base_host = 192.168.0.6; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.causal_keepalive_period = PT1S; evs.debug_log_mask = 0x1; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.info_log_mask = 0; evs.install_timeout = PT7.5S; evs.join_retrans_period = PT1S; evs.keepalive_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.use_aggregate = true; evs.user_send_window = 2; evs.version = 1; evs.view_forget_timeout = P1D; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.keep_plaintext_size = 128M; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.fc_single_primary = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.listen_addr = tcp://0.0.0.0:4567; gmcast.mcast_addr = ; gmcast.mcast_ttl = 1; gmcast.peer_timeout = PT3S; gmcast.segment = 0; gmcast.time_wait = PT5S; gmcast.version = 0; ist.recv_addr = 192.168.0.6; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.linger = PT20S; pc.npvo = false; pc.recovery = true; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = PT30S; pc.weight = 1; protonet.version = 0; repl.causal_read_timeout = PT30S; repl.commit_order = 3; repl.key_format = FLAT8; repl.max_ws_size = 2147483647; repl.proto_max = 10; socket.checksum = 2; socket.recv_buf_size = auto; socket.send_buf_size = auto;

Thanks for any tips.

Generated at Thu Feb 08 09:44:15 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.