Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Fixed
-
10.4.20, 10.4.21
-
Centos8
Description
Hi, Despite have add Primary Key to all tables, we still encounter the `Assertion `mode_ == m_local || transaction_.is_streaming()' failed.` Galera failure.
We operate 3 Mariadb Server 10.4.20 nodes (galera-4-26.4.6-1.el8.x86_64). Our aplications reach cluster using maxscale 2.2.9 configured with a read write splite router.
It seems to be related to a specfic request, ans some times both slaves fails and produce a core:
mysqld:
|
/home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.4.20/wsrep-lib/include/wsrep/client_state.hpp:668: int wsrep::client_state::bf_abort(wsrep::seqno): Assertion `mode_ == m_local || transaction_.is_streaming()' failed. |
210802 11:55:00 [ERROR] mysqld got signal 6 ; |
Please found in attachment:
- the Master general query log rd-devops-sql2.log
- A Slave core
- A Slave mysql error log
Attachments
Issue Links
- relates to
-
-
- Closed
-
Activity
Hello, sorry for jumping on this ticket. I have recently opened an Enterprise ticket, in which we got this exact same error having recently moved to the 10.4 series.
This was whilst running a stored procedure, that is being used to clear down a few tables. Up until now, I haven't been able to reproduce - however I have just been able to. I am now going to see if I can reproduce again, as I changed 2 variables since my last attempt
- using MySQL Workbench from a remote machine
- Using a different user, with less privileges - due to it being a remote machine.
Let me know if you want me to check anything in particular out.
Something like this :
MariaDB [INFORMATION_SCHEMA]> select TABLE_SCHEMA,TABLE_NAME,TABLE_COLLATION from TABLES where TABLE_NAME = 'CLIENT' \G |
*************************** 1. row *************************** |
TABLE_SCHEMA: keycloak
|
TABLE_NAME: CLIENT
|
TABLE_COLLATION: utf8_general_ci
|
1 row in set (0.001 sec) |
|
|
MariaDB [INFORMATION_SCHEMA]> SELECT CONSTRAINT_SCHEMA, TABLE_NAME, CONSTRAINT_NAME, REFERENCED_TABLE_NAME, REFERENCED_COLUMN_NAME FROM information_schema.key_column_usage where CONSTRAINT_NAME = 'FK_P56CTINXXB9GSK57FO49F9TAC' \G |
*************************** 1. row *************************** |
CONSTRAINT_SCHEMA: keycloak
|
TABLE_NAME: CLIENT
|
CONSTRAINT_NAME: FK_P56CTINXXB9GSK57FO49F9TAC
|
REFERENCED_TABLE_NAME: REALM
|
REFERENCED_COLUMN_NAME: ID
|
1 row in set (0.001 sec) |
|
|
MariaDB [INFORMATION_SCHEMA]> SHOW COLUMNS FROM CLIENT FROM keycloak where Field = 'ID' \G |
*************************** 1. row *************************** |
Field: ID
|
Type: varchar(36) |
Null: NO
|
Key: PRI
|
Default: NULL
|
Extra:
|
1 row in set (0.001 sec) |
After testing a recent build, I have reported my issue as a new ticket MDEV-26803
For what it is worth, on a large (9.5TB) Zabbix database I was experiencing this problem in epidemic proportions with MariaDB 10.4.19 and 10.4.21, but have not observed a single incident since upgrading to 10.4.22.
It seems I jinxed 10.4.22. Not more than an hour or two after I typed that previous comment claiming "all is fine" I had a node SIGABRT with 10.4.22. Doh!
Here is the full log from the crash:
Dec 02 21:15:02 mysqld[1566419]: mysqld: /home/buildbot/buildbot/build/mariadb-10.4.22/wsrep-lib/include/wsrep/client_state.hpp:668: int wsrep::client_state::bf_abort(wsrep::seqno): Assertion `mode_ == m_local || transaction_.is_streaming()' failed.
|
Dec 02 21:15:02 mysqld[1566419]: 211202 21:15:02 [ERROR] mysqld got signal 6 ;
|
Dec 02 21:15:02 mysqld[1566419]: This could be because you hit a bug. It is also possible that this binary
|
Dec 02 21:15:02 mysqld[1566419]: or one of the libraries it was linked against is corrupt, improperly built,
|
Dec 02 21:15:02 mysqld[1566419]: or misconfigured. This error can also be caused by malfunctioning hardware.
|
Dec 02 21:15:02 mysqld[1566419]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
Dec 02 21:15:02 mysqld[1566419]: We will try our best to scrape up some info that will hopefully help
|
Dec 02 21:15:02 mysqld[1566419]: diagnose the problem, but since we have already crashed,
|
Dec 02 21:15:02 mysqld[1566419]: something is definitely wrong and this may fail.
|
Dec 02 21:15:02 mysqld[1566419]: Server version: 10.4.22-MariaDB-1:10.4.22+maria~focal-log
|
Dec 02 21:15:02 mysqld[1566419]: key_buffer_size=134217728
|
Dec 02 21:15:02 mysqld[1566419]: read_buffer_size=131072
|
Dec 02 21:15:02 mysqld[1566419]: max_used_connections=3
|
Dec 02 21:15:02 mysqld[1566419]: max_threads=403
|
Dec 02 21:15:02 mysqld[1566419]: thread_count=73
|
Dec 02 21:15:02 mysqld[1566419]: It is possible that mysqld could use up to
|
Dec 02 21:15:02 mysqld[1566419]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6795376 K bytes of memory
|
Dec 02 21:15:02 mysqld[1566419]: Hope that's ok; if not, decrease some variables in the equation.
|
Dec 02 21:15:02 mysqld[1566419]: Thread pointer: 0x7e020c000c48
|
Dec 02 21:15:02 mysqld[1566419]: Attempting backtrace. You can use the following information to find out
|
Dec 02 21:15:02 mysqld[1566419]: where mysqld died. If you see no messages after this, something went
|
Dec 02 21:15:02 mysqld[1566419]: terribly wrong...
|
Dec 02 21:15:02 mysqld[1566419]: stack_bottom = 0x7e031c473e08 thread_stack 0x49000
|
Dec 02 21:15:09 mysqld[1566419]: 2021-12-02 21:15:09 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection to peer 1e77c150-8d79 with addr tcp://192.168.77.160:4567 timed out, no messages seen in PT3S, socket stats: rtt: 1257 rttvar: 1806 rto: 204000 lost: 0 last_data_recv: 3252 cwnd: 10 last_queued_since: 25298782 last_delivered_since: 6616014371 send_queue_length: 0 send_queue_bytes: 0 segment: 0 messages: 0
|
Dec 02 21:15:09 mysqld[1566419]: 2021-12-02 21:15:09 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://192.168.77.160:4567
|
Dec 02 21:15:09 mysqld[1566419]: 2021-12-02 21:15:09 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT6.87437S), skipping check
|
Dec 02 21:15:09 mysqld[1566419]: 2021-12-02 21:15:09 0 [Note] WSREP: evs::proto(2f8c7fd2-91e6, OPERATIONAL, view_id(REG,1e77c150-8d79,104)) suspecting node: 1e77c150-8d79
|
Dec 02 21:15:09 mysqld[1566419]: 2021-12-02 21:15:09 0 [Note] WSREP: evs::proto(2f8c7fd2-91e6, OPERATIONAL, view_id(REG,1e77c150-8d79,104)) suspected node without join message, declaring inactive
|
Dec 02 21:15:10 mysqld[1566419]: 2021-12-02 21:15:10 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') reconnecting to 1e77c150-8d79 (tcp://192.168.77.160:4567), attempt 0
|
Dec 02 21:15:10 mysqld[1566419]: 2021-12-02 21:15:10 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 1e77c150-8d79 tcp://192.168.77.160:4567
|
Dec 02 21:15:10 mysqld[1566419]: 2021-12-02 21:15:10 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') reconnecting to 58fa90ba-871c (tcp://192.168.77.162:4567), attempt 0
|
Dec 02 21:15:10 mysqld[1566419]: 2021-12-02 21:15:10 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 58fa90ba-871c tcp://192.168.77.162:4567
|
Dec 02 21:15:11 mysqld[1566419]: 2021-12-02 21:15:11 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') reconnecting to 1e77c150-8d79 (tcp://192.168.77.160:4567), attempt 0
|
Dec 02 21:15:11 mysqld[1566419]: 2021-12-02 21:15:11 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 1e77c150-8d79 tcp://192.168.77.160:4567
|
Dec 02 21:15:11 mysqld[1566419]: 2021-12-02 21:15:11 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') reconnecting to 58fa90ba-871c (tcp://192.168.77.162:4567), attempt 0
|
Dec 02 21:15:11 mysqld[1566419]: 2021-12-02 21:15:11 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 58fa90ba-871c tcp://192.168.77.162:4567
|
Dec 02 21:15:14 mysqld[1566419]: 2021-12-02 21:15:14 0 [Note] WSREP: evs::proto(2f8c7fd2-91e6, GATHER, view_id(REG,1e77c150-8d79,104)) suspecting node: 58fa90ba-871c
|
Dec 02 21:15:14 mysqld[1566419]: 2021-12-02 21:15:14 0 [Note] WSREP: evs::proto(2f8c7fd2-91e6, GATHER, view_id(REG,1e77c150-8d79,104)) suspected node without join message, declaring inactive
|
Dec 02 21:15:14 mysqld[1566419]: 2021-12-02 21:15:14 0 [Note] WSREP: view(view_id(NON_PRIM,1e77c150-8d79,104) memb {
|
Dec 02 21:15:14 mysqld[1566419]: 2f8c7fd2-91e6,0
|
Dec 02 21:15:14 mysqld[1566419]: } joined {
|
Dec 02 21:15:14 mysqld[1566419]: } left {
|
Dec 02 21:15:14 mysqld[1566419]: } partitioned {
|
Dec 02 21:15:14 mysqld[1566419]: 1e77c150-8d79,0
|
Dec 02 21:15:14 mysqld[1566419]: 58fa90ba-871c,0
|
Dec 02 21:15:14 mysqld[1566419]: })
|
Dec 02 21:15:14 mysqld[1566419]: 2021-12-02 21:15:14 0 [Note] WSREP: view(view_id(NON_PRIM,2f8c7fd2-91e6,105) memb {
|
Dec 02 21:15:14 mysqld[1566419]: 2f8c7fd2-91e6,0
|
Dec 02 21:15:14 mysqld[1566419]: } joined {
|
Dec 02 21:15:14 mysqld[1566419]: } left {
|
Dec 02 21:15:14 mysqld[1566419]: } partitioned {
|
Dec 02 21:15:14 mysqld[1566419]: 1e77c150-8d79,0
|
Dec 02 21:15:14 mysqld[1566419]: 58fa90ba-871c,0
|
Dec 02 21:15:14 mysqld[1566419]: })
|
Dec 02 21:15:14 mysqld[1566419]: 2021-12-02 21:15:14 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
|
Dec 02 21:15:14 mysqld[1566419]: 2021-12-02 21:15:14 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
|
Dec 02 21:15:14 mysqld[1566419]: 2021-12-02 21:15:14 0 [Note] WSREP: Received NON-PRIMARY.
|
Dec 02 21:15:14 mysqld[1566419]: 2021-12-02 21:15:14 0 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 4857087606)
|
Dec 02 21:15:14 mysqld[1566419]: 2021-12-02 21:15:14 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
|
Dec 02 21:15:14 mysqld[1566419]: 2021-12-02 21:15:14 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
|
Dec 02 21:15:14 mysqld[1566419]: 2021-12-02 21:15:14 0 [Note] WSREP: Received NON-PRIMARY.
|
Dec 02 21:15:15 mysqld[1566419]: 2021-12-02 21:15:15 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') turning message relay requesting off
|
Dec 02 21:15:15 mysqld[1566419]: 2021-12-02 21:15:15 0 [Note] WSREP: declaring 1e77c150-8d79 at tcp://192.168.77.160:4567 stable
|
Dec 02 21:15:15 mysqld[1566419]: 2021-12-02 21:15:15 0 [Note] WSREP: declaring 58fa90ba-871c at tcp://192.168.77.162:4567 stable
|
Dec 02 21:15:15 mysqld[1566419]: 2021-12-02 21:15:15 0 [Note] WSREP: Node 1e77c150-8d79 state prim
|
Dec 02 21:15:15 mysqld[1566419]: 2021-12-02 21:15:15 0 [Note] WSREP: view(view_id(PRIM,1e77c150-8d79,106) memb {
|
Dec 02 21:15:15 mysqld[1566419]: 1e77c150-8d79,0
|
Dec 02 21:15:15 mysqld[1566419]: 2f8c7fd2-91e6,0
|
Dec 02 21:15:15 mysqld[1566419]: 58fa90ba-871c,0
|
Dec 02 21:15:15 mysqld[1566419]: } joined {
|
Dec 02 21:15:15 mysqld[1566419]: } left {
|
Dec 02 21:15:15 mysqld[1566419]: } partitioned {
|
Dec 02 21:15:15 mysqld[1566419]: })
|
Dec 02 21:15:15 mysqld[1566419]: 2021-12-02 21:15:15 0 [Note] WSREP: save pc into disk
|
Dec 02 21:15:15 mysqld[1566419]: 2021-12-02 21:15:15 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 3
|
Dec 02 21:15:15 mysqld[1566419]: 2021-12-02 21:15:15 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
|
Dec 02 21:15:15 mysqld[1566419]: 2021-12-02 21:15:15 0 [Note] WSREP: STATE EXCHANGE: sent state msg: f1ed7f1a-53b4-11ec-a50b-9e152798ae9a
|
Dec 02 21:15:15 mysqld[1566419]: 2021-12-02 21:15:15 0 [Note] WSREP: STATE EXCHANGE: got state msg: f1ed7f1a-53b4-11ec-a50b-9e152798ae9a from 0 (zabbix-db04)
|
Dec 02 21:15:15 mysqld[1566419]: 2021-12-02 21:15:15 0 [Note] WSREP: STATE EXCHANGE: got state msg: f1ed7f1a-53b4-11ec-a50b-9e152798ae9a from 1 (zabbix-db05)
|
Dec 02 21:15:15 mysqld[1566419]: 2021-12-02 21:15:15 0 [Note] WSREP: STATE EXCHANGE: got state msg: f1ed7f1a-53b4-11ec-a50b-9e152798ae9a from 2 (zabbix-db06)
|
Dec 02 21:15:15 mysqld[1566419]: 2021-12-02 21:15:15 0 [Note] WSREP: Quorum results:
|
Dec 02 21:15:15 mysqld[1566419]: version = 6,
|
Dec 02 21:15:15 mysqld[1566419]: component = PRIMARY,
|
Dec 02 21:15:15 mysqld[1566419]: conf_id = 39,
|
Dec 02 21:15:15 mysqld[1566419]: members = 2/3 (joined/total),
|
Dec 02 21:15:15 mysqld[1566419]: act_id = 4857088711,
|
Dec 02 21:15:15 mysqld[1566419]: last_appl. = 4857088690,
|
Dec 02 21:15:15 mysqld[1566419]: protocols = 2/10/4 (gcs/repl/appl),
|
Dec 02 21:15:15 mysqld[1566419]: vote policy= 0,
|
Dec 02 21:15:15 mysqld[1566419]: group UUID = 810a5d0c-231c-11e9-b756-dbb7899d7dd0
|
Dec 02 21:15:15 mysqld[1566419]: 2021-12-02 21:15:15 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
|
Dec 02 21:15:15 mysqld[1566419]: 2021-12-02 21:15:15 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 4857088712)
|
Dec 02 21:15:18 mysqld[1566419]: /usr/sbin/mysqld(my_print_stacktrace+0x32)[0x564297590d12]
|
Dec 02 21:15:18 mysqld[1566419]: /usr/sbin/mysqld(handle_fatal_signal+0x55d)[0x564297042a5d]
|
Dec 02 21:15:23 mysqld[1566419]: 2021-12-02 21:15:23 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection to peer 1e77c150-8d79 with addr tcp://192.168.77.160:4567 timed out, no messages seen in PT3S, socket stats: rtt: 2620 rttvar: 3448 rto: 204000 lost: 0 last_data_recv: 2696 cwnd: 10 last_queued_since: 559365 last_delivered_since: 5870988113 send_queue_length: 0 send_queue_bytes: 0 segment: 0 messages: 0
|
Dec 02 21:15:23 mysqld[1566419]: 2021-12-02 21:15:23 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://192.168.77.160:4567
|
Dec 02 21:15:23 mysqld[1566419]: 2021-12-02 21:15:23 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT5.93833S), skipping check
|
Dec 02 21:15:23 mysqld[1566419]: 2021-12-02 21:15:23 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 1e77c150-8d79 tcp://192.168.77.160:4567
|
Dec 02 21:15:23 mysqld[1566419]: 2021-12-02 21:15:23 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 58fa90ba-871c tcp://192.168.77.162:4567
|
Dec 02 21:15:24 mysqld[1566419]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0)[0x7f348779e3c0]
|
Dec 02 21:15:24 mysqld[1566419]: 2021-12-02 21:15:24 0 [Note] WSREP: evs::proto(2f8c7fd2-91e6, OPERATIONAL, view_id(REG,1e77c150-8d79,106)) suspecting node: 1e77c150-8d79
|
Dec 02 21:15:24 mysqld[1566419]: 2021-12-02 21:15:24 0 [Note] WSREP: evs::proto(2f8c7fd2-91e6, OPERATIONAL, view_id(REG,1e77c150-8d79,106)) suspected node without join message, declaring inactive
|
Dec 02 21:15:24 mysqld[1566419]: 2021-12-02 21:15:24 0 [Note] WSREP: view(view_id(NON_PRIM,1e77c150-8d79,106) memb {
|
Dec 02 21:15:24 mysqld[1566419]: 2f8c7fd2-91e6,0
|
Dec 02 21:15:24 mysqld[1566419]: } joined {
|
Dec 02 21:15:24 mysqld[1566419]: } left {
|
Dec 02 21:15:24 mysqld[1566419]: } partitioned {
|
Dec 02 21:15:24 mysqld[1566419]: 1e77c150-8d79,0
|
Dec 02 21:15:24 mysqld[1566419]: 58fa90ba-871c,0
|
Dec 02 21:15:24 mysqld[1566419]: })
|
Dec 02 21:15:24 mysqld[1566419]: 2021-12-02 21:15:24 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
|
Dec 02 21:15:24 mysqld[1566419]: 2021-12-02 21:15:24 0 [Note] WSREP: view(view_id(NON_PRIM,2f8c7fd2-91e6,107) memb {
|
Dec 02 21:15:24 mysqld[1566419]: 2f8c7fd2-91e6,0
|
Dec 02 21:15:24 mysqld[1566419]: } joined {
|
Dec 02 21:15:24 mysqld[1566419]: } left {
|
Dec 02 21:15:24 mysqld[1566419]: } partitioned {
|
Dec 02 21:15:24 mysqld[1566419]: 1e77c150-8d79,0
|
Dec 02 21:15:24 mysqld[1566419]: 58fa90ba-871c,0
|
Dec 02 21:15:24 mysqld[1566419]: })
|
Dec 02 21:15:24 mysqld[1566419]: 2021-12-02 21:15:24 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
|
Dec 02 21:15:24 mysqld[1566419]: 2021-12-02 21:15:24 0 [Note] WSREP: Received NON-PRIMARY.
|
Dec 02 21:15:24 mysqld[1566419]: 2021-12-02 21:15:24 0 [Note] WSREP: Shifting PRIMARY -> OPEN (TO: 4857089108)
|
Dec 02 21:15:24 mysqld[1566419]: 2021-12-02 21:15:24 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
|
Dec 02 21:15:24 mysqld[1566419]: 2021-12-02 21:15:24 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
|
Dec 02 21:15:24 mysqld[1566419]: 2021-12-02 21:15:24 0 [Note] WSREP: Received NON-PRIMARY.
|
Dec 02 21:15:25 mysqld[1566419]: 2021-12-02 21:15:25 0 [Note] WSREP: declaring 1e77c150-8d79 at tcp://192.168.77.160:4567 stable
|
Dec 02 21:15:25 mysqld[1566419]: 2021-12-02 21:15:25 0 [Note] WSREP: declaring 58fa90ba-871c at tcp://192.168.77.162:4567 stable
|
Dec 02 21:15:25 mysqld[1566419]: 2021-12-02 21:15:25 0 [Note] WSREP: Node 1e77c150-8d79 state prim
|
Dec 02 21:15:25 mysqld[1566419]: 2021-12-02 21:15:25 0 [Note] WSREP: view(view_id(PRIM,1e77c150-8d79,108) memb {
|
Dec 02 21:15:25 mysqld[1566419]: 1e77c150-8d79,0
|
Dec 02 21:15:25 mysqld[1566419]: 2f8c7fd2-91e6,0
|
Dec 02 21:15:25 mysqld[1566419]: 58fa90ba-871c,0
|
Dec 02 21:15:25 mysqld[1566419]: } joined {
|
Dec 02 21:15:25 mysqld[1566419]: } left {
|
Dec 02 21:15:25 mysqld[1566419]: } partitioned {
|
Dec 02 21:15:25 mysqld[1566419]: })
|
Dec 02 21:15:25 mysqld[1566419]: 2021-12-02 21:15:25 0 [Note] WSREP: save pc into disk
|
Dec 02 21:15:25 mysqld[1566419]: 2021-12-02 21:15:25 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 3
|
Dec 02 21:15:25 mysqld[1566419]: 2021-12-02 21:15:25 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
|
Dec 02 21:15:25 mysqld[1566419]: 2021-12-02 21:15:25 0 [Note] WSREP: STATE EXCHANGE: sent state msg: f79778e9-53b4-11ec-9167-0b066174fefa
|
Dec 02 21:15:25 mysqld[1566419]: 2021-12-02 21:15:25 0 [Note] WSREP: STATE EXCHANGE: got state msg: f79778e9-53b4-11ec-9167-0b066174fefa from 0 (zabbix-db04)
|
Dec 02 21:15:25 mysqld[1566419]: 2021-12-02 21:15:25 0 [Note] WSREP: STATE EXCHANGE: got state msg: f79778e9-53b4-11ec-9167-0b066174fefa from 1 (zabbix-db05)
|
Dec 02 21:15:25 mysqld[1566419]: 2021-12-02 21:15:25 0 [Note] WSREP: STATE EXCHANGE: got state msg: f79778e9-53b4-11ec-9167-0b066174fefa from 2 (zabbix-db06)
|
Dec 02 21:15:25 mysqld[1566419]: 2021-12-02 21:15:25 0 [Note] WSREP: Quorum results:
|
Dec 02 21:15:25 mysqld[1566419]: version = 6,
|
Dec 02 21:15:25 mysqld[1566419]: component = PRIMARY,
|
Dec 02 21:15:25 mysqld[1566419]: conf_id = 40,
|
Dec 02 21:15:25 mysqld[1566419]: members = 2/3 (joined/total),
|
Dec 02 21:15:25 mysqld[1566419]: act_id = 4857089126,
|
Dec 02 21:15:25 mysqld[1566419]: last_appl. = 4857088690,
|
Dec 02 21:15:25 mysqld[1566419]: protocols = 2/10/4 (gcs/repl/appl),
|
Dec 02 21:15:25 mysqld[1566419]: vote policy= 0,
|
Dec 02 21:15:25 mysqld[1566419]: group UUID = 810a5d0c-231c-11e9-b756-dbb7899d7dd0
|
Dec 02 21:15:25 mysqld[1566419]: 2021-12-02 21:15:25 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
|
Dec 02 21:15:25 mysqld[1566419]: 2021-12-02 21:15:25 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 4857089127)
|
Dec 02 21:15:27 mysqld[1566419]: 2021-12-02 21:15:27 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') turning message relay requesting off
|
Dec 02 21:15:38 mysqld[1566419]: 2021-12-02 21:15:38 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection to peer 1e77c150-8d79 with addr tcp://192.168.77.160:4567 timed out, no messages seen in PT3S, socket stats: rtt: 1508 rttvar: 2174 rto: 204000 lost: 0 last_data_recv: 3732 cwnd: 10 last_queued_since: 412691 last_delivered_since: 6254106919 send_queue_length: 1 send_queue_bytes: 80 segment: 0 messages: 1
|
Dec 02 21:15:38 mysqld[1566419]: 2021-12-02 21:15:38 0 [Note] WSREP: Deferred close timer started for socket with remote endpoint: tcp://192.168.77.160:51202
|
Dec 02 21:15:38 mysqld[1566419]: 2021-12-02 21:15:38 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://192.168.77.160:4567
|
Dec 02 21:15:38 mysqld[1566419]: 2021-12-02 21:15:38 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT6.52657S), skipping check
|
Dec 02 21:15:38 mysqld[1566419]: 2021-12-02 21:15:38 0 [Note] WSREP: Deferred close timer handle_wait Operation aborted. for 0x7f13a003fb10
|
Dec 02 21:15:38 mysqld[1566419]: 2021-12-02 21:15:38 0 [Note] WSREP: Deferred close timer destruct
|
Dec 02 21:15:39 mysqld[1566419]: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x7f34872a418b]
|
Dec 02 21:15:39 mysqld[1566419]: 2021-12-02 21:15:39 0 [Note] WSREP: evs::proto(2f8c7fd2-91e6, OPERATIONAL, view_id(REG,1e77c150-8d79,108)) suspecting node: 1e77c150-8d79
|
Dec 02 21:15:39 mysqld[1566419]: 2021-12-02 21:15:39 0 [Note] WSREP: evs::proto(2f8c7fd2-91e6, OPERATIONAL, view_id(REG,1e77c150-8d79,108)) suspected node without join message, declaring inactive
|
Dec 02 21:15:39 mysqld[1566419]: /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b)[0x7f3487283859]
|
Dec 02 21:15:39 mysqld[1566419]: 2021-12-02 21:15:39 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') reconnecting to 1e77c150-8d79 (tcp://192.168.77.160:4567), attempt 0
|
Dec 02 21:15:39 mysqld[1566419]: 2021-12-02 21:15:39 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 1e77c150-8d79 tcp://192.168.77.160:4567
|
Dec 02 21:15:40 mysqld[1566419]: 2021-12-02 21:15:40 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') reconnecting to 58fa90ba-871c (tcp://192.168.77.162:4567), attempt 0
|
Dec 02 21:15:40 mysqld[1566419]: /lib/x86_64-linux-gnu/libc.so.6(+0x25729)[0x7f3487283729]
|
Dec 02 21:15:40 mysqld[1566419]: 2021-12-02 21:15:40 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 58fa90ba-871c tcp://192.168.77.162:4567
|
Dec 02 21:15:40 mysqld[1566419]: /lib/x86_64-linux-gnu/libc.so.6(+0x36f36)[0x7f3487294f36]
|
Dec 02 21:15:41 mysqld[1566419]: 2021-12-02 21:15:41 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') reconnecting to 1e77c150-8d79 (tcp://192.168.77.160:4567), attempt 0
|
Dec 02 21:15:41 mysqld[1566419]: 2021-12-02 21:15:41 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 1e77c150-8d79 tcp://192.168.77.160:4567
|
Dec 02 21:15:41 mysqld[1566419]: 2021-12-02 21:15:41 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') reconnecting to 58fa90ba-871c (tcp://192.168.77.162:4567), attempt 0
|
Dec 02 21:15:41 mysqld[1566419]: 2021-12-02 21:15:41 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 58fa90ba-871c tcp://192.168.77.162:4567
|
Dec 02 21:15:41 mysqld[1566419]: 2021-12-02 21:15:41 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 1e77c150-8d79 tcp://192.168.77.160:4567
|
Dec 02 21:15:43 mysqld[1566419]: 2021-12-02 21:15:43 0 [Note] WSREP: evs::proto(2f8c7fd2-91e6, GATHER, view_id(REG,1e77c150-8d79,108)) suspecting node: 58fa90ba-871c
|
Dec 02 21:15:43 mysqld[1566419]: 2021-12-02 21:15:43 0 [Note] WSREP: evs::proto(2f8c7fd2-91e6, GATHER, view_id(REG,1e77c150-8d79,108)) suspected node without join message, declaring inactive
|
Dec 02 21:15:44 mysqld[1566419]: 2021-12-02 21:15:44 0 [Note] WSREP: view(view_id(NON_PRIM,1e77c150-8d79,108) memb {
|
Dec 02 21:15:44 mysqld[1566419]: 2f8c7fd2-91e6,0
|
Dec 02 21:15:44 mysqld[1566419]: } joined {
|
Dec 02 21:15:44 mysqld[1566419]: } left {
|
Dec 02 21:15:44 mysqld[1566419]: } partitioned {
|
Dec 02 21:15:44 mysqld[1566419]: 1e77c150-8d79,0
|
Dec 02 21:15:44 mysqld[1566419]: 58fa90ba-871c,0
|
Dec 02 21:15:44 mysqld[1566419]: })
|
Dec 02 21:15:44 mysqld[1566419]: 2021-12-02 21:15:44 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
|
Dec 02 21:15:44 mysqld[1566419]: 2021-12-02 21:15:44 0 [Note] WSREP: view(view_id(NON_PRIM,2f8c7fd2-91e6,109) memb {
|
Dec 02 21:15:44 mysqld[1566419]: 2f8c7fd2-91e6,0
|
Dec 02 21:15:44 mysqld[1566419]: } joined {
|
Dec 02 21:15:44 mysqld[1566419]: } left {
|
Dec 02 21:15:44 mysqld[1566419]: } partitioned {
|
Dec 02 21:15:44 mysqld[1566419]: 1e77c150-8d79,0
|
Dec 02 21:15:44 mysqld[1566419]: 58fa90ba-871c,0
|
Dec 02 21:15:44 mysqld[1566419]: })
|
Dec 02 21:15:44 mysqld[1566419]: 2021-12-02 21:15:44 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
|
Dec 02 21:15:44 mysqld[1566419]: 2021-12-02 21:15:44 0 [Note] WSREP: Received NON-PRIMARY.
|
Dec 02 21:15:44 mysqld[1566419]: 2021-12-02 21:15:44 0 [Note] WSREP: Shifting PRIMARY -> OPEN (TO: 4857090486)
|
Dec 02 21:15:44 mysqld[1566419]: 2021-12-02 21:15:44 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
|
Dec 02 21:15:44 mysqld[1566419]: 2021-12-02 21:15:44 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
|
Dec 02 21:15:44 mysqld[1566419]: 2021-12-02 21:15:44 0 [Note] WSREP: Received NON-PRIMARY.
|
Dec 02 21:15:45 mysqld[1566419]: 2021-12-02 21:15:45 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') turning message relay requesting off
|
Dec 02 21:15:46 mysqld[1566419]: 2021-12-02 21:15:46 0 [Note] WSREP: declaring 1e77c150-8d79 at tcp://192.168.77.160:4567 stable
|
Dec 02 21:15:46 mysqld[1566419]: 2021-12-02 21:15:46 0 [Note] WSREP: declaring 58fa90ba-871c at tcp://192.168.77.162:4567 stable
|
Dec 02 21:15:46 mysqld[1566419]: 2021-12-02 21:15:46 0 [Note] WSREP: Node 1e77c150-8d79 state prim
|
Dec 02 21:15:46 mysqld[1566419]: 2021-12-02 21:15:46 0 [Note] WSREP: view(view_id(PRIM,1e77c150-8d79,110) memb {
|
Dec 02 21:15:46 mysqld[1566419]: 1e77c150-8d79,0
|
Dec 02 21:15:46 mysqld[1566419]: 2f8c7fd2-91e6,0
|
Dec 02 21:15:46 mysqld[1566419]: 58fa90ba-871c,0
|
Dec 02 21:15:46 mysqld[1566419]: } joined {
|
Dec 02 21:15:46 mysqld[1566419]: } left {
|
Dec 02 21:15:46 mysqld[1566419]: } partitioned {
|
Dec 02 21:15:46 mysqld[1566419]: })
|
Dec 02 21:15:46 mysqld[1566419]: 2021-12-02 21:15:46 0 [Note] WSREP: save pc into disk
|
Dec 02 21:15:46 mysqld[1566419]: 2021-12-02 21:15:46 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 3
|
Dec 02 21:15:46 mysqld[1566419]: 2021-12-02 21:15:46 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
|
Dec 02 21:15:46 mysqld[1566419]: 2021-12-02 21:15:46 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 03d428c1-53b5-11ec-be9c-4bd7ca174788
|
Dec 02 21:15:46 mysqld[1566419]: 2021-12-02 21:15:46 0 [Note] WSREP: STATE EXCHANGE: got state msg: 03d428c1-53b5-11ec-be9c-4bd7ca174788 from 0 (zabbix-db04)
|
Dec 02 21:15:46 mysqld[1566419]: 2021-12-02 21:15:46 0 [Note] WSREP: STATE EXCHANGE: got state msg: 03d428c1-53b5-11ec-be9c-4bd7ca174788 from 1 (zabbix-db05)
|
Dec 02 21:15:46 mysqld[1566419]: 2021-12-02 21:15:46 0 [Note] WSREP: STATE EXCHANGE: got state msg: 03d428c1-53b5-11ec-be9c-4bd7ca174788 from 2 (zabbix-db06)
|
Dec 02 21:15:46 mysqld[1566419]: 2021-12-02 21:15:46 0 [Note] WSREP: Quorum results:
|
Dec 02 21:15:46 mysqld[1566419]: version = 6,
|
Dec 02 21:15:46 mysqld[1566419]: component = PRIMARY,
|
Dec 02 21:15:46 mysqld[1566419]: conf_id = 42,
|
Dec 02 21:15:46 mysqld[1566419]: members = 2/3 (joined/total),
|
Dec 02 21:15:46 mysqld[1566419]: act_id = 4857091473,
|
Dec 02 21:15:46 mysqld[1566419]: last_appl. = 4857091469,
|
Dec 02 21:15:46 mysqld[1566419]: protocols = 2/10/4 (gcs/repl/appl),
|
Dec 02 21:15:46 mysqld[1566419]: vote policy= 0,
|
Dec 02 21:15:46 mysqld[1566419]: group UUID = 810a5d0c-231c-11e9-b756-dbb7899d7dd0
|
Dec 02 21:15:46 mysqld[1566419]: 2021-12-02 21:15:46 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
|
Dec 02 21:15:46 mysqld[1566419]: 2021-12-02 21:15:46 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 4857091474)
|
Dec 02 21:15:52 mysqld[1566419]: 2021-12-02 21:15:52 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection to peer 1e77c150-8d79 with addr tcp://192.168.77.160:4567 timed out, no messages seen in PT3S, socket stats: rtt: 1209 rttvar: 1476 rto: 204000 lost: 0 last_data_recv: 2284 cwnd: 10 last_queued_since: 363245 last_delivered_since: 5600232142 send_queue_length: 6 send_queue_bytes: 480 segment: 0 messages: 6
|
Dec 02 21:15:52 mysqld[1566419]: 2021-12-02 21:15:52 0 [Note] WSREP: Deferred close timer started for socket with remote endpoint: tcp://192.168.77.160:51208
|
Dec 02 21:15:52 mysqld[1566419]: 2021-12-02 21:15:52 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://192.168.77.160:4567
|
Dec 02 21:15:52 mysqld[1566419]: 2021-12-02 21:15:52 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT5.99973S), skipping check
|
Dec 02 21:15:52 mysqld[1566419]: 2021-12-02 21:15:52 0 [Note] WSREP: Deferred close timer handle_wait Operation aborted. for 0x56429cac2270
|
Dec 02 21:15:52 mysqld[1566419]: 2021-12-02 21:15:52 0 [Note] WSREP: Deferred close timer destruct
|
Dec 02 21:15:52 mysqld[1566419]: 2021-12-02 21:15:52 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 1e77c150-8d79 tcp://192.168.77.160:4567
|
Dec 02 21:15:52 mysqld[1566419]: 2021-12-02 21:15:52 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 58fa90ba-871c tcp://192.168.77.162:4567
|
Dec 02 21:15:52 mysqld[1566419]: /usr/sbin/mysqld(_Z14wsrep_bf_abortP3THDS0_+0x6e9)[0x564296fbd7f9]
|
Dec 02 21:15:53 mysqld[1566419]: 2021-12-02 21:15:53 0 [Note] WSREP: evs::proto(2f8c7fd2-91e6, OPERATIONAL, view_id(REG,1e77c150-8d79,110)) suspecting node: 1e77c150-8d79
|
Dec 02 21:15:53 mysqld[1566419]: 2021-12-02 21:15:53 0 [Note] WSREP: evs::proto(2f8c7fd2-91e6, OPERATIONAL, view_id(REG,1e77c150-8d79,110)) suspected node without join message, declaring inactive
|
Dec 02 21:15:53 mysqld[1566419]: 2021-12-02 21:15:53 0 [Note] WSREP: view(view_id(NON_PRIM,1e77c150-8d79,110) memb {
|
Dec 02 21:15:53 mysqld[1566419]: 2f8c7fd2-91e6,0
|
Dec 02 21:15:53 mysqld[1566419]: } joined {
|
Dec 02 21:15:53 mysqld[1566419]: } left {
|
Dec 02 21:15:53 mysqld[1566419]: } partitioned {
|
Dec 02 21:15:53 mysqld[1566419]: 1e77c150-8d79,0
|
Dec 02 21:15:53 mysqld[1566419]: 58fa90ba-871c,0
|
Dec 02 21:15:53 mysqld[1566419]: })
|
Dec 02 21:15:53 mysqld[1566419]: 2021-12-02 21:15:53 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
|
Dec 02 21:15:53 mysqld[1566419]: 2021-12-02 21:15:53 0 [Note] WSREP: view(view_id(NON_PRIM,2f8c7fd2-91e6,111) memb {
|
Dec 02 21:15:53 mysqld[1566419]: 2f8c7fd2-91e6,0
|
Dec 02 21:15:53 mysqld[1566419]: } joined {
|
Dec 02 21:15:53 mysqld[1566419]: } left {
|
Dec 02 21:15:53 mysqld[1566419]: } partitioned {
|
Dec 02 21:15:53 mysqld[1566419]: 1e77c150-8d79,0
|
Dec 02 21:15:53 mysqld[1566419]: 58fa90ba-871c,0
|
Dec 02 21:15:53 mysqld[1566419]: })
|
Dec 02 21:15:53 mysqld[1566419]: 2021-12-02 21:15:53 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
|
Dec 02 21:15:53 mysqld[1566419]: 2021-12-02 21:15:53 0 [Note] WSREP: Received NON-PRIMARY.
|
Dec 02 21:15:53 mysqld[1566419]: 2021-12-02 21:15:53 0 [Note] WSREP: Shifting PRIMARY -> OPEN (TO: 4857091694)
|
Dec 02 21:15:53 mysqld[1566419]: 2021-12-02 21:15:53 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
|
Dec 02 21:15:53 mysqld[1566419]: 2021-12-02 21:15:53 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
|
Dec 02 21:15:53 mysqld[1566419]: 2021-12-02 21:15:53 0 [Note] WSREP: Received NON-PRIMARY.
|
Dec 02 21:15:53 mysqld[1566419]: /usr/sbin/mysqld(wsrep_thd_bf_abort+0x1d)[0x564296fc56bd]
|
Dec 02 21:15:53 mysqld[1566419]: /usr/sbin/mysqld(+0xaab1b6)[0x5642971fb1b6]
|
Dec 02 21:15:54 mysqld[1566419]: /usr/sbin/mysqld(+0xb0cd84)[0x56429725cd84]
|
Dec 02 21:15:54 mysqld[1566419]: /usr/sbin/mysqld(+0xb12e62)[0x564297262e62]
|
Dec 02 21:15:55 mysqld[1566419]: 2021-12-02 21:15:55 0 [Note] WSREP: declaring 1e77c150-8d79 at tcp://192.168.77.160:4567 stable
|
Dec 02 21:15:55 mysqld[1566419]: 2021-12-02 21:15:55 0 [Note] WSREP: declaring 58fa90ba-871c at tcp://192.168.77.162:4567 stable
|
Dec 02 21:15:55 mysqld[1566419]: 2021-12-02 21:15:55 0 [Note] WSREP: Node 1e77c150-8d79 state prim
|
Dec 02 21:15:55 mysqld[1566419]: 2021-12-02 21:15:55 0 [Note] WSREP: view(view_id(PRIM,1e77c150-8d79,112) memb {
|
Dec 02 21:15:55 mysqld[1566419]: 1e77c150-8d79,0
|
Dec 02 21:15:55 mysqld[1566419]: 2f8c7fd2-91e6,0
|
Dec 02 21:15:55 mysqld[1566419]: 58fa90ba-871c,0
|
Dec 02 21:15:55 mysqld[1566419]: } joined {
|
Dec 02 21:15:55 mysqld[1566419]: } left {
|
Dec 02 21:15:55 mysqld[1566419]: } partitioned {
|
Dec 02 21:15:55 mysqld[1566419]: })
|
Dec 02 21:15:55 mysqld[1566419]: 2021-12-02 21:15:55 0 [Note] WSREP: save pc into disk
|
Dec 02 21:15:55 mysqld[1566419]: 2021-12-02 21:15:55 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 3
|
Dec 02 21:15:55 mysqld[1566419]: 2021-12-02 21:15:55 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
|
Dec 02 21:15:55 mysqld[1566419]: 2021-12-02 21:15:55 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 0931be05-53b5-11ec-bba0-7f8149696efd
|
Dec 02 21:15:55 mysqld[1566419]: 2021-12-02 21:15:55 0 [Note] WSREP: STATE EXCHANGE: got state msg: 0931be05-53b5-11ec-bba0-7f8149696efd from 0 (zabbix-db04)
|
Dec 02 21:15:55 mysqld[1566419]: 2021-12-02 21:15:55 0 [Note] WSREP: STATE EXCHANGE: got state msg: 0931be05-53b5-11ec-bba0-7f8149696efd from 1 (zabbix-db05)
|
Dec 02 21:15:55 mysqld[1566419]: 2021-12-02 21:15:55 0 [Note] WSREP: STATE EXCHANGE: got state msg: 0931be05-53b5-11ec-bba0-7f8149696efd from 2 (zabbix-db06)
|
Dec 02 21:15:55 mysqld[1566419]: 2021-12-02 21:15:55 0 [Note] WSREP: Quorum results:
|
Dec 02 21:15:55 mysqld[1566419]: version = 6,
|
Dec 02 21:15:55 mysqld[1566419]: component = PRIMARY,
|
Dec 02 21:15:55 mysqld[1566419]: conf_id = 43,
|
Dec 02 21:15:55 mysqld[1566419]: members = 2/3 (joined/total),
|
Dec 02 21:15:55 mysqld[1566419]: act_id = 4857091708,
|
Dec 02 21:15:55 mysqld[1566419]: last_appl. = 4857091469,
|
Dec 02 21:15:55 mysqld[1566419]: protocols = 2/10/4 (gcs/repl/appl),
|
Dec 02 21:15:55 mysqld[1566419]: vote policy= 0,
|
Dec 02 21:15:55 mysqld[1566419]: group UUID = 810a5d0c-231c-11e9-b756-dbb7899d7dd0
|
Dec 02 21:15:55 mysqld[1566419]: 2021-12-02 21:15:55 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
|
Dec 02 21:15:55 mysqld[1566419]: 2021-12-02 21:15:55 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 4857091709)
|
Dec 02 21:15:55 mysqld[1566419]: /usr/sbin/mysqld(+0xb9fdd1)[0x5642972efdd1]
|
Dec 02 21:15:55 mysqld[1566419]: /usr/sbin/mysqld(+0xac65b1)[0x5642972165b1]
|
Dec 02 21:15:56 mysqld[1566419]: 2021-12-02 21:15:56 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') turning message relay requesting off
|
Dec 02 21:15:56 mysqld[1566419]: /usr/sbin/mysqld(_ZN7handler10ha_rnd_posEPhS0_+0x50)[0x564297049690]
|
Dec 02 21:15:56 mysqld[1566419]: /usr/sbin/mysqld(+0x818150)[0x564296f68150]
|
Dec 02 21:15:57 mysqld[1566419]: /usr/sbin/mysqld(_ZN14Rows_log_event8find_rowEP14rpl_group_info+0x451)[0x56429715ef91]
|
Dec 02 21:15:57 mysqld[1566419]: /usr/sbin/mysqld(_ZN21Delete_rows_log_event11do_exec_rowEP14rpl_group_info+0x92)[0x56429715f252]
|
Dec 02 21:15:58 mysqld[1566419]: /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x23f)[0x56429715275f]
|
Dec 02 21:15:58 mysqld[1566419]: /usr/sbin/mysqld(_Z18wsrep_apply_eventsP3THDP14Relay_log_infoPKvm+0x1b9)[0x564296fbc979]
|
Dec 02 21:15:59 mysqld[1566419]: /usr/sbin/mysqld(_ZN21Wsrep_applier_service15apply_write_setERKN5wsrep7ws_metaERKNS0_12const_bufferERNS0_14mutable_bufferE+0xa3)[0x564296fa3fc3]
|
Dec 02 21:15:59 mysqld[1566419]: /usr/sbin/mysqld(+0xeaf580)[0x5642975ff580]
|
Dec 02 21:16:00 mysqld[1566419]: /usr/sbin/mysqld(_ZN5wsrep12server_state8on_applyERNS_21high_priority_serviceERKNS_9ws_handleERKNS_7ws_metaERKNS_12const_bufferE+0xc1)[0x564297600441]
|
Dec 02 21:16:00 mysqld[1566419]: /usr/sbin/mysqld(+0xec187c)[0x56429761187c]
|
Dec 02 21:16:05 mysqld[1566419]: 2021-12-02 21:16:05 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection to peer 1e77c150-8d79 with addr tcp://192.168.77.160:4567 timed out, no messages seen in PT3S, socket stats: rtt: 3289 rttvar: 6042 rto: 204000 lost: 0 last_data_recv: 2212 cwnd: 10 last_queued_since: 458625 last_delivered_since: 5346986204 send_queue_length: 0 send_queue_bytes: 0 segment: 0 messages: 0
|
Dec 02 21:16:05 mysqld[1566419]: 2021-12-02 21:16:05 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://192.168.77.160:4567
|
Dec 02 21:16:05 mysqld[1566419]: 2021-12-02 21:16:05 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT5.4279S), skipping check
|
Dec 02 21:16:05 mysqld[1566419]: 2021-12-02 21:16:05 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 58fa90ba-871c tcp://192.168.77.162:4567
|
Dec 02 21:16:05 mysqld[1566419]: 2021-12-02 21:16:05 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 1e77c150-8d79 tcp://192.168.77.160:4567
|
Dec 02 21:16:05 mysqld[1566419]: 2021-12-02 21:16:05 0 [Note] WSREP: view(view_id(NON_PRIM,1e77c150-8d79,112) memb {
|
Dec 02 21:16:05 mysqld[1566419]: 2f8c7fd2-91e6,0
|
Dec 02 21:16:05 mysqld[1566419]: } joined {
|
Dec 02 21:16:05 mysqld[1566419]: } left {
|
Dec 02 21:16:05 mysqld[1566419]: } partitioned {
|
Dec 02 21:16:05 mysqld[1566419]: 1e77c150-8d79,0
|
Dec 02 21:16:05 mysqld[1566419]: 58fa90ba-871c,0
|
Dec 02 21:16:05 mysqld[1566419]: })
|
Dec 02 21:16:05 mysqld[1566419]: 2021-12-02 21:16:05 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
|
Dec 02 21:16:05 mysqld[1566419]: 2021-12-02 21:16:05 0 [Note] WSREP: view(view_id(NON_PRIM,2f8c7fd2-91e6,113) memb {
|
Dec 02 21:16:05 mysqld[1566419]: 2f8c7fd2-91e6,0
|
Dec 02 21:16:05 mysqld[1566419]: } joined {
|
Dec 02 21:16:05 mysqld[1566419]: } left {
|
Dec 02 21:16:05 mysqld[1566419]: } partitioned {
|
Dec 02 21:16:05 mysqld[1566419]: 1e77c150-8d79,0
|
Dec 02 21:16:05 mysqld[1566419]: 58fa90ba-871c,0
|
Dec 02 21:16:05 mysqld[1566419]: })
|
Dec 02 21:16:05 mysqld[1566419]: 2021-12-02 21:16:05 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
|
Dec 02 21:16:05 mysqld[1566419]: 2021-12-02 21:16:05 0 [Note] WSREP: Received NON-PRIMARY.
|
Dec 02 21:16:05 mysqld[1566419]: 2021-12-02 21:16:05 0 [Note] WSREP: Shifting PRIMARY -> OPEN (TO: 4857092761)
|
Dec 02 21:16:05 mysqld[1566419]: 2021-12-02 21:16:05 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
|
Dec 02 21:16:05 mysqld[1566419]: 2021-12-02 21:16:05 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
|
Dec 02 21:16:05 mysqld[1566419]: 2021-12-02 21:16:05 0 [Note] WSREP: Received NON-PRIMARY.
|
Dec 02 21:16:05 mysqld[1566419]: /usr/lib/libgalera_smm.so(+0x639eb)[0x7f34865129eb]
|
Dec 02 21:16:06 mysqld[1566419]: /usr/lib/libgalera_smm.so(+0x7582e)[0x7f348652482e]
|
Dec 02 21:16:06 mysqld[1566419]: /usr/lib/libgalera_smm.so(+0x78408)[0x7f3486527408]
|
Dec 02 21:16:07 mysqld[1566419]: /usr/lib/libgalera_smm.so(+0xabd92)[0x7f348655ad92]
|
Dec 02 21:16:07 mysqld[1566419]: 2021-12-02 21:16:07 0 [Note] WSREP: declaring 1e77c150-8d79 at tcp://192.168.77.160:4567 stable
|
Dec 02 21:16:07 mysqld[1566419]: 2021-12-02 21:16:07 0 [Note] WSREP: declaring 58fa90ba-871c at tcp://192.168.77.162:4567 stable
|
Dec 02 21:16:07 mysqld[1566419]: 2021-12-02 21:16:07 0 [Note] WSREP: Node 1e77c150-8d79 state prim
|
Dec 02 21:16:07 mysqld[1566419]: 2021-12-02 21:16:07 0 [Note] WSREP: view(view_id(PRIM,1e77c150-8d79,114) memb {
|
Dec 02 21:16:07 mysqld[1566419]: 1e77c150-8d79,0
|
Dec 02 21:16:07 mysqld[1566419]: 2f8c7fd2-91e6,0
|
Dec 02 21:16:07 mysqld[1566419]: 58fa90ba-871c,0
|
Dec 02 21:16:07 mysqld[1566419]: } joined {
|
Dec 02 21:16:07 mysqld[1566419]: } left {
|
Dec 02 21:16:07 mysqld[1566419]: } partitioned {
|
Dec 02 21:16:07 mysqld[1566419]: })
|
Dec 02 21:16:07 mysqld[1566419]: 2021-12-02 21:16:07 0 [Note] WSREP: save pc into disk
|
Dec 02 21:16:07 mysqld[1566419]: 2021-12-02 21:16:07 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 3
|
Dec 02 21:16:07 mysqld[1566419]: 2021-12-02 21:16:07 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
|
Dec 02 21:16:07 mysqld[1566419]: 2021-12-02 21:16:07 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 10a57d2b-53b5-11ec-b6d6-efe5f64759fc
|
Dec 02 21:16:07 mysqld[1566419]: 2021-12-02 21:16:07 0 [Note] WSREP: STATE EXCHANGE: got state msg: 10a57d2b-53b5-11ec-b6d6-efe5f64759fc from 0 (zabbix-db04)
|
Dec 02 21:16:07 mysqld[1566419]: 2021-12-02 21:16:07 0 [Note] WSREP: STATE EXCHANGE: got state msg: 10a57d2b-53b5-11ec-b6d6-efe5f64759fc from 1 (zabbix-db05)
|
Dec 02 21:16:07 mysqld[1566419]: 2021-12-02 21:16:07 0 [Note] WSREP: STATE EXCHANGE: got state msg: 10a57d2b-53b5-11ec-b6d6-efe5f64759fc from 2 (zabbix-db06)
|
Dec 02 21:16:07 mysqld[1566419]: 2021-12-02 21:16:07 0 [Note] WSREP: Quorum results:
|
Dec 02 21:16:07 mysqld[1566419]: version = 6,
|
Dec 02 21:16:07 mysqld[1566419]: component = PRIMARY,
|
Dec 02 21:16:07 mysqld[1566419]: conf_id = 44,
|
Dec 02 21:16:07 mysqld[1566419]: members = 2/3 (joined/total),
|
Dec 02 21:16:07 mysqld[1566419]: act_id = 4857092776,
|
Dec 02 21:16:07 mysqld[1566419]: last_appl. = 4857091469,
|
Dec 02 21:16:07 mysqld[1566419]: protocols = 2/10/4 (gcs/repl/appl),
|
Dec 02 21:16:07 mysqld[1566419]: vote policy= 0,
|
Dec 02 21:16:07 mysqld[1566419]: group UUID = 810a5d0c-231c-11e9-b756-dbb7899d7dd0
|
Dec 02 21:16:07 mysqld[1566419]: 2021-12-02 21:16:07 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
|
Dec 02 21:16:07 mysqld[1566419]: 2021-12-02 21:16:07 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 4857092777)
|
Dec 02 21:16:07 mysqld[1566419]: /usr/lib/libgalera_smm.so(+0xabf1b)[0x7f348655af1b]
|
Dec 02 21:16:08 mysqld[1566419]: /usr/lib/libgalera_smm.so(+0xac112)[0x7f348655b112]
|
Dec 02 21:16:08 mysqld[1566419]: 2021-12-02 21:16:08 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') turning message relay requesting off
|
Dec 02 21:16:08 mysqld[1566419]: /usr/lib/libgalera_smm.so(+0x78e00)[0x7f3486527e00]
|
Dec 02 21:16:09 mysqld[1566419]: /usr/lib/libgalera_smm.so(+0x4efe1)[0x7f34864fdfe1]
|
Dec 02 21:16:19 mysqld[1566419]: 2021-12-02 21:16:19 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection to peer 1e77c150-8d79 with addr tcp://192.168.77.160:4567 timed out, no messages seen in PT3S, socket stats: rtt: 5633 rttvar: 10029 rto: 208000 lost: 0 last_data_recv: 2420 cwnd: 10 last_queued_since: 854370 last_delivered_since: 5590235758 send_queue_length: 0 send_queue_bytes: 0 segment: 0 messages: 0
|
Dec 02 21:16:19 mysqld[1566419]: 2021-12-02 21:16:19 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://192.168.77.160:4567
|
Dec 02 21:16:19 mysqld[1566419]: 2021-12-02 21:16:19 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT5.63521S), skipping check
|
Dec 02 21:16:19 mysqld[1566419]: 2021-12-02 21:16:19 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 1e77c150-8d79 tcp://192.168.77.160:4567
|
Dec 02 21:16:19 mysqld[1566419]: 2021-12-02 21:16:19 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 58fa90ba-871c tcp://192.168.77.162:4567
|
Dec 02 21:16:20 mysqld[1566419]: 2021-12-02 21:16:20 0 [Note] WSREP: evs::proto(2f8c7fd2-91e6, OPERATIONAL, view_id(REG,1e77c150-8d79,114)) suspecting node: 1e77c150-8d79
|
Dec 02 21:16:20 mysqld[1566419]: 2021-12-02 21:16:20 0 [Note] WSREP: evs::proto(2f8c7fd2-91e6, OPERATIONAL, view_id(REG,1e77c150-8d79,114)) suspected node without join message, declaring inactive
|
Dec 02 21:16:20 mysqld[1566419]: 2021-12-02 21:16:20 0 [Note] WSREP: view(view_id(NON_PRIM,1e77c150-8d79,114) memb {
|
Dec 02 21:16:20 mysqld[1566419]: 2f8c7fd2-91e6,0
|
Dec 02 21:16:20 mysqld[1566419]: } joined {
|
Dec 02 21:16:20 mysqld[1566419]: } left {
|
Dec 02 21:16:20 mysqld[1566419]: } partitioned {
|
Dec 02 21:16:20 mysqld[1566419]: 1e77c150-8d79,0
|
Dec 02 21:16:20 mysqld[1566419]: 58fa90ba-871c,0
|
Dec 02 21:16:20 mysqld[1566419]: })
|
Dec 02 21:16:20 mysqld[1566419]: 2021-12-02 21:16:20 0 [Note] WSREP: view(view_id(NON_PRIM,2f8c7fd2-91e6,115) memb {
|
Dec 02 21:16:20 mysqld[1566419]: 2f8c7fd2-91e6,0
|
Dec 02 21:16:20 mysqld[1566419]: } joined {
|
Dec 02 21:16:20 mysqld[1566419]: } left {
|
Dec 02 21:16:20 mysqld[1566419]: } partitioned {
|
Dec 02 21:16:20 mysqld[1566419]: 1e77c150-8d79,0
|
Dec 02 21:16:20 mysqld[1566419]: 58fa90ba-871c,0
|
Dec 02 21:16:20 mysqld[1566419]: })
|
Dec 02 21:16:20 mysqld[1566419]: 2021-12-02 21:16:20 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
|
Dec 02 21:16:20 mysqld[1566419]: 2021-12-02 21:16:20 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
|
Dec 02 21:16:20 mysqld[1566419]: 2021-12-02 21:16:20 0 [Note] WSREP: Received NON-PRIMARY.
|
Dec 02 21:16:20 mysqld[1566419]: 2021-12-02 21:16:20 0 [Note] WSREP: Shifting PRIMARY -> OPEN (TO: 4857094033)
|
Dec 02 21:16:20 mysqld[1566419]: 2021-12-02 21:16:20 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
|
Dec 02 21:16:20 mysqld[1566419]: 2021-12-02 21:16:20 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
|
Dec 02 21:16:20 mysqld[1566419]: 2021-12-02 21:16:20 0 [Note] WSREP: Received NON-PRIMARY.
|
Dec 02 21:16:21 mysqld[1566419]: 2021-12-02 21:16:21 0 [Note] WSREP: declaring 1e77c150-8d79 at tcp://192.168.77.160:4567 stable
|
Dec 02 21:16:21 mysqld[1566419]: 2021-12-02 21:16:21 0 [Note] WSREP: declaring 58fa90ba-871c at tcp://192.168.77.162:4567 stable
|
Dec 02 21:16:21 mysqld[1566419]: 2021-12-02 21:16:21 0 [Note] WSREP: Node 1e77c150-8d79 state prim
|
Dec 02 21:16:21 mysqld[1566419]: 2021-12-02 21:16:21 0 [Note] WSREP: view(view_id(PRIM,1e77c150-8d79,116) memb {
|
Dec 02 21:16:21 mysqld[1566419]: 1e77c150-8d79,0
|
Dec 02 21:16:21 mysqld[1566419]: 2f8c7fd2-91e6,0
|
Dec 02 21:16:21 mysqld[1566419]: 58fa90ba-871c,0
|
Dec 02 21:16:21 mysqld[1566419]: } joined {
|
Dec 02 21:16:21 mysqld[1566419]: } left {
|
Dec 02 21:16:21 mysqld[1566419]: } partitioned {
|
Dec 02 21:16:21 mysqld[1566419]: })
|
Dec 02 21:16:21 mysqld[1566419]: 2021-12-02 21:16:21 0 [Note] WSREP: save pc into disk
|
Dec 02 21:16:21 mysqld[1566419]: 2021-12-02 21:16:21 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 3
|
Dec 02 21:16:21 mysqld[1566419]: 2021-12-02 21:16:21 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
|
Dec 02 21:16:21 mysqld[1566419]: 2021-12-02 21:16:21 0 [Warning] WSREP: Last Applied Action message in non-primary configuration from member 0
|
Dec 02 21:16:21 mysqld[1566419]: 2021-12-02 21:16:21 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 18fe4842-53b5-11ec-bd11-4b161e80910e
|
Dec 02 21:16:21 mysqld[1566419]: 2021-12-02 21:16:21 0 [Note] WSREP: STATE EXCHANGE: got state msg: 18fe4842-53b5-11ec-bd11-4b161e80910e from 2 (zabbix-db06)
|
Dec 02 21:16:21 mysqld[1566419]: 2021-12-02 21:16:21 0 [Note] WSREP: STATE EXCHANGE: got state msg: 18fe4842-53b5-11ec-bd11-4b161e80910e from 0 (zabbix-db04)
|
Dec 02 21:16:21 mysqld[1566419]: 2021-12-02 21:16:21 0 [Note] WSREP: STATE EXCHANGE: got state msg: 18fe4842-53b5-11ec-bd11-4b161e80910e from 1 (zabbix-db05)
|
Dec 02 21:16:21 mysqld[1566419]: 2021-12-02 21:16:21 0 [Note] WSREP: Quorum results:
|
Dec 02 21:16:21 mysqld[1566419]: version = 6,
|
Dec 02 21:16:21 mysqld[1566419]: component = PRIMARY,
|
Dec 02 21:16:21 mysqld[1566419]: conf_id = 45,
|
Dec 02 21:16:21 mysqld[1566419]: members = 2/3 (joined/total),
|
Dec 02 21:16:21 mysqld[1566419]: act_id = 4857094048,
|
Dec 02 21:16:21 mysqld[1566419]: last_appl. = 4857091469,
|
Dec 02 21:16:21 mysqld[1566419]: protocols = 2/10/4 (gcs/repl/appl),
|
Dec 02 21:16:21 mysqld[1566419]: vote policy= 0,
|
Dec 02 21:16:21 mysqld[1566419]: group UUID = 810a5d0c-231c-11e9-b756-dbb7899d7dd0
|
Dec 02 21:16:21 mysqld[1566419]: 2021-12-02 21:16:21 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
|
Dec 02 21:16:21 mysqld[1566419]: 2021-12-02 21:16:21 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 4857094049)
|
Dec 02 21:16:23 mysqld[1566419]: 2021-12-02 21:16:23 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') turning message relay requesting off
|
Dec 02 21:16:28 mysqld[1566419]: /usr/sbin/mysqld(_ZN5wsrep18wsrep_provider_v2611run_applierEPNS_21high_priority_serviceE+0x12)[0x564297611e92]
|
Dec 02 21:16:28 mysqld[1566419]: /usr/sbin/mysqld(+0x86e7c3)[0x564296fbe7c3]
|
Dec 02 21:16:28 mysqld[1566419]: /usr/sbin/mysqld(_Z15start_wsrep_THDPv+0x2ec)[0x564296faef1c]
|
Dec 02 21:16:34 mysqld[1566419]: 2021-12-02 21:16:34 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection to peer 1e77c150-8d79 with addr tcp://192.168.77.160:4567 timed out, no messages seen in PT3S, socket stats: rtt: 2200 rttvar: 2183 rto: 204000 lost: 0 last_data_recv: 3048 cwnd: 10 last_queued_since: 420943 last_delivered_since: 6114827970 send_queue_length: 4 send_queue_bytes: 320 segment: 0 messages: 4
|
Dec 02 21:16:34 mysqld[1566419]: 2021-12-02 21:16:34 0 [Note] WSREP: Deferred close timer started for socket with remote endpoint: tcp://192.168.77.160:51218
|
Dec 02 21:16:34 mysqld[1566419]: 2021-12-02 21:16:34 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://192.168.77.160:4567
|
Dec 02 21:16:34 mysqld[1566419]: 2021-12-02 21:16:34 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT6.26427S), skipping check
|
Dec 02 21:16:34 mysqld[1566419]: 2021-12-02 21:16:34 0 [Note] WSREP: Deferred close timer handle_wait Operation aborted. for 0x56429a295c70
|
Dec 02 21:16:34 mysqld[1566419]: 2021-12-02 21:16:34 0 [Note] WSREP: Deferred close timer destruct
|
Dec 02 21:16:34 mysqld[1566419]: 2021-12-02 21:16:34 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 58fa90ba-871c tcp://192.168.77.162:4567
|
Dec 02 21:16:34 mysqld[1566419]: 2021-12-02 21:16:34 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 1e77c150-8d79 tcp://192.168.77.160:4567
|
Dec 02 21:16:34 mysqld[1566419]: 2021-12-02 21:16:34 0 [Note] WSREP: view(view_id(NON_PRIM,1e77c150-8d79,116) memb {
|
Dec 02 21:16:34 mysqld[1566419]: 2f8c7fd2-91e6,0
|
Dec 02 21:16:34 mysqld[1566419]: } joined {
|
Dec 02 21:16:34 mysqld[1566419]: } left {
|
Dec 02 21:16:34 mysqld[1566419]: } partitioned {
|
Dec 02 21:16:34 mysqld[1566419]: 1e77c150-8d79,0
|
Dec 02 21:16:34 mysqld[1566419]: 58fa90ba-871c,0
|
Dec 02 21:16:34 mysqld[1566419]: })
|
Dec 02 21:16:34 mysqld[1566419]: 2021-12-02 21:16:34 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
|
Dec 02 21:16:34 mysqld[1566419]: 2021-12-02 21:16:34 0 [Note] WSREP: view(view_id(NON_PRIM,2f8c7fd2-91e6,117) memb {
|
Dec 02 21:16:34 mysqld[1566419]: 2f8c7fd2-91e6,0
|
Dec 02 21:16:34 mysqld[1566419]: } joined {
|
Dec 02 21:16:34 mysqld[1566419]: } left {
|
Dec 02 21:16:34 mysqld[1566419]: } partitioned {
|
Dec 02 21:16:34 mysqld[1566419]: 1e77c150-8d79,0
|
Dec 02 21:16:34 mysqld[1566419]: 58fa90ba-871c,0
|
Dec 02 21:16:34 mysqld[1566419]: })
|
Dec 02 21:16:34 mysqld[1566419]: 2021-12-02 21:16:34 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
|
Dec 02 21:16:34 mysqld[1566419]: 2021-12-02 21:16:34 0 [Note] WSREP: Received NON-PRIMARY.
|
Dec 02 21:16:34 mysqld[1566419]: 2021-12-02 21:16:34 0 [Note] WSREP: Shifting PRIMARY -> OPEN (TO: 4857095264)
|
Dec 02 21:16:34 mysqld[1566419]: 2021-12-02 21:16:34 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
|
Dec 02 21:16:34 mysqld[1566419]: 2021-12-02 21:16:34 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
|
Dec 02 21:16:34 mysqld[1566419]: 2021-12-02 21:16:34 0 [Note] WSREP: Received NON-PRIMARY.
|
Dec 02 21:16:34 mysqld[1566419]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609)[0x7f3487792609]
|
Dec 02 21:16:35 mysqld[1566419]: 2021-12-02 21:16:35 0 [Note] WSREP: declaring 1e77c150-8d79 at tcp://192.168.77.160:4567 stable
|
Dec 02 21:16:35 mysqld[1566419]: 2021-12-02 21:16:35 0 [Note] WSREP: declaring 58fa90ba-871c at tcp://192.168.77.162:4567 stable
|
Dec 02 21:16:36 mysqld[1566419]: 2021-12-02 21:16:36 0 [Note] WSREP: Node 1e77c150-8d79 state prim
|
Dec 02 21:16:36 mysqld[1566419]: 2021-12-02 21:16:36 0 [Note] WSREP: view(view_id(PRIM,1e77c150-8d79,118) memb {
|
Dec 02 21:16:36 mysqld[1566419]: 1e77c150-8d79,0
|
Dec 02 21:16:36 mysqld[1566419]: 2f8c7fd2-91e6,0
|
Dec 02 21:16:36 mysqld[1566419]: 58fa90ba-871c,0
|
Dec 02 21:16:36 mysqld[1566419]: } joined {
|
Dec 02 21:16:36 mysqld[1566419]: } left {
|
Dec 02 21:16:36 mysqld[1566419]: } partitioned {
|
Dec 02 21:16:36 mysqld[1566419]: })
|
Dec 02 21:16:36 mysqld[1566419]: 2021-12-02 21:16:36 0 [Note] WSREP: save pc into disk
|
Dec 02 21:16:36 mysqld[1566419]: 2021-12-02 21:16:36 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 3
|
Dec 02 21:16:36 mysqld[1566419]: 2021-12-02 21:16:36 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
|
Dec 02 21:16:36 mysqld[1566419]: 2021-12-02 21:16:36 0 [Warning] WSREP: Action message in non-primary configuration from member 2
|
Dec 02 21:16:36 mysqld[1566419]: 2021-12-02 21:16:36 0 [Warning] WSREP: Action message in non-primary configuration from member 2
|
Dec 02 21:16:36 mysqld[1566419]: 2021-12-02 21:16:36 0 [Warning] WSREP: Action message in non-primary configuration from member 2
|
Dec 02 21:16:36 mysqld[1566419]: 2021-12-02 21:16:36 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 21efa719-53b5-11ec-b8ad-2b1fb78722e9
|
Dec 02 21:16:36 mysqld[1566419]: 2021-12-02 21:16:36 0 [Note] WSREP: STATE EXCHANGE: got state msg: 21efa719-53b5-11ec-b8ad-2b1fb78722e9 from 0 (zabbix-db04)
|
Dec 02 21:16:36 mysqld[1566419]: 2021-12-02 21:16:36 0 [Note] WSREP: STATE EXCHANGE: got state msg: 21efa719-53b5-11ec-b8ad-2b1fb78722e9 from 1 (zabbix-db05)
|
Dec 02 21:16:36 mysqld[1566419]: 2021-12-02 21:16:36 0 [Note] WSREP: STATE EXCHANGE: got state msg: 21efa719-53b5-11ec-b8ad-2b1fb78722e9 from 2 (zabbix-db06)
|
Dec 02 21:16:36 mysqld[1566419]: 2021-12-02 21:16:36 0 [Note] WSREP: Quorum results:
|
Dec 02 21:16:36 mysqld[1566419]: version = 6,
|
Dec 02 21:16:36 mysqld[1566419]: component = PRIMARY,
|
Dec 02 21:16:36 mysqld[1566419]: conf_id = 46,
|
Dec 02 21:16:36 mysqld[1566419]: members = 2/3 (joined/total),
|
Dec 02 21:16:36 mysqld[1566419]: act_id = 4857095281,
|
Dec 02 21:16:36 mysqld[1566419]: last_appl. = 4857091469,
|
Dec 02 21:16:36 mysqld[1566419]: protocols = 2/10/4 (gcs/repl/appl),
|
Dec 02 21:16:36 mysqld[1566419]: vote policy= 0,
|
Dec 02 21:16:36 mysqld[1566419]: group UUID = 810a5d0c-231c-11e9-b756-dbb7899d7dd0
|
Dec 02 21:16:36 mysqld[1566419]: 2021-12-02 21:16:36 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
|
Dec 02 21:16:36 mysqld[1566419]: 2021-12-02 21:16:36 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 4857095282)
|
Dec 02 21:16:37 mysqld[1566419]: 2021-12-02 21:16:37 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') turning message relay requesting off
|
Dec 02 21:16:48 mysqld[1566419]: 2021-12-02 21:16:48 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection to peer 1e77c150-8d79 with addr tcp://192.168.77.160:4567 timed out, no messages seen in PT3S, socket stats: rtt: 2007 rttvar: 2859 rto: 204000 lost: 0 last_data_recv: 2264 cwnd: 10 last_queued_since: 422869 last_delivered_since: 5570929836 send_queue_length: 4 send_queue_bytes: 320 segment: 0 messages: 4
|
Dec 02 21:16:48 mysqld[1566419]: 2021-12-02 21:16:48 0 [Note] WSREP: Deferred close timer started for socket with remote endpoint: tcp://192.168.77.160:51222
|
Dec 02 21:16:48 mysqld[1566419]: 2021-12-02 21:16:48 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://192.168.77.160:4567
|
Dec 02 21:16:48 mysqld[1566419]: 2021-12-02 21:16:48 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT5.97614S), skipping check
|
Dec 02 21:16:48 mysqld[1566419]: 2021-12-02 21:16:48 0 [Note] WSREP: Deferred close timer handle_wait Operation aborted. for 0x7f13a0044ea0
|
Dec 02 21:16:48 mysqld[1566419]: 2021-12-02 21:16:48 0 [Note] WSREP: Deferred close timer destruct
|
Dec 02 21:16:48 mysqld[1566419]: 2021-12-02 21:16:48 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 58fa90ba-871c tcp://192.168.77.162:4567
|
Dec 02 21:16:48 mysqld[1566419]: 2021-12-02 21:16:48 0 [Note] WSREP: (2f8c7fd2-91e6, 'tcp://0.0.0.0:4567') connection established to 1e77c150-8d79 tcp://192.168.77.160:4567
|
Dec 02 21:16:48 mysqld[1566419]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x43)[0x7f3487380293]
|
Dec 02 21:16:48 mysqld[1566419]: Trying to get some variables.
|
Dec 02 21:16:48 mysqld[1566419]: Some pointers may be invalid and cause the dump to abort.
|
Dec 02 21:16:48 mysqld[1566419]: Query (0x7f16e8a3da93): delete from problem where r_clock<>0 and r_clock<1638393301
|
Dec 02 21:16:48 mysqld[1566419]: Connection ID (thread ID): 66
|
Dec 02 21:16:48 mysqld[1566419]: Status: NOT_KILLED
|
Dec 02 21:16:48 mysqld[1566419]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on
|
Dec 02 21:16:48 mysqld[1566419]: The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
|
Dec 02 21:16:48 mysqld[1566419]: information that should help you find out what is causing the crash.
|
Dec 02 21:16:48 mysqld[1566419]: Writing a core file...
|
Dec 02 21:16:48 mysqld[1566419]: Working directory at /var/lib/mysql
|
Dec 02 21:16:48 mysqld[1566419]: Resource Limits:
|
Dec 02 21:16:48 mysqld[1566419]: Limit Soft Limit Hard Limit Units
|
Dec 02 21:16:48 mysqld[1566419]: Max cpu time unlimited unlimited seconds
|
Dec 02 21:16:48 mysqld[1566419]: Max file size unlimited unlimited bytes
|
Dec 02 21:16:48 mysqld[1566419]: Max data size unlimited unlimited bytes
|
Dec 02 21:16:48 mysqld[1566419]: Max stack size 8388608 unlimited bytes
|
Dec 02 21:16:48 mysqld[1566419]: Max core file size 0 unlimited bytes
|
Dec 02 21:16:48 mysqld[1566419]: Max resident set unlimited unlimited bytes
|
Dec 02 21:16:48 mysqld[1566419]: Max processes 5931800 5931800 processes
|
Dec 02 21:16:48 mysqld[1566419]: Max open files 65536 65536 files
|
Dec 02 21:16:48 mysqld[1566419]: Max locked memory 65536 65536 bytes
|
Dec 02 21:16:48 mysqld[1566419]: Max address space unlimited unlimited bytes
|
Dec 02 21:16:48 mysqld[1566419]: Max file locks unlimited unlimited locks
|
Dec 02 21:16:48 mysqld[1566419]: Max pending signals 5931800 5931800 signals
|
Dec 02 21:16:48 mysqld[1566419]: Max msgqueue size 819200 819200 bytes
|
Dec 02 21:16:48 mysqld[1566419]: Max nice priority 0 0
|
Dec 02 21:16:48 mysqld[1566419]: Max realtime priority 0 0
|
Dec 02 21:16:48 mysqld[1566419]: Max realtime timeout unlimited unlimited us
|
Dec 02 21:16:48 mysqld[1566419]: Core pattern: /var/crash/%e.%p
|
Dec 02 21:17:13 systemd[1]: mariadb.service: Main process exited, code=killed, status=6/ABRT
|
Dec 02 21:17:13 systemd[1]: mariadb.service: Failed with result 'signal'.
|
And here is what happened (another SIGABRT) when I simply attempted to restart MariaDB:
Dec 02 22:28:24 systemd[1]: Starting MariaDB 10.4.22 database server...
|
Dec 02 22:29:52 mysqld[2095405]: WSREP: Recovered position 810a5d0c-231c-11e9-b756-dbb7899d7dd0:4857087468
|
Dec 02 22:29:52 mysqld[2095962]: 2021-12-02 22:29:52 0 [Note] /usr/sbin/mysqld (mysqld 10.4.22-MariaDB-1:10.4.22+maria~focal-log) starting as process 2095962 ...
|
Dec 02 22:29:52 mysqld[2095962]: 2021-12-02 22:29:52 0 [Note] WSREP: Loading provider /usr/lib/libgalera_smm.so initial position: 810a5d0c-231c-11e9-b756-dbb7899d7dd0:4857087468
|
Dec 02 22:29:52 mysqld[2095962]: 2021-12-02 22:29:52 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/libgalera_smm.so'
|
Dec 02 22:29:52 mysqld[2095962]: 2021-12-02 22:29:52 0 [Note] WSREP: wsrep_load(): Galera 26.4.9(r819f29cb) by Codership Oy <info@codership.com> loaded successfully.
|
Dec 02 22:29:52 mysqld[2095962]: 2021-12-02 22:29:52 0 [Note] WSREP: CRC-32C: using 64-bit x86 acceleration.
|
Dec 02 22:29:52 mysqld[2095962]: 2021-12-02 22:29:52 0 [Note] WSREP: Found saved state: 810a5d0c-231c-11e9-b756-dbb7899d7dd0:-1, safe_to_bootstrap: 0
|
Dec 02 22:29:52 mysqld[2095962]: 2021-12-02 22:29:52 0 [Note] WSREP: GCache DEBUG: opened preamble:
|
Dec 02 22:29:52 mysqld[2095962]: Version: 2
|
Dec 02 22:29:52 mysqld[2095962]: UUID: 810a5d0c-231c-11e9-b756-dbb7899d7dd0
|
Dec 02 22:29:52 mysqld[2095962]: Seqno: -1 - -1
|
Dec 02 22:29:52 mysqld[2095962]: Offset: -1
|
Dec 02 22:29:52 mysqld[2095962]: Synced: 0
|
Dec 02 22:29:52 mysqld[2095962]: 2021-12-02 22:29:52 0 [Note] WSREP: Recovering GCache ring buffer: version: 2, UUID: 810a5d0c-231c-11e9-b756-dbb7899d7dd0, offset: -1
|
Dec 02 22:29:52 mysqld[2095962]: 2021-12-02 22:29:52 0 [Note] WSREP: GCache::RingBuffer initial scan... 0.0% ( 0/140928614424 bytes) complete.
|
Dec 02 22:30:02 mysqld[2095962]: 2021-12-02 22:30:02 0 [Note] WSREP: GCache::RingBuffer initial scan... 79.1% (111525388192/140928614424 bytes) complete.
|
Dec 02 22:30:05 mysqld[2095962]: 2021-12-02 22:30:05 0 [Note] WSREP: GCache::RingBuffer initial scan...100.0% (140928614424/140928614424 bytes) complete.
|
Dec 02 22:30:05 mysqld[2095962]: 2021-12-02 22:30:05 0 [Note] WSREP: Recovering GCache ring buffer: found gapless sequence 4846387206-4857087532
|
Dec 02 22:30:05 mysqld[2095962]: 2021-12-02 22:30:05 0 [Note] WSREP: GCache::RingBuffer unused buffers scan... 0.0% ( 0/140821082440 bytes) complete.
|
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: GCache::RingBuffer unused buffers scan...100.0% (140821082440/140821082440 bytes) complete.
|
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: GCache DEBUG: RingBuffer::recover(): found 7/10700334 locked buffers
|
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: GCache DEBUG: RingBuffer::recover(): free space: 107589352/140928614400
|
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 192.168.77.161; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 134400M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 2000000; gcs.fc_master_slave = yes; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.
|
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: Service thread queue flushed.
|
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: ####### Assign initial position for certification: 810a5d0c-231c-11e9-b756-dbb7899d7dd0:4857087468, protocol version: -1
|
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: Start replication
|
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: Connecting with bootstrap option: 0
|
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: Setting GCS initial position to 810a5d0c-231c-11e9-b756-dbb7899d7dd0:4857087468
|
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: protonet asio version 0
|
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: Using CRC-32C for message checksums.
|
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: backend: asio
|
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: gcomm thread scheduling priority set to other:0
|
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: restore pc from disk successfully
|
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: GMCast version 0
|
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: (2f8c7fd2-91e7, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
|
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: (2f8c7fd2-91e7, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
|
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: EVS version 1
|
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: gcomm: connecting to group 'galera_cluster', peer '192.168.77.160:,192.168.77.161:,192.168.77.162:'
|
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: (2f8c7fd2-91e7, 'tcp://0.0.0.0:4567') connection established to 1e77c150-8d79 tcp://192.168.77.160:4567
|
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: (2f8c7fd2-91e7, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
|
Dec 02 22:30:07 mysqld[2095962]: 2021-12-02 22:30:07 0 [Note] WSREP: (2f8c7fd2-91e7, 'tcp://0.0.0.0:4567') connection established to 58fa90ba-871c tcp://192.168.77.162:4567
|
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: gcomm: connected
|
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
|
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
|
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: Opened channel 'galera_cluster'
|
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 1 [Note] WSREP: Starting rollbacker thread 1
|
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 2 [Note] WSREP: Starting applier thread 2
|
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: EVS version upgrade 0 -> 1
|
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: declaring 1e77c150-8d79 at tcp://192.168.77.160:4567 stable
|
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: declaring 58fa90ba-871c at tcp://192.168.77.162:4567 stable
|
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: PC protocol upgrade 0 -> 1
|
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: Node 1e77c150-8d79 state prim
|
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: view(view_id(PRIM,1e77c150-8d79,120) memb {
|
Dec 02 22:30:08 mysqld[2095962]: 1e77c150-8d79,0
|
Dec 02 22:30:08 mysqld[2095962]: 2f8c7fd2-91e7,0
|
Dec 02 22:30:08 mysqld[2095962]: 58fa90ba-871c,0
|
Dec 02 22:30:08 mysqld[2095962]: } joined {
|
Dec 02 22:30:08 mysqld[2095962]: } left {
|
Dec 02 22:30:08 mysqld[2095962]: } partitioned {
|
Dec 02 22:30:08 mysqld[2095962]: })
|
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: save pc into disk
|
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: discarding pending addr without UUID: tcp://192.168.77.161:4567
|
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: discarding pending addr proto entry 0x55c63ede73c0
|
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: clear restored view
|
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 3
|
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
|
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 677c3d90-53bf-11ec-99c7-8a27c0e85aa9
|
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: STATE EXCHANGE: got state msg: 677c3d90-53bf-11ec-99c7-8a27c0e85aa9 from 0 (zabbix-db04)
|
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: STATE EXCHANGE: got state msg: 677c3d90-53bf-11ec-99c7-8a27c0e85aa9 from 2 (zabbix-db06)
|
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: STATE EXCHANGE: got state msg: 677c3d90-53bf-11ec-99c7-8a27c0e85aa9 from 1 (zabbix-db05)
|
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: Quorum results:
|
Dec 02 22:30:08 mysqld[2095962]: version = 6,
|
Dec 02 22:30:08 mysqld[2095962]: component = PRIMARY,
|
Dec 02 22:30:08 mysqld[2095962]: conf_id = 48,
|
Dec 02 22:30:08 mysqld[2095962]: members = 2/3 (joined/total),
|
Dec 02 22:30:08 mysqld[2095962]: act_id = 4857698808,
|
Dec 02 22:30:08 mysqld[2095962]: last_appl. = 4857698803,
|
Dec 02 22:30:08 mysqld[2095962]: protocols = 2/10/4 (gcs/repl/appl),
|
Dec 02 22:30:08 mysqld[2095962]: vote policy= 0,
|
Dec 02 22:30:08 mysqld[2095962]: group UUID = 810a5d0c-231c-11e9-b756-dbb7899d7dd0
|
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
|
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 4857698809)
|
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 2 [Note] WSREP: ####### processing CC 4857698809, local, ordered
|
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 2 [Note] WSREP: Process first view: 810a5d0c-231c-11e9-b756-dbb7899d7dd0 my uuid: 2f8c7fd2-5334-11ec-91e7-46483b39372d
|
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 2 [Note] WSREP: Server zabbix-db05 connected to cluster at position 810a5d0c-231c-11e9-b756-dbb7899d7dd0:4857698809 with ID 2f8c7fd2-5334-11ec-91e7-46483b39372d
|
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 2 [Note] WSREP: Server status change disconnected -> connected
|
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
|
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 2 [Note] WSREP: ####### My UUID: 2f8c7fd2-5334-11ec-91e7-46483b39372d
|
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 2 [Note] WSREP: Cert index reset to 00000000-0000-0000-0000-000000000000:-1 (proto: 10), state transfer needed: yes
|
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: Service thread queue flushed.
|
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 2 [Note] WSREP: ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: -1
|
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 2 [Note] WSREP: State transfer required:
|
Dec 02 22:30:08 mysqld[2095962]: Group state: 810a5d0c-231c-11e9-b756-dbb7899d7dd0:4857698809
|
Dec 02 22:30:08 mysqld[2095962]: Local state: 810a5d0c-231c-11e9-b756-dbb7899d7dd0:4857087468
|
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 2 [Note] WSREP: Server status change connected -> joiner
|
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
|
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: Joiner monitor thread started to monitor
|
Dec 02 22:30:08 mysqld[2095962]: 2021-12-02 22:30:08 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'joiner' --address '192.168.77.161' --datadir '/var/lib/mysql/' --parent '2095962' --mysqld-args --wsrep_start_position=810a5d0c-231c-11e9-b756-dbb7899d7dd0:4857087468'
|
Dec 02 22:30:08 mysqld[2096028]: WSREP_SST: [INFO] SSL configuration: CA='', CERT='', KEY='', MODE='DISABLED', encrypt='0' (20211202 22:30:08.213)
|
Dec 02 22:30:08 mysqld[2096028]: WSREP_SST: [INFO] Logging all stderr of SST/mariabackup to syslog (20211202 22:30:08.284)
|
Dec 02 22:30:08 -wsrep-sst-joiner[2096168]: Using socat as streamer
|
Dec 02 22:30:09 mysqld[2095962]: 2021-12-02 22:30:09 2 [Note] WSREP: ####### IST uuid:810a5d0c-231c-11e9-b756-dbb7899d7dd0 f: 4857087469, l: 4857698809, STRv: 3
|
Dec 02 22:30:09 mysqld[2095962]: 2021-12-02 22:30:09 2 [Note] WSREP: IST receiver addr using tcp://192.168.77.161:4568
|
Dec 02 22:30:09 mysqld[2095962]: 2021-12-02 22:30:09 2 [Note] WSREP: Prepared IST receiver for 4857087469-4857698809, listening at: tcp://192.168.77.161:4568
|
Dec 02 22:30:09 mysqld[2095962]: 2021-12-02 22:30:09 0 [Note] WSREP: Member 1.0 (zabbix-db05) requested state transfer from 'zabbix-db04'. Selected 0.0 (zabbix-db04)(SYNCED) as donor.
|
Dec 02 22:30:09 mysqld[2095962]: 2021-12-02 22:30:09 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 4857698990)
|
Dec 02 22:30:09 mysqld[2095962]: 2021-12-02 22:30:09 2 [Note] WSREP: Requesting state transfer: success, donor: 0
|
Dec 02 22:30:09 mysqld[2095962]: 2021-12-02 22:30:09 0 [Note] WSREP: 0.0 (zabbix-db04): State transfer to 1.0 (zabbix-db05) complete.
|
Dec 02 22:30:09 mysqld[2095962]: 2021-12-02 22:30:09 0 [Note] WSREP: Member 0.0 (zabbix-db04) synced with group.
|
Dec 02 22:30:09 -wsrep-sst-joiner[2096217]: Removing the sst_in_progress file
|
Dec 02 22:30:09 mysqld[2095962]: 2021-12-02 22:30:09 3 [Note] WSREP: SST received
|
Dec 02 22:30:09 mysqld[2095962]: 2021-12-02 22:30:09 3 [Note] WSREP: Server status change joiner -> initializing
|
Dec 02 22:30:09 mysqld[2095962]: 2021-12-02 22:30:09 3 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
|
Dec 02 22:30:09 mysqld[2095962]: 2021-12-02 22:30:09 0 [Note] InnoDB: Using Linux native AIO
|
Dec 02 22:30:09 mysqld[2095962]: 2021-12-02 22:30:09 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
|
Dec 02 22:30:09 mysqld[2095962]: 2021-12-02 22:30:09 0 [Note] InnoDB: Uses event mutexes
|
Dec 02 22:30:09 mysqld[2095962]: 2021-12-02 22:30:09 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
|
Dec 02 22:30:09 mysqld[2095962]: 2021-12-02 22:30:09 0 [Note] InnoDB: Number of pools: 1
|
Dec 02 22:30:09 mysqld[2095962]: 2021-12-02 22:30:09 0 [Note] InnoDB: Using SSE2 crc32 instructions
|
Dec 02 22:30:09 mysqld[2095962]: 2021-12-02 22:30:09 0 [Note] InnoDB: Initializing buffer pool, total size = 1087G, instances = 1, chunk size = 128M
|
Dec 02 22:30:10 mysqld[2095962]: 2021-12-02 22:30:10 0 [Note] WSREP: (2f8c7fd2-91e7, 'tcp://0.0.0.0:4567') turning message relay requesting off
|
Dec 02 22:30:49 mysqld[2095962]: 2021-12-02 22:30:49 0 [Note] InnoDB: Completed initialization of buffer pool
|
Dec 02 22:30:49 mysqld[2095962]: 2021-12-02 22:30:49 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
|
Dec 02 22:30:49 mysqld[2095962]: 2021-12-02 22:30:49 0 [Note] InnoDB: Setting log file ./ib_logfile101 size to 6442450944 bytes
|
Dec 02 22:30:50 mysqld[2095962]: 2021-12-02 22:30:50 0 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0
|
Dec 02 22:30:50 mysqld[2095962]: 2021-12-02 22:30:50 0 [Note] InnoDB: New log files created, LSN=192780380073929
|
Dec 02 22:30:50 mysqld[2095962]: 2021-12-02 22:30:50 0 [Note] InnoDB: 47 transaction(s) which must be rolled back or cleaned up in total 20286 row operations to undo
|
Dec 02 22:30:50 mysqld[2095962]: 2021-12-02 22:30:50 0 [Note] InnoDB: Trx id counter is 9733845087
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] InnoDB: Starting in background the rollback of recovered transactions
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] InnoDB: Creating shared tablespace for temporary tables
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] InnoDB: Waiting for purge to start
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] InnoDB: Rolled back recovered transaction 9733845042
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] InnoDB: Rolled back recovered transaction 9733845058
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] InnoDB: Rolled back recovered transaction 9733845034
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] InnoDB: 10.4.22 started; log sequence number 192780380073996; transaction id 9733845091
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] Plugin 'FEEDBACK' is disabled.
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] Server socket created on IP: '0.0.0.0'.
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] WSREP: wsrep_init_schema_and_SR (nil)
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] WSREP: Server initialized
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] WSREP: Server status change initializing -> initialized
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 3 [Note] WSREP: Server status change initialized -> joined
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 3 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 3 [Note] WSREP: Recovered position from storage: 810a5d0c-231c-11e9-b756-dbb7899d7dd0:4857087468
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 3 [Note] WSREP: Recovered view from SST:
|
Dec 02 22:30:59 mysqld[2095962]: id: 810a5d0c-231c-11e9-b756-dbb7899d7dd0:4850155776
|
Dec 02 22:30:59 mysqld[2095962]: status: primary
|
Dec 02 22:30:59 mysqld[2095962]: protocol_version: 4
|
Dec 02 22:30:59 mysqld[2095962]: capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
|
Dec 02 22:30:59 mysqld[2095962]: final: no
|
Dec 02 22:30:59 mysqld[2095962]: own_index: -1
|
Dec 02 22:30:59 mysqld[2095962]: members(3):
|
Dec 02 22:30:59 mysqld[2095962]: 0: 1e77c150-4403-11ec-8d79-83292a6d14d2, zabbix-db04
|
Dec 02 22:30:59 mysqld[2095962]: 1: 2f8c7fd2-5334-11ec-91e6-46483b39372d, zabbix-db05
|
Dec 02 22:30:59 mysqld[2095962]: 2: 58fa90ba-483b-11ec-871c-6b4c25d26355, zabbix-db06
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 3 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 14 [Note] WSREP: Starting applier thread 14
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 15 [Note] WSREP: Recovered cluster id 810a5d0c-231c-11e9-b756-dbb7899d7dd0
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 3 [Note] WSREP: SST received: 810a5d0c-231c-11e9-b756-dbb7899d7dd0:4857087468
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 3 [Note] WSREP: SST succeeded for position 810a5d0c-231c-11e9-b756-dbb7899d7dd0:4857087468
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] WSREP: Joiner monitor thread ended with total time 51 sec
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 2 [Note] WSREP: Installed new state from SST: 810a5d0c-231c-11e9-b756-dbb7899d7dd0:4857087468
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 2 [Note] WSREP: Receiving IST: 611341 writesets, seqnos 4857087469-4857698809
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] WSREP: ####### IST applying starts with 4857087469
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 31 [Note] WSREP: Starting applier thread 31
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 32 [Note] WSREP: Starting applier thread 32
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 42 [Note] WSREP: Starting applier thread 42
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 46 [Note] WSREP: Starting applier thread 46
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 21 [Note] WSREP: Starting applier thread 21
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 33 [Note] WSREP: Starting applier thread 33
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 64 [Note] WSREP: Starting applier thread 64
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] WSREP: ####### IST current seqno initialized to 4857087469
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 45 [Note] WSREP: Starting applier thread 45
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 48 [Note] WSREP: Starting applier thread 48
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 72 [Note] WSREP: Starting applier thread 72
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 53 [Note] WSREP: Starting applier thread 53
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 29 [Note] WSREP: Starting applier thread 29
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 47 [Note] WSREP: Starting applier thread 47
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 61 [Note] WSREP: Starting applier thread 61
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 30 [Note] WSREP: Starting applier thread 30
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 25 [Note] WSREP: Starting applier thread 25
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 65 [Note] WSREP: Starting applier thread 65
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 49 [Note] WSREP: Starting applier thread 49
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 19 [Note] WSREP: Starting applier thread 19
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 73 [Note] WSREP: Starting applier thread 73
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 70 [Note] WSREP: Starting applier thread 70
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 16 [Note] WSREP: Starting applier thread 16
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 28 [Note] WSREP: Starting applier thread 28
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 10 [Note] WSREP: Starting applier thread 10
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 52 [Note] WSREP: Starting applier thread 52
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 12 [Note] WSREP: Starting applier thread 12
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 23 [Note] WSREP: Starting applier thread 23
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 54 [Note] WSREP: Starting applier thread 54
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 17 [Note] WSREP: Starting applier thread 17
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 55 [Note] WSREP: Starting applier thread 55
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 35 [Note] WSREP: Starting applier thread 35
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 56 [Note] WSREP: Starting applier thread 56
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 24 [Note] WSREP: Starting applier thread 24
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 57 [Note] WSREP: Starting applier thread 57
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 34 [Note] WSREP: Starting applier thread 34
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 58 [Note] WSREP: Starting applier thread 58
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 36 [Note] WSREP: Starting applier thread 36
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 18 [Note] WSREP: Starting applier thread 18
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 59 [Note] WSREP: Starting applier thread 59
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 20 [Note] WSREP: Starting applier thread 20
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 60 [Note] WSREP: Starting applier thread 60
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 37 [Note] WSREP: Starting applier thread 37
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 38 [Note] WSREP: Starting applier thread 38
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 13 [Note] WSREP: Starting applier thread 13
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 39 [Note] WSREP: Starting applier thread 39
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 62 [Note] WSREP: Starting applier thread 62
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 63 [Note] WSREP: Starting applier thread 63
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 40 [Note] WSREP: Starting applier thread 40
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 22 [Note] WSREP: Starting applier thread 22
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 11 [Note] WSREP: Starting applier thread 11
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 66 [Note] WSREP: Starting applier thread 66
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] WSREP: Receiving IST... 0.0% ( 0/611341 events) complete.
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 67 [Note] WSREP: Starting applier thread 67
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 68 [Note] WSREP: Starting applier thread 68
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 26 [Note] WSREP: Starting applier thread 26
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 41 [Note] WSREP: Starting applier thread 41
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 69 [Note] WSREP: Starting applier thread 69
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 50 [Note] WSREP: Starting applier thread 50
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 27 [Note] WSREP: Starting applier thread 27
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 43 [Note] WSREP: Starting applier thread 43
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 51 [Note] WSREP: Starting applier thread 51
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 71 [Note] WSREP: Starting applier thread 71
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 44 [Note] WSREP: Starting applier thread 44
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] WSREP: Service thread queue flushed.
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] WSREP: ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:4857087468, protocol version: 5
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] Reading of all Master_info entries succeeded
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] Added new Master_info '' to hash table
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] /usr/sbin/mysqld: ready for connections.
|
Dec 02 22:30:59 mysqld[2095962]: Version: '10.4.22-MariaDB-1:10.4.22+maria~focal-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 21 [Warning] WSREP: no THD for trx: 9733845039
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 14 [Warning] WSREP: no THD for trx: 9733845033
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 42 [Warning] WSREP: no THD for trx: 9733845037
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 69 [Warning] WSREP: no THD for trx: 9733845076
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 66 [Warning] WSREP: no THD for trx: 9733845073
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 50 [Warning] WSREP: no THD for trx: 9733845077
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 44 [Warning] WSREP: no THD for trx: 9733845080
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 63 [Warning] WSREP: no THD for trx: 9733845069
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 41 [Warning] WSREP: no THD for trx: 9733845075
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 34 [Warning] WSREP: no THD for trx: 9733845065
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 67 [Warning] WSREP: no THD for trx: 9733845072
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 48 [Warning] WSREP: no THD for trx: 9733845043
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 64 [Warning] WSREP: no THD for trx: 9733845041
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 10 [Warning] WSREP: no THD for trx: 9733845056
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 28 [Warning] WSREP: no THD for trx: 9733845055
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 47 [Warning] WSREP: no THD for trx: 9733845047
|
Dec 02 22:30:59 systemd[1]: Started MariaDB 10.4.22 database server.
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 40 [Warning] WSREP: no THD for trx: 9733845070
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 39 [Warning] WSREP: no THD for trx: 9733845067
|
Dec 02 22:30:59 /etc/mysql/debian-start[2096564]: Upgrading MySQL tables if necessary.
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 62 [Warning] WSREP: no THD for trx: 9733845068
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 49 [Warning] WSREP: no THD for trx: 9733845052
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 31 [Warning] WSREP: no THD for trx: 9733845035
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 17 [Warning] WSREP: no THD for trx: 9733845060
|
Dec 02 22:30:59 /etc/mysql/debian-start[2096567]: /usr/bin/mysql_upgrade: the '--basedir' option is always ignored
|
Dec 02 22:30:59 /etc/mysql/debian-start[2096567]: Looking for 'mysql' as: /usr/bin/mysql
|
Dec 02 22:30:59 /etc/mysql/debian-start[2096567]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
|
Dec 02 22:30:59 /etc/mysql/debian-start[2096567]: This installation of MariaDB is already upgraded to 10.4.22-MariaDB, use --force if you still need to run mysql_upgrade
|
Dec 02 22:30:59 /etc/mysql/debian-start[2096575]: Checking for insecure root accounts.
|
Dec 02 22:30:59 mysqld[2096578]: ERROR 1047 (08S01) at line 1: WSREP has not yet prepared node for application use
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 29 [Warning] WSREP: no THD for trx: 9733845045
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 30 [Warning] WSREP: no THD for trx: 9733845049
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] InnoDB: Rolled back recovered transaction 9733845066
|
Dec 02 22:30:59 mysqld[2095962]: 2021-12-02 22:30:59 0 [Note] InnoDB: Rolled back recovered transaction 9733845050
|
Dec 02 22:31:00 mysqld[2095962]: 2021-12-02 22:31:00 0 [Note] InnoDB: Rolled back recovered transaction 9733845078
|
Dec 02 22:31:00 mysqld[2095962]: 2021-12-02 22:31:00 0 [Note] InnoDB: Rolled back recovered transaction 9733845046
|
Dec 02 22:31:00 mysqld[2095962]: 2021-12-02 22:31:00 0 [Note] InnoDB: Rolled back recovered transaction 9733845062
|
Dec 02 22:31:00 mysqld[2095962]: 2021-12-02 22:31:00 0 [Note] InnoDB: Rolled back recovered transaction 9733845070
|
Dec 02 22:31:00 mysqld[2095962]: 2021-12-02 22:31:00 0 [Note] InnoDB: Rolled back recovered transaction 9733845038
|
Dec 02 22:31:00 mysqld[2095962]: 2021-12-02 22:31:00 0 [Note] InnoDB: Rolled back recovered transaction 9733845054
|
Dec 02 22:31:00 mysqld[2095962]: 2021-12-02 22:31:00 0 [Note] InnoDB: Rolled back recovered transaction 9733845040
|
Dec 02 22:31:00 mysqld[2095962]: 2021-12-02 22:31:00 0 [Note] InnoDB: Rolled back recovered transaction 9733845072
|
Dec 02 22:31:00 mysqld[2095962]: 2021-12-02 22:31:00 0 [Note] InnoDB: Rolled back recovered transaction 9733845056
|
Dec 02 22:31:00 mysqld[2095962]: 2021-12-02 22:31:00 0 [Note] InnoDB: Rolled back recovered transaction 9733845080
|
Dec 02 22:31:00 mysqld[2095962]: 2021-12-02 22:31:00 0 [Note] InnoDB: Rolled back recovered transaction 9733845048
|
Dec 02 22:31:00 mysqld[2095962]: 2021-12-02 22:31:00 0 [Note] InnoDB: Rolled back recovered transaction 9733845064
|
Dec 02 22:31:00 mysqld[2095962]: 2021-12-02 22:31:00 0 [Note] InnoDB: Rolled back recovered transaction 9733845060
|
Dec 02 22:31:00 mysqld[2095962]: 2021-12-02 22:31:00 0 [Note] InnoDB: Rolled back recovered transaction 9733845076
|
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845044
|
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845068
|
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845036
|
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845052
|
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845035
|
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845067
|
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845051
|
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845043
|
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845075
|
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845059
|
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845079
|
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845047
|
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845063
|
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845055
|
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845071
|
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845039
|
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845073
|
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845041
|
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845057
|
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845049
|
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845065
|
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845033
|
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845053
|
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845037
|
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845069
|
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845061
|
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845045
|
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rolled back recovered transaction 9733845077
|
Dec 02 22:31:01 mysqld[2095962]: 2021-12-02 22:31:01 0 [Note] InnoDB: Rollback of non-prepared transactions completed
|
Dec 02 22:31:01 mysqld[2095962]: mysqld: /home/buildbot/buildbot/build/mariadb-10.4.22/wsrep-lib/include/wsrep/client_state.hpp:668: int wsrep::client_state::bf_abort(wsrep::seqno): Assertion `mode_ == m_local || transaction_.is_streaming()' failed.
|
Dec 02 22:31:01 mysqld[2095962]: 211202 22:31:01 [ERROR] mysqld got signal 6 ;
|
Dec 02 22:31:01 mysqld[2095962]: This could be because you hit a bug. It is also possible that this binary
|
Dec 02 22:31:01 mysqld[2095962]: or one of the libraries it was linked against is corrupt, improperly built,
|
Dec 02 22:31:01 mysqld[2095962]: or misconfigured. This error can also be caused by malfunctioning hardware.
|
Dec 02 22:31:01 mysqld[2095962]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
Dec 02 22:31:01 mysqld[2095962]: We will try our best to scrape up some info that will hopefully help
|
Dec 02 22:31:01 mysqld[2095962]: diagnose the problem, but since we have already crashed,
|
Dec 02 22:31:01 mysqld[2095962]: something is definitely wrong and this may fail.
|
Dec 02 22:31:01 mysqld[2095962]: Server version: 10.4.22-MariaDB-1:10.4.22+maria~focal-log
|
Dec 02 22:31:01 mysqld[2095962]: key_buffer_size=134217728
|
Dec 02 22:31:01 mysqld[2095962]: read_buffer_size=131072
|
Dec 02 22:31:01 mysqld[2095962]: max_used_connections=1
|
Dec 02 22:31:01 mysqld[2095962]: max_threads=403
|
Dec 02 22:31:01 mysqld[2095962]: thread_count=71
|
Dec 02 22:31:01 mysqld[2095962]: It is possible that mysqld could use up to
|
Dec 02 22:31:01 mysqld[2095962]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6795376 K bytes of memory
|
Dec 02 22:31:01 mysqld[2095962]: Hope that's ok; if not, decrease some variables in the equation.
|
Dec 02 22:31:01 mysqld[2095962]: Thread pointer: 0x7e01d8000c48
|
Dec 02 22:31:01 mysqld[2095962]: Attempting backtrace. You can use the following information to find out
|
Dec 02 22:31:01 mysqld[2095962]: where mysqld died. If you see no messages after this, something went
|
Dec 02 22:31:01 mysqld[2095962]: terribly wrong...
|
Dec 02 22:31:01 mysqld[2095962]: stack_bottom = 0x7f12b01dce08 thread_stack 0x49000
|
Dec 02 22:31:04 mysqld[2095962]: /usr/sbin/mysqld(my_print_stacktrace+0x32)[0x55c6369f7d12]
|
Dec 02 22:31:04 mysqld[2095962]: /usr/sbin/mysqld(handle_fatal_signal+0x55d)[0x55c6364a9a5d]
|
Dec 02 22:31:05 mysqld[2095962]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0)[0x7f338c51f3c0]
|
Dec 02 22:31:09 mysqld[2095962]: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x7f338c02518b]
|
Dec 02 22:31:09 mysqld[2095962]: /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b)[0x7f338c004859]
|
Dec 02 22:31:09 mysqld[2095962]: /lib/x86_64-linux-gnu/libc.so.6(+0x25729)[0x7f338c004729]
|
Dec 02 22:31:09 mysqld[2095962]: /lib/x86_64-linux-gnu/libc.so.6(+0x36f36)[0x7f338c015f36]
|
Dec 02 22:31:10 mysqld[2095962]: /usr/sbin/mysqld(_Z14wsrep_bf_abortP3THDS0_+0x6e9)[0x55c6364247f9]
|
Dec 02 22:31:11 mysqld[2095962]: /usr/sbin/mysqld(wsrep_thd_bf_abort+0x1d)[0x55c63642c6bd]
|
Dec 02 22:31:11 mysqld[2095962]: /usr/sbin/mysqld(+0xaab1b6)[0x55c6366621b6]
|
Dec 02 22:31:11 mysqld[2095962]: /usr/sbin/mysqld(+0xb0cd84)[0x55c6366c3d84]
|
Dec 02 22:31:11 mysqld[2095962]: /usr/sbin/mysqld(+0xb12e62)[0x55c6366c9e62]
|
Dec 02 22:31:11 mysqld[2095962]: /usr/sbin/mysqld(+0xb9fdd1)[0x55c636756dd1]
|
Dec 02 22:31:11 mysqld[2095962]: /usr/sbin/mysqld(+0xac65b1)[0x55c63667d5b1]
|
Dec 02 22:31:11 mysqld[2095962]: /usr/sbin/mysqld(_ZN7handler10ha_rnd_posEPhS0_+0x50)[0x55c6364b0690]
|
Dec 02 22:31:11 mysqld[2095962]: /usr/sbin/mysqld(+0x818150)[0x55c6363cf150]
|
Dec 02 22:31:11 mysqld[2095962]: /usr/sbin/mysqld(_ZN14Rows_log_event8find_rowEP14rpl_group_info+0x451)[0x55c6365c5f91]
|
Dec 02 22:31:11 mysqld[2095962]: /usr/sbin/mysqld(_ZN21Delete_rows_log_event11do_exec_rowEP14rpl_group_info+0x92)[0x55c6365c6252]
|
Dec 02 22:31:11 mysqld[2095962]: /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x23f)[0x55c6365b975f]
|
Dec 02 22:31:11 mysqld[2095962]: /usr/sbin/mysqld(_Z18wsrep_apply_eventsP3THDP14Relay_log_infoPKvm+0x1b9)[0x55c636423979]
|
Dec 02 22:31:11 mysqld[2095962]: /usr/sbin/mysqld(_ZN21Wsrep_applier_service15apply_write_setERKN5wsrep7ws_metaERKNS0_12const_bufferERNS0_14mutable_bufferE+0xa3)[0x55c63640afc3]
|
Dec 02 22:31:11 mysqld[2095962]: /usr/sbin/mysqld(+0xeaf580)[0x55c636a66580]
|
Dec 02 22:31:11 mysqld[2095962]: /usr/sbin/mysqld(_ZN5wsrep12server_state8on_applyERNS_21high_priority_serviceERKNS_9ws_handleERKNS_7ws_metaERKNS_12const_bufferE+0xc1)[0x55c636a67441]
|
Dec 02 22:31:11 mysqld[2095962]: /usr/sbin/mysqld(+0xec187c)[0x55c636a7887c]
|
Dec 02 22:31:12 mysqld[2095962]: /usr/lib/libgalera_smm.so(+0x639eb)[0x7f338b2939eb]
|
Dec 02 22:31:13 mysqld[2095962]: /usr/lib/libgalera_smm.so(+0x7582e)[0x7f338b2a582e]
|
Dec 02 22:31:13 mysqld[2095962]: src/trx_handle.cpp:391(galera::TrxHandleSlave::apply(void*, wsrep_cb_status (*)(void*, wsrep_ws_handle const*, unsigned int, wsrep_buf const*, wsrep_trx_meta const*, bool*), wsrep_trx_meta const&, bool&))[0x7f338b2bcdc4]
|
Dec 02 22:31:13 mysqld[2095962]: src/replicator_smm.cpp:504(galera::ReplicatorSMM::apply_trx(void*, galera::TrxHandleSlave&))[0x7f338b2bd462]
|
Dec 02 22:31:14 mysqld[2095962]: src/replicator_str.cpp:1131(galera::ReplicatorSMM::process_IST_writeset(void*, boost::shared_ptr<galera::TrxHandleSlave> const&))[0x7f338b2a8ea3]
|
Dec 02 22:31:14 mysqld[2095962]: src/replicator_smm.cpp:396(galera::ReplicatorSMM::async_recv(void*))[0x7f338b27efe1]
|
Dec 02 22:31:15 mysqld[2095962]: /usr/sbin/mysqld(_ZN5wsrep18wsrep_provider_v2611run_applierEPNS_21high_priority_serviceE+0x12)[0x55c636a78e92]
|
Dec 02 22:31:15 mysqld[2095962]: /usr/sbin/mysqld(+0x86e7c3)[0x55c6364257c3]
|
Dec 02 22:31:16 mysqld[2095962]: /usr/sbin/mysqld(_Z15start_wsrep_THDPv+0x2ec)[0x55c636415f1c]
|
Dec 02 22:31:17 mysqld[2095962]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609)[0x7f338c513609]
|
Dec 02 22:31:20 mysqld[2095962]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x43)[0x7f338c101293]
|
Dec 02 22:31:20 mysqld[2095962]: Trying to get some variables.
|
Dec 02 22:31:20 mysqld[2095962]: Some pointers may be invalid and cause the dump to abort.
|
Dec 02 22:31:20 mysqld[2095962]: Query (0x7f15f1893f5b): delete from problem where r_clock<>0 and r_clock<1638393301
|
Dec 02 22:31:20 mysqld[2095962]: Connection ID (thread ID): 14
|
Dec 02 22:31:20 mysqld[2095962]: Status: NOT_KILLED
|
Dec 02 22:31:20 mysqld[2095962]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on
|
Dec 02 22:31:20 mysqld[2095962]: The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
|
Dec 02 22:31:20 mysqld[2095962]: information that should help you find out what is causing the crash.
|
Dec 02 22:31:20 mysqld[2095962]: Writing a core file...
|
Dec 02 22:31:20 mysqld[2095962]: Working directory at /var/lib/mysql
|
Dec 02 22:31:20 mysqld[2095962]: Resource Limits:
|
Dec 02 22:31:20 mysqld[2095962]: Limit Soft Limit Hard Limit Units
|
Dec 02 22:31:20 mysqld[2095962]: Max cpu time unlimited unlimited seconds
|
Dec 02 22:31:20 mysqld[2095962]: Max file size unlimited unlimited bytes
|
Dec 02 22:31:20 mysqld[2095962]: Max data size unlimited unlimited bytes
|
Dec 02 22:31:20 mysqld[2095962]: Max stack size 8388608 unlimited bytes
|
Dec 02 22:31:20 mysqld[2095962]: Max core file size 0 unlimited bytes
|
Dec 02 22:31:20 mysqld[2095962]: Max resident set unlimited unlimited bytes
|
Dec 02 22:31:20 mysqld[2095962]: Max processes 5931800 5931800 processes
|
Dec 02 22:31:20 mysqld[2095962]: Max open files 65536 65536 files
|
Dec 02 22:31:20 mysqld[2095962]: Max locked memory 65536 65536 bytes
|
Dec 02 22:31:20 mysqld[2095962]: Max address space unlimited unlimited bytes
|
Dec 02 22:31:20 mysqld[2095962]: Max file locks unlimited unlimited locks
|
Dec 02 22:31:20 mysqld[2095962]: Max pending signals 5931800 5931800 signals
|
Dec 02 22:31:20 mysqld[2095962]: Max msgqueue size 819200 819200 bytes
|
Dec 02 22:31:20 mysqld[2095962]: Max nice priority 0 0
|
Dec 02 22:31:20 mysqld[2095962]: Max realtime priority 0 0
|
Dec 02 22:31:20 mysqld[2095962]: Max realtime timeout unlimited unlimited us
|
Dec 02 22:31:20 mysqld[2095962]: Core pattern: /var/crash/%e.%p
|
Dec 02 22:31:28 systemd[1]: mariadb.service: Main process exited, code=killed, status=6/ABRT
|
Dec 02 22:31:28 systemd[1]: mariadb.service: Failed with result 'signal'.
|
And again this happened on 5 Dec. I run a three-node Galera cluster with all connections routed to one node (the primary). Early on 5 Dec both "backup" nodes at exactly the same time (to within the one-second resolution in the logs) aborted with:
Dec 05 01:15:15 mysqld[687955]: mysqld: /home/buildbot/buildbot/build/mariadb-10.4.22/wsrep-lib/include/wsrep/client_state.hpp:668: int wsrep::client_state::bf_abort(wsrep::seqno): Assertion `mode_ == m_local || transaction_.is_streaming()' failed.
|
Dec 05 01:15:15 mysqld[687955]: 211205 1:15:15 [ERROR] mysqld got signal 6 ;
|
Of course, with both backup nodes failing the primary node loses quorum, leaving me dead in the water with no functioning database.
I cannot stress enough how critical an issue this is. My Galera cluster is a mission-critical, production system and for the past month+ (since upgrading to 10.4.19, then 10.4.21, and now with 10.4.22) the cluster is dramatically unstable. Every day is another opportunity for yet another crash.
Whatever I can do to help debug I will do. Do you want logs? I will provide them. Do you need extra or different logging? I will reconfigure my nodes to provide that. Just let me know! Thanks.
mreibert Do you get core dump? If you get can you open it on debugger by using MariaDB debug symbols package and run command thread apply all bt and provide output from that. Do you know what kind of query causes crash? If you do please provide affected table structure. Do you have tables with foreign keys using charset utf8 ?
jplindst — Unfortunately cores are disabled on my DB nodes; I will change that so next time it crashes I get the core.
Regarding the offending query, that too I do not have. This database backs a large and very busy Zabbix installation and the Zabbix servers are responsible for the queries. Is there some way to log and correlate those queries to the crash?
Regarding the charset, the Zabbix database on this cluster is latin1.
jplindst — I should mention there is also a database backing Grafana on this cluster, and that database is utf8mb4. However, whereas there are indexes on varchar columns I do not see a single FK constraint in this database.
jplindst — I received another SIGABRT and captured the core dump. The stack trace you requested is attached in reibert_mysqld-abort-stacktrace.txt
and the journal log is attached in reibert_mariadb.service.log.
Here is the scenario that lead to the assertion failure. The node (one of three in the cluster) was being brought online via a SST using mariabackup. The SST transferred normally (you'll see that took 6.5 hours; yes that is normal for this 9.5TB database). After the transfer the mariabackup "prepare" (apply) and "move" phases completed normally. (These took nearly an hour; again that is normal for this large database.) The server then comes online and begins the delta IST, which itself can be large since nearly 8 hours have passed since the SST began. Not long into that the mode_ == m_local || transaction_.is_streaming() assertion fails with the resulting SIGABRT.
Interestingly, near the end of the log we see a delete from events ... query; seeing a query in the log is actually not common. Could this be the query that triggered the assertion failure? That seems at least plausible since the stack trace indicates the assertion failure occurred when deleting a row in a writeset.
Please do not hesitate to ask if there is any other information I can provide.
Thank you. Your stack trace contains row_upd_check_references_constraint so there is foreign keys also in this case.
There are two databases on this cluster, one for Zabbix and one for Grafana. The Zabbix DB does have FKs, but that DB is entirely latin1. The Grafana DB is utf8mb4, but has no FKs.
The load on this cluster is dominated by the Zabbix DB, both in size and query frequency/volume. I suspect the problem is related to that, so I am not surprised there is a FK reference in the stack trace.
mreibert Can you try with provider configuration: cert.optimistic_pa=false ?
jplindst — Yes, I will try that.
In the mean time, I had another SIGABRT. The stack trace is attached in reibert_mysqld-abort-stacktrace-2.txt and the journal log (from server startup to stop) is attached in reibert_mariadb.service-2.log.
The scenario for this crash is a little different. This time I had only two nodes up and running. (The node that crashed several days ago is still down.) Nothing "special" was going on with the nodes; they had both been up for several days. So this was not an IST-triggered event but rather just a "normal" writeset application. The stack trace indicates there was a record lock conflict on a delete row. In that vein, I should note I have wsrep_slave_threads = 64 on these nodes.
On a potentially related topic, should I set wsrep_certification_rules = optimized ? From the documentation it seems that would have an effect only for a cluster that has multiple nodes taking writes (which is not my case; all writes are directed to one node), however Percona sets that by default on their distribution since it is "safer". (Percona also sets cert.optimistic_pa = no by default.)
mreibert I would not set wsrep_certification_rules = optimized in your case.
jplindst — thanks.
In other news, I now have cert.optimistic_pa = no on all (three) cluster nodes. With that setting I was able to SST the two down nodes back into the cluster with no errors or issues. Whether or not that is an actual fix, however, only time will tell. (Recall from my last core dump I had two nodes up for three days before one experienced the assertion failure "out of the blue".)
Keeping my fingers crossed …
mreibert your first stacktrace attached (reibert_mysqld-abort-stacktrace.txt) has one replication applier thread (Thread 73) in the middle of processing of Foreign key constraint's cascade operation (row_ins_foreign_check_on_constraint). This suggest that this crash relates to issue MDEV-26803, where the reason for the crashes was that cascading executions did not reliably record all rows manipulated through cascade operation in the replication write set. And this has resulted in unsafe parallel applying in replica nodes.
The second stacktrace has also ongoing foreign key checking, but in different stages.
seppo — Thank you for the update and reference to MDEV-26803. I see in that issue setting cert.optimistic_pa = no is a suggested workaround, as has been discussed in comments here. With the fix for MDEV-26803 in 10.4.23, would it be considered "safe" to return to cert.optimistic_pa = yes? To ask the question a little differently, is there a downside to staying with cert.optimistic_pa = no? After 1–2 months of very frequent crashes with 10.4.19–22, I have been rock solid for the past week after setting cert.optimistic_pa = no. So i am (I think understandably) hesitant to turn that back on … ever. 
mreibert "cert.optimistic_pa = no" configuration is a viable workaround for this and also as permanent configuration, if replication performance remains in acceptable level.
The "cert.optimistic_pa = no" configuration has global effect on all transactions, and will limit the concurrency of replication applying in replica nodes. Whereas, the fix for MDEV-26803, will monitor the transaction's foreign key constraint processing in the first node, and only when cascading delete operation really takes place, then that specific transaction only is marked as not safe for parallel applying.
Your stack trace contains at least one ongoing FK cascade delete operation, and this makes it plausible candidate for MDEV-26803 scenario.
seppo — thank you for the feedback.
At present my replication performance is holding fine with cert.optimistic_pa = no and I haven't experienced a single SIGABRT/SIGSEGV failure since making that change. So I will leave that disabled in my configuration. But if I do at some point start seeing replication fall behind I will consider re-enabling cert.optimistic_pa (after upgrading to a version that contains MDEV-26803, of course).
@Mark Reibert, we are experiencing the same issue from time to time and would like to set cert.optimistic_pa = no, however I'm not sure how to proceed safely applying this as a permanent configuration ?
Do I need to apply the setting on all nodes or can one node be set for testing?
Is adding such a line in /etc/mysql/mariadb.conf.d/50-server.cnf ok:
wsrep_provider_options = "cert.optimistic_pa = no" |
Or should I copy/paste all existing values from an already running node, which looks like below, with a lot of options set:
wsrep_provider_options base_dir = /var/lib/mysql/; base_host = 192.168.0.6; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.causal_keepalive_period = PT1S; evs.debug_log_mask = 0x1; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.info_log_mask = 0; evs.install_timeout = PT7.5S; evs.join_retrans_period = PT1S; evs.keepalive_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.use_aggregate = true; evs.user_send_window = 2; evs.version = 1; evs.view_forget_timeout = P1D; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.keep_plaintext_size = 128M; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.fc_single_primary = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.listen_addr = tcp://0.0.0.0:4567; gmcast.mcast_addr = ; gmcast.mcast_ttl = 1; gmcast.peer_timeout = PT3S; gmcast.segment = 0; gmcast.time_wait = PT5S; gmcast.version = 0; ist.recv_addr = 192.168.0.6; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.linger = PT20S; pc.npvo = false; pc.recovery = true; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = PT30S; pc.weight = 1; protonet.version = 0; repl.causal_read_timeout = PT30S; repl.commit_order = 3; repl.key_format = FLAT8; repl.max_ws_size = 2147483647; repl.proto_max = 10; socket.checksum = 2; socket.recv_buf_size = auto; socket.send_buf_size = auto; |
Thanks for any tips.
jfillatre does you database schema contain a table with foreign key constraint defined over a character (e.g. varchar) column, and the table is using some multi-byte character set (e.g. UTF8) ?