[MDEV-24852] MariaDB galera crush while LOAD DATA LOCAL INFILE Created: 2021-02-12  Updated: 2021-12-08  Resolved: 2021-11-01

Status: Closed
Project: MariaDB Server
Component/s: Galera
Affects Version/s: 10.2.25, 10.2.38
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Sergey Sokolov Assignee: Allen Lee (Inactive)
Resolution: Incomplete Votes: 0
Labels: need_feedback
Environment:

CentOS 7, 2xMaster-Master servers + 1 garbd

Attachments: Zip Archive CS0315197.zip    

 Description   

This bug is very similar to https://jira.mariadb.org/browse/MDEV-17243, but I didn't expect it in version 10.2.25.

2021-02-12 11:36:41 140042572732160 [ERROR] WSREP: FSM: no such a transition ABORTING -> REPLICATING
 
210212 11:36:41 [ERROR] mysqld got signal 6 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed,
 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.2.25-MariaDB-log
 
key_buffer_size=268435456
 
read_buffer_size=131072
 
max_used_connections=1179
 
max_threads=1202
 
thread_count=182
 
It is possible that mysqld could use up to
 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2903301 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x7f5ca439c4c8
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x7f5e33cd0d70 thread_stack 0x49000
 
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x561988cdb11e]
 
/usr/sbin/mysqld(handle_fatal_signal+0x30d)[0x56198875bd1d]
 
/lib64/libpthread.so.0(+0xf5e0)[0x7f709bb675e0]
 
/lib64/libc.so.6(gsignal+0x37)[0x7f709a0741f7]
 
/lib64/libc.so.6(abort+0x148)[0x7f709a0758e8]
 
/usr/lib64/galera/libgalera_smm.so(_ZN6galera3FSMINS_9TrxHandle5StateENS1_10TransitionENS_10EmptyGuardENS_11EmptyActionEE8shift_toES2_+0x17c)[0x7f7096c9035c]
 
/usr/lib64/galera/libgalera_smm.so(_ZN6galera13ReplicatorSMM9replicateEPNS_9TrxHandleEP14wsrep_trx_meta+0x171)[0x7f7096c86bf1]
 
/usr/lib64/galera/libgalera_smm.so(galera_pre_commit+0xaa)[0x7f7096c9784a]
 
/usr/sbin/mysqld(wsrep_run_wsrep_commit+0x74a)[0x5619886ececa]
 
/usr/sbin/mysqld(+0x60edc8)[0x5619886eddc8]
 
/usr/sbin/mysqld(_Z15ha_commit_transP3THDb+0x1a0)[0x56198875f1f0]
 
/usr/sbin/mysqld(_Z17trans_commit_stmtP3THD+0x57)[0x5619886a37d7]
 
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x7d5)[0x5619885c0cc5]
 
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x2de)[0x5619885c90fe]
 
/usr/sbin/mysqld(+0x4ea9af)[0x5619885c99af]
 
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x10a5)[0x5619885cb185]
 
/usr/sbin/mysqld(_Z10do_commandP3THD+0x165)[0x5619885cce25]
 
/usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1aa)[0x561988695c9a]
 
/usr/sbin/mysqld(handle_one_connection+0x3d)[0x561988695dbd]
 
/lib64/libpthread.so.0(+0x7e25)[0x7f709bb5fe25]
 
/lib64/libc.so.6(clone+0x6d)[0x7f709a13734d]
 
 
 
Trying to get some variables.
 
Some pointers may be invalid and cause the dump to abort.
 
Query (0x7f5ca4104d40): LOAD DATA LOCAL INFILE '/var/www/vhost/inom.int.autoopt.ru/releases/20210204081603/www/protected/config/../../../tmp/3b0dc23c437509d19a77763b84f97756/price_awk.txt' INTO TABLE `price_berg_procenka` CHARACTER SET utf8 FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\r\n' (art_search, name_tov,
 
Connection ID (thread ID): 595053953
 
Status: NOT_KILLED
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=off,derived_with_keys=off,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,par
 
 
 
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
 
information that should help you find out what is causing the crash.
 
Writing a core file...
 
Working directory at /home/mysql
 
Resource Limits:
 
Limit                     Soft Limit           Hard Limit           Units
 
Max cpu time              unlimited            unlimited            seconds
 
Max file size             unlimited            unlimited            bytes
 
Max data size             unlimited            unlimited            bytes
 
Max stack size            8388608              unlimited            bytes
 
Max core file size        0                    unlimited            bytes
 
Max resident set          unlimited            unlimited            bytes
 
Max processes             773671               773671               processes
 
Max open files            16364                16364                files
 
Max locked memory         65536                65536                bytes
 
Max address space         unlimited            unlimited            bytes
 
Max file locks            unlimited            unlimited            locks
 
Max pending signals       773671               773671               signals
 
Max msgqueue size         819200               819200               bytes
 
Max nice priority         0                    0
 
Max realtime priority     0                    0
 
Max realtime timeout      unlimited            unlimited            us
 
Core pattern: core



 Comments   
Comment by Sergey Sokolov [ 2021-02-15 ]

Today we have repeating of a crush. How we did it:

1. Executing of DDL statement cause error on slave node. This node not really slave, but all client traffic goes to other node:

2021-02-15 16:11:33 139933256070912 [ERROR] Slave SQL: Error 'Duplicate column name 'update_date_abcp'' on query. Default database: 'tp'. Query: 'ALTER TABLE `tbl_analogs`
 
ADD COLUMN `update_date_abcp`  int NULL AFTER `update_date`', Internal MariaDB error code: 1060
 
2021-02-15 16:11:33 139933256070912 [Warning] WSREP: RBR event 1 Query apply warning: 1, 909919642
 
2021-02-15 16:11:33 139933256070912 [Warning] WSREP: Ignoring error for TO isolated action: source: 96f6e54a-28bc-11eb-a8bf-662d9239dcc2 version: 3 local: 0 state: APPLYING flags: 65 conn_id: 17615001 trx_id: -1 seqnos (l: 916931069, g: 909919642, s: 909919641, d: 909919641, ts: 81618100969863216)
 
2021-02-15 16:13:47 139933260011264 [Warning] WSREP: certification interval for trx source: 96f6e54a-28bc-11eb-a8bf-662d9239dcc2 version: 3 local: 0 state: CERTIFYING flags: 1 conn_id: 17606327 trx_id: 59549489336 seqnos (l: 916947454, g: 909936026, s: 909919641, d: -1, ts: 81618234876835507) exceeds the limit of 16
 
384
 
2021-02-15 16:13:47 139933258495744 [Warning] WSREP: certification interval for trx source: 96f6e54a-28bc-11eb-a8bf-662d9239dcc2 version: 3 local: 0 state: CERTIFYING flags: 1 conn_id: 17606327 trx_id: 59549489338 seqnos (l: 916947455, g: 909936027, s: 909919641, d: -1, ts: 81618234877432539) exceeds the limit of 16
 
384
 
...
 
2021-02-15 16:13:50 139933257889536 [Warning] WSREP: certification interval for trx source: 96f6e54a-28bc-11eb-a8bf-662d9239dcc2 version: 3 local: 0 state: CERTIFYING flags: 1 conn_id: 17629471 trx_id: 59549489630 seqnos (l: 916947555, g: 909936127, s: 909919641, d: -1, ts: 81618237849855775) exceeds the limit of 16
 
384
 
2021-02-15 16:13:50 139933257586432 [Warning] WSREP: Attempt to trim certification index at 909919744, above safe-to-discard: 909919641
 
2021-02-15 16:13:50 139933257586432 [Warning] WSREP: certification interval for trx source: 96f6e54a-28bc-11eb-a8bf-662d9239dcc2 version: 3 local: 0 state: CERTIFYING flags: 1 conn_id: 17629471 trx_id: 59549489632 seqnos (l: 916947556, g: 909936128, s: 909919641, d: -1, ts: 81618237850392779) exceeds the limit of 16
 
384
 
...
 
2021-02-15 16:14:50 139933256374016 [Warning] WSREP: certification interval for trx source: 96f6e54a-28bc-11eb-a8bf-662d9239dcc2 version: 3 local: 0 state: CERTIFYING flags: 1 conn_id: 17635474 trx_id: 59549499542 seqnos (l: 916951406, g: 909939978, s: 909919641, d: -1, ts: 81618298304504173) exceeds the limit of 16
 
384
 
2021-02-15 16:15:27 139933137352448 [ERROR] Slave SQL: Error 'Duplicate column name 'update_date_abcp'' on query. Default database: 'tp'. Query: 'ALTER TABLE `tbl_analogs`
 
ADD COLUMN `update_date_abcp`  int NULL AFTER `update_date`', Internal MariaDB error code: 1060
 
2021-02-15 16:15:27 139933137352448 [Warning] WSREP: RBR event 1 Query apply warning: 1, 909941849
 
2021-02-15 16:15:27 139933137352448 [Warning] WSREP: Ignoring error for TO isolated action: source: 96f6e54a-28bc-11eb-a8bf-662d9239dcc2 version: 3 local: 0 state: APPLYING flags: 65 conn_id: 17615001 trx_id: -1 seqnos (l: 916953306, g: 909941849, s: 909941848, d: 909941848, ts: 81618335257637734)
 
2021-02-15 16:20:31 139933256070912 [ERROR] Slave SQL: Error 'Duplicate column name 'update_date_abcp'' on query. Default database: 'tp'. Query: 'ALTER TABLE `tbl_analogs`
 
ADD COLUMN `update_date_abcp`  int NULL AFTER `update_date`', Internal MariaDB error code: 1060
 
2021-02-15 16:20:31 139933256070912 [Warning] WSREP: RBR event 1 Query apply warning: 1, 909951618
 
2021-02-15 16:20:31 139933256070912 [Warning] WSREP: Ignoring error for TO isolated action: source: 96f6e54a-28bc-11eb-a8bf-662d9239dcc2 version: 3 local: 0 state: APPLYING flags: 65 conn_id: 17615001 trx_id: -1 seqnos (l: 916963141, g: 909951618, s: 909951617, d: 909951617, ts: 81618639089646933)
 
2021-02-15 16:22:50 139933256980224 [ERROR] Slave SQL: Error 'Duplicate column name 'update_date_abcp'' on query. Default database: 'tp'. Query: 'ALTER TABLE `tbl_analogs`
 
ADD COLUMN `update_date_abcp`  int NULL AFTER `update_date`,
 
AUTO_INCREMENT=5267861448', Internal MariaDB error code: 1060
 
2021-02-15 16:22:50 139933256980224 [Warning] WSREP: RBR event 1 Query apply warning: 1, 909957961
 
2021-02-15 16:22:50 139933256980224 [Warning] WSREP: Ignoring error for TO isolated action: source: 96f6e54a-28bc-11eb-a8bf-662d9239dcc2 version: 3 local: 0 state: APPLYING flags: 65 conn_id: 17615001 trx_id: -1 seqnos (l: 916969519, g: 909957961, s: 909957960, d: 909957960, ts: 81618778739039138)
 
2021-02-15 16:23:34 140005917837056 [Note] WSREP: (1aa0e05e, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://172.30.100.197:4567
 
2021-02-15 16:23:35 140005917837056 [Note] WSREP: (1aa0e05e, 'tcp://0.0.0.0:4567') reconnecting to 96f6e54a (tcp://172.30.100.197:4567), attempt 0
 
2021-02-15 16:23:36 140005917837056 [Note] WSREP: evs::proto(1aa0e05e, OPERATIONAL, view_id(REG,1aa0e05e,115)) suspecting node: 96f6e54a
 
2021-02-15 16:23:36 140005917837056 [Note] WSREP: evs::proto(1aa0e05e, OPERATIONAL, view_id(REG,1aa0e05e,115)) suspected node without join message, declaring inactive
 
2021-02-15 16:23:37 140005917837056 [Note] WSREP: view(view_id(NON_PRIM,1aa0e05e,115) memb {
 
        1aa0e05e,2
 
} joined {
 
} left {
 
} partitioned {
 
        96f6e54a,2
 
})
 
2021-02-15 16:23:37 140005917837056 [Note] WSREP: view(view_id(NON_PRIM,1aa0e05e,116) memb {
 
        1aa0e05e,2
 
} joined {
 
} left {
 
} partitioned {
 
        96f6e54a,2
 
})
 
2021-02-15 16:23:37 140005909444352 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
 
2021-02-15 16:23:37 140005909444352 [Note] WSREP: Flow-control interval: [16, 16]
 
2021-02-15 16:23:37 140005909444352 [Note] WSREP: Trying to continue unpaused monitor
 
2021-02-15 16:23:37 140005909444352 [Note] WSREP: Received NON-PRIMARY.
 
2021-02-15 16:23:37 140005909444352 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 909958467)
 
2021-02-15 16:23:37 140005909444352 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
 
2021-02-15 16:23:37 140005909444352 [Note] WSREP: Flow-control interval: [16, 16]
 
2021-02-15 16:23:37 140005909444352 [Note] WSREP: Trying to continue unpaused monitor
 
2021-02-15 16:23:37 140005909444352 [Note] WSREP: Received NON-PRIMARY.
 
2021-02-15 16:23:37 139933258798848 [Note] WSREP: New cluster view: global state: 96f73bc4-28bc-11eb-90e6-c3c240204854:909958467, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 3
 
2021-02-15 16:23:37 139933258798848 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
2021-02-15 16:23:37 139933258798848 [Note] WSREP: New cluster view: global state: 96f73bc4-28bc-11eb-90e6-c3c240204854:909958467, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 3
 
2021-02-15 16:23:37 139933258798848 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
2021-02-15 16:24:20 140005917837056 [Note] WSREP: (1aa0e05e, 'tcp://0.0.0.0:4567') reconnecting to 96f6e54a (tcp://172.30.100.197:4567), attempt 30
 
2021-02-15 16:25:05 140005917837056 [Note] WSREP: (1aa0e05e, 'tcp://0.0.0.0:4567') reconnecting to 96f6e54a (tcp://172.30.100.197:4567), attempt 60
 
2021-02-15 16:25:49 140005917837056 [Note] WSREP: (1aa0e05e, 'tcp://0.0.0.0:4567') reconnecting to 96f6e54a (tcp://172.30.100.197:4567), attempt 90
 
2021-02-15 16:26:34 140005917837056 [Note] WSREP: (1aa0e05e, 'tcp://0.0.0.0:4567') reconnecting to 96f6e54a (tcp://172.30.100.197:4567), attempt 120
 
2021-02-15 16:27:19 140005917837056 [Note] WSREP: (1aa0e05e, 'tcp://0.0.0.0:4567') reconnecting to 96f6e54a (tcp://172.30.100.197:4567), attempt 150
 
2021-02-15 16:28:04 140005917837056 [Note] WSREP: (1aa0e05e, 'tcp://0.0.0.0:4567') reconnecting to 96f6e54a (tcp://172.30.100.197:4567), attempt 180
 
2021-02-15 16:28:49 140005917837056 [Note] WSREP: (1aa0e05e, 'tcp://0.0.0.0:4567') reconnecting to 96f6e54a (tcp://172.30.100.197:4567), attempt 210
 
2021-02-15 16:29:34 140005917837056 [Note] WSREP: (1aa0e05e, 'tcp://0.0.0.0:4567') reconnecting to 96f6e54a (tcp://172.30.100.197:4567), attempt 240
 
2021-02-15 16:30:19 140005917837056 [Note] WSREP: (1aa0e05e, 'tcp://0.0.0.0:4567') reconnecting to 96f6e54a (tcp://172.30.100.197:4567), attempt 270
 
2021-02-15 16:31:04 140005917837056 [Note] WSREP: (1aa0e05e, 'tcp://0.0.0.0:4567') reconnecting to 96f6e54a (tcp://172.30.100.197:4567), attempt 300
 
2021-02-15 16:31:28 139933122459392 [Note] /usr/sbin/mysqld (initiated by: unknown): Normal shutdown
 
2021-02-15 16:31:28 139933122459392 [Note] WSREP: Stop replication
 
2021-02-15 16:31:28 139933122459392 [Note] WSREP: Closing send monitor...
 
2021-02-15 16:31:28 139933122459392 [Note] WSREP: Closed send monitor.
 
2021-02-15 16:31:28 139933122459392 [Note] WSREP: gcomm: joining thread
 
2021-02-15 16:31:28 139933122459392 [Note] WSREP: gcomm: closing backend
 
2021-02-15 16:31:28 139933122459392 [Note] WSREP: view((empty))
 
2021-02-15 16:31:28 139933122459392 [Note] WSREP: gcomm: closed
 
2021-02-15 16:31:28 140005909444352 [Note] WSREP: Received self-leave message.
 
2021-02-15 16:31:28 140005909444352 [Note] WSREP: Flow-control interval: [0, 0]
 
2021-02-15 16:31:28 140005909444352 [Note] WSREP: Trying to continue unpaused monitor
 
2021-02-15 16:31:28 140005909444352 [Note] WSREP: Received SELF-LEAVE. Closing connection.
 
2021-02-15 16:31:28 140005909444352 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 909958467)
 
2021-02-15 16:31:28 140005909444352 [Note] WSREP: RECV thread exiting 0: Success
 
2021-02-15 16:31:28 139933258798848 [Note] WSREP: New cluster view: global state: 96f73bc4-28bc-11eb-90e6-c3c240204854:909958467, view# -1: non-Primary, number of nodes: 0, my index: -1, protocol version 3
 
2021-02-15 16:31:28 139933258798848 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
2021-02-15 16:31:28 139933258798848 [Note] WSREP: applier thread exiting (code:0)
 
2021-02-15 16:31:28 139933122459392 [Note] WSREP: recv_thread() joined.
 
2021-02-15 16:31:28 139933122459392 [Note] WSREP: Closing replication queue.
 
2021-02-15 16:31:28 139933122459392 [Note] WSREP: Closing slave action queue.
 
2021-02-15 16:31:28 139933137958656 [Note] WSREP: applier thread exiting (code:6)
 
2021-02-15 16:31:28 139933256980224 [Note] WSREP: applier thread exiting (code:6)
 
2021-02-15 16:31:28 139933256677120 [Note] WSREP: applier thread exiting (code:6)
 
2021-02-15 16:31:28 139933257283328 [Note] WSREP: applier thread exiting (code:6)
 
2021-02-15 16:31:28 139933137655552 [Note] WSREP: applier thread exiting (code:6)
 
2021-02-15 16:31:28 139933257889536 [Note] WSREP: applier thread exiting (code:6)
 
2021-02-15 16:31:28 139933138261760 [Note] WSREP: applier thread exiting (code:6)
 
2021-02-15 16:31:28 139933256070912 [Note] WSREP: applier thread exiting (code:6)
 
2021-02-15 16:31:28 139933260617472 [Note] WSREP: applier thread exiting (code:6)
 
2021-02-15 16:31:28 139933259101952 [Note] WSREP: applier thread exiting (code:6)
 
2021-02-15 16:31:28 139933257586432 [Note] WSREP: applier thread exiting (code:6)
 
2021-02-15 16:31:28 139933259708160 [Note] WSREP: applier thread exiting (code:6)
 
2021-02-15 16:31:28 139933137352448 [Note] WSREP: applier thread exiting (code:6)
 
2021-02-15 16:31:28 139933136746240 [Note] WSREP: applier thread exiting (code:6)
 
2021-02-15 16:31:28 139933136443136 [Note] WSREP: applier thread exiting (code:6)
 
2021-02-15 16:31:28 139933259405056 [Note] WSREP: applier thread exiting (code:6)
 
2021-02-15 16:31:28 139933258495744 [Note] WSREP: applier thread exiting (code:6)
 
2021-02-15 16:31:28 139933256374016 [Note] WSREP: applier thread exiting (code:6)
 
2021-02-15 16:31:28 139933258192640 [Note] WSREP: applier thread exiting (code:6)
 
2021-02-15 16:31:28 139933137049344 [Note] WSREP: applier thread exiting (code:6)
 
2021-02-15 16:31:28 139933260011264 [Note] WSREP: applier thread exiting (code:6)
 
2021-02-15 16:31:28 139933260314368 [Note] WSREP: applier thread exiting (code:6)
 
2021-02-15 16:31:28 140011374540544 [Note] WSREP: applier thread exiting (code:6)
 
2021-02-15 16:31:30 140011374843648 [Note] WSREP: rollbacker thread exiting
 
2021-02-15 16:31:30 139933122459392 [Note] Event Scheduler: Killing the scheduler thread, thread id 9
 
2021-02-15 16:31:30 139933122459392 [Note] Event Scheduler: Waiting for the scheduler thread to reply
 
2021-02-15 16:31:30 139933122459392 [Note] Event Scheduler: Stopped
 
2021-02-15 16:31:30 139933122459392 [Note] Event Scheduler: Purging the queue. 0 events
 
2021-02-15 16:31:30 139933135398656 [Note] InnoDB: FTS optimize thread exiting.
 
2021-02-15 16:31:30 139933122459392 [Note] WSREP: dtor state: CLOSED
 
2021-02-15 16:31:30 139933122459392 [Note] WSREP: mon: entered 909013849 oooe fraction 0.0720851 oool fraction 0.0214276
 
2021-02-15 16:31:30 139933122459392 [Note] WSREP: mon: entered 909013849 oooe fraction 0.0863112 oool fraction 0.0214342
 
2021-02-15 16:31:30 139933122459392 [Note] WSREP: mon: entered 916970027 oooe fraction 0 oool fraction 2.1811e-09
 
2021-02-15 16:31:30 139933122459392 [Note] WSREP: cert index usage at exit 0
 
2021-02-15 16:31:30 139933122459392 [Note] WSREP: cert trx map usage at exit 508
 
2021-02-15 16:31:30 139933122459392 [Note] WSREP: deps set usage at exit 0
 
2021-02-15 16:31:30 139933122459392 [Note] WSREP: avg deps dist 6100.89
 
2021-02-15 16:31:30 139933122459392 [Note] WSREP: avg cert interval 0.430889
 
2021-02-15 16:31:30 139933122459392 [Note] WSREP: cert index size 701
 
2021-02-15 16:31:31 140011391629056 [Note] WSREP: Service thread queue flushed.
 
2021-02-15 16:31:31 139933122459392 [Note] WSREP: wsdb trx map usage 0 conn query map usage 0
 
2021-02-15 16:31:31 139933122459392 [Note] WSREP: MemPool(LocalTrxHandle): hit ratio: 0.903546, misses: 36086, in use: 0, in pool: 1024
 
2021-02-15 16:31:31 139933122459392 [Note] WSREP: MemPool(SlaveTrxHandle): hit ratio: 0.999894, misses: 96235, in use: 0, in pool: 2050
 
2021-02-15 16:31:31 139933122459392 [Note] WSREP: Shifting CLOSED -> DESTROYED (TO: 909958467)
 
2021-02-15 16:31:31 139933122459392 [Note] WSREP: Flushing memory map to disk...
 
2021-02-15 16:31:32 139933122459392 [Note] InnoDB: Starting shutdown...
 
2021-02-15 16:31:32 139933163439872 [Note] InnoDB: Dumping buffer pool(s) to /home/mysql/ib_buffer_pool
 
2021-02-15 16:31:32 139933163439872 [Note] InnoDB: Instance 0, restricted to 32767 pages due to innodb_buf_pool_dump_pct=25
 
2021-02-15 16:31:32 139933163439872 [Note] InnoDB: Instance 1, restricted to 32767 pages due to innodb_buf_pool_dump_pct=25
 
2021-02-15 16:31:32 139933163439872 [Note] InnoDB: Instance 2, restricted to 32767 pages due to innodb_buf_pool_dump_pct=25
 
2021-02-15 16:31:32 139933163439872 [Note] InnoDB: Instance 3, restricted to 32767 pages due to innodb_buf_pool_dump_pct=25
 
2021-02-15 16:31:32 139933163439872 [Note] InnoDB: Instance 4, restricted to 32767 pages due to innodb_buf_pool_dump_pct=25
 
2021-02-15 16:31:32 139933163439872 [Note] InnoDB: Instance 5, restricted to 32767 pages due to innodb_buf_pool_dump_pct=25
 
2021-02-15 16:31:32 139933163439872 [Note] InnoDB: Instance 6, restricted to 32768 pages due to innodb_buf_pool_dump_pct=25
 
2021-02-15 16:31:32 139933163439872 [Note] InnoDB: Instance 7, restricted to 32767 pages due to innodb_buf_pool_dump_pct=25
 
2021-02-15 16:31:32 139933163439872 [Note] InnoDB: Instance 8, restricted to 32767 pages due to innodb_buf_pool_dump_pct=25
 
2021-02-15 16:31:32 139933163439872 [Note] InnoDB: Instance 9, restricted to 32767 pages due to innodb_buf_pool_dump_pct=25
 
2021-02-15 16:31:32 139933163439872 [Note] InnoDB: Instance 10, restricted to 32768 pages due to innodb_buf_pool_dump_pct=25
 
2021-02-15 16:31:32 139933163439872 [Note] InnoDB: Instance 11, restricted to 32767 pages due to innodb_buf_pool_dump_pct=25
 
2021-02-15 16:31:32 139933163439872 [Note] InnoDB: Instance 12, restricted to 32767 pages due to innodb_buf_pool_dump_pct=25
 
2021-02-15 16:31:32 139933163439872 [Note] InnoDB: Instance 13, restricted to 32767 pages due to innodb_buf_pool_dump_pct=25
 
2021-02-15 16:31:32 139933163439872 [Note] InnoDB: Instance 14, restricted to 32768 pages due to innodb_buf_pool_dump_pct=25
 
2021-02-15 16:31:32 139933163439872 [Note] InnoDB: Instance 15, restricted to 32767 pages due to innodb_buf_pool_dump_pct=25
 
2021-02-15 16:31:32 139933163439872 [Note] InnoDB: Instance 16, restricted to 32767 pages due to innodb_buf_pool_dump_pct=25
 
2021-02-15 16:31:32 139933163439872 [Note] InnoDB: Instance 17, restricted to 32767 pages due to innodb_buf_pool_dump_pct=25
 
2021-02-15 16:31:32 139933163439872 [Note] InnoDB: Instance 18, restricted to 32768 pages due to innodb_buf_pool_dump_pct=25
 
2021-02-15 16:31:32 139933163439872 [Note] InnoDB: Instance 19, restricted to 32767 pages due to innodb_buf_pool_dump_pct=25
 
2021-02-15 16:31:32 139933163439872 [Note] InnoDB: Instance 20, restricted to 32767 pages due to innodb_buf_pool_dump_pct=25
 
2021-02-15 16:31:32 139933163439872 [Note] InnoDB: Instance 21, restricted to 32767 pages due to innodb_buf_pool_dump_pct=25
 
2021-02-15 16:31:32 139933163439872 [Note] InnoDB: Instance 22, restricted to 32768 pages due to innodb_buf_pool_dump_pct=25
 
2021-02-15 16:31:32 139933163439872 [Note] InnoDB: Instance 23, restricted to 32767 pages due to innodb_buf_pool_dump_pct=25
 
2021-02-15 16:31:32 139933163439872 [Note] InnoDB: Instance 24, restricted to 32767 pages due to innodb_buf_pool_dump_pct=25
 
2021-02-15 16:31:32 139933163439872 [Note] InnoDB: Instance 25, restricted to 32767 pages due to innodb_buf_pool_dump_pct=25
 
2021-02-15 16:31:32 139933163439872 [Note] InnoDB: Instance 26, restricted to 32768 pages due to innodb_buf_pool_dump_pct=25
 
2021-02-15 16:31:32 139933163439872 [Note] InnoDB: Instance 27, restricted to 32767 pages due to innodb_buf_pool_dump_pct=25
 
2021-02-15 16:31:32 139933163439872 [Note] InnoDB: Instance 28, restricted to 32767 pages due to innodb_buf_pool_dump_pct=25
 
2021-02-15 16:31:32 139933163439872 [Note] InnoDB: Instance 29, restricted to 32767 pages due to innodb_buf_pool_dump_pct=25
 
2021-02-15 16:31:32 139933163439872 [Note] InnoDB: Instance 30, restricted to 32768 pages due to innodb_buf_pool_dump_pct=25
 
2021-02-15 16:31:32 139933163439872 [Note] InnoDB: Instance 31, restricted to 32767 pages due to innodb_buf_pool_dump_pct=25
 
2021-02-15 16:31:32 139933163439872 [Note] InnoDB: Buffer pool(s) dump completed at 210215 16:31:32
 
2021-02-15 16:31:41 139933122459392 [Note] InnoDB: Shutdown completed; log sequence number 91926773212582
 
2021-02-15 16:31:41 139933122459392 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
 
2021-02-15 16:31:41 139933122459392 [Note] /usr/sbin/mysqld: Shutdown complete

On the master node (with client trafic) we have error looks like wroted in description:

2021-02-15 16:23:25 139941515069184 [ERROR] WSREP: FSM: no such a transition ABORTING -> REPLICATING
 
210215 16:23:25 [ERROR] mysqld got signal 6 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed,
 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.2.25-MariaDB-log
 
key_buffer_size=268435456
 
read_buffer_size=131072
 
max_used_connections=394
 
max_threads=1202
 
thread_count=196
 
It is possible that mysqld could use up to
 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2903301 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x7f45a40008c8
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x7f46ac4b7d70 thread_stack 0x49000
 
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x559d5af6011e]
 
/usr/sbin/mysqld(handle_fatal_signal+0x30d)[0x559d5a9e0d1d]
 
/lib64/libpthread.so.0(+0xf5e0)[0x7f58f9ec35e0]
 
/lib64/libc.so.6(gsignal+0x37)[0x7f58f83d01f7]
 
/lib64/libc.so.6(abort+0x148)[0x7f58f83d18e8]
 
/usr/lib64/galera/libgalera_smm.so(_ZN6galera3FSMINS_9TrxHandle5StateENS1_10TransitionENS_10EmptyGuardENS_11EmptyActionEE8shift_toES2_+0x17c)[0x7f58f4fec35c]
 
/usr/lib64/galera/libgalera_smm.so(_ZN6galera13ReplicatorSMM9replicateEPNS_9TrxHandleEP14wsrep_trx_meta+0x171)[0x7f58f4fe2bf1]
 
/usr/lib64/galera/libgalera_smm.so(galera_pre_commit+0xaa)[0x7f58f4ff384a]
 
/usr/sbin/mysqld(wsrep_run_wsrep_commit+0x74a)[0x559d5a971eca]
 
/usr/sbin/mysqld(+0x60edc8)[0x559d5a972dc8]
 
/usr/sbin/mysqld(_Z15ha_commit_transP3THDb+0x1a0)[0x559d5a9e41f0]
 
/usr/sbin/mysqld(_Z17trans_commit_stmtP3THD+0x57)[0x559d5a9287d7]
 
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x7d5)[0x559d5a845cc5]
 
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x2de)[0x559d5a84e0fe]
 
/usr/sbin/mysqld(+0x4ea9af)[0x559d5a84e9af]
 
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x10a5)[0x559d5a850185]
 
/usr/sbin/mysqld(_Z10do_commandP3THD+0x165)[0x559d5a851e25]
 
/usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1aa)[0x559d5a91ac9a]
 
/usr/sbin/mysqld(handle_one_connection+0x3d)[0x559d5a91adbd]
 
/lib64/libpthread.so.0(+0x7e25)[0x7f58f9ebbe25]
 
/lib64/libc.so.6(clone+0x6d)[0x7f58f849334d]
 
 
 
Trying to get some variables.
 
Some pointers may be invalid and cause the dump to abort.
 
Query (0x7f45a43e81d0): LOAD DATA LOCAL INFILE '/var/www/vhost/inom.int.autoopt.ru/releases/20210215102453/www/protected/config/../../../tmp/6704ce0b3bfdcc3925893e41508f861a/price.txt' INTO TABLE `price_konsul` CHARACTER SET utf8 FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\r\n' (art_supplier, kolvo, cena, cenar,
 
Connection ID (thread ID): 17692416
 
Status: NOT_KILLED
 
 
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=off,derived_with_keys=off,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,par
 
 
 
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
 
information that should help you find out what is causing the crash.
 
Writing a core file...
 
Working directory at /home/mysql
 
Resource Limits:
 
Limit                     Soft Limit           Hard Limit           Units
 
Max cpu time              unlimited            unlimited            seconds
 
Max file size             unlimited            unlimited            bytes
 
Max data size             unlimited            unlimited            bytes
 
Max stack size            8388608              unlimited            bytes
 
Max core file size        0                    unlimited            bytes
 
Max resident set          unlimited            unlimited            bytes
 
Max processes             773671               773671               processes
 
Max open files            16364                16364                files
 
Max locked memory         65536                65536                bytes
 
Max address space         unlimited            unlimited            bytes
 
Max file locks            unlimited            unlimited            locks
 
Max pending signals       773671               773671               signals
 
Max msgqueue size         819200               819200               bytes
 
Max nice priority         0                    0
 
Max realtime priority     0                    0
 
Max realtime timeout      unlimited            unlimited            us
 
Core pattern: core

Tonight i'll try upgrade both nodes to 10.2.36 MariaDB and 25.3.31 galera (now is 25.3.23).

Generated at Thu Feb 08 09:33:11 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.