[MDEV-17262] mysql crashed on galera server during data insertion while other node rejoined cluster - Jira

XML

Word

Printable

Details

Type: Bug
Status: Closed (View Workflow)
Priority: Critical
Resolution: Fixed
Affects Version/s: 10.2.14
Fix Version/s: 10.2.23, 10.3.14
Component/s: Galera
Labels:
None
Environment:
2 galera nodes; + 1 node rejoining cluster ; OS Fedora27

Description

mysql crashed on galera server during data insertion while other node rejoined cluster

mysql crashed when 3rd Node rejoined cluster with IST.
problem occurred during data loading ;
wsrep failed on the 3rd Node and Node 3 was manually rejoined:

[root@fedora6 ~]# systemctl  start mariadb.service

Node 3 rejoined successfully cluster but mysql crashed on Node1

2018-09-21 14:11:21 139852054320896 [ERROR] WSREP: FSM: no such a transition ABORTING -> REPLICATING

180921 14:11:21 [ERROR] mysqld got signal 6 ;

This could be because you hit a bug. It is also possible that this binary

or one of the libraries it was linked against is corrupt, improperly built,

or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help

diagnose the problem, but since we have already crashed,

something is definitely wrong and this may fail.

Server version: 10.2.14-MariaDB

key_buffer_size=134217728

read_buffer_size=131072

max_used_connections=24

max_threads=153

thread_count=18

It is possible that mysqld could use up to

key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467245 K  bytes of memory

Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f313c000c08

Attempting backtrace. You can use the following information to find out

where mysqld died. If you see no messages after this, something went

terribly wrong...

stack_bottom = 0x7f31d804ccd8 thread_stack 0x49000

/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55f9d36d118e]

/usr/sbin/mysqld(handle_fatal_signal+0x5a3)[0x55f9d31575f3]

sigaction.c:0(__restore_rt)[0x7f32711e81c0]

:0(__GI_raise)[0x7f326f118750]

:0(__GI_abort)[0x7f326f119d31]

/usr/lib64/galera/libgalera_smm.so(_ZN6galera3FSMINS_9TrxHandle5StateENS1_10TransitionENS_10EmptyGuardENS_11EmptyActionEE8shift_toES2_+0x190)[0x7f326b238850]

/usr/lib64/galera/libgalera_smm.so(_ZN6galera13ReplicatorSMM9replicateEPNS_9TrxHandleEP14wsrep_trx_meta+0x277)[0x7f326b231457]

/usr/lib64/galera/libgalera_smm.so(galera_pre_commit+0xb3)[0x7f326b251303]

/usr/sbin/mysqld(wsrep_run_wsrep_commit+0x987)[0x55f9d30ec147]

/usr/sbin/mysqld(+0x5e9d73)[0x55f9d30ecd73]

/usr/sbin/mysqld(_Z15ha_commit_transP3THDb+0x31f)[0x55f9d315a72f]

/usr/sbin/mysqld(_Z17trans_commit_stmtP3THD+0x5d)[0x55f9d309ff4d]

/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x3bf)[0x55f9d2fc0bff]

/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x2f3)[0x55f9d2fc9543]

/usr/sbin/mysqld(+0x4c6d16)[0x55f9d2fc9d16]

/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17b0)[0x55f9d2fcbad0]

/usr/sbin/mysqld(_Z10do_commandP3THD+0x230)[0x55f9d2fccfa0]

/usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x20a)[0x55f9d309130a]

/usr/sbin/mysqld(handle_one_connection+0x3d)[0x55f9d30914dd]

pthread_create.c:0(start_thread)[0x7f32711dd50b]

:0(__GI___clone)[0x7f326f1d938f]

Trying to get some variables.

Some pointers may be invalid and cause the dump to abort.

Query (0x7f313c00f2b0): LOAD DATA  LOCAL INFILE '/root/QA/mariadb-columnstore-tpcds/insert-data-tables/data/tpcds_2000/inventory.tbl' INTO TABLE `inventory` FIELDS  TERMINATED BY '|'  LINES TERMINATED BY '\n' IGNORE 0 LINES

Connection ID (thread ID): 23

Status: NOT_KILLED

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains

information that should help you find out what is causing the crash.

2018-09-21 13:57:17 139851287860992 [Note] WSREP: intermediate transaction commit in LOAD DATA

2018-09-21 13:57:24 139851383834368 [Note] WSREP: intermediate transaction commit in LOAD DATA

2018-09-21 13:57:27 139852054628096 [Note] WSREP: intermediate transaction commit in LOAD DATA

2018-09-21 13:57:32 139852189087488 [Note] WSREP: intermediate transaction commit in LOAD DATA

2018-09-21 13:57:36 139851288168192 [Note] WSREP: intermediate transaction commit in LOAD DATA

2018-09-21 13:57:56 139852054320896 [Note] WSREP: intermediate transaction commit in LOAD DATA

2018-09-21 13:57:56 139851290318592 [Note] WSREP: intermediate transaction commit in LOAD DATA

2018-09-21 14:01:32 139851383834368 [Note] WSREP: intermediate transaction commit in LOAD DATA

2018-09-21 14:01:43 139851287860992 [Note] WSREP: intermediate transaction commit in LOAD DATA

2018-09-21 14:01:45 139852054628096 [Note] WSREP: intermediate transaction commit in LOAD DATA

2018-09-21 14:01:53 139852189087488 [Note] WSREP: intermediate transaction commit in LOAD DATA

2018-09-21 14:02:03 139851288168192 [Note] WSREP: intermediate transaction commit in LOAD DATA

2018-09-21 14:02:14 139854486722304 [Note] WSREP: (29f5a00c, 'tcp://0.0.0.0:4567') connection established to cbf6deed tcp://172.20.2.236:4567

2018-09-21 14:02:14 139854486722304 [Note] WSREP: (29f5a00c, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:

2018-09-21 14:02:15 139854486722304 [Note] WSREP: declaring 48036483 at tcp://172.20.2.234:4567 stable

2018-09-21 14:02:15 139854486722304 [Note] WSREP: declaring cbf6deed at tcp://172.20.2.236:4567 stable

2018-09-21 14:02:15 139854486722304 [Note] WSREP: Node 29f5a00c state prim

2018-09-21 14:02:15 139854486722304 [Note] WSREP: view(view_id(PRIM,29f5a00c,63) memb {

        29f5a00c,0

        48036483,0

        cbf6deed,0

} joined {

} left {

} partitioned {

})

2018-09-21 14:02:15 139854486722304 [Note] WSREP: save pc into disk

2018-09-21 14:02:15 139854478329600 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3

2018-09-21 14:02:15 139854478329600 [Note] WSREP: STATE_EXCHANGE: sent state UUID: cc544de6-bd8d-11e8-843f-3346c7ce4927

2018-09-21 14:02:15 139854478329600 [Note] WSREP: STATE EXCHANGE: sent state msg: cc544de6-bd8d-11e8-843f-3346c7ce4927

2018-09-21 14:02:15 139854478329600 [Note] WSREP: STATE EXCHANGE: got state msg: cc544de6-bd8d-11e8-843f-3346c7ce4927 from 0 (dbcluster03)

2018-09-21 14:02:15 139854478329600 [Note] WSREP: STATE EXCHANGE: got state msg: cc544de6-bd8d-11e8-843f-3346c7ce4927 from 1 (dbcluster02)

2018-09-21 14:02:15 139854478329600 [Note] WSREP: STATE EXCHANGE: got state msg: cc544de6-bd8d-11e8-843f-3346c7ce4927 from 2 (dbcluster04)

2018-09-21 14:02:15 139854478329600 [Note] WSREP: Quorum results:

        version    = 4,

        component  = PRIMARY,

        conf_id    = 59,

        members    = 2/3 (joined/total),

        act_id     = 19514,

        last_appl. = 19478,

        protocols  = 0/8/3 (gcs/repl/appl),

        group UUID = a6f20744-ba5c-11e8-9e46-c71bbc8deae3

2018-09-21 14:02:15 139854478329600 [Note] WSREP: Flow-control interval: [28, 28]

2018-09-21 14:02:15 139852213118720 [Note] WSREP: New cluster view: global state: a6f20744-ba5c-11e8-9e46-c71bbc8deae3:19514, view# 60: Primary, number of nodes: 3, my index: 0, protocol version 3

2018-09-21 14:02:15 139852213118720 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

2018-09-21 14:02:15 139852213118720 [Note] WSREP: REPL Protocols: 8 (3, 2)

2018-09-21 14:02:15 139852213118720 [Note] WSREP: Assign initial position for certification: 19514, protocol version: 3

2018-09-21 14:02:15 139854520293120 [Note] WSREP: Service thread queue flushed.

2018-09-21 14:02:16 139852054320896 [Note] WSREP: intermediate transaction commit in LOAD DATA

2018-09-21 14:02:17 139854486722304 [Note] WSREP: (29f5a00c, 'tcp://0.0.0.0:4567') turning message relay requesting off

2018-09-21 14:04:05 139854486722304 [Note] WSREP: (29f5a00c, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://172.20.2.236:4567

2018-09-21 14:04:06 139854486722304 [Note] WSREP: (29f5a00c, 'tcp://0.0.0.0:4567') reconnecting to cbf6deed (tcp://172.20.2.236:4567), attempt 0

2018-09-21 14:04:10 139854486722304 [Note] WSREP: evs::proto(29f5a00c, OPERATIONAL, view_id(REG,29f5a00c,63)) suspecting node: cbf6deed

2018-09-21 14:04:10 139854486722304 [Note] WSREP: evs::proto(29f5a00c, OPERATIONAL, view_id(REG,29f5a00c,63)) suspected node without join message, declaring inactive

2018-09-21 14:04:10 139854486722304 [Note] WSREP: declaring 48036483 at tcp://172.20.2.234:4567 stable

2018-09-21 14:04:10 139854486722304 [Note] WSREP: Node 29f5a00c state prim

2018-09-21 14:04:10 139854486722304 [Note] WSREP: view(view_id(PRIM,29f5a00c,64) memb {

        29f5a00c,0

        48036483,0

} joined {

} left {

} partitioned {

        cbf6deed,0

})

2018-09-21 14:04:10 139854486722304 [Note] WSREP: save pc into disk

2018-09-21 14:04:10 139854486722304 [Note] WSREP: forgetting cbf6deed (tcp://172.20.2.236:4567)

2018-09-21 14:04:10 139854486722304 [Note] WSREP: (29f5a00c, 'tcp://0.0.0.0:4567') turning message relay requesting off

2018-09-21 14:04:10 139854478329600 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2

2018-09-21 14:04:10 139854478329600 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 10dccdb7-bd8e-11e8-a1a6-07e3b7e2b01c

2018-09-21 14:04:10 139854478329600 [Note] WSREP: STATE EXCHANGE: sent state msg: 10dccdb7-bd8e-11e8-a1a6-07e3b7e2b01c

2018-09-21 14:04:10 139854478329600 [Note] WSREP: STATE EXCHANGE: got state msg: 10dccdb7-bd8e-11e8-a1a6-07e3b7e2b01c from 0 (dbcluster03)

2018-09-21 14:04:10 139854478329600 [Note] WSREP: STATE EXCHANGE: got state msg: 10dccdb7-bd8e-11e8-a1a6-07e3b7e2b01c from 1 (dbcluster02)

2018-09-21 14:04:10 139854478329600 [Note] WSREP: Quorum results:

        version    = 4,

        component  = PRIMARY,

        conf_id    = 60,

        members    = 2/2 (joined/total),

        act_id     = 19521,

        last_appl. = 19478,

        protocols  = 0/8/3 (gcs/repl/appl),

        group UUID = a6f20744-ba5c-11e8-9e46-c71bbc8deae3

2018-09-21 14:04:10 139854478329600 [Note] WSREP: Flow-control interval: [23, 23]

2018-09-21 14:04:10 139852213118720 [Note] WSREP: New cluster view: global state: a6f20744-ba5c-11e8-9e46-c71bbc8deae3:19521, view# 61: Primary, number of nodes: 2, my index: 0, protocol version 3

2018-09-21 14:04:10 139852213118720 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

2018-09-21 14:04:10 139852213118720 [Note] WSREP: REPL Protocols: 8 (3, 2)

2018-09-21 14:04:10 139852213118720 [Note] WSREP: Assign initial position for certification: 19521, protocol version: 3

2018-09-21 14:04:10 139854520293120 [Note] WSREP: Service thread queue flushed.

2018-09-21 14:04:12 139854486722304 [Note] WSREP:  cleaning up cbf6deed (tcp://172.20.2.236:4567)

2018-09-21 14:04:14 139854486722304 [Note] WSREP: (29f5a00c, 'tcp://0.0.0.0:4567') connection established to cbf6deed tcp://172.20.2.236:4567

2018-09-21 14:04:14 139854486722304 [Note] WSREP: (29f5a00c, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:

2018-09-21 14:04:14 139854486722304 [Note] WSREP: declaring 48036483 at tcp://172.20.2.234:4567 stable

2018-09-21 14:04:14 139854486722304 [Note] WSREP: declaring cbf6deed at tcp://172.20.2.236:4567 stable

2018-09-21 14:04:14 139854486722304 [Note] WSREP: Node 29f5a00c state prim

2018-09-21 14:04:14 139854486722304 [Note] WSREP: view(view_id(PRIM,29f5a00c,65) memb {

        29f5a00c,0

        48036483,0

        cbf6deed,0

} joined {

} left {

} partitioned {

})

2018-09-21 14:04:14 139854486722304 [Note] WSREP: save pc into disk

2018-09-21 14:04:14 139854478329600 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3

2018-09-21 14:04:14 139854478329600 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 138c60be-bd8e-11e8-84e9-66dd1efd2b1d

2018-09-21 14:04:14 139854478329600 [Note] WSREP: STATE EXCHANGE: sent state msg: 138c60be-bd8e-11e8-84e9-66dd1efd2b1d

2018-09-21 14:04:14 139854478329600 [Note] WSREP: STATE EXCHANGE: got state msg: 138c60be-bd8e-11e8-84e9-66dd1efd2b1d from 0 (dbcluster03)

2018-09-21 14:04:14 139854478329600 [Note] WSREP: STATE EXCHANGE: got state msg: 138c60be-bd8e-11e8-84e9-66dd1efd2b1d from 1 (dbcluster02)

2018-09-21 14:04:15 139854478329600 [Note] WSREP: STATE EXCHANGE: got state msg: 138c60be-bd8e-11e8-84e9-66dd1efd2b1d from 2 (dbcluster04)

2018-09-21 14:04:15 139854478329600 [Note] WSREP: Quorum results:

        version    = 4,

        component  = PRIMARY,

        conf_id    = 61,

        members    = 2/3 (joined/total),

        act_id     = 19522,

        last_appl. = 19478,

        protocols  = 0/8/3 (gcs/repl/appl),

        group UUID = a6f20744-ba5c-11e8-9e46-c71bbc8deae3

2018-09-21 14:04:15 139854478329600 [Note] WSREP: Flow-control interval: [28, 28]

2018-09-21 14:04:15 139852213118720 [Note] WSREP: New cluster view: global state: a6f20744-ba5c-11e8-9e46-c71bbc8deae3:19522, view# 62: Primary, number of nodes: 3, my index: 0, protocol version 3

2018-09-21 14:04:15 139852213118720 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

2018-09-21 14:04:15 139852213118720 [Note] WSREP: REPL Protocols: 8 (3, 2)

2018-09-21 14:04:15 139852213118720 [Note] WSREP: Assign initial position for certification: 19522, protocol version: 3

2018-09-21 14:04:15 139854520293120 [Note] WSREP: Service thread queue flushed.

2018-09-21 14:04:17 139854486722304 [Note] WSREP: (29f5a00c, 'tcp://0.0.0.0:4567') turning message relay requesting off

2018-09-21 14:04:20 139852054320896 [Note] WSREP: intermediate transaction commit in LOAD DATA

2018-09-21 14:05:46 139851287860992 [Note] WSREP: intermediate transaction commit in LOAD DATA

2018-09-21 14:05:46 139851383834368 [Note] WSREP: intermediate transaction commit in LOAD DATA

2018-09-21 14:05:55 139852054628096 [Note] WSREP: intermediate transaction commit in LOAD DATA

2018-09-21 14:05:57 139854486722304 [Note] WSREP: (29f5a00c, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://172.20.2.236:4567

2018-09-21 14:05:58 139854486722304 [Note] WSREP: (29f5a00c, 'tcp://0.0.0.0:4567') reconnecting to cbf6deed (tcp://172.20.2.236:4567), attempt 0

2018-09-21 14:06:01 139852189087488 [Note] WSREP: intermediate transaction commit in LOAD DATA

2018-09-21 14:06:01 139854486722304 [Note] WSREP: evs::proto(29f5a00c, OPERATIONAL, view_id(REG,29f5a00c,65)) suspecting node: cbf6deed

2018-09-21 14:06:01 139854486722304 [Note] WSREP: evs::proto(29f5a00c, OPERATIONAL, view_id(REG,29f5a00c,65)) suspected node without join message, declaring inactive

2018-09-21 14:06:02 139854486722304 [Note] WSREP: declaring 48036483 at tcp://172.20.2.234:4567 stable

2018-09-21 14:06:02 139854486722304 [Note] WSREP: Node 29f5a00c state prim

2018-09-21 14:06:02 139854486722304 [Note] WSREP: view(view_id(PRIM,29f5a00c,66) memb {

        29f5a00c,0

        48036483,0

} joined {

} left {

} partitioned {

        cbf6deed,0

})

2018-09-21 14:06:02 139854486722304 [Note] WSREP: save pc into disk

2018-09-21 14:06:02 139854486722304 [Note] WSREP: forgetting cbf6deed (tcp://172.20.2.236:4567)

2018-09-21 14:06:02 139854486722304 [Note] WSREP: (29f5a00c, 'tcp://0.0.0.0:4567') turning message relay requesting off

2018-09-21 14:06:02 139854478329600 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2

2018-09-21 14:06:02 139854478329600 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 53ef0821-bd8e-11e8-8a11-5f4415a89186

2018-09-21 14:06:02 139854478329600 [Note] WSREP: STATE EXCHANGE: sent state msg: 53ef0821-bd8e-11e8-8a11-5f4415a89186

2018-09-21 14:06:02 139854478329600 [Note] WSREP: STATE EXCHANGE: got state msg: 53ef0821-bd8e-11e8-8a11-5f4415a89186 from 0 (dbcluster03)

2018-09-21 14:06:02 139854478329600 [Note] WSREP: STATE EXCHANGE: got state msg: 53ef0821-bd8e-11e8-8a11-5f4415a89186 from 1 (dbcluster02)

2018-09-21 14:06:02 139854478329600 [Note] WSREP: Quorum results:

        version    = 4,

        component  = PRIMARY,

        conf_id    = 62,

        members    = 2/2 (joined/total),

        act_id     = 19522,

        last_appl. = 19478,

        protocols  = 0/8/3 (gcs/repl/appl),

        group UUID = a6f20744-ba5c-11e8-9e46-c71bbc8deae3

2018-09-21 14:06:02 139854478329600 [Note] WSREP: Flow-control interval: [23, 23]

2018-09-21 14:06:02 139852213118720 [Note] WSREP: New cluster view: global state: a6f20744-ba5c-11e8-9e46-c71bbc8deae3:19522, view# 63: Primary, number of nodes: 2, my index: 0, protocol version 3

2018-09-21 14:06:02 139852213118720 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

2018-09-21 14:06:02 139852213118720 [Note] WSREP: REPL Protocols: 8 (3, 2)

2018-09-21 14:06:02 139852213118720 [Note] WSREP: Assign initial position for certification: 19522, protocol version: 3

2018-09-21 14:06:02 139854520293120 [Note] WSREP: Service thread queue flushed.

2018-09-21 14:06:04 139854486722304 [Note] WSREP: (29f5a00c, 'tcp://0.0.0.0:4567') connection established to cbf6deed tcp://172.20.2.236:4567

2018-09-21 14:06:04 139854486722304 [Warning] WSREP: discarding established (time wait) cbf6deed (tcp://172.20.2.236:4567)

2018-09-21 14:06:05 139854486722304 [Note] WSREP:  cleaning up cbf6deed (tcp://172.20.2.236:4567)

2018-09-21 14:06:07 139852054320896 [Note] WSREP: intermediate transaction commit in LOAD DATA

2018-09-21 14:06:08 139854486722304 [Note] WSREP: (29f5a00c, 'tcp://0.0.0.0:4567') connection established to cbf6deed tcp://172.20.2.236:4567

2018-09-21 14:06:08 139854486722304 [Note] WSREP: (29f5a00c, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:

2018-09-21 14:06:09 139854486722304 [Note] WSREP: declaring 48036483 at tcp://172.20.2.234:4567 stable

2018-09-21 14:06:09 139854486722304 [Note] WSREP: declaring cbf6deed at tcp://172.20.2.236:4567 stable

2018-09-21 14:06:09 139854486722304 [Note] WSREP: Node 29f5a00c state prim

2018-09-21 14:06:09 139854486722304 [Note] WSREP: view(view_id(PRIM,29f5a00c,68) memb {

        29f5a00c,0

        48036483,0

        cbf6deed,0

} joined {

} left {

} partitioned {

})

2018-09-21 14:06:09 139854486722304 [Note] WSREP: save pc into disk

2018-09-21 14:06:09 139854478329600 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3

2018-09-21 14:06:09 139854478329600 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 581b8061-bd8e-11e8-8b1e-e7e2caa8ad51

2018-09-21 14:06:09 139854478329600 [Note] WSREP: STATE EXCHANGE: sent state msg: 581b8061-bd8e-11e8-8b1e-e7e2caa8ad51

2018-09-21 14:06:09 139854478329600 [Note] WSREP: STATE EXCHANGE: got state msg: 581b8061-bd8e-11e8-8b1e-e7e2caa8ad51 from 0 (dbcluster03)

2018-09-21 14:06:09 139854478329600 [Note] WSREP: STATE EXCHANGE: got state msg: 581b8061-bd8e-11e8-8b1e-e7e2caa8ad51 from 1 (dbcluster02)

2018-09-21 14:06:09 139854478329600 [Note] WSREP: STATE EXCHANGE: got state msg: 581b8061-bd8e-11e8-8b1e-e7e2caa8ad51 from 2 (dbcluster04)

2018-09-21 14:06:09 139854478329600 [Note] WSREP: Quorum results:

        version    = 4,

        component  = PRIMARY,

        conf_id    = 63,

        members    = 2/3 (joined/total),

        act_id     = 19527,

        last_appl. = 19478,

        protocols  = 0/8/3 (gcs/repl/appl),

        group UUID = a6f20744-ba5c-11e8-9e46-c71bbc8deae3

2018-09-21 14:06:09 139854478329600 [Note] WSREP: Flow-control interval: [28, 28]

2018-09-21 14:06:09 139852213118720 [Note] WSREP: New cluster view: global state: a6f20744-ba5c-11e8-9e46-c71bbc8deae3:19527, view# 64: Primary, number of nodes: 3, my index: 0, protocol version 3

2018-09-21 14:06:09 139852213118720 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

2018-09-21 14:06:09 139852213118720 [Note] WSREP: REPL Protocols: 8 (3, 2)

2018-09-21 14:06:09 139852213118720 [Note] WSREP: Assign initial position for certification: 19527, protocol version: 3

2018-09-21 14:06:09 139854520293120 [Note] WSREP: Service thread queue flushed.

2018-09-21 14:06:11 139851288168192 [Note] WSREP: intermediate transaction commit in LOAD DATA

2018-09-21 14:06:12 139854486722304 [Note] WSREP: (29f5a00c, 'tcp://0.0.0.0:4567') turning message relay requesting off

2018-09-21 14:06:12 139852054320896 [Note] WSREP: intermediate transaction commit in LOAD DATA

2018-09-21 14:06:17 139854520293120 [Warning] WSREP: Failed to report last committed 19527, -110 (Connection timed out)

2018-09-21 14:07:19 139851290318592 [Note] WSREP: intermediate transaction commit in LOAD DATA

2018-09-21 14:07:51 139854486722304 [Note] WSREP: (29f5a00c, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://172.20.2.236:4567

2018-09-21 14:07:53 139854486722304 [Note] WSREP: (29f5a00c, 'tcp://0.0.0.0:4567') reconnecting to cbf6deed (tcp://172.20.2.236:4567), attempt 0

2018-09-21 14:07:56 139854486722304 [Note] WSREP: evs::proto(29f5a00c, OPERATIONAL, view_id(REG,29f5a00c,68)) suspecting node: cbf6deed

2018-09-21 14:07:56 139854486722304 [Note] WSREP: evs::proto(29f5a00c, OPERATIONAL, view_id(REG,29f5a00c,68)) suspected node without join message, declaring inactive

2018-09-21 14:07:56 139854486722304 [Note] WSREP: declaring 48036483 at tcp://172.20.2.234:4567 stable

2018-09-21 14:07:56 139854486722304 [Note] WSREP: Node 29f5a00c state prim

2018-09-21 14:07:56 139854486722304 [Note] WSREP: view(view_id(PRIM,29f5a00c,69) memb {

        29f5a00c,0

        48036483,0

} joined {

} left {

} partitioned {

        cbf6deed,0

})

2018-09-21 14:07:56 139854486722304 [Note] WSREP: save pc into disk

2018-09-21 14:07:56 139854486722304 [Note] WSREP: forgetting cbf6deed (tcp://172.20.2.236:4567)

2018-09-21 14:07:56 139854486722304 [Note] WSREP: (29f5a00c, 'tcp://0.0.0.0:4567') turning message relay requesting off

2018-09-21 14:07:56 139854478329600 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2

2018-09-21 14:07:56 139854478329600 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 97e5e590-bd8e-11e8-8113-22e7dbcd437b

2018-09-21 14:07:56 139854478329600 [Note] WSREP: STATE EXCHANGE: sent state msg: 97e5e590-bd8e-11e8-8113-22e7dbcd437b

2018-09-21 14:07:56 139854478329600 [Note] WSREP: STATE EXCHANGE: got state msg: 97e5e590-bd8e-11e8-8113-22e7dbcd437b from 0 (dbcluster03)

2018-09-21 14:07:56 139854478329600 [Note] WSREP: STATE EXCHANGE: got state msg: 97e5e590-bd8e-11e8-8113-22e7dbcd437b from 1 (dbcluster02)

2018-09-21 14:07:56 139854478329600 [Note] WSREP: Quorum results:

        version    = 4,

        component  = PRIMARY,

        conf_id    = 64,

        members    = 2/2 (joined/total),

        act_id     = 19528,

        last_appl. = 19478,

        protocols  = 0/8/3 (gcs/repl/appl),

        group UUID = a6f20744-ba5c-11e8-9e46-c71bbc8deae3

2018-09-21 14:07:56 139854478329600 [Note] WSREP: Flow-control interval: [23, 23]

2018-09-21 14:07:56 139852213118720 [Note] WSREP: New cluster view: global state: a6f20744-ba5c-11e8-9e46-c71bbc8deae3:19528, view# 65: Primary, number of nodes: 2, my index: 0, protocol version 3

2018-09-21 14:07:56 139852213118720 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

2018-09-21 14:07:56 139852213118720 [Note] WSREP: REPL Protocols: 8 (3, 2)

2018-09-21 14:07:56 139852213118720 [Note] WSREP: Assign initial position for certification: 19528, protocol version: 3

2018-09-21 14:07:57 139854520293120 [Note] WSREP: Service thread queue flushed.

2018-09-21 14:07:59 139854486722304 [Note] WSREP:  cleaning up cbf6deed (tcp://172.20.2.236:4567)

2018-09-21 14:08:00 139854486722304 [Note] WSREP: (29f5a00c, 'tcp://0.0.0.0:4567') connection established to cbf6deed tcp://172.20.2.236:4567

2018-09-21 14:08:00 139854486722304 [Note] WSREP: (29f5a00c, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:

2018-09-21 14:08:00 139854486722304 [Note] WSREP: declaring 48036483 at tcp://172.20.2.234:4567 stable

2018-09-21 14:08:00 139854486722304 [Note] WSREP: declaring cbf6deed at tcp://172.20.2.236:4567 stable

2018-09-21 14:08:00 139854486722304 [Note] WSREP: Node 29f5a00c state prim

2018-09-21 14:08:00 139854486722304 [Note] WSREP: view(view_id(PRIM,29f5a00c,70) memb {

        29f5a00c,0

        48036483,0

        cbf6deed,0

} joined {

} left {

} partitioned {

})

2018-09-21 14:08:00 139854486722304 [Note] WSREP: save pc into disk

2018-09-21 14:08:00 139854478329600 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3

2018-09-21 14:08:00 139854478329600 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 9a487cde-bd8e-11e8-86e2-1b2e12d0b640

2018-09-21 14:08:00 139854478329600 [Note] WSREP: STATE EXCHANGE: sent state msg: 9a487cde-bd8e-11e8-86e2-1b2e12d0b640

2018-09-21 14:08:00 139854478329600 [Note] WSREP: STATE EXCHANGE: got state msg: 9a487cde-bd8e-11e8-86e2-1b2e12d0b640 from 0 (dbcluster03)

2018-09-21 14:08:00 139854478329600 [Note] WSREP: STATE EXCHANGE: got state msg: 9a487cde-bd8e-11e8-86e2-1b2e12d0b640 from 1 (dbcluster02)

2018-09-21 14:08:00 139854478329600 [Note] WSREP: STATE EXCHANGE: got state msg: 9a487cde-bd8e-11e8-86e2-1b2e12d0b640 from 2 (dbcluster04)

2018-09-21 14:08:00 139854478329600 [Note] WSREP: Quorum results:

        version    = 4,

        component  = PRIMARY,

        conf_id    = 65,

        members    = 2/3 (joined/total),

        act_id     = 19529,

        last_appl. = 19478,

        protocols  = 0/8/3 (gcs/repl/appl),

        group UUID = a6f20744-ba5c-11e8-9e46-c71bbc8deae3

2018-09-21 14:08:00 139854478329600 [Note] WSREP: Flow-control interval: [28, 28]

2018-09-21 14:08:00 139852213118720 [Note] WSREP: New cluster view: global state: a6f20744-ba5c-11e8-9e46-c71bbc8deae3:19529, view# 66: Primary, number of nodes: 3, my index: 0, protocol version 3

2018-09-21 14:08:00 139852213118720 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

2018-09-21 14:08:00 139852213118720 [Note] WSREP: REPL Protocols: 8 (3, 2)

2018-09-21 14:08:00 139852213118720 [Note] WSREP: Assign initial position for certification: 19529, protocol version: 3

2018-09-21 14:08:03 139854486722304 [Note] WSREP: (29f5a00c, 'tcp://0.0.0.0:4567') turning message relay requesting off

2018-09-21 14:08:05 139854520293120 [Warning] WSREP: Failed to report last committed 19528, -110 (Connection timed out)

2018-09-21 14:08:05 139854520293120 [Note] WSREP: Service thread queue flushed.

2018-09-21 14:08:05 139852054320896 [Note] WSREP: commit failed for reason: 3

2018-09-21 14:08:05 139852054320896 [Note] WSREP: conflict state: 0

2018-09-21 14:08:05 139852054320896 [Note] WSREP: cluster conflict due to certification failure for threads:

2018-09-21 14:08:05 139852054320896 [Note] WSREP: Victim thread:

   THD: 23, mode: local, state: executing, conflict: cert failure, seqno: 19530

   SQL: LOAD DATA  LOCAL INFILE '/root/QA/mariadb-columnstore-tpcds/insert-data-tables/data/tpcds_2000/inventory.tbl' INTO TABLE `inventory` FIELDS  TERMINATED BY '|'  LINES TERMINATED BY '\n' IGNORE 0 LINES

2018-09-21 14:09:42 139854486722304 [Note] WSREP: (29f5a00c, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://172.20.2.236:4567

2018-09-21 14:09:43 139854486722304 [Note] WSREP: declaring 48036483 at tcp://172.20.2.234:4567 stable

2018-09-21 14:09:43 139854486722304 [Note] WSREP: forgetting cbf6deed (tcp://172.20.2.236:4567)

2018-09-21 14:09:43 139854486722304 [Note] WSREP: (29f5a00c, 'tcp://0.0.0.0:4567') turning message relay requesting off

2018-09-21 14:09:43 139854486722304 [Note] WSREP: Node 29f5a00c state prim

2018-09-21 14:09:43 139854486722304 [Note] WSREP: view(view_id(PRIM,29f5a00c,71) memb {

        29f5a00c,0

        48036483,0

} joined {

} left {

} partitioned {

        cbf6deed,0

})

2018-09-21 14:09:43 139854486722304 [Note] WSREP: save pc into disk

2018-09-21 14:09:43 139854486722304 [Note] WSREP: forgetting cbf6deed (tcp://172.20.2.236:4567)

2018-09-21 14:09:43 139854478329600 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2

2018-09-21 14:09:43 139854478329600 [Note] WSREP: STATE_EXCHANGE: sent state UUID: d7dbb0a4-bd8e-11e8-9857-675f880b91d2

2018-09-21 14:09:43 139854478329600 [Note] WSREP: STATE EXCHANGE: sent state msg: d7dbb0a4-bd8e-11e8-9857-675f880b91d2

2018-09-21 14:09:43 139854478329600 [Note] WSREP: STATE EXCHANGE: got state msg: d7dbb0a4-bd8e-11e8-9857-675f880b91d2 from 0 (dbcluster03)

2018-09-21 14:09:43 139854478329600 [Note] WSREP: STATE EXCHANGE: got state msg: d7dbb0a4-bd8e-11e8-9857-675f880b91d2 from 1 (dbcluster02)

2018-09-21 14:09:43 139854478329600 [Note] WSREP: Quorum results:

        version    = 4,

        component  = PRIMARY,

        conf_id    = 66,

        members    = 2/2 (joined/total),

        act_id     = 19530,

        last_appl. = 19478,

        protocols  = 0/8/3 (gcs/repl/appl),

        group UUID = a6f20744-ba5c-11e8-9e46-c71bbc8deae3

2018-09-21 14:09:43 139854478329600 [Note] WSREP: Flow-control interval: [23, 23]

2018-09-21 14:09:43 139852213118720 [Note] WSREP: New cluster view: global state: a6f20744-ba5c-11e8-9e46-c71bbc8deae3:19530, view# 67: Primary, number of nodes: 2, my index: 0, protocol version 3

2018-09-21 14:09:43 139852213118720 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

2018-09-21 14:09:43 139852213118720 [Note] WSREP: REPL Protocols: 8 (3, 2)

2018-09-21 14:09:43 139852213118720 [Note] WSREP: Assign initial position for certification: 19530, protocol version: 3

2018-09-21 14:09:43 139854520293120 [Note] WSREP: Service thread queue flushed.

2018-09-21 14:09:44 139851290318592 [Note] WSREP: commit failed for reason: 3

2018-09-21 14:09:44 139851290318592 [Note] WSREP: conflict state: 0

2018-09-21 14:09:44 139851290318592 [Note] WSREP: cluster conflict due to certification failure for threads:

2018-09-21 14:09:44 139851290318592 [Note] WSREP: Victim thread:

   THD: 32, mode: local, state: executing, conflict: cert failure, seqno: 19531

   SQL: LOAD DATA  LOCAL INFILE '/root/QA/mariadb-columnstore-tpcds/insert-data-tables/data/tpcds_2000/web_sales.tbl' INTO TABLE `web_sales` FIELDS  TERMINATED BY '|'  LINES TERMINATED BY '\n' IGNORE 0 LINES

2018-09-21 14:09:49 139854486722304 [Note] WSREP:  cleaning up cbf6deed (tcp://172.20.2.236:4567)

2018-09-21 14:09:50 139854486722304 [Note] WSREP: (29f5a00c, 'tcp://0.0.0.0:4567') connection established to cbf6deed tcp://172.20.2.236:4567

2018-09-21 14:09:50 139854486722304 [Note] WSREP: (29f5a00c, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:

2018-09-21 14:09:50 139854486722304 [Note] WSREP: declaring 48036483 at tcp://172.20.2.234:4567 stable

2018-09-21 14:09:50 139854486722304 [Note] WSREP: declaring cbf6deed at tcp://172.20.2.236:4567 stable

2018-09-21 14:09:50 139854486722304 [Note] WSREP: Node 29f5a00c state prim

2018-09-21 14:09:50 139854486722304 [Note] WSREP: view(view_id(PRIM,29f5a00c,72) memb {

        29f5a00c,0

        48036483,0

        cbf6deed,0

} joined {

} left {

} partitioned {

})

2018-09-21 14:09:50 139854486722304 [Note] WSREP: save pc into disk

2018-09-21 14:09:50 139854478329600 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3

2018-09-21 14:09:50 139854478329600 [Note] WSREP: STATE_EXCHANGE: sent state UUID: dc021662-bd8e-11e8-aaf5-af97e5bc218d

2018-09-21 14:09:50 139854478329600 [Note] WSREP: STATE EXCHANGE: sent state msg: dc021662-bd8e-11e8-aaf5-af97e5bc218d

2018-09-21 14:09:50 139854478329600 [Note] WSREP: STATE EXCHANGE: got state msg: dc021662-bd8e-11e8-aaf5-af97e5bc218d from 0 (dbcluster03)

2018-09-21 14:09:50 139854478329600 [Note] WSREP: STATE EXCHANGE: got state msg: dc021662-bd8e-11e8-aaf5-af97e5bc218d from 1 (dbcluster02)

2018-09-21 14:09:51 139854478329600 [Note] WSREP: STATE EXCHANGE: got state msg: dc021662-bd8e-11e8-aaf5-af97e5bc218d from 2 (dbcluster04)

2018-09-21 14:09:51 139854478329600 [Note] WSREP: Quorum results:

        version    = 4,

        component  = PRIMARY,

        conf_id    = 67,

        members    = 2/3 (joined/total),

        act_id     = 19531,

        last_appl. = 19478,

        protocols  = 0/8/3 (gcs/repl/appl),

        group UUID = a6f20744-ba5c-11e8-9e46-c71bbc8deae3

2018-09-21 14:09:51 139854478329600 [Note] WSREP: Flow-control interval: [28, 28]

2018-09-21 14:09:51 139852213118720 [Note] WSREP: New cluster view: global state: a6f20744-ba5c-11e8-9e46-c71bbc8deae3:19531, view# 68: Primary, number of nodes: 3, my index: 0, protocol version 3

2018-09-21 14:09:51 139852213118720 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

2018-09-21 14:09:51 139852213118720 [Note] WSREP: REPL Protocols: 8 (3, 2)

2018-09-21 14:09:51 139852213118720 [Note] WSREP: Assign initial position for certification: 19531, protocol version: 3

2018-09-21 14:09:51 139854520293120 [Note] WSREP: Service thread queue flushed.

2018-09-21 14:09:53 139854486722304 [Note] WSREP: (29f5a00c, 'tcp://0.0.0.0:4567') turning message relay requesting off

2018-09-21 14:09:54 139851383834368 [Note] WSREP: intermediate transaction commit in LOAD DATA

2018-09-21 14:10:07 139851287860992 [Note] WSREP: intermediate transaction commit in LOAD DATA

2018-09-21 14:10:08 139852054628096 [Note] WSREP: intermediate transaction commit in LOAD DATA

2018-09-21 14:10:25 139852189087488 [Note] WSREP: intermediate transaction commit in LOAD DATA

2018-09-21 14:11:21 139852054320896 [ERROR] WSREP: FSM: no such a transition ABORTING -> REPLICATING

180921 14:11:21 [ERROR] mysqld got signal 6 ;

This could be because you hit a bug. It is also possible that this binary

or one of the libraries it was linked against is corrupt, improperly built,

or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help

diagnose the problem, but since we have already crashed,

something is definitely wrong and this may fail.

Server version: 10.2.14-MariaDB

key_buffer_size=134217728

read_buffer_size=131072

max_used_connections=24

max_threads=153

thread_count=18

It is possible that mysqld could use up to

key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467245 K  bytes of memory

Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f313c000c08

Attempting backtrace. You can use the following information to find out

where mysqld died. If you see no messages after this, something went

terribly wrong...

stack_bottom = 0x7f31d804ccd8 thread_stack 0x49000

/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55f9d36d118e]

/usr/sbin/mysqld(handle_fatal_signal+0x5a3)[0x55f9d31575f3]

sigaction.c:0(__restore_rt)[0x7f32711e81c0]

:0(__GI_raise)[0x7f326f118750]

:0(__GI_abort)[0x7f326f119d31]

/usr/lib64/galera/libgalera_smm.so(_ZN6galera3FSMINS_9TrxHandle5StateENS1_10TransitionENS_10EmptyGuardENS_11EmptyActionEE8shift_toES2_+0x190)[0x7f326b238850]

/usr/lib64/galera/libgalera_smm.so(_ZN6galera13ReplicatorSMM9replicateEPNS_9TrxHandleEP14wsrep_trx_meta+0x277)[0x7f326b231457]

/usr/lib64/galera/libgalera_smm.so(galera_pre_commit+0xb3)[0x7f326b251303]

/usr/sbin/mysqld(wsrep_run_wsrep_commit+0x987)[0x55f9d30ec147]

/usr/sbin/mysqld(+0x5e9d73)[0x55f9d30ecd73]

/usr/sbin/mysqld(_Z15ha_commit_transP3THDb+0x31f)[0x55f9d315a72f]

/usr/sbin/mysqld(_Z17trans_commit_stmtP3THD+0x5d)[0x55f9d309ff4d]

/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x3bf)[0x55f9d2fc0bff]

/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x2f3)[0x55f9d2fc9543]

/usr/sbin/mysqld(+0x4c6d16)[0x55f9d2fc9d16]

/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17b0)[0x55f9d2fcbad0]

/usr/sbin/mysqld(_Z10do_commandP3THD+0x230)[0x55f9d2fccfa0]

/usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x20a)[0x55f9d309130a]

/usr/sbin/mysqld(handle_one_connection+0x3d)[0x55f9d30914dd]

pthread_create.c:0(start_thread)[0x7f32711dd50b]

:0(__GI___clone)[0x7f326f1d938f]

Trying to get some variables.

Some pointers may be invalid and cause the dump to abort.

Query (0x7f313c00f2b0): LOAD DATA  LOCAL INFILE '/root/QA/mariadb-columnstore-tpcds/insert-data-tables/data/tpcds_2000/inventory.tbl' INTO TABLE `inventory` FIELDS  TERMINATED BY '|'  LINES TERMINATED BY '\n' IGNORE 0 LINES

Connection ID (thread ID): 23

Status: NOT_KILLED

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains

Attachments

Issue Links

relates to

MDEV-25104 Galera crashes with "FSM: no such a transition COMMITTED -> CERTIFYING"

Closed

Activity

People

Assignee:: Jan Lindström (Inactive)

Reporter:: Zdravelina Sokolovska (Inactive)

Votes:: 0 Vote for this issue

Watchers:: 6 Start watching this issue

Dates

Created:: 2018-09-21 12:25

Updated:: 2024-07-07 23:38

Resolved:: 2019-03-18 05:40

Git Integration

Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.