[MDEV-11053] [ERROR] mysqld got signal 11 ; Created: 2016-10-13  Updated: 2017-12-13  Resolved: 2017-12-13

Status: Closed
Project: MariaDB Server
Component/s: Galera
Affects Version/s: 10.1.18
Fix Version/s: 10.1.30

Type: Bug Priority: Major
Reporter: Albert Assignee: Sachin Setiya (Inactive)
Resolution: Duplicate Votes: 0
Labels: None
Environment:

Ubuntu Server 16.04 64bit, SSD hard disk, 3 node Galera cluster

Issue Links:
Relates
relates to MDEV-9510 Segmentation fault in binlog thread c... Closed

 Description   

We got the following signal 11. We run a 3 node galera cluster

161013 16:14:59 [ERROR] mysqld got signal 11 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.1.18-MariaDB-1~xenial
 
key_buffer_size=134217728
 
read_buffer_size=2097152
 
max_used_connections=210
 
max_threads=514
 
thread_count=191
 
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 3299624 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x0x7f6fe7412008
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x7f714ea790d8 thread_stack 0x48400
 
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x561730dfa62e]
 
/usr/sbin/mysqld(handle_fatal_signal+0x2d5)[0x5617309493d5]
 
/lib/x86_64-linux-gnu/libpthread.so.0(+0x113d0)[0x7f714d1ea3d0]
 
/usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG13mark_xid_doneEmb+0x8b)[0x5617309fe31b]
 
/usr/sbin/mysqld(binlog_background_thread+0x2f3)[0x5617309fe8f3]
 
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76fa)[0x7f714d1e06fa]
 
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f714c88bb5d]
 
 
 
Trying to get some variables.
 
Some pointers may be invalid and cause the dump to abort.
 
Query (0x0): 
Connection ID (thread ID): 3
 
Status: NOT_KILLED
 
 
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=off
 
 
 
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
 
information that should help you find out what is causing the crash.

____________________________________________________________________
Log entries immediately before this error are listed below, but are about 19 mins prior to the crash.

2016-10-13 14:43:59 140124334881536 [Warning] Aborted connection 428 to db: 'mm' user: 'root' host: '10.1.0.17' (Got an error reading communication packets)
 
2016-10-13 14:44:09 140118458153728 [Warning] Aborted connection 430 to db: 'mm' user: 'root' host: '10.1.0.17' (Got an error reading communication packets)
 
2016-10-13 15:34:00 7f714668bb00InnoDB: transactions deadlock detected, dumping detailed information.
 
2016-10-13 15:34:00 7f714668bb00
 
*** (1) TRANSACTION:
 
TRANSACTION 31187032, ACTIVE 0 sec starting index read
 
mysql tables in use 1, locked 1
 
LOCK WAIT 4 lock struct(s), heap size 1184, 2 row lock(s), undo log entries 3
 
MySQL thread id 590, OS thread handle 0x7f6fd9bf2b00, query id 3150943 10.1.0.7 root updating
 
UPDATE User SET FollowerCount = FollowerCount + 1 WHERE UserId=7042
 
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
 
RECORD LOCKS space id 570 page no 15 n bits 96 index `PRIMARY` of table `mm`.`User` trx table locks 2 total table locks 167  trx id 31187032 lock_mode X locks rec but not gap waiting lock hold time 0 wait time before grant 0 
*** (2) TRANSACTION:
 
TRANSACTION 31184036, ACTIVE 5 sec starting index read
 
mysql tables in use 1, locked 1
 
5 lock struct(s), heap size 1184, 2 row lock(s), undo log entries 1
 
MySQL thread id 534, OS thread handle 0x7f714668bb00, query id 3149292 10.1.0.17 root updating
 
UPDATE User SET FriendCount = FriendCount - 1 WHERE (UserId=48780 OR UserId=7042) AND FriendCount > 0
 
*** (2) HOLDS THE LOCK(S):
 
RECORD LOCKS space id 570 page no 15 n bits 96 index `PRIMARY` of table `mm`.`User` trx table locks 2 total table locks 167  trx id 31184036 lock_mode X locks rec but not gap lock hold time 1 wait time before grant 1 
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
 
RECORD LOCKS space id 570 page no 1353 n bits 104 index `PRIMARY` of table `mm`.`User` trx table locks 2 total table locks 167  trx id 31184036 lock_mode X locks rec but not gap waiting lock hold time 0 wait time before grant 0 
*** WE ROLL BACK TRANSACTION (2)
 
2016-10-13 15:34:00 140124489300736 [ERROR] mysqld: Deadlock found when trying to get lock; try restarting transaction
 
2016-10-13 15:34:00 140124489300736 [ERROR] mysqld: Deadlock found when trying to get lock; try restarting transaction
 
2016-10-13 15:37:11 7f6fd83feb00InnoDB: transactions deadlock detected, dumping detailed information.
 
2016-10-13 15:37:11 7f6fd83feb00
 
*** (1) TRANSACTION:
 
TRANSACTION 31269941, ACTIVE 1 sec starting index read
 
mysql tables in use 1, locked 1
 
LOCK WAIT 4 lock struct(s), heap size 1184, 2 row lock(s), undo log entries 3
 
MySQL thread id 487, OS thread handle 0x7f71269f7b00, query id 3510118 10.1.0.17 root updating
 
UPDATE User SET FollowerCount = FollowerCount + 1 WHERE UserId=7042
 
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
 
RECORD LOCKS space id 570 page no 15 n bits 96 index `PRIMARY` of table `mm`.`User` trx table locks 2 total table locks 105  trx id 31269941 lock_mode X locks rec but not gap waiting lock hold time 0 wait time before grant 0 
*** (2) TRANSACTION:
 
TRANSACTION 31265639, ACTIVE 8 sec starting index read
 
mysql tables in use 1, locked 1
 
6 lock struct(s), heap size 1184, 2 row lock(s), undo log entries 1
 
MySQL thread id 630, OS thread handle 0x7f6fd83feb00, query id 3501445 10.1.0.7 root updating
 
UPDATE User SET FriendCount = FriendCount - 1 WHERE (UserId=59316 OR UserId=7042) AND FriendCount > 0
 
*** (2) HOLDS THE LOCK(S):
 
RECORD LOCKS space id 570 page no 15 n bits 96 index `PRIMARY` of table `mm`.`User` trx table locks 2 total table locks 105  trx id 31265639 lock_mode X locks rec but not gap lock hold time 3 wait time before grant 3 
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
 
RECORD LOCKS space id 570 page no 646 n bits 112 index `PRIMARY` of table `mm`.`User` trx table locks 2 total table locks 105  trx id 31265639 lock_mode X locks rec but not gap waiting lock hold time 0 wait time before grant 0 
*** WE ROLL BACK TRANSACTION (2)
 
2016-10-13 15:37:11 140118346164992 [ERROR] mysqld: Deadlock found when trying to get lock; try restarting transaction
 
2016-10-13 15:37:11 140118346164992 [ERROR] mysqld: Deadlock found when trying to get lock; try restarting transaction
 
2016-10-13 15:37:12 7f7146769b00InnoDB: transactions deadlock detected, dumping detailed information.
 
2016-10-13 15:37:12 7f7146769b00
 
*** (1) TRANSACTION:
 
TRANSACTION 31269306, ACTIVE 2 sec starting index read
 
mysql tables in use 1, locked 1
 
LOCK WAIT 4 lock struct(s), heap size 1184, 2 row lock(s), undo log entries 3
 
MySQL thread id 550, OS thread handle 0x7f70b4ec0b00, query id 3507731 10.1.0.7 root updating
 
UPDATE User SET FollowerCount = FollowerCount + 1 WHERE UserId=7042
 
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
 
RECORD LOCKS space id 570 page no 15 n bits 96 index `PRIMARY` of table `mm`.`User` trx table locks 2 total table locks 131  trx id 31269306 lock_mode X locks rec but not gap waiting lock hold time 2 wait time before grant 0 
*** (2) TRANSACTION:
 
TRANSACTION 31267821, ACTIVE 4 sec starting index read
 
mysql tables in use 1, locked 1
 
5 lock struct(s), heap size 1184, 2 row lock(s), undo log entries 1
 
MySQL thread id 480, OS thread handle 0x7f7146769b00, query id 3506114 10.1.0.17 root updating
 
UPDATE User SET FriendCount = FriendCount - 1 WHERE (UserId=59325 OR UserId=7042) AND FriendCount > 0
 
*** (2) HOLDS THE LOCK(S):
 
RECORD LOCKS space id 570 page no 15 n bits 96 index `PRIMARY` of table `mm`.`User` trx table locks 2 total table locks 131  trx id 31267821 lock_mode X locks rec but not gap lock hold time 3 wait time before grant 3 
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
 
RECORD LOCKS space id 570 page no 646 n bits 112 index `PRIMARY` of table `mm`.`User` trx table locks 2 total table locks 131  trx id 31267821 lock_mode X locks rec but not gap waiting lock hold time 0 wait time before grant 0 
*** WE ROLL BACK TRANSACTION (2)
 
2016-10-13 15:37:12 140124490210048 [ERROR] mysqld: Deadlock found when trying to get lock; try restarting transaction
 
2016-10-13 15:37:12 140124490210048 [ERROR] mysqld: Deadlock found when trying to get lock; try restarting transaction
 
2016-10-13 15:51:23 140124628146944 [Note] InnoDB: Online DDL : Start
 
2016-10-13 15:51:23 140124628146944 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
 
2016-10-13 15:51:23 140124628146944 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
 
2016-10-13 15:51:23 140124628146944 [Note] InnoDB: Online DDL : Completed
 
2016-10-13 15:56:43 140124331850496 [Warning] Aborted connection 454 to db: 'mm' user: 'root' host: '10.1.0.17' (Got an error reading communication packets)
 
2016-10-13 15:56:43 140118863137536 [Warning] Aborted connection 465 to db: 'mm' user: 'root' host: '10.1.0.17' (Got an error reading communication packets)
 
2016-10-13 15:56:43 140118863743744 [Warning] Aborted connection 458 to db: 'mm' user: 'root' host: '10.1.0.17' (Got an error reading communication packets)
 
2016-10-13 15:56:43 140122047523584 [Warning] Aborted connection 457 to db: 'mm' user: 'root' host: '10.1.0.17' (Got an error reading communication packets)
 
2016-10-13 15:56:43 140120772283136 [Warning] Aborted connection 455 to db: 'mm' user: 'root' host: '10.1.0.17' (Got an error reading communication packets)
 
2016-10-13 15:56:43 140124272102144 [Warning] Aborted connection 466 to db: 'mm' user: 'root' host: '10.1.0.17' (Got an error reading communication packets)
 
2016-10-13 15:56:43 140118458759936 [Warning] Aborted connection 448 to db: 'mm' user: 'root' host: '10.1.0.17' (Got an error reading communication packets)
 
2016-10-13 15:56:43 140118667569920 [Warning] Aborted connection 453 to db: 'mm' user: 'root' host: '10.1.0.17' (Got an error reading communication packets)
 
2016-10-13 15:56:43 140120773192448 [Warning] Aborted connection 442 to db: 'mm' user: 'root' host: '10.1.0.17' (Got an error reading communication packets)
 
2016-10-13 15:56:43 140118668782336 [Warning] Aborted connection 444 to db: 'mm' user: 'root' host: '10.1.0.17' (Got an error reading communication packets)
 
2016-10-13 15:56:43 140124488997632 [Warning] Aborted connection 445 to db: 'mm' user: 'root' host: '10.1.0.17' (Got an error reading communication packets)
 
2016-10-13 15:56:43 140118668176128 [Warning] Aborted connection 461 to db: 'mm' user: 'root' host: '10.1.0.17' (Got an error reading communication packets)
 
2016-10-13 15:56:43 140118668479232 [Warning] Aborted connection 464 to db: 'mm' user: 'root' host: '10.1.0.17' (Got an error reading communication packets)
 
2016-10-13 15:56:43 140118864046848 [Warning] Aborted connection 459 to db: 'mm' user: 'root' host: '10.1.0.17' (Got an error reading communication packets)
 
2016-10-13 15:56:43 140122046917376 [Warning] Aborted connection 451 to db: 'mm' user: 'root' host: '10.1.0.17' (Got an error reading communication packets)
 
2016-10-13 15:56:43 140118459063040 [Warning] Aborted connection 447 to db: 'mm' user: 'root' host: '10.1.0.17' (Got an error reading communication packets)
 
2016-10-13 15:56:43 140120771373824 [Warning] Aborted connection 467 to db: 'mm' user: 'root' host: '10.1.0.17' (Got an error reading communication packets)
 
2016-10-13 15:56:43 140124575476480 [Warning] Aborted connection 439 to db: 'mm' user: 'root' host: '10.1.0.17' (Got an error reading communication packets)
 
2016-10-13 15:56:43 140124272708352 [Warning] Aborted connection 456 to db: 'mm' user: 'root' host: '10.1.0.17' (Got an error reading communication packets)
 
2016-10-13 15:56:43 140118865865472 [Warning] Aborted connection 443 to db: 'mm' user: 'root' host: '10.1.0.17' (Got an error reading communication packets)
 
2016-10-13 15:56:43 140124271495936 [Warning] Aborted connection 463 to db: 'mm' user: 'root' host: '10.1.0.17' (Got an error reading communication packets)
 
2016-10-13 15:56:43 140124489603840 [Warning] Aborted connection 462 to db: 'mm' user: 'root' host: '10.1.0.17' (Got an error reading communication packets)
 
2016-10-13 15:56:43 140120771980032 [Warning] Aborted connection 460 to db: 'mm' user: 'root' host: '10.1.0.17' (Got an error reading communication packets)
 
2016-10-13 15:56:43 140118669085440 [Warning] Aborted connection 440 to db: 'mm' user: 'root' host: '10.1.0.17' (Got an error reading communication packets)
 
2016-10-13 15:56:43 140124334275328 [Warning] Aborted connection 450 to db: 'mm' user: 'root' host: '10.1.0.17' (Got an error reading communication packets)
 
2016-10-13 15:56:43 140118459366144 [Warning] Aborted connection 452 to db: 'mm' user: 'root' host: '10.1.0.17' (Got an error reading communication packets)
 
2016-10-13 15:56:43 140118862531328 [Warning] Aborted connection 441 to db: 'mm' user: 'root' host: '10.1.0.17' (Got an error reading communication packets)
 
2016-10-13 15:56:43 140124333062912 [Warning] Aborted connection 449 to db: 'mm' user: 'root' host: '10.1.0.17' (Got an error reading communication packets)
 
2016-10-13 15:56:43 140118667266816 [Warning] Aborted connection 432 to db: 'mm' user: 'root' host: '10.1.0.17' (Got an error reading communication packets)
 
2016-10-13 15:56:43 140118665448192 [Warning] Aborted connection 446 to db: 'mm' user: 'root' host: '10.1.0.17' (Got an error reading communication packets)
 
161013 16:14:59 [ERROR] mysqld got signal 11 ;



 Comments   
Comment by Brendan P [ 2017-07-13 ]

I've managed to hit this bug yet again 3 times in a row and ended up destroying the entire tablespace on recovery (have to do full state transfer if 3TB of data). This is also a 3-way Galera cluster. I thought I had reported it at one point but I guess I have not.

It appears to happen like Russian roulette when using pt-online-schema-change doing simple things like adding a basic INDEX to a table, rather predictably.

170713 15:40:32 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.1.25-MariaDB-1~xenial
key_buffer_size=16777216
read_buffer_size=524288
max_used_connections=733
max_threads=4097
thread_count=293
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 18937492 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f0ed6c12008
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f32f652e0d8 thread_stack 0x48400
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5575b37f6bde]
/usr/sbin/mysqld(handle_fatal_signal+0x2f5)[0x5575b3342415]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f32f7f23390]
/usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG13mark_xid_doneEmb+0x8b)[0x5575b33f779b]
/usr/sbin/mysqld(binlog_background_thread+0x2f3)[0x5575b33f7dd3]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f32f7f196ba]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f32f75c43dd]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x0):
Connection ID (thread ID): 3
Status: NOT_KILLED

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=off

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

2017-07-13 15:31:49 140130052470528 [Warning] WSREP: Failed to report last committed 35663404647, -4 (Interrupted system call)
170713 15:33:36 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.1.25-MariaDB-1~xenial
key_buffer_size=16777216
read_buffer_size=524288
max_used_connections=853
max_threads=4097
thread_count=581
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 18937492 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f4f75012008
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f73931bc0d8 thread_stack 0x48400
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55b9f6858bde]
/usr/sbin/mysqld(handle_fatal_signal+0x2f5)[0x55b9f63a4415]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f73961e9390]
/usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG13mark_xid_doneEmb+0x8b)[0x55b9f645979b]
/usr/sbin/mysqld(binlog_background_thread+0x2f3)[0x55b9f6459dd3]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f73961df6ba]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f739588a3dd]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x0):
Connection ID (thread ID): 3
Status: NOT_KILLED

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=off

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

170713 15:23:23 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.1.25-MariaDB-1~xenial
key_buffer_size=16777216
read_buffer_size=524288
max_used_connections=2050
max_threads=4097
thread_count=591
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 18937492 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f3193c12008
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f55b20d80d8 thread_stack 0x48400
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55843101fbde]
/usr/sbin/mysqld(handle_fatal_signal+0x2f5)[0x558430b6b415]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f55b3acd390]
/usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG13mark_xid_doneEmb+0x8b)[0x558430c2079b]
/usr/sbin/mysqld(binlog_background_thread+0x2f3)[0x558430c20dd3]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f55b3ac36ba]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f55b316e3dd]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x0):
Connection ID (thread ID): 3
Status: NOT_KILLED

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=off

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

Comment by Tim White [ 2017-09-25 ]

We have this same setup, and exact same issue. 3 Node Ubuntu 16.04 cluster died randomly the other night. Node 1 died first, followed by node 3 shortly after. Only shutting down the entire cluster and bring it backup as a new cluster worked (restarting nodes didn't work).

Mariadb Version 10.1.26-MariaDB-1~xenial

Sep 21 22:06:25 mariadb-2-1 mysqld[3979]: 170921 22:06:25 [ERROR] mysqld got signal 11 ;
 
Sep 21 22:06:25 mariadb-2-1 mysqld[3979]: This could be because you hit a bug. It is also possible that this binary
 
Sep 21 22:06:25 mariadb-2-1 mysqld[3979]: or one of the libraries it was linked against is corrupt, improperly built,
 
Sep 21 22:06:25 mariadb-2-1 mysqld[3979]: or misconfigured. This error can also be caused by malfunctioning hardware.
 
Sep 21 22:06:25 mariadb-2-1 mysqld[3979]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
Sep 21 22:06:25 mariadb-2-1 mysqld[3979]: We will try our best to scrape up some info that will hopefully help
 
Sep 21 22:06:25 mariadb-2-1 mysqld[3979]: diagnose the problem, but since we have already crashed,
 
Sep 21 22:06:25 mariadb-2-1 mysqld[3979]: something is definitely wrong and this may fail.
 
Sep 21 22:06:25 mariadb-2-1 mysqld[3979]: Server version: 10.1.26-MariaDB-1~xenial
 
Sep 21 22:06:25 mariadb-2-1 mysqld[3979]: key_buffer_size=268435456
 
Sep 21 22:06:25 mariadb-2-1 mysqld[3979]: read_buffer_size=2097152
 
Sep 21 22:06:25 mariadb-2-1 mysqld[3979]: max_used_connections=14
 
Sep 21 22:06:25 mariadb-2-1 mysqld[3979]: max_threads=152
 
Sep 21 22:06:25 mariadb-2-1 mysqld[3979]: thread_count=4
 
Sep 21 22:06:25 mariadb-2-1 mysqld[3979]: It is possible that mysqld could use up to
 
Sep 21 22:06:25 mariadb-2-1 mysqld[3979]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1199145 K  bytes of memory
 
Sep 21 22:06:25 mariadb-2-1 mysqld[3979]: Hope that's ok; if not, decrease some variables in the equation.
 
Sep 21 22:06:25 mariadb-2-1 mysqld[3979]: Thread pointer: 0x7fea6a812008
 
Sep 21 22:06:25 mariadb-2-1 mysqld[3979]: Attempting backtrace. You can use the following information to find out
 
Sep 21 22:06:25 mariadb-2-1 mysqld[3979]: where mysqld died. If you see no messages after this, something went
 
Sep 21 22:06:25 mariadb-2-1 mysqld[3979]: terribly wrong...
 
Sep 21 22:06:25 mariadb-2-1 mysqld[3979]: stack_bottom = 0x7fec869420d8 thread_stack 0x48400
 
Sep 21 22:06:25 mariadb-2-1 mysqld[3979]: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x33d8835efe]
 
Sep 21 22:06:25 mariadb-2-1 mysqld[3979]: /usr/sbin/mysqld(handle_fatal_signal+0x2f5)[0x33d837f435]
 
Sep 21 22:06:25 mariadb-2-1 mysqld[3979]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7fec850b3390]
 
Sep 21 22:06:25 mariadb-2-1 mysqld[3979]: /usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG13mark_xid_doneEmb+0x8b)[0x33d843537b]
 
Sep 21 22:06:25 mariadb-2-1 mysqld[3979]: /usr/sbin/mysqld(binlog_background_thread+0x2f3)[0x33d84359b3]
 
Sep 21 22:06:25 mariadb-2-1 mysqld[3979]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7fec850a96ba]
 
Sep 21 22:06:25 mariadb-2-1 mysqld[3979]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fec847543dd]
 
Sep 21 22:06:25 mariadb-2-1 mysqld[3979]: Trying to get some variables.
 
Sep 21 22:06:25 mariadb-2-1 mysqld[3979]: Some pointers may be invalid and cause the dump to abort.
 
Sep 21 22:06:25 mariadb-2-1 mysqld[3979]: Query (0x0):
 
Sep 21 22:06:25 mariadb-2-1 mysqld[3979]: Connection ID (thread ID): 3
 
Sep 21 22:06:25 mariadb-2-1 mysqld[3979]: Status: NOT_KILLED
 
Sep 21 22:06:25 mariadb-2-1 mysqld[3979]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=off
 
Sep 21 22:06:25 mariadb-2-1 mysqld[3979]: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
 
Sep 21 22:06:25 mariadb-2-1 mysqld[3979]: information that should help you find out what is causing the crash.
 
Sep 21 22:08:34 mariadb-2-1 systemd[1]: mariadb.service: Main process exited, code=killed, status=11/SEGV
 
Sep 21 22:08:34 mariadb-2-1 systemd[1]: mariadb.service: Unit entered failed state.
 
Sep 21 22:08:34 mariadb-2-1 systemd[1]: mariadb.service: Failed with result 'signal'.
 
Sep 21 22:08:39 mariadb-2-1 systemd[1]: mariadb.service: Service hold-off time over, scheduling restart.
 
Sep 21 22:08:39 mariadb-2-1 systemd[1]: Stopped MariaDB database server.
 
 
 
Sep 21 22:08:39 mariadb-2-1 systemd[1]: Starting MariaDB database server...
 
Sep 21 22:08:40 mariadb-2-1 mysqld[5507]: 2017-09-21 22:08:40 140564845336832 [Note] /usr/sbin/mysqld (mysqld 10.1.26-MariaDB-1~xenial) starting as process 5769 ...
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: Segmentation fault (core dumped)
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: WSREP: Failed to start mysqld for wsrep recovery: '2017-09-21 22:08:40 140564845336832 [Note] InnoDB: Using mutexes to ref count buffer pool pages
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: 2017-09-21 22:08:40 140564845336832 [Note] InnoDB: The InnoDB memory heap is disabled
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: 2017-09-21 22:08:40 140564845336832 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: 2017-09-21 22:08:40 140564845336832 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: 2017-09-21 22:08:40 140564845336832 [Note] InnoDB: Compressed tables use zlib 1.2.8
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: 2017-09-21 22:08:40 140564845336832 [Note] InnoDB: Using Linux native AIO
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: 2017-09-21 22:08:40 140564845336832 [Note] InnoDB: Using SSE crc32 instructions
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: 2017-09-21 22:08:40 140564845336832 [Note] InnoDB: Initializing buffer pool, size = 7.0G
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: 2017-09-21 22:08:40 140564845336832 [Note] InnoDB: Completed initialization of buffer pool
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: 2017-09-21 22:08:40 140564845336832 [Note] InnoDB: Highest supported file format is Barracuda.
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: 2017-09-21 22:08:40 140564845336832 [Note] InnoDB: Starting crash recovery from checkpoint LSN=32935648798
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: 2017-09-21 22:08:47 140564845336832 [Note] InnoDB: Restoring possible half-written data pages from the doublewrite buffer...
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: InnoDB: Transaction 18738117 was in the XA prepared state.
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: InnoDB: 1 transaction(s) which must be rolled back or cleaned up
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: InnoDB: in total 0 row operations to undo
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: InnoDB: Trx id counter is 18738432
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: 2017-09-21 22:08:48 140564845336832 [Note] InnoDB: Starting final batch to recover 6 pages from redo log
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: InnoDB: Last MySQL binlog file position 0 603, file name /var/log/mysql/mariadb-bin.000306
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: 2017-09-21 22:08:51 140564845336832 [Note] InnoDB: 128 rollback segment(s) are active.
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: 2017-09-21 22:08:51 140555946288896 [Note] InnoDB: Starting in background the rollback of recovered transactions
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: 2017-09-21 22:08:51 140555946288896 [Note] InnoDB: Rollback of non-prepared transactions completed
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: 2017-09-21 22:08:51 140564845336832 [Note] InnoDB: Waiting for purge to start
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: 2017-09-21 22:08:51 140564845336832 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.36-82.1 started; log sequence number 32935651364
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: 2017-09-21 22:08:51 140564845336832 [Warning] InnoDB: Skipping buffer pool dump/restore during wsrep recovery.
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: 2017-09-21 22:08:51 140564845336832 [Note] Plugin 'FEEDBACK' is disabled.
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: 2017-09-21 22:08:51 140564845336832 [Note] Recovering after a crash using /var/log/mysql/mariadb-bin
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: 2017-09-21 22:08:51 140564845336832 [Note] Starting crash recovery...
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: 2017-09-21 22:08:51 7fd7cdac9900  InnoDB: Starting recovery for XA transactions...
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: 2017-09-21 22:08:51 7fd7cdac9900  InnoDB: Transaction 18738117 in prepared state after recovery
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: 2017-09-21 22:08:51 7fd7cdac9900  InnoDB: Transaction contains changes to 1 rows
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: 2017-09-21 22:08:51 7fd7cdac9900  InnoDB: 1 transactions in prepared state after recovery
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: 2017-09-21 22:08:51 140564845336832 [Note] Found 1 prepared transaction(s) in InnoDB
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: 170921 22:08:51 [ERROR] mysqld got signal 11 ;
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: This could be because you hit a bug. It is also possible that this binary
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: or one of the libraries it was linked against is corrupt, improperly built,
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: or misconfigured. This error can also be caused by malfunctioning hardware.
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: We will try our best to scrape up some info that will hopefully help
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: diagnose the problem, but since we have already crashed,
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: something is definitely wrong and this may fail.
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: Server version: 10.1.26-MariaDB-1~xenial
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: key_buffer_size=268435456
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: read_buffer_size=2097152
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: max_used_connections=0
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: max_threads=152
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: thread_count=0
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: It is possible that mysqld could use up to
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1199145 K  bytes of memory
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: Hope that's ok; if not, decrease some variables in the equation.
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: Thread pointer: 0x0
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: Attempting backtrace. You can use the following information to find out
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: where mysqld died. If you see no messages after this, something went
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: terribly wrong...
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: stack_bottom = 0x0 thread_stack 0x48400
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x461407cefe]
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: /usr/sbin/mysqld(handle_fatal_signal+0x2f5)[0x4613bc6435]
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7fd7cc16c390]
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: /usr/sbin/mysqld(wsrep_on+0x18)[0x4613b60488]
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: /usr/sbin/mysqld(+0x80e602)[0x4613e35602]
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: /usr/sbin/mysqld(+0x80ec31)[0x4613e35c31]
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: /usr/sbin/mysqld(+0x5a0c1f)[0x4613bc7c1f]
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: /usr/sbin/mysqld(_Z24plugin_foreach_with_maskP3THDPFcS0_P13st_plugin_intPvEijS3_+0x2cf)[0x4613a4170f]
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: /usr/sbin/mysqld(_Z10ha_recoverP7st_hash+0xc6)[0x4613bca0c6]
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: /usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG7recoverEP11st_log_infoPKcP11st_io_cacheP28Format_description_log_eventb+0x55f)[0x4613c7d14f]
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: /usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG18do_binlog_recoveryEPKcb+0x372)[0x4613c7d7d2]
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: /usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG4openEPKc+0x30)[0x4613c809d0]
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: /usr/sbin/mysqld(+0x377615)[0x461399e615]
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: /usr/sbin/mysqld(_Z11mysqld_mainiPPc+0x1b09)[0x46139a18b9]
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7fd7cb726830]
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: /usr/sbin/mysqld(_start+0x29)[0x4613995649]
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
 
Sep 21 22:09:37 mariadb-2-1 mysqld[5507]: information that should help you find out what is causing the crash.'
 
Sep 21 22:09:37 mariadb-2-1 systemd[1]: mariadb.service: Control process exited, code=exited status=1
 
Sep 21 22:09:37 mariadb-2-1 systemd[1]: Failed to start MariaDB database server.
 
Sep 21 22:09:37 mariadb-2-1 systemd[1]: mariadb.service: Unit entered failed state.
 



Sep 21 22:06:28 mariadb-2-3 mysqld[59910]: 2017-09-21 22:06:28 139891660814080 [Note] WSREP: (1e376bbd, 'ssl://0.0.0.0:4567') connection to peer 08dd5586 with addr ssl://10.1.4.91:4567 timed out, no messages seen in PT3S
 
Sep 21 22:06:28 mariadb-2-3 mysqld[59910]: 2017-09-21 22:06:28 139891660814080 [Note] WSREP: (1e376bbd, 'ssl://0.0.0.0:4567') turning message relay requesting on, nonlive peers: ssl://10.1.4.91:4567
 
Sep 21 22:06:29 mariadb-2-3 mysqld[59910]: 2017-09-21 22:06:29 139891660814080 [Note] WSREP: (1e376bbd, 'ssl://0.0.0.0:4567') reconnecting to 08dd5586 (ssl://10.1.4.91:4567), attempt 0
 
Sep 21 22:06:30 mariadb-2-3 mysqld[59910]: 2017-09-21 22:06:30 139891660814080 [Note] WSREP: evs::proto(1e376bbd, OPERATIONAL, view_id(REG,08dd5586,169)) suspecting node: 08dd5586
 
Sep 21 22:06:30 mariadb-2-3 mysqld[59910]: 2017-09-21 22:06:30 139891660814080 [Note] WSREP: evs::proto(1e376bbd, OPERATIONAL, view_id(REG,08dd5586,169)) suspected node without join message, declaring inactive
 
Sep 21 22:06:31 mariadb-2-3 mysqld[59910]: 2017-09-21 22:06:31 139891660814080 [Note] WSREP: declaring 22680dcf at ssl://10.1.4.92:4567 stable
 
Sep 21 22:06:31 mariadb-2-3 mysqld[59910]: 2017-09-21 22:06:31 139891660814080 [Note] WSREP: Node 1e376bbd state prim
 
Sep 21 22:06:31 mariadb-2-3 mysqld[59910]: 2017-09-21 22:06:31 139891660814080 [Note] WSREP: view(view_id(PRIM,1e376bbd,170) memb {
 
Sep 21 22:06:31 mariadb-2-3 mysqld[59910]: #0111e376bbd,0
 
Sep 21 22:06:31 mariadb-2-3 mysqld[59910]: #01122680dcf,0
 
Sep 21 22:06:31 mariadb-2-3 mysqld[59910]: } joined {
 
Sep 21 22:06:31 mariadb-2-3 mysqld[59910]: } left {
 
Sep 21 22:06:31 mariadb-2-3 mysqld[59910]: } partitioned {
 
Sep 21 22:06:31 mariadb-2-3 mysqld[59910]: #01108dd5586,0
 
Sep 21 22:06:31 mariadb-2-3 mysqld[59910]: })
 
Sep 21 22:06:31 mariadb-2-3 mysqld[59910]: 2017-09-21 22:06:31 139891660814080 [Note] WSREP: save pc into disk
 
Sep 21 22:06:31 mariadb-2-3 mysqld[59910]: 2017-09-21 22:06:31 139891644036864 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
 
Sep 21 22:06:31 mariadb-2-3 mysqld[59910]: 2017-09-21 22:06:31 139891660814080 [Note] WSREP: forgetting 08dd5586 (ssl://10.1.4.91:4567)
 
Sep 21 22:06:31 mariadb-2-3 mysqld[59910]: 2017-09-21 22:06:31 139891660814080 [Note] WSREP: deleting entry ssl://10.1.4.91:4567
 
Sep 21 22:06:31 mariadb-2-3 mysqld[59910]: 2017-09-21 22:06:31 139891660814080 [Note] WSREP: (1e376bbd, 'ssl://0.0.0.0:4567') turning message relay requesting off
 
Sep 21 22:06:31 mariadb-2-3 mysqld[59910]: 2017-09-21 22:06:31 139891644036864 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 4e3d961a-9ec5-11e7-a6d2-2a9438a0bb8f
 
Sep 21 22:06:31 mariadb-2-3 mysqld[59910]: 2017-09-21 22:06:31 139891644036864 [Note] WSREP: STATE EXCHANGE: sent state msg: 4e3d961a-9ec5-11e7-a6d2-2a9438a0bb8f
 
Sep 21 22:06:31 mariadb-2-3 mysqld[59910]: 2017-09-21 22:06:31 139891644036864 [Note] WSREP: STATE EXCHANGE: got state msg: 4e3d961a-9ec5-11e7-a6d2-2a9438a0bb8f from 0 (mariadb-2-3)
 
Sep 21 22:06:31 mariadb-2-3 mysqld[59910]: 2017-09-21 22:06:31 139891644036864 [Note] WSREP: STATE EXCHANGE: got state msg: 4e3d961a-9ec5-11e7-a6d2-2a9438a0bb8f from 1 (mariadb-2-2)
 
Sep 21 22:06:31 mariadb-2-3 mysqld[59910]: 2017-09-21 22:06:31 139891644036864 [Note] WSREP: Quorum results:
 
Sep 21 22:06:31 mariadb-2-3 mysqld[59910]: #011version    = 4,
 
Sep 21 22:06:31 mariadb-2-3 mysqld[59910]: #011component  = PRIMARY,
 
Sep 21 22:06:31 mariadb-2-3 mysqld[59910]: #011conf_id    = 13,
 
Sep 21 22:06:31 mariadb-2-3 mysqld[59910]: #011members    = 2/2 (joined/total),
 
Sep 21 22:06:31 mariadb-2-3 mysqld[59910]: #011act_id     = 3881647,
 
Sep 21 22:06:31 mariadb-2-3 mysqld[59910]: #011last_appl. = 3881632,
 
Sep 21 22:06:31 mariadb-2-3 mysqld[59910]: #011protocols  = 0/7/3 (gcs/repl/appl),
 
Sep 21 22:06:31 mariadb-2-3 mysqld[59910]: #011group UUID = 95d66ad5-6762-11e7-9b19-6e1636e154cc
 
Sep 21 22:06:31 mariadb-2-3 mysqld[59910]: 2017-09-21 22:06:31 139891644036864 [Note] WSREP: Flow-control interval: [23, 23]
 
Sep 21 22:06:31 mariadb-2-3 mysqld[59910]: 2017-09-21 22:06:31 139892004408064 [Note] WSREP: New cluster view: global state: 95d66ad5-6762-11e7-9b19-6e1636e154cc:3881647, view# 14: Primary, number of nodes: 2, my index: 0, protocol version 3
 
Sep 21 22:06:31 mariadb-2-3 mysqld[59910]: 2017-09-21 22:06:31 139892004408064 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
Sep 21 22:06:31 mariadb-2-3 mysqld[59910]: 2017-09-21 22:06:31 139892004408064 [Note] WSREP: REPL Protocols: 7 (3, 2)
 
Sep 21 22:06:31 mariadb-2-3 mysqld[59910]: 2017-09-21 22:06:31 139892004408064 [Note] WSREP: Assign initial position for certification: 3881647, protocol version: 3
 
Sep 21 22:06:31 mariadb-2-3 mysqld[59910]: 2017-09-21 22:06:31 139891710375680 [Note] WSREP: Service thread queue flushed.
 
Sep 21 22:06:34 mariadb-2-3 mysqld[59910]: 2017-09-21 22:06:34 139891660814080 [Note] WSREP:  cleaning up 08dd5586 (ssl://10.1.4.91:4567)
 
Sep 21 22:24:13 mariadb-2-3 mysqld[59910]: 170921 22:24:13 [ERROR] mysqld got signal 11 ;
 
Sep 21 22:24:13 mariadb-2-3 mysqld[59910]: This could be because you hit a bug. It is also possible that this binary
 
Sep 21 22:24:13 mariadb-2-3 mysqld[59910]: or one of the libraries it was linked against is corrupt, improperly built,
 
Sep 21 22:24:13 mariadb-2-3 mysqld[59910]: or misconfigured. This error can also be caused by malfunctioning hardware.
 
Sep 21 22:24:13 mariadb-2-3 mysqld[59910]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
Sep 21 22:24:13 mariadb-2-3 mysqld[59910]: We will try our best to scrape up some info that will hopefully help
 
Sep 21 22:24:13 mariadb-2-3 mysqld[59910]: diagnose the problem, but since we have already crashed,
 
Sep 21 22:24:13 mariadb-2-3 mysqld[59910]: something is definitely wrong and this may fail.
 
Sep 21 22:24:13 mariadb-2-3 mysqld[59910]: Server version: 10.1.26-MariaDB-1~xenial
 
Sep 21 22:24:13 mariadb-2-3 mysqld[59910]: key_buffer_size=268435456
 
Sep 21 22:24:13 mariadb-2-3 mysqld[59910]: read_buffer_size=2097152
 
Sep 21 22:24:13 mariadb-2-3 mysqld[59910]: max_used_connections=13
 
Sep 21 22:24:13 mariadb-2-3 mysqld[59910]: max_threads=152
 
Sep 21 22:24:13 mariadb-2-3 mysqld[59910]: thread_count=8
 
Sep 21 22:24:13 mariadb-2-3 mysqld[59910]: It is possible that mysqld could use up to
 
Sep 21 22:24:13 mariadb-2-3 mysqld[59910]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1199145 K  bytes of memory
 
Sep 21 22:24:13 mariadb-2-3 mysqld[59910]: Hope that's ok; if not, decrease some variables in the equation.
 
Sep 21 22:24:13 mariadb-2-3 mysqld[59910]: Thread pointer: 0x7f3908c12008
 
Sep 21 22:24:13 mariadb-2-3 mysqld[59910]: Attempting backtrace. You can use the following information to find out
 
Sep 21 22:24:13 mariadb-2-3 mysqld[59910]: where mysqld died. If you see no messages after this, something went
 
Sep 21 22:24:13 mariadb-2-3 mysqld[59910]: terribly wrong...
 
Sep 21 22:24:13 mariadb-2-3 mysqld[59910]: stack_bottom = 0x7f3b2532b0d8 thread_stack 0x48400
 
Sep 21 22:24:13 mariadb-2-3 mysqld[59910]: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0xfba019efe]
 
Sep 21 22:24:13 mariadb-2-3 mysqld[59910]: /usr/sbin/mysqld(handle_fatal_signal+0x2f5)[0xfb9b63435]
 
Sep 21 22:24:13 mariadb-2-3 mysqld[59910]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f3b23a9c390]
 
Sep 21 22:24:13 mariadb-2-3 mysqld[59910]: /usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG13mark_xid_doneEmb+0x8b)[0xfb9c1937b]
 
Sep 21 22:24:13 mariadb-2-3 mysqld[59910]: /usr/sbin/mysqld(binlog_background_thread+0x2f3)[0xfb9c199b3]
 
Sep 21 22:24:13 mariadb-2-3 mysqld[59910]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f3b23a926ba]
 
Sep 21 22:24:13 mariadb-2-3 mysqld[59910]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f3b2313d3dd]
 
Sep 21 22:24:13 mariadb-2-3 mysqld[59910]: Trying to get some variables.
 
Sep 21 22:24:13 mariadb-2-3 mysqld[59910]: Some pointers may be invalid and cause the dump to abort.
 
Sep 21 22:24:13 mariadb-2-3 mysqld[59910]: Query (0x0):
 
Sep 21 22:24:13 mariadb-2-3 mysqld[59910]: Connection ID (thread ID): 3
 
Sep 21 22:24:13 mariadb-2-3 mysqld[59910]: Status: NOT_KILLED
 
Sep 21 22:24:13 mariadb-2-3 mysqld[59910]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=off
 
Sep 21 22:24:13 mariadb-2-3 mysqld[59910]: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
 
Sep 21 22:24:13 mariadb-2-3 mysqld[59910]: information that should help you find out what is causing the crash.
 
Sep 21 22:24:13 mariadb-2-3 systemd[1]: mariadb.service: Main process exited, code=killed, status=11/SEGV
 
Sep 21 22:24:13 mariadb-2-3 systemd[1]: mariadb.service: Unit entered failed state.
 
Sep 21 22:24:13 mariadb-2-3 systemd[1]: mariadb.service: Failed with result 'signal'.
 
Sep 21 22:24:18 mariadb-2-3 systemd[1]: mariadb.service: Service hold-off time over, scheduling restart.
 
Sep 21 22:24:18 mariadb-2-3 systemd[1]: Stopped MariaDB database server.

Comment by Andrei Elkin [ 2017-12-13 ]

Fixed by MDEV-9510.

Generated at Thu Feb 08 07:46:56 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.