Details
-
Bug
-
Status: Open (View Workflow)
-
Major
-
Resolution: Unresolved
-
10.1.21, 10.1.26
-
None
-
None
-
None
Description
We have MariaDB instance with very often 'signal 11' crashes. Most of them happen with reference to unlink_thd() function, which is not present anymore in MySQL 5.6+, so I assume this may be specific to MariaDB.
171010 12:33:21 server_audit: MariaDB Audit Plugin version 1.4.1 STARTED.
|
Version: '10.1.26-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server
|
It is not clear what exactly queries/transactions cause these crashes.
Example crash with related thread gdb traces (no full gdb attached due to sensitive information) below. Additional, slightly different occurences in attachment.
171010 12:28:49 [ERROR] mysqld got signal 11 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
Server version: 10.1.26-MariaDB
|
key_buffer_size=1073741824
|
read_buffer_size=131072
|
max_used_connections=1058
|
max_threads=2002
|
thread_count=457
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 5446182 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
Thread pointer: 0x7f282328f008
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x7f2783c90080 thread_stack 0x48400
|
*** ***: /usr/sbin/mysqld terminated
|
======= Backtrace: =========
|
/lib64/libc.so.6(__fortify_fail+0x37)[0x7f5e7fc7cd87]
|
/lib64/libc.so.6(+0x10df40)[0x7f5e7fc7af40]
|
/lib64/libc.so.6(+0x10fcf7)[0x7f5e7fc7ccf7]
|
/usr/sbin/mysqld(my_addr_resolve+0x48)[0x55f45696b088]
|
/usr/sbin/mysqld(my_print_stacktrace+0x1c2)[0x55f4569576c2]
|
/usr/sbin/mysqld(handle_fatal_signal+0x305)[0x55f45647aed5]
|
/lib64/libpthread.so.0(+0xf5e0)[0x7f5e818c95e0]
|
/usr/sbin/mysqld(_Z17ha_rollback_transP3THDb+0xab)[0x55f45647da1b]
|
/usr/sbin/mysqld(_Z14trans_rollbackP3THD+0x5a)[0x55f4563d6a9a]
|
/usr/sbin/mysqld(_ZN3THD7cleanupEv+0xe2)[0x55f4562c51b2]
|
/usr/sbin/mysqld(_Z10unlink_thdP3THD+0x17)[0x55f456257ad7]
|
/usr/sbin/mysqld(_Z29one_thread_per_connection_endP3THDb+0x20)[0x55f456257c90]
|
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x113)[0x55f4563c8963]
|
/usr/sbin/mysqld(handle_one_connection+0x40)[0x55f4563c8b80]
|
/usr/sbin/mysqld(+0x74be8d)[0x55f456607e8d]
|
/lib64/libpthread.so.0(+0x7e25)[0x7f5e818c1e25]
|
/lib64/libc.so.6(clone+0x6d)[0x7f5e7fc6534d]
|
======= Memory map: ========
|
55f455ebc000-55f456e46000 r-xp 00000000 fb:00 134770637 /usr/sbin/mysqld
|
55f457046000-55f4570f7000 r--p 00f8a000 fb:00 134770637 /usr/sbin/mysqld
|
55f4570f7000-55f4571a9000 rw-p 0103b000 fb:00 134770637 /usr/sbin/mysqld
|
(...)
|
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
|
Fatal signal 6 while backtracing
|
2017-10-10 12:32:44 139933296474368 [Note] Using unique option prefix 'myisam-recover' is error-prone and can break in the future. Please use the full name 'myisam-recover-options' instead.
|
2017-10-10 12:32:44 139933296474368 [Note] InnoDB: Using mutexes to ref count buffer pool pages
|
2017-10-10 12:32:44 139933296474368 [Note] InnoDB: The InnoDB memory heap is disabled
|
2017-10-10 12:32:44 139933296474368 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
|
2017-10-10 12:32:44 139933296474368 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
|
2017-10-10 12:32:44 139933296474368 [Note] InnoDB: Compressed tables use zlib 1.2.7
|
2017-10-10 12:32:44 139933296474368 [Note] InnoDB: Using Linux native AIO
|
2017-10-10 12:32:44 139933296474368 [Note] InnoDB: Using SSE crc32 instructions
|
2017-10-10 12:32:44 139933296474368 [Note] InnoDB: Initializing buffer pool, size = 200.0G
|
2017-10-10 12:32:51 139933296474368 [Note] InnoDB: Completed initialization of buffer pool
|
2017-10-10 12:32:51 139933296474368 [Note] InnoDB: Highest supported file format is Barracuda.
|
2017-10-10 12:32:51 139933296474368 [Note] InnoDB: Starting crash recovery from checkpoint LSN=49954283878443
|
2017-10-10 12:32:52 139933296474368 [Note] InnoDB: Restoring possible half-written data pages from the doublewrite buffer...
|
InnoDB: 4 transaction(s) which must be rolled back or cleaned up
|
InnoDB: in total 4 row operations to undo
|
InnoDB: Trx id counter is 36935782400
|
2017-10-10 12:33:04 139933296474368 [Note] InnoDB: Starting final batch to recover 127478 pages from redo log
|
2017-10-10 12:33:06 139701654628096 [Note] InnoDB: To recover: 92197 pages from log
|
InnoDB: Last MySQL binlog file position 0 575403140, file name /home/mysql/logs/mysql.002397
|
2017-10-10 12:33:11 139933296474368 [Note] InnoDB: 128 rollback segment(s) are active.
|
2017-10-10 12:33:11 139698190149376 [Note] InnoDB: Starting in background the rollback of recovered transactions
|
2017-10-10 12:33:11 7f0e04ffe700 InnoDB: Rolling back trx with id 36935781973, 1 rows to undo
|
2017-10-10 12:33:11 139933296474368 [Note] InnoDB: Waiting for purge to start
|
2017-10-10 12:33:11 139698190149376 [Note] InnoDB: Rollback of trx with id 36935781973 completed
|
2017-10-10 12:33:11 7f0e04ffe700 InnoDB: Rolling back trx with id 36935781972, 1 rows to undo
|
2017-10-10 12:33:11 139698190149376 [Note] InnoDB: Rollback of trx with id 36935781972 completed
|
2017-10-10 12:33:11 7f0e04ffe700 InnoDB: Rolling back trx with id 36935781970, 1 rows to undo
|
2017-10-10 12:33:11 139698190149376 [Note] InnoDB: Rollback of trx with id 36935781970 completed
|
2017-10-10 12:33:11 7f0e04ffe700 InnoDB: Rolling back trx with id 36935781952, 1 rows to undo
|
2017-10-10 12:33:11 139698190149376 [Note] InnoDB: Rollback of trx with id 36935781952 completed
|
2017-10-10 12:33:11 139698190149376 [Note] InnoDB: Rollback of non-prepared transactions completed
|
2017-10-10 12:33:11 139933296474368 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.36-82.1 started; log sequence number 49954847609095
|
2017-10-10 12:33:21 139933296474368 [Note] Plugin 'FEEDBACK' is disabled.
|
171010 12:33:21 server_audit: MariaDB Audit Plugin version 1.4.1 STARTED.
|
2017-10-10 12:33:21 139697242212096 [Note] InnoDB: Dumping buffer pool(s) not yet started
|
2017-10-10 12:33:21 139933296474368 [Note] Recovering after a crash using /home/mysql/logs/mysql
|
2017-10-10 12:33:22 139933296474368 [Note] Starting crash recovery...
|
2017-10-10 12:33:22 139933296474368 [Note] Crash recovery finished.
|
2017-10-10 12:33:22 139933296474368 [Note] Server socket created on IP: '::'.
|
2017-10-10 12:33:22 139933296474368 [Note] /usr/sbin/mysqld: ready for connections.
|
Version: '10.1.26-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server
|
Related thread gdb standard:
Thread 1 (Thread 0x7f2783c90b00 (LWP 22144)):
|
+bt
|
#0 0x00007f5e7fba21f7 in raise () from /lib64/libc.so.6
|
#1 0x00007f5e7fba38e8 in abort () from /lib64/libc.so.6
|
#2 0x00007f5e7fbe1f47 in __libc_message () from /lib64/libc.so.6
|
#3 0x00007f5e7fc7cd87 in __fortify_fail () from /lib64/libc.so.6
|
#4 0x00007f5e7fc7af40 in __chk_fail () from /lib64/libc.so.6
|
#5 0x00007f5e7fc7ccf7 in __fdelt_warn () from /lib64/libc.so.6
|
#6 0x000055f45696b088 in my_addr_resolve (ptr=0x55f45695752e <my_print_stacktrace+46>, loc=loc@entry=0x7f2783c8f3f0) at /home/buildbot/buildbot/build/mariadb-10.1.26/mysys/my_addr_resolve.c:162
|
#7 0x000055f4569576c2 in print_with_addr_resolve (n=<optimized out>, addrs=0x7f2783c8f410) at /home/buildbot/buildbot/build/mariadb-10.1.26/mysys/stacktrace.c:253
|
#8 my_print_stacktrace (stack_bottom=<optimized out>, thread_stack=295936) at /home/buildbot/buildbot/build/mariadb-10.1.26/mysys/stacktrace.c:271
|
#9 0x000055f45647aed5 in handle_fatal_signal (sig=11) at /home/buildbot/buildbot/build/mariadb-10.1.26/sql/signal_handler.cc:166
|
#10 <signal handler called>
|
#11 ha_rollback_trans (thd=thd@entry=0x7f282328f008, all=all@entry=true) at /home/buildbot/buildbot/build/mariadb-10.1.26/sql/handler.cc:1591
|
#12 0x000055f4563d6a9a in trans_rollback (thd=thd@entry=0x7f282328f008) at /home/buildbot/buildbot/build/mariadb-10.1.26/sql/transaction.cc:340
|
#13 0x000055f4562c51b2 in THD::cleanup (this=this@entry=0x7f282328f008) at /home/buildbot/buildbot/build/mariadb-10.1.26/sql/sql_class.cc:1582
|
#14 0x000055f4562578ea in thd_cleanup (thd=thd@entry=0x7f282328f008) at /home/buildbot/buildbot/build/mariadb-10.1.26/sql/mysqld.cc:2756
|
#15 0x000055f456257ad7 in unlink_thd (thd=0x7f282328f008) at /home/buildbot/buildbot/build/mariadb-10.1.26/sql/mysqld.cc:2824
|
#16 0x000055f456257c90 in one_thread_per_connection_end (thd=<optimized out>, put_in_cache=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.1.26/sql/mysqld.cc:2952
|
#17 0x000055f4563c8963 in do_handle_one_connection (thd_arg=thd_arg@entry=0x7f28084ed008) at /home/buildbot/buildbot/build/mariadb-10.1.26/sql/sql_connect.cc:1368
|
#18 0x000055f4563c8b80 in handle_one_connection (arg=arg@entry=0x7f28084ed008) at /home/buildbot/buildbot/build/mariadb-10.1.26/sql/sql_connect.cc:1261
|
#19 0x000055f456607e8d in pfs_spawn_thread (arg=0x7f283af7cc08) at /home/buildbot/buildbot/build/mariadb-10.1.26/storage/perfschema/pfs.cc:1860
|
#20 0x00007f5e818c1e25 in start_thread () from /lib64/libpthread.so.0
|
#21 0x00007f5e7fc6534d in clone () from /lib64/libc.so.6
|
+thread apply all bt
|
gdb full:
Thread 1 (Thread 0x7f2783c90b00 (LWP 22144)):
|
+bt full
|
#0 0x00007f5e7fba21f7 in raise () from /lib64/libc.so.6
|
No symbol table info available.
|
#1 0x00007f5e7fba38e8 in abort () from /lib64/libc.so.6
|
No symbol table info available.
|
#2 0x00007f5e7fbe1f47 in __libc_message () from /lib64/libc.so.6
|
No symbol table info available.
|
#3 0x00007f5e7fc7cd87 in __fortify_fail () from /lib64/libc.so.6
|
No symbol table info available.
|
#4 0x00007f5e7fc7af40 in __chk_fail () from /lib64/libc.so.6
|
No symbol table info available.
|
#5 0x00007f5e7fc7ccf7 in __fdelt_warn () from /lib64/libc.so.6
|
No symbol table info available.
|
#6 0x000055f45696b088 in my_addr_resolve (ptr=0x55f45695752e <my_print_stacktrace+46>, loc=loc@entry=0x7f2783c8f3f0) at /home/buildbot/buildbot/build/mariadb-10.1.26/mysys/my_addr_resolve.c:162
|
__d = <optimized out>
|
input = "\270\316\362\177^\177\000\000\000\000\000\000\000\000\000\000\270\316\362\177^\177\000\000\062\302\302\177^\177\000"
|
len = <optimized out>
|
total_bytes_read = 0
|
extra_bytes_read = 0
|
set = {
|
fds_bits = {[0] = 0 <repeats 16 times>}
|
}
|
timeout = {
|
tv_sec = 0,
|
tv_usec = 0
|
}
|
filename_start = -1
|
line_number_start = -1
|
i = <optimized out>
|
#7 0x000055f4569576c2 in print_with_addr_resolve (n=<optimized out>, addrs=0x7f2783c8f410) at /home/buildbot/buildbot/build/mariadb-10.1.26/mysys/stacktrace.c:253
|
loc = {
|
file = 0x7f2800000000 "\300\317\333|^\177",
|
func = 0x675f61d6be26ab00 <Address 0x675f61d6be26ab00 out of bounds>,
|
line = 1461355122
|
}
|
i = <optimized out>
|
err = <optimized out>
|
#8 my_print_stacktrace (stack_bottom=<optimized out>, thread_stack=295936) at /home/buildbot/buildbot/build/mariadb-10.1.26/mysys/stacktrace.c:271
|
addrs = {[0] = 0x55f45695752e <my_print_stacktrace+46>,
|
[1] = 0x55f45647aed5 <handle_fatal_signal(int)+773>,
|
[2] = 0x7f5e818c95e0 <__restore_rt>,
|
[3] = 0x55f45647da1b <ha_rollback_trans(THD*, bool)+171>,
|
[4] = 0x55f4563d6a9a <trans_rollback(THD*)+90>,
|
[5] = 0x55f4562c51b2 <THD::cleanup()+226>,
|
[6] = 0x55f456257ad7 <unlink_thd(THD*)+23>,
|
[7] = 0x55f456257c90 <one_thread_per_connection_end(THD*, bool)+32>,
|
[8] = 0x55f4563c8963 <do_handle_one_connection(THD*)+275>,
|
[9] = 0x55f4563c8b80 <handle_one_connection(void*)+64>,
|
[10] = 0x55f456607e8d <pfs_spawn_thread(void*)+365>,
|
[11] = 0x7f5e818c1e25 <start_thread+197>,
|
[12] = 0x7f5e7fc6534d <clone+109>,
|
[13] = 0x0,
|
[14] = 0x7f284705c0c2,
|
[15] = 0x7f284705c020,
|
[16] = 0x101000000a2,
|
[17] = 0x7f284705c128,
|
[18] = 0x7f284705c1ca,
|
[19] = 0x7f284705c1ca,
|
[20] = 0x7f284705c1ca,
|
[21] = 0x7f284705c1ca,
|
[22] = 0x0,
|
[23] = 0x112fd57b3f,
|
[24] = 0x0,
|
[25] = 0x55f40000000c,
|
[26] = 0x0,
|
[27] = 0x100007f,
|
[28] = 0x7f5e00000000,
|
[29] = 0x7f284705c1c5,
|
[30] = 0x7f284705c1ca,
|
[31] = 0x675f61d6be26ab00,
|
[32] = 0x0,
|
[33] = 0x7f2783c8f570,
|
[34] = 0x55f457136330 <dflt_key_cache>,
|
[35] = 0x55f4569570ec <my_write_stderr+28>,
|
[36] = 0x7f2783c8f570,
|
[37] = 0x7f5e818c86b9 <write+57>,
|
[38] = 0x0,
|
[39] = 0x55f4569571c3 <my_safe_printf_stderr+195>,
|
[40] = 0x0,
|
[41] = 0x3000000010,
|
[42] = 0x7f2783c8f850,
|
[43] = 0x7f2783c8f780,
|
[44] = 0x6974706d65747441,
|
[45] = 0x746b63616220676e,
|
[46] = 0x6f59202e65636172,
|
[47] = 0x7375206e61632075,
|
[48] = 0x6f66206568742065,
|
[49] = 0x20676e69776f6c6c,
|
[50] = 0x74616d726f666e69,
|
[51] = 0x66206f74206e6f69,
|
[52] = 0xa74756f20646e69,
|
[53] = 0x796d206572656877,
|
[54] = 0x65696420646c7173,
|
[55] = 0x6f79206649202e64,
|
[56] = 0x6f6e206565732075,
|
[57] = 0x6567617373656d20,
|
[58] = 0x2072657466612073,
|
[59] = 0x6f73202c73696874,
|
[60] = 0x20676e696874656d,
|
[61] = 0x7265740a746e6577,
|
[62] = 0x727720796c626972,
|
[63] = 0xa2e2e2e676e6f,
|
[64] = 0x616d207369687420,
|
[65] = 0xa2e6c6961662079,
|
[66] = 0x61206e616320000a,
|
[67] = 0x63206562206f736c,
|
[68] = 0x7962206465737561,
|
[69] = 0x636e75666c616d20,
|
[70] = 0x20676e696e6f6974,
|
[71] = 0x6572617764726168,
|
[72] = 0xa0a2e,
|
[73] = 0x0 <repeats 36 times>,
|
[109] = 0x675f61d6be26ab00,
|
[110] = 0xca,
|
[111] = 0x55f456a45a40,
|
[112] = 0x1f,
|
[113] = 0x7f5e818c86ad <write+45>,
|
[114] = 0x70,
|
[115] = 0x0,
|
[116] = 0x0,
|
[117] = 0x7f2783c8f810,
|
[118] = 0xc9,
|
[119] = 0x59dcaef1,
|
[120] = 0x18,
|
[121] = 0x7f2783c8f830,
|
[122] = 0x7f2783c8f7f0,
|
[123] = 0x7f2783c8f878,
|
[124] = 0x1,
|
[125] = 0x0,
|
[126] = 0x2f,
|
[127] = 0x55f456a4e8d4}
|
strings = 0x0
|
n = <optimized out>
|
#9 0x000055f45647aed5 in handle_fatal_signal (sig=11) at /home/buildbot/buildbot/build/mariadb-10.1.26/sql/signal_handler.cc:166
|
curr_time = 1507634929
|
tm = {
|
tm_sec = 49,
|
tm_min = 28,
|
tm_hour = 12,
|
tm_mday = 10,
|
tm_mon = 9,
|
tm_year = 117,
|
tm_wday = 2,
|
tm_yday = 282,
|
tm_isdst = 1,
|
tm_gmtoff = 3600,
|
tm_zone = 0x7f5e7c83b070 "BST"
|
}
|
print_invalid_query_pointer = false
|
#10 <signal handler called>
|
No symbol table info available.
|
#11 ha_rollback_trans (thd=thd@entry=0x7f282328f008, all=all@entry=true) at /home/buildbot/buildbot/build/mariadb-10.1.26/sql/handler.cc:1591
|
No locals.
|
#12 0x000055f4563d6a9a in trans_rollback (thd=thd@entry=0x7f282328f008) at /home/buildbot/buildbot/build/mariadb-10.1.26/sql/transaction.cc:340
|
res = <optimized out>
|
#13 0x000055f4562c51b2 in THD::cleanup (this=this@entry=0x7f282328f008) at /home/buildbot/buildbot/build/mariadb-10.1.26/sql/sql_class.cc:1582
|
No locals.
|
#14 0x000055f4562578ea in thd_cleanup (thd=thd@entry=0x7f282328f008) at /home/buildbot/buildbot/build/mariadb-10.1.26/sql/mysqld.cc:2756
|
No locals.
|
#15 0x000055f456257ad7 in unlink_thd (thd=0x7f282328f008) at /home/buildbot/buildbot/build/mariadb-10.1.26/sql/mysqld.cc:2824
|
No locals.
|
#16 0x000055f456257c90 in one_thread_per_connection_end (thd=<optimized out>, put_in_cache=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.1.26/sql/mysqld.cc:2952
|
wsrep_applier = false
|
#17 0x000055f4563c8963 in do_handle_one_connection (thd_arg=thd_arg@entry=0x7f28084ed008) at /home/buildbot/buildbot/build/mariadb-10.1.26/sql/sql_connect.cc:1368
|
create_user = <optimized out>
|
thd = 0x7f282328f008
|
#18 0x000055f4563c8b80 in handle_one_connection (arg=arg@entry=0x7f28084ed008) at /home/buildbot/buildbot/build/mariadb-10.1.26/sql/sql_connect.cc:1261
|
thd = 0x7f28084ed008
|
#19 0x000055f456607e8d in pfs_spawn_thread (arg=0x7f283af7cc08) at /home/buildbot/buildbot/build/mariadb-10.1.26/storage/perfschema/pfs.cc:1860
|
typed_arg = 0x7f283af7cc08
|
user_arg = 0x7f28084ed008
|
pfs = <optimized out>
|
user_start_routine = 0x55f4563c8b40 <handle_one_connection(void*)>
|
klass = <optimized out>
|
#20 0x00007f5e818c1e25 in start_thread () from /lib64/libpthread.so.0
|
No symbol table info available.
|
#21 0x00007f5e7fc6534d in clone () from /lib64/libc.so.6
|
No symbol table info available.
|
+quit
|
Attachments
Issue Links
- relates to
-
MDEV-13800 Crash during transaction rollback with thread_handling=pool-of-threads
-
- Closed
-
The configuration file indicates, although not conclusively, that it might be a replication slave. Is it so? If it's a slave, do crashes always happen on the slave only, and can you find out from the binary log what was the last binlog event that it was executing (after server restart it will show in the error log from which position it resumes replication)?
For a note, the attached occurrences actually contain four different types of crashes: