Details
-
Bug
-
Status: Open (View Workflow)
-
Major
-
Resolution: Unresolved
-
10.1.21, 10.1.26
-
None
-
None
-
None
Description
We have MariaDB instance with very often 'signal 11' crashes. Most of them happen with reference to unlink_thd() function, which is not present anymore in MySQL 5.6+, so I assume this may be specific to MariaDB.
171010 12:33:21 server_audit: MariaDB Audit Plugin version 1.4.1 STARTED.
|
Version: '10.1.26-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server
|
It is not clear what exactly queries/transactions cause these crashes.
Example crash with related thread gdb traces (no full gdb attached due to sensitive information) below. Additional, slightly different occurences in attachment.
171010 12:28:49 [ERROR] mysqld got signal 11 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
|
Server version: 10.1.26-MariaDB
|
key_buffer_size=1073741824
|
read_buffer_size=131072
|
max_used_connections=1058
|
max_threads=2002
|
thread_count=457
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 5446182 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
|
Thread pointer: 0x7f282328f008
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x7f2783c90080 thread_stack 0x48400
|
*** ***: /usr/sbin/mysqld terminated
|
======= Backtrace: =========
|
/lib64/libc.so.6(__fortify_fail+0x37)[0x7f5e7fc7cd87]
|
/lib64/libc.so.6(+0x10df40)[0x7f5e7fc7af40]
|
/lib64/libc.so.6(+0x10fcf7)[0x7f5e7fc7ccf7]
|
/usr/sbin/mysqld(my_addr_resolve+0x48)[0x55f45696b088]
|
/usr/sbin/mysqld(my_print_stacktrace+0x1c2)[0x55f4569576c2]
|
/usr/sbin/mysqld(handle_fatal_signal+0x305)[0x55f45647aed5]
|
/lib64/libpthread.so.0(+0xf5e0)[0x7f5e818c95e0]
|
/usr/sbin/mysqld(_Z17ha_rollback_transP3THDb+0xab)[0x55f45647da1b]
|
/usr/sbin/mysqld(_Z14trans_rollbackP3THD+0x5a)[0x55f4563d6a9a]
|
/usr/sbin/mysqld(_ZN3THD7cleanupEv+0xe2)[0x55f4562c51b2]
|
/usr/sbin/mysqld(_Z10unlink_thdP3THD+0x17)[0x55f456257ad7]
|
/usr/sbin/mysqld(_Z29one_thread_per_connection_endP3THDb+0x20)[0x55f456257c90]
|
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x113)[0x55f4563c8963]
|
/usr/sbin/mysqld(handle_one_connection+0x40)[0x55f4563c8b80]
|
/usr/sbin/mysqld(+0x74be8d)[0x55f456607e8d]
|
/lib64/libpthread.so.0(+0x7e25)[0x7f5e818c1e25]
|
/lib64/libc.so.6(clone+0x6d)[0x7f5e7fc6534d]
|
======= Memory map: ========
|
55f455ebc000-55f456e46000 r-xp 00000000 fb:00 134770637 /usr/sbin/mysqld
|
55f457046000-55f4570f7000 r--p 00f8a000 fb:00 134770637 /usr/sbin/mysqld
|
55f4570f7000-55f4571a9000 rw-p 0103b000 fb:00 134770637 /usr/sbin/mysqld
|
(...)
|
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
|
Fatal signal 6 while backtracing
|
2017-10-10 12:32:44 139933296474368 [Note] Using unique option prefix 'myisam-recover' is error-prone and can break in the future. Please use the full name 'myisam-recover-options' instead.
|
2017-10-10 12:32:44 139933296474368 [Note] InnoDB: Using mutexes to ref count buffer pool pages
|
2017-10-10 12:32:44 139933296474368 [Note] InnoDB: The InnoDB memory heap is disabled
|
2017-10-10 12:32:44 139933296474368 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
|
2017-10-10 12:32:44 139933296474368 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
|
2017-10-10 12:32:44 139933296474368 [Note] InnoDB: Compressed tables use zlib 1.2.7
|
2017-10-10 12:32:44 139933296474368 [Note] InnoDB: Using Linux native AIO
|
2017-10-10 12:32:44 139933296474368 [Note] InnoDB: Using SSE crc32 instructions
|
2017-10-10 12:32:44 139933296474368 [Note] InnoDB: Initializing buffer pool, size = 200.0G
|
2017-10-10 12:32:51 139933296474368 [Note] InnoDB: Completed initialization of buffer pool
|
2017-10-10 12:32:51 139933296474368 [Note] InnoDB: Highest supported file format is Barracuda.
|
2017-10-10 12:32:51 139933296474368 [Note] InnoDB: Starting crash recovery from checkpoint LSN=49954283878443
|
2017-10-10 12:32:52 139933296474368 [Note] InnoDB: Restoring possible half-written data pages from the doublewrite buffer...
|
InnoDB: 4 transaction(s) which must be rolled back or cleaned up
|
InnoDB: in total 4 row operations to undo
|
InnoDB: Trx id counter is 36935782400
|
2017-10-10 12:33:04 139933296474368 [Note] InnoDB: Starting final batch to recover 127478 pages from redo log
|
2017-10-10 12:33:06 139701654628096 [Note] InnoDB: To recover: 92197 pages from log
|
InnoDB: Last MySQL binlog file position 0 575403140, file name /home/mysql/logs/mysql.002397
|
2017-10-10 12:33:11 139933296474368 [Note] InnoDB: 128 rollback segment(s) are active.
|
2017-10-10 12:33:11 139698190149376 [Note] InnoDB: Starting in background the rollback of recovered transactions
|
2017-10-10 12:33:11 7f0e04ffe700 InnoDB: Rolling back trx with id 36935781973, 1 rows to undo
|
2017-10-10 12:33:11 139933296474368 [Note] InnoDB: Waiting for purge to start
|
2017-10-10 12:33:11 139698190149376 [Note] InnoDB: Rollback of trx with id 36935781973 completed
|
2017-10-10 12:33:11 7f0e04ffe700 InnoDB: Rolling back trx with id 36935781972, 1 rows to undo
|
2017-10-10 12:33:11 139698190149376 [Note] InnoDB: Rollback of trx with id 36935781972 completed
|
2017-10-10 12:33:11 7f0e04ffe700 InnoDB: Rolling back trx with id 36935781970, 1 rows to undo
|
2017-10-10 12:33:11 139698190149376 [Note] InnoDB: Rollback of trx with id 36935781970 completed
|
2017-10-10 12:33:11 7f0e04ffe700 InnoDB: Rolling back trx with id 36935781952, 1 rows to undo
|
2017-10-10 12:33:11 139698190149376 [Note] InnoDB: Rollback of trx with id 36935781952 completed
|
2017-10-10 12:33:11 139698190149376 [Note] InnoDB: Rollback of non-prepared transactions completed
|
2017-10-10 12:33:11 139933296474368 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.36-82.1 started; log sequence number 49954847609095
|
2017-10-10 12:33:21 139933296474368 [Note] Plugin 'FEEDBACK' is disabled.
|
171010 12:33:21 server_audit: MariaDB Audit Plugin version 1.4.1 STARTED.
|
2017-10-10 12:33:21 139697242212096 [Note] InnoDB: Dumping buffer pool(s) not yet started
|
2017-10-10 12:33:21 139933296474368 [Note] Recovering after a crash using /home/mysql/logs/mysql
|
2017-10-10 12:33:22 139933296474368 [Note] Starting crash recovery...
|
2017-10-10 12:33:22 139933296474368 [Note] Crash recovery finished.
|
2017-10-10 12:33:22 139933296474368 [Note] Server socket created on IP: '::'.
|
2017-10-10 12:33:22 139933296474368 [Note] /usr/sbin/mysqld: ready for connections.
|
Version: '10.1.26-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server
|
Related thread gdb standard:
Thread 1 (Thread 0x7f2783c90b00 (LWP 22144)):
|
+bt
|
#0 0x00007f5e7fba21f7 in raise () from /lib64/libc.so.6
|
#1 0x00007f5e7fba38e8 in abort () from /lib64/libc.so.6
|
#2 0x00007f5e7fbe1f47 in __libc_message () from /lib64/libc.so.6
|
#3 0x00007f5e7fc7cd87 in __fortify_fail () from /lib64/libc.so.6
|
#4 0x00007f5e7fc7af40 in __chk_fail () from /lib64/libc.so.6
|
#5 0x00007f5e7fc7ccf7 in __fdelt_warn () from /lib64/libc.so.6
|
#6 0x000055f45696b088 in my_addr_resolve (ptr=0x55f45695752e <my_print_stacktrace+46>, loc=loc@entry=0x7f2783c8f3f0) at /home/buildbot/buildbot/build/mariadb-10.1.26/mysys/my_addr_resolve.c:162
|
#7 0x000055f4569576c2 in print_with_addr_resolve (n=<optimized out>, addrs=0x7f2783c8f410) at /home/buildbot/buildbot/build/mariadb-10.1.26/mysys/stacktrace.c:253
|
#8 my_print_stacktrace (stack_bottom=<optimized out>, thread_stack=295936) at /home/buildbot/buildbot/build/mariadb-10.1.26/mysys/stacktrace.c:271
|
#9 0x000055f45647aed5 in handle_fatal_signal (sig=11) at /home/buildbot/buildbot/build/mariadb-10.1.26/sql/signal_handler.cc:166
|
#10 <signal handler called>
|
#11 ha_rollback_trans (thd=thd@entry=0x7f282328f008, all=all@entry=true) at /home/buildbot/buildbot/build/mariadb-10.1.26/sql/handler.cc:1591
|
#12 0x000055f4563d6a9a in trans_rollback (thd=thd@entry=0x7f282328f008) at /home/buildbot/buildbot/build/mariadb-10.1.26/sql/transaction.cc:340
|
#13 0x000055f4562c51b2 in THD::cleanup (this=this@entry=0x7f282328f008) at /home/buildbot/buildbot/build/mariadb-10.1.26/sql/sql_class.cc:1582
|
#14 0x000055f4562578ea in thd_cleanup (thd=thd@entry=0x7f282328f008) at /home/buildbot/buildbot/build/mariadb-10.1.26/sql/mysqld.cc:2756
|
#15 0x000055f456257ad7 in unlink_thd (thd=0x7f282328f008) at /home/buildbot/buildbot/build/mariadb-10.1.26/sql/mysqld.cc:2824
|
#16 0x000055f456257c90 in one_thread_per_connection_end (thd=<optimized out>, put_in_cache=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.1.26/sql/mysqld.cc:2952
|
#17 0x000055f4563c8963 in do_handle_one_connection (thd_arg=thd_arg@entry=0x7f28084ed008) at /home/buildbot/buildbot/build/mariadb-10.1.26/sql/sql_connect.cc:1368
|
#18 0x000055f4563c8b80 in handle_one_connection (arg=arg@entry=0x7f28084ed008) at /home/buildbot/buildbot/build/mariadb-10.1.26/sql/sql_connect.cc:1261
|
#19 0x000055f456607e8d in pfs_spawn_thread (arg=0x7f283af7cc08) at /home/buildbot/buildbot/build/mariadb-10.1.26/storage/perfschema/pfs.cc:1860
|
#20 0x00007f5e818c1e25 in start_thread () from /lib64/libpthread.so.0
|
#21 0x00007f5e7fc6534d in clone () from /lib64/libc.so.6
|
+thread apply all bt
|
gdb full:
Thread 1 (Thread 0x7f2783c90b00 (LWP 22144)):
|
+bt full
|
#0 0x00007f5e7fba21f7 in raise () from /lib64/libc.so.6
|
No symbol table info available.
|
#1 0x00007f5e7fba38e8 in abort () from /lib64/libc.so.6
|
No symbol table info available.
|
#2 0x00007f5e7fbe1f47 in __libc_message () from /lib64/libc.so.6
|
No symbol table info available.
|
#3 0x00007f5e7fc7cd87 in __fortify_fail () from /lib64/libc.so.6
|
No symbol table info available.
|
#4 0x00007f5e7fc7af40 in __chk_fail () from /lib64/libc.so.6
|
No symbol table info available.
|
#5 0x00007f5e7fc7ccf7 in __fdelt_warn () from /lib64/libc.so.6
|
No symbol table info available.
|
#6 0x000055f45696b088 in my_addr_resolve (ptr=0x55f45695752e <my_print_stacktrace+46>, loc=loc@entry=0x7f2783c8f3f0) at /home/buildbot/buildbot/build/mariadb-10.1.26/mysys/my_addr_resolve.c:162
|
__d = <optimized out>
|
input = "\270\316\362\177^\177\000\000\000\000\000\000\000\000\000\000\270\316\362\177^\177\000\000\062\302\302\177^\177\000"
|
len = <optimized out>
|
total_bytes_read = 0
|
extra_bytes_read = 0
|
set = {
|
fds_bits = {[0] = 0 <repeats 16 times>}
|
}
|
timeout = {
|
tv_sec = 0,
|
tv_usec = 0
|
}
|
filename_start = -1
|
line_number_start = -1
|
i = <optimized out>
|
#7 0x000055f4569576c2 in print_with_addr_resolve (n=<optimized out>, addrs=0x7f2783c8f410) at /home/buildbot/buildbot/build/mariadb-10.1.26/mysys/stacktrace.c:253
|
loc = {
|
file = 0x7f2800000000 "\300\317\333|^\177",
|
func = 0x675f61d6be26ab00 <Address 0x675f61d6be26ab00 out of bounds>,
|
line = 1461355122
|
}
|
i = <optimized out>
|
err = <optimized out>
|
#8 my_print_stacktrace (stack_bottom=<optimized out>, thread_stack=295936) at /home/buildbot/buildbot/build/mariadb-10.1.26/mysys/stacktrace.c:271
|
addrs = {[0] = 0x55f45695752e <my_print_stacktrace+46>,
|
[1] = 0x55f45647aed5 <handle_fatal_signal(int)+773>,
|
[2] = 0x7f5e818c95e0 <__restore_rt>,
|
[3] = 0x55f45647da1b <ha_rollback_trans(THD*, bool)+171>,
|
[4] = 0x55f4563d6a9a <trans_rollback(THD*)+90>,
|
[5] = 0x55f4562c51b2 <THD::cleanup()+226>,
|
[6] = 0x55f456257ad7 <unlink_thd(THD*)+23>,
|
[7] = 0x55f456257c90 <one_thread_per_connection_end(THD*, bool)+32>,
|
[8] = 0x55f4563c8963 <do_handle_one_connection(THD*)+275>,
|
[9] = 0x55f4563c8b80 <handle_one_connection(void*)+64>,
|
[10] = 0x55f456607e8d <pfs_spawn_thread(void*)+365>,
|
[11] = 0x7f5e818c1e25 <start_thread+197>,
|
[12] = 0x7f5e7fc6534d <clone+109>,
|
[13] = 0x0,
|
[14] = 0x7f284705c0c2,
|
[15] = 0x7f284705c020,
|
[16] = 0x101000000a2,
|
[17] = 0x7f284705c128,
|
[18] = 0x7f284705c1ca,
|
[19] = 0x7f284705c1ca,
|
[20] = 0x7f284705c1ca,
|
[21] = 0x7f284705c1ca,
|
[22] = 0x0,
|
[23] = 0x112fd57b3f,
|
[24] = 0x0,
|
[25] = 0x55f40000000c,
|
[26] = 0x0,
|
[27] = 0x100007f,
|
[28] = 0x7f5e00000000,
|
[29] = 0x7f284705c1c5,
|
[30] = 0x7f284705c1ca,
|
[31] = 0x675f61d6be26ab00,
|
[32] = 0x0,
|
[33] = 0x7f2783c8f570,
|
[34] = 0x55f457136330 <dflt_key_cache>,
|
[35] = 0x55f4569570ec <my_write_stderr+28>,
|
[36] = 0x7f2783c8f570,
|
[37] = 0x7f5e818c86b9 <write+57>,
|
[38] = 0x0,
|
[39] = 0x55f4569571c3 <my_safe_printf_stderr+195>,
|
[40] = 0x0,
|
[41] = 0x3000000010,
|
[42] = 0x7f2783c8f850,
|
[43] = 0x7f2783c8f780,
|
[44] = 0x6974706d65747441,
|
[45] = 0x746b63616220676e,
|
[46] = 0x6f59202e65636172,
|
[47] = 0x7375206e61632075,
|
[48] = 0x6f66206568742065,
|
[49] = 0x20676e69776f6c6c,
|
[50] = 0x74616d726f666e69,
|
[51] = 0x66206f74206e6f69,
|
[52] = 0xa74756f20646e69,
|
[53] = 0x796d206572656877,
|
[54] = 0x65696420646c7173,
|
[55] = 0x6f79206649202e64,
|
[56] = 0x6f6e206565732075,
|
[57] = 0x6567617373656d20,
|
[58] = 0x2072657466612073,
|
[59] = 0x6f73202c73696874,
|
[60] = 0x20676e696874656d,
|
[61] = 0x7265740a746e6577,
|
[62] = 0x727720796c626972,
|
[63] = 0xa2e2e2e676e6f,
|
[64] = 0x616d207369687420,
|
[65] = 0xa2e6c6961662079,
|
[66] = 0x61206e616320000a,
|
[67] = 0x63206562206f736c,
|
[68] = 0x7962206465737561,
|
[69] = 0x636e75666c616d20,
|
[70] = 0x20676e696e6f6974,
|
[71] = 0x6572617764726168,
|
[72] = 0xa0a2e,
|
[73] = 0x0 <repeats 36 times>,
|
[109] = 0x675f61d6be26ab00,
|
[110] = 0xca,
|
[111] = 0x55f456a45a40,
|
[112] = 0x1f,
|
[113] = 0x7f5e818c86ad <write+45>,
|
[114] = 0x70,
|
[115] = 0x0,
|
[116] = 0x0,
|
[117] = 0x7f2783c8f810,
|
[118] = 0xc9,
|
[119] = 0x59dcaef1,
|
[120] = 0x18,
|
[121] = 0x7f2783c8f830,
|
[122] = 0x7f2783c8f7f0,
|
[123] = 0x7f2783c8f878,
|
[124] = 0x1,
|
[125] = 0x0,
|
[126] = 0x2f,
|
[127] = 0x55f456a4e8d4}
|
strings = 0x0
|
n = <optimized out>
|
#9 0x000055f45647aed5 in handle_fatal_signal (sig=11) at /home/buildbot/buildbot/build/mariadb-10.1.26/sql/signal_handler.cc:166
|
curr_time = 1507634929
|
tm = {
|
tm_sec = 49,
|
tm_min = 28,
|
tm_hour = 12,
|
tm_mday = 10,
|
tm_mon = 9,
|
tm_year = 117,
|
tm_wday = 2,
|
tm_yday = 282,
|
tm_isdst = 1,
|
tm_gmtoff = 3600,
|
tm_zone = 0x7f5e7c83b070 "BST"
|
}
|
print_invalid_query_pointer = false
|
#10 <signal handler called>
|
No symbol table info available.
|
#11 ha_rollback_trans (thd=thd@entry=0x7f282328f008, all=all@entry=true) at /home/buildbot/buildbot/build/mariadb-10.1.26/sql/handler.cc:1591
|
No locals.
|
#12 0x000055f4563d6a9a in trans_rollback (thd=thd@entry=0x7f282328f008) at /home/buildbot/buildbot/build/mariadb-10.1.26/sql/transaction.cc:340
|
res = <optimized out>
|
#13 0x000055f4562c51b2 in THD::cleanup (this=this@entry=0x7f282328f008) at /home/buildbot/buildbot/build/mariadb-10.1.26/sql/sql_class.cc:1582
|
No locals.
|
#14 0x000055f4562578ea in thd_cleanup (thd=thd@entry=0x7f282328f008) at /home/buildbot/buildbot/build/mariadb-10.1.26/sql/mysqld.cc:2756
|
No locals.
|
#15 0x000055f456257ad7 in unlink_thd (thd=0x7f282328f008) at /home/buildbot/buildbot/build/mariadb-10.1.26/sql/mysqld.cc:2824
|
No locals.
|
#16 0x000055f456257c90 in one_thread_per_connection_end (thd=<optimized out>, put_in_cache=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.1.26/sql/mysqld.cc:2952
|
wsrep_applier = false
|
#17 0x000055f4563c8963 in do_handle_one_connection (thd_arg=thd_arg@entry=0x7f28084ed008) at /home/buildbot/buildbot/build/mariadb-10.1.26/sql/sql_connect.cc:1368
|
create_user = <optimized out>
|
thd = 0x7f282328f008
|
#18 0x000055f4563c8b80 in handle_one_connection (arg=arg@entry=0x7f28084ed008) at /home/buildbot/buildbot/build/mariadb-10.1.26/sql/sql_connect.cc:1261
|
thd = 0x7f28084ed008
|
#19 0x000055f456607e8d in pfs_spawn_thread (arg=0x7f283af7cc08) at /home/buildbot/buildbot/build/mariadb-10.1.26/storage/perfschema/pfs.cc:1860
|
typed_arg = 0x7f283af7cc08
|
user_arg = 0x7f28084ed008
|
pfs = <optimized out>
|
user_start_routine = 0x55f4563c8b40 <handle_one_connection(void*)>
|
klass = <optimized out>
|
#20 0x00007f5e818c1e25 in start_thread () from /lib64/libpthread.so.0
|
No symbol table info available.
|
#21 0x00007f5e7fc6534d in clone () from /lib64/libc.so.6
|
No symbol table info available.
|
+quit
|
Attachments
Issue Links
- relates to
-
-
- Closed
-