Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Fixed
-
10.1.41
-
None
-
Ubuntu 18.4, kernel 4.15.0-45-generic.
MariaDB 10.1.41
Description
The 10.1 branch has been stable for us for a while now.
10.2 crashes for us on a regular basis, so upgrading is not an option. (See MDEV-16923, and others)
Unfortunately, 10.1.40 crashed for us with an update on JOINs.
So we upgraded to 10.1.41, but since we did that the server crashes multiple times a day with the following log message:
Server version: 10.1.41-MariaDB-1~bionic
|
key_buffer_size=134217728
|
read_buffer_size=131072
|
max_used_connections=192
|
max_threads=502
|
thread_count=45
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1233831 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
|
Thread pointer: 0x7fa04ec10008
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x7f9f52fb3d68 thread_stack 0x49000
|
*** buffer overflow detected ***: /usr/sbin/mysqld terminated
|
Fatal signal 6 while backtracing
|
Attachments
Activity
Stack trace from george.ccto's occurrence of the failure is attached as gdb.txt
F.Y.I. -
A few days ago, we have updated MariaDB from 10.1.41 to 10.1.44 , it seems the problem disappeared (for a few days already).
In gdb.txt (as well as the Description), I see that the server crashed while trying to produce a stack trace:
|
mariadb-10.1.41 |
#6 0x000055f142b40c40 in my_addr_resolve (ptr=<optimized out>, loc=loc@entry=0x7fdb42ffd0b0) at /usr/src/debug/mariadb-10.1.41-linux-x86_64/server/src_0/mysys/my_addr_resolve.c:222
|
#7 0x000055f142b2ce82 in print_with_addr_resolve (n=<optimized out>, addrs=0x7fdb42ffd0d0) at /usr/src/debug/mariadb-10.1.41-linux-x86_64/server/src_0/mysys/stacktrace.c:253
|
#8 my_print_stacktrace (stack_bottom=<optimized out>, thread_stack=299008) at /usr/src/debug/mariadb-10.1.41-linux-x86_64/server/src_0/mysys/stacktrace.c:271
|
#9 0x000055f14264f5d5 in handle_fatal_signal (sig=6) at /usr/src/debug/mariadb-10.1.41-linux-x86_64/server/src_0/sql/signal_handler.cc:166
|
#10 <signal handler called>
|
#11 0x00007fdb7082d337 in raise () from /lib64/libc.so.6
|
#12 0x00007fdb7082ea28 in abort () from /lib64/libc.so.6
|
#13 0x000055f1429ca359 in srv_error_monitor_thread () at /usr/src/debug/mariadb-10.1.41-linux-x86_64/server/src_0/storage/xtradb/srv/srv0srv.cc:2410
|
That is, XtraDB intentionally initiated a kill of itself after a hang. In addition, the server crashed while trying to produce a stack trace. cvicentiu should be aware of this issue and try to remove the use of functions that are not safe to use in signal handlers. There is similar crash-while-crashing output in MDEV-19310.
Now, what could cause the hang? I was going to suggest MDEV-13983, but it turns out that that regression was fixed in 10.1.41.
Most threads are waiting for dict_sys->mutex while trying to open an InnoDB table.
Thread 5 (purge) is attempting to acquire dict_operation_lock in shared mode. That should prevent others from acquiring it in exclusive mode.
A possible cause of the deadlock could be between Thread 4 and Thread 2. This strongly looks like a duplicate of MDEV-20621, which was fixed in 10.1.42. Also george.ccto’s comment supports this.
CrewOne, do you use fulltext indexes on InnoDB or XtraDB tables? If not, your crash should be due to something else. Also, you mention "update on JOINs". Were there any messages in the server error log before the crash?
paladox, jcrespo, the InnoDB diagnostic output is next to useless for diagnosing hangs. We need stack traces of all threads during the hang, and to ease debugging, the information on who is holding the mutexes that almost everyone is waiting for:
gdb -ex 'set height 0' -ex 'thread apply all backtrace' -ex 'print *dict_operation_lock' -ex 'print *dict_sys.mutex' -ex detach -ex quit -p $(pgrep -x myqsld)
|
Hello MariaDB support,
We encountered the same - InnoDB: Assertion failure in thread 140581082752768 in file srv0srv.cc line 2410.
Do you have the procedure for coredump ?