Elena Stepanova added a comment - 2019-09-10 14:39 2019-09-10 8:31:33 0 [ERROR] [FATAL] InnoDB: Semaphore wait has lasted > 60 seconds. We intentionally crash the server because it appears to be hung. 190910 8:31:33 [ERROR] mysqld got signal 6 ;

Marko Mäkelä added a comment - 2019-09-10 15:16 f_razzoli , to better analyze this bug, we would need stack traces of all InnoDB threads during the hang: gdb -ex "set pagination 0" -ex "thread apply all bt" --batch -p $(pgrep mysqld) Alternatively, you should enable core dumps and provide the output of the following command: gdb -ex "set pagination 0" -ex "thread apply all bt" --batch /usr/sbin/mysqld core Starting with MariaDB Server 10.3, core dumps should be much smaller, because they will by default not include the InnoDB buffer pool (which we might actually need later to analyze the hang in more detail, in case some buf_block_t::lock are involved). We have similar reports in MDEV-13983 and MDEV-15135 .

Federico Razzoli added a comment - 2019-09-12 16:39 I've run the first command short after a crash: $ gdb -ex "set pagination 0" -ex "thread apply all bt" --batch -p $(pgrep mysqld) Thread 1 (process 3286): #0 0x0000000000456a43 in runtime.futex () #1 0x00000000004274ab in runtime.futexsleep () #2 0x0000000000b7b7c8 in runtime.m0 () #3 0x0000000000000000 in ?? () Unfortunately at the moment I don't have a core dump to submit. Does this make it impossible to investigate the bug?

Peanuts added a comment - 2019-09-14 12:45 Dump attached. Please let us know if this is what you are looking for.

Federico Razzoli added a comment - 2019-09-16 13:06 Please see the core dump uploaded by Peanuts. It comes from the same server.

Marko Mäkelä added a comment - 2019-09-30 15:45 Peanuts , thank you, the stack traces in dump.txt seem to suggest that some thread is holding dict_sys->mutex , which several threads are waiting for. If you still have that core dump (or can produce another), please show print dict_sys->mutex print dict_operation_lock and then try to find the thread that is holding those (or has submitted an exclusive latch request on dict_operation_lock ) by using the gdb command thread find 0x… It expects the thread identifier in hexadecimal, so you might want to use the print/x command. That is only the first step of analyzing the hang. Often hangs involve multiple mutexes or rw-latches. If you provide a core dump, then I’d also need download links to the executables and shared libraries ( ldd /usr/sbin/mysqld ). Otherwise the stack traces will typically be garbage.

Federico Razzoli added a comment - 2019-10-07 13:22 - edited Unfortunately mysqld is stripped. I have a dump to upload, from the most recent crash. But it's 2.2G, it compresses to 104M. And it seems I can't upload files > 10M here. So how can I share it? In the meanwhile, these are the libraries: # ldd /usr/sbin/mysqld linux-vdso.so.1 => (0x00007fff807bf000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f5d6d03d000) libaio.so.1 => /lib64/libaio.so.1 (0x00007f5d6ce3b000) libz.so.1 => /lib64/libz.so.1 (0x00007f5d6cc25000) libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f5d6c9ee000) libsystemd.so.0 => /lib64/libsystemd.so.0 (0x00007f5d6c7bd000) libssl.so.10 => /lib64/libssl.so.10 (0x00007f5d6c54b000) libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007f5d6c0e9000) libdl.so.2 => /lib64/libdl.so.2 (0x00007f5d6bee5000) libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f5d6bbde000) libm.so.6 => /lib64/libm.so.6 (0x00007f5d6b8dc000) libc.so.6 => /lib64/libc.so.6 (0x00007f5d6b50f000) /lib64/ld-linux-x86-64.so.2 (0x00007f5d6f164000) libfreebl3.so => /lib64/libfreebl3.so (0x00007f5d6b30c000) libcap.so.2 => /lib64/libcap.so.2 (0x00007f5d6b107000) librt.so.1 => /lib64/librt.so.1 (0x00007f5d6aeff000) libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f5d6acd8000) liblzma.so.5 => /lib64/liblzma.so.5 (0x00007f5d6aab2000) liblz4.so.1 => /lib64/liblz4.so.1 (0x00007f5d6a89d000) libgcrypt.so.11 => /lib64/libgcrypt.so.11 (0x00007f5d6a61c000) libgpg-error.so.0 => /lib64/libgpg-error.so.0 (0x00007f5d6a417000) libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f5d6a1fe000) libdw.so.1 => /lib64/libdw.so.1 (0x00007f5d69faf000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f5d69d99000) libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f5d69b4c000) libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f5d69863000) libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f5d6965f000) libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f5d6942c000) libattr.so.1 => /lib64/libattr.so.1 (0x00007f5d69227000) libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f5d68fc5000) libelf.so.1 => /lib64/libelf.so.1 (0x00007f5d68dad000) libbz2.so.1 => /lib64/libbz2.so.1 (0x00007f5d68b9d000) libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f5d6898d000) libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f5d68789000)

Marko Mäkelä added a comment - 2019-10-07 14:42 f_razzoli , is there a separate package available for your system that includes the debugging symbols? Where did you get the mysqld executable from? The core dump will be rather useless without having access to the debugging symbols and to the executable and the dynamic libraries. You can upload a .tar.bz2 file of the core dump (as well as all the listed .so files) to ftp://ftp.mariadb.com/uploads using anonymous FTP. Let us know the file name. It will only be accessible to employees of MariaDB Corporation.