[MDEV-15817] encryption tests fail futex(COND_thread_count/LOCK_thread_count) - Jira

XML

Word

Printable

Details

Type: Bug
Status: Open (View Workflow)
Priority: Major
Resolution: Unresolved
Affects Version/s: 10.3(EOL)
Fix Version/s: 10.4(EOL)
Component/s: Encryption, Server, Tests
Labels:
- crash
Environment:
x86_64 - ubuntu artful, clang-7, clang-6.0, gcc-7.2.0

Description

clang-7
dpkg -l \| grep clang-7
1:7~svn329460-1~exp1+0~20180406214534.323~1.gbp77c2bd

cmake ../mariadb-server/ -DCMAKE_CXX_COMPILER=clang++-7 -DCMAKE_C_COMPILER=clang-7 -DCMAKE_CFLAGS="-O3 -mcpu=skylake" -DCMAKE_CXXFLAGS="-O3 -mcpu=skylake"

CURRENT_TEST: binlog_encryption.encrypted_master_switch_to_unencrypted
--- /home/dan/mariadb-server-10.3/mysql-test/suite/binlog_encryption/encrypted_master_switch_to_unencrypted.result 2018-04-06 01:48:31.955687603 -0500
+++ /home/dan/mariadb-server-10.3/mysql-test/suite/binlog_encryption/encrypted_master_switch_to_unencrypted.reject 2018-04-08 18:23:26.526381135 -0500
@@ -72,6 +72,10 @@
SELECT COUNT(*) FROM table1_no_encryption;
COUNT(*)
8
+Warnings:
+Error 145 Table './test/table1_no_encryption' is marked as crashed and should be repaired
+Error 1194 Table 'table1_no_encryption' is marked as crashed and should be repaired
+Error 1034 1 client is using or hasn't closed the table properly
SELECT COUNT(*) FROM table2_to_encrypt;
COUNT(*)
8

mysqltest: Result length mismatch

- saving '/home/dan/build-mariadb-10.3_lastest_upstream/mysql-test/var/3/log/binlog_encryption.encrypted_master_switch_to_unencrypted-mix/' to '/home/dan/build-mariadb-10.3_lastest_upstream/mysql-test/var/log/binlog_encryption.encrypted_master_switch_to_unencrypted-mix/'
- found 'core' (0/5)

Trying 'dbx' to get a backtrace

Trying 'gdb' to get a backtrace from coredump /home/dan/build-mariadb-10.3_lastest_upstream/mysql-test/var/log/binlog_encryption.encrypted_master_switch_to_unencrypted-mix/mysqld.1/data/core
Core generated by '/home/dan/build-mariadb-10.3_lastest_upstream/sql/mysqld'
Output from gdb follows. The first stack trace is from the failing thread.
The following stack traces are from all threads (so the failing one is
duplicated).
--------------------------
[New LWP 149092]
[New LWP 149598]
[New LWP 149888]
[New LWP 149440]
[New LWP 149608]
[New LWP 149604]
[New LWP 149597]
[New LWP 149990]
[New LWP 149988]
[New LWP 150285]
[New LWP 149599]
[New LWP 149602]
[New LWP 150374]
[New LWP 153123]
[New LWP 149889]
[New LWP 150284]
[New LWP 150282]
[New LWP 149606]
[New LWP 149564]
[New LWP 149890]
[New LWP 149982]
[New LWP 149986]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/home/dan/build-mariadb-10.3_lastest_upstream/sql/mysqld --defaults-group-suffi'.
Program terminated with signal SIGABRT, Aborted.
#0 __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:57
57 ../sysdeps/unix/sysv/linux/pthread_kill.c: No such file or directory.
[Current thread is 1 (Thread 0x7fa57a84e8c0 (LWP 149092))]
#0 __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:57
#1 0x00000000007f5f08 in handle_fatal_signal (sig=6) at /home/dan/mariadb-server-10.3/sql/signal_handler.cc:305
#2 <signal handler called>
#3 0x00007fa57a42c072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x1572760 <COND_thread_count+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#4 __pthread_cond_wait_common (abstime=0x0, mutex=0x1572158 <LOCK_thread_count>, cond=0x1572738 <COND_thread_count>) at pthread_cond_wait.c:502
#5 __pthread_cond_wait (cond=0x1572738 <COND_thread_count>, mutex=0x1572158 <LOCK_thread_count>) at pthread_cond_wait.c:655
#6 0x000000000057f5a3 in inline_mysql_cond_wait (mutex=<optimized out>, src_line=6179, that=<optimized out>, src_file=<optimized out>) at /home/dan/mariadb-server-10.3/include/mysql/psi/mysql_thread.h:1167
#7 mysqld_main (argc=<optimized out>, argv=<optimized out>) at /home/dan/mariadb-server-10.3/sql/mysqld.cc:6179
#8 0x00007fa577f5e1c1 in __libc_start_main (main=0x57a9e0 <main(int, char**)>, argc=9, argv=0x7ffc0841a158, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffc0841a148) at ../csu/libc-start.c:308
#9 0x000000000057a92a in _start ()

Thread 22 (Thread 0x7fa557fff700 (LWP 149986)):
#0 0x00007fa57a42c072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x3b6aa70) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1 __pthread_cond_wait_common (abstime=0x0, mutex=0x3b6aa20, cond=0x3b6aa48) at pthread_cond_wait.c:502
#2 __pthread_cond_wait (cond=0x3b6aa48, mutex=0x3b6aa20) at pthread_cond_wait.c:655
#3 0x00000000009ce936 in os_event::wait (this=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:163
#4 os_event::wait_low (this=0x3b6aa10, reset_sig_count=9) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:333
#5 0x0000000000a46e74 in srv_resume_thread (slot=<optimized out>, sig_count=0, wait=255, timeout_usec=0) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0srv.cc:906
#6 srv_worker_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0srv.cc:2534
#7 0x00007fa57a4257fc in start_thread (arg=0x7fa557fff700) at pthread_create.c:465
#8 0x00007fa578051b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 21 (Thread 0x7fa563fff700 (LWP 149982)):
#0 0x00007fa57a4305f8 in __GI___nanosleep (requested_time=0x7fa563ffea48, remaining=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:27
#1 0x00000000009cef3f in os_thread_sleep (tm=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0thread.cc:230
#2 0x0000000000a460b5 in srv_master_sleep () at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0srv.cc:2358
#3 srv_master_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0srv.cc:2403
#4 0x00007fa57a4257fc in start_thread (arg=0x7fa563fff700) at pthread_create.c:465
#5 0x00007fa578051b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 20 (Thread 0x7fa5627fc700 (LWP 149890)):
#0 0x00007fa57a42c786 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7fa5627fbcd0, expected=0, futex_word=0x3b6abf0) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1 __pthread_cond_wait_common (abstime=0x7fa5627fbcd0, mutex=0x3b6aba0, cond=0x3b6abc8) at pthread_cond_wait.c:539
#2 __pthread_cond_timedwait (cond=0x3b6abc8, mutex=0x3b6aba0, abstime=0x7fa5627fbcd0) at pthread_cond_wait.c:667
#3 0x00000000009ce802 in os_event::timed_wait (this=<optimized out>, abstime=0x7fa5627fbcd0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:283
#4 0x00000000009cea66 in os_event::wait_time_low (this=0x3b6ab90, time_in_usec=<optimized out>, reset_sig_count=1) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:405
#5 0x0000000000a45590 in srv_monitor_thread () at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0srv.cc:1705
#6 0x00007fa57a4257fc in start_thread (arg=0x7fa5627fc700) at pthread_create.c:465
#7 0x00007fa578051b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 19 (Thread 0x7fa571a0d700 (LWP 149564)):
#0 0x00007fa57a42c786 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7fa571a0cc90, expected=0, futex_word=0x1dee9e8 <COND_checkpoint+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1 __pthread_cond_wait_common (abstime=0x7fa571a0cc90, mutex=0x1dee988 <LOCK_checkpoint>, cond=0x1dee9c0 <COND_checkpoint>) at pthread_cond_wait.c:539
#2 __pthread_cond_timedwait (cond=0x1dee9c0 <COND_checkpoint>, mutex=0x1dee988 <LOCK_checkpoint>, abstime=0x7fa571a0cc90) at pthread_cond_wait.c:667
#3 0x0000000000b86eff in inline_mysql_cond_timedwait (that=0x1dee9c0 <COND_checkpoint>, mutex=0x1dee988 <LOCK_checkpoint>, src_line=116, abstime=<optimized out>, src_file=<optimized out>) at /home/dan/mariadb-server-10.3/include/mysql/psi/mysql_thread.h:1204
#4 my_service_thread_sleep (control=0x14f4a80 <checkpoint_control>, sleep_time=<optimized out>) at /home/dan/mariadb-server-10.3/storage/maria/ma_servicethread.c:115
#5 0x0000000000b804ee in ma_checkpoint_background (arg=0x1e) at /home/dan/mariadb-server-10.3/storage/maria/ma_checkpoint.c:707
#6 0x0000000000c2c5e1 in pfs_spawn_thread (arg=0x3a659b8) at /home/dan/mariadb-server-10.3/storage/perfschema/pfs.cc:1862
#7 0x00007fa57a4257fc in start_thread (arg=0x7fa571a0d700) at pthread_create.c:465
#8 0x00007fa578051b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 18 (Thread 0x7fa5697fa700 (LWP 149606)):
#0 0x00007fa57a01463a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
#1 0x00000000009c6dad in LinuxAIOHandler::collect (this=0x7fa5697f9a90) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:1913
#2 0x00000000009c71b6 in LinuxAIOHandler::poll (this=0x7fa5697f9a90, m1=0x7fa5697f9b78, m2=0x7fa5697f9b80, request=0x7fa5697f9d30) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2058
#3 0x00000000009ca8d3 in os_aio_linux_handler (global_segment=<optimized out>, m1=<optimized out>, m2=<optimized out>, request=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2112
#4 os_aio_handler (segment=<optimized out>, m1=0x7fa5697f9b78, m2=0x7fa5697f9b80, request=0x7fa5697f9d30) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:5638
#5 0x0000000000b0ac5d in fil_aio_wait (segment=5) at /home/dan/mariadb-server-10.3/storage/innobase/fil/fil0fil.cc:4678
#6 0x0000000000a48548 in io_handler_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0start.cc:339
#7 0x00007fa57a4257fc in start_thread (arg=0x7fa5697fa700) at pthread_create.c:465
#8 0x00007fa578051b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 17 (Thread 0x7fa5567fc700 (LWP 150282)):
#0 0x00007fa57a42c072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x3b6ac70) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1 __pthread_cond_wait_common (abstime=0x0, mutex=0x3b6ac20, cond=0x3b6ac48) at pthread_cond_wait.c:502
#2 __pthread_cond_wait (cond=0x3b6ac48, mutex=0x3b6ac20) at pthread_cond_wait.c:655
#3 0x00000000009ce936 in os_event::wait (this=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:163
#4 os_event::wait_low (this=0x3b6ac10, reset_sig_count=1) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:333
#5 0x0000000000abc227 in buf_dump_thread () at /home/dan/mariadb-server-10.3/storage/innobase/buf/buf0dump.cc:828
#6 0x00007fa57a4257fc in start_thread (arg=0x7fa5567fc700) at pthread_create.c:465
#7 0x00007fa578051b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 16 (Thread 0x7fa555ffb700 (LWP 150284)):
#0 0x00007fa57a4305f8 in __GI___nanosleep (requested_time=0x7fa555ffa548, remaining=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:27
#1 0x00000000009cef3f in os_thread_sleep (tm=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0thread.cc:230
#2 0x0000000000aa665a in btr_defragment_thread () at /home/dan/mariadb-server-10.3/storage/innobase/btr/btr0defragment.cc:756
#3 0x00007fa57a4257fc in start_thread (arg=0x7fa555ffb700) at pthread_create.c:465
#4 0x00007fa578051b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 15 (Thread 0x7fa562ffd700 (LWP 149889)):
#0 0x00007fa57a42c786 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7fa562ffca00, expected=0, futex_word=0x3b6ab70) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1 __pthread_cond_wait_common (abstime=0x7fa562ffca00, mutex=0x3b6ab20, cond=0x3b6ab48) at pthread_cond_wait.c:539
#2 __pthread_cond_timedwait (cond=0x3b6ab48, mutex=0x3b6ab20, abstime=0x7fa562ffca00) at pthread_cond_wait.c:667
#3 0x00000000009ce802 in os_event::timed_wait (this=<optimized out>, abstime=0x7fa562ffca00) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:283
#4 0x00000000009cea66 in os_event::wait_time_low (this=0x3b6ab10, time_in_usec=<optimized out>, reset_sig_count=1) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:405
#5 0x0000000000a45c2f in srv_error_monitor_thread () at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0srv.cc:1869
#6 0x00007fa57a4257fc in start_thread (arg=0x7fa562ffd700) at pthread_create.c:465
#7 0x00007fa578051b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 14 (Thread 0x7fa570150700 (LWP 153123)):
#0 0x00007fa5780478c0 in __GI___select (nfds=0, readfds=0x0, writefds=0x0, exceptfds=0x0, timeout=0x7fa57014fca8) at ../sysdeps/unix/sysv/linux/select.c:41
#1 0x0000000000c68abe in my_sleep (m_seconds=<optimized out>) at /home/dan/mariadb-server-10.3/mysys/my_sleep.c:29
#2 0x000000000057b8ca in close_connections () at /home/dan/mariadb-server-10.3/sql/mysqld.cc:1785
#3 kill_server (sig_ptr=<optimized out>) at /home/dan/mariadb-server-10.3/sql/mysqld.cc:2033
#4 0x000000000057b00b in kill_server_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/sql/mysqld.cc:2065
#5 0x0000000000c2c5e1 in pfs_spawn_thread (arg=0x3d8f058) at /home/dan/mariadb-server-10.3/storage/perfschema/pfs.cc:1862
#6 0x00007fa57a4257fc in start_thread (arg=0x7fa570150700) at pthread_create.c:465
#7 0x00007fa578051b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 13 (Thread 0x7fa57030c700 (LWP 150374)):
#0 do_sigwait (sig=0x7fa57030bc60, set=<optimized out>) at ../sysdeps/unix/sysv/linux/sigwait.c:64
#1 __sigwait (set=0x7fa57030bcc8, sig=0x7fa57030bc60) at ../sysdeps/unix/sysv/linux/sigwait.c:96
#2 0x000000000057dfbb in signal_hand (arg=<optimized out>) at /home/dan/mariadb-server-10.3/sql/mysqld.cc:3550
#3 0x0000000000c2c5e1 in pfs_spawn_thread (arg=0x3d8f058) at /home/dan/mariadb-server-10.3/storage/perfschema/pfs.cc:1862
#4 0x00007fa57a4257fc in start_thread (arg=0x7fa57030c700) at pthread_create.c:465
#5 0x00007fa578051b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 12 (Thread 0x7fa56a7fc700 (LWP 149602)):
#0 0x00007fa57a01463a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
#1 0x00000000009c6dad in LinuxAIOHandler::collect (this=0x7fa56a7fba90) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:1913
#2 0x00000000009c71b6 in LinuxAIOHandler::poll (this=0x7fa56a7fba90, m1=0x7fa56a7fbb78, m2=0x7fa56a7fbb80, request=0x7fa56a7fbd30) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2058
#3 0x00000000009ca8d3 in os_aio_linux_handler (global_segment=<optimized out>, m1=<optimized out>, m2=<optimized out>, request=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2112
#4 os_aio_handler (segment=<optimized out>, m1=0x7fa56a7fbb78, m2=0x7fa56a7fbb80, request=0x7fa56a7fbd30) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:5638
#5 0x0000000000b0ac5d in fil_aio_wait (segment=3) at /home/dan/mariadb-server-10.3/storage/innobase/fil/fil0fil.cc:4678
#6 0x0000000000a48548 in io_handler_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0start.cc:339
#7 0x00007fa57a4257fc in start_thread (arg=0x7fa56a7fc700) at pthread_create.c:465
#8 0x00007fa578051b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 11 (Thread 0x7fa56affd700 (LWP 149599)):
#0 0x00007fa57a01463a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
#1 0x00000000009c6dad in LinuxAIOHandler::collect (this=0x7fa56affca90) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:1913
#2 0x00000000009c71b6 in LinuxAIOHandler::poll (this=0x7fa56affca90, m1=0x7fa56affcb78, m2=0x7fa56affcb80, request=0x7fa56affcd30) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2058
#3 0x00000000009ca8d3 in os_aio_linux_handler (global_segment=<optimized out>, m1=<optimized out>, m2=<optimized out>, request=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2112
#4 os_aio_handler (segment=<optimized out>, m1=0x7fa56affcb78, m2=0x7fa56affcb80, request=0x7fa56affcd30) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:5638
#5 0x0000000000b0ac5d in fil_aio_wait (segment=2) at /home/dan/mariadb-server-10.3/storage/innobase/fil/fil0fil.cc:4678
#6 0x0000000000a48548 in io_handler_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0start.cc:339
#7 0x00007fa57a4257fc in start_thread (arg=0x7fa56affd700) at pthread_create.c:465
#8 0x00007fa578051b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 10 (Thread 0x7fa5557fa700 (LWP 150285)):
#0 0x00007fa57a42c072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x3b6ad70) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1 __pthread_cond_wait_common (abstime=0x0, mutex=0x3b6ad20, cond=0x3b6ad48) at pthread_cond_wait.c:502
#2 __pthread_cond_wait (cond=0x3b6ad48, mutex=0x3b6ad20) at pthread_cond_wait.c:655
#3 0x00000000009ce936 in os_event::wait (this=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:163
#4 os_event::wait_low (this=0x3b6ad10, reset_sig_count=1) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:333
#5 0x0000000000aa9f5e in buf_resize_thread () at /home/dan/mariadb-server-10.3/storage/innobase/buf/buf0buf.cc:3125
#6 0x00007fa57a4257fc in start_thread (arg=0x7fa5557fa700) at pthread_create.c:465
#7 0x00007fa578051b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 9 (Thread 0x7fa5577fe700 (LWP 149988)):
#0 0x00007fa57a42c072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x3b6aaf4) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1 __pthread_cond_wait_common (abstime=0x0, mutex=0x3b6aaa0, cond=0x3b6aac8) at pthread_cond_wait.c:502
#2 __pthread_cond_wait (cond=0x3b6aac8, mutex=0x3b6aaa0) at pthread_cond_wait.c:655
#3 0x00000000009ce936 in os_event::wait (this=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:163
#4 os_event::wait_low (this=0x3b6aa90, reset_sig_count=2) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:333
#5 0x0000000000a46e74 in srv_resume_thread (slot=<optimized out>, sig_count=0, wait=255, timeout_usec=0) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0srv.cc:906
#6 srv_worker_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0srv.cc:2534
#7 0x00007fa57a4257fc in start_thread (arg=0x7fa5577fe700) at pthread_create.c:465
#8 0x00007fa578051b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 8 (Thread 0x7fa556ffd700 (LWP 149990)):
#0 0x00007fa57a42c072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x3b6a9f0) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1 __pthread_cond_wait_common (abstime=0x0, mutex=0x3b6a9a0, cond=0x3b6a9c8) at pthread_cond_wait.c:502
#2 __pthread_cond_wait (cond=0x3b6a9c8, mutex=0x3b6a9a0) at pthread_cond_wait.c:655
#3 0x00000000009ce936 in os_event::wait (this=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:163
#4 os_event::wait_low (this=0x3b6a990, reset_sig_count=7) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:333
#5 0x0000000000a46e74 in srv_resume_thread (slot=<optimized out>, sig_count=0, wait=255, timeout_usec=0) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0srv.cc:906
#6 srv_worker_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0srv.cc:2534
#7 0x00007fa57a4257fc in start_thread (arg=0x7fa556ffd700) at pthread_create.c:465
#8 0x00007fa578051b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 7 (Thread 0x7fa56bfff700 (LWP 149597)):
#0 0x00007fa57a01463a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
#1 0x00000000009c6dad in LinuxAIOHandler::collect (this=0x7fa56bffea90) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:1913
#2 0x00000000009c71b6 in LinuxAIOHandler::poll (this=0x7fa56bffea90, m1=0x7fa56bffeb78, m2=0x7fa56bffeb80, request=0x7fa56bffed30) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2058
#3 0x00000000009ca8d3 in os_aio_linux_handler (global_segment=<optimized out>, m1=<optimized out>, m2=<optimized out>, request=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2112
#4 os_aio_handler (segment=<optimized out>, m1=0x7fa56bffeb78, m2=0x7fa56bffeb80, request=0x7fa56bffed30) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:5638
#5 0x0000000000b0ac5d in fil_aio_wait (segment=0) at /home/dan/mariadb-server-10.3/storage/innobase/fil/fil0fil.cc:4678
#6 0x0000000000a48548 in io_handler_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0start.cc:339
#7 0x00007fa57a4257fc in start_thread (arg=0x7fa56bfff700) at pthread_create.c:465
#8 0x00007fa578051b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 6 (Thread 0x7fa569ffb700 (LWP 149604)):
#0 0x00007fa57a01463a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
#1 0x00000000009c6dad in LinuxAIOHandler::collect (this=0x7fa569ffaa90) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:1913
#2 0x00000000009c71b6 in LinuxAIOHandler::poll (this=0x7fa569ffaa90, m1=0x7fa569ffab78, m2=0x7fa569ffab80, request=0x7fa569ffad30) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2058
#3 0x00000000009ca8d3 in os_aio_linux_handler (global_segment=<optimized out>, m1=<optimized out>, m2=<optimized out>, request=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2112
#4 os_aio_handler (segment=<optimized out>, m1=0x7fa569ffab78, m2=0x7fa569ffab80, request=0x7fa569ffad30) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:5638
#5 0x0000000000b0ac5d in fil_aio_wait (segment=4) at /home/dan/mariadb-server-10.3/storage/innobase/fil/fil0fil.cc:4678
#6 0x0000000000a48548 in io_handler_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0start.cc:339
#7 0x00007fa57a4257fc in start_thread (arg=0x7fa569ffb700) at pthread_create.c:465
#8 0x00007fa578051b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 5 (Thread 0x7fa568ff9700 (LWP 149608)):
#0 0x00007fa57a42c786 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7fa568ff87d0, expected=0, futex_word=0x3b6acf4) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1 __pthread_cond_wait_common (abstime=0x7fa568ff87d0, mutex=0x3b6aca0, cond=0x3b6acc8) at pthread_cond_wait.c:539
#2 __pthread_cond_timedwait (cond=0x3b6acc8, mutex=0x3b6aca0, abstime=0x7fa568ff87d0) at pthread_cond_wait.c:667
#3 0x00000000009ce802 in os_event::timed_wait (this=<optimized out>, abstime=0x7fa568ff87d0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:283
#4 0x00000000009cea66 in os_event::wait_time_low (this=0x3b6ac90, time_in_usec=<optimized out>, reset_sig_count=2) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:405
#5 0x0000000000ac10ae in pc_sleep_if_needed (next_loop_time=<optimized out>, sig_count=2, cur_time=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/buf/buf0flu.cc:2674
#6 buf_flush_page_cleaner_coordinator () at /home/dan/mariadb-server-10.3/storage/innobase/buf/buf0flu.cc:3140
#7 0x00007fa57a4257fc in start_thread (arg=0x7fa568ff9700) at pthread_create.c:465
#8 0x00007fa578051b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 4 (Thread 0x7fa572f7e700 (LWP 149440)):
#0 0x00007fa57a42c786 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7fa572f7dcf0, expected=0, futex_word=0x1dfe098 <COND_timer+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1 __pthread_cond_wait_common (abstime=0x7fa572f7dcf0, mutex=0x1dfe040 <LOCK_timer>, cond=0x1dfe070 <COND_timer>) at pthread_cond_wait.c:539
#2 __pthread_cond_timedwait (cond=0x1dfe070 <COND_timer>, mutex=0x1dfe040 <LOCK_timer>, abstime=0x7fa572f7dcf0) at pthread_cond_wait.c:667
#3 0x0000000000c6f460 in inline_mysql_cond_timedwait (mutex=<optimized out>, src_line=292, that=<optimized out>, abstime=<optimized out>, src_file=<optimized out>) at /home/dan/mariadb-server-10.3/include/mysql/psi/mysql_thread.h:1204
#4 timer_handler (arg=<optimized out>) at /home/dan/mariadb-server-10.3/mysys/thr_timer.c:292
#5 0x0000000000c2c5e1 in pfs_spawn_thread (arg=0x3a4fae8) at /home/dan/mariadb-server-10.3/storage/perfschema/pfs.cc:1862
#6 0x00007fa57a4257fc in start_thread (arg=0x7fa572f7e700) at pthread_create.c:465
#7 0x00007fa578051b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 3 (Thread 0x7fa5637fe700 (LWP 149888)):
#0 0x00007fa57a42c786 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7fa5637fdcc0, expected=0, futex_word=0x3d0b7a0) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1 __pthread_cond_wait_common (abstime=0x7fa5637fdcc0, mutex=0x3d0b750, cond=0x3d0b778) at pthread_cond_wait.c:539
#2 __pthread_cond_timedwait (cond=0x3d0b778, mutex=0x3d0b750, abstime=0x7fa5637fdcc0) at pthread_cond_wait.c:667
#3 0x00000000009ce802 in os_event::timed_wait (this=<optimized out>, abstime=0x7fa5637fdcc0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:283
#4 0x00000000009cea66 in os_event::wait_time_low (this=0x3d0b740, time_in_usec=<optimized out>, reset_sig_count=1) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:405
#5 0x00000000009b30f0 in lock_wait_timeout_thread () at /home/dan/mariadb-server-10.3/storage/innobase/lock/lock0wait.cc:547
#6 0x00007fa57a4257fc in start_thread (arg=0x7fa5637fe700) at pthread_create.c:465
#7 0x00007fa578051b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 2 (Thread 0x7fa56b7fe700 (LWP 149598)):
#0 0x00007fa57a01463a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
#1 0x00000000009c6dad in LinuxAIOHandler::collect (this=0x7fa56b7fda90) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:1913
#2 0x00000000009c71b6 in LinuxAIOHandler::poll (this=0x7fa56b7fda90, m1=0x7fa56b7fdb78, m2=0x7fa56b7fdb80, request=0x7fa56b7fdd30) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2058
#3 0x00000000009ca8d3 in os_aio_linux_handler (global_segment=<optimized out>, m1=<optimized out>, m2=<optimized out>, request=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2112
#4 os_aio_handler (segment=<optimized out>, m1=0x7fa56b7fdb78, m2=0x7fa56b7fdb80, request=0x7fa56b7fdd30) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:5638
#5 0x0000000000b0ac5d in fil_aio_wait (segment=1) at /home/dan/mariadb-server-10.3/storage/innobase/fil/fil0fil.cc:4678
#6 0x0000000000a48548 in io_handler_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0start.cc:339
#7 0x00007fa57a4257fc in start_thread (arg=0x7fa56b7fe700) at pthread_create.c:465
#8 0x00007fa578051b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 1 (Thread 0x7fa57a84e8c0 (LWP 149092)):
#0 __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:57
#1 0x00000000007f5f08 in handle_fatal_signal (sig=6) at /home/dan/mariadb-server-10.3/sql/signal_handler.cc:305
#2 <signal handler called>
#3 0x00007fa57a42c072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x1572760 <COND_thread_count+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#4 __pthread_cond_wait_common (abstime=0x0, mutex=0x1572158 <LOCK_thread_count>, cond=0x1572738 <COND_thread_count>) at pthread_cond_wait.c:502
#5 __pthread_cond_wait (cond=0x1572738 <COND_thread_count>, mutex=0x1572158 <LOCK_thread_count>) at pthread_cond_wait.c:655
#6 0x000000000057f5a3 in inline_mysql_cond_wait (mutex=<optimized out>, src_line=6179, that=<optimized out>, src_file=<optimized out>) at /home/dan/mariadb-server-10.3/include/mysql/psi/mysql_thread.h:1167
#7 mysqld_main (argc=<optimized out>, argv=<optimized out>) at /home/dan/mariadb-server-10.3/sql/mysqld.cc:6179
#8 0x00007fa577f5e1c1 in __libc_start_main (main=0x57a9e0 <main(int, char**)>, argc=9, argv=0x7ffc0841a158, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffc0841a148) at ../csu/libc-start.c:308
#9 0x000000000057a92a in _start ()

gcc-7.2.0
encryption.innodb-force-corrupt 'ctr,innodb' w21 [ fail ] Found warnings/errors in server log file!
Test ended at 2018-04-08 19:06:13
line
Attempting backtrace. You can use the following information to find out
^ Found warnings in /home/dan/build-mariadb-10.3_lastest_upstream_gcc/mysql-test/var/21/log/mysqld.1.err
ok

- saving '/home/dan/build-mariadb-10.3_lastest_upstream_gcc/mysql-test/var/21/log/encryption.innodb-force-corrupt-ctr,innodb/' to '/home/dan/build-mariadb-10.3_lastest_upstream_gcc/mysql-test/var/log/encryption.innodb-force-corrupt-ctr,innodb/'
- found 'core' (0/5)

Trying 'dbx' to get a backtrace

Trying 'gdb' to get a backtrace from coredump /home/dan/build-mariadb-10.3_lastest_upstream_gcc/mysql-test/var/log/encryption.innodb-force-corrupt-ctr,innodb/mysqld.1/data/core
Core generated by '/home/dan/build-mariadb-10.3_lastest_upstream_gcc/sql/mysqld'
Output from gdb follows. The first stack trace is from the failing thread.
The following stack traces are from all threads (so the failing one is
duplicated).
--------------------------
[New LWP 54251]
[New LWP 57010]
[New LWP 54488]
[New LWP 57009]
[New LWP 57008]
[New LWP 57012]
[New LWP 54474]
[New LWP 57027]
[New LWP 57011]
[New LWP 57042]
[New LWP 57013]
[New LWP 57093]
[New LWP 57026]
[New LWP 57094]
[New LWP 57014]
[New LWP 57043]
[New LWP 57025]
[New LWP 57039]
[New LWP 57110]
[New LWP 57288]
[New LWP 57095]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/home/dan/build-mariadb-10.3_lastest_upstream_gcc/sql/mysqld --defaults-group-s'.
Program terminated with signal SIGABRT, Aborted.
#0 __pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:57
57 ../sysdeps/unix/sysv/linux/pthread_kill.c: No such file or directory.
[Current thread is 1 (Thread 0x7f9ed1dcdd40 (LWP 54251))]
#0 __pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:57
#1 0x000055c54904d7ea in my_write_core (sig=sig@entry=6) at /home/dan/mariadb-server-10.3/mysys/stacktrace.c:481
#2 0x000055c548ba183a in handle_fatal_signal (sig=6) at /home/dan/mariadb-server-10.3/sql/signal_handler.cc:305
#3 <signal handler called>
#4 0x00007f9ed019c072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55c54996f0c8 <COND_thread_count+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#5 __pthread_cond_wait_common (abstime=0x0, mutex=0x55c54996f820 <LOCK_thread_count>, cond=0x55c54996f0a0 <COND_thread_count>) at pthread_cond_wait.c:502
#6 __pthread_cond_wait (cond=cond@entry=0x55c54996f0a0 <COND_thread_count>, mutex=mutex@entry=0x55c54996f820 <LOCK_thread_count>) at pthread_cond_wait.c:655
#7 0x000055c548945b5a in inline_mysql_cond_wait (src_file=0x55c549091008 "/home/dan/mariadb-server-10.3/sql/mysqld.cc", src_line=6179, mutex=0x55c54996f820 <LOCK_thread_count>, that=0x55c54996f0a0 <COND_thread_count>) at /home/dan/mariadb-server-10.3/include/mysql/psi/mysql_thread.h:1167
#8 mysqld_main (argc=<optimized out>, argv=<optimized out>) at /home/dan/mariadb-server-10.3/sql/mysqld.cc:6179
#9 0x00007f9ecf4dc1c1 in __libc_start_main (main=0x55c548923eb0 <main(int, char**)>, argc=29, argv=0x7ffee3c77748, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffee3c77738) at ../csu/libc-start.c:308
#10 0x000055c5489386aa in _start ()



Thread 21 (Thread 0x7f79b99ac700 (LWP 54523)):
#0 do_sigwait (sig=0x7f79b99abc38, set=<optimized out>) at ../sysdeps/unix/sysv/linux/sigwait.c:64
#1 __sigwait (set=set@entry=0x7f79b99abc90, sig=sig@entry=0x7f79b99abc38) at ../sysdeps/unix/sysv/linux/sigwait.c:96
#2 0x0000556b2620b343 in signal_hand (arg=arg@entry=0x0) at /home/dan/mariadb-server-10.3/sql/mysqld.cc:3550
#3 0x0000556b268e0e7e in pfs_spawn_thread (arg=0x556b292b7928) at /home/dan/mariadb-server-10.3/storage/perfschema/pfs.cc:1862
#4 0x00007f79c226d7fc in start_thread (arg=0x7f79b99ac700) at pthread_create.c:465
#5 0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 20 (Thread 0x7f79b98ce700 (LWP 54641)):
#0 0x00007f79c2274072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x556b2723f0c8 <COND_thread_count+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1 __pthread_cond_wait_common (abstime=0x0, mutex=0x556b2723f820 <LOCK_thread_count>, cond=0x556b2723f0a0 <COND_thread_count>) at pthread_cond_wait.c:502
#2 __pthread_cond_wait (cond=cond@entry=0x556b2723f0a0 <COND_thread_count>, mutex=mutex@entry=0x556b2723f820 <LOCK_thread_count>) at pthread_cond_wait.c:655
#3 0x0000556b2620fa8a in inline_mysql_cond_wait (src_file=0x556b26961008 "/home/dan/mariadb-server-10.3/sql/mysqld.cc", src_line=1850, mutex=0x556b2723f820 <LOCK_thread_count>, that=0x556b2723f0a0 <COND_thread_count>) at /home/dan/mariadb-server-10.3/include/mysql/psi/mysql_thread.h:1167
#4 close_connections () at /home/dan/mariadb-server-10.3/sql/mysqld.cc:1850
#5 kill_server (sig_ptr=0x0) at /home/dan/mariadb-server-10.3/sql/mysqld.cc:2033
#6 0x0000556b2621015e in kill_server_thread (arg=arg@entry=0x7f79b99abc38) at /home/dan/mariadb-server-10.3/sql/mysqld.cc:2065
#7 0x0000556b268e0e7e in pfs_spawn_thread (arg=0x556b292b7928) at /home/dan/mariadb-server-10.3/storage/perfschema/pfs.cc:1862
#8 0x00007f79c226d7fc in start_thread (arg=0x7f79b98ce700) at pthread_create.c:465
#9 0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 19 (Thread 0x7f79a57fa700 (LWP 54503)):
#0 0x00007f79c2274072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x556b29094be4) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1 __pthread_cond_wait_common (abstime=0x0, mutex=0x556b29094b90, cond=0x556b29094bb8) at pthread_cond_wait.c:502
#2 __pthread_cond_wait (cond=cond@entry=0x556b29094bb8, mutex=mutex@entry=0x556b29094b90) at pthread_cond_wait.c:655
#3 0x0000556b2664233f in os_event::wait (this=0x556b29094b80) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:163
#4 os_event::wait_low (this=0x556b29094b80, reset_sig_count=2, reset_sig_count@entry=0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:333
#5 0x0000556b26642805 in os_event_wait_low (event=<optimized out>, reset_sig_count=reset_sig_count@entry=0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:527
#6 0x0000556b266c212d in srv_resume_thread (slot=slot@entry=0x556b271bdef8 <srv_sys+248>, sig_count=sig_count@entry=0, wait=wait@entry=true, timeout_usec=timeout_usec@entry=0) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0srv.cc:906
#7 0x0000556b266c723d in srv_worker_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0srv.cc:2534
#8 0x00007f79c226d7fc in start_thread (arg=0x7f79a57fa700) at pthread_create.c:465
#9 0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 18 (Thread 0x7f79927fc700 (LWP 54508)):
#0 0x00007f79c2274072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x556b29094f60) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1 __pthread_cond_wait_common (abstime=0x0, mutex=0x556b29094f10, cond=0x556b29094f38) at pthread_cond_wait.c:502
#2 __pthread_cond_wait (cond=cond@entry=0x556b29094f38, mutex=mutex@entry=0x556b29094f10) at pthread_cond_wait.c:655
#3 0x0000556b2664233f in os_event::wait (this=0x556b29094f00) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:163
#4 os_event::wait_low (this=0x556b29094f00, reset_sig_count=1, reset_sig_count@entry=0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:333
#5 0x0000556b26642805 in os_event_wait_low (event=<optimized out>, reset_sig_count=reset_sig_count@entry=0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:527
#6 0x0000556b26748f42 in buf_resize_thread () at /home/dan/mariadb-server-10.3/storage/innobase/buf/buf0buf.cc:3125
#7 0x00007f79c226d7fc in start_thread (arg=0x7f79927fc700) at pthread_create.c:465
#8 0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 17 (Thread 0x7f79a4ff9700 (LWP 54504)):
#0 0x00007f79c2274072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x556b29094c64) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1 __pthread_cond_wait_common (abstime=0x0, mutex=0x556b29094c10, cond=0x556b29094c38) at pthread_cond_wait.c:502
#2 __pthread_cond_wait (cond=cond@entry=0x556b29094c38, mutex=mutex@entry=0x556b29094c10) at pthread_cond_wait.c:655
#3 0x0000556b2664233f in os_event::wait (this=0x556b29094c00) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:163
#4 os_event::wait_low (this=0x556b29094c00, reset_sig_count=2, reset_sig_count@entry=0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:333
#5 0x0000556b26642805 in os_event_wait_low (event=<optimized out>, reset_sig_count=reset_sig_count@entry=0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:527
#6 0x0000556b266c212d in srv_resume_thread (slot=slot@entry=0x556b271bdf30 <srv_sys+304>, sig_count=sig_count@entry=0, wait=wait@entry=true, timeout_usec=timeout_usec@entry=0) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0srv.cc:906
#7 0x0000556b266c723d in srv_worker_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0srv.cc:2534
#8 0x00007f79c226d7fc in start_thread (arg=0x7f79a4ff9700) at pthread_create.c:465
#9 0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 16 (Thread 0x7f7992ffd700 (LWP 54507)):
#0 0x00007f79c22785f8 in __GI___nanosleep (requested_time=requested_time@entry=0x7f7992ffc690, remaining=remaining@entry=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:27
#1 0x0000556b266429cc in os_thread_sleep (tm=tm@entry=1000000) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0thread.cc:230
#2 0x0000556b2673b4ea in btr_defragment_thread () at /home/dan/mariadb-server-10.3/storage/innobase/btr/btr0defragment.cc:768
#3 0x00007f79c226d7fc in start_thread (arg=0x7f7992ffd700) at pthread_create.c:465
#4 0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 15 (Thread 0x7f79937fe700 (LWP 54506)):
#0 0x00007f79c2274072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x556b29094e60) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1 __pthread_cond_wait_common (abstime=0x0, mutex=0x556b29094e10, cond=0x556b29094e38) at pthread_cond_wait.c:502
#2 __pthread_cond_wait (cond=cond@entry=0x556b29094e38, mutex=mutex@entry=0x556b29094e10) at pthread_cond_wait.c:655
#3 0x0000556b2664233f in os_event::wait (this=0x556b29094e00) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:163
#4 os_event::wait_low (this=0x556b29094e00, reset_sig_count=1, reset_sig_count@entry=0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:333
#5 0x0000556b26642805 in os_event_wait_low (event=<optimized out>, reset_sig_count=reset_sig_count@entry=0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:527
#6 0x0000556b2674ff76 in buf_dump_thread () at /home/dan/mariadb-server-10.3/storage/innobase/buf/buf0dump.cc:828
#7 0x00007f79c226d7fc in start_thread (arg=0x7f79937fe700) at pthread_create.c:465
#8 0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 14 (Thread 0x7f79a7fff700 (LWP 54490)):
#0 0x00007f79c2274786 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7f79a7ffeb50, expected=0, futex_word=0x556b29094d60) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1 __pthread_cond_wait_common (abstime=0x7f79a7ffeb50, mutex=0x556b29094d10, cond=0x556b29094d38) at pthread_cond_wait.c:539
#2 __pthread_cond_timedwait (cond=cond@entry=0x556b29094d38, mutex=mutex@entry=0x556b29094d10, abstime=abstime@entry=0x7f79a7ffeb50) at pthread_cond_wait.c:667
#3 0x0000556b26642211 in os_event::timed_wait (this=this@entry=0x556b29094d00, abstime=abstime@entry=0x7f79a7ffeb50) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:283
#4 0x0000556b26642431 in os_event::wait_time_low (this=0x556b29094d00, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=1) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:405
#5 0x0000556b266427f5 in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:510
#6 0x0000556b266c5133 in srv_error_monitor_thread () at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0srv.cc:1869
#7 0x00007f79c226d7fc in start_thread (arg=0x7f79a7fff700) at pthread_create.c:465
#8 0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 13 (Thread 0x7f79b3fff700 (LWP 54477)):
#0 0x00007f79c388963a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
#1 0x0000556b2663d769 in LinuxAIOHandler::collect (this=this@entry=0x7f79b3ffeaf0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:1913
#2 0x0000556b2663dc10 in LinuxAIOHandler::poll (this=this@entry=0x7f79b3ffeaf0, m1=m1@entry=0x7f79b3ffeb90, m2=m2@entry=0x7f79b3ffeb98, request=request@entry=0x7f79b3ffeba0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2058
#3 0x0000556b266400ab in os_aio_linux_handler (request=0x7f79b3ffeba0, m2=0x7f79b3ffeb98, m1=0x7f79b3ffeb90, global_segment=2) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2112
#4 os_aio_handler (segment=segment@entry=2, m1=m1@entry=0x7f79b3ffeb90, m2=m2@entry=0x7f79b3ffeb98, request=request@entry=0x7f79b3ffeba0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:5638
#5 0x0000556b2679d1e4 in fil_aio_wait (segment=segment@entry=2) at /home/dan/mariadb-server-10.3/storage/innobase/fil/fil0fil.cc:4678
#6 0x0000556b266c8720 in io_handler_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0start.cc:339
#7 0x00007f79c226d7fc in start_thread (arg=0x7f79b3fff700) at pthread_create.c:465
#8 0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 12 (Thread 0x7f79b17fa700 (LWP 54501)):
#0 0x00007f79c22785f8 in __GI___nanosleep (requested_time=requested_time@entry=0x7f79b17f9d10, remaining=remaining@entry=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:27
#1 0x0000556b266429cc in os_thread_sleep (tm=tm@entry=1000000) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0thread.cc:230
#2 0x0000556b266c6c9c in srv_master_sleep () at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0srv.cc:2358
#3 srv_master_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0srv.cc:2403
#4 0x00007f79c226d7fc in start_thread (arg=0x7f79b17fa700) at pthread_create.c:465
#5 0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 11 (Thread 0x7f79a77fe700 (LWP 54491)):
#0 0x00007f79c2274786 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7f79a77fdc80, expected=0, futex_word=0x556b29094de0) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1 __pthread_cond_wait_common (abstime=0x7f79a77fdc80, mutex=0x556b29094d90, cond=0x556b29094db8) at pthread_cond_wait.c:539
#2 __pthread_cond_timedwait (cond=cond@entry=0x556b29094db8, mutex=mutex@entry=0x556b29094d90, abstime=abstime@entry=0x7f79a77fdc80) at pthread_cond_wait.c:667
#3 0x0000556b26642211 in os_event::timed_wait (this=this@entry=0x556b29094d80, abstime=abstime@entry=0x7f79a77fdc80) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:283
#4 0x0000556b26642431 in os_event::wait_time_low (this=0x556b29094d80, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=1) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:405
#5 0x0000556b266427f5 in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:510
#6 0x0000556b266c6320 in srv_monitor_thread () at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0srv.cc:1705
#7 0x00007f79c226d7fc in start_thread (arg=0x7f79a77fe700) at pthread_create.c:465
#8 0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 10 (Thread 0x7f79b0ff9700 (LWP 54489)):
#0 0x00007f79c2274786 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7f79b0ff8bf0, expected=0, futex_word=0x556b292358b0) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1 __pthread_cond_wait_common (abstime=0x7f79b0ff8bf0, mutex=0x556b29235860, cond=0x556b29235888) at pthread_cond_wait.c:539
#2 __pthread_cond_timedwait (cond=cond@entry=0x556b29235888, mutex=mutex@entry=0x556b29235860, abstime=abstime@entry=0x7f79b0ff8bf0) at pthread_cond_wait.c:667
#3 0x0000556b26642211 in os_event::timed_wait (this=this@entry=0x556b29235850, abstime=abstime@entry=0x7f79b0ff8bf0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:283
#4 0x0000556b26642431 in os_event::wait_time_low (this=this@entry=0x556b29235850, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=1) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:405
#5 0x0000556b266427f5 in os_event_wait_time_low (event=event@entry=0x556b29235850, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:510
#6 0x0000556b2661e5e1 in lock_wait_timeout_thread () at /home/dan/mariadb-server-10.3/storage/innobase/lock/lock0wait.cc:547
#7 0x00007f79c226d7fc in start_thread (arg=0x7f79b0ff9700) at pthread_create.c:465
#8 0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 9 (Thread 0x7f79b37fe700 (LWP 54478)):
#0 0x00007f79c388963a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
#1 0x0000556b2663d769 in LinuxAIOHandler::collect (this=this@entry=0x7f79b37fdaf0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:1913
#2 0x0000556b2663dc10 in LinuxAIOHandler::poll (this=this@entry=0x7f79b37fdaf0, m1=m1@entry=0x7f79b37fdb90, m2=m2@entry=0x7f79b37fdb98, request=request@entry=0x7f79b37fdba0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2058
#3 0x0000556b266400ab in os_aio_linux_handler (request=0x7f79b37fdba0, m2=0x7f79b37fdb98, m1=0x7f79b37fdb90, global_segment=3) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2112
#4 os_aio_handler (segment=segment@entry=3, m1=m1@entry=0x7f79b37fdb90, m2=m2@entry=0x7f79b37fdb98, request=request@entry=0x7f79b37fdba0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:5638
#5 0x0000556b2679d1e4 in fil_aio_wait (segment=segment@entry=3) at /home/dan/mariadb-server-10.3/storage/innobase/fil/fil0fil.cc:4678
#6 0x0000556b266c8720 in io_handler_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0start.cc:339
#7 0x00007f79c226d7fc in start_thread (arg=0x7f79b37fe700) at pthread_create.c:465
#8 0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 8 (Thread 0x7f79b1ffb700 (LWP 54481)):
#0 0x00007f79c2274786 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7f79b1ffaa90, expected=0, futex_word=0x556b29094ee4) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1 __pthread_cond_wait_common (abstime=0x7f79b1ffaa90, mutex=0x556b29094e90, cond=0x556b29094eb8) at pthread_cond_wait.c:539
#2 __pthread_cond_timedwait (cond=cond@entry=0x556b29094eb8, mutex=mutex@entry=0x556b29094e90, abstime=abstime@entry=0x7f79b1ffaa90) at pthread_cond_wait.c:667
#3 0x0000556b26642211 in os_event::timed_wait (this=this@entry=0x556b29094e80, abstime=abstime@entry=0x7f79b1ffaa90) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:283
#4 0x0000556b26642431 in os_event::wait_time_low (this=0x556b29094e80, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=reset_sig_count@entry=2) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:405
#5 0x0000556b266427f5 in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=reset_sig_count@entry=2) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:510
#6 0x0000556b2675764b in pc_sleep_if_needed (cur_time=1523232365070, sig_count=2, next_loop_time=1523232366070) at /home/dan/mariadb-server-10.3/storage/innobase/buf/buf0flu.cc:2675
#7 buf_flush_page_cleaner_coordinator () at /home/dan/mariadb-server-10.3/storage/innobase/buf/buf0flu.cc:3140
#8 0x00007f79c226d7fc in start_thread (arg=0x7f79b1ffb700) at pthread_create.c:465
#9 0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 7 (Thread 0x7f79b2ffd700 (LWP 54479)):
#0 0x00007f79c388963a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
#1 0x0000556b2663d769 in LinuxAIOHandler::collect (this=this@entry=0x7f79b2ffcaf0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:1913
#2 0x0000556b2663dc10 in LinuxAIOHandler::poll (this=this@entry=0x7f79b2ffcaf0, m1=m1@entry=0x7f79b2ffcb90, m2=m2@entry=0x7f79b2ffcb98, request=request@entry=0x7f79b2ffcba0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2058
#3 0x0000556b266400ab in os_aio_linux_handler (request=0x7f79b2ffcba0, m2=0x7f79b2ffcb98, m1=0x7f79b2ffcb90, global_segment=4) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2112
#4 os_aio_handler (segment=segment@entry=4, m1=m1@entry=0x7f79b2ffcb90, m2=m2@entry=0x7f79b2ffcb98, request=request@entry=0x7f79b2ffcba0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:5638
#5 0x0000556b2679d1e4 in fil_aio_wait (segment=segment@entry=4) at /home/dan/mariadb-server-10.3/storage/innobase/fil/fil0fil.cc:4678
#6 0x0000556b266c8720 in io_handler_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0start.cc:339
#7 0x00007f79c226d7fc in start_thread (arg=0x7f79b2ffd700) at pthread_create.c:465
#8 0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 6 (Thread 0x7f79b27fc700 (LWP 54480)):
#0 0x00007f79c388963a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
#1 0x0000556b2663d769 in LinuxAIOHandler::collect (this=this@entry=0x7f79b27fbaf0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:1913
#2 0x0000556b2663dc10 in LinuxAIOHandler::poll (this=this@entry=0x7f79b27fbaf0, m1=m1@entry=0x7f79b27fbb90, m2=m2@entry=0x7f79b27fbb98, request=request@entry=0x7f79b27fbba0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2058
#3 0x0000556b266400ab in os_aio_linux_handler (request=0x7f79b27fbba0, m2=0x7f79b27fbb98, m1=0x7f79b27fbb90, global_segment=5) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2112
#4 os_aio_handler (segment=segment@entry=5, m1=m1@entry=0x7f79b27fbb90, m2=m2@entry=0x7f79b27fbb98, request=request@entry=0x7f79b27fbba0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:5638
#5 0x0000556b2679d1e4 in fil_aio_wait (segment=segment@entry=5) at /home/dan/mariadb-server-10.3/storage/innobase/fil/fil0fil.cc:4678
#6 0x0000556b266c8720 in io_handler_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0start.cc:339
#7 0x00007f79c226d7fc in start_thread (arg=0x7f79b27fc700) at pthread_create.c:465
#8 0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 5 (Thread 0x7f79b97ac700 (LWP 54475)):
#0 0x00007f79c388963a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
#1 0x0000556b2663d769 in LinuxAIOHandler::collect (this=this@entry=0x7f79b97abaf0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:1913
#2 0x0000556b2663dc10 in LinuxAIOHandler::poll (this=this@entry=0x7f79b97abaf0, m1=m1@entry=0x7f79b97abb90, m2=m2@entry=0x7f79b97abb98, request=request@entry=0x7f79b97abba0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2058
#3 0x0000556b266400ab in os_aio_linux_handler (request=0x7f79b97abba0, m2=0x7f79b97abb98, m1=0x7f79b97abb90, global_segment=0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2112
#4 os_aio_handler (segment=segment@entry=0, m1=m1@entry=0x7f79b97abb90, m2=m2@entry=0x7f79b97abb98, request=request@entry=0x7f79b97abba0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:5638
#5 0x0000556b2679d1e4 in fil_aio_wait (segment=segment@entry=0) at /home/dan/mariadb-server-10.3/storage/innobase/fil/fil0fil.cc:4678
#6 0x0000556b266c8720 in io_handler_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0start.cc:339
#7 0x00007f79c226d7fc in start_thread (arg=0x7f79b97ac700) at pthread_create.c:465
#8 0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 4 (Thread 0x7f79b8fab700 (LWP 54476)):
#0 0x00007f79c388963a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
#1 0x0000556b2663d769 in LinuxAIOHandler::collect (this=this@entry=0x7f79b8faaaf0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:1913
#2 0x0000556b2663dc10 in LinuxAIOHandler::poll (this=this@entry=0x7f79b8faaaf0, m1=m1@entry=0x7f79b8faab90, m2=m2@entry=0x7f79b8faab98, request=request@entry=0x7f79b8faaba0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2058
#3 0x0000556b266400ab in os_aio_linux_handler (request=0x7f79b8faaba0, m2=0x7f79b8faab98, m1=0x7f79b8faab90, global_segment=1) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2112
#4 os_aio_handler (segment=segment@entry=1, m1=m1@entry=0x7f79b8faab90, m2=m2@entry=0x7f79b8faab98, request=request@entry=0x7f79b8faaba0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:5638
#5 0x0000556b2679d1e4 in fil_aio_wait (segment=segment@entry=1) at /home/dan/mariadb-server-10.3/storage/innobase/fil/fil0fil.cc:4678
#6 0x0000556b266c8720 in io_handler_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0start.cc:339
#7 0x00007f79c226d7fc in start_thread (arg=0x7f79b8fab700) at pthread_create.c:465
#8 0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 3 (Thread 0x7f79bb063700 (LWP 54470)):
#0 0x00007f79c2274786 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7f79bb062c70, expected=0, futex_word=0x556b27abeaa8 <COND_checkpoint+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1 __pthread_cond_wait_common (abstime=0x7f79bb062c70, mutex=0x556b27abeac0 <LOCK_checkpoint>, cond=0x556b27abea80 <COND_checkpoint>) at pthread_cond_wait.c:539
#2 __pthread_cond_timedwait (cond=cond@entry=0x556b27abea80 <COND_checkpoint>, mutex=mutex@entry=0x556b27abeac0 <LOCK_checkpoint>, abstime=abstime@entry=0x7f79bb062c70) at pthread_cond_wait.c:667
#3 0x0000556b268303ff in inline_mysql_cond_timedwait (src_file=0x556b26aad278 "/home/dan/mariadb-server-10.3/storage/maria/ma_servicethread.c", src_line=116, abstime=0x7f79bb062c70, mutex=0x556b27abeac0 <LOCK_checkpoint>, that=0x556b27abea80 <COND_checkpoint>) at /home/dan/mariadb-server-10.3/include/mysql/psi/mysql_thread.h:1204
#4 my_service_thread_sleep (control=control@entry=0x556b271c0900 <checkpoint_control>, sleep_time=sleep_time@entry=30000000000) at /home/dan/mariadb-server-10.3/storage/maria/ma_servicethread.c:115
#5 0x0000556b26829c9d in ma_checkpoint_background (arg=arg@entry=0x1e) at /home/dan/mariadb-server-10.3/storage/maria/ma_checkpoint.c:707
#6 0x0000556b268e0e7e in pfs_spawn_thread (arg=0x556b28f8fa88) at /home/dan/mariadb-server-10.3/storage/perfschema/pfs.cc:1862
#7 0x00007f79c226d7fc in start_thread (arg=0x7f79bb063700) at pthread_create.c:465
#8 0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 2 (Thread 0x7f79bc5d4700 (LWP 54463)):
#0 0x00007f79c2274786 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7f79bc5d3ce0, expected=0, futex_word=0x556b27ace548 <COND_timer+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1 __pthread_cond_wait_common (abstime=0x7f79bc5d3ce0, mutex=0x556b27ace560 <LOCK_timer>, cond=0x556b27ace520 <COND_timer>) at pthread_cond_wait.c:539
#2 __pthread_cond_timedwait (cond=cond@entry=0x556b27ace520 <COND_timer>, mutex=mutex@entry=0x556b27ace560 <LOCK_timer>, abstime=abstime@entry=0x7f79bc5d3ce0) at pthread_cond_wait.c:667
#3 0x0000556b269219cf in inline_mysql_cond_timedwait (that=0x556b27ace520 <COND_timer>, mutex=0x556b27ace560 <LOCK_timer>, src_file=0x556b26ac1768 "/home/dan/mariadb-server-10.3/mysys/thr_timer.c", src_line=292, abstime=0x7f79bc5d3ce0) at /home/dan/mariadb-server-10.3/include/mysql/psi/mysql_thread.h:1204
#4 timer_handler (arg=arg@entry=0x0) at /home/dan/mariadb-server-10.3/mysys/thr_timer.c:292
#5 0x0000556b268e0e7e in pfs_spawn_thread (arg=0x556b28f7bb18) at /home/dan/mariadb-server-10.3/storage/perfschema/pfs.cc:1862
#6 0x00007f79c226d7fc in start_thread (arg=0x7f79bc5d4700) at pthread_create.c:465
#7 0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 1 (Thread 0x7f79c3ea5d40 (LWP 54253)):
#0 __pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:57
#1 0x0000556b2691d7ea in my_write_core (sig=sig@entry=6) at /home/dan/mariadb-server-10.3/mysys/stacktrace.c:481
#2 0x0000556b2647183a in handle_fatal_signal (sig=6) at /home/dan/mariadb-server-10.3/sql/signal_handler.cc:305
#3 <signal handler called>
#4 0x00007f79c2274072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x556b2723f0c8 <COND_thread_count+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#5 __pthread_cond_wait_common (abstime=0x0, mutex=0x556b2723f820 <LOCK_thread_count>, cond=0x556b2723f0a0 <COND_thread_count>) at pthread_cond_wait.c:502
#6 __pthread_cond_wait (cond=cond@entry=0x556b2723f0a0 <COND_thread_count>, mutex=mutex@entry=0x556b2723f820 <LOCK_thread_count>) at pthread_cond_wait.c:655
#7 0x0000556b26215b5a in inline_mysql_cond_wait (src_file=0x556b26961008 "/home/dan/mariadb-server-10.3/sql/mysqld.cc", src_line=6179, mutex=0x556b2723f820 <LOCK_thread_count>, that=0x556b2723f0a0 <COND_thread_count>) at /home/dan/mariadb-server-10.3/include/mysql/psi/mysql_thread.h:1167
#8 mysqld_main (argc=<optimized out>, argv=<optimized out>) at /home/dan/mariadb-server-10.3/sql/mysqld.cc:6179
#9 0x00007f79c15b41c1 in __libc_start_main (main=0x556b261f3eb0 <main(int, char**)>, argc=28, argv=0x7ffce77e2ef8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffce77e2ee8) at ../csu/libc-start.c:308
#10 0x0000556b262086aa in _start ()

Compressed file /home/dan/build-mariadb-10.3_lastest_upstream_gcc/mysql-test/var/log/encryption.innodb-bad-key-change2-innodb/mysqld.1/data/core



encryption.innodb-bad-key-change2 'innodb' w44 [ fail ] Found warnings/errors in server log file!
Test ended at 2018-04-08 19:06:18
line
Attempting backtrace. You can use the following information to find out
^ Found warnings in /home/dan/build-mariadb-10.3_lastest_upstream_gcc/mysql-test/var/44/log/mysqld.1.err
ok

- saving '/home/dan/build-mariadb-10.3_lastest_upstream_gcc/mysql-test/var/44/log/encryption.innodb-bad-key-change2-innodb/' to '/home/dan/build-mariadb-10.3_lastest_upstream_gcc/mysql-test/var/log/encryption.innodb-bad-key-change2-innodb/'
- found 'core' (1/5)

Trying 'dbx' to get a backtrace

Trying 'gdb' to get a backtrace from coredump /home/dan/build-mariadb-10.3_lastest_upstream_gcc/mysql-test/var/log/encryption.innodb-bad-key-change2-innodb/mysqld.1/data/core
Core generated by '/home/dan/build-mariadb-10.3_lastest_upstream_gcc/sql/mysqld'
Output from gdb follows. The first stack trace is from the failing thread.
The following stack traces are from all threads (so the failing one is
duplicated).
--------------------------
[New LWP 54253]
[New LWP 54463]
[New LWP 54470]
[New LWP 54476]
[New LWP 54475]
[New LWP 54480]
[New LWP 54479]
[New LWP 54481]
[New LWP 54478]
[New LWP 54489]
[New LWP 54491]
[New LWP 54501]
[New LWP 54477]
[New LWP 54490]
[New LWP 54506]
[New LWP 54507]
[New LWP 54504]
[New LWP 54508]
[New LWP 54503]
[New LWP 54641]
[New LWP 54523]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/home/dan/build-mariadb-10.3_lastest_upstream_gcc/sql/mysqld --defaults-group-s'.
Program terminated with signal SIGABRT, Aborted.
#0 __pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:57
57 ../sysdeps/unix/sysv/linux/pthread_kill.c: No such file or directory.
[Current thread is 1 (Thread 0x7f79c3ea5d40 (LWP 54253))]
#0 __pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:57
#1 0x0000556b2691d7ea in my_write_core (sig=sig@entry=6) at /home/dan/mariadb-server-10.3/mysys/stacktrace.c:481
#2 0x0000556b2647183a in handle_fatal_signal (sig=6) at /home/dan/mariadb-server-10.3/sql/signal_handler.cc:305
#3 <signal handler called>
#4 0x00007f79c2274072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x556b2723f0c8 <COND_thread_count+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#5 __pthread_cond_wait_common (abstime=0x0, mutex=0x556b2723f820 <LOCK_thread_count>, cond=0x556b2723f0a0 <COND_thread_count>) at pthread_cond_wait.c:502
#6 __pthread_cond_wait (cond=cond@entry=0x556b2723f0a0 <COND_thread_count>, mutex=mutex@entry=0x556b2723f820 <LOCK_thread_count>) at pthread_cond_wait.c:655
#7 0x0000556b26215b5a in inline_mysql_cond_wait (src_file=0x556b26961008 "/home/dan/mariadb-server-10.3/sql/mysqld.cc", src_line=6179, mutex=0x556b2723f820 <LOCK_thread_count>, that=0x556b2723f0a0 <COND_thread_count>) at /home/dan/mariadb-server-10.3/include/mysql/psi/mysql_thread.h:1167
#8 mysqld_main (argc=<optimized out>, argv=<optimized out>) at /home/dan/mariadb-server-10.3/sql/mysqld.cc:6179
#9 0x00007f79c15b41c1 in __libc_start_main (main=0x556b261f3eb0 <main(int, char**)>, argc=28, argv=0x7ffce77e2ef8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffce77e2ee8) at ../csu/libc-start.c:308
#10 0x0000556b262086aa in _start ()

Thread 21 (Thread 0x7f79b99ac700 (LWP 54523)):
#0 do_sigwait (sig=0x7f79b99abc38, set=<optimized out>) at ../sysdeps/unix/sysv/linux/sigwait.c:64
#1 __sigwait (set=set@entry=0x7f79b99abc90, sig=sig@entry=0x7f79b99abc38) at ../sysdeps/unix/sysv/linux/sigwait.c:96
#2 0x0000556b2620b343 in signal_hand (arg=arg@entry=0x0) at /home/dan/mariadb-server-10.3/sql/mysqld.cc:3550
#3 0x0000556b268e0e7e in pfs_spawn_thread (arg=0x556b292b7928) at /home/dan/mariadb-server-10.3/storage/perfschema/pfs.cc:1862
#4 0x00007f79c226d7fc in start_thread (arg=0x7f79b99ac700) at pthread_create.c:465
#5 0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 20 (Thread 0x7f79b98ce700 (LWP 54641)):
#0 0x00007f79c2274072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x556b2723f0c8 <COND_thread_count+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1 __pthread_cond_wait_common (abstime=0x0, mutex=0x556b2723f820 <LOCK_thread_count>, cond=0x556b2723f0a0 <COND_thread_count>) at pthread_cond_wait.c:502
#2 __pthread_cond_wait (cond=cond@entry=0x556b2723f0a0 <COND_thread_count>, mutex=mutex@entry=0x556b2723f820 <LOCK_thread_count>) at pthread_cond_wait.c:655
#3 0x0000556b2620fa8a in inline_mysql_cond_wait (src_file=0x556b26961008 "/home/dan/mariadb-server-10.3/sql/mysqld.cc", src_line=1850, mutex=0x556b2723f820 <LOCK_thread_count>, that=0x556b2723f0a0 <COND_thread_count>) at /home/dan/mariadb-server-10.3/include/mysql/psi/mysql_thread.h:1167
#4 close_connections () at /home/dan/mariadb-server-10.3/sql/mysqld.cc:1850
#5 kill_server (sig_ptr=0x0) at /home/dan/mariadb-server-10.3/sql/mysqld.cc:2033
#6 0x0000556b2621015e in kill_server_thread (arg=arg@entry=0x7f79b99abc38) at /home/dan/mariadb-server-10.3/sql/mysqld.cc:2065
#7 0x0000556b268e0e7e in pfs_spawn_thread (arg=0x556b292b7928) at /home/dan/mariadb-server-10.3/storage/perfschema/pfs.cc:1862
#8 0x00007f79c226d7fc in start_thread (arg=0x7f79b98ce700) at pthread_create.c:465
#9 0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 19 (Thread 0x7f79a57fa700 (LWP 54503)):
#0 0x00007f79c2274072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x556b29094be4) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1 __pthread_cond_wait_common (abstime=0x0, mutex=0x556b29094b90, cond=0x556b29094bb8) at pthread_cond_wait.c:502
#2 __pthread_cond_wait (cond=cond@entry=0x556b29094bb8, mutex=mutex@entry=0x556b29094b90) at pthread_cond_wait.c:655
#3 0x0000556b2664233f in os_event::wait (this=0x556b29094b80) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:163
#4 os_event::wait_low (this=0x556b29094b80, reset_sig_count=2, reset_sig_count@entry=0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:333
#5 0x0000556b26642805 in os_event_wait_low (event=<optimized out>, reset_sig_count=reset_sig_count@entry=0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:527
#6 0x0000556b266c212d in srv_resume_thread (slot=slot@entry=0x556b271bdef8 <srv_sys+248>, sig_count=sig_count@entry=0, wait=wait@entry=true, timeout_usec=timeout_usec@entry=0) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0srv.cc:906
#7 0x0000556b266c723d in srv_worker_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0srv.cc:2534
#8 0x00007f79c226d7fc in start_thread (arg=0x7f79a57fa700) at pthread_create.c:465
#9 0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 18 (Thread 0x7f79927fc700 (LWP 54508)):
#0 0x00007f79c2274072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x556b29094f60) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1 __pthread_cond_wait_common (abstime=0x0, mutex=0x556b29094f10, cond=0x556b29094f38) at pthread_cond_wait.c:502
#2 __pthread_cond_wait (cond=cond@entry=0x556b29094f38, mutex=mutex@entry=0x556b29094f10) at pthread_cond_wait.c:655
#3 0x0000556b2664233f in os_event::wait (this=0x556b29094f00) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:163
#4 os_event::wait_low (this=0x556b29094f00, reset_sig_count=1, reset_sig_count@entry=0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:333
#5 0x0000556b26642805 in os_event_wait_low (event=<optimized out>, reset_sig_count=reset_sig_count@entry=0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:527
#6 0x0000556b26748f42 in buf_resize_thread () at /home/dan/mariadb-server-10.3/storage/innobase/buf/buf0buf.cc:3125
#7 0x00007f79c226d7fc in start_thread (arg=0x7f79927fc700) at pthread_create.c:465
#8 0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 17 (Thread 0x7f79a4ff9700 (LWP 54504)):
#0 0x00007f79c2274072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x556b29094c64) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1 __pthread_cond_wait_common (abstime=0x0, mutex=0x556b29094c10, cond=0x556b29094c38) at pthread_cond_wait.c:502
#2 __pthread_cond_wait (cond=cond@entry=0x556b29094c38, mutex=mutex@entry=0x556b29094c10) at pthread_cond_wait.c:655
#3 0x0000556b2664233f in os_event::wait (this=0x556b29094c00) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:163
#4 os_event::wait_low (this=0x556b29094c00, reset_sig_count=2, reset_sig_count@entry=0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:333
#5 0x0000556b26642805 in os_event_wait_low (event=<optimized out>, reset_sig_count=reset_sig_count@entry=0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:527
#6 0x0000556b266c212d in srv_resume_thread (slot=slot@entry=0x556b271bdf30 <srv_sys+304>, sig_count=sig_count@entry=0, wait=wait@entry=true, timeout_usec=timeout_usec@entry=0) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0srv.cc:906
#7 0x0000556b266c723d in srv_worker_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0srv.cc:2534
#8 0x00007f79c226d7fc in start_thread (arg=0x7f79a4ff9700) at pthread_create.c:465
#9 0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 16 (Thread 0x7f7992ffd700 (LWP 54507)):
#0 0x00007f79c22785f8 in __GI___nanosleep (requested_time=requested_time@entry=0x7f7992ffc690, remaining=remaining@entry=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:27
#1 0x0000556b266429cc in os_thread_sleep (tm=tm@entry=1000000) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0thread.cc:230
#2 0x0000556b2673b4ea in btr_defragment_thread () at /home/dan/mariadb-server-10.3/storage/innobase/btr/btr0defragment.cc:768
#3 0x00007f79c226d7fc in start_thread (arg=0x7f7992ffd700) at pthread_create.c:465
#4 0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 15 (Thread 0x7f79937fe700 (LWP 54506)):
#0 0x00007f79c2274072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x556b29094e60) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1 __pthread_cond_wait_common (abstime=0x0, mutex=0x556b29094e10, cond=0x556b29094e38) at pthread_cond_wait.c:502
#2 __pthread_cond_wait (cond=cond@entry=0x556b29094e38, mutex=mutex@entry=0x556b29094e10) at pthread_cond_wait.c:655
#3 0x0000556b2664233f in os_event::wait (this=0x556b29094e00) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:163
#4 os_event::wait_low (this=0x556b29094e00, reset_sig_count=1, reset_sig_count@entry=0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:333
#5 0x0000556b26642805 in os_event_wait_low (event=<optimized out>, reset_sig_count=reset_sig_count@entry=0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:527
#6 0x0000556b2674ff76 in buf_dump_thread () at /home/dan/mariadb-server-10.3/storage/innobase/buf/buf0dump.cc:828
#7 0x00007f79c226d7fc in start_thread (arg=0x7f79937fe700) at pthread_create.c:465
#8 0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 14 (Thread 0x7f79a7fff700 (LWP 54490)):
#0 0x00007f79c2274786 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7f79a7ffeb50, expected=0, futex_word=0x556b29094d60) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1 __pthread_cond_wait_common (abstime=0x7f79a7ffeb50, mutex=0x556b29094d10, cond=0x556b29094d38) at pthread_cond_wait.c:539
#2 __pthread_cond_timedwait (cond=cond@entry=0x556b29094d38, mutex=mutex@entry=0x556b29094d10, abstime=abstime@entry=0x7f79a7ffeb50) at pthread_cond_wait.c:667
#3 0x0000556b26642211 in os_event::timed_wait (this=this@entry=0x556b29094d00, abstime=abstime@entry=0x7f79a7ffeb50) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:283
#4 0x0000556b26642431 in os_event::wait_time_low (this=0x556b29094d00, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=1) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:405
#5 0x0000556b266427f5 in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:510
#6 0x0000556b266c5133 in srv_error_monitor_thread () at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0srv.cc:1869
#7 0x00007f79c226d7fc in start_thread (arg=0x7f79a7fff700) at pthread_create.c:465
#8 0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 13 (Thread 0x7f79b3fff700 (LWP 54477)):
#0 0x00007f79c388963a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
#1 0x0000556b2663d769 in LinuxAIOHandler::collect (this=this@entry=0x7f79b3ffeaf0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:1913
#2 0x0000556b2663dc10 in LinuxAIOHandler::poll (this=this@entry=0x7f79b3ffeaf0, m1=m1@entry=0x7f79b3ffeb90, m2=m2@entry=0x7f79b3ffeb98, request=request@entry=0x7f79b3ffeba0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2058
#3 0x0000556b266400ab in os_aio_linux_handler (request=0x7f79b3ffeba0, m2=0x7f79b3ffeb98, m1=0x7f79b3ffeb90, global_segment=2) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2112
#4 os_aio_handler (segment=segment@entry=2, m1=m1@entry=0x7f79b3ffeb90, m2=m2@entry=0x7f79b3ffeb98, request=request@entry=0x7f79b3ffeba0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:5638
#5 0x0000556b2679d1e4 in fil_aio_wait (segment=segment@entry=2) at /home/dan/mariadb-server-10.3/storage/innobase/fil/fil0fil.cc:4678
#6 0x0000556b266c8720 in io_handler_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0start.cc:339
#7 0x00007f79c226d7fc in start_thread (arg=0x7f79b3fff700) at pthread_create.c:465
#8 0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 12 (Thread 0x7f79b17fa700 (LWP 54501)):
#0 0x00007f79c22785f8 in __GI___nanosleep (requested_time=requested_time@entry=0x7f79b17f9d10, remaining=remaining@entry=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:27
#1 0x0000556b266429cc in os_thread_sleep (tm=tm@entry=1000000) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0thread.cc:230
#2 0x0000556b266c6c9c in srv_master_sleep () at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0srv.cc:2358
#3 srv_master_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0srv.cc:2403
#4 0x00007f79c226d7fc in start_thread (arg=0x7f79b17fa700) at pthread_create.c:465
#5 0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 11 (Thread 0x7f79a77fe700 (LWP 54491)):
#0 0x00007f79c2274786 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7f79a77fdc80, expected=0, futex_word=0x556b29094de0) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1 __pthread_cond_wait_common (abstime=0x7f79a77fdc80, mutex=0x556b29094d90, cond=0x556b29094db8) at pthread_cond_wait.c:539
#2 __pthread_cond_timedwait (cond=cond@entry=0x556b29094db8, mutex=mutex@entry=0x556b29094d90, abstime=abstime@entry=0x7f79a77fdc80) at pthread_cond_wait.c:667
#3 0x0000556b26642211 in os_event::timed_wait (this=this@entry=0x556b29094d80, abstime=abstime@entry=0x7f79a77fdc80) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:283
#4 0x0000556b26642431 in os_event::wait_time_low (this=0x556b29094d80, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=1) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:405
#5 0x0000556b266427f5 in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:510
#6 0x0000556b266c6320 in srv_monitor_thread () at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0srv.cc:1705
#7 0x00007f79c226d7fc in start_thread (arg=0x7f79a77fe700) at pthread_create.c:465
#8 0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 10 (Thread 0x7f79b0ff9700 (LWP 54489)):
#0 0x00007f79c2274786 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7f79b0ff8bf0, expected=0, futex_word=0x556b292358b0) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1 __pthread_cond_wait_common (abstime=0x7f79b0ff8bf0, mutex=0x556b29235860, cond=0x556b29235888) at pthread_cond_wait.c:539
#2 __pthread_cond_timedwait (cond=cond@entry=0x556b29235888, mutex=mutex@entry=0x556b29235860, abstime=abstime@entry=0x7f79b0ff8bf0) at pthread_cond_wait.c:667
#3 0x0000556b26642211 in os_event::timed_wait (this=this@entry=0x556b29235850, abstime=abstime@entry=0x7f79b0ff8bf0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:283
#4 0x0000556b26642431 in os_event::wait_time_low (this=this@entry=0x556b29235850, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=1) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:405
#5 0x0000556b266427f5 in os_event_wait_time_low (event=event@entry=0x556b29235850, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:510
#6 0x0000556b2661e5e1 in lock_wait_timeout_thread () at /home/dan/mariadb-server-10.3/storage/innobase/lock/lock0wait.cc:547
#7 0x00007f79c226d7fc in start_thread (arg=0x7f79b0ff9700) at pthread_create.c:465
#8 0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 9 (Thread 0x7f79b37fe700 (LWP 54478)):
#0 0x00007f79c388963a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
#1 0x0000556b2663d769 in LinuxAIOHandler::collect (this=this@entry=0x7f79b37fdaf0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:1913
#2 0x0000556b2663dc10 in LinuxAIOHandler::poll (this=this@entry=0x7f79b37fdaf0, m1=m1@entry=0x7f79b37fdb90, m2=m2@entry=0x7f79b37fdb98, request=request@entry=0x7f79b37fdba0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2058
#3 0x0000556b266400ab in os_aio_linux_handler (request=0x7f79b37fdba0, m2=0x7f79b37fdb98, m1=0x7f79b37fdb90, global_segment=3) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2112
#4 os_aio_handler (segment=segment@entry=3, m1=m1@entry=0x7f79b37fdb90, m2=m2@entry=0x7f79b37fdb98, request=request@entry=0x7f79b37fdba0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:5638
#5 0x0000556b2679d1e4 in fil_aio_wait (segment=segment@entry=3) at /home/dan/mariadb-server-10.3/storage/innobase/fil/fil0fil.cc:4678
#6 0x0000556b266c8720 in io_handler_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0start.cc:339
#7 0x00007f79c226d7fc in start_thread (arg=0x7f79b37fe700) at pthread_create.c:465
#8 0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 8 (Thread 0x7f79b1ffb700 (LWP 54481)):
#0 0x00007f79c2274786 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7f79b1ffaa90, expected=0, futex_word=0x556b29094ee4) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1 __pthread_cond_wait_common (abstime=0x7f79b1ffaa90, mutex=0x556b29094e90, cond=0x556b29094eb8) at pthread_cond_wait.c:539
#2 __pthread_cond_timedwait (cond=cond@entry=0x556b29094eb8, mutex=mutex@entry=0x556b29094e90, abstime=abstime@entry=0x7f79b1ffaa90) at pthread_cond_wait.c:667
#3 0x0000556b26642211 in os_event::timed_wait (this=this@entry=0x556b29094e80, abstime=abstime@entry=0x7f79b1ffaa90) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:283
#4 0x0000556b26642431 in os_event::wait_time_low (this=0x556b29094e80, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=reset_sig_count@entry=2) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:405
#5 0x0000556b266427f5 in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=reset_sig_count@entry=2) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:510
#6 0x0000556b2675764b in pc_sleep_if_needed (cur_time=1523232365070, sig_count=2, next_loop_time=1523232366070) at /home/dan/mariadb-server-10.3/storage/innobase/buf/buf0flu.cc:2675
#7 buf_flush_page_cleaner_coordinator () at /home/dan/mariadb-server-10.3/storage/innobase/buf/buf0flu.cc:3140
#8 0x00007f79c226d7fc in start_thread (arg=0x7f79b1ffb700) at pthread_create.c:465
#9 0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 7 (Thread 0x7f79b2ffd700 (LWP 54479)):
#0 0x00007f79c388963a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
#1 0x0000556b2663d769 in LinuxAIOHandler::collect (this=this@entry=0x7f79b2ffcaf0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:1913
#2 0x0000556b2663dc10 in LinuxAIOHandler::poll (this=this@entry=0x7f79b2ffcaf0, m1=m1@entry=0x7f79b2ffcb90, m2=m2@entry=0x7f79b2ffcb98, request=request@entry=0x7f79b2ffcba0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2058
#3 0x0000556b266400ab in os_aio_linux_handler (request=0x7f79b2ffcba0, m2=0x7f79b2ffcb98, m1=0x7f79b2ffcb90, global_segment=4) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2112
#4 os_aio_handler (segment=segment@entry=4, m1=m1@entry=0x7f79b2ffcb90, m2=m2@entry=0x7f79b2ffcb98, request=request@entry=0x7f79b2ffcba0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:5638
#5 0x0000556b2679d1e4 in fil_aio_wait (segment=segment@entry=4) at /home/dan/mariadb-server-10.3/storage/innobase/fil/fil0fil.cc:4678
#6 0x0000556b266c8720 in io_handler_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0start.cc:339
#7 0x00007f79c226d7fc in start_thread (arg=0x7f79b2ffd700) at pthread_create.c:465
#8 0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 6 (Thread 0x7f79b27fc700 (LWP 54480)):
#0 0x00007f79c388963a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
#1 0x0000556b2663d769 in LinuxAIOHandler::collect (this=this@entry=0x7f79b27fbaf0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:1913
#2 0x0000556b2663dc10 in LinuxAIOHandler::poll (this=this@entry=0x7f79b27fbaf0, m1=m1@entry=0x7f79b27fbb90, m2=m2@entry=0x7f79b27fbb98, request=request@entry=0x7f79b27fbba0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2058
#3 0x0000556b266400ab in os_aio_linux_handler (request=0x7f79b27fbba0, m2=0x7f79b27fbb98, m1=0x7f79b27fbb90, global_segment=5) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2112
#4 os_aio_handler (segment=segment@entry=5, m1=m1@entry=0x7f79b27fbb90, m2=m2@entry=0x7f79b27fbb98, request=request@entry=0x7f79b27fbba0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:5638
#5 0x0000556b2679d1e4 in fil_aio_wait (segment=segment@entry=5) at /home/dan/mariadb-server-10.3/storage/innobase/fil/fil0fil.cc:4678
#6 0x0000556b266c8720 in io_handler_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0start.cc:339
#7 0x00007f79c226d7fc in start_thread (arg=0x7f79b27fc700) at pthread_create.c:465
#8 0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 5 (Thread 0x7f79b97ac700 (LWP 54475)):
#0 0x00007f79c388963a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
#1 0x0000556b2663d769 in LinuxAIOHandler::collect (this=this@entry=0x7f79b97abaf0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:1913
#2 0x0000556b2663dc10 in LinuxAIOHandler::poll (this=this@entry=0x7f79b97abaf0, m1=m1@entry=0x7f79b97abb90, m2=m2@entry=0x7f79b97abb98, request=request@entry=0x7f79b97abba0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2058
#3 0x0000556b266400ab in os_aio_linux_handler (request=0x7f79b97abba0, m2=0x7f79b97abb98, m1=0x7f79b97abb90, global_segment=0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2112
#4 os_aio_handler (segment=segment@entry=0, m1=m1@entry=0x7f79b97abb90, m2=m2@entry=0x7f79b97abb98, request=request@entry=0x7f79b97abba0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:5638
#5 0x0000556b2679d1e4 in fil_aio_wait (segment=segment@entry=0) at /home/dan/mariadb-server-10.3/storage/innobase/fil/fil0fil.cc:4678
#6 0x0000556b266c8720 in io_handler_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0start.cc:339
#7 0x00007f79c226d7fc in start_thread (arg=0x7f79b97ac700) at pthread_create.c:465
#8 0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 4 (Thread 0x7f79b8fab700 (LWP 54476)):
#0 0x00007f79c388963a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
#1 0x0000556b2663d769 in LinuxAIOHandler::collect (this=this@entry=0x7f79b8faaaf0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:1913
#2 0x0000556b2663dc10 in LinuxAIOHandler::poll (this=this@entry=0x7f79b8faaaf0, m1=m1@entry=0x7f79b8faab90, m2=m2@entry=0x7f79b8faab98, request=request@entry=0x7f79b8faaba0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2058
#3 0x0000556b266400ab in os_aio_linux_handler (request=0x7f79b8faaba0, m2=0x7f79b8faab98, m1=0x7f79b8faab90, global_segment=1) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2112
#4 os_aio_handler (segment=segment@entry=1, m1=m1@entry=0x7f79b8faab90, m2=m2@entry=0x7f79b8faab98, request=request@entry=0x7f79b8faaba0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:5638
#5 0x0000556b2679d1e4 in fil_aio_wait (segment=segment@entry=1) at /home/dan/mariadb-server-10.3/storage/innobase/fil/fil0fil.cc:4678
#6 0x0000556b266c8720 in io_handler_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0start.cc:339
#7 0x00007f79c226d7fc in start_thread (arg=0x7f79b8fab700) at pthread_create.c:465
#8 0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 3 (Thread 0x7f79bb063700 (LWP 54470)):
#0 0x00007f79c2274786 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7f79bb062c70, expected=0, futex_word=0x556b27abeaa8 <COND_checkpoint+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1 __pthread_cond_wait_common (abstime=0x7f79bb062c70, mutex=0x556b27abeac0 <LOCK_checkpoint>, cond=0x556b27abea80 <COND_checkpoint>) at pthread_cond_wait.c:539
#2 __pthread_cond_timedwait (cond=cond@entry=0x556b27abea80 <COND_checkpoint>, mutex=mutex@entry=0x556b27abeac0 <LOCK_checkpoint>, abstime=abstime@entry=0x7f79bb062c70) at pthread_cond_wait.c:667
#3 0x0000556b268303ff in inline_mysql_cond_timedwait (src_file=0x556b26aad278 "/home/dan/mariadb-server-10.3/storage/maria/ma_servicethread.c", src_line=116, abstime=0x7f79bb062c70, mutex=0x556b27abeac0 <LOCK_checkpoint>, that=0x556b27abea80 <COND_checkpoint>) at /home/dan/mariadb-server-10.3/include/mysql/psi/mysql_thread.h:1204
#4 my_service_thread_sleep (control=control@entry=0x556b271c0900 <checkpoint_control>, sleep_time=sleep_time@entry=30000000000) at /home/dan/mariadb-server-10.3/storage/maria/ma_servicethread.c:115
#5 0x0000556b26829c9d in ma_checkpoint_background (arg=arg@entry=0x1e) at /home/dan/mariadb-server-10.3/storage/maria/ma_checkpoint.c:707
#6 0x0000556b268e0e7e in pfs_spawn_thread (arg=0x556b28f8fa88) at /home/dan/mariadb-server-10.3/storage/perfschema/pfs.cc:1862
#7 0x00007f79c226d7fc in start_thread (arg=0x7f79bb063700) at pthread_create.c:465
#8 0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 2 (Thread 0x7f79bc5d4700 (LWP 54463)):
#0 0x00007f79c2274786 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7f79bc5d3ce0, expected=0, futex_word=0x556b27ace548 <COND_timer+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1 __pthread_cond_wait_common (abstime=0x7f79bc5d3ce0, mutex=0x556b27ace560 <LOCK_timer>, cond=0x556b27ace520 <COND_timer>) at pthread_cond_wait.c:539
#2 __pthread_cond_timedwait (cond=cond@entry=0x556b27ace520 <COND_timer>, mutex=mutex@entry=0x556b27ace560 <LOCK_timer>, abstime=abstime@entry=0x7f79bc5d3ce0) at pthread_cond_wait.c:667
#3 0x0000556b269219cf in inline_mysql_cond_timedwait (that=0x556b27ace520 <COND_timer>, mutex=0x556b27ace560 <LOCK_timer>, src_file=0x556b26ac1768 "/home/dan/mariadb-server-10.3/mysys/thr_timer.c", src_line=292, abstime=0x7f79bc5d3ce0) at /home/dan/mariadb-server-10.3/include/mysql/psi/mysql_thread.h:1204
#4 timer_handler (arg=arg@entry=0x0) at /home/dan/mariadb-server-10.3/mysys/thr_timer.c:292
#5 0x0000556b268e0e7e in pfs_spawn_thread (arg=0x556b28f7bb18) at /home/dan/mariadb-server-10.3/storage/perfschema/pfs.cc:1862
#6 0x00007f79c226d7fc in start_thread (arg=0x7f79bc5d4700) at pthread_create.c:465
#7 0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 1 (Thread 0x7f79c3ea5d40 (LWP 54253)):
#0 __pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:57
#1 0x0000556b2691d7ea in my_write_core (sig=sig@entry=6) at /home/dan/mariadb-server-10.3/mysys/stacktrace.c:481
#2 0x0000556b2647183a in handle_fatal_signal (sig=6) at /home/dan/mariadb-server-10.3/sql/signal_handler.cc:305
#3 <signal handler called>
#4 0x00007f79c2274072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x556b2723f0c8 <COND_thread_count+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#5 __pthread_cond_wait_common (abstime=0x0, mutex=0x556b2723f820 <LOCK_thread_count>, cond=0x556b2723f0a0 <COND_thread_count>) at pthread_cond_wait.c:502
#6 __pthread_cond_wait (cond=cond@entry=0x556b2723f0a0 <COND_thread_count>, mutex=mutex@entry=0x556b2723f820 <LOCK_thread_count>) at pthread_cond_wait.c:655
#7 0x0000556b26215b5a in inline_mysql_cond_wait (src_file=0x556b26961008 "/home/dan/mariadb-server-10.3/sql/mysqld.cc", src_line=6179, mutex=0x556b2723f820 <LOCK_thread_count>, that=0x556b2723f0a0 <COND_thread_count>) at /home/dan/mariadb-server-10.3/include/mysql/psi/mysql_thread.h:1167
#8 mysqld_main (argc=<optimized out>, argv=<optimized out>) at /home/dan/mariadb-server-10.3/sql/mysqld.cc:6179
#9 0x00007f79c15b41c1 in __libc_start_main (main=0x556b261f3eb0 <main(int, char**)>, argc=28, argv=0x7ffce77e2ef8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffce77e2ee8) at ../csu/libc-start.c:308
#10 0x0000556b262086aa in _start ()

Compressed file /home/dan/build-mariadb-10.3_lastest_upstream_gcc/mysql-test/var/log/encryption.innodb-bad-key-change2-innodb/mysqld.1/data/core

Attachments

Issue Links

relates to

MDEV-13779 InnoDB fails to shut down purge workers, causing hang

Closed

MDEV-14379 encryption.innodb_encrypt_log_corruption failed in buildbot, crash

Closed

Activity

People

Assignee:: Sergei Golubchik

Reporter:: Daniel Black

Votes:: 0 Vote for this issue

Watchers:: 5 Start watching this issue

Dates

Created:: 2018-04-09 00:27

Updated:: 2023-04-27 14:19

Git Integration

Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.