Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-15817

encryption tests fail futex(COND_thread_count/LOCK_thread_count)

Details

    • Bug
    • Status: Open (View Workflow)
    • Major
    • Resolution: Unresolved
    • 10.3(EOL)
    • 10.4(EOL)
    • Encryption, Server, Tests
    • x86_64 - ubuntu artful, clang-7, clang-6.0, gcc-7.2.0

    Description

      clang-7

      dpkg -l | grep clang-7
      1:7~svn329460-1~exp1+0~20180406214534.323~1.gbp77c2bd
       
       cmake ../mariadb-server/ -DCMAKE_CXX_COMPILER=clang++-7 -DCMAKE_C_COMPILER=clang-7 -DCMAKE_CFLAGS="-O3 -mcpu=skylake" -DCMAKE_CXXFLAGS="-O3 -mcpu=skylake"
       
      CURRENT_TEST: binlog_encryption.encrypted_master_switch_to_unencrypted
      --- /home/dan/mariadb-server-10.3/mysql-test/suite/binlog_encryption/encrypted_master_switch_to_unencrypted.result      2018-04-06 01:48:31.955687603 -0500
      +++ /home/dan/mariadb-server-10.3/mysql-test/suite/binlog_encryption/encrypted_master_switch_to_unencrypted.reject      2018-04-08 18:23:26.526381135 -0500
      @@ -72,6 +72,10 @@
       SELECT COUNT(*) FROM table1_no_encryption;
       COUNT(*)
       8
      +Warnings:
      +Error  145     Table './test/table1_no_encryption' is marked as crashed and should be repaired
      +Error  1194    Table 'table1_no_encryption' is marked as crashed and should be repaired
      +Error  1034    1 client is using or hasn't closed the table properly
       SELECT COUNT(*) FROM table2_to_encrypt;
       COUNT(*)
       8
       
      mysqltest: Result length mismatch
       
       - saving '/home/dan/build-mariadb-10.3_lastest_upstream/mysql-test/var/3/log/binlog_encryption.encrypted_master_switch_to_unencrypted-mix/' to '/home/dan/build-mariadb-10.3_lastest_upstream/mysql-test/var/log/binlog_encryption.encrypted_master_switch_to_unencrypted-mix/'
       - found 'core' (0/5)
       
      Trying 'dbx' to get a backtrace
       
      Trying 'gdb' to get a backtrace from coredump /home/dan/build-mariadb-10.3_lastest_upstream/mysql-test/var/log/binlog_encryption.encrypted_master_switch_to_unencrypted-mix/mysqld.1/data/core
      Core generated by '/home/dan/build-mariadb-10.3_lastest_upstream/sql/mysqld'
      Output from gdb follows. The first stack trace is from the failing thread.
      The following stack traces are from all threads (so the failing one is
      duplicated).
      --------------------------
      [New LWP 149092]
      [New LWP 149598]
      [New LWP 149888]
      [New LWP 149440]
      [New LWP 149608]
      [New LWP 149604]
      [New LWP 149597]
      [New LWP 149990]
      [New LWP 149988]
      [New LWP 150285]
      [New LWP 149599]
      [New LWP 149602]
      [New LWP 150374]
      [New LWP 153123]
      [New LWP 149889]
      [New LWP 150284]
      [New LWP 150282]
      [New LWP 149606]
      [New LWP 149564]
      [New LWP 149890]
      [New LWP 149982]
      [New LWP 149986]
      [Thread debugging using libthread_db enabled]
      Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
      Core was generated by `/home/dan/build-mariadb-10.3_lastest_upstream/sql/mysqld --defaults-group-suffi'.
      Program terminated with signal SIGABRT, Aborted.
      #0  __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:57
      57      ../sysdeps/unix/sysv/linux/pthread_kill.c: No such file or directory.
      [Current thread is 1 (Thread 0x7fa57a84e8c0 (LWP 149092))]
      #0  __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:57
      #1  0x00000000007f5f08 in handle_fatal_signal (sig=6) at /home/dan/mariadb-server-10.3/sql/signal_handler.cc:305
      #2  <signal handler called>
      #3  0x00007fa57a42c072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x1572760 <COND_thread_count+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
      #4  __pthread_cond_wait_common (abstime=0x0, mutex=0x1572158 <LOCK_thread_count>, cond=0x1572738 <COND_thread_count>) at pthread_cond_wait.c:502
      #5  __pthread_cond_wait (cond=0x1572738 <COND_thread_count>, mutex=0x1572158 <LOCK_thread_count>) at pthread_cond_wait.c:655
      #6  0x000000000057f5a3 in inline_mysql_cond_wait (mutex=<optimized out>, src_line=6179, that=<optimized out>, src_file=<optimized out>) at /home/dan/mariadb-server-10.3/include/mysql/psi/mysql_thread.h:1167
      #7  mysqld_main (argc=<optimized out>, argv=<optimized out>) at /home/dan/mariadb-server-10.3/sql/mysqld.cc:6179
      #8  0x00007fa577f5e1c1 in __libc_start_main (main=0x57a9e0 <main(int, char**)>, argc=9, argv=0x7ffc0841a158, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffc0841a148) at ../csu/libc-start.c:308
      #9  0x000000000057a92a in _start ()
       
      Thread 22 (Thread 0x7fa557fff700 (LWP 149986)):
      #0  0x00007fa57a42c072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x3b6aa70) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
      #1  __pthread_cond_wait_common (abstime=0x0, mutex=0x3b6aa20, cond=0x3b6aa48) at pthread_cond_wait.c:502
      #2  __pthread_cond_wait (cond=0x3b6aa48, mutex=0x3b6aa20) at pthread_cond_wait.c:655
      #3  0x00000000009ce936 in os_event::wait (this=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:163
      #4  os_event::wait_low (this=0x3b6aa10, reset_sig_count=9) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:333
      #5  0x0000000000a46e74 in srv_resume_thread (slot=<optimized out>, sig_count=0, wait=255, timeout_usec=0) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0srv.cc:906
      #6  srv_worker_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0srv.cc:2534
      #7  0x00007fa57a4257fc in start_thread (arg=0x7fa557fff700) at pthread_create.c:465
      #8  0x00007fa578051b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 21 (Thread 0x7fa563fff700 (LWP 149982)):
      #0  0x00007fa57a4305f8 in __GI___nanosleep (requested_time=0x7fa563ffea48, remaining=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:27
      #1  0x00000000009cef3f in os_thread_sleep (tm=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0thread.cc:230
      #2  0x0000000000a460b5 in srv_master_sleep () at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0srv.cc:2358
      #3  srv_master_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0srv.cc:2403
      #4  0x00007fa57a4257fc in start_thread (arg=0x7fa563fff700) at pthread_create.c:465
      #5  0x00007fa578051b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 20 (Thread 0x7fa5627fc700 (LWP 149890)):
      #0  0x00007fa57a42c786 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7fa5627fbcd0, expected=0, futex_word=0x3b6abf0) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
      #1  __pthread_cond_wait_common (abstime=0x7fa5627fbcd0, mutex=0x3b6aba0, cond=0x3b6abc8) at pthread_cond_wait.c:539
      #2  __pthread_cond_timedwait (cond=0x3b6abc8, mutex=0x3b6aba0, abstime=0x7fa5627fbcd0) at pthread_cond_wait.c:667
      #3  0x00000000009ce802 in os_event::timed_wait (this=<optimized out>, abstime=0x7fa5627fbcd0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:283
      #4  0x00000000009cea66 in os_event::wait_time_low (this=0x3b6ab90, time_in_usec=<optimized out>, reset_sig_count=1) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:405
      #5  0x0000000000a45590 in srv_monitor_thread () at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0srv.cc:1705
      #6  0x00007fa57a4257fc in start_thread (arg=0x7fa5627fc700) at pthread_create.c:465
      #7  0x00007fa578051b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 19 (Thread 0x7fa571a0d700 (LWP 149564)):
      #0  0x00007fa57a42c786 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7fa571a0cc90, expected=0, futex_word=0x1dee9e8 <COND_checkpoint+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
      #1  __pthread_cond_wait_common (abstime=0x7fa571a0cc90, mutex=0x1dee988 <LOCK_checkpoint>, cond=0x1dee9c0 <COND_checkpoint>) at pthread_cond_wait.c:539
      #2  __pthread_cond_timedwait (cond=0x1dee9c0 <COND_checkpoint>, mutex=0x1dee988 <LOCK_checkpoint>, abstime=0x7fa571a0cc90) at pthread_cond_wait.c:667
      #3  0x0000000000b86eff in inline_mysql_cond_timedwait (that=0x1dee9c0 <COND_checkpoint>, mutex=0x1dee988 <LOCK_checkpoint>, src_line=116, abstime=<optimized out>, src_file=<optimized out>) at /home/dan/mariadb-server-10.3/include/mysql/psi/mysql_thread.h:1204
      #4  my_service_thread_sleep (control=0x14f4a80 <checkpoint_control>, sleep_time=<optimized out>) at /home/dan/mariadb-server-10.3/storage/maria/ma_servicethread.c:115
      #5  0x0000000000b804ee in ma_checkpoint_background (arg=0x1e) at /home/dan/mariadb-server-10.3/storage/maria/ma_checkpoint.c:707
      #6  0x0000000000c2c5e1 in pfs_spawn_thread (arg=0x3a659b8) at /home/dan/mariadb-server-10.3/storage/perfschema/pfs.cc:1862
      #7  0x00007fa57a4257fc in start_thread (arg=0x7fa571a0d700) at pthread_create.c:465
      #8  0x00007fa578051b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 18 (Thread 0x7fa5697fa700 (LWP 149606)):
      #0  0x00007fa57a01463a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
      #1  0x00000000009c6dad in LinuxAIOHandler::collect (this=0x7fa5697f9a90) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:1913
      #2  0x00000000009c71b6 in LinuxAIOHandler::poll (this=0x7fa5697f9a90, m1=0x7fa5697f9b78, m2=0x7fa5697f9b80, request=0x7fa5697f9d30) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2058
      #3  0x00000000009ca8d3 in os_aio_linux_handler (global_segment=<optimized out>, m1=<optimized out>, m2=<optimized out>, request=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2112
      #4  os_aio_handler (segment=<optimized out>, m1=0x7fa5697f9b78, m2=0x7fa5697f9b80, request=0x7fa5697f9d30) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:5638
      #5  0x0000000000b0ac5d in fil_aio_wait (segment=5) at /home/dan/mariadb-server-10.3/storage/innobase/fil/fil0fil.cc:4678
      #6  0x0000000000a48548 in io_handler_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0start.cc:339
      #7  0x00007fa57a4257fc in start_thread (arg=0x7fa5697fa700) at pthread_create.c:465
      #8  0x00007fa578051b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 17 (Thread 0x7fa5567fc700 (LWP 150282)):
      #0  0x00007fa57a42c072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x3b6ac70) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
      #1  __pthread_cond_wait_common (abstime=0x0, mutex=0x3b6ac20, cond=0x3b6ac48) at pthread_cond_wait.c:502
      #2  __pthread_cond_wait (cond=0x3b6ac48, mutex=0x3b6ac20) at pthread_cond_wait.c:655
      #3  0x00000000009ce936 in os_event::wait (this=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:163
      #4  os_event::wait_low (this=0x3b6ac10, reset_sig_count=1) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:333
      #5  0x0000000000abc227 in buf_dump_thread () at /home/dan/mariadb-server-10.3/storage/innobase/buf/buf0dump.cc:828
      #6  0x00007fa57a4257fc in start_thread (arg=0x7fa5567fc700) at pthread_create.c:465
      #7  0x00007fa578051b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 16 (Thread 0x7fa555ffb700 (LWP 150284)):
      #0  0x00007fa57a4305f8 in __GI___nanosleep (requested_time=0x7fa555ffa548, remaining=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:27
      #1  0x00000000009cef3f in os_thread_sleep (tm=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0thread.cc:230
      #2  0x0000000000aa665a in btr_defragment_thread () at /home/dan/mariadb-server-10.3/storage/innobase/btr/btr0defragment.cc:756
      #3  0x00007fa57a4257fc in start_thread (arg=0x7fa555ffb700) at pthread_create.c:465
      #4  0x00007fa578051b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 15 (Thread 0x7fa562ffd700 (LWP 149889)):
      #0  0x00007fa57a42c786 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7fa562ffca00, expected=0, futex_word=0x3b6ab70) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
      #1  __pthread_cond_wait_common (abstime=0x7fa562ffca00, mutex=0x3b6ab20, cond=0x3b6ab48) at pthread_cond_wait.c:539
      #2  __pthread_cond_timedwait (cond=0x3b6ab48, mutex=0x3b6ab20, abstime=0x7fa562ffca00) at pthread_cond_wait.c:667
      #3  0x00000000009ce802 in os_event::timed_wait (this=<optimized out>, abstime=0x7fa562ffca00) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:283
      #4  0x00000000009cea66 in os_event::wait_time_low (this=0x3b6ab10, time_in_usec=<optimized out>, reset_sig_count=1) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:405
      #5  0x0000000000a45c2f in srv_error_monitor_thread () at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0srv.cc:1869
      #6  0x00007fa57a4257fc in start_thread (arg=0x7fa562ffd700) at pthread_create.c:465
      #7  0x00007fa578051b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 14 (Thread 0x7fa570150700 (LWP 153123)):
      #0  0x00007fa5780478c0 in __GI___select (nfds=0, readfds=0x0, writefds=0x0, exceptfds=0x0, timeout=0x7fa57014fca8) at ../sysdeps/unix/sysv/linux/select.c:41
      #1  0x0000000000c68abe in my_sleep (m_seconds=<optimized out>) at /home/dan/mariadb-server-10.3/mysys/my_sleep.c:29
      #2  0x000000000057b8ca in close_connections () at /home/dan/mariadb-server-10.3/sql/mysqld.cc:1785
      #3  kill_server (sig_ptr=<optimized out>) at /home/dan/mariadb-server-10.3/sql/mysqld.cc:2033
      #4  0x000000000057b00b in kill_server_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/sql/mysqld.cc:2065
      #5  0x0000000000c2c5e1 in pfs_spawn_thread (arg=0x3d8f058) at /home/dan/mariadb-server-10.3/storage/perfschema/pfs.cc:1862
      #6  0x00007fa57a4257fc in start_thread (arg=0x7fa570150700) at pthread_create.c:465
      #7  0x00007fa578051b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 13 (Thread 0x7fa57030c700 (LWP 150374)):
      #0  do_sigwait (sig=0x7fa57030bc60, set=<optimized out>) at ../sysdeps/unix/sysv/linux/sigwait.c:64
      #1  __sigwait (set=0x7fa57030bcc8, sig=0x7fa57030bc60) at ../sysdeps/unix/sysv/linux/sigwait.c:96
      #2  0x000000000057dfbb in signal_hand (arg=<optimized out>) at /home/dan/mariadb-server-10.3/sql/mysqld.cc:3550
      #3  0x0000000000c2c5e1 in pfs_spawn_thread (arg=0x3d8f058) at /home/dan/mariadb-server-10.3/storage/perfschema/pfs.cc:1862
      #4  0x00007fa57a4257fc in start_thread (arg=0x7fa57030c700) at pthread_create.c:465
      #5  0x00007fa578051b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 12 (Thread 0x7fa56a7fc700 (LWP 149602)):
      #0  0x00007fa57a01463a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
      #1  0x00000000009c6dad in LinuxAIOHandler::collect (this=0x7fa56a7fba90) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:1913
      #2  0x00000000009c71b6 in LinuxAIOHandler::poll (this=0x7fa56a7fba90, m1=0x7fa56a7fbb78, m2=0x7fa56a7fbb80, request=0x7fa56a7fbd30) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2058
      #3  0x00000000009ca8d3 in os_aio_linux_handler (global_segment=<optimized out>, m1=<optimized out>, m2=<optimized out>, request=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2112
      #4  os_aio_handler (segment=<optimized out>, m1=0x7fa56a7fbb78, m2=0x7fa56a7fbb80, request=0x7fa56a7fbd30) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:5638
      #5  0x0000000000b0ac5d in fil_aio_wait (segment=3) at /home/dan/mariadb-server-10.3/storage/innobase/fil/fil0fil.cc:4678
      #6  0x0000000000a48548 in io_handler_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0start.cc:339
      #7  0x00007fa57a4257fc in start_thread (arg=0x7fa56a7fc700) at pthread_create.c:465
      #8  0x00007fa578051b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 11 (Thread 0x7fa56affd700 (LWP 149599)):
      #0  0x00007fa57a01463a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
      #1  0x00000000009c6dad in LinuxAIOHandler::collect (this=0x7fa56affca90) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:1913
      #2  0x00000000009c71b6 in LinuxAIOHandler::poll (this=0x7fa56affca90, m1=0x7fa56affcb78, m2=0x7fa56affcb80, request=0x7fa56affcd30) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2058
      #3  0x00000000009ca8d3 in os_aio_linux_handler (global_segment=<optimized out>, m1=<optimized out>, m2=<optimized out>, request=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2112
      #4  os_aio_handler (segment=<optimized out>, m1=0x7fa56affcb78, m2=0x7fa56affcb80, request=0x7fa56affcd30) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:5638
      #5  0x0000000000b0ac5d in fil_aio_wait (segment=2) at /home/dan/mariadb-server-10.3/storage/innobase/fil/fil0fil.cc:4678
      #6  0x0000000000a48548 in io_handler_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0start.cc:339
      #7  0x00007fa57a4257fc in start_thread (arg=0x7fa56affd700) at pthread_create.c:465
      #8  0x00007fa578051b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 10 (Thread 0x7fa5557fa700 (LWP 150285)):
      #0  0x00007fa57a42c072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x3b6ad70) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
      #1  __pthread_cond_wait_common (abstime=0x0, mutex=0x3b6ad20, cond=0x3b6ad48) at pthread_cond_wait.c:502
      #2  __pthread_cond_wait (cond=0x3b6ad48, mutex=0x3b6ad20) at pthread_cond_wait.c:655
      #3  0x00000000009ce936 in os_event::wait (this=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:163
      #4  os_event::wait_low (this=0x3b6ad10, reset_sig_count=1) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:333
      #5  0x0000000000aa9f5e in buf_resize_thread () at /home/dan/mariadb-server-10.3/storage/innobase/buf/buf0buf.cc:3125
      #6  0x00007fa57a4257fc in start_thread (arg=0x7fa5557fa700) at pthread_create.c:465
      #7  0x00007fa578051b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 9 (Thread 0x7fa5577fe700 (LWP 149988)):
      #0  0x00007fa57a42c072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x3b6aaf4) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
      #1  __pthread_cond_wait_common (abstime=0x0, mutex=0x3b6aaa0, cond=0x3b6aac8) at pthread_cond_wait.c:502
      #2  __pthread_cond_wait (cond=0x3b6aac8, mutex=0x3b6aaa0) at pthread_cond_wait.c:655
      #3  0x00000000009ce936 in os_event::wait (this=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:163
      #4  os_event::wait_low (this=0x3b6aa90, reset_sig_count=2) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:333
      #5  0x0000000000a46e74 in srv_resume_thread (slot=<optimized out>, sig_count=0, wait=255, timeout_usec=0) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0srv.cc:906
      #6  srv_worker_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0srv.cc:2534
      #7  0x00007fa57a4257fc in start_thread (arg=0x7fa5577fe700) at pthread_create.c:465
      #8  0x00007fa578051b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 8 (Thread 0x7fa556ffd700 (LWP 149990)):
      #0  0x00007fa57a42c072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x3b6a9f0) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
      #1  __pthread_cond_wait_common (abstime=0x0, mutex=0x3b6a9a0, cond=0x3b6a9c8) at pthread_cond_wait.c:502
      #2  __pthread_cond_wait (cond=0x3b6a9c8, mutex=0x3b6a9a0) at pthread_cond_wait.c:655
      #3  0x00000000009ce936 in os_event::wait (this=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:163
      #4  os_event::wait_low (this=0x3b6a990, reset_sig_count=7) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:333
      #5  0x0000000000a46e74 in srv_resume_thread (slot=<optimized out>, sig_count=0, wait=255, timeout_usec=0) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0srv.cc:906
      #6  srv_worker_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0srv.cc:2534
      #7  0x00007fa57a4257fc in start_thread (arg=0x7fa556ffd700) at pthread_create.c:465
      #8  0x00007fa578051b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 7 (Thread 0x7fa56bfff700 (LWP 149597)):
      #0  0x00007fa57a01463a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
      #1  0x00000000009c6dad in LinuxAIOHandler::collect (this=0x7fa56bffea90) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:1913
      #2  0x00000000009c71b6 in LinuxAIOHandler::poll (this=0x7fa56bffea90, m1=0x7fa56bffeb78, m2=0x7fa56bffeb80, request=0x7fa56bffed30) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2058
      #3  0x00000000009ca8d3 in os_aio_linux_handler (global_segment=<optimized out>, m1=<optimized out>, m2=<optimized out>, request=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2112
      #4  os_aio_handler (segment=<optimized out>, m1=0x7fa56bffeb78, m2=0x7fa56bffeb80, request=0x7fa56bffed30) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:5638
      #5  0x0000000000b0ac5d in fil_aio_wait (segment=0) at /home/dan/mariadb-server-10.3/storage/innobase/fil/fil0fil.cc:4678
      #6  0x0000000000a48548 in io_handler_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0start.cc:339
      #7  0x00007fa57a4257fc in start_thread (arg=0x7fa56bfff700) at pthread_create.c:465
      #8  0x00007fa578051b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 6 (Thread 0x7fa569ffb700 (LWP 149604)):
      #0  0x00007fa57a01463a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
      #1  0x00000000009c6dad in LinuxAIOHandler::collect (this=0x7fa569ffaa90) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:1913
      #2  0x00000000009c71b6 in LinuxAIOHandler::poll (this=0x7fa569ffaa90, m1=0x7fa569ffab78, m2=0x7fa569ffab80, request=0x7fa569ffad30) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2058
      #3  0x00000000009ca8d3 in os_aio_linux_handler (global_segment=<optimized out>, m1=<optimized out>, m2=<optimized out>, request=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2112
      #4  os_aio_handler (segment=<optimized out>, m1=0x7fa569ffab78, m2=0x7fa569ffab80, request=0x7fa569ffad30) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:5638
      #5  0x0000000000b0ac5d in fil_aio_wait (segment=4) at /home/dan/mariadb-server-10.3/storage/innobase/fil/fil0fil.cc:4678
      #6  0x0000000000a48548 in io_handler_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0start.cc:339
      #7  0x00007fa57a4257fc in start_thread (arg=0x7fa569ffb700) at pthread_create.c:465
      #8  0x00007fa578051b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 5 (Thread 0x7fa568ff9700 (LWP 149608)):
      #0  0x00007fa57a42c786 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7fa568ff87d0, expected=0, futex_word=0x3b6acf4) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
      #1  __pthread_cond_wait_common (abstime=0x7fa568ff87d0, mutex=0x3b6aca0, cond=0x3b6acc8) at pthread_cond_wait.c:539
      #2  __pthread_cond_timedwait (cond=0x3b6acc8, mutex=0x3b6aca0, abstime=0x7fa568ff87d0) at pthread_cond_wait.c:667
      #3  0x00000000009ce802 in os_event::timed_wait (this=<optimized out>, abstime=0x7fa568ff87d0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:283
      #4  0x00000000009cea66 in os_event::wait_time_low (this=0x3b6ac90, time_in_usec=<optimized out>, reset_sig_count=2) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:405
      #5  0x0000000000ac10ae in pc_sleep_if_needed (next_loop_time=<optimized out>, sig_count=2, cur_time=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/buf/buf0flu.cc:2674
      #6  buf_flush_page_cleaner_coordinator () at /home/dan/mariadb-server-10.3/storage/innobase/buf/buf0flu.cc:3140
      #7  0x00007fa57a4257fc in start_thread (arg=0x7fa568ff9700) at pthread_create.c:465
      #8  0x00007fa578051b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 4 (Thread 0x7fa572f7e700 (LWP 149440)):
      #0  0x00007fa57a42c786 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7fa572f7dcf0, expected=0, futex_word=0x1dfe098 <COND_timer+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
      #1  __pthread_cond_wait_common (abstime=0x7fa572f7dcf0, mutex=0x1dfe040 <LOCK_timer>, cond=0x1dfe070 <COND_timer>) at pthread_cond_wait.c:539
      #2  __pthread_cond_timedwait (cond=0x1dfe070 <COND_timer>, mutex=0x1dfe040 <LOCK_timer>, abstime=0x7fa572f7dcf0) at pthread_cond_wait.c:667
      #3  0x0000000000c6f460 in inline_mysql_cond_timedwait (mutex=<optimized out>, src_line=292, that=<optimized out>, abstime=<optimized out>, src_file=<optimized out>) at /home/dan/mariadb-server-10.3/include/mysql/psi/mysql_thread.h:1204
      #4  timer_handler (arg=<optimized out>) at /home/dan/mariadb-server-10.3/mysys/thr_timer.c:292
      #5  0x0000000000c2c5e1 in pfs_spawn_thread (arg=0x3a4fae8) at /home/dan/mariadb-server-10.3/storage/perfschema/pfs.cc:1862
      #6  0x00007fa57a4257fc in start_thread (arg=0x7fa572f7e700) at pthread_create.c:465
      #7  0x00007fa578051b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 3 (Thread 0x7fa5637fe700 (LWP 149888)):
      #0  0x00007fa57a42c786 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7fa5637fdcc0, expected=0, futex_word=0x3d0b7a0) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
      #1  __pthread_cond_wait_common (abstime=0x7fa5637fdcc0, mutex=0x3d0b750, cond=0x3d0b778) at pthread_cond_wait.c:539
      #2  __pthread_cond_timedwait (cond=0x3d0b778, mutex=0x3d0b750, abstime=0x7fa5637fdcc0) at pthread_cond_wait.c:667
      #3  0x00000000009ce802 in os_event::timed_wait (this=<optimized out>, abstime=0x7fa5637fdcc0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:283
      #4  0x00000000009cea66 in os_event::wait_time_low (this=0x3d0b740, time_in_usec=<optimized out>, reset_sig_count=1) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:405
      #5  0x00000000009b30f0 in lock_wait_timeout_thread () at /home/dan/mariadb-server-10.3/storage/innobase/lock/lock0wait.cc:547
      #6  0x00007fa57a4257fc in start_thread (arg=0x7fa5637fe700) at pthread_create.c:465
      #7  0x00007fa578051b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 2 (Thread 0x7fa56b7fe700 (LWP 149598)):
      #0  0x00007fa57a01463a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
      #1  0x00000000009c6dad in LinuxAIOHandler::collect (this=0x7fa56b7fda90) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:1913
      #2  0x00000000009c71b6 in LinuxAIOHandler::poll (this=0x7fa56b7fda90, m1=0x7fa56b7fdb78, m2=0x7fa56b7fdb80, request=0x7fa56b7fdd30) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2058
      #3  0x00000000009ca8d3 in os_aio_linux_handler (global_segment=<optimized out>, m1=<optimized out>, m2=<optimized out>, request=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2112
      #4  os_aio_handler (segment=<optimized out>, m1=0x7fa56b7fdb78, m2=0x7fa56b7fdb80, request=0x7fa56b7fdd30) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:5638
      #5  0x0000000000b0ac5d in fil_aio_wait (segment=1) at /home/dan/mariadb-server-10.3/storage/innobase/fil/fil0fil.cc:4678
      #6  0x0000000000a48548 in io_handler_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0start.cc:339
      #7  0x00007fa57a4257fc in start_thread (arg=0x7fa56b7fe700) at pthread_create.c:465
      #8  0x00007fa578051b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 1 (Thread 0x7fa57a84e8c0 (LWP 149092)):
      #0  __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:57
      #1  0x00000000007f5f08 in handle_fatal_signal (sig=6) at /home/dan/mariadb-server-10.3/sql/signal_handler.cc:305
      #2  <signal handler called>
      #3  0x00007fa57a42c072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x1572760 <COND_thread_count+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
      #4  __pthread_cond_wait_common (abstime=0x0, mutex=0x1572158 <LOCK_thread_count>, cond=0x1572738 <COND_thread_count>) at pthread_cond_wait.c:502
      #5  __pthread_cond_wait (cond=0x1572738 <COND_thread_count>, mutex=0x1572158 <LOCK_thread_count>) at pthread_cond_wait.c:655
      #6  0x000000000057f5a3 in inline_mysql_cond_wait (mutex=<optimized out>, src_line=6179, that=<optimized out>, src_file=<optimized out>) at /home/dan/mariadb-server-10.3/include/mysql/psi/mysql_thread.h:1167
      #7  mysqld_main (argc=<optimized out>, argv=<optimized out>) at /home/dan/mariadb-server-10.3/sql/mysqld.cc:6179
      #8  0x00007fa577f5e1c1 in __libc_start_main (main=0x57a9e0 <main(int, char**)>, argc=9, argv=0x7ffc0841a158, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffc0841a148) at ../csu/libc-start.c:308
      #9  0x000000000057a92a in _start ()
      

      gcc-7.2.0

      encryption.innodb-force-corrupt 'ctr,innodb' w21 [ fail ]  Found warnings/errors in server log file!
              Test ended at 2018-04-08 19:06:13
      line
      Attempting backtrace. You can use the following information to find out
      ^ Found warnings in /home/dan/build-mariadb-10.3_lastest_upstream_gcc/mysql-test/var/21/log/mysqld.1.err
      ok
       
       - saving '/home/dan/build-mariadb-10.3_lastest_upstream_gcc/mysql-test/var/21/log/encryption.innodb-force-corrupt-ctr,innodb/' to '/home/dan/build-mariadb-10.3_lastest_upstream_gcc/mysql-test/var/log/encryption.innodb-force-corrupt-ctr,innodb/'
       - found 'core' (0/5)
       
      Trying 'dbx' to get a backtrace
       
      Trying 'gdb' to get a backtrace from coredump /home/dan/build-mariadb-10.3_lastest_upstream_gcc/mysql-test/var/log/encryption.innodb-force-corrupt-ctr,innodb/mysqld.1/data/core
      Core generated by '/home/dan/build-mariadb-10.3_lastest_upstream_gcc/sql/mysqld'
      Output from gdb follows. The first stack trace is from the failing thread.
      The following stack traces are from all threads (so the failing one is
      duplicated).
      --------------------------
      [New LWP 54251]
      [New LWP 57010]
      [New LWP 54488]
      [New LWP 57009]
      [New LWP 57008]
      [New LWP 57012]
      [New LWP 54474]
      [New LWP 57027]
      [New LWP 57011]
      [New LWP 57042]
      [New LWP 57013]
      [New LWP 57093]
      [New LWP 57026]
      [New LWP 57094]
      [New LWP 57014]
      [New LWP 57043]
      [New LWP 57025]
      [New LWP 57039]
      [New LWP 57110]
      [New LWP 57288]
      [New LWP 57095]
      [Thread debugging using libthread_db enabled]
      Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
      Core was generated by `/home/dan/build-mariadb-10.3_lastest_upstream_gcc/sql/mysqld --defaults-group-s'.
      Program terminated with signal SIGABRT, Aborted.
      #0  __pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:57
      57      ../sysdeps/unix/sysv/linux/pthread_kill.c: No such file or directory.
      [Current thread is 1 (Thread 0x7f9ed1dcdd40 (LWP 54251))]
      #0  __pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:57
      #1  0x000055c54904d7ea in my_write_core (sig=sig@entry=6) at /home/dan/mariadb-server-10.3/mysys/stacktrace.c:481
      #2  0x000055c548ba183a in handle_fatal_signal (sig=6) at /home/dan/mariadb-server-10.3/sql/signal_handler.cc:305
      #3  <signal handler called>
      #4  0x00007f9ed019c072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55c54996f0c8 <COND_thread_count+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
      #5  __pthread_cond_wait_common (abstime=0x0, mutex=0x55c54996f820 <LOCK_thread_count>, cond=0x55c54996f0a0 <COND_thread_count>) at pthread_cond_wait.c:502
      #6  __pthread_cond_wait (cond=cond@entry=0x55c54996f0a0 <COND_thread_count>, mutex=mutex@entry=0x55c54996f820 <LOCK_thread_count>) at pthread_cond_wait.c:655
      #7  0x000055c548945b5a in inline_mysql_cond_wait (src_file=0x55c549091008 "/home/dan/mariadb-server-10.3/sql/mysqld.cc", src_line=6179, mutex=0x55c54996f820 <LOCK_thread_count>, that=0x55c54996f0a0 <COND_thread_count>) at /home/dan/mariadb-server-10.3/include/mysql/psi/mysql_thread.h:1167
      #8  mysqld_main (argc=<optimized out>, argv=<optimized out>) at /home/dan/mariadb-server-10.3/sql/mysqld.cc:6179
      #9  0x00007f9ecf4dc1c1 in __libc_start_main (main=0x55c548923eb0 <main(int, char**)>, argc=29, argv=0x7ffee3c77748, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffee3c77738) at ../csu/libc-start.c:308
      #10 0x000055c5489386aa in _start ()
       
       
       
      Thread 21 (Thread 0x7f79b99ac700 (LWP 54523)):
      #0  do_sigwait (sig=0x7f79b99abc38, set=<optimized out>) at ../sysdeps/unix/sysv/linux/sigwait.c:64
      #1  __sigwait (set=set@entry=0x7f79b99abc90, sig=sig@entry=0x7f79b99abc38) at ../sysdeps/unix/sysv/linux/sigwait.c:96
      #2  0x0000556b2620b343 in signal_hand (arg=arg@entry=0x0) at /home/dan/mariadb-server-10.3/sql/mysqld.cc:3550
      #3  0x0000556b268e0e7e in pfs_spawn_thread (arg=0x556b292b7928) at /home/dan/mariadb-server-10.3/storage/perfschema/pfs.cc:1862
      #4  0x00007f79c226d7fc in start_thread (arg=0x7f79b99ac700) at pthread_create.c:465
      #5  0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 20 (Thread 0x7f79b98ce700 (LWP 54641)):
      #0  0x00007f79c2274072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x556b2723f0c8 <COND_thread_count+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
      #1  __pthread_cond_wait_common (abstime=0x0, mutex=0x556b2723f820 <LOCK_thread_count>, cond=0x556b2723f0a0 <COND_thread_count>) at pthread_cond_wait.c:502
      #2  __pthread_cond_wait (cond=cond@entry=0x556b2723f0a0 <COND_thread_count>, mutex=mutex@entry=0x556b2723f820 <LOCK_thread_count>) at pthread_cond_wait.c:655
      #3  0x0000556b2620fa8a in inline_mysql_cond_wait (src_file=0x556b26961008 "/home/dan/mariadb-server-10.3/sql/mysqld.cc", src_line=1850, mutex=0x556b2723f820 <LOCK_thread_count>, that=0x556b2723f0a0 <COND_thread_count>) at /home/dan/mariadb-server-10.3/include/mysql/psi/mysql_thread.h:1167
      #4  close_connections () at /home/dan/mariadb-server-10.3/sql/mysqld.cc:1850
      #5  kill_server (sig_ptr=0x0) at /home/dan/mariadb-server-10.3/sql/mysqld.cc:2033
      #6  0x0000556b2621015e in kill_server_thread (arg=arg@entry=0x7f79b99abc38) at /home/dan/mariadb-server-10.3/sql/mysqld.cc:2065
      #7  0x0000556b268e0e7e in pfs_spawn_thread (arg=0x556b292b7928) at /home/dan/mariadb-server-10.3/storage/perfschema/pfs.cc:1862
      #8  0x00007f79c226d7fc in start_thread (arg=0x7f79b98ce700) at pthread_create.c:465
      #9  0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 19 (Thread 0x7f79a57fa700 (LWP 54503)):
      #0  0x00007f79c2274072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x556b29094be4) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
      #1  __pthread_cond_wait_common (abstime=0x0, mutex=0x556b29094b90, cond=0x556b29094bb8) at pthread_cond_wait.c:502
      #2  __pthread_cond_wait (cond=cond@entry=0x556b29094bb8, mutex=mutex@entry=0x556b29094b90) at pthread_cond_wait.c:655
      #3  0x0000556b2664233f in os_event::wait (this=0x556b29094b80) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:163
      #4  os_event::wait_low (this=0x556b29094b80, reset_sig_count=2, reset_sig_count@entry=0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:333
      #5  0x0000556b26642805 in os_event_wait_low (event=<optimized out>, reset_sig_count=reset_sig_count@entry=0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:527
      #6  0x0000556b266c212d in srv_resume_thread (slot=slot@entry=0x556b271bdef8 <srv_sys+248>, sig_count=sig_count@entry=0, wait=wait@entry=true, timeout_usec=timeout_usec@entry=0) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0srv.cc:906
      #7  0x0000556b266c723d in srv_worker_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0srv.cc:2534
      #8  0x00007f79c226d7fc in start_thread (arg=0x7f79a57fa700) at pthread_create.c:465
      #9  0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 18 (Thread 0x7f79927fc700 (LWP 54508)):
      #0  0x00007f79c2274072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x556b29094f60) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
      #1  __pthread_cond_wait_common (abstime=0x0, mutex=0x556b29094f10, cond=0x556b29094f38) at pthread_cond_wait.c:502
      #2  __pthread_cond_wait (cond=cond@entry=0x556b29094f38, mutex=mutex@entry=0x556b29094f10) at pthread_cond_wait.c:655
      #3  0x0000556b2664233f in os_event::wait (this=0x556b29094f00) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:163
      #4  os_event::wait_low (this=0x556b29094f00, reset_sig_count=1, reset_sig_count@entry=0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:333
      #5  0x0000556b26642805 in os_event_wait_low (event=<optimized out>, reset_sig_count=reset_sig_count@entry=0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:527
      #6  0x0000556b26748f42 in buf_resize_thread () at /home/dan/mariadb-server-10.3/storage/innobase/buf/buf0buf.cc:3125
      #7  0x00007f79c226d7fc in start_thread (arg=0x7f79927fc700) at pthread_create.c:465
      #8  0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 17 (Thread 0x7f79a4ff9700 (LWP 54504)):
      #0  0x00007f79c2274072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x556b29094c64) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
      #1  __pthread_cond_wait_common (abstime=0x0, mutex=0x556b29094c10, cond=0x556b29094c38) at pthread_cond_wait.c:502
      #2  __pthread_cond_wait (cond=cond@entry=0x556b29094c38, mutex=mutex@entry=0x556b29094c10) at pthread_cond_wait.c:655
      #3  0x0000556b2664233f in os_event::wait (this=0x556b29094c00) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:163
      #4  os_event::wait_low (this=0x556b29094c00, reset_sig_count=2, reset_sig_count@entry=0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:333
      #5  0x0000556b26642805 in os_event_wait_low (event=<optimized out>, reset_sig_count=reset_sig_count@entry=0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:527
      #6  0x0000556b266c212d in srv_resume_thread (slot=slot@entry=0x556b271bdf30 <srv_sys+304>, sig_count=sig_count@entry=0, wait=wait@entry=true, timeout_usec=timeout_usec@entry=0) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0srv.cc:906
      #7  0x0000556b266c723d in srv_worker_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0srv.cc:2534
      #8  0x00007f79c226d7fc in start_thread (arg=0x7f79a4ff9700) at pthread_create.c:465
      #9  0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 16 (Thread 0x7f7992ffd700 (LWP 54507)):
      #0  0x00007f79c22785f8 in __GI___nanosleep (requested_time=requested_time@entry=0x7f7992ffc690, remaining=remaining@entry=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:27
      #1  0x0000556b266429cc in os_thread_sleep (tm=tm@entry=1000000) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0thread.cc:230
      #2  0x0000556b2673b4ea in btr_defragment_thread () at /home/dan/mariadb-server-10.3/storage/innobase/btr/btr0defragment.cc:768
      #3  0x00007f79c226d7fc in start_thread (arg=0x7f7992ffd700) at pthread_create.c:465
      #4  0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 15 (Thread 0x7f79937fe700 (LWP 54506)):
      #0  0x00007f79c2274072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x556b29094e60) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
      #1  __pthread_cond_wait_common (abstime=0x0, mutex=0x556b29094e10, cond=0x556b29094e38) at pthread_cond_wait.c:502
      #2  __pthread_cond_wait (cond=cond@entry=0x556b29094e38, mutex=mutex@entry=0x556b29094e10) at pthread_cond_wait.c:655
      #3  0x0000556b2664233f in os_event::wait (this=0x556b29094e00) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:163
      #4  os_event::wait_low (this=0x556b29094e00, reset_sig_count=1, reset_sig_count@entry=0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:333
      #5  0x0000556b26642805 in os_event_wait_low (event=<optimized out>, reset_sig_count=reset_sig_count@entry=0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:527
      #6  0x0000556b2674ff76 in buf_dump_thread () at /home/dan/mariadb-server-10.3/storage/innobase/buf/buf0dump.cc:828
      #7  0x00007f79c226d7fc in start_thread (arg=0x7f79937fe700) at pthread_create.c:465
      #8  0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 14 (Thread 0x7f79a7fff700 (LWP 54490)):
      #0  0x00007f79c2274786 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7f79a7ffeb50, expected=0, futex_word=0x556b29094d60) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
      #1  __pthread_cond_wait_common (abstime=0x7f79a7ffeb50, mutex=0x556b29094d10, cond=0x556b29094d38) at pthread_cond_wait.c:539
      #2  __pthread_cond_timedwait (cond=cond@entry=0x556b29094d38, mutex=mutex@entry=0x556b29094d10, abstime=abstime@entry=0x7f79a7ffeb50) at pthread_cond_wait.c:667
      #3  0x0000556b26642211 in os_event::timed_wait (this=this@entry=0x556b29094d00, abstime=abstime@entry=0x7f79a7ffeb50) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:283
      #4  0x0000556b26642431 in os_event::wait_time_low (this=0x556b29094d00, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=1) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:405
      #5  0x0000556b266427f5 in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:510
      #6  0x0000556b266c5133 in srv_error_monitor_thread () at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0srv.cc:1869
      #7  0x00007f79c226d7fc in start_thread (arg=0x7f79a7fff700) at pthread_create.c:465
      #8  0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 13 (Thread 0x7f79b3fff700 (LWP 54477)):
      #0  0x00007f79c388963a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
      #1  0x0000556b2663d769 in LinuxAIOHandler::collect (this=this@entry=0x7f79b3ffeaf0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:1913
      #2  0x0000556b2663dc10 in LinuxAIOHandler::poll (this=this@entry=0x7f79b3ffeaf0, m1=m1@entry=0x7f79b3ffeb90, m2=m2@entry=0x7f79b3ffeb98, request=request@entry=0x7f79b3ffeba0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2058
      #3  0x0000556b266400ab in os_aio_linux_handler (request=0x7f79b3ffeba0, m2=0x7f79b3ffeb98, m1=0x7f79b3ffeb90, global_segment=2) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2112
      #4  os_aio_handler (segment=segment@entry=2, m1=m1@entry=0x7f79b3ffeb90, m2=m2@entry=0x7f79b3ffeb98, request=request@entry=0x7f79b3ffeba0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:5638
      #5  0x0000556b2679d1e4 in fil_aio_wait (segment=segment@entry=2) at /home/dan/mariadb-server-10.3/storage/innobase/fil/fil0fil.cc:4678
      #6  0x0000556b266c8720 in io_handler_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0start.cc:339
      #7  0x00007f79c226d7fc in start_thread (arg=0x7f79b3fff700) at pthread_create.c:465
      #8  0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 12 (Thread 0x7f79b17fa700 (LWP 54501)):
      #0  0x00007f79c22785f8 in __GI___nanosleep (requested_time=requested_time@entry=0x7f79b17f9d10, remaining=remaining@entry=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:27
      #1  0x0000556b266429cc in os_thread_sleep (tm=tm@entry=1000000) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0thread.cc:230
      #2  0x0000556b266c6c9c in srv_master_sleep () at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0srv.cc:2358
      #3  srv_master_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0srv.cc:2403
      #4  0x00007f79c226d7fc in start_thread (arg=0x7f79b17fa700) at pthread_create.c:465
      #5  0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 11 (Thread 0x7f79a77fe700 (LWP 54491)):
      #0  0x00007f79c2274786 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7f79a77fdc80, expected=0, futex_word=0x556b29094de0) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
      #1  __pthread_cond_wait_common (abstime=0x7f79a77fdc80, mutex=0x556b29094d90, cond=0x556b29094db8) at pthread_cond_wait.c:539
      #2  __pthread_cond_timedwait (cond=cond@entry=0x556b29094db8, mutex=mutex@entry=0x556b29094d90, abstime=abstime@entry=0x7f79a77fdc80) at pthread_cond_wait.c:667
      #3  0x0000556b26642211 in os_event::timed_wait (this=this@entry=0x556b29094d80, abstime=abstime@entry=0x7f79a77fdc80) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:283
      #4  0x0000556b26642431 in os_event::wait_time_low (this=0x556b29094d80, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=1) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:405
      #5  0x0000556b266427f5 in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:510
      #6  0x0000556b266c6320 in srv_monitor_thread () at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0srv.cc:1705
      #7  0x00007f79c226d7fc in start_thread (arg=0x7f79a77fe700) at pthread_create.c:465
      #8  0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 10 (Thread 0x7f79b0ff9700 (LWP 54489)):
      #0  0x00007f79c2274786 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7f79b0ff8bf0, expected=0, futex_word=0x556b292358b0) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
      #1  __pthread_cond_wait_common (abstime=0x7f79b0ff8bf0, mutex=0x556b29235860, cond=0x556b29235888) at pthread_cond_wait.c:539
      #2  __pthread_cond_timedwait (cond=cond@entry=0x556b29235888, mutex=mutex@entry=0x556b29235860, abstime=abstime@entry=0x7f79b0ff8bf0) at pthread_cond_wait.c:667
      #3  0x0000556b26642211 in os_event::timed_wait (this=this@entry=0x556b29235850, abstime=abstime@entry=0x7f79b0ff8bf0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:283
      #4  0x0000556b26642431 in os_event::wait_time_low (this=this@entry=0x556b29235850, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=1) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:405
      #5  0x0000556b266427f5 in os_event_wait_time_low (event=event@entry=0x556b29235850, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:510
      #6  0x0000556b2661e5e1 in lock_wait_timeout_thread () at /home/dan/mariadb-server-10.3/storage/innobase/lock/lock0wait.cc:547
      #7  0x00007f79c226d7fc in start_thread (arg=0x7f79b0ff9700) at pthread_create.c:465
      #8  0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 9 (Thread 0x7f79b37fe700 (LWP 54478)):
      #0  0x00007f79c388963a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
      #1  0x0000556b2663d769 in LinuxAIOHandler::collect (this=this@entry=0x7f79b37fdaf0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:1913
      #2  0x0000556b2663dc10 in LinuxAIOHandler::poll (this=this@entry=0x7f79b37fdaf0, m1=m1@entry=0x7f79b37fdb90, m2=m2@entry=0x7f79b37fdb98, request=request@entry=0x7f79b37fdba0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2058
      #3  0x0000556b266400ab in os_aio_linux_handler (request=0x7f79b37fdba0, m2=0x7f79b37fdb98, m1=0x7f79b37fdb90, global_segment=3) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2112
      #4  os_aio_handler (segment=segment@entry=3, m1=m1@entry=0x7f79b37fdb90, m2=m2@entry=0x7f79b37fdb98, request=request@entry=0x7f79b37fdba0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:5638
      #5  0x0000556b2679d1e4 in fil_aio_wait (segment=segment@entry=3) at /home/dan/mariadb-server-10.3/storage/innobase/fil/fil0fil.cc:4678
      #6  0x0000556b266c8720 in io_handler_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0start.cc:339
      #7  0x00007f79c226d7fc in start_thread (arg=0x7f79b37fe700) at pthread_create.c:465
      #8  0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 8 (Thread 0x7f79b1ffb700 (LWP 54481)):
      #0  0x00007f79c2274786 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7f79b1ffaa90, expected=0, futex_word=0x556b29094ee4) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
      #1  __pthread_cond_wait_common (abstime=0x7f79b1ffaa90, mutex=0x556b29094e90, cond=0x556b29094eb8) at pthread_cond_wait.c:539
      #2  __pthread_cond_timedwait (cond=cond@entry=0x556b29094eb8, mutex=mutex@entry=0x556b29094e90, abstime=abstime@entry=0x7f79b1ffaa90) at pthread_cond_wait.c:667
      #3  0x0000556b26642211 in os_event::timed_wait (this=this@entry=0x556b29094e80, abstime=abstime@entry=0x7f79b1ffaa90) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:283
      #4  0x0000556b26642431 in os_event::wait_time_low (this=0x556b29094e80, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=reset_sig_count@entry=2) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:405
      #5  0x0000556b266427f5 in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=reset_sig_count@entry=2) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:510
      #6  0x0000556b2675764b in pc_sleep_if_needed (cur_time=1523232365070, sig_count=2, next_loop_time=1523232366070) at /home/dan/mariadb-server-10.3/storage/innobase/buf/buf0flu.cc:2675
      #7  buf_flush_page_cleaner_coordinator () at /home/dan/mariadb-server-10.3/storage/innobase/buf/buf0flu.cc:3140
      #8  0x00007f79c226d7fc in start_thread (arg=0x7f79b1ffb700) at pthread_create.c:465
      #9  0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 7 (Thread 0x7f79b2ffd700 (LWP 54479)):
      #0  0x00007f79c388963a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
      #1  0x0000556b2663d769 in LinuxAIOHandler::collect (this=this@entry=0x7f79b2ffcaf0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:1913
      #2  0x0000556b2663dc10 in LinuxAIOHandler::poll (this=this@entry=0x7f79b2ffcaf0, m1=m1@entry=0x7f79b2ffcb90, m2=m2@entry=0x7f79b2ffcb98, request=request@entry=0x7f79b2ffcba0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2058
      #3  0x0000556b266400ab in os_aio_linux_handler (request=0x7f79b2ffcba0, m2=0x7f79b2ffcb98, m1=0x7f79b2ffcb90, global_segment=4) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2112
      #4  os_aio_handler (segment=segment@entry=4, m1=m1@entry=0x7f79b2ffcb90, m2=m2@entry=0x7f79b2ffcb98, request=request@entry=0x7f79b2ffcba0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:5638
      #5  0x0000556b2679d1e4 in fil_aio_wait (segment=segment@entry=4) at /home/dan/mariadb-server-10.3/storage/innobase/fil/fil0fil.cc:4678
      #6  0x0000556b266c8720 in io_handler_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0start.cc:339
      #7  0x00007f79c226d7fc in start_thread (arg=0x7f79b2ffd700) at pthread_create.c:465
      #8  0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 6 (Thread 0x7f79b27fc700 (LWP 54480)):
      #0  0x00007f79c388963a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
      #1  0x0000556b2663d769 in LinuxAIOHandler::collect (this=this@entry=0x7f79b27fbaf0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:1913
      #2  0x0000556b2663dc10 in LinuxAIOHandler::poll (this=this@entry=0x7f79b27fbaf0, m1=m1@entry=0x7f79b27fbb90, m2=m2@entry=0x7f79b27fbb98, request=request@entry=0x7f79b27fbba0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2058
      #3  0x0000556b266400ab in os_aio_linux_handler (request=0x7f79b27fbba0, m2=0x7f79b27fbb98, m1=0x7f79b27fbb90, global_segment=5) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2112
      #4  os_aio_handler (segment=segment@entry=5, m1=m1@entry=0x7f79b27fbb90, m2=m2@entry=0x7f79b27fbb98, request=request@entry=0x7f79b27fbba0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:5638
      #5  0x0000556b2679d1e4 in fil_aio_wait (segment=segment@entry=5) at /home/dan/mariadb-server-10.3/storage/innobase/fil/fil0fil.cc:4678
      #6  0x0000556b266c8720 in io_handler_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0start.cc:339
      #7  0x00007f79c226d7fc in start_thread (arg=0x7f79b27fc700) at pthread_create.c:465
      #8  0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 5 (Thread 0x7f79b97ac700 (LWP 54475)):
      #0  0x00007f79c388963a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
      #1  0x0000556b2663d769 in LinuxAIOHandler::collect (this=this@entry=0x7f79b97abaf0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:1913
      #2  0x0000556b2663dc10 in LinuxAIOHandler::poll (this=this@entry=0x7f79b97abaf0, m1=m1@entry=0x7f79b97abb90, m2=m2@entry=0x7f79b97abb98, request=request@entry=0x7f79b97abba0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2058
      #3  0x0000556b266400ab in os_aio_linux_handler (request=0x7f79b97abba0, m2=0x7f79b97abb98, m1=0x7f79b97abb90, global_segment=0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2112
      #4  os_aio_handler (segment=segment@entry=0, m1=m1@entry=0x7f79b97abb90, m2=m2@entry=0x7f79b97abb98, request=request@entry=0x7f79b97abba0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:5638
      #5  0x0000556b2679d1e4 in fil_aio_wait (segment=segment@entry=0) at /home/dan/mariadb-server-10.3/storage/innobase/fil/fil0fil.cc:4678
      #6  0x0000556b266c8720 in io_handler_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0start.cc:339
      #7  0x00007f79c226d7fc in start_thread (arg=0x7f79b97ac700) at pthread_create.c:465
      #8  0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 4 (Thread 0x7f79b8fab700 (LWP 54476)):
      #0  0x00007f79c388963a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
      #1  0x0000556b2663d769 in LinuxAIOHandler::collect (this=this@entry=0x7f79b8faaaf0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:1913
      #2  0x0000556b2663dc10 in LinuxAIOHandler::poll (this=this@entry=0x7f79b8faaaf0, m1=m1@entry=0x7f79b8faab90, m2=m2@entry=0x7f79b8faab98, request=request@entry=0x7f79b8faaba0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2058
      #3  0x0000556b266400ab in os_aio_linux_handler (request=0x7f79b8faaba0, m2=0x7f79b8faab98, m1=0x7f79b8faab90, global_segment=1) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2112
      #4  os_aio_handler (segment=segment@entry=1, m1=m1@entry=0x7f79b8faab90, m2=m2@entry=0x7f79b8faab98, request=request@entry=0x7f79b8faaba0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:5638
      #5  0x0000556b2679d1e4 in fil_aio_wait (segment=segment@entry=1) at /home/dan/mariadb-server-10.3/storage/innobase/fil/fil0fil.cc:4678
      #6  0x0000556b266c8720 in io_handler_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0start.cc:339
      #7  0x00007f79c226d7fc in start_thread (arg=0x7f79b8fab700) at pthread_create.c:465
      #8  0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 3 (Thread 0x7f79bb063700 (LWP 54470)):
      #0  0x00007f79c2274786 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7f79bb062c70, expected=0, futex_word=0x556b27abeaa8 <COND_checkpoint+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
      #1  __pthread_cond_wait_common (abstime=0x7f79bb062c70, mutex=0x556b27abeac0 <LOCK_checkpoint>, cond=0x556b27abea80 <COND_checkpoint>) at pthread_cond_wait.c:539
      #2  __pthread_cond_timedwait (cond=cond@entry=0x556b27abea80 <COND_checkpoint>, mutex=mutex@entry=0x556b27abeac0 <LOCK_checkpoint>, abstime=abstime@entry=0x7f79bb062c70) at pthread_cond_wait.c:667
      #3  0x0000556b268303ff in inline_mysql_cond_timedwait (src_file=0x556b26aad278 "/home/dan/mariadb-server-10.3/storage/maria/ma_servicethread.c", src_line=116, abstime=0x7f79bb062c70, mutex=0x556b27abeac0 <LOCK_checkpoint>, that=0x556b27abea80 <COND_checkpoint>) at /home/dan/mariadb-server-10.3/include/mysql/psi/mysql_thread.h:1204
      #4  my_service_thread_sleep (control=control@entry=0x556b271c0900 <checkpoint_control>, sleep_time=sleep_time@entry=30000000000) at /home/dan/mariadb-server-10.3/storage/maria/ma_servicethread.c:115
      #5  0x0000556b26829c9d in ma_checkpoint_background (arg=arg@entry=0x1e) at /home/dan/mariadb-server-10.3/storage/maria/ma_checkpoint.c:707
      #6  0x0000556b268e0e7e in pfs_spawn_thread (arg=0x556b28f8fa88) at /home/dan/mariadb-server-10.3/storage/perfschema/pfs.cc:1862
      #7  0x00007f79c226d7fc in start_thread (arg=0x7f79bb063700) at pthread_create.c:465
      #8  0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 2 (Thread 0x7f79bc5d4700 (LWP 54463)):
      #0  0x00007f79c2274786 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7f79bc5d3ce0, expected=0, futex_word=0x556b27ace548 <COND_timer+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
      #1  __pthread_cond_wait_common (abstime=0x7f79bc5d3ce0, mutex=0x556b27ace560 <LOCK_timer>, cond=0x556b27ace520 <COND_timer>) at pthread_cond_wait.c:539
      #2  __pthread_cond_timedwait (cond=cond@entry=0x556b27ace520 <COND_timer>, mutex=mutex@entry=0x556b27ace560 <LOCK_timer>, abstime=abstime@entry=0x7f79bc5d3ce0) at pthread_cond_wait.c:667
      #3  0x0000556b269219cf in inline_mysql_cond_timedwait (that=0x556b27ace520 <COND_timer>, mutex=0x556b27ace560 <LOCK_timer>, src_file=0x556b26ac1768 "/home/dan/mariadb-server-10.3/mysys/thr_timer.c", src_line=292, abstime=0x7f79bc5d3ce0) at /home/dan/mariadb-server-10.3/include/mysql/psi/mysql_thread.h:1204
      #4  timer_handler (arg=arg@entry=0x0) at /home/dan/mariadb-server-10.3/mysys/thr_timer.c:292
      #5  0x0000556b268e0e7e in pfs_spawn_thread (arg=0x556b28f7bb18) at /home/dan/mariadb-server-10.3/storage/perfschema/pfs.cc:1862
      #6  0x00007f79c226d7fc in start_thread (arg=0x7f79bc5d4700) at pthread_create.c:465
      #7  0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 1 (Thread 0x7f79c3ea5d40 (LWP 54253)):
      #0  __pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:57
      #1  0x0000556b2691d7ea in my_write_core (sig=sig@entry=6) at /home/dan/mariadb-server-10.3/mysys/stacktrace.c:481
      #2  0x0000556b2647183a in handle_fatal_signal (sig=6) at /home/dan/mariadb-server-10.3/sql/signal_handler.cc:305
      #3  <signal handler called>
      #4  0x00007f79c2274072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x556b2723f0c8 <COND_thread_count+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
      #5  __pthread_cond_wait_common (abstime=0x0, mutex=0x556b2723f820 <LOCK_thread_count>, cond=0x556b2723f0a0 <COND_thread_count>) at pthread_cond_wait.c:502
      #6  __pthread_cond_wait (cond=cond@entry=0x556b2723f0a0 <COND_thread_count>, mutex=mutex@entry=0x556b2723f820 <LOCK_thread_count>) at pthread_cond_wait.c:655
      #7  0x0000556b26215b5a in inline_mysql_cond_wait (src_file=0x556b26961008 "/home/dan/mariadb-server-10.3/sql/mysqld.cc", src_line=6179, mutex=0x556b2723f820 <LOCK_thread_count>, that=0x556b2723f0a0 <COND_thread_count>) at /home/dan/mariadb-server-10.3/include/mysql/psi/mysql_thread.h:1167
      #8  mysqld_main (argc=<optimized out>, argv=<optimized out>) at /home/dan/mariadb-server-10.3/sql/mysqld.cc:6179
      #9  0x00007f79c15b41c1 in __libc_start_main (main=0x556b261f3eb0 <main(int, char**)>, argc=28, argv=0x7ffce77e2ef8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffce77e2ee8) at ../csu/libc-start.c:308
      #10 0x0000556b262086aa in _start ()
       
      Compressed file /home/dan/build-mariadb-10.3_lastest_upstream_gcc/mysql-test/var/log/encryption.innodb-bad-key-change2-innodb/mysqld.1/data/core
       
       
       
      encryption.innodb-bad-key-change2 'innodb' w44 [ fail ]  Found warnings/errors in server log file!
              Test ended at 2018-04-08 19:06:18
      line
      Attempting backtrace. You can use the following information to find out
      ^ Found warnings in /home/dan/build-mariadb-10.3_lastest_upstream_gcc/mysql-test/var/44/log/mysqld.1.err
      ok
       
       - saving '/home/dan/build-mariadb-10.3_lastest_upstream_gcc/mysql-test/var/44/log/encryption.innodb-bad-key-change2-innodb/' to '/home/dan/build-mariadb-10.3_lastest_upstream_gcc/mysql-test/var/log/encryption.innodb-bad-key-change2-innodb/'
       - found 'core' (1/5)
       
      Trying 'dbx' to get a backtrace
       
      Trying 'gdb' to get a backtrace from coredump /home/dan/build-mariadb-10.3_lastest_upstream_gcc/mysql-test/var/log/encryption.innodb-bad-key-change2-innodb/mysqld.1/data/core
      Core generated by '/home/dan/build-mariadb-10.3_lastest_upstream_gcc/sql/mysqld'
      Output from gdb follows. The first stack trace is from the failing thread.
      The following stack traces are from all threads (so the failing one is
      duplicated).
      --------------------------
      [New LWP 54253]
      [New LWP 54463]
      [New LWP 54470]
      [New LWP 54476]
      [New LWP 54475]
      [New LWP 54480]
      [New LWP 54479]
      [New LWP 54481]
      [New LWP 54478]
      [New LWP 54489]
      [New LWP 54491]
      [New LWP 54501]
      [New LWP 54477]
      [New LWP 54490]
      [New LWP 54506]
      [New LWP 54507]
      [New LWP 54504]
      [New LWP 54508]
      [New LWP 54503]
      [New LWP 54641]
      [New LWP 54523]
      [Thread debugging using libthread_db enabled]
      Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
      Core was generated by `/home/dan/build-mariadb-10.3_lastest_upstream_gcc/sql/mysqld --defaults-group-s'.
      Program terminated with signal SIGABRT, Aborted.
      #0  __pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:57
      57      ../sysdeps/unix/sysv/linux/pthread_kill.c: No such file or directory.
      [Current thread is 1 (Thread 0x7f79c3ea5d40 (LWP 54253))]
      #0  __pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:57
      #1  0x0000556b2691d7ea in my_write_core (sig=sig@entry=6) at /home/dan/mariadb-server-10.3/mysys/stacktrace.c:481
      #2  0x0000556b2647183a in handle_fatal_signal (sig=6) at /home/dan/mariadb-server-10.3/sql/signal_handler.cc:305
      #3  <signal handler called>
      #4  0x00007f79c2274072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x556b2723f0c8 <COND_thread_count+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
      #5  __pthread_cond_wait_common (abstime=0x0, mutex=0x556b2723f820 <LOCK_thread_count>, cond=0x556b2723f0a0 <COND_thread_count>) at pthread_cond_wait.c:502
      #6  __pthread_cond_wait (cond=cond@entry=0x556b2723f0a0 <COND_thread_count>, mutex=mutex@entry=0x556b2723f820 <LOCK_thread_count>) at pthread_cond_wait.c:655
      #7  0x0000556b26215b5a in inline_mysql_cond_wait (src_file=0x556b26961008 "/home/dan/mariadb-server-10.3/sql/mysqld.cc", src_line=6179, mutex=0x556b2723f820 <LOCK_thread_count>, that=0x556b2723f0a0 <COND_thread_count>) at /home/dan/mariadb-server-10.3/include/mysql/psi/mysql_thread.h:1167
      #8  mysqld_main (argc=<optimized out>, argv=<optimized out>) at /home/dan/mariadb-server-10.3/sql/mysqld.cc:6179
      #9  0x00007f79c15b41c1 in __libc_start_main (main=0x556b261f3eb0 <main(int, char**)>, argc=28, argv=0x7ffce77e2ef8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffce77e2ee8) at ../csu/libc-start.c:308
      #10 0x0000556b262086aa in _start ()
       
      Thread 21 (Thread 0x7f79b99ac700 (LWP 54523)):
      #0  do_sigwait (sig=0x7f79b99abc38, set=<optimized out>) at ../sysdeps/unix/sysv/linux/sigwait.c:64
      #1  __sigwait (set=set@entry=0x7f79b99abc90, sig=sig@entry=0x7f79b99abc38) at ../sysdeps/unix/sysv/linux/sigwait.c:96
      #2  0x0000556b2620b343 in signal_hand (arg=arg@entry=0x0) at /home/dan/mariadb-server-10.3/sql/mysqld.cc:3550
      #3  0x0000556b268e0e7e in pfs_spawn_thread (arg=0x556b292b7928) at /home/dan/mariadb-server-10.3/storage/perfschema/pfs.cc:1862
      #4  0x00007f79c226d7fc in start_thread (arg=0x7f79b99ac700) at pthread_create.c:465
      #5  0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 20 (Thread 0x7f79b98ce700 (LWP 54641)):
      #0  0x00007f79c2274072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x556b2723f0c8 <COND_thread_count+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
      #1  __pthread_cond_wait_common (abstime=0x0, mutex=0x556b2723f820 <LOCK_thread_count>, cond=0x556b2723f0a0 <COND_thread_count>) at pthread_cond_wait.c:502
      #2  __pthread_cond_wait (cond=cond@entry=0x556b2723f0a0 <COND_thread_count>, mutex=mutex@entry=0x556b2723f820 <LOCK_thread_count>) at pthread_cond_wait.c:655
      #3  0x0000556b2620fa8a in inline_mysql_cond_wait (src_file=0x556b26961008 "/home/dan/mariadb-server-10.3/sql/mysqld.cc", src_line=1850, mutex=0x556b2723f820 <LOCK_thread_count>, that=0x556b2723f0a0 <COND_thread_count>) at /home/dan/mariadb-server-10.3/include/mysql/psi/mysql_thread.h:1167
      #4  close_connections () at /home/dan/mariadb-server-10.3/sql/mysqld.cc:1850
      #5  kill_server (sig_ptr=0x0) at /home/dan/mariadb-server-10.3/sql/mysqld.cc:2033
      #6  0x0000556b2621015e in kill_server_thread (arg=arg@entry=0x7f79b99abc38) at /home/dan/mariadb-server-10.3/sql/mysqld.cc:2065
      #7  0x0000556b268e0e7e in pfs_spawn_thread (arg=0x556b292b7928) at /home/dan/mariadb-server-10.3/storage/perfschema/pfs.cc:1862
      #8  0x00007f79c226d7fc in start_thread (arg=0x7f79b98ce700) at pthread_create.c:465
      #9  0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 19 (Thread 0x7f79a57fa700 (LWP 54503)):
      #0  0x00007f79c2274072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x556b29094be4) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
      #1  __pthread_cond_wait_common (abstime=0x0, mutex=0x556b29094b90, cond=0x556b29094bb8) at pthread_cond_wait.c:502
      #2  __pthread_cond_wait (cond=cond@entry=0x556b29094bb8, mutex=mutex@entry=0x556b29094b90) at pthread_cond_wait.c:655
      #3  0x0000556b2664233f in os_event::wait (this=0x556b29094b80) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:163
      #4  os_event::wait_low (this=0x556b29094b80, reset_sig_count=2, reset_sig_count@entry=0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:333
      #5  0x0000556b26642805 in os_event_wait_low (event=<optimized out>, reset_sig_count=reset_sig_count@entry=0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:527
      #6  0x0000556b266c212d in srv_resume_thread (slot=slot@entry=0x556b271bdef8 <srv_sys+248>, sig_count=sig_count@entry=0, wait=wait@entry=true, timeout_usec=timeout_usec@entry=0) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0srv.cc:906
      #7  0x0000556b266c723d in srv_worker_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0srv.cc:2534
      #8  0x00007f79c226d7fc in start_thread (arg=0x7f79a57fa700) at pthread_create.c:465
      #9  0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 18 (Thread 0x7f79927fc700 (LWP 54508)):
      #0  0x00007f79c2274072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x556b29094f60) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
      #1  __pthread_cond_wait_common (abstime=0x0, mutex=0x556b29094f10, cond=0x556b29094f38) at pthread_cond_wait.c:502
      #2  __pthread_cond_wait (cond=cond@entry=0x556b29094f38, mutex=mutex@entry=0x556b29094f10) at pthread_cond_wait.c:655
      #3  0x0000556b2664233f in os_event::wait (this=0x556b29094f00) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:163
      #4  os_event::wait_low (this=0x556b29094f00, reset_sig_count=1, reset_sig_count@entry=0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:333
      #5  0x0000556b26642805 in os_event_wait_low (event=<optimized out>, reset_sig_count=reset_sig_count@entry=0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:527
      #6  0x0000556b26748f42 in buf_resize_thread () at /home/dan/mariadb-server-10.3/storage/innobase/buf/buf0buf.cc:3125
      #7  0x00007f79c226d7fc in start_thread (arg=0x7f79927fc700) at pthread_create.c:465
      #8  0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 17 (Thread 0x7f79a4ff9700 (LWP 54504)):
      #0  0x00007f79c2274072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x556b29094c64) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
      #1  __pthread_cond_wait_common (abstime=0x0, mutex=0x556b29094c10, cond=0x556b29094c38) at pthread_cond_wait.c:502
      #2  __pthread_cond_wait (cond=cond@entry=0x556b29094c38, mutex=mutex@entry=0x556b29094c10) at pthread_cond_wait.c:655
      #3  0x0000556b2664233f in os_event::wait (this=0x556b29094c00) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:163
      #4  os_event::wait_low (this=0x556b29094c00, reset_sig_count=2, reset_sig_count@entry=0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:333
      #5  0x0000556b26642805 in os_event_wait_low (event=<optimized out>, reset_sig_count=reset_sig_count@entry=0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:527
      #6  0x0000556b266c212d in srv_resume_thread (slot=slot@entry=0x556b271bdf30 <srv_sys+304>, sig_count=sig_count@entry=0, wait=wait@entry=true, timeout_usec=timeout_usec@entry=0) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0srv.cc:906
      #7  0x0000556b266c723d in srv_worker_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0srv.cc:2534
      #8  0x00007f79c226d7fc in start_thread (arg=0x7f79a4ff9700) at pthread_create.c:465
      #9  0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 16 (Thread 0x7f7992ffd700 (LWP 54507)):
      #0  0x00007f79c22785f8 in __GI___nanosleep (requested_time=requested_time@entry=0x7f7992ffc690, remaining=remaining@entry=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:27
      #1  0x0000556b266429cc in os_thread_sleep (tm=tm@entry=1000000) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0thread.cc:230
      #2  0x0000556b2673b4ea in btr_defragment_thread () at /home/dan/mariadb-server-10.3/storage/innobase/btr/btr0defragment.cc:768
      #3  0x00007f79c226d7fc in start_thread (arg=0x7f7992ffd700) at pthread_create.c:465
      #4  0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 15 (Thread 0x7f79937fe700 (LWP 54506)):
      #0  0x00007f79c2274072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x556b29094e60) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
      #1  __pthread_cond_wait_common (abstime=0x0, mutex=0x556b29094e10, cond=0x556b29094e38) at pthread_cond_wait.c:502
      #2  __pthread_cond_wait (cond=cond@entry=0x556b29094e38, mutex=mutex@entry=0x556b29094e10) at pthread_cond_wait.c:655
      #3  0x0000556b2664233f in os_event::wait (this=0x556b29094e00) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:163
      #4  os_event::wait_low (this=0x556b29094e00, reset_sig_count=1, reset_sig_count@entry=0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:333
      #5  0x0000556b26642805 in os_event_wait_low (event=<optimized out>, reset_sig_count=reset_sig_count@entry=0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:527
      #6  0x0000556b2674ff76 in buf_dump_thread () at /home/dan/mariadb-server-10.3/storage/innobase/buf/buf0dump.cc:828
      #7  0x00007f79c226d7fc in start_thread (arg=0x7f79937fe700) at pthread_create.c:465
      #8  0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 14 (Thread 0x7f79a7fff700 (LWP 54490)):
      #0  0x00007f79c2274786 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7f79a7ffeb50, expected=0, futex_word=0x556b29094d60) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
      #1  __pthread_cond_wait_common (abstime=0x7f79a7ffeb50, mutex=0x556b29094d10, cond=0x556b29094d38) at pthread_cond_wait.c:539
      #2  __pthread_cond_timedwait (cond=cond@entry=0x556b29094d38, mutex=mutex@entry=0x556b29094d10, abstime=abstime@entry=0x7f79a7ffeb50) at pthread_cond_wait.c:667
      #3  0x0000556b26642211 in os_event::timed_wait (this=this@entry=0x556b29094d00, abstime=abstime@entry=0x7f79a7ffeb50) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:283
      #4  0x0000556b26642431 in os_event::wait_time_low (this=0x556b29094d00, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=1) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:405
      #5  0x0000556b266427f5 in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:510
      #6  0x0000556b266c5133 in srv_error_monitor_thread () at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0srv.cc:1869
      #7  0x00007f79c226d7fc in start_thread (arg=0x7f79a7fff700) at pthread_create.c:465
      #8  0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 13 (Thread 0x7f79b3fff700 (LWP 54477)):
      #0  0x00007f79c388963a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
      #1  0x0000556b2663d769 in LinuxAIOHandler::collect (this=this@entry=0x7f79b3ffeaf0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:1913
      #2  0x0000556b2663dc10 in LinuxAIOHandler::poll (this=this@entry=0x7f79b3ffeaf0, m1=m1@entry=0x7f79b3ffeb90, m2=m2@entry=0x7f79b3ffeb98, request=request@entry=0x7f79b3ffeba0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2058
      #3  0x0000556b266400ab in os_aio_linux_handler (request=0x7f79b3ffeba0, m2=0x7f79b3ffeb98, m1=0x7f79b3ffeb90, global_segment=2) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2112
      #4  os_aio_handler (segment=segment@entry=2, m1=m1@entry=0x7f79b3ffeb90, m2=m2@entry=0x7f79b3ffeb98, request=request@entry=0x7f79b3ffeba0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:5638
      #5  0x0000556b2679d1e4 in fil_aio_wait (segment=segment@entry=2) at /home/dan/mariadb-server-10.3/storage/innobase/fil/fil0fil.cc:4678
      #6  0x0000556b266c8720 in io_handler_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0start.cc:339
      #7  0x00007f79c226d7fc in start_thread (arg=0x7f79b3fff700) at pthread_create.c:465
      #8  0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 12 (Thread 0x7f79b17fa700 (LWP 54501)):
      #0  0x00007f79c22785f8 in __GI___nanosleep (requested_time=requested_time@entry=0x7f79b17f9d10, remaining=remaining@entry=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:27
      #1  0x0000556b266429cc in os_thread_sleep (tm=tm@entry=1000000) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0thread.cc:230
      #2  0x0000556b266c6c9c in srv_master_sleep () at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0srv.cc:2358
      #3  srv_master_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0srv.cc:2403
      #4  0x00007f79c226d7fc in start_thread (arg=0x7f79b17fa700) at pthread_create.c:465
      #5  0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 11 (Thread 0x7f79a77fe700 (LWP 54491)):
      #0  0x00007f79c2274786 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7f79a77fdc80, expected=0, futex_word=0x556b29094de0) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
      #1  __pthread_cond_wait_common (abstime=0x7f79a77fdc80, mutex=0x556b29094d90, cond=0x556b29094db8) at pthread_cond_wait.c:539
      #2  __pthread_cond_timedwait (cond=cond@entry=0x556b29094db8, mutex=mutex@entry=0x556b29094d90, abstime=abstime@entry=0x7f79a77fdc80) at pthread_cond_wait.c:667
      #3  0x0000556b26642211 in os_event::timed_wait (this=this@entry=0x556b29094d80, abstime=abstime@entry=0x7f79a77fdc80) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:283
      #4  0x0000556b26642431 in os_event::wait_time_low (this=0x556b29094d80, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=1) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:405
      #5  0x0000556b266427f5 in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:510
      #6  0x0000556b266c6320 in srv_monitor_thread () at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0srv.cc:1705
      #7  0x00007f79c226d7fc in start_thread (arg=0x7f79a77fe700) at pthread_create.c:465
      #8  0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 10 (Thread 0x7f79b0ff9700 (LWP 54489)):
      #0  0x00007f79c2274786 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7f79b0ff8bf0, expected=0, futex_word=0x556b292358b0) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
      #1  __pthread_cond_wait_common (abstime=0x7f79b0ff8bf0, mutex=0x556b29235860, cond=0x556b29235888) at pthread_cond_wait.c:539
      #2  __pthread_cond_timedwait (cond=cond@entry=0x556b29235888, mutex=mutex@entry=0x556b29235860, abstime=abstime@entry=0x7f79b0ff8bf0) at pthread_cond_wait.c:667
      #3  0x0000556b26642211 in os_event::timed_wait (this=this@entry=0x556b29235850, abstime=abstime@entry=0x7f79b0ff8bf0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:283
      #4  0x0000556b26642431 in os_event::wait_time_low (this=this@entry=0x556b29235850, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=1) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:405
      #5  0x0000556b266427f5 in os_event_wait_time_low (event=event@entry=0x556b29235850, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:510
      #6  0x0000556b2661e5e1 in lock_wait_timeout_thread () at /home/dan/mariadb-server-10.3/storage/innobase/lock/lock0wait.cc:547
      #7  0x00007f79c226d7fc in start_thread (arg=0x7f79b0ff9700) at pthread_create.c:465
      #8  0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 9 (Thread 0x7f79b37fe700 (LWP 54478)):
      #0  0x00007f79c388963a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
      #1  0x0000556b2663d769 in LinuxAIOHandler::collect (this=this@entry=0x7f79b37fdaf0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:1913
      #2  0x0000556b2663dc10 in LinuxAIOHandler::poll (this=this@entry=0x7f79b37fdaf0, m1=m1@entry=0x7f79b37fdb90, m2=m2@entry=0x7f79b37fdb98, request=request@entry=0x7f79b37fdba0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2058
      #3  0x0000556b266400ab in os_aio_linux_handler (request=0x7f79b37fdba0, m2=0x7f79b37fdb98, m1=0x7f79b37fdb90, global_segment=3) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2112
      #4  os_aio_handler (segment=segment@entry=3, m1=m1@entry=0x7f79b37fdb90, m2=m2@entry=0x7f79b37fdb98, request=request@entry=0x7f79b37fdba0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:5638
      #5  0x0000556b2679d1e4 in fil_aio_wait (segment=segment@entry=3) at /home/dan/mariadb-server-10.3/storage/innobase/fil/fil0fil.cc:4678
      #6  0x0000556b266c8720 in io_handler_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0start.cc:339
      #7  0x00007f79c226d7fc in start_thread (arg=0x7f79b37fe700) at pthread_create.c:465
      #8  0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 8 (Thread 0x7f79b1ffb700 (LWP 54481)):
      #0  0x00007f79c2274786 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7f79b1ffaa90, expected=0, futex_word=0x556b29094ee4) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
      #1  __pthread_cond_wait_common (abstime=0x7f79b1ffaa90, mutex=0x556b29094e90, cond=0x556b29094eb8) at pthread_cond_wait.c:539
      #2  __pthread_cond_timedwait (cond=cond@entry=0x556b29094eb8, mutex=mutex@entry=0x556b29094e90, abstime=abstime@entry=0x7f79b1ffaa90) at pthread_cond_wait.c:667
      #3  0x0000556b26642211 in os_event::timed_wait (this=this@entry=0x556b29094e80, abstime=abstime@entry=0x7f79b1ffaa90) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:283
      #4  0x0000556b26642431 in os_event::wait_time_low (this=0x556b29094e80, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=reset_sig_count@entry=2) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:405
      #5  0x0000556b266427f5 in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=reset_sig_count@entry=2) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0event.cc:510
      #6  0x0000556b2675764b in pc_sleep_if_needed (cur_time=1523232365070, sig_count=2, next_loop_time=1523232366070) at /home/dan/mariadb-server-10.3/storage/innobase/buf/buf0flu.cc:2675
      #7  buf_flush_page_cleaner_coordinator () at /home/dan/mariadb-server-10.3/storage/innobase/buf/buf0flu.cc:3140
      #8  0x00007f79c226d7fc in start_thread (arg=0x7f79b1ffb700) at pthread_create.c:465
      #9  0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 7 (Thread 0x7f79b2ffd700 (LWP 54479)):
      #0  0x00007f79c388963a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
      #1  0x0000556b2663d769 in LinuxAIOHandler::collect (this=this@entry=0x7f79b2ffcaf0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:1913
      #2  0x0000556b2663dc10 in LinuxAIOHandler::poll (this=this@entry=0x7f79b2ffcaf0, m1=m1@entry=0x7f79b2ffcb90, m2=m2@entry=0x7f79b2ffcb98, request=request@entry=0x7f79b2ffcba0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2058
      #3  0x0000556b266400ab in os_aio_linux_handler (request=0x7f79b2ffcba0, m2=0x7f79b2ffcb98, m1=0x7f79b2ffcb90, global_segment=4) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2112
      #4  os_aio_handler (segment=segment@entry=4, m1=m1@entry=0x7f79b2ffcb90, m2=m2@entry=0x7f79b2ffcb98, request=request@entry=0x7f79b2ffcba0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:5638
      #5  0x0000556b2679d1e4 in fil_aio_wait (segment=segment@entry=4) at /home/dan/mariadb-server-10.3/storage/innobase/fil/fil0fil.cc:4678
      #6  0x0000556b266c8720 in io_handler_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0start.cc:339
      #7  0x00007f79c226d7fc in start_thread (arg=0x7f79b2ffd700) at pthread_create.c:465
      #8  0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 6 (Thread 0x7f79b27fc700 (LWP 54480)):
      #0  0x00007f79c388963a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
      #1  0x0000556b2663d769 in LinuxAIOHandler::collect (this=this@entry=0x7f79b27fbaf0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:1913
      #2  0x0000556b2663dc10 in LinuxAIOHandler::poll (this=this@entry=0x7f79b27fbaf0, m1=m1@entry=0x7f79b27fbb90, m2=m2@entry=0x7f79b27fbb98, request=request@entry=0x7f79b27fbba0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2058
      #3  0x0000556b266400ab in os_aio_linux_handler (request=0x7f79b27fbba0, m2=0x7f79b27fbb98, m1=0x7f79b27fbb90, global_segment=5) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2112
      #4  os_aio_handler (segment=segment@entry=5, m1=m1@entry=0x7f79b27fbb90, m2=m2@entry=0x7f79b27fbb98, request=request@entry=0x7f79b27fbba0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:5638
      #5  0x0000556b2679d1e4 in fil_aio_wait (segment=segment@entry=5) at /home/dan/mariadb-server-10.3/storage/innobase/fil/fil0fil.cc:4678
      #6  0x0000556b266c8720 in io_handler_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0start.cc:339
      #7  0x00007f79c226d7fc in start_thread (arg=0x7f79b27fc700) at pthread_create.c:465
      #8  0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 5 (Thread 0x7f79b97ac700 (LWP 54475)):
      #0  0x00007f79c388963a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
      #1  0x0000556b2663d769 in LinuxAIOHandler::collect (this=this@entry=0x7f79b97abaf0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:1913
      #2  0x0000556b2663dc10 in LinuxAIOHandler::poll (this=this@entry=0x7f79b97abaf0, m1=m1@entry=0x7f79b97abb90, m2=m2@entry=0x7f79b97abb98, request=request@entry=0x7f79b97abba0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2058
      #3  0x0000556b266400ab in os_aio_linux_handler (request=0x7f79b97abba0, m2=0x7f79b97abb98, m1=0x7f79b97abb90, global_segment=0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2112
      #4  os_aio_handler (segment=segment@entry=0, m1=m1@entry=0x7f79b97abb90, m2=m2@entry=0x7f79b97abb98, request=request@entry=0x7f79b97abba0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:5638
      #5  0x0000556b2679d1e4 in fil_aio_wait (segment=segment@entry=0) at /home/dan/mariadb-server-10.3/storage/innobase/fil/fil0fil.cc:4678
      #6  0x0000556b266c8720 in io_handler_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0start.cc:339
      #7  0x00007f79c226d7fc in start_thread (arg=0x7f79b97ac700) at pthread_create.c:465
      #8  0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 4 (Thread 0x7f79b8fab700 (LWP 54476)):
      #0  0x00007f79c388963a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
      #1  0x0000556b2663d769 in LinuxAIOHandler::collect (this=this@entry=0x7f79b8faaaf0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:1913
      #2  0x0000556b2663dc10 in LinuxAIOHandler::poll (this=this@entry=0x7f79b8faaaf0, m1=m1@entry=0x7f79b8faab90, m2=m2@entry=0x7f79b8faab98, request=request@entry=0x7f79b8faaba0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2058
      #3  0x0000556b266400ab in os_aio_linux_handler (request=0x7f79b8faaba0, m2=0x7f79b8faab98, m1=0x7f79b8faab90, global_segment=1) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:2112
      #4  os_aio_handler (segment=segment@entry=1, m1=m1@entry=0x7f79b8faab90, m2=m2@entry=0x7f79b8faab98, request=request@entry=0x7f79b8faaba0) at /home/dan/mariadb-server-10.3/storage/innobase/os/os0file.cc:5638
      #5  0x0000556b2679d1e4 in fil_aio_wait (segment=segment@entry=1) at /home/dan/mariadb-server-10.3/storage/innobase/fil/fil0fil.cc:4678
      #6  0x0000556b266c8720 in io_handler_thread (arg=<optimized out>) at /home/dan/mariadb-server-10.3/storage/innobase/srv/srv0start.cc:339
      #7  0x00007f79c226d7fc in start_thread (arg=0x7f79b8fab700) at pthread_create.c:465
      #8  0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 3 (Thread 0x7f79bb063700 (LWP 54470)):
      #0  0x00007f79c2274786 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7f79bb062c70, expected=0, futex_word=0x556b27abeaa8 <COND_checkpoint+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
      #1  __pthread_cond_wait_common (abstime=0x7f79bb062c70, mutex=0x556b27abeac0 <LOCK_checkpoint>, cond=0x556b27abea80 <COND_checkpoint>) at pthread_cond_wait.c:539
      #2  __pthread_cond_timedwait (cond=cond@entry=0x556b27abea80 <COND_checkpoint>, mutex=mutex@entry=0x556b27abeac0 <LOCK_checkpoint>, abstime=abstime@entry=0x7f79bb062c70) at pthread_cond_wait.c:667
      #3  0x0000556b268303ff in inline_mysql_cond_timedwait (src_file=0x556b26aad278 "/home/dan/mariadb-server-10.3/storage/maria/ma_servicethread.c", src_line=116, abstime=0x7f79bb062c70, mutex=0x556b27abeac0 <LOCK_checkpoint>, that=0x556b27abea80 <COND_checkpoint>) at /home/dan/mariadb-server-10.3/include/mysql/psi/mysql_thread.h:1204
      #4  my_service_thread_sleep (control=control@entry=0x556b271c0900 <checkpoint_control>, sleep_time=sleep_time@entry=30000000000) at /home/dan/mariadb-server-10.3/storage/maria/ma_servicethread.c:115
      #5  0x0000556b26829c9d in ma_checkpoint_background (arg=arg@entry=0x1e) at /home/dan/mariadb-server-10.3/storage/maria/ma_checkpoint.c:707
      #6  0x0000556b268e0e7e in pfs_spawn_thread (arg=0x556b28f8fa88) at /home/dan/mariadb-server-10.3/storage/perfschema/pfs.cc:1862
      #7  0x00007f79c226d7fc in start_thread (arg=0x7f79bb063700) at pthread_create.c:465
      #8  0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 2 (Thread 0x7f79bc5d4700 (LWP 54463)):
      #0  0x00007f79c2274786 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7f79bc5d3ce0, expected=0, futex_word=0x556b27ace548 <COND_timer+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
      #1  __pthread_cond_wait_common (abstime=0x7f79bc5d3ce0, mutex=0x556b27ace560 <LOCK_timer>, cond=0x556b27ace520 <COND_timer>) at pthread_cond_wait.c:539
      #2  __pthread_cond_timedwait (cond=cond@entry=0x556b27ace520 <COND_timer>, mutex=mutex@entry=0x556b27ace560 <LOCK_timer>, abstime=abstime@entry=0x7f79bc5d3ce0) at pthread_cond_wait.c:667
      #3  0x0000556b269219cf in inline_mysql_cond_timedwait (that=0x556b27ace520 <COND_timer>, mutex=0x556b27ace560 <LOCK_timer>, src_file=0x556b26ac1768 "/home/dan/mariadb-server-10.3/mysys/thr_timer.c", src_line=292, abstime=0x7f79bc5d3ce0) at /home/dan/mariadb-server-10.3/include/mysql/psi/mysql_thread.h:1204
      #4  timer_handler (arg=arg@entry=0x0) at /home/dan/mariadb-server-10.3/mysys/thr_timer.c:292
      #5  0x0000556b268e0e7e in pfs_spawn_thread (arg=0x556b28f7bb18) at /home/dan/mariadb-server-10.3/storage/perfschema/pfs.cc:1862
      #6  0x00007f79c226d7fc in start_thread (arg=0x7f79bc5d4700) at pthread_create.c:465
      #7  0x00007f79c16a7b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 1 (Thread 0x7f79c3ea5d40 (LWP 54253)):
      #0  __pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:57
      #1  0x0000556b2691d7ea in my_write_core (sig=sig@entry=6) at /home/dan/mariadb-server-10.3/mysys/stacktrace.c:481
      #2  0x0000556b2647183a in handle_fatal_signal (sig=6) at /home/dan/mariadb-server-10.3/sql/signal_handler.cc:305
      #3  <signal handler called>
      #4  0x00007f79c2274072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x556b2723f0c8 <COND_thread_count+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
      #5  __pthread_cond_wait_common (abstime=0x0, mutex=0x556b2723f820 <LOCK_thread_count>, cond=0x556b2723f0a0 <COND_thread_count>) at pthread_cond_wait.c:502
      #6  __pthread_cond_wait (cond=cond@entry=0x556b2723f0a0 <COND_thread_count>, mutex=mutex@entry=0x556b2723f820 <LOCK_thread_count>) at pthread_cond_wait.c:655
      #7  0x0000556b26215b5a in inline_mysql_cond_wait (src_file=0x556b26961008 "/home/dan/mariadb-server-10.3/sql/mysqld.cc", src_line=6179, mutex=0x556b2723f820 <LOCK_thread_count>, that=0x556b2723f0a0 <COND_thread_count>) at /home/dan/mariadb-server-10.3/include/mysql/psi/mysql_thread.h:1167
      #8  mysqld_main (argc=<optimized out>, argv=<optimized out>) at /home/dan/mariadb-server-10.3/sql/mysqld.cc:6179
      #9  0x00007f79c15b41c1 in __libc_start_main (main=0x556b261f3eb0 <main(int, char**)>, argc=28, argv=0x7ffce77e2ef8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffce77e2ee8) at ../csu/libc-start.c:308
      #10 0x0000556b262086aa in _start ()
       
      Compressed file /home/dan/build-mariadb-10.3_lastest_upstream_gcc/mysql-test/var/log/encryption.innodb-bad-key-change2-innodb/mysqld.1/data/core
      

      Attachments

        Issue Links

          Activity

            danblack Daniel Black added a comment -

            clang-6.0, 10.3-0f6186c5509309c5d647f9631f04071a62a013df

             clang-6.0 --version
            clang version 6.0.1-svn329487-1~exp1~20180408152107.70 (branches/release_60)
             
            encryption.innodb-compressed-blob '8k,cbc,innodb,none' w60 [ fail ]  Found warnings/errors in server log file!
                    Test ended at 2018-04-08 19:19:12
            line
            Attempting backtrace. You can use the following information to find out
            ^ Found warnings in /home/dan/build-mariadb-10.3_lastest_upstream_clang-6/mysql-test/var/60/log/mysqld.1.err
            ok
             
             - saving '/home/dan/build-mariadb-10.3_lastest_upstream_clang-6/mysql-test/var/60/log/encryption.innodb-compressed-blob-8k,cbc,innodb,none/' to '/home/dan/build-mariadb-10.3_lastest_upstream_clang-6/mysql-test/var/log/encryption.innodb-compressed-blob-8k,cbc,innodb,none/'
             - found 'core' (0/5)
             
            Trying 'dbx' to get a backtrace
             
            Trying 'gdb' to get a backtrace from coredump /home/dan/build-mariadb-10.3_lastest_upstream_clang-6/mysql-test/var/log/encryption.innodb-compressed-blob-8k,cbc,innodb,none/mysqld.1/data/core
            Core generated by '/home/dan/build-mariadb-10.3_lastest_upstream_clang-6/sql/mysqld'
            Output from gdb follows. The first stack trace is from the failing thread.
            The following stack traces are from all threads (so the failing one is
            duplicated).
            --------------------------
            [New LWP 49983]
            [New LWP 50457]
            [New LWP 52135]
            [New LWP 52153]
            [New LWP 50452]
            [New LWP 52129]
            [New LWP 52152]
            [New LWP 52166]
            [New LWP 52154]
            [New LWP 52132]
            [New LWP 52188]
            [New LWP 52133]
            [New LWP 52189]
            [New LWP 52190]
            [New LWP 52193]
            [New LWP 52131]
            [New LWP 52130]
            [New LWP 52134]
            [New LWP 52192]
            [New LWP 52168]
            [New LWP 52169]
            [New LWP 52187]
            [New LWP 52194]
            [New LWP 52207]
            [New LWP 52241]
            [Thread debugging using libthread_db enabled]
            Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
            Core was generated by `/home/dan/build-mariadb-10.3_lastest_upstream_clang-6/sql/mysqld --defaults-gro'.
            Program terminated with signal SIGABRT, Aborted.
            #0  __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:57
            57      ../sysdeps/unix/sysv/linux/pthread_kill.c: No such file or directory.
            [Current thread is 1 (Thread 0x7ff8db08ed40 (LWP 49983))]
            #0  __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:57
            #1  0x000000000110698b in my_write_core ()
            #2  0x0000000000a29f5a in handle_fatal_signal ()
            #3  <signal handler called>
            #4  0x00007ff8dac6b072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x1b0d2a0 <COND_thread_count+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
            #5  __pthread_cond_wait_common (abstime=0x0, mutex=0x1b0cc98 <LOCK_thread_count>, cond=0x1b0d278 <COND_thread_count>) at pthread_cond_wait.c:502
            #6  __pthread_cond_wait (cond=0x1b0d278 <COND_thread_count>, mutex=0x1b0cc98 <LOCK_thread_count>) at pthread_cond_wait.c:655
            #7  0x000000000061ede6 in inline_mysql_cond_wait(st_mysql_cond*, st_mysql_mutex*, char const*, unsigned int) ()
            #8  0x0000000000620dea in mysqld_main(int, char**) ()
            #9  0x000000000061cfa2 in main ()
             
            Thread 25 (Thread 0x7ff8d0b04700 (LWP 52241)):
            #0  0x00007ff8dac6b072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x1b0d2a0 <COND_thread_count+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
            #1  __pthread_cond_wait_common (abstime=0x0, mutex=0x1b0cc98 <LOCK_thread_count>, cond=0x1b0d278 <COND_thread_count>) at pthread_cond_wait.c:502
            #2  __pthread_cond_wait (cond=0x1b0d278 <COND_thread_count>, mutex=0x1b0cc98 <LOCK_thread_count>) at pthread_cond_wait.c:655
            #3  0x000000000061ede6 in inline_mysql_cond_wait(st_mysql_cond*, st_mysql_mutex*, char const*, unsigned int) ()
            #4  0x000000000062a304 in close_connections() ()
            #5  0x000000000061daf0 in kill_server(void*) ()
            #6  0x000000000061d8cd in kill_server_thread ()
            #7  0x00000000010a8d45 in pfs_spawn_thread ()
            #8  0x00007ff8dac647fc in start_thread (arg=0x7ff8d0b04700) at pthread_create.c:465
            #9  0x00007ff8d8890b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
             
            Thread 24 (Thread 0x7ff8d0be2700 (LWP 52207)):
            #0  do_sigwait (sig=0x7ff8d0be1c8c, set=<optimized out>) at ../sysdeps/unix/sysv/linux/sigwait.c:64
            #1  __sigwait (set=0x7ff8d0be1c90, sig=0x7ff8d0be1c8c) at ../sysdeps/unix/sysv/linux/sigwait.c:96
            #2  0x000000000061f560 in signal_hand ()
            #3  0x00000000010a8d45 in pfs_spawn_thread ()
            #4  0x00007ff8dac647fc in start_thread (arg=0x7ff8d0be2700) at pthread_create.c:465
            #5  0x00007ff8d8890b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
             
            Thread 23 (Thread 0x7ff87f7fe700 (LWP 52194)):
            #0  0x00007ff8dac6b072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x3af0250) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
            #1  __pthread_cond_wait_common (abstime=0x0, mutex=0x3af0200, cond=0x3af0228) at pthread_cond_wait.c:502
            #2  __pthread_cond_wait (cond=0x3af0228, mutex=0x3af0200) at pthread_cond_wait.c:655
            #3  0x0000000000cefec0 in os_event::wait() ()
            #4  0x0000000000cef9ef in os_event::wait_low(long) ()
            #5  0x0000000000cefe0d in os_event_wait_low(os_event*, long) ()
            #6  0x0000000000e4abe9 in buf_resize_thread ()
            #7  0x00007ff8dac647fc in start_thread (arg=0x7ff87f7fe700) at pthread_create.c:465
            #8  0x00007ff8d8890b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
             
            Thread 22 (Thread 0x7ff89affd700 (LWP 52187)):
            #0  0x00007ff8dac6b072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x3af0150) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
            #1  __pthread_cond_wait_common (abstime=0x0, mutex=0x3af0100, cond=0x3af0128) at pthread_cond_wait.c:502
            #2  __pthread_cond_wait (cond=0x3af0128, mutex=0x3af0100) at pthread_cond_wait.c:655
            #3  0x0000000000cefec0 in os_event::wait() ()
            #4  0x0000000000cef9ef in os_event::wait_low(long) ()
            #5  0x0000000000cefe0d in os_event_wait_low(os_event*, long) ()
            #6  0x0000000000e63aef in buf_dump_thread ()
            #7  0x00007ff8dac647fc in start_thread (arg=0x7ff89affd700) at pthread_create.c:465
            #8  0x00007ff8d8890b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
             
            Thread 21 (Thread 0x7ff89bfff700 (LWP 52169)):
            #0  0x00007ff8dac6b072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x3aefed4) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
            #1  __pthread_cond_wait_common (abstime=0x0, mutex=0x3aefe80, cond=0x3aefea8) at pthread_cond_wait.c:502
            #2  __pthread_cond_wait (cond=0x3aefea8, mutex=0x3aefe80) at pthread_cond_wait.c:655
            #3  0x0000000000cefec0 in os_event::wait() ()
            #4  0x0000000000cef9ef in os_event::wait_low(long) ()
            #5  0x0000000000cefe0d in os_event_wait_low(os_event*, long) ()
            #6  0x0000000000db6537 in srv_resume_thread(srv_slot_t*, long, bool, unsigned long) ()
            #7  0x0000000000db6742 in srv_worker_thread ()
            #8  0x00007ff8dac647fc in start_thread (arg=0x7ff89bfff700) at pthread_create.c:465
            #9  0x00007ff8d8890b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
             
            Thread 20 (Thread 0x7ff8a0ff9700 (LWP 52168)):
            #0  0x00007ff8dac6b072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x3aeffd0) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
            #1  __pthread_cond_wait_common (abstime=0x0, mutex=0x3aeff80, cond=0x3aeffa8) at pthread_cond_wait.c:502
            #2  __pthread_cond_wait (cond=0x3aeffa8, mutex=0x3aeff80) at pthread_cond_wait.c:655
            #3  0x0000000000cefec0 in os_event::wait() ()
            #4  0x0000000000cef9ef in os_event::wait_low(long) ()
            #5  0x0000000000cefe0d in os_event_wait_low(os_event*, long) ()
            #6  0x0000000000db6537 in srv_resume_thread(srv_slot_t*, long, bool, unsigned long) ()
            #7  0x0000000000db6742 in srv_worker_thread ()
            #8  0x00007ff8dac647fc in start_thread (arg=0x7ff8a0ff9700) at pthread_create.c:465
            #9  0x00007ff8d8890b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
             
            Thread 19 (Thread 0x7ff898ff9700 (LWP 52192)):
            #0  0x00007ff8dac6b786 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7ff898ff8bf0, expected=0, futex_word=0x3ca2954) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
            #1  __pthread_cond_wait_common (abstime=0x7ff898ff8bf0, mutex=0x3ca2900, cond=0x3ca2928) at pthread_cond_wait.c:539
            #2  __pthread_cond_timedwait (cond=0x3ca2928, mutex=0x3ca2900, abstime=0x7ff898ff8bf0) at pthread_cond_wait.c:667
            #3  0x0000000000cef76d in os_event::timed_wait(timespec const*) ()
            #4  0x0000000000cefb8b in os_event::wait_time_low(unsigned long, long) ()
            #5  0x0000000000cefde5 in os_event_wait_time_low(os_event*, unsigned long, long) ()
            #6  0x0000000000eda1f3 in fil_crypt_thread ()
            #7  0x00007ff8dac647fc in start_thread (arg=0x7ff898ff9700) at pthread_create.c:465
            #8  0x00007ff8d8890b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
             
            Thread 18 (Thread 0x7ff8c9ffb700 (LWP 52134)):
            #0  0x00007ff8dac6b072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x3b71c54) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
            #1  __pthread_cond_wait_common (abstime=0x0, mutex=0x3b71c00, cond=0x3b71c28) at pthread_cond_wait.c:502
            #2  __pthread_cond_wait (cond=0x3b71c28, mutex=0x3b71c00) at pthread_cond_wait.c:655
            #3  0x0000000000cefec0 in os_event::wait() ()
            #4  0x0000000000cef9ef in os_event::wait_low(long) ()
            #5  0x0000000000cefe0d in os_event_wait_low(os_event*, long) ()
            #6  0x0000000000ce7789 in os_aio_simulated_handler(unsigned long, fil_node_t**, void**, IORequest*) ()
            #7  0x0000000000ce74be in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) ()
            #8  0x0000000000ed1863 in fil_aio_wait(unsigned long) ()
            #9  0x0000000000db9147 in io_handler_thread ()
            #10 0x00007ff8dac647fc in start_thread (arg=0x7ff8c9ffb700) at pthread_create.c:465
            #11 0x00007ff8d8890b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
             
            Thread 17 (Thread 0x7ff8cbfff700 (LWP 52130)):
            #0  0x00007ff8dac6b072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x3b71a50) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
            #1  __pthread_cond_wait_common (abstime=0x0, mutex=0x3b71a00, cond=0x3b71a28) at pthread_cond_wait.c:502
            #2  __pthread_cond_wait (cond=0x3b71a28, mutex=0x3b71a00) at pthread_cond_wait.c:655
            #3  0x0000000000cefec0 in os_event::wait() ()
            #4  0x0000000000cef9ef in os_event::wait_low(long) ()
            #5  0x0000000000cefe0d in os_event_wait_low(os_event*, long) ()
            #6  0x0000000000ce7789 in os_aio_simulated_handler(unsigned long, fil_node_t**, void**, IORequest*) ()
            #7  0x0000000000ce74be in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) ()
            #8  0x0000000000ed1863 in fil_aio_wait(unsigned long) ()
            #9  0x0000000000db9147 in io_handler_thread ()
            #10 0x00007ff8dac647fc in start_thread (arg=0x7ff8cbfff700) at pthread_create.c:465
            #11 0x00007ff8d8890b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
             
            Thread 16 (Thread 0x7ff8cb7fe700 (LWP 52131)):
            #0  0x00007ff8dac6b072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x3b71ad0) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
            #1  __pthread_cond_wait_common (abstime=0x0, mutex=0x3b71a80, cond=0x3b71aa8) at pthread_cond_wait.c:502
            #2  __pthread_cond_wait (cond=0x3b71aa8, mutex=0x3b71a80) at pthread_cond_wait.c:655
            #3  0x0000000000cefec0 in os_event::wait() ()
            #4  0x0000000000cef9ef in os_event::wait_low(long) ()
            #5  0x0000000000cefe0d in os_event_wait_low(os_event*, long) ()
            #6  0x0000000000ce7789 in os_aio_simulated_handler(unsigned long, fil_node_t**, void**, IORequest*) ()
            #7  0x0000000000ce74be in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) ()
            #8  0x0000000000ed1863 in fil_aio_wait(unsigned long) ()
            #9  0x0000000000db9147 in io_handler_thread ()
            #10 0x00007ff8dac647fc in start_thread (arg=0x7ff8cb7fe700) at pthread_create.c:465
            #11 0x00007ff8d8890b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
             
            Thread 15 (Thread 0x7ff87ffff700 (LWP 52193)):
            #0  0x00007ff8dac6f5f8 in __GI___nanosleep (requested_time=0x7ff87fffe438, remaining=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:27
            #1  0x0000000000cf0458 in os_thread_sleep(unsigned long) ()
            #2  0x0000000000e449f5 in btr_defragment_thread ()
            #3  0x00007ff8dac647fc in start_thread (arg=0x7ff87ffff700) at pthread_create.c:465
            #4  0x00007ff8d8890b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
             
            Thread 14 (Thread 0x7ff8997fa700 (LWP 52190)):
            #0  0x00007ff8dac6b786 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7ff8997f9bf0, expected=0, futex_word=0x3ca2954) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
            #1  __pthread_cond_wait_common (abstime=0x7ff8997f9bf0, mutex=0x3ca2900, cond=0x3ca2928) at pthread_cond_wait.c:539
            #2  __pthread_cond_timedwait (cond=0x3ca2928, mutex=0x3ca2900, abstime=0x7ff8997f9bf0) at pthread_cond_wait.c:667
            #3  0x0000000000cef76d in os_event::timed_wait(timespec const*) ()
            #4  0x0000000000cefb8b in os_event::wait_time_low(unsigned long, long) ()
            #5  0x0000000000cefde5 in os_event_wait_time_low(os_event*, unsigned long, long) ()
            #6  0x0000000000eda1f3 in fil_crypt_thread ()
            #7  0x00007ff8dac647fc in start_thread (arg=0x7ff8997fa700) at pthread_create.c:465
            #8  0x00007ff8d8890b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
             
            Thread 13 (Thread 0x7ff899ffb700 (LWP 52189)):
            #0  0x00007ff8dac6b786 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7ff899ffabf0, expected=0, futex_word=0x3ca2954) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
            #1  __pthread_cond_wait_common (abstime=0x7ff899ffabf0, mutex=0x3ca2900, cond=0x3ca2928) at pthread_cond_wait.c:539
            #2  __pthread_cond_timedwait (cond=0x3ca2928, mutex=0x3ca2900, abstime=0x7ff899ffabf0) at pthread_cond_wait.c:667
            #3  0x0000000000cef76d in os_event::timed_wait(timespec const*) ()
            #4  0x0000000000cefb8b in os_event::wait_time_low(unsigned long, long) ()
            #5  0x0000000000cefde5 in os_event_wait_time_low(os_event*, unsigned long, long) ()
            #6  0x0000000000eda1f3 in fil_crypt_thread ()
            #7  0x00007ff8dac647fc in start_thread (arg=0x7ff899ffb700) at pthread_create.c:465
            #8  0x00007ff8d8890b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
             
            Thread 12 (Thread 0x7ff8ca7fc700 (LWP 52133)):
            #0  0x00007ff8dac6b072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x3b71bd0) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
            #1  __pthread_cond_wait_common (abstime=0x0, mutex=0x3b71b80, cond=0x3b71ba8) at pthread_cond_wait.c:502
            #2  __pthread_cond_wait (cond=0x3b71ba8, mutex=0x3b71b80) at pthread_cond_wait.c:655
            #3  0x0000000000cefec0 in os_event::wait() ()
            #4  0x0000000000cef9ef in os_event::wait_low(long) ()
            #5  0x0000000000cefe0d in os_event_wait_low(os_event*, long) ()
            #6  0x0000000000ce7789 in os_aio_simulated_handler(unsigned long, fil_node_t**, void**, IORequest*) ()
            #7  0x0000000000ce74be in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) ()
            #8  0x0000000000ed1863 in fil_aio_wait(unsigned long) ()
            #9  0x0000000000db9147 in io_handler_thread ()
            #10 0x00007ff8dac647fc in start_thread (arg=0x7ff8ca7fc700) at pthread_create.c:465
            #11 0x00007ff8d8890b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
             
            Thread 11 (Thread 0x7ff89a7fc700 (LWP 52188)):
            #0  0x00007ff8dac6b786 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7ff89a7fbbf0, expected=0, futex_word=0x3ca2954) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
            #1  __pthread_cond_wait_common (abstime=0x7ff89a7fbbf0, mutex=0x3ca2900, cond=0x3ca2928) at pthread_cond_wait.c:539
            #2  __pthread_cond_timedwait (cond=0x3ca2928, mutex=0x3ca2900, abstime=0x7ff89a7fbbf0) at pthread_cond_wait.c:667
            #3  0x0000000000cef76d in os_event::timed_wait(timespec const*) ()
            #4  0x0000000000cefb8b in os_event::wait_time_low(unsigned long, long) ()
            #5  0x0000000000cefde5 in os_event_wait_time_low(os_event*, unsigned long, long) ()
            #6  0x0000000000eda1f3 in fil_crypt_thread ()
            #7  0x00007ff8dac647fc in start_thread (arg=0x7ff89a7fc700) at pthread_create.c:465
            #8  0x00007ff8d8890b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
             
            Thread 10 (Thread 0x7ff8caffd700 (LWP 52132)):
            #0  0x00007ff8dac6b072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x3b71b50) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
            #1  __pthread_cond_wait_common (abstime=0x0, mutex=0x3b71b00, cond=0x3b71b28) at pthread_cond_wait.c:502
            #2  __pthread_cond_wait (cond=0x3b71b28, mutex=0x3b71b00) at pthread_cond_wait.c:655
            #3  0x0000000000cefec0 in os_event::wait() ()
            #4  0x0000000000cef9ef in os_event::wait_low(long) ()
            #5  0x0000000000cefe0d in os_event_wait_low(os_event*, long) ()
            #6  0x0000000000ce7789 in os_aio_simulated_handler(unsigned long, fil_node_t**, void**, IORequest*) ()
            #7  0x0000000000ce74be in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) ()
            #8  0x0000000000ed1863 in fil_aio_wait(unsigned long) ()
            #9  0x0000000000db9147 in io_handler_thread ()
            #10 0x00007ff8dac647fc in start_thread (arg=0x7ff8caffd700) at pthread_create.c:465
            #11 0x00007ff8d8890b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
             
            Thread 9 (Thread 0x7ff8a2ffd700 (LWP 52154)):
            #0  0x00007ff8dac6b786 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7ff8a2ffccb0, expected=0, futex_word=0x3af00d0) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
            #1  __pthread_cond_wait_common (abstime=0x7ff8a2ffccb0, mutex=0x3af0080, cond=0x3af00a8) at pthread_cond_wait.c:539
            #2  __pthread_cond_timedwait (cond=0x3af00a8, mutex=0x3af0080, abstime=0x7ff8a2ffccb0) at pthread_cond_wait.c:667
            #3  0x0000000000cef76d in os_event::timed_wait(timespec const*) ()
            #4  0x0000000000cefb8b in os_event::wait_time_low(unsigned long, long) ()
            #5  0x0000000000cefde5 in os_event_wait_time_low(os_event*, unsigned long, long) ()
            #6  0x0000000000db4bed in srv_monitor_thread ()
            #7  0x00007ff8dac647fc in start_thread (arg=0x7ff8a2ffd700) at pthread_create.c:465
            #8  0x00007ff8d8890b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
             
            Thread 8 (Thread 0x7ff8c8df6700 (LWP 52166)):
            #0  0x00007ff8dac6f5f8 in __GI___nanosleep (requested_time=0x7ff8c8df5cd8, remaining=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:27
            #1  0x0000000000cf0458 in os_thread_sleep(unsigned long) ()
            #2  0x0000000000db5c22 in srv_master_sleep() ()
            #3  0x0000000000db5856 in srv_master_thread ()
            #4  0x00007ff8dac647fc in start_thread (arg=0x7ff8c8df6700) at pthread_create.c:465
            #5  0x00007ff8d8890b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
             
            Thread 7 (Thread 0x7ff8a3fff700 (LWP 52152)):
            #0  0x00007ff8dac6b786 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7ff8a3ffecd0, expected=0, futex_word=0x3cc4180) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
            #1  __pthread_cond_wait_common (abstime=0x7ff8a3ffecd0, mutex=0x3cc4130, cond=0x3cc4158) at pthread_cond_wait.c:539
            #2  __pthread_cond_timedwait (cond=0x3cc4158, mutex=0x3cc4130, abstime=0x7ff8a3ffecd0) at pthread_cond_wait.c:667
            #3  0x0000000000cef76d in os_event::timed_wait(timespec const*) ()
            #4  0x0000000000cefb8b in os_event::wait_time_low(unsigned long, long) ()
            #5  0x0000000000cefde5 in os_event_wait_time_low(os_event*, unsigned long, long) ()
            #6  0x0000000000cc06d6 in lock_wait_timeout_thread ()
            #7  0x00007ff8dac647fc in start_thread (arg=0x7ff8a3fff700) at pthread_create.c:465
            #8  0x00007ff8d8890b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
             
            Thread 6 (Thread 0x7ff8d09e2700 (LWP 52129)):
            #0  0x00007ff8dac6b072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x3b719d0) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
            #1  __pthread_cond_wait_common (abstime=0x0, mutex=0x3b71980, cond=0x3b719a8) at pthread_cond_wait.c:502
            #2  __pthread_cond_wait (cond=0x3b719a8, mutex=0x3b71980) at pthread_cond_wait.c:655
            #3  0x0000000000cefec0 in os_event::wait() ()
            #4  0x0000000000cef9ef in os_event::wait_low(long) ()
            #5  0x0000000000cefe0d in os_event_wait_low(os_event*, long) ()
            #6  0x0000000000ce7789 in os_aio_simulated_handler(unsigned long, fil_node_t**, void**, IORequest*) ()
            #7  0x0000000000ce74be in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) ()
            #8  0x0000000000ed1863 in fil_aio_wait(unsigned long) ()
            #9  0x0000000000db9147 in io_handler_thread ()
            #10 0x00007ff8dac647fc in start_thread (arg=0x7ff8d09e2700) at pthread_create.c:465
            #11 0x00007ff8d8890b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
             
            Thread 5 (Thread 0x7ff8d37bd700 (LWP 50452)):
            #0  0x00007ff8dac6b786 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7ff8d37bcce0, expected=0, futex_word=0x239cbf8 <COND_timer+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
            #1  __pthread_cond_wait_common (abstime=0x7ff8d37bcce0, mutex=0x239cba0 <LOCK_timer>, cond=0x239cbd0 <COND_timer>) at pthread_cond_wait.c:539
            #2  __pthread_cond_timedwait (cond=0x239cbd0 <COND_timer>, mutex=0x239cba0 <LOCK_timer>, abstime=0x7ff8d37bcce0) at pthread_cond_wait.c:667
            #3  0x000000000110c4e4 in inline_mysql_cond_timedwait ()
            #4  0x000000000110bd26 in timer_handler ()
            #5  0x00000000010a8d45 in pfs_spawn_thread ()
            #6  0x00007ff8dac647fc in start_thread (arg=0x7ff8d37bd700) at pthread_create.c:465
            #7  0x00007ff8d8890b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
             
            Thread 4 (Thread 0x7ff8a37fe700 (LWP 52153)):
            #0  0x00007ff8dac6b786 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7ff8a37fd930, expected=0, futex_word=0x3af0050) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
            #1  __pthread_cond_wait_common (abstime=0x7ff8a37fd930, mutex=0x3af0000, cond=0x3af0028) at pthread_cond_wait.c:539
            #2  __pthread_cond_timedwait (cond=0x3af0028, mutex=0x3af0000, abstime=0x7ff8a37fd930) at pthread_cond_wait.c:667
            #3  0x0000000000cef76d in os_event::timed_wait(timespec const*) ()
            #4  0x0000000000cefb8b in os_event::wait_time_low(unsigned long, long) ()
            #5  0x0000000000cefde5 in os_event_wait_time_low(os_event*, unsigned long, long) ()
            #6  0x0000000000db5329 in srv_error_monitor_thread ()
            #7  0x00007ff8dac647fc in start_thread (arg=0x7ff8a37fe700) at pthread_create.c:465
            #8  0x00007ff8d8890b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
             
            Thread 3 (Thread 0x7ff8c97fa700 (LWP 52135)):
            #0  0x00007ff8dac6b786 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7ff8c97f9680, expected=0, futex_word=0x3af01d4) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
            #1  __pthread_cond_wait_common (abstime=0x7ff8c97f9680, mutex=0x3af0180, cond=0x3af01a8) at pthread_cond_wait.c:539
            #2  __pthread_cond_timedwait (cond=0x3af01a8, mutex=0x3af0180, abstime=0x7ff8c97f9680) at pthread_cond_wait.c:667
            #3  0x0000000000cef76d in os_event::timed_wait(timespec const*) ()
            #4  0x0000000000cefb8b in os_event::wait_time_low(unsigned long, long) ()
            #5  0x0000000000cefde5 in os_event_wait_time_low(os_event*, unsigned long, long) ()
            #6  0x0000000000e6a6f1 in pc_sleep_if_needed(unsigned long, long, unsigned long) ()
            #7  0x0000000000e6921b in buf_flush_page_cleaner_coordinator ()
            #8  0x00007ff8dac647fc in start_thread (arg=0x7ff8c97fa700) at pthread_create.c:465
            #9  0x00007ff8d8890b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
             
            Thread 2 (Thread 0x7ff8d2244700 (LWP 50457)):
            #0  0x00007ff8dac6b786 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7ff8d2243c48, expected=0, futex_word=0x238d108 <COND_checkpoint+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
            #1  __pthread_cond_wait_common (abstime=0x7ff8d2243c48, mutex=0x238d0a8 <LOCK_checkpoint>, cond=0x238d0e0 <COND_checkpoint>) at pthread_cond_wait.c:539
            #2  __pthread_cond_timedwait (cond=0x238d0e0 <COND_checkpoint>, mutex=0x238d0a8 <LOCK_checkpoint>, abstime=0x7ff8d2243c48) at pthread_cond_wait.c:667
            #3  0x0000000000f8fb52 in inline_mysql_cond_timedwait ()
            #4  0x0000000000f8faa8 in my_service_thread_sleep ()
            #5  0x0000000000f8496e in ma_checkpoint_background ()
            #6  0x00000000010a8d45 in pfs_spawn_thread ()
            #7  0x00007ff8dac647fc in start_thread (arg=0x7ff8d2244700) at pthread_create.c:465
            #8  0x00007ff8d8890b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
             
            Thread 1 (Thread 0x7ff8db08ed40 (LWP 49983)):
            #0  __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:57
            #1  0x000000000110698b in my_write_core ()
            #2  0x0000000000a29f5a in handle_fatal_signal ()
            #3  <signal handler called>
            #4  0x00007ff8dac6b072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x1b0d2a0 <COND_thread_count+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
            #5  __pthread_cond_wait_common (abstime=0x0, mutex=0x1b0cc98 <LOCK_thread_count>, cond=0x1b0d278 <COND_thread_count>) at pthread_cond_wait.c:502
            #6  __pthread_cond_wait (cond=0x1b0d278 <COND_thread_count>, mutex=0x1b0cc98 <LOCK_thread_count>) at pthread_cond_wait.c:655
            #7  0x000000000061ede6 in inline_mysql_cond_wait(st_mysql_cond*, st_mysql_mutex*, char const*, unsigned int) ()
            #8  0x0000000000620dea in mysqld_main(int, char**) ()
            #9  0x000000000061cfa2 in main ()
             
            Compressed file /home/dan/build-mariadb-10.3_lastest_upstream_clang-6/mysql-test/var/log/encryption.innodb-compressed-blob-8k,cbc,innodb,none/mysqld.1/data/core
            

            danblack Daniel Black added a comment - clang-6.0, 10.3-0f6186c5509309c5d647f9631f04071a62a013df clang-6.0 --version clang version 6.0.1-svn329487-1~exp1~20180408152107.70 (branches/release_60)   encryption.innodb-compressed-blob '8k,cbc,innodb,none' w60 [ fail ] Found warnings/errors in server log file! Test ended at 2018-04-08 19:19:12 line Attempting backtrace. You can use the following information to find out ^ Found warnings in /home/dan/build-mariadb-10.3_lastest_upstream_clang-6/mysql-test/var/60/log/mysqld.1.err ok   - saving '/home/dan/build-mariadb-10.3_lastest_upstream_clang-6/mysql-test/var/60/log/encryption.innodb-compressed-blob-8k,cbc,innodb,none/' to '/home/dan/build-mariadb-10.3_lastest_upstream_clang-6/mysql-test/var/log/encryption.innodb-compressed-blob-8k,cbc,innodb,none/' - found 'core' (0/5)   Trying 'dbx' to get a backtrace   Trying 'gdb' to get a backtrace from coredump /home/dan/build-mariadb-10.3_lastest_upstream_clang-6/mysql-test/var/log/encryption.innodb-compressed-blob-8k,cbc,innodb,none/mysqld.1/data/core Core generated by '/home/dan/build-mariadb-10.3_lastest_upstream_clang-6/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 49983] [New LWP 50457] [New LWP 52135] [New LWP 52153] [New LWP 50452] [New LWP 52129] [New LWP 52152] [New LWP 52166] [New LWP 52154] [New LWP 52132] [New LWP 52188] [New LWP 52133] [New LWP 52189] [New LWP 52190] [New LWP 52193] [New LWP 52131] [New LWP 52130] [New LWP 52134] [New LWP 52192] [New LWP 52168] [New LWP 52169] [New LWP 52187] [New LWP 52194] [New LWP 52207] [New LWP 52241] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `/home/dan/build-mariadb-10.3_lastest_upstream_clang-6/sql/mysqld --defaults-gro'. Program terminated with signal SIGABRT, Aborted. #0 __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:57 57 ../sysdeps/unix/sysv/linux/pthread_kill.c: No such file or directory. [Current thread is 1 (Thread 0x7ff8db08ed40 (LWP 49983))] #0 __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:57 #1 0x000000000110698b in my_write_core () #2 0x0000000000a29f5a in handle_fatal_signal () #3 <signal handler called> #4 0x00007ff8dac6b072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x1b0d2a0 <COND_thread_count+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88 #5 __pthread_cond_wait_common (abstime=0x0, mutex=0x1b0cc98 <LOCK_thread_count>, cond=0x1b0d278 <COND_thread_count>) at pthread_cond_wait.c:502 #6 __pthread_cond_wait (cond=0x1b0d278 <COND_thread_count>, mutex=0x1b0cc98 <LOCK_thread_count>) at pthread_cond_wait.c:655 #7 0x000000000061ede6 in inline_mysql_cond_wait(st_mysql_cond*, st_mysql_mutex*, char const*, unsigned int) () #8 0x0000000000620dea in mysqld_main(int, char**) () #9 0x000000000061cfa2 in main ()   Thread 25 (Thread 0x7ff8d0b04700 (LWP 52241)): #0 0x00007ff8dac6b072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x1b0d2a0 <COND_thread_count+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88 #1 __pthread_cond_wait_common (abstime=0x0, mutex=0x1b0cc98 <LOCK_thread_count>, cond=0x1b0d278 <COND_thread_count>) at pthread_cond_wait.c:502 #2 __pthread_cond_wait (cond=0x1b0d278 <COND_thread_count>, mutex=0x1b0cc98 <LOCK_thread_count>) at pthread_cond_wait.c:655 #3 0x000000000061ede6 in inline_mysql_cond_wait(st_mysql_cond*, st_mysql_mutex*, char const*, unsigned int) () #4 0x000000000062a304 in close_connections() () #5 0x000000000061daf0 in kill_server(void*) () #6 0x000000000061d8cd in kill_server_thread () #7 0x00000000010a8d45 in pfs_spawn_thread () #8 0x00007ff8dac647fc in start_thread (arg=0x7ff8d0b04700) at pthread_create.c:465 #9 0x00007ff8d8890b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95   Thread 24 (Thread 0x7ff8d0be2700 (LWP 52207)): #0 do_sigwait (sig=0x7ff8d0be1c8c, set=<optimized out>) at ../sysdeps/unix/sysv/linux/sigwait.c:64 #1 __sigwait (set=0x7ff8d0be1c90, sig=0x7ff8d0be1c8c) at ../sysdeps/unix/sysv/linux/sigwait.c:96 #2 0x000000000061f560 in signal_hand () #3 0x00000000010a8d45 in pfs_spawn_thread () #4 0x00007ff8dac647fc in start_thread (arg=0x7ff8d0be2700) at pthread_create.c:465 #5 0x00007ff8d8890b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95   Thread 23 (Thread 0x7ff87f7fe700 (LWP 52194)): #0 0x00007ff8dac6b072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x3af0250) at ../sysdeps/unix/sysv/linux/futex-internal.h:88 #1 __pthread_cond_wait_common (abstime=0x0, mutex=0x3af0200, cond=0x3af0228) at pthread_cond_wait.c:502 #2 __pthread_cond_wait (cond=0x3af0228, mutex=0x3af0200) at pthread_cond_wait.c:655 #3 0x0000000000cefec0 in os_event::wait() () #4 0x0000000000cef9ef in os_event::wait_low(long) () #5 0x0000000000cefe0d in os_event_wait_low(os_event*, long) () #6 0x0000000000e4abe9 in buf_resize_thread () #7 0x00007ff8dac647fc in start_thread (arg=0x7ff87f7fe700) at pthread_create.c:465 #8 0x00007ff8d8890b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95   Thread 22 (Thread 0x7ff89affd700 (LWP 52187)): #0 0x00007ff8dac6b072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x3af0150) at ../sysdeps/unix/sysv/linux/futex-internal.h:88 #1 __pthread_cond_wait_common (abstime=0x0, mutex=0x3af0100, cond=0x3af0128) at pthread_cond_wait.c:502 #2 __pthread_cond_wait (cond=0x3af0128, mutex=0x3af0100) at pthread_cond_wait.c:655 #3 0x0000000000cefec0 in os_event::wait() () #4 0x0000000000cef9ef in os_event::wait_low(long) () #5 0x0000000000cefe0d in os_event_wait_low(os_event*, long) () #6 0x0000000000e63aef in buf_dump_thread () #7 0x00007ff8dac647fc in start_thread (arg=0x7ff89affd700) at pthread_create.c:465 #8 0x00007ff8d8890b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95   Thread 21 (Thread 0x7ff89bfff700 (LWP 52169)): #0 0x00007ff8dac6b072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x3aefed4) at ../sysdeps/unix/sysv/linux/futex-internal.h:88 #1 __pthread_cond_wait_common (abstime=0x0, mutex=0x3aefe80, cond=0x3aefea8) at pthread_cond_wait.c:502 #2 __pthread_cond_wait (cond=0x3aefea8, mutex=0x3aefe80) at pthread_cond_wait.c:655 #3 0x0000000000cefec0 in os_event::wait() () #4 0x0000000000cef9ef in os_event::wait_low(long) () #5 0x0000000000cefe0d in os_event_wait_low(os_event*, long) () #6 0x0000000000db6537 in srv_resume_thread(srv_slot_t*, long, bool, unsigned long) () #7 0x0000000000db6742 in srv_worker_thread () #8 0x00007ff8dac647fc in start_thread (arg=0x7ff89bfff700) at pthread_create.c:465 #9 0x00007ff8d8890b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95   Thread 20 (Thread 0x7ff8a0ff9700 (LWP 52168)): #0 0x00007ff8dac6b072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x3aeffd0) at ../sysdeps/unix/sysv/linux/futex-internal.h:88 #1 __pthread_cond_wait_common (abstime=0x0, mutex=0x3aeff80, cond=0x3aeffa8) at pthread_cond_wait.c:502 #2 __pthread_cond_wait (cond=0x3aeffa8, mutex=0x3aeff80) at pthread_cond_wait.c:655 #3 0x0000000000cefec0 in os_event::wait() () #4 0x0000000000cef9ef in os_event::wait_low(long) () #5 0x0000000000cefe0d in os_event_wait_low(os_event*, long) () #6 0x0000000000db6537 in srv_resume_thread(srv_slot_t*, long, bool, unsigned long) () #7 0x0000000000db6742 in srv_worker_thread () #8 0x00007ff8dac647fc in start_thread (arg=0x7ff8a0ff9700) at pthread_create.c:465 #9 0x00007ff8d8890b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95   Thread 19 (Thread 0x7ff898ff9700 (LWP 52192)): #0 0x00007ff8dac6b786 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7ff898ff8bf0, expected=0, futex_word=0x3ca2954) at ../sysdeps/unix/sysv/linux/futex-internal.h:205 #1 __pthread_cond_wait_common (abstime=0x7ff898ff8bf0, mutex=0x3ca2900, cond=0x3ca2928) at pthread_cond_wait.c:539 #2 __pthread_cond_timedwait (cond=0x3ca2928, mutex=0x3ca2900, abstime=0x7ff898ff8bf0) at pthread_cond_wait.c:667 #3 0x0000000000cef76d in os_event::timed_wait(timespec const*) () #4 0x0000000000cefb8b in os_event::wait_time_low(unsigned long, long) () #5 0x0000000000cefde5 in os_event_wait_time_low(os_event*, unsigned long, long) () #6 0x0000000000eda1f3 in fil_crypt_thread () #7 0x00007ff8dac647fc in start_thread (arg=0x7ff898ff9700) at pthread_create.c:465 #8 0x00007ff8d8890b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95   Thread 18 (Thread 0x7ff8c9ffb700 (LWP 52134)): #0 0x00007ff8dac6b072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x3b71c54) at ../sysdeps/unix/sysv/linux/futex-internal.h:88 #1 __pthread_cond_wait_common (abstime=0x0, mutex=0x3b71c00, cond=0x3b71c28) at pthread_cond_wait.c:502 #2 __pthread_cond_wait (cond=0x3b71c28, mutex=0x3b71c00) at pthread_cond_wait.c:655 #3 0x0000000000cefec0 in os_event::wait() () #4 0x0000000000cef9ef in os_event::wait_low(long) () #5 0x0000000000cefe0d in os_event_wait_low(os_event*, long) () #6 0x0000000000ce7789 in os_aio_simulated_handler(unsigned long, fil_node_t**, void**, IORequest*) () #7 0x0000000000ce74be in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) () #8 0x0000000000ed1863 in fil_aio_wait(unsigned long) () #9 0x0000000000db9147 in io_handler_thread () #10 0x00007ff8dac647fc in start_thread (arg=0x7ff8c9ffb700) at pthread_create.c:465 #11 0x00007ff8d8890b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95   Thread 17 (Thread 0x7ff8cbfff700 (LWP 52130)): #0 0x00007ff8dac6b072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x3b71a50) at ../sysdeps/unix/sysv/linux/futex-internal.h:88 #1 __pthread_cond_wait_common (abstime=0x0, mutex=0x3b71a00, cond=0x3b71a28) at pthread_cond_wait.c:502 #2 __pthread_cond_wait (cond=0x3b71a28, mutex=0x3b71a00) at pthread_cond_wait.c:655 #3 0x0000000000cefec0 in os_event::wait() () #4 0x0000000000cef9ef in os_event::wait_low(long) () #5 0x0000000000cefe0d in os_event_wait_low(os_event*, long) () #6 0x0000000000ce7789 in os_aio_simulated_handler(unsigned long, fil_node_t**, void**, IORequest*) () #7 0x0000000000ce74be in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) () #8 0x0000000000ed1863 in fil_aio_wait(unsigned long) () #9 0x0000000000db9147 in io_handler_thread () #10 0x00007ff8dac647fc in start_thread (arg=0x7ff8cbfff700) at pthread_create.c:465 #11 0x00007ff8d8890b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95   Thread 16 (Thread 0x7ff8cb7fe700 (LWP 52131)): #0 0x00007ff8dac6b072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x3b71ad0) at ../sysdeps/unix/sysv/linux/futex-internal.h:88 #1 __pthread_cond_wait_common (abstime=0x0, mutex=0x3b71a80, cond=0x3b71aa8) at pthread_cond_wait.c:502 #2 __pthread_cond_wait (cond=0x3b71aa8, mutex=0x3b71a80) at pthread_cond_wait.c:655 #3 0x0000000000cefec0 in os_event::wait() () #4 0x0000000000cef9ef in os_event::wait_low(long) () #5 0x0000000000cefe0d in os_event_wait_low(os_event*, long) () #6 0x0000000000ce7789 in os_aio_simulated_handler(unsigned long, fil_node_t**, void**, IORequest*) () #7 0x0000000000ce74be in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) () #8 0x0000000000ed1863 in fil_aio_wait(unsigned long) () #9 0x0000000000db9147 in io_handler_thread () #10 0x00007ff8dac647fc in start_thread (arg=0x7ff8cb7fe700) at pthread_create.c:465 #11 0x00007ff8d8890b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95   Thread 15 (Thread 0x7ff87ffff700 (LWP 52193)): #0 0x00007ff8dac6f5f8 in __GI___nanosleep (requested_time=0x7ff87fffe438, remaining=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:27 #1 0x0000000000cf0458 in os_thread_sleep(unsigned long) () #2 0x0000000000e449f5 in btr_defragment_thread () #3 0x00007ff8dac647fc in start_thread (arg=0x7ff87ffff700) at pthread_create.c:465 #4 0x00007ff8d8890b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95   Thread 14 (Thread 0x7ff8997fa700 (LWP 52190)): #0 0x00007ff8dac6b786 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7ff8997f9bf0, expected=0, futex_word=0x3ca2954) at ../sysdeps/unix/sysv/linux/futex-internal.h:205 #1 __pthread_cond_wait_common (abstime=0x7ff8997f9bf0, mutex=0x3ca2900, cond=0x3ca2928) at pthread_cond_wait.c:539 #2 __pthread_cond_timedwait (cond=0x3ca2928, mutex=0x3ca2900, abstime=0x7ff8997f9bf0) at pthread_cond_wait.c:667 #3 0x0000000000cef76d in os_event::timed_wait(timespec const*) () #4 0x0000000000cefb8b in os_event::wait_time_low(unsigned long, long) () #5 0x0000000000cefde5 in os_event_wait_time_low(os_event*, unsigned long, long) () #6 0x0000000000eda1f3 in fil_crypt_thread () #7 0x00007ff8dac647fc in start_thread (arg=0x7ff8997fa700) at pthread_create.c:465 #8 0x00007ff8d8890b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95   Thread 13 (Thread 0x7ff899ffb700 (LWP 52189)): #0 0x00007ff8dac6b786 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7ff899ffabf0, expected=0, futex_word=0x3ca2954) at ../sysdeps/unix/sysv/linux/futex-internal.h:205 #1 __pthread_cond_wait_common (abstime=0x7ff899ffabf0, mutex=0x3ca2900, cond=0x3ca2928) at pthread_cond_wait.c:539 #2 __pthread_cond_timedwait (cond=0x3ca2928, mutex=0x3ca2900, abstime=0x7ff899ffabf0) at pthread_cond_wait.c:667 #3 0x0000000000cef76d in os_event::timed_wait(timespec const*) () #4 0x0000000000cefb8b in os_event::wait_time_low(unsigned long, long) () #5 0x0000000000cefde5 in os_event_wait_time_low(os_event*, unsigned long, long) () #6 0x0000000000eda1f3 in fil_crypt_thread () #7 0x00007ff8dac647fc in start_thread (arg=0x7ff899ffb700) at pthread_create.c:465 #8 0x00007ff8d8890b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95   Thread 12 (Thread 0x7ff8ca7fc700 (LWP 52133)): #0 0x00007ff8dac6b072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x3b71bd0) at ../sysdeps/unix/sysv/linux/futex-internal.h:88 #1 __pthread_cond_wait_common (abstime=0x0, mutex=0x3b71b80, cond=0x3b71ba8) at pthread_cond_wait.c:502 #2 __pthread_cond_wait (cond=0x3b71ba8, mutex=0x3b71b80) at pthread_cond_wait.c:655 #3 0x0000000000cefec0 in os_event::wait() () #4 0x0000000000cef9ef in os_event::wait_low(long) () #5 0x0000000000cefe0d in os_event_wait_low(os_event*, long) () #6 0x0000000000ce7789 in os_aio_simulated_handler(unsigned long, fil_node_t**, void**, IORequest*) () #7 0x0000000000ce74be in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) () #8 0x0000000000ed1863 in fil_aio_wait(unsigned long) () #9 0x0000000000db9147 in io_handler_thread () #10 0x00007ff8dac647fc in start_thread (arg=0x7ff8ca7fc700) at pthread_create.c:465 #11 0x00007ff8d8890b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95   Thread 11 (Thread 0x7ff89a7fc700 (LWP 52188)): #0 0x00007ff8dac6b786 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7ff89a7fbbf0, expected=0, futex_word=0x3ca2954) at ../sysdeps/unix/sysv/linux/futex-internal.h:205 #1 __pthread_cond_wait_common (abstime=0x7ff89a7fbbf0, mutex=0x3ca2900, cond=0x3ca2928) at pthread_cond_wait.c:539 #2 __pthread_cond_timedwait (cond=0x3ca2928, mutex=0x3ca2900, abstime=0x7ff89a7fbbf0) at pthread_cond_wait.c:667 #3 0x0000000000cef76d in os_event::timed_wait(timespec const*) () #4 0x0000000000cefb8b in os_event::wait_time_low(unsigned long, long) () #5 0x0000000000cefde5 in os_event_wait_time_low(os_event*, unsigned long, long) () #6 0x0000000000eda1f3 in fil_crypt_thread () #7 0x00007ff8dac647fc in start_thread (arg=0x7ff89a7fc700) at pthread_create.c:465 #8 0x00007ff8d8890b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95   Thread 10 (Thread 0x7ff8caffd700 (LWP 52132)): #0 0x00007ff8dac6b072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x3b71b50) at ../sysdeps/unix/sysv/linux/futex-internal.h:88 #1 __pthread_cond_wait_common (abstime=0x0, mutex=0x3b71b00, cond=0x3b71b28) at pthread_cond_wait.c:502 #2 __pthread_cond_wait (cond=0x3b71b28, mutex=0x3b71b00) at pthread_cond_wait.c:655 #3 0x0000000000cefec0 in os_event::wait() () #4 0x0000000000cef9ef in os_event::wait_low(long) () #5 0x0000000000cefe0d in os_event_wait_low(os_event*, long) () #6 0x0000000000ce7789 in os_aio_simulated_handler(unsigned long, fil_node_t**, void**, IORequest*) () #7 0x0000000000ce74be in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) () #8 0x0000000000ed1863 in fil_aio_wait(unsigned long) () #9 0x0000000000db9147 in io_handler_thread () #10 0x00007ff8dac647fc in start_thread (arg=0x7ff8caffd700) at pthread_create.c:465 #11 0x00007ff8d8890b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95   Thread 9 (Thread 0x7ff8a2ffd700 (LWP 52154)): #0 0x00007ff8dac6b786 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7ff8a2ffccb0, expected=0, futex_word=0x3af00d0) at ../sysdeps/unix/sysv/linux/futex-internal.h:205 #1 __pthread_cond_wait_common (abstime=0x7ff8a2ffccb0, mutex=0x3af0080, cond=0x3af00a8) at pthread_cond_wait.c:539 #2 __pthread_cond_timedwait (cond=0x3af00a8, mutex=0x3af0080, abstime=0x7ff8a2ffccb0) at pthread_cond_wait.c:667 #3 0x0000000000cef76d in os_event::timed_wait(timespec const*) () #4 0x0000000000cefb8b in os_event::wait_time_low(unsigned long, long) () #5 0x0000000000cefde5 in os_event_wait_time_low(os_event*, unsigned long, long) () #6 0x0000000000db4bed in srv_monitor_thread () #7 0x00007ff8dac647fc in start_thread (arg=0x7ff8a2ffd700) at pthread_create.c:465 #8 0x00007ff8d8890b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95   Thread 8 (Thread 0x7ff8c8df6700 (LWP 52166)): #0 0x00007ff8dac6f5f8 in __GI___nanosleep (requested_time=0x7ff8c8df5cd8, remaining=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:27 #1 0x0000000000cf0458 in os_thread_sleep(unsigned long) () #2 0x0000000000db5c22 in srv_master_sleep() () #3 0x0000000000db5856 in srv_master_thread () #4 0x00007ff8dac647fc in start_thread (arg=0x7ff8c8df6700) at pthread_create.c:465 #5 0x00007ff8d8890b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95   Thread 7 (Thread 0x7ff8a3fff700 (LWP 52152)): #0 0x00007ff8dac6b786 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7ff8a3ffecd0, expected=0, futex_word=0x3cc4180) at ../sysdeps/unix/sysv/linux/futex-internal.h:205 #1 __pthread_cond_wait_common (abstime=0x7ff8a3ffecd0, mutex=0x3cc4130, cond=0x3cc4158) at pthread_cond_wait.c:539 #2 __pthread_cond_timedwait (cond=0x3cc4158, mutex=0x3cc4130, abstime=0x7ff8a3ffecd0) at pthread_cond_wait.c:667 #3 0x0000000000cef76d in os_event::timed_wait(timespec const*) () #4 0x0000000000cefb8b in os_event::wait_time_low(unsigned long, long) () #5 0x0000000000cefde5 in os_event_wait_time_low(os_event*, unsigned long, long) () #6 0x0000000000cc06d6 in lock_wait_timeout_thread () #7 0x00007ff8dac647fc in start_thread (arg=0x7ff8a3fff700) at pthread_create.c:465 #8 0x00007ff8d8890b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95   Thread 6 (Thread 0x7ff8d09e2700 (LWP 52129)): #0 0x00007ff8dac6b072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x3b719d0) at ../sysdeps/unix/sysv/linux/futex-internal.h:88 #1 __pthread_cond_wait_common (abstime=0x0, mutex=0x3b71980, cond=0x3b719a8) at pthread_cond_wait.c:502 #2 __pthread_cond_wait (cond=0x3b719a8, mutex=0x3b71980) at pthread_cond_wait.c:655 #3 0x0000000000cefec0 in os_event::wait() () #4 0x0000000000cef9ef in os_event::wait_low(long) () #5 0x0000000000cefe0d in os_event_wait_low(os_event*, long) () #6 0x0000000000ce7789 in os_aio_simulated_handler(unsigned long, fil_node_t**, void**, IORequest*) () #7 0x0000000000ce74be in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) () #8 0x0000000000ed1863 in fil_aio_wait(unsigned long) () #9 0x0000000000db9147 in io_handler_thread () #10 0x00007ff8dac647fc in start_thread (arg=0x7ff8d09e2700) at pthread_create.c:465 #11 0x00007ff8d8890b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95   Thread 5 (Thread 0x7ff8d37bd700 (LWP 50452)): #0 0x00007ff8dac6b786 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7ff8d37bcce0, expected=0, futex_word=0x239cbf8 <COND_timer+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205 #1 __pthread_cond_wait_common (abstime=0x7ff8d37bcce0, mutex=0x239cba0 <LOCK_timer>, cond=0x239cbd0 <COND_timer>) at pthread_cond_wait.c:539 #2 __pthread_cond_timedwait (cond=0x239cbd0 <COND_timer>, mutex=0x239cba0 <LOCK_timer>, abstime=0x7ff8d37bcce0) at pthread_cond_wait.c:667 #3 0x000000000110c4e4 in inline_mysql_cond_timedwait () #4 0x000000000110bd26 in timer_handler () #5 0x00000000010a8d45 in pfs_spawn_thread () #6 0x00007ff8dac647fc in start_thread (arg=0x7ff8d37bd700) at pthread_create.c:465 #7 0x00007ff8d8890b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95   Thread 4 (Thread 0x7ff8a37fe700 (LWP 52153)): #0 0x00007ff8dac6b786 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7ff8a37fd930, expected=0, futex_word=0x3af0050) at ../sysdeps/unix/sysv/linux/futex-internal.h:205 #1 __pthread_cond_wait_common (abstime=0x7ff8a37fd930, mutex=0x3af0000, cond=0x3af0028) at pthread_cond_wait.c:539 #2 __pthread_cond_timedwait (cond=0x3af0028, mutex=0x3af0000, abstime=0x7ff8a37fd930) at pthread_cond_wait.c:667 #3 0x0000000000cef76d in os_event::timed_wait(timespec const*) () #4 0x0000000000cefb8b in os_event::wait_time_low(unsigned long, long) () #5 0x0000000000cefde5 in os_event_wait_time_low(os_event*, unsigned long, long) () #6 0x0000000000db5329 in srv_error_monitor_thread () #7 0x00007ff8dac647fc in start_thread (arg=0x7ff8a37fe700) at pthread_create.c:465 #8 0x00007ff8d8890b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95   Thread 3 (Thread 0x7ff8c97fa700 (LWP 52135)): #0 0x00007ff8dac6b786 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7ff8c97f9680, expected=0, futex_word=0x3af01d4) at ../sysdeps/unix/sysv/linux/futex-internal.h:205 #1 __pthread_cond_wait_common (abstime=0x7ff8c97f9680, mutex=0x3af0180, cond=0x3af01a8) at pthread_cond_wait.c:539 #2 __pthread_cond_timedwait (cond=0x3af01a8, mutex=0x3af0180, abstime=0x7ff8c97f9680) at pthread_cond_wait.c:667 #3 0x0000000000cef76d in os_event::timed_wait(timespec const*) () #4 0x0000000000cefb8b in os_event::wait_time_low(unsigned long, long) () #5 0x0000000000cefde5 in os_event_wait_time_low(os_event*, unsigned long, long) () #6 0x0000000000e6a6f1 in pc_sleep_if_needed(unsigned long, long, unsigned long) () #7 0x0000000000e6921b in buf_flush_page_cleaner_coordinator () #8 0x00007ff8dac647fc in start_thread (arg=0x7ff8c97fa700) at pthread_create.c:465 #9 0x00007ff8d8890b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95   Thread 2 (Thread 0x7ff8d2244700 (LWP 50457)): #0 0x00007ff8dac6b786 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7ff8d2243c48, expected=0, futex_word=0x238d108 <COND_checkpoint+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205 #1 __pthread_cond_wait_common (abstime=0x7ff8d2243c48, mutex=0x238d0a8 <LOCK_checkpoint>, cond=0x238d0e0 <COND_checkpoint>) at pthread_cond_wait.c:539 #2 __pthread_cond_timedwait (cond=0x238d0e0 <COND_checkpoint>, mutex=0x238d0a8 <LOCK_checkpoint>, abstime=0x7ff8d2243c48) at pthread_cond_wait.c:667 #3 0x0000000000f8fb52 in inline_mysql_cond_timedwait () #4 0x0000000000f8faa8 in my_service_thread_sleep () #5 0x0000000000f8496e in ma_checkpoint_background () #6 0x00000000010a8d45 in pfs_spawn_thread () #7 0x00007ff8dac647fc in start_thread (arg=0x7ff8d2244700) at pthread_create.c:465 #8 0x00007ff8d8890b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95   Thread 1 (Thread 0x7ff8db08ed40 (LWP 49983)): #0 __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:57 #1 0x000000000110698b in my_write_core () #2 0x0000000000a29f5a in handle_fatal_signal () #3 <signal handler called> #4 0x00007ff8dac6b072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x1b0d2a0 <COND_thread_count+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88 #5 __pthread_cond_wait_common (abstime=0x0, mutex=0x1b0cc98 <LOCK_thread_count>, cond=0x1b0d278 <COND_thread_count>) at pthread_cond_wait.c:502 #6 __pthread_cond_wait (cond=0x1b0d278 <COND_thread_count>, mutex=0x1b0cc98 <LOCK_thread_count>) at pthread_cond_wait.c:655 #7 0x000000000061ede6 in inline_mysql_cond_wait(st_mysql_cond*, st_mysql_mutex*, char const*, unsigned int) () #8 0x0000000000620dea in mysqld_main(int, char**) () #9 0x000000000061cfa2 in main ()   Compressed file /home/dan/build-mariadb-10.3_lastest_upstream_clang-6/mysql-test/var/log/encryption.innodb-compressed-blob-8k,cbc,innodb,none/mysqld.1/data/core

            danblack, does the server really crash, or was the SIGABRT sent externally by the mysql-test-run framework? (Before 10.3, we used to ignore such shutdown failures, presumably sending SIGKILL straight away.)
            Can you please look at the mysqld.1.err and check if the abort message was issued 60 seconds after a shutdown message?

            MDEV-13779 tracks InnoDB hang during shutdown. Maybe it happens more easily when using encryption. I do not have an easy way to reproduce it yet.

            marko Marko Mäkelä added a comment - danblack , does the server really crash, or was the SIGABRT sent externally by the mysql-test-run framework? (Before 10.3, we used to ignore such shutdown failures, presumably sending SIGKILL straight away.) Can you please look at the mysqld.1.err and check if the abort message was issued 60 seconds after a shutdown message? MDEV-13779 tracks InnoDB hang during shutdown. Maybe it happens more easily when using encryption. I do not have an easy way to reproduce it yet.
            Elkin Andrei Elkin added a comment -

            Rasmus, sorry, I don't think this deals with replication anyhow. The provided stacks
            do not say that certainly. And I could not actually identify the tread that hit an assert (that's my reading of the case). I suggest this one to be closely examined by core server
            specialists (Svoj?).

            Cheers,
            Andrei

            Elkin Andrei Elkin added a comment - Rasmus, sorry, I don't think this deals with replication anyhow. The provided stacks do not say that certainly. And I could not actually identify the tread that hit an assert (that's my reading of the case). I suggest this one to be closely examined by core server specialists (Svoj?). Cheers, Andrei

            People

              serg Sergei Golubchik
              danblack Daniel Black
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.