Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-24872

galera.galera_insert_multi MTR failed: crash with SIGABRT

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: In Progress (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 10.2.37
    • Fix Version/s: 10.2
    • Component/s: Galera, Tests
    • Labels:
      None
    • Environment:
      kvm-deb-stretch-amd64

      Description

      stdio.log:

      10.2.37, 95003eab451cac2fd2c0552f374a85842f2773f2, kvm-deb-stretch-amd64

      galera.galera_insert_multi 'innodb'      w1 [ fail ]  timeout after 900 seconds
              Test ended at 2021-02-11 09:34:26
       
      Test case timeout after 900 seconds
       
      == /dev/shm/var/1/log/galera_insert_multi.log == 
      COUNT(*) = 0
      1
      connection node_2;
      SELECT COUNT(*) = 0 FROM t1;
      COUNT(*) = 0
      1
      DROP TABLE t1;
      connection node_1;
      CREATE TABLE t1 (f1 INTEGER PRIMARY KEY) ENGINE=InnoDB;
      SET AUTOCOMMIT = OFF;
      START TRANSACTION;
      INSERT INTO t1 VALUES (1), (2);
      connection node_2;
      SET AUTOCOMMIT = OFF;
      START TRANSACTION;
      INSERT INTO t1 VALUES (2), (1);
      connection node_1;
      COMMIT;
      connection node_2;
      COMMIT;
       
       == /dev/shm/var/1/tmp/analyze-timeout-mysqld.1.err ==
      mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/dev/shm/var/tmp/1/mysqld.1.sock' (111)
       
       == /dev/shm/var/1/tmp/analyze-timeout-mysqld.2.err ==
      mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/dev/shm/var/tmp/1/mysqld.2.sock' (111)
       
       
       - found 'core' (0/0)
       
      Trying 'dbx' to get a backtrace
       
      Trying 'gdb' to get a backtrace from coredump /dev/shm/var/1/log/galera.galera_insert_multi-innodb/mysqld.2/data/core
      Core generated by '/usr/sbin/mysqld'
      Output from gdb follows. The first stack trace is from the failing thread.
      The following stack traces are from all threads (so the failing one is
      duplicated).
      --------------------------
      [New LWP 5218]
      [New LWP 5618]
      [New LWP 5220]
      [New LWP 5615]
      [New LWP 5627]
      [New LWP 5219]
      [New LWP 5619]
      [New LWP 5641]
      [New LWP 5355]
      [New LWP 5639]
      [New LWP 5357]
      [New LWP 5620]
      [New LWP 5637]
      [New LWP 5640]
      [New LWP 5632]
      [New LWP 5621]
      [New LWP 5636]
      [New LWP 5624]
      [New LWP 5617]
      [New LWP 5625]
      [New LWP 5356]
      [New LWP 5635]
      [New LWP 5221]
      [New LWP 5629]
      [New LWP 5616]
      [New LWP 5626]
      [New LWP 5628]
      [New LWP 5630]
      [New LWP 5631]
      [New LWP 5633]
      [New LWP 5634]
      [New LWP 5638]
      [Thread debugging using libthread_db enabled]
      Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
      Core was generated by `/usr/sbin/mysqld --defaults-group-suffix=.2 --defaults-file=/dev/shm/var/1/my.c'.
      Program terminated with signal SIGABRT, Aborted.
      #0  __pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:57
      57	../sysdeps/unix/sysv/linux/pthread_kill.c: No such file or directory.
      [Current thread is 1 (Thread 0x7f62461a4440 (LWP 5218))]
      #0  __pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:57
      #1  0x000055880329a517 in my_write_core (sig=sig@entry=6) at ./mysys/stacktrace.c:382
      #2  0x0000558802d50abb in handle_fatal_signal (sig=6) at ./sql/signal_handler.cc:343
      #3  <signal handler called>
      #4  0x00007f624434e8bd in poll () at ../sysdeps/unix/syscall-template.S:84
      #5  0x00007f62443683be in __poll_chk (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>, fdslen=<optimized out>) at poll_chk.c:27
      #6  0x0000558802b16cbf in poll (__timeout=-1, __nfds=2, __fds=0x7ffca440c020) at /usr/include/x86_64-linux-gnu/bits/poll2.h:41
      #7  handle_connections_sockets () at ./sql/mysqld.cc:6724
      #8  0x0000558802b186ff in mysqld_main (argc=<optimized out>, argv=<optimized out>) at ./sql/mysqld.cc:6192
      #9  0x00007f624428f2e1 in __libc_start_main (main=0x558802af7b90 <main(int, char**)>, argc=22, argv=0x7ffca440c3c8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffca440c3b8) at ../csu/libc-start.c:291
      #10 0x0000558802b0aeea in _start ()
       
      Thread 32 (Thread 0x7f61f1ffb700 (LWP 5638)):
      #0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
      #1  0x000055880317df7e in inline_mysql_cond_timedwait (src_file=0x55880341b338 "/home/buildbot/buildbot/build/mariadb-10.2.37/storage/maria/ma_servicethread.c", src_line=116, abstime=0x7f61f1ffac50, mutex=0x5588043fcb20 <LOCK_checkpoint>, that=0x5588043fcae0 <COND_checkpoint>) at ./include/mysql/psi/mysql_thread.h:1182
      #2  my_service_thread_sleep (control=control@entry=0x558803aff9c0 <checkpoint_control>, sleep_time=<optimized out>) at ./storage/maria/ma_servicethread.c:115
      #3  0x00005588031759a1 in ma_checkpoint_background (arg=arg@entry=0x1e) at ./storage/maria/ma_checkpoint.c:709
      #4  0x000055880324c551 in pfs_spawn_thread (arg=0x55880614d608) at ./storage/perfschema/pfs.cc:1869
      #5  0x00007f6245e044a4 in start_thread (arg=0x7f61f1ffb700) at pthread_create.c:456
      #6  0x00007f6244357d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 31 (Thread 0x7f61f3fff700 (LWP 5634)):
      #0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
      #1  0x0000558802f454e6 in os_event::wait (this=0x558805fd9b10) at ./storage/innobase/os/os0event.cc:158
      #2  os_event::wait_low (reset_sig_count=1, this=0x558805fd9b10) at ./storage/innobase/os/os0event.cc:325
      #3  os_event_wait_low (event=0x558805fd9b10, reset_sig_count=reset_sig_count@entry=0) at ./storage/innobase/os/os0event.cc:507
      #4  0x0000558803075877 in buf_dump_thread () at ./storage/innobase/buf/buf0dump.cc:806
      #5  0x00007f6245e044a4 in start_thread (arg=0x7f61f3fff700) at pthread_create.c:456
      #6  0x00007f6244357d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 30 (Thread 0x7f6208ff9700 (LWP 5633)):
      #0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
      #1  0x0000558802f454e6 in os_event::wait (this=0x558805fd9950) at ./storage/innobase/os/os0event.cc:158
      #2  os_event::wait_low (reset_sig_count=321, this=0x558805fd9950) at ./storage/innobase/os/os0event.cc:325
      #3  os_event_wait_low (event=0x558805fd9950, reset_sig_count=reset_sig_count@entry=0) at ./storage/innobase/os/os0event.cc:507
      #4  0x0000558802fd75af in srv_resume_thread (sig_count=0, timeout_usec=0, wait=true, slot=0x558803afdc70 <srv_sys+304>) at ./storage/innobase/srv/srv0srv.cc:937
      #5  srv_worker_thread (arg=<optimized out>) at ./storage/innobase/srv/srv0srv.cc:2617
      #6  0x00007f6245e044a4 in start_thread (arg=0x7f6208ff9700) at pthread_create.c:456
      #7  0x00007f6244357d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 29 (Thread 0x7f6209ffb700 (LWP 5631)):
      #0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
      #1  0x0000558802f454e6 in os_event::wait (this=0x558805fd98e0) at ./storage/innobase/os/os0event.cc:158
      #2  os_event::wait_low (reset_sig_count=339, this=0x558805fd98e0) at ./storage/innobase/os/os0event.cc:325
      #3  os_event_wait_low (event=0x558805fd98e0, reset_sig_count=reset_sig_count@entry=0) at ./storage/innobase/os/os0event.cc:507
      #4  0x0000558802fd75af in srv_resume_thread (sig_count=0, timeout_usec=0, wait=true, slot=0x558803afdc38 <srv_sys+248>) at ./storage/innobase/srv/srv0srv.cc:937
      #5  srv_worker_thread (arg=<optimized out>) at ./storage/innobase/srv/srv0srv.cc:2617
      #6  0x00007f6245e044a4 in start_thread (arg=0x7f6209ffb700) at pthread_create.c:456
      #7  0x00007f6244357d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 28 (Thread 0x7f620a7fc700 (LWP 5630)):
      #0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
      #1  0x0000558802f454e6 in os_event::wait (this=0x558805fd9870) at ./storage/innobase/os/os0event.cc:158
      #2  os_event::wait_low (reset_sig_count=163, this=0x558805fd9870) at ./storage/innobase/os/os0event.cc:325
      #3  os_event_wait_low (event=0x558805fd9870, reset_sig_count=reset_sig_count@entry=163) at ./storage/innobase/os/os0event.cc:507
      #4  0x0000558802fd8a72 in srv_resume_thread (timeout_usec=0, wait=true, sig_count=163, slot=0x558803afdc00 <srv_sys+192>) at ./storage/innobase/srv/srv0srv.cc:937
      #5  srv_purge_coordinator_suspend (rseg_history_len=<optimized out>, slot=<optimized out>) at ./storage/innobase/srv/srv0srv.cc:2778
      #6  srv_purge_coordinator_thread (arg=<optimized out>) at ./storage/innobase/srv/srv0srv.cc:2865
      #7  0x00007f6245e044a4 in start_thread (arg=0x7f620a7fc700) at pthread_create.c:456
      #8  0x00007f6244357d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 27 (Thread 0x7f620b7fe700 (LWP 5628)):
      #0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
      #1  0x0000558802f44ea2 in os_event::timed_wait (this=this@entry=0x558806101170, abstime=abstime@entry=0x7f620b7fdbe0) at ./storage/innobase/os/os0event.cc:275
      #2  0x0000558802f450f1 in os_event::wait_time_low (this=0x558806101170, time_in_usec=time_in_usec@entry=10000000, reset_sig_count=1, reset_sig_count@entry=0) at ./storage/innobase/os/os0event.cc:385
      #3  0x0000558802f4549a in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=10000000, reset_sig_count=reset_sig_count@entry=0) at ./storage/innobase/os/os0event.cc:490
      #4  0x00005588030c8465 in dict_stats_thread () at ./storage/innobase/dict/dict0stats_bg.cc:475
      #5  0x00007f6245e044a4 in start_thread (arg=0x7f620b7fe700) at pthread_create.c:456
      #6  0x00007f6244357d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 26 (Thread 0x7f6214ff9700 (LWP 5626)):
      #0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
      #1  0x0000558802f44ea2 in os_event::timed_wait (this=this@entry=0x558805fd9aa0, abstime=abstime@entry=0x7f6214ff8c90) at ./storage/innobase/os/os0event.cc:275
      #2  0x0000558802f450f1 in os_event::wait_time_low (this=0x558805fd9aa0, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=1) at ./storage/innobase/os/os0event.cc:385
      #3  0x0000558802f4549a in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=<optimized out>) at ./storage/innobase/os/os0event.cc:490
      #4  0x0000558802fd5382 in srv_monitor_thread () at ./storage/innobase/srv/srv0srv.cc:1748
      #5  0x00007f6245e044a4 in start_thread (arg=0x7f6214ff9700) at pthread_create.c:456
      #6  0x00007f6244357d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 25 (Thread 0x7f622d7c8700 (LWP 5616)):
      #0  0x00007f62459e96ca in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
      #1  0x0000558802f3f261 in LinuxAIOHandler::collect (this=this@entry=0x7f622d7c7af0) at ./storage/innobase/os/os0file.cc:1936
      #2  0x0000558802f3f812 in LinuxAIOHandler::poll (this=this@entry=0x7f622d7c7af0, m1=m1@entry=0x7f622d7c7b90, m2=m2@entry=0x7f622d7c7b98, request=request@entry=0x7f622d7c7ba0) at ./storage/innobase/os/os0file.cc:2097
      #3  0x0000558802f40fd8 in os_aio_linux_handler (request=0x7f622d7c7ba0, m2=0x7f622d7c7b98, m1=0x7f622d7c7b90, global_segment=<optimized out>) at ./storage/innobase/os/os0file.cc:2151
      #4  os_aio_handler (segment=segment@entry=1, m1=m1@entry=0x7f622d7c7b90, m2=m2@entry=0x7f622d7c7b98, request=request@entry=0x7f622d7c7ba0) at ./storage/innobase/os/os0file.cc:5784
      #5  0x00005588030d3c0a in fil_aio_wait (segment=segment@entry=1) at ./storage/innobase/fil/fil0fil.cc:5115
      #6  0x0000558802fda478 in io_handler_thread (arg=<optimized out>) at ./storage/innobase/srv/srv0start.cc:332
      #7  0x00007f6245e044a4 in start_thread (arg=0x7f622d7c8700) at pthread_create.c:456
      #8  0x00007f6244357d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 24 (Thread 0x7f620affd700 (LWP 5629)):
      #0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
      #1  0x0000558802f44ea2 in os_event::timed_wait (this=this@entry=0x558806149a50, abstime=abstime@entry=0x7f620affca40) at ./storage/innobase/os/os0event.cc:275
      #2  0x0000558802f450f1 in os_event::wait_time_low (this=0x558806149a50, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=reset_sig_count@entry=1) at ./storage/innobase/os/os0event.cc:385
      #3  0x0000558802f4549a in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=reset_sig_count@entry=1) at ./storage/innobase/os/os0event.cc:490
      #4  0x000055880302b212 in ib_wqueue_timedwait (wq=wq@entry=0x558806137550, wait_in_usecs=wait_in_usecs@entry=5000000) at ./storage/innobase/ut/ut0wqueue.cc:156
      #5  0x0000558803110f4a in fts_optimize_thread (arg=<optimized out>) at ./storage/innobase/fts/fts0opt.cc:2829
      #6  0x00007f6245e044a4 in start_thread (arg=0x7f620affd700) at pthread_create.c:456
      #7  0x00007f6244357d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 23 (Thread 0x7f622fffe700 (LWP 5221)):
      #0  0x00007f6244358303 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
      #1  0x00007f623e59acff in asio::detail::epoll_reactor::run (this=0x558805f74060, block=block@entry=true, ops=...) at /usr/include/asio/detail/impl/epoll_reactor.ipp:391
      #2  0x00007f623e5e9519 in asio::detail::task_io_service::do_run_one (ec=..., this_thread=..., lock=..., this=0x558805f73f80) at /usr/include/asio/detail/impl/task_io_service.ipp:355
      #3  asio::detail::task_io_service::run (ec=..., this=0x558805f73f80) at /usr/include/asio/detail/impl/task_io_service.ipp:148
      #4  asio::io_service::run (this=<optimized out>) at /usr/include/asio/impl/io_service.ipp:58
      #5  gcomm::AsioProtonet::event_loop (this=<optimized out>, period=...) at ./gcomm/src/asio_protonet.cpp:116
      #6  0x00007f623e5cb469 in GCommConn::run (this=0x558805f73bb0) at ./gcs/src/gcs_gcomm.cpp:433
      #7  0x00007f623e5d4919 in GCommConn::run_fn (arg=<optimized out>) at ./gcs/src/gcs_gcomm.cpp:174
      #8  0x00007f6245e044a4 in start_thread (arg=0x7f622fffe700) at pthread_create.c:456
      #9  0x00007f6244357d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 22 (Thread 0x7f61f37fe700 (LWP 5635)):
      #0  0x00007f6245e0d7fd in nanosleep () at ../sysdeps/unix/syscall-template.S:84
      #1  0x0000558802f45750 in os_thread_sleep (tm=tm@entry=1000000) at ./storage/innobase/os/os0thread.cc:231
      #2  0x000055880305f042 in btr_defragment_thread () at ./storage/innobase/btr/btr0defragment.cc:722
      #3  0x00007f6245e044a4 in start_thread (arg=0x7f61f37fe700) at pthread_create.c:456
      #4  0x00007f6244357d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 21 (Thread 0x7f623c3c7700 (LWP 5356)):
      #0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
      #1  0x0000558802f454e6 in os_event::wait (this=0x558806077cb0) at ./storage/innobase/os/os0event.cc:158
      #2  os_event::wait_low (reset_sig_count=1, this=0x558806077cb0) at ./storage/innobase/os/os0event.cc:325
      #3  os_event_wait_low (event=0x558806077cb0, reset_sig_count=<optimized out>) at ./storage/innobase/os/os0event.cc:507
      #4  0x0000558802fe0dbc in sync_array_wait_event (arr=arr@entry=0x558805fd14e0, cell=@0x7f623c3c44d8: 0x7f623c20f050) at ./storage/innobase/sync/sync0arr.cc:471
      #5  0x0000558802fe3057 in rw_lock_s_lock_spin (lock=lock@entry=0x7f622dbe9950, pass=pass@entry=0, file_name=file_name@entry=0x5588033f6f80 "/home/buildbot/buildbot/build/mariadb-10.2.37/storage/innobase/row/row0row.cc", line=line@entry=976) at ./storage/innobase/sync/sync0rw.cc:373
      #6  0x0000558803060da0 in rw_lock_s_lock_func (pass=0, line=976, file_name=0x5588033f6f80 "/home/buildbot/buildbot/build/mariadb-10.2.37/storage/innobase/row/row0row.cc", lock=0x7f622dbe9950) at ./storage/innobase/include/sync0rw.ic:290
      #7  pfs_rw_lock_s_lock_func (lock=0x7f622dbe9950, file_name=0x5588033f6f80 "/home/buildbot/buildbot/build/mariadb-10.2.37/storage/innobase/row/row0row.cc", line=976, pass=0) at ./storage/innobase/include/sync0rw.ic:637
      #8  0x0000558803061105 in buf_page_mtr_lock (block=block@entry=0x7f622dbe98c0, rw_latch=rw_latch@entry=1, mtr=mtr@entry=0x7f623c3c5ac0, file=file@entry=0x5588033f6f80 "/home/buildbot/buildbot/build/mariadb-10.2.37/storage/innobase/row/row0row.cc", line=line@entry=976) at ./storage/innobase/buf/buf0buf.cc:4109
      #9  0x0000558803066f28 in buf_page_get_low (page_id=..., page_size=..., rw_latch=rw_latch@entry=1, guess=<optimized out>, mode=<optimized out>, file=<optimized out>, line=<optimized out>, mtr=<optimized out>, err=<optimized out>) at ./storage/innobase/buf/buf0buf.cc:4788
      #10 0x00005588030673e2 in buf_page_get_gen (page_id=..., page_id@entry=..., page_size=..., rw_latch=rw_latch@entry=1, guess=guess@entry=0x7f622dbe98c0, mode=mode@entry=10, file=file@entry=0x5588033f6f80 "/home/buildbot/buildbot/build/mariadb-10.2.37/storage/innobase/row/row0row.cc", line=<optimized out>, mtr=<optimized out>, err=<optimized out>) at ./storage/innobase/buf/buf0buf.cc:4836
      #11 0x0000558803046caf in btr_cur_search_to_nth_level_func (index=index@entry=0x7f621804ab68, level=level@entry=0, tuple=tuple@entry=0x7f6224012838, mode=mode@entry=PAGE_CUR_LE, latch_mode=latch_mode@entry=2, cursor=cursor@entry=0x7f6224012358, has_search_latch=<optimized out>, file=<optimized out>, line=<optimized out>, mtr=<optimized out>, autoinc=<optimized out>) at ./storage/innobase/btr/btr0cur.cc:1276
      #12 0x0000558802fa9a23 in btr_pcur_open_low (index=0x7f621804ab68, tuple=tuple@entry=0x7f6224012838, latch_mode=latch_mode@entry=2, cursor=cursor@entry=0x7f6224012358, line=line@entry=976, mtr=mtr@entry=0x7f623c3c5ac0, autoinc=0, file=0x5588033f6f80 "/home/buildbot/buildbot/build/mariadb-10.2.37/storage/innobase/row/row0row.cc", mode=PAGE_CUR_LE, level=0) at ./storage/innobase/include/btr0pcur.ic:457
      #13 0x0000558802fa9cdc in row_search_on_row_ref (pcur=pcur@entry=0x7f6224012358, mode=mode@entry=2, table=<optimized out>, ref=0x7f6224012838, mtr=mtr@entry=0x7f623c3c5ac0) at ./storage/innobase/row/row0row.cc:976
      #14 0x0000558802fc0603 in row_undo_search_clust_to_pcur (node=node@entry=0x7f62240122e8) at ./storage/innobase/row/row0undo.cc:180
      #15 0x00005588031370be in row_undo_ins_parse_undo_rec (node=node@entry=0x7f62240122e8, dict_locked=dict_locked@entry=0) at ./storage/innobase/row/row0uins.cc:386
      #16 0x0000558803138c76 in row_undo_ins (node=0x7f62240122e8, thr=0x7f6224012148) at ./storage/innobase/row/row0uins.cc:488
      #17 0x0000558802fc021d in row_undo (thr=0x7f6224012148, node=0x7f62240122e8) at ./storage/innobase/row/row0undo.cc:298
      #18 row_undo_step (thr=thr@entry=0x7f6224012148) at ./storage/innobase/row/row0undo.cc:351
      #19 0x0000558802f63b71 in que_thr_step (thr=<optimized out>) at ./storage/innobase/que/que0que.cc:1039
      #20 que_run_threads_low (thr=<optimized out>) at ./storage/innobase/que/que0que.cc:1103
      #21 que_run_threads (thr=<optimized out>) at ./storage/innobase/que/que0que.cc:1143
      #22 0x0000558803008609 in trx_rollback_to_savepoint_low (trx=trx@entry=0x7f622e3fc070, savept=savept@entry=0x0) at ./storage/innobase/trx/trx0roll.cc:107
      #23 0x0000558803008fdf in trx_rollback_for_mysql_low (trx=0x7f622e3fc070) at ./storage/innobase/trx/trx0roll.cc:169
      #24 trx_rollback_for_mysql (trx=trx@entry=0x7f622e3fc070) at ./storage/innobase/trx/trx0roll.cc:200
      #25 0x0000558802ecd7fe in innobase_rollback (hton=<optimized out>, thd=0x7f61cc0009a8, rollback_trx=<optimized out>) at ./storage/innobase/handler/ha_innodb.cc:4767
      #26 0x0000558802d539ba in ha_rollback_trans (thd=thd@entry=0x7f61cc0009a8, all=all@entry=true) at ./sql/handler.cc:1708
      #27 0x0000558802c96d5a in trans_rollback (thd=thd@entry=0x7f61cc0009a8) at ./sql/transaction.cc:415
      #28 0x0000558802cf743a in wsrep_client_rollback (thd=thd@entry=0x7f61cc0009a8) at ./sql/wsrep_thd.cc:61
      #29 0x0000558802cf79f3 in wsrep_rollback_process (thd=0x7f62240009a8) at ./sql/wsrep_thd.cc:679
      #30 0x0000558802ce8727 in start_wsrep_THD (arg=arg@entry=0x558805f938e8) at ./sql/wsrep_mysqld.cc:2173
      #31 0x000055880324c551 in pfs_spawn_thread (arg=0x558805f5b888) at ./storage/perfschema/pfs.cc:1869
      #32 0x00007f6245e044a4 in start_thread (arg=0x7f623c3c7700) at pthread_create.c:456
      #33 0x00007f6244357d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 20 (Thread 0x7f6215ffb700 (LWP 5625)):
      #0  0x00007f6245e0d7fd in nanosleep () at ../sysdeps/unix/syscall-template.S:84
      #1  0x0000558802f45750 in os_thread_sleep (tm=tm@entry=30000000) at ./storage/innobase/os/os0thread.cc:231
      #2  0x0000558802fe1efb in sync_array_print_long_waits (waiter=waiter@entry=0x7f6215ffabd0, sema=sema@entry=0x7f6215ffabd8) at ./storage/innobase/sync/sync0arr.cc:1106
      #3  0x0000558802fd5a68 in srv_error_monitor_thread () at ./storage/innobase/srv/srv0srv.cc:1877
      #4  0x00007f6245e044a4 in start_thread (arg=0x7f6215ffb700) at pthread_create.c:456
      #5  0x00007f6244357d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 19 (Thread 0x7f622cfc7700 (LWP 5617)):
      #0  0x00007f62459e96ca in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
      #1  0x0000558802f3f261 in LinuxAIOHandler::collect (this=this@entry=0x7f622cfc6af0) at ./storage/innobase/os/os0file.cc:1936
      #2  0x0000558802f3f812 in LinuxAIOHandler::poll (this=this@entry=0x7f622cfc6af0, m1=m1@entry=0x7f622cfc6b90, m2=m2@entry=0x7f622cfc6b98, request=request@entry=0x7f622cfc6ba0) at ./storage/innobase/os/os0file.cc:2097
      #3  0x0000558802f40fd8 in os_aio_linux_handler (request=0x7f622cfc6ba0, m2=0x7f622cfc6b98, m1=0x7f622cfc6b90, global_segment=<optimized out>) at ./storage/innobase/os/os0file.cc:2151
      #4  os_aio_handler (segment=segment@entry=2, m1=m1@entry=0x7f622cfc6b90, m2=m2@entry=0x7f622cfc6b98, request=request@entry=0x7f622cfc6ba0) at ./storage/innobase/os/os0file.cc:5784
      #5  0x00005588030d3c0a in fil_aio_wait (segment=segment@entry=2) at ./storage/innobase/fil/fil0fil.cc:5115
      #6  0x0000558802fda478 in io_handler_thread (arg=<optimized out>) at ./storage/innobase/srv/srv0start.cc:332
      #7  0x00007f6245e044a4 in start_thread (arg=0x7f622cfc7700) at pthread_create.c:456
      #8  0x00007f6244357d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 18 (Thread 0x7f62157fa700 (LWP 5624)):
      #0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
      #1  0x0000558802f44ea2 in os_event::timed_wait (this=this@entry=0x5588060efa40, abstime=abstime@entry=0x7f62157f9c00) at ./storage/innobase/os/os0event.cc:275
      #2  0x0000558802f450f1 in os_event::wait_time_low (this=this@entry=0x5588060efa40, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=23) at ./storage/innobase/os/os0event.cc:385
      #3  0x0000558802f4549a in os_event_wait_time_low (event=event@entry=0x5588060efa40, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=<optimized out>) at ./storage/innobase/os/os0event.cc:490
      #4  0x0000558802f1c410 in lock_wait_timeout_thread () at ./storage/innobase/lock/lock0wait.cc:525
      #5  0x00007f6245e044a4 in start_thread (arg=0x7f62157fa700) at pthread_create.c:456
      #6  0x00007f6244357d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 17 (Thread 0x7f61f2ffd700 (LWP 5636)):
      #0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
      #1  0x0000558802f454e6 in os_event::wait (this=0x558805fd9bf0) at ./storage/innobase/os/os0event.cc:158
      #2  os_event::wait_low (reset_sig_count=1, this=0x558805fd9bf0) at ./storage/innobase/os/os0event.cc:325
      #3  os_event_wait_low (event=0x558805fd9bf0, reset_sig_count=reset_sig_count@entry=0) at ./storage/innobase/os/os0event.cc:507
      #4  0x000055880306dcac in buf_resize_thread () at ./storage/innobase/buf/buf0buf.cc:3109
      #5  0x00007f6245e044a4 in start_thread (arg=0x7f61f2ffd700) at pthread_create.c:456
      #6  0x00007f6244357d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 16 (Thread 0x7f62167fc700 (LWP 5621)):
      #0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
      #1  0x0000558802f454e6 in os_event::wait (this=0x558806077cb0) at ./storage/innobase/os/os0event.cc:158
      #2  os_event::wait_low (reset_sig_count=1, this=0x558806077cb0) at ./storage/innobase/os/os0event.cc:325
      #3  os_event_wait_low (event=0x558806077cb0, reset_sig_count=<optimized out>) at ./storage/innobase/os/os0event.cc:507
      #4  0x0000558802fe0dbc in sync_array_wait_event (arr=arr@entry=0x558805fd14e0, cell=@0x7f62167fb428: 0x7f623c20f0d0) at ./storage/innobase/sync/sync0arr.cc:471
      #5  0x0000558802fe43e5 in rw_lock_sx_lock_func (lock=lock@entry=0x7f622dbe9950, pass=pass@entry=2, file_name=file_name@entry=0x558803406d58 "/home/buildbot/buildbot/build/mariadb-10.2.37/storage/innobase/buf/buf0flu.cc", line=line@entry=1212) at ./storage/innobase/sync/sync0rw.cc:838
      #6  0x000055880307aeb1 in pfs_rw_lock_sx_lock_func (pass=2, file_name=0x558803406d58 "/home/buildbot/buildbot/build/mariadb-10.2.37/storage/innobase/buf/buf0flu.cc", line=1212, lock=0x7f622dbe9950) at ./storage/innobase/include/sync0rw.ic:672
      #7  buf_flush_page (buf_pool=0x55880605b6f0, bpage=0x7f622dbe98c0, flush_type=BUF_FLUSH_LIST, sync=<optimized out>) at ./storage/innobase/buf/buf0flu.cc:1212
      #8  0x000055880307b85e in buf_flush_try_neighbors (page_id=..., page_id@entry=..., flush_type=flush_type@entry=BUF_FLUSH_LIST, n_flushed=n_flushed@entry=27, n_to_flush=n_to_flush@entry=200) at ./storage/innobase/buf/buf0flu.cc:1458
      #9  0x000055880307c2b0 in buf_flush_page_and_try_neighbors (flush_type=BUF_FLUSH_LIST, count=<synthetic pointer>, n_to_flush=<optimized out>, bpage=<optimized out>) at ./storage/innobase/buf/buf0flu.cc:1530
      #10 buf_do_flush_list_batch (buf_pool=buf_pool@entry=0x55880605b6f0, min_n=min_n@entry=200, lsn_limit=lsn_limit@entry=18446744073709551615) at ./storage/innobase/buf/buf0flu.cc:1789
      #11 0x000055880307d280 in buf_flush_batch (n=0x7f62167fb8c0, lsn_limit=<optimized out>, min_n=<optimized out>, flush_type=BUF_FLUSH_LIST, buf_pool=0x55880605b6f0) at ./storage/innobase/buf/buf0flu.cc:1858
      #12 buf_flush_do_batch (type=BUF_FLUSH_LIST, n=0x7f62167fb8c0, lsn_limit=<optimized out>, min_n=<optimized out>, buf_pool=0x55880605b6f0) at ./storage/innobase/buf/buf0flu.cc:2028
      #13 buf_flush_lists (min_n=<optimized out>, lsn_limit=<optimized out>, n_processed=<optimized out>) at ./storage/innobase/buf/buf0flu.cc:2134
      #14 0x0000558803080ce6 in buf_flush_page_cleaner_coordinator () at ./storage/innobase/buf/buf0flu.cc:3302
      #15 0x00007f6245e044a4 in start_thread (arg=0x7f62167fc700) at pthread_create.c:456
      #16 0x00007f6244357d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 15 (Thread 0x7f62097fa700 (LWP 5632)):
      #0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
      #1  0x0000558802f454e6 in os_event::wait (this=0x558805fd99c0) at ./storage/innobase/os/os0event.cc:158
      #2  os_event::wait_low (reset_sig_count=295, this=0x558805fd99c0) at ./storage/innobase/os/os0event.cc:325
      #3  os_event_wait_low (event=0x558805fd99c0, reset_sig_count=reset_sig_count@entry=0) at ./storage/innobase/os/os0event.cc:507
      #4  0x0000558802fd75af in srv_resume_thread (sig_count=0, timeout_usec=0, wait=true, slot=0x558803afdca8 <srv_sys+360>) at ./storage/innobase/srv/srv0srv.cc:937
      #5  srv_worker_thread (arg=<optimized out>) at ./storage/innobase/srv/srv0srv.cc:2617
      #6  0x00007f6245e044a4 in start_thread (arg=0x7f62097fa700) at pthread_create.c:456
      #7  0x00007f6244357d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 14 (Thread 0x7f623c0ac700 (LWP 5640)):
      #0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
      #1  0x0000558802f454e6 in os_event::wait (this=0x5588060ef860) at ./storage/innobase/os/os0event.cc:158
      #2  os_event::wait_low (reset_sig_count=2, this=0x5588060ef860) at ./storage/innobase/os/os0event.cc:325
      #3  os_event_wait_low (event=0x5588060ef860, reset_sig_count=<optimized out>) at ./storage/innobase/os/os0event.cc:507
      #4  0x0000558802fe0dbc in sync_array_wait_event (arr=0x558805fd14e0, cell=@0x7f623c0abbe8: 0x7f623c20f010) at ./storage/innobase/sync/sync0arr.cc:471
      #5  0x0000558802ee48ad in TTASEventMutex<GenericPolicy>::enter (line=<optimized out>, filename=<optimized out>, max_delay=<optimized out>, max_spins=<optimized out>, this=0x5588060d28c0) at ./storage/innobase/include/ib0mutex.h:516
      #6  PolicyMutex<TTASEventMutex<GenericPolicy> >::enter (this=0x5588060d28c0, n_spins=<optimized out>, n_delay=<optimized out>, name=<optimized out>, line=<optimized out>) at ./storage/innobase/include/ib0mutex.h:637
      #7  0x0000558802ed8c16 in bg_wsrep_kill_trx (void_arg=0x7f6218038650) at ./storage/innobase/handler/ha_innodb.cc:19524
      #8  0x0000558802ba745b in handle_manager (arg=arg@entry=0x0) at ./sql/sql_manager.cc:112
      #9  0x000055880324c551 in pfs_spawn_thread (arg=0x5588064c9138) at ./storage/perfschema/pfs.cc:1869
      #10 0x00007f6245e044a4 in start_thread (arg=0x7f623c0ac700) at pthread_create.c:456
      #11 0x00007f6244357d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 13 (Thread 0x7f61f27fc700 (LWP 5637)):
      #0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
      #1  0x0000558802ec745b in inline_mysql_cond_wait (src_file=0x5588033cba08 "/home/buildbot/buildbot/build/mariadb-10.2.37/storage/innobase/handler/ha_innodb.cc", src_line=348, mutex=0x7f61f27fbcc0, that=0x7f61f27fbcf0) at ./include/mysql/psi/mysql_thread.h:1156
      #2  thd_destructor_proxy () at ./storage/innobase/handler/ha_innodb.cc:348
      #3  0x000055880324c551 in pfs_spawn_thread (arg=0x55880614abf8) at ./storage/perfschema/pfs.cc:1869
      #4  0x00007f6245e044a4 in start_thread (arg=0x7f61f27fc700) at pthread_create.c:456
      #5  0x00007f6244357d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 12 (Thread 0x7f6216ffd700 (LWP 5620)):
      #0  0x00007f62459e96ca in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
      #1  0x0000558802f3f261 in LinuxAIOHandler::collect (this=this@entry=0x7f6216ffcaf0) at ./storage/innobase/os/os0file.cc:1936
      #2  0x0000558802f3f812 in LinuxAIOHandler::poll (this=this@entry=0x7f6216ffcaf0, m1=m1@entry=0x7f6216ffcb90, m2=m2@entry=0x7f6216ffcb98, request=request@entry=0x7f6216ffcba0) at ./storage/innobase/os/os0file.cc:2097
      #3  0x0000558802f40fd8 in os_aio_linux_handler (request=0x7f6216ffcba0, m2=0x7f6216ffcb98, m1=0x7f6216ffcb90, global_segment=<optimized out>) at ./storage/innobase/os/os0file.cc:2151
      #4  os_aio_handler (segment=segment@entry=5, m1=m1@entry=0x7f6216ffcb90, m2=m2@entry=0x7f6216ffcb98, request=request@entry=0x7f6216ffcba0) at ./storage/innobase/os/os0file.cc:5784
      #5  0x00005588030d3c0a in fil_aio_wait (segment=segment@entry=5) at ./storage/innobase/fil/fil0fil.cc:5115
      #6  0x0000558802fda478 in io_handler_thread (arg=<optimized out>) at ./storage/innobase/srv/srv0start.cc:332
      #7  0x00007f6245e044a4 in start_thread (arg=0x7f6216ffd700) at pthread_create.c:456
      #8  0x00007f6244357d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 11 (Thread 0x7f623c37c700 (LWP 5357)):
      #0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
      #1  0x00007f6245e06ce6 in __GI___pthread_mutex_lock (mutex=mutex@entry=0x7f61cc002248) at ../nptl/pthread_mutex_lock.c:135
      #2  0x0000558802cf8d1f in inline_mysql_mutex_lock (src_file=0x55880334aa20 "/home/buildbot/buildbot/build/mariadb-10.2.37/sql/wsrep_thd.cc", src_line=762, that=0x7f61cc002248) at ./include/mysql/psi/mysql_thread.h:675
      #3  wsrep_thd_is_BF (thd=thd@entry=0x7f61cc0009a8, sync=sync@entry=1 '\001') at ./sql/wsrep_thd.cc:762
      #4  0x0000558802b7e928 in thd_need_ordering_with (thd=0x7f62180009a8, other_thd=0x7f61cc0009a8) at ./sql/sql_class.cc:4740
      #5  0x0000558802f11fa2 in lock_rec_has_to_wait (for_locking=false, lock_is_on_supremum=<optimized out>, lock2=0x7f622e3fc278, type_mode=<optimized out>, trx=0x7f622e3fd0c8) at ./storage/innobase/lock/lock0lock.cc:795
      #6  lock_has_to_wait (lock1=0x7f622e3fd428, lock2=0x7f622e3fc278) at ./storage/innobase/lock/lock0lock.cc:878
      #7  0x0000558802f12248 in DeadlockChecker::search (this=this@entry=0x7f623c378440) at ./storage/innobase/lock/lock0lock.cc:7147
      #8  0x0000558802f180cf in DeadlockChecker::check_and_resolve (lock=lock@entry=0x7f622e3fd428, trx=trx@entry=0x7f622e3fd0c8) at ./storage/innobase/lock/lock0lock.cc:7291
      #9  0x0000558802f1836d in lock_rec_enqueue_waiting (c_lock=c_lock@entry=0x7f622e3fc278, type_mode=type_mode@entry=2563, block=block@entry=0x7f622dbe98c0, heap_no=heap_no@entry=2, index=index@entry=0x7f621804ab68, thr=thr@entry=0x7f61cc059670, prdt=0x0) at ./storage/innobase/lock/lock0lock.cc:1801
      #10 0x0000558802f18a66 in lock_rec_insert_check_and_lock (flags=flags@entry=0, rec=<optimized out>, block=0x7f622dbe98c0, index=index@entry=0x7f621804ab68, thr=thr@entry=0x7f61cc059670, mtr=0x7f623c3794b0, inherit=<optimized out>) at ./storage/innobase/lock/lock0lock.cc:5712
      #11 0x000055880303f1d9 in btr_cur_ins_lock_and_undo (flags=flags@entry=0, cursor=cursor@entry=0x7f623c378af0, entry=entry@entry=0x7f61cc09e338, thr=thr@entry=0x7f61cc059670, mtr=mtr@entry=0x7f623c3794b0, inherit=inherit@entry=0x7f623c378838) at ./storage/innobase/btr/btr0cur.cc:2983
      #12 0x0000558803049220 in btr_cur_optimistic_insert (flags=flags@entry=0, cursor=cursor@entry=0x7f623c378af0, offsets=offsets@entry=0x7f623c378a88, heap=heap@entry=0x7f623c378a80, entry=entry@entry=0x7f61cc09e338, rec=rec@entry=0x7f623c378f50, big_rec=0x7f623c378a78, n_ext=<optimized out>, thr=0x7f61cc059670, mtr=0x7f623c3794b0) at ./storage/innobase/btr/btr0cur.cc:3222
      #13 0x0000558802f7d837 in row_ins_clust_index_entry_low (flags=flags@entry=0, mode=<optimized out>, mode@entry=2, index=index@entry=0x7f621804ab68, n_uniq=n_uniq@entry=1, entry=entry@entry=0x7f61cc09e338, n_ext=n_ext@entry=0, thr=<optimized out>) at ./storage/innobase/row/row0ins.cc:2689
      #14 0x0000558802f835ce in row_ins_clust_index_entry (index=index@entry=0x7f621804ab68, entry=entry@entry=0x7f61cc09e338, thr=thr@entry=0x7f61cc059670, n_ext=n_ext@entry=0) at ./storage/innobase/row/row0ins.cc:3141
      #15 0x0000558802f83f6a in row_ins_index_entry (thr=0x7f61cc059670, entry=<optimized out>, index=<optimized out>) at ./storage/innobase/row/row0ins.cc:3260
      #16 row_ins_index_entry_step (thr=0x7f61cc059670, node=<optimized out>) at ./storage/innobase/row/row0ins.cc:3411
      #17 row_ins (thr=<optimized out>, node=<optimized out>) at ./storage/innobase/row/row0ins.cc:3548
      #18 row_ins_step (thr=thr@entry=0x7f61cc059670) at ./storage/innobase/row/row0ins.cc:3668
      #19 0x0000558802f95f7d in row_insert_for_mysql (mysql_rec=mysql_rec@entry=0x7f61cc0a3a18 "\377\001", prebuilt=0x7f61cc058f58) at ./storage/innobase/row/row0mysql.cc:1411
      #20 0x0000558802ede61b in ha_innobase::write_row (this=0x7f61cc0587f0, record=0x7f61cc0a3a18 "\377\001") at ./storage/innobase/handler/ha_innodb.cc:8164
      #21 0x0000558802d5abbe in handler::ha_write_row (this=0x7f61cc0587f0, buf=0x7f61cc0a3a18 "\377\001") at ./sql/handler.cc:6118
      #22 0x0000558802e3b342 in Rows_log_event::write_row (this=this@entry=0x7f6218034988, rgi=rgi@entry=0x7f621801b3c0, overwrite=false) at ./sql/log_event.cc:12874
      #23 0x0000558802e3b77d in Write_rows_log_event::do_exec_row (this=0x7f6218034988, rgi=0x7f621801b3c0) at ./sql/log_event.cc:13065
      #24 0x0000558802e2f850 in Rows_log_event::do_apply_event (this=0x7f6218034988, rgi=0x7f621801b3c0) at ./sql/log_event.cc:11101
      #25 0x0000558802cf60b4 in Log_event::apply_event (rgi=0x7f621801b3c0, this=0x7f6218034988) at ./sql/log_event.h:1454
      #26 wsrep_apply_events (buf_len=0, events_buf=0x7f623000bcc8, thd=0x7f62180009a8) at ./sql/wsrep_applier.cc:164
      #27 wsrep_apply_cb (ctx=0x7f62180009a8, buf=0x7f623000bcc8, buf_len=129, flags=<optimized out>, meta=<optimized out>) at ./sql/wsrep_applier.cc:267
      #28 0x00007f623e541eb8 in galera::TrxHandle::apply (this=this@entry=0x7f621801c640, recv_ctx=recv_ctx@entry=0x7f62180009a8, apply_cb=apply_cb@entry=0x558802cf5b60 <wsrep_apply_cb(void*, void const*, size_t, uint32_t, wsrep_trx_meta_t const*)>, meta=...) at ./galera/src/trx_handle.cpp:317
      #29 0x00007f623e551116 in apply_trx_ws (recv_ctx=recv_ctx@entry=0x7f62180009a8, apply_cb=0x558802cf5b60 <wsrep_apply_cb(void*, void const*, size_t, uint32_t, wsrep_trx_meta_t const*)>, commit_cb=0x558802cf64a0 <wsrep_commit_cb(void*, uint32_t, wsrep_trx_meta_t const*, wsrep_bool_t*, bool)>, trx=..., meta=...) at ./galera/src/replicator_smm.cpp:34
      #30 0x00007f623e554026 in galera::ReplicatorSMM::apply_trx (this=this@entry=0x558805f3adc0, recv_ctx=recv_ctx@entry=0x7f62180009a8, trx=trx@entry=0x7f621801c640) at ./galera/src/replicator_smm.cpp:454
      #31 0x00007f623e556bae in galera::ReplicatorSMM::process_trx (this=0x558805f3adc0, recv_ctx=0x7f62180009a8, trx=0x7f621801c640) at ./galera/src/replicator_smm.cpp:1258
      #32 0x00007f623e58ee38 in galera::GcsActionSource::dispatch (this=this@entry=0x558805f3b518, recv_ctx=recv_ctx@entry=0x7f62180009a8, act=..., exit_loop=@0x7f623c37b87f: false) at ./galera/src/gcs_action_source.cpp:115
      #33 0x00007f623e590206 in galera::GcsActionSource::process (this=0x558805f3b518, recv_ctx=0x7f62180009a8, exit_loop=@0x7f623c37b87f: false) at ./galera/src/gcs_action_source.cpp:180
      #34 0x00007f623e5572cb in galera::ReplicatorSMM::async_recv (this=0x558805f3adc0, recv_ctx=0x7f62180009a8) at ./galera/src/replicator_smm.cpp:362
      #35 0x00007f623e52b02b in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at ./galera/src/wsrep_provider.cpp:244
      #36 0x0000558802cf6ff8 in wsrep_replication_process (thd=0x7f62180009a8) at ./sql/wsrep_thd.cc:486
      #37 0x0000558802ce8727 in start_wsrep_THD (arg=arg@entry=0x558805f93518) at ./sql/wsrep_mysqld.cc:2173
      #38 0x000055880324c551 in pfs_spawn_thread (arg=0x558805f5bbe8) at ./storage/perfschema/pfs.cc:1869
      #39 0x00007f6245e044a4 in start_thread (arg=0x7f623c37c700) at pthread_create.c:456
      #40 0x00007f6244357d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 10 (Thread 0x7f623c0f7700 (LWP 5639)):
      #0  do_sigwait (set=<optimized out>, set@entry=0x7f623c0f6c90, sig=sig@entry=0x7f623c0f6c38) at ../sysdeps/unix/sysv/linux/sigwait.c:64
      #1  0x00007f6245e0dd0d in __sigwait (set=set@entry=0x7f623c0f6c90, sig=sig@entry=0x7f623c0f6c38) at ../sysdeps/unix/sysv/linux/sigwait.c:96
      #2  0x0000558802b0d48b in signal_hand (arg=arg@entry=0x0) at ./sql/mysqld.cc:3557
      #3  0x000055880324c551 in pfs_spawn_thread (arg=0x55880645b068) at ./storage/perfschema/pfs.cc:1869
      #4  0x00007f6245e044a4 in start_thread (arg=0x7f623c0f7700) at pthread_create.c:456
      #5  0x00007f6244357d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 9 (Thread 0x7f622f7fd700 (LWP 5355)):
      #0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
      #1  0x00007f623e5c85b8 in gu::Lock::wait (this=0x7f622f7fbd60, cond=...) at ./galerautils/src/gu_lock.hpp:59
      #2  RecvBuf::front (timeout=..., this=0x558805f73da8) at ./gcs/src/gcs_gcomm.cpp:114
      #3  gcomm_recv (backend=<optimized out>, msg=0x558805f43000, timeout=9223372035999999999) at ./gcs/src/gcs_gcomm.cpp:608
      #4  0x00007f623e5becc8 in core_msg_recv (timeout=9223372035999999999, recv_msg=0x558805f43000, backend=0x558805f43128) at ./gcs/src/gcs_core.cpp:469
      #5  gcs_core_recv (conn=0x558805f42fa0, recv_act=recv_act@entry=0x7f622f7fccf0, timeout=9223372035999999999) at ./gcs/src/gcs_core.cpp:1102
      #6  0x00007f623e5b9fb6 in gcs_recv_thread (arg=0x558805f42dc0) at ./gcs/src/gcs.cpp:1277
      #7  0x00007f6245e044a4 in start_thread (arg=0x7f622f7fd700) at pthread_create.c:456
      #8  0x00007f6244357d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 8 (Thread 0x7f623c061700 (LWP 5641)):
      #0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
      #1  0x00007f6245e06ce6 in __GI___pthread_mutex_lock (mutex=mutex@entry=0x7f61cc002248) at ../nptl/pthread_mutex_lock.c:135
      #2  0x0000558802ba7a61 in inline_mysql_mutex_lock (that=that@entry=0x7f61cc002248, src_file=src_file@entry=0x558803325f18 "/home/buildbot/buildbot/build/mariadb-10.2.37/sql/sql_parse.cc", src_line=src_line@entry=1268) at ./include/mysql/psi/mysql_thread.h:675
      #3  0x0000558802bbae76 in do_command (thd=0x7f61cc0009a8) at ./sql/sql_parse.cc:1268
      #4  0x0000558802c88492 in do_handle_one_connection (connect=connect@entry=0x5588064c60e8) at ./sql/sql_connect.cc:1336
      #5  0x0000558802c885ed in handle_one_connection (arg=arg@entry=0x5588064c60e8) at ./sql/sql_connect.cc:1241
      #6  0x000055880324c551 in pfs_spawn_thread (arg=0x558806465008) at ./storage/perfschema/pfs.cc:1869
      #7  0x00007f6245e044a4 in start_thread (arg=0x7f623c061700) at pthread_create.c:456
      #8  0x00007f6244357d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 7 (Thread 0x7f62177fe700 (LWP 5619)):
      #0  0x00007f62459e96ca in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
      #1  0x0000558802f3f261 in LinuxAIOHandler::collect (this=this@entry=0x7f62177fdaf0) at ./storage/innobase/os/os0file.cc:1936
      #2  0x0000558802f3f812 in LinuxAIOHandler::poll (this=this@entry=0x7f62177fdaf0, m1=m1@entry=0x7f62177fdb90, m2=m2@entry=0x7f62177fdb98, request=request@entry=0x7f62177fdba0) at ./storage/innobase/os/os0file.cc:2097
      #3  0x0000558802f40fd8 in os_aio_linux_handler (request=0x7f62177fdba0, m2=0x7f62177fdb98, m1=0x7f62177fdb90, global_segment=<optimized out>) at ./storage/innobase/os/os0file.cc:2151
      #4  os_aio_handler (segment=segment@entry=4, m1=m1@entry=0x7f62177fdb90, m2=m2@entry=0x7f62177fdb98, request=request@entry=0x7f62177fdba0) at ./storage/innobase/os/os0file.cc:5784
      #5  0x00005588030d3c0a in fil_aio_wait (segment=segment@entry=4) at ./storage/innobase/fil/fil0fil.cc:5115
      #6  0x0000558802fda478 in io_handler_thread (arg=<optimized out>) at ./storage/innobase/srv/srv0start.cc:332
      #7  0x00007f6245e044a4 in start_thread (arg=0x7f62177fe700) at pthread_create.c:456
      #8  0x00007f6244357d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 6 (Thread 0x7f623f159700 (LWP 5219)):
      #0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
      #1  0x000055880329ed81 in inline_mysql_cond_timedwait (that=0x55880440c500 <COND_timer>, mutex=0x55880440c540 <LOCK_timer>, src_file=0x55880342fed8 "/home/buildbot/buildbot/build/mariadb-10.2.37/mysys/thr_timer.c", src_line=292, abstime=0x7f623f158ce0) at ./include/mysql/psi/mysql_thread.h:1182
      #2  timer_handler (arg=arg@entry=0x0) at ./mysys/thr_timer.c:292
      #3  0x000055880324c551 in pfs_spawn_thread (arg=0x558805f0c098) at ./storage/perfschema/pfs.cc:1869
      #4  0x00007f6245e044a4 in start_thread (arg=0x7f623f159700) at pthread_create.c:456
      #5  0x00007f6244357d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 5 (Thread 0x7f620bfff700 (LWP 5627)):
      #0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
      #1  0x0000558802f454e6 in os_event::wait (this=0x55880610cf60) at ./storage/innobase/os/os0event.cc:158
      #2  os_event::wait_low (reset_sig_count=1, this=0x55880610cf60) at ./storage/innobase/os/os0event.cc:325
      #3  os_event_wait_low (event=0x55880610cf60, reset_sig_count=<optimized out>) at ./storage/innobase/os/os0event.cc:507
      #4  0x0000558802fe0dbc in sync_array_wait_event (arr=arr@entry=0x558805fd14e0, cell=@0x7f620bffebd0: 0x7f623c20f090) at ./storage/innobase/sync/sync0arr.cc:471
      #5  0x0000558802fe340a in rw_lock_x_lock_wait_func (line=<optimized out>, file_name=<optimized out>, threshold=<optimized out>, lock=<optimized out>) at ./storage/innobase/sync/sync0rw.cc:455
      #6  rw_lock_x_lock_low (line=2085, file_name=<optimized out>, pass=0, lock=0x558803bfb380 <dict_operation_lock>) at ./storage/innobase/sync/sync0rw.cc:511
      #7  rw_lock_x_lock_func (lock=lock@entry=0x558803bfb380 <dict_operation_lock>, pass=pass@entry=0, file_name=file_name@entry=0x5588033fd878 "/home/buildbot/buildbot/build/mariadb-10.2.37/storage/innobase/srv/srv0srv.cc", line=line@entry=2085) at ./storage/innobase/sync/sync0rw.cc:660
      #8  0x0000558802fd1af6 in pfs_rw_lock_x_lock_func (pass=0, line=2085, file_name=0x5588033fd878 "/home/buildbot/buildbot/build/mariadb-10.2.37/storage/innobase/srv/srv0srv.cc", lock=<optimized out>) at ./storage/innobase/include/sync0rw.ic:538
      #9  srv_master_evict_from_table_cache (pct_check=pct_check@entry=100) at ./storage/innobase/srv/srv0srv.cc:2085
      #10 0x0000558802fd6d66 in srv_master_do_idle_tasks () at ./storage/innobase/srv/srv0srv.cc:2333
      #11 srv_master_thread (arg=<optimized out>) at ./storage/innobase/srv/srv0srv.cc:2467
      #12 0x00007f6245e044a4 in start_thread (arg=0x7f620bfff700) at pthread_create.c:456
      #13 0x00007f6244357d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 4 (Thread 0x7f622effc700 (LWP 5615)):
      #0  0x00007f62459e96ca in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
      #1  0x0000558802f3f261 in LinuxAIOHandler::collect (this=this@entry=0x7f622effbaf0) at ./storage/innobase/os/os0file.cc:1936
      #2  0x0000558802f3f812 in LinuxAIOHandler::poll (this=this@entry=0x7f622effbaf0, m1=m1@entry=0x7f622effbb90, m2=m2@entry=0x7f622effbb98, request=request@entry=0x7f622effbba0) at ./storage/innobase/os/os0file.cc:2097
      #3  0x0000558802f40fd8 in os_aio_linux_handler (request=0x7f622effbba0, m2=0x7f622effbb98, m1=0x7f622effbb90, global_segment=<optimized out>) at ./storage/innobase/os/os0file.cc:2151
      #4  os_aio_handler (segment=segment@entry=0, m1=m1@entry=0x7f622effbb90, m2=m2@entry=0x7f622effbb98, request=request@entry=0x7f622effbba0) at ./storage/innobase/os/os0file.cc:5784
      #5  0x00005588030d3c0a in fil_aio_wait (segment=segment@entry=0) at ./storage/innobase/fil/fil0fil.cc:5115
      #6  0x0000558802fda478 in io_handler_thread (arg=<optimized out>) at ./storage/innobase/srv/srv0start.cc:332
      #7  0x00007f6245e044a4 in start_thread (arg=0x7f622effc700) at pthread_create.c:456
      #8  0x00007f6244357d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 3 (Thread 0x7f623e3cb700 (LWP 5220)):
      #0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
      #1  0x00007f623e58546d in gu::Lock::wait (this=0x7f623e3cab40, cond=...) at ./galerautils/src/gu_lock.hpp:59
      #2  galera::ServiceThd::thd_func (arg=0x558805f3b3e0) at ./galera/src/galera_service_thd.cpp:30
      #3  0x00007f6245e044a4 in start_thread (arg=0x7f623e3cb700) at pthread_create.c:456
      #4  0x00007f6244357d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 2 (Thread 0x7f6217fff700 (LWP 5618)):
      #0  0x00007f62459e96ca in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
      #1  0x0000558802f3f261 in LinuxAIOHandler::collect (this=this@entry=0x7f6217ffeaf0) at ./storage/innobase/os/os0file.cc:1936
      #2  0x0000558802f3f812 in LinuxAIOHandler::poll (this=this@entry=0x7f6217ffeaf0, m1=m1@entry=0x7f6217ffeb90, m2=m2@entry=0x7f6217ffeb98, request=request@entry=0x7f6217ffeba0) at ./storage/innobase/os/os0file.cc:2097
      #3  0x0000558802f40fd8 in os_aio_linux_handler (request=0x7f6217ffeba0, m2=0x7f6217ffeb98, m1=0x7f6217ffeb90, global_segment=<optimized out>) at ./storage/innobase/os/os0file.cc:2151
      #4  os_aio_handler (segment=segment@entry=3, m1=m1@entry=0x7f6217ffeb90, m2=m2@entry=0x7f6217ffeb98, request=request@entry=0x7f6217ffeba0) at ./storage/innobase/os/os0file.cc:5784
      #5  0x00005588030d3c0a in fil_aio_wait (segment=segment@entry=3) at ./storage/innobase/fil/fil0fil.cc:5115
      #6  0x0000558802fda478 in io_handler_thread (arg=<optimized out>) at ./storage/innobase/srv/srv0start.cc:332
      #7  0x00007f6245e044a4 in start_thread (arg=0x7f6217fff700) at pthread_create.c:456
      #8  0x00007f6244357d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 1 (Thread 0x7f62461a4440 (LWP 5218)):
      #0  __pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:57
      #1  0x000055880329a517 in my_write_core (sig=sig@entry=6) at ./mysys/stacktrace.c:382
      #2  0x0000558802d50abb in handle_fatal_signal (sig=6) at ./sql/signal_handler.cc:343
      #3  <signal handler called>
      #4  0x00007f624434e8bd in poll () at ../sysdeps/unix/syscall-template.S:84
      #5  0x00007f62443683be in __poll_chk (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>, fdslen=<optimized out>) at poll_chk.c:27
      #6  0x0000558802b16cbf in poll (__timeout=-1, __nfds=2, __fds=0x7ffca440c020) at /usr/include/x86_64-linux-gnu/bits/poll2.h:41
      #7  handle_connections_sockets () at ./sql/mysqld.cc:6724
      #8  0x0000558802b186ff in mysqld_main (argc=<optimized out>, argv=<optimized out>) at ./sql/mysqld.cc:6192
      #9  0x00007f624428f2e1 in __libc_start_main (main=0x558802af7b90 <main(int, char**)>, argc=22, argv=0x7ffca440c3c8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffca440c3b8) at ../csu/libc-start.c:291
      #10 0x0000558802b0aeea in _start ()
       
       - deleting it, already saved 0
       - found 'core' (0/0)
       
      Trying 'dbx' to get a backtrace
       
      Trying 'gdb' to get a backtrace from coredump /dev/shm/var/1/log/galera.galera_insert_multi-innodb/mysqld.1/data/core
      Core generated by '/usr/sbin/mysqld'
      Output from gdb follows. The first stack trace is from the failing thread.
      The following stack traces are from all threads (so the failing one is
      duplicated).
      --------------------------
      [New LWP 2597]
      [New LWP 2601]
      [New LWP 2598]
      [New LWP 2600]
      [New LWP 2608]
      [New LWP 2604]
      [New LWP 2625]
      [New LWP 2614]
      [New LWP 2616]
      [New LWP 2624]
      [New LWP 2621]
      [New LWP 2628]
      [New LWP 2620]
      [New LWP 2619]
      [New LWP 2617]
      [New LWP 2627]
      [New LWP 2631]
      [New LWP 2626]
      [New LWP 2622]
      [New LWP 2602]
      [New LWP 2599]
      [New LWP 3342]
      [New LWP 2623]
      [New LWP 2618]
      [New LWP 2615]
      [New LWP 2613]
      [New LWP 2612]
      [New LWP 2610]
      [New LWP 2609]
      [New LWP 2607]
      [New LWP 2606]
      [New LWP 2605]
      [New LWP 2603]
      [Thread debugging using libthread_db enabled]
      Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
      Core was generated by `/usr/sbin/mysqld --defaults-group-suffix=.1 --defaults-file=/dev/shm/var/1/my.c'.
      Program terminated with signal SIGABRT, Aborted.
      #0  __pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:57
      57	../sysdeps/unix/sysv/linux/pthread_kill.c: No such file or directory.
      [Current thread is 1 (Thread 0x7f390d7bd440 (LWP 2597))]
      #0  __pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:57
      #1  0x000055644ac4c517 in my_write_core (sig=sig@entry=6) at ./mysys/stacktrace.c:382
      #2  0x000055644a702abb in handle_fatal_signal (sig=6) at ./sql/signal_handler.cc:343
      #3  <signal handler called>
      #4  0x00007f390b9678bd in poll () at ../sysdeps/unix/syscall-template.S:84
      #5  0x00007f390b9813be in __poll_chk (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>, fdslen=<optimized out>) at poll_chk.c:27
      #6  0x000055644a4c8cbf in poll (__timeout=-1, __nfds=2, __fds=0x7ffd80136040) at /usr/include/x86_64-linux-gnu/bits/poll2.h:41
      #7  handle_connections_sockets () at ./sql/mysqld.cc:6724
      #8  0x000055644a4ca6ff in mysqld_main (argc=<optimized out>, argv=<optimized out>) at ./sql/mysqld.cc:6192
      #9  0x00007f390b8a82e1 in __libc_start_main (main=0x55644a4a9b90 <main(int, char**)>, argc=22, argv=0x7ffd801363e8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffd801363d8) at ../csu/libc-start.c:291
      #10 0x000055644a4bceea in _start ()
       
      Thread 33 (Thread 0x7f3904797700 (LWP 2603)):
      #0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
      #1  0x00007f3905d0951e in fifo_lock_get (q=0x7f39050e5010) at ./galerautils/src/gu_fifo.c:262
      #2  gu_fifo_get_head (q=0x7f39050e5010, err=err@entry=0x7f390479679c) at ./galerautils/src/gu_fifo.c:330
      #3  0x00007f3905bd0c95 in gcs_recv (conn=0x55644c509c40, action=action@entry=0x7f39047967f0) at ./gcs/src/gcs.cpp:1881
      #4  0x00007f3905ba91de in galera::Gcs::recv (act=..., this=<optimized out>) at ./galera/src/galera_gcs.hpp:119
      #5  galera::GcsActionSource::process (this=0x55644c502518, recv_ctx=0x7f38f40009a8, exit_loop=@0x7f390479687f: false) at ./galera/src/gcs_action_source.cpp:174
      #6  0x00007f3905b702cb in galera::ReplicatorSMM::async_recv (this=0x55644c501dc0, recv_ctx=0x7f38f40009a8) at ./galera/src/replicator_smm.cpp:362
      #7  0x00007f3905b4402b in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at ./galera/src/wsrep_provider.cpp:244
      #8  0x000055644a6a8ff8 in wsrep_replication_process (thd=0x7f38f40009a8) at ./sql/wsrep_thd.cc:486
      #9  0x000055644a69a727 in start_wsrep_THD (arg=arg@entry=0x55644c5530a8) at ./sql/wsrep_mysqld.cc:2173
      #10 0x000055644abfe551 in pfs_spawn_thread (arg=0x55644c552158) at ./storage/perfschema/pfs.cc:1869
      #11 0x00007f390d41d4a4 in start_thread (arg=0x7f3904797700) at pthread_create.c:456
      #12 0x00007f390b970d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 32 (Thread 0x7f38ebfff700 (LWP 2605)):
      #0  0x00007f390d0026ca in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
      #1  0x000055644a8f1261 in LinuxAIOHandler::collect (this=this@entry=0x7f38ebffeaf0) at ./storage/innobase/os/os0file.cc:1936
      #2  0x000055644a8f1812 in LinuxAIOHandler::poll (this=this@entry=0x7f38ebffeaf0, m1=m1@entry=0x7f38ebffeb90, m2=m2@entry=0x7f38ebffeb98, request=request@entry=0x7f38ebffeba0) at ./storage/innobase/os/os0file.cc:2097
      #3  0x000055644a8f2fd8 in os_aio_linux_handler (request=0x7f38ebffeba0, m2=0x7f38ebffeb98, m1=0x7f38ebffeb90, global_segment=<optimized out>) at ./storage/innobase/os/os0file.cc:2151
      #4  os_aio_handler (segment=segment@entry=1, m1=m1@entry=0x7f38ebffeb90, m2=m2@entry=0x7f38ebffeb98, request=request@entry=0x7f38ebffeba0) at ./storage/innobase/os/os0file.cc:5784
      #5  0x000055644aa85c0a in fil_aio_wait (segment=segment@entry=1) at ./storage/innobase/fil/fil0fil.cc:5115
      #6  0x000055644a98c478 in io_handler_thread (arg=<optimized out>) at ./storage/innobase/srv/srv0start.cc:332
      #7  0x00007f390d41d4a4 in start_thread (arg=0x7f38ebfff700) at pthread_create.c:456
      #8  0x00007f390b970d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 31 (Thread 0x7f38eb7fe700 (LWP 2606)):
      #0  0x00007f390d0026ca in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
      #1  0x000055644a8f1261 in LinuxAIOHandler::collect (this=this@entry=0x7f38eb7fdaf0) at ./storage/innobase/os/os0file.cc:1936
      #2  0x000055644a8f1812 in LinuxAIOHandler::poll (this=this@entry=0x7f38eb7fdaf0, m1=m1@entry=0x7f38eb7fdb90, m2=m2@entry=0x7f38eb7fdb98, request=request@entry=0x7f38eb7fdba0) at ./storage/innobase/os/os0file.cc:2097
      #3  0x000055644a8f2fd8 in os_aio_linux_handler (request=0x7f38eb7fdba0, m2=0x7f38eb7fdb98, m1=0x7f38eb7fdb90, global_segment=<optimized out>) at ./storage/innobase/os/os0file.cc:2151
      #4  os_aio_handler (segment=segment@entry=2, m1=m1@entry=0x7f38eb7fdb90, m2=m2@entry=0x7f38eb7fdb98, request=request@entry=0x7f38eb7fdba0) at ./storage/innobase/os/os0file.cc:5784
      #5  0x000055644aa85c0a in fil_aio_wait (segment=segment@entry=2) at ./storage/innobase/fil/fil0fil.cc:5115
      #6  0x000055644a98c478 in io_handler_thread (arg=<optimized out>) at ./storage/innobase/srv/srv0start.cc:332
      #7  0x00007f390d41d4a4 in start_thread (arg=0x7f38eb7fe700) at pthread_create.c:456
      #8  0x00007f390b970d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 30 (Thread 0x7f38eaffd700 (LWP 2607)):
      #0  0x00007f390d0026ca in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
      #1  0x000055644a8f1261 in LinuxAIOHandler::collect (this=this@entry=0x7f38eaffcaf0) at ./storage/innobase/os/os0file.cc:1936
      #2  0x000055644a8f1812 in LinuxAIOHandler::poll (this=this@entry=0x7f38eaffcaf0, m1=m1@entry=0x7f38eaffcb90, m2=m2@entry=0x7f38eaffcb98, request=request@entry=0x7f38eaffcba0) at ./storage/innobase/os/os0file.cc:2097
      #3  0x000055644a8f2fd8 in os_aio_linux_handler (request=0x7f38eaffcba0, m2=0x7f38eaffcb98, m1=0x7f38eaffcb90, global_segment=<optimized out>) at ./storage/innobase/os/os0file.cc:2151
      #4  os_aio_handler (segment=segment@entry=3, m1=m1@entry=0x7f38eaffcb90, m2=m2@entry=0x7f38eaffcb98, request=request@entry=0x7f38eaffcba0) at ./storage/innobase/os/os0file.cc:5784
      #5  0x000055644aa85c0a in fil_aio_wait (segment=segment@entry=3) at ./storage/innobase/fil/fil0fil.cc:5115
      #6  0x000055644a98c478 in io_handler_thread (arg=<optimized out>) at ./storage/innobase/srv/srv0start.cc:332
      #7  0x00007f390d41d4a4 in start_thread (arg=0x7f38eaffd700) at pthread_create.c:456
      #8  0x00007f390b970d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 29 (Thread 0x7f38e9ffb700 (LWP 2609)):
      #0  0x00007f390d0026ca in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
      #1  0x000055644a8f1261 in LinuxAIOHandler::collect (this=this@entry=0x7f38e9ffaaf0) at ./storage/innobase/os/os0file.cc:1936
      #2  0x000055644a8f1812 in LinuxAIOHandler::poll (this=this@entry=0x7f38e9ffaaf0, m1=m1@entry=0x7f38e9ffab90, m2=m2@entry=0x7f38e9ffab98, request=request@entry=0x7f38e9ffaba0) at ./storage/innobase/os/os0file.cc:2097
      #3  0x000055644a8f2fd8 in os_aio_linux_handler (request=0x7f38e9ffaba0, m2=0x7f38e9ffab98, m1=0x7f38e9ffab90, global_segment=<optimized out>) at ./storage/innobase/os/os0file.cc:2151
      #4  os_aio_handler (segment=segment@entry=5, m1=m1@entry=0x7f38e9ffab90, m2=m2@entry=0x7f38e9ffab98, request=request@entry=0x7f38e9ffaba0) at ./storage/innobase/os/os0file.cc:5784
      #5  0x000055644aa85c0a in fil_aio_wait (segment=segment@entry=5) at ./storage/innobase/fil/fil0fil.cc:5115
      #6  0x000055644a98c478 in io_handler_thread (arg=<optimized out>) at ./storage/innobase/srv/srv0start.cc:332
      #7  0x00007f390d41d4a4 in start_thread (arg=0x7f38e9ffb700) at pthread_create.c:456
      #8  0x00007f390b970d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 28 (Thread 0x7f38e97fa700 (LWP 2610)):
      #0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
      #1  0x000055644a8f6ea2 in os_event::timed_wait (this=this@entry=0x55644c597d70, abstime=abstime@entry=0x7f38e97f9970) at ./storage/innobase/os/os0event.cc:275
      #2  0x000055644a8f70f1 in os_event::wait_time_low (this=0x55644c597d70, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=reset_sig_count@entry=1051) at ./storage/innobase/os/os0event.cc:385
      #3  0x000055644a8f749a in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=reset_sig_count@entry=1051) at ./storage/innobase/os/os0event.cc:490
      #4  0x000055644aa326ea in pc_sleep_if_needed (cur_time=1162090, sig_count=1051, next_loop_time=1163090) at ./storage/innobase/buf/buf0flu.cc:2683
      #5  buf_flush_page_cleaner_coordinator () at ./storage/innobase/buf/buf0flu.cc:3149
      #6  0x00007f390d41d4a4 in start_thread (arg=0x7f38e97fa700) at pthread_create.c:456
      #7  0x00007f390b970d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 27 (Thread 0x7f38e3fff700 (LWP 2612)):
      #0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
      #1  0x000055644a8f6ea2 in os_event::timed_wait (this=this@entry=0x55644c6adc10, abstime=abstime@entry=0x7f38e3ffec00) at ./storage/innobase/os/os0event.cc:275
      #2  0x000055644a8f70f1 in os_event::wait_time_low (this=this@entry=0x55644c6adc10, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=1) at ./storage/innobase/os/os0event.cc:385
      #3  0x000055644a8f749a in os_event_wait_time_low (event=event@entry=0x55644c6adc10, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=<optimized out>) at ./storage/innobase/os/os0event.cc:490
      #4  0x000055644a8ce410 in lock_wait_timeout_thread () at ./storage/innobase/lock/lock0wait.cc:525
      #5  0x00007f390d41d4a4 in start_thread (arg=0x7f38e3fff700) at pthread_create.c:456
      #6  0x00007f390b970d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 26 (Thread 0x7f38e37fe700 (LWP 2613)):
      #0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
      #1  0x000055644a8f6ea2 in os_event::timed_wait (this=this@entry=0x55644c597c20, abstime=abstime@entry=0x7f38e37fdb60) at ./storage/innobase/os/os0event.cc:275
      #2  0x000055644a8f70f1 in os_event::wait_time_low (this=0x55644c597c20, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=1) at ./storage/innobase/os/os0event.cc:385
      #3  0x000055644a8f749a in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=<optimized out>) at ./storage/innobase/os/os0event.cc:490
      #4  0x000055644a987a1c in srv_error_monitor_thread () at ./storage/innobase/srv/srv0srv.cc:1912
      #5  0x00007f390d41d4a4 in start_thread (arg=0x7f38e37fe700) at pthread_create.c:456
      #6  0x00007f390b970d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 25 (Thread 0x7f38e8ff9700 (LWP 2615)):
      #0  0x00007f390d4267fd in nanosleep () at ../sysdeps/unix/syscall-template.S:84
      #1  0x000055644a8f7750 in os_thread_sleep (tm=tm@entry=1000000) at ./storage/innobase/os/os0thread.cc:231
      #2  0x000055644a988af2 in srv_master_sleep () at ./storage/innobase/srv/srv0srv.cc:2419
      #3  srv_master_thread (arg=<optimized out>) at ./storage/innobase/srv/srv0srv.cc:2459
      #4  0x00007f390d41d4a4 in start_thread (arg=0x7f38e8ff9700) at pthread_create.c:456
      #5  0x00007f390b970d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 24 (Thread 0x7f38e17fa700 (LWP 2618)):
      #0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
      #1  0x000055644a8f74e6 in os_event::wait (this=0x55644c597a60) at ./storage/innobase/os/os0event.cc:158
      #2  os_event::wait_low (reset_sig_count=669, this=0x55644c597a60) at ./storage/innobase/os/os0event.cc:325
      #3  os_event_wait_low (event=0x55644c597a60, reset_sig_count=reset_sig_count@entry=669) at ./storage/innobase/os/os0event.cc:507
      #4  0x000055644a98aa72 in srv_resume_thread (timeout_usec=0, wait=true, sig_count=669, slot=0x55644b4afc00 <srv_sys+192>) at ./storage/innobase/srv/srv0srv.cc:937
      #5  srv_purge_coordinator_suspend (rseg_history_len=<optimized out>, slot=<optimized out>) at ./storage/innobase/srv/srv0srv.cc:2778
      #6  srv_purge_coordinator_thread (arg=<optimized out>) at ./storage/innobase/srv/srv0srv.cc:2865
      #7  0x00007f390d41d4a4 in start_thread (arg=0x7f38e17fa700) at pthread_create.c:456
      #8  0x00007f390b970d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 23 (Thread 0x7f38daffd700 (LWP 2623)):
      #0  0x00007f390d4267fd in nanosleep () at ../sysdeps/unix/syscall-template.S:84
      #1  0x000055644a8f7750 in os_thread_sleep (tm=tm@entry=1000000) at ./storage/innobase/os/os0thread.cc:231
      #2  0x000055644aa11042 in btr_defragment_thread () at ./storage/innobase/btr/btr0defragment.cc:722
      #3  0x00007f390d41d4a4 in start_thread (arg=0x7f38daffd700) at pthread_create.c:456
      #4  0x00007f390b970d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 22 (Thread 0x7f38fcd33700 (LWP 3342)):
      #0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
      #1  0x000055644a4c0dc0 in inline_mysql_cond_timedwait (src_file=0x55644acc75e8 "/home/buildbot/buildbot/build/mariadb-10.2.37/sql/mysqld.cc", src_line=3021, abstime=0x7f38fcd32c30, mutex=0x55644b530600 <LOCK_thread_cache>, that=0x55644b532be0 <COND_thread_cache>) at ./include/mysql/psi/mysql_thread.h:1182
      #2  cache_thread (thd=0x7f38800009a8) at ./sql/mysqld.cc:3020
      #3  one_thread_per_connection_end (thd=0x7f38800009a8, put_in_cache=<optimized out>) at ./sql/mysqld.cc:3109
      #4  0x000055644a63a3f1 in do_handle_one_connection (connect=connect@entry=0x55644ca62cd8) at ./sql/sql_connect.cc:1355
      #5  0x000055644a63a5ed in handle_one_connection (arg=arg@entry=0x55644ca62cd8) at ./sql/sql_connect.cc:1241
      #6  0x000055644abfe551 in pfs_spawn_thread (arg=0x55644ca3cb58) at ./storage/perfschema/pfs.cc:1869
      #7  0x00007f390d41d4a4 in start_thread (arg=0x7f38fcd33700) at pthread_create.c:456
      #8  0x00007f390b970d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 21 (Thread 0x7f3904fe3700 (LWP 2599)):
      #0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
      #1  0x00007f3905b9e46d in gu::Lock::wait (this=0x7f3904fe2b40, cond=...) at ./galerautils/src/gu_lock.hpp:59
      #2  galera::ServiceThd::thd_func (arg=0x55644c5023e0) at ./galera/src/galera_service_thd.cpp:30
      #3  0x00007f390d41d4a4 in start_thread (arg=0x7f3904fe3700) at pthread_create.c:456
      #4  0x00007f390b970d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 20 (Thread 0x7f39047e2700 (LWP 2602)):
      #0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
      #1  0x000055644a6a971d in inline_mysql_cond_wait (src_file=0x55644acfca20 "/home/buildbot/buildbot/build/mariadb-10.2.37/sql/wsrep_thd.cc", src_line=632, mutex=<optimized out>, that=<optimized out>) at ./include/mysql/psi/mysql_thread.h:1145
      #2  wsrep_rollback_process (thd=0x7f38ec0009a8) at ./sql/wsrep_thd.cc:632
      #3  0x000055644a69a727 in start_wsrep_THD (arg=arg@entry=0x55644c552b78) at ./sql/wsrep_mysqld.cc:2173
      #4  0x000055644abfe551 in pfs_spawn_thread (arg=0x55644c551b88) at ./storage/perfschema/pfs.cc:1869
      #5  0x00007f390d41d4a4 in start_thread (arg=0x7f39047e2700) at pthread_create.c:456
      #6  0x00007f390b970d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 19 (Thread 0x7f38db7fe700 (LWP 2622)):
      #0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
      #1  0x000055644a8f74e6 in os_event::wait (this=0x55644c597d00) at ./storage/innobase/os/os0event.cc:158
      #2  os_event::wait_low (reset_sig_count=1, this=0x55644c597d00) at ./storage/innobase/os/os0event.cc:325
      #3  os_event_wait_low (event=0x55644c597d00, reset_sig_count=reset_sig_count@entry=0) at ./storage/innobase/os/os0event.cc:507
      #4  0x000055644aa27877 in buf_dump_thread () at ./storage/innobase/buf/buf0dump.cc:806
      #5  0x00007f390d41d4a4 in start_thread (arg=0x7f38db7fe700) at pthread_create.c:456
      #6  0x00007f390b970d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 18 (Thread 0x7f38d97fa700 (LWP 2626)):
      #0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
      #1  0x000055644ab2ff7e in inline_mysql_cond_timedwait (src_file=0x55644adcd338 "/home/buildbot/buildbot/build/mariadb-10.2.37/storage/maria/ma_servicethread.c", src_line=116, abstime=0x7f38d97f9c50, mutex=0x55644bdaeb20 <LOCK_checkpoint>, that=0x55644bdaeae0 <COND_checkpoint>) at ./include/mysql/psi/mysql_thread.h:1182
      #2  my_service_thread_sleep (control=control@entry=0x55644b4b19c0 <checkpoint_control>, sleep_time=<optimized out>) at ./storage/maria/ma_servicethread.c:115
      #3  0x000055644ab279a1 in ma_checkpoint_background (arg=arg@entry=0x1e) at ./storage/maria/ma_checkpoint.c:709
      #4  0x000055644abfe551 in pfs_spawn_thread (arg=0x55644c70d1b8) at ./storage/perfschema/pfs.cc:1869
      #5  0x00007f390d41d4a4 in start_thread (arg=0x7f38d97fa700) at pthread_create.c:456
      #6  0x00007f390b970d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 17 (Thread 0x7f390407b700 (LWP 2631)):
      #0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
      #1  0x000055644a4c0dc0 in inline_mysql_cond_timedwait (src_file=0x55644acc75e8 "/home/buildbot/buildbot/build/mariadb-10.2.37/sql/mysqld.cc", src_line=3021, abstime=0x7f390407ac30, mutex=0x55644b530600 <LOCK_thread_cache>, that=0x55644b532be0 <COND_thread_cache>) at ./include/mysql/psi/mysql_thread.h:1182
      #2  cache_thread (thd=0x7f388c0009a8) at ./sql/mysqld.cc:3020
      #3  one_thread_per_connection_end (thd=0x7f388c0009a8, put_in_cache=<optimized out>) at ./sql/mysqld.cc:3109
      #4  0x000055644a63a3f1 in do_handle_one_connection (connect=connect@entry=0x55644ca735d8) at ./sql/sql_connect.cc:1355
      #5  0x000055644a63a5ed in handle_one_connection (arg=arg@entry=0x55644ca735d8) at ./sql/sql_connect.cc:1241
      #6  0x000055644abfe551 in pfs_spawn_thread (arg=0x55644ca13008) at ./storage/perfschema/pfs.cc:1869
      #7  0x00007f390d41d4a4 in start_thread (arg=0x7f390407b700) at pthread_create.c:456
      #8  0x00007f390b970d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 16 (Thread 0x7f3904111700 (LWP 2627)):
      #0  do_sigwait (set=<optimized out>, set@entry=0x7f3904110c90, sig=sig@entry=0x7f3904110c38) at ../sysdeps/unix/sysv/linux/sigwait.c:64
      #1  0x00007f390d426d0d in __sigwait (set=set@entry=0x7f3904110c90, sig=sig@entry=0x7f3904110c38) at ../sysdeps/unix/sysv/linux/sigwait.c:96
      #2  0x000055644a4bf48b in signal_hand (arg=arg@entry=0x0) at ./sql/mysqld.cc:3557
      #3  0x000055644abfe551 in pfs_spawn_thread (arg=0x55644ca00458) at ./storage/perfschema/pfs.cc:1869
      #4  0x00007f390d41d4a4 in start_thread (arg=0x7f3904111700) at pthread_create.c:456
      #5  0x00007f390b970d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 15 (Thread 0x7f38e1ffb700 (LWP 2617)):
      #0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
      #1  0x000055644a8f6ea2 in os_event::timed_wait (this=this@entry=0x55644c7095f0, abstime=abstime@entry=0x7f38e1ffaa40) at ./storage/innobase/os/os0event.cc:275
      #2  0x000055644a8f70f1 in os_event::wait_time_low (this=0x55644c7095f0, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=reset_sig_count@entry=7) at ./storage/innobase/os/os0event.cc:385
      #3  0x000055644a8f749a in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=reset_sig_count@entry=7) at ./storage/innobase/os/os0event.cc:490
      #4  0x000055644a9dd212 in ib_wqueue_timedwait (wq=wq@entry=0x55644c6fcc70, wait_in_usecs=wait_in_usecs@entry=5000000) at ./storage/innobase/ut/ut0wqueue.cc:156
      #5  0x000055644aac2f4a in fts_optimize_thread (arg=<optimized out>) at ./storage/innobase/fts/fts0opt.cc:2829
      #6  0x00007f390d41d4a4 in start_thread (arg=0x7f38e1ffb700) at pthread_create.c:456
      #7  0x00007f390b970d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 14 (Thread 0x7f38e0ff9700 (LWP 2619)):
      #0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
      #1  0x000055644a8f74e6 in os_event::wait (this=0x55644c597b40) at ./storage/innobase/os/os0event.cc:158
      #2  os_event::wait_low (reset_sig_count=1418, this=0x55644c597b40) at ./storage/innobase/os/os0event.cc:325
      #3  os_event_wait_low (event=0x55644c597b40, reset_sig_count=reset_sig_count@entry=0) at ./storage/innobase/os/os0event.cc:507
      #4  0x000055644a9895af in srv_resume_thread (sig_count=0, timeout_usec=0, wait=true, slot=0x55644b4afc70 <srv_sys+304>) at ./storage/innobase/srv/srv0srv.cc:937
      #5  srv_worker_thread (arg=<optimized out>) at ./storage/innobase/srv/srv0srv.cc:2617
      #6  0x00007f390d41d4a4 in start_thread (arg=0x7f38e0ff9700) at pthread_create.c:456
      #7  0x00007f390b970d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 13 (Thread 0x7f38dbfff700 (LWP 2620)):
      #0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
      #1  0x000055644a8f74e6 in os_event::wait (this=0x55644c597ad0) at ./storage/innobase/os/os0event.cc:158
      #2  os_event::wait_low (reset_sig_count=1361, this=0x55644c597ad0) at ./storage/innobase/os/os0event.cc:325
      #3  os_event_wait_low (event=0x55644c597ad0, reset_sig_count=reset_sig_count@entry=0) at ./storage/innobase/os/os0event.cc:507
      #4  0x000055644a9895af in srv_resume_thread (sig_count=0, timeout_usec=0, wait=true, slot=0x55644b4afc38 <srv_sys+248>) at ./storage/innobase/srv/srv0srv.cc:937
      #5  srv_worker_thread (arg=<optimized out>) at ./storage/innobase/srv/srv0srv.cc:2617
      #6  0x00007f390d41d4a4 in start_thread (arg=0x7f38dbfff700) at pthread_create.c:456
      #7  0x00007f390b970d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 12 (Thread 0x7f39040c6700 (LWP 2628)):
      #0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
      #1  0x000055644a5595d8 in inline_mysql_cond_wait (that=0x55644b5347c0 <COND_manager>, mutex=0x55644b534800 <LOCK_manager>, src_file=0x55644acd7b98 "/home/buildbot/buildbot/build/mariadb-10.2.37/sql/sql_manager.cc", src_line=102) at ./include/mysql/psi/mysql_thread.h:1145
      #2  handle_manager (arg=arg@entry=0x0) at ./sql/sql_manager.cc:102
      #3  0x000055644abfe551 in pfs_spawn_thread (arg=0x55644ca76668) at ./storage/perfschema/pfs.cc:1869
      #4  0x00007f390d41d4a4 in start_thread (arg=0x7f39040c6700) at pthread_create.c:456
      #5  0x00007f390b970d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 11 (Thread 0x7f38d37fe700 (LWP 2621)):
      #0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
      #1  0x000055644a8f74e6 in os_event::wait (this=0x55644c597bb0) at ./storage/innobase/os/os0event.cc:158
      #2  os_event::wait_low (reset_sig_count=1351, this=0x55644c597bb0) at ./storage/innobase/os/os0event.cc:325
      #3  os_event_wait_low (event=0x55644c597bb0, reset_sig_count=reset_sig_count@entry=0) at ./storage/innobase/os/os0event.cc:507
      #4  0x000055644a9895af in srv_resume_thread (sig_count=0, timeout_usec=0, wait=true, slot=0x55644b4afca8 <srv_sys+360>) at ./storage/innobase/srv/srv0srv.cc:937
      #5  srv_worker_thread (arg=<optimized out>) at ./storage/innobase/srv/srv0srv.cc:2617
      #6  0x00007f390d41d4a4 in start_thread (arg=0x7f38d37fe700) at pthread_create.c:456
      #7  0x00007f390b970d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 10 (Thread 0x7f38da7fc700 (LWP 2624)):
      #0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
      #1  0x000055644a8f74e6 in os_event::wait (this=0x55644c597de0) at ./storage/innobase/os/os0event.cc:158
      #2  os_event::wait_low (reset_sig_count=1, this=0x55644c597de0) at ./storage/innobase/os/os0event.cc:325
      #3  os_event_wait_low (event=0x55644c597de0, reset_sig_count=reset_sig_count@entry=0) at ./storage/innobase/os/os0event.cc:507
      #4  0x000055644aa1fcac in buf_resize_thread () at ./storage/innobase/buf/buf0buf.cc:3109
      #5  0x00007f390d41d4a4 in start_thread (arg=0x7f38da7fc700) at pthread_create.c:456
      #6  0x00007f390b970d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 9 (Thread 0x7f38e27fc700 (LWP 2616)):
      #0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
      #1  0x000055644a8f6ea2 in os_event::timed_wait (this=this@entry=0x55644c6bf4a0, abstime=abstime@entry=0x7f38e27fbbe0) at ./storage/innobase/os/os0event.cc:275
      #2  0x000055644a8f70f1 in os_event::wait_time_low (this=0x55644c6bf4a0, time_in_usec=time_in_usec@entry=10000000, reset_sig_count=1, reset_sig_count@entry=0) at ./storage/innobase/os/os0event.cc:385
      #3  0x000055644a8f749a in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=10000000, reset_sig_count=reset_sig_count@entry=0) at ./storage/innobase/os/os0event.cc:490
      #4  0x000055644aa7a465 in dict_stats_thread () at ./storage/innobase/dict/dict0stats_bg.cc:475
      #5  0x00007f390d41d4a4 in start_thread (arg=0x7f38e27fc700) at pthread_create.c:456
      #6  0x00007f390b970d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 8 (Thread 0x7f38e2ffd700 (LWP 2614)):
      #0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
      #1  0x000055644a8f6ea2 in os_event::timed_wait (this=this@entry=0x55644c597c90, abstime=abstime@entry=0x7f38e2ffcc90) at ./storage/innobase/os/os0event.cc:275
      #2  0x000055644a8f70f1 in os_event::wait_time_low (this=0x55644c597c90, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=1) at ./storage/innobase/os/os0event.cc:385
      #3  0x000055644a8f749a in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=<optimized out>) at ./storage/innobase/os/os0event.cc:490
      #4  0x000055644a987382 in srv_monitor_thread () at ./storage/innobase/srv/srv0srv.cc:1748
      #5  0x00007f390d41d4a4 in start_thread (arg=0x7f38e2ffd700) at pthread_create.c:456
      #6  0x00007f390b970d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 7 (Thread 0x7f38d9ffb700 (LWP 2625)):
      #0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
      #1  0x000055644a87945b in inline_mysql_cond_wait (src_file=0x55644ad7da08 "/home/buildbot/buildbot/build/mariadb-10.2.37/storage/innobase/handler/ha_innodb.cc", src_line=348, mutex=0x7f38d9ffacc0, that=0x7f38d9ffacf0) at ./include/mysql/psi/mysql_thread.h:1156
      #2  thd_destructor_proxy () at ./storage/innobase/handler/ha_innodb.cc:348
      #3  0x000055644abfe551 in pfs_spawn_thread (arg=0x55644c70a798) at ./storage/perfschema/pfs.cc:1869
      #4  0x00007f390d41d4a4 in start_thread (arg=0x7f38d9ffb700) at pthread_create.c:456
      #5  0x00007f390b970d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 6 (Thread 0x7f38fcbc8700 (LWP 2604)):
      #0  0x00007f390d0026ca in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
      #1  0x000055644a8f1261 in LinuxAIOHandler::collect (this=this@entry=0x7f38fcbc7af0) at ./storage/innobase/os/os0file.cc:1936
      #2  0x000055644a8f1812 in LinuxAIOHandler::poll (this=this@entry=0x7f38fcbc7af0, m1=m1@entry=0x7f38fcbc7b90, m2=m2@entry=0x7f38fcbc7b98, request=request@entry=0x7f38fcbc7ba0) at ./storage/innobase/os/os0file.cc:2097
      #3  0x000055644a8f2fd8 in os_aio_linux_handler (request=0x7f38fcbc7ba0, m2=0x7f38fcbc7b98, m1=0x7f38fcbc7b90, global_segment=<optimized out>) at ./storage/innobase/os/os0file.cc:2151
      #4  os_aio_handler (segment=segment@entry=0, m1=m1@entry=0x7f38fcbc7b90, m2=m2@entry=0x7f38fcbc7b98, request=request@entry=0x7f38fcbc7ba0) at ./storage/innobase/os/os0file.cc:5784
      #5  0x000055644aa85c0a in fil_aio_wait (segment=segment@entry=0) at ./storage/innobase/fil/fil0fil.cc:5115
      #6  0x000055644a98c478 in io_handler_thread (arg=<optimized out>) at ./storage/innobase/srv/srv0start.cc:332
      #7  0x00007f390d41d4a4 in start_thread (arg=0x7f38fcbc8700) at pthread_create.c:456
      #8  0x00007f390b970d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 5 (Thread 0x7f38ea7fc700 (LWP 2608)):
      #0  0x00007f390d0026ca in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
      #1  0x000055644a8f1261 in LinuxAIOHandler::collect (this=this@entry=0x7f38ea7fbaf0) at ./storage/innobase/os/os0file.cc:1936
      #2  0x000055644a8f1812 in LinuxAIOHandler::poll (this=this@entry=0x7f38ea7fbaf0, m1=m1@entry=0x7f38ea7fbb90, m2=m2@entry=0x7f38ea7fbb98, request=request@entry=0x7f38ea7fbba0) at ./storage/innobase/os/os0file.cc:2097
      #3  0x000055644a8f2fd8 in os_aio_linux_handler (request=0x7f38ea7fbba0, m2=0x7f38ea7fbb98, m1=0x7f38ea7fbb90, global_segment=<optimized out>) at ./storage/innobase/os/os0file.cc:2151
      #4  os_aio_handler (segment=segment@entry=4, m1=m1@entry=0x7f38ea7fbb90, m2=m2@entry=0x7f38ea7fbb98, request=request@entry=0x7f38ea7fbba0) at ./storage/innobase/os/os0file.cc:5784
      #5  0x000055644aa85c0a in fil_aio_wait (segment=segment@entry=4) at ./storage/innobase/fil/fil0fil.cc:5115
      #6  0x000055644a98c478 in io_handler_thread (arg=<optimized out>) at ./storage/innobase/srv/srv0start.cc:332
      #7  0x00007f390d41d4a4 in start_thread (arg=0x7f38ea7fc700) at pthread_create.c:456
      #8  0x00007f390b970d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 4 (Thread 0x7f38fe7fc700 (LWP 2600)):
      #0  0x00007f390b971303 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
      #1  0x00007f3905bb3cff in asio::detail::epoll_reactor::run (this=0x55644c53af00, block=block@entry=true, ops=...) at /usr/include/asio/detail/impl/epoll_reactor.ipp:391
      #2  0x00007f3905c02519 in asio::detail::task_io_service::do_run_one (ec=..., this_thread=..., lock=..., this=0x55644c539fd0) at /usr/include/asio/detail/impl/task_io_service.ipp:355
      #3  asio::detail::task_io_service::run (ec=..., this=0x55644c539fd0) at /usr/include/asio/detail/impl/task_io_service.ipp:148
      #4  asio::io_service::run (this=<optimized out>) at /usr/include/asio/impl/io_service.ipp:58
      #5  gcomm::AsioProtonet::event_loop (this=<optimized out>, period=...) at ./gcomm/src/asio_protonet.cpp:116
      #6  0x00007f3905be4469 in GCommConn::run (this=0x55644c53a150) at ./gcs/src/gcs_gcomm.cpp:433
      #7  0x00007f3905bed919 in GCommConn::run_fn (arg=<optimized out>) at ./gcs/src/gcs_gcomm.cpp:174
      #8  0x00007f390d41d4a4 in start_thread (arg=0x7f38fe7fc700) at pthread_create.c:456
      #9  0x00007f390b970d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 3 (Thread 0x7f3906772700 (LWP 2598)):
      #0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
      #1  0x000055644ac50d81 in inline_mysql_cond_timedwait (that=0x55644bdbe500 <COND_timer>, mutex=0x55644bdbe540 <LOCK_timer>, src_file=0x55644ade1ed8 "/home/buildbot/buildbot/build/mariadb-10.2.37/mysys/thr_timer.c", src_line=292, abstime=0x7f3906771ce0) at ./include/mysql/psi/mysql_thread.h:1182
      #2  timer_handler (arg=arg@entry=0x0) at ./mysys/thr_timer.c:292
      #3  0x000055644abfe551 in pfs_spawn_thread (arg=0x55644c4d3098) at ./storage/perfschema/pfs.cc:1869
      #4  0x00007f390d41d4a4 in start_thread (arg=0x7f3906772700) at pthread_create.c:456
      #5  0x00007f390b970d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 2 (Thread 0x7f38fdffb700 (LWP 2601)):
      #0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
      #1  0x00007f3905be15b8 in gu::Lock::wait (this=0x7f38fdff9d60, cond=...) at ./galerautils/src/gu_lock.hpp:59
      #2  RecvBuf::front (timeout=..., this=0x55644c53a348) at ./gcs/src/gcs_gcomm.cpp:114
      #3  gcomm_recv (backend=<optimized out>, msg=0x55644c509e80, timeout=9223372035999999999) at ./gcs/src/gcs_gcomm.cpp:608
      #4  0x00007f3905bd7cc8 in core_msg_recv (timeout=9223372035999999999, recv_msg=0x55644c509e80, backend=0x55644c509fa8) at ./gcs/src/gcs_core.cpp:469
      #5  gcs_core_recv (conn=0x55644c509e20, recv_act=recv_act@entry=0x7f38fdffacf0, timeout=9223372035999999999) at ./gcs/src/gcs_core.cpp:1102
      #6  0x00007f3905bd2fb6 in gcs_recv_thread (arg=0x55644c509c40) at ./gcs/src/gcs.cpp:1277
      #7  0x00007f390d41d4a4 in start_thread (arg=0x7f38fdffb700) at pthread_create.c:456
      #8  0x00007f390b970d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
       
      Thread 1 (Thread 0x7f390d7bd440 (LWP 2597)):
      #0  __pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:57
      #1  0x000055644ac4c517 in my_write_core (sig=sig@entry=6) at ./mysys/stacktrace.c:382
      #2  0x000055644a702abb in handle_fatal_signal (sig=6) at ./sql/signal_handler.cc:343
      #3  <signal handler called>
      #4  0x00007f390b9678bd in poll () at ../sysdeps/unix/syscall-template.S:84
      #5  0x00007f390b9813be in __poll_chk (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>, fdslen=<optimized out>) at poll_chk.c:27
      #6  0x000055644a4c8cbf in poll (__timeout=-1, __nfds=2, __fds=0x7ffd80136040) at /usr/include/x86_64-linux-gnu/bits/poll2.h:41
      #7  handle_connections_sockets () at ./sql/mysqld.cc:6724
      #8  0x000055644a4ca6ff in mysqld_main (argc=<optimized out>, argv=<optimized out>) at ./sql/mysqld.cc:6192
      #9  0x00007f390b8a82e1 in __libc_start_main (main=0x55644a4a9b90 <main(int, char**)>, argc=22, argv=0x7ffd801363e8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffd801363d8) at ../csu/libc-start.c:291
      #10 0x000055644a4bceea in _start ()
       
       - deleting it, already saved 0
       - saving '/dev/shm/var/1/log/galera.galera_insert_multi-innodb/' to '/dev/shm/var/log/galera.galera_insert_multi-innodb/'
      

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              jplindst Jan Lindström
              Reporter:
              stepan.patryshev Stepan Patryshev (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated: