Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-20909

Galera test failure on galera.galera_gcs_fc_limit: Server crash with signal 6

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 10.4.9
    • Fix Version/s: 10.4.12
    • Component/s: Tests
    • Labels:
      None
    • Environment:
      OS: CentOS Linux release 7.6.1810 (Core).

      Description

      MariaDB Version 10.4.9-MariaDB-debug: Repository: MariaDB/server; branch 10.4; Revision 9afbb1069ab4178741b76adefacd3f6a0ed51369; debug built from sources.

      Galera lib 26.4.3(r4535): Repository: MariaDB/galera; branch mariadb-4.x; Revision 752664dc3c7065d8e0c73ac99d0028a5f84eb250; debug built from sources.

      It sporadically fails on SSD and always on HDD during suite running without --parallel.

      Run:

      ../mtr --suite galera --testcase-timeout=90 --force --big-test --max-test-fail=0 --timer --timestamp --timediff

      stdout.log: (the whole file is attached: 191027_HDD_stdout.log)
      Program terminated with signal 6, Aborted.

      191028  1:52:29 +5462.431 galera.galera_gcs_fc_limit 'innodb'      [ fail ]  timeout after 5400 seconds
              Test ended at 2019-10-28 01:52:29
       
      Test case timeout after 5400 seconds
       
      == /home/stepan/mariadb/10.4/github/mysql-test/var/log/galera_gcs_fc_limit.log == 
      connection node_2;
      connection node_1;
      CREATE TABLE t1 (f1 INTEGER NOT NULL PRIMARY KEY AUTO_INCREMENT,B INTEGER) ENGINE=InnoDB;
      INSERT INTO t1 VALUES (1,1);
      connection node_2;
      SET SESSION wsrep_sync_wait=15;
      SELECT COUNT(*) FROM t1;
      COUNT(*)
      1
      SET GLOBAL wsrep_provider_options = 'gcs.fc_limit=1';
      LOCK TABLE t1 WRITE;
      connection node_1;
      FLUSH STATUS;
      INSERT INTO t1 VALUES (2,2);
      INSERT INTO t1 VALUES (3,3);
      INSERT INTO t1 VALUES (4,4);
       
       == /home/stepan/mariadb/10.4/github/mysql-test/var/tmp/analyze-timeout-mysqld.2.err ==
      mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/mariadb/10.4/github/mysql-test/var/tmp/mysqld.2.soc' (111)
       
       == /home/stepan/mariadb/10.4/github/mysql-test/var/tmp/analyze-timeout-mysqld.1.err ==
      mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/mariadb/10.4/github/mysql-test/var/tmp/mysqld.1.soc' (111)
       
       
       - found 'core.31799' (0/5)
       
      Trying 'dbx' to get a backtrace
       
      Trying 'gdb' to get a backtrace from coredump /home/stepan/mariadb/10.4/github/mysql-test/var/log/galera.galera_gcs_fc_limit-innodb/mysqld.1/data/core.31799
      Core generated by '/home/stepan/mariadb/10.4/github/sql/mysqld'
      Output from gdb follows. The first stack trace is from the failing thread.
      The following stack traces are from all threads (so the failing one is
      duplicated).
      --------------------------
      [New LWP 31799]
      [New LWP 31804]
      [New LWP 31801]
      [New LWP 31806]
      [New LWP 31803]
      [New LWP 31805]
      [New LWP 31810]
      [New LWP 31812]
      [New LWP 31807]
      [New LWP 31821]
      [New LWP 31813]
      [New LWP 31822]
      [New LWP 31815]
      [New LWP 31828]
      [New LWP 31816]
      [New LWP 31802]
      [New LWP 31818]
      [New LWP 31823]
      [New LWP 31824]
      [New LWP 31832]
      [New LWP 31833]
      [New LWP 31827]
      [New LWP 31826]
      [New LWP 32066]
      [New LWP 31811]
      [New LWP 31817]
      [New LWP 31829]
      [New LWP 31825]
      [New LWP 31809]
      [New LWP 31808]
      [New LWP 31819]
      [New LWP 31800]
      [New LWP 31820]
      [Thread debugging using libthread_db enabled]
      Using host libthread_db library "/lib64/libthread_db.so.1".
      Core was generated by `/home/stepan/mariadb/10.4/github/sql/mysqld --defaults-group-suffix=.1 --defaul'.
      Program terminated with signal 6, Aborted.
      #0  0x00007fd30ed4f9d1 in pthread_kill () from /lib64/libpthread.so.0
      #0  0x00007fd30ed4f9d1 in pthread_kill () from /lib64/libpthread.so.0
      #1  0x000055612cc300f6 in my_write_core (sig=6) at /home/stepan/mariadb/10.4/github/mysys/stacktrace.c:481
      #2  0x000055612c39c964 in handle_fatal_signal (sig=6) at /home/stepan/mariadb/10.4/github/sql/signal_handler.cc:343
      #3  <signal handler called>
      #4  0x00007fd30d0fd20d in poll () from /lib64/libc.so.6
      #5  0x000055612bf012db in handle_connections_sockets () at /home/stepan/mariadb/10.4/github/sql/mysqld.cc:6455
      #6  0x000055612bf00576 in mysqld_main (argc=162, argv=0x55612f220130) at /home/stepan/mariadb/10.4/github/sql/mysqld.cc:5891
      #7  0x000055612bef4095 in main (argc=23, argv=0x7ffe548a0e98) at /home/stepan/mariadb/10.4/github/sql/main.cc:25
       
      Thread 33 (Thread 0x7fd2dbfff700 (LWP 31820)):
      #0  0x00007fd30ed51d1d in fsync () from /lib64/libpthread.so.0
      #1  0x000055612c6de5a7 in os_file_fsync_posix (file=19) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0file.cc:2366
      #2  0x000055612c6de8e2 in os_file_flush_func (file=19) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0file.cc:2459
      #3  0x000055612c9887f5 in pfs_os_file_flush_func (file=..., src_file=0x55612d054420 "/home/stepan/mariadb/10.4/github/storage/innobase/fil/fil0fil.cc", src_line=751) at /home/stepan/mariadb/10.4/github/storage/innobase/include/os0file.ic:406
      #4  0x000055612c98b186 in fil_flush_low (space=0x55612f746cc0, metadata=false) at /home/stepan/mariadb/10.4/github/storage/innobase/fil/fil0fil.cc:751
      #5  0x000055612c998b0f in fil_flush (space_id=4294967280) at /home/stepan/mariadb/10.4/github/storage/innobase/fil/fil0fil.cc:4462
      #6  0x000055612c6b13f8 in log_write_flush_to_disk_low () at /home/stepan/mariadb/10.4/github/storage/innobase/log/log0log.cc:809
      #7  0x000055612c6b1ede in log_write_up_to (lsn=97593560, flush_to_disk=true, rotate_key=false) at /home/stepan/mariadb/10.4/github/storage/innobase/log/log0log.cc:1046
      #8  0x000055612c6b2046 in log_buffer_sync_in_background (flush=true) at /home/stepan/mariadb/10.4/github/storage/innobase/log/log0log.cc:1091
      #9  0x000055612c805c7f in srv_sync_log_buffer_in_background () at /home/stepan/mariadb/10.4/github/storage/innobase/srv/srv0srv.cc:1997
      #10 0x000055612c80628c in srv_master_do_active_tasks () at /home/stepan/mariadb/10.4/github/storage/innobase/srv/srv0srv.cc:2156
      #11 0x000055612c806f93 in srv_master_thread (arg=0x0) at /home/stepan/mariadb/10.4/github/storage/innobase/srv/srv0srv.cc:2381
      #12 0x00007fd30ed4add5 in start_thread () from /lib64/libpthread.so.0
      #13 0x00007fd30d107ead in clone () from /lib64/libc.so.6
       
      Thread 32 (Thread 0x7fd3077e3700 (LWP 31800)):
      #0  0x00007fd30ed4ed12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x000055612cc35a6b in safe_cond_timedwait (cond=0x55612e266ce0 <COND_timer>, mp=0x55612e266c20 <LOCK_timer>, abstime=0x7fd3077e2d70, file=0x55612d0e8450 "/home/stepan/mariadb/10.4/github/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/mariadb/10.4/github/mysys/thr_mutex.c:546
      #2  0x000055612cc36aae in inline_mysql_cond_timedwait (that=0x55612e266ce0 <COND_timer>, mutex=0x55612e266c20 <LOCK_timer>, abstime=0x7fd3077e2d70, src_file=0x55612d0e8498 "/home/stepan/mariadb/10.4/github/mysys/thr_timer.c", src_line=292) at /home/stepan/mariadb/10.4/github/include/mysql/psi/mysql_thread.h:1204
      #3  0x000055612cc37671 in timer_handler (arg=0x0) at /home/stepan/mariadb/10.4/github/mysys/thr_timer.c:292
      #4  0x000055612cbbe4a5 in pfs_spawn_thread (arg=0x55612f3661c0) at /home/stepan/mariadb/10.4/github/storage/perfschema/pfs.cc:1862
      #5  0x00007fd30ed4add5 in start_thread () from /lib64/libpthread.so.0
      #6  0x00007fd30d107ead in clone () from /lib64/libc.so.6
       
      Thread 31 (Thread 0x7fd2d97fa700 (LWP 31819)):
      #0  0x00007fd30ed4ed12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x000055612c6ead00 in os_event::timed_wait (this=0x55612f676660, abstime=0x7fd2d97f9b70) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:275
      #2  0x000055612c6eafa9 in os_event::wait_time_low (this=0x55612f676660, time_in_usec=5000000, reset_sig_count=7) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:385
      #3  0x000055612c6eb149 in os_event_wait_time_low (event=0x55612f676660, time_in_usec=5000000, reset_sig_count=7) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:485
      #4  0x000055612c873563 in ib_wqueue_timedwait (wq=0x55612f66fa70, wait_in_usecs=5000000) at /home/stepan/mariadb/10.4/github/storage/innobase/ut/ut0wqueue.cc:163
      #5  0x000055612c9e623d in fts_optimize_thread (arg=0x55612f66fa70) at /home/stepan/mariadb/10.4/github/storage/innobase/fts/fts0opt.cc:2865
      #6  0x00007fd30ed4add5 in start_thread () from /lib64/libpthread.so.0
      #7  0x00007fd30d107ead in clone () from /lib64/libc.so.6
       
      Thread 30 (Thread 0x7fd2e77fe700 (LWP 31808)):
      #0  0x00007fd30e91b644 in __io_getevents_0_4 () from /lib64/libaio.so.1
      #1  0x000055612c6dcdad in LinuxAIOHandler::collect (this=0x7fd2e77fdb00) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0file.cc:1809
      #2  0x000055612c6dd3f6 in LinuxAIOHandler::poll (this=0x7fd2e77fdb00, m1=0x7fd2e77fdba8, m2=0x7fd2e77fdbb0, request=0x7fd2e77fdbe0) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0file.cc:1968
      #3  0x000055612c6dd559 in os_aio_linux_handler (global_segment=1, m1=0x7fd2e77fdba8, m2=0x7fd2e77fdbb0, request=0x7fd2e77fdbe0) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0file.cc:2022
      #4  0x000055612c6e22c0 in os_aio_handler (segment=1, m1=0x7fd2e77fdba8, m2=0x7fd2e77fdbb0, request=0x7fd2e77fdbe0) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0file.cc:5561
      #5  0x000055612c9984f9 in fil_aio_wait (segment=1) at /home/stepan/mariadb/10.4/github/storage/innobase/fil/fil0fil.cc:4357
      #6  0x000055612c809e42 in io_handler_thread (arg=0x55612da43988 <n+8>) at /home/stepan/mariadb/10.4/github/storage/innobase/srv/srv0start.cc:324
      #7  0x00007fd30ed4add5 in start_thread () from /lib64/libpthread.so.0
      #8  0x00007fd30d107ead in clone () from /lib64/libc.so.6
       
      Thread 29 (Thread 0x7fd2e6ffd700 (LWP 31809)):
      #0  0x00007fd30e91b644 in __io_getevents_0_4 () from /lib64/libaio.so.1
      #1  0x000055612c6dcdad in LinuxAIOHandler::collect (this=0x7fd2e6ffcb00) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0file.cc:1809
      #2  0x000055612c6dd3f6 in LinuxAIOHandler::poll (this=0x7fd2e6ffcb00, m1=0x7fd2e6ffcba8, m2=0x7fd2e6ffcbb0, request=0x7fd2e6ffcbe0) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0file.cc:1968
      #3  0x000055612c6dd559 in os_aio_linux_handler (global_segment=2, m1=0x7fd2e6ffcba8, m2=0x7fd2e6ffcbb0, request=0x7fd2e6ffcbe0) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0file.cc:2022
      #4  0x000055612c6e22c0 in os_aio_handler (segment=2, m1=0x7fd2e6ffcba8, m2=0x7fd2e6ffcbb0, request=0x7fd2e6ffcbe0) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0file.cc:5561
      #5  0x000055612c9984f9 in fil_aio_wait (segment=2) at /home/stepan/mariadb/10.4/github/storage/innobase/fil/fil0fil.cc:4357
      #6  0x000055612c809e42 in io_handler_thread (arg=0x55612da43990 <n+16>) at /home/stepan/mariadb/10.4/github/storage/innobase/srv/srv0start.cc:324
      #7  0x00007fd30ed4add5 in start_thread () from /lib64/libpthread.so.0
      #8  0x00007fd30d107ead in clone () from /lib64/libc.so.6
       
      Thread 28 (Thread 0x7fd2caffd700 (LWP 31825)):
      #0  0x00007fd30ed4e965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x000055612c6eb377 in os_event::wait (this=0x55612f4da4c0) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:158
      #2  0x000055612c6eae6b in os_event::wait_low (this=0x55612f4da4c0, reset_sig_count=1) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:325
      #3  0x000055612c6eb16e in os_event_wait_low (event=0x55612f4da4c0, reset_sig_count=0) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:502
      #4  0x000055612c908e86 in buf_dump_thread () at /home/stepan/mariadb/10.4/github/storage/innobase/buf/buf0dump.cc:833
      #5  0x00007fd30ed4add5 in start_thread () from /lib64/libpthread.so.0
      #6  0x00007fd30d107ead in clone () from /lib64/libc.so.6
       
      Thread 27 (Thread 0x7fd3040bb700 (LWP 31829)):
      #0  0x00007fd30d04140a in sigwaitinfo () from /lib64/libc.so.6
      #1  0x000055612bef4114 in my_sigwait (set=0x7fd3040bacf0, sig=0x7fd3040bacb4, code=0x7fd3040bacb8) at /home/stepan/mariadb/10.4/github/include/my_pthread.h:198
      #2  0x000055612befaa8d in signal_hand (arg=0x0) at /home/stepan/mariadb/10.4/github/sql/mysqld.cc:3243
      #3  0x000055612cbbe4a5 in pfs_spawn_thread (arg=0x55612f7b8770) at /home/stepan/mariadb/10.4/github/storage/perfschema/pfs.cc:1862
      #4  0x00007fd30ed4add5 in start_thread () from /lib64/libpthread.so.0
      #5  0x00007fd30d107ead in clone () from /lib64/libc.so.6
       
      Thread 26 (Thread 0x7fd2da7fc700 (LWP 31817)):
      #0  0x00007fd30ed4ed12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x000055612c6ead00 in os_event::timed_wait (this=0x55612f4da440, abstime=0x7fd2da7fbd30) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:275
      #2  0x000055612c6eafa9 in os_event::wait_time_low (this=0x55612f4da440, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:385
      #3  0x000055612c6eb149 in os_event_wait_time_low (event=0x55612f4da440, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:485
      #4  0x000055612c80520f in srv_monitor_thread () at /home/stepan/mariadb/10.4/github/storage/innobase/srv/srv0srv.cc:1687
      #5  0x00007fd30ed4add5 in start_thread () from /lib64/libpthread.so.0
      #6  0x00007fd30d107ead in clone () from /lib64/libc.so.6
       
      Thread 25 (Thread 0x7fd2e5ffb700 (LWP 31811)):
      #0  0x00007fd30e91b644 in __io_getevents_0_4 () from /lib64/libaio.so.1
      #1  0x000055612c6dcdad in LinuxAIOHandler::collect (this=0x7fd2e5ffab00) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0file.cc:1809
      #2  0x000055612c6dd3f6 in LinuxAIOHandler::poll (this=0x7fd2e5ffab00, m1=0x7fd2e5ffaba8, m2=0x7fd2e5ffabb0, request=0x7fd2e5ffabe0) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0file.cc:1968
      #3  0x000055612c6dd559 in os_aio_linux_handler (global_segment=4, m1=0x7fd2e5ffaba8, m2=0x7fd2e5ffabb0, request=0x7fd2e5ffabe0) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0file.cc:2022
      #4  0x000055612c6e22c0 in os_aio_handler (segment=4, m1=0x7fd2e5ffaba8, m2=0x7fd2e5ffabb0, request=0x7fd2e5ffabe0) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0file.cc:5561
      #5  0x000055612c9984f9 in fil_aio_wait (segment=4) at /home/stepan/mariadb/10.4/github/storage/innobase/fil/fil0fil.cc:4357
      #6  0x000055612c809e42 in io_handler_thread (arg=0x55612da439a0 <n+32>) at /home/stepan/mariadb/10.4/github/storage/innobase/srv/srv0start.cc:324
      #7  0x00007fd30ed4add5 in start_thread () from /lib64/libpthread.so.0
      #8  0x00007fd30d107ead in clone () from /lib64/libc.so.6
       
      Thread 24 (Thread 0x7fd2fc6a3700 (LWP 32066)):
      #0  0x00007fd30ed4ed12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x000055612cc35a6b in safe_cond_timedwait (cond=0x55612d9c5060 <COND_thread_cache>, mp=0x55612d9c7a20 <LOCK_thread_cache>, abstime=0x7fd2fc6a2c60, file=0x55612ccf3d68 "/home/stepan/mariadb/10.4/github/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/mariadb/10.4/github/mysys/thr_mutex.c:546
      #2  0x000055612bef4a15 in inline_mysql_cond_timedwait (that=0x55612d9c5060 <COND_thread_cache>, mutex=0x55612d9c7a20 <LOCK_thread_cache>, abstime=0x7fd2fc6a2c60, src_file=0x55612ccf5db8 "/home/stepan/mariadb/10.4/github/sql/mysqld.cc", src_line=2686) at /home/stepan/mariadb/10.4/github/include/mysql/psi/mysql_thread.h:1204
      #3  0x000055612bef9b2c in cache_thread (thd=0x7fd298000b00) at /home/stepan/mariadb/10.4/github/sql/mysqld.cc:2685
      #4  0x000055612bef9dd2 in one_thread_per_connection_end (thd=0x7fd298000b00, put_in_cache=true) at /home/stepan/mariadb/10.4/github/sql/mysqld.cc:2771
      #5  0x000055612c1bedcf in do_handle_one_connection (connect=0x55612f7d4510) at /home/stepan/mariadb/10.4/github/sql/sql_connect.cc:1423
      #6  0x000055612c1bea80 in handle_one_connection (arg=0x55612f7d4510) at /home/stepan/mariadb/10.4/github/sql/sql_connect.cc:1316
      #7  0x000055612cbbe4a5 in pfs_spawn_thread (arg=0x55612f323ee0) at /home/stepan/mariadb/10.4/github/storage/perfschema/pfs.cc:1862
      #8  0x00007fd30ed4add5 in start_thread () from /lib64/libpthread.so.0
      #9  0x00007fd30d107ead in clone () from /lib64/libc.so.6
       
      Thread 23 (Thread 0x7fd2ca7fc700 (LWP 31826)):
      #0  0x00007fd30ed51e3d in nanosleep () from /lib64/libpthread.so.0
      #1  0x000055612c6eb6b4 in os_thread_sleep (tm=1000000) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0thread.cc:225
      #2  0x000055612c8dddd3 in btr_defragment_thread () at /home/stepan/mariadb/10.4/github/storage/innobase/btr/btr0defragment.cc:705
      #3  0x00007fd30ed4add5 in start_thread () from /lib64/libpthread.so.0
      #4  0x00007fd30d107ead in clone () from /lib64/libc.so.6
       
      Thread 22 (Thread 0x7fd2c9ffb700 (LWP 31827)):
      #0  0x00007fd30ed4e965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x000055612c6eb377 in os_event::wait (this=0x55612f4da5c0) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:158
      #2  0x000055612c6eae6b in os_event::wait_low (this=0x55612f4da5c0, reset_sig_count=1) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:325
      #3  0x000055612c6eb16e in os_event_wait_low (event=0x55612f4da5c0, reset_sig_count=0) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:502
      #4  0x000055612c8ee7d9 in buf_resize_thread () at /home/stepan/mariadb/10.4/github/storage/innobase/buf/buf0buf.cc:3199
      #5  0x00007fd30ed4add5 in start_thread () from /lib64/libpthread.so.0
      #6  0x00007fd30d107ead in clone () from /lib64/libc.so.6
       
      Thread 21 (Thread 0x7fd2fc6ee700 (LWP 31833)):
      #0  0x00007fd30ed4ed12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x000055612cc35a6b in safe_cond_timedwait (cond=0x55612d9c5060 <COND_thread_cache>, mp=0x55612d9c7a20 <LOCK_thread_cache>, abstime=0x7fd2fc6edc60, file=0x55612ccf3d68 "/home/stepan/mariadb/10.4/github/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/mariadb/10.4/github/mysys/thr_mutex.c:546
      #2  0x000055612bef4a15 in inline_mysql_cond_timedwait (that=0x55612d9c5060 <COND_thread_cache>, mutex=0x55612d9c7a20 <LOCK_thread_cache>, abstime=0x7fd2fc6edc60, src_file=0x55612ccf5db8 "/home/stepan/mariadb/10.4/github/sql/mysqld.cc", src_line=2686) at /home/stepan/mariadb/10.4/github/include/mysql/psi/mysql_thread.h:1204
      #3  0x000055612bef9b2c in cache_thread (thd=0x7fd2a0000b00) at /home/stepan/mariadb/10.4/github/sql/mysqld.cc:2685
      #4  0x000055612bef9dd2 in one_thread_per_connection_end (thd=0x7fd2a0000b00, put_in_cache=true) at /home/stepan/mariadb/10.4/github/sql/mysqld.cc:2771
      #5  0x000055612c1bedcf in do_handle_one_connection (connect=0x55612f8b7a00) at /home/stepan/mariadb/10.4/github/sql/sql_connect.cc:1423
      #6  0x000055612c1bea80 in handle_one_connection (arg=0x55612f8b7a00) at /home/stepan/mariadb/10.4/github/sql/sql_connect.cc:1316
      #7  0x000055612cbbe4a5 in pfs_spawn_thread (arg=0x55612f8d8f40) at /home/stepan/mariadb/10.4/github/storage/perfschema/pfs.cc:1862
      #8  0x00007fd30ed4add5 in start_thread () from /lib64/libpthread.so.0
      #9  0x00007fd30d107ead in clone () from /lib64/libc.so.6
       
      Thread 20 (Thread 0x7fd304070700 (LWP 31832)):
      #0  0x00007fd30ed4e965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x000055612cc35769 in safe_cond_wait (cond=0x55612d9c50a0 <COND_slave_background>, mp=0x55612d9c8320 <LOCK_slave_background>, file=0x55612cd040c0 "/home/stepan/mariadb/10.4/github/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/mariadb/10.4/github/mysys/thr_mutex.c:492
      #2  0x000055612bf2708e in inline_mysql_cond_wait (that=0x55612d9c50a0 <COND_slave_background>, mutex=0x55612d9c8320 <LOCK_slave_background>, src_file=0x55612cd04c20 "/home/stepan/mariadb/10.4/github/sql/slave.cc", src_line=519) at /home/stepan/mariadb/10.4/github/include/mysql/psi/mysql_thread.h:1167
      #3  0x000055612bf289cb in handle_slave_background (arg=0x0) at /home/stepan/mariadb/10.4/github/sql/slave.cc:519
      #4  0x000055612cbbe4a5 in pfs_spawn_thread (arg=0x55612f8dde50) at /home/stepan/mariadb/10.4/github/storage/perfschema/pfs.cc:1862
      #5  0x00007fd30ed4add5 in start_thread () from /lib64/libpthread.so.0
      #6  0x00007fd30d107ead in clone () from /lib64/libc.so.6
       
      Thread 19 (Thread 0x7fd2cb7fe700 (LWP 31824)):
      #0  0x00007fd30ed4e965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x000055612c6eb377 in os_event::wait (this=0x55612f4da2c0) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:158
      #2  0x000055612c6eae6b in os_event::wait_low (this=0x55612f4da2c0, reset_sig_count=3439) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:325
      #3  0x000055612c6eb16e in os_event_wait_low (event=0x55612f4da2c0, reset_sig_count=0) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:502
      #4  0x000055612c803328 in srv_resume_thread (slot=0x55612d7ffe10 <srv_sys+1104>, sig_count=0, wait=true, timeout_usec=0) at /home/stepan/mariadb/10.4/github/storage/innobase/srv/srv0srv.cc:909
      #5  0x000055612c8075e9 in srv_worker_thread (arg=0x0) at /home/stepan/mariadb/10.4/github/storage/innobase/srv/srv0srv.cc:2517
      #6  0x00007fd30ed4add5 in start_thread () from /lib64/libpthread.so.0
      #7  0x00007fd30d107ead in clone () from /lib64/libc.so.6
       
      Thread 18 (Thread 0x7fd2c3fff700 (LWP 31823)):
      #0  0x00007fd30ed4e965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x000055612c6eb377 in os_event::wait (this=0x55612f4da240) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:158
      #2  0x000055612c6eae6b in os_event::wait_low (this=0x55612f4da240, reset_sig_count=4258) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:325
      #3  0x000055612c6eb16e in os_event_wait_low (event=0x55612f4da240, reset_sig_count=0) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:502
      #4  0x000055612c803328 in srv_resume_thread (slot=0x55612d7ffd10 <srv_sys+848>, sig_count=0, wait=true, timeout_usec=0) at /home/stepan/mariadb/10.4/github/storage/innobase/srv/srv0srv.cc:909
      #5  0x000055612c8075e9 in srv_worker_thread (arg=0x0) at /home/stepan/mariadb/10.4/github/storage/innobase/srv/srv0srv.cc:2517
      #6  0x00007fd30ed4add5 in start_thread () from /lib64/libpthread.so.0
      #7  0x00007fd30d107ead in clone () from /lib64/libc.so.6
       
      Thread 17 (Thread 0x7fd2d9ffb700 (LWP 31818)):
      #0  0x00007fd30ed4ed12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x000055612c6ead00 in os_event::timed_wait (this=0x55612f7473a0, abstime=0x7fd2d9ffad70) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:275
      #2  0x000055612c6eafa9 in os_event::wait_time_low (this=0x55612f7473a0, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:385
      #3  0x000055612c6eb149 in os_event_wait_time_low (event=0x55612f7473a0, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:485
      #4  0x000055612c97f6ca in dict_stats_thread () at /home/stepan/mariadb/10.4/github/storage/innobase/dict/dict0stats_bg.cc:476
      #5  0x00007fd30ed4add5 in start_thread () from /lib64/libpthread.so.0
      #6  0x00007fd30d107ead in clone () from /lib64/libc.so.6
       
      Thread 16 (Thread 0x7fd2ff67e700 (LWP 31802)):
      #0  0x00007fd30d108483 in epoll_wait () from /lib64/libc.so.6
      #1  0x00007fd306c4a548 in asio::detail::epoll_reactor::run (this=0x55612f3d2c30, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391
      #2  0x00007fd306c4ec40 in asio::detail::task_io_service::do_run_one (this=this@entry=0x55612f3d2ad0, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355
      #3  0x00007fd306c4eef2 in asio::detail::task_io_service::run (this=0x55612f3d2ad0, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148
      #4  0x00007fd306c6697d in asio::io_service::run (this=0x55612f3d27e0) at asio/asio/impl/io_service.ipp:58
      #5  0x00007fd306c670d1 in gcomm::AsioProtonet::event_loop (this=0x55612f3d2740, period=...) at gcomm/src/asio_protonet.cpp:116
      #6  0x00007fd306c85f3d in GCommConn::run (this=0x55612f3d2130) at gcs/src/gcs_gcomm.cpp:498
      #7  0x00007fd306c88fe9 in GCommConn::run_fn (arg=<optimized out>) at gcs/src/gcs_gcomm.cpp:209
      #8  0x00007fd30ed4add5 in start_thread () from /lib64/libpthread.so.0
      #9  0x00007fd30d107ead in clone () from /lib64/libc.so.6
       
      Thread 15 (Thread 0x7fd2daffd700 (LWP 31816)):
      #0  0x00007fd30ed4ed12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x000055612c6ead00 in os_event::timed_wait (this=0x55612f4da3c0, abstime=0x7fd2daffcbb0) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:275
      #2  0x000055612c6eafa9 in os_event::wait_time_low (this=0x55612f4da3c0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:385
      #3  0x000055612c6eb149 in os_event_wait_time_low (event=0x55612f4da3c0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:485
      #4  0x000055612c805789 in srv_error_monitor_thread () at /home/stepan/mariadb/10.4/github/storage/innobase/srv/srv0srv.cc:1851
      #5  0x00007fd30ed4add5 in start_thread () from /lib64/libpthread.so.0
      #6  0x00007fd30d107ead in clone () from /lib64/libc.so.6
       
      Thread 14 (Thread 0x7fd2c97fa700 (LWP 31828)):
      #0  0x00007fd30ed4e965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x000055612cc35769 in safe_cond_wait (cond=0x7fd2c97f9c70, mp=0x7fd2c97f9cb0, file=0x55612cf22ce0 "/home/stepan/mariadb/10.4/github/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/mariadb/10.4/github/mysys/thr_mutex.c:492
      #2  0x000055612c5de15d in inline_mysql_cond_wait (that=0x7fd2c97f9c70, mutex=0x7fd2c97f9cb0, src_file=0x55612cf247f8 "/home/stepan/mariadb/10.4/github/storage/innobase/handler/ha_innodb.cc", src_line=288) at /home/stepan/mariadb/10.4/github/include/mysql/psi/mysql_thread.h:1178
      #3  0x000055612c5e3a55 in thd_destructor_proxy () at /home/stepan/mariadb/10.4/github/storage/innobase/handler/ha_innodb.cc:288
      #4  0x000055612cbbe4a5 in pfs_spawn_thread (arg=0x55612f782c90) at /home/stepan/mariadb/10.4/github/storage/perfschema/pfs.cc:1862
      #5  0x00007fd30ed4add5 in start_thread () from /lib64/libpthread.so.0
      #6  0x00007fd30d107ead in clone () from /lib64/libc.so.6
       
      Thread 13 (Thread 0x7fd2db7fe700 (LWP 31815)):
      #0  0x00007fd30ed4ed12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x000055612c6ead00 in os_event::timed_wait (this=0x55612f715380, abstime=0x7fd2db7fdd50) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:275
      #2  0x000055612c6eafa9 in os_event::wait_time_low (this=0x55612f715380, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:385
      #3  0x000055612c6eb149 in os_event_wait_time_low (event=0x55612f715380, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:485
      #4  0x000055612c6ad527 in lock_wait_timeout_thread () at /home/stepan/mariadb/10.4/github/storage/innobase/lock/lock0wait.cc:522
      #5  0x00007fd30ed4add5 in start_thread () from /lib64/libpthread.so.0
      #6  0x00007fd30d107ead in clone () from /lib64/libc.so.6
       
      Thread 12 (Thread 0x7fd2cbfff700 (LWP 31822)):
      #0  0x00007fd30ed4e965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x000055612c6eb377 in os_event::wait (this=0x55612f4da340) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:158
      #2  0x000055612c6eae6b in os_event::wait_low (this=0x55612f4da340, reset_sig_count=3233) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:325
      #3  0x000055612c6eb16e in os_event_wait_low (event=0x55612f4da340, reset_sig_count=0) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:502
      #4  0x000055612c803328 in srv_resume_thread (slot=0x55612d7fff10 <srv_sys+1360>, sig_count=0, wait=true, timeout_usec=0) at /home/stepan/mariadb/10.4/github/storage/innobase/srv/srv0srv.cc:909
      #5  0x000055612c8075e9 in srv_worker_thread (arg=0x0) at /home/stepan/mariadb/10.4/github/storage/innobase/srv/srv0srv.cc:2517
      #6  0x00007fd30ed4add5 in start_thread () from /lib64/libpthread.so.0
      #7  0x00007fd30d107ead in clone () from /lib64/libc.so.6
       
      Thread 11 (Thread 0x7fd2e4ff9700 (LWP 31813)):
      #0  0x00007fd30ed4e965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x000055612c6eb377 in os_event::wait (this=0x55612f66eb90) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:158
      #2  0x000055612c6eae6b in os_event::wait_low (this=0x55612f66eb90, reset_sig_count=3795) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:325
      #3  0x000055612c6eb16e in os_event_wait_low (event=0x55612f66eb90, reset_sig_count=0) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:502
      #4  0x000055612c6b1a72 in log_write_up_to (lsn=97593560, flush_to_disk=true, rotate_key=false) at /home/stepan/mariadb/10.4/github/storage/innobase/log/log0log.cc:924
      #5  0x000055612c90f048 in buf_flush_write_block_low (bpage=0x7fd2fc90de00, flush_type=BUF_FLUSH_LIST, sync=false) at /home/stepan/mariadb/10.4/github/storage/innobase/buf/buf0flu.cc:999
      #6  0x000055612c90fd4b in buf_flush_page (buf_pool=0x55612f5c02b0, bpage=0x7fd2fc90de00, flush_type=BUF_FLUSH_LIST, sync=false) at /home/stepan/mariadb/10.4/github/storage/innobase/buf/buf0flu.cc:1210
      #7  0x000055612c910824 in buf_flush_try_neighbors (page_id=..., flush_type=BUF_FLUSH_LIST, n_flushed=2, n_to_flush=200) at /home/stepan/mariadb/10.4/github/storage/innobase/buf/buf0flu.cc:1437
      #8  0x000055612c910bde in buf_flush_page_and_try_neighbors (bpage=0x7fd2fc90de00, flush_type=BUF_FLUSH_LIST, n_to_flush=200, count=0x7fd2e4ff88b8) at /home/stepan/mariadb/10.4/github/storage/innobase/buf/buf0flu.cc:1511
      #9  0x000055612c9119e0 in buf_do_flush_list_batch (buf_pool=0x55612f5c02b0, min_n=200, lsn_limit=18446744073709551615) at /home/stepan/mariadb/10.4/github/storage/innobase/buf/buf0flu.cc:1769
      #10 0x000055612c9120bf in buf_flush_batch (buf_pool=0x55612f5c02b0, flush_type=BUF_FLUSH_LIST, min_n=200, lsn_limit=18446744073709551615, n=0x7fd2e4ff8b60) at /home/stepan/mariadb/10.4/github/storage/innobase/buf/buf0flu.cc:1839
      #11 0x000055612c912886 in buf_flush_do_batch (buf_pool=0x55612f5c02b0, type=BUF_FLUSH_LIST, min_n=200, lsn_limit=18446744073709551615, n=0x7fd2e4ff8b60) at /home/stepan/mariadb/10.4/github/storage/innobase/buf/buf0flu.cc:1997
      #12 0x000055612c912b96 in buf_flush_lists (min_n=200, lsn_limit=18446744073709551615, n_processed=0x7fd2e4ff8bd0) at /home/stepan/mariadb/10.4/github/storage/innobase/buf/buf0flu.cc:2103
      #13 0x000055612c916e17 in buf_flush_page_cleaner_coordinator () at /home/stepan/mariadb/10.4/github/storage/innobase/buf/buf0flu.cc:3255
      #14 0x00007fd30ed4add5 in start_thread () from /lib64/libpthread.so.0
      #15 0x00007fd30d107ead in clone () from /lib64/libc.so.6
       
      Thread 10 (Thread 0x7fd2d8ff9700 (LWP 31821)):
      #0  0x00007fd30ed4e965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x000055612c6eb377 in os_event::wait (this=0x55612f4da1c0) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:158
      #2  0x000055612c6eae6b in os_event::wait_low (this=0x55612f4da1c0, reset_sig_count=1154) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:325
      #3  0x000055612c6eb16e in os_event_wait_low (event=0x55612f4da1c0, reset_sig_count=1154) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:502
      #4  0x000055612c803328 in srv_resume_thread (slot=0x55612d7ffc10 <srv_sys+592>, sig_count=1154, wait=true, timeout_usec=0) at /home/stepan/mariadb/10.4/github/storage/innobase/srv/srv0srv.cc:909
      #5  0x000055612c807aa0 in srv_purge_coordinator_suspend (slot=0x55612d7ffc10 <srv_sys+592>, rseg_history_len=33) at /home/stepan/mariadb/10.4/github/storage/innobase/srv/srv0srv.cc:2657
      #6  0x000055612c807dd1 in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/mariadb/10.4/github/storage/innobase/srv/srv0srv.cc:2738
      #7  0x00007fd30ed4add5 in start_thread () from /lib64/libpthread.so.0
      #8  0x00007fd30d107ead in clone () from /lib64/libc.so.6
       
      Thread 9 (Thread 0x7fd2e7fff700 (LWP 31807)):
      #0  0x00007fd30e91b644 in __io_getevents_0_4 () from /lib64/libaio.so.1
      #1  0x000055612c6dcdad in LinuxAIOHandler::collect (this=0x7fd2e7ffeb00) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0file.cc:1809
      #2  0x000055612c6dd3f6 in LinuxAIOHandler::poll (this=0x7fd2e7ffeb00, m1=0x7fd2e7ffeba8, m2=0x7fd2e7ffebb0, request=0x7fd2e7ffebe0) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0file.cc:1968
      #3  0x000055612c6dd559 in os_aio_linux_handler (global_segment=0, m1=0x7fd2e7ffeba8, m2=0x7fd2e7ffebb0, request=0x7fd2e7ffebe0) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0file.cc:2022
      #4  0x000055612c6e22c0 in os_aio_handler (segment=0, m1=0x7fd2e7ffeba8, m2=0x7fd2e7ffebb0, request=0x7fd2e7ffebe0) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0file.cc:5561
      #5  0x000055612c9984f9 in fil_aio_wait (segment=0) at /home/stepan/mariadb/10.4/github/storage/innobase/fil/fil0fil.cc:4357
      #6  0x000055612c809e42 in io_handler_thread (arg=0x55612da43980 <n>) at /home/stepan/mariadb/10.4/github/storage/innobase/srv/srv0start.cc:324
      #7  0x00007fd30ed4add5 in start_thread () from /lib64/libpthread.so.0
      #8  0x00007fd30d107ead in clone () from /lib64/libc.so.6
       
      Thread 8 (Thread 0x7fd2e57fa700 (LWP 31812)):
      #0  0x00007fd30e91b644 in __io_getevents_0_4 () from /lib64/libaio.so.1
      #1  0x000055612c6dcdad in LinuxAIOHandler::collect (this=0x7fd2e57f9b00) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0file.cc:1809
      #2  0x000055612c6dd3f6 in LinuxAIOHandler::poll (this=0x7fd2e57f9b00, m1=0x7fd2e57f9ba8, m2=0x7fd2e57f9bb0, request=0x7fd2e57f9be0) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0file.cc:1968
      #3  0x000055612c6dd559 in os_aio_linux_handler (global_segment=5, m1=0x7fd2e57f9ba8, m2=0x7fd2e57f9bb0, request=0x7fd2e57f9be0) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0file.cc:2022
      #4  0x000055612c6e22c0 in os_aio_handler (segment=5, m1=0x7fd2e57f9ba8, m2=0x7fd2e57f9bb0, request=0x7fd2e57f9be0) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0file.cc:5561
      #5  0x000055612c9984f9 in fil_aio_wait (segment=5) at /home/stepan/mariadb/10.4/github/storage/innobase/fil/fil0fil.cc:4357
      #6  0x000055612c809e42 in io_handler_thread (arg=0x55612da439a8 <n+40>) at /home/stepan/mariadb/10.4/github/storage/innobase/srv/srv0start.cc:324
      #7  0x00007fd30ed4add5 in start_thread () from /lib64/libpthread.so.0
      #8  0x00007fd30d107ead in clone () from /lib64/libc.so.6
       
      Thread 7 (Thread 0x7fd2e67fc700 (LWP 31810)):
      #0  0x00007fd30e91b644 in __io_getevents_0_4 () from /lib64/libaio.so.1
      #1  0x000055612c6dcdad in LinuxAIOHandler::collect (this=0x7fd2e67fbb00) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0file.cc:1809
      #2  0x000055612c6dd3f6 in LinuxAIOHandler::poll (this=0x7fd2e67fbb00, m1=0x7fd2e67fbba8, m2=0x7fd2e67fbbb0, request=0x7fd2e67fbbe0) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0file.cc:1968
      #3  0x000055612c6dd559 in os_aio_linux_handler (global_segment=3, m1=0x7fd2e67fbba8, m2=0x7fd2e67fbbb0, request=0x7fd2e67fbbe0) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0file.cc:2022
      #4  0x000055612c6e22c0 in os_aio_handler (segment=3, m1=0x7fd2e67fbba8, m2=0x7fd2e67fbbb0, request=0x7fd2e67fbbe0) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0file.cc:5561
      #5  0x000055612c9984f9 in fil_aio_wait (segment=3) at /home/stepan/mariadb/10.4/github/storage/innobase/fil/fil0fil.cc:4357
      #6  0x000055612c809e42 in io_handler_thread (arg=0x55612da43998 <n+24>) at /home/stepan/mariadb/10.4/github/storage/innobase/srv/srv0start.cc:324
      #7  0x00007fd30ed4add5 in start_thread () from /lib64/libpthread.so.0
      #8  0x00007fd30d107ead in clone () from /lib64/libc.so.6
       
      Thread 6 (Thread 0x7fd3042e3700 (LWP 31805)):
      #0  0x00007fd30ed4e965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00007fd306b9be42 in fifo_lock_get (q=q@entry=0x7fd305f33010) at galerautils/src/gu_fifo.c:262
      #2  0x00007fd306b9c4b1 in gu_fifo_get_head (q=0x7fd305f33010, err=0x7fd3042e201c) at galerautils/src/gu_fifo.c:330
      #3  0x00007fd306c7fd1c in gcs_recv (conn=0x55612f3a2280, action=0x7fd3042e2060) at gcs/src/gcs.cpp:2024
      #4  0x00007fd306cacf6c in galera::GcsActionSource::process (this=0x55612f3a1c60, recv_ctx=0x7fd3042e2480, exit_loop=@0x7fd3042e224e: false) at galera/src/gcs_action_source.cpp:161
      #5  0x00007fd306ccbf24 in galera::ReplicatorSMM::async_recv (this=0x55612f399fa0, recv_ctx=0x7fd3042e2480) at galera/src/replicator_smm.cpp:391
      #6  0x000055612ccebe7a in wsrep::wsrep_provider_v26::run_applier (this=0x55612f3241f0, applier_ctx=0x7fd3042e2480) at /home/stepan/mariadb/10.4/github/wsrep-lib/src/wsrep_provider_v26.cpp:727
      #7  0x000055612c2bd787 in wsrep_replication_process (thd=0x7fd2e8000b00, arg=0x55612f3ebb60) at /home/stepan/mariadb/10.4/github/sql/wsrep_thd.cc:62
      #8  0x000055612c2ade70 in start_wsrep_THD (arg=0x55612f3ebb60) at /home/stepan/mariadb/10.4/github/sql/wsrep_mysqld.cc:2708
      #9  0x000055612cbbe4a5 in pfs_spawn_thread (arg=0x55612f3eabf0) at /home/stepan/mariadb/10.4/github/storage/perfschema/pfs.cc:1862
      #10 0x00007fd30ed4add5 in start_thread () from /lib64/libpthread.so.0
      #11 0x00007fd30d107ead in clone () from /lib64/libc.so.6
       
      Thread 5 (Thread 0x7fd2fee7d700 (LWP 31803)):
      #0  0x00007fd30ed4e965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00007fd306c844b7 in gu::Lock::wait (cond=..., this=0x7fd2fee7c850) at galerautils/src/gu_lock.hpp:45
      #2  0x00007fd306c86ffb in RecvBuf::front (this=this@entry=0x55612f3d2318, timeout=...) at gcs/src/gcs_gcomm.cpp:124
      #3  0x00007fd306c84b96 in gcomm_recv (backend=<optimized out>, msg=0x55612f3a2560, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673
      #4  0x00007fd306c78b2c in core_msg_recv (backend=backend@entry=0x55612f3a26c0, recv_msg=recv_msg@entry=0x55612f3a2560, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480
      #5  0x00007fd306c7bda4 in gcs_core_recv (conn=0x55612f3a2500, recv_act=recv_act@entry=0x7fd2fee7cd70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184
      #6  0x00007fd306c83028 in gcs_recv_thread (arg=0x55612f3a2280) at gcs/src/gcs.cpp:1398
      #7  0x00007fd30ed4add5 in start_thread () from /lib64/libpthread.so.0
      #8  0x00007fd30d107ead in clone () from /lib64/libc.so.6
       
      Thread 4 (Thread 0x7fd2fdcf0700 (LWP 31806)):
      #0  0x00007fd30ed4ed12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x000055612cc35a6b in safe_cond_timedwait (cond=0x55612e2545e0 <COND_checkpoint>, mp=0x55612e254520 <LOCK_checkpoint>, abstime=0x7fd2fdcefcb0, file=0x55612d09f880 "/home/stepan/mariadb/10.4/github/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/mariadb/10.4/github/mysys/thr_mutex.c:546
      #2  0x000055612ca8d109 in inline_mysql_cond_timedwait (that=0x55612e2545e0 <COND_checkpoint>, mutex=0x55612e254520 <LOCK_checkpoint>, abstime=0x7fd2fdcefcb0, src_file=0x55612d09f8c8 "/home/stepan/mariadb/10.4/github/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/mariadb/10.4/github/include/mysql/psi/mysql_thread.h:1204
      #3  0x000055612ca8d64d in my_service_thread_sleep (control=0x55612d806780 <checkpoint_control>, sleep_time=30000000000) at /home/stepan/mariadb/10.4/github/storage/maria/ma_servicethread.c:115
      #4  0x000055612ca8150a in ma_checkpoint_background (arg=0x1e) at /home/stepan/mariadb/10.4/github/storage/maria/ma_checkpoint.c:707
      #5  0x000055612cbbe4a5 in pfs_spawn_thread (arg=0x55612f4922d0) at /home/stepan/mariadb/10.4/github/storage/perfschema/pfs.cc:1862
      #6  0x00007fd30ed4add5 in start_thread () from /lib64/libpthread.so.0
      #7  0x00007fd30d107ead in clone () from /lib64/libc.so.6
       
      Thread 3 (Thread 0x7fd305e31700 (LWP 31801)):
      #0  0x00007fd30ed4e965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00007fd306ca9dd7 in gu::Lock::wait (cond=..., this=0x7fd305e30bf0) at galerautils/src/gu_lock.hpp:45
      #2  0x00007fd306ca9e69 in galera::ServiceThd::thd_func (arg=0x55612f39a6b8) at galera/src/galera_service_thd.cpp:30
      #3  0x00007fd30ed4add5 in start_thread () from /lib64/libpthread.so.0
      #4  0x00007fd30d107ead in clone () from /lib64/libc.so.6
       
      Thread 2 (Thread 0x7fd30432e700 (LWP 31804)):
      #0  0x00007fd30ed4e965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x000055612cc35769 in safe_cond_wait (cond=0x7fd2f4015a68, mp=0x7fd2f40159b8, file=0x55612cdb22c0 "/home/stepan/mariadb/10.4/github/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/mariadb/10.4/github/mysys/thr_mutex.c:492
      #2  0x000055612c2bd1b4 in inline_mysql_cond_wait (that=0x7fd2f4015a68, mutex=0x7fd2f40159b8, src_file=0x55612cdb24e0 "/home/stepan/mariadb/10.4/github/sql/wsrep_thd.h", src_line=69) at /home/stepan/mariadb/10.4/github/include/mysql/psi/mysql_thread.h:1178
      #3  0x000055612c2c0416 in Wsrep_thd_queue::pop_front (this=0x7fd2f4015960) at /home/stepan/mariadb/10.4/github/sql/wsrep_thd.h:69
      #4  0x000055612c2bee23 in wsrep_rollback_process (rollbacker=0x7fd2f4001d50, arg=0x55612f3e20b0) at /home/stepan/mariadb/10.4/github/sql/wsrep_thd.cc:266
      #5  0x000055612c2ade70 in start_wsrep_THD (arg=0x55612f3e20b0) at /home/stepan/mariadb/10.4/github/sql/wsrep_mysqld.cc:2708
      #6  0x000055612cbbe4a5 in pfs_spawn_thread (arg=0x55612f3eb960) at /home/stepan/mariadb/10.4/github/storage/perfschema/pfs.cc:1862
      #7  0x00007fd30ed4add5 in start_thread () from /lib64/libpthread.so.0
      #8  0x00007fd30d107ead in clone () from /lib64/libc.so.6
       
      Thread 1 (Thread 0x7fd30f15b880 (LWP 31799)):
      #0  0x00007fd30ed4f9d1 in pthread_kill () from /lib64/libpthread.so.0
      #1  0x000055612cc300f6 in my_write_core (sig=6) at /home/stepan/mariadb/10.4/github/mysys/stacktrace.c:481
      #2  0x000055612c39c964 in handle_fatal_signal (sig=6) at /home/stepan/mariadb/10.4/github/sql/signal_handler.cc:343
      #3  <signal handler called>
      #4  0x00007fd30d0fd20d in poll () from /lib64/libc.so.6
      #5  0x000055612bf012db in handle_connections_sockets () at /home/stepan/mariadb/10.4/github/sql/mysqld.cc:6455
      #6  0x000055612bf00576 in mysqld_main (argc=162, argv=0x55612f220130) at /home/stepan/mariadb/10.4/github/sql/mysqld.cc:5891
      #7  0x000055612bef4095 in main (argc=23, argv=0x7ffe548a0e98) at /home/stepan/mariadb/10.4/github/sql/main.cc:25
       
      Compressed file /home/stepan/mariadb/10.4/github/mysql-test/var/log/galera.galera_gcs_fc_limit-innodb/mysqld.1/data/core.31799
       - found 'core.1656' (1/5)
       
      Trying 'dbx' to get a backtrace
       
      Trying 'gdb' to get a backtrace from coredump /home/stepan/mariadb/10.4/github/mysql-test/var/log/galera.galera_gcs_fc_limit-innodb/mysqld.2/data/core.1656
      Core generated by '/home/stepan/mariadb/10.4/github/sql/mysqld'
      Output from gdb follows. The first stack trace is from the failing thread.
      The following stack traces are from all threads (so the failing one is
      duplicated).
      --------------------------
      [New LWP 1656]
      [New LWP 1861]
      [New LWP 1657]
      [New LWP 1658]
      [New LWP 1660]
      [New LWP 1659]
      [New LWP 1661]
      [New LWP 1864]
      [New LWP 1662]
      [New LWP 1874]
      [New LWP 1872]
      [New LWP 1867]
      [New LWP 1875]
      [New LWP 1883]
      [New LWP 1868]
      [New LWP 1877]
      [New LWP 1885]
      [New LWP 1876]
      [New LWP 1881]
      [New LWP 1880]
      [New LWP 1862]
      [New LWP 1884]
      [New LWP 1886]
      [New LWP 1866]
      [New LWP 2094]
      [New LWP 1882]
      [New LWP 1865]
      [New LWP 1879]
      [New LWP 1871]
      [New LWP 1878]
      [New LWP 1863]
      [New LWP 1873]
      [Thread debugging using libthread_db enabled]
      Using host libthread_db library "/lib64/libthread_db.so.1".
      Core was generated by `/home/stepan/mariadb/10.4/github/sql/mysqld --defaults-group-suffix=.2 --defaul'.
      Program terminated with signal 6, Aborted.
      #0  0x00007ff9375809d1 in pthread_kill () from /lib64/libpthread.so.0
      #0  0x00007ff9375809d1 in pthread_kill () from /lib64/libpthread.so.0
      #1  0x00005650e56980f6 in my_write_core (sig=6) at /home/stepan/mariadb/10.4/github/mysys/stacktrace.c:481
      #2  0x00005650e4e04964 in handle_fatal_signal (sig=6) at /home/stepan/mariadb/10.4/github/sql/signal_handler.cc:343
      #3  <signal handler called>
      #4  0x00007ff93592e20d in poll () from /lib64/libc.so.6
      #5  0x00005650e49692db in handle_connections_sockets () at /home/stepan/mariadb/10.4/github/sql/mysqld.cc:6455
      #6  0x00005650e4968576 in mysqld_main (argc=163, argv=0x5650e6dcf130) at /home/stepan/mariadb/10.4/github/sql/mysqld.cc:5891
      #7  0x00005650e495c095 in main (argc=23, argv=0x7ffe5325cce8) at /home/stepan/mariadb/10.4/github/sql/main.cc:25
       
      Thread 32 (Thread 0x7ff8f2ffd700 (LWP 1873)):
      #0  0x00007ff93757fd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00005650e5152d00 in os_event::timed_wait (this=0x5650e6fd7fd0, abstime=0x7ff8f2ffcd30) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:275
      #2  0x00005650e5152fa9 in os_event::wait_time_low (this=0x5650e6fd7fd0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:385
      #3  0x00005650e5153149 in os_event_wait_time_low (event=0x5650e6fd7fd0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:485
      #4  0x00005650e526d20f in srv_monitor_thread () at /home/stepan/mariadb/10.4/github/storage/innobase/srv/srv0srv.cc:1687
      #5  0x00007ff93757bdd5 in start_thread () from /lib64/libpthread.so.0
      #6  0x00007ff935938ead in clone () from /lib64/libc.so.6
       
      Thread 31 (Thread 0x7ff8ff7fe700 (LWP 1863)):
      #0  0x00007ff93714c644 in __io_getevents_0_4 () from /lib64/libaio.so.1
      #1  0x00005650e5144dad in LinuxAIOHandler::collect (this=0x7ff8ff7fdb00) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0file.cc:1809
      #2  0x00005650e51453f6 in LinuxAIOHandler::poll (this=0x7ff8ff7fdb00, m1=0x7ff8ff7fdba8, m2=0x7ff8ff7fdbb0, request=0x7ff8ff7fdbe0) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0file.cc:1968
      #3  0x00005650e5145559 in os_aio_linux_handler (global_segment=1, m1=0x7ff8ff7fdba8, m2=0x7ff8ff7fdbb0, request=0x7ff8ff7fdbe0) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0file.cc:2022
      #4  0x00005650e514a2c0 in os_aio_handler (segment=1, m1=0x7ff8ff7fdba8, m2=0x7ff8ff7fdbb0, request=0x7ff8ff7fdbe0) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0file.cc:5561
      #5  0x00005650e54004f9 in fil_aio_wait (segment=1) at /home/stepan/mariadb/10.4/github/storage/innobase/fil/fil0fil.cc:4357
      #6  0x00005650e5271e42 in io_handler_thread (arg=0x5650e64ab988 <n+8>) at /home/stepan/mariadb/10.4/github/storage/innobase/srv/srv0start.cc:324
      #7  0x00007ff93757bdd5 in start_thread () from /lib64/libpthread.so.0
      #8  0x00007ff935938ead in clone () from /lib64/libc.so.6
       
      Thread 30 (Thread 0x7ff8dbfff700 (LWP 1878)):
      #0  0x00007ff93757f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00005650e5153377 in os_event::wait (this=0x5650e6fd7dd0) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:158
      #2  0x00005650e5152e6b in os_event::wait_low (this=0x5650e6fd7dd0, reset_sig_count=460) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:325
      #3  0x00005650e515316e in os_event_wait_low (event=0x5650e6fd7dd0, reset_sig_count=0) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:502
      #4  0x00005650e526b328 in srv_resume_thread (slot=0x5650e6267d10 <srv_sys+848>, sig_count=0, wait=true, timeout_usec=0) at /home/stepan/mariadb/10.4/github/storage/innobase/srv/srv0srv.cc:909
      #5  0x00005650e526f5e9 in srv_worker_thread (arg=0x0) at /home/stepan/mariadb/10.4/github/storage/innobase/srv/srv0srv.cc:2517
      #6  0x00007ff93757bdd5 in start_thread () from /lib64/libpthread.so.0
      #7  0x00007ff935938ead in clone () from /lib64/libc.so.6
       
      Thread 29 (Thread 0x7ff8f37fe700 (LWP 1871)):
      #0  0x00007ff93757fd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00005650e5152d00 in os_event::timed_wait (this=0x5650e72d5120, abstime=0x7ff8f37fdd50) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:275
      #2  0x00005650e5152fa9 in os_event::wait_time_low (this=0x5650e72d5120, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:385
      #3  0x00005650e5153149 in os_event_wait_time_low (event=0x5650e72d5120, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:485
      #4  0x00005650e5115527 in lock_wait_timeout_thread () at /home/stepan/mariadb/10.4/github/storage/innobase/lock/lock0wait.cc:522
      #5  0x00007ff93757bdd5 in start_thread () from /lib64/libpthread.so.0
      #6  0x00007ff935938ead in clone () from /lib64/libc.so.6
       
      Thread 28 (Thread 0x7ff8db7fe700 (LWP 1879)):
      #0  0x00007ff93757f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00005650e5153377 in os_event::wait (this=0x5650e6fd7ed0) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:158
      #2  0x00005650e5152e6b in os_event::wait_low (this=0x5650e6fd7ed0, reset_sig_count=353) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:325
      #3  0x00005650e515316e in os_event_wait_low (event=0x5650e6fd7ed0, reset_sig_count=0) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:502
      #4  0x00005650e526b328 in srv_resume_thread (slot=0x5650e6267f10 <srv_sys+1360>, sig_count=0, wait=true, timeout_usec=0) at /home/stepan/mariadb/10.4/github/storage/innobase/srv/srv0srv.cc:909
      #5  0x00005650e526f5e9 in srv_worker_thread (arg=0x0) at /home/stepan/mariadb/10.4/github/storage/innobase/srv/srv0srv.cc:2517
      #6  0x00007ff93757bdd5 in start_thread () from /lib64/libpthread.so.0
      #7  0x00007ff935938ead in clone () from /lib64/libc.so.6
       
      Thread 27 (Thread 0x7ff8fe7fc700 (LWP 1865)):
      #0  0x00007ff93714c644 in __io_getevents_0_4 () from /lib64/libaio.so.1
      #1  0x00005650e5144dad in LinuxAIOHandler::collect (this=0x7ff8fe7fbb00) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0file.cc:1809
      #2  0x00005650e51453f6 in LinuxAIOHandler::poll (this=0x7ff8fe7fbb00, m1=0x7ff8fe7fbba8, m2=0x7ff8fe7fbbb0, request=0x7ff8fe7fbbe0) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0file.cc:1968
      #3  0x00005650e5145559 in os_aio_linux_handler (global_segment=3, m1=0x7ff8fe7fbba8, m2=0x7ff8fe7fbbb0, request=0x7ff8fe7fbbe0) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0file.cc:2022
      #4  0x00005650e514a2c0 in os_aio_handler (segment=3, m1=0x7ff8fe7fbba8, m2=0x7ff8fe7fbbb0, request=0x7ff8fe7fbbe0) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0file.cc:5561
      #5  0x00005650e54004f9 in fil_aio_wait (segment=3) at /home/stepan/mariadb/10.4/github/storage/innobase/fil/fil0fil.cc:4357
      #6  0x00005650e5271e42 in io_handler_thread (arg=0x5650e64ab998 <n+24>) at /home/stepan/mariadb/10.4/github/storage/innobase/srv/srv0start.cc:324
      #7  0x00007ff93757bdd5 in start_thread () from /lib64/libpthread.so.0
      #8  0x00007ff935938ead in clone () from /lib64/libc.so.6
       
      Thread 26 (Thread 0x7ff8d9ffb700 (LWP 1882)):
      #0  0x00007ff937582e3d in nanosleep () from /lib64/libpthread.so.0
      #1  0x00005650e51536b4 in os_thread_sleep (tm=1000000) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0thread.cc:225
      #2  0x00005650e5345dd3 in btr_defragment_thread () at /home/stepan/mariadb/10.4/github/storage/innobase/btr/btr0defragment.cc:705
      #3  0x00007ff93757bdd5 in start_thread () from /lib64/libpthread.so.0
      #4  0x00007ff935938ead in clone () from /lib64/libc.so.6
       
      Thread 25 (Thread 0x7ff91c8f2700 (LWP 2094)):
      #0  0x00007ff93757fd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00005650e569da6b in safe_cond_timedwait (cond=0x5650e642d060 <COND_thread_cache>, mp=0x5650e642fa20 <LOCK_thread_cache>, abstime=0x7ff91c8f1c60, file=0x5650e575bd68 "/home/stepan/mariadb/10.4/github/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/mariadb/10.4/github/mysys/thr_mutex.c:546
      #2  0x00005650e495ca15 in inline_mysql_cond_timedwait (that=0x5650e642d060 <COND_thread_cache>, mutex=0x5650e642fa20 <LOCK_thread_cache>, abstime=0x7ff91c8f1c60, src_file=0x5650e575ddb8 "/home/stepan/mariadb/10.4/github/sql/mysqld.cc", src_line=2686) at /home/stepan/mariadb/10.4/github/include/mysql/psi/mysql_thread.h:1204
      #3  0x00005650e4961b2c in cache_thread (thd=0x7ff8b0000b00) at /home/stepan/mariadb/10.4/github/sql/mysqld.cc:2685
      #4  0x00005650e4961dd2 in one_thread_per_connection_end (thd=0x7ff8b0000b00, put_in_cache=true) at /home/stepan/mariadb/10.4/github/sql/mysqld.cc:2771
      #5  0x00005650e4c26dcf in do_handle_one_connection (connect=0x5650e740f250) at /home/stepan/mariadb/10.4/github/sql/sql_connect.cc:1423
      #6  0x00005650e4c26a80 in handle_one_connection (arg=0x5650e740f250) at /home/stepan/mariadb/10.4/github/sql/sql_connect.cc:1316
      #7  0x00005650e56264a5 in pfs_spawn_thread (arg=0x5650e6ed2ee0) at /home/stepan/mariadb/10.4/github/storage/perfschema/pfs.cc:1862
      #8  0x00007ff93757bdd5 in start_thread () from /lib64/libpthread.so.0
      #9  0x00007ff935938ead in clone () from /lib64/libc.so.6
       
      Thread 24 (Thread 0x7ff8fdffb700 (LWP 1866)):
      #0  0x00007ff93714c644 in __io_getevents_0_4 () from /lib64/libaio.so.1
      #1  0x00005650e5144dad in LinuxAIOHandler::collect (this=0x7ff8fdffab00) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0file.cc:1809
      #2  0x00005650e51453f6 in LinuxAIOHandler::poll (this=0x7ff8fdffab00, m1=0x7ff8fdffaba8, m2=0x7ff8fdffabb0, request=0x7ff8fdffabe0) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0file.cc:1968
      #3  0x00005650e5145559 in os_aio_linux_handler (global_segment=4, m1=0x7ff8fdffaba8, m2=0x7ff8fdffabb0, request=0x7ff8fdffabe0) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0file.cc:2022
      #4  0x00005650e514a2c0 in os_aio_handler (segment=4, m1=0x7ff8fdffaba8, m2=0x7ff8fdffabb0, request=0x7ff8fdffabe0) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0file.cc:5561
      #5  0x00005650e54004f9 in fil_aio_wait (segment=4) at /home/stepan/mariadb/10.4/github/storage/innobase/fil/fil0fil.cc:4357
      #6  0x00005650e5271e42 in io_handler_thread (arg=0x5650e64ab9a0 <n+32>) at /home/stepan/mariadb/10.4/github/storage/innobase/srv/srv0start.cc:324
      #7  0x00007ff93757bdd5 in start_thread () from /lib64/libpthread.so.0
      #8  0x00007ff935938ead in clone () from /lib64/libc.so.6
       
      Thread 23 (Thread 0x7ff91c988700 (LWP 1886)):
      #0  0x00007ff93757f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00005650e569d769 in safe_cond_wait (cond=0x5650e642d0a0 <COND_slave_background>, mp=0x5650e6430320 <LOCK_slave_background>, file=0x5650e576c0c0 "/home/stepan/mariadb/10.4/github/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/mariadb/10.4/github/mysys/thr_mutex.c:492
      #2  0x00005650e498f08e in inline_mysql_cond_wait (that=0x5650e642d0a0 <COND_slave_background>, mutex=0x5650e6430320 <LOCK_slave_background>, src_file=0x5650e576cc20 "/home/stepan/mariadb/10.4/github/sql/slave.cc", src_line=519) at /home/stepan/mariadb/10.4/github/include/mysql/psi/mysql_thread.h:1167
      #3  0x00005650e49909cb in handle_slave_background (arg=0x0) at /home/stepan/mariadb/10.4/github/sql/slave.cc:519
      #4  0x00005650e56264a5 in pfs_spawn_thread (arg=0x5650e7481570) at /home/stepan/mariadb/10.4/github/storage/perfschema/pfs.cc:1862
      #5  0x00007ff93757bdd5 in start_thread () from /lib64/libpthread.so.0
      #6  0x00007ff935938ead in clone () from /lib64/libc.so.6
       
      Thread 22 (Thread 0x7ff8d8ff9700 (LWP 1884)):
      #0  0x00007ff93757f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00005650e569d769 in safe_cond_wait (cond=0x7ff8d8ff8c70, mp=0x7ff8d8ff8cb0, file=0x5650e598ace0 "/home/stepan/mariadb/10.4/github/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/mariadb/10.4/github/mysys/thr_mutex.c:492
      #2  0x00005650e504615d in inline_mysql_cond_wait (that=0x7ff8d8ff8c70, mutex=0x7ff8d8ff8cb0, src_file=0x5650e598c7f8 "/home/stepan/mariadb/10.4/github/storage/innobase/handler/ha_innodb.cc", src_line=288) at /home/stepan/mariadb/10.4/github/include/mysql/psi/mysql_thread.h:1178
      #3  0x00005650e504ba55 in thd_destructor_proxy () at /home/stepan/mariadb/10.4/github/storage/innobase/handler/ha_innodb.cc:288
      #4  0x00005650e56264a5 in pfs_spawn_thread (arg=0x5650e734c320) at /home/stepan/mariadb/10.4/github/storage/perfschema/pfs.cc:1862
      #5  0x00007ff93757bdd5 in start_thread () from /lib64/libpthread.so.0
      #6  0x00007ff935938ead in clone () from /lib64/libc.so.6
       
      Thread 21 (Thread 0x7ff8fffff700 (LWP 1862)):
      #0  0x00007ff93714c644 in __io_getevents_0_4 () from /lib64/libaio.so.1
      #1  0x00005650e5144dad in LinuxAIOHandler::collect (this=0x7ff8ffffeb00) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0file.cc:1809
      #2  0x00005650e51453f6 in LinuxAIOHandler::poll (this=0x7ff8ffffeb00, m1=0x7ff8ffffeba8, m2=0x7ff8ffffebb0, request=0x7ff8ffffebe0) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0file.cc:1968
      #3  0x00005650e5145559 in os_aio_linux_handler (global_segment=0, m1=0x7ff8ffffeba8, m2=0x7ff8ffffebb0, request=0x7ff8ffffebe0) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0file.cc:2022
      #4  0x00005650e514a2c0 in os_aio_handler (segment=0, m1=0x7ff8ffffeba8, m2=0x7ff8ffffebb0, request=0x7ff8ffffebe0) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0file.cc:5561
      #5  0x00005650e54004f9 in fil_aio_wait (segment=0) at /home/stepan/mariadb/10.4/github/storage/innobase/fil/fil0fil.cc:4357
      #6  0x00005650e5271e42 in io_handler_thread (arg=0x5650e64ab980 <n>) at /home/stepan/mariadb/10.4/github/storage/innobase/srv/srv0start.cc:324
      #7  0x00007ff93757bdd5 in start_thread () from /lib64/libpthread.so.0
      #8  0x00007ff935938ead in clone () from /lib64/libc.so.6
       
      Thread 20 (Thread 0x7ff8daffd700 (LWP 1880)):
      #0  0x00007ff93757f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00005650e5153377 in os_event::wait (this=0x5650e6fd7e50) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:158
      #2  0x00005650e5152e6b in os_event::wait_low (this=0x5650e6fd7e50, reset_sig_count=378) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:325
      #3  0x00005650e515316e in os_event_wait_low (event=0x5650e6fd7e50, reset_sig_count=0) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:502
      #4  0x00005650e526b328 in srv_resume_thread (slot=0x5650e6267e10 <srv_sys+1104>, sig_count=0, wait=true, timeout_usec=0) at /home/stepan/mariadb/10.4/github/storage/innobase/srv/srv0srv.cc:909
      #5  0x00005650e526f5e9 in srv_worker_thread (arg=0x0) at /home/stepan/mariadb/10.4/github/storage/innobase/srv/srv0srv.cc:2517
      #6  0x00007ff93757bdd5 in start_thread () from /lib64/libpthread.so.0
      #7  0x00007ff935938ead in clone () from /lib64/libc.so.6
       
      Thread 19 (Thread 0x7ff8da7fc700 (LWP 1881)):
      #0  0x00007ff93757f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00005650e5153377 in os_event::wait (this=0x5650e6fd8050) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:158
      #2  0x00005650e5152e6b in os_event::wait_low (this=0x5650e6fd8050, reset_sig_count=1) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:325
      #3  0x00005650e515316e in os_event_wait_low (event=0x5650e6fd8050, reset_sig_count=0) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:502
      #4  0x00005650e5370e86 in buf_dump_thread () at /home/stepan/mariadb/10.4/github/storage/innobase/buf/buf0dump.cc:833
      #5  0x00007ff93757bdd5 in start_thread () from /lib64/libpthread.so.0
      #6  0x00007ff935938ead in clone () from /lib64/libc.so.6
       
      Thread 18 (Thread 0x7ff8f17fa700 (LWP 1876)):
      #0  0x00007ff937582e3d in nanosleep () from /lib64/libpthread.so.0
      #1  0x00005650e51536b4 in os_thread_sleep (tm=1000000) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0thread.cc:225
      #2  0x00005650e526ed65 in srv_master_sleep () at /home/stepan/mariadb/10.4/github/storage/innobase/srv/srv0srv.cc:2334
      #3  0x00005650e526eecd in srv_master_thread (arg=0x0) at /home/stepan/mariadb/10.4/github/storage/innobase/srv/srv0srv.cc:2375
      #4  0x00007ff93757bdd5 in start_thread () from /lib64/libpthread.so.0
      #5  0x00007ff935938ead in clone () from /lib64/libc.so.6
       
      Thread 17 (Thread 0x7ff91c9d3700 (LWP 1885)):
      #0  0x00007ff93587240a in sigwaitinfo () from /lib64/libc.so.6
      #1  0x00005650e495c114 in my_sigwait (set=0x7ff91c9d2cf0, sig=0x7ff91c9d2cb4, code=0x7ff91c9d2cb8) at /home/stepan/mariadb/10.4/github/include/my_pthread.h:198
      #2  0x00005650e4962a8d in signal_hand (arg=0x0) at /home/stepan/mariadb/10.4/github/sql/mysqld.cc:3243
      #3  0x00005650e56264a5 in pfs_spawn_thread (arg=0x5650e6fe0680) at /home/stepan/mariadb/10.4/github/storage/perfschema/pfs.cc:1862
      #4  0x00007ff93757bdd5 in start_thread () from /lib64/libpthread.so.0
      #5  0x00007ff935938ead in clone () from /lib64/libc.so.6
       
      Thread 16 (Thread 0x7ff8f0ff9700 (LWP 1877)):
      #0  0x00007ff93757f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00005650e5153377 in os_event::wait (this=0x5650e6fd7d50) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:158
      #2  0x00005650e5152e6b in os_event::wait_low (this=0x5650e6fd7d50, reset_sig_count=127) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:325
      #3  0x00005650e515316e in os_event_wait_low (event=0x5650e6fd7d50, reset_sig_count=127) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:502
      #4  0x00005650e526b328 in srv_resume_thread (slot=0x5650e6267c10 <srv_sys+592>, sig_count=127, wait=true, timeout_usec=0) at /home/stepan/mariadb/10.4/github/storage/innobase/srv/srv0srv.cc:909
      #5  0x00005650e526faa0 in srv_purge_coordinator_suspend (slot=0x5650e6267c10 <srv_sys+592>, rseg_history_len=12) at /home/stepan/mariadb/10.4/github/storage/innobase/srv/srv0srv.cc:2657
      #6  0x00005650e526fdd1 in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/mariadb/10.4/github/storage/innobase/srv/srv0srv.cc:2738
      #7  0x00007ff93757bdd5 in start_thread () from /lib64/libpthread.so.0
      #8  0x00007ff935938ead in clone () from /lib64/libc.so.6
       
      Thread 15 (Thread 0x7ff8fcff9700 (LWP 1868)):
      #0  0x00007ff93757fd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00005650e5152d00 in os_event::timed_wait (this=0x5650e6fd80d0, abstime=0x7ff8fcff8b00) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:275
      #2  0x00005650e5152fa9 in os_event::wait_time_low (this=0x5650e6fd80d0, time_in_usec=613000, reset_sig_count=2) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:385
      #3  0x00005650e5153149 in os_event_wait_time_low (event=0x5650e6fd80d0, time_in_usec=613000, reset_sig_count=2) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:485
      #4  0x00005650e537d358 in pc_sleep_if_needed (next_loop_time=959894955, sig_count=2, cur_time=959894342) at /home/stepan/mariadb/10.4/github/storage/innobase/buf/buf0flu.cc:2643
      #5  0x00005650e537e4fc in buf_flush_page_cleaner_coordinator () at /home/stepan/mariadb/10.4/github/storage/innobase/buf/buf0flu.cc:3102
      #6  0x00007ff93757bdd5 in start_thread () from /lib64/libpthread.so.0
      #7  0x00007ff935938ead in clone () from /lib64/libc.so.6
       
      Thread 14 (Thread 0x7ff8d97fa700 (LWP 1883)):
      #0  0x00007ff93757f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00005650e5153377 in os_event::wait (this=0x5650e6fd8150) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:158
      #2  0x00005650e5152e6b in os_event::wait_low (this=0x5650e6fd8150, reset_sig_count=1) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:325
      #3  0x00005650e515316e in os_event_wait_low (event=0x5650e6fd8150, reset_sig_count=0) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:502
      #4  0x00005650e53567d9 in buf_resize_thread () at /home/stepan/mariadb/10.4/github/storage/innobase/buf/buf0buf.cc:3199
      #5  0x00007ff93757bdd5 in start_thread () from /lib64/libpthread.so.0
      #6  0x00007ff935938ead in clone () from /lib64/libc.so.6
       
      Thread 13 (Thread 0x7ff8f1ffb700 (LWP 1875)):
      #0  0x00007ff93757fd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00005650e5152d00 in os_event::timed_wait (this=0x5650e7239e70, abstime=0x7ff8f1ffab70) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:275
      #2  0x00005650e5152fa9 in os_event::wait_time_low (this=0x5650e7239e70, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:385
      #3  0x00005650e5153149 in os_event_wait_time_low (event=0x5650e7239e70, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:485
      #4  0x00005650e52db563 in ib_wqueue_timedwait (wq=0x5650e723f990, wait_in_usecs=5000000) at /home/stepan/mariadb/10.4/github/storage/innobase/ut/ut0wqueue.cc:163
      #5  0x00005650e544e23d in fts_optimize_thread (arg=0x5650e723f990) at /home/stepan/mariadb/10.4/github/storage/innobase/fts/fts0opt.cc:2865
      #6  0x00007ff93757bdd5 in start_thread () from /lib64/libpthread.so.0
      #7  0x00007ff935938ead in clone () from /lib64/libc.so.6
       
      Thread 12 (Thread 0x7ff8fd7fa700 (LWP 1867)):
      #0  0x00007ff93714c644 in __io_getevents_0_4 () from /lib64/libaio.so.1
      #1  0x00005650e5144dad in LinuxAIOHandler::collect (this=0x7ff8fd7f9b00) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0file.cc:1809
      #2  0x00005650e51453f6 in LinuxAIOHandler::poll (this=0x7ff8fd7f9b00, m1=0x7ff8fd7f9ba8, m2=0x7ff8fd7f9bb0, request=0x7ff8fd7f9be0) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0file.cc:1968
      #3  0x00005650e5145559 in os_aio_linux_handler (global_segment=5, m1=0x7ff8fd7f9ba8, m2=0x7ff8fd7f9bb0, request=0x7ff8fd7f9be0) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0file.cc:2022
      #4  0x00005650e514a2c0 in os_aio_handler (segment=5, m1=0x7ff8fd7f9ba8, m2=0x7ff8fd7f9bb0, request=0x7ff8fd7f9be0) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0file.cc:5561
      #5  0x00005650e54004f9 in fil_aio_wait (segment=5) at /home/stepan/mariadb/10.4/github/storage/innobase/fil/fil0fil.cc:4357
      #6  0x00005650e5271e42 in io_handler_thread (arg=0x5650e64ab9a8 <n+40>) at /home/stepan/mariadb/10.4/github/storage/innobase/srv/srv0start.cc:324
      #7  0x00007ff93757bdd5 in start_thread () from /lib64/libpthread.so.0
      #8  0x00007ff935938ead in clone () from /lib64/libc.so.6
       
      Thread 11 (Thread 0x7ff8f3fff700 (LWP 1872)):
      #0  0x00007ff93757fd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00005650e5152d00 in os_event::timed_wait (this=0x5650e6fd7f50, abstime=0x7ff8f3ffebb0) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:275
      #2  0x00005650e5152fa9 in os_event::wait_time_low (this=0x5650e6fd7f50, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:385
      #3  0x00005650e5153149 in os_event_wait_time_low (event=0x5650e6fd7f50, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:485
      #4  0x00005650e526d789 in srv_error_monitor_thread () at /home/stepan/mariadb/10.4/github/storage/innobase/srv/srv0srv.cc:1851
      #5  0x00007ff93757bdd5 in start_thread () from /lib64/libpthread.so.0
      #6  0x00007ff935938ead in clone () from /lib64/libc.so.6
       
      Thread 10 (Thread 0x7ff8f27fc700 (LWP 1874)):
      #0  0x00007ff93757fd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00005650e5152d00 in os_event::timed_wait (this=0x5650e73071e0, abstime=0x7ff8f27fbd70) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:275
      #2  0x00005650e5152fa9 in os_event::wait_time_low (this=0x5650e73071e0, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:385
      #3  0x00005650e5153149 in os_event_wait_time_low (event=0x5650e73071e0, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0event.cc:485
      #4  0x00005650e53e76ca in dict_stats_thread () at /home/stepan/mariadb/10.4/github/storage/innobase/dict/dict0stats_bg.cc:476
      #5  0x00007ff93757bdd5 in start_thread () from /lib64/libpthread.so.0
      #6  0x00007ff935938ead in clone () from /lib64/libc.so.6
       
      Thread 9 (Thread 0x7ff92c393700 (LWP 1662)):
      #0  0x00007ff93757f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00007ff92f3cce42 in fifo_lock_get (q=q@entry=0x7ff92f165010) at galerautils/src/gu_fifo.c:262
      #2  0x00007ff92f3cd4b1 in gu_fifo_get_head (q=0x7ff92f165010, err=0x7ff92c39201c) at galerautils/src/gu_fifo.c:330
      #3  0x00007ff92f4b0d1c in gcs_recv (conn=0x5650e6f60b70, action=0x7ff92c392060) at gcs/src/gcs.cpp:2024
      #4  0x00007ff92f4ddf6c in galera::GcsActionSource::process (this=0x5650e6f603e0, recv_ctx=0x7ff92c392480, exit_loop=@0x7ff92c39224e: false) at galera/src/gcs_action_source.cpp:161
      #5  0x00007ff92f4fcf24 in galera::ReplicatorSMM::async_recv (this=0x5650e6f48f90, recv_ctx=0x7ff92c392480) at galera/src/replicator_smm.cpp:391
      #6  0x00005650e5753e7a in wsrep::wsrep_provider_v26::run_applier (this=0x5650e6ec1920, applier_ctx=0x7ff92c392480) at /home/stepan/mariadb/10.4/github/wsrep-lib/src/wsrep_provider_v26.cpp:727
      #7  0x00005650e4d25787 in wsrep_replication_process (thd=0x7ff908000b00, arg=0x5650e6fa5b50) at /home/stepan/mariadb/10.4/github/sql/wsrep_thd.cc:62
      #8  0x00005650e4d15e70 in start_wsrep_THD (arg=0x5650e6fa5b50) at /home/stepan/mariadb/10.4/github/sql/wsrep_mysqld.cc:2708
      #9  0x00005650e56264a5 in pfs_spawn_thread (arg=0x5650e6f79790) at /home/stepan/mariadb/10.4/github/storage/perfschema/pfs.cc:1862
      #10 0x00007ff93757bdd5 in start_thread () from /lib64/libpthread.so.0
      #11 0x00007ff935938ead in clone () from /lib64/libc.so.6
       
      Thread 8 (Thread 0x7ff8feffd700 (LWP 1864)):
      #0  0x00007ff93714c644 in __io_getevents_0_4 () from /lib64/libaio.so.1
      #1  0x00005650e5144dad in LinuxAIOHandler::collect (this=0x7ff8feffcb00) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0file.cc:1809
      #2  0x00005650e51453f6 in LinuxAIOHandler::poll (this=0x7ff8feffcb00, m1=0x7ff8feffcba8, m2=0x7ff8feffcbb0, request=0x7ff8feffcbe0) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0file.cc:1968
      #3  0x00005650e5145559 in os_aio_linux_handler (global_segment=2, m1=0x7ff8feffcba8, m2=0x7ff8feffcbb0, request=0x7ff8feffcbe0) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0file.cc:2022
      #4  0x00005650e514a2c0 in os_aio_handler (segment=2, m1=0x7ff8feffcba8, m2=0x7ff8feffcbb0, request=0x7ff8feffcbe0) at /home/stepan/mariadb/10.4/github/storage/innobase/os/os0file.cc:5561
      #5  0x00005650e54004f9 in fil_aio_wait (segment=2) at /home/stepan/mariadb/10.4/github/storage/innobase/fil/fil0fil.cc:4357
      #6  0x00005650e5271e42 in io_handler_thread (arg=0x5650e64ab990 <n+16>) at /home/stepan/mariadb/10.4/github/storage/innobase/srv/srv0start.cc:324
      #7  0x00007ff93757bdd5 in start_thread () from /lib64/libpthread.so.0
      #8  0x00007ff935938ead in clone () from /lib64/libc.so.6
       
      Thread 7 (Thread 0x7ff92c3de700 (LWP 1661)):
      #0  0x00007ff93757f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00005650e569d769 in safe_cond_wait (cond=0x7ff914014758, mp=0x7ff9140146a8, file=0x5650e581a2c0 "/home/stepan/mariadb/10.4/github/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/mariadb/10.4/github/mysys/thr_mutex.c:492
      #2  0x00005650e4d251b4 in inline_mysql_cond_wait (that=0x7ff914014758, mutex=0x7ff9140146a8, src_file=0x5650e581a4e0 "/home/stepan/mariadb/10.4/github/sql/wsrep_thd.h", src_line=69) at /home/stepan/mariadb/10.4/github/include/mysql/psi/mysql_thread.h:1178
      #3  0x00005650e4d28416 in Wsrep_thd_queue::pop_front (this=0x7ff914014650) at /home/stepan/mariadb/10.4/github/sql/wsrep_thd.h:69
      #4  0x00005650e4d26e23 in wsrep_rollback_process (rollbacker=0x7ff914001d50, arg=0x5650e6faf290) at /home/stepan/mariadb/10.4/github/sql/wsrep_thd.cc:266
      #5  0x00005650e4d15e70 in start_wsrep_THD (arg=0x5650e6faf290) at /home/stepan/mariadb/10.4/github/sql/wsrep_mysqld.cc:2708
      #6  0x00005650e56264a5 in pfs_spawn_thread (arg=0x5650e6f792a0) at /home/stepan/mariadb/10.4/github/storage/perfschema/pfs.cc:1862
      #7  0x00007ff93757bdd5 in start_thread () from /lib64/libpthread.so.0
      #8  0x00007ff935938ead in clone () from /lib64/libc.so.6
       
      Thread 6 (Thread 0x7ff92cbdf700 (LWP 1659)):
      #0  0x00007ff935939483 in epoll_wait () from /lib64/libc.so.6
      #1  0x00007ff92f47b548 in asio::detail::epoll_reactor::run (this=0x5650e6f8de40, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391
      #2  0x00007ff92f47fc40 in asio::detail::task_io_service::do_run_one (this=this@entry=0x5650e6f8dce0, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355
      #3  0x00007ff92f47fef2 in asio::detail::task_io_service::run (this=0x5650e6f8dce0, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148
      #4  0x00007ff92f49797d in asio::io_service::run (this=0x5650e6f8ca40) at asio/asio/impl/io_service.ipp:58
      #5  0x00007ff92f4980d1 in gcomm::AsioProtonet::event_loop (this=0x5650e6f8c9a0, period=...) at gcomm/src/asio_protonet.cpp:116
      #6  0x00007ff92f4b6f3d in GCommConn::run (this=0x5650e6f8d4c0) at gcs/src/gcs_gcomm.cpp:498
      #7  0x00007ff92f4b9fe9 in GCommConn::run_fn (arg=<optimized out>) at gcs/src/gcs_gcomm.cpp:209
      #8  0x00007ff93757bdd5 in start_thread () from /lib64/libpthread.so.0
      #9  0x00007ff935938ead in clone () from /lib64/libc.so.6
       
      Thread 5 (Thread 0x7ff91fffe700 (LWP 1660)):
      #0  0x00007ff93757f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00007ff92f4b54b7 in gu::Lock::wait (cond=..., this=0x7ff91fffd850) at galerautils/src/gu_lock.hpp:45
      #2  0x00007ff92f4b7ffb in RecvBuf::front (this=this@entry=0x5650e6f8d6a8, timeout=...) at gcs/src/gcs_gcomm.cpp:124
      #3  0x00007ff92f4b5b96 in gcomm_recv (backend=<optimized out>, msg=0x5650e6f60e50, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673
      #4  0x00007ff92f4a9b2c in core_msg_recv (backend=backend@entry=0x5650e6f60fb0, recv_msg=recv_msg@entry=0x5650e6f60e50, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480
      #5  0x00007ff92f4acda4 in gcs_core_recv (conn=0x5650e6f60df0, recv_act=recv_act@entry=0x7ff91fffdd70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184
      #6  0x00007ff92f4b4028 in gcs_recv_thread (arg=0x5650e6f60b70) at gcs/src/gcs.cpp:1398
      #7  0x00007ff93757bdd5 in start_thread () from /lib64/libpthread.so.0
      #8  0x00007ff935938ead in clone () from /lib64/libc.so.6
       
      Thread 4 (Thread 0x7ff92f063700 (LWP 1658)):
      #0  0x00007ff93757f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00007ff92f4dadd7 in gu::Lock::wait (cond=..., this=0x7ff92f062bf0) at galerautils/src/gu_lock.hpp:45
      #2  0x00007ff92f4dae69 in galera::ServiceThd::thd_func (arg=0x5650e6f496a8) at galera/src/galera_service_thd.cpp:30
      #3  0x00007ff93757bdd5 in start_thread () from /lib64/libpthread.so.0
      #4  0x00007ff935938ead in clone () from /lib64/libc.so.6
       
      Thread 3 (Thread 0x7ff930014700 (LWP 1657)):
      #0  0x00007ff93757fd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00005650e569da6b in safe_cond_timedwait (cond=0x5650e6ccece0 <COND_timer>, mp=0x5650e6ccec20 <LOCK_timer>, abstime=0x7ff930013d70, file=0x5650e5b50450 "/home/stepan/mariadb/10.4/github/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/mariadb/10.4/github/mysys/thr_mutex.c:546
      #2  0x00005650e569eaae in inline_mysql_cond_timedwait (that=0x5650e6ccece0 <COND_timer>, mutex=0x5650e6ccec20 <LOCK_timer>, abstime=0x7ff930013d70, src_file=0x5650e5b50498 "/home/stepan/mariadb/10.4/github/mysys/thr_timer.c", src_line=292) at /home/stepan/mariadb/10.4/github/include/mysql/psi/mysql_thread.h:1204
      #3  0x00005650e569f671 in timer_handler (arg=0x0) at /home/stepan/mariadb/10.4/github/mysys/thr_timer.c:292
      #4  0x00005650e56264a5 in pfs_spawn_thread (arg=0x5650e6f15250) at /home/stepan/mariadb/10.4/github/storage/perfschema/pfs.cc:1862
      #5  0x00007ff93757bdd5 in start_thread () from /lib64/libpthread.so.0
      #6  0x00007ff935938ead in clone () from /lib64/libc.so.6
       
      Thread 2 (Thread 0x7ff91dfd5700 (LWP 1861)):
      #0  0x00007ff93757fd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00005650e569da6b in safe_cond_timedwait (cond=0x5650e6cbc5e0 <COND_checkpoint>, mp=0x5650e6cbc520 <LOCK_checkpoint>, abstime=0x7ff91dfd4cb0, file=0x5650e5b07880 "/home/stepan/mariadb/10.4/github/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/mariadb/10.4/github/mysys/thr_mutex.c:546
      #2  0x00005650e54f5109 in inline_mysql_cond_timedwait (that=0x5650e6cbc5e0 <COND_checkpoint>, mutex=0x5650e6cbc520 <LOCK_checkpoint>, abstime=0x7ff91dfd4cb0, src_file=0x5650e5b078c8 "/home/stepan/mariadb/10.4/github/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/mariadb/10.4/github/include/mysql/psi/mysql_thread.h:1204
      #3  0x00005650e54f564d in my_service_thread_sleep (control=0x5650e626e780 <checkpoint_control>, sleep_time=30000000000) at /home/stepan/mariadb/10.4/github/storage/maria/ma_servicethread.c:115
      #4  0x00005650e54e950a in ma_checkpoint_background (arg=0x1e) at /home/stepan/mariadb/10.4/github/storage/maria/ma_checkpoint.c:707
      #5  0x00005650e56264a5 in pfs_spawn_thread (arg=0x5650e6fcb4e0) at /home/stepan/mariadb/10.4/github/storage/perfschema/pfs.cc:1862
      #6  0x00007ff93757bdd5 in start_thread () from /lib64/libpthread.so.0
      #7  0x00007ff935938ead in clone () from /lib64/libc.so.6
       
      Thread 1 (Thread 0x7ff93798c880 (LWP 1656)):
      #0  0x00007ff9375809d1 in pthread_kill () from /lib64/libpthread.so.0
      #1  0x00005650e56980f6 in my_write_core (sig=6) at /home/stepan/mariadb/10.4/github/mysys/stacktrace.c:481
      #2  0x00005650e4e04964 in handle_fatal_signal (sig=6) at /home/stepan/mariadb/10.4/github/sql/signal_handler.cc:343
      #3  <signal handler called>
      #4  0x00007ff93592e20d in poll () from /lib64/libc.so.6
      #5  0x00005650e49692db in handle_connections_sockets () at /home/stepan/mariadb/10.4/github/sql/mysqld.cc:6455
      #6  0x00005650e4968576 in mysqld_main (argc=163, argv=0x5650e6dcf130) at /home/stepan/mariadb/10.4/github/sql/mysqld.cc:5891
      #7  0x00005650e495c095 in main (argc=23, argv=0x7ffe5325cce8) at /home/stepan/mariadb/10.4/github/sql/main.cc:25
       
      Compressed file /home/stepan/mariadb/10.4/github/mysql-test/var/log/galera.galera_gcs_fc_limit-innodb/mysqld.2/data/core.1656
       - saving '/home/stepan/mariadb/10.4/github/mysql-test/var/log/galera.galera_gcs_fc_limit-innodb/' to '/home/stepan/mariadb/10.4/github/mysql-test/var/log/galera.galera_gcs_fc_limit-innodb/'
      ***Warnings generated in error logs during shutdown after running tests: galera.galera_fk_multitable galera.galera_toi_ddl_fk_insert galera.galera_fk_selfreferential galera.galera_create_function galera.galera_commit_empty galera.MW-286 galera.galera_events galera.galera_change_user galera.galera_alter_engine_innodb galera.galera_bf_abort_lock_table galera.galera_FK_duplicate_client_insert galera.MW-258 galera.galera_ftwrl_drain galera.galera_rsu_add_pk galera.MW-357 galera.galera_toi_ddl_fk_update galera.MDEV-16509 galera.galera_fk_cascade_update galera.galera_ddl_multiline galera.galera_bf_abort galera.galera_bf_abort_ftwrl galera.galera_create_table_as_select galera.GCF-939 galera.MW-369 galera.galera_fk_setnull galera.galera_alter_table_force galera.galera_fk_cascade_delete galera.GAL-480 galera.galera_bf_abort_sleep galera.galera_create_table_like galera.MDEV-20225 galera.galera_toi_ddl_error galera.MW-328D galera.galera_var_load_data_splitting galera.MW-416 galera.galera_wsrep_desync_wsrep_on galera.galera_events2 galera.basic galera.MW-309 galera.galera_bf_abort_at_after_statement galera.galera_enum galera.GAL-382 galera.MW-402 galera.galera_fk_multicolumn galera.MW-285 galera.enforce_storage_engine galera.binlog_checksum galera.galera_binlog_cache_size galera.create galera.galera_admin galera.GAL-401 galera.galera_create_procedure galera.MW-292 galera.galera_alter_engine_myisam galera.galera_create_trigger galera.galera_rsu_drop_pk galera.galera_bf_abort_flush_for_export galera.galera_ftwrl galera.MW-328E galera.galera_delete_limit galera.galera_drop_database galera.galera_desync_overlapped galera.galera_bf_abort_for_update galera.galera_applier_ftwrl_table galera.galera_drop_multi galera.galera_defaults galera.fk galera.MW-252 galera.galera_bf_abort_get_lock galera.galera_gcs_fc_limit galera.galera_fk_mismatch galera.galera_fk_no_pk galera.ev51914 galera.galera_binlog_row_image galera.galera#505 galera.MW-259 galera.galera_bf_abort_shutdown galera.galera_fk_conflict galera.galera#500
       
      Attempting backtrace. You can use the following information to find out
      Attempting backtrace. You can use the following information to find out
      

      See also other logs attached 191027_HDD_galera_gcs_fc_limit.zip.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              jplindst Jan Lindström
              Reporter:
              stepan.patryshev Stepan Patryshev
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: