Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-20899

Galera test failure on galera_sr.galera_sr_cc_master: Server crash with signal 6

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Minor
    • Resolution: Cannot Reproduce
    • 10.4.9
    • N/A
    • Galera, Tests
    • CentOS: 7.6.1810.

    Description

      It sporadically fails only on the local suite runs with --parallel, not on the bb.

      MariaDB Version 10.4.9-MariaDB-debug: Repository: MariaDB/server; branch 10.4; Revision 2b5f4b3ed68585b310b7ebede474928ff90d9aa2; debug built from sources.

      Galera lib 26.4.3(r4535): Repository: MariaDB/galera; branch mariadb-4.x; Revision 752664dc3c7065d8e0c73ac99d0028a5f84eb250; debug built from sources.

      Run:

      ./mtr --suite galera_sr --force --big-test --max-test-fail=0 --parallel=auto --timer --timestamp --timediff
      

      stdout.log (191025_SSD_galera_sr_stdout.log):

      2019-10-25 22:28:03 2 [Warning] Aborted connection 2 to db: 'unconnected' user: 'unauthenticated' host: '' (This connection closed normally without authentication)
      mysqld: /home/stepan/mariadb/10.4/git/include/mysql/psi/mysql_socket.h:779: inline_mysql_socket_recv: Assertion `mysql_socket.fd != -1' failed.
      191025 22:28:03 [ERROR] mysqld got signal 6 ;

      191025 22:28:06 +5.780 galera_sr.galera_sr_cc_master 'innodb'   w4 [ fail ]
              Test ended at 2019-10-25 22:28:06
       
      CURRENT_TEST: galera_sr.galera_sr_cc_master
      mysqltest: At line 44: query 'SET GLOBAL wsrep_cluster_address = ''' failed: 2013: Lost connection to MySQL server during query
       
      The result from queries just before the failure was:
      < snip >
      SET AUTOCOMMIT=OFF;
      START TRANSACTION;
      INSERT INTO t1 VALUES (1);
      INSERT INTO t1 VALUES (2);
      INSERT INTO t1 VALUES (3);
      INSERT INTO t1 VALUES (4);
      INSERT INTO t1 VALUES (5);
      SELECT COUNT(*) FROM mysql.wsrep_streaming_log;
      COUNT(*)
      5
      connection node_1;
      SELECT COUNT(*) FROM mysql.wsrep_streaming_log;
      COUNT(*)
      5
      connect node_2a, 127.0.0.1, root, , test, $NODE_MYPORT_2;
      connection node_2a;
      SET SESSION wsrep_sync_wait=0;
      SET GLOBAL wsrep_cluster_address = '';
       
      More results from queries before failure can be found in /home/stepan/mariadb/10.4/git/mysql-test/var/4/log/galera_sr_cc_master.log
       
       
      Server [mysqld.2 - pid: 12603, winpid: 12603, exit: 256] failed during test run
      Server log from this test:
      ----------SERVER LOG START-----------
      2019-10-25 22:28:02 140 [Note] WSREP: Stop replication by 140
      2019-10-25 22:28:02 140 [Note] WSREP: Server status change synced -> disconnecting
      2019-10-25 22:28:02 140 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      2019-10-25 22:28:02 140 [Note] WSREP: Closing send monitor...
      2019-10-25 22:28:02 140 [Note] WSREP: Closed send monitor.
      2019-10-25 22:28:02 140 [Note] WSREP: gcomm: terminating thread
      2019-10-25 22:28:02 140 [Note] WSREP: gcomm: joining thread
      2019-10-25 22:28:02 140 [Note] WSREP: gcomm: closing backend
      2019-10-25 22:28:03 140 [Note] WSREP: view(view_id(NON_PRIM,7c15e3e0,2) memb {
      	7c79d6ce,0
      } joined {
      } left {
      } partitioned {
      	7c15e3e0,0
      })
      2019-10-25 22:28:03 140 [Note] WSREP: PC protocol downgrade 1 -> 0
      2019-10-25 22:28:03 140 [Note] WSREP: view((empty))
      2019-10-25 22:28:03 140 [Note] WSREP: gcomm: closed
      2019-10-25 22:28:03 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
      2019-10-25 22:28:03 0 [Note] WSREP: Flow-control interval: [16, 16]
      2019-10-25 22:28:03 0 [Note] WSREP: Received NON-PRIMARY.
      2019-10-25 22:28:03 0 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 216)
      2019-10-25 22:28:03 0 [Note] WSREP: New SELF-LEAVE.
      2019-10-25 22:28:03 0 [Note] WSREP: Flow-control interval: [0, 0]
      2019-10-25 22:28:03 0 [Note] WSREP: Received SELF-LEAVE. Closing connection.
      2019-10-25 22:28:03 0 [Note] WSREP: Shifting OPEN -> CLOSED (TO: -1)
      2019-10-25 22:28:03 0 [Note] WSREP: RECV thread exiting 0: Success
      2019-10-25 22:28:03 140 [Note] WSREP: recv_thread() joined.
      2019-10-25 22:28:03 140 [Note] WSREP: Closing replication queue.
      2019-10-25 22:28:03 140 [Note] WSREP: Closing slave action queue.
      2019-10-25 22:28:03 2 [Note] WSREP: ####### processing CC -1, local, ordered
      2019-10-25 22:28:03 2 [Note] WSREP: ####### My UUID: 7c79d6ce-f75d-11e9-8a80-13083ff677cf
      2019-10-25 22:28:03 2 [Note] WSREP: ####### ST not required
      2019-10-25 22:28:03 2 [Note] WSREP: ================================================
      View:
        id: 7c1cbeb7-f75d-11e9-a0f5-662e63655745:-1
        status: non-primary
        protocol_version: 4
        capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
        final: no
        own_index: 0
        members(1):
      	0: 7c79d6ce-f75d-11e9-8a80-13083ff677cf, cnt7glr11.localdomain
      =================================================
      2019-10-25 22:28:03 2 [Note] WSREP: Non-primary view
      2019-10-25 22:28:03 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      2019-10-25 22:28:03 2 [Note] WSREP: ####### processing CC -1, local, ordered
      2019-10-25 22:28:03 2 [Note] WSREP: ####### My UUID: 7c79d6ce-f75d-11e9-8a80-13083ff677cf
      2019-10-25 22:28:03 2 [Note] WSREP: ####### ST not required
      2019-10-25 22:28:03 2 [Note] WSREP: ================================================
      View:
        id: 7c1cbeb7-f75d-11e9-a0f5-662e63655745:-1
        status: non-primary
        protocol_version: 4
        capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
        final: yes
        own_index: -1
        members(0):
      =================================================
      2019-10-25 22:28:03 2 [Note] WSREP: Non-primary view
      2019-10-25 22:28:03 2 [Note] WSREP: Server status change disconnecting -> disconnected
      2019-10-25 22:28:03 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      2019-10-25 22:28:03 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      2019-10-25 22:28:03 2 [Note] WSREP: Applier thread exiting ret: 0 thd: 2
      2019-10-25 22:28:03 140 [Note] WSREP: killing local connection: 134
      2019-10-25 22:28:03 2 [Warning] Aborted connection 2 to db: 'unconnected' user: 'unauthenticated' host: '' (This connection closed normally without authentication)
      mysqld: /home/stepan/mariadb/10.4/git/include/mysql/psi/mysql_socket.h:779: inline_mysql_socket_recv: Assertion `mysql_socket.fd != -1' failed.
      191025 22:28:03 [ERROR] mysqld got signal 6 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
       
      To report this bug, see https://mariadb.com/kb/en/reporting-bugs
       
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed, 
      something is definitely wrong and this may fail.
       
      Server version: 10.4.9-MariaDB-debug-log
      key_buffer_size=1048576
      read_buffer_size=131072
      max_used_connections=2
      max_threads=153
      thread_count=9
      It is possible that mysqld could use up to 
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 63618 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x7f5d1000a040
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x7f5d3086cd30 thread_stack 0x49000
      /home/stepan/mariadb/10.4/git/sql/mysqld(my_print_stacktrace+0x40)[0x559feed99008]
      mysys/stacktrace.c:269(my_print_stacktrace)[0x559fee5055dd]
      sigaction.c:0(__restore_rt)[0x7f5d4b4a55d0]
      :0(__GI_raise)[0x7f5d49793207]
      :0(__GI_abort)[0x7f5d497948f8]
      :0(__assert_fail_base)[0x7f5d4978c026]
      :0(__GI___assert_fail)[0x7f5d4978c0d2]
      /home/stepan/mariadb/10.4/git/sql/mysqld(+0x1464638)[0x559feee07638]
      psi/mysql_socket.h:781(inline_mysql_socket_recv)[0x559feee07d2e]
      sql/net_serv.cc:962(my_real_read(st_net*, unsigned long*, char))[0x559fee6861e3]
      sql/net_serv.cc:1243(my_net_read_packet_reallen)[0x559fee686ae1]
      sql/net_serv.cc:1228(my_net_read_packet)[0x559fee686a94]
      sql/sql_parse.cc:1232(do_command(THD*))[0x559fee19d9de]
      sql/sql_connect.cc:1412(do_handle_one_connection(CONNECT*))[0x559fee327d51]
      sql/sql_connect.cc:1317(handle_one_connection)[0x559fee327a80]
      perfschema/pfs.cc:1864(pfs_spawn_thread)[0x559feed274a5]
      pthread_create.c:0(start_thread)[0x7f5d4b49ddd5]
      /lib64/libc.so.6(clone+0x6d)[0x7f5d4985aead]
       
      Trying to get some variables.
      Some pointers may be invalid and cause the dump to abort.
      Query (0x0): 
      Connection ID (thread ID): 134
      Status: KILL_CONNECTION
       
      Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on
       
      The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
      information that should help you find out what is causing the crash.
      Writing a core file...
      Working directory at /home/stepan/mariadb/10.4/git/mysql-test/var/4/mysqld.2/data
      Resource Limits:
      Limit                     Soft Limit           Hard Limit           Units     
      Max cpu time              unlimited            unlimited            seconds   
      Max file size             unlimited            unlimited            bytes     
      Max data size             unlimited            unlimited            bytes     
      Max stack size            8388608              unlimited            bytes     
      Max core file size        unlimited            unlimited            bytes     
      Max resident set          unlimited            unlimited            bytes     
      Max processes             4096                 23005                processes 
      Max open files            1024                 1024                 files     
      Max locked memory         65536                65536                bytes     
      Max address space         unlimited            unlimited            bytes     
      Max file locks            unlimited            unlimited            locks     
      Max pending signals       23005                23005                signals   
      Max msgqueue size         819200               819200               bytes     
      Max nice priority         0                    0                    
      Max realtime priority     0                    0                    
      Max realtime timeout      unlimited            unlimited            us        
      Core pattern: |/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t e %P %I %h
       
      ----------SERVER LOG END-------------
       
       
       - found 'core.12605' (0/5)
       
      Trying 'dbx' to get a backtrace
       
      Trying 'gdb' to get a backtrace from coredump /home/stepan/mariadb/10.4/git/mysql-test/var/4/log/galera_sr.galera_sr_cc_master-innodb/mysqld.2/data/core.12605
      Core generated by '/home/stepan/mariadb/10.4/git/sql/mysqld'
      Output from gdb follows. The first stack trace is from the failing thread.
      The following stack traces are from all threads (so the failing one is
      duplicated).
      --------------------------
      [New LWP 13163]
      [New LWP 12614]
      [New LWP 13078]
      [New LWP 12648]
      [New LWP 13082]
      [New LWP 14442]
      [New LWP 13131]
      [New LWP 13080]
      [New LWP 12613]
      [New LWP 13137]
      [New LWP 13132]
      [New LWP 13067]
      [New LWP 13138]
      [New LWP 13133]
      [New LWP 13130]
      [New LWP 13139]
      [New LWP 13135]
      [New LWP 13134]
      [New LWP 13155]
      [New LWP 13152]
      [New LWP 13136]
      [New LWP 13153]
      [New LWP 13151]
      [New LWP 13154]
      [New LWP 12605]
      [New LWP 13162]
      [New LWP 13081]
      [New LWP 13079]
      [New LWP 13077]
      [New LWP 13076]
      [Thread debugging using libthread_db enabled]
      Using host libthread_db library "/lib64/libthread_db.so.1".
      Core was generated by `/home/stepan/mariadb/10.4/git/sql/mysqld --defaults-group-suffix=.2 --defaults-'.
      Program terminated with signal 6, Aborted.
      #0  0x00007f5d4b4a29d1 in pthread_kill () from /lib64/libpthread.so.0
      #0  0x00007f5d4b4a29d1 in pthread_kill () from /lib64/libpthread.so.0
      #1  0x0000559feed990f6 in my_write_core (sig=6) at /home/stepan/mariadb/10.4/git/mysys/stacktrace.c:481
      #2  0x0000559fee505964 in handle_fatal_signal (sig=6) at /home/stepan/mariadb/10.4/git/sql/signal_handler.cc:343
      #3  <signal handler called>
      #4  0x00007f5d49793207 in raise () from /lib64/libc.so.6
      #5  0x00007f5d497948f8 in abort () from /lib64/libc.so.6
      #6  0x00007f5d4978c026 in __assert_fail_base () from /lib64/libc.so.6
      #7  0x00007f5d4978c0d2 in __assert_fail () from /lib64/libc.so.6
      #8  0x0000559feee07638 in inline_mysql_socket_recv (src_file=0x559fef391f70 "/home/stepan/mariadb/10.4/git/vio/viosocket.c", src_line=190, mysql_socket=..., buf=0x7f5d10028ba0, n=4, flags=64) at /home/stepan/mariadb/10.4/git/include/mysql/psi/mysql_socket.h:779
      #9  0x0000559feee07d2e in vio_read (vio=0x559ff0c0e6a0, buf=0x7f5d10028ba0 "\001", size=4) at /home/stepan/mariadb/10.4/git/vio/viosocket.c:189
      #10 0x0000559fee6861e3 in my_real_read (net=0x7f5d1000a338, complen=0x7f5d3086c810, header=1 '\001') at /home/stepan/mariadb/10.4/git/sql/net_serv.cc:962
      #11 0x0000559fee686ae1 in my_net_read_packet_reallen (net=0x7f5d1000a338, read_from_server=1 '\001', reallen=0x7f5d3086c878) at /home/stepan/mariadb/10.4/git/sql/net_serv.cc:1243
      #12 0x0000559fee686a94 in my_net_read_packet (net=0x7f5d1000a338, read_from_server=1 '\001') at /home/stepan/mariadb/10.4/git/sql/net_serv.cc:1227
      #13 0x0000559fee19d9de in do_command (thd=0x7f5d1000a040) at /home/stepan/mariadb/10.4/git/sql/sql_parse.cc:1232
      #14 0x0000559fee327d51 in do_handle_one_connection (connect=0x559ff0cff900) at /home/stepan/mariadb/10.4/git/sql/sql_connect.cc:1412
      #15 0x0000559fee327a80 in handle_one_connection (arg=0x559ff0cff900) at /home/stepan/mariadb/10.4/git/sql/sql_connect.cc:1316
      #16 0x0000559feed274a5 in pfs_spawn_thread (arg=0x559ff0d01c90) at /home/stepan/mariadb/10.4/git/storage/perfschema/pfs.cc:1862
      #17 0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
      #18 0x00007f5d4985aead in clone () from /lib64/libc.so.6
       
      Thread 30 (Thread 0x7f5d17fff700 (LWP 13076)):
      #0  0x00007f5d4b06e644 in __io_getevents_0_4 () from /lib64/libaio.so.1
      #1  0x0000559fee845dad in LinuxAIOHandler::collect (this=0x7f5d17ffeb00) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0file.cc:1809
      #2  0x0000559fee8463f6 in LinuxAIOHandler::poll (this=0x7f5d17ffeb00, m1=0x7f5d17ffeba8, m2=0x7f5d17ffebb0, request=0x7f5d17ffebe0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0file.cc:1968
      #3  0x0000559fee846559 in os_aio_linux_handler (global_segment=0, m1=0x7f5d17ffeba8, m2=0x7f5d17ffebb0, request=0x7f5d17ffebe0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0file.cc:2022
      #4  0x0000559fee84b2c0 in os_aio_handler (segment=0, m1=0x7f5d17ffeba8, m2=0x7f5d17ffebb0, request=0x7f5d17ffebe0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0file.cc:5561
      #5  0x0000559feeb014f9 in fil_aio_wait (segment=0) at /home/stepan/mariadb/10.4/git/storage/innobase/fil/fil0fil.cc:4357
      #6  0x0000559fee972e42 in io_handler_thread (arg=0x559fefbaa980 <n>) at /home/stepan/mariadb/10.4/git/storage/innobase/srv/srv0start.cc:324
      #7  0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
      #8  0x00007f5d4985aead in clone () from /lib64/libc.so.6
       
      Thread 29 (Thread 0x7f5d177fe700 (LWP 13077)):
      #0  0x00007f5d4b06e644 in __io_getevents_0_4 () from /lib64/libaio.so.1
      #1  0x0000559fee845dad in LinuxAIOHandler::collect (this=0x7f5d177fdb00) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0file.cc:1809
      #2  0x0000559fee8463f6 in LinuxAIOHandler::poll (this=0x7f5d177fdb00, m1=0x7f5d177fdba8, m2=0x7f5d177fdbb0, request=0x7f5d177fdbe0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0file.cc:1968
      #3  0x0000559fee846559 in os_aio_linux_handler (global_segment=1, m1=0x7f5d177fdba8, m2=0x7f5d177fdbb0, request=0x7f5d177fdbe0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0file.cc:2022
      #4  0x0000559fee84b2c0 in os_aio_handler (segment=1, m1=0x7f5d177fdba8, m2=0x7f5d177fdbb0, request=0x7f5d177fdbe0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0file.cc:5561
      #5  0x0000559feeb014f9 in fil_aio_wait (segment=1) at /home/stepan/mariadb/10.4/git/storage/innobase/fil/fil0fil.cc:4357
      #6  0x0000559fee972e42 in io_handler_thread (arg=0x559fefbaa988 <n+8>) at /home/stepan/mariadb/10.4/git/storage/innobase/srv/srv0start.cc:324
      #7  0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
      #8  0x00007f5d4985aead in clone () from /lib64/libc.so.6
       
      Thread 28 (Thread 0x7f5d167fc700 (LWP 13079)):
      #0  0x00007f5d4b06e644 in __io_getevents_0_4 () from /lib64/libaio.so.1
      #1  0x0000559fee845dad in LinuxAIOHandler::collect (this=0x7f5d167fbb00) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0file.cc:1809
      #2  0x0000559fee8463f6 in LinuxAIOHandler::poll (this=0x7f5d167fbb00, m1=0x7f5d167fbba8, m2=0x7f5d167fbbb0, request=0x7f5d167fbbe0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0file.cc:1968
      #3  0x0000559fee846559 in os_aio_linux_handler (global_segment=3, m1=0x7f5d167fbba8, m2=0x7f5d167fbbb0, request=0x7f5d167fbbe0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0file.cc:2022
      #4  0x0000559fee84b2c0 in os_aio_handler (segment=3, m1=0x7f5d167fbba8, m2=0x7f5d167fbbb0, request=0x7f5d167fbbe0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0file.cc:5561
      #5  0x0000559feeb014f9 in fil_aio_wait (segment=3) at /home/stepan/mariadb/10.4/git/storage/innobase/fil/fil0fil.cc:4357
      #6  0x0000559fee972e42 in io_handler_thread (arg=0x559fefbaa998 <n+24>) at /home/stepan/mariadb/10.4/git/storage/innobase/srv/srv0start.cc:324
      #7  0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
      #8  0x00007f5d4985aead in clone () from /lib64/libc.so.6
       
      Thread 27 (Thread 0x7f5d157fa700 (LWP 13081)):
      #0  0x00007f5d4b06e644 in __io_getevents_0_4 () from /lib64/libaio.so.1
      #1  0x0000559fee845dad in LinuxAIOHandler::collect (this=0x7f5d157f9b00) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0file.cc:1809
      #2  0x0000559fee8463f6 in LinuxAIOHandler::poll (this=0x7f5d157f9b00, m1=0x7f5d157f9ba8, m2=0x7f5d157f9bb0, request=0x7f5d157f9be0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0file.cc:1968
      #3  0x0000559fee846559 in os_aio_linux_handler (global_segment=5, m1=0x7f5d157f9ba8, m2=0x7f5d157f9bb0, request=0x7f5d157f9be0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0file.cc:2022
      #4  0x0000559fee84b2c0 in os_aio_handler (segment=5, m1=0x7f5d157f9ba8, m2=0x7f5d157f9bb0, request=0x7f5d157f9be0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0file.cc:5561
      #5  0x0000559feeb014f9 in fil_aio_wait (segment=5) at /home/stepan/mariadb/10.4/git/storage/innobase/fil/fil0fil.cc:4357
      #6  0x0000559fee972e42 in io_handler_thread (arg=0x559fefbaa9a8 <n+40>) at /home/stepan/mariadb/10.4/git/storage/innobase/srv/srv0start.cc:324
      #7  0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
      #8  0x00007f5d4985aead in clone () from /lib64/libc.so.6
       
      Thread 26 (Thread 0x7f5d40063700 (LWP 13162)):
      #0  0x00007f5d4b4a1965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x0000559feed9e769 in safe_cond_wait (cond=0x559fefb2c0a0 <COND_slave_background>, mp=0x559fefb2f320 <LOCK_slave_background>, file=0x559feee6d040 "/home/stepan/mariadb/10.4/git/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/mariadb/10.4/git/mysys/thr_mutex.c:492
      #2  0x0000559fee09008e in inline_mysql_cond_wait (that=0x559fefb2c0a0 <COND_slave_background>, mutex=0x559fefb2f320 <LOCK_slave_background>, src_file=0x559feee6db80 "/home/stepan/mariadb/10.4/git/sql/slave.cc", src_line=519) at /home/stepan/mariadb/10.4/git/include/mysql/psi/mysql_thread.h:1167
      #3  0x0000559fee0919cb in handle_slave_background (arg=0x0) at /home/stepan/mariadb/10.4/git/sql/slave.cc:519
      #4  0x0000559feed274a5 in pfs_spawn_thread (arg=0x559ff0c76690) at /home/stepan/mariadb/10.4/git/storage/perfschema/pfs.cc:1862
      #5  0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
      #6  0x00007f5d4985aead in clone () from /lib64/libc.so.6
       
      Thread 25 (Thread 0x7f5d4b8ae880 (LWP 12605)):
      #0  0x00007f5d4985020d in poll () from /lib64/libc.so.6
      #1  0x0000559fee06a2db in handle_connections_sockets () at /home/stepan/mariadb/10.4/git/sql/mysqld.cc:6455
      #2  0x0000559fee069576 in mysqld_main (argc=163, argv=0x559ff0642130) at /home/stepan/mariadb/10.4/git/sql/mysqld.cc:5891
      #3  0x0000559fee05d095 in main (argc=23, argv=0x7ffd7ed249d8) at /home/stepan/mariadb/10.4/git/sql/main.cc:25
       
      Thread 24 (Thread 0x7f5cf4ff9700 (LWP 13154)):
      #0  0x00007f5d4b4a1965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x0000559feed9e769 in safe_cond_wait (cond=0x7f5cf4ff8c70, mp=0x7f5cf4ff8cb0, file=0x559fef08adc0 "/home/stepan/mariadb/10.4/git/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/mariadb/10.4/git/mysys/thr_mutex.c:492
      #2  0x0000559fee74715d in inline_mysql_cond_wait (that=0x7f5cf4ff8c70, mutex=0x7f5cf4ff8cb0, src_file=0x559fef08c888 "/home/stepan/mariadb/10.4/git/storage/innobase/handler/ha_innodb.cc", src_line=288) at /home/stepan/mariadb/10.4/git/include/mysql/psi/mysql_thread.h:1178
      #3  0x0000559fee74ca55 in thd_destructor_proxy () at /home/stepan/mariadb/10.4/git/storage/innobase/handler/ha_innodb.cc:288
      #4  0x0000559feed274a5 in pfs_spawn_thread (arg=0x559ff0bb39e0) at /home/stepan/mariadb/10.4/git/storage/perfschema/pfs.cc:1862
      #5  0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
      #6  0x00007f5d4985aead in clone () from /lib64/libc.so.6
       
      Thread 23 (Thread 0x7f5cf67fc700 (LWP 13151)):
      #0  0x00007f5d4b4a1965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x0000559fee854377 in os_event::wait (this=0x559ff09094f0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:158
      #2  0x0000559fee853e6b in os_event::wait_low (this=0x559ff09094f0, reset_sig_count=1) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:325
      #3  0x0000559fee85416e in os_event_wait_low (event=0x559ff09094f0, reset_sig_count=0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:502
      #4  0x0000559feea71e86 in buf_dump_thread () at /home/stepan/mariadb/10.4/git/storage/innobase/buf/buf0dump.cc:833
      #5  0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
      #6  0x00007f5d4985aead in clone () from /lib64/libc.so.6
       
      Thread 22 (Thread 0x7f5cf57fa700 (LWP 13153)):
      #0  0x00007f5d4b4a1965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x0000559fee854377 in os_event::wait (this=0x559ff09095f0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:158
      #2  0x0000559fee853e6b in os_event::wait_low (this=0x559ff09095f0, reset_sig_count=1) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:325
      #3  0x0000559fee85416e in os_event_wait_low (event=0x559ff09095f0, reset_sig_count=0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:502
      #4  0x0000559feea577d9 in buf_resize_thread () at /home/stepan/mariadb/10.4/git/storage/innobase/buf/buf0buf.cc:3199
      #5  0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
      #6  0x00007f5d4985aead in clone () from /lib64/libc.so.6
       
      Thread 21 (Thread 0x7f5d08ff9700 (LWP 13136)):
      #0  0x00007f5d4b4a1965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x0000559fee854377 in os_event::wait (this=0x559ff09091f0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:158
      #2  0x0000559fee853e6b in os_event::wait_low (this=0x559ff09091f0, reset_sig_count=257) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:325
      #3  0x0000559fee85416e in os_event_wait_low (event=0x559ff09091f0, reset_sig_count=257) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:502
      #4  0x0000559fee96c328 in srv_resume_thread (slot=0x559fef966c10 <srv_sys+592>, sig_count=257, wait=true, timeout_usec=0) at /home/stepan/mariadb/10.4/git/storage/innobase/srv/srv0srv.cc:909
      #5  0x0000559fee970aa0 in srv_purge_coordinator_suspend (slot=0x559fef966c10 <srv_sys+592>, rseg_history_len=16) at /home/stepan/mariadb/10.4/git/storage/innobase/srv/srv0srv.cc:2657
      #6  0x0000559fee970dd1 in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/mariadb/10.4/git/storage/innobase/srv/srv0srv.cc:2738
      #7  0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
      #8  0x00007f5d4985aead in clone () from /lib64/libc.so.6
       
      Thread 20 (Thread 0x7f5cf5ffb700 (LWP 13152)):
      #0  0x00007f5d4b4a4e3d in nanosleep () from /lib64/libpthread.so.0
      #1  0x0000559fee8546b4 in os_thread_sleep (tm=1000000) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0thread.cc:225
      #2  0x0000559feea46dd3 in btr_defragment_thread () at /home/stepan/mariadb/10.4/git/storage/innobase/btr/btr0defragment.cc:705
      #3  0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
      #4  0x00007f5d4985aead in clone () from /lib64/libc.so.6
       
      Thread 19 (Thread 0x7f5d400ae700 (LWP 13155)):
      #0  0x00007f5d4979440a in sigwaitinfo () from /lib64/libc.so.6
      #1  0x0000559fee05d114 in my_sigwait (set=0x7f5d400adcf0, sig=0x7f5d400adcb4, code=0x7f5d400adcb8) at /home/stepan/mariadb/10.4/git/include/my_pthread.h:198
      #2  0x0000559fee063a8d in signal_hand (arg=0x0) at /home/stepan/mariadb/10.4/git/sql/mysqld.cc:3243
      #3  0x0000559feed274a5 in pfs_spawn_thread (arg=0x559ff08c3710) at /home/stepan/mariadb/10.4/git/storage/perfschema/pfs.cc:1862
      #4  0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
      #5  0x00007f5d4985aead in clone () from /lib64/libc.so.6
       
      Thread 18 (Thread 0x7f5d097fa700 (LWP 13134)):
      #0  0x00007f5d4b4a1d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x0000559fee853d00 in os_event::timed_wait (this=0x559ff0aadee0, abstime=0x7f5d097f9b70) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:275
      #2  0x0000559fee853fa9 in os_event::wait_time_low (this=0x559ff0aadee0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:385
      #3  0x0000559fee854149 in os_event_wait_time_low (event=0x559ff0aadee0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:485
      #4  0x0000559fee9dc563 in ib_wqueue_timedwait (wq=0x559ff0aaddb0, wait_in_usecs=5000000) at /home/stepan/mariadb/10.4/git/storage/innobase/ut/ut0wqueue.cc:163
      #5  0x0000559feeb4f23d in fts_optimize_thread (arg=0x559ff0aaddb0) at /home/stepan/mariadb/10.4/git/storage/innobase/fts/fts0opt.cc:2865
      #6  0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
      #7  0x00007f5d4985aead in clone () from /lib64/libc.so.6
       
      Thread 17 (Thread 0x7f5d0bfff700 (LWP 13135)):
      #0  0x00007f5d4b4a4e3d in nanosleep () from /lib64/libpthread.so.0
      #1  0x0000559fee8546b4 in os_thread_sleep (tm=1000000) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0thread.cc:225
      #2  0x0000559fee96fd65 in srv_master_sleep () at /home/stepan/mariadb/10.4/git/storage/innobase/srv/srv0srv.cc:2334
      #3  0x0000559fee96fecd in srv_master_thread (arg=0x0) at /home/stepan/mariadb/10.4/git/storage/innobase/srv/srv0srv.cc:2375
      #4  0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
      #5  0x00007f5d4985aead in clone () from /lib64/libc.so.6
       
      Thread 16 (Thread 0x7f5cf6ffd700 (LWP 13139)):
      #0  0x00007f5d4b4a1965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x0000559fee854377 in os_event::wait (this=0x559ff0909370) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:158
      #2  0x0000559fee853e6b in os_event::wait_low (this=0x559ff0909370, reset_sig_count=902) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:325
      #3  0x0000559fee85416e in os_event_wait_low (event=0x559ff0909370, reset_sig_count=0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:502
      #4  0x0000559fee96c328 in srv_resume_thread (slot=0x559fef966f10 <srv_sys+1360>, sig_count=0, wait=true, timeout_usec=0) at /home/stepan/mariadb/10.4/git/storage/innobase/srv/srv0srv.cc:909
      #5  0x0000559fee9705e9 in srv_worker_thread (arg=0x0) at /home/stepan/mariadb/10.4/git/storage/innobase/srv/srv0srv.cc:2517
      #6  0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
      #7  0x00007f5d4985aead in clone () from /lib64/libc.so.6
       
      Thread 15 (Thread 0x7f5d0b7fe700 (LWP 13130)):
      #0  0x00007f5d4b4a1d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x0000559fee853d00 in os_event::timed_wait (this=0x559ff0b446e0, abstime=0x7f5d0b7fdd50) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:275
      #2  0x0000559fee853fa9 in os_event::wait_time_low (this=0x559ff0b446e0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:385
      #3  0x0000559fee854149 in os_event_wait_time_low (event=0x559ff0b446e0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:485
      #4  0x0000559fee816527 in lock_wait_timeout_thread () at /home/stepan/mariadb/10.4/git/storage/innobase/lock/lock0wait.cc:522
      #5  0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
      #6  0x00007f5d4985aead in clone () from /lib64/libc.so.6
       
      Thread 14 (Thread 0x7f5d09ffb700 (LWP 13133)):
      #0  0x00007f5d4b4a1d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x0000559fee853d00 in os_event::timed_wait (this=0x559ff0b76580, abstime=0x7f5d09ffad70) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:275
      #2  0x0000559fee853fa9 in os_event::wait_time_low (this=0x559ff0b76580, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:385
      #3  0x0000559fee854149 in os_event_wait_time_low (event=0x559ff0b76580, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:485
      #4  0x0000559feeae86ca in dict_stats_thread () at /home/stepan/mariadb/10.4/git/storage/innobase/dict/dict0stats_bg.cc:476
      #5  0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
      #6  0x00007f5d4985aead in clone () from /lib64/libc.so.6
       
      Thread 13 (Thread 0x7f5cf77fe700 (LWP 13138)):
      #0  0x00007f5d4b4a1965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x0000559fee854377 in os_event::wait (this=0x559ff0909270) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:158
      #2  0x0000559fee853e6b in os_event::wait_low (this=0x559ff0909270, reset_sig_count=898) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:325
      #3  0x0000559fee85416e in os_event_wait_low (event=0x559ff0909270, reset_sig_count=0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:502
      #4  0x0000559fee96c328 in srv_resume_thread (slot=0x559fef966d10 <srv_sys+848>, sig_count=0, wait=true, timeout_usec=0) at /home/stepan/mariadb/10.4/git/storage/innobase/srv/srv0srv.cc:909
      #5  0x0000559fee9705e9 in srv_worker_thread (arg=0x0) at /home/stepan/mariadb/10.4/git/storage/innobase/srv/srv0srv.cc:2517
      #6  0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
      #7  0x00007f5d4985aead in clone () from /lib64/libc.so.6
       
      Thread 12 (Thread 0x7f5d31e6f700 (LWP 13067)):
      #0  0x00007f5d4b4a1d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x0000559feed9ea6b in safe_cond_timedwait (cond=0x559ff03bb5e0 <COND_checkpoint>, mp=0x559ff03bb520 <LOCK_checkpoint>, abstime=0x7f5d31e6ecb0, file=0x559fef206850 "/home/stepan/mariadb/10.4/git/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/mariadb/10.4/git/mysys/thr_mutex.c:546
      #2  0x0000559feebf6109 in inline_mysql_cond_timedwait (that=0x559ff03bb5e0 <COND_checkpoint>, mutex=0x559ff03bb520 <LOCK_checkpoint>, abstime=0x7f5d31e6ecb0, src_file=0x559fef206890 "/home/stepan/mariadb/10.4/git/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/mariadb/10.4/git/include/mysql/psi/mysql_thread.h:1204
      #3  0x0000559feebf664d in my_service_thread_sleep (control=0x559fef96d780 <checkpoint_control>, sleep_time=30000000000) at /home/stepan/mariadb/10.4/git/storage/maria/ma_servicethread.c:115
      #4  0x0000559feebea50a in ma_checkpoint_background (arg=0x1e) at /home/stepan/mariadb/10.4/git/storage/maria/ma_checkpoint.c:707
      #5  0x0000559feed274a5 in pfs_spawn_thread (arg=0x559ff08c1030) at /home/stepan/mariadb/10.4/git/storage/perfschema/pfs.cc:1862
      #6  0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
      #7  0x00007f5d4985aead in clone () from /lib64/libc.so.6
       
      Thread 11 (Thread 0x7f5d0a7fc700 (LWP 13132)):
      #0  0x00007f5d4b4a1d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x0000559fee853d00 in os_event::timed_wait (this=0x559ff0909470, abstime=0x7f5d0a7fbd30) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:275
      #2  0x0000559fee853fa9 in os_event::wait_time_low (this=0x559ff0909470, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:385
      #3  0x0000559fee854149 in os_event_wait_time_low (event=0x559ff0909470, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:485
      #4  0x0000559fee96e20f in srv_monitor_thread () at /home/stepan/mariadb/10.4/git/storage/innobase/srv/srv0srv.cc:1687
      #5  0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
      #6  0x00007f5d4985aead in clone () from /lib64/libc.so.6
       
      Thread 10 (Thread 0x7f5cf7fff700 (LWP 13137)):
      #0  0x00007f5d4b4a1965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x0000559fee854377 in os_event::wait (this=0x559ff09092f0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:158
      #2  0x0000559fee853e6b in os_event::wait_low (this=0x559ff09092f0, reset_sig_count=893) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:325
      #3  0x0000559fee85416e in os_event_wait_low (event=0x559ff09092f0, reset_sig_count=0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:502
      #4  0x0000559fee96c328 in srv_resume_thread (slot=0x559fef966e10 <srv_sys+1104>, sig_count=0, wait=true, timeout_usec=0) at /home/stepan/mariadb/10.4/git/storage/innobase/srv/srv0srv.cc:909
      #5  0x0000559fee9705e9 in srv_worker_thread (arg=0x0) at /home/stepan/mariadb/10.4/git/storage/innobase/srv/srv0srv.cc:2517
      #6  0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
      #7  0x00007f5d4985aead in clone () from /lib64/libc.so.6
       
      Thread 9 (Thread 0x7f5d43f36700 (LWP 12613)):
      #0  0x00007f5d4b4a1d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x0000559feed9ea6b in safe_cond_timedwait (cond=0x559ff03cdce0 <COND_timer>, mp=0x559ff03cdc20 <LOCK_timer>, abstime=0x7f5d43f35d70, file=0x559fef24ed80 "/home/stepan/mariadb/10.4/git/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/mariadb/10.4/git/mysys/thr_mutex.c:546
      #2  0x0000559feed9faae in inline_mysql_cond_timedwait (that=0x559ff03cdce0 <COND_timer>, mutex=0x559ff03cdc20 <LOCK_timer>, abstime=0x7f5d43f35d70, src_file=0x559fef24edc0 "/home/stepan/mariadb/10.4/git/mysys/thr_timer.c", src_line=292) at /home/stepan/mariadb/10.4/git/include/mysql/psi/mysql_thread.h:1204
      #3  0x0000559feeda0671 in timer_handler (arg=0x0) at /home/stepan/mariadb/10.4/git/mysys/thr_timer.c:292
      #4  0x0000559feed274a5 in pfs_spawn_thread (arg=0x559ff0788250) at /home/stepan/mariadb/10.4/git/storage/perfschema/pfs.cc:1862
      #5  0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
      #6  0x00007f5d4985aead in clone () from /lib64/libc.so.6
       
      Thread 8 (Thread 0x7f5d15ffb700 (LWP 13080)):
      #0  0x00007f5d4b06e644 in __io_getevents_0_4 () from /lib64/libaio.so.1
      #1  0x0000559fee845dad in LinuxAIOHandler::collect (this=0x7f5d15ffab00) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0file.cc:1809
      #2  0x0000559fee8463f6 in LinuxAIOHandler::poll (this=0x7f5d15ffab00, m1=0x7f5d15ffaba8, m2=0x7f5d15ffabb0, request=0x7f5d15ffabe0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0file.cc:1968
      #3  0x0000559fee846559 in os_aio_linux_handler (global_segment=4, m1=0x7f5d15ffaba8, m2=0x7f5d15ffabb0, request=0x7f5d15ffabe0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0file.cc:2022
      #4  0x0000559fee84b2c0 in os_aio_handler (segment=4, m1=0x7f5d15ffaba8, m2=0x7f5d15ffabb0, request=0x7f5d15ffabe0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0file.cc:5561
      #5  0x0000559feeb014f9 in fil_aio_wait (segment=4) at /home/stepan/mariadb/10.4/git/storage/innobase/fil/fil0fil.cc:4357
      #6  0x0000559fee972e42 in io_handler_thread (arg=0x559fefbaa9a0 <n+32>) at /home/stepan/mariadb/10.4/git/storage/innobase/srv/srv0start.cc:324
      #7  0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
      #8  0x00007f5d4985aead in clone () from /lib64/libc.so.6
       
      Thread 7 (Thread 0x7f5d0affd700 (LWP 13131)):
      #0  0x00007f5d4b4a1d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x0000559fee853d00 in os_event::timed_wait (this=0x559ff09093f0, abstime=0x7f5d0affcbb0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:275
      #2  0x0000559fee853fa9 in os_event::wait_time_low (this=0x559ff09093f0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:385
      #3  0x0000559fee854149 in os_event_wait_time_low (event=0x559ff09093f0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:485
      #4  0x0000559fee96e789 in srv_error_monitor_thread () at /home/stepan/mariadb/10.4/git/storage/innobase/srv/srv0srv.cc:1851
      #5  0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
      #6  0x00007f5d4985aead in clone () from /lib64/libc.so.6
       
      Thread 6 (Thread 0x7f5d30822700 (LWP 14442)):
      #0  0x00007f5d49821e2d in nanosleep () from /lib64/libc.so.6
      #1  0x00007f5d49821cc4 in sleep () from /lib64/libc.so.6
      #2  0x0000559fee415642 in wsrep_close_client_connections (wait_to_end=1 '\001', except_caller_thd=0x7f5cbc000b00) at /home/stepan/mariadb/10.4/git/sql/wsrep_mysqld.cc:2344
      #3  0x0000559fee40effc in wsrep_stop_replication (thd=0x7f5cbc000b00) at /home/stepan/mariadb/10.4/git/sql/wsrep_mysqld.cc:943
      #4  0x0000559fee4218ae in wsrep_cluster_address_update (self=0x559fefb548a0 <Sys_wsrep_cluster_address>, thd=0x7f5cbc000b00, type=SHOW_OPT_GLOBAL) at /home/stepan/mariadb/10.4/git/sql/wsrep_var.cc:483
      #5  0x0000559fee08a958 in sys_var::update (this=0x559fefb548a0 <Sys_wsrep_cluster_address>, thd=0x7f5cbc000b00, var=0x7f5cbc011f60) at /home/stepan/mariadb/10.4/git/sql/set_var.cc:209
      #6  0x0000559fee08c6f4 in set_var::update (this=0x7f5cbc011f60, thd=0x7f5cbc000b00) at /home/stepan/mariadb/10.4/git/sql/set_var.cc:837
      #7  0x0000559fee08c36e in sql_set_variables (thd=0x7f5cbc000b00, var_list=0x7f5cbc005930, free=true) at /home/stepan/mariadb/10.4/git/sql/set_var.cc:740
      #8  0x0000559fee1aa179 in mysql_execute_command (thd=0x7f5cbc000b00) at /home/stepan/mariadb/10.4/git/sql/sql_parse.cc:4944
      #9  0x0000559fee1b47b5 in mysql_parse (thd=0x7f5cbc000b00, rawbuf=0x7f5cbc011df8 "SET GLOBAL wsrep_cluster_address = ''", length=37, parser_state=0x7f5d30821030, is_com_multi=false, is_next_command=false) at /home/stepan/mariadb/10.4/git/sql/sql_parse.cc:7912
      #10 0x0000559fee1b3d68 in wsrep_mysql_parse (thd=0x7f5cbc000b00, rawbuf=0x7f5cbc011df8 "SET GLOBAL wsrep_cluster_address = ''", length=37, parser_state=0x7f5d30821030, is_com_multi=false, is_next_command=false) at /home/stepan/mariadb/10.4/git/sql/sql_parse.cc:7716
      #11 0x0000559fee19fa77 in dispatch_command (command=COM_QUERY, thd=0x7f5cbc000b00, packet=0x7f5cbc008331 "SET GLOBAL wsrep_cluster_address = ''", packet_length=37, is_com_multi=false, is_next_command=false) at /home/stepan/mariadb/10.4/git/sql/sql_parse.cc:1826
      #12 0x0000559fee19e18a in do_command (thd=0x7f5cbc000b00) at /home/stepan/mariadb/10.4/git/sql/sql_parse.cc:1359
      #13 0x0000559fee327d51 in do_handle_one_connection (connect=0x559ff0cff900) at /home/stepan/mariadb/10.4/git/sql/sql_connect.cc:1412
      #14 0x0000559fee327a80 in handle_one_connection (arg=0x559ff0cff900) at /home/stepan/mariadb/10.4/git/sql/sql_connect.cc:1316
      #15 0x0000559feed274a5 in pfs_spawn_thread (arg=0x559ff0745ee0) at /home/stepan/mariadb/10.4/git/storage/perfschema/pfs.cc:1862
      #16 0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
      #17 0x00007f5d4985aead in clone () from /lib64/libc.so.6
       
      Thread 5 (Thread 0x7f5d14ff9700 (LWP 13082)):
      #0  0x00007f5d4b4a1d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x0000559fee853d00 in os_event::timed_wait (this=0x559ff0909570, abstime=0x7f5d14ff8b00) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:275
      #2  0x0000559fee853fa9 in os_event::wait_time_low (this=0x559ff0909570, time_in_usec=213000, reset_sig_count=2) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:385
      #3  0x0000559fee854149 in os_event_wait_time_low (event=0x559ff0909570, time_in_usec=213000, reset_sig_count=2) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:485
      #4  0x0000559feea7e358 in pc_sleep_if_needed (next_loop_time=188004048, sig_count=2, cur_time=188003835) at /home/stepan/mariadb/10.4/git/storage/innobase/buf/buf0flu.cc:2643
      #5  0x0000559feea7f4fc in buf_flush_page_cleaner_coordinator () at /home/stepan/mariadb/10.4/git/storage/innobase/buf/buf0flu.cc:3102
      #6  0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
      #7  0x00007f5d4985aead in clone () from /lib64/libc.so.6
       
      Thread 4 (Thread 0x7f5d40321700 (LWP 12648)):
      #0  0x00007f5d4b4a1965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x0000559feed9e769 in safe_cond_wait (cond=0x7f5d24015a68, mp=0x7f5d240159b8, file=0x559feef1aa20 "/home/stepan/mariadb/10.4/git/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/mariadb/10.4/git/mysys/thr_mutex.c:492
      #2  0x0000559fee4261b4 in inline_mysql_cond_wait (that=0x7f5d24015a68, mutex=0x7f5d240159b8, src_file=0x559feef1ac30 "/home/stepan/mariadb/10.4/git/sql/wsrep_thd.h", src_line=69) at /home/stepan/mariadb/10.4/git/include/mysql/psi/mysql_thread.h:1178
      #3  0x0000559fee429416 in Wsrep_thd_queue::pop_front (this=0x7f5d24015960) at /home/stepan/mariadb/10.4/git/sql/wsrep_thd.h:69
      #4  0x0000559fee427e23 in wsrep_rollback_process (rollbacker=0x7f5d24001d50, arg=0x559ff0819dc0) at /home/stepan/mariadb/10.4/git/sql/wsrep_thd.cc:266
      #5  0x0000559fee416e70 in start_wsrep_THD (arg=0x559ff0819dc0) at /home/stepan/mariadb/10.4/git/sql/wsrep_mysqld.cc:2708
      #6  0x0000559feed274a5 in pfs_spawn_thread (arg=0x559ff07dc8f0) at /home/stepan/mariadb/10.4/git/storage/perfschema/pfs.cc:1862
      #7  0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
      #8  0x00007f5d4985aead in clone () from /lib64/libc.so.6
       
      Thread 3 (Thread 0x7f5d16ffd700 (LWP 13078)):
      #0  0x00007f5d4b06e644 in __io_getevents_0_4 () from /lib64/libaio.so.1
      #1  0x0000559fee845dad in LinuxAIOHandler::collect (this=0x7f5d16ffcb00) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0file.cc:1809
      #2  0x0000559fee8463f6 in LinuxAIOHandler::poll (this=0x7f5d16ffcb00, m1=0x7f5d16ffcba8, m2=0x7f5d16ffcbb0, request=0x7f5d16ffcbe0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0file.cc:1968
      #3  0x0000559fee846559 in os_aio_linux_handler (global_segment=2, m1=0x7f5d16ffcba8, m2=0x7f5d16ffcbb0, request=0x7f5d16ffcbe0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0file.cc:2022
      #4  0x0000559fee84b2c0 in os_aio_handler (segment=2, m1=0x7f5d16ffcba8, m2=0x7f5d16ffcbb0, request=0x7f5d16ffcbe0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0file.cc:5561
      #5  0x0000559feeb014f9 in fil_aio_wait (segment=2) at /home/stepan/mariadb/10.4/git/storage/innobase/fil/fil0fil.cc:4357
      #6  0x0000559fee972e42 in io_handler_thread (arg=0x559fefbaa990 <n+16>) at /home/stepan/mariadb/10.4/git/storage/innobase/srv/srv0start.cc:324
      #7  0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
      #8  0x00007f5d4985aead in clone () from /lib64/libc.so.6
       
      Thread 2 (Thread 0x7f5d42fa6700 (LWP 12614)):
      #0  0x00007f5d4b4a1965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00007f5d433fcdd7 in gu::Lock::wait (cond=..., this=0x7f5d42fa5bf0) at galerautils/src/gu_lock.hpp:45
      #2  0x00007f5d433fce69 in galera::ServiceThd::thd_func (arg=0x559ff07bc698) at galera/src/galera_service_thd.cpp:30
      #3  0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
      #4  0x00007f5d4985aead in clone () from /lib64/libc.so.6
       
      Thread 1 (Thread 0x7f5d3086d700 (LWP 13163)):
      #0  0x00007f5d4b4a29d1 in pthread_kill () from /lib64/libpthread.so.0
      #1  0x0000559feed990f6 in my_write_core (sig=6) at /home/stepan/mariadb/10.4/git/mysys/stacktrace.c:481
      #2  0x0000559fee505964 in handle_fatal_signal (sig=6) at /home/stepan/mariadb/10.4/git/sql/signal_handler.cc:343
      #3  <signal handler called>
      #4  0x00007f5d49793207 in raise () from /lib64/libc.so.6
      #5  0x00007f5d497948f8 in abort () from /lib64/libc.so.6
      #6  0x00007f5d4978c026 in __assert_fail_base () from /lib64/libc.so.6
      #7  0x00007f5d4978c0d2 in __assert_fail () from /lib64/libc.so.6
      #8  0x0000559feee07638 in inline_mysql_socket_recv (src_file=0x559fef391f70 "/home/stepan/mariadb/10.4/git/vio/viosocket.c", src_line=190, mysql_socket=..., buf=0x7f5d10028ba0, n=4, flags=64) at /home/stepan/mariadb/10.4/git/include/mysql/psi/mysql_socket.h:779
      #9  0x0000559feee07d2e in vio_read (vio=0x559ff0c0e6a0, buf=0x7f5d10028ba0 "\001", size=4) at /home/stepan/mariadb/10.4/git/vio/viosocket.c:189
      #10 0x0000559fee6861e3 in my_real_read (net=0x7f5d1000a338, complen=0x7f5d3086c810, header=1 '\001') at /home/stepan/mariadb/10.4/git/sql/net_serv.cc:962
      #11 0x0000559fee686ae1 in my_net_read_packet_reallen (net=0x7f5d1000a338, read_from_server=1 '\001', reallen=0x7f5d3086c878) at /home/stepan/mariadb/10.4/git/sql/net_serv.cc:1243
      #12 0x0000559fee686a94 in my_net_read_packet (net=0x7f5d1000a338, read_from_server=1 '\001') at /home/stepan/mariadb/10.4/git/sql/net_serv.cc:1227
      #13 0x0000559fee19d9de in do_command (thd=0x7f5d1000a040) at /home/stepan/mariadb/10.4/git/sql/sql_parse.cc:1232
      #14 0x0000559fee327d51 in do_handle_one_connection (connect=0x559ff0cff900) at /home/stepan/mariadb/10.4/git/sql/sql_connect.cc:1412
      #15 0x0000559fee327a80 in handle_one_connection (arg=0x559ff0cff900) at /home/stepan/mariadb/10.4/git/sql/sql_connect.cc:1316
      #16 0x0000559feed274a5 in pfs_spawn_thread (arg=0x559ff0d01c90) at /home/stepan/mariadb/10.4/git/storage/perfschema/pfs.cc:1862
      #17 0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
      #18 0x00007f5d4985aead in clone () from /lib64/libc.so.6
       
      Compressed file /home/stepan/mariadb/10.4/git/mysql-test/var/4/log/galera_sr.galera_sr_cc_master-innodb/mysqld.2/data/core.12605
       - saving '/home/stepan/mariadb/10.4/git/mysql-test/var/4/log/galera_sr.galera_sr_cc_master-innodb/' to '/home/stepan/mariadb/10.4/git/mysql-test/var/log/galera_sr.galera_sr_cc_master-innodb/'
      ***Warnings generated in error logs during shutdown after running tests: galera_sr.GCF-627 galera_sr.GCF-900 galera_sr.GCF-889 galera_sr.galera_sr_cc_master galera_sr.galera_sr_blob galera_sr.GCF-851 galera_sr.GCF-867 galera_sr.galera_sr_bf_abort galera_sr.GCF-845
       
      mysqld: /home/stepan/mariadb/10.4/git/include/mysql/psi/mysql_socket.h:779: inline_mysql_socket_recv: Assertion `mysql_socket.fd != -1' failed.
      Attempting backtrace. You can use the following information to find out
      

      All logs: 191025_SSD_galera_sr_cc_master.zip.

      Attachments

        Issue Links

          Activity

            People

              stepan.patryshev Stepan Patryshev (Inactive)
              stepan.patryshev Stepan Patryshev (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.