[MDEV-20899] Galera test failure on galera_sr.galera_sr_cc_master: Server crash with signal 6 Created: 2019-10-25  Updated: 2020-02-10  Resolved: 2020-02-10

Status: Closed
Project: MariaDB Server
Component/s: Galera, Tests
Affects Version/s: 10.4.9
Fix Version/s: N/A

Type: Bug Priority: Minor
Reporter: Stepan Patryshev (Inactive) Assignee: Stepan Patryshev (Inactive)
Resolution: Cannot Reproduce Votes: 0
Labels: galera_4
Environment:

CentOS: 7.6.1810.


Attachments: Zip Archive 191025_SSD_galera_sr_cc_master.zip     Text File 191025_SSD_galera_sr_stdout.log    
Issue Links:
Duplicate
duplicates MDEV-19616 Galera test failure on galera_sr.gale... Closed
Relates
relates to MDEV-19616 Galera test failure on galera_sr.gale... Closed

 Description   

It sporadically fails only on the local suite runs with --parallel, not on the bb.

MariaDB Version 10.4.9-MariaDB-debug: Repository: MariaDB/server; branch 10.4; Revision 2b5f4b3ed68585b310b7ebede474928ff90d9aa2; debug built from sources.

Galera lib 26.4.3(r4535): Repository: MariaDB/galera; branch mariadb-4.x; Revision 752664dc3c7065d8e0c73ac99d0028a5f84eb250; debug built from sources.

Run:

./mtr --suite galera_sr --force --big-test --max-test-fail=0 --parallel=auto --timer --timestamp --timediff

stdout.log (191025_SSD_galera_sr_stdout.log):

2019-10-25 22:28:03 2 [Warning] Aborted connection 2 to db: 'unconnected' user: 'unauthenticated' host: '' (This connection closed normally without authentication)
mysqld: /home/stepan/mariadb/10.4/git/include/mysql/psi/mysql_socket.h:779: inline_mysql_socket_recv: Assertion `mysql_socket.fd != -1' failed.
191025 22:28:03 [ERROR] mysqld got signal 6 ;

191025 22:28:06 +5.780 galera_sr.galera_sr_cc_master 'innodb'   w4 [ fail ]
        Test ended at 2019-10-25 22:28:06
 
CURRENT_TEST: galera_sr.galera_sr_cc_master
mysqltest: At line 44: query 'SET GLOBAL wsrep_cluster_address = ''' failed: 2013: Lost connection to MySQL server during query
 
The result from queries just before the failure was:
< snip >
SET AUTOCOMMIT=OFF;
START TRANSACTION;
INSERT INTO t1 VALUES (1);
INSERT INTO t1 VALUES (2);
INSERT INTO t1 VALUES (3);
INSERT INTO t1 VALUES (4);
INSERT INTO t1 VALUES (5);
SELECT COUNT(*) FROM mysql.wsrep_streaming_log;
COUNT(*)
5
connection node_1;
SELECT COUNT(*) FROM mysql.wsrep_streaming_log;
COUNT(*)
5
connect node_2a, 127.0.0.1, root, , test, $NODE_MYPORT_2;
connection node_2a;
SET SESSION wsrep_sync_wait=0;
SET GLOBAL wsrep_cluster_address = '';
 
More results from queries before failure can be found in /home/stepan/mariadb/10.4/git/mysql-test/var/4/log/galera_sr_cc_master.log
 
 
Server [mysqld.2 - pid: 12603, winpid: 12603, exit: 256] failed during test run
Server log from this test:
----------SERVER LOG START-----------
2019-10-25 22:28:02 140 [Note] WSREP: Stop replication by 140
2019-10-25 22:28:02 140 [Note] WSREP: Server status change synced -> disconnecting
2019-10-25 22:28:02 140 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2019-10-25 22:28:02 140 [Note] WSREP: Closing send monitor...
2019-10-25 22:28:02 140 [Note] WSREP: Closed send monitor.
2019-10-25 22:28:02 140 [Note] WSREP: gcomm: terminating thread
2019-10-25 22:28:02 140 [Note] WSREP: gcomm: joining thread
2019-10-25 22:28:02 140 [Note] WSREP: gcomm: closing backend
2019-10-25 22:28:03 140 [Note] WSREP: view(view_id(NON_PRIM,7c15e3e0,2) memb {
	7c79d6ce,0
} joined {
} left {
} partitioned {
	7c15e3e0,0
})
2019-10-25 22:28:03 140 [Note] WSREP: PC protocol downgrade 1 -> 0
2019-10-25 22:28:03 140 [Note] WSREP: view((empty))
2019-10-25 22:28:03 140 [Note] WSREP: gcomm: closed
2019-10-25 22:28:03 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
2019-10-25 22:28:03 0 [Note] WSREP: Flow-control interval: [16, 16]
2019-10-25 22:28:03 0 [Note] WSREP: Received NON-PRIMARY.
2019-10-25 22:28:03 0 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 216)
2019-10-25 22:28:03 0 [Note] WSREP: New SELF-LEAVE.
2019-10-25 22:28:03 0 [Note] WSREP: Flow-control interval: [0, 0]
2019-10-25 22:28:03 0 [Note] WSREP: Received SELF-LEAVE. Closing connection.
2019-10-25 22:28:03 0 [Note] WSREP: Shifting OPEN -> CLOSED (TO: -1)
2019-10-25 22:28:03 0 [Note] WSREP: RECV thread exiting 0: Success
2019-10-25 22:28:03 140 [Note] WSREP: recv_thread() joined.
2019-10-25 22:28:03 140 [Note] WSREP: Closing replication queue.
2019-10-25 22:28:03 140 [Note] WSREP: Closing slave action queue.
2019-10-25 22:28:03 2 [Note] WSREP: ####### processing CC -1, local, ordered
2019-10-25 22:28:03 2 [Note] WSREP: ####### My UUID: 7c79d6ce-f75d-11e9-8a80-13083ff677cf
2019-10-25 22:28:03 2 [Note] WSREP: ####### ST not required
2019-10-25 22:28:03 2 [Note] WSREP: ================================================
View:
  id: 7c1cbeb7-f75d-11e9-a0f5-662e63655745:-1
  status: non-primary
  protocol_version: 4
  capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
  final: no
  own_index: 0
  members(1):
	0: 7c79d6ce-f75d-11e9-8a80-13083ff677cf, cnt7glr11.localdomain
=================================================
2019-10-25 22:28:03 2 [Note] WSREP: Non-primary view
2019-10-25 22:28:03 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2019-10-25 22:28:03 2 [Note] WSREP: ####### processing CC -1, local, ordered
2019-10-25 22:28:03 2 [Note] WSREP: ####### My UUID: 7c79d6ce-f75d-11e9-8a80-13083ff677cf
2019-10-25 22:28:03 2 [Note] WSREP: ####### ST not required
2019-10-25 22:28:03 2 [Note] WSREP: ================================================
View:
  id: 7c1cbeb7-f75d-11e9-a0f5-662e63655745:-1
  status: non-primary
  protocol_version: 4
  capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
  final: yes
  own_index: -1
  members(0):
=================================================
2019-10-25 22:28:03 2 [Note] WSREP: Non-primary view
2019-10-25 22:28:03 2 [Note] WSREP: Server status change disconnecting -> disconnected
2019-10-25 22:28:03 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2019-10-25 22:28:03 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2019-10-25 22:28:03 2 [Note] WSREP: Applier thread exiting ret: 0 thd: 2
2019-10-25 22:28:03 140 [Note] WSREP: killing local connection: 134
2019-10-25 22:28:03 2 [Warning] Aborted connection 2 to db: 'unconnected' user: 'unauthenticated' host: '' (This connection closed normally without authentication)
mysqld: /home/stepan/mariadb/10.4/git/include/mysql/psi/mysql_socket.h:779: inline_mysql_socket_recv: Assertion `mysql_socket.fd != -1' failed.
191025 22:28:03 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
Server version: 10.4.9-MariaDB-debug-log
key_buffer_size=1048576
read_buffer_size=131072
max_used_connections=2
max_threads=153
thread_count=9
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 63618 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x7f5d1000a040
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f5d3086cd30 thread_stack 0x49000
/home/stepan/mariadb/10.4/git/sql/mysqld(my_print_stacktrace+0x40)[0x559feed99008]
mysys/stacktrace.c:269(my_print_stacktrace)[0x559fee5055dd]
sigaction.c:0(__restore_rt)[0x7f5d4b4a55d0]
:0(__GI_raise)[0x7f5d49793207]
:0(__GI_abort)[0x7f5d497948f8]
:0(__assert_fail_base)[0x7f5d4978c026]
:0(__GI___assert_fail)[0x7f5d4978c0d2]
/home/stepan/mariadb/10.4/git/sql/mysqld(+0x1464638)[0x559feee07638]
psi/mysql_socket.h:781(inline_mysql_socket_recv)[0x559feee07d2e]
sql/net_serv.cc:962(my_real_read(st_net*, unsigned long*, char))[0x559fee6861e3]
sql/net_serv.cc:1243(my_net_read_packet_reallen)[0x559fee686ae1]
sql/net_serv.cc:1228(my_net_read_packet)[0x559fee686a94]
sql/sql_parse.cc:1232(do_command(THD*))[0x559fee19d9de]
sql/sql_connect.cc:1412(do_handle_one_connection(CONNECT*))[0x559fee327d51]
sql/sql_connect.cc:1317(handle_one_connection)[0x559fee327a80]
perfschema/pfs.cc:1864(pfs_spawn_thread)[0x559feed274a5]
pthread_create.c:0(start_thread)[0x7f5d4b49ddd5]
/lib64/libc.so.6(clone+0x6d)[0x7f5d4985aead]
 
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x0): 
Connection ID (thread ID): 134
Status: KILL_CONNECTION
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on
 
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /home/stepan/mariadb/10.4/git/mysql-test/var/4/mysqld.2/data
Resource Limits:
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        unlimited            unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             4096                 23005                processes 
Max open files            1024                 1024                 files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       23005                23005                signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us        
Core pattern: |/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t e %P %I %h
 
----------SERVER LOG END-------------
 
 
 - found 'core.12605' (0/5)
 
Trying 'dbx' to get a backtrace
 
Trying 'gdb' to get a backtrace from coredump /home/stepan/mariadb/10.4/git/mysql-test/var/4/log/galera_sr.galera_sr_cc_master-innodb/mysqld.2/data/core.12605
Core generated by '/home/stepan/mariadb/10.4/git/sql/mysqld'
Output from gdb follows. The first stack trace is from the failing thread.
The following stack traces are from all threads (so the failing one is
duplicated).
--------------------------
[New LWP 13163]
[New LWP 12614]
[New LWP 13078]
[New LWP 12648]
[New LWP 13082]
[New LWP 14442]
[New LWP 13131]
[New LWP 13080]
[New LWP 12613]
[New LWP 13137]
[New LWP 13132]
[New LWP 13067]
[New LWP 13138]
[New LWP 13133]
[New LWP 13130]
[New LWP 13139]
[New LWP 13135]
[New LWP 13134]
[New LWP 13155]
[New LWP 13152]
[New LWP 13136]
[New LWP 13153]
[New LWP 13151]
[New LWP 13154]
[New LWP 12605]
[New LWP 13162]
[New LWP 13081]
[New LWP 13079]
[New LWP 13077]
[New LWP 13076]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/home/stepan/mariadb/10.4/git/sql/mysqld --defaults-group-suffix=.2 --defaults-'.
Program terminated with signal 6, Aborted.
#0  0x00007f5d4b4a29d1 in pthread_kill () from /lib64/libpthread.so.0
#0  0x00007f5d4b4a29d1 in pthread_kill () from /lib64/libpthread.so.0
#1  0x0000559feed990f6 in my_write_core (sig=6) at /home/stepan/mariadb/10.4/git/mysys/stacktrace.c:481
#2  0x0000559fee505964 in handle_fatal_signal (sig=6) at /home/stepan/mariadb/10.4/git/sql/signal_handler.cc:343
#3  <signal handler called>
#4  0x00007f5d49793207 in raise () from /lib64/libc.so.6
#5  0x00007f5d497948f8 in abort () from /lib64/libc.so.6
#6  0x00007f5d4978c026 in __assert_fail_base () from /lib64/libc.so.6
#7  0x00007f5d4978c0d2 in __assert_fail () from /lib64/libc.so.6
#8  0x0000559feee07638 in inline_mysql_socket_recv (src_file=0x559fef391f70 "/home/stepan/mariadb/10.4/git/vio/viosocket.c", src_line=190, mysql_socket=..., buf=0x7f5d10028ba0, n=4, flags=64) at /home/stepan/mariadb/10.4/git/include/mysql/psi/mysql_socket.h:779
#9  0x0000559feee07d2e in vio_read (vio=0x559ff0c0e6a0, buf=0x7f5d10028ba0 "\001", size=4) at /home/stepan/mariadb/10.4/git/vio/viosocket.c:189
#10 0x0000559fee6861e3 in my_real_read (net=0x7f5d1000a338, complen=0x7f5d3086c810, header=1 '\001') at /home/stepan/mariadb/10.4/git/sql/net_serv.cc:962
#11 0x0000559fee686ae1 in my_net_read_packet_reallen (net=0x7f5d1000a338, read_from_server=1 '\001', reallen=0x7f5d3086c878) at /home/stepan/mariadb/10.4/git/sql/net_serv.cc:1243
#12 0x0000559fee686a94 in my_net_read_packet (net=0x7f5d1000a338, read_from_server=1 '\001') at /home/stepan/mariadb/10.4/git/sql/net_serv.cc:1227
#13 0x0000559fee19d9de in do_command (thd=0x7f5d1000a040) at /home/stepan/mariadb/10.4/git/sql/sql_parse.cc:1232
#14 0x0000559fee327d51 in do_handle_one_connection (connect=0x559ff0cff900) at /home/stepan/mariadb/10.4/git/sql/sql_connect.cc:1412
#15 0x0000559fee327a80 in handle_one_connection (arg=0x559ff0cff900) at /home/stepan/mariadb/10.4/git/sql/sql_connect.cc:1316
#16 0x0000559feed274a5 in pfs_spawn_thread (arg=0x559ff0d01c90) at /home/stepan/mariadb/10.4/git/storage/perfschema/pfs.cc:1862
#17 0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
#18 0x00007f5d4985aead in clone () from /lib64/libc.so.6
 
Thread 30 (Thread 0x7f5d17fff700 (LWP 13076)):
#0  0x00007f5d4b06e644 in __io_getevents_0_4 () from /lib64/libaio.so.1
#1  0x0000559fee845dad in LinuxAIOHandler::collect (this=0x7f5d17ffeb00) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0file.cc:1809
#2  0x0000559fee8463f6 in LinuxAIOHandler::poll (this=0x7f5d17ffeb00, m1=0x7f5d17ffeba8, m2=0x7f5d17ffebb0, request=0x7f5d17ffebe0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0file.cc:1968
#3  0x0000559fee846559 in os_aio_linux_handler (global_segment=0, m1=0x7f5d17ffeba8, m2=0x7f5d17ffebb0, request=0x7f5d17ffebe0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0file.cc:2022
#4  0x0000559fee84b2c0 in os_aio_handler (segment=0, m1=0x7f5d17ffeba8, m2=0x7f5d17ffebb0, request=0x7f5d17ffebe0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0file.cc:5561
#5  0x0000559feeb014f9 in fil_aio_wait (segment=0) at /home/stepan/mariadb/10.4/git/storage/innobase/fil/fil0fil.cc:4357
#6  0x0000559fee972e42 in io_handler_thread (arg=0x559fefbaa980 <n>) at /home/stepan/mariadb/10.4/git/storage/innobase/srv/srv0start.cc:324
#7  0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
#8  0x00007f5d4985aead in clone () from /lib64/libc.so.6
 
Thread 29 (Thread 0x7f5d177fe700 (LWP 13077)):
#0  0x00007f5d4b06e644 in __io_getevents_0_4 () from /lib64/libaio.so.1
#1  0x0000559fee845dad in LinuxAIOHandler::collect (this=0x7f5d177fdb00) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0file.cc:1809
#2  0x0000559fee8463f6 in LinuxAIOHandler::poll (this=0x7f5d177fdb00, m1=0x7f5d177fdba8, m2=0x7f5d177fdbb0, request=0x7f5d177fdbe0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0file.cc:1968
#3  0x0000559fee846559 in os_aio_linux_handler (global_segment=1, m1=0x7f5d177fdba8, m2=0x7f5d177fdbb0, request=0x7f5d177fdbe0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0file.cc:2022
#4  0x0000559fee84b2c0 in os_aio_handler (segment=1, m1=0x7f5d177fdba8, m2=0x7f5d177fdbb0, request=0x7f5d177fdbe0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0file.cc:5561
#5  0x0000559feeb014f9 in fil_aio_wait (segment=1) at /home/stepan/mariadb/10.4/git/storage/innobase/fil/fil0fil.cc:4357
#6  0x0000559fee972e42 in io_handler_thread (arg=0x559fefbaa988 <n+8>) at /home/stepan/mariadb/10.4/git/storage/innobase/srv/srv0start.cc:324
#7  0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
#8  0x00007f5d4985aead in clone () from /lib64/libc.so.6
 
Thread 28 (Thread 0x7f5d167fc700 (LWP 13079)):
#0  0x00007f5d4b06e644 in __io_getevents_0_4 () from /lib64/libaio.so.1
#1  0x0000559fee845dad in LinuxAIOHandler::collect (this=0x7f5d167fbb00) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0file.cc:1809
#2  0x0000559fee8463f6 in LinuxAIOHandler::poll (this=0x7f5d167fbb00, m1=0x7f5d167fbba8, m2=0x7f5d167fbbb0, request=0x7f5d167fbbe0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0file.cc:1968
#3  0x0000559fee846559 in os_aio_linux_handler (global_segment=3, m1=0x7f5d167fbba8, m2=0x7f5d167fbbb0, request=0x7f5d167fbbe0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0file.cc:2022
#4  0x0000559fee84b2c0 in os_aio_handler (segment=3, m1=0x7f5d167fbba8, m2=0x7f5d167fbbb0, request=0x7f5d167fbbe0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0file.cc:5561
#5  0x0000559feeb014f9 in fil_aio_wait (segment=3) at /home/stepan/mariadb/10.4/git/storage/innobase/fil/fil0fil.cc:4357
#6  0x0000559fee972e42 in io_handler_thread (arg=0x559fefbaa998 <n+24>) at /home/stepan/mariadb/10.4/git/storage/innobase/srv/srv0start.cc:324
#7  0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
#8  0x00007f5d4985aead in clone () from /lib64/libc.so.6
 
Thread 27 (Thread 0x7f5d157fa700 (LWP 13081)):
#0  0x00007f5d4b06e644 in __io_getevents_0_4 () from /lib64/libaio.so.1
#1  0x0000559fee845dad in LinuxAIOHandler::collect (this=0x7f5d157f9b00) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0file.cc:1809
#2  0x0000559fee8463f6 in LinuxAIOHandler::poll (this=0x7f5d157f9b00, m1=0x7f5d157f9ba8, m2=0x7f5d157f9bb0, request=0x7f5d157f9be0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0file.cc:1968
#3  0x0000559fee846559 in os_aio_linux_handler (global_segment=5, m1=0x7f5d157f9ba8, m2=0x7f5d157f9bb0, request=0x7f5d157f9be0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0file.cc:2022
#4  0x0000559fee84b2c0 in os_aio_handler (segment=5, m1=0x7f5d157f9ba8, m2=0x7f5d157f9bb0, request=0x7f5d157f9be0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0file.cc:5561
#5  0x0000559feeb014f9 in fil_aio_wait (segment=5) at /home/stepan/mariadb/10.4/git/storage/innobase/fil/fil0fil.cc:4357
#6  0x0000559fee972e42 in io_handler_thread (arg=0x559fefbaa9a8 <n+40>) at /home/stepan/mariadb/10.4/git/storage/innobase/srv/srv0start.cc:324
#7  0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
#8  0x00007f5d4985aead in clone () from /lib64/libc.so.6
 
Thread 26 (Thread 0x7f5d40063700 (LWP 13162)):
#0  0x00007f5d4b4a1965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000559feed9e769 in safe_cond_wait (cond=0x559fefb2c0a0 <COND_slave_background>, mp=0x559fefb2f320 <LOCK_slave_background>, file=0x559feee6d040 "/home/stepan/mariadb/10.4/git/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/mariadb/10.4/git/mysys/thr_mutex.c:492
#2  0x0000559fee09008e in inline_mysql_cond_wait (that=0x559fefb2c0a0 <COND_slave_background>, mutex=0x559fefb2f320 <LOCK_slave_background>, src_file=0x559feee6db80 "/home/stepan/mariadb/10.4/git/sql/slave.cc", src_line=519) at /home/stepan/mariadb/10.4/git/include/mysql/psi/mysql_thread.h:1167
#3  0x0000559fee0919cb in handle_slave_background (arg=0x0) at /home/stepan/mariadb/10.4/git/sql/slave.cc:519
#4  0x0000559feed274a5 in pfs_spawn_thread (arg=0x559ff0c76690) at /home/stepan/mariadb/10.4/git/storage/perfschema/pfs.cc:1862
#5  0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f5d4985aead in clone () from /lib64/libc.so.6
 
Thread 25 (Thread 0x7f5d4b8ae880 (LWP 12605)):
#0  0x00007f5d4985020d in poll () from /lib64/libc.so.6
#1  0x0000559fee06a2db in handle_connections_sockets () at /home/stepan/mariadb/10.4/git/sql/mysqld.cc:6455
#2  0x0000559fee069576 in mysqld_main (argc=163, argv=0x559ff0642130) at /home/stepan/mariadb/10.4/git/sql/mysqld.cc:5891
#3  0x0000559fee05d095 in main (argc=23, argv=0x7ffd7ed249d8) at /home/stepan/mariadb/10.4/git/sql/main.cc:25
 
Thread 24 (Thread 0x7f5cf4ff9700 (LWP 13154)):
#0  0x00007f5d4b4a1965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000559feed9e769 in safe_cond_wait (cond=0x7f5cf4ff8c70, mp=0x7f5cf4ff8cb0, file=0x559fef08adc0 "/home/stepan/mariadb/10.4/git/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/mariadb/10.4/git/mysys/thr_mutex.c:492
#2  0x0000559fee74715d in inline_mysql_cond_wait (that=0x7f5cf4ff8c70, mutex=0x7f5cf4ff8cb0, src_file=0x559fef08c888 "/home/stepan/mariadb/10.4/git/storage/innobase/handler/ha_innodb.cc", src_line=288) at /home/stepan/mariadb/10.4/git/include/mysql/psi/mysql_thread.h:1178
#3  0x0000559fee74ca55 in thd_destructor_proxy () at /home/stepan/mariadb/10.4/git/storage/innobase/handler/ha_innodb.cc:288
#4  0x0000559feed274a5 in pfs_spawn_thread (arg=0x559ff0bb39e0) at /home/stepan/mariadb/10.4/git/storage/perfschema/pfs.cc:1862
#5  0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f5d4985aead in clone () from /lib64/libc.so.6
 
Thread 23 (Thread 0x7f5cf67fc700 (LWP 13151)):
#0  0x00007f5d4b4a1965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000559fee854377 in os_event::wait (this=0x559ff09094f0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:158
#2  0x0000559fee853e6b in os_event::wait_low (this=0x559ff09094f0, reset_sig_count=1) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:325
#3  0x0000559fee85416e in os_event_wait_low (event=0x559ff09094f0, reset_sig_count=0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:502
#4  0x0000559feea71e86 in buf_dump_thread () at /home/stepan/mariadb/10.4/git/storage/innobase/buf/buf0dump.cc:833
#5  0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f5d4985aead in clone () from /lib64/libc.so.6
 
Thread 22 (Thread 0x7f5cf57fa700 (LWP 13153)):
#0  0x00007f5d4b4a1965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000559fee854377 in os_event::wait (this=0x559ff09095f0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:158
#2  0x0000559fee853e6b in os_event::wait_low (this=0x559ff09095f0, reset_sig_count=1) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:325
#3  0x0000559fee85416e in os_event_wait_low (event=0x559ff09095f0, reset_sig_count=0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:502
#4  0x0000559feea577d9 in buf_resize_thread () at /home/stepan/mariadb/10.4/git/storage/innobase/buf/buf0buf.cc:3199
#5  0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f5d4985aead in clone () from /lib64/libc.so.6
 
Thread 21 (Thread 0x7f5d08ff9700 (LWP 13136)):
#0  0x00007f5d4b4a1965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000559fee854377 in os_event::wait (this=0x559ff09091f0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:158
#2  0x0000559fee853e6b in os_event::wait_low (this=0x559ff09091f0, reset_sig_count=257) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:325
#3  0x0000559fee85416e in os_event_wait_low (event=0x559ff09091f0, reset_sig_count=257) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:502
#4  0x0000559fee96c328 in srv_resume_thread (slot=0x559fef966c10 <srv_sys+592>, sig_count=257, wait=true, timeout_usec=0) at /home/stepan/mariadb/10.4/git/storage/innobase/srv/srv0srv.cc:909
#5  0x0000559fee970aa0 in srv_purge_coordinator_suspend (slot=0x559fef966c10 <srv_sys+592>, rseg_history_len=16) at /home/stepan/mariadb/10.4/git/storage/innobase/srv/srv0srv.cc:2657
#6  0x0000559fee970dd1 in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/mariadb/10.4/git/storage/innobase/srv/srv0srv.cc:2738
#7  0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
#8  0x00007f5d4985aead in clone () from /lib64/libc.so.6
 
Thread 20 (Thread 0x7f5cf5ffb700 (LWP 13152)):
#0  0x00007f5d4b4a4e3d in nanosleep () from /lib64/libpthread.so.0
#1  0x0000559fee8546b4 in os_thread_sleep (tm=1000000) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0thread.cc:225
#2  0x0000559feea46dd3 in btr_defragment_thread () at /home/stepan/mariadb/10.4/git/storage/innobase/btr/btr0defragment.cc:705
#3  0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f5d4985aead in clone () from /lib64/libc.so.6
 
Thread 19 (Thread 0x7f5d400ae700 (LWP 13155)):
#0  0x00007f5d4979440a in sigwaitinfo () from /lib64/libc.so.6
#1  0x0000559fee05d114 in my_sigwait (set=0x7f5d400adcf0, sig=0x7f5d400adcb4, code=0x7f5d400adcb8) at /home/stepan/mariadb/10.4/git/include/my_pthread.h:198
#2  0x0000559fee063a8d in signal_hand (arg=0x0) at /home/stepan/mariadb/10.4/git/sql/mysqld.cc:3243
#3  0x0000559feed274a5 in pfs_spawn_thread (arg=0x559ff08c3710) at /home/stepan/mariadb/10.4/git/storage/perfschema/pfs.cc:1862
#4  0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f5d4985aead in clone () from /lib64/libc.so.6
 
Thread 18 (Thread 0x7f5d097fa700 (LWP 13134)):
#0  0x00007f5d4b4a1d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000559fee853d00 in os_event::timed_wait (this=0x559ff0aadee0, abstime=0x7f5d097f9b70) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:275
#2  0x0000559fee853fa9 in os_event::wait_time_low (this=0x559ff0aadee0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:385
#3  0x0000559fee854149 in os_event_wait_time_low (event=0x559ff0aadee0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:485
#4  0x0000559fee9dc563 in ib_wqueue_timedwait (wq=0x559ff0aaddb0, wait_in_usecs=5000000) at /home/stepan/mariadb/10.4/git/storage/innobase/ut/ut0wqueue.cc:163
#5  0x0000559feeb4f23d in fts_optimize_thread (arg=0x559ff0aaddb0) at /home/stepan/mariadb/10.4/git/storage/innobase/fts/fts0opt.cc:2865
#6  0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f5d4985aead in clone () from /lib64/libc.so.6
 
Thread 17 (Thread 0x7f5d0bfff700 (LWP 13135)):
#0  0x00007f5d4b4a4e3d in nanosleep () from /lib64/libpthread.so.0
#1  0x0000559fee8546b4 in os_thread_sleep (tm=1000000) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0thread.cc:225
#2  0x0000559fee96fd65 in srv_master_sleep () at /home/stepan/mariadb/10.4/git/storage/innobase/srv/srv0srv.cc:2334
#3  0x0000559fee96fecd in srv_master_thread (arg=0x0) at /home/stepan/mariadb/10.4/git/storage/innobase/srv/srv0srv.cc:2375
#4  0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f5d4985aead in clone () from /lib64/libc.so.6
 
Thread 16 (Thread 0x7f5cf6ffd700 (LWP 13139)):
#0  0x00007f5d4b4a1965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000559fee854377 in os_event::wait (this=0x559ff0909370) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:158
#2  0x0000559fee853e6b in os_event::wait_low (this=0x559ff0909370, reset_sig_count=902) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:325
#3  0x0000559fee85416e in os_event_wait_low (event=0x559ff0909370, reset_sig_count=0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:502
#4  0x0000559fee96c328 in srv_resume_thread (slot=0x559fef966f10 <srv_sys+1360>, sig_count=0, wait=true, timeout_usec=0) at /home/stepan/mariadb/10.4/git/storage/innobase/srv/srv0srv.cc:909
#5  0x0000559fee9705e9 in srv_worker_thread (arg=0x0) at /home/stepan/mariadb/10.4/git/storage/innobase/srv/srv0srv.cc:2517
#6  0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f5d4985aead in clone () from /lib64/libc.so.6
 
Thread 15 (Thread 0x7f5d0b7fe700 (LWP 13130)):
#0  0x00007f5d4b4a1d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000559fee853d00 in os_event::timed_wait (this=0x559ff0b446e0, abstime=0x7f5d0b7fdd50) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:275
#2  0x0000559fee853fa9 in os_event::wait_time_low (this=0x559ff0b446e0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:385
#3  0x0000559fee854149 in os_event_wait_time_low (event=0x559ff0b446e0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:485
#4  0x0000559fee816527 in lock_wait_timeout_thread () at /home/stepan/mariadb/10.4/git/storage/innobase/lock/lock0wait.cc:522
#5  0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f5d4985aead in clone () from /lib64/libc.so.6
 
Thread 14 (Thread 0x7f5d09ffb700 (LWP 13133)):
#0  0x00007f5d4b4a1d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000559fee853d00 in os_event::timed_wait (this=0x559ff0b76580, abstime=0x7f5d09ffad70) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:275
#2  0x0000559fee853fa9 in os_event::wait_time_low (this=0x559ff0b76580, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:385
#3  0x0000559fee854149 in os_event_wait_time_low (event=0x559ff0b76580, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:485
#4  0x0000559feeae86ca in dict_stats_thread () at /home/stepan/mariadb/10.4/git/storage/innobase/dict/dict0stats_bg.cc:476
#5  0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f5d4985aead in clone () from /lib64/libc.so.6
 
Thread 13 (Thread 0x7f5cf77fe700 (LWP 13138)):
#0  0x00007f5d4b4a1965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000559fee854377 in os_event::wait (this=0x559ff0909270) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:158
#2  0x0000559fee853e6b in os_event::wait_low (this=0x559ff0909270, reset_sig_count=898) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:325
#3  0x0000559fee85416e in os_event_wait_low (event=0x559ff0909270, reset_sig_count=0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:502
#4  0x0000559fee96c328 in srv_resume_thread (slot=0x559fef966d10 <srv_sys+848>, sig_count=0, wait=true, timeout_usec=0) at /home/stepan/mariadb/10.4/git/storage/innobase/srv/srv0srv.cc:909
#5  0x0000559fee9705e9 in srv_worker_thread (arg=0x0) at /home/stepan/mariadb/10.4/git/storage/innobase/srv/srv0srv.cc:2517
#6  0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f5d4985aead in clone () from /lib64/libc.so.6
 
Thread 12 (Thread 0x7f5d31e6f700 (LWP 13067)):
#0  0x00007f5d4b4a1d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000559feed9ea6b in safe_cond_timedwait (cond=0x559ff03bb5e0 <COND_checkpoint>, mp=0x559ff03bb520 <LOCK_checkpoint>, abstime=0x7f5d31e6ecb0, file=0x559fef206850 "/home/stepan/mariadb/10.4/git/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/mariadb/10.4/git/mysys/thr_mutex.c:546
#2  0x0000559feebf6109 in inline_mysql_cond_timedwait (that=0x559ff03bb5e0 <COND_checkpoint>, mutex=0x559ff03bb520 <LOCK_checkpoint>, abstime=0x7f5d31e6ecb0, src_file=0x559fef206890 "/home/stepan/mariadb/10.4/git/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/mariadb/10.4/git/include/mysql/psi/mysql_thread.h:1204
#3  0x0000559feebf664d in my_service_thread_sleep (control=0x559fef96d780 <checkpoint_control>, sleep_time=30000000000) at /home/stepan/mariadb/10.4/git/storage/maria/ma_servicethread.c:115
#4  0x0000559feebea50a in ma_checkpoint_background (arg=0x1e) at /home/stepan/mariadb/10.4/git/storage/maria/ma_checkpoint.c:707
#5  0x0000559feed274a5 in pfs_spawn_thread (arg=0x559ff08c1030) at /home/stepan/mariadb/10.4/git/storage/perfschema/pfs.cc:1862
#6  0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f5d4985aead in clone () from /lib64/libc.so.6
 
Thread 11 (Thread 0x7f5d0a7fc700 (LWP 13132)):
#0  0x00007f5d4b4a1d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000559fee853d00 in os_event::timed_wait (this=0x559ff0909470, abstime=0x7f5d0a7fbd30) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:275
#2  0x0000559fee853fa9 in os_event::wait_time_low (this=0x559ff0909470, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:385
#3  0x0000559fee854149 in os_event_wait_time_low (event=0x559ff0909470, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:485
#4  0x0000559fee96e20f in srv_monitor_thread () at /home/stepan/mariadb/10.4/git/storage/innobase/srv/srv0srv.cc:1687
#5  0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f5d4985aead in clone () from /lib64/libc.so.6
 
Thread 10 (Thread 0x7f5cf7fff700 (LWP 13137)):
#0  0x00007f5d4b4a1965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000559fee854377 in os_event::wait (this=0x559ff09092f0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:158
#2  0x0000559fee853e6b in os_event::wait_low (this=0x559ff09092f0, reset_sig_count=893) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:325
#3  0x0000559fee85416e in os_event_wait_low (event=0x559ff09092f0, reset_sig_count=0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:502
#4  0x0000559fee96c328 in srv_resume_thread (slot=0x559fef966e10 <srv_sys+1104>, sig_count=0, wait=true, timeout_usec=0) at /home/stepan/mariadb/10.4/git/storage/innobase/srv/srv0srv.cc:909
#5  0x0000559fee9705e9 in srv_worker_thread (arg=0x0) at /home/stepan/mariadb/10.4/git/storage/innobase/srv/srv0srv.cc:2517
#6  0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f5d4985aead in clone () from /lib64/libc.so.6
 
Thread 9 (Thread 0x7f5d43f36700 (LWP 12613)):
#0  0x00007f5d4b4a1d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000559feed9ea6b in safe_cond_timedwait (cond=0x559ff03cdce0 <COND_timer>, mp=0x559ff03cdc20 <LOCK_timer>, abstime=0x7f5d43f35d70, file=0x559fef24ed80 "/home/stepan/mariadb/10.4/git/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/mariadb/10.4/git/mysys/thr_mutex.c:546
#2  0x0000559feed9faae in inline_mysql_cond_timedwait (that=0x559ff03cdce0 <COND_timer>, mutex=0x559ff03cdc20 <LOCK_timer>, abstime=0x7f5d43f35d70, src_file=0x559fef24edc0 "/home/stepan/mariadb/10.4/git/mysys/thr_timer.c", src_line=292) at /home/stepan/mariadb/10.4/git/include/mysql/psi/mysql_thread.h:1204
#3  0x0000559feeda0671 in timer_handler (arg=0x0) at /home/stepan/mariadb/10.4/git/mysys/thr_timer.c:292
#4  0x0000559feed274a5 in pfs_spawn_thread (arg=0x559ff0788250) at /home/stepan/mariadb/10.4/git/storage/perfschema/pfs.cc:1862
#5  0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f5d4985aead in clone () from /lib64/libc.so.6
 
Thread 8 (Thread 0x7f5d15ffb700 (LWP 13080)):
#0  0x00007f5d4b06e644 in __io_getevents_0_4 () from /lib64/libaio.so.1
#1  0x0000559fee845dad in LinuxAIOHandler::collect (this=0x7f5d15ffab00) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0file.cc:1809
#2  0x0000559fee8463f6 in LinuxAIOHandler::poll (this=0x7f5d15ffab00, m1=0x7f5d15ffaba8, m2=0x7f5d15ffabb0, request=0x7f5d15ffabe0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0file.cc:1968
#3  0x0000559fee846559 in os_aio_linux_handler (global_segment=4, m1=0x7f5d15ffaba8, m2=0x7f5d15ffabb0, request=0x7f5d15ffabe0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0file.cc:2022
#4  0x0000559fee84b2c0 in os_aio_handler (segment=4, m1=0x7f5d15ffaba8, m2=0x7f5d15ffabb0, request=0x7f5d15ffabe0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0file.cc:5561
#5  0x0000559feeb014f9 in fil_aio_wait (segment=4) at /home/stepan/mariadb/10.4/git/storage/innobase/fil/fil0fil.cc:4357
#6  0x0000559fee972e42 in io_handler_thread (arg=0x559fefbaa9a0 <n+32>) at /home/stepan/mariadb/10.4/git/storage/innobase/srv/srv0start.cc:324
#7  0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
#8  0x00007f5d4985aead in clone () from /lib64/libc.so.6
 
Thread 7 (Thread 0x7f5d0affd700 (LWP 13131)):
#0  0x00007f5d4b4a1d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000559fee853d00 in os_event::timed_wait (this=0x559ff09093f0, abstime=0x7f5d0affcbb0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:275
#2  0x0000559fee853fa9 in os_event::wait_time_low (this=0x559ff09093f0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:385
#3  0x0000559fee854149 in os_event_wait_time_low (event=0x559ff09093f0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:485
#4  0x0000559fee96e789 in srv_error_monitor_thread () at /home/stepan/mariadb/10.4/git/storage/innobase/srv/srv0srv.cc:1851
#5  0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f5d4985aead in clone () from /lib64/libc.so.6
 
Thread 6 (Thread 0x7f5d30822700 (LWP 14442)):
#0  0x00007f5d49821e2d in nanosleep () from /lib64/libc.so.6
#1  0x00007f5d49821cc4 in sleep () from /lib64/libc.so.6
#2  0x0000559fee415642 in wsrep_close_client_connections (wait_to_end=1 '\001', except_caller_thd=0x7f5cbc000b00) at /home/stepan/mariadb/10.4/git/sql/wsrep_mysqld.cc:2344
#3  0x0000559fee40effc in wsrep_stop_replication (thd=0x7f5cbc000b00) at /home/stepan/mariadb/10.4/git/sql/wsrep_mysqld.cc:943
#4  0x0000559fee4218ae in wsrep_cluster_address_update (self=0x559fefb548a0 <Sys_wsrep_cluster_address>, thd=0x7f5cbc000b00, type=SHOW_OPT_GLOBAL) at /home/stepan/mariadb/10.4/git/sql/wsrep_var.cc:483
#5  0x0000559fee08a958 in sys_var::update (this=0x559fefb548a0 <Sys_wsrep_cluster_address>, thd=0x7f5cbc000b00, var=0x7f5cbc011f60) at /home/stepan/mariadb/10.4/git/sql/set_var.cc:209
#6  0x0000559fee08c6f4 in set_var::update (this=0x7f5cbc011f60, thd=0x7f5cbc000b00) at /home/stepan/mariadb/10.4/git/sql/set_var.cc:837
#7  0x0000559fee08c36e in sql_set_variables (thd=0x7f5cbc000b00, var_list=0x7f5cbc005930, free=true) at /home/stepan/mariadb/10.4/git/sql/set_var.cc:740
#8  0x0000559fee1aa179 in mysql_execute_command (thd=0x7f5cbc000b00) at /home/stepan/mariadb/10.4/git/sql/sql_parse.cc:4944
#9  0x0000559fee1b47b5 in mysql_parse (thd=0x7f5cbc000b00, rawbuf=0x7f5cbc011df8 "SET GLOBAL wsrep_cluster_address = ''", length=37, parser_state=0x7f5d30821030, is_com_multi=false, is_next_command=false) at /home/stepan/mariadb/10.4/git/sql/sql_parse.cc:7912
#10 0x0000559fee1b3d68 in wsrep_mysql_parse (thd=0x7f5cbc000b00, rawbuf=0x7f5cbc011df8 "SET GLOBAL wsrep_cluster_address = ''", length=37, parser_state=0x7f5d30821030, is_com_multi=false, is_next_command=false) at /home/stepan/mariadb/10.4/git/sql/sql_parse.cc:7716
#11 0x0000559fee19fa77 in dispatch_command (command=COM_QUERY, thd=0x7f5cbc000b00, packet=0x7f5cbc008331 "SET GLOBAL wsrep_cluster_address = ''", packet_length=37, is_com_multi=false, is_next_command=false) at /home/stepan/mariadb/10.4/git/sql/sql_parse.cc:1826
#12 0x0000559fee19e18a in do_command (thd=0x7f5cbc000b00) at /home/stepan/mariadb/10.4/git/sql/sql_parse.cc:1359
#13 0x0000559fee327d51 in do_handle_one_connection (connect=0x559ff0cff900) at /home/stepan/mariadb/10.4/git/sql/sql_connect.cc:1412
#14 0x0000559fee327a80 in handle_one_connection (arg=0x559ff0cff900) at /home/stepan/mariadb/10.4/git/sql/sql_connect.cc:1316
#15 0x0000559feed274a5 in pfs_spawn_thread (arg=0x559ff0745ee0) at /home/stepan/mariadb/10.4/git/storage/perfschema/pfs.cc:1862
#16 0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
#17 0x00007f5d4985aead in clone () from /lib64/libc.so.6
 
Thread 5 (Thread 0x7f5d14ff9700 (LWP 13082)):
#0  0x00007f5d4b4a1d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000559fee853d00 in os_event::timed_wait (this=0x559ff0909570, abstime=0x7f5d14ff8b00) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:275
#2  0x0000559fee853fa9 in os_event::wait_time_low (this=0x559ff0909570, time_in_usec=213000, reset_sig_count=2) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:385
#3  0x0000559fee854149 in os_event_wait_time_low (event=0x559ff0909570, time_in_usec=213000, reset_sig_count=2) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0event.cc:485
#4  0x0000559feea7e358 in pc_sleep_if_needed (next_loop_time=188004048, sig_count=2, cur_time=188003835) at /home/stepan/mariadb/10.4/git/storage/innobase/buf/buf0flu.cc:2643
#5  0x0000559feea7f4fc in buf_flush_page_cleaner_coordinator () at /home/stepan/mariadb/10.4/git/storage/innobase/buf/buf0flu.cc:3102
#6  0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f5d4985aead in clone () from /lib64/libc.so.6
 
Thread 4 (Thread 0x7f5d40321700 (LWP 12648)):
#0  0x00007f5d4b4a1965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000559feed9e769 in safe_cond_wait (cond=0x7f5d24015a68, mp=0x7f5d240159b8, file=0x559feef1aa20 "/home/stepan/mariadb/10.4/git/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/mariadb/10.4/git/mysys/thr_mutex.c:492
#2  0x0000559fee4261b4 in inline_mysql_cond_wait (that=0x7f5d24015a68, mutex=0x7f5d240159b8, src_file=0x559feef1ac30 "/home/stepan/mariadb/10.4/git/sql/wsrep_thd.h", src_line=69) at /home/stepan/mariadb/10.4/git/include/mysql/psi/mysql_thread.h:1178
#3  0x0000559fee429416 in Wsrep_thd_queue::pop_front (this=0x7f5d24015960) at /home/stepan/mariadb/10.4/git/sql/wsrep_thd.h:69
#4  0x0000559fee427e23 in wsrep_rollback_process (rollbacker=0x7f5d24001d50, arg=0x559ff0819dc0) at /home/stepan/mariadb/10.4/git/sql/wsrep_thd.cc:266
#5  0x0000559fee416e70 in start_wsrep_THD (arg=0x559ff0819dc0) at /home/stepan/mariadb/10.4/git/sql/wsrep_mysqld.cc:2708
#6  0x0000559feed274a5 in pfs_spawn_thread (arg=0x559ff07dc8f0) at /home/stepan/mariadb/10.4/git/storage/perfschema/pfs.cc:1862
#7  0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
#8  0x00007f5d4985aead in clone () from /lib64/libc.so.6
 
Thread 3 (Thread 0x7f5d16ffd700 (LWP 13078)):
#0  0x00007f5d4b06e644 in __io_getevents_0_4 () from /lib64/libaio.so.1
#1  0x0000559fee845dad in LinuxAIOHandler::collect (this=0x7f5d16ffcb00) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0file.cc:1809
#2  0x0000559fee8463f6 in LinuxAIOHandler::poll (this=0x7f5d16ffcb00, m1=0x7f5d16ffcba8, m2=0x7f5d16ffcbb0, request=0x7f5d16ffcbe0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0file.cc:1968
#3  0x0000559fee846559 in os_aio_linux_handler (global_segment=2, m1=0x7f5d16ffcba8, m2=0x7f5d16ffcbb0, request=0x7f5d16ffcbe0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0file.cc:2022
#4  0x0000559fee84b2c0 in os_aio_handler (segment=2, m1=0x7f5d16ffcba8, m2=0x7f5d16ffcbb0, request=0x7f5d16ffcbe0) at /home/stepan/mariadb/10.4/git/storage/innobase/os/os0file.cc:5561
#5  0x0000559feeb014f9 in fil_aio_wait (segment=2) at /home/stepan/mariadb/10.4/git/storage/innobase/fil/fil0fil.cc:4357
#6  0x0000559fee972e42 in io_handler_thread (arg=0x559fefbaa990 <n+16>) at /home/stepan/mariadb/10.4/git/storage/innobase/srv/srv0start.cc:324
#7  0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
#8  0x00007f5d4985aead in clone () from /lib64/libc.so.6
 
Thread 2 (Thread 0x7f5d42fa6700 (LWP 12614)):
#0  0x00007f5d4b4a1965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f5d433fcdd7 in gu::Lock::wait (cond=..., this=0x7f5d42fa5bf0) at galerautils/src/gu_lock.hpp:45
#2  0x00007f5d433fce69 in galera::ServiceThd::thd_func (arg=0x559ff07bc698) at galera/src/galera_service_thd.cpp:30
#3  0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f5d4985aead in clone () from /lib64/libc.so.6
 
Thread 1 (Thread 0x7f5d3086d700 (LWP 13163)):
#0  0x00007f5d4b4a29d1 in pthread_kill () from /lib64/libpthread.so.0
#1  0x0000559feed990f6 in my_write_core (sig=6) at /home/stepan/mariadb/10.4/git/mysys/stacktrace.c:481
#2  0x0000559fee505964 in handle_fatal_signal (sig=6) at /home/stepan/mariadb/10.4/git/sql/signal_handler.cc:343
#3  <signal handler called>
#4  0x00007f5d49793207 in raise () from /lib64/libc.so.6
#5  0x00007f5d497948f8 in abort () from /lib64/libc.so.6
#6  0x00007f5d4978c026 in __assert_fail_base () from /lib64/libc.so.6
#7  0x00007f5d4978c0d2 in __assert_fail () from /lib64/libc.so.6
#8  0x0000559feee07638 in inline_mysql_socket_recv (src_file=0x559fef391f70 "/home/stepan/mariadb/10.4/git/vio/viosocket.c", src_line=190, mysql_socket=..., buf=0x7f5d10028ba0, n=4, flags=64) at /home/stepan/mariadb/10.4/git/include/mysql/psi/mysql_socket.h:779
#9  0x0000559feee07d2e in vio_read (vio=0x559ff0c0e6a0, buf=0x7f5d10028ba0 "\001", size=4) at /home/stepan/mariadb/10.4/git/vio/viosocket.c:189
#10 0x0000559fee6861e3 in my_real_read (net=0x7f5d1000a338, complen=0x7f5d3086c810, header=1 '\001') at /home/stepan/mariadb/10.4/git/sql/net_serv.cc:962
#11 0x0000559fee686ae1 in my_net_read_packet_reallen (net=0x7f5d1000a338, read_from_server=1 '\001', reallen=0x7f5d3086c878) at /home/stepan/mariadb/10.4/git/sql/net_serv.cc:1243
#12 0x0000559fee686a94 in my_net_read_packet (net=0x7f5d1000a338, read_from_server=1 '\001') at /home/stepan/mariadb/10.4/git/sql/net_serv.cc:1227
#13 0x0000559fee19d9de in do_command (thd=0x7f5d1000a040) at /home/stepan/mariadb/10.4/git/sql/sql_parse.cc:1232
#14 0x0000559fee327d51 in do_handle_one_connection (connect=0x559ff0cff900) at /home/stepan/mariadb/10.4/git/sql/sql_connect.cc:1412
#15 0x0000559fee327a80 in handle_one_connection (arg=0x559ff0cff900) at /home/stepan/mariadb/10.4/git/sql/sql_connect.cc:1316
#16 0x0000559feed274a5 in pfs_spawn_thread (arg=0x559ff0d01c90) at /home/stepan/mariadb/10.4/git/storage/perfschema/pfs.cc:1862
#17 0x00007f5d4b49ddd5 in start_thread () from /lib64/libpthread.so.0
#18 0x00007f5d4985aead in clone () from /lib64/libc.so.6
 
Compressed file /home/stepan/mariadb/10.4/git/mysql-test/var/4/log/galera_sr.galera_sr_cc_master-innodb/mysqld.2/data/core.12605
 - saving '/home/stepan/mariadb/10.4/git/mysql-test/var/4/log/galera_sr.galera_sr_cc_master-innodb/' to '/home/stepan/mariadb/10.4/git/mysql-test/var/log/galera_sr.galera_sr_cc_master-innodb/'
***Warnings generated in error logs during shutdown after running tests: galera_sr.GCF-627 galera_sr.GCF-900 galera_sr.GCF-889 galera_sr.galera_sr_cc_master galera_sr.galera_sr_blob galera_sr.GCF-851 galera_sr.GCF-867 galera_sr.galera_sr_bf_abort galera_sr.GCF-845
 
mysqld: /home/stepan/mariadb/10.4/git/include/mysql/psi/mysql_socket.h:779: inline_mysql_socket_recv: Assertion `mysql_socket.fd != -1' failed.
Attempting backtrace. You can use the following information to find out

All logs: 191025_SSD_galera_sr_cc_master.zip.



 Comments   
Comment by Stepan Patryshev (Inactive) [ 2019-12-23 ]

It's still reproduced on MariaDB Version 10.4.12-MariaDB-debug (088de81).

Comment by Stepan Patryshev (Inactive) [ 2020-01-22 ]

Decreased the priority since it failed sporadically only on local runs.

Comment by Stepan Patryshev (Inactive) [ 2020-02-10 ]

Closing since it does not fail on CI.

Generated at Thu Feb 08 09:03:02 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.