[MDEV-32115] Assertion `!recv_no_log_write' failed in bool log_checkpoint(bool) Created: 2023-09-07  Updated: 2023-11-04

Status: Open
Project: MariaDB Server
Component/s: Galera, Galera SST
Affects Version/s: 10.4
Fix Version/s: 10.4

Type: Bug Priority: Major
Reporter: Ramesh Sivaraman Assignee: Jan Lindström
Resolution: Unresolved Votes: 1
Labels: None

Attachments: File 10.4-recv_no_log_write.patch    
Issue Links:
Relates
relates to MDEV-24845 Oddities around innodb_fatal_semaphor... Closed

 Description   

The crash is sporadic and there is no reproducible test case yet.

#11 0x00007fb13562271b in __assert_fail_base (fmt=0x7fb1357bac30 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x55ec8b2db941 "!recv_no_log_write", file=0x55ec8b2db410 "/home/jan/mysql/10.4/storage/innobase/log/log0log.cc", line=1483, function=<optimized out>) at ./assert/assert.c:92
#12 0x00007fb135633596 in __GI___assert_fail (assertion=0x55ec8b2db941 "!recv_no_log_write", file=0x55ec8b2db410 "/home/jan/mysql/10.4/storage/innobase/log/log0log.cc", line=1483, function=0x55ec8b2dbfbb "bool log_checkpoint(bool)") at ./assert/assert.c:101
#13 0x000055ec8a9f56bd in log_checkpoint (sync=true) at /home/jan/mysql/10.4/storage/innobase/log/log0log.cc:1483
#14 0x000055ec8a9f5a6c in log_make_checkpoint () at /home/jan/mysql/10.4/storage/innobase/log/log0log.cc:1550
#15 0x000055ec8a9118a9 in sst_disable_innodb_writes () at /home/jan/mysql/10.4/storage/innobase/handler/ha_innodb.cc:1890
#16 0x000055ec8a9119ac in innodb_disable_internal_writes (disable=true) at /home/jan/mysql/10.4/storage/innobase/handler/ha_innodb.cc:1907
#17 0x000055ec8a55948a in plugin_disable_internal_writes (plugin=0x55ec8e15fa38, disable=0x7fb0da7fb8bc) at /home/jan/mysql/10.4/sql/handler.cc:881
#18 0x000055ec8a2124a3 in plugin_foreach_with_mask (thd=0x0, func=0x55ec8a559442 <plugin_disable_internal_writes(THD*, plugin_ref, void*)>, type=1, state_mask=8, arg=0x7fb0da7fb8bc) at /home/jan/mysql/10.4/sql/sql_plugin.cc:2477
#19 0x000055ec8a5594c7 in ha_disable_internal_writes (disable=true) at /home/jan/mysql/10.4/sql/handler.cc:888
#20 0x000055ec8a8d8ed5 in sst_flush_tables (thd=0x7fb0b8000d38) at /home/jan/mysql/10.4/sql/wsrep_sst.cc:1626
#21 0x000055ec8a8d9551 in sst_donor_thread (a=0x7fb1346fcac0) at /home/jan/mysql/10.4/sql/wsrep_sst.cc:1724



 Comments   
Comment by Marko Mäkelä [ 2023-09-07 ]

The Galera SST logic was rewritten in MDEV-24845. The failing assertion indicates that an unexpected log write took place.

It is possible that this failure is specific to MariaDB Server 10.4. Starting with 10.5, log checkpoints should be updated by the sole buf_flush_page_cleaner() thread.

Comment by Marko Mäkelä [ 2023-09-07 ]

janlindstrom, have you ever observed this failure in 10.5 or later major versions?

Can you try to patch the code so that the checks around recv_no_log_write are enabled in all builds, and then run some stress test on a RelWithDebInfo build? For faster execution time under rr record, use -DPLUGIN_PERFSCHEMA=NO.

Comment by Jan Lindström [ 2023-09-27 ]

marko If I correctly recall I have not seen this assertion in 10.5 or later major versions. Latest one I know (MariaDB Version 10.4.32-MariaDB-debug) is this:

 
galera.galera_ssl 'innodb'               w4 [ fail ]
        Test ended at 2023-09-27 01:51:11
 
CURRENT_TEST: galera.galera_ssl
 
 
Failed to start mysqld.2
mysqltest failed but provided no output
 
 
 - found 'core' (2/5)
Guessing that core was generated by '/home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/sql/mysqld'
Output from gdb follows. The first stack trace is from the failing thread.
The following stack traces are from all threads (so the failing one is
duplicated).
--------------------------
[New LWP 30578]
[New LWP 30392]
[New LWP 30396]
[New LWP 30394]
[New LWP 30393]
[New LWP 30417]
[New LWP 30410]
[New LWP 30399]
[New LWP 30411]
[New LWP 30397]
[New LWP 30419]
[New LWP 30398]
[New LWP 30395]
[New LWP 30400]
[New LWP 30401]
[New LWP 30422]
[New LWP 30404]
[New LWP 30412]
[New LWP 30416]
[New LWP 30403]
[New LWP 30418]
[New LWP 30415]
[New LWP 30577]
[New LWP 30405]
[New LWP 30413]
[New LWP 30420]
[New LWP 30576]
[New LWP 30425]
[New LWP 30402]
[New LWP 30421]
[New LWP 30391]
[New LWP 30409]
[New LWP 30414]
[New LWP 30408]
[New LWP 30407]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/sql/mysq'.
Program terminated with signal SIGABRT, Aborted.
#0  __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:57
57	../sysdeps/unix/sysv/linux/pthread_kill.c: No such file or directory.
[Current thread is 1 (Thread 0x7faab1ffb700 (LWP 30578))]
 
Thread 35 (Thread 0x7faade7fc700 (LWP 30407)):
#0  0x00007fab0a916065 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7faade7fbd30, expected=0, futex_word=0x55d8bf52a828) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  __pthread_cond_wait_common (abstime=0x7faade7fbd30, mutex=0x55d8bf52a7d8, cond=0x55d8bf52a800) at pthread_cond_wait.c:539
#2  __pthread_cond_timedwait (cond=0x55d8bf52a800, mutex=0x55d8bf52a7d8, abstime=0x7faade7fbd30) at pthread_cond_wait.c:667
#3  0x000055d8bce64d8c in os_event::timed_wait (this=0x55d8bf52a7c0, abstime=0x7faade7fbd30) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0event.cc:275
#4  0x000055d8bce6504b in os_event::wait_time_low (this=0x55d8bf52a7c0, time_in_usec=1000000, reset_sig_count=1) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0event.cc:385
#5  0x000055d8bce651b3 in os_event_wait_time_low (event=0x55d8bf52a7c0, time_in_usec=1000000, reset_sig_count=1) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0event.cc:485
#6  0x000055d8bce23d02 in lock_wait_timeout_thread () at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/lock/lock0wait.cc:527
#7  0x00007fab0a90f6db in start_thread (arg=0x7faade7fc700) at pthread_create.c:463
#8  0x00007fab09af561f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 34 (Thread 0x7faaddffb700 (LWP 30408)):
#0  0x00007fab0a916065 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7faaddffab70, expected=0, futex_word=0x55d8bf2f6a58) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  __pthread_cond_wait_common (abstime=0x7faaddffab70, mutex=0x55d8bf2f6a08, cond=0x55d8bf2f6a30) at pthread_cond_wait.c:539
#2  __pthread_cond_timedwait (cond=0x55d8bf2f6a30, mutex=0x55d8bf2f6a08, abstime=0x7faaddffab70) at pthread_cond_wait.c:667
#3  0x000055d8bce64d8c in os_event::timed_wait (this=0x55d8bf2f69f0, abstime=0x7faaddffab70) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0event.cc:275
#4  0x000055d8bce6504b in os_event::wait_time_low (this=0x55d8bf2f69f0, time_in_usec=1000000, reset_sig_count=1) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0event.cc:385
#5  0x000055d8bce651b3 in os_event_wait_time_low (event=0x55d8bf2f69f0, time_in_usec=1000000, reset_sig_count=1) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0event.cc:485
#6  0x000055d8bcf88892 in srv_error_monitor_thread () at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/srv/srv0srv.cc:1815
#7  0x00007fab0a90f6db in start_thread (arg=0x7faaddffb700) at pthread_create.c:463
#8  0x00007fab09af561f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 33 (Thread 0x7faaceffd700 (LWP 30414)):
#0  0x00007fab0a915ad3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55d8bf2f68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x55d8bf2f6888, cond=0x55d8bf2f68b0) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x55d8bf2f68b0, mutex=0x55d8bf2f6888) at pthread_cond_wait.c:655
#3  0x000055d8bce653ea in os_event::wait (this=0x55d8bf2f6870) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0event.cc:158
#4  0x000055d8bce64f26 in os_event::wait_low (this=0x55d8bf2f6870, reset_sig_count=25) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0event.cc:325
#5  0x000055d8bce651d8 in os_event_wait_low (event=0x55d8bf2f6870, reset_sig_count=0) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0event.cc:502
#6  0x000055d8bcf86407 in srv_resume_thread (slot=0x55d8bdea3480 <srv_sys+832>, sig_count=0, wait=true, timeout_usec=0) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/srv/srv0srv.cc:897
#7  0x000055d8bcf8a540 in srv_worker_thread (arg=0x0) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/srv/srv0srv.cc:2484
#8  0x00007fab0a90f6db in start_thread (arg=0x7faaceffd700) at pthread_create.c:463
#9  0x00007fab09af561f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 32 (Thread 0x7faadeffd700 (LWP 30409)):
#0  0x00007fab0a916065 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7faadeffcd10, expected=0, futex_word=0x55d8bf2f6ad8) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  __pthread_cond_wait_common (abstime=0x7faadeffcd10, mutex=0x55d8bf2f6a88, cond=0x55d8bf2f6ab0) at pthread_cond_wait.c:539
#2  __pthread_cond_timedwait (cond=0x55d8bf2f6ab0, mutex=0x55d8bf2f6a88, abstime=0x7faadeffcd10) at pthread_cond_wait.c:667
#3  0x000055d8bce64d8c in os_event::timed_wait (this=0x55d8bf2f6a70, abstime=0x7faadeffcd10) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0event.cc:275
#4  0x000055d8bce6504b in os_event::wait_time_low (this=0x55d8bf2f6a70, time_in_usec=5000000, reset_sig_count=1) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0event.cc:385
#5  0x000055d8bce651b3 in os_event_wait_time_low (event=0x55d8bf2f6a70, time_in_usec=5000000, reset_sig_count=1) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0event.cc:485
#6  0x000055d8bcf8834f in srv_monitor_thread () at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/srv/srv0srv.cc:1664
#7  0x00007fab0a90f6db in start_thread (arg=0x7faadeffd700) at pthread_create.c:463
#8  0x00007fab09af561f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 31 (Thread 0x7fab0bf5d200 (LWP 30391)):
#0  0x00007fab09ae8bb9 in __GI___poll (fds=0x7ffcb3bf4bb0, nfds=2, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x000055d8bc4f307d in handle_connections_sockets () at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/sql/mysqld.cc:6511
#2  0x000055d8bc4f22da in mysqld_main (argc=157, argv=0x55d8bf043820) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/sql/mysqld.cc:5947
#3  0x000055d8bc4e5fea in main (argc=23, argv=0x7ffcb3bf4eb8) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/sql/main.cc:25
 
Thread 30 (Thread 0x7fab04099700 (LWP 30421)):
#0  0x00007fab0a915ad3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55d8be06d46c <COND_manager+44>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x55d8be06d3a8 <LOCK_manager+40>, cond=0x55d8be06d440 <COND_manager>) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x55d8be06d440 <COND_manager>, mutex=0x55d8be06d3a8 <LOCK_manager+40>) at pthread_cond_wait.c:655
#3  0x000055d8bd25ebca in safe_cond_wait (cond=0x55d8be06d440 <COND_manager>, mp=0x55d8be06d380 <LOCK_manager>, file=0x55d8bd35cc20 "/home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/include/mysql/psi/mysql_thread.h", line=1174) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/mysys/thr_mutex.c:492
#4  0x000055d8bc626a59 in inline_mysql_cond_wait (that=0x55d8be06d440 <COND_manager>, mutex=0x55d8be06d380 <LOCK_manager>, src_file=0x55d8bd35cdc0 "/home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/sql/sql_manager.cc", src_line=102) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/include/mysql/psi/mysql_thread.h:1174
#5  0x000055d8bc626f63 in handle_manager (arg=0x0) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/sql/sql_manager.cc:102
#6  0x000055d8bccee829 in pfs_spawn_thread (arg=0x55d8bf5a18e0) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/perfschema/pfs.cc:1869
#7  0x00007fab0a90f6db in start_thread (arg=0x7fab04099700) at pthread_create.c:463
#8  0x00007fab09af561f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 29 (Thread 0x7faaf5078700 (LWP 30402)):
#0  0x00007fab0b6a763a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
#1  0x000055d8bce55453 in LinuxAIOHandler::collect (this=0x7faaf5077ad0) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0file.cc:1803
#2  0x000055d8bce55b62 in LinuxAIOHandler::poll (this=0x7faaf5077ad0, m1=0x7faaf5077b70, m2=0x7faaf5077b78, request=0x7faaf5077ba0) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0file.cc:1964
#3  0x000055d8bce55cd5 in os_aio_linux_handler (global_segment=3, m1=0x7faaf5077b70, m2=0x7faaf5077b78, request=0x7faaf5077ba0) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0file.cc:2018
#4  0x000055d8bce5aca3 in os_aio_handler (segment=3, m1=0x7faaf5077b70, m2=0x7faaf5077b78, request=0x7faaf5077ba0) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0file.cc:5296
#5  0x000055d8bd11e6d2 in fil_aio_wait (segment=3) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/fil/fil0fil.cc:4336
#6  0x000055d8bcf8d0db in io_handler_thread (arg=0x55d8be8ea198 <n+24>) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/srv/srv0start.cc:324
#7  0x00007fab0a90f6db in start_thread (arg=0x7faaf5078700) at pthread_create.c:463
#8  0x00007fab09af561f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 28 (Thread 0x7faafc0eb700 (LWP 30425)):
#0  0x00007fab0a916065 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7faafc0eac40, expected=0, futex_word=0x55d8be063c48 <COND_thread_cache+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  __pthread_cond_wait_common (abstime=0x7faafc0eac40, mutex=0x55d8be066548 <LOCK_thread_cache+40>, cond=0x55d8be063c20 <COND_thread_cache>) at pthread_cond_wait.c:539
#2  __pthread_cond_timedwait (cond=0x55d8be063c20 <COND_thread_cache>, mutex=0x55d8be066548 <LOCK_thread_cache+40>, abstime=0x7faafc0eac40) at pthread_cond_wait.c:667
#3  0x000055d8bd25ef8a in safe_cond_timedwait (cond=0x55d8be063c20 <COND_thread_cache>, mp=0x55d8be066520 <LOCK_thread_cache>, abstime=0x7faafc0eac40, file=0x55d8bd31c090 "/home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/include/mysql/psi/mysql_thread.h", line=1211) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/mysys/thr_mutex.c:546
#4  0x000055d8bc4e6a1a in inline_mysql_cond_timedwait (that=0x55d8be063c20 <COND_thread_cache>, mutex=0x55d8be066520 <LOCK_thread_cache>, abstime=0x7faafc0eac40, src_file=0x55d8bd31e2a8 "/home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/sql/mysqld.cc", src_line=2698) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/include/mysql/psi/mysql_thread.h:1211
#5  0x000055d8bc4ebb55 in cache_thread (thd=0x7faa9c000d50) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/sql/mysqld.cc:2697
#6  0x000055d8bc4ebdc5 in one_thread_per_connection_end (thd=0x7faa9c000d50, put_in_cache=true) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/sql/mysqld.cc:2783
#7  0x000055d8bc7c376d in do_handle_one_connection (connect=0x55d8bf5a1020) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/sql/sql_connect.cc:1431
#8  0x000055d8bc7c344d in handle_one_connection (arg=0x55d8bf5a1020) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/sql/sql_connect.cc:1324
#9  0x000055d8bccee829 in pfs_spawn_thread (arg=0x55d8bf714da0) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/perfschema/pfs.cc:1869
#10 0x00007fab0a90f6db in start_thread (arg=0x7faafc0eb700) at pthread_create.c:463
#11 0x00007fab09af561f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 27 (Thread 0x7faab2ffd700 (LWP 30576)):
#0  0x00007fab0a919474 in __libc_read (fd=58, buf=0x7faaec03b9f3, nbytes=5) at ../sysdeps/unix/sysv/linux/read.c:27
#1  0x00007fab0abd8c3e in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
#2  0x00007fab0abd407a in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
#3  0x00007fab0abd2f13 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
#4  0x00007fab0abd34c3 in BIO_read () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
#5  0x00007fab0b014913 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1
#6  0x00007fab0b018fbd in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1
#7  0x00007fab0b0166c2 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1
#8  0x00007fab0b01d995 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1
#9  0x00007fab0b028252 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1
#10 0x00007fab0b0283c9 in SSL_read_ex () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1
#11 0x00007fab04c37914 in AsioSslStreamEngine::do_read (this=0x7faaec0342c0, buf=0x7faab2ffc76f, max_count=1) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/galera/galerautils/src/gu_asio_stream_engine.cpp:203
#12 0x00007fab04c377e6 in AsioSslStreamEngine::read (this=0x7faaec0342c0, buf=0x7faab2ffc76f, max_count=1) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/galera/galerautils/src/gu_asio_stream_engine.cpp:172
#13 0x00007fab04c05e21 in gu::AsioStreamReact::read (this=0x7faaec031220, buf=...) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/galera/galerautils/src/gu_asio_stream_react.cpp:236
#14 0x00007fab04a7fbe2 in send_eof (p=..., socket=...) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/galera/galera/src/ist.cpp:640
#15 0x00007fab04a803e1 in galera::ist::Sender::send (this=0x7faaec02a820, first=3, last=2, preload_start=2) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/galera/galera/src/ist.cpp:735
#16 0x00007fab04a80763 in run_async_sender (arg=0x7faaec02a820) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/galera/galera/src/ist.cpp:760
#17 0x00007fab0a90f6db in start_thread (arg=0x7faab2ffd700) at pthread_create.c:463
#18 0x00007fab09af561f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 26 (Thread 0x7faab37fe700 (LWP 30420)):
#0  0x00007fab0a915ad3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7faab37fdc88) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7faab37fdcc8, cond=0x7faab37fdc60) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7faab37fdc60, mutex=0x7faab37fdcc8) at pthread_cond_wait.c:655
#3  0x000055d8bd25ebca in safe_cond_wait (cond=0x7faab37fdc60, mp=0x7faab37fdca0, file=0x55d8bd5dd100 "/home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/include/mysql/psi/mysql_thread.h", line=1185) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/mysys/thr_mutex.c:492
#4  0x000055d8bcd4bbe5 in inline_mysql_cond_wait (that=0x7faab37fdc60, mutex=0x7faab37fdca0, src_file=0x55d8bd5df870 "/home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/handler/ha_innodb.cc", src_line=310) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/include/mysql/psi/mysql_thread.h:1185
#5  0x000055d8bcd516a4 in thd_destructor_proxy () at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/handler/ha_innodb.cc:310
#6  0x000055d8bccee829 in pfs_spawn_thread (arg=0x55d8bf4a8840) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/perfschema/pfs.cc:1869
#7  0x00007fab0a90f6db in start_thread (arg=0x7faab37fe700) at pthread_create.c:463
#8  0x00007fab09af561f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 25 (Thread 0x7faacf7fe700 (LWP 30413)):
#0  0x00007fab0a915ad3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55d8bf2f685c) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x55d8bf2f6808, cond=0x55d8bf2f6830) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x55d8bf2f6830, mutex=0x55d8bf2f6808) at pthread_cond_wait.c:655
#3  0x000055d8bce653ea in os_event::wait (this=0x55d8bf2f67f0) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0event.cc:158
#4  0x000055d8bce64f26 in os_event::wait_low (this=0x55d8bf2f67f0, reset_sig_count=12) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0event.cc:325
#5  0x000055d8bce651d8 in os_event_wait_low (event=0x55d8bf2f67f0, reset_sig_count=12) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0event.cc:502
#6  0x000055d8bcf86407 in srv_resume_thread (slot=0x55d8bdea3388 <srv_sys+584>, sig_count=12, wait=true, timeout_usec=0) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/srv/srv0srv.cc:897
#7  0x000055d8bcf8aa2e in srv_purge_coordinator_suspend (slot=0x55d8bdea3388 <srv_sys+584>, rseg_history_len=20) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/srv/srv0srv.cc:2628
#8  0x000055d8bcf8ad70 in srv_purge_coordinator_thread (arg=0x0) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/srv/srv0srv.cc:2709
#9  0x00007fab0a90f6db in start_thread (arg=0x7faacf7fe700) at pthread_create.c:463
#10 0x00007fab09af561f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 24 (Thread 0x7faadf7fe700 (LWP 30405)):
#0  0x00007fab0a916065 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7faadf7fdac0, expected=0, futex_word=0x55d8bf2f6bdc) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  __pthread_cond_wait_common (abstime=0x7faadf7fdac0, mutex=0x55d8bf2f6b88, cond=0x55d8bf2f6bb0) at pthread_cond_wait.c:539
#2  __pthread_cond_timedwait (cond=0x55d8bf2f6bb0, mutex=0x55d8bf2f6b88, abstime=0x7faadf7fdac0) at pthread_cond_wait.c:667
#3  0x000055d8bce64d8c in os_event::timed_wait (this=0x55d8bf2f6b70, abstime=0x7faadf7fdac0) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0event.cc:275
#4  0x000055d8bce6504b in os_event::wait_time_low (this=0x55d8bf2f6b70, time_in_usec=972000, reset_sig_count=2) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0event.cc:385
#5  0x000055d8bce651b3 in os_event_wait_time_low (event=0x55d8bf2f6b70, time_in_usec=972000, reset_sig_count=2) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0event.cc:485
#6  0x000055d8bd09a3c2 in pc_sleep_if_needed (next_loop_time=56783714719, sig_count=2, cur_time=56783713747) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/buf/buf0flu.cc:2611
#7  0x000055d8bd09b5a7 in buf_flush_page_cleaner_coordinator () at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/buf/buf0flu.cc:3069
#8  0x00007fab0a90f6db in start_thread (arg=0x7faadf7fe700) at pthread_create.c:463
#9  0x00007fab09af561f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 23 (Thread 0x7faab27fc700 (LWP 30577)):
#0  0x00007fab0a916065 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7faab27fbd40, expected=0, futex_word=0x55d8be8a5668 <COND_wsrep_donor_monitor+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  __pthread_cond_wait_common (abstime=0x7faab27fbd40, mutex=0x55d8be8a5568 <LOCK_wsrep_donor_monitor+40>, cond=0x55d8be8a5640 <COND_wsrep_donor_monitor>) at pthread_cond_wait.c:539
#2  __pthread_cond_timedwait (cond=0x55d8be8a5640 <COND_wsrep_donor_monitor>, mutex=0x55d8be8a5568 <LOCK_wsrep_donor_monitor+40>, abstime=0x7faab27fbd40) at pthread_cond_wait.c:667
#3  0x000055d8bd25ef8a in safe_cond_timedwait (cond=0x55d8be8a5640 <COND_wsrep_donor_monitor>, mp=0x55d8be8a5540 <LOCK_wsrep_donor_monitor>, abstime=0x7faab27fbd40, file=0x55d8bd5d1a40 "/home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/include/mysql/psi/mysql_thread.h", line=1211) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/mysys/thr_mutex.c:546
#4  0x000055d8bcd1a314 in inline_mysql_cond_timedwait (that=0x55d8be8a5640 <COND_wsrep_donor_monitor>, mutex=0x55d8be8a5540 <LOCK_wsrep_donor_monitor>, abstime=0x7faab27fbd40, src_file=0x55d8bd5d1d08 "/home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/sql/wsrep_sst.cc", src_line=118) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/include/mysql/psi/mysql_thread.h:1211
#5  0x000055d8bcd1a6ab in wsrep_sst_donor_monitor_thread (arg=0x0) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/sql/wsrep_sst.cc:118
#6  0x000055d8bccee829 in pfs_spawn_thread (arg=0x55d8bf1d92e0) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/perfschema/pfs.cc:1869
#7  0x00007fab0a90f6db in start_thread (arg=0x7faab27fc700) at pthread_create.c:463
#8  0x00007fab09af561f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 22 (Thread 0x7faace7fc700 (LWP 30415)):
#0  0x00007fab0a915ad3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55d8bf2f695c) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x55d8bf2f6908, cond=0x55d8bf2f6930) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x55d8bf2f6930, mutex=0x55d8bf2f6908) at pthread_cond_wait.c:655
#3  0x000055d8bce653ea in os_event::wait (this=0x55d8bf2f68f0) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0event.cc:158
#4  0x000055d8bce64f26 in os_event::wait_low (this=0x55d8bf2f68f0, reset_sig_count=24) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0event.cc:325
#5  0x000055d8bce651d8 in os_event_wait_low (event=0x55d8bf2f68f0, reset_sig_count=0) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0event.cc:502
#6  0x000055d8bcf86407 in srv_resume_thread (slot=0x55d8bdea3578 <srv_sys+1080>, sig_count=0, wait=true, timeout_usec=0) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/srv/srv0srv.cc:897
#7  0x000055d8bcf8a540 in srv_worker_thread (arg=0x0) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/srv/srv0srv.cc:2484
#8  0x00007fab0a90f6db in start_thread (arg=0x7faace7fc700) at pthread_create.c:463
#9  0x00007fab09af561f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 21 (Thread 0x7faaccff9700 (LWP 30418)):
#0  0x00007fab0a919d50 in __GI___nanosleep (requested_time=0x7faaccff86d0, remaining=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:28
#1  0x000055d8bce6575b in os_thread_sleep (tm=1000000) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0thread.cc:231
#2  0x000055d8bd062935 in btr_defragment_thread () at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/btr/btr0defragment.cc:705
#3  0x00007fab0a90f6db in start_thread (arg=0x7faaccff9700) at pthread_create.c:463
#4  0x00007fab09af561f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 20 (Thread 0x7faaf4877700 (LWP 30403)):
#0  0x00007fab0b6a763a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
#1  0x000055d8bce55453 in LinuxAIOHandler::collect (this=0x7faaf4876ad0) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0file.cc:1803
#2  0x000055d8bce55b62 in LinuxAIOHandler::poll (this=0x7faaf4876ad0, m1=0x7faaf4876b70, m2=0x7faaf4876b78, request=0x7faaf4876ba0) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0file.cc:1964
#3  0x000055d8bce55cd5 in os_aio_linux_handler (global_segment=4, m1=0x7faaf4876b70, m2=0x7faaf4876b78, request=0x7faaf4876ba0) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0file.cc:2018
#4  0x000055d8bce5aca3 in os_aio_handler (segment=4, m1=0x7faaf4876b70, m2=0x7faaf4876b78, request=0x7faaf4876ba0) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0file.cc:5296
#5  0x000055d8bd11e6d2 in fil_aio_wait (segment=4) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/fil/fil0fil.cc:4336
#6  0x000055d8bcf8d0db in io_handler_thread (arg=0x55d8be8ea1a0 <n+32>) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/srv/srv0start.cc:324
#7  0x00007fab0a90f6db in start_thread (arg=0x7faaf4877700) at pthread_create.c:463
#8  0x00007fab09af561f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 19 (Thread 0x7faacdffb700 (LWP 30416)):
#0  0x00007fab0a915ad3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55d8bf2f69dc) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x55d8bf2f6988, cond=0x55d8bf2f69b0) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x55d8bf2f69b0, mutex=0x55d8bf2f6988) at pthread_cond_wait.c:655
#3  0x000055d8bce653ea in os_event::wait (this=0x55d8bf2f6970) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0event.cc:158
#4  0x000055d8bce64f26 in os_event::wait_low (this=0x55d8bf2f6970, reset_sig_count=24) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0event.cc:325
#5  0x000055d8bce651d8 in os_event_wait_low (event=0x55d8bf2f6970, reset_sig_count=0) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0event.cc:502
#6  0x000055d8bcf86407 in srv_resume_thread (slot=0x55d8bdea3670 <srv_sys+1328>, sig_count=0, wait=true, timeout_usec=0) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/srv/srv0srv.cc:897
#7  0x000055d8bcf8a540 in srv_worker_thread (arg=0x0) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/srv/srv0srv.cc:2484
#8  0x00007fab0a90f6db in start_thread (arg=0x7faacdffb700) at pthread_create.c:463
#9  0x00007fab09af561f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 18 (Thread 0x7faacffff700 (LWP 30412)):
#0  0x00007fab0a915ad3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55d8bf48847c) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x55d8bf488428, cond=0x55d8bf488450) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x55d8bf488450, mutex=0x55d8bf488428) at pthread_cond_wait.c:655
#3  0x000055d8bce653ea in os_event::wait (this=0x55d8bf488410) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0event.cc:158
#4  0x000055d8bce64f26 in os_event::wait_low (this=0x55d8bf488410, reset_sig_count=4) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0event.cc:325
#5  0x000055d8bce651d8 in os_event_wait_low (event=0x55d8bf488410, reset_sig_count=4) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0event.cc:502
#6  0x000055d8bcf96554 in sync_array_wait_event (arr=0x55d8bf1f7a70, cell=@0x7faacfffeb90: 0x55d8bf2ce360) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/sync/sync0arr.cc:471
#7  0x000055d8bcf99617 in rw_lock_s_lock_spin (lock=0x55d8bdea0138 <log_sys+824>, pass=0, file_name=0x55d8bd618200 "/home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/log/log0log.cc", line=1392) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/sync/sync0rw.cc:367
#8  0x000055d8bce24808 in rw_lock_s_lock_func (lock=0x55d8bdea0138 <log_sys+824>, pass=0, file_name=0x55d8bd618200 "/home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/log/log0log.cc", line=1392) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/include/sync0rw.inl:285
#9  0x000055d8bce24dc4 in pfs_rw_lock_s_lock_func (lock=0x55d8bdea0138 <log_sys+824>, pass=0, file_name=0x55d8bd618200 "/home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/log/log0log.cc", line=1392) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/include/sync0rw.inl:633
#10 0x000055d8bce29b59 in log_write_checkpoint_info (sync=true, end_lsn=78266) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/log/log0log.cc:1392
#11 0x000055d8bce2a184 in log_checkpoint (sync=true) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/log/log0log.cc:1536
#12 0x000055d8bcf895a1 in srv_master_do_active_tasks () at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/srv/srv0srv.cc:2158
#13 0x000055d8bcf89ecd in srv_master_thread (arg=0x0) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/srv/srv0srv.cc:2347
#14 0x00007fab0a90f6db in start_thread (arg=0x7faacffff700) at pthread_create.c:463
#15 0x00007fab09af561f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 17 (Thread 0x7faadffff700 (LWP 30404)):
#0  0x00007fab0b6a763a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
#1  0x000055d8bce55453 in LinuxAIOHandler::collect (this=0x7faadfffead0) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0file.cc:1803
#2  0x000055d8bce55b62 in LinuxAIOHandler::poll (this=0x7faadfffead0, m1=0x7faadfffeb70, m2=0x7faadfffeb78, request=0x7faadfffeba0) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0file.cc:1964
#3  0x000055d8bce55cd5 in os_aio_linux_handler (global_segment=5, m1=0x7faadfffeb70, m2=0x7faadfffeb78, request=0x7faadfffeba0) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0file.cc:2018
#4  0x000055d8bce5aca3 in os_aio_handler (segment=5, m1=0x7faadfffeb70, m2=0x7faadfffeb78, request=0x7faadfffeba0) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0file.cc:5296
#5  0x000055d8bd11e6d2 in fil_aio_wait (segment=5) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/fil/fil0fil.cc:4336
#6  0x000055d8bcf8d0db in io_handler_thread (arg=0x55d8be8ea1a8 <n+40>) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/srv/srv0start.cc:324
#7  0x00007fab0a90f6db in start_thread (arg=0x7faadffff700) at pthread_create.c:463
#8  0x00007fab09af561f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 16 (Thread 0x7fab0404e700 (LWP 30422)):
#0  0x00007fab09a1425c in __GI___sigtimedwait (set=<optimized out>, info=0x7fab0404dbf0, timeout=0x0) at ../sysdeps/unix/sysv/linux/sigtimedwait.c:42
#1  0x000055d8bc4e61da in my_sigwait (set=0x7fab0404dcd0, sig=0x7fab0404dc94, code=0x7fab0404dc98) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/include/my_pthread.h:196
#2  0x000055d8bc4eca77 in signal_hand (arg=0x0) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/sql/mysqld.cc:3254
#3  0x000055d8bccee829 in pfs_spawn_thread (arg=0x55d8bf5c6090) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/perfschema/pfs.cc:1869
#4  0x00007fab0a90f6db in start_thread (arg=0x7fab0404e700) at pthread_create.c:463
#5  0x00007fab09af561f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 15 (Thread 0x7faaf65fd700 (LWP 30401)):
#0  0x00007fab0b6a763a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
#1  0x000055d8bce55453 in LinuxAIOHandler::collect (this=0x7faaf65fcad0) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0file.cc:1803
#2  0x000055d8bce55b62 in LinuxAIOHandler::poll (this=0x7faaf65fcad0, m1=0x7faaf65fcb70, m2=0x7faaf65fcb78, request=0x7faaf65fcba0) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0file.cc:1964
#3  0x000055d8bce55cd5 in os_aio_linux_handler (global_segment=2, m1=0x7faaf65fcb70, m2=0x7faaf65fcb78, request=0x7faaf65fcba0) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0file.cc:2018
#4  0x000055d8bce5aca3 in os_aio_handler (segment=2, m1=0x7faaf65fcb70, m2=0x7faaf65fcb78, request=0x7faaf65fcba0) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0file.cc:5296
#5  0x000055d8bd11e6d2 in fil_aio_wait (segment=2) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/fil/fil0fil.cc:4336
#6  0x000055d8bcf8d0db in io_handler_thread (arg=0x55d8be8ea190 <n+16>) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/srv/srv0start.cc:324
#7  0x00007fab0a90f6db in start_thread (arg=0x7faaf65fd700) at pthread_create.c:463
#8  0x00007fab09af561f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 14 (Thread 0x7faaf6dfe700 (LWP 30400)):
#0  0x00007fab0a915ad3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55d8bf4880f8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x55d8bf4880a8, cond=0x55d8bf4880d0) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x55d8bf4880d0, mutex=0x55d8bf4880a8) at pthread_cond_wait.c:655
#3  0x000055d8bce653ea in os_event::wait (this=0x55d8bf488090) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0event.cc:158
#4  0x000055d8bce64f26 in os_event::wait_low (this=0x55d8bf488090, reset_sig_count=1) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0event.cc:325
#5  0x000055d8bce651d8 in os_event_wait_low (event=0x55d8bf488090, reset_sig_count=1) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0event.cc:502
#6  0x000055d8bcf96554 in sync_array_wait_event (arr=0x55d8bf1f7a70, cell=@0x7faaf6dfda98: 0x55d8bf2ce320) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/sync/sync0arr.cc:471
#7  0x000055d8bcd860a6 in TTASEventMutex<GenericPolicy>::enter (this=0x55d8bde9fe40 <log_sys+64>, max_spins=60, max_delay=4, filename=0x55d8bd618200 "/home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/log/log0log.cc", line=1282) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/include/ib0mutex.h:477
#8  0x000055d8bcd8402d in PolicyMutex<TTASEventMutex<GenericPolicy> >::enter (this=0x55d8bde9fe40 <log_sys+64>, n_spins=30, n_delay=4, name=0x55d8bd618200 "/home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/log/log0log.cc", line=1282) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/include/ib0mutex.h:594
#9  0x000055d8bce2909f in log_t::complete_checkpoint (this=0x55d8bde9fe00 <log_sys>) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/log/log0log.cc:1282
#10 0x000055d8bd11e97e in fil_aio_wait (segment=1) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/fil/fil0fil.cc:4386
#11 0x000055d8bcf8d0db in io_handler_thread (arg=0x55d8be8ea188 <n+8>) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/srv/srv0start.cc:324
#12 0x00007fab0a90f6db in start_thread (arg=0x7faaf6dfe700) at pthread_create.c:463
#13 0x00007fab09af561f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 13 (Thread 0x7faaf5879700 (LWP 30395)):
#0  0x00007fab0a915ad3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55d8bf1f0f44) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x55d8bf1f0ec0, cond=0x55d8bf1f0f18) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x55d8bf1f0f18, mutex=0x55d8bf1f0ec0) at pthread_cond_wait.c:655
#3  0x00007fab04a1c311 in gu_cond_wait_SYS (cond=0x55d8bf1f0f18, mutex=0x55d8bf1f0ec0) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/galera/galerautils/src/gu_threads.h:251
#4  0x00007fab04a336dc in gu::Lock::wait (this=0x7faaf5878940, cond=...) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/galera/galerautils/src/gu_lock.hpp:46
#5  0x00007fab04abd59e in RecvBuf::front (this=0x55d8bf1f0ec0, timeout=...) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/galera/gcs/src/gcs_gcomm.cpp:116
#6  0x00007fab04ab8c73 in gcomm_recv (backend=0x55d8bf1b9c58, msg=0x55d8bf1b9a98, timeout=9223372035999999999) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/galera/gcs/src/gcs_gcomm.cpp:614
#7  0x00007fab04a9ff15 in core_msg_recv (backend=0x55d8bf1b9c58, recv_msg=0x55d8bf1b9a98, timeout=9223372035999999999) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/galera/gcs/src/gcs_core.cpp:484
#8  0x00007fab04aa2492 in gcs_core_recv (conn=0x55d8bf1b9a20, recv_act=0x7faaf5878d10, timeout=9223372035999999999) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/galera/gcs/src/gcs_core.cpp:1205
#9  0x00007fab04a96e28 in gcs_recv_thread (arg=0x55d8bf1bc890) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/galera/gcs/src/gcs.cpp:1454
#10 0x00007fab0a90f6db in start_thread (arg=0x7faaf5879700) at pthread_create.c:463
#11 0x00007fab09af561f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 12 (Thread 0x7faafceed700 (LWP 30398)):
#0  0x00007fab0a916065 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7faafceecca0, expected=0, futex_word=0x55d8be89b768 <COND_checkpoint+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  __pthread_cond_wait_common (abstime=0x7faafceecca0, mutex=0x55d8be89b6a8 <LOCK_checkpoint+40>, cond=0x55d8be89b740 <COND_checkpoint>) at pthread_cond_wait.c:539
#2  __pthread_cond_timedwait (cond=0x55d8be89b740 <COND_checkpoint>, mutex=0x55d8be89b6a8 <LOCK_checkpoint+40>, abstime=0x7faafceecca0) at pthread_cond_wait.c:667
#3  0x000055d8bd25ef8a in safe_cond_timedwait (cond=0x55d8be89b740 <COND_checkpoint>, mp=0x55d8be89b680 <LOCK_checkpoint>, abstime=0x7faafceecca0, file=0x55d8bd587d00 "/home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/include/mysql/psi/mysql_thread.h", line=1211) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/mysys/thr_mutex.c:546
#4  0x000055d8bcc06209 in inline_mysql_cond_timedwait (that=0x55d8be89b740 <COND_checkpoint>, mutex=0x55d8be89b680 <LOCK_checkpoint>, abstime=0x7faafceecca0, src_file=0x55d8bd587d68 "/home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/maria/ma_servicethread.c", src_line=116) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/include/mysql/psi/mysql_thread.h:1211
#5  0x000055d8bcc0674e in my_service_thread_sleep (control=0x55d8bde90620 <checkpoint_control>, sleep_time=29000000000) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/maria/ma_servicethread.c:115
#6  0x000055d8bcbfa55a in ma_checkpoint_background (arg=0x1e) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/maria/ma_checkpoint.c:707
#7  0x000055d8bccee829 in pfs_spawn_thread (arg=0x55d8bf2af620) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/perfschema/pfs.cc:1869
#8  0x00007fab0a90f6db in start_thread (arg=0x7faafceed700) at pthread_create.c:463
#9  0x00007fab09af561f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 11 (Thread 0x7faab3fff700 (LWP 30419)):
#0  0x00007fab0a915ad3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55d8bf2f6c58) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x55d8bf2f6c08, cond=0x55d8bf2f6c30) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x55d8bf2f6c30, mutex=0x55d8bf2f6c08) at pthread_cond_wait.c:655
#3  0x000055d8bce653ea in os_event::wait (this=0x55d8bf2f6bf0) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0event.cc:158
#4  0x000055d8bce64f26 in os_event::wait_low (this=0x55d8bf2f6bf0, reset_sig_count=1) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0event.cc:325
#5  0x000055d8bce651d8 in os_event_wait_low (event=0x55d8bf2f6bf0, reset_sig_count=0) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0event.cc:502
#6  0x000055d8bd0734c5 in buf_resize_thread () at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/buf/buf0buf.cc:2965
#7  0x00007fab0a90f6db in start_thread (arg=0x7faab3fff700) at pthread_create.c:463
#8  0x00007fab09af561f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 10 (Thread 0x7fab042fa700 (LWP 30397)):
#0  0x00007fab0a915ad3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fab044470f8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7fab04447090, cond=0x7fab044470d0) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7fab044470d0, mutex=0x7fab04447090) at pthread_cond_wait.c:655
#3  0x00007fab04c3e415 in gu_cond_wait_SYS (cond=0x7fab044470d0, mutex=0x7fab04447090) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/galera/galerautils/src/gu_threads.h:251
#4  0x00007fab04c3ebc6 in fifo_lock_get (q=0x7fab04447010) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/galera/galerautils/src/gu_fifo.c:262
#5  0x00007fab04c3ee0f in gu_fifo_get_head (q=0x7fab04447010, err=0x7fab042f976c) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/galera/galerautils/src/gu_fifo.c:330
#6  0x00007fab04a993f3 in gcs_recv (conn=0x55d8bf1bc890, action=0x7fab042f9810) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/galera/gcs/src/gcs.cpp:2107
#7  0x00007fab04a356e5 in galera::Gcs::recv (this=0x55d8bf18c140, act=...) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/galera/galera/src/galera_gcs.hpp:133
#8  0x00007fab04a7a269 in galera::GcsActionSource::process (this=0x55d8bf1b9e80, recv_ctx=0x7fab042f9c80, exit_loop=@0x7fab042f99fa: false) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/galera/galera/src/gcs_action_source.cpp:165
#9  0x00007fab04a1eee2 in galera::ReplicatorSMM::async_recv (this=0x55d8bf18b7e0, recv_ctx=0x7fab042f9c80) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/galera/galera/src/replicator_smm.cpp:402
#10 0x00007fab049f9ae5 in galera_recv (gh=0x55d8bf174d40, recv_ctx=0x7fab042f9c80) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/galera/galera/src/wsrep_provider.cpp:264
#11 0x000055d8bd30fbae in wsrep::wsrep_provider_v26::run_applier (this=0x55d8bf0a31c0, applier_ctx=0x7fab042f9c80) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/wsrep-lib/src/wsrep_provider_v26.cpp:858
#12 0x000055d8bcd26ac4 in wsrep_replication_process (thd=0x7faaec000d50, arg=0x55d8bf207720) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/sql/wsrep_thd.cc:57
#13 0x000055d8bcd17284 in start_wsrep_THD (arg=0x55d8bf207720) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/sql/wsrep_mysqld.cc:3042
#14 0x000055d8bccee829 in pfs_spawn_thread (arg=0x55d8bf1d92e0) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/perfschema/pfs.cc:1869
#15 0x00007fab0a90f6db in start_thread (arg=0x7fab042fa700) at pthread_create.c:463
#16 0x00007fab09af561f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 9 (Thread 0x7faadcff9700 (LWP 30411)):
#0  0x00007fab0a916065 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7faadcff8b20, expected=0, futex_word=0x55d8bf48f3f8) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  __pthread_cond_wait_common (abstime=0x7faadcff8b20, mutex=0x55d8bf48f3a8, cond=0x55d8bf48f3d0) at pthread_cond_wait.c:539
#2  __pthread_cond_timedwait (cond=0x55d8bf48f3d0, mutex=0x55d8bf48f3a8, abstime=0x7faadcff8b20) at pthread_cond_wait.c:667
#3  0x000055d8bce64d8c in os_event::timed_wait (this=0x55d8bf48f390, abstime=0x7faadcff8b20) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0event.cc:275
#4  0x000055d8bce6504b in os_event::wait_time_low (this=0x55d8bf48f390, time_in_usec=5000000, reset_sig_count=1) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0event.cc:385
#5  0x000055d8bce651b3 in os_event_wait_time_low (event=0x55d8bf48f390, time_in_usec=5000000, reset_sig_count=1) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0event.cc:485
#6  0x000055d8bcff2fda in ib_wqueue_timedwait (wq=0x55d8bf48f1e0, wait_in_usecs=5000000) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/ut/ut0wqueue.cc:158
#7  0x000055d8bd16be41 in fts_optimize_thread (arg=0x55d8bf48f1e0) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/fts/fts0opt.cc:2849
#8  0x00007fab0a90f6db in start_thread (arg=0x7faadcff9700) at pthread_create.c:463
#9  0x00007fab09af561f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 8 (Thread 0x7faaf75ff700 (LWP 30399)):
#0  0x00007fab0b6a763a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
#1  0x000055d8bce55453 in LinuxAIOHandler::collect (this=0x7faaf75fead0) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0file.cc:1803
#2  0x000055d8bce55b62 in LinuxAIOHandler::poll (this=0x7faaf75fead0, m1=0x7faaf75feb70, m2=0x7faaf75feb78, request=0x7faaf75feba0) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0file.cc:1964
#3  0x000055d8bce55cd5 in os_aio_linux_handler (global_segment=0, m1=0x7faaf75feb70, m2=0x7faaf75feb78, request=0x7faaf75feba0) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0file.cc:2018
#4  0x000055d8bce5aca3 in os_aio_handler (segment=0, m1=0x7faaf75feb70, m2=0x7faaf75feb78, request=0x7faaf75feba0) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0file.cc:5296
#5  0x000055d8bd11e6d2 in fil_aio_wait (segment=0) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/fil/fil0fil.cc:4336
#6  0x000055d8bcf8d0db in io_handler_thread (arg=0x55d8be8ea180 <n>) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/srv/srv0start.cc:324
#7  0x00007fab0a90f6db in start_thread (arg=0x7faaf75ff700) at pthread_create.c:463
#8  0x00007fab09af561f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 7 (Thread 0x7faadd7fa700 (LWP 30410)):
#0  0x00007fab0a916065 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7faadd7f9d50, expected=0, futex_word=0x55d8bf55bfb8) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  __pthread_cond_wait_common (abstime=0x7faadd7f9d50, mutex=0x55d8bf55bf68, cond=0x55d8bf55bf90) at pthread_cond_wait.c:539
#2  __pthread_cond_timedwait (cond=0x55d8bf55bf90, mutex=0x55d8bf55bf68, abstime=0x7faadd7f9d50) at pthread_cond_wait.c:667
#3  0x000055d8bce64d8c in os_event::timed_wait (this=0x55d8bf55bf50, abstime=0x7faadd7f9d50) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0event.cc:275
#4  0x000055d8bce6504b in os_event::wait_time_low (this=0x55d8bf55bf50, time_in_usec=10000000, reset_sig_count=1) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0event.cc:385
#5  0x000055d8bce651b3 in os_event_wait_time_low (event=0x55d8bf55bf50, time_in_usec=10000000, reset_sig_count=0) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0event.cc:485
#6  0x000055d8bd106131 in dict_stats_thread () at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/dict/dict0stats_bg.cc:492
#7  0x00007fab0a90f6db in start_thread (arg=0x7faadd7fa700) at pthread_create.c:463
#8  0x00007fab09af561f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 6 (Thread 0x7faacd7fa700 (LWP 30417)):
#0  0x00007fab0a915ad3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55d8bf2f6b58) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x55d8bf2f6b08, cond=0x55d8bf2f6b30) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x55d8bf2f6b30, mutex=0x55d8bf2f6b08) at pthread_cond_wait.c:655
#3  0x000055d8bce653ea in os_event::wait (this=0x55d8bf2f6af0) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0event.cc:158
#4  0x000055d8bce64f26 in os_event::wait_low (this=0x55d8bf2f6af0, reset_sig_count=1) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0event.cc:325
#5  0x000055d8bce651d8 in os_event_wait_low (event=0x55d8bf2f6af0, reset_sig_count=0) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/os/os0event.cc:502
#6  0x000055d8bd08e2e7 in buf_dump_thread () at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/buf/buf0dump.cc:832
#7  0x00007fab0a90f6db in start_thread (arg=0x7faacd7fa700) at pthread_create.c:463
#8  0x00007fab09af561f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 5 (Thread 0x7faaff5fe700 (LWP 30393)):
#0  0x00007fab0a915ad3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55d8bf18c1f4) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x55d8bf18c170, cond=0x55d8bf18c1c8) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x55d8bf18c1c8, mutex=0x55d8bf18c170) at pthread_cond_wait.c:655
#3  0x00007fab04a1c311 in gu_cond_wait_SYS (cond=0x55d8bf18c1c8, mutex=0x55d8bf18c170) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/galera/galerautils/src/gu_threads.h:251
#4  0x00007fab04a336dc in gu::Lock::wait (this=0x7faaff5fdbb0, cond=...) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/galera/galerautils/src/gu_lock.hpp:46
#5  0x00007fab04a75312 in galera::ServiceThd::thd_func (arg=0x55d8bf18c150) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/galera/galera/src/galera_service_thd.cpp:31
#6  0x00007fab0a90f6db in start_thread (arg=0x7faaff5fe700) at pthread_create.c:463
#7  0x00007fab09af561f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 4 (Thread 0x7faafe07a700 (LWP 30394)):
#0  0x00007fab09af5947 in epoll_wait (epfd=8, events=0x7faafe0792e0, maxevents=128, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
#1  0x00007fab04be1dd2 in asio::detail::epoll_reactor::run (this=0x55d8bf1f2df0, usec=-1, ops=...) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/galera/asio/asio/detail/impl/epoll_reactor.ipp:470
#2  0x00007fab04be378e in asio::detail::scheduler::do_run_one (this=0x55d8bf1ec060, lock=..., this_thread=..., ec=...) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/galera/asio/asio/detail/impl/scheduler.ipp:430
#3  0x00007fab04be2efe in asio::detail::scheduler::run (this=0x55d8bf1ec060, ec=...) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/galera/asio/asio/detail/impl/scheduler.ipp:199
#4  0x00007fab04be3f5a in asio::io_context::run (this=0x55d8bf1f1110) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/galera/asio/asio/impl/io_context.ipp:62
#5  0x00007fab04bdbf1c in gu::AsioIoService::run (this=0x55d8bf1ed3e8) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/galera/galerautils/src/gu_asio.cpp:750
#6  0x00007fab04ad3592 in gcomm::AsioProtonet::event_loop (this=0x55d8bf1ed330, period=...) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/galera/gcomm/src/asio_protonet.cpp:103
#7  0x00007fab04ab7e54 in GCommConn::run (this=0x55d8bf1f0c90) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/galera/gcs/src/gcs_gcomm.cpp:439
#8  0x00007fab04ab6e2b in run_fn (arg=0x55d8bf1f0c90) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/galera/gcs/src/gcs_gcomm.cpp:307
#9  0x00007fab0a90f6db in start_thread (arg=0x7faafe07a700) at pthread_create.c:463
#10 0x00007fab09af561f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 3 (Thread 0x7fab04345700 (LWP 30396)):
#0  0x00007fab0a915ad3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7faae8018370) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7faae80182c0, cond=0x7faae8018348) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7faae8018348, mutex=0x7faae80182c0) at pthread_cond_wait.c:655
#3  0x000055d8bd25ebca in safe_cond_wait (cond=0x7faae8018348, mp=0x7faae8018298, file=0x55d8bd5d5220 "/home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/include/mysql/psi/mysql_thread.h", line=1185) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/mysys/thr_mutex.c:492
#4  0x000055d8bcd26626 in inline_mysql_cond_wait (that=0x7faae8018348, mutex=0x7faae8018298, src_file=0x55d8bd5d5538 "/home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/sql/wsrep_thd.h", src_line=69) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/include/mysql/psi/mysql_thread.h:1185
#5  0x000055d8bcd29c95 in Wsrep_thd_queue::pop_front (this=0x7faae8018240) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/sql/wsrep_thd.h:69
#6  0x000055d8bcd278ce in wsrep_rollback_process (rollbacker=0x7faae8001fa0, arg=0x55d8bf1edfd0) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/sql/wsrep_thd.cc:236
#7  0x000055d8bcd17284 in start_wsrep_THD (arg=0x55d8bf1edfd0) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/sql/wsrep_mysqld.cc:3042
#8  0x000055d8bccee829 in pfs_spawn_thread (arg=0x55d8bf1ba3b0) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/perfschema/pfs.cc:1869
#9  0x00007fab0a90f6db in start_thread (arg=0x7fab04345700) at pthread_create.c:463
#10 0x00007fab09af561f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 2 (Thread 0x7fab057ab700 (LWP 30392)):
#0  0x00007fab0a916065 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7fab057aad50, expected=0, futex_word=0x55d8be9022c8 <COND_timer+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  __pthread_cond_wait_common (abstime=0x7fab057aad50, mutex=0x55d8be902208 <LOCK_timer+40>, cond=0x55d8be9022a0 <COND_timer>) at pthread_cond_wait.c:539
#2  __pthread_cond_timedwait (cond=0x55d8be9022a0 <COND_timer>, mutex=0x55d8be902208 <LOCK_timer+40>, abstime=0x7fab057aad50) at pthread_cond_wait.c:667
#3  0x000055d8bd25ef8a in safe_cond_timedwait (cond=0x55d8be9022a0 <COND_timer>, mp=0x55d8be9021e0 <LOCK_timer>, abstime=0x7fab057aad50, file=0x55d8bd773cf0 "/home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/include/mysql/psi/mysql_thread.h", line=1211) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/mysys/thr_mutex.c:546
#4  0x000055d8bd25ffda in inline_mysql_cond_timedwait (that=0x55d8be9022a0 <COND_timer>, mutex=0x55d8be9021e0 <LOCK_timer>, abstime=0x7fab057aad50, src_file=0x55d8bd773d58 "/home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/mysys/thr_timer.c", src_line=291) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/include/mysql/psi/mysql_thread.h:1211
#5  0x000055d8bd260b8e in timer_handler (arg=0x0) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/mysys/thr_timer.c:291
#6  0x000055d8bccee829 in pfs_spawn_thread (arg=0x55d8bf163b00) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/perfschema/pfs.cc:1869
#7  0x00007fab0a90f6db in start_thread (arg=0x7fab057ab700) at pthread_create.c:463
#8  0x00007fab09af561f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 1 (Thread 0x7faab1ffb700 (LWP 30578)):
#0  __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:57
#1  0x000055d8bd25979d in my_write_core (sig=6) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/mysys/stacktrace.c:386
#2  0x000055d8bc975d40 in handle_fatal_signal (sig=6) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/sql/signal_handler.cc:372
#3  <signal handler called>
#4  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#5  0x00007fab09a147f1 in __GI_abort () at abort.c:79
#6  0x00007fab09a043fa in __assert_fail_base (fmt=0x7fab09b8b6c0 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x55d8bd61861b "!recv_no_log_write", file=file@entry=0x55d8bd618200 "/home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/log/log0log.cc", line=line@entry=1484, function=function@entry=0x55d8bd619f50 <log_checkpoint(bool)::__PRETTY_FUNCTION__> "bool log_checkpoint(bool)") at assert.c:92
#7  0x00007fab09a04472 in __GI___assert_fail (assertion=0x55d8bd61861b "!recv_no_log_write", file=0x55d8bd618200 "/home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/log/log0log.cc", line=1484, function=0x55d8bd619f50 <log_checkpoint(bool)::__PRETTY_FUNCTION__> "bool log_checkpoint(bool)") at assert.c:101
#8  0x000055d8bce29ed3 in log_checkpoint (sync=true) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/log/log0log.cc:1484
#9  0x000055d8bce2a225 in log_make_checkpoint () at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/log/log0log.cc:1551
#10 0x000055d8bcd529ea in sst_disable_innodb_writes () at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/handler/ha_innodb.cc:1892
#11 0x000055d8bcd52acb in innodb_disable_internal_writes (disable=true) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/innobase/handler/ha_innodb.cc:1909
#12 0x000055d8bc97ab82 in plugin_disable_internal_writes (plugin=0x55d8bf64d3b0, disable=0x7faab1ffa90c) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/sql/handler.cc:886
#13 0x000055d8bc651562 in plugin_foreach_with_mask (thd=0x0, func=0x55d8bc97ab3e <plugin_disable_internal_writes(THD*, plugin_ref, void*)>, type=1, state_mask=8, arg=0x7faab1ffa90c) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/sql/sql_plugin.cc:2521
#14 0x000055d8bc97abb8 in ha_disable_internal_writes (disable=true) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/sql/handler.cc:893
#15 0x000055d8bcd1e528 in sst_flush_tables (thd=0x7faa8c000d50) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/sql/wsrep_sst.cc:1640
#16 0x000055d8bcd1eb9f in sst_donor_thread (a=0x7fab042f8a10) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/sql/wsrep_sst.cc:1738
#17 0x000055d8bccee829 in pfs_spawn_thread (arg=0x7faaec03b7d0) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4-master/mysql-wsrep/storage/perfschema/pfs.cc:1869
#18 0x00007fab0a90f6db in start_thread (arg=0x7faab1ffb700) at pthread_create.c:463
#19 0x00007fab09af561f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Comment by Marko Mäkelä [ 2023-09-27 ]

janlindstrom, thank you. Did you try changing the assertions from ut_ad() to ut_a() so that they would be checked in a RelWithDebInfo build? That should allow more tests to be run in the same time, and improve the chances of hitting this failure.

Comment by Jan Lindström [ 2023-09-28 ]

marko I could do a PR containing exactly that change. I can't write MariaDB server repositories directly.

Comment by Marko Mäkelä [ 2023-09-28 ]

janlindstrom, I thought that you could apply 10.4-recv_no_log_write.patch (which I just created for you) into a local build and run some stress tests. I do not think that it is feasible to try to reproduce this in a CI environment. I don’t think that this can be analyzed without having a good enough execution trace, such as rr record. First it could be worthwhile to come up with a stress test that increases the failure probability, and then employ rr record.

The patch actually triggered one test failure, unrelated to Galera SST:

10.4 4e2594768d658194220ae5678a686ed680aa3217 RelWithDebInfo with patch

mariabackup.innodb_xa_rollback 'innodb'  w55 [ fail ]
2023-09-28 11:02:46 0 [Note] InnoDB: 1 transactions in prepared state after recovery
2023-09-28 11:02:46 0x7fcfb1bdb800  InnoDB: Assertion failure in file /mariadb/10.4/storage/innobase/mtr/mtr0mtr.cc line 429
InnoDB: Failing assertion: !m_modifications || !recv_no_log_write

This is most likely because I forgot to remove the ut_d() around assignments to recv_no_log_write in extra/mariabackup/xtrabackup.cc.

Comment by Jan Lindström [ 2023-09-29 ]

Same assertion was triggered on CI but same test does not crash locally or running full suite using --rr and stack trace is useless.

Thread 1 (Thread 0x7fa3617fa700 (LWP 16225)):
#0  __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:57
#1  0x0000564798628441 in my_write_core (sig=6) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4/mysql-wsrep/mysys/stacktrace.c:386
#2  0x0000564797d44da0 in handle_fatal_signal (sig=6) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4/mysql-wsrep/sql/signal_handler.cc:372
#3  <signal handler called>
#4  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#5  0x00007fa3c2be27f1 in __GI_abort () at abort.c:79
#6  0x00005647983bde1a in ut_dbg_assertion_failed (expr=0x5647989e306b "!recv_no_log_write", file=0x5647989e2c58 "/home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4/mysql-wsrep/storage/innobase/log/log0log.cc", line=1484) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4/mysql-wsrep/storage/innobase/ut/ut0dbg.cc:60
#7  0x00005647981f8d9d in log_checkpoint (sync=true) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4/mysql-wsrep/storage/innobase/log/log0log.cc:1484
#8  0x00005647981f90ef in log_make_checkpoint () at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4/mysql-wsrep/storage/innobase/log/log0log.cc:1551
#9  0x0000564798121a4a in sst_disable_innodb_writes () at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4/mysql-wsrep/storage/innobase/handler/ha_innodb.cc:1892
#10 0x0000564798121aea in innodb_disable_internal_writes (disable=true) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4/mysql-wsrep/storage/innobase/handler/ha_innodb.cc:1909
#11 0x0000564797d49be2 in plugin_disable_internal_writes (plugin=0x7fa34c183f20, disable=0x7fa3617f990c) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4/mysql-wsrep/sql/handler.cc:886
#12 0x0000564797a20594 in plugin_foreach_with_mask (thd=0x0, func=0x564797d49b9e <plugin_disable_internal_writes(THD*, plugin_ref, void*)>, type=1, state_mask=8, arg=0x7fa3617f990c) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4/mysql-wsrep/sql/sql_plugin.cc:2521
#13 0x0000564797d49c18 in ha_disable_internal_writes (disable=true) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4/mysql-wsrep/sql/handler.cc:893
#14 0x00005647980ed588 in sst_flush_tables (thd=0x7fa338000d50) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4/mysql-wsrep/sql/wsrep_sst.cc:1640
#15 0x00005647980edbff in sst_donor_thread (a=0x7fa3bc32ca20) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4/mysql-wsrep/sql/wsrep_sst.cc:1738
#16 0x00005647980bd889 in pfs_spawn_thread (arg=0x7fa3a00b4590) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4/mysql-wsrep/storage/perfschema/pfs.cc:1869
#17 0x00007fa3c3add6db in start_thread (arg=0x7fa3617fa700) at pthread_create.c:463
#18 0x00007fa3c2cc361f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Comment by Marko Mäkelä [ 2023-09-29 ]

The assertion failure catches the fact that some log records had been written before the previous log_make_checkpoint() completed. An execution trace would reveal where that offending write happened.

Were you able to reproduce this on 10.5 or later major versions? The page flushing and checkpoint logic is much simpler there, because everything is handled by the single buf_flush_page_cleaner thread.

Comment by Jan Lindström [ 2023-10-04 ]

marko After several rounds and history on Jenkings I can say on some confidence this problem does not reproduce 10.5 or later major versions.

Generated at Thu Feb 08 10:28:57 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.