[MDEV-21525] galera.galera_ftwrl MTR failed: Crash with signal SIGABRT Created: 2020-01-17  Updated: 2022-10-26  Resolved: 2022-10-26

Status: Closed
Project: MariaDB Server
Component/s: Galera, Tests
Affects Version/s: 10.4.12
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Stepan Patryshev (Inactive) Assignee: Ramesh Sivaraman
Resolution: Fixed Votes: 0
Labels: None
Environment:

kvm-deb-xenial-amd64


Issue Links:
Relates
relates to MDEV-13738 galera_applier_ftwrl_table_alter may ... Closed
relates to MDEV-14381 galera.galera_as_master_large, galera... Closed

 Description   

galera.galera_ftwrl failed on bb: Crash with signal SIGABRT.

stdio.log:

10.4.12 2cff807d3f2149d17e776fbddb0fc9e23fa64e1b

galera.galera_ftwrl 'innodb'             w1 [ fail ]  timeout after 900 seconds
        Test ended at 2019-12-31 07:28:33
 
Test case timeout after 900 seconds
 
== /dev/shm/var/1/log/galera_ftwrl.log == 
connection node_2;
connection node_1;
CREATE TABLE t1 (id INT PRIMARY KEY) ENGINE=InnoDB;
connection node_2;
SET GLOBAL wsrep_provider_options = "repl.causal_read_timeout=PT1S";
FLUSH TABLES WITH READ LOCK;
 
 == /dev/shm/var/1/tmp/analyze-timeout-mysqld.2.err ==
mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/dev/shm/var/tmp/1/mysqld.2.sock' (111)
 
 == /dev/shm/var/1/tmp/analyze-timeout-mysqld.1.err ==
mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/dev/shm/var/tmp/1/mysqld.1.sock' (111)
 
 
 - found 'core' (0/0)
 
Trying 'dbx' to get a backtrace
 
Trying 'gdb' to get a backtrace from coredump /dev/shm/var/1/log/galera.galera_ftwrl-innodb/mysqld.2/data/core
Core generated by '/usr/sbin/mysqld'
Output from gdb follows. The first stack trace is from the failing thread.
The following stack traces are from all threads (so the failing one is
duplicated).
--------------------------
[New LWP 6599]
[New LWP 6638]
[New LWP 6602]
[New LWP 6838]
[New LWP 6820]
[New LWP 6832]
[New LWP 6640]
[New LWP 6847]
[New LWP 6840]
[New LWP 6821]
[New LWP 6846]
[New LWP 6822]
[New LWP 6845]
[New LWP 6639]
[New LWP 6841]
[New LWP 6826]
[New LWP 6601]
[New LWP 6833]
[New LWP 6600]
[New LWP 6844]
[New LWP 6839]
[New LWP 6834]
[New LWP 6836]
[New LWP 6837]
[New LWP 6835]
[New LWP 6842]
[New LWP 6831]
[New LWP 6843]
[New LWP 6827]
[New LWP 6823]
[New LWP 6825]
[New LWP 6824]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/mysqld --defaults-group-suffix=.2 --defaults-file=/dev/shm/var/1/my.c'.
Program terminated with signal SIGABRT, Aborted.
#0  __pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:62
62	../sysdeps/unix/sysv/linux/pthread_kill.c: No such file or directory.
[Current thread is 1 (Thread 0x7ffbb22f18c0 (LWP 6599))]
#0  __pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:62
#1  0x0000562e25944e97 in my_write_core (sig=sig@entry=6) at /home/buildbot/buildbot/build/mariadb-10.4.12/mysys/stacktrace.c:481
#2  0x0000562e253d721e in handle_fatal_signal (sig=6) at /home/buildbot/buildbot/build/mariadb-10.4.12/sql/signal_handler.cc:343
#3  <signal handler called>
#4  0x00007ffbb04b874d in poll () at ../sysdeps/unix/syscall-template.S:84
#5  0x00007ffbb04d60be in __poll_chk (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>, fdslen=<optimized out>) at poll_chk.c:27
#6  0x0000562e2510130f in poll (__timeout=-1, __nfds=2, __fds=0x7ffff807ae30) at /usr/include/x86_64-linux-gnu/bits/poll2.h:41
#7  handle_connections_sockets () at /home/buildbot/buildbot/build/mariadb-10.4.12/sql/mysqld.cc:6456
#8  0x0000562e25102816 in mysqld_main (argc=163, argv=0x562e26db0848) at /home/buildbot/buildbot/build/mariadb-10.4.12/sql/mysqld.cc:5892
#9  0x00007ffbb03dd830 in __libc_start_main (main=0x562e250db9e0 <main(int, char**)>, argc=23, argv=0x7ffff807b568, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffff807b558) at ../csu/libc-start.c:291
#10 0x0000562e250f5199 in _start ()
 
Thread 32 (Thread 0x7ffb7a5fc700 (LWP 6824)):
#0  0x00007ffbb1d4964a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
#1  0x0000562e256ae2f8 in LinuxAIOHandler::collect (this=this@entry=0x7ffb7a5fbb00) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0file.cc:1813
#2  0x0000562e256ae962 in LinuxAIOHandler::poll (this=this@entry=0x7ffb7a5fbb00, m1=m1@entry=0x7ffb7a5fbb90, m2=m2@entry=0x7ffb7a5fbb98, request=request@entry=0x7ffb7a5fbba0) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0file.cc:1974
#3  0x0000562e256b2ae8 in os_aio_linux_handler (request=0x7ffb7a5fbba0, m2=0x7ffb7a5fbb98, m1=0x7ffb7a5fbb90, global_segment=3) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0file.cc:2028
#4  os_aio_handler (segment=segment@entry=3, m1=m1@entry=0x7ffb7a5fbb90, m2=m2@entry=0x7ffb7a5fbb98, request=request@entry=0x7ffb7a5fbba0) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0file.cc:5598
#5  0x0000562e25833105 in fil_aio_wait (segment=segment@entry=3) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/fil/fil0fil.cc:4357
#6  0x0000562e25745ed8 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/srv/srv0start.cc:324
#7  0x00007ffbb16e16ba in start_thread (arg=0x7ffb7a5fc700) at pthread_create.c:333
#8  0x00007ffbb04c441d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
Thread 31 (Thread 0x7ffb79dfb700 (LWP 6825)):
#0  0x00007ffbb1d4964a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
#1  0x0000562e256ae2f8 in LinuxAIOHandler::collect (this=this@entry=0x7ffb79dfab00) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0file.cc:1813
#2  0x0000562e256ae962 in LinuxAIOHandler::poll (this=this@entry=0x7ffb79dfab00, m1=m1@entry=0x7ffb79dfab90, m2=m2@entry=0x7ffb79dfab98, request=request@entry=0x7ffb79dfaba0) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0file.cc:1974
#3  0x0000562e256b2ae8 in os_aio_linux_handler (request=0x7ffb79dfaba0, m2=0x7ffb79dfab98, m1=0x7ffb79dfab90, global_segment=4) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0file.cc:2028
#4  os_aio_handler (segment=segment@entry=4, m1=m1@entry=0x7ffb79dfab90, m2=m2@entry=0x7ffb79dfab98, request=request@entry=0x7ffb79dfaba0) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0file.cc:5598
#5  0x0000562e25833105 in fil_aio_wait (segment=segment@entry=4) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/fil/fil0fil.cc:4357
#6  0x0000562e25745ed8 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/srv/srv0start.cc:324
#7  0x00007ffbb16e16ba in start_thread (arg=0x7ffb79dfb700) at pthread_create.c:333
#8  0x00007ffbb04c441d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
Thread 30 (Thread 0x7ffb7adfd700 (LWP 6823)):
#0  0x00007ffbb1d4964a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
#1  0x0000562e256ae2f8 in LinuxAIOHandler::collect (this=this@entry=0x7ffb7adfcb00) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0file.cc:1813
#2  0x0000562e256ae962 in LinuxAIOHandler::poll (this=this@entry=0x7ffb7adfcb00, m1=m1@entry=0x7ffb7adfcb90, m2=m2@entry=0x7ffb7adfcb98, request=request@entry=0x7ffb7adfcba0) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0file.cc:1974
#3  0x0000562e256b2ae8 in os_aio_linux_handler (request=0x7ffb7adfcba0, m2=0x7ffb7adfcb98, m1=0x7ffb7adfcb90, global_segment=2) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0file.cc:2028
#4  os_aio_handler (segment=segment@entry=2, m1=m1@entry=0x7ffb7adfcb90, m2=m2@entry=0x7ffb7adfcb98, request=request@entry=0x7ffb7adfcba0) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0file.cc:5598
#5  0x0000562e25833105 in fil_aio_wait (segment=segment@entry=2) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/fil/fil0fil.cc:4357
#6  0x0000562e25745ed8 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/srv/srv0start.cc:324
#7  0x00007ffbb16e16ba in start_thread (arg=0x7ffb7adfd700) at pthread_create.c:333
#8  0x00007ffbb04c441d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
Thread 29 (Thread 0x7ffb78df9700 (LWP 6827)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1  0x0000562e256b7a12 in os_event::timed_wait (this=this@entry=0x562e26f9a530, abstime=abstime@entry=0x7ffb78df8980) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0event.cc:275
#2  0x0000562e256b7c61 in os_event::wait_time_low (this=0x562e26f9a530, time_in_usec=time_in_usec@entry=986000, reset_sig_count=reset_sig_count@entry=2) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0event.cc:385
#3  0x0000562e256b801a in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=986000, reset_sig_count=reset_sig_count@entry=2) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0event.cc:485
#4  0x0000562e257e231c in pc_sleep_if_needed (cur_time=1177265, sig_count=2, next_loop_time=1178251) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/buf/buf0flu.cc:2640
#5  buf_flush_page_cleaner_coordinator () at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/buf/buf0flu.cc:3099
#6  0x00007ffbb16e16ba in start_thread (arg=0x7ffb78df9700) at pthread_create.c:333
#7  0x00007ffbb04c441d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
Thread 28 (Thread 0x7ffb5d7fa700 (LWP 6843)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000562e256b8066 in os_event::wait (this=0x562e26f9a5a0) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0event.cc:158
#2  os_event::wait_low (reset_sig_count=1, this=0x562e26f9a5a0) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0event.cc:325
#3  os_event_wait_low (event=0x562e26f9a5a0, reset_sig_count=reset_sig_count@entry=0) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0event.cc:502
#4  0x0000562e257cfbcc in buf_resize_thread () at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/buf/buf0buf.cc:3178
#5  0x00007ffbb16e16ba in start_thread (arg=0x7ffb5d7fa700) at pthread_create.c:333
#6  0x00007ffbb04c441d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
Thread 27 (Thread 0x7ffb6f7fe700 (LWP 6831)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1  0x0000562e256b7a12 in os_event::timed_wait (this=this@entry=0x562e271dd9d0, abstime=abstime@entry=0x7ffb6f7fdbf0) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0event.cc:275
#2  0x0000562e256b7c61 in os_event::wait_time_low (this=this@entry=0x562e271dd9d0, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=1) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0event.cc:385
#3  0x0000562e256b801a in os_event_wait_time_low (event=event@entry=0x562e271dd9d0, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0event.cc:485
#4  0x0000562e2568d319 in lock_wait_timeout_thread () at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/lock/lock0wait.cc:522
#5  0x00007ffbb16e16ba in start_thread (arg=0x7ffb6f7fe700) at pthread_create.c:333
#6  0x00007ffbb04c441d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
Thread 26 (Thread 0x7ffb5dffb700 (LWP 6842)):
#0  0x00007ffbb16eac1d in nanosleep () at ../sysdeps/unix/syscall-template.S:84
#1  0x0000562e256b82d0 in os_thread_sleep (tm=tm@entry=1000000) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0thread.cc:231
#2  0x0000562e257c1a47 in btr_defragment_thread () at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/btr/btr0defragment.cc:717
#3  0x00007ffbb16e16ba in start_thread (arg=0x7ffb5dffb700) at pthread_create.c:333
#4  0x00007ffbb04c441d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
Thread 25 (Thread 0x7ffb6dffb700 (LWP 6835)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1  0x0000562e256b7a12 in os_event::timed_wait (this=this@entry=0x562e271ae360, abstime=abstime@entry=0x7ffb6dffaa30) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0event.cc:275
#2  0x0000562e256b7c61 in os_event::wait_time_low (this=0x562e271ae360, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=reset_sig_count@entry=1) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0event.cc:385
#3  0x0000562e256b801a in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=reset_sig_count@entry=1) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0event.cc:485
#4  0x0000562e2578db0a in ib_wqueue_timedwait (wq=wq@entry=0x562e27212d30, wait_in_usecs=wait_in_usecs@entry=5000000) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/ut/ut0wqueue.cc:156
#5  0x0000562e2586d58a in fts_optimize_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/fts/fts0opt.cc:2836
#6  0x00007ffbb16e16ba in start_thread (arg=0x7ffb6dffb700) at pthread_create.c:333
#7  0x00007ffbb04c441d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
Thread 24 (Thread 0x7ffb6cff9700 (LWP 6837)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000562e256b8066 in os_event::wait (this=0x562e26f9a220) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0event.cc:158
#2  os_event::wait_low (reset_sig_count=105, this=0x562e26f9a220) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0event.cc:325
#3  os_event_wait_low (event=0x562e26f9a220, reset_sig_count=reset_sig_count@entry=105) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0event.cc:502
#4  0x0000562e257443a1 in srv_resume_thread (timeout_usec=0, wait=true, sig_count=105, slot=0x562e26302d40 <srv_sys+192>) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/srv/srv0srv.cc:909
#5  srv_purge_coordinator_suspend (rseg_history_len=<optimized out>, slot=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/srv/srv0srv.cc:2657
#6  srv_purge_coordinator_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/srv/srv0srv.cc:2738
#7  0x00007ffbb16e16ba in start_thread (arg=0x7ffb6cff9700) at pthread_create.c:333
#8  0x00007ffbb04c441d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
Thread 23 (Thread 0x7ffb6d7fa700 (LWP 6836)):
#0  0x00007ffbb16eac1d in nanosleep () at ../sysdeps/unix/syscall-template.S:84
#1  0x0000562e256b82d0 in os_thread_sleep (tm=tm@entry=1000000) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0thread.cc:231
#2  0x0000562e2574262a in srv_master_sleep () at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/srv/srv0srv.cc:2334
#3  srv_master_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/srv/srv0srv.cc:2375
#4  0x00007ffbb16e16ba in start_thread (arg=0x7ffb6d7fa700) at pthread_create.c:333
#5  0x00007ffbb04c441d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
Thread 22 (Thread 0x7ffb6e7fc700 (LWP 6834)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1  0x0000562e256b7a12 in os_event::timed_wait (this=this@entry=0x562e2720ee30, abstime=abstime@entry=0x7ffb6e7fbbe0) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0event.cc:275
#2  0x0000562e256b7c61 in os_event::wait_time_low (this=0x562e2720ee30, time_in_usec=time_in_usec@entry=10000000, reset_sig_count=1, reset_sig_count@entry=0) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0event.cc:385
#3  0x0000562e256b801a in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=10000000, reset_sig_count=reset_sig_count@entry=0) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0event.cc:485
#4  0x0000562e25826b40 in dict_stats_thread () at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/dict/dict0stats_bg.cc:476
#5  0x00007ffbb16e16ba in start_thread (arg=0x7ffb6e7fc700) at pthread_create.c:333
#6  0x00007ffbb04c441d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
Thread 21 (Thread 0x7ffb5f7fe700 (LWP 6839)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000562e256b8066 in os_event::wait (this=0x562e26f9a300) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0event.cc:158
#2  os_event::wait_low (reset_sig_count=270, this=0x562e26f9a300) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0event.cc:325
#3  os_event_wait_low (event=0x562e26f9a300, reset_sig_count=reset_sig_count@entry=0) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0event.cc:502
#4  0x0000562e257430f7 in srv_resume_thread (sig_count=0, timeout_usec=0, wait=true, slot=0x562e26302db0 <srv_sys+304>) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/srv/srv0srv.cc:909
#5  srv_worker_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/srv/srv0srv.cc:2517
#6  0x00007ffbb16e16ba in start_thread (arg=0x7ffb5f7fe700) at pthread_create.c:333
#7  0x00007ffbb04c441d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
Thread 20 (Thread 0x7ffb5cff9700 (LWP 6844)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000562e25637f53 in inline_mysql_cond_wait (src_file=0x562e25b673a0 "/home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/handler/ha_innodb.cc", src_line=288, mutex=0x7ffb5cff8cc0, that=0x7ffb5cff8cf0) at /home/buildbot/buildbot/build/mariadb-10.4.12/include/mysql/psi/mysql_thread.h:1178
#2  thd_destructor_proxy () at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/handler/ha_innodb.cc:288
#3  0x0000562e25587351 in pfs_spawn_thread (arg=0x562e271b8b58) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/perfschema/pfs.cc:1862
#4  0x00007ffbb16e16ba in start_thread (arg=0x7ffb5cff9700) at pthread_create.c:333
#5  0x00007ffbb04c441d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
Thread 19 (Thread 0x7ffbab181700 (LWP 6600)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1  0x0000562e25949a1d in inline_mysql_cond_timedwait (that=0x562e26c16bc0 <COND_timer>, mutex=0x562e26c16c00 <LOCK_timer>, src_file=0x562e25bb75c8 "/home/buildbot/buildbot/build/mariadb-10.4.12/mysys/thr_timer.c", src_line=292, abstime=0x7ffbab180ce0) at /home/buildbot/buildbot/build/mariadb-10.4.12/include/mysql/psi/mysql_thread.h:1204
#2  timer_handler (arg=arg@entry=0x0) at /home/buildbot/buildbot/build/mariadb-10.4.12/mysys/thr_timer.c:292
#3  0x0000562e25587351 in pfs_spawn_thread (arg=0x562e26eca928) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/perfschema/pfs.cc:1862
#4  0x00007ffbb16e16ba in start_thread (arg=0x7ffbab181700) at pthread_create.c:333
#5  0x00007ffbb04c441d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
Thread 18 (Thread 0x7ffb6ffff700 (LWP 6833)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1  0x0000562e256b7a12 in os_event::timed_wait (this=this@entry=0x562e26f9a450, abstime=abstime@entry=0x7ffb6fffec90) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0event.cc:275
#2  0x0000562e256b7c61 in os_event::wait_time_low (this=0x562e26f9a450, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=1) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0event.cc:385
#3  0x0000562e256b801a in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0event.cc:485
#4  0x0000562e25740e62 in srv_monitor_thread () at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/srv/srv0srv.cc:1687
#5  0x00007ffbb16e16ba in start_thread (arg=0x7ffb6ffff700) at pthread_create.c:333
#6  0x00007ffbb04c441d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
Thread 17 (Thread 0x7ffbaa3c8700 (LWP 6601)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffbaa6c0680 in gu::Lock::wait (this=<synthetic pointer>, cond=...) at galerautils/src/gu_lock.hpp:45
#2  galera::ServiceThd::thd_func (arg=0x562e26f06868) at galera/src/galera_service_thd.cpp:30
#3  0x00007ffbb16e16ba in start_thread (arg=0x7ffbaa3c8700) at pthread_create.c:333
#4  0x00007ffbb04c441d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
Thread 16 (Thread 0x7ffb795fa700 (LWP 6826)):
#0  0x00007ffbb1d4964a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
#1  0x0000562e256ae2f8 in LinuxAIOHandler::collect (this=this@entry=0x7ffb795f9b00) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0file.cc:1813
#2  0x0000562e256ae962 in LinuxAIOHandler::poll (this=this@entry=0x7ffb795f9b00, m1=m1@entry=0x7ffb795f9b90, m2=m2@entry=0x7ffb795f9b98, request=request@entry=0x7ffb795f9ba0) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0file.cc:1974
#3  0x0000562e256b2ae8 in os_aio_linux_handler (request=0x7ffb795f9ba0, m2=0x7ffb795f9b98, m1=0x7ffb795f9b90, global_segment=5) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0file.cc:2028
#4  os_aio_handler (segment=segment@entry=5, m1=m1@entry=0x7ffb795f9b90, m2=m2@entry=0x7ffb795f9b98, request=request@entry=0x7ffb795f9ba0) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0file.cc:5598
#5  0x0000562e25833105 in fil_aio_wait (segment=segment@entry=5) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/fil/fil0fil.cc:4357
#6  0x0000562e25745ed8 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/srv/srv0start.cc:324
#7  0x00007ffbb16e16ba in start_thread (arg=0x7ffb795fa700) at pthread_create.c:333
#8  0x00007ffbb04c441d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
Thread 15 (Thread 0x7ffb5e7fc700 (LWP 6841)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000562e256b8066 in os_event::wait (this=0x562e26f9a4c0) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0event.cc:158
#2  os_event::wait_low (reset_sig_count=1, this=0x562e26f9a4c0) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0event.cc:325
#3  os_event_wait_low (event=0x562e26f9a4c0, reset_sig_count=reset_sig_count@entry=0) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0event.cc:502
#4  0x0000562e257d76f7 in buf_dump_thread () at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/buf/buf0dump.cc:833
#5  0x00007ffbb16e16ba in start_thread (arg=0x7ffb5e7fc700) at pthread_create.c:333
#6  0x00007ffbb04c441d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
Thread 14 (Thread 0x7ffba83c4700 (LWP 6639)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000562e25349983 in inline_mysql_cond_wait (src_line=69, src_file=0x562e25a46e90 "/home/buildbot/buildbot/build/mariadb-10.4.12/sql/wsrep_thd.h", mutex=0x7ffb90012ae8, that=0x7ffb90012b18) at /home/buildbot/buildbot/build/mariadb-10.4.12/include/mysql/psi/mysql_thread.h:1178
#2  Wsrep_thd_queue::pop_front (this=0x7ffb90012a90) at /home/buildbot/buildbot/build/mariadb-10.4.12/sql/wsrep_thd.h:69
#3  wsrep_rollback_process (rollbacker=0x7ffb900009a8, arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.4.12/sql/wsrep_thd.cc:228
#4  0x0000562e2533946e in start_wsrep_THD (arg=arg@entry=0x562e26f440c0) at /home/buildbot/buildbot/build/mariadb-10.4.12/sql/wsrep_mysqld.cc:2723
#5  0x0000562e25587351 in pfs_spawn_thread (arg=0x562e26f32c28) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/perfschema/pfs.cc:1862
#6  0x00007ffbb16e16ba in start_thread (arg=0x7ffba83c4700) at pthread_create.c:333
#7  0x00007ffbb04c441d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
Thread 13 (Thread 0x7ffba8147700 (LWP 6845)):
#0  0x00007ffbb03f3b26 in __GI___sigwaitinfo (set=<optimized out>, set@entry=0x7ffba8146c10, info=info@entry=0x7ffba8146c90) at ../sysdeps/unix/sysv/linux/sigwaitinfo.c:56
#1  0x0000562e250f7a13 in my_sigwait (code=<synthetic pointer>, sig=0x7ffba8146c08, set=0x7ffba8146c10) at /home/buildbot/buildbot/build/mariadb-10.4.12/include/my_pthread.h:198
#2  signal_hand (arg=arg@entry=0x0) at /home/buildbot/buildbot/build/mariadb-10.4.12/sql/mysqld.cc:3245
#3  0x0000562e25587351 in pfs_spawn_thread (arg=0x562e26f8fa68) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/perfschema/pfs.cc:1862
#4  0x00007ffbb16e16ba in start_thread (arg=0x7ffba8147700) at pthread_create.c:333
#5  0x00007ffbb04c441d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
Thread 12 (Thread 0x7ffb7b5fe700 (LWP 6822)):
#0  0x00007ffbb1d4964a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
#1  0x0000562e256ae2f8 in LinuxAIOHandler::collect (this=this@entry=0x7ffb7b5fdb00) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0file.cc:1813
#2  0x0000562e256ae962 in LinuxAIOHandler::poll (this=this@entry=0x7ffb7b5fdb00, m1=m1@entry=0x7ffb7b5fdb90, m2=m2@entry=0x7ffb7b5fdb98, request=request@entry=0x7ffb7b5fdba0) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0file.cc:1974
#3  0x0000562e256b2ae8 in os_aio_linux_handler (request=0x7ffb7b5fdba0, m2=0x7ffb7b5fdb98, m1=0x7ffb7b5fdb90, global_segment=1) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0file.cc:2028
#4  os_aio_handler (segment=segment@entry=1, m1=m1@entry=0x7ffb7b5fdb90, m2=m2@entry=0x7ffb7b5fdb98, request=request@entry=0x7ffb7b5fdba0) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0file.cc:5598
#5  0x0000562e25833105 in fil_aio_wait (segment=segment@entry=1) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/fil/fil0fil.cc:4357
#6  0x0000562e25745ed8 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/srv/srv0start.cc:324
#7  0x00007ffbb16e16ba in start_thread (arg=0x7ffb7b5fe700) at pthread_create.c:333
#8  0x00007ffbb04c441d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
Thread 11 (Thread 0x7ffba80fd700 (LWP 6846)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000562e251161c1 in inline_mysql_cond_wait (that=<optimized out>, mutex=<optimized out>, src_file=0x562e25a13c58 "/home/buildbot/buildbot/build/mariadb-10.4.12/sql/slave.cc", src_line=522) at /home/buildbot/buildbot/build/mariadb-10.4.12/include/mysql/psi/mysql_thread.h:1167
#2  handle_slave_background (arg=arg@entry=0x0) at /home/buildbot/buildbot/build/mariadb-10.4.12/sql/slave.cc:522
#3  0x0000562e25587351 in pfs_spawn_thread (arg=0x562e272befc8) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/perfschema/pfs.cc:1862
#4  0x00007ffbb16e16ba in start_thread (arg=0x7ffba80fd700) at pthread_create.c:333
#5  0x00007ffbb04c441d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
Thread 10 (Thread 0x7ffb7bdff700 (LWP 6821)):
#0  0x00007ffbb1d4964a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
#1  0x0000562e256ae2f8 in LinuxAIOHandler::collect (this=this@entry=0x7ffb7bdfeb00) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0file.cc:1813
#2  0x0000562e256ae962 in LinuxAIOHandler::poll (this=this@entry=0x7ffb7bdfeb00, m1=m1@entry=0x7ffb7bdfeb90, m2=m2@entry=0x7ffb7bdfeb98, request=request@entry=0x7ffb7bdfeba0) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0file.cc:1974
#3  0x0000562e256b2ae8 in os_aio_linux_handler (request=0x7ffb7bdfeba0, m2=0x7ffb7bdfeb98, m1=0x7ffb7bdfeb90, global_segment=0) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0file.cc:2028
#4  os_aio_handler (segment=segment@entry=0, m1=m1@entry=0x7ffb7bdfeb90, m2=m2@entry=0x7ffb7bdfeb98, request=request@entry=0x7ffb7bdfeba0) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0file.cc:5598
#5  0x0000562e25833105 in fil_aio_wait (segment=segment@entry=0) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/fil/fil0fil.cc:4357
#6  0x0000562e25745ed8 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/srv/srv0start.cc:324
#7  0x00007ffbb16e16ba in start_thread (arg=0x7ffb7bdff700) at pthread_create.c:333
#8  0x00007ffbb04c441d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
Thread 9 (Thread 0x7ffb5effd700 (LWP 6840)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000562e256b8066 in os_event::wait (this=0x562e26f9a370) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0event.cc:158
#2  os_event::wait_low (reset_sig_count=276, this=0x562e26f9a370) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0event.cc:325
#3  os_event_wait_low (event=0x562e26f9a370, reset_sig_count=reset_sig_count@entry=0) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0event.cc:502
#4  0x0000562e257430f7 in srv_resume_thread (sig_count=0, timeout_usec=0, wait=true, slot=0x562e26302de8 <srv_sys+360>) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/srv/srv0srv.cc:909
#5  srv_worker_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/srv/srv0srv.cc:2517
#6  0x00007ffbb16e16ba in start_thread (arg=0x7ffb5effd700) at pthread_create.c:333
#7  0x00007ffbb04c441d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
Thread 8 (Thread 0x7ffba80b3700 (LWP 6847)):
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007ffbb16e3efe in __GI___pthread_mutex_lock (mutex=mutex@entry=0x7ffb8001a520) at ../nptl/pthread_mutex_lock.c:135
#2  0x0000562e252f6bb4 in inline_mysql_mutex_lock (src_line=3366, src_file=0x562e25a094d0 "/home/buildbot/buildbot/build/mariadb-10.4.12/sql/sql_class.h", that=0x7ffb8001a520) at /home/buildbot/buildbot/build/mariadb-10.4.12/include/mysql/psi/mysql_thread.h:697
#3  THD::exit_cond (src_function=0x562e25a0e676 "<unknown>", src_file=0x562e25a40980 "/home/buildbot/buildbot/build/mariadb-10.4.12/sql/rpl_parallel.cc", src_line=489, stage=0x7ffba80af2f0, this=0x7ffb80009fe8) at /home/buildbot/buildbot/build/mariadb-10.4.12/sql/sql_class.h:3366
#4  pool_mark_busy (pool=pool@entry=0x562e263aa4a0 <global_rpl_thread_pool>, thd=thd@entry=0x7ffb80009fe8) at /home/buildbot/buildbot/build/mariadb-10.4.12/sql/rpl_parallel.cc:489
#5  0x0000562e252f7be6 in rpl_pause_for_ftwrl (thd=thd@entry=0x7ffb80009fe8) at /home/buildbot/buildbot/build/mariadb-10.4.12/sql/rpl_parallel.cc:563
#6  0x0000562e251befb1 in mysql_execute_command (thd=thd@entry=0x7ffb80009fe8) at /home/buildbot/buildbot/build/mariadb-10.4.12/sql/sql_parse.cc:5453
#7  0x0000562e251c6afa in mysql_parse (thd=thd@entry=0x7ffb80009fe8, rawbuf=rawbuf@entry=0x7ffb8001a750 "FLUSH TABLES WITH READ LOCK", length=length@entry=27, parser_state=parser_state@entry=0x7ffba80b1ff0, is_com_multi=is_com_multi@entry=false, is_next_command=is_next_command@entry=false) at /home/buildbot/buildbot/build/mariadb-10.4.12/sql/sql_parse.cc:7901
#8  0x0000562e251c7389 in wsrep_mysql_parse (thd=thd@entry=0x7ffb80009fe8, rawbuf=0x7ffb8001a750 "FLUSH TABLES WITH READ LOCK", length=27, parser_state=parser_state@entry=0x7ffba80b1ff0, is_com_multi=is_com_multi@entry=false, is_next_command=is_next_command@entry=false) at /home/buildbot/buildbot/build/mariadb-10.4.12/sql/sql_parse.cc:7705
#9  0x0000562e251ca3bb in dispatch_command (command=command@entry=COM_QUERY, thd=thd@entry=0x7ffb80009fe8, packet=packet@entry=0x7ffb80001649 "FLUSH TABLES WITH READ LOCK", packet_length=packet_length@entry=27, is_com_multi=is_com_multi@entry=false, is_next_command=is_next_command@entry=false) at /home/buildbot/buildbot/build/mariadb-10.4.12/sql/sql_parse.cc:1827
#10 0x0000562e251ca98c in do_command (thd=0x7ffb80009fe8) at /home/buildbot/buildbot/build/mariadb-10.4.12/sql/sql_parse.cc:1360
#11 0x0000562e252a78ba in do_handle_one_connection (connect=connect@entry=0x562e273099d8) at /home/buildbot/buildbot/build/mariadb-10.4.12/sql/sql_connect.cc:1412
#12 0x0000562e252a797d in handle_one_connection (arg=arg@entry=0x562e273099d8) at /home/buildbot/buildbot/build/mariadb-10.4.12/sql/sql_connect.cc:1316
#13 0x0000562e25587351 in pfs_spawn_thread (arg=0x562e27316b48) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/perfschema/pfs.cc:1862
#14 0x00007ffbb16e16ba in start_thread (arg=0x7ffba80b3700) at pthread_create.c:333
#15 0x00007ffbb04c441d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
Thread 7 (Thread 0x7ffba837a700 (LWP 6640)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffbaa52c2ee in fifo_lock_get (q=0x7ffbaa4ca010) at galerautils/src/gu_fifo.c:262
#2  gu_fifo_get_head (q=0x7ffbaa4ca010, err=err@entry=0x7ffba8378d6c) at galerautils/src/gu_fifo.c:330
#3  0x00007ffbaa67dfb5 in gcs_recv (conn=0x562e26f1a7e0, action=action@entry=0x7ffba8378df0) at gcs/src/gcs.cpp:2024
#4  0x00007ffbaa6cbd16 in galera::Gcs::recv (act=..., this=<optimized out>) at galera/src/galera_gcs.hpp:120
#5  galera::GcsActionSource::process (this=0x562e26f0f5b0, recv_ctx=0x7ffba83793f0, exit_loop=@0x7ffba8378fff: false) at galera/src/gcs_action_source.cpp:161
#6  0x00007ffbaa6f44b3 in galera::ReplicatorSMM::async_recv (this=0x562e26f06110, recv_ctx=0x7ffba83793f0) at galera/src/replicator_smm.cpp:391
#7  0x00007ffbaa712e4b in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at galera/src/wsrep_provider.cpp:236
#8  0x0000562e259f743e in wsrep::wsrep_provider_v26::run_applier (this=<optimized out>, applier_ctx=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.4.12/wsrep-lib/src/wsrep_provider_v26.cpp:731
#9  0x0000562e2534ac92 in wsrep_replication_process (thd=0x7ffb840009a8, arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.4.12/sql/wsrep_thd.cc:62
#10 0x0000562e2533946e in start_wsrep_THD (arg=arg@entry=0x562e26f67d00) at /home/buildbot/buildbot/build/mariadb-10.4.12/sql/wsrep_mysqld.cc:2723
#11 0x0000562e25587351 in pfs_spawn_thread (arg=0x562e26f32f88) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/perfschema/pfs.cc:1862
#12 0x00007ffbb16e16ba in start_thread (arg=0x7ffba837a700) at pthread_create.c:333
#13 0x00007ffbb04c441d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
Thread 6 (Thread 0x7ffb6effd700 (LWP 6832)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1  0x0000562e256b7a12 in os_event::timed_wait (this=this@entry=0x562e26f9a3e0, abstime=abstime@entry=0x7ffb6effcb60) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0event.cc:275
#2  0x0000562e256b7c61 in os_event::wait_time_low (this=0x562e26f9a3e0, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=1) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0event.cc:385
#3  0x0000562e256b801a in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0event.cc:485
#4  0x0000562e2574152b in srv_error_monitor_thread () at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/srv/srv0srv.cc:1851
#5  0x00007ffbb16e16ba in start_thread (arg=0x7ffb6effd700) at pthread_create.c:333
#6  0x00007ffbb04c441d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
Thread 5 (Thread 0x7ffb99668700 (LWP 6820)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1  0x0000562e255d3e7c in inline_mysql_cond_timedwait (src_file=0x562e25b63810 "/home/buildbot/buildbot/build/mariadb-10.4.12/storage/maria/ma_servicethread.c", src_line=116, abstime=0x7ffb99667c60, mutex=0x562e26bbb440 <LOCK_checkpoint>, that=0x562e26bbb400 <COND_checkpoint>) at /home/buildbot/buildbot/build/mariadb-10.4.12/include/mysql/psi/mysql_thread.h:1204
#2  my_service_thread_sleep (control=control@entry=0x562e262f6520 <checkpoint_control>, sleep_time=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/maria/ma_servicethread.c:115
#3  0x0000562e255cb4c7 in ma_checkpoint_background (arg=arg@entry=0x1e) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/maria/ma_checkpoint.c:707
#4  0x0000562e25587351 in pfs_spawn_thread (arg=0x562e26f90cb8) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/perfschema/pfs.cc:1862
#5  0x00007ffbb16e16ba in start_thread (arg=0x7ffb99668700) at pthread_create.c:333
#6  0x00007ffbb04c441d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
Thread 4 (Thread 0x7ffb5ffff700 (LWP 6838)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000562e256b8066 in os_event::wait (this=0x562e26f9a290) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0event.cc:158
#2  os_event::wait_low (reset_sig_count=464, this=0x562e26f9a290) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0event.cc:325
#3  os_event_wait_low (event=0x562e26f9a290, reset_sig_count=reset_sig_count@entry=0) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/os/os0event.cc:502
#4  0x0000562e257430f7 in srv_resume_thread (sig_count=0, timeout_usec=0, wait=true, slot=0x562e26302d78 <srv_sys+248>) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/srv/srv0srv.cc:909
#5  srv_worker_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/srv/srv0srv.cc:2517
#6  0x00007ffbb16e16ba in start_thread (arg=0x7ffb5ffff700) at pthread_create.c:333
#7  0x00007ffbb04c441d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
Thread 3 (Thread 0x7ffb9bffe700 (LWP 6602)):
#0  0x00007ffbb04c4a13 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
#1  0x00007ffbaa64351d in asio::detail::epoll_reactor::run (this=0x562e26f47760, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391
#2  0x00007ffbaa65f8d4 in asio::detail::task_io_service::do_run_one (ec=..., this_thread=..., lock=..., this=0x562e26f47680) at asio/asio/detail/impl/task_io_service.ipp:355
#3  asio::detail::task_io_service::run (ec=..., this=0x562e26f47680) at asio/asio/detail/impl/task_io_service.ipp:148
#4  asio::io_service::run (this=<optimized out>) at asio/asio/impl/io_service.ipp:58
#5  gcomm::AsioProtonet::event_loop (this=<optimized out>, period=...) at gcomm/src/asio_protonet.cpp:116
#6  0x00007ffbaa68bf88 in GCommConn::run (this=0x562e26f46cc0) at gcs/src/gcs_gcomm.cpp:498
#7  0x00007ffbaa6938d9 in GCommConn::run_fn (arg=<optimized out>) at gcs/src/gcs_gcomm.cpp:209
#8  0x00007ffbb16e16ba in start_thread (arg=0x7ffb9bffe700) at pthread_create.c:333
#9  0x00007ffbb04c441d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
Thread 2 (Thread 0x7ffb9b7fd700 (LWP 6638)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffbaa689bab in gu::Lock::wait (this=<synthetic pointer>, cond=...) at galerautils/src/gu_lock.hpp:45
#2  RecvBuf::front (timeout=..., this=0x562e26f46ef8) at gcs/src/gcs_gcomm.cpp:124
#3  gcomm_recv (backend=<optimized out>, msg=0x562e26f1aac0, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673
#4  0x00007ffbaa679e98 in core_msg_recv (timeout=9223372035999999999, recv_msg=0x562e26f1aac0, backend=0x562e26f1ac20) at gcs/src/gcs_core.cpp:480
#5  gcs_core_recv (conn=0x562e26f1aa60, recv_act=recv_act@entry=0x7ffb9b7fcc80, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184
#6  0x00007ffbaa68612f in gcs_recv_thread (arg=0x562e26f1a7e0) at gcs/src/gcs.cpp:1398
#7  0x00007ffbb16e16ba in start_thread (arg=0x7ffb9b7fd700) at pthread_create.c:333
#8  0x00007ffbb04c441d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
Thread 1 (Thread 0x7ffbb22f18c0 (LWP 6599)):
#0  __pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:62
#1  0x0000562e25944e97 in my_write_core (sig=sig@entry=6) at /home/buildbot/buildbot/build/mariadb-10.4.12/mysys/stacktrace.c:481
#2  0x0000562e253d721e in handle_fatal_signal (sig=6) at /home/buildbot/buildbot/build/mariadb-10.4.12/sql/signal_handler.cc:343
#3  <signal handler called>
#4  0x00007ffbb04b874d in poll () at ../sysdeps/unix/syscall-template.S:84
#5  0x00007ffbb04d60be in __poll_chk (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>, fdslen=<optimized out>) at poll_chk.c:27
#6  0x0000562e2510130f in poll (__timeout=-1, __nfds=2, __fds=0x7ffff807ae30) at /usr/include/x86_64-linux-gnu/bits/poll2.h:41
#7  handle_connections_sockets () at /home/buildbot/buildbot/build/mariadb-10.4.12/sql/mysqld.cc:6456
#8  0x0000562e25102816 in mysqld_main (argc=163, argv=0x562e26db0848) at /home/buildbot/buildbot/build/mariadb-10.4.12/sql/mysqld.cc:5892
#9  0x00007ffbb03dd830 in __libc_start_main (main=0x562e250db9e0 <main(int, char**)>, argc=23, argv=0x7ffff807b568, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffff807b558) at ../csu/libc-start.c:291
#10 0x0000562e250f5199 in _start ()
 
Compressed file /dev/shm/var/1/log/galera.galera_ftwrl-innodb/mysqld.2/data/core
 - saving '/dev/shm/var/1/log/galera.galera_ftwrl-innodb/' to '/dev/shm/var/log/galera.galera_ftwrl-innodb/'
 
Retrying test galera.galera_ftwrl, attempt(2/3)...



 Comments   
Comment by Jan Lindström (Inactive) [ 2022-10-25 ]

ramesh Can you try to this test case on 10.4 and 10.5.

Comment by Ramesh Sivaraman [ 2022-10-26 ]

MTR test looks good on 10.4. and 10.5. Verified in CS and ES builds

worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019
galera.galera_ftwrl 'innodb'             [ pass ]   4086
--------------------------------------------------------------------------
The servers were restarted 0 times
Spent 4.086 of 34 seconds executing testcases
 
Completed: All 1 tests were successful.
$
 
$ cat /etc/os-release  | grep VERSION
VERSION="16.04.7 LTS (Xenial Xerus)"
VERSION_ID="16.04"
VERSION_CODENAME=xenial
$

Generated at Thu Feb 08 09:07:48 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.