Jan Lindström (Inactive) added a comment - 2022-12-07 05:34 bb-10.4-merge-jan bb-10.5-merge-jan bb-10.6-merge-jan

Daniel Black added a comment - 2022-12-13 02:05 In the merge to 10.5 I added disable on the following test (Galera 26.4.13 from fc36 packages) galera.galera_bf_abort_at_after_statement 'innodb' [ fail ] Test ended at 2022-12-13 12:36:39   CURRENT_TEST: galera.galera_bf_abort_at_after_statement --- /home/dan/repos/mariadb-server-10.5/mysql-test/suite/galera/r/galera_bf_abort_at_after_statement.result 2022-11-25 15:33:23.781512898 +1100 +++ /home/dan/repos/mariadb-server-10.5/mysql-test/suite/galera/r/galera_bf_abort_at_after_statement.reject 2022-12-13 12:36:29.449751519 +1100 @@ -14,6 +14,13 @@ UPDATE t1 SET val=3 WHERE id=1; connect node_2b, 127.0.0.1, root, , test, $NODE_MYPORT_2; set session wsrep_sync_wait=0; +Timeout in wait_condition.inc for SELECT COUNT(*) = 1 FROM t1 where id = 1 and val = 3 +Id User Host db Command Time State Info Progress +2 system user NULL Sleep 31 Update_rows_log_event::find_row(5) UPDATE t1 SET val=3 WHERE id=1 0.000 +1 system user NULL Sleep 36 wsrep aborter idle NULL 0.000 +10 root localhost:58978 test Killed 31 debug sync point: wsrep_after_statement_enter UPDATE t1 SET val=2 WHERE id=1 0.000 +11 root localhost:58984 test Sleep 31 NULL 0.000 +12 root localhost:59000 test Query 0 starting show full processlist 0.000 connection node_2a; SET DEBUG_SYNC = 'now SIGNAL continue'; connection node_2;   mysqltest: Result length mismatch

Daniel Black added a comment - 2022-12-14 00:30 re-disabled in 10.5: galera_sr.GCF-1060 , amd64 buster too , alma84 amd64 , rhel8 . galera_3nodes.galera_ssl_reload galera.galera_wan , ERROR WSREP: /home/buildbot/buildbot/build/gcs/src/gcs_state_msg.cpp:gcs_state_msg_get_quorum():947: Failed to establish quorum. galera_3nodes.GCF-354 : mysqltest: At line 39: query 'DROP TABLE test.t1' failed: 1047: WSREP has not yet prepared node for application use galera_3nodes.GCF-354 mysqltest: At line 30: query 'INSERT INTO test.t1 values (1)' failed: 1180: Got error 6 "No such device or address" during COMMIT

Daniel Black added a comment - 2022-12-15 07:17 - edited re-disabled for 10.5 mysql-wsrep-features#138 : At line 25: query 'DROP TABLE t1' failed: 2013: Lost connection to MySQL server during query as above 20 min timeout , no specifc test identified galera_ipv6_mysqldump : mysql_shutdown failed galera.galera_var_ignore_apply_errors : 28: "Server did not transition to READY state" same ^ 20 min timeout 20 min timeout 20 min timeout Note I didn't see any of these ^ failures in 10.6 when these tests where enabled.

Daniel Black added a comment - 2022-12-15 08:05 re-disable for 10.6 galera_3nodes.galera_ipv6_rsync_section/galera_ipv6_mysqldump , looks like the kvm-zyp-sles150-amd64 / kvm-zyp-sles15-amd64 /kvm-zyp-sles125-amd64 / kvm-zyp-opensuse150-amd64 isn't skipping these because of mysql-test/include/check_ipv6.inc or other ipv6 issues ^ and kvm-rpm-rocky9-amd64, kvm-rpm-rocky9-aarch64 and massive long list galera.galera_bf_kill_debug : timeout after 900 seconds galera.galera_ssl_upgrade : [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 130: Incorrect file format 'gtid_slave_pos'

Marko Mäkelä added a comment - 2022-12-16 08:53 Starting with 10.7, the test galera.galera_parallel_simple reliably times out while waiting for a condition. The following tests fail reliably for me locally, but they do not run at all on buildbot: galera_3nodes.galera_ipv6_mariabackup [ skipped ] Test needs --big-test galera_3nodes.galera_ipv6_mariabackup_section [ skipped ] Test needs --big-test galera_3nodes.galera_pc_bootstrap [ skipped ] Test needs --big-test

Marko Mäkelä added a comment - 2022-12-16 09:21 Starting with 10.9, the test galera_sr.MDEV-25718 times out while waiting for a condition very often.

Marko Mäkelä added a comment - 2022-12-16 09:50 It looks like the test galera.galera_var_notify_cmd may cause a hang at least starting with 10.6 . The name of the test does not occur anywhere in the log, but that is normal for this type of a hang. The normal 900-second (15-minute) timeout does not seem to apply to it. I noticed this because I got such a hang locally on 10.10. In the mysqld.1.err of the still active process, I see only one CURRENT_TEST line. The latest messages in it are 2022-12-16 11:24:53 5 [Note] WSREP: Cluster table is empty, not recovering transactions 2022-12-16 11:24:53 2 [Note] WSREP: Server status change initialized -> joined and those messages are 24 minutes old at the time of this writing.

Marko Mäkelä added a comment - 2022-12-16 09:58 Here are the stack traces of the mariadbd process for the hang of galera.galera_var_notify_cmd : 10.10 3217ecd7085a64b7e7ff89aeefbd9a09a139b6e4 Thread 14 (Thread 0x7f407d7fa6c0 (LWP 1038199) "mariadbd"): #0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7f407d7f9b00, op=137, expected=0, futex_word=0x561c233129dc) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (futex_word=futex_word@entry=0x561c233129dc, expected=expected@entry=0, clockid=clockid@entry=1, abstime=abstime@entry=0x7f407d7f9b00, private=private@entry=0, cancel=cancel@entry=true) at ./nptl/futex-internal.c:87 #2 0x00007f40b62a4d9b in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x561c233129dc, expected=expected@entry=0, clockid=clockid@entry=1, abstime=abstime@entry=0x7f407d7f9b00, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x00007f40b62a79cf in __pthread_cond_wait_common (abstime=<optimized out>, clockid=1, mutex=0x561c23312398, cond=0x561c233129b0) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_clockwait64 (abstime=<optimized out>, clockid=1, mutex=0x561c23312398, cond=0x561c233129b0) at ./nptl/pthread_cond_wait.c:682 #5 ___pthread_cond_clockwait64 (cond=0x561c233129b0, mutex=0x561c23312398, clockid=1, abstime=<optimized out>) at ./nptl/pthread_cond_wait.c:670 #6 0x0000561c2237823a in std::__condvar::wait_until (__abs_time=@0x7f407d7f9b00: {tv_sec = 12101, tv_nsec = 513386682}, __clock=1, __m=<optimized out>, this=0x561c233129b0) at /usr/include/c++/12/bits/std_mutex.h:169 #7 std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=this@entry=0x561c233129b0, __lock=@0x7f407d7f9bb0: {_M_device = 0x561c23312398, _M_owns = true}, __atime=@0x7f407d7f9b48: {__d = {__r = 12101513386682}}) at /usr/include/c++/12/condition_variable:200 #8 0x0000561c223782b1 in std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=@0x7f407d7f9b48: {__d = {__r = 12101513386682}}, __lock=@0x7f407d7f9bb0: {_M_device = 0x561c23312398, _M_owns = true}, this=0x561c233129b0) at /usr/include/c++/12/condition_variable:110 #9 std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (this=this@entry=0x561c233129b0, __lock=@0x7f407d7f9bb0: {_M_device = 0x561c23312398, _M_owns = true}, __rtime=@0x561c233123c0: {__r = 60000}) at /usr/include/c++/12/condition_variable:162 #10 0x0000561c22377077 in tpool::thread_pool_generic::wait_for_tasks (this=this@entry=0x561c23312280, lk=@0x7f407d7f9bb0: {_M_device = 0x561c23312398, _M_owns = true}, thread_data=thread_data@entry=0x561c233129b0) at /mariadb/10.10/tpool/tpool_generic.cc:480 #11 0x0000561c22377235 in tpool::thread_pool_generic::get_task (this=this@entry=0x561c23312280, thread_var=thread_var@entry=0x561c233129b0, t=t@entry=0x7f407d7f9bf8) at /mariadb/10.10/tpool/tpool_generic.cc:533 #12 0x0000561c2237731b in tpool::thread_pool_generic::worker_main (this=0x561c23312280, thread_var=0x561c233129b0) at /mariadb/10.10/tpool/tpool_generic.cc:578 #13 0x0000561c22378a1c in std::__invoke_impl<void, void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__f=<optimized out>, __t=<optimized out>) at /usr/include/c++/12/bits/invoke.h:74 #14 0x0000561c22378a37 in std::__invoke<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__fn=<optimized out>) at /usr/include/c++/12/bits/invoke.h:90 #15 std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::_M_invoke<0ul, 1ul, 2ul> (this=<optimized out>) at /usr/include/c++/12/bits/std_thread.h:252 #16 std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::operator() (this=<optimized out>) at /usr/include/c++/12/bits/std_thread.h:259 #17 std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> > >::_M_run (this=<optimized out>) at /usr/include/c++/12/bits/std_thread.h:210 #18 0x00007f40b60d44a3 in std::execute_native_thread_routine (__p=0x7f40ac003270) at ../../../../../src/libstdc++-v3/src/c++11/thread.cc:82 #19 0x00007f40b62a7fd4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #20 0x00007f40b632866c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81   Thread 13 (Thread 0x7f4091df36c0 (LWP 830344) "mariadbd"): #0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7f4091df2b00, op=137, expected=0, futex_word=0x561c233125dc) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (futex_word=futex_word@entry=0x561c233125dc, expected=expected@entry=0, clockid=clockid@entry=1, abstime=abstime@entry=0x7f4091df2b00, private=private@entry=0, cancel=cancel@entry=true) at ./nptl/futex-internal.c:87 #2 0x00007f40b62a4d9b in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x561c233125dc, expected=expected@entry=0, clockid=clockid@entry=1, abstime=abstime@entry=0x7f4091df2b00, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x00007f40b62a79cf in __pthread_cond_wait_common (abstime=<optimized out>, clockid=1, mutex=0x561c23312398, cond=0x561c233125b0) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_clockwait64 (abstime=<optimized out>, clockid=1, mutex=0x561c23312398, cond=0x561c233125b0) at ./nptl/pthread_cond_wait.c:682 #5 ___pthread_cond_clockwait64 (cond=0x561c233125b0, mutex=0x561c23312398, clockid=1, abstime=<optimized out>) at ./nptl/pthread_cond_wait.c:670 #6 0x0000561c2237823a in std::__condvar::wait_until (__abs_time=@0x7f4091df2b00: {tv_sec = 12129, tv_nsec = 519323851}, __clock=1, __m=<optimized out>, this=0x561c233125b0) at /usr/include/c++/12/bits/std_mutex.h:169 #7 std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=this@entry=0x561c233125b0, __lock=@0x7f4091df2bb0: {_M_device = 0x561c23312398, _M_owns = true}, __atime=@0x7f4091df2b48: {__d = {__r = 12129519323851}}) at /usr/include/c++/12/condition_variable:200 #8 0x0000561c223782b1 in std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=@0x7f4091df2b48: {__d = {__r = 12129519323851}}, __lock=@0x7f4091df2bb0: {_M_device = 0x561c23312398, _M_owns = true}, this=0x561c233125b0) at /usr/include/c++/12/condition_variable:110 #9 std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (this=this@entry=0x561c233125b0, __lock=@0x7f4091df2bb0: {_M_device = 0x561c23312398, _M_owns = true}, __rtime=@0x561c233123c0: {__r = 60000}) at /usr/include/c++/12/condition_variable:162 #10 0x0000561c22377077 in tpool::thread_pool_generic::wait_for_tasks (this=this@entry=0x561c23312280, lk=@0x7f4091df2bb0: {_M_device = 0x561c23312398, _M_owns = true}, thread_data=thread_data@entry=0x561c233125b--Type <RET> for more, q to quit, c to continue without paging--c 0) at /mariadb/10.10/tpool/tpool_generic.cc:480 #11 0x0000561c22377235 in tpool::thread_pool_generic::get_task (this=this@entry=0x561c23312280, thread_var=thread_var@entry=0x561c233125b0, t=t@entry=0x7f4091df2bf8) at /mariadb/10.10/tpool/tpool_generic.cc:533 #12 0x0000561c2237731b in tpool::thread_pool_generic::worker_main (this=0x561c23312280, thread_var=0x561c233125b0) at /mariadb/10.10/tpool/tpool_generic.cc:578 #13 0x0000561c22378a1c in std::__invoke_impl<void, void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__f=<optimized out>, __t=<optimized out>) at /usr/include/c++/12/bits/invoke.h:74 #14 0x0000561c22378a37 in std::__invoke<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__fn=<optimized out>) at /usr/include/c++/12/bits/invoke.h:90 #15 std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::_M_invoke<0ul, 1ul, 2ul> (this=<optimized out>) at /usr/include/c++/12/bits/std_thread.h:252 #16 std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::operator() (this=<optimized out>) at /usr/include/c++/12/bits/std_thread.h:259 #17 std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> > >::_M_run (this=<optimized out>) at /usr/include/c++/12/bits/std_thread.h:210 #18 0x00007f40b60d44a3 in std::execute_native_thread_routine (__p=0x7f40ac003240) at ../../../../../src/libstdc++-v3/src/c++11/thread.cc:82 #19 0x00007f40b62a7fd4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #20 0x00007f40b632866c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81   Thread 12 (Thread 0x7f40b00a76c0 (LWP 649768) "mariadbd"): #0 0x00007f40b625bbc2 in __GI___sigtimedwait (set=0x7f40b00a6ba0, info=0x7f40b00a6af0, timeout=0x0) at ../sysdeps/unix/sysv/linux/sigtimedwait.c:31 #1 0x0000561c21b344b0 in my_sigwait (set=set@entry=0x7f40b00a6ba0, sig=sig@entry=0x7f40b00a6b90, code=code@entry=0x7f40b00a6b94) at /mariadb/10.10/include/my_pthread.h:195 #2 0x0000561c21b3475d in signal_hand (arg=arg@entry=0x0) at /mariadb/10.10/sql/mysqld.cc:3201 #3 0x0000561c220653f0 in pfs_spawn_thread (arg=0x561c238fe998) at /mariadb/10.10/storage/perfschema/pfs.cc:2201 #4 0x00007f40b62a7fd4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #5 0x00007f40b632866c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81   Thread 11 (Thread 0x7f40b00f26c0 (LWP 649755) "mariadbd"): #0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x561c2270e4c8 <COND_manager+40>) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (futex_word=futex_word@entry=0x561c2270e4c8 <COND_manager+40>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0, cancel=cancel@entry=true) at ./nptl/futex-internal.c:87 #2 0x00007f40b62a4d9b in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x561c2270e4c8 <COND_manager+40>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x00007f40b62a73f8 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x561c2270e508 <LOCK_manager+40>, cond=0x561c2270e4a0 <COND_manager>) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_wait (cond=cond@entry=0x561c2270e4a0 <COND_manager>, mutex=mutex@entry=0x561c2270e508 <LOCK_manager+40>) at ./nptl/pthread_cond_wait.c:618 #5 0x0000561c223daa0e in safe_cond_wait (cond=cond@entry=0x561c2270e4a0 <COND_manager>, mp=mp@entry=0x561c2270e4e0 <LOCK_manager>, file=file@entry=0x561c214b0268 "/mariadb/10.10/mysys/my_thr_init.c", line=line@entry=596) at /mariadb/10.10/mysys/thr_mutex.c:494 #6 0x0000561c223d6277 in psi_cond_wait (that=0x561c2270e4a0 <COND_manager>, mutex=0x561c2270e4e0 <LOCK_manager>, file=<optimized out>, line=<optimized out>) at /mariadb/10.10/mysys/my_thr_init.c:596 #7 0x0000561c21dcf26d in inline_mysql_cond_wait (that=that@entry=0x561c2270e4a0 <COND_manager>, mutex=mutex@entry=0x561c2270e4e0 <LOCK_manager>, src_file=src_file@entry=0x561c21390f20 "/mariadb/10.10/sql/sql_manager.cc", src_line=src_line@entry=103) at /mariadb/10.10/include/mysql/psi/mysql_thread.h:1070 #8 0x0000561c21dcf3ca in handle_manager (arg=arg@entry=0x0) at /mariadb/10.10/sql/sql_manager.cc:103 #9 0x0000561c220653f0 in pfs_spawn_thread (arg=0x561c231341e8) at /mariadb/10.10/storage/perfschema/pfs.cc:2201 #10 0x00007f40b62a7fd4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #11 0x00007f40b632866c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81   Thread 10 (Thread 0x7f40933fe6c0 (LWP 649737) "mariadbd"): #0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x561c226191b0 <buf_pool+17584>) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (futex_word=futex_word@entry=0x561c226191b0 <buf_pool+17584>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0, cancel=cancel@entry=true) at ./nptl/futex-internal.c:87 #2 0x00007f40b62a4d9b in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x561c226191b0 <buf_pool+17584>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x00007f40b62a73f8 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x561c226190a8 <buf_pool+17320>, cond=0x561c22619188 <buf_pool+17544>) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_wait (cond=cond@entry=0x561c22619188 <buf_pool+17544>, mutex=mutex@entry=0x561c226190a8 <buf_pool+17320>) at ./nptl/pthread_cond_wait.c:618 #5 0x0000561c223daa0e in safe_cond_wait (cond=0x561c22619188 <buf_pool+17544>, mp=0x561c22619080 <buf_pool+17280>, file=0x561c21473680 "/mariadb/10.10/storage/innobase/buf/buf0flu.cc", line=2339) at /mariadb/10.10/mysys/thr_mutex.c:494 #6 0x0000561c22157dba in buf_flush_page_cleaner () at /mariadb/10.10/storage/innobase/buf/buf0flu.cc:2339 #7 0x0000561c221588cd in std::__invoke_impl<void, void (*)()> (__f=<optimized out>) at /usr/include/c++/12/bits/invoke.h:60 #8 std::__invoke<void (*)()> (__fn=<optimized out>) at /usr/include/c++/12/bits/invoke.h:96 #9 std::thread::_Invoker<std::tuple<void (*)()> >::_M_invoke<0ul> (this=<optimized out>) at /usr/include/c++/12/bits/std_thread.h:252 #10 std::thread::_Invoker<std::tuple<void (*)()> >::operator() (this=<optimized out>) at /usr/include/c++/12/bits/std_thread.h:259 #11 std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)()> > >::_M_run (this=<optimized out>) at /usr/include/c++/12/bits/std_thread.h:210 #12 0x00007f40b60d44a3 in std::execute_native_thread_routine (__p=0x561c2324fbc0) at ../../../../../src/libstdc++-v3/src/c++11/thread.cc:82 #13 0x00007f40b62a7fd4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #14 0x00007f40b632866c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81   Thread 9 (Thread 0x7f4093fff6c0 (LWP 649736) "mariadbd"): #0 _io_uring_get_cqe (ring=0x561c23356348, cqe_ptr=<optimized out>, data=data@entry=0x7f4093ffeb90) at ./src/queue.c:116 #1 0x00007f40b6c5dc39 in __io_uring_get_cqe (ring=<optimized out>, cqe_ptr=<optimized out>, submit=<optimized out>, wait_nr=<optimized out>, sigmask=<optimized out>) at ./src/queue.c:146 #2 0x0000561c223791d0 in io_uring_wait_cqe_nr (wait_nr=1, cqe_ptr=0x7f4093ffebf8, ring=0x561c23356348) at /usr/include/liburing.h:1175 #3 io_uring_wait_cqe (ring=ring@entry=0x561c23356348, cqe_ptr=cqe_ptr@entry=0x7f4093ffebf8) at /usr/include/liburing.h:1248 #4 0x0000561c223793ca in (anonymous namespace)::aio_uring::thread_routine (aio=0x561c23356340) at /mariadb/10.10/tpool/aio_liburing.cc:144 #5 0x0000561c22379074 in std::__invoke_impl<void, void (*)((anonymous namespace)::aio_uring*), (anonymous namespace)::aio_uring*> (__f=<optimized out>) at /usr/include/c++/12/bits/invoke.h:61 #6 std::__invoke<void (*)((anonymous namespace)::aio_uring*), (anonymous namespace)::aio_uring*> (__fn=<optimized out>) at /usr/include/c++/12/bits/invoke.h:96 #7 std::thread::_Invoker<std::tuple<void (*)((anonymous namespace)::aio_uring*), (anonymous namespace)::aio_uring*> >::_M_invoke<0, 1> (this=<optimized out>) at /usr/include/c++/12/bits/std_thread.h:252 #8 std::thread::_Invoker<std::tuple<void (*)((anonymous namespace)::aio_uring*), (anonymous namespace)::aio_uring*> >::operator() (this=<optimized out>) at /usr/include/c++/12/bits/std_thread.h:259 #9 std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)((anonymous namespace)::aio_uring*), (anonymous namespace)::aio_uring*> > >::_M_run (this=<optimized out>) at /usr/include/c++/12/bits/std_thread.h:210 #10 0x00007f40b60d44a3 in std::execute_native_thread_routine (__p=0x561c2324fa10) at ../../../../../src/libstdc++-v3/src/c++11/thread.cc:82 #11 0x00007f40b62a7fd4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #12 0x00007f40b632866c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81   Thread 8 (Thread 0x7f40a905c6c0 (LWP 649735) "mariadbd"): #0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7f40a905bba0, op=393, expected=0, futex_word=0x561c22f3f7e8 <COND_checkpoint+40>) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (futex_word=futex_word@entry=0x561c22f3f7e8 <COND_checkpoint+40>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f40a905bba0, private=private@entry=0, cancel=cancel@entry=true) at ./nptl/futex-internal.c:87 #2 0x00007f40b62a4d9b in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x561c22f3f7e8 <COND_checkpoint+40>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f40a905bba0, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x00007f40b62a76dc in __pthread_cond_wait_common (abstime=0x7f40a905bba0, clockid=0, mutex=0x561c22f3f828 <LOCK_checkpoint+40>, cond=0x561c22f3f7c0 <COND_checkpoint>) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_timedwait64 (cond=cond@entry=0x561c22f3f7c0 <COND_checkpoint>, mutex=mutex@entry=0x561c22f3f828 <LOCK_checkpoint+40>, abstime=abstime@entry=0x7f40a905bba0) at ./nptl/pthread_cond_wait.c:643 #5 0x0000561c223dace8 in safe_cond_timedwait (cond=cond@entry=0x561c22f3f7c0 <COND_checkpoint>, mp=mp@entry=0x561c22f3f800 <LOCK_checkpoint>, abstime=abstime@entry=0x7f40a905bba0, file=file@entry=0x561c214b0268 "/mariadb/10.10/mysys/my_thr_init.c", line=line@entry=609) at /mariadb/10.10/mysys/thr_mutex.c:548 #6 0x0000561c223d62fa in psi_cond_timedwait (that=0x561c22f3f7c0 <COND_checkpoint>, mutex=0x561c22f3f800 <LOCK_checkpoint>, abstime=0x7f40a905bba0, file=<optimized out>, line=<optimized out>) at /mariadb/10.10/mysys/my_thr_init.c:609 #7 0x0000561c2204ce70 in inline_mysql_cond_timedwait (that=<optimized out>, mutex=<optimized out>, abstime=abstime@entry=0x7f40a905bba0, src_file=src_file@entry=0x561c213efdd0 "/mariadb/10.10/storage/maria/ma_servicethread.c", src_line=src_line@entry=115) at /mariadb/10.10/include/mysql/psi/mysql_thread.h:1086 #8 0x0000561c2204d083 in my_service_thread_sleep (control=control@entry=0x561c2260b140 <checkpoint_control>, sleep_time=sleep_time@entry=30000000000) at /mariadb/10.10/storage/maria/ma_servicethread.c:115 #9 0x0000561c220455c1 in ma_checkpoint_background (arg=arg@entry=0x1e) at /mariadb/10.10/storage/maria/ma_checkpoint.c:725 #10 0x0000561c220653f0 in pfs_spawn_thread (arg=0x561c232fb868) at /mariadb/10.10/storage/perfschema/pfs.cc:2201 #11 0x00007f40b62a7fd4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #12 0x00007f40b632866c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81   Thread 7 (Thread 0x7f40b1ad86c0 (LWP 649218) "mariadbd"): #0 0x00007f40b62f2ac7 in __GI___wait4 (pid=649972, stat_loc=0x7f40b1ad618c, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30 #1 0x0000561c220b7e21 in wsp::process::wait (this=this@entry=0x7f40b1ad61a0) at /mariadb/10.10/sql/wsrep_utils.cc:380 #2 0x0000561c220c77b0 in wsrep_notify_status (status=status@entry=wsrep::server_state::s_joined, view=view@entry=0x0) at /mariadb/10.10/sql/wsrep_notify.cc:87 #3 0x0000561c220c60e7 in wsp::Config_state::set (this=0x561c231cff20, status=status@entry=wsrep::server_state::s_joined) at /mariadb/10.10/sql/wsrep_utils.h:213 #4 0x0000561c220c53d8 in Wsrep_server_service::log_state_change (this=<optimized out>, prev_state=wsrep::server_state::s_initialized, current_state=wsrep::server_state::s_joined) at /mariadb/10.10/sql/wsrep_server_service.cc:367 #5 0x0000561c22453e38 in wsrep::server_state::state (this=this@entry=0x561c231e0150, lock=@0x7f40b1ad68c0: {_M_device = 0x561c231e0438, _M_owns = true}, state=state@entry=wsrep::server_state::s_joined) at /mariadb/10.10/wsrep-lib/src/server_state.cpp:1361 #6 0x0000561c22454b8f in wsrep::server_state::on_primary_view (this=this@entry=0x561c231e0150, view=@0x7f40b1ad6b10: {state_id_ = {id_ = {data_ = {buf = "\177\213\314\373}#\021\355\272j\206\060\n\235\026\256"}}, seqno_ = {seqno_ = 1}}, view_seqno_ = {seqno_ = 1}, status_ = wsrep::view::primary, capabilities_ = 184703, own_index_ = 0, protocol_version_ = 4, members_ = std::vector of length 1, capacity 1 = {{id_ = {data_ = {buf = "\177\213\205R}#\021\355\227A\212\216]\256ݬ"}}, name_ = "jyty", incoming_ = "127.0.0.1:16280"}}}, high_priority_service=high_priority_service@entry=0x7f40b1ad7b60) at /mariadb/10.10/wsrep-lib/src/server_state.cpp:979 #7 0x0000561c22455aab in wsrep::server_state::on_view (this=this@entry=0x561c231e0150, view=@0x7f40b1ad6b10: {state_id_ = {id_ = {data_ = {buf = "\177\213\314\373}#\021\355\272j\206\060\n\235\026\256"}}, seqno_ = {seqno_ = 1}}, view_seqno_ = {seqno_ = 1}, status_ = wsrep::view::primary, capabilities_ = 184703, own_index_ = 0, protocol_version_ = 4, members_ = std::vector of length 1, capacity 1 = {{id_ = {data_ = {buf = "\177\213\205R}#\021\355\227A\212\216]\256ݬ"}}, name_ = "jyty", incoming_ = "127.0.0.1:16280"}}}, high_priority_service=high_priority_service@entry=0x7f40b1ad7b60) at /mariadb/10.10/wsrep-lib/src/server_state.cpp:1029 #8 0x0000561c2246257c in (anonymous namespace)::view_cb (app_ctx=0x561c231e0150, recv_ctx=0x7f40b1ad7b60, view_info=<optimized out>) at /mariadb/10.10/wsrep-lib/src/wsrep_provider_v26.cpp:404 #9 0x00007f40b185bb69 in galera::ReplicatorSMM::submit_view_info (this=<optimized out>, recv_ctx=<optimized out>, view_info=<optimized out>) at ./galera/src/replicator_smm.cpp:2431 #10 0x00007f40b185bba8 in galera::ReplicatorSMM::submit_ordered_view_info (this=0x561c23208a80, recv_ctx=<optimized out>, view_info=<optimized out>) at ./galera/src/replicator_smm.cpp:2821 #11 0x00007f40b1869d70 in galera::ReplicatorSMM::process_prim_conf_change (this=0x561c23208a80, recv_ctx=0x7f40b1ad7b60, conf=<optimized out>, my_index=<optimized out>, cc_buf=<optimized out>) at ./galera/src/replicator_smm.cpp:3012 #12 0x00007f40b186a073 in galera::ReplicatorSMM::process_conf_change (this=0x561c23208a80, recv_ctx=0x7f40b1ad7b60, cc=@0x7f40b1ad7590: {seqno_g = 0, seqno_l = 1, buf = 0x7f40b0dff518, size = 136, type = GCS_ACT_CCHANGE}) at ./galera/src/replicator_smm.cpp:2460 #13 0x00007f40b1892bb6 in galera::GcsActionSource::process (this=0x561c2323b1e0, recv_ctx=0x7f40b1ad7b60, exit_loop=@0x7f40b1ad779f: false) at ./galera/src/gcs_action_source.cpp:183 #14 0x00007f40b1868601 in galera::ReplicatorSMM::async_recv (this=0x561c23208a80, recv_ctx=0x7f40b1ad7b60) at ./galera/src/replicator_smm.cpp:398 #15 0x00007f40b1845bab in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at ./galera/src/wsrep_provider.cpp:238 #16 0x0000561c22461666 in wsrep::wsrep_provider_v26::run_applier (this=<optimized out>, applier_ctx=<optimized out>) at /mariadb/10.10/wsrep-lib/src/wsrep_provider_v26.cpp:858 #17 0x0000561c220d1057 in wsrep_replication_process (thd=0x7f4094000d58, arg=<optimized out>) at /mariadb/10.10/sql/wsrep_thd.cc:57 #18 0x0000561c220bff95 in start_wsrep_THD (arg=arg@entry=0x561c2324fa90) at /mariadb/10.10/sql/wsrep_mysqld.h:540 #19 0x0000561c220653f0 in pfs_spawn_thread (arg=0x561c2322afd8) at /mariadb/10.10/storage/perfschema/pfs.cc:2201 #20 0x00007f40b62a7fd4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #21 0x00007f40b632866c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81   Thread 6 (Thread 0x7f40b1b236c0 (LWP 649217) "mariadbd"): #0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7f40a0017310) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (futex_word=futex_word@entry=0x7f40a0017310, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0, cancel=cancel@entry=true) at ./nptl/futex-internal.c:87 #2 0x00007f40b62a4d9b in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f40a0017310, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x00007f40b62a73f8 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f40a0017260, cond=0x7f40a00172e8) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_wait (cond=cond@entry=0x7f40a00172e8, mutex=mutex@entry=0x7f40a0017260) at ./nptl/pthread_cond_wait.c:618 #5 0x0000561c223daa0e in safe_cond_wait (cond=0x7f40a00172e8, mp=0x7f40a0017238, file=0x561c2140e788 "/mariadb/10.10/include/mysql/psi/mysql_thread.h", line=1072) at /mariadb/10.10/mysys/thr_mutex.c:494 #6 0x0000561c220cfb1b in inline_mysql_cond_wait (that=that@entry=0x7f40a00172e8, mutex=mutex@entry=0x7f40a0017238, src_file=src_file@entry=0x561c2134e198 "/mariadb/10.10/sql/wsrep_thd.h", src_line=src_line@entry=69) at /mariadb/10.10/include/mysql/psi/mysql_thread.h:1072 #7 0x0000561c220d15a3 in Wsrep_thd_queue::pop_front (this=0x7f40a00171e0) at /mariadb/10.10/sql/wsrep_thd.h:69 #8 0x0000561c220d0dd5 in wsrep_rollback_process (rollbacker=0x7f40a0000d58, arg=<optimized out>) at /mariadb/10.10/sql/wsrep_thd.cc:236 #9 0x0000561c220bff95 in start_wsrep_THD (arg=arg@entry=0x561c23240b70) at /mariadb/10.10/sql/wsrep_mysqld.h:540 #10 0x0000561c220653f0 in pfs_spawn_thread (arg=0x561c2322ab88) at /mariadb/10.10/storage/perfschema/pfs.cc:2201 #11 0x00007f40b62a7fd4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #12 0x00007f40b632866c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 Thread 5 (Thread 0x7f40a9ffb6c0 (LWP 649216) "mariadbd"): #0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x561c231e8dc8) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (futex_word=futex_word@entry=0x561c231e8dc8, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0, cancel=cancel@entry=true) at ./nptl/futex-internal.c:87 #2 0x00007f40b62a4d9b in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x561c231e8dc8, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x00007f40b62a73f8 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x561c231e8d78, cond=0x561c231e8da0) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_wait (cond=0x561c231e8da0, mutex=0x561c231e8d78) at ./nptl/pthread_cond_wait.c:618 #5 0x00007f40b18c558a in gu::Lock::wait (this=<synthetic pointer>, cond=@0x561c231e8da0: {cond = {__data = {__wseq = {__value64 = 2, __value32 = {__low = 2, __high = 0}}, __g1_start = {__value64 = 0, __value32 = {__low = 0, __high = 0}}, __g_refs = {2, 0}, __g_size = {0, 0}, __g1_orig_size = 0, __wrefs = 8, __g_signals = {0, 0}}, __size = "\002", '\000' <repeats 15 times>, "\002", '\000' <repeats 19 times>, "\b\000\000\000\000\000\000\000\000\000\000", __align = 2}, ref_count = 1}) at ./galerautils/src/gu_lock.hpp:46 #6 RecvBuf::front (timeout=<optimized out>, this=0x561c231e8d78) at ./gcs/src/gcs_gcomm.cpp:114 #7 gcomm_recv (backend=<optimized out>, msg=0x561c23212890, timeout=9223372035999999999) at ./gcs/src/gcs_gcomm.cpp:610 #8 0x00007f40b18b3667 in core_msg_recv (timeout=9223372035999999999, recv_msg=0x561c23212890, backend=0x561c232129f0) at ./gcs/src/gcs_core.cpp:480 #9 gcs_core_recv (conn=0x561c23212830, recv_act=recv_act@entry=0x7f40a9ffab40, timeout=<optimized out>) at ./gcs/src/gcs_core.cpp:1201 #10 0x00007f40b18ae2ce in gcs_recv_thread (arg=0x561c232125a0) at ./gcs/src/gcs.cpp:1445 #11 0x00007f40b62a7fd4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #12 0x00007f40b632866c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 Thread 4 (Thread 0x7f40aa7fc6c0 (LWP 649202) "mariadbd"): #0 0x00007f40b6327cb6 in epoll_wait (epfd=6, events=0x7f40aa7fb230, maxevents=128, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30 #1 0x00007f40b199cef0 in asio::detail::epoll_reactor::run (this=0x561c2323b5a0, block=block@entry=true, ops=@0x7f40aa7fb938: {<asio::detail::noncopyable> = {<No data fields>}, front_ = 0x0, back_ = 0x0}) at ./asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007f40b199d93f in asio::detail::task_io_service::do_run_one (this=this@entry=0x561c2323e770, lock=@0x7f40aa7fb900: {<asio::detail::noncopyable> = {<No data fields>}, mutex_ = @0x561c2323e7a0, locked_ = false}, this_thread=@0x7f40aa7fb930: {<asio::detail::thread_info_base> = {<asio::detail::noncopyable> = {<No data fields>}, reusable_memory_ = 0x0}, private_op_queue = {<asio::detail::noncopyable> = {<No data fields>}, front_ = 0x0, back_ = 0x0}, private_outstanding_work = 0}, ec=std::error_code = { }) at ./asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007f40b1997f81 in asio::detail::task_io_service::run (ec=std::error_code = { }, this=0x561c2323e770) at ./asio/asio/detail/impl/task_io_service.ipp:148 #4 asio::io_service::run (this=<optimized out>) at ./asio/asio/impl/io_service.ipp:58 #5 gu::AsioIoService::run (this=<optimized out>) at ./galerautils/src/gu_asio.cpp:703 #6 0x00007f40b18c873c in GCommConn::run (this=0x561c231e8b80) at ./gcs/src/gcs_gcomm.cpp:435 #7 0x00007f40b18c8c89 in run_fn (arg=<optimized out>) at ./gcs/src/gcs_gcomm.cpp:304 #8 0x00007f40b62a7fd4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #9 0x00007f40b632866c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81   Thread 3 (Thread 0x7f40b0cfd6c0 (LWP 649072) "mariadbd"): #0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x561c23209268) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (futex_word=futex_word@entry=0x561c23209268, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0, cancel=cancel@entry=true) at ./nptl/futex-internal.c:87 #2 0x00007f40b62a4d9b in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x561c23209268, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x00007f40b62a73f8 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x561c23209218, cond=0x561c23209240) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_wait (cond=0x561c23209240, mutex=0x561c23209218) at ./nptl/pthread_cond_wait.c:618 #5 0x00007f40b188d81c in gu::Lock::wait (this=<synthetic pointer>, cond=@0x561c23209240: {cond = {__data = {__wseq = {__value64 = 6, __value32 = {__low = 6, __high = 0}}, __g1_start = {__value64 = 2, __value32 = {__low = 2, __high = 0}}, __g_refs = {2, 0}, __g_size = {0, 0}, __g1_orig_size = 4, __wrefs = 8, __g_signals = {0, 0}}, __size = "\006\000\000\000\000\000\000\000\002\000\000\000\000\000\000\000\002", '\000' <repeats 15 times>, "\004\000\000\000\b\000\000\000\000\000\000\000\000\000\000", __align = 6}, ref_count = 1}) at ./galerautils/src/gu_lock.hpp:46 #6 galera::ServiceThd::thd_func (arg=0x561c23209200) at ./galera/src/galera_service_thd.cpp:30 #7 0x00007f40b62a7fd4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #8 0x00007f40b632866c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81   Thread 2 (Thread 0x7f40b1bd66c0 (LWP 649011) "mariadbd"): #0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7f40b1bd5c20, op=393, expected=0, futex_word=0x561c22fd3bac <COND_timer+44>) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (futex_word=futex_word@entry=0x561c22fd3bac <COND_timer+44>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f40b1bd5c20, private=private@entry=0, cancel=cancel@entry=true) at ./nptl/futex-internal.c:87 #2 0x00007f40b62a4d9b in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x561c22fd3bac <COND_timer+44>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f40b1bd5c20, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x00007f40b62a76dc in __pthread_cond_wait_common (abstime=0x7f40b1bd5c20, clockid=0, mutex=0x561c22fd3be8 <LOCK_timer+40>, cond=0x561c22fd3b80 <COND_timer>) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_timedwait64 (cond=cond@entry=0x561c22fd3b80 <COND_timer>, mutex=mutex@entry=0x561c22fd3be8 <LOCK_timer+40>, abstime=abstime@entry=0x7f40b1bd5c20) at ./nptl/pthread_cond_wait.c:643 #5 0x0000561c223dace8 in safe_cond_timedwait (cond=cond@entry=0x561c22fd3b80 <COND_timer>, mp=mp@entry=0x561c22fd3bc0 <LOCK_timer>, abstime=abstime@entry=0x7f40b1bd5c20, file=file@entry=0x561c214b0268 "/mariadb/10.10/mysys/my_thr_init.c", line=line@entry=609) at /mariadb/10.10/mysys/thr_mutex.c:548 #6 0x0000561c223d62fa in psi_cond_timedwait (that=0x561c22fd3b80 <COND_timer>, mutex=0x561c22fd3bc0 <LOCK_timer>, abstime=0x7f40b1bd5c20, file=<optimized out>, line=<optimized out>) at /mariadb/10.10/mysys/my_thr_init.c:609 #7 0x0000561c223db228 in inline_mysql_cond_timedwait (that=that@entry=0x561c22fd3b80 <COND_timer>, mutex=mutex@entry=0x561c22fd3bc0 <LOCK_timer>, abstime=abstime@entry=0x7f40b1bd5c20, src_file=src_file@entry=0x561c213e9ac8 "/mariadb/10.10/mysys/thr_timer.c", src_line=src_line@entry=321) at /mariadb/10.10/include/mysql/psi/mysql_thread.h:1086 #8 0x0000561c223db2e2 in timer_handler (arg=arg@entry=0x0) at /mariadb/10.10/mysys/thr_timer.c:321 #9 0x0000561c220653f0 in pfs_spawn_thread (arg=0x561c231cf838) at /mariadb/10.10/storage/perfschema/pfs.cc:2201 #10 0x00007f40b62a7fd4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #11 0x00007f40b632866c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81   Thread 1 (Thread 0x7f40b5dd70c0 (LWP 648869) "mariadbd"): #0 futex_wait (private=0, expected=2, futex_word=0x561c22f8ce28 <LOCK_wsrep_server_state+40>) at ../sysdeps/nptl/futex-internal.h:146 #1 __GI___lll_lock_wait (futex=futex@entry=0x561c22f8ce28 <LOCK_wsrep_server_state+40>, private=0) at ./nptl/lowlevellock.c:49 #2 0x00007f40b62aa68f in __pthread_mutex_cond_lock (mutex=mutex@entry=0x561c22f8ce28 <LOCK_wsrep_server_state+40>) at ../nptl/pthread_mutex_lock.c:153 #3 0x00007f40b62a72f0 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x561c22f8ce28 <LOCK_wsrep_server_state+40>, cond=0x561c22f8cdc0 <COND_wsrep_server_state>) at ./nptl/pthread_cond_wait.c:607 #4 ___pthread_cond_wait (cond=cond@entry=0x561c22f8cdc0 <COND_wsrep_server_state>, mutex=mutex@entry=0x561c22f8ce28 <LOCK_wsrep_server_state+40>) at ./nptl/pthread_cond_wait.c:618 #5 0x0000561c223daa0e in safe_cond_wait (cond=0x561c22f8cdc0 <COND_wsrep_server_state>, mp=0x561c22f8ce00 <LOCK_wsrep_server_state>, file=0x561c2140e788 "/mariadb/10.10/include/mysql/psi/mysql_thread.h", line=1072) at /mariadb/10.10/mysys/thr_mutex.c:494 #6 0x0000561c21d28860 in inline_mysql_cond_wait (that=<optimized out>, mutex=<optimized out>, src_file=src_file@entry=0x561c21429d18 "/mariadb/10.10/sql/wsrep_condition_variable.h", src_line=src_line@entry=48) at /mariadb/10.10/include/mysql/psi/mysql_thread.h:1072 #7 0x0000561c21d37e6f in Wsrep_condition_variable::wait (this=0x561c231e0448, lock=<optimized out>) at /mariadb/10.10/sql/wsrep_condition_variable.h:48 #8 0x0000561c22453e55 in wsrep::server_state::state (this=this@entry=0x561c231e0150, lock=@0x7ffdc123b820: {_M_device = 0x561c231e0438, _M_owns = true}, state=state@entry=wsrep::server_state::s_initialized) at /mariadb/10.10/wsrep-lib/src/server_state.cpp:1366 #9 0x0000561c2245447b in wsrep::server_state::initialized (this=0x561c231e0150) at /mariadb/10.10/wsrep-lib/src/server_state.cpp:816 #10 0x0000561c220c1bb6 in wsrep_init_globals () at /mariadb/10.10/sql/wsrep_mysqld.cc:851 #11 0x0000561c21b3fc53 in mysqld_main (argc=<optimized out>, argv=<optimized out>) at /mariadb/10.10/sql/mysqld.cc:5837 #12 0x0000561c21b332e2 in main (argc=<optimized out>, argv=<optimized out>) at /mariadb/10.10/sql/main.cc:34

Marko Mäkelä added a comment - 2022-12-16 10:18 Note: Locally I get even more failures, apparently related to a newer SSL library, like these: CURRENT_TEST: galera_sr.MDEV-27615 mysqltest: At line 47: query 'reap' failed with wrong errno <Unknown> (2026): 'TLS/SSL error: unexpected eof while reading', instead of <Unknown> (2013)...   galera_3nodes.galera_dynamic_protocol 'innodb' w4 [ fail ] Found warnings/errors in server log file! Test ended at 2022-12-16 11:29:19 line 2022-12-16 11:28:58 0 [Warning] WSREP: read_completion_condition(): unexpected eof while reading (167772454: 'error:0A000126:SSL routines::unexpected eof while reading') 2022-12-16 11:28:58 0 [Warning] WSREP: read_handler(): unexpected eof while reading (167772454: 'error:0A000126:SSL routines::unexpected eof while reading') ^ Found warnings in /dev/shm/10.10/mysql-test/var/4/log/mysqld.3.err While I was looking for the Debian Sid failures on buildbot, I noticed lots of crashes on kvm-deb-sid-x86 like this: CURRENT_TEST: galera.galera_bf_abort_ps_bind mysqltest: At line 41: query 'update t set j='node2' where i=1' failed: <Unknown> (2013): Lost connection to server during query … #10 0x00ed8958 in vsnprintf (__ap=0xb0d93a68 "\002", __fmt=0x13d53c4 "wsrep_commit_empty for %llu client_state %s client_mode %s trans_state %s sql %s", __n=128, __s=0xb0d939bc "wsrep_commit_empty for 82112017988583426 client_state high priority client_mode executing trans_state NULL sql ") at /usr/include/i386-linux-gnu/bits/stdio2.h:85 #11 WSREP_LOG (fun=0xcfa120 <sql_print_information(char const*, ...)>, fmt=0x13d53c4 "wsrep_commit_empty for %llu client_state %s client_mode %s trans_state %s sql %s") at ./sql/wsrep_mysqld.cc:299 #12 0x00edec83 in wsrep_commit_empty (thd=0xb08006c8, all=true) at ./sql/wsrep_mysqld.cc:3693 The cause of the SIGSEGV is a vsnprintf format string mismatch: void wsrep_commit_empty(THD* thd, bool all) { DBUG_ENTER( "wsrep_commit_empty" ); WSREP_DEBUG( "wsrep_commit_empty for %llu client_state %s client_mode" " %s trans_state %s sql %s" , thd_get_thread_id(thd), … The function thd_get_thread_id() returns unsigned long , not unsigned long long , so the first parameter in the format string should be %lu instead of %llu . No warning is issued at compilation time, due to the lack of MDEV-21978 .

Sergei Golubchik added a comment - 2022-12-19 10:49 jplindst , there were more commits disabling more tests. Please, check them all before closing the bug

Marko Mäkelä added a comment - 2022-12-19 11:03 Moreover, many of the disabled tests refer only to this ticket, and hence this ticket should not be closed before all such tests have either been re-enabled or associated with more specific tickets.

Marko Mäkelä added a comment - 2023-11-20 08:07 The test galera_sr. MDEV-25718 is still disabled in 10.11.

Marko Mäkelä added a comment - 6 days ago The test galera_sr. MDEV-25718 is still disabled in 10.11. It currently is the only disabled test in that suite.