Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-24698

galera_sr.GCF-585 MTR failed with SIGABRT: no such a transition REPLICATING -> APPLYING

    XMLWordPrintable

Details

    Description

      galera_sr.GCF-585 failed on BB 10.5 CS and BB 10.6 CS with SIGABRT: "no such a transition REPLICATING -> APPLYING".
      It seems to be a sporadic issue.

      stdio.log:

      10.5.9, 927a882341eb1087e71d64de4e8cd89ab520de89, kvm-rpm-centos74-aarch64

      galera_sr.GCF-585 'innodb'               w2 [ fail ]
              Test ended at 2021-01-25 20:58:29
       
      CURRENT_TEST: galera_sr.GCF-585
      mysqltest: At line 44: query 'DROP TABLE t1' failed: 2013: Lost connection to MySQL server during query
       
      The result from queries just before the failure was:
      < snip >
      ERROR 40001: Deadlock found when trying to get lock; try restarting transaction
      alter table t1 drop primary key;
      drop table t1;
      CREATE TABLE t1 (f1 INTEGER PRIMARY KEY) ENGINE=InnoDB;
      INSERT INTO t1 VALUES (1),(2),(3),(4),(5),(6),(7),(8),(9),(10);
      SET SESSION wsrep_trx_fragment_size=1;
      SET AUTOCOMMIT=OFF;
      START TRANSACTION;
      update t1 set f1 = 100 where f1 = 10;
      connection node_2;
      INSERT INTO t1 VALUES (11),(12),(13),(14),(15),(16),(17),(18),(19),(20);
      SET SESSION wsrep_trx_fragment_size=1;
      SET SESSION innodb_lock_wait_timeout=1;
      SET AUTOCOMMIT=OFF;
      START TRANSACTION;
      delete from t1 where f1 > 10;
      ERROR 40001: Deadlock found when trying to get lock; try restarting transaction
      delete from t1 where f1 > 10 and f1 < 100;
      ERROR 40001: Deadlock found when trying to get lock; try restarting transaction
      DROP TABLE t1;
       
      More results from queries before failure can be found in /dev/shm/var/2/log/GCF-585.log
       
       
      Server [mysqld.2 - pid: 6712, winpid: 6712, exit: 256] failed during test run
      Server log from this test:
      ----------SERVER LOG START-----------
      2021-01-25 20:58:12 45 [ERROR] WSREP: FSM: no such a transition REPLICATING -> APPLYING
      210125 20:58:12 [ERROR] mysqld got signal 6 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
       
      To report this bug, see https://mariadb.com/kb/en/reporting-bugs
       
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed, 
      something is definitely wrong and this may fail.
       
      Server version: 10.5.9-MariaDB-log
      key_buffer_size=1048576
      read_buffer_size=131072
      max_used_connections=1
      max_threads=153
      thread_count=6
      It is possible that mysqld could use up to 
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 63646 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0xffff4c001298
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      2021-01-25 20:58:12 1 [Note] WSREP: MDL conflict, DDL vs SR
      schema:  test
      request: (1 	seqno 86 	wsrep (toi, exec, committed) cmd 0 9 	DROP TABLE t1)
      granted: (48 	seqno -1 	wsrep (high priority, exec, executing) cmd 0 161 	(null))
      stack_bottom = 0xffff525377b0 thread_stack 0x49000
      /usr/sbin/mariadbd(my_print_stacktrace+0x3c)[0xaaaacc0668c4]
      /usr/sbin/mariadbd(handle_fatal_signal+0x474)[0xaaaacbaa890c]
      addr2line: '': No such file
      [0xffff876806c0]
      :0(raise)[0xffff86e342c8]
      :0(abort)[0xffff86e35940]
      src/trx_handle.hpp:165(galera::TrxHandle::Transition::operator==(galera::TrxHandle::Transition const&) const)[0xffff81c3550c]
      :0(galera::ReplicatorSMM::to_isolation_end(galera::TrxHandleMaster&, wsrep_buf const*))[0xffff81c2a54c]
      src/gu_mutex.hpp:77(gu::Mutex::unlock() const)[0xffff81c430a0]
      /usr/sbin/mariadbd(_ZN5wsrep18wsrep_provider_v269leave_toiENS_9client_idERKNS_14mutable_bufferE+0x2c)[0xaaaacc0eea40]
      /usr/sbin/mariadbd(_ZN5wsrep12client_state14poll_enter_toiERSt11unique_lockINS_5mutexEERKSt6vectorINS_3keyESaIS6_EERKNS_12const_bufferERNS_7ws_metaEiNSt6chrono10time_pointINSG_3_V212steady_clockENSG_8durationIlSt5ratioILl1ELl1000000000EEEEEERb+0x2b4)[0xaaaacc0d4d60]
      /usr/sbin/mariadbd(_ZN5wsrep12client_state15enter_toi_localERKSt6vectorINS_3keyESaIS2_EERKNS_12const_bufferENSt6chrono10time_pointINSA_3_V212steady_clockENSA_8durationIlSt5ratioILl1ELl1000000000EEEEEE+0x7c)[0xaaaacc0d60f4]
      /usr/sbin/mariadbd(_Z24wsrep_to_isolation_beginP3THDPKcS2_PK10TABLE_LISTPK10Alter_infoPKSt6vectorIN5wsrep3keyESaISB_EEPK14HA_CREATE_INFO+0x5c4)[0xaaaacbd41ca0]
      /usr/sbin/mariadbd(_Z21mysql_execute_commandP3THD+0x4c58)[0xaaaacb8dbc30]
      /usr/sbin/mariadbd(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x228)[0xaaaacb8dca44]
      /usr/sbin/mariadbd(+0x65aac0)[0xaaaacb7caac0]
      /usr/sbin/mariadbd(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x1de0)[0xaaaacb8def58]
      /usr/sbin/mariadbd(_Z10do_commandP3THD+0xe4)[0xaaaacb8df734]
      /usr/sbin/mariadbd(_Z24do_handle_one_connectionP7CONNECTb+0x1a8)[0xaaaacb9a7218]
      /usr/sbin/mariadbd(handle_one_connection+0x3c)[0xaaaacb9a74e0]
      /usr/sbin/mariadbd(+0xb7291c)[0xaaaacbce291c]
      :0(start_thread)[0xffff87627bb0]
      :0(thread_start)[0xffff86edb4c0]
       
      Trying to get some variables.
      Some pointers may be invalid and cause the dump to abort.
      Query (0xffff4c010610): DROP TABLE t1
       
      Connection ID (thread ID): 45
      Status: NOT_KILLED
       
      Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off
       
      The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
      information that should help you find out what is causing the crash.
      Writing a core file...
      Working directory at /dev/shm/var/2/mysqld.2/data
      Resource Limits:
      Limit                     Soft Limit           Hard Limit           Units     
      Max cpu time              unlimited            unlimited            seconds   
      Max file size             unlimited            unlimited            bytes     
      Max data size             unlimited            unlimited            bytes     
      Max stack size            8388608              unlimited            bytes     
      Max core file size        unlimited            unlimited            bytes     
      Max resident set          unlimited            unlimited            bytes     
      Max processes             4096                 20007                processes 
      Max open files            1024                 1024                 files     
      Max locked memory         65536                65536                bytes     
      Max address space         unlimited            unlimited            bytes     
      Max file locks            unlimited            unlimited            locks     
      Max pending signals       20007                20007                signals   
      Max msgqueue size         819200               819200               bytes     
      Max nice priority         0                    0                    
      Max realtime priority     0                    0                    
      Max realtime timeout      unlimited            unlimited            us        
      Core pattern: core
       
      ----------SERVER LOG END-------------
       
       
       - found 'core.6714' (0/0)
       
      Trying 'dbx' to get a backtrace
       
      Trying 'gdb' to get a backtrace from coredump /dev/shm/var/2/log/galera_sr.GCF-585-innodb/mysqld.2/data/core.6714
      Core generated by '/usr/sbin/mariadbd'
      Output from gdb follows. The first stack trace is from the failing thread.
      The following stack traces are from all threads (so the failing one is
      duplicated).
      --------------------------
      [New LWP 6907]
      [New LWP 6718]
      [New LWP 6716]
      [New LWP 6893]
      [New LWP 6719]
      [New LWP 6894]
      [New LWP 6890]
      [New LWP 6898]
      [New LWP 6891]
      [New LWP 6899]
      [New LWP 6897]
      [New LWP 6903]
      [New LWP 6900]
      [New LWP 6904]
      [New LWP 6901]
      [New LWP 7226]
      [New LWP 6714]
      [New LWP 6717]
      [New LWP 6720]
      [New LWP 6721]
      [New LWP 6892]
      [New LWP 6895]
      [New LWP 6896]
      [Thread debugging using libthread_db enabled]
      Using host libthread_db library "/lib64/libthread_db.so.1".
      Core was generated by `/usr/sbin/mariadbd --defaults-group-suffix=.2 --defaults-file=/dev/shm/var/2/my'.
      Program terminated with signal 6, Aborted.
      #0  0x0000ffff8762cdfc in pthread_kill () from /lib64/libpthread.so.0
      #0  0x0000ffff8762cdfc in pthread_kill () from /lib64/libpthread.so.0
      #1  0x0000aaaacbaa8870 in handle_fatal_signal (sig=6) at /usr/src/debug/MariaDB-/src_0/sql/signal_handler.cc:330
      #2  <signal handler called>
      #3  0x0000ffff86e342c8 in raise () from /lib64/libc.so.6
      #4  0x0000ffff86e35940 in abort () from /lib64/libc.so.6
      #5  0x0000ffff81c3550c in galera::FSM<galera::TrxHandle::State, galera::TrxHandle::Transition, galera::EmptyGuard, galera::EmptyAction>::shift_to (this=this@entry=0xffff4cb0d6d8, state=state@entry=galera::TrxHandle::S_APPLYING, line=line@entry=1837) at galera/src/fsm.hpp:85
      #6  0x0000ffff81c2a54c in set_state (line=1837, state=galera::TrxHandle::S_APPLYING, this=0xffff4cb0d6d0) at galera/src/trx_handle.hpp:222
      #7  set_state (line=1837, state=galera::TrxHandle::S_APPLYING, this=0xffff4cb0d6d0) at galera/src/trx_handle.hpp:594
      #8  galera::ReplicatorSMM::to_isolation_end (this=0xaaab01058860, trx=..., err=<optimized out>) at galera/src/replicator_smm.cpp:1837
      #9  0x0000ffff81c430a0 in galera_to_execute_end (gh=<optimized out>, conn_id=45, err=0xffff52536280) at galera/src/wsrep_provider.cpp:1285
      #10 0x0000aaaacc0eea40 in wsrep::wsrep_provider_v26::leave_toi (this=<optimized out>, client_id=..., err=...) at /usr/src/debug/MariaDB-/src_0/wsrep-lib/src/wsrep_provider_v26.cpp:917
      #11 0x0000aaaacc0d4d60 in wsrep::client_state::poll_enter_toi (this=this@entry=0xffff4c0074d0, lock=..., keys=std::vector of length 1, capacity 1 = {...}, buffer=..., meta=..., flags=flags@entry=3, wait_until=wait_until@entry=..., timed_out=@0xffff52536590: false) at /usr/src/debug/MariaDB-/src_0/wsrep-lib/src/client_state.cpp:393
      #12 0x0000aaaacc0d60f4 in wsrep::client_state::enter_toi_local (this=this@entry=0xffff4c0074d0, keys=std::vector of length 1, capacity 1 = {...}, buffer=..., wait_until=wait_until@entry=...) at /usr/src/debug/MariaDB-/src_0/wsrep-lib/src/client_state.cpp:444
      #13 0x0000aaaacbd41ca0 in wsrep_TOI_begin (create_info=0x0, fk_tables=0x0, alter_info=0x0, table_list=<optimized out>, table=0x0, db=0x0, thd=0xffff4c001298) at /usr/src/debug/MariaDB-/src_0/sql/wsrep_mysqld.cc:2178
      #14 wsrep_to_isolation_begin (thd=thd@entry=0xffff4c001298, db_=db_@entry=0x0, table_=table_@entry=0x0, table_list=<optimized out>, alter_info=alter_info@entry=0x0, fk_tables=fk_tables@entry=0x0, create_info=create_info@entry=0x0) at /usr/src/debug/MariaDB-/src_0/sql/wsrep_mysqld.cc:2366
      #15 0x0000aaaacb8dbc30 in mysql_execute_command (thd=thd@entry=0xffff4c001298) at /usr/src/debug/MariaDB-/src_0/sql/sql_parse.cc:4972
      #16 0x0000aaaacb8dca44 in mysql_parse (thd=thd@entry=0xffff4c001298, rawbuf=rawbuf@entry=0xffff4c010610 "DROP TABLE t1", length=length@entry=13, parser_state=parser_state@entry=0xffff52536e40, is_com_multi=is_com_multi@entry=false, is_next_command=is_next_command@entry=false) at /usr/src/debug/MariaDB-/src_0/sql/sql_parse.cc:8062
      #17 0x0000aaaacb7caac0 in wsrep_mysql_parse (thd=thd@entry=0xffff4c001298, rawbuf=0xffff4c010610 "DROP TABLE t1", length=13, parser_state=parser_state@entry=0xffff52536e40, is_com_multi=is_com_multi@entry=false, is_next_command=is_next_command@entry=false) at /usr/src/debug/MariaDB-/src_0/sql/sql_parse.cc:7865
      #18 0x0000aaaacb8def58 in dispatch_command (command=command@entry=COM_QUERY, thd=thd@entry=0xffff4c001298, packet=packet@entry=0xffff4c0080f9 "DROP TABLE t1", packet_length=packet_length@entry=13, is_com_multi=is_com_multi@entry=false, is_next_command=is_next_command@entry=false) at /usr/src/debug/MariaDB-/src_0/sql/sql_parse.cc:1875
      #19 0x0000aaaacb8df734 in do_command (thd=0xffff4c001298) at /usr/src/debug/MariaDB-/src_0/sql/sql_parse.cc:1371
      #20 0x0000aaaacb9a7218 in do_handle_one_connection (connect=<optimized out>, connect@entry=0xaaab01ad0e08, put_in_cache=put_in_cache@entry=true) at /usr/src/debug/MariaDB-/src_0/sql/sql_connect.cc:1410
      #21 0x0000aaaacb9a74e0 in handle_one_connection (arg=arg@entry=0xaaab01ad0e08) at /usr/src/debug/MariaDB-/src_0/sql/sql_connect.cc:1312
      #22 0x0000aaaacbce291c in pfs_spawn_thread (arg=0xaaab01aedbc8) at /usr/src/debug/MariaDB-/src_0/storage/perfschema/pfs.cc:2201
      #23 0x0000ffff87627bb0 in start_thread () from /lib64/libpthread.so.0
      #24 0x0000ffff86edb4c0 in thread_start () from /lib64/libc.so.6
       
      Thread 23 (Thread 0xffff3ffff070 (LWP 6896)):
      #0  0x0000ffff8762c1ec in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
      #1  0x0000aaaacbff9c8c in __gthread_cond_timedwait (__abs_timeout=0xffff3fffe780, __mutex=<optimized out>, __cond=0xaaab01192780) at /usr/include/c++/4.8.2/aarch64-redhat-linux/bits/gthr-default.h:871
      #2  __wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=0xaaab01192780) at /usr/include/c++/4.8.2/condition_variable:160
      #3  wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=0xaaab01192780) at /usr/include/c++/4.8.2/condition_variable:100
      #4  wait_for<long, std::ratio<1l, 1000l> > (__rtime=..., __lock=..., this=0xaaab01192780) at /usr/include/c++/4.8.2/condition_variable:132
      #5  tpool::thread_pool_generic::wait_for_tasks (this=this@entry=0xaaab01182c40, lk=..., thread_data=thread_data@entry=0xaaab01192780) at /usr/src/debug/MariaDB-/src_0/tpool/tpool_generic.cc:446
      #6  0x0000aaaacbffa29c in tpool::thread_pool_generic::get_task (this=this@entry=0xaaab01182c40, thread_var=thread_var@entry=0xaaab01192780, t=t@entry=0xffff3fffe810) at /usr/src/debug/MariaDB-/src_0/tpool/tpool_generic.cc:499
      #7  0x0000aaaacbffb7e0 in tpool::thread_pool_generic::worker_main (this=0xaaab01182c40, thread_var=0xaaab01192780) at /usr/src/debug/MariaDB-/src_0/tpool/tpool_generic.cc:544
      #8  0x0000aaaacbffbd00 in operator()<tpool::worker_data*, void> (__object=<optimized out>, this=<optimized out>) at /usr/include/c++/4.8.2/functional:601
      #9  _M_invoke<0ul, 1ul> (this=<optimized out>) at /usr/include/c++/4.8.2/functional:1732
      #10 operator() (this=<optimized out>) at /usr/include/c++/4.8.2/functional:1720
      #11 std::thread::_Impl<std::_Bind_simple<std::_Mem_fn<void (tpool::thread_pool_generic::*)(tpool::worker_data*)> (tpool::thread_pool_generic*, tpool::worker_data*)> >::_M_run() (this=<optimized out>) at /usr/include/c++/4.8.2/thread:115
      #12 0x0000ffff87104348 in ?? () from /lib64/libstdc++.so.6
      #13 0x0000ffff87627bb0 in start_thread () from /lib64/libpthread.so.0
      #14 0x0000ffff86edb4c0 in thread_start () from /lib64/libc.so.6
       
      Thread 22 (Thread 0xffff50bff070 (LWP 6895)):
      #0  0x0000ffff8762c1ec in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
      #1  0x0000aaaacbff9c8c in __gthread_cond_timedwait (__abs_timeout=0xffff50bfe780, __mutex=<optimized out>, __cond=0xaaab01192800) at /usr/include/c++/4.8.2/aarch64-redhat-linux/bits/gthr-default.h:871
      #2  __wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=0xaaab01192800) at /usr/include/c++/4.8.2/condition_variable:160
      #3  wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=0xaaab01192800) at /usr/include/c++/4.8.2/condition_variable:100
      #4  wait_for<long, std::ratio<1l, 1000l> > (__rtime=..., __lock=..., this=0xaaab01192800) at /usr/include/c++/4.8.2/condition_variable:132
      #5  tpool::thread_pool_generic::wait_for_tasks (this=this@entry=0xaaab01182c40, lk=..., thread_data=thread_data@entry=0xaaab01192800) at /usr/src/debug/MariaDB-/src_0/tpool/tpool_generic.cc:446
      #6  0x0000aaaacbffa29c in tpool::thread_pool_generic::get_task (this=this@entry=0xaaab01182c40, thread_var=thread_var@entry=0xaaab01192800, t=t@entry=0xffff50bfe810) at /usr/src/debug/MariaDB-/src_0/tpool/tpool_generic.cc:499
      #7  0x0000aaaacbffb7e0 in tpool::thread_pool_generic::worker_main (this=0xaaab01182c40, thread_var=0xaaab01192800) at /usr/src/debug/MariaDB-/src_0/tpool/tpool_generic.cc:544
      #8  0x0000aaaacbffbd00 in operator()<tpool::worker_data*, void> (__object=<optimized out>, this=<optimized out>) at /usr/include/c++/4.8.2/functional:601
      #9  _M_invoke<0ul, 1ul> (this=<optimized out>) at /usr/include/c++/4.8.2/functional:1732
      #10 operator() (this=<optimized out>) at /usr/include/c++/4.8.2/functional:1720
      #11 std::thread::_Impl<std::_Bind_simple<std::_Mem_fn<void (tpool::thread_pool_generic::*)(tpool::worker_data*)> (tpool::thread_pool_generic*, tpool::worker_data*)> >::_M_run() (this=<optimized out>) at /usr/include/c++/4.8.2/thread:115
      #12 0x0000ffff87104348 in ?? () from /lib64/libstdc++.so.6
      #13 0x0000ffff87627bb0 in start_thread () from /lib64/libpthread.so.0
      #14 0x0000ffff86edb4c0 in thread_start () from /lib64/libc.so.6
       
      Thread 21 (Thread 0xffff523ff070 (LWP 6892)):
      #0  0x0000ffff8762bec8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
      #1  0x0000aaaacbf2d50c in inline_mysql_cond_wait (that=0xaaaacc8e71f0 <buf_pool+880>, mutex=0xaaaacc8e7180 <buf_pool+768>, src_file=<optimized out>, src_line=<optimized out>) at /usr/src/debug/MariaDB-/src_0/include/mysql/psi/mysql_thread.h:1233
      #2  0x0000aaaacbf317fc in buf_flush_page_cleaner () at /usr/src/debug/MariaDB-/src_0/storage/innobase/buf/buf0flu.cc:2098
      #3  0x0000ffff87627bb0 in start_thread () from /lib64/libpthread.so.0
      #4  0x0000ffff86edb4c0 in thread_start () from /lib64/libc.so.6
       
      Thread 20 (Thread 0xffff800d8070 (LWP 6721)):
      #0  0x0000ffff8762bec8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
      #1  0x0000ffff81ac35b4 in fifo_lock_get (q=0xffff81a70010) at galerautils/src/gu_fifo.c:262
      #2  gu_fifo_get_head (q=0xffff81a70010, err=0xffff800d70f0) at galerautils/src/gu_fifo.c:330
      #3  0x0000ffff81bdc7d4 in gcs_recv (conn=0xaaab01060b30, action=0xffff800d7150) at gcs/src/gcs.cpp:2047
      #4  0x0000ffff81c0f2ac in galera::GcsActionSource::process (this=0xaaab010a9440, recv_ctx=0xffff800d76f0, exit_loop=@0xffff800d7330: false) at galera/src/gcs_action_source.cpp:162
      #5  0x0000ffff81c2d5f4 in galera::ReplicatorSMM::async_recv (this=0xaaab01058860, recv_ctx=0xffff800d76f0) at galera/src/replicator_smm.cpp:390
      #6  0x0000ffff81c41448 in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at galera/src/wsrep_provider.cpp:236
      #7  0x0000aaaacc0ee790 in wsrep::wsrep_provider_v26::run_applier (this=<optimized out>, applier_ctx=<optimized out>) at /usr/src/debug/MariaDB-/src_0/wsrep-lib/src/wsrep_provider_v26.cpp:743
      #8  0x0000aaaacbd4d09c in wsrep_replication_process (thd=0xffff640009c8, arg=<optimized out>) at /usr/src/debug/MariaDB-/src_0/sql/wsrep_thd.cc:58
      #9  0x0000aaaacbd409ac in start_wsrep_THD (arg=arg@entry=0xaaab010bd880) at /usr/src/debug/MariaDB-/src_0/sql/wsrep_mysqld.cc:3039
      #10 0x0000aaaacbce291c in pfs_spawn_thread (arg=0xaaab01079508) at /usr/src/debug/MariaDB-/src_0/storage/perfschema/pfs.cc:2201
      #11 0x0000ffff87627bb0 in start_thread () from /lib64/libpthread.so.0
      #12 0x0000ffff86edb4c0 in thread_start () from /lib64/libc.so.6
       
      Thread 19 (Thread 0xffff80138070 (LWP 6720)):
      #0  0x0000ffff8762bec8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
      #1  0x0000aaaacbd4ca84 in inline_mysql_cond_wait (src_file=0xaaaacc285778 "/home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.5.9/sql/wsrep_thd.h", src_line=69, mutex=0xffff5c012eb8, that=0xffff5c012ef0) at /usr/src/debug/MariaDB-/src_0/include/mysql/psi/mysql_thread.h:1233
      #2  pop_front (this=0xffff5c012e60) at /usr/src/debug/MariaDB-/src_0/sql/wsrep_thd.h:69
      #3  wsrep_rollback_process (rollbacker=0xffff5c0009c8, arg=<optimized out>) at /usr/src/debug/MariaDB-/src_0/sql/wsrep_thd.cc:241
      #4  0x0000aaaacbd409ac in start_wsrep_THD (arg=arg@entry=0xaaab010c6380) at /usr/src/debug/MariaDB-/src_0/sql/wsrep_mysqld.cc:3039
      #5  0x0000aaaacbce291c in pfs_spawn_thread (arg=0xaaab010c5e38) at /usr/src/debug/MariaDB-/src_0/storage/perfschema/pfs.cc:2201
      #6  0x0000ffff87627bb0 in start_thread () from /lib64/libpthread.so.0
      #7  0x0000ffff86edb4c0 in thread_start () from /lib64/libc.so.6
       
      Thread 18 (Thread 0xffff8195f070 (LWP 6717)):
      #0  0x0000ffff8762bec8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
      #1  0x0000ffff81c06038 in wait (this=<synthetic pointer>, cond=...) at galerautils/src/gu_lock.hpp:46
      #2  galera::ServiceThd::thd_func (arg=0xaaab01058f28) at galera/src/galera_service_thd.cpp:30
      #3  0x0000ffff87627bb0 in start_thread () from /lib64/libpthread.so.0
      #4  0x0000ffff86edb4c0 in thread_start () from /lib64/libc.so.6
       
      Thread 17 (Thread 0xffff876cec80 (LWP 6714)):
      #0  0x0000ffff86ed0f8c in poll () from /lib64/libc.so.6
      #1  0x0000aaaacb822780 in handle_connections_sockets () at /usr/include/bits/poll2.h:41
      #2  0x0000aaaacb82399c in mysqld_main (argc=180, argv=0xaaab00e9f0e0) at /usr/src/debug/MariaDB-/src_0/sql/mysqld.cc:5669
      #3  0x0000ffff86e20d64 in __libc_start_main () from /lib64/libc.so.6
      #4  0x0000aaaacb818154 in _start ()
       
      Thread 16 (Thread 0xffff717df070 (LWP 7226)):
      #0  0x0000ffff8762c1ec in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
      #1  0x0000aaaacbff9c8c in __gthread_cond_timedwait (__abs_timeout=0xffff717de780, __mutex=<optimized out>, __cond=0xaaab01192480) at /usr/include/c++/4.8.2/aarch64-redhat-linux/bits/gthr-default.h:871
      #2  __wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=0xaaab01192480) at /usr/include/c++/4.8.2/condition_variable:160
      #3  wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=0xaaab01192480) at /usr/include/c++/4.8.2/condition_variable:100
      #4  wait_for<long, std::ratio<1l, 1000l> > (__rtime=..., __lock=..., this=0xaaab01192480) at /usr/include/c++/4.8.2/condition_variable:132
      #5  tpool::thread_pool_generic::wait_for_tasks (this=this@entry=0xaaab01182c40, lk=..., thread_data=thread_data@entry=0xaaab01192480) at /usr/src/debug/MariaDB-/src_0/tpool/tpool_generic.cc:446
      #6  0x0000aaaacbffa29c in tpool::thread_pool_generic::get_task (this=this@entry=0xaaab01182c40, thread_var=thread_var@entry=0xaaab01192480, t=t@entry=0xffff717de810) at /usr/src/debug/MariaDB-/src_0/tpool/tpool_generic.cc:499
      #7  0x0000aaaacbffb7e0 in tpool::thread_pool_generic::worker_main (this=0xaaab01182c40, thread_var=0xaaab01192480) at /usr/src/debug/MariaDB-/src_0/tpool/tpool_generic.cc:544
      #8  0x0000aaaacbffbd00 in operator()<tpool::worker_data*, void> (__object=<optimized out>, this=<optimized out>) at /usr/include/c++/4.8.2/functional:601
      #9  _M_invoke<0ul, 1ul> (this=<optimized out>) at /usr/include/c++/4.8.2/functional:1732
      #10 operator() (this=<optimized out>) at /usr/include/c++/4.8.2/functional:1720
      #11 std::thread::_Impl<std::_Bind_simple<std::_Mem_fn<void (tpool::thread_pool_generic::*)(tpool::worker_data*)> (tpool::thread_pool_generic*, tpool::worker_data*)> >::_M_run() (this=<optimized out>) at /usr/include/c++/4.8.2/thread:115
      #12 0x0000ffff87104348 in ?? () from /lib64/libstdc++.so.6
      #13 0x0000ffff87627bb0 in start_thread () from /lib64/libpthread.so.0
      #14 0x0000ffff86edb4c0 in thread_start () from /lib64/libc.so.6
       
      Thread 15 (Thread 0xffff3d7ff070 (LWP 6901)):
      #0  0x0000ffff8762c1ec in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
      #1  0x0000aaaacbff9c8c in __gthread_cond_timedwait (__abs_timeout=0xffff3d7fe780, __mutex=<optimized out>, __cond=0xaaab01192500) at /usr/include/c++/4.8.2/aarch64-redhat-linux/bits/gthr-default.h:871
      #2  __wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=0xaaab01192500) at /usr/include/c++/4.8.2/condition_variable:160
      #3  wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=0xaaab01192500) at /usr/include/c++/4.8.2/condition_variable:100
      #4  wait_for<long, std::ratio<1l, 1000l> > (__rtime=..., __lock=..., this=0xaaab01192500) at /usr/include/c++/4.8.2/condition_variable:132
      #5  tpool::thread_pool_generic::wait_for_tasks (this=this@entry=0xaaab01182c40, lk=..., thread_data=thread_data@entry=0xaaab01192500) at /usr/src/debug/MariaDB-/src_0/tpool/tpool_generic.cc:446
      #6  0x0000aaaacbffa29c in tpool::thread_pool_generic::get_task (this=this@entry=0xaaab01182c40, thread_var=thread_var@entry=0xaaab01192500, t=t@entry=0xffff3d7fe810) at /usr/src/debug/MariaDB-/src_0/tpool/tpool_generic.cc:499
      #7  0x0000aaaacbffb7e0 in tpool::thread_pool_generic::worker_main (this=0xaaab01182c40, thread_var=0xaaab01192500) at /usr/src/debug/MariaDB-/src_0/tpool/tpool_generic.cc:544
      #8  0x0000aaaacbffbd00 in operator()<tpool::worker_data*, void> (__object=<optimized out>, this=<optimized out>) at /usr/include/c++/4.8.2/functional:601
      #9  _M_invoke<0ul, 1ul> (this=<optimized out>) at /usr/include/c++/4.8.2/functional:1732
      #10 operator() (this=<optimized out>) at /usr/include/c++/4.8.2/functional:1720
      #11 std::thread::_Impl<std::_Bind_simple<std::_Mem_fn<void (tpool::thread_pool_generic::*)(tpool::worker_data*)> (tpool::thread_pool_generic*, tpool::worker_data*)> >::_M_run() (this=<optimized out>) at /usr/include/c++/4.8.2/thread:115
      #12 0x0000ffff87104348 in ?? () from /lib64/libstdc++.so.6
      #13 0x0000ffff87627bb0 in start_thread () from /lib64/libpthread.so.0
      #14 0x0000ffff86edb4c0 in thread_start () from /lib64/libc.so.6
       
      Thread 14 (Thread 0xffff52598070 (LWP 6904)):
      #0  0x0000ffff8762bec8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
      #1  0x0000aaaacb837a58 in handle_slave_background (arg=arg@entry=0x0) at /usr/src/debug/MariaDB-/src_0/include/mysql/psi/mysql_thread.h:1222
      #2  0x0000aaaacbce291c in pfs_spawn_thread (arg=0xaaab0116ca48) at /usr/src/debug/MariaDB-/src_0/storage/perfschema/pfs.cc:2201
      #3  0x0000ffff87627bb0 in start_thread () from /lib64/libpthread.so.0
      #4  0x0000ffff86edb4c0 in thread_start () from /lib64/libc.so.6
       
      Thread 13 (Thread 0xffff3dfff070 (LWP 6900)):
      #0  0x0000ffff8762c1ec in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
      #1  0x0000aaaacbff9c8c in __gthread_cond_timedwait (__abs_timeout=0xffff3dffe780, __mutex=<optimized out>, __cond=0xaaab01192580) at /usr/include/c++/4.8.2/aarch64-redhat-linux/bits/gthr-default.h:871
      #2  __wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=0xaaab01192580) at /usr/include/c++/4.8.2/condition_variable:160
      #3  wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=0xaaab01192580) at /usr/include/c++/4.8.2/condition_variable:100
      #4  wait_for<long, std::ratio<1l, 1000l> > (__rtime=..., __lock=..., this=0xaaab01192580) at /usr/include/c++/4.8.2/condition_variable:132
      #5  tpool::thread_pool_generic::wait_for_tasks (this=this@entry=0xaaab01182c40, lk=..., thread_data=thread_data@entry=0xaaab01192580) at /usr/src/debug/MariaDB-/src_0/tpool/tpool_generic.cc:446
      #6  0x0000aaaacbffa29c in tpool::thread_pool_generic::get_task (this=this@entry=0xaaab01182c40, thread_var=thread_var@entry=0xaaab01192580, t=t@entry=0xffff3dffe810) at /usr/src/debug/MariaDB-/src_0/tpool/tpool_generic.cc:499
      #7  0x0000aaaacbffb7e0 in tpool::thread_pool_generic::worker_main (this=0xaaab01182c40, thread_var=0xaaab01192580) at /usr/src/debug/MariaDB-/src_0/tpool/tpool_generic.cc:544
      #8  0x0000aaaacbffbd00 in operator()<tpool::worker_data*, void> (__object=<optimized out>, this=<optimized out>) at /usr/include/c++/4.8.2/functional:601
      #9  _M_invoke<0ul, 1ul> (this=<optimized out>) at /usr/include/c++/4.8.2/functional:1732
      #10 operator() (this=<optimized out>) at /usr/include/c++/4.8.2/functional:1720
      #11 std::thread::_Impl<std::_Bind_simple<std::_Mem_fn<void (tpool::thread_pool_generic::*)(tpool::worker_data*)> (tpool::thread_pool_generic*, tpool::worker_data*)> >::_M_run() (this=<optimized out>) at /usr/include/c++/4.8.2/thread:115
      #12 0x0000ffff87104348 in ?? () from /lib64/libstdc++.so.6
      #13 0x0000ffff87627bb0 in start_thread () from /lib64/libpthread.so.0
      #14 0x0000ffff86edb4c0 in thread_start () from /lib64/libc.so.6
       
      Thread 12 (Thread 0xffff525f8070 (LWP 6903)):
      #0  0x0000ffff86e35458 in sigwaitinfo () from /lib64/libc.so.6
      #1  0x0000aaaacb81a548 in signal_hand (arg=arg@entry=0x0) at /usr/src/debug/MariaDB-/src_0/include/my_pthread.h:196
      #2  0x0000aaaacbce291c in pfs_spawn_thread (arg=0xaaab0116be28) at /usr/src/debug/MariaDB-/src_0/storage/perfschema/pfs.cc:2201
      #3  0x0000ffff87627bb0 in start_thread () from /lib64/libpthread.so.0
      #4  0x0000ffff86edb4c0 in thread_start () from /lib64/libc.so.6
       
      Thread 11 (Thread 0xffff3f7ff070 (LWP 6897)):
      #0  0x0000ffff8762c1ec in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
      #1  0x0000aaaacbff9c8c in __gthread_cond_timedwait (__abs_timeout=0xffff3f7fe780, __mutex=<optimized out>, __cond=0xaaab01192700) at /usr/include/c++/4.8.2/aarch64-redhat-linux/bits/gthr-default.h:871
      #2  __wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=0xaaab01192700) at /usr/include/c++/4.8.2/condition_variable:160
      #3  wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=0xaaab01192700) at /usr/include/c++/4.8.2/condition_variable:100
      #4  wait_for<long, std::ratio<1l, 1000l> > (__rtime=..., __lock=..., this=0xaaab01192700) at /usr/include/c++/4.8.2/condition_variable:132
      #5  tpool::thread_pool_generic::wait_for_tasks (this=this@entry=0xaaab01182c40, lk=..., thread_data=thread_data@entry=0xaaab01192700) at /usr/src/debug/MariaDB-/src_0/tpool/tpool_generic.cc:446
      #6  0x0000aaaacbffa29c in tpool::thread_pool_generic::get_task (this=this@entry=0xaaab01182c40, thread_var=thread_var@entry=0xaaab01192700, t=t@entry=0xffff3f7fe810) at /usr/src/debug/MariaDB-/src_0/tpool/tpool_generic.cc:499
      #7  0x0000aaaacbffb7e0 in tpool::thread_pool_generic::worker_main (this=0xaaab01182c40, thread_var=0xaaab01192700) at /usr/src/debug/MariaDB-/src_0/tpool/tpool_generic.cc:544
      #8  0x0000aaaacbffbd00 in operator()<tpool::worker_data*, void> (__object=<optimized out>, this=<optimized out>) at /usr/include/c++/4.8.2/functional:601
      #9  _M_invoke<0ul, 1ul> (this=<optimized out>) at /usr/include/c++/4.8.2/functional:1732
      #10 operator() (this=<optimized out>) at /usr/include/c++/4.8.2/functional:1720
      #11 std::thread::_Impl<std::_Bind_simple<std::_Mem_fn<void (tpool::thread_pool_generic::*)(tpool::worker_data*)> (tpool::thread_pool_generic*, tpool::worker_data*)> >::_M_run() (this=<optimized out>) at /usr/include/c++/4.8.2/thread:115
      #12 0x0000ffff87104348 in ?? () from /lib64/libstdc++.so.6
      #13 0x0000ffff87627bb0 in start_thread () from /lib64/libpthread.so.0
      #14 0x0000ffff86edb4c0 in thread_start () from /lib64/libc.so.6
       
      Thread 10 (Thread 0xffff3e7ff070 (LWP 6899)):
      #0  0x0000ffff8762c1ec in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
      #1  0x0000aaaacbff9c8c in __gthread_cond_timedwait (__abs_timeout=0xffff3e7fe780, __mutex=<optimized out>, __cond=0xaaab01192600) at /usr/include/c++/4.8.2/aarch64-redhat-linux/bits/gthr-default.h:871
      #2  __wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=0xaaab01192600) at /usr/include/c++/4.8.2/condition_variable:160
      #3  wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=0xaaab01192600) at /usr/include/c++/4.8.2/condition_variable:100
      #4  wait_for<long, std::ratio<1l, 1000l> > (__rtime=..., __lock=..., this=0xaaab01192600) at /usr/include/c++/4.8.2/condition_variable:132
      #5  tpool::thread_pool_generic::wait_for_tasks (this=this@entry=0xaaab01182c40, lk=..., thread_data=thread_data@entry=0xaaab01192600) at /usr/src/debug/MariaDB-/src_0/tpool/tpool_generic.cc:446
      #6  0x0000aaaacbffa29c in tpool::thread_pool_generic::get_task (this=this@entry=0xaaab01182c40, thread_var=thread_var@entry=0xaaab01192600, t=t@entry=0xffff3e7fe810) at /usr/src/debug/MariaDB-/src_0/tpool/tpool_generic.cc:499
      #7  0x0000aaaacbffb7e0 in tpool::thread_pool_generic::worker_main (this=0xaaab01182c40, thread_var=0xaaab01192600) at /usr/src/debug/MariaDB-/src_0/tpool/tpool_generic.cc:544
      #8  0x0000aaaacbffbd00 in operator()<tpool::worker_data*, void> (__object=<optimized out>, this=<optimized out>) at /usr/include/c++/4.8.2/functional:601
      #9  _M_invoke<0ul, 1ul> (this=<optimized out>) at /usr/include/c++/4.8.2/functional:1732
      #10 operator() (this=<optimized out>) at /usr/include/c++/4.8.2/functional:1720
      #11 std::thread::_Impl<std::_Bind_simple<std::_Mem_fn<void (tpool::thread_pool_generic::*)(tpool::worker_data*)> (tpool::thread_pool_generic*, tpool::worker_data*)> >::_M_run() (this=<optimized out>) at /usr/include/c++/4.8.2/thread:115
      #12 0x0000ffff87104348 in ?? () from /lib64/libstdc++.so.6
      #13 0x0000ffff87627bb0 in start_thread () from /lib64/libpthread.so.0
      #14 0x0000ffff86edb4c0 in thread_start () from /lib64/libc.so.6
       
      Thread 9 (Thread 0xffff537ff070 (LWP 6891)):
      #0  0x0000ffff86ed66b4 in syscall () from /lib64/libc.so.6
      #1  0x0000aaaacbffdbc8 in my_getevents (min_nr=1, nr=256, ev=0xffff537fc820, ctx=<optimized out>) at /usr/src/debug/MariaDB-/src_0/tpool/aio_linux.cc:64
      #2  tpool::aio_linux::getevent_thread_routine (aio=0xaaab011ccb00) at /usr/src/debug/MariaDB-/src_0/tpool/aio_linux.cc:107
      #3  0x0000aaaacbffda28 in _M_invoke<0ul> (this=<optimized out>) at /usr/include/c++/4.8.2/functional:1732
      #4  operator() (this=<optimized out>) at /usr/include/c++/4.8.2/functional:1720
      #5  std::thread::_Impl<std::_Bind_simple<void (*(tpool::aio_linux*))(tpool::aio_linux*)> >::_M_run() (this=<optimized out>) at /usr/include/c++/4.8.2/thread:115
      #6  0x0000ffff87104348 in ?? () from /lib64/libstdc++.so.6
      #7  0x0000ffff87627bb0 in start_thread () from /lib64/libpthread.so.0
      #8  0x0000ffff86edb4c0 in thread_start () from /lib64/libc.so.6
       
      Thread 8 (Thread 0xffff3efff070 (LWP 6898)):
      #0  0x0000ffff8762c1ec in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
      #1  0x0000aaaacbff9c8c in __gthread_cond_timedwait (__abs_timeout=0xffff3effe780, __mutex=<optimized out>, __cond=0xaaab01192680) at /usr/include/c++/4.8.2/aarch64-redhat-linux/bits/gthr-default.h:871
      #2  __wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=0xaaab01192680) at /usr/include/c++/4.8.2/condition_variable:160
      #3  wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=0xaaab01192680) at /usr/include/c++/4.8.2/condition_variable:100
      #4  wait_for<long, std::ratio<1l, 1000l> > (__rtime=..., __lock=..., this=0xaaab01192680) at /usr/include/c++/4.8.2/condition_variable:132
      #5  tpool::thread_pool_generic::wait_for_tasks (this=this@entry=0xaaab01182c40, lk=..., thread_data=thread_data@entry=0xaaab01192680) at /usr/src/debug/MariaDB-/src_0/tpool/tpool_generic.cc:446
      #6  0x0000aaaacbffa29c in tpool::thread_pool_generic::get_task (this=this@entry=0xaaab01182c40, thread_var=thread_var@entry=0xaaab01192680, t=t@entry=0xffff3effe810) at /usr/src/debug/MariaDB-/src_0/tpool/tpool_generic.cc:499
      #7  0x0000aaaacbffb7e0 in tpool::thread_pool_generic::worker_main (this=0xaaab01182c40, thread_var=0xaaab01192680) at /usr/src/debug/MariaDB-/src_0/tpool/tpool_generic.cc:544
      #8  0x0000aaaacbffbd00 in operator()<tpool::worker_data*, void> (__object=<optimized out>, this=<optimized out>) at /usr/include/c++/4.8.2/functional:601
      #9  _M_invoke<0ul, 1ul> (this=<optimized out>) at /usr/include/c++/4.8.2/functional:1732
      #10 operator() (this=<optimized out>) at /usr/include/c++/4.8.2/functional:1720
      #11 std::thread::_Impl<std::_Bind_simple<std::_Mem_fn<void (tpool::thread_pool_generic::*)(tpool::worker_data*)> (tpool::thread_pool_generic*, tpool::worker_data*)> >::_M_run() (this=<optimized out>) at /usr/include/c++/4.8.2/thread:115
      #12 0x0000ffff87104348 in ?? () from /lib64/libstdc++.so.6
      #13 0x0000ffff87627bb0 in start_thread () from /lib64/libpthread.so.0
      #14 0x0000ffff86edb4c0 in thread_start () from /lib64/libc.so.6
       
      Thread 7 (Thread 0xffff53fff070 (LWP 6890)):
      #0  0x0000ffff8762c1ec in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
      #1  0x0000aaaacbc8153c in inline_mysql_cond_timedwait (src_file=0xaaaacc268ce0 "/home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.5.9/storage/maria/ma_servicethread.c", src_line=116, abstime=0xffff53ffe6f8, mutex=0xaaaacd1a09e8 <LOCK_checkpoint>, that=0xaaaacd1a0a28 <COND_checkpoint>) at /usr/src/debug/MariaDB-/src_0/include/mysql/psi/mysql_thread.h:1259
      #2  my_service_thread_sleep (control=control@entry=0xaaaacc8da100 <checkpoint_control>, sleep_time=<optimized out>) at /usr/src/debug/MariaDB-/src_0/storage/maria/ma_servicethread.c:115
      #3  0x0000aaaacbc7842c in ma_checkpoint_background (arg=arg@entry=0x1e) at /usr/src/debug/MariaDB-/src_0/storage/maria/ma_checkpoint.c:707
      #4  0x0000aaaacbce291c in pfs_spawn_thread (arg=0xaaab01168ec8) at /usr/src/debug/MariaDB-/src_0/storage/perfschema/pfs.cc:2201
      #5  0x0000ffff87627bb0 in start_thread () from /lib64/libpthread.so.0
      #6  0x0000ffff86edb4c0 in thread_start () from /lib64/libc.so.6
       
      Thread 6 (Thread 0xffff513ff070 (LWP 6894)):
      #0  0x0000ffff8762c1ec in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
      #1  0x0000aaaacbff9c8c in __gthread_cond_timedwait (__abs_timeout=0xffff513fe780, __mutex=<optimized out>, __cond=0xaaab01192880) at /usr/include/c++/4.8.2/aarch64-redhat-linux/bits/gthr-default.h:871
      #2  __wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=0xaaab01192880) at /usr/include/c++/4.8.2/condition_variable:160
      #3  wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=0xaaab01192880) at /usr/include/c++/4.8.2/condition_variable:100
      #4  wait_for<long, std::ratio<1l, 1000l> > (__rtime=..., __lock=..., this=0xaaab01192880) at /usr/include/c++/4.8.2/condition_variable:132
      #5  tpool::thread_pool_generic::wait_for_tasks (this=this@entry=0xaaab01182c40, lk=..., thread_data=thread_data@entry=0xaaab01192880) at /usr/src/debug/MariaDB-/src_0/tpool/tpool_generic.cc:446
      #6  0x0000aaaacbffa29c in tpool::thread_pool_generic::get_task (this=this@entry=0xaaab01182c40, thread_var=thread_var@entry=0xaaab01192880, t=t@entry=0xffff513fe810) at /usr/src/debug/MariaDB-/src_0/tpool/tpool_generic.cc:499
      #7  0x0000aaaacbffb7e0 in tpool::thread_pool_generic::worker_main (this=0xaaab01182c40, thread_var=0xaaab01192880) at /usr/src/debug/MariaDB-/src_0/tpool/tpool_generic.cc:544
      #8  0x0000aaaacbffbd00 in operator()<tpool::worker_data*, void> (__object=<optimized out>, this=<optimized out>) at /usr/include/c++/4.8.2/functional:601
      #9  _M_invoke<0ul, 1ul> (this=<optimized out>) at /usr/include/c++/4.8.2/functional:1732
      #10 operator() (this=<optimized out>) at /usr/include/c++/4.8.2/functional:1720
      #11 std::thread::_Impl<std::_Bind_simple<std::_Mem_fn<void (tpool::thread_pool_generic::*)(tpool::worker_data*)> (tpool::thread_pool_generic*, tpool::worker_data*)> >::_M_run() (this=<optimized out>) at /usr/include/c++/4.8.2/thread:115
      #12 0x0000ffff87104348 in ?? () from /lib64/libstdc++.so.6
      #13 0x0000ffff87627bb0 in start_thread () from /lib64/libpthread.so.0
      #14 0x0000ffff86edb4c0 in thread_start () from /lib64/libc.so.6
       
      Thread 5 (Thread 0xffff72fdf070 (LWP 6719)):
      #0  0x0000ffff8762bec8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
      #1  0x0000ffff81be230c in wait (this=<synthetic pointer>, cond=...) at galerautils/src/gu_lock.hpp:46
      #2  front (timeout=..., this=0xaaab010af2d0) at gcs/src/gcs_gcomm.cpp:114
      #3  gcomm_recv (backend=<optimized out>, msg=0xaaab01060e28, timeout=<optimized out>) at gcs/src/gcs_gcomm.cpp:608
      #4  0x0000ffff81bd46cc in core_msg_recv (timeout=9223372035999999999, recv_msg=0xaaab01060e28, backend=0xaaab01060f88) at gcs/src/gcs_core.cpp:480
      #5  gcs_core_recv (conn=0xaaab01060dc0, recv_act=recv_act@entry=0xffff72fde7a0, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1201
      #6  0x0000ffff81bdf358 in gcs_recv_thread (arg=0xaaab01060b30) at gcs/src/gcs.cpp:1413
      #7  0x0000ffff87627bb0 in start_thread () from /lib64/libpthread.so.0
      #8  0x0000ffff86edb4c0 in thread_start () from /lib64/libc.so.6
       
      Thread 4 (Thread 0xffff51bff070 (LWP 6893)):
      #0  0x0000ffff8762c1ec in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
      #1  0x0000aaaacbff9c8c in __gthread_cond_timedwait (__abs_timeout=0xffff51bfe780, __mutex=<optimized out>, __cond=0xaaab01192900) at /usr/include/c++/4.8.2/aarch64-redhat-linux/bits/gthr-default.h:871
      #2  __wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=0xaaab01192900) at /usr/include/c++/4.8.2/condition_variable:160
      #3  wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=0xaaab01192900) at /usr/include/c++/4.8.2/condition_variable:100
      #4  wait_for<long, std::ratio<1l, 1000l> > (__rtime=..., __lock=..., this=0xaaab01192900) at /usr/include/c++/4.8.2/condition_variable:132
      #5  tpool::thread_pool_generic::wait_for_tasks (this=this@entry=0xaaab01182c40, lk=..., thread_data=thread_data@entry=0xaaab01192900) at /usr/src/debug/MariaDB-/src_0/tpool/tpool_generic.cc:446
      #6  0x0000aaaacbffa29c in tpool::thread_pool_generic::get_task (this=this@entry=0xaaab01182c40, thread_var=thread_var@entry=0xaaab01192900, t=t@entry=0xffff51bfe810) at /usr/src/debug/MariaDB-/src_0/tpool/tpool_generic.cc:499
      #7  0x0000aaaacbffb7e0 in tpool::thread_pool_generic::worker_main (this=0xaaab01182c40, thread_var=0xaaab01192900) at /usr/src/debug/MariaDB-/src_0/tpool/tpool_generic.cc:544
      #8  0x0000aaaacbffbd00 in operator()<tpool::worker_data*, void> (__object=<optimized out>, this=<optimized out>) at /usr/include/c++/4.8.2/functional:601
      #9  _M_invoke<0ul, 1ul> (this=<optimized out>) at /usr/include/c++/4.8.2/functional:1732
      #10 operator() (this=<optimized out>) at /usr/include/c++/4.8.2/functional:1720
      #11 std::thread::_Impl<std::_Bind_simple<std::_Mem_fn<void (tpool::thread_pool_generic::*)(tpool::worker_data*)> (tpool::thread_pool_generic*, tpool::worker_data*)> >::_M_run() (this=<optimized out>) at /usr/include/c++/4.8.2/thread:115
      #12 0x0000ffff87104348 in ?? () from /lib64/libstdc++.so.6
      #13 0x0000ffff87627bb0 in start_thread () from /lib64/libpthread.so.0
      #14 0x0000ffff86edb4c0 in thread_start () from /lib64/libc.so.6
       
      Thread 3 (Thread 0xffff824cf070 (LWP 6716)):
      #0  0x0000ffff8762c1ec in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
      #1  0x0000aaaacc06ac38 in inline_mysql_cond_timedwait (that=0xaaaacd26f228 <COND_timer>, mutex=0xaaaacd26f1b8 <LOCK_timer>, src_file=0xaaaacc2e7dd8 "/home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.5.9/mysys/thr_timer.c", src_line=321, abstime=0xffff824ce7d8) at /usr/src/debug/MariaDB-/src_0/include/mysql/psi/mysql_thread.h:1259
      #2  timer_handler (arg=arg@entry=0x0) at /usr/src/debug/MariaDB-/src_0/mysys/thr_timer.c:321
      #3  0x0000aaaacbce291c in pfs_spawn_thread (arg=0xaaab01013d48) at /usr/src/debug/MariaDB-/src_0/storage/perfschema/pfs.cc:2201
      #4  0x0000ffff87627bb0 in start_thread () from /lib64/libpthread.so.0
      #5  0x0000ffff86edb4c0 in thread_start () from /lib64/libc.so.6
       
      Thread 2 (Thread 0xffff737df070 (LWP 6718)):
      #0  0x0000ffff86edb694 in epoll_pwait () from /lib64/libc.so.6
      #1  0x0000ffff81afd93c in asio::detail::epoll_reactor::run (this=0xaaab010af6c0, block=block@entry=true, ops=...) at /home/buildbot/buildbot/build/asio/asio/detail/impl/epoll_reactor.ipp:391
      #2  0x0000ffff81afe0f4 in asio::detail::task_io_service::do_run_one (this=this@entry=0xaaab010ae8c0, lock=..., this_thread=..., ec=...) at /home/buildbot/buildbot/build/asio/asio/detail/impl/task_io_service.ipp:355
      #3  0x0000ffff81afa33c in run (ec=..., this=0xaaab010ae8c0) at /home/buildbot/buildbot/build/asio/asio/detail/impl/task_io_service.ipp:148
      #4  run (this=<optimized out>) at /home/buildbot/buildbot/build/asio/asio/impl/io_service.ipp:58
      #5  gu::AsioIoService::run (this=this@entry=0xaaab010ae878) at galerautils/src/gu_asio.cpp:634
      #6  0x0000ffff81bbe8b8 in gcomm::AsioProtonet::event_loop (this=0xaaab010ae7d0, period=...) at gcomm/src/asio_protonet.cpp:103
      #7  0x0000ffff81be5380 in GCommConn::run (this=0xaaab010af140) at gcs/src/gcs_gcomm.cpp:433
      #8  0x0000ffff81bea040 in GCommConn::run_fn (arg=<optimized out>) at gcs/src/gcs_gcomm.cpp:174
      #9  0x0000ffff87627bb0 in start_thread () from /lib64/libpthread.so.0
      #10 0x0000ffff86edb4c0 in thread_start () from /lib64/libc.so.6
       
      Thread 1 (Thread 0xffff52538070 (LWP 6907)):
      #0  0x0000ffff8762cdfc in pthread_kill () from /lib64/libpthread.so.0
      #1  0x0000aaaacbaa8870 in handle_fatal_signal (sig=6) at /usr/src/debug/MariaDB-/src_0/sql/signal_handler.cc:330
      #2  <signal handler called>
      #3  0x0000ffff86e342c8 in raise () from /lib64/libc.so.6
      #4  0x0000ffff86e35940 in abort () from /lib64/libc.so.6
      #5  0x0000ffff81c3550c in galera::FSM<galera::TrxHandle::State, galera::TrxHandle::Transition, galera::EmptyGuard, galera::EmptyAction>::shift_to (this=this@entry=0xffff4cb0d6d8, state=state@entry=galera::TrxHandle::S_APPLYING, line=line@entry=1837) at galera/src/fsm.hpp:85
      #6  0x0000ffff81c2a54c in set_state (line=1837, state=galera::TrxHandle::S_APPLYING, this=0xffff4cb0d6d0) at galera/src/trx_handle.hpp:222
      #7  set_state (line=1837, state=galera::TrxHandle::S_APPLYING, this=0xffff4cb0d6d0) at galera/src/trx_handle.hpp:594
      #8  galera::ReplicatorSMM::to_isolation_end (this=0xaaab01058860, trx=..., err=<optimized out>) at galera/src/replicator_smm.cpp:1837
      #9  0x0000ffff81c430a0 in galera_to_execute_end (gh=<optimized out>, conn_id=45, err=0xffff52536280) at galera/src/wsrep_provider.cpp:1285
      #10 0x0000aaaacc0eea40 in wsrep::wsrep_provider_v26::leave_toi (this=<optimized out>, client_id=..., err=...) at /usr/src/debug/MariaDB-/src_0/wsrep-lib/src/wsrep_provider_v26.cpp:917
      #11 0x0000aaaacc0d4d60 in wsrep::client_state::poll_enter_toi (this=this@entry=0xffff4c0074d0, lock=..., keys=std::vector of length 1, capacity 1 = {...}, buffer=..., meta=..., flags=flags@entry=3, wait_until=wait_until@entry=..., timed_out=@0xffff52536590: false) at /usr/src/debug/MariaDB-/src_0/wsrep-lib/src/client_state.cpp:393
      #12 0x0000aaaacc0d60f4 in wsrep::client_state::enter_toi_local (this=this@entry=0xffff4c0074d0, keys=std::vector of length 1, capacity 1 = {...}, buffer=..., wait_until=wait_until@entry=...) at /usr/src/debug/MariaDB-/src_0/wsrep-lib/src/client_state.cpp:444
      #13 0x0000aaaacbd41ca0 in wsrep_TOI_begin (create_info=0x0, fk_tables=0x0, alter_info=0x0, table_list=<optimized out>, table=0x0, db=0x0, thd=0xffff4c001298) at /usr/src/debug/MariaDB-/src_0/sql/wsrep_mysqld.cc:2178
      #14 wsrep_to_isolation_begin (thd=thd@entry=0xffff4c001298, db_=db_@entry=0x0, table_=table_@entry=0x0, table_list=<optimized out>, alter_info=alter_info@entry=0x0, fk_tables=fk_tables@entry=0x0, create_info=create_info@entry=0x0) at /usr/src/debug/MariaDB-/src_0/sql/wsrep_mysqld.cc:2366
      #15 0x0000aaaacb8dbc30 in mysql_execute_command (thd=thd@entry=0xffff4c001298) at /usr/src/debug/MariaDB-/src_0/sql/sql_parse.cc:4972
      #16 0x0000aaaacb8dca44 in mysql_parse (thd=thd@entry=0xffff4c001298, rawbuf=rawbuf@entry=0xffff4c010610 "DROP TABLE t1", length=length@entry=13, parser_state=parser_state@entry=0xffff52536e40, is_com_multi=is_com_multi@entry=false, is_next_command=is_next_command@entry=false) at /usr/src/debug/MariaDB-/src_0/sql/sql_parse.cc:8062
      #17 0x0000aaaacb7caac0 in wsrep_mysql_parse (thd=thd@entry=0xffff4c001298, rawbuf=0xffff4c010610 "DROP TABLE t1", length=13, parser_state=parser_state@entry=0xffff52536e40, is_com_multi=is_com_multi@entry=false, is_next_command=is_next_command@entry=false) at /usr/src/debug/MariaDB-/src_0/sql/sql_parse.cc:7865
      #18 0x0000aaaacb8def58 in dispatch_command (command=command@entry=COM_QUERY, thd=thd@entry=0xffff4c001298, packet=packet@entry=0xffff4c0080f9 "DROP TABLE t1", packet_length=packet_length@entry=13, is_com_multi=is_com_multi@entry=false, is_next_command=is_next_command@entry=false) at /usr/src/debug/MariaDB-/src_0/sql/sql_parse.cc:1875
      #19 0x0000aaaacb8df734 in do_command (thd=0xffff4c001298) at /usr/src/debug/MariaDB-/src_0/sql/sql_parse.cc:1371
      #20 0x0000aaaacb9a7218 in do_handle_one_connection (connect=<optimized out>, connect@entry=0xaaab01ad0e08, put_in_cache=put_in_cache@entry=true) at /usr/src/debug/MariaDB-/src_0/sql/sql_connect.cc:1410
      #21 0x0000aaaacb9a74e0 in handle_one_connection (arg=arg@entry=0xaaab01ad0e08) at /usr/src/debug/MariaDB-/src_0/sql/sql_connect.cc:1312
      #22 0x0000aaaacbce291c in pfs_spawn_thread (arg=0xaaab01aedbc8) at /usr/src/debug/MariaDB-/src_0/storage/perfschema/pfs.cc:2201
      #23 0x0000ffff87627bb0 in start_thread () from /lib64/libpthread.so.0
      #24 0x0000ffff86edb4c0 in thread_start () from /lib64/libc.so.6
      

      Server logs.

      Attachments

        Activity

          People

            jplindst Jan Lindström (Inactive)
            stepan.patryshev Stepan Patryshev (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.