Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-32160

GCF-1060 test failure due to wsrep MDL conflict

    XMLWordPrintable

Details

    • Bug
    • Status: Stalled (View Workflow)
    • Critical
    • Resolution: Unresolved
    • 10.4
    • 10.4
    • Galera
    • None

    Description

      A crash that may be a regression after fixing MDEV-32051, but could also be an independent problem previously reported as MDEV-21677:

      CURRENT_TEST: galera_sr.GCF-1060
      2023-09-13  3:45:45 2 [Note] WSREP: MDL conflict, DDL vs SR
      schema:  test
      request: (2 	seqno 32 	wsrep (toi, exec, committed) cmd 0 8 	TRUNCATE TABLE t1)
      granted: (29 	seqno -1 	wsrep (high priority, exec, executing) cmd 0 161 	(null))
      2023-09-13  3:45:45 2 [Note] WSREP: MDL conflict, DDL vs SR
      schema:  test
      request: (2 	seqno 42 	wsrep (toi, exec, committed) cmd 0 8 	TRUNCATE TABLE t1)
      granted: (31 	seqno -1 	wsrep (high priority, exec, executing) cmd 0 161 	(null))
      2023-09-13  3:45:45 2 [Note] WSREP: MDL conflict, DDL vs SR
      schema:  test
      request: (2 	seqno 51 	wsrep (toi, exec, committed) cmd 0 8 	TRUNCATE TABLE t1)
      granted: (33 	seqno -1 	wsrep (high priority, exec, executing) cmd 0 161 	(null))
      2023-09-13  3:45:45 2 [Note] WSREP: MDL conflict, DDL vs SR
      schema:  test
      request: (2 	seqno 60 	wsrep (toi, exec, committed) cmd 0 8 	TRUNCATE TABLE t1)
      granted: (35 	seqno -1 	wsrep (high priority, exec, executing) cmd 0 161 	(null))
      2023-09-13  3:45:45 2 [Note] WSREP: MDL conflict, DDL vs SR
      schema:  test
      request: (2 	seqno 79 	wsrep (toi, exec, committed) cmd 0 8 	TRUNCATE TABLE t1)
      granted: (38 	seqno -1 	wsrep (high priority, exec, executing) cmd 0 161 	(null))
      2023-09-13  3:45:45 25 [Note] WSREP: MDL conflict, DDL vs SR
      schema:  test
      request: (2 	seqno 79 	wsrep (toi, exec, committed) cmd 0 8 	TRUNCATE TABLE t1)
      granted: (38 	seqno -1 	wsrep (high priority, exec, executing) cmd 0 161 	(null))
      230913  4:00:49 [ERROR] mysqld got signal 6 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
       
      To report this bug, see https://mariadb.com/kb/en/reporting-bugs
       
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed, 
      something is definitely wrong and this may fail.
       
      Server version: 10.4.32-MariaDB-log source revision: 5fe8d0d559eec0ec04a9d8ef679fa9c4eb308ff6
      key_buffer_size=1048576
      read_buffer_size=131072
      max_used_connections=1
      max_threads=153
      thread_count=10
      It is possible that mysqld could use up to 
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 63557 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x0
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x0 thread_stack 0x49000
      /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55e428355c1e]
      /usr/sbin/mysqld(handle_fatal_signal+0x515)[0x55e427da0c35]
      /lib64/libpthread.so.0(+0x168c0)[0x7fdd5ce228c0]
      /lib64/libc.so.6(__poll+0x4b)[0x7fdd5c1b01c9]
      /usr/sbin/mysqld(_Z26handle_connections_socketsv+0x1dc)[0x55e427ae4c8c]
      /usr/sbin/mysqld(_Z11mysqld_mainiPPc+0xdde)[0x55e427ae5ebe]
      /lib64/libc.so.6(__libc_start_main+0xef)[0x7fdd5c0da29d]
      /usr/sbin/mysqld(_start+0x2a)[0x55e427ad980a]
      The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
      information that should help you find out what is causing the crash.
      Writing a core file...
      Working directory at /dev/shm/var/1/mysqld.2/data
      Resource Limits:
      Limit                     Soft Limit           Hard Limit           Units     
      Max cpu time              unlimited            unlimited            seconds   
      Max file size             unlimited            unlimited            bytes     
      Max data size             unlimited            unlimited            bytes     
      Max stack size            8388608              unlimited            bytes     
      Max core file size        unlimited            unlimited            bytes     
      Max resident set          unlimited            unlimited            bytes     
      Max processes             31710                31710                processes 
      Max open files            1024                 1024                 files     
      Max locked memory         65536                65536                bytes     
      Max address space         unlimited            unlimited            bytes     
      Max file locks            unlimited            unlimited            locks     
      Max pending signals       31710                31710                signals   
      Max msgqueue size         819200               819200               bytes     
      Max nice priority         0                    0                    
      Max realtime priority     0                    0                    
      Max realtime timeout      unlimited            unlimited            us        
      Core pattern: core

      stack trace:

      [New LWP 13408]
      [New LWP 14165]
      [New LWP 13410]
      [New LWP 13411]
      [New LWP 14150]
      [New LWP 14159]
      [New LWP 14152]
      [New LWP 14155]
      [New LWP 14154]
      [New LWP 14173]
      [New LWP 14153]
      [New LWP 14160]
      [New LWP 14163]
      [New LWP 14171]
      [New LWP 14167]
      [New LWP 14168]
      [New LWP 14172]
      [New LWP 14177]
      [New LWP 14161]
      [New LWP 14166]
      [New LWP 14162]
      [New LWP 14170]
      [New LWP 14164]
      [New LWP 14169]
      [New LWP 14134]
      [New LWP 14174]
      [New LWP 13456]
      [New LWP 13457]
      [New LWP 14156]
      [New LWP 13412]
      [New LWP 14151]
      [New LWP 13458]
      [Thread debugging using libthread_db enabled]
      Using host libthread_db library "/lib64/libthread_db.so.1".
      Core was generated by `/usr/sbin/mysqld --defaults-group-suffix=.2 --defaults-file=/dev/shm/var/1/my.c'.
      Program terminated with signal SIGABRT, Aborted.
      #0  0x00007fdd5ce1f131 in pthread_kill () from /lib64/libpthread.so.0
      [Current thread is 1 (Thread 0x7fdd5e8418c0 (LWP 13408))]
       
      Thread 32 (Thread 0x7fdd545c3700 (LWP 13458)):
      #0  0x00007fdd5ce1d70c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x000055e427c9c1b4 in inline_mysql_cond_wait (src_file=0x55e42842a580 "/home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.4.32/sql/mdl.cc", src_line=1184, mutex=0x7fdd38000d90, that=0x7fdd38000dc0) at /usr/src/debug/MariaDB-10.4.32/src_0/include/mysql/psi/mysql_thread.h:1174
      #2  MDL_wait::timed_wait (this=this@entry=0x7fdd38000d90, owner=0x7fdd38000d20, abs_timeout=abs_timeout@entry=0x7fdd545bf0e0, set_status_on_timeout=set_status_on_timeout@entry=false, wait_state_name=<optimized out>) at /usr/src/debug/MariaDB-10.4.32/src_0/sql/mdl.cc:1184
      #3  0x000055e427c9dda3 in MDL_context::acquire_lock (this=this@entry=0x7fdd38000d90, mdl_request=<optimized out>, lock_wait_timeout=lock_wait_timeout@entry=86400) at /usr/src/debug/MariaDB-10.4.32/src_0/sql/mdl.cc:2377
      #4  0x000055e427c9e405 in MDL_context::acquire_locks (this=this@entry=0x7fdd38000d90, mdl_requests=mdl_requests@entry=0x7fdd545bf1e0, lock_wait_timeout=lock_wait_timeout@entry=86400) at /usr/src/debug/MariaDB-10.4.32/src_0/sql/mdl.cc:2503
      #5  0x000055e427b4459b in lock_table_names (thd=thd@entry=0x7fdd38000c48, options=..., tables_start=tables_start@entry=0x7fdd3800bde0, tables_end=tables_end@entry=0x0, lock_wait_timeout=86400, flags=flags@entry=0) at /usr/src/debug/MariaDB-10.4.32/src_0/sql/sql_base.cc:4151
      #6  0x000055e427caf68f in lock_table_names (flags=0, lock_wait_timeout=<optimized out>, table_list_end=0x0, table_list=0x7fdd3800bde0, thd=0x7fdd38000c48) at /usr/src/debug/MariaDB-10.4.32/src_0/sql/sql_base.h:247
      #7  Sql_cmd_truncate_table::lock_table (this=this@entry=0x7fdd3800c4d8, thd=thd@entry=0x7fdd38000c48, table_ref=table_ref@entry=0x7fdd3800bde0, hton_can_recreate=hton_can_recreate@entry=0x7fdd545bf460) at /usr/src/debug/MariaDB-10.4.32/src_0/sql/sql_truncate.cc:311
      #8  0x000055e427caf83e in Sql_cmd_truncate_table::truncate_table (this=this@entry=0x7fdd3800c4d8, thd=thd@entry=0x7fdd38000c48, table_ref=table_ref@entry=0x7fdd3800bde0) at /usr/src/debug/MariaDB-10.4.32/src_0/sql/sql_truncate.cc:438
      #9  0x000055e427cafbce in Sql_cmd_truncate_table::execute (this=0x7fdd3800c4d8, thd=0x7fdd38000c48) at /usr/src/debug/MariaDB-10.4.32/src_0/sql/sql_truncate.cc:528
      #10 0x000055e427ba642b in mysql_execute_command (thd=thd@entry=0x7fdd38000c48) at /usr/src/debug/MariaDB-10.4.32/src_0/sql/sql_parse.cc:6218
      #11 0x000055e427bacbba in mysql_parse (thd=0x7fdd38000c48, rawbuf=<optimized out>, length=17, parser_state=parser_state@entry=0x7fdd545c1580, is_com_multi=is_com_multi@entry=false, is_next_command=is_next_command@entry=false) at /usr/src/debug/MariaDB-10.4.32/src_0/sql/sql_parse.cc:8012
      #12 0x000055e427e9f7b7 in Query_log_event::do_apply_event (this=0x7fdd2808e368, rgi=0x7fdd3801c6a0, query_arg=0x7fdd38033b73 "TRUNCATE TABLE t1", q_len_arg=<optimized out>) at /usr/src/debug/MariaDB-10.4.32/src_0/sql/log_event.cc:5744
      #13 0x000055e428038db8 in Log_event::apply_event (rgi=0x7fdd3801c6a0, this=0x7fdd2808e368) at /usr/src/debug/MariaDB-10.4.32/src_0/sql/log_event.h:1492
      #14 wsrep_apply_events (thd=thd@entry=0x7fdd38000c48, rli=<optimized out>, events_buf=<optimized out>, buf_len=0) at /usr/src/debug/MariaDB-10.4.32/src_0/sql/wsrep_applier.cc:200
      #15 0x000055e42801e6d9 in Wsrep_high_priority_service::apply_toi (this=0x7fdd545c2bf0, ws_meta=..., data=...) at /usr/src/debug/MariaDB-10.4.32/src_0/sql/wsrep_high_priority_service.cc:394
      #16 0x000055e4283df2f3 in apply_toi (data=..., ws_meta=..., ws_handle=..., high_priority_service=..., provider=...) at /usr/src/debug/MariaDB-10.4.32/src_0/wsrep-lib/src/server_state.cpp:466
      #17 wsrep::server_state::on_apply (this=<optimized out>, high_priority_service=..., ws_handle=..., ws_meta=..., data=...) at /usr/src/debug/MariaDB-10.4.32/src_0/wsrep-lib/src/server_state.cpp:1118
      #18 0x000055e4283f1f74 in wsrep::high_priority_service::apply (data=..., ws_meta=..., ws_handle=..., this=0x7fdd545c2bf0) at /usr/src/debug/MariaDB-10.4.32/src_0/wsrep-lib/include/wsrep/high_priority_service.hpp:47
      #19 (anonymous namespace)::apply_cb (ctx=0x7fdd545c2bf0, wsh=<optimized out>, flags=<optimized out>, buf=<optimized out>, meta=<optimized out>, exit_loop=0x7fdd545c1e6f) at /usr/src/debug/MariaDB-10.4.32/src_0/wsrep-lib/src/wsrep_provider_v26.cpp:507
      #20 0x00007fdd5610a137 in galera::TrxHandleSlave::apply (this=this@entry=0x7fdd38056a10, recv_ctx=recv_ctx@entry=0x7fdd545c2bf0, apply_cb=0x55e4283f1e40 <(anonymous namespace)::apply_cb(void*, wsrep_ws_handle_t const*, uint32_t, wsrep_buf_t const*, wsrep_trx_meta_t const*, wsrep_bool_t*)>, meta=..., exit_loop=exit_loop@entry=@0x7fdd545c1e6f: false) at /home/buildbot/buildbot/build/galera/src/trx_handle.cpp:396
      #21 0x00007fdd5611f8d0 in galera::ReplicatorSMM::apply_trx (this=this@entry=0x55e42a1fd770, recv_ctx=recv_ctx@entry=0x7fdd545c2bf0, ts=...) at /home/buildbot/buildbot/build/galera/src/replicator_smm.cpp:516
      #22 0x00007fdd56125852 in galera::ReplicatorSMM::process_trx (this=0x55e42a1fd770, recv_ctx=0x7fdd545c2bf0, ts_ptr=...) at /home/buildbot/buildbot/build/galera/src/replicator_smm.cpp:2124
      #23 0x00007fdd5615ba99 in galera::GcsActionSource::process_writeset (this=0x55e42a230190, recv_ctx=0x7fdd545c2bf0, act=..., exit_loop=@0x7fdd545c27ff: false) at /home/buildbot/buildbot/build/galera/src/gcs_action_source.cpp:62
      #24 0x00007fdd5615ca32 in galera::GcsActionSource::dispatch (this=this@entry=0x55e42a230190, recv_ctx=recv_ctx@entry=0x7fdd545c2bf0, act=..., exit_loop=@0x7fdd545c27ff: false) at /home/buildbot/buildbot/build/galera/src/gcs_action_source.cpp:110
      #25 0x00007fdd5615d97b in galera::GcsActionSource::process (this=0x55e42a230190, recv_ctx=0x7fdd545c2bf0, exit_loop=@0x7fdd545c27ff: false) at /home/buildbot/buildbot/build/galera/src/gcs_action_source.cpp:186
      #26 0x00007fdd56120da0 in galera::ReplicatorSMM::async_recv (this=0x55e42a1fd770, recv_ctx=0x7fdd545c2bf0) at /home/buildbot/buildbot/build/galera/src/replicator_smm.cpp:402
      #27 0x00007fdd560f61c8 in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at /home/buildbot/buildbot/build/galera/src/wsrep_provider.cpp:264
      #28 0x000055e4283f273e in wsrep::wsrep_provider_v26::run_applier (this=<optimized out>, applier_ctx=<optimized out>) at /usr/src/debug/MariaDB-10.4.32/src_0/wsrep-lib/src/wsrep_provider_v26.cpp:858
      #29 0x000055e42803b08f in wsrep_replication_process (thd=0x7fdd38000c48, arg=<optimized out>) at /usr/src/debug/MariaDB-10.4.32/src_0/sql/wsrep_thd.cc:57
      #30 0x000055e428028de2 in start_wsrep_THD (arg=arg@entry=0x55e42a236310) at /usr/src/debug/MariaDB-10.4.32/src_0/sql/wsrep_mysqld.cc:3041
      #31 0x000055e42800e2ba in pfs_spawn_thread (arg=0x55e42a220168) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/perfschema/pfs.cc:1869
      #32 0x00007fdd5ce166ea in start_thread () from /lib64/libpthread.so.0
      #33 0x00007fdd5c1bca6f in clone () from /lib64/libc.so.6
       
      Thread 31 (Thread 0x7fdd26ffe700 (LWP 14151)):
      #0  0x00007fdd5d9be734 in ?? () from /lib64/libaio.so.1
      #1  0x000055e4280d8be1 in LinuxAIOHandler::collect (this=this@entry=0x7fdd26ffdae0) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0file.cc:1803
      #2  0x000055e4280d9256 in LinuxAIOHandler::poll (this=this@entry=0x7fdd26ffdae0, m1=m1@entry=0x7fdd26ffdb80, m2=m2@entry=0x7fdd26ffdb88, request=request@entry=0x7fdd26ffdb90) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0file.cc:1964
      #3  0x000055e4280e1704 in os_aio_linux_handler (request=0x7fdd26ffdb90, m2=0x7fdd26ffdb88, m1=0x7fdd26ffdb80, global_segment=<optimized out>) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0file.cc:2018
      #4  os_aio_handler (segment=segment@entry=1, m1=m1@entry=0x7fdd26ffdb80, m2=m2@entry=0x7fdd26ffdb88, request=request@entry=0x7fdd26ffdb90) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0file.cc:5296
      #5  0x000055e428265228 in fil_aio_wait (segment=segment@entry=1) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/fil/fil0fil.cc:4330
      #6  0x000055e4281753d0 in io_handler_thread (arg=<optimized out>) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/srv/srv0start.cc:324
      #7  0x00007fdd5ce166ea in start_thread () from /lib64/libpthread.so.0
      #8  0x00007fdd5c1bca6f in clone () from /lib64/libc.so.6
       
      Thread 30 (Thread 0x7fdd4ffff700 (LWP 13412)):
      #0  0x00007fdd5c1bcdff in epoll_wait () from /lib64/libc.so.6
      #1  0x00007fdd562c5ffa in asio::detail::epoll_reactor::run (this=0x55e42a239620, usec=<optimized out>, ops=...) at /home/buildbot/buildbot/build/asio/asio/detail/impl/epoll_reactor.ipp:470
      #2  0x00007fdd562c66b0 in asio::detail::scheduler::do_run_one (this=this@entry=0x55e42a2341e0, lock=..., this_thread=..., ec=std::error_code = {std::_V2::error_category: 0}) at /home/buildbot/buildbot/build/asio/asio/detail/impl/scheduler.ipp:430
      #3  0x00007fdd562c6a49 in asio::detail::scheduler::run (this=this@entry=0x55e42a2341e0, ec=std::error_code = {std::_V2::error_category: 0}) at /home/buildbot/buildbot/build/asio/asio/detail/impl/scheduler.ipp:199
      #4  0x00007fdd562c0cbc in asio::io_context::run (this=<optimized out>) at /home/buildbot/buildbot/build/asio/asio/impl/io_context.ipp:62
      #5  gu::AsioIoService::run (this=this@entry=0x55e42a2356a8) at /home/buildbot/buildbot/build/galerautils/src/gu_asio.cpp:750
      #6  0x00007fdd561c3fd6 in gcomm::AsioProtonet::event_loop (this=0x55e42a2355f0, period=...) at /home/buildbot/buildbot/build/gcomm/src/asio_protonet.cpp:103
      #7  0x00007fdd561a2021 in GCommConn::run (this=0x55e42a239150) at /home/buildbot/buildbot/build/gcs/src/gcs_gcomm.cpp:439
      #8  0x00007fdd561a25b6 in run_fn (arg=<optimized out>) at /home/buildbot/buildbot/build/gcs/src/gcs_gcomm.cpp:307
      #9  0x00007fdd5ce166ea in start_thread () from /lib64/libpthread.so.0
      #10 0x00007fdd5c1bca6f in clone () from /lib64/libc.so.6
       
      Thread 29 (Thread 0x7fdd247f9700 (LWP 14156)):
      #0  0x00007fdd5ce1da5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x000055e4280e2852 in os_event::timed_wait (this=this@entry=0x55e42a335e30, abstime=abstime@entry=0x7fdd247f8a70) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0event.cc:275
      #2  0x000055e4280e2eb1 in os_event::wait_time_low (reset_sig_count=908, time_in_usec=1000000, this=0x55e42a335e30) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0event.cc:385
      #3  os_event_wait_time_low (event=0x55e42a335e30, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=reset_sig_count@entry=908) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0event.cc:485
      #4  0x000055e428210a7b in pc_sleep_if_needed (cur_time=2812260, sig_count=908, next_loop_time=2813260) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/buf/buf0flu.cc:2611
      #5  buf_flush_page_cleaner_coordinator () at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/buf/buf0flu.cc:3069
      #6  0x00007fdd5ce166ea in start_thread () from /lib64/libpthread.so.0
      #7  0x00007fdd5c1bca6f in clone () from /lib64/libc.so.6
       
      Thread 28 (Thread 0x7fdd5460e700 (LWP 13457)):
      #0  0x00007fdd5ce1d70c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x000055e42803a193 in inline_mysql_cond_wait (src_line=69, src_file=0x55e42857ad88 "/home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.4.32/sql/wsrep_thd.h", mutex=0x7fdd44012e28, that=0x7fdd44012e58) at /usr/src/debug/MariaDB-10.4.32/src_0/include/mysql/psi/mysql_thread.h:1185
      #2  Wsrep_thd_queue::pop_front (this=0x7fdd44012dd0) at /usr/src/debug/MariaDB-10.4.32/src_0/sql/wsrep_thd.h:69
      #3  wsrep_rollback_process (rollbacker=0x7fdd44000c48, arg=<optimized out>) at /usr/src/debug/MariaDB-10.4.32/src_0/sql/wsrep_thd.cc:236
      #4  0x000055e428028de2 in start_wsrep_THD (arg=arg@entry=0x55e42a2362f0) at /usr/src/debug/MariaDB-10.4.32/src_0/sql/wsrep_mysqld.cc:3041
      #5  0x000055e42800e2ba in pfs_spawn_thread (arg=0x55e42a21fe08) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/perfschema/pfs.cc:1869
      #6  0x00007fdd5ce166ea in start_thread () from /lib64/libpthread.so.0
      #7  0x00007fdd5c1bca6f in clone () from /lib64/libc.so.6
       
      Thread 27 (Thread 0x7fdd4f7fe700 (LWP 13456)):
      #0  0x00007fdd5ce1d70c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00007fdd5619f9df in gu_cond_wait_SYS (mutex=0x55e42a239368, cond=0x55e42a2393a8) at /home/buildbot/buildbot/build/galerautils/src/gu_threads.h:251
      #2  gu::Lock::wait (this=<synthetic pointer>, cond=...) at /home/buildbot/buildbot/build/galerautils/src/gu_lock.hpp:46
      #3  RecvBuf::front (timeout=..., this=0x55e42a239368) at /home/buildbot/buildbot/build/gcs/src/gcs_gcomm.cpp:116
      #4  gcomm_recv (backend=<optimized out>, msg=0x55e42a207af8, timeout=9223372035999999999) at /home/buildbot/buildbot/build/gcs/src/gcs_gcomm.cpp:614
      #5  0x00007fdd56189040 in core_msg_recv (timeout=9223372035999999999, recv_msg=0x55e42a207af8, backend=0x55e42a207ca0) at /home/buildbot/buildbot/build/gcs/src/gcs_core.cpp:484
      #6  gcs_core_recv (conn=0x55e42a207a80, recv_act=recv_act@entry=0x7fdd4f7fdc70, timeout=9223372035999999999) at /home/buildbot/buildbot/build/gcs/src/gcs_core.cpp:1205
      #7  0x00007fdd56182c8b in gcs_recv_thread (arg=0x55e42a207790) at /home/buildbot/buildbot/build/gcs/src/gcs.cpp:1454
      #8  0x00007fdd5ce166ea in start_thread () from /lib64/libpthread.so.0
      #9  0x00007fdd5c1bca6f in clone () from /lib64/libc.so.6
       
      Thread 26 (Thread 0x7fdd54060700 (LWP 14174)):
      #0  0x00007fdd5c0f0ccc in sigtimedwait () from /lib64/libc.so.6
      #1  0x000055e427adbebb in my_sigwait (code=<synthetic pointer>, sig=0x7fdd5405fbe8, set=0x7fdd5405fbf0) at /usr/src/debug/MariaDB-10.4.32/src_0/include/my_pthread.h:196
      #2  signal_hand (arg=arg@entry=0x0) at /usr/src/debug/MariaDB-10.4.32/src_0/sql/mysqld.cc:3254
      #3  0x000055e42800e2ba in pfs_spawn_thread (arg=0x55e42a566ca8) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/perfschema/pfs.cc:1869
      #4  0x00007fdd5ce166ea in start_thread () from /lib64/libpthread.so.0
      #5  0x00007fdd5c1bca6f in clone () from /lib64/libc.so.6
       
      Thread 25 (Thread 0x7fdd4d051700 (LWP 14134)):
      #0  0x00007fdd5ce1da5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x000055e427f6cd1c in inline_mysql_cond_timedwait (src_file=0x55e428556a88 "/home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.4.32/storage/maria/ma_servicethread.c", src_line=116, abstime=0x7fdd4d050c30, mutex=0x55e4293ba620 <LOCK_checkpoint>, that=0x55e4293ba5e0 <COND_checkpoint>) at /usr/src/debug/MariaDB-10.4.32/src_0/include/mysql/psi/mysql_thread.h:1211
      #2  my_service_thread_sleep (control=control@entry=0x55e428afdfc0 <checkpoint_control>, sleep_time=sleep_time@entry=30000000000) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/maria/ma_servicethread.c:115
      #3  0x000055e427f60c20 in ma_checkpoint_background (arg=arg@entry=0x1e) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/maria/ma_checkpoint.c:707
      #4  0x000055e42800e2ba in pfs_spawn_thread (arg=0x55e42a2f2208) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/perfschema/pfs.cc:1869
      #5  0x00007fdd5ce166ea in start_thread () from /lib64/libpthread.so.0
      #6  0x00007fdd5c1bca6f in clone () from /lib64/libc.so.6
       
      Thread 24 (Thread 0x7fdd12ffd700 (LWP 14169)):
      #0  0x00007fdd5ce1d70c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x000055e4280e2fc0 in os_event::wait (this=0x55e42a335dc0) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0event.cc:158
      #2  os_event::wait_low (reset_sig_count=1, this=0x55e42a335dc0) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0event.cc:325
      #3  os_event_wait_low (event=0x55e42a335dc0, reset_sig_count=reset_sig_count@entry=0) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0event.cc:502
      #4  0x000055e42820664c in buf_dump_thread () at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/buf/buf0dump.cc:832
      #5  0x00007fdd5ce166ea in start_thread () from /lib64/libpthread.so.0
      #6  0x00007fdd5c1bca6f in clone () from /lib64/libc.so.6
       
      Thread 23 (Thread 0x7fdd215ee700 (LWP 14164)):
      #0  0x00007fdd5c180cc1 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
      #1  0x00007fdd5c1869c3 in nanosleep () from /lib64/libc.so.6
      #2  0x000055e4280e3230 in os_thread_sleep (tm=tm@entry=1000000) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0thread.cc:231
      #3  0x000055e428170b08 in srv_master_sleep () at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/srv/srv0srv.cc:2301
      #4  srv_master_thread (arg=<optimized out>) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/srv/srv0srv.cc:2341
      #5  0x00007fdd5ce166ea in start_thread () from /lib64/libpthread.so.0
      #6  0x00007fdd5c1bca6f in clone () from /lib64/libc.so.6
       
      Thread 22 (Thread 0x7fdd127fc700 (LWP 14170)):
      #0  0x00007fdd5c180cc1 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
      #1  0x00007fdd5c1869c3 in nanosleep () from /lib64/libc.so.6
      #2  0x000055e4280e3230 in os_thread_sleep (tm=tm@entry=1000000) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0thread.cc:231
      #3  0x000055e4281f0232 in btr_defragment_thread () at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/btr/btr0defragment.cc:717
      #4  0x00007fdd5ce166ea in start_thread () from /lib64/libpthread.so.0
      #5  0x00007fdd5c1bca6f in clone () from /lib64/libc.so.6
       
      Thread 21 (Thread 0x7fdd225f0700 (LWP 14162)):
      #0  0x00007fdd5ce1da5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x000055e4280e2852 in os_event::timed_wait (this=this@entry=0x55e42a4ff4e0, abstime=abstime@entry=0x7fdd225efc00) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0event.cc:275
      #2  0x000055e4280e2eb1 in os_event::wait_time_low (reset_sig_count=1, time_in_usec=10000000, this=0x55e42a4ff4e0) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0event.cc:385
      #3  os_event_wait_time_low (event=0x55e42a4ff4e0, time_in_usec=time_in_usec@entry=10000000, reset_sig_count=reset_sig_count@entry=0) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0event.cc:485
      #4  0x000055e4282580b7 in dict_stats_thread () at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/dict/dict0stats_bg.cc:492
      #5  0x00007fdd5ce166ea in start_thread () from /lib64/libpthread.so.0
      #6  0x00007fdd5c1bca6f in clone () from /lib64/libc.so.6
       
      Thread 20 (Thread 0x7fdd13fff700 (LWP 14166)):
      #0  0x00007fdd5ce1d70c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x000055e4280e2fc0 in os_event::wait (this=0x55e42a335b90) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0event.cc:158
      #2  os_event::wait_low (reset_sig_count=364, this=0x55e42a335b90) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0event.cc:325
      #3  os_event_wait_low (event=0x55e42a335b90, reset_sig_count=reset_sig_count@entry=0) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0event.cc:502
      #4  0x000055e4281719d9 in srv_resume_thread (sig_count=0, wait=true, timeout_usec=0, slot=0x55e428b0d5b8 <srv_sys+248>) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/srv/srv0srv.cc:897
      #5  srv_worker_thread (arg=<optimized out>) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/srv/srv0srv.cc:2484
      #6  0x00007fdd5ce166ea in start_thread () from /lib64/libpthread.so.0
      #7  0x00007fdd5c1bca6f in clone () from /lib64/libc.so.6
       
      Thread 19 (Thread 0x7fdd22df1700 (LWP 14161)):
      #0  0x00007fdd5ce1da5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x000055e4280e2852 in os_event::timed_wait (this=this@entry=0x55e42a335d50, abstime=abstime@entry=0x7fdd22df0c40) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0event.cc:275
      #2  0x000055e4280e2eb1 in os_event::wait_time_low (reset_sig_count=1, time_in_usec=5000000, this=0x55e42a335d50) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0event.cc:385
      #3  os_event_wait_time_low (event=0x55e42a335d50, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=<optimized out>) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0event.cc:485
      #4  0x000055e4281743ff in srv_monitor_thread () at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/srv/srv0srv.cc:1664
      #5  0x00007fdd5ce166ea in start_thread () from /lib64/libpthread.so.0
      #6  0x00007fdd5c1bca6f in clone () from /lib64/libc.so.6
       
      Thread 18 (Thread 0x7fdd4c34f700 (LWP 14177)):
      #0  0x00007fdd5ce1d70c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00007fdd5612da3b in gu_cond_wait_SYS (mutex=0x55e42a1feb08, cond=0x7fdd2800aa10) at /home/buildbot/buildbot/build/galerautils/src/gu_threads.h:251
      #2  gu::Lock::wait (this=<synthetic pointer>, cond=...) at /home/buildbot/buildbot/build/galerautils/src/gu_lock.hpp:46
      #3  galera::Monitor<galera::ReplicatorSMM::CommitOrder>::enter (this=this@entry=0x55e42a1feb08, obj=...) at /home/buildbot/buildbot/build/galera/src/monitor.hpp:194
      #4  0x00007fdd5611d41b in galera::ReplicatorSMM::commit_order_enter_local (this=this@entry=0x55e42a1fd770, trx=...) at /home/buildbot/buildbot/build/galera/src/replicator_smm.cpp:1241
      #5  0x00007fdd560fd2f6 in galera_commit_order_enter (gh=<optimized out>, ws_handle=<optimized out>, meta=<optimized out>) at /home/buildbot/buildbot/build/galera/src/wsrep_provider.cpp:758
      #6  0x000055e4283f36a0 in wsrep::wsrep_provider_v26::commit_order_enter (this=<optimized out>, ws_handle=..., ws_meta=...) at /usr/src/debug/MariaDB-10.4.32/src_0/wsrep-lib/src/wsrep_provider_v26.cpp:955
      #7  0x000055e4283e6347 in wsrep::transaction::release_commit_order (this=this@entry=0x7fdd28007d98, lock=...) at /usr/src/debug/MariaDB-10.4.32/src_0/wsrep-lib/src/transaction.cpp:786
      #8  0x000055e4283f02ab in wsrep::transaction::after_statement (this=this@entry=0x7fdd28007d98, lock=...) at /usr/src/debug/MariaDB-10.4.32/src_0/wsrep-lib/src/transaction.cpp:910
      #9  0x000055e4283cfc58 in wsrep::client_state::after_statement (this=this@entry=0x7fdd28007d30) at /usr/src/debug/MariaDB-10.4.32/src_0/wsrep-lib/src/client_state.cpp:265
      #10 0x000055e427b9d26c in wsrep_after_statement (thd=0x7fdd28001d48) at /usr/src/debug/MariaDB-10.4.32/src_0/sql/wsrep_trans_observer.h:440
      #11 0x000055e427bad494 in wsrep_mysql_parse (thd=0x7fdd28001d48, rawbuf=0x7fdd28010230 "INSERT INTO t1 VALUE (4, 'z')", length=29, parser_state=0x7fdd4c34e3b0, is_com_multi=false, is_next_command=false) at /usr/src/debug/MariaDB-10.4.32/src_0/sql/sql_parse.cc:7844
      #12 0x000055e427bb0149 in dispatch_command (command=command@entry=COM_QUERY, thd=thd@entry=0x7fdd28001d48, packet=packet@entry=0x7fdd28036409 "INSERT INTO t1 VALUE (4, 'z');", packet_length=packet_length@entry=30, is_com_multi=is_com_multi@entry=false, is_next_command=is_next_command@entry=false) at /usr/src/debug/MariaDB-10.4.32/src_0/sql/sql_parse.cc:1843
      #13 0x000055e427bb0656 in do_command (thd=0x7fdd28001d48) at /usr/src/debug/MariaDB-10.4.32/src_0/sql/sql_parse.cc:1378
      #14 0x000055e427c9438e in do_handle_one_connection (connect=connect@entry=0x55e42a5e0118) at /usr/src/debug/MariaDB-10.4.32/src_0/sql/sql_connect.cc:1420
      #15 0x000055e427c9444d in handle_one_connection (arg=arg@entry=0x55e42a5e0118) at /usr/src/debug/MariaDB-10.4.32/src_0/sql/sql_connect.cc:1324
      #16 0x000055e42800e2ba in pfs_spawn_thread (arg=0x55e42a566288) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/perfschema/pfs.cc:1869
      #17 0x00007fdd5ce166ea in start_thread () from /lib64/libpthread.so.0
      #18 0x00007fdd5c1bca6f in clone () from /lib64/libc.so.6
       
      Thread 17 (Thread 0x7fdd117fa700 (LWP 14172)):
      #0  0x00007fdd5ce1d70c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x000055e42805d1b3 in inline_mysql_cond_wait (src_file=0x55e42857d4c8 "/home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.4.32/storage/innobase/handler/ha_innodb.cc", src_line=310, mutex=0x7fdd117f9ca0, that=0x7fdd117f9cd0) at /usr/src/debug/MariaDB-10.4.32/src_0/include/mysql/psi/mysql_thread.h:1185
      #2  thd_destructor_proxy () at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/handler/ha_innodb.cc:310
      #3  0x000055e42800e2ba in pfs_spawn_thread (arg=0x55e42a4aa828) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/perfschema/pfs.cc:1869
      #4  0x00007fdd5ce166ea in start_thread () from /lib64/libpthread.so.0
      #5  0x00007fdd5c1bca6f in clone () from /lib64/libc.so.6
       
      Thread 16 (Thread 0x7fdd137fe700 (LWP 14168)):
      #0  0x00007fdd5ce1d70c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x000055e4280e2fc0 in os_event::wait (this=0x55e42a335c00) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0event.cc:158
      #2  os_event::wait_low (reset_sig_count=462, this=0x55e42a335c00) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0event.cc:325
      #3  os_event_wait_low (event=0x55e42a335c00, reset_sig_count=reset_sig_count@entry=0) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0event.cc:502
      #4  0x000055e4281719d9 in srv_resume_thread (sig_count=0, wait=true, timeout_usec=0, slot=0x55e428b0d5f0 <srv_sys+304>) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/srv/srv0srv.cc:897
      #5  srv_worker_thread (arg=<optimized out>) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/srv/srv0srv.cc:2484
      #6  0x00007fdd5ce166ea in start_thread () from /lib64/libpthread.so.0
      #7  0x00007fdd5c1bca6f in clone () from /lib64/libc.so.6
       
      Thread 15 (Thread 0x7fdd0b7fe700 (LWP 14167)):
      #0  0x00007fdd5ce1d70c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x000055e4280e2fc0 in os_event::wait (this=0x55e42a335c70) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0event.cc:158
      #2  os_event::wait_low (reset_sig_count=243, this=0x55e42a335c70) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0event.cc:325
      #3  os_event_wait_low (event=0x55e42a335c70, reset_sig_count=reset_sig_count@entry=0) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0event.cc:502
      #4  0x000055e4281719d9 in srv_resume_thread (sig_count=0, wait=true, timeout_usec=0, slot=0x55e428b0d628 <srv_sys+360>) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/srv/srv0srv.cc:897
      #5  srv_worker_thread (arg=<optimized out>) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/srv/srv0srv.cc:2484
      #6  0x00007fdd5ce166ea in start_thread () from /lib64/libpthread.so.0
      #7  0x00007fdd5c1bca6f in clone () from /lib64/libc.so.6
       
      Thread 14 (Thread 0x7fdd11ffb700 (LWP 14171)):
      #0  0x00007fdd5ce1d70c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x000055e4280e2fc0 in os_event::wait (this=0x55e42a335ea0) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0event.cc:158
      #2  os_event::wait_low (reset_sig_count=1, this=0x55e42a335ea0) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0event.cc:325
      #3  os_event_wait_low (event=0x55e42a335ea0, reset_sig_count=reset_sig_count@entry=0) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0event.cc:502
      #4  0x000055e4281fedb2 in buf_resize_thread () at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/buf/buf0buf.cc:2965
      #5  0x00007fdd5ce166ea in start_thread () from /lib64/libpthread.so.0
      #6  0x00007fdd5c1bca6f in clone () from /lib64/libc.so.6
       
      Thread 13 (Thread 0x7fdd21def700 (LWP 14163)):
      #0  0x00007fdd5ce1da5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x000055e4280e2852 in os_event::timed_wait (this=this@entry=0x55e42a495f50, abstime=abstime@entry=0x7fdd21deea20) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0event.cc:275
      #2  0x000055e4280e2eb1 in os_event::wait_time_low (reset_sig_count=1, time_in_usec=224, this=0x55e42a495f50) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0event.cc:385
      #3  os_event_wait_time_low (event=0x55e42a495f50, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=reset_sig_count@entry=1) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0event.cc:485
      #4  0x000055e4281bb600 in ib_wqueue_timedwait (wq=wq@entry=0x55e42a497be0, wait_in_usecs=wait_in_usecs@entry=5000000) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/ut/ut0wqueue.cc:158
      #5  0x000055e4282a2702 in fts_optimize_thread (arg=0x55e42a497be0) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/fts/fts0opt.cc:2849
      #6  0x00007fdd5ce166ea in start_thread () from /lib64/libpthread.so.0
      #7  0x00007fdd5c1bca6f in clone () from /lib64/libc.so.6
       
      Thread 12 (Thread 0x7fdd23df3700 (LWP 14160)):
      #0  0x00007fdd5ce1da5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x000055e4280e2852 in os_event::timed_wait (this=this@entry=0x55e42a335ce0, abstime=abstime@entry=0x7fdd23df2b40) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0event.cc:275
      #2  0x000055e4280e2eb1 in os_event::wait_time_low (reset_sig_count=1, time_in_usec=1000000, this=0x55e42a335ce0) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0event.cc:385
      #3  os_event_wait_time_low (event=0x55e42a335ce0, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=<optimized out>) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0event.cc:485
      #4  0x000055e42816fb0f in srv_error_monitor_thread () at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/srv/srv0srv.cc:1815
      #5  0x00007fdd5ce166ea in start_thread () from /lib64/libpthread.so.0
      #6  0x00007fdd5c1bca6f in clone () from /lib64/libc.so.6
       
      Thread 11 (Thread 0x7fdd25ffc700 (LWP 14153)):
      #0  0x00007fdd5d9be734 in ?? () from /lib64/libaio.so.1
      #1  0x000055e4280d8be1 in LinuxAIOHandler::collect (this=this@entry=0x7fdd25ffbae0) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0file.cc:1803
      #2  0x000055e4280d9256 in LinuxAIOHandler::poll (this=this@entry=0x7fdd25ffbae0, m1=m1@entry=0x7fdd25ffbb80, m2=m2@entry=0x7fdd25ffbb88, request=request@entry=0x7fdd25ffbb90) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0file.cc:1964
      #3  0x000055e4280e1704 in os_aio_linux_handler (request=0x7fdd25ffbb90, m2=0x7fdd25ffbb88, m1=0x7fdd25ffbb80, global_segment=<optimized out>) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0file.cc:2018
      #4  os_aio_handler (segment=segment@entry=3, m1=m1@entry=0x7fdd25ffbb80, m2=m2@entry=0x7fdd25ffbb88, request=request@entry=0x7fdd25ffbb90) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0file.cc:5296
      #5  0x000055e428265228 in fil_aio_wait (segment=segment@entry=3) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/fil/fil0fil.cc:4330
      #6  0x000055e4281753d0 in io_handler_thread (arg=<optimized out>) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/srv/srv0start.cc:324
      #7  0x00007fdd5ce166ea in start_thread () from /lib64/libpthread.so.0
      #8  0x00007fdd5c1bca6f in clone () from /lib64/libc.so.6
       
      Thread 10 (Thread 0x7fdd540ab700 (LWP 14173)):
      #0  0x00007fdd5ce1d70c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x000055e427b9c481 in inline_mysql_cond_wait (that=0x55e428b90cc0 <COND_manager>, mutex=0x55e428b90d00 <LOCK_manager>, src_file=0x55e428417658 "/home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.4.32/sql/sql_manager.cc", src_line=102) at /usr/src/debug/MariaDB-10.4.32/src_0/include/mysql/psi/mysql_thread.h:1174
      #2  handle_manager (arg=arg@entry=0x0) at /usr/src/debug/MariaDB-10.4.32/src_0/sql/sql_manager.cc:102
      #3  0x000055e42800e2ba in pfs_spawn_thread (arg=0x55e42a4af548) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/perfschema/pfs.cc:1869
      #4  0x00007fdd5ce166ea in start_thread () from /lib64/libpthread.so.0
      #5  0x00007fdd5c1bca6f in clone () from /lib64/libc.so.6
       
      Thread 9 (Thread 0x7fdd257fb700 (LWP 14154)):
      #0  0x00007fdd5d9be734 in ?? () from /lib64/libaio.so.1
      #1  0x000055e4280d8be1 in LinuxAIOHandler::collect (this=this@entry=0x7fdd257faae0) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0file.cc:1803
      #2  0x000055e4280d9256 in LinuxAIOHandler::poll (this=this@entry=0x7fdd257faae0, m1=m1@entry=0x7fdd257fab80, m2=m2@entry=0x7fdd257fab88, request=request@entry=0x7fdd257fab90) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0file.cc:1964
      #3  0x000055e4280e1704 in os_aio_linux_handler (request=0x7fdd257fab90, m2=0x7fdd257fab88, m1=0x7fdd257fab80, global_segment=<optimized out>) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0file.cc:2018
      #4  os_aio_handler (segment=segment@entry=4, m1=m1@entry=0x7fdd257fab80, m2=m2@entry=0x7fdd257fab88, request=request@entry=0x7fdd257fab90) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0file.cc:5296
      #5  0x000055e428265228 in fil_aio_wait (segment=segment@entry=4) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/fil/fil0fil.cc:4330
      #6  0x000055e4281753d0 in io_handler_thread (arg=<optimized out>) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/srv/srv0start.cc:324
      #7  0x00007fdd5ce166ea in start_thread () from /lib64/libpthread.so.0
      #8  0x00007fdd5c1bca6f in clone () from /lib64/libc.so.6
       
      Thread 8 (Thread 0x7fdd24ffa700 (LWP 14155)):
      #0  0x00007fdd5d9be734 in ?? () from /lib64/libaio.so.1
      #1  0x000055e4280d8be1 in LinuxAIOHandler::collect (this=this@entry=0x7fdd24ff9ae0) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0file.cc:1803
      #2  0x000055e4280d9256 in LinuxAIOHandler::poll (this=this@entry=0x7fdd24ff9ae0, m1=m1@entry=0x7fdd24ff9b80, m2=m2@entry=0x7fdd24ff9b88, request=request@entry=0x7fdd24ff9b90) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0file.cc:1964
      #3  0x000055e4280e1704 in os_aio_linux_handler (request=0x7fdd24ff9b90, m2=0x7fdd24ff9b88, m1=0x7fdd24ff9b80, global_segment=<optimized out>) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0file.cc:2018
      #4  os_aio_handler (segment=segment@entry=5, m1=m1@entry=0x7fdd24ff9b80, m2=m2@entry=0x7fdd24ff9b88, request=request@entry=0x7fdd24ff9b90) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0file.cc:5296
      #5  0x000055e428265228 in fil_aio_wait (segment=segment@entry=5) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/fil/fil0fil.cc:4330
      #6  0x000055e4281753d0 in io_handler_thread (arg=<optimized out>) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/srv/srv0start.cc:324
      #7  0x00007fdd5ce166ea in start_thread () from /lib64/libpthread.so.0
      #8  0x00007fdd5c1bca6f in clone () from /lib64/libc.so.6
       
      Thread 7 (Thread 0x7fdd267fd700 (LWP 14152)):
      #0  0x00007fdd5d9be734 in ?? () from /lib64/libaio.so.1
      #1  0x000055e4280d8be1 in LinuxAIOHandler::collect (this=this@entry=0x7fdd267fcae0) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0file.cc:1803
      #2  0x000055e4280d9256 in LinuxAIOHandler::poll (this=this@entry=0x7fdd267fcae0, m1=m1@entry=0x7fdd267fcb80, m2=m2@entry=0x7fdd267fcb88, request=request@entry=0x7fdd267fcb90) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0file.cc:1964
      #3  0x000055e4280e1704 in os_aio_linux_handler (request=0x7fdd267fcb90, m2=0x7fdd267fcb88, m1=0x7fdd267fcb80, global_segment=<optimized out>) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0file.cc:2018
      #4  os_aio_handler (segment=segment@entry=2, m1=m1@entry=0x7fdd267fcb80, m2=m2@entry=0x7fdd267fcb88, request=request@entry=0x7fdd267fcb90) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0file.cc:5296
      #5  0x000055e428265228 in fil_aio_wait (segment=segment@entry=2) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/fil/fil0fil.cc:4330
      #6  0x000055e4281753d0 in io_handler_thread (arg=<optimized out>) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/srv/srv0start.cc:324
      #7  0x00007fdd5ce166ea in start_thread () from /lib64/libpthread.so.0
      #8  0x00007fdd5c1bca6f in clone () from /lib64/libc.so.6
       
      Thread 6 (Thread 0x7fdd235f2700 (LWP 14159)):
      #0  0x00007fdd5ce1da5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x000055e4280e2852 in os_event::timed_wait (this=this@entry=0x55e42a4ce0d0, abstime=abstime@entry=0x7fdd235f1be0) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0event.cc:275
      #2  0x000055e4280e2eb1 in os_event::wait_time_low (reset_sig_count=1, time_in_usec=1000000, this=0x55e42a4ce0d0) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0event.cc:385
      #3  os_event_wait_time_low (event=event@entry=0x55e42a4ce0d0, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=<optimized out>) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0event.cc:485
      #4  0x000055e4280b8c19 in lock_wait_timeout_thread () at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/lock/lock0wait.cc:527
      #5  0x00007fdd5ce166ea in start_thread () from /lib64/libpthread.so.0
      #6  0x00007fdd5c1bca6f in clone () from /lib64/libc.so.6
       
      Thread 5 (Thread 0x7fdd277ff700 (LWP 14150)):
      #0  0x00007fdd5d9be734 in ?? () from /lib64/libaio.so.1
      #1  0x000055e4280d8be1 in LinuxAIOHandler::collect (this=this@entry=0x7fdd277feae0) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0file.cc:1803
      #2  0x000055e4280d9256 in LinuxAIOHandler::poll (this=this@entry=0x7fdd277feae0, m1=m1@entry=0x7fdd277feb80, m2=m2@entry=0x7fdd277feb88, request=request@entry=0x7fdd277feb90) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0file.cc:1964
      #3  0x000055e4280e1704 in os_aio_linux_handler (request=0x7fdd277feb90, m2=0x7fdd277feb88, m1=0x7fdd277feb80, global_segment=<optimized out>) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0file.cc:2018
      #4  os_aio_handler (segment=segment@entry=0, m1=m1@entry=0x7fdd277feb80, m2=m2@entry=0x7fdd277feb88, request=request@entry=0x7fdd277feb90) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0file.cc:5296
      #5  0x000055e428265228 in fil_aio_wait (segment=segment@entry=0) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/fil/fil0fil.cc:4330
      #6  0x000055e4281753d0 in io_handler_thread (arg=<optimized out>) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/srv/srv0start.cc:324
      #7  0x00007fdd5ce166ea in start_thread () from /lib64/libpthread.so.0
      #8  0x00007fdd5c1bca6f in clone () from /lib64/libc.so.6
       
      Thread 4 (Thread 0x7fdd55592700 (LWP 13411)):
      #0  0x00007fdd5ce1d70c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00007fdd56153316 in gu_cond_wait_SYS (mutex=0x55e42a1fe058, cond=0x55e42a1fe098) at /home/buildbot/buildbot/build/galerautils/src/gu_threads.h:251
      #2  gu::Lock::wait (this=<synthetic pointer>, cond=...) at /home/buildbot/buildbot/build/galerautils/src/gu_lock.hpp:46
      #3  galera::ServiceThd::thd_func (arg=0x55e42a1fe038) at /home/buildbot/buildbot/build/galera/src/galera_service_thd.cpp:31
      #4  0x00007fdd5ce166ea in start_thread () from /lib64/libpthread.so.0
      #5  0x00007fdd5c1bca6f in clone () from /lib64/libc.so.6
       
      Thread 3 (Thread 0x7fdd56b5e700 (LWP 13410)):
      #0  0x00007fdd5ce1da5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x000055e42835a4ad in inline_mysql_cond_timedwait (that=0x55e42941d5a0 <COND_timer>, mutex=0x55e42941d5e0 <LOCK_timer>, src_file=0x55e4285ce5c8 "/home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.4.32/mysys/thr_timer.c", src_line=291, abstime=0x7fdd56b5dcc0) at /usr/src/debug/MariaDB-10.4.32/src_0/include/mysql/psi/mysql_thread.h:1211
      #2  timer_handler (arg=arg@entry=0x0) at /usr/src/debug/MariaDB-10.4.32/src_0/mysys/thr_timer.c:291
      #3  0x000055e42800e2ba in pfs_spawn_thread (arg=0x55e42a1e7e58) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/perfschema/pfs.cc:1869
      #4  0x00007fdd5ce166ea in start_thread () from /lib64/libpthread.so.0
      #5  0x00007fdd5c1bca6f in clone () from /lib64/libc.so.6
       
      Thread 2 (Thread 0x7fdd20ded700 (LWP 14165)):
      #0  0x00007fdd5ce1d70c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x000055e4280e2fc0 in os_event::wait (this=0x55e42a335b20) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0event.cc:158
      #2  os_event::wait_low (reset_sig_count=85, this=0x55e42a335b20) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0event.cc:325
      #3  os_event_wait_low (event=0x55e42a335b20, reset_sig_count=reset_sig_count@entry=85) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/os/os0event.cc:502
      #4  0x000055e42817338a in srv_resume_thread (timeout_usec=0, wait=true, sig_count=85, slot=0x55e428b0d580 <srv_sys+192>) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/srv/srv0srv.cc:897
      #5  srv_purge_coordinator_suspend (rseg_history_len=<optimized out>, slot=<optimized out>) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/srv/srv0srv.cc:2628
      #6  srv_purge_coordinator_thread (arg=<optimized out>) at /usr/src/debug/MariaDB-10.4.32/src_0/storage/innobase/srv/srv0srv.cc:2709
      #7  0x00007fdd5ce166ea in start_thread () from /lib64/libpthread.so.0
      #8  0x00007fdd5c1bca6f in clone () from /lib64/libc.so.6
       
      Thread 1 (Thread 0x7fdd5e8418c0 (LWP 13408)):
      #0  0x00007fdd5ce1f131 in pthread_kill () from /lib64/libpthread.so.0
      #1  0x000055e428355a07 in my_write_core (sig=<optimized out>) at /usr/src/debug/MariaDB-10.4.32/src_0/mysys/stacktrace.c:386
      #2  0x000055e427da0b4a in handle_fatal_signal (sig=6) at /usr/src/debug/MariaDB-10.4.32/src_0/sql/signal_handler.cc:372
      #3  <signal handler called>
      #4  0x00007fdd5c1b01c9 in poll () from /lib64/libc.so.6
      #5  0x000055e427ae4c8c in poll (__timeout=-1, __nfds=2, __fds=0x7ffe6518d1a0) at /usr/include/bits/poll2.h:41
      #6  handle_connections_sockets () at /usr/src/debug/MariaDB-10.4.32/src_0/sql/mysqld.cc:6509
      #7  0x000055e427ae5ebe in mysqld_main (argc=<optimized out>, argv=<optimized out>) at /usr/src/debug/MariaDB-10.4.32/src_0/sql/mysqld.cc:5945
      #8  0x00007fdd5c0da29d in __libc_start_main () from /lib64/libc.so.6
      #9  0x000055e427ad980a in _start () at ../sysdeps/x86_64/start.S:120
       
       - deleting it, already saved 0
       - found 'core' (0/0)
      Core generated by '/usr/sbin/mysqld'
      Output from gdb follows. The first stack trace is from the failing thread.
      The following stack traces are from all threads (so the failing one is
      duplicated).
      --------------------------

      Attachments

        1. mysqld.1.err.1
          2.57 MB
        2. mysqld.2.err.1
          2.35 MB
        3. node_1.log
          2.53 MB
        4. node_2.log
          2.49 MB

        Issue Links

          Activity

            People

              denis.protivensky Denis Protivensky
              sysprg Julius Goryavsky
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.