[MDEV-27125] galera_new_cluster hangs with galera_recovery Created: 2021-11-26  Updated: 2022-02-11

Status: Open
Project: MariaDB Server
Component/s: Galera, Storage Engine - InnoDB
Affects Version/s: 10.5.9, 10.4.22
Fix Version/s: 10.4, 10.5

Type: Bug Priority: Major
Reporter: s Assignee: Thirunarayanan Balathandayuthapani
Resolution: Unresolved Votes: 0
Labels: hang

Attachments: PNG File image-2021-11-26-10-16-07-313.png     PNG File screenshot-1.png     PNG File screenshot-2.png     PNG File screenshot-3.png    
Issue Links:
Relates
relates to MDEV-14486 InnoDB hang on shutdown Confirmed

 Description   

My cluster crashed and i cannot restart the first node with galera_new_cluster.
The nodes are distributed geographically in France with medium latency of 12ms.
It remains on /usr/bin/galera_recovery without doing anything.

My cluster crashed and i cannot restart the first node with galera_new_cluster.

The nodes are distributed geographically in France with medium latency of 12ms.

It remains on /usr/bin/galera_recovery without doing anything.Htop screenshot

This is my Galera config on all nodes (values may change in order to match node IP etc)...

#
 
# * Galera-related settings
 
#
 
# See the examples of server wsrep.cnf files in /usr/share/mysql
 
# and read more at https://mariadb.com/kb/en/galera-cluster/
 
 
 
[galera]
 
# Mandatory settings
 
wsrep_provider           = /usr/lib/libgalera_smm.so
 
wsrep_on                 = ON
 
wsrep_cluster_name       = "xxxxCLUSTER"
 
wsrep_cluster_address    = "gcomm://xxx.xxx.xxx.100,xxx.xxx.xxx.101,xxx.xxx.xxx.102"
 
binlog_format            = ROW
 
default_storage_engine   = InnoDB
 
#innodb_autoinc_lock_mode = 2
 
 
 
wsrep_sst_method         = rsync
 
 
 
# Allow server to accept connections on all interfaces.
 
bind-address = 0.0.0.0
 
 
 
#wsrep_sst_donor          = "DB2"
 
 
 
# Optional settings
 
wsrep_slave_threads      = 32
 
wsrep_node_name          = "DB0"
 
wsrep_node_address       = "xxx.xxx.xxx.100"
 
 
 
#innodb_flush_log_at_trx_commit = 0

Since i've been using Galera for years in his MariaDB 10.x version i know that safe_to_bootstrap has to be set 1 ecc...But nothing works with galera_new_cluster and i cannot figure out why.

This is a typical log from wsrep_recovery

2021-03-27  2:06:14 0 [Note] /usr/sbin/mysqld (mysqld 10.5.9-MariaDB-1:10.5.9+maria~buster) starting as process 8286 ...
 
2021-03-27  2:06:14 0 [Note] InnoDB: Uses event mutexes
 
2021-03-27  2:06:14 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
 
2021-03-27  2:06:14 0 [Note] InnoDB: Number of pools: 1
 
2021-03-27  2:06:14 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
 
2021-03-27  2:06:14 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts)
 
2021-03-27  2:06:14 0 [Note] InnoDB: Using Linux native AIO
 
2021-03-27  2:06:14 0 [Note] InnoDB: Initializing buffer pool, total size = 1073741824, chunk size = 134217728
 
2021-03-27  2:06:14 0 [Note] InnoDB: Completed initialization of buffer pool
 
2021-03-27  2:06:14 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
 
2021-03-27  2:06:14 0 [Note] InnoDB: 128 rollback segments are active.
 
2021-03-27  2:06:14 0 [Note] InnoDB: Creating shared tablespace for temporary tables
 
2021-03-27  2:06:14 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
 
2021-03-27  2:06:14 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
 
2021-03-27  2:06:14 0 [Note] InnoDB: 10.5.9 started; log sequence number 57353507521; transaction id 2641101
 
2021-03-27  2:06:14 0 [Warning] InnoDB: Skipping buffer pool dump/restore during wsrep recovery.
 
2021-03-27  2:06:14 0 [Note] Plugin 'FEEDBACK' is disabled.
 
2021-03-27  2:06:14 0 [Note] Recovering after a crash using tc.log
 
2021-03-27  2:06:14 0 [Note] Starting crash recovery...
 
2021-03-27  2:06:14 0 [Note] Crash recovery finished.
 
2021-03-27  2:06:14 0 [Note] Server socket created on IP: '::'.
 
2021-03-27  2:06:14 0 [Note] WSREP: Recovered position: 2edc32a3-8e96-11eb-9165-6281b552400d:7



 Comments   
Comment by s [ 2021-11-26 ]

Comment by Jan Lindström (Inactive) [ 2021-11-26 ]

yz Why you use wsrep_new_cluster if you already have datadir with galera cluster? Why you do not just start that node again? Can you provide error logs and please use --wsrep-debug=1 so that we get additional information what goes wrong.

Comment by s [ 2021-11-26 ]

Comment by s [ 2021-11-26 ]

When node crash, I use 'mysqld --wsrep_recover' to get recoverd position, it hangs

Comment by s [ 2021-11-26 ]

I change version 10.4.22, it happens on too

Comment by s [ 2021-11-26 ]

Comment by s [ 2021-11-26 ]

@jplindst
It seems to hang on "InnoDB shutdown", maybe dead lock

Comment by s [ 2021-11-26 ]

@jplindst

(gdb) info threads
Id Target Id Frame
24 Thread 0x7fa84bf66700 (LWP 28852) "mysqld" pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
23 Thread 0x7fa842fae700 (LWP 28853) "mysqld" pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
22 Thread 0x7fa827fff700 (LWP 28854) "mysqld" 0x00007fa8505d1644 in __io_getevents_0_4 (ctx=0x7fa850a56000, min_nr=1, nr=256, events=0x559205559210, timeout=0x7fa827ffe610)
at io_getevents.c:25
21 Thread 0x7fa8277fe700 (LWP 28855) "mysqld" 0x00007fa8505d1644 in __io_getevents_0_4 (ctx=0x7fa850a53000, min_nr=1, nr=256, events=0x559205569460, timeout=0x7fa8277fd610)
at io_getevents.c:25
20 Thread 0x7fa826ffd700 (LWP 28856) "mysqld" 0x00007fa8505d1644 in __io_getevents_0_4 (ctx=0x7fa850c16000, min_nr=1, nr=256, events=0x559205542fe0, timeout=0x7fa826ffc610)
at io_getevents.c:25
19 Thread 0x7fa8267fc700 (LWP 28857) "mysqld" 0x00007fa8505d1644 in __io_getevents_0_4 (ctx=0x7fa850c13000, min_nr=1, nr=256, events=0x559205544fe0, timeout=0x7fa8267fb610)
at io_getevents.c:25
18 Thread 0x7fa825ffb700 (LWP 28858) "mysqld" 0x00007fa8505d1644 in __io_getevents_0_4 (ctx=0x7fa850c10000, min_nr=1, nr=256, events=0x559205546fe0, timeout=0x7fa825ffa610)
at io_getevents.c:25
17 Thread 0x7fa8257fa700 (LWP 28859) "mysqld" 0x00007fa8505d1644 in __io_getevents_0_4 (ctx=0x7fa850a59000, min_nr=1, nr=256, events=0x559205548fe0, timeout=0x7fa8257f9610)
at io_getevents.c:25
16 Thread 0x7fa824ff9700 (LWP 28860) "mysqld" 0x00007fa8505d1644 in __io_getevents_0_4 (ctx=0x7fa850a50000, min_nr=1, nr=256, events=0x5592055a36d0, timeout=0x7fa824ff8610)
at io_getevents.c:25
15 Thread 0x7fa8247f8700 (LWP 28861) "mysqld" 0x00007fa8505d1644 in __io_getevents_0_4 (ctx=0x7fa850a4d000, min_nr=1, nr=256, events=0x5592055a56d0, timeout=0x7fa8247f7610)
at io_getevents.c:25
14 Thread 0x7fa823ff7700 (LWP 28862) "mysqld" 0x00007fa8505d1644 in __io_getevents_0_4 (ctx=0x7fa850a4a000, min_nr=1, nr=256, events=0x5592055a76d0, timeout=0x7fa823ff6610)
at io_getevents.c:25
13 Thread 0x7fa8237f6700 (LWP 28863) "mysqld" 0x00007fa8505d1644 in __io_getevents_0_4 (ctx=0x7fa850a47000, min_nr=1, nr=256, events=0x5592055a96d0, timeout=0x7fa8237f5610)
at io_getevents.c:25
12 Thread 0x7fa822ff5700 (LWP 28864) "mysqld" pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
11 Thread 0x7fa821ff3700 (LWP 28867) "mysqld" pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
10 Thread 0x7fa8217f2700 (LWP 28868) "mysqld" pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
9 Thread 0x7fa8227f4700 (LWP 28869) "mysqld" pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
8 Thread 0x7fa81b7fe700 (LWP 28872) "mysqld" 0x00007fa84f8a3e4d in nanosleep () at ../sysdeps/unix/syscall-template.S:81
7 Thread 0x7fa81affd700 (LWP 28873) "mysqld" 0x00007fa84eda0e47 in sched_yield () at ../sysdeps/unix/syscall-template.S:81
6 Thread 0x7fa81a7fc700 (LWP 28874) "mysqld" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
5 Thread 0x7fa819ffb700 (LWP 28875) "mysqld" 0x00007fa84f8a3e4d in nanosleep () at ../sysdeps/unix/syscall-template.S:81
4 Thread 0x7fa8197fa700 (LWP 28876) "mysqld" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
3 Thread 0x7fa7fffff700 (LWP 28878) "mysqld" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
2 Thread 0x7fa7ff7fe700 (LWP 28879) "mysqld" 0x0000559201e27d7e in compare_exchange_strong (__m2=std::memory_order_relaxed, __m1=std::memory_order_acquire, __i2=1,
__i1=@0x7fa7ff7fdc40: 0, this=0x559202adbcd0 <srv_sys+80>) at /usr/include/c++/4.8.2/bits/atomic_base.h:577

  • 1 Thread 0x7fa850bd7880 (LWP 28851) "mysqld" 0x00007fa84f89df57 in pthread_join (threadid=140359522838272, thread_return=0x0) at pthread_join.c:92
    (gdb) thread apply all bt

Thread 24 (Thread 0x7fa84bf66700 (LWP 28852)):
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1 0x00005592020f1687 in inline_mysql_cond_timedwait (that=0x559203406080 <COND_timer>, mutex=0x5592034060c0 <LOCK_timer>,
src_file=0x5592023662e0 "/home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.4.22/mysys/thr_timer.c", src_line=292, abstime=0x7fa84bf65d50)
at /usr/src/debug/MariaDB-10.4.22/src_0/include/mysql/psi/mysql_thread.h:1222
#2 timer_handler (arg=<optimized out>) at /usr/src/debug/MariaDB-10.4.22/src_0/mysys/thr_timer.c:292
#3 0x00007fa84f89ce25 in start_thread (arg=0x7fa84bf66700) at pthread_create.c:308
#4 0x00007fa84edbc34d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 23 (Thread 0x7fa842fae700 (LWP 28853)):
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1 0x0000559201fcaba6 in inline_mysql_cond_timedwait (
src_file=0x559202340798 "/home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.4.22/storage/maria/ma_servicethread.c", src_line=116,
abstime=0x7fa842fadcd0, mutex=0x5592033f23c0 <LOCK_checkpoint>, that=0x5592033f2380 <COND_checkpoint>) at /usr/src/debug/MariaDB-10.4.22/src_0/include/mysql/psi/mysql_thread.h:1222
#2 my_service_thread_sleep (control=control@entry=0x559202adeaa0 <checkpoint_control>, sleep_time=<optimized out>)
at /usr/src/debug/MariaDB-10.4.22/src_0/storage/maria/ma_servicethread.c:115
#3 0x0000559201fc245e in ma_checkpoint_background (arg=0x1e) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/maria/ma_checkpoint.c:707
#4 0x00007fa84f89ce25 in start_thread (arg=0x7fa842fae700) at pthread_create.c:308
#5 0x00007fa84edbc34d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 22 (Thread 0x7fa827fff700 (LWP 28854)):
#0 0x00007fa8505d1644 in __io_getevents_0_4 (ctx=0x7fa850a56000, min_nr=1, nr=256, events=0x559205559210, timeout=0x7fa827ffe610) at io_getevents.c:25
#1 0x0000559201d9728a in LinuxAIOHandler::collect (this=this@entry=0x7fa827ffeae0) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0file.cc:1823
#2 0x0000559201d978b2 in LinuxAIOHandler::poll (this=this@entry=0x7fa827ffeae0, m1=m1@entry=0x7fa827ffebd0, m2=m2@entry=0x7fa827ffebe0, request=request@entry=0x7fa827ffec00)
at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0file.cc:1984
#3 0x0000559201d9f179 in os_aio_linux_handler (request=0x7fa827ffec00, m2=0x7fa827ffebe0, m1=0x7fa827ffebd0, global_segment=140360202320320)
at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0file.cc:2038
#4 os_aio_handler (segment=segment@entry=0, m1=m1@entry=0x7fa827ffebd0, m2=m2@entry=0x7fa827ffebe0, request=request@entry=0x7fa827ffec00)
at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0file.cc:5366
#5 0x0000559201f1f211 in fil_aio_wait (segment=segment@entry=0) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/fil/fil0fil.cc:4315
#6 0x0000559201e2f218 in io_handler_thread (arg=<optimized out>) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/srv/srv0start.cc:324
#7 0x00007fa84f89ce25 in start_thread (arg=0x7fa827fff700) at pthread_create.c:308
#8 0x00007fa84edbc34d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 21 (Thread 0x7fa8277fe700 (LWP 28855)):
#0 0x00007fa8505d1644 in __io_getevents_0_4 (ctx=0x7fa850a53000, min_nr=1, nr=256, events=0x559205569460, timeout=0x7fa8277fd610) at io_getevents.c:25
#1 0x0000559201d9728a in LinuxAIOHandler::collect (this=this@entry=0x7fa8277fdae0) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0file.cc:1823
#2 0x0000559201d978b2 in LinuxAIOHandler::poll (this=this@entry=0x7fa8277fdae0, m1=m1@entry=0x7fa8277fdbd0, m2=m2@entry=0x7fa8277fdbe0, request=request@entry=0x7fa8277fdc00)
at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0file.cc:1984
#3 0x0000559201d9f179 in os_aio_linux_handler (request=0x7fa8277fdc00, m2=0x7fa8277fdbe0, m1=0x7fa8277fdbd0, global_segment=140360193927616)
--Type <return> to continue, or q <return> to quit--
at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0file.cc:2038
#4 os_aio_handler (segment=segment@entry=1, m1=m1@entry=0x7fa8277fdbd0, m2=m2@entry=0x7fa8277fdbe0, request=request@entry=0x7fa8277fdc00)
at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0file.cc:5366
#5 0x0000559201f1f211 in fil_aio_wait (segment=segment@entry=1) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/fil/fil0fil.cc:4315
#6 0x0000559201e2f218 in io_handler_thread (arg=<optimized out>) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/srv/srv0start.cc:324
#7 0x00007fa84f89ce25 in start_thread (arg=0x7fa8277fe700) at pthread_create.c:308
#8 0x00007fa84edbc34d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 20 (Thread 0x7fa826ffd700 (LWP 28856)):
#0 0x00007fa8505d1644 in __io_getevents_0_4 (ctx=0x7fa850c16000, min_nr=1, nr=256, events=0x559205542fe0, timeout=0x7fa826ffc610) at io_getevents.c:25
#1 0x0000559201d9728a in LinuxAIOHandler::collect (this=this@entry=0x7fa826ffcae0) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0file.cc:1823
#2 0x0000559201d978b2 in LinuxAIOHandler::poll (this=this@entry=0x7fa826ffcae0, m1=m1@entry=0x7fa826ffcbd0, m2=m2@entry=0x7fa826ffcbe0, request=request@entry=0x7fa826ffcc00)
at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0file.cc:1984
#3 0x0000559201d9f179 in os_aio_linux_handler (request=0x7fa826ffcc00, m2=0x7fa826ffcbe0, m1=0x7fa826ffcbd0, global_segment=140360185534912)
at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0file.cc:2038
#4 os_aio_handler (segment=segment@entry=2, m1=m1@entry=0x7fa826ffcbd0, m2=m2@entry=0x7fa826ffcbe0, request=request@entry=0x7fa826ffcc00)
at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0file.cc:5366
#5 0x0000559201f1f211 in fil_aio_wait (segment=segment@entry=2) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/fil/fil0fil.cc:4315
#6 0x0000559201e2f218 in io_handler_thread (arg=<optimized out>) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/srv/srv0start.cc:324
#7 0x00007fa84f89ce25 in start_thread (arg=0x7fa826ffd700) at pthread_create.c:308
#8 0x00007fa84edbc34d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 19 (Thread 0x7fa8267fc700 (LWP 28857)):
#0 0x00007fa8505d1644 in __io_getevents_0_4 (ctx=0x7fa850c13000, min_nr=1, nr=256, events=0x559205544fe0, timeout=0x7fa8267fb610) at io_getevents.c:25
#1 0x0000559201d9728a in LinuxAIOHandler::collect (this=this@entry=0x7fa8267fbae0) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0file.cc:1823
#2 0x0000559201d978b2 in LinuxAIOHandler::poll (this=this@entry=0x7fa8267fbae0, m1=m1@entry=0x7fa8267fbbd0, m2=m2@entry=0x7fa8267fbbe0, request=request@entry=0x7fa8267fbc00)
at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0file.cc:1984
#3 0x0000559201d9f179 in os_aio_linux_handler (request=0x7fa8267fbc00, m2=0x7fa8267fbbe0, m1=0x7fa8267fbbd0, global_segment=140360177142208)
at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0file.cc:2038
#4 os_aio_handler (segment=segment@entry=3, m1=m1@entry=0x7fa8267fbbd0, m2=m2@entry=0x7fa8267fbbe0, request=request@entry=0x7fa8267fbc00)
at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0file.cc:5366
#5 0x0000559201f1f211 in fil_aio_wait (segment=segment@entry=3) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/fil/fil0fil.cc:4315
#6 0x0000559201e2f218 in io_handler_thread (arg=<optimized out>) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/srv/srv0start.cc:324
#7 0x00007fa84f89ce25 in start_thread (arg=0x7fa8267fc700) at pthread_create.c:308
#8 0x00007fa84edbc34d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 18 (Thread 0x7fa825ffb700 (LWP 28858)):
#0 0x00007fa8505d1644 in __io_getevents_0_4 (ctx=0x7fa850c10000, min_nr=1, nr=256, events=0x559205546fe0, timeout=0x7fa825ffa610) at io_getevents.c:25
#1 0x0000559201d9728a in LinuxAIOHandler::collect (this=this@entry=0x7fa825ffaae0) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0file.cc:1823
#2 0x0000559201d978b2 in LinuxAIOHandler::poll (this=this@entry=0x7fa825ffaae0, m1=m1@entry=0x7fa825ffabd0, m2=m2@entry=0x7fa825ffabe0, request=request@entry=0x7fa825ffac00)
at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0file.cc:1984
--Type <return> to continue, or q <return> to quit--
#3 0x0000559201d9f179 in os_aio_linux_handler (request=0x7fa825ffac00, m2=0x7fa825ffabe0, m1=0x7fa825ffabd0, global_segment=140360168749504)
at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0file.cc:2038
#4 os_aio_handler (segment=segment@entry=4, m1=m1@entry=0x7fa825ffabd0, m2=m2@entry=0x7fa825ffabe0, request=request@entry=0x7fa825ffac00)
at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0file.cc:5366
#5 0x0000559201f1f211 in fil_aio_wait (segment=segment@entry=4) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/fil/fil0fil.cc:4315
#6 0x0000559201e2f218 in io_handler_thread (arg=<optimized out>) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/srv/srv0start.cc:324
#7 0x00007fa84f89ce25 in start_thread (arg=0x7fa825ffb700) at pthread_create.c:308
#8 0x00007fa84edbc34d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 17 (Thread 0x7fa8257fa700 (LWP 28859)):

#0 0x00007fa8505d1644 in __io_getevents_0_4 (ctx=0x7fa850a59000, min_nr=1, nr=256, events=0x559205548fe0, timeout=0x7fa8257f9610) at io_getevents.c:25
#1 0x0000559201d9728a in LinuxAIOHandler::collect (this=this@entry=0x7fa8257f9ae0) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0file.cc:1823
#2 0x0000559201d978b2 in LinuxAIOHandler::poll (this=this@entry=0x7fa8257f9ae0, m1=m1@entry=0x7fa8257f9bd0, m2=m2@entry=0x7fa8257f9be0, request=request@entry=0x7fa8257f9c00)
at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0file.cc:1984
#3 0x0000559201d9f179 in os_aio_linux_handler (request=0x7fa8257f9c00, m2=0x7fa8257f9be0, m1=0x7fa8257f9bd0, global_segment=140360160356800)
at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0file.cc:2038
#4 os_aio_handler (segment=segment@entry=5, m1=m1@entry=0x7fa8257f9bd0, m2=m2@entry=0x7fa8257f9be0, request=request@entry=0x7fa8257f9c00)
at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0file.cc:5366
#5 0x0000559201f1f211 in fil_aio_wait (segment=segment@entry=5) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/fil/fil0fil.cc:4315
#6 0x0000559201e2f218 in io_handler_thread (arg=<optimized out>) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/srv/srv0start.cc:324
#7 0x00007fa84f89ce25 in start_thread (arg=0x7fa8257fa700) at pthread_create.c:308
#8 0x00007fa84edbc34d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 16 (Thread 0x7fa824ff9700 (LWP 28860)):

#0 0x00007fa8505d1644 in __io_getevents_0_4 (ctx=0x7fa850a50000, min_nr=1, nr=256, events=0x5592055a36d0, timeout=0x7fa824ff8610) at io_getevents.c:25
#1 0x0000559201d9728a in LinuxAIOHandler::collect (this=this@entry=0x7fa824ff8ae0) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0file.cc:1823
#2 0x0000559201d978b2 in LinuxAIOHandler::poll (this=this@entry=0x7fa824ff8ae0, m1=m1@entry=0x7fa824ff8bd0, m2=m2@entry=0x7fa824ff8be0, request=request@entry=0x7fa824ff8c00)
at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0file.cc:1984
#3 0x0000559201d9f179 in os_aio_linux_handler (request=0x7fa824ff8c00, m2=0x7fa824ff8be0, m1=0x7fa824ff8bd0, global_segment=140360151964096)
at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0file.cc:2038

#4 os_aio_handler (segment=segment@entry=6, m1=m1@entry=0x7fa824ff8bd0, m2=m2@entry=0x7fa824ff8be0, request=request@entry=0x7fa824ff8c00)
at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0file.cc:5366
#5 0x0000559201f1f211 in fil_aio_wait (segment=segment@entry=6) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/fil/fil0fil.cc:4315
#6 0x0000559201e2f218 in io_handler_thread (arg=<optimized out>) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/srv/srv0start.cc:324
#7 0x00007fa84f89ce25 in start_thread (arg=0x7fa824ff9700) at pthread_create.c:308
#8 0x00007fa84edbc34d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 15 (Thread 0x7fa8247f8700 (LWP 28861)):

#0 0x00007fa8505d1644 in __io_getevents_0_4 (ctx=0x7fa850a4d000, min_nr=1, nr=256, events=0x5592055a56d0, timeout=0x7fa8247f7610) at io_getevents.c:25
#1 0x0000559201d9728a in LinuxAIOHandler::collect (this=this@entry=0x7fa8247f7ae0) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0file.cc:1823
#2 0x0000559201d978b2 in LinuxAIOHandler::poll (this=this@entry=0x7fa8247f7ae0, m1=m1@entry=0x7fa8247f7bd0, m2=m2@entry=0x7fa8247f7be0, request=request@entry=0x7fa8247f7c00)
--Type <return> to continue, or q <return> to quit--
at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0file.cc:1984
#3 0x0000559201d9f179 in os_aio_linux_handler (request=0x7fa8247f7c00, m2=0x7fa8247f7be0, m1=0x7fa8247f7bd0, global_segment=140360143571392)
at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0file.cc:2038
#4 os_aio_handler (segment=segment@entry=7, m1=m1@entry=0x7fa8247f7bd0, m2=m2@entry=0x7fa8247f7be0, request=request@entry=0x7fa8247f7c00)
at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0file.cc:5366
#5 0x0000559201f1f211 in fil_aio_wait (segment=segment@entry=7) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/fil/fil0fil.cc:4315
#6 0x0000559201e2f218 in io_handler_thread (arg=<optimized out>) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/srv/srv0start.cc:324
#7 0x00007fa84f89ce25 in start_thread (arg=0x7fa8247f8700) at pthread_create.c:308
#8 0x00007fa84edbc34d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 14 (Thread 0x7fa823ff7700 (LWP 28862)):

#0 0x00007fa8505d1644 in __io_getevents_0_4 (ctx=0x7fa850a4a000, min_nr=1, nr=256, events=0x5592055a76d0, timeout=0x7fa823ff6610) at io_getevents.c:25
#1 0x0000559201d9728a in LinuxAIOHandler::collect (this=this@entry=0x7fa823ff6ae0) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0file.cc:1823
#2 0x0000559201d978b2 in LinuxAIOHandler::poll (this=this@entry=0x7fa823ff6ae0, m1=m1@entry=0x7fa823ff6bd0, m2=m2@entry=0x7fa823ff6be0, request=request@entry=0x7fa823ff6c00)
at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0file.cc:1984
#3 0x0000559201d9f179 in os_aio_linux_handler (request=0x7fa823ff6c00, m2=0x7fa823ff6be0, m1=0x7fa823ff6bd0, global_segment=140360135178688)
at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0file.cc:2038
#4 os_aio_handler (segment=segment@entry=8, m1=m1@entry=0x7fa823ff6bd0, m2=m2@entry=0x7fa823ff6be0, request=request@entry=0x7fa823ff6c00)
at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0file.cc:5366
#5 0x0000559201f1f211 in fil_aio_wait (segment=segment@entry=8) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/fil/fil0fil.cc:4315
#6 0x0000559201e2f218 in io_handler_thread (arg=<optimized out>) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/srv/srv0start.cc:324
#7 0x00007fa84f89ce25 in start_thread (arg=0x7fa823ff7700) at pthread_create.c:308
#8 0x00007fa84edbc34d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 13 (Thread 0x7fa8237f6700 (LWP 28863)):

#0 0x00007fa8505d1644 in __io_getevents_0_4 (ctx=0x7fa850a47000, min_nr=1, nr=256, events=0x5592055a96d0, timeout=0x7fa8237f5610) at io_getevents.c:25
#1 0x0000559201d9728a in LinuxAIOHandler::collect (this=this@entry=0x7fa8237f5ae0) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0file.cc:1823
#2 0x0000559201d978b2 in LinuxAIOHandler::poll (this=this@entry=0x7fa8237f5ae0, m1=m1@entry=0x7fa8237f5bd0, m2=m2@entry=0x7fa8237f5be0, request=request@entry=0x7fa8237f5c00)
at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0file.cc:1984

#3 0x0000559201d9f179 in os_aio_linux_handler (request=0x7fa8237f5c00, m2=0x7fa8237f5be0, m1=0x7fa8237f5bd0, global_segment=140360126785984)
at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0file.cc:2038
#4 os_aio_handler (segment=segment@entry=9, m1=m1@entry=0x7fa8237f5bd0, m2=m2@entry=0x7fa8237f5be0, request=request@entry=0x7fa8237f5c00)
at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0file.cc:5366
#5 0x0000559201f1f211 in fil_aio_wait (segment=segment@entry=9) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/fil/fil0fil.cc:4315

#6 0x0000559201e2f218 in io_handler_thread (arg=<optimized out>) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/srv/srv0start.cc:324
#7 0x00007fa84f89ce25 in start_thread (arg=0x7fa8237f6700) at pthread_create.c:308
#8 0x00007fa84edbc34d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 12 (Thread 0x7fa822ff5700 (LWP 28864)):

#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1 0x0000559201da0193 in os_event::timed_wait (this=this@entry=0x5592054f06d0, abstime=abstime@entry=0x7fa822ff4b20)
--Type <return> to continue, or q <return> to quit--
at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0event.cc:275

#2 0x0000559201da03a1 in os_event::wait_time_low (this=0x5592054f06d0, time_in_usec=time_in_usec@entry=999000, reset_sig_count=<optimized out>, reset_sig_count@entry=630)
at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0event.cc:385
#3 0x0000559201da065a in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=999000, reset_sig_count=reset_sig_count@entry=630)
at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0event.cc:485
#4 0x0000559201eca706 in pc_sleep_if_needed (cur_time=7859628390, sig_count=630, next_loop_time=7859629389) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/buf/buf0flu.cc:2638
#5 buf_flush_page_cleaner_coordinator () at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/buf/buf0flu.cc:3097
#6 0x00007fa84f89ce25 in start_thread (arg=0x7fa822ff5700) at pthread_create.c:308
#7 0x00007fa84edbc34d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 11 (Thread 0x7fa821ff3700 (LWP 28867)):
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1 0x0000559201da0193 in os_event::timed_wait (this=this@entry=0x559205b56060, abstime=abstime@entry=0x7fa821ff2c20)
at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0event.cc:275
#2 0x0000559201da03a1 in os_event::wait_time_low (this=this@entry=0x559205b56060, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=<optimized out>)
at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0event.cc:385
#3 0x0000559201da065a in os_event_wait_time_low (event=event@entry=0x559205b56060, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=<optimized out>)
at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0event.cc:485
#4 0x0000559201d7542f in lock_wait_timeout_thread () at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/lock/lock0wait.cc:514
#5 0x00007fa84f89ce25 in start_thread (arg=0x7fa821ff3700) at pthread_create.c:308
#6 0x00007fa84edbc34d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 10 (Thread 0x7fa8217f2700 (LWP 28868)):
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1 0x0000559201da0193 in os_event::timed_wait (this=this@entry=0x5592054f0580, abstime=abstime@entry=0x7fa8217f1bb0)
at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0event.cc:275
#2 0x0000559201da03a1 in os_event::wait_time_low (this=0x5592054f0580, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=<optimized out>)
at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0event.cc:385
#3 0x0000559201da065a in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=<optimized out>)
at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0event.cc:485
#4 0x0000559201e2c00c in srv_error_monitor_thread () at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/srv/srv0srv.cc:1848
#5 0x00007fa84f89ce25 in start_thread (arg=0x7fa8217f2700) at pthread_create.c:308
#6 0x00007fa84edbc34d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 9 (Thread 0x7fa8227f4700 (LWP 28869)):
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1 0x0000559201da0193 in os_event::timed_wait (this=this@entry=0x5592054f05f0, abstime=abstime@entry=0x7fa8227f3cc0)
at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0event.cc:275
#2 0x0000559201da03a1 in os_event::wait_time_low (this=0x5592054f05f0, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=<optimized out>)
at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0event.cc:385
#3 0x0000559201da065a in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=<optimized out>)
--Type <return> to continue, or q <return> to quit--
at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0event.cc:485
#4 0x0000559201e2b900 in srv_monitor_thread () at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/srv/srv0srv.cc:1684
#5 0x00007fa84f89ce25 in start_thread (arg=0x7fa8227f4700) at pthread_create.c:308
#6 0x00007fa84edbc34d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 8 (Thread 0x7fa81b7fe700 (LWP 28872)):
#0 0x00007fa84f8a3e4d in nanosleep () at ../sysdeps/unix/syscall-template.S:81
#1 0x0000559201da08e0 in os_thread_sleep (tm=tm@entry=1000000) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0thread.cc:231
#2 0x0000559201e2cb94 in srv_master_sleep () at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/srv/srv0srv.cc:2335
#3 srv_master_thread (arg=<optimized out>) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/srv/srv0srv.cc:2375
#4 0x00007fa84f89ce25 in start_thread (arg=0x7fa81b7fe700) at pthread_create.c:308
#5 0x00007fa84edbc34d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 7 (Thread 0x7fa81affd700 (LWP 28873)):
#0 0x00007fa84eda0e47 in sched_yield () at ../sysdeps/unix/syscall-template.S:81
#1 0x0000559201e494d5 in trx_purge_wait_for_workers_to_complete () at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/trx/trx0purge.cc:1240
#2 trx_purge (n_purge_threads=<optimized out>, truncate=<optimized out>) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/trx/trx0purge.cc:1299
#3 0x0000559201e2e2ab in srv_do_purge (n_total_purged=<synthetic pointer>) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/srv/srv0srv.cc:2622
#4 srv_purge_coordinator_thread (arg=<optimized out>) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/srv/srv0srv.cc:2754
#5 0x00007fa84f89ce25 in start_thread (arg=0x7fa81affd700) at pthread_create.c:308
#6 0x00007fa84edbc34d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 6 (Thread 0x7fa81a7fc700 (LWP 28874)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x0000559201da06a1 in wait (this=<optimized out>) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0event.cc:158
#2 wait_low (reset_sig_count=<optimized out>, this=0x5592054f0120) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0event.cc:325
#3 os_event_wait_low (event=0x5592054f0120, reset_sig_count=<optimized out>) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0event.cc:502
#4 0x0000559201e35298 in sync_array_wait_event (arr=0x5592054c7940, cell=@0x7fa81a7fbc90: 0x5592054c7a40) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/sync/sync0arr.cc:471
#5 0x0000559201e278e5 in enter (line=907,
filename=0x5592023224d8 "/home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.4.22/storage/innobase/srv/srv0srv.cc", max_delay=<optimized out>,
max_spins=60, this=0x559202adbcd0 <srv_sys+80>) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/include/ib0mutex.h:477
#6 enter (name=0x5592023224d8 "/home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.4.22/storage/innobase/srv/srv0srv.cc", line=907,
n_delay=<optimized out>, n_spins=<optimized out>, this=0x559202adbcd0 <srv_sys+80>) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/include/ib0mutex.h:594
#7 srv_resume_thread (slot=slot@entry=0x559202adbd78 <srv_sys+248>, wait=wait@entry=true, timeout_usec=0, sig_count=0)
at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/srv/srv0srv.cc:907
#8 0x0000559201e2d6e7 in srv_worker_thread (arg=<optimized out>) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/srv/srv0srv.cc:2518
#9 0x00007fa84f89ce25 in start_thread (arg=0x7fa81a7fc700) at pthread_create.c:308
#10 0x00007fa84edbc34d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 5 (Thread 0x7fa819ffb700 (LWP 28875)):
#0 0x00007fa84f8a3e4d in nanosleep () at ../sysdeps/unix/syscall-template.S:81
--Type <return> to continue, or q <return> to quit--
#1 0x0000559201da08e0 in os_thread_sleep (tm=tm@entry=1000000) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0thread.cc:231
#2 0x0000559201e0942c in row_purge_parse_undo_rec (thr=<optimized out>, updated_extern=0x7fa819ffaaa0, undo_rec=0x559205c74778 "", node=0x559205c74320)
at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/row/row0purge.cc:1107
#3 row_purge (thr=0x559205c74268, undo_rec=0x559205c74778 "", node=0x559205c74320) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/row/row0purge.cc:1261
#4 row_purge_step (thr=thr@entry=0x559205c74268) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/row/row0purge.cc:1342
#5 0x0000559201dbf307 in que_thr_step (thr=0x559205c74268) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/que/que0que.cc:1038
#6 que_run_threads_low (thr=<optimized out>) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/que/que0que.cc:1100
#7 que_run_threads (thr=thr@entry=0x559205c74268) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/que/que0que.cc:1140
#8 0x0000559201e2d69d in srv_task_execute () at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/srv/srv0srv.cc:2465
#9 srv_worker_thread (arg=<optimized out>) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/srv/srv0srv.cc:2520
#10 0x00007fa84f89ce25 in start_thread (arg=0x7fa819ffb700) at pthread_create.c:308
#11 0x00007fa84edbc34d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 4 (Thread 0x7fa8197fa700 (LWP 28876)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x0000559201da06a1 in wait (this=<optimized out>) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0event.cc:158
#2 wait_low (reset_sig_count=<optimized out>, this=0x5592054f0120) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0event.cc:325
#3 os_event_wait_low (event=0x5592054f0120, reset_sig_count=<optimized out>) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0event.cc:502
#4 0x0000559201e35298 in sync_array_wait_event (arr=0x5592054c7940, cell=@0x7fa8197f9c90: 0x5592054c79c0) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/sync/sync0arr.cc:471
#5 0x0000559201e278e5 in enter (line=907,
filename=0x5592023224d8 "/home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.4.22/storage/innobase/srv/srv0srv.cc", max_delay=<optimized out>,
max_spins=60, this=0x559202adbcd0 <srv_sys+80>) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/include/ib0mutex.h:477
#6 enter (name=0x5592023224d8 "/home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.4.22/storage/innobase/srv/srv0srv.cc", line=907,
n_delay=<optimized out>, n_spins=<optimized out>, this=0x559202adbcd0 <srv_sys+80>) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/include/ib0mutex.h:594
#7 srv_resume_thread (slot=slot@entry=0x559202adbde8 <srv_sys+360>, wait=wait@entry=true, timeout_usec=0, sig_count=0)
at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/srv/srv0srv.cc:907
#8 0x0000559201e2d6e7 in srv_worker_thread (arg=<optimized out>) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/srv/srv0srv.cc:2518
#9 0x00007fa84f89ce25 in start_thread (arg=0x7fa8197fa700) at pthread_create.c:308
#10 0x00007fa84edbc34d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 3 (Thread 0x7fa7fffff700 (LWP 28878)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x0000559201da06a1 in wait (this=<optimized out>) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0event.cc:158
#2 wait_low (reset_sig_count=<optimized out>, this=0x5592054f0740) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0event.cc:325
#3 os_event_wait_low (event=0x5592054f0740, reset_sig_count=reset_sig_count@entry=0) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/os/os0event.cc:502
#4 0x0000559201ebc232 in buf_resize_thread () at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/buf/buf0buf.cc:3142
#5 0x00007fa84f89ce25 in start_thread (arg=0x7fa7fffff700) at pthread_create.c:308
#6 0x00007fa84edbc34d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 2 (Thread 0x7fa7ff7fe700 (LWP 28879)):
#0 0x0000559201e27d7e in compare_exchange_strong (__m2=std::memory_order_relaxed, __m1=std::memory_order_acquire, __i2=1, __i1=@0x7fa7ff7fdc40: 0, this=0x559202adbcd0 <srv_sys+80>)
--Type <return> to continue, or q <return> to quit--
at /usr/include/c++/4.8.2/bits/atomic_base.h:577
#1 try_lock (this=0x559202adbcd0 <srv_sys+80>) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/include/ib0mutex.h:421
#2 enter (line=932, filename=0x5592023224d8 "/home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.4.22/storage/innobase/srv/srv0srv.cc",
max_delay=<optimized out>, max_spins=30, this=0x559202adbcd0 <srv_sys+80>) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/include/ib0mutex.h:452
#3 enter (name=0x5592023224d8 "/home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.4.22/storage/innobase/srv/srv0srv.cc", line=932,
n_delay=<optimized out>, n_spins=<optimized out>, this=0x559202adbcd0 <srv_sys+80>) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/include/ib0mutex.h:594
#4 srv_release_threads (n=<optimized out>, type=SRV_WORKER) at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/srv/srv0srv.cc:932
#5 0x0000559201e2ef9b in srv_purge_wakeup () at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/srv/srv0srv.cc:2857
#6 srv_purge_shutdown () at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/srv/srv0srv.cc:2869
#7 0x0000559201d1b240 in thd_destructor_proxy () at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/handler/ha_innodb.cc:325
#8 0x00007fa84f89ce25 in start_thread (arg=0x7fa7ff7fe700) at pthread_create.c:308
#9 0x00007fa84edbc34d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 1 (Thread 0x7fa850bd7880 (LWP 28851)):
#0 0x00007fa84f89df57 in pthread_join (threadid=140359522838272, thread_return=0x0) at pthread_join.c:92
#1 0x0000559201d195fa in innobase_end(handlerton *, enum ha_panic_function) () at /usr/src/debug/MariaDB-10.4.22/src_0/storage/innobase/handler/ha_innodb.cc:4414
#2 0x0000559201b77884 in ha_finalize_handlerton (plugin=0x559205408b40) at /usr/src/debug/MariaDB-10.4.22/src_0/sql/handler.cc:513
#3 0x000055920196fdc4 in plugin_deinitialize (plugin=0x559205408b40, ref_check=ref_check@entry=true) at /usr/src/debug/MariaDB-10.4.22/src_0/sql/sql_plugin.cc:1241
#4 0x0000559201972d2e in reap_plugins () at /usr/src/debug/MariaDB-10.4.22/src_0/sql/sql_plugin.cc:1317
#5 0x0000559201973783 in plugin_shutdown () at /usr/src/debug/MariaDB-10.4.22/src_0/sql/sql_plugin.cc:2013
#6 0x00005592018985ff in clean_up (print_message=<optimized out>) at /usr/src/debug/MariaDB-10.4.22/src_0/sql/mysqld.cc:1966
#7 0x000055920189befe in clean_up (print_message=<optimized out>) at /usr/src/debug/MariaDB-10.4.22/src_0/sql/mysqld.cc:8000
#8 unireg_abort (exit_code=exit_code@entry=0) at /usr/src/debug/MariaDB-10.4.22/src_0/sql/mysqld.cc:1879
#9 0x00005592018a3381 in mysqld_main (argc=16, argv=0x5592053d1a00) at /usr/src/debug/MariaDB-10.4.22/src_0/sql/mysqld.cc:5748
#10 0x00007fa84ece5c05 in __libc_start_main (main=0x55920186da30 <main(int, char**)>, argc=4, ubp_av=0x7fffc05ff518, init=<optimized out>, fini=<optimized out>,
rtld_fini=<optimized out>, stack_end=0x7fffc05ff508) at ../csu/libc-start.c:274
#11 0x0000559201896624 in _start ()
(gdb)

Comment by s [ 2021-11-26 ]

2021-11-26 17:14:14 0 [Warning] InnoDB: A long semaphore wait:
--Thread 140359959029504 has waited at srv0srv.cc line 907 for 2858.00 seconds the semaphore:
Mutex at 0x559202adbcd0, Mutex SRV_SYS created srv0srv.cc:1012, lock var 1

2021-11-26 17:14:14 0 [Warning] InnoDB: A long semaphore wait:
--Thread 140359975814912 has waited at srv0srv.cc line 907 for 2858.00 seconds the semaphore:
Mutex at 0x559202adbcd0, Mutex SRV_SYS created srv0srv.cc:1012, lock var 1

--Thread 140359959029504 has waited at srv0srv.cc line 907 for 2858.00 seconds the semaphore:
Mutex at 0x559202adbcd0, Mutex SRV_SYS created srv0srv.cc:1012, lock var 1

--Thread 140359975814912 has waited at srv0srv.cc line 907 for 2858.00 seconds the semaphore:
Mutex at 0x559202adbcd0, Mutex SRV_SYS created srv0srv.cc:1012, lock var 1

InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info:
InnoDB: Pending reads 0, writes 0

=====================================
2021-11-26 17:14:32 0x7fa8227f4700 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 18 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 0 srv_active, 0 srv_shutdown, 642 srv_idle
srv_master_thread log flush and writes: 642
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 746006
--Thread 140359975814912 has waited at srv0srv.cc line 907 for 0.00 seconds the semaphore:
Mutex at 0x559202adbcd0, Mutex SRV_SYS created srv0srv.cc:1012, lock var 1

OS WAIT ARRAY INFO: signal count 556248
RW-shared spins 2, rounds 30, OS waits 1
RW-excl spins 0, rounds 330, OS waits 11
RW-sx spins 0, rounds 0, OS waits 0
Spin rounds per wait: 15.00 RW-shared, 330.00 RW-excl, 0.00 RW-sx
------------
TRANSACTIONS
------------
Trx id counter 3273784
Purge done for trx's n:o < 3273783 undo n:o < 0 state: running
History list length 2
LIST OF TRANSACTIONS FOR EACH SESSION:
--------
FILE I/O
--------
I/O thread 0 state: waiting for completed aio requests (insert buffer thread)
I/O thread 1 state: waiting for completed aio requests (log thread)
I/O thread 2 state: waiting for completed aio requests (read thread)
I/O thread 3 state: waiting for completed aio requests (read thread)
I/O thread 4 state: waiting for completed aio requests (read thread)
I/O thread 5 state: waiting for completed aio requests (read thread)
I/O thread 6 state: waiting for completed aio requests (write thread)
I/O thread 7 state: waiting for completed aio requests (write thread)
I/O thread 8 state: waiting for completed aio requests (write thread)
I/O thread 9 state: waiting for completed aio requests (write thread)
Pending normal aio reads: [0, 0, 0, 0] , aio writes: [0, 0, 0, 0] ,
ibuf aio reads:, log i/o's:, sync i/o's:
Pending flushes (fsync) log: 0; buffer pool: 0
303 OS file reads, 133 OS file writes, 2 OS fsyncs
0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 1, free list len 0, seg size 2, 0 merges
merged operations:
insert 0, delete mark 0, delete 0
discarded operations:
insert 0, delete mark 0, delete 0
Hash table size 34679, node heap has 0 buffer(s)
Hash table size 34679, node heap has 0 buffer(s)
Hash table size 34679, node heap has 1 buffer(s)
Hash table size 34679, node heap has 1 buffer(s)
Hash table size 34679, node heap has 1 buffer(s)
Hash table size 34679, node heap has 1 buffer(s)
Hash table size 34679, node heap has 1 buffer(s)
Hash table size 34679, node heap has 1 buffer(s)
10.83 hash searches/s, 1.44 non-hash searches/s
—
LOG
—
Log sequence number 1204950549
Log flushed up to 1204950549
Pages flushed up to 1204950549
Last checkpoint at 1204950540
0 pending log flushes, 0 pending chkp writes
9 log i/o's done, 0.00 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total large memory allocated 167772160
Dictionary memory allocated 21960
Buffer pool size 8027
Free buffers 7602
Database pages 419
Old database pages 0
Modified db pages 0
Percent of dirty pages(LRU & free pages): 0.000
Max dirty pages percent: 75.000
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 0, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 288, created 131, written 131
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 419, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
--------------
ROW OPERATIONS
--------------
0 queries inside InnoDB, 0 queries in queue
0 read views open inside InnoDB
Process ID=28851, Main thread ID=140359992600320, state: sleeping
Number of rows inserted 0, updated 0, deleted 0, read 0
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s
Number of system rows inserted 0, updated 0, deleted 0, read 0
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s
----------------------------
END OF INNODB MONITOR OUTPUT
============================
InnoDB: ###### Diagnostic info printed to the standard error stream

Comment by Jan Lindström (Inactive) [ 2021-11-26 ]

yz What kernel version you have and do you use io_uring ? Can you please provide error logs.

Comment by s [ 2021-11-26 ]

@jplindst
kernel: 3.10.0-693.el7.x86_64

Comment by s [ 2021-11-26 ]

@jplindst

error log output stdout

Comment by Marko Mäkelä [ 2022-02-11 ]

jplindst, an io_uring interface (MDEV-24883) was added in MariaDB Server 10.6. MDEV-26674 cannot possibly affect earlier versions than that. From the pasted stack trace output you can see that the libaio interface (io_submit(), io_getevents()) is being used. Most threads are idle.

I do not see anything Galera specific in the stack trace output. This could be a duplicate of MDEV-14486.

Generated at Thu Feb 08 09:50:31 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.