[MDEV-26970] MariaDB InnoDB Plugin Failure - NUC/CIFS mounts Created: 2021-10-29  Updated: 2023-09-19

Status: Confirmed
Project: MariaDB Server
Component/s: Docker
Affects Version/s: 10.5
Fix Version/s: 10.5

Type: Bug Priority: Major
Reporter: Kevin Wilson Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: not-10.4
Environment:

Dell Optiplex 3070 USFF - Intel i5-9500T, 16Gb RAM, 8TB drive mounted in USB 3.0 Caddy

Windows 10 Pro (build 19043), running VirtualBox (6.1.28 r147628), running Ubuntu Server 20.04 LTS, running Docker 20.10.10 - Drives mapped via VirtualBox drivers

Also tested on ASRock Beebox with Intel Celeron N3850 8GB RAM, with remote drives (in Windows 10 sytem above) mounted as SMB 3.0 network shares.

Attachments: Text File command output.txt     File docker-compose-redacted.yml     Text File error messages.txt    
Issue Links:
Blocks
is blocked by MDEV-27593 Crashing on I/O error is unhelpful Closed
Relates
relates to MDEV-30266 Preparing a mariabackup on a mounted ... Open
relates to MDEV-30266 Preparing a mariabackup on a mounted ... Open

 Description   

I'm trying to deploy a new Photoprism docker instance, and as part of the docker-compose it sets up a MariaDB instance.

I seem to be getting consistent errors pertaining to Assertion Errors and Corrupted pages (see below).

My setup is as follows: Windows 10 Pro -> Virtualbox 6.1 -> Ubuntu Server 20.04 LTS -> Docker 20.10.10 -> MariaDB. The CPU is an Intel i5-9500T. The ubuntu VM is allocated 2 CPUs and 6GB RAM. I have also tried to set up Docker on an Ubuntu 20.04 server installed directly on an Intel N3850-powered NUC, with the same results. In the Virtualbox setup, the drives are mounted via the Virtualbox Shared Folder feature. In the NUC setup, the drives are mounted as SMB3.0 network shares.

The issue seems to be unique to the situation where the database needs to be created on a remote network drive. This drive is mounted in a USB caddy and connected to the Windows 10 Pro computer via USB 3.0. The file system is NTFS. Other docker containers on both Virtualbox and the NUC are able to read/write data to these shared drives without any issue.

I've tried versions 10.5, 10.6 and Latest of mariaDB, with the same behaviour observed in all three.

As suggested in the Discussions page, I tried innodb_use_native_aio=0 in the command but this didn't seem to have any effect.

I've included the output from the commands suggested in the discussion page (before the innodb_use_native_aio=0 was applied). There was no output after the "thread apply all bt full" command. The testdir folder also had no content. Not sure if this is what was expected.

Happy to help in whatever way I can

 Comments   
Comment by Daniel Black [ 2021-11-04 ]

Per discussion with wlad/marko "Failing assertion: cb->m_err == DB_SUCCESS" (in static void io_callback / storage/innobase/os/os0file.cc) should be changed to actually have the error so the art of debugging this is significantly easier.

Per command output.txt:

2021-10-29 11:47:02 0 [Warning] Can't create test file /var/lib/mysql/53a5df707bc7.lower-test
 
/usr/sbin/mariadbd: Can't change dir to '/var/lib/mysql/' (Errcode: 13 "Permission denied")

This needs to be resolved first, and in most likelihood, this will be the underlying cause. innodb_use_native_aio=0 was a workaround after the permissions were resolved, and on the assumption that this was a Virtualbox issue with libaio (which could be SMB as well).

Evidently root@53a5df707bc7:/# chown -R mysql: /var/lib/mysql wasn't sufficient to make this ownership effective. Examine this with ls -la /var/lib/mysql. Try with --user mysql on compose option. uid mapping between the docker container and the underlying storage make it not obvious how this is done.

What are the SMB and VirtualBox mount options inside the VM? What storage driver is Docker using? Which Docker version?

Comment by Kevin Wilson [ 2021-11-05 ]

For clarity - I've now been able to reproduce the same problem on a dedicated ASRock NUC running Ubuntu natively (i.e without Virtualbox).

The drives are mounted in Ubuntu with the following lines in the /etc/fstab file:

//192.168.1.99/Media /mnt/m cifs credentials=/home/user/.smbcredentials,iocharset=utf8,dir_mode=0777,file_mode=0777,vers=3.0
 
//192.168.1.99/Users /mnt/u cifs credentials=/home/user/.smbcredentials,iocharset=utf8,dir_mode=0777,file_mode=0777,vers=3.0

The dir_mode=0777 and file_mode=0777 are new additions, and this has changed the errors slightly to this:

mariadb       | 2021-11-05 10:28:23+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.6.4+maria~focal star
 
mariadb       | 2021-11-05 10:28:23+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
 
mariadb       | 2021-11-05 10:28:23+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.6.4+maria~focal star
 
mariadb       | 2021-11-05 10:28:24+00:00 [Note] [Entrypoint]: Initializing database files
 
mariadb       | Cannot change ownership of the database directories to the 'mysql'
 
mariadb       | user.  Check that you have the necessary permissions and try again.
 
mariadb       | chown: changing ownership of '/var/lib/mysql/': Operation not permitted
 
mariadb exited with code 1

I'm not able to do 'ls -la /var/lib/mysql', as when I try to connect to the docker container I get: "Error response from daemon: Container cf67035f2cc2ae00559976ec2c277b6b83954d5ecfdd760448e2fdea9e2881f9 is restarting, wait until the container is running"

The storage driver is as follows:

Storage Driver: overlay2
 
  Backing Filesystem: extfs
 
  Supports d_type: true
 
  Native Overlay Diff: true
 
  userxattr: false

Docker version is 20.10.10

Comment by Daniel Black [ 2022-01-24 ] 

$
 
$  sudo mount -t cifs -o credentials=/home/dan/.nascredentials,iocharset=utf8,dir_mode=0777,file_mode=0777,vers=3.0 //192.168.178.171/dan /mnt/nas
 
 
 
$ mkdir /mnt/nas/datadir108
 
 
 
~/repos/build-mariadb-server-10.8 
$ scripts/mysql_install_db --no-defaults --srcdir=$OLDPWD --builddir=$PWD --datadir=/mnt/nas/datadir108
 
Installing MariaDB/MySQL system tables in '/mnt/nas/datadir108' ...
 
2022-01-24 15:15:10 0x7f39e63fd640  InnoDB: Assertion failure in file /home/dan/repos/mariadb-server-10.8/storage/innobase/os/os0file.cc line 3596
 
InnoDB: Failing assertion: cb->m_err == DB_SUCCESS
 
2022-01-24 15:15:10 0x7f39e53fb640  InnoDB: Assertion failure in file /home/dan/repos/mariadb-server-10.8/storage/innobase/os/os0file.cc line 3596
 
InnoDB: Failing assertion: cb->m_err == DB_SUCCESS
 
InnoDB: We intentionally generate a memory trap.
 
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
 
InnoDB: If you get repeated assertion failures or crashes, even
 
InnoDB: immediately after the mariadbd startup, there may be
 
InnoDB: corruption in the InnoDB tablespace. Please refer to
 
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
 
InnoDB: about forcing recovery.
 
InnoDB: We intentionally generate a memory trap.
 
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
 
InnoDB: If you get repeated assertion failures or crashes, even
 
InnoDB: immediately after the mariadbd startup, there may be
 
InnoDB: corruption in the InnoDB tablespace. Please refer to
 
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
 
InnoDB: about forcing recovery.
 
220124 15:15:10 [ERROR] mysqld got signal 6 ;
 
scripts/mysql_install_db: line 537: 43865 Aborted                 (core dumped) "$mysqld_bootstrap" $defaults $defaults_group_suffix "$mysqld_opt" --bootstrap $silent_startup "--basedir=$basedir" "--datadir=$ldata" --log-warnings=0 --enforce-storage-engine="" "--plugin-dir=${plugindir}" $args --max_allowed_packet=8M --net_buffer_length=16K
 
 
 
#4  0x0000000000c037d0 in io_callback (cb=<optimized out>) at /home/dan/repos/mariadb-server-10.8/storage/innobase/os/os0file.cc:3596
 
#5  0x0000000000d6588d in tpool::task_group::execute (this=0x2fd8980, t=0x2ffe9e0) at /home/dan/repos/mariadb-server-10.8/tpool/task_group.cc:55
 
#6  0x0000000000d63976 in tpool::thread_pool_generic::worker_main (this=0x2f6d700, thread_var=0x2f7d320) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:549
 
#7  0x00007f3a0b6985c4 in execute_native_thread_routine () from /lib64/libstdc++.so.6
 
#8  0x00007f3a0b427a87 in start_thread () from /lib64/libc.so.6
 
#9  0x00007f3a0b4ac640 in clone3 () from /lib64/libc.so.6
 
(gdb) up
 
#1  0x00007f3a0b3dc6a6 in raise () from /lib64/libc.so.6
 
(gdb) up
 
#2  0x00007f3a0b3c67d3 in abort () from /lib64/libc.so.6
 
(gdb) up
 
#3  0x0000000000c9ecfb in ut_dbg_assertion_failed (expr=0x1152a4f "cb->m_err == DB_SUCCESS", file=<optimized out>, line=<optimized out>, line@entry=3596) at /home/dan/repos/mariadb-server-10.8/storage/innobase/ut/ut0dbg.cc:60
 
60		abort();
 
(gdb) p cb
 
No symbol "cb" in current context.
 
(gdb) up
 
#4  0x0000000000c037d0 in io_callback (cb=<optimized out>) at /home/dan/repos/mariadb-server-10.8/storage/innobase/os/os0file.cc:3596
 
3596	  ut_a(cb->m_err == DB_SUCCESS);
 
(gdb) p cb
 
$1 = <optimized out>
 
(gdb) up
 
#5  0x0000000000d6588d in tpool::task_group::execute (this=0x2fd8980, t=0x2ffe9e0) at /home/dan/repos/mariadb-server-10.8/tpool/task_group.cc:55
 
55	        t->m_func(t->m_arg);
 
(gdb)  p *((tpool::aiocb *) t->m_arg)
 
$2 = {<iovec> = {iov_base = 0x7f3a081e4000, iov_len = 16384}, m_fh = 12, m_opcode = tpool::aio_opcode::AIO_PWRITE, m_offset = 65536, m_buffer = 0x7f3a081e4000, m_len = 16384, m_callback = 0xc03740 <io_callback(tpool::aiocb*)>, m_group = 0x2fd8980, m_ret_len = 0, m_err = 22, m_internal = 0x0, m_internal_task = {_vptr$task = 0x1174848 <vtable for tpool::task+16>, m_func = 0xc03740 <io_callback(tpool::aiocb*)>, m_arg = 0x2ffe988, m_group = 0x2fd8980}, m_userdata = "\200\342\034\b:\177\000\000\000\000\000\000\000\000\000\000H]\006\003\000\000\000\000\021\000\000\000\000\000\000"}
 
 
Thread 5 (Thread 0x7f3a0b102280 (LWP 43865)):
 
#0  0x00007f3a0b42472a in __futex_abstimed_wait_common () from /lib64/libc.so.6
 
#1  0x00007f3a0b426ed9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libc.so.6
 
#2  0x0000000000ce1c68 in buf_flush_wait (lsn=10353) at /home/dan/repos/mariadb-server-10.8/storage/innobase/buf/buf0flu.cc:1789
 
#3  buf_flush_sync () at /home/dan/repos/mariadb-server-10.8/storage/innobase/buf/buf0flu.cc:2450
 
#4  0x0000000000c7a3be in srv_start (create_new_db=<optimized out>) at /home/dan/repos/mariadb-server-10.8/storage/innobase/srv/srv0start.cc:1417
 
#5  0x0000000000ba51e7 in innodb_init (p=<optimized out>) at /home/dan/repos/mariadb-server-10.8/storage/innobase/handler/ha_innodb.cc:4216
 
#6  0x0000000000935bd8 in ha_initialize_handlerton (plugin=0x2eb35d0) at /home/dan/repos/mariadb-server-10.8/sql/handler.cc:649
 
#7  0x0000000000736982 in plugin_initialize (tmp_root=<optimized out>, plugin=plugin@entry=0x2eb35d0, argc=argc@entry=0x165d600 <remaining_argc>, argv=argv@entry=0x2e757e8, options_only=false) at /home/dan/repos/mariadb-server-10.8/sql/sql_plugin.cc:1462
 
#8  0x00000000007361f1 in plugin_init (argc=0x165d600 <remaining_argc>, argv=0x2e757e8, flags=1) at /home/dan/repos/mariadb-server-10.8/sql/sql_plugin.cc:1755
 
#9  0x000000000063432c in init_server_components () at /home/dan/repos/mariadb-server-10.8/sql/mysqld.cc:5083
 
#10 0x00000000006310ba in mysqld_main (argc=<optimized out>, argv=<optimized out>) at /home/dan/repos/mariadb-server-10.8/sql/mysqld.cc:5698
 
#11 0x00007f3a0b3c7560 in __libc_start_call_main () from /lib64/libc.so.6
 
#12 0x00007f3a0b3c760c in __libc_start_main_impl () from /lib64/libc.so.6
 
#13 0x000000000062e2f5 in _start ()
 
 
 
(gdb)  p log_sys
 
$11 = {lsn = {<std::__atomic_base<unsigned long>> = {_M_i = 10353}, <No data fields>}, flushed_to_disk_lsn = {<std::__atomic_base<unsigned long>> = {_M_i = 10353}, <No data fields>}, check_flush_or_checkpoint_ = {_M_base = {_M_i = true}}, mutex = {m_mutex = {__data = {__lock = 1, __count = 0, __owner = 43869, __nusers = 1, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = "\001\000\000\000\000\000\000\000]\253\000\000\001", '\000' <repeats 26 times>, __align = 1}, m_psi = 0x0}, buf_free = 113, max_buf_free = 8321024, flush_order_mutex = {m_mutex = {__data = {__lock = 1, __count = 0, __owner = 43869, __nusers = 1, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = "\001\000\000\000\000\000\000\000]\253\000\000\001", '\000' <repeats 26 times>, __align = 1}, m_psi = 0x0}, buf = 0x7f39e8600000 "", flush_buf = 0x7f39e7600000 "", log = {format = 1346918739, subformat = 2, file_size = 100663296, lsn = 8764, lsn_offset = 2108, fd = {m_file = {_M_t = {<std::__uniq_ptr_impl<file_io, std::default_delete<file_io> >> = {_M_t = {<std::_Tuple_impl<0, file_io*, std::default_delete<file_io> >> = {<std::_Tuple_impl<1, std::default_delete<file_io> >> = {<std::_Head_base<1, std::default_delete<file_io>, true>> = {_M_head_impl = {<No data fields>}}, <No data fields>}, <std::_Head_base<0, file_io*, false>> = {_M_head_impl = 0x306c080}, <No data fields>}, <No data fields>}}, <No data fields>}}, m_path = {_M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0x1ee6478 <log_sys+248> "./ib_logfile101"}, _M_string_length = 15, {_M_local_buf = "./ib_logfile101", _M_allocated_capacity = 7453295065250213678}}}, scanned_lsn = 0}, buf_next_to_write = 113, write_lsn = 10353, current_flush_lsn = 0, pending_flushes = {<std::__atomic_base<unsigned long>> = {_M_i = 0}, <No data fields>}, flushes = {<std::__atomic_base<unsigned long>> = {_M_i = 10}, <No data fields>}, n_log_ios = 10, n_log_ios_old = 0, last_printout_time = 1642997706, log_capacity = 90595124, max_modified_age_async = 70724346, max_checkpoint_age = 80827823, next_checkpoint_no = 3, last_checkpoint_lsn = {m = {<std::__atomic_base<unsigned long>> = {_M_i = 8752}, <No data fields>}}, next_checkpoint_lsn = 8752, n_pending_checkpoint_writes = 0, checkpoint_buf = 0x3065600 "", m_initialised = true}
 
(gdb)  p log_sys.fd
 
There is no member or method named fd.
 
(gdb)  p log_sys.log
 
$12 = {format = 1346918739, subformat = 2, file_size = 100663296, lsn = 8764, lsn_offset = 2108, fd = {m_file = {_M_t = {<std::__uniq_ptr_impl<file_io, std::default_delete<file_io> >> = {_M_t = {<std::_Tuple_impl<0, file_io*, std::default_delete<file_io> >> = {<std::_Tuple_impl<1, std::default_delete<file_io> >> = {<std::_Head_base<1, std::default_delete<file_io>, true>> = {_M_head_impl = {<No data fields>}}, <No data fields>}, <std::_Head_base<0, file_io*, false>> = {_M_head_impl = 0x306c080}, <No data fields>}, <No data fields>}}, <No data fields>}}, m_path = {_M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0x1ee6478 <log_sys+248> "./ib_logfile101"}, _M_string_length = 15, {_M_local_buf = "./ib_logfile101", _M_allocated_capacity = 7453295065250213678}}}, scanned_lsn = 0}
 
(gdb)  p log_sys.fd
 
There is no member or method named fd.
 
(gdb)  p log_sys.log.fd
 
$13 = {m_file = {_M_t = {<std::__uniq_ptr_impl<file_io, std::default_delete<file_io> >> = {_M_t = {<std::_Tuple_impl<0, file_io*, std::default_delete<file_io> >> = {<std::_Tuple_impl<1, std::default_delete<file_io> >> = {<std::_Head_base<1, std::default_delete<file_io>, true>> = {_M_head_impl = {<No data fields>}}, <No data fields>}, <std::_Head_base<0, file_io*, false>> = {_M_head_impl = 0x306c080}, <No data fields>}, <No data fields>}}, <No data fields>}}, m_path = {_M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0x1ee6478 <log_sys+248> "./ib_logfile101"}, _M_string_length = 15, {_M_local_buf = "./ib_logfile101", _M_allocated_capacity = 7453295065250213678}}}

m_err = 22 -> EINVAL

Repeatable on 10.5 with --innodb_flush_method=O_DIRECT, the 10.5 default of fsync starts correctly.

Comment by Daniel Black [ 2022-01-24 ] 

(gdb) r
 
Starting program: /home/dan/repos/build-mariadb-server-10.5/sql/mysqld --no-defaults --skip-networking --datadir=/mnt/nas/datadir --verbose --innodb_flush_method=O_DIRECT
 
 
 
Thread 1 "mysqld" hit Breakpoint 2, os_file_set_nocache (fd=<optimized out>, file_name=<optimized out>, operation_name=<optimized out>) at /home/dan/repos/mariadb-server-10.5/storage/innobase/os/os0file.cc:3228
 
warning: Source file is more recent than executable.
 
3228		if (fcntl(fd, F_SETFL, O_DIRECT) == -1) {
 
(gdb) s
 
 
 
Thread 1 "mysqld" hit Breakpoint 1, 0x00007ffff75d5c60 in fcntl64 () from /lib64/libc.so.6
 
(gdb) finish
 
Run till exit from #0  0x00007ffff75d5c60 in fcntl64 () from /lib64/libc.so.6
 
0x0000000001bf152b in os_file_set_nocache (fd=<optimized out>, file_name=<optimized out>, operation_name=<optimized out>) at /home/dan/repos/mariadb-server-10.5/storage/innobase/os/os0file.cc:3228
 
3228		if (fcntl(fd, F_SETFL, O_DIRECT) == -1) {
 
(gdb) s
 
3259	}
 
(gdb) p errno
 
'errno' has unknown type; cast it to its declared type
 
(gdb) p (int) errno
 
$1 = 0

So in short the setting of the file mode to O_DIRECT succeeds but fails when used. The 10.6 work around is to set --innodb_flush_method=flush on CIFS mounts.

related (below), tmpfs will return EINVAL. btrfs is failing under some condition.

https://jira.mariadb.org/browse/MDEV-24854?focusedCommentId=211665&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-211665

Comment by Daniel Black [ 2022-01-25 ]

I've emailed linuxfs-devel developers about potential kernel based solutions.

Comment by Daniel Black [ 2022-02-07 ]

linuxfs-devel:

  • commitment to having fcntl(fd, F_SETFL, O_DIRECT) error if not available
  • cifs does have direct IO support, however not clear under what conditions, MDEV-26995 cache=none might the the conditions
  • if so we should have to check it somehow.
Comment by Daniel Black [ 2022-02-15 ]

cache=none as a mount option seems to be a pre-requisite for directio: https://lwn.net/Articles/756061/

With this, still observered failures:

observed failures in mysql_install_db-10.8-3b06415cb8dfc2b10d9604950f6ed945bc016425

 
+ mysqld_install_cmd_line
 
+ /home/dan/repos/build-mariadb-server-10.8/sql/mysqld --no-defaults --lc-messages-dir=/home/dan/repos/build-mariadb-server-10.8/sql/share/english/.. --bootstrap --silent-startup --basedir=/home/dan/repos/build-mariadb-server-10.8 --datadir=/mnt/nas/datadir108 --log-warnings=0 --enforce-storage-engine= --plugin-dir=/home/dan/repos/build-mariadb-server-10.8/plugin/auth_socket --loose-disable-plugin-file-key-management --max_allowed_packet=8M --net_buffer_length=16K
 
+ test 0 -eq 0
 
+ cat /home/dan/repos/mariadb-server-10.8/scripts/mysql_test_db.sql
 
+ test '!' -z ''
 
2022-02-15 17:32:26 0x7fd36cbfc640  InnoDB: Assertion failure in file /home/dan/repos/mariadb-server-10.8/storage/innobase/os/os0file.cc line 3571
 
InnoDB: Failing assertion: cb->m_err == DB_SUCCESS
 
InnoDB: We intentionally generate a memory trap.
 
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
 
InnoDB: If you get repeated assertion failures or crashes, even
 
InnoDB: immediately after the mariadbd startup, there may be
 
InnoDB: corruption in the InnoDB tablespace. Please refer to
 
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
 
InnoDB: about forcing recovery.
 
220215 17:32:26 [ERROR] mysqld got signal 6 ;
 
 
 
#0  0x00007fd3918ca88c in __pthread_kill_implementation () from /lib64/libc.so.6
 
[Current thread is 1 (Thread 0x7fd33ffff640 (LWP 1060236))]
 
(gdb) bt
 
#0  0x00007fd3918ca88c in __pthread_kill_implementation () from /lib64/libc.so.6
 
#1  0x00007fd39187d6a6 in raise () from /lib64/libc.so.6
 
#2  0x00007fd3918677d3 in abort () from /lib64/libc.so.6
 
#3  0x0000000000ca6e9b in ut_dbg_assertion_failed (expr=0x1160de8 "cb->m_err == DB_SUCCESS", file=<optimized out>, line=<optimized out>, line@entry=3571) at /home/dan/repos/mariadb-server-10.8/storage/innobase/ut/ut0dbg.cc:60
 
#4  0x0000000000c0c2d0 in io_callback (cb=<optimized out>) at /home/dan/repos/mariadb-server-10.8/storage/innobase/os/os0file.cc:3571
 
#5  0x0000000000d6d6dd in tpool::task_group::execute (this=0x2a238b0, t=0x2a48c98) at /home/dan/repos/mariadb-server-10.8/tpool/task_group.cc:55
 
#6  0x0000000000d6b7c6 in tpool::thread_pool_generic::worker_main (this=0x29b8630, thread_var=0x29c7ed0) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:549
 
#7  0x00007fd391b395c4 in execute_native_thread_routine () from /lib64/libstdc++.so.6
 
#8  0x00007fd3918c8b1a in start_thread () from /lib64/libc.so.6
 
#9  0x00007fd39194d650 in clone3 () from /lib64/libc.so.6
 
(gdb) thread apply all bt 
 
 
Thread 9 (Thread 0x7fd337fff640 (LWP 1060235)):
 
#0  0x00007fd3918c573a in __futex_abstimed_wait_common () from /lib64/libc.so.6
 
#1  0x00007fd3918c8482 in pthread_cond_clockwait@GLIBC_2.30 () from /lib64/libc.so.6
 
#2  0x0000000000d6b43c in std::__condvar::wait_until (this=0x29c7e50, __m=<error reading variable: Cannot access memory at address 0x89>, __clock=1, __abs_time=@0x7fd337ffecb8: {tv_sec = 177671, tv_nsec = 568528669}) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/std_mutex.h:169
 
#3  std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x29c7e50, __lock=@0x7fd337ffed00: {_M_device = 0x29b8740, _M_owns = true}, __atime=<optimized out>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:201
 
#4  std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x29c7e50, __lock=@0x7fd337ffed00: {_M_device = 0x29b8740, _M_owns = true}, __atime=<optimized out>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:111
 
#5  std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (this=0x29c7e50, __lock=@0x7fd337ffed00: {_M_device = 0x29b8740, _M_owns = true}, __rtime=@0x29b8768: {__r = 60000}) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:163
 
#6  tpool::thread_pool_generic::wait_for_tasks (this=this@entry=0x29b8630, lk=@0x7fd337ffed00: {_M_device = 0x29b8740, _M_owns = true}, thread_data=thread_data@entry=0x29c7e50) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:449
 
#7  0x0000000000d6b5ce in tpool::thread_pool_generic::get_task (this=this@entry=0x29b8630, thread_var=thread_var@entry=0x29c7e50, t=t@entry=0x7fd337ffed50) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:502
 
#8  0x0000000000d6b7a8 in tpool::thread_pool_generic::worker_main (this=0x29b8630, thread_var=0x29c7e50) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:547
 
#9  0x00007fd391b395c4 in execute_native_thread_routine () from /lib64/libstdc++.so.6
 
#10 0x00007fd3918c8b1a in start_thread () from /lib64/libc.so.6
 
#11 0x00007fd39194d650 in clone3 () from /lib64/libc.so.6
 
 
 
Thread 8 (Thread 0x7fd36cbfc640 (LWP 1059793)):
 
#0  0x00007fd39194079f in poll () from /lib64/libc.so.6
 
#1  0x0000000000de2de1 in addr_resolve (ptr=<optimized out>, loc=loc@entry=0x7fd36cbfa318) at /home/dan/repos/mariadb-server-10.8/mysys/my_addr_resolve.c:245
 
#2  0x0000000000de2cbc in my_addr_resolve (ptr=0xca6e9b, loc=loc@entry=0x7fd36cbfa318) at /home/dan/repos/mariadb-server-10.8/mysys/my_addr_resolve.c:330
 
#3  0x0000000000dcb11d in print_with_addr_resolve (addrs=0x7fd36cbf9f10, n=<optimized out>) at /home/dan/repos/mariadb-server-10.8/mysys/stacktrace.c:197
 
#4  my_print_stacktrace (stack_bottom=<optimized out>, thread_stack=<optimized out>, silent=<optimized out>) at /home/dan/repos/mariadb-server-10.8/mysys/stacktrace.c:216
 
#5  0x000000000093e3b4 in handle_fatal_signal (sig=6) at /home/dan/repos/mariadb-server-10.8/sql/signal_handler.cc:223
 
#6  <signal handler called>
 
#7  0x00007fd3918ca88c in __pthread_kill_implementation () from /lib64/libc.so.6
 
#8  0x00007fd39187d6a6 in raise () from /lib64/libc.so.6
 
#9  0x00007fd3918677d3 in abort () from /lib64/libc.so.6
 
#10 0x0000000000ca6e9b in ut_dbg_assertion_failed (expr=0x1160de8 "cb->m_err == DB_SUCCESS", file=<optimized out>, line=<optimized out>, line@entry=3571) at /home/dan/repos/mariadb-server-10.8/storage/innobase/ut/ut0dbg.cc:60
 
#11 0x0000000000c0c2d0 in io_callback (cb=<optimized out>) at /home/dan/repos/mariadb-server-10.8/storage/innobase/os/os0file.cc:3571
 
#12 0x0000000000d6d6dd in tpool::task_group::execute (this=0x2a238b0, t=0x2a48dc8) at /home/dan/repos/mariadb-server-10.8/tpool/task_group.cc:55
 
#13 0x0000000000d6b7c6 in tpool::thread_pool_generic::worker_main (this=0x29b8630, thread_var=0x29c8250) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:549
 
#14 0x00007fd391b395c4 in execute_native_thread_routine () from /lib64/libstdc++.so.6
 
#15 0x00007fd3918c8b1a in start_thread () from /lib64/libc.so.6
 
#16 0x00007fd39194d650 in clone3 () from /lib64/libc.so.6
 
 
 
Thread 7 (Thread 0x7fd3915cb280 (LWP 1059616)):
 
#0  0x00007fd391945ecd in syscall () from /lib64/libc.so.6
 
#1  0x00007fd3921d53b2 in __io_uring_submit () from /lib64/liburing.so.2
 
#2  0x0000000000d6e17b in (anonymous namespace)::aio_uring::submit_io (this=<optimized out>, cb=0x2a48fd0) at /home/dan/repos/mariadb-server-10.8/tpool/aio_liburing.cc:108
 
#3  0x0000000000c0c1a3 in tpool::thread_pool::submit_io (this=<optimized out>, cb=0x2a48fd0) at /home/dan/repos/mariadb-server-10.8/tpool/tpool.h:226
 
#4  os_aio (type=@0x7fff94f5ca48: {bpage = 0x7fd37c1a1dc0, slot = 0x0, node = 0x2ab06d8, type = IORequest::WRITE_ASYNC}, buf=0x7fd37c200000, offset=<optimized out>, n=16384) at /home/dan/repos/mariadb-server-10.8/storage/innobase/os/os0file.cc:3874
 
#5  0x0000000000d2218c in fil_space_t::io (this=0x2ab0578, type=@0x7fff94f5cad0: {bpage = 0x7fd37c1a1dc0, slot = 0x0, node = 0x0, type = IORequest::WRITE_ASYNC}, offset=0, len=1, buf=0x0, bpage=0x8) at /home/dan/repos/mariadb-server-10.8/storage/innobase/fil/fil0fil.cc:2810
 
#6  0x0000000000cea341 in buf_page_t::flush (this=this@entry=0x7fd37c1a1dc0, lru=<optimized out>, space=0x2ab0578) at /home/dan/repos/mariadb-server-10.8/storage/innobase/buf/buf0flu.cc:910
 
#7  0x0000000000ce77cd in buf_do_flush_list_batch (max_n=2000, lsn=18446744073709551615) at /home/dan/repos/mariadb-server-10.8/storage/innobase/buf/buf0flu.cc:1435
 
#8  buf_flush_list (max_n=<optimized out>, lsn=<optimized out>, lsn@entry=18446744073709551615) at /home/dan/repos/mariadb-server-10.8/storage/innobase/buf/buf0flu.cc:1510
 
#9  0x0000000000ce911d in buf_flush_buffer_pool () at /home/dan/repos/mariadb-server-10.8/storage/innobase/buf/buf0flu.cc:2440
 
#10 0x0000000000bf5418 in logs_empty_and_mark_files_at_shutdown () at /home/dan/repos/mariadb-server-10.8/storage/innobase/log/log0log.cc:902
 
#11 0x0000000000c8525b in innodb_shutdown () at /home/dan/repos/mariadb-server-10.8/storage/innobase/srv/srv0start.cc:1680
 
#12 0x0000000000bb3c78 in innobase_end () at /home/dan/repos/mariadb-server-10.8/storage/innobase/handler/ha_innodb.cc:4280
 
#13 0x000000000093f08e in ha_finalize_handlerton (plugin=0x28fe5e0) at /home/dan/repos/mariadb-server-10.8/sql/handler.cc:596
 
#14 0x000000000073cef2 in plugin_deinitialize (plugin=0x28fe5e0, ref_check=true) at /home/dan/repos/mariadb-server-10.8/sql/sql_plugin.cc:1270
 
#15 0x000000000073ba1a in reap_plugins () at /home/dan/repos/mariadb-server-10.8/sql/sql_plugin.cc:1341
 
#16 0x000000000073d272 in plugin_shutdown () at /home/dan/repos/mariadb-server-10.8/sql/sql_plugin.cc:2049
 
#17 0x00000000006378a5 in clean_up (print_message=false) at /home/dan/repos/mariadb-server-10.8/sql/mysqld.cc:1959
 
#18 0x0000000000637768 in unireg_abort (exit_code=0) at /home/dan/repos/mariadb-server-10.8/sql/mysqld.cc:1871
 
#19 0x00000000006399eb in mysqld_main (argc=<optimized out>, argv=<optimized out>) at /home/dan/repos/mariadb-server-10.8/sql/mysqld.cc:5793
 
#20 0x00007fd391868560 in __libc_start_call_main () from /lib64/libc.so.6
 
#21 0x00007fd39186860c in __libc_start_main_impl () from /lib64/libc.so.6
 
#22 0x00000000006369f5 in _start ()
 
 
 
Thread 6 (Thread 0x7fd36dbfe640 (LWP 1060215)):
 
#0  0x00007fd3918c573a in __futex_abstimed_wait_common () from /lib64/libc.so.6
 
#1  0x00007fd3918c8482 in pthread_cond_clockwait@GLIBC_2.30 () from /lib64/libc.so.6
 
#2  0x0000000000d6b43c in std::__condvar::wait_until (this=0x29c7dd0, __m=<error reading variable: Cannot access memory at address 0x89>, __clock=1, __abs_time=@0x7fd36dbfdcb8: {tv_sec = 177671, tv_nsec = 568514021}) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/std_mutex.h:169
 
#3  std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x29c7dd0, __lock=@0x7fd36dbfdd00: {_M_device = 0x29b8740, _M_owns = true}, __atime=<optimized out>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:201
 
#4  std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x29c7dd0, __lock=@0x7fd36dbfdd00: {_M_device = 0x29b8740, _M_owns = true}, __atime=<optimized out>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:111
 
#5  std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (this=0x29c7dd0, __lock=@0x7fd36dbfdd00: {_M_device = 0x29b8740, _M_owns = true}, __rtime=@0x29b8768: {__r = 60000}) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:163
 
#6  tpool::thread_pool_generic::wait_for_tasks (this=this@entry=0x29b8630, lk=@0x7fd36dbfdd00: {_M_device = 0x29b8740, _M_owns = true}, thread_data=thread_data@entry=0x29c7dd0) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:449
 
#7  0x0000000000d6b5ce in tpool::thread_pool_generic::get_task (this=this@entry=0x29b8630, thread_var=thread_var@entry=0x29c7dd0, t=t@entry=0x7fd36dbfdd50) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:502
 
#8  0x0000000000d6b7d4 in tpool::thread_pool_generic::worker_main (this=0x29b8630, thread_var=0x29c7dd0) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:547
 
#9  0x00007fd391b395c4 in execute_native_thread_routine () from /lib64/libstdc++.so.6
 
#10 0x00007fd3918c8b1a in start_thread () from /lib64/libc.so.6
 
#11 0x00007fd39194d650 in clone3 () from /lib64/libc.so.6
 
 
 
Thread 5 (Thread 0x7fd377fff640 (LWP 1059619)):
 
#0  0x00007fd391945ecd in syscall () from /lib64/libc.so.6
 
#1  0x00007fd3921d54f9 in _io_uring_get_cqe () from /lib64/liburing.so.2
 
#2  0x00007fd3921d5a9b in __io_uring_get_cqe () from /lib64/liburing.so.2
 
#3  0x0000000000d6e072 in io_uring_wait_cqe_nr (ring=0x2a4b9c8, cqe_ptr=0x7fd377ffed48, wait_nr=1) at /usr/include/liburing.h:635
 
#4  io_uring_wait_cqe (ring=0x2a4b9c8, cqe_ptr=0x7fd377ffed48) at /usr/include/liburing.h:655
 
#5  (anonymous namespace)::aio_uring::thread_routine (aio=0x2a4b9c0) at /home/dan/repos/mariadb-server-10.8/tpool/aio_liburing.cc:137
 
#6  0x00007fd391b395c4 in execute_native_thread_routine () from /lib64/libstdc++.so.6
 
#7  0x00007fd3918c8b1a in start_thread () from /lib64/libc.so.6
 
#8  0x00007fd39194d650 in clone3 () from /lib64/libc.so.6
 
 
 
Thread 4 (Thread 0x7fd37c1a0640 (LWP 1059657)):
 
#0  0x00007fd39187e43a in sigtimedwait () from /lib64/libc.so.6
 
#1  0x0000000000638bcb in my_sigwait (set=0x7fd37c19fc20, sig=0x7fd37c19fd20, code=<optimized out>) at /home/dan/repos/mariadb-server-10.8/include/my_pthread.h:195
 
#2  signal_hand (arg=<optimized out>) at /home/dan/repos/mariadb-server-10.8/sql/mysqld.cc:3152
 
#3  0x0000000000b21a4e in pfs_spawn_thread (arg=0x2e21018) at /home/dan/repos/mariadb-server-10.8/storage/perfschema/pfs.cc:2201
 
#4  0x00007fd3918c8b1a in start_thread () from /lib64/libc.so.6
 
#5  0x00007fd39194d650 in clone3 () from /lib64/libc.so.6
 
 
 
Thread 3 (Thread 0x7fd390c25640 (LWP 1059618)):
 
#0  0x00007fd3918c573a in __futex_abstimed_wait_common () from /lib64/libc.so.6
 
#1  0x00007fd3918c81b4 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libc.so.6
 
#2  0x0000000000ac73a3 in inline_mysql_cond_timedwait (that=0x1ea38b8 <COND_checkpoint+40>, mutex=0x189, abstime=0x7fd390c24d00, src_line=116, src_file=<optimized out>) at /home/dan/repos/mariadb-server-10.8/include/mysql/psi/mysql_thread.h:1088
 
#3  my_service_thread_sleep (control=0x15d4c70 <checkpoint_control>, sleep_time=30000000000) at /home/dan/repos/mariadb-server-10.8/storage/maria/ma_servicethread.c:115
 
#4  0x0000000000ac081f in ma_checkpoint_background (arg=<optimized out>) at /home/dan/repos/mariadb-server-10.8/storage/maria/ma_checkpoint.c:725
 
#5  0x00007fd3918c8b1a in start_thread () from /lib64/libc.so.6
 
#6  0x00007fd39194d650 in clone3 () from /lib64/libc.so.6
 
 
 
Thread 2 (Thread 0x7fd3922cb640 (LWP 1059617)):
 
#0  0x00007fd3918c573a in __futex_abstimed_wait_common () from /lib64/libc.so.6
 
#1  0x00007fd3918c81b4 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libc.so.6
 
#2  0x0000000000dcec62 in inline_mysql_cond_timedwait (abstime=0x7fd3922cad50, src_line=321, that=<optimized out>, mutex=<optimized out>, src_file=<optimized out>) at /home/dan/repos/mariadb-server-10.8/include/mysql/psi/mysql_thread.h:1088
 
#3  timer_handler (arg=<optimized out>) at /home/dan/repos/mariadb-server-10.8/mysys/thr_timer.c:321
 
#4  0x00007fd3918c8b1a in start_thread () from /lib64/libc.so.6
 
#5  0x00007fd39194d650 in clone3 () from /lib64/libc.so.6
 
 
 
Thread 1 (Thread 0x7fd33ffff640 (LWP 1060236)):
 
#0  0x00007fd3918ca88c in __pthread_kill_implementation () from /lib64/libc.so.6
 
#1  0x00007fd39187d6a6 in raise () from /lib64/libc.so.6
 
#2  0x00007fd3918677d3 in abort () from /lib64/libc.so.6
 
#3  0x0000000000ca6e9b in ut_dbg_assertion_failed (expr=0x1160de8 "cb->m_err == DB_SUCCESS", file=<optimized out>, line=<optimized out>, line@entry=3571) at /home/dan/repos/mariadb-server-10.8/storage/innobase/ut/ut0dbg.cc:60
 
#4  0x0000000000c0c2d0 in io_callback (cb=<optimized out>) at /home/dan/repos/mariadb-server-10.8/storage/innobase/os/os0file.cc:3571
 
#5  0x0000000000d6d6dd in tpool::task_group::execute (this=0x2a238b0, t=0x2a48c98) at /home/dan/repos/mariadb-server-10.8/tpool/task_group.cc:55
 
#6  0x0000000000d6b7c6 in tpool::thread_pool_generic::worker_main (this=0x29b8630, thread_var=0x29c7ed0) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:549
 
#7  0x00007fd391b395c4 in execute_native_thread_routine () from /lib64/libstdc++.so.6
 
#8  0x00007fd3918c8b1a in start_thread () from /lib64/libc.so.6
 
#9  0x00007fd39194d650 in clone3 () from /lib64/libc.so.6
 
(gdb) up
 
#1  0x00007fd39187d6a6 in raise () from /lib64/libc.so.6
 
(gdb) 
#2  0x00007fd3918677d3 in abort () from /lib64/libc.so.6
 
(gdb) 
#3  0x0000000000ca6e9b in ut_dbg_assertion_failed (expr=0x1160de8 "cb->m_err == DB_SUCCESS", file=<optimized out>, line=<optimized out>, line@entry=3571) at /home/dan/repos/mariadb-server-10.8/storage/innobase/ut/ut0dbg.cc:60
 
60		abort();
 
(gdb) 
#4  0x0000000000c0c2d0 in io_callback (cb=<optimized out>) at /home/dan/repos/mariadb-server-10.8/storage/innobase/os/os0file.cc:3571
 
3571	  ut_a(cb->m_err == DB_SUCCESS);
 
(gdb) 
#5  0x0000000000d6d6dd in tpool::task_group::execute (this=0x2a238b0, t=0x2a48c98) at /home/dan/repos/mariadb-server-10.8/tpool/task_group.cc:55
 
55	        t->m_func(t->m_arg);
 
(gdb)   p *((tpool::aiocb *) t->m_arg)
 
$1 = {<iovec> = {iov_base = 0x7fd37c204000, iov_len = 16384}, m_fh = 12, m_opcode = tpool::aio_opcode::AIO_PWRITE, m_offset = 3145728, m_buffer = 0x7fd37c204000, m_len = 16384, m_callback = 0xc0c240 <io_callback(tpool::aiocb*)>, m_group = 0x2a238b0, m_ret_len = 0, m_err = 4, m_internal = 0x0, m_internal_task = {_vptr$task = 0x1182828 <vtable for tpool::task+16>, m_func = 0xc0c240 <io_callback(tpool::aiocb*)>, m_arg = 0x2a48c40, m_group = 0x2a238b0}, m_userdata = "`\036\032|\323\177\000\000\000\000\000\000\000\000\000\000\330\006\253\002\000\000\000\000\021\000\000\000\323\177\000"}

m_err = 4 (EINTR). This maps the kernel errors:

kernel errors

 
[174428.702577] CIFS: Attempting to mount \\192.168.178.171\dan
 
[174504.683290] CIFS: VFS: \\192.168.178.171 Error -512 sending data on socket to server
 
[174504.815427] CIFS: VFS: cifs_setlk failed rc=-22
 
[175230.473589] CIFS: VFS: \\192.168.178.171 Error -512 sending data on socket to server
 
[175230.991370] CIFS: VFS: \\192.168.178.171 Error -512 sending data on socket to server
 
[175232.406643] CIFS: VFS: \\192.168.178.171 Error -512 sending data on socket to server
 
[175232.868178] CIFS: VFS: \\192.168.178.171 Error -512 sending data on socket to server
 
[175233.480149] CIFS: VFS: \\192.168.178.171 Error -512 sending data on socket to server
 
[176260.874005] CIFS: VFS: \\192.168.178.171 Error -512 sending data on socket to server
 
[176260.971757] CIFS: VFS: cifs_setlk failed rc=-22
 
[177282.983673] CIFS: VFS: \\192.168.178.171 Error -512 sending data on socket to server
 
[177607.913858] CIFS: VFS: \\192.168.178.171 Error -512 sending data on socket to server
 
[177608.525167] CIFS: VFS: \\192.168.178.171 Error -512 sending data on socket to server
 
[178379.697057] CIFS: VFS: \\192.168.178.171 Error -512 sending data on socket to server

Case 2 also on mysql_install_db:

10.8 fdatasync(thread 9, fd=12), interupts write

 
Core was generated by `/home/dan/repos/build-mariadb-server-10.8/sql/mysqld --no-defaults --lc-message'.
 
Program terminated with signal SIGABRT, Aborted.
 
#0  0x00007f39ff7fe88c in __pthread_kill_implementation () from /lib64/libc.so.6
 
[Current thread is 1 (Thread 0x7f39db1fe640 (LWP 1063553))]
 
(gdb) bt
 
#0  0x00007f39ff7fe88c in __pthread_kill_implementation () from /lib64/libc.so.6
 
#1  0x00007f39ff7b16a6 in raise () from /lib64/libc.so.6
 
#2  0x00007f39ff79b865 in abort () from /lib64/libc.so.6
 
#3  0x0000000000ca6e9b in ut_dbg_assertion_failed (expr=0x1160de8 "cb->m_err == DB_SUCCESS", file=<optimized out>, line=<optimized out>, line@entry=3571) at /home/dan/repos/mariadb-server-10.8/storage/innobase/ut/ut0dbg.cc:60
 
#4  0x0000000000c0c2d0 in io_callback (cb=<optimized out>) at /home/dan/repos/mariadb-server-10.8/storage/innobase/os/os0file.cc:3571
 
#5  0x0000000000d6d6dd in tpool::task_group::execute (this=0x33358b0, t=0x335b450) at /home/dan/repos/mariadb-server-10.8/tpool/task_group.cc:55
 
#6  0x0000000000d6b7c6 in tpool::thread_pool_generic::worker_main (this=0x32ca630, thread_var=0x32da2d0) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:549
 
#7  0x00007f39ffa6d5c4 in execute_native_thread_routine () from /lib64/libstdc++.so.6
 
#8  0x00007f39ff7fcb1a in start_thread () from /lib64/libc.so.6
 
#9  0x00007f39ff881650 in clone3 () from /lib64/libc.so.6
 
(gdb) up
 
#1  0x00007f39ff7b16a6 in raise () from /lib64/libc.so.6
 
(gdb) thread apply all bt
 
 
 
Thread 16 (Thread 0x7f3a001ff640 (LWP 1063541)):
 
#0  0x00007f39ff7f973a in __futex_abstimed_wait_common () from /lib64/libc.so.6
 
#1  0x00007f39ff7fc1b4 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libc.so.6
 
#2  0x0000000000dcec62 in inline_mysql_cond_timedwait (abstime=0x7f3a001fed50, src_line=321, that=<optimized out>, mutex=<optimized out>, src_file=<optimized out>) at /home/dan/repos/mariadb-server-10.8/include/mysql/psi/mysql_thread.h:1088
 
#3  timer_handler (arg=<optimized out>) at /home/dan/repos/mariadb-server-10.8/mysys/thr_timer.c:321
 
#4  0x00007f39ff7fcb1a in start_thread () from /lib64/libc.so.6
 
#5  0x00007f39ff881650 in clone3 () from /lib64/libc.so.6
 
 
 
Thread 15 (Thread 0x7f39b9ffb640 (LWP 1063573)):
 
#0  0x00007f39ff7f973a in __futex_abstimed_wait_common () from /lib64/libc.so.6
 
#1  0x00007f39ff7fc482 in pthread_cond_clockwait@GLIBC_2.30 () from /lib64/libc.so.6
 
#2  0x0000000000d6b43c in std::__condvar::wait_until (this=0x32d9dd0, __m=<error reading variable: Cannot access memory at address 0x89>, __clock=1, __abs_time=@0x7f39b9ffacb8: {tv_sec = 179179, tv_nsec = 584891184}) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/std_mutex.h:169
 
#3  std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x32d9dd0, __lock=@0x7f39b9ffad00: {_M_device = 0x32ca740, _M_owns = true}, __atime=<optimized out>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:201
 
#4  std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x32d9dd0, __lock=@0x7f39b9ffad00: {_M_device = 0x32ca740, _M_owns = true}, __atime=<optimized out>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:111
 
#5  std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (this=0x32d9dd0, __lock=@0x7f39b9ffad00: {_M_device = 0x32ca740, _M_owns = true}, __rtime=@0x32ca768: {__r = 60000}) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:163
 
#6  tpool::thread_pool_generic::wait_for_tasks (this=this@entry=0x32ca630, lk=@0x7f39b9ffad00: {_M_device = 0x32ca740, _M_owns = true}, thread_data=thread_data@entry=0x32d9dd0) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:449
 
#7  0x0000000000d6b5ce in tpool::thread_pool_generic::get_task (this=this@entry=0x32ca630, thread_var=thread_var@entry=0x32d9dd0, t=t@entry=0x7f39b9ffad50) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:502
 
#8  0x0000000000d6b7a8 in tpool::thread_pool_generic::worker_main (this=0x32ca630, thread_var=0x32d9dd0) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:547
 
#9  0x00007f39ffa6d5c4 in execute_native_thread_routine () from /lib64/libstdc++.so.6
 
#10 0x00007f39ff7fcb1a in start_thread () from /lib64/libc.so.6
 
#11 0x00007f39ff881650 in clone3 () from /lib64/libc.so.6
 
 
 
Thread 14 (Thread 0x7f39baffd640 (LWP 1063571)):
 
#0  0x00007f39ff7f973a in __futex_abstimed_wait_common () from /lib64/libc.so.6
 
#1  0x00007f39ff7fc482 in pthread_cond_clockwait@GLIBC_2.30 () from /lib64/libc.so.6
 
#2  0x0000000000d6b43c in std::__condvar::wait_until (this=0x32d9ed0, __m=<error reading variable: Cannot access memory at address 0x89>, __clock=1, __abs_time=@0x7f39baffccb8: {tv_sec = 179179, tv_nsec = 584961467}) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/std_mutex.h:169
 
#3  std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x32d9ed0, __lock=@0x7f39baffcd00: {_M_device = 0x32ca740, _M_owns = true}, __atime=<optimized out>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:201
 
#4  std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x32d9ed0, __lock=@0x7f39baffcd00: {_M_device = 0x32ca740, _M_owns = true}, __atime=<optimized out>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:111
 
#5  std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (this=0x32d9ed0, __lock=@0x7f39baffcd00: {_M_device = 0x32ca740, _M_owns = true}, __rtime=@0x32ca768: {__r = 60000}) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:163
 
#6  tpool::thread_pool_generic::wait_for_tasks (this=this@entry=0x32ca630, lk=@0x7f39baffcd00: {_M_device = 0x32ca740, _M_owns = true}, thread_data=thread_data@entry=0x32d9ed0) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:449
 
#7  0x0000000000d6b5ce in tpool::thread_pool_generic::get_task (this=this@entry=0x32ca630, thread_var=thread_var@entry=0x32d9ed0, t=t@entry=0x7f39baffcd50) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:502
 
#8  0x0000000000d6b7a8 in tpool::thread_pool_generic::worker_main (this=0x32ca630, thread_var=0x32d9ed0) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:547
 
#9  0x00007f39ffa6d5c4 in execute_native_thread_routine () from /lib64/libstdc++.so.6
 
#10 0x00007f39ff7fcb1a in start_thread () from /lib64/libc.so.6
 
#11 0x00007f39ff881650 in clone3 () from /lib64/libc.so.6
 
 
 
Thread 13 (Thread 0x7f39d89f9640 (LWP 1063568)):
 
#0  0x00007f39ff7f973a in __futex_abstimed_wait_common () from /lib64/libc.so.6
 
#1  0x00007f39ff7fc482 in pthread_cond_clockwait@GLIBC_2.30 () from /lib64/libc.so.6
 
#2  0x0000000000d6b43c in std::__condvar::wait_until (this=0x32da050, __m=<error reading variable: Cannot access memory at address 0x89>, __clock=1, __abs_time=@0x7f39d89f8cb8: {tv_sec = 179179, tv_nsec = 584950335}) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/std_mutex.h:169
 
#3  std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x32da050, __lock=@0x7f39d89f8d00: {_M_device = 0x32ca740, _M_owns = true}, __atime=<optimized out>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:201
 
#4  std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x32da050, __lock=@0x7f39d89f8d00: {_M_device = 0x32ca740, _M_owns = true}, __atime=<optimized out>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:111
 
#5  std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (this=0x32da050, __lock=@0x7f39d89f8d00: {_M_device = 0x32ca740, _M_owns = true}, __rtime=@0x32ca768: {__r = 60000}) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:163
 
#6  tpool::thread_pool_generic::wait_for_tasks (this=this@entry=0x32ca630, lk=@0x7f39d89f8d00: {_M_device = 0x32ca740, _M_owns = true}, thread_data=thread_data@entry=0x32da050) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:449
 
#7  0x0000000000d6b5ce in tpool::thread_pool_generic::get_task (this=this@entry=0x32ca630, thread_var=thread_var@entry=0x32da050, t=t@entry=0x7f39d89f8d50) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:502
 
#8  0x0000000000d6b7d4 in tpool::thread_pool_generic::worker_main (this=0x32ca630, thread_var=0x32da050) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:547
 
#9  0x00007f39ffa6d5c4 in execute_native_thread_routine () from /lib64/libstdc++.so.6
 
#10 0x00007f39ff7fcb1a in start_thread () from /lib64/libc.so.6
 
#11 0x00007f39ff881650 in clone3 () from /lib64/libc.so.6
 
 
 
Thread 12 (Thread 0x7f39fd3ff640 (LWP 1063546)):
 
#0  0x00007f39ff7f973a in __futex_abstimed_wait_common () from /lib64/libc.so.6
 
#1  0x00007f39ff7fc1b4 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libc.so.6
 
#2  0x0000000000ac73a3 in inline_mysql_cond_timedwait (that=0x1ea38b8 <COND_checkpoint+40>, mutex=0x189, abstime=0x7f39fd3fed00, src_line=116, src_file=<optimized out>) at /home/dan/repos/mariadb-server-10.8/include/mysql/psi/mysql_thread.h:1088
 
#3  my_service_thread_sleep (control=0x15d4c70 <checkpoint_control>, sleep_time=29000000000) at /home/dan/repos/mariadb-server-10.8/storage/maria/ma_servicethread.c:115
 
#4  0x0000000000ac081f in ma_checkpoint_background (arg=<optimized out>) at /home/dan/repos/mariadb-server-10.8/storage/maria/ma_checkpoint.c:725
 
#5  0x00007f39ff7fcb1a in start_thread () from /lib64/libc.so.6
 
#6  0x00007f39ff881650 in clone3 () from /lib64/libc.so.6
 
 
 
Thread 11 (Thread 0x7f39da1fc640 (LWP 1063565)):
 
#0  0x00007f39ff7f973a in __futex_abstimed_wait_common () from /lib64/libc.so.6
 
#1  0x00007f39ff7fc482 in pthread_cond_clockwait@GLIBC_2.30 () from /lib64/libc.so.6
 
#2  0x0000000000d6b43c in std::__condvar::wait_until (this=0x32da1d0, __m=<error reading variable: Cannot access memory at address 0x89>, __clock=1, __abs_time=@0x7f39da1fbcb8: {tv_sec = 179179, tv_nsec = 584899718}) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/std_mutex.h:169
 
#3  std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x32da1d0, __lock=@0x7f39da1fbd00: {_M_device = 0x32ca740, _M_owns = true}, __atime=<optimized out>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:201
 
#4  std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x32da1d0, __lock=@0x7f39da1fbd00: {_M_device = 0x32ca740, _M_owns = true}, __atime=<optimized out>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:111
 
#5  std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (this=0x32da1d0, __lock=@0x7f39da1fbd00: {_M_device = 0x32ca740, _M_owns = true}, __rtime=@0x32ca768: {__r = 60000}) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:163
 
#6  tpool::thread_pool_generic::wait_for_tasks (this=this@entry=0x32ca630, lk=@0x7f39da1fbd00: {_M_device = 0x32ca740, _M_owns = true}, thread_data=thread_data@entry=0x32da1d0) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:449
 
#7  0x0000000000d6b5ce in tpool::thread_pool_generic::get_task (this=this@entry=0x32ca630, thread_var=thread_var@entry=0x32da1d0, t=t@entry=0x7f39da1fbd50) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:502
 
#8  0x0000000000d6b7a8 in tpool::thread_pool_generic::worker_main (this=0x32ca630, thread_var=0x32da1d0) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:547
 
#9  0x00007f39ffa6d5c4 in execute_native_thread_routine () from /lib64/libstdc++.so.6
 
#10 0x00007f39ff7fcb1a in start_thread () from /lib64/libc.so.6
 
#11 0x00007f39ff881650 in clone3 () from /lib64/libc.so.6
 
 
 
Thread 10 (Thread 0x7f39ba7fc640 (LWP 1063572)):
 
#0  0x00007f39ff7f973a in __futex_abstimed_wait_common () from /lib64/libc.so.6
 
#1  0x00007f39ff7fc482 in pthread_cond_clockwait@GLIBC_2.30 () from /lib64/libc.so.6
 
#2  0x0000000000d6b43c in std::__condvar::wait_until (this=0x32d9e50, __m=<error reading variable: Cannot access memory at address 0x89>, __clock=1, __abs_time=@0x7f39ba7fbcb8: {tv_sec = 179179, tv_nsec = 584886846}) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/std_mutex.h:169
 
#3  std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x32d9e50, __lock=@0x7f39ba7fbd00: {_M_device = 0x32ca740, _M_owns = true}, __atime=<optimized out>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:201
 
#4  std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x32d9e50, __lock=@0x7f39ba7fbd00: {_M_device = 0x32ca740, _M_owns = true}, __atime=<optimized out>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:111
 
#5  std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (this=0x32d9e50, __lock=@0x7f39ba7fbd00: {_M_device = 0x32ca740, _M_owns = true}, __rtime=@0x32ca768: {__r = 60000}) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:163
 
#6  tpool::thread_pool_generic::wait_for_tasks (this=this@entry=0x32ca630, lk=@0x7f39ba7fbd00: {_M_device = 0x32ca740, _M_owns = true}, thread_data=thread_data@entry=0x32d9e50) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:449
 
#7  0x0000000000d6b5ce in tpool::thread_pool_generic::get_task (this=this@entry=0x32ca630, thread_var=thread_var@entry=0x32d9e50, t=t@entry=0x7f39ba7fbd50) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:502
 
#8  0x0000000000d6b7d4 in tpool::thread_pool_generic::worker_main (this=0x32ca630, thread_var=0x32d9e50) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:547
 
#9  0x00007f39ffa6d5c4 in execute_native_thread_routine () from /lib64/libstdc++.so.6
 
#10 0x00007f39ff7fcb1a in start_thread () from /lib64/libc.so.6
 
#11 0x00007f39ff881650 in clone3 () from /lib64/libc.so.6
 
 
 
Thread 9 (Thread 0x7f39db9ff640 (LWP 1063548)):
 
#0  0x00007f39ff8772fb in fdatasync () from /lib64/libc.so.6
 
#1  0x0000000000c09445 in os_file_sync_posix (file=12) at /home/dan/repos/mariadb-server-10.8/storage/innobase/os/os0file.cc:888
 
#2  os_file_flush_func (file=file@entry=12) at /home/dan/repos/mariadb-server-10.8/storage/innobase/os/os0file.cc:974
 
#3  0x0000000000d1bb98 in pfs_os_file_flush_func (file={m_file = 12, m_psi = <synthetic pointer>}, src_line=514, src_file=<optimized out>) at /home/dan/repos/mariadb-server-10.8/storage/innobase/include/os0file.inl:347
 
#4  fil_space_t::flush_low (this=this@entry=0x33c2578) at /home/dan/repos/mariadb-server-10.8/storage/innobase/fil/fil0fil.cc:514
 
#5  0x0000000000d1e326 in fil_flush_file_spaces () at /home/dan/repos/mariadb-server-10.8/storage/innobase/fil/fil0fil.cc:2904
 
#6  0x0000000000ce8148 in buf_flush_sync_for_checkpoint (lsn=13857) at /home/dan/repos/mariadb-server-10.8/storage/innobase/buf/buf0flu.cc:1956
 
#7  buf_flush_page_cleaner () at /home/dan/repos/mariadb-server-10.8/storage/innobase/buf/buf0flu.cc:2206
 
#8  0x00007f39ffa6d5c4 in execute_native_thread_routine () from /lib64/libstdc++.so.6
 
#9  0x00007f39ff7fcb1a in start_thread () from /lib64/libc.so.6
 
#10 0x00007f39ff881650 in clone3 () from /lib64/libc.so.6
 
 
 
Thread 8 (Thread 0x7f39ff4ff280 (LWP 1063539)):
 
#0  0x00007f39ff7f973a in __futex_abstimed_wait_common () from /lib64/libc.so.6
 
#1  0x00007f39ff7fbeb0 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libc.so.6
 
#2  0x0000000000ce941f in buf_flush_wait (lsn=13857) at /home/dan/repos/mariadb-server-10.8/storage/innobase/buf/buf0flu.cc:1833
 
#3  buf_flush_sync () at /home/dan/repos/mariadb-server-10.8/storage/innobase/buf/buf0flu.cc:2491
 
#4  0x0000000000c8326c in srv_start (create_new_db=<optimized out>) at /home/dan/repos/mariadb-server-10.8/storage/innobase/srv/srv0start.cc:1172
 
#5  0x0000000000bb00c9 in innodb_init (p=<optimized out>) at /home/dan/repos/mariadb-server-10.8/storage/innobase/handler/ha_innodb.cc:4203
 
#6  0x000000000093f198 in ha_initialize_handlerton (plugin=0x32105e0) at /home/dan/repos/mariadb-server-10.8/sql/handler.cc:649
 
#7  0x000000000073ccd2 in plugin_initialize (tmp_root=<optimized out>, plugin=plugin@entry=0x32105e0, argc=argc@entry=0x166e550 <remaining_argc>, argv=argv@entry=0x31d27e8, options_only=false) at /home/dan/repos/mariadb-server-10.8/sql/sql_plugin.cc:1462
 
#8  0x000000000073c541 in plugin_init (argc=0x166e550 <remaining_argc>, argv=0x31d27e8, flags=1) at /home/dan/repos/mariadb-server-10.8/sql/sql_plugin.cc:1755
 
#9  0x000000000063ca2c in init_server_components () at /home/dan/repos/mariadb-server-10.8/sql/mysqld.cc:5088
 
#10 0x00000000006397ba in mysqld_main (argc=<optimized out>, argv=<optimized out>) at /home/dan/repos/mariadb-server-10.8/sql/mysqld.cc:5704
 
#11 0x00007f39ff79c560 in __libc_start_call_main () from /lib64/libc.so.6
 
#12 0x00007f39ff79c60c in __libc_start_main_impl () from /lib64/libc.so.6
 
#13 0x00000000006369f5 in _start ()
 
 
 
Thread 7 (Thread 0x7f39d91fa640 (LWP 1063567)):
 
#0  0x00007f39ff7f973a in __futex_abstimed_wait_common () from /lib64/libc.so.6
 
#1  0x00007f39ff7fc482 in pthread_cond_clockwait@GLIBC_2.30 () from /lib64/libc.so.6
 
#2  0x0000000000d6b43c in std::__condvar::wait_until (this=0x32da0d0, __m=<error reading variable: Cannot access memory at address 0x89>, __clock=1, __abs_time=@0x7f39d91f9cb8: {tv_sec = 179179, tv_nsec = 584905003}) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/std_mutex.h:169
 
#3  std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x32da0d0, __lock=@0x7f39d91f9d00: {_M_device = 0x32ca740, _M_owns = true}, __atime=<optimized out>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:201
 
#4  std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x32da0d0, __lock=@0x7f39d91f9d00: {_M_device = 0x32ca740, _M_owns = true}, __atime=<optimized out>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:111
 
#5  std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (this=0x32da0d0, __lock=@0x7f39d91f9d00: {_M_device = 0x32ca740, _M_owns = true}, __rtime=@0x32ca768: {__r = 60000}) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:163
 
#6  tpool::thread_pool_generic::wait_for_tasks (this=this@entry=0x32ca630, lk=@0x7f39d91f9d00: {_M_device = 0x32ca740, _M_owns = true}, thread_data=thread_data@entry=0x32da0d0) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:449
 
#7  0x0000000000d6b5ce in tpool::thread_pool_generic::get_task (this=this@entry=0x32ca630, thread_var=thread_var@entry=0x32da0d0, t=t@entry=0x7f39d91f9d50) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:502
 
#8  0x0000000000d6b7a8 in tpool::thread_pool_generic::worker_main (this=0x32ca630, thread_var=0x32da0d0) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:547
 
#9  0x00007f39ffa6d5c4 in execute_native_thread_routine () from /lib64/libstdc++.so.6
 
#10 0x00007f39ff7fcb1a in start_thread () from /lib64/libc.so.6
 
#11 0x00007f39ff881650 in clone3 () from /lib64/libc.so.6
 
 
 
Thread 6 (Thread 0x7f39bb7fe640 (LWP 1063570)):
 
#0  0x00007f39ff7f973a in __futex_abstimed_wait_common () from /lib64/libc.so.6
 
#1  0x00007f39ff7fc482 in pthread_cond_clockwait@GLIBC_2.30 () from /lib64/libc.so.6
 
#2  0x0000000000d6b43c in std::__condvar::wait_until (this=0x32d9f50, __m=<error reading variable: Cannot access memory at address 0x89>, __clock=1, __abs_time=@0x7f39bb7fdcb8: {tv_sec = 179179, tv_nsec = 584944909}) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/std_mutex.h:169
 
#3  std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x32d9f50, __lock=@0x7f39bb7fdd00: {_M_device = 0x32ca740, _M_owns = true}, __atime=<optimized out>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:201
 
#4  std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x32d9f50, __lock=@0x7f39bb7fdd00: {_M_device = 0x32ca740, _M_owns = true}, __atime=<optimized out>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:111
 
#5  std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (this=0x32d9f50, __lock=@0x7f39bb7fdd00: {_M_device = 0x32ca740, _M_owns = true}, __rtime=@0x32ca768: {__r = 60000}) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:163
 
#6  tpool::thread_pool_generic::wait_for_tasks (this=this@entry=0x32ca630, lk=@0x7f39bb7fdd00: {_M_device = 0x32ca740, _M_owns = true}, thread_data=thread_data@entry=0x32d9f50) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:449
 
#7  0x0000000000d6b5ce in tpool::thread_pool_generic::get_task (this=this@entry=0x32ca630, thread_var=thread_var@entry=0x32d9f50, t=t@entry=0x7f39bb7fdd50) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:502
 
#8  0x0000000000d6b7a8 in tpool::thread_pool_generic::worker_main (this=0x32ca630, thread_var=0x32d9f50) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:547
 
#9  0x00007f39ffa6d5c4 in execute_native_thread_routine () from /lib64/libstdc++.so.6
 
#10 0x00007f39ff7fcb1a in start_thread () from /lib64/libc.so.6
 
#11 0x00007f39ff881650 in clone3 () from /lib64/libc.so.6
 
 
 
Thread 5 (Thread 0x7f39bbfff640 (LWP 1063569)):
 
#0  0x00007f39ff7f973a in __futex_abstimed_wait_common () from /lib64/libc.so.6
 
#1  0x00007f39ff7fc482 in pthread_cond_clockwait@GLIBC_2.30 () from /lib64/libc.so.6
 
#2  0x0000000000d6b43c in std::__condvar::wait_until (this=0x32d9fd0, __m=<error reading variable: Cannot access memory at address 0x89>, __clock=1, __abs_time=@0x7f39bbffecb8: {tv_sec = 179179, tv_nsec = 584956879}) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/std_mutex.h:169
 
#3  std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x32d9fd0, __lock=@0x7f39bbffed00: {_M_device = 0x32ca740, _M_owns = true}, __atime=<optimized out>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:201
 
#4  std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x32d9fd0, __lock=@0x7f39bbffed00: {_M_device = 0x32ca740, _M_owns = true}, __atime=<optimized out>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:111
 
#5  std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (this=0x32d9fd0, __lock=@0x7f39bbffed00: {_M_device = 0x32ca740, _M_owns = true}, __rtime=@0x32ca768: {__r = 60000}) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:163
 
#6  tpool::thread_pool_generic::wait_for_tasks (this=this@entry=0x32ca630, lk=@0x7f39bbffed00: {_M_device = 0x32ca740, _M_owns = true}, thread_data=thread_data@entry=0x32d9fd0) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:449
 
#7  0x0000000000d6b5ce in tpool::thread_pool_generic::get_task (this=this@entry=0x32ca630, thread_var=thread_var@entry=0x32d9fd0, t=t@entry=0x7f39bbffed50) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:502
 
#8  0x0000000000d6b7a8 in tpool::thread_pool_generic::worker_main (this=0x32ca630, thread_var=0x32d9fd0) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:547
 
#9  0x00007f39ffa6d5c4 in execute_native_thread_routine () from /lib64/libstdc++.so.6
 
#10 0x00007f39ff7fcb1a in start_thread () from /lib64/libc.so.6
 
#11 0x00007f39ff881650 in clone3 () from /lib64/libc.so.6
 
 
 
Thread 4 (Thread 0x7f39da9fd640 (LWP 1063554)):
 
#0  0x00007f39ff7f973a in __futex_abstimed_wait_common () from /lib64/libc.so.6
 
#1  0x00007f39ff7fc482 in pthread_cond_clockwait@GLIBC_2.30 () from /lib64/libc.so.6
 
#2  0x0000000000d6b43c in std::__condvar::wait_until (this=0x32da250, __m=<error reading variable: Cannot access memory at address 0x89>, __clock=1, __abs_time=@0x7f39da9fccb8: {tv_sec = 179179, tv_nsec = 584888915}) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/std_mutex.h:169
 
#3  std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x32da250, __lock=@0x7f39da9fcd00: {_M_device = 0x32ca740, _M_owns = true}, __atime=<optimized out>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:201
 
#4  std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x32da250, __lock=@0x7f39da9fcd00: {_M_device = 0x32ca740, _M_owns = true}, __atime=<optimized out>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:111
 
#5  std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (this=0x32da250, __lock=@0x7f39da9fcd00: {_M_device = 0x32ca740, _M_owns = true}, __rtime=@0x32ca768: {__r = 60000}) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:163
 
#6  tpool::thread_pool_generic::wait_for_tasks (this=this@entry=0x32ca630, lk=@0x7f39da9fcd00: {_M_device = 0x32ca740, _M_owns = true}, thread_data=thread_data@entry=0x32da250) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:449
 
#7  0x0000000000d6b5ce in tpool::thread_pool_generic::get_task (this=this@entry=0x32ca630, thread_var=thread_var@entry=0x32da250, t=t@entry=0x7f39da9fcd50) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:502
 
#8  0x0000000000d6b7d4 in tpool::thread_pool_generic::worker_main (this=0x32ca630, thread_var=0x32da250) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:547
 
#9  0x00007f39ffa6d5c4 in execute_native_thread_routine () from /lib64/libstdc++.so.6
 
#10 0x00007f39ff7fcb1a in start_thread () from /lib64/libc.so.6
 
#11 0x00007f39ff881650 in clone3 () from /lib64/libc.so.6
 
 
 
Thread 3 (Thread 0x7f39d99fb640 (LWP 1063566)):
 
#0  0x00007f39ff7f973a in __futex_abstimed_wait_common () from /lib64/libc.so.6
 
#1  0x00007f39ff7fc482 in pthread_cond_clockwait@GLIBC_2.30 () from /lib64/libc.so.6
 
#2  0x0000000000d6b43c in std::__condvar::wait_until (this=0x32da150, __m=<error reading variable: Cannot access memory at address 0x89>, __clock=1, __abs_time=@0x7f39d99facb8: {tv_sec = 179179, tv_nsec = 584892851}) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/std_mutex.h:169
 
#3  std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x32da150, __lock=@0x7f39d99fad00: {_M_device = 0x32ca740, _M_owns = true}, __atime=<optimized out>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:201
 
#4  std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x32da150, __lock=@0x7f39d99fad00: {_M_device = 0x32ca740, _M_owns = true}, __atime=<optimized out>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:111
 
#5  std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (this=0x32da150, __lock=@0x7f39d99fad00: {_M_device = 0x32ca740, _M_owns = true}, __rtime=@0x32ca768: {__r = 60000}) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:163
 
#6  tpool::thread_pool_generic::wait_for_tasks (this=this@entry=0x32ca630, lk=@0x7f39d99fad00: {_M_device = 0x32ca740, _M_owns = true}, thread_data=thread_data@entry=0x32da150) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:449
 
#7  0x0000000000d6b5ce in tpool::thread_pool_generic::get_task (this=this@entry=0x32ca630, thread_var=thread_var@entry=0x32da150, t=t@entry=0x7f39d99fad50) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:502
 
#8  0x0000000000d6b7d4 in tpool::thread_pool_generic::worker_main (this=0x32ca630, thread_var=0x32da150) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:547
 
#9  0x00007f39ffa6d5c4 in execute_native_thread_routine () from /lib64/libstdc++.so.6
 
#10 0x00007f39ff7fcb1a in start_thread () from /lib64/libc.so.6
 
#11 0x00007f39ff881650 in clone3 () from /lib64/libc.so.6
 
 
 
Thread 2 (Thread 0x7f39e9d1a640 (LWP 1063547)):
 
#0  0x00007f39ff879ecd in syscall () from /lib64/libc.so.6
 
#1  0x00007f3a001094f9 in _io_uring_get_cqe () from /lib64/liburing.so.2
 
#2  0x00007f3a00109a9b in __io_uring_get_cqe () from /lib64/liburing.so.2
 
#3  0x0000000000d6e072 in io_uring_wait_cqe_nr (ring=0x335d9c8, cqe_ptr=0x7f39e9d19d48, wait_nr=1) at /usr/include/liburing.h:635
 
#4  io_uring_wait_cqe (ring=0x335d9c8, cqe_ptr=0x7f39e9d19d48) at /usr/include/liburing.h:655
 
#5  (anonymous namespace)::aio_uring::thread_routine (aio=0x335d9c0) at /home/dan/repos/mariadb-server-10.8/tpool/aio_liburing.cc:137
 
#6  0x00007f39ffa6d5c4 in execute_native_thread_routine () from /lib64/libstdc++.so.6
 
#7  0x00007f39ff7fcb1a in start_thread () from /lib64/libc.so.6
 
#8  0x00007f39ff881650 in clone3 () from /lib64/libc.so.6
 
 
 
Thread 1 (Thread 0x7f39db1fe640 (LWP 1063553)):
 
#0  0x00007f39ff7fe88c in __pthread_kill_implementation () from /lib64/libc.so.6
 
#1  0x00007f39ff7b16a6 in raise () from /lib64/libc.so.6
 
#2  0x00007f39ff79b865 in abort () from /lib64/libc.so.6
 
#3  0x0000000000ca6e9b in ut_dbg_assertion_failed (expr=0x1160de8 "cb->m_err == DB_SUCCESS", file=<optimized out>, line=<optimized out>, line@entry=3571) at /home/dan/repos/mariadb-server-10.8/storage/innobase/ut/ut0dbg.cc:60
 
#4  0x0000000000c0c2d0 in io_callback (cb=<optimized out>) at /home/dan/repos/mariadb-server-10.8/storage/innobase/os/os0file.cc:3571
 
#5  0x0000000000d6d6dd in tpool::task_group::execute (this=0x33358b0, t=0x335b450) at /home/dan/repos/mariadb-server-10.8/tpool/task_group.cc:55
 
#6  0x0000000000d6b7c6 in tpool::thread_pool_generic::worker_main (this=0x32ca630, thread_var=0x32da2d0) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:549
 
#7  0x00007f39ffa6d5c4 in execute_native_thread_routine () from /lib64/libstdc++.so.6
 
#8  0x00007f39ff7fcb1a in start_thread () from /lib64/libc.so.6
 
#9  0x00007f39ff881650 in clone3 () from /lib64/libc.so.6
 
(gdb) up
 
#2  0x00007f39ff79b865 in abort () from /lib64/libc.so.6
 
(gdb) 
#3  0x0000000000ca6e9b in ut_dbg_assertion_failed (expr=0x1160de8 "cb->m_err == DB_SUCCESS", file=<optimized out>, line=<optimized out>, line@entry=3571) at /home/dan/repos/mariadb-server-10.8/storage/innobase/ut/ut0dbg.cc:60
 
60		abort();
 
(gdb) 
#4  0x0000000000c0c2d0 in io_callback (cb=<optimized out>) at /home/dan/repos/mariadb-server-10.8/storage/innobase/os/os0file.cc:3571
 
3571	  ut_a(cb->m_err == DB_SUCCESS);
 
(gdb) 
#5  0x0000000000d6d6dd in tpool::task_group::execute (this=0x33358b0, t=0x335b450) at /home/dan/repos/mariadb-server-10.8/tpool/task_group.cc:55
 
55	        t->m_func(t->m_arg);
 
(gdb)   p *((tpool::aiocb *) t->m_arg)
 
$1 = {<iovec> = {iov_base = 0x7f39fc5fc000, iov_len = 16384}, m_fh = 12, m_opcode = tpool::aio_opcode::AIO_PWRITE, m_offset = 180224, m_buffer = 0x7f39fc5fc000, m_len = 16384, m_callback = 0xc0c240 <io_callback(tpool::aiocb*)>, m_group = 0x33358b0, m_ret_len = 0, m_err = 4, m_internal = 0x0, m_internal_task = {_vptr$task = 0x1182828 <vtable for tpool::task+16>, m_func = 0xc0c240 <io_callback(tpool::aiocb*)>, m_arg = 0x335b3f8, m_group = 0x33358b0}, m_userdata = "\340\266\\\374\071\177\000\000\000\000\000\000\000\000\000\000\330&<\003\000\000\000\000\021\000\000\000\071\177\000"}

Generated at Thu Feb 08 09:49:20 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.