Daniel Black added a comment - 2021-11-04 01:25 - edited Per discussion with wlad / marko "Failing assertion: cb->m_err == DB_SUCCESS" (in static void io_callback / storage/innobase/os/os0file.cc) should be changed to actually have the error so the art of debugging this is significantly easier. Per command output.txt: 2021-10-29 11:47:02 0 [Warning] Can't create test file /var/lib/mysql/53a5df707bc7.lower-test /usr/sbin/mariadbd: Can't change dir to '/var/lib/mysql/' (Errcode: 13 "Permission denied") This needs to be resolved first, and in most likelihood, this will be the underlying cause. innodb_use_native_aio=0 was a workaround after the permissions were resolved, and on the assumption that this was a Virtualbox issue with libaio (which could be SMB as well). Evidently root@53a5df707bc7:/# chown -R mysql: /var/lib/mysql wasn't sufficient to make this ownership effective. Examine this with ls -la /var/lib/mysql . Try with --user mysql on compose option. uid mapping between the docker container and the underlying storage make it not obvious how this is done. What are the SMB and VirtualBox mount options inside the VM? What storage driver is Docker using? Which Docker version?

Kevin Wilson added a comment - 2021-11-05 10:44 - edited For clarity - I've now been able to reproduce the same problem on a dedicated ASRock NUC running Ubuntu natively (i.e without Virtualbox). The drives are mounted in Ubuntu with the following lines in the /etc/fstab file: //192.168.1.99/Media /mnt/m cifs credentials=/home/user/.smbcredentials,iocharset=utf8,dir_mode=0777,file_mode=0777,vers=3.0 //192.168.1.99/Users /mnt/u cifs credentials=/home/user/.smbcredentials,iocharset=utf8,dir_mode=0777,file_mode=0777,vers=3.0 The dir_mode=0777 and file_mode=0777 are new additions, and this has changed the errors slightly to this: mariadb | 2021 - 11 - 05 10 : 28 : 23 + 00 : 00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1 : 10.6 . 4 +maria~focal star mariadb | 2021 - 11 - 05 10 : 28 : 23 + 00 : 00 [Note] [Entrypoint]: Switching to dedicated user 'mysql' mariadb | 2021 - 11 - 05 10 : 28 : 23 + 00 : 00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1 : 10.6 . 4 +maria~focal star mariadb | 2021 - 11 - 05 10 : 28 : 24 + 00 : 00 [Note] [Entrypoint]: Initializing database files mariadb | Cannot change ownership of the database directories to the 'mysql' mariadb | user. Check that you have the necessary permissions and try again. mariadb | chown: changing ownership of '/var/lib/mysql/' : Operation not permitted mariadb exited with code 1 I'm not able to do 'ls -la /var/lib/mysql', as when I try to connect to the docker container I get: "Error response from daemon: Container cf67035f2cc2ae00559976ec2c277b6b83954d5ecfdd760448e2fdea9e2881f9 is restarting, wait until the container is running" The storage driver is as follows: Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Native Overlay Diff: true userxattr: false Docker version is 20.10.10

Daniel Black added a comment - 2022-01-24 05:02 - edited $ $ sudo mount -t cifs -o credentials=/home/dan/.nascredentials,iocharset=utf8,dir_mode=0777,file_mode=0777,vers=3.0 //192.168.178.171/dan /mnt/nas   $ mkdir /mnt/nas/datadir108   ~/repos/build-mariadb-server-10.8 $ scripts/mysql_install_db --no-defaults --srcdir=$OLDPWD --builddir=$PWD --datadir=/mnt/nas/datadir108 Installing MariaDB/MySQL system tables in '/mnt/nas/datadir108' ... 2022-01-24 15:15:10 0x7f39e63fd640 InnoDB: Assertion failure in file /home/dan/repos/mariadb-server-10.8/storage/innobase/os/os0file.cc line 3596 InnoDB: Failing assertion: cb->m_err == DB_SUCCESS 2022-01-24 15:15:10 0x7f39e53fb640 InnoDB: Assertion failure in file /home/dan/repos/mariadb-server-10.8/storage/innobase/os/os0file.cc line 3596 InnoDB: Failing assertion: cb->m_err == DB_SUCCESS InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to https://jira.mariadb.org/ InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mariadbd startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/ InnoDB: about forcing recovery. InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to https://jira.mariadb.org/ InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mariadbd startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/ InnoDB: about forcing recovery. 220124 15:15:10 [ERROR] mysqld got signal 6 ; scripts/mysql_install_db: line 537: 43865 Aborted (core dumped) "$mysqld_bootstrap" $defaults $defaults_group_suffix "$mysqld_opt" --bootstrap $silent_startup "--basedir=$basedir" "--datadir=$ldata" --log-warnings=0 --enforce-storage-engine="" "--plugin-dir=${plugindir}" $args --max_allowed_packet=8M --net_buffer_length=16K   #4 0x0000000000c037d0 in io_callback (cb=<optimized out>) at /home/dan/repos/mariadb-server-10.8/storage/innobase/os/os0file.cc:3596 #5 0x0000000000d6588d in tpool::task_group::execute (this=0x2fd8980, t=0x2ffe9e0) at /home/dan/repos/mariadb-server-10.8/tpool/task_group.cc:55 #6 0x0000000000d63976 in tpool::thread_pool_generic::worker_main (this=0x2f6d700, thread_var=0x2f7d320) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:549 #7 0x00007f3a0b6985c4 in execute_native_thread_routine () from /lib64/libstdc++.so.6 #8 0x00007f3a0b427a87 in start_thread () from /lib64/libc.so.6 #9 0x00007f3a0b4ac640 in clone3 () from /lib64/libc.so.6 (gdb) up #1 0x00007f3a0b3dc6a6 in raise () from /lib64/libc.so.6 (gdb) up #2 0x00007f3a0b3c67d3 in abort () from /lib64/libc.so.6 (gdb) up #3 0x0000000000c9ecfb in ut_dbg_assertion_failed (expr=0x1152a4f "cb->m_err == DB_SUCCESS", file=<optimized out>, line=<optimized out>, line@entry=3596) at /home/dan/repos/mariadb-server-10.8/storage/innobase/ut/ut0dbg.cc:60 60 abort(); (gdb) p cb No symbol "cb" in current context. (gdb) up #4 0x0000000000c037d0 in io_callback (cb=<optimized out>) at /home/dan/repos/mariadb-server-10.8/storage/innobase/os/os0file.cc:3596 3596 ut_a(cb->m_err == DB_SUCCESS); (gdb) p cb $1 = <optimized out> (gdb) up #5 0x0000000000d6588d in tpool::task_group::execute (this=0x2fd8980, t=0x2ffe9e0) at /home/dan/repos/mariadb-server-10.8/tpool/task_group.cc:55 55 t->m_func(t->m_arg); (gdb) p *((tpool::aiocb *) t->m_arg) $2 = {<iovec> = {iov_base = 0x7f3a081e4000, iov_len = 16384}, m_fh = 12, m_opcode = tpool::aio_opcode::AIO_PWRITE, m_offset = 65536, m_buffer = 0x7f3a081e4000, m_len = 16384, m_callback = 0xc03740 <io_callback(tpool::aiocb*)>, m_group = 0x2fd8980, m_ret_len = 0, m_err = 22, m_internal = 0x0, m_internal_task = {_vptr$task = 0x1174848 <vtable for tpool::task+16>, m_func = 0xc03740 <io_callback(tpool::aiocb*)>, m_arg = 0x2ffe988, m_group = 0x2fd8980}, m_userdata = "\200\342\034\b:\177\000\000\000\000\000\000\000\000\000\000H]\006\003\000\000\000\000\021\000\000\000\000\000\000"} Thread 5 (Thread 0x7f3a0b102280 (LWP 43865)): #0 0x00007f3a0b42472a in __futex_abstimed_wait_common () from /lib64/libc.so.6 #1 0x00007f3a0b426ed9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libc.so.6 #2 0x0000000000ce1c68 in buf_flush_wait (lsn=10353) at /home/dan/repos/mariadb-server-10.8/storage/innobase/buf/buf0flu.cc:1789 #3 buf_flush_sync () at /home/dan/repos/mariadb-server-10.8/storage/innobase/buf/buf0flu.cc:2450 #4 0x0000000000c7a3be in srv_start (create_new_db=<optimized out>) at /home/dan/repos/mariadb-server-10.8/storage/innobase/srv/srv0start.cc:1417 #5 0x0000000000ba51e7 in innodb_init (p=<optimized out>) at /home/dan/repos/mariadb-server-10.8/storage/innobase/handler/ha_innodb.cc:4216 #6 0x0000000000935bd8 in ha_initialize_handlerton (plugin=0x2eb35d0) at /home/dan/repos/mariadb-server-10.8/sql/handler.cc:649 #7 0x0000000000736982 in plugin_initialize (tmp_root=<optimized out>, plugin=plugin@entry=0x2eb35d0, argc=argc@entry=0x165d600 <remaining_argc>, argv=argv@entry=0x2e757e8, options_only=false) at /home/dan/repos/mariadb-server-10.8/sql/sql_plugin.cc:1462 #8 0x00000000007361f1 in plugin_init (argc=0x165d600 <remaining_argc>, argv=0x2e757e8, flags=1) at /home/dan/repos/mariadb-server-10.8/sql/sql_plugin.cc:1755 #9 0x000000000063432c in init_server_components () at /home/dan/repos/mariadb-server-10.8/sql/mysqld.cc:5083 #10 0x00000000006310ba in mysqld_main (argc=<optimized out>, argv=<optimized out>) at /home/dan/repos/mariadb-server-10.8/sql/mysqld.cc:5698 #11 0x00007f3a0b3c7560 in __libc_start_call_main () from /lib64/libc.so.6 #12 0x00007f3a0b3c760c in __libc_start_main_impl () from /lib64/libc.so.6 #13 0x000000000062e2f5 in _start ()   (gdb) p log_sys $11 = {lsn = {<std::__atomic_base<unsigned long>> = {_M_i = 10353}, <No data fields>}, flushed_to_disk_lsn = {<std::__atomic_base<unsigned long>> = {_M_i = 10353}, <No data fields>}, check_flush_or_checkpoint_ = {_M_base = {_M_i = true}}, mutex = {m_mutex = {__data = {__lock = 1, __count = 0, __owner = 43869, __nusers = 1, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = "\001\000\000\000\000\000\000\000]\253\000\000\001", '\000' <repeats 26 times>, __align = 1}, m_psi = 0x0}, buf_free = 113, max_buf_free = 8321024, flush_order_mutex = {m_mutex = {__data = {__lock = 1, __count = 0, __owner = 43869, __nusers = 1, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = "\001\000\000\000\000\000\000\000]\253\000\000\001", '\000' <repeats 26 times>, __align = 1}, m_psi = 0x0}, buf = 0x7f39e8600000 "", flush_buf = 0x7f39e7600000 "", log = {format = 1346918739, subformat = 2, file_size = 100663296, lsn = 8764, lsn_offset = 2108, fd = {m_file = {_M_t = {<std::__uniq_ptr_impl<file_io, std::default_delete<file_io> >> = {_M_t = {<std::_Tuple_impl<0, file_io*, std::default_delete<file_io> >> = {<std::_Tuple_impl<1, std::default_delete<file_io> >> = {<std::_Head_base<1, std::default_delete<file_io>, true>> = {_M_head_impl = {<No data fields>}}, <No data fields>}, <std::_Head_base<0, file_io*, false>> = {_M_head_impl = 0x306c080}, <No data fields>}, <No data fields>}}, <No data fields>}}, m_path = {_M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0x1ee6478 <log_sys+248> "./ib_logfile101"}, _M_string_length = 15, {_M_local_buf = "./ib_logfile101", _M_allocated_capacity = 7453295065250213678}}}, scanned_lsn = 0}, buf_next_to_write = 113, write_lsn = 10353, current_flush_lsn = 0, pending_flushes = {<std::__atomic_base<unsigned long>> = {_M_i = 0}, <No data fields>}, flushes = {<std::__atomic_base<unsigned long>> = {_M_i = 10}, <No data fields>}, n_log_ios = 10, n_log_ios_old = 0, last_printout_time = 1642997706, log_capacity = 90595124, max_modified_age_async = 70724346, max_checkpoint_age = 80827823, next_checkpoint_no = 3, last_checkpoint_lsn = {m = {<std::__atomic_base<unsigned long>> = {_M_i = 8752}, <No data fields>}}, next_checkpoint_lsn = 8752, n_pending_checkpoint_writes = 0, checkpoint_buf = 0x3065600 "", m_initialised = true} (gdb) p log_sys.fd There is no member or method named fd. (gdb) p log_sys.log $12 = {format = 1346918739, subformat = 2, file_size = 100663296, lsn = 8764, lsn_offset = 2108, fd = {m_file = {_M_t = {<std::__uniq_ptr_impl<file_io, std::default_delete<file_io> >> = {_M_t = {<std::_Tuple_impl<0, file_io*, std::default_delete<file_io> >> = {<std::_Tuple_impl<1, std::default_delete<file_io> >> = {<std::_Head_base<1, std::default_delete<file_io>, true>> = {_M_head_impl = {<No data fields>}}, <No data fields>}, <std::_Head_base<0, file_io*, false>> = {_M_head_impl = 0x306c080}, <No data fields>}, <No data fields>}}, <No data fields>}}, m_path = {_M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0x1ee6478 <log_sys+248> "./ib_logfile101"}, _M_string_length = 15, {_M_local_buf = "./ib_logfile101", _M_allocated_capacity = 7453295065250213678}}}, scanned_lsn = 0} (gdb) p log_sys.fd There is no member or method named fd. (gdb) p log_sys.log.fd $13 = {m_file = {_M_t = {<std::__uniq_ptr_impl<file_io, std::default_delete<file_io> >> = {_M_t = {<std::_Tuple_impl<0, file_io*, std::default_delete<file_io> >> = {<std::_Tuple_impl<1, std::default_delete<file_io> >> = {<std::_Head_base<1, std::default_delete<file_io>, true>> = {_M_head_impl = {<No data fields>}}, <No data fields>}, <std::_Head_base<0, file_io*, false>> = {_M_head_impl = 0x306c080}, <No data fields>}, <No data fields>}}, <No data fields>}}, m_path = {_M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0x1ee6478 <log_sys+248> "./ib_logfile101"}, _M_string_length = 15, {_M_local_buf = "./ib_logfile101", _M_allocated_capacity = 7453295065250213678}}} m_err = 22 -> EINVAL Repeatable on 10.5 with --innodb_flush_method=O_DIRECT, the 10.5 default of fsync starts correctly.

Daniel Black added a comment - 2022-01-24 05:16 (gdb) r Starting program: /home/dan/repos/build-mariadb-server-10.5/sql/mysqld --no-defaults --skip-networking --datadir=/mnt/nas/datadir --verbose --innodb_flush_method=O_DIRECT   Thread 1 "mysqld" hit Breakpoint 2, os_file_set_nocache (fd=<optimized out>, file_name=<optimized out>, operation_name=<optimized out>) at /home/dan/repos/mariadb-server-10.5/storage/innobase/os/os0file.cc:3228 warning: Source file is more recent than executable. 3228 if (fcntl(fd, F_SETFL, O_DIRECT) == -1) { (gdb) s   Thread 1 "mysqld" hit Breakpoint 1, 0x00007ffff75d5c60 in fcntl64 () from /lib64/libc.so.6 (gdb) finish Run till exit from #0 0x00007ffff75d5c60 in fcntl64 () from /lib64/libc.so.6 0x0000000001bf152b in os_file_set_nocache (fd=<optimized out>, file_name=<optimized out>, operation_name=<optimized out>) at /home/dan/repos/mariadb-server-10.5/storage/innobase/os/os0file.cc:3228 3228 if (fcntl(fd, F_SETFL, O_DIRECT) == -1) { (gdb) s 3259 } (gdb) p errno 'errno' has unknown type; cast it to its declared type (gdb) p (int) errno $1 = 0 So in short the setting of the file mode to O_DIRECT succeeds but fails when used. The 10.6 work around is to set --innodb_flush_method=flush on CIFS mounts. related (below), tmpfs will return EINVAL. btrfs is failing under some condition. https://jira.mariadb.org/browse/MDEV-24854?focusedCommentId=211665&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-211665

Daniel Black added a comment - 2022-01-25 09:43 I've emailed linuxfs-devel developers about potential kernel based solutions.

Daniel Black added a comment - 2022-02-07 04:26 linuxfs-devel: commitment to having fcntl(fd, F_SETFL, O_DIRECT) error if not available cifs does have direct IO support, however not clear under what conditions, MDEV-26995 cache=none might the the conditions if so we should have to check it somehow.

Daniel Black added a comment - 2022-02-15 07:08 cache=none as a mount option seems to be a pre-requisite for directio: https://lwn.net/Articles/756061/ With this, still observered failures: observed failures in mysql_install_db-10.8-3b06415cb8dfc2b10d9604950f6ed945bc016425 + mysqld_install_cmd_line + /home/dan/repos/build-mariadb-server-10.8/sql/mysqld --no-defaults --lc-messages-dir=/home/dan/repos/build-mariadb-server-10.8/sql/share/english/.. --bootstrap --silent-startup --basedir=/home/dan/repos/build-mariadb-server-10.8 --datadir=/mnt/nas/datadir108 --log-warnings=0 --enforce-storage-engine= --plugin-dir=/home/dan/repos/build-mariadb-server-10.8/plugin/auth_socket --loose-disable-plugin-file-key-management --max_allowed_packet=8M --net_buffer_length=16K + test 0 -eq 0 + cat /home/dan/repos/mariadb-server-10.8/scripts/mysql_test_db.sql + test '!' -z '' 2022-02-15 17:32:26 0x7fd36cbfc640 InnoDB: Assertion failure in file /home/dan/repos/mariadb-server-10.8/storage/innobase/os/os0file.cc line 3571 InnoDB: Failing assertion: cb->m_err == DB_SUCCESS InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to https://jira.mariadb.org/ InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mariadbd startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/ InnoDB: about forcing recovery. 220215 17:32:26 [ERROR] mysqld got signal 6 ;   #0 0x00007fd3918ca88c in __pthread_kill_implementation () from /lib64/libc.so.6 [Current thread is 1 (Thread 0x7fd33ffff640 (LWP 1060236))] (gdb) bt #0 0x00007fd3918ca88c in __pthread_kill_implementation () from /lib64/libc.so.6 #1 0x00007fd39187d6a6 in raise () from /lib64/libc.so.6 #2 0x00007fd3918677d3 in abort () from /lib64/libc.so.6 #3 0x0000000000ca6e9b in ut_dbg_assertion_failed (expr=0x1160de8 "cb->m_err == DB_SUCCESS", file=<optimized out>, line=<optimized out>, line@entry=3571) at /home/dan/repos/mariadb-server-10.8/storage/innobase/ut/ut0dbg.cc:60 #4 0x0000000000c0c2d0 in io_callback (cb=<optimized out>) at /home/dan/repos/mariadb-server-10.8/storage/innobase/os/os0file.cc:3571 #5 0x0000000000d6d6dd in tpool::task_group::execute (this=0x2a238b0, t=0x2a48c98) at /home/dan/repos/mariadb-server-10.8/tpool/task_group.cc:55 #6 0x0000000000d6b7c6 in tpool::thread_pool_generic::worker_main (this=0x29b8630, thread_var=0x29c7ed0) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:549 #7 0x00007fd391b395c4 in execute_native_thread_routine () from /lib64/libstdc++.so.6 #8 0x00007fd3918c8b1a in start_thread () from /lib64/libc.so.6 #9 0x00007fd39194d650 in clone3 () from /lib64/libc.so.6 (gdb) thread apply all bt   Thread 9 (Thread 0x7fd337fff640 (LWP 1060235)): #0 0x00007fd3918c573a in __futex_abstimed_wait_common () from /lib64/libc.so.6 #1 0x00007fd3918c8482 in pthread_cond_clockwait@GLIBC_2.30 () from /lib64/libc.so.6 #2 0x0000000000d6b43c in std::__condvar::wait_until (this=0x29c7e50, __m=<error reading variable: Cannot access memory at address 0x89>, __clock=1, __abs_time=@0x7fd337ffecb8: {tv_sec = 177671, tv_nsec = 568528669}) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/std_mutex.h:169 #3 std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x29c7e50, __lock=@0x7fd337ffed00: {_M_device = 0x29b8740, _M_owns = true}, __atime=<optimized out>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:201 #4 std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x29c7e50, __lock=@0x7fd337ffed00: {_M_device = 0x29b8740, _M_owns = true}, __atime=<optimized out>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:111 #5 std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (this=0x29c7e50, __lock=@0x7fd337ffed00: {_M_device = 0x29b8740, _M_owns = true}, __rtime=@0x29b8768: {__r = 60000}) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:163 #6 tpool::thread_pool_generic::wait_for_tasks (this=this@entry=0x29b8630, lk=@0x7fd337ffed00: {_M_device = 0x29b8740, _M_owns = true}, thread_data=thread_data@entry=0x29c7e50) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:449 #7 0x0000000000d6b5ce in tpool::thread_pool_generic::get_task (this=this@entry=0x29b8630, thread_var=thread_var@entry=0x29c7e50, t=t@entry=0x7fd337ffed50) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:502 #8 0x0000000000d6b7a8 in tpool::thread_pool_generic::worker_main (this=0x29b8630, thread_var=0x29c7e50) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:547 #9 0x00007fd391b395c4 in execute_native_thread_routine () from /lib64/libstdc++.so.6 #10 0x00007fd3918c8b1a in start_thread () from /lib64/libc.so.6 #11 0x00007fd39194d650 in clone3 () from /lib64/libc.so.6   Thread 8 (Thread 0x7fd36cbfc640 (LWP 1059793)): #0 0x00007fd39194079f in poll () from /lib64/libc.so.6 #1 0x0000000000de2de1 in addr_resolve (ptr=<optimized out>, loc=loc@entry=0x7fd36cbfa318) at /home/dan/repos/mariadb-server-10.8/mysys/my_addr_resolve.c:245 #2 0x0000000000de2cbc in my_addr_resolve (ptr=0xca6e9b, loc=loc@entry=0x7fd36cbfa318) at /home/dan/repos/mariadb-server-10.8/mysys/my_addr_resolve.c:330 #3 0x0000000000dcb11d in print_with_addr_resolve (addrs=0x7fd36cbf9f10, n=<optimized out>) at /home/dan/repos/mariadb-server-10.8/mysys/stacktrace.c:197 #4 my_print_stacktrace (stack_bottom=<optimized out>, thread_stack=<optimized out>, silent=<optimized out>) at /home/dan/repos/mariadb-server-10.8/mysys/stacktrace.c:216 #5 0x000000000093e3b4 in handle_fatal_signal (sig=6) at /home/dan/repos/mariadb-server-10.8/sql/signal_handler.cc:223 #6 <signal handler called> #7 0x00007fd3918ca88c in __pthread_kill_implementation () from /lib64/libc.so.6 #8 0x00007fd39187d6a6 in raise () from /lib64/libc.so.6 #9 0x00007fd3918677d3 in abort () from /lib64/libc.so.6 #10 0x0000000000ca6e9b in ut_dbg_assertion_failed (expr=0x1160de8 "cb->m_err == DB_SUCCESS", file=<optimized out>, line=<optimized out>, line@entry=3571) at /home/dan/repos/mariadb-server-10.8/storage/innobase/ut/ut0dbg.cc:60 #11 0x0000000000c0c2d0 in io_callback (cb=<optimized out>) at /home/dan/repos/mariadb-server-10.8/storage/innobase/os/os0file.cc:3571 #12 0x0000000000d6d6dd in tpool::task_group::execute (this=0x2a238b0, t=0x2a48dc8) at /home/dan/repos/mariadb-server-10.8/tpool/task_group.cc:55 #13 0x0000000000d6b7c6 in tpool::thread_pool_generic::worker_main (this=0x29b8630, thread_var=0x29c8250) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:549 #14 0x00007fd391b395c4 in execute_native_thread_routine () from /lib64/libstdc++.so.6 #15 0x00007fd3918c8b1a in start_thread () from /lib64/libc.so.6 #16 0x00007fd39194d650 in clone3 () from /lib64/libc.so.6   Thread 7 (Thread 0x7fd3915cb280 (LWP 1059616)): #0 0x00007fd391945ecd in syscall () from /lib64/libc.so.6 #1 0x00007fd3921d53b2 in __io_uring_submit () from /lib64/liburing.so.2 #2 0x0000000000d6e17b in (anonymous namespace)::aio_uring::submit_io (this=<optimized out>, cb=0x2a48fd0) at /home/dan/repos/mariadb-server-10.8/tpool/aio_liburing.cc:108 #3 0x0000000000c0c1a3 in tpool::thread_pool::submit_io (this=<optimized out>, cb=0x2a48fd0) at /home/dan/repos/mariadb-server-10.8/tpool/tpool.h:226 #4 os_aio (type=@0x7fff94f5ca48: {bpage = 0x7fd37c1a1dc0, slot = 0x0, node = 0x2ab06d8, type = IORequest::WRITE_ASYNC}, buf=0x7fd37c200000, offset=<optimized out>, n=16384) at /home/dan/repos/mariadb-server-10.8/storage/innobase/os/os0file.cc:3874 #5 0x0000000000d2218c in fil_space_t::io (this=0x2ab0578, type=@0x7fff94f5cad0: {bpage = 0x7fd37c1a1dc0, slot = 0x0, node = 0x0, type = IORequest::WRITE_ASYNC}, offset=0, len=1, buf=0x0, bpage=0x8) at /home/dan/repos/mariadb-server-10.8/storage/innobase/fil/fil0fil.cc:2810 #6 0x0000000000cea341 in buf_page_t::flush (this=this@entry=0x7fd37c1a1dc0, lru=<optimized out>, space=0x2ab0578) at /home/dan/repos/mariadb-server-10.8/storage/innobase/buf/buf0flu.cc:910 #7 0x0000000000ce77cd in buf_do_flush_list_batch (max_n=2000, lsn=18446744073709551615) at /home/dan/repos/mariadb-server-10.8/storage/innobase/buf/buf0flu.cc:1435 #8 buf_flush_list (max_n=<optimized out>, lsn=<optimized out>, lsn@entry=18446744073709551615) at /home/dan/repos/mariadb-server-10.8/storage/innobase/buf/buf0flu.cc:1510 #9 0x0000000000ce911d in buf_flush_buffer_pool () at /home/dan/repos/mariadb-server-10.8/storage/innobase/buf/buf0flu.cc:2440 #10 0x0000000000bf5418 in logs_empty_and_mark_files_at_shutdown () at /home/dan/repos/mariadb-server-10.8/storage/innobase/log/log0log.cc:902 #11 0x0000000000c8525b in innodb_shutdown () at /home/dan/repos/mariadb-server-10.8/storage/innobase/srv/srv0start.cc:1680 #12 0x0000000000bb3c78 in innobase_end () at /home/dan/repos/mariadb-server-10.8/storage/innobase/handler/ha_innodb.cc:4280 #13 0x000000000093f08e in ha_finalize_handlerton (plugin=0x28fe5e0) at /home/dan/repos/mariadb-server-10.8/sql/handler.cc:596 #14 0x000000000073cef2 in plugin_deinitialize (plugin=0x28fe5e0, ref_check=true) at /home/dan/repos/mariadb-server-10.8/sql/sql_plugin.cc:1270 #15 0x000000000073ba1a in reap_plugins () at /home/dan/repos/mariadb-server-10.8/sql/sql_plugin.cc:1341 #16 0x000000000073d272 in plugin_shutdown () at /home/dan/repos/mariadb-server-10.8/sql/sql_plugin.cc:2049 #17 0x00000000006378a5 in clean_up (print_message=false) at /home/dan/repos/mariadb-server-10.8/sql/mysqld.cc:1959 #18 0x0000000000637768 in unireg_abort (exit_code=0) at /home/dan/repos/mariadb-server-10.8/sql/mysqld.cc:1871 #19 0x00000000006399eb in mysqld_main (argc=<optimized out>, argv=<optimized out>) at /home/dan/repos/mariadb-server-10.8/sql/mysqld.cc:5793 #20 0x00007fd391868560 in __libc_start_call_main () from /lib64/libc.so.6 #21 0x00007fd39186860c in __libc_start_main_impl () from /lib64/libc.so.6 #22 0x00000000006369f5 in _start ()   Thread 6 (Thread 0x7fd36dbfe640 (LWP 1060215)): #0 0x00007fd3918c573a in __futex_abstimed_wait_common () from /lib64/libc.so.6 #1 0x00007fd3918c8482 in pthread_cond_clockwait@GLIBC_2.30 () from /lib64/libc.so.6 #2 0x0000000000d6b43c in std::__condvar::wait_until (this=0x29c7dd0, __m=<error reading variable: Cannot access memory at address 0x89>, __clock=1, __abs_time=@0x7fd36dbfdcb8: {tv_sec = 177671, tv_nsec = 568514021}) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/std_mutex.h:169 #3 std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x29c7dd0, __lock=@0x7fd36dbfdd00: {_M_device = 0x29b8740, _M_owns = true}, __atime=<optimized out>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:201 #4 std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x29c7dd0, __lock=@0x7fd36dbfdd00: {_M_device = 0x29b8740, _M_owns = true}, __atime=<optimized out>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:111 #5 std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (this=0x29c7dd0, __lock=@0x7fd36dbfdd00: {_M_device = 0x29b8740, _M_owns = true}, __rtime=@0x29b8768: {__r = 60000}) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:163 #6 tpool::thread_pool_generic::wait_for_tasks (this=this@entry=0x29b8630, lk=@0x7fd36dbfdd00: {_M_device = 0x29b8740, _M_owns = true}, thread_data=thread_data@entry=0x29c7dd0) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:449 #7 0x0000000000d6b5ce in tpool::thread_pool_generic::get_task (this=this@entry=0x29b8630, thread_var=thread_var@entry=0x29c7dd0, t=t@entry=0x7fd36dbfdd50) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:502 #8 0x0000000000d6b7d4 in tpool::thread_pool_generic::worker_main (this=0x29b8630, thread_var=0x29c7dd0) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:547 #9 0x00007fd391b395c4 in execute_native_thread_routine () from /lib64/libstdc++.so.6 #10 0x00007fd3918c8b1a in start_thread () from /lib64/libc.so.6 #11 0x00007fd39194d650 in clone3 () from /lib64/libc.so.6   Thread 5 (Thread 0x7fd377fff640 (LWP 1059619)): #0 0x00007fd391945ecd in syscall () from /lib64/libc.so.6 #1 0x00007fd3921d54f9 in _io_uring_get_cqe () from /lib64/liburing.so.2 #2 0x00007fd3921d5a9b in __io_uring_get_cqe () from /lib64/liburing.so.2 #3 0x0000000000d6e072 in io_uring_wait_cqe_nr (ring=0x2a4b9c8, cqe_ptr=0x7fd377ffed48, wait_nr=1) at /usr/include/liburing.h:635 #4 io_uring_wait_cqe (ring=0x2a4b9c8, cqe_ptr=0x7fd377ffed48) at /usr/include/liburing.h:655 #5 (anonymous namespace)::aio_uring::thread_routine (aio=0x2a4b9c0) at /home/dan/repos/mariadb-server-10.8/tpool/aio_liburing.cc:137 #6 0x00007fd391b395c4 in execute_native_thread_routine () from /lib64/libstdc++.so.6 #7 0x00007fd3918c8b1a in start_thread () from /lib64/libc.so.6 #8 0x00007fd39194d650 in clone3 () from /lib64/libc.so.6   Thread 4 (Thread 0x7fd37c1a0640 (LWP 1059657)): #0 0x00007fd39187e43a in sigtimedwait () from /lib64/libc.so.6 #1 0x0000000000638bcb in my_sigwait (set=0x7fd37c19fc20, sig=0x7fd37c19fd20, code=<optimized out>) at /home/dan/repos/mariadb-server-10.8/include/my_pthread.h:195 #2 signal_hand (arg=<optimized out>) at /home/dan/repos/mariadb-server-10.8/sql/mysqld.cc:3152 #3 0x0000000000b21a4e in pfs_spawn_thread (arg=0x2e21018) at /home/dan/repos/mariadb-server-10.8/storage/perfschema/pfs.cc:2201 #4 0x00007fd3918c8b1a in start_thread () from /lib64/libc.so.6 #5 0x00007fd39194d650 in clone3 () from /lib64/libc.so.6   Thread 3 (Thread 0x7fd390c25640 (LWP 1059618)): #0 0x00007fd3918c573a in __futex_abstimed_wait_common () from /lib64/libc.so.6 #1 0x00007fd3918c81b4 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libc.so.6 #2 0x0000000000ac73a3 in inline_mysql_cond_timedwait (that=0x1ea38b8 <COND_checkpoint+40>, mutex=0x189, abstime=0x7fd390c24d00, src_line=116, src_file=<optimized out>) at /home/dan/repos/mariadb-server-10.8/include/mysql/psi/mysql_thread.h:1088 #3 my_service_thread_sleep (control=0x15d4c70 <checkpoint_control>, sleep_time=30000000000) at /home/dan/repos/mariadb-server-10.8/storage/maria/ma_servicethread.c:115 #4 0x0000000000ac081f in ma_checkpoint_background (arg=<optimized out>) at /home/dan/repos/mariadb-server-10.8/storage/maria/ma_checkpoint.c:725 #5 0x00007fd3918c8b1a in start_thread () from /lib64/libc.so.6 #6 0x00007fd39194d650 in clone3 () from /lib64/libc.so.6   Thread 2 (Thread 0x7fd3922cb640 (LWP 1059617)): #0 0x00007fd3918c573a in __futex_abstimed_wait_common () from /lib64/libc.so.6 #1 0x00007fd3918c81b4 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libc.so.6 #2 0x0000000000dcec62 in inline_mysql_cond_timedwait (abstime=0x7fd3922cad50, src_line=321, that=<optimized out>, mutex=<optimized out>, src_file=<optimized out>) at /home/dan/repos/mariadb-server-10.8/include/mysql/psi/mysql_thread.h:1088 #3 timer_handler (arg=<optimized out>) at /home/dan/repos/mariadb-server-10.8/mysys/thr_timer.c:321 #4 0x00007fd3918c8b1a in start_thread () from /lib64/libc.so.6 #5 0x00007fd39194d650 in clone3 () from /lib64/libc.so.6   Thread 1 (Thread 0x7fd33ffff640 (LWP 1060236)): #0 0x00007fd3918ca88c in __pthread_kill_implementation () from /lib64/libc.so.6 #1 0x00007fd39187d6a6 in raise () from /lib64/libc.so.6 #2 0x00007fd3918677d3 in abort () from /lib64/libc.so.6 #3 0x0000000000ca6e9b in ut_dbg_assertion_failed (expr=0x1160de8 "cb->m_err == DB_SUCCESS", file=<optimized out>, line=<optimized out>, line@entry=3571) at /home/dan/repos/mariadb-server-10.8/storage/innobase/ut/ut0dbg.cc:60 #4 0x0000000000c0c2d0 in io_callback (cb=<optimized out>) at /home/dan/repos/mariadb-server-10.8/storage/innobase/os/os0file.cc:3571 #5 0x0000000000d6d6dd in tpool::task_group::execute (this=0x2a238b0, t=0x2a48c98) at /home/dan/repos/mariadb-server-10.8/tpool/task_group.cc:55 #6 0x0000000000d6b7c6 in tpool::thread_pool_generic::worker_main (this=0x29b8630, thread_var=0x29c7ed0) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:549 #7 0x00007fd391b395c4 in execute_native_thread_routine () from /lib64/libstdc++.so.6 #8 0x00007fd3918c8b1a in start_thread () from /lib64/libc.so.6 #9 0x00007fd39194d650 in clone3 () from /lib64/libc.so.6 (gdb) up #1 0x00007fd39187d6a6 in raise () from /lib64/libc.so.6 (gdb) #2 0x00007fd3918677d3 in abort () from /lib64/libc.so.6 (gdb) #3 0x0000000000ca6e9b in ut_dbg_assertion_failed (expr=0x1160de8 "cb->m_err == DB_SUCCESS", file=<optimized out>, line=<optimized out>, line@entry=3571) at /home/dan/repos/mariadb-server-10.8/storage/innobase/ut/ut0dbg.cc:60 60 abort(); (gdb) #4 0x0000000000c0c2d0 in io_callback (cb=<optimized out>) at /home/dan/repos/mariadb-server-10.8/storage/innobase/os/os0file.cc:3571 3571 ut_a(cb->m_err == DB_SUCCESS); (gdb) #5 0x0000000000d6d6dd in tpool::task_group::execute (this=0x2a238b0, t=0x2a48c98) at /home/dan/repos/mariadb-server-10.8/tpool/task_group.cc:55 55 t->m_func(t->m_arg); (gdb) p *((tpool::aiocb *) t->m_arg) $1 = {<iovec> = {iov_base = 0x7fd37c204000, iov_len = 16384}, m_fh = 12, m_opcode = tpool::aio_opcode::AIO_PWRITE, m_offset = 3145728, m_buffer = 0x7fd37c204000, m_len = 16384, m_callback = 0xc0c240 <io_callback(tpool::aiocb*)>, m_group = 0x2a238b0, m_ret_len = 0, m_err = 4, m_internal = 0x0, m_internal_task = {_vptr$task = 0x1182828 <vtable for tpool::task+16>, m_func = 0xc0c240 <io_callback(tpool::aiocb*)>, m_arg = 0x2a48c40, m_group = 0x2a238b0}, m_userdata = "`\036\032|\323\177\000\000\000\000\000\000\000\000\000\000\330\006\253\002\000\000\000\000\021\000\000\000\323\177\000"} m_err = 4 (EINTR). This maps the kernel errors: kernel errors [174428.702577] CIFS: Attempting to mount \\192.168.178.171\dan [174504.683290] CIFS: VFS: \\192.168.178.171 Error -512 sending data on socket to server [174504.815427] CIFS: VFS: cifs_setlk failed rc=-22 [175230.473589] CIFS: VFS: \\192.168.178.171 Error -512 sending data on socket to server [175230.991370] CIFS: VFS: \\192.168.178.171 Error -512 sending data on socket to server [175232.406643] CIFS: VFS: \\192.168.178.171 Error -512 sending data on socket to server [175232.868178] CIFS: VFS: \\192.168.178.171 Error -512 sending data on socket to server [175233.480149] CIFS: VFS: \\192.168.178.171 Error -512 sending data on socket to server [176260.874005] CIFS: VFS: \\192.168.178.171 Error -512 sending data on socket to server [176260.971757] CIFS: VFS: cifs_setlk failed rc=-22 [177282.983673] CIFS: VFS: \\192.168.178.171 Error -512 sending data on socket to server [177607.913858] CIFS: VFS: \\192.168.178.171 Error -512 sending data on socket to server [177608.525167] CIFS: VFS: \\192.168.178.171 Error -512 sending data on socket to server [178379.697057] CIFS: VFS: \\192.168.178.171 Error -512 sending data on socket to server Case 2 also on mysql_install_db: 10.8 fdatasync(thread 9, fd=12), interupts write Core was generated by `/home/dan/repos/build-mariadb-server-10.8/sql/mysqld --no-defaults --lc-message'. Program terminated with signal SIGABRT, Aborted. #0 0x00007f39ff7fe88c in __pthread_kill_implementation () from /lib64/libc.so.6 [Current thread is 1 (Thread 0x7f39db1fe640 (LWP 1063553))] (gdb) bt #0 0x00007f39ff7fe88c in __pthread_kill_implementation () from /lib64/libc.so.6 #1 0x00007f39ff7b16a6 in raise () from /lib64/libc.so.6 #2 0x00007f39ff79b865 in abort () from /lib64/libc.so.6 #3 0x0000000000ca6e9b in ut_dbg_assertion_failed (expr=0x1160de8 "cb->m_err == DB_SUCCESS", file=<optimized out>, line=<optimized out>, line@entry=3571) at /home/dan/repos/mariadb-server-10.8/storage/innobase/ut/ut0dbg.cc:60 #4 0x0000000000c0c2d0 in io_callback (cb=<optimized out>) at /home/dan/repos/mariadb-server-10.8/storage/innobase/os/os0file.cc:3571 #5 0x0000000000d6d6dd in tpool::task_group::execute (this=0x33358b0, t=0x335b450) at /home/dan/repos/mariadb-server-10.8/tpool/task_group.cc:55 #6 0x0000000000d6b7c6 in tpool::thread_pool_generic::worker_main (this=0x32ca630, thread_var=0x32da2d0) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:549 #7 0x00007f39ffa6d5c4 in execute_native_thread_routine () from /lib64/libstdc++.so.6 #8 0x00007f39ff7fcb1a in start_thread () from /lib64/libc.so.6 #9 0x00007f39ff881650 in clone3 () from /lib64/libc.so.6 (gdb) up #1 0x00007f39ff7b16a6 in raise () from /lib64/libc.so.6 (gdb) thread apply all bt   Thread 16 (Thread 0x7f3a001ff640 (LWP 1063541)): #0 0x00007f39ff7f973a in __futex_abstimed_wait_common () from /lib64/libc.so.6 #1 0x00007f39ff7fc1b4 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libc.so.6 #2 0x0000000000dcec62 in inline_mysql_cond_timedwait (abstime=0x7f3a001fed50, src_line=321, that=<optimized out>, mutex=<optimized out>, src_file=<optimized out>) at /home/dan/repos/mariadb-server-10.8/include/mysql/psi/mysql_thread.h:1088 #3 timer_handler (arg=<optimized out>) at /home/dan/repos/mariadb-server-10.8/mysys/thr_timer.c:321 #4 0x00007f39ff7fcb1a in start_thread () from /lib64/libc.so.6 #5 0x00007f39ff881650 in clone3 () from /lib64/libc.so.6   Thread 15 (Thread 0x7f39b9ffb640 (LWP 1063573)): #0 0x00007f39ff7f973a in __futex_abstimed_wait_common () from /lib64/libc.so.6 #1 0x00007f39ff7fc482 in pthread_cond_clockwait@GLIBC_2.30 () from /lib64/libc.so.6 #2 0x0000000000d6b43c in std::__condvar::wait_until (this=0x32d9dd0, __m=<error reading variable: Cannot access memory at address 0x89>, __clock=1, __abs_time=@0x7f39b9ffacb8: {tv_sec = 179179, tv_nsec = 584891184}) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/std_mutex.h:169 #3 std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x32d9dd0, __lock=@0x7f39b9ffad00: {_M_device = 0x32ca740, _M_owns = true}, __atime=<optimized out>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:201 #4 std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x32d9dd0, __lock=@0x7f39b9ffad00: {_M_device = 0x32ca740, _M_owns = true}, __atime=<optimized out>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:111 #5 std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (this=0x32d9dd0, __lock=@0x7f39b9ffad00: {_M_device = 0x32ca740, _M_owns = true}, __rtime=@0x32ca768: {__r = 60000}) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:163 #6 tpool::thread_pool_generic::wait_for_tasks (this=this@entry=0x32ca630, lk=@0x7f39b9ffad00: {_M_device = 0x32ca740, _M_owns = true}, thread_data=thread_data@entry=0x32d9dd0) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:449 #7 0x0000000000d6b5ce in tpool::thread_pool_generic::get_task (this=this@entry=0x32ca630, thread_var=thread_var@entry=0x32d9dd0, t=t@entry=0x7f39b9ffad50) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:502 #8 0x0000000000d6b7a8 in tpool::thread_pool_generic::worker_main (this=0x32ca630, thread_var=0x32d9dd0) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:547 #9 0x00007f39ffa6d5c4 in execute_native_thread_routine () from /lib64/libstdc++.so.6 #10 0x00007f39ff7fcb1a in start_thread () from /lib64/libc.so.6 #11 0x00007f39ff881650 in clone3 () from /lib64/libc.so.6   Thread 14 (Thread 0x7f39baffd640 (LWP 1063571)): #0 0x00007f39ff7f973a in __futex_abstimed_wait_common () from /lib64/libc.so.6 #1 0x00007f39ff7fc482 in pthread_cond_clockwait@GLIBC_2.30 () from /lib64/libc.so.6 #2 0x0000000000d6b43c in std::__condvar::wait_until (this=0x32d9ed0, __m=<error reading variable: Cannot access memory at address 0x89>, __clock=1, __abs_time=@0x7f39baffccb8: {tv_sec = 179179, tv_nsec = 584961467}) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/std_mutex.h:169 #3 std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x32d9ed0, __lock=@0x7f39baffcd00: {_M_device = 0x32ca740, _M_owns = true}, __atime=<optimized out>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:201 #4 std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x32d9ed0, __lock=@0x7f39baffcd00: {_M_device = 0x32ca740, _M_owns = true}, __atime=<optimized out>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:111 #5 std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (this=0x32d9ed0, __lock=@0x7f39baffcd00: {_M_device = 0x32ca740, _M_owns = true}, __rtime=@0x32ca768: {__r = 60000}) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:163 #6 tpool::thread_pool_generic::wait_for_tasks (this=this@entry=0x32ca630, lk=@0x7f39baffcd00: {_M_device = 0x32ca740, _M_owns = true}, thread_data=thread_data@entry=0x32d9ed0) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:449 #7 0x0000000000d6b5ce in tpool::thread_pool_generic::get_task (this=this@entry=0x32ca630, thread_var=thread_var@entry=0x32d9ed0, t=t@entry=0x7f39baffcd50) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:502 #8 0x0000000000d6b7a8 in tpool::thread_pool_generic::worker_main (this=0x32ca630, thread_var=0x32d9ed0) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:547 #9 0x00007f39ffa6d5c4 in execute_native_thread_routine () from /lib64/libstdc++.so.6 #10 0x00007f39ff7fcb1a in start_thread () from /lib64/libc.so.6 #11 0x00007f39ff881650 in clone3 () from /lib64/libc.so.6   Thread 13 (Thread 0x7f39d89f9640 (LWP 1063568)): #0 0x00007f39ff7f973a in __futex_abstimed_wait_common () from /lib64/libc.so.6 #1 0x00007f39ff7fc482 in pthread_cond_clockwait@GLIBC_2.30 () from /lib64/libc.so.6 #2 0x0000000000d6b43c in std::__condvar::wait_until (this=0x32da050, __m=<error reading variable: Cannot access memory at address 0x89>, __clock=1, __abs_time=@0x7f39d89f8cb8: {tv_sec = 179179, tv_nsec = 584950335}) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/std_mutex.h:169 #3 std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x32da050, __lock=@0x7f39d89f8d00: {_M_device = 0x32ca740, _M_owns = true}, __atime=<optimized out>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:201 #4 std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x32da050, __lock=@0x7f39d89f8d00: {_M_device = 0x32ca740, _M_owns = true}, __atime=<optimized out>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:111 #5 std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (this=0x32da050, __lock=@0x7f39d89f8d00: {_M_device = 0x32ca740, _M_owns = true}, __rtime=@0x32ca768: {__r = 60000}) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:163 #6 tpool::thread_pool_generic::wait_for_tasks (this=this@entry=0x32ca630, lk=@0x7f39d89f8d00: {_M_device = 0x32ca740, _M_owns = true}, thread_data=thread_data@entry=0x32da050) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:449 #7 0x0000000000d6b5ce in tpool::thread_pool_generic::get_task (this=this@entry=0x32ca630, thread_var=thread_var@entry=0x32da050, t=t@entry=0x7f39d89f8d50) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:502 #8 0x0000000000d6b7d4 in tpool::thread_pool_generic::worker_main (this=0x32ca630, thread_var=0x32da050) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:547 #9 0x00007f39ffa6d5c4 in execute_native_thread_routine () from /lib64/libstdc++.so.6 #10 0x00007f39ff7fcb1a in start_thread () from /lib64/libc.so.6 #11 0x00007f39ff881650 in clone3 () from /lib64/libc.so.6   Thread 12 (Thread 0x7f39fd3ff640 (LWP 1063546)): #0 0x00007f39ff7f973a in __futex_abstimed_wait_common () from /lib64/libc.so.6 #1 0x00007f39ff7fc1b4 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libc.so.6 #2 0x0000000000ac73a3 in inline_mysql_cond_timedwait (that=0x1ea38b8 <COND_checkpoint+40>, mutex=0x189, abstime=0x7f39fd3fed00, src_line=116, src_file=<optimized out>) at /home/dan/repos/mariadb-server-10.8/include/mysql/psi/mysql_thread.h:1088 #3 my_service_thread_sleep (control=0x15d4c70 <checkpoint_control>, sleep_time=29000000000) at /home/dan/repos/mariadb-server-10.8/storage/maria/ma_servicethread.c:115 #4 0x0000000000ac081f in ma_checkpoint_background (arg=<optimized out>) at /home/dan/repos/mariadb-server-10.8/storage/maria/ma_checkpoint.c:725 #5 0x00007f39ff7fcb1a in start_thread () from /lib64/libc.so.6 #6 0x00007f39ff881650 in clone3 () from /lib64/libc.so.6   Thread 11 (Thread 0x7f39da1fc640 (LWP 1063565)): #0 0x00007f39ff7f973a in __futex_abstimed_wait_common () from /lib64/libc.so.6 #1 0x00007f39ff7fc482 in pthread_cond_clockwait@GLIBC_2.30 () from /lib64/libc.so.6 #2 0x0000000000d6b43c in std::__condvar::wait_until (this=0x32da1d0, __m=<error reading variable: Cannot access memory at address 0x89>, __clock=1, __abs_time=@0x7f39da1fbcb8: {tv_sec = 179179, tv_nsec = 584899718}) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/std_mutex.h:169 #3 std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x32da1d0, __lock=@0x7f39da1fbd00: {_M_device = 0x32ca740, _M_owns = true}, __atime=<optimized out>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:201 #4 std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x32da1d0, __lock=@0x7f39da1fbd00: {_M_device = 0x32ca740, _M_owns = true}, __atime=<optimized out>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:111 #5 std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (this=0x32da1d0, __lock=@0x7f39da1fbd00: {_M_device = 0x32ca740, _M_owns = true}, __rtime=@0x32ca768: {__r = 60000}) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:163 #6 tpool::thread_pool_generic::wait_for_tasks (this=this@entry=0x32ca630, lk=@0x7f39da1fbd00: {_M_device = 0x32ca740, _M_owns = true}, thread_data=thread_data@entry=0x32da1d0) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:449 #7 0x0000000000d6b5ce in tpool::thread_pool_generic::get_task (this=this@entry=0x32ca630, thread_var=thread_var@entry=0x32da1d0, t=t@entry=0x7f39da1fbd50) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:502 #8 0x0000000000d6b7a8 in tpool::thread_pool_generic::worker_main (this=0x32ca630, thread_var=0x32da1d0) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:547 #9 0x00007f39ffa6d5c4 in execute_native_thread_routine () from /lib64/libstdc++.so.6 #10 0x00007f39ff7fcb1a in start_thread () from /lib64/libc.so.6 #11 0x00007f39ff881650 in clone3 () from /lib64/libc.so.6   Thread 10 (Thread 0x7f39ba7fc640 (LWP 1063572)): #0 0x00007f39ff7f973a in __futex_abstimed_wait_common () from /lib64/libc.so.6 #1 0x00007f39ff7fc482 in pthread_cond_clockwait@GLIBC_2.30 () from /lib64/libc.so.6 #2 0x0000000000d6b43c in std::__condvar::wait_until (this=0x32d9e50, __m=<error reading variable: Cannot access memory at address 0x89>, __clock=1, __abs_time=@0x7f39ba7fbcb8: {tv_sec = 179179, tv_nsec = 584886846}) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/std_mutex.h:169 #3 std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x32d9e50, __lock=@0x7f39ba7fbd00: {_M_device = 0x32ca740, _M_owns = true}, __atime=<optimized out>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:201 #4 std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x32d9e50, __lock=@0x7f39ba7fbd00: {_M_device = 0x32ca740, _M_owns = true}, __atime=<optimized out>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:111 #5 std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (this=0x32d9e50, __lock=@0x7f39ba7fbd00: {_M_device = 0x32ca740, _M_owns = true}, __rtime=@0x32ca768: {__r = 60000}) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:163 #6 tpool::thread_pool_generic::wait_for_tasks (this=this@entry=0x32ca630, lk=@0x7f39ba7fbd00: {_M_device = 0x32ca740, _M_owns = true}, thread_data=thread_data@entry=0x32d9e50) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:449 #7 0x0000000000d6b5ce in tpool::thread_pool_generic::get_task (this=this@entry=0x32ca630, thread_var=thread_var@entry=0x32d9e50, t=t@entry=0x7f39ba7fbd50) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:502 #8 0x0000000000d6b7d4 in tpool::thread_pool_generic::worker_main (this=0x32ca630, thread_var=0x32d9e50) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:547 #9 0x00007f39ffa6d5c4 in execute_native_thread_routine () from /lib64/libstdc++.so.6 #10 0x00007f39ff7fcb1a in start_thread () from /lib64/libc.so.6 #11 0x00007f39ff881650 in clone3 () from /lib64/libc.so.6   Thread 9 (Thread 0x7f39db9ff640 (LWP 1063548)): #0 0x00007f39ff8772fb in fdatasync () from /lib64/libc.so.6 #1 0x0000000000c09445 in os_file_sync_posix (file=12) at /home/dan/repos/mariadb-server-10.8/storage/innobase/os/os0file.cc:888 #2 os_file_flush_func (file=file@entry=12) at /home/dan/repos/mariadb-server-10.8/storage/innobase/os/os0file.cc:974 #3 0x0000000000d1bb98 in pfs_os_file_flush_func (file={m_file = 12, m_psi = <synthetic pointer>}, src_line=514, src_file=<optimized out>) at /home/dan/repos/mariadb-server-10.8/storage/innobase/include/os0file.inl:347 #4 fil_space_t::flush_low (this=this@entry=0x33c2578) at /home/dan/repos/mariadb-server-10.8/storage/innobase/fil/fil0fil.cc:514 #5 0x0000000000d1e326 in fil_flush_file_spaces () at /home/dan/repos/mariadb-server-10.8/storage/innobase/fil/fil0fil.cc:2904 #6 0x0000000000ce8148 in buf_flush_sync_for_checkpoint (lsn=13857) at /home/dan/repos/mariadb-server-10.8/storage/innobase/buf/buf0flu.cc:1956 #7 buf_flush_page_cleaner () at /home/dan/repos/mariadb-server-10.8/storage/innobase/buf/buf0flu.cc:2206 #8 0x00007f39ffa6d5c4 in execute_native_thread_routine () from /lib64/libstdc++.so.6 #9 0x00007f39ff7fcb1a in start_thread () from /lib64/libc.so.6 #10 0x00007f39ff881650 in clone3 () from /lib64/libc.so.6   Thread 8 (Thread 0x7f39ff4ff280 (LWP 1063539)): #0 0x00007f39ff7f973a in __futex_abstimed_wait_common () from /lib64/libc.so.6 #1 0x00007f39ff7fbeb0 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libc.so.6 #2 0x0000000000ce941f in buf_flush_wait (lsn=13857) at /home/dan/repos/mariadb-server-10.8/storage/innobase/buf/buf0flu.cc:1833 #3 buf_flush_sync () at /home/dan/repos/mariadb-server-10.8/storage/innobase/buf/buf0flu.cc:2491 #4 0x0000000000c8326c in srv_start (create_new_db=<optimized out>) at /home/dan/repos/mariadb-server-10.8/storage/innobase/srv/srv0start.cc:1172 #5 0x0000000000bb00c9 in innodb_init (p=<optimized out>) at /home/dan/repos/mariadb-server-10.8/storage/innobase/handler/ha_innodb.cc:4203 #6 0x000000000093f198 in ha_initialize_handlerton (plugin=0x32105e0) at /home/dan/repos/mariadb-server-10.8/sql/handler.cc:649 #7 0x000000000073ccd2 in plugin_initialize (tmp_root=<optimized out>, plugin=plugin@entry=0x32105e0, argc=argc@entry=0x166e550 <remaining_argc>, argv=argv@entry=0x31d27e8, options_only=false) at /home/dan/repos/mariadb-server-10.8/sql/sql_plugin.cc:1462 #8 0x000000000073c541 in plugin_init (argc=0x166e550 <remaining_argc>, argv=0x31d27e8, flags=1) at /home/dan/repos/mariadb-server-10.8/sql/sql_plugin.cc:1755 #9 0x000000000063ca2c in init_server_components () at /home/dan/repos/mariadb-server-10.8/sql/mysqld.cc:5088 #10 0x00000000006397ba in mysqld_main (argc=<optimized out>, argv=<optimized out>) at /home/dan/repos/mariadb-server-10.8/sql/mysqld.cc:5704 #11 0x00007f39ff79c560 in __libc_start_call_main () from /lib64/libc.so.6 #12 0x00007f39ff79c60c in __libc_start_main_impl () from /lib64/libc.so.6 #13 0x00000000006369f5 in _start ()   Thread 7 (Thread 0x7f39d91fa640 (LWP 1063567)): #0 0x00007f39ff7f973a in __futex_abstimed_wait_common () from /lib64/libc.so.6 #1 0x00007f39ff7fc482 in pthread_cond_clockwait@GLIBC_2.30 () from /lib64/libc.so.6 #2 0x0000000000d6b43c in std::__condvar::wait_until (this=0x32da0d0, __m=<error reading variable: Cannot access memory at address 0x89>, __clock=1, __abs_time=@0x7f39d91f9cb8: {tv_sec = 179179, tv_nsec = 584905003}) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/std_mutex.h:169 #3 std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x32da0d0, __lock=@0x7f39d91f9d00: {_M_device = 0x32ca740, _M_owns = true}, __atime=<optimized out>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:201 #4 std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x32da0d0, __lock=@0x7f39d91f9d00: {_M_device = 0x32ca740, _M_owns = true}, __atime=<optimized out>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:111 #5 std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (this=0x32da0d0, __lock=@0x7f39d91f9d00: {_M_device = 0x32ca740, _M_owns = true}, __rtime=@0x32ca768: {__r = 60000}) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:163 #6 tpool::thread_pool_generic::wait_for_tasks (this=this@entry=0x32ca630, lk=@0x7f39d91f9d00: {_M_device = 0x32ca740, _M_owns = true}, thread_data=thread_data@entry=0x32da0d0) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:449 #7 0x0000000000d6b5ce in tpool::thread_pool_generic::get_task (this=this@entry=0x32ca630, thread_var=thread_var@entry=0x32da0d0, t=t@entry=0x7f39d91f9d50) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:502 #8 0x0000000000d6b7a8 in tpool::thread_pool_generic::worker_main (this=0x32ca630, thread_var=0x32da0d0) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:547 #9 0x00007f39ffa6d5c4 in execute_native_thread_routine () from /lib64/libstdc++.so.6 #10 0x00007f39ff7fcb1a in start_thread () from /lib64/libc.so.6 #11 0x00007f39ff881650 in clone3 () from /lib64/libc.so.6   Thread 6 (Thread 0x7f39bb7fe640 (LWP 1063570)): #0 0x00007f39ff7f973a in __futex_abstimed_wait_common () from /lib64/libc.so.6 #1 0x00007f39ff7fc482 in pthread_cond_clockwait@GLIBC_2.30 () from /lib64/libc.so.6 #2 0x0000000000d6b43c in std::__condvar::wait_until (this=0x32d9f50, __m=<error reading variable: Cannot access memory at address 0x89>, __clock=1, __abs_time=@0x7f39bb7fdcb8: {tv_sec = 179179, tv_nsec = 584944909}) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/std_mutex.h:169 #3 std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x32d9f50, __lock=@0x7f39bb7fdd00: {_M_device = 0x32ca740, _M_owns = true}, __atime=<optimized out>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:201 #4 std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x32d9f50, __lock=@0x7f39bb7fdd00: {_M_device = 0x32ca740, _M_owns = true}, __atime=<optimized out>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:111 #5 std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (this=0x32d9f50, __lock=@0x7f39bb7fdd00: {_M_device = 0x32ca740, _M_owns = true}, __rtime=@0x32ca768: {__r = 60000}) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:163 #6 tpool::thread_pool_generic::wait_for_tasks (this=this@entry=0x32ca630, lk=@0x7f39bb7fdd00: {_M_device = 0x32ca740, _M_owns = true}, thread_data=thread_data@entry=0x32d9f50) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:449 #7 0x0000000000d6b5ce in tpool::thread_pool_generic::get_task (this=this@entry=0x32ca630, thread_var=thread_var@entry=0x32d9f50, t=t@entry=0x7f39bb7fdd50) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:502 #8 0x0000000000d6b7a8 in tpool::thread_pool_generic::worker_main (this=0x32ca630, thread_var=0x32d9f50) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:547 #9 0x00007f39ffa6d5c4 in execute_native_thread_routine () from /lib64/libstdc++.so.6 #10 0x00007f39ff7fcb1a in start_thread () from /lib64/libc.so.6 #11 0x00007f39ff881650 in clone3 () from /lib64/libc.so.6   Thread 5 (Thread 0x7f39bbfff640 (LWP 1063569)): #0 0x00007f39ff7f973a in __futex_abstimed_wait_common () from /lib64/libc.so.6 #1 0x00007f39ff7fc482 in pthread_cond_clockwait@GLIBC_2.30 () from /lib64/libc.so.6 #2 0x0000000000d6b43c in std::__condvar::wait_until (this=0x32d9fd0, __m=<error reading variable: Cannot access memory at address 0x89>, __clock=1, __abs_time=@0x7f39bbffecb8: {tv_sec = 179179, tv_nsec = 584956879}) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/std_mutex.h:169 #3 std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x32d9fd0, __lock=@0x7f39bbffed00: {_M_device = 0x32ca740, _M_owns = true}, __atime=<optimized out>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:201 #4 std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x32d9fd0, __lock=@0x7f39bbffed00: {_M_device = 0x32ca740, _M_owns = true}, __atime=<optimized out>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:111 #5 std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (this=0x32d9fd0, __lock=@0x7f39bbffed00: {_M_device = 0x32ca740, _M_owns = true}, __rtime=@0x32ca768: {__r = 60000}) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:163 #6 tpool::thread_pool_generic::wait_for_tasks (this=this@entry=0x32ca630, lk=@0x7f39bbffed00: {_M_device = 0x32ca740, _M_owns = true}, thread_data=thread_data@entry=0x32d9fd0) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:449 #7 0x0000000000d6b5ce in tpool::thread_pool_generic::get_task (this=this@entry=0x32ca630, thread_var=thread_var@entry=0x32d9fd0, t=t@entry=0x7f39bbffed50) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:502 #8 0x0000000000d6b7a8 in tpool::thread_pool_generic::worker_main (this=0x32ca630, thread_var=0x32d9fd0) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:547 #9 0x00007f39ffa6d5c4 in execute_native_thread_routine () from /lib64/libstdc++.so.6 #10 0x00007f39ff7fcb1a in start_thread () from /lib64/libc.so.6 #11 0x00007f39ff881650 in clone3 () from /lib64/libc.so.6   Thread 4 (Thread 0x7f39da9fd640 (LWP 1063554)): #0 0x00007f39ff7f973a in __futex_abstimed_wait_common () from /lib64/libc.so.6 #1 0x00007f39ff7fc482 in pthread_cond_clockwait@GLIBC_2.30 () from /lib64/libc.so.6 #2 0x0000000000d6b43c in std::__condvar::wait_until (this=0x32da250, __m=<error reading variable: Cannot access memory at address 0x89>, __clock=1, __abs_time=@0x7f39da9fccb8: {tv_sec = 179179, tv_nsec = 584888915}) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/std_mutex.h:169 #3 std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x32da250, __lock=@0x7f39da9fcd00: {_M_device = 0x32ca740, _M_owns = true}, __atime=<optimized out>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:201 #4 std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x32da250, __lock=@0x7f39da9fcd00: {_M_device = 0x32ca740, _M_owns = true}, __atime=<optimized out>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:111 #5 std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (this=0x32da250, __lock=@0x7f39da9fcd00: {_M_device = 0x32ca740, _M_owns = true}, __rtime=@0x32ca768: {__r = 60000}) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:163 #6 tpool::thread_pool_generic::wait_for_tasks (this=this@entry=0x32ca630, lk=@0x7f39da9fcd00: {_M_device = 0x32ca740, _M_owns = true}, thread_data=thread_data@entry=0x32da250) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:449 #7 0x0000000000d6b5ce in tpool::thread_pool_generic::get_task (this=this@entry=0x32ca630, thread_var=thread_var@entry=0x32da250, t=t@entry=0x7f39da9fcd50) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:502 #8 0x0000000000d6b7d4 in tpool::thread_pool_generic::worker_main (this=0x32ca630, thread_var=0x32da250) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:547 #9 0x00007f39ffa6d5c4 in execute_native_thread_routine () from /lib64/libstdc++.so.6 #10 0x00007f39ff7fcb1a in start_thread () from /lib64/libc.so.6 #11 0x00007f39ff881650 in clone3 () from /lib64/libc.so.6   Thread 3 (Thread 0x7f39d99fb640 (LWP 1063566)): #0 0x00007f39ff7f973a in __futex_abstimed_wait_common () from /lib64/libc.so.6 #1 0x00007f39ff7fc482 in pthread_cond_clockwait@GLIBC_2.30 () from /lib64/libc.so.6 #2 0x0000000000d6b43c in std::__condvar::wait_until (this=0x32da150, __m=<error reading variable: Cannot access memory at address 0x89>, __clock=1, __abs_time=@0x7f39d99facb8: {tv_sec = 179179, tv_nsec = 584892851}) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/std_mutex.h:169 #3 std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x32da150, __lock=@0x7f39d99fad00: {_M_device = 0x32ca740, _M_owns = true}, __atime=<optimized out>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:201 #4 std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x32da150, __lock=@0x7f39d99fad00: {_M_device = 0x32ca740, _M_owns = true}, __atime=<optimized out>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:111 #5 std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (this=0x32da150, __lock=@0x7f39d99fad00: {_M_device = 0x32ca740, _M_owns = true}, __rtime=@0x32ca768: {__r = 60000}) at /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/condition_variable:163 #6 tpool::thread_pool_generic::wait_for_tasks (this=this@entry=0x32ca630, lk=@0x7f39d99fad00: {_M_device = 0x32ca740, _M_owns = true}, thread_data=thread_data@entry=0x32da150) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:449 #7 0x0000000000d6b5ce in tpool::thread_pool_generic::get_task (this=this@entry=0x32ca630, thread_var=thread_var@entry=0x32da150, t=t@entry=0x7f39d99fad50) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:502 #8 0x0000000000d6b7d4 in tpool::thread_pool_generic::worker_main (this=0x32ca630, thread_var=0x32da150) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:547 #9 0x00007f39ffa6d5c4 in execute_native_thread_routine () from /lib64/libstdc++.so.6 #10 0x00007f39ff7fcb1a in start_thread () from /lib64/libc.so.6 #11 0x00007f39ff881650 in clone3 () from /lib64/libc.so.6   Thread 2 (Thread 0x7f39e9d1a640 (LWP 1063547)): #0 0x00007f39ff879ecd in syscall () from /lib64/libc.so.6 #1 0x00007f3a001094f9 in _io_uring_get_cqe () from /lib64/liburing.so.2 #2 0x00007f3a00109a9b in __io_uring_get_cqe () from /lib64/liburing.so.2 #3 0x0000000000d6e072 in io_uring_wait_cqe_nr (ring=0x335d9c8, cqe_ptr=0x7f39e9d19d48, wait_nr=1) at /usr/include/liburing.h:635 #4 io_uring_wait_cqe (ring=0x335d9c8, cqe_ptr=0x7f39e9d19d48) at /usr/include/liburing.h:655 #5 (anonymous namespace)::aio_uring::thread_routine (aio=0x335d9c0) at /home/dan/repos/mariadb-server-10.8/tpool/aio_liburing.cc:137 #6 0x00007f39ffa6d5c4 in execute_native_thread_routine () from /lib64/libstdc++.so.6 #7 0x00007f39ff7fcb1a in start_thread () from /lib64/libc.so.6 #8 0x00007f39ff881650 in clone3 () from /lib64/libc.so.6   Thread 1 (Thread 0x7f39db1fe640 (LWP 1063553)): #0 0x00007f39ff7fe88c in __pthread_kill_implementation () from /lib64/libc.so.6 #1 0x00007f39ff7b16a6 in raise () from /lib64/libc.so.6 #2 0x00007f39ff79b865 in abort () from /lib64/libc.so.6 #3 0x0000000000ca6e9b in ut_dbg_assertion_failed (expr=0x1160de8 "cb->m_err == DB_SUCCESS", file=<optimized out>, line=<optimized out>, line@entry=3571) at /home/dan/repos/mariadb-server-10.8/storage/innobase/ut/ut0dbg.cc:60 #4 0x0000000000c0c2d0 in io_callback (cb=<optimized out>) at /home/dan/repos/mariadb-server-10.8/storage/innobase/os/os0file.cc:3571 #5 0x0000000000d6d6dd in tpool::task_group::execute (this=0x33358b0, t=0x335b450) at /home/dan/repos/mariadb-server-10.8/tpool/task_group.cc:55 #6 0x0000000000d6b7c6 in tpool::thread_pool_generic::worker_main (this=0x32ca630, thread_var=0x32da2d0) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:549 #7 0x00007f39ffa6d5c4 in execute_native_thread_routine () from /lib64/libstdc++.so.6 #8 0x00007f39ff7fcb1a in start_thread () from /lib64/libc.so.6 #9 0x00007f39ff881650 in clone3 () from /lib64/libc.so.6 (gdb) up #2 0x00007f39ff79b865 in abort () from /lib64/libc.so.6 (gdb) #3 0x0000000000ca6e9b in ut_dbg_assertion_failed (expr=0x1160de8 "cb->m_err == DB_SUCCESS", file=<optimized out>, line=<optimized out>, line@entry=3571) at /home/dan/repos/mariadb-server-10.8/storage/innobase/ut/ut0dbg.cc:60 60 abort(); (gdb) #4 0x0000000000c0c2d0 in io_callback (cb=<optimized out>) at /home/dan/repos/mariadb-server-10.8/storage/innobase/os/os0file.cc:3571 3571 ut_a(cb->m_err == DB_SUCCESS); (gdb) #5 0x0000000000d6d6dd in tpool::task_group::execute (this=0x33358b0, t=0x335b450) at /home/dan/repos/mariadb-server-10.8/tpool/task_group.cc:55 55 t->m_func(t->m_arg); (gdb) p *((tpool::aiocb *) t->m_arg) $1 = {<iovec> = {iov_base = 0x7f39fc5fc000, iov_len = 16384}, m_fh = 12, m_opcode = tpool::aio_opcode::AIO_PWRITE, m_offset = 180224, m_buffer = 0x7f39fc5fc000, m_len = 16384, m_callback = 0xc0c240 <io_callback(tpool::aiocb*)>, m_group = 0x33358b0, m_ret_len = 0, m_err = 4, m_internal = 0x0, m_internal_task = {_vptr$task = 0x1182828 <vtable for tpool::task+16>, m_func = 0xc0c240 <io_callback(tpool::aiocb*)>, m_arg = 0x335b3f8, m_group = 0x33358b0}, m_userdata = "\340\266\\\374\071\177\000\000\000\000\000\000\000\000\000\000\330&<\003\000\000\000\000\021\000\000\000\071\177\000"}