Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Incomplete
-
10.6.5
-
production, docker, linux
25GB of memory free
CPU about 15%
About 260 databases on the server
Description
Few seconds after creating new database and during table migration into it, mariadb server failed with following error lines.
Our configuration:
[mysqld]
|
max_allowed_packet=64M
|
back_log=1000
|
log_warnings=3
|
max_connections=1512
|
max_connect_errors=4294967295
|
key_buffer_size=1G
|
tmp_table_size=512M
|
thread_cache_size=200
|
innodb_rollback_on_timeout=1
|
innodb_buffer_pool_size=8G
|
innodb_log_file_size=4G
|
innodb_flush_log_at_trx_commit=1
|
innodb_file_per_table=1
|
First occurrence log:
2022-08-15 7:30:26 273422829 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
|
2022-08-15 7:30:26 273422829 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
|
2022-08-15 7:30:26 273422829 [Note] InnoDB: Online DDL : Applying log to index
|
2022-08-15 7:30:27 273422829 [Note] InnoDB: Cannot close file ./db_195/user_sessions.ibd because of pending fsync
|
2022-08-15 07:30:27 0x7efcc0474700 InnoDB: Assertion failure in file /home/buildbot/buildbot/build/mariadb-10.6.5/storage/innobase/trx/trx0trx.cc line 918
|
InnoDB: Failing assertion: trx->lock.table_locks.empty()
|
InnoDB: We intentionally generate a memory trap.
|
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
InnoDB: If you get repeated assertion failures or crashes, even
|
InnoDB: immediately after the mariadbd startup, there may be
|
InnoDB: corruption in the InnoDB tablespace. Please refer to
|
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
|
InnoDB: about forcing recovery.
|
220815 7:30:27 [ERROR] mysqld got signal 6 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
Server version: 10.6.5-MariaDB-1:10.6.5+maria~focal
|
key_buffer_size=1073741824
|
read_buffer_size=131072
|
max_used_connections=339
|
max_threads=1026
|
thread_count=200
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 3307842 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
Thread pointer: 0x7efa94000c58
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x7efcc0473d98 thread_stack 0x49000
|
mysqld(my_print_stacktrace+0x32)[0x5630e4505442]
|
Printing to addr2line failed
|
mysqld(handle_fatal_signal+0x485)[0x5630e3fb86a5]
|
/lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0)[0x7effa64163c0]
|
UPDATE 2022-08-24:
Today this error occurred again, under same conditions. We created new DB named "db_276" and started to migrate tables.
2022-08-24 7:07:27 16520720 [Note] InnoDB: Cannot close file ./db_103/events.ibd because of pending fsync
|
2022-08-24 7:07:27 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
|
2022-08-24 7:07:27 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
|
2022-08-24 7:07:27 16520720 [Note] InnoDB: Online DDL : Applying log to index
|
2022-08-24 7:07:27 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
|
2022-08-24 7:07:27 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
|
2022-08-24 7:07:27 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
|
2022-08-24 7:07:27 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
|
2022-08-24 7:07:27 16520720 [Note] InnoDB: Online DDL : Applying log to index
|
2022-08-24 7:07:28 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
|
2022-08-24 7:07:28 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
|
2022-08-24 7:07:28 16520720 [Note] InnoDB: Online DDL : Applying log to index
|
2022-08-24 7:07:28 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
|
2022-08-24 7:07:28 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
|
2022-08-24 7:07:28 16520720 [Note] InnoDB: Online DDL : Applying log to index
|
2022-08-24 7:07:28 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
|
2022-08-24 7:07:28 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
|
2022-08-24 7:07:29 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
|
2022-08-24 7:07:29 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
|
2022-08-24 7:07:30 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
|
2022-08-24 7:07:30 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
|
2022-08-24 7:07:30 16520720 [Note] InnoDB: Online DDL : Applying log to index
|
2022-08-24 7:07:30 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
|
2022-08-24 7:07:30 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
|
2022-08-24 7:07:30 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
|
2022-08-24 7:07:30 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
|
2022-08-24 7:07:30 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
|
2022-08-24 7:07:30 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
|
2022-08-24 7:07:30 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
|
2022-08-24 7:07:30 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
|
2022-08-24 7:07:30 16520720 [Note] InnoDB: Online DDL : Applying log to index
|
2022-08-24 7:07:31 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
|
2022-08-24 7:07:31 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
|
2022-08-24 7:07:31 16520720 [Note] InnoDB: Online DDL : Applying log to index
|
2022-08-24 7:07:31 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
|
2022-08-24 7:07:31 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
|
2022-08-24 7:07:31 16520720 [Note] InnoDB: Online DDL : Applying log to index
|
2022-08-24 7:07:31 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
|
2022-08-24 7:07:31 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
|
2022-08-24 7:07:31 16520720 [Note] InnoDB: Online DDL : Applying log to index
|
2022-08-24 7:07:31 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
|
2022-08-24 7:07:31 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
|
2022-08-24 7:07:31 16520720 [Note] InnoDB: Online DDL : Applying log to index
|
2022-08-24 7:07:31 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
|
2022-08-24 7:07:31 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
|
2022-08-24 7:07:31 16520720 [Note] InnoDB: Online DDL : Applying log to index
|
2022-08-24 7:07:32 16520720 [Note] InnoDB: Cannot close file ./db_103/events.ibd because of pending fsync
|
2022-08-24 7:07:32 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
|
2022-08-24 7:07:32 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
|
2022-08-24 7:07:32 16520720 [Note] InnoDB: Online DDL : Applying log to index
|
2022-08-24 7:07:32 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
|
2022-08-24 7:07:32 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
|
2022-08-24 7:07:32 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
|
2022-08-24 7:07:32 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
|
2022-08-24 7:07:33 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
|
2022-08-24 7:07:33 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
|
2022-08-24 7:07:33 16520720 [Note] InnoDB: Online DDL : Applying log to index
|
2022-08-24 7:07:33 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
|
2022-08-24 7:07:33 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
|
2022-08-24 7:07:33 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
|
2022-08-24 7:07:33 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
|
2022-08-24 7:07:33 16520720 [Note] InnoDB: Online DDL : Applying log to index
|
2022-08-24 7:07:34 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
|
2022-08-24 7:07:34 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
|
2022-08-24 7:07:34 16520720 [Note] InnoDB: Online DDL : Applying log to index
|
2022-08-24 7:07:34 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
|
2022-08-24 7:07:34 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
|
2022-08-24 7:07:34 16520720 [Note] InnoDB: Online DDL : Applying log to index
|
2022-08-24 7:07:34 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
|
2022-08-24 7:07:34 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
|
2022-08-24 7:07:34 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
|
2022-08-24 7:07:34 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
|
2022-08-24 7:07:34 16520720 [Note] InnoDB: Online DDL : Applying log to index
|
2022-08-24 7:07:35 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
|
2022-08-24 7:07:35 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
|
2022-08-24 7:07:35 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
|
2022-08-24 7:07:35 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
|
2022-08-24 7:07:36 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
|
2022-08-24 7:07:36 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
|
2022-08-24 7:07:36 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
|
2022-08-24 7:07:36 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
|
2022-08-24 7:07:37 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
|
2022-08-24 7:07:37 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
|
2022-08-24 7:07:37 16520720 [Note] InnoDB: Online DDL : Applying log to index
|
2022-08-24 7:07:37 16520720 [Note] InnoDB: Cannot close file ./db_103/events.ibd because of pending fsync
|
2022-08-24 07:07:37 0x7fb5dc3de700 InnoDB: Assertion failure in file /home/buildbot/buildbot/build/mariadb-10.6.5/storage/innobase/trx/trx0trx.cc line 918
|
InnoDB: Failing assertion: trx->lock.table_locks.empty()
|
InnoDB: We intentionally generate a memory trap.
|
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
InnoDB: If you get repeated assertion failures or crashes, even
|
InnoDB: immediately after the mariadbd startup, there may be
|
InnoDB: corruption in the InnoDB tablespace. Please refer to
|
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
|
InnoDB: about forcing recovery.
|
220824 7:07:37 [ERROR] mysqld got signal 6 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
Server version: 10.6.5-MariaDB-1:10.6.5+maria~focal
|
key_buffer_size=1073741824
|
read_buffer_size=131072
|
max_used_connections=641
|
max_threads=1514
|
thread_count=201
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 4382437 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
Thread pointer: 0x7fb3ac000c58
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x7fb5dc3ddd98 thread_stack 0x49000
|
mysqld(my_print_stacktrace+0x32)[0x559b6f5e7442]
|
mysqld(handle_fatal_signal+0x485)[0x559b6f09a6a5]
|