Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Fixed
-
10.6.8, 10.7.4, 10.8.3, 10.9.1, 10.10.1
-
None
Description
10.8.4
Yesterday, after starting to get the same issue described in MDEV-29364 (after a few days of not getting that following a restart),
I restarted MariaDB and then created a fresh backup.
Transferred to the other server, and ran the prepare.
Prepare seems to run OK (this runs in a docker container btw), then it runs a number of restarts and SQL queries to sort out a new random root password (same process that's been running for years)
However, with yesterday's backup, it consistently crashes after a while, with:
80 Segmentation fault (core dumped) mysqld -u root
And a core file is created. The core file seems to be unreadable if I open with Notepad++,
but the "mysql.err" file contains possibly more readable information.
Here are some excerpts of the "mysql.err":
2022-08-30 8:40:24 0 [Note] InnoDB: Rolled back recovered transaction 45987095395
|
2022-08-30 8:40:24 0 [Note] InnoDB: Rolled back recovered transaction 45987095393
|
2022-08-30 8:40:24 0 [Note] InnoDB: Rolled back recovered transaction 45987095399
|
2022-08-30 8:40:24 0 [Note] InnoDB: Rollback of non-prepared transactions completed
|
2022-08-30 8:40:25 0 [Note] Server socket created on IP: '0.0.0.0'.
|
2022-08-30 8:40:25 1 [ERROR] InnoDB: Database page corruption on disk or a failed read of file './mysql/gtid_slave_pos.ibd' page [page id: space=20, page number=3]. You may have to recover from a backup.
|
2022-08-30 8:40:25 1 [Note] InnoDB: Page dump (16384 bytes):
|
2022-08-30 8:40:25 1 [Note] InnoDB: .............. [huge dump] ..............
|
2022-08-30 8:40:25 1 [Note] InnoDB: End of page dump
|
2022-08-30 8:40:25 1 [ERROR] InnoDB: File './mysql/gtid_slave_pos.ibd' is corrupted
|
2022-08-30 8:40:25 1 [Note] InnoDB: You can use CHECK TABLE to scan your table for corruption. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery.
|
2022-08-30 8:40:25 1 [ERROR] InnoDB: Database page corruption on disk or a failed read of file './mysql/gtid_slave_pos.ibd' page [page id: space=20, page number=3]. You may have to recover from a backup.
|
2022-08-30 8:40:25 1 [Note] InnoDB: Page dump (16384 bytes):
|
2022-08-30 8:40:25 1 [Note] InnoDB: .............. [huge dump] ..............
|
2022-08-30 8:40:25 1 [Note] InnoDB: End of page dump
|
2022-08-30 8:40:25 1 [Note] InnoDB: You can use CHECK TABLE to scan your table for corruption. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery.
|
2022-08-30 8:40:25 1 [ERROR] InnoDB: Database page corruption on disk or a failed read of file './mysql/gtid_slave_pos.ibd' page [page id: space=20, page number=3]. You may have to recover from a backup.
|
2022-08-30 8:40:25 1 [Note] InnoDB: Page dump (16384 bytes):
|
2022-08-30 8:40:25 1 [Note] InnoDB: .............. [huge dump] ..............
|
2022-08-30 8:40:25 1 [Note] InnoDB: End of page dump
|
2022-08-30 8:40:25 1 [Note] InnoDB: You can use CHECK TABLE to scan your table for corruption. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery.
|
2022-08-30 8:40:25 1 [ERROR] InnoDB: Database page corruption on disk or a failed read of file './mysql/gtid_slave_pos.ibd' page [page id: space=20, page number=3]. You may have to recover from a backup.
|
2022-08-30 8:40:25 1 [Note] InnoDB: Page dump (16384 bytes):
|
2022-08-30 8:40:25 1 [Note] InnoDB: .............. [huge dump] ..............
|
2022-08-30 8:40:25 1 [Note] InnoDB: End of page dump
|
2022-08-30 8:40:25 1 [Note] InnoDB: You can use CHECK TABLE to scan your table for corruption. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery.
|
2022-08-30 8:40:25 1 [ERROR] InnoDB: Database page corruption on disk or a failed read of file './mysql/gtid_slave_pos.ibd' page [page id: space=20, page number=3]. You may have to recover from a backup.
|
2022-08-30 8:40:25 1 [Note] InnoDB: Page dump (16384 bytes):
|
2022-08-30 8:40:25 1 [Note] InnoDB: .............. [huge dump] ..............
|
2022-08-30 8:40:25 1 [Note] InnoDB: End of page dump
|
2022-08-30 8:40:25 1 [Note] InnoDB: You can use CHECK TABLE to scan your table for corruption. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery.
|
2022-08-30 8:40:25 1 [ERROR] InnoDB: Database page corruption on disk or a failed read of file './mysql/gtid_slave_pos.ibd' page [page id: space=20, page number=3]. You may have to recover from a backup.
|
2022-08-30 8:40:25 1 [Note] InnoDB: Page dump (16384 bytes):
|
2022-08-30 8:40:25 1 [Note] InnoDB: .............. [huge dump] ..............
|
2022-08-30 8:40:25 1 [Note] InnoDB: End of page dump
|
2022-08-30 8:40:25 1 [Note] InnoDB: You can use CHECK TABLE to scan your table for corruption. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery.
|
2022-08-30 8:40:25 1 [ERROR] InnoDB: Database page corruption on disk or a failed read of file './mysql/gtid_slave_pos.ibd' page [page id: space=20, page number=3]. You may have to recover from a backup.
|
2022-08-30 8:40:25 1 [Note] InnoDB: Page dump (16384 bytes):
|
2022-08-30 8:40:25 1 [Note] InnoDB: .............. [huge dump] ..............
|
(many of the same)
At some point it starts showing my own databases/tables:
2022-08-30 8:41:05 0 [ERROR] InnoDB: Failed to read page 4631 from file './my_db/my_table.ibd': Page read from tablespace is corrupted.
|
2022-08-30 8:41:05 0 [Note] InnoDB: [small dump]
|
2022-08-30 8:41:05 0 [Note] InnoDB: [small dump]
|
2022-08-30 8:41:05 0 [Note] InnoDB: [small dump]
|
2022-08-30 8:41:05 0 [ERROR] InnoDB: Database page corruption on disk or a failed read of file './my_db/my_table.ibd' page [page id: space=393, page number=5773]. You may have to recover from a backup.
|
2022-08-30 8:41:05 0 [Note] InnoDB: [small dump]
|
2022-08-30 8:41:05 0 [Note] InnoDB: Page dump (16384 bytes):
|
...
2022-08-30 8:41:05 0 [Note] InnoDB: End of page dump
|
2022-08-30 8:41:05 0 [Note] InnoDB: [small dump]
|
2022-08-30 8:41:05 0 [Note] InnoDB: [small dump]
|
2022-08-30 8:41:05 0 [Note] InnoDB: [small dump]
|
2022-08-30 8:41:05 0 [Note] InnoDB: [small dump]
|
2022-08-30 8:41:05 0 [Note] InnoDB: [small dump]
|
2022-08-30 8:41:05 0 [Note] InnoDB: [small dump]
|
2022-08-30 8:41:05 0 [Note] InnoDB: [small dump]
|
2022-08-30 8:41:05 0 [Note] InnoDB: [small dump]
|
2022-08-30 8:41:05 0 [Note] InnoDB: [small dump]
|
2022-08-30 8:41:05 0 [Note] InnoDB: [small dump]
|
2022-08-30 8:41:05 0 [Note] InnoDB: [small dump]
|
2022-08-30 8:41:05 0 [Note] InnoDB: [small dump]
|
2022-08-30 8:41:05 0 [Note] InnoDB: You can use CHECK TABLE to scan your table for corruption. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery.
|
2022-08-30 8:41:05 0 [Note] InnoDB: [small dump]
|
2022-08-30 8:41:05 0 [Note] InnoDB: [small dump]
|
2022-08-30 8:41:05 0 [ERROR] InnoDB: Failed to read page 5210 from file './my_db/my_table.ibd': Page read from tablespace is corrupted.
|
2022-08-30 8:41:05 0 [Note] InnoDB: [small dump]
|
2022-08-30 8:41:05 0 [Note] InnoDB: [small dump]
|
2022-08-30 8:41:05 0 [ERROR] InnoDB: Database page corruption on disk or a failed read of file './my_db/my_table.ibd' page [page id: space=393, page number=5774]. You may have to recover from a backup.
|
...
...
continues with a lot of my databases/tables
...
...
and then the final bit:
2022-08-30 11:07:20 0 [Note] InnoDB: End of page dump
|
2022-08-30 11:07:20 0 [Note] InnoDB: You can use CHECK TABLE to scan your table for corruption. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery.
|
220830 11:07:20 [ERROR] mysqld got signal 11 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
 |
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
 |
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
 |
Server version: 10.8.4-MariaDB
|
key_buffer_size=134217728
|
read_buffer_size=131072
|
max_used_connections=2
|
max_threads=12
|
thread_count=2
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 919329 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
 |
Thread pointer: 0x55ea6b2d75b8
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x7f9879e60be8 thread_stack 0x49000
|
addr2line: 'mysqld': No such file
|
Printing to addr2line failed
|
mysqld(my_print_stacktrace+0x2e)[0x55ea67ebfd9e]
|
mysqld(handle_fatal_signal+0x307)[0x55ea6792aef7]
|
sigaction.c:0(__restore_rt)[0x7f9db45f4630]
|
addr2line: 'mysqld': No such file
|
mysqld(+0xeab38c)[0x55ea67db138c]
|
mysqld(+0xeb8c09)[0x55ea67dbec09]
|
mysqld(+0xebafe5)[0x55ea67dc0fe5]
|
mysqld(+0xebbb66)[0x55ea67dc1b66]
|
mysqld(+0xeaf98c)[0x55ea67db598c]
|
mysqld(+0xdd6baf)[0x55ea67cdcbaf]
|
mysqld(+0xd91c08)[0x55ea67c97c08]
|
mysqld(+0xdf6480)[0x55ea67cfc480]
|
mysqld(_ZN5tpool10task_group7executeEPNS_4taskE+0xa6)[0x55ea67e4a536]
|
mysqld(_ZN5tpool19thread_pool_generic11worker_mainEPNS_11worker_dataE+0x61)[0x55ea67e48ee1]
|
??:0(std::this_thread::__sleep_for(std::chrono::duration<long, std::ratio<1l, 1l> >, std::chrono::duration<long, std::ratio<1l, 1000000000l> >))[0x7f9db418e330]
|
pthread_create.c:0(start_thread)[0x7f9db45ecea5]
|
??:0(__clone)[0x7f9db3b07b0d]
|
 |
Trying to get some variables.
|
Some pointers may be invalid and cause the dump to abort.
|
Query (0x0): (null)
|
Connection ID (thread ID): 0
|
Status: NOT_KILLED
|
 |
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off
|
 |
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
|
information that should help you find out what is causing the crash.
|
 |
We think the query pointer is invalid, but we will try to print it anyway.
|
Query:
|
 |
Writing a core file...
|
Working directory at /var/lib/mysql
|
Resource Limits:
|
Limit Soft Limit Hard Limit Units
|
Max cpu time unlimited unlimited seconds
|
Max file size unlimited unlimited bytes
|
Max data size unlimited unlimited bytes
|
Max stack size 8388608 unlimited bytes
|
Max core file size unlimited unlimited bytes
|
Max resident set unlimited unlimited bytes
|
Max processes unlimited unlimited processes
|
Max open files 1048576 1048576 files
|
Max locked memory 65536 65536 bytes
|
Max address space unlimited unlimited bytes
|
Max file locks unlimited unlimited locks
|
Max pending signals 127822 127822 signals
|
Max msgqueue size 819200 819200 bytes
|
Max nice priority 0 0
|
Max realtime priority 0 0
|
Max realtime timeout unlimited unlimited us
|
Core pattern: |/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t e %P %I %h
|
 |
Kernel version: Linux version 3.10.0-1160.62.1.el7.x86_64 (mockbuild@kbuilder.bsys.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Tue Apr 5 16:57:59 UTC 2022
|
Do you think this could be caused by MDEV-29383?
Thanks.
Today's backup, however, seems to have gone well. I have a stable Snapshot, finally.
(on other days, it's MDEV-28974 that keeps affecting my daily snapshots)
It's really hard to get valid backups...
.
.
Attachments
Issue Links
- duplicates
-
-
- Closed
-