Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Fixed
-
10.6.8, 10.7.4, 10.8.3, 10.9.1, 10.10.1
-
None
Description
10.8.4
Yesterday, after starting to get the same issue described in MDEV-29364 (after a few days of not getting that following a restart),
I restarted MariaDB and then created a fresh backup.
Transferred to the other server, and ran the prepare.
Prepare seems to run OK (this runs in a docker container btw), then it runs a number of restarts and SQL queries to sort out a new random root password (same process that's been running for years)
However, with yesterday's backup, it consistently crashes after a while, with:
80 Segmentation fault (core dumped) mysqld -u root
And a core file is created. The core file seems to be unreadable if I open with Notepad++,
but the "mysql.err" file contains possibly more readable information.
Here are some excerpts of the "mysql.err":
2022-08-30 8:40:24 0 [Note] InnoDB: Rolled back recovered transaction 45987095395
|
2022-08-30 8:40:24 0 [Note] InnoDB: Rolled back recovered transaction 45987095393
|
2022-08-30 8:40:24 0 [Note] InnoDB: Rolled back recovered transaction 45987095399
|
2022-08-30 8:40:24 0 [Note] InnoDB: Rollback of non-prepared transactions completed
|
2022-08-30 8:40:25 0 [Note] Server socket created on IP: '0.0.0.0'.
|
2022-08-30 8:40:25 1 [ERROR] InnoDB: Database page corruption on disk or a failed read of file './mysql/gtid_slave_pos.ibd' page [page id: space=20, page number=3]. You may have to recover from a backup.
|
2022-08-30 8:40:25 1 [Note] InnoDB: Page dump (16384 bytes):
|
2022-08-30 8:40:25 1 [Note] InnoDB: .............. [huge dump] ..............
|
2022-08-30 8:40:25 1 [Note] InnoDB: End of page dump
|
2022-08-30 8:40:25 1 [ERROR] InnoDB: File './mysql/gtid_slave_pos.ibd' is corrupted
|
2022-08-30 8:40:25 1 [Note] InnoDB: You can use CHECK TABLE to scan your table for corruption. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery.
|
2022-08-30 8:40:25 1 [ERROR] InnoDB: Database page corruption on disk or a failed read of file './mysql/gtid_slave_pos.ibd' page [page id: space=20, page number=3]. You may have to recover from a backup.
|
2022-08-30 8:40:25 1 [Note] InnoDB: Page dump (16384 bytes):
|
2022-08-30 8:40:25 1 [Note] InnoDB: .............. [huge dump] ..............
|
2022-08-30 8:40:25 1 [Note] InnoDB: End of page dump
|
2022-08-30 8:40:25 1 [Note] InnoDB: You can use CHECK TABLE to scan your table for corruption. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery.
|
2022-08-30 8:40:25 1 [ERROR] InnoDB: Database page corruption on disk or a failed read of file './mysql/gtid_slave_pos.ibd' page [page id: space=20, page number=3]. You may have to recover from a backup.
|
2022-08-30 8:40:25 1 [Note] InnoDB: Page dump (16384 bytes):
|
2022-08-30 8:40:25 1 [Note] InnoDB: .............. [huge dump] ..............
|
2022-08-30 8:40:25 1 [Note] InnoDB: End of page dump
|
2022-08-30 8:40:25 1 [Note] InnoDB: You can use CHECK TABLE to scan your table for corruption. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery.
|
2022-08-30 8:40:25 1 [ERROR] InnoDB: Database page corruption on disk or a failed read of file './mysql/gtid_slave_pos.ibd' page [page id: space=20, page number=3]. You may have to recover from a backup.
|
2022-08-30 8:40:25 1 [Note] InnoDB: Page dump (16384 bytes):
|
2022-08-30 8:40:25 1 [Note] InnoDB: .............. [huge dump] ..............
|
2022-08-30 8:40:25 1 [Note] InnoDB: End of page dump
|
2022-08-30 8:40:25 1 [Note] InnoDB: You can use CHECK TABLE to scan your table for corruption. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery.
|
2022-08-30 8:40:25 1 [ERROR] InnoDB: Database page corruption on disk or a failed read of file './mysql/gtid_slave_pos.ibd' page [page id: space=20, page number=3]. You may have to recover from a backup.
|
2022-08-30 8:40:25 1 [Note] InnoDB: Page dump (16384 bytes):
|
2022-08-30 8:40:25 1 [Note] InnoDB: .............. [huge dump] ..............
|
2022-08-30 8:40:25 1 [Note] InnoDB: End of page dump
|
2022-08-30 8:40:25 1 [Note] InnoDB: You can use CHECK TABLE to scan your table for corruption. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery.
|
2022-08-30 8:40:25 1 [ERROR] InnoDB: Database page corruption on disk or a failed read of file './mysql/gtid_slave_pos.ibd' page [page id: space=20, page number=3]. You may have to recover from a backup.
|
2022-08-30 8:40:25 1 [Note] InnoDB: Page dump (16384 bytes):
|
2022-08-30 8:40:25 1 [Note] InnoDB: .............. [huge dump] ..............
|
2022-08-30 8:40:25 1 [Note] InnoDB: End of page dump
|
2022-08-30 8:40:25 1 [Note] InnoDB: You can use CHECK TABLE to scan your table for corruption. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery.
|
2022-08-30 8:40:25 1 [ERROR] InnoDB: Database page corruption on disk or a failed read of file './mysql/gtid_slave_pos.ibd' page [page id: space=20, page number=3]. You may have to recover from a backup.
|
2022-08-30 8:40:25 1 [Note] InnoDB: Page dump (16384 bytes):
|
2022-08-30 8:40:25 1 [Note] InnoDB: .............. [huge dump] ..............
|
(many of the same)
At some point it starts showing my own databases/tables:
2022-08-30 8:41:05 0 [ERROR] InnoDB: Failed to read page 4631 from file './my_db/my_table.ibd': Page read from tablespace is corrupted.
|
2022-08-30 8:41:05 0 [Note] InnoDB: [small dump]
|
2022-08-30 8:41:05 0 [Note] InnoDB: [small dump]
|
2022-08-30 8:41:05 0 [Note] InnoDB: [small dump]
|
2022-08-30 8:41:05 0 [ERROR] InnoDB: Database page corruption on disk or a failed read of file './my_db/my_table.ibd' page [page id: space=393, page number=5773]. You may have to recover from a backup.
|
2022-08-30 8:41:05 0 [Note] InnoDB: [small dump]
|
2022-08-30 8:41:05 0 [Note] InnoDB: Page dump (16384 bytes):
|
...
2022-08-30 8:41:05 0 [Note] InnoDB: End of page dump
|
2022-08-30 8:41:05 0 [Note] InnoDB: [small dump]
|
2022-08-30 8:41:05 0 [Note] InnoDB: [small dump]
|
2022-08-30 8:41:05 0 [Note] InnoDB: [small dump]
|
2022-08-30 8:41:05 0 [Note] InnoDB: [small dump]
|
2022-08-30 8:41:05 0 [Note] InnoDB: [small dump]
|
2022-08-30 8:41:05 0 [Note] InnoDB: [small dump]
|
2022-08-30 8:41:05 0 [Note] InnoDB: [small dump]
|
2022-08-30 8:41:05 0 [Note] InnoDB: [small dump]
|
2022-08-30 8:41:05 0 [Note] InnoDB: [small dump]
|
2022-08-30 8:41:05 0 [Note] InnoDB: [small dump]
|
2022-08-30 8:41:05 0 [Note] InnoDB: [small dump]
|
2022-08-30 8:41:05 0 [Note] InnoDB: [small dump]
|
2022-08-30 8:41:05 0 [Note] InnoDB: You can use CHECK TABLE to scan your table for corruption. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery.
|
2022-08-30 8:41:05 0 [Note] InnoDB: [small dump]
|
2022-08-30 8:41:05 0 [Note] InnoDB: [small dump]
|
2022-08-30 8:41:05 0 [ERROR] InnoDB: Failed to read page 5210 from file './my_db/my_table.ibd': Page read from tablespace is corrupted.
|
2022-08-30 8:41:05 0 [Note] InnoDB: [small dump]
|
2022-08-30 8:41:05 0 [Note] InnoDB: [small dump]
|
2022-08-30 8:41:05 0 [ERROR] InnoDB: Database page corruption on disk or a failed read of file './my_db/my_table.ibd' page [page id: space=393, page number=5774]. You may have to recover from a backup.
|
...
...
continues with a lot of my databases/tables
...
...
and then the final bit:
2022-08-30 11:07:20 0 [Note] InnoDB: End of page dump
|
2022-08-30 11:07:20 0 [Note] InnoDB: You can use CHECK TABLE to scan your table for corruption. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery.
|
220830 11:07:20 [ERROR] mysqld got signal 11 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
|
Server version: 10.8.4-MariaDB
|
key_buffer_size=134217728
|
read_buffer_size=131072
|
max_used_connections=2
|
max_threads=12
|
thread_count=2
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 919329 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
|
Thread pointer: 0x55ea6b2d75b8
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x7f9879e60be8 thread_stack 0x49000
|
addr2line: 'mysqld': No such file
|
Printing to addr2line failed
|
mysqld(my_print_stacktrace+0x2e)[0x55ea67ebfd9e]
|
mysqld(handle_fatal_signal+0x307)[0x55ea6792aef7]
|
sigaction.c:0(__restore_rt)[0x7f9db45f4630]
|
addr2line: 'mysqld': No such file
|
mysqld(+0xeab38c)[0x55ea67db138c]
|
mysqld(+0xeb8c09)[0x55ea67dbec09]
|
mysqld(+0xebafe5)[0x55ea67dc0fe5]
|
mysqld(+0xebbb66)[0x55ea67dc1b66]
|
mysqld(+0xeaf98c)[0x55ea67db598c]
|
mysqld(+0xdd6baf)[0x55ea67cdcbaf]
|
mysqld(+0xd91c08)[0x55ea67c97c08]
|
mysqld(+0xdf6480)[0x55ea67cfc480]
|
mysqld(_ZN5tpool10task_group7executeEPNS_4taskE+0xa6)[0x55ea67e4a536]
|
mysqld(_ZN5tpool19thread_pool_generic11worker_mainEPNS_11worker_dataE+0x61)[0x55ea67e48ee1]
|
??:0(std::this_thread::__sleep_for(std::chrono::duration<long, std::ratio<1l, 1l> >, std::chrono::duration<long, std::ratio<1l, 1000000000l> >))[0x7f9db418e330]
|
pthread_create.c:0(start_thread)[0x7f9db45ecea5]
|
??:0(__clone)[0x7f9db3b07b0d]
|
|
|
Trying to get some variables.
|
Some pointers may be invalid and cause the dump to abort.
|
Query (0x0): (null)
|
Connection ID (thread ID): 0
|
Status: NOT_KILLED
|
|
|
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off
|
|
|
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
|
information that should help you find out what is causing the crash.
|
|
|
We think the query pointer is invalid, but we will try to print it anyway.
|
Query:
|
|
|
Writing a core file...
|
Working directory at /var/lib/mysql
|
Resource Limits:
|
Limit Soft Limit Hard Limit Units
|
Max cpu time unlimited unlimited seconds
|
Max file size unlimited unlimited bytes
|
Max data size unlimited unlimited bytes
|
Max stack size 8388608 unlimited bytes
|
Max core file size unlimited unlimited bytes
|
Max resident set unlimited unlimited bytes
|
Max processes unlimited unlimited processes
|
Max open files 1048576 1048576 files
|
Max locked memory 65536 65536 bytes
|
Max address space unlimited unlimited bytes
|
Max file locks unlimited unlimited locks
|
Max pending signals 127822 127822 signals
|
Max msgqueue size 819200 819200 bytes
|
Max nice priority 0 0
|
Max realtime priority 0 0
|
Max realtime timeout unlimited unlimited us
|
Core pattern: |/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t e %P %I %h
|
|
|
Kernel version: Linux version 3.10.0-1160.62.1.el7.x86_64 (mockbuild@kbuilder.bsys.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Tue Apr 5 16:57:59 UTC 2022
|
Do you think this could be caused by MDEV-29383?
Thanks.
Today's backup, however, seems to have gone well. I have a stable Snapshot, finally.
(on other days, it's MDEV-28974 that keeps affecting my daily snapshots)
It's really hard to get valid backups...
.
.
Attachments
Issue Links
- duplicates
-
-
- Closed
-
Activity
My attempt to find and download the correct package files into Debian apparently failed. I can resolve the short addresses, but they do not make any sense, that is, they were resolving to unrelated functions that are not calling each other.
Here is what I did:
- Download MariaDB-server-10.8.4-1.el8.x86_64.rpm from one of the mirrors that mariadb.org points to.
- Extract the server executable usr/sbin/mariadbd, for example, using the GNOME file-roller.
- Download MariaDB-server-debuginfo-10.8.4-1.el8.x86_64.rpm from the same mirror.
- Extract all of it to the current directory.
- Invoke the debugger:
gdb --ex 'set debug-file-directory usr/lib/debug' --ex 'file mariadbd' - Inside the debugger, type commands like list *0xdf6480 to resolve the addresses one at a time.
I repeated the exercise with MariaDB-server-10.8.4-1.el7.x86_64.rpm and the corresponding debuginfo. The stack traces are suddenly making some sense:
|
mariadb-10.8.4 |
dict_sys_t::remove(dict_table_t*, bool, bool) (/usr/src/debug/MariaDB-/src_0/storage/innobase/dict/dict0dict.cc:1885)
|
dict_load_table_one(st_::span<char const> const&, dict_err_ignore_t, dict_names_t&) (/usr/src/debug/MariaDB-/src_0/storage/innobase/include/mem0mem.inl:407)
|
dict_sys_t::load_table(st_::span<char const> const&, dict_err_ignore_t) (/usr/src/debug/MariaDB-/src_0/storage/innobase/dict/dict0load.cc:2495)
|
dict_load_table_on_id(unsigned long, dict_err_ignore_t) (/usr/src/debug/MariaDB-/src_0/storage/innobase/dict/dict0load.cc:2571)
|
dict_table_open_on_id<true>(unsigned long, bool, dict_table_op_t, THD*, MDL_ticket**) (/usr/src/debug/MariaDB-/src_0/storage/innobase/dict/dict0dict.cc:864)
|
row_purge_step(que_thr_t*) (/usr/src/debug/MariaDB-/src_0/storage/innobase/row/row0purge.cc:944)
|
que_run_threads(que_thr_t*) (/usr/src/debug/MariaDB-/src_0/storage/innobase/que/que0que.cc:653)
|
purge_worker_callback(void*) (/usr/src/debug/MariaDB-/src_0/include/mysql/psi/mysql_thread.h:745).
|
Yes, sometimes the current instruction in a stack frame is referring to some inlined code. The important part here is that dict_load_table_one() was invoking dict_sys_t::remove() to evict the definition of a corrupted table. Let us check the relevant part of the output of disassemble dict_load_table_one in GDB:
0x0000000000eb8bec <+3084>: je 0xeb8ef0 <dict_load_table_one(st_::span<char const> const&, dict_err_ignore_t, dict_names_t&)+3856>
|
0x0000000000eb8bf2 <+3090>: mov -0x1c00(%rbp),%rsi
|
0x0000000000eb8bf9 <+3097>: lea 0xb88200(%rip),%rdi # 0x1a40e00 <dict_sys>
|
0x0000000000eb8c00 <+3104>: xor %ecx,%ecx
|
0x0000000000eb8c02 <+3106>: xor %edx,%edx
|
0x0000000000eb8c04 <+3108>: call 0xeab250 <_ZN10dict_sys_t6removeEP12dict_table_tbb>
|
0x0000000000eb8c09 <+3113>: mov -0x1c28(%rbp),%rax
|
Because we have the debug information, we can check which code is associated with that preceding conditional branch (je):
(gdb) list *0xeb8bec
|
0xeb8bec is in dict_load_table_one(st_::span<char const> const&, dict_err_ignore_t, dict_names_t&) (/usr/src/debug/MariaDB-/src_0/storage/innobase/dict/dict0load.cc:2348).
|
2343
|
2344 mem_heap_t* heap = mem_heap_create(32000);
|
2345
|
2346 dict_load_tablespace(table, ignore_err);
|
2347
|
2348 if (dict_load_columns(table, heap) || dict_load_virtual(table)) {
|
2349 evict:
|
2350 dict_sys.remove(table);
|
2351 mem_heap_free(heap);
|
2352 DBUG_RETURN(nullptr);
|
At this point, the table had not been added to dict_sys yet, and therefore the call dict_sys.remove(table) is incorrect. Instead, we would need dict_mem_table_free(). This is something that was recently fixed as part of MDEV-29440.
Hey marko
Strange that with el7 you find a match that makes more sense.
I'm definitely using the el8 release, since I'm on AlmaLinux 8.6.
# yum list installed | grep maria
|
MariaDB-backup.x86_64 10.8.4-1.el8 @mariadb-main
|
MariaDB-client.x86_64 10.8.4-1.el8 @mariadb-main
|
MariaDB-common.x86_64 10.8.4-1.el8 @mariadb-main
|
MariaDB-devel.x86_64 10.8.4-1.el8 @mariadb-main
|
MariaDB-server.x86_64 10.8.4-1.el8 @mariadb-main
|
MariaDB-shared.x86_64 10.8.4-1.el8 @mariadb-main
|
galera-4.x86_64 26.4.12-1.el8 @mariadb-main
|
|
|
# yum info mariadb
|
Name : MariaDB
|
Version : 10.8.4
|
Release : 1.el8
|
Architecture : src
|
Size : 91 M
|
Source : None
|
Repository : mariadb-main
|
...
|
marko ignore what I said above. The crash happened on the Docker container which runs CentOS 7. So el7 is the right package to look at! Sorry.
nunop, I am glad that you were able to tweak something that works for you until 10.8.5 is available.
When it comes to this bug report, I think that this is about avoiding a server crash when the data is corrupted in a particular way. To fix that, I would need to resolve the following stack trace that you posted earlier:
mysqld(+0xeab38c)[0x55ea67db138c]mysqld(+0xeb8c09)[0x55ea67dbec09]mysqld(+0xebafe5)[0x55ea67dc0fe5]mysqld(+0xebbb66)[0x55ea67dc1b66]mysqld(+0xeaf98c)[0x55ea67db598c]mysqld(+0xdd6baf)[0x55ea67cdcbaf]mysqld(+0xd91c08)[0x55ea67c97c08]mysqld(+0xdf6480)[0x55ea67cfc480]mysqld(_ZN5tpool10task_group7executeEPNS_4taskE+0xa6)[0x55ea67e4a536]I will download the package that you used, and try to resolve this manually.