Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Fixed
-
10.6.15, 10.4(EOL), 10.5, 10.6, 10.7(EOL), 10.8(EOL), 10.9(EOL), 10.10(EOL), 10.11, 11.0(EOL), 11.1(EOL), 11.2(EOL), 11.3(EOL), 11.4
Description
We utilize mariabackup for partial backup and restore operations. However, our automated restore tests have encountered failures at times due to corrupted indexes. After some investigation, I have isolated the issue. It appears not to be a problem with mariabackup or the backup itself, as I was able to reproduce the issue on a local instance as well.
#1 Test Case:
Backup the database files:
rsync -av /var/lib/mysql/db1 /var/tmp/
|
Recreate database db1:
DROP DATABASE db1;
|
CREATE DATABASE db1;
|
Create the database schema without secondary indexes (excerpt):
CREATE TABLE `questionnaire` (
|
`id` varchar(64) NOT NULL,
|
.....
|
`created` datetime DEFAULT NULL,
|
`updated` datetime DEFAULT NULL,
|
`deleted` tinyint(1) unsigned NOT NULL DEFAULT 0,
|
PRIMARY KEY (`id`)
|
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
|
ALTER TABLE `questionnaire` DISCARD TABLESPACE;
|
cp -p /var/tmp/db1/*.ibd /var/lib/mysql/db1/
|
ALTER TABLE `questionnaire` IMPORT TABLESPACE;
|
Check database integrity before restart:
mysqlcheck db1 questionnaire
|
|
|
db1.questionnaire: OK
|
Check database integrity after restart:
mysqlcheck db1 questionnaire
|
db1.questionnaire: Warning: InnoDB: Index PRIMARY is marked as corrupted
|
error: Corrupt
|
#2 Test Case:
In this case, I considered what would happen if I removed the PRIMARY KEY from the table definition. The steps are the same as above, but without the PRIMARY KEY (id).
This led to a crash:
2024-02-21 10:22:28 37 [ERROR] InnoDB: Trying to read 16384 bytes at 15225848610816 outside the bounds of the file: ./db1/questionnaire.ibd
|
2024-02-21 10:22:28 37 [ERROR] InnoDB: File './db1/questionnaire.ibd' is corrupted
|
2024-02-21 10:22:28 37 [Note] InnoDB: Phase III - Flush changes to disk
|
2024-02-21 10:22:28 37 [Note] InnoDB: Phase IV - Flush complete
|
2024-02-21 10:22:28 37 [ERROR] InnoDB: Trying to read 16384 bytes at 13850385334272 outside the bounds of the file: ./db1/questionnaire.ibd
|
2024-02-21 10:22:28 37 [ERROR] InnoDB: Trying to read 16384 bytes at 13850385334272 outside the bounds of the file: ./db1/questionnaire.ibd
|
240221 10:22:28 [ERROR] mysqld got signal 8 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
|
Server version: 10.6.15-MariaDB-1:10.6.15+maria~deb11 source revision: 0d16eb35bc981023ce2f4912e8ecde68ca381f4e
|
key_buffer_size=134217728
|
read_buffer_size=131072
|
max_used_connections=1
|
max_threads=153
|
thread_count=1
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467985 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
|
Thread pointer: 0x7f2cd8000c58
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x7f2d67ffecd8 thread_stack 0x49000
|
Printing to addr2line failed
|
/usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x55846a04f2ee]
|
/usr/sbin/mariadbd(handle_fatal_signal+0x485)[0x558469b14825]
|
/lib/x86_64-linux-gnu/libpthread.so.0(+0x13140)[0x7f2d6e1c0140]
|
/usr/sbin/mariadbd(+0xe664f3)[0x558469f844f3]
|
/usr/sbin/mariadbd(+0xe6845b)[0x558469f8645b]
|
/usr/sbin/mariadbd(+0xcfb233)[0x558469e19233]
|
/usr/sbin/mariadbd(_Z34mysql_discard_or_import_tablespaceP3THDP10TABLE_LISTb+0xf9)[0x5584699787f9]
|
/usr/sbin/mariadbd(_ZN33Sql_cmd_discard_import_tablespace7executeEP3THD+0xac)[0x5584699f0fec]
|
/usr/sbin/mariadbd(_Z21mysql_execute_commandP3THDb+0x4a7e)[0x5584698d7b3e]
|
/usr/sbin/mariadbd(_Z11mysql_parseP3THDPcjP12Parser_state+0x1e7)[0x5584698d8d67]
|
/usr/sbin/mariadbd(_Z16dispatch_command19enum_server_commandP3THDPcjb+0x156d)[0x5584698db5fd]
|
/usr/sbin/mariadbd(_Z10do_commandP3THDb+0x138)[0x5584698dd328]
|
/usr/sbin/mariadbd(_Z24do_handle_one_connectionP7CONNECTb+0x3af)[0x5584699ec9af]
|
/usr/sbin/mariadbd(handle_one_connection+0x5d)[0x5584699eccfd]
|
/usr/sbin/mariadbd(+0xc43ce2)[0x558469d61ce2]
|
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7ea7)[0x7f2d6e1b4ea7]
|
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f2d6ddbba2f]
|
|
|
Trying to get some variables.
|
Some pointers may be invalid and cause the dump to abort.
|
Query (0x7f2cd8010be0): ALTER TABLE `questionnaire` IMPORT TABLESPACE
|
|
|
Connection ID (thread ID): 37
|
Status: NOT_KILLED
|
|
|
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off,hash_join_cardinality=off
|
|
|
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
|
information that should help you find out what is causing the crash.
|
Writing a core file...
|
Working directory at /var/lib/mysql
|
Resource Limits:
|
Limit Soft Limit Hard Limit Units
|
Max cpu time unlimited unlimited seconds
|
Max file size unlimited unlimited bytes
|
Max data size unlimited unlimited bytes
|
Max stack size 8388608 unlimited bytes
|
Max core file size unlimited unlimited bytes
|
Max resident set unlimited unlimited bytes
|
Max processes unlimited unlimited processes
|
Max open files 1073741816 1073741816 files
|
Max locked memory 8388608 8388608 bytes
|
Max address space unlimited unlimited bytes
|
Max file locks unlimited unlimited locks
|
Max pending signals 126657 126657 signals
|
Max msgqueue size 819200 819200 bytes
|
Max nice priority 0 0
|
Max realtime priority 0 0
|
Max realtime timeout unlimited unlimited us
|
Core pattern: |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h
|
|
|
Kernel version: Linux version 6.7.4-200.fc39.x86_64 (mockbuild@de0c58eb5f524c20963d3b29334043cc) (gcc (GCC) 13.2.1 20231205 (Red Hat 13.2.1-6), GNU ld version 2.40-14.fc39) #1 SMP PREEMPT_DYNAMIC Mon Feb 5 22:21:14 UTC 2024
|
Attachments
Issue Links
- relates to
-
-
- Closed
-
Activity
thanks for approving the changes on github. What does the "Stalled" status here mean?
The "stalled" ticket state in this system can mean almost anything. The states do not even distinguish between "review approved" and "review rejected". In this case, we thought that the fix can be applied without subjecting it to additional stress testing.
Thanks @Marko.
Will there also be a fix for our affecting version 10.6. (our main problem exists on that version)?
Yes, the 10.4 branch will be merged to all other branches before the upcoming maintenance releases, which should be out in a couple of weeks.
The fix of
MDEV-18543was incomplete. I can reproduce this with the following change to an existing test:diff --git a/mysql-test/suite/innodb/t/instant_alter_import.test b/mysql-test/suite/innodb/t/instant_alter_import.testindex 7bd5645436c..50eda83f4b0 100644--- a/mysql-test/suite/innodb/t/instant_alter_import.test+++ b/mysql-test/suite/innodb/t/instant_alter_import.test@@ -79,6 +79,7 @@ flush tables t2 for export;unlock tables;alter table t1 import tablespace;+--source include/restart_mysqld.incselect * from t1;10.4 662bb176b412993a085fe329af559ddc3dc83ec3 with the above change
CURRENT_TEST: innodb.instant_alter_importmysqltest: At line 83: query 'select * from t1' failed: 2013: Lost connection to MySQL server during query…mysqld: /mariadb/10.4/storage/innobase/include/dict0mem.h:2373: bool dict_index_t::is_instant() const: Assertion `n_core_fields <= n_fields || table->n_dropped()' failed.I filed https://github.com/MariaDB/server/pull/3190 for the fix.