[MDEV-19711] Assertion failure during IMPORT TABLESPACE Created: 2019-06-07  Updated: 2022-06-08  Resolved: 2022-06-07

Status: Closed
Project: MariaDB Server
Component/s: Data Definition - Alter Table, Storage Engine - InnoDB
Affects Version/s: 10.1.40
Fix Version/s: 10.6.9, 10.7.5, 10.8.4, 10.9.2

Type: Bug Priority: Major
Reporter: Markus Nägele Assignee: Marko Mäkelä
Resolution: Fixed Votes: 0
Labels: None
Environment:

CentOS Linux release 7.6.1810 (Core)
Dell PowerEdge R740 384 GB RAM


Issue Links:
Relates
relates to MDEV-13542 Crashing on a corrupted page is unhel... Closed
relates to MDEV-14170 InnoDB: Failing assertion: sibling_mo... Closed

 Description   

MariaDB is running into an assertion failure, when trying to import a tablespace from another MariaDB instance. The steps are as followed:

  • I have a fresh installation of MariaDB and create this table:

CREATE TABLE `jobdata` (
	`OID` BIGINT(20) NOT NULL AUTO_INCREMENT,
	`IDJOB` INT(11) NOT NULL,
	`IDATTR` INT(11) NOT NULL,
	`SIDE` CHAR(1) NOT NULL,
	`VALUE` VARCHAR(100) NULL DEFAULT NULL,
	`BIGVALUE` LONGTEXT NULL,
	`OptimisticLockField` INT(11) NULL DEFAULT NULL,
	`GCRecord` INT(11) NULL DEFAULT NULL,
	`DVALUE` DOUBLE NULL DEFAULT NULL,
	PRIMARY KEY (`OID`),
	INDEX `iDValue` (`IDATTR`, `SIDE`, `DVALUE`),
	INDEX `Search` (`IDJOB`, `IDATTR`, `SIDE`, `VALUE`),
	INDEX `iValue` (`IDATTR`, `SIDE`, `VALUE`),
	INDEX `iIDJOB_JOBDATA` (`IDJOB`),
	INDEX `iIDATTR_JOBDATA` (`IDATTR`)
)
COLLATE='utf8_general_ci'
ENGINE=InnoDB;

  • Discard the tablespace:

    ALTER TABLE jobdata DISCARD TABLESPACE; 
    

  • Copy the jobdata.ibd from a dfferent instance (same definition as above of course) into the data directory
  • Trying to import the tablespace:

    ALTER TABLE jobdata IMPORT TABLESPACE; 
    

    After this statement the system is quite busy for about 2 minutes. Then the deamon restarts itself with the follwoing log messages:

2019-06-07 17:11:05 140326383437568 [Note] InnoDB: Sync to disk
2019-06-07 17:11:15 140326383437568 [Note] InnoDB: Sync to disk - done!
2019-06-07 17:11:15 140326383437568 [Note] InnoDB: Phase I - Update all pages
2019-06-07 17:12:13 140326383437568 [Note] InnoDB: Sync to disk
2019-06-07 17:12:22 140326383437568 [Note] InnoDB: Sync to disk - done!
2019-06-07 17:12:22 140326383437568 [Note] InnoDB: Phase II - Purge records from index PRIMARY
2019-06-07 17:12:31 7fa0483cc700  InnoDB: Assertion failure in thread 140326383437568 in file btr0pcur.cc line 458
InnoDB: Failing assertion: btr_page_get_prev(next_page, mtr) == buf_block_get_page_no(btr_pcur_get_block(cursor))
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
190607 17:12:31 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
 
Server version: 10.1.40-MariaDB
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=4
max_threads=2002
thread_count=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 4528960 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x7f4707246008
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7fa0483cbcb0 thread_stack 0x49000
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x56385819c4ee]
/usr/sbin/mysqld(handle_fatal_signal+0x305)[0x563857cbfe15]
sigaction.c:0(__restore_rt)[0x7fa0480ff5d0]
:0(__GI_raise)[0x7fa04619e207]
:0(__GI_abort)[0x7fa04619f8f8]
/usr/sbin/mysqld(+0x99afc0)[0x563858096fc0]
/usr/sbin/mysqld(+0x8f825e)[0x563857ff425e]
/usr/sbin/mysqld(+0x8f853b)[0x563857ff453b]
/usr/sbin/mysqld(+0x8faa1d)[0x563857ff6a1d]
/usr/sbin/mysqld(+0x85f0a2)[0x563857f5b0a2]
/usr/sbin/mysqld(_Z34mysql_discard_or_import_tablespaceP3THDP10TABLE_LISTb+0xe7)[0x563857bbe9e7]
/usr/sbin/mysqld(_ZN33Sql_cmd_discard_import_tablespace7executeEP3THD+0xad)[0x563857c0da0d]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x3bc9)[0x563857b33b09]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x342)[0x563857b39342]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x26f9)[0x563857b3cd19]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x177)[0x563857b3d6a7]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x18a)[0x563857c0b4fa]
/usr/sbin/mysqld(handle_one_connection+0x40)[0x563857c0b6a0]
pthread_create.c:0(start_thread)[0x7fa0480f7dd5]
/lib64/libc.so.6(clone+0x6d)[0x7fa046265ead]
 
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7f470a023020): is an invalid pointer
Connection ID (thread ID): 8
Status: NOT_KILLED
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=off
 
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
 
We think the query pointer is invalid, but we will try to print it anyway.
Query: ALTER TABLE jobdata import TABLESPACE

Additional information:

  • The jobdata.ibd is quite big, about 20 GB
  • The following options are set in /etc/my.cnf.d/server.cnf

    [mysqld]
    datadir=/var/lib/mysql
    socket=/var/lib/mysql/mysql.sock
    user=mysql
    skip-name-resolve
    character-set-server=utf8mb4
    collation-server=utf8mb4_unicode_ci
    log-bin=/var/log/mysql/mysql-bin
    binlog-format=MIXED
    sync-binlog=2
    log-slave-updates
    expire-logs-days=5
    server-id=3
    max_connections = 2000
    lower_case_table_names=1
    query_cache_size = 0
    query_cache_type = 0
    tmp_table_size = 128M
    max_heap_table_size = 64M
    thread_cache_size = 4
    table_cache = 512
    innodb_stats_on_metadata = 0
    max_allowed_packet = 32M
    slow_query_log = ON
    innodb_log_file_size = 100M
    innodb_buffer_pool_size = 329G
    

  • I also tried with an innodb_buffer_pool_size of 16G with the same result.
  • The ibd that shall be imported comes from an instance with MariaDB 10.1.21

What could cause this failure?



 Comments   
Comment by Marko Mäkelä [ 2020-10-23 ]

Sorry, I noticed this report only now. InnoDB is not very good at handling corruption. Especially IMPORT TABLESPACE and CHECK TABLE should really be more resilient and kind-of expect the input to be corrupted.

If the file jobdata.ibd was copied from a different server without any precautions (such as between a completed FLUSH TABLE jobdata FOR EXPORT; and UNLOCK TABLES;), then it can be corrupted to begin with.

Comment by Marko Mäkelä [ 2022-05-16 ]

It looks possible that this type of crash will be fixed by MDEV-13542.

Comment by Marko Mäkelä [ 2022-06-07 ]

This crash on corrupted data was fixed by MDEV-13542.

Generated at Thu Feb 08 08:53:46 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.