[MDEV-21407] Crash when restarting server after IMPORT TABLESPACE Created: 2019-12-30  Updated: 2020-05-04  Resolved: 2020-04-15

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - InnoDB
Affects Version/s: 10.1, 10.4.11, 10.2, 10.3, 10.4
Fix Version/s: 10.1.45, 10.2.32, 10.3.23, 10.5.3

Type: Bug Priority: Major
Reporter: Michael Chang Assignee: Marko Mäkelä
Resolution: Duplicate Votes: 5
Labels: None
Environment:

Tested on Arch Linux and Ubuntu 18.04 with MariaDB repo

Attachments: File mariadb.log    
Issue Links:
Blocks
blocks MDEV-18762 Support easy restore of partial backup Closed
Duplicate
duplicates MDEV-21422 InnoDB: Assertion failure in file /ho... Closed
Relates
relates to MDEV-21549 IMPORT TABLESPACE fails to adjust all... Closed
relates to MDEV-22431 IMPORT TABLESPACE, service restart Closed

 Description   

After importing a tablespace (created with mariabackup --prepare --export or FLUSH TABLE ... FOR EXPORT), if the server is restarted before writing to the new table, reading the table crashes the server.

Started with empty datadir, nothing set in my.cnf.

mariadb> create database test;
 
Query OK, 1 row affected (0.001 sec)
 
mariadb> create database test2;
 
Query OK, 1 row affected (0.000 sec)
 
mariadb> create table test.foo(id int not null primary key);
 
Query OK, 0 rows affected (0.185 sec)
 
mariadb> create table test2.foo(id int not null primary key);
 
Query OK, 0 rows affected (0.143 sec)
 
mariadb> insert into test.foo values (1);
 
Query OK, 1 row affected (0.031 sec)
 
mariadb> alter table test2.foo discard tablespace;
 
Query OK, 0 rows affected (0.045 sec)
 
mariadb> flush table test.foo for export;
 
Query OK, 0 rows affected (0.001 sec)
 
-- Copy test/foo.cfg and test/foo.ibd to test2/
 
mariadb> unlock tables;
 
Query OK, 0 rows affected (0.000 sec)
 
mariadb> alter table test2.foo import tablespace;
 
Query OK, 0 rows affected (0.200 sec)
 
mariadb> select * from test2.foo;
 
+----+
 
| id |
 
+----+
 
|  1 |
 
+----+
 
1 row in set (0.001 sec)
 
mariadb> exit
 
-- Restart MariaDB server
 
mariadb> select * from test2.foo;
 
ERROR 2013 (HY000): Lost connection to MySQL server during query

The server log (attached) shows:

2019-12-30  9:12:54 0 [Note] /usr/bin/mysqld: ready for connections.
 
Version: '10.4.11-MariaDB'  socket: '/run/mysqld/mysqld.sock'  port: 3306  Arch Linux
 
2019-12-30 09:13:02 0x7f91f41ae700  InnoDB: Assertion failure in file /build/mariadb/src/mariadb-10.4.11/storage/innobase/btr/btr0btr.cc line 204
 
InnoDB: Failing assertion: mach_read_from_4(seg_header + FSEG_HDR_SPACE) == space

It seems like doing any write to the new table prevents the issue.

 Comments   
Comment by Alice Sherepa [ 2020-01-07 ]

Thank you for the report! Reproducible with MariaDB 10.2-10.4:

--source include/have_innodb.inc
 
 
 
 create database test1;
 
 create database test2;
 
 create table test1.foo(id int not null primary key) engine=innodb;
 
 create table test2.foo(id int not null primary key) engine=innodb;
 
 insert into test1.foo values (1);
 
 alter table test2.foo discard tablespace;
 
 flush table test1.foo for export;
 
 
 
  --let $datadir = `SELECT @@datadir`
 
 
 
  --replace_result $datadir <DATADIR>
 
  --copy_file $datadir/test1/foo.ibd $datadir/test2/foo.ibd
 
  --copy_file $datadir/test1/foo.cfg $datadir/test2/foo.cfg
 
 
 
unlock tables;
 
 
 
alter table test2.foo import tablespace;
 
select * from test2.foo;
 
 
 
--source include/restart_mysqld.inc
 
select * from test2.foo;
 
 
 
drop database test1;
 
drop database test2;

10.2 0b4ae6724f2707841c50f88d

 
#4  0x00007f136e157428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
 
#5  0x00007f136e15902a in __GI_abort () at abort.c:89
 
#6  0x0000560fd2ab939e in ut_dbg_assertion_failed (expr=0x560fd2fdd988 "mach_read_from_4(seg_header + FSEG_HDR_SPACE) == space", file=0x560fd2fdd950 "/10.2/storage/innobase/btr/btr0btr.cc", line=205) at /10.2/storage/innobase/ut/ut0dbg.cc:60
 
#7  0x0000560fd2ac29c4 in btr_root_fseg_validate (seg_header=0x7f1362efc04a "", space=5) at /10.2/storage/innobase/btr/btr0btr.cc:205
 
#8  0x0000560fd2ac2bc2 in btr_root_block_get (index=0x7f1310135558, mode=4, mtr=0x7f13601a9be0) at /10.2/storage/innobase/btr/btr0btr.cc:251
 
#9  0x0000560fd2ac2c5e in btr_root_get (index=0x7f1310135558, mtr=0x7f13601a9be0) at /10.2/storage/innobase/btr/btr0btr.cc:274
 
#10 0x0000560fd2ac3613 in btr_get_size (index=0x7f1310135558, flag=2, mtr=0x7f13601a9be0) at /10.2/storage/innobase/btr/btr0btr.cc:615
 
#11 0x0000560fd2b85351 in dict_stats_update_transient_for_index (index=0x7f1310135558) at /10.2/storage/innobase/dict/dict0stats.cc:863
 
#12 0x0000560fd2b8563e in dict_stats_update_transient (table=0x7f1310137fd8) at /10.2/storage/innobase/dict/dict0stats.cc:950
 
#13 0x0000560fd2b8aabd in dict_stats_update (table=0x7f1310137fd8, stats_upd_option=DICT_STATS_RECALC_TRANSIENT) at /10.2/storage/innobase/dict/dict0stats.cc:3349
 
#14 0x0000560fd28a0b8a in dict_stats_init (table=0x7f1310137fd8) at /10.2/storage/innobase/include/dict0stats.ic:165
 
#15 0x0000560fd28aa5b7 in ha_innobase::open (this=0x7f1310133be8, name=0x7f1310137c20 "./test2/foo") at /10.2/storage/innobase/handler/ha_innodb.cc:6258
 
#16 0x0000560fd256b472 in handler::ha_open (this=0x7f1310133be8, table_arg=0x7f1310132fe0, name=0x7f1310137c20 "./test2/foo", mode=2, test_if_locked=18) at /10.2/sql/handler.cc:2585
 
#17 0x0000560fd23ed3ea in open_table_from_share (thd=0x7f1310000af0, share=0x7f1310137708, alias=0x7f1310014730 "foo", db_stat=33, prgflag=8, ha_open_flags=18, outparam=0x7f1310132fe0, is_create_table=false) at /10.2/sql/table.cc:3389
 
#18 0x0000560fd2270187 in open_table (thd=0x7f1310000af0, table_list=0x7f1310014738, ot_ctx=0x7f13601aaed0) at /10.2/sql/sql_base.cc:1931
 
#19 0x0000560fd227304a in open_and_process_table (thd=0x7f1310000af0, tables=0x7f1310014738, counter=0x7f13601aaf64, flags=0, prelocking_strategy=0x7f13601aafe0, has_prelocking_list=false, ot_ctx=0x7f13601aaed0) at /10.2/sql/sql_base.cc:3566
 
#20 0x0000560fd2273f3a in open_tables (thd=0x7f1310000af0, options=..., start=0x7f13601aaf48, counter=0x7f13601aaf64, flags=0, prelocking_strategy=0x7f13601aafe0) at /10.2/sql/sql_base.cc:4035
 
#21 0x0000560fd2275738 in open_and_lock_tables (thd=0x7f1310000af0, options=..., tables=0x7f1310014738, derived=true, flags=0, prelocking_strategy=0x7f13601aafe0) at /10.2/sql/sql_base.cc:4793
 
#22 0x0000560fd2268502 in open_and_lock_tables (thd=0x7f1310000af0, tables=0x7f1310014738, derived=true, flags=0) at /10.2/sql/sql_base.h:509
 
#23 0x0000560fd22eccfe in execute_sqlcom_select (thd=0x7f1310000af0, all_tables=0x7f1310014738) at /10.2/sql/sql_parse.cc:6146
 
#24 0x0000560fd22e3ba6 in mysql_execute_command (thd=0x7f1310000af0) at /10.2/sql/sql_parse.cc:3532
 
#25 0x0000560fd22f1035 in mysql_parse (thd=0x7f1310000af0, rawbuf=0x7f1310014548 "select * from test2.foo", length=23, parser_state=0x7f13601ac080, is_com_multi=false, is_next_command=false) at /10.2/sql/sql_parse.cc:7740
 
#26 0x0000560fd22df30a in dispatch_command (command=COM_QUERY, thd=0x7f1310000af0, packet=0x7f13100088a1 "select * from test2.foo", packet_length=23, is_com_multi=false, is_next_command=false) at /10.2/sql/sql_parse.cc:1831
 
#27 0x0000560fd22ddc1e in do_command (thd=0x7f1310000af0) at /10.2/sql/sql_parse.cc:1385
 
#28 0x0000560fd24347c9 in do_handle_one_connection (connect=0x560fd5eb1ac0) at /10.2/sql/sql_connect.cc:1336
 
#29 0x0000560fd2434534 in handle_one_connection (arg=0x560fd5eb1ac0) at /10.2/sql/sql_connect.cc:1241
 
#30 0x0000560fd2c5dedc in pfs_spawn_thread (arg=0x560fd5f73720) at /10.2/storage/perfschema/pfs.cc:1862
 
#31 0x00007f136ed946ba in start_thread (arg=0x7f13601ad700) at pthread_create.c:333
 
#32 0x00007f136e22941d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Comment by Alice Sherepa [ 2020-01-08 ] 

./mtr encryption.innodb-bad-key-change --mysqld=--innodb-checksum-algorithm=none

10.1 cb204e11eaf4c473ce5d5a10a21de1

 
Version: '10.1.44-MariaDB-debug' 
2020-01-08 12:24:27 7f0b230b6700  InnoDB: Assertion failure in thread 139685809317632 in file btr0btr.cc line 708
 
InnoDB: Failing assertion: mach_read_from_4(seg_header + FSEG_HDR_SPACE) == space
 
InnoDB: We intentionally generate a memory trap.
 
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
 
InnoDB: If you get repeated assertion failures or crashes, even
 
InnoDB: immediately after the mysqld startup, there may be
 
InnoDB: corruption in the InnoDB tablespace. Please refer to
 
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
 
InnoDB: about forcing recovery.
 
200108 12:24:27 [ERROR] mysqld got signal 6 ;
 
 
 
#4  0x00007f0b20c38428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
 
#5  0x00007f0b20c3a02a in __GI_abort () at abort.c:89
 
#6  0x00007f0b191e3a16 in btr_root_fseg_validate (seg_header=0x7f0b16eb804a "\374\200\204\240?ZU\267\"\034\063E\341\252\312\360BOg\245\252@\032x=99\225\245\202\021\262x\001\324\351x\376P\233\317\215\067\366\207W\031?\036r\274@\345K\a\240\354v{9\217s-K\270x\342'\202\002\346\066\354\313{TQ\203\061k\335\rk\005\236\304v\004mx!\317\060\367\bO\320\307=\277\tK\343O.\306M*\312\341\307\232\327\266\337\255\316op\300\003\330\347X/\270\253\036z\325G\265\060\340\024>\273\317H\352j\225\310\230(\367\n\307~S~\246s\340V\024\240\353wz\365\221l\336\036\272W?\341\005\351\005g\234`F\363X\227\320\325\372\323\251\213x\304\230}\255\366\375\376\372\346\070+\360\021z"..., space=7) at /git/10.1/storage/innobase/btr/btr0btr.cc:708
 
#7  0x00007f0b191e3bf7 in btr_root_block_get (index=0x7f0b09d0d678, mode=2, mtr=0x7f0b230b2780) at /git/10.1/storage/innobase/btr/btr0btr.cc:759
 
#8  0x00007f0b191e3c8b in btr_root_get (index=0x7f0b09d0d678, mtr=0x7f0b230b2780) at /git/10.1/storage/innobase/btr/btr0btr.cc:780
 
#9  0x00007f0b191e4a97 in btr_get_size_and_reserved (index=0x7f0b09d0d678, flag=2, used=0x7f0b230b2748, mtr=0x7f0b230b2780) at /git/10.1/storage/innobase/btr/btr0btr.cc:1281
 
#10 0x00007f0b191e498c in btr_get_size (index=0x7f0b09d0d678, flag=2, mtr=0x7f0b230b2780) at /git/10.1/storage/innobase/btr/btr0btr.cc:1245
 
#11 0x00007f0b1927ef6d in dict_stats_update_transient_for_index (index=0x7f0b09d0d678) at /git/10.1/storage/innobase/dict/dict0stats.cc:902
 
#12 0x00007f0b1927f1d6 in dict_stats_update_transient (table=0x7f0b09cf4378) at /git/10.1/storage/innobase/dict/dict0stats.cc:988
 
#13 0x00007f0b19283dea in dict_stats_update (table=0x7f0b09cf4378, stats_upd_option=DICT_STATS_RECALC_TRANSIENT) at /git/10.1/storage/innobase/dict/dict0stats.cc:3473
 
#14 0x00007f0b192e69a0 in dict_stats_init (table=0x7f0b09cf4378) at /git/10.1/storage/innobase/include/dict0stats.ic:174
 
#15 0x00007f0b192ed9b2 in ha_innodb::open (this=0x7f0b09c8c088, name=0x7f0b09cb0960 "./test/t2", mode=2, test_if_locked=18) at /git/10.1/storage/innobase/handler/ha_innodb.cc:5947
 
#16 0x00005566fd18108a in handler::ha_open (this=0x7f0b09c8c088, table_arg=0x7f0b09cd1e70, name=0x7f0b09cb0960 "./test/t2", mode=2, test_if_locked=18) at /git/10.1/sql/handler.cc:2569
 
#17 0x00005566fd02bb39 in open_table_from_share (thd=0x7f0b16764070, share=0x7f0b09cb0488, alias=0x7f0b09c42258 "t2", db_stat=39, prgflag=44, ha_open_flags=18, outparam=0x7f0b09cd1e70, is_create_table=false) at /git/10.1/sql/table.cc:2988
 
#18 0x00005566fcecc15b in open_table (thd=0x7f0b16764070, table_list=0x7f0b09c42260, ot_ctx=0x7f0b230b3f50) at /git/10.1/sql/sql_base.cc:2591
 
#19 0x00005566fceceab3 in open_and_process_table (thd=0x7f0b16764070, tables=0x7f0b09c42260, counter=0x7f0b230b3fe4, flags=0, prelocking_strategy=0x7f0b230b4060, has_prelocking_list=false, ot_ctx=0x7f0b230b3f50) at /git/10.1/sql/sql_base.cc:4168
 
#20 0x00005566fcecf898 in open_tables (thd=0x7f0b16764070, options=..., start=0x7f0b230b3fc8, counter=0x7f0b230b3fe4, flags=0, prelocking_strategy=0x7f0b230b4060) at /git/10.1/sql/sql_base.cc:4629
 
#21 0x00005566fced0f94 in open_and_lock_tables (thd=0x7f0b16764070, options=..., tables=0x7f0b09c42260, derived=true, flags=0, prelocking_strategy=0x7f0b230b4060) at /git/10.1/sql/sql_base.cc:5377
 
#22 0x00005566fcec4416 in open_and_lock_tables (thd=0x7f0b16764070, tables=0x7f0b09c42260, derived=true, flags=0) at /git/10.1/sql/sql_base.h:547
 
#23 0x00005566fcf3e71c in execute_sqlcom_select (thd=0x7f0b16764070, all_tables=0x7f0b09c42260) at /git/10.1/sql/sql_parse.cc:5612
 
#24 0x00005566fcf359f0 in mysql_execute_command (thd=0x7f0b16764070) at /git/10.1/sql/sql_parse.cc:3038
 
#25 0x00005566fcf4282b in mysql_parse (thd=0x7f0b16764070, rawbuf=0x7f0b09c42088 "SELECT * FROM t2", length=16, parser_state=0x7f0b230b5060) at /git/10.1/sql/sql_parse.cc:7209
 
#26 0x00005566fcf31980 in dispatch_command (command=COM_QUERY, thd=0x7f0b16764070, packet=0x7f0b1865b071 "SELECT * FROM t2", packet_length=16) at /git/10.1/sql/sql_parse.cc:1499
 
#27 0x00005566fcf30702 in do_command (thd=0x7f0b16764070) at /git/10.1/sql/sql_parse.cc:1131
 
#28 0x00005566fd06ef17 in do_handle_one_connection (thd_arg=0x7f0b16764070) at /git/10.1/sql/sql_connect.cc:1331
 
#29 0x00005566fd06ec51 in handle_one_connection (arg=0x7f0b16764070) at /git/10.1/sql/sql_connect.cc:1242
 
#30 0x00005566fd77dc54 in pfs_spawn_thread (arg=0x7f0b1864d3f0) at /git/10.1/storage/perfschema/pfs.cc:1861
 
#31 0x00007f0b2165f6ba in start_thread (arg=0x7f0b230b6700) at pthread_create.c:333
 
#32 0x00007f0b20d0a41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109


#4  0x00007fc8820ac428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
 
#5  0x00007fc8820ae02a in __GI_abort () at abort.c:89
 
#6  0x00007fc87a5e3bbb in btr_root_block_get (index=0x7fc86b10d678, mode=2, mtr=0x7fc884526780) at /git/10.1/storage/innobase/btr/btr0btr.cc:753
 
#7  0x00007fc87a5e3c8b in btr_root_get (index=0x7fc86b10d678, mtr=0x7fc884526780) at /git/10.1/storage/innobase/btr/btr0btr.cc:780
 
#8  0x00007fc87a5e4a97 in btr_get_size_and_reserved (index=0x7fc86b10d678, flag=2, used=0x7fc884526748, mtr=0x7fc884526780) at /git/10.1/storage/innobase/btr/btr0btr.cc:1281
 
#9  0x00007fc87a5e498c in btr_get_size (index=0x7fc86b10d678, flag=2, mtr=0x7fc884526780) at /git/10.1/storage/innobase/btr/btr0btr.cc:1245
 
#10 0x00007fc87a67ef6d in dict_stats_update_transient_for_index (index=0x7fc86b10d678) at /git/10.1/storage/innobase/dict/dict0stats.cc:902
 
#11 0x00007fc87a67f1d6 in dict_stats_update_transient (table=0x7fc86b0f4378) at /git/10.1/storage/innobase/dict/dict0stats.cc:988
 
#12 0x00007fc87a683dea in dict_stats_update (table=0x7fc86b0f4378, stats_upd_option=DICT_STATS_RECALC_TRANSIENT) at /git/10.1/storage/innobase/dict/dict0stats.cc:3473
 
#13 0x00007fc87a6e69a0 in dict_stats_init (table=0x7fc86b0f4378) at /git/10.1/storage/innobase/include/dict0stats.ic:174
 
#14 0x00007fc87a6ed9b2 in ha_innodb::open (this=0x7fc86b08c088, name=0x7fc86b0b0960 "./test/t2", mode=2, test_if_locked=18) at /git/10.1/storage/innobase/handler/ha_innodb.cc:5947
 
#15 0x000056485c66f08a in handler::ha_open (this=0x7fc86b08c088, table_arg=0x7fc86b0d1e70, name=0x7fc86b0b0960 "./test/t2", mode=2, test_if_locked=18) at /git/10.1/sql/handler.cc:2569
 
#16 0x000056485c519b39 in open_table_from_share (thd=0x7fc877b64070, share=0x7fc86b0b0488, alias=0x7fc86b042258 "t2", db_stat=39, prgflag=44, ha_open_flags=18, outparam=0x7fc86b0d1e70, is_create_table=false) at /git/10.1/sql/table.cc:2988
 
#17 0x000056485c3ba15b in open_table (thd=0x7fc877b64070, table_list=0x7fc86b042260, ot_ctx=0x7fc884527f50) at /git/10.1/sql/sql_base.cc:2591
 
#18 0x000056485c3bcab3 in open_and_process_table (thd=0x7fc877b64070, tables=0x7fc86b042260, counter=0x7fc884527fe4, flags=0, prelocking_strategy=0x7fc884528060, has_prelocking_list=false, ot_ctx=0x7fc884527f50) at /git/10.1/sql/sql_base.cc:4168
 
#19 0x000056485c3bd898 in open_tables (thd=0x7fc877b64070, options=..., start=0x7fc884527fc8, counter=0x7fc884527fe4, flags=0, prelocking_strategy=0x7fc884528060) at /git/10.1/sql/sql_base.cc:4629
 
#20 0x000056485c3bef94 in open_and_lock_tables (thd=0x7fc877b64070, options=..., tables=0x7fc86b042260, derived=true, flags=0, prelocking_strategy=0x7fc884528060) at /git/10.1/sql/sql_base.cc:5377
 
#21 0x000056485c3b2416 in open_and_lock_tables (thd=0x7fc877b64070, tables=0x7fc86b042260, derived=true, flags=0) at /git/10.1/sql/sql_base.h:547
 
#22 0x000056485c42c71c in execute_sqlcom_select (thd=0x7fc877b64070, all_tables=0x7fc86b042260) at /git/10.1/sql/sql_parse.cc:5612
 
#23 0x000056485c4239f0 in mysql_execute_command (thd=0x7fc877b64070) at /git/10.1/sql/sql_parse.cc:3038
 
#24 0x000056485c43082b in mysql_parse (thd=0x7fc877b64070, rawbuf=0x7fc86b042088 "SELECT * FROM t2", length=16, parser_state=0x7fc884529060) at /git/10.1/sql/sql_parse.cc:7209
 
#25 0x000056485c41f980 in dispatch_command (command=COM_QUERY, thd=0x7fc877b64070, packet=0x7fc879a5b071 "SELECT * FROM t2", packet_length=16) at /git/10.1/sql/sql_parse.cc:1499
 
#26 0x000056485c41e702 in do_command (thd=0x7fc877b64070) at /git/10.1/sql/sql_parse.cc:1131
 
#27 0x000056485c55cf17 in do_handle_one_connection (thd_arg=0x7fc877b64070) at /git/10.1/sql/sql_connect.cc:1331
 
#28 0x000056485c55cc51 in handle_one_connection (arg=0x7fc877b64070) at /git/10.1/sql/sql_connect.cc:1242
 
#29 0x000056485cc6bc54 in pfs_spawn_thread (arg=0x7fc879a4d3f0) at /git/10.1/storage/perfschema/pfs.cc:1861
 
#30 0x00007fc882ad36ba in start_thread (arg=0x7fc88452a700) at pthread_create.c:333
 
#31 0x00007fc88217e41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Comment by Milan Sorm [ 2020-01-13 ]

Same situation on CentOS 7.3.

Situation:
Upgrading MariaDB 10.3.20 to 10.3.21 in production, tested on several servers, upgraded on 20+ servers. Only one server reported several Table doesn't exists for mysql database and one user database. Software using user database cannot work. InnoDB engine used on all tables.

Next steps:
Used mysqlfrm utility to generate structure of non-existing *.frm files in both databases. Than copied all *.ibd files outside of structure, droping non-existing tables and discarding their tablespaces. After that tables re-created from generated structures, *.ibd files returned, corrected permissions and ownership and imported tablespace. MariaDB restarted to testing user software. Everything seems fine...

Next day:
Backup of database cannot be done. User software failing for MariaDB reconnecting issue. After investigation the same error report in messages as reported soon. Any select from any database crashed MariaDB. User software cannot be used.

What do you recommend to resurrect normal production? I can go to backups of course and re-install the whole MariaDB with all databases.
The pity is that user software is backuping software Bacula and last changes on tapes are written only in database which didn't backup. I have to try restore database dump created immediately after last backup to solve the situation.

Is it any real chance how to restore functionallity without reinstalling everything from backup?

Thanks in advance.

Jan 13 23:39:08 lorlen mysqld: 2020-01-13 23:39:08 0x7f07f40f9700  InnoDB: Assertion failure in file /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.3.21/storage/innobase/btr/btr0btr.cc line 205
 
Jan 13 23:39:08 lorlen mysqld: InnoDB: Failing assertion: mach_read_from_4(seg_header + FSEG_HDR_SPACE) == space
 
Jan 13 23:39:08 lorlen mysqld: InnoDB: We intentionally generate a memory trap.
 
Jan 13 23:39:08 lorlen mysqld: InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
 
Jan 13 23:39:08 lorlen mysqld: InnoDB: If you get repeated assertion failures or crashes, even
 
Jan 13 23:39:08 lorlen mysqld: InnoDB: immediately after the mysqld startup, there may be
 
Jan 13 23:39:08 lorlen mysqld: InnoDB: corruption in the InnoDB tablespace. Please refer to
 
Jan 13 23:39:08 lorlen mysqld: InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
 
Jan 13 23:39:08 lorlen mysqld: InnoDB: about forcing recovery.
 
Jan 13 23:39:08 lorlen mysqld: 200113 23:39:08 [ERROR] mysqld got signal 6 ;
 
Jan 13 23:39:08 lorlen mysqld: This could be because you hit a bug. It is also possible that this binary
 
Jan 13 23:39:08 lorlen mysqld: or one of the libraries it was linked against is corrupt, improperly built,
 
Jan 13 23:39:08 lorlen mysqld: or misconfigured. This error can also be caused by malfunctioning hardware.
 
Jan 13 23:39:08 lorlen mysqld: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
Jan 13 23:39:08 lorlen mysqld: We will try our best to scrape up some info that will hopefully help
 
Jan 13 23:39:08 lorlen mysqld: diagnose the problem, but since we have already crashed,
 
Jan 13 23:39:08 lorlen mysqld: something is definitely wrong and this may fail.
 
Jan 13 23:39:08 lorlen mysqld: Server version: 10.3.21-MariaDB
 
Jan 13 23:39:08 lorlen mysqld: key_buffer_size=134217728
 
Jan 13 23:39:08 lorlen mysqld: read_buffer_size=131072
 
Jan 13 23:39:08 lorlen mysqld: max_used_connections=1
 
Jan 13 23:39:08 lorlen mysqld: max_threads=153
 
Jan 13 23:39:08 lorlen mysqld: thread_count=8
 
Jan 13 23:39:08 lorlen mysqld: It is possible that mysqld could use up to
 
Jan 13 23:39:08 lorlen mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467424 K  bytes of memory
 
Jan 13 23:39:08 lorlen mysqld: Hope that's ok; if not, decrease some variables in the equation.
 
Jan 13 23:39:08 lorlen mysqld: Thread pointer: 0x7f07f0000a88
 
Jan 13 23:39:08 lorlen mysqld: Attempting backtrace. You can use the following information to find out
 
Jan 13 23:39:08 lorlen mysqld: where mysqld died. If you see no messages after this, something went
 
Jan 13 23:39:08 lorlen mysqld: terribly wrong...
 
Jan 13 23:39:08 lorlen mysqld: stack_bottom = 0x7f07f40f8d30 thread_stack 0x49000
 
Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55e1e2346c0e]
 
Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55e1e1de640f]
 
Jan 13 23:39:08 lorlen mysqld: sigaction.c:0(__restore_rt)[0x7f0801bec5f0]
 
Jan 13 23:39:08 lorlen mysqld: :0(__GI_raise)[0x7f07ffebd337]
 
Jan 13 23:39:08 lorlen mysqld: :0(__GI_abort)[0x7f07ffebea28]
 
Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(+0x4d3b5c)[0x55e1e1b24b5c]
 
Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(+0xa8084f)[0x55e1e20d184f]
 
Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(+0xa826bd)[0x55e1e20d36bd]
 
Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(+0xa82891)[0x55e1e20d3891]
 
Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(+0x4d4302)[0x55e1e1b25302]
 
Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(+0xb03377)[0x55e1e2154377]
 
Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(+0xb0378d)[0x55e1e215478d]
 
Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(+0xaf2dbb)[0x55e1e2143dbb]
 
Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(+0x93e489)[0x55e1e1f8f489]
 
Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(_ZN7handler7ha_openEP5TABLEPKcijP11st_mem_rootP4ListI6StringE+0x47)[0x55e1e1deaff7]
 
Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(_Z21open_table_from_shareP3THDP11TABLE_SHAREPK25st_mysql_const_lex_stringjjjP5TABLEbP4ListI6StringE+0xbb7)[0x55e1e1cbbc07]
 
Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(_Z10open_tableP3THDP10TABLE_LISTP18Open_table_context+0xaac)[0x55e1e1ba9ebc]
 
Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(_Z11open_tablesP3THDRK14DDL_options_stPP10TABLE_LISTPjjP19Prelocking_strategy+0xd57)[0x55e1e1bad307]
 
Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(_Z29mysqld_show_create_get_fieldsP3THDP10TABLE_LISTP4ListI4ItemEP6String+0x1a2)[0x55e1e1c69da2]
 
Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(_Z18mysqld_show_createP3THDP10TABLE_LIST+0x132)[0x55e1e1c6a6b2]
 
Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x3387)[0x55e1e1c06b47]
 
Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x22b)[0x55e1e1c0c44b]
 
Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x1ca6)[0x55e1e1c0f036]
 
Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(_Z10do_commandP3THD+0x13e)[0x55e1e1c0ff7e]
 
Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x221)[0x55e1e1ce4e91]
 
Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55e1e1ce4f5d]
 
Jan 13 23:39:08 lorlen mysqld: pthread_create.c:0(start_thread)[0x7f0801be4e65]
 
Jan 13 23:39:08 lorlen mysqld: /lib64/libc.so.6(clone+0x6d)[0x7f07fff8588d]
 
Jan 13 23:39:08 lorlen mysqld: Trying to get some variables.
 
Jan 13 23:39:08 lorlen mysqld: Some pointers may be invalid and cause the dump to abort.
 
Jan 13 23:39:08 lorlen mysqld: Query (0x7f07f000f4c0): show create table `BaseFiles`
 
Jan 13 23:39:08 lorlen mysqld: Connection ID (thread ID): 10
 
Jan 13 23:39:08 lorlen mysqld: Status: NOT_KILLED
 
Jan 13 23:39:08 lorlen mysqld: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on
 
Jan 13 23:39:08 lorlen mysqld: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
 
Jan 13 23:39:08 lorlen mysqld: information that should help you find out what is causing the crash.
 
Jan 13 23:39:08 lorlen mysqld: Writing a core file...
 
Jan 13 23:39:08 lorlen mysqld: Working directory at /var/lib/mysql
 
Jan 13 23:39:08 lorlen mysqld: Resource Limits:
 
Jan 13 23:39:08 lorlen mysqld: Limit                     Soft Limit           Hard Limit           Units
 
Jan 13 23:39:08 lorlen mysqld: Max cpu time              unlimited            unlimited            seconds
 
Jan 13 23:39:08 lorlen mysqld: Max file size             unlimited            unlimited            bytes
 
Jan 13 23:39:08 lorlen mysqld: Max data size             unlimited            unlimited            bytes
 
Jan 13 23:39:08 lorlen mysqld: Max stack size            8388608              unlimited            bytes
 
Jan 13 23:39:08 lorlen mysqld: Max core file size        0                    unlimited            bytes
 
Jan 13 23:39:08 lorlen mysqld: Max resident set          unlimited            unlimited            bytes
 
Jan 13 23:39:08 lorlen mysqld: Max processes             7258                 7258                 processes
 
Jan 13 23:39:08 lorlen mysqld: Max open files            16364                16364                files
 
Jan 13 23:39:08 lorlen mysqld: Max locked memory         65536                65536                bytes
 
Jan 13 23:39:08 lorlen mysqld: Max address space         unlimited            unlimited            bytes
 
Jan 13 23:39:08 lorlen mysqld: Max file locks            unlimited            unlimited            locks
 
Jan 13 23:39:08 lorlen mysqld: Max pending signals       7258                 7258                 signals
 
Jan 13 23:39:08 lorlen mysqld: Max msgqueue size         819200               819200               bytes
 
Jan 13 23:39:08 lorlen mysqld: Max nice priority         0                    0
 
Jan 13 23:39:08 lorlen mysqld: Max realtime priority     0                    0
 
Jan 13 23:39:08 lorlen mysqld: Max realtime timeout      unlimited            unlimited            us
 
Jan 13 23:39:08 lorlen systemd: mariadb.service: main process exited, code=killed, status=6/ABRT
 
Jan 13 23:39:08 lorlen systemd: Unit mariadb.service entered failed state.

Comment by Marko Mäkelä [ 2020-01-14 ]

seally1186, it will take some time before we can analyze this bug. I suspect that the IMPORT TABLESPACE code is failing to adjust the FSEG_HDR_SPACE field in BTR_SEG_TOP and BTR_SEG_LEAF of the index root page. It could be that the reason that it avoids a crash when the server is not being restarted is that some metadata in dict_index_t will already be initialized during the import, and a call to btr_root_block_get() will be avoided.

A possible workaround could be to recompile the server without that assertion, or to make sure that UNIV_BTR_DEBUG is not defined during the compilation. The field FSEG_HDR_SPACE should actually be unnecessary, because the ‘inode’ page should always be in the same tablespace with the pointer.

Comment by Michael Chang [ 2020-01-19 ]

Thank you for the update and workaround. I did some quick testing, and it does seem that commenting out the assert prevents the crash and allows the server to read the data files.

Comment by Marko Mäkelä [ 2020-01-21 ]

As noted in MDEV-21549, the server can crash even without a shutdown.

kevg, please check all my remarks in MDEV-21549. There are several problems related to this.

Comment by Marko Mäkelä [ 2020-04-15 ]

It turns out that MDEV-21549 fixed exactly this problem. For completeness, I backported the fix to 10.1

Generated at Thu Feb 08 09:06:54 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.