[MDEV-21407] Crash when restarting server after IMPORT TABLESPACE - Jira

Details

Type: Bug
Status: Closed (View Workflow)
Priority: Major
Resolution: Duplicate
Affects Version/s: 10.4.11, 10.1(EOL), 10.2(EOL), 10.3(EOL), 10.4(EOL)
Fix Version/s: 10.1.45, 10.2.32, 10.3.23, 10.5.3
Component/s: Storage Engine - InnoDB
Labels:
None
Environment:
Tested on Arch Linux and Ubuntu 18.04 with MariaDB repo

Description

After importing a tablespace (created with mariabackup --prepare --export or FLUSH TABLE ... FOR EXPORT), if the server is restarted before writing to the new table, reading the table crashes the server.

Started with empty datadir, nothing set in my.cnf.

mariadb> create database test;

Query OK, 1 row affected (0.001 sec)

mariadb> create database test2;

Query OK, 1 row affected (0.000 sec)

mariadb> create table test.foo(id int not null primary key);

Query OK, 0 rows affected (0.185 sec)

mariadb> create table test2.foo(id int not null primary key);

Query OK, 0 rows affected (0.143 sec)

mariadb> insert into test.foo values (1);

Query OK, 1 row affected (0.031 sec)

mariadb> alter table test2.foo discard tablespace;

Query OK, 0 rows affected (0.045 sec)

mariadb> flush table test.foo for export;

Query OK, 0 rows affected (0.001 sec)

-- Copy test/foo.cfg and test/foo.ibd to test2/

mariadb> unlock tables;

Query OK, 0 rows affected (0.000 sec)

mariadb> alter table test2.foo import tablespace;

Query OK, 0 rows affected (0.200 sec)

mariadb> select * from test2.foo;

+----+

| id |

+----+

|  1 |

+----+

1 row in set (0.001 sec)

mariadb> exit

-- Restart MariaDB server

mariadb> select * from test2.foo;

ERROR 2013 (HY000): Lost connection to MySQL server during query

The server log (attached) shows:

2019-12-30  9:12:54 0 [Note] /usr/bin/mysqld: ready for connections.

Version: '10.4.11-MariaDB'  socket: '/run/mysqld/mysqld.sock'  port: 3306  Arch Linux

2019-12-30 09:13:02 0x7f91f41ae700  InnoDB: Assertion failure in file /build/mariadb/src/mariadb-10.4.11/storage/innobase/btr/btr0btr.cc line 204

InnoDB: Failing assertion: mach_read_from_4(seg_header + FSEG_HDR_SPACE) == space

It seems like doing any write to the new table prevents the issue.

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

mariadb.log
11 kB
2019-12-30 20:15

Issue Links

blocks

MDEV-18762 Support easy restore of partial backup

Closed

duplicates

MDEV-21422 InnoDB: Assertion failure in file /home/buildbot/buildbot/build/mariadb-10.4.11/storage/innobase/btr/btr0btr.cc

Closed

relates to

MDEV-21549 IMPORT TABLESPACE fails to adjust all tablespace ID in root pages

Closed

MDEV-22431 IMPORT TABLESPACE, service restart

Closed

Activity

Ascending order - Click to sort in descending order

Alice Sherepa added a comment - 2020-01-07 14:10

Thank you for the report! Reproducible with MariaDB 10.2-10.4:

--source include/have_innodb.inc

 create database test1;

 create database test2;

 create table test1.foo(id int not null primary key) engine=innodb;

 create table test2.foo(id int not null primary key) engine=innodb;

 insert into test1.foo values (1);

 alter table test2.foo discard tablespace;

 flush table test1.foo for export;

  --let $datadir = `SELECT @@datadir`

  --replace_result $datadir <DATADIR>

  --copy_file $datadir/test1/foo.ibd $datadir/test2/foo.ibd

  --copy_file $datadir/test1/foo.cfg $datadir/test2/foo.cfg

unlock tables;

alter table test2.foo import tablespace;

select * from test2.foo;

--source include/restart_mysqld.inc

select * from test2.foo;

drop database test1;

drop database test2;

10.2 0b4ae6724f2707841c50f88d
#4 0x00007f136e157428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
#5 0x00007f136e15902a in __GI_abort () at abort.c:89
#6 0x0000560fd2ab939e in ut_dbg_assertion_failed (expr=0x560fd2fdd988 "mach_read_from_4(seg_header + FSEG_HDR_SPACE) == space", file=0x560fd2fdd950 "/10.2/storage/innobase/btr/btr0btr.cc", line=205) at /10.2/storage/innobase/ut/ut0dbg.cc:60
#7 0x0000560fd2ac29c4 in btr_root_fseg_validate (seg_header=0x7f1362efc04a "", space=5) at /10.2/storage/innobase/btr/btr0btr.cc:205
#8 0x0000560fd2ac2bc2 in btr_root_block_get (index=0x7f1310135558, mode=4, mtr=0x7f13601a9be0) at /10.2/storage/innobase/btr/btr0btr.cc:251
#9 0x0000560fd2ac2c5e in btr_root_get (index=0x7f1310135558, mtr=0x7f13601a9be0) at /10.2/storage/innobase/btr/btr0btr.cc:274
#10 0x0000560fd2ac3613 in btr_get_size (index=0x7f1310135558, flag=2, mtr=0x7f13601a9be0) at /10.2/storage/innobase/btr/btr0btr.cc:615
#11 0x0000560fd2b85351 in dict_stats_update_transient_for_index (index=0x7f1310135558) at /10.2/storage/innobase/dict/dict0stats.cc:863
#12 0x0000560fd2b8563e in dict_stats_update_transient (table=0x7f1310137fd8) at /10.2/storage/innobase/dict/dict0stats.cc:950
#13 0x0000560fd2b8aabd in dict_stats_update (table=0x7f1310137fd8, stats_upd_option=DICT_STATS_RECALC_TRANSIENT) at /10.2/storage/innobase/dict/dict0stats.cc:3349
#14 0x0000560fd28a0b8a in dict_stats_init (table=0x7f1310137fd8) at /10.2/storage/innobase/include/dict0stats.ic:165
#15 0x0000560fd28aa5b7 in ha_innobase::open (this=0x7f1310133be8, name=0x7f1310137c20 "./test2/foo") at /10.2/storage/innobase/handler/ha_innodb.cc:6258
#16 0x0000560fd256b472 in handler::ha_open (this=0x7f1310133be8, table_arg=0x7f1310132fe0, name=0x7f1310137c20 "./test2/foo", mode=2, test_if_locked=18) at /10.2/sql/handler.cc:2585
#17 0x0000560fd23ed3ea in open_table_from_share (thd=0x7f1310000af0, share=0x7f1310137708, alias=0x7f1310014730 "foo", db_stat=33, prgflag=8, ha_open_flags=18, outparam=0x7f1310132fe0, is_create_table=false) at /10.2/sql/table.cc:3389
#18 0x0000560fd2270187 in open_table (thd=0x7f1310000af0, table_list=0x7f1310014738, ot_ctx=0x7f13601aaed0) at /10.2/sql/sql_base.cc:1931
#19 0x0000560fd227304a in open_and_process_table (thd=0x7f1310000af0, tables=0x7f1310014738, counter=0x7f13601aaf64, flags=0, prelocking_strategy=0x7f13601aafe0, has_prelocking_list=false, ot_ctx=0x7f13601aaed0) at /10.2/sql/sql_base.cc:3566
#20 0x0000560fd2273f3a in open_tables (thd=0x7f1310000af0, options=..., start=0x7f13601aaf48, counter=0x7f13601aaf64, flags=0, prelocking_strategy=0x7f13601aafe0) at /10.2/sql/sql_base.cc:4035
#21 0x0000560fd2275738 in open_and_lock_tables (thd=0x7f1310000af0, options=..., tables=0x7f1310014738, derived=true, flags=0, prelocking_strategy=0x7f13601aafe0) at /10.2/sql/sql_base.cc:4793
#22 0x0000560fd2268502 in open_and_lock_tables (thd=0x7f1310000af0, tables=0x7f1310014738, derived=true, flags=0) at /10.2/sql/sql_base.h:509
#23 0x0000560fd22eccfe in execute_sqlcom_select (thd=0x7f1310000af0, all_tables=0x7f1310014738) at /10.2/sql/sql_parse.cc:6146
#24 0x0000560fd22e3ba6 in mysql_execute_command (thd=0x7f1310000af0) at /10.2/sql/sql_parse.cc:3532
#25 0x0000560fd22f1035 in mysql_parse (thd=0x7f1310000af0, rawbuf=0x7f1310014548 "select * from test2.foo", length=23, parser_state=0x7f13601ac080, is_com_multi=false, is_next_command=false) at /10.2/sql/sql_parse.cc:7740
#26 0x0000560fd22df30a in dispatch_command (command=COM_QUERY, thd=0x7f1310000af0, packet=0x7f13100088a1 "select * from test2.foo", packet_length=23, is_com_multi=false, is_next_command=false) at /10.2/sql/sql_parse.cc:1831
#27 0x0000560fd22ddc1e in do_command (thd=0x7f1310000af0) at /10.2/sql/sql_parse.cc:1385
#28 0x0000560fd24347c9 in do_handle_one_connection (connect=0x560fd5eb1ac0) at /10.2/sql/sql_connect.cc:1336
#29 0x0000560fd2434534 in handle_one_connection (arg=0x560fd5eb1ac0) at /10.2/sql/sql_connect.cc:1241
#30 0x0000560fd2c5dedc in pfs_spawn_thread (arg=0x560fd5f73720) at /10.2/storage/perfschema/pfs.cc:1862
#31 0x00007f136ed946ba in start_thread (arg=0x7f13601ad700) at pthread_create.c:333
#32 0x00007f136e22941d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Alice Sherepa added a comment - 2020-01-07 14:10 Thank you for the report! Reproducible with MariaDB 10.2-10.4: --source include/have_innodb.inc create database test1; create database test2; create table test1.foo(id int not null primary key ) engine=innodb; create table test2.foo(id int not null primary key ) engine=innodb; insert into test1.foo values (1); alter table test2.foo discard tablespace; flush table test1.foo for export; --let $datadir = `SELECT @@datadir` --replace_result $datadir <DATADIR> --copy_file $datadir/test1/foo.ibd $datadir/test2/foo.ibd --copy_file $datadir/test1/foo.cfg $datadir/test2/foo.cfg unlock tables; alter table test2.foo import tablespace; select * from test2.foo; --source include/restart_mysqld.inc select * from test2.foo; drop database test1; drop database test2; 10.2 0b4ae6724f2707841c50f88d #4 0x00007f136e157428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54 #5 0x00007f136e15902a in __GI_abort () at abort.c:89 #6 0x0000560fd2ab939e in ut_dbg_assertion_failed (expr=0x560fd2fdd988 "mach_read_from_4(seg_header + FSEG_HDR_SPACE) == space", file=0x560fd2fdd950 "/10.2/storage/innobase/btr/btr0btr.cc", line=205) at /10.2/storage/innobase/ut/ut0dbg.cc:60 #7 0x0000560fd2ac29c4 in btr_root_fseg_validate (seg_header=0x7f1362efc04a "", space=5) at /10.2/storage/innobase/btr/btr0btr.cc:205 #8 0x0000560fd2ac2bc2 in btr_root_block_get (index=0x7f1310135558, mode=4, mtr=0x7f13601a9be0) at /10.2/storage/innobase/btr/btr0btr.cc:251 #9 0x0000560fd2ac2c5e in btr_root_get (index=0x7f1310135558, mtr=0x7f13601a9be0) at /10.2/storage/innobase/btr/btr0btr.cc:274 #10 0x0000560fd2ac3613 in btr_get_size (index=0x7f1310135558, flag=2, mtr=0x7f13601a9be0) at /10.2/storage/innobase/btr/btr0btr.cc:615 #11 0x0000560fd2b85351 in dict_stats_update_transient_for_index (index=0x7f1310135558) at /10.2/storage/innobase/dict/dict0stats.cc:863 #12 0x0000560fd2b8563e in dict_stats_update_transient (table=0x7f1310137fd8) at /10.2/storage/innobase/dict/dict0stats.cc:950 #13 0x0000560fd2b8aabd in dict_stats_update (table=0x7f1310137fd8, stats_upd_option=DICT_STATS_RECALC_TRANSIENT) at /10.2/storage/innobase/dict/dict0stats.cc:3349 #14 0x0000560fd28a0b8a in dict_stats_init (table=0x7f1310137fd8) at /10.2/storage/innobase/include/dict0stats.ic:165 #15 0x0000560fd28aa5b7 in ha_innobase::open (this=0x7f1310133be8, name=0x7f1310137c20 "./test2/foo") at /10.2/storage/innobase/handler/ha_innodb.cc:6258 #16 0x0000560fd256b472 in handler::ha_open (this=0x7f1310133be8, table_arg=0x7f1310132fe0, name=0x7f1310137c20 "./test2/foo", mode=2, test_if_locked=18) at /10.2/sql/handler.cc:2585 #17 0x0000560fd23ed3ea in open_table_from_share (thd=0x7f1310000af0, share=0x7f1310137708, alias=0x7f1310014730 "foo", db_stat=33, prgflag=8, ha_open_flags=18, outparam=0x7f1310132fe0, is_create_table=false) at /10.2/sql/table.cc:3389 #18 0x0000560fd2270187 in open_table (thd=0x7f1310000af0, table_list=0x7f1310014738, ot_ctx=0x7f13601aaed0) at /10.2/sql/sql_base.cc:1931 #19 0x0000560fd227304a in open_and_process_table (thd=0x7f1310000af0, tables=0x7f1310014738, counter=0x7f13601aaf64, flags=0, prelocking_strategy=0x7f13601aafe0, has_prelocking_list=false, ot_ctx=0x7f13601aaed0) at /10.2/sql/sql_base.cc:3566 #20 0x0000560fd2273f3a in open_tables (thd=0x7f1310000af0, options=..., start=0x7f13601aaf48, counter=0x7f13601aaf64, flags=0, prelocking_strategy=0x7f13601aafe0) at /10.2/sql/sql_base.cc:4035 #21 0x0000560fd2275738 in open_and_lock_tables (thd=0x7f1310000af0, options=..., tables=0x7f1310014738, derived=true, flags=0, prelocking_strategy=0x7f13601aafe0) at /10.2/sql/sql_base.cc:4793 #22 0x0000560fd2268502 in open_and_lock_tables (thd=0x7f1310000af0, tables=0x7f1310014738, derived=true, flags=0) at /10.2/sql/sql_base.h:509 #23 0x0000560fd22eccfe in execute_sqlcom_select (thd=0x7f1310000af0, all_tables=0x7f1310014738) at /10.2/sql/sql_parse.cc:6146 #24 0x0000560fd22e3ba6 in mysql_execute_command (thd=0x7f1310000af0) at /10.2/sql/sql_parse.cc:3532 #25 0x0000560fd22f1035 in mysql_parse (thd=0x7f1310000af0, rawbuf=0x7f1310014548 "select * from test2.foo", length=23, parser_state=0x7f13601ac080, is_com_multi=false, is_next_command=false) at /10.2/sql/sql_parse.cc:7740 #26 0x0000560fd22df30a in dispatch_command (command=COM_QUERY, thd=0x7f1310000af0, packet=0x7f13100088a1 "select * from test2.foo", packet_length=23, is_com_multi=false, is_next_command=false) at /10.2/sql/sql_parse.cc:1831 #27 0x0000560fd22ddc1e in do_command (thd=0x7f1310000af0) at /10.2/sql/sql_parse.cc:1385 #28 0x0000560fd24347c9 in do_handle_one_connection (connect=0x560fd5eb1ac0) at /10.2/sql/sql_connect.cc:1336 #29 0x0000560fd2434534 in handle_one_connection (arg=0x560fd5eb1ac0) at /10.2/sql/sql_connect.cc:1241 #30 0x0000560fd2c5dedc in pfs_spawn_thread (arg=0x560fd5f73720) at /10.2/storage/perfschema/pfs.cc:1862 #31 0x00007f136ed946ba in start_thread (arg=0x7f13601ad700) at pthread_create.c:333 #32 0x00007f136e22941d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Alice Sherepa added a comment - 2020-01-08 11:30

./mtr encryption.innodb-bad-key-change --mysqld=--innodb-checksum-algorithm=none

10.1 cb204e11eaf4c473ce5d5a10a21de1
Version: '10.1.44-MariaDB-debug'
2020-01-08 12:24:27 7f0b230b6700 InnoDB: Assertion failure in thread 139685809317632 in file btr0btr.cc line 708
InnoDB: Failing assertion: mach_read_from_4(seg_header + FSEG_HDR_SPACE) == space
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
200108 12:24:27 [ERROR] mysqld got signal 6 ;

#4 0x00007f0b20c38428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
#5 0x00007f0b20c3a02a in __GI_abort () at abort.c:89
#6 0x00007f0b191e3a16 in btr_root_fseg_validate (seg_header=0x7f0b16eb804a "\374\200\204\240?ZU\267\"\034\063E\341\252\312\360BOg\245\252@\032x=99\225\245\202\021\262x\001\324\351x\376P\233\317\215\067\366\207W\031?\036r\274@\345K\a\240\354v{9\217s-K\270x\342'\202\002\346\066\354\313{TQ\203\061k\335\rk\005\236\304v\004mx!\317\060\367\bO\320\307=\277\tK\343O.\306M*\312\341\307\232\327\266\337\255\316op\300\003\330\347X/\270\253\036z\325G\265\060\340\024>\273\317H\352j\225\310\230(\367\n\307~S~\246s\340V\024\240\353wz\365\221l\336\036\272W?\341\005\351\005g\234`F\363X\227\320\325\372\323\251\213x\304\230}\255\366\375\376\372\346\070+\360\021z"..., space=7) at /git/10.1/storage/innobase/btr/btr0btr.cc:708
#7 0x00007f0b191e3bf7 in btr_root_block_get (index=0x7f0b09d0d678, mode=2, mtr=0x7f0b230b2780) at /git/10.1/storage/innobase/btr/btr0btr.cc:759
#8 0x00007f0b191e3c8b in btr_root_get (index=0x7f0b09d0d678, mtr=0x7f0b230b2780) at /git/10.1/storage/innobase/btr/btr0btr.cc:780
#9 0x00007f0b191e4a97 in btr_get_size_and_reserved (index=0x7f0b09d0d678, flag=2, used=0x7f0b230b2748, mtr=0x7f0b230b2780) at /git/10.1/storage/innobase/btr/btr0btr.cc:1281
#10 0x00007f0b191e498c in btr_get_size (index=0x7f0b09d0d678, flag=2, mtr=0x7f0b230b2780) at /git/10.1/storage/innobase/btr/btr0btr.cc:1245
#11 0x00007f0b1927ef6d in dict_stats_update_transient_for_index (index=0x7f0b09d0d678) at /git/10.1/storage/innobase/dict/dict0stats.cc:902
#12 0x00007f0b1927f1d6 in dict_stats_update_transient (table=0x7f0b09cf4378) at /git/10.1/storage/innobase/dict/dict0stats.cc:988
#13 0x00007f0b19283dea in dict_stats_update (table=0x7f0b09cf4378, stats_upd_option=DICT_STATS_RECALC_TRANSIENT) at /git/10.1/storage/innobase/dict/dict0stats.cc:3473
#14 0x00007f0b192e69a0 in dict_stats_init (table=0x7f0b09cf4378) at /git/10.1/storage/innobase/include/dict0stats.ic:174
#15 0x00007f0b192ed9b2 in ha_innodb::open (this=0x7f0b09c8c088, name=0x7f0b09cb0960 "./test/t2", mode=2, test_if_locked=18) at /git/10.1/storage/innobase/handler/ha_innodb.cc:5947
#16 0x00005566fd18108a in handler::ha_open (this=0x7f0b09c8c088, table_arg=0x7f0b09cd1e70, name=0x7f0b09cb0960 "./test/t2", mode=2, test_if_locked=18) at /git/10.1/sql/handler.cc:2569
#17 0x00005566fd02bb39 in open_table_from_share (thd=0x7f0b16764070, share=0x7f0b09cb0488, alias=0x7f0b09c42258 "t2", db_stat=39, prgflag=44, ha_open_flags=18, outparam=0x7f0b09cd1e70, is_create_table=false) at /git/10.1/sql/table.cc:2988
#18 0x00005566fcecc15b in open_table (thd=0x7f0b16764070, table_list=0x7f0b09c42260, ot_ctx=0x7f0b230b3f50) at /git/10.1/sql/sql_base.cc:2591
#19 0x00005566fceceab3 in open_and_process_table (thd=0x7f0b16764070, tables=0x7f0b09c42260, counter=0x7f0b230b3fe4, flags=0, prelocking_strategy=0x7f0b230b4060, has_prelocking_list=false, ot_ctx=0x7f0b230b3f50) at /git/10.1/sql/sql_base.cc:4168
#20 0x00005566fcecf898 in open_tables (thd=0x7f0b16764070, options=..., start=0x7f0b230b3fc8, counter=0x7f0b230b3fe4, flags=0, prelocking_strategy=0x7f0b230b4060) at /git/10.1/sql/sql_base.cc:4629
#21 0x00005566fced0f94 in open_and_lock_tables (thd=0x7f0b16764070, options=..., tables=0x7f0b09c42260, derived=true, flags=0, prelocking_strategy=0x7f0b230b4060) at /git/10.1/sql/sql_base.cc:5377
#22 0x00005566fcec4416 in open_and_lock_tables (thd=0x7f0b16764070, tables=0x7f0b09c42260, derived=true, flags=0) at /git/10.1/sql/sql_base.h:547
#23 0x00005566fcf3e71c in execute_sqlcom_select (thd=0x7f0b16764070, all_tables=0x7f0b09c42260) at /git/10.1/sql/sql_parse.cc:5612
#24 0x00005566fcf359f0 in mysql_execute_command (thd=0x7f0b16764070) at /git/10.1/sql/sql_parse.cc:3038
#25 0x00005566fcf4282b in mysql_parse (thd=0x7f0b16764070, rawbuf=0x7f0b09c42088 "SELECT * FROM t2", length=16, parser_state=0x7f0b230b5060) at /git/10.1/sql/sql_parse.cc:7209
#26 0x00005566fcf31980 in dispatch_command (command=COM_QUERY, thd=0x7f0b16764070, packet=0x7f0b1865b071 "SELECT * FROM t2", packet_length=16) at /git/10.1/sql/sql_parse.cc:1499
#27 0x00005566fcf30702 in do_command (thd=0x7f0b16764070) at /git/10.1/sql/sql_parse.cc:1131
#28 0x00005566fd06ef17 in do_handle_one_connection (thd_arg=0x7f0b16764070) at /git/10.1/sql/sql_connect.cc:1331
#29 0x00005566fd06ec51 in handle_one_connection (arg=0x7f0b16764070) at /git/10.1/sql/sql_connect.cc:1242
#30 0x00005566fd77dc54 in pfs_spawn_thread (arg=0x7f0b1864d3f0) at /git/10.1/storage/perfschema/pfs.cc:1861
#31 0x00007f0b2165f6ba in start_thread (arg=0x7f0b230b6700) at pthread_create.c:333
#32 0x00007f0b20d0a41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

#4  0x00007fc8820ac428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54

#5  0x00007fc8820ae02a in __GI_abort () at abort.c:89

#6  0x00007fc87a5e3bbb in btr_root_block_get (index=0x7fc86b10d678, mode=2, mtr=0x7fc884526780) at /git/10.1/storage/innobase/btr/btr0btr.cc:753

#7  0x00007fc87a5e3c8b in btr_root_get (index=0x7fc86b10d678, mtr=0x7fc884526780) at /git/10.1/storage/innobase/btr/btr0btr.cc:780

#8  0x00007fc87a5e4a97 in btr_get_size_and_reserved (index=0x7fc86b10d678, flag=2, used=0x7fc884526748, mtr=0x7fc884526780) at /git/10.1/storage/innobase/btr/btr0btr.cc:1281

#9  0x00007fc87a5e498c in btr_get_size (index=0x7fc86b10d678, flag=2, mtr=0x7fc884526780) at /git/10.1/storage/innobase/btr/btr0btr.cc:1245

#10 0x00007fc87a67ef6d in dict_stats_update_transient_for_index (index=0x7fc86b10d678) at /git/10.1/storage/innobase/dict/dict0stats.cc:902

#11 0x00007fc87a67f1d6 in dict_stats_update_transient (table=0x7fc86b0f4378) at /git/10.1/storage/innobase/dict/dict0stats.cc:988

#12 0x00007fc87a683dea in dict_stats_update (table=0x7fc86b0f4378, stats_upd_option=DICT_STATS_RECALC_TRANSIENT) at /git/10.1/storage/innobase/dict/dict0stats.cc:3473

#13 0x00007fc87a6e69a0 in dict_stats_init (table=0x7fc86b0f4378) at /git/10.1/storage/innobase/include/dict0stats.ic:174

#14 0x00007fc87a6ed9b2 in ha_innodb::open (this=0x7fc86b08c088, name=0x7fc86b0b0960 "./test/t2", mode=2, test_if_locked=18) at /git/10.1/storage/innobase/handler/ha_innodb.cc:5947

#15 0x000056485c66f08a in handler::ha_open (this=0x7fc86b08c088, table_arg=0x7fc86b0d1e70, name=0x7fc86b0b0960 "./test/t2", mode=2, test_if_locked=18) at /git/10.1/sql/handler.cc:2569

#16 0x000056485c519b39 in open_table_from_share (thd=0x7fc877b64070, share=0x7fc86b0b0488, alias=0x7fc86b042258 "t2", db_stat=39, prgflag=44, ha_open_flags=18, outparam=0x7fc86b0d1e70, is_create_table=false) at /git/10.1/sql/table.cc:2988

#17 0x000056485c3ba15b in open_table (thd=0x7fc877b64070, table_list=0x7fc86b042260, ot_ctx=0x7fc884527f50) at /git/10.1/sql/sql_base.cc:2591

#18 0x000056485c3bcab3 in open_and_process_table (thd=0x7fc877b64070, tables=0x7fc86b042260, counter=0x7fc884527fe4, flags=0, prelocking_strategy=0x7fc884528060, has_prelocking_list=false, ot_ctx=0x7fc884527f50) at /git/10.1/sql/sql_base.cc:4168

#19 0x000056485c3bd898 in open_tables (thd=0x7fc877b64070, options=..., start=0x7fc884527fc8, counter=0x7fc884527fe4, flags=0, prelocking_strategy=0x7fc884528060) at /git/10.1/sql/sql_base.cc:4629

#20 0x000056485c3bef94 in open_and_lock_tables (thd=0x7fc877b64070, options=..., tables=0x7fc86b042260, derived=true, flags=0, prelocking_strategy=0x7fc884528060) at /git/10.1/sql/sql_base.cc:5377

#21 0x000056485c3b2416 in open_and_lock_tables (thd=0x7fc877b64070, tables=0x7fc86b042260, derived=true, flags=0) at /git/10.1/sql/sql_base.h:547

#22 0x000056485c42c71c in execute_sqlcom_select (thd=0x7fc877b64070, all_tables=0x7fc86b042260) at /git/10.1/sql/sql_parse.cc:5612

#23 0x000056485c4239f0 in mysql_execute_command (thd=0x7fc877b64070) at /git/10.1/sql/sql_parse.cc:3038

#24 0x000056485c43082b in mysql_parse (thd=0x7fc877b64070, rawbuf=0x7fc86b042088 "SELECT * FROM t2", length=16, parser_state=0x7fc884529060) at /git/10.1/sql/sql_parse.cc:7209

#25 0x000056485c41f980 in dispatch_command (command=COM_QUERY, thd=0x7fc877b64070, packet=0x7fc879a5b071 "SELECT * FROM t2", packet_length=16) at /git/10.1/sql/sql_parse.cc:1499

#26 0x000056485c41e702 in do_command (thd=0x7fc877b64070) at /git/10.1/sql/sql_parse.cc:1131

#27 0x000056485c55cf17 in do_handle_one_connection (thd_arg=0x7fc877b64070) at /git/10.1/sql/sql_connect.cc:1331

#28 0x000056485c55cc51 in handle_one_connection (arg=0x7fc877b64070) at /git/10.1/sql/sql_connect.cc:1242

#29 0x000056485cc6bc54 in pfs_spawn_thread (arg=0x7fc879a4d3f0) at /git/10.1/storage/perfschema/pfs.cc:1861

#30 0x00007fc882ad36ba in start_thread (arg=0x7fc88452a700) at pthread_create.c:333

#31 0x00007fc88217e41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Alice Sherepa added a comment - 2020-01-08 11:30 ./mtr encryption.innodb-bad-key-change --mysqld=--innodb-checksum-algorithm=none 10.1 cb204e11eaf4c473ce5d5a10a21de1 Version: '10.1.44-MariaDB-debug' 2020-01-08 12:24:27 7f0b230b6700 InnoDB: Assertion failure in thread 139685809317632 in file btr0btr.cc line 708 InnoDB: Failing assertion: mach_read_from_4(seg_header + FSEG_HDR_SPACE) == space InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to https://jira.mariadb.org/ InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 200108 12:24:27 [ERROR] mysqld got signal 6 ; #4 0x00007f0b20c38428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54 #5 0x00007f0b20c3a02a in __GI_abort () at abort.c:89 #6 0x00007f0b191e3a16 in btr_root_fseg_validate (seg_header=0x7f0b16eb804a "\374\200\204\240?ZU\267\"\034\063E\341\252\312\360BOg\245\252@\032x=99\225\245\202\021\262x\001\324\351x\376P\233\317\215\067\366\207W\031?\036r\274@\345K\a\240\354v{9\217s-K\270x\342'\202\002\346\066\354\313{TQ\203\061k\335\rk\005\236\304v\004mx!\317\060\367\bO\320\307=\277\tK\343O.\306M*\312\341\307\232\327\266\337\255\316op\300\003\330\347X/\270\253\036z\325G\265\060\340\024>\273\317H\352j\225\310\230(\367\n\307~S~\246s\340V\024\240\353wz\365\221l\336\036\272W?\341\005\351\005g\234`F\363X\227\320\325\372\323\251\213x\304\230}\255\366\375\376\372\346\070+\360\021z"..., space=7) at /git/10.1/storage/innobase/btr/btr0btr.cc:708 #7 0x00007f0b191e3bf7 in btr_root_block_get (index=0x7f0b09d0d678, mode=2, mtr=0x7f0b230b2780) at /git/10.1/storage/innobase/btr/btr0btr.cc:759 #8 0x00007f0b191e3c8b in btr_root_get (index=0x7f0b09d0d678, mtr=0x7f0b230b2780) at /git/10.1/storage/innobase/btr/btr0btr.cc:780 #9 0x00007f0b191e4a97 in btr_get_size_and_reserved (index=0x7f0b09d0d678, flag=2, used=0x7f0b230b2748, mtr=0x7f0b230b2780) at /git/10.1/storage/innobase/btr/btr0btr.cc:1281 #10 0x00007f0b191e498c in btr_get_size (index=0x7f0b09d0d678, flag=2, mtr=0x7f0b230b2780) at /git/10.1/storage/innobase/btr/btr0btr.cc:1245 #11 0x00007f0b1927ef6d in dict_stats_update_transient_for_index (index=0x7f0b09d0d678) at /git/10.1/storage/innobase/dict/dict0stats.cc:902 #12 0x00007f0b1927f1d6 in dict_stats_update_transient (table=0x7f0b09cf4378) at /git/10.1/storage/innobase/dict/dict0stats.cc:988 #13 0x00007f0b19283dea in dict_stats_update (table=0x7f0b09cf4378, stats_upd_option=DICT_STATS_RECALC_TRANSIENT) at /git/10.1/storage/innobase/dict/dict0stats.cc:3473 #14 0x00007f0b192e69a0 in dict_stats_init (table=0x7f0b09cf4378) at /git/10.1/storage/innobase/include/dict0stats.ic:174 #15 0x00007f0b192ed9b2 in ha_innodb::open (this=0x7f0b09c8c088, name=0x7f0b09cb0960 "./test/t2", mode=2, test_if_locked=18) at /git/10.1/storage/innobase/handler/ha_innodb.cc:5947 #16 0x00005566fd18108a in handler::ha_open (this=0x7f0b09c8c088, table_arg=0x7f0b09cd1e70, name=0x7f0b09cb0960 "./test/t2", mode=2, test_if_locked=18) at /git/10.1/sql/handler.cc:2569 #17 0x00005566fd02bb39 in open_table_from_share (thd=0x7f0b16764070, share=0x7f0b09cb0488, alias=0x7f0b09c42258 "t2", db_stat=39, prgflag=44, ha_open_flags=18, outparam=0x7f0b09cd1e70, is_create_table=false) at /git/10.1/sql/table.cc:2988 #18 0x00005566fcecc15b in open_table (thd=0x7f0b16764070, table_list=0x7f0b09c42260, ot_ctx=0x7f0b230b3f50) at /git/10.1/sql/sql_base.cc:2591 #19 0x00005566fceceab3 in open_and_process_table (thd=0x7f0b16764070, tables=0x7f0b09c42260, counter=0x7f0b230b3fe4, flags=0, prelocking_strategy=0x7f0b230b4060, has_prelocking_list=false, ot_ctx=0x7f0b230b3f50) at /git/10.1/sql/sql_base.cc:4168 #20 0x00005566fcecf898 in open_tables (thd=0x7f0b16764070, options=..., start=0x7f0b230b3fc8, counter=0x7f0b230b3fe4, flags=0, prelocking_strategy=0x7f0b230b4060) at /git/10.1/sql/sql_base.cc:4629 #21 0x00005566fced0f94 in open_and_lock_tables (thd=0x7f0b16764070, options=..., tables=0x7f0b09c42260, derived=true, flags=0, prelocking_strategy=0x7f0b230b4060) at /git/10.1/sql/sql_base.cc:5377 #22 0x00005566fcec4416 in open_and_lock_tables (thd=0x7f0b16764070, tables=0x7f0b09c42260, derived=true, flags=0) at /git/10.1/sql/sql_base.h:547 #23 0x00005566fcf3e71c in execute_sqlcom_select (thd=0x7f0b16764070, all_tables=0x7f0b09c42260) at /git/10.1/sql/sql_parse.cc:5612 #24 0x00005566fcf359f0 in mysql_execute_command (thd=0x7f0b16764070) at /git/10.1/sql/sql_parse.cc:3038 #25 0x00005566fcf4282b in mysql_parse (thd=0x7f0b16764070, rawbuf=0x7f0b09c42088 "SELECT * FROM t2", length=16, parser_state=0x7f0b230b5060) at /git/10.1/sql/sql_parse.cc:7209 #26 0x00005566fcf31980 in dispatch_command (command=COM_QUERY, thd=0x7f0b16764070, packet=0x7f0b1865b071 "SELECT * FROM t2", packet_length=16) at /git/10.1/sql/sql_parse.cc:1499 #27 0x00005566fcf30702 in do_command (thd=0x7f0b16764070) at /git/10.1/sql/sql_parse.cc:1131 #28 0x00005566fd06ef17 in do_handle_one_connection (thd_arg=0x7f0b16764070) at /git/10.1/sql/sql_connect.cc:1331 #29 0x00005566fd06ec51 in handle_one_connection (arg=0x7f0b16764070) at /git/10.1/sql/sql_connect.cc:1242 #30 0x00005566fd77dc54 in pfs_spawn_thread (arg=0x7f0b1864d3f0) at /git/10.1/storage/perfschema/pfs.cc:1861 #31 0x00007f0b2165f6ba in start_thread (arg=0x7f0b230b6700) at pthread_create.c:333 #32 0x00007f0b20d0a41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 #4 0x00007fc8820ac428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54 #5 0x00007fc8820ae02a in __GI_abort () at abort.c:89 #6 0x00007fc87a5e3bbb in btr_root_block_get (index=0x7fc86b10d678, mode=2, mtr=0x7fc884526780) at /git/10.1/storage/innobase/btr/btr0btr.cc:753 #7 0x00007fc87a5e3c8b in btr_root_get (index=0x7fc86b10d678, mtr=0x7fc884526780) at /git/10.1/storage/innobase/btr/btr0btr.cc:780 #8 0x00007fc87a5e4a97 in btr_get_size_and_reserved (index=0x7fc86b10d678, flag=2, used=0x7fc884526748, mtr=0x7fc884526780) at /git/10.1/storage/innobase/btr/btr0btr.cc:1281 #9 0x00007fc87a5e498c in btr_get_size (index=0x7fc86b10d678, flag=2, mtr=0x7fc884526780) at /git/10.1/storage/innobase/btr/btr0btr.cc:1245 #10 0x00007fc87a67ef6d in dict_stats_update_transient_for_index (index=0x7fc86b10d678) at /git/10.1/storage/innobase/dict/dict0stats.cc:902 #11 0x00007fc87a67f1d6 in dict_stats_update_transient (table=0x7fc86b0f4378) at /git/10.1/storage/innobase/dict/dict0stats.cc:988 #12 0x00007fc87a683dea in dict_stats_update (table=0x7fc86b0f4378, stats_upd_option=DICT_STATS_RECALC_TRANSIENT) at /git/10.1/storage/innobase/dict/dict0stats.cc:3473 #13 0x00007fc87a6e69a0 in dict_stats_init (table=0x7fc86b0f4378) at /git/10.1/storage/innobase/include/dict0stats.ic:174 #14 0x00007fc87a6ed9b2 in ha_innodb::open (this=0x7fc86b08c088, name=0x7fc86b0b0960 "./test/t2", mode=2, test_if_locked=18) at /git/10.1/storage/innobase/handler/ha_innodb.cc:5947 #15 0x000056485c66f08a in handler::ha_open (this=0x7fc86b08c088, table_arg=0x7fc86b0d1e70, name=0x7fc86b0b0960 "./test/t2", mode=2, test_if_locked=18) at /git/10.1/sql/handler.cc:2569 #16 0x000056485c519b39 in open_table_from_share (thd=0x7fc877b64070, share=0x7fc86b0b0488, alias=0x7fc86b042258 "t2", db_stat=39, prgflag=44, ha_open_flags=18, outparam=0x7fc86b0d1e70, is_create_table=false) at /git/10.1/sql/table.cc:2988 #17 0x000056485c3ba15b in open_table (thd=0x7fc877b64070, table_list=0x7fc86b042260, ot_ctx=0x7fc884527f50) at /git/10.1/sql/sql_base.cc:2591 #18 0x000056485c3bcab3 in open_and_process_table (thd=0x7fc877b64070, tables=0x7fc86b042260, counter=0x7fc884527fe4, flags=0, prelocking_strategy=0x7fc884528060, has_prelocking_list=false, ot_ctx=0x7fc884527f50) at /git/10.1/sql/sql_base.cc:4168 #19 0x000056485c3bd898 in open_tables (thd=0x7fc877b64070, options=..., start=0x7fc884527fc8, counter=0x7fc884527fe4, flags=0, prelocking_strategy=0x7fc884528060) at /git/10.1/sql/sql_base.cc:4629 #20 0x000056485c3bef94 in open_and_lock_tables (thd=0x7fc877b64070, options=..., tables=0x7fc86b042260, derived=true, flags=0, prelocking_strategy=0x7fc884528060) at /git/10.1/sql/sql_base.cc:5377 #21 0x000056485c3b2416 in open_and_lock_tables (thd=0x7fc877b64070, tables=0x7fc86b042260, derived=true, flags=0) at /git/10.1/sql/sql_base.h:547 #22 0x000056485c42c71c in execute_sqlcom_select (thd=0x7fc877b64070, all_tables=0x7fc86b042260) at /git/10.1/sql/sql_parse.cc:5612 #23 0x000056485c4239f0 in mysql_execute_command (thd=0x7fc877b64070) at /git/10.1/sql/sql_parse.cc:3038 #24 0x000056485c43082b in mysql_parse (thd=0x7fc877b64070, rawbuf=0x7fc86b042088 "SELECT * FROM t2", length=16, parser_state=0x7fc884529060) at /git/10.1/sql/sql_parse.cc:7209 #25 0x000056485c41f980 in dispatch_command (command=COM_QUERY, thd=0x7fc877b64070, packet=0x7fc879a5b071 "SELECT * FROM t2", packet_length=16) at /git/10.1/sql/sql_parse.cc:1499 #26 0x000056485c41e702 in do_command (thd=0x7fc877b64070) at /git/10.1/sql/sql_parse.cc:1131 #27 0x000056485c55cf17 in do_handle_one_connection (thd_arg=0x7fc877b64070) at /git/10.1/sql/sql_connect.cc:1331 #28 0x000056485c55cc51 in handle_one_connection (arg=0x7fc877b64070) at /git/10.1/sql/sql_connect.cc:1242 #29 0x000056485cc6bc54 in pfs_spawn_thread (arg=0x7fc879a4d3f0) at /git/10.1/storage/perfschema/pfs.cc:1861 #30 0x00007fc882ad36ba in start_thread (arg=0x7fc88452a700) at pthread_create.c:333 #31 0x00007fc88217e41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Milan Sorm added a comment - 2020-01-13 22:53 - edited

Same situation on CentOS 7.3.

Situation:
Upgrading MariaDB 10.3.20 to 10.3.21 in production, tested on several servers, upgraded on 20+ servers. Only one server reported several Table doesn't exists for mysql database and one user database. Software using user database cannot work. InnoDB engine used on all tables.

Next steps:
Used mysqlfrm utility to generate structure of non-existing *.frm files in both databases. Than copied all *.ibd files outside of structure, droping non-existing tables and discarding their tablespaces. After that tables re-created from generated structures, *.ibd files returned, corrected permissions and ownership and imported tablespace. MariaDB restarted to testing user software. Everything seems fine...

Next day:
Backup of database cannot be done. User software failing for MariaDB reconnecting issue. After investigation the same error report in messages as reported soon. Any select from any database crashed MariaDB. User software cannot be used.

What do you recommend to resurrect normal production? I can go to backups of course and re-install the whole MariaDB with all databases.
The pity is that user software is backuping software Bacula and last changes on tapes are written only in database which didn't backup. I have to try restore database dump created immediately after last backup to solve the situation.

Is it any real chance how to restore functionallity without reinstalling everything from backup?

Thanks in advance.

Jan 13 23:39:08 lorlen mysqld: 2020-01-13 23:39:08 0x7f07f40f9700  InnoDB: Assertion failure in file /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.3.21/storage/innobase/btr/btr0btr.cc line 205

Jan 13 23:39:08 lorlen mysqld: InnoDB: Failing assertion: mach_read_from_4(seg_header + FSEG_HDR_SPACE) == space

Jan 13 23:39:08 lorlen mysqld: InnoDB: We intentionally generate a memory trap.

Jan 13 23:39:08 lorlen mysqld: InnoDB: Submit a detailed bug report to https://jira.mariadb.org/

Jan 13 23:39:08 lorlen mysqld: InnoDB: If you get repeated assertion failures or crashes, even

Jan 13 23:39:08 lorlen mysqld: InnoDB: immediately after the mysqld startup, there may be

Jan 13 23:39:08 lorlen mysqld: InnoDB: corruption in the InnoDB tablespace. Please refer to

Jan 13 23:39:08 lorlen mysqld: InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/

Jan 13 23:39:08 lorlen mysqld: InnoDB: about forcing recovery.

Jan 13 23:39:08 lorlen mysqld: 200113 23:39:08 [ERROR] mysqld got signal 6 ;

Jan 13 23:39:08 lorlen mysqld: This could be because you hit a bug. It is also possible that this binary

Jan 13 23:39:08 lorlen mysqld: or one of the libraries it was linked against is corrupt, improperly built,

Jan 13 23:39:08 lorlen mysqld: or misconfigured. This error can also be caused by malfunctioning hardware.

Jan 13 23:39:08 lorlen mysqld: To report this bug, see https://mariadb.com/kb/en/reporting-bugs

Jan 13 23:39:08 lorlen mysqld: We will try our best to scrape up some info that will hopefully help

Jan 13 23:39:08 lorlen mysqld: diagnose the problem, but since we have already crashed,

Jan 13 23:39:08 lorlen mysqld: something is definitely wrong and this may fail.

Jan 13 23:39:08 lorlen mysqld: Server version: 10.3.21-MariaDB

Jan 13 23:39:08 lorlen mysqld: key_buffer_size=134217728

Jan 13 23:39:08 lorlen mysqld: read_buffer_size=131072

Jan 13 23:39:08 lorlen mysqld: max_used_connections=1

Jan 13 23:39:08 lorlen mysqld: max_threads=153

Jan 13 23:39:08 lorlen mysqld: thread_count=8

Jan 13 23:39:08 lorlen mysqld: It is possible that mysqld could use up to

Jan 13 23:39:08 lorlen mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467424 K  bytes of memory

Jan 13 23:39:08 lorlen mysqld: Hope that's ok; if not, decrease some variables in the equation.

Jan 13 23:39:08 lorlen mysqld: Thread pointer: 0x7f07f0000a88

Jan 13 23:39:08 lorlen mysqld: Attempting backtrace. You can use the following information to find out

Jan 13 23:39:08 lorlen mysqld: where mysqld died. If you see no messages after this, something went

Jan 13 23:39:08 lorlen mysqld: terribly wrong...

Jan 13 23:39:08 lorlen mysqld: stack_bottom = 0x7f07f40f8d30 thread_stack 0x49000

Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55e1e2346c0e]

Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55e1e1de640f]

Jan 13 23:39:08 lorlen mysqld: sigaction.c:0(__restore_rt)[0x7f0801bec5f0]

Jan 13 23:39:08 lorlen mysqld: :0(__GI_raise)[0x7f07ffebd337]

Jan 13 23:39:08 lorlen mysqld: :0(__GI_abort)[0x7f07ffebea28]

Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(+0x4d3b5c)[0x55e1e1b24b5c]

Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(+0xa8084f)[0x55e1e20d184f]

Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(+0xa826bd)[0x55e1e20d36bd]

Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(+0xa82891)[0x55e1e20d3891]

Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(+0x4d4302)[0x55e1e1b25302]

Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(+0xb03377)[0x55e1e2154377]

Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(+0xb0378d)[0x55e1e215478d]

Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(+0xaf2dbb)[0x55e1e2143dbb]

Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(+0x93e489)[0x55e1e1f8f489]

Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(_ZN7handler7ha_openEP5TABLEPKcijP11st_mem_rootP4ListI6StringE+0x47)[0x55e1e1deaff7]

Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(_Z21open_table_from_shareP3THDP11TABLE_SHAREPK25st_mysql_const_lex_stringjjjP5TABLEbP4ListI6StringE+0xbb7)[0x55e1e1cbbc07]

Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(_Z10open_tableP3THDP10TABLE_LISTP18Open_table_context+0xaac)[0x55e1e1ba9ebc]

Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(_Z11open_tablesP3THDRK14DDL_options_stPP10TABLE_LISTPjjP19Prelocking_strategy+0xd57)[0x55e1e1bad307]

Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(_Z29mysqld_show_create_get_fieldsP3THDP10TABLE_LISTP4ListI4ItemEP6String+0x1a2)[0x55e1e1c69da2]

Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(_Z18mysqld_show_createP3THDP10TABLE_LIST+0x132)[0x55e1e1c6a6b2]

Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x3387)[0x55e1e1c06b47]

Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x22b)[0x55e1e1c0c44b]

Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x1ca6)[0x55e1e1c0f036]

Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(_Z10do_commandP3THD+0x13e)[0x55e1e1c0ff7e]

Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x221)[0x55e1e1ce4e91]

Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55e1e1ce4f5d]

Jan 13 23:39:08 lorlen mysqld: pthread_create.c:0(start_thread)[0x7f0801be4e65]

Jan 13 23:39:08 lorlen mysqld: /lib64/libc.so.6(clone+0x6d)[0x7f07fff8588d]

Jan 13 23:39:08 lorlen mysqld: Trying to get some variables.

Jan 13 23:39:08 lorlen mysqld: Some pointers may be invalid and cause the dump to abort.

Jan 13 23:39:08 lorlen mysqld: Query (0x7f07f000f4c0): show create table `BaseFiles`

Jan 13 23:39:08 lorlen mysqld: Connection ID (thread ID): 10

Jan 13 23:39:08 lorlen mysqld: Status: NOT_KILLED

Jan 13 23:39:08 lorlen mysqld: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on

Jan 13 23:39:08 lorlen mysqld: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains

Jan 13 23:39:08 lorlen mysqld: information that should help you find out what is causing the crash.

Jan 13 23:39:08 lorlen mysqld: Writing a core file...

Jan 13 23:39:08 lorlen mysqld: Working directory at /var/lib/mysql

Jan 13 23:39:08 lorlen mysqld: Resource Limits:

Jan 13 23:39:08 lorlen mysqld: Limit                     Soft Limit           Hard Limit           Units

Jan 13 23:39:08 lorlen mysqld: Max cpu time              unlimited            unlimited            seconds

Jan 13 23:39:08 lorlen mysqld: Max file size             unlimited            unlimited            bytes

Jan 13 23:39:08 lorlen mysqld: Max data size             unlimited            unlimited            bytes

Jan 13 23:39:08 lorlen mysqld: Max stack size            8388608              unlimited            bytes

Jan 13 23:39:08 lorlen mysqld: Max core file size        0                    unlimited            bytes

Jan 13 23:39:08 lorlen mysqld: Max resident set          unlimited            unlimited            bytes

Jan 13 23:39:08 lorlen mysqld: Max processes             7258                 7258                 processes

Jan 13 23:39:08 lorlen mysqld: Max open files            16364                16364                files

Jan 13 23:39:08 lorlen mysqld: Max locked memory         65536                65536                bytes

Jan 13 23:39:08 lorlen mysqld: Max address space         unlimited            unlimited            bytes

Jan 13 23:39:08 lorlen mysqld: Max file locks            unlimited            unlimited            locks

Jan 13 23:39:08 lorlen mysqld: Max pending signals       7258                 7258                 signals

Jan 13 23:39:08 lorlen mysqld: Max msgqueue size         819200               819200               bytes

Jan 13 23:39:08 lorlen mysqld: Max nice priority         0                    0

Jan 13 23:39:08 lorlen mysqld: Max realtime priority     0                    0

Jan 13 23:39:08 lorlen mysqld: Max realtime timeout      unlimited            unlimited            us

Jan 13 23:39:08 lorlen systemd: mariadb.service: main process exited, code=killed, status=6/ABRT

Jan 13 23:39:08 lorlen systemd: Unit mariadb.service entered failed state.

Milan Sorm added a comment - 2020-01-13 22:53 - edited Same situation on CentOS 7.3. Situation: Upgrading MariaDB 10.3.20 to 10.3.21 in production, tested on several servers, upgraded on 20+ servers. Only one server reported several Table doesn't exists for mysql database and one user database. Software using user database cannot work. InnoDB engine used on all tables. Next steps: Used mysqlfrm utility to generate structure of non-existing *.frm files in both databases. Than copied all *.ibd files outside of structure, droping non-existing tables and discarding their tablespaces. After that tables re-created from generated structures, *.ibd files returned, corrected permissions and ownership and imported tablespace. MariaDB restarted to testing user software. Everything seems fine... Next day: Backup of database cannot be done. User software failing for MariaDB reconnecting issue. After investigation the same error report in messages as reported soon. Any select from any database crashed MariaDB. User software cannot be used. What do you recommend to resurrect normal production? I can go to backups of course and re-install the whole MariaDB with all databases. The pity is that user software is backuping software Bacula and last changes on tapes are written only in database which didn't backup. I have to try restore database dump created immediately after last backup to solve the situation. Is it any real chance how to restore functionallity without reinstalling everything from backup? Thanks in advance. Jan 13 23:39:08 lorlen mysqld: 2020-01-13 23:39:08 0x7f07f40f9700 InnoDB: Assertion failure in file /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.3.21/storage/innobase/btr/btr0btr.cc line 205 Jan 13 23:39:08 lorlen mysqld: InnoDB: Failing assertion: mach_read_from_4(seg_header + FSEG_HDR_SPACE) == space Jan 13 23:39:08 lorlen mysqld: InnoDB: We intentionally generate a memory trap. Jan 13 23:39:08 lorlen mysqld: InnoDB: Submit a detailed bug report to https://jira.mariadb.org/ Jan 13 23:39:08 lorlen mysqld: InnoDB: If you get repeated assertion failures or crashes, even Jan 13 23:39:08 lorlen mysqld: InnoDB: immediately after the mysqld startup, there may be Jan 13 23:39:08 lorlen mysqld: InnoDB: corruption in the InnoDB tablespace. Please refer to Jan 13 23:39:08 lorlen mysqld: InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/ Jan 13 23:39:08 lorlen mysqld: InnoDB: about forcing recovery. Jan 13 23:39:08 lorlen mysqld: 200113 23:39:08 [ERROR] mysqld got signal 6 ; Jan 13 23:39:08 lorlen mysqld: This could be because you hit a bug. It is also possible that this binary Jan 13 23:39:08 lorlen mysqld: or one of the libraries it was linked against is corrupt, improperly built, Jan 13 23:39:08 lorlen mysqld: or misconfigured. This error can also be caused by malfunctioning hardware. Jan 13 23:39:08 lorlen mysqld: To report this bug, see https://mariadb.com/kb/en/reporting-bugs Jan 13 23:39:08 lorlen mysqld: We will try our best to scrape up some info that will hopefully help Jan 13 23:39:08 lorlen mysqld: diagnose the problem, but since we have already crashed, Jan 13 23:39:08 lorlen mysqld: something is definitely wrong and this may fail. Jan 13 23:39:08 lorlen mysqld: Server version: 10.3.21-MariaDB Jan 13 23:39:08 lorlen mysqld: key_buffer_size=134217728 Jan 13 23:39:08 lorlen mysqld: read_buffer_size=131072 Jan 13 23:39:08 lorlen mysqld: max_used_connections=1 Jan 13 23:39:08 lorlen mysqld: max_threads=153 Jan 13 23:39:08 lorlen mysqld: thread_count=8 Jan 13 23:39:08 lorlen mysqld: It is possible that mysqld could use up to Jan 13 23:39:08 lorlen mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467424 K bytes of memory Jan 13 23:39:08 lorlen mysqld: Hope that's ok; if not, decrease some variables in the equation. Jan 13 23:39:08 lorlen mysqld: Thread pointer: 0x7f07f0000a88 Jan 13 23:39:08 lorlen mysqld: Attempting backtrace. You can use the following information to find out Jan 13 23:39:08 lorlen mysqld: where mysqld died. If you see no messages after this, something went Jan 13 23:39:08 lorlen mysqld: terribly wrong... Jan 13 23:39:08 lorlen mysqld: stack_bottom = 0x7f07f40f8d30 thread_stack 0x49000 Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55e1e2346c0e] Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55e1e1de640f] Jan 13 23:39:08 lorlen mysqld: sigaction.c:0(__restore_rt)[0x7f0801bec5f0] Jan 13 23:39:08 lorlen mysqld: :0(__GI_raise)[0x7f07ffebd337] Jan 13 23:39:08 lorlen mysqld: :0(__GI_abort)[0x7f07ffebea28] Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(+0x4d3b5c)[0x55e1e1b24b5c] Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(+0xa8084f)[0x55e1e20d184f] Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(+0xa826bd)[0x55e1e20d36bd] Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(+0xa82891)[0x55e1e20d3891] Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(+0x4d4302)[0x55e1e1b25302] Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(+0xb03377)[0x55e1e2154377] Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(+0xb0378d)[0x55e1e215478d] Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(+0xaf2dbb)[0x55e1e2143dbb] Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(+0x93e489)[0x55e1e1f8f489] Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(_ZN7handler7ha_openEP5TABLEPKcijP11st_mem_rootP4ListI6StringE+0x47)[0x55e1e1deaff7] Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(_Z21open_table_from_shareP3THDP11TABLE_SHAREPK25st_mysql_const_lex_stringjjjP5TABLEbP4ListI6StringE+0xbb7)[0x55e1e1cbbc07] Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(_Z10open_tableP3THDP10TABLE_LISTP18Open_table_context+0xaac)[0x55e1e1ba9ebc] Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(_Z11open_tablesP3THDRK14DDL_options_stPP10TABLE_LISTPjjP19Prelocking_strategy+0xd57)[0x55e1e1bad307] Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(_Z29mysqld_show_create_get_fieldsP3THDP10TABLE_LISTP4ListI4ItemEP6String+0x1a2)[0x55e1e1c69da2] Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(_Z18mysqld_show_createP3THDP10TABLE_LIST+0x132)[0x55e1e1c6a6b2] Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x3387)[0x55e1e1c06b47] Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x22b)[0x55e1e1c0c44b] Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x1ca6)[0x55e1e1c0f036] Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(_Z10do_commandP3THD+0x13e)[0x55e1e1c0ff7e] Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x221)[0x55e1e1ce4e91] Jan 13 23:39:08 lorlen mysqld: /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55e1e1ce4f5d] Jan 13 23:39:08 lorlen mysqld: pthread_create.c:0(start_thread)[0x7f0801be4e65] Jan 13 23:39:08 lorlen mysqld: /lib64/libc.so.6(clone+0x6d)[0x7f07fff8588d] Jan 13 23:39:08 lorlen mysqld: Trying to get some variables. Jan 13 23:39:08 lorlen mysqld: Some pointers may be invalid and cause the dump to abort. Jan 13 23:39:08 lorlen mysqld: Query (0x7f07f000f4c0): show create table `BaseFiles` Jan 13 23:39:08 lorlen mysqld: Connection ID (thread ID): 10 Jan 13 23:39:08 lorlen mysqld: Status: NOT_KILLED Jan 13 23:39:08 lorlen mysqld: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on Jan 13 23:39:08 lorlen mysqld: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains Jan 13 23:39:08 lorlen mysqld: information that should help you find out what is causing the crash. Jan 13 23:39:08 lorlen mysqld: Writing a core file... Jan 13 23:39:08 lorlen mysqld: Working directory at /var/lib/mysql Jan 13 23:39:08 lorlen mysqld: Resource Limits: Jan 13 23:39:08 lorlen mysqld: Limit Soft Limit Hard Limit Units Jan 13 23:39:08 lorlen mysqld: Max cpu time unlimited unlimited seconds Jan 13 23:39:08 lorlen mysqld: Max file size unlimited unlimited bytes Jan 13 23:39:08 lorlen mysqld: Max data size unlimited unlimited bytes Jan 13 23:39:08 lorlen mysqld: Max stack size 8388608 unlimited bytes Jan 13 23:39:08 lorlen mysqld: Max core file size 0 unlimited bytes Jan 13 23:39:08 lorlen mysqld: Max resident set unlimited unlimited bytes Jan 13 23:39:08 lorlen mysqld: Max processes 7258 7258 processes Jan 13 23:39:08 lorlen mysqld: Max open files 16364 16364 files Jan 13 23:39:08 lorlen mysqld: Max locked memory 65536 65536 bytes Jan 13 23:39:08 lorlen mysqld: Max address space unlimited unlimited bytes Jan 13 23:39:08 lorlen mysqld: Max file locks unlimited unlimited locks Jan 13 23:39:08 lorlen mysqld: Max pending signals 7258 7258 signals Jan 13 23:39:08 lorlen mysqld: Max msgqueue size 819200 819200 bytes Jan 13 23:39:08 lorlen mysqld: Max nice priority 0 0 Jan 13 23:39:08 lorlen mysqld: Max realtime priority 0 0 Jan 13 23:39:08 lorlen mysqld: Max realtime timeout unlimited unlimited us Jan 13 23:39:08 lorlen systemd: mariadb.service: main process exited, code=killed, status=6/ABRT Jan 13 23:39:08 lorlen systemd: Unit mariadb.service entered failed state.

Marko Mäkelä added a comment - 2020-01-14 07:24

seally1186, it will take some time before we can analyze this bug. I suspect that the IMPORT TABLESPACE code is failing to adjust the FSEG_HDR_SPACE field in BTR_SEG_TOP and BTR_SEG_LEAF of the index root page. It could be that the reason that it avoids a crash when the server is not being restarted is that some metadata in dict_index_t will already be initialized during the import, and a call to btr_root_block_get() will be avoided.

A possible workaround could be to recompile the server without that assertion, or to make sure that UNIV_BTR_DEBUG is not defined during the compilation. The field FSEG_HDR_SPACE should actually be unnecessary, because the ‘inode’ page should always be in the same tablespace with the pointer.

Marko Mäkelä added a comment - 2020-01-14 07:24 seally1186 , it will take some time before we can analyze this bug. I suspect that the IMPORT TABLESPACE code is failing to adjust the FSEG_HDR_SPACE field in BTR_SEG_TOP and BTR_SEG_LEAF of the index root page. It could be that the reason that it avoids a crash when the server is not being restarted is that some metadata in dict_index_t will already be initialized during the import, and a call to btr_root_block_get() will be avoided. A possible workaround could be to recompile the server without that assertion, or to make sure that UNIV_BTR_DEBUG is not defined during the compilation. The field FSEG_HDR_SPACE should actually be unnecessary, because the ‘inode’ page should always be in the same tablespace with the pointer.

Michael Chang added a comment - 2020-01-19 21:20

Thank you for the update and workaround. I did some quick testing, and it does seem that commenting out the assert prevents the crash and allows the server to read the data files.

Michael Chang added a comment - 2020-01-19 21:20 Thank you for the update and workaround. I did some quick testing, and it does seem that commenting out the assert prevents the crash and allows the server to read the data files.

Marko Mäkelä added a comment - 2020-01-21 17:17

As noted in ~~MDEV-21549~~, the server can crash even without a shutdown.

kevg, please check all my remarks in ~~MDEV-21549~~. There are several problems related to this.

Marko Mäkelä added a comment - 2020-01-21 17:17 As noted in MDEV-21549 , the server can crash even without a shutdown. kevg , please check all my remarks in MDEV-21549 . There are several problems related to this.

Marko Mäkelä added a comment - 2020-04-15 12:08

It turns out that ~~MDEV-21549~~ fixed exactly this problem. For completeness, I backported the fix to 10.1

Marko Mäkelä added a comment - 2020-04-15 12:08 It turns out that MDEV-21549 fixed exactly this problem. For completeness, I backported the fix to 10.1

People

Assignee:: Marko Mäkelä

Reporter:: Michael Chang

Votes:: 5 Vote for this issue

Watchers:: 11 Start watching this issue

Dates

Created:: 2019-12-30 20:34

Updated:: 2020-05-04 08:26

Resolved:: 2020-04-15 12:08

Git Integration

Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.

MariaDB Server

Details

Description

Attachments

Attachments

Issue Links

Activity

People

Dates

Git Integration