[MDEV-26155] Mangled data after IMPORT tablespace up till assert for DDL Created: 2021-07-15  Updated: 2024-01-05

Status: Open
Project: MariaDB Server
Component/s: Storage Engine - InnoDB
Affects Version/s: 10.6.3
Fix Version/s: 10.6

Type: Bug Priority: Major
Reporter: Matthias Leich Assignee: Marko Mäkelä
Resolution: Unresolved Votes: 0
Labels: None

Attachments: File ML_importB.test    
Issue Links:
Blocks
is blocked by MDEV-24621 In bulk insert, pre-sort and build in... Closed
Relates
relates to MDEV-26137 ALTER TABLE IMPORT enhancement Closed

 Description   

CREATE TABLE `t1` (
`col_varchar_255` varchar(255),
`col_text` text,
`col_int` int) ENGINE=innodb ;
INSERT /*! IGNORE */ INTO t1 VALUES 
('director', 'rifle', 9) ,
('commissioner', 'examination', 4) ,
('y', 'intervention', NULL) ,
('assure', 'previously', 2) ,
('m', 'appointment', 559480832) ,
('opposed', 'miracle', NULL) ,
('planet', 'prayer', -1602027520) ,
('hug', 'write', 5) ,
('y', 'versus', 5) ,
('n', 'preliminary', 64290816) ;
FLUSH TABLES t1 FOR EXPORT ;
'copy ibd file to safe place''
UNLOCK TABLES ;
CREATE TABLE imp_t1 (
`col_varchar_255` varchar(255) PRIMARY KEY,
`col_text` text,
`col_int` int
) ENGINE=innodb ;
ALTER TABLE imp_t1 DISCARD TABLESPACE ;
'copy ibd file around'
ALTER TABLE imp_t1 IMPORT TABLESPACE ;
Warnings:
Warning 1810    IO Read error: (2, No such file or directory) Error opening './test/imp_t1.cfg', will attempt to import without schema verification
##### imp_t1 has crippled data
SELECT '->' S1, col_varchar_255, '<->' S2, col_text, '<->' S3, col_int, '<-' S4 FROM t1;
S1      col_varchar_255 S2      col_text        S3      col_int S4
->      director        <->     rifle   <->     9       <-
->      commissioner    <->     examination     <->     4       <-
->      y       <->     intervention    <->     NULL    <-
->      assure  <->     previously      <->     2       <-
->      m       <->     appointment     <->     559480832       <-
->      opposed <->     miracle <->     NULL    <-
->      planet  <->     prayer  <->     -1602027520     <-
->      hug     <->     write   <->     5       <-
->      y       <->     versus  <->     5       <-
->      n       <->     preliminary     <->     64290816        <-
SELECT '->' S1, col_varchar_255, '<->' S2, col_text, '<->' S3, col_int, '<-' S4 FROM imp_t1;
S1      col_varchar_255 S2      col_text        S3      col_int S4
->              <->     recto   <->     -227382938      <-
->              <->     sionerexami     <->     -295603095      <-
->              <->     yinterv <->     -445746071      <-
->              <->     assureprev      <->     -378571405      <-
->              <->     mappoi  <->     -294359707      <-
->              <->     pposedm <->     -378379933      <-
->              <->     planet  <->     -260939399      <-
->              <->     hu      <->     -411602327      <-
->              <->     y       <->     -161123725      <-
->              <->     npreli  <->     -311857567      <-
##### expect to crash with btr0bulk.cc:176: void PageBulk::insertPage(rec_t*, rec_offs*) [with PageBulk::format <anonymous> = PageBulk::DYNAMIC; rec_t = unsigned char; rec_offs = short unsigned int]: Assertion `cmp_rec_rec(rec, old_rec, offsets, old_offsets, m_index) > 0' failed. ####
ALTER TABLE imp_t1 ADD KEY idx ( col_varchar_255 ) ;
ERROR HY000: Lost connection to server during query
DROP TABLE t1, imp_t1 ;
ERROR HY000: Server has gone away
 
mariadbd: /data/Server/bb-10.6-MDEV-26131/storage/innobase/btr/btr0bulk.cc:176: void PageBulk::insertPage(rec_t*, rec_offs*) [with PageBulk::format <anonymous> = PageBulk::DYNAMIC; rec_t = unsigned char; rec_offs = short unsigned int]: Assertion `cmp_rec_rec(rec, old_rec, offsets, old_offsets, m_index) > 0' failed.
210715  9:48:57 [ERROR] mysqld got signal 6 ;
 
mysys/stacktrace.c:212(my_print_stacktrace)[0x5598436e122e]
sql/signal_handler.cc:225(handle_fatal_signal)[0x55984245f373]
sigaction.c:0(__restore_rt)[0x7fdc7ad8b3c0]
linux/raise.c:51(__GI_raise)[0x7fdc7a87718b]
stdlib/abort.c:81(__GI_abort)[0x7fdc7a856859]
intl/loadmsgcat.c:509(get_sysdep_segment_value)[0x7fdc7a856729]
:0(__GI___assert_fail)[0x7fdc7a867f36]
btr/btr0bulk.cc:179(void PageBulk::insertPage<(PageBulk::format)1>(unsigned char*, unsigned short*))[0x559843288472]
btr/btr0bulk.cc:335(PageBulk::insert(unsigned char const*, unsigned short*))[0x5598432846ae]
btr/btr0bulk.cc:1130(BtrBulk::insert(dtuple_t*, unsigned long))[0x5598432826ef]
include/btr0bulk.h:302(BtrBulk::insert(dtuple_t*))[0x55984305a2c8]
row/row0merge.cc:3630(row_merge_insert_index_tuples(dict_index_t*, dict_table_t const*, pfs_os_file_t const&, unsigned char*, row_merge_buf_t const*, BtrBulk*, unsigned long, double, double, unsigned char*, unsigned long, ut_stage_alter_t*))[0x5598430bf5d5]
row/row0merge.cc:2622(row_merge_read_clustered_index(trx_t*, TABLE*, dict_table_t const*, dict_table_t*, bool, dict_index_t**, dict_index_t*, fts_psort_t*, merge_file_t*, unsigned long const*, unsigned long, dtuple_t const*, dict_add_v_col_t const*, unsigned long const*, unsigned long, ib_sequence_t&, unsigned char*, bool, pfs_os_file_t*, ut_stage_alter_t*, double, unsigned char*, TABLE*, bool))[0x5598430b842a]
row/row0merge.cc:4548(row_merge_build_indexes(trx_t*, dict_table_t*, dict_table_t*, bool, dict_index_t**, unsigned long const*, unsigned long, TABLE*, dtuple_t const*, unsigned long const*, unsigned long, ib_sequence_t&, bool, ut_stage_alter_t*, dict_add_v_col_t const*, TABLE*, bool))[0x5598430c471f]
handler/handler0alter.cc:8463(ha_innobase::inplace_alter_table(TABLE*, Alter_inplace_info*))[0x559842e09bcc]
sql/handler.h:4684(handler::ha_inplace_alter_table(TABLE*, Alter_inplace_info*))[0x559841f2a8d6]
sql/sql_table.cc:7378(mysql_inplace_alter_table(THD*, TABLE_LIST*, TABLE*, TABLE*, Alter_inplace_info*, MDL_request*, st_ddl_log_state*, TRIGGER_RENAME_PARAM*, Alter_table_ctx*))[0x559841f09f34]
sql/sql_table.cc:10207(mysql_alter_table(THD*, st_mysql_const_lex_string const*, st_mysql_const_lex_string const*, HA_CREATE_INFO*, TABLE_LIST*, Alter_info*, unsigned int, st_order*, bool, bool))[0x559841f1da3a]
sql/sql_alter.cc:550(Sql_cmd_alter_table::execute(THD*))[0x5598420a9fe5]
sql/sql_parse.cc:5993(mysql_execute_command(THD*, bool))[0x559841cb0d1a]
sql/sql_parse.cc:8026(mysql_parse(THD*, char*, unsigned int, Parser_state*))[0x559841cbd1ec]
sql/sql_parse.cc:1898(dispatch_command(enum_server_command, THD*, char*, unsigned int, bool))[0x559841c9548b]
sql/sql_parse.cc:1404(do_command(THD*, bool))[0x559841c92863]
sql/sql_connect.cc:1410(do_handle_one_connection(CONNECT*, bool))[0x5598420904c2]
sql/sql_connect.cc:1314(handle_one_connection)[0x55984208fe24]
nptl/pthread_create.c:478(start_thread)[0x7fdc7ad7f609]
x86_64/clone.S:97(__GI___clone)[0x7fdc7a953293]
 
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x62b0000d2238): ALTER TABLE imp_t1 ADD KEY idx ( col_varchar_255 )
 
Connection ID (thread ID): 4
Status: NOT_KILLED
 
origin/bb-10.6-MDEV-26131 ffc41e1cf56809f82e97daad4869fecac4b46fed 2021-07-13T23:31:16+05:30  build with debug and ASAN.
Some build without debug has the same mangled data but does not assert.
 
origin/10.6, 10.6 315380a4d16ddb16461d906a23be341e354c30ed 2021-07-01T19:06:53+03:00  build with debug
The same mangled data and
mariadbd: /data/Server/10.6G/storage/innobase/btr/btr0bulk.cc:176: void PageBulk::insertPage(rec_t*, rec_offs*) [with PageBulk::format <anonymous> = PageBulk::DYNAMIC; rec_t = unsigned char; rec_offs = short unsigned int]: Assertion `cmp_rec_rec(rec, old_rec, offsets, old_offsets, m_index) > 0' failed.
Thread 1 (Thread 0x7f4152884700 (LWP 3855140)):
#0  __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
#1  0x000055c131a124a1 in my_write_core (sig=6) at /data/Server/10.6G/mysys/stacktrace.c:424
#2  0x000055c1311da6d3 in handle_fatal_signal (sig=6) at /data/Server/10.6G/sql/signal_handler.cc:344
#3  <signal handler called>
#4  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#5  0x00007f41582e8859 in __GI_abort () at abort.c:79
#6  0x00007f41582e8729 in __assert_fail_base (fmt=0x7f415847e588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x55c131f62ea0 "cmp_rec_rec(rec, old_rec, offsets, old_offsets, m_index) > 0", file=0x55c131f61b28 "/data/Server/10.6G/storage/innobase/btr/btr0bulk.cc", line=176, function=<optimized out>) at assert.c:92
#7  0x00007f41582f9f36 in __GI___assert_fail (assertion=0x55c131f62ea0 "cmp_rec_rec(rec, old_rec, offsets, old_offsets, m_index) > 0", file=0x55c131f61b28 "/data/Server/10.6G/storage/innobase/btr/btr0bulk.cc", line=176, function=0x55c131f62f38 "void PageBulk::insertPage(rec_t*, rec_offs*) [with PageBulk::format <anonymous> = PageBulk::DYNAMIC; rec_t = unsigned char; rec_offs = short unsigned int]") at assert.c:101
#8  0x000055c1318181c8 in PageBulk::insertPage<(PageBulk::format)1> (this=0x7f4128069700, rec=0x7f4128069e7e "", offsets=0x7f4128069e80) at /data/Server/10.6G/storage/innobase/btr/btr0bulk.cc:176
#9  0x000055c13181654a in PageBulk::insert (this=0x7f4128069700, rec=0x7f4128069e7e "", offsets=0x7f4128069e80) at /data/Server/10.6G/storage/innobase/btr/btr0bulk.cc:334
#10 0x000055c1318153b3 in BtrBulk::insert (this=0x7f415287ee70, tuple=0x7f41280596f8, level=0) at /data/Server/10.6G/storage/innobase/btr/btr0bulk.cc:1128
#11 0x000055c1316fae46 in BtrBulk::insert (this=0x7f415287ee70, tuple=0x7f41280596f8) at /data/Server/10.6G/storage/innobase/include/btr0bulk.h:301
#12 0x000055c13172fa2a in row_merge_insert_index_tuples (index=0x7f4128057f10, old_table=0x7f4128175050, fd=..., block=0x0, row_buf=0x7f41281100f0, btr_bulk=0x7f415287ee70, table_total_rows=9, pct_progress=0, pct_cost=50, crypt_block=0x0, space=6, stage=0x0) at /data/Server/10.6G/storage/innobase/row/row0merge.cc:3630
#13 0x000055c13172c62a in row_merge_read_clustered_index (trx=0x7f4153c001b8, table=0x7f41528803f0, old_table=0x7f4128175050, new_table=0x7f4128175050, online=true, index=0x7f4128057480, fts_sort_idx=0x0, psort_info=0x0, files=0x7f412807ec30, key_numbers=0x7f4128057488, n_index=1, defaults=0x0, add_v=0x0, col_map=0x0, add_autoinc=18446744073709551615, sequence=..., block=0x7f41404f9000 <error: Cannot access memory at address 0x7f41404f9000>, skip_pk_sort=false, tmpfd=0x7f415287f5a0, stage=0x7f412810b500, pct_cost=50, crypt_block=0x0, eval_table=0x7f41528803f0, allow_not_null=false) at /data/Server/10.6G/storage/innobase/row/row0merge.cc:2622
#14 0x000055c131732568 in row_merge_build_indexes (trx=0x7f4153c001b8, old_table=0x7f4128175050, new_table=0x7f4128175050, online=true, indexes=0x7f4128057480, key_numbers=0x7f4128057488, n_indexes=1, table=0x7f41528803f0, defaults=0x0, col_map=0x0, add_autoinc=18446744073709551615, sequence=..., skip_pk_sort=false, stage=0x7f412810b500, add_v=0x0, eval_table=0x7f41528803f0, allow_not_null=false) at /data/Server/10.6G/storage/innobase/row/row0merge.cc:4548
#15 0x000055c1315dab3c in ha_innobase::inplace_alter_table (this=0x7f4128050fb0, altered_table=0x7f41528803f0, ha_alter_info=0x7f4152880330) at /data/Server/10.6G/storage/innobase/handler/handler0alter.cc:8463
#16 0x000055c130f71e69 in handler::ha_inplace_alter_table (this=0x7f4128050fb0, altered_table=0x7f41528803f0, ha_alter_info=0x7f4152880330) at /data/Server/10.6G/sql/handler.h:4681
#17 0x000055c130f63d2d in mysql_inplace_alter_table (thd=0x7f4128000d48, table_list=0x7f41280142d0, table=0x7f412804bd58, altered_table=0x7f41528803f0, ha_alter_info=0x7f4152880330, target_mdl_request=0x7f4152880bf0, ddl_log_state=0x7f41528802f0, trigger_param=0x7f41528807c0, alter_ctx=0x7f4152881750) at /data/Server/10.6G/sql/sql_table.cc:7378
#18 0x000055c130f6c3e7 in mysql_alter_table (thd=0x7f4128000d48, new_db=0x7f4128005758, new_name=0x7f4128005b70, create_info=0x7f4152882540, table_list=0x7f41280142d0, alter_info=0x7f4152882450, order_num=0, order=0x0, ignore=false, if_exists=false) at /data/Server/10.6G/sql/sql_table.cc:10202
#19 0x000055c131015231 in Sql_cmd_alter_table::execute (this=0x7f4128014aa0, thd=0x7f4128000d48) at /data/Server/10.6G/sql/sql_alter.cc:550
#20 0x000055c130e76112 in mysql_execute_command (thd=0x7f4128000d48, is_called_from_prepared_stmt=false) at /data/Server/10.6G/sql/sql_parse.cc:5995
#21 0x000055c130e7be88 in mysql_parse (thd=0x7f4128000d48, rawbuf=0x7f41280141c0 "ALTER TABLE imp_t1 ADD KEY idx ( col_varchar_255 )", length=50, parser_state=0x7f4152883580) at /data/Server/10.6G/sql/sql_parse.cc:8028
#22 0x000055c130e68a01 in dispatch_command (command=COM_QUERY, thd=0x7f4128000d48, packet=0x7f4128087079 "ALTER TABLE imp_t1 ADD KEY idx ( col_varchar_255 ) ", packet_length=51, blocking=true) at /data/Server/10.6G/sql/sql_parse.cc:1898
#23 0x000055c130e6750b in do_command (thd=0x7f4128000d48, blocking=true) at /data/Server/10.6G/sql/sql_parse.cc:1406
#24 0x000055c13100ad3a in do_handle_one_connection (connect=0x55c13438d618, put_in_cache=true) at /data/Server/10.6G/sql/sql_connect.cc:1410
#25 0x000055c13100aa9b in handle_one_connection (arg=0x55c1343841a8) at /data/Server/10.6G/sql/sql_connect.cc:1312
#26 0x00007f4158811609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#27 0x00007f41583e5293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
I am aware that the CREATE TABLE statement of the target table differs a bit.
Nevertheless I would appreciate if the server/InnoDB would somehow detect that there seems to be something
wrong and abort the import of the tablespace.



 Comments   
Comment by Marko Mäkelä [ 2021-07-20 ]

I think that the important thing to note is

Warning 1810    IO Read error: (2, No such file or directory) Error opening './test/imp_t1.cfg', will attempt to import without schema verification

The IMPORT TABLESPACE feature is basically unsafe by design. I believe that it is theoretically impossible to detect every possible case of metadata mismatch if the user is trying to import a corrupted file, or trying to import a file using wrong metadata.

I think that a reasonable path forward would be to remove the DISCARD TABLESPACE/IMPORT TABLESPACE feature altogether and to prefer logical transfer. I hope that MDEV-24621 will make logical imports reasonably fast. Logical dumps could be sped up by MDEV-16232.

Comment by Marko Mäkelä [ 2024-01-05 ]

MDEV-26137 introduced a simplified workflow where CREATE TABLE and ALTER TABLE…DISCARD TABLESPACE are not used. The table would be imported based on .frm and .ibd files. As far as I understand, in that workflow, the only way you could get this type of error would be when the copied .frm and .ibd files do not correspond to each other.

Generated at Thu Feb 08 09:43:09 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.