[MDEV-22305] Server crash or DDL mismatch in myrocks::ha_rocksdb::info after adding partition Created: 2020-04-19  Updated: 2023-11-28

Status: Open
Project: MariaDB Server
Component/s: Partitioning, Storage Engine - RocksDB
Affects Version/s: 10.2, 10.3, 10.4, 10.5, 10.6, 10.7, 10.8, 10.9, 10.10
Fix Version/s: 10.4, 10.5, 10.6

Type: Bug Priority: Major
Reporter: Elena Stepanova Assignee: Sergei Petrunia
Resolution: Unresolved Votes: 1
Labels: None

Issue Links:
Relates
relates to MDEV-22304 Inconsistent behavior upon adding for... Open

 Description   

Note: It might be a side-effect of MDEV-22304, the first ALTER in the test is only possible because of it; or maybe it just produces the right structure, which I couldn't create otherwise.

--source include/have_partition.inc
 
INSTALL SONAME 'ha_rocksdb';
 
CREATE TABLE t1 (id INT, a VARCHAR(8192)) ENGINE=RocksDB;
ALTER  TABLE t1 DROP FOREIGN KEY xx, ADD FOREIGN KEY (a) REFERENCES x(x);
ALTER TABLE t1 PARTITION BY RANGE (id) (PARTITION p1 VALUES LESS THAN (10));
ALTER  TABLE t1 ADD KEY (a);
ALTER  TABLE t1 ADD PARTITION (PARTITION p2 VALUES LESS THAN (20));
SHOW CREATE TABLE t1;

10.4 632b1deb

#3  <signal handler called>
#4  0x00007ff9670be859 in myrocks::ha_rocksdb::info (this=0x61d0002dbf10, flag=26) at /data/src/10.4/storage/rocksdb/ha_rocksdb.cc:11043
#5  0x00007ff967099c91 in myrocks::ha_rocksdb::open (this=0x61d0002dbf10, name=0x7ff968629750 "./test/t1#P#p1", mode=2, test_if_locked=1042) at /data/src/10.4/storage/rocksdb/ha_rocksdb.cc:6786
#6  0x00005595d19f3d77 in handler::ha_open (this=0x61d0002dbf10, table_arg=0x6200000490f0, name=0x7ff968629750 "./test/t1#P#p1", mode=2, test_if_locked=1042, mem_root=0x0, partitions_to_open=0x0) at /data/src/10.4/sql/handler.cc:2754
#7  0x00005595d2c86e09 in ha_partition::open_read_partitions (this=0x61d0002db510, name_buff=0x7ff968629750 "./test/t1#P#p1", name_buff_size=513) at /data/src/10.4/sql/ha_partition.cc:8509
#8  0x00005595d2c5ddf6 in ha_partition::open (this=0x61d0002db510, name=0x61b000052398 "./test/t1", mode=2, test_if_locked=18) at /data/src/10.4/sql/ha_partition.cc:3591
#9  0x00005595d19f3d77 in handler::ha_open (this=0x61d0002db510, table_arg=0x6200000490f0, name=0x61b000052398 "./test/t1", mode=2, test_if_locked=18, mem_root=0x0, partitions_to_open=0x0) at /data/src/10.4/sql/handler.cc:2754
#10 0x00005595d1520236 in open_table_from_share (thd=0x62b00009a270, share=0x61b000051e10, alias=0x62b0000a13e0, db_stat=33, prgflag=8, ha_open_flags=18, outparam=0x6200000490f0, is_create_table=false, partitions_to_open=0x0) at /data/src/10.4/sql/table.cc:3949
#11 0x00005595d109a041 in open_table (thd=0x62b00009a270, table_list=0x62b0000a1398, ot_ctx=0x7ff96862a5c0) at /data/src/10.4/sql/sql_base.cc:2082
#12 0x00005595d10a3373 in open_and_process_table (thd=0x62b00009a270, tables=0x62b0000a1398, counter=0x7ff96862a790, flags=1024, prelocking_strategy=0x7ff96862a6a0, has_prelocking_list=false, ot_ctx=0x7ff96862a5c0) at /data/src/10.4/sql/sql_base.cc:3850
#13 0x00005595d10a5e68 in open_tables (thd=0x62b00009a270, options=..., start=0x7ff96862a7d0, counter=0x7ff96862a790, flags=1024, prelocking_strategy=0x7ff96862a6a0) at /data/src/10.4/sql/sql_base.cc:4322
#14 0x00005595d107cecd in open_tables (thd=0x62b00009a270, tables=0x7ff96862a7d0, counter=0x7ff96862a790, flags=1024) at /data/src/10.4/sql/sql_base.h:473
#15 0x00005595d13c3dc5 in mysqld_show_create_get_fields (thd=0x62b00009a270, table_list=0x62b0000a1398, field_list=0x7ff96862ab80, buffer=0x7ff96862abc0) at /data/src/10.4/sql/sql_show.cc:1305
#16 0x00005595d13c4b71 in mysqld_show_create (thd=0x62b00009a270, table_list=0x62b0000a1398) at /data/src/10.4/sql/sql_show.cc:1412
#17 0x00005595d12264db in mysql_execute_command (thd=0x62b00009a270) at /data/src/10.4/sql/sql_parse.cc:4331
#18 0x00005595d1240c91 in mysql_parse (thd=0x62b00009a270, rawbuf=0x62b0000a1290 "SHOW CREATE TABLE t1", length=20, parser_state=0x7ff96862dac0, is_com_multi=false, is_next_command=false) at /data/src/10.4/sql/sql_parse.cc:7900
#19 0x00005595d12169af in dispatch_command (command=COM_QUERY, thd=0x62b00009a270, packet=0x629000244271 "SHOW CREATE TABLE t1", packet_length=20, is_com_multi=false, is_next_command=false) at /data/src/10.4/sql/sql_parse.cc:1841
#20 0x00005595d12133da in do_command (thd=0x62b00009a270) at /data/src/10.4/sql/sql_parse.cc:1359
#21 0x00005595d15ea5a4 in do_handle_one_connection (connect=0x6110000439f0) at /data/src/10.4/sql/sql_connect.cc:1412
#22 0x00005595d15e9e46 in handle_one_connection (arg=0x6110000439f0) at /data/src/10.4/sql/sql_connect.cc:1316
#23 0x00007ff98ab5dfa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#24 0x00007ff98a3ac4cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Reproducible on 10.2-10.5, debug and non-debug alike.
If it doesn't crash for you, try ASAN builds. Initially the failure on just slightly different test cases was observed as use-after-poison on ASAN builds, e.g.

==32178==ERROR: AddressSanitizer: use-after-poison on address 0x61900004b468 at pc 0x7f353309082a bp 0x7f3517911f90 sp 0x7f3517911f88
READ of size 8 at 0x61900004b468 thread T50
    #0 0x7f3533090829 in myrocks::ha_rocksdb::info(unsigned int) /data/src/10.4/storage/rocksdb/ha_rocksdb.cc:11043
    #1 0x7f353306bc90 in myrocks::ha_rocksdb::open(char const*, int, unsigned int) /data/src/10.4/storage/rocksdb/ha_rocksdb.cc:6786
    #2 0x55732e133d76 in handler::ha_open(TABLE*, char const*, int, unsigned int, st_mem_root*, List<String>*) /data/src/10.4/sql/handler.cc:2754
    #3 0x55732f3c6e08 in ha_partition::open_read_partitions(char*, unsigned long) /data/src/10.4/sql/ha_partition.cc:8509
    #4 0x55732f39ddf5 in ha_partition::open(char const*, int, unsigned int) /data/src/10.4/sql/ha_partition.cc:3591
    #5 0x55732e133d76 in handler::ha_open(TABLE*, char const*, int, unsigned int, st_mem_root*, List<String>*) /data/src/10.4/sql/handler.cc:2754
    #6 0x55732dc60235 in open_table_from_share(THD*, TABLE_SHARE*, st_mysql_const_lex_string const*, unsigned int, unsigned int, unsigned int, TABLE*, bool, List<String>*) /data/src/10.4/sql/table.cc:3949
    #7 0x55732d7da040 in open_table(THD*, TABLE_LIST*, Open_table_context*) /data/src/10.4/sql/sql_base.cc:2082
    #8 0x55732d7e3372 in open_and_process_table /data/src/10.4/sql/sql_base.cc:3850
    #9 0x55732d7e5e67 in open_tables(THD*, DDL_options_st const&, TABLE_LIST**, unsigned int*, unsigned int, Prelocking_strategy*) /data/src/10.4/sql/sql_base.cc:4322
    #10 0x55732d7bcecc in open_tables(THD*, TABLE_LIST**, unsigned int*, unsigned int) /data/src/10.4/sql/sql_base.h:473
    #11 0x55732db03dc4 in mysqld_show_create_get_fields(THD*, TABLE_LIST*, List<Item>*, String*) /data/src/10.4/sql/sql_show.cc:1305
    #12 0x55732db04b70 in mysqld_show_create(THD*, TABLE_LIST*) /data/src/10.4/sql/sql_show.cc:1412
    #13 0x55732d9664da in mysql_execute_command(THD*) /data/src/10.4/sql/sql_parse.cc:4331
    #14 0x55732d980c90 in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) /data/src/10.4/sql/sql_parse.cc:7900
    #15 0x55732d9569ae in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /data/src/10.4/sql/sql_parse.cc:1841
    #16 0x55732d9533d9 in do_command(THD*) /data/src/10.4/sql/sql_parse.cc:1359
    #17 0x55732dd2a5a3 in do_handle_one_connection(CONNECT*) /data/src/10.4/sql/sql_connect.cc:1412
    #18 0x55732dd29e45 in handle_one_connection /data/src/10.4/sql/sql_connect.cc:1316
    #19 0x7f3540169fa2 in start_thread /build/glibc-vjB4T1/glibc-2.28/nptl/pthread_create.c:486
    #20 0x7f353f9b84ce in clone (/lib/x86_64-linux-gnu/libc.so.6+0xf94ce)
 
0x61900004b468 is located 1000 bytes inside of 1100-byte region [0x61900004b080,0x61900004b4cc)
allocated by thread T50 here:
    #0 0x7f354026c330 in __interceptor_malloc (/lib/x86_64-linux-gnu/libasan.so.5+0xe9330)
    #1 0x55732f491a77 in sf_malloc /data/src/10.4/mysys/safemalloc.c:118
    #2 0x55732f460d2d in my_malloc /data/src/10.4/mysys/my_malloc.c:101
    #3 0x55732f43dafe in alloc_root /data/src/10.4/mysys/my_alloc.c:250
    #4 0x55732f43f223 in strmake_root /data/src/10.4/mysys/my_alloc.c:480
    #5 0x55732dc5d70a in open_table_from_share(THD*, TABLE_SHARE*, st_mysql_const_lex_string const*, unsigned int, unsigned int, unsigned int, TABLE*, bool, List<String>*) /data/src/10.4/sql/table.cc:3622
    #6 0x55732d7da040 in open_table(THD*, TABLE_LIST*, Open_table_context*) /data/src/10.4/sql/sql_base.cc:2082
    #7 0x55732d7e3372 in open_and_process_table /data/src/10.4/sql/sql_base.cc:3850
    #8 0x55732d7e5e67 in open_tables(THD*, DDL_options_st const&, TABLE_LIST**, unsigned int*, unsigned int, Prelocking_strategy*) /data/src/10.4/sql/sql_base.cc:4322
    #9 0x55732d7bcecc in open_tables(THD*, TABLE_LIST**, unsigned int*, unsigned int) /data/src/10.4/sql/sql_base.h:473
    #10 0x55732db03dc4 in mysqld_show_create_get_fields(THD*, TABLE_LIST*, List<Item>*, String*) /data/src/10.4/sql/sql_show.cc:1305
    #11 0x55732db04b70 in mysqld_show_create(THD*, TABLE_LIST*) /data/src/10.4/sql/sql_show.cc:1412
    #12 0x55732d9664da in mysql_execute_command(THD*) /data/src/10.4/sql/sql_parse.cc:4331
    #13 0x55732d980c90 in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) /data/src/10.4/sql/sql_parse.cc:7900
    #14 0x55732d9569ae in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /data/src/10.4/sql/sql_parse.cc:1841
    #15 0x55732d9533d9 in do_command(THD*) /data/src/10.4/sql/sql_parse.cc:1359
    #16 0x55732dd2a5a3 in do_handle_one_connection(CONNECT*) /data/src/10.4/sql/sql_connect.cc:1412
    #17 0x55732dd29e45 in handle_one_connection /data/src/10.4/sql/sql_connect.cc:1316
    #18 0x7f3540169fa2 in start_thread /build/glibc-vjB4T1/glibc-2.28/nptl/pthread_create.c:486
 
Thread T50 created by T0 here:
    #0 0x7f35401d3db0 in __interceptor_pthread_create (/lib/x86_64-linux-gnu/libasan.so.5+0x50db0)
    #1 0x55732f4c3def in spawn_thread_noop /data/src/10.4/mysys/psi_noop.c:187
    #2 0x55732d6736c5 in inline_mysql_thread_create /data/src/10.4/include/mysql/psi/mysql_thread.h:1275
    #3 0x55732d68ac78 in create_thread_to_handle_connection(CONNECT*) /data/src/10.4/sql/mysqld.cc:6243
    #4 0x55732d68b3cd in create_new_thread(CONNECT*) /data/src/10.4/sql/mysqld.cc:6313
    #5 0x55732d68b8a7 in handle_accepted_socket(st_mysql_socket, st_mysql_socket) /data/src/10.4/sql/mysqld.cc:6411
    #6 0x55732d68c754 in handle_connections_sockets() /data/src/10.4/sql/mysqld.cc:6569
    #7 0x55732d68a3dc in mysqld_main(int, char**) /data/src/10.4/sql/mysqld.cc:5901
    #8 0x55732d671594 in main /data/src/10.4/sql/main.cc:25
    #9 0x7f353f8e309a in __libc_start_main ../csu/libc-start.c:308
 
SUMMARY: AddressSanitizer: use-after-poison /data/src/10.4/storage/rocksdb/ha_rocksdb.cc:11043 in myrocks::ha_rocksdb::info(unsigned int)
Shadow bytes around the buggy address:
  0x0c3280001630: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c3280001640: 00 00 00 00 00 00 00 00 f7 00 00 00 00 00 00 00
  0x0c3280001650: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c3280001660: 00 00 00 00 00 00 f7 00 00 00 00 00 00 00 00 00
  0x0c3280001670: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c3280001680: 00 00 f7 00 00 00 00 00 f7 02 f7 00 00[f7]03 f7
  0x0c3280001690: 07 f7 00 00 f7 03 f7 f7 f7 04 fa fa fa fa fa fa
  0x0c32800016a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c32800016b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c32800016c0: f7 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c32800016d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==32178==ABORTING



 Comments   
Comment by Elena Stepanova [ 2022-09-17 ]

Current builds don't crash, at least on my machine, but still produce a corruption-like error:

10.3 32bab2ce

mysqltest: At line 17: query 'SHOW CREATE TABLE t1' failed: 1815: Internal error: MyRocks: DDL mismatch. Check the error log for details

error log

2022-09-17 23:38:14 4 [ERROR] MyRocks: DDL mismatch: .frm file has 1 indexes, MyRocks has 3 (1 hidden pk)

Generated at Thu Feb 08 09:13:44 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.