[MDEV-26677] SIGABRT in myrocks::rdb_i_s_global_info_fill_table / [ERROR] RocksDB: Failed to get column family flags from CF with id = 2. MyRocks data dictionary may be corrupted. - Jira

Details

Type: Bug
Status: Confirmed (View Workflow)
Priority: Major
Resolution: Unresolved
Affects Version/s: 10.2(EOL), 10.3(EOL), 10.4(EOL), 10.5, 10.6, 10.7(EOL), 10.11, 11.1(EOL), 11.2(EOL), 11.4, 11.6(EOL), 11.7(EOL)
Fix Version/s: 10.5, 10.6, 10.11, 11.4
Component/s: Storage Engine - RocksDB
Labels:
- corruption
- sporadic

Description

The crash is highly sporadic. The "MyRocks data dictionary may be corrupted." message is very lightly sporadic and shows almost always. The testcase for both the message and the crash is identical. To make the crash a bit less sporadic, I added a SELECT SLEEP(10); to the end of the testcase. CLI Based replay crash reproducibility.

INSTALL SONAME 'ha_rocksdb';

SET GLOBAL rocksdb_update_cf_options='DEFAULT={write_buffer_size=8m};';

SELECT COUNT(*) FROM information_schema.rocksdb_global_info;

SELECT SLEEP(10);

This leads to either the error message below, plus a crash, or just the error message without (or with a delayed/upcoming) crash:

10.7.0 d552e092c9f3e20da078d1b62b976f629f73d3a4 (Optimized)
Version: '10.7.0-MariaDB' socket: '/test/MD180921-mariadb-10.7.0-linux-x86_64-opt/socket.sock' port: 38673 MariaDB Server
2021-09-25 14:09:25 4 [Note] RocksDB: Got ENOENT when listing column families
2021-09-25 14:09:25 4 [Note] RocksDB: assuming that we're creating a new database
2021-09-25 14:09:25 4 [Note] RocksDB: Column Families at start:
2021-09-25 14:09:25 4 [Note] cf=default
2021-09-25 14:09:25 4 [Note] write_buffer_size=67108864
2021-09-25 14:09:25 4 [Note] target_file_size_base=67108864
2021-09-25 14:09:25 4 [Note] RocksDB: creating a column family __system__
2021-09-25 14:09:25 4 [Note] write_buffer_size=67108864
2021-09-25 14:09:25 4 [Note] target_file_size_base=67108864
2021-09-25 14:09:25 4 [Note] RocksDB: Table_store: loaded DDL data for 0 tables
2021-09-25 14:09:25 4 [Note] RocksDB: global statistics using get_sched_indexer_t indexer
2021-09-25 14:09:25 4 [Note] MyRocks storage engine plugin has been successfully initialized.
2021-09-25 14:09:25 4 [Note] RocksDB: creating a column family DEFAULT
2021-09-25 14:09:25 4 [Note] write_buffer_size=67108864
2021-09-25 14:09:25 4 [Note] target_file_size_base=67108864
2021-09-25 14:09:25 4 [Note] MyRocks: options for column family 'DEFAULT' have been successfully updated.
2021-09-25 14:09:25 4 [ERROR] RocksDB: Failed to get column family flags from CF with id = 2. MyRocks data dictionary may be corrupted.

10.7.0 d552e092c9f3e20da078d1b62b976f629f73d3a4 (Optimized)
Core was generated by `/test/MD180921-mariadb-10.7.0-linux-x86_64-opt/bin/mysqld --no-defaults --core-'.
Program terminated with signal SIGABRT, Aborted.
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
[Current thread is 1 (Thread 0x1538a8cc2700 (LWP 3844856))]
(gdb) bt
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1 0x00001538b410e859 in __GI_abort () at abort.c:79
#2 0x000015387e27e181 in myrocks::rdb_i_s_global_info_fill_table (thd=0x153830000c58, tables=0x153830010f08, cond=<optimized out>) at /test/10.7_opt/storage/rocksdb/rdb_i_s.cc:803
#3 0x0000564b5c0dec3e in get_schema_tables_result (join=join@entry=0x1538300120e8, executed_place=executed_place@entry=PROCESSED_BY_JOIN_EXEC) at /test/10.7_opt/sql/sql_show.cc:8830
#4 0x0000564b5c0c22df in JOIN::exec_inner (this=0x1538300120e8) at /test/10.7_opt/sql/sql_select.cc:4691
#5 0x0000564b5c0c2b38 in JOIN::exec (this=this@entry=0x1538300120e8) at /test/10.7_opt/sql/sql_select.cc:4513
#6 0x0000564b5c0c0bda in mysql_select (thd=0x153830000c58, tables=0x153830010f08, fields=<optimized out>, conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=<optimized out>, result=0x1538300120c0, unit=0x153830004df0, select_lex=0x153830010868) at /test/10.7_opt/sql/sql_select.cc:4991
#7 0x0000564b5c0c13a7 in handle_select (thd=thd@entry=0x153830000c58, lex=lex@entry=0x153830004d28, result=result@entry=0x1538300120c0, setup_tables_done_option=setup_tables_done_option@entry=0) at /test/10.7_opt/sql/sql_select.cc:545
#8 0x0000564b5c044811 in execute_sqlcom_select (thd=0x153830000c58, all_tables=0x153830010f08) at /test/10.7_opt/sql/sql_parse.cc:6253
#9 0x0000564b5c0525a5 in mysql_execute_command (thd=0x153830000c58, is_called_from_prepared_stmt=<optimized out>) at /test/10.7_opt/sql/sql_parse.cc:3944
#10 0x0000564b5c03f826 in mysql_parse (thd=0x153830000c58, rawbuf=<optimized out>, length=<optimized out>, parser_state=<optimized out>) at /test/10.7_opt/sql/sql_parse.cc:8028
#11 0x0000564b5c04b705 in dispatch_command (command=COM_QUERY, thd=0x153830000c58, packet=<optimized out>, packet_length=<optimized out>, blocking=<optimized out>) at /test/10.7_opt/sql/sql_class.h:1358
#12 0x0000564b5c04d5d7 in do_command (thd=0x153830000c58, blocking=blocking@entry=true) at /test/10.7_opt/sql/sql_parse.cc:1402
#13 0x0000564b5c168ab7 in do_handle_one_connection (connect=<optimized out>, put_in_cache=true) at /test/10.7_opt/sql/sql_connect.cc:1418
#14 0x0000564b5c168dfd in handle_one_connection (arg=arg@entry=0x564b5f55a6e8) at /test/10.7_opt/sql/sql_connect.cc:1312
#15 0x0000564b5c4bba18 in pfs_spawn_thread (arg=0x564b5f4eaa38) at /test/10.7_opt/storage/perfschema/pfs.cc:2201
#16 0x00001538b461d609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#17 0x00001538b420b293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Bug confirmed present in:
MariaDB: 10.2.41 (dbg), 10.2.41 (opt), 10.3.32 (dbg), 10.3.32 (opt), 10.4.22 (dbg), 10.4.22 (opt), 10.5.13 (dbg), 10.5.13 (opt), 10.6.5 (dbg), 10.6.5 (opt), 10.7.0 (dbg), 10.7.0 (opt)

Attachments

Activity

Ascending order - Click to sort in descending order

Roel Van de Paar added a comment - 2021-09-25 04:24 - edited

Verify any fix with this testcase also please:

# mysqld options required for replay: --sql_mode=

INSTALL SONAME 'ha_rocksdb';

SET GLOBAL rocksdb_update_cf_options='cf1={write_buffer_size=8m;target_file_size_base=2m};cf2={write_buffer_size=16m;max_bytes_for_level_multiplier=8};cf3={target_file_size_base=4m};';

SELECT COUNT(*) FROM information_schema.rocksdb_global_info;

SELECT SLEEP(10);

Roel Van de Paar added a comment - 2021-09-25 04:24 - edited Verify any fix with this testcase also please: # mysqld options required for replay: --sql_mode= INSTALL SONAME 'ha_rocksdb' ; SET GLOBAL rocksdb_update_cf_options= 'cf1={write_buffer_size=8m;target_file_size_base=2m};cf2={write_buffer_size=16m;max_bytes_for_level_multiplier=8};cf3={target_file_size_base=4m};' ; SELECT COUNT (*) FROM information_schema.rocksdb_global_info; SELECT SLEEP(10);

Roel Van de Paar added a comment - 2024-09-15 23:42 - edited

Also,

INSTALL SONAME 'ha_rocksdb';

SET GLOBAL rocksdb_update_cf_options='cf0={prefix_extractor=capped:0};';

SELECT * FROM information_schema.rocksdb_global_info;

Leads to:

CS 11.7.0 5bbda9711131845ae6b4315a268b4d1710943a85 (Debug)
2024-09-16 9:43:05 4 [ERROR] RocksDB: Failed to get column family flags from CF with id = 2. MyRocks data dictionary may be corrupted.

CS 11.7.0 5bbda9711131845ae6b4315a268b4d1710943a85 (Debug)
Core was generated by `/test/MD090924-mariadb-11.7.0-linux-x86_64-dbg/bin/mariadbd --no-defaults --max'.
Program terminated with signal SIGABRT, Aborted.
#0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>)at ./nptl/pthread_kill.c:44
[Current thread is 1 (LWP 754050)]
(gdb) bt
#0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>)at ./nptl/pthread_kill.c:44
#1 __pthread_kill_internal (signo=6, threadid=<optimized out>)at ./nptl/pthread_kill.c:78
#2 __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6)at ./nptl/pthread_kill.c:89
#3 0x0000150d8f84526e in __GI_raise (sig=sig@entry=6)at ../sysdeps/posix/raise.c:26
#4 0x0000150d8f8288ff in __GI_abort () at ./stdlib/abort.c:79
#5 0x0000150d6474874a in myrocks::rdb_i_s_global_info_fill_table (thd=0x150d30000d58, tables=0x150d3001b290, cond=<optimized out>)at /test/server_dbg/storage/rocksdb/rdb_i_s.cc:802
#6 0x000055f90ea59cba in get_schema_tables_result (join=join@entry=0x150d3001c4f0, executed_place=executed_place@entry=PROCESSED_BY_JOIN_EXEC)at /test/server_dbg/sql/sql_show.cc:9605
#7 0x000055f90ea28cf2 in JOIN::exec_inner (this=this@entry=0x150d3001c4f0)at /test/server_dbg/sql/sql_select.cc:4990
#8 0x000055f90ea29806 in JOIN::exec (this=this@entry=0x150d3001c4f0)at /test/server_dbg/sql/sql_select.cc:4812
#9 0x000055f90ea275eb in mysql_select (thd=thd@entry=0x150d30000d58, tables=0x150d3001b290, fields=@0x150d3001aee0: {<base_list> = {<Sql_alloc> = {<No data fields>}, first = 0x150d3001b210, last = 0x150d3001d0d8, elements = 3}, <No data fields>}, conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=2701396736, result=0x150d3001c4c8, unit=0x150d30005250, select_lex=0x150d3001ac28) at /test/server_dbg/sql/sql_select.cc:5345
#10 0x000055f90ea27e14 in handle_select (thd=thd@entry=0x150d30000d58, lex=lex@entry=0x150d30005170, result=result@entry=0x150d3001c4c8, setup_tables_done_option=setup_tables_done_option@entry=0)at /test/server_dbg/sql/sql_select.cc:628
#11 0x000055f90e9876c0 in execute_sqlcom_select (thd=thd@entry=0x150d30000d58, all_tables=0x150d3001b290) at /test/server_dbg/sql/sql_parse.cc:6154
#12 0x000055f90e992e3a in mysql_execute_command (thd=thd@entry=0x150d30000d58, is_called_from_prepared_stmt=is_called_from_prepared_stmt@entry=false)at /test/server_dbg/sql/sql_parse.cc:3954
#13 0x000055f90e99978c in mysql_parse (thd=thd@entry=0x150d30000d58, rawbuf=<optimized out>, length=<optimized out>, parser_state=parser_state@entry=0x150d8c94c260)at /test/server_dbg/sql/sql_parse.cc:7876
#14 0x000055f90e99bc23 in dispatch_command (command=command@entry=COM_QUERY, thd=thd@entry=0x150d30000d58, packet=packet@entry=0x150d3000b329 "SELECT * FROM information_schema.rocksdb_global_info", packet_length=packet_length@entry=52, blocking=blocking@entry=true) at /test/server_dbg/sql/sql_class.h:256
#15 0x000055f90e99df71 in do_command (thd=0x150d30000d58, blocking=blocking@entry=true) at /test/server_dbg/sql/sql_parse.cc:1405
#16 0x000055f90eb16218 in do_handle_one_connection (connect=<optimized out>, connect@entry=0x55f912d32a48, put_in_cache=put_in_cache@entry=true)at /test/server_dbg/sql/sql_connect.cc:1448
#17 0x000055f90eb16524 in handle_one_connection (arg=arg@entry=0x55f912d32a48)at /test/server_dbg/sql/sql_connect.cc:1350
#18 0x000055f90ef6af79 in pfs_spawn_thread (arg=0x55f912c9c9d8)at /test/server_dbg/storage/perfschema/pfs.cc:2198
#19 0x0000150d8f89ca94 in start_thread (arg=<optimized out>)at ./nptl/pthread_create.c:447
#20 0x0000150d8f929c3c in clone3 ()at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Bug confirmed present in:
MariaDB: 10.5.27 (dbg), 10.5.27 (opt), 10.6.20 (dbg), 10.6.20 (opt), 10.11.10 (dbg), 10.11.10 (opt), 11.1.7 (dbg), 11.1.7 (opt), 11.2.6 (dbg), 11.2.6 (opt), 11.4.4 (dbg), 11.4.4 (opt), 11.6.2 (dbg), 11.6.2 (opt), 11.7.0 (dbg), 11.7.0 (opt)

This testcase is either not sporadic, or has only low sporadicity.

Roel Van de Paar added a comment - 2024-09-15 23:42 - edited Also, INSTALL SONAME 'ha_rocksdb' ; SET GLOBAL rocksdb_update_cf_options= 'cf0={prefix_extractor=capped:0};' ; SELECT * FROM information_schema.rocksdb_global_info; Leads to: CS 11.7.0 5bbda9711131845ae6b4315a268b4d1710943a85 (Debug) 2024-09-16 9:43:05 4 [ERROR] RocksDB: Failed to get column family flags from CF with id = 2. MyRocks data dictionary may be corrupted. CS 11.7.0 5bbda9711131845ae6b4315a268b4d1710943a85 (Debug) Core was generated by `/test/MD090924-mariadb-11.7.0-linux-x86_64-dbg/bin/mariadbd --no-defaults --max'. Program terminated with signal SIGABRT, Aborted. #0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>)at ./nptl/pthread_kill.c:44 [Current thread is 1 (LWP 754050)] (gdb) bt #0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>)at ./nptl/pthread_kill.c:44 #1 __pthread_kill_internal (signo=6, threadid=<optimized out>)at ./nptl/pthread_kill.c:78 #2 __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6)at ./nptl/pthread_kill.c:89 #3 0x0000150d8f84526e in __GI_raise (sig=sig@entry=6)at ../sysdeps/posix/raise.c:26 #4 0x0000150d8f8288ff in __GI_abort () at ./stdlib/abort.c:79 #5 0x0000150d6474874a in myrocks::rdb_i_s_global_info_fill_table (thd=0x150d30000d58, tables=0x150d3001b290, cond=<optimized out>)at /test/server_dbg/storage/rocksdb/rdb_i_s.cc:802 #6 0x000055f90ea59cba in get_schema_tables_result (join=join@entry=0x150d3001c4f0, executed_place=executed_place@entry=PROCESSED_BY_JOIN_EXEC)at /test/server_dbg/sql/sql_show.cc:9605 #7 0x000055f90ea28cf2 in JOIN::exec_inner (this=this@entry=0x150d3001c4f0)at /test/server_dbg/sql/sql_select.cc:4990 #8 0x000055f90ea29806 in JOIN::exec (this=this@entry=0x150d3001c4f0)at /test/server_dbg/sql/sql_select.cc:4812 #9 0x000055f90ea275eb in mysql_select (thd=thd@entry=0x150d30000d58, tables=0x150d3001b290, fields=@0x150d3001aee0: {<base_list> = {<Sql_alloc> = {<No data fields>}, first = 0x150d3001b210, last = 0x150d3001d0d8, elements = 3}, <No data fields>}, conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=2701396736, result=0x150d3001c4c8, unit=0x150d30005250, select_lex=0x150d3001ac28) at /test/server_dbg/sql/sql_select.cc:5345 #10 0x000055f90ea27e14 in handle_select (thd=thd@entry=0x150d30000d58, lex=lex@entry=0x150d30005170, result=result@entry=0x150d3001c4c8, setup_tables_done_option=setup_tables_done_option@entry=0)at /test/server_dbg/sql/sql_select.cc:628 #11 0x000055f90e9876c0 in execute_sqlcom_select (thd=thd@entry=0x150d30000d58, all_tables=0x150d3001b290) at /test/server_dbg/sql/sql_parse.cc:6154 #12 0x000055f90e992e3a in mysql_execute_command (thd=thd@entry=0x150d30000d58, is_called_from_prepared_stmt=is_called_from_prepared_stmt@entry=false)at /test/server_dbg/sql/sql_parse.cc:3954 #13 0x000055f90e99978c in mysql_parse (thd=thd@entry=0x150d30000d58, rawbuf=<optimized out>, length=<optimized out>, parser_state=parser_state@entry=0x150d8c94c260)at /test/server_dbg/sql/sql_parse.cc:7876 #14 0x000055f90e99bc23 in dispatch_command (command=command@entry=COM_QUERY, thd=thd@entry=0x150d30000d58, packet=packet@entry=0x150d3000b329 "SELECT * FROM information_schema.rocksdb_global_info", packet_length=packet_length@entry=52, blocking=blocking@entry=true) at /test/server_dbg/sql/sql_class.h:256 #15 0x000055f90e99df71 in do_command (thd=0x150d30000d58, blocking=blocking@entry=true) at /test/server_dbg/sql/sql_parse.cc:1405 #16 0x000055f90eb16218 in do_handle_one_connection (connect=<optimized out>, connect@entry=0x55f912d32a48, put_in_cache=put_in_cache@entry=true)at /test/server_dbg/sql/sql_connect.cc:1448 #17 0x000055f90eb16524 in handle_one_connection (arg=arg@entry=0x55f912d32a48)at /test/server_dbg/sql/sql_connect.cc:1350 #18 0x000055f90ef6af79 in pfs_spawn_thread (arg=0x55f912c9c9d8)at /test/server_dbg/storage/perfschema/pfs.cc:2198 #19 0x0000150d8f89ca94 in start_thread (arg=<optimized out>)at ./nptl/pthread_create.c:447 #20 0x0000150d8f929c3c in clone3 ()at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 Bug confirmed present in: MariaDB: 10.5.27 (dbg), 10.5.27 (opt), 10.6.20 (dbg), 10.6.20 (opt), 10.11.10 (dbg), 10.11.10 (opt), 11.1.7 (dbg), 11.1.7 (opt), 11.2.6 (dbg), 11.2.6 (opt), 11.4.4 (dbg), 11.4.4 (opt), 11.6.2 (dbg), 11.6.2 (opt), 11.7.0 (dbg), 11.7.0 (opt) This testcase is either not sporadic, or has only low sporadicity.

MariaDB Server

SIGABRT in myrocks::rdb_i_s_global_info_fill_table / [ERROR] RocksDB: Failed to get column family flags from CF with id = 2. MyRocks data dictionary may be corrupted.

Details

Description

Attachments

Activity

People

Dates

Git Integration