[MDEV-26551] InnoDB crash on multiple concurrent SHOW TABLE STATUS Created: 2021-09-06  Updated: 2022-04-13  Resolved: 2022-03-16

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - InnoDB
Affects Version/s: 10.6.5
Fix Version/s: 10.6.8, 10.7.4, 10.8.3

Type: Bug Priority: Major
Reporter: Matthias Leich Assignee: Marko Mäkelä
Resolution: Fixed Votes: 0
Labels: affects-tests, race, regression-10.6

Issue Links:
Duplicate
is duplicated by MDEV-28305 Innodb assertion - DICT_TF_HAS_DATA_D... Closed
Problem/Incident
is caused by MDEV-24258 Merge dict_sys.mutex into dict_sys.latch Closed

 Description    

origin/10.6 4690442411b75ea0fc1a6aacabe767d6fb1d1f62 2021-09-02T17:17:18+03:00
 
 
 
# 2021-09-03T15:40:52 [2887877] | [rr 2891843 1869221]2021-09-03 15:15:44 0x150317ab3700[rr 2891843 1869224]  InnoDB: Assertion failure in file /data/Server/10.6A/storage/innobase/dict/dict0load.cc line 2126
 
# 2021-09-03T15:40:52 [2887877] | [rr 2891843 1869226]InnoDB: Failing assertion: DICT_TF_HAS_DATA_DIR(table->flags)
 
...
 
Query (0x62b000173238): CREATE OR REPLACE TABLE F LIKE `t3`
 
# 2021-09-03T15:40:52 [2887877] | Connection ID (thread ID): 23
 
# 2021-09-03T15:40:52 [2887877] | [rr 2891843 1899510]Status: KILL_TIMEOUT
 
 
 
pluto:/data/Results/1630674257/TBR-1165/dev/shm/vardir/1630674257/117/1/rr
 
(rr) bt
 
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
 
#1  0x00007f7b9ef94859 in __GI_abort () at abort.c:79
 
#2  0x0000558dd7ef98b9 in ut_dbg_assertion_failed (expr=0x558dd912ca80 "DICT_TF_HAS_DATA_DIR(table->flags)", file=0x558dd9129e00 "/data/Server/10.6A/storage/innobase/dict/dict0load.cc", line=2126)
 
    at /data/Server/10.6A/storage/innobase/ut/ut0dbg.cc:60
 
#3  0x0000558dd8077fad in dict_save_data_dir_path (table=0x6180000e1908, filepath=0x60200002a1f0 "./test/t3.ibd") at /data/Server/10.6A/storage/innobase/dict/dict0load.cc:2126
 
#4  0x0000558dd807858c in dict_get_and_save_data_dir_path (table=0x6180000e1908, dict_locked=false) at /data/Server/10.6A/storage/innobase/dict/dict0load.cc:2163
 
#5  0x0000558dd7a25577 in ha_innobase::update_create_info (this=0x61d000983ab8, create_info=0x150317aaf240) at /data/Server/10.6A/storage/innobase/handler/ha_innodb.cc:11378
 
#6  0x0000558dd6bbd4b3 in mysql_prepare_alter_table (thd=0x62b00016c218, table=0x619003314e98, create_info=0x150317aaf240, alter_info=0x150317aaf110, alter_ctx=0x150317aaf510) at /data/Server/10.6A/sql/sql_table.cc:8555
 
#7  0x0000558dd6ba528b in mysql_create_like_table (thd=0x62b00016c218, table=0x62b0001733b8, src_table=0x62b000173b10, create_info=0x150317ab0d20) at /data/Server/10.6A/sql/sql_table.cc:5121
 
#8  0x0000558dd6bd2b67 in Sql_cmd_create_table_like::execute (this=0x62b000173330, thd=0x62b00016c218) at /data/Server/10.6A/sql/sql_table.cc:11753
 
#9  0x0000558dd6959942 in mysql_execute_command (thd=0x62b00016c218, is_called_from_prepared_stmt=false) at /data/Server/10.6A/sql/sql_parse.cc:5997
 
#10 0x0000558dd6965e14 in mysql_parse (thd=0x62b00016c218, rawbuf=0x62b000173238 "CREATE OR REPLACE TABLE F LIKE `t3` /* E_R Thread10 QNO 15 CON_ID 23 */", length=71, parser_state=0x150317ab1b20) at /data/Server/10.6A/sql/sql_parse.cc:8030
 
#11 0x0000558dd693e07f in dispatch_command (command=COM_QUERY, thd=0x62b00016c218, packet=0x6290002d0219 "CREATE OR REPLACE TABLE F LIKE `t3` /* E_R Thread10 QNO 15 CON_ID 23 */ ", packet_length=72, blocking=true)
 
    at /data/Server/10.6A/sql/sql_parse.cc:1896
 
#12 0x0000558dd693b457 in do_command (thd=0x62b00016c218, blocking=true) at /data/Server/10.6A/sql/sql_parse.cc:1404
 
#13 0x0000558dd6d3a81a in do_handle_one_connection (connect=0x608000003638, put_in_cache=true) at /data/Server/10.6A/sql/sql_connect.cc:1418
 
#14 0x0000558dd6d3a0a6 in handle_one_connection (arg=0x608000003638) at /data/Server/10.6A/sql/sql_connect.cc:1312
 
#15 0x0000648e56d41609 in start_thread (arg=<optimized out>) at pthread_create.c:477
 
#16 0x00007f7b9f091293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
(rr) quit
 
 
 
RQG
 
===
 
git clone https://github.com/mleich1/rqg --branch experimental RQG
 
 
 
perl rqg.pl \
 
--grammar=conf/mariadb/innodb_compression_encryption.yy \
 
--gendata=conf/mariadb/innodb_compression_encryption.zz \
 
--max_gd_duration=1800 \
 
--mysqld=--loose-innodb_encryption_rotate_key_age=2 \
 
--mysqld=--loose-innodb_lock_schedule_algorithm=fcfs \
 
--mysqld=--loose-idle_write_transaction_timeout=0 \
 
--mysqld=--loose-idle_transaction_timeout=0 \
 
--mysqld=--loose-idle_readonly_transaction_timeout=0 \
 
--mysqld=--connect_timeout=60 \
 
--mysqld=--interactive_timeout=28800 \
 
--mysqld=--slave_net_timeout=60 \
 
--mysqld=--net_read_timeout=30 \
 
--mysqld=--net_write_timeout=60 \
 
--mysqld=--loose-table_lock_wait_timeout=50 \
 
--mysqld=--wait_timeout=28800 \
 
--mysqld=--lock-wait-timeout=86400 \
 
--mysqld=--innodb-lock-wait-timeout=50 \
 
--no-mask \
 
--queries=10000000 \
 
--seed=random \
 
--reporters=Backtrace \
 
--reporters=ErrorLog \
 
--reporters=Deadlock1 \
 
--validators=None \
 
--mysqld=--log_output=none \
 
--mysqld=--log_bin_trust_function_creators=1 \
 
--mysqld=--loose-debug_assert_on_not_freed_memory=0 \
 
--engine=InnoDB \
 
--restart_timeout=240 \
 
--mysqld=--plugin-load-add=file_key_management.so \
 
--mysqld=--loose-file-key-management-filename=$RQG_HOME/conf/mariadb/encryption_keys.txt \
 
--duration=300 \
 
--mysqld=--loose-innodb_fatal_semaphore_wait_threshold=300 \
 
--mysqld=--loose-innodb-sync-debug \
 
--mysqld=--innodb_stats_persistent=off \
 
--mysqld=--innodb_adaptive_hash_index=on \
 
--mysqld=--loose-innodb_evict_tables_on_commit_debug=off \
 
--mysqld=--loose-max-statement-time=30 \
 
--threads=33 \
 
--mysqld=--innodb-use-native-aio=0 \
 
--rr=Extended \
 
--rr_options=--chaos --wait \
 
--mysqld=--innodb_page_size=4K \
 
--mysqld=--innodb-buffer-pool-size=256M \
 
--no_mask \
 
--workdir=<local settings> \
 
--vardir=<local settings> \
 
--mtr-build-thread=<local settings> \
 
--basedir1=<local settings> \
 
--script_debug=_nix



 Comments   
Comment by Alice Sherepa [ 2021-10-21 ]

also with insert select from information_schema.tables 

2021-10-21 12:06:31 0x7fe398155700  InnoDB: Assertion failure in file /10.6/storage/innobase/dict/dict0load.cc line 2139
 
InnoDB: Failing assertion: DICT_TF_HAS_DATA_DIR(table->flags)
 
InnoDB: We intentionally generate a memory trap.
 
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
 
InnoDB: If you get repeated assertion failures or crashes, even
 
InnoDB: immediately after the mariadbd startup, there may be
 
InnoDB: corruption in the InnoDB tablespace. Please refer to
 
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
 
InnoDB: about forcing recovery.
 
211019 12:06:31 [ERROR] mysqld got signal 6 ;
 
 
 
linux/raise.c:51(__GI_raise)[0x7fe3c07317bb]
 
stdlib/abort.c:81(__GI_abort)[0x7fe3c071c535]
 
ut/ut0dbg.cc:60(_GLOBAL__sub_D_00099_0_ut0dbg.cc)[0x562c7e48712c]
 
dict/dict0load.cc:2141(dict_save_data_dir_path(dict_table_t*, char const*))[0x562c7e6067d5]
 
dict/dict0load.cc:2179(dict_get_and_save_data_dir_path(dict_table_t*, bool))[0x562c7e606e6a]
 
handler/ha_innodb.cc:11376(ha_innobase::update_create_info(HA_CREATE_INFO*))[0x562c7df8058f]
 
sql/sql_show.cc:5629(get_schema_tables_record(THD*, TABLE_LIST*, TABLE*, bool, st_mysql_const_lex_string const*, st_mysql_const_lex_string const*))[0x562c7ce2bb99]
 
sql/sql_show.cc:4705(fill_schema_table_by_open(THD*, st_mem_root*, bool, TABLE*, st_schema_table*, st_mysql_const_lex_string*, st_mysql_const_lex_string*, Open_tables_backup*, bool))[0x562c7ce24101]
 
sql/sql_show.cc:5315(get_all_tables(THD*, TABLE_LIST*, Item*))[0x562c7ce27c17]
 
sql/sql_show.cc:8837(get_schema_tables_result(JOIN*, enum_schema_table_state))[0x562c7ce59e1d]
 
sql/sql_select.cc:4693(JOIN::exec_inner())[0x562c7cd2cabc]
 
sql/sql_select.cc:4516(JOIN::exec())[0x562c7cd2a8b4]
 
sql/sql_select.cc:4996(mysql_select(THD*, TABLE_LIST*, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*))[0x562c7cd2ec89]
 
sql/sql_select.cc:545(handle_select(THD*, LEX*, select_result*, unsigned long))[0x562c7ccff887]
 
sql/sql_parse.cc:4711(mysql_execute_command(THD*, bool))[0x562c7cc5ca76]
 
sql/sql_parse.cc:8030(mysql_parse(THD*, char*, unsigned int, Parser_state*))[0x562c7cc73742]
 
sql/sql_parse.cc:1898(dispatch_command(enum_server_command, THD*, char*, unsigned int, bool))[0x562c7cc49e8c]
 
sql/sql_parse.cc:1404(do_command(THD*, bool))[0x562c7cc46b75]
 
sql/sql_connect.cc:1418(do_handle_one_connection(CONNECT*, bool))[0x562c7d097e64]
 
sql/sql_connect.cc:1314(handle_one_connection)[0x562c7d0976e9]
 
perfschema/pfs.cc:2203(pfs_spawn_thread)[0x562c7dd6ed38]
 
nptl/pthread_create.c:487(start_thread)[0x7fe3c0be8fa3]
 
x86_64/clone.S:97(clone)[0x7fe3c07f34cf]
 
 
 
Query (0x62b0001812a8):  insert t WITH cte1 AS ( SELECT TABLE_SCHEMA,TABLE_NAME,TABLE_TYPE,TABLE_ROWS,TABLE_COLLATION,TABLE_COMMENT FROM information_schema.tables WHERE TABLE_SCHEMA LIKE 'test%' ORDER BY 1,2 ) SELECT * FROM cte1

Comment by Marko Mäkelä [ 2021-10-21 ]

alice, do you have a SQL test case?

Comment by Alice Sherepa [ 2021-10-21 ]

no, sorry, it was with rqg and not reproducible right away

Comment by Elena Stepanova [ 2021-10-27 ]

marko: I doubt a proper MTR test case is possible without the analysis and likely some synchronization, but this reproduces a similar failure for me relatively easily on 10.6 and 10.7 debug builds (for reproducing purposes only, not for the regression suite!):

--source include/have_innodb.inc
 
 
 
CREATE TABLE t1 (pk INT) ENGINE=InnoDB;
 
CREATE TABLE t2 (pk INT) ENGINE=InnoDB;
 
CREATE TABLE t3 (pk INT) ENGINE=InnoDB;
 
CREATE TABLE t4 (pk INT) ENGINE=InnoDB;
 
CREATE TABLE t5 (pk INT) ENGINE=InnoDB;
 
CREATE TABLE t6 (pk INT) ENGINE=InnoDB;
 
CREATE TABLE t7 (pk INT) ENGINE=InnoDB;
 
CREATE TABLE t8 (pk INT) ENGINE=InnoDB;
 
CREATE TABLE t9 (pk INT) ENGINE=InnoDB;
 
 
 
--connect (con1,localhost,root,,test)
 
--let $run=100000
 
--disable_result_log
 
while ($run)
 
{
 
  --send SHOW TABLE STATUS
 
  --connection default
 
    SHOW TABLE STATUS;
 
  --connection con1
 
  --reap
 
  --dec $run
 
}
 
 
 
# Cleanup
 
--disconnect con1
 
--connection default
 
DROP TABLE IF EXISTS t1, t2, t3, t4, t5, t6, t7, t8, t9;

In theory one table is enough, but more tables seem to make it better reproducible.

It is apparently the same failure as alice posted, as it's all the same I_S at the end, but I'm not quite sure it's the exact same failure as mleich initially reported.

10.6 58fe6b47

 
2021-10-27 16:27:10 0x7fa33661c700  InnoDB: Assertion failure in file /data/src/10.6/storage/innobase/dict/dict0load.cc line 2139
 
InnoDB: Failing assertion: DICT_TF_HAS_DATA_DIR(table->flags)
 
 
 
#3  <signal handler called>
 
#4  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
 
#5  0x00007fa34144f859 in __GI_abort () at abort.c:79
 
#6  0x000055c60e5c2f0e in ut_dbg_assertion_failed (expr=0x55c60ed09268 "DICT_TF_HAS_DATA_DIR(table->flags)", file=0x55c60ed07f08 "/data/src/10.6/storage/innobase/dict/dict0load.cc", line=2139) at /data/src/10.6/storage/innobase/ut/ut0dbg.cc:60
 
#7  0x000055c60e692934 in dict_save_data_dir_path (table=0x7fa308239138, filepath=0x7fa308223c48 "./test/t3.ibd") at /data/src/10.6/storage/innobase/dict/dict0load.cc:2139
 
#8  0x000055c60e692c88 in dict_get_and_save_data_dir_path (table=0x7fa308239138, dict_locked=false) at /data/src/10.6/storage/innobase/dict/dict0load.cc:2176
 
#9  0x000055c60e34a61e in ha_innobase::update_create_info (this=0x7fa30c0659b0, create_info=0x7fa3366177d0) at /data/src/10.6/storage/innobase/handler/ha_innodb.cc:11387
 
#10 0x000055c60dc2ad5e in get_schema_tables_record (thd=0x7fa30c000db8, tables=0x7fa30c031b40, table=0x7fa30c02e610, res=false, db_name=0x7fa30c019a40, table_name=0x7fa30c046248) at /data/src/10.6/sql/sql_show.cc:5633
 
#11 0x000055c60dc27bfd in fill_schema_table_by_open (thd=0x7fa30c000db8, mem_root=0x7fa336619a80, is_show_fields_or_keys=false, table=0x7fa30c02e610, schema_table=0x55c60f2c72e0 <schema_tables+2176>, orig_db_name=0x7fa30c019a40, orig_table_name=0x7fa30c046248, open_tables_state_backup=0x7fa336619ac0, can_deadlock=false) at /data/src/10.6/sql/sql_show.cc:4710
 
#12 0x000055c60dc29616 in get_all_tables (thd=0x7fa30c000db8, tables=0x7fa30c015ef0, cond=0x0) at /data/src/10.6/sql/sql_show.cc:5320
 
#13 0x000055c60dc3a987 in get_schema_tables_result (join=0x7fa30c0173e8, executed_place=PROCESSED_BY_JOIN_EXEC) at /data/src/10.6/sql/sql_show.cc:8842
 
#14 0x000055c60dbc69c8 in JOIN::exec_inner (this=0x7fa30c0173e8) at /data/src/10.6/sql/sql_select.cc:4694
 
#15 0x000055c60dbc5d2b in JOIN::exec (this=0x7fa30c0173e8) at /data/src/10.6/sql/sql_select.cc:4515
 
#16 0x000055c60dbc7678 in mysql_select (thd=0x7fa30c000db8, tables=0x7fa30c015ef0, fields=..., conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=2684619520, result=0x7fa30c0173c0, unit=0x7fa30c005120, select_lex=0x7fa30c005918) at /data/src/10.6/sql/sql_select.cc:4994
 
#17 0x000055c60dbb6841 in handle_select (thd=0x7fa30c000db8, lex=0x7fa30c005058, result=0x7fa30c0173c0, setup_tables_done_option=0) at /data/src/10.6/sql/sql_select.cc:545
 
#18 0x000055c60db76e63 in execute_sqlcom_select (thd=0x7fa30c000db8, all_tables=0x7fa30c015ef0) at /data/src/10.6/sql/sql_parse.cc:6256
 
#19 0x000055c60db6e0cb in mysql_execute_command (thd=0x7fa30c000db8, is_called_from_prepared_stmt=false) at /data/src/10.6/sql/sql_parse.cc:3946
 
#20 0x000055c60db7bcb3 in mysql_parse (thd=0x7fa30c000db8, rawbuf=0x7fa30c014160 "SHOW TABLE STATUS", length=17, parser_state=0x7fa33661b480) at /data/src/10.6/sql/sql_parse.cc:8030
 
#21 0x000055c60db680e3 in dispatch_command (command=COM_QUERY, thd=0x7fa30c000db8, packet=0x7fa30c00b879 "SHOW TABLE STATUS", packet_length=17, blocking=true) at /data/src/10.6/sql/sql_parse.cc:1896
 
#22 0x000055c60db66a7f in do_command (thd=0x7fa30c000db8, blocking=true) at /data/src/10.6/sql/sql_parse.cc:1404
 
#23 0x000055c60dd25123 in do_handle_one_connection (connect=0x55c610983ff8, put_in_cache=true) at /data/src/10.6/sql/sql_connect.cc:1418
 
#24 0x000055c60dd24db3 in handle_one_connection (arg=0x55c610983ff8) at /data/src/10.6/sql/sql_connect.cc:1312
 
#25 0x000055c60e25604d in pfs_spawn_thread (arg=0x55c6109840d8) at /data/src/10.6/storage/perfschema/pfs.cc:2201
 
#26 0x00007fa341979609 in start_thread (arg=<optimized out>) at pthread_create.c:477
 
#27 0x00007fa34154c293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Not rr-able, at least on my machine.

Comment by Alice Sherepa [ 2021-11-10 ]

I am getting it on 10.7, with small variations

InnoDB: Assertion failure in file /10.7/storage/innobase/dict/dict0load.cc line 2081
 
InnoDB: Failing assertion: DICT_TF_HAS_DATA_DIR(table->flags)
 
 
 
dict/dict0load.cc:2083(dict_save_data_dir_path(dict_table_t*, char const*))[0x55ea3c90f3b1]
 
dict/dict0load.cc:2121(dict_get_and_save_data_dir_path(dict_table_t*, bool))[0x55ea3c90fad9]
 
handler/ha_innodb.cc:11379(ha_innobase::update_create_info(HA_CREATE_INFO*))[0x55ea3c271d93]
 
sql/ha_partition.cc:2354(ha_partition::update_create_info(HA_CREATE_INFO*))[0x55ea3c005bf8]
 
sql/sql_show.cc:1874(add_table_options(THD*, TABLE*, Table_specification_st*, bool, bool, String*))[0x55ea3b175463]
 
sql/sql_show.cc:2458(show_create_table_ex(THD*, TABLE_LIST*, char const*, char const*, String*, Table_specification_st*, enum_with_db_name))[0x55ea3b179d4c]
 
sql/sql_show.cc:2029(show_create_table(THD*, TABLE_LIST*, String*, Table_specification_st*, enum_with_db_name))[0x55ea3b17695c]
 
sql/sql_show.cc:1249(mysqld_show_create_get_fields(THD*, TABLE_LIST*, List<Item>*, String*))[0x55ea3b171121]
 
sql/sql_show.cc:1323(mysqld_show_create(THD*, TABLE_LIST*))[0x55ea3b171987]
 
sql/sql_parse.cc:4368(mysql_execute_command(THD*, bool))[0x55ea3af8593a]
 
sql/sql_parse.cc:8028(mysql_parse(THD*, char*, unsigned int, Parser_state*))[0x55ea3af9f831]
 
sql/sql_parse.cc:1896(dispatch_command(enum_server_command, THD*, char*, unsigned int, bool))[0x55ea3af75fbd]
 
sql/sql_parse.cc:1402(do_command(THD*, bool))[0x55ea3af72cb2]
 
sql/sql_connect.cc:1418(do_handle_one_connection(CONNECT*, bool))[0x55ea3b3fcf78]
 
sql/sql_connect.cc:1314(handle_one_connection)[0x55ea3b3fc7fd]
 
perfschema/pfs.cc:2203(pfs_spawn_thread)[0x55ea3c05f900]
 
nptl/pthread_create.c:487(start_thread)[0x7f6634eddfa3]
 
x86_64/clone.S:97(clone)[0x7f6634ae84cf]
 
 
 
Query (0x6290007c62a8): SHOW CREATE TABLE `t`

also on slave:

InnoDB: Assertion failure in file /10.7/storage/innobase/dict/dict0load.cc line 2081
 
InnoDB: Failing assertion: DICT_TF_HAS_DATA_DIR(table->flags)
 
 
 
dict/dict0load.cc:2083(dict_save_data_dir_path(dict_table_t*, char const*))[0x56271d7973b1]
 
dict/dict0load.cc:2121(dict_get_and_save_data_dir_path(dict_table_t*, bool))[0x56271d797ad9]
 
handler/ha_innodb.cc:11379(ha_innobase::update_create_info(HA_CREATE_INFO*))[0x56271d0f9d93]
 
sql/sql_show.cc:5637(get_schema_tables_record(THD*, TABLE_LIST*, TABLE*, bool, st_mysql_const_lex_string const*, st_mysql_const_lex_string const*))[0x56271c019fcb]
 
sql/sql_show.cc:4713(fill_schema_table_by_open(THD*, st_mem_root*, bool, TABLE*, st_schema_table*, st_mysql_const_lex_string*, st_mysql_const_lex_string*, Open_tables_backup*, bool))[0x56271c01251c]
 
sql/sql_show.cc:5323(get_all_tables(THD*, TABLE_LIST*, Item*))[0x56271c016038]
 
sql/sql_show.cc:8838(get_schema_tables_result(JOIN*, enum_schema_table_state))[0x56271c0480af]
 
sql/sql_select.cc:4691(JOIN::exec_inner())[0x56271bf1d804]
 
sql/sql_select.cc:4514(JOIN::exec())[0x56271bf1b5fc]
 
sql/sql_select.cc:4995(mysql_select(THD*, TABLE_LIST*, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*))[0x56271bf1f9ed]
 
sql/sql_select.cc:545(handle_select(THD*, LEX*, select_result*, unsigned long))[0x56271bef055b]
 
sql/sql_parse.cc:4709(mysql_execute_command(THD*, bool))[0x56271be10bab]
 
sql/sql_parse.cc:8028(mysql_parse(THD*, char*, unsigned int, Parser_state*))[0x56271be27831]
 
sql/log_event_server.cc:1918(Query_log_event::do_apply_event(rpl_group_info*, char const*, unsigned int))[0x56271ca404d9]
 
sql/log_event_server.cc:1592(Query_log_event::do_apply_event(rpl_group_info*))[0x56271ca3d6f7]
 
sql/log_event.h:1516(Log_event::apply_event(rpl_group_info*))[0x56271bb9a9e9]
 
sql/slave.cc:3881(apply_event_and_update_pos_apply(Log_event*, THD*, rpl_group_info*, int))[0x56271bb7cccc]
 
sql/slave.cc:4068(apply_event_and_update_pos_for_parallel(Log_event*, THD*, rpl_group_info*))[0x56271bb7da35]
 
sql/rpl_parallel.cc:62(rpt_handle_event(rpl_parallel_thread::queued_event*, rpl_parallel_thread*))[0x56271c3f1c25]
 
sql/rpl_parallel.cc:1376(handle_rpl_parallel_thread)[0x56271c3f96c6]
 
perfschema/pfs.cc:2203(pfs_spawn_thread)[0x56271cee7900]
 
nptl/pthread_create.c:487(start_thread)[0x7f4b267a8fa3]
 
x86_64/clone.S:97(clone)[0x7f4b263b34cf]

Comment by Alice Sherepa [ 2021-12-22 ]

preview-10.8-MDEV-11675-rpl-lag-free-alter 8fc5d21de2530bf7ca9edd0

 
2021-12-22 10:30:43 0x7f23a11a4700  InnoDB: Assertion failure in file /10.8/storage/innobase/dict/dict0load.cc line 2089
 
InnoDB: Failing assertion: DICT_TF_HAS_DATA_DIR(table->flags)
 
InnoDB: We intentionally generate a memory trap.
 
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
 
InnoDB: If you get repeated assertion failures or crashes, even
 
InnoDB: immediately after the mariadbd startup, there may be
 
InnoDB: corruption in the InnoDB tablespace. Please refer to
 
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
 
InnoDB: about forcing recovery.
 
211222 10:30:43 [ERROR] mysqld got signal 6 ;
 
 
 
Server version: 10.8.0-MariaDB-debug-log
 
 
 
??:0(gsignal)[0x7f23c757218b]
 
??:0(abort)[0x7f23c7551859]
 
ut/ut0dbg.cc:60(_sub_D_00099_0)[0x559ec254b6ff]
 
dict/dict0load.cc:2091(dict_save_data_dir_path(dict_table_t*, char const*))[0x559ec26d1fe8]
 
dict/dict0load.cc:2129(dict_get_and_save_data_dir_path(dict_table_t*, bool))[0x559ec26d2714]
 
handler/ha_innodb.cc:11373(ha_innobase::update_create_info(HA_CREATE_INFO*))[0x559ec202743d]
 
sql/sql_table.cc:8810(mysql_prepare_alter_table(THD*, TABLE*, HA_CREATE_INFO*, Alter_info*, Alter_table_ctx*))[0x559ec0fab679]
 
sql/sql_table.cc:5261(mysql_create_like_table(THD*, TABLE_LIST*, TABLE_LIST*, Table_specification_st*))[0x559ec0f91180]
 
sql/sql_table.cc:12173(Sql_cmd_create_table_like::execute(THD*))[0x559ec0fc469c]
 
sql/sql_parse.cc:5989(mysql_execute_command(THD*, bool))[0x559ec0cd5161]
 
sql/sql_parse.cc:8028(mysql_parse(THD*, char*, unsigned int, Parser_state*))[0x559ec0ce2987]
 
sql/sql_parse.cc:1896(dispatch_command(enum_server_command, THD*, char*, unsigned int, bool))[0x559ec0cb8a84]
 
sql/sql_parse.cc:1402(do_command(THD*, bool))[0x559ec0cb57a8]
 
sql/sql_connect.cc:1418(do_handle_one_connection(CONNECT*, bool))[0x559ec115957d]
 
sql/sql_connect.cc:1314(handle_one_connection)[0x559ec1158e09]
 
perfschema/pfs.cc:2203(pfs_spawn_thread)[0x559ec1e0910d]
 
nptl/pthread_create.c:478(start_thread)[0x7f23c7a7b609]
 
??:0(clone)[0x7f23c764e293]
 
 
 
Query (0x629001e0a2a8): CREATE TABLE IF NOT EXISTS t LIKE `BB`

Comment by Marko Mäkelä [ 2022-03-16 ]

The problem is a race condition between two SHOW TABLE STATUS. Here is a little better test case.

--source include/have_innodb.inc
 
CREATE TABLE t1 (pk INT) ENGINE=InnoDB;
 
delimiter |;
 
create procedure showtime()
 
begin
 
   declare n int default 1000000;
 
   while (n > 0) do
 
   SHOW TABLE STATUS;
 
   set n = n - 1;
 
   end while;
 
end|
 
delimiter ;|
 
--connect (con1,localhost,root,,test)
 
send call showtime();
 
--connect (con2,localhost,root,,test)
 
send call showtime();
 
--connect (con3,localhost,root,,test)
 
send call showtime();
 
--connect (con4,localhost,root,,test)
 
send call showtime();
 
--connect (con5,localhost,root,,test)
 
send call showtime();
 
 
 
--connection default
 
call showtime();
 
 
 
DROP TABLE t1;

The race condition is obvious in the code:

void dict_get_and_save_data_dir_path(dict_table_t* table, bool dict_locked)
 
{
 
	ut_ad(!table->is_temporary());
 
	ut_ad(!table->space || table->space->id == table->space_id);
 
 
 
	if (!table->data_dir_path && table->space_id && table->space) {
 
		if (!dict_locked) {
 
			dict_sys.freeze(SRW_LOCK_CALL);
 
		}
 
 
 
		table->flags |= 1 << DICT_TF_POS_DATA_DIR
 
			& ((1U << DICT_TF_BITS) - 1);
 
		dict_save_data_dir_path(table,
 
					table->space->chain.start->name);
 
 
 
		if (table->data_dir_path == NULL) {
 
			/* Since we did not set the table data_dir_path,
 
			unset the flag.  This does not change
 
			SYS_TABLES or FSP_SPACE_FLAGS on the header page
 
			of the tablespace, but it makes dict_table_t
 
			consistent. */
 
			table->flags &= ~DICT_TF_MASK_DATA_DIR
 
				& ((1U << DICT_TF_BITS) - 1);
 
		}
 
 
 
		if (!dict_locked) {
 
			dict_sys.unfreeze();
 
		}
 
	}
 
}

Before MDEV-24258, this code was protected by the exclusive dict_sys.mutex, and there was no problem.

That said, this code deserves to be simplified (or eliminated). Thanks to the fact that the SYS_DATAFILES table was removed in MDEV-22343 and MDEV-12266 made dict_table_t::space a pointer, we should not need table->data_dir_path at all, and may simply refer to table->space->chain.start->name directly to determine whether a DATA DIRECTORY attribute is in effect.

Comment by Marko Mäkelä [ 2022-03-16 ]

It turns out that table->data_dir_path is storing something derived from the DATA DIRECTORY attribute on CREATE TABLE, before the tablespace (fil_space_t) has been created. We can’t remove it easily, but we can protect the metadata with dict_table_t::lock_mutex.

Generated at Thu Feb 08 09:46:10 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.