[MDEV-30286] InnoDB: Failing assertion: sym_node->table != NULL Created: 2022-12-21  Updated: 2024-02-07  Resolved: 2024-02-07

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - InnoDB
Affects Version/s: 10.5.16
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: guozhentang Assignee: Marko Mäkelä
Resolution: Incomplete Votes: 0
Labels: None
Environment:

centos 7.6


Issue Links:
Relates
relates to MDEV-25506 Atomic DDL: .frm file is removed and ... Closed

 Description   

Crash on missing innodb stats table

error.log

2022-09-16 15:49:13 14 [Note] InnoDB: Resuming purge
2022-09-16 15:49:18 39 [ERROR] InnoDB: Table `mysql`.`innodb_index_stats` not found.
2022-09-16 15:49:18 46 [ERROR] InnoDB: Fetch of persistent statistics requested for table `test`.`ti` but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead.
2022-09-16 15:49:18 19 [ERROR] InnoDB: Fetch of persistent statistics requested for table `test`.`ti` but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead.
2022-09-16 15:49:18 27 [ERROR] InnoDB: Fetch of persistent statistics requested for table `mysql`.`#sql-alter-82ea-1b` but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead.
2022-09-16 15:49:18 0x7f687c184700  InnoDB: Assertion failure in file /data/fuxi_ci_workspace/6323ea4dfb6d3909ff2a059d/storage/innobase/pars/pars0pars.cc line 771
InnoDB: Failing assertion: sym_node->table != NULL
2022-09-16 15:49:18 39 [ERROR] InnoDB: Fetch of persistent statistics requested for table `test`.`ti` but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead.
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
InnoDB: about forcing recovery.
220916 15:49:18 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
 
 
Server version: 10.5.16-MariaDB
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=50
max_threads=1001
thread_count=51
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2333216 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x7f67c80009b8
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f687c183df8 thread_stack 0x49000
mysys/stacktrace.c:213(my_print_stacktrace)[0x5651463ddb1e]
sql/signal_handler.cc:225(handle_fatal_signal)[0x565145e38c05]
sigaction.c:0(__restore_rt)[0x7f68a249d6d0]
:0(__GI_raise)[0x7f68a20f7277]
:0(__GI_abort)[0x7f68a20f8968]
ut/ut0rbt.cc:458(rbt_eject_node(ib_rbt_node_t*, ib_rbt_node_t*) [clone .part.0])[0x565145b0974a]
include/ut0ut.h:335(pars_retrieve_table_def)[0x565145aee91a]
include/que0que.inl:209(que_node_get_next)[0x5651461d179c]
innobase/pars0grm.cc:2411(yyparse())[0x56514635e89f]
pars/pars0pars.cc:2003(pars_sql(pars_info_t*, char const*))[0x5651461d4e94]
include/ib0mutex.h:745(PolicyMutex<TTASEventMutex<GenericPolicy> >::pfs_exit())[0x5651461d89fe]
dict/dict0stats.cc:3096(dict_stats_fetch_from_ps)[0x5651462f5788]
sql/sql_lifo_buffer.h:202(ha_innobase::info_low(unsigned int, bool))[0x565146144a88]
sql/sql_lifo_buffer.h:202(ha_innobase::open(char const*, int, unsigned int))[0x56514614a568]
??:0(handler::ha_open(TABLE*, char const*, int, unsigned int, st_mem_root*, List<String>*))[0x565145e3d276]
sql/table.h:546(open_table_from_share(THD*, TABLE_SHARE*, st_mysql_const_lex_string const*, unsigned int, unsigned int, unsigned int, TABLE*, bool, List<String>*))[0x565145d013e2]
??:0(open_table(THD*, TABLE_LIST*, Open_table_context*))[0x565145bc4599]
??:0(open_tables(THD*, DDL_options_st const&, TABLE_LIST**, unsigned int*, unsigned int, Prelocking_strategy*))[0x565145bc7886]
??:0(open_and_lock_tables(THD*, DDL_options_st const&, TABLE_LIST*, bool, unsigned int, Prelocking_strategy*))[0x565145bc7fe5]
??:0(mysql_insert(THD*, TABLE_LIST*, List<Item>&, List<List<Item> >&, List<Item>&, List<Item>&, enum_duplicates, bool, select_result*))[0x565145bfb1e4]
??:0(mysql_execute_command(THD*))[0x565145c2e48b]
??:0(mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool))[0x565145c318ba]
sql_parse.cc:0(dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) [clone .localalias])[0x565145c331fb]
??:0(do_command(THD*))[0x565145c34d43]
sql/sql_connect.cc:1419(do_handle_one_connection(CONNECT*, bool))[0x565145d30771]
sql/sql_connect.cc:1313(handle_one_connection)[0x565145d30bfd]
perfschema/pfs.cc:2204(pfs_spawn_thread)[0x5651460b889b]
 
pthread_create.c:0(start_thread)[0x7f68a2495e25]
??:0(__clone)[0x7f68a21bfbad]
 
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7f67c800fed0): INSERT INTO ti VALUES (23806,-7662318326977267671,'Rbu9jMJN629R8LaDE9UVHlZg','idyvFcTtorQ8WjHNEKAme8elnspCAmp27uKuEgDuKOa8PicyRdEEIVBXQxBJeoEn1opCVfcj2ITfxb0OyJEK4VX04LFxQGKOYB0YERN59vI7q73DshCRpGdmUYudp7a05zMxv3yhpbhFUYfIjEmSDlge2V4wYu99klqW4auF7H7seDd8puB6x0','RkbD5ni54ovQuQrtSorHH2NfscBNO6kdoMy2XD','R9OcRYgS4Oe5jxFQqCABx7174pn9smGP00fFI9fce9Yj0Kpc486kVtzl52OL6qABMNKVqV9tFeP8t2ca38wshFpxoBV0rNl0fIUw8sXeBbCp7Xzp6X1sk6wnH49ceWlnHxa5eJ10uBMvAj19pcEFMfTu0hu0KZNG6S','l','cf',6)
 
Connection ID (thread ID): 28
Status: NOT_KILLED
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off

core:
(gdb) bt
#0  0x00007f68a249aa01 in pthread_kill () from /lib64/libpthread.so.0
#1  0x00005651463dd919 in my_write_core (sig=<optimized out>) at /data/fuxi_ci_workspace/6323ea4dfb6d3909ff2a059d/mysys/stacktrace.c:424
#2  0x0000565145e38c20 in handle_fatal_signal (sig=6) at /data/fuxi_ci_workspace/6323ea4dfb6d3909ff2a059d/sql/signal_handler.cc:344
#3  <signal handler called>
#4  0x00007f68a20f7277 in raise () from /lib64/libc.so.6
#5  0x00007f68a20f8968 in abort () from /lib64/libc.so.6
#6  0x0000565145b0974a in ut_dbg_assertion_failed (expr=expr@entry=0x56514663bba4 "sym_node->table != NULL", 
    file=file@entry=0x56514663b530 "/data/fuxi_ci_workspace/6323ea4dfb6d3909ff2a059d/storage/innobase/pars/pars0pars.cc", line=line@entry=771)
    at /data/fuxi_ci_workspace/6323ea4dfb6d3909ff2a059d/storage/innobase/ut/ut0dbg.cc:60
#7  0x0000565145aee91a in pars_retrieve_table_def (sym_node=0x7f67c8061ee8) at /data/fuxi_ci_workspace/6323ea4dfb6d3909ff2a059d/storage/innobase/pars/pars0pars.cc:771
#8  pars_retrieve_table_def (sym_node=0x7f67c8061ee8) at /data/fuxi_ci_workspace/6323ea4dfb6d3909ff2a059d/storage/innobase/pars/pars0pars.cc:753
#9  0x00005651461d179c in pars_retrieve_table_list_defs (sym_node=0x7f67c8061ee8) at /data/fuxi_ci_workspace/6323ea4dfb6d3909ff2a059d/storage/innobase/pars/pars0pars.cc:792
#10 pars_select_statement(sel_node_t*, sym_node_t*, void*, pars_res_word_t*, pars_res_word_t*, order_node_t*) ()
    at /data/fuxi_ci_workspace/6323ea4dfb6d3909ff2a059d/storage/innobase/pars/pars0pars.cc:923
#11 0x000056514635e89f in yyparse () at pars0grm.y:308
#12 0x00005651461d4e94 in pars_sql(pars_info_t*, char const*) () at /data/fuxi_ci_workspace/6323ea4dfb6d3909ff2a059d/storage/innobase/pars/pars0pars.cc:2001
#13 0x00005651461d89fe in que_eval_sql(pars_info_t*, char const*, bool, trx_t*) () at /data/fuxi_ci_workspace/6323ea4dfb6d3909ff2a059d/storage/innobase/que/que0que.cc:1112
#14 0x00005651462f5788 in dict_stats_fetch_from_ps (table=0x7f67c805e220) at /data/fuxi_ci_workspace/6323ea4dfb6d3909ff2a059d/storage/innobase/dict/dict0stats.cc:3039
#15 dict_stats_update(dict_table_t*, dict_stats_upd_option_t) () at /data/fuxi_ci_workspace/6323ea4dfb6d3909ff2a059d/storage/innobase/dict/dict0stats.cc:3318
#16 0x0000565146144a88 in dict_stats_init (table=0x7f678c035050) at /data/fuxi_ci_workspace/6323ea4dfb6d3909ff2a059d/storage/innobase/include/dict0stats.inl:165
#17 dict_stats_init (table=0x7f678c035050) at /data/fuxi_ci_workspace/6323ea4dfb6d3909ff2a059d/storage/innobase/include/dict0stats.inl:147
#18 ha_innobase::info_low(unsigned int, bool) () at /data/fuxi_ci_workspace/6323ea4dfb6d3909ff2a059d/storage/innobase/handler/ha_innodb.cc:14360
#19 0x000056514614a568 in info (flag=282, this=0x7f67c805bc80) at /data/fuxi_ci_workspace/6323ea4dfb6d3909ff2a059d/storage/innobase/handler/ha_innodb.cc:14623
#20 ha_innobase::open(char const*, int, unsigned int) () at /data/fuxi_ci_workspace/6323ea4dfb6d3909ff2a059d/storage/innobase/handler/ha_innodb.cc:6015
#21 0x0000565145e3d276 in handler::ha_open (this=0x7f67c805bc80, table_arg=table_arg@entry=0x7f67c8035528, name=0x7f67a804d1c8 "./test/ti", mode=mode@entry=2, 
    test_if_locked=test_if_locked@entry=18, mem_root=mem_root@entry=0x0, partitions_to_open=0x0) at /data/fuxi_ci_workspace/6323ea4dfb6d3909ff2a059d/sql/handler.cc:3013
#22 0x0000565145d013e2 in open_table_from_share (thd=thd@entry=0x7f67c80009b8, share=share@entry=0x7f67a804cd30, alias=alias@entry=0x7f67c8010378, db_stat=db_stat@entry=33, 
    prgflag=prgflag@entry=8, ha_open_flags=18, outparam=<optimized out>, is_create_table=<optimized out>, partitions_to_open=<optimized out>)
    at /data/fuxi_ci_workspace/6323ea4dfb6d3909ff2a059d/sql/table.cc:4322
#23 0x0000565145bc4599 in open_table(THD*, TABLE_LIST*, Open_table_context*) () at /data/fuxi_ci_workspace/6323ea4dfb6d3909ff2a059d/sql/sql_base.cc:2014
#24 0x0000565145bc7886 in open_and_process_table (ot_ctx=0x7f687c182b20, has_prelocking_list=false, prelocking_strategy=0x7f687c182cd0, flags=0, counter=0x7f687c182bbc, 
    tables=0x7f67c8010330, thd=0x7f67c80009b8) at /data/fuxi_ci_workspace/6323ea4dfb6d3909ff2a059d/sql/sql_base.cc:3805
#25 open_tables (thd=thd@entry=0x7f67c80009b8, options=..., start=start@entry=0x7f687c182ba8, counter=counter@entry=0x7f687c182bbc, flags=flags@entry=0, prelocking_strategy=0x7f687c182cd0)
    at /data/fuxi_ci_workspace/6323ea4dfb6d3909ff2a059d/sql/sql_base.cc:4288
#26 0x0000565145bc7fe5 in open_and_lock_tables (thd=thd@entry=0x7f67c80009b8, options=..., tables=<optimized out>, tables@entry=0x7f67c8010330, derived=derived@entry=true, 
    flags=flags@entry=0, prelocking_strategy=<optimized out>) at /data/fuxi_ci_workspace/6323ea4dfb6d3909ff2a059d/sql/sql_base.cc:5235
#27 0x0000565145bfb1e4 in open_and_lock_tables (flags=<optimized out>, derived=<optimized out>, tables=<optimized out>, thd=<optimized out>)
    at /data/fuxi_ci_workspace/6323ea4dfb6d3909ff2a059d/sql/sql_base.h:507
#28 open_and_lock_for_insert_delayed (table_list=<optimized out>, thd=<optimized out>) at /data/fuxi_ci_workspace/6323ea4dfb6d3909ff2a059d/sql/sql_insert.cc:626
#29 mysql_insert(THD*, TABLE_LIST*, List<Item>&, List<List<Item> >&, List<Item>&, List<Item>&, enum_duplicates, bool, select_result*) ()
    at /data/fuxi_ci_workspace/6323ea4dfb6d3909ff2a059d/sql/sql_insert.cc:752
#30 0x0000565145c2e48b in mysql_execute_command(THD*) () at /data/fuxi_ci_workspace/6323ea4dfb6d3909ff2a059d/sql/sql_parse.cc:4625
#31 0x0000565145c318ba in mysql_parse (thd=thd@entry=0x7f67c80009b8, rawbuf=<optimized out>, length=<optimized out>, parser_state=parser_state@entry=0x7f687c183520, 
---Type <return> to continue, or q <return> to quit---
    is_com_multi=is_com_multi@entry=false, is_next_command=is_next_command@entry=false) at /data/fuxi_ci_workspace/6323ea4dfb6d3909ff2a059d/sql/sql_parse.cc:8173
#32 0x0000565145c331fb in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) [clone .localalias] ()
    at /data/fuxi_ci_workspace/6323ea4dfb6d3909ff2a059d/sql/sql_class.h:1290
#33 0x0000565145c34d43 in do_command (thd=0x7f67c80009b8) at /data/fuxi_ci_workspace/6323ea4dfb6d3909ff2a059d/sql/sql_parse.cc:1376
#34 0x0000565145d30771 in do_handle_one_connection (connect=<optimized out>, connect@entry=0x565147b6dfd8, put_in_cache=put_in_cache@entry=true)
    at /data/fuxi_ci_workspace/6323ea4dfb6d3909ff2a059d/sql/sql_connect.cc:1419
#35 0x0000565145d30bfd in handle_one_connection (arg=arg@entry=0x565147b6dfd8) at /data/fuxi_ci_workspace/6323ea4dfb6d3909ff2a059d/sql/sql_connect.cc:1313
#36 0x00005651460b889b in pfs_spawn_thread (arg=0x565147b6e048) at /data/fuxi_ci_workspace/6323ea4dfb6d3909ff2a059d/storage/perfschema/pfs.cc:2201
#37 0x00007f68a2495e25 in start_thread () from /lib64/libpthread.so.0
#38 0x00007f68a21bfbad in clone () from /lib64/libc.so.6
(gdb) f 7
#7  0x0000565145aee91a in pars_retrieve_table_def (sym_node=0x7f67c8061ee8) at /data/fuxi_ci_workspace/6323ea4dfb6d3909ff2a059d/storage/innobase/pars/pars0pars.cc:771
771    /data/fuxi_ci_workspace/6323ea4dfb6d3909ff2a059d/storage/innobase/pars/pars0pars.cc: No such file or directory.
(gdb) p sym_node->table
$1 = (dict_table_t *) 0x0



 Comments   
Comment by Marko Mäkelä [ 2023-10-02 ]

Hi! Can you give the exact steps for reproducing this bug?

Also, can you try what would happen if you started up MariaDB Server 10.6 on a copy of this data directory? I would hope that MDEV-25506 and related changes fixed this crash.

Comment by Marko Mäkelä [ 2024-01-05 ]

As far as I can understand, the assertion fails because one of the tables mysql.innodb_index_stats or mysql.innodb_table_stats does not exist or is corrupted or unaccessible. I believe that in MariaDB Server 10.6, we should be able to open the InnoDB table even if the persistent statistics are unaccessible.

guozhentang, did you try to start up MariaDB Server 10.6 on a copy of the data directory?

Generated at Thu Feb 08 10:15:07 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.