[MDEV-16931] Slave crashes on DROP DATABASE command when there is an orphaned table in the schema Created: 2018-08-09  Updated: 2018-10-05  Resolved: 2018-10-05

Status: Closed
Project: MariaDB Server
Component/s: Server
Affects Version/s: 10.1.31
Fix Version/s: N/A

Type: Bug Priority: Critical
Reporter: Chris Calender (Inactive) Assignee: Unassigned
Resolution: Incomplete Votes: 0
Labels: need_feedback


 Description   

We see a slave crash on a DROP DATABASE command (from the master) when there is an orphaned table in the schema (on the slave).

Here is the stack trace:

180808 5:39:57 [ERROR] mysqld got signal 11 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed,
 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.1.31-MariaDB
 
key_buffer_size=67108864
 
read_buffer_size=131072
 
max_used_connections=0
 
max_threads=1002
 
thread_count=0
 
It is possible that mysqld could use up to
 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1240673 K bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x7ed464412008
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x7f0ab1cc1070 thread_stack 0x48400
 
mysys/stacktrace.c:268(my_print_stacktrace)[0x55f7f54531ce]
 
sql/signal_handler.cc:168(handle_fatal_signal)[0x55f7f4f76fb5]
 
sigaction.c:0(__restore_rt)[0x7f0ab7c9b680]
 
include/dict0dict.ic:166(dict_col_get_no)[0x55f7f528916b]
 
row/row0upd.cc:1795(row_upd_store_row(upd_node_t*))[0x55f7f529692c]
 
row/row0upd.cc:2543(row_upd_del_mark_clust_rec)[0x55f7f5297667]
 
row/row0upd.cc:2845(row_upd)[0x55f7f5298497]
 
que/que0que.cc:1071(que_thr_step)[0x55f7f524f000]
 
que/que0que.cc:1277(que_eval_sql(pars_info_t*, char const*, unsigned long, trx_t*))[0x55f7f524f7d2]
 
row/row0mysql.cc:4340(row_drop_table_for_mysql(char const*, trx_t*, bool, unsigned long, bool))[0x55f7f5279461]
 
handler/ha_innodb.cc:13182(ha_innobase::delete_table(char const*))[0x55f7f51b5b11]
 
sql/handler.cc:2382(ha_delete_table(THD*, handlerton*, char const*, char const*, char const*, bool))[0x55f7f4f7efe5]
 
sql/sql_table.cc:2472(mysql_rm_table_no_locks(THD*, TABLE_LIST*, bool, bool, bool, bool, bool))[0x55f7f4e734bd]
 
sql/sql_db.cc:918(mysql_rm_db_internal(THD*, char*, bool, bool))[0x55f7f4dc66f3]
 
sql/sql_parse.cc:4461(mysql_execute_command(THD*))[0x55f7f4df0177]
 
sql/sql_parse.cc:7353(mysql_parse(THD*, char*, unsigned int, Parser_state*))[0x55f7f4df3852]
 
sql/log_event.cc:4460(Query_log_event::do_apply_event(rpl_group_info*, char const*, unsigned int))[0x55f7f50504b1]
 
sql/log_event.h:1344(Log_event::apply_event(rpl_group_info*))[0x55f7f4d6b1db]
 
sql/mysqld.h:746(thread_safe_increment64)[0x55f7f4f08e9e]
 
sql/rpl_parallel.cc:1260(handle_rpl_parallel_thread)[0x55f7f4f0d20e]
 
pthread_create.c:0(start_thread)[0x7f0ab7c93dd5]
 
/lib64/libc.so.6(clone+0x6d)[0x7f0ab5e04b3d]
 
 
 
Trying to get some variables.
 
Some pointers may be invalid and cause the dump to abort.
 
Query (0x7ed4624462b6): is an invalid pointer
 
Connection ID (thread ID): 11
 
Status: NOT_KILLED
 
 
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=on,mrr_cost_based=on,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=off
 
 
 
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
 
information that should help you find out what is causing the crash.
 
 
 
We think the query pointer is invalid, but we will try to print it anyway.
 
Query: drop database db1

Unfortunately, I am currently unable to reproduce, and the instance was recovered from a backup, so I cannot see what happened at the file level.

I presume the table was orphaned because it showed in SHOW TABLES output, but SELECTs, INSERTs, UPDATEs, and even DROP TABLE all failed.

The log shows it as using XtraDB. However, line 166 in dict0dict.ic are the same in the source code for both InooDB and XtraDB.

 Comments   
Comment by Elena Stepanova [ 2018-08-12 ]

How exactly (with which errors) DROP TABLE and DML were failing?

Comment by Chris Calender (Inactive) [ 2018-08-13 ]

elenst Unfortunately, they could not provide me with the exact errors they saw.

However, I mocked up this example, and they said those looked like the errors they saw:

1. I created a table, t1 ("create table t1 (id int);").

2. Then I stopped the mariadb instance, so I could remove the t1.ibd file.

3. I then restarted mariadb.

4. Then I performed the actions they performed and saw similar results:

mysql> show tables;
 
+------------------+
 
| Tables_in_d20985 |
 
+------------------+
 
| t1 |
 
+------------------+
 
1 row in set (0.00 sec)
 
 
 
mysql> select * from t1;
 
ERROR 1932 (42S02): Table 'd20985.t1' doesn't exist in engine
 
mysql>
 
mysql> insert into t1 values (4);
 
ERROR 1932 (42S02): Table 'd20985.t1' doesn't exist in engine
 
mysql>
 
mysql> create table t1 (id int);
 
ERROR 1050 (42S01): Table 't1' already exists"

However, 2 major differences in my test compared to what they saw:

1. I was able to issue a DROP TABLE (on the slave and/or master) and it worked as expected. Their DROP TABLE failed, hence their attempt to DROP DATABASE.

2. I see errors in the error log about missing .ibd files, but they said they did not see any error in the error log (which I don't see a mention about it in the above error log snippet, which I see it straight-away in my error log).

Comment by Elena Stepanova [ 2018-08-13 ]

Yes, your mock-up only exhibits valid and expected behavior.
Here is another question then. From the error log above, at least this

2018-08-08 05:39:57 7f0ab8011b00 InnoDB: Error: Table "mysql"."innodb_index_stats" not found.

and most likely this

wrong number of columns in SYS_INDEXES record

indicates that there was either a problematic upgrade, or some tampering with the data directory (copying files from different versions around, partial backup/restore, etc.).
Which was it, and if it was an upgrade/crossgrade, then from which version?

Generated at Thu Feb 08 08:32:37 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.