[MDEV-14021] Failing assertion: table->get_ref_count() == 0 on ALTER TABLE with implicit ALGORITHM=INPLACE Created: 2017-10-06  Updated: 2018-04-04  Resolved: 2017-10-24

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - InnoDB
Affects Version/s: 10.2.8, 10.2
Fix Version/s: 10.2.10, 10.3.3

Type: Bug Priority: Major
Reporter: Hartmut Holzgraefe Assignee: Marko Mäkelä
Resolution: Fixed Votes: 0
Labels: None

Issue Links:
Relates
relates to MDEV-13918 Race condition between INFORMATION_SC... Closed
relates to MDEV-15776 Failing assertion: table->get_ref_cou... Closed

 Description   

A production server repeatedly crashed when trying to do an

ALTER TABLE ... ADD COLUMN ... AFTER ...

The assertion failure happened in row_merge_drop_table() from
storage/innobase/row/row0merge.cc which is only called when
using ALGORITHM=INPLACE, so explicitly usign =COPY helped
as a workaround here.

        /* There must be no open transactions on the table. */
 
        ut_a(table->get_ref_count() == 0);

Several other client connections were active in parallel to ALTER TABLE.

On a test system with less client traffic the problem was not reproducible.

2017-09-13 06:04:31 0x7f79cfacb700  InnoDB: Assertion failure in file /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.8/storage/innobase/row/row0merge.cc line 4569
 
InnoDB: Failing assertion: table->get_ref_count() == 0
 
InnoDB: We intentionally generate a memory trap.
 
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
 
InnoDB: If you get repeated assertion failures or crashes, even
 
InnoDB: immediately after the mysqld startup, there may be
 
InnoDB: corruption in the InnoDB tablespace. Please refer to
 
InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
 
InnoDB: about forcing recovery.
 
170913  6:04:31 [ERROR] mysqld got signal 6 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.2.8-MariaDB-log
 
key_buffer_size=33554432
 
read_buffer_size=131072
 
max_used_connections=110
 
max_threads=502
 
thread_count=117
 
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1135719 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x7f7834000bd8
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x7f79cfacad40 thread_stack 0x49000
 
(my_addr_resolve failure: fork)
 
/usr/sbin/mysqld(my_print_stacktrace+0x2e) [0x7f80e3506a7e]
 
/usr/sbin/mysqld(handle_fatal_signal+0x30d) [0x7f80e2f4cdbd]
 
/lib64/libpthread.so.0(+0xf370) [0x7f80e24b5370]
 
/lib64/libc.so.6(gsignal+0x37) [0x7f80e0a3e1d7]
 
/lib64/libc.so.6(abort+0x148) [0x7f80e0a3f8c8]
 
/usr/sbin/mysqld(+0x423578) [0x7f80e2d08578]
 
/usr/sbin/mysqld(+0x91613c) [0x7f80e31fb13c]
 
/usr/sbin/mysqld(+0x86475f) [0x7f80e314975f]
 
/usr/sbin/mysqld(+0x4180b1) [0x7f80e2cfd0b1]
 
/usr/sbin/mysqld(mysql_alter_table(THD*, char*, char*, HA_CREATE_INFO*, TABLE_LIST*, Alter_info*, unsigned int, st_order*, bool)+0x3a29) [0x7f80e2e46d99]
 
/usr/sbin/mysqld(Sql_cmd_alter_table::execute(THD*)+0x63a) [0x7f80e2e8f3ea]
 
/usr/sbin/mysqld(mysql_execute_command(THD*)+0x14eb) [0x7f80e2dbd3fb]
 
/usr/sbin/mysqld(mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool)+0x2de) [0x7f80e2dc50ae]
 
/usr/sbin/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool)+0x203d) [0x7f80e2dc805d]
 
/usr/sbin/mysqld(do_command(THD*)+0x149) [0x7f80e2dc8c69]
 
/usr/sbin/mysqld(do_handle_one_connection(CONNECT*)+0x1aa) [0x7f80e2e8c5da]
 
/usr/sbin/mysqld(handle_one_connection+0x3d) [0x7f80e2e8c6fd]
 
/usr/sbin/mysqld(+0x81bc0d) [0x7f80e3100c0d]
 
/lib64/libpthread.so.0(+0x7dc5) [0x7f80e24addc5]
 
/lib64/libc.so.6(clone+0x6d) [0x7f80e0b0076d]
 
 
 
Trying to get some variables.
 
Some pointers may be invalid and cause the dump to abort.
 
Query (0x7f783400f010): Alter table `...dbname...`.`...tablename...`       add column `...some_column...` text NULL after `...other_column...`
 
Connection ID (thread ID): 595304
 
Status: NOT_KILLED
 
 
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on
 
 
 
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
 
information that should help you find out what is causing the crash.



 Comments   
Comment by Elena Stepanova [ 2017-10-09 ]

Reproducible with a 2-thread concurrent test. On some reason, MTR refuses to cooperate and triggers MDEV-13918 instead, so here is the RQG version of the test:

git clone https://github.com/elenst/rqg --branch=experimental rqg-mdev14021
 
cd rqg-mdev14021
 
# To start the server and run the flow
 
$ perl ./runall-new.pl --grammar=conf/mariadb/bugs/mdev14021.yy --threads=2 --skip-gendata --duration=300 --basedir=<your basedir> --vardir=<your vardir>
 
# or, on a server already running on port 3306
 
perl ./gentest.pl --grammar=conf/mariadb/bugs/mdev14021.yy --threads=2 --duration=600 --queries=100M --dsn="dbi:mysql:user=root:host=127.0.0.1:port=3306:database=test"

10.2 a659291e85437

 
2017-10-09 19:08:18 0x7f104d134700  InnoDB: Assertion failure in file /data/src/10.2/storage/innobase/row/row0merge.cc line 4495
 
InnoDB: Failing assertion: table->get_ref_count() == 0
 
 
 
# 2017-10-09T19:08:28 [22088] #4  0x00007f106494b3fa in abort () from /lib/x86_64-linux-gnu/libc.so.6
 
# 2017-10-09T19:08:28 [22088] #5  0x000055ccb5bd49e3 in ut_dbg_assertion_failed (expr=expr@entry=0x55ccb63f8907 "table->get_ref_count() == 0", file=file@entry=0x55ccb63f7f08 "/data/src/10.2/storage/innobase/row/row0merge.cc", line=line@entry=4495) at /data/src/10.2/storage/innobase/ut/ut0dbg.cc:61
 
# 2017-10-09T19:08:28 [22088] #6  0x000055ccb60dc41c in row_merge_drop_table (trx=trx@entry=0x7f104f215198, table=table@entry=0x7f0ff0432a40) at /data/src/10.2/storage/innobase/row/row0merge.cc:4495
 
# 2017-10-09T19:08:28 [22088] #7  0x000055ccb61ded7b in dict_table_close_and_drop (trx=0x7f104f215198, table=0x7f0ff0432a40) at /data/src/10.2/storage/innobase/dict/dict0dict.cc:575
 
# 2017-10-09T19:08:28 [22088] #8  0x000055ccb6048cd0 in rollback_inplace_alter_table (prebuilt=<optimized out>, table=0x7f0ff0433848, ha_alter_info=0x7f104d12fbb0) at /data/src/10.2/storage/innobase/handler/handler0alter.cc:6734
 
# 2017-10-09T19:08:28 [22088] #9  ha_innobase::commit_inplace_alter_table (this=<optimized out>, altered_table=0x7f0ff0394178, ha_alter_info=0x7f104d12fbb0, commit=<optimized out>) at /data/src/10.2/storage/innobase/handler/handler0alter.cc:8419
 
# 2017-10-09T19:08:28 [22088] #10 0x000055ccb5d0e647 in mysql_inplace_alter_table (target_mdl_request=0x7f104d12fc40, alter_ctx=0x7f104d130800, inplace_supported=HA_ALTER_INPLACE_NO_LOCK_AFTER_PREPARE, ha_alter_info=0x7f104d12fbb0, altered_table=0x7f0ff0394178, table=0x7f0ff0433848, table_list=0x7f0ff0011148, thd=0x7f0ff00009a8) at /data/src/10.2/sql/sql_table.cc:7447
 
# 2017-10-09T19:08:28 [22088] #11 mysql_alter_table (thd=0x7f0ff00009a8, new_db=<optimized out>, new_name=<optimized out>, create_info=<optimized out>, table_list=0x7f0ff0011148, alter_info=<optimized out>, order_num=0, order=0x0, ignore=false) at /data/src/10.2/sql/sql_table.cc:9323
 
# 2017-10-09T19:08:28 [22088] #12 0x000055ccb5d55fc5 in Sql_cmd_alter_table::execute (this=0x2, thd=0x7f0ff00009a8) at /data/src/10.2/sql/sql_alter.cc:324
 
# 2017-10-09T19:08:28 [22088] #13 0x000055ccb5c7fd33 in mysql_execute_command (thd=0x7f0ff00009a8) at /data/src/10.2/sql/sql_parse.cc:6203
 
# 2017-10-09T19:08:28 [22088] #14 0x000055ccb5c8803a in mysql_parse (thd=0x7f0ff00009a8, rawbuf=<optimized out>, length=<optimized out>, parser_state=<optimized out>, is_com_multi=<optimized out>, is_next_command=<optimized out>) at /data/src/10.2/sql/sql_parse.cc:7875
 
# 2017-10-09T19:08:28 [22088] #15 0x000055ccb5c8bacc in dispatch_command (command=command@entry=COM_QUERY, thd=thd@entry=0x7f0ff00009a8, packet=packet@entry=0x7f0ff0006c59 "ALTER TABLE t1 ADD COLUMN IF NOT EXISTS c INT PRIMARY KEY /* QNO 8 CON_ID 12 */ ", packet_length=packet_length@entry=80, is_com_multi=is_com_multi@entry=false, is_next_command=is_next_command@entry=false) at /data/src/10.2/sql/sql_parse.cc:1812
 
# 2017-10-09T19:08:28 [22088] #16 0x000055ccb5c8c47e in do_command (thd=0x7f0ff00009a8) at /data/src/10.2/sql/sql_parse.cc:1360
 
# 2017-10-09T19:08:28 [22088] #17 0x000055ccb5d53304 in do_handle_one_connection (connect=connect@entry=0x55ccb91c5ec8) at /data/src/10.2/sql/sql_connect.cc:1354
 
# 2017-10-09T19:08:28 [22088] #18 0x000055ccb5d534a4 in handle_one_connection (arg=0x55ccb91c5ec8) at /data/src/10.2/sql/sql_connect.cc:1260
 
# 2017-10-09T19:08:28 [22088] #19 0x00007f1066887494 in start_thread (arg=0x7f104d134700) at pthread_create.c:333
 
# 2017-10-09T19:08:28 [22088] #20 0x00007f10649ff93f in clone () from /lib/x86_64-linux-gnu/libc.so.6

For the reference, the grammar:

conf/mariadb/bugs/mdev14021.yy

 
query_init:
 
  CREATE TABLE IF NOT EXISTS t1 (f INT) ENGINE=InnoDB; INSERT INTO t1 () VALUES ();
 
 
 
thread1:
 
  ALTER TABLE t1 ADD COLUMN IF NOT EXISTS c INT PRIMARY KEY; ALTER TABLE t1 DROP COLUMN IF EXISTS c;
 
    
thread2:
 
  SELECT COUNT(*) FROM INFORMATION_SCHEMA.INNODB_SYS_TABLESTATS;

Generated at Thu Feb 08 08:10:17 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.