[MDEV-15471] Assertion `new_clustered == ctx->need_rebuild()' failed in ha_innobase::commit_inplace_alter_table Created: 2018-03-05  Updated: 2023-04-27

Status: Stalled
Project: MariaDB Server
Component/s: Data Definition - Alter Table, Storage Engine - InnoDB
Affects Version/s: 10.3.3, 10.3, 10.4
Fix Version/s: 10.4

Type: Bug Priority: Major
Reporter: Elena Stepanova Assignee: Oleksandr Byelkin
Resolution: Unresolved Votes: 0
Labels: crash, fulltext, online-ddl

Attachments: File MDEV-15471.test     File dt.7z     HTML File threads    
Issue Links:
Relates
relates to MDEV-16329 Engine-independent online ALTER TABLE Closed

 Description   

https://api.travis-ci.org/v3/job/348679822/log.txt

10.3 d70573564cb2701076083ea7f55ae9539fad13b4

 
mysqld: /home/travis/src/storage/innobase/handler/handler0alter.cc:9195: virtual bool ha_innobase::commit_inplace_alter_table(TABLE*, Alter_inplace_info*, bool): Assertion `new_clustered == ctx->need_rebuild()' failed.
 
180304  2:46:51 [ERROR] mysqld got signal 6 ;
 
 
 
Some pointers may be invalid and cause the dump to abort.
 
Query (0x7f24a001f4d8): ALTER TABLE t1 ADD FULLTEXT KEY ftidx ( col4 ) /* QNO 1163 CON_ID 17 */
 
Connection ID (thread ID): 17
 
Status: NOT_KILLED


#7  0x00007f2502363ca2 in __GI___assert_fail (assertion=0x5600de9ba0d0 "new_clustered == ctx->need_rebuild()", file=0x5600de9b80a0 "/home/travis/src/storage/innobase/handler/handler0alter.cc", line=9195, function=0x5600de9bff60 <ha_innobase::commit_inplace_alter_table(TABLE*, Alter_inplace_info*, bool)::__PRETTY_FUNCTION__> "virtual bool ha_innobase::commit_inplace_alter_table(TABLE*, Alter_inplace_info*, bool)") at assert.c:101
 
#8  0x00005600de1d62fd in ha_innobase::commit_inplace_alter_table (this=0x7f24a80d15b8, altered_table=0x7f24a00e7340, ha_alter_info=0x7f24ec93d200, commit=true) at /home/travis/src/storage/innobase/handler/handler0alter.cc:9195
 
#9  0x00005600ddf81e12 in handler::ha_commit_inplace_alter_table (this=0x7f24a80d15b8, altered_table=0x7f24a00e7340, ha_alter_info=0x7f24ec93d200, commit=true) at /home/travis/src/sql/handler.cc:4403
 
#10 0x00005600de6ff1b7 in ha_partition::commit_inplace_alter_table (this=0x7f24a80d0d08, altered_table=0x7f24a00e7340, ha_alter_info=0x7f24ec93d200, commit=true) at /home/travis/src/sql/ha_partition.cc:10025
 
#11 0x00005600ddf81e12 in handler::ha_commit_inplace_alter_table (this=0x7f24a80d0d08, altered_table=0x7f24a00e7340, ha_alter_info=0x7f24ec93d200, commit=true) at /home/travis/src/sql/handler.cc:4403
 
#12 0x00005600ddd616de in mysql_inplace_alter_table (thd=0x7f24a000c610, table_list=0x7f24a001f610, table=0x7f24a80d00c0, altered_table=0x7f24a00e7340, ha_alter_info=0x7f24ec93d200, inplace_supported=HA_ALTER_INPLACE_SHARED_LOCK_AFTER_PREPARE, target_mdl_request=0x7f24ec93d370, alter_ctx=0x7f24ec93e020) at /home/travis/src/sql/sql_table.cc:7683
 
#13 0x00005600ddd679a0 in mysql_alter_table (thd=0x7f24a000c610, new_db=0x7f24a0010c58, new_name=0x7f24a0011008, create_info=0x7f24ec93ec00, table_list=0x7f24a001f610, alter_info=0x7f24ec93eb50, order_num=0, order=0x0, ignore=false) at /home/travis/src/sql/sql_table.cc:9847
 
#14 0x00005600dddee17a in Sql_cmd_alter_table::execute (this=0x7f24a001fd30, thd=0x7f24a000c610) at /home/travis/src/sql/sql_alter.cc:334
 
#15 0x00005600ddc92ca3 in mysql_execute_command (thd=0x7f24a000c610) at /home/travis/src/sql/sql_parse.cc:6280
 
#16 0x00005600ddc9782a in mysql_parse (thd=0x7f24a000c610, rawbuf=0x7f24a001f4d8 "ALTER TABLE t1 ADD FULLTEXT KEY ftidx ( col4 ) /* QNO 1163 CON_ID 17 */", length=71, parser_state=0x7f24ec9405e0, is_com_multi=false, is_next_command=false) at /home/travis/src/sql/sql_parse.cc:8009
 
#17 0x00005600ddc84f23 in dispatch_command (command=COM_QUERY, thd=0x7f24a000c610, packet=0x7f24a0016d11 "ALTER TABLE t1 ADD FULLTEXT KEY ftidx ( col4 ) /* QNO 1163 CON_ID 17 */ ", packet_length=72, is_com_multi=false, is_next_command=false) at /home/travis/src/sql/sql_parse.cc:1838
 
#18 0x00005600ddc8396f in do_command (thd=0x7f24a000c610) at /home/travis/src/sql/sql_parse.cc:1383
 
#19 0x00005600ddde8bef in do_handle_one_connection (connect=0x5600e020a050) at /home/travis/src/sql/sql_connect.cc:1402
 
#20 0x00005600ddde897c in handle_one_connection (arg=0x5600e020a050) at /home/travis/src/sql/sql_connect.cc:1308
 
#21 0x00007f2502f25184 in start_thread (arg=0x7f24ec941700) at pthread_create.c:312
 
#22 0x00007f250243203d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

experimental e42a181399012909fbda32f7e9b7be8b416175c9

 
perl /home/travis/rqg/runall-new.pl --duration=350 --threads=6 --seed=1520131581 --reporters=Backtrace,ErrorLog,Deadlock --validators=TransformerNoComparator --transformers=ConvertSubqueriesToViews,ConvertTablesToDerived,Count,DisableOptimizations,Distinct,EnableOptimizations,ExecuteAsCTE,ExecuteAsDeleteReturning,ExecuteAsDerived,ExecuteAsExcept,ExecuteAsExecuteImmediate,ExecuteAsInsertSelect,ExecuteAsIntersect,ExecuteAsSelectItem,ExecuteAsUnion,ExecuteAsUpdateDelete,ExecuteAsView,ExecuteAsWhereSubquery,Having,InlineSubqueries,LimitRowsExamined,OrderBy,StraightJoin,ExecuteAsPreparedTwice,ExecuteAsTrigger,ExecuteAsSPTwice,ExecuteAsFunctionTwice --redefine=conf/mariadb/general-workarounds.yy --mysqld=--log_output=FILE --mysqld=--log_bin_trust_function_creators=1 --mysqld=--log-bin --mysqld=--max-statement-time=30 --mysqld=--loose-debug_assert_on_not_freed_memory=0 --views --redefine=conf/mariadb/versioning.yy --basedir=/home/travis/server --grammar=conf/runtime/alter_online.yy --gendata=conf/runtime/alter_online.zz --engine=Aria --mysqld=--default-storage-engine=Aria --vardir=/home/travis/logs/vardir1_4

Not reproducible right away

 Comments   
Comment by Elena Stepanova [ 2018-04-06 ]

New occurrence on 10.3: https://api.travis-ci.org/v3/job/361732380/log.txt

Comment by Elena Stepanova [ 2018-05-06 ]

New occurrence: https://api.travis-ci.org/v3/job/375320577/log.txt

Comment by Alice Sherepa [ 2018-09-04 ]

Reproducible on 10.3 39c6775a3514d0e7edff9e9,
there is datadir attached in dt.7z (table t1 in db test, crashing query is
ALTER TABLE t1 ADD FULLTEXT KEY ftidx ( col4(5)) )

mysqld: src/storage/innobase/handler/handler0alter.cc:9436: virtual bool ha_innobase::commit_inplace_alter_table(TABLE*, Alter_inplace_info*, bool): Assertion `new_clustered == ctx->need_rebuild()' failed.
 
180904 17:41:55 [ERROR] mysqld got signal 6 ;
 
 
 
Server version: 10.3.10-MariaDB-debug
 
linux/raise.c:54(__GI_raise)[0x7fe092d94428]
 
stdlib/abort.c:91(__GI_abort)[0x7fe092d9602a]
 
assert/assert.c:92(__assert_fail_base)[0x7fe092d8cbd7]
 
/lib/x86_64-linux-gnu/libc.so.6(+0x2dc82)[0x7fe092d8cc82]
 
handler/handler0alter.cc:9438(ha_innobase::commit_inplace_alter_table(TABLE*, Alter_inplace_info*, bool))[0x55aa756fd217]
 
sql/handler.cc:4481(handler::ha_commit_inplace_alter_table(TABLE*, Alter_inplace_info*, bool))[0x55aa75425f9e]
 
sql/ha_partition.cc:10089(ha_partition::commit_inplace_alter_table(TABLE*, Alter_inplace_info*, bool))[0x55aa75aea4e7]
 
sql/handler.cc:4481(handler::ha_commit_inplace_alter_table(TABLE*, Alter_inplace_info*, bool))[0x55aa75425f9e]
 
sql/sql_table.cc:7618(mysql_inplace_alter_table(THD*, TABLE_LIST*, TABLE*, TABLE*, Alter_inplace_info*, enum_alter_inplace_result, MDL_request*, Alter_table_ctx*))[0x55aa752720ab]
 
sql/sql_table.cc:9712(mysql_alter_table(THD*, st_mysql_const_lex_string const*, st_mysql_const_lex_string const*, HA_CREATE_INFO*, TABLE_LIST*, Alter_info*, unsigned int, st_order*, bool))[0x55aa7527c6a9]
 
sql/sql_alter.cc:495(Sql_cmd_alter_table::execute(THD*))[0x55aa752e6a70]
 
sql/sql_parse.cc:6285(mysql_execute_command(THD*))[0x55aa751d0156]
 
sql/sql_parse.cc:8089(mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool))[0x55aa751d1db8]
 
sql/sql_parse.cc:1874(dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool))[0x55aa751d43e7]
 
sql/sql_parse.cc:1395(do_command(THD*))[0x55aa751d7485]
 
sql/sql_connect.cc:1402(do_handle_one_connection(CONNECT*))[0x55aa752e2d49]
 
sql/sql_connect.cc:1310(handle_one_connection)[0x55aa752e2efb]
 
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7fe0937bb6ba]
 
x86_64/clone.S:111(clone)[0x7fe092e6641d]
 
 
 
Query (0x7fe0200137c8): ALTER TABLE t1 ADD FULLTEXT KEY ftidx ( col4(5))

Comment by Marko Mäkelä [ 2019-03-22 ]

In dt.7z, the table consists of two partitions. The first partition is missing the InnoDB-internal hidden index on the hidden column FTS_DOC_ID_INDEX(FTS_DOC_ID), but the second partition does contain them. Hence, only the first partition would be rebuilt. Both partitions are in sync with the .frm file, because the hidden index and column are only known to InnoDB. However, the commit of ALTER TABLE ought to be atomic for partitioned tables, and hence all partitions should have equivalent schema. It turns out that MariaDB did merge the MySQL 5.6.11 bug fix.

The assertion is saying that all partitions should be either rebuilt or modified in-place, but a mixture is not allowed.

Unrelated to this test case, which hints that something has been broken in an earlier ALTER TABLE operation, the assertion may be wrong. With operations that increment innodb_instant_alter_column and avoid a table rebuild by abandoning the canonical format, it could happen that some partitions avoid a rebuild, while others will use an instantaneous operation. Hence, we should evaluate ctx->need_rebuild() for every partition separately. The hidden indexed column FTS_DOC_ID can never be added without rebuilding the table, though.

Does this assertion really not fail on 10.2?

Comment by Marko Mäkelä [ 2019-03-22 ]

I am afraid that the logic that all partitions must be rebuilt, or all must not be rebuilt, is very hard-wired. For the rebuild case, an InnoDB internal transaction object would be allocated for the first partition already. Lifting that restriction would require changes to the way how ha_innobase::prepare_inplace_alter_table() is invoked on partitioned tables. We would need some kind of ‘atomic prepare’, just like we have the ‘atomic commit’ where ha_innobase::commit_inplace_alter_table() is invoked only once.

Also, the actual transaction commit is more complex for the ‘rebuild’ phase, because it requires extra calls to fil_mtr_rename_log(). This code would be removed in MDEV-16329 Cross-engine ALTER ONLINE TABLE.

elenst or alice, can you please try to find out which ALTER TABLE operations were attempted to be executed on the partitioned table before the crashing one?

Comment by Matthias Leich [ 2019-03-22 ] 

MTR based replay test (MDEV-15471.test)
 
Printed here just for information because attached too.
 
 
 
--source include/have_innodb.inc
 
--source include/have_partition.inc
 
 
 
# col2  VARCHAR(500) but also TEXT end with the assert.
 
CREATE TABLE t1 (col1 INT, col2 VARCHAR(500), FULLTEXT KEY ftidx ( col2 ))
 
ENGINE = InnoDB WITH SYSTEM VERSIONING
 
PARTITION BY system_time LIMIT 6 ( PARTITION ver_p1 HISTORY, PARTITION ver_pn CURRENT );
 
ALTER TABLE t1 DROP KEY ftidx ;
 
ALTER TABLE t1 ADD PARTITION (PARTITION ver_p2 HISTORY) ;
 
ALTER TABLE t1 ADD FULLTEXT KEY ftidx ( col2 );
 
 
 
DROP TABLE t1;
 
 
 
Replay on
 
10.3.14 commit 1c60f40868f6067be81a50198e1c72df08814adf 2019-03-22
 
10.4.4    commit ca80e14a882b1928350ef0522607cd1011048941 2019-03-22

Comment by Thirunarayanan Balathandayuthapani [ 2019-10-01 ]

The following test case repeats the issue without system versioning:

--source include/have_innodb.inc
 
--source include/have_partition.inc
 
 
 
CREATE TABLE t1 (a INT, b VARCHAR(10), fulltext ftidx(b)) ENGINE=InnoDB
 
PARTITION BY RANGE(a)
 
(PARTITION pa VALUES LESS THAN (3),
 
PARTITION pb VALUES LESS THAN (6));
 
ALTER TABLE t1 DROP KEY ftidx ;
 
ALTER TABLE t1 ADD PARTITION (PARTITION pc VALUES LESS THAN (10)) ;
 
ALTER TABLE t1 ADD FULLTEXT KEY ftidx ( b );
 
DROP TABLE t1;

The following commit enables the full text index in partition (10.3.3):

commit 2f09b28e0f7f8e4291003a9b46c62b54f3bf4454
 
Author: Monty <monty@mariadb.org>
 
Date:   Tue Sep 26 00:56:10 2017 +0300
 
 
 
    Adding Full Text Search support to partitions
 
    
    Contains Spiral patches:
 
    007_mariadb-10.2.0.partition_fulltext.diff  MDEV-7705
 
    038_mariadb-10.2.0.partition_fulltext2.diff MDEV-7734

InnoDB doesn't drop FTS_DOC_ID and FTS_DOC_ID_INDEX even if all fts indexes are dropped. So when new partition is added, TABLE object
doesn't aware of hidden fulltext FTS_DOC_ID. So it leads to build the partition without fts_doc_id hidden column.
Subsequent alter does add fulltext index and it fails because few partitions doesn't need to rebuild (it has FTS_DOC_ID) and other partition doesn't
have FTS_DOC_ID, it requires rebuild.

The following patch disables the FULL_TEXT flag for partition. It is used for testing purpose only as of now.

diff --git a/sql/ha_partition.cc b/sql/ha_partition.cc
 
index 5a78249644d..c87207cb4eb 100644
 
--- a/sql/ha_partition.cc
 
+++ b/sql/ha_partition.cc
 
@@ -74,6 +74,7 @@
 
                                        HA_REC_NOT_IN_SEQ | \
 
                                        HA_CAN_REPAIR)
 
 #define PARTITION_DISABLED_TABLE_FLAGS (HA_CAN_GEOMETRY | \
 
+                                       HA_CAN_FULLTEXT | \
 
                                         HA_DUPLICATE_POS | \
 
                                         HA_CAN_INSERT_DELAYED | \
 
                                         HA_READ_BEFORE_WRITE_REMOVAL |\

Matthias can use the above patch to disable FTS in partition.

Comment by Marko Mäkelä [ 2019-10-01 ]

The assertion is failing, because the native ALTER TABLE in InnoDB is based on the assumption that each partition of the table has equivalent schema and therefore the ALTER TABLE will invoke a similar operation on each partition.

In the test case, some partitions would require rebuilding (to add a hidden FTS_DOC_ID column that is not visible in TABLE_SHARE) while others could be modified in-place, just to add the FULLTEXT INDEX.

An easy fix could be to ensure that fulltext indexes remain disabled on partitioned InnoDB tables.

A more complex fix would be to rewrite all ALTER TABLE code in some way that guarantees that each partition inside InnoDB will be created in the same way. It might be sufficient to do this for ADD PARTITION and similar operations. If we do that, some tests for fulltext search with partitioned InnoDB tables should be added to the regression test suite, and the fix should be extensively tested with RQG.

Generated at Thu Feb 08 08:21:34 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.