[MDEV-33063] Assertion `thd->transaction.stmt.is_empty()' failed in bool trans_check(THD*) when using --slave_parallel_threads Created: 2023-12-19  Updated: 2023-12-22

Status: Confirmed
Project: MariaDB Server
Component/s: Replication
Affects Version/s: 10.4, 10.5, 10.6, 10.11, 11.0, 11.1, 11.2, 11.3, 11.4
Fix Version/s: 10.4, 10.5, 10.6, 10.11, 11.0, 11.1, 11.2, 11.3, 11.4

Type: Bug Priority: Major
Reporter: Ramesh Sivaraman Assignee: Andrei Elkin
Resolution: Unresolved Votes: 0
Labels: debug


 Description   

Test case

BASEDIR=/test/MD011223-mariadb-10.4.33-linux-x86_64-dbg
WORKDIR=/test
 
${BASEDIR}/bin/mariadb-admin -uroot --socket=${WORKDIR}/source/mysql.sock shutdown 2>/dev/null
${BASEDIR}/bin/mariadb-admin -uroot --socket=${WORKDIR}/replica/mysql.sock shutdown 2>/dev/null
 
rm -rf ${WORKDIR}/source ${WORKDIR}/replica
 
echo "Initializing database"
${BASEDIR}/scripts/mariadb-install-db --no-defaults --force --auth-root-authentication-method=normal  --basedir=${BASEDIR} --datadir=${WORKDIR}/source  > /dev/null 2>&1
${BASEDIR}/scripts/mariadb-install-db --no-defaults --force --auth-root-authentication-method=normal  --basedir=${BASEDIR} --datadir=${WORKDIR}/replica  > /dev/null 2>&1
 
echo "Starting source and replica"
 
${BASEDIR}/bin/mariadbd --no-defaults --core-file --basedir=${BASEDIR} --tmpdir=${WORKDIR}/source --datadir=${WORKDIR}/source --socket=${WORKDIR}/source/mysql.sock --log-error=${WORKDIR}/source/mysql.err --port=4401 --log_bin=binlog --binlog_format=ROW --log_bin_trust_function_creators=1 --server_id=1 &
 
${BASEDIR}/bin/mariadbd --no-defaults --core-file --basedir=${BASEDIR} --tmpdir=${WORKDIR}/replica --datadir=${WORKDIR}/replica --socket=${WORKDIR}/replica/mysql.sock --log-error=${WORKDIR}/replica/mysql.err --port=4402 --slave-parallel-threads=11  --server_id=2 & 
 
sleep 5;
 
echo "creating replication user"
${BASEDIR}/bin/mariadb -uroot --socket=${WORKDIR}/source/mysql.sock -e"DELETE FROM mysql.user WHERE user='';"  > /dev/null 2>&1
${BASEDIR}/bin/mariadb -uroot --socket=${WORKDIR}/source/mysql.sock -e"GRANT REPLICATION SLAVE ON *.* TO 'repl_user'@'%' IDENTIFIED BY 'repl_pass'; FLUSH PRIVILEGES;"  > /dev/null 2>&1
 
echo "starting replication"
 
${BASEDIR}/bin/mariadb -uroot --socket=${WORKDIR}/replica/mysql.sock -e"CHANGE MASTER TO MASTER_HOST='127.0.0.1', MASTER_PORT=4401, MASTER_USER='repl_user', MASTER_PASSWORD='repl_pass', MASTER_USE_GTID=slave_pos ;"  > /dev/null 2>&1
${BASEDIR}/bin/mariadb -uroot --socket=${WORKDIR}/replica/mysql.sock -e"START SLAVE;"  > /dev/null 2>&1
 
echo "Run test case"
 
${BASEDIR}/bin/mariadb -uroot --socket=${WORKDIR}/source/mysql.sock test <<'EOF'
ALTER TABLE mysql.gtid_slave_pos CHANGE seq_no seq_no CHAR(4);
CREATE TABLE t (c FLOAT,c2 FLOAT);
INSERT INTO t VALUES (0,0),(DEFAULT,DEFAULT);
CREATE SEQUENCE s START WITH 9223372036854775805 MINVALUE 9223372036854775804 MAXVALUE 9223372036854775806 CACHE 1 cycle;
EOF

Leads to

10.4.33 5f89045221717533994da01fc270d4851ccdc06c (Debug)

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50	../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
[Current thread is 1 (Thread 0x14e708cbe700 (LWP 734255))]
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x000014e72f2a2859 in __GI_abort () at abort.c:79
#2  0x000014e72f2a2729 in __assert_fail_base (fmt=0x14e72f438588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x55cbe49fd550 "thd->transaction.stmt.is_empty()", 
    file=0x55cbe4a31698 "/test/10.4_dbg/sql/transaction.cc", line=70, function=<optimized out>) at assert.c:92
#3  0x000014e72f2b3fd6 in __GI___assert_fail (assertion=assertion@entry=0x55cbe49fd550 "thd->transaction.stmt.is_empty()", file=file@entry=0x55cbe4a31698 "/test/10.4_dbg/sql/transaction.cc", 
    line=line@entry=70, function=function@entry=0x55cbe4a31927 "bool trans_check(THD*)") at assert.c:101
#4  0x000055cbe3ec51b8 in trans_check (thd=thd@entry=0x14e69c000d28) at /test/10.4_dbg/sql/handler.h:1712
#5  0x000055cbe3ec5c64 in trans_commit_implicit (thd=thd@entry=0x14e69c000d28) at /test/10.4_dbg/sql/transaction.cc:286
#6  0x000055cbe3f62809 in sequence_insert (thd=thd@entry=0x14e69c000d28, lex=0x14e69c004b90, org_table_list=org_table_list@entry=0x14e69c00a460) at /test/10.4_dbg/sql/sql_sequence.cc:380
#7  0x000055cbe3e40658 in mysql_create_table_no_lock (thd=thd@entry=0x14e69c000d28, db=db@entry=0x14e69c00a478, table_name=table_name@entry=0x14e69c00a488, create_info=create_info@entry=0x14e708cbbde0, 
    alter_info=alter_info@entry=0x14e708cbbd20, is_trans=is_trans@entry=0x14e708cbbc97, create_table_mode=0, table_list=0x14e69c00a460) at /test/10.4_dbg/sql/sql_table.cc:5311
#8  0x000055cbe3e40943 in mysql_create_table (thd=thd@entry=0x14e69c000d28, create_table=create_table@entry=0x14e69c00a460, create_info=create_info@entry=0x14e708cbbde0, 
    alter_info=alter_info@entry=0x14e708cbbd20) at /test/10.4_dbg/sql/sql_table.cc:5457
#9  0x000055cbe3e42309 in Sql_cmd_create_table_like::execute (this=<optimized out>, thd=0x14e69c000d28) at /test/10.4_dbg/sql/sql_table.cc:11943
#10 0x000055cbe3d7f6c0 in mysql_execute_command (thd=thd@entry=0x14e69c000d28) at /test/10.4_dbg/sql/sql_parse.cc:6266
#11 0x000055cbe3d81d6c in mysql_parse (thd=0x14e69c000d28, rawbuf=<optimized out>, length=<optimized out>, parser_state=parser_state@entry=0x14e708cbd8c0, is_com_multi=is_com_multi@entry=false, 
    is_next_command=is_next_command@entry=false) at /test/10.4_dbg/sql/sql_parse.cc:8062
#12 0x000055cbe418de33 in Query_log_event::do_apply_event (this=0x14e6b0023978, rgi=0x14e6b0022f50, 
    query_arg=0x14e6b0023ab3 "CREATE SEQUENCE s START WITH 9223372036854775805 MINVALUE 9223372036854775804 MAXVALUE 9223372036854775806 CACHE 1 cycle", q_len_arg=<optimized out>)
    at /test/10.4_dbg/sql/sql_class.h:1242
#13 0x000055cbe418e85e in Query_log_event::do_apply_event (this=<optimized out>, rgi=<optimized out>) at /test/10.4_dbg/sql/log_event.cc:5436
#14 0x000055cbe3ca077b in Log_event::apply_event (rgi=0x14e6b0022f50, this=0x14e6b0023978) at /test/10.4_dbg/sql/log_event.h:1494
#15 apply_event_and_update_pos_apply (ev=ev@entry=0x14e6b0023978, thd=thd@entry=0x14e69c000d28, rgi=rgi@entry=0x14e6b0022f50, reason=reason@entry=0) at /test/10.4_dbg/sql/slave.cc:3819
#16 0x000055cbe3caae30 in apply_event_and_update_pos_for_parallel (ev=ev@entry=0x14e6b0023978, thd=thd@entry=0x14e69c000d28, rgi=rgi@entry=0x14e6b0022f50) at /test/10.4_dbg/sql/slave.cc:4005
#17 0x000055cbe3f1d6c9 in rpt_handle_event (qev=qev@entry=0x14e6b0023ba8, rpt=rpt@entry=0x14e6b000bc28) at /test/10.4_dbg/sql/rpl_parallel.cc:64
#18 0x000055cbe3f22b0b in handle_rpl_parallel_thread (arg=0x14e6b000bc28) at /test/10.4_dbg/sql/rpl_parallel.cc:1480
#19 0x000014e72f7b3609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#20 0x000014e72f39f133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95



 Comments   
Comment by Roel Van de Paar [ 2023-12-22 ]

Issue reproduced on 10.6 and 11.3. For some reason, MTR seems unable to reproduce this issue. This should do it:

--source include/have_binlog_format_row.inc
--source include/have_innodb.inc
--source include/have_sequence.inc
--source include/master-slave.inc
--connection slave
STOP SLAVE;
SET GLOBAL slave_parallel_threads=10;
START SLAVE;
SELECT SLEEP(2);
--connection master
ALTER TABLE mysql.gtid_slave_pos CHANGE seq_no seq_no CHAR(4);
CREATE TABLE t (c FLOAT,c2 FLOAT) ENGINE=InnoDB;
INSERT INTO t VALUES (0,0),(DEFAULT,DEFAULT);
CREATE SEQUENCE s START WITH 9223372036854775805 MINVALUE 9223372036854775804 MAXVALUE 9223372036854775806 CACHE 1 CYCLE;

But where a non-MTR server crashes:

10.6.16 6f091434f313996436b4f01d7e7d62a9f3786958 (Debug)

...
2023-12-22 15:12:18 16 [ERROR] Slave (additional info): Failed to open mysql.gtid_slave_pos Error_code: 1944
2023-12-22 15:12:18 16 [Warning] Slave: Failed to open mysql.gtid_slave_pos Error_code: 1944
2023-12-22 15:12:18 16 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'binlog.000001' position 1303; GTID position '0-1-3'
2023-12-22 15:12:18 6 [Note] Error reading relay log event: slave SQL thread was killed
mariadbd: /test/10.6_dbg/sql/transaction.cc:79: bool trans_check(THD*): Assertion `thd->transaction->stmt.is_empty()' failed.

MTR keeps going (same build):

10.6.16 6f091434f313996436b4f01d7e7d62a9f3786958 (Debug)

...
2023-12-22 15:16:13 30 [ERROR] Slave (additional info): Failed to open mysql.gtid_slave_pos Error_code: 1944
2023-12-22 15:16:13 30 [Warning] Slave: Failed to open mysql.gtid_slave_pos Error_code: 1944
2023-12-22 15:16:13 30 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'master-bin.000001' position 673; GTID position ''
2023-12-22 15:16:13 21 [Note] Error reading relay log event: slave SQL thread was killed
2023-12-22 15:16:13 31 [ERROR] Incorrect definition of table mysql.gtid_slave_pos: expected column 'seq_no' at position 3 to have type bigint(20) unsigned, found type char(4).
2023-12-22 15:16:13 31 [Warning] Slave SQL: Failed to update GTID state in mysql.gtid_slave_pos, slave state may become inconsistent: 1944: Failed to open mysql.gtid_slave_pos, Gtid 0-1-4, Internal MariaDB error code: 1942
2023-12-22 15:16:13 21 [Note] Slave SQL thread exiting, replication stopped in log 'master-bin.000001' at position 1141; GTID position '', master: 127.0.0.1:16000
...

Note that --slave_parallel_threads is required.

Generated at Thu Feb 08 10:36:06 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.