[MDEV-31755] Replica's DML event deadlocks with online alter table Created: 2023-07-20  Updated: 2023-08-16  Resolved: 2023-08-16

Status: Closed
Project: MariaDB Server
Component/s: Data Definition - Alter Table
Affects Version/s: None
Fix Version/s: 11.2.1

Type: Bug Priority: Critical
Reporter: Nikita Malyavin Assignee: Nikita Malyavin
Resolution: Fixed Votes: 0
Labels: None

Attachments: HTML File slave_status     HTML File status     HTML File threads_slave     HTML File threads_slave_full     HTML File variables    
Issue Links:
Problem/Incident
is caused by MDEV-16329 Engine-independent online ALTER TABLE Closed

 Description   

A deadlock of two phase binlogged "start" ALTER with the following in binlog order DML statement on the parallel slave is caused
by partly missed integration of a MDEV-16329's feature with the MDEV-11675 framework. A stronger than required MDL lock
by ALTER prevents the DML execution on slave (while it succeed on master).
The following mtr test exposes it

source include/master-slave.inc;
 
 
 
--connection slave
 
source include/stop_slave.inc;
 
 
 
#
 
# MDEV-31755 Replica's DML event deadlocks wit online alter table
 
#
 
# Three threads for SA,U,CA
 
--let $slave_parallel_threads=`select @@global.slave_parallel_threads`
 
--let $slave_parallel_mode=   `select @@global.slave_parallel_mode`
 
set global slave_parallel_threads=3;
 
set global slave_parallel_mode= optimistic;
 
 
 
--connection master
 
create table t (id int, a int, b text, primary key (id));
 
insert into t values (1,10,''),(2,20,'');
 
 
 
set @@session.binlog_alter_two_phase=1;
 
set debug_sync= 'alter_table_online_progress signal ready wait_for go';
 
send alter table t force, algorithm=copy, lock=none;
 
 
 
connect (con1,localhost,root,,);
 
set debug_sync= 'now wait_for ready';
 
 
 
update t set a = 1;
 
 
 
set debug_sync= 'now signal go';
 
 
 
--connection master
 
--reap
 
--source include/save_master_gtid.inc
 
 
 
--connection slave
 
source include/start_slave.inc;
 
--source include/sync_with_master_gtid.inc



 Comments   
Comment by Kristian Nielsen [ 2023-07-20 ]

How can I view the details of this issue? The Google drive link doesn't seem to work.

Comment by Elena Stepanova [ 2023-07-20 ]

The google drive link should have never been there. I hope nikitamalyavin will write a proper description soon enough.
Meanwhile, I've attached some of the files from that archive:
status variables threads_slave_full threads_slave slave_status

I cannot however attach the actual binlog which I guess is the most important, if you need it, I'll upload it to the foundation FTP's public when I find out how it has been uploaded to ftp.mariadb.org/public/MDEV-31755-deadlocked_replication_of_alter.tar.gz

Comment by Kristian Nielsen [ 2023-07-21 ]

Thanks, Elena, Nikita.

I was a bit confused at first having only threads_slave_full to look into, as the GTIDs and sub_ids are a bit strange. But it's possible that they are just incorrect values shown by GDB. It makes sense that different locking for the ALTER on master and slave can cause this kind of hang.

It's something that worries be in general: the START ALTER feels fragile since it can hang if any kind of different lock conflict occurs against a later query. It feels like we need the lock wait report and kill for metadata locks that we currently have for InnoDB row locks. (But that's probably a separate issue from this particular issue).

- Kristian.

Comment by Nikita Malyavin [ 2023-07-21 ]

knielsen I have counted 6 locking systems in the server. Some of them (like innodb's row/table locks, or my_safe_mutex, or MDL) have their own lock detection systems, others don't, at all. It'd be nice to generalize it all in one deadlock detection module and spread to the locks not covered.

Not sure though, can we do it here – in general, condvars can't be deadlock-detected (we don't know who else can signal), but here is the only producer that is known to us, so we should be able.

Generated at Thu Feb 08 10:26:13 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.