[MDEV-25607] Auto-generated DELETE from HEAP table can break replication Created: 2021-05-05  Updated: 2023-12-15

Status: In Review
Project: MariaDB Server
Component/s: Replication, Storage Engine - Memory
Affects Version/s: 10.2, 10.3, 10.4, 10.5, 10.6, 10.7, 10.8, 10.9
Fix Version/s: 10.4, 10.5, 10.6

Type: Bug Priority: Critical
Reporter: Elena Stepanova Assignee: Andrei Elkin
Resolution: Unresolved Votes: 1
Labels: None


 Description   

After server restart, a DELETE query is written into the binary log for every HEAP table to reflect the restart emptying them. It is written unconditionally, regardless whether it's actually executable or not. If it is not, it causes replication abort.

In the example test case below DELETE causes an error because the table has a DELETE trigger which refers to a non-existing table.

--source include/master-slave.inc
 
reset master;
 
create table t (a int) engine=MEMORY;
create trigger tr after delete on t for each row update t2 set a = 1;
insert into t values (1);
 
--let $rpl_server_number= 1
--source include/rpl_restart_server.inc
 
check table t;
 
--sync_slave_with_master
 
# Cleanup
--connection master
drop table t;
--source include/rpl_end.inc

10.2 e788738e

master-bin.000002	4	Format_desc	1	256	Server ver: 10.2.38-MariaDB-debug-log, Binlog ver: 4
master-bin.000002	256	Gtid_list	1	299	[0-1-3]
master-bin.000002	299	Binlog_checkpoint	1	343	master-bin.000002
master-bin.000002	343	Gtid	1	385	GTID 0-1-4
master-bin.000002	385	Query	1	474	DELETE FROM `test`.`t`

Last_Errno	1146
Last_Error	Error 'Table 'test.t2' doesn't exist' on query. Default database: 'test'. Query: 'DELETE FROM `test`.`t`'



 Comments   
Comment by Brandon Nesterenko [ 2021-06-07 ]

Conceptually, the fix for this would depend on what we would determine to be the underlying cause.. What would be the contributing "issue(s)" here (as opposed to allowable input/behavior)?
1. An incomplete binlog recording of the master dying. Additionally, couldn't this issue "silently" break replication, even if the slave didn't die? I.e., if the trigger was executable, wouldn't it continue to persist on the slave when it would be deleted from the master after it died?
2. The slave crashing because it can't execute the trigger.
3. The trigger having invalid references.
4. Anything/something else?

Each problem would then have a different fix:
1. Write more events in the binlog to more accurately the state of the master, e.g. additionally include DROP TRIGGER statements in the binlog for tables with MEMORY based storage engines (before the DELETE).
2. Change the error handling logic on the slave to allow certain "fatal" errors in certain situations.
3. Validate the trigger query (on the master?).

My thought would be just #1. Elkin

Comment by Andrei Elkin [ 2021-06-08 ]

bnestere: There's no crashing really, rather the slave stops with an error.
According to the description (the slave error) the slave does not have t2 table.
I'd say simply does not have.

But that's an inconsistency with the master in the very setup that is required.
I have to involve elenst again to help me to understand what is the problem or is there any.
I am puzzle 'cos if t were not of Memory type, and the user DELETE:d from it, it'd be the same error.

Comment by Andrei Elkin [ 2021-06-08 ]

elenst: Finally I understood the "magic". Needed to have the 2nd chance . DELETE ends up in the binlog bypassing execution , so it misses its chance to hit the slave side error, on master.

Comment by Elena Stepanova [ 2021-06-08 ]

That's exactly right.

The essence of this report is that DELETE which, if it had been executed, would have caused an error on the master and thus would have never been written into the binary log, gets written there directly by the special HEAP logic, gets replicated to the slave, expectedly ends up with an error on the slave, and thus causes replication abort due to the error code mismatch (0 vs non-zero). There is no previous discrepancy between master and slave, besides the inevitable loss of contents of the memory table after master restart.

Hypothetically it is not anyhow limited to triggers, it can be any error which satisfies the conditions above. Triggers were just the only example I came up with for DELETE failure on a memory table.

Surely, as bnestere describes, there can be numerous reasons for error code mismatches causing replication abort. They would be different problems though, or non-bugs if, for example, they are caused by previous schema/data discrepancy.

The problem is, I don't see a way out of it. Even if you don't write DELETE on the master, or if you ignore the event failure on the slave – still, we will end up with discrepancy in the table data, which will very likely make replication abort anyway, just later. Somehow the table on the slave has to be emptied to preserve consistency.

Comment by Andrei Elkin [ 2021-06-09 ]

elenst, (to The problem is) indeed! So we should only binlog it but with a hint to slave(s) which is not call triggers - bnestere, right? And the best hint to my knowledge would be logging TRUNCATE instead of DELETE !

Comment by Brandon Nesterenko [ 2021-06-09 ]

Elkin Nice. Just did some testing/debugging with truncate, and I agree, I think truncate would work well here.

Comment by Brandon Nesterenko [ 2021-06-09 ]

Hi Andrei,

The fix is ready for review.

Patch

Buildbot

Comment by Andrei Elkin [ 2023-12-15 ]

The status confirmed. The review should be completed, to make the patch into the upcoming CS release.

Generated at Thu Feb 08 09:38:58 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.