[MDEV-23691] S3 storage engine: delayed slave can drop the table Created: 2020-09-08  Updated: 2020-10-21  Resolved: 2020-10-21

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - S3
Affects Version/s: 10.5
Fix Version/s: 10.5.7

Type: Bug Priority: Critical
Reporter: Sergei Petrunia Assignee: Michael Widenius
Resolution: Fixed Votes: 0
Labels: None


 Description   

The s3_slave_ignore_updates option is supposed to allow running a master-slave setup where both master and slave are pointed at the same S3 storage. Yet, in some scenarios, slave will drop the table, destroying the data that the master has put there.

Put this into suite/s3/rpl_a.cnf:

!include ../rpl/my.cnf
!include ./my.cnf
!include ./slave.cnf
 
[mysqld.2]
s3_slave_ignore_updates=1
 
[mysqld.1]
s3_replicate_alter_as_create_select=1

Put this into suite/s3/rpl_a.test:

--source include/have_s3.inc
--source include/have_innodb.inc
 
--source include/have_binlog_format_mixed.inc
--source include/master-slave.inc

--disable_warnings
drop table if exists ten, t100, t101;
--enable_warnings
 
create table ten(a int primary key);
insert into ten values (0),(1),(2),(3),(4),(5),(6),(7),(8),(9);
 
connection slave;
stop slave;
connection master;

 
#
# Create version 1 of the table
#
create /*or replace*/ table t100 (
  pk varchar(100)
) engine = 'innodb';
 
insert into t100 values ('old data');
alter table t100 engine=s3;

#
# Create version 2 of the table
#
drop table t100;
create /*or replace*/ table t100 (
  pk varchar(100)
) engine= innodb;
insert into t100 select 'new data' from ten;
alter table t100 engine=s3;
 
select *, 'before slave start' from t100;

#
#  Now, start the slave
#
connection slave;
start slave;
connection master;
 
--sync_slave_with_master
connection master;
 
flush tables;
select *, 'after slave start' from t100;
 
--sync_slave_with_master
 
# This will fail with an error:
# mysqltest: At line 49: query 'select *, 'after slave start' from 
# t100' failed: 1146: Table 'test.t100' doesn't exist
select * from t100;



 Comments   
Comment by Sergei Petrunia [ 2020-09-08 ]

Debug printout on the slave (the printout patch: https://gist.github.com/spetrunia/a3a941eeec55273fc806bf88d872e6c3 )

AAA query start slave
AAA query create table ten(a int primary key)
AAA query insert into ten values (0),(1),(2),(3),(4),(5),(6),(7),(8),(9)
AAA query COMMIT
AAA query create /*or replace*/ table t100 (
pk varchar(100)
) engine = 'innodb'
AAA ha_s3::delete_table(./test/t100)
AAA query insert into t100 values ('old data')
AAA query alter table t100 engine=s3
AAA query DROP TABLE IF EXISTS `t100` /* generated by server */
AAA ha_s3::delete_table(./test/t100)
AAA query create /*or replace*/ table t100 (
pk varchar(100)
) engine= innodb
AAA query insert into t100 select 'new data' from ten
AAA query alter table t100 engine=s3
AAA query flush tables

Note that some of these statements might be ignored on the slave.

But also note the calls to ha_s3::delete_table().

Comment by Sergei Petrunia [ 2020-09-08 ]

A bit more detail about what happens on the slave here:

AAA query create /*or replace*/ table t100 (
            pk varchar(100)
          ) engine = 'innodb'
AAA     ha_s3::delete_table(./test/t100)
AAA     ha_innobase::create(name=./test/t100)

AAA query insert into t100 values ('old data')
AAA     ha_innobase::write_row()

AAA query alter table t100 engine=s3
AAA     ha_check_if_updates_are_ignored(): ignoring for stmt=alter table t100 engine=s3

So, the slave will drop the table in S3 when creating the InnoDB table (the first segment).

This is wrong (as the table in S3 might be the newer version that has more data)
On the other hand, that's the only option as the slave needs to create the InnoDB table and one is not allowed to have two tables with the same name in different engines.

Then, ALTER TABLE ... engine=s3 is ignored. This is why we get "Table doesn't exist" error in the original testcase.

Comment by Michael Widenius [ 2020-09-12 ]

The slave should not be able to drop an S3 table.

There is a bug in 10.5 introduced with the fix for force drop table that affects S3 that I have been working on fixing for the last two days (hope to have a final patch by tomorrow)

Comment by Michael Widenius [ 2020-10-21 ]

Pushed into 10.5 tree

Generated at Thu Feb 08 09:24:18 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.