[MDEV-24844] slave replicate only DDL from master after failover Created: 2021-02-11  Updated: 2021-03-15

Status: Open
Project: MariaDB Server
Component/s: Platform Debian, Replication
Affects Version/s: 10.1.22
Fix Version/s: None

Type: Bug Priority: Critical
Reporter: youngchen Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None
Environment:

OS: Debian GNU/Linux 9.8 (stretch)

Attachments: Text File binlog_new_master.log     PNG File binlog_on_new_slave.png     File my-3313.cnf     File mysql-bin.002415.part     PNG File old_master_crash.png     PNG File relay_log_on_new_slave.png     PNG File show_slave_status.png    

 Description   

Background:

  • My cluster was M -> S replication(row-based), M crashed with some bug, see screenshot old_master_crash.png
  • Then S was promoted to be new master

Description:

  • A new slave instance was deployed by dba and data was loaded by mydumper&myloader from new master (standard operating procedure, works for all other failover scenario)
  • All the things look good by command "SHOW SLAVE STATUS", however, only DDL can be applyed from relay log automaticlly, DML is not applyed anymore...
  • All the things look good from replay log on slave, see screenshot "replay_log_on_new_slave.png"
  • Only DDL can be found from slave binlog
  • server_id is defferent between new master and new slave
  • Tried to deploy a new slave, not working
  • please see mysql config file "my-3313.cnf" of new slave with attachment

Kindly advise the probable cause and how to fix this promblem.



 Comments   
Comment by Alice Sherepa [ 2021-02-11 ]

I will describe as I understand the situation, please correct me.
Currently, you have M->S (row-based replication?). (Master was a slave before the crash)
my-3313.cnf - is the configuration of a new slave? Please add also .cnf from the master.
Then you perform create table + insert on the master and on the slave the table was created, but no data was inserted. Could you please attach binlog file from the master?
Can you reproduce this with a newer MariaDB release (10.1 is EOL)?

Comment by youngchen [ 2021-02-12 ]

binlog_new_master.log

Hi, thanks for your quick response, please see my answer below:

row-based replication?
– yes

my-3313.cnf - is the configuration of a new slave?
– yes

Then you perform create table + insert on the master and on the slave the table was created, but no data was inserted.
– yes

Can you reproduce this with a newer MariaDB release (10.1 is EOL)?
– let me have a try and get back to you

please see binlog of new master from attachment "binlog_new_master.log"

Comment by youngchen [ 2021-02-12 ]

Tried to setup a new slave with 10.3.18, not working...

Comment by Alice Sherepa [ 2021-02-12 ]

Could you please add the actual binlog file, not the binlog_new_master.log, because it is not clear how it is produced. One of suspicion is the option --base64-output=DECODE-ROWS (as described here https://www.percona.com/blog/2015/07/30/why-base64-outputdecode-rows-does-not-print-row-events-in-mysql-binary-logs/)

Comment by youngchen [ 2021-02-12 ]

mysql-bin.002415.part

Comment by youngchen [ 2021-02-12 ]

Please see the binlog file "mysql-bin.002415.part", thanks.

Generated at Thu Feb 08 09:33:07 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.