Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-24844

slave replicate only DDL from master after failover

Details

    • Bug
    • Status: Open (View Workflow)
    • Critical
    • Resolution: Unresolved
    • 10.1.22
    • None
    • None
    • OS: Debian GNU/Linux 9.8 (stretch)

    Description

      Background:

      • My cluster was M -> S replication(row-based), M crashed with some bug, see screenshot old_master_crash.png
      • Then S was promoted to be new master

      Description:

      • A new slave instance was deployed by dba and data was loaded by mydumper&myloader from new master (standard operating procedure, works for all other failover scenario)
      • All the things look good by command "SHOW SLAVE STATUS", however, only DDL can be applyed from relay log automaticlly, DML is not applyed anymore...
      • All the things look good from replay log on slave, see screenshot "replay_log_on_new_slave.png"
      • Only DDL can be found from slave binlog
      • server_id is defferent between new master and new slave
      • Tried to deploy a new slave, not working
      • please see mysql config file "my-3313.cnf" of new slave with attachment

      Kindly advise the probable cause and how to fix this promblem.

      Attachments

        1. show_slave_status.png
          show_slave_status.png
          136 kB
        2. relay_log_on_new_slave.png
          relay_log_on_new_slave.png
          217 kB
        3. old_master_crash.png
          old_master_crash.png
          226 kB
        4. mysql-bin.002415.part
          8 kB
        5. my-3313.cnf
          5 kB
        6. binlog_on_new_slave.png
          binlog_on_new_slave.png
          171 kB
        7. binlog_new_master.log
          8 kB

        Activity

          alice Alice Sherepa added a comment -

          I will describe as I understand the situation, please correct me.
          Currently, you have M->S (row-based replication?). (Master was a slave before the crash)
          my-3313.cnf - is the configuration of a new slave? Please add also .cnf from the master.
          Then you perform create table + insert on the master and on the slave the table was created, but no data was inserted. Could you please attach binlog file from the master?
          Can you reproduce this with a newer MariaDB release (10.1 is EOL)?

          alice Alice Sherepa added a comment - I will describe as I understand the situation, please correct me. Currently, you have M->S (row-based replication?). (Master was a slave before the crash) my-3313.cnf - is the configuration of a new slave? Please add also .cnf from the master. Then you perform create table + insert on the master and on the slave the table was created, but no data was inserted. Could you please attach binlog file from the master? Can you reproduce this with a newer MariaDB release (10.1 is EOL)?
          nevfo youngchen added a comment -

          binlog_new_master.log

          Hi, thanks for your quick response, please see my answer below:

          row-based replication?
          – yes

          my-3313.cnf - is the configuration of a new slave?
          – yes

          Then you perform create table + insert on the master and on the slave the table was created, but no data was inserted.
          – yes

          Can you reproduce this with a newer MariaDB release (10.1 is EOL)?
          – let me have a try and get back to you

          please see binlog of new master from attachment "binlog_new_master.log"

          nevfo youngchen added a comment - binlog_new_master.log Hi, thanks for your quick response, please see my answer below: row-based replication? – yes my-3313.cnf - is the configuration of a new slave? – yes Then you perform create table + insert on the master and on the slave the table was created, but no data was inserted. – yes Can you reproduce this with a newer MariaDB release (10.1 is EOL)? – let me have a try and get back to you please see binlog of new master from attachment "binlog_new_master.log"
          nevfo youngchen added a comment -

          Tried to setup a new slave with 10.3.18, not working...

          nevfo youngchen added a comment - Tried to setup a new slave with 10.3.18, not working...
          alice Alice Sherepa added a comment - - edited

          Could you please add the actual binlog file, not the binlog_new_master.log, because it is not clear how it is produced. One of suspicion is the option --base64-output=DECODE-ROWS (as described here https://www.percona.com/blog/2015/07/30/why-base64-outputdecode-rows-does-not-print-row-events-in-mysql-binary-logs/)

          alice Alice Sherepa added a comment - - edited Could you please add the actual binlog file, not the binlog_new_master.log, because it is not clear how it is produced. One of suspicion is the option --base64-output=DECODE-ROWS (as described here https://www.percona.com/blog/2015/07/30/why-base64-outputdecode-rows-does-not-print-row-events-in-mysql-binary-logs/ )
          nevfo youngchen added a comment - mysql-bin.002415.part
          nevfo youngchen added a comment -

          Please see the binlog file "mysql-bin.002415.part", thanks.

          nevfo youngchen added a comment - Please see the binlog file "mysql-bin.002415.part", thanks.

          People

            Unassigned Unassigned
            nevfo youngchen
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.