Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-40958

Intermittent network issue disables semi-sync for the rest of the replication

    XMLWordPrintable

Details

    Description

      Reproduction

      rpl_semi_sync_slave_reply_fail.patch

      diff --git a/mysql-test/suite/rpl/t/rpl_semi_sync_slave_reply_fail.test b/mysql-test/suite/rpl/t/rpl_semi_sync_slave_reply_fail.test
      index 84462ed6426..ef78d2987d7 100644
      --- a/mysql-test/suite/rpl/t/rpl_semi_sync_slave_reply_fail.test
      +++ b/mysql-test/suite/rpl/t/rpl_semi_sync_slave_reply_fail.test
      @@ -16,7 +16,7 @@
       #    4 - Remove the debug simulation and do some more DML operations on master
       #        and wait for them to be replicated.
       #    5 - Slave will be able to replicate and data is consistent on both master
      -#        and slave. Semi sync will be automatically turned on.
      +#        and slave. **Semi sync will be automatically turned on.**
       #
       # ==== References ====
       #
      @@ -71,14 +71,29 @@ SET GLOBAL debug_dbug= @save_debug;
       --connection master
       insert into t1 values (10);
       --sync_slave_with_master
      +#TODO: properly wait for the master to reactivate semi-sync
      +--sleep 1
      +--connection master
      +insert into t1 values (11);
      +--sync_slave_with_master
       
       --connection slave
       --echo # Compare the tables on master and slave.
       --let $diff_tables= master:t1, slave:t1
       --source include/diff_tables.inc
       
      +SHOW STATUS LIKE 'Rpl_semi_sync_slave_send_ack'; # Expected: 2; Actual: 0
       --connection master
      -set statement sql_log_bin=0 for call mtr.add_suppression("Read semi-sync reply magic number error");
      +SHOW STATUS LIKE 'Rpl_semi_sync_master_%';
      +#       Expected Actual
      +# %clients     1 1
      +# %status     ON OFF
      +# %request_ack 2 1
      +# %get_ack     2 0
      +# %no_tx       2 3
      +# %yes_tx      1 0
      +
      +# Expected: Errors/warnings were found in logfiles...; Actual: nothing
       SET @save_debug_master= @@global.debug_dbug;
       SET GLOBAL debug_dbug="+d,semisync_corrupt_magic";
       insert into t1 values (11);
      

      Code Analysis

      If ACK replying fails, the IO thread turns off ACK replying, carries on, and never retries ACK replying again.

              if (repl_semisync_slave.slave_reply(mi))
              {
                /*
                  Master is not responding (gone away?) or it has turned semi sync
                  off. Turning off semi-sync responses as there is no point in sending
                  data to the master if the master not receiving the messages.
                  This also stops the logs from getting filled with
                  "Semi-sync slave net_flush() reply failed" messages.
                  On reconnect semi sync will be turned on again, if the
                  master has semi-sync enabled.
       
                  We check mi->abort_slave to see if the io thread was
                  killed and in this case we do not need an error message as
                  we know what is going on.
                 */
                if (!mi->abort_slave)
                  sql_print_error("Master server does not read semi-sync messages "
                                  "last_error: %s (%d). "
                                  "Fallback to asynchronous replication",
                                  mi->mysql->net.last_error,
                                  mi->mysql->net.last_errno);
                mi->semi_sync_reply_enabled= 0;
              }
      

      The only instance of mi->semi_sync_reply_enabled= 1 is in the connection establishment procedure, but unless the download also fails, there is no reconnect step after the above block.
      About the Repl_semi_sync_slave::slave_reply() failures causes described:

      • The master turned off semi-sync:
        This is not true: this function only sends, not receives.
        In fact, the master does not reply to ACK replies whether or not it is in semi-sync mode.
      • The master became unreachable:
        In this case, the IO thread should eventually fail to receive events anyway; turning off ACK replying only harms the recovery of intermittent network issues.

      The above block should be reverted to before its introduction by MDEV-32551, including the misleading error message.
      As for preventing error spam, perhaps explicitly try to reconnect when ACK replying fails, or abort the IO thread entirely if that fails.

      Attachments

        Issue Links

          Activity

            People

              ParadoxV5 Jimmy Hú
              ParadoxV5 Jimmy Hú
              Brandon Nesterenko Brandon Nesterenko
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - Not Specified
                  Not Specified
                  Logged:
                  Time Spent - 1.5h
                  1.5h

                  Git Integration

                    Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.