Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-21117

refine the server binlog-based recovery for semisync

Details

    Description

      When run after master server crash --tc-heuristic-recover=rollback produces inconsistent server state with binlog still containing transactions that were rolled back by the option.
      Such way recovered server may not be used for replication. E.g when such way recovered
      ex-master is demoted into slave its binlog state needs further correction to subtract
      the rolled back transactions from its binlog status. Otherwise the "new" slave might claim
      those transactions as locally present at the (gtid) master-slave connection protocol. At the same time the actual "new" master may never have seen those transactions (because they never arrived at it when it was formerly slave, due to the crash).

      This issue should be fixed with refining the recovery procedure with truncating binlog to cut off the prepared rolled back transactions. The method is also known as pioneered by FB
      https://percona.community/blog/2018/08/23/question-about-semi-synchronous-replication-answer-with-all-the-details/

      Once a transaction reaches the binary logs it should roll forward.

      Attachments

        Issue Links

          Activity

            Elkin Andrei Elkin added a comment -

            Documentation is updated with the following diff.

            Elkin Andrei Elkin added a comment - Documentation is updated with the following diff .

            Currently we have a problems with the binary logs:

            • We truncate transactions from the binary log that was already fully written there (and thus may be on the slave)
            • When we truncate a half-written-transaction we don't take into account that a slave may have already got part of it and will reuse the GTID id if reconnecting to a master that died in the middle of writing the transaction to the binary log.

            The fix should be:

            • All fully written transactions in the binary log should roll forwards. The half or not written one should do rollback.
            • In case of the master-slave connection closing before the slave gets a full transaction, it should reconnect to the master and ask for the 'transaction after the last fully received GTID.
              Alternatively it could use the last binlog file + position (which should work fine as the master binlog was truncated).
            monty Michael Widenius added a comment - Currently we have a problems with the binary logs: We truncate transactions from the binary log that was already fully written there (and thus may be on the slave) When we truncate a half-written-transaction we don't take into account that a slave may have already got part of it and will reuse the GTID id if reconnecting to a master that died in the middle of writing the transaction to the binary log. The fix should be: All fully written transactions in the binary log should roll forwards. The half or not written one should do rollback. In case of the master-slave connection closing before the slave gets a full transaction, it should reconnect to the master and ask for the 'transaction after the last fully received GTID. Alternatively it could use the last binlog file + position (which should work fine as the master binlog was truncated).
            Elkin Andrei Elkin added a comment -

            monty, to the fully written ones, when a server restarts having both rpl_semi_sync_MASTER,SLAVE_enabled, the semisync slave recovery does not check the first of the two. Hence a server intended to be a master, that is having rpl_semi_sync_MASTER_enabled = true rolls back transactions in doubt unnecessarily and possibly harmfully.

            That's an issue. It could be trivially fixed with adding the check of rpl_semi_sync_MASTER_enabled and when it's true it would override.

            The half-written is an issue only because the slave is reconnecting to a master that passed through the semisync recovery with the intent to become slave (such master may discard from its binlog a transaction being received by slave.
            It will disappear when the new rpl_semi_sync_MASTER_enabled-based decision will be implemented.

            Elkin Andrei Elkin added a comment - monty , to the fully written ones, when a server restarts having both rpl_semi_sync_MASTER,SLAVE_enabled , the semisync slave recovery does not check the first of the two. Hence a server intended to be a master, that is having rpl_semi_sync_MASTER_enabled = true rolls back transactions in doubt unnecessarily and possibly harmfully. That's an issue. It could be trivially fixed with adding the check of rpl_semi_sync_MASTER_enabled and when it's true it would override. The half-written is an issue only because the slave is reconnecting to a master that passed through the semisync recovery with the intent to become slave (such master may discard from its binlog a transaction being received by slave. It will disappear when the new rpl_semi_sync_MASTER_enabled -based decision will be implemented.
            Elkin Andrei Elkin added a comment -

            monty, I filed MDEV-33424 for this plan which is about 1 line of code.

            Elkin Andrei Elkin added a comment - monty , I filed MDEV-33424 for this plan which is about 1 line of code.
            monty Michael Widenius added a comment - - edited

            MDEV-33424 is not acceptable as it does not solve the issue at hand. Please fix this issue the way I have described!
            To make things clear, this is how to fix this issue:

            • rpl_semi_sync_slave_enable should not have anything to do with recovery.
            • We should only delete things from the binary log that was found to be half written during recovery. This could be a new option that could default to always one.
              I personally don't think we need an option for this as this should always be safe to do (and should save disk space)
            • We should that if a slave re-connects it should continue from the next transaction after the last full transaction it has read.
              In other words. if the slave reads GTID-1 and master crashes while reading GTID-2 then the slave should ask for 'next transaction after GTD-1'
              Alternatively it can ask the transaction based on the last binlog position it was using.
            monty Michael Widenius added a comment - - edited MDEV-33424 is not acceptable as it does not solve the issue at hand. Please fix this issue the way I have described! To make things clear, this is how to fix this issue: rpl_semi_sync_slave_enable should not have anything to do with recovery. We should only delete things from the binary log that was found to be half written during recovery. This could be a new option that could default to always one. I personally don't think we need an option for this as this should always be safe to do (and should save disk space) We should that if a slave re-connects it should continue from the next transaction after the last full transaction it has read. In other words. if the slave reads GTID-1 and master crashes while reading GTID-2 then the slave should ask for 'next transaction after GTD-1' Alternatively it can ask the transaction based on the last binlog position it was using.

            People

              bnestere Brandon Nesterenko
              Elkin Andrei Elkin
              Votes:
              3 Vote for this issue
              Watchers:
              24 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.