Uploaded image for project: 'MariaDB MaxScale'
  1. MariaDB MaxScale
  2. MXS-6572

Throttle writes to a server until replication catches up

    XMLWordPrintable

Details

    • New Feature
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Fixed
    • None
    • 26.10.0
    • readwritesplit
    • None
    • MXS-SPRINT-278, MXS-SPRINT-279
    • Hide
      Writes to servers in MaxScale can now be throttled with the `maxctrl set server
      <name> throttle` command. If a readwritesplit service uses a server that's being
      throttled, it will add a linearly increasing delay to each COMMIT until
      replication lag catches up after which transactions proceed as fast as the
      replicas can process.
      Show
      Writes to servers in MaxScale can now be throttled with the `maxctrl set server <name> throttle` command. If a readwritesplit service uses a server that's being throttled, it will add a linearly increasing delay to each COMMIT until replication lag catches up after which transactions proceed as fast as the replicas can process.

    Description

      The new SERVER_TOO_MUCH_LAG status that's to be added in MXS-6571 needs to be detected by readwritesplit. If detected, the router should behave as if sync_transaction=soft is in effect but with a very low initial value of sync_transaction_timeout. While the SERVER_TOO_MUCH_LAG status is observed, the router should gradually increase the timeout for the transaction sync until the status bit disappears. The exact algorithm for the incrementing needs some research and experimentation but an initial version of linear increase of 5ms per minute should be a reasonable starting point.

      The value of sync_transaction_timeout could act as the hard limit above which the timeouts would never increase. This guarantees a reasonable rate of progress without triggering timeouts. The current default of 10 seconds could be lowered to 1 second to make it so that a commit will at most take one second. This is more suitable for the sync_transaction=soft mode which is a feature that's more generally usable compared to the synchronous replication of the sync_transaction=hard mode.

      Implementation

      Added two new settings: sync_transaction_throttle_rate which controls the rate at which throttling is introduced and sync_transaction_probe which allows the lag probing to be disabled. The throttle rate defaults to 5ms per minute which means that at 60 minutes there's at most 300ms of delay added to commits and at 8 hours it'll be 2.4 seconds.

      A new throttle state can be manually set on servers which, if set on the current primary, causes transaction throttling to start. This is a manual alternative to the queued switchover to be introduced in MXS-6571.

      In order for transaction throttling to work with readwritesplit, the router must be configured as follows. sync_transaction must be set to soft in order for the throttling to work. sync_transaction_timeout should be set to the maximum delay that is acceptable for a client. Setting it to 1s means a client will see at most one second of delay for a COMMIT. sync_transaction_max_lag should be set to a highest value of replication lag that is still acceptable. Something like 1800s will make it so that only extreme lag will trigger unconditional transaction synchronization and seems like a good starting point for a general throttling mechanism. sync_transaction_probe should be set to false to avoid delaying any transactions while throttling is not going on. It can be left on if transaction latency probing is useful but for the pure throttling use-case it's not that useful.

      Attachments

        Issue Links

          Activity

            People

              markus makela markus makela
              markus makela markus makela
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - 5d
                  5d
                  Remaining:
                  Remaining Estimate - 0d
                  0d
                  Logged:
                  Time Spent - 1d Time Not Required
                  1d

                  Git Integration

                    Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.