React to SERVER_TOO_MUCH_LAG status

The new SERVER_TOO_MUCH_LAG status that's to be added in MXS-6571 needs to be detected by readwritesplit. If detected, the router should behave as if sync_transaction=soft is in effect but with a very low initial value of sync_transaction_timeout. While the SERVER_TOO_MUCH_LAG status is observed, the router should gradually increase the timeout for the transaction sync until the status bit disappears. The exact algorithm for the incrementing needs some research and experimentation but an initial version of linear increase of 5ms per minute should be a reasonable starting point.

The value of sync_transaction_timeout could act as the hard limit above which the timeouts would never increase. This guarantees a reasonable rate of progress without triggering timeouts. The current default of 10 seconds could be lowered to 1 second to make it so that a commit will at most take one second. This is more suitable for the sync_transaction=soft mode which is a feature that's more generally usable compared to the synchronous replication of the sync_transaction=hard mode.

Implementation

Added two new settings: sync_transaction_throttle_rate which controls the rate at which throttling is introduced and sync_transaction_probe which allows the lag probing to be disabled. The throttle rate defaults to 5ms per minute which means that at 60 minutes there's at most 300ms of delay added to commits and at 8 hours it'll be 2.4 seconds.

A new throttle state can be manually set on servers which, if set on the current primary, causes transaction throttling to start. This is a manual alternative to the queued switchover to be introduced in MXS-6571.

In order for transaction throttling to work with readwritesplit, the router must be configured as follows. sync_transaction must be set to soft in order for the throttling to work. sync_transaction_timeout should be set to the maximum delay that is acceptable for a client. Setting it to 1s means a client will see at most one second of delay for a COMMIT. sync_transaction_max_lag should be set to a highest value of replication lag that is still acceptable. Something like 1800s will make it so that only extreme lag will trigger unconditional transaction synchronization and seems like a good starting point for a general throttling mechanism. sync_transaction_probe should be set to false to avoid delaying any transactions while throttling is not going on. It can be left on if transaction latency probing is useful but for the pure throttling use-case it's not that useful.