[MDEV-21322] report slave progress to the master Created: 2019-12-16  Updated: 2023-12-22

Status: Stalled
Project: MariaDB Server
Component/s: Replication
Fix Version/s: 11.5

Type: Task Priority: Critical
Reporter: Anel Husakovic Assignee: Brandon Nesterenko
Resolution: Unresolved Votes: 0
Labels: contribution, replication-monitoring, semisync

Issue Links:
Relates
relates to MDBF-573 Create the blog for a semi-sync repli... Closed
relates to MDEV-31556 rpl_semi_sync_master_[get,request]_ac... Open
relates to MDEV-31557 rpl_semi_sync_master_net_wait_num is ... Open
relates to MDEV-18475 extend show slave hosts Open

 Description   

When using the semi-sync protocol, track each replica's progress. The SHOW REPLICA HOSTS output should be extended with two fields: one representing the GTID state last sent to the replica, and the other representing the GTID state last ACK'd from the replica.

Additionally, treat rpl_semi_sync_master_timeout=0 as a special case where transactions do not await an ACK, but still report progress from the ack_thread on replica reply.

When running SHOW REPLICA HOSTS instead of 4 columns

SHOW REPLICA HOSTS;
Server_id	Host	Port	Master_id

we want to have additional columns

Server_id  Host       Port    Master_id   Gtid_State_Sent   Gtid_State_Ack
3          127.0.0.1  16002   1           0-1-5,1-2-10      0-1-4,1-2-7
4          127.0.0.1  16003   1           0-1-5,1-2-10      0-1-2,1-2-3

Additional notes:
1. Replicas with semi-sync disabled should have empty values for Gtid_State_Ack.
2. Updating a replica's Gtid_State_Ack should be done by the ack thread after validating that the transaction was sent to the replica
3. The new columns should always be present, regardless of rpl_semi_sync_master_enabled, as the primary can disable semi-sync dynamically, and it could still be useful to display the information.
4. Per design, when rpl_semi_sync_master_timeout=0, than rpl_semi_sync_master_status, if enabled earlier, should stay ON , instead of switching off to async and incrementing the counter updates.
5. Per desing, in order to distinguish replica type between async, semi-sync stalled and semi-sync active replica, suggestion is to add new column Replica_type with replica type values depending on the state of replica.



 Comments   
Comment by Andrei Elkin [ 2020-06-06 ]

The idea of 'reusing' in my understanding should be implemented as the following. The semisync mode slave acks on master event receive. The acks are handled by a special master server thread. The ack consists of file:pos and the slave id known too. So SHOW-SLAVE-HOSTS handler should be able to see those data. We'd like to add up the last acked GTID as well to the S-S-H reporting.

anel: What is meant in The semisync can optionally be switched to a "dummy" mode, where the master will no longer wait for slaves ? Semisync always is able to temporarily turning OFF, on master. And then master acts as if the formerly semisync slave is normal.

Comment by Jack [ 2020-08-15 ]

Thank you for your review

I'm the author of the patchset (https://github.com/MariaDB/server/pull/1427), sadly I have not seen the notification on jira

I'll update soon the code to show GTID as well

About the dummy mode:
Today, when a slave vanishes, the master have to wait rpl_semi_sync_master_timeout msec, during which all CRUD queries are blocked
When that timeout is over, the semi-sync is turned off and operation can continue
If the timeout is too low, the replication will die unnecessarily
If the timeout is too high, the death of a slave will have a large impact on the master

So, I introduced a dummy semi-sync mode, where the master never actually waits for a slave. In that mode, the master never waits for a slave

Comment by Jack [ 2020-08-23 ]

The GTID is now shown as well:
```
MariaDB [(none)]> show slave hosts;
--------------------------------------------------------------------------------------

Server_id Host Port Master_id File Position Gtid

--------------------------------------------------------------------------------------

12345 slave1 3306 10 log-bin.000039 4 0-10-246940,10-10-949282
1234 slave2 3306 10 log-bin.000039 4 0-10-246940,10-10-949282

---------------------------------------------------------------------------------
```

All the test are passing
I dropped the "dummy" mode, I'll make another PR later

Thanks

Comment by Andrei Elkin [ 2020-08-25 ]

To how to specify the new "dummy" mode, it was suggested (by danblack) on Github, to consider to add up 'NEVER' value to @@global.rpl_semi_sync_master_wait_point.

Comment by Andrei Elkin [ 2020-08-25 ]

NONE actually reads better (as it answers to the point "question" in the variable name).

Comment by Daniel Black [ 2020-08-26 ]

Yes that was the question, what actually sounds better. Glad you've decided.

Comment by Andrei Elkin [ 2020-09-03 ]

anel: Salute! Maybe reassigning it back to you makes sense, as we (engineering/replication) don't seem to be able to implement it any time soon. Hopefully by 10.6 GA-ing the idea contributor will come up with the patch fulfilling the requirements.

Comment by Anel Husakovic [ 2020-09-04 ]

Salute Elkin, ok, danblack and I will proceed further with this PR.

Comment by Brandon Nesterenko [ 2022-06-10 ]

Setting myself to review the existing patch

Comment by Anel Husakovic [ 2022-11-28 ]

I'm assigning to myself to finish this MDEV based on PR review.

Comment by Anel Husakovic [ 2023-06-22 ]

Hi bnestere , Elkin, should we use this MDEV (or another) to allow show replica hosts show everything in 1 screen, like how long in time or packets are replicas delayed and if possible other things, to allow users extended view expressing the necessary informations about the state of transactions ?

Comment by Andrei Elkin [ 2023-06-26 ]

anel, to whether to use this MDEV to cover up yet more slave state details (if I got the question correctly) I'd vote to do that under another ticket.
Clearly there's a lot to cover on the design level including which side (primary or replica) is responsible for displaying what, and if it's the replica
we'd also need to consider performance_schema as placeholder/interface as a candidate (not least 'cos it provides selectivity by fields).

Comment by Brandon Nesterenko [ 2023-08-16 ]

Review done via Slack

Comment by Kristian Nielsen [ 2023-08-17 ]

> Review done via Slack

Please don't use non-public channels for review, it makes it impossible for other to follow
Use preferably the developers@ mailing list for reviews, look at Sergei Golubchik's excellent reviews there for inspiration. Or use a channel on Zulip if you need an interactive review sessions.

- Kristian.

Comment by Anel Husakovic [ 2023-08-17 ]

Hi Kristian,
the most of the review is done on the PR itself.
Only thing what is left regarding the PR is that Brandon explained to me is how to use rr to debug timeouts for sync_with_master, at the end of the test, since there was single MSAN worker that failed (no log available now). I'm currently trying to repeat locally and understand why happens and fix, still in learning phase.

Comment by Brandon Nesterenko [ 2023-08-18 ]

Thanks anel for the explanation

And knielsen, to the general case, thanks for a great suggestion to use Zulip for interactive sessions! i had indeed used Slack (somewhat reluctantly) for them before.

Generated at Thu Feb 08 09:06:15 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.