Details
-
Task
-
Status: Open (View Workflow)
-
Major
-
Resolution: Unresolved
-
None
-
None
Description
The current parallel paradigm for replication ties transactions to slave workers for their entire life-cycle. That is, there are parallel-worker-threads created and able to work on transactions, and once a task is assigned to a parallel worker thread, that thread is restricted to only work on that transaction until it is completed. This can be a large bottleneck due to the restriction that the commit-order of transactions on the master must be preserved on the slave because transactions can be ready-to-commit before their prior-master-committed transactions. This is because the slave threads will go idle while waiting for their prior transactions to commit. It is particularly problematic for mixed size workloads (i.e. varying from small to large transactions) and transactions with overlapping data binlogged close together when running with slave-parallel-mode=optimistic.
A few ideas have come about to solve this:
1. Change the parallel execution method to be task-based, so instead of having a thread idle, it can simply switch to pick up the next available task to continue execution on some other transaction.
2. Create a thread-pool for parallel replication that allows for a maximum of slave-parallel-threads to be doing meaningful work. That is, when an existing thread would go idle while waiting for a prior transaction to commit, it would allow a new thread to pick up a transaction and begin executing it.
Attachments
Issue Links
- is part of
-
MDEV-30458 Consolidate Serial Replica to Parallel Replica with 1 Worker Thread
-
- Open
-
-
MDEV-37582 Reduce Slave Lag
-
- Open
-
- relates to
-
MDEV-16404 Balanced replication parallel applier
-
- Stalled
-
-
MDEV-37583 Streaming Replication
-
- Open
-