Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-37587

Increase Parallel Replication Concurrency

    XMLWordPrintable

Details

    • Task
    • Status: Open (View Workflow)
    • Major
    • Resolution: Unresolved
    • None
    • Replication
    • None

    Description

      The current parallel paradigm for replication ties transactions to slave workers for their entire life-cycle. That is, there are parallel-worker-threads created and able to work on transactions, and once a task is assigned to a parallel worker thread, that thread is restricted to only work on that transaction until it is completed. This can be a large bottleneck due to the restriction that the commit-order of transactions on the master must be preserved on the slave because transactions can be ready-to-commit before their prior-master-committed transactions. This is because the slave threads will go idle while waiting for their prior transactions to commit. It is particularly problematic for mixed size workloads (i.e. varying from small to large transactions) and transactions with overlapping data binlogged close together when running with slave-parallel-mode=optimistic.

      A few ideas have come about to solve this:
      1. Change the parallel execution method to be task-based, so instead of having a thread idle, it can simply switch to pick up the next available task to continue execution on some other transaction.
      2. Create a thread-pool for parallel replication that allows for a maximum of slave-parallel-threads to be doing meaningful work. That is, when an existing thread would go idle while waiting for a prior transaction to commit, it would allow a new thread to pick up a transaction and begin executing it.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              bnestere Brandon Nesterenko
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.