Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-13915

STOP SLAVE takes very long time on a busy system

Details

    Description

      150     system user             NULL    Slave_IO        66361   Waiting for master to send event        NULL    0.000
      152     system user             NULL    Slave_worker    0       Write_rows_log_event::write_row(-1)     NULL    0.000
      153     system user             NULL    Slave_worker    0       Write_rows_log_event::write_row(-1)     NULL    0.000
      154     system user             NULL    Slave_worker    0       Write_rows_log_event::write_row(-1)     NULL    0.000
      155     system user             NULL    Slave_worker    0       Write_rows_log_event::write_row(-1)     NULL    0.000
      156     system user             NULL    Slave_worker    0       Write_rows_log_event::write_row(-1)     NULL    0.000
      157     system user             NULL    Slave_worker    0       Write_rows_log_event::write_row(-1)     NULL    0.000
      158     system user             NULL    Slave_worker    0       Write_rows_log_event::write_row(-1)     NULL    0.000
      159     system user             NULL    Slave_worker    0       Write_rows_log_event::write_row(-1)     NULL    0.000
      160     system user             NULL    Slave_worker    0       Write_rows_log_event::write_row(-1)     NULL    0.000
      162     system user             NULL    Slave_worker    0       Write_rows_log_event::write_row(-1)     NULL    0.000
      161     system user             NULL    Slave_worker    0       Write_rows_log_event::write_row(-1)     NULL    0.000
      163     system user             NULL    Slave_worker    0       Write_rows_log_event::write_row(-1)     NULL    0.000
      151     system user             NULL    Slave_SQL       1090    Reading event from the relay log        NULL    0.000
      2415    root    127.0.0.1:42392 NULL    Query   835     Killing slave   STOP SLAVE      0.000
      

      slave machine configuration:

      sync_master_info          = 500000
      sync_relay_log            = 100000
      sync_relay_log_info       = 500000
      slave_parallel_max_queued = 67108864
      slave_parallel_mode       = optimistic
      slave_parallel_threads    = 12
      

      Is that normal?

      Attachments

        Issue Links

          Activity

            Something that is missing from the discussion here is that the main reason STOP SLAVE is slow in parallel replication is not because it doesn't roll back running transactions. The main problem is that in many cases parallel replication will replicate all queued events (@@slave_parallel_max_queued).

            I think this is a left-over of when only conservative mode existed. The current STOP SLAVE mechanism is seen in do_gco_wait(), it continues until the current GCO is completed (wait_count > entry->stop_count). But in optimistic mode, the GCO can be very large, potentially all queued events, thus stop is delayed longer than needed.

            I think a much simpler solution is to fix this, so that stop_count is initialised to largest_started_sub_id, and compared against rgi->gtid_sub_id.

            This will not rollback an existing long-running transaction, but I think that's actually good. Forcing stop immediately will cause massive rollback when many threads are configured (Jean Francois Gagné tested using > 1000 threads), which seems undesirable. And forcing stop does not guarantee fast stop anyway, a long-running statement will not be aborted.

            knielsen Kristian Nielsen added a comment - Something that is missing from the discussion here is that the main reason STOP SLAVE is slow in parallel replication is not because it doesn't roll back running transactions. The main problem is that in many cases parallel replication will replicate all queued events (@@slave_parallel_max_queued). I think this is a left-over of when only conservative mode existed. The current STOP SLAVE mechanism is seen in do_gco_wait(), it continues until the current GCO is completed (wait_count > entry->stop_count). But in optimistic mode, the GCO can be very large, potentially all queued events, thus stop is delayed longer than needed. I think a much simpler solution is to fix this, so that stop_count is initialised to largest_started_sub_id, and compared against rgi->gtid_sub_id. This will not rollback an existing long-running transaction, but I think that's actually good. Forcing stop immediately will cause massive rollback when many threads are configured (Jean Francois Gagné tested using > 1000 threads), which seems undesirable. And forcing stop does not guarantee fast stop anyway, a long-running statement will not be aborted.

            I implemented a much simpler fix for this in the branch knielsen_faster_stop_slave:

            https://github.com/MariaDB/server/commits/knielsen_faster_stop_slave

            knielsen Kristian Nielsen added a comment - I implemented a much simpler fix for this in the branch knielsen_faster_stop_slave: https://github.com/MariaDB/server/commits/knielsen_faster_stop_slave
            knielsen Kristian Nielsen added a comment - And a better fix for the MDEV-31448 fix included in Brandon's patch for this issue: https://github.com/MariaDB/server/commits/knielsen_faster_stop_slave https://github.com/MariaDB/server/commit/6ce9c839997c1fc78c2989540dd21155a96fb419
            knielsen Kristian Nielsen added a comment - - edited

            I now pushed another patch to the branch knielsen_faster_stop_slave:

            https://github.com/MariaDB/server/commits/knielsen_faster_stop_slave
            https://github.com/MariaDB/server/commit/c20ce0dff404890df59f1fd305bd85be0ada86f8

            This implements a STOP SLAVE FORCE option, which can be used to optionally force a quick slave stop by rolling back all active transactions on the next event or row operation, if no non-transactional event groups are blocking it. I included the test cases from Brandon's pull request, they are passing with this patch (replacing STOP SLAVE with STOP SLAVE FORCE).

            I think this is the way to go if we want this rollback functionality, leaving the option to not roll back a lot of work needlessly. I also think this is a new feature and should only go to development version (11.1?). Comments welcome, of course.

            The simple fix earlier on my branch (without forcing rollback) should be sufficient to solve the user/customers problem. I think that can go in 10.5 if we want (or even 10.4, it should be safe). Some of the other bug fixes on the branch related to stop may also be appropriate for some earlier branch, suggestions wellcome.

            knielsen Kristian Nielsen added a comment - - edited I now pushed another patch to the branch knielsen_faster_stop_slave: https://github.com/MariaDB/server/commits/knielsen_faster_stop_slave https://github.com/MariaDB/server/commit/c20ce0dff404890df59f1fd305bd85be0ada86f8 This implements a STOP SLAVE FORCE option, which can be used to optionally force a quick slave stop by rolling back all active transactions on the next event or row operation, if no non-transactional event groups are blocking it. I included the test cases from Brandon's pull request, they are passing with this patch (replacing STOP SLAVE with STOP SLAVE FORCE). I think this is the way to go if we want this rollback functionality, leaving the option to not roll back a lot of work needlessly. I also think this is a new feature and should only go to development version (11.1?). Comments welcome, of course. The simple fix earlier on my branch (without forcing rollback) should be sufficient to solve the user/customers problem. I think that can go in 10.5 if we want (or even 10.4, it should be safe). Some of the other bug fixes on the branch related to stop may also be appropriate for some earlier branch, suggestions wellcome.

            I have pushed the fix for slow STOP SLAVE to 10.4 (The STOP SLAVE FORCE feature can go to a development branch later if we wish).
            So I suggest to close this bug?

            - Kristian.

            knielsen Kristian Nielsen added a comment - I have pushed the fix for slow STOP SLAVE to 10.4 (The STOP SLAVE FORCE feature can go to a development branch later if we wish). So I suggest to close this bug? - Kristian.

            People

              bnestere Brandon Nesterenko
              mxu Michael Xu
              Votes:
              3 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.