Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-5289

master server starts slave parallel threads

Details

    Description

      When you run this command in a slave
      set global slave_parallel_threads=10;
      you get a rightful error

      ERROR 1198 (HY000): This operation cannot be performed as you have a running slave ''; run STOP SLAVE '' first

      However, if you run the same command in a master, the statement is accepted, and the thread started.

      master [localhost] {msandbox} ((none)) > select ID, DB, state, time_ms, memory_used from information_schema .PROCESSLIST where USER='system user';
      Empty set (0.01 sec)
       
      master [localhost] {msandbox} ((none)) > set global slave_parallel_threads=10;
      Query OK, 0 rows affected (0.00 sec)
       
      master [localhost] {msandbox} ((none)) > select ID, DB, state, time_ms, memory_used from information_schema .PROCESSLIST where USER='system user';
      +----+------+----------------------------------+----------+-------------+
      | ID | DB   | state                            | time_ms  | memory_used |
      +----+------+----------------------------------+----------+-------------+
      | 47 | NULL | Waiting for work from SQL thread | 2207.683 |       34704 |
      | 46 | NULL | Waiting for work from SQL thread | 2207.686 |       34704 |
      | 45 | NULL | Waiting for work from SQL thread | 2207.722 |       34704 |
      | 44 | NULL | Waiting for work from SQL thread | 2207.723 |       34704 |
      | 43 | NULL | Waiting for work from SQL thread | 2207.754 |       34704 |
      | 42 | NULL | Waiting for work from SQL thread | 2207.757 |       34704 |
      | 41 | NULL | Waiting for work from SQL thread | 2207.765 |       34704 |
      | 40 | NULL | Waiting for work from SQL thread | 2207.801 |       34704 |
      | 39 | NULL | Waiting for work from SQL thread | 2207.843 |       34704 |
      | 38 | NULL | Waiting for work from SQL thread | 2207.851 |       34704 |
      +----+------+----------------------------------+----------+-------------+
      10 rows in set (0.01 sec)

      Attachments

        Activity

          datacharmer Giuseppe Maxia created issue -
          elenst Elena Stepanova made changes -
          Field Original Value New Value
          Description When you run this command in a slave
          set global slave_parallel_threads=10;
          you get a rightful error
          ERROR 1198 (HY000): This operation cannot be performed as you have a running slave ''; run STOP SLAVE '' first

          However, if you run the same command in a master, the statement is accepted, and the thread started.

          master [localhost] {msandbox} ((none)) > select ID, DB, state, time_ms, memory_used from information_schema .PROCESSLIST where USER='system user';
          Empty set (0.01 sec)

          master [localhost] {msandbox} ((none)) > set global slave_parallel_threads=10;
          Query OK, 0 rows affected (0.00 sec)

          master [localhost] {msandbox} ((none)) > select ID, DB, state, time_ms, memory_used from information_schema .PROCESSLIST where USER='system user';
          +----+------+----------------------------------+----------+-------------+
          | ID | DB | state | time_ms | memory_used |
          +----+------+----------------------------------+----------+-------------+
          | 47 | NULL | Waiting for work from SQL thread | 2207.683 | 34704 |
          | 46 | NULL | Waiting for work from SQL thread | 2207.686 | 34704 |
          | 45 | NULL | Waiting for work from SQL thread | 2207.722 | 34704 |
          | 44 | NULL | Waiting for work from SQL thread | 2207.723 | 34704 |
          | 43 | NULL | Waiting for work from SQL thread | 2207.754 | 34704 |
          | 42 | NULL | Waiting for work from SQL thread | 2207.757 | 34704 |
          | 41 | NULL | Waiting for work from SQL thread | 2207.765 | 34704 |
          | 40 | NULL | Waiting for work from SQL thread | 2207.801 | 34704 |
          | 39 | NULL | Waiting for work from SQL thread | 2207.843 | 34704 |
          | 38 | NULL | Waiting for work from SQL thread | 2207.851 | 34704 |
          +----+------+----------------------------------+----------+-------------+
          10 rows in set (0.01 sec)



          When you run this command in a slave
          set global slave_parallel_threads=10;
          you get a rightful error
          {code:sql}
          ERROR 1198 (HY000): This operation cannot be performed as you have a running slave ''; run STOP SLAVE '' first
          {code}

          However, if you run the same command in a master, the statement is accepted, and the thread started.

          {code:sql}
          master [localhost] {msandbox} ((none)) > select ID, DB, state, time_ms, memory_used from information_schema .PROCESSLIST where USER='system user';
          Empty set (0.01 sec)

          master [localhost] {msandbox} ((none)) > set global slave_parallel_threads=10;
          Query OK, 0 rows affected (0.00 sec)

          master [localhost] {msandbox} ((none)) > select ID, DB, state, time_ms, memory_used from information_schema .PROCESSLIST where USER='system user';
          +----+------+----------------------------------+----------+-------------+
          | ID | DB | state | time_ms | memory_used |
          +----+------+----------------------------------+----------+-------------+
          | 47 | NULL | Waiting for work from SQL thread | 2207.683 | 34704 |
          | 46 | NULL | Waiting for work from SQL thread | 2207.686 | 34704 |
          | 45 | NULL | Waiting for work from SQL thread | 2207.722 | 34704 |
          | 44 | NULL | Waiting for work from SQL thread | 2207.723 | 34704 |
          | 43 | NULL | Waiting for work from SQL thread | 2207.754 | 34704 |
          | 42 | NULL | Waiting for work from SQL thread | 2207.757 | 34704 |
          | 41 | NULL | Waiting for work from SQL thread | 2207.765 | 34704 |
          | 40 | NULL | Waiting for work from SQL thread | 2207.801 | 34704 |
          | 39 | NULL | Waiting for work from SQL thread | 2207.843 | 34704 |
          | 38 | NULL | Waiting for work from SQL thread | 2207.851 | 34704 |
          +----+------+----------------------------------+----------+-------------+
          10 rows in set (0.01 sec)
          {code}

          Is it wrong? There is no slave running on the master (I assume there is none, right?), so why would it throw ER_SLAVE_MUST_STOP?
          And we can't forbid setting the option before configuring server as a slave, because then it will be impossible to put the value in the config file.

          Maybe I misunderstand the nature of the problem, could you please elaborate on it?

          elenst Elena Stepanova added a comment - Is it wrong? There is no slave running on the master (I assume there is none, right?), so why would it throw ER_SLAVE_MUST_STOP? And we can't forbid setting the option before configuring server as a slave, because then it will be impossible to put the value in the config file. Maybe I misunderstand the nature of the problem, could you please elaborate on it?

          It is wrong, because slave threads are started in a server that is NOT configured as a slave. I have no idea what these threads do in the master, but surely I would not want additional threads running in a production server.

          The right behavior should be do nothing. Then if I run 'start slave', I should get an error that the server is not configured as a slave, and all is well.
          This is what MySQL 5.6 does for its corresponding variable, and it is the right thing, because this kind of protection allows you to configure all your servers in the same way, but only the active slaves get the additional threads.

          datacharmer Giuseppe Maxia added a comment - It is wrong, because slave threads are started in a server that is NOT configured as a slave. I have no idea what these threads do in the master, but surely I would not want additional threads running in a production server. The right behavior should be do nothing . Then if I run 'start slave', I should get an error that the server is not configured as a slave, and all is well. This is what MySQL 5.6 does for its corresponding variable, and it is the right thing, because this kind of protection allows you to configure all your servers in the same way, but only the active slaves get the additional threads.

          I see, yes, it makes sense that the value should be configurable, but the actual thread start should only happen when the server starts serving as a slave.

          elenst Elena Stepanova added a comment - I see, yes, it makes sense that the value should be configurable, but the actual thread start should only happen when the server starts serving as a slave.
          elenst Elena Stepanova made changes -
          Fix Version/s 10.0.7 [ 14100 ]
          Assignee Kristian Nielsen [ knielsen ]

          I do not think this is worth it, it just causes needless complexity.

          Those are just background threads. They are waiting for work, if no work
          arrives (because no SQL threads are running), then they will just sit idle.

          It is just like the InnoDB background threads. If you enable InnoDB you get
          all the InnoDB threads started, even if you do not ever create any InnoDB
          tables; they just will not be doing anything.

          knielsen Kristian Nielsen added a comment - I do not think this is worth it, it just causes needless complexity. Those are just background threads. They are waiting for work, if no work arrives (because no SQL threads are running), then they will just sit idle. It is just like the InnoDB background threads. If you enable InnoDB you get all the InnoDB threads started, even if you do not ever create any InnoDB tables; they just will not be doing anything.
          knielsen Kristian Nielsen made changes -
          Resolution Won't Fix [ 2 ]
          Status Open [ 1 ] Closed [ 6 ]

          BTW, the MySQL 5.6 variable is quite different if I understand correctly.

          In MariaDB, the threads just form a thread pool, shared among any configured slaves. They do not do anything except sit ready, waiting for work from slave SQL threads.

          In the future, one could imagine a more advanced thread pool management, eg. that
          could dynamically spawn and shutdown threads depending on the current needs and
          load on the replication. But in this first version, a simple fixed-size pool is what we
          have.

          knielsen Kristian Nielsen added a comment - BTW, the MySQL 5.6 variable is quite different if I understand correctly. In MariaDB, the threads just form a thread pool, shared among any configured slaves. They do not do anything except sit ready, waiting for work from slave SQL threads. In the future, one could imagine a more advanced thread pool management, eg. that could dynamically spawn and shutdown threads depending on the current needs and load on the replication. But in this first version, a simple fixed-size pool is what we have.

          I agree that MySQL 5.6 behavior is quite different.
          What I wanted to stress is that the server should not start any task related to slave role unless it is configured as such.
          This, is also how Tungsten Replicator works. You can configure a service to run parallel threads, but the threads are only enabled if the service role changes to 'slave'. When a slave is promoted to master, it shelves its additional threads and resumes working in a single thread.
          I would expect the same thing from any server. Tasks that are only valid for a slave should not be enabled by default when the service is running as master.

          datacharmer Giuseppe Maxia added a comment - I agree that MySQL 5.6 behavior is quite different. What I wanted to stress is that the server should not start any task related to slave role unless it is configured as such. This, is also how Tungsten Replicator works. You can configure a service to run parallel threads, but the threads are only enabled if the service role changes to 'slave'. When a slave is promoted to master, it shelves its additional threads and resumes working in a single thread. I would expect the same thing from any server. Tasks that are only valid for a slave should not be enabled by default when the service is running as master.

          Right, so the pool of threads could magically spring into life upon the first CHANGE MASTER statement, and be teared down when the last slave is removed by RESET SLAVE ALL.

          But I do not have bandwidth to work on that kind of stuff...

          knielsen Kristian Nielsen added a comment - Right, so the pool of threads could magically spring into life upon the first CHANGE MASTER statement, and be teared down when the last slave is removed by RESET SLAVE ALL. But I do not have bandwidth to work on that kind of stuff...
          knielsen Kristian Nielsen added a comment - - edited

          I changed my mind a bit, would be nice to not have the threads started unnecessarily, even if they are not harmful as such. So re-opening.

          knielsen Kristian Nielsen added a comment - - edited I changed my mind a bit, would be nice to not have the threads started unnecessarily, even if they are not harmful as such. So re-opening.
          knielsen Kristian Nielsen made changes -
          Resolution Won't Fix [ 2 ]
          Status Closed [ 6 ] Reopened [ 4 ]
          knielsen Kristian Nielsen made changes -
          Priority Critical [ 2 ] Minor [ 4 ]
          knielsen Kristian Nielsen made changes -
          Fix Version/s 10.0.8 [ 14200 ]
          Fix Version/s 10.0.7 [ 14100 ]
          serg Sergei Golubchik made changes -
          Fix Version/s 10.0.9 [ 14400 ]
          Fix Version/s 10.0.8 [ 14200 ]
          serg Sergei Golubchik made changes -
          Fix Version/s 10.0.10 [ 14500 ]
          Fix Version/s 10.0.9 [ 14400 ]
          serg Sergei Golubchik made changes -
          Fix Version/s 10.0.11 [ 15200 ]
          Fix Version/s 10.0.10 [ 14500 ]
          serg Sergei Golubchik made changes -
          Fix Version/s 10.0.12 [ 15201 ]
          Fix Version/s 10.0.11 [ 15200 ]

          Idea for how to implement this in a way that works with the current code without introducing extra locking overhead:

          1. Do not start the worker threads in the pool at server start.

          2. In start_slave_threads(), first startup the worker threads, if we did not already.

          3. At the end of handle_slave_sql(), if we are shutting down the last SQL driver thread, shut down the pool of worker threads also.

          And do the similar thing for the slave background thread also. This has to be started briefly during server startup (to load mysql.gtid_slave_pos), but we can at that point stop it again, and then start and stop it along with the worker threads in the pool.

          knielsen Kristian Nielsen added a comment - Idea for how to implement this in a way that works with the current code without introducing extra locking overhead: 1. Do not start the worker threads in the pool at server start. 2. In start_slave_threads(), first startup the worker threads, if we did not already. 3. At the end of handle_slave_sql(), if we are shutting down the last SQL driver thread, shut down the pool of worker threads also. And do the similar thing for the slave background thread also. This has to be started briefly during server startup (to load mysql.gtid_slave_pos), but we can at that point stop it again, and then start and stop it along with the worker threads in the pool.
          serg Sergei Golubchik made changes -
          Workflow defaullt [ 29717 ] MariaDB v2 [ 43620 ]
          Status Reopened [ 4 ] Stalled [ 10000 ]
          serg Sergei Golubchik made changes -
          Fix Version/s 10.0.13 [ 16000 ]
          Fix Version/s 10.0.12 [ 15201 ]
          knielsen Kristian Nielsen made changes -
          Priority Minor [ 4 ] Major [ 3 ]
          knielsen Kristian Nielsen made changes -
          Fix Version/s 10.0 [ 16000 ]
          serg Sergei Golubchik made changes -
          Fix Version/s 10.0 [ 16000 ]
          knielsen Kristian Nielsen made changes -
          Status Stalled [ 10000 ] In Progress [ 3 ]

          Pushed to 10.0.18:

          http://lists.askmonty.org/pipermail/commits/2015-March/007562.html

          With this patch, parallel replication worker threads are not spawned until
          needed (when an SQL thread is started), and they will be de-spawned if all SQL
          threads are stopped.

          Thanks, Giuseppe, for reporting this!

          knielsen Kristian Nielsen added a comment - Pushed to 10.0.18: http://lists.askmonty.org/pipermail/commits/2015-March/007562.html With this patch, parallel replication worker threads are not spawned until needed (when an SQL thread is started), and they will be de-spawned if all SQL threads are stopped. Thanks, Giuseppe, for reporting this!
          knielsen Kristian Nielsen made changes -
          Component/s Replication [ 10100 ]
          Fix Version/s 10.0.18 [ 18702 ]
          Fix Version/s 10.0 [ 16000 ]
          Resolution Fixed [ 1 ]
          Status In Progress [ 3 ] Closed [ 6 ]
          ratzpo Rasmus Johansson (Inactive) made changes -
          Workflow MariaDB v2 [ 43620 ] MariaDB v3 [ 66779 ]
          serg Sergei Golubchik made changes -
          Workflow MariaDB v3 [ 66779 ] MariaDB v4 [ 147229 ]

          People

            knielsen Kristian Nielsen
            datacharmer Giuseppe Maxia
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.