Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-10297

Threadpool : add priorization features

Details

    • Task
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Fixed
    • 10.2.2
    • OTHER
    • None
    • 10.2.2-1, 10.2.2-2, 10.2.2-3

    Description

      There is feature that is there in Percona tree 5.5+ and also in MySQL/Enterprise
      but not in MariaDB is priorization

      • By default, connection that is in transaction gets a priority boost (which translated to
        in transaction= low prio, outside of transaction => high prio). The first statement after "begin" does not need to be high prio (this transaction does not actually use any resources)
      • We need to have a variable that will allow to specify per-connection or global scheduling behavior (auto = based on transaction status, high - always high prio, low- always low prio)
        This way, it would be easy to return back to FIFO scheduling if someone prefers that (i.e set global threadpool_priority=high). Or for a connection to be "nice" (setting priority = low regardless of transaction status), if it does not unimportant work . This variable is akin to Percona's threadpool_high_prio_mode=[transactions|statements|none] , but with an understandable name.
      • On Unix ,we'll use separate queues for high priority and low priority items. high priority queue is checked before low prio when an event is dequeued.
      • On Windows threadpool, it is possible to 'yield' low prio work items.
      • On Unix, to avoid starvation of the low prio items, periodically the old items in low prio queue need to be merged at the end of high prio queue. We can use 'thread_pool_kickup_timer' logic, like MySQL Enterprise is doing it.
      • We won't be using 'throttling of low prio queue' like Percona, as restricting the number of active+waiting threads can easily lead to a deadlock (single global lock or table lock would suffice to check that)

      Attachments

        Activity

          wlad Vladislav Vaintroub created issue -
          wlad Vladislav Vaintroub made changes -
          Field Original Value New Value
          Assignee Vladislav Vaintroub [ wlad ]
          ratzpo Rasmus Johansson (Inactive) made changes -
          Sprint 10.2.2-1 [ 79 ]
          ratzpo Rasmus Johansson (Inactive) made changes -
          Rank Ranked higher
          ratzpo Rasmus Johansson (Inactive) made changes -
          Sprint 10.2.2-1 [ 79 ] 10.2.2-1, 10.2.2-2 [ 79, 82 ]
          ratzpo Rasmus Johansson (Inactive) made changes -
          Sprint 10.2.2-1, 10.2.2-2 [ 79, 82 ] 10.2.2-1, 10.2.2-2, 10.2.2-3 [ 79, 82, 83 ]
          wlad Vladislav Vaintroub made changes -
          Description Main feature that is there in Percona tree 5.5+ and later an not in hours is
          1. Priorization mode (in-transaction queries can be prioritized higher, nontransaction can be prioritized higher).
          https://www.percona.com/blog/2014/01/29/percona-server-thread-pool-improvements
          has benchmark details.
          2. Some smaller stuff to play well with performance schema (idle waits for network), instrumentation for threads
          There is feature that is there in Percona tree 5.5+ and also in MySQL/Enterprise
          but not in MariaDB is priorization

          - By default, connection that is in transaction gets a priority boost (which translated to
          in transaction= low prio, outside of transaction => high prio). The first statement after "begin" does not need to be high prio (this transaction does not actually use any resources)

          - We need to have a variable that will allow to specify per-connection or global scheduling behavior (auto = based on transaction status, high - always high prio, low- always low prio)
          This way, it would be easy to return back to FIFO scheduling if someone prefers that (i.e set global threadpool_priority=high). Or for a connection to be "nice" (setting priority = low regardless of transaction status), if it does not unimportant work . This variable is akin to Percona's threadpool_high_prio_mode=[transactions|statements|none] , but with an understandable name.

          - On Unix ,we'll use separate queues for high priority and low priority items. high priority queue is checked before low prio when an event is dequeued.

          - To avoid starvation of the low prio items, periodically the old items in low prio queue need to be merged at the end of high prio queue. We can use 'thread_pool_kickup_timer' logic, like MySQL Enterprise is doing it.

          - We won't be using 'throttling of low prio queue' like Percona, as restricting the number of active+waiting threads can easily lead to a deadlock (single global lock or table lock would suffice to check that)
          Summary Port percona threadpool improvements to MariaDB tree Threadpool : add priorization features
          wlad Vladislav Vaintroub made changes -
          Description There is feature that is there in Percona tree 5.5+ and also in MySQL/Enterprise
          but not in MariaDB is priorization

          - By default, connection that is in transaction gets a priority boost (which translated to
          in transaction= low prio, outside of transaction => high prio). The first statement after "begin" does not need to be high prio (this transaction does not actually use any resources)

          - We need to have a variable that will allow to specify per-connection or global scheduling behavior (auto = based on transaction status, high - always high prio, low- always low prio)
          This way, it would be easy to return back to FIFO scheduling if someone prefers that (i.e set global threadpool_priority=high). Or for a connection to be "nice" (setting priority = low regardless of transaction status), if it does not unimportant work . This variable is akin to Percona's threadpool_high_prio_mode=[transactions|statements|none] , but with an understandable name.

          - On Unix ,we'll use separate queues for high priority and low priority items. high priority queue is checked before low prio when an event is dequeued.

          - To avoid starvation of the low prio items, periodically the old items in low prio queue need to be merged at the end of high prio queue. We can use 'thread_pool_kickup_timer' logic, like MySQL Enterprise is doing it.

          - We won't be using 'throttling of low prio queue' like Percona, as restricting the number of active+waiting threads can easily lead to a deadlock (single global lock or table lock would suffice to check that)
          There is feature that is there in Percona tree 5.5+ and also in MySQL/Enterprise
          but not in MariaDB is priorization

          - By default, connection that is in transaction gets a priority boost (which translated to
          in transaction= low prio, outside of transaction => high prio). The first statement after "begin" does not need to be high prio (this transaction does not actually use any resources)

          - We need to have a variable that will allow to specify per-connection or global scheduling behavior (auto = based on transaction status, high - always high prio, low- always low prio)
          This way, it would be easy to return back to FIFO scheduling if someone prefers that (i.e set global threadpool_priority=high). Or for a connection to be "nice" (setting priority = low regardless of transaction status), if it does not unimportant work . This variable is akin to Percona's threadpool_high_prio_mode=[transactions|statements|none] , but with an understandable name.

          - On Unix ,we'll use separate queues for high priority and low priority items. high priority queue is checked before low prio when an event is dequeued.

          - On Windows threadpool, it is possible to 'yield' low prio work items.

          - On Unix, to avoid starvation of the low prio items, periodically the old items in low prio queue need to be merged at the end of high prio queue. We can use 'thread_pool_kickup_timer' logic, like MySQL Enterprise is doing it.

          - We won't be using 'throttling of low prio queue' like Percona, as restricting the number of active+waiting threads can easily lead to a deadlock (single global lock or table lock would suffice to check that)
          wlad Vladislav Vaintroub made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          wlad Vladislav Vaintroub made changes -
          Assignee Vladislav Vaintroub [ wlad ] Sergei Golubchik [ serg ]
          Status In Progress [ 3 ] In Review [ 10002 ]
          wlad Vladislav Vaintroub made changes -
          Fix Version/s 10.2 [ 14601 ]
          Fix Version/s 10.1 [ 16100 ]
          serg Sergei Golubchik made changes -
          Assignee Sergei Golubchik [ serg ] Vladislav Vaintroub [ wlad ]
          Status In Review [ 10002 ] Stalled [ 10000 ]
          wlad Vladislav Vaintroub made changes -
          Component/s OTHER [ 10125 ]
          Fix Version/s 10.2.2 [ 22013 ]
          Fix Version/s 10.2 [ 14601 ]
          Resolution Fixed [ 1 ]
          Status Stalled [ 10000 ] Closed [ 6 ]
          serg Sergei Golubchik made changes -
          Workflow MariaDB v3 [ 76251 ] MariaDB v4 [ 132908 ]

          People

            wlad Vladislav Vaintroub
            wlad Vladislav Vaintroub
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.