Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-14505

Threads_running becomes scalability bottleneck

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Fixed
    • 5.5(EOL), 10.0(EOL), 10.1(EOL), 10.2(EOL), 10.3(EOL)
    • 10.3.3
    • Server

    Description

      thread_running is global shared variable, which is updated twice per query in dispatch_command() using atomic-add.

      It started to appear on radar in OLTP RO benchmark. E.g here're numbers for 2 socket/20 cores/40 threads Intel Broadwell system:

         0,70%  mysqld               [.] dispatch_command
      		 
      ...
      		 
             │      inline void thread_safe_increment32(int32 *value)
      		 
             │      {
      		 
             │        (void) my_atomic_add32_explicit(value, 1, MY_MEMORY_ORDER_RELAXED);
      		 
        0,59 │ 680:   lea    thread_running,%rax
      		 
       16,07 │        lock   addl   $0x1,(%rax)

      If we remove inc_thread_running() and dec_thread_running() we get slightly better throughput and dispatch_command() goes down in profiler:

         0,58%  mysqld               [.] dispatch_command
      		 
      ...
      		 
             │      /* increment query_id and return it.  */
      		 
             │      inline __attribute__((warn_unused_result)) query_id_t next_query_id()
      		 
             │      {
      		 
             │        return my_atomic_add64_explicit(&global_query_id, 1, MY_MEMORY_ORDER_RELAXED);
      		 
        0,49 │ 290:   mov    $0x1,%edx
      		 
       18,02 │        lock   xadd   %rdx,(%r14)

      So bottleneck is shifted to global_query_id counter, which is subject for another bug.

      I expect much higher scalability impact on more powerful hardware.

      Attachments

        Issue Links

          is blocked by

          Bug - A problem which impairs or prevents the functions of the product. MDEV-18287 Status threads_running show wrong value since 10.3

          • Major - Major loss of function.
          • Closed

          Activity

            Descending order - Click to sort in ascending order
            svoj Sergey Vojtovich added a comment -

            Patch was approved by wlad.

            svoj Sergey Vojtovich added a comment - Patch was approved by wlad .
            svoj Sergey Vojtovich added a comment -

            ssethia, dthompson FYI: I added this under MDEV-14442, because it may have negative impact for ARM as well. Nevertheless it is generic scalability bottleneck, so feel free to move it out of scope of arm64 optimisations.

            svoj Sergey Vojtovich added a comment - ssethia , dthompson FYI: I added this under MDEV-14442 , because it may have negative impact for ARM as well. Nevertheless it is generic scalability bottleneck, so feel free to move it out of scope of arm64 optimisations.

            People

              svoj Sergey Vojtovich
              svoj Sergey Vojtovich
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.