Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-15016

multiple page cleaner threads use a lot of CPU on idle server

    XMLWordPrintable

Details

    • 10.1.31

    Description

      Issue described in https://lists.launchpad.net/maria-discuss/msg04975.html as follows
      "
      On a 4-core (4 physical, 8 logical cores), 3.5 Ghz box, MariaDB 10.3.4
      consumes typically 35-40% of available CPU even when completely idle
      according to Task Manager. Please see image.

      I cannot tell if previous 10.3 versions also did, as I don't have 10.3
      running constantly in the backgorund. I just observed this now. This does
      not happen with MariaDB 10.1, MariaDB 10.2, MySQL 5.6, MySQL 5.7 and MySQL
      8.0 (all those I have starting with Windows on this box), where the
      CPU-load is neglible unless the server is "doing real work". As a result, the CPU clock frequency is always highest possible 3.5 Ghz
      (this CPU model is a 4th generation Intel with the option to scale down CPU
      frequency to around 40-50% of max. when there is no need for more) when
      MariaDB 10.3.4 is running.

      "

      Examination of the attached stacktraces taken from the process when it in this "idle" state, shows 3 CPUs being occupied by something, that for most time seems to be some busy-look acquiring innodb mutex
      .
      my.ini uses all defaults, with the exception of
      innodb_buffer_pool_size=4091M

      Setting innodb_page_cleaners = 1, according to reporter, fixed the issue.

      Relevant callstack information :

       
      		 
      . 15  Id: 2ce8.2d38 Suspend: 0 Teb: 0000005f`55863000 Unfrozen
      		 
            Start: mysqld!buf_flush_page_cleaner_worker (00007ff7`6aec9370)
      		 
            Priority: 0  Priority class: 32  Affinity: ff
      		 
      Child-SP          RetAddr           Call Site
      		 
      0000005f`56cffd40 00007ff7`6aec945d ntdll!RtlLeaveCriticalSection+0x29
      		 
      0000005f`56cffd70 00007ffa`6d831fe4 mysqld!buf_flush_page_cleaner_worker+0xed [d:\winx64-packages\build\src\storage\innobase\buf\buf0flu.cc @ 3574]
      		 
      0000005f`56cffdb0 00007ffa`6ff0efb1 kernel32!BaseThreadInitThunk+0x14
      		 
      0000005f`56cffde0 00000000`00000000 ntdll!RtlUserThreadStart+0x21
      		 
       
      		 
      . 16  Id: 2ce8.2d3c Suspend: 0 Teb: 0000005f`55865000 Unfrozen
      		 
            Start: mysqld!buf_flush_page_cleaner_worker (00007ff7`6aec9370)
      		 
            Priority: 0  Priority class: 32  Affinity: ff
      		 
      Child-SP          RetAddr           Call Site
      		 
      0000005f`56dff6c0 00007ff7`6aec8503 mysqld!PolicyMutex<TTASEventMutex<GenericPolicy> >::enter+0x26 [d:\winx64-packages\build\src\storage\innobase\include\ib0mutex.h @ 622]
      		 
      0000005f`56dff740 00007ff7`6aec944b mysqld!pc_flush_slot+0x43 [d:\winx64-packages\build\src\storage\innobase\buf\buf0flu.cc @ 2841]
      		 
      0000005f`56dff7b0 00007ffa`6d831fe4 mysqld!buf_flush_page_cleaner_worker+0xdb [d:\winx64-packages\build\src\storage\innobase\buf\buf0flu.cc @ 3574]
      		 
      0000005f`56dff7f0 00007ffa`6ff0efb1 kernel32!BaseThreadInitThunk+0x14
      		 
      0000005f`56dff820 00000000`00000000 ntdll!RtlUserThreadStart+0x21
      		 
       
      		 
      . 17  Id: 2ce8.2d40 Suspend: 0 Teb: 0000005f`55867000 Unfrozen
      		 
            Start: mysqld!buf_flush_page_cleaner_worker (00007ff7`6aec9370)
      		 
            Priority: 0  Priority class: 32  Affinity: ff
      		 
      Child-SP          RetAddr           Call Site
      		 
      0000005f`56eff750 00007ff7`6aec8503 mysqld!PolicyMutex<TTASEventMutex<GenericPolicy> >::enter+0x57 [d:\winx64-packages\build\src\storage\innobase\include\ib0mutex.h @ 635]
      		 
      0000005f`56eff7d0 00007ff7`6aec944b mysqld!pc_flush_slot+0x43 [d:\winx64-packages\build\src\storage\innobase\buf\buf0flu.cc @ 2841]
      		 
      0000005f`56eff840 00007ffa`6d831fe4 mysqld!buf_flush_page_cleaner_worker+0xdb [d:\winx64-packages\build\src\storage\innobase\buf\buf0flu.cc @ 3574]
      		 
      0000005f`56eff880 00007ffa`6ff0efb1 kernel32!BaseThreadInitThunk+0x14
      		 
      0000005f`56eff8b0 00000000`00000000 ntdll!RtlUserThreadStart+0x21


      . 18  Id: c10.1c30 Suspend: 0 Teb: 00000012`c5100000 Unfrozen
      		 
            Start: mysqld!buf_flush_page_cleaner_worker (00007ff7`6aec9370)
      		 
            Priority: 0  Priority class: 32  Affinity: ff
      		 
      Child-SP          RetAddr           Call Site
      		 
      00000012`c63ff970 00007ffa`6d831fe4 mysqld!buf_flush_page_cleaner_worker+0xe8 [d:\winx64-packages\build\src\storage\innobase\buf\buf0flu.cc @ 3574]
      		 
      00000012`c63ff9b0 00007ffa`6ff0efb1 kernel32!BaseThreadInitThunk+0x14
      		 
      00000012`c63ff9e0 00000000`00000000 ntdll!RtlUserThreadStart+0x21
      		 
       
      		 
      . 19  Id: c10.a04 Suspend: 0 Teb: 00000012`c5102000 Unfrozen
      		 
            Start: mysqld!buf_flush_page_cleaner_worker (00007ff7`6aec9370)
      		 
            Priority: 0  Priority class: 32  Affinity: ff
      		 
      Child-SP          RetAddr           Call Site
      		 
      00000012`c64ff6d0 00007ff7`6ae0ef0f ntdll!RtlEnterCriticalSection+0x13
      		 
      (Inline Function) --------`-------- mysqld!OSMutex::enter+0xa [d:\winx64-packages\build\src\storage\innobase\include\sync0types.h @ 468]
      		 
      00000012`c64ff700 00007ff7`6aec945d mysqld!os_event::wait_low+0x1f [d:\winx64-packages\build\src\storage\innobase\os\os0event.cc @ 330]
      		 
      00000012`c64ff730 00007ffa`6d831fe4 mysqld!buf_flush_page_cleaner_worker+0xed [d:\winx64-packages\build\src\storage\innobase\buf\buf0flu.cc @ 3574]
      		 
      00000012`c64ff770 00007ffa`6ff0efb1 kernel32!BaseThreadInitThunk+0x14
      		 
      00000012`c64ff7a0 00000000`00000000 ntdll!RtlUserThreadStart+0x21
      		 
       
      		 
      . 20  Id: c10.161c Suspend: 0 Teb: 00000012`c5104000 Unfrozen
      		 
            Start: mysqld!buf_flush_page_cleaner_worker (00007ff7`6aec9370)
      		 
            Priority: 0  Priority class: 32  Affinity: ff
      		 
      Child-SP          RetAddr           Call Site
      		 
      00000012`c65ff9e8 00007ffa`6feec17f ntdll!NtWaitForAlertByThreadId+0x14
      		 
      00000012`c65ff9f0 00007ffa`6feebf4b ntdll!RtlpWaitOnAddressWithTimeout+0x43
      		 
      00000012`c65ffa20 00007ffa`6fec0c49 ntdll!RtlpWaitOnCriticalSection+0x15f
      		 
      00000012`c65ffad0 00007ffa`6fec0b60 ntdll!RtlpEnterCriticalSectionContended+0xd9
      		 
      00000012`c65ffb00 00007ff7`6ae0ef0f ntdll!RtlEnterCriticalSection+0x40
      		 
      (Inline Function) --------`-------- mysqld!OSMutex::enter+0xa [d:\winx64-packages\build\src\storage\innobase\include\sync0types.h @ 468]
      		 
      00000012`c65ffb30 00007ff7`6aec945d mysqld!os_event::wait_low+0x1f [d:\winx64-packages\build\src\storage\innobase\os\os0event.cc @ 330]
      		 
      00000012`c65ffb60 00007ffa`6d831fe4 mysqld!buf_flush_page_cleaner_worker+0xed [d:\winx64-packages\build\src\storage\innobase\buf\buf0flu.cc @ 3574]
      		 
      00000012`c65ffba0 00007ffa`6ff0efb1 kernel32!BaseThreadInitThunk+0x14
      		 
      00000012`c65ffbd0 00000000`00000000 ntdll!RtlUserThreadStart+0x21

      Attachments

        1. mysqld.DMP.bz2.001
          10.00 MB
          Vladislav Vaintroub
        2. mysqld.DMP.bz2.002
          10.00 MB
          Vladislav Vaintroub
        3. mysqld.DMP.bz2.003
          1.68 MB
          Vladislav Vaintroub
        4. mysqld0.DMP.xz.001
          10.00 MB
          Vladislav Vaintroub
        5. mysqld0.DMP.xz.002
          6.22 MB
          Vladislav Vaintroub
        6. stacks1.txt
          30 kB
          Vladislav Vaintroub
        7. stacks2.txt
          30 kB
          Vladislav Vaintroub

        Issue Links

          is caused by

          Task - A task that needs to be done. MDEV-11025 Make number of page cleaner threads variable dynamic

          • Major - Major loss of function.
          • Closed
          relates to

          Task - A task that needs to be done. MDEV-15053 Reduce buf_pool_t::mutex contention

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Task - A task that needs to be done. MDEV-15058 Remove multiple InnoDB buffer pool instances

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-9784 High CPU Usage When Absolutely Idle on OSX

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-11738 Mariadb uses 100% of several of my 8 cpus doing nothing

          • Major - Major loss of function.
          • Closed

          Activity

            People

              marko Marko Mäkelä
              wlad Vladislav Vaintroub
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.