Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-24789

Performance regression after MDEV-24671

Details

    Description

      The fix of MDEV-24671 introduced a serious performance regression that is observable at 32 concurrent connections.

      My current hypothesis based on some initial investigation is that the changed sizeof(trx->lock) caused an increase of cache misses.

      Attachments

        Issue Links

          Activity

            marko Marko Mäkelä created issue -
            marko Marko Mäkelä made changes -
            Field Original Value New Value
            marko Marko Mäkelä made changes -
            Status Open [ 1 ] In Progress [ 3 ]

            Reducing sizeof(trx_lock_t) did not help (I did it anyway), but some refactoring to reduce the hold time of lock_sys.mutex and lock_sys.wait_mutex seems to have done the trick. Thanks to axel for helping with the analysis and validation!

            marko Marko Mäkelä added a comment - Reducing sizeof(trx_lock_t) did not help (I did it anyway), but some refactoring to reduce the hold time of lock_sys.mutex and lock_sys.wait_mutex seems to have done the trick. Thanks to axel for helping with the analysis and validation!
            marko Marko Mäkelä made changes -
            issue.field.resolutiondate 2021-02-05 17:15:44.0 2021-02-05 17:15:44.751
            marko Marko Mäkelä made changes -
            Fix Version/s 10.6.0 [ 24431 ]
            Fix Version/s 10.6 [ 24028 ]
            Resolution Fixed [ 1 ]
            Status In Progress [ 3 ] Closed [ 6 ]

            Some performance regression is still present. The function lock_wait() that was introduced in MDEV-24671 was holding lock_sys.wait_mutex for unnecessarily long time.

            marko Marko Mäkelä added a comment - Some performance regression is still present. The function lock_wait() that was introduced in MDEV-24671 was holding lock_sys.wait_mutex for unnecessarily long time.
            marko Marko Mäkelä made changes -
            Resolution Fixed [ 1 ]
            Status Closed [ 6 ] Stalled [ 10000 ]
            marko Marko Mäkelä made changes -
            Status Stalled [ 10000 ] In Progress [ 3 ]
            marko Marko Mäkelä made changes -
            Fix Version/s 10.6 [ 24028 ]
            Fix Version/s 10.6.0 [ 24431 ]

            Even after reducing the lock_sys.wait_mutex hold time in lock_wait() to the minimum, some regression is present, and more work needs to be done.

            marko Marko Mäkelä added a comment - Even after reducing the lock_sys.wait_mutex hold time in lock_wait() to the minimum, some regression is present, and more work needs to be done.
            marko Marko Mäkelä made changes -

            It looks like MDEV-24671 emphasized a pre-existing bottleneck on log_sys.mutex, which we hope to address in MDEV-14425. The remaining performance regression was only observed on RAM disk, not on real storage.

            That said, the latest change that is being tested should reduce contention on lock_sys.latch and lock_sys.wait_mutex to the absolute minimum.

            marko Marko Mäkelä added a comment - It looks like MDEV-24671 emphasized a pre-existing bottleneck on log_sys.mutex , which we hope to address in MDEV-14425 . The remaining performance regression was only observed on RAM disk, not on real storage. That said, the latest change that is being tested should reduce contention on lock_sys.latch and lock_sys.wait_mutex to the absolute minimum.
            marko Marko Mäkelä made changes -
            issue.field.resolutiondate 2021-03-02 12:41:36.0 2021-03-02 12:41:36.866
            marko Marko Mäkelä made changes -
            Fix Version/s 10.6.0 [ 24431 ]
            Fix Version/s 10.6 [ 24028 ]
            Resolution Fixed [ 1 ]
            Status In Progress [ 3 ] Closed [ 6 ]
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            serg Sergei Golubchik made changes -
            Workflow MariaDB v3 [ 118851 ] MariaDB v4 [ 158862 ]

            People

              marko Marko Mäkelä
              marko Marko Mäkelä
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.