Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-33009

Server hangs for a long time with innodb_undo_log_truncate=ON

Details

    Description

      After implementing MDEV-32757, we are seeing a performance anomaly with innodb_undo_log_truncate=ON. The server is not actually hung or deadlocked (it will eventually recover), but buf_pool.mutex is being occupied for an extremely long time (several minutes).

      1. trx_purge_truncate_history() writes the message InnoDB: Truncating and is about to truncate an undo log tablespace.
      2. trx_purge_truncate_history() is busy-looping in a scan of buf_pool.flush_list because one of the pages belonging to the undo tablespace is write-fixed.
      3. During the time trx_purge_truncate_history() releases and re-acquires buf_pool.flush_list_mutex, buf_flush_page_cleaner (which is holding buf_pool.mutex in buf_do_flush_list_batch()) cannot grab it, in this Ubuntu 18.04 version of GNU libc and Linux kernel (4.15.0-112-generic). This could be similar to MDEV-31343 and MDEV-30180, which could only be reproduced in the same particular environment.
      4. Most threads are blocked because the buf_flush_page_cleaner thread is holding buf_pool.mutex.

      There is some indication that buf_flush_list_batch() may be making some progress (writing out some pages), but it would be extremely slow.

      Attachments

        Issue Links

          Activity

            marko Marko Mäkelä created issue -
            marko Marko Mäkelä made changes -
            Field Original Value New Value
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            marko Marko Mäkelä made changes -
            Status In Progress [ 3 ] In Testing [ 10301 ]
            marko Marko Mäkelä made changes -
            Assignee Marko Mäkelä [ marko ] Axel Schwenke [ axel ]
            axel Axel Schwenke made changes -
            Attachment mariadb-ES-10.5-MDEV-33009.zip [ 72684 ]
            axel Axel Schwenke made changes -
            Attachment 10.6-MDEV-33009-1.png [ 72685 ]
            axel Axel Schwenke made changes -
            marko Marko Mäkelä made changes -
            axel Axel Schwenke made changes -
            Status In Testing [ 10301 ] Stalled [ 10000 ]
            axel Axel Schwenke made changes -
            Assignee Axel Schwenke [ axel ] Marko Mäkelä [ marko ]
            marko Marko Mäkelä made changes -
            Environment Ubuntu 18.04 on AMD64 Ubuntu 18.04 on AMD64
            Ubuntu 20.04 on AMD64
            marko Marko Mäkelä made changes -
            Status Stalled [ 10000 ] In Progress [ 3 ]
            marko Marko Mäkelä made changes -
            Status In Progress [ 3 ] In Testing [ 10301 ]
            marko Marko Mäkelä made changes -
            Assignee Marko Mäkelä [ marko ] Axel Schwenke [ axel ]
            axel Axel Schwenke made changes -
            Status In Testing [ 10301 ] Stalled [ 10000 ]
            axel Axel Schwenke made changes -
            Assignee Axel Schwenke [ axel ] Marko Mäkelä [ marko ]
            axel Axel Schwenke made changes -
            axel Axel Schwenke made changes -
            axel Axel Schwenke made changes -
            Attachment 24x5_high_threads.pdf [ 72702 ]
            marko Marko Mäkelä made changes -
            axel Axel Schwenke made changes -
            Attachment MDEV-33009-9682add5cdf.png [ 72706 ]
            axel Axel Schwenke made changes -
            Attachment MDEV-33062-3ef0e678b1c.png [ 72707 ]
            marko Marko Mäkelä made changes -
            issue.field.resolutiondate 2023-12-19 15:47:41.0 2023-12-19 15:47:41.396
            marko Marko Mäkelä made changes -
            Fix Version/s 10.5.24 [ 29517 ]
            Fix Version/s 10.6.17 [ 29518 ]
            Fix Version/s 10.11.7 [ 29519 ]
            Fix Version/s 11.0.5 [ 29520 ]
            Fix Version/s 11.1.4 [ 29024 ]
            Fix Version/s 11.2.3 [ 29521 ]
            Fix Version/s 11.3.2 [ 29522 ]
            Fix Version/s 10.5 [ 23123 ]
            Fix Version/s 10.6 [ 24028 ]
            Fix Version/s 10.11 [ 27614 ]
            Fix Version/s 11.0 [ 28320 ]
            Fix Version/s 11.1 [ 28549 ]
            Fix Version/s 11.3 [ 28565 ]
            Fix Version/s 11.2 [ 28603 ]
            Resolution Fixed [ 1 ]
            Status Stalled [ 10000 ] Closed [ 6 ]
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -

            People

              marko Marko Mäkelä
              marko Marko Mäkelä
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.