Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-30000

make mariadb-backup to force an innodb checkpoint

Details

    Description

      Since MDEV-23855 InnoDB can do checkpoints much more rarely. This improves performance under write-heavy load, but makes mariadb-backup create huge incremental backups (since the last checkpoint).

      One way to solve it would be to let mariadb-backup to force an InnoDB checkpoint before a backup. There are many ways of doing it, innodb.page_cleaner test shows one of them, marko knows more. Perhaps it should be optional, but, likely, enabled by default.

      Alternatively, InnoDB can force a checkpoint automatically when entering a certain backup stage.

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MDEV-24537 innodb_max_dirty_pages_pct_lwm=0 lost its special meaning

          • Blocker - Blocks development and/or testing work, production could not run.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-24949 Enabling idle flushing (possible regression from MDEV-23855)

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-26626 InnoDB fails to advance the log checkpoint

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-34062 mariadb-backup --backup is extremely slow at copying ib_logfile0

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-31410 mariadb-backup prepare crash with InnoDB: Missing FILE_CREATE, FILE_DELETE or FILE_MODIFY before FILE_CHECKPOINT

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-31446 mariabackup loop on Read redo log up to LSN

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. MDEV-33838 Data still present in backup / on disk after table delete or/and import from sql

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. MDEV-35298 Issues with --extra-lsndir in MariaDB Incremental Backups Not Updating LSN as Expected

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. MDEV-36159 mariabackup failed after upgrade 10.11.10

          • Critical - Crashes, loss of data, severe memory leak.
          • In Progress

          Task - A task that needs to be done. MDEV-36254 Document MDEV-30000: make mariadb-backup force an innodb checkpoint

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. MDEV-36270 mariabackup.incremental_compressed fails in 10.11+

          • Major - Major loss of function.
          • Closed
          links to

          Web Link Question on Launchpad

          Activity

            Ascending order - Click to sort in descending order
            View 18 older comments
            serg Sergei Golubchik added a comment -

            I think that

            SET GLOBAL innodb_log_checkpoint_now=ON;

            would be the best approach. It clearly says what's to be done, doesn't make any system-wide changes that need to be restored (so, no problems if the client suddenly disconnects). Let's do that?

            serg Sergei Golubchik added a comment - I think that SET GLOBAL innodb_log_checkpoint_now= ON ; would be the best approach. It clearly says what's to be done, doesn't make any system-wide changes that need to be restored (so, no problems if the client suddenly disconnects). Let's do that?
            marko Marko Mäkelä added a comment -

            brevilo, I don’t think that setting innodb_log_buffer_size larger than a few megabytes should help at all. The parameter innodb_log_file_size should be much more relevant.

            marko Marko Mäkelä added a comment - brevilo , I don’t think that setting innodb_log_buffer_size larger than a few megabytes should help at all. The parameter innodb_log_file_size should be much more relevant.
            brevilo Oliver Bock added a comment - - edited

            Ooops, my bad, I meant innodb_log_file_size = 64 GB, not innodb_log_buffer_size (still at the default of 16 MB!).

            I fixed my previous comment above but since that was meant as a reply to your's it kind of lost its context now Maybe it helps anyway...

            brevilo Oliver Bock added a comment - - edited Ooops, my bad, I meant innodb_log_file_size = 64 GB , not innodb_log_buffer_size (still at the default of 16 MB!). I fixed my previous comment above but since that was meant as a reply to your's it kind of lost its context now Maybe it helps anyway...
            serg Sergei Golubchik added a comment -

            10.11 and up looks appropriate

            serg Sergei Golubchik added a comment - 10.11 and up looks appropriate
            marko Marko Mäkelä added a comment -

            I introduced the parameter --skip-innodb-log-checkpoint-on to backup, so that the extra checkpoint at the start of the backup can be avoided when necessary. This is necessary in the regression test encryption.doublewrite_debug.

            marko Marko Mäkelä added a comment - I introduced the parameter --skip-innodb-log-checkpoint-on to backup, so that the extra checkpoint at the start of the backup can be avoided when necessary. This is necessary in the regression test encryption.doublewrite_debug .

            People

              marko Marko Mäkelä
              serg Sergei Golubchik
              Votes:
              9 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.