Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-40932

SET GLOBAL innodb_log_archive=OFF may still break recovery

    XMLWordPrintable

Details

    • Not for Release Notes

    Description

      An assertion may fail in log_t::set_archive(archive=false), indicating that crash recovery could be broken as the result of executing SET GLOBAL innodb_log_archive=OFF.

      mleich reproduced the failure in the MDEV-14992 development branch during the execution of a BACKUP SERVER statement:

      MDEV-14992 4e71fc51f26ecab8ed556b352fa492ea343429d0

      2026-08-26 13:38:21 30 [Note] InnoDB: setting innodb_log_archive=1 at innodb_log_recovery_start=151323484
      2026-08-26 13:38:25 19 [Note] InnoDB: Online DDL : …
      mariadbd: /data/Server/MDEV-14992B/storage/innobase/log/log0log.cc:918: bool log_t::set_archive(my_bool, THD*, bool): Assertion `get_sequence_bit(get_lsn())' failed.
      

      I analyzed the execution trace:

      ssh pluto
      rr replay /data/results/1787747058/NMB-2531/1/rr/latest-trace
      

      We had innodb_log_file_size=209715200 and log_sys.first_lsn=0x3000 during the entire run, so there is an identity mapping between log file offsets and any LSN up to innodb_log_file_size.

      During the execution of BACKUP SERVER, innodb_log_archive had been temporarily enabled. At that point of time, the current LSN is still well below the file size, that is, all records had been written with the sequence bit 1.

      The assertion fails in the following piece of code that is restoring the old setting innodb_log_archive:

      log_t::set_archive()

            if (!get_sequence_bit(checkpoint))
              /*
                In the innodb_log_archive=ON format, mtr_t::finish_writer()
                always writes the sequence bit as 1. Ensure that the
                innodb_log_archive=OFF recovery will expect this value.
              */
              first_lsn-= capacity();
            ut_ad(get_sequence_bit(checkpoint));
            ut_ad(get_sequence_bit(get_lsn()));
            header_rewrite(false);
      

      At this time we have log_sys.last_checkpoint_lsn=0x8afd100 that is below the log_sys.file_size=0xc800000. The current LSN is 0xc812118 (0x12118 bytes after the end of the file). However, the last written LSN is within that file: log_sys.write_lsn=0xc7f5a41.

      If we ignored the failing assertion and proceeded to rewrite the log file header at log_sys.first_lsn=0x3000, things would be fine until any further log records are written from log_sys.buf to the file. Any subsequent records that would be appended to log_sys.buf would carry the sequence bit 0, indicating that the circular file had wrapped around an odd number of times (1 in this case). However, the mini-transactions that end between 0xc800000 and 0xc812118 would carry the incorrect sequence bit value 1. This would break crash recovery until a further log checkpoint is written. Recovery would stop at the last mini-transaction that ended before the wrap-around at 0xc800000.

      I believe that the following patch should prevent this corruption:

      diff --git a/storage/innobase/log/log0log.cc b/storage/innobase/log/log0log.cc
      index 16e7983e705..1c78c25e4a5 100644
      --- a/storage/innobase/log/log0log.cc
      +++ b/storage/innobase/log/log0log.cc
      @@ -847,7 +847,9 @@ bool log_t::set_archive(my_bool archive, THD *thd, bool backup) noexcept
             */
             circular_recovery_from_sequence_bit_0= !get_sequence_bit(wait_lsn);
           }
      -    else if (circular_recovery_from_sequence_bit_0 || checkpoint < first_lsn)
      +    else if (circular_recovery_from_sequence_bit_0 ||
      +             checkpoint < first_lsn ||
      +             get_sequence_bit(checkpoint) != get_sequence_bit(wait_lsn))
           {
             /*
               Some records after the latest checkpoint may have been written
      

      I think that we must rely on stress testing for validating this fix.

      Attachments

        Issue Links

          Activity

            People

              marko Marko Mäkelä
              marko Marko Mäkelä
              Marko Mäkelä Marko Mäkelä
              Thirunarayanan Balathandayuthapani Thirunarayanan Balathandayuthapani
              Matthias Leich Matthias Leich
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0d
                  0d
                  Logged:
                  Time Spent - 4.75h
                  4.75h

                  Git Integration

                    Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.