Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-27199

Require ib_logfile0 to exist unless innodb_force_recovery=6

Details

    Description

      The primary purpose of the parameter innodb_force_recovery is to allow data to be dumped from a corrupted database.

      Ironically, the parameter may also be used to corrupt a database. Until MDEV-19514 in MariaDB Server 10.5, innodb_force_recovery=4 and larger settings could cause corruption of secondary indexes.

      In all versions, the setting innodb_force_recovery=6 will cause the redo log to be ignored. This setting was somewhat redundant, because a similar effect could be achieved by replacing ib_logfile0 with a logically empty log at the desired log sequence number, or by deleting the ib_logfile0 altogether.

      We must not allow InnoDB to start up when the log file is missing or physically empty, except when the option innodb_force_recovery=6 is specified, to ignore the redo log altogether and to start the database in read-only mode.

      Previously, to allow InnoDB to automatically create a lost log file, a field FIL_PAGE_FILE_FLUSH_LSN was written to the first page of the system tablespace on shutdown, bypassing the redo log and the doublewrite buffer. If the server was killed in the middle of that write, the entire server could fail to start up, due to a corrupted first page of the system tablespace. Because we will no longer automatically create a new log file, we will not update that field in the system tablespace either.

      Attachments

        Issue Links

          Activity

            MDEV-14425 will make InnoDB refuse to start up without ib_logfile0 unless innodb_force_recovery=6 is specified. In preview-10.8-MDEV-14425-innodb there is a change that removes the updates of the field FIL_PAGE_FILE_FLUSH_LSN in the system tablespace, because due to MDEV-14425, that field is no longer being read.

            marko Marko Mäkelä added a comment - MDEV-14425 will make InnoDB refuse to start up without ib_logfile0 unless innodb_force_recovery=6 is specified. In preview-10.8- MDEV-14425 -innodb there is a change that removes the updates of the field FIL_PAGE_FILE_FLUSH_LSN in the system tablespace, because due to MDEV-14425 , that field is no longer being read.

            The tree
            origin/bb-10.8-MDEV-14425 614e46b89ffe7357e5b72ea0d0fd3f490567a384 2022-01-13T20:32:56+02:00
            which contains the patch for MDEV-27199 behaved well in RQG testing.
            Bad effects observed exist in the main trees too and are known.
            

            mleich Matthias Leich added a comment - The tree origin/bb-10.8-MDEV-14425 614e46b89ffe7357e5b72ea0d0fd3f490567a384 2022-01-13T20:32:56+02:00 which contains the patch for MDEV-27199 behaved well in RQG testing. Bad effects observed exist in the main trees too and are known.

            Thing with ib_logfile0 is that people may be used to removing it from offline backups. I certainly did that many times, and if I did, I expect many others do that too. That one can remove redo log after regular shutdown, is common knowledge. It should not take 20GB in the backup, when logically, it is empty. If there is not much effort to retain that feature, I'd retain it, but otherwise write a big Warning into the release notes.

            wlad Vladislav Vaintroub added a comment - Thing with ib_logfile0 is that people may be used to removing it from offline backups. I certainly did that many times, and if I did, I expect many others do that too. That one can remove redo log after regular shutdown, is common knowledge. It should not take 20GB in the backup, when logically, it is empty. If there is not much effort to retain that feature, I'd retain it, but otherwise write a big Warning into the release notes.

            wlad, before MDEV-14425, mariadb-backup --prepare created a zero-length ib_logfile0 as a dummy placeholder. In MDEV-14425, the size of that dummy file was increased to 12304 (0x3010) bytes. The code change attached to this MDEV-27199 will merely remove all updates of FIL_PAGE_FILE_FLUSH_LSN in the first page of the system tablespace.

            If the server is started up with a zero-sized ib_logfile0, in MDEV-14425 we will assume that we are upgrading after a backup had been prepared. The start LSN will then be read from FIL_PAGE_FILE_FLUSH_LSN, and a new log file will be created starting from exactly that LSN.

            If a user tries to be smart and creates a zero-sized ib_logfile0 without manually updating the FIL_PAGE_FILE_FLUSH_LSN in the system tablespace to a recent enough LSN, then scary error messages like "page LSN is in the future" may be issued. As always, if some log was discarded while some changes had already been written to data pages, all sort of corruption may occur.

            If the database was initialized with a server that never updates the FIL_PAGE_FILE_FLUSH_LSN field, then any server startup attempts with a zero-size ib_logfile0 will be refused because of an invalid LSN. If that field was ever updated with a valid LSN by an older server, this safety mechanism cannot work, and the server may "rewind" to an earlier LSN.

            I believe that any operations that would rebuild the redo log should be left to code that has been developed and tested. Users should not be encouraged to directly manipulate the log files themselves.

            marko Marko Mäkelä added a comment - wlad , before MDEV-14425 , mariadb-backup --prepare created a zero-length ib_logfile0 as a dummy placeholder. In MDEV-14425 , the size of that dummy file was increased to 12304 (0x3010) bytes. The code change attached to this MDEV-27199 will merely remove all updates of FIL_PAGE_FILE_FLUSH_LSN in the first page of the system tablespace. If the server is started up with a zero-sized ib_logfile0 , in MDEV-14425 we will assume that we are upgrading after a backup had been prepared. The start LSN will then be read from FIL_PAGE_FILE_FLUSH_LSN , and a new log file will be created starting from exactly that LSN. If a user tries to be smart and creates a zero-sized ib_logfile0 without manually updating the FIL_PAGE_FILE_FLUSH_LSN in the system tablespace to a recent enough LSN, then scary error messages like "page LSN is in the future" may be issued. As always, if some log was discarded while some changes had already been written to data pages, all sort of corruption may occur. If the database was initialized with a server that never updates the FIL_PAGE_FILE_FLUSH_LSN field, then any server startup attempts with a zero-size ib_logfile0 will be refused because of an invalid LSN. If that field was ever updated with a valid LSN by an older server, this safety mechanism cannot work, and the server may "rewind" to an earlier LSN. I believe that any operations that would rebuild the redo log should be left to code that has been developed and tested. Users should not be encouraged to directly manipulate the log files themselves.

            It would be great if the above was clarified in the documentation.

            Roel Roel Van de Paar added a comment - It would be great if the above was clarified in the documentation.

            greenman, can you please make sure that this as well as MDEV-14425 will be documented?

            marko Marko Mäkelä added a comment - greenman , can you please make sure that this as well as MDEV-14425 will be documented?

            Users who run performance tests (such as wlad) might want to initialize a server with test data, shut it down and then back up the files, to be restored before read/write performance test runs. That would save the trouble of loading data before each test run, and guarantee that each test is starting from the same state.

            Such users could be accustomed to deleting the ib_logfile0 to save space in the backup. That used to be safe after a clean shutdown. Now that we no longer store the latest LSN anywhere else than the log file, the log file must not be deleted. To save space in the backup, the server could be restarted with a smaller innodb_log_file_size and then shut down. Alternatively, the database could be copied with mariadb-backup and prepared with mariadb-backup --prepare to create a minimal log file, and then backed up.

            marko Marko Mäkelä added a comment - Users who run performance tests (such as wlad ) might want to initialize a server with test data, shut it down and then back up the files, to be restored before read/write performance test runs. That would save the trouble of loading data before each test run, and guarantee that each test is starting from the same state. Such users could be accustomed to deleting the ib_logfile0 to save space in the backup. That used to be safe after a clean shutdown. Now that we no longer store the latest LSN anywhere else than the log file, the log file must not be deleted. To save space in the backup, the server could be restarted with a smaller innodb_log_file_size and then shut down. Alternatively, the database could be copied with mariadb-backup and prepared with mariadb-backup --prepare to create a minimal log file, and then backed up.

            MDEV-27812 would allow a DBA to shrink the log file before shutting down the server. The log could simply be resized to its minimal size:

            SET GLOBAL innodb_log_file_size=4096*1024;
            

            marko Marko Mäkelä added a comment - MDEV-27812 would allow a DBA to shrink the log file before shutting down the server. The log could simply be resized to its minimal size: SET GLOBAL innodb_log_file_size=4096*1024;

            People

              marko Marko Mäkelä
              marko Marko Mäkelä
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.