Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-36270

mariabackup.incremental_compressed fails in 10.11+

Details

    Description

      The tests fails since less than a week (first failure March 7th, 2025) in 10.11 and higher versions. First failure was in MDEV-30000 pull request, and it failed ever since it was merged.

      It fails multiple times a day, on all Windows builders.

      The error message is

      2025-03-11 14:26:44 0 [ERROR] InnoDB: The size of tablespace file 'test/t#p#p0.ibd' is only 32768, should be at least 65536!
      2025-03-11 14:26:44 0 [ERROR] InnoDB: Recovery cannot access file ./test/t#p#p0.ibd (tablespace 9)
      

      Previously, such errors were seen in 10.4, and were not really attended by anyone.

      Attachments

        Issue Links

          Activity

            Can you see if disabling the MDEV-30000 functionality in the backup command would help? That is, add --skip-innodb-log-checkpoint-now to the invocation?

            marko Marko Mäkelä added a comment - Can you see if disabling the MDEV-30000 functionality in the backup command would help? That is, add --skip-innodb-log-checkpoint-now to the invocation?

            I think that this needs to be debugged and the root cause analyzed on the Windows platform. My previous comment is not a suggested fix; it is only a first step of troubleshooting. We also have an open bug MDEV-36201 in this area, but I don’t think that it shares a root cause with this one.

            marko Marko Mäkelä added a comment - I think that this needs to be debugged and the root cause analyzed on the Windows platform. My previous comment is not a suggested fix; it is only a first step of troubleshooting. We also have an open bug MDEV-36201 in this area, but I don’t think that it shares a root cause with this one.

            marko, yes, the first step is to debug and such. And we have a Windows machine to do this debugging, by the same person who caused the break I'm here to help you with debug techniques, but not spend time on something that I have no clue of. In the past, I remember, files that were undersized were extended, then they were just ignored by mariabackup, but at some point it has changed, so I do not know why Innodb fails or even requires that files have some specific size.

            wlad Vladislav Vaintroub added a comment - marko , yes, the first step is to debug and such. And we have a Windows machine to do this debugging, by the same person who caused the break I'm here to help you with debug techniques, but not spend time on something that I have no clue of. In the past, I remember, files that were undersized were extended, then they were just ignored by mariabackup, but at some point it has changed, so I do not know why Innodb fails or even requires that files have some specific size.

            I think that "caused by MDEV-30000" is an incorrect claim. There is correlation, but not causation. This test has failed in the past too, not as frequently as now, but it has failed. Here is an example:

            10.4 ee59ca7ff1620dd74b9bc1449878c5d1acee0f53

            mariabackup.incremental_compressed 'innodb' w32 [ fail ]
            2024-04-26 18:58:30 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=69905
            2024-04-26 18:58:30 0 [ERROR] InnoDB: The size of tablespace file 'test\t#p#p0.ibd' is only 32768, should be at least 65536!
            2024-04-26 18:58:30 0 [Warning] InnoDB: We do not continue the crash recovery, because the table may become corrupt if we cannot apply the log records in the InnoDB log to it. To fix the problem and start mysqld:
            2024-04-26 18:58:30 0 [Note] InnoDB: 1) If there is a permission problem in the file and mysqld cannot open the file, you should modify the permissions.
            2024-04-26 18:58:30 0 [Note] InnoDB: 2) If the tablespace is not needed, or you can restore an older version from a backup, then you can remove the .ibd file, and use --innodb_force_recovery=1 to force startup without this file.
            2024-04-26 18:58:30 0 [Note] InnoDB: 3) If the file system or the disk is broken, and you cannot remove the .ibd file, you can set --innodb_force_recovery.
            2024-04-26 18:58:30 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
            [00] FATAL ERROR: 2024-04-26 18:58:31 mariabackup: innodb_init() returned 11 (Generic error).
            mysqltest: At line 20: exec of 'D:\Buildbot\amd64-windows-packages\build\extra\mariabackup\RelWithDebInfo\mariabackup.exe --prepare --target-dir=D:/Buildbot/amd64-windows-packages/build/mysql-test/var/32/tmp/backup --incremental-dir=D:/Buildbot/amd64-windows-packages/build/mysql-test/var/32/tmp/backup_inc1 --user=root >  D:/Buildbot/amd64-windows-packages/build/mysql-test/var/tmp/32/backup_prepare_1.log' failed, error: 1, status: 1, errno: 2
            

            A possible fix would be to adjust the deferred tablespace recovery that was introduced when MDEV-24626 was implemented. But we need to reproduce this within a debugger first. I have seen this test failing on Microsoft Windows only, and I am not productive in that environment.

            marko Marko Mäkelä added a comment - I think that "caused by MDEV-30000 " is an incorrect claim. There is correlation, but not causation. This test has failed in the past too, not as frequently as now, but it has failed. Here is an example: 10.4 ee59ca7ff1620dd74b9bc1449878c5d1acee0f53 mariabackup.incremental_compressed 'innodb' w32 [ fail ] … 2024-04-26 18:58:30 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=69905 2024-04-26 18:58:30 0 [ERROR] InnoDB: The size of tablespace file 'test\t#p#p0.ibd' is only 32768, should be at least 65536! 2024-04-26 18:58:30 0 [Warning] InnoDB: We do not continue the crash recovery, because the table may become corrupt if we cannot apply the log records in the InnoDB log to it. To fix the problem and start mysqld: 2024-04-26 18:58:30 0 [Note] InnoDB: 1) If there is a permission problem in the file and mysqld cannot open the file, you should modify the permissions. 2024-04-26 18:58:30 0 [Note] InnoDB: 2) If the tablespace is not needed, or you can restore an older version from a backup, then you can remove the .ibd file, and use --innodb_force_recovery=1 to force startup without this file. 2024-04-26 18:58:30 0 [Note] InnoDB: 3) If the file system or the disk is broken, and you cannot remove the .ibd file, you can set --innodb_force_recovery. 2024-04-26 18:58:30 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error [00] FATAL ERROR: 2024-04-26 18:58:31 mariabackup: innodb_init() returned 11 (Generic error). mysqltest: At line 20: exec of 'D:\Buildbot\amd64-windows-packages\build\extra\mariabackup\RelWithDebInfo\mariabackup.exe --prepare --target-dir=D:/Buildbot/amd64-windows-packages/build/mysql-test/var/32/tmp/backup --incremental-dir=D:/Buildbot/amd64-windows-packages/build/mysql-test/var/32/tmp/backup_inc1 --user=root > D:/Buildbot/amd64-windows-packages/build/mysql-test/var/tmp/32/backup_prepare_1.log' failed, error: 1, status: 1, errno: 2 A possible fix would be to adjust the deferred tablespace recovery that was introduced when MDEV-24626 was implemented. But we need to reproduce this within a debugger first. I have seen this test failing on Microsoft Windows only, and I am not productive in that environment.
            wlad Vladislav Vaintroub added a comment - - edited

            Yes, it has failed in the past. I said that, too. in 10.4, a year ago.

            I'm unable to reproduce it locally though
            mtr --repeat=100 did not help

            ...
            mariabackup.incremental_compressed 'undo3' w2 [ 99 pass ]   9762
            mariabackup.incremental_compressed 'undo3' w2 [ 100 pass ]  16080
            --------------------------------------------------------------------------
            The servers were restarted 0 times
            Completed: All 200 tests were successful.
             
            
            

            Also, I'm not exactly productive on any other environments but Windows. If I have to, I fix on Linux, and FreeBSD, and even Solaris. AIX, when it has to be.

            wlad Vladislav Vaintroub added a comment - - edited Yes, it has failed in the past. I said that, too. in 10.4, a year ago. I'm unable to reproduce it locally though mtr --repeat=100 did not help ... mariabackup.incremental_compressed 'undo3' w2 [ 99 pass ] 9762 mariabackup.incremental_compressed 'undo3' w2 [ 100 pass ] 16080 -------------------------------------------------------------------------- The servers were restarted 0 times Completed: All 200 tests were successful.   Also, I'm not exactly productive on any other environments but Windows. If I have to, I fix on Linux, and FreeBSD, and even Solaris. AIX, when it has to be.

            People

              thiru Thirunarayanan Balathandayuthapani
              wlad Vladislav Vaintroub
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.