[MDEV-22913] Error: xtrabackup_apply_delta() upon preparing incremental backup Created: 2020-06-16  Updated: 2022-09-14  Resolved: 2022-09-14

Status: Closed
Project: MariaDB Server
Component/s: mariabackup
Affects Version/s: 10.5
Fix Version/s: 10.4.27, 10.5.18, 10.6.11, 10.7.7, 10.8.6, 10.9.4, 10.10.2

Type: Bug Priority: Major
Reporter: Elena Stepanova Assignee: Marko Mäkelä
Resolution: Duplicate Votes: 0
Labels: None

Attachments: File mbackup_backup_0.log     File mbackup_backup_1.log     File mbackup_backup_2.log     File mbackup_backup_3.log     File mbackup_prepare_0.log     File mbackup_prepare_1.log     File mbackup_prepare_2.log    
Issue Links:
Duplicate
duplicates MDEV-18589 Assertion `fil_space_t::physical_size... Closed
Relates
relates to MDEV-18550 InnoDB: Failing assertion: success du... Open
relates to MDEV-18589 Assertion `fil_space_t::physical_size... Closed
relates to MDEV-28473 field_ref_zero is not initialized in ... Closed
relates to MDEV-22918 innodb_init() returned 11 (Generic er... Closed

 Description   

This report is about a non-debug 10.5 build. Debug builds fail with an assertion failure from MDEV-18589. However, according to MDEV-18589, on non-debug builds it didn't have the failure described here, so I'm not sure it's the same problem.

10.5 e290e5a7

[00] 2020-06-16 15:12:21 page size for /dev/shm/var_xbackup_apply/backup_2//ibdata1.delta is 16384 bytes
[00] 2020-06-16 15:12:21 Applying /dev/shm/var_xbackup_apply/backup_2//ibdata1.delta to ./ibdata1...
[00] 2020-06-16 15:12:21 page size for /dev/shm/var_xbackup_apply/backup_2//test/alt_t1.ibd.delta is 8192 bytes
[00] 2020-06-16 15:12:21 Can't create file ./test/alt_t1.ibd
[00] 2020-06-16 15:12:21 error: can't open ./test/alt_t1.ibd
[00] 2020-06-16 15:12:21 Error: xtrabackup_apply_delta(): failed to apply /dev/shm/var_xbackup_apply/backup_2//test/alt_t1.ibd.delta to ./test/alt_t1.ibd.

To reproduce:

git clone https://github.com/MariaDB/randgen --branch mdev22913 rqg-mdev22913
rqg-mdev22913
perl ./runall-trials.pl --basedir=<basedir> --trials=5 --skip-gendata --duration=150 --threads=4 --scenario=MariaBackupIncremental --grammar=mdev22913.yy --vardir=/dev/shm/var_mdev22913

Remember to set the basedir on the command line.
For reproducing, use a 10.5 non-debug build.

The grammar in the test above only contains single query:

query:
  CREATE OR REPLACE TABLE { 't'.$prng->int(1,9) } (b BIT) ENGINE=InnoDB ROW_FORMAT COMPRESSED;

It currently fails every time for me, but there is always room for non-determinism.
The command line above runs the test 5 times, until it fails (in any way). Given the number of problems with incremental backup (or even any backup), it can occasionally fail with a different problem than the one described here. If you want to ignore all other ones and only focus on this one, run with the extra option

--output="Error: xtrabackup_apply_delta"

Then it will ignore other errors and automatically continue running the configured 5 trials until it hits this exact failure.

Logs are attached.

In the community server, this failure was only observed on 10.5 non-debug.

10.4 and 10.5 debug and sometimes 10.4 non-debug fail with MDEV-18589 instead.
10.4 non-debug also fails non-deterministically with

2020-06-16 20:13:22 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
[00] FATAL ERROR: 2020-06-16 20:13:23 mariabackup: innodb_init() returned 11 (Generic error).

(to be filed separately).

10.2 non-debug fails non-deterministically with MDEV-18550.
10.3 non-debug fails non-deterministically with

[00] FATAL ERROR: 2020-06-16 20:03:57 xtrabackup_copy_logfile() failed.



 Comments   
Comment by Marko Mäkelä [ 2022-09-14 ]

I suspected that this might have been fixed by MDEV-28473, but I was able to reproduce the following:

10.5 fb70bb44d096f4117e9454740aa4fa0651a1a69e

# 2022-09-14T10:30:37 [462207] | [00] 2022-09-14 10:30:36 mariabackup: Generating a list of tablespaces
# 2022-09-14T10:30:37 [462207] | 2022-09-14 10:30:36 0 [ERROR] InnoDB: Operating system error number 2 in a file operation.
# 2022-09-14T10:30:37 [462207] | 2022-09-14 10:30:36 0 [ERROR] InnoDB: The error means the system cannot find the path specified.
# 2022-09-14T10:30:37 [462207] | 2022-09-14 10:30:36 0 [ERROR] InnoDB: If you are installing InnoDB, remember that you must create directories yourself, InnoDB does not create them.
# 2022-09-14T10:30:37 [462207] | 2022-09-14 10:30:36 0 [ERROR] InnoDB: Cannot open datafile for read-only: './test/t5.ibd' OS error: 71
# 2022-09-14T10:30:37 [462207] | [00] FATAL ERROR: 2022-09-14 10:30:36 Can't open datafile test/t5
# 2022-09-14T10:30:37 [462207] ----------------------------------
# 2022-09-14T10:30:37 [462287][ERROR] Executor::MySQL::execute: Failed to reconnect after getting STATUS_SERVER_CRASHED
# 2022-09-14T10:30:37 [462287] Executor::MySQL::execute: Query: CREATE /* QNO 6205 CON_ID 9 */ OR REPLACE TABLE t8 (b BIT) ENGINE=InnoDB ROW_FORMAT COMPRESSED failed: 2013 Lost connection to server during query (STATUS_SERVER_CRASHED)

Comment by Marko Mäkelä [ 2022-09-14 ]

I could reproduce a failure of this on 10.5 without the fix of MDEV-18589. After the fix, the test passed:

10.5 fb70bb44d096f4117e9454740aa4fa0651a1a69e + MDEV-18589

# 2022-09-14T12:28:37 [511102] For table `test`.`t9` : status : OK
# 2022-09-14T12:28:37 [511102] #-------------------------#
# 2022-09-14T12:28:37 [511102] #-- Stopping the server --#
# 2022-09-14T12:28:37 [511102] #-------------------------#
# 2022-09-14T12:28:37 [511102] Stopping server on port 19300
# 2022-09-14T12:28:38 [511102] Server has been stopped
# 2022-09-14T12:28:38 [511102] [511102] /mariadb/rqg-mdev22913/runall-new.pl will exit with exit status STATUS_OK (0)
# 2022-09-14T12:28:38 [509225] Trial 5 ended with exit status STATUS_OK (0)
# 2022-09-14T12:28:38 [509225] Exit status STATUS_OK and the list of desired status codes is empty, result will be ignored
# 2022-09-14T12:28:38 [509225] ./runall-trials.pl will exit with exit status FAILED TO REPRODUCE (0)

Generated at Thu Feb 08 09:18:25 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.