mariadb-backup --prepare: Frequent [ERROR] InnoDB: OPT_PAGE_CHECKSUM mismatch on

origin/10.6 bb2e125d076d9475cbe04fc8c24fbaee39d76bda 2024-04-18T07:14:56+03:00

with and without with ibuf.patch.diff applied

Scenario:

1. Start the DB server and generate some initial data

2. Several sessions run some DML mix concurrent.

3. During 2. is ongoing run loops consisting of

    3.1 mariadb-backup --backup  to some location A

    3.2 mariadb-backup --prepare on location A

    3.3 Start a DB server on location A and check the integrity of the data

3.2 fails in one of these loops fails with

[rr 1609519 2196]2024-04-18 13:21:58 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=56848,56848

[rr 1609519 2358]2024-04-18 13:21:58 0 [Note] InnoDB: Opened 16 undo tablespaces

[rr 1609519 2404]2024-04-18 13:21:58 0 [Note] InnoDB: To recover: 104 pages

[rr 1609519 5067]2024-04-18 13:21:59 0 [ERROR] InnoDB: OPT_PAGE_CHECKSUM mismatch on [page id: space=21, page number=17]

[rr 1609519 5071]2024-04-18 13:21:59 0 [ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.

[rr 1609519 5075]2024-04-18 13:21:59 0 [Note] InnoDB: Set innodb_force_recovery=1 to ignore corrupted pages.

[rr 1609519 5079]2024-04-18 13:21:59 0 [ERROR] InnoDB: Unable to apply log to corrupted page [page id: space=21, page number=17]

[rr 1609519 5143]2024-04-18 13:21:59 0 [ERROR] InnoDB: OPT_PAGE_CHECKSUM mismatch on [page id: space=22, page number=3]

[rr 1609519 5147]2024-04-18 13:21:59 0 [ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.

[rr 1609519 5151]2024-04-18 13:21:59 0 [Note] InnoDB: Set innodb_force_recovery=1 to ignore corrupted pages.

[rr 1609519 5155]2024-04-18 13:21:59 0 [ERROR] InnoDB: Unable to apply log to corrupted page [page id: space=22, page number=3]

[rr 1609519 5384]2024-04-18 13:21:59 0 [ERROR] InnoDB: OPT_PAGE_CHECKSUM mismatch on [page id: space=23, page number=3]

[rr 1609519 5388]2024-04-18 13:21:59 0 [ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.

[rr 1609519 5392]2024-04-18 13:21:59 0 [Note] InnoDB: Set innodb_force_recovery=1 to ignore corrupted pages.

[rr 1609519 5396]2024-04-18 13:21:59 0 [ERROR] InnoDB: Unable to apply log to corrupted page [page id: space=23, page number=3]

[rr 1609519 6086]2024-04-18 13:22:00 0 [ERROR] InnoDB: Plugin initialization aborted at srv0start.cc[1532] with error Data structure corruption

[rr 1609519 6090][00] FATAL ERROR: 2024-04-18 13:22:00 mariabackup: innodb_init() returned 37 (Data structure corruption).

# Source DB server

sdp:/data1/results/1713460012/MB-1937$ _RR_TRACE_DIR=./1/rr rr replay --mark-stdio

# mariadb-backup --backup

sdp:/data1/results/1713460012/MB-1937$ _RR_TRACE_DIR=./1_clone/rr rr replay --mark-stdio mariadb-backup-0

# File backup of the location A made after executing mariadb-backup --backup

sdp:/data1/results/1713460012/MB-1937/1_clone/fbackup

# mariadb-backup --prepare

sdp:/data1/results/1713460012/MB-1937$ _RR_TRACE_DIR=./1_clone/rr rr replay --mark-stdio mariadb-backup-1

1298 tests with that scenario were executed and 84 of them failed with that error.

Hence I set the priority to "Blocker".

The rr traces above belong to some RQG test which fiddles with partitioned and non partitioned tables. Other tests which do not use partitioned tables show the same problem.

# git clone https://github.com/mleich1/rqg --branch <pick the right branch> RQG

#

# GIT_SHOW: HEAD -> master, origin/master, origin/HEAD ea2185ca59d4adbbfcd52987437aef1eac0aedad 2024-04-17T12:52:30+02:00

# rqg.pl  : Version 4.5.0 (2023-12)

#

# $RQG_HOME/rqg.pl \

# --gendata=conf/engines/many_indexes.zz \

# --max_gd_duration=900 \

# --grammar=conf/engines/many_indexes.yy \

# --reporters=Mariabackup_linux \

# --mysqld=--loose-innodb-log-file-size=200M \

# --mysqld=--loose-innodb_lock_schedule_algorithm=fcfs \

# --mysqld=--loose-idle_write_transaction_timeout=0 \

# --mysqld=--loose-idle_transaction_timeout=0 \

# --mysqld=--loose-idle_readonly_transaction_timeout=0 \

# --mysqld=--connect_timeout=60 \

# --mysqld=--interactive_timeout=28800 \

# --mysqld=--slave_net_timeout=60 \

# --mysqld=--net_read_timeout=30 \

# --mysqld=--net_write_timeout=60 \

# --mysqld=--loose-table_lock_wait_timeout=50 \

# --mysqld=--wait_timeout=28800 \

# --no_mask \

# --queries=10000000 \

# --seed=random \

# --reporters=None \

# --reporters=ErrorLog \

# --reporters=Deadlock \

# --validators=None \

# --mysqld=--log_output=none \

# --mysqld=--log_bin_trust_function_creators=1 \

# --mysqld=--loose-debug_assert_on_not_freed_memory=0 \

# --engine=InnoDB \

# --restart_timeout=240 \

# --mysqld=--plugin-load-add=file_key_management.so \

# --mysqld=--loose-file-key-management-filename=$RQG_HOME/conf/mariadb/encryption_keys.txt \

# --mysqld=--plugin-load-add=provider_lzo.so \

# --mysqld=--plugin-load-add=provider_bzip2.so \

# --mysqld=--plugin-load-add=provider_lzma.so \

# --mysqld=--plugin-load-add=provider_snappy.so \

# --mysqld=--plugin-load-add=provider_lz4.so \

# --mysqld=--loose-innodb_compression_level=1 \

# --duration=300 \

# --mysqld=--loose-innodb_fatal_semaphore_wait_threshold=300 \

# --mysqld=--lock-wait-timeout=86400 \

# --mysqld=--innodb-lock-wait-timeout=50 \

# --mysqld=--loose-innodb_fast_shutdown=1 \

# --mysqld=--sql_mode=traditional \

# --mysqld=--innodb_file_per_table=1 \

# --mysqld=--loose-innodb_read_only_compressed=OFF \

# --mysqld=--innodb_stats_persistent=off \

# --mysqld=--innodb_adaptive_hash_index=off \

# --mysqld=--innodb_random_read_ahead=OFF \

# --mysqld=--log-bin \

# --mysqld=--sync-binlog=1 \

# --mysqld=--loose-innodb_evict_tables_on_commit_debug=off \

# --threads=2 \

# --rr=Extended \

# --rr_options=--chaos --wait \

# --mysqld=--innodb_undo_log_truncate=OFF \

# --mysqld=--innodb_undo_tablespaces=16 \

# --mysqld=--innodb_rollback_on_timeout=OFF \

# --mysqld=--innodb_page_size=64K \

# --mysqld=--innodb-buffer-pool-size=256M \

# <some local settings>