[MDEV-24521] Partitioned InnoDB table appears corrupted after restoring full backup Created: 2021-01-04  Updated: 2023-08-16

Status: Open
Project: MariaDB Server
Component/s: mariabackup, Partitioning
Affects Version/s: 10.3, 10.4
Fix Version/s: 10.4

Type: Bug Priority: Major
Reporter: Elena Stepanova Assignee: Marko Mäkelä
Resolution: Unresolved Votes: 0
Labels: None

Attachments: File mbackup_backup_6.log    

 Description   

The original test runs random DML/DDL in a few threads and takes full backups from the server periodically. Then it stops the flow and the server, prepares the backups one by one, each time starting the server on the restored backup and performing data health checks. At some points, an error is revealed:

10.3 e59c1ce

 
MariaDB [test]> check table t1 extended;
 
+---------+-------+----------+-------------------------------------------------------------------+
 
| Table   | Op    | Msg_type | Msg_text                                                          |
 
+---------+-------+----------+-------------------------------------------------------------------+
 
| test.t1 | check | error    | Partition ver_p1 returned error                                   |
 
| test.t1 | check | Warning  | InnoDB: Index 'FTS_DOC_ID_INDEX' contains 0 entries, should be 3. |
 
| test.t1 | check | error    | Corrupt                                                           |
 
+---------+-------+----------+-------------------------------------------------------------------+


2021-01-03 21:09:25 8 [ERROR] InnoDB: Flagged corruption of `FTS_DOC_ID_INDEX` in table `test`.`t1` /* Partition `ver_p1` */ in CHECK TABLE; Wrong count

In the server log, it is the first and only error occurring during the test run.
mariabackup backup log does not show any errors (attached as mbackup_backup_6.log).

The issue is sporadic in the sense of creating the initial problematic; but once such a backup is created, the failure upon prepare => restart => check is reliably reproducible.

Similar failures have been observed in tests on at least 10.2, 10.3 and 10.4. I'm setting affect version to 10.3 and 10.4, because the data provided within this ticket relates to a system-versioned partitioned table, so it's not directly applicable to 10.2.

The backups, logs and rr profiles for each start of server and mariabackup are available.

For a reference, here is the test I used to produce the data:

randgen elenst-dev bf2f4069

 
perl ./runall-trials.pl --trials=10 --duration=350 --threads=4 --seed=1609549322 --reporters=Backtrace,ErrorLog,Deadlock --skip-gendata --gendata-advanced --engine=InnoDB --views --grammar=conf/mariadb/generic-dml.yy --redefine=conf/mariadb/bulk_insert.yy --filter=conf/mariadb/10.4-combo-filter.ff --mysqld=--log_output=FILE --mysqld=--max-statement-time=20 --mysqld=--lock-wait-timeout=10 --mysqld=--loose-innodb-lock-wait-timeout=5 --mysqld=--loose-debug_assert_on_not_freed_memory=0 --mysqld=--innodb-compression-algorithm=zlib --mysqld=--innodb-page-size=16K --mysqld=--innodb-encrypt-tables --mysqld=--innodb-encrypt-log --mysqld=--innodb-encryption-threads=4 --mysqld=--aria-encrypt-tables=1 --mysqld=--encrypt-tmp-disk-tables=1 --mysqld=--encrypt-binlog --mysqld=--file-key-management --mysqld=--file-key-management-filename=`pwd`/../10.3/mysql-test/std_data/keys.txt --mysqld=--plugin-load-add=file_key_management --scenario=MariaBackupFull --redefine=conf/mariadb/alter_table.yy --redefine=conf/mariadb/modules/admin.yy --redefine=conf/mariadb/versioning.yy --redefine=conf/mariadb/sequences.yy --basedir1=`pwd`/../10.3 --mysqld=--innodb-buffer-pool-size=512M --vardir1=`pwd`/../var --mtr-build-thread=55 --rr



 Comments   
Comment by Elena Stepanova [ 2021-01-23 ]

The same or something similar also happens with (at least) non-partitioned system-versioned tables, like this:

10.3 e59c1cef

 
2020-12-31  1:46:54 8 [ERROR] InnoDB: Flagged corruption of `FTS_DOC_ID_INDEX` in table `test`.`alt_t3` in CHECK TABLE; Wrong count
 
 
 
# 2020-12-31T01:46:54 [79925][ERROR] For InnoDB SYSTEM VERSIONED ROW_FORMAT=Dynamic `test`.`alt_t3` : error : Corrupt

For now I'll assume it to be the same problem.

Comment by Marko Mäkelä [ 2023-08-16 ]

System-versioned tables do not play together with FULLTEXT INDEX until MDEV-25004 and MDEV-30528 are fixed.

I am having trouble executing the script against the current 10.4. I did not find any boot.sql and I do not know which directory var should point to.

Generated at Thu Feb 08 09:30:37 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.