[MDEV-18204] RocksDB failed to start due to problems validating data dictionary against .frm files after restoring incremental backup Created: 2019-01-10  Updated: 2019-02-19  Resolved: 2019-02-18

Status: Closed
Project: MariaDB Server
Component/s: mariabackup, Storage Engine - RocksDB
Affects Version/s: 10.2, 10.3, 10.4
Fix Version/s: 10.4.3, 10.2.23, 10.3.13

Type: Bug Priority: Major
Reporter: Elena Stepanova Assignee: Vladislav Vaintroub
Resolution: Fixed Votes: 0
Labels: None

Attachments: File backup_0.tar.gz     File backup_before_prepare_0.tar.gz     File backup_before_prepare_1.tar.gz     File backup_before_prepare_2.tar.gz     File mbackup_backup_0.log     File mbackup_backup_1.log     File mbackup_backup_2.log     File mbackup_prepare_0.log     File mbackup_prepare_1.log     File mbackup_prepare_2.log     File mbackup_restore.log     File mysql.err     File trial.log    

 Description   

10.4 30da40bb8c3

 
2019-01-10 19:10:10 0 [Note] RocksDB: Got ENOENT when listing column families
 
2019-01-10 19:10:10 0 [Note] RocksDB:   assuming that we're creating a new database
 
2019-01-10 19:10:10 0 [Note] RocksDB: Column Families at start:
 
2019-01-10 19:10:10 0 [Note]   cf=default
 
2019-01-10 19:10:10 0 [Note]     write_buffer_size=67108864
 
2019-01-10 19:10:10 0 [Note]     target_file_size_base=67108864
 
2019-01-10 19:10:11 0 [Note] RocksDB: creating a column family __system__
 
2019-01-10 19:10:11 0 [Note]     write_buffer_size=67108864
 
2019-01-10 19:10:11 0 [Note]     target_file_size_base=67108864
 
2019-01-10 19:10:11 0 [Warning] RocksDB: Schema mismatch - A .frm file exists for table test.t8, but that table is not registered in RocksDB
 
2019-01-10 19:10:11 0 [Warning] RocksDB: Schema mismatch - A .frm file exists for table test.t7, but that table is not registered in RocksDB
 
2019-01-10 19:10:11 0 [Warning] RocksDB: Schema mismatch - A .frm file exists for table test.t2, but that table is not registered in RocksDB
 
2019-01-10 19:10:11 0 [Warning] RocksDB: Schema mismatch - A .frm file exists for table test.t6, but that table is not registered in RocksDB
 
2019-01-10 19:10:11 0 [Warning] RocksDB: Schema mismatch - A .frm file exists for table test.t3, but that table is not registered in RocksDB
 
2019-01-10 19:10:11 0 [Warning] RocksDB: Schema mismatch - A .frm file exists for table test.t1, but that table is not registered in RocksDB
 
2019-01-10 19:10:11 0 [Warning] RocksDB: Schema mismatch - A .frm file exists for table test.t9, but that table is not registered in RocksDB
 
2019-01-10 19:10:11 0 [Warning] RocksDB: Schema mismatch - A .frm file exists for table test.t4, but that table is not registered in RocksDB
 
2019-01-10 19:10:11 0 [Warning] RocksDB: Schema mismatch - A .frm file exists for table test.t5, but that table is not registered in RocksDB
 
2019-01-10 19:10:11 0 [ERROR] RocksDB: Problems validating data dictionary against .frm files, exiting
 
2019-01-10 19:10:11 0 [ERROR] RocksDB: Failed to initialize DDL manager.
 
2019-01-10 19:10:11 0 [ERROR] Plugin 'ROCKSDB' init function returned error.
 
2019-01-10 19:10:11 0 [ERROR] Plugin 'ROCKSDB' registration as a STORAGE ENGINE failed.

Backup dirs are attached.
backup_before_prepare_0 is the initial full backup
backup_before_prepare_1 is the 1st incremental backup
backup_before_prepare_2 is the 1st incremental backup
backup_0 is the prepared backup after all prepare were executed on it.

To reproduce the failure, restore backup_0.tar.gz and start the server with

--plugin-load-add=ha_rocksdb --binlog-format=row

other options can be default.

If you have doubts about the correctness of prepare, you can run it on non-prepared backups.

mariabackup logs from all stages are attached with hopefully self-explanatory names (mbackup*.log).


Notes on the origin of the report

Please be aware it's not necessarily reproducible the same way.

elenst-dev 0ebbe452a69f mariadb-toolbox 17ab77fe268d

 
perl ./runall-new.pl --basedir=/home/travis/server/10.4 --vardir=/home/travis/logs/vardir --duration=350 --threads=4 --seed=1547101908 --reporters=Backtrace,ErrorLog,Deadlock --skip-gendata --gendata-advanced --views --grammar=conf/mariadb/generic-dml.yy --redefine=conf/mariadb/bulk_insert.yy --mysqld=--log_output=FILE --mysqld=--max-statement-time=20 --mysqld=--lock-wait-timeout=10 --mysqld=--loose-innodb-lock-wait-timeout=5 --mysqld=--loose-debug_assert_on_not_freed_memory=0 --filter=/home/travis/mariadb-toolbox/travis/10.4-combo-filter.ff --mysqld=--plugin-load-add=ha_rocksdb --mysqld=--binlog-format=row --engine=RocksDB --scenario=MariaBackupIncremental

The test log is attached as trial.log

On 10.2: https://travis-ci.org/elenst/travis-tests/jobs/481810951
Generated at Thu Feb 08 08:42:17 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.