[MDEV-14872] rocksdb fails to load Created: 2018-01-05  Updated: 2020-05-18

Status: Confirmed
Project: MariaDB Server
Component/s: Storage Engine - RocksDB
Affects Version/s: 10.2.12
Fix Version/s: 10.2

Type: Bug Priority: Major
Reporter: Philip orleans Assignee: Sergei Petrunia
Resolution: Unresolved Votes: 0
Labels: None
Environment:

centos 7

Issue Links:
Relates
relates to MDEV-22609 RocksDB engine fails to upgrade from ... Open

 Description   

After the update some tables that were created previously with rocksdb fail to load and the whole server fails to start. I need to remove rocksdb from the configuration and then the server starts with only innodb.

2018-01-05  8:05:14 139957746784384 [Warning] RocksDB: Schema mismatch - A .frm file exists for table mysql.innodb_index_stats, but that table is not registered in RocksDB
 
2018-01-05  8:05:14 139957746784384 [Warning] RocksDB: Schema mismatch - A .frm file exists for table mysql.innodb_table_stats, but that table is not registered in RocksDB
 
2018-01-05  8:05:14 139957746784384 [Warning] RocksDB: Schema mismatch - A .frm file exists for table lrn.dips, but that table is not registered in RocksDB
 
2018-01-05  8:05:14 139957746784384 [Warning] RocksDB: Schema mismatch - A .frm file exists for table lrn.directdips, but that table is not registered in RocksDB
 
2018-01-05  8:05:14 139957746784384 [ERROR] RocksDB: Problems validating data dictionary against .frm files, exiting
 
2018-01-05  8:05:14 139957746784384 [ERROR] RocksDB: Failed to initialize DDL manager.
 
2018-01-05  8:05:14 139957746784384 [ERROR] Plugin 'ROCKSDB' init function returned error.
 
2018-01-05  8:05:14 139957746784384 [ERROR] Plugin 'ROCKSDB' registration as a STORAGE ENGINE failed.
 
2018-01-05  8:05:15 139957746784384 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
 
2018-01-05  8:05:15 139957746784384 [Note] InnoDB: Uses event mutexes
 
2018-01-05  8:05:15 139957746784384 [Note] InnoDB: Compressed tables use zlib 1.2.7
 
2018-01-05  8:05:15 139957746784384 [Note] InnoDB: Using Linux native AIO
 
2018-01-05  8:05:15 139957746784384 [Note] InnoDB: Number of pools: 1
 
2018-01-05  8:05:15 139957746784384 [Note] InnoDB: Using SSE2 crc32 instructions
 
2018-01-05  8:05:15 139957746784384 [Note] InnoDB: Initializing buffer pool, total size = 8G, instances = 32, chunk size = 128M
 
2018-01-05  8:05:15 139957746784384 [Note] InnoDB: Completed initialization of buffer pool
 
2018-01-05  8:05:15 139944131741440 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
 
2018-01-05  8:05:15 139957746784384 [Note] InnoDB: Highest supported file format is Barracuda.
 
2018-01-05  8:05:16 139957746784384 [Note] InnoDB: 128 out of 128 rollback segments are active.
 
2018-01-05  8:05:16 139957746784384 [Note] InnoDB: Creating shared tablespace for temporary tables
 
2018-01-05  8:05:16 139957746784384 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
 
2018-01-05  8:05:16 139957746784384 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
 
2018-01-05  8:05:16 139957746784384 [Note] InnoDB: 5.7.20 started; log sequence number 2036632093
 
2018-01-05  8:05:16 139957746784384 [Note] Plugin 'FEEDBACK' is disabled.
 
2018-01-05  8:05:16 139957746784384 [ERROR] Unknown/unsupported storage engine: rocksdb
 
2018-01-05  8:05:16 139957746784384 [ERROR] Aborting



 Comments   
Comment by Oren Bissick (Inactive) [ 2018-01-05 ]

i am having this same issue on upgrading to 10.2.12.

Comment by Philip orleans [ 2018-01-05 ]

The isue happens with existing tables of the same engine RocksDB. I converted tables from Innodb and they work fine.

Comment by Oren Bissick (Inactive) [ 2018-01-05 ]

You converted from RocksDB to InnoDB?

Comment by Philip orleans [ 2018-01-06 ]

No, I had a bunch of Innodb tables, and convereted them to Rocksdb. They work. But existing rocksdb tables do not.

Comment by Philip orleans [ 2018-01-06 ]

I need some help urgent. I want to go back to 10.2.11 and convert all my tables to innodb, upgrade and then convert the tables to the new version of rocksdb. How do I revert MaraDB to the version before the current one? I am at a loss about that. This will solve my problem.

Comment by Daniel Black [ 2018-01-06 ]

https://mariadb.com/kb/en/library/yum/ has an example of creating a repository for an older version. Test upgrades and major changes on a test system and then there wouldn't be an emergency. Also don't change mysql.X tables to rocksdb, that's not supported or sane. You did catch that RocksDB is Alpha right https://github.com/MariaDB/server/blob/10.2/storage/rocksdb/ha_rocksdb.h#L1415?

Comment by Philip orleans [ 2018-01-06 ]

I successfully installed veesion 10.2.11 and reverted all tables to innodb, then upgraded
However, on version 10.2.12, the variable
default_storage_engine=rocksdb
is ignored, and the software always sets it to innodb
show variables like '%default_storage_%';
------------------------------+

Variable_name Value

------------------------------+

default_storage_engine InnoDB

but my /etc/my.cnf says =rocksdb
and rocskdb is loaded

Comment by Elena Stepanova [ 2018-01-07 ]

philip_38: why are your mysql.innodb_index_stats tables and such RocksDB? Did you convert them too? What's the point of that, what did you try to achieve?
obissick, you said you are having the same issue, do you also have mysql.innodb_*_stats tables converted to RocksDB?

Comment by Elena Stepanova [ 2018-01-07 ]

The problem with RocksDB not being able to read tables created in previous versions is reproducible.

2018-01-07  2:26:13 140306935647104 [Warning] RocksDB: Schema mismatch - A .frm file exists for table test.t1, but that table is not registered in RocksDB
 
2018-01-07  2:26:13 140306935647104 [ERROR] RocksDB: Problems validating data dictionary against .frm files, exiting
 
2018-01-07  2:26:13 140306935647104 [ERROR] RocksDB: Failed to initialize DDL manager.
 
2018-01-07  2:26:13 140306935647104 [ERROR] Plugin 'ROCKSDB' init function returned error.
 
2018-01-07  2:26:13 140306935647104 [ERROR] Plugin 'ROCKSDB' registration as a STORAGE ENGINE failed.

While RocksDB is still in Alpha stage and compatibility with previous releases are not guaranteed, I am assigning to psergey to check the reason of it and confirm that it was expected (or fix it). If it's expected, it should probably be mentioned in release notes.

However, for those who got a problem with InnoDB stat tables – I don't expect anything good can come out of converting them into RocksDB; unless you know exactly why you are doing it, please don't.

Comment by Andrew Sun [ 2020-05-17 ]

I attempted to upgrade MariaDB (and its MyRocks plugin) from 10.4.12 to 10.4.13, and I ran into this very same issue:

mariadb-ssd_1           | 2020-05-17 19:29:46 0 [Warning] RocksDB: Schema mismatch - A .frm file exists for table [redacted], but that table is not registered in RocksDB
 
mariadb-ssd_1           | 2020-05-17 19:29:46 0 [Warning] RocksDB: Schema mismatch - A .frm file exists for table [redacted], but that table is not registered in RocksDB
 
mariadb-ssd_1           | 2020-05-17 19:29:46 0 [Warning] RocksDB: Schema mismatch - A .frm file exists for table [redacted], but that table is not registered in RocksDB
 
mariadb-ssd_1           | 2020-05-17 19:29:46 0 [Warning] RocksDB: Schema mismatch - A .frm file exists for table [redacted], but that table is not registered in RocksDB
 
mariadb-ssd_1           | 2020-05-17 19:29:46 0 [Warning] RocksDB: Schema mismatch - A .frm file exists for table [redacted], but that table is not registered in RocksDB
 
mariadb-ssd_1           | 2020-05-17 19:29:46 0 [Warning] RocksDB: Schema mismatch - A .frm file exists for table [redacted], but that table is not registered in RocksDB
 
mariadb-ssd_1           | 2020-05-17 19:29:46 0 [Warning] RocksDB: Schema mismatch - A .frm file exists for table [redacted], but that table is not registered in RocksDB
 
mariadb-ssd_1           | 2020-05-17 19:29:46 0 [Warning] RocksDB: Schema mismatch - A .frm file exists for table [redacted], but that table is not registered in RocksDB
 
mariadb-ssd_1           | 2020-05-17 19:29:46 0 [Warning] RocksDB: Schema mismatch - A .frm file exists for table [redacted], but that table is not registered in RocksDB
 
mariadb-ssd_1           | 2020-05-17 19:29:46 0 [Warning] RocksDB: Schema mismatch - A .frm file exists for table [redacted], but that table is not registered in RocksDB
 
mariadb-ssd_1           | 2020-05-17 19:29:46 0 [ERROR] RocksDB: Problems validating data dictionary against .frm files, exiting
 
mariadb-ssd_1           | 2020-05-17 19:29:46 0 [ERROR] RocksDB: Failed to initialize DDL manager.
 
mariadb-ssd_1           | 2020-05-17 19:29:46 0 [ERROR] Plugin 'ROCKSDB' init function returned error.
 
mariadb-ssd_1           | 2020-05-17 19:29:46 0 [ERROR] Plugin 'ROCKSDB' registration as a STORAGE ENGINE failed.

Setting rocksdb_validate_tables=0 as a possible workaround results in this:

mariadb-ssd_1           | 2020-05-17 19:31:52 0 [ERROR] RocksDB: Table_store load error, Status Code: 2, Status: Corruption: ZSTD not supported or corrupted ZSTD compressed block contents
 
mariadb-ssd_1           | 2020-05-17 19:31:52 0 [ERROR] RocksDB: Failed to initialize DDL manager.
 
mariadb-ssd_1           | 2020-05-17 19:31:52 0 [ERROR] Plugin 'ROCKSDB' init function returned error.
 
mariadb-ssd_1           | 2020-05-17 19:31:52 0 [ERROR] Plugin 'ROCKSDB' registration as a STORAGE ENGINE failed.

This was all done on the official MariaDB Docker image (Ubuntu Bionic) with the mariadb-plugin-rocksdb also installed.

It may also be relevant that I have five other column families in MyRocks besides the default column family, each with their own RocksDB options.

Edit: decided to create MDEV-22609 in a separate ticket

Generated at Thu Feb 08 08:16:56 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.