Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Not a Bug
-
None
-
None
-
AWS RDS mariadb 10.11.10
-
Not for Release Notes
Description
A snippet from startup (error) log of one mariadb RDS instance in AWS:
2025-05-15 12:24:09 0 [Note] Starting MariaDB 10.11.10-MariaDB-log source revision server_uid uWFwUsR0pESQKIMmx955OhKBzJI= as process 598 |
|
2025-05-15 12:24:09 0 [Note] RocksDB: 2 column families found |
|
2025-05-15 12:24:09 0 [Note] RocksDB: Column Families at start: |
|
2025-05-15 12:24:09 0 [Note] cf=default |
|
2025-05-15 12:24:09 0 [Note] write_buffer_size=67108864 |
|
2025-05-15 12:24:09 0 [Note] target_file_size_base=67108864 |
|
2025-05-15 12:24:09 0 [Note] cf=__system__ |
|
2025-05-15 12:24:09 0 [Note] write_buffer_size=67108864 |
|
2025-05-15 12:24:09 0 [Note] target_file_size_base=67108864 |
|
2025-05-15 12:48:12 0 [Note] RocksDB: Table_store: loaded DDL data for 0 tables |
|
2025-05-15 12:48:12 0 [Note] RocksDB: global statistics using get_sched_indexer_t indexer |
|
2025-05-15 12:48:12 0 [Note] MyRocks storage engine plugin has been successfully initialized.
|
RocksDB initialization takes 24 minutes in this case.
The RDS contains many schemas (6000+) with many tables (each schema has 250+ tables).
None of the tables use MyRocks plugin.
There is no way to disable the MyRocks plugin and it is making the reboots slow.
What can be done?
AWS mariadb support's answer about the slow reboot problem:
The prolonged delay during the failover is due to the Atomic DDL functionality introduced in MariaDB 10.6. From MariaDB 10.6.1, we have improved readability for DDL (Data Definition Language) operations to make most of them atomic, and the rest crash-safe, even if the server crashes in the middle of an operation.The design of Atomic/Crash-safe DDL (
MDEV-17567) allows it to work with all storage engines.>> Please find the below link for your reference:
[+] https://mariadb.com/kb/en/atomic-ddl/During crash recovery, MyRocks storage engine is enabled by default in RDS MariaDB and it scans for RocksDB tables' FRM files to validate that the stored metadata matches what RocksDB has in it cache.
Kindly note that, this issue is not related to RDS rather it is an expected behaviour of MariaDB. FRM files scanning may be skipped with rocksdb_validate_tables=2 parameter, however RDS MariaDB does not allow modification of this parameter at the moment which has "default rocksdb_validate_tables=1".
While we currently don't have a direct solution to disable this parameter, I want to assure you that our internal team is actively working on a fix. Though implementing the solution may take some time, we are committed to addressing this issue completely. In the meantime, you may experience longer reboot times . We understand this may impact your operations, and we assure you that we are currently prioritizing this for resolution. We appreciate your patience during this process .