[MDEV-13609] [OOM because of in-mem DDL] Converting InnoDB Table to MyRocks Storage engine crashes server Created: 2017-08-21  Updated: 2018-11-27

Status: Open
Project: MariaDB Server
Component/s: Storage Engine - RocksDB
Affects Version/s: 10.2.7
Fix Version/s: 10.4

Type: Bug Priority: Major
Reporter: Thomas Boyd (Inactive) Assignee: Sergei Petrunia
Resolution: Unresolved Votes: 2
Labels: None

Issue Links:
Relates
relates to MDEV-13599 rocksdb.add_index_inplace_sstfilewrit... Closed

 Description   

Using alter table command to convert large inno table (100GB+) to rocksdb crashes server.



 Comments   
Comment by Daniel Black [ 2017-08-22 ]

Do you have a backtrace from the mysql log. The original table format (SHOW CREATE TABLE) and the ALTER TABLE statement?

What is your server configuration?

How long after issuing the ALTER TABLE statement does the crash occur?

Comment by Sergei Petrunia [ 2017-08-22 ]

The symptoms (posted privately) look a bit similar to MDEV-13599.

Comment by Sergei Petrunia [ 2017-08-24 ]

It turns out, when MyRocks uses default settings, it will try to do ALTER TABLE .... ENGINE=ROCKSDB as one big transaction. If the table is sufficiently big, mysqld will die with an OOM error.

There is a workaround:

set @@rocksdb_bulk_load_size= 100*1000;
set @@rocksdb_commit_in_the_middle= 1;
ALTER TABLE .... ENGINE=ROCKSDB;

This is not a pleasant user experience though, so I have filed https://github.com/facebook/mysql-5.6/issues/692 .

Comment by Sergei Petrunia [ 2018-05-11 ]

In current MariaDB (and FB's upstream), the crash is avoided by having @@rocksdb_max_row_locks to be set sufficiently low so that one hits that limit before the server consumes so much memory that it is killed by the OOM killer.

Talking to the upstream team, they have acknowledged that task #692 is good to have and doable but there hasn't been any progress on it.

rocksdb_max_row_locks error looks like so:

MariaDB [test]> create table ten_m (pk int primary key, a int) engine=innodb;
Query OK, 0 rows affected (0.01 sec)
 
MariaDB [test]> insert into ten_m select A.a + 1000*1000 * B.a, 1234 from one_m A, ten B;
Query OK, 10000000 rows affected (24.17 sec)
Records: 10000000  Duplicates: 0  Warnings: 0
 
MariaDB [test]> alter table ten_m engine=rocksdb;
ERROR 4067 (HY000): Status error 10 received from RocksDB: Operation aborted: Failed to acquire lock due to max_num_locks limit

The default settings are:

MariaDB [test]> select @@rocksdb_max_row_locks;        
+-------------------------+
| @@rocksdb_max_row_locks |
+-------------------------+
|                 1048576 |
+-------------------------+
1 row in set (0.00 sec)

Comment by Juan Telleria [ 2018-11-27 ]

A workaround could also be:

SET STATEMENT 
     rocksdb_bulk_load = 1
FOR
     ALTER TABLE .... ENGINE=ROCKSDB;

Generated at Thu Feb 08 08:06:56 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.