[MDEV-13609] [OOM because of in-mem DDL] Converting InnoDB Table to MyRocks Storage engine crashes server Created: 2017-08-21 Updated: 2018-11-27 |
|
| Status: | Open |
| Project: | MariaDB Server |
| Component/s: | Storage Engine - RocksDB |
| Affects Version/s: | 10.2.7 |
| Fix Version/s: | 10.4 |
| Type: | Bug | Priority: | Major |
| Reporter: | Thomas Boyd (Inactive) | Assignee: | Sergei Petrunia |
| Resolution: | Unresolved | Votes: | 2 |
| Labels: | None | ||
| Issue Links: |
|
||||||||
| Description |
|
Using alter table command to convert large inno table (100GB+) to rocksdb crashes server. |
| Comments |
| Comment by Daniel Black [ 2017-08-22 ] | ||||||||||||||||
|
Do you have a backtrace from the mysql log. The original table format (SHOW CREATE TABLE) and the ALTER TABLE statement? What is your server configuration? How long after issuing the ALTER TABLE statement does the crash occur? | ||||||||||||||||
| Comment by Sergei Petrunia [ 2017-08-22 ] | ||||||||||||||||
|
The symptoms (posted privately) look a bit similar to | ||||||||||||||||
| Comment by Sergei Petrunia [ 2017-08-24 ] | ||||||||||||||||
|
It turns out, when MyRocks uses default settings, it will try to do ALTER TABLE .... ENGINE=ROCKSDB as one big transaction. If the table is sufficiently big, mysqld will die with an OOM error. There is a workaround:
This is not a pleasant user experience though, so I have filed https://github.com/facebook/mysql-5.6/issues/692 . | ||||||||||||||||
| Comment by Sergei Petrunia [ 2018-05-11 ] | ||||||||||||||||
|
In current MariaDB (and FB's upstream), the crash is avoided by having @@rocksdb_max_row_locks to be set sufficiently low so that one hits that limit before the server consumes so much memory that it is killed by the OOM killer. Talking to the upstream team, they have acknowledged that task #692 is good to have and doable but there hasn't been any progress on it. rocksdb_max_row_locks error looks like so:
The default settings are:
| ||||||||||||||||
| Comment by Juan Telleria [ 2018-11-27 ] | ||||||||||||||||
|
A workaround could also be:
|