[MDEV-17777] RocksDB corruption after out of disk error Created: 2018-11-20  Updated: 2023-04-27

Status: Open
Project: MariaDB Server
Component/s: Storage Engine - RocksDB
Affects Version/s: 10.3.9
Fix Version/s: 10.4

Type: Bug Priority: Major
Reporter: Anders Karlsson Assignee: Sergei Petrunia
Resolution: Unresolved Votes: 1
Labels: None
Environment:

CentOS Linux release 7.5.1804



 Description   

While loading data into RocksDB it ran out of disk space and there was a crash. After this, MyRocks will not initialize. Crash dump:

2018-11-20  0:26:55 0 [ERROR] RocksDB: Error detected in background, Status Code: 5, Status: IO error: No space left on deviceWhile appending to file: ./#rocksdb/014150.sst: No space left on device
2018-11-20  0:26:55 0 [ERROR] RocksDB: BackgroundErrorReason: 0
2018-11-20  0:26:55 0 [ERROR] LibRocksDB:[/home/buildbot/buildbot/build/storage/rocksdb/rocksdb/db/db_impl_compaction_flush.cc:1293] Waiting after background flush error: IO error: No space left on deviceWhile appending to file: ./#rocksdb/014150.sst: No space left on deviceAccumulated background error counts: 1
2018-11-20  0:26:55 11 [ERROR] RocksDB: failed to write to WAL, Status Code: 5, Status: IO error: No space left on deviceWhile appending to file: ./#rocksdb/014150.sst: No space left on device
2018-11-20  0:26:55 11 [ERROR] MyRocks: aborting on WAL write error.
181120  0:26:55 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
 
Server version: 10.3.9-MariaDB
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=3
max_threads=153
thread_count=9
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467396 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x7f2b240008c8
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f2bc0616e70 thread_stack 0x49000

After this crash, restarting MariaDB fails and MyRocks show this error:

2018-11-20  9:49:02 0 [ERROR] RocksDB: Error opening instance, Status Code: 2, Status: Corruption: truncated header
2018-11-20  9:49:02 0 [ERROR] Plugin 'ROCKSDB' init function returned error.
2018-11-20  9:49:02 0 [ERROR] Plugin 'ROCKSDB' registration as a STORAGE ENGINE failed.


Generated at Thu Feb 08 08:39:02 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.