[MDEV-13975] [out of disk space assert] I was uploading 100 million records to a rocsdb table and Mariadb crashed Created: 2017-10-01  Updated: 2017-12-12

Status: Open
Project: MariaDB Server
Component/s: Storage Engine - RocksDB
Affects Version/s: 10.2
Fix Version/s: 10.2

Type: Bug Priority: Major
Reporter: Philip orleans Assignee: Sergei Petrunia
Resolution: Unresolved Votes: 0
Labels: rocksdb

Issue Links:
Relates
relates to MDEV-14047 "Too many open files" Confirmed

 Description   

The drive was out of space, but it should not crash/

2017-10-01 13:25:08 139821712402176 [ERROR] LibRocksDB:[/home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.9/storage/rocksdb/rocksdb/db/db_impl_compaction_flush.cc:1257] Waiting after background flush error: IO error: Error allocating space to file : ./.rocksdb/004708.sstError : No space left on deviceAccumulated background error counts: 1
2017-10-01 13:25:08 139821934552832 [ERROR] RocksDB: failed to write to WAL, Status Code: 5, Status: IO error: Error allocating space to file : ./.rocksdb/004708.sstError : No space left on device
2017-10-01 13:25:08 139821934552832 [ERROR] MyRocks: aborting on WAL write error.
2017-10-01 13:25:08 139821934053120 [ERROR] RocksDB: failed to write to WAL, Status Code: 5, Status: IO error: Error allocating space to file : ./.rocksdb/004708.sstError : No space left on device
2017-10-01 13:25:08 139821934053120 [ERROR] MyRocks: aborting on WAL write error.
171001 13:25:08 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
2017-10-01 13:25:17 140164155345024 [Note] RocksDB: 2 column families found
2017-10-01 13:25:17 140164155345024 [Note] RocksDB: Column Families at start:
2017-10-01 13:25:17 140164155345024 [Note]   cf=default
2017-10-01 13:25:17 140164155345024 [Note]     write_buffer_size=134217728
2017-10-01 13:25:17 140164155345024 [Note]     target_file_size_base=67108864
2017-10-01 13:25:17 140164155345024 [Note]   cf=__system__
2017-10-01 13:25:17 140164155345024 [Note]     write_buffer_size=134217728
2017-10-01 13:25:17 140164155345024 [Note]     target_file_size_base=67108864
mysqld: /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.9/storage/rocksdb/rocksdb/env/posix_logger.h:153: virtual void rocksdb::PosixLogger::Logv(const char*, __va_list_tag*): Assertion `sz == write_size' failed.
171001 13:25:17 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
Server version: 10.2.9-MariaDB
key_buffer_size=16384
read_buffer_size=262144
max_used_connections=0
max_threads=65537
thread_count=0
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 553658729 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x3c000
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5560a1f043ee]
/usr/sbin/mysqld(handle_fatal_signal+0x30d)[0x5560a19816fd]
/lib64/libpthread.so.0(+0xf5e0)[0x7f7a8271d5e0]
/lib64/libc.so.6(gsignal+0x37)[0x7f7a80c2a1f7]
/lib64/libc.so.6(abort+0x148)[0x7f7a80c2b8e8]
/lib64/libc.so.6(+0x2e266)[0x7f7a80c23266]
/lib64/libc.so.6(+0x2e312)[0x7f7a80c23312]
/usr/lib64/mysql/plugin/ha_rocksdb.so(_ZN7rocksdb11PosixLogger4LogvEPKcP13__va_list_tag+0x3e4)[0x7f7a73ca86f4]
/usr/lib64/mysql/plugin/ha_rocksdb.so(_ZN7rocksdb6Logger4LogvENS_12In[root@demo mysql]# 


Generated at Thu Feb 08 08:09:51 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.