[MDEV-30194] RocksDB, MariaDB crash Created: 2022-12-10  Updated: 2022-12-14

Status: Open
Project: MariaDB Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major
Reporter: poiuty Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None


 Description    

Dec 10 00:15:46 main mariadbd[1435]: 2022-12-10  0:15:46 0 [ERROR] RocksDB: Error detected in background, Status Code: 5, Status: IO error: No space left on deviceWhile appending to file: ./#rocksdb/581104.sst: No space left on device
 
Dec 10 00:15:46 main mariadbd[1435]: 2022-12-10  0:15:46 0 [ERROR] RocksDB: BackgroundErrorReason: 0
 
Dec 10 00:15:46 main mariadbd[1435]: 2022-12-10  0:15:46 0 [ERROR] LibRocksDB:[db_impl/db_impl_compaction_flush.cc:2249] Waiting after background flush error: IO error: No space left on deviceWhile appending to file: ./#rocksdb/581104.sst: No space left on deviceAccumulated background error counts: 1
 
Dec 10 00:15:46 main mariadbd[1435]: 2022-12-10  0:15:46 1156551 [ERROR] RocksDB: Error detected in background, Status Code: 5, Status: IO error: No space left on deviceWhile appending to file: ./#rocksdb/581104.sst: No space left on device
 
Dec 10 00:15:46 main mariadbd[1435]: 2022-12-10  0:15:46 1156551 [ERROR] RocksDB: BackgroundErrorReason: 2
 
Dec 10 00:15:46 main mariadbd[1435]: 2022-12-10  0:15:46 1156551 [ERROR] RocksDB: failed to write to WAL, Status Code: 5, Status: IO error: No space left on deviceWhile appending to file: ./#rocksdb/581104.sst: No space left on device
 
Dec 10 00:15:46 main mariadbd[1435]: 2022-12-10  0:15:46 1156551 [ERROR] MyRocks: aborting on WAL write error.
 
Dec 10 00:15:46 main mariadbd[1435]: 221210  0:15:46 [ERROR] mysqld got signal 6 ;
 
Dec 10 00:15:46 main mariadbd[1435]: This could be because you hit a bug. It is also possible that this binary
 
Dec 10 00:15:46 main mariadbd[1435]: or one of the libraries it was linked against is corrupt, improperly built,
 
Dec 10 00:15:46 main mariadbd[1435]: or misconfigured. This error can also be caused by malfunctioning hardware.
 
Dec 10 00:15:46 main mariadbd[1435]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
Dec 10 00:15:46 main mariadbd[1435]: We will try our best to scrape up some info that will hopefully help
 
Dec 10 00:15:46 main mariadbd[1435]: diagnose the problem, but since we have already crashed,
 
Dec 10 00:15:46 main mariadbd[1435]: something is definitely wrong and this may fail.
 
Dec 10 00:15:46 main mariadbd[1435]: Server version: 10.7.7-MariaDB-1:10.7.7+maria~deb11-log
 
Dec 10 00:15:46 main mariadbd[1435]: key_buffer_size=33554432
 
Dec 10 00:15:46 main mariadbd[1435]: read_buffer_size=131072
 
Dec 10 00:15:46 main mariadbd[1435]: max_used_connections=27
 
Dec 10 00:15:46 main mariadbd[1435]: max_threads=102
 
Dec 10 00:15:46 main mariadbd[1435]: thread_count=27
 
Dec 10 00:15:46 main mariadbd[1435]: It is possible that mysqld could use up to
 
Dec 10 00:15:46 main mariadbd[1435]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 257379 K  bytes of memory
 
Dec 10 00:15:46 main mariadbd[1435]: Hope that's ok; if not, decrease some variables in the equation.
 
Dec 10 00:15:46 main mariadbd[1435]: Thread pointer: 0x7febb4000c58
 
Dec 10 00:15:46 main mariadbd[1435]: Attempting backtrace. You can use the following information to find out
 
Dec 10 00:15:46 main mariadbd[1435]: where mysqld died. If you see no messages after this, something went
 
Dec 10 00:15:46 main mariadbd[1435]: terribly wrong...
 
Dec 10 00:15:46 main mariadbd[1435]: stack_bottom = 0x7ff4a00d8d18 thread_stack 0x49000
 
Dec 10 00:15:46 main mariadbd[1435]: ??:0(my_print_stacktrace)[0x55c48e11dc9e]
 
Dec 10 00:15:46 main mariadbd[1435]: ??:0(handle_fatal_signal)[0x55c48dc04145]
 
Dec 10 00:15:46 main mariadbd[1435]: ??:0(__restore_rt)[0x7ff4c2a47140]
 
Dec 10 00:15:46 main mariadbd[1435]: ??:0(gsignal)[0x7ff4c257ece1]
 
Dec 10 00:15:46 main mariadbd[1435]: ??:0(abort)[0x7ff4c2568537]
 
Dec 10 00:15:46 main mariadbd[1435]: ??:0(myrocks::rdb_handle_io_error(rocksdb::Status, myrocks::RDB_IO_ERROR_TYPE))[0x7ff4bbb02d8d]
 
Dec 10 00:15:46 main mariadbd[1435]: ??:0(myrocks::Rdb_transaction_impl::commit_no_binlog())[0x7ff4bbb2d812]
 
Dec 10 00:15:46 main mariadbd[1435]: ??:0(myrocks::timeout_message(char const*, char const*, char const*))[0x7ff4bbaffd68]
 
Dec 10 00:15:46 main mariadbd[1435]: ??:0(ha_check_and_coalesce_trx_read_only(THD*, Ha_trx_info*, bool))[0x55c48dc07a29]
 
Dec 10 00:15:46 main mariadbd[1435]: ??:0(ha_commit_trans(THD*, bool))[0x55c48dc14b36]
 
Dec 10 00:15:46 main mariadbd[1435]: ??:0(trans_commit(THD*))[0x55c48daf7183]
 
Dec 10 00:15:46 main mariadbd[1435]: ??:0(mysql_execute_command(THD*, bool))[0x55c48d9c535d]
 
Dec 10 00:15:46 main mariadbd[1435]: ??:0(mysql_parse(THD*, char*, unsigned int, Parser_state*))[0x55c48d9c8197]
 
Dec 10 00:15:46 main mariadbd[1435]: ??:0(dispatch_command(enum_server_command, THD*, char*, unsigned int, bool))[0x55c48d9caa5d]
 
Dec 10 00:15:46 main mariadbd[1435]: ??:0(do_command(THD*, bool))[0x55c48d9cc5c8]
 
Dec 10 00:15:46 main mariadbd[1435]: ??:0(do_handle_one_connection(CONNECT*, bool))[0x55c48dae71ef]
 
Dec 10 00:15:46 main mariadbd[1435]: ??:0(handle_one_connection)[0x55c48dae753d]
 
Dec 10 00:15:46 main mariadbd[1435]: ??:0(MyCTX_nopad::finish(unsigned char*, unsigned int*))[0x55c48de24c22]
 
Dec 10 00:15:46 main mariadbd[1435]: ??:0(start_thread)[0x7ff4c2a3bea7]
 
Dec 10 00:15:46 main mariadbd[1435]: ??:0(clone)[0x7ff4c2642a2f]
 
Dec 10 00:15:46 main mariadbd[1435]: Trying to get some variables.
 
Dec 10 00:15:46 main mariadbd[1435]: Some pointers may be invalid and cause the dump to abort.
 
Dec 10 00:15:46 main mariadbd[1435]: Query (0x7febb4010920): COMMIT
 
Dec 10 00:15:46 main mariadbd[1435]: Connection ID (thread ID): 1156551
 
Dec 10 00:15:46 main mariadbd[1435]: Status: NOT_KILLED
 
Dec 10 00:15:46 main mariadbd[1435]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off
 
Dec 10 00:15:46 main mariadbd[1435]: The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
 
Dec 10 00:15:46 main mariadbd[1435]: information that should help you find out what is causing the crash.
 
Dec 10 00:15:46 main mariadbd[1435]: Writing a core file...
 
Dec 10 00:15:46 main mariadbd[1435]: Working directory at /var/lib/mysql
 
Dec 10 00:15:46 main mariadbd[1435]: Resource Limits:
 
Dec 10 00:15:46 main mariadbd[1435]: Limit                     Soft Limit           Hard Limit           Units
 
Dec 10 00:15:46 main mariadbd[1435]: Max cpu time              unlimited            unlimited            seconds
 
Dec 10 00:15:46 main mariadbd[1435]: Max file size             unlimited            unlimited            bytes
 
Dec 10 00:15:46 main mariadbd[1435]: Max data size             unlimited            unlimited            bytes
 
Dec 10 00:15:46 main mariadbd[1435]: Max stack size            8388608              unlimited            bytes
 
Dec 10 00:15:46 main mariadbd[1435]: Max core file size        0                    unlimited            bytes
 
Dec 10 00:15:46 main mariadbd[1435]: Max resident set          unlimited            unlimited            bytes
 
Dec 10 00:15:46 main mariadbd[1435]: Max processes             514767               514767               processes
 
Dec 10 00:15:46 main mariadbd[1435]: Max open files            32768                32768                files
 
Dec 10 00:15:46 main mariadbd[1435]: Max locked memory         524288               524288               bytes
 
Dec 10 00:15:46 main mariadbd[1435]: Max address space         unlimited            unlimited            bytes
 
Dec 10 00:15:46 main mariadbd[1435]: Max file locks            unlimited            unlimited            locks
 
Dec 10 00:15:46 main mariadbd[1435]: Max pending signals       514767               514767               signals
 
Dec 10 00:15:46 main mariadbd[1435]: Max msgqueue size         819200               819200               bytes
 
Dec 10 00:15:46 main mariadbd[1435]: Max nice priority         0                    0
 
Dec 10 00:15:46 main mariadbd[1435]: Max realtime priority     0                    0
 
Dec 10 00:15:46 main mariadbd[1435]: Max realtime timeout      unlimited            unlimited            us
 
Dec 10 00:15:46 main mariadbd[1435]: Core pattern: core
 
Dec 10 00:15:46 main mariadbd[1435]: Kernel version: Linux version 5.10.0-19-amd64 (debian-kernel@lists.debian.org) (gcc-10 (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP Debian 5.10.149-2 (2022-10-21)
 
Dec 10 00:15:51 main mariadbd[3482145]: 2022-12-10  0:15:51 0 [Note] /usr/sbin/mariadbd (server 10.7.7-MariaDB-1:10.7.7+maria~deb11-log) starting as process 3482145 ...
 
Dec 10 00:15:51 main mariadbd[3482145]: 2022-12-10  0:15:51 0 [Note] RocksDB: 2 column families found
 
Dec 10 00:15:51 main mariadbd[3482145]: 2022-12-10  0:15:51 0 [Note] RocksDB: Column Families at start:
 
Dec 10 00:15:51 main mariadbd[3482145]: 2022-12-10  0:15:51 0 [Note]   cf=default
 
Dec 10 00:15:51 main mariadbd[3482145]: 2022-12-10  0:15:51 0 [Note]     write_buffer_size=67108864
 
Dec 10 00:15:51 main mariadbd[3482145]: 2022-12-10  0:15:51 0 [Note]     target_file_size_base=33554432
 
Dec 10 00:15:51 main mariadbd[3482145]: 2022-12-10  0:15:51 0 [Note]   cf=__system__
 
Dec 10 00:15:51 main mariadbd[3482145]: 2022-12-10  0:15:51 0 [Note]     write_buffer_size=67108864
 
Dec 10 00:15:51 main mariadbd[3482145]: 2022-12-10  0:15:51 0 [Note]     target_file_size_base=67108864
 
Dec 10 00:15:52 main mariadbd[3482145]: 2022-12-10  0:15:52 0 [ERROR] RocksDB: Error opening instance, Status Code: 5, Status: IO error: No space left on deviceWhile appending to file: ./#rocksdb/581104.sst: No space left on device
 
Dec 10 00:15:52 main mariadbd[3482145]: 2022-12-10  0:15:52 0 [ERROR] Plugin 'ROCKSDB' init function returned error.
 
Dec 10 00:15:52 main mariadbd[3482145]: 2022-12-10  0:15:52 0 [ERROR] Plugin 'ROCKSDB' registration as a STORAGE ENGINE failed.
 
Dec 10 00:15:52 main mariadbd[3482145]: 2022-12-10  0:15:52 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
 
Dec 10 00:15:52 main mariadbd[3482145]: 2022-12-10  0:15:52 0 [Note] InnoDB: Number of transaction pools: 1
 
Dec 10 00:15:52 main mariadbd[3482145]: 2022-12-10  0:15:52 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
 
Dec 10 00:15:52 main mariadbd[3482145]: 2022-12-10  0:15:52 0 [Note] InnoDB: Using liburing
 
Dec 10 00:15:52 main mariadbd[3482145]: 2022-12-10  0:15:52 0 [Note] InnoDB: Initializing buffer pool, total size = 34359738368, chunk size = 134217728
 
Dec 10 00:15:52 main mariadbd[3482145]: 2022-12-10  0:15:52 0 [Note] InnoDB: Completed initialization of buffer pool
 
Dec 10 00:15:52 main mariadbd[3482145]: 2022-12-10  0:15:52 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=201143428594,201162725253
 
Dec 10 00:16:04 main mariadbd[3482145]: 2022-12-10  0:16:04 0 [Note] InnoDB: Starting final batch to recover  34907 pages from redo log.
 
Dec 10 00:16:07 main mariadbd[3482145]: 2022-12-10  0:16:07 0 [Note] InnoDB: To recover: 7866 pages from log
 
Dec 10 00:16:07 main mariadbd[3482145]: 2022-12-10  0:16:07 0 [Note] InnoDB: 128 rollback segments are active.
 
Dec 10 00:16:07 main mariadbd[3482145]: 2022-12-10  0:16:07 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1"
 
Dec 10 00:16:07 main mariadbd[3482145]: 2022-12-10  0:16:07 0 [Note] InnoDB: Creating shared tablespace for temporary tables
 
Dec 10 00:16:07 main mariadbd[3482145]: 2022-12-10  0:16:07 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
 
Dec 10 00:16:07 main mariadbd[3482145]: 2022-12-10  0:16:07 0 [ERROR] InnoDB: preallocating 12582912 bytes for file ./ibtmp1 failed with error 28
 
Dec 10 00:16:07 main mariadbd[3482145]: 2022-12-10  0:16:07 0 [ERROR] InnoDB: Could not set the file size of './ibtmp1'. Probably out of disk space
 
Dec 10 00:16:07 main mariadbd[3482145]: 2022-12-10  0:16:07 0 [ERROR] InnoDB: Unable to create the shared innodb_temporary
 
Dec 10 00:16:07 main mariadbd[3482145]: 2022-12-10  0:16:07 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
 
Dec 10 00:16:07 main mariadbd[3482145]: 2022-12-10  0:16:07 0 [Note] InnoDB: FTS optimize thread exiting.
 
Dec 10 00:16:07 main mariadbd[3482145]: 2022-12-10  0:16:07 0 [Note] InnoDB: Starting shutdown...
 
Dec 10 00:16:08 main mariadbd[3482145]: 2022-12-10  0:16:08 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1"
 
Dec 10 00:16:08 main mariadbd[3482145]: 2022-12-10  0:16:08 0 [ERROR] Plugin 'InnoDB' init function returned error.
 
Dec 10 00:16:08 main mariadbd[3482145]: 2022-12-10  0:16:08 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
 
Dec 10 00:16:08 main mariadbd[3482145]: 2022-12-10  0:16:08 0 [Note] Plugin 'FEEDBACK' is disabled.
 
Dec 10 00:16:08 main mariadbd[3482145]: 2022-12-10  0:16:08 0 [ERROR] Unknown/unsupported storage engine: InnoDB
 
Dec 10 00:16:08 main mariadbd[3482145]: 2022-12-10  0:16:08 0 [ERROR] Aborting

After restart MariaDB

Dec 10 10:48:59 main mariadbd[3495662]: 2022-12-10 10:48:59 0 [Note] /usr/sbin/mariadbd (server 10.7.7-MariaDB-1:10.7.7+maria~deb11-log) starting as process 3495662 ...
 
Dec 10 10:48:59 main mariadbd[3495662]: 2022-12-10 10:48:59 0 [Note] RocksDB: 2 column families found
 
Dec 10 10:48:59 main mariadbd[3495662]: 2022-12-10 10:48:59 0 [Note] RocksDB: Column Families at start:
 
Dec 10 10:48:59 main mariadbd[3495662]: 2022-12-10 10:48:59 0 [Note]   cf=default
 
Dec 10 10:48:59 main mariadbd[3495662]: 2022-12-10 10:48:59 0 [Note]     write_buffer_size=67108864
 
Dec 10 10:48:59 main mariadbd[3495662]: 2022-12-10 10:48:59 0 [Note]     target_file_size_base=33554432
 
Dec 10 10:48:59 main mariadbd[3495662]: 2022-12-10 10:48:59 0 [Note]   cf=__system__
 
Dec 10 10:48:59 main mariadbd[3495662]: 2022-12-10 10:48:59 0 [Note]     write_buffer_size=67108864
 
Dec 10 10:48:59 main mariadbd[3495662]: 2022-12-10 10:48:59 0 [Note]     target_file_size_base=67108864
 
Dec 10 10:49:00 main mariadbd[3495662]: 2022-12-10 10:49:00 0 [Note] RocksDB: Table_store: loaded DDL data for 3 tables
 
Dec 10 10:49:00 main mariadbd[3495662]: 2022-12-10 10:49:00 0 [Note] RocksDB: global statistics using get_sched_indexer_t indexer
 
Dec 10 10:49:00 main mariadbd[3495662]: 2022-12-10 10:49:00 0 [Note] MyRocks storage engine plugin has been successfully initialized.
 
Dec 10 10:49:00 main mariadbd[3495662]: 2022-12-10 10:49:00 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
 
Dec 10 10:49:00 main mariadbd[3495662]: 2022-12-10 10:49:00 0 [Note] InnoDB: Number of transaction pools: 1
 
Dec 10 10:49:00 main mariadbd[3495662]: 2022-12-10 10:49:00 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
 
Dec 10 10:49:00 main mariadbd[3495662]: 2022-12-10 10:49:00 0 [Note] InnoDB: Using liburing
 
Dec 10 10:49:00 main mariadbd[3495662]: 2022-12-10 10:49:00 0 [Note] InnoDB: Initializing buffer pool, total size = 34359738368, chunk size = 134217728
 
Dec 10 10:49:00 main mariadbd[3495662]: 2022-12-10 10:49:00 0 [Note] InnoDB: Completed initialization of buffer pool
 
Dec 10 10:49:00 main mariadbd[3495662]: 2022-12-10 10:49:00 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=201143428594,201143428594
 
Dec 10 10:49:13 main mariadbd[3495662]: 2022-12-10 10:49:13 0 [Note] InnoDB: Starting final batch to recover  34907 pages from redo log.
 
Dec 10 10:49:15 main mariadbd[3495662]: 2022-12-10 10:49:15 0 [Note] InnoDB: To recover: 12912 pages from log
 
Dec 10 10:49:16 main mariadbd[3495662]: 2022-12-10 10:49:16 0 [Note] InnoDB: 128 rollback segments are active.
 
Dec 10 10:49:16 main mariadbd[3495662]: 2022-12-10 10:49:16 0 [Note] InnoDB: Creating shared tablespace for temporary tables
 
Dec 10 10:49:16 main mariadbd[3495662]: 2022-12-10 10:49:16 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
 
Dec 10 10:49:16 main mariadbd[3495662]: 2022-12-10 10:49:16 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
 
Dec 10 10:49:16 main mariadbd[3495662]: 2022-12-10 10:49:16 0 [Note] InnoDB: 10.7.7 started; log sequence number 202450436185; transaction id 554514390
 
Dec 10 10:49:16 main mariadbd[3495662]: 2022-12-10 10:49:16 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
 
Dec 10 10:49:16 main mariadbd[3495662]: 2022-12-10 10:49:16 0 [Note] Plugin 'FEEDBACK' is disabled.
 
Dec 10 10:49:16 main mariadbd[3495662]: 2022-12-10 10:49:16 0 [Note] Recovering after a crash using tc.log
 
Dec 10 10:49:16 main mariadbd[3495662]: 2022-12-10 10:49:16 0 [Note] Starting table crash recovery...
 
Dec 10 10:49:16 main mariadbd[3495662]: 2022-12-10 10:49:16 0 [Note] Crash table recovery finished.
 
Dec 10 10:49:16 main mariadbd[3495662]: 2022-12-10 10:49:16 0 [Note] Server socket created on IP: '127.0.0.1'.
 
Dec 10 10:49:16 main mariadbd[3495662]: 2022-12-10 10:49:16 0 [Note] /usr/sbin/mariadbd: ready for connections.
 
Dec 10 10:49:16 main mariadbd[3495662]: Version: '10.7.7-MariaDB-1:10.7.7+maria~deb11-log'  socket: '/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution
 
Dec 10 10:49:18 main mariadbd[3495662]: 2022-12-10 10:49:18 0 [Note] InnoDB: Buffer pool(s) load completed at 221210 10:49:18



 Comments   
Comment by Lewis Chan [ 2022-12-14 ]

You should help to tell what's your environment is and how to reproduce it.

Generated at Thu Feb 08 10:14:24 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.