[MDEV-18427] RocksDB crash Created: 2019-01-30  Updated: 2023-04-27

Status: Open
Project: MariaDB Server
Component/s: Storage Engine - RocksDB
Affects Version/s: 10.3.12
Fix Version/s: 10.4

Type: Bug Priority: Major
Reporter: Philip orleans Assignee: Sergei Petrunia
Resolution: Unresolved Votes: 0
Labels: None
Environment:

windows server 2019, box with 512 GB RAM



 Description   

190130 2:55:08 [ERROR] mysqld got exception 0xc0000005 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.3.12-MariaDB
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=6
max_threads=65537
thread_count=8
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 136184 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x25316fff148
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
ha_rocksdb.dll!???
ha_rocksdb.dll!???
ha_rocksdb.dll!???
ha_rocksdb.dll!???
ha_rocksdb.dll!???
ha_rocksdb.dll!???
ha_rocksdb.dll!???
ha_rocksdb.dll!???
ha_rocksdb.dll!???
ha_rocksdb.dll!???
ha_rocksdb.dll!???
ha_rocksdb.dll!???
ha_rocksdb.dll!???
ha_rocksdb.dll!???
ha_rocksdb.dll!???
mysqld.exe!handler::ha_index_next()[handler.cc:2873]
mysqld.exe!join_read_next()[sql_select.cc:20410]
mysqld.exe!sub_select()[sql_select.cc:19415]
mysqld.exe!do_select()[sql_select.cc:18936]
mysqld.exe!JOIN::exec_inner()[sql_select.cc:4040]
mysqld.exe!JOIN::exec()[sql_select.cc:3835]
mysqld.exe!mysql_select()[sql_select.cc:4241]
mysqld.exe!handle_select()[sql_select.cc:373]
mysqld.exe!execute_sqlcom_select()[sql_parse.cc:6548]
mysqld.exe!mysql_execute_command()[sql_parse.cc:3769]
mysqld.exe!mysql_parse()[sql_parse.cc:8097]
mysqld.exe!dispatch_command()[sql_parse.cc:1853]
mysqld.exe!do_command()[sql_parse.cc:1395]
mysqld.exe!threadpool_process_request()[threadpool_common.cc:358]
mysqld.exe!tp_callback()[threadpool_common.cc:186]
ntdll.dll!RtlInitializeCriticalSection()
ntdll.dll!RtlReleaseSRWLockExclusive()
KERNEL32.DLL!BaseThreadInitThunk()
ntdll.dll!RtlUserThreadStart()

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x2540e7b1e10): SELECT did,lrn,ocn INTO OUTFILE 'g:\\temp
lrn.txt' FIELDS TERMINATED BY ',' from asterisk.npadata where lrn > 0 order by did asc
Connection ID (thread ID): 697
Status: NOT_KILLED

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on
-----------------------------------
confoguration

[mysqld]
datadir=C:/Program Files/MariaDB 10.3/data
port=3306
innodb_buffer_pool_size=65523M
innodb_force_recovery=0
innodb_purge_threads=4
innodb_page_size=16384
innodb_buffer_pool_instances=64
#innodb_buffer_pool_size=90G
innodb_buffer_pool_dump_at_shutdown=OFF
innodb_buffer_pool_load_at_startup=OFF
innodb_doublewrite=1
innodb_file_per_table=1
innodb_use_atomic_writes=1
innodb_compression_level=6
innodb_compression_algorithm=zlib
innodb_use_native_aio=1
innodb_flush_log_at_trx_commit=2
innodb_flush_log_at_timeout=10
innodb_strict_mode=0
innodb_log_file_size=15G
innodb_log_files_in_group=2
innodb_open_files=4000
innodb_log_buffer_size=64M
innodb_thread_concurrency=0
innodb_flush_method=O_DIRECT_NO_FSYNC
innodb_max_dirty_pages_pct=90
innodb_lru_scan_depth=1024
innodb_page_cleaners=4
innodb_stats_persistent = 1
innodb_adaptive_flushing = 1
innodb_flush_neighbors = 0
innodb_read_io_threads = 4
innodb_write_io_threads = 2
innodb_io_capacity=2000
innodb_io_capacity_max=4000
innodb_adaptive_hash_index=1
rocksdb_commit_in_the_middle=1
rocksdb_max_open_files=-1
rocksdb_allow_concurrent_memtable_write=1
rocksdb_blind_delete_primary_key=1
rocksdb_tmpdir=/tmp
rocksdb_bulk_load_size=1000
rocksdb_max_total_wal_size=14G
rocksdb_block_size=16384
rocksdb_block_cache_size=90G
rocksdb_table_cache_numshardbits=19
rocksdb_default_cf_options=write_buffer_size=256m;target_file_size_base=64m;max_bytes_for_level_base=512m;level0_file_num_compaction_trigger=4;level0_slowdown_writes_trigger=256;level0_stop_writes_trigger=256;max_write_buffer_number=16;compression_per_level=kNoCompression:kNoCompression:kNoCompression:kZlibCompression:kZlibCompression:kZlibCompression;bottommost_compression=kZlibCompression;compression_opts=-14:1:0;block_based_table_factory=

{cache_index_and_filter_blocks=1;filter_policy=bloomfilter:10:false;whole_key_filtering=1}

;level_compaction_dynamic_level_bytes=true;optimize_filters_for_hits=true
rocksdb_override_cf_options=system=

{memtable=skip_list:16}

rocksdb_write_disable_wal=1
rocksdb_flush_log_at_trx_commit=2
rocksdb_strict_collation_check=off
rocksdb_max_background_jobs=24
rocksdb_bytes_per_sync=16777216
rocksdb_wal_bytes_per_sync=4194304
rocksdb_compaction_sequential_deletes_count_sd=1
rocksdb_compaction_sequential_deletes=199999
rocksdb_compaction_sequential_deletes_window=200000
rocksdb_default_cf_options="write_buffer_size=256m;target_file_size_base=32m;max_bytes_for_level_base=512m;max_write_buffer_number=4;level0_file_num_compaction_trigger=4;level0_slowdown_writes_trigger=20;level0_stop_writes_trigger=30;max_write_buffer_number=4;block_based_table_factory=

{cache_index_and_filter_blocks=1;filter_policy=bloomfilter:10:false;whole_key_filtering=0}

;level_compaction_dynamic_level_bytes=true;optimize_filters_for_hits=true;memtable_prefix_bloom_size_ratio=0.05;prefix_extractor=capped:12;compaction_pri=kMinOverlappingRatio;compression=kZlibCompression;bottommost_compression=kZlibCompression;compression_opts=-14:4:0"
rocksdb_max_subcompactions=16
rocksdb_compaction_readahead_size=16m
rocksdb_use_direct_reads=ON
rocksdb_use_direct_io_for_flush_and_compaction=ON
rocksdb_max_row_locks=1073741824
rocksdb-table-stats-sampling-pct=15
[client]
port=3306
plugin-dir=C:/Program Files/MariaDB 10.3/lib/plugin
------------------------------------
table has 1.7 BN records



 Comments   
Comment by Sergei Petrunia [ 2019-01-31 ]

non-resolved symbols in ha_rocksdb.dll

ha_rocksdb.dll!???
ha_rocksdb.dll!???
ha_rocksdb.dll!???
ha_rocksdb.dll!???

do not give much clue about what happened.

(Now trying to figure out what's up with the symbols)

Comment by Sergei Petrunia [ 2019-01-31 ]

Re the debug symbols (Thanks to wlad for the info): They are built but are not included into the default package.

One can get it from here: https://downloads.mariadb.org/mariadb/10.3.12/ , the file is mariadb-10.3.12-winx64-debugsymbols.zip.

One can take the whole tarball, or just get the ha_rocksdb.pdb file and copy it next to the mysqld.exe.

philip_38, would it be possible for you to install the debug symbols?

Comment by Sergei Petrunia [ 2019-01-31 ]

Another suggestion: add core-file=1 into my.ini file. This will create a minidump on crash (file name mysqld.dmp). The file will help us to analyze the crash.

Comment by Philip orleans [ 2020-08-26 ]

I use the Linux version and could not find any my.ini to include core-file=1. Also, of I include core-file=1 in /etc/my.cnf, the server does not start.
So how do we get a dump in case of a crash in Linux?

Generated at Thu Feb 08 08:44:01 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.