[MDEV-18246] MyRocks ColumnFamilySet(): Assertion `last_ref' failed on MariaDB startup Created: 2019-01-15  Updated: 2019-06-21  Resolved: 2019-06-21

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - RocksDB
Affects Version/s: 10.2.18, 10.2.21
Fix Version/s: 10.2.26, 10.3.17, 10.4.7

Type: Bug Priority: Major
Reporter: Jonas Krauss Assignee: Sergei Petrunia
Resolution: Duplicate Votes: 4
Labels: crash
Environment:

Debian Linux version 4.9.0-8-amd64 (debian-kernel@lists.debian.org) (gcc version 6.3.0 20170516 (Debian 6.3.0-18+deb9u1) ) #1 SMP Debian 4.9.110-3+deb9u6 (2018-10-08)

jemalloc

64GB RAM machine with RAID1 SSD 256GB for logs and 10TB HDD for tables

Attachments: File mariadb.log     File my.cnf     File rocksdb.log    
Issue Links:
Blocks
is blocked by MDEV-19795 Merge upstream MyRocks: June 2019 Closed
Duplicate
duplicates MDEV-19395 Server fails to crash recover after b... Closed

 Description   

We had a database setup running in production for quite some time (> 5 months) and it crashed during last weekend without obvious reason. MariaDB error log shows this on startup of the server:

2019-01-15 10:07:21 140060699941120 [Note] RocksDB: 2 column families found
 
2019-01-15 10:07:21 140060699941120 [Note] RocksDB: Column Families at start:
 
2019-01-15 10:07:21 140060699941120 [Note]   cf=default
 
2019-01-15 10:07:21 140060699941120 [Note]     write_buffer_size=134217728
 
2019-01-15 10:07:21 140060699941120 [Note]     target_file_size_base=268435456
 
2019-01-15 10:07:21 140060699941120 [Note]   cf=__system__
 
2019-01-15 10:07:21 140060699941120 [Note]     write_buffer_size=134217728
 
2019-01-15 10:07:21 140060699941120 [Note]     target_file_size_base=268435456
 
mysqld: /home/buildbot/buildbot/build/mariadb-10.2.21/storage/rocksdb/rocksdb/db/column_family.cc:1238: rocksdb::ColumnFamilySet::~ColumnFamilySet(): Assertion `last_ref' failed.
 
190115 10:07:32 [ERROR] mysqld got signal 6 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.2.21-MariaDB-10.2.21+maria~stretch-log
 
key_buffer_size=134217728
 
read_buffer_size=131072
 
max_used_connections=0
 
max_threads=3002
 
thread_count=0
 
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6727382 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x0
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x0 thread_stack 0x30000
 
*** buffer overflow detected ***: /usr/sbin/mysqld terminated
 
======= Backtrace: =========
 
/lib/x86_64-linux-gnu/libc.so.6(+0x70bfb)[0x7f626a34ebfb]
 
/lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7f626a3d71f7]
 
/lib/x86_64-linux-gnu/libc.so.6(+0xf7330)[0x7f626a3d5330]
 
/lib/x86_64-linux-gnu/libc.so.6(+0xf916a)[0x7f626a3d716a]
 
/usr/sbin/mysqld(my_addr_resolve+0xe2)[0x55c80b8c0e82]
 
/usr/sbin/mysqld(my_print_stacktrace+0x1bb)[0x55c80b8a773b]
 
/usr/sbin/mysqld(handle_fatal_signal+0x41d)[0x55c80b346c7d]
 
/lib/x86_64-linux-gnu/libpthread.so.0(+0x110c0)[0x7f626be7c0c0]
 
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcf)[0x7f626a310fff]
 
/lib/x86_64-linux-gnu/libc.so.6(abort+0x16a)[0x7f626a31242a]
 
/lib/x86_64-linux-gnu/libc.so.6(+0x2be67)[0x7f626a309e67]
 
/lib/x86_64-linux-gnu/libc.so.6(+0x2bf12)[0x7f626a309f12]
 
/usr/lib/mysql/plugin/ha_rocksdb.so(_ZN7rocksdb15ColumnFamilySetD1Ev+0x1de)[0x7f61de475b0e]
 
/usr/lib/mysql/plugin/ha_rocksdb.so(_ZN7rocksdb10VersionSetD1Ev+0x4d)[0x7f61de3205cd]
 
/usr/lib/mysql/plugin/ha_rocksdb.so(_ZN7rocksdb6DBImpl11CloseHelperEv+0x647)[0x7f61de260627]
 
/usr/lib/mysql/plugin/ha_rocksdb.so(_ZN7rocksdb6DBImplD1Ev+0x69d)[0x7f61de26fa1d]
 
/usr/lib/mysql/plugin/ha_rocksdb.so(_ZN7rocksdb6DBImplD0Ev+0x11)[0x7f61de26fa81]
 
/usr/lib/mysql/plugin/ha_rocksdb.so(_ZN7rocksdb24PessimisticTransactionDBD2Ev+0x32c)[0x7f61de42e06c]
 
/usr/lib/mysql/plugin/ha_rocksdb.so(_ZN7rocksdb19WriteCommittedTxnDBD0Ev+0x1f)[0x7f61de4346df]
 
/usr/lib/mysql/plugin/ha_rocksdb.so(_ZN7rocksdb13TransactionDB6WrapDBEPNS_2DBERKNS_20TransactionDBOptionsERKSt6vectorImSaImEERKS6_IPNS_18ColumnFamilyHandleESaISC_EEPPS0_+0x12a)[0x7f61de430e3a]
 
/usr/lib/mysql/plugin/ha_rocksdb.so(_ZN7rocksdb13TransactionDB4OpenERKNS_9DBOptionsERKNS_20TransactionDBOptionsERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt6vectorINS_22ColumnFamilyDescriptorESaISG_EEPSF_IPNS_18ColumnFamilyHandleESaISM_EEPPS0_+0x1341)[0x7f61de432251]
 
/usr/lib/mysql/plugin/ha_rocksdb.so(+0x20e556)[0x7f61de201556]
 
/usr/sbin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x62)[0x55c80b3490a2]
 
/usr/sbin/mysqld(+0x4cdf75)[0x55c80b1bbf75]
 
/usr/sbin/mysqld(_Z11plugin_initPiPPci+0x7da)[0x55c80b1bd62a]
 
/usr/sbin/mysqld(+0x427e7c)[0x55c80b115e7c]
 
/usr/sbin/mysqld(_Z11mysqld_mainiPPc+0x1d48)[0x55c80b11add8]
 
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7f626a2fe2e1]
 
/usr/sbin/mysqld(_start+0x2a)[0x55c80b10e40a]
 
======= Memory map: ========
 
55c80acee000-55c80bdcb000 r-xp 00000000 09:02 8528713                    /usr/sbin/mysqld
 
55c80bfca000-55c80c09a000 r--p 010dc000 09:02 8528713                    /usr/sbin/mysqld
 
55c80c09a000-55c80c151000 rw-p 011ac000 09:02 8528713                    /usr/sbin/mysqld
 
...

We tried the various recovery modes of rocksdb_wal_recovery_mode (0,2,3) but none seems to have any effect, the output in the log is exactly the same.

Here is the rocksdb config part of the my.cnf:

plugin-load-add					= ha_rocksdb.so
 
default-storage-engine				= rocksdb
 
rocksdb_unsafe_for_binlog			= 1 # enables statement based replication
 
rocksdb_datadir					= /var/local/mysql/rocksdb
 
rocksdb_wal_dir					= /var/mysql_logs/rocksdb
 
rocksdb_tmpdir					= /var/mysql_logs/rocksdb
 
rocksdb_flush_log_at_trx_commit			= 0
 
rocksdb_use_direct_io_for_flush_and_compaction	= 0
 
rocksdb_use_direct_reads			= 0
 
rocksdb_max_open_files				= -1
 
rocksdb_max_background_jobs			= 8
 
rocksdb_max_total_wal_size			= 4G
 
rocksdb_block_size				= 64K
 
rocksdb_block_cache_size			= 16G
 
rocksdb_table_cache_numshardbits		= 6
 
rocksdb_new_table_reader_for_compaction_inputs	= 1
 
rocksdb_compaction_readahead_size		= 4M
 
#rocksdb_db_write_buffer_size			= 0 # max write buffer across all column families, zero = disabled
 
#rocksdb_wal_recovery_mode			= 1
 
rocksdb_bytes_per_sync				= 4M
 
rocksdb_wal_bytes_per_sync			= 4M
 
rocksdb_rate_limiter_bytes_per_sec		= 80M # MB/s. Increase if you're running on higher spec machines
 
rocksdb_compaction_sequential_deletes_count_sd	= 1
 
rocksdb_compaction_sequential_deletes		= 199999
 
rocksdb_compaction_sequential_deletes_window	= 200000
 
#rocksdb_rpl_lookup_rows=0
 
rocksdb_default_cf_options=write_buffer_size=128m;max_write_buffer_number=4;target_file_size_base=256m;max_bytes_for_level_base=2560m;target_file_size_multiplier=2;level0_file_num_compaction_trigger=4;level0_slowdown_writes_trigger=10;level0_stop_writes_trigger=15;compression=kSnappyCompression;bottommost_compression=kZlibCompression;compression_opts=-14:1:0;block_based_table_factory={cache_index_and_filter_blocks=1;filter_policy=bloomfilter:10:false;whole_key_filtering=1};level_compaction_dynamic_level_bytes=true;optimize_filters_for_hits=true;compaction_pri=kMinOverlappingRatio

Google did not return helpful results, most similar thing anyone had is this in my search: https://github.com/facebook/mysql-5.6/issues/391

Checking the rocksdb log it seems that the recovery phase is successfully completed, yet the rocksdb engine receives a shutdown signal from the MariaDB process. The log does not show any error message just before it goes into shutdown, so my best guess is that the shutdown signal is not internally produced but comes from MariaDB). Here is the regarding log:

...
 
Recovering log #70212 mode 1
 
2019/01/14-17:57:35.828286 7fd584e99900 [/home/buildbot/buildbot/build/mariadb-10.2.21/storage/rocksdb/rocksdb/db/db_impl_open.cc:552] Recovering log #70215 mode 1
 
2019/01/14-17:57:35.828310 7fd584e99900 [/home/buildbot/buildbot/build/mariadb-10.2.21/storage/rocksdb/rocksdb/db/db_impl_open.cc:552] Recovering log #70218 mode 1
 
2019/01/14-17:57:35.828642 7fd584e99900 [/home/buildbot/buildbot/build/mariadb-10.2.21/storage/rocksdb/rocksdb/db/version_set.cc:2854] Creating manifest 70220
 
2019/01/14-17:57:35.937947 7fd584e99900 [WARN] [/home/buildbot/buildbot/build/mariadb-10.2.21/storage/rocksdb/rocksdb/db/version_set.cc:2496] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed.
 
2019/01/14-17:57:35.979604 7fd584e99900 EVENT_LOG_v1 {"time_micros": 1547485055979597, "job": 1, "event": "recovery_finished"}
 
2019/01/14-17:57:35.992961 7fd584e99900 [/home/buildbot/buildbot/build/mariadb-10.2.21/storage/rocksdb/rocksdb/db/db_impl_files.cc:477] [JOB 2] Delete info log file /var/local/mysql/rocksdb//LOG.old.1547441122341707
 
2019/01/14-17:57:36.059847 7fd584e99900 [/home/buildbot/buildbot/build/mariadb-10.2.21/storage/rocksdb/rocksdb/db/db_impl_open.cc:1221] DB pointer 0x7fd4fea2ae00
 
2019/01/14-17:57:38.210653 7fd584e99900 [/home/buildbot/buildbot/build/mariadb-10.2.21/storage/rocksdb/rocksdb/db/db_impl.cc:295] Shutdown: canceling all background work

Another observation is that rocksdb created ~6600 wal files in the regarding folder, yet most of them are just empty files. Only about 50 files are filled with content up to the point in time the crash occurred. So maybe something is wrong here. We have ~2500 files in the rocksdb datadir, a couply of them are also log files which correspond to files created in the wal dir.

I have the feeling the high number of files and the assertion failure might be connected, but it is just a guess. I could imagine that it is actually just a configuration problem which results in too many files being opened simultaneously leading to the assertion failure, just guessing wildly here.

I increased hard and soft limits for open files to 20000 for the mysql user, however this did not solve the problem. I also tried setting a limit to rocksdb max open files, this changed the error message in the MariaDB log, however it still failed with the same assertion failure but not with a buffer overflow anymore. I upgraded MariaDB from 10.2.18 to 10.2.21 but this did not solve the problem either.

I am happy to provide more details on request or to try out other things. Any support or ideas are greatly appreciated.

 Comments   
Comment by YOSHINORI MATSUNOBU [ 2019-01-15 ]

Could you update MySQL err log (*err.log) and RocksDB log files (under $rocksdb_datadir/LOG, default rocksdb_datadir is $datadir/.rocksdb) so that we can look further what was going on?

Comment by Sergei Petrunia [ 2019-01-15 ]

(under $rocksdb_datadir/LOG, default rocksdb_datadir is $datadir/.rocksdb)

It's #rocksdb in MariaDB by default.

jkrauss, in case you will be trying to reproduce the crash, could you please install the debug symbols:

sudo apt-get install mariadb-server-10.2-dbgsym mariadb-plugin-rocksdb-dbgsym

.. although the first step is to get the log files.

Comment by Jonas Krauss [ 2019-01-16 ]

Just uploaded the log files of a startup attempt a few minutes ago. Also attached the my.cnf just in case.

@Sergei Patrunia I will install the debug symbols and attach additional log files here, soon.

Thanks.

Comment by Jonas Krauss [ 2019-01-16 ]

Tried installing the debug symbols however setup fails due to MariaDB not starting up correctly. Is this a problem? I did not see any additional entries in the log files checking them after debug symbols were installed, but maybe I just overlooked them.

sudo apt-get install mariadb-server-10.2-dbgsym mariadb-plugin-rocksdb-dbgsym
 
Reading package lists... Done
 
Building dependency tree       
Reading state information... Done
 
mariadb-plugin-rocksdb-dbgsym is already the newest version (10.2.21+maria~stretch).
 
mariadb-server-10.2-dbgsym is already the newest version (10.2.21+maria~stretch).
 
0 upgraded, 0 newly installed, 0 to remove and 21 not upgraded.
 
5 not fully installed or removed.
 
After this operation, 0 B of additional disk space will be used.
 
Do you want to continue? [Y/n] 
Setting up mariadb-server-10.2 (10.2.21+maria~stretch) ...
 
*** buffer overflow detected ***: /usr/sbin/mysqld terminated
 
======= Backtrace: =========
 
/lib/x86_64-linux-gnu/libc.so.6(+0x70bfb)[0x7f62fd1d8bfb]
 
/lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7f62fd2611f7]
 
/lib/x86_64-linux-gnu/libc.so.6(+0xf7330)[0x7f62fd25f330]
 
/lib/x86_64-linux-gnu/libc.so.6(+0xf916a)[0x7f62fd26116a]
 
/usr/sbin/mysqld(my_addr_resolve+0xe2)[0x55d70032fe82]
 
/usr/sbin/mysqld(my_print_stacktrace+0x1bb)[0x55d70031673b]
 
/usr/sbin/mysqld(handle_fatal_signal+0x41d)[0x55d6ffdb5c7d]
 
/lib/x86_64-linux-gnu/libpthread.so.0(+0x110c0)[0x7f62fed060c0]
 
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcf)[0x7f62fd19afff]
 
/lib/x86_64-linux-gnu/libc.so.6(abort+0x16a)[0x7f62fd19c42a]
 
/lib/x86_64-linux-gnu/libc.so.6(+0x2be67)[0x7f62fd193e67]
 
/lib/x86_64-linux-gnu/libc.so.6(+0x2bf12)[0x7f62fd193f12]
 
/usr/lib/mysql/plugin/ha_rocksdb.so(_ZN7rocksdb15ColumnFamilySetD1Ev+0x1de)[0x7f6271475b0e]
 
/usr/lib/mysql/plugin/ha_rocksdb.so(_ZN7rocksdb10VersionSetD1Ev+0x4d)[0x7f62713205cd]
 
/usr/lib/mysql/plugin/ha_rocksdb.so(_ZN7rocksdb6DBImpl11CloseHelperEv+0x647)[0x7f6271260627]
 
/usr/lib/mysql/plugin/ha_rocksdb.so(_ZN7rocksdb6DBImplD1Ev+0x69d)[0x7f627126fa1d]
 
/usr/lib/mysql/plugin/ha_rocksdb.so(_ZN7rocksdb6DBImplD0Ev+0x11)[0x7f627126fa81]
 
/usr/lib/mysql/plugin/ha_rocksdb.so(_ZN7rocksdb24PessimisticTransactionDBD2Ev+0x32c)[0x7f627142e06c]
 
/usr/lib/mysql/plugin/ha_rocksdb.so(_ZN7rocksdb19WriteCommittedTxnDBD0Ev+0x1f)[0x7f62714346df]
 
/usr/lib/mysql/plugin/ha_rocksdb.so(_ZN7rocksdb13TransactionDB6WrapDBEPNS_2DBERKNS_20TransactionDBOptionsERKSt6vectorImSaImEERKS6_IPNS_18ColumnFamilyHandleESaISC_EEPPS0_+0x12a)[0x7f6271430e3a]
 
/usr/lib/mysql/plugin/ha_rocksdb.so(_ZN7rocksdb13TransactionDB4OpenERKNS_9DBOptionsERKNS_20TransactionDBOptionsERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt6vectorINS_22ColumnFamilyDescriptorESaISG_EEPSF_IPNS_18ColumnFamilyHandleESaISM_EEPPS0_+0x1341)[0x7f6271432251]
 
/usr/lib/mysql/plugin/ha_rocksdb.so(+0x20e556)[0x7f6271201556]
 
/usr/sbin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x62)[0x55d6ffdb80a2]
 
/usr/sbin/mysqld(+0x4cdf75)[0x55d6ffc2af75]
 
/usr/sbin/mysqld(_Z11plugin_initPiPPci+0x7da)[0x55d6ffc2c62a]
 
/usr/sbin/mysqld(+0x427e7c)[0x55d6ffb84e7c]
 
/usr/sbin/mysqld(_Z11mysqld_mainiPPc+0x1d48)[0x55d6ffb89dd8]
 
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7f62fd1882e1]
 
/usr/sbin/mysqld(_start+0x2a)[0x55d6ffb7d40a]
 
======= Memory map: ========
 
55d6ff75d000-55d70083a000 r-xp 00000000 09:02 8528713                    /usr/sbin/mysqld
 
55d700a39000-55d700b09000 r--p 010dc000 09:02 8528713                    /usr/sbin/mysqld
 
55d700b09000-55d700bc0000 rw-p 011ac000 09:02 8528713                    /usr/sbin/mysqld
 
55d700bc0000-55d701452000 rw-p 00000000 00:00 0 
7f6261400000-7f6264400000 rw-p 00000000 00:00 0 
7f6265800000-7f6265c00000 rw-p 00000000 00:00 0 
7f6267000000-7f6267400000 rw-p 00000000 00:00 0 
7f6267bfd000-7f6267bfe000 ---p 00000000 00:00 0 
7f6267bfe000-7f62683fe000 rw-p 00000000 00:00 0 
7f62683fe000-7f62683ff000 ---p 00000000 00:00 0 
7f62683ff000-7f6268bff000 rw-p 00000000 00:00 0 
7f6268bff000-7f6268c00000 ---p 00000000 00:00 0 
7f6268c00000-7f6269800000 rw-p 00000000 00:00 0 
7f6269bf8000-7f6269bf9000 ---p 00000000 00:00 0 
7f6269bf9000-7f626a3f9000 rw-p 00000000 00:00 0 
7f626c800000-7f626e400000 rw-p 00000000 00:00 0 
7f626e7fe000-7f626e7ff000 ---p 00000000 00:00 0 
7f626e7ff000-7f626efff000 rw-p 00000000 00:00 0 
7f626efff000-7f626f000000 ---p 00000000 00:00 0 
7f626f000000-7f626fc00000 rw-p 00000000 00:00 0 
7f626ffff000-7f6270000000 ---p 00000000 00:00 0 
7f6270000000-7f6270c00000 rw-p 00000000 00:00 0 
7f6270ff3000-7f62715be000 r-xp 00000000 09:02 8650949                    /usr/lib/mysql/plugin/ha_rocksdb.so
 
7f62715be000-7f62717be000 ---p 005cb000 09:02 8650949                    /usr/lib/mysql/plugin/ha_rocksdb.so
 
7f62717be000-7f62717d1000 r--p 005cb000 09:02 8650949                    /usr/lib/mysql/plugin/ha_rocksdb.so
 
7f62717d1000-7f62717d7000 rw-p 005de000 09:02 8650949                    /usr/lib/mysql/plugin/ha_rocksdb.so
 
7f62717d7000-7f6271800000 rw-p 00000000 00:00 0 
7f6271800000-7f627a000000 rw-p 00000000 00:00 0 
7f627a1f7000-7f627a1fe000 r-xp 00000000 09:02 8791975                    /usr/lib/x86_64-linux-gnu/libsnappy.so.1.3.0
 
7f627a1fe000-7f627a3fd000 ---p 00007000 09:02 8791975                    /usr/lib/x86_64-linux-gnu/libsnappy.so.1.3.0
 
7f627a3fd000-7f627a3fe000 r--p 00006000 09:02 8791975                    /usr/lib/x86_64-linux-gnu/libsnappy.so.1.3.0
 
7f627a3fe000-7f627a3ff000 rw-p 00007000 09:02 8791975                    /usr/lib/x86_64-linux-gnu/libsnappy.so.1.3.0
 
7f627a3ff000-7f627a400000 ---p 00000000 00:00 0 
7f627a400000-7f62fac00000 rw-p 00000000 00:00 0 
7f62fadca000-7f62fadd4000 r-xp 00000000 09:02 3279609                    /lib/x86_64-linux-gnu/libnss_files-2.24.so
 
7f62fadd4000-7f62fafd4000 ---p 0000a000 09:02 3279609                    /lib/x86_64-linux-gnu/libnss_files-2.24.so
 
7f62fafd4000-7f62fafd5000 r--p 0000a000 09:02 3279609                    /lib/x86_64-linux-gnu/libnss_files-2.24.so
 
7f62fafd5000-7f62fafd6000 rw-p 0000b000 09:02 3279609                    /lib/x86_64-linux-gnu/libnss_files-2.24.so
 
7f62fafd6000-7f62fafdc000 rw-p 00000000 00:00 0 
7f62fafdc000-7f62fafe7000 r-xp 00000000 09:02 3279709                    /lib/x86_64-linux-gnu/libnss_nis-2.24.so
 
7f62fafe7000-7f62fb1e6000 ---p 0000b000 09:02 3279709                    /lib/x86_64-linux-gnu/libnss_nis-2.24.so
 
7f62fb1e6000-7f62fb1e7000 r--p 0000a000 09:02 3279709                    /lib/x86_64-linux-gnu/libnss_nis-2.24.so
 
7f62fb1e7000-7f62fb1e8000 rw-p 0000b000 09:02 3279709                    /lib/x86_64-linux-gnu/libnss_nis-2.24.so
 
7f62fb1e8000-7f62fb1fc000 r-xp 00000000 09:02 3279667                    /lib/x86_64-linux-gnu/libnsl-2.24.so
 
7f62fb1fc000-7f62fb3fc000 ---p 00014000 09:02 3279667                    /lib/x86_64-linux-gnu/libnsl-2.24.so
 
7f62fb3fc000-7f62fb3fd000 r--p 00014000 09:02 3279667                    /lib/x86_64-linux-gnu/libnsl-2.24.so
 
7f62fb3fd000-7f62fb3fe000 rw-p 00015000 09:02 3279667                    /lib/x86_64-linux-gnu/libnsl-2.24.so
 
7f62fb3fe000-7f62fb400000 rw-p 00000000 00:00 0 
7f62fb400000-7f62fbc00000 rw-p 00000000 00:00 0 
7f62fbd4a000-7f62fbd51000 r-xp 00000000 09:02 3279662                    /lib/x86_64-linux-gnu/libnss_compat-2.24.so
 
7f62fbd51000-7f62fbf50000 ---p 00007000 09:02 3279662                    /lib/x86_64-linux-gnu/libnss_compat-2.24.so
 
7f62fbf50000-7f62fbf51000 r--p 00006000 09:02 3279662                    /lib/x86_64-linux-gnu/libnss_compat-2.24.so
 
7f62fbf51000-7f62fbf52000 rw-p 00007000 09:02 3279662                    /lib/x86_64-linux-gnu/libnss_compat-2.24.so
 
7f62fbf52000-7f62fbf65000 r-xp 00000000 09:02 3279659                    /lib/x86_64-linux-gnu/libgpg-error.so.0.21.0
 
7f62fbf65000-7f62fc164000 ---p 00013000 09:02 3279659                    /lib/x86_64-linux-gnu/libgpg-error.so.0.21.0
 
7f62fc164000-7f62fc165000 r--p 00012000 09:02 3279659                    /lib/x86_64-linux-gnu/libgpg-error.so.0.21.0
 
7f62fc165000-7f62fc166000 rw-p 00013000 09:02 3279659                    /lib/x86_64-linux-gnu/libgpg-error.so.0.21.0
 
7f62fc166000-7f62fc1d8000 r-xp 00000000 09:02 3279691                    /lib/x86_64-linux-gnu/libpcre.so.3.13.3
 
7f62fc1d8000-7f62fc3d7000 ---p 00072000 09:02 3279691                    /lib/x86_64-linux-gnu/libpcre.so.3.13.3
 
7f62fc3d7000-7f62fc3d8000 r--p 00071000 09:02 3279691                    /lib/x86_64-linux-gnu/libpcre.so.3.13.3
 
7f62fc3d8000-7f62fc3d9000 rw-p 00072000 09:02 3279691                    /lib/x86_64-linux-gnu/libpcre.so.3.13.3
 
7f62fc3d9000-7f62fc3ef000 r-xp 00000000 09:02 3279689                    /lib/x86_64-linux-gnu/libgcc_s.so.1
 
7f62fc3ef000-7f62fc5ee000 ---p 00016000 09:02 3279689                    /lib/x86_64-linux-gnu/libgcc_s.so.1
 
7f62fc5ee000-7f62fc5ef000 r--p 00015000 09:02 3279689                    /lib/x86_64-linux-gnu/libgcc_s.so.1
 
7f62fc5ef000-7f62fc5f0000 rw-p 00016000 09:02 3279689                    /lib/x86_64-linux-gnu/libgcc_s.so.1
 
7f62fc5f0000-7f62fc6f7000 r-xp 00000000 09:02 3279696                    /lib/x86_64-linux-gnu/libgcrypt.so.20.1.6
 
7f62fc6f7000-7f62fc8f7000 ---p 00107000 09:02 3279696                    /lib/x86_64-linux-gnu/libgcrypt.so.20.1.6
 
7f62fc8f7000-7f62fc8f9000 r--p 00107000 09:02 3279696                    /lib/x86_64-linux-gnu/libgcrypt.so.20.1.6
 
7f62fc8f9000-7f62fc900000 rw-p 00109000 09:02 3279696                    /lib/x86_64-linux-gnu/libgcrypt.so.20.1.6
 
7f62fc900000-7f62fc911000 r-xp 00000000 09:02 8789557                    /usr/lib/x86_64-linux-gnu/liblz4.so.1.7.1
 
7f62fc911000-7f62fcb10000 ---p 00011000 09:02 8789557                    /usr/lib/x86_64-linux-gnu/liblz4.so.1.7.1
 
7f62fcb10000-7f62fcb11000 r--p 00010000 09:02 8789557                    /usr/lib/x86_64-linux-gnu/liblz4.so.1.7.1
 
7f62fcb11000-7f62fcb12000 rw-p 00011000 09:02 8789557                    /usr/lib/x86_64-linux-gnu/liblz4.so.1.7.1
 
7f62fcb12000-7f62fcb37000 r-xp 00000000 09:02 3279716                    /lib/x86_64-linux-gnu/liblzma.so.5.2.2
 
7f62fcb37000-7f62fcd36000 ---p 00025000 09:02 3279716                    /lib/x86_64-linux-gnu/liblzma.so.5.2.2
 
7f62fcd36000-7f62fcd37000 r--p 00024000 09:02 3279716                    /lib/x86_64-linux-gnu/liblzma.so.5.2.2
 
7f62fcd37000-7f62fcd38000 rw-p 00025000 09:02 3279716                    /lib/x86_64-linux-gnu/liblzma.so.5.2.2
 
7f62fcd38000-7f62fcd3f000 r-xp 00000000 09:02 3279661                    /lib/x86_64-linux-gnu/librt-2.24.so
 
7f62fcd3f000-7f62fcf3e000 ---p 00007000 09:02 3279661                    /lib/x86_64-linux-gnu/librt-2.24.so
 
7f62fcf3e000-7f62fcf3f000 r--p 00006000 09:02 3279661                    /lib/x86_64-linux-gnu/librt-2.24.so
 
7f62fcf3f000-7f62fcf40000 rw-p 00007000 09:02 3279661                    /lib/x86_64-linux-gnu/librt-2.24.so
 
7f62fcf40000-7f62fcf65000 r-xp 00000000 09:02 3279700                    /lib/x86_64-linux-gnu/libselinux.so.1
 
7f62fcf65000-7f62fd164000 ---p 00025000 09:02 3279700                    /lib/x86_64-linux-gnu/libselinux.so.1
 
7f62fd164000-7f62fd165000 r--p 00024000 09:02 3279700                    /lib/x86_64-linux-gnu/libselinux.so.1
 
7f62fd165000-7f62fd166000 rw-p 00025000 09:02 3279700                    /lib/x86_64-linux-gnu/libselinux.so.1
 
7f62fd166000-7f62fd168000 rw-p 00000000 00:00 0 
7f62fd168000-7f62fd2fd000 r-xp 00000000 09:02 3279626                    /lib/x86_64-linux-gnu/libc-2.24.so
 
7f62fd2fd000-7f62fd4fd000 ---p 00195000 09:02 3279626                    /lib/x86_64-linux-gnu/libc-2.24.so
 
7f62fd4fd000-7f62fd501000 r--p 00195000 09:02 3279626                    /lib/x86_64-linux-gnu/libc-2.24.so
 
7f62fd501000-7f62fd503000 rw-p 00199000 09:02 3279626                    /lib/x86_64-linux-gnu/libc-2.24.so
 
7f62fd503000-7f62fd507000 rw-p 00000000 00:00 0 
7f62fd507000-7f62fd60a000 r-xp 00000000 09:02 3279580                    /lib/x86_64-linux-gnu/libm-2.24.so
 
7f62fd60a000-7f62fd809000 ---p 00103000 09:02 3279580                    /lib/x86_64-linux-gnu/libm-2.24.so
 
7f62fd809000-7f62fd80a000 r--p 00102000 09:02 3279580                    /lib/x86_64-linux-gnu/libm-2.24.so
 
7f62fd80a000-7f62fd80b000 rw-p 00103000 09:02 3279580                    /lib/x86_64-linux-gnu/libm-2.24.so
 
7f62fd80b000-7f62fd97d000 r-xp 00000000 09:02 8789485                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22
 
7f62fd97d000-7f62fdb7d000 ---p 00172000 09:02 8789485                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22
 
7f62fdb7d000-7f62fdb87000 r--p 00172000 09:02 8789485                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22
 
7f62fdb87000-7f62fdb89000 rw-p 0017c000 09:02 8789485                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22
 
7f62fdb89000-7f62fdb8d000 rw-p 00000000 00:00 0 
7f62fdb8d000-7f62fdb90000 r-xp 00000000 09:02 3279685                    /lib/x86_64-linux-gnu/libdl-2.24.so
 
7f62fdb90000-7f62fdd8f000 ---p 00003000 09:02 3279685                    /lib/x86_64-linux-gnu/libdl-2.24.so
 
7f62fdd8f000-7f62fdd90000 r--p 00002000 09:02 3279685                    /lib/x86_64-linux-gnu/libdl-2.24.so
 
7f62fdd90000-7f62fdd91000 rw-p 00003000 09:02 3279685                    /lib/x86_64-linux-gnu/libdl-2.24.so
 
7f62fdd91000-7f62fdffb000 r-xp 00000000 09:02 8783685                    /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
 
7f62fdffb000-7f62fe1fb000 ---p 0026a000 09:02 8783685                    /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
 
7f62fe1fb000-7f62fe219000 r--p 0026a000 09:02 8783685                    /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
 
7f62fe219000-7f62fe227000 rw-p 00288000 09:02 8783685                    /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
 
7f62fe227000-7f62fe22a000 rw-p 00000000 00:00 0 
7f62fe22a000-7f62fe28d000 r-xp 00000000 09:02 8790549                    /usr/lib/x86_64-linux-gnu/libssl.so.1.1
 
7f62fe28d000-7f62fe48c000 ---p 00063000 09:02 8790549                    /usr/lib/x86_64-linux-gnu/libssl.so.1.1
 
7f62fe48c000-7f62fe490000 r--p 00062000 09:02 8790549                    /usr/lib/x86_64-linux-gnu/libssl.so.1.1
 
7f62fe490000-7f62fe496000 rw-p 00066000 09:02 8790549                    /usr/lib/x86_64-linux-gnu/libssl.so.1.1
 
7f62fe496000-7f62fe49e000 r-xp 00000000 09:02 3279727                    /lib/x86_64-linux-gnu/libcrypt-2.24.so
 
7f62fe49e000-7f62fe69e000 ---p 00008000 09:02 3279727                    /lib/x86_64-linux-gnu/libcrypt-2.24.so
 
7f62fe69e000-7f62fe69f000 r--p 00008000 09:02 3279727                    /lib/x86_64-linux-gnu/libcrypt-2.24.so
 
7f62fe69f000-7f62fe6a0000 rw-p 00009000 09:02 3279727                    /lib/x86_64-linux-gnu/libcrypt-2.24.so
 
7f62fe6a0000-7f62fe6ce000 rw-p 00000000 00:00 0 
7f62fe6ce000-7f62fe6e7000 r-xp 00000000 09:02 3279719                    /lib/x86_64-linux-gnu/libz.so.1.2.8
 
7f62fe6e7000-7f62fe8e6000 ---p 00019000 09:02 3279719                    /lib/x86_64-linux-gnu/libz.so.1.2.8
 
7f62fe8e6000-7f62fe8e7000 r--p 00018000 09:02 3279719                    /lib/x86_64-linux-gnu/libz.so.1.2.8
 
7f62fe8e7000-7f62fe8e8000 rw-p 00019000 09:02 3279719                    /lib/x86_64-linux-gnu/libz.so.1.2.8
 
7f62fe8e8000-7f62fe8f2000 r-xp 00000000 09:02 8789529                    /usr/lib/x86_64-linux-gnu/libnuma.so.1.0.0
 
7f62fe8f2000-7f62feaf1000 ---p 0000a000 09:02 8789529                    /usr/lib/x86_64-linux-gnu/libnuma.so.1.0.0
 
7f62feaf1000-7f62feaf2000 r--p 00009000 09:02 8789529                    /usr/lib/x86_64-linux-gnu/libnuma.so.1.0.0
 
7f62feaf2000-7f62feaf3000 rw-p 0000a000 09:02 8789529                    /usr/lib/x86_64-linux-gnu/libnuma.so.1.0.0
 
7f62feaf3000-7f62feaf4000 r-xp 00000000 09:02 3276936                    /lib/x86_64-linux-gnu/libaio.so.1.0.1
 
7f62feaf4000-7f62fecf3000 ---p 00001000 09:02 3276936                    /lib/x86_64-linux-gnu/libaio.so.1.0.1
 
7f62fecf3000-7f62fecf4000 r--p 00000000 09:02 3276936                    /lib/x86_64-linux-gnu/libaio.so.1.0.1
 
7f62fecf4000-7f62fecf5000 rw-p 00000000 00:00 0 
7f62fecf5000-7f62fed0d000 r-xp 00000000 09:02 3279608                    /lib/x86_64-linux-gnu/libpthread-2.24.so
 
7f62fed0d000-7f62fef0c000 ---p 00018000 09:02 3279608                    /lib/x86_64-linux-gnu/libpthread-2.24.so
 
7f62fef0c000-7f62fef0d000 r--p 00017000 09:02 3279608                    /lib/x86_64-linux-gnu/libpthread-2.24.so
 
7f62fef0d000-7f62fef0e000 rw-p 00018000 09:02 3279608                    /lib/x86_64-linux-gnu/libpthread-2.24.so
 
7f62fef0e000-7f62fef12000 rw-p 00000000 00:00 0 
7f62fef12000-7f62fef46000 r-xp 00000000 09:02 8783246                    /usr/lib/x86_64-linux-gnu/libjemalloc.so.1
 
7f62fef46000-7f62ff145000 ---p 00034000 09:02 8783246                    /usr/lib/x86_64-linux-gnu/libjemalloc.so.1
 
7f62ff145000-7f62ff147000 r--p 00033000 09:02 8783246                    /usr/lib/x86_64-linux-gnu/libjemalloc.so.1
 
7f62ff147000-7f62ff148000 rw-p 00035000 09:02 8783246                    /usr/lib/x86_64-linux-gnu/libjemalloc.so.1
 
7f62ff148000-7f62ff149000 rw-p 00000000 00:00 0 
7f62ff149000-7f62ff16c000 r-xp 00000000 09:02 3279590                    /lib/x86_64-linux-gnu/ld-2.24.so
 
7f62ff2cf000-7f62ff2d8000 rw-p 00000000 00:00 0 
7f62ff2d8000-7f62ff35c000 r-xp 00000000 09:02 3276939                    /lib/x86_64-linux-gnu/libsystemd.so.0.17.0
 
7f62ff35c000-7f62ff35d000 ---p 00084000 09:02 3276939                    /lib/x86_64-linux-gnu/libsystemd.so.0.17.0
 
7f62ff35d000-7f62ff360000 r--p 00084000 09:02 3276939                    /lib/x86_64-linux-gnu/libsystemd.so.0.17.0
 
7f62ff360000-7f62ff361000 rw-p 00087000 09:02 3276939                    /lib/x86_64-linux-gnu/libsystemd.so.0.17.0
 
7f62ff361000-7f62ff362000 rw-p 00000000 00:00 0 
7f62ff362000-7f62ff364000 rw-p 00000000 00:00 0 
7f62ff367000-7f62ff368000 rw-p 00000000 00:00 0 
7f62ff369000-7f62ff36c000 rw-p 00000000 00:00 0 
7f62ff36c000-7f62ff36d000 r--p 00023000 09:02 3279590                    /lib/x86_64-linux-gnu/ld-2.24.so
 
7f62ff36d000-7f62ff36e000 rw-p 00024000 09:02 3279590                    /lib/x86_64-linux-gnu/ld-2.24.so
 
7f62ff36e000-7f62ff36f000 rw-p 00000000 00:00 0 
7ffe82616000-7ffe82637000 rw-p 00000000 00:00 0                          [stack]
 
7ffe82797000-7ffe82799000 r--p 00000000 00:00 0                          [vvar]
 
7ffe82799000-7ffe8279b000 r-xp 00000000 00:00 0                          [vdso]
 
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
 
/var/lib/dpkg/info/mariadb-server-10.2.postinst: line 56: 13775 Done                    echo "$replace_query"
 
     13776 Aborted                 | $MYSQL_BOOTSTRAP 2>&1
 
     13777                       | $ERR_LOGGER
 
Job for mariadb.service failed because a fatal signal was delivered to the control process.
 
See "systemctl status mariadb.service" and "journalctl -xe" for details.
 
invoke-rc.d: initscript mysql, action "start" failed.
 
● mariadb.service - MariaDB 10.2.21 database server
 
   Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
 
  Drop-In: /etc/systemd/system/mariadb.service.d
 
           └─migrated-from-my.cnf-settings.conf
 
   Active: failed (Result: signal) since Wed 2019-01-16 12:09:19 CET; 15ms ago
 
     Docs: man:mysqld(8)
 
           https://mariadb.com/kb/en/library/systemd/
 
  Process: 14225 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=killed, signal=ABRT)
 
  Process: 13879 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, status=0/SUCCESS)
 
  Process: 13876 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
 
  Process: 13874 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
 
 Main PID: 14225 (code=killed, signal=ABRT)
 
    Tasks: 2 (limit: 4915)
 
   CGroup: /system.slice/mariadb.service
 
           ├─13633 /usr/sbin/mysqld
 
           └─14260 /usr/sbin/mysqld
 
 
 
Jan 16 12:06:07 DBMaster systemd[1]: Starting MariaDB 10.2.21 database server...
 
Jan 16 12:06:08 DBMaster mysqld[14225]: 2019-01-16 12:06:08 140524373297408 [Note] /usr/sbin/mysqld (mysqld 10.2.21-MariaDB-1…14225 ...
 
Jan 16 12:06:19 DBMaster systemd[1]: mariadb.service: Main process exited, code=killed, status=6/ABRT
 
Jan 16 12:07:49 DBMaster systemd[1]: mariadb.service: State 'stop-sigterm' timed out. Skipping SIGKILL.
 
Jan 16 12:09:19 DBMaster systemd[1]: mariadb.service: State 'stop-final-sigterm' timed out. Skipping SIGKILL. Entering failed mode.
 
Jan 16 12:09:19 DBMaster systemd[1]: Failed to start MariaDB 10.2.21 database server.
 
Jan 16 12:09:19 DBMaster systemd[1]: mariadb.service: Unit entered failed state.
 
Jan 16 12:09:19 DBMaster systemd[1]: mariadb.service: Failed with result 'signal'.
 
Hint: Some lines were ellipsized, use -l to show in full.
 
dpkg: error processing package mariadb-server-10.2 (--configure):
 
 subprocess installed post-installation script returned error exit status 1
 
dpkg: dependency problems prevent configuration of mariadb-server:
 
 mariadb-server depends on mariadb-server-10.2 (>= 10.2.21+maria~stretch); however:
 
  Package mariadb-server-10.2 is not configured yet.
 
 
 
dpkg: error processing package mariadb-server (--configure):
 
 dependency problems - leaving unconfigured
 
dpkg: dependency problems prevent configuration of mariadb-server-10.2-dbgsym:
 
 mariadb-server-10.2-dbgsym depends on mariadb-server-10.2 (= 10.2.21+maria~stretch); however:
 
  Package mariadb-server-10.2 is not configured yet.
 
 
 
dpkg: error processing package mariadb-server-10.2-dbgsym (--configure):
 
 dependency problems - leaving unconfigured
 
dpkg: dependency problems prevent configuration of mariadb-plugin-rocksdb:
 
 mariadb-plugin-rocksdb depends on mariadb-server-10.2 (= 10.2.21+maria~stretch); however:
 
  Package mariadb-server-10.2 is not configured yet.
 
 
 
dpkg: error processing package mariadb-plugin-rocksdb (--configure):
 
 dependency problems - leaving unconfigured
 
dpkg: dependency problems prevent configuration of mariadb-plugin-rocksdb-dbgsym:
 
 mariadb-plugin-rocksdb-dbgsym depends on mariadb-plugin-rocksdb (= 10.2.21+maria~stretch); however:
 
  Package mariadb-plugin-rocksdb is not configured yet.
 
 
 
dpkg: error processing package mariadb-plugin-rocksdb-dbgsym (--configure):
 
 dependency problems - leaving unconfigured
 
Errors were encountered while processing:
 
 mariadb-server-10.2
 
 mariadb-server
 
 mariadb-server-10.2-dbgsym
 
 mariadb-plugin-rocksdb
 
 mariadb-plugin-rocksdb-dbgsym
 
E: Sub-process /usr/bin/dpkg returned an error code (1)

Comment by YOSHINORI MATSUNOBU [ 2019-01-16 ]

I took at logs with my colleague, and we think that RocksDB hit some bugs that couldn't recover from WAL. Could you try removing WAL files (rocksdb_wal_dir/*.log, archiving just in case), restarting mysqld and see if it restarts? Uncommitted transactions in WAL will be lost, but I assume it's ok since rocksdb_flush_log_at_trx_commit=0, and if it is a replica, it can recover from master.

Did operating system restart when the problem happened?

Comment by Jonas Krauss [ 2019-01-17 ]

Thanks for having a look and the suggestion. I moved the WAL files out of the folder and restarted the server. Mysqld successfully passed the rocksdb init part and went into the Innodb recovery process. Here is the current status of the mysql log (we are Europe/Berlin timezone):

2019-01-17  9:51:47 139809435699456 [Note] RocksDB: 2 column families found
 
2019-01-17  9:51:47 139809435699456 [Note] RocksDB: Column Families at start:
 
2019-01-17  9:51:47 139809435699456 [Note]   cf=default
 
2019-01-17  9:51:47 139809435699456 [Note]     write_buffer_size=134217728
 
2019-01-17  9:51:47 139809435699456 [Note]     target_file_size_base=268435456
 
2019-01-17  9:51:47 139809435699456 [Note]   cf=__system__
 
2019-01-17  9:51:47 139809435699456 [Note]     write_buffer_size=134217728
 
2019-01-17  9:51:47 139809435699456 [Note]     target_file_size_base=268435456
 
2019-01-17  9:51:48 139809435699456 [Note] RocksDB: Table_store: loaded DDL data for 58 tables
 
2019-01-17  9:51:49 139809435699456 [Note] RocksDB: global statistics using get_sched_indexer_t indexer
 
2019-01-17  9:51:49 139809435699456 [Note] MyRocks storage engine plugin has been successfully initialized.
 
2019-01-17  9:51:49 139809435699456 [Warning] InnoDB: Using innodb_file_format is deprecated and the parameter may be removed in future releases. See https://mariadb.com/kb/en/library/xtradbinnodb-file-format/
 
2019-01-17  9:51:49 139809435699456 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
 
2019-01-17  9:51:49 139809435699456 [Note] InnoDB: Uses event mutexes
 
2019-01-17  9:51:49 139809435699456 [Note] InnoDB: Compressed tables use zlib 1.2.8
 
2019-01-17  9:51:49 139809435699456 [Note] InnoDB: Using Linux native AIO
 
2019-01-17  9:51:49 139809435699456 [Note] InnoDB: Number of pools: 1
 
2019-01-17  9:51:49 139809435699456 [Note] InnoDB: Using SSE2 crc32 instructions
 
2019-01-17  9:51:49 139809435699456 [Note] InnoDB: Initializing buffer pool, total size = 8G, instances = 2, chunk size = 128M
 
2019-01-17  9:51:49 139809435699456 [Note] InnoDB: Completed initialization of buffer pool
 
2019-01-17  9:51:49 139797095999232 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
 
2019-01-17  9:51:49 139809435699456 [Note] InnoDB: Highest supported file format is Barracuda.
 
2019-01-17  9:51:49 139809435699456 [Note] InnoDB: Starting crash recovery from checkpoint LSN=22573949945612
 
2019-01-17  9:51:59 139809435699456 [Note] InnoDB: Transaction 65935829956 was in the XA prepared state.
 
2019-01-17  9:52:00 139809435699456 [Note] InnoDB: Transaction 65935890300 was in the XA prepared state.
 
2019-01-17  9:52:00 139809435699456 [Note] InnoDB: Transaction 65935889783 was in the XA prepared state.
 
2019-01-17  9:52:00 139809435699456 [Note] InnoDB: Transaction 65935889784 was in the XA prepared state.
 
2019-01-17  9:52:00 139809435699456 [Note] InnoDB: Transaction 65935889785 was in the XA prepared state.
 
2019-01-17  9:52:00 139809435699456 [Note] InnoDB: Transaction 65935889790 was in the XA prepared state.
 
2019-01-17  9:52:00 139809435699456 [Note] InnoDB: Transaction 65935889793 was in the XA prepared state.
 
2019-01-17  9:52:00 139809435699456 [Note] InnoDB: Transaction 65935889794 was in the XA prepared state.
 
2019-01-17  9:52:00 139809435699456 [Note] InnoDB: Transaction 65935889796 was in the XA prepared state.
 
2019-01-17  9:52:00 139809435699456 [Note] InnoDB: Transaction 65935890451 was in the XA prepared state.
 
2019-01-17  9:52:00 139809435699456 [Note] InnoDB: Transaction 65935890587 was in the XA prepared state.
 
2019-01-17  9:52:00 139809435699456 [Note] InnoDB: Transaction 65935889811 was in the XA prepared state.
 
2019-01-17  9:52:00 139809435699456 [Note] InnoDB: Transaction 65935889816 was in the XA prepared state.
 
2019-01-17  9:52:00 139809435699456 [Note] InnoDB: Transaction 65935889817 was in the XA prepared state.
 
2019-01-17  9:52:00 139809435699456 [Note] InnoDB: Transaction 65935889818 was in the XA prepared state.
 
2019-01-17  9:52:00 139809435699456 [Note] InnoDB: Transaction 65935889819 was in the XA prepared state.
 
2019-01-17  9:52:00 139809435699456 [Note] InnoDB: Transaction 65935889820 was in the XA prepared state.
 
2019-01-17  9:52:00 139809435699456 [Note] InnoDB: Transaction 65935889821 was in the XA prepared state.
 
2019-01-17  9:52:00 139809435699456 [Note] InnoDB: Transaction 65935889825 was in the XA prepared state.
 
2019-01-17  9:52:00 139809435699456 [Note] InnoDB: Transaction 65935890477 was in the XA prepared state.
 
2019-01-17  9:52:00 139809435699456 [Note] InnoDB: Transaction 65935889848 was in the XA prepared state.
 
2019-01-17  9:52:00 139809435699456 [Note] InnoDB: Transaction 65935889849 was in the XA prepared state.
 
2019-01-17  9:52:00 139809435699456 [Note] InnoDB: Transaction 65935889850 was in the XA prepared state.
 
2019-01-17  9:52:00 139809435699456 [Note] InnoDB: Transaction 65935889851 was in the XA prepared state.
 
2019-01-17  9:52:00 139809435699456 [Note] InnoDB: Transaction 65935889852 was in the XA prepared state.
 
2019-01-17  9:52:01 139809435699456 [Note] InnoDB: Transaction 65935889859 was in the XA prepared state.
 
2019-01-17  9:52:01 139809435699456 [Note] InnoDB: Transaction 65935890381 was in the XA prepared state.
 
2019-01-17  9:52:01 139809435699456 [Note] InnoDB: Transaction 65935890770 was in the XA prepared state.
 
2019-01-17  9:52:01 139809435699456 [Note] InnoDB: Transaction 65935890272 was in the XA prepared state.
 
2019-01-17  9:52:01 139809435699456 [Note] InnoDB: Transaction 65935889729 was in the XA prepared state.
 
2019-01-17  9:52:01 139809435699456 [Note] InnoDB: Transaction 65935890147 was in the XA prepared state.
 
2019-01-17  9:52:01 139809435699456 [Note] InnoDB: Transaction 65935890020 was in the XA prepared state.
 
2019-01-17  9:52:01 139809435699456 [Note] InnoDB: Transaction 65935890405 was in the XA prepared state.
 
2019-01-17  9:52:01 139809435699456 [Note] InnoDB: Transaction 65935890407 was in the XA prepared state.
 
2019-01-17  9:52:01 139809435699456 [Note] InnoDB: Transaction 65935890408 was in the XA prepared state.
 
2019-01-17  9:52:01 139809435699456 [Note] InnoDB: 36 transaction(s) which must be rolled back or cleaned up in total 379048 row operations to undo
 
2019-01-17  9:52:01 139809435699456 [Note] InnoDB: Trx id counter is 65935892480
 
2019-01-17  9:52:01 139809435699456 [Note] InnoDB: Starting final batch to recover 210786 pages from redo log.
 
2019-01-17  9:52:04 139797146355456 [Note] InnoDB: To recover: 207914 pages from log
 
2019-01-17  9:52:19 139797154748160 [Note] InnoDB: To recover: 188846 pages from log
 
2019-01-17  9:52:34 139797163140864 [Note] InnoDB: To recover: 168890 pages from log
 
2019-01-17  9:52:49 139797154748160 [Note] InnoDB: To recover: 150500 pages from log
 
2019-01-17  9:53:04 139797163140864 [Note] InnoDB: To recover: 133295 pages from log
 
2019-01-17  9:53:19 139797163140864 [Note] InnoDB: To recover: 113649 pages from log
 
2019-01-17  9:53:34 139797146355456 [Note] InnoDB: To recover: 95119 pages from log
 
2019-01-17  9:53:49 139797146355456 [Note] InnoDB: To recover: 76231 pages from log
 
2019-01-17  9:54:04 139797163140864 [Note] InnoDB: To recover: 56298 pages from log
 
2019-01-17  9:54:19 139797137962752 [Note] InnoDB: To recover: 35437 pages from log
 
2019-01-17  9:54:34 139797163140864 [Note] InnoDB: To recover: 16051 pages from log
 
2019-01-17  9:54:46 139809435699456 [Note] InnoDB: Last binlog file '/var/mysql_logs/mysql-bin.000644', position 740602989
 
2019-01-17 12:09:01 139809435699456 [Note] InnoDB: Upgrading redo log: 2*402653184 bytes; LSN=22574377492835
 
2019-01-17 12:09:01 139809435699456 [Note] InnoDB: Starting to delete and rewrite log files.
 
2019-01-17 12:09:02 139809435699456 [Note] InnoDB: Setting log file /var/mysql_logs/innodb/ib_logfile101 size to 402653184 bytes
 
2019-01-17 12:09:02 139809435699456 [Note] InnoDB: Setting log file /var/mysql_logs/innodb/ib_logfile1 size to 402653184 bytes
 
2019-01-17 12:09:02 139809435699456 [Note] InnoDB: Renaming log file /var/mysql_logs/innodb/ib_logfile101 to /var/mysql_logs/innodb/ib_logfile0
 
2019-01-17 12:09:02 139809435699456 [Note] InnoDB: New log files created, LSN=22574377492835
 
2019-01-17 12:09:07 139809435699456 [Note] InnoDB: 128 out of 128 rollback segments are active.
 
2019-01-17 12:09:07 139797070870272 [Note] InnoDB: Starting in background the rollback of recovered transactions
 
2019-01-17 12:09:07 139797070870272 [Note] InnoDB: To roll back: 1 transactions, 379048 rows
 
2019-01-17 12:09:07 139809435699456 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
 
2019-01-17 12:09:07 139809435699456 [Note] InnoDB: Creating shared tablespace for temporary tables
 
2019-01-17 12:09:07 139809435699456 [Note] InnoDB: Setting file '/var/local/mysql/data/ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
 
2019-01-17 12:09:07 139809435699456 [Note] InnoDB: File '/var/local/mysql/data/ibtmp1' size is now 12 MB.
 
2019-01-17 12:09:07 139809435699456 [Note] InnoDB: Waiting for purge to start
 
2019-01-17 12:09:07 139809435699456 [Note] InnoDB: 5.7.24 started; log sequence number 22574377492835
 
2019-01-17 12:09:07 139796894668544 [Note] InnoDB: Loading buffer pool(s) from /var/local/mysql/data/ib_buffer_pool
 
2019-01-17 12:09:07 139809435699456 [Note] Plugin 'FEEDBACK' is disabled.
 
2019-01-17 12:09:07 139809435699456 [Note] Recovering after a crash using /var/mysql_logs/mysql-bin
 
2019-01-17 12:09:08 139796894668544 [Note] InnoDB: Buffer pool(s) load completed at 190117 12:09:08
 
2019-01-17 12:09:09 139809435699456 [Note] Starting crash recovery...
 
2019-01-17 12:09:09 139809435699456 [Note] InnoDB: Starting recovery for XA transactions...
 
...
 
2019-01-17 12:48:37 139797070870272 [Note] InnoDB: To roll back: 1 transactions, 16970 rows
 
2019-01-17 12:48:52 139797070870272 [Note] InnoDB: To roll back: 1 transactions, 16509 rows
 
2019-01-17 12:49:07 139797070870272 [Note] InnoDB: To roll back: 1 transactions, 16043 rows
 
2019-01-17 12:49:22 139797070870272 [Note] InnoDB: To roll back: 1 transactions, 15565 rows
 
2019-01-17 12:49:37 139797070870272 [Note] InnoDB: To roll back: 1 transactions, 15046 rows
 
2019-01-17 12:49:52 139797070870272 [Note] InnoDB: To roll back: 1 transactions, 14579 rows
 
2019-01-17 12:50:07 139797070870272 [Note] InnoDB: To roll back: 1 transactions, 1638 rows
 
2019-01-17 12:50:08 139797070870272 [Note] InnoDB: Rolled back recovered transaction 65935842105
 
2019-01-17 12:50:08 139797070870272 [Note] InnoDB: Rollback of non-prepared transactions completed

Log has not changed since 12:50 (we are now at 14:02) but I guess it is doing some internal data cleaning or whatever. I still cannot connect to the mysql server, neither with the cli client nor via network.

Unfortunately I do not know what happened at the time when the problem occurred. It could be the server restarted but I am not sure, I will try to find out. If I find anything I will post here.

Comment by Jonas Krauss [ 2019-01-18 ]

The server has gone through innodb recovery successfully and came back online (without the previously moved WAL entries). It is now replicating from another master and this also seems to go through so far.

I tried to find out what caused the crash, unfortunately I cannot report a success for this matter as of right now. For some reason the mysql log was not archived around the crash which could either be because there simply were not log entries or because the server crashed so badly it did not have a change to log anything. Statuses of the two RAIDs we are using on the server looked ok, so I figured it is rather unlikely that it is a file system or hardware error.

I am speculating that it could have to do with a sort of batch processing which was running in addition to the regular workout during the time of the server crash. This process has not run before so when thinking of potential causes it seems plausible to consider that. This process looping long running INSERT INTO ... SELECT ... FROM ... statements. Each insert took around half an hour, sometimes longer using GROUP_CONCAT among other aggregations. It looked at 1M+ rows in two tables condensing them into 300-500K rows for the insert. If that sounds like something which could be a problem I can deliver more infos on table structures etc.

Comment by Jonas Krauss [ 2019-01-21 ]

Further update with some more observations during replication catching up:

At the point replication is right now it constantly gets a semaphore timeout in innodb, here is the regarding log entry:

InnoDB: ###### Diagnostic info printed to the standard error stream
 
2019-01-21 13:36:32 140482785044224 [Warning] InnoDB: A long semaphore wait:
 
--Thread 140495150864128 has waited at fts0fts.cc line 3612 for 799.00 seconds the semaphore:
 
X-lock on RW-latch at 0x7fc794465470 created in file fts0fts.cc line 633
 
a writer (thread id 140482751473408) has reserved it in mode  exclusive
 
number of readers 0, waiters flag 1, lock_word: 0
 
Last time read locked in file not yet reserved line 0
 
Last time write locked in file fts0fts.cc line 4367
 
2019-01-21 13:36:32 140482785044224 [Note] InnoDB: A semaphore wait:
 
--Thread 140495150864128 has waited at fts0fts.cc line 3612 for 799.00 seconds the semaphore:
 
X-lock on RW-latch at 0x7fc794465470 created in file fts0fts.cc line 633
 
a writer (thread id 140482751473408) has reserved it in mode  exclusive
 
number of readers 0, waiters flag 1, lock_word: 0
 
Last time read locked in file not yet reserved line 0
 
Last time write locked in file fts0fts.cc line 4367
 
InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info:
 
InnoDB: Pending reads 1, writes 0
 
 
 
=====================================
 
2019-01-21 13:36:48 0x7fc4b1ffd700 INNODB MONITOR OUTPUT
 
=====================================
 
Per second averages calculated from the last 20 seconds
 
-----------------
 
BACKGROUND THREAD
 
-----------------
 
srv_master_thread loops: 1291 srv_active, 0 srv_shutdown, 3433 srv_idle
 
srv_master_thread log flush and writes: 4724
 
----------
 
SEMAPHORES
 
----------
 
OS WAIT ARRAY INFO: reservation count 16380
 
--Thread 140495150864128 has waited at fts0fts.cc line 3612 for 815.00 seconds the semaphore:
 
X-lock on RW-latch at 0x7fc794465470 created in file fts0fts.cc line 633
 
a writer (thread id 140482751473408) has reserved it in mode  exclusive
 
number of readers 0, waiters flag 1, lock_word: 0
 
Last time read locked in file not yet reserved line 0
 
Last time write locked in file fts0fts.cc line 4367
 
OS WAIT ARRAY INFO: signal count 16042
 
RW-shared spins 0, rounds 28411, OS waits 13759
 
RW-excl spins 0, rounds 6698, OS waits 154
 
RW-sx spins 218, rounds 1341, OS waits 11
 
Spin rounds per wait: 28411.00 RW-shared, 6698.00 RW-excl, 6.15 RW-sx
 
------------
 
TRANSACTIONS
 
------------
 
Trx id counter 65957160689
 
Purge done for trx's n:o < 65957160689 undo n:o < 0 state: running but idle
 
History list length 42
 
LIST OF TRANSACTIONS FOR EACH SESSION:
 
---TRANSACTION 421967727486728, not started
 
0 lock struct(s), heap size 1136, 0 row lock(s)
 
---TRANSACTION 421967727482504, not started
 
0 lock struct(s), heap size 1136, 0 row lock(s)
 
---TRANSACTION 421967727478280, not started
 
0 lock struct(s), heap size 1136, 0 row lock(s)
 
---TRANSACTION 421967727474056, not started
 
0 lock struct(s), heap size 1136, 0 row lock(s)
 
---TRANSACTION 421967727465608, not started
 
0 lock struct(s), heap size 1136, 0 row lock(s)
 
---TRANSACTION 421967727469832, not started
 
0 lock struct(s), heap size 1136, 0 row lock(s)
 
---TRANSACTION 421967727444488, not started
 
0 lock struct(s), heap size 1136, 0 row lock(s)
 
---TRANSACTION 421967727452936, not started
 
0 lock struct(s), heap size 1136, 0 row lock(s)
 
---TRANSACTION 421967727448712, not started
 
0 lock struct(s), heap size 1136, 0 row lock(s)
 
---TRANSACTION 65957160681, ACTIVE 815 sec doing SYNC index
 
5 lock struct(s), heap size 1136, 0 row lock(s), undo log entries 435269
 
---TRANSACTION 65957160671, ACTIVE 815 sec adding doc id to FTS DELETED
 
1 lock struct(s), heap size 1136, 0 row lock(s), undo log entries 981
 
---TRANSACTION 65957160646, ACTIVE 842 sec committing
 
190 lock struct(s), heap size 57552, 1347 row lock(s), undo log entries 3932
 
MySQL thread id 465, OS thread handle 140495150864128, query id 5615 Unlocking tables
 
--------
 
FILE I/O
 
--------
 
I/O thread 0 state: waiting for completed aio requests (insert buffer thread)
 
I/O thread 1 state: waiting for completed aio requests (log thread)
 
I/O thread 2 state: waiting for completed aio requests (read thread)
 
I/O thread 3 state: waiting for completed aio requests (read thread)
 
I/O thread 4 state: waiting for completed aio requests (read thread)
 
I/O thread 5 state: waiting for completed aio requests (read thread)
 
I/O thread 6 state: waiting for completed aio requests (write thread)
 
I/O thread 7 state: waiting for completed aio requests (write thread)
 
I/O thread 8 state: waiting for completed aio requests (write thread)
 
I/O thread 9 state: waiting for completed aio requests (write thread)
 
Pending normal aio reads: [0, 0, 0, 0] , aio writes: [0, 0, 0, 1] ,
 
 ibuf aio reads:, log i/o's:, sync i/o's:
 
Pending flushes (fsync) log: 0; buffer pool: 1
 
241972 OS file reads, 156177 OS file writes, 12436 OS fsyncs
 
1 pending reads, 0 pending writes
 
17.70 reads/s, 16384 avg bytes/read, 43.90 writes/s, 2.85 fsyncs/s
 
-------------------------------------
 
INSERT BUFFER AND ADAPTIVE HASH INDEX
 
-------------------------------------
 
Ibuf: size 1, free list len 28619, seg size 28621, 57 merges
 
merged operations:
 
 insert 193, delete mark 956, delete 0
 
discarded operations:
 
 insert 0, delete mark 0, delete 0
 
Hash table size 2212699, node heap has 1 buffer(s)
 
Hash table size 2212699, node heap has 1012 buffer(s)
 
Hash table size 2212699, node heap has 8 buffer(s)
 
Hash table size 2212699, node heap has 153 buffer(s)
 
Hash table size 2212699, node heap has 1 buffer(s)
 
Hash table size 2212699, node heap has 6 buffer(s)
 
Hash table size 2212699, node heap has 2 buffer(s)
 
Hash table size 2212699, node heap has 2 buffer(s)
 
0.00 hash searches/s, 126.44 non-hash searches/s
 
---
 
LOG
 
---
 
Log sequence number 22603203587983
 
Log flushed up to   22603203587983
 
Pages flushed up to 22603102245652
 
Last checkpoint at  22603102245652
 
0 pending log flushes, 0 pending chkp writes
 
5311 log i/o's done, 1.30 log i/o's/second
 
----------------------
 
BUFFER POOL AND MEMORY
 
----------------------
 
Total large memory allocated 8799649792
 
Dictionary memory allocated 1022550
 
Buffer pool size   524288
 
Free buffers       280444
 
Database pages     242656
 
Old database pages 89614
 
Modified db pages  89340
 
Percent of dirty pages(LRU & free pages): 17.079
 
Max dirty pages percent: 90.000
 
Pending reads 1
 
Pending writes: LRU 0, flush list 23, single page 0
 
Pages made young 8, not young 0
 
0.00 youngs/s, 0.00 non-youngs/s
 
Pages read 239441, created 3214, written 148299
 
17.70 reads/s, 1.65 creates/s, 41.90 writes/s
 
Buffer pool hit rate 967 / 1000, young-making rate 0 / 1000 not 0 / 1000
 
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
 
LRU len: 242656, unzip_LRU len: 0
 
I/O sum[0]:cur[1750], unzip sum[0]:cur[0]
 
----------------------
 
INDIVIDUAL BUFFER POOL INFO
 
----------------------
 
---BUFFER POOL 0
 
Buffer pool size   262144
 
Free buffers       139910
 
Database pages     121642
 
Old database pages 44923
 
Modified db pages  44514
 
Percent of dirty pages(LRU & free pages): 17.019
 
Max dirty pages percent: 90.000
 
Pending reads 0
 
Pending writes: LRU 0, flush list 23, single page 0
 
Pages made young 6, not young 0
 
0.00 youngs/s, 0.00 non-youngs/s
 
Pages read 120041, created 1601, written 74548
 
9.50 reads/s, 0.70 creates/s, 22.90 writes/s
 
Buffer pool hit rate 958 / 1000, young-making rate 0 / 1000 not 0 / 1000
 
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
 
LRU len: 121642, unzip_LRU len: 0
 
I/O sum[0]:cur[875], unzip sum[0]:cur[0]
 
---BUFFER POOL 1
 
Buffer pool size   262144
 
Free buffers       140534
 
Database pages     121014
 
Old database pages 44691
 
Modified db pages  44826
 
Percent of dirty pages(LRU & free pages): 17.139
 
Max dirty pages percent: 90.000
 
Pending reads 1
 
Pending writes: LRU 0, flush list 0, single page 0
 
Pages made young 2, not young 0
 
0.00 youngs/s, 0.00 non-youngs/s
 
Pages read 119400, created 1613, written 73751
 
8.20 reads/s, 0.95 creates/s, 19.00 writes/s
 
Buffer pool hit rate 973 / 1000, young-making rate 0 / 1000 not 0 / 1000
 
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
 
LRU len: 121014, unzip_LRU len: 0
 
I/O sum[0]:cur[875], unzip sum[0]:cur[0]
 
--------------
 
ROW OPERATIONS
 
--------------
 
0 queries inside InnoDB, 0 queries in queue
 
0 read views open inside InnoDB
 
Process ID=24747, Main thread ID=140482768258816, state: making checkpoint
 
Number of rows inserted 188, updated 6295, deleted 28, read 12192
 
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s
 
Number of system rows inserted 1314, updated 0, deleted 1313, read 1319
 
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s
 
----------------------------
 
END OF INNODB MONITOR OUTPUT
 
============================

During this state the processlist shows all slave threads either idle or waiting for the previous trx to be committed aside from one thread which hangs in state "Unlocking tables". As this did not change for several minutes I send a shutdown signal to the server which would then lead to the following log eventually:

InnoDB: ###### Diagnostic info printed to the standard error stream
 
2019-01-21 13:39:06 140482785044224 [ERROR] [FATAL] InnoDB: Semaphore wait has lasted > 600 seconds. We intentionally crash the server because it appears to be hung.
 
190121 13:39:06 [ERROR] mysqld got signal 6 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.2.21-MariaDB-10.2.21+maria~stretch-log
 
key_buffer_size=134217728
 
read_buffer_size=131072
 
max_used_connections=15
 
max_threads=3002
 
thread_count=17
 
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6727382 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x0
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x0 thread_stack 0x30000
 
*** buffer overflow detected ***: /usr/sbin/mysqld terminated
 
======= Backtrace: =========
 
/lib/x86_64-linux-gnu/libc.so.6(+0x70bfb)[0x7fc79600dbfb]
 
/lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7fc7960961f7]
 
/lib/x86_64-linux-gnu/libc.so.6(+0xf7330)[0x7fc796094330]
 
/lib/x86_64-linux-gnu/libc.so.6(+0xf916a)[0x7fc79609616a]
 
/usr/sbin/mysqld(my_addr_resolve+0xe2)[0x562b9f47ee82]
 
/usr/sbin/mysqld(my_print_stacktrace+0x1bb)[0x562b9f46573b]
 
/usr/sbin/mysqld(handle_fatal_signal+0x41d)[0x562b9ef04c7d]
 
/lib/x86_64-linux-gnu/libpthread.so.0(+0x110c0)[0x7fc797b3b0c0]
 
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcf)[0x7fc795fcffff]
 
/lib/x86_64-linux-gnu/libc.so.6(abort+0x16a)[0x7fc795fd142a]
 
/usr/sbin/mysqld(+0x93f431)[0x562b9f1eb431]
 
/usr/sbin/mysqld(+0x8e90c6)[0x562b9f1950c6]
 
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7494)[0x7fc797b31494]
 
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7fc796085acf]
 
======= Memory map: ========
 
562b9e8ac000-562b9f989000 r-xp 00000000 09:02 8528713                    /usr/sbin/mysqld
 
562b9fb88000-562b9fc58000 r--p 010dc000 09:02 8528713                    /usr/sbin/mysqld
 
562b9fc58000-562b9fd0f000 rw-p 011ac000 09:02 8528713                    /usr/sbin/mysqld
 
562b9fd0f000-562ba05a1000 rw-p 00000000 00:00 0 
7fc367400000-7fc4a3400000 rw-p 00000000 00:00 0 
7fc4a37ff000-7fc4a3800000 ---p 00000000 00:00 0 
7fc4a3800000-7fc4abc00000 rw-p 00000000 00:00 0 
7fc4abff2000-7fc4abff3000 ---p 00000000 00:00 0 
7fc4abff3000-7fc4ac7f3000 rw-p 00000000 00:00 0 
7fc4ac7f3000-7fc4ac7f4000 ---p 00000000 00:00 0 
7fc4ac7f4000-7fc4acff4000 rw-p 00000000 00:00 0 
7fc4acff4000-7fc4acff5000 ---p 00000000 00:00 0 
7fc4acff5000-7fc4ad7f5000 rw-p 00000000 00:00 0 
7fc4ad7f5000-7fc4ad7f6000 ---p 00000000 00:00 0 
7fc4ad7f6000-7fc4adff6000 rw-p 00000000 00:00 0 
7fc4adff6000-7fc4adff7000 ---p 00000000 00:00 0 
7fc4adff7000-7fc4ae7f7000 rw-p 00000000 00:00 0 
7fc4ae7f7000-7fc4ae7f8000 ---p 00000000 00:00 0 
7fc4ae7f8000-7fc4aeff8000 rw-p 00000000 00:00 0 
7fc4aeff8000-7fc4aeff9000 ---p 00000000 00:00 0 
7fc4aeff9000-7fc4af7f9000 rw-p 00000000 00:00 0 
7fc4af7f9000-7fc4af7fa000 ---p 00000000 00:00 0 
7fc4af7fa000-7fc4afffa000 rw-p 00000000 00:00 0 
7fc4afffa000-7fc4afffb000 ---p 00000000 00:00 0 
7fc4afffb000-7fc4b07fb000 rw-p 00000000 00:00 0 
7fc4b07fb000-7fc4b07fc000 ---p 00000000 00:00 0 
7fc4b07fc000-7fc4b0ffc000 rw-p 00000000 00:00 0 
7fc4b0ffc000-7fc4b0ffd000 ---p 00000000 00:00 0 
7fc4b0ffd000-7fc4b17fd000 rw-p 00000000 00:00 0 
7fc4b17fd000-7fc4b17fe000 ---p 00000000 00:00 0 
7fc4b17fe000-7fc4b1ffe000 rw-p 00000000 00:00 0 
7fc4b1ffe000-7fc4b1fff000 ---p 00000000 00:00 0 
7fc4b1fff000-7fc4b27ff000 rw-p 00000000 00:00 0 
7fc4b27ff000-7fc4b2800000 ---p 00000000 00:00 0 
7fc4b2800000-7fc4b6400000 rw-p 00000000 00:00 0 
7fc4b67ff000-7fc4b6800000 ---p 00000000 00:00 0 
7fc4b6800000-7fc4b7400000 rw-p 00000000 00:00 0 
7fc4b77f5000-7fc4b77f6000 ---p 00000000 00:00 0 
7fc4b77f6000-7fc4b7ff6000 rw-p 00000000 00:00 0 
7fc4b7ff6000-7fc4b7ff7000 ---p 00000000 00:00 0 
7fc4b7ff7000-7fc4b87f7000 rw-p 00000000 00:00 0 
7fc4b87f7000-7fc4b87f8000 ---p 00000000 00:00 0 
7fc4b87f8000-7fc4b8ff8000 rw-p 00000000 00:00 0 
7fc4b8ff8000-7fc4b8ff9000 ---p 00000000 00:00 0 
7fc4b8ff9000-7fc4b97f9000 rw-p 00000000 00:00 0 
7fc4b97f9000-7fc4b97fa000 ---p 00000000 00:00 0 
7fc4b97fa000-7fc4b9ffa000 rw-p 00000000 00:00 0 
7fc4b9ffa000-7fc4b9ffb000 ---p 00000000 00:00 0 
7fc4b9ffb000-7fc4ba7fb000 rw-p 00000000 00:00 0 
7fc4ba7fb000-7fc4ba7fc000 ---p 00000000 00:00 0 
7fc4ba7fc000-7fc4baffc000 rw-p 00000000 00:00 0 
7fc4baffc000-7fc4baffd000 ---p 00000000 00:00 0 
7fc4baffd000-7fc4bb7fd000 rw-p 00000000 00:00 0 
7fc4bb7fd000-7fc4bb7fe000 ---p 00000000 00:00 0 
7fc4bb7fe000-7fc4bbffe000 rw-p 00000000 00:00 0 
7fc4bbffe000-7fc4bbfff000 ---p 00000000 00:00 0 
7fc4bbfff000-7fc4bc7ff000 rw-p 00000000 00:00 0 
7fc4bc7ff000-7fc4bc800000 ---p 00000000 00:00 0 
7fc4bc800000-7fc4e5c00000 rw-p 00000000 00:00 0 
7fc4e5ce0000-7fc4ee400000 rw-p 00000000 00:00 0 
7fc4ee4e0000-7fc4f6c00000 rw-p 00000000 00:00 0 
7fc4f6ce0000-7fc4ff400000 rw-p 00000000 00:00 0 
7fc4ff5c0000-7fc510000000 rw-p 00000000 00:00 0 
7fc5100e0000-7fc518800000 rw-p 00000000 00:00 0 
7fc5189c0000-7fc529400000 rw-p 00000000 00:00 0 
7fc5294e0000-7fc531c00000 rw-p 00000000 00:00 0 
7fc531dc0000-7fc542800000 rw-p 00000000 00:00 0 
7fc5428e0000-7fc54b000000 rw-p 00000000 00:00 0 
7fc54b1c0000-7fc55bc00000 rw-p 00000000 00:00 0 
7fc55bce0000-7fc564400000 rw-p 00000000 00:00 0 
7fc5645c0000-7fc575000000 rw-p 00000000 00:00 0 
7fc5750e0000-7fc57d800000 rw-p 00000000 00:00 0 
7fc57d9c0000-7fc58e400000 rw-p 00000000 00:00 0 
7fc58e4e0000-7fc596c00000 rw-p 00000000 00:00 0 
7fc596dc0000-7fc5a7800000 rw-p 00000000 00:00 0 
7fc5a78e0000-7fc5b0000000 rw-p 00000000 00:00 0 
7fc5b01c0000-7fc5c0c00000 rw-p 00000000 00:00 0 
7fc5c0ce0000-7fc5c9400000 rw-p 00000000 00:00 0 
7fc5c95c0000-7fc5da000000 rw-p 00000000 00:00 0 
7fc5da0e0000-7fc5e2800000 rw-p 00000000 00:00 0 
7fc5e29c0000-7fc6fd000000 rw-p 00000000 00:00 0 
7fc6fd000000-7fc6fdc00000 rw-p 00000000 00:00 0 
7fc6fdc00000-7fc6ff800000 rw-p 00000000 00:00 0 
7fc6ff800000-7fc6ffc00000 rw-p 00000000 00:00 0 
7fc6fffff000-7fc700000000 ---p 00000000 00:00 0 
7fc700000000-7fc700c00000 rw-p 00000000 00:00 0 
7fc700ffd000-7fc700ffe000 ---p 00000000 00:00 0 
7fc700ffe000-7fc7017fe000 rw-p 00000000 00:00 0 
7fc7017fe000-7fc7017ff000 ---p 00000000 00:00 0 
7fc7017ff000-7fc701fff000 rw-p 00000000 00:00 0 
7fc701fff000-7fc702000000 ---p 00000000 00:00 0 
7fc702000000-7fc707000000 rw-p 00000000 00:00 0 
7fc7073fe000-7fc7073ff000 ---p 00000000 00:00 0 
7fc7073ff000-7fc707bff000 rw-p 00000000 00:00 0 
7fc707bff000-7fc707c00000 ---p 00000000 00:00 0 
7fc707c00000-7fc708800000 rw-p 00000000 00:00 0 
7fc708bff000-7fc708c00000 ---p 00000000 00:00 0 
7fc708c00000-7fc709800000 rw-p 00000000 00:00 0 
7fc709bf3000-7fc70a1be000 r-xp 00000000 09:02 8650949                    /usr/lib/mysql/plugin/ha_rocksdb.so
 
7fc70a1be000-7fc70a3be000 ---p 005cb000 09:02 8650949                    /usr/lib/mysql/plugin/ha_rocksdb.so
 
7fc70a3be000-7fc70a3d1000 r--p 005cb000 09:02 8650949                    /usr/lib/mysql/plugin/ha_rocksdb.so
 
7fc70a3d1000-7fc70a3d7000 rw-p 005de000 09:02 8650949                    /usr/lib/mysql/plugin/ha_rocksdb.so
 
7fc70a3d7000-7fc70a400000 rw-p 00000000 00:00 0 
7fc70a400000-7fc712c00000 rw-p 00000000 00:00 0 
7fc712ea8000-7fc712ea9000 ---p 00000000 00:00 0 
7fc712ea9000-7fc712ed9000 rw-p 00000000 00:00 0 
7fc712ed9000-7fc712eda000 ---p 00000000 00:00 0 
7fc712eda000-7fc712f0a000 rw-p 00000000 00:00 0 
7fc712f0a000-7fc712f0b000 ---p 00000000 00:00 0 
7fc712f0b000-7fc712f3b000 rw-p 00000000 00:00 0 
7fc712f3b000-7fc712f3c000 ---p 00000000 00:00 0 
7fc712f3c000-7fc712f6c000 rw-p 00000000 00:00 0 
7fc712f6c000-7fc712f6d000 ---p 00000000 00:00 0 
7fc712f6d000-7fc712f9d000 rw-p 00000000 00:00 0 
7fc712f9d000-7fc712f9e000 ---p 00000000 00:00 0 
7fc712f9e000-7fc712fce000 rw-p 00000000 00:00 0 
7fc712fce000-7fc712fcf000 ---p 00000000 00:00 0 
7fc712fcf000-7fc712fff000 rw-p 00000000 00:00 0 
7fc712fff000-7fc713000000 ---p 00000000 00:00 0 
7fc713000000-7fc793800000 rw-p 00000000 00:00 0 
7fc793801000-7fc793802000 ---p 00000000 00:00 0 
7fc793802000-7fc793832000 rw-p 00000000 00:00 0 
7fc793832000-7fc793833000 ---p 00000000 00:00 0 
7fc793833000-7fc793863000 rw-p 00000000 00:00 0 
7fc793863000-7fc793864000 ---p 00000000 00:00 0 
7fc793864000-7fc793894000 rw-p 00000000 00:00 0 
7fc793894000-7fc793895000 ---p 00000000 00:00 0 
7fc793895000-7fc7938c5000 rw-p 00000000 00:00 0 
7fc7938c5000-7fc7938c6000 ---p 00000000 00:00 0 
7fc7938c6000-7fc7938f6000 rw-p 00000000 00:00 0 
7fc7938f6000-7fc7938f7000 ---p 00000000 00:00 0 
7fc7938f7000-7fc793927000 rw-p 00000000 00:00 0 
7fc793927000-7fc793928000 ---p 00000000 00:00 0 
7fc793928000-7fc793958000 rw-p 00000000 00:00 0 
7fc793958000-7fc793959000 ---p 00000000 00:00 0 
7fc793959000-7fc793989000 rw-p 00000000 00:00 0 
7fc793989000-7fc79398a000 ---p 00000000 00:00 0 
7fc79398a000-7fc7939ba000 rw-p 00000000 00:00 0 
7fc7939ba000-7fc7939c1000 r-xp 00000000 09:02 8791975                    /usr/lib/x86_64-linux-gnu/libsnappy.so.1.3.0
 
7fc7939c1000-7fc793bc0000 ---p 00007000 09:02 8791975                    /usr/lib/x86_64-linux-gnu/libsnappy.so.1.3.0
 
7fc793bc0000-7fc793bc1000 r--p 00006000 09:02 8791975                    /usr/lib/x86_64-linux-gnu/libsnappy.so.1.3.0
 
7fc793bc1000-7fc793bc2000 rw-p 00007000 09:02 8791975                    /usr/lib/x86_64-linux-gnu/libsnappy.so.1.3.0
 
7fc793bc2000-7fc793bcc000 r-xp 00000000 09:02 3279609                    /lib/x86_64-linux-gnu/libnss_files-2.24.so
 
7fc793bcc000-7fc793dcc000 ---p 0000a000 09:02 3279609                    /lib/x86_64-linux-gnu/libnss_files-2.24.so
 
7fc793dcc000-7fc793dcd000 r--p 0000a000 09:02 3279609                    /lib/x86_64-linux-gnu/libnss_files-2.24.so
 
7fc793dcd000-7fc793dce000 rw-p 0000b000 09:02 3279609                    /lib/x86_64-linux-gnu/libnss_files-2.24.so
 
7fc793dce000-7fc793dd4000 rw-p 00000000 00:00 0 
7fc793dd4000-7fc793ddf000 r-xp 00000000 09:02 3279709                    /lib/x86_64-linux-gnu/libnss_nis-2.24.so
 
7fc793ddf000-7fc793fde000 ---p 0000b000 09:02 3279709                    /lib/x86_64-linux-gnu/libnss_nis-2.24.so
 
7fc793fde000-7fc793fdf000 r--p 0000a000 09:02 3279709                    /lib/x86_64-linux-gnu/libnss_nis-2.24.so
 
7fc793fdf000-7fc793fe0000 rw-p 0000b000 09:02 3279709                    /lib/x86_64-linux-gnu/libnss_nis-2.24.so
 
7fc793fe0000-7fc793ff4000 r-xp 00000000 09:02 3279667                    /lib/x86_64-linux-gnu/libnsl-2.24.so
 
7fc793ff4000-7fc7941f4000 ---p 00014000 09:02 3279667                    /lib/x86_64-linux-gnu/libnsl-2.24.so
 
7fc7941f4000-7fc7941f5000 r--p 00014000 09:02 3279667                    /lib/x86_64-linux-gnu/libnsl-2.24.so
 
7fc7941f5000-7fc7941f6000 rw-p 00015000 09:02 3279667                    /lib/x86_64-linux-gnu/libnsl-2.24.so
 
7fc7941f6000-7fc7941f8000 rw-p 00000000 00:00 0 
7fc7941f8000-7fc7941ff000 r-xp 00000000 09:02 3279662                    /lib/x86_64-linux-gnu/libnss_compat-2.24.so
 
7fc7941ff000-7fc7943fe000 ---p 00007000 09:02 3279662                    /lib/x86_64-linux-gnu/libnss_compat-2.24.so
 
7fc7943fe000-7fc7943ff000 r--p 00006000 09:02 3279662                    /lib/x86_64-linux-gnu/libnss_compat-2.24.so
 
7fc7943ff000-7fc794400000 rw-p 00007000 09:02 3279662                    /lib/x86_64-linux-gnu/libnss_compat-2.24.so
 
7fc794400000-7fc794c00000 rw-p 00000000 00:00 0 
7fc794c30000-7fc794c31000 ---p 00000000 00:00 0 
7fc794c31000-7fc794c61000 rw-p 00000000 00:00 0 
7fc794c61000-7fc794c62000 ---p 00000000 00:00 0 
7fc794c62000-7fc794c92000 rw-p 00000000 00:00 0 
7fc794c92000-7fc794c93000 ---p 00000000 00:00 0 
7fc794c93000-7fc794cc3000 rw-p 00000000 00:00 0 
7fc794cc3000-7fc794cc4000 ---p 00000000 00:00 0 
7fc794cc4000-7fc794cf4000 rw-p 00000000 00:00 0 
7fc794cf4000-7fc794cf5000 ---p 00000000 00:00 0 
7fc794cf5000-7fc794d25000 rw-p 00000000 00:00 0 
7fc794d25000-7fc794d26000 ---p 00000000 00:00 0 
7fc794d26000-7fc794d56000 rw-p 00000000 00:00 0 
7fc794d56000-7fc794d57000 ---p 00000000 00:00 0 
7fc794d57000-7fc794d87000 rw-p 00000000 00:00 0 
7fc794d87000-7fc794d9a000 r-xp 00000000 09:02 3279659                    /lib/x86_64-linux-gnu/libgpg-error.so.0.21.0
 
7fc794d9a000-7fc794f99000 ---p 00013000 09:02 3279659                    /lib/x86_64-linux-gnu/libgpg-error.so.0.21.0
 
7fc794f99000-7fc794f9a000 r--p 00012000 09:02 3279659                    /lib/x86_64-linux-gnu/libgpg-error.so.0.21.0
 
7fc794f9a000-7fc794f9b000 rw-p 00013000 09:02 3279659                    /lib/x86_64-linux-gnu/libgpg-error.so.0.21.0
 
7fc794f9b000-7fc79500d000 r-xp 00000000 09:02 3279691                    /lib/x86_64-linux-gnu/libpcre.so.3.13.3
 
7fc79500d000-7fc79520c000 ---p 00072000 09:02 3279691                    /lib/x86_64-linux-gnu/libpcre.so.3.13.3
 
7fc79520c000-7fc79520d000 r--p 00071000 09:02 3279691                    /lib/x86_64-linux-gnu/libpcre.so.3.13.3
 
7fc79520d000-7fc79520e000 rw-p 00072000 09:02 3279691                    /lib/x86_64-linux-gnu/libpcre.so.3.13.3
 
7fc79520e000-7fc795224000 r-xp 00000000 09:02 3279689                    /lib/x86_64-linux-gnu/libgcc_s.so.1
 
7fc795224000-7fc795423000 ---p 00016000 09:02 3279689                    /lib/x86_64-linux-gnu/libgcc_s.so.1
 
7fc795423000-7fc795424000 r--p 00015000 09:02 3279689                    /lib/x86_64-linux-gnu/libgcc_s.so.1
 
7fc795424000-7fc795425000 rw-p 00016000 09:02 3279689                    /lib/x86_64-linux-gnu/libgcc_s.so.1
 
7fc795425000-7fc79552c000 r-xp 00000000 09:02 3279696                    /lib/x86_64-linux-gnu/libgcrypt.so.20.1.6
 
7fc79552c000-7fc79572c000 ---p 00107000 09:02 3279696                    /lib/x86_64-linux-gnu/libgcrypt.so.20.1.6
 
7fc79572c000-7fc79572e000 r--p 00107000 09:02 3279696                    /lib/x86_64-linux-gnu/libgcrypt.so.20.1.6
 
7fc79572e000-7fc795735000 rw-p 00109000 09:02 3279696                    /lib/x86_64-linux-gnu/libgcrypt.so.20.1.6
 
7fc795735000-7fc795746000 r-xp 00000000 09:02 8789557                    /usr/lib/x86_64-linux-gnu/liblz4.so.1.7.1
 
7fc795746000-7fc795945000 ---p 00011000 09:02 8789557                    /usr/lib/x86_64-linux-gnu/liblz4.so.1.7.1
 
7fc795945000-7fc795946000 r--p 00010000 09:02 8789557                    /usr/lib/x86_64-linux-gnu/liblz4.so.1.7.1
 
7fc795946000-7fc795947000 rw-p 00011000 09:02 8789557                    /usr/lib/x86_64-linux-gnu/liblz4.so.1.7.1
 
7fc795947000-7fc79596c000 r-xp 00000000 09:02 3279716                    /lib/x86_64-linux-gnu/liblzma.so.5.2.2
 
7fc79596c000-7fc795b6b000 ---p 00025000 09:02 3279716                    /lib/x86_64-linux-gnu/liblzma.so.5.2.2
 
7fc795b6b000-7fc795b6c000 r--p 00024000 09:02 3279716                    /lib/x86_64-linux-gnu/liblzma.so.5.2.2
 
7fc795b6c000-7fc795b6d000 rw-p 00025000 09:02 3279716                    /lib/x86_64-linux-gnu/liblzma.so.5.2.2
 
7fc795b6d000-7fc795b74000 r-xp 00000000 09:02 3279661                    /lib/x86_64-linux-gnu/librt-2.24.so
 
7fc795b74000-7fc795d73000 ---p 00007000 09:02 3279661                    /lib/x86_64-linux-gnu/librt-2.24.so
 
7fc795d73000-7fc795d74000 r--p 00006000 09:02 3279661                    /lib/x86_64-linux-gnu/librt-2.24.so
 
7fc795d74000-7fc795d75000 rw-p 00007000 09:02 3279661                    /lib/x86_64-linux-gnu/librt-2.24.so
 
7fc795d75000-7fc795d9a000 r-xp 00000000 09:02 3279700                    /lib/x86_64-linux-gnu/libselinux.so.1
 
7fc795d9a000-7fc795f99000 ---p 00025000 09:02 3279700                    /lib/x86_64-linux-gnu/libselinux.so.1
 
7fc795f99000-7fc795f9a000 r--p 00024000 09:02 3279700                    /lib/x86_64-linux-gnu/libselinux.so.1
 
7fc795f9a000-7fc795f9b000 rw-p 00025000 09:02 3279700                    /lib/x86_64-linux-gnu/libselinux.so.1
 
7fc795f9b000-7fc795f9d000 rw-p 00000000 00:00 0 
7fc795f9d000-7fc796132000 r-xp 00000000 09:02 3279626                    /lib/x86_64-linux-gnu/libc-2.24.so
 
7fc796132000-7fc796332000 ---p 00195000 09:02 3279626                    /lib/x86_64-linux-gnu/libc-2.24.so
 
7fc796332000-7fc796336000 r--p 00195000 09:02 3279626                    /lib/x86_64-linux-gnu/libc-2.24.so
 
7fc796336000-7fc796338000 rw-p 00199000 09:02 3279626                    /lib/x86_64-linux-gnu/libc-2.24.so
 
7fc796338000-7fc79633c000 rw-p 00000000 00:00 0 
7fc79633c000-7fc79643f000 r-xp 00000000 09:02 3279580                    /lib/x86_64-linux-gnu/libm-2.24.so
 
7fc79643f000-7fc79663e000 ---p 00103000 09:02 3279580                    /lib/x86_64-linux-gnu/libm-2.24.so
 
7fc79663e000-7fc79663f000 r--p 00102000 09:02 3279580                    /lib/x86_64-linux-gnu/libm-2.24.so
 
7fc79663f000-7fc796640000 rw-p 00103000 09:02 3279580                    /lib/x86_64-linux-gnu/libm-2.24.so
 
7fc796640000-7fc7967b2000 r-xp 00000000 09:02 8789485                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22
 
7fc7967b2000-7fc7969b2000 ---p 00172000 09:02 8789485                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22
 
7fc7969b2000-7fc7969bc000 r--p 00172000 09:02 8789485                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22
 
7fc7969bc000-7fc7969be000 rw-p 0017c000 09:02 8789485                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22
 
7fc7969be000-7fc7969c2000 rw-p 00000000 00:00 0 
7fc7969c2000-7fc7969c5000 r-xp 00000000 09:02 3279685                    /lib/x86_64-linux-gnu/libdl-2.24.so
 
7fc7969c5000-7fc796bc4000 ---p 00003000 09:02 3279685                    /lib/x86_64-linux-gnu/libdl-2.24.so
 
7fc796bc4000-7fc796bc5000 r--p 00002000 09:02 3279685                    /lib/x86_64-linux-gnu/libdl-2.24.so
 
7fc796bc5000-7fc796bc6000 rw-p 00003000 09:02 3279685                    /lib/x86_64-linux-gnu/libdl-2.24.so
 
7fc796bc6000-7fc796e30000 r-xp 00000000 09:02 8783685                    /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
 
7fc796e30000-7fc797030000 ---p 0026a000 09:02 8783685                    /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
 
7fc797030000-7fc79704e000 r--p 0026a000 09:02 8783685                    /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
 
7fc79704e000-7fc79705c000 rw-p 00288000 09:02 8783685                    /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
 
7fc79705c000-7fc79705f000 rw-p 00000000 00:00 0 
7fc79705f000-7fc7970c2000 r-xp 00000000 09:02 8790549                    /usr/lib/x86_64-linux-gnu/libssl.so.1.1
 
7fc7970c2000-7fc7972c1000 ---p 00063000 09:02 8790549                    /usr/lib/x86_64-linux-gnu/libssl.so.1.1
 
7fc7972c1000-7fc7972c5000 r--p 00062000 09:02 8790549                    /usr/lib/x86_64-linux-gnu/libssl.so.1.1
 
7fc7972c5000-7fc7972cb000 rw-p 00066000 09:02 8790549                    /usr/lib/x86_64-linux-gnu/libssl.so.1.1
 
7fc7972cb000-7fc7972d3000 r-xp 00000000 09:02 3279727                    /lib/x86_64-linux-gnu/libcrypt-2.24.so
 
7fc7972d3000-7fc7974d3000 ---p 00008000 09:02 3279727                    /lib/x86_64-linux-gnu/libcrypt-2.24.so
 
7fc7974d3000-7fc7974d4000 r--p 00008000 09:02 3279727                    /lib/x86_64-linux-gnu/libcrypt-2.24.so
 
7fc7974d4000-7fc7974d5000 rw-p 00009000 09:02 3279727                    /lib/x86_64-linux-gnu/libcrypt-2.24.so
 
7fc7974d5000-7fc797503000 rw-p 00000000 00:00 0 
7fc797503000-7fc79751c000 r-xp 00000000 09:02 3279719                    /lib/x86_64-linux-gnu/libz.so.1.2.8
 
7fc79751c000-7fc79771b000 ---p 00019000 09:02 3279719                    /lib/x86_64-linux-gnu/libz.so.1.2.8
 
7fc79771b000-7fc79771c000 r--p 00018000 09:02 3279719                    /lib/x86_64-linux-gnu/libz.so.1.2.8
 
7fc79771c000-7fc79771d000 rw-p 00019000 09:02 3279719                    /lib/x86_64-linux-gnu/libz.so.1.2.8
 
7fc79771d000-7fc797727000 r-xp 00000000 09:02 8789529                    /usr/lib/x86_64-linux-gnu/libnuma.so.1.0.0
 
7fc797727000-7fc797926000 ---p 0000a000 09:02 8789529                    /usr/lib/x86_64-linux-gnu/libnuma.so.1.0.0
 
7fc797926000-7fc797927000 r--p 00009000 09:02 8789529                    /usr/lib/x86_64-linux-gnu/libnuma.so.1.0.0
 
7fc797927000-7fc797928000 rw-p 0000a000 09:02 8789529                    /usr/lib/x86_64-linux-gnu/libnuma.so.1.0.0
 
7fc797928000-7fc797929000 r-xp 00000000 09:02 3276936                    /lib/x86_64-linux-gnu/libaio.so.1.0.1
 
7fc797929000-7fc797b28000 ---p 00001000 09:02 3276936                    /lib/x86_64-linux-gnu/libaio.so.1.0.1
 
7fc797b28000-7fc797b29000 r--p 00000000 09:02 3276936                    /lib/x86_64-linux-gnu/libaio.so.1.0.1
 
7fc797b29000-7fc797b2a000 rw-p 00000000 00:00 0 
7fc797b2a000-7fc797b42000 r-xp 00000000 09:02 3279608                    /lib/x86_64-linux-gnu/libpthread-2.24.so
 
7fc797b42000-7fc797d41000 ---p 00018000 09:02 3279608                    /lib/x86_64-linux-gnu/libpthread-2.24.so
 
7fc797d41000-7fc797d42000 r--p 00017000 09:02 3279608                    /lib/x86_64-linux-gnu/libpthread-2.24.so
 
7fc797d42000-7fc797d43000 rw-p 00018000 09:02 3279608                    /lib/x86_64-linux-gnu/libpthread-2.24.so
 
7fc797d43000-7fc797d47000 rw-p 00000000 00:00 0 
7fc797d47000-7fc797d7b000 r-xp 00000000 09:02 8783246                    /usr/lib/x86_64-linux-gnu/libjemalloc.so.1
 
7fc797d7b000-7fc797f7a000 ---p 00034000 09:02 8783246                    /usr/lib/x86_64-linux-gnu/libjemalloc.so.1
 
7fc797f7a000-7fc797f7c000 r--p 00033000 09:02 8783246                    /usr/lib/x86_64-linux-gnu/libjemalloc.so.1
 
7fc797f7c000-7fc797f7d000 rw-p 00035000 09:02 8783246                    /usr/lib/x86_64-linux-gnu/libjemalloc.so.1
 
7fc797f7d000-7fc797f7e000 rw-p 00000000 00:00 0 
7fc797f7e000-7fc797fa1000 r-xp 00000000 09:02 3279590                    /lib/x86_64-linux-gnu/ld-2.24.so
 
7fc797fac000-7fc797fad000 ---p 00000000 00:00 0 
7fc797fad000-7fc797fdd000 rw-p 00000000 00:00 0 
7fc797fdd000-7fc797fde000 ---p 00000000 00:00 0 
7fc797fde000-7fc79800e000 rw-p 00000000 00:00 0 
7fc79800e000-7fc79800f000 ---p 00000000 00:00 0 
7fc79800f000-7fc79803f000 rw-p 00000000 00:00 0 
7fc79803f000-7fc798040000 ---p 00000000 00:00 0 
7fc798040000-7fc798070000 rw-p 00000000 00:00 0 
7fc798070000-7fc798071000 ---p 00000000 00:00 0 
7fc798071000-7fc7980a1000 rw-p 00000000 00:00 0 
7fc7980a1000-7fc7980a2000 ---p 00000000 00:00 0 
7fc7980a2000-7fc7980d2000 rw-p 00000000 00:00 0 
7fc7980d2000-7fc7980d7000 rw-s 00000000 00:0e 2691719                    /[aio] (deleted)
 
7fc7980d7000-7fc7980dc000 rw-s 00000000 00:0e 2691718                    /[aio] (deleted)
 
7fc7980dc000-7fc7980e1000 rw-s 00000000 00:0e 2691717                    /[aio] (deleted)
 
7fc7980e1000-7fc7980e6000 rw-s 00000000 00:0e 2691716                    /[aio] (deleted)
 
7fc7980e6000-7fc7980eb000 rw-s 00000000 00:0e 2691715                    /[aio] (deleted)
 
7fc7980eb000-7fc7980f0000 rw-s 00000000 00:0e 2691714                    /[aio] (deleted)
 
7fc7980f0000-7fc7980f5000 rw-s 00000000 00:0e 2691713                    /[aio] (deleted)
 
7fc7980f5000-7fc7980fa000 rw-s 00000000 00:0e 2691712                    /[aio] (deleted)
 
7fc7980fa000-7fc7980ff000 rw-s 00000000 00:0e 2691711                    /[aio] (deleted)
 
7fc7980ff000-7fc798104000 rw-s 00000000 00:0e 2691710                    /[aio] (deleted)
 
7fc798104000-7fc79810d000 rw-p 00000000 00:00 0 
7fc79810d000-7fc798191000 r-xp 00000000 09:02 3276939                    /lib/x86_64-linux-gnu/libsystemd.so.0.17.0
 
7fc798191000-7fc798192000 ---p 00084000 09:02 3276939                    /lib/x86_64-linux-gnu/libsystemd.so.0.17.0
 
7fc798192000-7fc798195000 r--p 00084000 09:02 3276939                    /lib/x86_64-linux-gnu/libsystemd.so.0.17.0
 
7fc798195000-7fc798196000 rw-p 00087000 09:02 3276939                    /lib/x86_64-linux-gnu/libsystemd.so.0.17.0
 
7fc798196000-7fc798197000 rw-p 00000000 00:00 0 
7fc798197000-7fc798199000 rw-p 00000000 00:00 0 
7fc79819a000-7fc79819b000 rw-p 00000000 00:00 0 
7fc79819b000-7fc79819d000 rw-s 00000000 00:0e 2691720                    /[aio] (deleted)
 
7fc79819d000-7fc79819e000 rw-s 00000000 00:0e 2691709                    /[aio] (deleted)
 
7fc79819e000-7fc7981a1000 rw-p 00000000 00:00 0 
7fc7981a1000-7fc7981a2000 r--p 00023000 09:02 3279590                    /lib/x86_64-linux-gnu/ld-2.24.so
 
7fc7981a2000-7fc7981a3000 rw-p 00024000 09:02 3279590                    /lib/x86_64-linux-gnu/ld-2.24.so
 
7fc7981a3000-7fc7981a4000 rw-p 00000000 00:00 0 
7fff05d6b000-7fff05d8c000 rw-p 00000000 00:00 0                          [stack]
 
7fff05daa000-7fff05dac000 r--p 00000000 00:00 0                          [vvar]
 
7fff05dac000-7fff05dae000 r-xp 00000000 00:00 0                          [vdso]
 
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
 
Fatal signal 11 while backtracing

Does any of this point in a certain direction I could dig deeper into? Let me know if I should post more logging entries or try out other things, thanks.

Comment by Michal Wazgird [ 2019-05-20 ]

I have run into similar problem yesterday. Machine run out of RAM and crashed. After the crash I got similar error. All recovery options failed, I had to start form backup.

This is my log:

Starting MariaDB 10.3.15 database server...
 
2019-05-19 13:10:45 0 [Warning] 'THREAD_CONCURRENCY' is deprecated and will be removed in a future release.
 
2019-05-19 13:10:45 0 [Note] /usr/sbin/mysqld (mysqld 10.3.15-MariaDB-log) starting as process 7559 ...
 
2019-05-19 13:10:45 0 [Note] Using unique option prefix 'myisam_recover' is error-prone and can break in the future. Please use the full name 'myisam-recover-options' instead.
 
2019-05-19 13:10:45 0 [Note] RocksDB: 2 column families found
 
2019-05-19 13:10:45 0 [Note] RocksDB: Column Families at start:
 
2019-05-19 13:10:45 0 [Note]   cf=default
 
2019-05-19 13:10:45 0 [Note]     write_buffer_size=536870912
 
2019-05-19 13:10:45 0 [Note]     target_file_size_base=268435456
 
2019-05-19 13:10:45 0 [Note]   cf=__system__
 
2019-05-19 13:10:45 0 [Note]     write_buffer_size=536870912
 
2019-05-19 13:10:45 0 [Note]     target_file_size_base=268435456
 
mysqld: /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.3.15/storage/rocksdb/rocksdb/db/column_family.cc:1238: rocksdb::ColumnFamilySet::~ColumnFamilySet(): Assertion `last_ref' failed.
 
190519 13:10:59 [ERROR] mysqld got signal 6 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed,
 
something is definitely wrong and this may fail.
 
Server version: 10.3.15-MariaDB-log
 
key_buffer_size=536870912
 
read_buffer_size=268435456
 
max_used_connections=0
 
max_threads=1002
 
thread_count=0
 
It is possible that mysqld could use up to
 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 271423482 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x0
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x0 thread_stack 0x3c000
 
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55741b30b91e]
 
/usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55741ada853f]
 
sigaction.c:0(__restore_rt)[0x7f9519c3e5d0]
 
:0(__GI_raise)[0x7f9517f11207]
 
:0(__GI_abort)[0x7f9517f128f8]
 
:0(__assert_fail_base)[0x7f9517f0a026]
 
:0(__GI___assert_fail)[0x7f9517f0a0d2]
 
/usr/lib64/mysql/plugin/ha_rocksdb.so(_ZN7rocksdb15ColumnFamilySetD1Ev+0x1b9)[0x7f95107fc879]
 
/usr/lib64/mysql/plugin/ha_rocksdb.so(_ZN7rocksdb10VersionSetD1Ev+0x4c)[0x7f95106d9abc]
 
/usr/lib64/mysql/plugin/ha_rocksdb.so(_ZN7rocksdb6DBImpl11CloseHelperEv+0x64c)[0x7f951063da6c]
 
/usr/lib64/mysql/plugin/ha_rocksdb.so(_ZN7rocksdb6DBImplD1Ev+0x6b5)[0x7f951064c405]
 
/usr/lib64/mysql/plugin/ha_rocksdb.so(_ZN7rocksdb6DBImplD0Ev+0x11)[0x7f951064c5f1]
 
/usr/lib64/mysql/plugin/ha_rocksdb.so(_ZN7rocksdb24PessimisticTransactionDBD2Ev+0x160)[0x7f95107bf9b0]
 
/usr/lib64/mysql/plugin/ha_rocksdb.so(_ZN7rocksdb19WriteCommittedTxnDBD0Ev+0x1f)[0x7f95107c4baf]
 
/usr/lib64/mysql/plugin/ha_rocksdb.so(_ZN7rocksdb13TransactionDB6WrapDBEPNS_2DBERKNS_20TransactionDBOptionsERKSt6vectorImSaImEERKS6_IPNS_18ColumnFamilyHandleESaISC_EEPPS0_+0x311)[0x7f95107c0cf1]
 
/usr/lib64/mysql/plugin/ha_rocksdb.so(_ZN7rocksdb13TransactionDB4OpenERKNS_9DBOptionsERKNS_20TransactionDBOptionsERKSsRKSt6vectorINS_22ColumnFamilyDescriptorESaISA_EEPS9_IPNS_18ColumnFamilyHandleESaISG_EEPPS0_+0xa4f)[0x7f95107c1b2f]
 
/usr/lib64/mysql/plugin/ha_rocksdb.so(+0x231f98)[0x7f95105e3f98]
 
/usr/sbin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x64)[0x55741adaabc4]
 
/usr/sbin/mysqld(+0x5ba110)[0x55741abe4110]
 
/usr/sbin/mysqld(_Z11plugin_initPiPPci+0x9ba)[0x55741abe528a]
 
/usr/sbin/mysqld(+0x4eb531)[0x55741ab15531]
 
/usr/sbin/mysqld(_Z11mysqld_mainiPPc+0x4ae)[0x55741ab1bbae]
 
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f9517efd3d5]
 
/usr/sbin/mysqld(+0x4e50cd)[0x55741ab0f0cd]
 
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
 
information that should help you find out what is causing the crash.
 
Writing a core file...
 
Working directory at /db/mysql
 
Resource Limits:
 
Limit                     Soft Limit           Hard Limit           Units
 
Max cpu time              unlimited            unlimited            seconds
 
Max file size             unlimited            unlimited            bytes
 
Max data size             unlimited            unlimited            bytes
 
Max stack size            8388608              unlimited            bytes
 
Max core file size        0                    unlimited            bytes
 
mariadb.service: main process exited, code=killed, status=6/ABRT

Comment by Sergei Petrunia [ 2019-06-03 ]

With 99% chance, this is the same issue as MDEV-19395 (for that issue, now investigation results are available - new RocksDB has a fix and we need to merge it).

Comment by Sergei Petrunia [ 2019-06-21 ]

The fix for this is now in MariaDB's 10.2 branch, it will be merged to other branches. see https://jira.mariadb.org/browse/MDEV-19395?focusedCommentId=129717&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-129717 for details

Comment by Sergei Petrunia [ 2019-06-21 ]

Marking as duplicate of MDEV-19395.

Generated at Thu Feb 08 08:42:37 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.