[MDEV-17985] rocksdb crashes mariadb 10.3 Created: 2018-12-12  Updated: 2018-12-13  Resolved: 2018-12-13

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - RocksDB
Affects Version/s: 10.3.11
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Philip orleans Assignee: Unassigned
Resolution: Not a Bug Votes: 0
Labels: None
Environment:

Centos 7 fully updated


 Description   

2018-12-12 15:32:18 140415786465472 [ERROR] Can't open shared library '/usr/lib64/mysql/plugin/ha_rocksdb.so' (errno: 2, undefined symbol: _ZN7handler11engine_nameEv)
2018-12-12 15:32:18 140415786465472 [ERROR] Couldn't load plugins from 'ha_rocksdb.so'.
2018-12-12 15:32:18 140415786465472 [Note] InnoDB: innodb_page_size=4096
2018-12-12 15:32:18 140415786465472 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2018-12-12 15:32:18 140415786465472 [Note] InnoDB: Uses event mutexes
2018-12-12 15:32:18 140415786465472 [Note] InnoDB: Compressed tables use zlib 1.2.7
2018-12-12 15:32:18 140415786465472 [Note] InnoDB: Using Linux native AIO
2018-12-12 15:32:18 140415786465472 [Note] InnoDB: Number of pools: 1
2018-12-12 15:32:18 140415786465472 [Note] InnoDB: Using SSE2 crc32 instructions
2018-12-12 15:32:18 140415786465472 [Note] InnoDB: Initializing buffer pool, total size = 75G, instances = 8, chunk size = 128M
2018-12-12 15:32:30 140415786465472 [Note] InnoDB: Completed initialization of buffer pool
2018-12-12 15:32:32 140312068454144 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2018-12-12 15:32:32 140415786465472 [Note] InnoDB: Highest supported file format is Barracuda.
2018-12-12 15:32:34 140415786465472 [Note] InnoDB: 128 out of 128 rollback segments are active.
2018-12-12 15:32:34 140415786465472 [Note] InnoDB: Creating shared tablespace for temporary tables
2018-12-12 15:32:34 140415786465472 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2018-12-12 15:32:34 140415786465472 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2018-12-12 15:32:34 140415786465472 [Note] InnoDB: Waiting for purge to start
2018-12-12 15:32:34 140415786465472 [Note] InnoDB: 5.7.24 started; log sequence number 55960749776
2018-12-12 15:32:34 140415786465472 [Note] Plugin 'FEEDBACK' is disabled.
2018-12-12 15:32:34 140415786465472 [ERROR] /usr/sbin/mysqld: unknown variable 'rocksdb_commit_in_the_middle=1'
2018-12-12 15:32:34 140415786465472 [ERROR] Aborting

 Comments   
Comment by Elena Stepanova [ 2018-12-12 ]

Please paste the output of rpm -qa | grep -i maria

Comment by Philip orleans [ 2018-12-13 ]

MariaDB-server-10.3.11-1.el7.centos.x86_64
MariaDB-common-10.3.11-1.el7.centos.x86_64
MariaDB-client-10.3.11-1.el7.centos.x86_64
MariaDB-compat-10.3.11-1.el7.centos.x86_64
MariaDB-tokudb-engine-10.3.11-1.el7.centos.x86_64

I removed RocksDB.

Comment by Elena Stepanova [ 2018-12-13 ]

The RocksDB was the most interesting one – the error (first line in the log snippet) suggests that the RocksDB version didn't much the server version, which is why the library didn't load, which further led to RocksDB variables being unrecognizable.
Do you have any information about the version of RocksDB package which was installed at the time?

Comment by Philip orleans [ 2018-12-13 ]

Please close the case. I just reinstalled the engine with Yum, and it did not crash.
I guess yum did not remove the old version on upgrade, although I did remove the engine prior to upgrading from 10.2 to10.3, trying to about the crash on the "replace"

Just by curiosity, when I shutdown the server I get
[Warning] InnoDB: Failed to set memory to DODUMP: Invalid argument ptr 0x7f9cee400000 size 209715
is this something to worry about?

Comment by Elena Stepanova [ 2018-12-13 ]

Here is what marko said about it in a comment to MDEV-10814:

The reason for this turned out to be a Linux kernel bug, for which danblack contributed a fix for the Linux 4.19 kernel:

mm: madvise(MADV_DODUMP): allow hugetlbfs pages

This was also included in the backport queue for older kernels.

The messages in the MariaDB server error log can be ignored, and they should disappear after upgrading the kernel.

philip_38, which kernel version do you have?

Comment by Philip orleans [ 2018-12-13 ]

I have 4.15.18-9-pve
thanks for the information

Comment by Daniel Black [ 2018-12-13 ]

Released in stable kernel's 4.18.14, 4.14.76, 4.9.133, 4.4.161, and 3.18.124, and will be in 3.16.62. Redhat has confirmed it will be in their kernels (probably out already).

Without this fix, the impact is that a core dump may not contain all the information.

Generated at Thu Feb 08 08:40:38 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.