[MDEV-15164] Assertion `ikey_.type == kTypeValue' failed in rocksdb::CompactionIterator::NextFromInput Created: 2018-02-01  Updated: 2023-11-28

Status: Confirmed
Project: MariaDB Server
Component/s: Storage Engine - RocksDB
Affects Version/s: 10.2, 10.3, 10.4, 10.5, 10.6, 10.7, 10.8, 10.9, 10.10
Fix Version/s: 10.4, 10.5, 10.6

Type: Bug Priority: Major
Reporter: Elena Stepanova Assignee: Sergei Petrunia
Resolution: Unresolved Votes: 0
Labels: None

Issue Links:
Blocks
blocks MDEV-16523 Assertion `!level_and_file.second->be... Open

 Description   

10.2 d18a66147cff

 
mysqld: /data/src/10.2/storage/rocksdb/rocksdb/db/compaction_iterator.cc:314: void rocksdb::CompactionIterator::NextFromInput(): Assertion `ikey_.type == kTypeValue' failed.
 
180330 17:39:12 [ERROR] mysqld got signal 6 ;
 
 
 
#7  0x00007fa3eacd6ee2 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
 
#8  0x00007fa3e5ccbf50 in rocksdb::CompactionIterator::NextFromInput (this=0x7fa3d4006520) at /data/src/10.2/storage/rocksdb/rocksdb/db/compaction_iterator.cc:314
 
#9  0x00007fa3e5ccb4fd in rocksdb::CompactionIterator::Next (this=0x7fa3d4006520) at /data/src/10.2/storage/rocksdb/rocksdb/db/compaction_iterator.cc:156
 
#10 0x00007fa3e5cd2dfb in rocksdb::CompactionJob::ProcessKeyValueCompaction (this=0x7fa3dfffdbc0, sub_compact=0x7fa3d4003a60) at /data/src/10.2/storage/rocksdb/rocksdb/db/compaction_job.cc:886
 
#11 0x00007fa3e5cd0e22 in rocksdb::CompactionJob::Run (this=0x7fa3dfffdbc0) at /data/src/10.2/storage/rocksdb/rocksdb/db/compaction_job.cc:535
 
#12 0x00007fa3e5a15d61 in rocksdb::DBImpl::BackgroundCompaction (this=0x55ef44a05070, made_progress=0x7fa3dfffe157, job_context=0x7fa3dfffe1c0, log_buffer=0x7fa3dfffe3c0, prepicked_compaction=0x7fa3c4002bc0) at /data/src/10.2/storage/rocksdb/rocksdb/db/db_impl_compaction_flush.cc:1687
 
#13 0x00007fa3e5a13cea in rocksdb::DBImpl::BackgroundCallCompaction (this=0x55ef44a05070, prepicked_compaction=0x7fa3c4002bc0, bg_thread_pri=rocksdb::Env::LOW) at /data/src/10.2/storage/rocksdb/rocksdb/db/db_impl_compaction_flush.cc:1344
 
#14 0x00007fa3e5a12e0b in rocksdb::DBImpl::BGWorkCompaction (arg=0x7fa3c4002be0) at /data/src/10.2/storage/rocksdb/rocksdb/db/db_impl_compaction_flush.cc:1163
 
#15 0x00007fa3e5d5c6b4 in rocksdb::ThreadPoolImpl::<lambda()>::operator()(void) const (__closure=0x7fa3c4002630) at /data/src/10.2/storage/rocksdb/rocksdb/util/threadpool_imp.cc:424
 
#16 0x00007fa3e5d5cb4f in std::_Function_handler<void(), rocksdb::ThreadPoolImpl::Schedule(void (*)(void*), void*, void*, void (*)(void*))::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/4.9/functional:2039
 
#17 0x00007fa3e5c53cae in std::function<void ()>::operator()() const (this=0x7fa3dfffedd0) at /usr/include/c++/4.9/functional:2439
 
#18 0x00007fa3e5d5ba73 in rocksdb::ThreadPoolImpl::Impl::BGThread (this=0x55ef447b2dd0, thread_id=0) at /data/src/10.2/storage/rocksdb/rocksdb/util/threadpool_imp.cc:237
 
#19 0x00007fa3e5d5bb5a in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper (arg=0x55ef44a02e60) at /data/src/10.2/storage/rocksdb/rocksdb/util/threadpool_imp.cc:261
 
#20 0x00007fa3e5d64242 in std::_Bind_simple<void* (*(rocksdb::BGThreadMetadata*))(void*)>::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=0x55ef449e44d8) at /usr/include/c++/4.9/functional:1700
 
#21 0x00007fa3e5d6413b in std::_Bind_simple<void* (*(rocksdb::BGThreadMetadata*))(void*)>::operator()() (this=0x55ef449e44d8) at /usr/include/c++/4.9/functional:1688
 
#22 0x00007fa3e5d640b8 in std::thread::_Impl<std::_Bind_simple<void* (*(rocksdb::BGThreadMetadata*))(void*)> >::_M_run() (this=0x55ef449e44c0) at /usr/include/c++/4.9/thread:115
 
#23 0x00007fa3eb61d200 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
 
#24 0x00007fa3ec9ad494 in start_thread (arg=0x7fa3dffff700) at pthread_create.c:333
 
#25 0x00007fa3ead9393f in clone () from /lib/x86_64-linux-gnu/libc.so.6


# Run with --mysqld=--plugin-load-add=ha_rocksdb --repeat=N
 
 
 
--source include/have_partition.inc 
 
 
CREATE TABLE t1 (a INT) ENGINE=RocksDB;
 
CREATE TABLE t2 (pk INT PRIMARY KEY, f VARCHAR(1), KEY(f)) ENGINE=RocksDB;
 
INSERT INTO t2 VALUES 
(1,'a'),(2,'b'),(3,'c'),(4,'d'),(5,'e'),(6,'f'),(7,'g'),(8,'h'),(9,'i'),(10,'j');
 
 
 
--connect (con1,localhost,root,,test)
 
BEGIN;
 
SELECT * FROM t1;
 
 
 
--connection default
 
 
 
ALTER TABLE t2 PARTITION BY KEY() PARTITIONS 6;
 
 
 
--connect (con2,localhost,root,,test)
 
XA START 'xid1';
 
UPDATE t2 SET f = 'x';
 
--error ER_XAER_RMFAIL
 
GRANT SELECT ON something TO somebody;
 
 
 
--connection default
 
ALTER TABLE t2 PARTITION BY KEY() PARTITIONS 8;
 
 
 
--connection con2
 
XA END 'xid1';
 
XA PREPARE 'xid1';
 
XA COMMIT 'xid1';
 
 
 
sleep 10; 
 
 
# Cleanup
 
--disconnect con2
 
--connection con1
 
COMMIT;
 
--disconnect con1
 
--connection default
 
DROP TABLE t1, t2;

No visible effect on a non-debug build.

 Comments   
Comment by Alice Sherepa [ 2020-06-16 ]

also on 10.5:

10.5 7803601dcb8e40746a1

 
 
 /10.5/storage/rocksdb/rocksdb/db/compaction/compaction_iterator.cc:350: void rocksdb::CompactionIterator::NextFromInput(): Assertion `ikey_.type == kTypeValue' failed.
 
200616 15:11:34 [ERROR] mysqld got signal 6 ;
 
 
 
Server version: 10.5.4-MariaDB-debug-log
 
 
 
linux/raise.c:51(__GI_raise)[0x7f03f66c6535]
 
stdlib/abort.c:81(__GI_abort)[0x7f03f66c640f]
 
intl/loadmsgcat.c:1177(_nl_load_domain)[0x7f03f66d4102]
 
compaction/compaction_iterator.cc:351(rocksdb::CompactionIterator::NextFromInput())[0x7f03eb0aa32e]
 
compaction/compaction_iterator.cc:177(rocksdb::CompactionIterator::Next())[0x7f03eb0a4e7c]
 
compaction/compaction_job.cc:960(rocksdb::CompactionJob::ProcessKeyValueCompaction(rocksdb::CompactionJob::SubcompactionState*))[0x7f03ea8f8f8d]
 
compaction/compaction_job.cc:590(rocksdb::CompactionJob::Run())[0x7f03ea8f3674]
 
db_impl/db_impl_compaction_flush.cc:2770(rocksdb::DBImpl::BackgroundCompaction(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority))[0x7f03ea8efe44]
 
db_impl/db_impl_compaction_flush.cc:2323(rocksdb::DBImpl::BackgroundCallCompaction(rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority))[0x7f03eaef2027]
 
db_impl/db_impl_compaction_flush.cc:2100(rocksdb::DBImpl::BGWorkCompaction(void*))[0x7f03eaef17c3]
 
bits/invoke.h:60(void std::__invoke_impl<void, void (*&)(void*), void*&>(std::__invoke_other, void (*&)(void*), void*&))[0x7f03eaef037a]
 
bits/invoke.h:96(std::__invoke_result<void (*&)(void*), void*&>::type std::__invoke<void (*&)(void*), void*&>(void (*&)(void*), void*&))[0x7f03eaeee2c1]
 
8/functional:480(void std::_Bind<void (*(void*))(void*)>::operator()<, void>())[0x7f03eaeeb783]
 
bits/std_function.h:299(std::_Function_handler<void (), std::_Bind<void (*(void*))(void*)> >::_M_invoke(std::_Any_data const&))[0x7f03ea61add6]
 
bits/std_function.h:687(std::function<void ()>::operator()() const)[0x7f03eaee81b2]
 
util/threadpool_imp.cc:222(rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long))[0x7f03eaef2ccf]
 
util/threadpool_imp.cc:308(rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*))[0x7f03eaef2c41]
 
bits/invoke.h:60(void std::__invoke_impl<void, void (*)(void*), rocksdb::BGThreadMetadata*>(std::__invoke_other, void (*&&)(void*), rocksdb::BGThreadMetadata*&&))[0x7f03eaef2bb0]
 
??:0(std::error_code::default_error_condition() const)[0x7f03f6abfb2f]
 
nptl/pthread_create.c:487(start_thread)[0x7f03f7196fa3]
 
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f03f679d4cf]

Comment by Elena Stepanova [ 2022-08-25 ]

The test case from the description no longer works for current versions.
It still behaves as described on the revision which the bug report was filed for, so there is no typo in the test case.
But something changed since then (even in 10.2 before it went EOL), so now the second ALTER stalls waiting for a lock, which probably should happen but didn't happen before.

All in all, we need a new test case for this, as it also prevents us from reproducing MDEV-16523.

Comment by Elena Stepanova [ 2022-08-29 ]

A new test case. It is even worse than the previous one in terms of probability, but it fails eventually if it's given enough time. I can't make it any better as the test case itself doesn't contain any concurrency, whatever causes non-determinism is in the internals.
Run with --mysqld=--plugin-load-add=ha_rocksdb and as big --repeat=N as possible, at least hundreds, it's highly volatile. On the bright side, the test case is rr-rable (although it takes even longer).

--source include/have_partition.inc
 
--source include/have_sequence.inc
 
 
 
CREATE TABLE t1 (
 
  pk INTEGER AUTO_INCREMENT,
 
  a INT,
 
  b INT,
 
  c DATE,
 
  d DATE,
 
  e TIME,
 
  f TIME,
 
  g DATETIME,
 
  h DATETIME,
 
  i VARCHAR(1),
 
  j VARCHAR(1),
 
  PRIMARY KEY (pk),
 
  KEY (b),
 
  KEY (i, b)
 
) ENGINE=RocksDB;
 
 
 
CREATE TABLE t3 LIKE t1;
 
CREATE TABLE t5 LIKE t1;
 
CREATE TABLE t2 LIKE t1;
 
CREATE TABLE t4 LIKE t1;
 
 
 
INSERT INTO t5 (pk) SELECT NULL FROM seq_1_to_500;
 
INSERT INTO t4 (pk) SELECT NULL FROM seq_1_to_100;
 
 
 
REPLACE INTO t1 SELECT * FROM t5;
 
ALTER TABLE t1 PARTITION BY KEY(pk);
 
START TRANSACTION;
 
INSERT INTO t3 (pk) VALUES (NULL);
 
 
 
--connect (con1,localhost,root,,test)
 
CREATE TABLE t6 ENGINE=RocksDB SELECT * FROM t4;
 
DELETE FROM t1 WHERE pk IN (1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,17,18,19);
 
ALTER TABLE t2 PARTITION BY KEY() PARTITIONS 2;
 
REPLACE INTO t1 SELECT * FROM t5;
 
CREATE TABLE t7 ENGINE=RocksDB SELECT * FROM t5;
 
DROP TABLE t7;
 
 
 
DELETE FROM t1 WHERE pk = 2;
 
ALTER TABLE t1 REMOVE PARTITIONING;
 
CREATE TABLE t8 ENGINE=RocksDB SELECT * FROM t5;
 
DROP TABLE t8;
 
 
 
# Seems necessary, maybe to allow the compaction start
 
--sleep 1
 
 
 
# Cleanup
 
--connection default
 
COMMIT;
 
DROP TABLE t6, t5, t4, t3, t2, t1;

10.4 e4043152

 
#2  0x00007f643631740f in __assert_fail_base (fmt=0x7f6436480128 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x7f642bd0cda2 "ikey_.type == kTypeValue", 
    file=0x7f642bd0cbc0 "/data/src/10.4/storage/rocksdb/rocksdb/db/compaction/compaction_iterator.cc", line=350, function=<optimized out>) at assert.c:92
 
#3  0x00007f6436326662 in __GI___assert_fail (assertion=0x7f642bd0cda2 "ikey_.type == kTypeValue", 
    file=0x7f642bd0cbc0 "/data/src/10.4/storage/rocksdb/rocksdb/db/compaction/compaction_iterator.cc", line=350, 
    function=0x7f642bd0cd08 "void rocksdb::CompactionIterator::NextFromInput()") at assert.c:101
 
#4  0x00007f642bbc8827 in rocksdb::CompactionIterator::NextFromInput (this=0x7f641c003180) at /data/src/10.4/storage/rocksdb/rocksdb/db/compaction/compaction_iterator.cc:350
 
#5  0x00007f642bbc7cdc in rocksdb::CompactionIterator::Next (this=0x7f641c003180) at /data/src/10.4/storage/rocksdb/rocksdb/db/compaction/compaction_iterator.cc:174
 
#6  0x00007f642bbdafb3 in rocksdb::CompactionJob::ProcessKeyValueCompaction (this=0x7f642ad1f710, sub_compact=0x7f641c001500)
 
    at /data/src/10.4/storage/rocksdb/rocksdb/db/compaction/compaction_job.cc:959
 
#7  0x00007f642bbd8b14 in rocksdb::CompactionJob::Run (this=0x7f642ad1f710) at /data/src/10.4/storage/rocksdb/rocksdb/db/compaction/compaction_job.cc:587
 
#8  0x00007f642b8b2877 in rocksdb::DBImpl::BackgroundCompaction (this=0x55d633c2b280, made_progress=0x7f642ad1fede, job_context=0x7f642ad1ff40, log_buffer=0x7f642ad20110, 
    prepicked_compaction=0x7f6423ae3890, thread_pri=rocksdb::Env::LOW) at /data/src/10.4/storage/rocksdb/rocksdb/db/db_impl/db_impl_compaction_flush.cc:2770
 
#9  0x00007f642b8afa71 in rocksdb::DBImpl::BackgroundCallCompaction (this=0x55d633c2b280, prepicked_compaction=0x7f6423ae3890, bg_thread_pri=rocksdb::Env::LOW)
 
    at /data/src/10.4/storage/rocksdb/rocksdb/db/db_impl/db_impl_compaction_flush.cc:2323
 
#10 0x00007f642b8ae1a5 in rocksdb::DBImpl::BGWorkCompaction (arg=0x7f64208a9f40) at /data/src/10.4/storage/rocksdb/rocksdb/db/db_impl/db_impl_compaction_flush.cc:2098
 
#11 0x00007f642bb1b15c in std::__invoke_impl<void, void (*&)(void*), void*&> (__f=@0x7f64044f4ce0: 0x7f642b8ae0cc <rocksdb::DBImpl::BGWorkCompaction(void*)>)
 
    at /usr/include/c++/10/bits/invoke.h:60
 
#12 0x00007f642bb1adeb in std::__invoke<void (*&)(void*), void*&> (__fn=@0x7f64044f4ce0: 0x7f642b8ae0cc <rocksdb::DBImpl::BGWorkCompaction(void*)>)
 
    at /usr/include/c++/10/bits/invoke.h:95
 
#13 0x00007f642bb1a6ea in std::_Bind<void (*(void*))(void*)>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) (this=0x7f64044f4ce0, __args=...)
 
    at /usr/include/c++/10/functional:416
 
#14 0x00007f642bb19efa in std::_Bind<void (*(void*))(void*)>::operator()<, void>() (this=0x7f64044f4ce0) at /usr/include/c++/10/functional:499
 
#15 0x00007f642bb196b4 in std::__invoke_impl<void, std::_Bind<void (*(void*))(void*)>&>(std::__invoke_other, std::_Bind<void (*(void*))(void*)>&) (__f=...)
 
    at /usr/include/c++/10/bits/invoke.h:60
 
#16 0x00007f642bb1897f in std::__invoke_r<void, std::_Bind<void (*(void*))(void*)>&>(std::_Bind<void (*(void*))(void*)>&) (__fn=...) at /usr/include/c++/10/bits/invoke.h:153
 
#17 0x00007f642bb1788b in std::_Function_handler<void (), std::_Bind<void (*(void*))(void*)> >::_M_invoke(std::_Any_data const&) (__functor=...)
 
    at /usr/include/c++/10/bits/std_function.h:291
 
#18 0x00007f642b77a40e in std::function<void ()>::operator()() const (this=0x7f642ad20c40) at /usr/include/c++/10/bits/std_function.h:622
 
#19 0x00007f642bb13ac4 in rocksdb::ThreadPoolImpl::Impl::BGThread (this=0x55d633bc69a0, thread_id=0) at /data/src/10.4/storage/rocksdb/rocksdb/util/threadpool_imp.cc:265
 
#20 0x00007f642bb13c3f in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper (arg=0x55d633c26540) at /data/src/10.4/storage/rocksdb/rocksdb/util/threadpool_imp.cc:306
 
#21 0x00007f642bb1b6a6 in std::__invoke_impl<void, void (*)(void*), rocksdb::BGThreadMetadata*> (
 
    __f=@0x55d633c265d0: 0x7f642bb13b32 <rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*)>) at /usr/include/c++/10/bits/invoke.h:60
 
#22 0x00007f642bb1b61b in std::__invoke<void (*)(void*), rocksdb::BGThreadMetadata*> (
 
    __fn=@0x55d633c265d0: 0x7f642bb13b32 <rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*)>) at /usr/include/c++/10/bits/invoke.h:95
 
#23 0x00007f642bb1b58b in std::thread::_Invoker<std::tuple<void (*)(void*), rocksdb::BGThreadMetadata*> >::_M_invoke<0ul, 1ul> (this=0x55d633c265c8)
 
    at /usr/include/c++/10/thread:264
 
#24 0x00007f642bb1b544 in std::thread::_Invoker<std::tuple<void (*)(void*), rocksdb::BGThreadMetadata*> >::operator() (this=0x55d633c265c8) at /usr/include/c++/10/thread:271
 
#25 0x00007f642bb1b528 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(void*), rocksdb::BGThreadMetadata*> > >::_M_run (this=0x55d633c265c0)
 
    at /usr/include/c++/10/thread:215
 
#26 0x00007f64366e3ed0 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
 
--Type <RET> for more, q to quit, c to continue without paging--
 
#27 0x00007f64367f0ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
 
#28 0x00007f64363efdef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

While the failure itself doesn't go anywhere, changes in DDL/locking/whatever make the server-side behavior change with time. This test case also seem to be failing somewhat more willingly on 10.3-10.4, less willingly on 10.5-10.6, and very reluctantly on higher versions. Hopefully the problem will be fixed before this test case expires, too.

Generated at Thu Feb 08 08:19:10 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.