[MDEV-21927] RocksDB: Failed to get column family flags from cf, SIGABRT in myrocks::Rdb_drop_index_thread::run Created: 2020-03-12  Updated: 2023-04-27

Status: Confirmed
Project: MariaDB Server
Component/s: Storage Engine - RocksDB
Affects Version/s: 10.2, 10.3, 10.4, 10.5, 10.6, 10.7
Fix Version/s: 10.4, 10.5, 10.6

Type: Bug Priority: Major
Reporter: Elena Stepanova Assignee: Sergei Petrunia
Resolution: Unresolved Votes: 0
Labels: None


 Description    

INSTALL SONAME 'ha_rocksdb';
 
 
 
CREATE TABLE t1 (pk INT PRIMARY KEY, a INT) ENGINE=RocksDB;
 
INSERT INTO t1 VALUES (1,1),(2,1);
 
 
 
--error ER_DUP_ENTRY
 
ALTER TABLE t1 ADD UNIQUE (a) COMMENT 'foo';
 
CREATE TABLE t2 (b INT) ENGINE=RocksDB;
 
DROP TABLE t2;

10.4 b7362d5f

 
2020-03-12 18:50:02 0 [ERROR] RocksDB: Failed to get column family flags from cf id 2. MyRocks data dictionary may get corrupted.
 
200312 18:50:02 [ERROR] mysqld got signal 6 ;
 
 
 
#3  <signal handler called>
 
#4  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
 
#5  0x00007fd12e0a142a in __GI_abort () at abort.c:89
 
#6  0x00007fd0d3638976 in myrocks::Rdb_drop_index_thread::run (this=0x7fd0d3feb5e0 <myrocks::rdb_drop_idx_thread>) at /data/src/10.4/storage/rocksdb/ha_rocksdb.cc:11604
 
#7  0x00007fd0d36e5206 in myrocks::Rdb_thread::thread_func (thread_ptr=0x7fd0d3feb5e0 <myrocks::rdb_drop_idx_thread>) at /data/src/10.4/storage/rocksdb/rdb_threads.cc:34
 
#8  0x00007fd1300214a4 in start_thread (arg=0x7fd0bb7f6700) at pthread_create.c:456
 
#9  0x00007fd12e155d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Reproducible on 10.2-10.5, debug and non-debug alike.
Stack traces may vary, and SIGABRT can happen in a thread other than 1st.

 Comments   
Comment by Sergei Petrunia [ 2021-05-07 ]

Doesn't reproduce on the upstream. This commit made changed the execution: 

commit 876d9a3fafe36d0cf6b285b1496f1de94eb89739
 
Author:	bintong <bintong@devvm19843.prn1.facebook.com>  Thu Nov  8 04:09:22 2018
 
Committer:	Facebook Github Bot <facebook-github-bot@users.noreply.github.com>  Tue Sep  3 03:43:46 2019

Comment by Sergei Petrunia [ 2021-05-07 ]

A bit more detail: debug printouts from both upstream MyRocks and MariaDB's
MyRocks (the upstream is ahead of MariaDB's) :

ALTER TABLE t1 ADD UNIQUE (a) COMMENT 'foo';


  AAABBB: CreateColumnFamily foo
 
  AAABBB: Rdb_dict_manager::add_cf_flags 2   ## this is only called on upstream's MyRocks
 
  AAABBB: Rdb_dict_manager::start_ongoing_index_operation 2,3127628307,2
 
  AAABBB: Rdb_dict_manager::rollback_ongoing_index_creation: 2, 3127628307, 2  
  AAABBB: Rdb_dict_manager::start_ongoing_index_operation 2,3127628307,2

I'm not sure what is the purpose of the second start_ongoing_index_operation
call. Current MyRocks has it, too, so it is not the problem.

CREATE TABLE t2 (b INT) ENGINE=RocksDB;
 
DROP TABLE t2;


  AAABBB: Rdb_dict_manager::start_ongoing_index_operation 0,3127628307,3
 
  AAABBB: Rdb_drop_index_thread: looking at indexes
 
  AAABBB: __ index 2,2 # Here, MariaDB's MyRocks crashes. 
  AAABBB: __ index 0,3

The crash happens because add_cf_flags call wasn't made.

Comment by Sergei Petrunia [ 2021-05-07 ]

Looking at MariaDB's MyRocks:
add_cf_flags() is only called from Rdb_tbl_def::put_dict()
which is called from Rdb_ddl_manager::put_and_write which is called from ha_rocksdb::commit_inplace_alter_table. However, commit_inplace_alter_table is only called when ALTER TABLE succeeds.

That is, in order to reproduce this bug, the ALTER TABLE needs to create a column family for the new index and then fail.

Generated at Thu Feb 08 09:10:53 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.