[MDEV-21799] Assertion `is_storage_available(tuple - packed_tuple, 0)' failed in myrocks::Rdb_key_def::pack_record upon INSERT into VARBINARY(0) or VARCHAR | heap-buffer-overflow on address from myrocks::Rdb_field_packing::fill_hidden_pk_val Created: 2020-02-21  Updated: 2023-11-28

Status: Confirmed
Project: MariaDB Server
Component/s: Storage Engine - RocksDB
Affects Version/s: 10.2, 10.3, 10.4, 10.5, 10.6, 10.7, 10.8, 10.9
Fix Version/s: 10.4, 10.5, 10.6

Type: Bug Priority: Critical
Reporter: Elena Stepanova Assignee: Sergei Petrunia
Resolution: Unresolved Votes: 0
Labels: ASAN, affects-tests, hang, memory_corruption

Issue Links:
Relates
relates to MDEV-29857 ASAN heap-buffer-overflow in myrocks:... Open
relates to MDEV-30610 Update RocksDB to the latest upstream... Open

 Description   

INSTALL SONAME 'ha_rocksdb';
 
CREATE TABLE t1 (a VARBINARY(0), KEY(a)) ENGINE=RocksDB;
INSERT IGNORE INTO t1 VALUES (1);

10.2 3ce49a0a

mysqld: /data/src/10.2/storage/rocksdb/rdb_datadic.cc:1443: uint myrocks::Rdb_key_def::pack_record(const TABLE*, uchar*, const uchar*, uchar*, myrocks::Rdb_string_writer*, bool, longlong, uint, uint*, const char*) const: Assertion `is_storage_available(tuple - packed_tuple, 0)' failed.
200221 15:48:42 [ERROR] mysqld got signal 6 ;
 
#6  0x00007f0977f1fe67 in __assert_fail_base (fmt=<optimized out>, assertion=assertion@entry=0x7f0931ac9c70 "is_storage_available(tuple - packed_tuple, 0)", file=file@entry=0x7f0931ac9688 "/data/src/10.2/storage/rocksdb/rdb_datadic.cc", line=line@entry=1443, function=function@entry=0x7f0931acb860 <myrocks::Rdb_key_def::pack_record(TABLE const*, unsigned char*, unsigned char const*, unsigned char*, myrocks::Rdb_string_writer*, bool, long long, unsigned int, unsigned int*, char const*) const::__PRETTY_FUNCTION__> "uint myrocks::Rdb_key_def::pack_record(const TABLE*, uchar*, const uchar*, uchar*, myrocks::Rdb_string_writer*, bool, longlong, uint, uint*, const char*) const") at assert.c:92
#7  0x00007f0977f1ff12 in __GI___assert_fail (assertion=0x7f0931ac9c70 "is_storage_available(tuple - packed_tuple, 0)", file=0x7f0931ac9688 "/data/src/10.2/storage/rocksdb/rdb_datadic.cc", line=1443, function=0x7f0931acb860 <myrocks::Rdb_key_def::pack_record(TABLE const*, unsigned char*, unsigned char const*, unsigned char*, myrocks::Rdb_string_writer*, bool, long long, unsigned int, unsigned int*, char const*) const::__PRETTY_FUNCTION__> "uint myrocks::Rdb_key_def::pack_record(const TABLE*, uchar*, const uchar*, uchar*, myrocks::Rdb_string_writer*, bool, longlong, uint, uint*, const char*) const") at assert.c:101
#8  0x00007f09316c274f in myrocks::Rdb_key_def::pack_record (this=0x7f0914050c50, tbl=0x7f0914055a70, pack_buffer=0x7f09140576d0 '\245' <repeats 16 times>, "h4z\025\217\217\217\217\225", record=0x7f0914056680 "\376", packed_tuple=0x7f091405ec90 "", unpack_info=0x7f0914057018, should_store_row_debug_checksums=false, hidden_pk_id=1, n_key_parts=2, n_null_fields=0x0, ttl_bytes=0x7f0914057841 '\217' <repeats 15 times>, "\020y\005\024\t\177") at /data/src/10.2/storage/rocksdb/rdb_datadic.cc:1443
#9  0x00007f093163e57a in myrocks::ha_rocksdb::update_write_sk (this=0x7f0914056ad8, table_arg=0x7f0914055a70, kd=..., row_info=..., bulk_load_sk=false) at /data/src/10.2/storage/rocksdb/ha_rocksdb.cc:10141
#10 0x00007f093163ea6c in myrocks::ha_rocksdb::update_write_indexes (this=0x7f0914056ad8, row_info=..., pk_changed=false) at /data/src/10.2/storage/rocksdb/ha_rocksdb.cc:10241
#11 0x00007f093163ed28 in myrocks::ha_rocksdb::update_write_row (this=0x7f0914056ad8, old_data=0x0, new_data=0x7f0914056680 "\376", skip_unique_check=false) at /data/src/10.2/storage/rocksdb/ha_rocksdb.cc:10324
#12 0x00007f093163ca7c in myrocks::ha_rocksdb::write_row (this=0x7f0914056ad8, buf=0x7f0914056680 "\376") at /data/src/10.2/storage/rocksdb/ha_rocksdb.cc:9533
#13 0x0000561d839cf5a8 in handler::ha_write_row (this=0x7f0914056ad8, buf=0x7f0914056680 "\376") at /data/src/10.2/sql/handler.cc:6089
#14 0x0000561d8371ea0a in write_record (thd=0x7f0914000af0, table=0x7f0914055a70, info=0x7f097567a530) at /data/src/10.2/sql/sql_insert.cc:1941
#15 0x0000561d8371c4a2 in mysql_insert (thd=0x7f0914000af0, table_list=0x7f0914012328, fields=..., values_list=..., update_fields=..., update_values=..., duplic=DUP_ERROR, ignore=true) at /data/src/10.2/sql/sql_insert.cc:1066
#16 0x0000561d83743515 in mysql_execute_command (thd=0x7f0914000af0) at /data/src/10.2/sql/sql_parse.cc:4166
#17 0x0000561d8374ecba in mysql_parse (thd=0x7f0914000af0, rawbuf=0x7f0914012238 "INSERT IGNORE INTO t1 VALUES (1)", length=32, parser_state=0x7f097567b250, is_com_multi=false, is_next_command=false) at /data/src/10.2/sql/sql_parse.cc:7739
#18 0x0000561d8373d013 in dispatch_command (command=COM_QUERY, thd=0x7f0914000af0, packet=0x7f0914066ca1 "INSERT IGNORE INTO t1 VALUES (1)", packet_length=32, is_com_multi=false, is_next_command=false) at /data/src/10.2/sql/sql_parse.cc:1831
#19 0x0000561d8373b967 in do_command (thd=0x7f0914000af0) at /data/src/10.2/sql/sql_parse.cc:1384
#20 0x0000561d838909a3 in do_handle_one_connection (connect=0x561d871f0cf0) at /data/src/10.2/sql/sql_connect.cc:1336
#21 0x0000561d8389070e in handle_one_connection (arg=0x561d871f0cf0) at /data/src/10.2/sql/sql_connect.cc:1241
#22 0x00007f0979ea84a4 in start_thread (arg=0x7f097567c700) at pthread_create.c:456
#23 0x00007f0977fdcd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Reproducible on 10.2-10.5.
No obvious problem on a non-debug build.



 Comments   
Comment by Roel Van de Paar [ 2021-09-29 ]

Repeat testcase a second time in case necessary.

SET sql_mode='',collation_server=2;
INSTALL plugin RocksDB SONAME 'ha_rocksdb';
DROP DATABASE test;
CREATE DATABASE test;
USE test;
CREATE TABLE t (c VARCHAR(100), KEY(c (10))) ENGINE=RocksDB;
INSERT INTO t VALUES ('aaaaaaaaaaaaaa');

Leads to:

10.7.0 d552e092c9f3e20da078d1b62b976f629f73d3a4 (Debug)

mysqld: /test/10.7_dbg/storage/rocksdb/rdb_datadic.cc:1447: uint myrocks::Rdb_key_def::pack_record(const TABLE*, uchar*, const uchar*, uchar*, myrocks::Rdb_string_writer*, bool, longlong, uint, uint*, const char*) const: Assertion `is_storage_available(tuple - packed_tuple, 0)' failed.

10.7.0 d552e092c9f3e20da078d1b62b976f629f73d3a4 (Debug)

Core was generated by `/test/MD180921-mariadb-10.7.0-linux-x86_64-dbg/bin/mysqld --no-defaults --core-'.
Program terminated with signal SIGABRT, Aborted.
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
[Current thread is 1 (Thread 0x150b0407a700 (LWP 2804024))]
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x0000150b07e5a859 in __GI_abort () at abort.c:79
#2  0x0000150b07e5a729 in __assert_fail_base (fmt=0x150b07ff0588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x150ad3acd910 "is_storage_available(tuple - packed_tuple, 0)", file=0x150ad3acca70 "/test/10.7_dbg/storage/rocksdb/rdb_datadic.cc", line=1447, function=<optimized out>) at assert.c:92
#3  0x0000150b07e6bf36 in __GI___assert_fail (assertion=assertion@entry=0x150ad3acd910 "is_storage_available(tuple - packed_tuple, 0)", file=file@entry=0x150ad3acca70 "/test/10.7_dbg/storage/rocksdb/rdb_datadic.cc", line=line@entry=1447, function=function@entry=0x150ad3ace580 "uint myrocks::Rdb_key_def::pack_record(const TABLE*, uchar*, const uchar*, uchar*, myrocks::Rdb_string_writer*, bool, longlong, uint, uint*, const char*) const") at assert.c:101
#4  0x0000150ad3789626 in myrocks::Rdb_key_def::pack_record (this=this@entry=0x150ab02c2b10, tbl=tbl@entry=0x150ab02554b8, pack_buffer=0x150ab0270688 '\003' <repeats 14 times>, "\001", '\003' <repeats 14 times>, "\001", '\003' <repeats 14 times>, "\001\003\003\003\003\003\003\003\003\003", '\245' <repeats 18 times>, "h4z\025\217\217\217\217", <incomplete sequence \345>, record=0x150ab0256678 "\376\016", 'a' <repeats 14 times>, packed_tuple=0x150ab02571e8 "", unpack_info=unpack_info@entry=0x150ab026ed78, should_store_row_debug_checksums=false, hidden_pk_id=1, n_key_parts=2, n_null_fields=0x0, ttl_bytes=0x150ab025e391 "\200\353B\202U") at /test/10.7_dbg/storage/rocksdb/./rdb_datadic.h:773
#5  0x0000150ad374cda2 in myrocks::ha_rocksdb::update_write_sk (this=this@entry=0x150ab026e790, table_arg=0x150ab02554b8, kd=@0x150ab02c2b10: {static CF_FLAGS_TO_IGNORE = 4, m_index_number = 256, m_index_number_storage_form = "\000\000\001", m_cf_handle = 0x150ab0212b70, m_index_dict_version = 6, m_index_type = 2 '\002', m_kv_format_version = 13, m_is_reverse_cf = false, m_is_per_partition_cf = false, m_name = "c", m_stats = {m_gl_index_id = {cf_id = 0, index_id = 0}, m_data_size = 0, m_rows = 0, m_actual_disk_size = 0, m_entry_deletes = 0, m_entry_single_deletes = 0, m_entry_merges = 0, m_entry_others = 0, m_distinct_keys_per_prefix = std::vector of length 2, capacity 2 = {0, 0}, m_name = ""}, m_index_flags_bitmap = 0, m_total_index_flags_length = 0, m_ttl_rec_offset = 4294967295, m_ttl_duration = 0, m_ttl_column = "", m_pk_key_parts = 1, m_pk_part_no = 0x150ab025c478, m_pack_info = 0x150ab0013868, m_keyno = 0, m_key_parts = 2, m_ttl_pk_key_part_offset = 4294967295, m_ttl_field_index = 4294967295, m_prefix_extractor = std::shared_ptr<const class rocksdb::SliceTransform> (empty) = {
        get() = 0x0
      }, m_maxlength = 67, m_mutex = {m_mutex = {global = pthread_mutex_t = {
            Type = Error check,
            Status = Not acquired,
            Robust = No,
            Shared = No,
            Protocol = None
          }, mutex = pthread_mutex_t = {
            Type = Adaptive,
            Status = Not acquired,
            Robust = No,
            Shared = No,
            Protocol = None
          }, file = 0x150ad3acca70 "/test/10.7_dbg/storage/rocksdb/rdb_datadic.cc", name = 0x150ad3ac6d33 "m_mutex", line = 406, count = 0, create_flags = 0, active_flags = 0, id = 219, thread = 0, locked_mutex = 0x150ab0235218, used_mutex = 0x150ab0235288, prev = 0x0, next = 0x0}, m_psi = 0x0}}, row_info=@0x150b04078970: {tx = 0x150ab0272250, new_data = 0x150ab0256678 "\376\016", 'a' <repeats 14 times>, old_data = 0x0, new_pk_slice = {data_ = 0x150ab0263588 "", size_ = 12}, old_pk_slice = {data_ = 0x150ad3b13fe0 "", size_ = 0}, old_pk_rec = {data_ = 0x150ad3b13fe0 "", size_ = 0}, new_pk_unpack_info = 0x0, hidden_pk_id = 1, skip_unique_check = false}, bulk_load_sk=bulk_load_sk@entry=false) at /test/10.7_dbg/storage/rocksdb/ha_rocksdb.cc:10206
#6  0x0000150ad374d627 in myrocks::ha_rocksdb::update_write_indexes (this=this@entry=0x150ab026e790, row_info=@0x150b04078970: {tx = 0x150ab0272250, new_data = 0x150ab0256678 "\376\016", 'a' <repeats 14 times>, old_data = 0x0, new_pk_slice = {data_ = 0x150ab0263588 "", size_ = 12}, old_pk_slice = {data_ = 0x150ad3b13fe0 "", size_ = 0}, old_pk_rec = {data_ = 0x150ad3b13fe0 "", size_ = 0}, new_pk_unpack_info = 0x0, hidden_pk_id = 1, skip_unique_check = false}, pk_changed=pk_changed@entry=false) at /usr/include/c++/9/bits/shared_ptr_base.h:1309
#7  0x0000150ad374d866 in myrocks::ha_rocksdb::update_write_row (this=this@entry=0x150ab026e790, old_data=old_data@entry=0x0, new_data=new_data@entry=0x150ab0256678 "\376\016", 'a' <repeats 14 times>, skip_unique_check=<optimized out>) at /test/10.7_dbg/storage/rocksdb/ha_rocksdb.cc:10390
#8  0x0000150ad374daa4 in myrocks::ha_rocksdb::write_row (this=0x150ab026e790, buf=0x150ab0256678 "\376\016", 'a' <repeats 14 times>) at /test/10.7_dbg/storage/rocksdb/ha_rocksdb.cc:9590
#9  0x00005582430db7b7 in handler::ha_write_row (this=0x150ab026e790, buf=0x150ab0256678 "\376\016", 'a' <repeats 14 times>) at /test/10.7_dbg/sql/handler.cc:7516
#10 0x0000558242d803da in write_record (thd=thd@entry=0x150ab0000db8, table=table@entry=0x150ab02554b8, info=info@entry=0x150b04078cc0, sink=sink@entry=0x0) at /test/10.7_dbg/sql/sql_insert.cc:2136
#11 0x0000558242d8c028 in mysql_insert (thd=thd@entry=0x150ab0000db8, table_list=0x150ab0013d60, fields=@0x150ab0005fb8: {<base_list> = {<Sql_alloc> = {<No data fields>}, first = 0x5582443c64a0 <end_of_list>, last = 0x150ab0005fb8, elements = 0}, <No data fields>}, values_list=@0x150ab0006000: {<base_list> = {<Sql_alloc> = {<No data fields>}, first = 0x150ab0014978, last = 0x150ab0014978, elements = 1}, <No data fields>}, update_fields=@0x150ab0005fe8: {<base_list> = {<Sql_alloc> = {<No data fields>}, first = 0x5582443c64a0 <end_of_list>, last = 0x150ab0005fe8, elements = 0}, <No data fields>}, update_values=@0x150ab0005fd0: {<base_list> = {<Sql_alloc> = {<No data fields>}, first = 0x5582443c64a0 <end_of_list>, last = 0x150ab0005fd0, elements = 0}, <No data fields>}, duplic=DUP_ERROR, ignore=false, result=0x0) at /test/10.7_dbg/sql/sql_insert.cc:1124
#12 0x0000558242dd2112 in mysql_execute_command (thd=thd@entry=0x150ab0000db8, is_called_from_prepared_stmt=is_called_from_prepared_stmt@entry=false) at /test/10.7_dbg/sql/sql_parse.cc:4563
#13 0x0000558242dbcd4b in mysql_parse (thd=thd@entry=0x150ab0000db8, rawbuf=<optimized out>, length=<optimized out>, parser_state=parser_state@entry=0x150b04079400) at /test/10.7_dbg/sql/sql_parse.cc:8028
#14 0x0000558242dcb944 in dispatch_command (command=command@entry=COM_QUERY, thd=thd@entry=0x150ab0000db8, packet=packet@entry=0x150ab000b729 "INSERT INTO t VALUES ('", 'a' <repeats 14 times>, "')", packet_length=packet_length@entry=39, blocking=blocking@entry=true) at /test/10.7_dbg/sql/sql_class.h:1358
#15 0x0000558242dced4a in do_command (thd=0x150ab0000db8, blocking=blocking@entry=true) at /test/10.7_dbg/sql/sql_parse.cc:1402
#16 0x0000558242f44ed8 in do_handle_one_connection (connect=<optimized out>, connect@entry=0x558246790478, put_in_cache=put_in_cache@entry=true) at /test/10.7_dbg/sql/sql_connect.cc:1418
#17 0x0000558242f454dd in handle_one_connection (arg=arg@entry=0x558246790478) at /test/10.7_dbg/sql/sql_connect.cc:1312
#18 0x00005582433ae47e in pfs_spawn_thread (arg=0x55824668f288) at /test/10.7_dbg/storage/perfschema/pfs.cc:2201
#19 0x0000150b08369609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#20 0x0000150b07f57293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Bug confirmed present in:
MariaDB: 10.2.41 (dbg), 10.3.32 (dbg), 10.4.22 (dbg), 10.5.13 (dbg), 10.6.5 (dbg), 10.7.0 (dbg)

Bug (or feature/syntax) confirmed not present in:
MariaDB: 10.2.41 (opt), 10.3.32 (opt), 10.4.22 (opt), 10.5.13 (opt), 10.6.5 (opt), 10.7.0 (opt)
MySQL: 5.5.62 (dbg), 5.5.62 (opt), 5.6.51 (dbg), 5.6.51 (opt), 5.7.35 (dbg), 5.7.35 (opt), 8.0.26 (dbg), 8.0.26 (opt)

Comment by Roel Van de Paar [ 2021-09-29 ]

There are more issues here it seems. Using the original testcase, I get:

10.5.13 c430aa72abbdccb1ece7f0d65b49a6b48e7c5ba7 (Optimized)

2021-09-29 16:02:38 0 [Note] /test/MD180921-mariadb-10.5.13-linux-x86_64-opt/bin/mysqld (initiated by: root[root] @ localhost []): Normal shutdown
2021-09-29 16:02:38 0 [Note] Event Scheduler: Purging the queue. 0 events
2021-09-29 16:02:38 0 [Note] InnoDB: FTS optimize thread exiting.
free(): invalid next size (fast)

So I ran it again under ASAN/UBSAN (using dbg build) and got this:

10.5.13 a1b0f2358620335e9495242270cf47a1366c9a6a (Debug)

2021-09-29 16:20:04 4 [Note] MyRocks storage engine plugin has been successfully initialized.
=================================================================
==605817==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60e00001a4dc at pc 0x5558b1e4afdd bp 0x1490ab3d2560 sp 0x1490ab3d1d08
WRITE of size 8 at 0x60e00001a4dc thread T13
    #0 0x5558b1e4afdc in __interceptor_memcpy.part.0 (/test/UBASAN_MD060921-mariadb-10.5.13-linux-x86_64-dbg/bin/mariadbd+0x81e8fdc)
    #1 0x1490a50f2370 in myrocks::Rdb_field_packing::fill_hidden_pk_val(unsigned char**, long long) const /test/10.5_dbg_san/storage/rocksdb/rdb_datadic.cc:3500
    #2 0x1490a510543d in myrocks::Rdb_key_def::pack_record(TABLE const*, unsigned char*, unsigned char const*, unsigned char*, myrocks::Rdb_string_writer*, bool, long long, unsigned int, unsigned int*, char const*) const /test/10.5_dbg_san/storage/rocksdb/rdb_datadic.cc:1365
    #3 0x1490a4ef6bdc in myrocks::ha_rocksdb::update_write_sk(TABLE const*, myrocks::Rdb_key_def const&, myrocks::ha_rocksdb::update_row_info const&, bool) /test/10.5_dbg_san/storage/rocksdb/ha_rocksdb.cc:10185
    #4 0x1490a4efb95f in myrocks::ha_rocksdb::update_write_indexes(myrocks::ha_rocksdb::update_row_info const&, bool) /test/10.5_dbg_san/storage/rocksdb/ha_rocksdb.cc:10284
    #5 0x1490a4f02e08 in myrocks::ha_rocksdb::update_write_row(unsigned char const*, unsigned char const*, bool) /test/10.5_dbg_san/storage/rocksdb/ha_rocksdb.cc:10368
    #6 0x1490a4f03db9 in myrocks::ha_rocksdb::write_row(unsigned char const*) /test/10.5_dbg_san/storage/rocksdb/ha_rocksdb.cc:9577
    #7 0x5558b3e1187e in handler::ha_write_row(unsigned char const*) /test/10.5_dbg_san/sql/handler.cc:7168
    #8 0x5558b24a6750 in write_record(THD*, TABLE*, st_copy_info*, select_result*) /test/10.5_dbg_san/sql/sql_insert.cc:2129
    #9 0x5558b25180c1 in mysql_insert(THD*, TABLE_LIST*, List<Item>&, List<List<Item> >&, List<Item>&, List<Item>&, enum_duplicates, bool, select_result*) /test/10.5_dbg_san/sql/sql_insert.cc:1118
    #10 0x5558b27936ad in mysql_execute_command(THD*) /test/10.5_dbg_san/sql/sql_parse.cc:4624
    #11 0x5558b26e8c6a in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) /test/10.5_dbg_san/sql/sql_parse.cc:8100
    #12 0x5558b276277d in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /test/10.5_dbg_san/sql/sql_parse.cc:1891
    #13 0x5558b2779a5c in do_command(THD*) /test/10.5_dbg_san/sql/sql_parse.cc:1370
    #14 0x5558b3168fc7 in do_handle_one_connection(CONNECT*, bool) /test/10.5_dbg_san/sql/sql_connect.cc:1418
    #15 0x5558b316c27f in handle_one_connection /test/10.5_dbg_san/sql/sql_connect.cc:1312
    #16 0x5558b56fdde7 in pfs_spawn_thread /test/10.5_dbg_san/storage/perfschema/pfs.cc:2201
    #17 0x1490ce192608 in start_thread /build/glibc-eX1tMB/glibc-2.31/nptl/pthread_create.c:477
    #18 0x1490cd408292 in __clone (/lib/x86_64-linux-gnu/libc.so.6+0x122292)
 
0x60e00001a4dc is located 0 bytes to the right of 156-byte region [0x60e00001a440,0x60e00001a4dc)
allocated by thread T13 here:
    #0 0x5558b1ebd5f8 in __interceptor_malloc (/test/UBASAN_MD060921-mariadb-10.5.13-linux-x86_64-dbg/bin/mariadbd+0x825b5f8)
    #1 0x5558b6fc362f in sf_malloc /test/10.5_dbg_san/mysys/safemalloc.c:121
    #2 0x5558b6f810f8 in my_malloc /test/10.5_dbg_san/mysys/my_malloc.c:90
    #3 0x1490a4e1263e in myrocks::ha_rocksdb::alloc_key_buffers(TABLE const*, myrocks::Rdb_tbl_def const*, bool) /test/10.5_dbg_san/storage/rocksdb/ha_rocksdb.cc:6580
    #4 0x1490a4f0958c in myrocks::ha_rocksdb::open(char const*, int, unsigned int) /test/10.5_dbg_san/storage/rocksdb/ha_rocksdb.cc:6733
    #5 0x5558b3dac402 in handler::ha_open(TABLE*, char const*, int, unsigned int, st_mem_root*, List<String>*) /test/10.5_dbg_san/sql/handler.cc:2997
    #6 0x5558b2f82066 in open_table_from_share(THD*, TABLE_SHARE*, st_mysql_const_lex_string const*, unsigned int, unsigned int, unsigned int, TABLE*, bool, List<String>*) /test/10.5_dbg_san/sql/table.cc:4234
    #7 0x5558b22a3aaa in open_table(THD*, TABLE_LIST*, Open_table_context*) /test/10.5_dbg_san/sql/sql_base.cc:2001
    #8 0x5558b22c388c in open_and_process_table /test/10.5_dbg_san/sql/sql_base.cc:3786
    #9 0x5558b22c388c in open_tables(THD*, DDL_options_st const&, TABLE_LIST**, unsigned int*, unsigned int, Prelocking_strategy*) /test/10.5_dbg_san/sql/sql_base.cc:4269
    #10 0x5558b22c4d0c in open_and_lock_tables(THD*, DDL_options_st const&, TABLE_LIST*, bool, unsigned int, Prelocking_strategy*) /test/10.5_dbg_san/sql/sql_base.cc:5215
    #11 0x5558b250ab8b in open_and_lock_tables(THD*, TABLE_LIST*, bool, unsigned int) /test/10.5_dbg_san/sql/sql_base.h:507
    #12 0x5558b250ab8b in mysql_insert(THD*, TABLE_LIST*, List<Item>&, List<List<Item> >&, List<Item>&, List<Item>&, enum_duplicates, bool, select_result*) /test/10.5_dbg_san/sql/sql_insert.cc:757
    #13 0x5558b27936ad in mysql_execute_command(THD*) /test/10.5_dbg_san/sql/sql_parse.cc:4624
    #14 0x5558b26e8c6a in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) /test/10.5_dbg_san/sql/sql_parse.cc:8100
    #15 0x5558b276277d in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /test/10.5_dbg_san/sql/sql_parse.cc:1891
    #16 0x5558b2779a5c in do_command(THD*) /test/10.5_dbg_san/sql/sql_parse.cc:1370
    #17 0x5558b3168fc7 in do_handle_one_connection(CONNECT*, bool) /test/10.5_dbg_san/sql/sql_connect.cc:1418
    #18 0x5558b316c27f in handle_one_connection /test/10.5_dbg_san/sql/sql_connect.cc:1312
    #19 0x5558b56fdde7 in pfs_spawn_thread /test/10.5_dbg_san/storage/perfschema/pfs.cc:2201
    #20 0x1490ce192608 in start_thread /build/glibc-eX1tMB/glibc-2.31/nptl/pthread_create.c:477
 
Thread T13 created by T0 here:
    #0 0x5558b1dea265 in pthread_create (/test/UBASAN_MD060921-mariadb-10.5.13-linux-x86_64-dbg/bin/mariadbd+0x8188265)
    #1 0x5558b570e8ff in my_thread_create /test/10.5_dbg_san/storage/perfschema/my_thread.h:48
    #2 0x5558b570e8ff in pfs_spawn_thread_v1 /test/10.5_dbg_san/storage/perfschema/pfs.cc:2252
    #3 0x5558b1f1768d in inline_mysql_thread_create /test/10.5_dbg_san/include/mysql/psi/mysql_thread.h:1323
    #4 0x5558b1f1768d in create_thread_to_handle_connection(CONNECT*) /test/10.5_dbg_san/sql/mysqld.cc:6010
    #5 0x5558b1f2b34f in create_new_thread(CONNECT*) /test/10.5_dbg_san/sql/mysqld.cc:6069
    #6 0x5558b1f2b9e6 in handle_accepted_socket(st_mysql_socket, st_mysql_socket) /test/10.5_dbg_san/sql/mysqld.cc:6134
    #7 0x5558b1f2d474 in handle_connections_sockets() /test/10.5_dbg_san/sql/mysqld.cc:6261
    #8 0x5558b1f30f07 in mysqld_main(int, char**) /test/10.5_dbg_san/sql/mysqld.cc:5656
    #9 0x5558b1efebaa in main /test/10.5_dbg_san/sql/main.cc:25
    #10 0x1490cd30d0b2 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x270b2)
 
SUMMARY: AddressSanitizer: heap-buffer-overflow (/test/UBASAN_MD060921-mariadb-10.5.13-linux-x86_64-dbg/bin/mariadbd+0x81e8fdc) in __interceptor_memcpy.part.0
Shadow bytes around the buggy address:
  0x0c1c7fffb440: 00 00 00 00 00 00 04 fa fa fa fa fa fa fa fa fa
  0x0c1c7fffb450: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c1c7fffb460: 00 00 04 fa fa fa fa fa fa fa fa fa 00 00 00 00
  0x0c1c7fffb470: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 04
  0x0c1c7fffb480: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
=>0x0c1c7fffb490: 00 00 00 00 00 00 00 00 00 00 00[04]fa fa fa fa
  0x0c1c7fffb4a0: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c1c7fffb4b0: 00 00 00 00 00 00 00 04 fa fa fa fa fa fa fa fa
  0x0c1c7fffb4c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c1c7fffb4d0: 00 00 00 04 fa fa fa fa fa fa fa fa 00 00 00 00
  0x0c1c7fffb4e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 04
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==605817==ABORTING
210929 16:20:06 [ERROR] mysqld got signal 6 ;

Comment by Roel Van de Paar [ 2021-09-29 ]

The secondary testcase produces the crash, but does not produce an ASAN error on 10.5 debug nor on 10.7 debug, so there may be intertwined issues (one being the memory one, the other the crash, for example).

Comment by Roel Van de Paar [ 2021-09-29 ]

The secondary testcase, on optimized, can produce this warning:

10.5.13 c430aa72abbdccb1ece7f0d65b49a6b48e7c5ba7 (Optimized)

2021-09-29 16:26:11 4 [Warning] RocksDB: you're trying to create an index with a multi-level collation latin2_czech_cs
2021-09-29 16:26:11 4 [Warning] MyRocks will handle this collation internally  as if it had a NO_PAD attribute.

There may even be three issues here (memory corruption, crash from original testcase with VARBINARY(0), crash from secondary collation testcase. Crashes will end up with very similar stacks, though other stacks were observed also, ref below. Alternatively, the two crashes may be closely related + the memory corruption)

Comment by Roel Van de Paar [ 2021-09-29 ]

This issue can cause sporadic hangs. If not hanging then crash is observed. Using the orignal testcase (with drop/create test db and looped twice), observed client hang on connect to 10.2.41 (Optimized) past server crash (server crash already showing in error log, PID still live and not defunct). gdb breakin trace on PID:

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
__lll_lock_wait_private (futex=futex@entry=0x1554d5d47b80 <main_arena>) at ./lowlevellock.c:35
35	./lowlevellock.c: No such file or directory.
(gdb) bt
#0  __lll_lock_wait_private (futex=futex@entry=0x1554d5d47b80 <main_arena>) at ./lowlevellock.c:35
#1  0x00001554d5bf94ab in __GI___libc_malloc (bytes=bytes@entry=504) at malloc.c:3064
#2  0x0000564817cbc7c3 in my_malloc (size=size@entry=496, my_flags=my_flags@entry=16)
    at /test/10.2_opt/mysys/my_malloc.c:101
#3  0x0000564817d1f665 in mysql_socket_vio_new (mysql_socket=..., type=type@entry=VIO_TYPE_SOCKET, flags=flags@entry=1)
    at /test/10.2_opt/vio/vio.c:253
#4  0x0000564817519925 in handle_connections_sockets () at /test/10.2_opt/sql/mysqld.cc:6912
#5  0x000056481751a701 in mysqld_main (argc=<optimized out>, argv=<optimized out>) at /test/10.2_opt/sql/mysqld.cc:6192
#6  0x00001554d5b830b3 in __libc_start_main (main=0x5648174fb500 <main(int, char**)>, argc=10, argv=0x7ffe3bbb3ec8, 
    init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffe3bbb3eb8)
    at ../csu/libc-start.c:308
#7  0x000056481750d39e in _start () at /test/10.2_opt/sql/sql_list.h:196
(gdb) thread apply all bt
 
Thread 39 (Thread 0x1554a4bda700 (LWP 1599299)):
#0  futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x1554a4bd9dc0, clockid=<optimized out>, expected=0, futex_word=0x1554a6c84af0 <myrocks::rdb_mc_thread+144>) at ../sysdeps/nptl/futex-internal.h:320
#1  __pthread_cond_wait_common (abstime=0x1554a4bd9dc0, clockid=<optimized out>, mutex=0x1554a6c84a98 <myrocks::rdb_mc_thread+56>, cond=0x1554a6c84ac8 <myrocks::rdb_mc_thread+104>) at pthread_cond_wait.c:520
#2  __pthread_cond_timedwait (cond=cond@entry=0x1554a6c84ac8 <myrocks::rdb_mc_thread+104>, mutex=mutex@entry=0x1554a6c84a98 <myrocks::rdb_mc_thread+56>, abstime=abstime@entry=0x1554a4bd9dc0) at pthread_cond_wait.c:656
#3  0x00001554a67b719e in inline_mysql_cond_timedwait (src_file=0x1554a6b6dbb8 "/test/10.2_opt/storage/rocksdb/ha_rocksdb.cc", src_line=13561, abstime=0x1554a4bd9dc0, mutex=0x1554a6c84a98 <myrocks::rdb_mc_thread+56>, that=0x1554a6c84ac8 <myrocks::rdb_mc_thread+104>) at /test/10.2_opt/include/mysql/psi/mysql_thread.h:1193
#4  myrocks::Rdb_manual_compaction_thread::run (this=0x1554a6c84a60 <myrocks::rdb_mc_thread>) at /test/10.2_opt/storage/rocksdb/ha_rocksdb.cc:13561
#5  0x00001554a680f7f0 in myrocks::Rdb_thread::thread_func (thread_ptr=0x1554a6c84a60 <myrocks::rdb_mc_thread>) at /test/10.2_opt/storage/rocksdb/rdb_threads.cc:34
#6  myrocks::Rdb_thread::thread_func (thread_ptr=0x1554a6c84a60 <myrocks::rdb_mc_thread>) at /test/10.2_opt/storage/rocksdb/rdb_threads.cc:29
#7  0x00001554d608a609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#8  0x00001554d5c7e293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 38 (Thread 0x1554a49d9700 (LWP 1599298)):
#0  futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x1554a49d8cb0, clockid=<optimized out>, expected=0, futex_word=0x1554a6c847f0 <myrocks::rdb_drop_idx_thread+144>) at ../sysdeps/nptl/futex-internal.h:320
#1  __pthread_cond_wait_common (abstime=0x1554a49d8cb0, clockid=<optimized out>, mutex=0x1554a6c84798 <myrocks::rdb_drop_idx_thread+56>, cond=0x1554a6c847c8 <myrocks::rdb_drop_idx_thread+104>) at pthread_cond_wait.c:520
#2  __pthread_cond_timedwait (cond=cond@entry=0x1554a6c847c8 <myrocks::rdb_drop_idx_thread+104>, mutex=mutex@entry=0x1554a6c84798 <myrocks::rdb_drop_idx_thread+56>, abstime=abstime@entry=0x1554a49d8cb0) at pthread_cond_wait.c:656
#3  0x00001554a67c92ba in inline_mysql_cond_timedwait (src_file=0x1554a6b6dbb8 "/test/10.2_opt/storage/rocksdb/ha_rocksdb.cc", src_line=11526, abstime=0x1554a49d8cb0, mutex=0x1554a6c84798 <myrocks::rdb_drop_idx_thread+56>, that=0x1554a6c847c8 <myrocks::rdb_drop_idx_thread+104>) at /test/10.2_opt/include/mysql/psi/mysql_thread.h:1193
#4  myrocks::Rdb_drop_index_thread::run (this=0x1554a6c84760 <myrocks::rdb_drop_idx_thread>) at /test/10.2_opt/storage/rocksdb/ha_rocksdb.cc:11526
#5  0x00001554a680f7f0 in myrocks::Rdb_thread::thread_func (thread_ptr=0x1554a6c84760 <myrocks::rdb_drop_idx_thread>) at /test/10.2_opt/storage/rocksdb/rdb_threads.cc:34
#6  myrocks::Rdb_thread::thread_func (thread_ptr=0x1554a6c84760 <myrocks::rdb_drop_idx_thread>) at /test/10.2_opt/storage/rocksdb/rdb_threads.cc:29
#7  0x00001554d608a609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#8  0x00001554d5c7e293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 37 (Thread 0x1554a47d8700 (LWP 1599297)):
#0  futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x1554a47d7dc0, clockid=<optimized out>, expected=0, futex_word=0x1554a6c84c10 <myrocks::rdb_bg_thread+144>) at ../sysdeps/nptl/futex-internal.h:320
#1  __pthread_cond_wait_common (abstime=0x1554a47d7dc0, clockid=<optimized out>, mutex=0x1554a6c84bb8 <myrocks::rdb_bg_thread+56>, cond=0x1554a6c84be8 <myrocks::rdb_bg_thread+104>) at pthread_cond_wait.c:520
#2  __pthread_cond_timedwait (cond=cond@entry=0x1554a6c84be8 <myrocks::rdb_bg_thread+104>, mutex=mutex@entry=0x1554a6c84bb8 <myrocks::rdb_bg_thread+56>, abstime=abstime@entry=0x1554a47d7dc0) at pthread_cond_wait.c:656
#3  0x00001554a67cb152 in inline_mysql_cond_timedwait (src_file=0x1554a6b6dbb8 "/test/10.2_opt/storage/rocksdb/ha_rocksdb.cc", src_line=13465, abstime=0x1554a47d7dc0, mutex=0x1554a6c84bb8 <myrocks::rdb_bg_thread+56>, that=0x1554a6c84be8 <myrocks::rdb_bg_thread+104>) at /test/10.2_opt/include/mysql/psi/mysql_thread.h:1193
#4  myrocks::Rdb_background_thread::run (this=0x1554a6c84b80 <myrocks::rdb_bg_thread>) at /test/10.2_opt/storage/rocksdb/ha_rocksdb.cc:13465
#5  0x00001554a680f7f0 in myrocks::Rdb_thread::thread_func (thread_ptr=0x1554a6c84b80 <myrocks::rdb_bg_thread>) at /test/10.2_opt/storage/rocksdb/rdb_threads.cc:34
#6  myrocks::Rdb_thread::thread_func (thread_ptr=0x1554a6c84b80 <myrocks::rdb_bg_thread>) at /test/10.2_opt/storage/rocksdb/rdb_threads.cc:29
#7  0x00001554d608a609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#8  0x00001554d5c7e293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 36 (Thread 0x1554a45d7700 (LWP 1599050)):
#0  futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x1554a45d6d00, clockid=<optimized out>, expected=0, futex_word=0x155478208878) at ../sysdeps/nptl/futex-internal.h:320
#1  __pthread_cond_wait_common (abstime=0x1554a45d6d00, clockid=<optimized out>, mutex=0x155478208808, cond=0x155478208850) at pthread_cond_wait.c:520
#2  __pthread_cond_timedwait (cond=cond@entry=0x155478208850, mutex=0x155478208808, abstime=abstime@entry=0x1554a45d6d00) at pthread_cond_wait.c:656
#3  0x00001554a6ab7e02 in rocksdb::port::CondVar::TimedWait (this=this@entry=0x155478208850, abs_time_us=<optimized out>) at /test/10.2_opt/storage/rocksdb/rocksdb/port/port_posix.cc:120
#4  0x00001554a69993eb in rocksdb::InstrumentedCondVar::TimedWaitInternal (this=0x155478208850, abs_time_us=<optimized out>) at /test/10.2_opt/storage/rocksdb/rocksdb/monitoring/instrumented_mutex.cc:66
#5  0x00001554a69994cc in rocksdb::InstrumentedCondVar::TimedWait (this=this@entry=0x155478208850, abs_time_us=abs_time_us@entry=1632897676064446) at /test/10.2_opt/storage/rocksdb/rocksdb/monitoring/instrumented_mutex.cc:55
#6  0x00001554a68490e3 in rocksdb::RepeatableThread::wait (delay=<optimized out>, this=0x1554782087b0) at /test/10.2_opt/storage/rocksdb/rocksdb/util/repeatable_thread.h:90
#7  rocksdb::RepeatableThread::thread (this=0x1554782087b0) at /test/10.2_opt/storage/rocksdb/rocksdb/util/repeatable_thread.h:126
#8  rocksdb::RepeatableThread::RepeatableThread(std::function<void ()>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, unsigned long, unsigned long)::{lambda()#1}::operator()() const (__closure=<optimized out>) at /test/10.2_opt/storage/rocksdb/rocksdb/util/repeatable_thread.h:38
#9  std::__invoke_impl<void, rocksdb::RepeatableThread::RepeatableThread(std::function<void ()>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, unsigned long, unsigned long)::{lambda()#1}>(std::__invoke_other, rocksdb::RepeatableThread::RepeatableThread(std::function<void ()>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, unsigned long, unsigned long)::{lambda()#1}&&) (__f=...) at /usr/include/c++/9/bits/invoke.h:60
#10 std::__invoke<rocksdb::RepeatableThread::RepeatableThread(std::function<void ()>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, unsigned long, unsigned long)::{lambda()#1}>(std::__invoke_result&&, (rocksdb::RepeatableThread::RepeatableThread(std::function<void ()>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, unsigned long, unsigned long)::{lambda()#1}&&)...) (__fn=...) at /usr/include/c++/9/bits/invoke.h:95
#11 std::thread::_Invoker<std::tuple<rocksdb::RepeatableThread::RepeatableThread(std::function<void ()>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, unsigned long, unsigned long)::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=<optimized out>) at /usr/include/c++/9/thread:244
#12 std::thread::_Invoker<std::tuple<rocksdb::RepeatableThread::RepeatableThread(std::function<void ()>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, unsigned long, unsigned long)::{lambda()#1}> >::operator()() (this=<optimized out>) at /usr/include/c++/9/thread:251
#13 std::thread::_State_impl<std::thread::_Invoker<std::tuple<rocksdb::RepeatableThread::RepeatableThread(std::function<void ()>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, unsigned long, unsigned long)::{lambda()#1}> > >::_M_run() (this=<optimized out>) at /usr/include/c++/9/thread:195
#14 0x00001554d5f75de4 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#15 0x00001554d608a609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#16 0x00001554d5c7e293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 35 (Thread 0x1554a43d6700 (LWP 1599049)):
#0  futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x1554a43d5d00, clockid=<optimized out>, expected=0, futex_word=0x15547802b538) at ../sysdeps/nptl/futex-internal.h:320
#1  __pthread_cond_wait_common (abstime=0x1554a43d5d00, clockid=<optimized out>, mutex=0x15547802b4c8, cond=0x15547802b510) at pthread_cond_wait.c:520
#2  __pthread_cond_timedwait (cond=cond@entry=0x15547802b510, mutex=0x15547802b4c8, abstime=abstime@entry=0x1554a43d5d00) at pthread_cond_wait.c:656
#3  0x00001554a6ab7e02 in rocksdb::port::CondVar::TimedWait (this=this@entry=0x15547802b510, abs_time_us=<optimized out>) at /test/10.2_opt/storage/rocksdb/rocksdb/port/port_posix.cc:120
#4  0x00001554a69993eb in rocksdb::InstrumentedCondVar::TimedWaitInternal (this=0x15547802b510, abs_time_us=<optimized out>) at /test/10.2_opt/storage/rocksdb/rocksdb/monitoring/instrumented_mutex.cc:66
#5  0x00001554a69994cc in rocksdb::InstrumentedCondVar::TimedWait (this=this@entry=0x15547802b510, abs_time_us=abs_time_us@entry=1632897676066044) at /test/10.2_opt/storage/rocksdb/rocksdb/monitoring/instrumented_mutex.cc:55
#6  0x00001554a68490e3 in rocksdb::RepeatableThread::wait (delay=<optimized out>, this=0x15547802b470) at /test/10.2_opt/storage/rocksdb/rocksdb/util/repeatable_thread.h:90
#7  rocksdb::RepeatableThread::thread (this=0x15547802b470) at /test/10.2_opt/storage/rocksdb/rocksdb/util/repeatable_thread.h:126
#8  rocksdb::RepeatableThread::RepeatableThread(std::function<void ()>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, unsigned long, unsigned long)::{lambda()#1}::operator()() const (__closure=<optimized out>) at /test/10.2_opt/storage/rocksdb/rocksdb/util/repeatable_thread.h:38
#9  std::__invoke_impl<void, rocksdb::RepeatableThread::RepeatableThread(std::function<void ()>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, unsigned long, unsigned long)::{lambda()#1}>(std::__invoke_other, rocksdb::RepeatableThread::RepeatableThread(std::function<void ()>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, unsigned long, unsigned long)::{lambda()#1}&&) (__f=...) at /usr/include/c++/9/bits/invoke.h:60
#10 std::__invoke<rocksdb::RepeatableThread::RepeatableThread(std::function<void ()>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, unsigned long, unsigned long)::{lambda()#1}>(std::__invoke_result&&, (rocksdb::RepeatableThread::RepeatableThread(std::function<void ()>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, unsigned long, unsigned long)::{lambda()#1}&&)...) (__fn=...) at /usr/include/c++/9/bits/invoke.h:95
#11 std::thread::_Invoker<std::tuple<rocksdb::RepeatableThread::RepeatableThread(std::function<void ()>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, unsigned long, unsigned long)::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=<optimized out>) at /usr/include/c++/9/thread:244
#12 std::thread::_Invoker<std::tuple<rocksdb::RepeatableThread::RepeatableThread(std::function<void ()>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, unsigned long, unsigned long)::{lambda()#1}> >::operator()() (this=<optimized out>) at /usr/include/c++/9/thread:251
#13 std::thread::_State_impl<std::thread::_Invoker<std::tuple<rocksdb::RepeatableThread::RepeatableThread(std::function<void ()>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, unsigned long, unsigned long)::{lambda()#1}> > >::_M_run() (this=<optimized out>) at /usr/include/c++/9/thread:195
#14 0x00001554d5f75de4 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#15 0x00001554d608a609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#16 0x00001554d5c7e293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 34 (Thread 0x1554a61e5700 (LWP 1598998)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x15547801e590) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x15547801e540, cond=0x15547801e568) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=0x15547801e568, mutex=0x15547801e540) at pthread_cond_wait.c:638
#3  0x00001554d5f6fe30 in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4  0x00001554a6a5fa2d in rocksdb::ThreadPoolImpl::Impl::BGThread (this=0x15547801e4d0, thread_id=0) at /test/10.2_opt/storage/rocksdb/rocksdb/util/threadpool_imp.cc:196
#5  0x00001554a6a5fdbe in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper (arg=0x1554781f6f60) at /test/10.2_opt/storage/rocksdb/rocksdb/util/threadpool_imp.cc:306
#6  0x00001554d5f75de4 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#7  0x00001554d608a609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#8  0x00001554d5c7e293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 33 (Thread 0x1554a63e6700 (LWP 1598997)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x15547801e250) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x15547801e200, cond=0x15547801e228) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=0x15547801e228, mutex=0x15547801e200) at pthread_cond_wait.c:638
#3  0x00001554d5f6fe30 in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4  0x00001554a6a5fa2d in rocksdb::ThreadPoolImpl::Impl::BGThread (this=0x15547801e190, thread_id=0) at /test/10.2_opt/storage/rocksdb/rocksdb/util/threadpool_imp.cc:196
#5  0x00001554a6a5fdbe in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper (arg=0x1554781f6a50) at /test/10.2_opt/storage/rocksdb/rocksdb/util/threadpool_imp.cc:306
#6  0x00001554d5f75de4 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#7  0x00001554d608a609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#8  0x00001554d5c7e293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 32 (Thread 0x1554a65e7700 (LWP 1598996)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x1554781eea10) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x1554781ee908, cond=0x1554781ee9e8) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=cond@entry=0x1554781ee9e8, mutex=0x1554781ee908) at pthread_cond_wait.c:638
#3  0x00001554a6ab7d81 in rocksdb::port::CondVar::Wait (this=this@entry=0x1554781ee9e8) at /test/10.2_opt/storage/rocksdb/rocksdb/port/port_posix.cc:106
#4  0x00001554a699918a in rocksdb::InstrumentedCondVar::WaitInternal (this=this@entry=0x1554781ee9e8) at /test/10.2_opt/storage/rocksdb/rocksdb/monitoring/instrumented_mutex.cc:48
#5  0x00001554a6999230 in rocksdb::InstrumentedCondVar::Wait (this=this@entry=0x1554781ee9e8) at /test/10.2_opt/storage/rocksdb/rocksdb/monitoring/instrumented_mutex.cc:41
#6  0x00001554a697d598 in rocksdb::DeleteScheduler::BackgroundEmptyTrash (this=0x1554781ee8e8) at /test/10.2_opt/storage/rocksdb/rocksdb/file/delete_scheduler.cc:196
#7  0x00001554d5f75de4 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#8  0x00001554d608a609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#9  0x00001554d5c7e293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 31 (Thread 0x1554d4070700 (LWP 1585530)):
#0  __lll_lock_wait_private (futex=futex@entry=0x1554d5d47b80 <main_arena>) at ./lowlevellock.c:35
#1  0x00001554d5bf9049 in __malloc_fork_lock_parent () at arena.c:156
#2  0x00001554d5c421aa in __libc_fork () at ../sysdeps/nptl/fork.c:73
#3  0x0000564817cdb9b4 in start_addr2line_fork (binary_path=0x7ffe3bbb447c "/test/MD180921-mariadb-10.2.41-linux-x86_64-opt/bin/mysqld") at /test/10.2_opt/mysys/my_addr_resolve.c:192
#4  start_addr2line_fork (binary_path=0x7ffe3bbb447c "/test/MD180921-mariadb-10.2.41-linux-x86_64-opt/bin/mysqld") at /test/10.2_opt/mysys/my_addr_resolve.c:175
#5  0x0000564817cdbae8 in my_addr_resolve (ptr=0x564817cc09f2 <my_print_stacktrace+50>, loc=loc@entry=0x1554d406cb80) at /test/10.2_opt/mysys/my_addr_resolve.c:241
#6  0x0000564817cc0a4b in print_with_addr_resolve (n=<optimized out>, addrs=0x1554d406cba0) at /test/10.2_opt/mysys/stacktrace.c:158
#7  my_print_stacktrace (stack_bottom=<optimized out>, thread_stack=299008) at /test/10.2_opt/mysys/stacktrace.c:176
#8  0x000056481776585b in handle_fatal_signal (sig=11) at /test/10.2_opt/sql/signal_handler.cc:218
#9  <signal handler called>
#10 _int_free (av=0x1554d5d47b80 <main_arena>, p=0x155478251480, have_lock=<optimized out>) at malloc.c:4316
#11 0x00001554a67fa17d in String::free (this=0x1554d406d6c0) at /test/10.2_opt/sql/./sql_string.h:351
#12 String::~String (this=0x1554d406d6c0, __in_chrg=<optimized out>) at /test/10.2_opt/sql/./sql_string.h:187
#13 myrocks::Rdb_field_packing::fill_hidden_pk_val (this=0x1554782b7970, dst=0x1554d406d768, hidden_pk_id=<optimized out>) at /test/10.2_opt/storage/rocksdb/rdb_datadic.cc:3492
#14 0x00001554a67ff2e8 in myrocks::Rdb_key_def::pack_record (this=this@entry=0x15547820ce00, tbl=tbl@entry=0x15547822b1b8, pack_buffer=0x1554781f5628 "\260g xT\025", record=0x15547822a278 "\376", packed_tuple=0x155478251478 "", unpack_info=unpack_info@entry=0x155478060310, should_store_row_debug_checksums=false, hidden_pk_id=1, n_key_parts=2, n_null_fields=0x0, ttl_bytes=0x155478257431 "") at /test/10.2_opt/storage/rocksdb/rdb_datadic.cc:1362
#15 0x00001554a67c5765 in myrocks::ha_rocksdb::update_write_sk (this=this@entry=0x15547805fdd0, table_arg=0x15547822b1b8, kd=..., row_info=..., bulk_load_sk=bulk_load_sk@entry=false) at /test/10.2_opt/storage/rocksdb/ha_rocksdb.cc:10131
#16 0x00001554a67c5eb4 in myrocks::ha_rocksdb::update_write_indexes (pk_changed=<optimized out>, row_info=..., this=0x15547805fdd0) at /usr/include/c++/9/bits/shared_ptr_base.h:1020
#17 myrocks::ha_rocksdb::update_write_indexes (this=0x15547805fdd0, row_info=..., pk_changed=<optimized out>) at /test/10.2_opt/storage/rocksdb/ha_rocksdb.cc:10210
#18 0x00001554a67c6034 in myrocks::ha_rocksdb::update_write_row (this=this@entry=0x15547805fdd0, old_data=old_data@entry=0x0, new_data=new_data@entry=0x15547822a278 "\376", skip_unique_check=<optimized out>) at /test/10.2_opt/storage/rocksdb/ha_rocksdb.cc:10315
#19 0x00001554a67c617b in myrocks::ha_rocksdb::write_row (this=0x15547805fdd0, buf=0x15547822a278 "\376") at /test/10.2_opt/storage/rocksdb/ha_rocksdb.cc:9524
#20 0x000056481777093f in handler::ha_write_row (this=0x15547805fdd0, buf=0x15547822a278 "\376") at /test/10.2_opt/sql/handler.cc:6118
#21 0x000056481759522d in write_record (thd=thd@entry=0x155478000c48, table=table@entry=0x15547822b1b8, info=info@entry=0x1554d406dc30) at /test/10.2_opt/sql/sql_insert.cc:1940
#22 0x000056481759f268 in mysql_insert (thd=thd@entry=0x155478000c48, table_list=<optimized out>, fields=..., values_list=..., update_fields=..., update_values=..., duplic=<optimized out>, ignore=<optimized out>) at /test/10.2_opt/sql/sql_insert.cc:1066
#23 0x00005648175b4733 in mysql_execute_command (thd=<optimized out>) at /test/10.2_opt/sql/sql_parse.cc:4217
#24 0x00005648175bb45a in mysql_parse (thd=thd@entry=0x155478000c48, rawbuf=<optimized out>, length=<optimized out>, parser_state=parser_state@entry=0x1554d406f560, is_com_multi=is_com_multi@entry=false, is_next_command=is_next_command@entry=false) at /test/10.2_opt/sql/sql_parse.cc:7793
#25 0x00005648175be41d in dispatch_command (command=COM_QUERY, thd=0x155478000c48, packet=0x155478006fb9 "INSERT IGNORE INTO t1 VALUES (1)", packet_length=32, is_com_multi=<optimized out>, is_next_command=<optimized out>) at /test/10.2_opt/sql/sql_class.h:1109
#26 0x00005648175bf85d in do_command (thd=0x155478000c48) at /test/10.2_opt/sql/sql_parse.cc:1381
#27 0x00005648176986c6 in do_handle_one_connection (connect=connect@entry=0x56481a7e4d98) at /test/10.2_opt/sql/sql_connect.cc:1336
#28 0x000056481769883f in handle_one_connection (arg=0x56481a7e4d98) at /test/10.2_opt/sql/sql_connect.cc:1241
#29 0x00001554d608a609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#30 0x00001554d5c7e293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 30 (Thread 0x1554d40bb700 (LWP 1585349)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x5648183de6e8 <COND_manager+40>) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5648183de700 <LOCK_manager>, cond=0x5648183de6c0 <COND_manager>) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=cond@entry=0x5648183de6c0 <COND_manager>, mutex=mutex@entry=0x5648183de700 <LOCK_manager>) at pthread_cond_wait.c:638
#3  0x00005648175aa56b in inline_mysql_cond_wait (that=0x5648183de6c0 <COND_manager>, mutex=0x5648183de700 <LOCK_manager>, src_file=0x564817d58080 "/test/10.2_opt/sql/sql_manager.cc", src_line=102) at /test/10.2_opt/include/mysql/psi/mysql_thread.h:1156
#4  handle_manager (arg=<optimized out>) at /test/10.2_opt/sql/sql_manager.cc:102
#5  0x00001554d608a609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6  0x00001554d5c7e293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 29 (Thread 0x1554d4106700 (LWP 1585348)):
#0  0x00001554d5ba3322 in __GI___sigtimedwait (set=set@entry=0x1554d4105da0, info=info@entry=0x1554d4105c70, timeout=timeout@entry=0x0) at ../sysdeps/unix/sysv/linux/sigtimedwait.c:29
#1  0x00001554d6095f6c in __sigwait (set=set@entry=0x1554d4105da0, sig=sig@entry=0x1554d4105d48) at ../sysdeps/unix/sysv/linux/sigwait.c:28
#2  0x000056481750f8db in signal_hand (arg=<optimized out>) at /test/10.2_opt/sql/mysqld.cc:3557
#3  0x00001554d608a609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#4  0x00001554d5c7e293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 28 (Thread 0x1554a6ead700 (LWP 1585347)):
#0  futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x1554a6eacd70, clockid=<optimized out>, expected=0, futex_word=0x564818c57bc8 <COND_checkpoint+40>) at ../sysdeps/nptl/futex-internal.h:320
#1  __pthread_cond_wait_common (abstime=0x1554a6eacd70, clockid=<optimized out>, mutex=0x564818c57be0 <LOCK_checkpoint>, cond=0x564818c57ba0 <COND_checkpoint>) at pthread_cond_wait.c:520
#2  __pthread_cond_timedwait (cond=cond@entry=0x564818c57ba0 <COND_checkpoint>, mutex=mutex@entry=0x564818c57be0 <LOCK_checkpoint>, abstime=abstime@entry=0x1554a6eacd70) at pthread_cond_wait.c:656
#3  0x0000564817b72837 in inline_mysql_cond_timedwait (src_file=0x564817e4cdb8 "/test/10.2_opt/storage/maria/ma_servicethread.c", src_line=115, abstime=0x1554a6eacd70, mutex=0x564818c57be0 <LOCK_checkpoint>, that=0x564818c57ba0 <COND_checkpoint>) at /test/10.2_opt/include/mysql/psi/mysql_thread.h:1193
#4  my_service_thread_sleep (control=control@entry=0x56481835b8c0 <checkpoint_control>, sleep_time=sleep_time@entry=30000000000) at /test/10.2_opt/storage/maria/ma_servicethread.c:115
#5  0x0000564817b6ae3a in ma_checkpoint_background (arg=0x1e) at /test/10.2_opt/storage/maria/ma_checkpoint.c:709
#6  0x00001554d608a609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#7  0x00001554d5c7e293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 27 (Thread 0x1554a75fa700 (LWP 1585345)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x1554a75f9e28) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x1554a75f9dd0, cond=0x1554a75f9e00) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=cond@entry=0x1554a75f9e00, mutex=mutex@entry=0x1554a75f9dd0) at pthread_cond_wait.c:638
#3  0x00005648178f9ec9 in inline_mysql_cond_wait (src_file=0x564817dfca80 "/test/10.2_opt/storage/innobase/handler/ha_innodb.cc", src_line=347, mutex=0x1554a75f9dd0, that=0x1554a75f9e00) at /test/10.2_opt/include/mysql/psi/mysql_thread.h:1156
#4  thd_destructor_proxy () at /test/10.2_opt/storage/innobase/handler/ha_innodb.cc:347
#5  0x00001554d608a609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6  0x00001554d5c7e293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 26 (Thread 0x1554a77fb700 (LWP 1585344)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x56481a0fea40) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x56481a0fe9f0, cond=0x56481a0fea18) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=cond@entry=0x56481a0fea18, mutex=mutex@entry=0x56481a0fe9f0) at pthread_cond_wait.c:638
#3  0x000056481797a161 in os_event::wait (this=0x56481a0fe9e0) at /test/10.2_opt/storage/innobase/include/sync0types.h:484
#4  os_event::wait_low (this=0x56481a0fe9e0, reset_sig_count=1) at /test/10.2_opt/storage/innobase/os/os0event.cc:325
#5  0x0000564817a8023a in buf_resize_thread () at /test/10.2_opt/storage/innobase/buf/buf0buf.cc:3110
#6  0x00001554d608a609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#7  0x00001554d5c7e293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 25 (Thread 0x1554a79fc700 (LWP 1585343)):
#0  0x00001554d5c3c3bf in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x1554a79fb760, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78
#1  0x00001554d5c42047 in __GI___nanosleep (requested_time=requested_time@entry=0x1554a79fb760, remaining=remaining@entry=0x0) at nanosleep.c:27
#2  0x000056481797a684 in os_thread_sleep (tm=tm@entry=1000000) at /test/10.2_opt/storage/innobase/os/os0thread.cc:231
#3  0x0000564817a6f722 in btr_defragment_thread () at /test/10.2_opt/storage/innobase/btr/btr0defragment.cc:723
#4  0x00001554d608a609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#5  0x00001554d5c7e293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 24 (Thread 0x1554a7bfd700 (LWP 1585342)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x56481a0fe960) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x56481a0fe910, cond=0x56481a0fe938) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=cond@entry=0x56481a0fe938, mutex=mutex@entry=0x56481a0fe910) at pthread_cond_wait.c:638
#3  0x000056481797a161 in os_event::wait (this=0x56481a0fe900) at /test/10.2_opt/storage/innobase/include/sync0types.h:484
#4  os_event::wait_low (this=0x56481a0fe900, reset_sig_count=1) at /test/10.2_opt/storage/innobase/os/os0event.cc:325
#5  0x0000564817a85e68 in buf_dump_thread () at /test/10.2_opt/storage/innobase/buf/buf0dump.cc:806
#6  0x00001554d608a609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#7  0x00001554d5c7e293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 23 (Thread 0x1554a7dfe700 (LWP 1585341)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x56481a0fe810) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x56481a0fe7c0, cond=0x56481a0fe7e8) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=cond@entry=0x56481a0fe7e8, mutex=mutex@entry=0x56481a0fe7c0) at pthread_cond_wait.c:638
#3  0x000056481797a161 in os_event::wait (this=0x56481a0fe7b0) at /test/10.2_opt/storage/innobase/include/sync0types.h:484
#4  os_event::wait_low (this=0x56481a0fe7b0, reset_sig_count=1) at /test/10.2_opt/storage/innobase/os/os0event.cc:325
#5  0x00005648179fcd7e in srv_resume_thread (sig_count=0, timeout_usec=0, wait=true, slot=0x564818359d68 <srv_sys+360>) at /test/10.2_opt/storage/innobase/srv/srv0srv.cc:2621
#6  srv_worker_thread (arg=<optimized out>) at /test/10.2_opt/storage/innobase/srv/srv0srv.cc:2621
#7  0x00001554d608a609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#8  0x00001554d5c7e293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 22 (Thread 0x1554a7fff700 (LWP 1585340)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x56481a0fe7a0) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x56481a0fe750, cond=0x56481a0fe778) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=cond@entry=0x56481a0fe778, mutex=mutex@entry=0x56481a0fe750) at pthread_cond_wait.c:638
#3  0x000056481797a161 in os_event::wait (this=0x56481a0fe740) at /test/10.2_opt/storage/innobase/include/sync0types.h:484
#4  os_event::wait_low (this=0x56481a0fe740, reset_sig_count=1) at /test/10.2_opt/storage/innobase/os/os0event.cc:325
#5  0x00005648179fcd7e in srv_resume_thread (sig_count=0, timeout_usec=0, wait=true, slot=0x564818359d30 <srv_sys+304>) at /test/10.2_opt/storage/innobase/srv/srv0srv.cc:2621
#6  srv_worker_thread (arg=<optimized out>) at /test/10.2_opt/storage/innobase/srv/srv0srv.cc:2621
#7  0x00001554d608a609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#8  0x00001554d5c7e293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 21 (Thread 0x1554bc30c700 (LWP 1585339)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x56481a0fe730) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x56481a0fe6e0, cond=0x56481a0fe708) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=cond@entry=0x56481a0fe708, mutex=mutex@entry=0x56481a0fe6e0) at pthread_cond_wait.c:638
#3  0x000056481797a161 in os_event::wait (this=0x56481a0fe6d0) at /test/10.2_opt/storage/innobase/include/sync0types.h:484
#4  os_event::wait_low (this=0x56481a0fe6d0, reset_sig_count=1) at /test/10.2_opt/storage/innobase/os/os0event.cc:325
#5  0x00005648179fcd7e in srv_resume_thread (sig_count=0, timeout_usec=0, wait=true, slot=0x564818359cf8 <srv_sys+248>) at /test/10.2_opt/storage/innobase/srv/srv0srv.cc:2621
#6  srv_worker_thread (arg=<optimized out>) at /test/10.2_opt/storage/innobase/srv/srv0srv.cc:2621
#7  0x00001554d608a609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#8  0x00001554d5c7e293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 20 (Thread 0x1554bc50d700 (LWP 1585338)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x56481a0fe6c0) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x56481a0fe670, cond=0x56481a0fe698) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=cond@entry=0x56481a0fe698, mutex=mutex@entry=0x56481a0fe670) at pthread_cond_wait.c:638
#3  0x000056481797a161 in os_event::wait (this=0x56481a0fe660) at /test/10.2_opt/storage/innobase/include/sync0types.h:484
#4  os_event::wait_low (this=0x56481a0fe660, reset_sig_count=1) at /test/10.2_opt/storage/innobase/os/os0event.cc:325
#5  0x00005648179fe04a in srv_resume_thread (timeout_usec=0, wait=true, sig_count=1, slot=0x564818359cc0 <srv_sys+192>) at /test/10.2_opt/storage/innobase/srv/srv0srv.cc:2782
#6  srv_purge_coordinator_suspend (rseg_history_len=<optimized out>, slot=<optimized out>) at /test/10.2_opt/storage/innobase/srv/srv0srv.cc:2782
#7  srv_purge_coordinator_thread (arg=<optimized out>) at /test/10.2_opt/storage/innobase/srv/srv0srv.cc:2877
#8  0x00001554d608a609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#9  0x00001554d5c7e293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 19 (Thread 0x1554bc70e700 (LWP 1585336)):
#0  futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x1554bc70db40, clockid=<optimized out>, expected=0, futex_word=0x56481a47f480) at ../sysdeps/nptl/futex-internal.h:320
#1  __pthread_cond_wait_common (abstime=0x1554bc70db40, clockid=<optimized out>, mutex=0x56481a47f430, cond=0x56481a47f458) at pthread_cond_wait.c:520
#2  __pthread_cond_timedwait (cond=0x56481a47f458, mutex=0x56481a47f430, abstime=abstime@entry=0x1554bc70db40) at pthread_cond_wait.c:656
#3  0x000056481797a0c6 in os_event::timed_wait (this=<optimized out>, abstime=0x1554bc70db40) at /test/10.2_opt/storage/innobase/include/sync0types.h:484
#4  0x000056481797a1f1 in os_event::wait_time_low (this=0x56481a47f420, time_in_usec=<optimized out>, reset_sig_count=1) at /test/10.2_opt/storage/innobase/os/os0event.cc:385
#5  0x0000564817a3eb68 in ib_wqueue_timedwait (wq=wq@entry=0x56481a0ce5c0, wait_in_usecs=wait_in_usecs@entry=5000000) at /test/10.2_opt/storage/innobase/ut/ut0wqueue.cc:154
#6  0x0000564817b0ea39 in fts_optimize_thread (arg=<optimized out>) at /test/10.2_opt/storage/innobase/fts/fts0opt.cc:2826
#7  0x00001554d608a609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#8  0x00001554d5c7e293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 18 (Thread 0x1554bc90f700 (LWP 1585335)):
#0  futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x1554bc90ed10, clockid=<optimized out>, expected=0, futex_word=0x56481a43c590) at ../sysdeps/nptl/futex-internal.h:320
#1  __pthread_cond_wait_common (abstime=0x1554bc90ed10, clockid=<optimized out>, mutex=0x56481a43c540, cond=0x56481a43c568) at pthread_cond_wait.c:520
#2  __pthread_cond_timedwait (cond=0x56481a43c568, mutex=0x56481a43c540, abstime=abstime@entry=0x1554bc90ed10) at pthread_cond_wait.c:656
#3  0x000056481797a0c6 in os_event::timed_wait (this=<optimized out>, abstime=0x1554bc90ed10) at /test/10.2_opt/storage/innobase/include/sync0types.h:484
#4  0x000056481797a1f1 in os_event::wait_time_low (this=0x56481a43c530, time_in_usec=<optimized out>, reset_sig_count=1) at /test/10.2_opt/storage/innobase/os/os0event.cc:385
#5  0x0000564817acc31e in dict_stats_thread () at /test/10.2_opt/storage/innobase/dict/dict0stats_bg.cc:489
#6  0x00001554d608a609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#7  0x00001554d5c7e293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 17 (Thread 0x1554d4e3f700 (LWP 1585334)):
#0  0x00001554d5c3c3bf in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x1554d4e3ed70, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78
#1  0x00001554d5c42047 in __GI___nanosleep (requested_time=requested_time@entry=0x1554d4e3ed70, remaining=remaining@entry=0x0) at nanosleep.c:27
#2  0x000056481797a684 in os_thread_sleep (tm=tm@entry=1000000) at /test/10.2_opt/storage/innobase/os/os0thread.cc:231
#3  0x00005648179fbcb5 in srv_master_sleep () at /test/10.2_opt/storage/innobase/srv/srv0srv.cc:2416
#4  srv_master_thread (arg=<optimized out>) at /test/10.2_opt/storage/innobase/srv/srv0srv.cc:2456
#5  0x00001554d608a609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6  0x00001554d5c7e293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 16 (Thread 0x1554bcb10700 (LWP 1585329)):
#0  futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x1554bcb0fd40, clockid=<optimized out>, expected=0, futex_word=0x56481a0fe8f0) at ../sysdeps/nptl/futex-internal.h:320
#1  __pthread_cond_wait_common (abstime=0x1554bcb0fd40, clockid=<optimized out>, mutex=0x56481a0fe8a0, cond=0x56481a0fe8c8) at pthread_cond_wait.c:520
#2  __pthread_cond_timedwait (cond=0x56481a0fe8c8, mutex=0x56481a0fe8a0, abstime=abstime@entry=0x1554bcb0fd40) at pthread_cond_wait.c:656
#3  0x000056481797a0c6 in os_event::timed_wait (this=<optimized out>, abstime=0x1554bcb0fd40) at /test/10.2_opt/storage/innobase/include/sync0types.h:484
#4  0x000056481797a1f1 in os_event::wait_time_low (this=0x56481a0fe890, time_in_usec=<optimized out>, reset_sig_count=1) at /test/10.2_opt/storage/innobase/os/os0event.cc:385
#5  0x00005648179fa7f0 in srv_monitor_thread () at /test/10.2_opt/storage/innobase/srv/srv0srv.cc:1745
#6  0x00001554d608a609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#7  0x00001554d5c7e293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 15 (Thread 0x1554bcd11700 (LWP 1585328)):
#0  futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x1554bcd10c50, clockid=<optimized out>, expected=0, futex_word=0x56481a0fe880) at ../sysdeps/nptl/futex-internal.h:320
#1  __pthread_cond_wait_common (abstime=0x1554bcd10c50, clockid=<optimized out>, mutex=0x56481a0fe830, cond=0x56481a0fe858) at pthread_cond_wait.c:520
#2  __pthread_cond_timedwait (cond=0x56481a0fe858, mutex=0x56481a0fe830, abstime=abstime@entry=0x1554bcd10c50) at pthread_cond_wait.c:656
#3  0x000056481797a0c6 in os_event::timed_wait (this=<optimized out>, abstime=0x1554bcd10c50) at /test/10.2_opt/storage/innobase/include/sync0types.h:484
#4  0x000056481797a1f1 in os_event::wait_time_low (this=0x56481a0fe820, time_in_usec=<optimized out>, reset_sig_count=1) at /test/10.2_opt/storage/innobase/os/os0event.cc:385
#5  0x00005648179fb0c7 in srv_error_monitor_thread () at /test/10.2_opt/storage/innobase/srv/srv0srv.cc:1909
#6  0x00001554d608a609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#7  0x00001554d5c7e293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 14 (Thread 0x1554d4c3e700 (LWP 1585327)):
#0  futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x1554d4c3dce0, clockid=<optimized out>, expected=0, futex_word=0x56481a43a7a0) at ../sysdeps/nptl/futex-internal.h:320
#1  __pthread_cond_wait_common (abstime=0x1554d4c3dce0, clockid=<optimized out>, mutex=0x56481a43a750, cond=0x56481a43a778) at pthread_cond_wait.c:520
#2  __pthread_cond_timedwait (cond=0x56481a43a778, mutex=0x56481a43a750, abstime=abstime@entry=0x1554d4c3dce0) at pthread_cond_wait.c:656
#3  0x000056481797a0c6 in os_event::timed_wait (this=<optimized out>, abstime=0x1554d4c3dce0) at /test/10.2_opt/storage/innobase/include/sync0types.h:484
#4  0x000056481797a1f1 in os_event::wait_time_low (this=0x56481a43a740, time_in_usec=<optimized out>, reset_sig_count=1) at /test/10.2_opt/storage/innobase/os/os0event.cc:385
#5  0x000056481795aa87 in lock_wait_timeout_thread () at /test/10.2_opt/storage/innobase/lock/lock0wait.cc:530
#6  0x00001554d608a609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#7  0x00001554d5c7e293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 13 (Thread 0x1554bd117700 (LWP 1585315)):
#0  futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x1554bd116ba0, clockid=<optimized out>, expected=0, futex_word=0x56481a0fe9d4) at ../sysdeps/nptl/futex-internal.h:320
#1  __pthread_cond_wait_common (abstime=0x1554bd116ba0, clockid=<optimized out>, mutex=0x56481a0fe980, cond=0x56481a0fe9a8) at pthread_cond_wait.c:520
#2  __pthread_cond_timedwait (cond=0x56481a0fe9a8, mutex=0x56481a0fe980, abstime=abstime@entry=0x1554bd116ba0) at pthread_cond_wait.c:656
#3  0x000056481797a0c6 in os_event::timed_wait (this=<optimized out>, abstime=0x1554bd116ba0) at /test/10.2_opt/storage/innobase/include/sync0types.h:484
#4  0x000056481797a1f1 in os_event::wait_time_low (this=0x56481a0fe970, time_in_usec=<optimized out>, reset_sig_count=466) at /test/10.2_opt/storage/innobase/os/os0event.cc:385
#5  0x00005648174e0c51 in pc_sleep_if_needed (cur_time=16802165, sig_count=466, next_loop_time=16803165) at /test/10.2_opt/storage/innobase/buf/buf0flu.cc:2682
#6  buf_flush_page_cleaner_coordinator () at /test/10.2_opt/storage/innobase/buf/buf0flu.cc:3148
#7  0x00001554d608a609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#8  0x00001554d5c7e293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 12 (Thread 0x1554bd318700 (LWP 1585314)):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00001554d619527b in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1
#2  0x0000564817973c4d in LinuxAIOHandler::collect (this=0x1554bd317ba0) at /test/10.2_opt/storage/innobase/os/os0file.cc:1940
#3  0x0000564817977d33 in LinuxAIOHandler::poll (this=0x1554bd317ba0, m1=0x1554bd317c70, m2=0x1554bd317c78, request=0x1554bd317c80) at /test/10.2_opt/storage/innobase/os/os0file.cc:2101
#4  0x00005648179791b3 in os_aio_linux_handler (request=0x1554bd317c80, m2=0x1554bd317c78, m1=0x1554bd317c70, global_segment=<optimized out>) at /usr/include/c++/9/bits/stl_vector.h:915
#5  os_aio_handler (segment=segment@entry=9, m1=m1@entry=0x1554bd317c70, m2=m2@entry=0x1554bd317c78, request=request@entry=0x1554bd317c80) at /test/10.2_opt/storage/innobase/os/os0file.cc:5818
#6  0x0000564817ad624c in fil_aio_wait (segment=segment@entry=9) at /test/10.2_opt/storage/innobase/fil/fil0fil.cc:5153
#7  0x00005648179ff818 in io_handler_thread (arg=<optimized out>) at /test/10.2_opt/storage/innobase/srv/srv0start.cc:332
#8  0x00001554d608a609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#9  0x00001554d5c7e293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 11 (Thread 0x1554bd519700 (LWP 1585313)):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00001554d619527b in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1
#2  0x0000564817973c4d in LinuxAIOHandler::collect (this=0x1554bd518ba0) at /test/10.2_opt/storage/innobase/os/os0file.cc:1940
#3  0x0000564817977d33 in LinuxAIOHandler::poll (this=0x1554bd518ba0, m1=0x1554bd518c70, m2=0x1554bd518c78, request=0x1554bd518c80) at /test/10.2_opt/storage/innobase/os/os0file.cc:2101
#4  0x00005648179791b3 in os_aio_linux_handler (request=0x1554bd518c80, m2=0x1554bd518c78, m1=0x1554bd518c70, global_segment=<optimized out>) at /usr/include/c++/9/bits/stl_vector.h:915
#5  os_aio_handler (segment=segment@entry=8, m1=m1@entry=0x1554bd518c70, m2=m2@entry=0x1554bd518c78, request=request@entry=0x1554bd518c80) at /test/10.2_opt/storage/innobase/os/os0file.cc:5818
#6  0x0000564817ad624c in fil_aio_wait (segment=segment@entry=8) at /test/10.2_opt/storage/innobase/fil/fil0fil.cc:5153
#7  0x00005648179ff818 in io_handler_thread (arg=<optimized out>) at /test/10.2_opt/storage/innobase/srv/srv0start.cc:332
#8  0x00001554d608a609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#9  0x00001554d5c7e293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 10 (Thread 0x1554bd71a700 (LWP 1585312)):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00001554d619527b in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1
#2  0x0000564817973c4d in LinuxAIOHandler::collect (this=0x1554bd719ba0) at /test/10.2_opt/storage/innobase/os/os0file.cc:1940
#3  0x0000564817977d33 in LinuxAIOHandler::poll (this=0x1554bd719ba0, m1=0x1554bd719c70, m2=0x1554bd719c78, request=0x1554bd719c80) at /test/10.2_opt/storage/innobase/os/os0file.cc:2101
#4  0x00005648179791b3 in os_aio_linux_handler (request=0x1554bd719c80, m2=0x1554bd719c78, m1=0x1554bd719c70, global_segment=<optimized out>) at /usr/include/c++/9/bits/stl_vector.h:915
#5  os_aio_handler (segment=segment@entry=7, m1=m1@entry=0x1554bd719c70, m2=m2@entry=0x1554bd719c78, request=request@entry=0x1554bd719c80) at /test/10.2_opt/storage/innobase/os/os0file.cc:5818
#6  0x0000564817ad624c in fil_aio_wait (segment=segment@entry=7) at /test/10.2_opt/storage/innobase/fil/fil0fil.cc:5153
#7  0x00005648179ff818 in io_handler_thread (arg=<optimized out>) at /test/10.2_opt/storage/innobase/srv/srv0start.cc:332
#8  0x00001554d608a609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#9  0x00001554d5c7e293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 9 (Thread 0x1554bd91b700 (LWP 1585310)):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00001554d619527b in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1
#2  0x0000564817973c4d in LinuxAIOHandler::collect (this=0x1554bd91aba0) at /test/10.2_opt/storage/innobase/os/os0file.cc:1940
#3  0x0000564817977d33 in LinuxAIOHandler::poll (this=0x1554bd91aba0, m1=0x1554bd91ac70, m2=0x1554bd91ac78, request=0x1554bd91ac80) at /test/10.2_opt/storage/innobase/os/os0file.cc:2101
#4  0x00005648179791b3 in os_aio_linux_handler (request=0x1554bd91ac80, m2=0x1554bd91ac78, m1=0x1554bd91ac70, global_segment=<optimized out>) at /usr/include/c++/9/bits/stl_vector.h:915
#5  os_aio_handler (segment=segment@entry=6, m1=m1@entry=0x1554bd91ac70, m2=m2@entry=0x1554bd91ac78, request=request@entry=0x1554bd91ac80) at /test/10.2_opt/storage/innobase/os/os0file.cc:5818
#6  0x0000564817ad624c in fil_aio_wait (segment=segment@entry=6) at /test/10.2_opt/storage/innobase/fil/fil0fil.cc:5153
#7  0x00005648179ff818 in io_handler_thread (arg=<optimized out>) at /test/10.2_opt/storage/innobase/srv/srv0start.cc:332
#8  0x00001554d608a609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#9  0x00001554d5c7e293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 8 (Thread 0x1554bdb1c700 (LWP 1585309)):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00001554d619527b in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1
#2  0x0000564817973c4d in LinuxAIOHandler::collect (this=0x1554bdb1bba0) at /test/10.2_opt/storage/innobase/os/os0file.cc:1940
#3  0x0000564817977d33 in LinuxAIOHandler::poll (this=0x1554bdb1bba0, m1=0x1554bdb1bc70, m2=0x1554bdb1bc78, request=0x1554bdb1bc80) at /test/10.2_opt/storage/innobase/os/os0file.cc:2101
#4  0x00005648179791b3 in os_aio_linux_handler (request=0x1554bdb1bc80, m2=0x1554bdb1bc78, m1=0x1554bdb1bc70, global_segment=<optimized out>) at /usr/include/c++/9/bits/stl_vector.h:915
#5  os_aio_handler (segment=segment@entry=5, m1=m1@entry=0x1554bdb1bc70, m2=m2@entry=0x1554bdb1bc78, request=request@entry=0x1554bdb1bc80) at /test/10.2_opt/storage/innobase/os/os0file.cc:5818
#6  0x0000564817ad624c in fil_aio_wait (segment=segment@entry=5) at /test/10.2_opt/storage/innobase/fil/fil0fil.cc:5153
#7  0x00005648179ff818 in io_handler_thread (arg=<optimized out>) at /test/10.2_opt/storage/innobase/srv/srv0start.cc:332
#8  0x00001554d608a609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#9  0x00001554d5c7e293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 7 (Thread 0x1554bdd1d700 (LWP 1585308)):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00001554d619527b in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1
#2  0x0000564817973c4d in LinuxAIOHandler::collect (this=0x1554bdd1cba0) at /test/10.2_opt/storage/innobase/os/os0file.cc:1940
#3  0x0000564817977d33 in LinuxAIOHandler::poll (this=0x1554bdd1cba0, m1=0x1554bdd1cc70, m2=0x1554bdd1cc78, request=0x1554bdd1cc80) at /test/10.2_opt/storage/innobase/os/os0file.cc:2101
#4  0x00005648179791b3 in os_aio_linux_handler (request=0x1554bdd1cc80, m2=0x1554bdd1cc78, m1=0x1554bdd1cc70, global_segment=<optimized out>) at /usr/include/c++/9/bits/stl_vector.h:915
#5  os_aio_handler (segment=segment@entry=4, m1=m1@entry=0x1554bdd1cc70, m2=m2@entry=0x1554bdd1cc78, request=request@entry=0x1554bdd1cc80) at /test/10.2_opt/storage/innobase/os/os0file.cc:5818
#6  0x0000564817ad624c in fil_aio_wait (segment=segment@entry=4) at /test/10.2_opt/storage/innobase/fil/fil0fil.cc:5153
#7  0x00005648179ff818 in io_handler_thread (arg=<optimized out>) at /test/10.2_opt/storage/innobase/srv/srv0start.cc:332
#8  0x00001554d608a609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#9  0x00001554d5c7e293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 6 (Thread 0x1554d4328700 (LWP 1585307)):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00001554d619527b in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1
#2  0x0000564817973c4d in LinuxAIOHandler::collect (this=0x1554d4327ba0) at /test/10.2_opt/storage/innobase/os/os0file.cc:1940
#3  0x0000564817977d33 in LinuxAIOHandler::poll (this=0x1554d4327ba0, m1=0x1554d4327c70, m2=0x1554d4327c78, request=0x1554d4327c80) at /test/10.2_opt/storage/innobase/os/os0file.cc:2101
#4  0x00005648179791b3 in os_aio_linux_handler (request=0x1554d4327c80, m2=0x1554d4327c78, m1=0x1554d4327c70, global_segment=<optimized out>) at /usr/include/c++/9/bits/stl_vector.h:915
#5  os_aio_handler (segment=segment@entry=3, m1=m1@entry=0x1554d4327c70, m2=m2@entry=0x1554d4327c78, request=request@entry=0x1554d4327c80) at /test/10.2_opt/storage/innobase/os/os0file.cc:5818
#6  0x0000564817ad624c in fil_aio_wait (segment=segment@entry=3) at /test/10.2_opt/storage/innobase/fil/fil0fil.cc:5153
#7  0x00005648179ff818 in io_handler_thread (arg=<optimized out>) at /test/10.2_opt/storage/innobase/srv/srv0start.cc:332
#8  0x00001554d608a609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#9  0x00001554d5c7e293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 5 (Thread 0x1554d4529700 (LWP 1585306)):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00001554d619527b in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1
#2  0x0000564817973c4d in LinuxAIOHandler::collect (this=0x1554d4528ba0) at /test/10.2_opt/storage/innobase/os/os0file.cc:1940
#3  0x0000564817977d33 in LinuxAIOHandler::poll (this=0x1554d4528ba0, m1=0x1554d4528c70, m2=0x1554d4528c78, request=0x1554d4528c80) at /test/10.2_opt/storage/innobase/os/os0file.cc:2101
#4  0x00005648179791b3 in os_aio_linux_handler (request=0x1554d4528c80, m2=0x1554d4528c78, m1=0x1554d4528c70, global_segment=<optimized out>) at /usr/include/c++/9/bits/stl_vector.h:915
#5  os_aio_handler (segment=segment@entry=2, m1=m1@entry=0x1554d4528c70, m2=m2@entry=0x1554d4528c78, request=request@entry=0x1554d4528c80) at /test/10.2_opt/storage/innobase/os/os0file.cc:5818
#6  0x0000564817ad624c in fil_aio_wait (segment=segment@entry=2) at /test/10.2_opt/storage/innobase/fil/fil0fil.cc:5153
#7  0x00005648179ff818 in io_handler_thread (arg=<optimized out>) at /test/10.2_opt/storage/innobase/srv/srv0start.cc:332
#8  0x00001554d608a609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#9  0x00001554d5c7e293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 4 (Thread 0x1554d472a700 (LWP 1585305)):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00001554d619527b in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1
#2  0x0000564817973c4d in LinuxAIOHandler::collect (this=0x1554d4729ba0) at /test/10.2_opt/storage/innobase/os/os0file.cc:1940
#3  0x0000564817977d33 in LinuxAIOHandler::poll (this=0x1554d4729ba0, m1=0x1554d4729c70, m2=0x1554d4729c78, request=0x1554d4729c80) at /test/10.2_opt/storage/innobase/os/os0file.cc:2101
#4  0x00005648179791b3 in os_aio_linux_handler (request=0x1554d4729c80, m2=0x1554d4729c78, m1=0x1554d4729c70, global_segment=<optimized out>) at /usr/include/c++/9/bits/stl_vector.h:915
#5  os_aio_handler (segment=segment@entry=1, m1=m1@entry=0x1554d4729c70, m2=m2@entry=0x1554d4729c78, request=request@entry=0x1554d4729c80) at /test/10.2_opt/storage/innobase/os/os0file.cc:5818
#6  0x0000564817ad624c in fil_aio_wait (segment=segment@entry=1) at /test/10.2_opt/storage/innobase/fil/fil0fil.cc:5153
#7  0x00005648179ff818 in io_handler_thread (arg=<optimized out>) at /test/10.2_opt/storage/innobase/srv/srv0start.cc:332
#8  0x00001554d608a609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#9  0x00001554d5c7e293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 3 (Thread 0x1554d492b700 (LWP 1585304)):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00001554d619527b in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1
#2  0x0000564817973c4d in LinuxAIOHandler::collect (this=0x1554d492aba0) at /test/10.2_opt/storage/innobase/os/os0file.cc:1940
#3  0x0000564817977d33 in LinuxAIOHandler::poll (this=0x1554d492aba0, m1=0x1554d492ac70, m2=0x1554d492ac78, request=0x1554d492ac80) at /test/10.2_opt/storage/innobase/os/os0file.cc:2101
#4  0x00005648179791b3 in os_aio_linux_handler (request=0x1554d492ac80, m2=0x1554d492ac78, m1=0x1554d492ac70, global_segment=<optimized out>) at /usr/include/c++/9/bits/stl_vector.h:915
#5  os_aio_handler (segment=segment@entry=0, m1=m1@entry=0x1554d492ac70, m2=m2@entry=0x1554d492ac78, request=request@entry=0x1554d492ac80) at /test/10.2_opt/storage/innobase/os/os0file.cc:5818
#6  0x0000564817ad624c in fil_aio_wait (segment=segment@entry=0) at /test/10.2_opt/storage/innobase/fil/fil0fil.cc:5153
#7  0x00005648179ff818 in io_handler_thread (arg=<optimized out>) at /test/10.2_opt/storage/innobase/srv/srv0start.cc:332
#8  0x00001554d608a609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#9  0x00001554d5c7e293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 2 (Thread 0x1554d5829700 (LWP 1585111)):
#0  futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x1554d5828df0, clockid=<optimized out>, expected=0, futex_word=0x564818c67608 <COND_timer+40>) at ../sysdeps/nptl/futex-internal.h:320
#1  __pthread_cond_wait_common (abstime=0x1554d5828df0, clockid=<optimized out>, mutex=0x564818c67620 <LOCK_timer>, cond=0x564818c675e0 <COND_timer>) at pthread_cond_wait.c:520
#2  __pthread_cond_timedwait (cond=cond@entry=0x564818c675e0 <COND_timer>, mutex=mutex@entry=0x564818c67620 <LOCK_timer>, abstime=abstime@entry=0x1554d5828df0) at pthread_cond_wait.c:656
#3  0x0000564817cc4c9a in inline_mysql_cond_timedwait (that=0x564818c675e0 <COND_timer>, mutex=0x564818c67620 <LOCK_timer>, src_file=0x564817e74f28 "/test/10.2_opt/mysys/thr_timer.c", src_line=292, abstime=0x1554d5828df0) at /test/10.2_opt/include/mysql/psi/mysql_thread.h:1193
#4  timer_handler (arg=<optimized out>) at /test/10.2_opt/mysys/thr_timer.c:292
#5  0x00001554d608a609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6  0x00001554d5c7e293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 1 (Thread 0x1554d59a47c0 (LWP 1585094)):
#0  __lll_lock_wait_private (futex=futex@entry=0x1554d5d47b80 <main_arena>) at ./lowlevellock.c:35
#1  0x00001554d5bf94ab in __GI___libc_malloc (bytes=bytes@entry=504) at malloc.c:3064
#2  0x0000564817cbc7c3 in my_malloc (size=size@entry=496, my_flags=my_flags@entry=16) at /test/10.2_opt/mysys/my_malloc.c:101
#3  0x0000564817d1f665 in mysql_socket_vio_new (mysql_socket=..., type=type@entry=VIO_TYPE_SOCKET, flags=flags@entry=1)
#9  0x00001554d5c7e293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 2 (Thread 0x1554d5829700 (LWP 1585111)):
#0  futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x1554d5828df0, clockid=<optimized out>, expected=0, futex_word=0x564818c67608 <COND_timer+40>) at ../sysdeps/nptl/futex-internal.h:320
#1  __pthread_cond_wait_common (abstime=0x1554d5828df0, clockid=<optimized out>, mutex=0x564818c67620 <LOCK_timer>, cond=0x564818c675e0 <COND_timer>) at pthread_cond_wait.c:520
#2  __pthread_cond_timedwait (cond=cond@entry=0x564818c675e0 <COND_timer>, mutex=mutex@entry=0x564818c67620 <LOCK_timer>, abstime=abstime@entry=0x1554d5828df0) at pthread_cond_wait.c:656
#3  0x0000564817cc4c9a in inline_mysql_cond_timedwait (that=0x564818c675e0 <COND_timer>, mutex=0x564818c67620 <LOCK_timer>, src_file=0x564817e74f28 "/test/10.2_opt/mysys/thr_timer.c", src_line=292, abstime=0x1554d5828df0) at /test/10.2_opt/include/mysql/psi/mysql_thread.h:1193
#4  timer_handler (arg=<optimized out>) at /test/10.2_opt/mysys/thr_timer.c:292
#5  0x00001554d608a609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6  0x00001554d5c7e293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 1 (Thread 0x1554d59a47c0 (LWP 1585094)):
#0  __lll_lock_wait_private (futex=futex@entry=0x1554d5d47b80 <main_arena>) at ./lowlevellock.c:35
#1  0x00001554d5bf94ab in __GI___libc_malloc (bytes=bytes@entry=504) at malloc.c:3064
#2  0x0000564817cbc7c3 in my_malloc (size=size@entry=496, my_flags=my_flags@entry=16) at /test/10.2_opt/mysys/my_malloc.c:101
#3  0x0000564817d1f665 in mysql_socket_vio_new (mysql_socket=..., type=type@entry=VIO_TYPE_SOCKET, flags=flags@entry=1) at /test/10.2_opt/vio/vio.c:253
#4  0x0000564817519925 in handle_connections_sockets () at /test/10.2_opt/sql/mysqld.cc:6912
#5  0x000056481751a701 in mysqld_main (argc=<optimized out>, argv=<optimized out>) at /test/10.2_opt/sql/mysqld.cc:6192
#6  0x00001554d5b830b3 in __libc_start_main (main=0x5648174fb500 <main(int, char**)>, argc=10, argv=0x7ffe3bbb3ec8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffe3bbb3eb8) at ../csu/libc-start.c:308
#7  0x000056481750d39e in _start () at /test/10.2_opt/sql/sql_list.h:196

Comment by Roel Van de Paar [ 2021-09-29 ]

Full report for first testcase (looped twice to ensure crashes).

INSTALL SONAME 'ha_rocksdb';
DROP DATABASE test;
CREATE DATABASE test;
USE test;
CREATE TABLE t1 (a VARBINARY(0), KEY(a)) ENGINE=RocksDB;
INSERT IGNORE INTO t1 VALUES (1);
INSTALL SONAME 'ha_rocksdb';
DROP DATABASE test;
CREATE DATABASE test;
USE test;
CREATE TABLE t1 (a VARBINARY(0), KEY(a)) ENGINE=RocksDB;
INSERT IGNORE INTO t1 VALUES (1);

Leads to:

10.7.0 d552e092c9f3e20da078d1b62b976f629f73d3a4 (Debug)

mysqld: /test/10.7_dbg/storage/rocksdb/rdb_datadic.cc:1447: uint myrocks::Rdb_key_def::pack_record(const TABLE*, uchar*, const uchar*, uchar*, myrocks::Rdb_string_writer*, bool, longlong, uint, uint*, const char*) const: Assertion `is_storage_available(tuple - packed_tuple, 0)' failed.

10.7.0 d552e092c9f3e20da078d1b62b976f629f73d3a4 (Debug)

Core was generated by `/test/MD180921-mariadb-10.7.0-linux-x86_64-dbg/bin/mysqld --no-defaults --core-'.
Program terminated with signal SIGABRT, Aborted.
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
[Current thread is 1 (Thread 0x152e18506700 (LWP 1551416))]
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x0000152e2e395859 in __GI_abort () at abort.c:79
#2  0x0000152e2e395729 in __assert_fail_base (fmt=0x152e2e52b588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x152e09ebf910 "is_storage_available(tuple - packed_tuple, 0)", file=0x152e09ebea70 "/test/10.7_dbg/storage/rocksdb/rdb_datadic.cc", line=1447, function=<optimized out>) at assert.c:92
#3  0x0000152e2e3a6f36 in __GI___assert_fail (assertion=assertion@entry=0x152e09ebf910 "is_storage_available(tuple - packed_tuple, 0)", file=file@entry=0x152e09ebea70 "/test/10.7_dbg/storage/rocksdb/rdb_datadic.cc", line=line@entry=1447, function=function@entry=0x152e09ec0580 "uint myrocks::Rdb_key_def::pack_record(const TABLE*, uchar*, const uchar*, uchar*, myrocks::Rdb_string_writer*, bool, longlong, uint, uint*, const char*) const") at assert.c:101
#4  0x0000152e09b7b626 in myrocks::Rdb_key_def::pack_record (this=this@entry=0x152de02cded0, tbl=tbl@entry=0x152de025e668, pack_buffer=0x152de02d33e8 '\245' <repeats 16 times>, "h4z\025", record=0x152de02653b8 "\376", packed_tuple=0x152de0243b38 "", unpack_info=unpack_info@entry=0x152de0265e28, should_store_row_debug_checksums=false, hidden_pk_id=1, n_key_parts=2, n_null_fields=0x0, ttl_bytes=0x152de026f901 "\240\253R\362U") at /test/10.7_dbg/storage/rocksdb/./rdb_datadic.h:773
#5  0x0000152e09b3eda2 in myrocks::ha_rocksdb::update_write_sk (this=this@entry=0x152de0265840, table_arg=0x152de025e668, kd=@0x152de02cded0: {static CF_FLAGS_TO_IGNORE = 4, m_index_number = 256, m_index_number_storage_form = "\000\000\001", m_cf_handle = 0x152de021df30, m_index_dict_version = 6, m_index_type = 2 '\002', m_kv_format_version = 13, m_is_reverse_cf = false, m_is_per_partition_cf = false, m_name = "a", m_stats = {m_gl_index_id = {cf_id = 0, index_id = 0}, m_data_size = 0, m_rows = 0, m_actual_disk_size = 0, m_entry_deletes = 0, m_entry_single_deletes = 0, m_entry_merges = 0, m_entry_others = 0, m_distinct_keys_per_prefix = std::vector of length 2, capacity 2 = {0, 0}, m_name = ""}, m_index_flags_bitmap = 0, m_total_index_flags_length = 0, m_ttl_rec_offset = 4294967295, m_ttl_duration = 0, m_ttl_column = "", m_pk_key_parts = 1, m_pk_part_no = 0x55f2563aafe8, m_pack_info = 0x152de0013868, m_keyno = 0, m_key_parts = 2, m_ttl_pk_key_part_offset = 4294967295, m_ttl_field_index = 4294967295, m_prefix_extractor = std::shared_ptr<const class rocksdb::SliceTransform> (empty) = {
        get() = 0x0
      }, m_maxlength = 13, m_mutex = {m_mutex = {global = pthread_mutex_t = {
            Type = Error check,
            Status = Not acquired,
            Robust = No,
            Shared = No,
            Protocol = None
          }, mutex = pthread_mutex_t = {
            Type = Adaptive,
            Status = Not acquired,
            Robust = No,
            Shared = No,
            Protocol = None
          }, file = 0x152e09ebea70 "/test/10.7_dbg/storage/rocksdb/rdb_datadic.cc", name = 0x152e09eb8d33 "m_mutex", line = 406, count = 0, create_flags = 0, active_flags = 0, id = 219, thread = 0, locked_mutex = 0x152de026dcd8, used_mutex = 0x152de026dd48, prev = 0x0, next = 0x0}, m_psi = 0x0}}, row_info=@0x152e18504970: {tx = 0x152de027d900, new_data = 0x152de02653b8 "\376", old_data = 0x0, new_pk_slice = {data_ = 0x152de0240888 "", size_ = 12}, old_pk_slice = {data_ = 0x152e09f05fe0 "", size_ = 0}, old_pk_rec = {data_ = 0x152e09f05fe0 "", size_ = 0}, new_pk_unpack_info = 0x0, hidden_pk_id = 1, skip_unique_check = false}, bulk_load_sk=bulk_load_sk@entry=false) at /test/10.7_dbg/storage/rocksdb/ha_rocksdb.cc:10206
#6  0x0000152e09b3f627 in myrocks::ha_rocksdb::update_write_indexes (this=this@entry=0x152de0265840, row_info=@0x152e18504970: {tx = 0x152de027d900, new_data = 0x152de02653b8 "\376", old_data = 0x0, new_pk_slice = {data_ = 0x152de0240888 "", size_ = 12}, old_pk_slice = {data_ = 0x152e09f05fe0 "", size_ = 0}, old_pk_rec = {data_ = 0x152e09f05fe0 "", size_ = 0}, new_pk_unpack_info = 0x0, hidden_pk_id = 1, skip_unique_check = false}, pk_changed=pk_changed@entry=false) at /usr/include/c++/9/bits/shared_ptr_base.h:1309
#7  0x0000152e09b3f866 in myrocks::ha_rocksdb::update_write_row (this=this@entry=0x152de0265840, old_data=old_data@entry=0x0, new_data=new_data@entry=0x152de02653b8 "\376", skip_unique_check=<optimized out>) at /test/10.7_dbg/storage/rocksdb/ha_rocksdb.cc:10390
#8  0x0000152e09b3faa4 in myrocks::ha_rocksdb::write_row (this=0x152de0265840, buf=0x152de02653b8 "\376") at /test/10.7_dbg/storage/rocksdb/ha_rocksdb.cc:9590
#9  0x000055f252cdd7b7 in handler::ha_write_row (this=0x152de0265840, buf=0x152de02653b8 "\376") at /test/10.7_dbg/sql/handler.cc:7516
#10 0x000055f2529823da in write_record (thd=thd@entry=0x152de0000db8, table=table@entry=0x152de025e668, info=info@entry=0x152e18504cc0, sink=sink@entry=0x0) at /test/10.7_dbg/sql/sql_insert.cc:2136
#11 0x000055f25298e028 in mysql_insert (thd=thd@entry=0x152de0000db8, table_list=0x152de0013d58, fields=@0x152de0005fb8: {<base_list> = {<Sql_alloc> = {<No data fields>}, first = 0x55f253fc84a0 <end_of_list>, last = 0x152de0005fb8, elements = 0}, <No data fields>}, values_list=@0x152de0006000: {<base_list> = {<Sql_alloc> = {<No data fields>}, first = 0x152de0014960, last = 0x152de0014960, elements = 1}, <No data fields>}, update_fields=@0x152de0005fe8: {<base_list> = {<Sql_alloc> = {<No data fields>}, first = 0x55f253fc84a0 <end_of_list>, last = 0x152de0005fe8, elements = 0}, <No data fields>}, update_values=@0x152de0005fd0: {<base_list> = {<Sql_alloc> = {<No data fields>}, first = 0x55f253fc84a0 <end_of_list>, last = 0x152de0005fd0, elements = 0}, <No data fields>}, duplic=DUP_ERROR, ignore=true, result=0x0) at /test/10.7_dbg/sql/sql_insert.cc:1124
#12 0x000055f2529d4112 in mysql_execute_command (thd=thd@entry=0x152de0000db8, is_called_from_prepared_stmt=is_called_from_prepared_stmt@entry=false) at /test/10.7_dbg/sql/sql_parse.cc:4563
#13 0x000055f2529bed4b in mysql_parse (thd=thd@entry=0x152de0000db8, rawbuf=<optimized out>, length=<optimized out>, parser_state=parser_state@entry=0x152e18505400) at /test/10.7_dbg/sql/sql_parse.cc:8028
#14 0x000055f2529cd944 in dispatch_command (command=command@entry=COM_QUERY, thd=thd@entry=0x152de0000db8, packet=packet@entry=0x152de000b729 "INSERT IGNORE INTO t1 VALUES (1)", packet_length=packet_length@entry=32, blocking=blocking@entry=true) at /test/10.7_dbg/sql/sql_class.h:1358
#15 0x000055f2529d0d4a in do_command (thd=0x152de0000db8, blocking=blocking@entry=true) at /test/10.7_dbg/sql/sql_parse.cc:1402
#16 0x000055f252b46ed8 in do_handle_one_connection (connect=<optimized out>, connect@entry=0x55f2569ad068, put_in_cache=put_in_cache@entry=true) at /test/10.7_dbg/sql/sql_connect.cc:1418
#17 0x000055f252b474dd in handle_one_connection (arg=arg@entry=0x55f2569ad068) at /test/10.7_dbg/sql/sql_connect.cc:1312
#18 0x000055f252fb047e in pfs_spawn_thread (arg=0x55f2568ac2b8) at /test/10.7_dbg/storage/perfschema/pfs.cc:2201
#19 0x0000152e2e8a4609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#20 0x0000152e2e492293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

10.2.41 (Optimized)

Core was generated by `/test/MD180921-mariadb-10.2.41-linux-x86_64-opt/bin/mysqld --no-defaults --core'.
Program terminated with signal SIGABRT, Aborted.
#0  __pthread_kill (threadid=<optimized out>, signo=signo@entry=6)
    at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
[Current thread is 1 (Thread 0x14aaa089a700 (LWP 3314512))]
(gdb) bt
#0  __pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
#1  0x000055560dca87bf in my_write_core (sig=sig@entry=6) at /test/10.2_opt/mysys/stacktrace.c:382
#2  0x000055560d74d878 in handle_fatal_signal (sig=6) at /test/10.2_opt/sql/signal_handler.cc:355
#3  <signal handler called>
#4  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#5  0x000014aad316e859 in __GI_abort () at abort.c:79
#6  0x000014aad31d93ee in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x14aad3303285 "%s\n") at ../sysdeps/posix/libc_fatal.c:155
#7  0x000014aad31e147c in malloc_printerr (str=str@entry=0x14aad3305600 "free(): invalid next size (fast)") at malloc.c:5347
#8  0x000014aad31e2d2c in _int_free (av=0x14aa74000020, p=0x14aa74210fb0, have_lock=0) at malloc.c:4249
#9  0x000014aaa2f654fe in __gnu_cxx::new_allocator<long>::deallocate (this=0x14aa742b5568, __p=<optimized out>) at /usr/include/c++/9/ext/new_allocator.h:119
#10 std::allocator_traits<std::allocator<long> >::deallocate (__a=@0x14aa742b5568: {<__gnu_cxx::new_allocator<long>> = {<No data fields>}, <No data fields>}, __n=<optimized out>, __p=<optimized out>) at /usr/include/c++/9/bits/alloc_traits.h:470
#11 std::_Vector_base<long, std::allocator<long> >::_M_deallocate (this=0x14aa742b5568, __n=<optimized out>, __p=<optimized out>) at /usr/include/c++/9/bits/stl_vector.h:351
#12 std::_Vector_base<long, std::allocator<long> >::~_Vector_base (this=0x14aa742b5568, __in_chrg=<optimized out>) at /usr/include/c++/9/bits/stl_vector.h:332
#13 std::vector<long, std::allocator<long> >::~vector (this=0x14aa742b5568, __in_chrg=<optimized out>) at /usr/include/c++/9/bits/stl_vector.h:680
#14 myrocks::Rdb_index_stats::~Rdb_index_stats (this=0x14aa742b5528, __in_chrg=<optimized out>) at /test/10.2_opt/storage/rocksdb/././properties_collector.h:47
#15 myrocks::Rdb_key_def::~Rdb_key_def (this=0x14aa742b54f0, __in_chrg=<optimized out>) at /test/10.2_opt/storage/rocksdb/rdb_datadic.cc:381
#16 0x000014aaa2f6e7ad in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x14aa742b54e0) at /usr/include/c++/9/bits/shared_ptr_base.h:148
#17 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x14aa742b54e0) at /usr/include/c++/9/bits/shared_ptr_base.h:148
#18 std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=<optimized out>, __in_chrg=<optimized out>) at /usr/include/c++/9/bits/shared_ptr_base.h:730
#19 std::__shared_ptr<myrocks::Rdb_key_def, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (this=<optimized out>, __in_chrg=<optimized out>) at /usr/include/c++/9/bits/shared_ptr_base.h:1169
#20 std::__shared_ptr<myrocks::Rdb_key_def, (__gnu_cxx::_Lock_policy)2>::operator= (__r=<optimized out>, this=<optimized out>) at /usr/include/c++/9/bits/shared_ptr_base.h:1265
#21 std::shared_ptr<myrocks::Rdb_key_def>::operator= (__r=<optimized out>, this=<optimized out>) at /usr/include/c++/9/bits/shared_ptr.h:335
#22 myrocks::Rdb_tbl_def::~Rdb_tbl_def (this=0x14aa7420edd0, __in_chrg=<optimized out>) at /test/10.2_opt/storage/rocksdb/rdb_datadic.cc:3512
#23 0x000014aaa2f714df in myrocks::Rdb_ddl_manager::remove (this=this@entry=0x14aaa33f2200 <myrocks::ddl_manager>, tbl=tbl@entry=0x14aa7420edd0, batch=batch@entry=0x14aa5001e9b0, lock=lock@entry=true) at /test/10.2_opt/storage/rocksdb/rdb_datadic.cc:4388
#24 0x000014aaa2f1d342 in myrocks::ha_rocksdb::delete_table (this=0x14aa5000fc08, tbl=0x14aa7420edd0) at /test/10.2_opt/storage/rocksdb/ha_rocksdb.cc:11626
#25 0x000055560d755c9d in ha_delete_table (thd=thd@entry=0x14aa50000c48, table_type=<optimized out>, path=path@entry=0x14aaa08973d0 "./test/t1", db=db@entry=0x14aa5000f368 "test", alias=0x14aa5000f380 "t1", generate_warning=generate_warning@entry=false) at /test/10.2_opt/sql/handler.cc:2443
#26 0x000055560d625e27 in mysql_rm_table_no_locks (thd=thd@entry=0x14aa50000c48, tables=tables@entry=0x14aa5000f388, if_exists=if_exists@entry=true, drop_temporary=drop_temporary@entry=false, drop_view=drop_view@entry=true, dont_log_query=dont_log_query@entry=true, dont_free_locks=<optimized out>) at /test/10.2_opt/sql/sql_table.cc:2451
#27 0x000055560d5726d4 in mysql_rm_db_internal (thd=0x14aa50000c48, db=0x14aa5000f368 "test", if_exists=<optimized out>, silent=<optimized out>) at /test/10.2_opt/sql/sql_db.cc:919
#28 0x000055560d59d67c in mysql_execute_command (thd=<optimized out>) at /test/10.2_opt/sql/structs.h:530
#29 0x000055560d5a345a in mysql_parse (thd=thd@entry=0x14aa50000c48, rawbuf=<optimized out>, length=<optimized out>, parser_state=parser_state@entry=0x14aaa0899560, is_com_multi=is_com_multi@entry=false, is_next_command=is_next_command@entry=false) at /test/10.2_opt/sql/sql_parse.cc:7793
#30 0x000055560d5a641d in dispatch_command (command=COM_QUERY, thd=0x14aa50000c48, packet=0x14aa50006fb9 "DROP DATABASE test", packet_length=18, is_com_multi=<optimized out>, is_next_command=<optimized out>) at /test/10.2_opt/sql/sql_class.h:1109
#31 0x000055560d5a785d in do_command (thd=0x14aa50000c48) at /test/10.2_opt/sql/sql_parse.cc:1381
#32 0x000055560d6806c6 in do_handle_one_connection (connect=connect@entry=0x5556104ef4d8) at /test/10.2_opt/sql/sql_connect.cc:1336
#33 0x000055560d68083f in handle_one_connection (arg=0x5556104ef4d8) at /test/10.2_opt/sql/sql_connect.cc:1241
#34 0x000014aad3677609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#35 0x000014aad326b293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

10.3.32 b112c9dfaacbcb7c3548414c6f402114663223dc (Optimized)

Core was generated by `/test/MD180921-mariadb-10.3.32-linux-x86_64-opt/bin/mysqld --no-defaults --core'.
Program terminated with signal SIGABRT, Aborted.
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
[Current thread is 1 (Thread 0x151bdc0a1700 (LWP 1585888))]
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x0000151bdda8e859 in __GI_abort () at abort.c:79
#2  0x0000151bddaf93ee in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x151bddc23285 "%s\n") at ../sysdeps/posix/libc_fatal.c:155
#3  0x0000151bddb0147c in malloc_printerr (str=str@entry=0x151bddc2143a "corrupted size vs. prev_size") at malloc.c:5347
#4  0x0000151bddb01aeb in unlink_chunk (p=p@entry=0x151b80226f50, av=0x151b80000020) at malloc.c:1454
#5  0x0000151bddb0300b in _int_free (av=0x151b80000020, p=0x151b80226e40, have_lock=<optimized out>, have_lock@entry=0) at malloc.c:4342
#6  0x0000151bddb06a0d in __GI___libc_free (mem=0x151b80226e50) at malloc.c:3125
#7  tcache_thread_shutdown () at malloc.c:2964
#8  __malloc_arena_thread_freeres () at arena.c:951
#9  0x0000151bddb0b220 in __libc_thread_freeres () at thread-freeres.c:38
#10 0x0000151bddc6462f in start_thread (arg=<optimized out>) at pthread_create.c:491
#11 0x0000151bddb8b293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

10.4.22 03a10706ecfa5cf0a6252d436e341483f5c0827c (Optimized)

Core was generated by `/test/MD180921-mariadb-10.4.22-linux-x86_64-opt/bin/mysqld --no-defaults --core'.
Program terminated with signal SIGABRT, Aborted.
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
[Current thread is 1 (Thread 0x14fe9406e700 (LWP 3210537))]
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x000014fe9721f859 in __GI_abort () at abort.c:79
#2  0x000014fe9728a3ee in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x14fe973b4285 "%s\n") at ../sysdeps/posix/libc_fatal.c:155
#3  0x000014fe9729247c in malloc_printerr (str=str@entry=0x14fe973b6600 "free(): invalid next size (fast)") at malloc.c:5347
#4  0x000014fe97293d2c in _int_free (av=0x14fe2c000020, p=0x14fe2c20f0d0, have_lock=0) at malloc.c:4249
#5  0x000014fe4b91fe24 in __gnu_cxx::new_allocator<std::shared_ptr<rocksdb::EventListener> >::deallocate (this=0x14fe94068980, __p=<optimized out>) at /usr/include/c++/9/ext/new_allocator.h:119
#6  std::allocator_traits<std::allocator<std::shared_ptr<rocksdb::EventListener> > >::deallocate (__a=@0x14fe94068980: {<__gnu_cxx::new_allocator<std::shared_ptr<rocksdb::EventListener> >> = {<No data fields>}, <No data fields>}, __n=<optimized out>, __p=<optimized out>) at /usr/include/c++/9/bits/alloc_traits.h:470
#7  std::_Vector_base<std::shared_ptr<rocksdb::EventListener>, std::allocator<std::shared_ptr<rocksdb::EventListener> > >::_M_deallocate (this=0x14fe94068980, __n=<optimized out>, __p=<optimized out>) at /usr/include/c++/9/bits/stl_vector.h:351
#8  std::_Vector_base<std::shared_ptr<rocksdb::EventListener>, std::allocator<std::shared_ptr<rocksdb::EventListener> > >::~_Vector_base (this=0x14fe94068980, __in_chrg=<optimized out>) at /usr/include/c++/9/bits/stl_vector.h:332
#9  std::vector<std::shared_ptr<rocksdb::EventListener>, std::allocator<std::shared_ptr<rocksdb::EventListener> > >::~vector (this=0x14fe94068980, __in_chrg=<optimized out>) at /usr/include/c++/9/bits/stl_vector.h:680
#10 rocksdb::DBOptions::~DBOptions (this=0x14fe940687c0, __in_chrg=<optimized out>) at /test/10.4_opt/storage/rocksdb/rocksdb/include/rocksdb/options.h:350
#11 0x000014fe4b97a904 in rocksdb::DBImpl::GetOptions (this=0x14fe2c200e10, column_family=<optimized out>) at /usr/include/c++/9/ext/atomicity.h:96
#12 0x000014fe4b94fcd3 in rocksdb::StackableDB::GetOptions (column_family=<optimized out>, this=<optimized out>) at /test/10.4_opt/storage/rocksdb/rocksdb/include/rocksdb/utilities/stackable_db.h:308
#13 myrocks::Rdb_key_def::setup (this=0x14fe2c217dd0, tbl=tbl@entry=0x14fe94069890, tbl_def=tbl_def@entry=0x14fe2c257960) at /test/10.4_opt/storage/rocksdb/rdb_datadic.cc:582
#14 0x000014fe4b903272 in myrocks::ha_rocksdb::create_key_def (this=<optimized out>, table_arg=0x14fe94069890, i=1, tbl_def_arg=0x14fe2c257960, new_key_def=0x14fe2c2186d8, cf_info=<optimized out>, ttl_duration=0, ttl_column="") at /test/10.4_opt/storage/rocksdb/ha_rocksdb.cc:7515
#15 0x000014fe4b91144c in myrocks::ha_rocksdb::create_key_defs (this=0x14fe2c259fe0, table_arg=0x14fe94069890, tbl_def_arg=0x14fe2c257960, old_table_arg=0x0, old_tbl_def_arg=0x0) at /usr/include/c++/9/array:185
#16 0x000014fe4b9116ac in myrocks::ha_rocksdb::create_table (this=0x14fe2c259fe0, table_name=<optimized out>, table_arg=0x14fe94069890, auto_increment_value=<optimized out>) at /test/10.4_opt/storage/rocksdb/ha_rocksdb.cc:7693
#17 0x000014fe4b911bae in myrocks::ha_rocksdb::create (this=0x14fe2c259fe0, name=0x14fe9406b570 "./test/t1", table_arg=0x14fe94069890, create_info=0x14fe9406b940) at /test/10.4_opt/storage/rocksdb/ha_rocksdb.cc:7839
#18 0x0000556a28a03cf4 in handler::ha_create (this=<optimized out>, name=<optimized out>, form=0x14fe94069890, info_arg=0x14fe9406b940) at /test/10.4_opt/sql/handler.cc:4804
#19 0x0000556a28a0469a in ha_create_table (thd=thd@entry=0x14fe2c000c48, path=path@entry=0x14fe9406b570 "./test/t1", db=0x14fe2c0107e0 "test", table_name=0x14fe2c0100e0 "t1", create_info=create_info@entry=0x14fe9406b940, frm=frm@entry=0x14fe9406b560) at /test/10.4_opt/sql/handler.cc:5269
#20 0x0000556a28873169 in create_table_impl (thd=thd@entry=0x14fe2c000c48, orig_db=@0x14fe2c010130: {str = 0x14fe2c0107e0 "test", length = 4}, orig_table_name=@0x14fe2c010140: {str = 0x14fe2c0100e0 "t1", length = 2}, db=@0x14fe2c010130: {str = 0x14fe2c0107e0 "test", length = 4}, table_name=@0x14fe2c010140: {str = 0x14fe2c0100e0 "t1", length = 2}, path=path@entry=0x14fe9406b570 "./test/t1", options={m_options = DDL_options_st::OPT_NONE}, create_info=0x14fe9406b940, alter_info=0x14fe9406b880, create_table_mode=0, is_trans=0x14fe9406b7f7, key_info=0x14fe9406b558, key_count=0x14fe9406b554, frm=0x14fe9406b560) at /test/10.4_opt/sql/sql_table.cc:5152
#21 0x0000556a2887359f in mysql_create_table_no_lock (thd=thd@entry=0x14fe2c000c48, db=db@entry=0x14fe2c010130, table_name=table_name@entry=0x14fe2c010140, create_info=create_info@entry=0x14fe9406b940, alter_info=0x14fe9406b880, is_trans=is_trans@entry=0x14fe9406b7f7, create_table_mode=0, table_list=0x14fe2c010118) at /test/10.4_opt/sql/sql_table.cc:5236
#22 0x0000556a288737eb in mysql_create_table (thd=thd@entry=0x14fe2c000c48, create_table=create_table@entry=0x14fe2c010118, create_info=create_info@entry=0x14fe9406b940, alter_info=alter_info@entry=0x14fe9406b880) at /test/10.4_opt/sql/sql_table.cc:5331
#23 0x0000556a28874bc4 in Sql_cmd_create_table_like::execute (this=<optimized out>, thd=0x14fe2c000c48) at /test/10.4_opt/sql/sql_table.cc:11593
#24 0x0000556a287cd700 in mysql_execute_command (thd=0x14fe2c000c48) at /test/10.4_opt/sql/sql_parse.cc:6192
#25 0x0000556a287d47e7 in mysql_parse (thd=0x14fe2c000c48, rawbuf=<optimized out>, length=<optimized out>, parser_state=<optimized out>, is_com_multi=<optimized out>, is_next_command=<optimized out>) at /test/10.4_opt/sql/sql_parse.cc:7995
#26 0x0000556a287d6c9d in dispatch_command (command=COM_QUERY, thd=0x14fe2c000c48, packet=<optimized out>, packet_length=<optimized out>, is_com_multi=<optimized out>, is_next_command=<optimized out>) at /test/10.4_opt/sql/sql_class.h:1201
#27 0x0000556a287d8e6e in do_command (thd=0x14fe2c000c48) at /test/10.4_opt/sql/sql_parse.cc:1373
#28 0x0000556a288cd50e in do_handle_one_connection (connect=connect@entry=0x556a2bbf86a8) at /test/10.4_opt/sql/sql_connect.cc:1420
#29 0x0000556a288cd63f in handle_one_connection (arg=0x556a2bbf86a8) at /test/10.4_opt/sql/sql_connect.cc:1316
#30 0x000014fe9772e609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#31 0x000014fe9731c293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

10.5.13 c430aa72abbdccb1ece7f0d65b49a6b48e7c5ba7 (Optimized)

Core was generated by `/test/MD180921-mariadb-10.5.13-linux-x86_64-opt/bin/mysqld --no-defaults --core'.
Program terminated with signal SIGABRT, Aborted.
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
[Current thread is 1 (Thread 0x14561811d700 (LWP 1579833))]
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x0000145619769859 in __GI_abort () at abort.c:79
#2  0x00001456197d43ee in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x1456198fe285 "%s\n") at ../sysdeps/posix/libc_fatal.c:155
#3  0x00001456197dc47c in malloc_printerr (str=str@entry=0x145619900600 "free(): invalid next size (fast)") at malloc.c:5347
#4  0x00001456197ddd2c in _int_free (av=0x1455c8000020, p=0x1455c8079500, have_lock=0) at malloc.c:4249
#5  0x00001455f4f16e69 in __gnu_cxx::new_allocator<int>::deallocate (this=0x145618119318, __p=<optimized out>) at /usr/include/c++/9/ext/new_allocator.h:119
#6  std::allocator_traits<std::allocator<int> >::deallocate (__a=@0x145618119318: {<__gnu_cxx::new_allocator<int>> = {<No data fields>}, <No data fields>}, __n=<optimized out>, __p=<optimized out>) at /usr/include/c++/9/bits/alloc_traits.h:470
#7  std::_Vector_base<int, std::allocator<int> >::_M_deallocate (this=0x145618119318, __n=<optimized out>, __p=<optimized out>) at /usr/include/c++/9/bits/stl_vector.h:351
#8  std::_Vector_base<int, std::allocator<int> >::~_Vector_base (this=0x145618119318, __in_chrg=<optimized out>) at /usr/include/c++/9/bits/stl_vector.h:332
#9  std::vector<int, std::allocator<int> >::~vector (this=0x145618119318, __in_chrg=<optimized out>) at /usr/include/c++/9/bits/stl_vector.h:680
#10 rocksdb::AdvancedColumnFamilyOptions::~AdvancedColumnFamilyOptions (this=0x145618119270, __in_chrg=<optimized out>) at /test/10.5_opt/storage/rocksdb/rocksdb/include/rocksdb/advanced_options.h:154
#11 rocksdb::ColumnFamilyOptions::~ColumnFamilyOptions (this=0x145618119270, __in_chrg=<optimized out>) at /test/10.5_opt/storage/rocksdb/rocksdb/include/rocksdb/options.h:84
#12 0x00001455f4f71130 in rocksdb::DBImpl::GetOptions (this=0x1455c8201940, column_family=<optimized out>) at /usr/include/c++/9/ext/atomicity.h:96
#13 0x00001455f4f465c3 in rocksdb::StackableDB::GetOptions (column_family=<optimized out>, this=<optimized out>) at /test/10.5_opt/storage/rocksdb/rocksdb/include/rocksdb/utilities/stackable_db.h:308
#14 myrocks::Rdb_key_def::setup (this=0x1455c82337f0, tbl=tbl@entry=0x14561811a5a0, tbl_def=tbl_def@entry=0x1455c8234060) at /test/10.5_opt/storage/rocksdb/rdb_datadic.cc:583
#15 0x00001455f4ef9842 in myrocks::ha_rocksdb::create_key_def (this=<optimized out>, table_arg=0x14561811a5a0, i=0, tbl_def_arg=0x1455c8234060, new_key_def=0x1455c82b8e08, cf_info=<optimized out>, ttl_duration=0, ttl_column="") at /test/10.5_opt/storage/rocksdb/ha_rocksdb.cc:7534
#16 0x00001455f4f07adc in myrocks::ha_rocksdb::create_key_defs (this=0x1455c82140f0, table_arg=0x14561811a5a0, tbl_def_arg=0x1455c8234060, old_table_arg=0x0, old_tbl_def_arg=0x0) at /usr/include/c++/9/array:185
#17 0x00001455f4f07d3c in myrocks::ha_rocksdb::create_table (this=0x1455c82140f0, table_name=<optimized out>, table_arg=0x14561811a5a0, auto_increment_value=<optimized out>) at /test/10.5_opt/storage/rocksdb/ha_rocksdb.cc:7712
#18 0x00001455f4f0823e in myrocks::ha_rocksdb::create (this=0x1455c82140f0, name=0x14561811b8a0 "./test/t1", table_arg=0x14561811a5a0, create_info=0x14561811bc70) at /test/10.5_opt/storage/rocksdb/ha_rocksdb.cc:7858
#19 0x0000564aca4f6654 in handler::ha_create (this=<optimized out>, name=<optimized out>, form=0x14561811a5a0, info_arg=0x14561811bc70) at /test/10.5_opt/sql/handler.cc:5110
#20 0x0000564aca4f6fea in ha_create_table (thd=thd@entry=0x1455c8000c58, path=path@entry=0x14561811b8a0 "./test/t1", db=0x1455c8010c00 "test", table_name=0x1455c80104f0 "t1", create_info=create_info@entry=0x14561811bc70, frm=frm@entry=0x14561811b890) at /test/10.5_opt/sql/handler.cc:5575
#21 0x0000564aca37d633 in create_table_impl (thd=thd@entry=0x1455c8000c58, orig_db=@0x1455c8010540: {str = 0x1455c8010c00 "test", length = 4}, orig_table_name=@0x1455c8010550: {str = 0x1455c80104f0 "t1", length = 2}, db=@0x1455c8010540: {str = 0x1455c8010c00 "test", length = 4}, table_name=@0x1455c8010550: {str = 0x1455c80104f0 "t1", length = 2}, path=path@entry=0x14561811b8a0 "./test/t1", options={m_options = DDL_options_st::OPT_NONE}, create_info=0x14561811bc70, alter_info=0x14561811bba0, create_table_mode=0, is_trans=0x14561811bb17, key_info=0x14561811b888, key_count=0x14561811b884, frm=0x14561811b890) at /test/10.5_opt/sql/sql_table.cc:5392
#22 0x0000564aca37df0f in mysql_create_table_no_lock (thd=thd@entry=0x1455c8000c58, db=db@entry=0x1455c8010540, table_name=table_name@entry=0x1455c8010550, create_info=create_info@entry=0x14561811bc70, alter_info=alter_info@entry=0x14561811bba0, is_trans=is_trans@entry=0x14561811bb17, create_table_mode=0, table_list=0x1455c8010528) at /test/10.5_opt/sql/sql_table.cc:5476
#23 0x0000564aca37e1f2 in mysql_create_table (thd=thd@entry=0x1455c8000c58, create_table=create_table@entry=0x1455c8010528, create_info=create_info@entry=0x14561811bc70, alter_info=alter_info@entry=0x14561811bba0) at /test/10.5_opt/sql/sql_table.cc:5580
#24 0x0000564aca37f5f9 in Sql_cmd_create_table_like::execute (this=<optimized out>, thd=0x1455c8000c58) at /test/10.5_opt/sql/sql_table.cc:12199
#25 0x0000564aca2dac4e in mysql_execute_command (thd=0x1455c8000c58) at /test/10.5_opt/sql/sql_parse.cc:6056
#26 0x0000564aca2ca1e3 in mysql_parse (thd=0x1455c8000c58, rawbuf=<optimized out>, length=<optimized out>, parser_state=<optimized out>, is_com_multi=<optimized out>, is_next_command=<optimized out>) at /test/10.5_opt/sql/sql_parse.cc:8100
#27 0x0000564aca2d69c5 in dispatch_command (command=COM_QUERY, thd=0x1455c8000c58, packet=<optimized out>, packet_length=<optimized out>, is_com_multi=<optimized out>, is_next_command=<optimized out>) at /test/10.5_opt/sql/sql_class.h:1290
#28 0x0000564aca2d8f32 in do_command (thd=0x1455c8000c58) at /test/10.5_opt/sql/sql_parse.cc:1370
#29 0x0000564aca3df3d1 in do_handle_one_connection (connect=<optimized out>, connect@entry=0x564acdea89a8, put_in_cache=put_in_cache@entry=true) at /test/10.5_opt/sql/sql_connect.cc:1418
#30 0x0000564aca3df84d in handle_one_connection (arg=arg@entry=0x564acdea89a8) at /test/10.5_opt/sql/sql_connect.cc:1312
#31 0x0000564aca76f1d9 in pfs_spawn_thread (arg=0x564acde17128) at /test/10.5_opt/storage/perfschema/pfs.cc:2201
#32 0x0000145619c78609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#33 0x0000145619866293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

10.6.5 1e9c922fa726b22f4522f2a4de0fcb6595404086 (Optimized)

Core was generated by `/test/MD180921-mariadb-10.6.5-linux-x86_64-opt/bin/mysqld --no-defaults --core-'.
Program terminated with signal SIGABRT, Aborted.
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
[Current thread is 1 (Thread 0x1542ebfa3800 (LWP 1581728))]
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00001542ec180859 in __GI_abort () at abort.c:79
#2  0x00001542ec1eb3ee in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x1542ec315285 "%s\n") at ../sysdeps/posix/libc_fatal.c:155
#3  0x00001542ec1f347c in malloc_printerr (str=str@entry=0x1542ec317600 "free(): invalid next size (fast)") at malloc.c:5347
#4  0x00001542ec1f4d2c in _int_free (av=0x154284000020, p=0x1542842b9b00, have_lock=0) at malloc.c:4249
#5  0x00001542b6ef4793 in myrocks::ha_rocksdb::free_key_buffers (this=this@entry=0x1542842153e0) at /test/10.6_opt/storage/rocksdb/ha_rocksdb.cc:6648
#6  0x00001542b6ef4938 in myrocks::ha_rocksdb::close (this=0x1542842153e0) at /test/10.6_opt/storage/rocksdb/ha_rocksdb.cc:6856
#7  0x000056159a916cb9 in closefrm (table=table@entry=0x15428425aa88) at /test/10.6_opt/sql/table.cc:4431
#8  0x000056159a9f1d5d in intern_close_table (table=0x15428425aa88) at /test/10.6_opt/sql/table_cache.cc:220
#9  tc_purge () at /test/10.6_opt/sql/table_cache.cc:312
#10 0x000056159a7d2812 in purge_tables () at /test/10.6_opt/sql/sql_base.cc:327
#11 0x000056159a9f08b1 in tdc_start_shutdown () at /test/10.6_opt/sql/table_cache.cc:634
#12 0x000056159a76683d in clean_up (print_message=print_message@entry=true) at /test/10.6_opt/sql/mysqld.cc:1919
#13 0x000056159a771692 in clean_up (print_message=true) at /test/10.6_opt/sql/mysqld.cc:5844
#14 mysqld_main (argc=<optimized out>, argv=<optimized out>) at /test/10.6_opt/sql/mysqld.cc:5844
#15 0x00001542ec1820b3 in __libc_start_main (main=0x56159a735230 <main(int, char**)>, argc=10, argv=0x7fff0b37bfc8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fff0b37bfb8) at ../csu/libc-start.c:308
#16 0x000056159a76541e in _start () at /test/10.6_opt/sql/mysqld.cc:4475

Debug stacks are always the same (ref a previous comment above).

Bug confirmed present in:
MariaDB: 10.2.41 (dbg), 10.2.41 (opt), 10.3.32 (dbg), 10.3.32 (opt), 10.4.22 (dbg), 10.4.22 (opt), 10.5.13 (dbg), 10.5.13 (opt), 10.6.5 (dbg), 10.6.5 (opt), 10.7.0 (dbg), 10.7.0 (opt)

Bug (or feature/syntax) confirmed not present in:
MySQL: 5.5.62 (dbg), 5.5.62 (opt), 5.6.51 (dbg), 5.6.51 (opt), 5.7.35 (dbg), 5.7.35 (opt), 8.0.26 (dbg), 8.0.26 (opt)

Comment by Roel Van de Paar [ 2021-09-29 ]

Summary of UniqueID's seen across versions

SIGABRT|__libc_message|malloc_printerr|_int_free|__gnu_cxx::new_allocator<int>::deallocate
SIGABRT|__libc_message|malloc_printerr|_int_free|__gnu_cxx::new_allocator<long>::deallocate
SIGABRT|__libc_message|malloc_printerr|_int_free|__gnu_cxx::new_allocator<std::shared_ptr<rocksdb::EventListener> >::deallocate
SIGABRT|__libc_message|malloc_printerr|_int_free|myrocks::ha_rocksdb::free_key_buffers
SIGABRT|__libc_message|malloc_printerr|munmap_chunk|cleanup_variables
SIGABRT|__libc_message|malloc_printerr|munmap_chunk|free_root
SIGABRT|__libc_message|malloc_printerr|munmap_chunk|myrocks::ha_rocksdb::free_key_buffers
SIGABRT|__libc_message|malloc_printerr|munmap_chunk|std::__shared_count<shared_ptr_base.h
SIGABRT|__libc_message|malloc_printerr|unlink_chunk|_int_free
is_storage_available(tuple - packed_tuple, 0)|SIGABRT|myrocks::Rdb_key_def::pack_record|myrocks::ha_rocksdb::update_write_sk|myrocks::ha_rocksdb::update_write_indexes|myrocks::ha_rocksdb::update_write_row
is_storage_available(tuple - packed_tuple, pack_info->m_max_image_len)|SIGABRT|myrocks::Rdb_key_def::pack_field|myrocks::Rdb_key_def::pack_record|myrocks::ha_rocksdb::update_write_sk|myrocks::ha_rocksdb::update_write_indexes

Comment by Roel Van de Paar [ 2021-09-29 ]

This bug significantly affects testing as the optimized stacks for these issues are broad filters. A fix would be appreciated.

Comment by Roel Van de Paar [ 2021-09-29 ]

Another related assertion/stack with this testcase variation:

SET sql_mode='',@@collation_server=2;
INSTALL plugin RocksDB SONAME 'ha_rocksdb';
DROP DATABASE test;
CREATE DATABASE test;
USE test;
CREATE TABLE t (pk INT PRIMARY KEY, c VARCHAR(100), KEY(c (10))) ENGINE=RocksDB;
INSERT INTO t VALUES (19, 'aøaaaøaaaaaaaaøaaaa');
SELECT SLEEP (3);

Leads to:

10.7.0 d552e092c9f3e20da078d1b62b976f629f73d3a4 (Debug)

mysqld: /test/10.7_dbg/storage/rocksdb/rdb_datadic.cc:1231: uchar* myrocks::Rdb_key_def::pack_field(Field*, myrocks::Rdb_field_packing*, uchar*, uchar*, uchar*, myrocks::Rdb_string_writer*, uint*) const: Assertion `is_storage_available(tuple - packed_tuple, pack_info->m_max_image_len)' failed.

10.7.0 d552e092c9f3e20da078d1b62b976f629f73d3a4 (Debug)

Core was generated by `/test/MD180921-mariadb-10.7.0-linux-x86_64-dbg/bin/mysqld --no-defaults --core-'.
Program terminated with signal SIGABRT, Aborted.
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
[Current thread is 1 (Thread 0x14b130c59700 (LWP 4057769))]
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x000014b145b0c859 in __GI_abort () at abort.c:79
#2  0x000014b145b0c729 in __assert_fail_base (fmt=0x14b145ca2588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x14b1216cb2e0 "is_storage_available(tuple - packed_tuple, pack_info->m_max_image_len)", file=0x14b1216caa70 "/test/10.7_dbg/storage/rocksdb/rdb_datadic.cc", line=1231, function=<optimized out>) at assert.c:92
#3  0x000014b145b1df36 in __GI___assert_fail (assertion=assertion@entry=0x14b1216cb2e0 "is_storage_available(tuple - packed_tuple, pack_info->m_max_image_len)", file=file@entry=0x14b1216caa70 "/test/10.7_dbg/storage/rocksdb/rdb_datadic.cc", line=line@entry=1231, function=function@entry=0x14b1216cb250 "uchar* myrocks::Rdb_key_def::pack_field(Field*, myrocks::Rdb_field_packing*, uchar*, uchar*, uchar*, myrocks::Rdb_string_writer*, uint*) const") at assert.c:101
#4  0x000014b121381fab in myrocks::Rdb_key_def::pack_field (this=this@entry=0x14b0f82c4330, field=field@entry=0x14b0f8259aa0, pack_info=0x14b0f80138e0, tuple=<optimized out>, packed_tuple=packed_tuple@entry=0x14b0f826c738 "", pack_buffer=pack_buffer@entry=0x14b0f82694c8 '\003' <repeats 16 times>, "\001", '\003' <repeats 16 times>, "\001", '\003' <repeats 16 times>, "\001\003\227\003\245\245\245\245\245\245\245\245\245\245h4z\025", '\217' <repeats 12 times>, <incomplete sequence \345>, unpack_info=0x14b0f825a3c8, n_null_fields=0x0) at /test/10.7_dbg/storage/rocksdb/./rdb_datadic.h:773
#5  0x000014b121387099 in myrocks::Rdb_key_def::pack_record (this=this@entry=0x14b0f82c4330, tbl=tbl@entry=0x14b0f8268998, pack_buffer=0x14b0f82694c8 '\003' <repeats 16 times>, "\001", '\003' <repeats 16 times>, "\001", '\003' <repeats 16 times>, "\001\003\227\003\245\245\245\245\245\245\245\245\245\245h4z\025", '\217' <repeats 12 times>, <incomplete sequence \345>, record=0x14b0f8259958 "\376\023", packed_tuple=0x14b0f826c738 "", unpack_info=unpack_info@entry=0x14b0f825a3c8, should_store_row_debug_checksums=false, hidden_pk_id=0, n_key_parts=2, n_null_fields=0x0, ttl_bytes=0x14b0f8232541 "\240\340'\234U") at /test/10.7_dbg/storage/rocksdb/rdb_datadic.cc:1382
#6  0x000014b12134ada2 in myrocks::ha_rocksdb::update_write_sk (this=this@entry=0x14b0f8259de0, table_arg=0x14b0f8268998, kd=@0x14b0f82c4330: {static CF_FLAGS_TO_IGNORE = 4, m_index_number = 257, m_index_number_storage_form = "\000\000\001\001", m_cf_handle = 0x14b0f8212b70, m_index_dict_version = 6, m_index_type = 2 '\002', m_kv_format_version = 13, m_is_reverse_cf = false, m_is_per_partition_cf = false, m_name = "c", m_stats = {m_gl_index_id = {cf_id = 0, index_id = 0}, m_data_size = 0, m_rows = 0, m_actual_disk_size = 0, m_entry_deletes = 0, m_entry_single_deletes = 0, m_entry_merges = 0, m_entry_others = 0, m_distinct_keys_per_prefix = std::vector of length 2, capacity 2 = {0, 0}, m_name = ""}, m_index_flags_bitmap = 0, m_total_index_flags_length = 0, m_ttl_rec_offset = 4294967295, m_ttl_duration = 0, m_ttl_column = "", m_pk_key_parts = 1, m_pk_part_no = 0x14b0f825c4b8, m_pack_info = 0x14b0f8013868, m_keyno = 1, m_key_parts = 2, m_ttl_pk_key_part_offset = 4294967295, m_ttl_field_index = 4294967295, m_prefix_extractor = std::shared_ptr<const class rocksdb::SliceTransform> (empty) = {
        get() = 0x0
      }, m_maxlength = 63, m_mutex = {m_mutex = {global = pthread_mutex_t = {
            Type = Error check,
            Status = Not acquired,
            Robust = No,
            Shared = No,
            Protocol = None
          }, mutex = pthread_mutex_t = {
            Type = Adaptive,
            Status = Not acquired,
            Robust = No,
            Shared = No,
            Protocol = None
          }, file = 0x14b1216caa70 "/test/10.7_dbg/storage/rocksdb/rdb_datadic.cc", name = 0x14b1216c4d33 "m_mutex", line = 406, count = 0, create_flags = 0, active_flags = 0, id = 220, thread = 0, locked_mutex = 0x14b0f826c5b8, used_mutex = 0x14b0f826c628, prev = 0x0, next = 0x0}, m_psi = 0x0}}, row_info=@0x14b130c57970: {tx = 0x14b0f82724e0, new_data = 0x14b0f8259958 "\376\023", old_data = 0x0, new_pk_slice = {data_ = 0x14b0f8237628 "", size_ = 8}, old_pk_slice = {data_ = 0x14b121711fe0 "", size_ = 0}, old_pk_rec = {data_ = 0x14b121711fe0 "", size_ = 0}, new_pk_unpack_info = 0x14b0f825a3e0, hidden_pk_id = 0, skip_unique_check = false}, bulk_load_sk=bulk_load_sk@entry=false) at /test/10.7_dbg/storage/rocksdb/ha_rocksdb.cc:10206
#7  0x000014b12134b627 in myrocks::ha_rocksdb::update_write_indexes (this=this@entry=0x14b0f8259de0, row_info=@0x14b130c57970: {tx = 0x14b0f82724e0, new_data = 0x14b0f8259958 "\376\023", old_data = 0x0, new_pk_slice = {data_ = 0x14b0f8237628 "", size_ = 8}, old_pk_slice = {data_ = 0x14b121711fe0 "", size_ = 0}, old_pk_rec = {data_ = 0x14b121711fe0 "", size_ = 0}, new_pk_unpack_info = 0x14b0f825a3e0, hidden_pk_id = 0, skip_unique_check = false}, pk_changed=pk_changed@entry=false) at /usr/include/c++/9/bits/shared_ptr_base.h:1309
#8  0x000014b12134b866 in myrocks::ha_rocksdb::update_write_row (this=this@entry=0x14b0f8259de0, old_data=old_data@entry=0x0, new_data=new_data@entry=0x14b0f8259958 "\376\023", skip_unique_check=<optimized out>) at /test/10.7_dbg/storage/rocksdb/ha_rocksdb.cc:10390
#9  0x000014b12134baa4 in myrocks::ha_rocksdb::write_row (this=0x14b0f8259de0, buf=0x14b0f8259958 "\376\023") at /test/10.7_dbg/storage/rocksdb/ha_rocksdb.cc:9590
#10 0x0000559c2802d7b7 in handler::ha_write_row (this=0x14b0f8259de0, buf=0x14b0f8259958 "\376\023") at /test/10.7_dbg/sql/handler.cc:7516
#11 0x0000559c27cd23da in write_record (thd=thd@entry=0x14b0f8000db8, table=table@entry=0x14b0f8268998, info=info@entry=0x14b130c57cc0, sink=sink@entry=0x0) at /test/10.7_dbg/sql/sql_insert.cc:2136
#12 0x0000559c27cde028 in mysql_insert (thd=thd@entry=0x14b0f8000db8, table_list=0x14b0f8013d80, fields=@0x14b0f8005fb8: {<base_list> = {<Sql_alloc> = {<No data fields>}, first = 0x559c293184a0 <end_of_list>, last = 0x14b0f8005fb8, elements = 0}, <No data fields>}, values_list=@0x14b0f8006000: {<base_list> = {<Sql_alloc> = {<No data fields>}, first = 0x14b0f8014a38, last = 0x14b0f8014a38, elements = 1}, <No data fields>}, update_fields=@0x14b0f8005fe8: {<base_list> = {<Sql_alloc> = {<No data fields>}, first = 0x559c293184a0 <end_of_list>, last = 0x14b0f8005fe8, elements = 0}, <No data fields>}, update_values=@0x14b0f8005fd0: {<base_list> = {<Sql_alloc> = {<No data fields>}, first = 0x559c293184a0 <end_of_list>, last = 0x14b0f8005fd0, elements = 0}, <No data fields>}, duplic=DUP_ERROR, ignore=false, result=0x0) at /test/10.7_dbg/sql/sql_insert.cc:1124
#13 0x0000559c27d24112 in mysql_execute_command (thd=thd@entry=0x14b0f8000db8, is_called_from_prepared_stmt=is_called_from_prepared_stmt@entry=false) at /test/10.7_dbg/sql/sql_parse.cc:4563
#14 0x0000559c27d0ed4b in mysql_parse (thd=thd@entry=0x14b0f8000db8, rawbuf=<optimized out>, length=<optimized out>, parser_state=parser_state@entry=0x14b130c58400) at /test/10.7_dbg/sql/sql_parse.cc:8028
#15 0x0000559c27d1d944 in dispatch_command (command=command@entry=COM_QUERY, thd=thd@entry=0x14b0f8000db8, packet=packet@entry=0x14b0f800b729 "INSERT INTO t VALUES (19, 'aøaaaøaaaaaaaaøaaaa')", packet_length=packet_length@entry=51, blocking=blocking@entry=true) at /test/10.7_dbg/sql/sql_class.h:1358
#16 0x0000559c27d20d4a in do_command (thd=0x14b0f8000db8, blocking=blocking@entry=true) at /test/10.7_dbg/sql/sql_parse.cc:1402
#17 0x0000559c27e96ed8 in do_handle_one_connection (connect=<optimized out>, connect@entry=0x559c2ae86078, put_in_cache=put_in_cache@entry=true) at /test/10.7_dbg/sql/sql_connect.cc:1418
#18 0x0000559c27e974dd in handle_one_connection (arg=arg@entry=0x559c2ae86078) at /test/10.7_dbg/sql/sql_connect.cc:1312
#19 0x0000559c2830047e in pfs_spawn_thread (arg=0x559c2ad852c8) at /test/10.7_dbg/storage/perfschema/pfs.cc:2201
#20 0x000014b14601b609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#21 0x000014b145c09293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Bug confirmed present in:
MariaDB: 10.2.41 (dbg), 10.2.41 (opt), 10.3.32 (dbg), 10.3.32 (opt), 10.4.22 (dbg), 10.4.22 (opt), 10.5.13 (dbg), 10.5.13 (opt), 10.6.5 (dbg), 10.6.5 (opt), 10.7.0 (dbg), 10.7.0 (opt)

Bug (or feature/syntax) confirmed not present in:
MySQL: 5.5.62 (dbg), 5.5.62 (opt), 5.6.51 (dbg), 5.6.51 (opt), 5.7.35 (dbg), 5.7.35 (opt), 8.0.26 (dbg), 8.0.26 (opt)

Comment by Daniel Black [ 2023-04-16 ]

Given how deep in the stack this is, I'm linking to a submodule update - MDEV-30610 and will reevaluate after an update.

Generated at Thu Feb 08 09:09:53 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.