[MDEV-16644] rocksdb_rpl.rpl_binlog_xid_count failed in buildbot, server crashed Created: 2018-06-30  Updated: 2023-04-27

Status: Open
Project: MariaDB Server
Component/s: Replication, Storage Engine - RocksDB, Tests
Affects Version/s: 10.2, 10.3, 10.4, 10.5, 10.6, 10.7
Fix Version/s: 10.4, 10.5, 10.6

Type: Bug Priority: Major
Reporter: Elena Stepanova Assignee: Andrei Elkin
Resolution: Unresolved Votes: 0
Labels: None


 Description   

http://buildbot.askmonty.org/buildbot/builders/kvm-bintar-trusty-amd64/builds/6558

10.3 88aaf590ac9fa8c8030a5831cebd867a

rocksdb_rpl.rpl_binlog_xid_count 'row'   w4 [ fail ]
        Test ended at 2018-06-25 11:28:07
 
CURRENT_TEST: rocksdb_rpl.rpl_binlog_xid_count
mysqltest: At line 17: query 'INSERT INTO t SET a=repeat('a', $size)' failed: 2013: Lost connection to MySQL server during query
 
The result from queries just before the failure was:
< snip >
INSERT INTO t SET a=repeat('a', 4096/2);
INSERT INTO t SET a=repeat('a', 4096);
INSERT INTO t SET a=repeat('a', 4096/2);
INSERT INTO t SET a=repeat('a', 4096);
INSERT INTO t SET a=repeat('a', 4096/2);
INSERT INTO t SET a=repeat('a', 4096);
INSERT INTO t SET a=repeat('a', 4096/2);
INSERT INTO t SET a=repeat('a', 4096);
INSERT INTO t SET a=repeat('a', 4096/2);
INSERT INTO t SET a=repeat('a', 4096);
INSERT INTO t SET a=repeat('a', 4096/2);
INSERT INTO t SET a=repeat('a', 4096);
INSERT INTO t SET a=repeat('a', 4096/2);
INSERT INTO t SET a=repeat('a', 4096);
INSERT INTO t SET a=repeat('a', 4096/2);
INSERT INTO t SET a=repeat('a', 4096);
INSERT INTO t SET a=repeat('a', 4096/2);
INSERT INTO t SET a=repeat('a', 4096);
INSERT INTO t SET a=repeat('a', 4096/2);
INSERT INTO t SET a=repeat('a', 4096);
 
More results from queries before failure can be found in /usr/local/mariadb-10.3.8-linux-x86_64/mysql-test/var/4/log/rpl_binlog_xid_count.log
 
 
Server [mysqld.1 - pid: 5657, winpid: 5657, exit: 256] failed during test run
Server log from this test:
----------SERVER LOG START-----------
2018-06-25 11:28:01 0 [Note] /usr/local/mariadb-10.3.8-linux-x86_64/bin/mysqld (mysqld 10.3.8-MariaDB-log) starting as process 5658 ...
2018-06-25 11:28:01 0 [Warning] Could not increase number of max_open_files to more than 1024 (request: 4182)
2018-06-25 11:28:01 0 [Note] RocksDB: Got ENOENT when listing column families
2018-06-25 11:28:01 0 [Note] RocksDB:   assuming that we're creating a new database
2018-06-25 11:28:01 0 [Note] RocksDB: Column Families at start:
2018-06-25 11:28:01 0 [Note]   cf=default
2018-06-25 11:28:01 0 [Note]     write_buffer_size=67108864
2018-06-25 11:28:01 0 [Note]     target_file_size_base=67108864
2018-06-25 11:28:01 0 [Note] RocksDB: creating a column family __system__
2018-06-25 11:28:01 0 [Note]     write_buffer_size=67108864
2018-06-25 11:28:01 0 [Note]     target_file_size_base=67108864
2018-06-25 11:28:01 0 [Note] RocksDB: Table_store: loaded DDL data for 0 tables
2018-06-25 11:28:01 0 [Note] RocksDB: global statistics using get_sched_indexer_t indexer
2018-06-25 11:28:01 0 [Note] MyRocks storage engine plugin has been successfully initialized.
2018-06-25 11:28:01 0 [Note] Plugin 'partition' is disabled.
2018-06-25 11:28:01 0 [Note] InnoDB: Using Linux native AIO
2018-06-25 11:28:01 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2018-06-25 11:28:01 0 [Note] InnoDB: Uses event mutexes
2018-06-25 11:28:01 0 [Note] InnoDB: Compressed tables use zlib 1.2.3
2018-06-25 11:28:01 0 [Note] InnoDB: Number of pools: 1
2018-06-25 11:28:01 0 [Note] InnoDB: Using generic crc32 instructions
2018-06-25 11:28:01 0 [Note] InnoDB: Initializing buffer pool, total size = 8M, instances = 1, chunk size = 8M
2018-06-25 11:28:01 0 [Note] InnoDB: Completed initialization of buffer pool
2018-06-25 11:28:01 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2018-06-25 11:28:02 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
2018-06-25 11:28:02 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2018-06-25 11:28:02 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2018-06-25 11:28:02 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2018-06-25 11:28:02 0 [Note] InnoDB: Waiting for purge to start
2018-06-25 11:28:02 0 [Note] InnoDB: 10.3.8 started; log sequence number 1642313; transaction id 34
2018-06-25 11:28:02 0 [Note] InnoDB: Loading buffer pool(s) from /run/shm/var_auto_7pQX/4/mysqld.1/data/ib_buffer_pool
2018-06-25 11:28:02 0 [Note] InnoDB: Buffer pool(s) load completed at 180625 11:28:02
2018-06-25 11:28:02 0 [Note] Plugin 'SEQUENCE' is disabled.
2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_SYS_DATAFILES' is disabled.
2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_SYS_TABLESTATS' is disabled.
2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_LOCKS' is disabled.
2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_TABLESPACES_SCRUBBING' is disabled.
2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_CMPMEM' is disabled.
2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_CMP_PER_INDEX' is disabled.
2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_CMP' is disabled.
2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_TABLESPACES_ENCRYPTION' is disabled.
2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_CMP_RESET' is disabled.
2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_SYS_INDEXES' is disabled.
2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_BUFFER_PAGE_LRU' is disabled.
2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_CMPMEM_RESET' is disabled.
2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_SYS_COLUMNS' is disabled.
2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_FT_INDEX_TABLE' is disabled.
2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_CMP_PER_INDEX_RESET' is disabled.
2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_FT_INDEX_CACHE' is disabled.
2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_SYS_FOREIGN_COLS' is disabled.
2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_FT_BEING_DELETED' is disabled.
2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_TRX' is disabled.
2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_SYS_FOREIGN' is disabled.
2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_SYS_TABLES' is disabled.
2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_FT_DEFAULT_STOPWORD' is disabled.
2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_FT_CONFIG' is disabled.
2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_SYS_TABLESPACES' is disabled.
2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_MUTEXES' is disabled.
2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_SYS_SEMAPHORE_WAITS' is disabled.
2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_SYS_FIELDS' is disabled.
2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_METRICS' is disabled.
2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_FT_DELETED' is disabled.
2018-06-25 11:28:02 0 [Note] Plugin 'user_variables' is disabled.
2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_BUFFER_POOL_STATS' is disabled.
2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_LOCK_WAITS' is disabled.
2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_SYS_VIRTUAL' is disabled.
2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_BUFFER_PAGE' is disabled.
2018-06-25 11:28:02 0 [Note] Plugin 'FEEDBACK' is disabled.
2018-06-25 11:28:02 0 [Warning] /usr/local/mariadb-10.3.8-linux-x86_64/bin/mysqld: unknown variable 'loose-feedback-debug-startup-interval=20'
2018-06-25 11:28:02 0 [Warning] /usr/local/mariadb-10.3.8-linux-x86_64/bin/mysqld: unknown variable 'loose-feedback-debug-first-interval=60'
2018-06-25 11:28:02 0 [Warning] /usr/local/mariadb-10.3.8-linux-x86_64/bin/mysqld: unknown variable 'loose-feedback-debug-interval=60'
2018-06-25 11:28:02 0 [Warning] /usr/local/mariadb-10.3.8-linux-x86_64/bin/mysqld: unknown option '--loose-pam-debug'
2018-06-25 11:28:02 0 [Warning] /usr/local/mariadb-10.3.8-linux-x86_64/bin/mysqld: unknown option '--loose-enable-rocksdb_lock_info'
2018-06-25 11:28:02 0 [Warning] /usr/local/mariadb-10.3.8-linux-x86_64/bin/mysqld: unknown variable 'loose-debug-sync-timeout=300'
2018-06-25 11:28:02 0 [Note] Server socket created on IP: '127.0.0.1'.
2018-06-25 11:28:02 0 [Note] /usr/local/mariadb-10.3.8-linux-x86_64/bin/mysqld: ready for connections.
Version: '10.3.8-MariaDB-log'  socket: '/usr/local/mariadb-10.3.8-linux-x86_64/mysql-test/var/tmp/4/mysqld.1.sock'  port: 16060  MariaDB Server
2018-06-25 11:28:02 10 [Note] RocksDB: Manual memtable flush.
180625 11:28:03 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
Server version: 10.3.8-MariaDB-log
key_buffer_size=1048576
read_buffer_size=131072
max_used_connections=1
max_threads=153
thread_count=8
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 63163 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x7f05d80009a8
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f0626423e20 thread_stack 0x49000
/usr/local/mariadb-10.3.8-linux-x86_64/bin/mysqld(my_print_stacktrace+0x2e)[0x7f063e937e6e]
/usr/local/mariadb-10.3.8-linux-x86_64/bin/mysqld(handle_fatal_signal+0x357)[0x7f063e3d6d97]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10330)[0x7f063cdd7330]
/usr/local/mariadb-10.3.8-linux-x86_64/bin/mysqld(_ZN13MYSQL_BIN_LOG13mark_xid_doneEmb+0x97)[0x7f063e4bf9a7]
/usr/local/mariadb-10.3.8-linux-x86_64/bin/mysqld(binlog_background_thread+0x305)[0x7f063e4bff65]
/usr/local/mariadb-10.3.8-linux-x86_64/bin/mysqld(+0x92afcd)[0x7f063e582fcd]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8182)[0x7f063cdcf182]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f063c4efb2d]
 
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x0): 
Connection ID (thread ID): 6
Status: NOT_KILLED
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on
 
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Writing a core file at /usr/local/mariadb-10.3.8-linux-x86_64/mysql-test/var/4/mysqld.1/data/
----------SERVER LOG END-------------
 
 
 - found 'core' (0/0)
 
Trying 'dbx' to get a backtrace
gdb not found, cannot get the stack trace
 
Trying 'lldb' to get a backtrace from coredump /usr/local/mariadb-10.3.8-linux-x86_64/mysql-test/var/4/log/rocksdb_rpl.rpl_binlog_xid_count-row/mysqld.1/data/core
Compressed file /usr/local/mariadb-10.3.8-linux-x86_64/mysql-test/var/4/log/rocksdb_rpl.rpl_binlog_xid_count-row/mysqld.1/data/core
 - saving '/usr/local/mariadb-10.3.8-linux-x86_64/mysql-test/var/4/log/rocksdb_rpl.rpl_binlog_xid_count-row/' to '/usr/local/mariadb-10.3.8-linux-x86_64/mysql-test/var/log/rocksdb_rpl.rpl_binlog_xid_count-row/'



 Comments   
Comment by Alice Sherepa [ 2021-04-21 ]

http://buildbot.askmonty.org/buildbot/builders/kvm-zyp-sles150-amd64/builds/5398/steps/mtr-rocksdb/logs/stdio

10.4 ceed768eead2c858b0ace76e1ed35fd7

rocksdb_rpl.rpl_binlog_xid_count 'row-write-committed' w4 [ fail ]
        Test ended at 2021-04-21 11:37:55
 
CURRENT_TEST: rocksdb_rpl.rpl_binlog_xid_count
mysqltest: At line 17: query 'INSERT INTO t SET a=repeat('a', $size)' failed: 2013: Lost connection to MySQL server during query
 
The result from queries just before the failure was:
< snip >
INSERT INTO t SET a=repeat('a', 4096);
INSERT INTO t SET a=repeat('a', 4096/2);
INSERT INTO t SET a=repeat('a', 4096);
INSERT INTO t SET a=repeat('a', 4096/2);
INSERT INTO t SET a=repeat('a', 4096);
INSERT INTO t SET a=repeat('a', 4096/2);
INSERT INTO t SET a=repeat('a', 4096);
INSERT INTO t SET a=repeat('a', 4096/2);
INSERT INTO t SET a=repeat('a', 4096);
INSERT INTO t SET a=repeat('a', 4096/2);
INSERT INTO t SET a=repeat('a', 4096);
INSERT INTO t SET a=repeat('a', 4096/2);
INSERT INTO t SET a=repeat('a', 4096);
INSERT INTO t SET a=repeat('a', 4096/2);
INSERT INTO t SET a=repeat('a', 4096);
INSERT INTO t SET a=repeat('a', 4096/2);
INSERT INTO t SET a=repeat('a', 4096);
INSERT INTO t SET a=repeat('a', 4096/2);
INSERT INTO t SET a=repeat('a', 4096);
 
More results from queries before failure can be found in /dev/shm/var/4/log/rpl_binlog_xid_count.log
 
 
Server [mysqld.1 - pid: 15488, winpid: 15488, exit: 256] failed during test run
Server log from this test:
----------SERVER LOG START-----------
2021-04-21 11:37:47 0 [Note] /usr/sbin/mysqld (mysqld 10.4.19-MariaDB-log) starting as process 15489 ...
2021-04-21 11:37:47 0 [Warning] Could not increase number of max_open_files to more than 1024 (request: 32186)
2021-04-21 11:37:47 0 [Note] RocksDB: Got ENOENT when listing column families
2021-04-21 11:37:47 0 [Note] RocksDB:   assuming that we're creating a new database
2021-04-21 11:37:47 0 [Note] RocksDB: Column Families at start:
2021-04-21 11:37:47 0 [Note]   cf=default
2021-04-21 11:37:47 0 [Note]     write_buffer_size=67108864
2021-04-21 11:37:47 0 [Note]     target_file_size_base=67108864
2021-04-21 11:37:47 0 [Note] RocksDB: creating a column family __system__
2021-04-21 11:37:47 0 [Note]     write_buffer_size=67108864
2021-04-21 11:37:47 0 [Note]     target_file_size_base=67108864
2021-04-21 11:37:47 0 [Note] RocksDB: Table_store: loaded DDL data for 0 tables
2021-04-21 11:37:47 0 [Note] RocksDB: global statistics using get_sched_indexer_t indexer
2021-04-21 11:37:47 0 [Note] MyRocks storage engine plugin has been successfully initialized.
2021-04-21 11:37:47 0 [Note] Plugin 'partition' is disabled.
2021-04-21 11:37:47 0 [Note] Plugin 'SEQUENCE' is disabled.
2021-04-21 11:37:47 0 [Note] InnoDB: Using Linux native AIO
2021-04-21 11:37:47 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-04-21 11:37:47 0 [Note] InnoDB: Uses event mutexes
2021-04-21 11:37:47 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2021-04-21 11:37:47 0 [Note] InnoDB: Number of pools: 1
2021-04-21 11:37:47 0 [Note] InnoDB: Using generic crc32 instructions
2021-04-21 11:37:47 0 [Note] mysqld: O_TMPFILE is not supported on /dev/shm/var/tmp/4/mysqld.1 (disabling future attempts)
2021-04-21 11:37:47 0 [Note] InnoDB: Initializing buffer pool, total size = 8M, instances = 1, chunk size = 8M
2021-04-21 11:37:47 0 [Note] InnoDB: Completed initialization of buffer pool
2021-04-21 11:37:47 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2021-04-21 11:37:47 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
2021-04-21 11:37:47 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2021-04-21 11:37:47 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2021-04-21 11:37:47 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2021-04-21 11:37:47 0 [Note] InnoDB: Waiting for purge to start
2021-04-21 11:37:47 0 [Note] InnoDB: 10.4.19 started; log sequence number 68170; transaction id 33
2021-04-21 11:37:47 0 [Note] Plugin 'INNODB_SYS_DATAFILES' is disabled.
2021-04-21 11:37:47 0 [Note] Plugin 'INNODB_SYS_TABLESTATS' is disabled.
2021-04-21 11:37:47 0 [Note] Plugin 'INNODB_LOCKS' is disabled.
2021-04-21 11:37:47 0 [Note] Plugin 'INNODB_MUTEXES' is disabled.
2021-04-21 11:37:47 0 [Note] Plugin 'INNODB_CMPMEM' is disabled.
2021-04-21 11:37:47 0 [Note] Plugin 'INNODB_CMP_PER_INDEX' is disabled.
2021-04-21 11:37:47 0 [Note] Plugin 'INNODB_CMP' is disabled.
2021-04-21 11:37:47 0 [Note] Plugin 'INNODB_TABLESPACES_ENCRYPTION' is disabled.
2021-04-21 11:37:47 0 [Note] Plugin 'INNODB_CMP_RESET' is disabled.
2021-04-21 11:37:47 0 [Note] Plugin 'INNODB_SYS_INDEXES' is disabled.
2021-04-21 11:37:47 0 [Note] Plugin 'INNODB_BUFFER_PAGE_LRU' is disabled.
2021-04-21 11:37:47 0 [Note] Plugin 'INNODB_CMPMEM_RESET' is disabled.
2021-04-21 11:37:47 0 [Note] Plugin 'INNODB_SYS_COLUMNS' is disabled.
2021-04-21 11:37:47 0 [Note] Plugin 'INNODB_FT_INDEX_TABLE' is disabled.
2021-04-21 11:37:47 0 [Note] Plugin 'INNODB_CMP_PER_INDEX_RESET' is disabled.
2021-04-21 11:37:47 0 [Note] Plugin 'INNODB_FT_INDEX_CACHE' is disabled.
2021-04-21 11:37:47 0 [Note] Plugin 'INNODB_SYS_FOREIGN_COLS' is disabled.
2021-04-21 11:37:47 0 [Note] Plugin 'INNODB_FT_BEING_DELETED' is disabled.
2021-04-21 11:37:47 0 [Note] Plugin 'INNODB_TRX' is disabled.
2021-04-21 11:37:47 0 [Note] Plugin 'INNODB_SYS_TABLES' is disabled.
2021-04-21 11:37:47 0 [Note] Plugin 'INNODB_FT_DEFAULT_STOPWORD' is disabled.
2021-04-21 11:37:47 0 [Note] Plugin 'INNODB_FT_CONFIG' is disabled.
2021-04-21 11:37:47 0 [Note] Plugin 'INNODB_SYS_TABLESPACES' is disabled.
2021-04-21 11:37:47 0 [Note] Plugin 'INNODB_TABLESPACES_SCRUBBING' is disabled.
2021-04-21 11:37:47 0 [Note] Plugin 'INNODB_SYS_SEMAPHORE_WAITS' is disabled.
2021-04-21 11:37:47 0 [Note] Plugin 'INNODB_SYS_FIELDS' is disabled.
2021-04-21 11:37:47 0 [Note] Plugin 'INNODB_METRICS' is disabled.
2021-04-21 11:37:47 0 [Note] Plugin 'INNODB_FT_DELETED' is disabled.
2021-04-21 11:37:47 0 [Note] Plugin 'user_variables' is disabled.
2021-04-21 11:37:47 0 [Note] Plugin 'INNODB_BUFFER_POOL_STATS' is disabled.
2021-04-21 11:37:47 0 [Note] Plugin 'INNODB_LOCK_WAITS' is disabled.
2021-04-21 11:37:47 0 [Note] InnoDB: Loading buffer pool(s) from /dev/shm/var/4/mysqld.1/data/ib_buffer_pool
2021-04-21 11:37:47 0 [Note] InnoDB: Buffer pool(s) load completed at 210421 11:37:47
2021-04-21 11:37:47 0 [Note] Plugin 'INNODB_SYS_FOREIGN' is disabled.
2021-04-21 11:37:47 0 [Note] Plugin 'INNODB_SYS_VIRTUAL' is disabled.
2021-04-21 11:37:47 0 [Note] Plugin 'INNODB_BUFFER_PAGE' is disabled.
2021-04-21 11:37:47 0 [Note] Plugin 'FEEDBACK' is disabled.
2021-04-21 11:37:47 0 [Note] Plugin 'unix_socket' is disabled.
2021-04-21 11:37:47 0 [Warning] /usr/sbin/mysqld: unknown variable 'loose-feedback-debug-startup-interval=20'
2021-04-21 11:37:47 0 [Warning] /usr/sbin/mysqld: unknown variable 'loose-feedback-debug-first-interval=60'
2021-04-21 11:37:47 0 [Warning] /usr/sbin/mysqld: unknown variable 'loose-feedback-debug-interval=60'
2021-04-21 11:37:47 0 [Warning] /usr/sbin/mysqld: unknown option '--loose-pam-debug'
2021-04-21 11:37:47 0 [Warning] /usr/sbin/mysqld: unknown option '--loose-aria'
2021-04-21 11:37:47 0 [Warning] /usr/sbin/mysqld: unknown option '--loose-enable-rocksdb_lock_info'
2021-04-21 11:37:47 0 [Warning] /usr/sbin/mysqld: unknown variable 'loose-debug-sync-timeout=300'
2021-04-21 11:37:47 0 [Note] Server socket created on IP: '127.0.0.1'.
2021-04-21 11:37:47 0 [Note] /usr/sbin/mysqld: ready for connections.
Version: '10.4.19-MariaDB-log'  socket: '/dev/shm/var/tmp/4/mysqld.1.sock'  port: 16020  MariaDB Server
2021-04-21 11:37:48 10 [Note] RocksDB: Manual memtable flush.
210421 11:37:48 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
Server version: 10.4.19-MariaDB-log
key_buffer_size=1048576
read_buffer_size=131072
max_used_connections=1
max_threads=153
thread_count=7
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 63542 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x7f7b40000bf8
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f7ba405acd8 thread_stack 0x49000
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5572079fb02e]
/usr/sbin/mysqld(handle_fatal_signal+0x515)[0x55720746c255]
/lib64/libpthread.so.0(+0x132f0)[0x7f7badcb02f0]
/usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG13mark_xid_doneEmb+0x98)[0x557207551a08]
/usr/sbin/mysqld(binlog_background_thread+0x385)[0x557207552145]
/usr/sbin/mysqld(+0xde98fa)[0x5572079a58fa]
/lib64/libpthread.so.0(+0x84f9)[0x7f7badca54f9]
/lib64/libc.so.6(clone+0x3f)[0x7f7bad0c0eef]
 
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x0): (null)
Connection ID (thread ID): 6
Status: NOT_KILLED
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on
 
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
 
We think the query pointer is invalid, but we will try to print it anyway. 
Query: 
 
Writing a core file...
Working directory at /dev/shm/var/4/mysqld.1/data
Resource Limits:
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        unlimited            unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             23754                23754                processes 
Max open files            1024                 1024                 files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       23754                23754                signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us        
Core pattern: |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %e
 
----------SERVER LOG END-------------

10.2 b3dd95e035c10ad30c52f582ce519b17

rocksdb_rpl.rpl_binlog_xid_count 'row-write-committed' w1 [ fail ]
        Test ended at 2020-07-23 07:49:04
 
CURRENT_TEST: rocksdb_rpl.rpl_binlog_xid_count
mysqltest: At line 17: query 'INSERT INTO t SET a=repeat('a', $size)' failed: 2013: Lost connection to MySQL server during query
 
The result from queries just before the failure was:
< snip >
INSERT INTO t SET a=repeat('a', 4096/2);
INSERT INTO t SET a=repeat('a', 4096);
INSERT INTO t SET a=repeat('a', 4096/2);
INSERT INTO t SET a=repeat('a', 4096);
INSERT INTO t SET a=repeat('a', 4096/2);
INSERT INTO t SET a=repeat('a', 4096);
INSERT INTO t SET a=repeat('a', 4096/2);
INSERT INTO t SET a=repeat('a', 4096);
INSERT INTO t SET a=repeat('a', 4096/2);
INSERT INTO t SET a=repeat('a', 4096);
INSERT INTO t SET a=repeat('a', 4096/2);
INSERT INTO t SET a=repeat('a', 4096);
INSERT INTO t SET a=repeat('a', 4096/2);
INSERT INTO t SET a=repeat('a', 4096);
INSERT INTO t SET a=repeat('a', 4096/2);
INSERT INTO t SET a=repeat('a', 4096);
INSERT INTO t SET a=repeat('a', 4096/2);
INSERT INTO t SET a=repeat('a', 4096);
INSERT INTO t SET a=repeat('a', 4096/2);
INSERT INTO t SET a=repeat('a', 4096);
 
More results from queries before failure can be found in /dev/shm/var/1/log/rpl_binlog_xid_count.log
 
 
Server [mysqld.1 - pid: 367, winpid: 367, exit: 256] failed during test run
Server log from this test:
----------SERVER LOG START-----------
2020-07-23  7:49:00 140279326938176 [Note] /usr/sbin/mysqld (mysqld 10.2.33-MariaDB-10.2.33+maria~stretch-log) starting as process 394 ...
2020-07-23  7:49:00 140279326938176 [Warning] Could not increase number of max_open_files to more than 1024 (request: 32186)
2020-07-23  7:49:01 140279326938176 [Note] RocksDB: Got ENOENT when listing column families
2020-07-23  7:49:01 140279326938176 [Note] RocksDB:   assuming that we're creating a new database
2020-07-23  7:49:01 140279326938176 [Note] RocksDB: Column Families at start:
2020-07-23  7:49:01 140279326938176 [Note]   cf=default
2020-07-23  7:49:01 140279326938176 [Note]     write_buffer_size=67108864
2020-07-23  7:49:01 140279326938176 [Note]     target_file_size_base=67108864
2020-07-23  7:49:01 140279326938176 [Note] RocksDB: creating a column family __system__
2020-07-23  7:49:01 140279326938176 [Note]     write_buffer_size=67108864
2020-07-23  7:49:01 140279326938176 [Note]     target_file_size_base=67108864
2020-07-23  7:49:01 140279326938176 [Note] RocksDB: Table_store: loaded DDL data for 0 tables
2020-07-23  7:49:01 140279326938176 [Note] RocksDB: global statistics using get_sched_indexer_t indexer
2020-07-23  7:49:01 140279326938176 [Note] MyRocks storage engine plugin has been successfully initialized.
2020-07-23  7:49:01 140279326938176 [Note] Plugin 'SEQUENCE' is disabled.
2020-07-23  7:49:01 140279326938176 [Note] Plugin 'partition' is disabled.
2020-07-23  7:49:01 140279326938176 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2020-07-23  7:49:01 140279326938176 [Note] InnoDB: Uses event mutexes
2020-07-23  7:49:01 140279326938176 [Note] InnoDB: Compressed tables use zlib 1.2.8
2020-07-23  7:49:01 140279326938176 [Note] InnoDB: Using Linux native AIO
2020-07-23  7:49:01 140279326938176 [Note] InnoDB: Number of pools: 1
2020-07-23  7:49:01 140279326938176 [Note] InnoDB: Using generic crc32 instructions
2020-07-23  7:49:01 140279326938176 [Note] InnoDB: Initializing buffer pool, total size = 8M, instances = 1, chunk size = 8M
2020-07-23  7:49:01 140279326938176 [Note] InnoDB: Completed initialization of buffer pool
2020-07-23  7:49:01 140278639871744 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2020-07-23  7:49:01 140279326938176 [Note] InnoDB: Highest supported file format is Barracuda.
2020-07-23  7:49:01 140279326938176 [Note] InnoDB: 128 out of 128 rollback segments are active.
2020-07-23  7:49:01 140279326938176 [Note] InnoDB: Creating shared tablespace for temporary tables
2020-07-23  7:49:01 140279326938176 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2020-07-23  7:49:01 140279326938176 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2020-07-23  7:49:01 140279326938176 [Note] InnoDB: Waiting for purge to start
2020-07-23  7:49:01 140279326938176 [Note] InnoDB: 5.7.31 started; log sequence number 1628244
2020-07-23  7:49:01 140277926852352 [Note] InnoDB: Loading buffer pool(s) from /dev/shm/var/1/mysqld.1/data/ib_buffer_pool
2020-07-23  7:49:01 140277926852352 [Note] InnoDB: Buffer pool(s) load completed at 200723  7:49:01
2020-07-23  7:49:01 140279326938176 [Note] Plugin 'INNODB_SYS_DATAFILES' is disabled.
2020-07-23  7:49:01 140279326938176 [Note] Plugin 'INNODB_SYS_TABLESTATS' is disabled.
2020-07-23  7:49:01 140279326938176 [Note] Plugin 'INNODB_LOCKS' is disabled.
2020-07-23  7:49:01 140279326938176 [Note] Plugin 'INNODB_MUTEXES' is disabled.
2020-07-23  7:49:01 140279326938176 [Note] Plugin 'INNODB_CMPMEM' is disabled.
2020-07-23  7:49:01 140279326938176 [Note] Plugin 'INNODB_CMP_PER_INDEX' is disabled.
2020-07-23  7:49:01 140279326938176 [Note] Plugin 'INNODB_CMP' is disabled.
2020-07-23  7:49:01 140279326938176 [Note] Plugin 'INNODB_TABLESPACES_ENCRYPTION' is disabled.
2020-07-23  7:49:01 140279326938176 [Note] Plugin 'INNODB_CMP_RESET' is disabled.
2020-07-23  7:49:01 140279326938176 [Note] Plugin 'INNODB_SYS_INDEXES' is disabled.
2020-07-23  7:49:01 140279326938176 [Note] Plugin 'INNODB_BUFFER_PAGE_LRU' is disabled.
2020-07-23  7:49:01 140279326938176 [Note] Plugin 'INNODB_CMPMEM_RESET' is disabled.
2020-07-23  7:49:01 140279326938176 [Note] Plugin 'INNODB_SYS_COLUMNS' is disabled.
2020-07-23  7:49:01 140279326938176 [Note] Plugin 'INNODB_FT_INDEX_TABLE' is disabled.
2020-07-23  7:49:01 140279326938176 [Note] Plugin 'INNODB_CMP_PER_INDEX_RESET' is disabled.
2020-07-23  7:49:01 140279326938176 [Note] Plugin 'INNODB_FT_INDEX_CACHE' is disabled.
2020-07-23  7:49:01 140279326938176 [Note] Plugin 'INNODB_SYS_FOREIGN_COLS' is disabled.
2020-07-23  7:49:01 140279326938176 [Note] Plugin 'INNODB_FT_BEING_DELETED' is disabled.
2020-07-23  7:49:01 140279326938176 [Note] Plugin 'INNODB_TRX' is disabled.
2020-07-23  7:49:01 140279326938176 [Note] Plugin 'INNODB_SYS_TABLES' is disabled.
2020-07-23  7:49:01 140279326938176 [Note] Plugin 'INNODB_FT_DEFAULT_STOPWORD' is disabled.
2020-07-23  7:49:01 140279326938176 [Note] Plugin 'INNODB_FT_CONFIG' is disabled.
2020-07-23  7:49:01 140279326938176 [Note] Plugin 'INNODB_SYS_TABLESPACES' is disabled.
2020-07-23  7:49:01 140279326938176 [Note] Plugin 'INNODB_TABLESPACES_SCRUBBING' is disabled.
2020-07-23  7:49:01 140279326938176 [Note] Plugin 'INNODB_SYS_SEMAPHORE_WAITS' is disabled.
2020-07-23  7:49:01 140279326938176 [Note] Plugin 'INNODB_SYS_FIELDS' is disabled.
2020-07-23  7:49:01 140279326938176 [Note] Plugin 'INNODB_METRICS' is disabled.
2020-07-23  7:49:01 140279326938176 [Note] Plugin 'INNODB_FT_DELETED' is disabled.
2020-07-23  7:49:01 140279326938176 [Note] Plugin 'user_variables' is disabled.
2020-07-23  7:49:01 140279326938176 [Note] Plugin 'INNODB_BUFFER_POOL_STATS' is disabled.
2020-07-23  7:49:01 140279326938176 [Note] Plugin 'INNODB_LOCK_WAITS' is disabled.
2020-07-23  7:49:01 140279326938176 [Note] Plugin 'INNODB_SYS_FOREIGN' is disabled.
2020-07-23  7:49:01 140279326938176 [Note] Plugin 'INNODB_SYS_VIRTUAL' is disabled.
2020-07-23  7:49:01 140279326938176 [Note] Plugin 'INNODB_BUFFER_PAGE' is disabled.
2020-07-23  7:49:01 140279326938176 [Note] Plugin 'FEEDBACK' is disabled.
2020-07-23  7:49:01 140279326938176 [Warning] /usr/sbin/mysqld: unknown variable 'loose-feedback-debug-startup-interval=20'
2020-07-23  7:49:01 140279326938176 [Warning] /usr/sbin/mysqld: unknown variable 'loose-feedback-debug-first-interval=60'
2020-07-23  7:49:01 140279326938176 [Warning] /usr/sbin/mysqld: unknown variable 'loose-feedback-debug-interval=60'
2020-07-23  7:49:01 140279326938176 [Warning] /usr/sbin/mysqld: unknown option '--loose-pam-debug'
2020-07-23  7:49:01 140279326938176 [Warning] /usr/sbin/mysqld: unknown option '--loose-enable-rocksdb_lock_info'
2020-07-23  7:49:01 140279326938176 [Warning] /usr/sbin/mysqld: unknown variable 'loose-debug-sync-timeout=300'
2020-07-23  7:49:01 140279326938176 [Note] Server socket created on IP: '127.0.0.1'.
2020-07-23  7:49:01 140279326938176 [Note] /usr/sbin/mysqld: ready for connections.
Version: '10.2.33-MariaDB-10.2.33+maria~stretch-log'  socket: '/dev/shm/var/tmp/1/mysqld.1.sock'  port: 16000  mariadb.org binary distribution
2020-07-23  7:49:02 140279138817792 [Note] RocksDB: Manual memtable flush.
200723  7:49:02 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
Server version: 10.2.33-MariaDB-10.2.33+maria~stretch-log
key_buffer_size=1048576
read_buffer_size=131072
max_used_connections=1
max_threads=153
thread_count=8
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 63019 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x7f94ec0009a8
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f95484b1ce8 thread_stack 0x49000
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x559fd949e40e]
/usr/sbin/mysqld(handle_fatal_signal+0x3ad)[0x559fd8f55a3d]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x110e0)[0x7f95533a30e0]
/usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG13mark_xid_doneEmb+0x97)[0x559fd90240c7]
/usr/sbin/mysqld(binlog_background_thread+0x386)[0x559fd9024836]
/usr/sbin/mysqld(+0xb7e041)[0x559fd9450041]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x74a4)[0x7f95533994a4]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f95518ecd0f]
 
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x0): 
Connection ID (thread ID): 6
Status: NOT_KILLED
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on
 
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /dev/shm/var/1/mysqld.1/data
Resource Limits:
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        unlimited            unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             23799                23799                processes 
Max open files            1024                 1024                 files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       23799                23799                signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us        
Core pattern: core
 
----------SERVER LOG END-------------
 
 
 - found 'core' (0/0)
 
Trying 'dbx' to get a backtrace
 
Trying 'gdb' to get a backtrace from coredump /dev/shm/var/1/log/rocksdb_rpl.rpl_binlog_xid_count-row-write-committed/mysqld.1/data/core
Core generated by '/usr/sbin/mysqld'
Output from gdb follows. The first stack trace is from the failing thread.
The following stack traces are from all threads (so the failing one is
duplicated).
--------------------------
[New LWP 1071]
[New LWP 706]
[New LWP 705]
[New LWP 394]
[New LWP 725]
[New LWP 708]
[New LWP 836]
[New LWP 759]
[New LWP 835]
[New LWP 841]
[New LWP 837]
[New LWP 972]
[New LWP 839]
[New LWP 973]
[New LWP 1089]
[New LWP 974]
[New LWP 1070]
[New LWP 1072]
[New LWP 1021]
[New LWP 1175]
[New LWP 1020]
[New LWP 1019]
[New LWP 1018]
[New LWP 935]
[New LWP 936]
[New LWP 934]
[New LWP 760]
[New LWP 758]
[New LWP 724]
[New LWP 707]
[New LWP 840]
[New LWP 838]
[New LWP 975]
[New LWP 976]
[New LWP 978]
[New LWP 977]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/mysqld --defaults-group-suffix=.1 --defaults-file=/dev/shm/var/1/my.c'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  __pthread_kill (threadid=<optimized out>, signo=signo@entry=11) at ../sysdeps/unix/sysv/linux/pthread_kill.c:57
57	../sysdeps/unix/sysv/linux/pthread_kill.c: No such file or directory.
[Current thread is 1 (Thread 0x7f95484b2700 (LWP 1071))]
#0  __pthread_kill (threadid=<optimized out>, signo=signo@entry=11) at ../sysdeps/unix/sysv/linux/pthread_kill.c:57
#1  0x0000559fd949dfc7 in my_write_core (sig=sig@entry=11) at ./mysys/stacktrace.c:477
#2  0x0000559fd8f559db in handle_fatal_signal (sig=11) at ./sql/signal_handler.cc:343
#3  <signal handler called>
#4  0x0000559fd90240c7 in MYSQL_BIN_LOG::mark_xid_done (this=this@entry=0x559fd9da8ca0 <mysql_bin_log>, binlog_id=44, write_checkpoint=write_checkpoint@entry=true) at ./sql/log.cc:9735
#5  0x0000559fd9024836 in binlog_background_thread (arg=arg@entry=0x0) at ./sql/log.cc:9931
#6  0x0000559fd9450041 in pfs_spawn_thread (arg=0x559fdb996c58) at ./storage/perfschema/pfs.cc:1869
#7  0x00007f95533994a4 in start_thread (arg=0x7f95484b2700) at pthread_create.c:456
#8  0x00007f95518ecd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
 
Thread 36 (Thread 0x7f951d7fa700 (LWP 977)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000559fd914a0b6 in os_event::wait (this=0x559fdb9e3650) at ./storage/innobase/os/os0event.cc:158
#2  os_event::wait_low (reset_sig_count=2, this=0x559fdb9e3650) at ./storage/innobase/os/os0event.cc:325
#3  os_event_wait_low (event=0x559fdb9e3650, reset_sig_count=reset_sig_count@entry=0) at ./storage/innobase/os/os0event.cc:507
#4  0x0000559fd91db43f in srv_resume_thread (sig_count=0, timeout_usec=0, wait=true, slot=0x559fd9cffc38 <srv_sys+248>) at ./storage/innobase/srv/srv0srv.cc:940
#5  srv_worker_thread (arg=<optimized out>) at ./storage/innobase/srv/srv0srv.cc:2620
#6  0x00007f95533994a4 in start_thread (arg=0x7f951d7fa700) at pthread_create.c:456
#7  0x00007f95518ecd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
 
Thread 35 (Thread 0x7f951cff9700 (LWP 978)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000559fd914a0b6 in os_event::wait (this=0x559fdb9e3730) at ./storage/innobase/os/os0event.cc:158
#2  os_event::wait_low (reset_sig_count=4, this=0x559fdb9e3730) at ./storage/innobase/os/os0event.cc:325
#3  os_event_wait_low (event=0x559fdb9e3730, reset_sig_count=reset_sig_count@entry=0) at ./storage/innobase/os/os0event.cc:507
#4  0x0000559fd91db43f in srv_resume_thread (sig_count=0, timeout_usec=0, wait=true, slot=0x559fd9cffca8 <srv_sys+360>) at ./storage/innobase/srv/srv0srv.cc:940
#5  srv_worker_thread (arg=<optimized out>) at ./storage/innobase/srv/srv0srv.cc:2620
#6  0x00007f95533994a4 in start_thread (arg=0x7f951cff9700) at pthread_create.c:456
#7  0x00007f95518ecd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
 
Thread 34 (Thread 0x7f951dffb700 (LWP 976)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000559fd914a0b6 in os_event::wait (this=0x559fdb9e36c0) at ./storage/innobase/os/os0event.cc:158
#2  os_event::wait_low (reset_sig_count=9, this=0x559fdb9e36c0) at ./storage/innobase/os/os0event.cc:325
#3  os_event_wait_low (event=0x559fdb9e36c0, reset_sig_count=reset_sig_count@entry=0) at ./storage/innobase/os/os0event.cc:507
#4  0x0000559fd91db43f in srv_resume_thread (sig_count=0, timeout_usec=0, wait=true, slot=0x559fd9cffc70 <srv_sys+304>) at ./storage/innobase/srv/srv0srv.cc:940
#5  srv_worker_thread (arg=<optimized out>) at ./storage/innobase/srv/srv0srv.cc:2620
#6  0x00007f95533994a4 in start_thread (arg=0x7f951dffb700) at pthread_create.c:456
#7  0x00007f95518ecd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
 
Thread 33 (Thread 0x7f951e7fc700 (LWP 975)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000559fd914a0b6 in os_event::wait (this=0x559fdb9e35e0) at ./storage/innobase/os/os0event.cc:158
#2  os_event::wait_low (reset_sig_count=1, this=0x559fdb9e35e0) at ./storage/innobase/os/os0event.cc:325
#3  os_event_wait_low (event=0x559fdb9e35e0, reset_sig_count=reset_sig_count@entry=1) at ./storage/innobase/os/os0event.cc:507
#4  0x0000559fd91dc902 in srv_resume_thread (timeout_usec=0, wait=true, sig_count=1, slot=0x559fd9cffc00 <srv_sys+192>) at ./storage/innobase/srv/srv0srv.cc:940
#5  srv_purge_coordinator_suspend (rseg_history_len=<optimized out>, slot=<optimized out>) at ./storage/innobase/srv/srv0srv.cc:2781
#6  srv_purge_coordinator_thread (arg=<optimized out>) at ./storage/innobase/srv/srv0srv.cc:2868
#7  0x00007f95533994a4 in start_thread (arg=0x7f951e7fc700) at pthread_create.c:456
#8  0x00007f95518ecd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
 
Thread 32 (Thread 0x7f952bfff700 (LWP 838)):
#0  0x00007f9552f7e6ca in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
#1  0x0000559fd9143e41 in LinuxAIOHandler::collect (this=this@entry=0x7f952bffeaf0) at ./storage/innobase/os/os0file.cc:1936
#2  0x0000559fd91443f2 in LinuxAIOHandler::poll (this=this@entry=0x7f952bffeaf0, m1=m1@entry=0x7f952bffeb90, m2=m2@entry=0x7f952bffeb98, request=request@entry=0x7f952bffeba0) at ./storage/innobase/os/os0file.cc:2097
#3  0x0000559fd9145ba8 in os_aio_linux_handler (request=0x7f952bffeba0, m2=0x7f952bffeb98, m1=0x7f952bffeb90, global_segment=<optimized out>) at ./storage/innobase/os/os0file.cc:2151
#4  os_aio_handler (segment=segment@entry=3, m1=m1@entry=0x7f952bffeb90, m2=m2@entry=0x7f952bffeb98, request=request@entry=0x7f952bffeba0) at ./storage/innobase/os/os0file.cc:5781
#5  0x0000559fd92d6c3a in fil_aio_wait (segment=segment@entry=3) at ./storage/innobase/fil/fil0fil.cc:5098
#6  0x0000559fd91de308 in io_handler_thread (arg=<optimized out>) at ./storage/innobase/srv/srv0start.cc:332
#7  0x00007f95533994a4 in start_thread (arg=0x7f952bfff700) at pthread_create.c:456
#8  0x00007f95518ecd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
 
Thread 31 (Thread 0x7f952affd700 (LWP 840)):
#0  0x00007f9552f7e6ca in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
#1  0x0000559fd9143e41 in LinuxAIOHandler::collect (this=this@entry=0x7f952affcaf0) at ./storage/innobase/os/os0file.cc:1936
#2  0x0000559fd91443f2 in LinuxAIOHandler::poll (this=this@entry=0x7f952affcaf0, m1=m1@entry=0x7f952affcb90, m2=m2@entry=0x7f952affcb98, request=request@entry=0x7f952affcba0) at ./storage/innobase/os/os0file.cc:2097
#3  0x0000559fd9145ba8 in os_aio_linux_handler (request=0x7f952affcba0, m2=0x7f952affcb98, m1=0x7f952affcb90, global_segment=<optimized out>) at ./storage/innobase/os/os0file.cc:2151
#4  os_aio_handler (segment=segment@entry=5, m1=m1@entry=0x7f952affcb90, m2=m2@entry=0x7f952affcb98, request=request@entry=0x7f952affcba0) at ./storage/innobase/os/os0file.cc:5781
#5  0x0000559fd92d6c3a in fil_aio_wait (segment=segment@entry=5) at ./storage/innobase/fil/fil0fil.cc:5098
#6  0x0000559fd91de308 in io_handler_thread (arg=<optimized out>) at ./storage/innobase/srv/srv0start.cc:332
#7  0x00007f95533994a4 in start_thread (arg=0x7f952affd700) at pthread_create.c:456
#8  0x00007f95518ecd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
 
Thread 30 (Thread 0x7f954aaf3700 (LWP 707)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007f9551f5b50c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x00007f954b9860f6 in rocksdb::ThreadPoolImpl::Impl::BGThread (this=this@entry=0x559fdb8d43e0, thread_id=thread_id@entry=0) at ./storage/rocksdb/rocksdb/util/threadpool_imp.cc:196
#3  0x00007f954b9864f9 in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper (arg=0x559fdb94c7d0) at ./storage/rocksdb/rocksdb/util/threadpool_imp.cc:306
#4  0x00007f9551f60e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007f95533994a4 in start_thread (arg=0x7f954aaf3700) at pthread_create.c:456
#6  0x00007f95518ecd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
 
Thread 29 (Thread 0x7f9531ffb700 (LWP 724)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1  0x00007f954b9eb59e in rocksdb::port::CondVar::TimedWait (this=this@entry=0x559fdb95ed90, abs_time_us=<optimized out>) at ./storage/rocksdb/rocksdb/port/port_posix.cc:120
#2  0x00007f954b8be672 in rocksdb::InstrumentedCondVar::TimedWaitInternal (this=this@entry=0x559fdb95ed90, abs_time_us=<optimized out>, abs_time_us@entry=1595505541045749) at ./storage/rocksdb/rocksdb/monitoring/instrumented_mutex.cc:66
#3  0x00007f954b8be768 in rocksdb::InstrumentedCondVar::TimedWait (this=this@entry=0x559fdb95ed90, abs_time_us=abs_time_us@entry=1595505541045749) at ./storage/rocksdb/rocksdb/monitoring/instrumented_mutex.cc:55
#4  0x00007f954b76ea4b in rocksdb::RepeatableThread::wait (delay=<optimized out>, this=0x559fdb95ecf0) at ./storage/rocksdb/rocksdb/util/repeatable_thread.h:90
#5  rocksdb::RepeatableThread::thread (this=0x559fdb95ecf0) at ./storage/rocksdb/rocksdb/util/repeatable_thread.h:117
#6  rocksdb::RepeatableThread::RepeatableThread(std::function<void ()>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, unsigned long, unsigned long)::{lambda()#1}::operator()() const (__closure=<optimized out>) at ./storage/rocksdb/rocksdb/util/repeatable_thread.h:38
#7  std::_Bind_simple<rocksdb::RepeatableThread::RepeatableThread(std::function<void ()>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, unsigned long, unsigned long)::{lambda()#1} ()>::_M_invoke<>(std::_Index_tuple<>) (this=<optimized out>) at /usr/include/c++/6/functional:1391
#8  std::_Bind_simple<rocksdb::RepeatableThread::RepeatableThread(std::function<void ()>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, unsigned long, unsigned long)::{lambda()#1} ()>::operator()() (this=<optimized out>) at /usr/include/c++/6/functional:1380
#9  std::thread::_State_impl<std::_Bind_simple<rocksdb::RepeatableThread::RepeatableThread(std::function<void ()>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, unsigned long, unsigned long)::{lambda()#1} ()> >::_M_run() (this=<optimized out>) at /usr/include/c++/6/thread:197
#10 0x00007f9551f60e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#11 0x00007f95533994a4 in start_thread (arg=0x7f9531ffb700) at pthread_create.c:456
#12 0x00007f95518ecd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
 
Thread 28 (Thread 0x7f9532ffd700 (LWP 758)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1  0x00007f954b6ee3d6 in inline_mysql_cond_timedwait (src_file=0x7f954ba9c798 "/home/buildbot/buildbot/build/mariadb-10.2.33/storage/rocksdb/ha_rocksdb.cc", src_line=13466, abstime=0x7f9532ffcc90, mutex=0x7f954bdc4bb8 <myrocks::rdb_bg_thread+56>, that=0x7f954bdc4be8 <myrocks::rdb_bg_thread+104>) at ./include/mysql/psi/mysql_thread.h:1193
#2  myrocks::Rdb_background_thread::run (this=0x7f954bdc4b80 <myrocks::rdb_bg_thread>) at ./storage/rocksdb/ha_rocksdb.cc:13466
#3  0x00007f954b7340a2 in myrocks::Rdb_thread::thread_func (thread_ptr=thread_ptr@entry=0x7f954bdc4b80 <myrocks::rdb_bg_thread>) at ./storage/rocksdb/rdb_threads.cc:34
#4  0x0000559fd9450041 in pfs_spawn_thread (arg=0x559fdb995a58) at ./storage/perfschema/pfs.cc:1869
#5  0x00007f95533994a4 in start_thread (arg=0x7f9532ffd700) at pthread_create.c:456
#6  0x00007f95518ecd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
 
Thread 27 (Thread 0x7f9549af1700 (LWP 760)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1  0x00007f954b6e1d26 in inline_mysql_cond_timedwait (src_file=0x7f954ba9c798 "/home/buildbot/buildbot/build/mariadb-10.2.33/storage/rocksdb/ha_rocksdb.cc", src_line=13562, abstime=0x7f9549af0cb0, mutex=0x7f954bdc4a98 <myrocks::rdb_mc_thread+56>, that=0x7f954bdc4ac8 <myrocks::rdb_mc_thread+104>) at ./include/mysql/psi/mysql_thread.h:1193
#2  myrocks::Rdb_manual_compaction_thread::run (this=0x7f954bdc4a60 <myrocks::rdb_mc_thread>) at ./storage/rocksdb/ha_rocksdb.cc:13562
#3  0x00007f954b7340a2 in myrocks::Rdb_thread::thread_func (thread_ptr=thread_ptr@entry=0x7f954bdc4a60 <myrocks::rdb_mc_thread>) at ./storage/rocksdb/rdb_threads.cc:34
#4  0x0000559fd9450041 in pfs_spawn_thread (arg=0x559fdb995cb8) at ./storage/perfschema/pfs.cc:1869
#5  0x00007f95533994a4 in start_thread (arg=0x7f9549af1700) at pthread_create.c:456
#6  0x00007f95518ecd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
 
Thread 26 (Thread 0x7f95297fa700 (LWP 934)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1  0x0000559fd9149a72 in os_event::timed_wait (this=this@entry=0x559fdbbe1750, abstime=abstime@entry=0x7f95297f9c00) at ./storage/innobase/os/os0event.cc:275
#2  0x0000559fd9149cc1 in os_event::wait_time_low (this=this@entry=0x559fdbbe1750, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=1) at ./storage/innobase/os/os0event.cc:385
#3  0x0000559fd914a06a in os_event_wait_time_low (event=event@entry=0x559fdbbe1750, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=<optimized out>) at ./storage/innobase/os/os0event.cc:490
#4  0x0000559fd9121550 in lock_wait_timeout_thread () at ./storage/innobase/lock/lock0wait.cc:525
#5  0x00007f95533994a4 in start_thread (arg=0x7f95297fa700) at pthread_create.c:456
#6  0x00007f95518ecd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
 
Thread 25 (Thread 0x7f951ffff700 (LWP 936)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1  0x0000559fd9149a72 in os_event::timed_wait (this=this@entry=0x559fdb9e3810, abstime=abstime@entry=0x7f951fffec90) at ./storage/innobase/os/os0event.cc:275
#2  0x0000559fd9149cc1 in os_event::wait_time_low (this=0x559fdb9e3810, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=1) at ./storage/innobase/os/os0event.cc:385
#3  0x0000559fd914a06a in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=<optimized out>) at ./storage/innobase/os/os0event.cc:490
#4  0x0000559fd91d9212 in srv_monitor_thread () at ./storage/innobase/srv/srv0srv.cc:1751
#5  0x00007f95533994a4 in start_thread (arg=0x7f951ffff700) at pthread_create.c:456
#6  0x00007f95518ecd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
 
Thread 24 (Thread 0x7f9528ff9700 (LWP 935)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1  0x0000559fd9149a72 in os_event::timed_wait (this=this@entry=0x559fdb9e37a0, abstime=abstime@entry=0x7f9528ff8b60) at ./storage/innobase/os/os0event.cc:275
#2  0x0000559fd9149cc1 in os_event::wait_time_low (this=0x559fdb9e37a0, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=1) at ./storage/innobase/os/os0event.cc:385
#3  0x0000559fd914a06a in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=<optimized out>) at ./storage/innobase/os/os0event.cc:490
#4  0x0000559fd91d98ac in srv_error_monitor_thread () at ./storage/innobase/srv/srv0srv.cc:1915
#5  0x00007f95533994a4 in start_thread (arg=0x7f9528ff9700) at pthread_create.c:456
#6  0x00007f95518ecd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
 
Thread 23 (Thread 0x7f94fffff700 (LWP 1018)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000559fd914a0b6 in os_event::wait (this=0x559fdb9e3880) at ./storage/innobase/os/os0event.cc:158
#2  os_event::wait_low (reset_sig_count=1, this=0x559fdb9e3880) at ./storage/innobase/os/os0event.cc:325
#3  os_event_wait_low (event=0x559fdb9e3880, reset_sig_count=reset_sig_count@entry=0) at ./storage/innobase/os/os0event.cc:507
#4  0x0000559fd92792f7 in buf_dump_thread () at ./storage/innobase/buf/buf0dump.cc:806
#5  0x00007f95533994a4 in start_thread (arg=0x7f94fffff700) at pthread_create.c:456
#6  0x00007f95518ecd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
 
Thread 22 (Thread 0x7f94ff7fe700 (LWP 1019)):
#0  0x00007f95533a27fd in nanosleep () at ../sysdeps/unix/syscall-template.S:84
#1  0x0000559fd914a320 in os_thread_sleep (tm=tm@entry=1000000) at ./storage/innobase/os/os0thread.cc:231
#2  0x0000559fd9262e42 in btr_defragment_thread () at ./storage/innobase/btr/btr0defragment.cc:727
#3  0x00007f95533994a4 in start_thread (arg=0x7f94ff7fe700) at pthread_create.c:456
#4  0x00007f95518ecd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
 
Thread 21 (Thread 0x7f94feffd700 (LWP 1020)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000559fd914a0b6 in os_event::wait (this=0x559fdb9e3960) at ./storage/innobase/os/os0event.cc:158
#2  os_event::wait_low (reset_sig_count=1, this=0x559fdb9e3960) at ./storage/innobase/os/os0event.cc:325
#3  os_event_wait_low (event=0x559fdb9e3960, reset_sig_count=reset_sig_count@entry=0) at ./storage/innobase/os/os0event.cc:507
#4  0x0000559fd92717ac in buf_resize_thread () at ./storage/innobase/buf/buf0buf.cc:3127
#5  0x00007f95533994a4 in start_thread (arg=0x7f94feffd700) at pthread_create.c:456
#6  0x00007f95518ecd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
 
Thread 20 (Thread 0x7f95483d1700 (LWP 1175)):
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007f955339bce6 in __GI___pthread_mutex_lock (mutex=mutex@entry=0x559fd9da9138 <mysql_bin_log+1176>) at ../nptl/pthread_mutex_lock.c:135
#2  0x0000559fd9023f6f in inline_mysql_mutex_lock (src_file=0x559fd95c5af8 "/home/buildbot/buildbot/build/mariadb-10.2.33/sql/log.cc", src_line=9665, that=0x559fd9da9138 <mysql_bin_log+1176>) at ./include/mysql/psi/mysql_thread.h:675
#3  MYSQL_BIN_LOG::mark_xids_active (this=this@entry=0x559fd9da8ca0 <mysql_bin_log>, binlog_id=binlog_id@entry=49, xid_count=xid_count@entry=1) at ./sql/log.cc:9665
#4  0x0000559fd902762f in MYSQL_BIN_LOG::rotate (this=this@entry=0x559fd9da8ca0 <mysql_bin_log>, force_rotate=force_rotate@entry=false, check_purge=check_purge@entry=0x7f95483ce80e) at ./sql/log.cc:6660
#5  0x0000559fd9027fae in MYSQL_BIN_LOG::trx_group_commit_leader (this=this@entry=0x559fd9da8ca0 <mysql_bin_log>, leader=leader@entry=0x7f95483ce8d0) at ./sql/log.cc:7949
#6  0x0000559fd90281c3 in MYSQL_BIN_LOG::write_transaction_to_binlog_events (this=this@entry=0x559fd9da8ca0 <mysql_bin_log>, entry=entry@entry=0x7f95483ce8d0) at ./sql/log.cc:7641
#7  0x0000559fd9028471 in MYSQL_BIN_LOG::write_transaction_to_binlog (this=this@entry=0x559fd9da8ca0 <mysql_bin_log>, thd=thd@entry=0x7f94dc0009a8, cache_mngr=cache_mngr@entry=0x7f94dc0ac9c8, end_ev=end_ev@entry=0x7f95483cea30, all=<optimized out>, using_stmt_cache=using_stmt_cache@entry=true, using_trx_cache=true) at ./sql/log.cc:7315
#8  0x0000559fd90286b2 in binlog_flush_cache (thd=thd@entry=0x7f94dc0009a8, cache_mngr=cache_mngr@entry=0x7f94dc0ac9c8, end_ev=end_ev@entry=0x7f95483cea30, all=all@entry=false, using_stmt=using_stmt@entry=true, using_trx=using_trx@entry=true) at ./sql/log.cc:1743
#9  0x0000559fd9029272 in binlog_commit_flush_xid_caches (thd=thd@entry=0x7f94dc0009a8, cache_mngr=cache_mngr@entry=0x7f94dc0ac9c8, all=all@entry=false, xid=xid@entry=121) at ./sql/log.cc:1851
#10 0x0000559fd9029434 in MYSQL_BIN_LOG::log_and_order (this=<optimized out>, thd=0x7f94dc0009a8, xid=121, all=<optimized out>, need_prepare_ordered=<optimized out>, need_commit_ordered=<optimized out>) at ./sql/log.cc:9621
#11 0x0000559fd8f5917c in ha_commit_trans (thd=thd@entry=0x7f94dc0009a8, all=all@entry=false) at ./sql/handler.cc:1497
#12 0x0000559fd8e9c5e7 in trans_commit_stmt (thd=thd@entry=0x7f94dc0009a8) at ./sql/transaction.cc:510
#13 0x0000559fd8db608c in mysql_execute_command (thd=thd@entry=0x7f94dc0009a8) at ./sql/sql_parse.cc:6035
#14 0x0000559fd8dbdd5e in mysql_parse (thd=thd@entry=0x7f94dc0009a8, rawbuf=<optimized out>, length=37, parser_state=parser_state@entry=0x7f95483d04a0, is_com_multi=is_com_multi@entry=false, is_next_command=is_next_command@entry=false) at ./sql/sql_parse.cc:7733
#15 0x0000559fd8dc05f0 in dispatch_command (command=command@entry=COM_QUERY, thd=thd@entry=0x7f94dc0009a8, packet=packet@entry=0x7f94dc006d99 "INSERT INTO t SET a=repeat('a', 4096)", packet_length=packet_length@entry=37, is_com_multi=is_com_multi@entry=false, is_next_command=is_next_command@entry=false) at ./sql/sql_parse.cc:1824
#16 0x0000559fd8dc13b0 in do_command (thd=0x7f94dc0009a8) at ./sql/sql_parse.cc:1377
#17 0x0000559fd8e8dcd2 in do_handle_one_connection (connect=connect@entry=0x559fdbfa1758) at ./sql/sql_connect.cc:1336
#18 0x0000559fd8e8de2d in handle_one_connection (arg=arg@entry=0x559fdbfa1758) at ./sql/sql_connect.cc:1241
#19 0x0000559fd9450041 in pfs_spawn_thread (arg=0x559fdbf58268) at ./storage/perfschema/pfs.cc:1869
#20 0x00007f95533994a4 in start_thread (arg=0x7f95483d1700) at pthread_create.c:456
#21 0x00007f95518ecd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
 
Thread 19 (Thread 0x7f94fe7fc700 (LWP 1021)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000559fd90cc88b in inline_mysql_cond_wait (src_file=0x559fd95cd368 "/home/buildbot/buildbot/build/mariadb-10.2.33/storage/innobase/handler/ha_innodb.cc", src_line=328, mutex=0x7f94fe7fbcc0, that=0x7f94fe7fbcf0) at ./include/mysql/psi/mysql_thread.h:1156
#2  thd_destructor_proxy () at ./storage/innobase/handler/ha_innodb.cc:328
#3  0x0000559fd9450041 in pfs_spawn_thread (arg=0x559fdbbbefd8) at ./storage/perfschema/pfs.cc:1869
#4  0x00007f95533994a4 in start_thread (arg=0x7f94fe7fc700) at pthread_create.c:456
#5  0x00007f95518ecd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
 
Thread 18 (Thread 0x7f9548467700 (LWP 1072)):
#0  do_sigwait (set=<optimized out>, set@entry=0x7f9548466c90, sig=sig@entry=0x7f9548466c38) at ../sysdeps/unix/sysv/linux/sigwait.c:64
#1  0x00007f95533a2d0d in __sigwait (set=set@entry=0x7f9548466c90, sig=sig@entry=0x7f9548466c38) at ../sysdeps/unix/sysv/linux/sigwait.c:96
#2  0x0000559fd8d140ab in signal_hand (arg=arg@entry=0x0) at ./sql/mysqld.cc:3511
#3  0x0000559fd9450041 in pfs_spawn_thread (arg=0x559fdbf530f8) at ./storage/perfschema/pfs.cc:1869
#4  0x00007f95533994a4 in start_thread (arg=0x7f9548467700) at pthread_create.c:456
#5  0x00007f95518ecd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
 
Thread 17 (Thread 0x7f94fdffb700 (LWP 1070)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1  0x0000559fd938214e in inline_mysql_cond_timedwait (src_file=0x559fd961d278 "/home/buildbot/buildbot/build/mariadb-10.2.33/storage/maria/ma_servicethread.c", src_line=116, abstime=0x7f94fdffac50, mutex=0x559fda5febe0 <LOCK_checkpoint>, that=0x559fda5feba0 <COND_checkpoint>) at ./include/mysql/psi/mysql_thread.h:1182
#2  my_service_thread_sleep (control=control@entry=0x559fd9d019c0 <checkpoint_control>, sleep_time=sleep_time@entry=29000000000) at ./storage/maria/ma_servicethread.c:115
#3  0x0000559fd9379b76 in ma_checkpoint_background (arg=arg@entry=0x1e) at ./storage/maria/ma_checkpoint.c:709
#4  0x0000559fd9450041 in pfs_spawn_thread (arg=0x559fdbf30f28) at ./storage/perfschema/pfs.cc:1869
#5  0x00007f95533994a4 in start_thread (arg=0x7f94fdffb700) at pthread_create.c:456
#6  0x00007f95518ecd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
 
Thread 16 (Thread 0x7f951effd700 (LWP 974)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1  0x0000559fd9149a72 in os_event::timed_wait (this=this@entry=0x559fdbbbde30, abstime=abstime@entry=0x7f951effca40) at ./storage/innobase/os/os0event.cc:275
#2  0x0000559fd9149cc1 in os_event::wait_time_low (this=0x559fdbbbde30, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=reset_sig_count@entry=1) at ./storage/innobase/os/os0event.cc:385
#3  0x0000559fd914a06a in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=reset_sig_count@entry=1) at ./storage/innobase/os/os0event.cc:490
#4  0x0000559fd922f142 in ib_wqueue_timedwait (wq=wq@entry=0x559fdbc1e350, wait_in_usecs=wait_in_usecs@entry=5000000) at ./storage/innobase/ut/ut0wqueue.cc:156
#5  0x0000559fd931508a in fts_optimize_thread (arg=<optimized out>) at ./storage/innobase/fts/fts0opt.cc:2820
#6  0x00007f95533994a4 in start_thread (arg=0x7f951effd700) at pthread_create.c:456
#7  0x00007f95518ecd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
 
Thread 15 (Thread 0x7f954841c700 (LWP 1089)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000559fd8d3246e in inline_mysql_cond_wait (that=<optimized out>, mutex=<optimized out>, src_file=0x559fd9522648 "/home/buildbot/buildbot/build/mariadb-10.2.33/sql/slave.cc", src_line=337) at ./include/mysql/psi/mysql_thread.h:1145
#2  handle_slave_background (arg=arg@entry=0x0) at ./sql/slave.cc:337
#3  0x0000559fd9450041 in pfs_spawn_thread (arg=0x559fdbf3f488) at ./storage/perfschema/pfs.cc:1869
#4  0x00007f95533994a4 in start_thread (arg=0x7f954841c700) at pthread_create.c:456
#5  0x00007f95518ecd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
 
Thread 14 (Thread 0x7f951f7fe700 (LWP 973)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1  0x0000559fd9149a72 in os_event::timed_wait (this=this@entry=0x559fdbc12fe0, abstime=abstime@entry=0x7f951f7fdbe0) at ./storage/innobase/os/os0event.cc:275
#2  0x0000559fd9149cc1 in os_event::wait_time_low (this=0x559fdbc12fe0, time_in_usec=time_in_usec@entry=10000000, reset_sig_count=1, reset_sig_count@entry=0) at ./storage/innobase/os/os0event.cc:385
#3  0x0000559fd914a06a in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=10000000, reset_sig_count=reset_sig_count@entry=0) at ./storage/innobase/os/os0event.cc:490
#4  0x0000559fd92cb7d5 in dict_stats_thread () at ./storage/innobase/dict/dict0stats_bg.cc:475
#5  0x00007f95533994a4 in start_thread (arg=0x7f951f7fe700) at pthread_create.c:456
#6  0x00007f95518ecd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
 
Thread 13 (Thread 0x7f952b7fe700 (LWP 839)):
#0  0x00007f9552f7e6ca in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
#1  0x0000559fd9143e41 in LinuxAIOHandler::collect (this=this@entry=0x7f952b7fdaf0) at ./storage/innobase/os/os0file.cc:1936
#2  0x0000559fd91443f2 in LinuxAIOHandler::poll (this=this@entry=0x7f952b7fdaf0, m1=m1@entry=0x7f952b7fdb90, m2=m2@entry=0x7f952b7fdb98, request=request@entry=0x7f952b7fdba0) at ./storage/innobase/os/os0file.cc:2097
#3  0x0000559fd9145ba8 in os_aio_linux_handler (request=0x7f952b7fdba0, m2=0x7f952b7fdb98, m1=0x7f952b7fdb90, global_segment=<optimized out>) at ./storage/innobase/os/os0file.cc:2151
#4  os_aio_handler (segment=segment@entry=4, m1=m1@entry=0x7f952b7fdb90, m2=m2@entry=0x7f952b7fdb98, request=request@entry=0x7f952b7fdba0) at ./storage/innobase/os/os0file.cc:5781
#5  0x0000559fd92d6c3a in fil_aio_wait (segment=segment@entry=4) at ./storage/innobase/fil/fil0fil.cc:5098
#6  0x0000559fd91de308 in io_handler_thread (arg=<optimized out>) at ./storage/innobase/srv/srv0start.cc:332
#7  0x00007f95533994a4 in start_thread (arg=0x7f952b7fe700) at pthread_create.c:456
#8  0x00007f95518ecd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
 
Thread 12 (Thread 0x7f9529ffb700 (LWP 972)):
#0  0x00007f95533a27fd in nanosleep () at ../sysdeps/unix/syscall-template.S:84
#1  0x0000559fd914a320 in os_thread_sleep (tm=tm@entry=1000000) at ./storage/innobase/os/os0thread.cc:231
#2  0x0000559fd91da982 in srv_master_sleep () at ./storage/innobase/srv/srv0srv.cc:2422
#3  srv_master_thread (arg=<optimized out>) at ./storage/innobase/srv/srv0srv.cc:2462
#4  0x00007f95533994a4 in start_thread (arg=0x7f9529ffb700) at pthread_create.c:456
#5  0x00007f95518ecd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
 
Thread 11 (Thread 0x7f9530ff9700 (LWP 837)):
#0  0x00007f9552f7e6ca in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
#1  0x0000559fd9143e41 in LinuxAIOHandler::collect (this=this@entry=0x7f9530ff8af0) at ./storage/innobase/os/os0file.cc:1936
#2  0x0000559fd91443f2 in LinuxAIOHandler::poll (this=this@entry=0x7f9530ff8af0, m1=m1@entry=0x7f9530ff8b90, m2=m2@entry=0x7f9530ff8b98, request=request@entry=0x7f9530ff8ba0) at ./storage/innobase/os/os0file.cc:2097
#3  0x0000559fd9145ba8 in os_aio_linux_handler (request=0x7f9530ff8ba0, m2=0x7f9530ff8b98, m1=0x7f9530ff8b90, global_segment=<optimized out>) at ./storage/innobase/os/os0file.cc:2151
#4  os_aio_handler (segment=segment@entry=2, m1=m1@entry=0x7f9530ff8b90, m2=m2@entry=0x7f9530ff8b98, request=request@entry=0x7f9530ff8ba0) at ./storage/innobase/os/os0file.cc:5781
#5  0x0000559fd92d6c3a in fil_aio_wait (segment=segment@entry=2) at ./storage/innobase/fil/fil0fil.cc:5098
#6  0x0000559fd91de308 in io_handler_thread (arg=<optimized out>) at ./storage/innobase/srv/srv0start.cc:332
#7  0x00007f95533994a4 in start_thread (arg=0x7f9530ff9700) at pthread_create.c:456
#8  0x00007f95518ecd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
 
Thread 10 (Thread 0x7f952a7fc700 (LWP 841)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1  0x0000559fd9149a72 in os_event::timed_wait (this=this@entry=0x559fdb9e38f0, abstime=abstime@entry=0x7f952a7fb970) at ./storage/innobase/os/os0event.cc:275
#2  0x0000559fd9149cc1 in os_event::wait_time_low (this=0x559fdb9e38f0, time_in_usec=time_in_usec@entry=979000, reset_sig_count=reset_sig_count@entry=4) at ./storage/innobase/os/os0event.cc:385
#3  0x0000559fd914a06a in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=979000, reset_sig_count=reset_sig_count@entry=4) at ./storage/innobase/os/os0event.cc:490
#4  0x0000559fd928414a in pc_sleep_if_needed (cur_time=868551, sig_count=4, next_loop_time=869530) at ./storage/innobase/buf/buf0flu.cc:2683
#5  buf_flush_page_cleaner_coordinator () at ./storage/innobase/buf/buf0flu.cc:3149
#6  0x00007f95533994a4 in start_thread (arg=0x7f952a7fc700) at pthread_create.c:456
#7  0x00007f95518ecd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
 
Thread 9 (Thread 0x7f9533fff700 (LWP 835)):
#0  0x00007f9552f7e6ca in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
#1  0x0000559fd9143e41 in LinuxAIOHandler::collect (this=this@entry=0x7f9533ffeaf0) at ./storage/innobase/os/os0file.cc:1936
#2  0x0000559fd91443f2 in LinuxAIOHandler::poll (this=this@entry=0x7f9533ffeaf0, m1=m1@entry=0x7f9533ffeb90, m2=m2@entry=0x7f9533ffeb98, request=request@entry=0x7f9533ffeba0) at ./storage/innobase/os/os0file.cc:2097
#3  0x0000559fd9145ba8 in os_aio_linux_handler (request=0x7f9533ffeba0, m2=0x7f9533ffeb98, m1=0x7f9533ffeb90, global_segment=<optimized out>) at ./storage/innobase/os/os0file.cc:2151
#4  os_aio_handler (segment=segment@entry=0, m1=m1@entry=0x7f9533ffeb90, m2=m2@entry=0x7f9533ffeb98, request=request@entry=0x7f9533ffeba0) at ./storage/innobase/os/os0file.cc:5781
#5  0x0000559fd92d6c3a in fil_aio_wait (segment=segment@entry=0) at ./storage/innobase/fil/fil0fil.cc:5098
#6  0x0000559fd91de308 in io_handler_thread (arg=<optimized out>) at ./storage/innobase/srv/srv0start.cc:332
#7  0x00007f95533994a4 in start_thread (arg=0x7f9533fff700) at pthread_create.c:456
#8  0x00007f95518ecd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
 
Thread 8 (Thread 0x7f95337fe700 (LWP 759)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1  0x00007f954b6ed9ba in inline_mysql_cond_timedwait (src_file=0x7f954ba9c798 "/home/buildbot/buildbot/build/mariadb-10.2.33/storage/rocksdb/ha_rocksdb.cc", src_line=11527, abstime=0x7f95337fdb80, mutex=0x7f954bdc4798 <myrocks::rdb_drop_idx_thread+56>, that=0x7f954bdc47c8 <myrocks::rdb_drop_idx_thread+104>) at ./include/mysql/psi/mysql_thread.h:1193
#2  myrocks::Rdb_drop_index_thread::run (this=0x7f954bdc4760 <myrocks::rdb_drop_idx_thread>) at ./storage/rocksdb/ha_rocksdb.cc:11527
#3  0x00007f954b7340a2 in myrocks::Rdb_thread::thread_func (thread_ptr=thread_ptr@entry=0x7f954bdc4760 <myrocks::rdb_drop_idx_thread>) at ./storage/rocksdb/rdb_threads.cc:34
#4  0x0000559fd9450041 in pfs_spawn_thread (arg=0x559fdb995358) at ./storage/perfschema/pfs.cc:1869
#5  0x00007f95533994a4 in start_thread (arg=0x7f95337fe700) at pthread_create.c:456
#6  0x00007f95518ecd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
 
Thread 7 (Thread 0x7f95317fa700 (LWP 836)):
#0  0x00007f9552f7e6ca in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
#1  0x0000559fd9143e41 in LinuxAIOHandler::collect (this=this@entry=0x7f95317f9af0) at ./storage/innobase/os/os0file.cc:1936
#2  0x0000559fd91443f2 in LinuxAIOHandler::poll (this=this@entry=0x7f95317f9af0, m1=m1@entry=0x7f95317f9b90, m2=m2@entry=0x7f95317f9b98, request=request@entry=0x7f95317f9ba0) at ./storage/innobase/os/os0file.cc:2097
#3  0x0000559fd9145ba8 in os_aio_linux_handler (request=0x7f95317f9ba0, m2=0x7f95317f9b98, m1=0x7f95317f9b90, global_segment=<optimized out>) at ./storage/innobase/os/os0file.cc:2151
#4  os_aio_handler (segment=segment@entry=1, m1=m1@entry=0x7f95317f9b90, m2=m2@entry=0x7f95317f9b98, request=request@entry=0x7f95317f9ba0) at ./storage/innobase/os/os0file.cc:5781
#5  0x0000559fd92d6c3a in fil_aio_wait (segment=segment@entry=1) at ./storage/innobase/fil/fil0fil.cc:5098
#6  0x0000559fd91de308 in io_handler_thread (arg=<optimized out>) at ./storage/innobase/srv/srv0start.cc:332
#7  0x00007f95533994a4 in start_thread (arg=0x7f95317fa700) at pthread_create.c:456
#8  0x00007f95518ecd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
 
Thread 6 (Thread 0x7f954a2f2700 (LWP 708)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007f9551f5b50c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x00007f954b9860f6 in rocksdb::ThreadPoolImpl::Impl::BGThread (this=this@entry=0x559fdb8d4720, thread_id=thread_id@entry=0) at ./storage/rocksdb/rocksdb/util/threadpool_imp.cc:196
#3  0x00007f954b9864f9 in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper (arg=0x559fdb94c810) at ./storage/rocksdb/rocksdb/util/threadpool_imp.cc:306
#4  0x00007f9551f60e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007f95533994a4 in start_thread (arg=0x7f954a2f2700) at pthread_create.c:456
#6  0x00007f95518ecd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
 
Thread 5 (Thread 0x7f95327fc700 (LWP 725)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1  0x00007f954b9eb59e in rocksdb::port::CondVar::TimedWait (this=this@entry=0x559fdb95eea0, abs_time_us=<optimized out>) at ./storage/rocksdb/rocksdb/port/port_posix.cc:120
#2  0x00007f954b8be672 in rocksdb::InstrumentedCondVar::TimedWaitInternal (this=this@entry=0x559fdb95eea0, abs_time_us=<optimized out>, abs_time_us@entry=1595505541042089) at ./storage/rocksdb/rocksdb/monitoring/instrumented_mutex.cc:66
#3  0x00007f954b8be768 in rocksdb::InstrumentedCondVar::TimedWait (this=this@entry=0x559fdb95eea0, abs_time_us=abs_time_us@entry=1595505541042089) at ./storage/rocksdb/rocksdb/monitoring/instrumented_mutex.cc:55
#4  0x00007f954b76ea4b in rocksdb::RepeatableThread::wait (delay=<optimized out>, this=0x559fdb95ee00) at ./storage/rocksdb/rocksdb/util/repeatable_thread.h:90
#5  rocksdb::RepeatableThread::thread (this=0x559fdb95ee00) at ./storage/rocksdb/rocksdb/util/repeatable_thread.h:117
#6  rocksdb::RepeatableThread::RepeatableThread(std::function<void ()>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, unsigned long, unsigned long)::{lambda()#1}::operator()() const (__closure=<optimized out>) at ./storage/rocksdb/rocksdb/util/repeatable_thread.h:38
#7  std::_Bind_simple<rocksdb::RepeatableThread::RepeatableThread(std::function<void ()>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, unsigned long, unsigned long)::{lambda()#1} ()>::_M_invoke<>(std::_Index_tuple<>) (this=<optimized out>) at /usr/include/c++/6/functional:1391
#8  std::_Bind_simple<rocksdb::RepeatableThread::RepeatableThread(std::function<void ()>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, unsigned long, unsigned long)::{lambda()#1} ()>::operator()() (this=<optimized out>) at /usr/include/c++/6/functional:1380
#9  std::thread::_State_impl<std::_Bind_simple<rocksdb::RepeatableThread::RepeatableThread(std::function<void ()>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, unsigned long, unsigned long)::{lambda()#1} ()> >::_M_run() (this=<optimized out>) at /usr/include/c++/6/thread:197
#10 0x00007f9551f60e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#11 0x00007f95533994a4 in start_thread (arg=0x7f95327fc700) at pthread_create.c:456
#12 0x00007f95518ecd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
 
Thread 4 (Thread 0x7f9553739440 (LWP 394)):
#0  0x00007f95518e38bd in poll () at ../sysdeps/unix/syscall-template.S:84
#1  0x00007f95518fd3be in __poll_chk (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>, fdslen=<optimized out>) at poll_chk.c:27
#2  0x0000559fd8d1d90f in poll (__timeout=-1, __nfds=2, __fds=0x7fff777387e0) at /usr/include/x86_64-linux-gnu/bits/poll2.h:41
#3  handle_connections_sockets () at ./sql/mysqld.cc:6671
#4  0x0000559fd8d1f34d in mysqld_main (argc=<optimized out>, argv=<optimized out>) at ./sql/mysqld.cc:6137
#5  0x00007f95518242e1 in __libc_start_main (main=0x559fd8cfe730 <main(int, char**)>, argc=25, argv=0x7fff77738b88, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fff77738b78) at ../csu/libc-start.c:291
#6  0x0000559fd8d11aea in _start ()
 
Thread 3 (Thread 0x7f954c6ee700 (LWP 705)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1  0x0000559fd94a2a71 in inline_mysql_cond_timedwait (that=0x559fda60e5c0 <COND_timer>, mutex=0x559fda60e600 <LOCK_timer>, src_file=0x559fd9631e48 "/home/buildbot/buildbot/build/mariadb-10.2.33/mysys/thr_timer.c", src_line=292, abstime=0x7f954c6edce0) at ./include/mysql/psi/mysql_thread.h:1182
#2  timer_handler (arg=arg@entry=0x0) at ./mysys/thr_timer.c:292
#3  0x0000559fd9450041 in pfs_spawn_thread (arg=0x559fdb8b4788) at ./storage/perfschema/pfs.cc:1869
#4  0x00007f95533994a4 in start_thread (arg=0x7f954c6ee700) at pthread_create.c:456
#5  0x00007f95518ecd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
 
Thread 2 (Thread 0x7f954b2f4700 (LWP 706)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007f954b9eb51d in rocksdb::port::CondVar::Wait (this=this@entry=0x559fdb91a558) at ./storage/rocksdb/rocksdb/port/port_posix.cc:106
#2  0x00007f954b8be3a4 in rocksdb::InstrumentedCondVar::WaitInternal (this=this@entry=0x559fdb91a558) at ./storage/rocksdb/rocksdb/monitoring/instrumented_mutex.cc:48
#3  0x00007f954b8be460 in rocksdb::InstrumentedCondVar::Wait (this=this@entry=0x559fdb91a558) at ./storage/rocksdb/rocksdb/monitoring/instrumented_mutex.cc:41
#4  0x00007f954b89c768 in rocksdb::DeleteScheduler::BackgroundEmptyTrash (this=0x559fdb91a458) at ./storage/rocksdb/rocksdb/file/delete_scheduler.cc:196
#5  0x00007f9551f60e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007f95533994a4 in start_thread (arg=0x7f954b2f4700) at pthread_create.c:456
#7  0x00007f95518ecd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
 
Thread 1 (Thread 0x7f95484b2700 (LWP 1071)):
#0  __pthread_kill (threadid=<optimized out>, signo=signo@entry=11) at ../sysdeps/unix/sysv/linux/pthread_kill.c:57
#1  0x0000559fd949dfc7 in my_write_core (sig=sig@entry=11) at ./mysys/stacktrace.c:477
#2  0x0000559fd8f559db in handle_fatal_signal (sig=11) at ./sql/signal_handler.cc:343
#3  <signal handler called>
#4  0x0000559fd90240c7 in MYSQL_BIN_LOG::mark_xid_done (this=this@entry=0x559fd9da8ca0 <mysql_bin_log>, binlog_id=44, write_checkpoint=write_checkpoint@entry=true) at ./sql/log.cc:9735
#5  0x0000559fd9024836 in binlog_background_thread (arg=arg@entry=0x0) at ./sql/log.cc:9931
#6  0x0000559fd9450041 in pfs_spawn_thread (arg=0x559fdb996c58) at ./storage/perfschema/pfs.cc:1869
#7  0x00007f95533994a4 in start_thread (arg=0x7f95484b2700) at pthread_create.c:456
#8  0x00007f95518ecd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
 
Compressed file /dev/shm/var/1/log/rocksdb_rpl.rpl_binlog_xid_count-row-write-committed/mysqld.1/data/core
 - saving '/dev/shm/var/1/log/rocksdb_rpl.rpl_binlog_xid_count-row-write-commit
 

10.3 4f26aea51b22eaf65a44d5f4d8587d20

200518  7:18:34 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
Server version: 10.3.24-MariaDB-1:10.3.24+maria~sid-log
key_buffer_size=1048576
read_buffer_size=131072
max_used_connections=1
max_threads=153
thread_count=8
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 63199 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x7f5b78000c48
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f5bdb9fdda8 thread_stack 0x49000
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x56010971167e]
/usr/sbin/mysqld(handle_fatal_signal+0x54d)[0x5601091dbe9d]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x14110)[0x7f5bf8cc2110]
/usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG24commit_checkpoint_notifyEPv+0x50)[0x5601092b6250]
/usr/lib/mysql/plugin/ha_rocksdb.so(+0x1ab2d3)[0x7f5bf34c22d3]
/usr/sbin/mysqld(+0x7b1062)[0x5601091dc062]
/usr/sbin/mysqld(_Z24plugin_foreach_with_maskP3THDPFcS0_P13st_plugin_intPvEijS3_+0x1a6)[0x560108ffd4a6]
/usr/sbin/mysqld(_Z28ha_commit_checkpoint_requestPvPFvS_E+0x2c)[0x5601091dea5c]
/usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG21do_checkpoint_requestEm+0x7f)[0x5601092c464f]
/usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG20checkpoint_and_purgeEm+0x12)[0x5601092c46f2]
/usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG23trx_group_commit_leaderEPNS_18group_commit_entryE+0x51f)[0x5601092c8d4f]
/usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG34write_transaction_to_binlog_eventsEPNS_18group_commit_entryE+0x8b)[0x5601092c920b]
/usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG27write_transaction_to_binlogEP3THDP17binlog_cache_mngrP9Log_eventbbb+0x1d2)[0x5601092c9682]
/usr/sbin/mysqld(+0x89e796)[0x5601092c9796]
/usr/sbin/mysqld(+0x89ea24)[0x5601092c9a24]
/usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG13log_and_orderEP3THDybbb+0x44)[0x5601092c9b94]
/usr/sbin/mysqld(_Z15ha_commit_transP3THDb+0x59b)[0x5601091df61b]
/usr/sbin/mysqld(_Z17trans_commit_stmtP3THD+0x58)[0x5601090e82c8]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0xc52)[0x560108feb4d2]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x1f3)[0x560108ff33b3]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x104d)[0x560108ff575d]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x129)[0x560108ff7789]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x236)[0x5601090daaf6]
/usr/sbin/mysqld(handle_one_connection+0x3b)[0x5601090daccb]
/usr/sbin/mysqld(+0xc96b72)[0x5601096c1b72]
nptl/pthread_create.c:480(start_thread)[0x7f5bf8cb6f27]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f5bf88d431f]
 
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7f5b7800f7a0): INSERT INTO t SET a=repeat('a', 4096)
Connection ID (thread ID): 10
Status: NOT_KILLED

Comment by Alice Sherepa [ 2021-11-03 ]

https://buildbot.askmonty.org/buildbot/builders/kvm-bintar-trusty-amd64/builds/25771/steps/mtr-rocksdb/logs/stdio

Generated at Thu Feb 08 08:30:28 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.