rocksdb_rpl.rpl_binlog_xid_count failed in buildbot, server crashed

rocksdb_rpl.rpl_binlog_xid_count 'row'   w4 [ fail ]

        Test ended at 2018-06-25 11:28:07

CURRENT_TEST: rocksdb_rpl.rpl_binlog_xid_count

mysqltest: At line 17: query 'INSERT INTO t SET a=repeat('a', $size)' failed: 2013: Lost connection to MySQL server during query

The result from queries just before the failure was:

< snip >

INSERT INTO t SET a=repeat('a', 4096/2);

INSERT INTO t SET a=repeat('a', 4096);

INSERT INTO t SET a=repeat('a', 4096/2);

INSERT INTO t SET a=repeat('a', 4096);

INSERT INTO t SET a=repeat('a', 4096/2);

INSERT INTO t SET a=repeat('a', 4096);

INSERT INTO t SET a=repeat('a', 4096/2);

INSERT INTO t SET a=repeat('a', 4096);

INSERT INTO t SET a=repeat('a', 4096/2);

INSERT INTO t SET a=repeat('a', 4096);

INSERT INTO t SET a=repeat('a', 4096/2);

INSERT INTO t SET a=repeat('a', 4096);

INSERT INTO t SET a=repeat('a', 4096/2);

INSERT INTO t SET a=repeat('a', 4096);

INSERT INTO t SET a=repeat('a', 4096/2);

INSERT INTO t SET a=repeat('a', 4096);

INSERT INTO t SET a=repeat('a', 4096/2);

INSERT INTO t SET a=repeat('a', 4096);

INSERT INTO t SET a=repeat('a', 4096/2);

INSERT INTO t SET a=repeat('a', 4096);

More results from queries before failure can be found in /usr/local/mariadb-10.3.8-linux-x86_64/mysql-test/var/4/log/rpl_binlog_xid_count.log

Server [mysqld.1 - pid: 5657, winpid: 5657, exit: 256] failed during test run

Server log from this test:

----------SERVER LOG START-----------

2018-06-25 11:28:01 0 [Note] /usr/local/mariadb-10.3.8-linux-x86_64/bin/mysqld (mysqld 10.3.8-MariaDB-log) starting as process 5658 ...

2018-06-25 11:28:01 0 [Warning] Could not increase number of max_open_files to more than 1024 (request: 4182)

2018-06-25 11:28:01 0 [Note] RocksDB: Got ENOENT when listing column families

2018-06-25 11:28:01 0 [Note] RocksDB:   assuming that we're creating a new database

2018-06-25 11:28:01 0 [Note] RocksDB: Column Families at start:

2018-06-25 11:28:01 0 [Note]   cf=default

2018-06-25 11:28:01 0 [Note]     write_buffer_size=67108864

2018-06-25 11:28:01 0 [Note]     target_file_size_base=67108864

2018-06-25 11:28:01 0 [Note] RocksDB: creating a column family __system__

2018-06-25 11:28:01 0 [Note]     write_buffer_size=67108864

2018-06-25 11:28:01 0 [Note]     target_file_size_base=67108864

2018-06-25 11:28:01 0 [Note] RocksDB: Table_store: loaded DDL data for 0 tables

2018-06-25 11:28:01 0 [Note] RocksDB: global statistics using get_sched_indexer_t indexer

2018-06-25 11:28:01 0 [Note] MyRocks storage engine plugin has been successfully initialized.

2018-06-25 11:28:01 0 [Note] Plugin 'partition' is disabled.

2018-06-25 11:28:01 0 [Note] InnoDB: Using Linux native AIO

2018-06-25 11:28:01 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins

2018-06-25 11:28:01 0 [Note] InnoDB: Uses event mutexes

2018-06-25 11:28:01 0 [Note] InnoDB: Compressed tables use zlib 1.2.3

2018-06-25 11:28:01 0 [Note] InnoDB: Number of pools: 1

2018-06-25 11:28:01 0 [Note] InnoDB: Using generic crc32 instructions

2018-06-25 11:28:01 0 [Note] InnoDB: Initializing buffer pool, total size = 8M, instances = 1, chunk size = 8M

2018-06-25 11:28:01 0 [Note] InnoDB: Completed initialization of buffer pool

2018-06-25 11:28:01 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().

2018-06-25 11:28:02 0 [Note] InnoDB: 128 out of 128 rollback segments are active.

2018-06-25 11:28:02 0 [Note] InnoDB: Creating shared tablespace for temporary tables

2018-06-25 11:28:02 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...

2018-06-25 11:28:02 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.

2018-06-25 11:28:02 0 [Note] InnoDB: Waiting for purge to start

2018-06-25 11:28:02 0 [Note] InnoDB: 10.3.8 started; log sequence number 1642313; transaction id 34

2018-06-25 11:28:02 0 [Note] InnoDB: Loading buffer pool(s) from /run/shm/var_auto_7pQX/4/mysqld.1/data/ib_buffer_pool

2018-06-25 11:28:02 0 [Note] InnoDB: Buffer pool(s) load completed at 180625 11:28:02

2018-06-25 11:28:02 0 [Note] Plugin 'SEQUENCE' is disabled.

2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_SYS_DATAFILES' is disabled.

2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_SYS_TABLESTATS' is disabled.

2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_LOCKS' is disabled.

2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_TABLESPACES_SCRUBBING' is disabled.

2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_CMPMEM' is disabled.

2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_CMP_PER_INDEX' is disabled.

2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_CMP' is disabled.

2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_TABLESPACES_ENCRYPTION' is disabled.

2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_CMP_RESET' is disabled.

2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_SYS_INDEXES' is disabled.

2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_BUFFER_PAGE_LRU' is disabled.

2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_CMPMEM_RESET' is disabled.

2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_SYS_COLUMNS' is disabled.

2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_FT_INDEX_TABLE' is disabled.

2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_CMP_PER_INDEX_RESET' is disabled.

2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_FT_INDEX_CACHE' is disabled.

2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_SYS_FOREIGN_COLS' is disabled.

2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_FT_BEING_DELETED' is disabled.

2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_TRX' is disabled.

2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_SYS_FOREIGN' is disabled.

2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_SYS_TABLES' is disabled.

2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_FT_DEFAULT_STOPWORD' is disabled.

2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_FT_CONFIG' is disabled.

2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_SYS_TABLESPACES' is disabled.

2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_MUTEXES' is disabled.

2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_SYS_SEMAPHORE_WAITS' is disabled.

2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_SYS_FIELDS' is disabled.

2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_METRICS' is disabled.

2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_FT_DELETED' is disabled.

2018-06-25 11:28:02 0 [Note] Plugin 'user_variables' is disabled.

2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_BUFFER_POOL_STATS' is disabled.

2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_LOCK_WAITS' is disabled.

2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_SYS_VIRTUAL' is disabled.

2018-06-25 11:28:02 0 [Note] Plugin 'INNODB_BUFFER_PAGE' is disabled.

2018-06-25 11:28:02 0 [Note] Plugin 'FEEDBACK' is disabled.

2018-06-25 11:28:02 0 [Warning] /usr/local/mariadb-10.3.8-linux-x86_64/bin/mysqld: unknown variable 'loose-feedback-debug-startup-interval=20'

2018-06-25 11:28:02 0 [Warning] /usr/local/mariadb-10.3.8-linux-x86_64/bin/mysqld: unknown variable 'loose-feedback-debug-first-interval=60'

2018-06-25 11:28:02 0 [Warning] /usr/local/mariadb-10.3.8-linux-x86_64/bin/mysqld: unknown variable 'loose-feedback-debug-interval=60'

2018-06-25 11:28:02 0 [Warning] /usr/local/mariadb-10.3.8-linux-x86_64/bin/mysqld: unknown option '--loose-pam-debug'

2018-06-25 11:28:02 0 [Warning] /usr/local/mariadb-10.3.8-linux-x86_64/bin/mysqld: unknown option '--loose-enable-rocksdb_lock_info'

2018-06-25 11:28:02 0 [Warning] /usr/local/mariadb-10.3.8-linux-x86_64/bin/mysqld: unknown variable 'loose-debug-sync-timeout=300'

2018-06-25 11:28:02 0 [Note] Server socket created on IP: '127.0.0.1'.

2018-06-25 11:28:02 0 [Note] /usr/local/mariadb-10.3.8-linux-x86_64/bin/mysqld: ready for connections.

Version: '10.3.8-MariaDB-log'  socket: '/usr/local/mariadb-10.3.8-linux-x86_64/mysql-test/var/tmp/4/mysqld.1.sock'  port: 16060  MariaDB Server

2018-06-25 11:28:02 10 [Note] RocksDB: Manual memtable flush.

180625 11:28:03 [ERROR] mysqld got signal 11 ;

This could be because you hit a bug. It is also possible that this binary

or one of the libraries it was linked against is corrupt, improperly built,

or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help

diagnose the problem, but since we have already crashed, 

something is definitely wrong and this may fail.

Server version: 10.3.8-MariaDB-log

key_buffer_size=1048576

read_buffer_size=131072

max_used_connections=1

max_threads=153

thread_count=8

It is possible that mysqld could use up to 

key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 63163 K  bytes of memory

Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f05d80009a8

Attempting backtrace. You can use the following information to find out

where mysqld died. If you see no messages after this, something went

terribly wrong...

stack_bottom = 0x7f0626423e20 thread_stack 0x49000

/usr/local/mariadb-10.3.8-linux-x86_64/bin/mysqld(my_print_stacktrace+0x2e)[0x7f063e937e6e]

/usr/local/mariadb-10.3.8-linux-x86_64/bin/mysqld(handle_fatal_signal+0x357)[0x7f063e3d6d97]

/lib/x86_64-linux-gnu/libpthread.so.0(+0x10330)[0x7f063cdd7330]

/usr/local/mariadb-10.3.8-linux-x86_64/bin/mysqld(_ZN13MYSQL_BIN_LOG13mark_xid_doneEmb+0x97)[0x7f063e4bf9a7]

/usr/local/mariadb-10.3.8-linux-x86_64/bin/mysqld(binlog_background_thread+0x305)[0x7f063e4bff65]

/usr/local/mariadb-10.3.8-linux-x86_64/bin/mysqld(+0x92afcd)[0x7f063e582fcd]

/lib/x86_64-linux-gnu/libpthread.so.0(+0x8182)[0x7f063cdcf182]

/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f063c4efb2d]

Trying to get some variables.

Some pointers may be invalid and cause the dump to abort.

Query (0x0): 

Connection ID (thread ID): 6

Status: NOT_KILLED

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains

information that should help you find out what is causing the crash.

Writing a core file at /usr/local/mariadb-10.3.8-linux-x86_64/mysql-test/var/4/mysqld.1/data/

----------SERVER LOG END-------------

 - found 'core' (0/0)

Trying 'dbx' to get a backtrace

gdb not found, cannot get the stack trace

Trying 'lldb' to get a backtrace from coredump /usr/local/mariadb-10.3.8-linux-x86_64/mysql-test/var/4/log/rocksdb_rpl.rpl_binlog_xid_count-row/mysqld.1/data/core

Compressed file /usr/local/mariadb-10.3.8-linux-x86_64/mysql-test/var/4/log/rocksdb_rpl.rpl_binlog_xid_count-row/mysqld.1/data/core

 - saving '/usr/local/mariadb-10.3.8-linux-x86_64/mysql-test/var/4/log/rocksdb_rpl.rpl_binlog_xid_count-row/' to '/usr/local/mariadb-10.3.8-linux-x86_64/mysql-test/var/log/rocksdb_rpl.rpl_binlog_xid_count-row/'