innodb.ibuf_not_empty failed in buildbot with Assertion `0' failed in ReadView::check_trx_id_sanity

innodb.ibuf_not_empty 'innodb'           w1 [ fail ]

        Test ended at 2018-10-19 09:28:26

CURRENT_TEST: innodb.ibuf_not_empty

mysqltest: At line 50: query 'check table t1' failed: 2013: Lost connection to MySQL server during query

The result from queries just before the failure was:

< snip >

INDEX(b))

ENGINE=InnoDB STATS_PERSISTENT=0;

SET GLOBAL innodb_change_buffering_debug = 1;

INSERT INTO t1 VALUES(0,'x',1);

INSERT INTO t1 SELECT 0,b,c FROM t1;

INSERT INTO t1 SELECT 0,b,c FROM t1;

INSERT INTO t1 SELECT 0,b,c FROM t1;

INSERT INTO t1 SELECT 0,b,c FROM t1;

INSERT INTO t1 SELECT 0,b,c FROM t1;

INSERT INTO t1 SELECT 0,b,c FROM t1;

INSERT INTO t1 SELECT 0,b,c FROM t1;

INSERT INTO t1 SELECT 0,b,c FROM t1;

INSERT INTO t1 SELECT 0,b,c FROM t1;

INSERT INTO t1 SELECT 0,b,c FROM t1;

INSERT INTO t1 SELECT 0,b,c FROM t1;

INSERT INTO t1 SELECT 0,b,c FROM t1;

check table t1;

More results from queries before failure can be found in /dev/shm/var/1/log/ibuf_not_empty.log

Server [mysqld.1 - pid: 9022, winpid: 9022, exit: 256] failed during test run

Server log from this test:

----------SERVER LOG START-----------

2018-10-19  9:28:21 0 [Note] /usr/sbin/mysqld (initiated by: root[root] @ localhost []): Normal shutdown

2018-10-19  9:28:21 0 [Note] Event Scheduler: Purging the queue. 0 events

2018-10-19  9:28:21 0 [Note] InnoDB: FTS optimize thread exiting.

2018-10-19  9:28:21 0 [Note] InnoDB: Starting shutdown...

2018-10-19  9:28:21 0 [Note] InnoDB: Dumping buffer pool(s) to /dev/shm/var/1/mysqld.1/data/ib_buffer_pool

2018-10-19  9:28:21 0 [Note] InnoDB: Instance 0, restricted to 127 pages due to innodb_buf_pool_dump_pct=25

2018-10-19  9:28:21 0 [Note] InnoDB: Buffer pool(s) dump completed at 181019  9:28:21

2018-10-19  9:28:22 0 [Note] InnoDB: Shutdown completed; log sequence number 11564414; transaction id 10170

2018-10-19  9:28:22 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"

2018-10-19  9:28:22 0 [Note] Debug sync points hit:                   65470

2018-10-19  9:28:22 0 [Note] Debug sync points executed:              0

2018-10-19  9:28:22 0 [Note] Debug sync points max active per thread: 0

2018-10-19  9:28:22 0 [Note] /usr/sbin/mysqld: Shutdown complete

2018-10-19  9:28:24 0 [Note] /usr/sbin/mysqld (mysqld 10.3.11-MariaDB-debug-log) starting as process 9024 ...

2018-10-19  9:28:24 0 [Warning] Could not increase number of max_open_files to more than 1024 (request: 4182)

2018-10-19  9:28:24 0 [Warning] Changed limits: max_open_files: 1024  max_connections: 151 (was 151)  table_cache: 421 (was 2000)

2018-10-19  9:28:24 0 [Note] Plugin 'partition' is disabled.

2018-10-19  9:28:24 0 [Note] InnoDB: Using Linux native AIO

2018-10-19  9:28:24 0 [Note] InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!!

2018-10-19  9:28:24 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins

2018-10-19  9:28:24 0 [Note] InnoDB: Uses event mutexes

2018-10-19  9:28:24 0 [Note] InnoDB: Compressed tables use zlib 1.2.7

2018-10-19  9:28:24 0 [Note] InnoDB: Number of pools: 1

2018-10-19  9:28:24 0 [Note] InnoDB: Using generic crc32 instructions

2018-10-19  9:28:24 0 [Note] InnoDB: Disabling background log and ibuf IO write threads.

2018-10-19  9:28:24 0 [Note] InnoDB: Initializing buffer pool, total size = 8M, instances = 1, chunk size = 8M

2018-10-19  9:28:24 0 [Note] InnoDB: Completed initialization of buffer pool

2018-10-19  9:28:24 0 [Note] InnoDB: innodb_force_recovery=6 skips redo log apply

2018-10-19  9:28:24 0 [Note] InnoDB: 10.3.11 started; log sequence number 0; transaction id 0

2018-10-19  9:28:24 0 [Note] InnoDB: !!! innodb_force_recovery is set to 6 !!!

2018-10-19  9:28:24 0 [Note] Plugin 'SEQUENCE' is disabled.

2018-10-19  9:28:24 0 [Note] Plugin 'INNODB_SYS_DATAFILES' is disabled.

2018-10-19  9:28:24 0 [Note] Plugin 'INNODB_SYS_TABLESTATS' is disabled.

2018-10-19  9:28:24 0 [Note] Plugin 'INNODB_TABLESPACES_SCRUBBING' is disabled.

2018-10-19  9:28:24 0 [Note] Plugin 'INNODB_CMP' is disabled.

2018-10-19  9:28:24 0 [Note] Plugin 'INNODB_FT_DELETED' is disabled.

2018-10-19  9:28:24 0 [Note] Plugin 'INNODB_CMP_RESET' is disabled.

2018-10-19  9:28:24 0 [Note] Plugin 'INNODB_TABLESPACES_ENCRYPTION' is disabled.

2018-10-19  9:28:24 0 [Note] Plugin 'INNODB_CMPMEM_RESET' is disabled.

2018-10-19  9:28:24 0 [Note] Plugin 'FEEDBACK' is disabled.

2018-10-19  9:28:24 0 [Note] Plugin 'INNODB_FT_INDEX_TABLE' is disabled.

2018-10-19  9:28:24 0 [Note] Plugin 'INNODB_CMP_PER_INDEX_RESET' is disabled.

2018-10-19  9:28:24 0 [Note] Plugin 'user_variables' is disabled.

2018-10-19  9:28:24 0 [Note] Plugin 'INNODB_FT_INDEX_CACHE' is disabled.

2018-10-19  9:28:24 0 [Note] Plugin 'INNODB_FT_BEING_DELETED' is disabled.

2018-10-19  9:28:24 0 [Note] Plugin 'INNODB_FT_DEFAULT_STOPWORD' is disabled.

2018-10-19  9:28:24 0 [Note] Plugin 'INNODB_FT_CONFIG' is disabled.

2018-10-19  9:28:24 0 [Note] Plugin 'INNODB_SYS_TABLESPACES' is disabled.

2018-10-19  9:28:24 0 [Note] Plugin 'INNODB_MUTEXES' is disabled.

2018-10-19  9:28:24 0 [Note] Plugin 'INNODB_SYS_SEMAPHORE_WAITS' is disabled.

2018-10-19  9:28:24 0 [Warning] /usr/sbin/mysqld: unknown option '--loose-pam-debug'

2018-10-19  9:28:24 0 [Note] Server socket created on IP: '127.0.0.1'.

2018-10-19  9:28:24 0 [Note] Reading of all Master_info entries succeded

2018-10-19  9:28:24 0 [Note] Added new Master_info '' to hash table

2018-10-19  9:28:24 0 [Note] /usr/sbin/mysqld: ready for connections.

Version: '10.3.11-MariaDB-debug-log'  socket: '/dev/shm/var/tmp/1/mysqld.1.sock'  port: 16060  MariaDB Server

2018-10-19  9:28:24 3 [ERROR] InnoDB: Failed to find tablespace for table `test`.`t1` in the cache. Attempting to load the tablespace with space id 1003

2018-10-19  9:28:24 3 [Warning] InnoDB: Allocated tablespace ID 1003 for test/t1, old maximum was 0

2018-10-19  9:28:24 3 [Warning] InnoDB: A transaction id in a record of table `test`.`t1` is newer than the system-wide maximum.

mysqld: /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.3.11/storage/innobase/trx/trx0sys.cc:67: static void ReadView::check_trx_id_sanity(trx_id_t, const table_name_t&): Assertion `0' failed.

181019  9:28:24 [ERROR] mysqld got signal 6 ;

This could be because you hit a bug. It is also possible that this binary

or one of the libraries it was linked against is corrupt, improperly built,

or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help

diagnose the problem, but since we have already crashed, 

something is definitely wrong and this may fail.

Server version: 10.3.11-MariaDB-debug-log

key_buffer_size=1048576

read_buffer_size=131072

max_used_connections=1

max_threads=153

thread_count=2

It is possible that mysqld could use up to 

key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 63260 K  bytes of memory

Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7fc52c000b00

Attempting backtrace. You can use the following information to find out

where mysqld died. If you see no messages after this, something went

terribly wrong...

stack_bottom = 0x7fc54c5f7d48 thread_stack 0x49000

/usr/sbin/mysqld(my_print_stacktrace+0x2b)[0x558f4d4e7a9f]

/usr/sbin/mysqld(handle_fatal_signal+0x33e)[0x558f4ce1b2fe]

sigaction.c:0(__restore_rt)[0x7fc55770a5e0]

:0(__GI_raise)[0x7fc555c171f7]

:0(__GI_abort)[0x7fc555c188e8]

:0(__assert_fail_base)[0x7fc555c10266]

:0(__GI___assert_fail)[0x7fc555c10312]

/usr/sbin/mysqld(+0xbd4dac)[0x558f4d215dac]

/usr/sbin/mysqld(+0xa4b4b6)[0x558f4d08c4b6]

/usr/sbin/mysqld(+0xb5f8b7)[0x558f4d1a08b7]

trx/trx0sys.cc:67(ReadView::check_trx_id_sanity(unsigned long, table_name_t const&))[0x558f4d164ef3]

include/read0types.h:217(ReadView::changes_visible(unsigned long, table_name_t const&) const)[0x558f4d02d438]

row/row0sel.cc:5146(row_search_mvcc(unsigned char*, page_cur_mode_t, row_prebuilt_t*, unsigned long, unsigned long))[0x558f4ce2558d]

include/row0sel.ic:137(row_scan_index_for_mysql(row_prebuilt_t*, dict_index_t const*, unsigned long*))[0x558f4cd04895]

handler/ha_innodb.cc:14487(ha_innobase::check(THD*, st_ha_check_opt*))[0x558f4cd05cd4]

sql/handler.cc:4199(handler::ha_check(THD*, st_ha_check_opt*))[0x558f4cbff1a7]

sql/sql_admin.cc:803(mysql_admin_table(THD*, TABLE_LIST*, st_ha_check_opt*, char const*, thr_lock_type, bool, bool, unsigned int, int (*)(THD*, TABLE_LIST*, st_ha_check_opt*), int (handler::*)(THD*, st_ha_check_opt*), int (*)(THD*, TABLE_LIST*, st_ha_check_opt*)))[0x558f4cc0090d]

sql/sql_parse.cc:6285(mysql_execute_command(THD*))[0x558f4cc02a53]

sql/sql_parse.cc:8092(mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool))[0x558f4cc04dc1]

sql/sql_parse.cc:1850(dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool))[0x558f4ccf8f40]

sql/sql_connect.cc:1402(do_handle_one_connection(CONNECT*))[0x558f4ccf906d]

sql/sql_connect.cc:1310(handle_one_connection)[0x558f4cffb4aa]

pthread_create.c:0(start_thread)[0x7fc557702e25]

/lib64/libc.so.6(clone+0x6d)[0x7fc555cda34d]

Trying to get some variables.

Some pointers may be invalid and cause the dump to abort.

Query (0x7fc52c016e18): check table t1

Connection ID (thread ID): 3

Status: NOT_KILLED

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on