[MDEV-13644] encryption.innochecksum, innodb.innodb-page_compression_default, failed in bb with Assertion `prev != 0 && next != 0' failed Created: 2017-08-25  Updated: 2022-07-27  Resolved: 2022-07-27

Status: Closed
Project: MariaDB Server
Component/s: Server, Tests
Affects Version/s: 10.2, 10.3
Fix Version/s: 10.2.42, 10.3.33, 10.4.23, 10.5.14, 10.6.6, 10.7.2

Type: Bug Priority: Major
Reporter: Alice Sherepa Assignee: Sergei Golubchik
Resolution: Duplicate Votes: 0
Labels: None

Attachments: HTML File threads    
Issue Links:
Duplicate
duplicates MDEV-26755 innodb.undo_truncate: ilink::assert_l... Closed

 Description   

http://buildbot.askmonty.org/buildbot/builders/kvm-fulltest2/builds/9336/steps/test_2/logs/stdio

encryption.innochecksum '8k,cbc,innodb'  w4 [ fail ]  Found warnings/errors in server log file!
 
        Test ended at 2017-08-24 13:57:32
 
line
 
mysqld: /home/buildbot/buildbot/build/mariadb-10.3.1/sql/sql_list.h:623: void ilink::assert_linked(): Assertion `prev != 0 && next != 0' failed.
 
mysqld: /home/buildbot/buildbot/build/mariadb-10.3.1/sql/sql_list.h:623: void ilink::assert_linked(): Assertion `prev != 0 && next != 0' failed.
 
mysqld: /home/buildbot/buildbot/build/mariadb-10.3.1/sql/sql_list.h:623: void ilink::assert_linked(): Assertion `prev != 0 && next != 0' failed.
 
Attempting backtrace. You can use the following information to find out
 
^ Found warnings in /mnt/buildbot/build/mariadb-10.3.1/mysql-test/var/4/log/mysqld.1.err


innodb.innodb-page_compression_default 'innodb' w2 [ fail ]  Found warnings/errors in server log file!
 
        Test ended at 2017-08-24 14:45:00
 
line
 
mysqld: /home/buildbot/buildbot/build/mariadb-10.3.1/sql/sql_list.h:623: void ilink::assert_linked(): Assertion `prev != 0 && next != 0' failed.
 
Attempting backtrace. You can use the following information to find out
 
mysqld: /home/buildbot/buildbot/build/mariadb-10.3.1/sql/sql_list.h:623: void ilink::assert_linked(): Assertion `prev != 0 && next != 0' failed.
 
^ Found warnings in /mnt/buildbot/build/mariadb-10.3.1/mysql-test/var/2/log/mysqld.1.err



 Comments   
Comment by Elena Stepanova [ 2017-08-25 ]

The failure happens on shutdown:

2017-08-25 13:24:17 0 [Note] /data/bld/10.3/bin/mysqld (root[root] @ localhost []): Normal shutdown
 
 
 
2017-08-25 13:24:17 0 [Note] InnoDB: FTS optimize thread exiting.
 
2017-08-25 13:24:17 0 [Note] Event Scheduler: Purging the queue. 0 events
 
mysqld: /data/src/10.3/sql/sql_list.h:623: void ilink::assert_linked(): Assertion `prev != 0 && next != 0' failed.
 
170825 13:24:45 [ERROR] mysqld got signal 6 ;

Reproducible locally by running with --repeat

worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019
 
encryption.innochecksum '8k,cbc,innodb'  [ pass ]  24706
 
encryption.innochecksum '8k,cbc,innodb'  [ pass ]  25370
 
encryption.innochecksum '8k,cbc,innodb'  [ pass ]  29768
 
encryption.innochecksum '8k,cbc,innodb'  [ pass ]  32904
 
encryption.innochecksum '8k,cbc,innodb'  [ fail ]  Found warnings/errors in server log file!
 
        Test ended at 2017-08-25 13:24:51
 
line
 
mysqld: /data/src/10.3/sql/sql_list.h:623: void ilink::assert_linked(): Assertion `prev != 0 && next != 0' failed.
 
Attempting backtrace. You can use the following information to find out
 
mysqld: /data/src/10.3/sql/sql_list.h:623: void ilink::assert_linked(): Assertion `prev != 0 && next != 0' failed.
 
^ Found warnings in /data/bld/10.3/mysql-test/var/log/mysqld.1.err
 
ok

10.3 77c41fa72512

 
#3  <signal handler called>
 
#4  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
 
#5  0x0000563e0a071e8c in safe_cond_wait (cond=0x563e0ac7d2c0 <COND_thread_count>, mp=0x563e0ac7bfc0 <LOCK_thread_count>, file=0x563e0a0d6440 "/data/src/10.3/include/mysql/psi/mysql_thread.h", line=1138) at /data/src/10.3/mysys/thr_mutex.c:492
 
#6  0x0000563e0954dbb7 in inline_mysql_cond_wait (that=0x563e0ac7d2c0 <COND_thread_count>, mutex=0x563e0ac7bfc0 <LOCK_thread_count>, src_file=0x563e0a0d81da "/data/src/10.3/sql/mysqld.cc", src_line=6128) at /data/src/10.3/include/mysql/psi/mysql_thread.h:1138
 
#7  0x0000563e09558df5 in mysqld_main (argc=144, argv=0x563e0d129370) at /data/src/10.3/sql/mysqld.cc:6128
 
#8  0x0000563e0954d4e0 in main (argc=22, argv=0x7ffc30fd3538) at /data/src/10.3/sql/main.cc:25

Couldn't reproduce on 10.2.

Please also note, that the test (when it passes) takes 4 times longer on 10.3 than on 10.2. There was an addition to the test in 10.3 (extra restart), but I'm not sure it accounts for all extra time:

10.2

 
perl ./mtr encryption.innochecksum,8k,cbc,innodb --mem
 
...
 
encryption.innochecksum '8k,cbc,innodb'  [ pass ]   6509

10.3

 
perl ./mtr encryption.innochecksum,8k,cbc,innodb --mem
 
...
 
encryption.innochecksum '8k,cbc,innodb'  [ pass ]  23778

Comment by Jan Lindström (Inactive) [ 2017-08-31 ]

This does not look InnoDB problem, can you find someone better fitted for this issue?

Comment by Elena Stepanova [ 2017-09-15 ]

The test case below seems to reproduce the failure reliably, although I'm not 100% sure whether it's a valid test (since it uses the artificial debug injection). 

--disable_abort_on_error
 
 
 
connect con1, localhost, root;
 
set global debug_dbug='+d,only_kill_system_threads';
 
send shutdown;
 
connection default;
 
kill connection_id();
 
source include/start_mysqld.inc;

10.2 74f677fcc22

 
2017-09-15 22:12:49 139688852338432 [Note] /data/bld/10.2/bin/mysqld (root[root] @ localhost []): Normal shutdown
 
 
 
2017-09-15 22:12:49 139688852338432 [Note] Event Scheduler: Purging the queue. 0 events
 
2017-09-15 22:13:09 139688852338432 [Warning] /data/bld/10.2/bin/mysqld: Forcing close of thread 5  user: 'root'
 
 
 
mysqld: /data/src/10.2/sql/sql_list.h:665: void ilink::assert_linked(): Assertion `prev != 0 && next != 0' failed.
 
170915 22:13:09 [ERROR] mysqld got signal 6 ;


#7  0x00007f0bde47aee2 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
 
#8  0x0000559516dffb5f in ilink::assert_linked (this=0x7f0bbc000b00) at /data/src/10.2/sql/sql_list.h:665
 
#9  0x0000559516e00d44 in unlink_not_visible_thd (thd=0x7f0bbc000b00) at /data/src/10.2/sql/sql_class.h:4387
 
#10 0x0000559516df2afb in unlink_thd (thd=0x7f0bbc000b00) at /data/src/10.2/sql/mysqld.cc:2907
 
#11 0x0000559516df2f84 in one_thread_per_connection_end (thd=0x7f0bbc000b00, put_in_cache=true) at /data/src/10.2/sql/mysqld.cc:3049
 
#12 0x00005595170193f4 in do_handle_one_connection (connect=0x559519d3fbb0) at /data/src/10.2/sql/sql_connect.cc:1373
 
#13 0x0000559517019098 in handle_one_connection (arg=0x559519d3fbb0) at /data/src/10.2/sql/sql_connect.cc:1260
 
#14 0x0000559517435996 in pfs_spawn_thread (arg=0x559519d4a9a0) at /data/src/10.2/storage/perfschema/pfs.cc:1862
 
#15 0x00007f0be03bf494 in start_thread (arg=0x7f0bd870d700) at pthread_create.c:333
 
#16 0x00007f0bde53793f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Reproducible on 10.2 and 10.3. It's possible that the problem (if it's a problem) also exists in previous versions, but they don't have the debug injection.

Comment by Alice Sherepa [ 2017-10-12 ]

http://buildbot.askmonty.org/buildbot/builders/kvm-fulltest2/builds/9991/steps/test_2/logs/stdio

innodb.innodb-wl5522 'innodb'            w2 [ fail ]  Found warnings/errors in server log file!
 
        Test ended at 2017-10-11 20:45:08
 
line
 
mysqld: /home/buildbot/buildbot/build/mariadb-10.3.3/sql/sql_list.h:623: void ilink::assert_linked(): Assertion `prev != 0 && next != 0' failed.
 
mysqld: /home/buildbot/buildbot/build/mariadb-10.3.3/sql/sql_list.h:623: void ilink::assert_linked(): Assertion `prev != 0 && next != 0' failed.

Comment by Elena Stepanova [ 2018-03-08 ]

Still happens on current 10.2 in my non-MTR tests

10.2 a050189773c911e0b9689aa212c44aaa0cea7b66

 
mysqld: /home/travis/src/sql/sql_list.h:668: void ilink::assert_linked(): Assertion `prev != 0 && next != 0' failed.
 
180308  5:57:45 [ERROR] mysqld got signal 6 ;
 
 
 
Query (0x0): 
Connection ID (thread ID): 14
 
Status: KILL_SERVER


#6  0x00007fc8084bfbf6 in __assert_fail_base (fmt=0x7fc808614058 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x55c4cf057298 "prev != 0 && next != 0", file=file@entry=0x55c4cf057278 "/home/travis/src/sql/sql_list.h", line=line@entry=668, function=function@entry=0x55c4cf061210 <ilink::assert_linked()::__PRETTY_FUNCTION__> "void ilink::assert_linked()") at assert.c:92
 
#7  0x00007fc8084bfca2 in __GI___assert_fail (assertion=0x55c4cf057298 "prev != 0 && next != 0", file=0x55c4cf057278 "/home/travis/src/sql/sql_list.h", line=668, function=0x55c4cf061210 <ilink::assert_linked()::__PRETTY_FUNCTION__> "void ilink::assert_linked()") at assert.c:101
 
#8  0x000055c4ce5173e3 in ilink::assert_linked (this=0x7fc7c0008a50) at /home/travis/src/sql/sql_list.h:668
 
#9  0x000055c4ce5185d2 in unlink_not_visible_thd (thd=0x7fc7c0008a50) at /home/travis/src/sql/sql_class.h:4479
 
#10 0x000055c4ce50a0c5 in unlink_thd (thd=0x7fc7c0008a50) at /home/travis/src/sql/mysqld.cc:2913
 
#11 0x000055c4ce50a54e in one_thread_per_connection_end (thd=0x7fc7c0008a50, put_in_cache=true) at /home/travis/src/sql/mysqld.cc:3055
 
#12 0x000055c4ce7359f3 in do_handle_one_connection (connect=0x55c4d2b596d0) at /home/travis/src/sql/sql_connect.cc:1354
 
#13 0x000055c4ce735697 in handle_one_connection (arg=0x55c4d2b596d0) at /home/travis/src/sql/sql_connect.cc:1241
 
#14 0x00007fc809081184 in start_thread (arg=0x7fc805c92700) at pthread_create.c:312
 
#15 0x00007fc80858e03d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

threads

Comment by Alice Sherepa [ 2018-04-25 ]

also on 10.3 

mysqld: /home/alice/git/10.3/sql/sql_list.h:626: void ilink::assert_linked(): Assertion `prev != 0 && next != 0' failed.
 
180425  5:43:28 [ERROR] mysqld got signal 6 ;
 
 
 
Server version: 10.3.7-MariaDB-debug-log
 
 
 
stack_bottom = 0x7f5c9006bec0 thread_stack 0x49000
 
/home/alice/git/10.3/sql/mysqld(my_print_stacktrace+0x40)[0x55e81aaaa829]
 
mysys/stacktrace.c:269(my_print_stacktrace)[0x55e81a298a6e]
 
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f5cc0afa390]
 
linux/raise.c:54(__GI_raise)[0x7f5cbfeb3428]
 
stdlib/abort.c:91(__GI_abort)[0x7f5cbfeb502a]
 
assert/assert.c:92(__assert_fail_base)[0x7f5cbfeabbd7]
 
/lib/x86_64-linux-gnu/libc.so.6(+0x2dc82)[0x7f5cbfeabc82]
 
sql/sql_list.h:627(ilink::assert_linked())[0x55e819e933ea]
 
sql/sql_class.h:4776(unlink_not_visible_thd(THD*))[0x55e819e948f4]
 
sql/mysqld.cc:2942(unlink_thd(THD*))[0x55e819e858a7]
 
sql/mysqld.cc:3087(one_thread_per_connection_end(THD*, bool))[0x55e819e85da1]
 
sql/sql_connect.cc:1421(do_handle_one_connection(CONNECT*))[0x55e81a10be27]
 
sql/sql_connect.cc:1309(handle_one_connection)[0x55e81a10bacd]
 
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f5cc0af06ba]
 
x86_64/clone.S:111(clone)[0x7f5cbff8541d]

Comment by Elena Stepanova [ 2018-05-04 ]

Here is a test case which doesn't require debug injections and is applicable both to debug and non-debug builds. On debug it causes the assertion failure as reported, on non-debug the server just hangs on shutdown. 

--source include/have_sequence.inc
 
--source include/have_innodb.inc
 
 
 
CREATE TABLE t1 (i INT) ENGINE=InnoDB;
 
INSERT INTO t1 SELECT seq FROM seq_1_to_100000;
 
 
 
--connect (con1,localhost,root,,test)
 
--send
 
  SELECT * FROM t1;
 
 
 
--connection default
 
--sleep 1
 
--source include/restart_mysqld.inc
 
 
 
--echo # All done.
 
 
 
# Cleanup
 
DROP TABLE t1;

10.2 debug da3c5c3c9ad53

 
2018-05-04 23:48:46 139909167965952 [Note] /data/bld/10.2/bin/mysqld (initiated by: root[root] @ localhost []): Normal shutdown
 
2018-05-04 23:48:46 139908870031104 [Note] InnoDB: FTS optimize thread exiting.
 
2018-05-04 23:48:46 139909167965952 [Note] Event Scheduler: Purging the queue. 0 events
 
2018-05-04 23:49:07 139909167965952 [Warning] /data/bld/10.2/bin/mysqld: Forcing close of thread 10  user: 'root'
 
 
 
mysqld: /data/src/10.2/sql/sql_list.h:668: void ilink::assert_linked(): Assertion `prev != 0 && next != 0' failed.
 
180504 23:49:46 [ERROR] mysqld got signal 6 ;
 
 
 
#7  0x00007f3f2c045ee2 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
 
#8  0x0000561cd627687f in ilink::assert_linked (this=0x7f3ec8000b00) at /data/src/10.2/sql/sql_list.h:668
 
#9  0x0000561cd6277b36 in unlink_not_visible_thd (thd=0x7f3ec8000b00) at /data/src/10.2/sql/sql_class.h:4479
 
#10 0x0000561cd626941a in unlink_thd (thd=0x7f3ec8000b00) at /data/src/10.2/sql/mysqld.cc:2913
 
#11 0x0000561cd62698e3 in one_thread_per_connection_end (thd=0x7f3ec8000b00, put_in_cache=true) at /data/src/10.2/sql/mysqld.cc:3059
 
#12 0x0000561cd6494e8d in do_handle_one_connection (connect=0x561cd8c45a60) at /data/src/10.2/sql/sql_connect.cc:1354
 
#13 0x0000561cd6494b31 in handle_one_connection (arg=0x561cd8c45a60) at /data/src/10.2/sql/sql_connect.cc:1241
 
#14 0x0000561cd68b56a8 in pfs_spawn_thread (arg=0x561cd8c642b0) at /data/src/10.2/storage/perfschema/pfs.cc:1862
 
#15 0x00007f3f2dd1c494 in start_thread (arg=0x7f3f24468700) at pthread_create.c:333
 
#16 0x00007f3f2c10293f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Same build, same test case, stack traces from the alive server, before MTR kills it:

Thread 3 (Thread 0x7f079471a700 (LWP 2344)):
 
#0  0x00007f07a3be24ed in poll () from /lib/x86_64-linux-gnu/libc.so.6
 
#1  0x000055be240c81c1 in vio_io_wait (vio=0x55be26414790, event=VIO_IO_EVENT_WRITE, timeout=60000) at /data/src/10.2/vio/viosocket.c:945
 
#2  0x000055be240c69d6 in vio_socket_io_wait (vio=0x55be26414790, event=VIO_IO_EVENT_WRITE) at /data/src/10.2/vio/viosocket.c:108
 
#3  0x000055be240c6ff6 in vio_write (vio=0x55be26414790, buf=0x7f0740008950 "05\006", size=16384) at /data/src/10.2/vio/viosocket.c:294
 
#4  0x000055be235c2e9a in net_real_write (net=0x7f0740000df0, packet=0x7f0740008950 "05\006", len=16384) at /data/src/10.2/sql/net_serv.cc:678
 
#5  0x000055be235c2afb in net_write_buff (net=0x7f0740000df0, packet=0x7f0794717c20 "\006", len=4) at /data/src/10.2/sql/net_serv.cc:570
 
#6  0x000055be235c2717 in my_net_write (net=0x7f0740000df0, packet=0x7f074000ca90 "\005\062\064\060\064\064est\002t1\002t1\001i\001i\f?", len=6) at /data/src/10.2/sql/net_serv.cc:436
 
#7  0x000055be235cae76 in Protocol::write (this=0x7f07400010c0) at /data/src/10.2/sql/protocol.cc:953
 
#8  0x000055be2364d05d in select_send::send_data (this=0x7f07400119f0, items=...) at /data/src/10.2/sql/sql_class.cc:2813
 
#9  0x000055be23706502 in end_send (join=0x7f0740011a10, join_tab=0x7f0740012eb8, end_of_records=false) at /data/src/10.2/sql/sql_select.cc:19861
 
#10 0x000055be23703fde in evaluate_join_record (join=0x7f0740011a10, join_tab=0x7f0740012b08, error=0) at /data/src/10.2/sql/sql_select.cc:18913
 
#11 0x000055be23703a82 in sub_select (join=0x7f0740011a10, join_tab=0x7f0740012b08, end_of_records=false) at /data/src/10.2/sql/sql_select.cc:18732
 
#12 0x000055be23702e81 in do_select (join=0x7f0740011a10, procedure=0x0) at /data/src/10.2/sql/sql_select.cc:18237
 
#13 0x000055be236dd583 in JOIN::exec_inner (this=0x7f0740011a10) at /data/src/10.2/sql/sql_select.cc:3568
 
#14 0x000055be236dca30 in JOIN::exec (this=0x7f0740011a10) at /data/src/10.2/sql/sql_select.cc:3363
 
#15 0x000055be236ddbfb in mysql_select (thd=0x7f0740000b00, tables=0x7f0740011310, wild_num=1, fields=..., conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=2147748608, result=0x7f07400119f0, unit=0x7f07400046a8, select_lex=0x7f0740004de0) at /data/src/10.2/sql/sql_select.cc:3763
 
#16 0x000055be236d2213 in handle_select (thd=0x7f0740000b00, lex=0x7f07400045e0, result=0x7f07400119f0, setup_tables_done_option=0) at /data/src/10.2/sql/sql_select.cc:376
 
#17 0x000055be2369ddc2 in execute_sqlcom_select (thd=0x7f0740000b00, all_tables=0x7f0740011310) at /data/src/10.2/sql/sql_parse.cc:6472
 
#18 0x000055be23693e49 in mysql_execute_command (thd=0x7f0740000b00) at /data/src/10.2/sql/sql_parse.cc:3483
 
#19 0x000055be236a1846 in mysql_parse (thd=0x7f0740000b00, rawbuf=0x7f0740011138 "SELECT * FROM t1", length=16, parser_state=0x7f0794719200, is_com_multi=false, is_next_command=false) at /data/src/10.2/sql/sql_parse.cc:7924
 
#20 0x000055be2368f6db in dispatch_command (command=COM_QUERY, thd=0x7f0740000b00, packet=0x7f0740008951 "5\006", packet_length=16, is_com_multi=false, is_next_command=false) at /data/src/10.2/sql/sql_parse.cc:1820
 
#21 0x000055be2368e03e in do_command (thd=0x7f0740000b00) at /data/src/10.2/sql/sql_parse.cc:1374
 
#22 0x000055be237dcda4 in do_handle_one_connection (connect=0x55be264b5820) at /data/src/10.2/sql/sql_connect.cc:1335
 
#23 0x000055be237dcb31 in handle_one_connection (arg=0x55be264b5820) at /data/src/10.2/sql/sql_connect.cc:1241
 
#24 0x000055be23bfd6a8 in pfs_spawn_thread (arg=0x55be264d4070) at /data/src/10.2/storage/perfschema/pfs.cc:1862
 
#25 0x00007f07a5805494 in start_thread (arg=0x7f079471a700) at pthread_create.c:333
 
#26 0x00007f07a3beb93f in clone () from /lib/x86_64-linux-gnu/libc.so.6
 
 
 
Thread 2 (Thread 0x7f07946d0700 (LWP 2346)):
 
#0  0x00007f07a3be4233 in select () from /lib/x86_64-linux-gnu/libc.so.6
 
#1  0x000055be2405e353 in my_sleep (m_seconds=20000) at /data/src/10.2/mysys/my_sleep.c:29
 
#2  0x000055be235aea9d in close_connections () at /data/src/10.2/sql/mysqld.cc:1763
 
#3  0x000055be235af372 in kill_server (sig_ptr=0x0) at /data/src/10.2/sql/mysqld.cc:2011
 
#4  0x000055be235af3ed in kill_server_thread (arg=0x7f07947f7e00) at /data/src/10.2/sql/mysqld.cc:2043
 
#5  0x000055be23bfd6a8 in pfs_spawn_thread (arg=0x7f075c0013f0) at /data/src/10.2/storage/perfschema/pfs.cc:1862
 
#6  0x00007f07a5805494 in start_thread (arg=0x7f07946d0700) at pthread_create.c:333
 
#7  0x00007f07a3beb93f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Comment by Elena Stepanova [ 2018-06-30 ]

on 10.3

innodb.innodb-page_compression_snappy 'innodb' w1 [ fail ]  Found warnings/errors in server log file!
 
        Test ended at 2018-06-26 16:53:27
 
line
 
mysqld: /home/buildbot/buildbot/build/mariadb-10.3.8/sql/sql_list.h:631: void ilink::assert_linked(): Assertion `prev != 0 && next != 0' failed.
 
Attempting backtrace. You can use the following information to find out
 
mysqld: /home/buildbot/buildbot/build/mariadb-10.3.8/sql/sql_list.h:631: void ilink::assert_linked(): Assertion `prev != 0 && next != 0' failed.
 
^ Found warnings in /mnt/buildbot/build/mariadb-10.3.8/mysql-test/var/1/log/mysqld.1.err

http://buildbot.askmonty.org/buildbot/builders/kvm-fulltest2/builds/13263

10.3 b27ec709350e13c5cdc08dcdcaeb31b1

 
innodb.innodb-wl5522 'innodb'            w3 [ fail ]  Found warnings/errors in server log file!
 
        Test ended at 2018-06-18 23:39:01
 
line
 
mysqld: /home/buildbot/buildbot/build/mariadb-10.3.8/sql/sql_list.h:631: void ilink::assert_linked(): Assertion `prev != 0 && next != 0' failed.
 
mysqld: /home/buildbot/buildbot/build/mariadb-10.3.8/sql/sql_list.h:631: void ilink::assert_linked(): Assertion `prev != 0 && next != 0' failed.
 
Attempting backtrace. You can use the following information to find out
 
^ Found warnings in /mnt/buildbot/build/mariadb-10.3.8/mysql-test/var/3/log/mysqld.1.err
 
ok
 
 
 
 - found 'core' (1/0)
 
 
 
Trying 'dbx' to get a backtrace
 
 
 
Trying 'gdb' to get a backtrace from coredump /mnt/buildbot/build/mariadb-10.3.8/mysql-test/var/3/log/innodb.innodb-wl5522-innodb/mysqld.1/data/core
 
Core generated by '/mnt/buildbot/build/mariadb-10.3.8/sql/mysqld'
 
Output from gdb follows. The first stack trace is from the failing thread.
 
The following stack traces are from all threads (so the failing one is
 
duplicated).
 
--------------------------
 
[New LWP 23683]
 
[New LWP 23690]
 
[New LWP 23657]
 
[New LWP 23689]
 
[New LWP 23635]
 
[New LWP 23654]
 
[New LWP 23662]
 
[New LWP 23688]
 
[New LWP 23693]
 
[New LWP 24479]
 
[New LWP 23672]
 
[New LWP 23682]
 
[New LWP 23674]
 
[New LWP 23660]
 
[New LWP 23673]
 
[New LWP 23679]
 
[New LWP 23661]
 
[New LWP 23659]
 
[New LWP 23658]
 
[New LWP 23656]
 
[New LWP 23691]
 
[New LWP 23655]
 
[New LWP 23680]
 
[New LWP 23681]
 
[Thread debugging using libthread_db enabled]
 
Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1".
 
Core was generated by `/mnt/buildbot/build/mariadb-10.3.8/sql/mysqld --defaults-group-suffix=.1 --defa'.
 
Program terminated with signal SIGABRT, Aborted.
 
#0  0xb7747c31 in __kernel_vsyscall ()
 
[Current thread is 1 (Thread 0xa77fcb40 (LWP 23683))]
 
#0  0xb7747c31 in __kernel_vsyscall ()
 
#1  0xb7025e89 in raise () from /lib/i386-linux-gnu/libc.so.6
 
#2  0xb70273e7 in abort () from /lib/i386-linux-gnu/libc.so.6
 
#3  0xb701ed07 in ?? () from /lib/i386-linux-gnu/libc.so.6
 
#4  0xb701ed8b in __assert_fail () from /lib/i386-linux-gnu/libc.so.6
 
#5  0x8037f398 in ilink::assert_linked (this=0xa6e01330) at /home/buildbot/buildbot/build/mariadb-10.3.8/sql/sql_list.h:631
 
#6  0x803809cb in unlink_not_visible_thd (thd=0xa6e01330) at /home/buildbot/buildbot/build/mariadb-10.3.8/sql/sql_class.h:4828
 
#7  0x80431fd5 in destroy_thd (thd=0xa6e01330) at /home/buildbot/buildbot/build/mariadb-10.3.8/sql/sql_class.cc:4821
 
#8  0x80a62b73 in innobase_destroy_background_thd (thd=0xa6e01330) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/handler/ha_innodb.cc:1652
 
#9  0x80c2c38d in srv_worker_thread (arg=0x0) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/srv/srv0srv.cc:2503
 
#10 0xb73a02b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#11 0xb70e116e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 24 (Thread 0xa87feb40 (LWP 23681)):
 
#0  0xb7747c31 in __kernel_vsyscall ()
 
#1  0xb73a8d32 in __lll_lock_wait () from /lib/i386-linux-gnu/libpthread.so.0
 
#2  0xb73a28be in pthread_mutex_lock () from /lib/i386-linux-gnu/libpthread.so.0
 
#3  0x80ade86b in OSMutex::enter (this=0x838296e8) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/include/sync0types.h:457
 
#4  0x80b32ef8 in OSTrackMutex<GenericPolicy>::enter (this=0x838296e4) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/include/ib0mutex.h:94
 
#5  0x80b325fe in PolicyMutex<OSTrackMutex<GenericPolicy> >::enter (this=0x838296e4, n_spins=30, n_delay=4, name=0x8121b4d0 "/home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/sync/sync0arr.cc", line=336) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/include/ib0mutex.h:592
 
#6  0x80c3877f in sync_array_reserve_cell (arr=0x838296d8, object=0x81735918 <srv_sys+88>, type=1, file=0x812169d4 "/home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/srv/srv0srv.cc", line=918) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/sync/sync0arr.cc:336
 
#7  0x80a5e1e2 in sync_array_get_and_reserve_cell (object=0x81735918 <srv_sys+88>, type=1, file=0x812169d4 "/home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/srv/srv0srv.cc", line=918, cell=0xa87fe198) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/include/sync0arr.ic:76
 
#8  0x80a8dae0 in TTASEventMutex<GenericPolicy>::enter (this=0x81735918 <srv_sys+88>, max_spins=60, max_delay=4, filename=0x812169d4 "/home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/srv/srv0srv.cc", line=918) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/include/ib0mutex.h:460
 
#9  0x80a8bfd8 in PolicyMutex<TTASEventMutex<GenericPolicy> >::enter (this=0x81735918 <srv_sys+88>, n_spins=30, n_delay=4, name=0x812169d4 "/home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/srv/srv0srv.cc", line=918) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/include/ib0mutex.h:592
 
#10 0x80c28515 in srv_resume_thread (slot=0x817359d0 <srv_sys+272>, sig_count=0, wait=true, timeout_usec=0) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/srv/srv0srv.cc:918
 
#11 0x80c2c2fb in srv_worker_thread (arg=0x0) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/srv/srv0srv.cc:2483
 
#12 0xb73a02b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#13 0xb70e116e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 23 (Thread 0xa8fffb40 (LWP 23680)):
 
#0  0x8094a248 in insert_events_waits_history_long (wait=0xb525cacc) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/perfschema/pfs_events_waits.cc:127
 
#1  0x80979585 in end_mutex_wait_v1 (locker=0xa8fff200, rc=0) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/perfschema/pfs.cc:3525
 
#2  0x80a8dbc7 in PolicyMutex<TTASEventMutex<GenericPolicy> >::pfs_end (this=0x81735918 <srv_sys+88>, locker=0xa8fff200, ret=0) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/include/ib0mutex.h:738
 
#3  0x80a8c00c in PolicyMutex<TTASEventMutex<GenericPolicy> >::enter (this=0x81735918 <srv_sys+88>, n_spins=30, n_delay=4, name=0x812169d4 "/home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/srv/srv0srv.cc", line=943) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/include/ib0mutex.h:596
 
#4  0x80c28696 in srv_release_threads (type=SRV_WORKER, n=3) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/srv/srv0srv.cc:943
 
#5  0x80c2cb5b in srv_purge_coordinator_thread (arg=0x0) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/srv/srv0srv.cc:2748
 
#6  0xb73a02b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#7  0xb70e116e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 22 (Thread 0xb17ffb40 (LWP 23655)):
 
#0  0xb7747c31 in __kernel_vsyscall ()
 
#1  0xb73a5e56 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0
 
#2  0x80ed7eee in safe_cond_timedwait (cond=0x82105a40 <COND_checkpoint>, mp=0x821059c0 <LOCK_checkpoint>, abstime=0xb17ff19c, file=0x81169904 "/home/buildbot/buildbot/build/mariadb-10.3.8/include/mysql/psi/mysql_thread.h", line=1204) at /home/buildbot/buildbot/build/mariadb-10.3.8/mysys/thr_mutex.c:546
 
#3  0x809d4426 in inline_mysql_cond_timedwait (that=0x82105a40 <COND_checkpoint>, mutex=0x821059c0 <LOCK_checkpoint>, abstime=0xb17ff19c, src_file=0x81169954 "/home/buildbot/buildbot/build/mariadb-10.3.8/storage/maria/ma_servicethread.c", src_line=116) at /home/buildbot/buildbot/build/mariadb-10.3.8/include/mysql/psi/mysql_thread.h:1204
 
#4  0x809d4969 in my_service_thread_sleep (control=0x8172ccb8 <checkpoint_control>, sleep_time=30000000000) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/maria/ma_servicethread.c:115
 
#5  0x809c8b50 in ma_checkpoint_background (arg=0x1e) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/maria/ma_checkpoint.c:707
 
#6  0x80975d12 in pfs_spawn_thread (arg=0x837c5c40) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/perfschema/pfs.cc:1862
 
#7  0xb73a02b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#8  0xb70e116e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 21 (Thread 0xa53fcb40 (LWP 23691)):
 
#0  0xb71c65a9 in ?? () from /lib/i386-linux-gnu/libgcc_s.so.1
 
#1  0xb7116afe in dl_iterate_phdr () from /lib/i386-linux-gnu/libc.so.6
 
#2  0xb71c753a in _Unwind_Find_FDE () from /lib/i386-linux-gnu/libgcc_s.so.1
 
#3  0xb71c4153 in ?? () from /lib/i386-linux-gnu/libgcc_s.so.1
 
#4  0xb71c5a08 in _Unwind_Backtrace () from /lib/i386-linux-gnu/libgcc_s.so.1
 
#5  0xb70ee6e7 in backtrace () from /lib/i386-linux-gnu/libc.so.6
 
#6  0x80ed2efa in my_print_stacktrace (stack_bottom=0xa53fc17c "\266\037C\200\024\247p\201\024\247p\201h\302?\245\n#\246\200`\b@\254\024\247p\201h\302?\245\374\"\246\200\024\247p\201", thread_stack=299008, silent=0 '\000') at /home/buildbot/buildbot/build/mariadb-10.3.8/mysys/stacktrace.c:269
 
#7  0x807607a0 in handle_fatal_signal (sig=6) at /home/buildbot/buildbot/build/mariadb-10.3.8/sql/signal_handler.cc:166
 
#8  <signal handler called>
 
#9  0xb7747c31 in __kernel_vsyscall ()
 
#10 0xb7025e89 in raise () from /lib/i386-linux-gnu/libc.so.6
 
#11 0xb70273e7 in abort () from /lib/i386-linux-gnu/libc.so.6
 
#12 0xb701ed07 in ?? () from /lib/i386-linux-gnu/libc.so.6
 
#13 0xb701ed8b in __assert_fail () from /lib/i386-linux-gnu/libc.so.6
 
#14 0x8037f398 in ilink::assert_linked (this=0xac400860) at /home/buildbot/buildbot/build/mariadb-10.3.8/sql/sql_list.h:631
 
#15 0x803809cb in unlink_not_visible_thd (thd=0xac400860) at /home/buildbot/buildbot/build/mariadb-10.3.8/sql/sql_class.h:4828
 
#16 0x80431fd5 in destroy_thd (thd=0xac400860) at /home/buildbot/buildbot/build/mariadb-10.3.8/sql/sql_class.cc:4821
 
#17 0x80a6230a in thd_destructor_proxy () at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/handler/ha_innodb.cc:330
 
#18 0x80975d12 in pfs_spawn_thread (arg=0x839d4230) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/perfschema/pfs.cc:1862
 
#19 0xb73a02b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#20 0xb70e116e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 20 (Thread 0xafdc6b40 (LWP 23656)):
 
#0  0xb7713570 in ?? () from /lib/i386-linux-gnu/libaio.so.1
 
#1  0xb77135e6 in io_getevents () from /lib/i386-linux-gnu/libaio.so.1
 
#2  0x80b28d97 in LinuxAIOHandler::collect (this=0xafdc60e0) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/os/os0file.cc:1933
 
#3  0x80b292f1 in LinuxAIOHandler::poll (this=0xafdc60e0, m1=0xafdc6150, m2=0xafdc6154, request=0xafdc6178) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/os/os0file.cc:2078
 
#4  0x80b29455 in os_aio_linux_handler (global_segment=0, m1=0xafdc6150, m2=0xafdc6154, request=0xafdc6178) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/os/os0file.cc:2132
 
#5  0x80b2e3de in os_aio_handler (segment=0, m1=0xafdc6150, m2=0xafdc6154, request=0xafdc6178) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/os/os0file.cc:5658
 
#6  0x80d8d9fa in fil_aio_wait (segment=0) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/fil/fil0fil.cc:4559
 
#7  0x80c2f091 in io_handler_thread (arg=0x82138a20 <n>) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/srv/srv0start.cc:331
 
#8  0xb73a02b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#9  0xb70e116e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 19 (Thread 0xaedc4b40 (LWP 23658)):
 
#0  0xb7713570 in ?? () from /lib/i386-linux-gnu/libaio.so.1
 
#1  0xb77135e6 in io_getevents () from /lib/i386-linux-gnu/libaio.so.1
 
#2  0x80b28d97 in LinuxAIOHandler::collect (this=0xaedc40e0) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/os/os0file.cc:1933
 
#3  0x80b292f1 in LinuxAIOHandler::poll (this=0xaedc40e0, m1=0xaedc4150, m2=0xaedc4154, request=0xaedc4178) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/os/os0file.cc:2078
 
#4  0x80b29455 in os_aio_linux_handler (global_segment=2, m1=0xaedc4150, m2=0xaedc4154, request=0xaedc4178) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/os/os0file.cc:2132
 
#5  0x80b2e3de in os_aio_handler (segment=2, m1=0xaedc4150, m2=0xaedc4154, request=0xaedc4178) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/os/os0file.cc:5658
 
#6  0x80d8d9fa in fil_aio_wait (segment=2) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/fil/fil0fil.cc:4559
 
#7  0x80c2f091 in io_handler_thread (arg=0x82138a28 <n+8>) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/srv/srv0start.cc:331
 
#8  0xb73a02b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#9  0xb70e116e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 18 (Thread 0xae5c3b40 (LWP 23659)):
 
#0  0xb7713570 in ?? () from /lib/i386-linux-gnu/libaio.so.1
 
#1  0xb77135e6 in io_getevents () from /lib/i386-linux-gnu/libaio.so.1
 
#2  0x80b28d97 in LinuxAIOHandler::collect (this=0xae5c30e0) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/os/os0file.cc:1933
 
#3  0x80b292f1 in LinuxAIOHandler::poll (this=0xae5c30e0, m1=0xae5c3150, m2=0xae5c3154, request=0xae5c3178) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/os/os0file.cc:2078
 
#4  0x80b29455 in os_aio_linux_handler (global_segment=3, m1=0xae5c3150, m2=0xae5c3154, request=0xae5c3178) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/os/os0file.cc:2132
 
#5  0x80b2e3de in os_aio_handler (segment=3, m1=0xae5c3150, m2=0xae5c3154, request=0xae5c3178) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/os/os0file.cc:5658
 
#6  0x80d8d9fa in fil_aio_wait (segment=3) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/fil/fil0fil.cc:4559
 
#7  0x80c2f091 in io_handler_thread (arg=0x82138a2c <n+12>) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/srv/srv0start.cc:331
 
#8  0xb73a02b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#9  0xb70e116e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 17 (Thread 0xad5c1b40 (LWP 23661)):
 
#0  0xb7713570 in ?? () from /lib/i386-linux-gnu/libaio.so.1
 
#1  0xb77135e6 in io_getevents () from /lib/i386-linux-gnu/libaio.so.1
 
#2  0x80b28d97 in LinuxAIOHandler::collect (this=0xad5c10e0) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/os/os0file.cc:1933
 
#3  0x80b292f1 in LinuxAIOHandler::poll (this=0xad5c10e0, m1=0xad5c1150, m2=0xad5c1154, request=0xad5c1178) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/os/os0file.cc:2078
 
#4  0x80b29455 in os_aio_linux_handler (global_segment=5, m1=0xad5c1150, m2=0xad5c1154, request=0xad5c1178) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/os/os0file.cc:2132
 
#5  0x80b2e3de in os_aio_handler (segment=5, m1=0xad5c1150, m2=0xad5c1154, request=0xad5c1178) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/os/os0file.cc:5658
 
#6  0x80d8d9fa in fil_aio_wait (segment=5) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/fil/fil0fil.cc:4559
 
#7  0x80c2f091 in io_handler_thread (arg=0x82138a34 <n+20>) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/srv/srv0start.cc:331
 
#8  0xb73a02b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#9  0xb70e116e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 16 (Thread 0xac1fab40 (LWP 23679)):
 
#0  0xb7747c31 in __kernel_vsyscall ()
 
#1  0xb73a98aa in nanosleep () from /lib/i386-linux-gnu/libpthread.so.0
 
#2  0x80b36d5d in os_thread_sleep (tm=1000000) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/os/os0thread.cc:230
 
#3  0x80c2bc28 in srv_master_sleep () at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/srv/srv0srv.cc:2325
 
#4  0x80c2bd67 in srv_master_thread (arg=0x0) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/srv/srv0srv.cc:2366
 
#5  0xb73a02b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#6  0xb70e116e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 15 (Thread 0xab1f8b40 (LWP 23673)):
 
#0  0xb7747c31 in __kernel_vsyscall ()
 
#1  0xb73a5e56 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0
 
#2  0x80b3616a in os_event::timed_wait (this=0x8384dcb8, abstime=0xab1f8120) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/os/os0event.cc:283
 
#3  0x80b364c0 in os_event::wait_time_low (this=0x8384dcb8, time_in_usec=1000000, reset_sig_count=1) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/os/os0event.cc:405
 
#4  0x80b366cc in os_event_wait_time_low (event=0x8384dcb8, time_in_usec=1000000, reset_sig_count=1) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/os/os0event.cc:505
 
#5  0x80c2a74d in srv_error_monitor_thread () at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/srv/srv0srv.cc:1853
 
#6  0xb73a02b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#7  0xb70e116e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 14 (Thread 0xaddc2b40 (LWP 23660)):
 
#0  0xb7713570 in ?? () from /lib/i386-linux-gnu/libaio.so.1
 
#1  0xb77135e6 in io_getevents () from /lib/i386-linux-gnu/libaio.so.1
 
#2  0x80b28d97 in LinuxAIOHandler::collect (this=0xaddc20e0) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/os/os0file.cc:1933
 
#3  0x80b292f1 in LinuxAIOHandler::poll (this=0xaddc20e0, m1=0xaddc2150, m2=0xaddc2154, request=0xaddc2178) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/os/os0file.cc:2078
 
#4  0x80b29455 in os_aio_linux_handler (global_segment=4, m1=0xaddc2150, m2=0xaddc2154, request=0xaddc2178) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/os/os0file.cc:2132
 
#5  0x80b2e3de in os_aio_handler (segment=4, m1=0xaddc2150, m2=0xaddc2154, request=0xaddc2178) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/os/os0file.cc:5658
 
#6  0x80d8d9fa in fil_aio_wait (segment=4) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/fil/fil0fil.cc:4559
 
#7  0x80c2f091 in io_handler_thread (arg=0x82138a30 <n+16>) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/srv/srv0start.cc:331
 
#8  0xb73a02b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#9  0xb70e116e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 13 (Thread 0xaa9f7b40 (LWP 23674)):
 
#0  0xb7747c31 in __kernel_vsyscall ()
 
#1  0xb73a5e56 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0
 
#2  0x80b3616a in os_event::timed_wait (this=0x8384dd18, abstime=0xaa9f7210) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/os/os0event.cc:283
 
#3  0x80b364c0 in os_event::wait_time_low (this=0x8384dd18, time_in_usec=5000000, reset_sig_count=1) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/os/os0event.cc:405
 
#4  0x80b366cc in os_event_wait_time_low (event=0x8384dd18, time_in_usec=5000000, reset_sig_count=1) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/os/os0event.cc:505
 
#5  0x80c2a1f3 in srv_monitor_thread () at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/srv/srv0srv.cc:1689
 
#6  0xb73a02b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#7  0xb70e116e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 12 (Thread 0xa7ffdb40 (LWP 23682)):
 
#0  0xb7747c31 in __kernel_vsyscall ()
 
#1  0xb73a8d32 in __lll_lock_wait () from /lib/i386-linux-gnu/libpthread.so.0
 
#2  0xb73a28be in pthread_mutex_lock () from /lib/i386-linux-gnu/libpthread.so.0
 
#3  0x80ade86b in OSMutex::enter (this=0x838296e8) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/include/sync0types.h:457
 
#4  0x80b32ef8 in OSTrackMutex<GenericPolicy>::enter (this=0x838296e4) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/include/ib0mutex.h:94
 
#5  0x80b325fe in PolicyMutex<OSTrackMutex<GenericPolicy> >::enter (this=0x838296e4, n_spins=30, n_delay=4, name=0x8121b4d0 "/home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/sync/sync0arr.cc", line=336) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/include/ib0mutex.h:592
 
#6  0x80c3877f in sync_array_reserve_cell (arr=0x838296d8, object=0x81735918 <srv_sys+88>, type=1, file=0x812169d4 "/home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/srv/srv0srv.cc", line=918) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/sync/sync0arr.cc:336
 
#7  0x80a5e1e2 in sync_array_get_and_reserve_cell (object=0x81735918 <srv_sys+88>, type=1, file=0x812169d4 "/home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/srv/srv0srv.cc", line=918, cell=0xa7ffd198) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/include/sync0arr.ic:76
 
#8  0x80a8dae0 in TTASEventMutex<GenericPolicy>::enter (this=0x81735918 <srv_sys+88>, max_spins=60, max_delay=4, filename=0x812169d4 "/home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/srv/srv0srv.cc", line=918) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/include/ib0mutex.h:460
 
#9  0x80a8bfd8 in PolicyMutex<TTASEventMutex<GenericPolicy> >::enter (this=0x81735918 <srv_sys+88>, n_spins=30, n_delay=4, name=0x812169d4 "/home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/srv/srv0srv.cc", line=918) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/include/ib0mutex.h:592
 
#10 0x80c28515 in srv_resume_thread (slot=0x817359b4 <srv_sys+244>, sig_count=0, wait=true, timeout_usec=0) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/srv/srv0srv.cc:918
 
#11 0x80c2c2fb in srv_worker_thread (arg=0x0) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/srv/srv0srv.cc:2483
 
#12 0xb73a02b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#13 0xb70e116e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 11 (Thread 0xab9f9b40 (LWP 23672)):
 
#0  0xb7747c31 in __kernel_vsyscall ()
 
#1  0xb73a5e56 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0
 
#2  0x80b3616a in os_event::timed_wait (this=0x83975588, abstime=0xab9f9220) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/os/os0event.cc:283
 
#3  0x80b364c0 in os_event::wait_time_low (this=0x83975588, time_in_usec=1000000, reset_sig_count=1) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/os/os0event.cc:405
 
#4  0x80b366cc in os_event_wait_time_low (event=0x83975588, time_in_usec=1000000, reset_sig_count=1) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/os/os0event.cc:505
 
#5  0x80b01423 in lock_wait_timeout_thread () at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/lock/lock0wait.cc:543
 
#6  0xb73a02b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#7  0xb70e116e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 10 (Thread 0xaffc6b40 (LWP 24479)):
 
#0  0xb7747c31 in __kernel_vsyscall ()
 
#1  0xb73a5a8c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0
 
#2  0x80ed7c43 in safe_cond_wait (cond=0x818dc760 <COND_thread_count>, mp=0x818db9c0 <LOCK_thread_count>, file=0x80f5c900 "/home/buildbot/buildbot/build/mariadb-10.3.8/include/mysql/psi/mysql_thread.h", line=1167) at /home/buildbot/buildbot/build/mariadb-10.3.8/mysys/thr_mutex.c:492
 
#3  0x8036ca6e in inline_mysql_cond_wait (that=0x818dc760 <COND_thread_count>, mutex=0x818db9c0 <LOCK_thread_count>, src_file=0x80f5e8f4 "/home/buildbot/buildbot/build/mariadb-10.3.8/sql/mysqld.cc", src_line=1850) at /home/buildbot/buildbot/build/mariadb-10.3.8/include/mysql/psi/mysql_thread.h:1167
 
#4  0x8036f1fa in close_connections () at /home/buildbot/buildbot/build/mariadb-10.3.8/sql/mysqld.cc:1850
 
#5  0x8036f6cb in kill_server (sig_ptr=0x0) at /home/buildbot/buildbot/build/mariadb-10.3.8/sql/mysqld.cc:2034
 
#6  0x8036f74e in kill_server_thread (arg=0xb0fee1b0) at /home/buildbot/buildbot/build/mariadb-10.3.8/sql/mysqld.cc:2066
 
#7  0x80975d12 in pfs_spawn_thread (arg=0xb0e01878) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/perfschema/pfs.cc:1862
 
#8  0xb73a02b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#9  0xb70e116e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 9 (Thread 0xb0feeb40 (LWP 23693)):
 
#0  0xb7747c31 in __kernel_vsyscall ()
 
#1  0xb73aa1cd in do_sigwait () from /lib/i386-linux-gnu/libpthread.so.0
 
#2  0xb73aa25e in sigwait () from /lib/i386-linux-gnu/libpthread.so.0
 
#3  0x8037290b in signal_hand (arg=0x0) at /home/buildbot/buildbot/build/mariadb-10.3.8/sql/mysqld.cc:3553
 
#4  0x80975d12 in pfs_spawn_thread (arg=0x839ed4d8) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/perfschema/pfs.cc:1862
 
#5  0xb73a02b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#6  0xb70e116e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 8 (Thread 0xa6bffb40 (LWP 23688)):
 
#0  0xb7747c31 in __kernel_vsyscall ()
 
#1  0xb73a5a8c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0
 
#2  0x80b369ae in os_event::wait (this=0x8384dd78) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/os/os0event.cc:163
 
#3  0x80b3633d in os_event::wait_low (this=0x8384dd78, reset_sig_count=1) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/os/os0event.cc:333
 
#4  0x80b366fe in os_event_wait_low (event=0x8384dd78, reset_sig_count=0) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/os/os0event.cc:522
 
#5  0x80d0b778 in buf_dump_thread () at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/buf/buf0dump.cc:837
 
#6  0xb73a02b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#7  0xb70e116e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 7 (Thread 0xacdc0b40 (LWP 23662)):
 
#0  0xb7747c31 in __kernel_vsyscall ()
 
#1  0xb73a5e56 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0
 
#2  0x80b3616a in os_event::timed_wait (this=0x8384ddd8, abstime=0xacdc0090) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/os/os0event.cc:283
 
#3  0x80b364c0 in os_event::wait_time_low (this=0x8384ddd8, time_in_usec=986000, reset_sig_count=2) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/os/os0event.cc:405
 
#4  0x80b366cc in os_event_wait_time_low (event=0x8384ddd8, time_in_usec=986000, reset_sig_count=2) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/os/os0event.cc:505
 
#5  0x80d166b0 in pc_sleep_if_needed (next_loop_time=371063796, sig_count=2, cur_time=371062810) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/buf/buf0flu.cc:2688
 
#6  0x80d1760e in buf_flush_page_cleaner_coordinator () at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/buf/buf0flu.cc:3147
 
#7  0xb73a02b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#8  0xb70e116e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 6 (Thread 0xb2c21b40 (LWP 23654)):
 
#0  0xb7747c31 in __kernel_vsyscall ()
 
#1  0xb73a5e56 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0
 
#2  0x80ed7eee in safe_cond_timedwait (cond=0x8214d3a0 <COND_timer>, mp=0x8214d320 <LOCK_timer>, abstime=0xb2c21250, file=0x812df07c "/home/buildbot/buildbot/build/mariadb-10.3.8/include/mysql/psi/mysql_thread.h", line=1204) at /home/buildbot/buildbot/build/mariadb-10.3.8/mysys/thr_mutex.c:546
 
#3  0x80ed8dfb in inline_mysql_cond_timedwait (that=0x8214d3a0 <COND_timer>, mutex=0x8214d320 <LOCK_timer>, abstime=0xb2c21250, src_file=0x812df0cc "/home/buildbot/buildbot/build/mariadb-10.3.8/mysys/thr_timer.c", src_line=292) at /home/buildbot/buildbot/build/mariadb-10.3.8/include/mysql/psi/mysql_thread.h:1204
 
#4  0x80ed996b in timer_handler (arg=0x0) at /home/buildbot/buildbot/build/mariadb-10.3.8/mysys/thr_timer.c:292
 
#5  0x80975d12 in pfs_spawn_thread (arg=0x8377a4b0) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/perfschema/pfs.cc:1862
 
#6  0xb73a02b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#7  0xb70e116e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 5 (Thread 0xb6e677c0 (LWP 23635)):
 
#0  0xb7747c31 in __kernel_vsyscall ()
 
#1  0xb73a5a8c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0
 
#2  0x80ed7c43 in safe_cond_wait (cond=0x818dc760 <COND_thread_count>, mp=0x818db9c0 <LOCK_thread_count>, file=0x80f5c900 "/home/buildbot/buildbot/build/mariadb-10.3.8/include/mysql/psi/mysql_thread.h", line=1167) at /home/buildbot/buildbot/build/mariadb-10.3.8/mysys/thr_mutex.c:492
 
#3  0x8036ca6e in inline_mysql_cond_wait (that=0x818dc760 <COND_thread_count>, mutex=0x818db9c0 <LOCK_thread_count>, src_file=0x80f5e8f4 "/home/buildbot/buildbot/build/mariadb-10.3.8/sql/mysqld.cc", src_line=6202) at /home/buildbot/buildbot/build/mariadb-10.3.8/include/mysql/psi/mysql_thread.h:1167
 
#4  0x80378498 in mysqld_main (argc=146, argv=0x83638058) at /home/buildbot/buildbot/build/mariadb-10.3.8/sql/mysqld.cc:6202
 
#5  0x8036c2bb in main (argc=23, argv=0xbfe85094) at /home/buildbot/buildbot/build/mariadb-10.3.8/sql/main.cc:25
 
 
 
Thread 4 (Thread 0xa63feb40 (LWP 23689)):
 
#0  0xb7747c31 in __kernel_vsyscall ()
 
#1  0xb73a98aa in nanosleep () from /lib/i386-linux-gnu/libpthread.so.0
 
#2  0x80b36d5d in os_thread_sleep (tm=1000000) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/os/os0thread.cc:230
 
#3  0x80ce64a1 in btr_defragment_thread () at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/btr/btr0defragment.cc:705
 
#4  0xb73a02b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#5  0xb70e116e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 3 (Thread 0xaf5c5b40 (LWP 23657)):
 
#0  0xb7713570 in ?? () from /lib/i386-linux-gnu/libaio.so.1
 
#1  0xb77135e6 in io_getevents () from /lib/i386-linux-gnu/libaio.so.1
 
#2  0x80b28d97 in LinuxAIOHandler::collect (this=0xaf5c50e0) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/os/os0file.cc:1933
 
#3  0x80b292f1 in LinuxAIOHandler::poll (this=0xaf5c50e0, m1=0xaf5c5150, m2=0xaf5c5154, request=0xaf5c5178) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/os/os0file.cc:2078
 
#4  0x80b29455 in os_aio_linux_handler (global_segment=1, m1=0xaf5c5150, m2=0xaf5c5154, request=0xaf5c5178) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/os/os0file.cc:2132
 
#5  0x80b2e3de in os_aio_handler (segment=1, m1=0xaf5c5150, m2=0xaf5c5154, request=0xaf5c5178) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/os/os0file.cc:5658
 
#6  0x80d8d9fa in fil_aio_wait (segment=1) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/fil/fil0fil.cc:4559
 
#7  0x80c2f091 in io_handler_thread (arg=0x82138a24 <n+4>) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/srv/srv0start.cc:331
 
#8  0xb73a02b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#9  0xb70e116e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 2 (Thread 0xa5bfdb40 (LWP 23690)):
 
#0  0xb7747c31 in __kernel_vsyscall ()
 
#1  0xb73a5a8c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0
 
#2  0x80b369ae in os_event::wait (this=0x8384de38) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/os/os0event.cc:163
 
#3  0x80b3633d in os_event::wait_low (this=0x8384de38, reset_sig_count=1) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/os/os0event.cc:333
 
#4  0x80b366fe in os_event_wait_low (event=0x8384de38, reset_sig_count=0) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/os/os0event.cc:522
 
#5  0x80cf4be0 in buf_resize_thread () at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/buf/buf0buf.cc:3259
 
#6  0xb73a02b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#7  0xb70e116e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 1 (Thread 0xa77fcb40 (LWP 23683)):
 
#0  0xb7747c31 in __kernel_vsyscall ()
 
#1  0xb7025e89 in raise () from /lib/i386-linux-gnu/libc.so.6
 
#2  0xb70273e7 in abort () from /lib/i386-linux-gnu/libc.so.6
 
#3  0xb701ed07 in ?? () from /lib/i386-linux-gnu/libc.so.6
 
#4  0xb701ed8b in __assert_fail () from /lib/i386-linux-gnu/libc.so.6
 
#5  0x8037f398 in ilink::assert_linked (this=0xa6e01330) at /home/buildbot/buildbot/build/mariadb-10.3.8/sql/sql_list.h:631
 
#6  0x803809cb in unlink_not_visible_thd (thd=0xa6e01330) at /home/buildbot/buildbot/build/mariadb-10.3.8/sql/sql_class.h:4828
 
#7  0x80431fd5 in destroy_thd (thd=0xa6e01330) at /home/buildbot/buildbot/build/mariadb-10.3.8/sql/sql_class.cc:4821
 
#8  0x80a62b73 in innobase_destroy_background_thd (thd=0xa6e01330) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/handler/ha_innodb.cc:1652
 
#9  0x80c2c38d in srv_worker_thread (arg=0x0) at /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/srv/srv0srv.cc:2503
 
#10 0xb73a02b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#11 0xb70e116e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Compressed file /mnt/buildbot/build/mariadb-10.3.8/mysql-test/var/3/log/innodb.innodb-wl5522-innodb/mysqld.1/data/core
 
 - skipping '/mnt/buildbot/build/mariadb-10.3.8/mysql-test/var/3/log/innodb.innodb-wl5522-innodb/'

Comment by Elena Stepanova [ 2018-10-01 ]

on 10.3: http://buildbot.askmonty.org/buildbot/builders/kvm-fulltest2/builds/13952

innodb.innodb-page_compression_tables 'innodb' w3 [ fail ]  Found warnings/errors in server log file!
 
        Test ended at 2018-08-16 17:34:09
 
line
 
mysqld: /home/buildbot/buildbot/build/mariadb-10.3.10/sql/sql_list.h:634: void ilink::assert_linked(): Assertion `prev != 0 && next != 0' failed.
 
mysqld: /home/buildbot/buildbot/build/mariadb-10.3.10/sql/sql_list.h:634: void ilink::assert_linked(): Assertion `prev != 0 && next != 0' failed.
 
mysqld: /home/buildbot/buildbot/build/mariadb-10.3.10/sql/sql_list.h:634: void ilink::assert_linked(): Assertion `prev != 0 && next != 0' failed.
 
Attempting backtrace. You can use the following information to find out
 
^ Found warnings in /mnt/buildbot/build/mariadb-10.3.10/mysql-test/var/3/log/mysqld.1.err
 
ok
 
 
 
 - found 'core' (3/0)
 
 
 
Trying 'dbx' to get a backtrace
 
worker[3] > Restart  - not started
 
 
 
Trying 'gdb' to get a backtrace from coredump /mnt/buildbot/build/mariadb-10.3.10/mysql-test/var/3/log/innodb.innodb-page_compression_tables-innodb/mysqld.1/data/core
 
Core generated by '/mnt/buildbot/build/mariadb-10.3.10/sql/mysqld'
 
Output from gdb follows. The first stack trace is from the failing thread.
 
The following stack traces are from all threads (so the failing one is
 
duplicated).
 
--------------------------
 
[New LWP 14313]
 
[New LWP 14316]
 
[New LWP 14317]
 
[New LWP 14294]
 
[New LWP 14304]
 
[New LWP 14295]
 
[New LWP 14301]
 
[New LWP 14298]
 
[New LWP 14515]
 
[New LWP 14299]
 
[New LWP 14311]
 
[New LWP 14310]
 
[New LWP 14319]
 
[New LWP 14314]
 
[New LWP 14315]
 
[New LWP 14297]
 
[New LWP 14312]
 
[New LWP 14296]
 
[New LWP 14305]
 
[New LWP 14306]
 
[New LWP 14300]
 
[New LWP 14288]
 
[New LWP 14292]
 
[New LWP 14318]
 
[Thread debugging using libthread_db enabled]
 
Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1".
 
Core was generated by `/mnt/buildbot/build/mariadb-10.3.10/sql/mysqld --defaults-group-suffix=.1 --def'.
 
Program terminated with signal SIGABRT, Aborted.
 
#0  0xb77d9c31 in __kernel_vsyscall ()
 
[Current thread is 1 (Thread 0xa80fdb40 (LWP 14313))]
 
#0  0xb77d9c31 in __kernel_vsyscall ()
 
#1  0xb70b7e89 in raise () from /lib/i386-linux-gnu/libc.so.6
 
#2  0xb70b93e7 in abort () from /lib/i386-linux-gnu/libc.so.6
 
#3  0xb70b0d07 in ?? () from /lib/i386-linux-gnu/libc.so.6
 
#4  0xb70b0d8b in __assert_fail () from /lib/i386-linux-gnu/libc.so.6
 
#5  0x8033c8f4 in ilink::assert_linked (this=0xa6d13ce0) at /home/buildbot/buildbot/build/mariadb-10.3.10/sql/sql_list.h:634
 
#6  0x8033df1d in unlink_not_visible_thd (thd=0xa6d13ce0) at /home/buildbot/buildbot/build/mariadb-10.3.10/sql/sql_class.h:4848
 
#7  0x803ef23c in destroy_thd (thd=0xa6d13ce0) at /home/buildbot/buildbot/build/mariadb-10.3.10/sql/sql_class.cc:4719
 
#8  0x80a22e7c in innobase_destroy_background_thd (thd=0xa6d13ce0) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/handler/ha_innodb.cc:1652
 
#9  0x80beef81 in srv_worker_thread (arg=0x0) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/srv/srv0srv.cc:2503
 
#10 0xb74322b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#11 0xb717316e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 24 (Thread 0xa54fcb40 (LWP 14318)):
 
#0  0xb71a8cec in _dl_addr () from /lib/i386-linux-gnu/libc.so.6
 
#1  0xb744a47d in dladdr () from /lib/i386-linux-gnu/libdl.so.2
 
#2  0x80eb1441 in my_addr_resolve (ptr=0x80e95342 <my_print_stacktrace+60>, loc=0xa54fb6e4) at /home/buildbot/buildbot/build/mariadb-10.3.10/mysys/my_addr_resolve.c:212
 
#3  0x80e9527d in print_with_addr_resolve (addrs=0xa54fb730, n=15) at /home/buildbot/buildbot/build/mariadb-10.3.10/mysys/stacktrace.c:254
 
#4  0x80e95378 in my_print_stacktrace (stack_bottom=0xa54fc17c "\035\362>\200\064\347l\201\064\347l\201h\302O\245\023&\242\200\310\006\020\251\064\347l\201h\302O\245\005&\242\200\064\347l\201", thread_stack=299008, silent=0 '\000') at /home/buildbot/buildbot/build/mariadb-10.3.10/mysys/stacktrace.c:273
 
#5  0x8071fdd2 in handle_fatal_signal (sig=6) at /home/buildbot/buildbot/build/mariadb-10.3.10/sql/signal_handler.cc:166
 
#6  <signal handler called>
 
#7  0xb77d9c31 in __kernel_vsyscall ()
 
#8  0xb70b7e89 in raise () from /lib/i386-linux-gnu/libc.so.6
 
#9  0xb70b93e7 in abort () from /lib/i386-linux-gnu/libc.so.6
 
#10 0xb70b0d07 in ?? () from /lib/i386-linux-gnu/libc.so.6
 
#11 0xb70b0d8b in __assert_fail () from /lib/i386-linux-gnu/libc.so.6
 
#12 0x8033c8f4 in ilink::assert_linked (this=0xa91006c8) at /home/buildbot/buildbot/build/mariadb-10.3.10/sql/sql_list.h:634
 
#13 0x8033df1d in unlink_not_visible_thd (thd=0xa91006c8) at /home/buildbot/buildbot/build/mariadb-10.3.10/sql/sql_class.h:4848
 
#14 0x803ef23c in destroy_thd (thd=0xa91006c8) at /home/buildbot/buildbot/build/mariadb-10.3.10/sql/sql_class.cc:4719
 
#15 0x80a22613 in thd_destructor_proxy () at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/handler/ha_innodb.cc:330
 
#16 0x809356e0 in pfs_spawn_thread (arg=0x82d4e998) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/perfschema/pfs.cc:1862
 
#17 0xb74322b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#18 0xb717316e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 23 (Thread 0xb2cb3b40 (LWP 14292)):
 
#0  0xb77d9c31 in __kernel_vsyscall ()
 
#1  0xb7437e56 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0
 
#2  0x80e9a336 in safe_cond_timedwait (cond=0x82111480 <COND_timer>, mp=0x82111400 <LOCK_timer>, abstime=0xb2cb3250, file=0x812a3698 "/home/buildbot/buildbot/build/mariadb-10.3.10/include/mysql/psi/mysql_thread.h", line=1204) at /home/buildbot/buildbot/build/mariadb-10.3.10/mysys/thr_mutex.c:546
 
#3  0x80e9b243 in inline_mysql_cond_timedwait (that=0x82111480 <COND_timer>, mutex=0x82111400 <LOCK_timer>, abstime=0xb2cb3250, src_file=0x812a36e8 "/home/buildbot/buildbot/build/mariadb-10.3.10/mysys/thr_timer.c", src_line=292) at /home/buildbot/buildbot/build/mariadb-10.3.10/include/mysql/psi/mysql_thread.h:1204
 
#4  0x80e9bdb3 in timer_handler (arg=0x0) at /home/buildbot/buildbot/build/mariadb-10.3.10/mysys/thr_timer.c:292
 
#5  0x809356e0 in pfs_spawn_thread (arg=0x82b00500) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/perfschema/pfs.cc:1862
 
#6  0xb74322b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#7  0xb717316e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 22 (Thread 0xb6ef97c0 (LWP 14288)):
 
#0  0xb77d9c31 in __kernel_vsyscall ()
 
#1  0xb7437a8c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0
 
#2  0x80e9a08b in safe_cond_wait (cond=0x818a0760 <COND_thread_count>, mp=0x8189f9c0 <LOCK_thread_count>, file=0x80f1eb44 "/home/buildbot/buildbot/build/mariadb-10.3.10/include/mysql/psi/mysql_thread.h", line=1167) at /home/buildbot/buildbot/build/mariadb-10.3.10/mysys/thr_mutex.c:492
 
#3  0x80329e2e in inline_mysql_cond_wait (that=0x818a0760 <COND_thread_count>, mutex=0x8189f9c0 <LOCK_thread_count>, src_file=0x80f20b34 "/home/buildbot/buildbot/build/mariadb-10.3.10/sql/mysqld.cc", src_line=6208) at /home/buildbot/buildbot/build/mariadb-10.3.10/include/mysql/psi/mysql_thread.h:1167
 
#4  0x80335922 in mysqld_main (argc=146, argv=0x829be088) at /home/buildbot/buildbot/build/mariadb-10.3.10/sql/mysqld.cc:6208
 
#5  0x8032965b in main (argc=23, argv=0xbffa0d54) at /home/buildbot/buildbot/build/mariadb-10.3.10/sql/main.cc:25
 
 
 
Thread 21 (Thread 0xad6c1b40 (LWP 14300)):
 
#0  0xb77a5570 in ?? () from /lib/i386-linux-gnu/libaio.so.1
 
#1  0xb77a55e6 in io_getevents () from /lib/i386-linux-gnu/libaio.so.1
 
#2  0x80aea46f in LinuxAIOHandler::collect (this=0xad6c10e0) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/os/os0file.cc:1933
 
#3  0x80aea9c9 in LinuxAIOHandler::poll (this=0xad6c10e0, m1=0xad6c1150, m2=0xad6c1154, request=0xad6c1178) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/os/os0file.cc:2078
 
#4  0x80aeab2d in os_aio_linux_handler (global_segment=5, m1=0xad6c1150, m2=0xad6c1154, request=0xad6c1178) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/os/os0file.cc:2132
 
#5  0x80aefa34 in os_aio_handler (segment=5, m1=0xad6c1150, m2=0xad6c1154, request=0xad6c1178) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/os/os0file.cc:5683
 
#6  0x80d4fff0 in fil_aio_wait (segment=5) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/fil/fil0fil.cc:4551
 
#7  0x80bf1c85 in io_handler_thread (arg=0x820fcb34 <n+20>) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/srv/srv0start.cc:331
 
#8  0xb74322b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#9  0xb717316e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 20 (Thread 0xaaaf7b40 (LWP 14306)):
 
#0  0xb77d9c31 in __kernel_vsyscall ()
 
#1  0xb7437e56 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0
 
#2  0x80af77c0 in os_event::timed_wait (this=0x82bc8bc8, abstime=0xaaaf7210) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/os/os0event.cc:283
 
#3  0x80af7b16 in os_event::wait_time_low (this=0x82bc8bc8, time_in_usec=5000000, reset_sig_count=1) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/os/os0event.cc:405
 
#4  0x80af7d22 in os_event_wait_time_low (event=0x82bc8bc8, time_in_usec=5000000, reset_sig_count=1) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/os/os0event.cc:505
 
#5  0x80becde7 in srv_monitor_thread () at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/srv/srv0srv.cc:1689
 
#6  0xb74322b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#7  0xb717316e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 19 (Thread 0xab2f8b40 (LWP 14305)):
 
#0  0xb77d9c31 in __kernel_vsyscall ()
 
#1  0xb7437e56 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0
 
#2  0x80af77c0 in os_event::timed_wait (this=0x82bc8b68, abstime=0xab2f8120) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/os/os0event.cc:283
 
#3  0x80af7b16 in os_event::wait_time_low (this=0x82bc8b68, time_in_usec=1000000, reset_sig_count=1) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/os/os0event.cc:405
 
#4  0x80af7d22 in os_event_wait_time_low (event=0x82bc8b68, time_in_usec=1000000, reset_sig_count=1) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/os/os0event.cc:505
 
#5  0x80bed341 in srv_error_monitor_thread () at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/srv/srv0srv.cc:1853
 
#6  0xb74322b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#7  0xb717316e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 18 (Thread 0xaf6c5b40 (LWP 14296)):
 
#0  0xb77a5570 in ?? () from /lib/i386-linux-gnu/libaio.so.1
 
#1  0xb77a55e6 in io_getevents () from /lib/i386-linux-gnu/libaio.so.1
 
#2  0x80aea46f in LinuxAIOHandler::collect (this=0xaf6c50e0) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/os/os0file.cc:1933
 
#3  0x80aea9c9 in LinuxAIOHandler::poll (this=0xaf6c50e0, m1=0xaf6c5150, m2=0xaf6c5154, request=0xaf6c5178) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/os/os0file.cc:2078
 
#4  0x80aeab2d in os_aio_linux_handler (global_segment=1, m1=0xaf6c5150, m2=0xaf6c5154, request=0xaf6c5178) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/os/os0file.cc:2132
 
#5  0x80aefa34 in os_aio_handler (segment=1, m1=0xaf6c5150, m2=0xaf6c5154, request=0xaf6c5178) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/os/os0file.cc:5683
 
#6  0x80d4fff0 in fil_aio_wait (segment=1) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/fil/fil0fil.cc:4551
 
#7  0x80bf1c85 in io_handler_thread (arg=0x820fcb24 <n+4>) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/srv/srv0start.cc:331
 
#8  0xb74322b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#9  0xb717316e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 17 (Thread 0xa88feb40 (LWP 14312)):
 
#0  0xb77d9c31 in __kernel_vsyscall ()
 
#1  0xb7157bfc in sched_yield () from /lib/i386-linux-gnu/libc.so.6
 
#2  0x80af8356 in os_thread_yield () at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/os/os0thread.cc:211
 
#3  0x80a4e822 in TTASEventMutex<GenericPolicy>::enter (this=0x816f9918 <srv_sys+88>, max_spins=60, max_delay=4, filename=0x811da63c "/home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/srv/srv0srv.cc", line=918) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/include/ib0mutex.h:451
 
#4  0x80a4cd6a in PolicyMutex<TTASEventMutex<GenericPolicy> >::enter (this=0x816f9918 <srv_sys+88>, n_spins=30, n_delay=4, name=0x811da63c "/home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/srv/srv0srv.cc", line=918) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/include/ib0mutex.h:592
 
#5  0x80beb109 in srv_resume_thread (slot=0x816f9998 <srv_sys+216>, sig_count=0, wait=true, timeout_usec=0) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/srv/srv0srv.cc:918
 
#6  0x80beeeef in srv_worker_thread (arg=0x0) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/srv/srv0srv.cc:2483
 
#7  0xb74322b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#8  0xb717316e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 16 (Thread 0xaeec4b40 (LWP 14297)):
 
#0  0xb77a5570 in ?? () from /lib/i386-linux-gnu/libaio.so.1
 
#1  0xb77a55e6 in io_getevents () from /lib/i386-linux-gnu/libaio.so.1
 
#2  0x80aea46f in LinuxAIOHandler::collect (this=0xaeec40e0) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/os/os0file.cc:1933
 
#3  0x80aea9c9 in LinuxAIOHandler::poll (this=0xaeec40e0, m1=0xaeec4150, m2=0xaeec4154, request=0xaeec4178) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/os/os0file.cc:2078
 
#4  0x80aeab2d in os_aio_linux_handler (global_segment=2, m1=0xaeec4150, m2=0xaeec4154, request=0xaeec4178) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/os/os0file.cc:2132
 
#5  0x80aefa34 in os_aio_handler (segment=2, m1=0xaeec4150, m2=0xaeec4154, request=0xaeec4178) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/os/os0file.cc:5683
 
#6  0x80d4fff0 in fil_aio_wait (segment=2) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/fil/fil0fil.cc:4551
 
#7  0x80bf1c85 in io_handler_thread (arg=0x820fcb28 <n+8>) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/srv/srv0start.cc:331
 
#8  0xb74322b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#9  0xb717316e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 15 (Thread 0xa6cffb40 (LWP 14315)):
 
#0  0xb77d9c31 in __kernel_vsyscall ()
 
#1  0xb7437a8c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0
 
#2  0x80af8004 in os_event::wait (this=0x82bc8c28) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/os/os0event.cc:163
 
#3  0x80af7993 in os_event::wait_low (this=0x82bc8c28, reset_sig_count=1) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/os/os0event.cc:333
 
#4  0x80af7d54 in os_event_wait_low (event=0x82bc8c28, reset_sig_count=0) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/os/os0event.cc:522
 
#5  0x80ccdcde in buf_dump_thread () at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/buf/buf0dump.cc:837
 
#6  0xb74322b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#7  0xb717316e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 14 (Thread 0xa78fcb40 (LWP 14314)):
 
#0  0xb77d9c31 in __kernel_vsyscall ()
 
#1  0xb716ee46 in munmap () from /lib/i386-linux-gnu/libc.so.6
 
#2  0xb70b0cf0 in ?? () from /lib/i386-linux-gnu/libc.so.6
 
#3  0xb70b0d8b in __assert_fail () from /lib/i386-linux-gnu/libc.so.6
 
#4  0x8033c8f4 in ilink::assert_linked (this=0xa6f00700) at /home/buildbot/buildbot/build/mariadb-10.3.10/sql/sql_list.h:634
 
#5  0x8033df1d in unlink_not_visible_thd (thd=0xa6f00700) at /home/buildbot/buildbot/build/mariadb-10.3.10/sql/sql_class.h:4848
 
#6  0x803ef23c in destroy_thd (thd=0xa6f00700) at /home/buildbot/buildbot/build/mariadb-10.3.10/sql/sql_class.cc:4719
 
#7  0x80a22e7c in innobase_destroy_background_thd (thd=0xa6f00700) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/handler/ha_innodb.cc:1652
 
#8  0x80beef81 in srv_worker_thread (arg=0x0) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/srv/srv0srv.cc:2503
 
#9  0xb74322b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#10 0xb717316e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 13 (Thread 0xb00c6b40 (LWP 14319)):
 
#0  0xb77d9c31 in __kernel_vsyscall ()
 
#1  0xb743c1cd in do_sigwait () from /lib/i386-linux-gnu/libpthread.so.0
 
#2  0xb743c25e in sigwait () from /lib/i386-linux-gnu/libpthread.so.0
 
#3  0x8032fd67 in signal_hand (arg=0x0) at /home/buildbot/buildbot/build/mariadb-10.3.10/sql/mysqld.cc:3554
 
#4  0x809356e0 in pfs_spawn_thread (arg=0x82d68068) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/perfschema/pfs.cc:1862
 
#5  0xb74322b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#6  0xb717316e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 12 (Thread 0xac2fab40 (LWP 14310)):
 
#0  0xb77d9c31 in __kernel_vsyscall ()
 
#1  0xb743b8aa in nanosleep () from /lib/i386-linux-gnu/libpthread.so.0
 
#2  0x80af83b3 in os_thread_sleep (tm=1000000) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/os/os0thread.cc:230
 
#3  0x80bee81c in srv_master_sleep () at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/srv/srv0srv.cc:2325
 
#4  0x80bee95b in srv_master_thread (arg=0x0) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/srv/srv0srv.cc:2366
 
#5  0xb74322b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#6  0xb717316e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 11 (Thread 0xa90ffb40 (LWP 14311)):
 
#0  0x80beb29a in srv_release_threads (type=SRV_WORKER, n=3) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/srv/srv0srv.cc:945
 
#1  0x80bef74f in srv_purge_coordinator_thread (arg=0x0) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/srv/srv0srv.cc:2748
 
#2  0xb74322b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#3  0xb717316e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 10 (Thread 0xadec2b40 (LWP 14299)):
 
#0  0xb77a5570 in ?? () from /lib/i386-linux-gnu/libaio.so.1
 
#1  0xb77a55e6 in io_getevents () from /lib/i386-linux-gnu/libaio.so.1
 
#2  0x80aea46f in LinuxAIOHandler::collect (this=0xadec20e0) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/os/os0file.cc:1933
 
#3  0x80aea9c9 in LinuxAIOHandler::poll (this=0xadec20e0, m1=0xadec2150, m2=0xadec2154, request=0xadec2178) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/os/os0file.cc:2078
 
#4  0x80aeab2d in os_aio_linux_handler (global_segment=4, m1=0xadec2150, m2=0xadec2154, request=0xadec2178) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/os/os0file.cc:2132
 
#5  0x80aefa34 in os_aio_handler (segment=4, m1=0xadec2150, m2=0xadec2154, request=0xadec2178) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/os/os0file.cc:5683
 
#6  0x80d4fff0 in fil_aio_wait (segment=4) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/fil/fil0fil.cc:4551
 
#7  0x80bf1c85 in io_handler_thread (arg=0x820fcb30 <n+16>) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/srv/srv0start.cc:331
 
#8  0xb74322b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#9  0xb717316e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 9 (Thread 0xac675b40 (LWP 14515)):
 
#0  0xb77d9c31 in __kernel_vsyscall ()
 
#1  0xb7437a8c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0
 
#2  0x80e9a08b in safe_cond_wait (cond=0x818a0760 <COND_thread_count>, mp=0x8189f9c0 <LOCK_thread_count>, file=0x80f1eb44 "/home/buildbot/buildbot/build/mariadb-10.3.10/include/mysql/psi/mysql_thread.h", line=1167) at /home/buildbot/buildbot/build/mariadb-10.3.10/mysys/thr_mutex.c:492
 
#3  0x80329e2e in inline_mysql_cond_wait (that=0x818a0760 <COND_thread_count>, mutex=0x8189f9c0 <LOCK_thread_count>, src_file=0x80f20b34 "/home/buildbot/buildbot/build/mariadb-10.3.10/sql/mysqld.cc", src_line=1851) at /home/buildbot/buildbot/build/mariadb-10.3.10/include/mysql/psi/mysql_thread.h:1167
 
#4  0x8032c5ba in close_connections () at /home/buildbot/buildbot/build/mariadb-10.3.10/sql/mysqld.cc:1851
 
#5  0x8032ca8b in kill_server (sig_ptr=0x0) at /home/buildbot/buildbot/build/mariadb-10.3.10/sql/mysqld.cc:2035
 
#6  0x8032cb0e in kill_server_thread (arg=0xb00c61b0) at /home/buildbot/buildbot/build/mariadb-10.3.10/sql/mysqld.cc:2067
 
#7  0x809356e0 in pfs_spawn_thread (arg=0xb0f018d0) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/perfschema/pfs.cc:1862
 
#8  0xb74322b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#9  0xb717316e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 8 (Thread 0xae6c3b40 (LWP 14298)):
 
#0  0xb77a5570 in ?? () from /lib/i386-linux-gnu/libaio.so.1
 
#1  0xb77a55e6 in io_getevents () from /lib/i386-linux-gnu/libaio.so.1
 
#2  0x80aea46f in LinuxAIOHandler::collect (this=0xae6c30e0) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/os/os0file.cc:1933
 
#3  0x80aea9c9 in LinuxAIOHandler::poll (this=0xae6c30e0, m1=0xae6c3150, m2=0xae6c3154, request=0xae6c3178) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/os/os0file.cc:2078
 
#4  0x80aeab2d in os_aio_linux_handler (global_segment=3, m1=0xae6c3150, m2=0xae6c3154, request=0xae6c3178) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/os/os0file.cc:2132
 
#5  0x80aefa34 in os_aio_handler (segment=3, m1=0xae6c3150, m2=0xae6c3154, request=0xae6c3178) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/os/os0file.cc:5683
 
#6  0x80d4fff0 in fil_aio_wait (segment=3) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/fil/fil0fil.cc:4551
 
#7  0x80bf1c85 in io_handler_thread (arg=0x820fcb2c <n+12>) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/srv/srv0start.cc:331
 
#8  0xb74322b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#9  0xb717316e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 7 (Thread 0xacec0b40 (LWP 14301)):
 
#0  0xb77d9c31 in __kernel_vsyscall ()
 
#1  0xb7437e56 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0
 
#2  0x80af77c0 in os_event::timed_wait (this=0x82bc8c88, abstime=0xacec0080) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/os/os0event.cc:283
 
#3  0x80af7b16 in os_event::wait_time_low (this=0x82bc8c88, time_in_usec=1000000, reset_sig_count=2) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/os/os0event.cc:405
 
#4  0x80af7d22 in os_event_wait_time_low (event=0x82bc8c88, time_in_usec=1000000, reset_sig_count=2) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/os/os0event.cc:505
 
#5  0x80cd8bc0 in pc_sleep_if_needed (next_loop_time=1151902660, sig_count=2, cur_time=1151901660) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/buf/buf0flu.cc:2687
 
#6  0x80cd9b1e in buf_flush_page_cleaner_coordinator () at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/buf/buf0flu.cc:3146
 
#7  0xb74322b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#8  0xb717316e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 6 (Thread 0xafec6b40 (LWP 14295)):
 
#0  0xb77a5570 in ?? () from /lib/i386-linux-gnu/libaio.so.1
 
#1  0xb77a55e6 in io_getevents () from /lib/i386-linux-gnu/libaio.so.1
 
#2  0x80aea46f in LinuxAIOHandler::collect (this=0xafec60e0) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/os/os0file.cc:1933
 
#3  0x80aea9c9 in LinuxAIOHandler::poll (this=0xafec60e0, m1=0xafec6150, m2=0xafec6154, request=0xafec6178) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/os/os0file.cc:2078
 
#4  0x80aeab2d in os_aio_linux_handler (global_segment=0, m1=0xafec6150, m2=0xafec6154, request=0xafec6178) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/os/os0file.cc:2132
 
#5  0x80aefa34 in os_aio_handler (segment=0, m1=0xafec6150, m2=0xafec6154, request=0xafec6178) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/os/os0file.cc:5683
 
#6  0x80d4fff0 in fil_aio_wait (segment=0) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/fil/fil0fil.cc:4551
 
#7  0x80bf1c85 in io_handler_thread (arg=0x820fcb20 <n>) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/srv/srv0start.cc:331
 
#8  0xb74322b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#9  0xb717316e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 5 (Thread 0xabaf9b40 (LWP 14304)):
 
#0  0xb77d9c31 in __kernel_vsyscall ()
 
#1  0xb7437e56 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0
 
#2  0x80af77c0 in os_event::timed_wait (this=0x82cf0420, abstime=0xabaf9220) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/os/os0event.cc:283
 
#3  0x80af7b16 in os_event::wait_time_low (this=0x82cf0420, time_in_usec=1000000, reset_sig_count=1) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/os/os0event.cc:405
 
#4  0x80af7d22 in os_event_wait_time_low (event=0x82cf0420, time_in_usec=1000000, reset_sig_count=1) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/os/os0event.cc:505
 
#5  0x80ac29e7 in lock_wait_timeout_thread () at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/lock/lock0wait.cc:543
 
#6  0xb74322b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#7  0xb717316e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 4 (Thread 0xb1843b40 (LWP 14294)):
 
#0  0xb77d9c31 in __kernel_vsyscall ()
 
#1  0xb7437e56 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0
 
#2  0x80e9a336 in safe_cond_timedwait (cond=0x820c9b20 <COND_checkpoint>, mp=0x820c9aa0 <LOCK_checkpoint>, abstime=0xb184319c, file=0x8112c3e8 "/home/buildbot/buildbot/build/mariadb-10.3.10/include/mysql/psi/mysql_thread.h", line=1204) at /home/buildbot/buildbot/build/mariadb-10.3.10/mysys/thr_mutex.c:546
 
#3  0x80994583 in inline_mysql_cond_timedwait (that=0x820c9b20 <COND_checkpoint>, mutex=0x820c9aa0 <LOCK_checkpoint>, abstime=0xb184319c, src_file=0x8112c438 "/home/buildbot/buildbot/build/mariadb-10.3.10/storage/maria/ma_servicethread.c", src_line=116) at /home/buildbot/buildbot/build/mariadb-10.3.10/include/mysql/psi/mysql_thread.h:1204
 
#4  0x80994ac6 in my_service_thread_sleep (control=0x816f0cb8 <checkpoint_control>, sleep_time=30000000000) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/maria/ma_servicethread.c:115
 
#5  0x80988cad in ma_checkpoint_background (arg=0x1e) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/maria/ma_checkpoint.c:707
 
#6  0x809356e0 in pfs_spawn_thread (arg=0x82b2b2a0) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/perfschema/pfs.cc:1862
 
#7  0xb74322b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#8  0xb717316e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 3 (Thread 0xa5cfdb40 (LWP 14317)):
 
#0  0xb77d9c31 in __kernel_vsyscall ()
 
#1  0xb7437a8c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0
 
#2  0x80af8004 in os_event::wait (this=0x82bc8ce8) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/os/os0event.cc:163
 
#3  0x80af7993 in os_event::wait_low (this=0x82bc8ce8, reset_sig_count=1) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/os/os0event.cc:333
 
#4  0x80af7d54 in os_event_wait_low (event=0x82bc8ce8, reset_sig_count=0) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/os/os0event.cc:522
 
#5  0x80cb7284 in buf_resize_thread () at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/buf/buf0buf.cc:3259
 
#6  0xb74322b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#7  0xb717316e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 2 (Thread 0xa64feb40 (LWP 14316)):
 
#0  0xb77d9c31 in __kernel_vsyscall ()
 
#1  0xb743b8aa in nanosleep () from /lib/i386-linux-gnu/libpthread.so.0
 
#2  0x80af83b3 in os_thread_sleep (tm=1000000) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/os/os0thread.cc:230
 
#3  0x80ca8b7f in btr_defragment_thread () at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/btr/btr0defragment.cc:705
 
#4  0xb74322b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#5  0xb717316e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 1 (Thread 0xa80fdb40 (LWP 14313)):
 
#0  0xb77d9c31 in __kernel_vsyscall ()
 
#1  0xb70b7e89 in raise () from /lib/i386-linux-gnu/libc.so.6
 
#2  0xb70b93e7 in abort () from /lib/i386-linux-gnu/libc.so.6
 
#3  0xb70b0d07 in ?? () from /lib/i386-linux-gnu/libc.so.6
 
#4  0xb70b0d8b in __assert_fail () from /lib/i386-linux-gnu/libc.so.6
 
#5  0x8033c8f4 in ilink::assert_linked (this=0xa6d13ce0) at /home/buildbot/buildbot/build/mariadb-10.3.10/sql/sql_list.h:634
 
#6  0x8033df1d in unlink_not_visible_thd (thd=0xa6d13ce0) at /home/buildbot/buildbot/build/mariadb-10.3.10/sql/sql_class.h:4848
 
#7  0x803ef23c in destroy_thd (thd=0xa6d13ce0) at /home/buildbot/buildbot/build/mariadb-10.3.10/sql/sql_class.cc:4719
 
#8  0x80a22e7c in innobase_destroy_background_thd (thd=0xa6d13ce0) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/handler/ha_innodb.cc:1652
 
#9  0x80beef81 in srv_worker_thread (arg=0x0) at /home/buildbot/buildbot/build/mariadb-10.3.10/storage/innobase/srv/srv0srv.cc:2503
 
#10 0xb74322b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#11 0xb717316e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Compressed file /mnt/buildbot/build/mariadb-10.3.10/mysql-test/var/3/log/innodb.innodb-page_compression_tables-innodb/mysqld.1/data/core
 
 - skipping '/mnt/buildbot/build/mariadb-10.3.10/mysql-test/var/3/log/innodb.innodb-page_compression_tables-innodb/'

Comment by Marko Mäkelä [ 2022-07-27 ]

I think that this was fixed in MDEV-26755 a while ago.

Generated at Thu Feb 08 08:07:12 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.