[MDEV-21571] main.mysqld_option_err failed in buildbot, server crashed in ut_list_prepend on bootstrap Created: 2020-01-26  Updated: 2023-11-29  Resolved: 2023-11-29

Status: Closed
Project: MariaDB Server
Component/s: Server, Storage Engine - InnoDB, Tests
Affects Version/s: 10.2
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Elena Stepanova Assignee: Marko Mäkelä
Resolution: Cannot Reproduce Votes: 0
Labels: None


 Description   

http://buildbot.askmonty.org/buildbot/builders/kvm-bintar-centos6-amd64-debug/builds/2462

10.2 7aa443ca7dc58f301a8c6051e2623498

main.mysqld_option_err                   w3 [ fail ]
        Test ended at 2020-01-23 18:27:53
 
CURRENT_TEST: main.mysqld_option_err
sh: line 1: 28647 Aborted                 (core dumped) /usr/local/mariadb-10.2.31-linux-x86_64/bin/mysqld --no-defaults --disable-getopt-prefix-matching --bootstrap --basedir=/usr/local/mariadb-10.2.31-linux-x86_64 --datadir=/usr/local/mariadb-10.2.31-linux-x86_64/mysql-test/var/3/mysqld.1/data --plugin-dir=/usr/local/mariadb-10.2.31-linux-x86_64/lib/plugin/ --default-storage-engine=myisam --loose-skip-plugin-aria --loose-skip-plugin-feedback --loose-skip-plugin-innodb --loose-skip-plugin-innodb-buffer-page --loose-skip-plugin-innodb-buffer-page-lru --loose-skip-plugin-innodb-buffer-pool-stats --loose-skip-plugin-innodb-cmp --loose-skip-plugin-innodb-cmp-per-index --loose-skip-plugin-innodb-cmp-per-index-reset --loose-skip-plugin-innodb-cmp-reset --loose-skip-plugin-innodb-cmpmem --loose-skip-plugin-innodb-cmpmem-reset --loose-skip-plugin-innodb-ft-being-deleted --loose-skip-plugin-innodb-ft-config --loose-skip-plugin-innodb-ft-default-stopword --loose-skip-plugin-innodb-ft-deleted --loose-skip-plugin-innodb-ft-index-cache --loose-skip-plugin-innodb-ft-index-table --loose-skip-plugin-innodb-lock-waits --loose-skip-plugin-innodb-locks --loose-skip-plugin-innodb-metrics --loose-skip-plugin-innodb-mutexes --loose-skip-plugin-innodb-sys-columns --loose-skip-plugin-innodb-sys-datafiles --loose-skip-plugin-innodb-sys-fields --loose-skip-plugin-innodb-sys-foreign --loose-skip-plugin-innodb-sys-foreign-cols --loose-skip-plugin-innodb-sys-indexes --loose-skip-plugin-innodb-sys-semaphore-waits --loose-skip-plugin-innodb-sys-tables --loose-skip-plugin-innodb-sys-tablespaces --loose-skip-plugin-innodb-sys-tablestats --loose-skip-plugin-innodb-sys-virtual --loose-skip-plugin-innodb-tablespaces-encryption --loose-skip-plugin-innodb-tablespaces-scrubbing --loose-skip-plugin-innodb-trx --loose-skip-plugin-partition --loose-skip-plugin-sequence --loose-skip-plugin-user-variables --loose-innodb --loose-innodb-log-file-size=5M --disable-sync-frm --tmpdir=/usr/local/mariadb-10.2.31-linux-x86_64/mysql-test/var/3/tmp/ --core-file --console --lc-messages-dir=/usr/local/mariadb-10.2.31-linux-x86_64/share/ --character-sets-dir=/usr/local/mariadb-10.2.31-linux-x86_64/share/charsets --skip-networking --datadir=/usr/local/mariadb-10.2.31-linux-x86_64/mysql-test/var/3/tmp/mysqld_option_err --skip-grant-tables --default-storage-engine=nonexistentengine >> /usr/local/mariadb-10.2.31-linux-x86_64/mysql-test/var/3/tmp/mysqld_option_err/mysqltest.log 2>&1
mysqltest: At line 34: command "$MYSQLD_BOOTSTRAP_CMD --skip-networking --datadir=$MYSQLTEST_VARDIR/tmp/mysqld_option_err --skip-grant-tables --default-storage-engine=nonexistentengine >>$MYSQLTEST_VARDIR/tmp/mysqld_option_err/mysqltest.log 2>&1" failed with wrong error: 134
 
The result from queries just before the failure was:
Test that unknown option is not silently ignored.
Test bad binlog format.
Test bad default storage engine.
 
worker[3] > Restart  - not started
 - found 'core.28647' (0/0)
 
Trying 'dbx' to get a backtrace
 
Trying 'gdb' to get a backtrace from coredump /usr/local/mariadb-10.2.31-linux-x86_64/mysql-test/var/3/log/main.mysqld_option_err/tmp/mysqld_option_err/core.28647
Core generated by '/usr/local/mariadb-10.2.31-linux-x86_64/bin/mysqld'
Output from gdb follows. The first stack trace is from the failing thread.
The following stack traces are from all threads (so the failing one is
duplicated).
--------------------------
[New Thread 28673]
[New Thread 28653]
[New Thread 28676]
[New Thread 28655]
[New Thread 28647]
[New Thread 28656]
[New Thread 28650]
[New Thread 28658]
[New Thread 28654]
[New Thread 28659]
[New Thread 28657]
[New Thread 28660]
[New Thread 28661]
[New Thread 28662]
[New Thread 28668]
[New Thread 28663]
[New Thread 28671]
[Thread debugging using libthread_db enabled]
Core was generated by `/usr/local/mariadb-10.2.31-linux-x86_64/bin/mysqld --no-defaults --disable-geto'.
Program terminated with signal 6, Aborted.
#0  0x00007f34b2d1863c in pthread_kill () from /lib64/libpthread.so.0
#0  0x00007f34b2d1863c in pthread_kill () from /lib64/libpthread.so.0
#1  0x00007f34b3edf955 in my_write_core (sig=6) at /home/buildbot/buildbot/build/mysys/stacktrace.c:477
#2  0x00007f34b387978e in handle_fatal_signal (sig=6) at /home/buildbot/buildbot/build/sql/signal_handler.cc:343
#3  <signal handler called>
#4  0x00007f34b1be0885 in raise () from /lib64/libc.so.6
#5  0x00007f34b1be2065 in abort () from /lib64/libc.so.6
#6  0x00007f34b1bd99fe in __assert_fail_base () from /lib64/libc.so.6
#7  0x00007f34b1bd9ac0 in __assert_fail () from /lib64/libc.so.6
#8  0x00007f34b3bd40a7 in ut_list_prepend<ut_list_base<rw_lock_t, ut_list_node<rw_lock_t> rw_lock_t::*> > (list=Unhandled dwarf expression opcode 0xf3
) at /home/buildbot/buildbot/build/storage/innobase/include/ut0lst.h:160
#9  0x00007f34b3bd3496 in rw_lock_create_func (lock=0x7f34b49436b8, level=Unhandled dwarf expression opcode 0xf3
) at /home/buildbot/buildbot/build/storage/innobase/sync/sync0rw.cc:244
#10 0x00007f34b3bbdf08 in pfs_rw_lock_create_func (key=Unhandled dwarf expression opcode 0xf3
) at /home/buildbot/buildbot/build/storage/innobase/include/sync0rw.ic:508
#11 0x00007f34b3bc1a86 in srv_worker_thread (arg=Unhandled dwarf expression opcode 0xf3
) at /home/buildbot/buildbot/build/storage/innobase/srv/srv0srv.cc:2605
#12 0x00007f34b2d137f1 in start_thread () from /lib64/libpthread.so.0
#13 0x00007f34b1c93ccd in clone () from /lib64/libc.so.6
 
Thread 17 (Thread 0x7f348a1fc700 (LWP 28671)):
#0  pfs_end (type=SRV_WORKER, n=3) at /home/buildbot/buildbot/build/storage/innobase/include/ib0mutex.h:782
#1  enter (type=SRV_WORKER, n=3) at /home/buildbot/buildbot/build/storage/innobase/include/ib0mutex.h:641
#2  srv_release_threads (type=SRV_WORKER, n=3) at /home/buildbot/buildbot/build/storage/innobase/srv/srv0srv.cc:966
#3  0x00007f34b3bc1612 in srv_purge_coordinator_thread (arg=Unhandled dwarf expression opcode 0xf3
) at /home/buildbot/buildbot/build/storage/innobase/srv/srv0srv.cc:2924
#4  0x00007f34b2d137f1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f34b1c93ccd in clone () from /lib64/libc.so.6
 
Thread 16 (Thread 0x7f349302e700 (LWP 28663)):
#0  0x00007f34b2d1775b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f34b3ae6bb1 in os_event::timed_wait (this=Unhandled dwarf expression opcode 0xf3
) at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:275
#2  0x00007f34b3ae6ea8 in os_event::wait_time_low (this=0x7f34b579fdc0, time_in_usec=Unhandled dwarf expression opcode 0xf3
) at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:385
#3  0x00007f34b3ae72a1 in os_event_wait_time_low (event=Unhandled dwarf expression opcode 0xf3
) at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:490
#4  0x00007f34b3c9c430 in pc_sleep_if_needed () at /home/buildbot/buildbot/build/storage/innobase/buf/buf0flu.cc:2707
#5  buf_flush_page_cleaner_coordinator () at /home/buildbot/buildbot/build/storage/innobase/buf/buf0flu.cc:3173
#6  0x00007f34b2d137f1 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f34b1c93ccd in clone () from /lib64/libc.so.6
 
Thread 15 (Thread 0x7f348bfff700 (LWP 28668)):
#0  0x00007f34b2d1accd in nanosleep () from /lib64/libpthread.so.0
#1  0x00007f34b3ae74ce in os_thread_sleep (tm=Unhandled dwarf expression opcode 0xf3
) at /home/buildbot/buildbot/build/storage/innobase/os/os0thread.cc:231
#2  0x00007f34b3bbffcd in srv_master_sleep (arg=Unhandled dwarf expression opcode 0xf3
) at /home/buildbot/buildbot/build/storage/innobase/srv/srv0srv.cc:2414
#3  srv_master_thread (arg=Unhandled dwarf expression opcode 0xf3
) at /home/buildbot/buildbot/build/storage/innobase/srv/srv0srv.cc:2455
#4  0x00007f34b2d137f1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f34b1c93ccd in clone () from /lib64/libc.so.6
 
Thread 14 (Thread 0x7f3493a2f700 (LWP 28662)):
#0  0x00007f34b2b0b614 in ?? () from /lib64/libaio.so.1
#1  0x00007f34b3ae0b6a in LinuxAIOHandler::collect (this=0x7f3493a2ec00) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:1936
#2  0x00007f34b3ae283a in LinuxAIOHandler::poll (this=0x7f3493a2ec00, m1=0x7f3493a2eca0, m2=0x7f3493a2eca8, request=0x7f3493a2ecc0) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2097
#3  0x00007f34b3ae3aec in os_aio_linux_handler (segment=9, m1=0x7f3493a2eca0, m2=0x7f3493a2eca8, request=0x7f3493a2ecc0) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2151
#4  os_aio_handler (segment=9, m1=0x7f3493a2eca0, m2=0x7f3493a2eca8, request=0x7f3493a2ecc0) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:5781
#5  0x00007f34b3d03ca0 in fil_aio_wait (segment=9) at /home/buildbot/buildbot/build/storage/innobase/fil/fil0fil.cc:5116
#6  0x00007f34b3bc5df0 in io_handler_thread (arg=Unhandled dwarf expression opcode 0xf3
) at /home/buildbot/buildbot/build/storage/innobase/srv/srv0start.cc:332
#7  0x00007f34b2d137f1 in start_thread () from /lib64/libpthread.so.0
#8  0x00007f34b1c93ccd in clone () from /lib64/libc.so.6
 
Thread 13 (Thread 0x7f3494430700 (LWP 28661)):
#0  0x00007f34b2b0b614 in ?? () from /lib64/libaio.so.1
#1  0x00007f34b3ae0b6a in LinuxAIOHandler::collect (this=0x7f349442fc00) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:1936
#2  0x00007f34b3ae283a in LinuxAIOHandler::poll (this=0x7f349442fc00, m1=0x7f349442fca0, m2=0x7f349442fca8, request=0x7f349442fcc0) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2097
#3  0x00007f34b3ae3aec in os_aio_linux_handler (segment=8, m1=0x7f349442fca0, m2=0x7f349442fca8, request=0x7f349442fcc0) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2151
#4  os_aio_handler (segment=8, m1=0x7f349442fca0, m2=0x7f349442fca8, request=0x7f349442fcc0) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:5781
#5  0x00007f34b3d03ca0 in fil_aio_wait (segment=8) at /home/buildbot/buildbot/build/storage/innobase/fil/fil0fil.cc:5116
#6  0x00007f34b3bc5df0 in io_handler_thread (arg=Unhandled dwarf expression opcode 0xf3
) at /home/buildbot/buildbot/build/storage/innobase/srv/srv0start.cc:332
#7  0x00007f34b2d137f1 in start_thread () from /lib64/libpthread.so.0
#8  0x00007f34b1c93ccd in clone () from /lib64/libc.so.6
 
Thread 12 (Thread 0x7f3494e31700 (LWP 28660)):
#0  0x00007f34b2b0b614 in ?? () from /lib64/libaio.so.1
#1  0x00007f34b3ae0b6a in LinuxAIOHandler::collect (this=0x7f3494e30c00) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:1936
#2  0x00007f34b3ae283a in LinuxAIOHandler::poll (this=0x7f3494e30c00, m1=0x7f3494e30ca0, m2=0x7f3494e30ca8, request=0x7f3494e30cc0) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2097
#3  0x00007f34b3ae3aec in os_aio_linux_handler (segment=7, m1=0x7f3494e30ca0, m2=0x7f3494e30ca8, request=0x7f3494e30cc0) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2151
#4  os_aio_handler (segment=7, m1=0x7f3494e30ca0, m2=0x7f3494e30ca8, request=0x7f3494e30cc0) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:5781
#5  0x00007f34b3d03ca0 in fil_aio_wait (segment=7) at /home/buildbot/buildbot/build/storage/innobase/fil/fil0fil.cc:5116
#6  0x00007f34b3bc5df0 in io_handler_thread (arg=Unhandled dwarf expression opcode 0xf3
) at /home/buildbot/buildbot/build/storage/innobase/srv/srv0start.cc:332
#7  0x00007f34b2d137f1 in start_thread () from /lib64/libpthread.so.0
#8  0x00007f34b1c93ccd in clone () from /lib64/libc.so.6
 
Thread 11 (Thread 0x7f3496c34700 (LWP 28657)):
#0  0x00007f34b2b0b614 in ?? () from /lib64/libaio.so.1
#1  0x00007f34b3ae0b6a in LinuxAIOHandler::collect (this=0x7f3496c33c00) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:1936
#2  0x00007f34b3ae283a in LinuxAIOHandler::poll (this=0x7f3496c33c00, m1=0x7f3496c33ca0, m2=0x7f3496c33ca8, request=0x7f3496c33cc0) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2097
#3  0x00007f34b3ae3aec in os_aio_linux_handler (segment=4, m1=0x7f3496c33ca0, m2=0x7f3496c33ca8, request=0x7f3496c33cc0) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2151
#4  os_aio_handler (segment=4, m1=0x7f3496c33ca0, m2=0x7f3496c33ca8, request=0x7f3496c33cc0) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:5781
#5  0x00007f34b3d03ca0 in fil_aio_wait (segment=4) at /home/buildbot/buildbot/build/storage/innobase/fil/fil0fil.cc:5116
#6  0x00007f34b3bc5df0 in io_handler_thread (arg=Unhandled dwarf expression opcode 0xf3
) at /home/buildbot/buildbot/build/storage/innobase/srv/srv0start.cc:332
#7  0x00007f34b2d137f1 in start_thread () from /lib64/libpthread.so.0
#8  0x00007f34b1c93ccd in clone () from /lib64/libc.so.6
 
Thread 10 (Thread 0x7f3495832700 (LWP 28659)):
#0  0x00007f34b2b0b614 in ?? () from /lib64/libaio.so.1
#1  0x00007f34b3ae0b6a in LinuxAIOHandler::collect (this=0x7f3495831c00) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:1936
#2  0x00007f34b3ae283a in LinuxAIOHandler::poll (this=0x7f3495831c00, m1=0x7f3495831ca0, m2=0x7f3495831ca8, request=0x7f3495831cc0) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2097
#3  0x00007f34b3ae3aec in os_aio_linux_handler (segment=6, m1=0x7f3495831ca0, m2=0x7f3495831ca8, request=0x7f3495831cc0) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2151
#4  os_aio_handler (segment=6, m1=0x7f3495831ca0, m2=0x7f3495831ca8, request=0x7f3495831cc0) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:5781
#5  0x00007f34b3d03ca0 in fil_aio_wait (segment=6) at /home/buildbot/buildbot/build/storage/innobase/fil/fil0fil.cc:5116
#6  0x00007f34b3bc5df0 in io_handler_thread (arg=Unhandled dwarf expression opcode 0xf3
) at /home/buildbot/buildbot/build/storage/innobase/srv/srv0start.cc:332
#7  0x00007f34b2d137f1 in start_thread () from /lib64/libpthread.so.0
#8  0x00007f34b1c93ccd in clone () from /lib64/libc.so.6
 
Thread 9 (Thread 0x7f3498a37700 (LWP 28654)):
#0  0x00007f34b2b0b614 in ?? () from /lib64/libaio.so.1
#1  0x00007f34b3ae0b6a in LinuxAIOHandler::collect (this=0x7f3498a36c00) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:1936
#2  0x00007f34b3ae283a in LinuxAIOHandler::poll (this=0x7f3498a36c00, m1=0x7f3498a36ca0, m2=0x7f3498a36ca8, request=0x7f3498a36cc0) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2097
#3  0x00007f34b3ae3aec in os_aio_linux_handler (segment=1, m1=0x7f3498a36ca0, m2=0x7f3498a36ca8, request=0x7f3498a36cc0) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2151
#4  os_aio_handler (segment=1, m1=0x7f3498a36ca0, m2=0x7f3498a36ca8, request=0x7f3498a36cc0) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:5781
#5  0x00007f34b3d03ca0 in fil_aio_wait (segment=1) at /home/buildbot/buildbot/build/storage/innobase/fil/fil0fil.cc:5116
#6  0x00007f34b3bc5df0 in io_handler_thread (arg=Unhandled dwarf expression opcode 0xf3
) at /home/buildbot/buildbot/build/storage/innobase/srv/srv0start.cc:332
#7  0x00007f34b2d137f1 in start_thread () from /lib64/libpthread.so.0
#8  0x00007f34b1c93ccd in clone () from /lib64/libc.so.6
 
Thread 8 (Thread 0x7f3496233700 (LWP 28658)):
#0  0x00007f34b2b0b614 in ?? () from /lib64/libaio.so.1
#1  0x00007f34b3ae0b6a in LinuxAIOHandler::collect (this=0x7f3496232c00) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:1936
#2  0x00007f34b3ae283a in LinuxAIOHandler::poll (this=0x7f3496232c00, m1=0x7f3496232ca0, m2=0x7f3496232ca8, request=0x7f3496232cc0) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2097
#3  0x00007f34b3ae3aec in os_aio_linux_handler (segment=5, m1=0x7f3496232ca0, m2=0x7f3496232ca8, request=0x7f3496232cc0) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2151
#4  os_aio_handler (segment=5, m1=0x7f3496232ca0, m2=0x7f3496232ca8, request=0x7f3496232cc0) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:5781
#5  0x00007f34b3d03ca0 in fil_aio_wait (segment=5) at /home/buildbot/buildbot/build/storage/innobase/fil/fil0fil.cc:5116
#6  0x00007f34b3bc5df0 in io_handler_thread (arg=Unhandled dwarf expression opcode 0xf3
) at /home/buildbot/buildbot/build/storage/innobase/srv/srv0start.cc:332
#7  0x00007f34b2d137f1 in start_thread () from /lib64/libpthread.so.0
#8  0x00007f34b1c93ccd in clone () from /lib64/libc.so.6
 
Thread 7 (Thread 0x7f34b1794700 (LWP 28650)):
#0  0x00007f34b2d1775b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f34b3ee5033 in safe_cond_timedwait (cond=0x7f34b5257920, mp=0x7f34b5257860, abstime=0x7f34b1793e10, file=0x7f34b3f8f578 "/home/buildbot/buildbot/build/include/mysql/psi/mysql_thread.h", line=1193) at /home/buildbot/buildbot/build/mysys/thr_mutex.c:545
#2  0x00007f34b3ee570c in inline_mysql_cond_timedwait (arg=Unhandled dwarf expression opcode 0xf3
) at /home/buildbot/buildbot/build/include/mysql/psi/mysql_thread.h:1193
#3  timer_handler (arg=Unhandled dwarf expression opcode 0xf3
) at /home/buildbot/buildbot/build/mysys/thr_timer.c:292
#4  0x00007f34b2d137f1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f34b1c93ccd in clone () from /lib64/libc.so.6
 
Thread 6 (Thread 0x7f3497635700 (LWP 28656)):
#0  0x00007f34b2b0b614 in ?? () from /lib64/libaio.so.1
#1  0x00007f34b3ae0b6a in LinuxAIOHandler::collect (this=0x7f3497634c00) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:1936
#2  0x00007f34b3ae283a in LinuxAIOHandler::poll (this=0x7f3497634c00, m1=0x7f3497634ca0, m2=0x7f3497634ca8, request=0x7f3497634cc0) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2097
#3  0x00007f34b3ae3aec in os_aio_linux_handler (segment=3, m1=0x7f3497634ca0, m2=0x7f3497634ca8, request=0x7f3497634cc0) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2151
#4  os_aio_handler (segment=3, m1=0x7f3497634ca0, m2=0x7f3497634ca8, request=0x7f3497634cc0) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:5781
#5  0x00007f34b3d03ca0 in fil_aio_wait (segment=3) at /home/buildbot/buildbot/build/storage/innobase/fil/fil0fil.cc:5116
#6  0x00007f34b3bc5df0 in io_handler_thread (arg=Unhandled dwarf expression opcode 0xf3
) at /home/buildbot/buildbot/build/storage/innobase/srv/srv0start.cc:332
#7  0x00007f34b2d137f1 in start_thread () from /lib64/libpthread.so.0
#8  0x00007f34b1c93ccd in clone () from /lib64/libc.so.6
 
Thread 5 (Thread 0x7f34b313a720 (LWP 28647)):
#0  0x00007f34b2d1accd in nanosleep () from /lib64/libpthread.so.0
#1  0x00007f34b3ae74ce in os_thread_sleep (tm=Unhandled dwarf expression opcode 0xf3
) at /home/buildbot/buildbot/build/storage/innobase/os/os0thread.cc:231
#2  0x00007f34b3abbb89 in logs_empty_and_mark_files_at_shutdown () at /home/buildbot/buildbot/build/storage/innobase/log/log0log.cc:1836
#3  0x00007f34b3bc69a5 in innodb_shutdown () at /home/buildbot/buildbot/build/storage/innobase/srv/srv0start.cc:2794
#4  0x00007f34b3a321ca in innobase_end () at /home/buildbot/buildbot/build/storage/innobase/handler/ha_innodb.cc:4475
#5  0x00007f34b387c03d in ha_finalize_handlerton (plugin=0x7f34b5764250) at /home/buildbot/buildbot/build/sql/handler.cc:494
#6  0x00007f34b36bd6c4 in plugin_deinitialize (plugin=0x7f34b5764250, ref_check=true) at /home/buildbot/buildbot/build/sql/sql_plugin.cc:1220
#7  0x00007f34b36c10e9 in reap_plugins () at /home/buildbot/buildbot/build/sql/sql_plugin.cc:1296
#8  0x00007f34b36c1d6d in plugin_shutdown () at /home/buildbot/buildbot/build/sql/sql_plugin.cc:1957
#9  0x00007f34b35fde1a in clean_up (print_message=true) at /home/buildbot/buildbot/build/sql/mysqld.cc:2225
#10 0x00007f34b3600c53 in unireg_abort (exit_code=1) at /home/buildbot/buildbot/build/sql/mysqld.cc:2143
#11 0x00007f34b36050ef in init_server_components () at /home/buildbot/buildbot/build/sql/mysqld.cc:5438
#12 0x00007f34b3608bc0 in mysqld_main (argc=59, argv=0x7f34b571cb10) at /home/buildbot/buildbot/build/sql/mysqld.cc:5929
#13 0x00007f34b35fb1c9 in main (argc=Unhandled dwarf expression opcode 0xf3
) at /home/buildbot/buildbot/build/sql/main.cc:25
 
Thread 4 (Thread 0x7f3498036700 (LWP 28655)):
#0  0x00007f34b2b0b614 in ?? () from /lib64/libaio.so.1
#1  0x00007f34b3ae0b6a in LinuxAIOHandler::collect (this=0x7f3498035c00) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:1936
#2  0x00007f34b3ae283a in LinuxAIOHandler::poll (this=0x7f3498035c00, m1=0x7f3498035ca0, m2=0x7f3498035ca8, request=0x7f3498035cc0) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2097
#3  0x00007f34b3ae3aec in os_aio_linux_handler (segment=2, m1=0x7f3498035ca0, m2=0x7f3498035ca8, request=0x7f3498035cc0) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2151
#4  os_aio_handler (segment=2, m1=0x7f3498035ca0, m2=0x7f3498035ca8, request=0x7f3498035cc0) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:5781
#5  0x00007f34b3d03ca0 in fil_aio_wait (segment=2) at /home/buildbot/buildbot/build/storage/innobase/fil/fil0fil.cc:5116
#6  0x00007f34b3bc5df0 in io_handler_thread (arg=Unhandled dwarf expression opcode 0xf3
) at /home/buildbot/buildbot/build/storage/innobase/srv/srv0start.cc:332
#7  0x00007f34b2d137f1 in start_thread () from /lib64/libpthread.so.0
#8  0x00007f34b1c93ccd in clone () from /lib64/libc.so.6
 
Thread 3 (Thread 0x7f347abfd700 (LWP 28676)):
#0  0x00007f34b2d1accd in nanosleep () from /lib64/libpthread.so.0
#1  0x00007f34b3ae74ce in os_thread_sleep (tm=Unhandled dwarf expression opcode 0xf3
) at /home/buildbot/buildbot/build/storage/innobase/os/os0thread.cc:231
#2  0x00007f34b3c6f6f0 in btr_defragment_thread () at /home/buildbot/buildbot/build/storage/innobase/btr/btr0defragment.cc:712
#3  0x00007f34b2d137f1 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f34b1c93ccd in clone () from /lib64/libc.so.6
 
Thread 2 (Thread 0x7f3499438700 (LWP 28653)):
#0  0x00007f34b2b0b614 in ?? () from /lib64/libaio.so.1
#1  0x00007f34b3ae0b6a in LinuxAIOHandler::collect (this=0x7f3499437c00) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:1936
#2  0x00007f34b3ae283a in LinuxAIOHandler::poll (this=0x7f3499437c00, m1=0x7f3499437ca0, m2=0x7f3499437ca8, request=0x7f3499437cc0) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2097
#3  0x00007f34b3ae3aec in os_aio_linux_handler (segment=0, m1=0x7f3499437ca0, m2=0x7f3499437ca8, request=0x7f3499437cc0) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2151
#4  os_aio_handler (segment=0, m1=0x7f3499437ca0, m2=0x7f3499437ca8, request=0x7f3499437cc0) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:5781
#5  0x00007f34b3d03ca0 in fil_aio_wait (segment=0) at /home/buildbot/buildbot/build/storage/innobase/fil/fil0fil.cc:5116
#6  0x00007f34b3bc5df0 in io_handler_thread (arg=Unhandled dwarf expression opcode 0xf3
) at /home/buildbot/buildbot/build/storage/innobase/srv/srv0start.cc:332
#7  0x00007f34b2d137f1 in start_thread () from /lib64/libpthread.so.0
#8  0x00007f34b1c93ccd in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f3488dfa700 (LWP 28673)):
#0  0x00007f34b2d1863c in pthread_kill () from /lib64/libpthread.so.0
#1  0x00007f34b3edf955 in my_write_core (sig=6) at /home/buildbot/buildbot/build/mysys/stacktrace.c:477
#2  0x00007f34b387978e in handle_fatal_signal (sig=6) at /home/buildbot/buildbot/build/sql/signal_handler.cc:343
#3  <signal handler called>
#4  0x00007f34b1be0885 in raise () from /lib64/libc.so.6
#5  0x00007f34b1be2065 in abort () from /lib64/libc.so.6
#6  0x00007f34b1bd99fe in __assert_fail_base () from /lib64/libc.so.6
#7  0x00007f34b1bd9ac0 in __assert_fail () from /lib64/libc.so.6
#8  0x00007f34b3bd40a7 in ut_list_prepend<ut_list_base<rw_lock_t, ut_list_node<rw_lock_t> rw_lock_t::*> > (list=Unhandled dwarf expression opcode 0xf3
) at /home/buildbot/buildbot/build/storage/innobase/include/ut0lst.h:160
#9  0x00007f34b3bd3496 in rw_lock_create_func (lock=0x7f34b49436b8, level=Unhandled dwarf expression opcode 0xf3
) at /home/buildbot/buildbot/build/storage/innobase/sync/sync0rw.cc:244
#10 0x00007f34b3bbdf08 in pfs_rw_lock_create_func (key=Unhandled dwarf expression opcode 0xf3
) at /home/buildbot/buildbot/build/storage/innobase/include/sync0rw.ic:508
#11 0x00007f34b3bc1a86 in srv_worker_thread (arg=Unhandled dwarf expression opcode 0xf3
) at /home/buildbot/buildbot/build/storage/innobase/srv/srv0srv.cc:2605
#12 0x00007f34b2d137f1 in start_thread () from /lib64/libpthread.so.0
#13 0x00007f34b1c93ccd in clone () from /lib64/libc.so.6
 
Compressed file /usr/local/mariadb-10.2.31-linux-x86_64/mysql-test/var/3/log/main.mysqld_option_err/tmp/mysqld_option_err/core.28647
 - skipping '/usr/local/mariadb-10.2.31-linux-x86_64/mysql-test/var/3/log/main.mysqld_option_err/'



 Comments   
Comment by Marko Mäkelä [ 2020-01-27 ]

It looks like a purge worker thread is exceptionally slow to start up, and it crashes early when creating slot->debug_sync-lock, while the server is already shutting down. I cannot immediately see the problem. The code should be protected by rw_lock_list_mutex.

Comment by Elena Stepanova [ 2020-05-05 ]

Looks the same or related
http://buildbot.askmonty.org/buildbot/builders/kvm-bintar-centos6-amd64-debug/builds/2595/steps/mtr/logs/stdio

10.2 bc862c4ebeb9d706465d580ed58c2357

perfschema.bad_option_1                  w2 [ fail ]
        Test ended at 2020-04-01 10:49:09
 
CURRENT_TEST: perfschema.bad_option_1
/usr/local/mariadb-10.2.32-linux-x86_64/bin/mysqltest: Error on delete of '/usr/local/mariadb-10.2.32-linux-x86_64/mysql-test/var/2/tmp/bad_option_1.txt' (Errcode: 2 "No such file or directory")
sh: line 1:   723 Aborted                 (core dumped) /usr/local/mariadb-10.2.32-linux-x86_64/bin/mysqld --no-defaults --disable-getopt-prefix-matching --bootstrap --basedir=/usr/local/mariadb-10.2.32-linux-x86_64 --datadir=/usr/local/mariadb-10.2.32-linux-x86_64/mysql-test/var/2/mysqld.1/data --plugin-dir=/usr/local/mariadb-10.2.32-linux-x86_64/lib/plugin/ --default-storage-engine=myisam --loose-skip-plugin-aria --loose-skip-plugin-feedback --loose-skip-plugin-innodb --loose-skip-plugin-innodb-buffer-page --loose-skip-plugin-innodb-buffer-page-lru --loose-skip-plugin-innodb-buffer-pool-stats --loose-skip-plugin-innodb-cmp --loose-skip-plugin-innodb-cmp-per-index --loose-skip-plugin-innodb-cmp-per-index-reset --loose-skip-plugin-innodb-cmp-reset --loose-skip-plugin-innodb-cmpmem --loose-skip-plugin-innodb-cmpmem-reset --loose-skip-plugin-innodb-ft-being-deleted --loose-skip-plugin-innodb-ft-config --loose-skip-plugin-innodb-ft-default-stopword --loose-skip-plugin-innodb-ft-deleted --loose-skip-plugin-innodb-ft-index-cache --loose-skip-plugin-innodb-ft-index-table --loose-skip-plugin-innodb-lock-waits --loose-skip-plugin-innodb-locks --loose-skip-plugin-innodb-metrics --loose-skip-plugin-innodb-mutexes --loose-skip-plugin-innodb-sys-columns --loose-skip-plugin-innodb-sys-datafiles --loose-skip-plugin-innodb-sys-fields --loose-skip-plugin-innodb-sys-foreign --loose-skip-plugin-innodb-sys-foreign-cols --loose-skip-plugin-innodb-sys-indexes --loose-skip-plugin-innodb-sys-semaphore-waits --loose-skip-plugin-innodb-sys-tables --loose-skip-plugin-innodb-sys-tablespaces --loose-skip-plugin-innodb-sys-tablestats --loose-skip-plugin-innodb-sys-virtual --loose-skip-plugin-innodb-tablespaces-encryption --loose-skip-plugin-innodb-tablespaces-scrubbing --loose-skip-plugin-innodb-trx --loose-skip-plugin-partition --loose-skip-plugin-sequence --loose-skip-plugin-user-variables --loose-innodb --loose-innodb-log-file-size=5M --disable-sync-frm --tmpdir=/usr/local/mariadb-10.2.32-linux-x86_64/mysql-test/var/2/tmp/ --core-file --console --lc-messages-dir=/usr/local/mariadb-10.2.32-linux-x86_64/share/ --character-sets-dir=/usr/local/mariadb-10.2.32-linux-x86_64/share/charsets --loose-console --performance-schema-enabled=maybe > /usr/local/mariadb-10.2.32-linux-x86_64/mysql-test/var/2/tmp/bad_option_1.txt 2>&1
mysqltest: At line 11: command "$MYSQLD_BOOTSTRAP_CMD --loose-console --performance-schema-enabled=maybe > $outfile 2>&1" failed with wrong error: 134
 
 - found 'core.723' (0/0)
 
Trying 'dbx' to get a backtrace
 
Trying 'gdb' to get a backtrace from coredump /usr/local/mariadb-10.2.32-linux-x86_64/mysql-test/var/2/log/perfschema.bad_option_1/mysqld.1/data/core.723
Core generated by '/usr/local/mariadb-10.2.32-linux-x86_64/bin/mysqld'
Output from gdb follows. The first stack trace is from the failing thread.
The following stack traces are from all threads (so the failing one is
duplicated).
--------------------------
[New Thread 765]
[New Thread 767]
[New Thread 723]
[New Thread 762]
[New Thread 729]
[New Thread 734]
[New Thread 735]
[New Thread 736]
[New Thread 744]
[New Thread 746]
[New Thread 748]
[New Thread 759]
[New Thread 737]
[New Thread 738]
[New Thread 745]
[New Thread 747]
[New Thread 749]
[Thread debugging using libthread_db enabled]
Core was generated by `/usr/local/mariadb-10.2.32-linux-x86_64/bin/mysqld --no-defaults --disable-geto'.
Program terminated with signal 6, Aborted.
#0  0x00007f18b1b5663c in pthread_kill () from /lib64/libpthread.so.0
#0  0x00007f18b1b5663c in pthread_kill () from /lib64/libpthread.so.0
#1  0x00007f18b2d3b8b5 in my_write_core ()
#2  0x00007f18b26b70e8 in handle_fatal_signal ()
#3  <signal handler called>
#4  0x00007f18b0a1e885 in raise () from /lib64/libc.so.6
#5  0x00007f18b0a20065 in abort () from /lib64/libc.so.6
#6  0x00007f18b0a179fe in __assert_fail_base () from /lib64/libc.so.6
#7  0x00007f18b0a17ac0 in __assert_fail () from /lib64/libc.so.6
#8  0x00007f18b2a03dd4 in rw_lock_create_func(rw_lock_t*, latch_level_t, char const*, unsigned int) () at /home/buildbot/buildbot/build/storage/innobase/include/ut0lst.h:160
#9  0x00007f18b29eefef in pfs_rw_lock_create_func(unsigned int, rw_lock_t*, latch_level_t, char const*, unsigned int) () at /home/buildbot/buildbot/build/storage/innobase/include/sync0rw.ic:508
#10 0x00007f18b29f21cb in srv_worker_thread () at /home/buildbot/buildbot/build/storage/innobase/srv/srv0srv.cc:2607
#11 0x00007f18b1b517f1 in start_thread () from /lib64/libpthread.so.0
#12 0x00007f18b0ad1ccd in clone () from /lib64/libc.so.6
 
Thread 17 (Thread 0x7f18918d4700 (LWP 749)):
#0  0x00007f18b1b5575b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f18b2918609 in os_event::timed_wait(timespec const*) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:275
#2  0x00007f18b29188e1 in os_event::wait_time_low(unsigned long, long) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:385
#3  0x00007f18b2918cb9 in os_event_wait_time_low(os_event*, unsigned long, long) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:490
#4  0x00007f18b2ac7ce7 in buf_flush_page_cleaner_coordinator () at /home/buildbot/buildbot/build/storage/innobase/buf/buf0flu.cc:2707
#5  0x00007f18b1b517f1 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f18b0ad1ccd in clone () from /lib64/libc.so.6
 
Thread 16 (Thread 0x7f1892cd6700 (LWP 747)):
#0  0x00007f18b1949614 in ?? () from /lib64/libaio.so.1
#1  0x00007f18b2912b89 in LinuxAIOHandler::collect() () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:1936
#2  0x00007f18b291554e in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2097
#3  0x00007f18b2915cfc in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2151
#4  0x00007f18b2b2ce68 in fil_aio_wait(unsigned long) () at /home/buildbot/buildbot/build/storage/innobase/fil/fil0fil.cc:5112
#5  0x00007f18b29f682f in io_handler_thread () at /home/buildbot/buildbot/build/storage/innobase/srv/srv0start.cc:332
#6  0x00007f18b1b517f1 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f18b0ad1ccd in clone () from /lib64/libc.so.6
 
Thread 15 (Thread 0x7f18940d8700 (LWP 745)):
#0  0x00007f18b1949614 in ?? () from /lib64/libaio.so.1
#1  0x00007f18b2912b89 in LinuxAIOHandler::collect() () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:1936
#2  0x00007f18b291554e in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2097
#3  0x00007f18b2915cfc in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2151
#4  0x00007f18b2b2ce68 in fil_aio_wait(unsigned long) () at /home/buildbot/buildbot/build/storage/innobase/fil/fil0fil.cc:5112
#5  0x00007f18b29f682f in io_handler_thread () at /home/buildbot/buildbot/build/storage/innobase/srv/srv0start.cc:332
#6  0x00007f18b1b517f1 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f18b0ad1ccd in clone () from /lib64/libc.so.6
 
Thread 14 (Thread 0x7f18954da700 (LWP 738)):
#0  0x00007f18b1949614 in ?? () from /lib64/libaio.so.1
#1  0x00007f18b2912b89 in LinuxAIOHandler::collect() () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:1936
#2  0x00007f18b291554e in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2097
#3  0x00007f18b2915cfc in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2151
#4  0x00007f18b2b2ce68 in fil_aio_wait(unsigned long) () at /home/buildbot/buildbot/build/storage/innobase/fil/fil0fil.cc:5112
#5  0x00007f18b29f682f in io_handler_thread () at /home/buildbot/buildbot/build/storage/innobase/srv/srv0start.cc:332
#6  0x00007f18b1b517f1 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f18b0ad1ccd in clone () from /lib64/libc.so.6
 
Thread 13 (Thread 0x7f1895edb700 (LWP 737)):
#0  0x00007f18b1949614 in ?? () from /lib64/libaio.so.1
#1  0x00007f18b2912b89 in LinuxAIOHandler::collect() () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:1936
#2  0x00007f18b291554e in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2097
#3  0x00007f18b2915cfc in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2151
#4  0x00007f18b2b2ce68 in fil_aio_wait(unsigned long) () at /home/buildbot/buildbot/build/storage/innobase/fil/fil0fil.cc:5112
#5  0x00007f18b29f682f in io_handler_thread () at /home/buildbot/buildbot/build/storage/innobase/srv/srv0start.cc:332
#6  0x00007f18b1b517f1 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f18b0ad1ccd in clone () from /lib64/libc.so.6
 
Thread 12 (Thread 0x7f188b5fe700 (LWP 759)):
#0  0x00007f18b1b58ccd in nanosleep () from /lib64/libpthread.so.0
#1  0x00007f18b2918ecb in os_thread_sleep(unsigned long) ()
#2  0x00007f18b29f0b31 in srv_master_thread () at /home/buildbot/buildbot/build/storage/innobase/srv/srv0srv.cc:2416
#3  0x00007f18b1b517f1 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f18b0ad1ccd in clone () from /lib64/libc.so.6
 
Thread 11 (Thread 0x7f18922d5700 (LWP 748)):
#0  0x00007f18b1949614 in ?? () from /lib64/libaio.so.1
#1  0x00007f18b2912b89 in LinuxAIOHandler::collect() () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:1936
#2  0x00007f18b291554e in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2097
#3  0x00007f18b2915cfc in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2151
#4  0x00007f18b2b2ce68 in fil_aio_wait(unsigned long) () at /home/buildbot/buildbot/build/storage/innobase/fil/fil0fil.cc:5112
#5  0x00007f18b29f682f in io_handler_thread () at /home/buildbot/buildbot/build/storage/innobase/srv/srv0start.cc:332
#6  0x00007f18b1b517f1 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f18b0ad1ccd in clone () from /lib64/libc.so.6
 
Thread 10 (Thread 0x7f18936d7700 (LWP 746)):
#0  0x00007f18b1949614 in ?? () from /lib64/libaio.so.1
#1  0x00007f18b2912b89 in LinuxAIOHandler::collect() () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:1936
#2  0x00007f18b291554e in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2097
#3  0x00007f18b2915cfc in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2151
#4  0x00007f18b2b2ce68 in fil_aio_wait(unsigned long) () at /home/buildbot/buildbot/build/storage/innobase/fil/fil0fil.cc:5112
#5  0x00007f18b29f682f in io_handler_thread () at /home/buildbot/buildbot/build/storage/innobase/srv/srv0start.cc:332
#6  0x00007f18b1b517f1 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f18b0ad1ccd in clone () from /lib64/libc.so.6
 
Thread 9 (Thread 0x7f1894ad9700 (LWP 744)):
#0  0x00007f18b1949614 in ?? () from /lib64/libaio.so.1
#1  0x00007f18b2912b89 in LinuxAIOHandler::collect() () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:1936
#2  0x00007f18b291554e in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2097
#3  0x00007f18b2915cfc in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2151
#4  0x00007f18b2b2ce68 in fil_aio_wait(unsigned long) () at /home/buildbot/buildbot/build/storage/innobase/fil/fil0fil.cc:5112
#5  0x00007f18b29f682f in io_handler_thread () at /home/buildbot/buildbot/build/storage/innobase/srv/srv0start.cc:332
#6  0x00007f18b1b517f1 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f18b0ad1ccd in clone () from /lib64/libc.so.6
 
Thread 8 (Thread 0x7f18968dc700 (LWP 736)):
#0  0x00007f18b1949614 in ?? () from /lib64/libaio.so.1
#1  0x00007f18b2912b89 in LinuxAIOHandler::collect() () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:1936
#2  0x00007f18b291554e in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2097
#3  0x00007f18b2915cfc in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2151
#4  0x00007f18b2b2ce68 in fil_aio_wait(unsigned long) () at /home/buildbot/buildbot/build/storage/innobase/fil/fil0fil.cc:5112
#5  0x00007f18b29f682f in io_handler_thread () at /home/buildbot/buildbot/build/storage/innobase/srv/srv0start.cc:332
#6  0x00007f18b1b517f1 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f18b0ad1ccd in clone () from /lib64/libc.so.6
 
Thread 7 (Thread 0x7f18972dd700 (LWP 735)):
#0  0x00007f18b1949614 in ?? () from /lib64/libaio.so.1
#1  0x00007f18b2912b89 in LinuxAIOHandler::collect() () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:1936
#2  0x00007f18b291554e in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2097
#3  0x00007f18b2915cfc in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2151
#4  0x00007f18b2b2ce68 in fil_aio_wait(unsigned long) () at /home/buildbot/buildbot/build/storage/innobase/fil/fil0fil.cc:5112
#5  0x00007f18b29f682f in io_handler_thread () at /home/buildbot/buildbot/build/storage/innobase/srv/srv0start.cc:332
#6  0x00007f18b1b517f1 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f18b0ad1ccd in clone () from /lib64/libc.so.6
 
Thread 6 (Thread 0x7f1897cde700 (LWP 734)):
#0  0x00007f18b1949614 in ?? () from /lib64/libaio.so.1
#1  0x00007f18b2912b89 in LinuxAIOHandler::collect() () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:1936
#2  0x00007f18b291554e in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2097
#3  0x00007f18b2915cfc in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2151
#4  0x00007f18b2b2ce68 in fil_aio_wait(unsigned long) () at /home/buildbot/buildbot/build/storage/innobase/fil/fil0fil.cc:5112
#5  0x00007f18b29f682f in io_handler_thread () at /home/buildbot/buildbot/build/storage/innobase/srv/srv0start.cc:332
#6  0x00007f18b1b517f1 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f18b0ad1ccd in clone () from /lib64/libc.so.6
 
Thread 5 (Thread 0x7f18b05d2700 (LWP 729)):
#0  0x00007f18b1b5575b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f18b2d40ee6 in safe_cond_timedwait ()
#2  0x00007f18b2d415dd in timer_handler ()
#3  0x00007f18b1b517f1 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f18b0ad1ccd in clone () from /lib64/libc.so.6
 
Thread 4 (Thread 0x7f18897fb700 (LWP 762)):
#0  0x00007f18b29f2033 in srv_release_threads(srv_thread_type, unsigned long) () at /home/buildbot/buildbot/build/storage/innobase/include/ib0mutex.h:467
#1  0x00007f18b29f3598 in srv_purge_coordinator_thread () at /home/buildbot/buildbot/build/storage/innobase/srv/srv0srv.cc:2926
#2  0x00007f18b1b517f1 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f18b0ad1ccd in clone () from /lib64/libc.so.6
 
Thread 3 (Thread 0x7f18b1f78720 (LWP 723)):
#0  0x00007f18b1b58ccd in nanosleep () from /lib64/libpthread.so.0
#1  0x00007f18b2918ecb in os_thread_sleep(unsigned long) ()
#2  0x00007f18b28edf9d in logs_empty_and_mark_files_at_shutdown() () at /home/buildbot/buildbot/build/storage/innobase/log/log0log.cc:1836
#3  0x00007f18b29f6e3c in innodb_shutdown() () at /home/buildbot/buildbot/build/storage/innobase/srv/srv0start.cc:2794
#4  0x00007f18b286a994 in innobase_end(handlerton*, ha_panic_function) () at /home/buildbot/buildbot/build/storage/innobase/handler/ha_innodb.cc:4475
#5  0x00007f18b26b9a6f in ha_finalize_handlerton(st_plugin_int*) () at /home/buildbot/buildbot/build/sql/handler.cc:494
#6  0x00007f18b24edb52 in plugin_deinitialize(st_plugin_int*, bool) () at /home/buildbot/buildbot/build/sql/sql_plugin.cc:1220
#7  0x00007f18b24f19f4 in reap_plugins() () at /home/buildbot/buildbot/build/sql/sql_plugin.cc:1296
#8  0x00007f18b24f2458 in plugin_shutdown() () at /home/buildbot/buildbot/build/sql/sql_plugin.cc:1957
#9  0x00007f18b24245ef in clean_up(bool) () at /home/buildbot/buildbot/build/sql/mysqld.cc:2225
#10 0x00007f18b2429a37 in unireg_abort () at /home/buildbot/buildbot/build/sql/mysqld.cc:2143
#11 0x00007f18b242bd1b in init_server_components() () at /home/buildbot/buildbot/build/sql/mysqld.cc:5377
#12 0x00007f18b2431e70 in mysqld_main(int, char**) () at /home/buildbot/buildbot/build/sql/mysqld.cc:5937
#13 0x00007f18b2423dde in main ()
 
Thread 2 (Thread 0x7f186ffff700 (LWP 767)):
#0  0x00007f18b1b58ccd in nanosleep () from /lib64/libpthread.so.0
#1  0x00007f18b2918ecb in os_thread_sleep(unsigned long) ()
#2  0x00007f18b2a9dff8 in btr_defragment_thread () at /home/buildbot/buildbot/build/storage/innobase/btr/btr0defragment.cc:715
#3  0x00007f18b1b517f1 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f18b0ad1ccd in clone () from /lib64/libc.so.6
 
Thread 1 (Thread 0x7f187f5fe700 (LWP 765)):
#0  0x00007f18b1b5663c in pthread_kill () from /lib64/libpthread.so.0
#1  0x00007f18b2d3b8b5 in my_write_core ()
#2  0x00007f18b26b70e8 in handle_fatal_signal ()
#3  <signal handler called>
#4  0x00007f18b0a1e885 in raise () from /lib64/libc.so.6
#5  0x00007f18b0a20065 in abort () from /lib64/libc.so.6
#6  0x00007f18b0a179fe in __assert_fail_base () from /lib64/libc.so.6
#7  0x00007f18b0a17ac0 in __assert_fail () from /lib64/libc.so.6
#8  0x00007f18b2a03dd4 in rw_lock_create_func(rw_lock_t*, latch_level_t, char const*, unsigned int) () at /home/buildbot/buildbot/build/storage/innobase/include/ut0lst.h:160
#9  0x00007f18b29eefef in pfs_rw_lock_create_func(unsigned int, rw_lock_t*, latch_level_t, char const*, unsigned int) () at /home/buildbot/buildbot/build/storage/innobase/include/sync0rw.ic:508
#10 0x00007f18b29f21cb in srv_worker_thread () at /home/buildbot/buildbot/build/storage/innobase/srv/srv0srv.cc:2607
#11 0x00007f18b1b517f1 in start_thread () from /lib64/libpthread.so.0
#12 0x00007f18b0ad1ccd in clone () from /lib64/libc.so.6

Comment by Marko Mäkelä [ 2023-11-29 ]

The latest failure was almost a year ago:

bb-10.3-MDEV-30052 d79150eeef769ee3b54ff6abbeda5c4bc1df9864

perfschema.bad_option_1                  w15 [ fail ]
        Test ended at 2023-01-19 20:55:58
 
CURRENT_TEST: perfschema.bad_option_1
/home/buildbot/aarch64-ubuntu-2004-debug/build/client//mysqltest: Error on delete of '/home/buildbot/aarch64-ubuntu-2004-debug/build/mysql-test/var/15/tmp/bad_option_1.txt' (Errcode: 2 "No such file or directory")
mysqltest: At line 11: command "$MYSQLD_BOOTSTRAP_CMD --loose-console --performance-schema-enabled=maybe > $outfile 2>&1" failed with wrong error: 134
 
 - saving '/home/buildbot/aarch64-ubuntu-2004-debug/build/mysql-test/var/15/log/perfschema.bad_option_1/' to '/home/buildbot/aarch64-ubuntu-2004-debug/build/mysql-test/var/log/perfschema.bad_option_1/'

I did not find any failure for 10.5 or later. We may thank MDEV-16264 for fixing this.

Generated at Thu Feb 08 09:08:09 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.