[MDEV-22956] rpl.rpl_old_master failed in buildbot with InnoDB: Failing assertion: list.count > 0 Created: 2020-06-19  Updated: 2023-04-27

Status: Open
Project: MariaDB Server
Component/s: Storage Engine - InnoDB, Tests
Affects Version/s: 10.2, 10.3, 10.4
Fix Version/s: 10.4

Type: Bug Priority: Major
Reporter: Elena Stepanova Assignee: Marko Mäkelä
Resolution: Unresolved Votes: 1
Labels: None

Issue Links:
Relates
relates to MDEV-24489 Server crashes: double free or corrup... Closed
relates to MDEV-24495 InnoDB: Assertion failure in file ut0... Closed
relates to MDEV-24505 innodb.innodb-ucs2 test failed, asse... Closed

 Description   

http://buildbot.askmonty.org/buildbot/builders/kvm-rpm-centos73-ppc64le/builds/11012

2020-06-13 17:50:06 0x3fff8e73ebf0  InnoDB: Assertion failure in file /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.4.14/storage/innobase/include/ut0lst.h line 334
InnoDB: Failing assertion: list.count > 0

10.4 805340936aa47493886bafd119863d83

rpl.rpl_old_master 'innodb,stmt'         w4 [ fail ]  Found warnings/errors in server log file!
        Test ended at 2020-06-13 17:50:28
line
Attempting backtrace. You can use the following information to find out
^ Found warnings in /dev/shm/var/4/log/mysqld.1.err
ok
 
 - found 'core.25559' (0/0)
 
Trying 'dbx' to get a backtrace
 
Trying 'gdb' to get a backtrace from coredump /dev/shm/var/4/log/rpl.rpl_old_master-innodb,stmt/mysqld.1/data/core.25559
Core generated by '/usr/sbin/mysqld'
Output from gdb follows. The first stack trace is from the failing thread.
The following stack traces are from all threads (so the failing one is
duplicated).
--------------------------
[New LWP 25559]
[New LWP 25592]
[New LWP 25563]
[New LWP 25562]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/sbin/mysqld --defaults-group-suffix=.1 --defaults-file=/dev/shm/var/4/my.c'.
Program terminated with signal 6, Aborted.
#0  0x00003fff8e68f150 in pthread_kill () from /lib64/libpthread.so.0
#0  0x00003fff8e68f150 in pthread_kill () from /lib64/libpthread.so.0
#1  0x00000000270dfc40 in my_write_core (sig=<optimized out>) at /usr/src/debug/MariaDB-10.4.14/src_0/mysys/stacktrace.c:481
#2  0x0000000026a132e8 in handle_fatal_signal (sig=<optimized out>) at /usr/src/debug/MariaDB-10.4.14/src_0/sql/signal_handler.cc:343
#3  <signal handler called>
#4  0x00003fff8dd5eb18 in raise () from /lib64/libc.so.6
#5  0x00003fff8dd60c9c in abort () from /lib64/libc.so.6
#6  0x0000000026637694 in ut_dbg_assertion_failed (expr=0x2730ecc8 "list.count > 0", file=0x2730ecd8 "/home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.4.14/storage/innobase/include/ut0lst.h", line=<optimized out>) at /usr/src/debug/MariaDB-10.4.14/src_0/storage/innobase/ut/ut0dbg.cc:60
#7  0x0000000026d9c31c in ut_list_remove<ut_list_base<rw_lock_t, ut_list_node<rw_lock_t> rw_lock_t::*>, GenericGetNode<rw_lock_t> > (get_node=..., node=..., list=...) at /usr/src/debug/MariaDB-10.4.14/src_0/storage/innobase/include/ut0lst.h:334
#8  ut_list_remove<ut_list_base<rw_lock_t, ut_list_node<rw_lock_t> rw_lock_t::*> > (elem=0x1003ac96d50, list=...) at /usr/src/debug/MariaDB-10.4.14/src_0/storage/innobase/include/ut0lst.h:386
#9  rw_lock_free_func (lock=0x1003ac96d50) at /usr/src/debug/MariaDB-10.4.14/src_0/storage/innobase/sync/sync0rw.cc:260
#10 0x0000000026ed1908 in pfs_rw_lock_free_func (lock=0x1003ac96d50) at /usr/src/debug/MariaDB-10.4.14/src_0/storage/innobase/include/sync0rw.ic:605
#11 ha_clear (table=0x1003ac94480) at /usr/src/debug/MariaDB-10.4.14/src_0/storage/innobase/ha/ha0ha.cc:171
#12 0x0000000026e31e50 in buf_pool_free_instance (buf_pool=0x1003ac6ab90) at /usr/src/debug/MariaDB-10.4.14/src_0/storage/innobase/buf/buf0buf.cc:2070
#13 buf_pool_free (n_instances=1) at /usr/src/debug/MariaDB-10.4.14/src_0/storage/innobase/buf/buf0buf.cc:2136
#14 0x0000000026d95780 in innodb_shutdown () at /usr/src/debug/MariaDB-10.4.14/src_0/storage/innobase/srv/srv0start.cc:2541
#15 0x0000000026c59860 in innobase_end () at /usr/src/debug/MariaDB-10.4.14/src_0/storage/innobase/handler/ha_innodb.cc:4311
#16 0x0000000026a179c4 in ha_finalize_handlerton (plugin=0x1003aa9de80) at /usr/src/debug/MariaDB-10.4.14/src_0/sql/handler.cc:496
#17 0x0000000026784bd4 in plugin_deinitialize (plugin=0x1003aa9de80, ref_check=true) at /usr/src/debug/MariaDB-10.4.14/src_0/sql/sql_plugin.cc:1241
#18 0x0000000026788848 in reap_plugins () at /usr/src/debug/MariaDB-10.4.14/src_0/sql/sql_plugin.cc:1317
#19 0x0000000026789810 in plugin_shutdown () at /usr/src/debug/MariaDB-10.4.14/src_0/sql/sql_plugin.cc:1986
#20 0x0000000026672ad4 in clean_up (print_message=true) at /usr/src/debug/MariaDB-10.4.14/src_0/sql/mysqld.cc:1999
#21 0x0000000026680678 in clean_up (print_message=true) at /usr/src/debug/MariaDB-10.4.14/src_0/sql/mysqld.cc:5935
#22 mysqld_main (argc=149, argv=0x1003a92bc48) at /usr/src/debug/MariaDB-10.4.14/src_0/sql/mysqld.cc:5941
#23 0x00000000266426a0 in main (argc=<optimized out>, argv=<optimized out>) at /usr/src/debug/MariaDB-10.4.14/src_0/sql/main.cc:25
 
Thread 4 (Thread 0x3fff890af070 (LWP 25562)):
#0  0x00003fff8e68e144 in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
#1  0x00000000270e6728 in inline_mysql_cond_timedwait (that=0x2825b118 <COND_timer>, mutex=0x2825b0b0 <LOCK_timer>, src_file=0x2736c7f8 "/home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.4.14/mysys/thr_timer.c", src_line=292, abstime=0x3fff890ae598) at /usr/src/debug/MariaDB-10.4.14/src_0/include/mysql/psi/mysql_thread.h:1211
#2  timer_handler (arg=<optimized out>) at /usr/src/debug/MariaDB-10.4.14/src_0/mysys/thr_timer.c:292
#3  0x0000000027075f10 in pfs_spawn_thread (arg=0x1003aa89b28) at /usr/src/debug/MariaDB-10.4.14/src_0/storage/perfschema/pfs.cc:1869
#4  0x00003fff8e688728 in start_thread () from /lib64/libpthread.so.0
#5  0x00003fff8de3d210 in clone () from /lib64/libc.so.6
 
Thread 3 (Thread 0x3fff8384f070 (LWP 25563)):
#0  0x00003fff8e68e144 in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
#1  0x0000000026f755bc in inline_mysql_cond_timedwait (src_file=0x27356c48 "/home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.4.14/storage/maria/ma_servicethread.c", src_line=116, abstime=<optimized out>, mutex=0x2824a520 <LOCK_checkpoint>, that=0x2824a558 <COND_checkpoint>) at /usr/src/debug/MariaDB-10.4.14/src_0/include/mysql/psi/mysql_thread.h:1211
#2  my_service_thread_sleep (control=0x2791b890 <checkpoint_control>, sleep_time=<optimized out>) at /usr/src/debug/MariaDB-10.4.14/src_0/storage/maria/ma_servicethread.c:115
#3  0x0000000026f69354 in ma_checkpoint_background (arg=0x1e) at /usr/src/debug/MariaDB-10.4.14/src_0/storage/maria/ma_checkpoint.c:707
#4  0x0000000027075f10 in pfs_spawn_thread (arg=0x1003ab40fa8) at /usr/src/debug/MariaDB-10.4.14/src_0/storage/perfschema/pfs.cc:1869
#5  0x00003fff8e688728 in start_thread () from /lib64/libpthread.so.0
#6  0x00003fff8de3d210 in clone () from /lib64/libc.so.6
 
Thread 2 (Thread 0x3fff825e8070 (LWP 25592)):
#0  0x00003fff8dd60448 in sigwaitinfo () from /lib64/libc.so.6
#1  0x000000002667385c in my_sigwait (code=<synthetic pointer>, sig=0x3fff825e74e0, set=0x3fff825e74f0) at /usr/src/debug/MariaDB-10.4.14/src_0/include/my_pthread.h:196
#2  signal_hand (arg=<optimized out>) at /usr/src/debug/MariaDB-10.4.14/src_0/sql/mysqld.cc:3253
#3  0x0000000027075f10 in pfs_spawn_thread (arg=0x1003afeeb78) at /usr/src/debug/MariaDB-10.4.14/src_0/storage/perfschema/pfs.cc:1869
#4  0x00003fff8e688728 in start_thread () from /lib64/libpthread.so.0
#5  0x00003fff8de3d210 in clone () from /lib64/libc.so.6
 
Thread 1 (Thread 0x3fff8e73ebf0 (LWP 25559)):
#0  0x00003fff8e68f150 in pthread_kill () from /lib64/libpthread.so.0
#1  0x00000000270dfc40 in my_write_core (sig=<optimized out>) at /usr/src/debug/MariaDB-10.4.14/src_0/mysys/stacktrace.c:481
#2  0x0000000026a132e8 in handle_fatal_signal (sig=<optimized out>) at /usr/src/debug/MariaDB-10.4.14/src_0/sql/signal_handler.cc:343
#3  <signal handler called>
#4  0x00003fff8dd5eb18 in raise () from /lib64/libc.so.6
#5  0x00003fff8dd60c9c in abort () from /lib64/libc.so.6
#6  0x0000000026637694 in ut_dbg_assertion_failed (expr=0x2730ecc8 "list.count > 0", file=0x2730ecd8 "/home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.4.14/storage/innobase/include/ut0lst.h", line=<optimized out>) at /usr/src/debug/MariaDB-10.4.14/src_0/storage/innobase/ut/ut0dbg.cc:60
#7  0x0000000026d9c31c in ut_list_remove<ut_list_base<rw_lock_t, ut_list_node<rw_lock_t> rw_lock_t::*>, GenericGetNode<rw_lock_t> > (get_node=..., node=..., list=...) at /usr/src/debug/MariaDB-10.4.14/src_0/storage/innobase/include/ut0lst.h:334
#8  ut_list_remove<ut_list_base<rw_lock_t, ut_list_node<rw_lock_t> rw_lock_t::*> > (elem=0x1003ac96d50, list=...) at /usr/src/debug/MariaDB-10.4.14/src_0/storage/innobase/include/ut0lst.h:386
#9  rw_lock_free_func (lock=0x1003ac96d50) at /usr/src/debug/MariaDB-10.4.14/src_0/storage/innobase/sync/sync0rw.cc:260
#10 0x0000000026ed1908 in pfs_rw_lock_free_func (lock=0x1003ac96d50) at /usr/src/debug/MariaDB-10.4.14/src_0/storage/innobase/include/sync0rw.ic:605
#11 ha_clear (table=0x1003ac94480) at /usr/src/debug/MariaDB-10.4.14/src_0/storage/innobase/ha/ha0ha.cc:171
#12 0x0000000026e31e50 in buf_pool_free_instance (buf_pool=0x1003ac6ab90) at /usr/src/debug/MariaDB-10.4.14/src_0/storage/innobase/buf/buf0buf.cc:2070
#13 buf_pool_free (n_instances=1) at /usr/src/debug/MariaDB-10.4.14/src_0/storage/innobase/buf/buf0buf.cc:2136
#14 0x0000000026d95780 in innodb_shutdown () at /usr/src/debug/MariaDB-10.4.14/src_0/storage/innobase/srv/srv0start.cc:2541
#15 0x0000000026c59860 in innobase_end () at /usr/src/debug/MariaDB-10.4.14/src_0/storage/innobase/handler/ha_innodb.cc:4311
#16 0x0000000026a179c4 in ha_finalize_handlerton (plugin=0x1003aa9de80) at /usr/src/debug/MariaDB-10.4.14/src_0/sql/handler.cc:496
#17 0x0000000026784bd4 in plugin_deinitialize (plugin=0x1003aa9de80, ref_check=true) at /usr/src/debug/MariaDB-10.4.14/src_0/sql/sql_plugin.cc:1241
#18 0x0000000026788848 in reap_plugins () at /usr/src/debug/MariaDB-10.4.14/src_0/sql/sql_plugin.cc:1317
#19 0x0000000026789810 in plugin_shutdown () at /usr/src/debug/MariaDB-10.4.14/src_0/sql/sql_plugin.cc:1986
#20 0x0000000026672ad4 in clean_up (print_message=true) at /usr/src/debug/MariaDB-10.4.14/src_0/sql/mysqld.cc:1999
#21 0x0000000026680678 in clean_up (print_message=true) at /usr/src/debug/MariaDB-10.4.14/src_0/sql/mysqld.cc:5935
#22 mysqld_main (argc=149, argv=0x1003a92bc48) at /usr/src/debug/MariaDB-10.4.14/src_0/sql/mysqld.cc:5941
#23 0x00000000266426a0 in main (argc=<optimized out>, argv=<optimized out>) at /usr/src/debug/MariaDB-10.4.14/src_0/sql/main.cc:25



 Comments   
Comment by Elena Stepanova [ 2020-08-04 ]

http://buildbot.askmonty.org/buildbot/builders/kvm-deb-jessie-x86/builds/9896/steps/mtr/logs/stdio

10.4 b99fa1e767

sql_sequence.read_only 'innodb'          w3 [ fail ]  Found warnings/errors in server log file!
        Test ended at 2020-07-04 18:03:50
line
Attempting backtrace. You can use the following information to find out
^ Found warnings in /dev/shm/var/3/log/mysqld.1.err
ok
 
 - found 'core' (0/0)
 
Trying 'dbx' to get a backtrace
 
Trying 'gdb' to get a backtrace from coredump /dev/shm/var/3/log/sql_sequence.read_only-innodb/mysqld.1/data/core
Core generated by '/usr/sbin/mysqld'
Output from gdb follows. The first stack trace is from the failing thread.
The following stack traces are from all threads (so the failing one is
duplicated).
--------------------------
[New LWP 17134]
[New LWP 17137]
[New LWP 17168]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/i386-linux-gnu/i686/cmov/libthread_db.so.1".
Core was generated by `/usr/sbin/mysqld --defaults-group-suffix=.1 --defaults-file=/dev/shm/var/3/my.c'.
Program terminated with signal SIGABRT, Aborted.
#0  0xb653cd3c in __kernel_vsyscall ()
#0  0xb653cd3c in __kernel_vsyscall ()
#1  0xb6522d03 in __pthread_kill (threadid=3051718464, signo=6) at ../nptl/sysdeps/unix/sysv/linux/pthread_kill.c:61
#2  0xb71407cc in my_write_core ()
#3  0xb6b95042 in handle_fatal_signal ()
#4  <signal handler called>
#5  0xb653cd3c in __kernel_vsyscall ()
#6  0xb5fb6367 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#7  0xb5fb7a23 in __GI_abort () at abort.c:89
#8  0xb685888c in ?? ()
#9  0xb6e498f0 in ?? ()
#10 0xb6f5d7ad in ?? ()
#11 0xb6ecc4d1 in ?? ()
#12 0xb6e436fa in ?? ()
#13 0xb6d28282 in ?? ()
#14 0xb6b9821b in ha_finalize_handlerton(st_plugin_int*) ()
#15 0xb696373b in ?? ()
#16 0xb696770d in ?? ()
#17 0xb6968504 in plugin_shutdown() ()
#18 0xb688434c in ?? ()
#19 0xb68915d7 in mysqld_main(int, char**) ()
#20 0xb685f2f7 in main ()
 
Thread 3 (Thread 0xaff5cb40 (LWP 17168)):
#0  0xb653cd3c in __kernel_vsyscall ()
#1  0xb5fb73fa in do_sigwaitinfo (set=0xfffffffc, set@entry=0xaff5c1a4, info=info@entry=0xaff5c224) at ../nptl/sysdeps/unix/sysv/linux/../../../../../sysdeps/unix/sysv/linux/sigwaitinfo.c:54
#2  0xb5fb7499 in __GI___sigwaitinfo (set=0xaff5c1a4, info=0xaff5c224) at ../nptl/sysdeps/unix/sysv/linux/../../../../../sysdeps/unix/sysv/linux/sigwaitinfo.c:81
#3  0xb6885ff4 in signal_hand ()
#4  0xb70df4eb in ?? ()
#5  0xb651decb in start_thread (arg=0xaff5cb40) at pthread_create.c:309
#6  0xb6073d0e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129
 
Thread 2 (Thread 0xb1bbdb40 (LWP 17137)):
#0  0xb653cd3c in __kernel_vsyscall ()
#1  0xb6521fe5 in pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_timedwait.S:245
#2  0xb714531d in ?? ()
#3  0xb70df4eb in ?? ()
#4  0xb651decb in start_thread (arg=0xb1bbdb40) at pthread_create.c:309
#5  0xb6073d0e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129
 
Thread 1 (Thread 0xb5e58740 (LWP 17134)):
#0  0xb653cd3c in __kernel_vsyscall ()
#1  0xb6522d03 in __pthread_kill (threadid=3051718464, signo=6) at ../nptl/sysdeps/unix/sysv/linux/pthread_kill.c:61
#2  0xb71407cc in my_write_core ()
#3  0xb6b95042 in handle_fatal_signal ()
#4  <signal handler called>
#5  0xb653cd3c in __kernel_vsyscall ()
#6  0xb5fb6367 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#7  0xb5fb7a23 in __GI_abort () at abort.c:89
#8  0xb685888c in ?? ()
#9  0xb6e498f0 in ?? ()
#10 0xb6f5d7ad in ?? ()
#11 0xb6ecc4d1 in ?? ()
#12 0xb6e436fa in ?? ()
#13 0xb6d28282 in ?? ()
#14 0xb6b9821b in ha_finalize_handlerton(st_plugin_int*) ()
#15 0xb696373b in ?? ()
#16 0xb696770d in ?? ()
#17 0xb6968504 in plugin_shutdown() ()
#18 0xb688434c in ?? ()
#19 0xb68915d7 in mysqld_main(int, char**) ()
#20 0xb685f2f7 in main ()

10.4 b99fa1e767

2020-07-04 18:03:44 0 [Note] /usr/sbin/mysqld (initiated by: root[root] @ localhost []): Normal shutdown
2020-07-04 18:03:44 0 [Note] Event Scheduler: Purging the queue. 0 events
2020-07-04 18:03:44 0 [Note] InnoDB: FTS optimize thread exiting.
2020-07-04 18:03:44 0 [Note] InnoDB: Starting shutdown...
2020-07-04 18:03:44 0 [Note] InnoDB: Dumping buffer pool(s) to /dev/shm/var/3/mysqld.1/data/ib_buffer_pool
2020-07-04 18:03:44 0 [Note] InnoDB: Instance 0, restricted to 126 pages due to innodb_buf_pool_dump_pct=25
2020-07-04 18:03:44 0 [Note] InnoDB: Buffer pool(s) dump completed at 200704 18:03:44
2020-07-04 18:03:46 0xb5e58740  InnoDB: Assertion failure in file /home/buildbot/buildbot/build/mariadb-10.4.14/storage/innobase/include/ut0lst.h line 334
InnoDB: Failing assertion: list.count > 0
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
InnoDB: about forcing recovery.
200704 18:03:46 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
Server version: 10.4.14-MariaDB-1:10.4.14+maria~jessie-log
key_buffer_size=1048576
read_buffer_size=131072
max_used_connections=2
max_threads=153
thread_count=0
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 62159 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x49000
/usr/sbin/mysqld(my_print_stacktrace+0x28)[0xb7140be8]
/usr/sbin/mysqld(handle_fatal_signal+0x301)[0xb6b94fc1]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xb653cd18]
linux-gate.so.1(__kernel_vsyscall+0x10)[0xb653cd3c]
/lib/i386-linux-gnu/i686/cmov/libc.so.6(gsignal+0x47)[0xb5fb6367]
/lib/i386-linux-gnu/i686/cmov/libc.so.6(abort+0x143)[0xb5fb7a23]
/usr/sbin/mysqld(+0x2f788c)[0xb685888c]
/usr/sbin/mysqld(+0x8e88f0)[0xb6e498f0]
/usr/sbin/mysqld(+0x9fc7ad)[0xb6f5d7ad]
/usr/sbin/mysqld(+0x96b4d1)[0xb6ecc4d1]
/usr/sbin/mysqld(+0x8e26fa)[0xb6e436fa]
/usr/sbin/mysqld(+0x7c7282)[0xb6d28282]
/usr/sbin/mysqld(_Z22ha_finalize_handlertonP13st_plugin_int+0x3b)[0xb6b9821b]
/usr/sbin/mysqld(+0x40273b)[0xb696373b]
/usr/sbin/mysqld(+0x40670d)[0xb696770d]
/usr/sbin/mysqld(_Z15plugin_shutdownv+0xd4)[0xb6968504]
/usr/sbin/mysqld(+0x32334c)[0xb688434c]
/usr/sbin/mysqld(_Z11mysqld_mainiPPc+0x1fe7)[0xb68915d7]
/usr/sbin/mysqld(main+0x27)[0xb685f2f7]
/lib/i386-linux-gnu/i686/cmov/libc.so.6(__libc_start_main+0xf3)[0xb5fa1a63]
/usr/sbin/mysqld(+0x321c00)[0xb6882c00]
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /dev/shm/var/3/mysqld.1/data
Resource Limits:
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        unlimited            unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             48521                48521                processes 
Max open files            1024                 1024                 files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       48521                48521                signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us        
Core pattern: core

Comment by Alice Sherepa [ 2020-12-31 ]

http://buildbot.askmonty.org/buildbot/builders/kvm-deb-stretch-x86/builds/8464/steps/mtr/logs/stdio

10.2 42e1815ad850384fad292534665ff61b

sys_vars.innodb_buffer_pool_dump_at_shutdown_basic 'innodb' w3 [ fail ]  Found warnings/errors in server log file!
        Test ended at 2020-10-27 12:34:30
line
Attempting backtrace. You can use the following information to find out
^ Found warnings in /dev/shm/var/3/log/mysqld.1.err
ok
 
 - found 'core' (0/0)
 
Trying 'dbx' to get a backtrace
 
Trying 'gdb' to get a backtrace from coredump /dev/shm/var/3/log/sys_vars.innodb_buffer_pool_dump_at_shutdown_basic-innodb/mysqld.1/data/core
Core generated by '/usr/sbin/mysqld'
Output from gdb follows. The first stack trace is from the failing thread.
The following stack traces are from all threads (so the failing one is
duplicated).
--------------------------
[New LWP 12357]
[New LWP 5167]
[New LWP 5141]
[New LWP 5136]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/mysqld --defaults-group-suffix=.1 --defaults-file=/dev/shm/var/3/my.c'.
Program terminated with signal SIGABRT, Aborted.
#0  0xb7747cf9 in __kernel_vsyscall ()
[Current thread is 1 (Thread 0xa4e69b40 (LWP 12357))]
#0  0xb7747cf9 in __kernel_vsyscall ()
#1  0xb772ce41 in __pthread_kill (threadid=2766576448, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:55
#2  0x80a83eac in my_write_core (sig=6) at ./mysys/stacktrace.c:382
#3  0x804fbf0b in handle_fatal_signal (sig=6) at ./sql/signal_handler.cc:343
#4  <signal handler called>
#5  0xb7747cf9 in __kernel_vsyscall ()
#6  0xb6ff3dd0 in __libc_signal_restore_set (set=0xa4e68a30) at ../sysdeps/unix/sysv/linux/nptl-signals.h:79
#7  __GI_raise (sig=6) at ../sysdeps/unix/sysv/linux/raise.c:48
#8  0xb6ff5297 in __GI_abort () at abort.c:89
#9  0x8027b392 in ut_dbg_assertion_failed (expr=0x80bc00e3 "list.count > 0", file=0x80bbf958 "/home/buildbot/buildbot/build/mariadb-10.2.35/storage/innobase/include/ut0lst.h", line=334) at ./storage/innobase/ut/ut0dbg.cc:60
#10 0x807aa516 in ut_list_remove<ut_list_base<rw_lock_t, ut_list_node<rw_lock_t> rw_lock_t::*>, GenericGetNode<rw_lock_t> > (get_node=..., node=..., list=...) at ./storage/innobase/include/ut0lst.h:334
#11 ut_list_remove<ut_list_base<rw_lock_t, ut_list_node<rw_lock_t> rw_lock_t::*> > (elem=0x82684e9c, list=...) at ./storage/innobase/include/ut0lst.h:386
#12 rw_lock_free_func (lock=<optimized out>) at ./storage/innobase/sync/sync0rw.cc:267
#13 0x808c819a in pfs_rw_lock_free_func (lock=<optimized out>) at ./storage/innobase/include/sync0rw.ic:603
#14 ha_clear (table=0x826838f8) at ./storage/innobase/ha/ha0ha.cc:168
#15 0x80832ace in buf_pool_free_instance (buf_pool=0x826625b8) at ./storage/innobase/buf/buf0buf.cc:2017
#16 buf_pool_free (n_instances=1) at ./storage/innobase/buf/buf0buf.cc:2100
#17 0x807a233e in innodb_shutdown () at ./storage/innobase/srv/srv0start.cc:2907
#18 0x8067f84c in innobase_end () at ./storage/innobase/handler/ha_innodb.cc:4420
#19 0x804fe56b in ha_finalize_handlerton (plugin=0x825c0e60) at ./sql/handler.cc:494
#20 0x8034e5c8 in plugin_deinitialize (plugin=0x825c0e60, ref_check=ref_check@entry=true) at ./sql/sql_plugin.cc:1220
#21 0x803531cd in reap_plugins () at ./sql/sql_plugin.cc:1296
#22 0x80354295 in plugin_shutdown () at ./sql/sql_plugin.cc:1957
#23 0x80294ddb in clean_up (print_message=print_message@entry=true) at ./sql/mysqld.cc:2223
#24 0x802955e1 in clean_up (print_message=true) at ./sql/mysqld.cc:2104
#25 unireg_end () at ./sql/mysqld.cc:2098
#26 0x8029844d in kill_server (sig_ptr=0x0) at ./sql/mysqld.cc:2025
#27 0x802988fc in kill_server_thread (arg=0xad9b0184) at ./sql/mysqld.cc:2048
#28 0x80a36dcb in pfs_spawn_thread (arg=<optimized out>) at ./storage/perfschema/pfs.cc:1869
#29 0xb772627a in start_thread (arg=0xa4e69b40) at pthread_create.c:456
#30 0xb70b0366 in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:110
 
Thread 4 (Thread 0xb6de57c0 (LWP 5136)):
#0  0xb7747cf9 in __kernel_vsyscall ()
#1  0xb772bbeb in pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/i386/pthread_cond_wait.S:187
#2  0x802a01c1 in inline_mysql_cond_wait (that=0x810e1ca0 <COND_thread_count>, mutex=0x810e2084 <LOCK_thread_count>, src_file=0x80aff088 "/home/buildbot/buildbot/build/mariadb-10.2.35/sql/mysqld.cc", src_line=6162) at ./include/mysql/psi/mysql_thread.h:1145
#3  mysqld_main (argc=<optimized out>, argv=<optimized out>) at ./sql/mysqld.cc:6162
#4  0x80280507 in main (argc=22, argv=0xbfea3ab4) at ./sql/main.cc:25
 
Thread 3 (Thread 0xb2cd5b40 (LWP 5141)):
#0  0xb7747cf9 in __kernel_vsyscall ()
#1  0xb772bfb5 in pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/i386/i686/../pthread_cond_timedwait.S:245
#2  0x80a886f4 in inline_mysql_cond_timedwait (that=0x8194f260 <COND_timer>, mutex=0x8194f294 <LOCK_timer>, src_file=0x80c13e04 "/home/buildbot/buildbot/build/mariadb-10.2.35/mysys/thr_timer.c", src_line=292, abstime=0xb2cd5224) at ./include/mysql/psi/mysql_thread.h:1182
#3  timer_handler (arg=0x0) at ./mysys/thr_timer.c:292
#4  0x80a36dcb in pfs_spawn_thread (arg=<optimized out>) at ./storage/perfschema/pfs.cc:1869
#5  0xb772627a in start_thread (arg=0xb2cd5b40) at pthread_create.c:456
#6  0xb70b0366 in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:110
 
Thread 2 (Thread 0xad9b0b40 (LWP 5167)):
#0  0xb7747cf9 in __kernel_vsyscall ()
#1  0xb773006d in do_sigwait (set=<optimized out>, set@entry=0xad9b01b4, sig=sig@entry=0xad9b0184) at ../sysdeps/unix/sysv/linux/sigwait.c:61
#2  0xb77300fe in __sigwait (set=0xad9b01b4, sig=0xad9b0184) at ../sysdeps/unix/sysv/linux/sigwait.c:96
#3  0x8029426a in signal_hand (arg=0x0) at ./sql/mysqld.cc:3510
#4  0x80a36dcb in pfs_spawn_thread (arg=<optimized out>) at ./storage/perfschema/pfs.cc:1869
#5  0xb772627a in start_thread (arg=0xad9b0b40) at pthread_create.c:456
#6  0xb70b0366 in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:110
 
Thread 1 (Thread 0xa4e69b40 (LWP 12357)):
#0  0xb7747cf9 in __kernel_vsyscall ()
#1  0xb772ce41 in __pthread_kill (threadid=2766576448, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:55
#2  0x80a83eac in my_write_core (sig=6) at ./mysys/stacktrace.c:382
#3  0x804fbf0b in handle_fatal_signal (sig=6) at ./sql/signal_handler.cc:343
#4  <signal handler called>
#5  0xb7747cf9 in __kernel_vsyscall ()
#6  0xb6ff3dd0 in __libc_signal_restore_set (set=0xa4e68a30) at ../sysdeps/unix/sysv/linux/nptl-signals.h:79
#7  __GI_raise (sig=6) at ../sysdeps/unix/sysv/linux/raise.c:48
#8  0xb6ff5297 in __GI_abort () at abort.c:89
#9  0x8027b392 in ut_dbg_assertion_failed (expr=0x80bc00e3 "list.count > 0", file=0x80bbf958 "/home/buildbot/buildbot/build/mariadb-10.2.35/storage/innobase/include/ut0lst.h", line=334) at ./storage/innobase/ut/ut0dbg.cc:60
#10 0x807aa516 in ut_list_remove<ut_list_base<rw_lock_t, ut_list_node<rw_lock_t> rw_lock_t::*>, GenericGetNode<rw_lock_t> > (get_node=..., node=..., list=...) at ./storage/innobase/include/ut0lst.h:334
#11 ut_list_remove<ut_list_base<rw_lock_t, ut_list_node<rw_lock_t> rw_lock_t::*> > (elem=0x82684e9c, list=...) at ./storage/innobase/include/ut0lst.h:386
#12 rw_lock_free_func (lock=<optimized out>) at ./storage/innobase/sync/sync0rw.cc:267
#13 0x808c819a in pfs_rw_lock_free_func (lock=<optimized out>) at ./storage/innobase/include/sync0rw.ic:603
#14 ha_clear (table=0x826838f8) at ./storage/innobase/ha/ha0ha.cc:168
#15 0x80832ace in buf_pool_free_instance (buf_pool=0x826625b8) at ./storage/innobase/buf/buf0buf.cc:2017
#16 buf_pool_free (n_instances=1) at ./storage/innobase/buf/buf0buf.cc:2100
#17 0x807a233e in innodb_shutdown () at ./storage/innobase/srv/srv0start.cc:2907
#18 0x8067f84c in innobase_end () at ./storage/innobase/handler/ha_innodb.cc:4420
#19 0x804fe56b in ha_finalize_handlerton (plugin=0x825c0e60) at ./sql/handler.cc:494
#20 0x8034e5c8 in plugin_deinitialize (plugin=0x825c0e60, ref_check=ref_check@entry=true) at ./sql/sql_plugin.cc:1220
#21 0x803531cd in reap_plugins () at ./sql/sql_plugin.cc:1296
#22 0x80354295 in plugin_shutdown () at ./sql/sql_plugin.cc:1957
#23 0x80294ddb in clean_up (print_message=print_message@entry=true) at ./sql/mysqld.cc:2223
#24 0x802955e1 in clean_up (print_message=true) at ./sql/mysqld.cc:2104
#25 unireg_end () at ./sql/mysqld.cc:2098
#26 0x8029844d in kill_server (sig_ptr=0x0) at ./sql/mysqld.cc:2025
#27 0x802988fc in kill_server_thread (arg=0xad9b0184) at ./sql/mysqld.cc:2048
#28 0x80a36dcb in pfs_spawn_thread (arg=<optimized out>) at ./storage/perfschema/pfs.cc:1869
#29 0xb772627a in start_thread (arg=0xa4e69b40) at pthread_create.c:456
#30 0xb70b0366 in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:110
 
 - deleting it, already saved 0
 - saving '/dev/shm/var/3/log/sys_vars.innodb_buffer_pool_dump_at_shutdown_basic-innodb/' to '/dev/shm/var/log/sys_vars.innodb_buffer_pool_dump_at_shutdown_basic-innodb/'

Generated at Thu Feb 08 09:18:45 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.