rpl.rpl_old_master failed in buildbot with InnoDB: Failing assertion: list.count > 0

2020-06-13 17:50:06 0x3fff8e73ebf0  InnoDB: Assertion failure in file /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.4.14/storage/innobase/include/ut0lst.h line 334

InnoDB: Failing assertion: list.count > 0

rpl.rpl_old_master 'innodb,stmt'         w4 [ fail ]  Found warnings/errors in server log file!

        Test ended at 2020-06-13 17:50:28

line

Attempting backtrace. You can use the following information to find out

^ Found warnings in /dev/shm/var/4/log/mysqld.1.err

ok

 - found 'core.25559' (0/0)

Trying 'dbx' to get a backtrace

Trying 'gdb' to get a backtrace from coredump /dev/shm/var/4/log/rpl.rpl_old_master-innodb,stmt/mysqld.1/data/core.25559

Core generated by '/usr/sbin/mysqld'

Output from gdb follows. The first stack trace is from the failing thread.

The following stack traces are from all threads (so the failing one is

duplicated).

--------------------------

[New LWP 25559]

[New LWP 25592]

[New LWP 25563]

[New LWP 25562]

[Thread debugging using libthread_db enabled]

Using host libthread_db library "/lib64/libthread_db.so.1".

Core was generated by `/usr/sbin/mysqld --defaults-group-suffix=.1 --defaults-file=/dev/shm/var/4/my.c'.

Program terminated with signal 6, Aborted.

#0  0x00003fff8e68f150 in pthread_kill () from /lib64/libpthread.so.0

#0  0x00003fff8e68f150 in pthread_kill () from /lib64/libpthread.so.0

#1  0x00000000270dfc40 in my_write_core (sig=<optimized out>) at /usr/src/debug/MariaDB-10.4.14/src_0/mysys/stacktrace.c:481

#2  0x0000000026a132e8 in handle_fatal_signal (sig=<optimized out>) at /usr/src/debug/MariaDB-10.4.14/src_0/sql/signal_handler.cc:343

#3  <signal handler called>

#4  0x00003fff8dd5eb18 in raise () from /lib64/libc.so.6

#5  0x00003fff8dd60c9c in abort () from /lib64/libc.so.6

#6  0x0000000026637694 in ut_dbg_assertion_failed (expr=0x2730ecc8 "list.count > 0", file=0x2730ecd8 "/home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.4.14/storage/innobase/include/ut0lst.h", line=<optimized out>) at /usr/src/debug/MariaDB-10.4.14/src_0/storage/innobase/ut/ut0dbg.cc:60

#7  0x0000000026d9c31c in ut_list_remove<ut_list_base<rw_lock_t, ut_list_node<rw_lock_t> rw_lock_t::*>, GenericGetNode<rw_lock_t> > (get_node=..., node=..., list=...) at /usr/src/debug/MariaDB-10.4.14/src_0/storage/innobase/include/ut0lst.h:334

#8  ut_list_remove<ut_list_base<rw_lock_t, ut_list_node<rw_lock_t> rw_lock_t::*> > (elem=0x1003ac96d50, list=...) at /usr/src/debug/MariaDB-10.4.14/src_0/storage/innobase/include/ut0lst.h:386

#9  rw_lock_free_func (lock=0x1003ac96d50) at /usr/src/debug/MariaDB-10.4.14/src_0/storage/innobase/sync/sync0rw.cc:260

#10 0x0000000026ed1908 in pfs_rw_lock_free_func (lock=0x1003ac96d50) at /usr/src/debug/MariaDB-10.4.14/src_0/storage/innobase/include/sync0rw.ic:605

#11 ha_clear (table=0x1003ac94480) at /usr/src/debug/MariaDB-10.4.14/src_0/storage/innobase/ha/ha0ha.cc:171

#12 0x0000000026e31e50 in buf_pool_free_instance (buf_pool=0x1003ac6ab90) at /usr/src/debug/MariaDB-10.4.14/src_0/storage/innobase/buf/buf0buf.cc:2070

#13 buf_pool_free (n_instances=1) at /usr/src/debug/MariaDB-10.4.14/src_0/storage/innobase/buf/buf0buf.cc:2136

#14 0x0000000026d95780 in innodb_shutdown () at /usr/src/debug/MariaDB-10.4.14/src_0/storage/innobase/srv/srv0start.cc:2541

#15 0x0000000026c59860 in innobase_end () at /usr/src/debug/MariaDB-10.4.14/src_0/storage/innobase/handler/ha_innodb.cc:4311

#16 0x0000000026a179c4 in ha_finalize_handlerton (plugin=0x1003aa9de80) at /usr/src/debug/MariaDB-10.4.14/src_0/sql/handler.cc:496

#17 0x0000000026784bd4 in plugin_deinitialize (plugin=0x1003aa9de80, ref_check=true) at /usr/src/debug/MariaDB-10.4.14/src_0/sql/sql_plugin.cc:1241

#18 0x0000000026788848 in reap_plugins () at /usr/src/debug/MariaDB-10.4.14/src_0/sql/sql_plugin.cc:1317

#19 0x0000000026789810 in plugin_shutdown () at /usr/src/debug/MariaDB-10.4.14/src_0/sql/sql_plugin.cc:1986

#20 0x0000000026672ad4 in clean_up (print_message=true) at /usr/src/debug/MariaDB-10.4.14/src_0/sql/mysqld.cc:1999

#21 0x0000000026680678 in clean_up (print_message=true) at /usr/src/debug/MariaDB-10.4.14/src_0/sql/mysqld.cc:5935

#22 mysqld_main (argc=149, argv=0x1003a92bc48) at /usr/src/debug/MariaDB-10.4.14/src_0/sql/mysqld.cc:5941

#23 0x00000000266426a0 in main (argc=<optimized out>, argv=<optimized out>) at /usr/src/debug/MariaDB-10.4.14/src_0/sql/main.cc:25

Thread 4 (Thread 0x3fff890af070 (LWP 25562)):

#0  0x00003fff8e68e144 in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0

#1  0x00000000270e6728 in inline_mysql_cond_timedwait (that=0x2825b118 <COND_timer>, mutex=0x2825b0b0 <LOCK_timer>, src_file=0x2736c7f8 "/home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.4.14/mysys/thr_timer.c", src_line=292, abstime=0x3fff890ae598) at /usr/src/debug/MariaDB-10.4.14/src_0/include/mysql/psi/mysql_thread.h:1211

#2  timer_handler (arg=<optimized out>) at /usr/src/debug/MariaDB-10.4.14/src_0/mysys/thr_timer.c:292

#3  0x0000000027075f10 in pfs_spawn_thread (arg=0x1003aa89b28) at /usr/src/debug/MariaDB-10.4.14/src_0/storage/perfschema/pfs.cc:1869

#4  0x00003fff8e688728 in start_thread () from /lib64/libpthread.so.0

#5  0x00003fff8de3d210 in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x3fff8384f070 (LWP 25563)):

#0  0x00003fff8e68e144 in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0

#1  0x0000000026f755bc in inline_mysql_cond_timedwait (src_file=0x27356c48 "/home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.4.14/storage/maria/ma_servicethread.c", src_line=116, abstime=<optimized out>, mutex=0x2824a520 <LOCK_checkpoint>, that=0x2824a558 <COND_checkpoint>) at /usr/src/debug/MariaDB-10.4.14/src_0/include/mysql/psi/mysql_thread.h:1211

#2  my_service_thread_sleep (control=0x2791b890 <checkpoint_control>, sleep_time=<optimized out>) at /usr/src/debug/MariaDB-10.4.14/src_0/storage/maria/ma_servicethread.c:115

#3  0x0000000026f69354 in ma_checkpoint_background (arg=0x1e) at /usr/src/debug/MariaDB-10.4.14/src_0/storage/maria/ma_checkpoint.c:707

#4  0x0000000027075f10 in pfs_spawn_thread (arg=0x1003ab40fa8) at /usr/src/debug/MariaDB-10.4.14/src_0/storage/perfschema/pfs.cc:1869

#5  0x00003fff8e688728 in start_thread () from /lib64/libpthread.so.0

#6  0x00003fff8de3d210 in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x3fff825e8070 (LWP 25592)):

#0  0x00003fff8dd60448 in sigwaitinfo () from /lib64/libc.so.6

#1  0x000000002667385c in my_sigwait (code=<synthetic pointer>, sig=0x3fff825e74e0, set=0x3fff825e74f0) at /usr/src/debug/MariaDB-10.4.14/src_0/include/my_pthread.h:196

#2  signal_hand (arg=<optimized out>) at /usr/src/debug/MariaDB-10.4.14/src_0/sql/mysqld.cc:3253

#3  0x0000000027075f10 in pfs_spawn_thread (arg=0x1003afeeb78) at /usr/src/debug/MariaDB-10.4.14/src_0/storage/perfschema/pfs.cc:1869

#4  0x00003fff8e688728 in start_thread () from /lib64/libpthread.so.0

#5  0x00003fff8de3d210 in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x3fff8e73ebf0 (LWP 25559)):

#0  0x00003fff8e68f150 in pthread_kill () from /lib64/libpthread.so.0

#1  0x00000000270dfc40 in my_write_core (sig=<optimized out>) at /usr/src/debug/MariaDB-10.4.14/src_0/mysys/stacktrace.c:481

#2  0x0000000026a132e8 in handle_fatal_signal (sig=<optimized out>) at /usr/src/debug/MariaDB-10.4.14/src_0/sql/signal_handler.cc:343

#3  <signal handler called>

#4  0x00003fff8dd5eb18 in raise () from /lib64/libc.so.6

#5  0x00003fff8dd60c9c in abort () from /lib64/libc.so.6

#6  0x0000000026637694 in ut_dbg_assertion_failed (expr=0x2730ecc8 "list.count > 0", file=0x2730ecd8 "/home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.4.14/storage/innobase/include/ut0lst.h", line=<optimized out>) at /usr/src/debug/MariaDB-10.4.14/src_0/storage/innobase/ut/ut0dbg.cc:60

#7  0x0000000026d9c31c in ut_list_remove<ut_list_base<rw_lock_t, ut_list_node<rw_lock_t> rw_lock_t::*>, GenericGetNode<rw_lock_t> > (get_node=..., node=..., list=...) at /usr/src/debug/MariaDB-10.4.14/src_0/storage/innobase/include/ut0lst.h:334

#8  ut_list_remove<ut_list_base<rw_lock_t, ut_list_node<rw_lock_t> rw_lock_t::*> > (elem=0x1003ac96d50, list=...) at /usr/src/debug/MariaDB-10.4.14/src_0/storage/innobase/include/ut0lst.h:386

#9  rw_lock_free_func (lock=0x1003ac96d50) at /usr/src/debug/MariaDB-10.4.14/src_0/storage/innobase/sync/sync0rw.cc:260

#10 0x0000000026ed1908 in pfs_rw_lock_free_func (lock=0x1003ac96d50) at /usr/src/debug/MariaDB-10.4.14/src_0/storage/innobase/include/sync0rw.ic:605

#11 ha_clear (table=0x1003ac94480) at /usr/src/debug/MariaDB-10.4.14/src_0/storage/innobase/ha/ha0ha.cc:171

#12 0x0000000026e31e50 in buf_pool_free_instance (buf_pool=0x1003ac6ab90) at /usr/src/debug/MariaDB-10.4.14/src_0/storage/innobase/buf/buf0buf.cc:2070

#13 buf_pool_free (n_instances=1) at /usr/src/debug/MariaDB-10.4.14/src_0/storage/innobase/buf/buf0buf.cc:2136

#14 0x0000000026d95780 in innodb_shutdown () at /usr/src/debug/MariaDB-10.4.14/src_0/storage/innobase/srv/srv0start.cc:2541

#15 0x0000000026c59860 in innobase_end () at /usr/src/debug/MariaDB-10.4.14/src_0/storage/innobase/handler/ha_innodb.cc:4311

#16 0x0000000026a179c4 in ha_finalize_handlerton (plugin=0x1003aa9de80) at /usr/src/debug/MariaDB-10.4.14/src_0/sql/handler.cc:496

#17 0x0000000026784bd4 in plugin_deinitialize (plugin=0x1003aa9de80, ref_check=true) at /usr/src/debug/MariaDB-10.4.14/src_0/sql/sql_plugin.cc:1241

#18 0x0000000026788848 in reap_plugins () at /usr/src/debug/MariaDB-10.4.14/src_0/sql/sql_plugin.cc:1317

#19 0x0000000026789810 in plugin_shutdown () at /usr/src/debug/MariaDB-10.4.14/src_0/sql/sql_plugin.cc:1986

#20 0x0000000026672ad4 in clean_up (print_message=true) at /usr/src/debug/MariaDB-10.4.14/src_0/sql/mysqld.cc:1999

#21 0x0000000026680678 in clean_up (print_message=true) at /usr/src/debug/MariaDB-10.4.14/src_0/sql/mysqld.cc:5935

#22 mysqld_main (argc=149, argv=0x1003a92bc48) at /usr/src/debug/MariaDB-10.4.14/src_0/sql/mysqld.cc:5941

#23 0x00000000266426a0 in main (argc=<optimized out>, argv=<optimized out>) at /usr/src/debug/MariaDB-10.4.14/src_0/sql/main.cc:25