[MDEV-33343] spider.mdev_28739_simple fails in buildbot Created: 2024-01-31  Updated: 2024-02-01  Resolved: 2024-01-31

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - Spider, Tests
Affects Version/s: 10.4, 10.5, 10.6, 10.11, 11.0, 11.1, 11.2
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Sergei Golubchik Assignee: Yuchen Pei
Resolution: Duplicate Votes: 0
Labels: None

Issue Links:
Duplicate
duplicates MDEV-32997 sporadic SIGABRT at my_sleep (timeout... Open

 Description   

The server crashes on shutdown. For example (https://buildbot.mariadb.net/buildbot/builders/kvm-fulltest/builds/48734/steps/mtr_ps/logs/stdio)

spider/bugfix.mdev_28739_simple          w2 [ fail ]  Found warnings/errors in server log file!
        Test ended at 2024-01-30 14:12:23
line
Attempting backtrace. You can use the following information to find out
^ Found warnings in /mnt/buildbot/build/mariadb-10.5.24/mysql-test/var/2/log/mysqld.1.1.err
ok
 
 - found 'core' (0/0)
Core generated by '/mnt/buildbot/build/mariadb-10.5.24/sql/mariadbd'
Output from gdb follows. The first stack trace is from the failing thread.
The following stack traces are from all threads (so the failing one is
duplicated).
--------------------------
[New LWP 17350]
[New LWP 17408]
[New LWP 17375]
[New LWP 17857]
[New LWP 17856]
[New LWP 17410]
[New LWP 17409]
[New LWP 17403]
[New LWP 17392]
[New LWP 17391]
[New LWP 17374]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/mnt/buildbot/build/mariadb-10.5.24/sql/mariadbd --defaults-group-suffix=.1.1 -'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007faf7f15859f in pthread_kill () from /lib/x86_64-linux-gnu/libpthread.so.0
[Current thread is 1 (Thread 0x7faf7ffd38c0 (LWP 17350))]
 
Thread 11 (Thread 0x7faf7fed5700 (LWP 17374)):
#0  0x00007faf7f157709 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x000055aff40306ae in inline_mysql_cond_timedwait (that=0x55aff5419da0 <COND_timer>, mutex=0x55aff5419de0 <LOCK_timer>, src_file=0x55aff4334948 "/home/buildbot/buildbot/build/mariadb-10.5.24/mysys/thr_timer.c", src_line=321, abstime=0x7faf7fed4d10) at /home/buildbot/buildbot/build/mariadb-10.5.24/include/mysql/psi/mysql_thread.h:1270
#2  timer_handler (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.5.24/mysys/thr_timer.c:321
#3  0x00007faf7f1516ba in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#4  0x00007faf7e5f882d in clone () from /lib/x86_64-linux-gnu/libc.so.6
 
Thread 10 (Thread 0x7faf77bfe700 (LWP 17391)):
#0  0x00007faf7e5f28e9 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x000055aff409d1f8 in my_getevents (min_nr=1, nr=256, ev=0x7faf77bfbd40, ctx=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.5.24/tpool/aio_linux.cc:64
#2  tpool::aio_linux::getevent_thread_routine (aio=0x55aff65ae630) at /home/buildbot/buildbot/build/mariadb-10.5.24/tpool/aio_linux.cc:108
#3  0x00007faf7ec7cc80 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4  0x00007faf7f1516ba in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5  0x00007faf7e5f882d in clone () from /lib/x86_64-linux-gnu/libc.so.6
 
Thread 9 (Thread 0x7faf768fd700 (LWP 17392)):
#0  0x00007faf7f157360 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x000055aff3f36ee8 in buf_flush_page_cleaner () at /home/buildbot/buildbot/build/mariadb-10.5.24/storage/innobase/buf/buf0flu.cc:2298
#2  0x00007faf7f1516ba in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#3  0x00007faf7e5f882d in clone () from /lib/x86_64-linux-gnu/libc.so.6
 
Thread 8 (Thread 0x7faf697fa700 (LWP 17403)):
#0  0x00007faf7f157709 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x000055aff4099334 in __gthread_cond_timedwait (__abs_timeout=0x7faf697f9ca0, __mutex=<optimized out>, __cond=0x55aff65651b0) at /usr/include/x86_64-linux-gnu/c++/5/bits/gthr-default.h:871
#2  std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=0x55aff65651b0) at /usr/include/c++/5/condition_variable:165
#3  std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=0x55aff65651b0) at /usr/include/c++/5/condition_variable:105
#4  std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (__rtime=..., __lock=..., this=0x55aff65651b0) at /usr/include/c++/5/condition_variable:137
#5  tpool::thread_pool_generic::wait_for_tasks (this=this@entry=0x55aff6564a80, lk=..., thread_data=thread_data@entry=0x55aff65651b0) at /home/buildbot/buildbot/build/mariadb-10.5.24/tpool/tpool_generic.cc:499
#6  0x000055aff4099d19 in tpool::thread_pool_generic::get_task (this=this@entry=0x55aff6564a80, thread_var=thread_var@entry=0x55aff65651b0, t=t@entry=0x7faf697f9d48) at /home/buildbot/buildbot/build/mariadb-10.5.24/tpool/tpool_generic.cc:552
#7  0x000055aff4099e8d in tpool::thread_pool_generic::worker_main (this=0x55aff6564a80, thread_var=0x55aff65651b0) at /home/buildbot/buildbot/build/mariadb-10.5.24/tpool/tpool_generic.cc:597
#8  0x00007faf7ec7cc80 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#9  0x00007faf7f1516ba in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007faf7e5f882d in clone () from /lib/x86_64-linux-gnu/libc.so.6
 
Thread 7 (Thread 0x7faf7c0a8700 (LWP 17409)):
#0  0x00007faf7f157360 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x000055aff3b01efb in inline_mysql_cond_wait (that=0x55aff4b77ca0 <mysql_bin_log+3232>, mutex=0x55aff4b77c70 <mysql_bin_log+3184>, src_file=0x55aff4276498 "/home/buildbot/buildbot/build/mariadb-10.5.24/sql/log.cc", src_line=10490) at /home/buildbot/buildbot/build/mariadb-10.5.24/include/mysql/psi/mysql_thread.h:1233
#2  binlog_background_thread (arg=arg@entry=0x0) at /home/buildbot/buildbot/build/mariadb-10.5.24/sql/log.cc:10490
#3  0x000055aff3c8693b in pfs_spawn_thread (arg=0x55aff67cb468) at /home/buildbot/buildbot/build/mariadb-10.5.24/storage/perfschema/pfs.cc:2201
#4  0x00007faf7f1516ba in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5  0x00007faf7e5f882d in clone () from /lib/x86_64-linux-gnu/libc.so.6
 
Thread 6 (Thread 0x7faf7c05e700 (LWP 17410)):
#0  0x00007faf7e528b26 in sigwaitinfo () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x000055aff373768b in my_sigwait (code=<synthetic pointer>, sig=0x7faf7c05dc08, set=0x7faf7c05dc10) at /home/buildbot/buildbot/build/mariadb-10.5.24/include/my_pthread.h:193
#2  signal_hand (arg=arg@entry=0x0) at /home/buildbot/buildbot/build/mariadb-10.5.24/sql/mysqld.cc:3007
#3  0x000055aff3c8693b in pfs_spawn_thread (arg=0x55aff698ef08) at /home/buildbot/buildbot/build/mariadb-10.5.24/storage/perfschema/pfs.cc:2201
#4  0x00007faf7f1516ba in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5  0x00007faf7e5f882d in clone () from /lib/x86_64-linux-gnu/libc.so.6
 
Thread 5 (Thread 0x7faf5f7fe700 (LWP 17856)):
#0  0x00007faf7f157360 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007faf7425389f in inline_mysql_cond_wait (src_file=0x7faf742be280 "/home/buildbot/buildbot/build/mariadb-10.5.24/storage/spider/spd_table.cc", src_line=10019, mutex=0x7faf1408f560, that=0x7faf1408f528) at /home/buildbot/buildbot/build/mariadb-10.5.24/include/mysql/psi/mysql_thread.h:1233
#2  spider_table_bg_sts_action (arg=0x7faf1408f508) at /home/buildbot/buildbot/build/mariadb-10.5.24/storage/spider/spd_table.cc:10019
#3  0x00007faf7f1516ba in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#4  0x00007faf7e5f882d in clone () from /lib/x86_64-linux-gnu/libc.so.6
 
Thread 4 (Thread 0x7faf5effd700 (LWP 17857)):
#0  0x00007faf7f157360 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007faf7425441f in inline_mysql_cond_wait (src_file=0x7faf742be280 "/home/buildbot/buildbot/build/mariadb-10.5.24/storage/spider/spd_table.cc", src_line=10171, mutex=0x7faf1408f630, that=0x7faf1408f5f8) at /home/buildbot/buildbot/build/mariadb-10.5.24/include/mysql/psi/mysql_thread.h:1233
#2  spider_table_bg_crd_action (arg=0x7faf1408f5d8) at /home/buildbot/buildbot/build/mariadb-10.5.24/storage/spider/spd_table.cc:10171
#3  0x00007faf7f1516ba in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#4  0x00007faf7e5f882d in clone () from /lib/x86_64-linux-gnu/libc.so.6
 
Thread 3 (Thread 0x7faf7c9f3700 (LWP 17375)):
#0  0x00007faf7f157709 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x000055aff3c152e6 in inline_mysql_cond_timedwait (src_file=0x55aff42a6750 "/home/buildbot/buildbot/build/mariadb-10.5.24/storage/maria/ma_servicethread.c", src_line=116, abstime=0x7faf7c9f2c90, mutex=0x55aff5379e20 <LOCK_checkpoint>, that=0x55aff5379de0 <COND_checkpoint>) at /home/buildbot/buildbot/build/mariadb-10.5.24/include/mysql/psi/mysql_thread.h:1270
#2  my_service_thread_sleep (control=control@entry=0x55aff4ab47c0 <checkpoint_control>, sleep_time=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.5.24/storage/maria/ma_servicethread.c:115
#3  0x000055aff3c0c5e7 in ma_checkpoint_background (arg=0x1e) at /home/buildbot/buildbot/build/mariadb-10.5.24/storage/maria/ma_checkpoint.c:725
#4  0x00007faf7f1516ba in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5  0x00007faf7e5f882d in clone () from /lib/x86_64-linux-gnu/libc.so.6
 
Thread 2 (Thread 0x7faf7c0f2700 (LWP 17408)):
#0  0x00007faf7f157360 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x000055aff37ff2eb in inline_mysql_cond_wait (that=0x55aff4b466c0 <COND_manager>, mutex=0x55aff4b46700 <LOCK_manager>, src_file=0x55aff4128bb8 "/home/buildbot/buildbot/build/mariadb-10.5.24/sql/sql_manager.cc", src_line=103) at /home/buildbot/buildbot/build/mariadb-10.5.24/include/mysql/psi/mysql_thread.h:1233
#2  handle_manager (arg=arg@entry=0x0) at /home/buildbot/buildbot/build/mariadb-10.5.24/sql/sql_manager.cc:103
#3  0x000055aff3c8693b in pfs_spawn_thread (arg=0x55aff67f5af8) at /home/buildbot/buildbot/build/mariadb-10.5.24/storage/perfschema/pfs.cc:2201
#4  0x00007faf7f1516ba in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5  0x00007faf7e5f882d in clone () from /lib/x86_64-linux-gnu/libc.so.6
 
Thread 1 (Thread 0x7faf7ffd38c0 (LWP 17350)):
#0  0x00007faf7f15859f in pthread_kill () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x000055aff402c177 in my_write_core (sig=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.5.24/mysys/stacktrace.c:424
#2  0x000055aff3a1992e in handle_fatal_signal (sig=6) at /home/buildbot/buildbot/build/mariadb-10.5.24/sql/signal_handler.cc:357
#3  <signal handler called>
#4  0x00007faf7e5ee9e3 in select () from /lib/x86_64-linux-gnu/libc.so.6
#5  0x000055aff4029d6f in my_sleep (m_seconds=m_seconds@entry=1000) at /home/buildbot/buildbot/build/mariadb-10.5.24/mysys/my_sleep.c:29
#6  0x000055aff3741b2d in close_connections () at /home/buildbot/buildbot/build/mariadb-10.5.24/sql/mysqld.cc:1813
#7  mysqld_main (argc=163, argv=0x55aff645fcf0) at /home/buildbot/buildbot/build/mariadb-10.5.24/sql/mysqld.cc:5739
#8  0x00007faf7e512830 in __libc_start_main () from /lib/x86_64-linux-gnu/libc.so.6
#9  0x000055aff3734f19 in _start ()
 
 - deleting it, already saved 0
 - saving '/mnt/buildbot/build/mariadb-10.5.24/mysql-test/var/2/log/spider/bugfix.mdev_28739_simple/' to '/mnt/buildbot/build/mariadb-10.5.24/mysql-test/var/log/bugfix.mdev_28739_simple/'

More failures in X-ref https://buildbot.mariadb.net/ci/reports/cross_reference#branch=&revision=&platform=&fail_name=spider/bugfix.mdev_28739_simple&fail_variant=&fail_info_full=&typ=&info=&dt=&limit=100&fail_info_short=



 Comments   
Comment by Yuchen Pei [ 2024-01-31 ]

dup of MDEV-32997, judging from the testcase and the stack

Generated at Thu Feb 08 10:38:12 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.