[MDEV-17340] innodb.undo_truncate failed in buildbot, server hung Created: 2018-10-01  Updated: 2022-02-11  Resolved: 2022-02-11

Status: Closed
Project: MariaDB Server
Component/s: Locking, Server, Tests
Affects Version/s: 10.2
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Elena Stepanova Assignee: Marko Mäkelä
Resolution: Cannot Reproduce Votes: 0
Labels: None

Issue Links:
Relates
relates to MDEV-26733 assert on shutdown lock->lock_word ==... Open

 Description   

http://buildbot.askmonty.org/buildbot/builders/kvm-zyp-sles12-amd64/builds/5768

innodb.undo_truncate '16k,2,innodb'      w2 [ fail ]  timeout after 900 seconds
 
        Test ended at 2018-09-28 04:34:58
 
 
 
Test case timeout after 900 seconds
 
 
 
== /dev/shm/var/2/log/undo_truncate.log == 
SET i = i + 1;
 
END WHILE;
 
END |
 
CREATE PROCEDURE populate_t2()
 
BEGIN
 
DECLARE i INT DEFAULT 1;
 
while (i <= 20000) DO
 
insert into t2 values (i, 'a');
 
SET i = i + 1;
 
END WHILE;
 
END |
 
connect  con1,localhost,root,,;
 
begin;
 
call populate_t1();
 
connect  con2,localhost,root,,;
 
begin;
 
call populate_t2();
 
connection con1;
 
update t1 set c = 'mysql';
 
connection con2;
 
 
 
 == /dev/shm/var/2/tmp/analyze-timeout-mysqld.1.err ==
 
mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/dev/shm/var/tmp/2/mysqld.1.sock' (111)
 
 
 
 
 
 - found 'core' (2/0)
 
 
 
Trying 'dbx' to get a backtrace
 
 
 
Trying 'gdb' to get a backtrace from coredump /dev/shm/var/2/log/innodb.undo_truncate-16k,2,innodb/mysqld.1/data/core
 
Core generated by '/usr/sbin/mysqld'
 
Output from gdb follows. The first stack trace is from the failing thread.
 
The following stack traces are from all threads (so the failing one is
 
duplicated).
 
--------------------------
 
 
 
warning: /etc/gdbinit.d/gdb-heap.py: No such file or directory
 
[New LWP 10117]
 
[New LWP 10147]
 
[New LWP 10148]
 
[New LWP 10153]
 
[New LWP 10150]
 
[New LWP 10178]
 
[New LWP 10149]
 
[New LWP 10161]
 
[New LWP 10154]
 
[New LWP 10146]
 
[New LWP 10177]
 
[New LWP 10151]
 
[New LWP 10174]
 
[New LWP 10157]
 
[New LWP 10152]
 
[New LWP 10171]
 
[New LWP 10158]
 
[New LWP 10170]
 
[New LWP 10163]
 
[New LWP 10162]
 
[New LWP 10167]
 
[New LWP 10164]
 
[New LWP 10169]
 
[New LWP 10165]
 
[New LWP 10166]
 
[New LWP 10168]
 
[New LWP 10156]
 
[New LWP 10159]
 
[New LWP 10160]
 
[Thread debugging using libthread_db enabled]
 
Using host libthread_db library "/lib64/libthread_db.so.1".
 
Core was generated by `/usr/sbin/mysqld --defaults-group-suffix=.1 --defaults-file=/dev/shm/var/2/my.c'.
 
Program terminated with signal SIGABRT, Aborted.
 
#0  0x00007f5e6d5040d1 in pthread_kill () from /lib64/libpthread.so.0
 
#0  0x00007f5e6d5040d1 in pthread_kill () from /lib64/libpthread.so.0
 
#1  0x00007f5e6dfc67f9 in handle_fatal_signal (sig=6) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/sql/signal_handler.cc:305
 
#2  <signal handler called>
 
#3  0x00007f5e6b9243cd in poll () from /lib64/libc.so.6
 
#4  0x00007f5e6ddb682f in poll (__timeout=-1, __nfds=2, __fds=0x7fff447672a0) at /usr/include/bits/poll2.h:41
 
#5  handle_connections_sockets () at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/sql/mysqld.cc:6609
 
#6  0x00007f5e6ddb9b60 in mysqld_main (argc=146, argv=0x7f5e7107ea68) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/sql/mysqld.cc:6075
 
#7  0x00007f5e6b868b05 in __libc_start_main () from /lib64/libc.so.6
 
#8  0x00007f5e6ddad51d in _start () at ../sysdeps/x86_64/start.S:112
 
 
 
Thread 29 (Thread 0x7f5e51ffb700 (LWP 10160)):
 
#0  0x00007f5e6d503408 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
 
#1  0x00007f5e6e1e26b1 in os_event::timed_wait (this=this@entry=0x7f5e718f9ef0, abstime=abstime@entry=0x7f5e51ffad60) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0event.cc:286
 
#2  0x00007f5e6e1e28df in os_event::wait_time_low (this=0x7f5e718f9ef0, time_in_usec=time_in_usec@entry=10000000, reset_sig_count=<optimized out>, reset_sig_count@entry=0) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0event.cc:413
 
#3  0x00007f5e6e1e2c0a in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=10000000, reset_sig_count=reset_sig_count@entry=0) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0event.cc:518
 
#4  0x00007f5e6e369ec3 in dict_stats_thread () at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/dict/dict0stats_bg.cc:460
 
#5  0x00007f5e6d4ff0a4 in start_thread () from /lib64/libpthread.so.0
 
#6  0x00007f5e6b92c7fd in clone () from /lib64/libc.so.6
 
 
 
Thread 28 (Thread 0x7f5e53fff700 (LWP 10159)):
 
#0  0x00007f5e6d50614d in nanosleep () from /lib64/libpthread.so.0
 
#1  0x00007f5e6e1e2e70 in os_thread_sleep (tm=tm@entry=1000000) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0thread.cc:230
 
#2  0x00007f5e6e27222d in srv_master_sleep () at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/srv/srv0srv.cc:2385
 
#3  srv_master_thread (arg=<optimized out>) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/srv/srv0srv.cc:2426
 
#4  0x00007f5e6d4ff0a4 in start_thread () from /lib64/libpthread.so.0
 
#5  0x00007f5e6b92c7fd in clone () from /lib64/libc.so.6
 
 
 
Thread 27 (Thread 0x7f5e537fe700 (LWP 10156)):
 
#0  0x00007f5e6d503408 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
 
#1  0x00007f5e6e1e26b1 in os_event::timed_wait (this=this@entry=0x7f5e718c6750, abstime=abstime@entry=0x7f5e537fdd40) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0event.cc:286
 
#2  0x00007f5e6e1e28df in os_event::wait_time_low (this=this@entry=0x7f5e718c6750, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=<optimized out>, reset_sig_count@entry=1) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0event.cc:413
 
#3  0x00007f5e6e1e2c0a in os_event_wait_time_low (event=event@entry=0x7f5e718c6750, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=reset_sig_count@entry=1) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0event.cc:518
 
#4  0x00007f5e6e1b53c4 in lock_wait_timeout_thread () at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/lock/lock0wait.cc:547
 
#5  0x00007f5e6d4ff0a4 in start_thread () from /lib64/libpthread.so.0
 
#6  0x00007f5e6b92c7fd in clone () from /lib64/libc.so.6
 
 
 
Thread 26 (Thread 0x7f5e497fa700 (LWP 10168)):
 
#0  0x00007f5e6d50305f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
 
#1  0x00007f5e6e1e2c6b in wait (this=<optimized out>) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0event.cc:166
 
#2  wait_low (reset_sig_count=<optimized out>, this=0x7f5e712c17d0) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0event.cc:336
 
#3  os_event_wait_low (event=0x7f5e712c17d0, reset_sig_count=reset_sig_count@entry=0) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0event.cc:535
 
#4  0x00007f5e6e30ddff in buf_resize_thread () at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/buf/buf0buf.cc:3214
 
#5  0x00007f5e6d4ff0a4 in start_thread () from /lib64/libpthread.so.0
 
#6  0x00007f5e6b92c7fd in clone () from /lib64/libc.so.6
 
 
 
Thread 25 (Thread 0x7f5e4a7fc700 (LWP 10166)):
 
#0  0x00007f5e6d50305f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
 
#1  0x00007f5e6e1e2c6b in wait (this=<optimized out>) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0event.cc:166
 
#2  wait_low (reset_sig_count=<optimized out>, this=0x7f5e712c16d0) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0event.cc:336
 
#3  os_event_wait_low (event=0x7f5e712c16d0, reset_sig_count=reset_sig_count@entry=0) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0event.cc:535
 
#4  0x00007f5e6e3163f9 in buf_dump_thread () at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/buf/buf0dump.cc:809
 
#5  0x00007f5e6d4ff0a4 in start_thread () from /lib64/libpthread.so.0
 
#6  0x00007f5e6b92c7fd in clone () from /lib64/libc.so.6
 
 
 
Thread 24 (Thread 0x7f5e4affd700 (LWP 10165)):
 
#0  0x00007f5e6d50305f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
 
#1  0x00007f5e6e1e2c6b in wait (this=<optimized out>) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0event.cc:166
 
#2  wait_low (reset_sig_count=<optimized out>, this=0x7f5e712c1550) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0event.cc:336
 
#3  os_event_wait_low (event=0x7f5e712c1550, reset_sig_count=reset_sig_count@entry=0) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0event.cc:535
 
#4  0x00007f5e6e272d21 in srv_resume_thread (wait=true, slot=0x7f5e6ed54ea8 <srv_sys+360>, sig_count=<optimized out>, timeout_usec=<optimized out>) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/srv/srv0srv.cc:930
 
#5  srv_worker_thread (arg=<optimized out>) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/srv/srv0srv.cc:2580
 
#6  0x00007f5e6d4ff0a4 in start_thread () from /lib64/libpthread.so.0
 
#7  0x00007f5e6b92c7fd in clone () from /lib64/libc.so.6
 
 
 
Thread 23 (Thread 0x7f5e48ff9700 (LWP 10169)):
 
#0  0x00007f5e6d50305f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
 
#1  0x00007f5e6e160d0b in inline_mysql_cond_wait (mutex=0x7f5e48ff8e00, that=0x7f5e48ff8e30, src_file=<optimized out>, src_line=<optimized out>) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/include/mysql/psi/mysql_thread.h:1149
 
#2  inline_mysql_cond_wait (mutex=0x7f5e48ff8e00, that=0x7f5e48ff8e30, src_file=<optimized out>, src_line=<optimized out>) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/handler/ha_innodb.cc:312
 
#3  thd_destructor_proxy () at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/handler/ha_innodb.cc:334
 
#4  0x00007f5e6e1464ad in pfs_spawn_thread (arg=0x7f5e716a5518) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/perfschema/pfs.cc:1862
 
#5  0x00007f5e6d4ff0a4 in start_thread () from /lib64/libpthread.so.0
 
#6  0x00007f5e6b92c7fd in clone () from /lib64/libc.so.6
 
 
 
Thread 22 (Thread 0x7f5e4b7fe700 (LWP 10164)):
 
#0  0x00007f5e6d50305f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
 
#1  0x00007f5e6e1e2c6b in wait (this=<optimized out>) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0event.cc:166
 
#2  wait_low (reset_sig_count=<optimized out>, this=0x7f5e712c1450) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0event.cc:336
 
#3  os_event_wait_low (event=0x7f5e712c1450, reset_sig_count=reset_sig_count@entry=0) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0event.cc:535
 
#4  0x00007f5e6e272d21 in srv_resume_thread (wait=true, slot=0x7f5e6ed54e38 <srv_sys+248>, sig_count=<optimized out>, timeout_usec=<optimized out>) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/srv/srv0srv.cc:930
 
#5  srv_worker_thread (arg=<optimized out>) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/srv/srv0srv.cc:2580
 
#6  0x00007f5e6d4ff0a4 in start_thread () from /lib64/libpthread.so.0
 
#7  0x00007f5e6b92c7fd in clone () from /lib64/libc.so.6
 
 
 
Thread 21 (Thread 0x7f5e49ffb700 (LWP 10167)):
 
#0  0x00007f5e6d50614d in nanosleep () from /lib64/libpthread.so.0
 
#1  0x00007f5e6e1e2e70 in os_thread_sleep (tm=tm@entry=1000000) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0thread.cc:230
 
#2  0x00007f5e6e2fe83a in btr_defragment_thread () at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/btr/btr0defragment.cc:721
 
#3  0x00007f5e6d4ff0a4 in start_thread () from /lib64/libpthread.so.0
 
#4  0x00007f5e6b92c7fd in clone () from /lib64/libc.so.6
 
 
 
Thread 20 (Thread 0x7f5e50ff9700 (LWP 10162)):
 
#0  0x00007f5e6d50305f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
 
#1  0x00007f5e6e1e2c6b in wait (this=<optimized out>) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0event.cc:166
 
#2  wait_low (reset_sig_count=<optimized out>, this=0x7f5e712c13d0) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0event.cc:336
 
#3  os_event_wait_low (event=0x7f5e712c13d0, reset_sig_count=reset_sig_count@entry=2) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0event.cc:535
 
#4  0x00007f5e6e27530b in srv_resume_thread (timeout_usec=0, wait=true, sig_count=2, slot=0x7f5e6ed54e00 <srv_sys+192>) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/srv/srv0srv.cc:930
 
#5  srv_purge_coordinator_suspend (rseg_history_len=0, slot=0x7f5e6ed54e00 <srv_sys+192>) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/srv/srv0srv.cc:2732
 
#6  srv_purge_coordinator_thread (arg=<optimized out>) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/srv/srv0srv.cc:2819
 
#7  0x00007f5e6d4ff0a4 in start_thread () from /lib64/libpthread.so.0
 
#8  0x00007f5e6b92c7fd in clone () from /lib64/libc.so.6
 
 
 
Thread 19 (Thread 0x7f5e4bfff700 (LWP 10163)):
 
#0  0x00007f5e6d50305f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
 
#1  0x00007f5e6e1e2c6b in wait (this=<optimized out>) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0event.cc:166
 
#2  wait_low (reset_sig_count=<optimized out>, this=0x7f5e712c14d0) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0event.cc:336
 
#3  os_event_wait_low (event=0x7f5e712c14d0, reset_sig_count=reset_sig_count@entry=0) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0event.cc:535
 
#4  0x00007f5e6e272d21 in srv_resume_thread (wait=true, slot=0x7f5e6ed54e70 <srv_sys+304>, sig_count=<optimized out>, timeout_usec=<optimized out>) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/srv/srv0srv.cc:930
 
#5  srv_worker_thread (arg=<optimized out>) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/srv/srv0srv.cc:2580
 
#6  0x00007f5e6d4ff0a4 in start_thread () from /lib64/libpthread.so.0
 
#7  0x00007f5e6b92c7fd in clone () from /lib64/libc.so.6
 
 
 
Thread 18 (Thread 0x7f5e6424d700 (LWP 10170)):
 
#0  0x00007f5e6d5065c9 in do_sigwait () from /lib64/libpthread.so.0
 
#1  0x00007f5e6d506653 in sigwait () from /lib64/libpthread.so.0
 
#2  0x00007f5e6ddafc1f in signal_hand (arg=arg@entry=0x0) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/sql/mysqld.cc:3510
 
#3  0x00007f5e6e1464ad in pfs_spawn_thread (arg=0x7f5e716c9d08) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/perfschema/pfs.cc:1862
 
#4  0x00007f5e6d4ff0a4 in start_thread () from /lib64/libpthread.so.0
 
#5  0x00007f5e6b92c7fd in clone () from /lib64/libc.so.6
 
 
 
Thread 17 (Thread 0x7f5e527fc700 (LWP 10158)):
 
#0  0x00007f5e6d503408 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
 
#1  0x00007f5e6e1e26b1 in os_event::timed_wait (this=this@entry=0x7f5e712c1650, abstime=abstime@entry=0x7f5e527fbda0) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0event.cc:286
 
#2  0x00007f5e6e1e28df in os_event::wait_time_low (this=0x7f5e712c1650, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=<optimized out>) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0event.cc:413
 
#3  0x00007f5e6e1e2c0a in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=<optimized out>) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0event.cc:518
 
#4  0x00007f5e6e2706e0 in srv_monitor_thread () at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/srv/srv0srv.cc:1732
 
#5  0x00007f5e6d4ff0a4 in start_thread () from /lib64/libpthread.so.0
 
#6  0x00007f5e6b92c7fd in clone () from /lib64/libc.so.6
 
 
 
Thread 16 (Thread 0x7f5e64203700 (LWP 10171)):
 
#0  0x00007f5e6d50305f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
 
#1  0x00007f5e6ddcc629 in inline_mysql_cond_wait (src_line=334, that=<optimized out>, mutex=<optimized out>, src_file=<optimized out>) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/include/mysql/psi/mysql_thread.h:1138
 
#2  handle_slave_background (arg=arg@entry=0x0) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/sql/slave.cc:334
 
#3  0x00007f5e6e1464ad in pfs_spawn_thread (arg=0x7f5e7170b8d8) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/perfschema/pfs.cc:1862
 
#4  0x00007f5e6d4ff0a4 in start_thread () from /lib64/libpthread.so.0
 
#5  0x00007f5e6b92c7fd in clone () from /lib64/libc.so.6
 
 
 
Thread 15 (Thread 0x7f5e5dffb700 (LWP 10152)):
 
#0  0x00007f5e6d0cf6a4 in ?? () from /lib64/libaio.so.1
 
#1  0x00007f5e6e1da1ac in LinuxAIOHandler::collect (this=this@entry=0x7f5e5dffac50) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0file.cc:1930
 
#2  0x00007f5e6e1da792 in LinuxAIOHandler::poll (this=this@entry=0x7f5e5dffac50, m1=m1@entry=0x7f5e5dffacf0, m2=m2@entry=0x7f5e5dffad00, request=request@entry=0x7f5e5dffad10) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0file.cc:2075
 
#3  0x00007f5e6e1e1a8e in os_aio_linux_handler (request=0x7f5e5dffad10, m2=0x7f5e5dffad00, m1=0x7f5e5dffacf0, global_segment=4) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0file.cc:2129
 
#4  os_aio_handler (segment=segment@entry=4, m1=m1@entry=0x7f5e5dffacf0, m2=m2@entry=0x7f5e5dffad00, request=request@entry=0x7f5e5dffad10) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0file.cc:5734
 
#5  0x00007f5e6e375917 in fil_aio_wait (segment=segment@entry=4) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/fil/fil0fil.cc:5129
 
#6  0x00007f5e6e2765b8 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/srv/srv0start.cc:337
 
#7  0x00007f5e6d4ff0a4 in start_thread () from /lib64/libpthread.so.0
 
#8  0x00007f5e6b92c7fd in clone () from /lib64/libc.so.6
 
 
 
Thread 14 (Thread 0x7f5e52ffd700 (LWP 10157)):
 
#0  0x00007f5e6d503408 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
 
#1  0x00007f5e6e1e26b1 in os_event::timed_wait (this=this@entry=0x7f5e712c15d0, abstime=abstime@entry=0x7f5e52ffccc0) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0event.cc:286
 
#2  0x00007f5e6e1e28df in os_event::wait_time_low (this=0x7f5e712c15d0, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=<optimized out>) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0event.cc:413
 
#3  0x00007f5e6e1e2c0a in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=<optimized out>) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0event.cc:518
 
#4  0x00007f5e6e2710cc in srv_error_monitor_thread () at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/srv/srv0srv.cc:1896
 
#5  0x00007f5e6d4ff0a4 in start_thread () from /lib64/libpthread.so.0
 
#6  0x00007f5e6b92c7fd in clone () from /lib64/libc.so.6
 
 
 
Thread 13 (Thread 0x7f5e641b9700 (LWP 10174)):
 
#0  0x00007f5e6d503408 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
 
#1  0x00007f5e6ddb18df in inline_mysql_cond_timedwait (src_line=2973, abstime=0x7f5e641b8dc0, mutex=0x7f5e6edd3660 <LOCK_thread_cache>, that=0x7f5e6edd5e40 <COND_thread_cache>, src_file=<optimized out>) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/include/mysql/psi/mysql_thread.h:1175
 
#2  cache_thread (thd=0x7f5e140009a8) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/sql/mysqld.cc:2972
 
#3  one_thread_per_connection_end (thd=0x7f5e140009a8, put_in_cache=<optimized out>) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/sql/mysqld.cc:3061
 
#4  0x00007f5e6df0bfae in do_handle_one_connection (connect=connect@entry=0x7f5e716cdf18) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/sql/sql_connect.cc:1354
 
#5  0x00007f5e6df0c144 in handle_one_connection (arg=arg@entry=0x7f5e716cdf18) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/sql/sql_connect.cc:1241
 
#6  0x00007f5e6e1464ad in pfs_spawn_thread (arg=0x7f5e716ce178) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/perfschema/pfs.cc:1862
 
#7  0x00007f5e6d4ff0a4 in start_thread () from /lib64/libpthread.so.0
 
#8  0x00007f5e6b92c7fd in clone () from /lib64/libc.so.6
 
 
 
Thread 12 (Thread 0x7f5e5e7fc700 (LWP 10151)):
 
#0  0x00007f5e6d0cf6a4 in ?? () from /lib64/libaio.so.1
 
#1  0x00007f5e6e1da1ac in LinuxAIOHandler::collect (this=this@entry=0x7f5e5e7fbc50) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0file.cc:1930
 
#2  0x00007f5e6e1da792 in LinuxAIOHandler::poll (this=this@entry=0x7f5e5e7fbc50, m1=m1@entry=0x7f5e5e7fbcf0, m2=m2@entry=0x7f5e5e7fbd00, request=request@entry=0x7f5e5e7fbd10) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0file.cc:2075
 
#3  0x00007f5e6e1e1a8e in os_aio_linux_handler (request=0x7f5e5e7fbd10, m2=0x7f5e5e7fbd00, m1=0x7f5e5e7fbcf0, global_segment=3) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0file.cc:2129
 
#4  os_aio_handler (segment=segment@entry=3, m1=m1@entry=0x7f5e5e7fbcf0, m2=m2@entry=0x7f5e5e7fbd00, request=request@entry=0x7f5e5e7fbd10) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0file.cc:5734
 
#5  0x00007f5e6e375917 in fil_aio_wait (segment=segment@entry=3) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/fil/fil0fil.cc:5129
 
#6  0x00007f5e6e2765b8 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/srv/srv0start.cc:337
 
#7  0x00007f5e6d4ff0a4 in start_thread () from /lib64/libpthread.so.0
 
#8  0x00007f5e6b92c7fd in clone () from /lib64/libc.so.6
 
 
 
Thread 11 (Thread 0x7f5e6416f700 (LWP 10177)):
 
#0  0x00007f5e6e30078c in buf_page_hash_get_low (buf_pool=0x7f5e71409c50, page_id=...) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/include/buf0buf.ic:1097
 
#1  0x00007f5e6e30788c in buf_page_get_gen (page_id=..., page_size=..., rw_latch=rw_latch@entry=8, guess=guess@entry=0x0, mode=mode@entry=10, file=file@entry=0x7f5e6e64ae30 "/home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/row/row0uins.cc", line=line@entry=157, mtr=mtr@entry=0x7f5e6416d970, err=err@entry=0x7f5e6416c970) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/buf/buf0buf.cc:4390
 
#2  0x00007f5e6e2e3879 in btr_cur_search_to_nth_level (index=index@entry=0x7f5e140ae600, level=level@entry=0, tuple=tuple@entry=0x7f5e08030300, mode=PAGE_CUR_LE, latch_mode=33, cursor=cursor@entry=0x7f5e0802fab0, has_search_latch=has_search_latch@entry=0, file=file@entry=0x7f5e6e64ae30 "/home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/row/row0uins.cc", line=line@entry=157, mtr=mtr@entry=0x7f5e6416d970, autoinc=autoinc@entry=0) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/btr/btr0cur.cc:1218
 
#3  0x00007f5e6e2ecfcd in btr_pcur_open_with_no_init_func (mtr=0x7f5e6416d970, line=157, file=0x7f5e6e64ae30 "/home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/row/row0uins.cc", cursor=0x7f5e0802fab0, latch_mode=<optimized out>, mode=<optimized out>, tuple=0x7f5e08030300, index=0x7f5e140ae600, has_search_latch=<optimized out>) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/include/btr0pcur.ic:545
 
#4  btr_pcur_restore_position_func (latch_mode=latch_mode@entry=65569, cursor=cursor@entry=0x7f5e0802fab0, file=file@entry=0x7f5e6e64ae30 "/home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/row/row0uins.cc", line=line@entry=157, mtr=mtr@entry=0x7f5e6416d970) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/btr/btr0pcur.cc:356
 
#5  0x00007f5e6e3e5c49 in row_undo_ins_remove_clust_rec (node=node@entry=0x7f5e0802fa40) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/row/row0uins.cc:157
 
#6  0x00007f5e6e3e6453 in row_undo_ins (node=0x7f5e0802fa40, thr=0x7f5e0802f890) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/row/row0uins.cc:525
 
#7  0x00007f5e6e25d85b in row_undo (thr=0x7f5e0802f890, node=0x7f5e0802fa40) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/row/row0undo.cc:299
 
#8  row_undo_step (thr=thr@entry=0x7f5e0802f890) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/row/row0undo.cc:352
 
#9  0x00007f5e6e203927 in que_thr_step (thr=0x7f5e0802f890) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/que/que0que.cc:1047
 
#10 que_run_threads_low (thr=0x7f5e0802f890) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/que/que0que.cc:1111
 
#11 que_run_threads (thr=0x7f5e0802f890) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/que/que0que.cc:1151
 
#12 0x00007f5e6e2a2f15 in trx_rollback_to_savepoint_low (trx=0x7f5e64cc4118, savept=savept@entry=0x0) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/trx/trx0roll.cc:112
 
#13 0x00007f5e6e2a390b in trx_rollback_for_mysql_low (trx=<optimized out>) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/trx/trx0roll.cc:174
 
#14 trx_rollback_for_mysql (trx=trx@entry=0x7f5e64cc4118) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/trx/trx0roll.cc:202
 
#15 0x00007f5e6e161569 in innobase_rollback (hton=<optimized out>, thd=0x7f5e080009a8, rollback_trx=true) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/handler/ha_innodb.cc:4834
 
#16 0x00007f5e6dfc9552 in ha_rollback_trans (thd=thd@entry=0x7f5e080009a8, all=all@entry=true) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/sql/handler.cc:1673
 
#17 0x00007f5e6df1985a in trans_rollback (thd=thd@entry=0x7f5e080009a8) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/sql/transaction.cc:415
 
#18 0x00007f5e6de19299 in THD::cleanup (this=this@entry=0x7f5e080009a8) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/sql/sql_class.cc:1368
 
#19 0x00007f5e6ddb15e9 in unlink_thd (thd=thd@entry=0x7f5e080009a8) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/sql/mysqld.cc:2911
 
#20 0x00007f5e6ddb1647 in one_thread_per_connection_end (thd=0x7f5e080009a8, put_in_cache=true) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/sql/mysqld.cc:3060
 
#21 0x00007f5e6df0bfae in do_handle_one_connection (connect=connect@entry=0x7f5e716cdf18) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/sql/sql_connect.cc:1354
 
#22 0x00007f5e6df0c144 in handle_one_connection (arg=arg@entry=0x7f5e716cdf18) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/sql/sql_connect.cc:1241
 
#23 0x00007f5e6e1464ad in pfs_spawn_thread (arg=0x7f5e7117a6e8) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/perfschema/pfs.cc:1862
 
#24 0x00007f5e6d4ff0a4 in start_thread () from /lib64/libpthread.so.0
 
#25 0x00007f5e6b92c7fd in clone () from /lib64/libc.so.6
 
 
 
Thread 10 (Thread 0x7f5e66c2f700 (LWP 10146)):
 
#0  0x00007f5e6d503408 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
 
#1  0x00007f5e6e4f9624 in inline_mysql_cond_timedwait (abstime=0x7f5e66c2ee20, that=<optimized out>, mutex=<optimized out>, src_file=<optimized out>, src_line=<optimized out>) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/include/mysql/psi/mysql_thread.h:1175
 
#2  timer_handler (arg=arg@entry=0x0) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/mysys/thr_timer.c:292
 
#3  0x00007f5e6e1464ad in pfs_spawn_thread (arg=0x7f5e711b6468) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/perfschema/pfs.cc:1862
 
#4  0x00007f5e6d4ff0a4 in start_thread () from /lib64/libpthread.so.0
 
#5  0x00007f5e6b92c7fd in clone () from /lib64/libc.so.6
 
 
 
Thread 9 (Thread 0x7f5e5cff9700 (LWP 10154)):
 
#0  0x00007f5e6d503408 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
 
#1  0x00007f5e6e1e26b1 in os_event::timed_wait (this=this@entry=0x7f5e712c1750, abstime=abstime@entry=0x7f5e5cff8ba0) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0event.cc:286
 
#2  0x00007f5e6e1e28df in os_event::wait_time_low (this=0x7f5e712c1750, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=<optimized out>, reset_sig_count@entry=94) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0event.cc:413
 
#3  0x00007f5e6e1e2c0a in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=reset_sig_count@entry=94) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0event.cc:518
 
#4  0x00007f5e6e31c9f8 in pc_sleep_if_needed (cur_time=1538123647985, sig_count=94, next_loop_time=1538123648985) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/buf/buf0flu.cc:2705
 
#5  buf_flush_page_cleaner_coordinator () at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/buf/buf0flu.cc:3171
 
#6  0x00007f5e6d4ff0a4 in start_thread () from /lib64/libpthread.so.0
 
#7  0x00007f5e6b92c7fd in clone () from /lib64/libc.so.6
 
 
 
Thread 8 (Thread 0x7f5e517fa700 (LWP 10161)):
 
#0  0x00007f5e6d503408 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
 
#1  0x00007f5e6e1e26b1 in os_event::timed_wait (this=this@entry=0x7f5e716a4470, abstime=abstime@entry=0x7f5e517f9c10) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0event.cc:286
 
#2  0x00007f5e6e1e28df in os_event::wait_time_low (this=0x7f5e716a4470, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=<optimized out>, reset_sig_count@entry=1) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0event.cc:413
 
#3  0x00007f5e6e1e2c0a in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=reset_sig_count@entry=1) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0event.cc:518
 
#4  0x00007f5e6e2c396b in ib_wqueue_timedwait (wq=wq@entry=0x7f5e716a4330, wait_in_usecs=wait_in_usecs@entry=5000000) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/ut/ut0wqueue.cc:163
 
#5  0x00007f5e6e3bbce2 in fts_optimize_thread (arg=0x7f5e716a4330) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/fts/fts0opt.cc:2899
 
#6  0x00007f5e6d4ff0a4 in start_thread () from /lib64/libpthread.so.0
 
#7  0x00007f5e6b92c7fd in clone () from /lib64/libc.so.6
 
 
 
Thread 7 (Thread 0x7f5e5f7fe700 (LWP 10149)):
 
#0  0x00007f5e6d0cf6a4 in ?? () from /lib64/libaio.so.1
 
#1  0x00007f5e6e1da1ac in LinuxAIOHandler::collect (this=this@entry=0x7f5e5f7fdc50) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0file.cc:1930
 
#2  0x00007f5e6e1da792 in LinuxAIOHandler::poll (this=this@entry=0x7f5e5f7fdc50, m1=m1@entry=0x7f5e5f7fdcf0, m2=m2@entry=0x7f5e5f7fdd00, request=request@entry=0x7f5e5f7fdd10) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0file.cc:2075
 
#3  0x00007f5e6e1e1a8e in os_aio_linux_handler (request=0x7f5e5f7fdd10, m2=0x7f5e5f7fdd00, m1=0x7f5e5f7fdcf0, global_segment=1) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0file.cc:2129
 
#4  os_aio_handler (segment=segment@entry=1, m1=m1@entry=0x7f5e5f7fdcf0, m2=m2@entry=0x7f5e5f7fdd00, request=request@entry=0x7f5e5f7fdd10) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0file.cc:5734
 
#5  0x00007f5e6e375917 in fil_aio_wait (segment=segment@entry=1) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/fil/fil0fil.cc:5129
 
#6  0x00007f5e6e2765b8 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/srv/srv0start.cc:337
 
#7  0x00007f5e6d4ff0a4 in start_thread () from /lib64/libpthread.so.0
 
#8  0x00007f5e6b92c7fd in clone () from /lib64/libc.so.6
 
 
 
Thread 6 (Thread 0x7f5e64125700 (LWP 10178)):
 
#0  0x00007f5e6d5057bc in __lll_lock_wait () from /lib64/libpthread.so.0
 
#1  0x00007f5e6d501489 in _L_lock_909 () from /lib64/libpthread.so.0
 
#2  0x00007f5e6d5012b0 in pthread_mutex_lock () from /lib64/libpthread.so.0
 
#3  0x00007f5e6e4f93af in rw_pr_wrlock (rwlock=0x7f5e7117ab10) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/mysys/thr_rwlock.c:392
 
#4  0x00007f5e6df122ab in inline_mysql_prlock_wrlock (that=0x7f5e7117ab10, src_file=<optimized out>, src_line=<optimized out>) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/include/mysql/psi/mysql_thread.h:960
 
#5  0x00007f5e6df12a59 in MDL_map::find_or_insert (this=this@entry=0x7f5e6ede04c0 <mdl_locks>, pins=<optimized out>, mdl_key=mdl_key@entry=0x7f5e64120830) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/sql/mdl.cc:750
 
#6  0x00007f5e6df1416a in MDL_context::try_acquire_lock_impl (this=this@entry=0x7f5e0c000ac8, mdl_request=mdl_request@entry=0x7f5e64120810, out_ticket=out_ticket@entry=0x7f5e641205f0) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/sql/mdl.cc:1915
 
#7  0x00007f5e6df145d0 in MDL_context::acquire_lock (this=this@entry=0x7f5e0c000ac8, mdl_request=mdl_request@entry=0x7f5e64120810, lock_wait_timeout=86400) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/sql/mdl.cc:2067
 
#8  0x00007f5e6ddff668 in open_table (thd=thd@entry=0x7f5e0c0009a8, table_list=table_list@entry=0x7f5e0c017980, ot_ctx=ot_ctx@entry=0x7f5e64120d00) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/sql/sql_base.cc:1742
 
#9  0x00007f5e6de029eb in open_and_process_table (ot_ctx=0x7f5e64120d00, has_prelocking_list=false, prelocking_strategy=0x7f5e64120ee0, flags=0, counter=0x7f5e64120d90, tables=0x7f5e0c017980, lex=0x7f5e0c01ca90, thd=0x7f5e0c0009a8) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/sql/sql_base.cc:3488
 
#10 open_tables (thd=thd@entry=0x7f5e0c0009a8, options=..., start=start@entry=0x7f5e64120d88, counter=counter@entry=0x7f5e64120d90, flags=flags@entry=0, prelocking_strategy=prelocking_strategy@entry=0x7f5e64120ee0) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/sql/sql_base.cc:4011
 
#11 0x00007f5e6de02d34 in open_and_lock_tables (thd=thd@entry=0x7f5e0c0009a8, options=..., tables=tables@entry=0x7f5e0c017980, derived=derived@entry=true, flags=flags@entry=0, prelocking_strategy=prelocking_strategy@entry=0x7f5e64120ee0) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/sql/sql_base.cc:4766
 
#12 0x00007f5e6de2e35b in open_and_lock_tables (flags=0, derived=true, tables=0x7f5e0c017980, thd=0x7f5e0c0009a8) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/sql/sql_base.h:506
 
#13 mysql_insert (thd=thd@entry=0x7f5e0c0009a8, table_list=0x7f5e0c017980, fields=..., values_list=..., update_fields=..., update_values=..., duplic=DUP_ERROR, ignore=false) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/sql/sql_insert.cc:758
 
#14 0x00007f5e6de44619 in mysql_execute_command (thd=0x7f5e0c0009a8) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/sql/sql_parse.cc:4435
 
#15 0x00007f5e6e0c773e in sp_instr_stmt::exec_core (this=0x7f5e0c018150, thd=<optimized out>, nextp=0x7f5e641227f0) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/sql/sp_head.cc:3246
 
#16 0x00007f5e6e0cd419 in sp_lex_keeper::reset_lex_and_exec_core (this=this@entry=0x7f5e0c018190, thd=thd@entry=0x7f5e0c0009a8, nextp=nextp@entry=0x7f5e641227f0, open_tables=open_tables@entry=false, instr=instr@entry=0x7f5e0c018150) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/sql/sp_head.cc:3009
 
#17 0x00007f5e6e0cd955 in sp_instr_stmt::execute (this=0x7f5e0c018150, thd=0x7f5e0c0009a8, nextp=0x7f5e641227f0) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/sql/sp_head.cc:3162
 
#18 0x00007f5e6e0ca7c2 in sp_head::execute (this=this@entry=0x7f5e0c016420, thd=thd@entry=0x7f5e0c0009a8, merge_da_on_success=merge_da_on_success@entry=true) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/sql/sp_head.cc:1327
 
#19 0x00007f5e6e0cbbde in sp_head::execute_procedure (this=0x7f5e0c016420, thd=thd@entry=0x7f5e0c0009a8, args=0x7f5e0c005100) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/sql/sp_head.cc:2116
 
#20 0x00007f5e6de3bc0b in do_execute_sp (thd=thd@entry=0x7f5e0c0009a8, sp=<optimized out>) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/sql/sql_parse.cc:2912
 
#21 0x00007f5e6de41cde in mysql_execute_command (thd=thd@entry=0x7f5e0c0009a8) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/sql/sql_parse.cc:5835
 
#22 0x00007f5e6de491aa in mysql_parse (thd=thd@entry=0x7f5e0c0009a8, rawbuf=<optimized out>, length=18, parser_state=parser_state@entry=0x7f5e64124220, is_com_multi=is_com_multi@entry=false, is_next_command=is_next_command@entry=false) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/sql/sql_parse.cc:8011
 
#23 0x00007f5e6de4b5ce in dispatch_command (command=command@entry=COM_QUERY, thd=thd@entry=0x7f5e0c0009a8, packet=<optimized out>, packet_length=201388096, packet_length@entry=18, is_com_multi=is_com_multi@entry=false, is_next_command=is_next_command@entry=false) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/sql/sql_parse.cc:1824
 
#24 0x00007f5e6de4bbbd in do_command (thd=0x7f5e0c0009a8) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/sql/sql_parse.cc:1378
 
#25 0x00007f5e6df0c01f in do_handle_one_connection (connect=connect@entry=0x7f5e716cdf18) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/sql/sql_connect.cc:1335
 
#26 0x00007f5e6df0c144 in handle_one_connection (arg=arg@entry=0x7f5e716cdf18) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/sql/sql_connect.cc:1241
 
#27 0x00007f5e6e1464ad in pfs_spawn_thread (arg=0x7f5e716ce4b8) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/perfschema/pfs.cc:1862
 
#28 0x00007f5e6d4ff0a4 in start_thread () from /lib64/libpthread.so.0
 
#29 0x00007f5e6b92c7fd in clone () from /lib64/libc.so.6
 
 
 
Thread 5 (Thread 0x7f5e5effd700 (LWP 10150)):
 
#0  0x00007f5e6d0cf6a4 in ?? () from /lib64/libaio.so.1
 
#1  0x00007f5e6e1da1ac in LinuxAIOHandler::collect (this=this@entry=0x7f5e5effcc50) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0file.cc:1930
 
#2  0x00007f5e6e1da792 in LinuxAIOHandler::poll (this=this@entry=0x7f5e5effcc50, m1=m1@entry=0x7f5e5effccf0, m2=m2@entry=0x7f5e5effcd00, request=request@entry=0x7f5e5effcd10) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0file.cc:2075
 
#3  0x00007f5e6e1e1a8e in os_aio_linux_handler (request=0x7f5e5effcd10, m2=0x7f5e5effcd00, m1=0x7f5e5effccf0, global_segment=2) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0file.cc:2129
 
#4  os_aio_handler (segment=segment@entry=2, m1=m1@entry=0x7f5e5effccf0, m2=m2@entry=0x7f5e5effcd00, request=request@entry=0x7f5e5effcd10) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0file.cc:5734
 
#5  0x00007f5e6e375917 in fil_aio_wait (segment=segment@entry=2) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/fil/fil0fil.cc:5129
 
#6  0x00007f5e6e2765b8 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/srv/srv0start.cc:337
 
#7  0x00007f5e6d4ff0a4 in start_thread () from /lib64/libpthread.so.0
 
#8  0x00007f5e6b92c7fd in clone () from /lib64/libc.so.6
 
 
 
Thread 4 (Thread 0x7f5e5d7fa700 (LWP 10153)):
 
#0  0x00007f5e6d0cf6a4 in ?? () from /lib64/libaio.so.1
 
#1  0x00007f5e6e1da1ac in LinuxAIOHandler::collect (this=this@entry=0x7f5e5d7f9c50) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0file.cc:1930
 
#2  0x00007f5e6e1da792 in LinuxAIOHandler::poll (this=this@entry=0x7f5e5d7f9c50, m1=m1@entry=0x7f5e5d7f9cf0, m2=m2@entry=0x7f5e5d7f9d00, request=request@entry=0x7f5e5d7f9d10) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0file.cc:2075
 
#3  0x00007f5e6e1e1a8e in os_aio_linux_handler (request=0x7f5e5d7f9d10, m2=0x7f5e5d7f9d00, m1=0x7f5e5d7f9cf0, global_segment=5) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0file.cc:2129
 
#4  os_aio_handler (segment=segment@entry=5, m1=m1@entry=0x7f5e5d7f9cf0, m2=m2@entry=0x7f5e5d7f9d00, request=request@entry=0x7f5e5d7f9d10) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0file.cc:5734
 
#5  0x00007f5e6e375917 in fil_aio_wait (segment=segment@entry=5) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/fil/fil0fil.cc:5129
 
#6  0x00007f5e6e2765b8 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/srv/srv0start.cc:337
 
#7  0x00007f5e6d4ff0a4 in start_thread () from /lib64/libpthread.so.0
 
#8  0x00007f5e6b92c7fd in clone () from /lib64/libc.so.6
 
 
 
Thread 3 (Thread 0x7f5e5ffff700 (LWP 10148)):
 
#0  0x00007f5e6d0cf6a4 in ?? () from /lib64/libaio.so.1
 
#1  0x00007f5e6e1da1ac in LinuxAIOHandler::collect (this=this@entry=0x7f5e5fffec50) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0file.cc:1930
 
#2  0x00007f5e6e1da792 in LinuxAIOHandler::poll (this=this@entry=0x7f5e5fffec50, m1=m1@entry=0x7f5e5fffecf0, m2=m2@entry=0x7f5e5fffed00, request=request@entry=0x7f5e5fffed10) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0file.cc:2075
 
#3  0x00007f5e6e1e1a8e in os_aio_linux_handler (request=0x7f5e5fffed10, m2=0x7f5e5fffed00, m1=0x7f5e5fffecf0, global_segment=0) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0file.cc:2129
 
#4  os_aio_handler (segment=segment@entry=0, m1=m1@entry=0x7f5e5fffecf0, m2=m2@entry=0x7f5e5fffed00, request=request@entry=0x7f5e5fffed10) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/os/os0file.cc:5734
 
#5  0x00007f5e6e375917 in fil_aio_wait (segment=segment@entry=0) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/fil/fil0fil.cc:5129
 
#6  0x00007f5e6e2765b8 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/srv/srv0start.cc:337
 
#7  0x00007f5e6d4ff0a4 in start_thread () from /lib64/libpthread.so.0
 
#8  0x00007f5e6b92c7fd in clone () from /lib64/libc.so.6
 
 
 
Thread 2 (Thread 0x7f5e658c3700 (LWP 10147)):
 
#0  0x00007f5e6d503408 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
 
#1  0x00007f5e6e42620b in inline_mysql_cond_timedwait (abstime=0x7f5e658c2da0, mutex=0x7f5e6f657b40 <LOCK_checkpoint>, that=0x7f5e6f657b80 <COND_checkpoint>, src_file=<optimized out>, src_line=<optimized out>) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/include/mysql/psi/mysql_thread.h:1175
 
#2  my_service_thread_sleep (control=control@entry=0x7f5e6ed56be0 <checkpoint_control>, sleep_time=<optimized out>) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/maria/ma_servicethread.c:115
 
#3  0x00007f5e6e41ea59 in ma_checkpoint_background (arg=arg@entry=0x1e) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/maria/ma_checkpoint.c:709
 
#4  0x00007f5e6e1464ad in pfs_spawn_thread (arg=0x7f5e71279108) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/perfschema/pfs.cc:1862
 
#5  0x00007f5e6d4ff0a4 in start_thread () from /lib64/libpthread.so.0
 
#6  0x00007f5e6b92c7fd in clone () from /lib64/libc.so.6
 
 
 
Thread 1 (Thread 0x7f5e6d91c780 (LWP 10117)):
 
#0  0x00007f5e6d5040d1 in pthread_kill () from /lib64/libpthread.so.0
 
#1  0x00007f5e6dfc67f9 in handle_fatal_signal (sig=6) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/sql/signal_handler.cc:305
 
#2  <signal handler called>
 
#3  0x00007f5e6b9243cd in poll () from /lib64/libc.so.6
 
#4  0x00007f5e6ddb682f in poll (__timeout=-1, __nfds=2, __fds=0x7fff447672a0) at /usr/include/bits/poll2.h:41
 
#5  handle_connections_sockets () at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/sql/mysqld.cc:6609
 
#6  0x00007f5e6ddb9b60 in mysqld_main (argc=146, argv=0x7f5e7107ea68) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/sql/mysqld.cc:6075
 
#7  0x00007f5e6b868b05 in __libc_start_main () from /lib64/libc.so.6
 
#8  0x00007f5e6ddad51d in _start () at ../sysdeps/x86_64/start.S:112



 Comments   
Comment by Marko Mäkelä [ 2019-11-11 ]

The logs are no longer available on buildbot, and no excerpt from the server error is present. At first I was suspecting that this could duplicate MDEV-19176, which is a hang during startup, because of running out of buffer pool. But, this test does not restart the server, so the cause should be something else.

Comment by Marko Mäkelä [ 2021-10-01 ]

I was thinking whether this could be related to MDEV-13564 or MDEV-26450, but then I realized that the problem really could be MDL. Why would a trylock operation block?

Thread 6 (Thread 0x7f5e64125700 (LWP 10178)):
 
#0  0x00007f5e6d5057bc in __lll_lock_wait () from /lib64/libpthread.so.0
 
#1  0x00007f5e6d501489 in _L_lock_909 () from /lib64/libpthread.so.0
 
#2  0x00007f5e6d5012b0 in pthread_mutex_lock () from /lib64/libpthread.so.0
 
#3  0x00007f5e6e4f93af in rw_pr_wrlock (rwlock=0x7f5e7117ab10) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/mysys/thr_rwlock.c:392
 
#4  0x00007f5e6df122ab in inline_mysql_prlock_wrlock (that=0x7f5e7117ab10, src_file=<optimized out>, src_line=<optimized out>) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/include/mysql/psi/mysql_thread.h:960
 
#5  0x00007f5e6df12a59 in MDL_map::find_or_insert (this=this@entry=0x7f5e6ede04c0 <mdl_locks>, pins=<optimized out>, mdl_key=mdl_key@entry=0x7f5e64120830) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/sql/mdl.cc:750
 
#6  0x00007f5e6df1416a in MDL_context::try_acquire_lock_impl (this=this@entry=0x7f5e0c000ac8, mdl_request=mdl_request@entry=0x7f5e64120810, out_ticket=out_ticket@entry=0x7f5e641205f0) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/sql/mdl.cc:1915
 
#7  0x00007f5e6df145d0 in MDL_context::acquire_lock (this=this@entry=0x7f5e0c000ac8, mdl_request=mdl_request@entry=0x7f5e64120810, lock_wait_timeout=86400) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/sql/mdl.cc:2067
 
#8  0x00007f5e6ddff668 in open_table (thd=thd@entry=0x7f5e0c0009a8, table_list=table_list@entry=0x7f5e0c017980, ot_ctx=ot_ctx@entry=0x7f5e64120d00) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/sql/sql_base.cc:1742

At the same time, we had this executing in another thread. Note: that thread would not acquire any MDL in this version. (Unfortunately the exact commit hash is not available.)

#15 0x00007f5e6e161569 in innobase_rollback (hton=<optimized out>, thd=0x7f5e080009a8, rollback_trx=true) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/storage/innobase/handler/ha_innodb.cc:4834
 
#16 0x00007f5e6dfc9552 in ha_rollback_trans (thd=thd@entry=0x7f5e080009a8, all=all@entry=true) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/sql/handler.cc:1673
 
#17 0x00007f5e6df1985a in trans_rollback (thd=thd@entry=0x7f5e080009a8) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/sql/transaction.cc:415
 
#18 0x00007f5e6de19299 in THD::cleanup (this=this@entry=0x7f5e080009a8) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/sql/sql_class.cc:1368
 
#19 0x00007f5e6ddb15e9 in unlink_thd (thd=thd@entry=0x7f5e080009a8) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/sql/mysqld.cc:2911
 
#20 0x00007f5e6ddb1647 in one_thread_per_connection_end (thd=0x7f5e080009a8, put_in_cache=true) at /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.19/sql/mysqld.cc:3060

Maybe this just is some race condition between MDL and unlink_thd()?

Comment by Vladislav Vaintroub [ 2021-10-08 ]

he trylock operation blocks, because it either finds a conflict, or not. it can fail only if there is a allocation error or something like that, so the comment for this function.

Comment by Marko Mäkelä [ 2022-02-11 ]

This particular type of test failure does not seem to have repeated ever since in the 10.2 branch.

There are other failures, though, for various reasons. MDEV-13564 removed the use of a separate log file for undo tablespace truncation. Before an MLOG_MEMSET record was introduced in 10.4, the mini-transaction log for reinitializing an undo tablespace would be very long, and the tests in 10.2 and 10.3 could fail due to that.

As of this writing, the latest fixes to undo log truncation include MDEV-26450 and MDEV-26864.

Generated at Thu Feb 08 08:35:43 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.