[MDEV-17438] rpl.show_status_stop_slave_race-7126 fails with timeout on Windows Created: 2018-10-12  Updated: 2020-12-08  Resolved: 2020-09-11

Status: Closed
Project: MariaDB Server
Component/s: Platform Windows, Replication, Tests
Affects Version/s: 10.3
Fix Version/s: 10.1.48, 10.2.35, 10.3.26, 10.4.16, 10.5.7

Type: Bug Priority: Critical
Reporter: Vladislav Vaintroub Assignee: Sachin Setiya (Inactive)
Resolution: Fixed Votes: 0
Labels: None


 Description   

worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019
rpl.show_status_stop_slave_race-7126 'mix' [ fail ]
        Test ended at 2018-10-12 16:33:01
 
CURRENT_TEST: rpl.show_status_stop_slave_race-7126
R:\build\client\Debug\mysqlslap.exe: Cannot run query START SLAVE; STOP SLAVE; S
HOW GLOBAL STATUS ERROR : Misconfigured slave: MASTER_HOST was not set; Fix in c
onfig file or with CHANGE MASTER TO
Rmysqltest: In included file "./include/wait_for_slave_param.inc":
included from ./include/wait_for_slave_sql_to_stop.inc at line 36:
included from ./include/wait_for_slave_to_stop.inc at line 34:
included from D:/work/10.3/mysql-test/suite/rpl/t/show_status_stop_slave_race-71
26.test at line 19:
At line 119: Timeout in include/wait_for_slave_param.inc
 
The result from queries just before the failure was:
< snip >
SHOW BINARY LOGS;
Log_name        File_size
master-bin.000001       664
 
**** SHOW BINLOG EVENTS on server_1 ****
binlog_name = 'master-bin.000001'
SHOW BINLOG EVENTS IN 'master-bin.000001';
Log_name        Pos     Event_type      Server_id       End_log_pos     Info
master-bin.000001       4       Format_desc     1       256     Server ver: 10.3
.11-MariaDB-debug-log, Binlog ver: 4
master-bin.000001       256     Gtid_list       1       285     []
master-bin.000001       285     Binlog_checkpoint       1       329     master-b
in.000001
master-bin.000001       329     Gtid    1       371     BEGIN GTID 0-1-1
master-bin.000001       371     Query   1       592     use `mtr`; INSERT INTO t
est_suppressions (pattern) VALUES ( NAME_CONST('pattern',_latin1'Master is confi
gured to log replication events' COLLATE 'latin1_swedish_ci'))
master-bin.000001       592     Query   1       664     COMMIT
 
**** SHOW RELAYLOG EVENTS on server_1 ****
relaylog_name = 'No such row'
SHOW RELAYLOG EVENTS IN 'No such row';
ERROR HY000: Error when executing command SHOW BINLOG EVENTS: Could not find tar
get log
connection slave;
 
More results from queries before failure can be found in R:\build\mysql-test\var
\log\show_status_stop_slave_race-7126.log
 
 - saving 'R:/build/mysql-test/var/log/rpl.show_status_stop_slave_race-7126-mix/
' to 'R:/build/mysql-test/var/log/rpl.show_status_stop_slave_race-7126-mix/'
 
Only  1  of 3 completed.
--------------------------------------------------------------------------
The servers were restarted 0 times
Spent 0.000 of 631 seconds executing testcases
 
Failure: Failed 1/1 tests, 0.00% were successful.
 
Failing test(s): rpl.show_status_stop_slave_race-7126



 Comments   
Comment by Elena Stepanova [ 2018-10-13 ]

The test also fails (crashes) in buildbot on non-Windows builders. Please check cross-reference to determine whether it's the same problem or it needs a separate bug report.

Comment by Andrei Elkin [ 2020-08-27 ]

sachin.setiya.007: Could you please try reproducing it now as the first step. If it's still there (expected), maybe you could spot something. I tend to suspect the test.

Comment by Sachin Setiya (Inactive) [ 2020-09-08 ]

Not able to find buildbot failure on windows with cross reference. Although it fails in linux but with different error(in asan and normal builds), Tried on my windows installation it is failing.

Comment by Alice Sherepa [ 2020-09-09 ]

recent failures of this test
on 10.2 http://buildbot.askmonty.org/buildbot/builders/kvm-deb-stretch-ppc64le/builds/6957/steps/mtr/logs/stdio

rpl.show_status_stop_slave_race-7126 'row' w3 [ fail ]  timeout after 900 seconds
        Test ended at 2020-09-09 10:35:05
 
Test case timeout after 900 seconds
 
== /dev/shm/var/3/log/show_status_stop_slave_race-7126.log == 
include/master-slave.inc
[connection master]
call mtr.add_suppression("Master is configured to log replication events");
connection slave;
 
 == /dev/shm/var/3/tmp/analyze-timeout-mysqld.1.err ==
mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/dev/shm/var/tmp/3/mysqld.1.sock' (111)
 
 == /dev/shm/var/3/tmp/analyze-timeout-mysqld.2.err ==
mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/dev/shm/var/tmp/3/mysqld.2.sock' (111)
 
 
 - found 'core' (0/0)
 
Trying 'dbx' to get a backtrace
 
Trying 'gdb' to get a backtrace from coredump /dev/shm/var/3/log/rpl.show_status_stop_slave_race-7126-row/mysqld.2/data/core
Core generated by '/usr/sbin/mysqld'
Output from gdb follows. The first stack trace is from the failing thread.
The following stack traces are from all threads (so the failing one is
duplicated).
--------------------------
[New LWP 24113]
[New LWP 24115]
[New LWP 24120]
[New LWP 24118]
[New LWP 24132]
[New LWP 24124]
[New LWP 24140]
[New LWP 24127]
[New LWP 24139]
[New LWP 24133]
[New LWP 24138]
[New LWP 24136]
[New LWP 24137]
[New LWP 24143]
[New LWP 24142]
[New LWP 24144]
[New LWP 24241]
[New LWP 24340]
[New LWP 24131]
[New LWP 24130]
[New LWP 24240]
[New LWP 24128]
[New LWP 24237]
[New LWP 24126]
[New LWP 24141]
[New LWP 24123]
[New LWP 24135]
[New LWP 24121]
[New LWP 24134]
[New LWP 24117]
[New LWP 24129]
[New LWP 24125]
[New LWP 24119]
[New LWP 24116]
[New LWP 24114]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/powerpc64le-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/mysqld --defaults-group-suffix=.2 --defaults-file=/dev/shm/var/3/my.c'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00003fffb20c2898 in __pthread_kill (threadid=<optimized out>, signo=<optimized out>) at ../sysdeps/unix/sysv/linux/pthread_kill.c:55
55	../sysdeps/unix/sysv/linux/pthread_kill.c: No such file or directory.
[Current thread is 1 (Thread 0x3fffb2169ec0 (LWP 24113))]
#0  0x00003fffb20c2898 in __pthread_kill (threadid=<optimized out>, signo=<optimized out>) at ../sysdeps/unix/sysv/linux/pthread_kill.c:55
#1  0x0000000051d116c0 in my_write_core (sig=<optimized out>) at ./mysys/stacktrace.c:382
#2  0x0000000051684c08 in handle_fatal_signal (sig=<optimized out>) at ./sql/signal_handler.cc:343
#3  <signal handler called>
#4  0x00003fffb1814238 in poll () at ../sysdeps/unix/syscall-template.S:84
#5  0x00003fffb183a634 in __poll_chk (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>, fdslen=<optimized out>) at poll_chk.c:27
#6  0x00000000513aada8 in poll (__timeout=-1, __nfds=2, __fds=0x3fffd0d96fd0) at /usr/include/powerpc64le-linux-gnu/bits/poll2.h:41
#7  handle_connections_sockets () at ./sql/mysqld.cc:6669
#8  0x00000000513acd44 in mysqld_main (argc=<optimized out>, argv=<optimized out>) at ./sql/mysqld.cc:6137
#9  0x0000000051385710 in main (argc=<optimized out>, argv=<optimized out>) at ./sql/main.cc:25
 
Thread 35 (Thread 0x3fffacfff000 (LWP 24114)):
#0  0x00003fffb20c1264 in __pthread_cond_timedwait (cond=0x52cfb178 <COND_timer>, mutex=0x52cfb110 <LOCK_timer>, abstime=0x3fffacffe4b8) at pthread_cond_timedwait.c:200
#1  0x0000000051d17f28 in inline_mysql_cond_timedwait (that=0x52cfb178 <COND_timer>, mutex=0x52cfb110 <LOCK_timer>, src_file=0x51ecc1d8 "/home/buildbot/buildbot/build/mariadb-10.2.34/mysys/thr_timer.c", src_line=292, abstime=0x3fffacffe4b8) at ./include/mysql/psi/mysql_thread.h:1182
#2  timer_handler (arg=<optimized out>) at ./mysys/thr_timer.c:292
#3  0x0000000051caa0a8 in pfs_spawn_thread (arg=0x1002a5d6488) at ./storage/perfschema/pfs.cc:1869
#4  0x00003fffb20b80fc in start_thread (arg=0x3fffacfff000) at pthread_create.c:458
#5  0x00003fffb1821748 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:85
 
Thread 34 (Thread 0x3fffa6b9f000 (LWP 24116)):
#0  0x00003fffb20407f8 in ?? () from /lib/powerpc64le-linux-gnu/libaio.so.1
#1  0x00003fffb2040860 in io_getevents () from /lib/powerpc64le-linux-gnu/libaio.so.1
#2  0x000000005191aec0 in LinuxAIOHandler::collect (this=0x3fffa6b9e288) at ./storage/innobase/os/os0file.cc:1936
#3  0x0000000051923188 in LinuxAIOHandler::poll (this=0x3fffa6b9e288, m1=0x3fffa6b9e380, m2=0x3fffa6b9e388, request=0x3fffa6b9e390) at ./storage/innobase/os/os0file.cc:2097
#4  0x0000000051923a28 in os_aio_linux_handler (request=0x3fffa6b9e390, m2=0x3fffa6b9e388, m1=0x3fffa6b9e380, global_segment=<optimized out>) at ./storage/innobase/os/os0file.cc:2151
#5  os_aio_handler (segment=<optimized out>, m1=0x3fffa6b9e380, m2=0x3fffa6b9e388, request=0x3fffa6b9e390) at ./storage/innobase/os/os0file.cc:5781
#6  0x0000000051ae5030 in fil_aio_wait (segment=<optimized out>) at ./storage/innobase/fil/fil0fil.cc:5098
#7  0x00000000519ca038 in io_handler_thread (arg=<optimized out>) at ./storage/innobase/srv/srv0start.cc:332
#8  0x00003fffb20b80fc in start_thread (arg=0x3fffa6b9f000) at pthread_create.c:458
#9  0x00003fffb1821748 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:85
 
Thread 33 (Thread 0x3fffa536f000 (LWP 24119)):
#0  0x00003fffb20407f8 in ?? () from /lib/powerpc64le-linux-gnu/libaio.so.1
#1  0x00003fffb2040860 in io_getevents () from /lib/powerpc64le-linux-gnu/libaio.so.1
#2  0x000000005191aec0 in LinuxAIOHandler::collect (this=0x3fffa536e288) at ./storage/innobase/os/os0file.cc:1936
#3  0x0000000051923188 in LinuxAIOHandler::poll (this=0x3fffa536e288, m1=0x3fffa536e380, m2=0x3fffa536e388, request=0x3fffa536e390) at ./storage/innobase/os/os0file.cc:2097
#4  0x0000000051923a28 in os_aio_linux_handler (request=0x3fffa536e390, m2=0x3fffa536e388, m1=0x3fffa536e380, global_segment=<optimized out>) at ./storage/innobase/os/os0file.cc:2151
#5  os_aio_handler (segment=<optimized out>, m1=0x3fffa536e380, m2=0x3fffa536e388, request=0x3fffa536e390) at ./storage/innobase/os/os0file.cc:5781
#6  0x0000000051ae5030 in fil_aio_wait (segment=<optimized out>) at ./storage/innobase/fil/fil0fil.cc:5098
#7  0x00000000519ca038 in io_handler_thread (arg=<optimized out>) at ./storage/innobase/srv/srv0start.cc:332
#8  0x00003fffb20b80fc in start_thread (arg=0x3fffa536f000) at pthread_create.c:458
#9  0x00003fffb1821748 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:85
 
Thread 32 (Thread 0x3fffa20ff000 (LWP 24125)):
#0  0x00003fffb20c1264 in __pthread_cond_timedwait (cond=0x1002a644d48, mutex=0x1002a644d20, abstime=0x3fffa20fe450) at pthread_cond_timedwait.c:200
#1  0x0000000051925f98 in os_event::timed_wait (this=<error reading variable: value has been optimized out>, abstime=<optimized out>) at ./storage/innobase/os/os0event.cc:275
#2  0x00000000519262a8 in os_event::wait_time_low (this=0x1002a644d10, time_in_usec=<optimized out>, reset_sig_count=<optimized out>) at ./storage/innobase/os/os0event.cc:385
#3  0x00000000519c3490 in srv_monitor_thread () at ./storage/innobase/srv/srv0srv.cc:1751
#4  0x00003fffb20b80fc in start_thread (arg=0x3fffa20ff000) at pthread_create.c:458
#5  0x00003fffb1821748 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:85
 
Thread 31 (Thread 0x3fffa08cf000 (LWP 24129)):
#0  0x00003fffb20c0ca4 in __pthread_cond_wait (cond=0x1002a644b18, mutex=0x1002a644af0) at pthread_cond_wait.c:188
#1  0x0000000051926954 in os_event::wait (this=<optimized out>) at ./storage/innobase/os/os0event.cc:158
#2  os_event::wait_low (reset_sig_count=<optimized out>, this=0x1002a644ae0) at ./storage/innobase/os/os0event.cc:325
#3  os_event_wait_low (event=0x1002a644ae0, reset_sig_count=<optimized out>) at ./storage/innobase/os/os0event.cc:507
#4  0x00000000519c85f0 in srv_resume_thread (timeout_usec=0, wait=true, sig_count=1, slot=0x523c6cc0 <srv_sys+192>) at ./storage/innobase/srv/srv0srv.cc:940
#5  srv_purge_coordinator_suspend (rseg_history_len=<optimized out>, slot=<optimized out>) at ./storage/innobase/srv/srv0srv.cc:2781
#6  srv_purge_coordinator_thread (arg=<optimized out>) at ./storage/innobase/srv/srv0srv.cc:2868
#7  0x00003fffb20b80fc in start_thread (arg=0x3fffa08cf000) at pthread_create.c:458
#8  0x00003fffb1821748 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:85
 
Thread 30 (Thread 0x3fffa638f000 (LWP 24117)):
#0  0x00003fffb20407f8 in ?? () from /lib/powerpc64le-linux-gnu/libaio.so.1
#1  0x00003fffb2040860 in io_getevents () from /lib/powerpc64le-linux-gnu/libaio.so.1
#2  0x000000005191aec0 in LinuxAIOHandler::collect (this=0x3fffa638e288) at ./storage/innobase/os/os0file.cc:1936
#3  0x0000000051923188 in LinuxAIOHandler::poll (this=0x3fffa638e288, m1=0x3fffa638e380, m2=0x3fffa638e388, request=0x3fffa638e390) at ./storage/innobase/os/os0file.cc:2097
#4  0x0000000051923a28 in os_aio_linux_handler (request=0x3fffa638e390, m2=0x3fffa638e388, m1=0x3fffa638e380, global_segment=<optimized out>) at ./storage/innobase/os/os0file.cc:2151
#5  os_aio_handler (segment=<optimized out>, m1=0x3fffa638e380, m2=0x3fffa638e388, request=0x3fffa638e390) at ./storage/innobase/os/os0file.cc:5781
#6  0x0000000051ae5030 in fil_aio_wait (segment=<optimized out>) at ./storage/innobase/fil/fil0fil.cc:5098
#7  0x00000000519ca038 in io_handler_thread (arg=<optimized out>) at ./storage/innobase/srv/srv0start.cc:332
#8  0x00003fffb20b80fc in start_thread (arg=0x3fffa638f000) at pthread_create.c:458
#9  0x00003fffb1821748 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:85
 
Thread 29 (Thread 0x3fff91fbf000 (LWP 24134)):
#0  0x00003fffb20c6ec8 in nanosleep () at ../sysdeps/unix/syscall-template.S:84
#1  0x0000000051926df4 in os_thread_sleep (tm=<optimized out>) at ./storage/innobase/os/os0thread.cc:231
#2  0x0000000051a5eb5c in btr_defragment_thread () at ./storage/innobase/btr/btr0defragment.cc:727
#3  0x00003fffb20b80fc in start_thread (arg=0x3fff91fbf000) at pthread_create.c:458
#4  0x00003fffb1821748 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:85
 
Thread 28 (Thread 0x3fffa434f000 (LWP 24121)):
#0  0x00003fffb20c1264 in __pthread_cond_timedwait (cond=0x1002a644e28, mutex=0x1002a644e00, abstime=0x3fffa434e100) at pthread_cond_timedwait.c:200
#1  0x0000000051925f98 in os_event::timed_wait (this=<error reading variable: value has been optimized out>, abstime=<optimized out>) at ./storage/innobase/os/os0event.cc:275
#2  0x00000000519262a8 in os_event::wait_time_low (this=0x1002a644df0, time_in_usec=<optimized out>, reset_sig_count=<optimized out>) at ./storage/innobase/os/os0event.cc:385
#3  0x0000000051a87464 in pc_sleep_if_needed (cur_time=1757659, sig_count=905, next_loop_time=1758659) at ./storage/innobase/buf/buf0flu.cc:2683
#4  buf_flush_page_cleaner_coordinator () at ./storage/innobase/buf/buf0flu.cc:3149
#5  0x00003fffb20b80fc in start_thread (arg=0x3fffa434f000) at pthread_create.c:458
#6  0x00003fffb1821748 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:85
 
Thread 27 (Thread 0x3fff917af000 (LWP 24135)):
#0  0x00003fffb20c0ca4 in __pthread_cond_wait (cond=0x1002a644e98, mutex=0x1002a644e70) at pthread_cond_wait.c:188
#1  0x0000000051926954 in os_event::wait (this=<optimized out>) at ./storage/innobase/os/os0event.cc:158
#2  os_event::wait_low (reset_sig_count=<optimized out>, this=0x1002a644e60) at ./storage/innobase/os/os0event.cc:325
#3  os_event_wait_low (event=0x1002a644e60, reset_sig_count=<optimized out>) at ./storage/innobase/os/os0event.cc:507
#4  0x0000000051a6f6d4 in buf_resize_thread () at ./storage/innobase/buf/buf0buf.cc:3127
#5  0x00003fffb20b80fc in start_thread (arg=0x3fff917af000) at pthread_create.c:458
#6  0x00003fffb1821748 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:85
 
Thread 26 (Thread 0x3fffa311f000 (LWP 24123)):
#0  0x00003fffb20c1264 in __pthread_cond_timedwait (cond=0x1002a75ada8, mutex=0x1002a75ad80, abstime=0x3fffa311e3b0) at pthread_cond_timedwait.c:200
#1  0x0000000051925f98 in os_event::timed_wait (this=<error reading variable: value has been optimized out>, abstime=<optimized out>) at ./storage/innobase/os/os0event.cc:275
#2  0x00000000519262a8 in os_event::wait_time_low (this=0x1002a75ad70, time_in_usec=<optimized out>, reset_sig_count=<optimized out>) at ./storage/innobase/os/os0event.cc:385
#3  0x00000000518f5308 in lock_wait_timeout_thread () at ./storage/innobase/lock/lock0wait.cc:525
#4  0x00003fffb20b80fc in start_thread (arg=0x3fffa311f000) at pthread_create.c:458
#5  0x00003fffb1821748 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:85
 
Thread 25 (Thread 0x3fffa7548000 (LWP 24141)):
#0  0x00003fffb20c1264 in __pthread_cond_timedwait (cond=0x52448fe0 <COND_thread_cache>, mutex=0x52446a70 <LOCK_thread_cache>, abstime=0x3fffa75473f8) at pthread_cond_timedwait.c:200
#1  0x00000000513a0c3c in inline_mysql_cond_timedwait (src_file=0x51db2a50 "/home/buildbot/buildbot/build/mariadb-10.2.34/sql/mysqld.cc", src_line=2974, abstime=0x3fffa75473f8, mutex=0x52446a70 <LOCK_thread_cache>, that=0x52448fe0 <COND_thread_cache>) at ./include/mysql/psi/mysql_thread.h:1182
#2  cache_thread (thd=0x3fff5c0009a8) at ./sql/mysqld.cc:2973
#3  one_thread_per_connection_end (thd=0x3fff5c0009a8, put_in_cache=<optimized out>) at ./sql/mysqld.cc:3062
#4  0x00000000515738d8 in do_handle_one_connection (connect=<optimized out>) at ./sql/sql_connect.cc:1355
#5  0x0000000051573bc4 in handle_one_connection (arg=0x1002ab42d88) at ./sql/sql_connect.cc:1241
#6  0x0000000051caa0a8 in pfs_spawn_thread (arg=0x1002aadeba8) at ./storage/perfschema/pfs.cc:1869
#7  0x00003fffb20b80fc in start_thread (arg=0x3fffa7548000) at pthread_create.c:458
#8  0x00003fffb1821748 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:85
 
Thread 24 (Thread 0x3fffa392f000 (LWP 24126)):
#0  0x00003fffb20c6ec8 in nanosleep () at ../sysdeps/unix/syscall-template.S:84
#1  0x0000000051926df4 in os_thread_sleep (tm=<optimized out>) at ./storage/innobase/os/os0thread.cc:231
#2  0x00000000519c5488 in srv_master_sleep () at ./storage/innobase/srv/srv0srv.cc:2422
#3  srv_master_thread (arg=<optimized out>) at ./storage/innobase/srv/srv0srv.cc:2462
#4  0x00003fffb20b80fc in start_thread (arg=0x3fffa392f000) at pthread_create.c:458
#5  0x00003fffb1821748 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:85
 
Thread 23 (Thread 0x3fff906a8000 (LWP 24237)):
#0  0x00003fffb20c1264 in __pthread_cond_timedwait (cond=0x52448fe0 <COND_thread_cache>, mutex=0x52446a70 <LOCK_thread_cache>, abstime=0x3fff906a73f8) at pthread_cond_timedwait.c:200
#1  0x00000000513a0c3c in inline_mysql_cond_timedwait (src_file=0x51db2a50 "/home/buildbot/buildbot/build/mariadb-10.2.34/sql/mysqld.cc", src_line=2974, abstime=0x3fff906a73f8, mutex=0x52446a70 <LOCK_thread_cache>, that=0x52448fe0 <COND_thread_cache>) at ./include/mysql/psi/mysql_thread.h:1182
#2  cache_thread (thd=0x3fff380009a8) at ./sql/mysqld.cc:2973
#3  one_thread_per_connection_end (thd=0x3fff380009a8, put_in_cache=<optimized out>) at ./sql/mysqld.cc:3062
#4  0x00000000515738d8 in do_handle_one_connection (connect=<optimized out>) at ./sql/sql_connect.cc:1355
#5  0x0000000051573bc4 in handle_one_connection (arg=0x1002ab40458) at ./sql/sql_connect.cc:1241
#6  0x0000000051caa0a8 in pfs_spawn_thread (arg=0x1002ab100d8) at ./storage/perfschema/pfs.cc:1869
#7  0x00003fffb20b80fc in start_thread (arg=0x3fff906a8000) at pthread_create.c:458
#8  0x00003fffb1821748 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:85
 
Thread 22 (Thread 0x3fffa10df000 (LWP 24128)):
#0  0x00003fffb20c1264 in __pthread_cond_timedwait (cond=0x1002a7ae5e8, mutex=0x1002a7ae5c0, abstime=0x3fffa10de1a0) at pthread_cond_timedwait.c:200
#1  0x0000000051925f98 in os_event::timed_wait (this=<error reading variable: value has been optimized out>, abstime=<optimized out>) at ./storage/innobase/os/os0event.cc:275
#2  0x00000000519262a8 in os_event::wait_time_low (this=0x1002a7ae5b0, time_in_usec=<optimized out>, reset_sig_count=<optimized out>) at ./storage/innobase/os/os0event.cc:385
#3  0x0000000051a25664 in ib_wqueue_timedwait (wq=0x1002a7a4380, wait_in_usecs=<optimized out>) at ./storage/innobase/ut/ut0wqueue.cc:156
#4  0x0000000051b2fa64 in fts_optimize_thread (arg=0x1002a7a4380) at ./storage/innobase/fts/fts0opt.cc:2820
#5  0x00003fffb20b80fc in start_thread (arg=0x3fffa10df000) at pthread_create.c:458
#6  0x00003fffb1821748 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:85
 
Thread 21 (Thread 0x3fff90638000 (LWP 24240)):
#0  0x00003fffb20c57c8 in __lll_lock_wait (futex=0x200, private=<optimized out>) at lowlevellock.c:46
#1  0x00003fffb20ca734 in __lll_lock_elision (lock=<optimized out>, adapt_count=<optimized out>, pshared=<optimized out>) at ../sysdeps/unix/sysv/linux/powerpc/elision-lock.c:85
#2  0x00003fffb20bb978 in __GI___pthread_mutex_lock (mutex=0x1002ab4da60) at ../nptl/pthread_mutex_lock.c:92
#3  0x00000000513c8690 in inline_mysql_mutex_lock (that=0x1002ab4da60, src_line=<optimized out>, src_file=0x51dbbeb0 "/home/buildbot/buildbot/build/mariadb-10.2.34/sql/slave.cc") at ./include/mysql/psi/mysql_thread.h:675
#4  0x000000005149aa3c in start_slave (thd=0x3fff3c0009a8, mi=0x1002ab4c1c0, net_report=<optimized out>) at ./sql/sql_repl.cc:3007
#5  0x0000000051475514 in mysql_execute_command (thd=0x3fff3c0009a8) at ./sql/sql_parse.cc:3838
#6  0x00000000514784ac in mysql_parse (thd=0x3fff3c0009a8, rawbuf=0x3fff3c00f0b0 "START SLAVE; STOP SLAVE; SHOW GLOBAL STATUS", length=<optimized out>, parser_state=0x3fff90636bd0, is_com_multi=<optimized out>, is_next_command=<optimized out>) at ./sql/sql_parse.cc:7733
#7  0x000000005147a304 in dispatch_command (command=<optimized out>, thd=0x3fff3c0009a8, packet=<optimized out>, packet_length=<optimized out>, is_com_multi=<optimized out>, is_next_command=<optimized out>) at ./sql/sql_parse.cc:1824
#8  0x000000005147bb64 in do_command (thd=0x3fff3c0009a8) at ./sql/sql_parse.cc:1377
#9  0x00000000515739e4 in do_handle_one_connection (connect=<optimized out>) at ./sql/sql_connect.cc:1336
#10 0x0000000051573bc4 in handle_one_connection (arg=0x1002ab40458) at ./sql/sql_connect.cc:1241
#11 0x0000000051caa0a8 in pfs_spawn_thread (arg=0x1002ab10428) at ./storage/perfschema/pfs.cc:1869
#12 0x00003fffb20b80fc in start_thread (arg=0x3fff90638000) at pthread_create.c:458
#13 0x00003fffb1821748 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:85
 
Thread 20 (Thread 0x3fff93fff000 (LWP 24130)):
#0  0x00003fffb20c0ca4 in __pthread_cond_wait (cond=0x1002a644b88, mutex=0x1002a644b60) at pthread_cond_wait.c:188
#1  0x0000000051926954 in os_event::wait (this=<optimized out>) at ./storage/innobase/os/os0event.cc:158
#2  os_event::wait_low (reset_sig_count=<optimized out>, this=0x1002a644b50) at ./storage/innobase/os/os0event.cc:325
#3  os_event_wait_low (event=0x1002a644b50, reset_sig_count=<optimized out>) at ./storage/innobase/os/os0event.cc:507
#4  0x00000000519c6784 in srv_resume_thread (sig_count=0, timeout_usec=0, wait=true, slot=0x523c6cf8 <srv_sys+248>) at ./storage/innobase/srv/srv0srv.cc:940
#5  srv_worker_thread (arg=<optimized out>) at ./storage/innobase/srv/srv0srv.cc:2620
#6  0x00003fffb20b80fc in start_thread (arg=0x3fff93fff000) at pthread_create.c:458
#7  0x00003fffb1821748 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:85
 
Thread 19 (Thread 0x3fff937ef000 (LWP 24131)):
#0  0x00003fffb20c0ca4 in __pthread_cond_wait (cond=0x1002a644c68, mutex=0x1002a644c40) at pthread_cond_wait.c:188
#1  0x0000000051926954 in os_event::wait (this=<optimized out>) at ./storage/innobase/os/os0event.cc:158
#2  os_event::wait_low (reset_sig_count=<optimized out>, this=0x1002a644c30) at ./storage/innobase/os/os0event.cc:325
#3  os_event_wait_low (event=0x1002a644c30, reset_sig_count=<optimized out>) at ./storage/innobase/os/os0event.cc:507
#4  0x00000000519c6784 in srv_resume_thread (sig_count=0, timeout_usec=0, wait=true, slot=0x523c6d68 <srv_sys+360>) at ./storage/innobase/srv/srv0srv.cc:940
#5  srv_worker_thread (arg=<optimized out>) at ./storage/innobase/srv/srv0srv.cc:2620
#6  0x00003fffb20b80fc in start_thread (arg=0x3fff937ef000) at pthread_create.c:458
#7  0x00003fffb1821748 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:85
 
Thread 18 (Thread 0x3fff90558000 (LWP 24340)):
#0  0x00003fffb20c57c8 in __lll_lock_wait (futex=0x200, private=<optimized out>) at lowlevellock.c:46
#1  0x00003fffb20ca734 in __lll_lock_elision (lock=<optimized out>, adapt_count=<optimized out>, pshared=<optimized out>) at ../sysdeps/unix/sysv/linux/powerpc/elision-lock.c:85
#2  0x00003fffb20bb978 in __GI___pthread_mutex_lock (mutex=0x1002ab4dfe8) at ../nptl/pthread_mutex_lock.c:92
#3  0x00000000517a1d40 in inline_mysql_mutex_lock (that=0x1002ab4dfe8, src_line=5082, src_file=0x51e5edb0 "/home/buildbot/buildbot/build/mariadb-10.2.34/sql/log.cc") at ./include/mysql/psi/mysql_thread.h:675
#4  0x00000000517b60dc in MYSQL_BIN_LOG::new_file_impl (this=0x1002ab4dfe0, need_lock=<optimized out>) at ./sql/log.cc:5082
#5  0x00000000517b6354 in MYSQL_BIN_LOG::new_file (this=<optimized out>) at ./sql/log.cc:5047
#6  0x00000000513d4634 in rotate_relay_log (mi=0x1002ab4c1c0) at ./sql/slave.cc:7499
#7  0x00000000513d47a4 in process_io_rotate (mi=0x1002ab4c1c0, rev=0x3fff90555a58) at ./sql/slave.cc:5576
#8  0x00000000513d565c in queue_event (mi=0x1002ab4c1c0, buf=0x3fff4404c259 "", event_len=<optimized out>) at ./sql/slave.cc:6004
#9  0x00000000513d7384 in handle_slave_io (arg=0x1002ab4c1c0) at ./sql/slave.cc:4580
#10 0x0000000051caa0a8 in pfs_spawn_thread (arg=0x3fff30006258) at ./storage/perfschema/pfs.cc:1869
#11 0x00003fffb20b80fc in start_thread (arg=0x3fff90558000) at pthread_create.c:458
#12 0x00003fffb1821748 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:85
 
Thread 17 (Thread 0x3fff905c8000 (LWP 24241)):
#0  0x00003fffb20c1264 in __pthread_cond_timedwait (cond=0x1002ab4f5b8, mutex=0x1002ab4f550, abstime=0x3fff905c5278) at pthread_cond_timedwait.c:200
#1  0x00000000513ca75c in inline_mysql_cond_timedwait (src_file=0x51dbbeb0 "/home/buildbot/buildbot/build/mariadb-10.2.34/sql/slave.cc", src_line=882, abstime=0x3fff905c5278, mutex=<optimized out>, that=0x1002ab4f5b8) at ./include/mysql/psi/mysql_thread.h:1182
#2  terminate_slave_thread (thd=0x3fff400008c8, term_lock=0x1002ab4f550, term_cond=0x1002ab4f5b8, slave_running=0x1002ab4ff28, skip_lock=false) at ./sql/slave.cc:882
#3  0x00000000513cc860 in terminate_slave_thread (skip_lock=false, slave_running=0x1002ab4ff28, term_cond=0x1002ab4f5b8, term_lock=0x1002ab4f550, thd=0x3fff400008c8) at ./sql/slave.cc:833
#4  terminate_slave_threads (mi=0x1002ab4c1c0, thread_mask=<optimized out>, skip_lock=<optimized out>) at ./sql/slave.cc:732
#5  0x000000005149b050 in stop_slave (thd=0x3fff300009a8, mi=0x1002ab4c1c0, net_report=<optimized out>) at ./sql/sql_repl.cc:3220
#6  0x0000000051471a48 in mysql_execute_command (thd=0x3fff300009a8) at ./sql/sql_parse.cc:3874
#7  0x00000000514784ac in mysql_parse (thd=0x3fff300009a8, rawbuf=0x3fff3000f0bd "STOP SLAVE; SHOW GLOBAL STATUS", length=<optimized out>, parser_state=0x3fff905c6bd0, is_com_multi=<optimized out>, is_next_command=<optimized out>) at ./sql/sql_parse.cc:7733
#8  0x000000005147a660 in dispatch_command (command=<optimized out>, thd=<optimized out>, packet=<optimized out>, packet_length=<optimized out>, is_com_multi=<optimized out>, is_next_command=<optimized out>) at ./sql/sql_parse.cc:1913
#9  0x000000005147bb64 in do_command (thd=0x3fff300009a8) at ./sql/sql_parse.cc:1377
#10 0x00000000515739e4 in do_handle_one_connection (connect=<optimized out>) at ./sql/sql_connect.cc:1336
#11 0x0000000051573bc4 in handle_one_connection (arg=0x1002ab401e8) at ./sql/sql_connect.cc:1241
#12 0x0000000051caa0a8 in pfs_spawn_thread (arg=0x1002ab10978) at ./storage/perfschema/pfs.cc:1869
#13 0x00003fffb20b80fc in start_thread (arg=0x3fff905c8000) at pthread_create.c:458
#14 0x00003fffb1821748 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:85
 
Thread 16 (Thread 0x3fffa00b8000 (LWP 24144)):
#0  0x00003fffb20c1264 in __pthread_cond_timedwait (cond=0x52448fe0 <COND_thread_cache>, mutex=0x52446a70 <LOCK_thread_cache>, abstime=0x3fffa00b73f8) at pthread_cond_timedwait.c:200
#1  0x00000000513a0c3c in inline_mysql_cond_timedwait (src_file=0x51db2a50 "/home/buildbot/buildbot/build/mariadb-10.2.34/sql/mysqld.cc", src_line=2974, abstime=0x3fffa00b73f8, mutex=0x52446a70 <LOCK_thread_cache>, that=0x52448fe0 <COND_thread_cache>) at ./include/mysql/psi/mysql_thread.h:1182
#2  cache_thread (thd=0x3fff480009a8) at ./sql/mysqld.cc:2973
#3  one_thread_per_connection_end (thd=0x3fff480009a8, put_in_cache=<optimized out>) at ./sql/mysqld.cc:3062
#4  0x00000000515738d8 in do_handle_one_connection (connect=<optimized out>) at ./sql/sql_connect.cc:1355
#5  0x0000000051573bc4 in handle_one_connection (arg=0x1002ab42d88) at ./sql/sql_connect.cc:1241
#6  0x0000000051caa0a8 in pfs_spawn_thread (arg=0x1002ab5ab78) at ./storage/perfschema/pfs.cc:1869
#7  0x00003fffb20b80fc in start_thread (arg=0x3fffa00b8000) at pthread_create.c:458
#8  0x00003fffb1821748 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:85
 
Thread 15 (Thread 0x3fffa74d8000 (LWP 24142)):
#0  0x00003fffb20c1264 in __pthread_cond_timedwait (cond=0x52448fe0 <COND_thread_cache>, mutex=0x52446a70 <LOCK_thread_cache>, abstime=0x3fffa74d73f8) at pthread_cond_timedwait.c:200
#1  0x00000000513a0c3c in inline_mysql_cond_timedwait (src_file=0x51db2a50 "/home/buildbot/buildbot/build/mariadb-10.2.34/sql/mysqld.cc", src_line=2974, abstime=0x3fffa74d73f8, mutex=0x52446a70 <LOCK_thread_cache>, that=0x52448fe0 <COND_thread_cache>) at ./include/mysql/psi/mysql_thread.h:1182
#2  cache_thread (thd=0x3fff500009d8) at ./sql/mysqld.cc:2973
#3  one_thread_per_connection_end (thd=0x3fff500009d8, put_in_cache=<optimized out>) at ./sql/mysqld.cc:3062
#4  0x00000000515738d8 in do_handle_one_connection (connect=<optimized out>) at ./sql/sql_connect.cc:1355
#5  0x0000000051573bc4 in handle_one_connection (arg=0x1002ab42d88) at ./sql/sql_connect.cc:1241
#6  0x0000000051caa0a8 in pfs_spawn_thread (arg=0x1002aadeef8) at ./storage/perfschema/pfs.cc:1869
#7  0x00003fffb20b80fc in start_thread (arg=0x3fffa74d8000) at pthread_create.c:458
#8  0x00003fffb1821748 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:85
 
Thread 14 (Thread 0x3fffa7468000 (LWP 24143)):
#0  0x00003fffb20c1264 in __pthread_cond_timedwait (cond=0x52448fe0 <COND_thread_cache>, mutex=0x52446a70 <LOCK_thread_cache>, abstime=0x3fffa74673f8) at pthread_cond_timedwait.c:200
#1  0x00000000513a0c3c in inline_mysql_cond_timedwait (src_file=0x51db2a50 "/home/buildbot/buildbot/build/mariadb-10.2.34/sql/mysqld.cc", src_line=2974, abstime=0x3fffa74673f8, mutex=0x52446a70 <LOCK_thread_cache>, that=0x52448fe0 <COND_thread_cache>) at ./include/mysql/psi/mysql_thread.h:1182
#2  cache_thread (thd=0x3fff540009a8) at ./sql/mysqld.cc:2973
#3  one_thread_per_connection_end (thd=0x3fff540009a8, put_in_cache=<optimized out>) at ./sql/mysqld.cc:3062
#4  0x00000000515738d8 in do_handle_one_connection (connect=<optimized out>) at ./sql/sql_connect.cc:1355
#5  0x0000000051573bc4 in handle_one_connection (arg=0x1002ab42d88) at ./sql/sql_connect.cc:1241
#6  0x0000000051caa0a8 in pfs_spawn_thread (arg=0x1002aadf248) at ./storage/perfschema/pfs.cc:1869
#7  0x00003fffb20b80fc in start_thread (arg=0x3fffa7468000) at pthread_create.c:458
#8  0x00003fffb1821748 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:85
 
Thread 13 (Thread 0x3fff6f84f000 (LWP 24137)):
#0  0x00003fffb20c1264 in __pthread_cond_timedwait (cond=0x52ceb4a0 <COND_checkpoint>, mutex=0x52ceb468 <LOCK_checkpoint>, abstime=0x3fff6f84e3a8) at pthread_cond_timedwait.c:200
#1  0x0000000051bb0f1c in inline_mysql_cond_timedwait (src_file=0x51eb8140 "/home/buildbot/buildbot/build/mariadb-10.2.34/storage/maria/ma_servicethread.c", src_line=116, abstime=<optimized out>, mutex=0x52ceb468 <LOCK_checkpoint>, that=0x52ceb4a0 <COND_checkpoint>) at ./include/mysql/psi/mysql_thread.h:1182
#2  my_service_thread_sleep (control=0x523c88c8 <checkpoint_control>, sleep_time=<optimized out>) at ./storage/maria/ma_servicethread.c:115
#3  0x0000000051ba5498 in ma_checkpoint_background (arg=0x1e) at ./storage/maria/ma_checkpoint.c:709
#4  0x0000000051caa0a8 in pfs_spawn_thread (arg=0x1002a7b20d8) at ./storage/perfschema/pfs.cc:1869
#5  0x00003fffb20b80fc in start_thread (arg=0x3fff6f84f000) at pthread_create.c:458
#6  0x00003fffb1821748 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:85
 
Thread 12 (Thread 0x3fff90f9f000 (LWP 24136)):
#0  0x00003fffb20c0ca4 in __pthread_cond_wait (cond=0x3fff90f9e4e0, mutex=0x3fff90f9e4b0) at pthread_cond_wait.c:188
#1  0x00000000518920ec in inline_mysql_cond_wait (src_file=0x51e66c40 "/home/buildbot/buildbot/build/mariadb-10.2.34/storage/innobase/handler/ha_innodb.cc", src_line=329, mutex=0x3fff90f9e4b0, that=0x3fff90f9e4e0) at ./include/mysql/psi/mysql_thread.h:1156
#2  thd_destructor_proxy () at ./storage/innobase/handler/ha_innodb.cc:329
#3  0x0000000051caa0a8 in pfs_spawn_thread (arg=0x1002a7af6d8) at ./storage/perfschema/pfs.cc:1869
#4  0x00003fffb20b80fc in start_thread (arg=0x3fff90f9f000) at pthread_create.c:458
#5  0x00003fffb1821748 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:85
 
Thread 11 (Thread 0x3fffac0f8000 (LWP 24138)):
#0  0x00003fffb20c0ca4 in __pthread_cond_wait (cond=0x5246f0d8 <mysql_bin_log+3144>, mutex=0x5246f0a8 <mysql_bin_log+3096>) at pthread_cond_wait.c:188
#1  0x00000000517b3514 in inline_mysql_cond_wait (that=0x5246f0d8 <mysql_bin_log+3144>, mutex=0x5246f0a8 <mysql_bin_log+3096>, src_file=0x51e5edb0 "/home/buildbot/buildbot/build/mariadb-10.2.34/sql/log.cc", src_line=9905) at ./include/mysql/psi/mysql_thread.h:1145
#2  binlog_background_thread (arg=<optimized out>) at ./sql/log.cc:9905
#3  0x0000000051caa0a8 in pfs_spawn_thread (arg=0x1002a7b4778) at ./storage/perfschema/pfs.cc:1869
#4  0x00003fffb20b80fc in start_thread (arg=0x3fffac0f8000) at pthread_create.c:458
#5  0x00003fffb1821748 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:85
 
Thread 10 (Thread 0x3fff927cf000 (LWP 24133)):
#0  0x00003fffb20c0ca4 in __pthread_cond_wait (cond=0x1002a644db8, mutex=0x1002a644d90) at pthread_cond_wait.c:188
#1  0x0000000051926954 in os_event::wait (this=<optimized out>) at ./storage/innobase/os/os0event.cc:158
#2  os_event::wait_low (reset_sig_count=<optimized out>, this=0x1002a644d80) at ./storage/innobase/os/os0event.cc:325
#3  os_event_wait_low (event=0x1002a644d80, reset_sig_count=<optimized out>) at ./storage/innobase/os/os0event.cc:507
#4  0x0000000051a78db0 in buf_dump_thread () at ./storage/innobase/buf/buf0dump.cc:806
#5  0x00003fffb20b80fc in start_thread (arg=0x3fff927cf000) at pthread_create.c:458
#6  0x00003fffb1821748 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:85
 
Thread 9 (Thread 0x3fffac088000 (LWP 24139)):
#0  0x00003fffb20c7ac8 in do_sigwait (set=<optimized out>, set@entry=0x3fffac087470, sig=sig@entry=0x3fffac087420) at ../sysdeps/unix/sysv/linux/sigwait.c:61
#1  0x00003fffb20c7b88 in __sigwait (set=0x3fffac087470, sig=0x3fffac087420) at ../sysdeps/unix/sysv/linux/sigwait.c:96
#2  0x000000005139e48c in signal_hand (arg=<optimized out>) at ./sql/mysqld.cc:3510
#3  0x0000000051caa0a8 in pfs_spawn_thread (arg=0x1002aad8228) at ./storage/perfschema/pfs.cc:1869
#4  0x00003fffb20b80fc in start_thread (arg=0x3fffac088000) at pthread_create.c:458
#5  0x00003fffb1821748 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:85
 
Thread 8 (Thread 0x3fffa18ef000 (LWP 24127)):
#0  0x00003fffb20c1264 in __pthread_cond_timedwait (cond=0x1002a76c5e8, mutex=0x1002a76c5c0, abstime=0x3fffa18ee340) at pthread_cond_timedwait.c:200
#1  0x0000000051925f98 in os_event::timed_wait (this=<error reading variable: value has been optimized out>, abstime=<optimized out>) at ./storage/innobase/os/os0event.cc:275
#2  0x00000000519262a8 in os_event::wait_time_low (this=0x1002a76c5b0, time_in_usec=<optimized out>, reset_sig_count=<optimized out>) at ./storage/innobase/os/os0event.cc:385
#3  0x0000000051ad8624 in dict_stats_thread () at ./storage/innobase/dict/dict0stats_bg.cc:475
#4  0x00003fffb20b80fc in start_thread (arg=0x3fffa18ef000) at pthread_create.c:458
#5  0x00003fffb1821748 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:85
 
Thread 7 (Thread 0x3fffa75b8000 (LWP 24140)):
#0  0x00003fffb20c0ca4 in __pthread_cond_wait (cond=0x52448fa8 <COND_slave_background>, mutex=0x524467d0 <LOCK_slave_background>) at pthread_cond_wait.c:188
#1  0x00000000513cab54 in inline_mysql_cond_wait (that=<optimized out>, mutex=<optimized out>, src_file=0x51dbbeb0 "/home/buildbot/buildbot/build/mariadb-10.2.34/sql/slave.cc", src_line=337) at ./include/mysql/psi/mysql_thread.h:1145
#2  handle_slave_background (arg=<optimized out>) at ./sql/slave.cc:337
#3  0x0000000051caa0a8 in pfs_spawn_thread (arg=0x1002ab42b48) at ./storage/perfschema/pfs.cc:1869
#4  0x00003fffb20b80fc in start_thread (arg=0x3fffa75b8000) at pthread_create.c:458
#5  0x00003fffb1821748 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:85
 
Thread 6 (Thread 0x3fffa290f000 (LWP 24124)):
#0  0x00003fffb20c1264 in __pthread_cond_timedwait (cond=0x1002a644cd8, mutex=0x1002a644cb0, abstime=0x3fffa290e2e0) at pthread_cond_timedwait.c:200
#1  0x0000000051925f98 in os_event::timed_wait (this=<error reading variable: value has been optimized out>, abstime=<optimized out>) at ./storage/innobase/os/os0event.cc:275
#2  0x00000000519262a8 in os_event::wait_time_low (this=0x1002a644ca0, time_in_usec=<optimized out>, reset_sig_count=<optimized out>) at ./storage/innobase/os/os0event.cc:385
#3  0x00000000519c3e2c in srv_error_monitor_thread () at ./storage/innobase/srv/srv0srv.cc:1915
#4  0x00003fffb20b80fc in start_thread (arg=0x3fffa290f000) at pthread_create.c:458
#5  0x00003fffb1821748 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:85
 
Thread 5 (Thread 0x3fff92fdf000 (LWP 24132)):
#0  0x00003fffb20c0ca4 in __pthread_cond_wait (cond=0x1002a644bf8, mutex=0x1002a644bd0) at pthread_cond_wait.c:188
#1  0x0000000051926954 in os_event::wait (this=<optimized out>) at ./storage/innobase/os/os0event.cc:158
#2  os_event::wait_low (reset_sig_count=<optimized out>, this=0x1002a644bc0) at ./storage/innobase/os/os0event.cc:325
#3  os_event_wait_low (event=0x1002a644bc0, reset_sig_count=<optimized out>) at ./storage/innobase/os/os0event.cc:507
#4  0x00000000519c6784 in srv_resume_thread (sig_count=0, timeout_usec=0, wait=true, slot=0x523c6d30 <srv_sys+304>) at ./storage/innobase/srv/srv0srv.cc:940
#5  srv_worker_thread (arg=<optimized out>) at ./storage/innobase/srv/srv0srv.cc:2620
#6  0x00003fffb20b80fc in start_thread (arg=0x3fff92fdf000) at pthread_create.c:458
#7  0x00003fffb1821748 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:85
 
Thread 4 (Thread 0x3fffa5b7f000 (LWP 24118)):
#0  0x00003fffb20407f8 in ?? () from /lib/powerpc64le-linux-gnu/libaio.so.1
#1  0x00003fffb2040860 in io_getevents () from /lib/powerpc64le-linux-gnu/libaio.so.1
#2  0x000000005191aec0 in LinuxAIOHandler::collect (this=0x3fffa5b7e288) at ./storage/innobase/os/os0file.cc:1936
#3  0x0000000051923188 in LinuxAIOHandler::poll (this=0x3fffa5b7e288, m1=0x3fffa5b7e380, m2=0x3fffa5b7e388, request=0x3fffa5b7e390) at ./storage/innobase/os/os0file.cc:2097
#4  0x0000000051923a28 in os_aio_linux_handler (request=0x3fffa5b7e390, m2=0x3fffa5b7e388, m1=0x3fffa5b7e380, global_segment=<optimized out>) at ./storage/innobase/os/os0file.cc:2151
#5  os_aio_handler (segment=<optimized out>, m1=0x3fffa5b7e380, m2=0x3fffa5b7e388, request=0x3fffa5b7e390) at ./storage/innobase/os/os0file.cc:5781
#6  0x0000000051ae5030 in fil_aio_wait (segment=<optimized out>) at ./storage/innobase/fil/fil0fil.cc:5098
#7  0x00000000519ca038 in io_handler_thread (arg=<optimized out>) at ./storage/innobase/srv/srv0start.cc:332
#8  0x00003fffb20b80fc in start_thread (arg=0x3fffa5b7f000) at pthread_create.c:458
#9  0x00003fffb1821748 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:85
 
Thread 3 (Thread 0x3fffa4b5f000 (LWP 24120)):
#0  0x00003fffb20407f8 in ?? () from /lib/powerpc64le-linux-gnu/libaio.so.1
#1  0x00003fffb2040860 in io_getevents () from /lib/powerpc64le-linux-gnu/libaio.so.1
#2  0x000000005191aec0 in LinuxAIOHandler::collect (this=0x3fffa4b5e288) at ./storage/innobase/os/os0file.cc:1936
#3  0x0000000051923188 in LinuxAIOHandler::poll (this=0x3fffa4b5e288, m1=0x3fffa4b5e380, m2=0x3fffa4b5e388, request=0x3fffa4b5e390) at ./storage/innobase/os/os0file.cc:2097
#4  0x0000000051923a28 in os_aio_linux_handler (request=0x3fffa4b5e390, m2=0x3fffa4b5e388, m1=0x3fffa4b5e380, global_segment=<optimized out>) at ./storage/innobase/os/os0file.cc:2151
#5  os_aio_handler (segment=<optimized out>, m1=0x3fffa4b5e380, m2=0x3fffa4b5e388, request=0x3fffa4b5e390) at ./storage/innobase/os/os0file.cc:5781
#6  0x0000000051ae5030 in fil_aio_wait (segment=<optimized out>) at ./storage/innobase/fil/fil0fil.cc:5098
#7  0x00000000519ca038 in io_handler_thread (arg=<optimized out>) at ./storage/innobase/srv/srv0start.cc:332
#8  0x00003fffb20b80fc in start_thread (arg=0x3fffa4b5f000) at pthread_create.c:458
#9  0x00003fffb1821748 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:85
 
Thread 2 (Thread 0x3fffa73af000 (LWP 24115)):
#0  0x00003fffb20407f8 in ?? () from /lib/powerpc64le-linux-gnu/libaio.so.1
#1  0x00003fffb2040860 in io_getevents () from /lib/powerpc64le-linux-gnu/libaio.so.1
#2  0x000000005191aec0 in LinuxAIOHandler::collect (this=0x3fffa73ae288) at ./storage/innobase/os/os0file.cc:1936
#3  0x0000000051923188 in LinuxAIOHandler::poll (this=0x3fffa73ae288, m1=0x3fffa73ae380, m2=0x3fffa73ae388, request=0x3fffa73ae390) at ./storage/innobase/os/os0file.cc:2097
#4  0x0000000051923a28 in os_aio_linux_handler (request=0x3fffa73ae390, m2=0x3fffa73ae388, m1=0x3fffa73ae380, global_segment=<optimized out>) at ./storage/innobase/os/os0file.cc:2151
#5  os_aio_handler (segment=<optimized out>, m1=0x3fffa73ae380, m2=0x3fffa73ae388, request=0x3fffa73ae390) at ./storage/innobase/os/os0file.cc:5781
#6  0x0000000051ae5030 in fil_aio_wait (segment=<optimized out>) at ./storage/innobase/fil/fil0fil.cc:5098
#7  0x00000000519ca038 in io_handler_thread (arg=<optimized out>) at ./storage/innobase/srv/srv0start.cc:332
#8  0x00003fffb20b80fc in start_thread (arg=0x3fffa73af000) at pthread_create.c:458
#9  0x00003fffb1821748 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:85
 
Thread 1 (Thread 0x3fffb2169ec0 (LWP 24113)):
#0  0x00003fffb20c2898 in __pthread_kill (threadid=<optimized out>, signo=<optimized out>) at ../sysdeps/unix/sysv/linux/pthread_kill.c:55
#1  0x0000000051d116c0 in my_write_core (sig=<optimized out>) at ./mysys/stacktrace.c:382
#2  0x0000000051684c08 in handle_fatal_signal (sig=<optimized out>) at ./sql/signal_handler.cc:343
#3  <signal handler called>
#4  0x00003fffb1814238 in poll () at ../sysdeps/unix/syscall-template.S:84
#5  0x00003fffb183a634 in __poll_chk (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>, fdslen=<optimized out>) at poll_chk.c:27
#6  0x00000000513aada8 in poll (__timeout=-1, __nfds=2, __fds=0x3fffd0d96fd0) at /usr/include/powerpc64le-linux-gnu/bits/poll2.h:41
#7  handle_connections_sockets () at ./sql/mysqld.cc:6669
#8  0x00000000513acd44 in mysqld_main (argc=<optimized out>, argv=<optimized out>) at ./sql/mysqld.cc:6137
#9  0x0000000051385710 in main (argc=<optimized out>, argv=<optimized out>) at ./sql/main.cc:25
 
 - deleting it, already saved 0
 - found 'core' (0/0)
 
Trying 'dbx' to get a backtrace
 
Trying 'gdb' to get a backtrace from coredump /dev/shm/var/3/log/rpl.show_status_stop_slave_race-7126-row/mysqld.1/data/core
Core generated by '/usr/sbin/mysqld'
Output from gdb follows. The first stack trace is from the failing thread.
The following stack traces are from all threads (so the failing one is
duplicated).
--------------------------
[New LWP 21379]
[New LWP 21384]
[New LWP 21387]
[New LWP 21388]
[New LWP 21454]
[New LWP 21452]
[New LWP 21708]
[New LWP 21382]
[New LWP 23320]
[New LWP 21453]
[New LWP 23319]
[New LWP 21385]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/powerpc64le-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/mysqld --defaults-group-suffix=.1 --defaults-file=/dev/shm/var/3/my.c'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00003fffaacd2898 in __pthread_kill (threadid=<optimized out>, signo=<optimized out>) at ../sysdeps/unix/sysv/linux/pthread_kill.c:55
55	../sysdeps/unix/sysv/linux/pthread_kill.c: No such file or directory.
[Current thread is 1 (Thread 0x3fffaad79ec0 (LWP 21379))]
#0  0x00003fffaacd2898 in __pthread_kill (threadid=<optimized out>, signo=<optimized out>) at ../sysdeps/unix/sysv/linux/pthread_kill.c:55
#1  0x00000000366216c0 in my_write_core (sig=<optimized out>) at ./mysys/stacktrace.c:382
#2  0x0000000035f94c08 in handle_fatal_signal (sig=<optimized out>) at ./sql/signal_handler.cc:343
#3  <signal handler called>
#4  0x00003fffaa424238 in poll () at ../sysdeps/unix/syscall-template.S:84
#5  0x00003fffaa44a634 in __poll_chk (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>, fdslen=<optimized out>) at poll_chk.c:27
#6  0x0000000035cbada8 in poll (__timeout=-1, __nfds=2, __fds=0x3fffea4bf000) at /usr/include/powerpc64le-linux-gnu/bits/poll2.h:41
#7  handle_connections_sockets () at ./sql/mysqld.cc:6669
#8  0x0000000035cbcd44 in mysqld_main (argc=<optimized out>, argv=<optimized out>) at ./sql/mysqld.cc:6137
#9  0x0000000035c95710 in main (argc=<optimized out>, argv=<optimized out>) at ./sql/main.cc:25
 
Thread 12 (Thread 0x3fff9fff8000 (LWP 21385)):
#0  0x00003fffaacd0ca4 in __pthread_cond_wait (cond=0x36d7f0d8 <mysql_bin_log+3144>, mutex=0x36d7f0a8 <mysql_bin_log+3096>) at pthread_cond_wait.c:188
#1  0x00000000360c3514 in inline_mysql_cond_wait (that=0x36d7f0d8 <mysql_bin_log+3144>, mutex=0x36d7f0a8 <mysql_bin_log+3096>, src_file=0x3676edb0 "/home/buildbot/buildbot/build/mariadb-10.2.34/sql/log.cc", src_line=9905) at ./include/mysql/psi/mysql_thread.h:1145
#2  binlog_background_thread (arg=<optimized out>) at ./sql/log.cc:9905
#3  0x00000000365ba0a8 in pfs_spawn_thread (arg=0x10000702fc8) at ./storage/perfschema/pfs.cc:1869
#4  0x00003fffaacc80fc in start_thread (arg=0x3fff9fff8000) at pthread_create.c:458
#5  0x00003fffaa431748 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:85
 
Thread 11 (Thread 0x3fff9fb98000 (LWP 23319)):
#0  0x00003fffaacd1264 in __pthread_cond_timedwait (cond=0x36d58fe0 <COND_thread_cache>, mutex=0x36d56a70 <LOCK_thread_cache>, abstime=0x3fff9fb973f8) at pthread_cond_timedwait.c:200
#1  0x0000000035cb0c3c in inline_mysql_cond_timedwait (src_file=0x366c2a50 "/home/buildbot/buildbot/build/mariadb-10.2.34/sql/mysqld.cc", src_line=2974, abstime=0x3fff9fb973f8, mutex=0x36d56a70 <LOCK_thread_cache>, that=0x36d58fe0 <COND_thread_cache>) at ./include/mysql/psi/mysql_thread.h:1182
#2  cache_thread (thd=0x3fff680009a8) at ./sql/mysqld.cc:2973
#3  one_thread_per_connection_end (thd=0x3fff680009a8, put_in_cache=<optimized out>) at ./sql/mysqld.cc:3062
#4  0x0000000035e838d8 in do_handle_one_connection (connect=<optimized out>) at ./sql/sql_connect.cc:1355
#5  0x0000000035e83bc4 in handle_one_connection (arg=0x10000798e68) at ./sql/sql_connect.cc:1241
#6  0x00000000365ba0a8 in pfs_spawn_thread (arg=0x1000076a8e8) at ./storage/perfschema/pfs.cc:1869
#7  0x00003fffaacc80fc in start_thread (arg=0x3fff9fb98000) at pthread_create.c:458
#8  0x00003fffaa431748 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:85
 
Thread 10 (Thread 0x3fff9fce8000 (LWP 21453)):
#0  0x00003fffaacd1264 in __pthread_cond_timedwait (cond=0x36d58fe0 <COND_thread_cache>, mutex=0x36d56a70 <LOCK_thread_cache>, abstime=0x3fff9fce73f8) at pthread_cond_timedwait.c:200
#1  0x0000000035cb0c3c in inline_mysql_cond_timedwait (src_file=0x366c2a50 "/home/buildbot/buildbot/build/mariadb-10.2.34/sql/mysqld.cc", src_line=2974, abstime=0x3fff9fce73f8, mutex=0x36d56a70 <LOCK_thread_cache>, that=0x36d58fe0 <COND_thread_cache>) at ./include/mysql/psi/mysql_thread.h:1182
#2  cache_thread (thd=0x3fff7c0009a8) at ./sql/mysqld.cc:2973
#3  one_thread_per_connection_end (thd=0x3fff7c0009a8, put_in_cache=<optimized out>) at ./sql/mysqld.cc:3062
#4  0x0000000035e838d8 in do_handle_one_connection (connect=<optimized out>) at ./sql/sql_connect.cc:1355
#5  0x0000000035e83bc4 in handle_one_connection (arg=0x100007030f8) at ./sql/sql_connect.cc:1241
#6  0x00000000365ba0a8 in pfs_spawn_thread (arg=0x100007374e8) at ./storage/perfschema/pfs.cc:1869
#7  0x00003fffaacc80fc in start_thread (arg=0x3fff9fce8000) at pthread_create.c:458
#8  0x00003fffaa431748 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:85
 
Thread 9 (Thread 0x3fff9fb28000 (LWP 23320)):
#0  0x00003fffaacd1264 in __pthread_cond_timedwait (cond=0x36d58fe0 <COND_thread_cache>, mutex=0x36d56a70 <LOCK_thread_cache>, abstime=0x3fff9fb273f8) at pthread_cond_timedwait.c:200
#1  0x0000000035cb0c3c in inline_mysql_cond_timedwait (src_file=0x366c2a50 "/home/buildbot/buildbot/build/mariadb-10.2.34/sql/mysqld.cc", src_line=2974, abstime=0x3fff9fb273f8, mutex=0x36d56a70 <LOCK_thread_cache>, that=0x36d58fe0 <COND_thread_cache>) at ./include/mysql/psi/mysql_thread.h:1182
#2  cache_thread (thd=0x3fff6c0009a8) at ./sql/mysqld.cc:2973
#3  one_thread_per_connection_end (thd=0x3fff6c0009a8, put_in_cache=<optimized out>) at ./sql/mysqld.cc:3062
#4  0x0000000035e838d8 in do_handle_one_connection (connect=<optimized out>) at ./sql/sql_connect.cc:1355
#5  0x0000000035e83bc4 in handle_one_connection (arg=0x10000798e68) at ./sql/sql_connect.cc:1241
#6  0x00000000365ba0a8 in pfs_spawn_thread (arg=0x100007735b8) at ./storage/perfschema/pfs.cc:1869
#7  0x00003fffaacc80fc in start_thread (arg=0x3fff9fb28000) at pthread_create.c:458
#8  0x00003fffaa431748 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:85
 
Thread 8 (Thread 0x3fffa5c0f000 (LWP 21382)):
#0  0x00003fffaacd1264 in __pthread_cond_timedwait (cond=0x3760b178 <COND_timer>, mutex=0x3760b110 <LOCK_timer>, abstime=0x3fffa5c0e4b8) at pthread_cond_timedwait.c:200
#1  0x0000000036627f28 in inline_mysql_cond_timedwait (that=0x3760b178 <COND_timer>, mutex=0x3760b110 <LOCK_timer>, src_file=0x367dc1d8 "/home/buildbot/buildbot/build/mariadb-10.2.34/mysys/thr_timer.c", src_line=292, abstime=0x3fffa5c0e4b8) at ./include/mysql/psi/mysql_thread.h:1182
#2  timer_handler (arg=<optimized out>) at ./mysys/thr_timer.c:292
#3  0x00000000365ba0a8 in pfs_spawn_thread (arg=0x100006063e8) at ./storage/perfschema/pfs.cc:1869
#4  0x00003fffaacc80fc in start_thread (arg=0x3fffa5c0f000) at pthread_create.c:458
#5  0x00003fffaa431748 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:85
 
Thread 7 (Thread 0x3fff9fc08000 (LWP 21708)):
#0  0x00003fffaacd1264 in __pthread_cond_timedwait (cond=0x36d7e990 <mysql_bin_log+1280>, mutex=0x36d7e8f8 <mysql_bin_log+1128>, abstime=0x3fff9fc035e0) at pthread_cond_timedwait.c:200
#1  0x00000000360bd510 in inline_mysql_cond_timedwait (src_file=0x3676edb0 "/home/buildbot/buildbot/build/mariadb-10.2.34/sql/log.cc", src_line=8371, abstime=0x3fff9fc035e0, mutex=<optimized out>, that=<optimized out>) at ./include/mysql/psi/mysql_thread.h:1182
#2  MYSQL_BIN_LOG::wait_for_update_binlog_end_pos (this=0x36d7e490 <mysql_bin_log>, thd=0x3fff740009a8, timeout=0x3fff9fc035e0) at ./sql/log.cc:8370
#3  0x0000000035da5520 in wait_new_events (end_pos_ptr=<synthetic pointer>, binlog_end_pos_filename=0x3fff9fc03658 "./master-bin.000001", linfo=<optimized out>, info=<optimized out>) at ./sql/sql_repl.cc:2425
#4  get_binlog_end_pos (info=0x3fff9fc05210, log=0x3fff9fc03ee8, linfo=0x3fff9fc04fc0) at ./sql/sql_repl.cc:2545
#5  0x0000000035da8864 in send_one_binlog_file (start_pos=<optimized out>, linfo=0x3fff9fc04fc0, log=0x3fff9fc03ee8, info=0x3fff9fc05210) at ./sql/sql_repl.cc:2705
#6  mysql_binlog_send (thd=0x3fff740009a8, log_ident=<optimized out>, pos=<optimized out>, flags=<optimized out>) at ./sql/sql_repl.cc:2842
#7  0x0000000035d8b0e0 in dispatch_command (command=<optimized out>, thd=0x3fff740009a8, packet=0x3fff7400adb9 "", packet_length=<optimized out>, is_com_multi=<optimized out>, is_next_command=<optimized out>) at ./sql/sql_parse.cc:2076
#8  0x0000000035d8bb64 in do_command (thd=0x3fff740009a8) at ./sql/sql_parse.cc:1377
#9  0x0000000035e839e4 in do_handle_one_connection (connect=<optimized out>) at ./sql/sql_connect.cc:1336
#10 0x0000000035e83bc4 in handle_one_connection (arg=0x100007030f8) at ./sql/sql_connect.cc:1241
#11 0x00000000365ba0a8 in pfs_spawn_thread (arg=0x100005d8e78) at ./storage/perfschema/pfs.cc:1869
#12 0x00003fffaacc80fc in start_thread (arg=0x3fff9fc08000) at pthread_create.c:458
#13 0x00003fffaa431748 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:85
 
Thread 6 (Thread 0x3fff9fd58000 (LWP 21452)):
#0  0x00003fffaacd1264 in __pthread_cond_timedwait (cond=0x36d58fe0 <COND_thread_cache>, mutex=0x36d56a70 <LOCK_thread_cache>, abstime=0x3fff9fd573f8) at pthread_cond_timedwait.c:200
#1  0x0000000035cb0c3c in inline_mysql_cond_timedwait (src_file=0x366c2a50 "/home/buildbot/buildbot/build/mariadb-10.2.34/sql/mysqld.cc", src_line=2974, abstime=0x3fff9fd573f8, mutex=0x36d56a70 <LOCK_thread_cache>, that=0x36d58fe0 <COND_thread_cache>) at ./include/mysql/psi/mysql_thread.h:1182
#2  cache_thread (thd=0x3fff780009a8) at ./sql/mysqld.cc:2973
#3  one_thread_per_connection_end (thd=0x3fff780009a8, put_in_cache=<optimized out>) at ./sql/mysqld.cc:3062
#4  0x0000000035e838d8 in do_handle_one_connection (connect=<optimized out>) at ./sql/sql_connect.cc:1355
#5  0x0000000035e83bc4 in handle_one_connection (arg=0x100007030f8) at ./sql/sql_connect.cc:1241
#6  0x00000000365ba0a8 in pfs_spawn_thread (arg=0x10000737198) at ./storage/perfschema/pfs.cc:1869
#7  0x00003fffaacc80fc in start_thread (arg=0x3fff9fd58000) at pthread_create.c:458
#8  0x00003fffaa431748 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:85
 
Thread 5 (Thread 0x3fff9fc78000 (LWP 21454)):
#0  0x00003fffaacd1264 in __pthread_cond_timedwait (cond=0x36d58fe0 <COND_thread_cache>, mutex=0x36d56a70 <LOCK_thread_cache>, abstime=0x3fff9fc773f8) at pthread_cond_timedwait.c:200
#1  0x0000000035cb0c3c in inline_mysql_cond_timedwait (src_file=0x366c2a50 "/home/buildbot/buildbot/build/mariadb-10.2.34/sql/mysqld.cc", src_line=2974, abstime=0x3fff9fc773f8, mutex=0x36d56a70 <LOCK_thread_cache>, that=0x36d58fe0 <COND_thread_cache>) at ./include/mysql/psi/mysql_thread.h:1182
#2  cache_thread (thd=0x3fff700009a8) at ./sql/mysqld.cc:2973
#3  one_thread_per_connection_end (thd=0x3fff700009a8, put_in_cache=<optimized out>) at ./sql/mysqld.cc:3062
#4  0x0000000035e838d8 in do_handle_one_connection (connect=<optimized out>) at ./sql/sql_connect.cc:1355
#5  0x0000000035e83bc4 in handle_one_connection (arg=0x100007030f8) at ./sql/sql_connect.cc:1241
#6  0x00000000365ba0a8 in pfs_spawn_thread (arg=0x10000737838) at ./storage/perfschema/pfs.cc:1869
#7  0x00003fffaacc80fc in start_thread (arg=0x3fff9fc78000) at pthread_create.c:458
#8  0x00003fffaa431748 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:85
 
Thread 4 (Thread 0x3fff9ff18000 (LWP 21388)):
#0  0x00003fffaacd0ca4 in __pthread_cond_wait (cond=0x36d58fa8 <COND_slave_background>, mutex=0x36d567d0 <LOCK_slave_background>) at pthread_cond_wait.c:188
#1  0x0000000035cdab54 in inline_mysql_cond_wait (that=<optimized out>, mutex=<optimized out>, src_file=0x366cbeb0 "/home/buildbot/buildbot/build/mariadb-10.2.34/sql/slave.cc", src_line=337) at ./include/mysql/psi/mysql_thread.h:1145
#2  handle_slave_background (arg=<optimized out>) at ./sql/slave.cc:337
#3  0x00000000365ba0a8 in pfs_spawn_thread (arg=0x10000785478) at ./storage/perfschema/pfs.cc:1869
#4  0x00003fffaacc80fc in start_thread (arg=0x3fff9ff18000) at pthread_create.c:458
#5  0x00003fffaa431748 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:85
 
Thread 3 (Thread 0x3fff9ff88000 (LWP 21387)):
#0  0x00003fffaacd7ac8 in do_sigwait (set=<optimized out>, set@entry=0x3fff9ff87470, sig=sig@entry=0x3fff9ff87420) at ../sysdeps/unix/sysv/linux/sigwait.c:61
#1  0x00003fffaacd7b88 in __sigwait (set=0x3fff9ff87470, sig=0x3fff9ff87420) at ../sysdeps/unix/sysv/linux/sigwait.c:96
#2  0x0000000035cae48c in signal_hand (arg=<optimized out>) at ./sql/mysqld.cc:3510
#3  0x00000000365ba0a8 in pfs_spawn_thread (arg=0x10000731608) at ./storage/perfschema/pfs.cc:1869
#4  0x00003fffaacc80fc in start_thread (arg=0x3fff9ff88000) at pthread_create.c:458
#5  0x00003fffaa431748 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:85
 
Thread 2 (Thread 0x3fffa485f000 (LWP 21384)):
#0  0x00003fffaacd1264 in __pthread_cond_timedwait (cond=0x375fb4a0 <COND_checkpoint>, mutex=0x375fb468 <LOCK_checkpoint>, abstime=0x3fffa485e3a8) at pthread_cond_timedwait.c:200
#1  0x00000000364c0f1c in inline_mysql_cond_timedwait (src_file=0x367c8140 "/home/buildbot/buildbot/build/mariadb-10.2.34/storage/maria/ma_servicethread.c", src_line=116, abstime=<optimized out>, mutex=0x375fb468 <LOCK_checkpoint>, that=0x375fb4a0 <COND_checkpoint>) at ./include/mysql/psi/mysql_thread.h:1182
#2  my_service_thread_sleep (control=0x36cd88c8 <checkpoint_control>, sleep_time=<optimized out>) at ./storage/maria/ma_servicethread.c:115
#3  0x00000000364b5498 in ma_checkpoint_background (arg=0x1e) at ./storage/maria/ma_checkpoint.c:709
#4  0x00000000365ba0a8 in pfs_spawn_thread (arg=0x10000702fc8) at ./storage/perfschema/pfs.cc:1869
#5  0x00003fffaacc80fc in start_thread (arg=0x3fffa485f000) at pthread_create.c:458
#6  0x00003fffaa431748 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:85
 
Thread 1 (Thread 0x3fffaad79ec0 (LWP 21379)):
#0  0x00003fffaacd2898 in __pthread_kill (threadid=<optimized out>, signo=<optimized out>) at ../sysdeps/unix/sysv/linux/pthread_kill.c:55
#1  0x00000000366216c0 in my_write_core (sig=<optimized out>) at ./mysys/stacktrace.c:382
#2  0x0000000035f94c08 in handle_fatal_signal (sig=<optimized out>) at ./sql/signal_handler.cc:343
#3  <signal handler called>
#4  0x00003fffaa424238 in poll () at ../sysdeps/unix/syscall-template.S:84
#5  0x00003fffaa44a634 in __poll_chk (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>, fdslen=<optimized out>) at poll_chk.c:27
#6  0x0000000035cbada8 in poll (__timeout=-1, __nfds=2, __fds=0x3fffea4bf000) at /usr/include/powerpc64le-linux-gnu/bits/poll2.h:41
#7  handle_connections_sockets () at ./sql/mysqld.cc:6669
#8  0x0000000035cbcd44 in mysqld_main (argc=<optimized out>, argv=<optimized out>) at ./sql/mysqld.cc:6137
#9  0x0000000035c95710 in main (argc=<optimized out>, argv=<optimized out>) at ./sql/main.cc:25
 
 - deleting it, already saved 0
 - saving '/dev/shm/var/3/log/rpl.show_status_stop_slave_race-7126-row/' to '/dev/shm/var/log/rpl.show_status_stop_slave_race-7126-row/'

on 10.4 with ASAN 2020-08-21 http://buildbot.askmonty.org/buildbot/builders/kvm-asan/builds/4253/steps/mtr_nm/logs/stdio :

=================================================================
==49024==ERROR: AddressSanitizer: heap-use-after-free on address 0x6150000b0208 at pc 0x5631a2b3be83 bp 0x7fa8a45b1a80 sp 0x7fa8a45b1a70
READ of size 4 at 0x6150000b0208 thread T52
    #0 0x5631a2b3be82 in vio_socket_shutdown /home/buildbot/buildbot/build/mariadb-10.4.15/vio/viosocket.c:325
    #1 0x5631a0d7f005 in THD::awake_no_mutex(killed_state) /home/buildbot/buildbot/build/mariadb-10.4.15/sql/sql_class.cc:1891
    #2 0x5631a0c47ce5 in terminate_slave_thread /home/buildbot/buildbot/build/mariadb-10.4.15/sql/slave.cc:1231
    #3 0x5631a0c4d93d in terminate_slave_threads(Master_info*, int, bool) /home/buildbot/buildbot/build/mariadb-10.4.15/sql/slave.cc:1108
    #4 0x5631a0c4d93d in terminate_slave_threads(Master_info*, int, bool) /home/buildbot/buildbot/build/mariadb-10.4.15/sql/slave.cc:1064
    #5 0x5631a0f0a7c9 in stop_slave(THD*, Master_info*, bool) /home/buildbot/buildbot/build/mariadb-10.4.15/sql/sql_repl.cc:3316
    #6 0x5631a0e96341 in mysql_execute_command(THD*) /home/buildbot/buildbot/build/mariadb-10.4.15/sql/sql_parse.cc:4238
    #7 0x5631a0eaa1bf in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) /home/buildbot/buildbot/build/mariadb-10.4.15/sql/sql_parse.cc:7896
    #8 0x5631a0eb37ff in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /home/buildbot/buildbot/build/mariadb-10.4.15/sql/sql_parse.cc:1935
    #9 0x5631a0eb8b2a in do_command(THD*) /home/buildbot/buildbot/build/mariadb-10.4.15/sql/sql_parse.cc:1352
    #10 0x5631a12344b6 in do_handle_one_connection(CONNECT*) /home/buildbot/buildbot/build/mariadb-10.4.15/sql/sql_connect.cc:1412
    #11 0x5631a1234a3e in handle_one_connection /home/buildbot/buildbot/build/mariadb-10.4.15/sql/sql_connect.cc:1316
    #12 0x5631a27ad678 in pfs_spawn_thread /home/buildbot/buildbot/build/mariadb-10.4.15/storage/perfschema/pfs.cc:1869
    #13 0x7fa8bcddc608 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x9608)
    #14 0x7fa8bc91e102 in __clone (/lib/x86_64-linux-gnu/libc.so.6+0x122102)
 
0x6150000b0208 is located 8 bytes inside of 504-byte region [0x6150000b0200,0x6150000b03f8)
freed by thread T195 here:
    #0 0x7fa8bcf647cf in __interceptor_free (/lib/x86_64-linux-gnu/libasan.so.5+0x10d7cf)
    #1 0x5631a156c93d in end_server /home/buildbot/buildbot/build/mariadb-10.4.15/sql-common/client.c:695
    #2 0x5631a157525c in mysql_close /home/buildbot/buildbot/build/mariadb-10.4.15/sql-common/client.c:3470
    #3 0x5631a157525c in mysql_close /home/buildbot/buildbot/build/mariadb-10.4.15/sql-common/client.c:3463
    #4 0x5631a0c678f1 in handle_slave_io /home/buildbot/buildbot/build/mariadb-10.4.15/sql/slave.cc:5090
    #5 0x5631a27ad678 in pfs_spawn_thread /home/buildbot/buildbot/build/mariadb-10.4.15/storage/perfschema/pfs.cc:1869
    #6 0x7fa8bcddc608 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x9608)
 
previously allocated by thread T195 here:
    #0 0x7fa8bcf64bc8 in malloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10dbc8)
    #1 0x5631a2a0a496 in my_malloc /home/buildbot/buildbot/build/mariadb-10.4.15/mysys/my_malloc.c:101
    #2 0x5631a2b3a0dd in mysql_socket_vio_new /home/buildbot/buildbot/build/mariadb-10.4.15/vio/vio.c:221
    #3 0x5631a2b3aeb7 in vio_new /home/buildbot/buildbot/build/mariadb-10.4.15/vio/vio.c:240
    #4 0x5631a1579f20 in mysql_real_connect /home/buildbot/buildbot/build/mariadb-10.4.15/sql-common/client.c:2857
    #5 0x5631a0c46f73 in connect_to_master /home/buildbot/buildbot/build/mariadb-10.4.15/sql/slave.cc:7311
    #6 0x5631a0c66f0c in safe_connect /home/buildbot/buildbot/build/mariadb-10.4.15/sql/slave.cc:7223
    #7 0x5631a0c66f0c in handle_slave_io /home/buildbot/buildbot/build/mariadb-10.4.15/sql/slave.cc:4762
    #8 0x5631a27ad678 in pfs_spawn_thread /home/buildbot/buildbot/build/mariadb-10.4.15/storage/perfschema/pfs.cc:1869
    #9 0x7fa8bcddc608 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x9608)
 
Thread T52 created by T0 here:
    #0 0x7fa8bce91805 in pthread_create (/lib/x86_64-linux-gnu/libasan.so.5+0x3a805)
    #1 0x5631a27b4dce in spawn_thread_v1 /home/buildbot/buildbot/build/mariadb-10.4.15/storage/perfschema/pfs.cc:1919
    #2 0x5631a0bf82ce in inline_mysql_thread_create /home/buildbot/buildbot/build/mariadb-10.4.15/include/mysql/psi/mysql_thread.h:1275
    #3 0x5631a0bf82ce in create_thread_to_handle_connection(CONNECT*) /home/buildbot/buildbot/build/mariadb-10.4.15/sql/mysqld.cc:6261
    #4 0x5631a0c045f2 in create_new_thread(CONNECT*) /home/buildbot/buildbot/build/mariadb-10.4.15/sql/mysqld.cc:6331
    #5 0x5631a0c04c02 in handle_accepted_socket(st_mysql_socket, st_mysql_socket) /home/buildbot/buildbot/build/mariadb-10.4.15/sql/mysqld.cc:6429
    #6 0x5631a0c05e1f in handle_connections_sockets() /home/buildbot/buildbot/build/mariadb-10.4.15/sql/mysqld.cc:6587
    #7 0x5631a0c078b2 in mysqld_main(int, char**) /home/buildbot/buildbot/build/mariadb-10.4.15/sql/mysqld.cc:5919
    #8 0x7fa8bc8230b2 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x270b2)
 
Thread T195 created by T52 here:
    #0 0x7fa8bce91805 in pthread_create (/lib/x86_64-linux-gnu/libasan.so.5+0x3a805)
    #1 0x5631a27b4dce in spawn_thread_v1 /home/buildbot/buildbot/build/mariadb-10.4.15/storage/perfschema/pfs.cc:1919
    #2 0x5631a0c4df45 in inline_mysql_thread_create /home/buildbot/buildbot/build/mariadb-10.4.15/include/mysql/psi/mysql_thread.h:1275
    #3 0x5631a0c4df45 in start_slave_thread(unsigned int, void* (*)(void*), st_mysql_mutex*, st_mysql_mutex*, st_mysql_cond*, unsigned int volatile*, unsigned long volatile*, Master_info*) /home/buildbot/buildbot/build/mariadb-10.4.15/sql/slave.cc:1296
    #4 0x5631a0c4e7ab in start_slave_threads(THD*, bool, bool, Master_info*, char const*, char const*, int) /home/buildbot/buildbot/build/mariadb-10.4.15/sql/slave.cc:1412
    #5 0x5631a0f09a88 in start_slave(THD*, Master_info*, bool) /home/buildbot/buildbot/build/mariadb-10.4.15/sql/sql_repl.cc:3240
    #6 0x5631a0e944a6 in mysql_execute_command(THD*) /home/buildbot/buildbot/build/mariadb-10.4.15/sql/sql_parse.cc:4202
    #7 0x5631a0eaa1bf in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) /home/buildbot/buildbot/build/mariadb-10.4.15/sql/sql_parse.cc:7896
    #8 0x5631a0eb2d66 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /home/buildbot/buildbot/build/mariadb-10.4.15/sql/sql_parse.cc:1834
    #9 0x5631a0eb8b2a in do_command(THD*) /home/buildbot/buildbot/build/mariadb-10.4.15/sql/sql_parse.cc:1352
    

Comment by Andrei Elkin [ 2020-09-10 ]

Smells of deadlock. The recent trace shows at mtr timeout aborting
the slave's IO thread (Thread 18) is waiting for a mutex, apparently locked by STOP-SLAVE (thread 17) that is waiting
for a notification.

Comment by Sachin Setiya (Inactive) [ 2020-09-10 ]

alice that is different failure.
In windows it fails differently.

Comment by Sachin Setiya (Inactive) [ 2020-09-10 ]

So the issue is 
mysqlslap does not differentiate between sockets on windows
For example this
C:\Users\Sachin\Desktop\Mariadb\server\bld\client\Debug>C:\Users\Sachin\Desktop\Mariadb\server\bld\client\Debug\mysqlslap.exe --defaults-file=C:/Users/Sachin/Desktop/Mariadb/server/bld/mysql-test/var/my.cnf --silent --socket=C:/Users/Sachin/Desktop/Mariadb/server/bld/mysql-test/var/tmp/mysqld.2.sock -q "create table a2(a int)"  --create-schema=test  
NO Issue table is created but on Mysqld.1 instead of mysqld.2
Proof
Changing the socket
C:\Users\Sachin\Desktop\Mariadb\server\bld\client\Debug>C:\Users\Sachin\Desktop\Mariadb\server\bld\client\Debug\mysqlslap.exe --defaults-file=C:/Users/Sachin/Desktop/Mariadb/server/bld/mysql-test/var/my.cnf --silent --socket=C:/Users/Sachin/Desktop/Mariadb/server/bld/mysql-test/var/tmp/mysqld.1.sock -q "create table a2(a int)"  --create-schema=test
C:\Users\Sachin\Desktop\Mariadb\server\bld\client\Debug\mysqlslap.exe: Cannot run query create table a2(a int) ERROR : Table 'a2' already exists
In fact the socket does not event exist
I tried this socket C:/Users/Sachin/Desktop/Mariadb/server/bld/mysql-test/var/tmp/xyz
C:\Users\Sachin\Desktop\Mariadb\server\bld\client\Debug>C:\Users\Sachin\Desktop\Mariadb\server\bld\client\Debug\mysqlslap.exe --defaults-file=C:/Users/Sachin/Desktop/Mariadb/server/bld/mysql-test/var/my.cnf --silent --socket=C:/Users/Sachin/Desktop/Mariadb/server/bld/mysql-test/var/tmp/xyz -q "create table a2(a int)"  --create-schema=test
C:\Users\Sachin\Desktop\Mariadb\server\bld\client\Debug>C:\Users\Sachin\Desktop\Mariadb\server\bld\client\Debug\mysqlslap.exe --defaults-file=C:/Users/Sachin/Desktop/Mariadb/server/bld/mysql-test/var/my.cnf --silent --socket=C:/Users/Sachin/Desktop/Mariadb/server/bld/mysql-test/var/tmp/xyz -q "create table a2(a int)"  --create-schema=test
C:\Users\Sachin\Desktop\Mariadb\server\bld\client\Debug\mysqlslap.exe: Cannot run query create table a2(a int) ERROR : Table 'a2' already exists
 
 
socket does not exist and still  mysqlslap is able to connect .
In linux it works without any issue

Generated at Thu Feb 08 08:36:28 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.