[MDEV-7098] spider/bg.spider_fixes failed in buildbot with safe_mutex: Trying to unlock mutex conn->mta_conn_mutex that wasn't locked at storage/spider/spd_db_conn.cc, line 671 Created: 2014-11-13  Updated: 2023-11-27  Resolved: 2020-09-07

Status: Closed
Project: MariaDB Server
Component/s: Locking, Storage Engine - Spider, Tests
Affects Version/s: 10.0, 10.1, 10.2
Fix Version/s: 10.1.48, 10.2.35, 10.3.26, 10.4.16, 10.5.7, 10.6.0

Type: Bug Priority: Major
Reporter: Elena Stepanova Assignee: Kentoku Shiba (Inactive)
Resolution: Fixed Votes: 1
Labels: buildbot, powerpc, spider, tests

Issue Links:
Duplicate
Relates
relates to MDEV-7069 Fix buildbot failures in main server ... Stalled

 Description   

It's a one-time failure so far, but it looks nasty.
http://buildbot.askmonty.org/buildbot/builders/bintar-rhel6-p8-debug/builds/218/steps/test/logs/stdio

CURRENT_TEST: spider/bg.spider_fixes
mysqltest: At line 1117: query 'SELECT MAX(id) FROM t1' failed: 2013: Lost connection to MySQL server during query
 
The result from queries just before the failure was:
< snip >
1554
2331
3108
3885
4662
5439
6216
SET INSERT_ID=5000;
INSERT INTO t1 () VALUES ();
SELECT LAST_INSERT_ID();
LAST_INSERT_ID()
5000
SELECT MAX(id) FROM t1;
MAX(id)
6216
INSERT INTO t1 (id) VALUES (10000);
SELECT LAST_INSERT_ID();
LAST_INSERT_ID()
5000
SELECT MAX(id) FROM t1;

Server [mysqld.1.1 - pid: 27307, winpid: 27307, exit: 256] failed during test run
Server log from this test:
----------SERVER LOG START-----------
safe_mutex: Trying to unlock mutex conn->mta_conn_mutex that wasn't locked at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/spider/spd_db_conn.cc, line 671
Last used at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/spider/spd_conn.cc, line: 2379
141112 11:36:49 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see http://kb.askmonty.org/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
Server version: 10.0.15-MariaDB-debug-log
key_buffer_size=1048576
read_buffer_size=131072
max_used_connections=4
max_threads=153
thread_count=3
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 62987 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x0xfff896ac070
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0xfff77ff6680 thread_stack 0x48000
:0(00000017.plt_call.strsignal@@GLIBC_2.3)[0x10d25084]
sql/signal_handler.cc:153(00000017.plt_call.strsignal@@GLIBC_2.3)[0x1050f618]
linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0xfff8ab30448]
/opt/at7.0/lib64/power7/libc.so.6(gsignal-0x16f708)[0xfff8a26f8f0]
/opt/at7.0/lib64/power7/libc.so.6(abort-0x16dab4)[0xfff8a2719c4]
:0(00000017.plt_call.strsignal@@GLIBC_2.3)[0x10d2bae0]
/home/buildbot/maria-slave/power8-vlp01-debug/build/mysql-test/var/3/plugins/ha_spider.so(+0x74dc4)[0xfff85e34dc4]
/home/buildbot/maria-slave/power8-vlp01-debug/build/mysql-test/var/3/plugins/ha_spider.so(_Z17spider_db_errornoP14st_spider_conn-0x135b74)[0xfff85e38984]
/home/buildbot/maria-slave/power8-vlp01-debug/build/mysql-test/var/3/plugins/ha_spider.so(_ZN15spider_db_mysql6commitEPi-0x7503c)[0xfff85efe8bc]
/home/buildbot/maria-slave/power8-vlp01-debug/build/mysql-test/var/3/plugins/ha_spider.so(_Z16spider_db_commitP14st_spider_conn-0x133534)[0xfff85e3b09c]
/home/buildbot/maria-slave/power8-vlp01-debug/build/mysql-test/var/3/plugins/ha_spider.so(_Z13spider_commitP10handlertonP3THDb-0x13cae0)[0xfff85e31580]
sql/handler.cc:1527(00000017.plt_call.strsignal@@GLIBC_2.3)[0x10514548]
sql/handler.cc:1508(00000017.plt_call.strsignal@@GLIBC_2.3)[0x105143e8]
sql/handler.cc:1385(00000017.plt_call.strsignal@@GLIBC_2.3)[0x10513c38]
sql/transaction.cc:394(00000017.plt_call.strsignal@@GLIBC_2.3)[0x1040f35c]
sql/sql_parse.cc:5126(00000017.plt_call.strsignal@@GLIBC_2.3)[0x102914e8]
sql/sql_parse.cc:6407(00000017.plt_call.strsignal@@GLIBC_2.3)[0x10294bb0]
sql/sql_parse.cc:1299(00000017.plt_call.strsignal@@GLIBC_2.3)[0x1028655c]
sql/sql_parse.cc:996(00000017.plt_call.strsignal@@GLIBC_2.3)[0x1028572c]
sql/sql_connect.cc:1379(00000017.plt_call.strsignal@@GLIBC_2.3)[0x103f3af0]
sql/sql_connect.cc:1293(00000017.plt_call.strsignal@@GLIBC_2.3)[0x103f3754]
/opt/at7.0/lib64/power7/libpthread.so.0(+0xc548)[0xfff8aafc548]
/opt/at7.0/lib64/power7/libc.so.6(clone-0xa3978)[0xfff8a343c90]
 
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0xfff77410888): SELECT MAX(id) FROM t1
Connection ID (thread ID): 21
Status: NOT_KILLED
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on
 
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Writing a core file
----------SERVER LOG END-------------
 
 
 - saving '/home/buildbot/maria-slave/power8-vlp01-debug/build/mysql-test/var/3/log/spider/bg.spider_fixes/' to '/home/buildbot/maria-slave/power8-vlp01-debug/build/mysql-test/var/log/bg.spider_fixes/'
 - found 'core.27308' (0/5)
 
Trying 'dbx' to get a backtrace
 
Trying 'gdb' to get a backtrace
Core generated by '/home/buildbot/maria-slave/power8-vlp01-debug/build/sql/mysqld'
Output from gdb follows. The first stack trace is from the failing thread.
The following stack traces are from all threads (so the failing one is
duplicated).
--------------------------
[New Thread 27698]
[New Thread 27310]
[New Thread 27316]
[New Thread 27382]
[New Thread 27708]
[New Thread 27315]
[New Thread 27380]
[New Thread 27314]
[New Thread 27713]
[New Thread 27312]
[New Thread 27385]
[New Thread 27384]
[New Thread 27386]
[New Thread 27388]
[New Thread 27383]
[New Thread 27317]
[New Thread 27710]
[New Thread 27308]
[New Thread 27387]
[New Thread 27714]
[New Thread 27389]
[New Thread 27390]
[New Thread 27397]
[New Thread 27697]
[New Thread 27381]
[New Thread 27313]
 
warning: Ignoring non-absolute filename: <linux-vdso64.so.1>
Missing separate debuginfo for linux-vdso64.so.1
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/05/920648c05dcd5d47af1fe7b0befecb469a256d
 
warning: File "/opt/at7.0/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".
To enable execution of this file add
	add-auto-load-safe-path /opt/at7.0/lib64/libthread_db-1.0.so
line to your configuration file "/home/buildbot/.gdbinit".
To completely disable this security protection add
	set auto-load safe-path /
line to your configuration file "/home/buildbot/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
	info "(gdb)Auto-loading safe path"
 
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
 
warning: File "/opt/at7.0/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".
 
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
 
warning: File "/opt/at7.0/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".
 
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
 
warning: File "/opt/at7.0/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".
 
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
 
warning: File "/opt/at7.0/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".
 
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
 
warning: File "/opt/at7.0/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".
 
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
 
warning: File "/opt/at7.0/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".
 
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
 
warning: File "/opt/at7.0/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".
 
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
 
warning: File "/opt/at7.0/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".
 
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
 
warning: File "/opt/at7.0/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".
 
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
 
warning: File "/opt/at7.0/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".
 
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
 
warning: File "/opt/at7.0/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".
 
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
 
warning: File "/opt/at7.0/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".
 
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
 
warning: File "/opt/at7.0/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".
 
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
 
warning: File "/opt/at7.0/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".
 
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
Core was generated by `/home/buildbot/maria-slave/power8-vlp01-debug/build/sql/mysqld --defaults-group'.
Program terminated with signal 6, Aborted.
#0  0x00000fff8ab03a40 in .pthread_kill () from /opt/at7.0/lib64/power7/libpthread.so.0
#0  0x00000fff8ab03a40 in .pthread_kill () from /opt/at7.0/lib64/power7/libpthread.so.0
#1  0x0000000010d251d0 in .my_write_core ()
#2  0x000000001050fa48 in .handle_fatal_signal () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/signal_handler.cc:262
#3  <signal handler called>
#4  0x00000fff8a26f8f0 in .raise () from /opt/at7.0/lib64/power7/libc.so.6
#5  0x00000fff8a2718f4 in .abort () from /opt/at7.0/lib64/power7/libc.so.6
#6  0x0000000010d2bae0 in .safe_mutex_unlock ()
#7  0x00000fff85e34dc4 in .inline_mysql_mutex_unlock () at /home/buildbot/maria-slave/power8-vlp01-debug/build/include/mysql/psi/mysql_thread.h:761
#8  0x00000fff85e38984 in ._Z17spider_db_errornoP14st_spider_conn () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/spider/spd_db_conn.cc:671
#9  0x00000fff85efe8bc in ._ZN15spider_db_mysql6commitEPi () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/spider/spd_db_mysql.cc:2087
#10 0x00000fff85e3b09c in ._Z16spider_db_commitP14st_spider_conn () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/spider/spd_db_conn.cc:1148
#11 0x00000fff85e31580 in ._Z13spider_commitP10handlertonP3THDb () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/spider/spd_trx.cc:3327
#12 0x0000000010514548 in ._ZL18commit_one_phase_2P3THDbP9THD_TRANSb () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/handler.cc:1527
#13 0x00000000105143e8 in ._Z19ha_commit_one_phaseP3THDb () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/handler.cc:1508
#14 0x0000000010513c38 in ._Z15ha_commit_transP3THDb () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/handler.cc:1385
#15 0x000000001040f35c in ._Z17trans_commit_stmtP3THD () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/transaction.cc:394
#16 0x00000000102914e8 in ._Z21mysql_execute_commandP3THD () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/sql_parse.cc:5126
#17 0x0000000010294bb0 in ._Z11mysql_parseP3THDPcjP12Parser_state () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/sql_parse.cc:6407
#18 0x000000001028655c in ._Z16dispatch_command19enum_server_commandP3THDPcj () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/sql_parse.cc:1299
#19 0x000000001028572c in ._Z10do_commandP3THD () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/sql_parse.cc:996
#20 0x00000000103f3af0 in ._Z24do_handle_one_connectionP3THD () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/sql_connect.cc:1379
#21 0x00000000103f3754 in .handle_one_connection () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/sql_connect.cc:1293
#22 0x00000fff8aafc548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0
#23 0x00000fff8a343c90 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:106
 
Thread 26 (Thread 27313):
#0  0x0000008078cd0958 in ?? () from /lib64/libaio.so.1
#1  0x0000008078cd09bc in .io_getevents () from /lib64/libaio.so.1
#2  0x000000001092b5e8 in ._ZL20os_aio_linux_collectP14os_aio_array_tmm () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/os/os0file.cc:5098
#3  0x000000001092ba6c in ._Z19os_aio_linux_handlemPP10fil_node_tPPvPmS4_ () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/os/os0file.cc:5244
#4  0x0000000010b71118 in ._Z12fil_aio_waitm () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/fil/fil0fil.cc:5662
#5  0x0000000010a2edb0 in .io_handler_thread () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/srv/srv0start.cc:499
#6  0x00000fff8aafc548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0
#7  0x00000fff8a343c90 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:106
 
Thread 25 (Thread 27381):
#0  0x00000fff8ab025a4 in .pthread_cond_timedwait () from /opt/at7.0/lib64/power7/libpthread.so.0
#1  0x000000001092e64c in ._ZL18os_cond_wait_timedP14pthread_cond_tP15os_fast_mutex_tPK8timespec () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/os/os0sync.cc:177
#2  0x000000001092f220 in ._Z22os_event_wait_time_lowP8os_eventml () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/os/os0sync.cc:723
#3  0x0000000010a27d18 in .srv_error_monitor_thread () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/srv/srv0srv.cc:2241
#4  0x00000fff8aafc548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0
#5  0x00000fff8a343c90 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:106
 
Thread 24 (Thread 27697):
#0  0x00000fff8a334dd0 in .__GI___poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x0000000010d931fc in .vio_io_wait ()
#2  0x0000000010d91414 in .vio_socket_io_wait ()
#3  0x0000000010d91660 in .vio_read ()
#4  0x000000001019ea48 in ._ZL12my_real_readP6st_netPmc ()
#5  0x000000001019f4c0 in .my_net_read_packet ()
#6  0x0000000010285440 in ._Z10do_commandP3THD () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/sql_parse.cc:929
#7  0x00000000103f3af0 in ._Z24do_handle_one_connectionP3THD () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/sql_connect.cc:1379
#8  0x00000000103f3754 in .handle_one_connection () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/sql_connect.cc:1293
#9  0x00000fff8aafc548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0
#10 0x00000fff8a343c90 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:106
 
Thread 23 (Thread 27397):
#0  0x00000fff8ab07f18 in .__libc_sigwait () from /opt/at7.0/lib64/power7/libpthread.so.0
#1  0x000000001018c8e4 in .signal_hand () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/mysqld.cc:3343
#2  0x00000fff8aafc548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0
#3  0x00000fff8a343c90 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:106
 
Thread 22 (Thread 27390):
#0  0x00000fff8ab02130 in .pthread_cond_wait () from /opt/at7.0/lib64/power7/libpthread.so.0
#1  0x0000000010d2bef4 in .safe_cond_wait ()
#2  0x0000000010643920 in .inline_mysql_cond_wait () at /home/buildbot/maria-slave/power8-vlp01-debug/build/include/mysql/psi/mysql_thread.h:1165
#3  0x0000000010660694 in .binlog_background_thread () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/log.cc:9133
#4  0x00000fff8aafc548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0
#5  0x00000fff8a343c90 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:106
 
Thread 21 (Thread 27389):
#0  0x00000fff8ab025a4 in .pthread_cond_timedwait () from /opt/at7.0/lib64/power7/libpthread.so.0
#1  0x000000001092e64c in ._ZL18os_cond_wait_timedP14pthread_cond_tP15os_fast_mutex_tPK8timespec () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/os/os0sync.cc:177
#2  0x000000001092f220 in ._Z22os_event_wait_time_lowP8os_eventml () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/os/os0sync.cc:723
#3  0x0000000010a85030 in ._Z19ib_wqueue_timedwaitP11ib_wqueue_tl () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/ut/ut0wqueue.cc:154
#4  0x0000000010bab2a8 in ._Z19fts_optimize_threadPv () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/fts/fts0opt.cc:2996
#5  0x00000fff8aafc548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0
#6  0x00000fff8a343c90 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:106
 
Thread 20 (Thread 27714):
#0  0x00000fff8ab02130 in .pthread_cond_wait () from /opt/at7.0/lib64/power7/libpthread.so.0
#1  0x0000000010d2bef4 in .safe_cond_wait ()
#2  0x00000fff85e5f330 in .inline_mysql_cond_wait () at /home/buildbot/maria-slave/power8-vlp01-debug/build/include/mysql/psi/mysql_thread.h:1165
#3  0x00000fff85e6656c in ._Z21spider_bg_conn_actionPv () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/spider/spd_conn.cc:2282
#4  0x00000fff8aafc548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0
#5  0x00000fff8a343c90 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:106
 
Thread 19 (Thread 27387):
#0  0x00000fff8ab02130 in .pthread_cond_wait () from /opt/at7.0/lib64/power7/libpthread.so.0
#1  0x000000001092e7a8 in ._ZL12os_cond_waitP14pthread_cond_tP15os_fast_mutex_t () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/os/os0sync.cc:214
#2  0x000000001092efec in ._Z17os_event_wait_lowP8os_eventl () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/os/os0sync.cc:610
#3  0x0000000010aefae4 in .buf_dump_thread ()
#4  0x00000fff8aafc548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0
#5  0x00000fff8a343c90 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:106
 
Thread 18 (Thread 27308):
#0  0x00000fff8a334dd0 in .__GI___poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00000000101931dc in ._Z26handle_connections_socketsv () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/mysqld.cc:6103
#2  0x0000000010191f04 in ._Z11mysqld_mainiPPc () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/mysqld.cc:5535
#3  0x0000000010184208 in .main ()
 
Thread 17 (Thread 27710):
#0  0x00000fff8ab02130 in .pthread_cond_wait () from /opt/at7.0/lib64/power7/libpthread.so.0
#1  0x0000000010d2bef4 in .safe_cond_wait ()
#2  0x00000fff85e5f330 in .inline_mysql_cond_wait () at /home/buildbot/maria-slave/power8-vlp01-debug/build/include/mysql/psi/mysql_thread.h:1165
#3  0x00000fff85e6656c in ._Z21spider_bg_conn_actionPv () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/spider/spd_conn.cc:2282
#4  0x00000fff8aafc548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0
#5  0x00000fff8a343c90 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:106
 
Thread 16 (Thread 27317):
#0  0x0000008078cd0958 in ?? () from /lib64/libaio.so.1
#1  0x0000008078cd09bc in .io_getevents () from /lib64/libaio.so.1
#2  0x000000001092b5e8 in ._ZL20os_aio_linux_collectP14os_aio_array_tmm () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/os/os0file.cc:5098
#3  0x000000001092ba6c in ._Z19os_aio_linux_handlemPP10fil_node_tPPvPmS4_ () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/os/os0file.cc:5244
#4  0x0000000010b71118 in ._Z12fil_aio_waitm () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/fil/fil0fil.cc:5662
#5  0x0000000010a2edb0 in .io_handler_thread () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/srv/srv0start.cc:499
#6  0x00000fff8aafc548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0
#7  0x00000fff8a343c90 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:106
 
Thread 15 (Thread 27383):
#0  0x00000fff8a337784 in .__select () at ../sysdeps/unix/syscall-template.S:81
#1  0x00000000109302f8 in ._Z15os_thread_sleepm ()
#2  0x0000000010a29b44 in ._ZL16srv_master_sleepv () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/srv/srv0srv.cc:2969
#3  0x0000000010a29d5c in .srv_master_thread () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/srv/srv0srv.cc:3017
#4  0x00000fff8aafc548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0
#5  0x00000fff8a343c90 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:106
 
Thread 14 (Thread 27388):
#0  0x00000fff8ab025a4 in .pthread_cond_timedwait () from /opt/at7.0/lib64/power7/libpthread.so.0
#1  0x000000001092e64c in ._ZL18os_cond_wait_timedP14pthread_cond_tP15os_fast_mutex_tPK8timespec () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/os/os0sync.cc:177
#2  0x000000001092f220 in ._Z22os_event_wait_time_lowP8os_eventml () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/os/os0sync.cc:723
#3  0x0000000010b59d24 in .dict_stats_thread () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/dict/dict0stats_bg.cc:349
#4  0x00000fff8aafc548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0
#5  0x00000fff8a343c90 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:106
 
Thread 13 (Thread 27386):
#0  0x00000fff8a337784 in .__select () at ../sysdeps/unix/syscall-template.S:81
#1  0x00000000109302f8 in ._Z15os_thread_sleepm ()
#2  0x0000000010afa280 in ._ZL28page_cleaner_sleep_if_neededm () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/buf/buf0flu.cc:2578
#3  0x0000000010afacb8 in .buf_flush_lru_manager_thread () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/buf/buf0flu.cc:2849
#4  0x00000fff8aafc548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0
#5  0x00000fff8a343c90 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:106
 
Thread 12 (Thread 27384):
#0  0x00000fff8ab02130 in .pthread_cond_wait () from /opt/at7.0/lib64/power7/libpthread.so.0
#1  0x000000001092e7a8 in ._ZL12os_cond_waitP14pthread_cond_tP15os_fast_mutex_t () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/os/os0sync.cc:214
#2  0x000000001092efec in ._Z17os_event_wait_lowP8os_eventl () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/os/os0sync.cc:610
#3  0x0000000010a2ab00 in ._ZL29srv_purge_coordinator_suspendP10srv_slot_tm () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/srv/srv0srv.cc:3326
#4  0x0000000010a2b1e8 in .srv_purge_coordinator_thread () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/srv/srv0srv.cc:3448
#5  0x00000fff8aafc548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0
#6  0x00000fff8a343c90 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:106
 
Thread 11 (Thread 27385):
#0  0x00000fff8a337784 in .__select () at ../sysdeps/unix/syscall-template.S:81
#1  0x00000000109302f8 in ._Z15os_thread_sleepm ()
#2  0x0000000010afa280 in ._ZL28page_cleaner_sleep_if_neededm () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/buf/buf0flu.cc:2578
#3  0x0000000010afa698 in .buf_flush_page_cleaner_thread () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/buf/buf0flu.cc:2697
#4  0x00000fff8aafc548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0
#5  0x00000fff8a343c90 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:106
 
Thread 10 (Thread 27312):
#0  0x0000008078cd0958 in ?? () from /lib64/libaio.so.1
#1  0x0000008078cd09bc in .io_getevents () from /lib64/libaio.so.1
#2  0x000000001092b5e8 in ._ZL20os_aio_linux_collectP14os_aio_array_tmm () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/os/os0file.cc:5098
#3  0x000000001092ba6c in ._Z19os_aio_linux_handlemPP10fil_node_tPPvPmS4_ () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/os/os0file.cc:5244
#4  0x0000000010b71118 in ._Z12fil_aio_waitm () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/fil/fil0fil.cc:5662
#5  0x0000000010a2edb0 in .io_handler_thread () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/srv/srv0start.cc:499
#6  0x00000fff8aafc548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0
#7  0x00000fff8a343c90 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:106
 
Thread 9 (Thread 27713):
#0  0x00000fff8ab02130 in .pthread_cond_wait () from /opt/at7.0/lib64/power7/libpthread.so.0
#1  0x0000000010d2bef4 in .safe_cond_wait ()
#2  0x00000fff85e5f330 in .inline_mysql_cond_wait () at /home/buildbot/maria-slave/power8-vlp01-debug/build/include/mysql/psi/mysql_thread.h:1165
#3  0x00000fff85e6656c in ._Z21spider_bg_conn_actionPv () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/spider/spd_conn.cc:2282
#4  0x00000fff8aafc548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0
#5  0x00000fff8a343c90 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:106
 
Thread 8 (Thread 27314):
#0  0x0000008078cd0958 in ?? () from /lib64/libaio.so.1
#1  0x0000008078cd09bc in .io_getevents () from /lib64/libaio.so.1
#2  0x000000001092b5e8 in ._ZL20os_aio_linux_collectP14os_aio_array_tmm () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/os/os0file.cc:5098
#3  0x000000001092ba6c in ._Z19os_aio_linux_handlemPP10fil_node_tPPvPmS4_ () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/os/os0file.cc:5244
#4  0x0000000010b71118 in ._Z12fil_aio_waitm () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/fil/fil0fil.cc:5662
#5  0x0000000010a2edb0 in .io_handler_thread () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/srv/srv0start.cc:499
#6  0x00000fff8aafc548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0
#7  0x00000fff8a343c90 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:106
 
Thread 7 (Thread 27380):
#0  0x00000fff8ab025a4 in .pthread_cond_timedwait () from /opt/at7.0/lib64/power7/libpthread.so.0
#1  0x000000001092e64c in ._ZL18os_cond_wait_timedP14pthread_cond_tP15os_fast_mutex_tPK8timespec () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/os/os0sync.cc:177
#2  0x000000001092f220 in ._Z22os_event_wait_time_lowP8os_eventml () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/os/os0sync.cc:723
#3  0x00000000108f509c in .lock_wait_timeout_thread () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/lock/lock0wait.cc:499
#4  0x00000fff8aafc548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0
#5  0x00000fff8a343c90 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:106
 
Thread 6 (Thread 27315):
#0  0x0000008078cd0958 in ?? () from /lib64/libaio.so.1
#1  0x0000008078cd09bc in .io_getevents () from /lib64/libaio.so.1
#2  0x000000001092b5e8 in ._ZL20os_aio_linux_collectP14os_aio_array_tmm () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/os/os0file.cc:5098
#3  0x000000001092ba6c in ._Z19os_aio_linux_handlemPP10fil_node_tPPvPmS4_ () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/os/os0file.cc:5244
#4  0x0000000010b71118 in ._Z12fil_aio_waitm () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/fil/fil0fil.cc:5662
#5  0x0000000010a2edb0 in .io_handler_thread () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/srv/srv0start.cc:499
#6  0x00000fff8aafc548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0
#7  0x00000fff8a343c90 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:106
 
Thread 5 (Thread 27708):
#0  0x00000fff8ab025a4 in .pthread_cond_timedwait () from /opt/at7.0/lib64/power7/libpthread.so.0
#1  0x0000000010d2c238 in .safe_cond_timedwait ()
#2  0x0000000010643aa0 in .inline_mysql_cond_timedwait () at /home/buildbot/maria-slave/power8-vlp01-debug/build/include/mysql/psi/mysql_thread.h:1202
#3  0x000000001065bf98 in ._ZN13MYSQL_BIN_LOG23wait_for_update_bin_logEP3THDPK8timespec () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/log.cc:7636
#4  0x00000000102c51f8 in ._Z17mysql_binlog_sendP3THDPcyt () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/sql_repl.cc:2547
#5  0x0000000010287194 in ._Z16dispatch_command19enum_server_commandP3THDPcj () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/sql_parse.cc:1524
#6  0x000000001028572c in ._Z10do_commandP3THD () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/sql_parse.cc:996
#7  0x00000000103f3af0 in ._Z24do_handle_one_connectionP3THD () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/sql_connect.cc:1379
#8  0x00000000103f3754 in .handle_one_connection () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/sql_connect.cc:1293
#9  0x00000fff8aafc548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0
#10 0x00000fff8a343c90 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:106
 
Thread 4 (Thread 27382):
#0  0x00000fff8ab025a4 in .pthread_cond_timedwait () from /opt/at7.0/lib64/power7/libpthread.so.0
#1  0x000000001092e64c in ._ZL18os_cond_wait_timedP14pthread_cond_tP15os_fast_mutex_tPK8timespec () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/os/os0sync.cc:177
#2  0x000000001092f220 in ._Z22os_event_wait_time_lowP8os_eventml () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/os/os0sync.cc:723
#3  0x0000000010a27320 in .srv_monitor_thread () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/srv/srv0srv.cc:1977
#4  0x00000fff8aafc548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0
#5  0x00000fff8a343c90 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:106
 
Thread 3 (Thread 27316):
#0  0x0000008078cd0958 in ?? () from /lib64/libaio.so.1
#1  0x0000008078cd09bc in .io_getevents () from /lib64/libaio.so.1
#2  0x000000001092b5e8 in ._ZL20os_aio_linux_collectP14os_aio_array_tmm () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/os/os0file.cc:5098
#3  0x000000001092ba6c in ._Z19os_aio_linux_handlemPP10fil_node_tPPvPmS4_ () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/os/os0file.cc:5244
#4  0x0000000010b71118 in ._Z12fil_aio_waitm () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/fil/fil0fil.cc:5662
#5  0x0000000010a2edb0 in .io_handler_thread () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/srv/srv0start.cc:499
#6  0x00000fff8aafc548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0
#7  0x00000fff8a343c90 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:106
 
Thread 2 (Thread 27310):
#0  0x00000fff8ab025a4 in .pthread_cond_timedwait () from /opt/at7.0/lib64/power7/libpthread.so.0
#1  0x0000000010d2c238 in .safe_cond_timedwait ()
#2  0x00000000107c5598 in .inline_mysql_cond_timedwait ()
#3  0x00000000107c5bb8 in .my_service_thread_sleep ()
#4  0x00000000107b63dc in .ma_checkpoint_background ()
#5  0x00000fff8aafc548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0
#6  0x00000fff8a343c90 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:106
 
Thread 1 (Thread 27698):
#0  0x00000fff8ab03a40 in .pthread_kill () from /opt/at7.0/lib64/power7/libpthread.so.0
#1  0x0000000010d251d0 in .my_write_core ()
#2  0x000000001050fa48 in .handle_fatal_signal () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/signal_handler.cc:262
#3  <signal handler called>
#4  0x00000fff8a26f8f0 in .raise () from /opt/at7.0/lib64/power7/libc.so.6
#5  0x00000fff8a2718f4 in .abort () from /opt/at7.0/lib64/power7/libc.so.6
#6  0x0000000010d2bae0 in .safe_mutex_unlock ()
#7  0x00000fff85e34dc4 in .inline_mysql_mutex_unlock () at /home/buildbot/maria-slave/power8-vlp01-debug/build/include/mysql/psi/mysql_thread.h:761
#8  0x00000fff85e38984 in ._Z17spider_db_errornoP14st_spider_conn () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/spider/spd_db_conn.cc:671
#9  0x00000fff85efe8bc in ._ZN15spider_db_mysql6commitEPi () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/spider/spd_db_mysql.cc:2087
#10 0x00000fff85e3b09c in ._Z16spider_db_commitP14st_spider_conn () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/spider/spd_db_conn.cc:1148
#11 0x00000fff85e31580 in ._Z13spider_commitP10handlertonP3THDb () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/spider/spd_trx.cc:3327
#12 0x0000000010514548 in ._ZL18commit_one_phase_2P3THDbP9THD_TRANSb () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/handler.cc:1527
#13 0x00000000105143e8 in ._Z19ha_commit_one_phaseP3THDb () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/handler.cc:1508
#14 0x0000000010513c38 in ._Z15ha_commit_transP3THDb () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/handler.cc:1385
#15 0x000000001040f35c in ._Z17trans_commit_stmtP3THD () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/transaction.cc:394
#16 0x00000000102914e8 in ._Z21mysql_execute_commandP3THD () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/sql_parse.cc:5126
#17 0x0000000010294bb0 in ._Z11mysql_parseP3THDPcjP12Parser_state () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/sql_parse.cc:6407
#18 0x000000001028655c in ._Z16dispatch_command19enum_server_commandP3THDPcj () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/sql_parse.cc:1299
#19 0x000000001028572c in ._Z10do_commandP3THD () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/sql_parse.cc:996
#20 0x00000000103f3af0 in ._Z24do_handle_one_connectionP3THD () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/sql_connect.cc:1379
#21 0x00000000103f3754 in .handle_one_connection () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/sql_connect.cc:1293
#22 0x00000fff8aafc548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0
#23 0x00000fff8a343c90 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:106
 
Compressed file /home/buildbot/maria-slave/power8-vlp01-debug/build/mysql-test/var/log/bg.spider_fixes/mysqld.1.1/data/core.27308



 Comments   
Comment by Daniel Black [ 2016-06-22 ]

Last Built Revision: Revision 21479a6bb3cc37ffa1fb4f53a0ab71d22c7ad445 (origin/10.0)

ppc64le - ubuntu-16.04:

+ cmake /source -DCMAKE_BUILD_TYPE=Release -DBUILD_CONFIG=mysql_release -DDOWNLOAD_BOOST=1 -DWITH_BOOST=build_boost -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ '-DCMAKE_C_FLAGS_RELWITHDEBINFO=-m64 -O3 -g -mtune=native' '-DCMAKE_CXX_FLAGS_RELWITHDEBINFO=-m64 -O3 -g -mtune=native' '-DCMAKE_C_FLAGS=-m64 -O3 -g -mtune=native' '-DCMAKE_CXX_FLAGS=-m64 -O3 -g -mtune=native' -DWITH_SSL=system -DCMAKE_INSTALL_PREFIX=/mariadb--10.0 -DPLUGIN_OQGRAPH=NO -DWITHOUT_OQGRAPH=1
-- Running cmake version 3.5.1
-- The C compiler identification is Clang 3.8.0
-- The CXX compiler identification is Clang 3.8.0

output:

spider/bg.spider_fixes                   w2 [ fail ]
        Test ended at 2016-06-21 20:24:37
 
CURRENT_TEST: spider/bg.spider_fixes
mysqltest: At line 1120: query 'SELECT MAX(id) FROM t1' failed: 1156: Got packets out of order
 
The result from queries just before the failure was:
< snip >
SET INSERT_ID=5000;
INSERT INTO t1 () VALUES ();
SELECT LAST_INSERT_ID();
LAST_INSERT_ID()
5000
SELECT MAX(id) FROM t1;
MAX(id)
6216
INSERT INTO t1 (id) VALUES (10000);
SELECT LAST_INSERT_ID();
LAST_INSERT_ID()
5000
SELECT MAX(id) FROM t1;
MAX(id)
10000
INSERT INTO t1 (id) VALUES (1000);
SELECT LAST_INSERT_ID();
LAST_INSERT_ID()
5000
SELECT MAX(id) FROM t1;
 
More results from queries before failure can be found in /build/build/mysql-test/var/2/log/spider_fixes.log
 
Warnings from just before the error:
Error 1156 Got packets out of order 
Error 12701 Remote MySQL server has gone away

Comment by Elena Stepanova [ 2016-08-21 ]

It was also observed on spider/bg.direct_aggregate:
http://buildbot.askmonty.org/buildbot/builders/kvm-deb-jessie-x86/builds/1029/steps/test_5/logs/stdio

spider/bg.direct_aggregate               w2 [ fail ]
        Test ended at 2016-07-18 12:10:22
 
CURRENT_TEST: spider/bg.direct_aggregate
mysqltest: At line 134: query 'SELECT MIN(a) FROM ta_l' failed: 1156: Got packets out of order
 
The result from queries just before the failure was:
< snip >
(4, 'd', '2003-11-30 05:01:03'),
(5, 'c', '2001-12-31 23:59:59');
 
direct_aggregating test
SHOW STATUS LIKE 'Spider_direct_aggregate';
Variable_name	Value
Spider_direct_aggregate	0
SELECT COUNT(*) FROM ta_l;
COUNT(*)
5
SHOW STATUS LIKE 'Spider_direct_aggregate';
Variable_name	Value
Spider_direct_aggregate	0
SELECT MAX(a) FROM ta_l;
MAX(a)
5
SHOW STATUS LIKE 'Spider_direct_aggregate';
Variable_name	Value
Spider_direct_aggregate	0
SELECT MIN(a) FROM ta_l;
 
More results from queries before failure can be found in /dev/shm/var/2/log/direct_aggregate.log
 
Warnings from just before the error:
Error 1156 Got packets out of order 
Error 12701 Remote MySQL server has gone away

Comment by Elena Stepanova [ 2016-08-21 ]

Yet another variation, might also be related (logs are not available):

0.1	825427fa9f4fa5c56301f392d10d254f	kvm-deb-wheezy-amd64	2016-06-03 12:30:43	5824	nm	Normal run, no --ps-protocol	spider/bg.direct_aggregate_part		
spider/bg.direct_aggregate_part          w2 [ fail ]
        Test ended at 2016-06-03 05:44:29
 
CURRENT_TEST: spider/bg.direct_aggregate_part
mysqltest: At line 138: query 'SELECT MAX(a) FROM ta_l2' failed: 1159: Got timeout reading communication packets
 
The result from queries just before the failure was:
< snip >
1
1
 
with partition test
CREATE TABLE ta_l2 (
a INT,
b CHAR(1),
c DATETIME,
PRIMARY KEY(a)
) MASTER_1_ENGINE MASTER_1_COMMENT2_P_2_1
SHOW STATUS LIKE 'Spider_direct_aggregate';
Variable_name	Value
Spider_direct_aggregate	0
SELECT COUNT(*) FROM ta_l2;
COUNT(*)
5
SHOW STATUS LIKE 'Spider_direct_aggregate';
Variable_name	Value
Spider_direct_aggregate	0
SELECT MAX(a) FROM ta_l2;
 
More results from queries before failure can be found in /run/shm/var/2/log/direct_aggregate_part.log
 
Warnings from just before the error:
Error 1159 Got timeout reading communication packets 
Error 1159
 
 - saving '/run/shm/var/2/log/spider/bg.direct_aggregate_part/' to '/run/shm/var/log/bg.direct_aggregate_part/'
 
Retrying test spider/bg.direct_aggregate_part, attempt(2/3)...

Comment by Elena Stepanova [ 2016-11-20 ]

Fresh with logs (not that they help):
http://buildbot.askmonty.org/buildbot/builders/kvm-deb-yakkety-amd64/builds/148/steps/test_5/logs/stdio

spider/bg.direct_aggregate               w3 [ fail ]
        Test ended at 2016-11-16 10:38:10
 
CURRENT_TEST: spider/bg.direct_aggregate
mysqltest: At line 132: query 'SELECT MAX(a) FROM ta_l' failed: 1159: Got timeout reading communication packets
 
The result from queries just before the failure was:
< snip >
PRIMARY KEY(a)
) MASTER_1_ENGINE MASTER_1_CHARSET MASTER_1_COMMENT_2_1
INSERT INTO ta_l (a, b, c) VALUES
(1, 'a', '2008-08-01 10:21:39'),
(2, 'b', '2000-01-01 00:00:00'),
(3, 'e', '2007-06-04 20:03:11'),
(4, 'd', '2003-11-30 05:01:03'),
(5, 'c', '2001-12-31 23:59:59');
 
direct_aggregating test
SHOW STATUS LIKE 'Spider_direct_aggregate';
Variable_name	Value
Spider_direct_aggregate	0
SELECT COUNT(*) FROM ta_l;
COUNT(*)
5
SHOW STATUS LIKE 'Spider_direct_aggregate';
Variable_name	Value
Spider_direct_aggregate	0
SELECT MAX(a) FROM ta_l;
 
More results from queries before failure can be found in /dev/shm/var/3/log/direct_aggregate.log
 
Warnings from just before the error:
Error 1159 Got timeout reading communication packets 
Error 1159

Comment by Elena Stepanova [ 2019-05-07 ]

http://buildbot.askmonty.org/buildbot/builders/kvm-fulltest2/builds/17126

10.2 e124ff17

spider/bg.spider3_fixes_part             w2 [ fail ]
        Test ended at 2019-04-08 18:03:44
 
CURRENT_TEST: spider/bg.spider3_fixes_part
mysqltest: At line 253: query 'SELECT MAX(id) FROM t1' failed: 2013: Lost connection to MySQL server during query
 
The result from queries just before the failure was:
< snip >
INSERT INTO t1 () VALUES ();
SELECT LAST_INSERT_ID();
LAST_INSERT_ID()
5000
SELECT MAX(id) FROM t1;
MAX(id)
6216
MASTER_1_AUTO_INCREMENT_OFFSET4
INSERT INTO t2 () VALUES ();
SELECT LAST_INSERT_ID();
LAST_INSERT_ID()
6993
SELECT MAX(id) FROM t2;
MAX(id)
6993
INSERT INTO t1 (id) VALUES (10000);
SELECT LAST_INSERT_ID();
LAST_INSERT_ID()
6993
SELECT MAX(id) FROM t1;
 
More results from queries before failure can be found in /mnt/buildbot/build/mariadb-10.2.24/mysql-test/var/2/log/spider3_fixes_part.log
 
 
Server [mysqld.1.1 - pid: 1269, winpid: 1269, exit: 256] failed during test run
Server log from this test:
----------SERVER LOG START-----------
2019-04-08 18:03:36 3068504000 [Note] /mnt/buildbot/build/mariadb-10.2.24/sql/mysqld (mysqld 10.2.24-MariaDB-debug-log) starting as process 1271 ...
2019-04-08 18:03:36 3068504000 [Warning] Could not increase number of max_open_files to more than 1024 (request: 4186)
2019-04-08 18:03:36 3068504000 [Note] Plugin 'SEQUENCE' is disabled.
2019-04-08 18:03:36 3068504000 [Note] InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!!
2019-04-08 18:03:36 3068504000 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2019-04-08 18:03:36 3068504000 [Note] InnoDB: Uses event mutexes
2019-04-08 18:03:36 3068504000 [Note] InnoDB: Compressed tables use zlib 1.2.8
2019-04-08 18:03:36 3068504000 [Note] InnoDB: Using Linux native AIO
2019-04-08 18:03:36 3068504000 [Note] InnoDB: Number of pools: 1
2019-04-08 18:03:36 3068504000 [Note] InnoDB: Using generic crc32 instructions
2019-04-08 18:03:36 3068504000 [Note] InnoDB: Initializing buffer pool, total size = 8M, instances = 1, chunk size = 8M
2019-04-08 18:03:36 3068504000 [Note] InnoDB: Completed initialization of buffer pool
2019-04-08 18:03:37 2969283392 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2019-04-08 18:03:37 3068504000 [Note] InnoDB: Highest supported file format is Barracuda.
2019-04-08 18:03:37 3068504000 [Note] InnoDB: 128 out of 128 rollback segments are active.
2019-04-08 18:03:37 3068504000 [Note] InnoDB: Creating shared tablespace for temporary tables
2019-04-08 18:03:37 3068504000 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2019-04-08 18:03:37 3068504000 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2019-04-08 18:03:38 3068504000 [Note] InnoDB: Waiting for purge to start
2019-04-08 18:03:38 3068504000 [Note] InnoDB: 5.7.25 started; log sequence number 1633840
2019-04-08 18:03:38 2868902720 [Note] InnoDB: Loading buffer pool(s) from /dev/shm/var_auto_SSQJ/2/mysqld.1.1/data/ib_buffer_pool
2019-04-08 18:03:38 3068504000 [Note] Plugin 'INNODB_SYS_DATAFILES' is disabled.
2019-04-08 18:03:38 3068504000 [Note] Plugin 'INNODB_SYS_TABLESTATS' is disabled.
2019-04-08 18:03:38 3068504000 [Note] Plugin 'INNODB_LOCKS' is disabled.
2019-04-08 18:03:38 3068504000 [Note] Plugin 'INNODB_TABLESPACES_SCRUBBING' is disabled.
2019-04-08 18:03:38 3068504000 [Note] Plugin 'INNODB_CMPMEM' is disabled.
2019-04-08 18:03:38 3068504000 [Note] Plugin 'INNODB_CMP_PER_INDEX' is disabled.
2019-04-08 18:03:38 3068504000 [Note] Plugin 'INNODB_CMP' is disabled.
2019-04-08 18:03:38 3068504000 [Note] Plugin 'INNODB_FT_DELETED' is disabled.
2019-04-08 18:03:38 3068504000 [Note] Plugin 'INNODB_CMP_RESET' is disabled.
2019-04-08 18:03:38 3068504000 [Note] Plugin 'INNODB_LOCK_WAITS' is disabled.
2019-04-08 18:03:38 3068504000 [Note] Plugin 'INNODB_TABLESPACES_ENCRYPTION' is disabled.
2019-04-08 18:03:38 3068504000 [Note] Plugin 'INNODB_BUFFER_PAGE_LRU' is disabled.
2019-04-08 18:03:38 3068504000 [Note] Plugin 'INNODB_SYS_FIELDS' is disabled.
2019-04-08 18:03:38 3068504000 [Note] Plugin 'INNODB_CMPMEM_RESET' is disabled.
2019-04-08 18:03:38 3068504000 [Note] Plugin 'FEEDBACK' is disabled.
2019-04-08 18:03:38 3068504000 [Note] Plugin 'INNODB_SYS_COLUMNS' is disabled.
2019-04-08 18:03:38 3068504000 [Note] Plugin 'INNODB_FT_INDEX_TABLE' is disabled.
2019-04-08 18:03:38 3068504000 [Note] Plugin 'INNODB_CMP_PER_INDEX_RESET' is disabled.
2019-04-08 18:03:38 2868902720 [Note] InnoDB: Buffer pool(s) load completed at 190408 18:03:38
2019-04-08 18:03:38 3068504000 [Note] Plugin 'user_variables' is disabled.
2019-04-08 18:03:38 3068504000 [Note] Plugin 'INNODB_FT_INDEX_CACHE' is disabled.
2019-04-08 18:03:38 3068504000 [Note] Plugin 'INNODB_SYS_FOREIGN_COLS' is disabled.
2019-04-08 18:03:38 3068504000 [Note] Plugin 'INNODB_FT_BEING_DELETED' is disabled.
2019-04-08 18:03:38 3068504000 [Note] Plugin 'INNODB_BUFFER_POOL_STATS' is disabled.
2019-04-08 18:03:38 3068504000 [Note] Plugin 'INNODB_TRX' is disabled.
2019-04-08 18:03:38 3068504000 [Note] Plugin 'INNODB_SYS_FOREIGN' is disabled.
2019-04-08 18:03:38 3068504000 [Note] Plugin 'INNODB_SYS_TABLES' is disabled.
2019-04-08 18:03:38 3068504000 [Note] Plugin 'INNODB_FT_DEFAULT_STOPWORD' is disabled.
2019-04-08 18:03:38 3068504000 [Note] Plugin 'INNODB_FT_CONFIG' is disabled.
2019-04-08 18:03:38 3068504000 [Note] Plugin 'INNODB_BUFFER_PAGE' is disabled.
2019-04-08 18:03:38 3068504000 [Note] Plugin 'INNODB_SYS_TABLESPACES' is disabled.
2019-04-08 18:03:38 3068504000 [Note] Plugin 'INNODB_METRICS' is disabled.
2019-04-08 18:03:38 3068504000 [Note] Plugin 'INNODB_SYS_INDEXES' is disabled.
2019-04-08 18:03:38 3068504000 [Note] Plugin 'INNODB_SYS_VIRTUAL' is disabled.
2019-04-08 18:03:38 3068504000 [Note] Plugin 'INNODB_MUTEXES' is disabled.
2019-04-08 18:03:38 3068504000 [Note] Plugin 'INNODB_SYS_SEMAPHORE_WAITS' is disabled.
2019-04-08 18:03:38 3068504000 [Warning] /mnt/buildbot/build/mariadb-10.2.24/sql/mysqld: unknown option '--loose-pam-debug'
2019-04-08 18:03:38 3068504000 [Warning] /mnt/buildbot/build/mariadb-10.2.24/sql/mysqld: unknown variable 'loose_handlersocket_port=20000'
2019-04-08 18:03:38 3068504000 [Warning] /mnt/buildbot/build/mariadb-10.2.24/sql/mysqld: unknown variable 'loose_handlersocket_port_wr=20001'
2019-04-08 18:03:38 3068504000 [Warning] /mnt/buildbot/build/mariadb-10.2.24/sql/mysqld: unknown variable 'loose_handlersocket_threads=2'
2019-04-08 18:03:38 3068504000 [Warning] /mnt/buildbot/build/mariadb-10.2.24/sql/mysqld: unknown variable 'loose_handlersocket_threads_wr=1'
2019-04-08 18:03:38 3068504000 [Warning] /mnt/buildbot/build/mariadb-10.2.24/sql/mysqld: unknown variable 'loose_handlersocket_support_merge_table=0'
2019-04-08 18:03:38 3068504000 [Warning] /mnt/buildbot/build/mariadb-10.2.24/sql/mysqld: unknown variable 'loose_handlersocket_direct_update_mode=2'
2019-04-08 18:03:38 3068504000 [Warning] /mnt/buildbot/build/mariadb-10.2.24/sql/mysqld: unknown variable 'loose_handlersocket_unlimited_boundary=65536'
2019-04-08 18:03:38 3068504000 [Warning] /mnt/buildbot/build/mariadb-10.2.24/sql/mysqld: unknown variable 'loose_handlersocket_bulk_insert=0'
2019-04-08 18:03:38 3068504000 [Warning] /mnt/buildbot/build/mariadb-10.2.24/sql/mysqld: unknown variable 'loose_handlersocket_bulk_insert_timeout=0'
2019-04-08 18:03:38 3068504000 [Warning] /mnt/buildbot/build/mariadb-10.2.24/sql/mysqld: unknown variable 'loose_handlersocket_general_log=1'
2019-04-08 18:03:38 3068504000 [Warning] /mnt/buildbot/build/mariadb-10.2.24/sql/mysqld: unknown variable 'loose_handlersocket_timeout=30'
2019-04-08 18:03:38 3068504000 [Warning] /mnt/buildbot/build/mariadb-10.2.24/sql/mysqld: unknown variable 'loose_handlersocket_close_table_interval=2'
2019-04-08 18:03:38 3068504000 [Note] Server socket created on IP: '127.0.0.1'.
2019-04-08 18:03:38 3068504000 [Note] /mnt/buildbot/build/mariadb-10.2.24/sql/mysqld: ready for connections.
Version: '10.2.24-MariaDB-debug-log'  socket: '/mnt/buildbot/build/mariadb-10.2.24/mysql-test/var/tmp/2/mysqld.1.1.sock'  port: 16000  Source distribution

safe_mutex: Trying to unlock mutex conn->mta_conn_mutex at /home/buildbot/buildbot/build/mariadb-10.2.24/storage/spider/spd_db_mysql.cc, line 2094 that was locked by another thread at: /home/buildbot/buildbot/build/mariadb-10.2.24/storage/spider/spd_conn.cc, line: 2402
190408 18:03:40 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
Server version: 10.2.24-MariaDB-debug-log
key_buffer_size=1048576
read_buffer_size=131072
max_used_connections=3
max_threads=153
thread_count=12
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 61893 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0xab2007b8
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0xab3f025c thread_stack 0x49000
/mnt/buildbot/build/mariadb-10.2.24/sql/mysqld(my_print_stacktrace+0x33)[0x80d120ae]
/mnt/buildbot/build/mariadb-10.2.24/sql/mysqld(handle_fatal_signal+0x3e1)[0x805f9c42]
addr2line: '': No such file
[0xb773ac14]
[0xb773ac31]
/lib/i386-linux-gnu/libc.so.6(gsignal+0x39)[0xb7018e89]
/lib/i386-linux-gnu/libc.so.6(abort+0x157)[0xb701a3e7]
/mnt/buildbot/build/mariadb-10.2.24/sql/mysqld(safe_mutex_unlock+0x142)[0x80d16aa3]
/mnt/buildbot/build/mariadb-10.2.24/mysql-test/var/2/plugins/ha_spider.so(+0xe237e)[0xa8d8737e]
psi/mysql_thread.h:747(inline_mysql_mutex_unlock)[0xa8d8f888]
spider/spd_db_mysql.cc:2094(spider_db_mysql::commit(int*))[0xa8cedff9]
spider/spd_db_conn.cc:1196(spider_db_commit(st_spider_conn*))[0xa8ce5e95]
sql/handler.cc:1570(commit_one_phase_2(THD*, bool, THD_TRANS*, bool))[0x805fd645]
sql/handler.cc:1551(ha_commit_one_phase(THD*, bool))[0x805fd543]
sql/handler.cc:1418(ha_commit_trans(THD*, bool))[0x805fce09]
sql/transaction.cc:510(trans_commit_stmt(THD*))[0x804e75d4]
sql/sql_parse.cc:6296(mysql_execute_command(THD*))[0x803a29f1]
sql/sql_parse.cc:8013(mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool))[0x803a7063]
sql/sql_parse.cc:1832(dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool))[0x80393c29]
sql/sql_parse.cc:1386(do_command(THD*))[0x80392434]
sql/sql_connect.cc:1335(do_handle_one_connection(CONNECT*))[0x804d21b5]
sql/sql_connect.cc:1241(handle_one_connection)[0x804d1f2d]
/lib/i386-linux-gnu/libpthread.so.0(+0x62b5)[0xb73932b5]
/lib/i386-linux-gnu/libc.so.6(clone+0x6e)[0xb70d416e]
 
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0xab20e168): SELECT MAX(id) FROM t1
Connection ID (thread ID): 11
Status: NOT_KILLED
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on

Generated at Thu Feb 08 07:16:59 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.