main.connect2 failed in buildbot, server crash

main.connect2                            w1 [ fail ]

        Test ended at 2017-08-18 03:28:12

CURRENT_TEST: main.connect2

mysqltest: At line 35: query 'connect con1,localhost,root,,test,$MASTER_EXTRA_PORT,' failed: 2013: Lost connection to MySQL server at 'handshake: reading inital communication packet', system error: 11

The result from queries just before the failure was:

< snip >

1

disconnect con1;

connection default;

set global debug_dbug='+d,simulate_failed_connection_1';

connect(localhost,root,,test,MASTER_PORT,MASTER_SOCKET);

connect con1,localhost,root,,test,,;

Got one of the listed errors

connection default;

set global debug_dbug=@old_debug;

set global debug_dbug='+d,simulate_failed_connection_2';

connect(localhost,root,,test,MASTER_PORT,MASTER_SOCKET);

connect con1,localhost,root,,test,,;

Got one of the listed errors

connection default;

set global debug_dbug=@old_debug;

connect con1,localhost,root,,test,,;

select 1;

1

1

disconnect con1;

More results from queries before failure can be found in /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/mysql-test/var/1/log/connect2.log

Server [mysqld.1 - pid: 22381, winpid: 22381, exit: 256] failed during test run

Server log from this test:

----------SERVER LOG START-----------

2017-08-18  3:28:11 70367216113808 [Note] /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/mysqld (mysqld 10.2.8-MariaDB-debug-log) starting as process 22382 ...

2017-08-18  3:28:11 70367216113808 [Warning] Could not increase number of max_open_files to more than 1024 (request: 4163)

2017-08-18  3:28:11 70367216113808 [Note] Plugin 'SEQUENCE' is disabled.

2017-08-18  3:28:11 70367216113808 [Note] Plugin 'partition' is disabled.

2017-08-18  3:28:11 70367216113808 [Note] Plugin 'InnoDB' is disabled.

2017-08-18  3:28:11 70367216113808 [Note] Plugin 'INNODB_SYS_DATAFILES' is disabled.

2017-08-18  3:28:11 70367216113808 [Note] Plugin 'INNODB_SYS_TABLESTATS' is disabled.

2017-08-18  3:28:11 70367216113808 [Note] Plugin 'INNODB_LOCKS' is disabled.

2017-08-18  3:28:11 70367216113808 [Note] Plugin 'INNODB_MUTEXES' is disabled.

2017-08-18  3:28:11 70367216113808 [Note] Plugin 'INNODB_CMPMEM' is disabled.

2017-08-18  3:28:11 70367216113808 [Note] Plugin 'INNODB_CMP_PER_INDEX' is disabled.

2017-08-18  3:28:11 70367216113808 [Note] Plugin 'INNODB_CMP' is disabled.

2017-08-18  3:28:11 70367216113808 [Note] Plugin 'INNODB_FT_DELETED' is disabled.

2017-08-18  3:28:11 70367216113808 [Note] Plugin 'INNODB_CMP_RESET' is disabled.

2017-08-18  3:28:11 70367216113808 [Note] Plugin 'INNODB_LOCK_WAITS' is disabled.

2017-08-18  3:28:11 70367216113808 [Note] Plugin 'INNODB_TABLESPACES_ENCRYPTION' is disabled.

2017-08-18  3:28:11 70367216113808 [Note] Plugin 'INNODB_BUFFER_PAGE_LRU' is disabled.

2017-08-18  3:28:11 70367216113808 [Note] Plugin 'INNODB_SYS_FIELDS' is disabled.

2017-08-18  3:28:11 70367216113808 [Note] Plugin 'INNODB_CMPMEM_RESET' is disabled.

2017-08-18  3:28:11 70367216113808 [Note] Plugin 'FEEDBACK' is disabled.

2017-08-18  3:28:11 70367216113808 [Note] Plugin 'INNODB_SYS_COLUMNS' is disabled.

2017-08-18  3:28:11 70367216113808 [Note] Plugin 'INNODB_FT_INDEX_TABLE' is disabled.

2017-08-18  3:28:11 70367216113808 [Note] Plugin 'INNODB_CMP_PER_INDEX_RESET' is disabled.

2017-08-18  3:28:11 70367216113808 [Note] Plugin 'user_variables' is disabled.

2017-08-18  3:28:11 70367216113808 [Note] Plugin 'INNODB_FT_INDEX_CACHE' is disabled.

2017-08-18  3:28:11 70367216113808 [Note] Plugin 'INNODB_SYS_FOREIGN_COLS' is disabled.

2017-08-18  3:28:11 70367216113808 [Note] Plugin 'INNODB_FT_BEING_DELETED' is disabled.

2017-08-18  3:28:11 70367216113808 [Note] Plugin 'INNODB_BUFFER_POOL_STATS' is disabled.

2017-08-18  3:28:11 70367216113808 [Note] Plugin 'INNODB_TRX' is disabled.

2017-08-18  3:28:11 70367216113808 [Note] Plugin 'INNODB_SYS_FOREIGN' is disabled.

2017-08-18  3:28:11 70367216113808 [Note] Plugin 'INNODB_SYS_TABLES' is disabled.

2017-08-18  3:28:11 70367216113808 [Note] Plugin 'INNODB_FT_DEFAULT_STOPWORD' is disabled.

2017-08-18  3:28:11 70367216113808 [Note] Plugin 'INNODB_FT_CONFIG' is disabled.

2017-08-18  3:28:11 70367216113808 [Note] Plugin 'INNODB_BUFFER_PAGE' is disabled.

2017-08-18  3:28:11 70367216113808 [Note] Plugin 'INNODB_SYS_TABLESPACES' is disabled.

2017-08-18  3:28:11 70367216113808 [Note] Plugin 'INNODB_METRICS' is disabled.

2017-08-18  3:28:11 70367216113808 [Note] Plugin 'INNODB_SYS_INDEXES' is disabled.

2017-08-18  3:28:11 70367216113808 [Note] Plugin 'INNODB_SYS_VIRTUAL' is disabled.

2017-08-18  3:28:11 70367216113808 [Note] Plugin 'INNODB_TABLESPACES_SCRUBBING' is disabled.

2017-08-18  3:28:11 70367216113808 [Note] Plugin 'INNODB_SYS_SEMAPHORE_WAITS' is disabled.

2017-08-18  3:28:11 70367216113808 [Warning] /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/mysqld: unknown option '--loose-pam-debug'

2017-08-18  3:28:11 70367216113808 [Note] Server socket created on IP: '127.0.0.1'.

2017-08-18  3:28:11 70367216113808 [Note] Server socket created on IP: '127.0.0.1'.

2017-08-18  3:28:11 70367216113808 [Note] Reading of all Master_info entries succeded

2017-08-18  3:28:11 70367216113808 [Note] Added new Master_info '' to hash table

2017-08-18  3:28:11 70367216113808 [Note] /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/mysqld: ready for connections.

Version: '10.2.8-MariaDB-debug-log'  socket: '/home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/mysql-test/var/tmp/1/mysqld.1.sock'  port: 16020  Source distribution

170818  3:28:11 [ERROR] mysqld got signal 11 ;

This could be because you hit a bug. It is also possible that this binary

or one of the libraries it was linked against is corrupt, improperly built,

or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help

diagnose the problem, but since we have already crashed, 

something is definitely wrong and this may fail.

Server version: 10.2.8-MariaDB-debug-log

key_buffer_size=1048576

read_buffer_size=131072

max_used_connections=3

max_threads=65538

thread_count=3

It is possible that mysqld could use up to 

key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 25170958 K  bytes of memory

Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x3fff7c000cd0

Attempting backtrace. You can use the following information to find out

where mysqld died. If you see no messages after this, something went

terribly wrong...

stack_bottom = 0x3fff9d927450 thread_stack 0x49000

/home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/mysqld(my_print_stacktrace-0xb235fc)[0x4ecb8794]

/home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/mysqld(handle_fatal_signal-0x14f8b6c)[0x4e2270dc]

linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0x3fffa4e40478]

/opt/at8.0/lib64/power8/libc.so.6(+0x1111e0)[0x3fffa45a11e0]

[0x3fff9d9264a0]

/opt/at8.0/lib64/power8/libc.so.6(fnmatch-0xe6f70)[0x3fffa45a29c8]

/home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/mysqld(+0x1641200)[0x4ecf1200]

/home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/mysqld(_db_keyword_-0xaee39c)[0x4ecf196c]

/home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/mysqld(_db_pargs_-0xaef4fc)[0x4ecf06d4]

/home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/mysqld(my_free-0xb2cbfc)[0x4ecae8ac]

/home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/mysqld(safe_mutex_free_deadlock_data-0xb1b90c)[0x4ecc0bec]

/home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/mysqld(safe_mutex_destroy-0xb1bab8)[0x4ecc0a28]

/home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/mysqld(+0x7b12bc)[0x4de612bc]

mysys/stacktrace.c:267(my_print_stacktrace)[0x4de682d4]

sql/signal_handler.cc:166(handle_fatal_signal)[0x4de6897c]

/home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/mysqld(+0xa645c8)[0x4e1145c8]

psi/mysql_thread.h:640(inline_mysql_mutex_destroy)[0x4e114204]

sql/threadpool_generic.cc:1581(worker_main)[0x4e1bfb28]

perfschema/pfs.cc:1862(pfs_spawn_thread)[0x4ec34234]

/opt/at8.0/lib64/power8/libpthread.so.0(+0xb9e0)[0x3fffa4e0b9e0]

/opt/at8.0/lib64/power8/libc.so.6(clone-0xaa8e8)[0x3fffa45e18d0]

Trying to get some variables.

Some pointers may be invalid and cause the dump to abort.

Query (0x0): 

Connection ID (thread ID): 5

Status: KILL_CONNECTION

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains

information that should help you find out what is causing the crash.

Writing a core file

----------SERVER LOG END-------------

 - saving '/home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/mysql-test/var/1/log/main.connect2/' to '/home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/mysql-test/var/log/main.connect2/'

 - found 'core.22382' (2/5)

Trying 'dbx' to get a backtrace

Trying 'gdb' to get a backtrace from coredump /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/mysql-test/var/log/main.connect2/mysqld.1/data/core.22382

Core generated by '/home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/mysqld'

Output from gdb follows. The first stack trace is from the failing thread.

The following stack traces are from all threads (so the failing one is

duplicated).

--------------------------

[New LWP 22397]

[New LWP 22384]

[New LWP 22387]

[New LWP 22398]

[New LWP 22382]

[New LWP 22386]

[New LWP 22390]

[New LWP 22383]

[New LWP 22394]

[New LWP 22385]

[New LWP 22395]

[New LWP 22391]

[New LWP 22396]

warning: Could not load shared library symbols for linux-vdso64.so.1.

Do you need "set solib-search-path" or "set sysroot"?

warning: File "/opt/at8.0/lib64/power8/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load:/usr/bin/mono-gdb.py".

To enable execution of this file add

	add-auto-load-safe-path /opt/at8.0/lib64/power8/libthread_db-1.0.so

line to your configuration file "/home/buildbot/.gdbinit".

To completely disable this security protection add

	set auto-load safe-path /

line to your configuration file "/home/buildbot/.gdbinit".

For more information about this security protection see the

"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:

	info "(gdb)Auto-loading safe path"

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.

warning: File "/opt/at8.0/lib64/power8/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load:/usr/bin/mono-gdb.py".

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.

Core was generated by `/home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/mysqld --default'.

Program terminated with signal 11, Segmentation fault.

#0  0x00003fffa4e14f00 in __pthread_kill (threadid=<optimized out>, signo=<optimized out>) at ../sysdeps/unix/sysv/linux/pthread_kill.c:60

60	../sysdeps/unix/sysv/linux/pthread_kill.c: No such file or directory.

#0  0x00003fffa4e14f00 in __pthread_kill (threadid=<optimized out>, signo=<optimized out>) at ../sysdeps/unix/sysv/linux/pthread_kill.c:60

#1  0x000000004ecb892c in my_write_core (sig=11) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/mysys/stacktrace.c:477

#2  0x000000004e227664 in handle_fatal_signal (sig=11) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/signal_handler.cc:299

#3  <signal handler called>

#4  internal_fnmatch (pattern=<optimized out>, string=string@entry=0x4f084ce8 "my", string_end=0x4f084cea "", no_leading_period=no_leading_period@entry=0, flags=flags@entry=0, ends=ends@entry=0x0, alloca_used=alloca_used@entry=0) at fnmatch_loop.c:62

#5  0x00003fffa45a29c8 in __fnmatch (pattern=0x8f8f8f8f8f8f8f98 <Address 0x8f8f8f8f8f8f8f98 out of bounds>, string=0x4f084ce8 "my", flags=<optimized out>) at fnmatch.c:458

#6  0x000000004ecf1200 in InList (linkp=0x8f8f8f8f8f8f8f8f, cp=0x4f084ce8 "my", exact_match=0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/dbug/dbug.c:1515

#7  0x000000004ecf196c in _db_keyword_ (cs=0x3fff7c000c00, keyword=0x4f084ce8 "my", strict=0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/dbug/dbug.c:1717

#8  0x000000004ecf06d4 in _db_pargs_ (_line_=210, keyword=0x4f084ce8 "my") at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/dbug/dbug.c:1236

#9  0x000000004ecae8ac in my_free (ptr=0x3fff7c006c50) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/mysys/my_malloc.c:210

#10 0x000000004ecc0bec in safe_mutex_free_deadlock_data (mp=0x3fff7c0025e0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/mysys/thr_mutex.c:666

#11 0x000000004ecc0a28 in safe_mutex_destroy (mp=0x3fff7c0025e0, file=0x4edd2518 "/home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/sql_class.cc", line=1597) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/mysys/thr_mutex.c:603

#12 0x000000004de612bc in inline_mysql_mutex_destroy (that=0x3fff7c0025e0, src_file=0x4edd2518 "/home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/sql_class.cc", src_line=1597) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/include/mysql/psi/mysql_thread.h:640

#13 0x000000004de682d4 in THD::~THD (this=0x3fff7c000cd0, __in_chrg=<optimized out>) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/sql_class.cc:1597

#14 0x000000004de6897c in THD::~THD (this=0x3fff7c000cd0, __in_chrg=<optimized out>) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/sql_class.cc:1645

#15 0x000000004e1145c8 in threadpool_remove_connection (thd=0x3fff7c000cd0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/threadpool_common.cc:302

#16 0x000000004e114204 in tp_callback (c=0x1001c6ae5a0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/threadpool_common.cc:216

#17 0x000000004e1bfb28 in worker_main (param=0x1001c694bb0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/threadpool_generic.cc:1581

#18 0x000000004ec34234 in pfs_spawn_thread (arg=0x3fff7c010cd0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/perfschema/pfs.cc:1862

#19 0x00003fffa4e0b9e0 in start_thread (arg=0x3fff9d928180) at pthread_create.c:312

#20 0x00003fffa45e18d0 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 13 (LWP 22396):

#0  0x00003fffa45e206c in .epoll_wait () at ../sysdeps/unix/syscall-template.S:81

#1  0x000000004e1bc7b0 in io_poll_wait (pollfd=11, native_events=0x3fff9d97f470, maxevents=1024, timeout_ms=-1) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/threadpool_generic.cc:320

#2  0x000000004e1bd620 in listener (current_thread=0x3fff9d987628, thread_group=0x1001c694bb0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/threadpool_generic.cc:788

#3  0x000000004e1be75c in get_event (current_thread=0x3fff9d987628, thread_group=0x1001c694bb0, abstime=0x3fff9d9875f8) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/threadpool_generic.cc:1233

#4  0x000000004e1bfae8 in worker_main (param=0x1001c694bb0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/threadpool_generic.cc:1577

#5  0x000000004ec34234 in pfs_spawn_thread (arg=0x1001c55fc70) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/perfschema/pfs.cc:1862

#6  0x00003fffa4e0b9e0 in start_thread (arg=0x3fff9d988180) at pthread_create.c:312

#7  0x00003fffa45e18d0 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 12 (LWP 22391):

#0  0x00003fffa45e206c in .epoll_wait () at ../sysdeps/unix/syscall-template.S:81

#1  0x000000004e1bc7b0 in io_poll_wait (pollfd=13, native_events=0x3fff9da9f470, maxevents=1024, timeout_ms=-1) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/threadpool_generic.cc:320

#2  0x000000004e1bd620 in listener (current_thread=0x3fff9daa7628, thread_group=0x1001c694eb0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/threadpool_generic.cc:788

#3  0x000000004e1be75c in get_event (current_thread=0x3fff9daa7628, thread_group=0x1001c694eb0, abstime=0x3fff9daa75f8) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/threadpool_generic.cc:1233

#4  0x000000004e1bfae8 in worker_main (param=0x1001c694eb0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/threadpool_generic.cc:1577

#5  0x000000004ec34234 in pfs_spawn_thread (arg=0x3fff80010cd0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/perfschema/pfs.cc:1862

#6  0x00003fffa4e0b9e0 in start_thread (arg=0x3fff9daa8180) at pthread_create.c:312

#7  0x00003fffa45e18d0 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 11 (LWP 22395):

#0  0x00003fffa45e206c in .epoll_wait () at ../sysdeps/unix/syscall-template.S:81

#1  0x000000004e1bc7b0 in io_poll_wait (pollfd=10, native_events=0x3fff9d9df470, maxevents=1024, timeout_ms=-1) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/threadpool_generic.cc:320

#2  0x000000004e1bd620 in listener (current_thread=0x3fff9d9e7628, thread_group=0x1001c694a30) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/threadpool_generic.cc:788

#3  0x000000004e1be75c in get_event (current_thread=0x3fff9d9e7628, thread_group=0x1001c694a30, abstime=0x3fff9d9e75f8) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/threadpool_generic.cc:1233

#4  0x000000004e1bfae8 in worker_main (param=0x1001c694a30) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/threadpool_generic.cc:1577

#5  0x000000004ec34234 in pfs_spawn_thread (arg=0x3fff78010d70) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/perfschema/pfs.cc:1862

#6  0x00003fffa4e0b9e0 in start_thread (arg=0x3fff9d9e8180) at pthread_create.c:312

#7  0x00003fffa45e18d0 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 10 (LWP 22385):

#0  0x00003fffa4e13960 in __pthread_cond_timedwait (cond=0x4f814700 <pool_timer+176>, mutex=0x4f814678 <pool_timer+40>, abstime=0x3fff9e3ce660) at pthread_cond_timedwait.c:199

#1  0x000000004ecc06a4 in safe_cond_timedwait (cond=0x4f814700 <pool_timer+176>, mp=0x4f814650 <pool_timer>, abstime=0x3fff9e3ce660, file=0x4ee9a110 "/home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/include/mysql/psi/mysql_thread.h", line=1175) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/mysys/thr_mutex.c:545

#2  0x000000004e1bc050 in inline_mysql_cond_timedwait (that=0x4f814700 <pool_timer+176>, mutex=0x4f814650 <pool_timer>, abstime=0x3fff9e3ce660, src_file=0x4ee9a1e0 "/home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/threadpool_generic.cc", src_line=637) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/include/mysql/psi/mysql_thread.h:1175

#3  0x000000004e1bcf0c in timer_thread (param=0x4f814650 <pool_timer>) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/threadpool_generic.cc:637

#4  0x000000004ec34234 in pfs_spawn_thread (arg=0x1001c6a53f0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/perfschema/pfs.cc:1862

#5  0x00003fffa4e0b9e0 in start_thread (arg=0x3fff9e3cf180) at pthread_create.c:312

#6  0x00003fffa45e18d0 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 9 (LWP 22394):

#0  0x00003fffa4e13960 in __pthread_cond_timedwait (cond=0x3fff9da47648, mutex=0x1001c694a58, abstime=0x3fff9da475f8) at pthread_cond_timedwait.c:199

#1  0x000000004ecc06a4 in safe_cond_timedwait (cond=0x3fff9da47648, mp=0x1001c694a30, abstime=0x3fff9da475f8, file=0x4ee9a110 "/home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/include/mysql/psi/mysql_thread.h", line=1175) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/mysys/thr_mutex.c:545

#2  0x000000004e1bc050 in inline_mysql_cond_timedwait (that=0x3fff9da47648, mutex=0x1001c694a30, abstime=0x3fff9da475f8, src_file=0x4ee9a1e0 "/home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/threadpool_generic.cc", src_line=1276) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/include/mysql/psi/mysql_thread.h:1175

#3  0x000000004e1be8b4 in get_event (current_thread=0x3fff9da47628, thread_group=0x1001c694a30, abstime=0x3fff9da475f8) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/threadpool_generic.cc:1276

#4  0x000000004e1bfae8 in worker_main (param=0x1001c694a30) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/threadpool_generic.cc:1577

#5  0x000000004ec34234 in pfs_spawn_thread (arg=0x1001c6ab3d0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/perfschema/pfs.cc:1862

#6  0x00003fffa4e0b9e0 in start_thread (arg=0x3fff9da48180) at pthread_create.c:312

#7  0x00003fffa45e18d0 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 8 (LWP 22383):

#0  0x00003fffa4e13960 in __pthread_cond_timedwait (cond=0x5009bca8 <COND_timer>, mutex=0x5009bc20 <LOCK_timer+40>, abstime=0x3fff9ff6e680) at pthread_cond_timedwait.c:199

#1  0x000000004ecc06a4 in safe_cond_timedwait (cond=0x5009bca8 <COND_timer>, mp=0x5009bbf8 <LOCK_timer>, abstime=0x3fff9ff6e680, file=0x4f086cb0 "/home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/include/mysql/psi/mysql_thread.h", line=1175) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/mysys/thr_mutex.c:545

#2  0x000000004ecc1f74 in inline_mysql_cond_timedwait (that=0x5009bca8 <COND_timer>, mutex=0x5009bbf8 <LOCK_timer>, abstime=0x3fff9ff6e680, src_file=0x4f086d20 "/home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/mysys/thr_timer.c", src_line=292) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/include/mysql/psi/mysql_thread.h:1175

#3  0x000000004ecc2fa0 in timer_handler (arg=0x0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/mysys/thr_timer.c:292

#4  0x000000004ec34234 in pfs_spawn_thread (arg=0x1001c56c860) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/perfschema/pfs.cc:1862

#5  0x00003fffa4e0b9e0 in start_thread (arg=0x3fff9ff6f180) at pthread_create.c:312

#6  0x00003fffa45e18d0 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 7 (LWP 22390):

#0  0x00003fffa4e13960 in __pthread_cond_timedwait (cond=0x3fff9db07648, mutex=0x1001c694ed8, abstime=0x3fff9db075f8) at pthread_cond_timedwait.c:199

#1  0x000000004ecc06a4 in safe_cond_timedwait (cond=0x3fff9db07648, mp=0x1001c694eb0, abstime=0x3fff9db075f8, file=0x4ee9a110 "/home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/include/mysql/psi/mysql_thread.h", line=1175) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/mysys/thr_mutex.c:545

#2  0x000000004e1bc050 in inline_mysql_cond_timedwait (that=0x3fff9db07648, mutex=0x1001c694eb0, abstime=0x3fff9db075f8, src_file=0x4ee9a1e0 "/home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/threadpool_generic.cc", src_line=1276) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/include/mysql/psi/mysql_thread.h:1175

#3  0x000000004e1be8b4 in get_event (current_thread=0x3fff9db07628, thread_group=0x1001c694eb0, abstime=0x3fff9db075f8) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/threadpool_generic.cc:1276

#4  0x000000004e1bfae8 in worker_main (param=0x1001c694eb0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/threadpool_generic.cc:1577

#5  0x000000004ec34234 in pfs_spawn_thread (arg=0x1001c6ac010) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/perfschema/pfs.cc:1862

#6  0x00003fffa4e0b9e0 in start_thread (arg=0x3fff9db08180) at pthread_create.c:312

#7  0x00003fffa45e18d0 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 6 (LWP 22386):

#0  0x00003fffa4e19554 in do_sigwait (sig=0x3fff9dbc75e0, set=<optimized out>) at ../sysdeps/unix/sysv/linux/sigwait.c:61

#1  __sigwait (set=<optimized out>, sig=0x3fff9dbc75e0) at ../sysdeps/unix/sysv/linux/sigwait.c:98

#2  0x000000004dd9746c in signal_hand (arg=0x0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/mysqld.cc:3499

#3  0x000000004ec34234 in pfs_spawn_thread (arg=0x1001c6a69e0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/perfschema/pfs.cc:1862

#4  0x00003fffa4e0b9e0 in start_thread (arg=0x3fff9dbc8180) at pthread_create.c:312

#5  0x00003fffa45e18d0 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 5 (LWP 22382):

#0  0x00003fffa45d1d10 in .__GI___poll () at ../sysdeps/unix/syscall-template.S:81

#1  0x000000004dd9f894 in handle_connections_sockets () at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/mysqld.cc:6577

#2  0x000000004dd9e648 in mysqld_main (argc=131, argv=0x1001c3dbff0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/mysqld.cc:6043

#3  0x000000004dd8d3e8 in main (argc=6, argv=0x3fffc7af93a8) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/main.cc:25

Thread 4 (LWP 22398):

#0  0x00003fffa4e13960 in __pthread_cond_timedwait (cond=0x3fff9d8c7648, mutex=0x1001c694a58, abstime=0x3fff9d8c75f8) at pthread_cond_timedwait.c:199

#1  0x000000004ecc06a4 in safe_cond_timedwait (cond=0x3fff9d8c7648, mp=0x1001c694a30, abstime=0x3fff9d8c75f8, file=0x4ee9a110 "/home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/include/mysql/psi/mysql_thread.h", line=1175) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/mysys/thr_mutex.c:545

#2  0x000000004e1bc050 in inline_mysql_cond_timedwait (that=0x3fff9d8c7648, mutex=0x1001c694a30, abstime=0x3fff9d8c75f8, src_file=0x4ee9a1e0 "/home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/threadpool_generic.cc", src_line=1276) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/include/mysql/psi/mysql_thread.h:1175

#3  0x000000004e1be8b4 in get_event (current_thread=0x3fff9d8c7628, thread_group=0x1001c694a30, abstime=0x3fff9d8c75f8) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/threadpool_generic.cc:1276

#4  0x000000004e1bfae8 in worker_main (param=0x1001c694a30) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/threadpool_generic.cc:1577

#5  0x000000004ec34234 in pfs_spawn_thread (arg=0x3fff78033f30) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/perfschema/pfs.cc:1862

#6  0x00003fffa4e0b9e0 in start_thread (arg=0x3fff9d8c8180) at pthread_create.c:312

#7  0x00003fffa45e18d0 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 3 (LWP 22387):

#0  0x00003fffa4e13470 in __pthread_cond_wait (cond=0x4f7ea688 <COND_slave_background>, mutex=0x4f7ed710 <LOCK_slave_background+40>) at pthread_cond_wait.c:188

#1  0x000000004ecc02d4 in safe_cond_wait (cond=0x4f7ea688 <COND_slave_background>, mp=0x4f7ed6e8 <LOCK_slave_background>, file=0x4edbfb60 "/home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/include/mysql/psi/mysql_thread.h", line=1138) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/mysys/thr_mutex.c:491

#2  0x000000004ddd48e4 in inline_mysql_cond_wait (that=0x4f7ea688 <COND_slave_background>, mutex=0x4f7ed6e8 <LOCK_slave_background>, src_file=0x4edc0598 "/home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/slave.cc", src_line=334) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/include/mysql/psi/mysql_thread.h:1138

#3  0x000000004ddd6048 in handle_slave_background (arg=0x0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/slave.cc:334

#4  0x000000004ec34234 in pfs_spawn_thread (arg=0x1001c74fdf0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/perfschema/pfs.cc:1862

#5  0x00003fffa4e0b9e0 in start_thread (arg=0x3fff9db68180) at pthread_create.c:312

#6  0x00003fffa45e18d0 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 2 (LWP 22384):

#0  0x00003fffa4e13960 in __pthread_cond_timedwait (cond=0x5008a440 <COND_checkpoint>, mutex=0x5008a3b8 <LOCK_checkpoint+40>, abstime=0x3fff9ebce570) at pthread_cond_timedwait.c:199

#1  0x000000004ecc06a4 in safe_cond_timedwait (cond=0x5008a440 <COND_checkpoint>, mp=0x5008a390 <LOCK_checkpoint>, abstime=0x3fff9ebce570, file=0x4f0475a8 "/home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/include/mysql/psi/mysql_thread.h", line=1175) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/mysys/thr_mutex.c:545

#2  0x000000004ea94774 in inline_mysql_cond_timedwait (that=0x5008a440 <COND_checkpoint>, mutex=0x5008a390 <LOCK_checkpoint>, abstime=0x3fff9ebce570, src_file=0x4f047628 "/home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/maria/ma_servicethread.c", src_line=116) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/include/mysql/psi/mysql_thread.h:1175

#3  0x000000004ea94ed0 in my_service_thread_sleep (control=0x4f51bdc0 <checkpoint_control>, sleep_time=29000000000) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/maria/ma_servicethread.c:115

#4  0x000000004ea83bf4 in ma_checkpoint_background (arg=0x1e) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/maria/ma_checkpoint.c:708

#5  0x000000004ec34234 in pfs_spawn_thread (arg=0x1001c670710) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/perfschema/pfs.cc:1862

#6  0x00003fffa4e0b9e0 in start_thread (arg=0x3fff9ebcf180) at pthread_create.c:312

#7  0x00003fffa45e18d0 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 1 (LWP 22397):

#0  0x00003fffa4e14f00 in __pthread_kill (threadid=<optimized out>, signo=<optimized out>) at ../sysdeps/unix/sysv/linux/pthread_kill.c:60

#1  0x000000004ecb892c in my_write_core (sig=11) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/mysys/stacktrace.c:477

#2  0x000000004e227664 in handle_fatal_signal (sig=11) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/signal_handler.cc:299

#3  <signal handler called>

#4  internal_fnmatch (pattern=<optimized out>, string=string@entry=0x4f084ce8 "my", string_end=0x4f084cea "", no_leading_period=no_leading_period@entry=0, flags=flags@entry=0, ends=ends@entry=0x0, alloca_used=alloca_used@entry=0) at fnmatch_loop.c:62

#5  0x00003fffa45a29c8 in __fnmatch (pattern=0x8f8f8f8f8f8f8f98 <Address 0x8f8f8f8f8f8f8f98 out of bounds>, string=0x4f084ce8 "my", flags=<optimized out>) at fnmatch.c:458

#6  0x000000004ecf1200 in InList (linkp=0x8f8f8f8f8f8f8f8f, cp=0x4f084ce8 "my", exact_match=0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/dbug/dbug.c:1515

#7  0x000000004ecf196c in _db_keyword_ (cs=0x3fff7c000c00, keyword=0x4f084ce8 "my", strict=0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/dbug/dbug.c:1717

#8  0x000000004ecf06d4 in _db_pargs_ (_line_=210, keyword=0x4f084ce8 "my") at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/dbug/dbug.c:1236

#9  0x000000004ecae8ac in my_free (ptr=0x3fff7c006c50) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/mysys/my_malloc.c:210

#10 0x000000004ecc0bec in safe_mutex_free_deadlock_data (mp=0x3fff7c0025e0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/mysys/thr_mutex.c:666

#11 0x000000004ecc0a28 in safe_mutex_destroy (mp=0x3fff7c0025e0, file=0x4edd2518 "/home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/sql_class.cc", line=1597) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/mysys/thr_mutex.c:603

#12 0x000000004de612bc in inline_mysql_mutex_destroy (that=0x3fff7c0025e0, src_file=0x4edd2518 "/home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/sql_class.cc", src_line=1597) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/include/mysql/psi/mysql_thread.h:640

#13 0x000000004de682d4 in THD::~THD (this=0x3fff7c000cd0, __in_chrg=<optimized out>) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/sql_class.cc:1597

#14 0x000000004de6897c in THD::~THD (this=0x3fff7c000cd0, __in_chrg=<optimized out>) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/sql_class.cc:1645

#15 0x000000004e1145c8 in threadpool_remove_connection (thd=0x3fff7c000cd0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/threadpool_common.cc:302

#16 0x000000004e114204 in tp_callback (c=0x1001c6ae5a0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/threadpool_common.cc:216

#17 0x000000004e1bfb28 in worker_main (param=0x1001c694bb0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/threadpool_generic.cc:1581

#18 0x000000004ec34234 in pfs_spawn_thread (arg=0x3fff7c010cd0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/perfschema/pfs.cc:1862

#19 0x00003fffa4e0b9e0 in start_thread (arg=0x3fff9d928180) at pthread_create.c:312

#20 0x00003fffa45e18d0 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96