Shutdown crash presumably due to master thread activity

Thread 1 (Thread 24994):

#0  0x00000fffa3a63a40 in .pthread_kill () from /opt/at7.0/lib64/power7/libpthread.so.0

#1  0x00000000109eebb8 in .my_write_core ()

#2  0x00000000103e3a3c in .handle_fatal_signal ()

#3  <signal handler called>

#4  0x00000fffa320f2f0 in .raise () from /opt/at7.0/lib64/power7/libc.so.6

#5  0x00000fffa32112f4 in .abort () from /opt/at7.0/lib64/power7/libc.so.6

#6  0x000000001079ef54 in .buf_flush_page_cleaner_thread () at /home/buildbot/maria-slave/p8-rhel6-bintar/build/storage/xtradb/buf/buf0flu.cc:2803

#7  0x00000fffa3a5c548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0

#8  0x00000fffa32e8454 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:104

innodb_fts.innodb_fts_stopword_charset 'xtradb' w2 [ fail ]  Found warnings/errors in server log file!

        Test ended at 2017-02-09 07:47:28

line

Attempting backtrace. You can use the following information to find out

^ Found warnings in /home/buildbot/maria-slave/p8-rhel6-bintar/build/mysql-test/var/2/log/mysqld.1.err

ok

 - saving '/home/buildbot/maria-slave/p8-rhel6-bintar/build/mysql-test/var/2/log/innodb_fts.innodb_fts_stopword_charset-xtradb/' to '/home/buildbot/maria-slave/p8-rhel6-bintar/build/mysql-test/var/log/innodb_fts.innodb_fts_stopword_charset-xtradb/'

 - found 'core.24981' (0/5)

Trying 'dbx' to get a backtrace

Trying 'gdb' to get a backtrace from coredump /home/buildbot/maria-slave/p8-rhel6-bintar/build/mysql-test/var/log/innodb_fts.innodb_fts_stopword_charset-xtradb/mysqld.1/data/core.24981

Core generated by '/home/buildbot/maria-slave/p8-rhel6-bintar/build/sql/mysqld'

Output from gdb follows. The first stack trace is from the failing thread.

The following stack traces are from all threads (so the failing one is

duplicated).

--------------------------

[New Thread 24994]

[New Thread 25000]

[New Thread 24992]

[New Thread 25032]

[New Thread 24981]

[New Thread 24982]

[New Thread 24983]

[New Thread 24984]

[New Thread 24985]

[New Thread 24987]

[New Thread 24986]

warning: Ignoring non-absolute filename: <linux-vdso64.so.1>

Missing separate debuginfo for linux-vdso64.so.1

Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/b4/c87d71a8818890c1b89fb25fa9a70233e32a89

warning: File "/opt/at7.0/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".

To enable execution of this file add

	add-auto-load-safe-path /opt/at7.0/lib64/libthread_db-1.0.so

line to your configuration file "/home/buildbot/.gdbinit".

To completely disable this security protection add

	set auto-load safe-path /

line to your configuration file "/home/buildbot/.gdbinit".

For more information about this security protection see the

"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:

	info "(gdb)Auto-loading safe path"

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.

warning: File "/opt/at7.0/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.

warning: File "/opt/at7.0/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.

warning: File "/opt/at7.0/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.

warning: File "/opt/at7.0/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.

warning: File "/opt/at7.0/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.

warning: File "/opt/at7.0/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.

warning: File "/opt/at7.0/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.

warning: File "/opt/at7.0/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.

warning: File "/opt/at7.0/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.

warning: File "/opt/at7.0/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.

warning: File "/opt/at7.0/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.

warning: File "/opt/at7.0/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.

warning: File "/opt/at7.0/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.

Core was generated by `/home/buildbot/maria-slave/p8-rhel6-bintar/build/sql/mysqld --defaults-group-su'.

Program terminated with signal 6, Aborted.

#0  0x00000fffa3a63a40 in .pthread_kill () from /opt/at7.0/lib64/power7/libpthread.so.0

#0  0x00000fffa3a63a40 in .pthread_kill () from /opt/at7.0/lib64/power7/libpthread.so.0

#1  0x00000000109eebb8 in .my_write_core ()

#2  0x00000000103e3a3c in .handle_fatal_signal ()

#3  <signal handler called>

#4  0x00000fffa320f2f0 in .raise () from /opt/at7.0/lib64/power7/libc.so.6

#5  0x00000fffa32112f4 in .abort () from /opt/at7.0/lib64/power7/libc.so.6

#6  0x000000001079ef54 in .buf_flush_page_cleaner_thread () at /home/buildbot/maria-slave/p8-rhel6-bintar/build/storage/xtradb/buf/buf0flu.cc:2803

#7  0x00000fffa3a5c548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0

#8  0x00000fffa32e8454 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:104

Thread 11 (Thread 24986):

#0  0x00000fffa3a67220 in .__libc_fsync () at ../sysdeps/unix/syscall-template.S:81

#1  0x000000001068b8e4 in ._Z18os_file_flush_funci () at /home/buildbot/maria-slave/p8-rhel6-bintar/build/storage/xtradb/os/os0file.cc:2482

#2  0x00000000107f04c8 in ._Z9fil_flushm () at /home/buildbot/maria-slave/p8-rhel6-bintar/build/storage/xtradb/include/os0file.ic:355

#3  0x00000000107f13cc in ._Z21fil_flush_file_spacesm () at /home/buildbot/maria-slave/p8-rhel6-bintar/build/storage/xtradb/fil/fil0fil.cc:6014

#4  0x0000000010790fdc in ._Z16buf_dblwr_updatePK10buf_page_t11buf_flush_t () at /home/buildbot/maria-slave/p8-rhel6-bintar/build/storage/xtradb/buf/buf0dblwr.cc:655

#5  0x0000000010797918 in ._Z24buf_flush_write_completeP10buf_page_t () at /home/buildbot/maria-slave/p8-rhel6-bintar/build/storage/xtradb/buf/buf0flu.cc:734

#6  0x000000001078b7c8 in ._Z20buf_page_io_completeP10buf_page_t () at /home/buildbot/maria-slave/p8-rhel6-bintar/build/storage/xtradb/buf/buf0buf.cc:4639

#7  0x00000000107ec38c in ._Z12fil_aio_waitm () at /home/buildbot/maria-slave/p8-rhel6-bintar/build/storage/xtradb/fil/fil0fil.cc:5820

#8  0x000000001070f790 in .io_handler_thread () at /home/buildbot/maria-slave/p8-rhel6-bintar/build/storage/xtradb/srv/srv0start.cc:532

#9  0x00000fffa3a5c548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0

#10 0x00000fffa32e8454 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:104

Thread 10 (Thread 24987):

#0  0x00000fffa3a10958 in ?? () from /lib64/libaio.so.1

#1  0x00000fffa3a109bc in .io_getevents () from /lib64/libaio.so.1

#2  0x000000001068f6c8 in ._Z19os_aio_linux_handlemPP10fil_node_tPPvPmS4_ () at /home/buildbot/maria-slave/p8-rhel6-bintar/build/storage/xtradb/os/os0file.cc:5179

#3  0x00000000107ec224 in ._Z12fil_aio_waitm () at /home/buildbot/maria-slave/p8-rhel6-bintar/build/storage/xtradb/fil/fil0fil.cc:5784

#4  0x000000001070f790 in .io_handler_thread () at /home/buildbot/maria-slave/p8-rhel6-bintar/build/storage/xtradb/srv/srv0start.cc:532

#5  0x00000fffa3a5c548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0

#6  0x00000fffa32e8454 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:104

Thread 9 (Thread 24985):

#0  0x00000fffa3a10958 in ?? () from /lib64/libaio.so.1

#1  0x00000fffa3a109bc in .io_getevents () from /lib64/libaio.so.1

#2  0x000000001068f6c8 in ._Z19os_aio_linux_handlemPP10fil_node_tPPvPmS4_ () at /home/buildbot/maria-slave/p8-rhel6-bintar/build/storage/xtradb/os/os0file.cc:5179

#3  0x00000000107ec224 in ._Z12fil_aio_waitm () at /home/buildbot/maria-slave/p8-rhel6-bintar/build/storage/xtradb/fil/fil0fil.cc:5784

#4  0x000000001070f790 in .io_handler_thread () at /home/buildbot/maria-slave/p8-rhel6-bintar/build/storage/xtradb/srv/srv0start.cc:532

#5  0x00000fffa3a5c548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0

#6  0x00000fffa32e8454 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:104

Thread 8 (Thread 24984):

#0  0x00000fffa3a10958 in ?? () from /lib64/libaio.so.1

#1  0x00000fffa3a109bc in .io_getevents () from /lib64/libaio.so.1

#2  0x000000001068f6c8 in ._Z19os_aio_linux_handlemPP10fil_node_tPPvPmS4_ () at /home/buildbot/maria-slave/p8-rhel6-bintar/build/storage/xtradb/os/os0file.cc:5179

#3  0x00000000107ec224 in ._Z12fil_aio_waitm () at /home/buildbot/maria-slave/p8-rhel6-bintar/build/storage/xtradb/fil/fil0fil.cc:5784

#4  0x000000001070f790 in .io_handler_thread () at /home/buildbot/maria-slave/p8-rhel6-bintar/build/storage/xtradb/srv/srv0start.cc:532

#5  0x00000fffa3a5c548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0

#6  0x00000fffa32e8454 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:104

Thread 7 (Thread 24983):

#0  0x00000fffa3a10958 in ?? () from /lib64/libaio.so.1

#1  0x00000fffa3a109bc in .io_getevents () from /lib64/libaio.so.1

#2  0x000000001068f6c8 in ._Z19os_aio_linux_handlemPP10fil_node_tPPvPmS4_ () at /home/buildbot/maria-slave/p8-rhel6-bintar/build/storage/xtradb/os/os0file.cc:5179

#3  0x00000000107ec224 in ._Z12fil_aio_waitm () at /home/buildbot/maria-slave/p8-rhel6-bintar/build/storage/xtradb/fil/fil0fil.cc:5784

#4  0x000000001070f790 in .io_handler_thread () at /home/buildbot/maria-slave/p8-rhel6-bintar/build/storage/xtradb/srv/srv0start.cc:532

#5  0x00000fffa3a5c548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0

#6  0x00000fffa32e8454 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:104

Thread 6 (Thread 24982):

#0  0x00000fffa3a10958 in ?? () from /lib64/libaio.so.1

#1  0x00000fffa3a109bc in .io_getevents () from /lib64/libaio.so.1

#2  0x000000001068f6c8 in ._Z19os_aio_linux_handlemPP10fil_node_tPPvPmS4_ () at /home/buildbot/maria-slave/p8-rhel6-bintar/build/storage/xtradb/os/os0file.cc:5179

#3  0x00000000107ec224 in ._Z12fil_aio_waitm () at /home/buildbot/maria-slave/p8-rhel6-bintar/build/storage/xtradb/fil/fil0fil.cc:5784

#4  0x000000001070f790 in .io_handler_thread () at /home/buildbot/maria-slave/p8-rhel6-bintar/build/storage/xtradb/srv/srv0start.cc:532

#5  0x00000fffa3a5c548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0

#6  0x00000fffa32e8454 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:104

Thread 5 (Thread 24981):

#0  0x00000fffa3a62130 in .pthread_cond_wait () from /opt/at7.0/lib64/power7/libpthread.so.0

#1  0x000000001014e5c8 in ._Z11mysqld_mainiPPc () at /home/buildbot/maria-slave/p8-rhel6-bintar/build/include/mysql/psi/mysql_thread.h:1154

#2  0x0000000010129e58 in .main () at /home/buildbot/maria-slave/p8-rhel6-bintar/build/sql/main.cc:25

Thread 4 (Thread 25032):

#0  0x00000fffa32dbf44 in .__select () at ../sysdeps/unix/syscall-template.S:81

#1  0x00000000106931c4 in ._Z15os_thread_sleepm ()

#2  0x000000001066f9b0 in ._Z37logs_empty_and_mark_files_at_shutdownv () at /home/buildbot/maria-slave/p8-rhel6-bintar/build/storage/xtradb/log/log0log.cc:3631

#3  0x0000000010713b04 in ._Z27innobase_shutdown_for_mysqlv () at /home/buildbot/maria-slave/p8-rhel6-bintar/build/storage/xtradb/srv/srv0start.cc:3002

#4  0x000000001060778c in ._ZL12innobase_endP10handlerton17ha_panic_function () at /home/buildbot/maria-slave/p8-rhel6-bintar/build/storage/xtradb/handler/ha_innodb.cc:4188

#5  0x00000000103e6db4 in ._Z22ha_finalize_handlertonP13st_plugin_int () at /home/buildbot/maria-slave/p8-rhel6-bintar/build/sql/handler.cc:440

#6  0x000000001021ef2c in ._ZL12reap_pluginsv () at /home/buildbot/maria-slave/p8-rhel6-bintar/build/sql/sql_plugin.cc:1176

#7  0x0000000000000000 in ?? ()

Thread 3 (Thread 24992):

#0  0x00000fffa3a62130 in .pthread_cond_wait () from /opt/at7.0/lib64/power7/libpthread.so.0

#1  0x00000000106921dc in ._Z17os_event_wait_lowP8os_eventl ()

#2  0x000000001070c5d4 in .srv_master_thread () at /home/buildbot/maria-slave/p8-rhel6-bintar/build/storage/xtradb/srv/srv0srv.cc:1010

#3  0x00000fffa3a5c548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0

#4  0x00000fffa32e8454 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:104

Thread 2 (Thread 25000):

#0  0x00000fffa3a67f18 in .__libc_sigwait () from /opt/at7.0/lib64/power7/libpthread.so.0

#1  0x00000000101404dc in .signal_hand () at /home/buildbot/maria-slave/p8-rhel6-bintar/build/sql/mysqld.cc:3278

#2  0x00000000109916dc in .pfs_spawn_thread ()

#3  0x00000fffa3a5c548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0

#4  0x00000fffa32e8454 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:104

Thread 1 (Thread 24994):

#0  0x00000fffa3a63a40 in .pthread_kill () from /opt/at7.0/lib64/power7/libpthread.so.0

#1  0x00000000109eebb8 in .my_write_core ()

#2  0x00000000103e3a3c in .handle_fatal_signal ()

#3  <signal handler called>

#4  0x00000fffa320f2f0 in .raise () from /opt/at7.0/lib64/power7/libc.so.6

#5  0x00000fffa32112f4 in .abort () from /opt/at7.0/lib64/power7/libc.so.6

#6  0x000000001079ef54 in .buf_flush_page_cleaner_thread () at /home/buildbot/maria-slave/p8-rhel6-bintar/build/storage/xtradb/buf/buf0flu.cc:2803

#7  0x00000fffa3a5c548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0

#8  0x00000fffa32e8454 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:104