[MDEV-11638] Encryption causes race conditions in InnoDB shutdown - Jira

Details

Type: Bug
Status: Closed (View Workflow)
Priority: Major
Resolution: Fixed
Affects Version/s: 10.2(EOL)
Fix Version/s: 10.1.21, 10.2.4
Component/s: Storage Engine - InnoDB, Tests
Labels:
None

Sprint:
10.0.29

Description

http://buildbot.askmonty.org/buildbot/builders/p8-rhel71-bintar-debug/builds/1484/steps/test/logs/stdio
(it's not limited to p8, but the output is better in this build).

encryption.innodb_onlinealter_encryption 'innodb' w2 [ fail ]  Found warnings/errors in server log file!

        Test ended at 2016-12-22 11:21:54

line

2016-12-22 11:21:48 70365707497856 [Warning] InnoDB: Trying to access missing tablespace 38

Attempting backtrace. You can use the following information to find out

2016-12-22 11:21:48 70366043798912 [ERROR] [FATAL] InnoDB: Page [page id: space=38, page number=743] still fixed or dirty

^ Found warnings in /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/mysql-test/var/2/log/mysqld.1.err

Program terminated with signal 6, Aborted.

#0  0x00003fff80a9eeb8 in __GI_raise (sig=<optimized out>) at ../sysdeps/unix/sysv/linux/raise.c:55

55	../sysdeps/unix/sysv/linux/raise.c: No such file or directory.

#0  0x00003fff80a9eeb8 in __GI_raise (sig=<optimized out>) at ../sysdeps/unix/sysv/linux/raise.c:55

#1  0x00003fff80aa19bc in __GI_abort () at abort.c:89

#2  0x000000005c4f8fdc in ut_dbg_assertion_failed (expr=0x0, file=0x5cc70e48 "/home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/ut/ut0ut.cc", line=949) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/ut/ut0dbg.cc:67

#3  0x000000005c4fe1d4 in ib::fatal::~fatal (this=0x3fff5f0b6610, __in_chrg=<optimized out>) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/ut/ut0ut.cc:949

#4  0x000000005c599430 in buf_all_freed_instance (buf_pool=0x1002ca9b2b0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/buf/buf0buf.cc:6230

#5  0x000000005c59ccc8 in buf_all_freed () at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/buf/buf0buf.cc:7219

#6  0x000000005c2b8ae0 in logs_empty_and_mark_files_at_shutdown () at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/log/log0log.cc:2415

#7  0x000000005c46f9c8 in innobase_shutdown_for_mysql () at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/srv/srv0start.cc:2801

#8  0x000000005c1d94b4 in innobase_end (hton=0x1002c70e160, type=HA_PANIC_CLOSE) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/handler/ha_innodb.cc:4844

#9  0x000000005becf0d0 in ha_finalize_handlerton (plugin=0x1002c6d04c8) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/handler.cc:451

#10 0x000000005bb9c914 in plugin_deinitialize (plugin=0x1002c6d04c8, ref_check=true) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/sql_plugin.cc:1217

#11 0x000000005bb9cfc8 in reap_plugins () at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/sql_plugin.cc:1294

#12 0x000000005bb9f80c in plugin_shutdown () at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/sql_plugin.cc:1945

#13 0x000000005ba46984 in clean_up (print_message=true) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/mysqld.cc:2201

#14 0x000000005ba462c4 in unireg_end () at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/mysqld.cc:2076

#15 0x000000005ba4608c in kill_server (sig_ptr=0x0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/mysqld.cc:2003

#16 0x000000005ba4610c in kill_server_thread (arg=0x3fff5f1d7280) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/mysqld.cc:2026

#17 0x000000005c93da00 in pfs_spawn_thread (arg=0x3fff1c001380) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/perfschema/pfs.cc:1862

#18 0x00003fff812f7cec in start_thread (arg=0x3fff5f0b7d80) at pthread_create.c:312

#19 0x00003fff80b91140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 12 (LWP 17819):

#0  __libc_fork () at ../sysdeps/nptl/fork.c:215

#1  0x00003fff81303f48 in __fork () at pt-fork.c:25

#2  0x000000005c9eea10 in my_addr_resolve_init () at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/mysys/my_addr_resolve.c:245

#3  0x000000005c9bfe1c in print_with_addr_resolve (addrs=0x3fff4affa720, n=18) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/mysys/stacktrace.c:244

#4  0x000000005c9c0028 in my_print_stacktrace (stack_bottom=0x0, thread_stack=297984) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/mysys/stacktrace.c:271

#5  0x000000005becc3b4 in handle_fatal_signal (sig=6) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/signal_handler.cc:166

#6  <signal handler called>

#7  0x00003fff80a9eeb8 in __GI_raise (sig=<optimized out>) at ../sysdeps/unix/sysv/linux/raise.c:55

#8  0x00003fff80aa19bc in __GI_abort () at abort.c:89

#9  0x000000005c4f8fdc in ut_dbg_assertion_failed (expr=0x5cc8a6e8 "found", file=0x5cc889c0 "/home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/buf/buf0buf.cc", line=4180) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/ut/ut0dbg.cc:67

#10 0x000000005c591948 in buf_page_get_gen (page_id=..., page_size=..., rw_latch=1, guess=0x0, mode=10, file=0x5cc0b708 "/home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/ibuf/ibuf0ibuf.cc", line=4584, mtr=0x3fff4affd7d0, err=0x3fff4affc178) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/buf/buf0buf.cc:4180

#11 0x000000005c546094 in btr_cur_search_to_nth_level (index=0x1002cbc6568, level=0, tuple=0x3fff18010238, mode=PAGE_CUR_GE, latch_mode=2, cursor=0x3fff4affd418, has_search_latch=0, file=0x5cc0b708 "/home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/ibuf/ibuf0ibuf.cc", line=4584, mtr=0x3fff4affd7d0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/btr/btr0cur.cc:1108

#12 0x000000005c55d880 in btr_pcur_open_low (index=0x1002cbc6568, level=0, tuple=0x3fff18010238, mode=PAGE_CUR_GE, latch_mode=2, cursor=0x3fff4affd418, file=0x5cc0b708 "/home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/ibuf/ibuf0ibuf.cc", line=4584, mtr=0x3fff4affd7d0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/include/btr0pcur.ic:457

#13 0x000000005c55f5b8 in btr_pcur_open_on_user_rec_func (index=0x1002cbc6568, tuple=0x3fff18010238, mode=PAGE_CUR_GE, latch_mode=2, cursor=0x3fff4affd418, file=0x5cc0b708 "/home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/ibuf/ibuf0ibuf.cc", line=4584, mtr=0x3fff4affd7d0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/btr/btr0pcur.cc:599

#14 0x000000005c278104 in ibuf_merge_or_delete_for_page (block=0x0, page_id=..., page_size=0x3fff4affe0f8, update_ibuf_bitmap=0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/ibuf/ibuf0ibuf.cc:4584

#15 0x000000005c592e58 in buf_page_get_gen (page_id=..., page_size=..., rw_latch=2, guess=0x0, mode=16, file=0x5ccb00b8 "/home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/fil/fil0crypt.cc", line=1927, mtr=0x3fff4affe100, err=0x3fff4affe004) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/buf/buf0buf.cc:4577

#16 0x000000005c673928 in fil_crypt_get_page_throttle_func (state=0x3fff4affe6e0, space=38, page_size=..., offset=743, mtr=0x3fff4affe100, sleeptime_ms=0x3fff4affe0b0, file=0x5ccb00b8 "/home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/fil/fil0crypt.cc", line=1927) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/fil/fil0crypt.cc:1812

#17 0x000000005c673d98 in fil_crypt_rotate_page (key_state=0x3fff4affe6d4, state=0x3fff4affe6e0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/fil/fil0crypt.cc:1924

#18 0x000000005c6744d0 in fil_crypt_rotate_pages (key_state=0x3fff4affe6d4, state=0x3fff4affe6e0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/fil/fil0crypt.cc:2092

#19 0x000000005c674f70 in fil_crypt_thread (arg=0x0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/fil/fil0crypt.cc:2324

#20 0x00003fff812f7cec in start_thread (arg=0x3fff4afff180) at pthread_create.c:312

#21 0x00003fff80b91140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 11 (LWP 17814):

#0  0x00003fff812ff7d0 in __pthread_cond_wait (cond=0x1002c747090, mutex=0x1002c747068) at pthread_cond_wait.c:188

#1  0x000000005c301f10 in os_event::wait (this=0x1002c747050) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/os/os0event.cc:166

#2  0x000000005c3013e0 in os_event::wait_low (this=0x1002c747050, reset_sig_count=1) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/os/os0event.cc:336

#3  0x000000005c301a3c in os_event_wait_low (event=0x1002c747050, reset_sig_count=0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/os/os0event.cc:535

#4  0x000000005c45fa88 in srv_master_thread (arg=0x0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/srv/srv0srv.cc:2650

#5  0x00003fff812f7cec in start_thread (arg=0x3fff59def180) at pthread_create.c:312

#6  0x00003fff80b91140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 10 (LWP 17804):

#0  0x00003fff81240738 in __io_getevents_0_4 () from /lib64/libaio.so.1

#1  0x00003fff812407a0 in io_getevents () from /lib64/libaio.so.1

#2  0x000000005c2ec838 in LinuxAIOHandler::collect (this=0x3fff5e82e5f0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/os/os0file.cc:2519

#3  0x000000005c2ece70 in LinuxAIOHandler::poll (this=0x3fff5e82e5f0, m1=0x3fff5e82e6e0, m2=0x3fff5e82e6e8, request=0x3fff5e82e6f0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/os/os0file.cc:2665

#4  0x000000005c2ed040 in os_aio_linux_handler (global_segment=1, m1=0x3fff5e82e6e0, m2=0x3fff5e82e6e8, request=0x3fff5e82e6f0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/os/os0file.cc:2721

#5  0x000000005c2f33a8 in os_aio_handler (segment=1, m1=0x3fff5e82e6e0, m2=0x3fff5e82e6e8, request=0x3fff5e82e6f0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/os/os0file.cc:6450

#6  0x000000005c6626c8 in fil_aio_wait (segment=1) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/fil/fil0fil.cc:5810

#7  0x000000005c466e80 in io_handler_thread (arg=0x5d3d5a38 <n+8>) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/srv/srv0start.cc:331

#8  0x00003fff812f7cec in start_thread (arg=0x3fff5e82f180) at pthread_create.c:312

#9  0x00003fff80b91140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 9 (LWP 17808):

#0  0x00003fff81240738 in __io_getevents_0_4 () from /lib64/libaio.so.1

#1  0x00003fff812407a0 in io_getevents () from /lib64/libaio.so.1

#2  0x000000005c2ec838 in LinuxAIOHandler::collect (this=0x3fff5c82e5f0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/os/os0file.cc:2519

#3  0x000000005c2ece70 in LinuxAIOHandler::poll (this=0x3fff5c82e5f0, m1=0x3fff5c82e6e0, m2=0x3fff5c82e6e8, request=0x3fff5c82e6f0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/os/os0file.cc:2665

#4  0x000000005c2ed040 in os_aio_linux_handler (global_segment=5, m1=0x3fff5c82e6e0, m2=0x3fff5c82e6e8, request=0x3fff5c82e6f0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/os/os0file.cc:2721

#5  0x000000005c2f33a8 in os_aio_handler (segment=5, m1=0x3fff5c82e6e0, m2=0x3fff5c82e6e8, request=0x3fff5c82e6f0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/os/os0file.cc:6450

#6  0x000000005c6626c8 in fil_aio_wait (segment=5) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/fil/fil0fil.cc:5810

#7  0x000000005c466e80 in io_handler_thread (arg=0x5d3d5a58 <n+40>) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/srv/srv0start.cc:331

#8  0x00003fff812f7cec in start_thread (arg=0x3fff5c82f180) at pthread_create.c:312

#9  0x00003fff80b91140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 8 (LWP 17807):

#0  0x00003fff81240738 in __io_getevents_0_4 () from /lib64/libaio.so.1

#1  0x00003fff812407a0 in io_getevents () from /lib64/libaio.so.1

#2  0x000000005c2ec838 in LinuxAIOHandler::collect (this=0x3fff5d02e5f0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/os/os0file.cc:2519

#3  0x000000005c2ece70 in LinuxAIOHandler::poll (this=0x3fff5d02e5f0, m1=0x3fff5d02e6e0, m2=0x3fff5d02e6e8, request=0x3fff5d02e6f0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/os/os0file.cc:2665

#4  0x000000005c2ed040 in os_aio_linux_handler (global_segment=4, m1=0x3fff5d02e6e0, m2=0x3fff5d02e6e8, request=0x3fff5d02e6f0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/os/os0file.cc:2721

#5  0x000000005c2f33a8 in os_aio_handler (segment=4, m1=0x3fff5d02e6e0, m2=0x3fff5d02e6e8, request=0x3fff5d02e6f0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/os/os0file.cc:6450

#6  0x000000005c6626c8 in fil_aio_wait (segment=4) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/fil/fil0fil.cc:5810

#7  0x000000005c466e80 in io_handler_thread (arg=0x5d3d5a50 <n+32>) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/srv/srv0start.cc:331

#8  0x00003fff812f7cec in start_thread (arg=0x3fff5d02f180) at pthread_create.c:312

#9  0x00003fff80b91140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 7 (LWP 17805):

#0  0x00003fff81240738 in __io_getevents_0_4 () from /lib64/libaio.so.1

#1  0x00003fff812407a0 in io_getevents () from /lib64/libaio.so.1

#2  0x000000005c2ec838 in LinuxAIOHandler::collect (this=0x3fff5e02e5f0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/os/os0file.cc:2519

#3  0x000000005c2ece70 in LinuxAIOHandler::poll (this=0x3fff5e02e5f0, m1=0x3fff5e02e6e0, m2=0x3fff5e02e6e8, request=0x3fff5e02e6f0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/os/os0file.cc:2665

#4  0x000000005c2ed040 in os_aio_linux_handler (global_segment=2, m1=0x3fff5e02e6e0, m2=0x3fff5e02e6e8, request=0x3fff5e02e6f0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/os/os0file.cc:2721

#5  0x000000005c2f33a8 in os_aio_handler (segment=2, m1=0x3fff5e02e6e0, m2=0x3fff5e02e6e8, request=0x3fff5e02e6f0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/os/os0file.cc:6450

#6  0x000000005c6626c8 in fil_aio_wait (segment=2) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/fil/fil0fil.cc:5810

#7  0x000000005c466e80 in io_handler_thread (arg=0x5d3d5a40 <n+16>) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/srv/srv0start.cc:331

#8  0x00003fff812f7cec in start_thread (arg=0x3fff5e02f180) at pthread_create.c:312

#9  0x00003fff80b91140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 6 (LWP 17829):

#0  0x00003fff81305714 in do_sigwait (sig=0x3fff5f1d7280, set=<optimized out>) at ../sysdeps/unix/sysv/linux/sigwait.c:61

#1  __sigwait (set=<optimized out>, sig=0x3fff5f1d7280) at ../sysdeps/unix/sysv/linux/sigwait.c:98

#2  0x000000005ba4a624 in signal_hand (arg=0x0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/mysqld.cc:3560

#3  0x000000005c93da00 in pfs_spawn_thread (arg=0x1002cee7b40) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/perfschema/pfs.cc:1862

#4  0x00003fff812f7cec in start_thread (arg=0x3fff5f1d7d80) at pthread_create.c:312

#5  0x00003fff80b91140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 5 (LWP 17806):

#0  0x00003fff81240738 in __io_getevents_0_4 () from /lib64/libaio.so.1

#1  0x00003fff812407a0 in io_getevents () from /lib64/libaio.so.1

#2  0x000000005c2ec838 in LinuxAIOHandler::collect (this=0x3fff5d82e5f0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/os/os0file.cc:2519

#3  0x000000005c2ece70 in LinuxAIOHandler::poll (this=0x3fff5d82e5f0, m1=0x3fff5d82e6e0, m2=0x3fff5d82e6e8, request=0x3fff5d82e6f0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/os/os0file.cc:2665

#4  0x000000005c2ed040 in os_aio_linux_handler (global_segment=3, m1=0x3fff5d82e6e0, m2=0x3fff5d82e6e8, request=0x3fff5d82e6f0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/os/os0file.cc:2721

#5  0x000000005c2f33a8 in os_aio_handler (segment=3, m1=0x3fff5d82e6e0, m2=0x3fff5d82e6e8, request=0x3fff5d82e6f0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/os/os0file.cc:6450

#6  0x000000005c6626c8 in fil_aio_wait (segment=3) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/fil/fil0fil.cc:5810

#7  0x000000005c466e80 in io_handler_thread (arg=0x5d3d5a48 <n+24>) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/srv/srv0start.cc:331

#8  0x00003fff812f7cec in start_thread (arg=0x3fff5d82f180) at pthread_create.c:312

#9  0x00003fff80b91140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 4 (LWP 17801):

#0  0x00003fff812ffcc0 in __pthread_cond_timedwait (cond=0x5dc006b0 <COND_timer>, mutex=0x5dc00628 <LOCK_timer+40>, abstime=0x3fff7c61e720) at pthread_cond_timedwait.c:199

#1  0x000000005c9c7eec in safe_cond_timedwait (cond=0x5dc006b0 <COND_timer>, mp=0x5dc00600 <LOCK_timer>, abstime=0x3fff7c61e720, file=0x5cd234f8 "/home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/include/mysql/psi/mysql_thread.h", line=1175) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/mysys/thr_mutex.c:545

#2  0x000000005c9c97b4 in inline_mysql_cond_timedwait (that=0x5dc006b0 <COND_timer>, mutex=0x5dc00600 <LOCK_timer>, abstime=0x3fff7c61e720, src_file=0x5cd23568 "/home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/mysys/thr_timer.c", src_line=292) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/include/mysql/psi/mysql_thread.h:1175

#3  0x000000005c9ca7cc in timer_handler (arg=0x0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/mysys/thr_timer.c:292

#4  0x000000005c93da00 in pfs_spawn_thread (arg=0x1002c6cc080) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/perfschema/pfs.cc:1862

#5  0x00003fff812f7cec in start_thread (arg=0x3fff7c61f180) at pthread_create.c:312

#6  0x00003fff80b91140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 3 (LWP 17803):

#0  0x00003fff81240738 in __io_getevents_0_4 () from /lib64/libaio.so.1

#1  0x00003fff812407a0 in io_getevents () from /lib64/libaio.so.1

#2  0x000000005c2ec838 in LinuxAIOHandler::collect (this=0x3fff5f02e5f0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/os/os0file.cc:2519

#3  0x000000005c2ece70 in LinuxAIOHandler::poll (this=0x3fff5f02e5f0, m1=0x3fff5f02e6e0, m2=0x3fff5f02e6e8, request=0x3fff5f02e6f0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/os/os0file.cc:2665

#4  0x000000005c2ed040 in os_aio_linux_handler (global_segment=0, m1=0x3fff5f02e6e0, m2=0x3fff5f02e6e8, request=0x3fff5f02e6f0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/os/os0file.cc:2721

#5  0x000000005c2f33a8 in os_aio_handler (segment=0, m1=0x3fff5f02e6e0, m2=0x3fff5f02e6e8, request=0x3fff5f02e6f0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/os/os0file.cc:6450

#6  0x000000005c6626c8 in fil_aio_wait (segment=0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/fil/fil0fil.cc:5810

#7  0x000000005c466e80 in io_handler_thread (arg=0x5d3d5a30 <n>) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/srv/srv0start.cc:331

#8  0x00003fff812f7cec in start_thread (arg=0x3fff5f02f180) at pthread_create.c:312

#9  0x00003fff80b91140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 2 (LWP 17800):

#0  0x00003fff812ff7d0 in __pthread_cond_wait (cond=0x5d34db40 <COND_thread_count>, mutex=0x5d34c9e8 <LOCK_thread_count+40>) at pthread_cond_wait.c:188

#1  0x000000005c9c7b10 in safe_cond_wait (cond=0x5d34db40 <COND_thread_count>, mp=0x5d34c9c0 <LOCK_thread_count>, file=0x5cab96b8 "/home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/include/mysql/psi/mysql_thread.h", line=1138) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/mysys/thr_mutex.c:491

#2  0x000000005ba41560 in inline_mysql_cond_wait (that=0x5d34db40 <COND_thread_count>, mutex=0x5d34c9c0 <LOCK_thread_count>, src_file=0x5cabb690 "/home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/mysqld.cc", src_line=6098) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/include/mysql/psi/mysql_thread.h:1138

#3  0x000000005ba5157c in mysqld_main (argc=151, argv=0x1002c53c180) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/mysqld.cc:6098

#4  0x000000005ba40638 in main (argc=30, argv=0x3fffe9c7fce8) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/main.cc:25

Thread 1 (LWP 18606):

#0  0x00003fff80a9eeb8 in __GI_raise (sig=<optimized out>) at ../sysdeps/unix/sysv/linux/raise.c:55

#1  0x00003fff80aa19bc in __GI_abort () at abort.c:89

#2  0x000000005c4f8fdc in ut_dbg_assertion_failed (expr=0x0, file=0x5cc70e48 "/home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/ut/ut0ut.cc", line=949) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/ut/ut0dbg.cc:67

#3  0x000000005c4fe1d4 in ib::fatal::~fatal (this=0x3fff5f0b6610, __in_chrg=<optimized out>) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/ut/ut0ut.cc:949

#4  0x000000005c599430 in buf_all_freed_instance (buf_pool=0x1002ca9b2b0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/buf/buf0buf.cc:6230

#5  0x000000005c59ccc8 in buf_all_freed () at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/buf/buf0buf.cc:7219

#6  0x000000005c2b8ae0 in logs_empty_and_mark_files_at_shutdown () at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/log/log0log.cc:2415

#7  0x000000005c46f9c8 in innobase_shutdown_for_mysql () at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/srv/srv0start.cc:2801

#8  0x000000005c1d94b4 in innobase_end (hton=0x1002c70e160, type=HA_PANIC_CLOSE) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/handler/ha_innodb.cc:4844

#9  0x000000005becf0d0 in ha_finalize_handlerton (plugin=0x1002c6d04c8) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/handler.cc:451

#10 0x000000005bb9c914 in plugin_deinitialize (plugin=0x1002c6d04c8, ref_check=true) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/sql_plugin.cc:1217

#11 0x000000005bb9cfc8 in reap_plugins () at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/sql_plugin.cc:1294

#12 0x000000005bb9f80c in plugin_shutdown () at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/sql_plugin.cc:1945

#13 0x000000005ba46984 in clean_up (print_message=true) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/mysqld.cc:2201

#14 0x000000005ba462c4 in unireg_end () at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/mysqld.cc:2076

#15 0x000000005ba4608c in kill_server (sig_ptr=0x0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/mysqld.cc:2003

#16 0x000000005ba4610c in kill_server_thread (arg=0x3fff5f1d7280) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/mysqld.cc:2026

#17 0x000000005c93da00 in pfs_spawn_thread (arg=0x3fff1c001380) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/perfschema/pfs.cc:1862

#18 0x00003fff812f7cec in start_thread (arg=0x3fff5f0b7d80) at pthread_create.c:312

#19 0x00003fff80b91140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

encryption.innodb-bad-key-change3 - failed several times on d6a1f9f10f21bfb5198b81c40e097556 and 08f79bdeda89f5b8e09d99e7dc07fc9c
http://buildbot.askmonty.org/buildbot/builders/p8-rhel71-bintar-debug/builds/1484/steps/test/logs/stdio

encryption.innodb-bad-key-change3 'innodb' w2 [ fail ]  Found warnings/errors in server log file!

        Test ended at 2016-12-22 11:29:29

line

Attempting backtrace. You can use the following information to find out

^ Found warnings in /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/mysql-test/var/2/log/mysqld.1.err

Program terminated with signal 6, Aborted.

#0  0x00003fff82cc0ef0 in __pthread_kill (threadid=<optimized out>, signo=<optimized out>) at ../sysdeps/unix/sysv/linux/pthread_kill.c:60

60	../sysdeps/unix/sysv/linux/pthread_kill.c: No such file or directory.

#0  0x00003fff82cc0ef0 in __pthread_kill (threadid=<optimized out>, signo=<optimized out>) at ../sysdeps/unix/sysv/linux/pthread_kill.c:60

#1  0x000000003e0e0180 in my_write_core (sig=6) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/mysys/stacktrace.c:477

#2  0x000000003d5ec924 in handle_fatal_signal (sig=6) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/signal_handler.cc:294

#3  <signal handler called>

#4  0x00003fff8245eeb8 in __GI_raise (sig=<optimized out>) at ../sysdeps/unix/sysv/linux/raise.c:55

#5  0x00003fff824619bc in __GI_abort () at abort.c:89

#6  0x00003fff824ac224 in __libc_message (do_abort=<optimized out>, fmt=<optimized out>) at ../sysdeps/posix/libc_fatal.c:175

#7  0x00003fff824bbd58 in malloc_printerr (ptr=<optimized out>, str=0x3fff825c2b70 "munmap_chunk(): invalid pointer", action=3) at malloc.c:4958

#8  munmap_chunk (p=<optimized out>) at malloc.c:2815

#9  __GI___libc_free (mem=<optimized out>) at malloc.c:2940

#10 0x00003fff82796898 in operator delete(void*) () from /opt/at8.0/lib64/power8/libstdc++.so.6

#11 0x000000003dba9948 in LatchCounter::~LatchCounter (this=0x100078ed658, __in_chrg=<optimized out>) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/include/sync0types.h:586

#12 0x000000003dbacd80 in LatchMeta<LatchCounter>::~LatchMeta (this=0x100078ed630, __in_chrg=<optimized out>) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/include/sync0types.h:773

#13 0x000000003dba9238 in sync_latch_meta_destroy () at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/sync/sync0debug.cc:1593

#14 0x000000003dba9698 in sync_check_close () at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/sync/sync0debug.cc:1811

#15 0x000000003db8fcdc in innobase_shutdown_for_mysql () at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/srv/srv0start.cc:2886

#16 0x000000003d8f94b4 in innobase_end (hton=0x100078cc800, type=HA_PANIC_CLOSE) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/handler/ha_innodb.cc:4844

#17 0x000000003d5ef0d0 in ha_finalize_handlerton (plugin=0x10007890428) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/handler.cc:451

#18 0x000000003d2bc914 in plugin_deinitialize (plugin=0x10007890428, ref_check=true) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/sql_plugin.cc:1217

#19 0x000000003d2bcfc8 in reap_plugins () at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/sql_plugin.cc:1294

#20 0x000000003d2bf80c in plugin_shutdown () at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/sql_plugin.cc:1945

#21 0x000000003d166984 in clean_up (print_message=true) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/mysqld.cc:2201

#22 0x000000003d1662c4 in unireg_end () at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/mysqld.cc:2076

#23 0x000000003d16608c in kill_server (sig_ptr=0x0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/mysqld.cc:2003

#24 0x000000003d16610c in kill_server_thread (arg=0x3fff60bc7280) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/mysqld.cc:2026

#25 0x000000003e05da00 in pfs_spawn_thread (arg=0x3fff28001380) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/perfschema/pfs.cc:1862

#26 0x00003fff82cb7cec in start_thread (arg=0x3fff60aa7d80) at pthread_create.c:312

#27 0x00003fff82551140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 4 (LWP 24982):

#0  0x00003fff82cbfcc0 in __pthread_cond_timedwait (cond=0x3f3206b0 <COND_timer>, mutex=0x3f320628 <LOCK_timer+40>, abstime=0x3fff7dfde720) at pthread_cond_timedwait.c:199

#1  0x000000003e0e7eec in safe_cond_timedwait (cond=0x3f3206b0 <COND_timer>, mp=0x3f320600 <LOCK_timer>, abstime=0x3fff7dfde720, file=0x3e4434f8 "/home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/include/mysql/psi/mysql_thread.h", line=1175) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/mysys/thr_mutex.c:545

#2  0x000000003e0e97b4 in inline_mysql_cond_timedwait (that=0x3f3206b0 <COND_timer>, mutex=0x3f320600 <LOCK_timer>, abstime=0x3fff7dfde720, src_file=0x3e443568 "/home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/mysys/thr_timer.c", src_line=292) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/include/mysql/psi/mysql_thread.h:1175

#3  0x000000003e0ea7cc in timer_handler (arg=0x0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/mysys/thr_timer.c:292

#4  0x000000003e05da00 in pfs_spawn_thread (arg=0x1000788bfe0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/perfschema/pfs.cc:1862

#5  0x00003fff82cb7cec in start_thread (arg=0x3fff7dfdf180) at pthread_create.c:312

#6  0x00003fff82551140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 3 (LWP 25006):

#0  0x00003fff82cc5714 in do_sigwait (sig=0x3fff60bc7280, set=<optimized out>) at ../sysdeps/unix/sysv/linux/sigwait.c:61

#1  __sigwait (set=<optimized out>, sig=0x3fff60bc7280) at ../sysdeps/unix/sysv/linux/sigwait.c:98

#2  0x000000003d16a624 in signal_hand (arg=0x0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/mysqld.cc:3560

#3  0x000000003e05da00 in pfs_spawn_thread (arg=0x1000809cde0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/perfschema/pfs.cc:1862

#4  0x00003fff82cb7cec in start_thread (arg=0x3fff60bc7d80) at pthread_create.c:312

#5  0x00003fff82551140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 2 (LWP 24981):

#0  0x00003fff82cbf7d0 in __pthread_cond_wait (cond=0x3ea6db40 <COND_thread_count>, mutex=0x3ea6c9e8 <LOCK_thread_count+40>) at pthread_cond_wait.c:188

#1  0x000000003e0e7b10 in safe_cond_wait (cond=0x3ea6db40 <COND_thread_count>, mp=0x3ea6c9c0 <LOCK_thread_count>, file=0x3e1d96b8 "/home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/include/mysql/psi/mysql_thread.h", line=1138) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/mysys/thr_mutex.c:491

#2  0x000000003d161560 in inline_mysql_cond_wait (that=0x3ea6db40 <COND_thread_count>, mutex=0x3ea6c9c0 <LOCK_thread_count>, src_file=0x3e1db690 "/home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/mysqld.cc", src_line=6098) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/include/mysql/psi/mysql_thread.h:1138

#3  0x000000003d17157c in mysqld_main (argc=143, argv=0x100076fc180) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/mysqld.cc:6098

#4  0x000000003d160638 in main (argc=22, argv=0x3ffffddac5e8) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/main.cc:25

Thread 1 (LWP 25019):

#0  0x00003fff82cc0ef0 in __pthread_kill (threadid=<optimized out>, signo=<optimized out>) at ../sysdeps/unix/sysv/linux/pthread_kill.c:60

#1  0x000000003e0e0180 in my_write_core (sig=6) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/mysys/stacktrace.c:477

#2  0x000000003d5ec924 in handle_fatal_signal (sig=6) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/signal_handler.cc:294

#3  <signal handler called>

#4  0x00003fff8245eeb8 in __GI_raise (sig=<optimized out>) at ../sysdeps/unix/sysv/linux/raise.c:55

#5  0x00003fff824619bc in __GI_abort () at abort.c:89

#6  0x00003fff824ac224 in __libc_message (do_abort=<optimized out>, fmt=<optimized out>) at ../sysdeps/posix/libc_fatal.c:175

#7  0x00003fff824bbd58 in malloc_printerr (ptr=<optimized out>, str=0x3fff825c2b70 "munmap_chunk(): invalid pointer", action=3) at malloc.c:4958

#8  munmap_chunk (p=<optimized out>) at malloc.c:2815

#9  __GI___libc_free (mem=<optimized out>) at malloc.c:2940

#10 0x00003fff82796898 in operator delete(void*) () from /opt/at8.0/lib64/power8/libstdc++.so.6

#11 0x000000003dba9948 in LatchCounter::~LatchCounter (this=0x100078ed658, __in_chrg=<optimized out>) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/include/sync0types.h:586

#12 0x000000003dbacd80 in LatchMeta<LatchCounter>::~LatchMeta (this=0x100078ed630, __in_chrg=<optimized out>) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/include/sync0types.h:773

#13 0x000000003dba9238 in sync_latch_meta_destroy () at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/sync/sync0debug.cc:1593

#14 0x000000003dba9698 in sync_check_close () at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/sync/sync0debug.cc:1811

#15 0x000000003db8fcdc in innobase_shutdown_for_mysql () at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/srv/srv0start.cc:2886

#16 0x000000003d8f94b4 in innobase_end (hton=0x100078cc800, type=HA_PANIC_CLOSE) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/handler/ha_innodb.cc:4844

#17 0x000000003d5ef0d0 in ha_finalize_handlerton (plugin=0x10007890428) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/handler.cc:451

#18 0x000000003d2bc914 in plugin_deinitialize (plugin=0x10007890428, ref_check=true) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/sql_plugin.cc:1217

#19 0x000000003d2bcfc8 in reap_plugins () at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/sql_plugin.cc:1294

#20 0x000000003d2bf80c in plugin_shutdown () at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/sql_plugin.cc:1945

#21 0x000000003d166984 in clean_up (print_message=true) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/mysqld.cc:2201

#22 0x000000003d1662c4 in unireg_end () at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/mysqld.cc:2076

#23 0x000000003d16608c in kill_server (sig_ptr=0x0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/mysqld.cc:2003

#24 0x000000003d16610c in kill_server_thread (arg=0x3fff60bc7280) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/mysqld.cc:2026

#25 0x000000003e05da00 in pfs_spawn_thread (arg=0x3fff28001380) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/perfschema/pfs.cc:1862

#26 0x00003fff82cb7cec in start_thread (arg=0x3fff60aa7d80) at pthread_create.c:312

#27 0x00003fff82551140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Attachments

Issue Links

blocks

MDEV-12052 Shutdown crash presumably due to master thread activity

Closed

relates to

MDEV-7069 Fix buildbot failures in main server trees

Stalled

MDEV-9359 encryption.create_or_replace fails sporadically in buildbot: failing assertion: mutex->magic_n == MUTEX_MAGIC_N

Closed

Activity

Ascending order - Click to sort in descending order

Elena Stepanova created issue - 2016-12-22 21:40

Elena Stepanova made changes - 2016-12-22 21:40

Field	Original Value	New Value
Link		This issue relates to MDEV-7069 [ MDEV-7069 ]

Elena Stepanova made changes - 2016-12-22 21:48

Summary

encryption.innodb_onlinealter_encryption fails in buildbot (crashes on shutdown)

encryption.innodb_onlinealter_encryption, encryption.innodb-bad-key-change3 fail in buildbot (crash on shutdown)

Elena Stepanova made changes - 2016-12-22 21:50

Description

Happened twice on commit 08f79bdeda89f5b8e09d99e7dc07fc9c (the previous one is a suspect, but it was mainly skipped by buildbot due to the arrival of a newer push).
This particular failure hasn't happened before. The test used to fail, but differently.

http://buildbot.askmonty.org/buildbot/builders/p8-rhel71-bintar-debug/builds/1484/steps/test/logs/stdio
(it's not limited to p8, but the output is better in this build).

{noformat}
encryption.innodb_onlinealter_encryption 'innodb' w2 [ fail ] Found warnings/errors in server log file!
Test ended at 2016-12-22 11:21:54
line
2016-12-22 11:21:48 70365707497856 [Warning] InnoDB: Trying to access missing tablespace 38
Attempting backtrace. You can use the following information to find out
2016-12-22 11:21:48 70366043798912 [ERROR] [FATAL] InnoDB: Page [page id: space=38, page number=743] still fixed or dirty
^ Found warnings in /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/mysql-test/var/2/log/mysqld.1.err
{noformat}
{noformat}
Program terminated with signal 6, Aborted.
#0 0x00003fff80a9eeb8 in __GI_raise (sig=<optimized out>) at ../sysdeps/unix/sysv/linux/raise.c:55
55 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
#0 0x00003fff80a9eeb8 in __GI_raise (sig=<optimized out>) at ../sysdeps/unix/sysv/linux/raise.c:55
#1 0x00003fff80aa19bc in __GI_abort () at abort.c:89
#2 0x000000005c4f8fdc in ut_dbg_assertion_failed (expr=0x0, file=0x5cc70e48 "/home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/ut/ut0ut.cc", line=949) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/ut/ut0dbg.cc:67
#3 0x000000005c4fe1d4 in ib::fatal::~fatal (this=0x3fff5f0b6610, __in_chrg=<optimized out>) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/ut/ut0ut.cc:949
#4 0x000000005c599430 in buf_all_freed_instance (buf_pool=0x1002ca9b2b0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/buf/buf0buf.cc:6230
#5 0x000000005c59ccc8 in buf_all_freed () at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/buf/buf0buf.cc:7219
#6 0x000000005c2b8ae0 in logs_empty_and_mark_files_at_shutdown () at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/log/log0log.cc:2415
#7 0x000000005c46f9c8 in innobase_shutdown_for_mysql () at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/srv/srv0start.cc:2801
#8 0x000000005c1d94b4 in innobase_end (hton=0x1002c70e160, type=HA_PANIC_CLOSE) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/handler/ha_innodb.cc:4844
#9 0x000000005becf0d0 in ha_finalize_handlerton (plugin=0x1002c6d04c8) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/handler.cc:451
#10 0x000000005bb9c914 in plugin_deinitialize (plugin=0x1002c6d04c8, ref_check=true) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/sql_plugin.cc:1217
#11 0x000000005bb9cfc8 in reap_plugins () at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/sql_plugin.cc:1294
#12 0x000000005bb9f80c in plugin_shutdown () at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/sql_plugin.cc:1945
#13 0x000000005ba46984 in clean_up (print_message=true) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/mysqld.cc:2201
#14 0x000000005ba462c4 in unireg_end () at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/mysqld.cc:2076
#15 0x000000005ba4608c in kill_server (sig_ptr=0x0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/mysqld.cc:2003
#16 0x000000005ba4610c in kill_server_thread (arg=0x3fff5f1d7280) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/mysqld.cc:2026
#17 0x000000005c93da00 in pfs_spawn_thread (arg=0x3fff1c001380) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/perfschema/pfs.cc:1862
#18 0x00003fff812f7cec in start_thread (arg=0x3fff5f0b7d80) at pthread_create.c:312
#19 0x00003fff80b91140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 12 (LWP 17819):
#0 __libc_fork () at ../sysdeps/nptl/fork.c:215
#1 0x00003fff81303f48 in __fork () at pt-fork.c:25
#2 0x000000005c9eea10 in my_addr_resolve_init () at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/mysys/my_addr_resolve.c:245
#3 0x000000005c9bfe1c in print_with_addr_resolve (addrs=0x3fff4affa720, n=18) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/mysys/stacktrace.c:244
#4 0x000000005c9c0028 in my_print_stacktrace (stack_bottom=0x0, thread_stack=297984) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/mysys/stacktrace.c:271
#5 0x000000005becc3b4 in handle_fatal_signal (sig=6) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/signal_handler.cc:166
#6 <signal handler called>
#7 0x00003fff80a9eeb8 in __GI_raise (sig=<optimized out>) at ../sysdeps/unix/sysv/linux/raise.c:55
#8 0x00003fff80aa19bc in __GI_abort () at abort.c:89
#9 0x000000005c4f8fdc in ut_dbg_assertion_failed (expr=0x5cc8a6e8 "found", file=0x5cc889c0 "/home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/buf/buf0buf.cc", line=4180) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/ut/ut0dbg.cc:67
#10 0x000000005c591948 in buf_page_get_gen (page_id=..., page_size=..., rw_latch=1, guess=0x0, mode=10, file=0x5cc0b708 "/home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/ibuf/ibuf0ibuf.cc", line=4584, mtr=0x3fff4affd7d0, err=0x3fff4affc178) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/buf/buf0buf.cc:4180
#11 0x000000005c546094 in btr_cur_search_to_nth_level (index=0x1002cbc6568, level=0, tuple=0x3fff18010238, mode=PAGE_CUR_GE, latch_mode=2, cursor=0x3fff4affd418, has_search_latch=0, file=0x5cc0b708 "/home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/ibuf/ibuf0ibuf.cc", line=4584, mtr=0x3fff4affd7d0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/btr/btr0cur.cc:1108
#12 0x000000005c55d880 in btr_pcur_open_low (index=0x1002cbc6568, level=0, tuple=0x3fff18010238, mode=PAGE_CUR_GE, latch_mode=2, cursor=0x3fff4affd418, file=0x5cc0b708 "/home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/ibuf/ibuf0ibuf.cc", line=4584, mtr=0x3fff4affd7d0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/include/btr0pcur.ic:457
#13 0x000000005c55f5b8 in btr_pcur_open_on_user_rec_func (index=0x1002cbc6568, tuple=0x3fff18010238, mode=PAGE_CUR_GE, latch_mode=2, cursor=0x3fff4affd418, file=0x5cc0b708 "/home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/ibuf/ibuf0ibuf.cc", line=4584, mtr=0x3fff4affd7d0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/btr/btr0pcur.cc:599
#14 0x000000005c278104 in ibuf_merge_or_delete_for_page (block=0x0, page_id=..., page_size=0x3fff4affe0f8, update_ibuf_bitmap=0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/ibuf/ibuf0ibuf.cc:4584
#15 0x000000005c592e58 in buf_page_get_gen (page_id=..., page_size=..., rw_latch=2, guess=0x0, mode=16, file=0x5ccb00b8 "/home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/fil/fil0crypt.cc", line=1927, mtr=0x3fff4affe100, err=0x3fff4affe004) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/buf/buf0buf.cc:4577
#16 0x000000005c673928 in fil_crypt_get_page_throttle_func (state=0x3fff4affe6e0, space=38, page_size=..., offset=743, mtr=0x3fff4affe100, sleeptime_ms=0x3fff4affe0b0, file=0x5ccb00b8 "/home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/fil/fil0crypt.cc", line=1927) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/fil/fil0crypt.cc:1812
#17 0x000000005c673d98 in fil_crypt_rotate_page (key_state=0x3fff4affe6d4, state=0x3fff4affe6e0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/fil/fil0crypt.cc:1924
#18 0x000000005c6744d0 in fil_crypt_rotate_pages (key_state=0x3fff4affe6d4, state=0x3fff4affe6e0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/fil/fil0crypt.cc:2092
#19 0x000000005c674f70 in fil_crypt_thread (arg=0x0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/fil/fil0crypt.cc:2324
#20 0x00003fff812f7cec in start_thread (arg=0x3fff4afff180) at pthread_create.c:312
#21 0x00003fff80b91140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 11 (LWP 17814):
#0 0x00003fff812ff7d0 in __pthread_cond_wait (cond=0x1002c747090, mutex=0x1002c747068) at pthread_cond_wait.c:188
#1 0x000000005c301f10 in os_event::wait (this=0x1002c747050) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/os/os0event.cc:166
#2 0x000000005c3013e0 in os_event::wait_low (this=0x1002c747050, reset_sig_count=1) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/os/os0event.cc:336
#3 0x000000005c301a3c in os_event_wait_low (event=0x1002c747050, reset_sig_count=0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/os/os0event.cc:535
#4 0x000000005c45fa88 in srv_master_thread (arg=0x0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/srv/srv0srv.cc:2650
#5 0x00003fff812f7cec in start_thread (arg=0x3fff59def180) at pthread_create.c:312
#6 0x00003fff80b91140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 10 (LWP 17804):
#0 0x00003fff81240738 in __io_getevents_0_4 () from /lib64/libaio.so.1
#1 0x00003fff812407a0 in io_getevents () from /lib64/libaio.so.1
#2 0x000000005c2ec838 in LinuxAIOHandler::collect (this=0x3fff5e82e5f0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/os/os0file.cc:2519
#3 0x000000005c2ece70 in LinuxAIOHandler::poll (this=0x3fff5e82e5f0, m1=0x3fff5e82e6e0, m2=0x3fff5e82e6e8, request=0x3fff5e82e6f0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/os/os0file.cc:2665
#4 0x000000005c2ed040 in os_aio_linux_handler (global_segment=1, m1=0x3fff5e82e6e0, m2=0x3fff5e82e6e8, request=0x3fff5e82e6f0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/os/os0file.cc:2721
#5 0x000000005c2f33a8 in os_aio_handler (segment=1, m1=0x3fff5e82e6e0, m2=0x3fff5e82e6e8, request=0x3fff5e82e6f0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/os/os0file.cc:6450
#6 0x000000005c6626c8 in fil_aio_wait (segment=1) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/fil/fil0fil.cc:5810
#7 0x000000005c466e80 in io_handler_thread (arg=0x5d3d5a38 <n+8>) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/srv/srv0start.cc:331
#8 0x00003fff812f7cec in start_thread (arg=0x3fff5e82f180) at pthread_create.c:312
#9 0x00003fff80b91140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 9 (LWP 17808):
#0 0x00003fff81240738 in __io_getevents_0_4 () from /lib64/libaio.so.1
#1 0x00003fff812407a0 in io_getevents () from /lib64/libaio.so.1
#2 0x000000005c2ec838 in LinuxAIOHandler::collect (this=0x3fff5c82e5f0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/os/os0file.cc:2519
#3 0x000000005c2ece70 in LinuxAIOHandler::poll (this=0x3fff5c82e5f0, m1=0x3fff5c82e6e0, m2=0x3fff5c82e6e8, request=0x3fff5c82e6f0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/os/os0file.cc:2665
#4 0x000000005c2ed040 in os_aio_linux_handler (global_segment=5, m1=0x3fff5c82e6e0, m2=0x3fff5c82e6e8, request=0x3fff5c82e6f0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/os/os0file.cc:2721
#5 0x000000005c2f33a8 in os_aio_handler (segment=5, m1=0x3fff5c82e6e0, m2=0x3fff5c82e6e8, request=0x3fff5c82e6f0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/os/os0file.cc:6450
#6 0x000000005c6626c8 in fil_aio_wait (segment=5) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/fil/fil0fil.cc:5810
#7 0x000000005c466e80 in io_handler_thread (arg=0x5d3d5a58 <n+40>) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/srv/srv0start.cc:331
#8 0x00003fff812f7cec in start_thread (arg=0x3fff5c82f180) at pthread_create.c:312
#9 0x00003fff80b91140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 8 (LWP 17807):
#0 0x00003fff81240738 in __io_getevents_0_4 () from /lib64/libaio.so.1
#1 0x00003fff812407a0 in io_getevents () from /lib64/libaio.so.1
#2 0x000000005c2ec838 in LinuxAIOHandler::collect (this=0x3fff5d02e5f0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/os/os0file.cc:2519
#3 0x000000005c2ece70 in LinuxAIOHandler::poll (this=0x3fff5d02e5f0, m1=0x3fff5d02e6e0, m2=0x3fff5d02e6e8, request=0x3fff5d02e6f0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/os/os0file.cc:2665
#4 0x000000005c2ed040 in os_aio_linux_handler (global_segment=4, m1=0x3fff5d02e6e0, m2=0x3fff5d02e6e8, request=0x3fff5d02e6f0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/os/os0file.cc:2721
#5 0x000000005c2f33a8 in os_aio_handler (segment=4, m1=0x3fff5d02e6e0, m2=0x3fff5d02e6e8, request=0x3fff5d02e6f0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/os/os0file.cc:6450
#6 0x000000005c6626c8 in fil_aio_wait (segment=4) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/fil/fil0fil.cc:5810
#7 0x000000005c466e80 in io_handler_thread (arg=0x5d3d5a50 <n+32>) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/srv/srv0start.cc:331
#8 0x00003fff812f7cec in start_thread (arg=0x3fff5d02f180) at pthread_create.c:312
#9 0x00003fff80b91140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 7 (LWP 17805):
#0 0x00003fff81240738 in __io_getevents_0_4 () from /lib64/libaio.so.1
#1 0x00003fff812407a0 in io_getevents () from /lib64/libaio.so.1
#2 0x000000005c2ec838 in LinuxAIOHandler::collect (this=0x3fff5e02e5f0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/os/os0file.cc:2519
#3 0x000000005c2ece70 in LinuxAIOHandler::poll (this=0x3fff5e02e5f0, m1=0x3fff5e02e6e0, m2=0x3fff5e02e6e8, request=0x3fff5e02e6f0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/os/os0file.cc:2665
#4 0x000000005c2ed040 in os_aio_linux_handler (global_segment=2, m1=0x3fff5e02e6e0, m2=0x3fff5e02e6e8, request=0x3fff5e02e6f0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/os/os0file.cc:2721
#5 0x000000005c2f33a8 in os_aio_handler (segment=2, m1=0x3fff5e02e6e0, m2=0x3fff5e02e6e8, request=0x3fff5e02e6f0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/os/os0file.cc:6450
#6 0x000000005c6626c8 in fil_aio_wait (segment=2) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/fil/fil0fil.cc:5810
#7 0x000000005c466e80 in io_handler_thread (arg=0x5d3d5a40 <n+16>) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/srv/srv0start.cc:331
#8 0x00003fff812f7cec in start_thread (arg=0x3fff5e02f180) at pthread_create.c:312
#9 0x00003fff80b91140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 6 (LWP 17829):
#0 0x00003fff81305714 in do_sigwait (sig=0x3fff5f1d7280, set=<optimized out>) at ../sysdeps/unix/sysv/linux/sigwait.c:61
#1 __sigwait (set=<optimized out>, sig=0x3fff5f1d7280) at ../sysdeps/unix/sysv/linux/sigwait.c:98
#2 0x000000005ba4a624 in signal_hand (arg=0x0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/mysqld.cc:3560
#3 0x000000005c93da00 in pfs_spawn_thread (arg=0x1002cee7b40) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/perfschema/pfs.cc:1862
#4 0x00003fff812f7cec in start_thread (arg=0x3fff5f1d7d80) at pthread_create.c:312
#5 0x00003fff80b91140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 5 (LWP 17806):
#0 0x00003fff81240738 in __io_getevents_0_4 () from /lib64/libaio.so.1
#1 0x00003fff812407a0 in io_getevents () from /lib64/libaio.so.1
#2 0x000000005c2ec838 in LinuxAIOHandler::collect (this=0x3fff5d82e5f0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/os/os0file.cc:2519
#3 0x000000005c2ece70 in LinuxAIOHandler::poll (this=0x3fff5d82e5f0, m1=0x3fff5d82e6e0, m2=0x3fff5d82e6e8, request=0x3fff5d82e6f0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/os/os0file.cc:2665
#4 0x000000005c2ed040 in os_aio_linux_handler (global_segment=3, m1=0x3fff5d82e6e0, m2=0x3fff5d82e6e8, request=0x3fff5d82e6f0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/os/os0file.cc:2721
#5 0x000000005c2f33a8 in os_aio_handler (segment=3, m1=0x3fff5d82e6e0, m2=0x3fff5d82e6e8, request=0x3fff5d82e6f0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/os/os0file.cc:6450
#6 0x000000005c6626c8 in fil_aio_wait (segment=3) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/fil/fil0fil.cc:5810
#7 0x000000005c466e80 in io_handler_thread (arg=0x5d3d5a48 <n+24>) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/srv/srv0start.cc:331
#8 0x00003fff812f7cec in start_thread (arg=0x3fff5d82f180) at pthread_create.c:312
#9 0x00003fff80b91140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 4 (LWP 17801):
#0 0x00003fff812ffcc0 in __pthread_cond_timedwait (cond=0x5dc006b0 <COND_timer>, mutex=0x5dc00628 <LOCK_timer+40>, abstime=0x3fff7c61e720) at pthread_cond_timedwait.c:199
#1 0x000000005c9c7eec in safe_cond_timedwait (cond=0x5dc006b0 <COND_timer>, mp=0x5dc00600 <LOCK_timer>, abstime=0x3fff7c61e720, file=0x5cd234f8 "/home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/include/mysql/psi/mysql_thread.h", line=1175) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/mysys/thr_mutex.c:545
#2 0x000000005c9c97b4 in inline_mysql_cond_timedwait (that=0x5dc006b0 <COND_timer>, mutex=0x5dc00600 <LOCK_timer>, abstime=0x3fff7c61e720, src_file=0x5cd23568 "/home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/mysys/thr_timer.c", src_line=292) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/include/mysql/psi/mysql_thread.h:1175
#3 0x000000005c9ca7cc in timer_handler (arg=0x0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/mysys/thr_timer.c:292
#4 0x000000005c93da00 in pfs_spawn_thread (arg=0x1002c6cc080) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/perfschema/pfs.cc:1862
#5 0x00003fff812f7cec in start_thread (arg=0x3fff7c61f180) at pthread_create.c:312
#6 0x00003fff80b91140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 3 (LWP 17803):
#0 0x00003fff81240738 in __io_getevents_0_4 () from /lib64/libaio.so.1
#1 0x00003fff812407a0 in io_getevents () from /lib64/libaio.so.1
#2 0x000000005c2ec838 in LinuxAIOHandler::collect (this=0x3fff5f02e5f0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/os/os0file.cc:2519
#3 0x000000005c2ece70 in LinuxAIOHandler::poll (this=0x3fff5f02e5f0, m1=0x3fff5f02e6e0, m2=0x3fff5f02e6e8, request=0x3fff5f02e6f0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/os/os0file.cc:2665
#4 0x000000005c2ed040 in os_aio_linux_handler (global_segment=0, m1=0x3fff5f02e6e0, m2=0x3fff5f02e6e8, request=0x3fff5f02e6f0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/os/os0file.cc:2721
#5 0x000000005c2f33a8 in os_aio_handler (segment=0, m1=0x3fff5f02e6e0, m2=0x3fff5f02e6e8, request=0x3fff5f02e6f0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/os/os0file.cc:6450
#6 0x000000005c6626c8 in fil_aio_wait (segment=0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/fil/fil0fil.cc:5810
#7 0x000000005c466e80 in io_handler_thread (arg=0x5d3d5a30 <n>) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/srv/srv0start.cc:331
#8 0x00003fff812f7cec in start_thread (arg=0x3fff5f02f180) at pthread_create.c:312
#9 0x00003fff80b91140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 2 (LWP 17800):
#0 0x00003fff812ff7d0 in __pthread_cond_wait (cond=0x5d34db40 <COND_thread_count>, mutex=0x5d34c9e8 <LOCK_thread_count+40>) at pthread_cond_wait.c:188
#1 0x000000005c9c7b10 in safe_cond_wait (cond=0x5d34db40 <COND_thread_count>, mp=0x5d34c9c0 <LOCK_thread_count>, file=0x5cab96b8 "/home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/include/mysql/psi/mysql_thread.h", line=1138) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/mysys/thr_mutex.c:491
#2 0x000000005ba41560 in inline_mysql_cond_wait (that=0x5d34db40 <COND_thread_count>, mutex=0x5d34c9c0 <LOCK_thread_count>, src_file=0x5cabb690 "/home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/mysqld.cc", src_line=6098) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/include/mysql/psi/mysql_thread.h:1138
#3 0x000000005ba5157c in mysqld_main (argc=151, argv=0x1002c53c180) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/mysqld.cc:6098
#4 0x000000005ba40638 in main (argc=30, argv=0x3fffe9c7fce8) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/main.cc:25

Thread 1 (LWP 18606):
#0 0x00003fff80a9eeb8 in __GI_raise (sig=<optimized out>) at ../sysdeps/unix/sysv/linux/raise.c:55
#1 0x00003fff80aa19bc in __GI_abort () at abort.c:89
#2 0x000000005c4f8fdc in ut_dbg_assertion_failed (expr=0x0, file=0x5cc70e48 "/home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/ut/ut0ut.cc", line=949) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/ut/ut0dbg.cc:67
#3 0x000000005c4fe1d4 in ib::fatal::~fatal (this=0x3fff5f0b6610, __in_chrg=<optimized out>) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/ut/ut0ut.cc:949
#4 0x000000005c599430 in buf_all_freed_instance (buf_pool=0x1002ca9b2b0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/buf/buf0buf.cc:6230
#5 0x000000005c59ccc8 in buf_all_freed () at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/buf/buf0buf.cc:7219
#6 0x000000005c2b8ae0 in logs_empty_and_mark_files_at_shutdown () at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/log/log0log.cc:2415
#7 0x000000005c46f9c8 in innobase_shutdown_for_mysql () at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/srv/srv0start.cc:2801
#8 0x000000005c1d94b4 in innobase_end (hton=0x1002c70e160, type=HA_PANIC_CLOSE) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/handler/ha_innodb.cc:4844
#9 0x000000005becf0d0 in ha_finalize_handlerton (plugin=0x1002c6d04c8) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/handler.cc:451
#10 0x000000005bb9c914 in plugin_deinitialize (plugin=0x1002c6d04c8, ref_check=true) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/sql_plugin.cc:1217
#11 0x000000005bb9cfc8 in reap_plugins () at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/sql_plugin.cc:1294
#12 0x000000005bb9f80c in plugin_shutdown () at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/sql_plugin.cc:1945
#13 0x000000005ba46984 in clean_up (print_message=true) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/mysqld.cc:2201
#14 0x000000005ba462c4 in unireg_end () at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/mysqld.cc:2076
#15 0x000000005ba4608c in kill_server (sig_ptr=0x0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/mysqld.cc:2003
#16 0x000000005ba4610c in kill_server_thread (arg=0x3fff5f1d7280) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/mysqld.cc:2026
#17 0x000000005c93da00 in pfs_spawn_thread (arg=0x3fff1c001380) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/perfschema/pfs.cc:1862
#18 0x00003fff812f7cec in start_thread (arg=0x3fff5f0b7d80) at pthread_create.c:312
#19 0x00003fff80b91140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96
{noformat}

encryption.innodb-bad-key-change3 - failed several times on d6a1f9f10f21bfb5198b81c40e097556 and 08f79bdeda89f5b8e09d99e7dc07fc9c
http://buildbot.askmonty.org/buildbot/builders/p8-rhel71-bintar-debug/builds/1484/steps/test/logs/stdio
{noformat}
encryption.innodb-bad-key-change3 'innodb' w2 [ fail ] Found warnings/errors in server log file!
Test ended at 2016-12-22 11:29:29
line
Attempting backtrace. You can use the following information to find out
^ Found warnings in /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/mysql-test/var/2/log/mysqld.1.err
{noformat}
{noformat}
Program terminated with signal 6, Aborted.
#0 0x00003fff82cc0ef0 in __pthread_kill (threadid=<optimized out>, signo=<optimized out>) at ../sysdeps/unix/sysv/linux/pthread_kill.c:60
60 ../sysdeps/unix/sysv/linux/pthread_kill.c: No such file or directory.
#0 0x00003fff82cc0ef0 in __pthread_kill (threadid=<optimized out>, signo=<optimized out>) at ../sysdeps/unix/sysv/linux/pthread_kill.c:60
#1 0x000000003e0e0180 in my_write_core (sig=6) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/mysys/stacktrace.c:477
#2 0x000000003d5ec924 in handle_fatal_signal (sig=6) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/signal_handler.cc:294
#3 <signal handler called>
#4 0x00003fff8245eeb8 in __GI_raise (sig=<optimized out>) at ../sysdeps/unix/sysv/linux/raise.c:55
#5 0x00003fff824619bc in __GI_abort () at abort.c:89
#6 0x00003fff824ac224 in __libc_message (do_abort=<optimized out>, fmt=<optimized out>) at ../sysdeps/posix/libc_fatal.c:175
#7 0x00003fff824bbd58 in malloc_printerr (ptr=<optimized out>, str=0x3fff825c2b70 "munmap_chunk(): invalid pointer", action=3) at malloc.c:4958
#8 munmap_chunk (p=<optimized out>) at malloc.c:2815
#9 __GI___libc_free (mem=<optimized out>) at malloc.c:2940
#10 0x00003fff82796898 in operator delete(void*) () from /opt/at8.0/lib64/power8/libstdc++.so.6
#11 0x000000003dba9948 in LatchCounter::~LatchCounter (this=0x100078ed658, __in_chrg=<optimized out>) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/include/sync0types.h:586
#12 0x000000003dbacd80 in LatchMeta<LatchCounter>::~LatchMeta (this=0x100078ed630, __in_chrg=<optimized out>) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/include/sync0types.h:773
#13 0x000000003dba9238 in sync_latch_meta_destroy () at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/sync/sync0debug.cc:1593
#14 0x000000003dba9698 in sync_check_close () at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/sync/sync0debug.cc:1811
#15 0x000000003db8fcdc in innobase_shutdown_for_mysql () at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/srv/srv0start.cc:2886
#16 0x000000003d8f94b4 in innobase_end (hton=0x100078cc800, type=HA_PANIC_CLOSE) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/handler/ha_innodb.cc:4844
#17 0x000000003d5ef0d0 in ha_finalize_handlerton (plugin=0x10007890428) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/handler.cc:451
#18 0x000000003d2bc914 in plugin_deinitialize (plugin=0x10007890428, ref_check=true) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/sql_plugin.cc:1217
#19 0x000000003d2bcfc8 in reap_plugins () at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/sql_plugin.cc:1294
#20 0x000000003d2bf80c in plugin_shutdown () at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/sql_plugin.cc:1945
#21 0x000000003d166984 in clean_up (print_message=true) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/mysqld.cc:2201
#22 0x000000003d1662c4 in unireg_end () at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/mysqld.cc:2076
#23 0x000000003d16608c in kill_server (sig_ptr=0x0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/mysqld.cc:2003
#24 0x000000003d16610c in kill_server_thread (arg=0x3fff60bc7280) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/mysqld.cc:2026
#25 0x000000003e05da00 in pfs_spawn_thread (arg=0x3fff28001380) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/perfschema/pfs.cc:1862
#26 0x00003fff82cb7cec in start_thread (arg=0x3fff60aa7d80) at pthread_create.c:312
#27 0x00003fff82551140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 4 (LWP 24982):
#0 0x00003fff82cbfcc0 in __pthread_cond_timedwait (cond=0x3f3206b0 <COND_timer>, mutex=0x3f320628 <LOCK_timer+40>, abstime=0x3fff7dfde720) at pthread_cond_timedwait.c:199
#1 0x000000003e0e7eec in safe_cond_timedwait (cond=0x3f3206b0 <COND_timer>, mp=0x3f320600 <LOCK_timer>, abstime=0x3fff7dfde720, file=0x3e4434f8 "/home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/include/mysql/psi/mysql_thread.h", line=1175) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/mysys/thr_mutex.c:545
#2 0x000000003e0e97b4 in inline_mysql_cond_timedwait (that=0x3f3206b0 <COND_timer>, mutex=0x3f320600 <LOCK_timer>, abstime=0x3fff7dfde720, src_file=0x3e443568 "/home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/mysys/thr_timer.c", src_line=292) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/include/mysql/psi/mysql_thread.h:1175
#3 0x000000003e0ea7cc in timer_handler (arg=0x0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/mysys/thr_timer.c:292
#4 0x000000003e05da00 in pfs_spawn_thread (arg=0x1000788bfe0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/perfschema/pfs.cc:1862
#5 0x00003fff82cb7cec in start_thread (arg=0x3fff7dfdf180) at pthread_create.c:312
#6 0x00003fff82551140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 3 (LWP 25006):
#0 0x00003fff82cc5714 in do_sigwait (sig=0x3fff60bc7280, set=<optimized out>) at ../sysdeps/unix/sysv/linux/sigwait.c:61
#1 __sigwait (set=<optimized out>, sig=0x3fff60bc7280) at ../sysdeps/unix/sysv/linux/sigwait.c:98
#2 0x000000003d16a624 in signal_hand (arg=0x0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/mysqld.cc:3560
#3 0x000000003e05da00 in pfs_spawn_thread (arg=0x1000809cde0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/perfschema/pfs.cc:1862
#4 0x00003fff82cb7cec in start_thread (arg=0x3fff60bc7d80) at pthread_create.c:312
#5 0x00003fff82551140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 2 (LWP 24981):
#0 0x00003fff82cbf7d0 in __pthread_cond_wait (cond=0x3ea6db40 <COND_thread_count>, mutex=0x3ea6c9e8 <LOCK_thread_count+40>) at pthread_cond_wait.c:188
#1 0x000000003e0e7b10 in safe_cond_wait (cond=0x3ea6db40 <COND_thread_count>, mp=0x3ea6c9c0 <LOCK_thread_count>, file=0x3e1d96b8 "/home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/include/mysql/psi/mysql_thread.h", line=1138) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/mysys/thr_mutex.c:491
#2 0x000000003d161560 in inline_mysql_cond_wait (that=0x3ea6db40 <COND_thread_count>, mutex=0x3ea6c9c0 <LOCK_thread_count>, src_file=0x3e1db690 "/home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/mysqld.cc", src_line=6098) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/include/mysql/psi/mysql_thread.h:1138
#3 0x000000003d17157c in mysqld_main (argc=143, argv=0x100076fc180) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/mysqld.cc:6098
#4 0x000000003d160638 in main (argc=22, argv=0x3ffffddac5e8) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/main.cc:25

Thread 1 (LWP 25019):
#0 0x00003fff82cc0ef0 in __pthread_kill (threadid=<optimized out>, signo=<optimized out>) at ../sysdeps/unix/sysv/linux/pthread_kill.c:60
#1 0x000000003e0e0180 in my_write_core (sig=6) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/mysys/stacktrace.c:477
#2 0x000000003d5ec924 in handle_fatal_signal (sig=6) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/signal_handler.cc:294
#3 <signal handler called>
#4 0x00003fff8245eeb8 in __GI_raise (sig=<optimized out>) at ../sysdeps/unix/sysv/linux/raise.c:55
#5 0x00003fff824619bc in __GI_abort () at abort.c:89
#6 0x00003fff824ac224 in __libc_message (do_abort=<optimized out>, fmt=<optimized out>) at ../sysdeps/posix/libc_fatal.c:175
#7 0x00003fff824bbd58 in malloc_printerr (ptr=<optimized out>, str=0x3fff825c2b70 "munmap_chunk(): invalid pointer", action=3) at malloc.c:4958
#8 munmap_chunk (p=<optimized out>) at malloc.c:2815
#9 __GI___libc_free (mem=<optimized out>) at malloc.c:2940
#10 0x00003fff82796898 in operator delete(void*) () from /opt/at8.0/lib64/power8/libstdc++.so.6
#11 0x000000003dba9948 in LatchCounter::~LatchCounter (this=0x100078ed658, __in_chrg=<optimized out>) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/include/sync0types.h:586
#12 0x000000003dbacd80 in LatchMeta<LatchCounter>::~LatchMeta (this=0x100078ed630, __in_chrg=<optimized out>) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/include/sync0types.h:773
#13 0x000000003dba9238 in sync_latch_meta_destroy () at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/sync/sync0debug.cc:1593
#14 0x000000003dba9698 in sync_check_close () at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/sync/sync0debug.cc:1811
#15 0x000000003db8fcdc in innobase_shutdown_for_mysql () at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/srv/srv0start.cc:2886
#16 0x000000003d8f94b4 in innobase_end (hton=0x100078cc800, type=HA_PANIC_CLOSE) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/innobase/handler/ha_innodb.cc:4844
#17 0x000000003d5ef0d0 in ha_finalize_handlerton (plugin=0x10007890428) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/handler.cc:451
#18 0x000000003d2bc914 in plugin_deinitialize (plugin=0x10007890428, ref_check=true) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/sql_plugin.cc:1217
#19 0x000000003d2bcfc8 in reap_plugins () at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/sql_plugin.cc:1294
#20 0x000000003d2bf80c in plugin_shutdown () at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/sql_plugin.cc:1945
#21 0x000000003d166984 in clean_up (print_message=true) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/mysqld.cc:2201
#22 0x000000003d1662c4 in unireg_end () at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/mysqld.cc:2076
#23 0x000000003d16608c in kill_server (sig_ptr=0x0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/mysqld.cc:2003
#24 0x000000003d16610c in kill_server_thread (arg=0x3fff60bc7280) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/mysqld.cc:2026
#25 0x000000003e05da00 in pfs_spawn_thread (arg=0x3fff28001380) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/perfschema/pfs.cc:1862
#26 0x00003fff82cb7cec in start_thread (arg=0x3fff60aa7d80) at pthread_create.c:312
#27 0x00003fff82551140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96
{noformat}

Elena Stepanova made changes - 2016-12-22 21:51

Summary

encryption.innodb_onlinealter_encryption, encryption.innodb-bad-key-change3 fail in buildbot (crash on shutdown)

encryption.innodb_onlinealter_encryption, encryption.innodb-bad-key-change3, encryption.innodb-discard-import fail in buildbot (crash on shutdown)

Elena Stepanova added a comment - 2016-12-29 22:22

Variation of a failure in innobase_shutdown_for_mysql:

#3  <signal handler called>

#4  0x00007f7740fef067 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56

#5  0x00007f7740ff0448 in __GI_abort () at abort.c:89

#6  0x00007f774376e284 in ut_dbg_assertion_failed (expr=expr@entry=0x7f7743c32b03 "node->n_pending == 0", file=file@entry=0x7f7743c30090 "/data/src/10.2/storage/innobase/fil/fil0fil.cc", line=line@entry=880) at /data/src/10.2/storage/innobase/ut/ut0dbg.cc:67

#7  0x00007f774385ce3e in fil_node_close_file (node=node@entry=0x7f773cff9780) at /data/src/10.2/storage/innobase/fil/fil0fil.cc:880

#8  0x00007f774385ec0d in fil_close_all_files () at /data/src/10.2/storage/innobase/fil/fil0fil.cc:1820

#9  0x00007f77435e697a in logs_empty_and_mark_files_at_shutdown () at /data/src/10.2/storage/innobase/log/log0log.cc:2409

#10 0x00007f7743701b1e in innobase_shutdown_for_mysql () at /data/src/10.2/storage/innobase/srv/srv0start.cc:2802

#11 0x00007f774355b346 in innobase_end (hton=<optimized out>, type=<optimized out>) at /data/src/10.2/storage/innobase/handler/ha_innodb.cc:4862

#12 0x00007f774336d2e2 in ha_finalize_handlerton (plugin=0x7f773bfdd880) at /data/src/10.2/sql/handler.cc:451

#13 0x00007f77431974a2 in plugin_deinitialize (plugin=0x7f773bfdd880, ref_check=ref_check@entry=true) at /data/src/10.2/sql/sql_plugin.cc:1217

#14 0x00007f774319bba0 in reap_plugins () at /data/src/10.2/sql/sql_plugin.cc:1294

#15 0x00007f774319c5e6 in plugin_shutdown () at /data/src/10.2/sql/sql_plugin.cc:1945

#16 0x00007f77430d2e41 in clean_up (print_message=print_message@entry=true) at /data/src/10.2/sql/mysqld.cc:2201

#17 0x00007f77430d75ae in unireg_end () at /data/src/10.2/sql/mysqld.cc:2076

#18 0x00007f77430de21b in kill_server (sig_ptr=sig_ptr@entry=0x0) at /data/src/10.2/sql/mysqld.cc:2003

#19 0x00007f77430de28b in kill_server_thread (arg=arg@entry=0x7f774421d884) at /data/src/10.2/sql/mysqld.cc:2026

#20 0x00007f77435329d7 in pfs_spawn_thread (arg=0x7f771802c0f0) at /data/src/10.2/storage/perfschema/pfs.cc:1862

#21 0x00007f77428820a4 in start_thread (arg=0x7f7744140300) at pthread_create.c:309

#22 0x00007f77410a287d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Elena Stepanova added a comment - 2016-12-29 22:22 Variation of a failure in innobase_shutdown_for_mysql : #3 <signal handler called> #4 0x00007f7740fef067 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56 #5 0x00007f7740ff0448 in __GI_abort () at abort.c:89 #6 0x00007f774376e284 in ut_dbg_assertion_failed (expr=expr@entry=0x7f7743c32b03 "node->n_pending == 0", file=file@entry=0x7f7743c30090 "/data/src/10.2/storage/innobase/fil/fil0fil.cc", line=line@entry=880) at /data/src/10.2/storage/innobase/ut/ut0dbg.cc:67 #7 0x00007f774385ce3e in fil_node_close_file (node=node@entry=0x7f773cff9780) at /data/src/10.2/storage/innobase/fil/fil0fil.cc:880 #8 0x00007f774385ec0d in fil_close_all_files () at /data/src/10.2/storage/innobase/fil/fil0fil.cc:1820 #9 0x00007f77435e697a in logs_empty_and_mark_files_at_shutdown () at /data/src/10.2/storage/innobase/log/log0log.cc:2409 #10 0x00007f7743701b1e in innobase_shutdown_for_mysql () at /data/src/10.2/storage/innobase/srv/srv0start.cc:2802 #11 0x00007f774355b346 in innobase_end (hton=<optimized out>, type=<optimized out>) at /data/src/10.2/storage/innobase/handler/ha_innodb.cc:4862 #12 0x00007f774336d2e2 in ha_finalize_handlerton (plugin=0x7f773bfdd880) at /data/src/10.2/sql/handler.cc:451 #13 0x00007f77431974a2 in plugin_deinitialize (plugin=0x7f773bfdd880, ref_check=ref_check@entry=true) at /data/src/10.2/sql/sql_plugin.cc:1217 #14 0x00007f774319bba0 in reap_plugins () at /data/src/10.2/sql/sql_plugin.cc:1294 #15 0x00007f774319c5e6 in plugin_shutdown () at /data/src/10.2/sql/sql_plugin.cc:1945 #16 0x00007f77430d2e41 in clean_up (print_message=print_message@entry=true) at /data/src/10.2/sql/mysqld.cc:2201 #17 0x00007f77430d75ae in unireg_end () at /data/src/10.2/sql/mysqld.cc:2076 #18 0x00007f77430de21b in kill_server (sig_ptr=sig_ptr@entry=0x0) at /data/src/10.2/sql/mysqld.cc:2003 #19 0x00007f77430de28b in kill_server_thread (arg=arg@entry=0x7f774421d884) at /data/src/10.2/sql/mysqld.cc:2026 #20 0x00007f77435329d7 in pfs_spawn_thread (arg=0x7f771802c0f0) at /data/src/10.2/storage/perfschema/pfs.cc:1862 #21 0x00007f77428820a4 in start_thread (arg=0x7f7744140300) at pthread_create.c:309 #22 0x00007f77410a287d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Marko Mäkelä added a comment - 2017-01-03 15:35 - edited

The assertion node->n_pending==0 fails for the InnoDB system tablespace.
The call to fil_close_all_files() in logs_empty_and_mark_files_at_shutdown() expects all files to be in an idle state. Somehow we failed to properly finish an I/O request at shutdown.

The buf_pool_ptr->flush_list is empty, but it is possible that a write from the buffer pool to the file buffers was completed, and the fsync() is still pending. There are 413 pages in the buffer pool. Many of these pages were modified while the server was running.
One example is (buf_page_t *) 0x7f772b3a79c0 which is space 0 (system tablespace), page 8 (root page of the clustered index of the SYS_TABLES table), newest_modification = 1804026. I suppose that this page was recently changed due to a DROP TABLE that was executed at the end of the test, just before the shutdown.

Interestingly, log_sys->lsn == log_sys->last_checkpoint_lsn == log_sys->next_checkpoint_lsn == 1804880. That is 854 bytes after the newest_modification of the above block. This should be OK, because newest_modification is never reset to 0. The changes probably were already written to the data file.

There are no clues in the buffer pool. The fil_node_t for which the assertion fails is this one:

(gdb) p *fil_system.space_list.start.chain.start

$110 = {space = 0x7f7740413a80, name = 0x7f7740418370 "./ibdata1",

  is_open = true, handle = 3, sync_event = 0x7f772cbf0c00,

  is_raw_disk = false, size = 768, init_size = 768,

  max_size = 18446744073709551614, n_pending = 1, n_pending_flushes = 0,

  being_extended = false, modification_counter = 200, flush_counter = 200,

  chain = {prev = 0x0, next = 0x0}, LRU = {prev = 0x0, next = 0x0},

  punch_hole = true, block_size = 4096, atomic_write = false, magic_n = 89389}

The node->n_pending is incremented by fil_node_prepare_for_io() and decremented by fil_node_complete_io(). The only case where a call to the latter could be missed is when a call to fil_aio_wait() is skipped or os_aio_handler() fails to return the fil_node_t pointer.

The only caller of fil_aio_wait() is io_handler_thread(). Maybe the termination condition needs to be revised so that pending work by encryption threads will be properly finished?

        while (srv_shutdown_state != SRV_SHUTDOWN_EXIT_THREADS

               || buf_page_cleaner_is_active

               || !os_aio_all_slots_free()) {

                fil_aio_wait(segment);

Marko Mäkelä added a comment - 2017-01-03 15:35 - edited The assertion node->n_pending==0 fails for the InnoDB system tablespace. The call to fil_close_all_files() in logs_empty_and_mark_files_at_shutdown() expects all files to be in an idle state. Somehow we failed to properly finish an I/O request at shutdown. The buf_pool_ptr->flush_list is empty, but it is possible that a write from the buffer pool to the file buffers was completed, and the fsync() is still pending. There are 413 pages in the buffer pool. Many of these pages were modified while the server was running. One example is (buf_page_t *) 0x7f772b3a79c0 which is space 0 (system tablespace), page 8 (root page of the clustered index of the SYS_TABLES table), newest_modification = 1804026. I suppose that this page was recently changed due to a DROP TABLE that was executed at the end of the test, just before the shutdown. Interestingly, log_sys->lsn == log_sys->last_checkpoint_lsn == log_sys->next_checkpoint_lsn == 1804880. That is 854 bytes after the newest_modification of the above block. This should be OK, because newest_modification is never reset to 0. The changes probably were already written to the data file. There are no clues in the buffer pool. The fil_node_t for which the assertion fails is this one: (gdb) p *fil_system.space_list.start.chain.start $110 = {space = 0x7f7740413a80, name = 0x7f7740418370 "./ibdata1", is_open = true, handle = 3, sync_event = 0x7f772cbf0c00, is_raw_disk = false, size = 768, init_size = 768, max_size = 18446744073709551614, n_pending = 1, n_pending_flushes = 0, being_extended = false, modification_counter = 200, flush_counter = 200, chain = {prev = 0x0, next = 0x0}, LRU = {prev = 0x0, next = 0x0}, punch_hole = true, block_size = 4096, atomic_write = false, magic_n = 89389} The node->n_pending is incremented by fil_node_prepare_for_io() and decremented by fil_node_complete_io(). The only case where a call to the latter could be missed is when a call to fil_aio_wait() is skipped or os_aio_handler() fails to return the fil_node_t pointer. The only caller of fil_aio_wait() is io_handler_thread(). Maybe the termination condition needs to be revised so that pending work by encryption threads will be properly finished? while (srv_shutdown_state != SRV_SHUTDOWN_EXIT_THREADS || buf_page_cleaner_is_active || !os_aio_all_slots_free()) { fil_aio_wait(segment); }

Rasmus Johansson (Inactive) made changes - 2017-01-04 10:00

Sprint

10.0.29 [ 128 ]

Rasmus Johansson (Inactive) made changes - 2017-01-04 10:00

Rank

Ranked higher

Marko Mäkelä added a comment - 2017-01-04 13:34

I cleaned up the test encryption.innodb_scrub (~~MDEV-8139~~) so that it creates and populates all tables, then does one slow shutdown, and checks the files while the server is offline. That test fails once in 10 to 20 runs. With that, I believe that I finally have a reliable way of repeating the root cause of these failures.

I suspect that the root cause is that srv_any_background_threads_are_active() fails to account for the encryption threads.

Marko Mäkelä added a comment - 2017-01-04 13:34 I cleaned up the test encryption.innodb_scrub ( MDEV-8139 ) so that it creates and populates all tables, then does one slow shutdown, and checks the files while the server is offline. That test fails once in 10 to 20 runs. With that, I believe that I finally have a reliable way of repeating the root cause of these failures. I suspect that the root cause is that srv_any_background_threads_are_active() fails to account for the encryption threads.

Marko Mäkelä made changes - 2017-01-04 13:34

Status

Open [ 1 ]

In Progress [ 3 ]

Marko Mäkelä made changes - 2017-01-04 16:44

Summary

encryption.innodb_onlinealter_encryption, encryption.innodb-bad-key-change3, encryption.innodb-discard-import fail in buildbot (crash on shutdown)

Encryption causes race conditions in InnoDB shutdown

Marko Mäkelä added a comment - 2017-01-04 18:55

See the penultimate commit in bb-10.1-mdev-8139.

Marko Mäkelä added a comment - 2017-01-04 18:55 See the penultimate commit in bb-10.1-mdev-8139.

Marko Mäkelä made changes - 2017-01-04 18:55

Assignee	Marko Mäkelä [ marko ]	Jan Lindström [ jplindst ]
Status	In Progress [ 3 ]	In Review [ 10002 ]

Jan Lindström (Inactive) added a comment - 2017-01-05 07:12

ok to push.

Jan Lindström (Inactive) added a comment - 2017-01-05 07:12 ok to push.

Jan Lindström (Inactive) made changes - 2017-01-05 07:12

Assignee	Jan Lindström [ jplindst ]	Marko Mäkelä [ marko ]
Status	In Review [ 10002 ]	Stalled [ 10000 ]

Marko Mäkelä made changes - 2017-01-05 08:53

Fix Version/s		10.1.21 [ 22113 ]
Fix Version/s		10.2.4 [ 22116 ]
Fix Version/s	10.2 [ 14601 ]
Resolution		Fixed [ 1 ]
Status	Stalled [ 10000 ]	Closed [ 6 ]

Elena Stepanova made changes - 2017-01-09 21:48

Resolution	Fixed [ 1 ]
Status	Closed [ 6 ]	Stalled [ 10000 ]

Marko Mäkelä made changes - 2017-01-10 07:47

Link

This issue relates to ~~MDEV-9359~~ [ ~~MDEV-9359~~ ]

Elena Stepanova made changes - 2017-01-10 11:39

Comment

[ The failure first appeared on bb-10.1-mdev8139 commit ffb38c9771bc7, and later in 10.1 tree.
Again, it is not limited at all to P8, but P8 produces a better stack trace, hence it is used in the example below.
{noformat:title=10.1 384f4d1e36cf2ba8791b15c83ac601b9}
encryption.innodb_encryption-page-compression 'xtradb' w2 [ fail ]
Test ended at 2017-01-07 19:49:05

CURRENT_TEST: encryption.innodb_encryption-page-compression
mysqltest: At line 127: query 'select count(*) from innodb_page_compressed3 where c1 < 500000' failed: 2013: Lost connection to MySQL server during query

The result from queries just before the failure was:
< snip >
variable_value >= 0
1
SELECT variable_value >= 0 FROM information_schema.global_status WHERE variable_name = 'innodb_num_pages_decrypted';
variable_value >= 0
1
SELECT variable_value >= 0 FROM information_schema.global_status WHERE variable_name = 'innodb_num_pages_page_compressed';
variable_value >= 0
1
SELECT variable_value >= 0 FROM information_schema.global_status WHERE variable_name = 'innodb_num_pages_page_decompressed';
variable_value >= 0
1
SET GLOBAL innodb_encryption_threads = 4;
SET GLOBAL innodb_encrypt_tables = off;
select count(*) from innodb_page_compressed1 where c1 < 500000;
count(*)
5000
select count(*) from innodb_page_compressed2 where c1 < 500000;
count(*)
5000
select count(*) from innodb_page_compressed3 where c1 < 500000;

More results from queries before failure can be found in /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/mysql-test/var/2/log/innodb_encryption-page-compression.log
{noformat}
{noformat}
Server [mysqld.1 - pid: 16847, winpid: 16847, exit: 256] failed during test run
Server log from this test:
----------SERVER LOG START-----------
2017-01-07 19:47:08 70367272794992 [Note] /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/sql/mysqld (mysqld 10.1.21-MariaDB-debug) starting as process 16425 ...
2017-01-07 19:47:08 70367272794992 [Warning] Could not increase number of max_open_files to more than 1024 (request: 4162)
2017-01-07 19:47:08 70367272794992 [Note] Plugin 'SEQUENCE' is disabled.
2017-01-07 19:47:08 70367272794992 [Note] Plugin 'partition' is disabled.
innodb_open_files should not be greater than the open_files_limit.
2017-01-07 19:47:09 70367272794992 [Note] InnoDB: innodb_empty_free_list_algorithm has been changed to legacy because of small buffer pool size. In order to use backoff, increase buffer pool at least up to 20MB.

2017-01-07 19:47:09 70367272794992 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2017-01-07 19:47:09 70367272794992 [Note] InnoDB: InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!!
2017-01-07 19:47:09 70367272794992 [Note] InnoDB: InnoDB: !!!!!!!! UNIV_SYNC_DEBUG switched on !!!!!!!!!
2017-01-07 19:47:09 70367272794992 [Note] InnoDB: The InnoDB memory heap is disabled
2017-01-07 19:47:09 70367272794992 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2017-01-07 19:47:09 70367272794992 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2017-01-07 19:47:09 70367272794992 [Note] InnoDB: Compressed tables use zlib 1.2.8
2017-01-07 19:47:09 70367272794992 [Note] InnoDB: Using Linux native AIO
2017-01-07 19:47:09 70367272794992 [Note] InnoDB: Using POWER8 crc32 instructions
2017-01-07 19:47:09 70367272794992 [Note] InnoDB: Initializing buffer pool, size = 8.0M
2017-01-07 19:47:09 70367272794992 [Note] InnoDB: Completed initialization of buffer pool
2017-01-07 19:47:09 70367272794992 [Note] InnoDB: Created tablespace for space 0 name ./ibdata1 key_id 1 encryption 0.
2017-01-07 19:47:09 70367272794992 [Note] InnoDB: Created tablespace for space 4294967280 name ./ib_logfile0 key_id 0 encryption 0.
2017-01-07 19:47:09 70367272794992 [Note] InnoDB: Created tablespace for space 4294967281 name arch_log_space key_id 0 encryption 0.
2017-01-07 19:47:09 70367272794992 [Note] InnoDB: Highest supported file format is Barracuda.
2017-01-07 19:47:11 70367272794992 [Note] InnoDB: Created tablespace for space 2 name mysql/innodb_index_stats key_id 0 encryption 0.
2017-01-07 19:47:11 70367272794992 [Note] InnoDB: Created tablespace for space 1 name mysql/innodb_table_stats key_id 0 encryption 0.
2017-01-07 19:47:12 70367272794992 [Note] InnoDB: 128 rollback segment(s) are active.
2017-01-07 19:47:12 70367272794992 [Note] InnoDB: Waiting for purge to start
2017-01-07 19:47:12 70367272794992 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.34-79.1 started; log sequence number 1629767
2017-01-07 19:47:12 70367272794992 [Note] InnoDB: Creating #1 thread id 70366936428864 total threads 4.
2017-01-07 19:47:12 70367272794992 [Note] InnoDB: Creating #2 thread id 70366928040256 total threads 4.
2017-01-07 19:47:12 70367272794992 [Note] InnoDB: Creating #3 thread id 70366919651648 total threads 4.
2017-01-07 19:47:12 70367272794992 [Note] InnoDB: Creating #4 thread id 70366911263040 total threads 4.
2017-01-07 19:47:12 70366961594688 [Note] InnoDB: Dumping buffer pool(s) not yet started
2017-01-07 19:47:12 70367272794992 [Note] Plugin 'INNODB_SYS_DATAFILES' is disabled.
2017-01-07 19:47:12 70367272794992 [Note] Plugin 'INNODB_SYS_TABLESTATS' is disabled.
2017-01-07 19:47:12 70367272794992 [Note] Plugin 'INNODB_SYS_INDEXES' is disabled.
2017-01-07 19:47:12 70367272794992 [Note] Plugin 'XTRADB_RSEG' is disabled.
2017-01-07 19:47:12 70367272794992 [Note] Plugin 'CHANGED_PAGE_BITMAPS' is disabled.
2017-01-07 19:47:12 70367272794992 [Note] Plugin 'INNODB_FT_BEING_DELETED' is disabled.
2017-01-07 19:47:12 70367272794992 [Note] Plugin 'INNODB_LOCK_WAITS' is disabled.
2017-01-07 19:47:12 70367272794992 [Note] Plugin 'XTRADB_INTERNAL_HASH_TABLES' is disabled.
2017-01-07 19:47:12 70367272794992 [Note] Plugin 'INNODB_SYS_FIELDS' is disabled.
2017-01-07 19:47:12 70367272794992 [Note] Plugin 'INNODB_CMPMEM_RESET' is disabled.
2017-01-07 19:47:12 70367272794992 [Note] Plugin 'FEEDBACK' is disabled.
2017-01-07 19:47:12 70367272794992 [Note] Plugin 'INNODB_CMP' is disabled.
2017-01-07 19:47:12 70367272794992 [Note] Plugin 'INNODB_FT_INDEX_TABLE' is disabled.
2017-01-07 19:47:12 70367272794992 [Note] Plugin 'INNODB_SYS_TABLESPACES' is disabled.
2017-01-07 19:47:12 70367272794992 [Note] Plugin 'INNODB_MUTEXES' is disabled.
2017-01-07 19:47:12 70367272794992 [Note] Plugin 'INNODB_CMP_RESET' is disabled.
2017-01-07 19:47:12 70367272794992 [Note] Plugin 'INNODB_FT_INDEX_CACHE' is disabled.
2017-01-07 19:47:12 70367272794992 [Note] Plugin 'INNODB_FT_DEFAULT_STOPWORD' is disabled.
2017-01-07 19:47:12 70367272794992 [Note] Plugin 'INNODB_SYS_COLUMNS' is disabled.
2017-01-07 19:47:12 70367272794992 [Note] Plugin 'INNODB_FT_CONFIG' is disabled.
2017-01-07 19:47:12 70367272794992 [Note] Plugin 'INNODB_CMP_PER_INDEX_RESET' is disabled.
2017-01-07 19:47:12 70367272794992 [Note] Plugin 'XTRADB_READ_VIEW' is disabled.
2017-01-07 19:47:12 70367272794992 [Note] Plugin 'INNODB_SYS_SEMAPHORE_WAITS' is disabled.
2017-01-07 19:47:12 70367272794992 [Note] Plugin 'INNODB_CHANGED_PAGES' is disabled.
2017-01-07 19:47:13 70367272794992 [Note] Plugin 'INNODB_FT_DELETED' is disabled.
2017-01-07 19:47:13 70367272794992 [Note] Plugin 'INNODB_TABLESPACES_SCRUBBING' is disabled.
2017-01-07 19:47:13 70367272794992 [Warning] /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/sql/mysqld: unknown option '--loose-pam-debug'
2017-01-07 19:47:13 70367272794992 [Note] Server socket created on IP: '127.0.0.1'.
2017-01-07 19:47:13 70367272794992 [Note] /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/sql/mysqld: ready for connections.
Version: '10.1.21-MariaDB-debug' socket: '/var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/mysql-test/var/tmp/2/mysqld.1.sock' port: 16040 Source distribution
2017-01-07 19:47:14 70367261390144 [Note] InnoDB: Created tablespace for space 4 name test/innodb_normal key_id 1 encryption 0.
2017-01-07 19:47:14 70367261390144 [Note] InnoDB: Created tablespace for space 5 name test/innodb_page_compressed1 key_id 1 encryption 0.
2017-01-07 19:47:14 70367261390144 [Note] InnoDB: Created tablespace for space 6 name test/innodb_page_compressed2 key_id 1 encryption 0.
2017-01-07 19:47:14 70367261390144 [Note] InnoDB: Created tablespace for space 7 name test/innodb_page_compressed3 key_id 1 encryption 0.
2017-01-07 19:47:15 70367261390144 [Note] InnoDB: Created tablespace for space 8 name test/innodb_page_compressed4 key_id 1 encryption 0.
2017-01-07 19:47:15 70367261390144 [Note] InnoDB: Created tablespace for space 9 name test/innodb_page_compressed5 key_id 1 encryption 0.
2017-01-07 19:47:15 70367261390144 [Note] InnoDB: Created tablespace for space 10 name test/innodb_page_compressed6 key_id 1 encryption 0.
2017-01-07 19:47:15 70367261390144 [Note] InnoDB: Created tablespace for space 11 name test/innodb_page_compressed7 key_id 1 encryption 0.
2017-01-07 19:47:15 70367261390144 [Note] InnoDB: Created tablespace for space 12 name test/innodb_page_compressed8 key_id 1 encryption 0.
2017-01-07 19:47:16 70367261390144 [Note] InnoDB: Created tablespace for space 13 name test/innodb_page_compressed9 key_id 1 encryption 0.
2017-01-07 19:47:19 70366928040256 [Note] InnoDB: Read page 0 from tablespace for space 2 name mysql/innodb_index_stats key_id 0 encryption 0 handle 75.
2017-01-07 19:47:20 70366936428864 [Note] InnoDB: Read page 0 from tablespace for space 1 name mysql/innodb_table_stats key_id 0 encryption 0 handle 76.
2017-01-07 19:48:13 70367260996928 [Note] /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/sql/mysqld: Normal shutdown

2017-01-07 19:48:13 70367260996928 [Note] Event Scheduler: Purging the queue. 0 events
2017-01-07 19:48:13 70366944817472 [Note] InnoDB: FTS optimize thread exiting.
2017-01-07 19:48:13 70367260996928 [Note] InnoDB: Starting shutdown...
2017-01-07 19:48:14 70367260996928 [Note] InnoDB: Waiting for page_cleaner to finish flushing of buffer pool
2017-01-07 19:48:15 70367260996928 [Note] InnoDB: Shutdown completed; log sequence number 5764705
2017-01-07 19:48:15 70367260996928 [Note] Debug sync points hit: 567208
2017-01-07 19:48:15 70367260996928 [Note] Debug sync points executed: 0
2017-01-07 19:48:15 70367260996928 [Note] Debug sync points max active per thread: 0
2017-01-07 19:48:15 70367260996928 [Note] /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/sql/mysqld: Shutdown complete

2017-01-07 19:48:16 70366796282736 [Note] /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/sql/mysqld (mysqld 10.1.21-MariaDB-debug) starting as process 16849 ...
2017-01-07 19:48:16 70366796282736 [Warning] Could not increase number of max_open_files to more than 1024 (request: 4162)
2017-01-07 19:48:16 70366796282736 [Note] Plugin 'SEQUENCE' is disabled.
2017-01-07 19:48:16 70366796282736 [Note] Plugin 'partition' is disabled.
innodb_open_files should not be greater than the open_files_limit.
2017-01-07 19:48:16 70366796282736 [Note] InnoDB: innodb_empty_free_list_algorithm has been changed to legacy because of small buffer pool size. In order to use backoff, increase buffer pool at least up to 20MB.

2017-01-07 19:48:16 70366796282736 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2017-01-07 19:48:16 70366796282736 [Note] InnoDB: InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!!
2017-01-07 19:48:16 70366796282736 [Note] InnoDB: InnoDB: !!!!!!!! UNIV_SYNC_DEBUG switched on !!!!!!!!!
2017-01-07 19:48:16 70366796282736 [Note] InnoDB: The InnoDB memory heap is disabled
2017-01-07 19:48:16 70366796282736 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2017-01-07 19:48:16 70366796282736 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2017-01-07 19:48:16 70366796282736 [Note] InnoDB: Compressed tables use zlib 1.2.8
2017-01-07 19:48:16 70366796282736 [Note] InnoDB: Using Linux native AIO
2017-01-07 19:48:16 70366796282736 [Note] InnoDB: Using POWER8 crc32 instructions
2017-01-07 19:48:16 70366796282736 [Note] InnoDB: Initializing buffer pool, size = 8.0M
2017-01-07 19:48:16 70366796282736 [Note] InnoDB: Completed initialization of buffer pool
2017-01-07 19:48:16 70366796282736 [Note] InnoDB: Created tablespace for space 0 name ./ibdata1 key_id 1 encryption 0.
2017-01-07 19:48:16 70366796282736 [Note] InnoDB: Created tablespace for space 4294967280 name ./ib_logfile0 key_id 0 encryption 0.
2017-01-07 19:48:16 70366796282736 [Note] InnoDB: Created tablespace for space 4294967281 name arch_log_space key_id 0 encryption 0.
2017-01-07 19:48:16 70366796282736 [Note] InnoDB: Highest supported file format is Barracuda.
2017-01-07 19:48:16 70366796282736 [Note] InnoDB: Created tablespace for space 2 name mysql/innodb_index_stats key_id 0 encryption 0.
2017-01-07 19:48:16 70366796282736 [Note] InnoDB: Created tablespace for space 1 name mysql/innodb_table_stats key_id 0 encryption 0.
2017-01-07 19:48:16 70366796282736 [Note] InnoDB: Created tablespace for space 4 name test/innodb_normal key_id 0 encryption 0.
2017-01-07 19:48:16 70366796282736 [Note] InnoDB: Created tablespace for space 5 name test/innodb_page_compressed1 key_id 0 encryption 0.
2017-01-07 19:48:16 70366796282736 [Note] InnoDB: Created tablespace for space 6 name test/innodb_page_compressed2 key_id 0 encryption 0.
2017-01-07 19:48:16 70366796282736 [Note] InnoDB: Created tablespace for space 7 name test/innodb_page_compressed3 key_id 0 encryption 0.
2017-01-07 19:48:16 70366796282736 [Note] InnoDB: Created tablespace for space 8 name test/innodb_page_compressed4 key_id 0 encryption 0.
2017-01-07 19:48:16 70366796282736 [Note] InnoDB: Created tablespace for space 9 name test/innodb_page_compressed5 key_id 0 encryption 0.
2017-01-07 19:48:16 70366796282736 [Note] InnoDB: Created tablespace for space 10 name test/innodb_page_compressed6 key_id 0 encryption 0.
2017-01-07 19:48:16 70366796282736 [Note] InnoDB: Created tablespace for space 11 name test/innodb_page_compressed7 key_id 0 encryption 0.
2017-01-07 19:48:16 70366796282736 [Note] InnoDB: Created tablespace for space 12 name test/innodb_page_compressed8 key_id 0 encryption 0.
2017-01-07 19:48:16 70366796282736 [Note] InnoDB: Created tablespace for space 13 name test/innodb_page_compressed9 key_id 0 encryption 0.
2017-01-07 19:48:17 70366796282736 [Note] InnoDB: 128 rollback segment(s) are active.
2017-01-07 19:48:17 70366796282736 [Note] InnoDB: Waiting for purge to start
2017-01-07 19:48:17 70366796282736 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.34-79.1 started; log sequence number 5764705
2017-01-07 19:48:18 70366796282736 [Note] InnoDB: Creating #1 thread id 70366462472512 total threads 4.
2017-01-07 19:48:18 70366796282736 [Note] InnoDB: Creating #2 thread id 70366454083904 total threads 4.
2017-01-07 19:48:18 70366796282736 [Note] InnoDB: Creating #3 thread id 70366445695296 total threads 4.
2017-01-07 19:48:18 70366796282736 [Note] InnoDB: Creating #4 thread id 70366437306688 total threads 4.
2017-01-07 19:48:18 70366487638336 [Note] InnoDB: Dumping buffer pool(s) not yet started
2017-01-07 19:48:18 70366796282736 [Note] Plugin 'INNODB_SYS_DATAFILES' is disabled.
2017-01-07 19:48:18 70366796282736 [Note] Plugin 'INNODB_SYS_TABLESTATS' is disabled.
2017-01-07 19:48:18 70366796282736 [Note] Plugin 'INNODB_SYS_INDEXES' is disabled.
2017-01-07 19:48:18 70366796282736 [Note] Plugin 'XTRADB_RSEG' is disabled.
2017-01-07 19:48:18 70366796282736 [Note] Plugin 'CHANGED_PAGE_BITMAPS' is disabled.
2017-01-07 19:48:18 70366796282736 [Note] Plugin 'INNODB_FT_BEING_DELETED' is disabled.
2017-01-07 19:48:18 70366796282736 [Note] Plugin 'INNODB_LOCK_WAITS' is disabled.
2017-01-07 19:48:18 70366796282736 [Note] Plugin 'XTRADB_INTERNAL_HASH_TABLES' is disabled.
2017-01-07 19:48:18 70366796282736 [Note] Plugin 'INNODB_SYS_FIELDS' is disabled.
2017-01-07 19:48:18 70366796282736 [Note] Plugin 'INNODB_CMPMEM_RESET' is disabled.
2017-01-07 19:48:18 70366796282736 [Note] Plugin 'FEEDBACK' is disabled.
2017-01-07 19:48:18 70366796282736 [Note] Plugin 'INNODB_CMP' is disabled.
2017-01-07 19:48:18 70366796282736 [Note] Plugin 'INNODB_FT_INDEX_TABLE' is disabled.
2017-01-07 19:48:18 70366796282736 [Note] Plugin 'INNODB_SYS_TABLESPACES' is disabled.
2017-01-07 19:48:18 70366796282736 [Note] Plugin 'INNODB_MUTEXES' is disabled.
2017-01-07 19:48:18 70366796282736 [Note] Plugin 'INNODB_CMP_RESET' is disabled.
2017-01-07 19:48:18 70366796282736 [Note] Plugin 'INNODB_FT_INDEX_CACHE' is disabled.
2017-01-07 19:48:18 70366796282736 [Note] Plugin 'INNODB_FT_DEFAULT_STOPWORD' is disabled.
2017-01-07 19:48:18 70366796282736 [Note] Plugin 'INNODB_SYS_COLUMNS' is disabled.
2017-01-07 19:48:18 70366796282736 [Note] Plugin 'INNODB_FT_CONFIG' is disabled.
2017-01-07 19:48:18 70366796282736 [Note] Plugin 'INNODB_CMP_PER_INDEX_RESET' is disabled.
2017-01-07 19:48:18 70366796282736 [Note] Plugin 'XTRADB_READ_VIEW' is disabled.
2017-01-07 19:48:18 70366796282736 [Note] Plugin 'INNODB_SYS_SEMAPHORE_WAITS' is disabled.
2017-01-07 19:48:18 70366796282736 [Note] Plugin 'INNODB_CHANGED_PAGES' is disabled.
2017-01-07 19:48:18 70366796282736 [Note] Plugin 'INNODB_FT_DELETED' is disabled.
2017-01-07 19:48:18 70366796282736 [Note] Plugin 'INNODB_TABLESPACES_SCRUBBING' is disabled.
2017-01-07 19:48:18 70366796282736 [Warning] /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/sql/mysqld: unknown option '--loose-pam-debug'
2017-01-07 19:48:18 70366796282736 [Note] Server socket created on IP: '127.0.0.1'.
2017-01-07 19:48:18 70366796282736 [Note] /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/sql/mysqld: ready for connections.
Version: '10.1.21-MariaDB-debug' socket: '/var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/mysql-test/var/tmp/2/mysqld.1.sock' port: 16040 Source distribution
2017-01-07 19:48:18 70366578373952 [Note] InnoDB: Read page 0 from tablespace for space 5 name test/innodb_page_compressed1 key_id 1 encryption 0 handle 43.
2017-01-07 19:48:19 70366578373952 [Note] InnoDB: Read page 0 from tablespace for space 6 name test/innodb_page_compressed2 key_id 1 encryption 0 handle 44.
2017-01-07 19:48:19 70366578373952 [Note] InnoDB: Read page 0 from tablespace for space 7 name test/innodb_page_compressed3 key_id 1 encryption 0 handle 45.
2017-01-07 19:48:20 70366578373952 [Note] InnoDB: Read page 0 from tablespace for space 8 name test/innodb_page_compressed4 key_id 1 encryption 0 handle 46.
2017-01-07 19:48:20 70366578373952 [Note] InnoDB: Read page 0 from tablespace for space 9 name test/innodb_page_compressed5 key_id 1 encryption 0 handle 47.
2017-01-07 19:48:20 70366578373952 [Note] InnoDB: Read page 0 from tablespace for space 10 name test/innodb_page_compressed6 key_id 1 encryption 0 handle 48.
2017-01-07 19:48:21 70366578373952 [Note] InnoDB: Read page 0 from tablespace for space 11 name test/innodb_page_compressed7 key_id 1 encryption 0 handle 49.
2017-01-07 19:48:21 70366578373952 [Note] InnoDB: Read page 0 from tablespace for space 12 name test/innodb_page_compressed8 key_id 1 encryption 0 handle 50.
2017-01-07 19:48:22 70366578373952 [Note] InnoDB: Read page 0 from tablespace for space 13 name test/innodb_page_compressed9 key_id 1 encryption 0 handle 51.
2017-01-07 19:48:38 70366462472512 [Note] InnoDB: Read page 0 from tablespace for space 2 name mysql/innodb_index_stats key_id 1 encryption 0 handle 52.
2017-01-07 19:48:38 70366437306688 [Note] InnoDB: Read page 0 from tablespace for space 1 name mysql/innodb_table_stats key_id 1 encryption 0 handle 53.
2017-01-07 19:48:40 70366462472512 [Note] InnoDB: Read page 0 from tablespace for space 4 name test/innodb_normal key_id 1 encryption 0 handle 54.
2017-01-07 19:48:40 70366437306688 [Note] InnoDB: Read page 0 from tablespace for space 4 name test/innodb_normal key_id 1 encryption 0 handle 54.
{noformat}
{noformat}
2017-01-07 19:48:40 3fff77fff140 InnoDB: Assertion failure in thread 70366462472512 in file fil0crypt.cc line 2220
InnoDB: Failing assertion: crypt_data->rotate_state.active_threads > 0
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
170107 19:48:40 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.1.21-MariaDB-debug
key_buffer_size=1048576
read_buffer_size=131072
max_used_connections=1
max_threads=153
thread_count=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 62983 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x48400
/var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/sql/mysqld(my_print_stacktrace+0x3c)[0x5718c578]
/var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/sql/mysqld(handle_fatal_signal+0x4f4)[0x56865db4]
linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0x3fff8bde0478]
/opt/at8.0/lib64/power8/libc.so.6(gsignal+0x48)[0x3fff8b56eeb8]
/opt/at8.0/lib64/power8/libc.so.6(abort+0x2ac)[0x3fff8b571a5c]
/var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/sql/mysqld(+0xe3405c)[0x56e7405c]
/var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/sql/mysqld(+0xe34504)[0x56e74504]
/opt/at8.0/lib64/power8/libpthread.so.0(+0x7cec)[0x3fff8bda7cec]
/opt/at8.0/lib64/power8/libc.so.6(clone+0x98)[0x3fff8b661140]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Writing a core file
----------SERVER LOG END-------------

- saving '/var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/mysql-test/var/2/log/encryption.innodb_encryption-page-compression-xtradb/' to '/var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/mysql-test/var/log/encryption.innodb_encryption-page-compression-xtradb/'
- found 'core' (1/5)

Trying 'dbx' to get a backtrace

Trying 'gdb' to get a backtrace
Core generated by '/var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/sql/mysqld'
Output from gdb follows. The first stack trace is from the failing thread.
The following stack traces are from all threads (so the failing one is
duplicated).

{noformat}
{noformat}
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/opt/at8.0/lib64/power8/libthread_db.so.1".
Core was generated by `/var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/sql/mysqld --default'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00003fff8bdb0ef0 in __pthread_kill (threadid=<optimized out>, signo=<optimized out>) at ../sysdeps/unix/sysv/linux/pthread_kill.c:60
60 ../sysdeps/unix/sysv/linux/pthread_kill.c: No such file or directory.
#0 0x00003fff8bdb0ef0 in __pthread_kill (threadid=<optimized out>, signo=<optimized out>) at ../sysdeps/unix/sysv/linux/pthread_kill.c:60
#1 0x000000005718c718 in my_write_core (sig=6) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/mysys/stacktrace.c:477
#2 0x0000000056866324 in handle_fatal_signal (sig=6) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/sql/signal_handler.cc:296
#3 <signal handler called>
#4 0x00003fff8b56eeb8 in __GI_raise (sig=<optimized out>) at ../sysdeps/unix/sysv/linux/raise.c:55
#5 0x00003fff8b5719bc in __GI_abort () at abort.c:89
#6 0x0000000056e7405c in fil_crypt_complete_rotate_space (key_state=0x3fff77ffe694, state=0x3fff77ffe6a0) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/fil/fil0crypt.cc:2220
#7 0x0000000056e74504 in fil_crypt_thread (arg=0x0) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/fil/fil0crypt.cc:2355
#8 0x00003fff8bda7cec in start_thread (arg=0x3fff77fff140) at pthread_create.c:312
#9 0x00003fff8b661140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 27 (Thread 0x3fff7afff140 (LWP 16878)):
#0 0x00003fff8bdb4908 in fsync () at ../sysdeps/unix/syscall-template.S:81
#1 0x0000000056bd35a0 in os_file_fsync (file=6) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/os/os0file.cc:2698
#2 0x0000000056bd3780 in os_file_flush_func (file=6) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/os/os0file.cc:2798
#3 0x0000000056e52800 in pfs_os_file_flush_func (file=6, src_file=0x5742a500 "/var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/fil/fil0fil.cc", src_line=1015) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/include/os0file.ic:371
#4 0x0000000056e598fc in fil_flush_low (space=0x3fff83660478) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/fil/fil0fil.cc:1015
#5 0x0000000056e65ee8 in fil_flush (space_id=0) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/fil/fil0fil.cc:6340
#6 0x0000000056dca6f0 in buf_dblwr_flush_buffered_writes () at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/buf/buf0dblwr.cc:1027
#7 0x0000000056dd7010 in buf_flush_common (flush_type=BUF_FLUSH_LIST, page_count=20) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/buf/buf0flu.cc:1843
#8 0x0000000056dd7b60 in buf_flush_list (min_n=200, lsn_limit=18446744073709551615, n_processed=0x3fff7affe6c8) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/buf/buf0flu.cc:2126
#9 0x0000000056dd86e8 in page_cleaner_do_flush_batch (n_to_flush=200, lsn_limit=18446744073709551615) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/buf/buf0flu.cc:2410
#10 0x0000000056dd99e8 in buf_flush_page_cleaner_thread (arg=0x0) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/buf/buf0flu.cc:2792
#11 0x00003fff8bda7cec in start_thread (arg=0x3fff7afff140) at pthread_create.c:312
#12 0x00003fff8b661140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 26 (Thread 0x3fff7ee87540 (LWP 16890)):
#0 0x0000000056c188f8 in thr_get_trx (thr=0x3fff81f5c2c0) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/include/que0que.ic:35
#1 0x0000000056c1aa34 in que_thr_stop_for_mysql_no_error (thr=0x3fff74b23708, trx=0x3fff74b60478) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/que/que0que.cc:883
#2 0x0000000056cbd160 in row_search_for_mysql (buf=0x3fff7e7e2488 "\375$\006", mode=1, prebuilt=0x3fff74b23078, match_mode=0, direction=1) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/row/row0sel.cc:5235
#3 0x0000000056b09584 in ha_innobase::general_fetch (this=0x3fff7eb93088, buf=0x3fff7e7e2488 "\375$\006", direction=1, match_mode=0) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/handler/ha_innodb.cc:10244
#4 0x0000000056b09d98 in ha_innobase::rnd_next (this=0x3fff7eb93088, buf=0x3fff7e7e2488 "\375$\006") at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/handler/ha_innodb.cc:10456
#5 0x000000005686f70c in handler::ha_rnd_next (this=0x3fff7eb93088, buf=0x3fff7e7e2488 "\375$\006") at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/sql/handler.cc:2578
#6 0x0000000056a583ac in rr_sequential (info=0x3fff7ea18510) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/sql/records.cc:470
#7 0x00000000565e61d4 in sub_select (join=0x3fff7ea16d80, join_tab=0x3fff7ea18450, end_of_records=false) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/sql/sql_select.cc:18323
#8 0x00000000565e55b8 in do_select (join=0x3fff7ea16d80, fields=0x3fff7ea17170, table=0x0, procedure=0x0) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/sql/sql_select.cc:17959
#9 0x00000000565b83a8 in JOIN::exec_inner (this=0x3fff7ea16d80) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/sql/sql_select.cc:3215
#10 0x00000000565b56cc in JOIN::exec (this=0x3fff7ea16d80) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/sql/sql_select.cc:2505
#11 0x00000000565b8e00 in mysql_select (thd=0x3fff85556070, rref_pointer_array=0x3fff8555a430, tables=0x3fff7ea163b0, wild_num=0, fields=..., conds=0x3fff7ea16b40, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=2147748608, result=0x3fff7ea16d60, unit=0x3fff85559ab8, select_lex=0x3fff8555a1b8) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/sql/sql_select.cc:3439
#12 0x00000000565adfac in handle_select (thd=0x3fff85556070, lex=0x3fff855599f0, result=0x3fff7ea16d60, setup_tables_done_option=0) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/sql/sql_select.cc:384
#13 0x000000005656b710 in execute_sqlcom_select (thd=0x3fff85556070, all_tables=0x3fff7ea163b0) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/sql/sql_parse.cc:5899
#14 0x00000000565613b4 in mysql_execute_command (thd=0x3fff85556070) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/sql/sql_parse.cc:2971
#15 0x000000005656f814 in mysql_parse (thd=0x3fff85556070, rawbuf=0x3fff7ea16088 "select count(*) from innodb_page_compressed3 where c1 < 500000", length=62, parser_state=0x3fff7ee85dd0) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/sql/sql_parse.cc:7322
#16 0x000000005655c6c8 in dispatch_command (command=COM_QUERY, thd=0x3fff85556070, packet=0x3fff8a439071 "", packet_length=62) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/sql/sql_parse.cc:1488
#17 0x000000005655b174 in do_command (thd=0x3fff85556070) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/sql/sql_parse.cc:1109
#18 0x00000000566f39e8 in do_handle_one_connection (thd_arg=0x3fff85556070) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/sql/sql_connect.cc:1350
#19 0x00000000566f3648 in handle_one_connection (arg=0x3fff85556070) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/sql/sql_connect.cc:1262
#20 0x0000000057102fec in pfs_spawn_thread (arg=0x3fff8367d5f0) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/perfschema/pfs.cc:1860
#21 0x00003fff8bda7cec in start_thread (arg=0x3fff7ee87540) at pthread_create.c:312
#22 0x00003fff8b661140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 25 (Thread 0x3fff767ff140 (LWP 16886)):
#0 0x0000000056b248f8 in os_atomic_test_and_set (ptr=0x5793e5a8 <sync_thread_mutex+104> "\001") at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/include/os0sync.h:529
#1 0x0000000056cfdc80 in ib_mutex_test_and_set (mutex=0x5793e540 <sync_thread_mutex>) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/include/sync0sync.ic:86
#2 0x0000000056cfdf58 in mutex_enter_func (mutex=0x5793e540 <sync_thread_mutex>, file_name=0x573f15f0 "/var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/sync/sync0sync.cc", line=1200) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/include/sync0sync.ic:268
#3 0x0000000056cfe0a4 in pfs_mutex_enter_func (mutex=0x5793e540 <sync_thread_mutex>, file_name=0x573f15f0 "/var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/sync/sync0sync.cc", line=1200) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/include/sync0sync.ic:375
#4 0x0000000056d00124 in sync_thread_add_level (latch=0x3fff8a7d29b8, level=142, relock=0) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/sync/sync0sync.cc:1200
#5 0x0000000056cff344 in mutex_set_debug_info (mutex=0x3fff8a7d29b8, file_name=0x57413b88 "/var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/buf/buf0flu.cc", line=1869) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/sync/sync0sync.cc:729
#6 0x0000000056dce58c in mutex_enter_func (mutex=0x3fff8a7d29b8, file_name=0x57413b88 "/var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/buf/buf0flu.cc", line=1869) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/include/sync0sync.ic:271
#7 0x0000000056dce88c in pfs_mutex_enter_func (mutex=0x3fff8a7d29b8, file_name=0x57413b88 "/var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/buf/buf0flu.cc", line=1869) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/include/sync0sync.ic:375
#8 0x0000000056dd7128 in buf_flush_start (buf_pool=0x3fff8a7d2478, flush_type=BUF_FLUSH_LIST) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/buf/buf0flu.cc:1869
#9 0x0000000056dd791c in buf_flush_list (min_n=18446744073709551614, lsn_limit=9452341, n_processed=0x3fff767fe0a8) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/buf/buf0flu.cc:2090
#10 0x0000000056e73ca8 in fil_crypt_flush_space (state=0x3fff767fe6a0, space=4) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/fil/fil0crypt.cc:2150
#11 0x0000000056e741ec in fil_crypt_complete_rotate_space (key_state=0x3fff767fe694, state=0x3fff767fe6a0) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/fil/fil0crypt.cc:2262
#12 0x0000000056e74504 in fil_crypt_thread (arg=0x0) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/fil/fil0crypt.cc:2355
#13 0x00003fff8bda7cec in start_thread (arg=0x3fff767ff140) at pthread_create.c:312
#14 0x00003fff8b661140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 24 (Thread 0x3fff787ff140 (LWP 16882)):
#0 0x00003fff8bdafcc0 in __pthread_cond_timedwait (cond=0x3fff85792188, mutex=0x3fff85792150, abstime=0x3fff787fe5f0) at pthread_cond_timedwait.c:199
#1 0x0000000056bdb58c in os_cond_wait_timed (cond=0x3fff85792188, fast_mutex=0x3fff85792150, abstime=0x3fff787fe5f0) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/os/os0sync.cc:159
#2 0x0000000056bdbfa8 in os_event_wait_time_low (event=0x3fff85792150, time_in_usec=5000000, reset_sig_count=1) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/os/os0sync.cc:591
#3 0x0000000056d4b2c8 in ib_wqueue_timedwait (wq=0x3fff8aee3d78, wait_in_usecs=5000000) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/ut/ut0wqueue.cc:154
#4 0x0000000056eb2a88 in fts_optimize_thread (arg=0x3fff8aee3d78) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/fts/fts0opt.cc:3065
#5 0x00003fff8bda7cec in start_thread (arg=0x3fff787ff140) at pthread_create.c:312
#6 0x00003fff8b661140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 23 (Thread 0x3fff78fff140 (LWP 16881)):
#0 0x00003fff8bdafcc0 in __pthread_cond_timedwait (cond=0x3fff85791b68, mutex=0x3fff85791b30, abstime=0x3fff78ffe700) at pthread_cond_timedwait.c:199
#1 0x0000000056bdb58c in os_cond_wait_timed (cond=0x3fff85791b68, fast_mutex=0x3fff85791b30, abstime=0x3fff78ffe700) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/os/os0sync.cc:159
#2 0x0000000056bdbfa8 in os_event_wait_time_low (event=0x3fff85791b30, time_in_usec=10000000, reset_sig_count=1) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/os/os0sync.cc:591
#3 0x0000000056e48a70 in dict_stats_thread () at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/dict/dict0stats_bg.cc:542
#4 0x00003fff8bda7cec in start_thread (arg=0x3fff78fff140) at pthread_create.c:312
#5 0x00003fff8b661140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 22 (Thread 0x3fff7c7ff140 (LWP 16876)):
#0 0x00003fff8b653c14 in select () at ../sysdeps/unix/syscall-template.S:81
#1 0x0000000056bdcc80 in os_thread_sleep (tm=1000000) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/os/os0thread.cc:303
#2 0x0000000056ce9274 in srv_master_sleep () at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/srv/srv0srv.cc:3139
#3 0x0000000056ce94a4 in srv_master_thread (arg=0x0) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/srv/srv0srv.cc:3189
#4 0x00003fff8bda7cec in start_thread (arg=0x3fff7c7ff140) at pthread_create.c:312
#5 0x00003fff8b661140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 21 (Thread 0x3fff7cfff140 (LWP 16875)):
#0 0x00003fff8bdafcc0 in __pthread_cond_timedwait (cond=0x3fff85790358, mutex=0x3fff85790320, abstime=0x3fff7cffe670) at pthread_cond_timedwait.c:199
#1 0x0000000056bdb58c in os_cond_wait_timed (cond=0x3fff85790358, fast_mutex=0x3fff85790320, abstime=0x3fff7cffe670) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/os/os0sync.cc:159
#2 0x0000000056bdbfa8 in os_event_wait_time_low (event=0x3fff85790320, time_in_usec=5000000, reset_sig_count=1) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/os/os0sync.cc:591
#3 0x0000000056ce67c8 in srv_monitor_thread () at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/srv/srv0srv.cc:2135
#4 0x00003fff8bda7cec in start_thread (arg=0x3fff7cfff140) at pthread_create.c:312
#5 0x00003fff8b661140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 20 (Thread 0x3fff7d7ff140 (LWP 16874)):
#0 0x00003fff8bdafcc0 in __pthread_cond_timedwait (cond=0x3fff857902e8, mutex=0x3fff857902b0, abstime=0x3fff7d7fe650) at pthread_cond_timedwait.c:199
#1 0x0000000056bdb58c in os_cond_wait_timed (cond=0x3fff857902e8, fast_mutex=0x3fff857902b0, abstime=0x3fff7d7fe650) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/os/os0sync.cc:159
#2 0x0000000056bdbfa8 in os_event_wait_time_low (event=0x3fff857902b0, time_in_usec=1000000, reset_sig_count=1) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/os/os0sync.cc:591
#3 0x0000000056ce720c in srv_error_monitor_thread () at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/srv/srv0srv.cc:2406
#4 0x00003fff8bda7cec in start_thread (arg=0x3fff7d7ff140) at pthread_create.c:312
#5 0x00003fff8b661140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 19 (Thread 0x3fff7e3ff140 (LWP 16873)):
#0 0x00003fff8bdafcc0 in __pthread_cond_timedwait (cond=0x3fff857918c8, mutex=0x3fff85791890, abstime=0x3fff7e3fe6e0) at pthread_cond_timedwait.c:199
#1 0x0000000056bdb58c in os_cond_wait_timed (cond=0x3fff857918c8, fast_mutex=0x3fff85791890, abstime=0x3fff7e3fe6e0) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/os/os0sync.cc:159
#2 0x0000000056bdbfa8 in os_event_wait_time_low (event=0x3fff85791890, time_in_usec=1000000, reset_sig_count=1) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/os/os0sync.cc:591
#3 0x0000000056b969d0 in lock_wait_timeout_thread () at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/lock/lock0wait.cc:531
#4 0x00003fff8bda7cec in start_thread (arg=0x3fff7e3ff140) at pthread_create.c:312
#5 0x00003fff8b661140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 18 (Thread 0x3fff7f74f140 (LWP 16868)):
#0 0x00003fff8bd60688 in ?? () from /lib/powerpc64le-linux-gnu/libaio.so.1
#1 0x00003fff8bd606f0 in io_getevents () from /lib/powerpc64le-linux-gnu/libaio.so.1
#2 0x0000000056bd7d58 in os_aio_linux_collect (array=0x3fff837401a0, segment=1, seg_size=256) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/os/os0file.cc:5432
#3 0x0000000056bd8278 in os_aio_linux_handle (global_seg=5, message1=0x3fff7f74e6d8, message2=0x3fff7f74e6e0, type=0x3fff7f74e6e8, space_id=0x3fff7f74e6f0) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/os/os0file.cc:5585
#4 0x0000000056e65c68 in fil_aio_wait (segment=5) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/fil/fil0fil.cc:6276
#5 0x0000000056ceef48 in io_handler_thread (arg=0x57926c40 <n+40>) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/srv/srv0start.cc:546
#6 0x00003fff8bda7cec in start_thread (arg=0x3fff7f74f140) at pthread_create.c:312
#7 0x00003fff8b661140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 17 (Thread 0x3fff7ff4f140 (LWP 16867)):
#0 0x00003fff8bd60688 in ?? () from /lib/powerpc64le-linux-gnu/libaio.so.1
#1 0x00003fff8bd606f0 in io_getevents () from /lib/powerpc64le-linux-gnu/libaio.so.1
#2 0x0000000056bd7d58 in os_aio_linux_collect (array=0x3fff837401a0, segment=0, seg_size=256) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/os/os0file.cc:5432
#3 0x0000000056bd8278 in os_aio_linux_handle (global_seg=4, message1=0x3fff7ff4e6d8, message2=0x3fff7ff4e6e0, type=0x3fff7ff4e6e8, space_id=0x3fff7ff4e6f0) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/os/os0file.cc:5585
#4 0x0000000056e65c68 in fil_aio_wait (segment=4) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/fil/fil0fil.cc:6276
#5 0x0000000056ceef48 in io_handler_thread (arg=0x57926c38 <n+32>) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/srv/srv0start.cc:546
#6 0x00003fff8bda7cec in start_thread (arg=0x3fff7ff4f140) at pthread_create.c:312
#7 0x00003fff8b661140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 16 (Thread 0x3fff8074f140 (LWP 16866)):
#0 0x00003fff8bd60688 in ?? () from /lib/powerpc64le-linux-gnu/libaio.so.1
#1 0x00003fff8bd606f0 in io_getevents () from /lib/powerpc64le-linux-gnu/libaio.so.1
#2 0x0000000056bd7d58 in os_aio_linux_collect (array=0x3fff837400b0, segment=1, seg_size=256) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/os/os0file.cc:5432
#3 0x0000000056bd8278 in os_aio_linux_handle (global_seg=3, message1=0x3fff8074e6d8, message2=0x3fff8074e6e0, type=0x3fff8074e6e8, space_id=0x3fff8074e6f0) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/os/os0file.cc:5585
#4 0x0000000056e65c68 in fil_aio_wait (segment=3) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/fil/fil0fil.cc:6276
#5 0x0000000056ceef48 in io_handler_thread (arg=0x57926c30 <n+24>) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/srv/srv0start.cc:546
#6 0x00003fff8bda7cec in start_thread (arg=0x3fff8074f140) at pthread_create.c:312
#7 0x00003fff8b661140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 15 (Thread 0x3fff797ff140 (LWP 16880)):
#0 0x00003fff8bdaf7d0 in __pthread_cond_wait (cond=0x3fff857903c8, mutex=0x3fff85790390) at pthread_cond_wait.c:188
#1 0x0000000056bdb700 in os_cond_wait (cond=0x3fff857903c8, fast_mutex=0x3fff85790390) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/os/os0sync.cc:196
#2 0x0000000056bdbd14 in os_event_wait_low (event=0x3fff85790390, reset_sig_count=1) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/os/os0sync.cc:477
#3 0x0000000056dcde88 in buf_dump_thread () at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/buf/buf0dump.cc:699
#4 0x00003fff8bda7cec in start_thread (arg=0x3fff797ff140) at pthread_create.c:312
#5 0x00003fff8b661140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 14 (Thread 0x3fff80f4f140 (LWP 16865)):
#0 0x00003fff8bd60688 in ?? () from /lib/powerpc64le-linux-gnu/libaio.so.1
#1 0x00003fff8bd606f0 in io_getevents () from /lib/powerpc64le-linux-gnu/libaio.so.1
#2 0x0000000056bd7d58 in os_aio_linux_collect (array=0x3fff837400b0, segment=0, seg_size=256) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/os/os0file.cc:5432
#3 0x0000000056bd8278 in os_aio_linux_handle (global_seg=2, message1=0x3fff80f4e6d8, message2=0x3fff80f4e6e0, type=0x3fff80f4e6e8, space_id=0x3fff80f4e6f0) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/os/os0file.cc:5585
#4 0x0000000056e65c68 in fil_aio_wait (segment=2) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/fil/fil0fil.cc:6276
#5 0x0000000056ceef48 in io_handler_thread (arg=0x57926c28 <n+16>) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/srv/srv0start.cc:546
#6 0x00003fff8bda7cec in start_thread (arg=0x3fff80f4f140) at pthread_create.c:312
#7 0x00003fff8b661140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 13 (Thread 0x3fff7a7ff140 (LWP 16879)):
#0 0x00003fff8b653c14 in select () at ../sysdeps/unix/syscall-template.S:81
#1 0x0000000056bdcc80 in os_thread_sleep (tm=286000) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/os/os0thread.cc:303
#2 0x0000000056dd9494 in page_cleaner_sleep_if_needed (next_loop_time=1483836520835) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/buf/buf0flu.cc:2645
#3 0x0000000056dd9f68 in buf_flush_lru_manager_thread (arg=0x0) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/buf/buf0flu.cc:2919
#4 0x00003fff8bda7cec in start_thread (arg=0x3fff7a7ff140) at pthread_create.c:312
#5 0x00003fff8b661140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 12 (Thread 0x3fff8174f140 (LWP 16864)):
#0 0x00003fff8bd60688 in ?? () from /lib/powerpc64le-linux-gnu/libaio.so.1
#1 0x00003fff8bd606f0 in io_getevents () from /lib/powerpc64le-linux-gnu/libaio.so.1
#2 0x0000000056bd7d58 in os_aio_linux_collect (array=0x3fff83740100, segment=0, seg_size=256) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/os/os0file.cc:5432
#3 0x0000000056bd8278 in os_aio_linux_handle (global_seg=1, message1=0x3fff8174e6d8, message2=0x3fff8174e6e0, type=0x3fff8174e6e8, space_id=0x3fff8174e6f0) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/os/os0file.cc:5585
#4 0x0000000056e65c68 in fil_aio_wait (segment=1) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/fil/fil0fil.cc:6276
#5 0x0000000056ceef48 in io_handler_thread (arg=0x57926c20 <n+8>) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/srv/srv0start.cc:546
#6 0x00003fff8bda7cec in start_thread (arg=0x3fff8174f140) at pthread_create.c:312
#7 0x00003fff8b661140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 11 (Thread 0x3fff7bfff140 (LWP 16877)):
#0 0x00003fff8bdaf7d0 in __pthread_cond_wait (cond=0x3fff85790278, mutex=0x3fff85790240) at pthread_cond_wait.c:188
#1 0x0000000056bdb700 in os_cond_wait (cond=0x3fff85790278, fast_mutex=0x3fff85790240) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/os/os0sync.cc:196
#2 0x0000000056bdbd14 in os_event_wait_low (event=0x3fff85790240, reset_sig_count=10) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/os/os0sync.cc:477
#3 0x0000000056cea2bc in srv_purge_coordinator_suspend (slot=0x3fff8ae482c8, rseg_history_len=9) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/srv/srv0srv.cc:3499
#4 0x0000000056cea9dc in srv_purge_coordinator_thread (arg=0x0) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/srv/srv0srv.cc:3621
#5 0x00003fff8bda7cec in start_thread (arg=0x3fff7bfff140) at pthread_create.c:312
#6 0x00003fff8b661140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 10 (Thread 0x3fff81f4f140 (LWP 16863)):
#0 0x00003fff8bd60688 in ?? () from /lib/powerpc64le-linux-gnu/libaio.so.1
#1 0x00003fff8bd606f0 in io_getevents () from /lib/powerpc64le-linux-gnu/libaio.so.1
#2 0x0000000056bd7d58 in os_aio_linux_collect (array=0x3fff83740150, segment=0, seg_size=256) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/os/os0file.cc:5432
#3 0x0000000056bd8278 in os_aio_linux_handle (global_seg=0, message1=0x3fff81f4e6d8, message2=0x3fff81f4e6e0, type=0x3fff81f4e6e8, space_id=0x3fff81f4e6f0) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/os/os0file.cc:5585
#4 0x0000000056e65c68 in fil_aio_wait (segment=0) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/fil/fil0fil.cc:6276
#5 0x0000000056ceef48 in io_handler_thread (arg=0x57926c18 <n>) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/srv/srv0start.cc:546
#6 0x00003fff8bda7cec in start_thread (arg=0x3fff81f4f140) at pthread_create.c:312
#7 0x00003fff8b661140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 9 (Thread 0x3fff833ff140 (LWP 16858)):
#0 0x00003fff8bdafcc0 in __pthread_cond_timedwait (cond=0x581448a8 <COND_checkpoint>, mutex=0x58144820 <LOCK_checkpoint+40>, abstime=0x3fff833fe5e0) at pthread_cond_timedwait.c:199
#1 0x0000000057194470 in safe_cond_timedwait (cond=0x581448a8 <COND_checkpoint>, mp=0x581447f8 <LOCK_checkpoint>, abstime=0x3fff833fe5e0, file=0x57456eb0 "/var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/include/mysql/psi/mysql_thread.h", line=1191) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/mysys/thr_mutex.c:547
#2 0x0000000056f56a88 in inline_mysql_cond_timedwait (that=0x581448a8 <COND_checkpoint>, mutex=0x581447f8 <LOCK_checkpoint>, abstime=0x3fff833fe5e0, src_file=0x57456f30 "/var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/maria/ma_servicethread.c", src_line=116) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/include/mysql/psi/mysql_thread.h:1191
#3 0x0000000056f57198 in my_service_thread_sleep (control=0x576fa898 <checkpoint_control>, sleep_time=29000000000) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/maria/ma_servicethread.c:115
#4 0x0000000056f45dcc in ma_checkpoint_background (arg=0x1e) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/maria/ma_checkpoint.c:708
#5 0x0000000057102fec in pfs_spawn_thread (arg=0x3fff8aea3ff0) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/perfschema/pfs.cc:1860
#6 0x00003fff8bda7cec in start_thread (arg=0x3fff833ff140) at pthread_create.c:312
#7 0x00003fff8b661140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 8 (Thread 0x3fff84bff140 (LWP 16857)):
#0 0x00003fff8bdafcc0 in __pthread_cond_timedwait (cond=0x58155e70 <COND_timer>, mutex=0x58155de8 <LOCK_timer+40>, abstime=0x3fff84bfe6e0) at pthread_cond_timedwait.c:199
#1 0x0000000057194470 in safe_cond_timedwait (cond=0x58155e70 <COND_timer>, mp=0x58155dc0 <LOCK_timer>, abstime=0x3fff84bfe6e0, file=0x57497f70 "/var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/include/mysql/psi/mysql_thread.h", line=1191) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/mysys/thr_mutex.c:547
#2 0x0000000057195d34 in inline_mysql_cond_timedwait (that=0x58155e70 <COND_timer>, mutex=0x58155dc0 <LOCK_timer>, abstime=0x3fff84bfe6e0, src_file=0x57497fe0 "/var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/mysys/thr_timer.c", src_line=292) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/include/mysql/psi/mysql_thread.h:1191
#3 0x0000000057196d4c in timer_handler (arg=0x0) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/mysys/thr_timer.c:292
#4 0x0000000057102fec in pfs_spawn_thread (arg=0x3fff8aea1cf0) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/perfschema/pfs.cc:1860
#5 0x00003fff8bda7cec in start_thread (arg=0x3fff84bff140) at pthread_create.c:312
#6 0x00003fff8b661140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 7 (Thread 0x3fff8be57b70 (LWP 16849)):
#0 0x00003fff8b651438 in poll () at ../sysdeps/unix/syscall-template.S:81
#1 0x000000005644cae8 in handle_connections_sockets () at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/sql/mysqld.cc:6539
#2 0x000000005644b5e8 in mysqld_main (argc=151, argv=0x3fff8af510c0) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/sql/mysqld.cc:5964
#3 0x000000005643b148 in main (argc=27, argv=0x3fffc637e8f8) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/sql/main.cc:25

Thread 6 (Thread 0x3fff7eee7540 (LWP 16889)):
#0 0x00003fff8bdaf7d0 in __pthread_cond_wait (cond=0x578b4c08 <COND_slave_background>, mutex=0x578b7be0 <LOCK_slave_background+40>) at pthread_cond_wait.c:188
#1 0x0000000057194098 in safe_cond_wait (cond=0x578b4c08 <COND_slave_background>, mp=0x578b7bb8 <LOCK_slave_background>, file=0x5727f2a0 "/var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/include/mysql/psi/mysql_thread.h", line=1154) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/mysys/thr_mutex.c:493
#2 0x000000005647b968 in inline_mysql_cond_wait (that=0x578b4c08 <COND_slave_background>, mutex=0x578b7bb8 <LOCK_slave_background>, src_file=0x5727fc58 "/var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/sql/slave.cc", src_line=341) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/include/mysql/psi/mysql_thread.h:1154
#3 0x000000005647d240 in handle_slave_background (arg=0x0) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/sql/slave.cc:341
#4 0x0000000057102fec in pfs_spawn_thread (arg=0x3fff8367c6f0) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/perfschema/pfs.cc:1860
#5 0x00003fff8bda7cec in start_thread (arg=0x3fff7eee7540) at pthread_create.c:312
#6 0x00003fff8b661140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 5 (Thread 0x3fff7ef47540 (LWP 16888)):
#0 0x00003fff8bdb5714 in do_sigwait (sig=0x3fff7ef46a40, set=<optimized out>) at ../sysdeps/unix/sysv/linux/sigwait.c:61
#1 __sigwait (set=<optimized out>, sig=0x3fff7ef46a40) at ../sysdeps/unix/sysv/linux/sigwait.c:98
#2 0x0000000056444b90 in signal_hand (arg=0x0) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/sql/mysqld.cc:3465
#3 0x0000000057102fec in pfs_spawn_thread (arg=0x3fff8367b7f0) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/perfschema/pfs.cc:1860
#4 0x00003fff8bda7cec in start_thread (arg=0x3fff7ef47540) at pthread_create.c:312
#5 0x00003fff8b661140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 4 (Thread 0x3fff75fff140 (LWP 16887)):
#0 0x00003fff8b653c14 in select () at ../sysdeps/unix/syscall-template.S:81
#1 0x0000000056bdcc80 in os_thread_sleep (tm=1000000) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/os/os0thread.cc:303
#2 0x0000000056daadf0 in btr_defragment_thread (arg=0x0) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/btr/btr0defragment.cc:758
#3 0x00003fff8bda7cec in start_thread (arg=0x3fff75fff140) at pthread_create.c:312
#4 0x00003fff8b661140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 3 (Thread 0x3fff76fff140 (LWP 16885)):
#0 0x00003fff8bdafcc0 in __pthread_cond_timedwait (cond=0x3fff857915b8, mutex=0x3fff85791580, abstime=0x3fff76ffe030) at pthread_cond_timedwait.c:199
#1 0x0000000056bdb58c in os_cond_wait_timed (cond=0x3fff857915b8, fast_mutex=0x3fff85791580, abstime=0x3fff76ffe030) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/os/os0sync.cc:159
#2 0x0000000056bdbfa8 in os_event_wait_time_low (event=0x3fff85791580, time_in_usec=36000, reset_sig_count=1) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/os/os0sync.cc:591
#3 0x0000000056e73abc in fil_crypt_rotate_page (key_state=0x3fff76ffe694, state=0x3fff76ffe6a0) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/fil/fil0crypt.cc:2094
#4 0x0000000056e73b9c in fil_crypt_rotate_pages (key_state=0x3fff76ffe694, state=0x3fff76ffe6a0) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/fil/fil0crypt.cc:2125
#5 0x0000000056e7448c in fil_crypt_thread (arg=0x0) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/fil/fil0crypt.cc:2348
#6 0x00003fff8bda7cec in start_thread (arg=0x3fff76fff140) at pthread_create.c:312
#7 0x00003fff8b661140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 2 (Thread 0x3fff777ff140 (LWP 16884)):
#0 0x00003fff8bdafcc0 in __pthread_cond_timedwait (cond=0x3fff857915b8, mutex=0x3fff85791580, abstime=0x3fff777fe030) at pthread_cond_timedwait.c:199
#1 0x0000000056bdb58c in os_cond_wait_timed (cond=0x3fff857915b8, fast_mutex=0x3fff85791580, abstime=0x3fff777fe030) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/os/os0sync.cc:159
#2 0x0000000056bdbfa8 in os_event_wait_time_low (event=0x3fff85791580, time_in_usec=35000, reset_sig_count=1) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/os/os0sync.cc:591
#3 0x0000000056e73abc in fil_crypt_rotate_page (key_state=0x3fff777fe694, state=0x3fff777fe6a0) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/fil/fil0crypt.cc:2094
#4 0x0000000056e73b9c in fil_crypt_rotate_pages (key_state=0x3fff777fe694, state=0x3fff777fe6a0) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/fil/fil0crypt.cc:2125
#5 0x0000000056e7448c in fil_crypt_thread (arg=0x0) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/fil/fil0crypt.cc:2348
#6 0x00003fff8bda7cec in start_thread (arg=0x3fff777ff140) at pthread_create.c:312
#7 0x00003fff8b661140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 1 (Thread 0x3fff77fff140 (LWP 16883)):
#0 0x00003fff8bdb0ef0 in __pthread_kill (threadid=<optimized out>, signo=<optimized out>) at ../sysdeps/unix/sysv/linux/pthread_kill.c:60
#1 0x000000005718c718 in my_write_core (sig=6) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/mysys/stacktrace.c:477
#2 0x0000000056866324 in handle_fatal_signal (sig=6) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/sql/signal_handler.cc:296
#3 <signal handler called>
#4 0x00003fff8b56eeb8 in __GI_raise (sig=<optimized out>) at ../sysdeps/unix/sysv/linux/raise.c:55
#5 0x00003fff8b5719bc in __GI_abort () at abort.c:89
#6 0x0000000056e7405c in fil_crypt_complete_rotate_space (key_state=0x3fff77ffe694, state=0x3fff77ffe6a0) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/fil/fil0crypt.cc:2220
#7 0x0000000056e74504 in fil_crypt_thread (arg=0x0) at /var/lib/buildbot/maria-slave/p8-trusty-bintar-debug/build/storage/xtradb/fil/fil0crypt.cc:2355
#8 0x00003fff8bda7cec in start_thread (arg=0x3fff77fff140) at pthread_create.c:312
#9 0x00003fff8b661140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96
{noformat} ]

Elena Stepanova made changes - 2017-01-10 11:40

Resolution		Fixed [ 1 ]
Status	Stalled [ 10000 ]	Closed [ 6 ]

Marko Mäkelä made changes - 2017-01-10 11:43

Comment

[ The posted crash does not occur at shutdown, but during normal processing. Before the crash, the server was restarted, and an UPDATE without any WHERE condition was performed on several tables. The crash occurs during a SELECT.
I think we should group this failure with ~~MDEV-9359~~. Only if any problems remain during shutdown should ~~MDEV-11638~~ be reopened. ]

Marko Mäkelä made changes - 2017-01-10 11:44

Comment

[ Commit ffb38c9771bc7 only modifies some tests (~~MDEV-8139~~) that were previously disabled.
Commit ffb38c9771bc7~ (719321e78e69) refactors the shutdown.
Commit ffb38c9771bc7~2 (0f8e17af92cd0) cherry-picked a fix of Bug#24450908 from MySQL 5.7. That has nothing to do with encryption.

If any of these commits is to blame for the crash, that would be the shutdown refactoring. I carefully reviewed it again. It did not change fil_crypt_complete_rotate_space() or fil_crypt_thread(), and normal operation outside shutdown should not be affected by the change to rotate_thread_t::should_shutdown(), by the removal of fil_crypt_threads_end() or by exposing fil_crypt_threads_event so that
logs_empty_and_mark_files_at_shutdown() can signal it.

One more change was related to moving the initialization and shutdown of the log_scrub_thread. It is now being initialized with the rest of the redo log subsystem. But, the test encryption.innodb_encryption-page-compression does not innodb_scrub_log, and the crash occurred very much after server startup.

So, I would recommend to file the encryption.innodb_encryption-page-compression crash with ~~MDEV-9359~~ even if it only started occurring now. ]

Marko Mäkelä made changes - 2017-05-19 14:07

Link

This issue blocks ~~MDEV-12052~~ [ ~~MDEV-12052~~ ]

Sergei Golubchik made changes - 2021-12-06 21:44

Workflow

MariaDB v3 [ 78871 ]

MariaDB v4 [ 151426 ]

People

Assignee:: Marko Mäkelä

Reporter:: Elena Stepanova

Votes:: 0 Vote for this issue

Watchers:: 3 Start watching this issue

Dates

Created:: 2016-12-22 21:40

Updated:: 2017-05-20 05:41

Resolved:: 2017-01-10 11:40

Git Integration

Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.

MariaDB Server

Details

Description

Attachments

Issue Links

Activity

People

Dates

Git Integration