innodb.innodb_bug59641 fails sporadically in buildbot: InnoDB: Failing assertion: current_rec != insert_rec in file page0cur.c line 1052

InnoDB: Progress in percents: 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 160612 17:16:22  InnoDB: Assertion failure in thread 70366885572992 in file page0cur.c line 1052

InnoDB: Failing assertion: current_rec != insert_rec

160612 17:16:22 [ERROR] mysqld got signal 6 ;

This could be because you hit a bug. It is also possible that this binary

or one of the libraries it was linked against is corrupt, improperly built,

or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see http://kb.askmonty.org/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help

diagnose the problem, but since we have already crashed, 

something is definitely wrong and this may fail.

Server version: 5.5.50-MariaDB-debug

key_buffer_size=1048576

read_buffer_size=131072

max_used_connections=0

max_threads=153

thread_count=0

It is possible that mysqld could use up to 

key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 62567 K  bytes of memory

Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0x0

Attempting backtrace. You can use the following information to find out

where mysqld died. If you see no messages after this, something went

terribly wrong...

stack_bottom = 0x0 thread_stack 0x48000

mysys/stacktrace.c:246(my_print_stacktrace)[0x10beab8c]

sql/signal_handler.cc:153(handle_fatal_signal)[0x1049ece0]

linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0x3fff9aac0478]

/opt/at8.0/lib64/power8/libc.so.6(gsignal+0x48)[0x3fff9a53ee38]

/opt/at8.0/lib64/power8/libc.so.6(abort+0x2ac)[0x3fff9a5419dc]

page/page0cur.c:1052(page_cur_insert_rec_low)[0x10b26904]

include/page0cur.ic:296(page_cur_rec_insert)[0x10b2491c]

page/page0cur.c:923(page_cur_parse_insert_rec)[0x10b2649c]

log/log0recv.c:1120(recv_parse_or_apply_log_rec_body)[0x10af591c]

log/log0recv.c:1690(recv_recover_page_func)[0x10af78bc]

buf/buf0buf.c:4045(buf_page_io_complete)[0x10a1a720]

fil/fil0fil.c:5734(fil_aio_wait)[0x10a906b8]

srv/srv0start.c:497(io_handler_thread)[0x1097b920]

/opt/at8.0/lib64/power8/libpthread.so.0(+0x7cac)[0x3fff9aa87cac]

/opt/at8.0/lib64/power8/libc.so.6(clone+0x98)[0x3fff9a62a7c0]

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains

information that should help you find out what is causing the crash.

Writing a core file

----------SERVER LOG END-------------

mysqltest failed but provided no output

The result from queries just before the failure was:

< snip >

call mtr.add_suppression("Found 3 prepared XA transactions");

flush tables;

CREATE TABLE t(a INT PRIMARY KEY, b INT)ENGINE=InnoDB;

INSERT INTO t VALUES(2,2),(4,4),(8,8),(16,16),(32,32);

COMMIT;

XA START '123';

INSERT INTO t VALUES(1,1);

XA END '123';

XA PREPARE '123';

XA START '456';

INSERT INTO t VALUES(3,47),(5,67);

UPDATE t SET b=2*b WHERE a BETWEEN 5 AND 8;

XA END '456';

XA PREPARE '456';

XA START '789';

UPDATE t SET b=4*a WHERE a=32;

XA END '789';

XA PREPARE '789';

#0  0x00003fff9aa90ed0 in __pthread_kill (threadid=<optimized out>, signo=<optimized out>) at ../sysdeps/unix/sysv/linux/pthread_kill.c:60

60	../sysdeps/unix/sysv/linux/pthread_kill.c: No such file or directory.

#0  0x00003fff9aa90ed0 in __pthread_kill (threadid=<optimized out>, signo=<optimized out>) at ../sysdeps/unix/sysv/linux/pthread_kill.c:60

#1  0x0000000010bead2c in my_write_core (sig=6) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/mysys/stacktrace.c:457

#2  0x000000001049f17c in handle_fatal_signal (sig=6) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/signal_handler.cc:262

#3  <signal handler called>

#4  0x00003fff9a53ee38 in __GI_raise (sig=<optimized out>) at ../sysdeps/unix/sysv/linux/raise.c:55

#5  0x00003fff9a54193c in __GI_abort () at abort.c:89

#6  0x0000000010b26904 in page_cur_insert_rec_low (current_rec=0x3fff92f12e02 "", index=0x3fff80003848, rec=0x3fff9137d806 "", offsets=0x3fff9137dc18, mtr=0x3fff9137e0e8) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/xtradb/page/page0cur.c:1052

#7  0x0000000010b2491c in page_cur_rec_insert (cursor=0x3fff9137dc00, rec=0x3fff9137d806 "", index=0x3fff80003848, offsets=0x3fff9137dc18, mtr=0x3fff9137e0e8) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/xtradb/include/page0cur.ic:296

#8  0x0000000010b2649c in page_cur_parse_insert_rec (is_short=0, ptr=0x1000ea10653 "\261", end_ptr=0x1000ea10654 "", block=0x3fff92b9d640, index=0x3fff80003848, mtr=0x3fff9137e0e8) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/xtradb/page/page0cur.c:923

#9  0x0000000010af591c in recv_parse_or_apply_log_rec_body (type=38 '&', ptr=0x1000ea10650 ".\002\002\261", end_ptr=0x1000ea10654 "", block=0x3fff92b9d640, mtr=0x3fff9137e0e8, space_id=0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/xtradb/log/log0recv.c:1120

#10 0x0000000010af78bc in recv_recover_page_func (just_read_in=1, block=0x3fff92b9d640) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/xtradb/log/log0recv.c:1685

#11 0x0000000010a1a720 in buf_page_io_complete (bpage=0x3fff92b9d640) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/xtradb/buf/buf0buf.c:4045

#12 0x0000000010a906b8 in fil_aio_wait (segment=3) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/xtradb/fil/fil0fil.c:5734

#13 0x000000001097b920 in io_handler_thread (arg=0x11957c38 <n+24>) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/xtradb/srv/srv0start.c:497

#14 0x00003fff9aa87cac in start_thread (arg=0x3fff9137f180) at pthread_create.c:312

#15 0x00003fff9a62a7c0 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 8 (LWP 17425):

#0  0x00003fff9aa8fca0 in __pthread_cond_timedwait (cond=0x11953b00 <COND_checkpoint>, mutex=0x11953a80 <LOCK_checkpoint+40>, abstime=0x3fff9a4fe610) at pthread_cond_timedwait.c:199

#1  0x0000000010bf7070 in safe_cond_timedwait (cond=0x11953b00 <COND_checkpoint>, mp=0x11953a58 <LOCK_checkpoint>, abstime=0x3fff9a4fe610, file=0x10d5d090 "/home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/maria/ma_servicethread.c", line=116) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/mysys/thr_mutex.c:547

#2  0x000000001077a960 in inline_mysql_cond_timedwait (that=0x11953b00 <COND_checkpoint>, mutex=0x11953a58 <LOCK_checkpoint>, abstime=0x3fff9a4fe610, src_file=0x10d5d090 "/home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/maria/ma_servicethread.c", src_line=116) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/include/mysql/psi/mysql_thread.h:1017

#3  0x000000001077b07c in my_service_thread_sleep (control=0x10ff56b8 <checkpoint_control>, sleep_time=29000000000) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/maria/ma_servicethread.c:115

#4  0x0000000010769d34 in ma_checkpoint_background (arg=0x1e) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/maria/ma_checkpoint.c:704

#5  0x00000000108e0860 in pfs_spawn_thread (arg=0x1000de38630) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/perfschema/pfs.cc:1015

#6  0x00003fff9aa87cac in start_thread (arg=0x3fff9a4ff180) at pthread_create.c:312

#7  0x00003fff9a62a7c0 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 7 (LWP 17423):

#0  0x00003fff9a61d314 in select () at ../sysdeps/unix/syscall-template.S:81

#1  0x0000000010b1b070 in os_thread_sleep (tm=500000) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/xtradb/os/os0thread.c:261

#2  0x0000000010af81f0 in recv_apply_hashed_log_recs (allow_ibuf=0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/xtradb/log/log0recv.c:1919

#3  0x0000000010af9cf8 in recv_scan_log_recs (available_memory=4177920, store_to_hash=1, buf=0x1000e904200 "", len=65536, start_lsn=8501248, contiguous_lsn=0x3ffff43f8e58, group_scanned_lsn=0x3ffff43f8e50) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/xtradb/log/log0recv.c:2863

#4  0x0000000010af9e50 in recv_group_scan_log_recs (group=0x1000ea0ddb8, contiguous_lsn=0x3ffff43f8e58, group_scanned_lsn=0x3ffff43f8e50) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/xtradb/log/log0recv.c:2906

#5  0x0000000010afa4f4 in recv_recovery_from_checkpoint_start_func (min_flushed_lsn=0, max_flushed_lsn=0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/xtradb/log/log0recv.c:3193

#6  0x000000001097ef64 in innobase_start_or_create_for_mysql () at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/xtradb/srv/srv0start.c:1968

#7  0x00000000108eeb1c in innobase_init (p=0x1000deca4f0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/xtradb/handler/ha_innodb.cc:3246

#8  0x00000000104a18c4 in ha_initialize_handlerton (plugin=0x1000de17078) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/handler.cc:470

#9  0x00000000102638fc in plugin_initialize (tmp_root=0x3ffff43f9e10, plugin=0x1000de17078, argc=0x11137e70 <remaining_argc>, argv=0x1000dd98c20, options_only=false) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/sql_plugin.cc:1372

#10 0x000000001026449c in plugin_init (argc=0x11137e70 <remaining_argc>, argv=0x1000dd98c20, flags=0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/sql_plugin.cc:1597

#11 0x000000001015a96c in init_server_components () at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/mysqld.cc:4349

#12 0x000000001015c184 in mysqld_main (argc=90, argv=0x1000dd98c20) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/mysqld.cc:4946

#13 0x0000000010150678 in main (argc=12, argv=0x3ffff43fa6b8) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/main.cc:25

Thread 6 (LWP 17428):

#0  0x0000000010a21f88 in buf_flush_insert_sorted_into_flush_list (buf_pool=0x1000e879a18, block=0x3fff92ba3b80, lsn=7720640) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/xtradb/buf/buf0flu.c:429

#1  0x0000000010af2a9c in buf_flush_recv_note_modification (block=0x3fff92ba3b80, start_lsn=7720640, end_lsn=7764075) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/xtradb/include/buf0flu.ic:127

#2  0x0000000010af7a88 in recv_recover_page_func (just_read_in=1, block=0x3fff92ba3b80) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/xtradb/log/log0recv.c:1730

#3  0x0000000010a1a720 in buf_page_io_complete (bpage=0x3fff92ba3b80) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/xtradb/buf/buf0buf.c:4045

#4  0x0000000010a906b8 in fil_aio_wait (segment=2) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/xtradb/fil/fil0fil.c:5734

#5  0x000000001097b920 in io_handler_thread (arg=0x11957c30 <n+16>) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/xtradb/srv/srv0start.c:497

#6  0x00003fff9aa87cac in start_thread (arg=0x3fff91b7f180) at pthread_create.c:312

#7  0x00003fff9a62a7c0 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 5 (LWP 17427):

#0  0x00003fff9aa30738 in __io_getevents_0_4 () from /lib64/libaio.so.1

#1  0x00003fff9aa307a0 in io_getevents () from /lib64/libaio.so.1

#2  0x0000000010b16244 in os_aio_linux_collect (array=0x1000e80df20, segment=0, seg_size=256) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/xtradb/os/os0file.c:4703

#3  0x0000000010b167b8 in os_aio_linux_handle (global_seg=1, message1=0x3fff9237e718, message2=0x3fff9237e720, type=0x3fff9237e728, space_id=0x3fff9237e730) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/xtradb/os/os0file.c:4849

#4  0x0000000010a904d4 in fil_aio_wait (segment=1) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/xtradb/fil/fil0fil.c:5697

#5  0x000000001097b920 in io_handler_thread (arg=0x11957c28 <n+8>) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/xtradb/srv/srv0start.c:497

#6  0x00003fff9aa87cac in start_thread (arg=0x3fff9237f180) at pthread_create.c:312

#7  0x00003fff9a62a7c0 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 4 (LWP 17431):

#0  0x00003fff9aa30738 in __io_getevents_0_4 () from /lib64/libaio.so.1

#1  0x00003fff9aa307a0 in io_getevents () from /lib64/libaio.so.1

#2  0x0000000010b16244 in os_aio_linux_collect (array=0x1000e83ac20, segment=1, seg_size=256) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/xtradb/os/os0file.c:4703

#3  0x0000000010b167b8 in os_aio_linux_handle (global_seg=5, message1=0x3fff9037e718, message2=0x3fff9037e720, type=0x3fff9037e728, space_id=0x3fff9037e730) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/xtradb/os/os0file.c:4849

#4  0x0000000010a904d4 in fil_aio_wait (segment=5) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/xtradb/fil/fil0fil.c:5697

#5  0x000000001097b920 in io_handler_thread (arg=0x11957c48 <n+40>) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/xtradb/srv/srv0start.c:497

#6  0x00003fff9aa87cac in start_thread (arg=0x3fff9037f180) at pthread_create.c:312

#7  0x00003fff9a62a7c0 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 3 (LWP 17430):

#0  0x00003fff9aa30738 in __io_getevents_0_4 () from /lib64/libaio.so.1

#1  0x00003fff9aa307a0 in io_getevents () from /lib64/libaio.so.1

#2  0x0000000010b16244 in os_aio_linux_collect (array=0x1000e83ac20, segment=0, seg_size=256) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/xtradb/os/os0file.c:4703

#3  0x0000000010b167b8 in os_aio_linux_handle (global_seg=4, message1=0x3fff90b7e718, message2=0x3fff90b7e720, type=0x3fff90b7e728, space_id=0x3fff90b7e730) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/xtradb/os/os0file.c:4849

#4  0x0000000010a904d4 in fil_aio_wait (segment=4) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/xtradb/fil/fil0fil.c:5697

#5  0x000000001097b920 in io_handler_thread (arg=0x11957c40 <n+32>) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/xtradb/srv/srv0start.c:497

#6  0x00003fff9aa87cac in start_thread (arg=0x3fff90b7f180) at pthread_create.c:312

#7  0x00003fff9a62a7c0 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 2 (LWP 17426):

#0  0x00003fff9aa30738 in __io_getevents_0_4 () from /lib64/libaio.so.1

#1  0x00003fff9aa307a0 in io_getevents () from /lib64/libaio.so.1

#2  0x0000000010b16244 in os_aio_linux_collect (array=0x1000e7fe8a0, segment=0, seg_size=256) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/xtradb/os/os0file.c:4703

#3  0x0000000010b167b8 in os_aio_linux_handle (global_seg=0, message1=0x3fff92b7e718, message2=0x3fff92b7e720, type=0x3fff92b7e728, space_id=0x3fff92b7e730) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/xtradb/os/os0file.c:4849

#4  0x0000000010a904d4 in fil_aio_wait (segment=0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/xtradb/fil/fil0fil.c:5697

#5  0x000000001097b920 in io_handler_thread (arg=0x11957c20 <n>) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/xtradb/srv/srv0start.c:497

#6  0x00003fff9aa87cac in start_thread (arg=0x3fff92b7f180) at pthread_create.c:312

#7  0x00003fff9a62a7c0 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 1 (LWP 17429):

#0  0x00003fff9aa90ed0 in __pthread_kill (threadid=<optimized out>, signo=<optimized out>) at ../sysdeps/unix/sysv/linux/pthread_kill.c:60

#1  0x0000000010bead2c in my_write_core (sig=6) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/mysys/stacktrace.c:457

#2  0x000000001049f17c in handle_fatal_signal (sig=6) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/sql/signal_handler.cc:262

#3  <signal handler called>

#4  0x00003fff9a53ee38 in __GI_raise (sig=<optimized out>) at ../sysdeps/unix/sysv/linux/raise.c:55

#5  0x00003fff9a54193c in __GI_abort () at abort.c:89

#6  0x0000000010b26904 in page_cur_insert_rec_low (current_rec=0x3fff92f12e02 "", index=0x3fff80003848, rec=0x3fff9137d806 "", offsets=0x3fff9137dc18, mtr=0x3fff9137e0e8) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/xtradb/page/page0cur.c:1052

#7  0x0000000010b2491c in page_cur_rec_insert (cursor=0x3fff9137dc00, rec=0x3fff9137d806 "", index=0x3fff80003848, offsets=0x3fff9137dc18, mtr=0x3fff9137e0e8) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/xtradb/include/page0cur.ic:296

#8  0x0000000010b2649c in page_cur_parse_insert_rec (is_short=0, ptr=0x1000ea10653 "\261", end_ptr=0x1000ea10654 "", block=0x3fff92b9d640, index=0x3fff80003848, mtr=0x3fff9137e0e8) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/xtradb/page/page0cur.c:923

#9  0x0000000010af591c in recv_parse_or_apply_log_rec_body (type=38 '&', ptr=0x1000ea10650 ".\002\002\261", end_ptr=0x1000ea10654 "", block=0x3fff92b9d640, mtr=0x3fff9137e0e8, space_id=0) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/xtradb/log/log0recv.c:1120

#10 0x0000000010af78bc in recv_recover_page_func (just_read_in=1, block=0x3fff92b9d640) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/xtradb/log/log0recv.c:1685

#11 0x0000000010a1a720 in buf_page_io_complete (bpage=0x3fff92b9d640) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/xtradb/buf/buf0buf.c:4045

#12 0x0000000010a906b8 in fil_aio_wait (segment=3) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/xtradb/fil/fil0fil.c:5734

#13 0x000000001097b920 in io_handler_thread (arg=0x11957c38 <n+24>) at /home/buildbot/maria-slave/power8-vlp06-bintar-debug/build/storage/xtradb/srv/srv0start.c:497

#14 0x00003fff9aa87cac in start_thread (arg=0x3fff9137f180) at pthread_create.c:312

#15 0x00003fff9a62a7c0 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96