sys_vars.wsrep_desync_basic fails with SIGSEGV on ppc64el

sys_vars.wsrep_desync_basic              w6 [ fail ]

        Test ended at 2015-07-08 19:40:22

CURRENT_TEST: sys_vars.wsrep_desync_basic

mysqltest: At line 20: query 'SET @@global.wsrep_desync=OFF' failed: 2013: Lost connection to MySQL server during query

The result from queries just before the failure was:

< snip >

#

call mtr.add_suppression("WSREP: SET desync failed 9 for SET @@global.wsrep_desync=ON");

# save the initial value

SET @wsrep_desync_global_saved = @@global.wsrep_desync;

# default

SELECT @@global.wsrep_desync;

@@global.wsrep_desync

0

# scope

SELECT @@session.wsrep_desync;

ERROR HY000: Variable 'wsrep_desync' is a GLOBAL variable

SET @@global.wsrep_desync=OFF;

More results from queries before failure can be found in /var/lib/jenkins/workspace/mariadb-10.1/build/mysql-test/var/6/log/wsrep_desync_basic.log

Server [mysqld.1 - pid: 3044, winpid: 3044, exit: 256] failed during test run

Server log from this test:

----------SERVER LOG START-----------

150708 19:40:21 [ERROR] mysqld got signal 11 ;

This could be because you hit a bug. It is also possible that this binary

or one of the libraries it was linked against is corrupt, improperly built,

or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see http://kb.askmonty.org/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help

diagnose the problem, but since we have already crashed, 

something is definitely wrong and this may fail.

Server version: 10.1.6-MariaDB-log

key_buffer_size=1048576

read_buffer_size=131072

max_used_connections=5

max_threads=153

thread_count=1

It is possible that mysqld could use up to 

key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 62877 K  bytes of memory

Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0x3fffb12402d8

Attempting backtrace. You can use the following information to find out

where mysqld died. If you see no messages after this, something went

terribly wrong...

stack_bottom = 0x3fff7d0d66c8 thread_stack 0x48000

mysys/stacktrace.c:246(my_print_stacktrace)[0x3fff8439349c]

sql/signal_handler.cc:153(handle_fatal_signal)[0x3fff83e6ffd0]

[0x3fff837e0478]

sql/wsrep_var.cc:511(wsrep_desync_update(sys_var*, THD*, enum_var_type))[0x3fff83e0ab90]

sql/sql_select.cc:371(handle_select(THD*, LEX*, select_result*, unsigned long))[0x3fff83d07b48]

sql/set_var.cc:198(sys_var::update(THD*, set_var*))[0x3fff83c0bac8]

sql/set_var.cc:793(set_var::update(THD*))[0x3fff83c0bf9c]

sql/set_var.cc:695(sql_set_variables(THD*, List<set_var_base>*, bool))[0x3fff83c0dd9c]

sql/sql_parse.cc:4248(mysql_execute_command(THD*))[0x3fff83ca1030]

sql/sql_parse.cc:7181(mysql_parse(THD*, char*, unsigned int, Parser_state*))[0x3fff83ca4ee8]

sql/sql_parse.cc:1470(dispatch_command(enum_server_command, THD*, char*, unsigned int))[0x3fff83ca81e8]

sql/sql_parse.cc:1093(do_command(THD*))[0x3fff83ca8a04]

sql/sql_connect.cc:1350(do_handle_one_connection(THD*))[0x3fff83d95df4]

sql/sql_connect.cc:1262(handle_one_connection)[0x3fff83d960ac]

perfschema/pfs.cc:1860(pfs_spawn_thread)[0x3fff84017638]

/lib/powerpc64le-linux-gnu/libpthread.so.0(+0x833c)[0x3fff8340833c]

/lib/powerpc64le-linux-gnu/libc.so.6(clone+0xe4)[0x3fff830bb014]

Trying to get some variables.

Some pointers may be invalid and cause the dump to abort.

Query (0x3fff6000ad00): SET @@global.wsrep_desync=OFF

Connection ID (thread ID): 676

Status: NOT_KILLED

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains

information that should help you find out what is causing the crash.

Writing a core file

----------SERVER LOG END-------------

 - saving '/var/lib/jenkins/workspace/mariadb-10.1/build/mysql-test/var/6/log/sys_vars.wsrep_desync_basic/' to '/var/lib/jenkins/workspace/mariadb-10.1/build/mysql-test/var/log/sys_vars.wsrep_desync_basic/'

 - found 'core' (0/5)

Trying 'dbx' to get a backtrace

Trying 'gdb' to get a backtrace

Core generated by '/var/lib/jenkins/workspace/mariadb-10.1/build/sql/mysqld'

Output from gdb follows. The first stack trace is from the failing thread.

The following stack traces are from all threads (so the failing one is

duplicated).

--------------------------

[New LWP 13785]

[New LWP 3045]

[New LWP 3068]

[New LWP 3067]

[New LWP 3066]

[Thread debugging using libthread_db enabled]

Using host libthread_db library "/lib/powerpc64le-linux-gnu/libthread_db.so.1".

Core was generated by `/var/lib/jenkins/workspace/mariadb-10.1/build/sql/mysqld --defaults-group-suffi'.

Program terminated with signal SIGSEGV, Segmentation fault.

#0  0x00003fff834126a0 in __pthread_kill (threadid=<optimized out>, signo=<optimized out>) at ../sysdeps/unix/sysv/linux/pthread_kill.c:58

58	../sysdeps/unix/sysv/linux/pthread_kill.c: No such file or directory.

#0  0x00003fff834126a0 in __pthread_kill (threadid=<optimized out>, signo=<optimized out>) at ../sysdeps/unix/sysv/linux/pthread_kill.c:58

#1  0x00003fff84392f18 in my_write_core (sig=<optimized out>) at /var/lib/jenkins/workspace/mariadb-10.1/mysys/stacktrace.c:456

#2  0x00003fff83e70070 in handle_fatal_signal (sig=<optimized out>) at /var/lib/jenkins/workspace/mariadb-10.1/sql/signal_handler.cc:266

#3  <signal handler called>

#4  wsrep_desync_update (self=<optimized out>, thd=0x3fffb12402d8, type=SHOW_OPT_GLOBAL) at /var/lib/jenkins/workspace/mariadb-10.1/sql/wsrep_var.cc:511

#5  0x00003fff83c0bac8 in sys_var::update (this=0x3fff848e1a80 <Sys_wsrep_desync>, thd=0x3fffb12402d8, var=<optimized out>) at /var/lib/jenkins/workspace/mariadb-10.1/sql/set_var.cc:198

#6  0x00003fff83c0bf9c in set_var::update (this=<optimized out>, thd=<optimized out>) at /var/lib/jenkins/workspace/mariadb-10.1/sql/set_var.cc:793

#7  0x00003fff83c0dd9c in sql_set_variables (thd=0x3fffb12402d8, var_list=0x3fffb1244828, free=<optimized out>) at /var/lib/jenkins/workspace/mariadb-10.1/sql/set_var.cc:695

#8  0x00003fff83ca1030 in mysql_execute_command (thd=0x3fffb12402d8) at /var/lib/jenkins/workspace/mariadb-10.1/sql/sql_parse.cc:4248

#9  0x00003fff83ca4ee8 in mysql_parse (thd=0x3fffb12402d8, rawbuf=<optimized out>, length=<optimized out>, parser_state=0x3fff7d0d5a00) at /var/lib/jenkins/workspace/mariadb-10.1/sql/sql_parse.cc:7181

#10 0x00003fff83ca81e8 in dispatch_command (command=<optimized out>, thd=<optimized out>, packet=<optimized out>, packet_length=<optimized out>) at /var/lib/jenkins/workspace/mariadb-10.1/sql/sql_parse.cc:1470

#11 0x00003fff83ca8a04 in do_command (thd=0x3fffb12402d8) at /var/lib/jenkins/workspace/mariadb-10.1/sql/sql_parse.cc:1093

#12 0x00003fff83d95df4 in do_handle_one_connection (thd_arg=<optimized out>) at /var/lib/jenkins/workspace/mariadb-10.1/sql/sql_connect.cc:1350

#13 0x00003fff83d960ac in handle_one_connection (arg=0x3fffb12402d8) at /var/lib/jenkins/workspace/mariadb-10.1/sql/sql_connect.cc:1262

#14 0x00003fff84017638 in pfs_spawn_thread (arg=0x3fffb125e118) at /var/lib/jenkins/workspace/mariadb-10.1/storage/perfschema/pfs.cc:1860

#15 0x00003fff8340833c in start_thread (arg=0x3fff7d0d7180) at pthread_create.c:335

#16 0x00003fff830bb014 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 5 (Thread 0x3fff7ed9f180 (LWP 3066)):

#0  0x00003fff834111cc in __pthread_cond_timedwait (cond=0x3fff851577a8 <COND_timer>, mutex=0x3fff85157740 <LOCK_timer>, abstime=0x3fff7ed9e6b8) at pthread_cond_timedwait.c:198

#1  0x00003fff84399178 in inline_mysql_cond_timedwait (that=0x3fff851577a8 <COND_timer>, mutex=0x3fff85157740 <LOCK_timer>, src_file=0x3fff844efe18 "/var/lib/jenkins/workspace/mariadb-10.1/mysys/thr_timer.c", src_line=292, abstime=0x3fff7ed9e6b8) at /var/lib/jenkins/workspace/mariadb-10.1/include/mysql/psi/mysql_thread.h:1191

#2  timer_handler (arg=<optimized out>) at /var/lib/jenkins/workspace/mariadb-10.1/mysys/thr_timer.c:292

#3  0x00003fff84017638 in pfs_spawn_thread (arg=0x3fffb1124848) at /var/lib/jenkins/workspace/mariadb-10.1/storage/perfschema/pfs.cc:1860

#4  0x00003fff8340833c in start_thread (arg=0x3fff7ed9f180) at pthread_create.c:335

#5  0x00003fff830bb014 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 4 (Thread 0x3fff7d9ff180 (LWP 3067)):

#0  0x00003fff834111cc in __pthread_cond_timedwait (cond=0x3fff8514e558 <COND_checkpoint>, mutex=0x3fff8514e528 <LOCK_checkpoint>, abstime=0x3fff7d9fe5e8) at pthread_cond_timedwait.c:198

#1  0x00003fff842a6b1c in inline_mysql_cond_timedwait (src_file=0x3fff844e7068 "/var/lib/jenkins/workspace/mariadb-10.1/storage/maria/ma_servicethread.c", src_line=119, abstime=<optimized out>, mutex=0x3fff8514e528 <LOCK_checkpoint>, that=0x3fff8514e558 <COND_checkpoint>) at /var/lib/jenkins/workspace/mariadb-10.1/include/mysql/psi/mysql_thread.h:1191

#2  my_service_thread_sleep (control=0x3fff848591c0 <checkpoint_control>, sleep_time=<optimized out>) at /var/lib/jenkins/workspace/mariadb-10.1/storage/maria/ma_servicethread.c:118

#3  0x00003fff8429d524 in ma_checkpoint_background (arg=0x1e) at /var/lib/jenkins/workspace/mariadb-10.1/storage/maria/ma_checkpoint.c:711

#4  0x00003fff84017638 in pfs_spawn_thread (arg=0x3fffb114eca8) at /var/lib/jenkins/workspace/mariadb-10.1/storage/perfschema/pfs.cc:1860

#5  0x00003fff8340833c in start_thread (arg=0x3fff7d9ff180) at pthread_create.c:335

#6  0x00003fff830bb014 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 3 (Thread 0x3fff7d1f7180 (LWP 3068)):

#0  0x00003fff83417a44 in do_sigwait (sig=0x3fff7d1f6630, set=<optimized out>) at ../sysdeps/unix/sysv/linux/sigwait.c:61

#1  __sigwait (set=<optimized out>, sig=0x3fff7d1f6630) at ../sysdeps/unix/sysv/linux/sigwait.c:96

#2  0x00003fff83bccb8c in signal_hand (arg=<optimized out>) at /var/lib/jenkins/workspace/mariadb-10.1/sql/mysqld.cc:3455

#3  0x00003fff84017638 in pfs_spawn_thread (arg=0x3fffb1223608) at /var/lib/jenkins/workspace/mariadb-10.1/storage/perfschema/pfs.cc:1860

#4  0x00003fff8340833c in start_thread (arg=0x3fff7d1f7180) at pthread_create.c:335

#5  0x00003fff830bb014 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96

Thread 2 (Thread 0x3fff83859850 (LWP 3045)):

#0  0x00003fff830a99b8 in poll () at ../sysdeps/unix/syscall-template.S:81

#1  0x00003fff830d5954 in __poll_chk (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>, fdslen=<optimized out>) at poll_chk.c:27

#2  0x00003fff83bcf5e8 in poll (__timeout=-1, __nfds=2, __fds=<optimized out>) at /usr/include/powerpc64le-linux-gnu/bits/poll2.h:41

#3  handle_connections_sockets () at /var/lib/jenkins/workspace/mariadb-10.1/sql/mysqld.cc:6445

#4  0x00003fff83bd92b0 in mysqld_main (argc=126, argv=0x3fffb10098e8) at /var/lib/jenkins/workspace/mariadb-10.1/sql/mysqld.cc:5873

#5  0x00003fff83bb5478 in main (argc=<optimized out>, argv=<optimized out>) at /var/lib/jenkins/workspace/mariadb-10.1/sql/main.cc:25

Thread 1 (Thread 0x3fff7d0d7180 (LWP 13785)):

#0  0x00003fff834126a0 in __pthread_kill (threadid=<optimized out>, signo=<optimized out>) at ../sysdeps/unix/sysv/linux/pthread_kill.c:58

#1  0x00003fff84392f18 in my_write_core (sig=<optimized out>) at /var/lib/jenkins/workspace/mariadb-10.1/mysys/stacktrace.c:456

#2  0x00003fff83e70070 in handle_fatal_signal (sig=<optimized out>) at /var/lib/jenkins/workspace/mariadb-10.1/sql/signal_handler.cc:266

#3  <signal handler called>

#4  wsrep_desync_update (self=<optimized out>, thd=0x3fffb12402d8, type=SHOW_OPT_GLOBAL) at /var/lib/jenkins/workspace/mariadb-10.1/sql/wsrep_var.cc:511

#5  0x00003fff83c0bac8 in sys_var::update (this=0x3fff848e1a80 <Sys_wsrep_desync>, thd=0x3fffb12402d8, var=<optimized out>) at /var/lib/jenkins/workspace/mariadb-10.1/sql/set_var.cc:198

#6  0x00003fff83c0bf9c in set_var::update (this=<optimized out>, thd=<optimized out>) at /var/lib/jenkins/workspace/mariadb-10.1/sql/set_var.cc:793

#7  0x00003fff83c0dd9c in sql_set_variables (thd=0x3fffb12402d8, var_list=0x3fffb1244828, free=<optimized out>) at /var/lib/jenkins/workspace/mariadb-10.1/sql/set_var.cc:695

#8  0x00003fff83ca1030 in mysql_execute_command (thd=0x3fffb12402d8) at /var/lib/jenkins/workspace/mariadb-10.1/sql/sql_parse.cc:4248

#9  0x00003fff83ca4ee8 in mysql_parse (thd=0x3fffb12402d8, rawbuf=<optimized out>, length=<optimized out>, parser_state=0x3fff7d0d5a00) at /var/lib/jenkins/workspace/mariadb-10.1/sql/sql_parse.cc:7181

#10 0x00003fff83ca81e8 in dispatch_command (command=<optimized out>, thd=<optimized out>, packet=<optimized out>, packet_length=<optimized out>) at /var/lib/jenkins/workspace/mariadb-10.1/sql/sql_parse.cc:1470

#11 0x00003fff83ca8a04 in do_command (thd=0x3fffb12402d8) at /var/lib/jenkins/workspace/mariadb-10.1/sql/sql_parse.cc:1093

#12 0x00003fff83d95df4 in do_handle_one_connection (thd_arg=<optimized out>) at /var/lib/jenkins/workspace/mariadb-10.1/sql/sql_connect.cc:1350

#13 0x00003fff83d960ac in handle_one_connection (arg=0x3fffb12402d8) at /var/lib/jenkins/workspace/mariadb-10.1/sql/sql_connect.cc:1262

#14 0x00003fff84017638 in pfs_spawn_thread (arg=0x3fffb125e118) at /var/lib/jenkins/workspace/mariadb-10.1/storage/perfschema/pfs.cc:1860

#15 0x00003fff8340833c in start_thread (arg=0x3fff7d0d7180) at pthread_create.c:335

#16 0x00003fff830bb014 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96