sys_vars.wsrep_provider_basic failed in buildbot with Assertion `status_var.local_memory_used == 0 || !debug_assert_on_not_freed_memory' failed

sys_vars.wsrep_provider_basic            w2 [ fail ]

        Test ended at 2019-05-03 17:50:27

CURRENT_TEST: sys_vars.wsrep_provider_basic

mysqltest: At line 37: query 'SET @@global.wsrep_provider = @wsrep_provider_global_saved' failed: 2013: Lost connection to MySQL server during query

The result from queries just before the failure was:

< snip >

SELECT @@global.wsrep_provider;

@@global.wsrep_provider

none

# invalid values

SET @@global.wsrep_provider='/invalid/libgalera_smm.so';

ERROR 42000: Variable 'wsrep_provider' can't be set to the value of '/invalid/libgalera_smm.so'

SET @@global.wsrep_provider=NULL;

ERROR 42000: Variable 'wsrep_provider' can't be set to the value of 'NULL'

SELECT @@global.wsrep_provider;

@@global.wsrep_provider

none

SET @@global.wsrep_provider=1;

ERROR 42000: Incorrect argument type to variable 'wsrep_provider'

SELECT @@global.wsrep_provider;

@@global.wsrep_provider

none

# restore the initial value

SET @@global.wsrep_provider = @wsrep_provider_global_saved;

More results from queries before failure can be found in /mnt/buildbot/build/mariadb-10.3.15/mysql-test/var/2/log/wsrep_provider_basic.log

Server [mysqld.1 - pid: 3561, winpid: 3561, exit: 256] failed during test run

Server log from this test:

----------SERVER LOG START-----------

2019-05-03 17:50:22 1855 [Note] WSREP: Stop replication

2019-05-03 17:50:22 1855 [Note] WSREP: Provider was not loaded, in stop replication

2019-05-03 17:50:22 1855 [Note] WSREP: Read nil XID from storage engines, skipping position init

2019-05-03 17:50:22 1855 [Note] WSREP: wsrep_load(): loading provider library 'none'

2019-05-03 17:50:22 1855 [Note] WSREP: Stop replication

mysqld: /home/buildbot/buildbot/build/mariadb-10.3.15/sql/sql_class.cc:1670: virtual THD::~THD(): Assertion `status_var.local_memory_used == 0 || !debug_assert_on_not_freed_memory' failed.

190503 17:50:22 [ERROR] mysqld got signal 6 ;

This could be because you hit a bug. It is also possible that this binary

or one of the libraries it was linked against is corrupt, improperly built,

or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help

diagnose the problem, but since we have already crashed, 

something is definitely wrong and this may fail.

Server version: 10.3.15-MariaDB-debug-log

key_buffer_size=1048576

read_buffer_size=131072

max_used_connections=5

max_threads=153

thread_count=4

It is possible that mysqld could use up to 

key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 62003 K  bytes of memory

Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0xb17021b0

Attempting backtrace. You can use the following information to find out

where mysqld died. If you see no messages after this, something went

terribly wrong...

stack_bottom = 0xb08ff21c thread_stack 0x49000

mysqld: /home/buildbot/buildbot/build/mariadb-10.3.15/sql/sql_class.cc:1670: virtual THD::~THD(): Assertion `status_var.local_memory_used == 0 || !debug_assert_on_not_freed_memory' failed.

----------SERVER LOG END-------------

 - found 'core' (0/0)

Trying 'dbx' to get a backtrace

Trying 'gdb' to get a backtrace from coredump /mnt/buildbot/build/mariadb-10.3.15/mysql-test/var/2/log/sys_vars.wsrep_provider_basic/mysqld.1/data/core

Core generated by '/mnt/buildbot/build/mariadb-10.3.15/sql/mysqld'

Output from gdb follows. The first stack trace is from the failing thread.

The following stack traces are from all threads (so the failing one is

duplicated).

--------------------------

[New LWP 3719]

[New LWP 3720]

[New LWP 3562]

[New LWP 3565]

[New LWP 3571]

[New LWP 3716]

[New LWP 4374]

[New LWP 3570]

[New LWP 3569]

[Thread debugging using libthread_db enabled]

Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1".

Core was generated by `/mnt/buildbot/build/mariadb-10.3.15/sql/mysqld --defaults-group-suffix=.1 --def'.

Program terminated with signal SIGABRT, Aborted.

#0  0xb76f3c31 in __kernel_vsyscall ()

[Current thread is 1 (Thread 0xb08b5b40 (LWP 3719))]

#0  0xb76f3c31 in __kernel_vsyscall ()

#1  0xb6fd1e89 in raise () from /lib/i386-linux-gnu/libc.so.6

#2  0xb6fd33e7 in abort () from /lib/i386-linux-gnu/libc.so.6

#3  0xb6fcad07 in ?? () from /lib/i386-linux-gnu/libc.so.6

#4  0xb6fcad8b in __assert_fail () from /lib/i386-linux-gnu/libc.so.6

#5  0x804c82d9 in THD::~THD (this=0xb0d01458, __in_chrg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.3.15/sql/sql_class.cc:1669

#6  0x804c8675 in THD::~THD (this=0xb0d01458, __in_chrg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.3.15/sql/sql_class.cc:1676

#7  0x804115fb in one_thread_per_connection_end (thd=0xb0d01458, put_in_cache=true) at /home/buildbot/buildbot/build/mariadb-10.3.15/sql/mysqld.cc:3095

#8  0x8067ac4f in do_handle_one_connection (connect=0x83dc7e40) at /home/buildbot/buildbot/build/mariadb-10.3.15/sql/sql_connect.cc:1421

#9  0x8067a8bb in handle_one_connection (arg=0x83dc7e40) at /home/buildbot/buildbot/build/mariadb-10.3.15/sql/sql_connect.cc:1308

#10 0x80a25018 in pfs_spawn_thread (arg=0x83e01e90) at /home/buildbot/buildbot/build/mariadb-10.3.15/storage/perfschema/pfs.cc:1862

#11 0xb734c2b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0

#12 0xb708d16e in clone () from /lib/i386-linux-gnu/libc.so.6

Thread 9 (Thread 0xb18b7b40 (LWP 3569)):

#0  0xb76f3c31 in __kernel_vsyscall ()

#1  0xb73561cd in do_sigwait () from /lib/i386-linux-gnu/libpthread.so.0

#2  0xb735625e in sigwait () from /lib/i386-linux-gnu/libpthread.so.0

#3  0x80412119 in signal_hand (arg=0x0) at /home/buildbot/buildbot/build/mariadb-10.3.15/sql/mysqld.cc:3557

#4  0x80a25018 in pfs_spawn_thread (arg=0x83d7a220) at /home/buildbot/buildbot/build/mariadb-10.3.15/storage/perfschema/pfs.cc:1862

#5  0xb734c2b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0

#6  0xb708d16e in clone () from /lib/i386-linux-gnu/libc.so.6

Thread 8 (Thread 0xb186db40 (LWP 3570)):

#0  0xb76f3c31 in __kernel_vsyscall ()

#1  0xb7351a8c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0

#2  0x80f762fb in safe_cond_wait (cond=0x81980a60 <COND_slave_background>, mp=0x819831c0 <LOCK_slave_background>, file=0x8100a360 "/home/buildbot/buildbot/build/mariadb-10.3.15/include/mysql/psi/mysql_thread.h", line=1167) at /home/buildbot/buildbot/build/mariadb-10.3.15/mysys/thr_mutex.c:492

#3  0x8043cf03 in inline_mysql_cond_wait (that=0x81980a60 <COND_slave_background>, mutex=0x819831c0 <LOCK_slave_background>, src_file=0x8100afa8 "/home/buildbot/buildbot/build/mariadb-10.3.15/sql/slave.cc", src_line=513) at /home/buildbot/buildbot/build/mariadb-10.3.15/include/mysql/psi/mysql_thread.h:1167

#4  0x8043e3f5 in handle_slave_background (arg=0x0) at /home/buildbot/buildbot/build/mariadb-10.3.15/sql/slave.cc:513

#5  0x80a25018 in pfs_spawn_thread (arg=0x83dd0f60) at /home/buildbot/buildbot/build/mariadb-10.3.15/storage/perfschema/pfs.cc:1862

#6  0xb734c2b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0

#7  0xb708d16e in clone () from /lib/i386-linux-gnu/libc.so.6

Thread 7 (Thread 0xb16ffb40 (LWP 4374)):

#0  0xb76f3c31 in __kernel_vsyscall ()

#1  0xb7351e56 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0

#2  0x80f765a6 in safe_cond_timedwait (cond=0x821acce0 <COND_checkpoint>, mp=0x821acc60 <LOCK_checkpoint>, abstime=0xb16ff19c, file=0x81214724 "/home/buildbot/buildbot/build/mariadb-10.3.15/include/mysql/psi/mysql_thread.h", line=1204) at /home/buildbot/buildbot/build/mariadb-10.3.15/mysys/thr_mutex.c:546

#3  0x80a84082 in inline_mysql_cond_timedwait (that=0x821acce0 <COND_checkpoint>, mutex=0x821acc60 <LOCK_checkpoint>, abstime=0xb16ff19c, src_file=0x81214774 "/home/buildbot/buildbot/build/mariadb-10.3.15/storage/maria/ma_servicethread.c", src_line=116) at /home/buildbot/buildbot/build/mariadb-10.3.15/include/mysql/psi/mysql_thread.h:1204

#4  0x80a845c5 in my_service_thread_sleep (control=0x817d3ef8 <checkpoint_control>, sleep_time=29000000000) at /home/buildbot/buildbot/build/mariadb-10.3.15/storage/maria/ma_servicethread.c:115

#5  0x80a785ef in ma_checkpoint_background (arg=0x1e) at /home/buildbot/buildbot/build/mariadb-10.3.15/storage/maria/ma_checkpoint.c:707

#6  0x80a25018 in pfs_spawn_thread (arg=0x83e01e90) at /home/buildbot/buildbot/build/mariadb-10.3.15/storage/perfschema/pfs.cc:1862

#7  0xb734c2b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0

#8  0xb708d16e in clone () from /lib/i386-linux-gnu/libc.so.6

Thread 6 (LWP 3716):

#0  0xb76f3c31 in __kernel_vsyscall ()

#1  0xb734c381 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0

#2  0xb708d16e in clone () from /lib/i386-linux-gnu/libc.so.6

Thread 5 (Thread 0xb0efeb40 (LWP 3571)):

#0  0xb76f3c31 in __kernel_vsyscall ()

#1  0xb705633a in nanosleep () from /lib/i386-linux-gnu/libc.so.6

#2  0xb705626d in sleep () from /lib/i386-linux-gnu/libc.so.6

#3  0x80729976 in wsrep_close_client_connections (wait_to_end=1 '\001', except_caller_thd=0x0) at /home/buildbot/buildbot/build/mariadb-10.3.15/sql/wsrep_mysqld.cc:2319

#4  0x80725c58 in wsrep_stop_replication (thd=0xb09005d8) at /home/buildbot/buildbot/build/mariadb-10.3.15/sql/wsrep_mysqld.cc:927

#5  0x80732d55 in wsrep_provider_update (self=0x8199c000 <Sys_wsrep_provider>, thd=0xb09005d8, type=SHOW_OPT_GLOBAL) at /home/buildbot/buildbot/build/mariadb-10.3.15/sql/wsrep_var.cc:348

#6  0x8043829a in sys_var::update (this=0x8199c000 <Sys_wsrep_provider>, thd=0xb09005d8, var=0xb09e7a00) at /home/buildbot/buildbot/build/mariadb-10.3.15/sql/set_var.cc:209

#7  0x80439d22 in set_var::update (this=0xb09e7a00, thd=0xb09005d8) at /home/buildbot/buildbot/build/mariadb-10.3.15/sql/set_var.cc:837

#8  0x804399e8 in sql_set_variables (thd=0xb09005d8, var_list=0xb09035ec, free=true) at /home/buildbot/buildbot/build/mariadb-10.3.15/sql/set_var.cc:740

#9  0x8053157c in mysql_execute_command (thd=0xb09005d8) at /home/buildbot/buildbot/build/mariadb-10.3.15/sql/sql_parse.cc:5140

#10 0x8053adcf in mysql_parse (thd=0xb09005d8, rawbuf=0xb09e7860 "SET @@global.wsrep_provider = @wsrep_provider_global_saved", length=58, parser_state=0xb0efdb98, is_com_multi=false, is_next_command=false) at /home/buildbot/buildbot/build/mariadb-10.3.15/sql/sql_parse.cc:8091

#11 0x80527064 in dispatch_command (command=COM_QUERY, thd=0xb09005d8, packet=0xb0905cf9 "SET @@global.wsrep_provider = @wsrep_provider_global_saved", packet_length=58, is_com_multi=false, is_next_command=false) at /home/buildbot/buildbot/build/mariadb-10.3.15/sql/sql_parse.cc:1858

#12 0x80525929 in do_command (thd=0xb09005d8) at /home/buildbot/buildbot/build/mariadb-10.3.15/sql/sql_parse.cc:1404

#13 0x8067ab6e in do_handle_one_connection (connect=0x83e01fc0) at /home/buildbot/buildbot/build/mariadb-10.3.15/sql/sql_connect.cc:1402

#14 0x8067a8bb in handle_one_connection (arg=0x83e01fc0) at /home/buildbot/buildbot/build/mariadb-10.3.15/sql/sql_connect.cc:1308

#15 0x80a25018 in pfs_spawn_thread (arg=0x83e16bc0) at /home/buildbot/buildbot/build/mariadb-10.3.15/storage/perfschema/pfs.cc:1862

#16 0xb734c2b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0

#17 0xb708d16e in clone () from /lib/i386-linux-gnu/libc.so.6

Thread 4 (Thread 0xb2bcdb40 (LWP 3565)):

#0  0xb76f3c31 in __kernel_vsyscall ()

#1  0xb7351e56 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0

#2  0x80f765a6 in safe_cond_timedwait (cond=0x821f4500 <COND_timer>, mp=0x821f4480 <LOCK_timer>, abstime=0xb2bcd250, file=0x81389260 "/home/buildbot/buildbot/build/mariadb-10.3.15/include/mysql/psi/mysql_thread.h", line=1204) at /home/buildbot/buildbot/build/mariadb-10.3.15/mysys/thr_mutex.c:546

#3  0x80f774b3 in inline_mysql_cond_timedwait (that=0x821f4500 <COND_timer>, mutex=0x821f4480 <LOCK_timer>, abstime=0xb2bcd250, src_file=0x813892b0 "/home/buildbot/buildbot/build/mariadb-10.3.15/mysys/thr_timer.c", src_line=292) at /home/buildbot/buildbot/build/mariadb-10.3.15/include/mysql/psi/mysql_thread.h:1204

#4  0x80f78023 in timer_handler (arg=0x0) at /home/buildbot/buildbot/build/mariadb-10.3.15/mysys/thr_timer.c:292

#5  0x80a25018 in pfs_spawn_thread (arg=0x83cb5c48) at /home/buildbot/buildbot/build/mariadb-10.3.15/storage/perfschema/pfs.cc:1862

#6  0xb734c2b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0

#7  0xb708d16e in clone () from /lib/i386-linux-gnu/libc.so.6

Thread 3 (Thread 0xb6e137c0 (LWP 3562)):

#0  0xb76f3c31 in __kernel_vsyscall ()

#1  0xb7082f8f in poll () from /lib/i386-linux-gnu/libc.so.6

#2  0x8041896b in handle_connections_sockets () at /home/buildbot/buildbot/build/mariadb-10.3.15/sql/mysqld.cc:6737

#3  0x80417c5d in mysqld_main (argc=129, argv=0x83b738f0) at /home/buildbot/buildbot/build/mariadb-10.3.15/sql/mysqld.cc:6206

#4  0x8040ba3b in main (argc=6, argv=0xbfd1eb04) at /home/buildbot/buildbot/build/mariadb-10.3.15/sql/main.cc:25

Thread 2 (Thread 0xb08ffb40 (LWP 3720)):

#0  0xb70c2cec in _dl_addr () from /lib/i386-linux-gnu/libc.so.6

#1  0xb736447d in dladdr () from /lib/i386-linux-gnu/libdl.so.2

#2  0x80f8d697 in my_addr_resolve (ptr=0x80f715b2 <my_print_stacktrace+60>, loc=0xb08fe734) at /home/buildbot/buildbot/build/mariadb-10.3.15/mysys/my_addr_resolve.c:212

#3  0x80f714ed in print_with_addr_resolve (addrs=0xb08fe780, n=16) at /home/buildbot/buildbot/build/mariadb-10.3.15/mysys/stacktrace.c:254

#4  0x80f715e8 in my_print_stacktrace (stack_bottom=0xb08ff21c "\260!p\261", thread_stack=299008, silent=0 '\000') at /home/buildbot/buildbot/build/mariadb-10.3.15/mysys/stacktrace.c:273

#5  0x8080e920 in handle_fatal_signal (sig=6) at /home/buildbot/buildbot/build/mariadb-10.3.15/sql/signal_handler.cc:207

#6  <signal handler called>

#7  0xb76f3c31 in __kernel_vsyscall ()

#8  0xb6fd1e89 in raise () from /lib/i386-linux-gnu/libc.so.6

#9  0xb6fd33e7 in abort () from /lib/i386-linux-gnu/libc.so.6

#10 0xb6fcad07 in ?? () from /lib/i386-linux-gnu/libc.so.6

#11 0xb6fcad8b in __assert_fail () from /lib/i386-linux-gnu/libc.so.6

#12 0x804c82d9 in THD::~THD (this=0xb17021b0, __in_chrg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.3.15/sql/sql_class.cc:1669

#13 0x804c8675 in THD::~THD (this=0xb17021b0, __in_chrg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.3.15/sql/sql_class.cc:1676

#14 0x804115fb in one_thread_per_connection_end (thd=0xb17021b0, put_in_cache=true) at /home/buildbot/buildbot/build/mariadb-10.3.15/sql/mysqld.cc:3095

#15 0x8067ac4f in do_handle_one_connection (connect=0x83dc7bf0) at /home/buildbot/buildbot/build/mariadb-10.3.15/sql/sql_connect.cc:1421

#16 0x8067a8bb in handle_one_connection (arg=0x83dc7bf0) at /home/buildbot/buildbot/build/mariadb-10.3.15/sql/sql_connect.cc:1308

#17 0x80a25018 in pfs_spawn_thread (arg=0x83dfada8) at /home/buildbot/buildbot/build/mariadb-10.3.15/storage/perfschema/pfs.cc:1862

#18 0xb734c2b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0

#19 0xb708d16e in clone () from /lib/i386-linux-gnu/libc.so.6

Thread 1 (Thread 0xb08b5b40 (LWP 3719)):

#0  0xb76f3c31 in __kernel_vsyscall ()

#1  0xb6fd1e89 in raise () from /lib/i386-linux-gnu/libc.so.6

#2  0xb6fd33e7 in abort () from /lib/i386-linux-gnu/libc.so.6

#3  0xb6fcad07 in ?? () from /lib/i386-linux-gnu/libc.so.6

#4  0xb6fcad8b in __assert_fail () from /lib/i386-linux-gnu/libc.so.6

#5  0x804c82d9 in THD::~THD (this=0xb0d01458, __in_chrg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.3.15/sql/sql_class.cc:1669

#6  0x804c8675 in THD::~THD (this=0xb0d01458, __in_chrg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.3.15/sql/sql_class.cc:1676

#7  0x804115fb in one_thread_per_connection_end (thd=0xb0d01458, put_in_cache=true) at /home/buildbot/buildbot/build/mariadb-10.3.15/sql/mysqld.cc:3095

#8  0x8067ac4f in do_handle_one_connection (connect=0x83dc7e40) at /home/buildbot/buildbot/build/mariadb-10.3.15/sql/sql_connect.cc:1421

#9  0x8067a8bb in handle_one_connection (arg=0x83dc7e40) at /home/buildbot/buildbot/build/mariadb-10.3.15/sql/sql_connect.cc:1308

#10 0x80a25018 in pfs_spawn_thread (arg=0x83e01e90) at /home/buildbot/buildbot/build/mariadb-10.3.15/storage/perfschema/pfs.cc:1862

#11 0xb734c2b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0

#12 0xb708d16e in clone () from /lib/i386-linux-gnu/libc.so.6

Compressed file /mnt/buildbot/build/mariadb-10.3.15/mysql-test/var/2/log/sys_vars.wsrep_provider_basic/mysqld.1/data/core

 - saving '/mnt/buildbot/build/mariadb-10.3.15/mysql-test/var/2/log/sys_vars.wsrep_provider_basic/' to '/mnt/buildbot/build/mariadb-10.3.15/mysql-test/var/log/sys_vars.wsrep_provider_basic/'