[MDEV-7140] Sporadic buildbot failures in rpl.rpl_semi_sync_uninstall_plugin, 2013: 'Lost connection to MySQL server Created: 2014-11-19  Updated: 2023-11-29  Resolved: 2023-11-29

Status: Closed
Project: MariaDB Server
Component/s: Tests
Affects Version/s: 5.5, 10.0, 10.1, 10.2
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Elena Stepanova Assignee: Andrei Elkin
Resolution: Won't Fix Votes: 0
Labels: buildbot, tests

Issue Links:
Relates
relates to MDEV-10892 rpl.rpl_semi_sync_uninstall_plugin fa... Closed
relates to MDEV-7069 Fix buildbot failures in main server ... Stalled
relates to MDEV-24561 rpl.rpl_semi_sync_uninstall_plugin fa... Closed

 Description   

http://buildbot.askmonty.org/buildbot/builders/kvm-deb-trusty-amd64/builds/606/steps/test_4/logs/stdio

CURRENT_TEST: rpl.rpl_semi_sync_uninstall_plugin
 
--- /usr/share/mysql/mysql-test/suite/rpl/r/rpl_semi_sync_uninstall_plugin.result	2014-11-19 07:12:09.000000000 +0000
 
+++ /run/shm/var/4/log/rpl_semi_sync_uninstall_plugin.reject	2014-11-19 15:00:33.507204650 +0000
 
@@ -20,28 +20,22 @@
 
 rpl_semi_sync_slave	DELETED
 
 [connection master]
 
 UNINSTALL PLUGIN rpl_semi_sync_master;
 
-Warnings:
 
-Warning	1620	Plugin is busy and will be uninstalled on shutdown
 
 select plugin_name,plugin_status from information_schema.plugins where plugin_name like 'rpl_%';
 
 plugin_name	plugin_status
 
-rpl_semi_sync_master	DELETED
 
 CREATE TABLE t1(i int);
 
 INSERT INTO t1 values (2);
 
 DROP TABLE t1;
 
 [connection slave]
 
 show status like "Rpl_semi_sync_slave_status";
 
 Variable_name	Value
 
-Rpl_semi_sync_slave_status	ON
 
+Rpl_semi_sync_slave_status	OFF
 
 [connection master]
 
 show status like "Rpl_semi_sync_master_status";
 
 Variable_name	Value
 
-Rpl_semi_sync_master_status	ON
 
 show status like "Rpl_semi_sync_master_clients";
 
 Variable_name	Value
 
-Rpl_semi_sync_master_clients	1
 
 select plugin_name,plugin_status from information_schema.plugins where plugin_name like 'rpl_%';
 
 plugin_name	plugin_status
 
-rpl_semi_sync_master	DELETED
 
 [connection slave]
 
 include/stop_slave.inc
 
 select plugin_name,plugin_status from information_schema.plugins where plugin_name like 'rpl_%';
 
 
 
mysqltest: Result length mismatch
 
 
 
 - saving '/run/shm/var/4/log/rpl.rpl_semi_sync_uninstall_plugin-stmt/' to '/run/shm/var/log/rpl.rpl_semi_sync_uninstall_plugin-stmt/'
 
 
 
Retrying test rpl.rpl_semi_sync_uninstall_plugin, attempt(2/3)...
 
 
 
worker[4] > Restart  - not started
 
worker[4] > Restart  - not started
 
***Warnings generated in error logs during shutdown after running tests: rpl.rpl_semi_sync_uninstall_plugin
 
 
 
141119 15:00:33 [Warning] Master server does not support semi-sync, fallback to asynchronous replication
 
 
 
worker[4] > Restart [mysqld.1 - pid: 19176, winpid: 19176] - running with different options '--binlog-format=statement --log-bin=master-bin --log-bin=master-bin' != '--log-bin=master-bin --binlog-format=statement --log-bin=master-bin'
 
rpl.rpl_semi_sync_uninstall_plugin 'stmt' w4 [ retry-pass ]    130

Also observed on 10.0, 10.1

Another kind – timeout:
http://buildbot.askmonty.org/buildbot/builders/labrador/builds/4438/steps/test/logs/stdio

rpl.rpl_semi_sync_uninstall_plugin 'stmt' [ fail ]  timeout after 900 seconds
 
        Test ended at 2015-01-06 17:51:50
 
 
 
Test case timeout after 900 seconds
 
 
 
== /private/var/lib/buildslave/maria-slave/labrador/build/mysql-test/var/log/rpl_semi_sync_uninstall_plugin.log == 
UNINSTALL PLUGIN rpl_semi_sync_master;
 
CREATE TABLE t1(i int);
 
INSERT INTO t1 values (1);
 
DROP TABLE t1;
 
[connection slave]
 
include/install_semisync.inc
 
[connection slave]
 
UNINSTALL PLUGIN rpl_semi_sync_slave;
 
Warnings:
 
Warning	1620	Plugin is busy and will be uninstalled on shutdown
 
select plugin_name,plugin_status from information_schema.plugins where plugin_name like 'rpl_%';
 
plugin_name	plugin_status
 
rpl_semi_sync_slave	DELETED
 
[connection master]
 
UNINSTALL PLUGIN rpl_semi_sync_master;
 
select plugin_name,plugin_status from information_schema.plugins where plugin_name like 'rpl_%';
 
plugin_name	plugin_status
 
CREATE TABLE t1(i int);
 
INSERT INTO t1 values (2);
 
DROP TABLE t1;
 
 
 
 == /private/var/lib/buildslave/maria-slave/labrador/build/mysql-test/var/tmp/analyze-timeout-mysqld.1.err ==
 
mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/private/var/lib/buildslave/maria-slave/labrador/build/mysql-test/var/tmp/mysqld.1.sock' (61 "Connection refused")
 
 
 
 == /private/var/lib/buildslave/maria-slave/labrador/build/mysql-test/var/tmp/analyze-timeout-mysqld.2.err ==
 
mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/private/var/lib/buildslave/maria-slave/labrador/build/mysql-test/var/tmp/mysqld.2.sock' (61 "Connection refused")



 Comments   
Comment by Elena Stepanova [ 2015-01-11 ]

Kristian,

You mentioned that you'd been looking into some semisync-related failures recently, maybe you figured something out regarding these, too?

Comment by Elena Stepanova [ 2016-10-17 ]

Recent variation:
http://buildbot.askmonty.org/buildbot/builders/kvm-deb-precise-x86/builds/4119/steps/test_4/logs/stdio

rpl.rpl_semi_sync_uninstall_plugin 'stmt' w3 [ fail ]
 
        Test ended at 2016-10-14 14:59:18
 
 
 
CURRENT_TEST: rpl.rpl_semi_sync_uninstall_plugin
 
=== SHOW MASTER STATUS ===
 
---- 1. ----
 
File	slave-bin.000001
 
Position	659
 
Binlog_Do_DB	
Binlog_Ignore_DB	
==========================
 
 
 
=== SHOW SLAVE STATUS ===
 
---- 1. ----
 
Slave_IO_State	
Master_Host	127.0.0.1
 
Master_User	root
 
Master_Port	16020
 
Connect_Retry	1
 
Master_Log_File	master-bin.000001
 
Read_Master_Log_Pos	659
 
Relay_Log_File	slave-relay-bin.000002
 
Relay_Log_Pos	944
 
Relay_Master_Log_File	master-bin.000001
 
Slave_IO_Running	No
 
Slave_SQL_Running	Yes
 
Replicate_Do_DB	
Replicate_Ignore_DB	
Replicate_Do_Table	
Replicate_Ignore_Table	
Replicate_Wild_Do_Table	
Replicate_Wild_Ignore_Table	
Last_Errno	0
 
Last_Error	
Skip_Counter	0
 
Exec_Master_Log_Pos	659
 
Relay_Log_Space	1238
 
Until_Condition	None
 
Until_Log_File	
Until_Log_Pos	0
 
Master_SSL_Allowed	No
 
Master_SSL_CA_File	
Master_SSL_CA_Path	
Master_SSL_Cert	
Master_SSL_Cipher	
Master_SSL_Key	
Seconds_Behind_Master	
Master_SSL_Verify_Server_Cert	No
 
Last_IO_Errno	1593
 
Last_IO_Error	Fatal error: Failed to run 'after_read_event' hook
 
Last_SQL_Errno	0
 
Last_SQL_Error	
Replicate_Ignore_Server_Ids	
Master_Server_Id	1
 
=========================
 
 
 
=== SHOW PROCESSLIST ===
 
---- 1. ----
 
Id	2
 
User	root
 
Host	localhost:42791
 
db	test
 
Command	Sleep
 
Time	300
 
State	
Info	
Progress	0.000
 
---- 2. ----
 
Id	3
 
User	root
 
Host	localhost:42792
 
db	test
 
Command	Sleep
 
Time	300
 
State	
Info	
Progress	0.000
 
---- 3. ----
 
Id	5
 
User	system user
 
Host	
db	
Command	Connect
 
Time	300
 
State	Slave has read all relay log; waiting for the slave I/O thread to update it
 
Info	
Progress	0.000
 
---- 4. ----
 
Id	6
 
User	root
 
Host	localhost:42798
 
db	test
 
Command	Query
 
Time	0
 
State	
Info	SHOW PROCESSLIST
 
Progress	0.000
 
---- 5. ----
 
Id	7
 
User	root
 
Host	localhost:42799
 
db	test
 
Command	Sleep
 
Time	300
 
State	
Info	
Progress	0.000
 
========================
 
 
 
analyze: sync_with_master
 
mysqltest: At line 71: sync_slave_with_master failed: 'select master_pos_wait('master-bin.000001', 1073, 300)' returned -1 indicating timeout after 300 seconds
 
 
 
The result from queries just before the failure was:
 
< snip >
 
UNINSTALL PLUGIN rpl_semi_sync_master;
 
CREATE TABLE t1(i int);
 
INSERT INTO t1 values (1);
 
DROP TABLE t1;
 
[connection slave]
 
include/install_semisync.inc
 
[connection slave]
 
UNINSTALL PLUGIN rpl_semi_sync_slave;
 
Warnings:
 
Warning	1620	Plugin is busy and will be uninstalled on shutdown
 
select plugin_name,plugin_status from information_schema.plugins where plugin_name like 'rpl_%';
 
plugin_name	plugin_status
 
rpl_semi_sync_slave	DELETED
 
[connection master]
 
UNINSTALL PLUGIN rpl_semi_sync_master;
 
select plugin_name,plugin_status from information_schema.plugins where plugin_name like 'rpl_%';
 
plugin_name	plugin_status
 
CREATE TABLE t1(i int);
 
INSERT INTO t1 values (2);
 
DROP TABLE t1;
 
 
 
More results from queries before failure can be found in /run/shm/var/3/log/rpl_semi_sync_uninstall_plugin.log
 
 
 
 
 
 == /run/shm/var/3/tmp/analyze-sync_with_master-mysqld.1.err ==
 
 
 
############################## default ##############################
 
 
 
**** SHOW WARNINGS on default ****
 
SHOW WARNINGS;
 
Level	Code	Message
 
 
 
**** SELECT replication-related variables on default ****
 
SELECT NOW(), @@SERVER_ID;
 
NOW()	@@SERVER_ID
 
2016-10-14 14:59:18	1
 
 
 
**** SHOW SLAVE STATUS on default ****
 
SHOW SLAVE STATUS;
 
 
 
**** SHOW MASTER STATUS on default ****
 
SHOW MASTER STATUS;
 
File	master-bin.000001
 
Position	1073
 
Binlog_Do_DB	
Binlog_Ignore_DB	
 
 
**** SHOW SLAVE HOSTS on default ****
 
SHOW SLAVE HOSTS;
 
 
 
**** SHOW PROCESSLIST on default ****
 
SHOW PROCESSLIST;
 
Id	User	Host	db	Command	Time	State	Info	Progress
 
9	root	localhost	NULL	Query	0	NULL	SHOW PROCESSLIST	0.000
 
 
 
**** SHOW BINARY LOGS on default ****
 
SHOW BINARY LOGS;
 
Log_name	File_size
 
master-bin.000001	1073
 
 
 
**** SHOW BINLOG EVENTS on default ****
 
binlog_name = 'master-bin.000001'
 
SHOW BINLOG EVENTS IN 'master-bin.000001';
 
Log_name	Pos	Event_type	Server_id	End_log_pos	Info
 
master-bin.000001	4	Format_desc	1	245	Server ver: 5.5.53-MariaDB-1~precise, Binlog ver: 4
 
master-bin.000001	245	Query	1	330	use `test`; CREATE TABLE t1(i int)
 
master-bin.000001	330	Query	1	398	BEGIN
 
master-bin.000001	398	Query	1	486	use `test`; INSERT INTO t1 values (1)
 
master-bin.000001	486	Query	1	555	COMMIT
 
master-bin.000001	555	Query	1	659	use `test`; DROP TABLE `t1` /* generated by server */
 
master-bin.000001	659	Query	1	744	use `test`; CREATE TABLE t1(i int)
 
master-bin.000001	744	Query	1	812	BEGIN
 
master-bin.000001	812	Query	1	900	use `test`; INSERT INTO t1 values (2)
 
master-bin.000001	900	Query	1	969	COMMIT
 
master-bin.000001	969	Query	1	1073	use `test`; DROP TABLE `t1` /* generated by server */
 
 
 
**** SHOW RELAYLOG EVENTS on default ****
 
relaylog_name = 'No such row'
 
SHOW RELAYLOG EVENTS IN 'No such row';
 
Log_name	Pos	Event_type	Server_id	End_log_pos	Info
 
 
 
 == /run/shm/var/3/tmp/analyze-sync_with_master-mysqld.2.err ==
 
 
 
############################## default ##############################
 
 
 
**** SHOW WARNINGS on default ****
 
SHOW WARNINGS;
 
Level	Code	Message
 
 
 
**** SELECT replication-related variables on default ****
 
SELECT NOW(), @@SERVER_ID;
 
NOW()	@@SERVER_ID
 
2016-10-14 14:59:18	2
 
 
 
**** SHOW SLAVE STATUS on default ****
 
SHOW SLAVE STATUS;
 
Slave_IO_State	
Master_Host	127.0.0.1
 
Master_User	root
 
Master_Port	16020
 
Connect_Retry	1
 
Master_Log_File	master-bin.000001
 
Read_Master_Log_Pos	659
 
Relay_Log_File	slave-relay-bin.000002
 
Relay_Log_Pos	944
 
Relay_Master_Log_File	master-bin.000001
 
Slave_IO_Running	No
 
Slave_SQL_Running	Yes
 
Replicate_Do_DB	
Replicate_Ignore_DB	
Replicate_Do_Table	
Replicate_Ignore_Table	
Replicate_Wild_Do_Table	
Replicate_Wild_Ignore_Table	
Last_Errno	0
 
Last_Error	
Skip_Counter	0
 
Exec_Master_Log_Pos	659
 
Relay_Log_Space	1238
 
Until_Condition	None
 
Until_Log_File	
Until_Log_Pos	0
 
Master_SSL_Allowed	No
 
Master_SSL_CA_File	
Master_SSL_CA_Path	
Master_SSL_Cert	
Master_SSL_Cipher	
Master_SSL_Key	
Seconds_Behind_Master	NULL
 
Master_SSL_Verify_Server_Cert	No
 
Last_IO_Errno	1593
 
Last_IO_Error	Fatal error: Failed to run 'after_read_event' hook
 
Last_SQL_Errno	0
 
Last_SQL_Error	
Replicate_Ignore_Server_Ids	
Master_Server_Id	1
 
 
 
**** SHOW MASTER STATUS on default ****
 
SHOW MASTER STATUS;
 
File	slave-bin.000001
 
Position	659
 
Binlog_Do_DB	
Binlog_Ignore_DB	
 
 
**** SHOW SLAVE HOSTS on default ****
 
SHOW SLAVE HOSTS;
 
 
 
**** SHOW PROCESSLIST on default ****
 
SHOW PROCESSLIST;
 
Id	User	Host	db	Command	Time	State	Info	Progress
 
5	system user		NULL	Connect	300	Slave has read all relay log; waiting for the slave I/O thread to update it	NULL	0.000
 
9	root	localhost	NULL	Query	0	NULL	SHOW PROCESSLIST	0.000
 
 
 
**** SHOW BINARY LOGS on default ****
 
SHOW BINARY LOGS;
 
Log_name	File_size
 
slave-bin.000001	659
 
 
 
**** SHOW BINLOG EVENTS on default ****
 
binlog_name = 'slave-bin.000001'
 
SHOW BINLOG EVENTS IN 'slave-bin.000001';
 
Log_name	Pos	Event_type	Server_id	End_log_pos	Info
 
slave-bin.000001	4	Format_desc	2	245	Server ver: 5.5.53-MariaDB-1~precise, Binlog ver: 4
 
slave-bin.000001	245	Query	1	330	use `test`; CREATE TABLE t1(i int)
 
slave-bin.000001	330	Query	1	398	BEGIN
 
slave-bin.000001	398	Query	1	486	use `test`; INSERT INTO t1 values (1)
 
slave-bin.000001	486	Query	1	555	COMMIT
 
slave-bin.000001	555	Query	1	659	use `test`; DROP TABLE `t1` /* generated by server */
 
 
 
**** SHOW RELAYLOG EVENTS on default ****
 
relaylog_name = 'slave-relay-bin.000002'
 
SHOW RELAYLOG EVENTS IN 'slave-relay-bin.000002';
 
Log_name	Pos	Event_type	Server_id	End_log_pos	Info
 
slave-relay-bin.000002	4	Format_desc	2	245	Server ver: 5.5.53-MariaDB-1~precise, Binlog ver: 4
 
slave-relay-bin.000002	245	Rotate	1	0	master-bin.000001;pos=4
 
slave-relay-bin.000002	289	Format_desc	1	245	Server ver: 5.5.53-MariaDB-1~precise, Binlog ver: 4
 
slave-relay-bin.000002	530	Query	1	330	use `test`; CREATE TABLE t1(i int)
 
slave-relay-bin.000002	615	Query	1	398	BEGIN
 
slave-relay-bin.000002	683	Query	1	486	use `test`; INSERT INTO t1 values (1)
 
slave-relay-bin.000002	771	Query	1	555	COMMIT
 
slave-relay-bin.000002	840	Query	1	659	use `test`; DROP TABLE `t1` /* generated by server */

Comment by Elena Stepanova [ 2016-11-06 ]

Yet another variation:
http://buildbot.askmonty.org/buildbot/builders/kvm-deb-trusty-amd64/builds/3869/steps/test_4/logs/stdio

rpl.rpl_semi_sync_uninstall_plugin 'stmt' w2 [ fail ]
 
        Test ended at 2016-11-03 20:37:10
 
 
 
CURRENT_TEST: rpl.rpl_semi_sync_uninstall_plugin
 
--- /usr/share/mysql/mysql-test/suite/rpl/r/rpl_semi_sync_uninstall_plugin.result	2016-11-03 16:32:20.000000000 +0000
 
+++ /run/shm/var/2/log/rpl_semi_sync_uninstall_plugin.reject	2016-11-03 20:37:10.286035085 +0000
 
@@ -52,8 +52,10 @@
 
 [connection master]
 
 show status like "Rpl_semi_sync_master_clients";
 
 Variable_name	Value
 
+Rpl_semi_sync_master_clients	1
 
 select plugin_name,plugin_status from information_schema.plugins where plugin_name like 'rpl_%';
 
 plugin_name	plugin_status
 
+rpl_semi_sync_master	DELETED
 
 CREATE TABLE t1(i int);
 
 INSERT INTO t1 values (3);
 
 DROP TABLE t1;
 
 
 
mysqltest: Result length mismatch

Comment by Elena Stepanova [ 2016-11-20 ]

Crash:
http://buildbot.askmonty.org/buildbot/builders/kvm-deb-jessie-x86/builds/1430/steps/test_5/logs/stdio

rpl.rpl_semi_sync_uninstall_plugin 'stmt' w1 [ fail ]
 
        Test ended at 2016-11-16 12:56:45
 
 
 
CURRENT_TEST: rpl.rpl_semi_sync_uninstall_plugin
 
mysqltest: In included file "./include/wait_condition.inc": 
included from /usr/share/mysql/mysql-test/suite/rpl/t/rpl_semi_sync_uninstall_plugin.test at line 124:
 
At line 43: query 'let $success= `$wait_condition`' failed with wrong errno 2013: 'Lost connection to MySQL server during query', instead of 0...
 
 
 
The result from queries just before the failure was:
 
< snip >
 
show status like "Rpl_semi_sync_slave_status";
 
Variable_name	Value
 
Rpl_semi_sync_slave_status	ON
 
[connection master]
 
show status like "Rpl_semi_sync_master_status";
 
Variable_name	Value
 
Rpl_semi_sync_master_status	ON
 
show status like "Rpl_semi_sync_master_clients";
 
Variable_name	Value
 
Rpl_semi_sync_master_clients	1
 
select plugin_name,plugin_status from information_schema.plugins where plugin_name like 'rpl_%';
 
plugin_name	plugin_status
 
rpl_semi_sync_master	DELETED
 
[connection slave]
 
include/stop_slave.inc
 
select plugin_name,plugin_status from information_schema.plugins where plugin_name like 'rpl_%';
 
plugin_name	plugin_status
 
[connection master]
 
create table t2 (a int);
 
drop table t2;


2016-11-16 12:56:45 2960564032 [Warning] Timeout waiting for reply of binlog (file: master-bin.000001, pos: 1981), semi-sync up to file master-bin.000001, position 1857.
 
2016-11-16 12:56:45 2960564032 [Note] Semi-sync replication switched OFF.
 
2016-11-16 12:56:45 2959957824 [Note] unregister_replicator OK
 
161116 12:56:45 [ERROR] mysqld got signal 11 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.1.20-MariaDB-1~jessie
 
key_buffer_size=1048576
 
read_buffer_size=131072
 
max_used_connections=7
 
max_threads=153
 
thread_count=5
 
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 61770 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x0xaff24008
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0xb076966c thread_stack 0x48400
 
/usr/sbin/mysqld(my_print_stacktrace+0x28)[0xb7249748]
 
/usr/sbin/mysqld(handle_fatal_signal+0x2cf)[0xb6d5fb8f]
 
linux-gate.so.1(__kernel_sigreturn+0x0)[0xb6923d18]
 
/usr/sbin/mysqld(+0x931439)[0xb7278439]
 
/usr/sbin/mysqld(my_copy_fix_mb+0x3e)[0xb72678ee]
 
/usr/sbin/mysqld(_ZN13String_copier16well_formed_copyEPK15charset_info_stPcjS2_PKcjj+0x69)[0xb6c47f39]
 
/usr/sbin/mysqld(_ZN15Field_varstring5storeEPKcjPK15charset_info_st+0x43)[0xb6d55453]
 
/usr/sbin/mysqld(+0x2e7933)[0xb6c2e933]
 
/usr/sbin/mysqld(_Z24plugin_foreach_with_maskP3THDPFcS0_P13st_plugin_intPvEijS3_+0x27b)[0xb6bd37db]
 
/usr/sbin/mysqld(_Z12fill_pluginsP3THDP10TABLE_LISTP4Item+0x2b)[0xb6c24c4b]
 
/usr/sbin/mysqld(_Z24get_schema_tables_resultP4JOIN23enum_schema_table_state+0x241)[0xb6c3b181]
 
/usr/sbin/mysqld(_ZN4JOIN10exec_innerEv+0x840)[0xb6c20c30]
 
/usr/sbin/mysqld(_ZN4JOIN4execEv+0x4f)[0xb6c22e8f]
 
/usr/sbin/mysqld(_Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe8)[0xb6c1f658]
 
/usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x133)[0xb6c1ffd3]
 
/usr/sbin/mysqld(+0x273dfc)[0xb6bbadfc]
 
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x6d32)[0xb6bc7b42]
 
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x28c)[0xb6bcbc2c]
 
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x2247)[0xb6bcee07]
 
/usr/sbin/mysqld(_Z10do_commandP3THD+0x188)[0xb6bcf588]
 
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x1e4)[0xb6c9d6f4]
 
/usr/sbin/mysqld(handle_one_connection+0x3e)[0xb6c9d89e]
 
/usr/sbin/mysqld(+0x8ade7b)[0xb71f4e7b]
 
/lib/i386-linux-gnu/i686/cmov/libpthread.so.0(+0x6efb)[0xb6904efb]
 
/lib/i386-linux-gnu/i686/cmov/libc.so.6(clone+0x5e)[0xb63b0d0e]
 
 
 
Trying to get some variables.
 
Some pointers may be invalid and cause the dump to abort.
 
Query (0xac014018): select count(*) = 0 from information_schema.plugins where plugin_name like 'rpl_%'
 
Connection ID (thread ID): 8
 
Status: NOT_KILLED

Comment by Alice Sherepa [ 2017-08-15 ]

rpl.rpl_semi_sync_uninstall_plugin fails on 5.5 http://buildbot.askmonty.org/buildbot/builders/kvm-deb-trusty-x86/builds/4302/steps/mtr/logs/stdio

Comment by Sergey Vojtovich [ 2017-10-11 ]

Both this one and MDEV-10892 seem to have the same cause. It is not semisync plugin, but rather race condition in plugin_foreach_with_mask(). It iterates plugins without mutex protection.

And of course it has to be fixed independently of if we want to make semisync static or not.

Comment by Sergey Vojtovich [ 2017-10-11 ]

In fact there's some versioning:

    if (unlikely(version != plugin_array_version))
 
    ...
 
    plugin= plugins[idx];
 
    /* It will stop iterating on first engine error when "func" returns TRUE */
 
    if (plugin && func(thd, plugin_int_to_ref(plugin), arg))
 
        goto err;

...which doesn't make any sense. It is silly to assume that if version didn't change by the time of this check, it won't change by the time of func() call.

Comment by Sergey Vojtovich [ 2017-10-11 ]

This function is being used in quite a few places like handler, audit, acl, wsrep, I_S. All this code is subject for race conditions when ran concurrently with [UN]INSTALL PLUGIN.

A simple fix would be to temporarily increment ref_count. It guarantees that plugin won't disappear while we access it's structures, but doesn't protect from concurrent modifications, e.g. state or ref_count may change.

Another option is to hold LOCK_plugin while func() is being called. Most probably it is going to be quite complex to implement.

Comment by Sergey Vojtovich [ 2017-10-12 ]

There seem to be more issues than I described. Specifically my hypothesis doesn't seem to explain failure from bug description. I reassign this bug to Elkin for further analysis. What I described above is to be fixed in MDEV-10892, current patch: https://github.com/MariaDB/server/commit/96e982bb1a72db47a41b97bb632b595b347d2956

Comment by Alice Sherepa [ 2020-10-29 ]

10.2 http://buildbot.askmonty.org/buildbot/builders/kvm-rpm-centos74-amd64-debug/builds/5090/steps/mtr_centos_mix/logs/stdio

rpl.rpl_semi_sync_uninstall_plugin 'stmt' w4 [ fail ]
 
        Test ended at 2020-10-26 11:01:32
 
 
 
CURRENT_TEST: rpl.rpl_semi_sync_uninstall_plugin
 
mysqltest: In included file "./include/wait_condition.inc": 
included from /usr/share/mysql-test/suite/rpl/t/rpl_semi_sync_uninstall_plugin.test at line 120:
 
At line 43: query 'let $success= `$wait_condition`' failed with wrong errno 2013: 'Lost connection to MySQL server during query', instead of 0...
 
 
 
The result from queries just before the failure was:
 
< snip >
 
show status like "Rpl_semi_sync_slave_status";
 
Variable_name	Value
 
Rpl_semi_sync_slave_status	ON
 
connection master;
 
show status like "Rpl_semi_sync_master_status";
 
Variable_name	Value
 
Rpl_semi_sync_master_status	ON
 
show status like "Rpl_semi_sync_master_clients";
 
Variable_name	Value
 
Rpl_semi_sync_master_clients	1
 
select plugin_name,plugin_status from information_schema.plugins where plugin_name like 'rpl_%';
 
plugin_name	plugin_status
 
rpl_semi_sync_master	DELETED
 
connection slave;
 
include/stop_slave.inc
 
select plugin_name,plugin_status from information_schema.plugins where plugin_name like 'rpl_%';
 
plugin_name	plugin_status
 
connection master;
 
create table t2 (a int);
 
drop table t2;
 
 
 
More results from queries before failure can be found in /dev/shm/var/4/log/rpl_semi_sync_uninstall_plugin.log
 
 
 
 
 
Server [mysqld.1 - pid: 14785, winpid: 14785, exit: 256] failed during test run
 
Server log from this test:
 
----------SERVER LOG START-----------
 
2020-10-26 11:01:14 140450741406016 [Note] /usr/sbin/mysqld (mysqld 10.2.35-MariaDB-debug-log) starting as process 14805 ...
 
2020-10-26 11:01:14 140450741406016 [Warning] Could not increase number of max_open_files to more than 1024 (request: 32186)
 
2020-10-26 11:01:14 140450741406016 [Warning] Changed limits: max_open_files: 1024  max_connections: 151 (was 151)  table_cache: 421 (was 2000)
 
2020-10-26 11:01:14 140450741406016 [Note] Plugin 'SEQUENCE' is disabled.
 
2020-10-26 11:01:14 140450741406016 [Note] Plugin 'partition' is disabled.
 
2020-10-26 11:01:14 140450741406016 [Note] Plugin 'InnoDB' is disabled.
 
2020-10-26 11:01:14 140450741406016 [Note] Plugin 'INNODB_SYS_DATAFILES' is disabled.
 
2020-10-26 11:01:14 140450741406016 [Note] Plugin 'INNODB_SYS_TABLESTATS' is disabled.
 
2020-10-26 11:01:14 140450741406016 [Note] Plugin 'INNODB_LOCKS' is disabled.
 
2020-10-26 11:01:14 140450741406016 [Note] Plugin 'INNODB_MUTEXES' is disabled.
 
2020-10-26 11:01:14 140450741406016 [Note] Plugin 'INNODB_CMPMEM' is disabled.
 
2020-10-26 11:01:14 140450741406016 [Note] Plugin 'INNODB_CMP_PER_INDEX' is disabled.
 
2020-10-26 11:01:14 140450741406016 [Note] Plugin 'INNODB_CMP' is disabled.
 
2020-10-26 11:01:14 140450741406016 [Note] Plugin 'INNODB_FT_DELETED' is disabled.
 
2020-10-26 11:01:14 140450741406016 [Note] Plugin 'INNODB_CMP_RESET' is disabled.
 
2020-10-26 11:01:14 140450741406016 [Note] Plugin 'INNODB_LOCK_WAITS' is disabled.
 
2020-10-26 11:01:14 140450741406016 [Note] Plugin 'INNODB_TABLESPACES_ENCRYPTION' is disabled.
 
2020-10-26 11:01:14 140450741406016 [Note] Plugin 'INNODB_BUFFER_PAGE_LRU' is disabled.
 
2020-10-26 11:01:14 140450741406016 [Note] Plugin 'INNODB_SYS_FIELDS' is disabled.
 
2020-10-26 11:01:14 140450741406016 [Note] Plugin 'INNODB_CMPMEM_RESET' is disabled.
 
2020-10-26 11:01:14 140450741406016 [Note] Plugin 'FEEDBACK' is disabled.
 
2020-10-26 11:01:14 140450741406016 [Note] Plugin 'INNODB_SYS_COLUMNS' is disabled.
 
2020-10-26 11:01:14 140450741406016 [Note] Plugin 'INNODB_FT_INDEX_TABLE' is disabled.
 
2020-10-26 11:01:14 140450741406016 [Note] Plugin 'INNODB_CMP_PER_INDEX_RESET' is disabled.
 
2020-10-26 11:01:14 140450741406016 [Note] Plugin 'user_variables' is disabled.
 
2020-10-26 11:01:14 140450741406016 [Note] Plugin 'INNODB_FT_INDEX_CACHE' is disabled.
 
2020-10-26 11:01:14 140450741406016 [Note] Plugin 'INNODB_SYS_FOREIGN_COLS' is disabled.
 
2020-10-26 11:01:14 140450741406016 [Note] Plugin 'INNODB_FT_BEING_DELETED' is disabled.
 
2020-10-26 11:01:14 140450741406016 [Note] Plugin 'INNODB_BUFFER_POOL_STATS' is disabled.
 
2020-10-26 11:01:14 140450741406016 [Note] Plugin 'INNODB_TRX' is disabled.
 
2020-10-26 11:01:14 140450741406016 [Note] Plugin 'INNODB_SYS_FOREIGN' is disabled.
 
2020-10-26 11:01:14 140450741406016 [Note] Plugin 'INNODB_SYS_TABLES' is disabled.
 
2020-10-26 11:01:14 140450741406016 [Note] Plugin 'INNODB_FT_DEFAULT_STOPWORD' is disabled.
 
2020-10-26 11:01:14 140450741406016 [Note] Plugin 'INNODB_FT_CONFIG' is disabled.
 
2020-10-26 11:01:14 140450741406016 [Note] Plugin 'INNODB_BUFFER_PAGE' is disabled.
 
2020-10-26 11:01:14 140450741406016 [Note] Plugin 'INNODB_SYS_TABLESPACES' is disabled.
 
2020-10-26 11:01:14 140450741406016 [Note] Plugin 'INNODB_METRICS' is disabled.
 
2020-10-26 11:01:14 140450741406016 [Note] Plugin 'INNODB_SYS_INDEXES' is disabled.
 
2020-10-26 11:01:14 140450741406016 [Note] Plugin 'INNODB_SYS_VIRTUAL' is disabled.
 
2020-10-26 11:01:14 140450741406016 [Note] Plugin 'INNODB_TABLESPACES_SCRUBBING' is disabled.
 
2020-10-26 11:01:14 140450741406016 [Note] Plugin 'INNODB_SYS_SEMAPHORE_WAITS' is disabled.
 
2020-10-26 11:01:14 140450741406016 [Warning] /usr/sbin/mysqld: unknown option '--loose-pam-debug'
 
2020-10-26 11:01:14 140450741406016 [Note] Server socket created on IP: '127.0.0.1'.
 
2020-10-26 11:01:14 140450741406016 [Note] Reading of all Master_info entries succeeded
 
2020-10-26 11:01:14 140450741406016 [Note] Added new Master_info '' to hash table
 
2020-10-26 11:01:14 140450741406016 [Note] /usr/sbin/mysqld: ready for connections.
 
Version: '10.2.35-MariaDB-debug-log'  socket: '/dev/shm/var/tmp/4/mysqld.1.sock'  port: 16040  MariaDB Server
 
2020-10-26 11:01:15 140450740180736 [Note] Deleted Master_info file '/dev/shm/var/4/mysqld.1/data/master.info'.
 
2020-10-26 11:01:15 140450740180736 [Note] Deleted Master_info file '/dev/shm/var/4/mysqld.1/data/relay-log.info'.
 
2020-10-26 11:01:15 140450644608768 [Note] Start binlog_dump to slave_server(2), pos(master-bin.000001, 4), using_gtid(0), gtid('')
 
2020-10-26 11:01:15 140450644305664 [Note] unregister_replicator OK
 
2020-10-26 11:01:15 140450644305664 [Note] Semi-sync replication initialized for transactions.
 
2020-10-26 11:01:15 140450644305664 [Note] Semi-sync replication enabled on the master.
 
2020-10-26 11:01:15 140450644608768 [Note] Stop asynchronous binlog_dump to slave (server_id: 2)
 
2020-10-26 11:01:15 140450641532672 [Note] Start binlog_dump to slave_server(2), pos(master-bin.000001, 1464), using_gtid(0), gtid('')
 
2020-10-26 11:01:15 140450641532672 [Note] Start semi-sync binlog_dump to slave (server_id: 2), pos(./master-bin.000001, 1464)
 
2020-10-26 11:01:16 140450641532672 [ERROR] Read semi-sync reply network error:  (errno: 1158)
 
2020-10-26 11:01:16 140450641532672 [Note] Stop semi-sync binlog_dump to slave (server_id: 2)
 
2020-10-26 11:01:16 140450641532672 [Warning] Aborted connection 11 to db: 'unconnected' user: 'root' host: 'localhost' (Unknown error)
 
2020-10-26 11:01:26 140450644305664 [Warning] Timeout waiting for reply of binlog (file: master-bin.000001, pos: 2084), semi-sync up to file master-bin.000001, position 1952.
 
2020-10-26 11:01:26 140450644305664 [Note] Semi-sync replication switched OFF.
 
2020-10-26 11:01:26 140450641532672 [Note] unregister_replicator OK
 
201026 11:01:26 [ERROR] mysqld got signal 11 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.2.35-MariaDB-debug-log
 
key_buffer_size=1048576
 
read_buffer_size=131072
 
max_used_connections=7
 
max_threads=153
 
thread_count=9
 
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 63104 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x556026392070
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x7fbd36c31c88 thread_stack 0x49000
 
/usr/sbin/mysqld(my_print_stacktrace+0x2b)[0x55601f0e55f9]
 
/usr/sbin/mysqld(handle_fatal_signal+0x30c)[0x55601ea8ab93]
 
addr2line: Dwarf Error: Offset (3816030) greater than or equal to .debug_str size (133351).
 
addr2line: Dwarf Error: Offset (66453762) greater than or equal to .debug_str size (133351).
 
addr2line: Dwarf Error: Offset (6516329) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (6514020) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (1178796290) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (5390925) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (1273784) greater than or equal to .debug_str size (133351).
 
addr2line: Dwarf Error: Offset (1023536813) greater than or equal to .debug_str size (133351).
 
addr2line: Dwarf Error: Offset (1270421) greater than or equal to .debug_str size (133351).
 
addr2line: Dwarf Error: Offset (1158551555) greater than or equal to .debug_str size (133351).
 
addr2line: Dwarf Error: Offset (6579297) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (3238134144) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (453074177) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (1111240193) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (16777263) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (29426177) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (870871) greater than or equal to .debug_str size (133351).
 
addr2line: Dwarf Error: Offset (869989) greater than or equal to .debug_str size (133351).
 
addr2line: Dwarf Error: Offset (3096661) greater than or equal to .debug_str size (133351).
 
addr2line: Dwarf Error: Offset (946336404) greater than or equal to .debug_str size (133351).
 
addr2line: Dwarf Error: Offset (2409103810) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (7173491) greater than or equal to .debug_str size (3937801).
 
sigaction.c:0(__restore_rt)[0x7fbd3c0e95e0]
 
/usr/sbin/mysqld(+0xd63df6)[0x55601f116df6]
 
/usr/sbin/mysqld(+0xd63eb5)[0x55601f116eb5]
 
/usr/sbin/mysqld(my_copy_fix_mb+0x4d)[0x55601f106b55]
 
/usr/sbin/mysqld(_ZN13String_copier16well_formed_copyEPK15charset_info_stPcjS2_PKcjj+0x75)[0x55601e940471]
 
/usr/sbin/mysqld(_ZN15Field_varstring5storeEPKcjPK15charset_info_st+0x111)[0x55601ea7f9f5]
 
addr2line: Dwarf Error: Offset (3816030) greater than or equal to .debug_str size (133351).
 
addr2line: Dwarf Error: Offset (66453762) greater than or equal to .debug_str size (133351).
 
addr2line: Dwarf Error: Offset (6516329) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (6514020) greater than or equal to .debug_str size (3937801).
 
/usr/sbin/mysqld(+0x5747db)[0x55601e9277db]
 
/usr/sbin/mysqld(_Z24plugin_foreach_with_maskP3THDPFcS0_PP13st_plugin_intPvEijS4_+0x1ef)[0x55601e8ca917]
 
/usr/sbin/mysqld(_Z12fill_pluginsP3THDP10TABLE_LISTP4Item+0x4e)[0x55601e91ea6c]
 
addr2line: Dwarf Error: Offset (1178796290) greater than or equal to .debug_str size (3937801).
 
/usr/sbin/mysqld(_Z24get_schema_tables_resultP4JOIN23enum_schema_table_state+0x2c5)[0x55601e934e53]
 
/usr/sbin/mysqld(_ZN4JOIN10exec_innerEv+0x71f)[0x55601e91a83f]
 
addr2line: Dwarf Error: Offset (5390925) greater than or equal to .debug_str size (3937801).
 
/usr/sbin/mysqld(_ZN4JOIN4execEv+0x8f)[0x55601e91b11f]
 
/usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x84b)[0x55601e9198ea]
 
/usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x139)[0x55601e919b78]
 
/usr/sbin/mysqld(+0x4fbe56)[0x55601e8aee56]
 
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x164c)[0x55601e8b7dd3]
 
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x1bc)[0x55601e8c072f]
 
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0xee6)[0x55601e8c26a2]
 
addr2line: Dwarf Error: Offset (1273784) greater than or equal to .debug_str size (133351).
 
addr2line: Dwarf Error: Offset (1023536813) greater than or equal to .debug_str size (133351).
 
addr2line: Dwarf Error: Offset (1270421) greater than or equal to .debug_str size (133351).
 
addr2line: Dwarf Error: Offset (1158551555) greater than or equal to .debug_str size (133351).
 
/usr/sbin/mysqld(_Z10do_commandP3THD+0x568)[0x55601e8c4928]
 
addr2line: Dwarf Error: Offset (6579297) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (3238134144) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (453074177) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (1111240193) greater than or equal to .debug_str size (3937801).
 
/usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x192)[0x55601e9ac5ac]
 
addr2line: Dwarf Error: Offset (16777263) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (29426177) greater than or equal to .debug_str size (3937801).
 
/usr/sbin/mysqld(handle_one_connection+0x3d)[0x55601e9ac6e5]
 
addr2line: Dwarf Error: Offset (870871) greater than or equal to .debug_str size (133351).
 
addr2line: Dwarf Error: Offset (869989) greater than or equal to .debug_str size (133351).
 
addr2line: Dwarf Error: Offset (3096661) greater than or equal to .debug_str size (133351).
 
addr2line: Dwarf Error: Offset (946336404) greater than or equal to .debug_str size (133351).
 
addr2line: Dwarf Error: Offset (2409103810) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (7173491) greater than or equal to .debug_str size (3937801).
 
/usr/sbin/mysqld(+0xce9ed6)[0x55601f09ced6]
 
addr2line: Dwarf Error: Offset (40453238) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (39666808) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (1162084995) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (40453238) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (40453238) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (39666808) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (39666808) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (1162084995) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (1162084995) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (7500912) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (1296577539) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (1947296921) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (1296577539) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (1869886857) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (3942842858) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (4211278330) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (4211278330) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (3942842858) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (4211278330) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (4211278330) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (3942842858) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (4211278330) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (4211278330) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (3942842858) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (4211278330) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (4211278330) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (3942842858) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (4211278330) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (4211278330) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (3942842858) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (4211278330) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (4211278330) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (3942842858) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (4211278330) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (4211278330) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (3942842858) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (4211278330) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (4211278330) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (91161724) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (91161724) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (91161724) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (3942842858) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (3942842858) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (4211278330) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (4211278330) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (4211278330) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (4211278330) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (1869886857) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (3266249541) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (5390925) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (3942842858) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (4211278330) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (4211278330) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (1869886857) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (4211278330) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (4211278330) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (3942842858) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (3942842858) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (4211278330) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (4211278330) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (3942842858) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (4211278330) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (4211278330) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (3266249541) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (30933830) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (3266249541) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (30933830) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (3266249541) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (30933830) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (4211278330) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (4211278330) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (3942842858) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Could not find abbrev number 7659.
 
addr2line: Dwarf Error: Offset (1076627413) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (168624504) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Could not find abbrev number 7659.
 
addr2line: Dwarf Error: Could not find abbrev number 7659.
 
addr2line: Dwarf Error: Could not find abbrev number 7659.
 
addr2line: Dwarf Error: Offset (1076627413) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (168624504) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (1076627413) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (168624504) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Could not find abbrev number 7659.
 
addr2line: Dwarf Error: Offset (42336295) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (3942842858) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (4211278330) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (4211278330) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (1073753946) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (3942842858) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (4211278330) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (4211278330) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (3741516189) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (3741516189) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (3741516189) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (3741516189) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (3741516189) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (757183238) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (757183238) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (757183238) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (757183238) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (757183238) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (3741516189) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (757183238) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (7628147) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (3355640826) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (3355640826) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (3355640826) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (3355640826) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (3355640826) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (3942842858) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (4211278330) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (4211278330) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (27362604) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (50331648) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (27362604) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (27362604) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (27362604) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (7628147) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (27362604) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (2542797623) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (28542252) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (27362604) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (757183234) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (757183238) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (757183234) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (757183238) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (757183234) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (757183238) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (757183238) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (757183238) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (757183238) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (50331649) greater than or equal to .debug_str size (3937801).
 
addr2line: Dwarf Error: Offset (870871) greater than or equal to .debug_str size (133351).
 
addr2line: Dwarf Error: Offset (870871) greater than or equal to .debug_str size (133351).
 
addr2line: Dwarf Error: Offset (870871) greater than or equal to .debug_str size (133351).
 
addr2line: Dwarf Error: Offset (870871) greater than or equal to .debug_str size (133351).
 
addr2line: Dwarf Error: Offset (870871) greater than or equal to .debug_str size (133351).
 
addr2line: Dwarf Error: Offset (870871) greater than or equal to .debug_str size (133351).
 
addr2line: Dwarf Error: Offset (870871) greater than or equal to .debug_str size (133351).
 
addr2line: Dwarf Error: Offset (870871) greater than or equal to .debug_str size (133351).
 
addr2line: Dwarf Error: Offset (870871) greater than or equal to .debug_str size (133351).
 
addr2line: Dwarf Error: Offset (870871) greater than or equal to .debug_str size (133351).
 
addr2line: Dwarf Error: Offset (870871) greater than or equal to .debug_str size (133351).
 
addr2line: Dwarf Error: Offset (870871) greater than or equal to .debug_str size (133351).
 
pthread_create.c:0(start_thread)[0x7fbd3c0e1e25]
 
/lib64/libc.so.6(clone+0x6d)[0x7fbd3a6b834d]
 
 
 
Trying to get some variables.
 
Some pointers may be invalid and cause the dump to abort.
 
Query (0x5560263a2088): select count(*) = 0 from information_schema.plugins where plugin_name like 'rpl_%'
 
 
 
Connection ID (thread ID): 9
 
Status: NOT_KILLED
 
 
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on
 
 
 
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
 
information that should help you find out what is causing the crash.
 
Writing a core file...
 
Working directory at /dev/shm/var/4/mysqld.1/data
 
Resource Limits:
 
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        unlimited            unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             4096                 23142                processes 
Max open files            1024                 1024                 files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       23142                23142                signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us        
Core pattern: core
 
 
 
----------SERVER LOG END-------------
 
 
 
 
 
 - found 'core.14805' (0/0)
 
 
 
Trying 'dbx' to get a backtrace
 
gdb not found, cannot get the stack trace
 
 - deleting it, already saved 0
 
 - saving '/dev/shm/var/4/log/rpl.rpl_semi_sync_uninstall_plugin-stmt/' to '/dev/shm/var/log/rpl.rpl_semi_sync_uninstall_plugin-stmt/'

Comment by Alice Sherepa [ 2021-03-04 ]

http://buildbot.askmonty.org/buildbot/builders/kvm-fulltest2/builds/27228

Comment by Alice Sherepa [ 2021-04-27 ]

from http://buildbot.askmonty.org/buildbot/builders/kvm-rpm-centos74-amd64-debug/builds/6256/steps/mtr/logs/stdio 10.2 a35cde8cd8364edc0a2375

rpl.rpl_semi_sync_uninstall_plugin 'stmt' w2 [ fail ]
 
        Test ended at 2021-04-27 00:01:26
 
 
 
CURRENT_TEST: rpl.rpl_semi_sync_uninstall_plugin
 
mysqltest: In included file "./include/wait_condition.inc": 
included from /usr/share/mysql-test/suite/rpl/t/rpl_semi_sync_uninstall_plugin.test at line 120:
 
At line 43: query 'let $success= `$wait_condition`' failed with wrong errno 2013: 'Lost connection to MySQL server during query', instead of 0...
 
 
 
210427  0:01:19 [ERROR] mysqld got signal 11 ;
 
 
 
Server version: 10.2.38-MariaDB-debug-log
 
 
 
stack_bottom = 0x7f2fc085fc88 thread_stack 0x49000
 
mysys/stacktrace.c:172(my_print_stacktrace)[0x5649dc857469]
 
sql/signal_handler.cc:221(handle_fatal_signal)[0x5649dc1f9c61]
 
sigaction.c:0(__restore_rt)[0x7f2fc5d175e0]
 
strings/ctype-utf8.c:5394(my_charlen_utf8)[0x5649dc095c07]
 
strings/ctype-mb.ic:187(my_well_formed_char_length_utf8)[0x5649dc038a3d]
 
strings/ctype-mb.c:406(my_copy_fix_mb)[0x5649dc08ce5c]
 
sql/sql_string.cc:1086(String_copier::well_formed_copy(charset_info_st const*, char*, unsigned int, charset_info_st const*, char const*, unsigned int, unsigned int))[0x5649dc0a32ea]
 
sql/field.cc:7708(Field_varstring::store(char const*, unsigned int, charset_info_st const*))[0x5649dc088c2e]
 
sql/sql_show.cc:179(schema_table_store_record(THD*, TABLE*))[0x5649dc08950d]
 
sql/sql_plugin.cc:2411(plugin_foreach_with_mask(THD*, char (*)(THD*, st_plugin_int**, void*), int, unsigned int, void*))[0x5649dc087cd3]
 
sql/sql_show.cc:286(fill_plugins(THD*, TABLE_LIST*, Item*))[0x5649dc087f61]
 
sql/sql_show.cc:8472(get_schema_tables_result(JOIN*, enum_schema_table_state))[0x5649dc01cc36]
 
sql/sql_select.cc:3614(JOIN::exec_inner())[0x5649dc025f68]
 
sql/sql_select.cc:3447(JOIN::exec())[0x5649dc02e819]
 
sql/sql_select.cc:3851(mysql_select(THD*, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*))[0x5649dc030776]
 
sql/sql_select.cc:373(handle_select(THD*, LEX*, select_result*, unsigned long))[0x5649dc032a38]
 
sql/sql_parse.cc:6274(st_select_lex::add_table_to_list(THD*, Table_ident*, st_mysql_lex_string*, unsigned long, thr_lock_type, enum_mdl_type, List<Index_hint>*, List<String>*, st_mysql_lex_string*))[0x5649dc11af01]
 
sql/sql_parse.cc:3585(mysql_execute_command(THD*))[0x5649dc11b03a]
 
sql/sql_parse.cc:7797(mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool))[0x5649dc80ed56]
 
pthread_create.c:0(start_thread)[0x7f2fc5d0fe25]
 
/lib64/libc.so.6(clone+0x6d)[0x7f2fc42e634d]
 
 
 
Trying to get some variables.
 
Some pointers may be invalid and cause the dump to abort.
 
Query (0x5649e408e088): select count(*) = 0 from information_schema.plugins where plugin_name like 'rpl_%'
 
 
 
Connection ID (thread ID): 9
 
Status: NOT_KILLED

Comment by Alice Sherepa [ 2021-05-28 ]

http://buildbot.askmonty.org/buildbot/builders/kvm-rpm-centos74-amd64-debug/builds/6524/steps/mtr/logs/stdio

Generated at Thu Feb 08 07:17:19 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.