[MDEV-24036] galera_3nodes.galera_ipv6_mysqldump: rare random crashes during shutdown Created: 2020-10-27  Updated: 2021-09-29  Resolved: 2021-05-05

Status: Closed
Project: MariaDB Server
Component/s: Galera, Galera SST
Affects Version/s: 10.2.37, 10.3.28, 10.4.17, 10.5.9, 10.6.0
Fix Version/s: 10.2.38, 10.3.29, 10.4.19, 10.5.10, 10.6.1

Type: Bug Priority: Critical
Reporter: Julius Goryavsky Assignee: Julius Goryavsky
Resolution: Fixed Votes: 0
Labels: None

Attachments: Zip Archive MDEV-24036_102_logs_201215.zip     File mysqld.1.err.2     File mysqld.2.err.2     File mysqld.3.err.2    
Issue Links:
Issue split
split from MDEV-23576 galera_3nodes.galera_ipv6_mysqldump M... Closed

 Description   

Sometimes (in rare cases) there is a failure on node number 2 during shutdown, which does not replay after restart of this test:

galera_3nodes.galera_ipv6_mysqldump 'innodb' w2 [ fail ]
 
        Test ended at 2020-10-22 06:17:57
 
 
 
CURRENT_TEST: galera_3nodes.galera_ipv6_mysqldump
 
mysqltest: In included file "./include/shutdown_mysqld.inc": 
included from ./include/restart_mysqld.inc at line 10:
 
included from /usr/share/mysql/mysql-test/suite/galera_3nodes/t/galera_ipv6_mysqldump.test at line 84:
 
At line 40: mysql_shutdown failed
 
 
 
The result from queries just before the failure was:
 
< snip >
 
connection node_1;
 
CREATE USER 'sst';
 
GRANT ALL PRIVILEGES ON *.* TO 'sst';
 
SET GLOBAL wsrep_sst_auth = 'sst:';
 
connection node_2;
 
SET GLOBAL wsrep_sst_method = 'mysqldump';
 
Shutting down server ...
 
connection node_1;
 
Cleaning var directory ...
 
CREATE TABLE t1 (f1 INTEGER NOT NULL PRIMARY KEY) ENGINE=InnoDB;
 
INSERT INTO t1 VALUES (1);
 
connection node_2;
 
Starting server ...
 
SELECT COUNT(*) AS EXPECT_1 FROM t1;
 
EXPECT_1
 
1
 
DROP TABLE t1;
 
SELECT VARIABLE_VALUE LIKE '%[::1]%' FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_incoming_addresses';
 
VARIABLE_VALUE LIKE '%[::1]%'
 
1
 
 
 
More results from queries before failure can be found in /dev/shm/var/2/log/galera_ipv6_mysqldump.log
 
 
 
 - saving '/dev/shm/var/2/log/galera_3nodes.galera_ipv6_mysqldump-innodb/' to '/dev/shm/var/log/galera_3nodes.galera_ipv6_mysqldump-innodb/'
 
 
 
Retrying test galera_3nodes.galera_ipv6_mysqldump, attempt(2/3)...
 
 
 
worker[2] > Restart  - not started
 
worker[2] > Restart  - not started
 
worker[2] > Restart  - not started
 
galera_3nodes.galera_ipv6_mariabackup_section 'innodb' w1 [ pass ]  13629

snippet of the log for node 2:

2020-10-22  6:17:56 2 [Note] WSREP: Non-primary view
 
2020-10-22  6:17:56 2 [Note] WSREP: Server status change disconnecting -> disconnected
 
2020-10-22  6:17:56 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
2020-10-22  6:17:56 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
2020-10-22  6:17:56 2 [Note] WSREP: Applier thread exiting ret: 0 thd: 2
 
2020-10-22  6:17:56 2 [Warning] Aborted connection 2 to db: 'unconnected' user: 'unauthenticated' host: '' (This connection closed normally without authentication)
 
2020-10-22  6:17:56 0 [Note] WSREP: killing local connection: 13



 Comments   
Comment by Stepan Patryshev (Inactive) [ 2020-12-17 ]

galera_3nodes.galera_ipv6_mysqldump failed also on BB, 10.2 CS, but without proper output.
stdio.log:

10.2.37, 74223c33d1aeffac28b155c59646dc3df58e33fb, kvm-deb-xenial-x86

 
galera_3nodes.galera_ipv6_mysqldump 'innodb' w1 [ fail ]
 
        Test ended at 2020-12-15 06:06:43
 
 
 
CURRENT_TEST: galera_3nodes.galera_ipv6_mysqldump
 
 
 
 
 
Failed to start mysqld.2
 
mysqltest failed but provided no output
 
 
 
 
 
 - saving '/dev/shm/var/1/log/galera_3nodes.galera_ipv6_mysqldump-innodb/' to '/dev/shm/var/log/galera_3nodes.galera_ipv6_mysqldump-innodb/'
 
 
 
Retrying test galera_3nodes.galera_ipv6_mysqldump, attempt(2/3)...
 
 
 
worker[1] > Restart  - not started
 
worker[1] > Restart  - not started
 
worker[1] > Restart  - not started
 
***Warnings generated in error logs during shutdown after running tests: galera_3nodes.galera_ipv6_mysqldump
 
 
 
2020-12-15  6:06:42 3069016000 [ERROR] Aborting

Server logs.

Comment by Stepan Patryshev (Inactive) [ 2020-12-17 ]

It affects also 10.5 CS on BB.

stdio.log:

10.5.9, 6bb3949eb37bbe674a3550b502b3f60474cc0d46, kvm-deb-groovy-aarch64

 
galera_3nodes.galera_ipv6_mysqldump 'innodb' w2 [ fail ]
 
        Test ended at 2020-12-17 03:48:13
 
 
 
CURRENT_TEST: galera_3nodes.galera_ipv6_mysqldump
 
mysqltest: In included file "./include/shutdown_mysqld.inc": 
included from /usr/share/mysql/mysql-test/suite/galera_3nodes/t/galera_ipv6_mysqldump.test at line 44:
 
At line 50: mysql_shutdown failed
 
 
 
The result from queries just before the failure was:
 
connection node_2;
 
connection node_1;
 
call mtr.add_suppression("WSREP: wsrep_sst_method is set to 'mysqldump' yet mysqld bind_address is set to'");
 
call mtr.add_suppression("Failed to load slave replication state from table mysql.gtid_slave_pos");
 
connection node_1;
 
connection node_2;
 
connection node_3;
 
connection node_1;
 
CREATE USER 'sst';
 
GRANT ALL PRIVILEGES ON *.* TO 'sst';
 
SET GLOBAL wsrep_sst_auth = 'sst:';
 
connection node_2;
 
SET GLOBAL wsrep_sst_method = 'mysqldump';
 
Shutting down server ...
 
 
 
 - skipping '/dev/shm/var/2/log/galera_3nodes.galera_ipv6_mysqldump-innodb/'
 
 
 
Retrying test galera_3nodes.galera_ipv6_mysqldump, attempt(2/3)...
 
 
 
worker[2] > Restart  - not started
 
worker[2] > Restart  - not started
 
worker[2] > Restart  - not started
 
galera_3nodes.galera_ipv6_mariabackup 'innodb' w1 [ retry-pass ]  13699
 
 
 
Retrying test galera_3nodes.galera_ipv6_mariabackup, attempt(3/3)...
 
 
 
worker[1] > Restart  - not started
 
worker[1] > Restart  - not started
 
worker[1] > Restart  - not started
 
galera_3nodes.galera_ipv6_mysqldump 'innodb' w2 [ retry-pass ]  15911
 
 
 
Retrying test galera_3nodes.galera_ipv6_mysqldump, attempt(3/3)...
 
 
 
worker[2] > Restart  - not started
 
worker[2] > Restart  - not started
 
worker[2] > Restart  - not started
 
galera_3nodes.galera_ipv6_mariabackup 'innodb' w1 [ retry-pass ]  13436
 
worker[1] > Restart  - using different config file
 
worker[1] > Restart  - using different config file
 
worker[1] > Restart  - using different config file
 
galera_3nodes.galera_ipv6_mysqldump 'innodb' w2 [ retry-pass ]  16805
 
worker[2] > Restart  - using different config file
 
worker[2] > Restart  - using different config file
 
worker[2] > Restart  - using different config file

Comment by Stepan Patryshev (Inactive) [ 2020-12-17 ]

It affects also 10.6.0 CS on BB.

Comment by Julius Goryavsky [ 2021-05-05 ]

All SST modifications for mysqldump from this task have been moved to MDEV-24962

Generated at Thu Feb 08 09:26:57 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.