[MDEV-23576] galera_3nodes.galera_ipv6_mysqldump MTR failed: WSREP_SST: [ERROR] rsync daemon port '16008' has been taken Created: 2020-08-25  Updated: 2021-10-04  Resolved: 2020-10-27

Status: Closed
Project: MariaDB Server
Component/s: Galera, Tests
Affects Version/s: 10.2.34, 10.3.25, 10.4.15, 10.5.6
Fix Version/s: 10.2.35, 10.3.26, 10.4.16, 10.5.7, 10.6.0

Type: Bug Priority: Major
Reporter: Stepan Patryshev (Inactive) Assignee: Julius Goryavsky
Resolution: Fixed Votes: 0
Labels: None
Environment:

kvm-zyp-sles123-amd64


Issue Links:
Duplicate
duplicates MDEV-21770 `galera_3nodes.galera_ipv6_mariabacku... Closed
Issue split
split to MDEV-24036 galera_3nodes.galera_ipv6_mysqldump: ... Closed

 Description   

galera_3nodes.galera_ipv6_mysqldump failed on bb 10.2 and bb 10.3: "WSREP_SST: [ERROR] rsync daemon port '16008' has been taken".

stdio.log:

10.2.34 0be70a1b773ce66ef1803fcce19522fd9c60c07d

galera_3nodes.galera_ipv6_mysqldump 'innodb' w1 [ fail ]
        Test ended at 2020-08-25 13:32:03
 
CURRENT_TEST: galera_3nodes.galera_ipv6_mariabackup
CURRENT_TEST: galera_3nodes.galera_ipv6_mariabackup_section
CURRENT_TEST: galera_3nodes.galera_ipv6_mariabackup
CURRENT_TEST: galera_3nodes.galera_ipv6_mysqldump
 
 
Failed to start mysqld.2
 
 
 - skipping '/dev/shm/var/1/log/galera_3nodes.galera_ipv6_mysqldump-innodb/'
 
Retrying test galera_3nodes.galera_ipv6_mysqldump, attempt(2/3)...
 
***Warnings generated in error logs during shutdown after running tests: galera_3nodes.galera_ipv6_mysqldump
 
WSREP_SST: [ERROR] rsync daemon port '16008' has been taken (20200825 13:31:53.933)
2020-08-25 13:31:53 140068462999296 [ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_rsync --role 'joiner' --address '[::1]:16008' --datadir '/dev/shm/var/1/mysqld.2/data/' --defaults-file '/dev/shm/var/1/my.cnf' --defaults-group-suffix '.2' --parent '27002' --binlog-index 'mysqld-bin.index' --mysqld-args --defaults-group-suffix=.2 --defaults-file=/dev/shm/var/1/my.cnf --log-output=file --innodb --innodb-cmpmem --innodb-cmp-per-index --innodb-trx --innodb-locks --innodb-metrics --innodb-buffer-pool-stats --innodb-buffer-page --innodb-buffer-page-lru --innodb-sys-columns --innodb-sys-fields --innodb-sys-foreign --innodb-sys-foreign-cols --innodb-sys-indexes --innodb-sys-tables --innodb-sys-virtual --bind-address=:: --core-file --loose-debug-sync-timeout=300
2020-08-25 13:31:53 140068462999296 [ERROR] WSREP: Process completed with error: wsrep_sst_rsync --role 'joiner' --address '[::1]:16008' --datadir '/dev/shm/var/1/mysqld.2/data/' --defaults-file '/dev/shm/var/1/my.cnf' --defaults-group-suffix '.2' --parent '27002' --binlog-index 'mysqld-bin.index' --mysqld-args --defaults-group-suffix=.2 --defaults-file=/dev/shm/var/1/my.cnf --log-output=file --innodb --innodb-cmpmem --innodb-cmp-per-index --innodb-trx --innodb-locks --innodb-metrics --innodb-buffer-pool-stats --innodb-buffer-page --innodb-buffer-page-lru --innodb-sys-columns --innodb-sys-fields --innodb-sys-foreign --innodb-sys-foreign-cols --innodb-sys-indexes --innodb-sys-tables --innodb-sys-virtual --bind-address=:: --core-file --loose-debug-sync-timeout=300: 16 (Device or resource busy)
2020-08-25 13:31:53 140068463302400 [ERROR] WSREP: Failed to prepare for 'rsync' SST. Unrecoverable.
2020-08-25 13:31:53 140068463302400 [ERROR] Aborting
Warning: Memory not freed: 1427624

On bb 10.4 and bb 10.5 it failed with another output: "mysql_shutdown failed".

stdio.log:

10.4.15 88e70f4caea34e0d7677b1fa646151b8b87dd3ab

galera_3nodes.galera_ipv6_mysqldump 'innodb' w1 [ fail ]
        Test ended at 2020-08-25 06:58:54
 
CURRENT_TEST: galera_3nodes.galera_ipv6_mysqldump
mysqltest: In included file "./include/shutdown_mysqld.inc": 
included from ./include/restart_mysqld.inc at line 10:
included from /usr/share/mysql-test/suite/galera_3nodes/t/galera_ipv6_mysqldump.test at line 83:
At line 40: mysql_shutdown failed
 
The result from queries just before the failure was:
< snip >
connection node_1;
CREATE USER 'sst';
GRANT ALL PRIVILEGES ON *.* TO 'sst';
SET GLOBAL wsrep_sst_auth = 'sst:';
connection node_2;
SET GLOBAL wsrep_sst_method = 'mysqldump';
Shutting down server ...
connection node_1;
Cleaning var directory ...
CREATE TABLE t1 (f1 INTEGER) ENGINE=InnoDB;
INSERT INTO t1 VALUES (1);
connection node_2;
Starting server ...
SELECT COUNT(*) = 1 FROM t1;
COUNT(*) = 1
1
DROP TABLE t1;
SELECT VARIABLE_VALUE LIKE '%[::1]%' FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_incoming_addresses';
VARIABLE_VALUE LIKE '%[::1]%'



 Comments   
Comment by Julius Goryavsky [ 2020-10-21 ]

This is a duplicate of MDEV-21770, corrected here: https://github.com/MariaDB/server/commit/ac3c19de6967abbaaa661f72526b233d11bd6868

Comment by Marko Mäkelä [ 2020-10-22 ]

Because the test keeps failing even after the supposed MDEV-21770 fix, I disabled the test on 10.4.

Comment by Julius Goryavsky [ 2020-10-22 ]

marko Thanks, this is something new, not related to MDEV-21770 and original MDEV-23576 - in this case it crashes during server shutdown. I'll figure out why this is happening - this is some kind of new problem, different from the one in MDEV-21770 and from the original MDEV-23576:

galera_3nodes.galera_ipv6_mysqldump 'innodb' w2 [ fail ]
        Test ended at 2020-10-22 06:12:07
 
CURRENT_TEST: galera_3nodes.galera_ipv6_mysqldump
mysqltest: In included file "./include/shutdown_mysqld.inc": 
included from /usr/share/mysql/mysql-test/suite/galera_3nodes/t/galera_ipv6_mysqldump.test at line 44:
At line 40: mysql_shutdown failed

and (node-2 log):

2020-10-22  6:12:06 2 [Note] WSREP: Non-primary view
2020-10-22  6:12:06 2 [Note] WSREP: Server status change disconnecting -> disconnected
2020-10-22  6:12:06 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2020-10-22  6:12:06 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2020-10-22  6:12:06 2 [Note] WSREP: Applier thread exiting ret: 0 thd: 2
2020-10-22  6:12:06 2 [Warning] Aborted connection 2 to db: 'unconnected' user: 'unauthenticated' host: '' (This connection closed normally without authentication)
2020-10-22  6:12:06 0 [Note] WSREP: killing local connection: 15

Comment by Julius Goryavsky [ 2020-10-27 ]

Closed as duplicate of the MDEV-21770, but created a separate task for the detected problem during shutdown: https://jira.mariadb.org/browse/MDEV-24036

Generated at Thu Feb 08 09:23:27 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.