[MDEV-24890] galera_3nodes.galera_ipv6_rsync MTR failed: rsync daemon port '16031' has been taken Created: 2021-02-16  Updated: 2021-09-29  Resolved: 2021-05-22

Status: Closed
Project: MariaDB Server
Component/s: Galera, Tests
Affects Version/s: 10.3.28, 10.6.0
Fix Version/s: 10.5.10, 10.6.1, 10.2.39, 10.3.30, 10.4.20

Type: Bug Priority: Major
Reporter: Stepan Patryshev (Inactive) Assignee: Julius Goryavsky
Resolution: Fixed Votes: 0
Labels: None
Environment:

kvm-deb-bionic-amd64


Attachments: Zip Archive 210112_galera_ipv6_rsync_103_CS_logs.zip     HTML File 210112_stdio_103_CS.html    
Issue Links:
Duplicate
duplicates MDEV-23580 galera_3nodes.galera_ipv6_rsync_secti... Closed

 Description   

galera_3nodes.galera_ipv6_rsync failed on BB 10.3 CS and BB 10.6 CS: "rsync daemon port '16031' has been taken".
It seems to be a sporadic issue.

stdio.log:

10.3.28, 1be707286ef13041a67e6e6ca30125470245a898, kvm-deb-bionic-amd64

galera_3nodes.galera_ipv6_rsync 'innodb' w2 [ fail ]
        Test ended at 2021-01-12 03:59:00
 
CURRENT_TEST: galera_3nodes.galera_ipv6_rsync
 
 
Failed to start mysqld.3
mysqltest failed but provided no output
 
 
 - skipping '/dev/shm/var/2/log/galera_3nodes.galera_ipv6_rsync-innodb/'
 
Retrying test galera_3nodes.galera_ipv6_rsync, attempt(2/3)...
 
worker[2] > Restart  - not started
worker[2] > Restart  - not started
worker[2] > Restart  - not started
***Warnings generated in error logs during shutdown after running tests: galera_3nodes.galera_ipv6_rsync
 
WSREP_SST: [ERROR] rsync daemon port '16031' has been taken (20210112 03:58:46.128)
2021-01-12  3:58:46 0 [ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_rsync --role 'joiner' --address '[::1]:16031' --datadir '/dev/shm/var/2/mysqld.3/data/' --defaults-file '/dev/shm/var/2/my.cnf' --defaults-group-suffix '.3' --parent '30628' --binlog-index 'mysqld-bin.index' --mysqld-args --defaults-group-suffix=.3 --defaults-file=/dev/shm/var/2/my.cnf --log-output=file --innodb --innodb-cmpmem --innodb-cmp-per-index --innodb-trx --innodb-locks --innodb-lock-waits --innodb-metrics --innodb-buffer-pool-stats --innodb-buffer-page --innodb-buffer-page-lru --innodb-sys-columns --innodb-sys-fields --innodb-sys-foreign --innodb-sys-foreign-cols --innodb-sys-indexes --innodb-sys-tables --innodb-sys-virtual --core-file --loose-debug-sync-timeout=300
2021-01-12  3:58:46 0 [ERROR] WSREP: Process completed with error: wsrep_sst_rsync --role 'joiner' --address '[::1]:16031' --datadir '/dev/shm/var/2/mysqld.3/data/' --defaults-file '/dev/shm/var/2/my.cnf' --defaults-group-suffix '.3' --parent '30628' --binlog-index 'mysqld-bin.index' --mysqld-args --defaults-group-suffix=.3 --defaults-file=/dev/shm/var/2/my.cnf --log-output=file --innodb --innodb-cmpmem --innodb-cmp-per-index --innodb-trx --innodb-locks --innodb-lock-waits --innodb-metrics --innodb-buffer-pool-stats --innodb-buffer-page --innodb-buffer-page-lru --innodb-sys-columns --innodb-sys-fields --innodb-sys-foreign --innodb-sys-foreign-cols --innodb-sys-indexes --innodb-sys-tables --innodb-sys-virtual --core-file --loose-debug-sync-timeout=300: 16 (Device or resource busy)
2021-01-12  3:58:46 2 [ERROR] WSREP: Failed to prepare for 'rsync' SST. Unrecoverable.
2021-01-12  3:58:46 2 [ERROR] Aborting
Warning: Memory not freed: 1381120
 
galera_3nodes.galera_ipv6_rsync_section 'innodb' w1 [ pass ]  14117
worker[1] > Restart  - using different config file
worker[1] > Restart  - using different config file
worker[1] > Restart  - using different config file
worker[2] mysql-test-run: WARNING: Process [mysqld.2 - pid: 31409, winpid: 31409, exit: 256] died after mysql-test-run waited 31.4 seconds for /dev/shm/var/2/run/mysqld.2.pid to be created.
galera_3nodes.galera_ipv6_rsync 'innodb' w2 [ retry-fail ]
        Test ended at 2021-01-12 03:59:32
 
CURRENT_TEST: galera_3nodes.galera_ipv6_rsync
 
 
Failed to start mysqld.2
mysqltest failed but provided no output
 
 
 - skipping '/dev/shm/var/2/log/galera_3nodes.galera_ipv6_rsync-innodb/'
 
Test galera_3nodes.galera_ipv6_rsync has failed 2 times, no more retries!
 
worker[2] > Restart  - using different config file
worker[2] > Restart  - using different config file
worker[2] > Restart  - using different config file
***Warnings generated in error logs during shutdown after running tests: galera_3nodes.galera_ipv6_rsync
 
2021-01-12  3:59:31 0 [ERROR] Aborting
WSREP_SST: [ERROR] rsync daemon port '16031' has been taken (20210112 03:59:02.289)
2021-01-12  3:59:02 0 [ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_rsync --role 'joiner' --address '[::1]:16031' --datadir '/dev/shm/var/2/mysqld.3/data/' --defaults-file '/dev/shm/var/2/my.cnf' --defaults-group-suffix '.3' --parent '31412' --binlog-index 'mysqld-bin.index' --mysqld-args --defaults-group-suffix=.3 --defaults-file=/dev/shm/var/2/my.cnf --log-output=file --innodb --innodb-cmpmem --innodb-cmp-per-index --innodb-trx --innodb-locks --innodb-lock-waits --innodb-metrics --innodb-buffer-pool-stats --innodb-buffer-page --innodb-buffer-page-lru --innodb-sys-columns --innodb-sys-fields --innodb-sys-foreign --innodb-sys-foreign-cols --innodb-sys-indexes --innodb-sys-tables --innodb-sys-virtual --core-file --loose-debug-sync-timeout=300
2021-01-12  3:59:02 0 [ERROR] WSREP: Process completed with error: wsrep_sst_rsync --role 'joiner' --address '[::1]:16031' --datadir '/dev/shm/var/2/mysqld.3/data/' --defaults-file '/dev/shm/var/2/my.cnf' --defaults-group-suffix '.3' --parent '31412' --binlog-index 'mysqld-bin.index' --mysqld-args --defaults-group-suffix=.3 --defaults-file=/dev/shm/var/2/my.cnf --log-output=file --innodb --innodb-cmpmem --innodb-cmp-per-index --innodb-trx --innodb-locks --innodb-lock-waits --innodb-metrics --innodb-buffer-pool-stats --innodb-buffer-page --innodb-buffer-page-lru --innodb-sys-columns --innodb-sys-fields --innodb-sys-foreign --innodb-sys-foreign-cols --innodb-sys-indexes --innodb-sys-tables --innodb-sys-virtual --core-file --loose-debug-sync-timeout=300: 16 (Device or resource busy)
2021-01-12  3:59:02 2 [ERROR] WSREP: Failed to prepare for 'rsync' SST. Unrecoverable.
2021-01-12  3:59:02 2 [ERROR] Aborting
Warning: Memory not freed: 1381120

Server logs.



 Comments   
Comment by Julius Goryavsky [ 2021-05-22 ]

Fixed as part of https://jira.mariadb.org/browse/MDEV-23580

Generated at Thu Feb 08 09:33:29 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.