galera_3nodes.galera_ipv6_rsync failed on BB 10.3 CS and BB 10.6 CS: "rsync daemon port '16031' has been taken".
It seems to be a sporadic issue.
stdio.log
:
10.3.28, 1be707286ef13041a67e6e6ca30125470245a898, kvm-deb-bionic-amd64
|
galera_3nodes.galera_ipv6_rsync 'innodb' w2 [ fail ]
|
Test ended at 2021-01-12 03:59:00
|
|
CURRENT_TEST: galera_3nodes.galera_ipv6_rsync
|
|
|
Failed to start mysqld.3
|
mysqltest failed but provided no output
|
|
|
- skipping '/dev/shm/var/2/log/galera_3nodes.galera_ipv6_rsync-innodb/'
|
|
Retrying test galera_3nodes.galera_ipv6_rsync, attempt(2/3)...
|
|
worker[2] > Restart - not started
|
worker[2] > Restart - not started
|
worker[2] > Restart - not started
|
***Warnings generated in error logs during shutdown after running tests: galera_3nodes.galera_ipv6_rsync
|
|
WSREP_SST: [ERROR] rsync daemon port '16031' has been taken (20210112 03:58:46.128)
|
2021-01-12 3:58:46 0 [ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_rsync --role 'joiner' --address '[::1]:16031' --datadir '/dev/shm/var/2/mysqld.3/data/' --defaults-file '/dev/shm/var/2/my.cnf' --defaults-group-suffix '.3' --parent '30628' --binlog-index 'mysqld-bin.index' --mysqld-args --defaults-group-suffix=.3 --defaults-file=/dev/shm/var/2/my.cnf --log-output=file --innodb --innodb-cmpmem --innodb-cmp-per-index --innodb-trx --innodb-locks --innodb-lock-waits --innodb-metrics --innodb-buffer-pool-stats --innodb-buffer-page --innodb-buffer-page-lru --innodb-sys-columns --innodb-sys-fields --innodb-sys-foreign --innodb-sys-foreign-cols --innodb-sys-indexes --innodb-sys-tables --innodb-sys-virtual --core-file --loose-debug-sync-timeout=300
|
2021-01-12 3:58:46 0 [ERROR] WSREP: Process completed with error: wsrep_sst_rsync --role 'joiner' --address '[::1]:16031' --datadir '/dev/shm/var/2/mysqld.3/data/' --defaults-file '/dev/shm/var/2/my.cnf' --defaults-group-suffix '.3' --parent '30628' --binlog-index 'mysqld-bin.index' --mysqld-args --defaults-group-suffix=.3 --defaults-file=/dev/shm/var/2/my.cnf --log-output=file --innodb --innodb-cmpmem --innodb-cmp-per-index --innodb-trx --innodb-locks --innodb-lock-waits --innodb-metrics --innodb-buffer-pool-stats --innodb-buffer-page --innodb-buffer-page-lru --innodb-sys-columns --innodb-sys-fields --innodb-sys-foreign --innodb-sys-foreign-cols --innodb-sys-indexes --innodb-sys-tables --innodb-sys-virtual --core-file --loose-debug-sync-timeout=300: 16 (Device or resource busy)
|
2021-01-12 3:58:46 2 [ERROR] WSREP: Failed to prepare for 'rsync' SST. Unrecoverable.
|
2021-01-12 3:58:46 2 [ERROR] Aborting
|
Warning: Memory not freed: 1381120
|
|
galera_3nodes.galera_ipv6_rsync_section 'innodb' w1 [ pass ] 14117
|
worker[1] > Restart - using different config file
|
worker[1] > Restart - using different config file
|
worker[1] > Restart - using different config file
|
worker[2] mysql-test-run: WARNING: Process [mysqld.2 - pid: 31409, winpid: 31409, exit: 256] died after mysql-test-run waited 31.4 seconds for /dev/shm/var/2/run/mysqld.2.pid to be created.
|
galera_3nodes.galera_ipv6_rsync 'innodb' w2 [ retry-fail ]
|
Test ended at 2021-01-12 03:59:32
|
|
CURRENT_TEST: galera_3nodes.galera_ipv6_rsync
|
|
|
Failed to start mysqld.2
|
mysqltest failed but provided no output
|
|
|
- skipping '/dev/shm/var/2/log/galera_3nodes.galera_ipv6_rsync-innodb/'
|
|
Test galera_3nodes.galera_ipv6_rsync has failed 2 times, no more retries!
|
|
worker[2] > Restart - using different config file
|
worker[2] > Restart - using different config file
|
worker[2] > Restart - using different config file
|
***Warnings generated in error logs during shutdown after running tests: galera_3nodes.galera_ipv6_rsync
|
|
2021-01-12 3:59:31 0 [ERROR] Aborting
|
WSREP_SST: [ERROR] rsync daemon port '16031' has been taken (20210112 03:59:02.289)
|
2021-01-12 3:59:02 0 [ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_rsync --role 'joiner' --address '[::1]:16031' --datadir '/dev/shm/var/2/mysqld.3/data/' --defaults-file '/dev/shm/var/2/my.cnf' --defaults-group-suffix '.3' --parent '31412' --binlog-index 'mysqld-bin.index' --mysqld-args --defaults-group-suffix=.3 --defaults-file=/dev/shm/var/2/my.cnf --log-output=file --innodb --innodb-cmpmem --innodb-cmp-per-index --innodb-trx --innodb-locks --innodb-lock-waits --innodb-metrics --innodb-buffer-pool-stats --innodb-buffer-page --innodb-buffer-page-lru --innodb-sys-columns --innodb-sys-fields --innodb-sys-foreign --innodb-sys-foreign-cols --innodb-sys-indexes --innodb-sys-tables --innodb-sys-virtual --core-file --loose-debug-sync-timeout=300
|
2021-01-12 3:59:02 0 [ERROR] WSREP: Process completed with error: wsrep_sst_rsync --role 'joiner' --address '[::1]:16031' --datadir '/dev/shm/var/2/mysqld.3/data/' --defaults-file '/dev/shm/var/2/my.cnf' --defaults-group-suffix '.3' --parent '31412' --binlog-index 'mysqld-bin.index' --mysqld-args --defaults-group-suffix=.3 --defaults-file=/dev/shm/var/2/my.cnf --log-output=file --innodb --innodb-cmpmem --innodb-cmp-per-index --innodb-trx --innodb-locks --innodb-lock-waits --innodb-metrics --innodb-buffer-pool-stats --innodb-buffer-page --innodb-buffer-page-lru --innodb-sys-columns --innodb-sys-fields --innodb-sys-foreign --innodb-sys-foreign-cols --innodb-sys-indexes --innodb-sys-tables --innodb-sys-virtual --core-file --loose-debug-sync-timeout=300: 16 (Device or resource busy)
|
2021-01-12 3:59:02 2 [ERROR] WSREP: Failed to prepare for 'rsync' SST. Unrecoverable.
|
2021-01-12 3:59:02 2 [ERROR] Aborting
|
Warning: Memory not freed: 1381120
|
Server logs
.
- duplicates
-
MDEV-23580
galera_3nodes.galera_ipv6_rsync_section MTR failed: WSREP_SST: [ERROR] rsync daemon port '16008' has been taken
-
-
Closed
{"report":{"fcp":1065.1000000238419,"ttfb":339.3000000715256,"pageVisibility":"visible","entityId":96998,"key":"jira.project.issue.view-issue","isInitial":true,"threshold":1000,"elementTimings":{},"userDeviceMemory":8,"userDeviceProcessors":64,"apdex":0.5,"journeyId":"b0a2d326-5651-42d0-a957-588bb7860d34","navigationType":0,"readyForUser":1183.2000000476837,"redirectCount":0,"resourceLoadedEnd":1331.8000000715256,"resourceLoadedStart":350,"resourceTiming":[{"duration":150.40000009536743,"initiatorType":"link","name":"https://jira.mariadb.org/s/2c21342762a6a02add1c328bed317ffd-CDN/lu2bu7/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/css/_super/batch.css","startTime":350,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":350,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":500.40000009536743,"responseStart":0,"secureConnectionStart":0},{"duration":163.10000002384186,"initiatorType":"link","name":"https://jira.mariadb.org/s/7ebd35e77e471bc30ff0eba799ebc151-CDN/lu2bu7/820016/12ta74/8679b4946efa1a0bb029a3a22206fb5d/_/download/contextbatch/css/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true","startTime":350.3000000715256,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":350.3000000715256,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":513.4000000953674,"responseStart":0,"secureConnectionStart":0},{"duration":231.30000007152557,"initiatorType":"script","name":"https://jira.mariadb.org/s/fbf975c0cce4b1abf04784eeae9ba1f4-CDN/lu2bu7/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/js/_super/batch.js?locale=en","startTime":350.5,"connectEnd":350.5,"connectStart":350.5,"domainLookupEnd":350.5,"domainLookupStart":350.5,"fetchStart":350.5,"redirectEnd":0,"redirectStart":0,"requestStart":350.5,"responseEnd":581.8000000715256,"responseStart":581.8000000715256,"secureConnectionStart":350.5},{"duration":306.10000002384186,"initiatorType":"script","name":"https://jira.mariadb.org/s/099b33461394b8015fc36c0a4b96e19f-CDN/lu2bu7/820016/12ta74/8679b4946efa1a0bb029a3a22206fb5d/_/download/contextbatch/js/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en&slack-enabled=true","startTime":350.60000002384186,"connectEnd":350.60000002384186,"connectStart":350.60000002384186,"domainLookupEnd":350.60000002384186,"domainLookupStart":350.60000002384186,"fetchStart":350.60000002384186,"redirectEnd":0,"redirectStart":0,"requestStart":350.60000002384186,"responseEnd":656.7000000476837,"responseStart":656.7000000476837,"secureConnectionStart":350.60000002384186},{"duration":310.2999999523163,"initiatorType":"script","name":"https://jira.mariadb.org/s/94c15bff32baef80f4096a08aceae8bc-CDN/lu2bu7/820016/12ta74/c92c0caa9a024ae85b0ebdbed7fb4bd7/_/download/contextbatch/js/atl.global,-_super/batch.js?locale=en","startTime":350.8000000715256,"connectEnd":350.8000000715256,"connectStart":350.8000000715256,"domainLookupEnd":350.8000000715256,"domainLookupStart":350.8000000715256,"fetchStart":350.8000000715256,"redirectEnd":0,"redirectStart":0,"requestStart":350.8000000715256,"responseEnd":661.1000000238419,"responseStart":661.1000000238419,"secureConnectionStart":350.8000000715256},{"duration":310.5,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bu7/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-en/jira.webresources:calendar-en.js","startTime":351.10000002384186,"connectEnd":351.10000002384186,"connectStart":351.10000002384186,"domainLookupEnd":351.10000002384186,"domainLookupStart":351.10000002384186,"fetchStart":351.10000002384186,"redirectEnd":0,"redirectStart":0,"requestStart":351.10000002384186,"responseEnd":661.6000000238419,"responseStart":661.6000000238419,"secureConnectionStart":351.10000002384186},{"duration":311.10000002384186,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bu7/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-localisation-moment/jira.webresources:calendar-localisation-moment.js","startTime":351.2000000476837,"connectEnd":351.2000000476837,"connectStart":351.2000000476837,"domainLookupEnd":351.2000000476837,"domainLookupStart":351.2000000476837,"fetchStart":351.2000000476837,"redirectEnd":0,"redirectStart":0,"requestStart":351.2000000476837,"responseEnd":662.3000000715256,"responseStart":662.3000000715256,"secureConnectionStart":351.2000000476837},{"duration":375.39999997615814,"initiatorType":"link","name":"https://jira.mariadb.org/s/b04b06a02d1959df322d9cded3aeecc1-CDN/lu2bu7/820016/12ta74/a2ff6aa845ffc9a1d22fe23d9ee791fc/_/download/contextbatch/css/jira.global.look-and-feel,-_super/batch.css","startTime":351.40000009536743,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":351.40000009536743,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":726.8000000715256,"responseStart":0,"secureConnectionStart":0},{"duration":311.60000002384186,"initiatorType":"script","name":"https://jira.mariadb.org/rest/api/1.0/shortcuts/820016/47140b6e0a9bc2e4913da06536125810/shortcuts.js?context=issuenavigation&context=issueaction","startTime":351.5,"connectEnd":351.5,"connectStart":351.5,"domainLookupEnd":351.5,"domainLookupStart":351.5,"fetchStart":351.5,"redirectEnd":0,"redirectStart":0,"requestStart":351.5,"responseEnd":663.1000000238419,"responseStart":663.1000000238419,"secureConnectionStart":351.5},{"duration":375.3000000715256,"initiatorType":"link","name":"https://jira.mariadb.org/s/3ac36323ba5e4eb0af2aa7ac7211b4bb-CDN/lu2bu7/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/css/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.css?jira.create.linked.issue=true","startTime":351.60000002384186,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":351.60000002384186,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":726.9000000953674,"responseStart":0,"secureConnectionStart":0},{"duration":312,"initiatorType":"script","name":"https://jira.mariadb.org/s/3339d87fa2538a859872f2df449bf8d0-CDN/lu2bu7/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/js/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.js?jira.create.linked.issue=true&locale=en","startTime":351.8000000715256,"connectEnd":351.8000000715256,"connectStart":351.8000000715256,"domainLookupEnd":351.8000000715256,"domainLookupStart":351.8000000715256,"fetchStart":351.8000000715256,"redirectEnd":0,"redirectStart":0,"requestStart":351.8000000715256,"responseEnd":663.8000000715256,"responseStart":663.8000000715256,"secureConnectionStart":351.8000000715256},{"duration":971.6999999284744,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bu7/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-js/jira.webresources:bigpipe-js.js","startTime":352.8000000715256,"connectEnd":352.8000000715256,"connectStart":352.8000000715256,"domainLookupEnd":352.8000000715256,"domainLookupStart":352.8000000715256,"fetchStart":352.8000000715256,"redirectEnd":0,"redirectStart":0,"requestStart":352.8000000715256,"responseEnd":1324.5,"responseStart":1324.5,"secureConnectionStart":352.8000000715256},{"duration":951.2000000476837,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bu7/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-init/jira.webresources:bigpipe-init.js","startTime":380.60000002384186,"connectEnd":380.60000002384186,"connectStart":380.60000002384186,"domainLookupEnd":380.60000002384186,"domainLookupStart":380.60000002384186,"fetchStart":380.60000002384186,"redirectEnd":0,"redirectStart":0,"requestStart":380.60000002384186,"responseEnd":1331.8000000715256,"responseStart":1331.8000000715256,"secureConnectionStart":380.60000002384186},{"duration":585.0999999046326,"initiatorType":"xmlhttprequest","name":"https://jira.mariadb.org/rest/webResources/1.0/resources","startTime":739.9000000953674,"connectEnd":739.9000000953674,"connectStart":739.9000000953674,"domainLookupEnd":739.9000000953674,"domainLookupStart":739.9000000953674,"fetchStart":739.9000000953674,"redirectEnd":0,"redirectStart":0,"requestStart":739.9000000953674,"responseEnd":1325,"responseStart":1325,"secureConnectionStart":739.9000000953674},{"duration":283.59999990463257,"initiatorType":"script","name":"https://www.google-analytics.com/analytics.js","startTime":1055.9000000953674,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":1055.9000000953674,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":1339.5,"responseStart":0,"secureConnectionStart":0},{"duration":142.80000007152557,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bu7/820016/12ta74/e65b778d185daf5aee24936755b43da6/_/download/contextbatch/js/browser-metrics-plugin.contrib,-_super,-atl.general/batch.js?agile_global_admin_condition=true&jag=true&slack-enabled=true","startTime":1338,"connectEnd":1338,"connectStart":1338,"domainLookupEnd":1338,"domainLookupStart":1338,"fetchStart":1338,"redirectEnd":0,"redirectStart":0,"requestStart":1338,"responseEnd":1480.8000000715256,"responseStart":1480.8000000715256,"secureConnectionStart":1338}],"fetchStart":0,"domainLookupStart":0,"domainLookupEnd":0,"connectStart":0,"connectEnd":0,"requestStart":148,"responseStart":339,"responseEnd":355,"domLoading":347,"domInteractive":1427,"domContentLoadedEventStart":1427,"domContentLoadedEventEnd":1479,"domComplete":1768,"loadEventStart":1768,"loadEventEnd":1768,"userAgent":"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)","marks":[{"name":"bigPipe.sidebar-id.start","time":1405.6000000238419},{"name":"bigPipe.sidebar-id.end","time":1406.4000000953674},{"name":"bigPipe.activity-panel-pipe-id.start","time":1406.6000000238419},{"name":"bigPipe.activity-panel-pipe-id.end","time":1408},{"name":"activityTabFullyLoaded","time":1496.4000000953674}],"measures":[],"correlationId":"1f2fd696b3d0eb","effectiveType":"4g","downlink":10,"rtt":0,"serverDuration":119,"dbReadsTimeInMs":25,"dbConnsTimeInMs":36,"applicationHash":"9d11dbea5f4be3d4cc21f03a88dd11d8c8687422","experiments":[]}}