galera_3nodes.galera_ist_gcache_rollover MTR failed: WSREP: exception caused by message: {v=0,t=1,ut=255,o=4,s=0,sr=0,as=1,f=6,src=50524cfe,srcvid=view_id(REG,50524cfe,4),insvid=view_id(UNKNOWN,00000000,0),ru=00000000,r=[-1,-1],fs=75,nl=(}

galera_3nodes.galera_ist_gcache_rollover 'innodb' w1 [ fail ]

        Test ended at 2020-08-25 11:48:06

CURRENT_TEST: galera_3nodes.galera_ipv6_xtrabackup-v2

CURRENT_TEST: galera_3nodes.galera_ipv6_xtrabackup-v2

CURRENT_TEST: galera_3nodes.galera_ist_gcache_rollover

Failed to start mysqld.3

 - skipping '/dev/shm/var/1/log/galera_3nodes.galera_ist_gcache_rollover-innodb/'

Retrying test galera_3nodes.galera_ist_gcache_rollover, attempt(2/3)...

***Warnings generated in error logs during shutdown after running tests: galera_3nodes.galera_ist_gcache_rollover

2020-08-25 11:48:04 3054057280 [ERROR] WSREP: caught exception in PC, state dump to stderr follows:

2020-08-25 11:48:04 3054057280 [ERROR] WSREP: exception caused by message: {v=0,t=1,ut=255,o=4,s=0,sr=0,as=1,f=6,src=50524cfe,srcvid=view_id(REG,50524cfe,4),insvid=view_id(UNKNOWN,00000000,0),ru=00000000,r=[-1,-1],fs=75,nl=(

 }2020-08-25 11:48:04 3054057280 [ERROR] WSREP: failed to open gcomm backend connection: 131: 505b83f7 last prims not consistent (FATAL)

2020-08-25 11:48:04 3054057280 [ERROR] WSREP: gcs/src/gcs_core.cpp:gcs_core_open():209: Failed to open backend connection: -131 (State not recoverable)

2020-08-25 11:48:04 3054057280 [ERROR] WSREP: gcs connect failed: State not recoverable

2020-08-25 11:48:04 3054057280 [ERROR] Aborting

galera_3nodes.galera_ist_gcache_rollover 'innodb' w2 [ fail ]

        Test ended at 2020-08-25 17:51:09

CURRENT_TEST: galera_3nodes.galera_ipv6_mariabackup_section

CURRENT_TEST: galera_3nodes.galera_ipv6_rsync_section

CURRENT_TEST: galera_3nodes.galera_ist_gcache_rollover

Failed to start mysqld.3

worker[2] > Restart  - using different config file

worker[2] > Restart  - using different config file

worker[2] > Restart  - using different config file

 - skipping '/dev/shm/var/2/log/galera_3nodes.galera_ist_gcache_rollover-innodb/'

Retrying test galera_3nodes.galera_ist_gcache_rollover, attempt(2/3)...

***Warnings generated in error logs during shutdown after running tests: galera_3nodes.galera_ist_gcache_rollover

WSREP_SST: [ERROR] rsync daemon port '16031' has been taken (20200825 17:50:25.449)

2020-08-25 17:50:25 0 [ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_rsync --role 'joiner' --address '127.0.0.1:16031' --datadir '/dev/shm/var/2/mysqld.3/data/' --defaults-file '/dev/shm/var/2/my.cnf' --defaults-group-suffix '.3' --parent '32484' --binlog-index 'mysqld-bin.index' --mysqld-args --defaults-group-suffix=.3 --defaults-file=/dev/shm/var/2/my.cnf --log-output=file --innodb --innodb-cmpmem --innodb-cmp-per-index --innodb-trx --innodb-locks --innodb-lock-waits --innodb-metrics --innodb-buffer-pool-stats --innodb-buffer-page --innodb-buffer-page-lru --innodb-sys-columns --innodb-sys-fields --innodb-sys-foreign --innodb-sys-foreign-cols --innodb-sys-indexes --innodb-sys-tables --innodb-sys-virtual --core-file --loose-debug-sync-timeout=300

2020-08-25 17:50:25 0 [ERROR] WSREP: Process completed with error: wsrep_sst_rsync --role 'joiner' --address '127.0.0.1:16031' --datadir '/dev/shm/var/2/mysqld.3/data/' --defaults-file '/dev/shm/var/2/my.cnf' --defaults-group-suffix '.3' --parent '32484' --binlog-index 'mysqld-bin.index' --mysqld-args --defaults-group-suffix=.3 --defaults-file=/dev/shm/var/2/my.cnf --log-output=file --innodb --innodb-cmpmem --innodb-cmp-per-index --innodb-trx --innodb-locks --innodb-lock-waits --innodb-metrics --innodb-buffer-pool-stats --innodb-buffer-page --innodb-buffer-page-lru --innodb-sys-columns --innodb-sys-fields --innodb-sys-foreign --innodb-sys-foreign-cols --innodb-sys-indexes --innodb-sys-tables --innodb-sys-virtual --core-file --loose-debug-sync-timeout=300: 16 (Device or resource busy)

2020-08-25 17:50:25 2 [ERROR] WSREP: Failed to prepare for 'rsync' SST. Unrecoverable.

2020-08-25 17:50:25 2 [ERROR] Aborting

Warning: Memory not freed: 1381136

worker[2] mysql-test-run: WARNING: Waited 60 seconds for /dev/shm/var/2/run/mysqld.3.pid to be created, still waiting for 120 seconds...

worker[2] mysql-test-run: WARNING: Waited 120 seconds for /dev/shm/var/2/run/mysqld.3.pid to be created, still waiting for 60 seconds...

worker[2] mysql-test-run: WARNING: Timeout after mysql-test-run waited 180 seconds for the process [mysqld.3 - pid: 979, winpid: 979] to create a pid file.

galera_3nodes.galera_ipv6_rsync_section 'innodb' w2 [ retry-fail ]

        Test ended at 2020-08-25 17:54:14

CURRENT_TEST: galera_3nodes.galera_ipv6_mariabackup_section

CURRENT_TEST: galera_3nodes.galera_ipv6_rsync_section

CURRENT_TEST: galera_3nodes.galera_ist_gcache_rollover

CURRENT_TEST: galera_3nodes.galera_ipv6_rsync_section

Failed to start mysqld.3

worker[2] > Restart  - using different config file

worker[2] > Restart  - using different config file

worker[2] > Restart  - using different config file

 - skipping '/dev/shm/var/2/log/galera_3nodes.galera_ipv6_rsync_section-innodb/'

Test galera_3nodes.galera_ipv6_rsync_section has failed 2 times, no more retries!

***Warnings generated in error logs during shutdown after running tests: galera_3nodes.galera_ipv6_rsync_section

WSREP_SST: [ERROR] rsync daemon port '16031' has been taken (20200825 17:51:11.334)

2020-08-25 17:51:11 0 [ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_rsync --role 'joiner' --address '[::1]:16031' --datadir '/dev/shm/var/2/mysqld.3/data/' --defaults-file '/dev/shm/var/2/my.cnf' --defaults-group-suffix '.3' --parent '980' --binlog-index 'mysqld-bin.index' --mysqld-args --defaults-group-suffix=.3 --defaults-file=/dev/shm/var/2/my.cnf --log-output=file --innodb --innodb-cmpmem --innodb-cmp-per-index --innodb-trx --innodb-locks --innodb-lock-waits --innodb-metrics --innodb-buffer-pool-stats --innodb-buffer-page --innodb-buffer-page-lru --innodb-sys-columns --innodb-sys-fields --innodb-sys-foreign --innodb-sys-foreign-cols --innodb-sys-indexes --innodb-sys-tables --innodb-sys-virtual --bind-address=:: --core-file --loose-debug-sync-timeout=300

2020-08-25 17:51:11 0 [ERROR] WSREP: Process completed with error: wsrep_sst_rsync --role 'joiner' --address '[::1]:16031' --datadir '/dev/shm/var/2/mysqld.3/data/' --defaults-file '/dev/shm/var/2/my.cnf' --defaults-group-suffix '.3' --parent '980' --binlog-index 'mysqld-bin.index' --mysqld-args --defaults-group-suffix=.3 --defaults-file=/dev/shm/var/2/my.cnf --log-output=file --innodb --innodb-cmpmem --innodb-cmp-per-index --innodb-trx --innodb-locks --innodb-lock-waits --innodb-metrics --innodb-buffer-pool-stats --innodb-buffer-page --innodb-buffer-page-lru --innodb-sys-columns --innodb-sys-fields --innodb-sys-foreign --innodb-sys-foreign-cols --innodb-sys-indexes --innodb-sys-tables --innodb-sys-virtual --bind-address=:: --core-file --loose-debug-sync-timeout=300: 16 (Device or resource busy)

2020-08-25 17:51:11 1 [ERROR] WSREP: Failed to prepare for 'rsync' SST. Unrecoverable.

2020-08-25 17:51:11 1 [ERROR] Aborting

wsrep.mysql_tzinfo_to_sql_symlink 'innodb' w2 [ pass ]     34

worker[2] > Restart [mysqld.1 - pid: 1307, winpid: 1307] - using different config file

galera_3nodes.galera_ist_gcache_rollover 'innodb' w2 [ skipped ]  debug_sync is not available

Retrying test galera_3nodes.galera_ist_gcache_rollover, attempt(3/3)...