[MDEV-24650] galera_3nodes.galera_pc_bootstrap MTR failed: Could not execute 'check-testcase' before testcase Created: 2021-01-21  Updated: 2021-09-29  Resolved: 2021-06-01

Status: Closed
Project: MariaDB Server
Component/s: Galera, Tests
Affects Version/s: 10.2.36, 10.3, 10.4
Fix Version/s: 10.2.39, 10.3.30, 10.4.20

Type: Bug Priority: Major
Reporter: Stepan Patryshev (Inactive) Assignee: Ramesh Sivaraman
Resolution: Done Votes: 0
Labels: None
Environment:

sles 12

Attachments: Zip Archive 210116_102ES_galera_pc_bootstrap_logs.zip     Text File 210116_102ES_galera_pc_bootstrap_mtr.log    

 Description   

galera_3nodes.galera_pc_bootstrap failed on 10.2/10.3/10.4 ES: "Could not execute 'check-testcase' before testcase ".
It seems to be a sporadic issue. It did not fail on CS during the last month.

stdio.log:

10.2.36-10, f423f105cb7e94b86175658df6ce6990395a3f3d, sles 12

 
galera_3nodes.galera_pc_bootstrap 'innodb' [ fail ]
 
        Test ended at 2021-01-16 01:30:27
 
 
 
CURRENT_TEST: galera_3nodes.galera_pc_bootstrap
 
 
 
 
 
Could not execute 'check-testcase' before testcase 'galera_3nodes.galera_pc_bootstrap' (res: 1):
 
mysqltest: Logging to '/var/tmp/mtr/tmp/check-mysqld_3.log'.
 
mysqltest: Results saved in '/var/tmp/mtr/tmp/check-mysqld_3.result'.
 
mysqltest: Connecting to server localhost:16002 (socket /var/tmp/mtr/tmp/mysqld.3.sock) as 'root', connection 'default', attempt 0 ...
 
mysqltest: ... Connected.
 
mysqltest: Start processing test commands from './include/check-testcase.test' ...
 
mysqltest: At line 81: query 'call mtr.check_testcase()' failed: 1047: WSREP has not yet prepared node for application use
 
not ok
 
mysqltest failed but provided no output
 
 
 
 
 
 - saving '/var/tmp/mtr/log/galera_3nodes.galera_pc_bootstrap-innodb/' to '/var/tmp/mtr/log/galera_3nodes.galera_pc_bootstrap-innodb/'
 
 
 
Retrying test galera_3nodes.galera_pc_bootstrap, attempt(2/3)...
 
 
 
galera_3nodes.galera_pc_bootstrap 'innodb' [ retry-pass ]  37312
 
 
 
Retrying test galera_3nodes.galera_pc_bootstrap, attempt(3/3)...

Server logs.

 Comments   
Comment by Jan Lindström (Inactive) [ 2021-01-22 ]

Most likely earlier test did not clean database state correctly and caused this.

Comment by Stepan Patryshev (Inactive) [ 2021-01-22 ]

It failed on 10.3 ES with another output:

10.3.27-10, e21d261e7883547608a77a42a6e9626964e67045, debian buster

 
CURRENT_TEST: galera_3nodes.galera_pc_bootstrap
 
 
 
 
 
Failed to start mysqld.3
 
 
 
 
 
 - skipping '/var/tmp/mtr/log/galera_3nodes.galera_pc_bootstrap-innodb/'
 
 
 
Retrying test galera_3nodes.galera_pc_bootstrap, attempt(2/3)...
 
 
 
***Warnings generated in error logs during shutdown after running tests: galera_3nodes.galera_pc_bootstrap
 
 
 
WSREP_SST: [ERROR] rsync daemon port '16011' has been taken (20210102 07:49:03.304)
 
2021-01-02  7:49:03 0 [ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_rsync --role 'joiner' --address '127.0.0.1:16011' --datadir '/var/tmp/mtr/mysqld.3/data/' --defaults-file '/var/tmp/mtr/my.cnf' --defaults-group-suffix '.3' --parent '4048' --binlog-index 'mysqld-bin.index' --mysqld-args --defaults-group-suffix=.3 --defaults-file=/var/tmp/mtr/my.cnf --log-output=file --innodb --innodb-cmpmem --innodb-cmp-per-index --innodb-trx --innodb-locks --innodb-lock-waits --innodb-metrics --innodb-buffer-pool-stats --innodb-buffer-page --innodb-buffer-page-lru --innodb-sys-columns --innodb-sys-fields --innodb-sys-foreign --innodb-sys-foreign-cols --innodb-sys-indexes --innodb-sys-tables --innodb-sys-virtual --core-file --loose-debug-sync-timeout=300
 
2021-01-02  7:49:03 0 [ERROR] WSREP: Process completed with error: wsrep_sst_rsync --role 'joiner' --address '127.0.0.1:16011' --datadir '/var/tmp/mtr/mysqld.3/data/' --defaults-file '/var/tmp/mtr/my.cnf' --defaults-group-suffix '.3' --parent '4048' --binlog-index 'mysqld-bin.index' --mysqld-args --defaults-group-suffix=.3 --defaults-file=/var/tmp/mtr/my.cnf --log-output=file --innodb --innodb-cmpmem --innodb-cmp-per-index --innodb-trx --innodb-locks --innodb-lock-waits --innodb-metrics --innodb-buffer-pool-stats --innodb-buffer-page --innodb-buffer-page-lru --innodb-sys-columns --innodb-sys-fields --innodb-sys-foreign --innodb-sys-foreign-cols --innodb-sys-indexes --innodb-sys-tables --innodb-sys-virtual --core-file --loose-debug-sync-timeout=300: 16 (Device or resource busy)
 
2021-01-02  7:49:03 2 [ERROR] WSREP: Failed to prepare for 'rsync' SST. Unrecoverable.
 
2021-01-02  7:49:03 2 [ERROR] Aborting
 
Warning: Memory not freed: 1381480
 
 
 
worker[1] mysql-test-run: WARNING: Process [mysqld.3 - pid: 4368, winpid: 4368, exit: 256] died after mysql-test-run waited 17.8 seconds for /var/tmp/mtr/run/mysqld.3.pid to be created.

Comment by Ramesh Sivaraman [ 2021-05-12 ]

galera_3nodes.galera_pc_bootstrap MTR test is running fine in BB. Removed the test from disabled list and pushed changes to 10.3 CS branch. 

Logging: mysql-test-run.pl  --suite=galera,wsrep,galera_3nodes --max-test-fail=0 --testcase-timeout=20 --big-test --retry=3 --parallel=1 --max-save-core=0 --max-save-datadir=1 --force --vardir=/var/tmp/mtr --xml-report=mtr.xml
 
vardir: /var/tmp/mtr
 
Removing old var directory...
 
Creating var directory '/var/tmp/mtr'...
 
Checking supported features...
 
MariaDB Version 10.3.29-MariaDB
 
[..]
 
==============================================================================
 
 
 
TEST                                      RESULT   TIME (ms) or COMMENT
 
--------------------------------------------------------------------------
 
 
 
worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019
 
[..]
 
galera_3nodes.galera_pc_bootstrap 'innodb' [ pass ]  39358
 
[..]
 
--------------------------------------------------------------------------
 
The servers were restarted 106 times
 
Spent 2331.418 of 3928 seconds executing testcases
 
 
 
Completed: All 267 tests were successful.
 
 
 
42 tests were skipped, 23 by the test itself.

Generated at Thu Feb 08 09:31:36 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.