Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-24650

galera_3nodes.galera_pc_bootstrap MTR failed: Could not execute 'check-testcase' before testcase

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Done
    • 10.2.36, 10.3(EOL), 10.4(EOL)
    • 10.2.39, 10.3.30, 10.4.20
    • Galera, Tests
    • None
    • sles 12

    Description

      galera_3nodes.galera_pc_bootstrap failed on 10.2/10.3/10.4 ES: "Could not execute 'check-testcase' before testcase ".
      It seems to be a sporadic issue. It did not fail on CS during the last month.

      stdio.log:

      10.2.36-10, f423f105cb7e94b86175658df6ce6990395a3f3d, sles 12

      galera_3nodes.galera_pc_bootstrap 'innodb' [ fail ]
              Test ended at 2021-01-16 01:30:27
       
      CURRENT_TEST: galera_3nodes.galera_pc_bootstrap
       
       
      Could not execute 'check-testcase' before testcase 'galera_3nodes.galera_pc_bootstrap' (res: 1):
      mysqltest: Logging to '/var/tmp/mtr/tmp/check-mysqld_3.log'.
      mysqltest: Results saved in '/var/tmp/mtr/tmp/check-mysqld_3.result'.
      mysqltest: Connecting to server localhost:16002 (socket /var/tmp/mtr/tmp/mysqld.3.sock) as 'root', connection 'default', attempt 0 ...
      mysqltest: ... Connected.
      mysqltest: Start processing test commands from './include/check-testcase.test' ...
      mysqltest: At line 81: query 'call mtr.check_testcase()' failed: 1047: WSREP has not yet prepared node for application use
      not ok
      mysqltest failed but provided no output
       
       
       - saving '/var/tmp/mtr/log/galera_3nodes.galera_pc_bootstrap-innodb/' to '/var/tmp/mtr/log/galera_3nodes.galera_pc_bootstrap-innodb/'
       
      Retrying test galera_3nodes.galera_pc_bootstrap, attempt(2/3)...
       
      galera_3nodes.galera_pc_bootstrap 'innodb' [ retry-pass ]  37312
       
      Retrying test galera_3nodes.galera_pc_bootstrap, attempt(3/3)... 
      

      Server logs.

      Attachments

        Activity

          Most likely earlier test did not clean database state correctly and caused this.

          jplindst Jan Lindström (Inactive) added a comment - Most likely earlier test did not clean database state correctly and caused this.

          It failed on 10.3 ES with another output:

          10.3.27-10, e21d261e7883547608a77a42a6e9626964e67045, debian buster

          CURRENT_TEST: galera_3nodes.galera_pc_bootstrap
           
           
          Failed to start mysqld.3
           
           
           - skipping '/var/tmp/mtr/log/galera_3nodes.galera_pc_bootstrap-innodb/'
           
          Retrying test galera_3nodes.galera_pc_bootstrap, attempt(2/3)...
           
          ***Warnings generated in error logs during shutdown after running tests: galera_3nodes.galera_pc_bootstrap
           
          WSREP_SST: [ERROR] rsync daemon port '16011' has been taken (20210102 07:49:03.304)
          2021-01-02  7:49:03 0 [ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_rsync --role 'joiner' --address '127.0.0.1:16011' --datadir '/var/tmp/mtr/mysqld.3/data/' --defaults-file '/var/tmp/mtr/my.cnf' --defaults-group-suffix '.3' --parent '4048' --binlog-index 'mysqld-bin.index' --mysqld-args --defaults-group-suffix=.3 --defaults-file=/var/tmp/mtr/my.cnf --log-output=file --innodb --innodb-cmpmem --innodb-cmp-per-index --innodb-trx --innodb-locks --innodb-lock-waits --innodb-metrics --innodb-buffer-pool-stats --innodb-buffer-page --innodb-buffer-page-lru --innodb-sys-columns --innodb-sys-fields --innodb-sys-foreign --innodb-sys-foreign-cols --innodb-sys-indexes --innodb-sys-tables --innodb-sys-virtual --core-file --loose-debug-sync-timeout=300
          2021-01-02  7:49:03 0 [ERROR] WSREP: Process completed with error: wsrep_sst_rsync --role 'joiner' --address '127.0.0.1:16011' --datadir '/var/tmp/mtr/mysqld.3/data/' --defaults-file '/var/tmp/mtr/my.cnf' --defaults-group-suffix '.3' --parent '4048' --binlog-index 'mysqld-bin.index' --mysqld-args --defaults-group-suffix=.3 --defaults-file=/var/tmp/mtr/my.cnf --log-output=file --innodb --innodb-cmpmem --innodb-cmp-per-index --innodb-trx --innodb-locks --innodb-lock-waits --innodb-metrics --innodb-buffer-pool-stats --innodb-buffer-page --innodb-buffer-page-lru --innodb-sys-columns --innodb-sys-fields --innodb-sys-foreign --innodb-sys-foreign-cols --innodb-sys-indexes --innodb-sys-tables --innodb-sys-virtual --core-file --loose-debug-sync-timeout=300: 16 (Device or resource busy)
          2021-01-02  7:49:03 2 [ERROR] WSREP: Failed to prepare for 'rsync' SST. Unrecoverable.
          2021-01-02  7:49:03 2 [ERROR] Aborting
          Warning: Memory not freed: 1381480
           
          worker[1] mysql-test-run: WARNING: Process [mysqld.3 - pid: 4368, winpid: 4368, exit: 256] died after mysql-test-run waited 17.8 seconds for /var/tmp/mtr/run/mysqld.3.pid to be created.
          

          stepan.patryshev Stepan Patryshev (Inactive) added a comment - It failed on 10.3 ES with another output: 10.3.27-10, e21d261e7883547608a77a42a6e9626964e67045, debian buster CURRENT_TEST: galera_3nodes.galera_pc_bootstrap     Failed to start mysqld.3     - skipping '/var/tmp/mtr/log/galera_3nodes.galera_pc_bootstrap-innodb/'   Retrying test galera_3nodes.galera_pc_bootstrap, attempt(2/3)...   ***Warnings generated in error logs during shutdown after running tests: galera_3nodes.galera_pc_bootstrap   WSREP_SST: [ERROR] rsync daemon port '16011' has been taken (20210102 07:49:03.304) 2021-01-02 7:49:03 0 [ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_rsync --role 'joiner' --address '127.0.0.1:16011' --datadir '/var/tmp/mtr/mysqld.3/data/' --defaults-file '/var/tmp/mtr/my.cnf' --defaults-group-suffix '.3' --parent '4048' --binlog-index 'mysqld-bin.index' --mysqld-args --defaults-group-suffix=.3 --defaults-file=/var/tmp/mtr/my.cnf --log-output=file --innodb --innodb-cmpmem --innodb-cmp-per-index --innodb-trx --innodb-locks --innodb-lock-waits --innodb-metrics --innodb-buffer-pool-stats --innodb-buffer-page --innodb-buffer-page-lru --innodb-sys-columns --innodb-sys-fields --innodb-sys-foreign --innodb-sys-foreign-cols --innodb-sys-indexes --innodb-sys-tables --innodb-sys-virtual --core-file --loose-debug-sync-timeout=300 2021-01-02 7:49:03 0 [ERROR] WSREP: Process completed with error: wsrep_sst_rsync --role 'joiner' --address '127.0.0.1:16011' --datadir '/var/tmp/mtr/mysqld.3/data/' --defaults-file '/var/tmp/mtr/my.cnf' --defaults-group-suffix '.3' --parent '4048' --binlog-index 'mysqld-bin.index' --mysqld-args --defaults-group-suffix=.3 --defaults-file=/var/tmp/mtr/my.cnf --log-output=file --innodb --innodb-cmpmem --innodb-cmp-per-index --innodb-trx --innodb-locks --innodb-lock-waits --innodb-metrics --innodb-buffer-pool-stats --innodb-buffer-page --innodb-buffer-page-lru --innodb-sys-columns --innodb-sys-fields --innodb-sys-foreign --innodb-sys-foreign-cols --innodb-sys-indexes --innodb-sys-tables --innodb-sys-virtual --core-file --loose-debug-sync-timeout=300: 16 (Device or resource busy) 2021-01-02 7:49:03 2 [ERROR] WSREP: Failed to prepare for 'rsync' SST. Unrecoverable. 2021-01-02 7:49:03 2 [ERROR] Aborting Warning: Memory not freed: 1381480   worker[1] mysql-test-run: WARNING: Process [mysqld.3 - pid: 4368, winpid: 4368, exit: 256] died after mysql-test-run waited 17.8 seconds for /var/tmp/mtr/run/mysqld.3.pid to be created.

          galera_3nodes.galera_pc_bootstrap MTR test is running fine in BB. Removed the test from disabled list and pushed changes to 10.3 CS branch.

          Logging: mysql-test-run.pl  --suite=galera,wsrep,galera_3nodes --max-test-fail=0 --testcase-timeout=20 --big-test --retry=3 --parallel=1 --max-save-core=0 --max-save-datadir=1 --force --vardir=/var/tmp/mtr --xml-report=mtr.xml
          vardir: /var/tmp/mtr
          Removing old var directory...
          Creating var directory '/var/tmp/mtr'...
          Checking supported features...
          MariaDB Version 10.3.29-MariaDB
          [..]
          ==============================================================================
           
          TEST                                      RESULT   TIME (ms) or COMMENT
          --------------------------------------------------------------------------
           
          worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019
          [..]
          galera_3nodes.galera_pc_bootstrap 'innodb' [ pass ]  39358
          [..]
          --------------------------------------------------------------------------
          The servers were restarted 106 times
          Spent 2331.418 of 3928 seconds executing testcases
           
          Completed: All 267 tests were successful.
           
          42 tests were skipped, 23 by the test itself.
          

          ramesh Ramesh Sivaraman added a comment - galera_3nodes.galera_pc_bootstrap MTR test is running fine in BB. Removed the test from disabled list and pushed changes to 10.3 CS branch. Logging: mysql-test-run.pl --suite=galera,wsrep,galera_3nodes --max-test-fail=0 --testcase-timeout=20 --big-test --retry=3 --parallel=1 --max-save-core=0 --max-save-datadir=1 --force --vardir=/var/tmp/mtr --xml-report=mtr.xml vardir: /var/tmp/mtr Removing old var directory... Creating var directory '/var/tmp/mtr'... Checking supported features... MariaDB Version 10.3.29-MariaDB [..] ==============================================================================   TEST RESULT TIME (ms) or COMMENT --------------------------------------------------------------------------   worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019 [..] galera_3nodes.galera_pc_bootstrap 'innodb' [ pass ] 39358 [..] -------------------------------------------------------------------------- The servers were restarted 106 times Spent 2331.418 of 3928 seconds executing testcases   Completed: All 267 tests were successful.   42 tests were skipped, 23 by the test itself.

          People

            ramesh Ramesh Sivaraman
            stepan.patryshev Stepan Patryshev (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.