Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-18593

galera.galera_gcache_recover_full_gcache: Test failure: galera_gcache_recover_full_gcache.test: assert_grep.inc failed

Details

    Description

      MariaDB Server: 10.4.3-MariaDB-debug built from sources, branch=10.4, commit: ce6505f890956f81354269a991e69f20babae8e4.
      Galera4 lib: debug built from sources, commit: fbf047d76c4d95bc1ef1c465409efdca11256225.

      Run:

      ./mtr galera.galera_gcache_recover_full_gcache --force --max-test-fail=0 --repeat=3
      ./mtr --suite galera --force --big-test --max-test-fail=0
      

      stdout.log: (the whole file is attached: 190213_desktop_stdout.log)

      galera.galera_gcache_recover_full_gcache 'innodb' [ pass ]  56778
      galera.galera_gcache_recover_full_gcache 'innodb' [ 2 pass ]  53569
      galera.galera_gcache_recover_full_gcache 'innodb' [ 3 fail ]
       
      CURRENT_TEST: galera.galera_gcache_recover_full_gcache
      mysqltest: In included file "./include/assert_grep.inc": 
      included from /home/stepan/galera/git/10.4/server/mysql-test/suite/galera/t/galera_gcache_recover_full_gcache.test at line 65:
      At line 152: assert_grep.inc failed.
       
      The result from queries just before the failure was:
      < snip >
      connection node_1;
      Performing --wsrep-recover ...
      Using --wsrep-start-position when starting mysqld ...
      connection node_2;
      Performing --wsrep-recover ...
      Using --wsrep-start-position when starting mysqld ...
      connection node_1;
      include/diff_servers.inc [servers=1 2]
      connection node_1;
      DROP TABLE t1;
      CALL mtr.add_suppression("Skipped GCache ring buffer recovery");
      include/assert_grep.inc [IST first seqno [24] not found from cache, falling back to SST]
      include/assert_grep.inc failed!
      assert_text: 'IST first seqno [24] not found from cache, falling back to SST'
      assert_file: '/home/stepan/galera/git/10.4/server/mysql-test/var/log/mysqld.1.err'
      assert_select: 'IST first seqno [24] not found from cache, falling back to SST'
      assert_match: ''
      assert_count: '1'
      assert_only_after: 'starting as process'
      number of matching lines: 0
       
      Completed: Failed 1/3 tests, 66.67% were successful.
      

      See also attached 190215_galera.galera_gcache_recover_full_gcache.zip.

      Note: This test is failed sporadically.

      Attachments

        Issue Links

          Activity

            Closing it as not reproduced.

            stepan.patryshev Stepan Patryshev (Inactive) added a comment - Closing it as not reproduced.

            Reopening it since it sporadically fails again on the local suite runs, but only on HDD and with --parallel=auto.

            MariaDB Version 10.4.9-MariaDB-debug: Repository: MariaDB/server; branch 10.4; Revision 9afbb1069ab4178741b76adefacd3f6a0ed51369; debug built from sources.

            Galera lib 26.4.3(r4535): Repository: MariaDB/galera; branch mariadb-4.x; Revision 752664dc3c7065d8e0c73ac99d0028a5f84eb250; debug built from sources.

            Run:

            ./mtr --suite galera --parallel=auto --testcase-timeout=90 --force --big-test --max-test-fail=0 --timer --timestamp --timediff
            

            stdout.log:

            191029 13:55:05 +0.042 galera.galera_gcache_recover_full_gcache 'innodb' w2 [ fail ]
                    Test ended at 2019-10-29 13:55:05
             
            CURRENT_TEST: galera.galera_gcache_recover_full_gcache
            mysqltest: In included file "./include/assert_grep.inc": 
            included from /home/stepan/mariadb/10.4/github/mysql-test/suite/galera/t/galera_gcache_recover_full_gcache.test at line 65:
            At line 152: assert_grep.inc failed.
             
            The result from queries just before the failure was:
            < snip >
            connection node_1;
            Performing --wsrep-recover ...
            Using --wsrep-start-position when starting mysqld ...
            connection node_2;
            Performing --wsrep-recover ...
            Using --wsrep-start-position when starting mysqld ...
            connection node_1;
            include/diff_servers.inc [servers=1 2]
            connection node_1;
            DROP TABLE t1;
            CALL mtr.add_suppression("Skipped GCache ring buffer recovery");
            include/assert_grep.inc [IST first seqno [24] not found from cache, falling back to SST]
            include/assert_grep.inc failed!
            assert_text: 'IST first seqno [24] not found from cache, falling back to SST'
            assert_file: '/home/stepan/mariadb/10.4/github/mysql-test/var/2/log/mysqld.1.err'
            assert_select: 'IST first seqno [24] not found from cache, falling back to SST'
            assert_match: ''
            assert_count: '1'
            assert_only_after: 'starting as process'
            number of matching lines: 0
            

            Other logs.

            stepan.patryshev Stepan Patryshev (Inactive) added a comment - - edited Reopening it since it sporadically fails again on the local suite runs, but only on HDD and with --parallel=auto. MariaDB Version 10.4.9-MariaDB-debug: Repository: MariaDB/server; branch 10.4; Revision 9afbb1069ab4178741b76adefacd3f6a0ed51369; debug built from sources. Galera lib 26.4.3(r4535): Repository: MariaDB/galera; branch mariadb-4.x; Revision 752664dc3c7065d8e0c73ac99d0028a5f84eb250; debug built from sources. Run: ./mtr --suite galera --parallel=auto --testcase-timeout=90 --force --big-test --max-test-fail=0 --timer --timestamp --timediff stdout.log : 191029 13:55:05 +0.042 galera.galera_gcache_recover_full_gcache 'innodb' w2 [ fail ] Test ended at 2019-10-29 13:55:05   CURRENT_TEST: galera.galera_gcache_recover_full_gcache mysqltest: In included file "./include/assert_grep.inc": included from /home/stepan/mariadb/10.4/github/mysql-test/suite/galera/t/galera_gcache_recover_full_gcache.test at line 65: At line 152: assert_grep.inc failed.   The result from queries just before the failure was: < snip > connection node_1; Performing --wsrep-recover ... Using --wsrep-start-position when starting mysqld ... connection node_2; Performing --wsrep-recover ... Using --wsrep-start-position when starting mysqld ... connection node_1; include/diff_servers.inc [servers=1 2] connection node_1; DROP TABLE t1; CALL mtr.add_suppression("Skipped GCache ring buffer recovery"); include/assert_grep.inc [IST first seqno [24] not found from cache, falling back to SST] include/assert_grep.inc failed! assert_text: 'IST first seqno [24] not found from cache, falling back to SST' assert_file: '/home/stepan/mariadb/10.4/github/mysql-test/var/2/log/mysqld.1.err' assert_select: 'IST first seqno [24] not found from cache, falling back to SST' assert_match: '' assert_count: '1' assert_only_after: 'starting as process' number of matching lines: 0 Other logs .

            Decreasing the priority since it is failed only locally on HDD.

            stepan.patryshev Stepan Patryshev (Inactive) added a comment - Decreasing the priority since it is failed only locally on HDD.

            Closing since it does not fail on CI.

            stepan.patryshev Stepan Patryshev (Inactive) added a comment - Closing since it does not fail on CI.

            Reopened it since it failed on Jenkins, 10.3 ES, dddb4468aefb02789f978c303b26f557991f9678:

            CURRENT_TEST: galera.galera_gcache_recover_full_gcache
            mysqltest: In included file "./include/assert_grep.inc": 
            included from /home/jenkins/workspace/10.3e-MTR-GALERA/BuildType/Debug/label/sles-15/mysql-test/suite/galera/t/galera_gcache_recover_full_gcache.test at line 57:
            At line 152: assert_grep.inc failed.
             
            The result from queries just before the failure was:
            < snip >
            connection node_1;
            Performing --wsrep-recover ...
            Using --wsrep-start-position when starting mysqld ...
            connection node_2;
            Performing --wsrep-recover ...
            Using --wsrep-start-position when starting mysqld ...
            connection node_1;
            include/diff_servers.inc [servers=1 2]
            connection node_1;
            DROP TABLE t1;
            CALL mtr.add_suppression("Skipped GCache ring buffer recovery");
            include/assert_grep.inc [IST first seqno 2 not found from cache, falling back to SST]
            include/assert_grep.inc failed!
            assert_text: 'IST first seqno 2 not found from cache, falling back to SST'
            assert_file: '/var/tmp/mtr/1/log/mysqld.1.err'
            assert_select: 'IST first seqno 2 not found from cache, falling back to SST'
            assert_match: ''
            assert_count: '1'
            assert_only_after: 'starting as process'
            number of matching lines: 0
             
            More results from queries before failure can be found in /var/tmp/mtr/1/log/galera_gcache_recover_full_gcache.log
            

            stepan.patryshev Stepan Patryshev (Inactive) added a comment - - edited Reopened it since it failed on Jenkins, 10.3 ES, dddb4468aefb02789f978c303b26f557991f9678: CURRENT_TEST: galera.galera_gcache_recover_full_gcache mysqltest: In included file "./include/assert_grep.inc": included from /home/jenkins/workspace/10.3e-MTR-GALERA/BuildType/Debug/label/sles-15/mysql-test/suite/galera/t/galera_gcache_recover_full_gcache.test at line 57: At line 152: assert_grep.inc failed.   The result from queries just before the failure was: < snip > connection node_1; Performing --wsrep-recover ... Using --wsrep-start-position when starting mysqld ... connection node_2; Performing --wsrep-recover ... Using --wsrep-start-position when starting mysqld ... connection node_1; include/diff_servers.inc [servers=1 2] connection node_1; DROP TABLE t1; CALL mtr.add_suppression("Skipped GCache ring buffer recovery"); include/assert_grep.inc [IST first seqno 2 not found from cache, falling back to SST] include/assert_grep.inc failed! assert_text: 'IST first seqno 2 not found from cache, falling back to SST' assert_file: '/var/tmp/mtr/1/log/mysqld.1.err' assert_select: 'IST first seqno 2 not found from cache, falling back to SST' assert_match: '' assert_count: '1' assert_only_after: 'starting as process' number of matching lines: 0   More results from queries before failure can be found in /var/tmp/mtr/1/log/galera_gcache_recover_full_gcache.log

            It failed also on Jenkins, 10.5 ES.

            stdio.log:

            10.5.2-0 e7ca9ec6f6bd93883eeb35ebb2b09ffcbde35ad6, Build Debug, sles-12

            galera.galera_gcache_recover_full_gcache 'innodb' w2 [ fail ]
                    Test ended at 2020-04-17 02:30:13
             
            CURRENT_TEST: galera.galera_gcache_recover_full_gcache
            mysqltest: In included file "./include/assert_grep.inc": 
            included from /home/jenkins/workspace/10.5e-MTR-GALERA/BuildType/Debug/label/sles-12/mysql-test/suite/galera/t/galera_gcache_recover_full_gcache.test at line 65:
            At line 152: assert_grep.inc failed.
             
            The result from queries just before the failure was:
            < snip >
            connection node_1;
            Performing --wsrep-recover ...
            Using --wsrep-start-position when starting mysqld ...
            connection node_2;
            Performing --wsrep-recover ...
            Using --wsrep-start-position when starting mysqld ...
            connection node_1;
            include/diff_servers.inc [servers=1 2]
            connection node_1;
            DROP TABLE t1;
            CALL mtr.add_suppression("Skipped GCache ring buffer recovery");
            include/assert_grep.inc [IST first seqno [24] not found from cache, falling back to SST]
            include/assert_grep.inc failed!
            assert_text: 'IST first seqno [24] not found from cache, falling back to SST'
            assert_file: '/var/tmp/mtr/2/log/mysqld.1.err'
            assert_select: 'IST first seqno [24] not found from cache, falling back to SST'
            assert_match: ''
            assert_count: '1'
            assert_only_after: 'starting as process'
            number of matching lines: 0
             
            More results from queries before failure can be found in /var/tmp/mtr/2/log/galera_gcache_recover_full_gcache.log
             
             - saving '/var/tmp/mtr/2/log/galera.galera_gcache_recover_full_gcache-innodb/' to '/var/tmp/mtr/log/galera.galera_gcache_recover_full_gcache-innodb/'
            galera.galera_nbo_sst_slave 'innodb'     w6 [ fail ]
                    Test ended at 2020-04-17 02:30:14
            

            Server logs.

            stepan.patryshev Stepan Patryshev (Inactive) added a comment - It failed also on Jenkins, 10.5 ES. stdio.log : 10.5.2-0 e7ca9ec6f6bd93883eeb35ebb2b09ffcbde35ad6, Build Debug, sles-12 galera.galera_gcache_recover_full_gcache 'innodb' w2 [ fail ] Test ended at 2020-04-17 02:30:13   CURRENT_TEST: galera.galera_gcache_recover_full_gcache mysqltest: In included file "./include/assert_grep.inc": included from /home/jenkins/workspace/10.5e-MTR-GALERA/BuildType/Debug/label/sles-12/mysql-test/suite/galera/t/galera_gcache_recover_full_gcache.test at line 65: At line 152: assert_grep.inc failed.   The result from queries just before the failure was: < snip > connection node_1; Performing --wsrep-recover ... Using --wsrep-start-position when starting mysqld ... connection node_2; Performing --wsrep-recover ... Using --wsrep-start-position when starting mysqld ... connection node_1; include/diff_servers.inc [servers=1 2] connection node_1; DROP TABLE t1; CALL mtr.add_suppression("Skipped GCache ring buffer recovery"); include/assert_grep.inc [IST first seqno [24] not found from cache, falling back to SST] include/assert_grep.inc failed! assert_text: 'IST first seqno [24] not found from cache, falling back to SST' assert_file: '/var/tmp/mtr/2/log/mysqld.1.err' assert_select: 'IST first seqno [24] not found from cache, falling back to SST' assert_match: '' assert_count: '1' assert_only_after: 'starting as process' number of matching lines: 0   More results from queries before failure can be found in /var/tmp/mtr/2/log/galera_gcache_recover_full_gcache.log   - saving '/var/tmp/mtr/2/log/galera.galera_gcache_recover_full_gcache-innodb/' to '/var/tmp/mtr/log/galera.galera_gcache_recover_full_gcache-innodb/' galera.galera_nbo_sst_slave 'innodb' w6 [ fail ] Test ended at 2020-04-17 02:30:14 Server logs .

            It failed also on Jenkins, 10.2 ES 80101dc0473e07f66c9f7c9fadc51abe4681ba28, RelWithDebInfo,rhel-7.

            stepan.patryshev Stepan Patryshev (Inactive) added a comment - It failed also on Jenkins, 10.2 ES 80101dc0473e07f66c9f7c9fadc51abe4681ba28, RelWithDebInfo,rhel-7.

            It failed also on Jenkins, 10.4 ES, 00abe03ad1da3719e06f7112000a331ee2b6786a, Build RelWithDebInfo, ubuntu-2004.

            stepan.patryshev Stepan Patryshev (Inactive) added a comment - It failed also on Jenkins, 10.4 ES, 00abe03ad1da3719e06f7112000a331ee2b6786a, Build RelWithDebInfo, ubuntu-2004.

            People

              jplindst Jan Lindström (Inactive)
              stepan.patryshev Stepan Patryshev (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.