Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-18593

galera.galera_gcache_recover_full_gcache: Test failure: galera_gcache_recover_full_gcache.test: assert_grep.inc failed

Details

    Description

      MariaDB Server: 10.4.3-MariaDB-debug built from sources, branch=10.4, commit: ce6505f890956f81354269a991e69f20babae8e4.
      Galera4 lib: debug built from sources, commit: fbf047d76c4d95bc1ef1c465409efdca11256225.

      Run:

      ./mtr galera.galera_gcache_recover_full_gcache --force --max-test-fail=0 --repeat=3
      ./mtr --suite galera --force --big-test --max-test-fail=0
      

      stdout.log: (the whole file is attached: 190213_desktop_stdout.log)

      galera.galera_gcache_recover_full_gcache 'innodb' [ pass ]  56778
      galera.galera_gcache_recover_full_gcache 'innodb' [ 2 pass ]  53569
      galera.galera_gcache_recover_full_gcache 'innodb' [ 3 fail ]
       
      CURRENT_TEST: galera.galera_gcache_recover_full_gcache
      mysqltest: In included file "./include/assert_grep.inc": 
      included from /home/stepan/galera/git/10.4/server/mysql-test/suite/galera/t/galera_gcache_recover_full_gcache.test at line 65:
      At line 152: assert_grep.inc failed.
       
      The result from queries just before the failure was:
      < snip >
      connection node_1;
      Performing --wsrep-recover ...
      Using --wsrep-start-position when starting mysqld ...
      connection node_2;
      Performing --wsrep-recover ...
      Using --wsrep-start-position when starting mysqld ...
      connection node_1;
      include/diff_servers.inc [servers=1 2]
      connection node_1;
      DROP TABLE t1;
      CALL mtr.add_suppression("Skipped GCache ring buffer recovery");
      include/assert_grep.inc [IST first seqno [24] not found from cache, falling back to SST]
      include/assert_grep.inc failed!
      assert_text: 'IST first seqno [24] not found from cache, falling back to SST'
      assert_file: '/home/stepan/galera/git/10.4/server/mysql-test/var/log/mysqld.1.err'
      assert_select: 'IST first seqno [24] not found from cache, falling back to SST'
      assert_match: ''
      assert_count: '1'
      assert_only_after: 'starting as process'
      number of matching lines: 0
       
      Completed: Failed 1/3 tests, 66.67% were successful.
      

      See also attached 190215_galera.galera_gcache_recover_full_gcache.zip.

      Note: This test is failed sporadically.

      Attachments

        Issue Links

          Activity

            stepan.patryshev Stepan Patryshev (Inactive) created issue -
            stepan.patryshev Stepan Patryshev (Inactive) made changes -
            Field Original Value New Value
            stepan.patryshev Stepan Patryshev (Inactive) made changes -
            stepan.patryshev Stepan Patryshev (Inactive) made changes -
            Assignee Stepan Patryshev [ stepan.patryshev ] Jan Lindström [ jplindst ]

            Closing it as not reproduced.

            stepan.patryshev Stepan Patryshev (Inactive) added a comment - Closing it as not reproduced.
            stepan.patryshev Stepan Patryshev (Inactive) made changes -
            issue.field.resolutiondate 2019-03-29 11:43:12.0 2019-03-29 11:43:12.87
            stepan.patryshev Stepan Patryshev (Inactive) made changes -
            Fix Version/s 10.4.4 [ 23310 ]
            Fix Version/s 10.4 [ 22408 ]
            Resolution Cannot Reproduce [ 5 ]
            Status Open [ 1 ] Closed [ 6 ]
            stepan.patryshev Stepan Patryshev (Inactive) made changes -
            Attachment 191029_HDD_galera_suite.zip [ 49334 ]
            Attachment 191029_HDD_stdout.log [ 49335 ]

            Reopening it since it sporadically fails again on the local suite runs, but only on HDD and with --parallel=auto.

            MariaDB Version 10.4.9-MariaDB-debug: Repository: MariaDB/server; branch 10.4; Revision 9afbb1069ab4178741b76adefacd3f6a0ed51369; debug built from sources.

            Galera lib 26.4.3(r4535): Repository: MariaDB/galera; branch mariadb-4.x; Revision 752664dc3c7065d8e0c73ac99d0028a5f84eb250; debug built from sources.

            Run:

            ./mtr --suite galera --parallel=auto --testcase-timeout=90 --force --big-test --max-test-fail=0 --timer --timestamp --timediff
            

            stdout.log:

            191029 13:55:05 +0.042 galera.galera_gcache_recover_full_gcache 'innodb' w2 [ fail ]
                    Test ended at 2019-10-29 13:55:05
             
            CURRENT_TEST: galera.galera_gcache_recover_full_gcache
            mysqltest: In included file "./include/assert_grep.inc": 
            included from /home/stepan/mariadb/10.4/github/mysql-test/suite/galera/t/galera_gcache_recover_full_gcache.test at line 65:
            At line 152: assert_grep.inc failed.
             
            The result from queries just before the failure was:
            < snip >
            connection node_1;
            Performing --wsrep-recover ...
            Using --wsrep-start-position when starting mysqld ...
            connection node_2;
            Performing --wsrep-recover ...
            Using --wsrep-start-position when starting mysqld ...
            connection node_1;
            include/diff_servers.inc [servers=1 2]
            connection node_1;
            DROP TABLE t1;
            CALL mtr.add_suppression("Skipped GCache ring buffer recovery");
            include/assert_grep.inc [IST first seqno [24] not found from cache, falling back to SST]
            include/assert_grep.inc failed!
            assert_text: 'IST first seqno [24] not found from cache, falling back to SST'
            assert_file: '/home/stepan/mariadb/10.4/github/mysql-test/var/2/log/mysqld.1.err'
            assert_select: 'IST first seqno [24] not found from cache, falling back to SST'
            assert_match: ''
            assert_count: '1'
            assert_only_after: 'starting as process'
            number of matching lines: 0
            

            Other logs.

            stepan.patryshev Stepan Patryshev (Inactive) added a comment - - edited Reopening it since it sporadically fails again on the local suite runs, but only on HDD and with --parallel=auto. MariaDB Version 10.4.9-MariaDB-debug: Repository: MariaDB/server; branch 10.4; Revision 9afbb1069ab4178741b76adefacd3f6a0ed51369; debug built from sources. Galera lib 26.4.3(r4535): Repository: MariaDB/galera; branch mariadb-4.x; Revision 752664dc3c7065d8e0c73ac99d0028a5f84eb250; debug built from sources. Run: ./mtr --suite galera --parallel=auto --testcase-timeout=90 --force --big-test --max-test-fail=0 --timer --timestamp --timediff stdout.log : 191029 13:55:05 +0.042 galera.galera_gcache_recover_full_gcache 'innodb' w2 [ fail ] Test ended at 2019-10-29 13:55:05   CURRENT_TEST: galera.galera_gcache_recover_full_gcache mysqltest: In included file "./include/assert_grep.inc": included from /home/stepan/mariadb/10.4/github/mysql-test/suite/galera/t/galera_gcache_recover_full_gcache.test at line 65: At line 152: assert_grep.inc failed.   The result from queries just before the failure was: < snip > connection node_1; Performing --wsrep-recover ... Using --wsrep-start-position when starting mysqld ... connection node_2; Performing --wsrep-recover ... Using --wsrep-start-position when starting mysqld ... connection node_1; include/diff_servers.inc [servers=1 2] connection node_1; DROP TABLE t1; CALL mtr.add_suppression("Skipped GCache ring buffer recovery"); include/assert_grep.inc [IST first seqno [24] not found from cache, falling back to SST] include/assert_grep.inc failed! assert_text: 'IST first seqno [24] not found from cache, falling back to SST' assert_file: '/home/stepan/mariadb/10.4/github/mysql-test/var/2/log/mysqld.1.err' assert_select: 'IST first seqno [24] not found from cache, falling back to SST' assert_match: '' assert_count: '1' assert_only_after: 'starting as process' number of matching lines: 0 Other logs .
            stepan.patryshev Stepan Patryshev (Inactive) made changes -
            Resolution Cannot Reproduce [ 5 ]
            Status Closed [ 6 ] Stalled [ 10000 ]
            stepan.patryshev Stepan Patryshev (Inactive) made changes -
            Affects Version/s 10.4.9 [ 23906 ]
            stepan.patryshev Stepan Patryshev (Inactive) made changes -
            Fix Version/s 10.4 [ 22408 ]
            Fix Version/s 10.4.4 [ 23310 ]
            stepan.patryshev Stepan Patryshev (Inactive) made changes -
            stepan.patryshev Stepan Patryshev (Inactive) made changes -
            stepan.patryshev Stepan Patryshev (Inactive) made changes -
            Priority Major [ 3 ] Minor [ 4 ]

            Decreasing the priority since it is failed only locally on HDD.

            stepan.patryshev Stepan Patryshev (Inactive) added a comment - Decreasing the priority since it is failed only locally on HDD.

            Closing since it does not fail on CI.

            stepan.patryshev Stepan Patryshev (Inactive) added a comment - Closing since it does not fail on CI.
            stepan.patryshev Stepan Patryshev (Inactive) made changes -
            Fix Version/s N/A [ 14700 ]
            Fix Version/s 10.4 [ 22408 ]
            Assignee Jan Lindström [ jplindst ] Stepan Patryshev [ stepan.patryshev ]
            Resolution Cannot Reproduce [ 5 ]
            Status Stalled [ 10000 ] Closed [ 6 ]
            stepan.patryshev Stepan Patryshev (Inactive) made changes -

            Reopened it since it failed on Jenkins, 10.3 ES, dddb4468aefb02789f978c303b26f557991f9678:

            CURRENT_TEST: galera.galera_gcache_recover_full_gcache
            mysqltest: In included file "./include/assert_grep.inc": 
            included from /home/jenkins/workspace/10.3e-MTR-GALERA/BuildType/Debug/label/sles-15/mysql-test/suite/galera/t/galera_gcache_recover_full_gcache.test at line 57:
            At line 152: assert_grep.inc failed.
             
            The result from queries just before the failure was:
            < snip >
            connection node_1;
            Performing --wsrep-recover ...
            Using --wsrep-start-position when starting mysqld ...
            connection node_2;
            Performing --wsrep-recover ...
            Using --wsrep-start-position when starting mysqld ...
            connection node_1;
            include/diff_servers.inc [servers=1 2]
            connection node_1;
            DROP TABLE t1;
            CALL mtr.add_suppression("Skipped GCache ring buffer recovery");
            include/assert_grep.inc [IST first seqno 2 not found from cache, falling back to SST]
            include/assert_grep.inc failed!
            assert_text: 'IST first seqno 2 not found from cache, falling back to SST'
            assert_file: '/var/tmp/mtr/1/log/mysqld.1.err'
            assert_select: 'IST first seqno 2 not found from cache, falling back to SST'
            assert_match: ''
            assert_count: '1'
            assert_only_after: 'starting as process'
            number of matching lines: 0
             
            More results from queries before failure can be found in /var/tmp/mtr/1/log/galera_gcache_recover_full_gcache.log
            

            stepan.patryshev Stepan Patryshev (Inactive) added a comment - - edited Reopened it since it failed on Jenkins, 10.3 ES, dddb4468aefb02789f978c303b26f557991f9678: CURRENT_TEST: galera.galera_gcache_recover_full_gcache mysqltest: In included file "./include/assert_grep.inc": included from /home/jenkins/workspace/10.3e-MTR-GALERA/BuildType/Debug/label/sles-15/mysql-test/suite/galera/t/galera_gcache_recover_full_gcache.test at line 57: At line 152: assert_grep.inc failed.   The result from queries just before the failure was: < snip > connection node_1; Performing --wsrep-recover ... Using --wsrep-start-position when starting mysqld ... connection node_2; Performing --wsrep-recover ... Using --wsrep-start-position when starting mysqld ... connection node_1; include/diff_servers.inc [servers=1 2] connection node_1; DROP TABLE t1; CALL mtr.add_suppression("Skipped GCache ring buffer recovery"); include/assert_grep.inc [IST first seqno 2 not found from cache, falling back to SST] include/assert_grep.inc failed! assert_text: 'IST first seqno 2 not found from cache, falling back to SST' assert_file: '/var/tmp/mtr/1/log/mysqld.1.err' assert_select: 'IST first seqno 2 not found from cache, falling back to SST' assert_match: '' assert_count: '1' assert_only_after: 'starting as process' number of matching lines: 0   More results from queries before failure can be found in /var/tmp/mtr/1/log/galera_gcache_recover_full_gcache.log
            stepan.patryshev Stepan Patryshev (Inactive) made changes -
            Resolution Cannot Reproduce [ 5 ]
            Status Closed [ 6 ] Stalled [ 10000 ]
            stepan.patryshev Stepan Patryshev (Inactive) made changes -
            Affects Version/s 10.3 [ 22126 ]
            Affects Version/s 10.4.3 [ 23230 ]
            Affects Version/s 10.4.9 [ 23906 ]
            stepan.patryshev Stepan Patryshev (Inactive) made changes -
            Fix Version/s 10.3 [ 22126 ]
            Fix Version/s N/A [ 14700 ]
            stepan.patryshev Stepan Patryshev (Inactive) made changes -
            Priority Minor [ 4 ] Critical [ 2 ]
            stepan.patryshev Stepan Patryshev (Inactive) made changes -
            Assignee Stepan Patryshev [ stepan.patryshev ] Jan Lindström [ jplindst ]
            stepan.patryshev Stepan Patryshev (Inactive) made changes -
            stepan.patryshev Stepan Patryshev (Inactive) made changes -
            stepan.patryshev Stepan Patryshev (Inactive) made changes -
            Attachment 200417_galera_gcache_recover_full_gcache.zip [ 51481 ]
            Attachment 200417_stdout_10.5e.log [ 51482 ]

            It failed also on Jenkins, 10.5 ES.

            stdio.log:

            10.5.2-0 e7ca9ec6f6bd93883eeb35ebb2b09ffcbde35ad6, Build Debug, sles-12

            galera.galera_gcache_recover_full_gcache 'innodb' w2 [ fail ]
                    Test ended at 2020-04-17 02:30:13
             
            CURRENT_TEST: galera.galera_gcache_recover_full_gcache
            mysqltest: In included file "./include/assert_grep.inc": 
            included from /home/jenkins/workspace/10.5e-MTR-GALERA/BuildType/Debug/label/sles-12/mysql-test/suite/galera/t/galera_gcache_recover_full_gcache.test at line 65:
            At line 152: assert_grep.inc failed.
             
            The result from queries just before the failure was:
            < snip >
            connection node_1;
            Performing --wsrep-recover ...
            Using --wsrep-start-position when starting mysqld ...
            connection node_2;
            Performing --wsrep-recover ...
            Using --wsrep-start-position when starting mysqld ...
            connection node_1;
            include/diff_servers.inc [servers=1 2]
            connection node_1;
            DROP TABLE t1;
            CALL mtr.add_suppression("Skipped GCache ring buffer recovery");
            include/assert_grep.inc [IST first seqno [24] not found from cache, falling back to SST]
            include/assert_grep.inc failed!
            assert_text: 'IST first seqno [24] not found from cache, falling back to SST'
            assert_file: '/var/tmp/mtr/2/log/mysqld.1.err'
            assert_select: 'IST first seqno [24] not found from cache, falling back to SST'
            assert_match: ''
            assert_count: '1'
            assert_only_after: 'starting as process'
            number of matching lines: 0
             
            More results from queries before failure can be found in /var/tmp/mtr/2/log/galera_gcache_recover_full_gcache.log
             
             - saving '/var/tmp/mtr/2/log/galera.galera_gcache_recover_full_gcache-innodb/' to '/var/tmp/mtr/log/galera.galera_gcache_recover_full_gcache-innodb/'
            galera.galera_nbo_sst_slave 'innodb'     w6 [ fail ]
                    Test ended at 2020-04-17 02:30:14
            

            Server logs.

            stepan.patryshev Stepan Patryshev (Inactive) added a comment - It failed also on Jenkins, 10.5 ES. stdio.log : 10.5.2-0 e7ca9ec6f6bd93883eeb35ebb2b09ffcbde35ad6, Build Debug, sles-12 galera.galera_gcache_recover_full_gcache 'innodb' w2 [ fail ] Test ended at 2020-04-17 02:30:13   CURRENT_TEST: galera.galera_gcache_recover_full_gcache mysqltest: In included file "./include/assert_grep.inc": included from /home/jenkins/workspace/10.5e-MTR-GALERA/BuildType/Debug/label/sles-12/mysql-test/suite/galera/t/galera_gcache_recover_full_gcache.test at line 65: At line 152: assert_grep.inc failed.   The result from queries just before the failure was: < snip > connection node_1; Performing --wsrep-recover ... Using --wsrep-start-position when starting mysqld ... connection node_2; Performing --wsrep-recover ... Using --wsrep-start-position when starting mysqld ... connection node_1; include/diff_servers.inc [servers=1 2] connection node_1; DROP TABLE t1; CALL mtr.add_suppression("Skipped GCache ring buffer recovery"); include/assert_grep.inc [IST first seqno [24] not found from cache, falling back to SST] include/assert_grep.inc failed! assert_text: 'IST first seqno [24] not found from cache, falling back to SST' assert_file: '/var/tmp/mtr/2/log/mysqld.1.err' assert_select: 'IST first seqno [24] not found from cache, falling back to SST' assert_match: '' assert_count: '1' assert_only_after: 'starting as process' number of matching lines: 0   More results from queries before failure can be found in /var/tmp/mtr/2/log/galera_gcache_recover_full_gcache.log   - saving '/var/tmp/mtr/2/log/galera.galera_gcache_recover_full_gcache-innodb/' to '/var/tmp/mtr/log/galera.galera_gcache_recover_full_gcache-innodb/' galera.galera_nbo_sst_slave 'innodb' w6 [ fail ] Test ended at 2020-04-17 02:30:14 Server logs .
            stepan.patryshev Stepan Patryshev (Inactive) made changes -
            Affects Version/s 10.5.2 [ 24030 ]
            stepan.patryshev Stepan Patryshev (Inactive) made changes -
            Fix Version/s 10.5 [ 23123 ]

            It failed also on Jenkins, 10.2 ES 80101dc0473e07f66c9f7c9fadc51abe4681ba28, RelWithDebInfo,rhel-7.

            stepan.patryshev Stepan Patryshev (Inactive) added a comment - It failed also on Jenkins, 10.2 ES 80101dc0473e07f66c9f7c9fadc51abe4681ba28, RelWithDebInfo,rhel-7.
            stepan.patryshev Stepan Patryshev (Inactive) made changes -
            Affects Version/s 10.2 [ 14601 ]
            stepan.patryshev Stepan Patryshev (Inactive) made changes -
            Fix Version/s 10.2 [ 14601 ]

            It failed also on Jenkins, 10.4 ES, 00abe03ad1da3719e06f7112000a331ee2b6786a, Build RelWithDebInfo, ubuntu-2004.

            stepan.patryshev Stepan Patryshev (Inactive) added a comment - It failed also on Jenkins, 10.4 ES, 00abe03ad1da3719e06f7112000a331ee2b6786a, Build RelWithDebInfo, ubuntu-2004.
            stepan.patryshev Stepan Patryshev (Inactive) made changes -
            Affects Version/s 10.4 [ 22408 ]
            stepan.patryshev Stepan Patryshev (Inactive) made changes -
            Fix Version/s 10.4 [ 22408 ]
            jplindst Jan Lindström (Inactive) made changes -
            Status Stalled [ 10000 ] In Progress [ 3 ]
            jplindst Jan Lindström (Inactive) made changes -
            issue.field.resolutiondate 2020-10-06 09:14:13.0 2020-10-06 09:14:13.419
            jplindst Jan Lindström (Inactive) made changes -
            Fix Version/s 10.2.34 [ 24505 ]
            Fix Version/s 10.3.25 [ 24506 ]
            Fix Version/s 10.4.15 [ 24507 ]
            Fix Version/s 10.5.6 [ 24508 ]
            Fix Version/s 10.6.0 [ 24431 ]
            Fix Version/s 10.2 [ 14601 ]
            Fix Version/s 10.3 [ 22126 ]
            Fix Version/s 10.4 [ 22408 ]
            Fix Version/s 10.5 [ 23123 ]
            Resolution Fixed [ 1 ]
            Status In Progress [ 3 ] Closed [ 6 ]
            ralf.gebhardt Ralf Gebhardt made changes -
            Fix Version/s 10.5.7 [ 25019 ]
            ralf.gebhardt Ralf Gebhardt made changes -
            Fix Version/s 10.5.6 [ 24508 ]
            ralf.gebhardt Ralf Gebhardt made changes -
            Fix Version/s 10.4.16 [ 25020 ]
            ralf.gebhardt Ralf Gebhardt made changes -
            Fix Version/s 10.4.15 [ 24507 ]
            ralf.gebhardt Ralf Gebhardt made changes -
            Fix Version/s 10.3.26 [ 25021 ]
            ralf.gebhardt Ralf Gebhardt made changes -
            Fix Version/s 10.3.25 [ 24506 ]
            ralf.gebhardt Ralf Gebhardt made changes -
            Fix Version/s 10.2.35 [ 25022 ]
            ralf.gebhardt Ralf Gebhardt made changes -
            Fix Version/s 10.2.34 [ 24505 ]
            stepan.patryshev Stepan Patryshev (Inactive) made changes -
            stepan.patryshev Stepan Patryshev (Inactive) made changes -
            serg Sergei Golubchik made changes -
            Workflow MariaDB v3 [ 92626 ] MariaDB v4 [ 155711 ]

            People

              jplindst Jan Lindström (Inactive)
              stepan.patryshev Stepan Patryshev (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.