[MDEV-18593] galera.galera_gcache_recover_full_gcache: Test failure: galera_gcache_recover_full_gcache.test: assert_grep.inc failed Created: 2019-02-15  Updated: 2020-10-29  Resolved: 2020-10-06

Status: Closed
Project: MariaDB Server
Component/s: Galera, Tests
Affects Version/s: 10.5.2, 10.2, 10.3, 10.4
Fix Version/s: 10.2.35, 10.3.26, 10.4.16, 10.5.7, 10.6.0

Type: Bug Priority: Critical
Reporter: Stepan Patryshev (Inactive) Assignee: Jan Lindström (Inactive)
Resolution: Fixed Votes: 0
Labels: galera_4, tests
Environment:

OS: CentOS Linux release 7.6.1810 (Core).

Attachments: Text File 190213_desktop_stdout.log     Zip Archive 190215_galera.galera_gcache_recover_full_gcache.zip     Zip Archive 191029_HDD_galera_suite.zip     Text File 191029_HDD_stdout.log     Zip Archive 200417_galera_gcache_recover_full_gcache.zip     Text File 200417_stdout_10.5e.log    
Issue Links:
Relates
relates to MDEV-18111 Galera 4: run galera test suite Closed

 Description   

MariaDB Server: 10.4.3-MariaDB-debug built from sources, branch=10.4, commit: ce6505f890956f81354269a991e69f20babae8e4.
Galera4 lib: debug built from sources, commit: fbf047d76c4d95bc1ef1c465409efdca11256225.

Run: 

./mtr galera.galera_gcache_recover_full_gcache --force --max-test-fail=0 --repeat=3
 
./mtr --suite galera --force --big-test --max-test-fail=0

stdout.log: (the whole file is attached: 190213_desktop_stdout.log)

galera.galera_gcache_recover_full_gcache 'innodb' [ pass ]  56778
 
galera.galera_gcache_recover_full_gcache 'innodb' [ 2 pass ]  53569
 
galera.galera_gcache_recover_full_gcache 'innodb' [ 3 fail ]
 
 
 
CURRENT_TEST: galera.galera_gcache_recover_full_gcache
 
mysqltest: In included file "./include/assert_grep.inc": 
included from /home/stepan/galera/git/10.4/server/mysql-test/suite/galera/t/galera_gcache_recover_full_gcache.test at line 65:
 
At line 152: assert_grep.inc failed.
 
 
 
The result from queries just before the failure was:
 
< snip >
 
connection node_1;
 
Performing --wsrep-recover ...
 
Using --wsrep-start-position when starting mysqld ...
 
connection node_2;
 
Performing --wsrep-recover ...
 
Using --wsrep-start-position when starting mysqld ...
 
connection node_1;
 
include/diff_servers.inc [servers=1 2]
 
connection node_1;
 
DROP TABLE t1;
 
CALL mtr.add_suppression("Skipped GCache ring buffer recovery");
 
include/assert_grep.inc [IST first seqno [24] not found from cache, falling back to SST]
 
include/assert_grep.inc failed!
 
assert_text: 'IST first seqno [24] not found from cache, falling back to SST'
 
assert_file: '/home/stepan/galera/git/10.4/server/mysql-test/var/log/mysqld.1.err'
 
assert_select: 'IST first seqno [24] not found from cache, falling back to SST'
 
assert_match: ''
 
assert_count: '1'
 
assert_only_after: 'starting as process'
 
number of matching lines: 0
 
 
 
Completed: Failed 1/3 tests, 66.67% were successful.

See also attached 190215_galera.galera_gcache_recover_full_gcache.zip.

Note: This test is failed sporadically.



 Comments   
Comment by Stepan Patryshev (Inactive) [ 2019-03-29 ]

Closing it as not reproduced.

Comment by Stepan Patryshev (Inactive) [ 2019-10-31 ]

Reopening it since it sporadically fails again on the local suite runs, but only on HDD and with --parallel=auto.

MariaDB Version 10.4.9-MariaDB-debug: Repository: MariaDB/server; branch 10.4; Revision 9afbb1069ab4178741b76adefacd3f6a0ed51369; debug built from sources.

Galera lib 26.4.3(r4535): Repository: MariaDB/galera; branch mariadb-4.x; Revision 752664dc3c7065d8e0c73ac99d0028a5f84eb250; debug built from sources.

Run:

./mtr --suite galera --parallel=auto --testcase-timeout=90 --force --big-test --max-test-fail=0 --timer --timestamp --timediff

stdout.log:

191029 13:55:05 +0.042 galera.galera_gcache_recover_full_gcache 'innodb' w2 [ fail ]
 
        Test ended at 2019-10-29 13:55:05
 
 
 
CURRENT_TEST: galera.galera_gcache_recover_full_gcache
 
mysqltest: In included file "./include/assert_grep.inc": 
included from /home/stepan/mariadb/10.4/github/mysql-test/suite/galera/t/galera_gcache_recover_full_gcache.test at line 65:
 
At line 152: assert_grep.inc failed.
 
 
 
The result from queries just before the failure was:
 
< snip >
 
connection node_1;
 
Performing --wsrep-recover ...
 
Using --wsrep-start-position when starting mysqld ...
 
connection node_2;
 
Performing --wsrep-recover ...
 
Using --wsrep-start-position when starting mysqld ...
 
connection node_1;
 
include/diff_servers.inc [servers=1 2]
 
connection node_1;
 
DROP TABLE t1;
 
CALL mtr.add_suppression("Skipped GCache ring buffer recovery");
 
include/assert_grep.inc [IST first seqno [24] not found from cache, falling back to SST]
 
include/assert_grep.inc failed!
 
assert_text: 'IST first seqno [24] not found from cache, falling back to SST'
 
assert_file: '/home/stepan/mariadb/10.4/github/mysql-test/var/2/log/mysqld.1.err'
 
assert_select: 'IST first seqno [24] not found from cache, falling back to SST'
 
assert_match: ''
 
assert_count: '1'
 
assert_only_after: 'starting as process'
 
number of matching lines: 0

Other logs.

Comment by Stepan Patryshev (Inactive) [ 2020-01-03 ]

Decreasing the priority since it is failed only locally on HDD.

Comment by Stepan Patryshev (Inactive) [ 2020-02-10 ]

Closing since it does not fail on CI.

Comment by Stepan Patryshev (Inactive) [ 2020-04-13 ]

Reopened it since it failed on Jenkins, 10.3 ES, dddb4468aefb02789f978c303b26f557991f9678:

CURRENT_TEST: galera.galera_gcache_recover_full_gcache
 
mysqltest: In included file "./include/assert_grep.inc": 
included from /home/jenkins/workspace/10.3e-MTR-GALERA/BuildType/Debug/label/sles-15/mysql-test/suite/galera/t/galera_gcache_recover_full_gcache.test at line 57:
 
At line 152: assert_grep.inc failed.
 
 
 
The result from queries just before the failure was:
 
< snip >
 
connection node_1;
 
Performing --wsrep-recover ...
 
Using --wsrep-start-position when starting mysqld ...
 
connection node_2;
 
Performing --wsrep-recover ...
 
Using --wsrep-start-position when starting mysqld ...
 
connection node_1;
 
include/diff_servers.inc [servers=1 2]
 
connection node_1;
 
DROP TABLE t1;
 
CALL mtr.add_suppression("Skipped GCache ring buffer recovery");
 
include/assert_grep.inc [IST first seqno 2 not found from cache, falling back to SST]
 
include/assert_grep.inc failed!
 
assert_text: 'IST first seqno 2 not found from cache, falling back to SST'
 
assert_file: '/var/tmp/mtr/1/log/mysqld.1.err'
 
assert_select: 'IST first seqno 2 not found from cache, falling back to SST'
 
assert_match: ''
 
assert_count: '1'
 
assert_only_after: 'starting as process'
 
number of matching lines: 0
 
 
 
More results from queries before failure can be found in /var/tmp/mtr/1/log/galera_gcache_recover_full_gcache.log

Comment by Stepan Patryshev (Inactive) [ 2020-04-28 ]

It failed also on Jenkins, 10.5 ES.

stdio.log:

10.5.2-0 e7ca9ec6f6bd93883eeb35ebb2b09ffcbde35ad6, Build Debug, sles-12

 
galera.galera_gcache_recover_full_gcache 'innodb' w2 [ fail ]
 
        Test ended at 2020-04-17 02:30:13
 
 
 
CURRENT_TEST: galera.galera_gcache_recover_full_gcache
 
mysqltest: In included file "./include/assert_grep.inc": 
included from /home/jenkins/workspace/10.5e-MTR-GALERA/BuildType/Debug/label/sles-12/mysql-test/suite/galera/t/galera_gcache_recover_full_gcache.test at line 65:
 
At line 152: assert_grep.inc failed.
 
 
 
The result from queries just before the failure was:
 
< snip >
 
connection node_1;
 
Performing --wsrep-recover ...
 
Using --wsrep-start-position when starting mysqld ...
 
connection node_2;
 
Performing --wsrep-recover ...
 
Using --wsrep-start-position when starting mysqld ...
 
connection node_1;
 
include/diff_servers.inc [servers=1 2]
 
connection node_1;
 
DROP TABLE t1;
 
CALL mtr.add_suppression("Skipped GCache ring buffer recovery");
 
include/assert_grep.inc [IST first seqno [24] not found from cache, falling back to SST]
 
include/assert_grep.inc failed!
 
assert_text: 'IST first seqno [24] not found from cache, falling back to SST'
 
assert_file: '/var/tmp/mtr/2/log/mysqld.1.err'
 
assert_select: 'IST first seqno [24] not found from cache, falling back to SST'
 
assert_match: ''
 
assert_count: '1'
 
assert_only_after: 'starting as process'
 
number of matching lines: 0
 
 
 
More results from queries before failure can be found in /var/tmp/mtr/2/log/galera_gcache_recover_full_gcache.log
 
 
 
 - saving '/var/tmp/mtr/2/log/galera.galera_gcache_recover_full_gcache-innodb/' to '/var/tmp/mtr/log/galera.galera_gcache_recover_full_gcache-innodb/'
 
galera.galera_nbo_sst_slave 'innodb'     w6 [ fail ]
 
        Test ended at 2020-04-17 02:30:14

Server logs.

Comment by Stepan Patryshev (Inactive) [ 2020-05-11 ]

It failed also on Jenkins, 10.2 ES 80101dc0473e07f66c9f7c9fadc51abe4681ba28, RelWithDebInfo,rhel-7.

Comment by Stepan Patryshev (Inactive) [ 2020-05-15 ]

It failed also on Jenkins, 10.4 ES, 00abe03ad1da3719e06f7112000a331ee2b6786a, Build RelWithDebInfo, ubuntu-2004.

Generated at Thu Feb 08 08:45:17 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.