[MDEV-26705] MTR Easily fails with 'File exists' on multiple runs of the same (any) testcase when using --mem Created: 2021-09-28  Updated: 2021-09-28  Resolved: 2021-09-28

Status: Closed
Project: MariaDB Server
Component/s: Tests
Affects Version/s: N/A
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Roel Van de Paar Assignee: Roel Van de Paar
Resolution: Not a Bug Votes: 0
Labels: affects-tests


 Description   

Start (any) repeated test in MTR (using parallel=1), for example main/create.test:

/test/MD180921-mariadb-10.7.0-linux-x86_64-dbg/mysql-test$ ./mysql-test-run --repeat=99999 --parallel=1 --mem --force --retry=99999 --retry-failure=99999 --suite=main create | tee long_run.txt

Or you can use /dev/shm (same outcome):

MTR_MEM=/dev/shm ./mysql-test-run --repeat=99999 --parallel=1 --mem --force --retry=99999 --retry-failure=99999 --suite=main create | tee long_run.txt

And it will quickly (20 to 500 testcase runs) fail with:

10.7.0 d552e092c9f3e20da078d1b62b976f629f73d3a4 (Debug)

 
...
 
main.create                              [ 13 pass ]   5546
 
main.create                              [ 14 pass ]   5549
 
main.create                              [ 15 pass ]   5580
 
main.create                              [ 16 pass ]   5755
 
========================================
 
mkdir failed
 
err: 'File exists'
 
path: '/test/MD180921-mariadb-10.7.0-linux-x86_64-dbg/mysql-test/var'
 
dir: 'var'
 
...

Issue pops up frequently around 15-30 repeats. Seems like a (sporadic) race condition.

 Comments   
Comment by Marko Mäkelä [ 2021-09-28 ]

I believe that the claim "any testcase" is exaggerated. Just recently, related to MDEV-26450, I ran thousands of repetitions of the test innodb.undo_truncate. Earlier, related to MDEV-25506, I ran some DDL tests hundreds or thousands of times.

That said, there are (too) many tests that fail when run repeatedly. Mostly they should be outside the InnoDB test suites.

Comment by Roel Van de Paar [ 2021-09-28 ]

marko On further checking, I found the issue only appears when using --mem. Thanks

Comment by Roel Van de Paar [ 2021-09-28 ]

marko I do not believe that 'any testcase' for this particular MTR issue 'File exists' is exaggerated. For example,

MTR_MEM=; ./mysql-test-run --repeat=99999 --parallel=1 --mem --retry=99999 --retry-failure=99999 --suite=innodb innodb | tee long_run.txt

Leads to:

10.7.0 d552e092c9f3e20da078d1b62b976f629f73d3a4 (Debug)

 
innodb.innodb 'innodb'                   [ 43 pass ]   4296
 
innodb.innodb 'innodb'                   [ 44 pass ]   4219
 
innodb.innodb 'innodb'                   [ 45 pass ]   4124
 
innodb.innodb 'innodb'                   [ 46 pass ]   4005
 
innodb.innodb 'innodb'                   [ 47 pass ]   4249
 
innodb.innodb 'innodb'                   [ 48 pass ]   4333
 
========================================
 
mkdir failed
 
err: 'File exists'
 
path: '/test/MD180921-mariadb-10.7.0-linux-x86_64-dbg/mysql-test/var'
 
dir: 'var'
 
----------------------------------

Comment by Vladislav Vaintroub [ 2021-09-28 ]

I think your Linux got some problems. What would poor assignee do with it?

Comment by Roel Van de Paar [ 2021-09-28 ]

wlad Correct. One of the server monitoring scripts which was cleaning up /dev/shm had a shortcoming to avoid MTR var_... directories. Will fix that one. Thanks.

Comment by Roel Van de Paar [ 2021-09-28 ]

Fixed in tmpfs_cleanup.sh called by disk space monitor (ds).

Generated at Thu Feb 08 09:47:19 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.