[MDEV-202] Overlays do not support nested test suites which exist in MTR Created: 2012-03-25  Updated: 2012-04-26  Resolved: 2012-04-26

Status: Closed
Project: MariaDB Server
Component/s: None
Affects Version/s: 5.5.21
Fix Version/s: 5.5.23

Type: Bug Priority: Minor
Reporter: Elena Stepanova Assignee: Sergei Golubchik
Resolution: Fixed Votes: 0
Labels: None

Issue Links:
Blocks
blocks MDEV-11 Generic storage engine test suite Closed

 Description   

MTR supports nested test suites, such as suite/engines/funcs.
It is run by providing the suite value as

--suite=engines/funcs.

Overlays do not support this. If I create storage/<engine>/mysql-test/engines/funcs and then run

perl mtr --suite=engines/funcs- 

it dies at lib/mtr_cases.pm line 383.

It seems that the change below fixes it, although I might fail to predict some deeper consequences.

@@ -380,7 +380,7 @@
     local %file_combinations = ();
     local %file_in_overlay = ();
 
-    die unless m@/(?:storage|plugin)/(\w+)/mysql-test/\w+$@;
+    die "Overlay not found" unless m@/(?:storage|plugin)/(\w+)/mysql-test/[\w\\\/]+$@;
     next unless defined $over and ($over eq '' or $over eq $1);
     push @cases, 
     # don't add cases that take *all* data from the parent suite

Please replace the death message with something appropriate, but it's better to say something than die quietly.



 Comments   
Comment by Elena Stepanova [ 2012-03-25 ]

Also, for the same nested suites, the following works in MySQL, but makes MariaDB version die at the same line 383 (regardless whether I have overlays for engines subsuites or not):

perl ./mtr --suite=engines/*

It is not perfect in MySQL either, as it produces dirty output like

engines/*.db_create_drop [ pass ] 24

(see the asterisk instead of the subsuite name). Still, dying is not good.
Otherwise, I am going to create a separate feature request for providing a convenient way to launch nested subsuites.

Comment by Elena Stepanova [ 2012-03-30 ]

Since this report itself turned out to be a feature request, I will put my suggestions here.

1. Durability

Overlays should sustain any level of nesting, as long as it is supported in MTR itself. By "sustain" I mean that MTR should not die running an internal suite (with subsuites) due to the presence of its overlays, full or partial. Lets say we have in MTR
mysql-test/suite/engine/test1.test
mysql-test/suite/engine/basic/test2.test
mysql-test/suite/engine/basic/types/test3.test

Now we can run
perl ./mtr --suite=engine
perl ./mtr --suite=engine/basic
perl ./mtr --suite=engine/basic/types

We should still be able to do so, no matter whether an engine or plugin has any of
mysql-test/suite/engine
mysql-test/suite/engine/basic
mysql-test/suite/engine/basic/types
folders.

Now it is not so – if I have storage/my_engine/mysql-test/engine/basic folder and try to run MTR as
perl ./mtr --suite=engine/basic
(even without the dash), it dies.

2. Supported level of nesting

At the same time, I don't think it's strictly necessary to actually support the same level of nesting in overlays as it is done in MTR. It is somewhat nice to have, mainly to avoid confusion, but if there is any problem implementing it, I believe that 2 levels (parent and child suite) should be enough for the most real-life purposes.

3. Syntax

I do find it important whether nested overlays should be triggered as
--suite=engine-my_engine/basic or
--suite=engine/basic-my_engine
Whichever is easier to implement should be fine.

It would be good to avoid making a user launch it as
--suite=engine-my_engine/basic-my_engine
as it seems redundant (especially if we decide to support more than 2 levels), but it is not critical either.

4. Add-ons

I previously suggested to make MTR execute all subsuites of 'engine' if it is run as
--suite=engine
but thinking about it, it is probably not a good idea, since the 'engine' suite can have tests of its own, and if we implement such a feature, it will be impossible to execute only tests from the 'engine' suite, without subsuites.

A nice-to-have (but low priority) feature could be to improve currently half-supported syntax --suite=engine/*, to print correct test full names in the output, and not to die on middle-level expansions.
To exend the example above, now, if we have

mysql-test/suite/engine/test1.test
mysql-test/suite/engine/basic/test2.test
mysql-test/suite/engine/basic/types/test3.test

executing mtr --suite=engine/* would run test2, and mtr --suite=engine/basic/* would run test3. But it's a corner case only that works – if there are more than one subsuite on either level, MTR would die, e.g. if we have

mysql-test/suite/engine/test1.test
mysql-test/suite/engine/basic/test2.test
mysql-test/suite/engine/basic/types/test3.test
mysql-test/suite/engine/basic/indexes/test4.test

then mtr --suite=engine/* would still run test2, but mtr --suite=engine/basic/* would die.

More consistent behavior would be nice: -suite=suite1/* to execute all subsuites of suite1 (but not their subsuites), and --suite=suite1-my_engine/* to work accordingly, as well as --suite=suite1/*.

Comment by Elena Stepanova [ 2012-04-25 ]

OK to push to 5.5. I will keep trying the new feature while working on MDEV-11.

Comment by Sergei Golubchik [ 2012-04-26 ]

pushed into 5.5

Generated at Thu Feb 08 06:27:00 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.