Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-202

Overlays do not support nested test suites which exist in MTR

Details

    • Bug
    • Status: Closed (View Workflow)
    • Minor
    • Resolution: Fixed
    • 5.5.21
    • 5.5.23
    • None
    • None

    Description

      MTR supports nested test suites, such as suite/engines/funcs.
      It is run by providing the suite value as

      --suite=engines/funcs.

      Overlays do not support this. If I create storage/<engine>/mysql-test/engines/funcs and then run

      perl mtr --suite=engines/funcs- 

      it dies at lib/mtr_cases.pm line 383.

      It seems that the change below fixes it, although I might fail to predict some deeper consequences.

      @@ -380,7 +380,7 @@
           local %file_combinations = ();
           local %file_in_overlay = ();
       
      -    die unless m@/(?:storage|plugin)/(\w+)/mysql-test/\w+$@;
      +    die "Overlay not found" unless m@/(?:storage|plugin)/(\w+)/mysql-test/[\w\\\/]+$@;
           next unless defined $over and ($over eq '' or $over eq $1);
           push @cases, 
           # don't add cases that take *all* data from the parent suite

      Please replace the death message with something appropriate, but it's better to say something than die quietly.

      Attachments

        Issue Links

          Activity

            elenst Elena Stepanova added a comment - - edited

            Also, for the same nested suites, the following works in MySQL, but makes MariaDB version die at the same line 383 (regardless whether I have overlays for engines subsuites or not):

            perl ./mtr --suite=engines/*

            It is not perfect in MySQL either, as it produces dirty output like

            engines/*.db_create_drop [ pass ] 24

            (see the asterisk instead of the subsuite name). Still, dying is not good.
            Otherwise, I am going to create a separate feature request for providing a convenient way to launch nested subsuites.

            elenst Elena Stepanova added a comment - - edited Also, for the same nested suites, the following works in MySQL, but makes MariaDB version die at the same line 383 (regardless whether I have overlays for engines subsuites or not): perl ./mtr --suite=engines/* It is not perfect in MySQL either, as it produces dirty output like engines/*.db_create_drop [ pass ] 24 (see the asterisk instead of the subsuite name). Still, dying is not good. Otherwise, I am going to create a separate feature request for providing a convenient way to launch nested subsuites.

            Since this report itself turned out to be a feature request, I will put my suggestions here.

            1. Durability

            Overlays should sustain any level of nesting, as long as it is supported in MTR itself. By "sustain" I mean that MTR should not die running an internal suite (with subsuites) due to the presence of its overlays, full or partial. Lets say we have in MTR
            mysql-test/suite/engine/test1.test
            mysql-test/suite/engine/basic/test2.test
            mysql-test/suite/engine/basic/types/test3.test

            Now we can run
            perl ./mtr --suite=engine
            perl ./mtr --suite=engine/basic
            perl ./mtr --suite=engine/basic/types

            We should still be able to do so, no matter whether an engine or plugin has any of
            mysql-test/suite/engine
            mysql-test/suite/engine/basic
            mysql-test/suite/engine/basic/types
            folders.

            Now it is not so – if I have storage/my_engine/mysql-test/engine/basic folder and try to run MTR as
            perl ./mtr --suite=engine/basic
            (even without the dash), it dies.

            2. Supported level of nesting

            At the same time, I don't think it's strictly necessary to actually support the same level of nesting in overlays as it is done in MTR. It is somewhat nice to have, mainly to avoid confusion, but if there is any problem implementing it, I believe that 2 levels (parent and child suite) should be enough for the most real-life purposes.

            3. Syntax

            I do find it important whether nested overlays should be triggered as
            --suite=engine-my_engine/basic or
            --suite=engine/basic-my_engine
            Whichever is easier to implement should be fine.

            It would be good to avoid making a user launch it as
            --suite=engine-my_engine/basic-my_engine
            as it seems redundant (especially if we decide to support more than 2 levels), but it is not critical either.

            4. Add-ons

            I previously suggested to make MTR execute all subsuites of 'engine' if it is run as
            --suite=engine
            but thinking about it, it is probably not a good idea, since the 'engine' suite can have tests of its own, and if we implement such a feature, it will be impossible to execute only tests from the 'engine' suite, without subsuites.

            A nice-to-have (but low priority) feature could be to improve currently half-supported syntax --suite=engine/*, to print correct test full names in the output, and not to die on middle-level expansions.
            To exend the example above, now, if we have

            mysql-test/suite/engine/test1.test
            mysql-test/suite/engine/basic/test2.test
            mysql-test/suite/engine/basic/types/test3.test

            executing mtr --suite=engine/* would run test2, and mtr --suite=engine/basic/* would run test3. But it's a corner case only that works – if there are more than one subsuite on either level, MTR would die, e.g. if we have

            mysql-test/suite/engine/test1.test
            mysql-test/suite/engine/basic/test2.test
            mysql-test/suite/engine/basic/types/test3.test
            mysql-test/suite/engine/basic/indexes/test4.test

            then mtr --suite=engine/* would still run test2, but mtr --suite=engine/basic/* would die.

            More consistent behavior would be nice: -suite=suite1/* to execute all subsuites of suite1 (but not their subsuites), and --suite=suite1-my_engine/* to work accordingly, as well as --suite=suite1/*.

            elenst Elena Stepanova added a comment - Since this report itself turned out to be a feature request, I will put my suggestions here. 1. Durability Overlays should sustain any level of nesting, as long as it is supported in MTR itself. By "sustain" I mean that MTR should not die running an internal suite (with subsuites) due to the presence of its overlays, full or partial. Lets say we have in MTR mysql-test/suite/engine/test1.test mysql-test/suite/engine/basic/test2.test mysql-test/suite/engine/basic/types/test3.test Now we can run perl ./mtr --suite=engine perl ./mtr --suite=engine/basic perl ./mtr --suite=engine/basic/types We should still be able to do so, no matter whether an engine or plugin has any of mysql-test/suite/engine mysql-test/suite/engine/basic mysql-test/suite/engine/basic/types folders. Now it is not so – if I have storage/my_engine/mysql-test/engine/basic folder and try to run MTR as perl ./mtr --suite=engine/basic (even without the dash), it dies. 2. Supported level of nesting At the same time, I don't think it's strictly necessary to actually support the same level of nesting in overlays as it is done in MTR. It is somewhat nice to have, mainly to avoid confusion, but if there is any problem implementing it, I believe that 2 levels (parent and child suite) should be enough for the most real-life purposes. 3. Syntax I do find it important whether nested overlays should be triggered as --suite=engine-my_engine/basic or --suite=engine/basic-my_engine Whichever is easier to implement should be fine. It would be good to avoid making a user launch it as --suite=engine-my_engine/basic-my_engine as it seems redundant (especially if we decide to support more than 2 levels), but it is not critical either. 4. Add-ons I previously suggested to make MTR execute all subsuites of 'engine' if it is run as --suite=engine but thinking about it, it is probably not a good idea, since the 'engine' suite can have tests of its own, and if we implement such a feature, it will be impossible to execute only tests from the 'engine' suite, without subsuites. A nice-to-have (but low priority) feature could be to improve currently half-supported syntax --suite=engine/*, to print correct test full names in the output, and not to die on middle-level expansions. To exend the example above, now, if we have mysql-test/suite/engine/test1.test mysql-test/suite/engine/basic/test2.test mysql-test/suite/engine/basic/types/test3.test executing mtr --suite=engine/* would run test2, and mtr --suite=engine/basic/* would run test3. But it's a corner case only that works – if there are more than one subsuite on either level, MTR would die, e.g. if we have mysql-test/suite/engine/test1.test mysql-test/suite/engine/basic/test2.test mysql-test/suite/engine/basic/types/test3.test mysql-test/suite/engine/basic/indexes/test4.test then mtr --suite=engine/* would still run test2, but mtr --suite=engine/basic/* would die. More consistent behavior would be nice: - suite=suite1/* to execute all subsuites of suite1 (but not their subsuites), and --suite=suite1-my_engine/* to work accordingly, as well as --suite=suite1 /*.

            OK to push to 5.5. I will keep trying the new feature while working on MDEV-11.

            elenst Elena Stepanova added a comment - OK to push to 5.5. I will keep trying the new feature while working on MDEV-11 .

            pushed into 5.5

            serg Sergei Golubchik added a comment - pushed into 5.5

            People

              serg Sergei Golubchik
              elenst Elena Stepanova
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.