Details

    Description

      Hi,
      we have a nice tiny feature in Fedora and RHEL, I want to share with other distros.

      When we solve some troubles with tests, we list them with comments to a standalone file. Then, we add '--skip-test-list' option, to the testsuite.
      It help us quickly enable / disable test in 'developement' time, and use different lists for different architectures.

      The code is already in place, I only offer a better documentation and a suggestions to ship that file ( mysql-test/skipped-tests.list ) by default (even if it would be empty - since you disable / enable tests directly in each test file)

      Take a glimpse at the attachement. It is a patch we use and it is written for Red Hat, but adding macros to the paths should solve it.
      Take a look and tell me, what you think.

      I don't know how others, but RPM distros - like SUSE - could look into Fedora specfile and ease their maintainers lifes with this

      Attachments

        Activity

          elenst Elena Stepanova added a comment - - edited

          Actually, we already have the file itself in 10.0/10.1, it's called unstable-tests, and it's far from empty. The file is used when a reliable subset of tests is needed (e.g. when tests are a part of the build process), to avoid sporadic and irrelevant failures. Naturally, something can always be added to it based on distribution or architecture specifics (it is also known to happen).
          The list is updated before every release. It uses pessimistic approach – tests are unstable unless are proven to be stable enough. Basic rules for the list are these:

          For 10.2 it is not maintained yet, but it will be by the time 10.2 goes GA.

          Inclusion:

          • tests which have failed in buildbot in the corresponding branch since the previous release go to the list (and bug reports are created for them);
          • tests for which we have open bug reports remain on the list until the bug reports are closed;
          • tests which have been added or modified since the previous release, directly or via include files, go to the list (trivial modifications can be ignored on case by case basis).

          Exclusion:

          • tests for which we do not have open bugs and which have not been modified and have not failed since the previous release (sometimes over two release cycles, depending on their length) are excluded from the list.

          On an additional note, if you are want to enable/disable tests really quickly, depending on your process, it might be easier to use --skip-test rather than --skip-test-list.

          elenst Elena Stepanova added a comment - - edited Actually, we already have the file itself in 10.0/10.1, it's called unstable-tests , and it's far from empty. The file is used when a reliable subset of tests is needed (e.g. when tests are a part of the build process), to avoid sporadic and irrelevant failures. Naturally, something can always be added to it based on distribution or architecture specifics (it is also known to happen). The list is updated before every release. It uses pessimistic approach – tests are unstable unless are proven to be stable enough. Basic rules for the list are these: For 10.2 it is not maintained yet, but it will be by the time 10.2 goes GA. Inclusion: tests which have failed in buildbot in the corresponding branch since the previous release go to the list (and bug reports are created for them); tests for which we have open bug reports remain on the list until the bug reports are closed; tests which have been added or modified since the previous release, directly or via include files, go to the list (trivial modifications can be ignored on case by case basis). Exclusion: tests for which we do not have open bugs and which have not been modified and have not failed since the previous release (sometimes over two release cycles, depending on their length) are excluded from the list. On an additional note, if you are want to enable/disable tests really quickly, depending on your process, it might be easier to use --skip-test rather than --skip-test-list .
          mschorm Michal Schorm added a comment -

          Great!
          That is good to know.
          I changed Fedora's package to use the "unstable-tests" lists instead, while extending the list with ours.

          However, I still believe, the comment in the patch should be reviewed.
          After changing certain words about Red Hat implementation, I believe, it better describe how to run the testsuite, what basic option you have and why you need to run it on local server.

          mschorm Michal Schorm added a comment - Great! That is good to know. I changed Fedora's package to use the "unstable-tests" lists instead, while extending the list with ours. However, I still believe, the comment in the patch should be reviewed. After changing certain words about Red Hat implementation, I believe, it better describe how to run the testsuite, what basic option you have and why you need to run it on local server.

          https://github.com/MariaDB/server/commit/9b897d663a116c5484dade0404870975fd5a2e68

          I've added suggested notes with some changes, and while at that, have also done other modifications.

          mschorm, I delayed this simple request a lot, sorry about that. Please check the RedHat-related part, if it's already become outdated, I will fix it further.

          elenst Elena Stepanova added a comment - https://github.com/MariaDB/server/commit/9b897d663a116c5484dade0404870975fd5a2e68 I've added suggested notes with some changes, and while at that, have also done other modifications. mschorm , I delayed this simple request a lot, sorry about that. Please check the RedHat-related part, if it's already become outdated, I will fix it further.
          mschorm Michal Schorm added a comment -

          Actually, beacuse this discussion, I removed the mentioned 'rh-skipped-tests.list' and all RH tests, that I want to skip are now appended to the 'unstable-tests' file - as it should be.
          So you can drop the mentions of that file.

          The important part is about "How to run the testsuite at RH family OSs".

          mschorm Michal Schorm added a comment - Actually, beacuse this discussion, I removed the mentioned 'rh-skipped-tests.list' and all RH tests, that I want to skip are now appended to the 'unstable-tests' file - as it should be. So you can drop the mentions of that file. The important part is about "How to run the testsuite at RH family OSs".

          Thanks. I've removed the mention of rh-skipped-tests.list.

          elenst Elena Stepanova added a comment - Thanks. I've removed the mention of rh-skipped-tests.list.

          People

            elenst Elena Stepanova
            mschorm Michal Schorm
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.