Uploaded image for project: 'MariaDB Foundation Development'
  1. MariaDB Foundation Development
  2. MDBF-1001

Add a builder that checks that test cases actually fail if bug is not fixed

Details

    • Task
    • Status: Open (View Workflow)
    • Major
    • Resolution: Unresolved
    • None
    • None
    • Buildbot
    • None

    Description

      Idea:

      Check that for any particular bugfix patch (TODO: define what that means), the test cases introduced actually fail if the bug fix is not compiled in.

      Pseudoscript:

      git checkout HEAD~1
      		 
      cmake . && make
      		 
      git checkout HEAD -- <test-files-added-in-HEAD-commit>
      		 
      ./mtr <test-files>  # This should fail. If it fails, the test is succesful.

      Attachments

        Activity

          Ascending order - Click to sort in descending order
          marko Marko Mäkelä added a comment -

          In some cases it is only possible to write a test that occasionally fails when the fix is not present. A recent example is MDEV-35647.

          marko Marko Mäkelä added a comment - In some cases it is only possible to write a test that occasionally fails when the fix is not present. A recent example is MDEV-35647 .
          marko Marko Mäkelä added a comment -

          Sometimes, as part of a fix, additional #ifndef DBUG_OFF or #ifdef ENABLED_DEBUG_SYNC instrumentation would be added that the test case depends on, sometimes resulting in a hang, sometimes in an absence of a failure, or sometimes in a failure that is different from the one that the fix is supposed to address. MDEV-36227 is a recent example of this. How would these cases be categorized, and where would the "exception database" be saved and made available for review? This would seem to involve some manual work in any case.

          marko Marko Mäkelä added a comment - Sometimes, as part of a fix, additional #ifndef DBUG_OFF or #ifdef ENABLED_DEBUG_SYNC instrumentation would be added that the test case depends on, sometimes resulting in a hang, sometimes in an absence of a failure, or sometimes in a failure that is different from the one that the fix is supposed to address. MDEV-36227 is a recent example of this. How would these cases be categorized, and where would the "exception database" be saved and made available for review? This would seem to involve some manual work in any case.

          People

            Unassigned Unassigned
            cvicentiu Vicențiu Ciorbaru
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated: