mtr extensions for storage engine testing

Simplify testing of different storage engines in mtr

Jump to Summary

The main goal

allow to test new engines against the whole test suite (or most of it) easily.

The current approach (that PBXT is using) — copy many tests from the main suite to another suite. Apply the necessary changes. The main problem — it's not really maintainable, because tests in the main suite are constantly modified (extended, corrected, etc), and the copied suite cannot catch up.

So, instead we need to be able to run exiting tests, not copies, with different engines. And adapt the results accordingly, because results might differ slightly.

An engine might want to do either one of the following:

• simply run the a test (or all tests) from another suite, with different
  settings (e.g. --default-storage-engine). In a sense it's like adding another combination to existing tests.
• alter the test and then run it. That can also be done in two ways:
  • add a prologue and/or epilogue.
  • modifying the test in the middle. this can be done with a patch.

related mtr problems

they will be solved together with the main issue:

• when a test suite produces different result file in different environments we have to duplicate it. generally, it's done by copying the original file into include/, and creating "test files" that don't do anything else but requiring the correct execution environment (have_binlog_format_raw, have_64bit, not_windows) and including the original tets file. This workaround allows to have different (although, still, nearly identical) result files for the same original test file.

• combination and suite.pm files are per suite. In innodb suite we can run all tests both for innodb and xtradb. But there are tests in the main and rpl suites that use xtradb. Currently there's no way to run them for innodb too. Similarly, select.test is run few times with different optimizer settings. This could be done with combinations to avoid duplicating result files.

Features to implement

• replacing result file for a combination
  • result file may optionally have a combination name after the test name, before the .result extension. For example, innodb-lock,xtradb.result. If such a file is present, mtr will use it instead of the innodb-lock.result file.

• modifying result file for a combination
  • same as replacing, with .rdiff instead of .result

• A plugin can extend an existing suite. Currently mtr automatically picks up a test suite XXX if it's located in the storage/*/mtr/XXX or in plugin/XXX/tests (it is quite inconsistent now: a storage engine may have many suites, but a plugin — only one, with the suite name being plugin name). The point is that a storage engine may have many suites in its mtr/ subdirectory. So, if, say, pbxt would like to run the whole innodb suite with pbxt, a quite logical way of doing it would be creating a storage/pbxt/mtr/innodb/suite.opt file.

• extending test with a prologue/epilogue
  • easy: create a new test file and source the original test file into it.
    • if it'll have the same name as the original test file (in another test suite), the result file can be a diff.
    • the same should work for includes.

• modifying test file with a patch
  • possible solution: use .tdiff instead of .test
  • not part of this task, though

• no support for .pm files per test
  • too complex and not needed for now

• supporting combinations per test
  • for a test file sometest.test, currently we support test-specific options in the sometest.opt and test-specific config in the sometest.cnf.
  • it will be extended to support sometest.combinations.
  • of course, this should also work for include files.

Implementation details

• All compiled-in plugins need to be disabled by default. mtr will do it automatically — there will be no need to modify default_my.cnf and add loose-disable-pbxt, loose-disable-innodb, loose-disable-feedback, etc. This way a new plugin can be added, with its own directory, its own suites or overlays - but all other suites will not be affected by it. Unless this plugin is mandatory and cannot be disabled, of course.

• many combinations may apply to the same test file, if it includes two files, and both have their own combination files.
  • In this case mtr will do a Cartesian product. like, row+xtradb, row+innodb, stmt+xtradb, stmt+innodb, etc.

• As before, a suite.pm may specify what combinations to skip, no matter whether they come from a suite-wide combinations file or a more specific testname.combinations file. As a special case, it can disable the complete testname.combinations file explicitly (almost the same as disabling all combinations from that file, but allows to specify a skip comment).

• mtr will consistenctly use storage/*/mysql-test and plugin/*/mysql-test directories (storage/*/mtr is already occupied in innodb/xtradb by mtr0mtr.c, etc).
  • in a sense plugin's (engine's) mysql-test directory will be an overlay of the main mysql-test directory.
  • every new suite XXX is virtually mapped into mysql-test/suite/XXX directory. It means that if FederatedX engine would like to add a new combination to the federated suite (located in storage/federated/mysql-test/federated), it will do it in storage/federatedx/mysql-test/federated/combinations, even though federated suite is not in mysql-test/suite/federated.
  • overlay never affects the original suite. Think of it as a second layer in an image editor (in, say, Gimp). It may obscure or visually extend part of the background layer, but it does not modify it. mtr executes the "background layer" as is, then it executes the "view" of both layers.
  • when running tests for the overlay mtr will only run tests that are affected by the overlay (when a test file itself is overlayed, or one of the include files, or overlay adds new mysqld options, or my.cnf template, or a master.sh or slave.sh script).
  • overlays don't stack. Every overlay is an overlay "layer" on top of the original "background" layer. There can never be an overlay on top of the overlay.

• a test in the overlay will be shown as suite-overlay.testname. Remember that normally the test is shown as suite.testname. Notation suite-overlay will also work when selecting a suite or a test as in

  $ ./mtr --suite main-pbxt

  $ ./mtr main-pbxt.create

  specifying a suite name without an overlay will mean this suite only. For example, running

  ./mtr main.create

  will not run main-pbxt.create test. To run a test in all overlays, one should use

  ./mtr main-.create

  that is, the empty overlay name.

• mysqltest will not natively interpret diffs. Instead, mtr will create a .result~ file when it finds a .rdiff file. The file will be created in the same difrectory where .rdiff file is, and only if .result~ file does not exist already (or if it's older than .rdiff file). If result file directory is not writable, then it'll be created in var/tmp/.

• we'll apply patches using the external patch tool. This way we'll support all patch formats, and won't require installing perl patch modules on test machines. If patch is not available, the corresponding tests will be skipped.

• determining a proper result file can be quite complicated. For a test foobar.test in the combination pair aa+bb, that is run in the overlay "rty" of the suite "qwe", in other words, for the test that mtr prints as

  qwe-rty.foobar 'aa,bb' [ pass ]

  The result can be expected in

  • either .rdiff or .result file
  • either in the overlay or in the original suite
  • with or without combinations in the file name.
    which means any of the following 15 file names can be used:
  1. rty/r/foo,aa,bb.result
  2. rty/r/foo,aa,bb.rdiff
  3. qwe/r/foo,aa,bb.result
  4. qwe/r/foo,aa,bb.rdiff
  5. rty/r/foo,aa.result
  6. rty/r/foo,aa.rdiff
  7. qwe/r/foo,aa.result
  8. qwe/r/foo,aa.rdiff
  9. rty/r/foo,bb.result
  10. rty/r/foo,bb.rdiff
  11. qwe/r/foo,bb.result
  12. qwe/r/foo,bb.rdiff
  13. rty/r/foo.result
  14. rty/r/foo.rdiff
  15. qwe/r/foo.result
      They are listed, precisely, in the order of preference. mtr will walk that list from top to bottom and the first file that is found will be used.
      If this found file is a .rdiff, mtr continues walking down the list until the first .result file is found. A .rdiff is applied to that .result.

Summary

To put it short, this issue is about adding three new concepts to mysql-test:

• per-test combination files will allow to run tests with different settings, independently from a suite. Currently only tests in the "innodb" suite are run for innodb and xtradb, other tests are run for xtradb only. This feature will allow to run all such tests for both innodb and xtradb.

• overlays will allow new engines to run the whole test suite (or only individual tests) for this engine, without copying test files to a new suite.

• diffs will allow to avoid copying tests and results when the copy should only slightly differ from the original. It will greatly increase the maintainability as all changes to the original will automatically affect the copy.