Thanks for creating this!

monty and I discussed the lack of the assert command in MariaDB's implementation last month in Vancouver, and how having this would make it easier to write MTR tests in a style+structure that's familiar to users of unit-testing frameworks like JUnit/Pytest/etc.

PR created: https://github.com/MariaDB/server/pull/2169

• copy_files_wildcard looks quite a logical addition.
• force_cpdir — isn't it the same as rmdir + mkdir + copy_files_wildcard ?
• assert — it's conceptually wrong here. mysqltest isn't JUnit/Pytest/etc. An "assert" in mysqltest is "just print the result", everything that goes to the output is an assert. mysqltest language is not a conventional procedural language and it's wrong to try to use it as such.

Hi Sergei Golubchik. Thanks for your reply. Here are some of my thoughts.

Re "force_cpdir — isn't it the same as rmdir + mkdir + copy_files_wildcard"

It is true until we have nested directories to copy. Then without "force_cpdir", users will have to use "rmdir + mkdir + ... + mkdir + copy_files_wildcard + ... + copy_files_wildcard" with the assumption that they know what the directory structure is like. Comparing to this, "force_cpdir" allows users to copy recursively.

Re "assert — it's conceptually wrong here"

It's a good point that for MTR "everything that goes to the output is an assert". However in my opinion, not all contents in the output are at the same level of importance. As a result when our change affect the test output, we usually need to decide whether a alternation is reasonable. For example in such a situation:

1. A user creates a test case where a query is explicitly expecting 10 rows. He/she generated the .result file and test passed. Everything looks good.
2. Later another user makes some changes, that alter a large number of test outputs including the output above.
3. He/she thinks it is insignificant that the row number now becomes 11, so simply overwrites it with "--record". Or even worse that he/she doesn't even notice this line.

If the person who creates the 10-row test case isn't around, a bug could very likely sneak through. If "assert (...)" is used in the .test file, then it convey the idea that this is an important check point, and it will never be overwritten by "--record".

Alternatively the initial test writer could leave some comment with "--echo" to make it visible in .result, but that doesn't come as convenient as "assert" in my opinion.

force_cpdir — right, I see. Perhaps, then, copy_files_wildcard should copy directories recursively then? In fact, I'd personally prefer to have only one command, copy_files that does wildcards and copy recursively, but someone has already added copy_files_wildcard instead of extending the existing command. Let's not do that again?

assert — well, nothing helps against somebody who is determined to update the results, I've seen people updating asserts too. In my opinion the main goal of a .test file (besides actually, verifying some behavior) is good debug-ability, when a test fails, it should be as easy as possible to understand what's going on. A diff between expected and actual results help much more than an abort like in source include/assert.inc. I sometimes use the pattern like

select count(col) as 'must be 1' from t1;

that makes it somewhat more difficult to update the result and not notice the incongruity.

serg wrote:

assert — In my opinion the main goal of a .test file (besides actually, verifying some behavior) is good debug-ability, when a test fails, it should be as easy as possible to understand what's going on. A diff between expected and actual results help much more than an abort like in source include/assert.inc.

MTR has a split between ".test" files (contain the meta-commands and SQL commands to run) and ".result" files (which contain most of the meta-commands and SQL commands and also the results/output of those commands). As I told monty last year, this is quite an error-prone design compared to the test frameworks of most modern programming languages.

For example, if you were testing MariaDB with PyTest or Python's unittest…

cur = mariadb.connect(…).cursor()

cur.execute("DO SOME QUERY");

assert( list(cur) == [ [ 'EXPECTED', 'RESULT', 'ROW', 1 ] ] )

… you'd have the test queries and their results in one place. With MTR, we have them split across two files, which makes it hard to understand what the tests are doing, and easy to overlook mistakes given the high level of verbosity and redundancy of the .result file.

Adding assert to the MTR language can go a long way towards making it possible to use MTR in a concise form that closely resembles the modern test frameworks of other programming languages.