Details
-
New Feature
-
Status: Open (View Workflow)
-
Minor
-
Resolution: Unresolved
-
None
-
None
-
None
Description
Here in MCS we use at least two test suites to discover introduction of defects into new or changed code. Both test suites are not optimized for the discovery of defect sources, as one example, tests in MTR suite usually contain dozen(s) of DDL/DML and retrieval clauses and MTR itself does not do any analysis apart from providing diff.
While these suites and tools are infinitely valuable as they are, they can be enhanced to provide more succinct representation of failure and even provide shorter test cases derived from failing tests. These changes would make life of developers easier.
The simplest step to make life easier is to annotate MTR's query results with the line range of the test file of query being executed. Results will indicate failures such as completely empty results and also directly point to the queries that diverge. Right now developer has to manually count queries, look at the query results structure and logged results structure to be sure it is wrong one, etc, and then pinpoint problematic query. With the annotation of output we can directly go to the test file and tell "this is the query that has wrong results" or "this query returns empty set and this is why its results are not present".
More complex way is to track queries executed and use difference in the results of MariaDB and MCS to figure out the least possible set of queries that diverge.
For example, we may have a CREATE TABLE, some INSERTs and then some SELECTs. Problem in some these SELECTs can be due to interactions between INSERTs and problematic selects and between SELECTs themselves (both are related to extentmap global state). The tool can eliminate some of the queries preceding problematic ones and compare execution on the MariaDB/ISAM and on MCS. Minimal set of queries that diverge on these two engines can be reported to the developer as a special test case he or she can use to better find source of the defect.