Type:
Task
Priority:
Minor
Resolution:
Unresolved
Affects Version/s:
None
Every default run of mysql-test-run script takes a lot of time (tens of minutes to many hours, depending on the build and computer configuration).
But most of that time is wasted by running wrong tests. Some of the tests are related, if one of the fails, others will fail too. Some of the tests just almost never fail, while others fail much more often. Some of the test execute a specific part of the source code and if that part of the code isn't changed in a particular revision, these tests will certainly not fail.
I would like to know what tests are most useful to run for every particular revision on every particular test platform. In my experiments one can catch 90% of the problems by only running 10% of the tests.
It doesn't mean we will always run 10% of the tests only. It would make sense to run the complete big test suite before releases or on specific builders. But many builders can test must faster with only a small reduction of the test coverage.
How to experiment
There is no need to run tests on many platforms for this. We have historical data from the buildbot for many years. They contain the information what revisions were tested on what builders, what files were modified in what revision, what tests failed where and so on. One can use these data to analyze and select the best test running strategy.
The goal is to run as little tests as possible, while still being able to detect as many test failures as possible.
What to take into account (ideas):
probability of a test to fail
depending on the builder, on the combination
depending on the changed files, changed lines/functions/etc
inter-test correlations
individual tests within a big test file
Assorted thoughts
what to do what a new builder/test/combination is added? we don't have prior probabilities yes
don't use all the data, instead use a sliding window — the failure rates may change over time
average over different combinations or builders
or don't average and treat triplets (test,combination,builder) as individual "tests"
optimize for time, not for a number of tests — differen builders run with different speed, different tests take different time too
emulate the filter bubble (ignore not predicted falures), have a solution to break it
{"searchers":{"groups":[{"searchers":[{"name":"Project","id":"project","key":"issue.field.project","isShown":true,"lastViewed":1739932125857},{"name":"Summary","id":"summary","key":"issue.field.summary","isShown":true},{"name":"Type","id":"issuetype","key":"issue.field.issuetype","isShown":true,"lastViewed":1739932125857},{"name":"Status","id":"status","key":"issue.field.status","isShown":true,"lastViewed":1739932125864},{"name":"Priority","id":"priority","key":"issue.field.priority","isShown":true},{"name":"Resolution","id":"resolution","key":"issue.field.resolution","isShown":true},{"name":"Creator","id":"creator","key":"issue.field.creator","isShown":true},{"name":"Affects Version","id":"version","key":"issue.field.affectsversions","isShown":true},{"name":"Fix Version","id":"fixfor","key":"issue.field.fixversions","isShown":true},{"name":"Component","id":"component","key":"issue.field.components","isShown":true},{"name":"% Limits","id":"workratio","key":"issue.field.workratio","isShown":true},{"name":"Link types","id":"issue_link_type","key":"issue.field.issuelinks","isShown":true},{"name":"Environment","id":"environment","key":"issue.field.environment","isShown":true},{"name":"Description","id":"description","key":"issue.field.description","isShown":true},{"name":"Comment","id":"comment","key":"issue.field.comment","isShown":true},{"name":"Label","id":"labels","key":"issue.field.labels","isShown":true,"lastViewed":1739932125866},{"name":"Query","id":"text","key":"text","isShown":true},{"name":"Development","id":"customfield_11700","key":"com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary","isShown":true},{"name":"Epic Colour","id":"customfield_10603","key":"com.pyxis.greenhopper.jira:gh-epic-color","isShown":false},{"name":"Epic Link","id":"customfield_10600","key":"com.pyxis.greenhopper.jira:gh-epic-link","isShown":true},{"name":"Epic Name","id":"customfield_10601","key":"com.pyxis.greenhopper.jira:gh-epic-label","isShown":true},{"name":"Epic Status","id":"customfield_10602","key":"com.pyxis.greenhopper.jira:gh-epic-status","isShown":false},{"name":"Epic/Theme","id":"customfield_10103","key":"com.atlassian.jira.plugin.system.customfieldtypes:labels","isShown":true},{"name":"Flagged","id":"customfield_13101","key":"com.atlassian.jira.plugin.system.customfieldtypes:multicheckboxes","isShown":true},{"name":"Global Rank","id":"customfield_10000","key":"com.pyxis.greenhopper.jira:gh-global-rank","isShown":true},{"name":"PM Planning","id":"customfield_11404","key":"com.atlassian.jira.plugin.system.customfieldtypes:labels","isShown":true},{"name":"Python Version","id":"customfield_12404","key":"com.atlassian.jira.plugin.system.customfieldtypes:textfield","isShown":false},{"name":"Rank","id":"customfield_10700","key":"com.pyxis.greenhopper.jira:gh-lexo-rank","isShown":true},{"name":"Rank (Obsolete)","id":"customfield_10104","key":"com.atlassian.jira.plugin.system.customfieldtypes:float","isShown":false},{"name":"Rank (Obsolete)","id":"customfield_10300","key":"com.pyxis.greenhopper.jira:gh-global-rank","isShown":true},{"name":"Roadmap Score","id":"customfield_10106","key":"com.atlassian.jira.plugin.system.customfieldtypes:float","isShown":false},{"name":"Sprint","id":"customfield_10400","key":"com.pyxis.greenhopper.jira:gh-sprint","isShown":true},{"name":"Story Points","id":"customfield_13100","key":"com.atlassian.jira.plugin.system.customfieldtypes:float","isShown":false},{"name":"Support case ID","id":"customfield_11500","key":"com.atlassian.jira.plugin.system.customfieldtypes:labels","isShown":true},{"name":"Zendesk inactive tickets","id":"customfield_13002","key":"com.atlassian.jira.plugin.system.customfieldtypes:labels","isShown":true},{"name":"gitBranch","id":"customfield_12400","key":"com.xiplink.jira.git.jira_git_plugin:gitbranches","isShown":true},{"name":"gitCommitsReferenced","id":"customfield_12401","key":"com.xiplink.jira.git.jira_git_plugin:gitcommitsreferenced","isShown":true},{"name":"jqltField","id":"customfield_11300","key":"com.j-tricks.jql-plugin:jqlt-field","isShown":true}],"type":"DETAILS","title":"Details"},{"searchers":[{"name":"Created Date","id":"created","key":"issue.field.created","isShown":true},{"name":"Updated Date","id":"updated","key":"issue.field.updated","isShown":true},{"name":"Resolution Date","id":"resolutiondate","key":"issue.field.resolution.date","isShown":true},{"name":"Due Date","id":"duedate","key":"issue.field.duedate","isShown":true}],"type":"DATES","title":"Dates"},{"searchers":[{"name":"Assignee","id":"assignee","key":"issue.field.assignee","isShown":true,"lastViewed":1739932125865},{"name":"Reporter","id":"reporter","key":"issue.field.reporter","isShown":true},{"name":"Assigned for Review","id":"customfield_12700","key":"com.atlassian.jira.plugin.system.customfieldtypes:userpicker","isShown":true},{"name":"Assigned for Testing","id":"customfield_12701","key":"com.atlassian.jira.plugin.system.customfieldtypes:userpicker","isShown":true}],"type":"PEOPLE","title":"People"}]},"values":{"issuetype":{"name":"Type","editHtml":"\n\n\n\n <div class=\"field-group aui-field-issuetype\" >\n <label for=\"searcher-type\">Type</label> <select class=\"select js-default-checkboxmultiselect\" id=\"searcher-type\" multiple=\"multiple\" name=\"type\"\n data-placeholder-text=\"Find Issue Types...\" size=\"5\">\n <optgroup>\n \n <option class=\" \"\n id=\"type_-2\"\n title=\"All Standard Issue Types\"\n value=\"-2\">All Standard Issue Types</option>\n \n <option class=\" \"\n id=\"type_-3\"\n title=\"All Sub-Task Issue Types\"\n value=\"-3\">All Sub-Task Issue Types</option>\n </optgroup>\n\n <optgroup label=\"Standard Issue Types\">\n \n <option class=\" imagebacked 12800 10202 12701 \"\n data-icon=\"/secure/viewavatar?size=xsmall&avatarId=14103&avatarType=issuetype\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_1\"\n title=\"Bug\"\n value=\"1\">Bug</option>\n \n <option class=\" imagebacked 12800 10202 12701 \"\n data-icon=\"/secure/viewavatar?size=xsmall&avatarId=14107&avatarType=issuetype\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_5\"\n title=\"Epic\"\n value=\"5\">Epic</option>\n \n <option class=\" imagebacked 12800 10202 12701 \"\n data-icon=\"/secure/viewavatar?size=xsmall&avatarId=14111&avatarType=issuetype\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_2\"\n title=\"New Feature\"\n value=\"2\">New Feature</option>\n \n <option class=\" imagebacked 12800 10202 12701 \"\n data-icon=\"/secure/viewavatar?size=xsmall&avatarId=14118&avatarType=issuetype\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_3\"\n title=\"Task\"\n value=\"3\">Task</option>\n </optgroup>\n\n <optgroup label=\"Sub-Task Issue Types\">\n \n <option class=\" imagebacked 12800 12701 \"\n data-icon=\"/secure/viewavatar?size=xsmall&avatarId=14116&avatarType=issuetype\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_10000\"\n title=\"Sub-Task\"\n value=\"10000\">Sub-Task</option>\n \n <option class=\" imagebacked 10202 12701 \"\n data-icon=\"/images/icons/issuetypes/task_agile.png\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_7\"\n title=\"Technical task\"\n value=\"7\">Technical task</option>\n </optgroup>\n </select>\n </div>\n ","validSearcher":true,"isShown":true},"project":{"name":"Project","editHtml":"\n <div class=\"field-group aui-field-project\" >\n <label for=\"searcher-pid\">Project</label> \n\n <select class=\"select js-default-checkboxmultiselect\" data-placeholder-text=\"Find Projects...\" id=\"searcher-pid\" multiple=\"multiple\" name=\"pid\" size=\"5\">\n <optgroup label=\"All Projects\">\n \n <option data-icon=\"/secure/projectavatar?pid=10902&size=small\"\n title=\"MariaDB ColumnStore\"\n value=\"10902\">\n MariaDB ColumnStore (MCOL)\n </option>\n \n <option data-icon=\"/secure/projectavatar?pid=10300&size=small\"\n title=\"MariaDB Connector/C\"\n value=\"10300\">\n MariaDB Connector/C (CONC)\n </option>\n \n <option data-icon=\"/secure/projectavatar?pid=11801&size=small\"\n title=\"MariaDB Connector/C++\"\n value=\"11801\">\n MariaDB Connector/C++ (CONCPP)\n </option>\n \n <option data-icon=\"/secure/projectavatar?pid=10301&size=small\"\n title=\"MariaDB Connector/J\"\n value=\"10301\">\n MariaDB Connector/J (CONJ)\n </option>\n \n <option data-icon=\"/secure/projectavatar?pid=11300&size=small\"\n title=\"MariaDB Connector/node.js\"\n value=\"11300\">\n MariaDB Connector/node.js (CONJS)\n </option>\n \n <option data-icon=\"/secure/projectavatar?pid=10400&size=small\"\n title=\"MariaDB Connector/ODBC\"\n value=\"10400\">\n MariaDB Connector/ODBC (ODBC)\n </option>\n \n <option data-icon=\"/secure/projectavatar?pid=11400&size=small\"\n title=\"MariaDB Connector/Python\"\n value=\"11400\">\n MariaDB Connector/Python (CONPY)\n </option>\n \n <option data-icon=\"/secure/projectavatar?pid=11902&size=small\"\n title=\"MariaDB Connector/R2DBC\"\n value=\"11902\">\n MariaDB Connector/R2DBC (R2DBC)\n </option>\n \n <option data-icon=\"/secure/projectavatar?pid=11901&size=small\"\n title=\"MariaDB Foundation Development\"\n value=\"11901\">\n MariaDB Foundation Development (MDBF)\n </option>\n \n <option data-icon=\"/secure/projectavatar?pid=10600&size=small\"\n title=\"MariaDB MaxScale\"\n value=\"10600\">\n MariaDB MaxScale (MXS)\n </option>\n \n <option data-icon=\"/secure/projectavatar?pid=10000&size=small\"\n title=\"MariaDB Server\"\n value=\"10000\">\n MariaDB Server (MDEV)\n </option>\n </optgroup>\n </select>\n </div>\n \n","validSearcher":true,"isShown":true},"assignee":{"name":"Assignee","editHtml":"\n \n <div class=\"field-group aui-field-userlist\" >\n <label for=\"searcher-assigneeSelect\">Assignee</label> <fieldset rel=\"assignee\" class=\"hidden user-group-searcher-params\">\n </fieldset>\n <select class=\"js-usergroup-checkboxmultiselect\" multiple=\"multiple\" id=\"assignee\" name=\"assignee\" data-placeholder-text=\"Enter username or group\">\n <optgroup>\n <option class=\"headerOption\" data-icon=\"https://jira.mariadb.org/secure/useravatar?size=xsmall&avatarId=10123\" value=\"empty\" title=\"Unassigned\">Unassigned</option>\n </optgroup>\n <optgroup>\n </optgroup>\n </select>\n <input type=\"hidden\" name=\"check_prev_assignee\" value=\"true\">\n </div>\n \n","validSearcher":true,"isShown":true},"status":{"name":"Status","editHtml":"\n <div class=\"field-group aui-field-constants\" >\n <label for=\"searcher-status\">Status</label> \n <select class=\"select js-default-checkboxmultiselectstatuslozenge\" data-placeholder-text=\"Find Statuses...\" id=\"searcher-status\" multiple=\"multiple\" name=\"status\" size=\"4\" data-status-lozenge=\"true\">\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/open.png\" value=\"1\" title=\"Open\" data-simple-status=\"{"id":"1","name":"Open","description":"","iconUrl":"/images/icons/statuses/open.png","statusCategory":{"id":2,"key":"new","colorName":"blue-gray"}}\">Open</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/status_generic.gif\" value=\"10101\" title=\"Confirmed\" data-simple-status=\"{"id":"10101","name":"Confirmed","description":"","iconUrl":"/images/icons/status_generic.gif","statusCategory":{"id":4,"key":"indeterminate","colorName":"yellow"}}\">Confirmed</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/inprogress.png\" value=\"3\" title=\"In Progress\" data-simple-status=\"{"id":"3","name":"In Progress","description":"This issue is being actively worked on at the moment by the assignee.","iconUrl":"/images/icons/statuses/inprogress.png","statusCategory":{"id":4,"key":"indeterminate","colorName":"yellow"}}\">In Progress</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/down.png\" value=\"10000\" title=\"Stalled\" data-simple-status=\"{"id":"10000","name":"Stalled","description":"This issue is not being actively worked on at the moment.","iconUrl":"/images/icons/statuses/down.png","statusCategory":{"id":4,"key":"indeterminate","colorName":"yellow"}}\">Stalled</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/information.png\" value=\"10002\" title=\"In Review\" data-simple-status=\"{"id":"10002","name":"In Review","description":"The current assignee is reviewing the work done in this issue.","iconUrl":"/images/icons/statuses/information.png","statusCategory":{"id":4,"key":"indeterminate","colorName":"yellow"}}\">In Review</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/generic.png\" value=\"10301\" title=\"In Testing\" data-simple-status=\"{"id":"10301","name":"In Testing","description":"","iconUrl":"/images/icons/statuses/generic.png","statusCategory":{"id":4,"key":"indeterminate","colorName":"yellow"}}\">In Testing</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/closed.png\" value=\"6\" title=\"Closed\" data-simple-status=\"{"id":"6","name":"Closed","description":"","iconUrl":"/images/icons/statuses/closed.png","statusCategory":{"id":3,"key":"done","colorName":"green"}}\">Closed</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/closed.png\" value=\"10001\" title=\"Done\" data-simple-status=\"{"id":"10001","name":"Done","description":"","iconUrl":"/images/icons/statuses/closed.png","statusCategory":{"id":3,"key":"done","colorName":"green"}}\">Done</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/open.png\" value=\"10003\" title=\"To Do\" data-simple-status=\"{"id":"10003","name":"To Do","description":"","iconUrl":"/images/icons/statuses/open.png","statusCategory":{"id":2,"key":"new","colorName":"blue-gray"}}\">To Do</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/status_generic.gif\" value=\"10102\" title=\"Verified\" data-simple-status=\"{"id":"10102","name":"Verified","description":"Issue is tested and confirmed as fixed or done","iconUrl":"/images/icons/status_generic.gif","statusCategory":{"id":3,"key":"done","colorName":"green"}}\">Verified</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/generic.png\" value=\"10408\" title=\"Release done\" data-simple-status=\"{"id":"10408","name":"Release done","description":"-Commit & merges are done\\r\\n-Packages are built & tested\\r\\n-Packages of packages are created\\r\\n-Packages are uploaded live on MariaDB.com","iconUrl":"/images/icons/statuses/generic.png","statusCategory":{"id":4,"key":"indeterminate","colorName":"yellow"}}\">Release done</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/generic.png\" value=\"10501\" title=\"Needs Feedback\" data-simple-status=\"{"id":"10501","name":"Needs Feedback","description":"","iconUrl":"/images/icons/statuses/generic.png","statusCategory":{"id":2,"key":"new","colorName":"blue-gray"}}\">Needs Feedback</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/generic.png\" value=\"10701\" title=\"Fixed\" data-simple-status=\"{"id":"10701","name":"Fixed","description":"","iconUrl":"/images/icons/statuses/generic.png","statusCategory":{"id":3,"key":"done","colorName":"green"}}\">Fixed</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/generic.png\" value=\"10702\" title=\"Duplicate\" data-simple-status=\"{"id":"10702","name":"Duplicate","description":"","iconUrl":"/images/icons/statuses/generic.png","statusCategory":{"id":3,"key":"done","colorName":"green"}}\">Duplicate</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/generic.png\" value=\"10703\" title=\"Retest\" data-simple-status=\"{"id":"10703","name":"Retest","description":"","iconUrl":"/images/icons/statuses/generic.png","statusCategory":{"id":3,"key":"done","colorName":"green"}}\">Retest</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/generic.png\" value=\"10704\" title=\"Re-opened\" data-simple-status=\"{"id":"10704","name":"Re-opened","description":"","iconUrl":"/images/icons/statuses/generic.png","statusCategory":{"id":2,"key":"new","colorName":"blue-gray"}}\">Re-opened</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/generic.png\" value=\"10801\" title=\"Published on Staging\" data-simple-status=\"{"id":"10801","name":"Published on Staging","description":"For the issue type \\"Release\\" in the poject TODO, when new build are published on staging.","iconUrl":"/images/icons/statuses/generic.png","statusCategory":{"id":4,"key":"indeterminate","colorName":"yellow"}}\">Published on Staging</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/generic.png\" value=\"10802\" title=\"Won't Fix\" data-simple-status=\"{"id":"10802","name":"Won't Fix","description":"","iconUrl":"/images/icons/statuses/generic.png","statusCategory":{"id":3,"key":"done","colorName":"green"}}\">Won't Fix</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/generic.png\" value=\"10803\" title=\"Invalid\" data-simple-status=\"{"id":"10803","name":"Invalid","description":"","iconUrl":"/images/icons/statuses/generic.png","statusCategory":{"id":3,"key":"done","colorName":"green"}}\">Invalid</option>\n </select>\n </div>\n \n","validSearcher":true,"isShown":true},"labels":{"name":"Label","viewHtml":" <div class=\"searcherValue\">\n \n <label class=\"fieldLabel\" for=\"fieldlabels\">Label:</label><span id=\"fieldlabels\" class=\"fieldValue\">\n \n re-evaluate\n</span></div>\n","editHtml":"\n <div class=\"field-group aui-field-labels\" >\n <label for=\"searcher-labels\">Labels</label> <select class=\"js-label-checkboxmultiselect\" multiple=\"multiple\" id=\"searcher-labels\" name=\"labels\" data-placeholder-text=\"Find Labels...\">\n <option value=\"re-evaluate\" title=\"re-evaluate\" selected=\"selected\">re-evaluate</option>\n </select>\n </div>\n \n","jql":"labels = re-evaluate","validSearcher":true,"isShown":true}}}
[{"id":-1,"name":"My open issues","jql":"assignee = currentUser() AND resolution = Unresolved order by updated DESC","isSystem":true,"sharePermissions":[],"requiresLogin":true},{"id":-2,"name":"Reported by me","jql":"reporter = currentUser() order by created DESC","isSystem":true,"sharePermissions":[],"requiresLogin":true},{"id":-4,"name":"All issues","jql":"order by created DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-5,"name":"Open issues","jql":"resolution = Unresolved order by priority DESC,updated DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-9,"name":"Done issues","jql":"statusCategory = Done order by updated DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-3,"name":"Viewed recently","jql":"issuekey in issueHistory() order by lastViewed DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-6,"name":"Created recently","jql":"created >= -1w order by created DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-7,"name":"Resolved recently","jql":"resolutiondate >= -1w order by updated DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-8,"name":"Updated recently","jql":"updated >= -1w order by updated DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false}]
0.3
0
It could be a reasonably good time to bring it to life, but there is a major problem – we have had so many broken builds in the recent past, our buildbot history (on which the algorithm bases its choices) is completely messed up