https://buildbot.mariadb.org/#/builders/amd64-ubuntu-2004-fulltest has been running all spider tests reliably for a while according to cross-reference. Failures where the WIP branches from ycp.
Valgrind was also running spider tests reliably for a while (until non-spider related failure).
Testing main suite only and calling it protected is too much of a fudge.
Nice idea of codebase triggered protected branches - no idea how that would be implemented. Feedback from code touch by storage engine/plugin to where the tests are run would be useful.
Spider tests seems about ~2 minutes locally, so added https://github.com/MariaDB/buildbot/pull/101
Thanks danblack! I left a comment in the PR regarding running almost all spider tests instead of just the ones immediately under the "main" spider suite. That will run about 7.5x as many tests, so conditional on spider codebase touching may still be needed.
I did an experiment by breaking one spider test in each spider (sub)suite.
I got a green checkmark at github 
:
https://github.com/MariaDB/server/commit/3db3392f81d0e55292b16ef82acd78715d3c99b4
The grid shows that only 3 out of the 109 builds run spider tests. And these 3 builds only run the main spider suite, under storage/spider/mysql-test/spider/t/, see [1]:
- amd64-ubuntu-1804-clang10-asan
- amd64-ubuntu-2004-fulltest
- amd64-ubuntu-2204-valgrind
[1] https://buildbot.mariadb.org/#/grid?branch=bb-11.0-mdbf-535-do-not-merge
Most of the spider bugfix related tests are placed under the bugfix subsuite, i.e. storage/spider/mysql-test/spider/bugfix/t/, so it would be great if we can include all subsuites in the CI too.
I have not had a look at the buildbot code regarding this observation yet.
Thanks danblack, this sounds like MDEV-30722.
It works: https://buildbot.mariadb.org/#/builders/221/builds/24081
for commit https://github.com/MariaDB/server/commit/0ae05d7d37e39ecd782299f2c0cc00c1a61cf504
In build properties tab:
spider_changed "YES"
in compile[1]:
sh -c 'cmake . -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER=g++ -DPLUGIN_TOKUDB=NO -DPLUGIN_MROONGA=NO -DPLUGIN_SPIDER=YES -DPLUGIN_OQGRAPH=NO -DPLUGIN_PERFSCHEMA=YES -DPLUGIN_SPHINX=NO && make -j7 package'
[1] https://buildbot.mariadb.org/#/builders/221/builds/24081/steps/6/logs/stdio
In test[2]:
argv: [b'sh', b'-c', b'\n cd mysql-test &&\n exec perl mysql-test-run.pl --verbose-restart --force --retry=3 --max-save-core=1 --max-save-datadir=1 --max-test-fail=20 --mem --parallel=$(expr 7 \\* 2) --suite=main,spider,spider/bg,spider/bugfix,spider/feature,spider/regression/e1121,spider/regression/e112122 --skip-test="^stack_crash$|^float$|^derived_split_innodb$|^mysql_client_test$|^kill$|^processlist_not_embedded$|^sp-big$"\n ']
[2] https://buildbot.mariadb.org/#/builders/221/builds/24081/steps/7/logs/stdio
I think we can close this ticket now.
Oh wait, it does not show the failed spider tests. Take [1] for example, the failing tests are
Failing test(s): spider/bugfix.mdev_19866 innodb.undo_truncate
But the grid view[2] only shows innodb.undo_truncate as the failed tests:
amd64-ubuntu-2204-debug-ps failed test (failure)
innodb.undo_truncate
[1] https://buildbot.mariadb.org/#/builders/534/builds/7967
[2] https://buildbot.mariadb.org/#/grid?branch=bb-10.5-mdbf-535-test
After some discussions with danblack, it seems to be the regex not accounting for subsuites containing forward slashes like "foo/bar.quux" at [1]:
_line_re = re.compile( |
r"^([-._0-9a-zA-z]+)( '[-_ a-zA-Z]+')?\s+(w[0-9]+\s+)?\[ (fail|pass) \]\s*(.*)$") |
ycp the regex issue (at least the case with /) should be solved now. If you encounter any other cases please ping me.
vladbogo Thank you. I just re-ran the same builder
amd64-ubuntu-2204-debug-ps on the commit I used above, but it still
does not show the failure spider/bugfix.mdev_19866:
- https://buildbot.mariadb.org/#/builders/534/builds/8216
- https://buildbot.mariadb.org/#/grid?branch=bb-10.5-mdbf-535-test
I'm gonna try a brand new commit now, see if it makes any difference.
Seems like the regexp fix works for most of the builders, but a few builders (e.g. amd64-centos-7) still do not show the new spider failure:
https://buildbot.mariadb.org/#/grid?branch=bb-10.9-mdbf-535-test
Do you know what could have caused this vladbogo?
Hi. Buildbot uses a multi master setup where sub parts of the builders are managed by different master processes. The master process that handled amd64-centos-7 was not yet restarted with the new changes. That particular master handles all protected branches builders and there were running builds when I propagated the changes. Unfortunately buildbot requires a restart to propagate new changes which cancels all running builds. However, the changes will be propagated to all masters. Thanks for checking!
vladbogo I see, thanks for explaining. Do you know when the change will be propagated to all masters?
vladbogo Yes I just checked amd64-centos-7 on a recent build and it did run the spider tests[1], thank you for fixing this.
[1] https://buildbot.mariadb.org/#/builders/146/builds/25142