I noticed that Travis-CI has stopped passing for the 10.5 branch:
Last successful one:
First failing one:
(there are also a couple cancelled in between)
However, since it is not fixed, it was disabled by me in https://github.com/MariaDB/server/commit/a135f0ab88d63b9a8976d6b3010f27766c38873d (when https://github.com/MariaDB/server/pull/1484 was merged by Marko Mäkelä).
This change to mysql-test/unstable-tests was lost in a merge commit. The fix to this is trivial: add back the line in mysql-test/unstable-tests.
However, the underlying issue here is that current MariaDB Server practices allow Travis-CI to be broken, and effectively after that:
- All new and updated pull requests at https://github.com/MariaDB/server/pulls will start to fail, communicating indirectly to both contributors and reviewers that the code is broken and not worth reviewing until at least the CI passes
- Any new contributors branching of the latest development git branch will have a failing CI as the starting point, which most likely puts them off.
- Quality deteriorates, since once the CI starts failing, people start to ignore all results from the CI and more and more failures start to creep in.
And so on. I hope you get the point why failing CI is bad and how it is counter-productive and wastes a lot of human resources that is away from productive development.
Now what can be do about this?
Is there a need for more education? Travis-CI was added as the first and only CI system accessible to outside contributors in August 2016. Surely all developers have had a chance to learn about it? Or is there some obstacles? Should we maybe organize a webinar where we quickly go through what Travis-CI is, what the lines in .travis.yml mean and how to browser Travis-CI.org to look at build results or debug them?
Are the Travis-CI tests bad? There are no open bug reports on Jira about any complaints about Travis-CI.
I think the underlying problem here is the same reason why there are so many failures on buildbot.askmonty.org and buildbot.mariadb.org as well. Way too many people are taking the wrong tradeoff in the decision about "Just get it done and move on, don't wait for tests" vs "Work on something else, only merge once tests complete".
What do you think? What should be done about this to improve the situation, to improve the quality of MariaDB both by current developers and future contributors, and speed up the progress by having less breakage and steps backwards?