Relatively fresh occasion on 5.5-galera (Feb 17, 2016):
http://buildbot.askmonty.org/buildbot/builders/kvm-bintar-centos5-amd64/builds/2356/steps/test/logs/mysqld.1.err.1
http://buildbot.askmonty.org/buildbot/builders/kvm-bintar-centos5-amd64/builds/2356/steps/test/logs/stdio
So, it's not specific either to 32-bit platform or the particular builder.
There are 5
occasions on labrador, and it does not even run MTR with parallel workers. So, the theory that a parallel test somehow interferes can be safely abandoned.
http://buildbot.askmonty.org/buildbot/builders/labrador/builds/5554 (bb-10.0-serg)
http://buildbot.askmonty.org/buildbot/builders/labrador/builds/5566
http://buildbot.askmonty.org/buildbot/builders/labrador/builds/5567
http://buildbot.askmonty.org/buildbot/builders/labrador/builds/5582
http://buildbot.askmonty.org/buildbot/builders/labrador/builds/5818
All but one are on 5.5 trees. It starts getting strange – given that 10.x have many more pushes and test runs, it should hit the problem more frequently.
Occasions on p8:
http://buildbot.askmonty.org/buildbot/builders/p8-rhel7-bintar/builds/304
http://buildbot.askmonty.org/buildbot/builders/p8-rhel7-bintar/builds/586
http://buildbot.askmonty.org/buildbot/builders/p8-rhel7-bintar-debug/builds/1036
http://buildbot.askmonty.org/buildbot/builders/p8-rhel7-bintar-debug/builds/109
http://buildbot.askmonty.org/buildbot/builders/p8-rhel7-bintar-debug/builds/1164
http://buildbot.askmonty.org/buildbot/builders/p8-rhel7-bintar-debug/builds/140 
http://buildbot.askmonty.org/buildbot/builders/p8-rhel7-bintar-debug/builds/196
http://buildbot.askmonty.org/buildbot/builders/p8-rhel7-bintar-debug/builds/820
(maybe there are more, didn't check vlp04-06).
Reproduced locally on 5.5.44 release build by running perl ./mtr rpl.rpl_mdev382 --repeat=100
The failure is very easily and reliably reproducible with this change:
diff --git a/mysql-test/suite/rpl/t/rpl_mdev382.test b/mysql-test/suite/rpl/t/rpl_mdev382.test
|
index 21925e0..c26e9e8 100644
|
--- a/mysql-test/suite/rpl/t/rpl_mdev382.test
|
+++ b/mysql-test/suite/rpl/t/rpl_mdev382.test
|
@@ -210,6 +210,7 @@ EOF
|
--shutdown_server 30
|
|
--remove_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect
|
+--sleep 1
|
--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect
|
restart-rpl_mdev382.test
|
EOF
|
The problem is that after finding mysqld.1.expect file containing wait, mysql-test-run.pl keeps reading it in a loop. So, there is a short window between remove and write when the file does not exist, if MTR happens to attempt to read at this moment, the failure occurs.
The fix is to use append_file instead.
It has already been fixed in 10.x, hidden among other fixes in this commit:
commit 6b97512b2170fc6a77288cef6e5a85cad773ae4e
|
Author: unknown <knielsen@knielsen-hq.org>
|
Date: Wed Apr 24 13:05:40 2013 +0200
|
|
|
Add missing check for thd->killed in mysql_binlog_send().
|
|
The slave dump thread running on the master only checked thd->killed whenever
|
it reached the end of a binlog file, not between events. This could
|
unnecessarily delay server shutdown.
|
|
This was found by code inspection while tracking down some occasional "forcing
|
close of thread..." errors in Buildbot. Hopefully this will fix the failures,
|
but the fix is correct in any case.
|
|
Also increase the wait during server shutdown, 2 seconds is a bit tight in
|
case of heavy I/O stall, and it seems better to delay shutdown a bit than
|
force-kill threads unnecessarily.
|
|
Also fix some races in test cases that restart the mysqld server. The .expect
|
file should be changed with --append_file, --remove_file + --write_file
|
creates a short window where mysqld can error out due to .expect file missing.
|
So it's a 5.5-only fix.
https://github.com/MariaDB/server/commit/67b4a6f576db102d652b401c3d0150556826f641