Right, currently master_slave.inc (or rpl_init.inc) must come after any have_xxx.inc checks that can cause the test case to be skipped.

The cleanup-after-failure involves restarting (and re-installing I think) all mysqld's, that would be too expensive to do for every skipped test.

It used to be that some have_xxx tests were interpreted before even running mysqltest. Then the test case never starts, and master-slave.inc cannot run. I prefered that as it is much faster, but I wonder if perhaps it is not like that anymore, since have_binlog_format_row can trigger this. Maybe that's also why this problem becomes worse?

It is in any case kind of wrong not to do skip tests before doing real work like starting slaves. Maybe just fix all of them at once? Using something like this to find them:

find mysql-test -type f -print0|xargs -0 grep -A10 -E 'master-slave\.inc|rpl_init\.inc' | grep -E 'source include/not_|source include/have_'

There are quite a few... but shouldn't be hard to just do all the reorderings once and for all?

It used to be that some have_xxx tests were interpreted before even running mysqltest. Then the test case never starts, and master-slave.inc cannot run.

Probably some are still do, but it is not always possible. More often than not these files check the variables of the currently running server – so, the server must be started before the check is performed.

It is in any case kind of wrong not to do skip tests before doing real work like starting slaves

I agree that we should try to do it whenever possible, but again, I'm afraid it's not always possible. I was going to do it anyway, whenever possible, and see in how many cases it's not.

shouldn't be hard to just do all the reorderings once and for all?

Now, that's certainly impossible. We can fix existing tests, but there should certainly be new ones.

Now, that's certainly impossible. We can fix existing tests, but there should certainly be new ones.

Hehe, yes, that's true.

Maybe it would be best if check_testcase was also run for skipped tests?
From the code it looks like the "before" check_testase is run anyway.
I am thinking of this code:

      if ( $res == 0 ) {

	my $check_res;

	if ( $opt_check_testcases and

	     $check_res= check_testcase($tinfo, "after"))

	{

  ...

      elsif ( $res == 62 ) {

	# Testcase itself tell us to skip this one

Maybe add in the $res==62 case a call to check_testcase() similar to the $res==0 case?

Might make tests take longer, but perhaps not too much, that can be measured. And random failures that depent on test order are troublesome. Anyway, just a suggestion.