Now, back to comparison with mysqld--help, which mostly works smoothly with its windows rdiff.

It turns out mysqld--help does exactly the same that I did in my "radical" solution. Only it does it much thicker, it just unconditionally replaces all "32-bit-looking" values with the corresponding "64-bit-looking" values, and also suppresses some inconvenient variables completely. Still, it has to have the windows rdiff because for Windows it's not just about different values, there are some variables that only exist on Windows, and some that do not exist on Windows. So, again, it correlates with my existing ugly solution, where I get rid of 32-bit rdiffs, but introduce win rdiffs instead. And of course, xtradb rdiff will have to stay as well.

So, my conclusion is this: we cannot target only mature versions with these tests, and maintaining them on early releases is real pain (I had a chance to feel it while working on this). So, the radical solution wins, I am going to get back to it, maybe will be able to make it somewhat less ugly.

But even then, adding or removing variables can break the context of remaining rdiffs. It happens with mysqld--help as well (for example, see 55e99b29339dae833448ded6d0ca3d31f673d02a); only, without plugin variables it happens rarely. We are going to have this problem with our XtraDB rdiff much more often. When other problems are solved, I will need to get back to this one.

Second solution (attached as mdev6887-2.patch) does type and value replacements unconditionally. It's mysqld--help style, which is of course not very precise.
Additional logic for InnoDB version number.
32-bit rdiffs are removed, Windows rdiffs are introduced.

I tried 10.1.1=>10.1.2=>10.1.3=>10.1.4=>10.1.5=>10.1.6 upgrade to see how much maintenance problems it solves.

10.1.1 => 10.1.2

Linux:

server, innodb, xtradb fail

after updating results => xtradb fails (mismatches, rdiff applies smoothly)

Windows (after fixing Linux tests):

server fails (1 hunk, broken context), debug does not build, can't check the rest

10.1.2 => 10.1.3

Linux:

all fail (additions, changes)

after updating results => xtradb fails (1 hunk, broken context)

Windows:

debug and innodb fail (broken context, maybe in the previous version too)

10.1.3 => 10.1.4

Linux:

aria passes, other fail 

after updating results => xtradb fails, 2 hunks, broken context

Windows: 

debug and xtradb fail (broken context)

10.1.4 => 10.1.5

Linux:

server and wsrep fail

Windows:

okay

10.1.5 => 10.1.6

Linux:

wsrep, server, innodb, xtradb failed (additions, changes)

after updating result files: okay

WIndows:

okay

So, only in the last 2 versions all rdiffs worked smoothly, previously they needed to be re-created. Sometimes it's inevitable, when changes affect variables directly covered by rdiffs; but more often it's about the context, neighboring variables get changed, so the hunks cannot be applied.

If it's often about the context, this can be fixed, I guess. We can make sure the context never changes. Either by having less context (diff -U1 for example), or by providing constant context:

select '' as before1, '' as before2, '' as before3, *, '' as after1, '' as after2, '' as after3 from

Yes, I've been thinking about both possibilities.
I don't think we can afford -U1. I don't know all dark magic of patch, but it appears to me that -U1 might do more harm than good. With the current rdiffs, I'm almost certain it would. Here is a typical hunk:

@@ -144,7 +144,7 @@

 VARIABLE_TYPE  BIGINT UNSIGNED

 VARIABLE_COMMENT       The size of the transactional cache for updates to transactional engines for the binary log. If you often use transactions containing many statements, you can increase this to get more performance

 NUMERIC_MIN_VALUE      4096

-NUMERIC_MAX_VALUE      18446744073709551615

+NUMERIC_MAX_VALUE      4294967295

 NUMERIC_BLOCK_SIZE     4096

 ENUM_VALUE_LIST        NULL

 READ_ONLY      NO

So, 1-line context would make it completely indistinguishable; and line numbers alone cannot be enough, because they change even more often than the context.

If we get rid of 32-bit rdiffs, there will be less examples like that, but they still exist:

@@ -1021,7 +1021,7 @@

 COMMAND_LINE_ARGUMENT  NULL

 VARIABLE_NAME  HAVE_CRYPT

 SESSION_VALUE  NULL

-GLOBAL_VALUE   YES

+GLOBAL_VALUE   NO

 GLOBAL_VALUE_ORIGIN    COMPILE-TIME

 DEFAULT_VALUE  NULL

 VARIABLE_SCOPE GLOBAL

For the second solution, again my lack of expertise in patch magic does not allow me to be sure, but I suspect patches which add variables will fail miserably, because they won't know where exactly to add them.

I was going to experiment with both, but I start having a suspicion that some context problems might stay unavoidable.

On a separate note, performance_schema variables should probably be extracted into their own test, because as it is now, the test should fail if the server is compiled without P_S.