I implemented an improvement in MDEV-14425 that allows the server to start up with a 0-sized ib_logfile0 if the FIL_PAGE_FILE_FLUSH_LSN in the system tablespace looks valid (8204 used to be the minimum LSN value). This should allow an upgrade straight after executing mariadb-backup --prepare on the older server version.

The following messages might be made less verbose for the mariadb-backup --prepare in 10.8, but they are correctly identifying the cause:

2022-01-06 15:00:03 0 [ERROR] InnoDB: Upgrade after a crash is not supported. The redo log was created with Backup 10.7.2-MariaDB.

2022-01-06 15:00:03 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error

[00] 2022-01-06 15:00:03 mariadb-backup: srv_start() returned 11 (Generic error).

I will rephrase the first message to say mariadb-backup --prepare instead of InnoDB, and I will suppress the second one.

An attempt to run a newer mariadb-backup --backup against an older server should result in a message like this:

[00] 2022-01-06 15:00:03 Error: cannot process redo log before MariaDB 10.8

As far as I understand, Galera snapshot transfer does not work for a major upgrade at least in the following cases:

• Upgrading from anything older than 10.2: The redo log format was changed in MariaDB Server 10.2.2 (which incorporated that change from MySQL 5.7.9). We may ignore this, because older versions than 10.2 have reached their end of life already, and everyone should have upgraded already.
• 10.2 to 10.3 or 10.4: While the redo log format is compatible, MDEV-12288 changed the undo log format in a way that prevents an upgrade when certain incomplete (or XA PREPARE) transactions are recovered. A check to prevent startup in this case was added in MDEV-15912.
• Upgrade to 10.5: The redo log format was changed in MDEV-12353, and a crash-upgrade is not supported.
• Upgrade to 10.8 with MDEV-14425: The redo log format will be changed, and a crash-upgrade is not supported..

To fix this, I suggest that the snapshot transfer (SST) scripts (both rsync and mariabackup or mariadb-backup variants) be improved in one of the following ways:

1. Add a special SST option or script that would be used (manually or automatically) if the joiner refuses to handle the stream. That special script would apply log on the donor side.
2. Implement a "joiner proxy" that would apply the old-format log in a special way (using old-version executables), and then pass the files to the final joiner.

Note that the XA PREPARE transactions whose existence would prevent an upgrade from 10.2 or earlier will not be automatically removed by a server startup and clean shutdown (something that the rsync method normally skips), nor by mariabackup --prepare or mariadb-backup --prepare.

seppo Can we really reliable do something about this or should we just document that rolling upgrade is not supported between major releases?

I expect that MDEV-25975 fixed this for upgrading from 10.3 or later to later major versions. With that fix, a snapshot transfer should always use a logically empty redo log file. It is important that the SST donor includes a fix of MDEV-25975; the version of the joiner does not matter.

ramesh, can you please test the upgrades again to confirm or refute my claim?

I analyzed one test failure that was with rsync SST, from 10.4 cbdf62ae907ad42ceb7a65e070b821bb45e07be9 (including the fix of MDEV-25975). In that failure, the upgrade to 10.5 failed because the ib_logfile0 contained records that were apparently written by change buffer merge. This bug was corrected in MDEV-28247, which is the very next commit in the 10.4 branch.

When it comes to upgrading from 10.2, there are two problems: neither MDEV-25975 nor MDEV-28247 will be fixed in that branch, and there could be problems related to the 10.3 undo log format changes (such as MDEV-12288) that could prevent an upgrade; the 10.3 or later server would refuse to start up with a message that was introduced in MDEV-15912.

The planned use case for cluster upgrading, is to carry out the node upgrade fast enough so that it can join back through IST. gcache should be adjusted to allow enough headroom for the upgrading. If upgraded node joins back through SST, it will pollute the once upgraded data directory, I feel it is pointless even to try to support SST in cluster upgrading.

marko mariabackup SST works fine in 10.4 > 10.5 upgrade using latest builds. Also no SST failures were found in the 10.5> 10.6 and 10.6> 10.7 upgrades. But mariabackup --prepare does not in 10.7 > 10.8 upgrade

2022-04-07 20:34:25 0 [ERROR] mariadb-backup --prepare is not possible The redo log was created with Backup 10.7.4-MariaDB.

[00] 2022-04-07 20:34:25 mariadb-backup: srv_start() returned 11 (Generic error).

Thank you, ramesh. A MDEV-14425 follow-up fix for the 10.8 mariadb-backup is needed, so that a logically empty older-format redo log file will be tolerated.

ramesh, I see that there is an typo in the message:

diff --git a/storage/innobase/log/log0recv.cc b/storage/innobase/log/log0recv.cc

index 27dce5fa17d..46e3b8a7548 100644

--- a/storage/innobase/log/log0recv.cc

+++ b/storage/innobase/log/log0recv.cc

@@ -1871,7 +1871,7 @@ dberr_t recv_sys_t::find_checkpoint()

     sql_print_error("%s The redo log was created with %s%s",

                     srv_operation == SRV_OPERATION_NORMAL

                     ? "InnoDB: Upgrade after a crash is not supported."

-                    : "mariadb-backup --prepare is not possible", creator,

+                    : "mariadb-backup --prepare is not possible.", creator,

                     (err == DB_ERROR ? "." : ", and it appears corrupted."));

     return err;

   }

But, the root cause of that failure should be that the old-format redo log was not logically empty. Can you provide a copy of that log, and double-check that you tested the currently latest 10.7 revision 2d8e38bc9477aa00b371ed14d95390bede70c5cb that is the oldest 10.7 revision to contain the fix of MDEV-25975?

So far, the reason for the failures appears to be that writes are not being blocked in 10.5 to 10.7 during the Galera snapshot transfer.

I will need a copy of a logically nonempty ib_logfile0 as well as an rr record trace of the donor in order to debug this.

ramesh, the rr replay trace that you claimed to be for a Galera SST donor is not showing any sign of writes being disabled:

Thread 1 received signal SIGUSR1, User defined signal 1.

0x0000000070000002 in ?? ()

(rr) i b

Num     Type           Disp Enb Address            What

1       breakpoint     keep y   0x00005590fd2a9e86 in innodb_disable_internal_writes(bool) at /test/mtest/10.7/storage/innobase/handler/ha_innodb.cc:2003

2       breakpoint     keep y   0x00005590fd275a43 in sst_flush_tables(THD*) 

                                                   at /test/mtest/10.7/sql/wsrep_sst.cc:1514

(rr) when

Current event: 6697222

(rr) c

Continuing.

Thread 1 received signal SIGKILL, Killed.

0x0000000070000002 in ?? ()

(rr) when

Current event: 6728393

(rr) p server_version_source_revision

$1 = 0x5590fd90e628 "2d8e38bc9477aa00b371ed14d95390bede70c5cb"

jplindst, please determine and fix the root cause. I believe that already 10.5 should be affected by this.

marko Root cause is

(rr) p wsrep_sst_method

$2 = 0x5590fef5f503 "mariabackup"

jplindst, I understand that for normal Galera mariabackup SST it makes sense not to block writes, because allowing concurrent writes should be the only benefit over the simpler rsync based snapshot transfer.

If writes are not blocked, I do not think that mariabackup based SST ugprade can possibly work when upgrading from an earlier major version to 10.2 or later, 10.5 or later (MDEV-12353), or 10.8 or later (MDEV-14492).

A minimal solution could be to simply document that mariabackup SST cannot be used for upgrades. This is already enforced by mariabackup or mariadb-backup.

An alternative would be to implement an environment variable in the mariabackup SST script that allows an older-version mariabackup --prepare to be invoked. Something like this:

diff --git a/scripts/wsrep_sst_mariabackup.sh b/scripts/wsrep_sst_mariabackup.sh

index 067a777fb9c..37ff8d10c09 100644

--- a/scripts/wsrep_sst_mariabackup.sh

+++ b/scripts/wsrep_sst_mariabackup.sh

@@ -102,6 +102,7 @@ if [ -z "$BACKUP_BIN" ]; then

     wsrep_log_error 'mariabackup binary not found in path'

     exit 42

 fi

+: ${BACKUP_PREPARE_BIN="$BACKUP_BIN"}

 DATA="$WSREP_SST_OPT_DATA"

 INFO_FILE='xtrabackup_galera_info'

@@ -934,7 +935,7 @@ setup_commands()

     if [ -n "$INNODB_FORCE_RECOVERY" ]; then

         recovery=" --innodb-force-recovery=$INNODB_FORCE_RECOVERY"

     fi

-    INNOAPPLY="$BACKUP_BIN --prepare$disver$recovery${iapts:+ }$iapts$INNOEXTRA --target-dir='$DATA' --datadir='$DATA'$mysqld_args $INNOAPPLY"

+    INNOAPPLY="$BACKUP_PREPARE_BIN --prepare$disver$recovery${iapts:+ }$iapts$INNOEXTRA --target-dir='$DATA' --datadir='$DATA'$mysqld_args $INNOAPPLY"

     INNOMOVE="$BACKUP_BIN$WSREP_SST_OPT_CONF --move-back$disver${impts:+ }$impts$INNOEXTRA --force-non-empty-directories --target-dir='$DATA' --datadir='${TDATA:-$DATA}' $INNOMOVE"

     INNOBACKUP="$BACKUP_BIN$WSREP_SST_OPT_CONF --backup$disver${iopts:+ }$iopts$tmpopts$INNOEXTRA --galera-info --stream=$sfmt --target-dir='$itmpdir' --datadir='$DATA'$mysqld_args $INNOBACKUP"

 }

If the environment variable BACKUP_PREPARE_BIN is set for invoking the joiner, the correct executable would be used for applying the log.

marko with older version mariabackup(hard-coded older binary location in 10.8 sst script) SST worked fine in 10.7 > 10.8 upgrade

ramesh@galapq:/dev/shm/qa$ cat node3/mysql_upgrade_info 

10.8.3-MariaDB

ramesh@galapq:/dev/shm/qa$ cat mariabackup.prepare.log.2022.04.12-14.33.21.938721816

/dev/shm/qa/mariadb-10.7.4-linux-x86_64//bin/mariabackup based on MariaDB server 10.7.4-MariaDB Linux (x86_64)

[00] 2022-04-12 14:28:45 cd to /dev/shm/qa/node3/.sst/

[00] 2022-04-12 14:28:45 open files limit requested 0, set to 1048576

[00] 2022-04-12 14:28:45 Renaming ./mysql/wsrep_streaming_log.new to ./mysql/wsrep_streaming_log.ibd

[..]

2022-04-12 14:28:45 0 [Note] InnoDB: Setting O_DIRECT on file ./ibdata1 failed

2022-04-12 14:28:45 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=41933,41933

2022-04-12 14:28:45 0 [Note] InnoDB: Starting final batch to recover  25 pages from redo log.

[00] 2022-04-12 14:28:45 Last binlog file , position 0

[00] 2022-04-12 14:28:45 completed OK!

ramesh@galapq:/dev/shm/qa$ 

ramesh, thank you. Automating the solution in something like a package installation script would seem to be somewhat challenging, so maybe this work-around should only be documented with a detailed example. An easier way could be that users must simply enable rsync based snapshot transfer before major-version upgrades.

Theoretically, an upgrade with mariabackup could work if the user made a copy (or hard link) of the old backup executable and then pointed an environment variable or a configuration parameter to it, for the duration of upgrading the server. An interface for specifying the full path to the executable would be needed in wsrep_sst_mariabackup, as demonstrated by the patch that I posted. After the upgrade, the old executable could be removed and the configuration restored to normal (so that a snapshot transfer will continue to work from a current-version donor).

I quickly checked the Debian packaging scripts. Baking such upgrade logic directly into those scripts would be very challenging to implement and test. Something like the following would happen:

1. mariadb-server-10.7.prerm prepares for the removal of the 10.7 executables (tries to shut down the server)
2. the 10.7 executables are removed
3. the 10.8 executables are installed
4. mariadb-server-10.8.preinst tries to shut down any old server process again
5. mariadb-server-10.8.postinst starts the 10.8 server

marko Agree with using rsync SST during major upgrades. The 10.7> 10.8 upgrade works well on rsync SST.

I read this issue on request by marko but I am not sure if I understood it fully. Event with potentially lacking understanding I would strongly advice the following:

1. Only change redo log format on major releases, never in minor releases. If this happens, make sure the release notes of the major version says that the new version redo log format is no longer backwards compatible and sysadmins and automation software needs to do X, Y, and Z when upgrading to new major version.
2. This is hard to fix in Linux distro packaging and anyways doing such tricks in the package manager is against the purpose of packaging scripts. To begin with, there is no guarantee that the server is even running when the upgrade happens. The distro packages primary function is to update plain files. Packagign scripts should not have code about data format detection and conversion. Secondly, any fix in packaging scripts in deb must also be done in rpm and in all other packaging scripts whatever they are. Would be hard to maintain and likely cause a lot of failures anyway.
3. The only place outside the server and client/utility binaries that handles the start/stop/restart process is the systemd and init files. They could potentially have some logic about doing graceful restart in certain situations, and those files are shared across all Linux distros and container systems etc so the maintenance burden could be manageable.
4. Best place to fix this is in the server and client/helper binaries by the same people do introduce a potential backwards incompatible change. This is identical to what Marko wrote in the comment from April 14th about solutions 1&2.

But indeed, priority number 1 is to document such changes in release notes of major releases. Those are read by sysadmins and that is the place when sysadmins expect they might have to occasionally do something extra to make the upgrade work right.

Thank you, otto. This ticket is indeed about upgrades between major versions. We do avoid changes to file formats after a release series has been declared generally available (GA). There are some exceptions, such as MDEV-11623, MDEV-13564, MDEV-25004, perhaps also MDEV-25440 if you consider it to be a file format change. When implementing such changes, special attention must be (and has been) paid on upgrades, as well as downgrades within the same release series (say, 10.3.35 to 10.3.34).

I see that the default setting is wsrep_sst_method=rsync. According to the tests conducted by ramesh, major version upgrades using that method will work reliably if the donor (running the older major version) includes the fixes of MDEV-25975 and MDEV-28247.

I updated the following pages to document the limitations around major version upgrades:
https://mariadb.com/kb/en/introduction-to-state-snapshot-transfers-ssts/
https://mariadb.com/kb/en/mariabackup-sst-method/