[MDEV-31536] Galera mariadb-backup to work as SST between major versions Created: 2023-06-25  Updated: 2024-01-17

Status: Open
Project: MariaDB Server
Component/s: Galera SST
Fix Version/s: None

Type: Task Priority: Major
Reporter: Daniel Black Assignee: Daniel Black
Resolution: Unresolved Votes: 2
Labels: None

Issue Links:
Relates
relates to MDEV-27437 Galera snapshot transfer fails to upg... Closed
relates to MDEV-28483 Include the details about 10.4 to 10.... In Progress
relates to MDEV-29162 Major upgrade instructions missing fo... Closed
relates to MDEV-31506 Updating cluster to a new major versi... Confirmed

 Description   

For a HA solution like galera to require downtime to perform an upgrade inconvenience for the users. A traditional async replication based upgrade can be done without downtime, so our HA product Galera should be able to as well/

Problem: (MDEV-27347 summary)

The mariadb-backup sst doesn't work between major version as the donor streams the result to the joiner. This stream and the saved version afterwards has not had its redo log applied and is effectively crashed. The joiner mariadb-backup being a newer version won't apply the changes in the prepare stage.

Fix:

The mariadb-backup prepare part of the SST could be performed by the donor using a sshfs mount SSHFS exported by the JOINER. This enables the mariadb-backup of the right version to apply the redo log without error.

tested (out of a sst context) with:

socat TCP-LISTEN:4444,reuseaddr EXEC:"/usr/libexec/openssh/sftp-server -e -d $DATA"

The xtrabackup_info info file contains the donor server version so this can be made conditional.

Donor:

localdir=$(mktemp -d)
 
socat TCP:$REMOTEIP:4444  EXEC:"sshfs \:$DATA $localdir -o passive" &
 
# or sshfs $REMOTEIP:$DATA $localdir -o directport=4444 &
 
fusepid=$!
 
mariabackup --target-dir=$localdir --prepare
 
kill $fusepid

Note, not socat -u - bidirection is needed

Alternate untested modification for the sst script on the joiner side:

diff --git a/scripts/wsrep_sst_mariabackup.sh b/scripts/wsrep_sst_mariabackup.sh
 
index d6334052f24..c00fc57b390 100644
 
--- a/scripts/wsrep_sst_mariabackup.sh
 
+++ b/scripts/wsrep_sst_mariabackup.sh
 
@@ -1512,7 +1512,10 @@ else # joiner
 
 
         wsrep_log_info "Preparing the backup at $DATA"
 
         setup_commands
 
-        timeit 'mariadb-backup prepare stage' "$INNOAPPLY"
 
+        DONOR_SERVER_VERSION=$(grep server_version $DATA/xtrabackup_info)
 
+        DONOR_SERVER_VERSION=${DONOR_SERVER_VERSION#* = }
 
+        DONOR_SERVER_VERSION=${DONOR_SERVER_VERSION%%-*}
 
+        timeit 'mariadb-backup prepare stage' docker run --rm -v $DATA:/var/lib/mysql:z mariadb:$DONOR_SERVER_VERSION mariadb-backup --prepare > "$INNOAPPLYLOG" 2>&1
 
         if [ $? -ne 0 ]; then
 
             wsrep_log_error "mariadb-backup apply finished with errors." \
 
                             "Check syslog or '$INNOAPPLYLOG' for details."

--defaults-file=/var/lib/mysql/backup-my.cnf might be needed as the first arg on mariadb-backup if non-standard innodb options related to storage are used.
Generated at Thu Feb 08 10:24:36 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.