We have this issue as well on Debian Jessie with Mariadb 10.2.10
Hi,
Any news / plans for this issue?
BR
johan
johan-severalnines did suggested workaround help you? You can try also remove 'u' letter from the first line of wsrep_sst_xtrabackup-v2 script, so it is just "#!/bin/bash -e" or put `set +u` at start of sst script.
Hi,
Thanks, but I rather have this fixed and released than resorting to workarounds.
In any case I changed to use mariabackup instead, which might have been the intention with this 
.
BR
johan
Here's another good workaround that doesn't require editing any files included in the distribution:
mkdir -p /etc/systemd/system/mariadb.service.d/
|
printf '[Service]\nEnvironment="WSREP_SST_OPT_PORT=4444"\n' > /etc/systemd/system/mariadb.service.d/MDEV-14256.conf
|
systemctl daemon-reload
|
Can we take a moment to just reflect on how absurd it is that this bug was released to production? A simple test case which is, invoke an SST transfer with xtrabackup-v2 would have caught this issue. People like me are facing production outages due to this issue. A lot of faith has just been lost in the MariaDB product.
Yeah, I agree with Eric that the tests for mariadb 10.2 could really use some improvements. I expect some bugs in the first tagged stable release of a new major version (10.2.6) but the following releases have also had some bugs that can make it unusable in certain usecases.
This has also been mentioned by "DEZILLIUM LIMITED" in his description at https://jira.mariadb.org/browse/MDEV-14255 :
Quote:
MariaDB is broken (again) on Debian.
|
10.2.6: broken libmariadb3
|
10.2.7: unreleased for Debian
|
10.2.8: broken libmariadb3
|
10.2.9: working
|
10.2.10: broken sst
|
At least for Debian 8 and 9 this means we had exactly 1 working MariaDB 10.2 stable release, which was 10.2.9. And now, it's broken again.
With 10.1 we never had big issues like this, and we still don't. I get that no one puts in these bugs on purpose and I appreciate the effort put in by the dev team, but it would be really re-assuring to have some feedback from the dev team regarding the prevention of these issues. Are there any plans to improve the tests? Does the release of mariadb 10.2 not feel messier than 10.1 to you?
As a sidenote, since everyone
running MariaDB 10.2/Galera now with xtrabackup-v2 will break their installation when they upgrade to the latest version, is this not hotfix worthy?
This will indeed break it for everyone using xtrabackup-v2 but they might not realise it until it is too late. The upgrade procedure itself might only trigger an IST and they would not realise that SST is broken until they reboot a node later on etc.
All but one of my production nodes broke upon upgrade (via nightly cron job). The only reason one node survived is because it had a broken yum config which prevented new packages from being updated. I was able to run my infrastructure on one node for 24 hours until I found the problem but this could easily have caused a production outage for me.
The bugs affects those systems which don't have explicit port specified in wsrep configuration.
To demonstrate the problem with docker containers following scripts may be used:
tearup - will create docker image with installed 10.2 Server and xtrabackup 2.4
test - will create two containers using image from tearup step and try to setup cluster between them.
When this line is uncommented - the fix is applied and no problem happens anymore.
sachin.setiya.007 please review following patch to address the problem in 10.2
diff --git a/scripts/wsrep_sst_xtrabackup-v2.sh b/scripts/wsrep_sst_xtrabackup-v2.sh |
index 40e686d4d6b..f6bda7db499 100644
|
--- a/scripts/wsrep_sst_xtrabackup-v2.sh |
+++ b/scripts/wsrep_sst_xtrabackup-v2.sh |
@@ -638,12 +638,12 @@ kill_xtrabackup()
|
setup_ports()
|
{
|
if [[ "$WSREP_SST_OPT_ROLE" == "donor" ]];then |
- SST_PORT=$WSREP_SST_OPT_PORT
|
+ SST_PORT=${WSREP_SST_OPT_PORT:-}
|
REMOTEIP=$WSREP_SST_OPT_HOST
|
lsn=$(echo $WSREP_SST_OPT_PATH | awk -F '[/]' '{ print $2 }') |
sst_ver=$(echo $WSREP_SST_OPT_PATH | awk -F '[/]' '{ print $3 }') |
else |
- SST_PORT=$WSREP_SST_OPT_PORT
|
+ SST_PORT=${WSREP_SST_OPT_PORT:-}
|
fi |
}
|
|
anikitin what change introduced this bug? How comes it didn't fail before?
serg it looks it was merged in this commit, before that it did parse address expression directly:
https://github.com/MariaDB/server/commit/83664e21e4fb6755c8c0c90d3dee8819d36928c9#diff-cca56af3f0ce3e7f4fbc13dc62cc2823R640
I'm not sure that was it. Old code used
'--address') |
readonly WSREP_SST_OPT_ADDR="$2" |
...
|
SST_PORT=$(echo ${WSREP_SST_OPT_ADDR} | awk -F ':' '{ print $2 }') |
That would've thrown an error if --address is not used. New code does
'--address') |
readonly WSREP_SST_OPT_ADDR="$2" |
...
|
readonly WSREP_SST_OPT_PORT=$(echo $WSREP_SST_OPT_ADDR | \ |
cut -d ']' -f 2 | cut -s -d ':' -f 2 | cut -d '/' -f 1) |
...
|
SST_PORT=$WSREP_SST_OPT_PORT
|
Assuming that --address is used (because the old code didn't fail), I don't see how WSREP_SST_OPT_PORT could be unset. There was a later relevant commit — 4c2c057d404 — but I don't see how it could've left WSREP_SST_OPT_PORT unset either.
Yes, correct. Then after fix for MDEV-13968 it became WSREP_SST_OPT_ADDR_PORT which was initialized in --address and WSREP_SST_OPT_PORT left unset
Actually probably patch from duplicate MDEV-14299 is better than mine suggested above
Committed a patch
serg The patch is good and I've verified it by patching 10.2.10 in docker image from earlier case like this https://github.com/AndriiNikitin/bugs/blob/master/MDEV-14256-test1.sh#L24