[MDEV-9884] existing /var/lib/mysql/.sst directory (with contents) causes SST to fail with xtrabackup-v2 Created: 2016-04-07  Updated: 2016-05-05  Resolved: 2016-04-27

Status: Closed
Project: MariaDB Server
Component/s: Galera SST
Affects Version/s: 5.5.48-galera
Fix Version/s: 10.1.14, 5.5.50-galera, 10.0.25-galera

Type: Bug Priority: Major
Reporter: Evan Jardine-Skinner Assignee: Nirbhay Choubey (Inactive)
Resolution: Fixed Votes: 1
Labels: None
Environment:

Debian 7 64bit
xtrabackup version 2.3.3 based on MySQL server 5.6.24 Linux (x86_64) (revision id: 525ca7d)


 Description   

When an SST runs using xtrabackup-v2 I see the following error in syslog on the joiner and the SST is unable to complete no matter how many times I restart mariadb:

-wsrep-sst-joiner: #007xbstream: Can't create/write to file './xtrabackup_logfile' (Errcode: 17 - File exists)

Whilst debugging this I eventually found there was still an .sst directory inside /var/lib/mysql that had a bunch of files in it, including xtrabackup_logfle. Once I deleted this directory hen the SST worked.

My suggested fix it to modify this file:

/usr/bin/wsrep_sst_xtrabackup-v2

and change this line:

mkdir -p ${DATA}/.sst

to:

rm -rf ${DATA}/.sst
mkdir ${DATA}/.sst

to ensure the .sst directory is always empty as that is what xtrabackup seems to expect. The .sst directory is in the list to be deleted for the default cpat options anyway, but that options does not clear the .sst directory soon enough to allow the sst to work.

 Comments   
Comment by Evan Jardine-Skinner [ 2016-04-07 ]

I think this happened due to mysql running out of connections during the SST (not the SSTs fault. It was caused by a load of runaway processes all accessing the db), so the SST just seems to hang forever. If the node then gets rebooted then of course another SST is able to run but the .sst dir stil lexists and is full of stuff.

Comment by Andrew Garner [ 2016-04-27 ]

FWIW, Percona XtraDB Cluster has fixed this in this commit:

https://github.com/percona/percona-xtradb-cluster/commit/b3ee75949ed82b88f355ca2e26431350cc1c89ac

MariaDB did not fix this for some reason, so a failed SST requires manual intervention or future attempts will continue to fail.

Comment by Evan Jardine-Skinner [ 2016-04-28 ]

Thank you very much

Comment by Evan Jardine-Skinner [ 2016-04-28 ]

Hi,

I just downloaded the released source (from here https://downloads.mariadb.org/interstitial/mariadb-galera-5.5.49/source/mariadb-galera-5.5.49.tar.gz/from/http%3A//mirrors.coreix.net/mariadb/) for mariadb-galera-5.5.49 and this fix is not in there. Should the fix version have been 5.5.50 ?

Comment by Nirbhay Choubey (Inactive) [ 2016-04-28 ]

Ev Hello! Sorry, the fix version was incorrectly tagged. The patch was pushed a day after 5.5.49-galera got released, so will be part of 5.5.50-galera.

Generated at Thu Feb 08 07:38:03 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.