Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-9884

existing /var/lib/mysql/.sst directory (with contents) causes SST to fail with xtrabackup-v2

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Fixed
    • 5.5.48-galera
    • Galera SST
    • None
    • Debian 7 64bit
      xtrabackup version 2.3.3 based on MySQL server 5.6.24 Linux (x86_64) (revision id: 525ca7d)

    Description

      When an SST runs using xtrabackup-v2 I see the following error in syslog on the joiner and the SST is unable to complete no matter how many times I restart mariadb:

      -wsrep-sst-joiner: #007xbstream: Can't create/write to file './xtrabackup_logfile' (Errcode: 17 - File exists)

      Whilst debugging this I eventually found there was still an .sst directory inside /var/lib/mysql that had a bunch of files in it, including xtrabackup_logfle. Once I deleted this directory hen the SST worked.

      My suggested fix it to modify this file:

      /usr/bin/wsrep_sst_xtrabackup-v2

      and change this line:

      mkdir -p ${DATA}/.sst

      to:

      rm -rf ${DATA}/.sst
      mkdir ${DATA}/.sst

      to ensure the .sst directory is always empty as that is what xtrabackup seems to expect. The .sst directory is in the list to be deleted for the default cpat options anyway, but that options does not clear the .sst directory soon enough to allow the sst to work.

      Attachments

        Issue Links

          Activity

            I think this happened due to mysql running out of connections during the SST (not the SSTs fault. It was caused by a load of runaway processes all accessing the db), so the SST just seems to hang forever. If the node then gets rebooted then of course another SST is able to run but the .sst dir stil lexists and is full of stuff.

            Ev Evan Jardine-Skinner added a comment - I think this happened due to mysql running out of connections during the SST (not the SSTs fault. It was caused by a load of runaway processes all accessing the db), so the SST just seems to hang forever. If the node then gets rebooted then of course another SST is able to run but the .sst dir stil lexists and is full of stuff.
            andrew.garner Andrew Garner added a comment -

            FWIW, Percona XtraDB Cluster has fixed this in this commit:

            https://github.com/percona/percona-xtradb-cluster/commit/b3ee75949ed82b88f355ca2e26431350cc1c89ac

            MariaDB did not fix this for some reason, so a failed SST requires manual intervention or future attempts will continue to fail.

            andrew.garner Andrew Garner added a comment - FWIW, Percona XtraDB Cluster has fixed this in this commit: https://github.com/percona/percona-xtradb-cluster/commit/b3ee75949ed82b88f355ca2e26431350cc1c89ac MariaDB did not fix this for some reason, so a failed SST requires manual intervention or future attempts will continue to fail.

            Thank you very much

            Ev Evan Jardine-Skinner added a comment - Thank you very much

            Hi,

            I just downloaded the released source (from here https://downloads.mariadb.org/interstitial/mariadb-galera-5.5.49/source/mariadb-galera-5.5.49.tar.gz/from/http%3A//mirrors.coreix.net/mariadb/) for mariadb-galera-5.5.49 and this fix is not in there. Should the fix version have been 5.5.50 ?

            Ev Evan Jardine-Skinner added a comment - Hi, I just downloaded the released source (from here https://downloads.mariadb.org/interstitial/mariadb-galera-5.5.49/source/mariadb-galera-5.5.49.tar.gz/from/http%3A//mirrors.coreix.net/mariadb/ ) for mariadb-galera-5.5.49 and this fix is not in there. Should the fix version have been 5.5.50 ?
            nirbhay_c Nirbhay Choubey (Inactive) added a comment - - edited

            Ev Hello! Sorry, the fix version was incorrectly tagged. The patch was pushed a day after 5.5.49-galera got released, so will be part of 5.5.50-galera.

            nirbhay_c Nirbhay Choubey (Inactive) added a comment - - edited Ev Hello! Sorry, the fix version was incorrectly tagged. The patch was pushed a day after 5.5.49-galera got released, so will be part of 5.5.50-galera.

            People

              nirbhay_c Nirbhay Choubey (Inactive)
              Ev Evan Jardine-Skinner
              Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.