[MCOL-3786] In 1.4.2, columnstoreRestore checks wrong directory for datadir Created: 2020-02-12  Updated: 2023-10-26  Resolved: 2020-04-08

Status: Closed
Project: MariaDB ColumnStore
Component/s: None
Affects Version/s: 1.4.2
Fix Version/s: 1.4.3

Type: Bug Priority: Major
Reporter: Geoff Montee (Inactive) Assignee: Daniel Lee (Inactive)
Resolution: Fixed Votes: 0
Labels: None

Issue Links:
PartOf
is part of MCOL-3847 MariaDB Website is missing columnstor... Closed

 Description   

In 1.4.2, columnstoreBackup backs up the MariaDB datadir to $umBackupdir/mysql. See the following log snippet:

...
 
[2020-02-11 23:48:11] Backing up UMs
 
[2020-02-11 23:48:11] Backing up um1
 
[2020-02-11 23:48:11] rsync -a -z --delete --link-dest=../backup.1/um1 root@172.30.0.83:/var/lib/mysql /tmp/backup//um1
 
[2020-02-11 23:48:13] DONE
 
...

Based on the above output, we can guess that /var/lib/mysql will be backed up as /tmp/backup/um1/mysql. That is exactly what happens:

$ sudo ls -ld /tmp/backup/um1/mysql
 
drwxr-xr-x. 8 mysql mysql 4096 Feb 11 22:58 /tmp/backup/um1/mysql

Unfortunately, when columnstoreRestore tries to restore /var/lib/mysql, it expects to find the backup at /tmp/backup/um1/db. See the following log snippet:

...
 
[2020-02-12 00:26:56] Restoring UMs
 
[2020-02-12 00:26:56] Restoring um1...
 
[2020-02-12 00:26:56] rsync -a -z --delete /tmp/backup//um1/db/ root@172.30.0.83:/var/lib/mysql
 
[2020-02-12 00:26:56] ERROR: Rsync command failed.
 
[2020-02-12 00:26:56] ERROR Command: rsync -a -z --delete /tmp/backup//um1/db/ root@172.30.0.83:/var/lib/mysql
 
[2020-02-12 00:26:56] Restore Status: Failed

The problem seems to be caused by an incorrect path in this section of the code in columnstoreRestore:

###############################################################################
 
# Restore UM
 
###############################################################################
 
restoreUM () {
 
printLog "Restoring UMs"
 
for (( moduleID=1; moduleID<=$restore_umModuleCount; moduleID++ ))
 
do
 
    umBackupDir="um""$moduleID"
 
    if [ -e $BACKUP_SERVER_LOCATION/$umBackupDir ]; then
 
        printLog "Restoring $umBackupDir..."
 
        executeRsync $DRY_RUN "" $BACKUP_SERVER_LOCATION/$umBackupDir/db/ $REMOTE_USER@${restore_umModuleIP[$moduleID]}:/var/lib/mysql
 
    fi
 
printLog "DONE"
 
done
 
}



 Comments   
Comment by Geoff Montee (Inactive) [ 2020-02-12 ]

I made the following basic change to columnstoreRestore:

--- columnstoreRestore.old      2020-02-12 02:12:01.353118477 +0000
 
+++ /usr/bin/columnstoreRestore 2020-02-12 02:14:40.294720582 +0000
 
@@ -814,7 +814,7 @@
 
     umBackupDir="um""$moduleID"
 
     if [ -e $BACKUP_SERVER_LOCATION/$umBackupDir ]; then
 
         printLog "Restoring $umBackupDir..."
 
-        executeRsync $DRY_RUN "" $BACKUP_SERVER_LOCATION/$umBackupDir/db/ $REMOTE_USER@${restore_umModuleIP[$moduleID]}:/var/lib/mysql
 
+        executeRsync $DRY_RUN "" $BACKUP_SERVER_LOCATION/$umBackupDir/mysql/ $REMOTE_USER@${restore_umModuleIP[$moduleID]}:/var/lib/mysql
 
     fi
 
 printLog "DONE"
 
 done

With this change, I was able to restore my backup.

Comment by David Hill (Inactive) [ 2020-02-21 ]

A user did run into this issue of backup not working. I just wanted to add issue number for future reference

Comment by Ben Thompson (Inactive) [ 2020-03-19 ]

Mostly done by Andrew I added some changes to it and add S3 support per MCOL-3667
https://github.com/mariadb-corporation/mariadb-columnstore-tools/pull/48

Comment by Geoff Montee (Inactive) [ 2020-03-27 ]

ben.thompson,

It looks like your fix may be incomplete.

Your pull request doesn't even fix the specific line that I pointed out in the issue description above. In your current branch, we can see that the restoreUM() function is still copying from $umBackupDir/db/, rather than $umBackupDir/mysql/:

###############################################################################
 
# Restore UM
 
###############################################################################
 
restoreUM () {
 
printLog "Restoring UMs"
 
for (( moduleID=1; moduleID<=$restore_umModuleCount; moduleID++ )) 
do
 
    umBackupDir="um""$moduleID"
 
    if [ -e $BACKUP_SERVER_LOCATION/$umBackupDir ]; then
 
        printLog "Restoring $umBackupDir..."
 
        executeRsync $DRY_RUN "" $BACKUP_SERVER_LOCATION/$umBackupDir/db/ $REMOTE_USER@${restore_umModuleIP[$moduleID]}:@MARIADIR@
 
    fi
 
printLog "DONE"
 
done
 
}

https://github.com/mariadb-corporation/mariadb-columnstore-tools/blob/d37fe97125e2f49e876282c4139257b5585cff71/backuprestore/columnstoreRestore.in#L817

I see that the restorePMs() function also most likely needs to be fixed:

https://github.com/mariadb-corporation/mariadb-columnstore-tools/blob/d37fe97125e2f49e876282c4139257b5585cff71/backuprestore/columnstoreRestore.in#L799

It seems to me that some developer still needs to take a closer look at backuprestore/columnstoreRestore.in, because it doesn't quite look fixed to me yet.

Comment by Ben Thompson (Inactive) [ 2020-04-02 ]

the directory structure is labeled differently but that files system is just local to the backup directory. the destination its restoring should be correct with the @MARIADIR@

renaming the backup directory location from db to mysql is not needed for functionality but could be renamed for consistency.

Comment by Geoff Montee (Inactive) [ 2020-04-02 ]

ben.thompson,

Ah, I see. Thanks for the clarification. I do see now that the code has changed a bit from the original. Thanks!

Comment by Daniel Lee (Inactive) [ 2020-04-08 ]

Build verified:

[dlee@master tools]$ git status

  1. On branch develop
    nothing to commit, working directory clean
    [dlee@master tools]$ git show
    commit cc983d522bfb44ab8b44feaa023133be54da7b13
    Merge: 0c06d9d 28c6b75
    Author: Patrick LeBlanc <43503225+pleblanc1976@users.noreply.github.com>
    Date: Fri Mar 20 09:11:35 2020 -0500

Merge pull request #48 from mariadb-corporation/S3-update

S3 update and fixes

Verified backup and restore for single server, 2pm combo, and 1um2pm configurations. Also verified single server using S3 storage on AWS.

Comment by Daniel Lee (Inactive) [ 2020-04-08 ]

Verified on ColumnStore 1.4.3-6

Generated at Thu Feb 08 02:45:26 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.