[MCOL-1573] recommended change to the rsync.sh script Created: 2018-07-20  Updated: 2023-10-26  Resolved: 2020-04-15

Status: Closed
Project: MariaDB ColumnStore
Component/s: ?
Affects Version/s: 1.1.5
Fix Version/s: N/A

Type: New Feature Priority: Minor
Reporter: David Hill (Inactive) Assignee: Todd Stoffel (Inactive)
Resolution: Won't Do Votes: 2
Labels: None

Attachments: File rsync.sh    
Issue Links:
PartOf
includes MCOL-1063 Multi server upgrade ERROR in replica... Closed
includes MCOL-1665 test schema not rsync during replicat... Closed
Epic Link: ColumnStore Utility Improvements

 Description   

Input from a community user about improvement in the rsync.sh script, which will equalize the front-end um from um1 (pm1) to the other nodes when mysql-replication is enabled

I notice that when we call mcsadmin enableMySQLReplication one of the steps is the call to the script /usr/local/mariadb/columnstore/bin/rsync.sh

If I understand well this is used to create the "zero point" each time, before activate the standard mysql replication.

What I'm not understanding is why this script not use the delete option of the rsync.

Here the problem:
1) I disable the MySQL replication with mcsadmin disableMySQLReplication
2) On the Primary Front-End MariaDB ColumnStore Module (pm1 /um1) I call a simple DROP TABLE or a DROP DATABASE
3) I enable the MySQL replication with mcsadmin enableMySQLReplication

In this case what I'm expecting is that also on the other UM (or pm if combinated) the table or the database that i drop were removed.
Instead all the ibd and frm files remain in place.

Other thing I not understand is why we exclude "test/" (test database), is useful for some reason? (according to https://github.com/mariadb-corporation/mariadb-columnstore-server/blob/master/scripts/mysql_secure_installation.sh I've removed it from PM1)
I have also not idea if it's better to rsync also the mysql database or not.

I suggest you to upgrade the rsync.sh script with the "--delete" option, the "-axvz" options instead of "-vopgr" and removing the --exclude=mysql/ and --exclude=test/ clause, somethings like:

set COMMAND "rsync -axvz --delete -e ssh --exclude=infinidb_vtable/ --exclude=infinidb_querystats/ --exclude=calpontsys/ --include=/ --include=/* --exclude=* $INSTALLDIR/mysql/db/ $USERNAME@$SERVER:$INSTALLDIR/mysql/db/"



 Comments   
Comment by Nico [ 2019-01-23 ]

According to MCOL-1063 and after some test it's better:

set COMMAND "rsync -az --stats --delete -e ssh --exclude=mysql/ --exclude=infinidb_vtable/ --exclude=infinidb_querystats/ --exclude=calpontsys/ --include=*/ --include=*/* --exclude=* $INSTALLDIR/mysql/db/ $USERNAME@$SERVER:$INSTALLDIR/mysql/db/"

Comment by Nico [ 2019-01-25 ]

After some test I refactor a bit the rsync.sh
rsync.sh
In this version "expect" doesn't wait the timeout to go over in some case were it sent the password. In the previous version the second declaration of expect never match nothing in case of 'ssh' and so it wait for the timeout.
I'm also increasing the timeout to 600 sec. The previous one on big database can fall in problems.

I sill have a problem when I call it with ma enableMySQLReplication :
If the rsync takes more then X seconds (I think a minute) the command above return:
**** enableRep Failed : API Failure return in enableMySQLRep API

I check in the debug.log and I found this:
{{Jan 25 09:40:49 prod-cs-1121 ProcessManager[2929]: 49.305394 |0|0|0| E 17 CAL0000: line: 6901 sendMsgProcMon: ProcMon Msg timeout on module pm1
Jan 25 09:40:49 prod-cs-1121 ProcessManager[2929]: 49.305508 |0|0|0| E 17 CAL0000: line: 11251 setMySQLReplication: ERROR: Error getting MySQL Replication Master Information
Jan 25 09:40:49 prod-cs-1121 ProcessManager[2929]: 49.305547 |0|0|0| I 17 CAL0000: Enable MySQL Replication status: 1}}

Calling ma enableMySQLReplication more times, rsync takes progressive less time and the replication finish correctly.
So, I think you need to increase the timeout for this command, it's still too short, I think 600sec is enough.

I hope can be included in next releases.

Thanks

Comment by Todd Stoffel (Inactive) [ 2020-04-15 ]

OAM is being deprecated and replaced by an enhanced API and the MaxScale orchestration project.

Generated at Thu Feb 08 02:29:45 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.