mysql failed to log into server on a member of the Galera cluster when it reboots after restore

Hi:

There is a GALERA issue when a member of the Galera Cluster reboots after the restore operation. In this case, mysql (mysteriously) fails to connect to the server, specifically:

[root@mdbaas-demo-app-1 ~]# mysql -p

Enter password:

ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)

There is a manual workaround for this situation - on the impacted member, run the folowing commands:

1. systemctl stop mariadb

2. DATADIR=/mariadb/data # Note: This is the default, set to match your configuration

3. mv $DATADIR/grastate.dat $DATADIR/grastate.dat.OLD

4. systemctl start mariadb

The above manual workaround needs to be reapplied as needed whenever you reboot AFTER executing the restore operation for the Galera configuration. The manual workaround works until the next restore operation. In other words, subsequent reboots continue to work until the operator performs another restore operation. At that point, the workaround needs to be re-applied again.

We only see this issue in Galera configuration, not in Master/2-slave, or Master/Master or standalone configurations. Also, I think the node that performed the restore does not have this problem, only the nodes that get replicated are having this issue.

The backup/restore method was using mysqldump. Wondering if you have any suggestions on why this is happening?

p.s. I do not have a lab setup on this now, if more info is needed, let me know, and I'll find some time to do it.

Thanks,