I have since been able to replicate the problem on a stand-alone node without Galera and SST involved. After upgrading from 10.2 to 10.3, nightly mariabackup tool started throwing the same error about PROCESS privilege being missing from the user. Granting the user ALL privileges fixed the issue.
Can you provide output from show grants for sst user and user executing the sst.
Unfortunately, I didn't snapshot the privileges before I set the user to GRANT ALL PRIVILEGES. However, I'm pretty certain that the original privileges were set as recommended by your documentation:
GRANT RELOAD, PROCESS, LOCK TABLES, REPLICATION CLIENT ON . TO 'mariabackup'@'localhost';
Did this list of required privileges change between 10.2 and 10.3?
In my understanding similar access rights should be granted for user used on sst i.e. on your case galeraSST
So it now happened to us on both a Galera SST AND a stand-alone node. In both cases it complained about PROCESS privilege not being set. But in both cases, the user had PROCESS privilege and only when we gave the user ALL privileges mariabackup started working.
Hi kvasserman,
What is your value of wsrep_sst_auth?
https://mariadb.com/kb/en/library/mariabackup-sst-method/#authentication-and-privileges
As jplindst pointed out, you've provided the privileges of the 'mariabackup'@'localhost' user, but your output suggests that the SST was using the galeraSST user instead:
Feb 11 00:50:48 maria3 -innobackupex-backup: 190211 00:50:48 Connecting to MySQL server host: localhost, user: galeraSST, password: set, port: 3306, socket: /var/run/mysqld/mysqld.sock
|
wsrep_sst_auth=galeraSST:<password reducted>
as I have explained galeraSST user had PROCESS privilege, but mariabackup failed with the above error until I gave that user ALL privileges.
It then happened on a completely different server that is not a Galera node and SST was not involved. The same issue, the user already had PROCESS privilege, but mariabackup failed complaining that the user doesn't have it, until I gave that user ALL privileges.
In both cases, problem happened after upgrade from 10.2 to 10.3
Can you , for diagnostic purposes, paste the output for following command, for the galeraSST user?
SHOW GRANTS FOR 'galeraSST'@'localhost'
You don't believe me that it NOW has ALL privileges and is working with this configuration? 
Here it is:
GRANT ALL PRIVILEGES ON . TO 'galeraSST'@'localhost' IDENTIFIED BY PASSWORD '*XXXXXXXXXX'
Since I had to give the user all privileges to make it work, I don't have the old privileges. But they were set as per your documentation.
I believe you that all privileges work, however we do have a unit test that tests just required privileges.
So either the parsing went somehow wrong, or something else was fishy in your case, but it is hard to tell what, without obtaining original GRANT, where mariabackup would fail.
Here is the test, it succeeds if RELOAD and PROCESS are granted on all databases
and it fails otherwise
https://github.com/MariaDB/server/blob/10.4/mysql-test/suite/mariabackup/backup_grants.test
I know, it's unfortunate that I didn't save the old GRANTs but I'm pretty sure they were:
GRANT RELOAD, PROCESS, LOCK TABLES, REPLICATION CLIENT ON . TO 'galeraSST'@'localhost' .....
I just did a manual test with mariabackup and mysqld from 10.3.13 using the privileges in the documentation, and that worked:
$ sudo mariabackup --backup \
|
> --target-dir=/home/ec2-user/backup/ \
|
> --user=mariabackup --password=mypassword
|
[00] 2019-03-13 17:41:03 Connecting to MySQL server host: localhost, user: mariabackup, password: set, port: not set, socket: not set
|
[00] 2019-03-13 17:41:03 Using server version 10.3.13-MariaDB-log
|
[00] 2019-03-13 17:41:03 mariabackup based on MariaDB server 10.3.13-MariaDB Linux (x86_64)
|
[00] 2019-03-13 17:41:03 uses posix_fadvise().
|
[00] 2019-03-13 17:41:03 cd to /var/lib/mysql/
|
...
|
[00] 2019-03-13 17:41:05 completed OK!
|
$ mysql -u root --execute "SHOW GRANTS FOR 'mariabackup'@'localhost'"
|
+---------------------------------------------------------------------------------------------------------------------------------------------------------------+
|
| Grants for mariabackup@localhost |
|
+---------------------------------------------------------------------------------------------------------------------------------------------------------------+
|
| GRANT RELOAD, PROCESS, LOCK TABLES, REPLICATION CLIENT ON *.* TO 'mariabackup'@'localhost' IDENTIFIED BY PASSWORD '*FABE5482D5AADF36D028AC443D117BE1180B9725' |
|
+---------------------------------------------------------------------------------------------------------------------------------------------------------------+
|
Did you create user in 10.2 and then upgraded to 10.3? Because this is when I got the error: after upgrading to 10.3. It worked well in 10.2.
There is really not enough info to process it, thus I added some code to dump current grants in mariabackup output, in case it ends with "Insufficient privileges".
If bug reappears, or will be reproducible somehow, this will make fixing it easy.