Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-18544

"missing required privilege PROCESS on *.*" using mariabackup for SST

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Incomplete
    • 10.3.12
    • N/A
    • Galera SST, mariabackup
    • None
    • Ubuntu 18.04

    Description

      We had a 10.2 Galera cluster that used mariabackup as SST method. Everything worked fine, until we've upgraded to 10.3.

      With 10.3, SST started failing on donor node with "missing required privilege PROCESS on ." error. When we checked the privileges on the account used for SST, it already had PROCESS privilege (and that didn't change since 10.2). We tried granting it again, just in case, that didn't help.

      We attempted mariabackup --backup with the same account and it worked fine. So the problem appears to be specific to mariabackup used as SST. Only after we've granted ALL privileges to the account, SST began to function.

      Here is an excerpt from syslog on the donor node that is related to this issue:

      ----------
      Feb 11 00:50:38 maria3 mysqld[26260]: 2019-02-11 0:50:38 0 [Note] WSREP: Member 1.0 (maria1) requested state transfer from 'maria3,maria2,'. Selected 2.0 (maria3)(SYNCED) as
      donor.
      Feb 11 00:50:38 maria3 mysqld[26260]: 2019-02-11 0:50:38 0 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 863845639)
      Feb 11 00:50:38 maria3 mysqld[26260]: 2019-02-11 0:50:38 12 [Note] WSREP: IST request: d7a81744-d48f-11e7-abeb-9efa07dcbffd:863832526-863845639|ssl://maria1:4568
      Feb 11 00:50:38 maria3 mysqld[26260]: 2019-02-11 0:50:38 12 [Note] WSREP: IST first seqno 863832527 not found from cache, falling back to SST
      Feb 11 00:50:38 maria3 mysqld[26260]: 2019-02-11 0:50:38 12 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      Feb 11 00:50:38 maria3 mysqld[26260]: 2019-02-11 0:50:38 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'donor' --address 'maria1:4444/xtrabackup_sst//1' --socket '/
      var/run/mysqld/mysqld.sock' --datadir '/data/maria/' --binlog '/data/logs/mariadb-bin' --gtid 'd7a81744-d48f-11e7-abeb-9efa07dcbffd:863845639' --gtid-domain-id '0''
      Feb 11 00:50:38 maria3 mysqld[26260]: 2019-02-11 0:50:38 12 [Note] WSREP: sst_donor_thread signaled with 0
      Feb 11 00:50:38 maria3 mysqld[26260]: WSREP_SST: [INFO] Logging all stderr of SST/Innobackupex to syslog (20190211 00:50:38.513)
      Feb 11 00:50:38 maria3 -wsrep-sst-donor: Streaming with xbstream
      Feb 11 00:50:38 maria3 -wsrep-sst-donor: Using socat as streamer
      Feb 11 00:50:38 maria3 -wsrep-sst-donor: Using /tmp/tmp.VNQEQmWK7T as innobackupex temporary directory
      Feb 11 00:50:38 maria3 -wsrep-sst-donor: Streaming GTID file before SST
      Feb 11 00:50:38 maria3 -wsrep-sst-donor: Evaluating mbstream -c ${INFO_FILE} | pigz --fast | socat -u stdio TCP:maria1:4444 ; RC=( ${PIPESTATUS[@]} )
      Feb 11 00:50:38 maria3 -wsrep-sst-donor: Sleeping before data transfer for SST
      Feb 11 00:50:39 maria3 mysqld[26260]: 2019-02-11 0:50:39 0 [Note] WSREP: (e565f0f1, 'ssl://0.0.0.0:4567') turning message relay requesting off
      Feb 11 00:50:48 maria3 -wsrep-sst-donor: Streaming the backup to joiner at maria1 4444
      Feb 11 00:50:48 maria3 -wsrep-sst-donor: Evaluating mariabackup --innobackupex $tmpopts $INNOEXTRA --galera-info --stream=$sfmt $itmpdir 2> >(logger -p daemon.err -t -in
      nobackupex-backup) | pigz --fast | socat -u stdio TCP:maria1:4444; RC=( ${PIPESTATUS[@]} )
      Feb 11 00:50:48 maria3 -innobackupex-backup: 190211 00:50:48 innobackupex: Starting the backup operation
      Feb 11 00:50:48 maria3 -innobackupex-backup:
      Feb 11 00:50:48 maria3 -innobackupex-backup: IMPORTANT: Please check that the backup run completes successfully.
      Feb 11 00:50:48 maria3 -innobackupex-backup: At the end of a successful backup run innobackupex
      Feb 11 00:50:48 maria3 -innobackupex-backup: prints "completed OK!".
      Feb 11 00:50:48 maria3 -innobackupex-backup:
      Feb 11 00:50:48 maria3 -innobackupex-backup: 190211 00:50:48 Connecting to MySQL server host: localhost, user: galeraSST, password: set, port: 3306, socket: /var/run/mysqld/m
      ysqld.sock
      Feb 11 00:50:48 maria3 -innobackupex-backup: Using server version 10.3.12-MariaDB-1:10.3.12+maria~bionic-log
      Feb 11 00:50:48 maria3 -innobackupex-backup: Error: missing required privilege PROCESS on .
      Feb 11 00:50:48 maria3 -wsrep-sst-donor: mariabackup finished with error: 1. Check /data/maria//innobackup.backup.log
      Feb 11 00:50:48 maria3 -wsrep-sst-donor: Cleanup after exit with status:22
      Feb 11 00:50:48 maria3 -wsrep-sst-donor: Cleaning up temporary directories
      Feb 11 00:50:48 maria3 mysqld[26260]: 2019-02-11 0:50:48 0 [ERROR] WSREP: Failed to read from: wsrep_sst_mariabackup --role 'donor' --address 'maria1:4444/xtrabackup_sst//1' --socket '/var/run/mysqld/mysqld.sock' --datadir '/data/maria/' --binlog '/data/logs/mariadb-bin' --gtid 'd7a81744-d48f-11e7-abeb-9efa07dcbffd:863845639' --gtid-domain-id '0'
      Feb 11 00:50:48 maria3 mysqld[26260]: 2019-02-11 0:50:48 0 [ERROR] WSREP: Process completed with error: wsrep_sst_mariabackup --role 'donor' --address 'maria1:4444/xtrabackup_sst//1' --socket '/var/run/mysqld/mysqld.sock' --datadir '/data/maria/' --binlog '/data/logs/mariadb-bin' --gtid 'd7a81744-d48f-11e7-abeb-9efa07dcbffd:863845639' --gtid-domain-id '0': 22 (Invalid argument)
      Feb 11 00:50:48 maria3 mysqld[26260]: 2019-02-11 0:50:48 0 [ERROR] WSREP: Command did not run: wsrep_sst_mariabackup --role 'donor' --address 'maria1:4444/xtrabackup_sst//1' --socket '/var/run/mysqld/mysqld.sock' --datadir '/data/maria/' --binlog '/data/logs/mariadb-bin' --gtid 'd7a81744-d48f-11e7-abeb-9efa07dcbffd:863845639' --gtid-domain-id '0'
      Feb 11 00:50:48 maria3 mysqld[26260]: 2019-02-11 0:50:48 0 [Warning] WSREP: 2.0 (maria3): State transfer to 1.0 (maria1) failed: -22 (Invalid argument)
      Feb 11 00:50:48 maria3 mysqld[26260]: 2019-02-11 0:50:48 0 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 863845666)

      Attachments

        Activity

          wlad Vladislav Vaintroub added a comment - - edited

          I believe you that all privileges work, however we do have a unit test that tests just required privileges.

          So either the parsing went somehow wrong, or something else was fishy in your case, but it is hard to tell what, without obtaining original GRANT, where mariabackup would fail.

          Here is the test, it succeeds if RELOAD and PROCESS are granted on all databases
          and it fails otherwise
          https://github.com/MariaDB/server/blob/10.4/mysql-test/suite/mariabackup/backup_grants.test

          wlad Vladislav Vaintroub added a comment - - edited I believe you that all privileges work, however we do have a unit test that tests just required privileges. So either the parsing went somehow wrong, or something else was fishy in your case, but it is hard to tell what, without obtaining original GRANT, where mariabackup would fail. Here is the test, it succeeds if RELOAD and PROCESS are granted on all databases and it fails otherwise https://github.com/MariaDB/server/blob/10.4/mysql-test/suite/mariabackup/backup_grants.test

          I know, it's unfortunate that I didn't save the old GRANTs but I'm pretty sure they were:

          GRANT RELOAD, PROCESS, LOCK TABLES, REPLICATION CLIENT ON . TO 'galeraSST'@'localhost' .....

          kvasserman Konstantin Vasserman added a comment - I know, it's unfortunate that I didn't save the old GRANTs but I'm pretty sure they were: GRANT RELOAD, PROCESS, LOCK TABLES, REPLICATION CLIENT ON . TO 'galeraSST'@'localhost' .....

          I just did a manual test with mariabackup and mysqld from 10.3.13 using the privileges in the documentation, and that worked:

          $ sudo mariabackup --backup \
          >    --target-dir=/home/ec2-user/backup/ \
          >    --user=mariabackup --password=mypassword
          [00] 2019-03-13 17:41:03 Connecting to MySQL server host: localhost, user: mariabackup, password: set, port: not set, socket: not set
          [00] 2019-03-13 17:41:03 Using server version 10.3.13-MariaDB-log
          [00] 2019-03-13 17:41:03 mariabackup based on MariaDB server 10.3.13-MariaDB Linux (x86_64)
          [00] 2019-03-13 17:41:03 uses posix_fadvise().
          [00] 2019-03-13 17:41:03 cd to /var/lib/mysql/
          ...
          [00] 2019-03-13 17:41:05 completed OK!
          $ mysql -u root --execute "SHOW GRANTS FOR 'mariabackup'@'localhost'"
          +---------------------------------------------------------------------------------------------------------------------------------------------------------------+
          | Grants for mariabackup@localhost                                                                                                                              |
          +---------------------------------------------------------------------------------------------------------------------------------------------------------------+
          | GRANT RELOAD, PROCESS, LOCK TABLES, REPLICATION CLIENT ON *.* TO 'mariabackup'@'localhost' IDENTIFIED BY PASSWORD '*FABE5482D5AADF36D028AC443D117BE1180B9725' |
          +---------------------------------------------------------------------------------------------------------------------------------------------------------------+
          

          GeoffMontee Geoff Montee (Inactive) added a comment - I just did a manual test with mariabackup and mysqld from 10.3.13 using the privileges in the documentation, and that worked: $ sudo mariabackup --backup \ > --target-dir=/home/ec2-user/backup/ \ > --user=mariabackup --password=mypassword [00] 2019-03-13 17:41:03 Connecting to MySQL server host: localhost, user: mariabackup, password: set, port: not set, socket: not set [00] 2019-03-13 17:41:03 Using server version 10.3.13-MariaDB-log [00] 2019-03-13 17:41:03 mariabackup based on MariaDB server 10.3.13-MariaDB Linux (x86_64) [00] 2019-03-13 17:41:03 uses posix_fadvise(). [00] 2019-03-13 17:41:03 cd to /var/lib/mysql/ ... [00] 2019-03-13 17:41:05 completed OK! $ mysql -u root --execute "SHOW GRANTS FOR 'mariabackup'@'localhost'" +---------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Grants for mariabackup@localhost | +---------------------------------------------------------------------------------------------------------------------------------------------------------------+ | GRANT RELOAD, PROCESS, LOCK TABLES, REPLICATION CLIENT ON *.* TO 'mariabackup'@'localhost' IDENTIFIED BY PASSWORD '*FABE5482D5AADF36D028AC443D117BE1180B9725' | +---------------------------------------------------------------------------------------------------------------------------------------------------------------+

          Did you create user in 10.2 and then upgraded to 10.3? Because this is when I got the error: after upgrading to 10.3. It worked well in 10.2.

          kvasserman Konstantin Vasserman added a comment - Did you create user in 10.2 and then upgraded to 10.3? Because this is when I got the error: after upgrading to 10.3. It worked well in 10.2.

          There is really not enough info to process it, thus I added some code to dump current grants in mariabackup output, in case it ends with "Insufficient privileges".

          If bug reappears, or will be reproducible somehow, this will make fixing it easy.

          wlad Vladislav Vaintroub added a comment - There is really not enough info to process it, thus I added some code to dump current grants in mariabackup output, in case it ends with "Insufficient privileges". If bug reappears, or will be reproducible somehow, this will make fixing it easy.

          People

            wlad Vladislav Vaintroub
            kvasserman Konstantin Vasserman
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.