Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-17713

MariaDB 10.1.26 fails SST with xtrabackup-v2/encrypt=4

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 10.1.26
    • Fix Version/s: 10.1
    • Component/s: Galera SST
    • Labels:
      None
    • Environment:
      Debian 9 / MariaDB 10.1.26-0+deb9u1 through standard Debian repo

      Description

      Hello,

      We're using MariaDB 10.1 configured with SSL on Debian 9 and have been using it for about a year now without issues. As far as I am aware, SST has worked previously but since we don't need it very often (normally just IST) I am not sure when the issue was introduced.

      We recently needed a full SST for one of our nodes within the 3-node Galera cluster. I will include the config options I think are relevant:

      /etc/mysql/my.cnf

      [client]
      <snipped>
      ssl=1
      ssl-ca=/etc/ssl/certs/icingasql01-mariadb-ca.pem
      ssl-cert=/etc/ssl/certs/icingasql01-mariadb-cert.pem
      ssl-key=/etc/ssl/private/icingasql01-mariadb-key.pem
       
      [mysqld_safe]
      <snipped>
       
      [mysqld]
      <snipped>
      ssl-ca=/etc/ssl/certs/icingasql01-mariadb-ca.pem
      ssl-cert=/etc/ssl/certs/icingasql01-mariadb-cert.pem
      ssl-key=/etc/ssl/private/icingasql01-mariadb-key.pem
       
      [mysqldump]
      <snipped>
       
      [isamchk]
      <snipped>
      

      /etc/mysql/conf.d/00-galera.cnf:

      [mysqld]
      wsrep_on=ON
      wsrep_provider=/usr/lib/galera/libgalera_smm.so
      wsrep_cluster_name='galera01_icinga'
      wsrep_node_address=<redacted>
      wsrep_cluster_address='gcomm://<redacted>'
      wsrep_sst_method=xtrabackup-v2
      wsrep_sst_auth=sst_xtrabackup:<redacted>
      wsrep_retry_autocommit=4
      wsrep_slave_threads=4
      wsrep_provider_options="gcache.size=256M;socket.ssl=yes;socket.ssl_key=/etc/ssl/private/icingasql01-mariadb-key.pem;socket.ssl_cert=/etc/ssl/certs/icingasql01-mariadb-cert.pem;socket.ssl_ca=/etc/ssl/certs/icingasql01-mariadb-ca.pem"
      binlog_format=ROW
      innodb_autoinc_lock_mode=2
      innodb_doublewrite=1
      innodb_log_file_size=512M
      innodb_flush_log_at_trx_commit=0
      enforce_storage_engine=innodb
      sql_mode="NO_AUTO_CREATE_USER"
      [sst]
      encrypt=4
      ssl-ca=/etc/ssl/certs/icingasql01-mariadb-ca.pem
      ssl-cert=/etc/ssl/certs/icingasql01-mariadb-cert.pem
      ssl-key=/etc/ssl/private/icingasql01-mariadb-key.pem
      

      The certificates have chmod 0440 and ownership root:ssl-cert. The mysql user is part of the ssl-cert group and is able to access the cert/key from as far as I can see.

      During the SST, the donor shows the following error:

      WSREP_SST: [ERROR] innobackupex finished with error: 1.  Check /var/lib/mysql//innobackup.backup.log (20181114 12:00:13.940)
      

      I have added a set -x in the xtrabackup script and ran the exact command being executed, this shows the following (identical to what is in the above logfile)

      innobackupex --no-version-check --user=sst_xtrabackup --password=<redacted> --socket=/var/run/mysqld/mysqld.sock --galera-info --stream=xbstream /tmp/tmp.oFMtzcQalr
      xtrabackup: recognized server arguments: --datadir=/var/lib/mysql --tmpdir=/tmp --innodb_buffer_pool_size=997M --innodb_log_buffer_size=16M --innodb_file_per_table=1 --innodb_open_files=2000 --innodb_io_capacity=4000 --innodb_flush_method=O_DIRECT --open_files_limit=65536 --innodb_doublewrite=1 --innodb_log_file_size=512M --innodb_flush_log_at_trx_commit=0
      WARNING: --ssl is deprecated and will be removed in a future version. Use --ssl-mode instead.
      xtrabackup: recognized client arguments: --datadir=/var/lib/mysql --tmpdir=/tmp --innodb_buffer_pool_size=997M --innodb_log_buffer_size=16M --innodb_file_per_table=1 --innodb_open_files=2000 --innodb_io_capacity=4000 --innodb_flush_method=O_DIRECT --open_files_limit=65536 --innodb_doublewrite=1 --innodb_log_file_size=512M --innodb_flush_log_at_trx_commit=0 --ssl=1 --ssl-ca=/etc/ssl/certs/icingasql02-mariadb-ca.pem --ssl-cert=/etc/ssl/certs/icingasql02-mariadb-cert.pem --ssl-key=/etc/ssl/private/icingasql02-mariadb-key.pem
      encryption: using gcrypt 1.7.6-beta                                            
      181114 12:02:40 innobackupex: Starting the backup operation
                                  
      IMPORTANT: Please check that the backup run completes successfully.          
                 At the end of a successful backup run innobackupex            
                 prints "completed OK!".                                             
                                                          
      181114 12:02:40 Connecting to MySQL server host: localhost, user: sst_xtrabackup, password: set, port: 3306, socket: /var/run/mysqld/mysqld.sock
      Failed to connect to MySQL server: Access denied for user 'sst_xtrabackup'@'localhost' (using password: YES).
      

      I have the following xtrabackup scripts:

      apt-cache policy percona-xtrabackup-24
      percona-xtrabackup-24:
        Installed: 2.4.12-1.stretch
        Candidate: 2.4.12-1.stretch
        Version table:
       *** 2.4.12-1.stretch 500
              500 http://repo.percona.com/apt stretch/main amd64 Packages
              100 /var/lib/dpkg/status
      

      Manually connecting with this user does work. When I try to connect with --no-defaults, it does not work. This is because SSL is required. We configured the SST user like this:

      GRANT ALL PRIVILEGES ON *.* TO 'sst_xtrabackup'@'127.0.0.1' IDENTIFIED BY PASSWORD '<redacted>' REQUIRE SSL WITH GRANT OPTION
      

      After I removed the REQUIRE SSL part, SST did work correctly. However since we have galera running across DCs we want to use SSL instead. Is this misconfiguration or a bug?

        Attachments

          Activity

            People

            • Assignee:
              jplindst Jan Lindström
              Reporter:
              nielsh Niels Hendriks
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: