Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-15910

Galera SST Log encryption error with mariabackup

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 10.1, 10.2, 10.3
    • Fix Version/s: 10.1, 10.2, 10.3
    • Component/s: Galera SST
    • Labels:
      None

      Description

      I am able to successfully use galera with mariabackup with no encryption of SST logs (default encrypt=0).
      however, the galera SST log file is unencrypted when using mariabackup with galera.

      edit:
      I am also able to successfully use galera with xtrapackup-v2 with SST encryption=3
      but I get an error with SST transfer on the joiner node if i switch the backup method to mariabackup.

      ============
      I tried setting the encrypt=1 setting as described on https://www.percona.com/doc/percona-xtradb-cluster/LATEST/manual/xtrabackup_sst.html
      and get the error message:
      WSREP_SST: [ERROR] Xtrabackup-based encryption is currently not supported with MariaBackup (20180417 18:32:36.154)

      response from Vladislav: this mode is not supported for mariabackup!
      ============
      when using with encrypt=2, a different error results-
      2018-04-17 18:53:43 139677235410688 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'joiner' --address '10.180.48.128' --datadir '/var/lib/mysql/' --parent '1' --binlog '/var/log/mysql/mariadb-bin' '
      WSREP_SST: [INFO] Logging all stderr of SST/Innobackupex to syslog (20180417 18:53:43.358)
      2018-04-17 18:53:43 139677235410688 [ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_mariabackup --role 'joiner' --address '10.180.48.128' --datadir '/var/lib/mysql/' --parent '1' --binlog '/var/log/mysql/mariadb-bin'
      Read: '(null)'
      2018-04-17 18:53:43 139677235410688 [ERROR] WSREP: Process completed with error: wsrep_sst_mariabackup --role 'joiner' --address '10.180.48.128' --datadir '/var/lib/mysql/' --parent '1' --binlog '/var/log/mysql/mariadb-bin' : 22 (Invalid argument)
      2018-04-17 18:53:43 139677441746688 [ERROR] WSREP: Failed to prepare for 'mariabackup' SST. Unrecoverable.
      2018-04-17 18:53:43 139677441746688 [ERROR] Aborting

      response from Vladislav: possibley a galera error!
      =====
      new edit:
      using encrypt mode=3 with mariabackup
      settings in my.cnf:
      [sst]
      streamfmt=xbstream
      encrypt = 3
      tca = /var/lib/mysql/ca-cert.pem
      tkey = /var/lib/mysql/server1-key.pem
      tcert = /var/lib/mysql/server1-cert.pem

      error log on joiner node:
      2018-04-18 16:09:40 140673811871488 [Note] WSREP: Service thread queue flushed.
      2018-04-18 16:09:40 140673803175680 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (c088b3c0-4322-11e8-8adf-6e9903c6f8b1): 1 (Operation not permitted)
      at galera/src/replicator_str.cpp:prepare_for_IST():482. IST will be unavailable.
      2018-04-18 16:09:40 140673560635136 [Note] WSREP: Member 2.0 (b36490ef44c1) requested state transfer from 'any'. Selected 0.0 (54d23c7d7634)(SYNCED) as donor.
      2018-04-18 16:09:40 140673560635136 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 0)
      2018-04-18 16:09:40 140673803175680 [Note] WSREP: Requesting state transfer: success, donor: 0
      2018-04-18 16:09:40 140673803175680 [Note] WSREP: GCache history reset: 00000000-0000-0000-0000-000000000000:0 -> c088b3c0-4322-11e8-8adf-6e9903c6f8b1:0
      2018-04-18 16:09:41 140673552242432 [ERROR] WSREP: Process completed with error: wsrep_sst_mariabackup --role 'joiner' --address '10.180.48.128' --datadir '/var/lib/mysql/' --parent '1' --binlog '/var/log/mysql/mariadb-bin' : 32 (Broken pipe)
      2018-04-18 16:09:41 140673552242432 [ERROR] WSREP: Failed to read uuid:seqno and wsrep_gtid_domain_id from joiner script.
      2018-04-18 16:09:41 140673960392896 [ERROR] WSREP: SST failed: 32 (Broken pipe)
      2018-04-18 16:09:41 140673960392896 [ERROR] Aborting

      2018-04-18 16:09:41 140673560635136 [Warning] WSREP: 0.0 (54d23c7d7634): State transfer to 2.0 (b36490ef44c1) failed: -32 (Broken pipe)
      2018-04-18 16:09:41 140673560635136 [ERROR] WSREP: gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():737: Will never receive state. Need to abort.
      180418 16:09:41 [ERROR] mysqld got signal 11 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.

      To report this bug, see https://mariadb.com/kb/en/reporting-bugs

      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed,
      something is definitely wrong and this may fail.
      =====================

      Not sure if this issue belongs on the galera Jira or Mariabackup jira board

        Attachments

          Activity

            People

            • Assignee:
              jplindst Jan Lindström
              Reporter:
              kaushik netmaster
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: