Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 10.1.18
    • Fix Version/s: N/A
    • Component/s: Galera SST
    • Labels:
      None
    • Environment:
      Debian 8.6, packages from MariaDB repo, Galera replication enabled
    • Sprint:
      10.1.20

      Description

      Enabling SSL for SST results in a a split brain when a new node joins the cluster:

      Oct 1 15:01:20 1 -innobackupex-backup: 161001 15:01:20 [01] Encrypting and streaming ./ibdata1
      Oct 1 15:01:21 1 -wsrep-sst-donor: 2016/10/01 15:01:21 socat[15723] E write(13, 0xc16220, 8192): Broken pipe
      Oct 1 15:01:21 1 -innobackupex-backup: #007innobackupex: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)
      Oct 1 15:01:21 1 -innobackupex-backup: xb_stream_write_data() failed.
      Oct 1 15:01:21 1 -innobackupex-backup: encrypt: write to the destination file failed.
      Oct 1 15:01:21 1 -innobackupex-backup: #007innobackupex: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)
      Oct 1 15:01:21 1 -innobackupex-backup: [01] xtrabackup: Error: xtrabackup_copy_datafile() failed.
      Oct 1 15:01:21 1 -innobackupex-backup: [01] xtrabackup: Error: failed to copy datafile.
      Oct 1 15:01:21 1 mysqld[15281]: 2016-10-01 15:01:21 140606113225472 [Warning] Aborted connection 19 to db: 'unconnected' user: 'root' host: 'localhost' (Got an error re
      ading communication packets)
      Oct 1 15:01:21 1 -wsrep-sst-donor: innobackupex finished with error: 1. Check /var/lib/mysql//innobackup.backup.log
      Oct 1 15:01:21 1 -wsrep-sst-donor: Cleanup after exit with status:22
      Oct 1 15:01:21 1 -wsrep-sst-donor: Cleaning up temporary directories
      Oct 1 15:01:21 1 mysqld[15281]: 2016-10-01 15:01:21 140604830906112 [ERROR] WSREP: Failed to read from: wsrep_sst_xtrabackup-v2 --role 'donor' --address '1.1.1.1
      :4444/xtrabackup_sst//1' --socket '/var/run/mysqld/mysqld.sock' --datadir '/var/lib/mysql/' '' --gtid '2320744f-86e5-11e6-9fd8-87f46ed48225:2' --gtid-domain-id '0'
      Oct 1 15:01:21 1 mysqld[15281]: 2016-10-01 15:01:21 140604830906112 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '1.1.1.1:4444/xtrabackup_sst//1' --socket '/var/run/mysqld/mysqld.sock' --datadir '/var/lib/mysql/' '' --gtid '2320744f-86e5-11e6-9fd8-87f46ed48225:2' --gtid-domain
      -id '0': 22 (Invalid argument)
      Oct 1 15:01:21 1 mysqld[15281]: 2016-10-01 15:01:21 140604830906112 [ERROR] WSREP: Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '1.1.1.1
      :4444/xtrabackup_sst//1' --socket '/var/run/mysqld/mysqld.sock' --datadir '/var/lib/mysql/' '' --gtid '2320744f-86e5-11e6-9fd8-87f46ed48225:2' --gtid-domain-id '0'
      Oct 1 15:01:21 1 mysqld[15281]: 2016-10-01 15:01:21 140605682349824 [Warning] WSREP: 1.0 (server1): State transfer to 0.0 (server2) failed: -22 (Invalid argument)

      Logs have been sanitized of course (IP/hostnames).

      Configuration files were taken from a production MariaDB cluster running Galera replication. The only changes made were:
      1) Enabled SSL where applicable (server/client/wsrep/SST)
      2) Changed passwords (verified to be correct on both nodes)
      3) Copied debian.cnf from server1 to server2

      Here are the .conf settings:
      [sst]
      encrypt=1
      encrypt-algo=AES128
      encrypt-key=[some long key]
      tca=/somepath/ca.pem
      tcert=/somepath/combined.pem

      [some long key] is of course edited, there is a string generated from openssl, without the square brackets.
      somepath is a sanitized path

      Tried with encrypt=1 and encrypt=3 and still failing.

      wsrep_sst_method=xtrabackup-v2 declared in a [mysqld] section.

      SST succeeds when streamfmt is set to tar, but of course that is unencrypted. This is not a firewall issue, it has been verified that there are rules enabling all the nodes to talk to each other.

      Tried with 10.1.17 and 10.1.18 released today.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                anikitin Andrii Nikitin (Inactive)
                Reporter:
                dezillium DEZILLIUM LIMITED
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: