Details
-
Bug
-
Status: Closed (View Workflow)
-
Critical
-
Resolution: Fixed
-
10.1.18
-
None
-
Debian 8.6, packages from MariaDB repo, Galera replication enabled
-
10.1.20
Description
Enabling SSL for SST results in a a split brain when a new node joins the cluster:
Oct 1 15:01:20 1 -innobackupex-backup: 161001 15:01:20 [01] Encrypting and streaming ./ibdata1
Oct 1 15:01:21 1 -wsrep-sst-donor: 2016/10/01 15:01:21 socat[15723] E write(13, 0xc16220, 8192): Broken pipe
Oct 1 15:01:21 1 -innobackupex-backup: #007innobackupex: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)
Oct 1 15:01:21 1 -innobackupex-backup: xb_stream_write_data() failed.
Oct 1 15:01:21 1 -innobackupex-backup: encrypt: write to the destination file failed.
Oct 1 15:01:21 1 -innobackupex-backup: #007innobackupex: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)
Oct 1 15:01:21 1 -innobackupex-backup: [01] xtrabackup: Error: xtrabackup_copy_datafile() failed.
Oct 1 15:01:21 1 -innobackupex-backup: [01] xtrabackup: Error: failed to copy datafile.
Oct 1 15:01:21 1 mysqld[15281]: 2016-10-01 15:01:21 140606113225472 [Warning] Aborted connection 19 to db: 'unconnected' user: 'root' host: 'localhost' (Got an error re
ading communication packets)
Oct 1 15:01:21 1 -wsrep-sst-donor: innobackupex finished with error: 1. Check /var/lib/mysql//innobackup.backup.log
Oct 1 15:01:21 1 -wsrep-sst-donor: Cleanup after exit with status:22
Oct 1 15:01:21 1 -wsrep-sst-donor: Cleaning up temporary directories
Oct 1 15:01:21 1 mysqld[15281]: 2016-10-01 15:01:21 140604830906112 [ERROR] WSREP: Failed to read from: wsrep_sst_xtrabackup-v2 --role 'donor' --address '1.1.1.1
:4444/xtrabackup_sst//1' --socket '/var/run/mysqld/mysqld.sock' --datadir '/var/lib/mysql/' '' --gtid '2320744f-86e5-11e6-9fd8-87f46ed48225:2' --gtid-domain-id '0'
Oct 1 15:01:21 1 mysqld[15281]: 2016-10-01 15:01:21 140604830906112 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '1.1.1.1:4444/xtrabackup_sst//1' --socket '/var/run/mysqld/mysqld.sock' --datadir '/var/lib/mysql/' '' --gtid '2320744f-86e5-11e6-9fd8-87f46ed48225:2' --gtid-domain
-id '0': 22 (Invalid argument)
Oct 1 15:01:21 1 mysqld[15281]: 2016-10-01 15:01:21 140604830906112 [ERROR] WSREP: Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '1.1.1.1
:4444/xtrabackup_sst//1' --socket '/var/run/mysqld/mysqld.sock' --datadir '/var/lib/mysql/' '' --gtid '2320744f-86e5-11e6-9fd8-87f46ed48225:2' --gtid-domain-id '0'
Oct 1 15:01:21 1 mysqld[15281]: 2016-10-01 15:01:21 140605682349824 [Warning] WSREP: 1.0 (server1): State transfer to 0.0 (server2) failed: -22 (Invalid argument)
Logs have been sanitized of course (IP/hostnames).
Configuration files were taken from a production MariaDB cluster running Galera replication. The only changes made were:
1) Enabled SSL where applicable (server/client/wsrep/SST)
2) Changed passwords (verified to be correct on both nodes)
3) Copied debian.cnf from server1 to server2
Here are the .conf settings:
[sst]
encrypt=1
encrypt-algo=AES128
encrypt-key=[some long key]
tca=/somepath/ca.pem
tcert=/somepath/combined.pem
[some long key] is of course edited, there is a string generated from openssl, without the square brackets.
somepath is a sanitized path
Tried with encrypt=1 and encrypt=3 and still failing.
wsrep_sst_method=xtrabackup-v2 declared in a [mysqld] section.
SST succeeds when streamfmt is set to tar, but of course that is unencrypted. This is not a firewall issue, it has been verified that there are rules enabling all the nodes to talk to each other.
Tried with 10.1.17 and 10.1.18 released today.
Attachments
Issue Links
- relates to
-
MDEV-9403 When using xtrabackup-v2 SST, socat + SSL fails on CentOS/RHEL 6
- Closed
-
MDEV-14010 merge issue in wsrep_sst_xtrabackup-v2
- Closed
-
MDEV-14011 (draft) wsrep_sst_xtrabackup-v2 sst error
- Closed