Right now, when using rsync SST, the rsync messages go to syslog. This is confusing as a system administrator as MariaDB has its own log, and even contains SOME messages regarding rsync. The issue is that these errors in the MariaDB log regarding rsync are highly cryptic or misleading, since the real information was actually sent to syslog. Since the rsync SST is part of the MariaDB Galera process, all logs regarding this particular process should remain together.
When rsync fails for any reason, MariaDB's error log will show something like the following. The error message implies that MariaDB possibly crashed (terminated unexpectedly) when the actual issue was rsync erroring out.
2020-02-11 15:11:23 0 [Note] WSREP: PC protocol downgrade 1 -> 0
2020-02-11 15:11:23 0 [Note] WSREP: view((empty))
2020-02-11 15:11:23 0 [Note] WSREP: gcomm: closed
2020-02-11 15:11:23 0 [Note] WSREP: mysqld: Terminated.
WSREP_SST: [ERROR] Parent mysqld process (PID:26635) terminated unexpectedly. (20200211 15:11:24.N)
/usr/local/bin/wsrep_sst_rsync: line 506: kill: (-26635) - No such process
WSREP_SST: [INFO] Joiner cleanup. rsync PID: 26693 (20200211 15:11:24.N)
WSREP_SST: [INFO] Joiner cleanup done. (20200211 15:11:24.N)
This particular instance came up due to a ZFS configuration that I had on a particular server. However, other rsync errors happen from time to time too, such as networking errors. When diagnosing the issue with
MDEV-21709, this is the error message presented in the syslog which would have been significantly more helpful (much less time consuming as a sysadmin) to just have it in the MariaDB error.log, so less hunting/debugging/coalescing logs is needed in the field.
Feb 11 00:09:49 sea-cospix-maria-02 rsyncd: rsync: readdir(".zfs" (in rsync_sst)): Invalid argument (22)
Feb 11 00:09:49 sea-cospix-maria-02 rsyncd: rsync: delete_file: rmdir(.zfs/snapshot) failed: Operation not supported (45)