We have MariaDB 10.4.12 Galera 4 DR ( 3 node Streaming Replication ) , where MariaDB Service is UP on Primary Node and on DR Nodes MariaDB Service is down.
We have uninstalled MariaDB 10.4.12 & Galera 4 Software on both DR nodes without touching Primary Node.
Data is getting Loaded on Primary Node.
We once again installed MariaDB 10.4.12 & Galera 4 Software from Open Source Repository on both DR nodes and deleted content of mysql folder.
When we try to start MariaDB service on DR node , We can see on Primary node wsrep_cluster_size = 2 and it is trying to recover data on DR Nodes and DR Nodes is trying to Join Galera Cluster
It Recovers all data on failed Node but it fails to start the MariaDB Service on DR Nodes.
Below are the contents of Error Log
2020-05-10 15:07:42 0 [ERROR] WSREP: Process completed with error: wsrep_sst_rsync --role 'joiner' --address '10.21.138.33' --datadir '/opt/db/data/mysql/' --parent '50025' --mysqld-args --wsrep_start_position=00000000-0000-0000-0000-000000000000:-1: 3 (No such process)
2020-05-10 15:07:42 0 [ERROR] WSREP: Failed to read uuid:seqno and wsrep_gtid_domain_id from joiner script.
2020-05-10 15:07:42 3 [Note] WSREP: SST received
2020-05-10 15:07:42 3 [Note] WSREP: SST received: 00000000-0000-0000-0000-000000000000:-1
2020-05-10 15:07:42 2 [ERROR] WSREP: Application received wrong state:
2020-05-10 15:07:42 2 [ERROR] WSREP: Application state transfer failed. This is unrecoverable condition, restart required.
rsynced version 3.1.2 starting , listening on port 4444
Core pattern : /usr/libexec/abrt-hook-ccpp %s %c %p %u %g
/usr/bin/galery_recovery : line 71 : 9257 Aborted /usr/sbin/mysqld --user=mysql --wsrep_recover --disable-log-error
2020-03-30 14:17:21 140409262663424 [Warning] Aborted connection 1835102 to db: 'unconnected' user: 'unauthenticated' host: '10.21.47.13' (Got an error reading communication packets)
2020-03-30 16:16:43 140642978871040 [Warning] WSREP: Failed to report last committed 495492282, -110 (Connection timed out)
failed to start MariaDB 10.4.12 database server