I got a notification from DigitalOcean that one of the nodes in my cluster was on a blade with faulty hardware. They decided to hot migrate my VM to a different blade.

After migration was completed, the node never "rejoined the cluster" instead acted idle. After a day of being idle, the whole cluster went into a sync deadlock.

Read actions were still allowed, however, write actions would get stuck in the "Committing" status and never finish. Rebooting the offending node did not fix the issue because the remaining nodes were still stuck in sync deadlock. I had to reboot the whole cluster to get the read/write to work again.

1) I cannot replicate this issue, because I do not have the ability to mess with network/VM settings on the back end like the DigitalOcean admins can.
2) There are no stack traces in the logs, no errors, so I cannot show more information.
3) Doing "service mysql stop" would not kill wsrep PID, i had to kill -9 it myself
4) I have a large DB (300GB) the nodes got so out of sync that after reboot, the whole DB had to be rsynced over to the new node. This took 2x longer than initially when i added the nodes.