We received the below error on our production server, which caused the database server to crash. Despite being a 3 node cluster, we direct all read and write operations to one node, using the other 2 nodes as hot spares. Upon failure, all access redirected to one of the other nodes in the cluster with no issue.

I have attached the syslog containing all the events that occured afterwards in case they are useful. Of note, it appears that the first restart of the server, had trouble synchronising because of an unexpected primary key. I suspect our on-call engineers then deleted the DB, forcing an SST. I have trimmed the log where it is copying and moving several files.

Oct 21 15:51:58 MEDX-C1 mysqld: 2016-10-21 15:51:58 7f3905a40b00  InnoDB: Assertion failure in thread 139882884500224 in file rem0rec.cc line 580