Details
-
Bug
-
Status: Closed (View Workflow)
-
Critical
-
Resolution: Cannot Reproduce
-
10.5.15, 10.6.7
-
None
Description
We detected a galera entire galera cluster blocked no DDL operations could be run until we kill -9 writer node behaviour was on 10.6 and 10.5 series
There were such entries in error log:
[Warning] Aborted connection 22233 to db: 'unconnected' user: 'unauthenticated' host: 'IP' (This connection closed normally without authentication)
Root Cause
The Node stuck 'hangout ' was eventually tracked down to an insert update query:
INSERT INTO `bets_cashout_offers` (BetId, CashoutAmount, OfferedTime)VALUES (?betId, ?guaranteedPrice, ?offeredTime)ON DUPLICATE KEY UPDATE CashoutAmount = ?guaranteedPrice, OfferedTime = ?offeredTime;";
the problem was the query was being run as follows:
Connection.Query(sql, new
changing it to the following resolves the issue:
Connection.Execute(sql, new { betId, guaranteedPrice, offeredTime }
);
Given enought time with Connection.Query (30 minutes give or take it will hang entire cluster)