Backport MCOL-5352 to 23.02: first DDL/DML after a PrimProc restart fails once with MCS-2045

Summary

MCOL-5352 ("Truncate table failed after PrimProc restarted") is closed, fixed in 23.10.2 by commit 7f14dae5c ("Properly handle PP restart for DML/DDL operations"). That commit is not in 23.02.24 (git cherry shows it absent). Please backport it to 23.02.

Impact

Seen on our 3-node test cluster (23.02.24): after a PrimProc restart the first DDL or DML statement fails once with MCS-2045 ("At least one PrimProc closed the connection unexpectedly"), and the next one works.

Where it happens (tag 23.02.24, 4e11b0ed9)

• dbcon/joblist/distributedenginecomm.cpp writeToClient() catch block (about :1089-1131): the connection to PrimProc is re-established only after a write has failed. The failed write clears the session queues and pushes a zero-length message, which turns the statement in flight into MCS-2045. It then re-runs Setup(), and the next statement succeeds.
• The same mechanism in mysqld's own DEC is described in the comment on MCOL-6602.

Suggested fix

Cherry-pick 7f14dae5c to the 23.02 branch. It is sizeable (31 files, about 700 lines in dbcon/ddlpackageproc, dbcon/dmlpackageproc, ddlproc and dmlproc), so expect conflicts to resolve.

Related

• MCOL-5352: the original fix. This ticket only asks for the 23.02 backport.

Source

Found while investigating a customer issue on 23.02.24. Reproduction steps and logs from our test cluster can be shared as needed.