Multiple engine daemons stop accepting connections after a TCP RST during accept()

On a multi-node cluster, a single TCP connection to an internal ColumnStore port that is closed with RST during the accept handshake throws:

CompressedInetStreamSocket::accept sync: Connection reset by peer

The daemons do not handle this consistently. WriteEngineServer and DDLProc catch the exception, keep running with the same PID, but never accept on their port again. The process stays up, so mcs cluster status reports it online and the cluster read-write, while every DDL/DML that needs it fails. Recovery requires restarting the affected service or the cluster.

This was discovered after a customer had issues during a Nessus scan similar to MCOL-5339 and MCOL-5378, but this is now easily repeatable.

Steps to reproduce:

• Confirm the cluster is read-write and WriteEngineServer is listening on the primary:

  ss -ltnp | grep :8630

  mcs cluster status

• From any host that can reach the primary, open one connection to 8630 and close it with

  RST: python3 -c "import socket,struct;s=socket.create_connection(('PRIMARY_IP',8630));s.setsockopt(socket.SOL_SOCKET,socket.SO_LINGER,struct.pack('ii',1,0));s.close()"

• Check the listener and the process:

  ss -ltnp | grep :8630

  pgrep -a WriteEngineServer

  mcs cluster status

• Run a DDL:

  CREATE TABLE test.t (a INT) ENGINE=Columnstore;

Relevant log entry:

writeengineserver[7129]: 29.360489 |0|0|0| C 32 CAL0000: WriteEngineServer : Exception caught on accept(): CompressedInetStreamSocket::accept sync: Connection reset by peer %%10%%