4QA
This change should be tested using multinode cluster based on NFS where all dbroots are writable from all nodes. The scenario that we fix by this issue is:

• shutdown a cluster
• check /var/lib/columnstore/data1/systemFiles/dbrm/* timestamps. The current stable release writes multiple times and there is a race b/w nodes saving dbrm files.
  JFYI save_brm is called from mcs-workernode@ 1 and @ 2 systemd units. You can add an artificial delay to be able to detect the race.
  This fix must resolve the race so that the primary writes dbrm files so that check /var/lib/columnstore/data1/systemFiles/dbrm/* timestamps shows that dbrm files are written only once.

Plz test a failure scenario when the primary is lost before the shutdown and: a) failover finishes, b) failover doesn't have time to converge but the cluster is shutdown

Build verified: 23.02 (Drone build# 6492)

Verified on a 3 node cluster with NFS shared storage

Checked timestamps for all files in the dbrm directory every .5 seconds during the test scenarios.

Timestamp for dbrm files changed only one time.
1) cluster stop
2) failover completed
3) failed node rejoining cluster
4) cluster stopped during failover

For scenario #4, stopping cluster while primary node is down caused this messages, which is expected since s1pm2 was the primary node.

[rocky8:root@rocky8~]# mcs cluster stop
{
"timestamp": "2023-01-18 14:27:40.921970",
"s1pm2": "HTTPSConnectionPool(host='s1pm2', port=8640): Max retries exceeded with url: /cmapi/0.4.0/node/shutdown (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f4afa702dd0>: Failed to establish a new connection: [Errno 113] No route to host'))",
"s1pm3":

,
"s1pm1":

}