mcs-loadbrm pull of BRM files from the primary's CMAPI has no integrity check, retry or consistent snapshot

Summary

On a replica, mcs-loadbrm fetches the extent map, journal, VBBM and VSS from the primary's CMAPI as four separate HTTP requests, each a single urlopen().read() with no retry, checksum or common snapshot. A cut transfer fails the load; a transfer of files from different save generations, or a file read while savebrm rewrites it, loads "successfully" with a bad image. If the primary's CMAPI is unreachable the script silently loads local files instead.

Impact

• A failed pull fails the load, and the DBRM worker then starts anyway on an empty image (see the sibling ticket on starting after a failed load). This is how a short transfer turns into an empty node.
• The "successful but wrong image" cases (below) give a replica an EM that differs from the controller's. It is noticed only at the next write that allocates an extent, when DBRM goes read-only. The code paths are verified; that they happen in practice is inferred.
• The log is misleading: "Cmapi is not running on primary node. Skip loading metafiles." is printed after every successful pull, and the real unreachable-primary fallback logs nothing.

Where it happens (tag 23.02.24, 4e11b0ed9)

Client, oam/install_scripts/mcs-loadbrm.py.in

• get_meta (:203-250): one urlopen(request, context=ctx, timeout=60).read() (:229-230). No size, checksum or footprint check. Short-body detection is whatever http.client does: with a Content-Length it raises IncompleteRead; without one, a cut body is accepted at EOF. Exceptions are logged and re-raised.
• Replica path (:371-388): cmapi_available(primary_address), then a loop over em, journal, vbbm, vss writing each to /tmp/columnstore_tmp_files/rdwrscratch/BRM_saves_<type> (:29). Any exception exits 1 (:397-402), with no retry, leaving earlier files unloaded in that directory.
• The else: at :393-396 belongs to the for, not to if cmapi_available, so the "Skip loading metafiles" line prints after every successful pull. When the primary's CMAPI is unreachable nothing is pulled, nothing is logged, and load_brm loads the configured local files at the hard-coded un-suffixed prefix (:414), not what BRM_saves_current names.

Server, cmapi/cmapi_server/controllers/endpoints.py

• get_brm_bytes (:877-888): re-reads BRM_saves_current, then read_bytes() of the named file, per request. No lock against a concurrent save and no generation in the response.
• get_footprint (:920-931) returns a hard-coded dummy MD5 for every element, and loadbrm never calls it.

Ways to load a bad image with exit code 0 (code paths verified, triggers inferred)

1. Consistent-but-short body: the file is read while savebrm rewrites it in place; Content-Length matches the short body, so the client sees a complete response.
2. Mixed snapshot: a save on the primary between two of the four requests gives an EM from one generation and VBBM/VSS/journal from another.
3. Primary CMAPI unreachable: silent fallback to local files, which may be this node's own stale save.

Evidence

We did not reproduce a truncated transfer on demand, and this ticket does not rest on a reproduction.

• On our 3-node test cluster (23.02.24) the EM transfer from the new primary's CMAPI was cut short (client saw IncompleteRead, about 340 KB into a 4.2 MB body, clean FIN, no reset) during two earlier failovers under memory pressure. One later, traced attempt (strace plus packet capture) transferred the full EM and loaded cleanly. We have not found what closes the connection. Candidates (inferred, unproven) are CMAPI's bundled cheroot connection management under load, or an external restart of the CMAPI service. Pinning it down needs strace and a capture on the serving node during a failover that actually cuts.
• The customer logs show no IncompleteRead on the pull. They do show a replica whose load failed and whose worker started anyway, and a live replica worker restart with a "successful" reload followed minutes later by an image inconsistency. That second event is consistent with a successful-but-stale load; it is not proven.
• The weaknesses in the previous section come from reading the code above, not from reproducing each one.

Suggested fix

• Serve one bundle per pull (EM, journal, VBBM, VSS plus a generation ID and sha256), read under a lock or from an immutable save generation, and have the client verify it before calling load_brm. Or at minimum check one generation across the four files.
• Retry with backoff on a short or failed transfer.
• Move the else: to the if cmapi_available branch, log the local-file fallback loudly, and load the file BRM_saves_current names.
• Whatever closes the socket, a cut transfer then becomes a retry rather than an empty node; this also needs the sibling fix to not start the worker after a failed load.

Related

• Sibling new tickets: DBRM starts on a failed load (consequence of a failed pull) and load_brm reports success on bad input (a different cause of a bad load).
• Existing MCOL-6258, MCOL-5687 and MCOL-4168 are about other mcs-loadbrm problems; none covers integrity of the pull from the primary.

Source

Found while investigating a customer issue on 23.02.24. Logs and analysis can be shared as needed.