Distributed cpimport (mode 1/2) loses rows but reports success when SystemTempFileDir is shared between PMs

Summary

In distributed cpimport (mode 1 and 2) the coordinator builds one BRM report file name per job and sends it to every PM. Each PM's cpimport writes its report to that name and its WriteEngine Server (WES) re-reads it by name. If SystemTempFileDir is on storage shared by the PMs, the PMs overwrite each other's reports, and the coordinator never checks that each PM's report covers that PM's work. The load is logged as successful while one PM's extents are never published, so its rows are on disk but invisible, and the next load into the table fails.

Impact

Reproduced on our 3-node test cluster (23.02.24) with SystemTempFileDir on a shared filesystem and a distributed cpimport db tbl run from one node (mode 1, 1M-row chunks per PM):

• With the shared temp dir, 3 of 3 loads lost one PM's rows. cpimport returned 0 every time. The summary said "2000000 rows inserted" for a 3M-row file while 1M rows were visible, and "3000000 rows inserted" while 2M were visible.
• With a node-local temp dir, 3 of 3 loads were exact (1M rows per DBRoot).
• The lost DBRoot's extent stays at HWM 0 with no CP data. It can show as unavailable or as available, so grepping for unavailable extents alone does not find it.
• The next cpimport into that table aborts with "There is no extent information for table" and can leave the table lock behind.
  Not seen at a customer site: the cluster where we found this loads in mode 3 (no -R), so this is a mode 1/2 defect only.

Where it happens (tag 23.02.24, 4e11b0ed9)

1. writeengine/splitter/we_cmdargs.cpp:1316-1323 getBrmRptFileName(): the name is <SystemTempFileDir>/BrmRpt<mon><day><hour><min><sec><coordinator pid>.rpt, built once on the coordinator. Nothing in it identifies the PM or DBRoot. It goes to every PM as -R <name> (mode 1/2 only, we_cmdargs.cpp:211-215) and as a WE_CLT_SRV_BRMRPT message to WES.
2. writeengine/bulk/we_brmreporter.cpp:378-380 openRptFile(): fRptFile.open(name) on an ofstream, so the file is truncated on open. In this mode cpimport does not touch BRM itself; the report is the only way HWM and CP reach the coordinator.
3. writeengine/server/we_dataloader.cpp:558-569 onCpimportSuccess(): after its cpimport exits, WES re-opens the same path by name, parses it and sends the content stamped with its own PM id. The file is not removed on success. On a shared directory all PMs use one file, so a later PM's truncate-and-write can replace the content before an earlier PM's WES has read it. Likely result: the coordinator gets a copy of one PM's report in place of another's (interleaving inferred from the outcome, not traced).
4. writeengine/splitter/we_sdhandler.cpp:2145-2150 check4AllBrmReports() only checks that each PM sent a message. we_sdhandler.cpp:1377-1380 then calls updateCPAndHWMInBRM() and logs "BRM updated successfully". Nothing checks that a report belongs to the sending PM, that every (oid, DBRoot) touched has an HWM line, or that a PM's ROWS: count matches the rows the splitter sent it. The row total is the sum of whatever ROWS: lines arrived.
5. versioning/BRM/extentmap.cpp:5121-5123: for a bulk load the only move to AVAILABLE is setLocalHWM(). With no HWM line for a PM's extents they stay at HWM 0, and extentmap.cpp:4866-4870 getDbRootHWMInfo() throws on the next load ("@bug 5349").

With a node-local temp dir each PM has its own file, which is why the default (/tmp/columnstore_tmp_files) does not show this.

How to reproduce

1. 3-node cluster, set SystemTempFileDir to a directory on a filesystem mounted on all three nodes, restart the cluster so all nodes agree on the config.
2. Create a table and run a plain distributed load from one node: cpimport db tbl file (mode 1, no -m/-P) with a file of a few million rows.
3. cpimport returns 0 and prints a row total. SELECT COUNT returns fewer rows (one PM's share is missing). Compare extent HWMs per DBRoot with editem/the extent map: the missing PM's extent is at HWM 0.
4. Repeat with a node-local SystemTempFileDir: counts match.

Suggested fix

• Make the report name unique per PM. WES already tokenizes the cpimport command line and edits it before exec (WEDataLoader::setupCpimport()), so it can rewrite the -R argument and its own fBrmRptFileName with a PM suffix; alternatively the coordinator can build one name per PM.
• Have the coordinator verify each report before applying it: ROWS: equals the rows sent to that PM, and every DBRoot written by the job has HWM lines. Fail the job and roll back otherwise, instead of logging success.
• Independently, see the linked improvement for keeping SystemTempFileDir node-local.

Related

• No existing MCOL found for this report path. The name is built the same way on stable-23.10 and develop.
• Same root configuration as the shared-SystemTempFileDir improvement (spill-file wipe by another node's ExeMgr start), but a different code path and failure.

Source

Found while investigating a customer issue on 23.02.24. Reproduction steps and logs from our test cluster can be shared as needed.