Keep SystemTempFileDir node-local: a shared directory lets one node's ExeMgr start delete other nodes' spill files

Summary

ExeMgr deletes and recreates its temp subdirectories (joins, aggregates) every time it starts. SystemTempFileDir has no per-node component, so when it points at storage shared by several nodes, one node's ExeMgr start deletes the live disk-spill files of every other node's running queries. Nothing warns that a shared value is a problem. This is requested as an improvement: keep the temp dir node-local by construction, and warn or refuse when it is not.

Impact

• On a customer cluster (Sep-Oct 2026) with SystemTempFileDir on a shared export and disk-based aggregation enabled, running aggregations failed with MCS-2056 ("disk-based aggregation: No such file or directory") a few seconds after another node restarted.
• Reproduced on our 3-node test cluster (23.02.24) with a shared SystemTempFileDir: when one node's ExeMgr started it logged "Exception whilst cleaning tmpdir: remove_all: Stale file handle" for a spill file named after another node's PrimProc pid, and a running aggregation on that other node then failed with MCS-2056.
• A shared temp dir also triggers silent row loss in distributed cpimport (see the linked cpimport report-file ticket).

Where it happens (tag 23.02.24, 4e11b0ed9)

• primitives/primproc/serviceexemgr.cpp:160-168: at startup, for each spill section, ExeMgr takes config->getTempFileDir(purpose) and, when the disk-based option is allowed, runs boost::filesystem::remove_all(tmpPrefix) followed by create_directories(tmpPrefix). The code comment already calls this "quite scary".
• utils/configcpp/configcpp.cpp:616-630 getTempFileDir() appends only joins/ or aggregates/ to SystemTempFileDir; there is no host or PM component.
• oam/etc/Columnstore.xml:69-72: the shipped default is the local /tmp/columnstore_tmp_files. The only comment says ExeMgr deletes the joins and aggregates subdirectories on start. It does not say the directory must not be shared.
• Other users of the directory build one fixed name per job (the cpimport BRM report, Job_<id>.xml), so they also assume it is per node.

Evidence

Code reading at the tag, the test-cluster reproduction above, and MCS-2056 log bursts at a customer site that followed other nodes' restarts.

Suggested fix

• Add a per-node component below SystemTempFileDir (hostname or PM id) in getTempFileDir(), so each ExeMgr only ever cleans its own subtree. Alternatively remove only files owned by this node's PIDs.
• Warn in the log, or refuse to enable disk-based join/aggregation, when SystemTempFileDir is on a network filesystem.
• Update the Columnstore.xml comment and the docs to say the directory must be node-local.

Related

• MCOL-2101 (older tmpfiles.d handling for the temp directory): about cleanup of the same location outside the process, not about several nodes sharing it.

Source

Found while investigating a customer issue on 23.02.24. Reproduction steps and logs from our test cluster can be shared as needed.