PrimProc crashes with SIGSEGV in TupleHashJoinStep::abort() during query teardown (empty entry in joiners vector)

PrimProc crashes with signal 11 during query teardown. We have seen this on more than one production system, on 25.10.4 and also on 23.02.20, always with the same stack:

The crash typically comes right after a query has finished (directly after "CAL0042: End SQL statement" in debug.log). A pattern we see often in the logs is two sessions running the exact same query at the same time, and the crash happens when one of them is torn down. Many of the affected queries use a multi column IN list like `(col1,col2,col3) IN ((1,1,2026),(2,1,2026),...)`, which gets compiled into a hash join. But we also have cases where a normal single session teardown after query completion was enough, so the concurrency only makes it more likely, it is not required.

Signal: 11

joblist::TupleHashJoinStep::abort()+0x47

joblist::JobList::abort()

joblist::TupleJobList::abort()

joblist::TupleJobList::~TupleJobList()

the problem is maybe here, tuplehashjoin.cpp around line 2049:

cpp

void TupleHashJoinStep::abort()

{

  JobStep::abort();

  boost::mutex::scoped_lock sl(djsLock);

  // To prevent potential endless loop in bucketsToTables()

  for (auto& joiner : joiners)

    joiner->abort();

There is no null check on joiner. The vector is prepared with `joiners.resize(smallDLs.size());` (line 190), so it holds empty shared_ptrs until the setup code fills the entries per index (lines 230/235). And `~TupleJobList()` calls `abort()` unconditionally (joblist.cpp:667). So when a job list is destroyed while its hash join is still being set up (or is already torn down), abort() walks over an empty shared_ptr and PrimProc dies.

Some history: this loop only exists in the stable branches. It came in with the counting allocator work from MCOL-5797 (PR #3468, commit f594d2768) and was never on develop. That also fits the versions where we see the crash (MCOL-5797 went into 23.02.13 and 23.10.4).

We checked MCOL-6427 , they do not cover this. MCOL-6427 (PR #4048) fixes the same defect class in the drain loops of tupleaggregatestep.cpp, but in tuplehashjoin.cpp it only added an infer-ignore comment at smallRunnerFcn() saying the entries are always populated there. That may be true for smallRunnerFcn(), but the stack traces show that empty entries ARE reached through the abort() path.

Two notes for the fix:

1. A plain `if (joiner)` is probably not enough. The setup code writes the joiners entries while abort() reads them under djsLock. Concurrent write and read of a shared_ptr is a data race on its own, so the entries should be assigned under the same lock. Otherwise the reliable null deref just turns into a rarer race.

2. Some traces show a nested JobList::abort -> TupleJobList::abort during destruction, so abort() can re-enter while the destructor runs. The unconditional abort() in ~TupleJobList() should be guarded against that as well.

Stack traces and log extracts are available in the linked support case.

1. 1. How to repeat

No clean reproduction yet, it is timing dependent. Run the same query (multi column tuple IN filter on a large fact table, so it goes through TupleHashJoinStep) from two sessions at the same moment and close one session while both are in the join phase. Under a typical BI workload (many identical concurrent report queries) this fires several times a day.

1. 1. Workaround

None on the server side. Reducing duplicate query fire from the BI layer lowers the frequency but does not prevent it.