Server crash when aggregate and window function co-exist with expression in PARTITION BY

Description

Queries combining a regular aggregate function (AVG, SUM, COUNT, MIN, MAX, COUNT DISTINCT, GROUP_CONCAT) with a window function (STD OVER, AVG OVER, VARIANCE OVER, DENSE_RANK, FIRST_VALUE, LAST_VALUE, LEAD, LAG, NTILE, NTH_VALUE) crash the server with Lost connection (ERROR 2013).

Related to MCOL-5430 (wrong results with WF + aggregate) — same root cause area (combineJobStepsByTable / TAS hash key miscalculation), but under additional conditions it escalates from wrong results to a server crash.

Crash conditions (all required)

1. Aggregate + window function in the same SELECT
2. Expression (not a bare column) in PARTITION BY — e.g. abs(col), col+2
3. Two or more columns in ORDER BY inside the window specification
4. ~1500+ rows in the table

Without condition 2 or 3 → wrong results (MCOL-5430), not crash.

Minimal reproducer

DROP DATABASE IF EXISTS mcol5430_repro;

CREATE DATABASE mcol5430_repro;

USE mcol5430_repro;

CREATE TABLE t (a INT, b INT) ENGINE=Columnstore;

INSERT INTO t SELECT seq, seq % 150 FROM seq_1_to_1500;

-- CRASHES the server:

SELECT AVG(b),

       STD(b) OVER (PARTITION BY abs(b)

                     ORDER BY b, a

                     ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING)

FROM t ORDER BY 1, 2;

More crash variants (all confirmed independently)

-- SUM + DENSE_RANK

SELECT SUM(b), DENSE_RANK() OVER (PARTITION BY abs(b) ORDER BY b, a)

FROM t ORDER BY 1, 2;

-- GROUP_CONCAT inside FIRST_VALUE

SELECT a, FIRST_VALUE(GROUP_CONCAT(b)) OVER (PARTITION BY abs(b)+2 ORDER BY b, a ROWS UNBOUNDED PRECEDING)

FROM t ORDER BY 1, 2;

-- COUNT DISTINCT + VARIANCE

SELECT COUNT(DISTINCT b), VARIANCE(b) OVER (PARTITION BY abs(b) ORDER BY b, a ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)

FROM t ORDER BY 1, 2;

Environment

• MariaDB 11.4.11-8, Columnstore 25.10.4 (stable-23.10)
• Build: RelWithDebInfo
• OS: Ubuntu 24.04