Attached stack.pdf which shows a current build of develop tree. The allocations in MB when doing this query 20 times simultaneously:

select sum(l_quantity) from lineitem group by l_quantity;

Captured using:

sudo /usr/local/mariadb/columnstore/bin/mcsadmin  stopexemgrs

LD_PRELOAD=/lib64/libtcmalloc.so HEAPPROFILE=/tmp/heapprof /usr/local/mariadb/columnstore/bin/ExeMgr

... run queries and ctrl-c ...

pprof --pdf --lines --alloc_space /usr/local/mariadb/columnstore/bin/ExeMgr /tmp/heapprof.00??.heap > /home/linuxjedi/stack.pdf

Be sure to benchmark whatever sol'n is chosen. IIRC, in the pool allocator it's using right now, allocation just means moving a pointer. An off-the-shelf allocator will likely slow things down. How much is the question. Performance is important here since the caller is trying to allocate potentially millions of little things.

An alternative if the benchmarks aren't great.

I'd suggest we have it start with a small window size (maybe 1MB), and have it grow up to some max (64MB) as more is allocated.

fwiw, 64MB was chosen b/c it is a threshold value in the default linux allocator (this was before we switched to jemalloc). IIRC, above that value, it will bypass most of the allocator's logic and allocate a segment using sbrk(), which was a performance gain for its use cases. Since we're now using jemalloc in CS, we could experiment with reducing that #, or get fancy.

went ahead and fixed it, just needed to not explicitly tell it to allocate 64MB per chunk. By default it allocates 4096 (page size) * sizeof(element), which is much less. Will check for performance issues then check it in...

This, and the new PM hash table functionality, are in
https://github.com/mariadb-corporation/mariadb-columnstore-engine/pull/920

Build tested: 1.2.5-1, 1.4.1-1 for comparison

Build verified: 1.4.1-1
engine commit:
57724e5

stack: single node

Executed the query 20 times simultaneously on a 10g dbt3 database, on both 1.2.5-1 to 1.4.1-1. Heap memory utilization info are captured in the attached pdf files.