Build tested: 1.0.7-1
This is was identified during the 1TB DBT3. More specifically, the following query was being tested:
select c_name, c_custkey, o_orderkey, o_orderdate, o_totalprice, sum(l_quantity) from customer, orders, lineitem where o_orderkey in ( select l_orderkey from lineitem group by l_orderkey having sum(l_quantity) > 313 ) and c_custkey = o_custkey and o_orderkey = l_orderkey group by c_name, c_custkey, o_orderkey, o_orderdate, o_totalprice order by o_totalprice desc, o_orderdate LIMIT 100;
Memory configuration:
244 gb memory in each server. TotalUMMemory was set to 50% and numBlock (PrimProc cache) was set to 40%.
After starting ColumnStore, I executed the above query and that was the only query being process in the entire system. After the query was successfully completed, Only one of the 36 cores to churn at 100% (the rest were at 0%) for 4 minutes to free memory, from the peak of 54% it to 30% and remained at that level.
Two questions:
1) Why would it take 4 minutes to free 24%, or 58.5gb of memory?
This is a big issue, as in a real environment (or my DBT3 test of 22 queries), the next query will be starting running immediately.
2) Why it remains at 30% utilization?
I reported the issue at InfiniDB and was told that memory used by ExeMgr will not be completed released. They will be released when processing the next query and internal objects are being reinitialized. So I ran a select count
query and ExeMgr utilization dropped to 20%.
When 20% of memory utilization remained, I executed the same query and ExeMgr memory utilization went up to 70% and disk swapping kicked in. After swap space exhausted, ColumnStore restarted itself.