Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Fixed
-
1.0.11
-
None
-
2019-01, 2019-02, 2019-03
Description
Build tested: 1.0.11-1, 1.0.9-1
Using a VM that's configured with 60 GB memory. I did the following test:
1) create a 1gb dbt3 database
The lineitem table is about 750mb data and 6001215 rows. There are 1500000 unique order keys in the table.
2) execute query
select l_orderkey, group_concat(l_partkey) from lineitem group by l_orderkey;
1500000 rows in set, 1 warning (4 min 13.48 sec)
3) Check cal trace
MariaDB [tpch1]> select calgettrace();
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
calgettrace() |
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Desc Mode Table TableOID ReferencedColumns PIO LIO PBE Elapsed Rows
BPS PM orders 3044 (o_custkey,o_orderkey) 1474 1486 0 0.102 1500000
BPS PM lineitem 3092 (l_orderkey,l_partkey) 0 5866 0 215.113 6001215
HJS PM lineitem-orders 3092 - - - - ----- -
TAS UM - - - - - - 244.862 1500000
4) getcalstat
-----------------------------------------------------------+
Query Stats: MaxMemPct-68; NumTempFiles-0; TempFileSpace-0B; ApproxPhyI/O-5870; CacheI/O-5886; BlocksTouched-5866; PartitionBlocksEliminated-0; MsgBytesIn-57MB; MsgBytesOut-2KB; Mode-Distributed |
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
1 row in set (0.01 sec)
ExeMgr's memory utilization was peaked at 40gb. Although group_concat() is known to use more memory, but using 40gb memory to process two columns totaling 44mg of data seems to be excessive.
I also did the same test on 1.0.9-1 and it showed the same behavior.