[MCOL-4031] Sorting code doesn't honor existing memory limits for ExeMgr. Created: 2020-06-02  Updated: 2021-02-04  Resolved: 2021-02-04

Status: Closed
Project: MariaDB ColumnStore
Component/s: None
Affects Version/s: 1.2.5, 1.4.4, 1.5.3
Fix Version/s: 6.1.1

Type: Bug Priority: Major
Reporter: Roman Assignee: Daniel Lee (Inactive)
Resolution: Fixed Votes: 1
Labels: None

Sprint: 2021-2

 Description   

ExeMgr can go out of allowed memory limits doing sorting.

Try to sort unreasonable amount of records that and instead of an error the system goes swapping then crashes or got killed by OOM.

The intermediate workaround is to apply dedicated cgroup rules to ExeMgr.



 Comments   
Comment by Daniel Lee (Inactive) [ 2021-02-04 ]

Build verified: 6.1.1 (drone #1620)

Reproduced the issue in 5.5.1 and verified that it has been fixed in 6.1.1

VM has 4 gb of memory
loaded 10g lineitem dataset twice
executed "select l_comment from lineitem order by l_comment;"
monitored memory usage

5.4.1

top - 15:41:05 up 42 min, 2 users, load average: 7.00, 2.66, 1.93
Tasks: 119 total, 1 running, 118 sleeping, 0 stopped, 0 zombie
%Cpu(s): 4.5 us, 6.5 sy, 0.0 ni, 14.1 id, 72.6 wa, 1.9 hi, 0.4 si, 0.0 st
MiB Mem : 3938.4 total, 102.1 free, 3790.7 used, 45.6 buff/cache
MiB Swap: 2048.0 total, 400.4 free, 1647.6 used. 8.1 avail Mem

MariaDB [tpch50]> select l_comment from lineitem order by l_comment;
ERROR 1815 (HY000): Internal error: InetStreamSocket::readToMagic: Remote is closed

6.1.1

top - 15:43:03 up 34 min, 2 users, load average: 1.27, 1.89, 1.70
Tasks: 122 total, 1 running, 121 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.1 us, 0.1 sy, 0.0 ni, 99.8 id, 0.0 wa, 0.1 hi, 0.0 si, 0.0 st
MiB Mem : 3938.4 total, 763.7 free, 1706.2 used, 1468.4 buff/cache
MiB Swap: 2048.0 total, 1884.9 free, 163.1 used. 1977.2 avail Mem

MariaDB [tpch50]> select l_comment from lineitem order by l_comment;
ERROR 1815 (HY000): Internal error: TupleAnnexStep::executeParallelOrderBy() IDB-2018: Not enough memory to process the LIMIT. Consider raising TotalUmMemory or reducing memory usage.

Generated at Thu Feb 08 02:47:16 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.