[MCOL-4562] Disk Based Join is not working Created: 2021-02-25  Updated: 2021-03-01  Resolved: 2021-02-25

Status: Closed
Project: MariaDB ColumnStore
Component/s: N/A
Affects Version/s: None
Fix Version/s: N/A

Type: Task Priority: Major
Reporter: Developer Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: need_feedback
Environment:

We have multinode system which contains 1 UM and 3 PM


 Description   

We have tried to enable DiskBasedJoin but it is not working. It is still giving error as below for one of the our query.
ERROR 1815 (HY000): Internal error: IDB-2003: Aggregation/Distinct memory limit is exceeded.

Can you please check report by downloading from below link and let us know what we are missing to configure.

https://drive.google.com/file/d/14tnaKcgejcGRxvgrhaLCcv-G3hJx8EV5/view?usp=sharing

 Comments   
Comment by Todd Stoffel (Inactive) [ 2021-02-25 ]

Disk based aggregation is different than disk based joins. Please see MCOL-563

Comment by Developer [ 2021-02-25 ]

In that case what can be solution for my issue? Can you please guide me?

Comment by Todd Stoffel (Inactive) [ 2021-02-25 ]

shashank9898 You will need to add more RAM to your system and/or adjust your NumBlockPct and TotalUmMemory settings.

https://mariadb.com/kb/en/mariadb-columnstore-performance-related-configuration-settings/#memory-management-numblockspct-and-totalummemory

Comment by Developer [ 2021-03-01 ]

Can you please tell me what should be ideal value for NumBlockPct and TotalUmMemory settings as per my hardware details? I am using amazon insatances for UM and PM. We have 1 UM and 3 PM all are m4.2xlarge. Current value of NumBlockPct is 70 and TotalUmMemory is 50% So please check and help us to set appropriate values.

Instance : m4.2xlarge
vCPU* : 8
Mem (GiB) : 32
Storage : EBS-only
Dedicated EBS Bandwidth (Mbps) : 1,000
Network Performance : High

Thanks.

Comment by Todd Stoffel (Inactive) [ 2021-03-01 ]

shashank9898 with those settings, you have allocated 120% of your RAM to MariaDB and left nothing for the system. The combination of those numbers should probably never exceed 75% of your available RAM. Further adjustments would need to be made if you were also using a sizable innodb buffer pool. An m4.2xlarge instance has 32 GB of RAM. From your comments it sounds like your workload requires more than that.

Generated at Thu Feb 08 02:51:17 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.