[MCOL-5250] Disk-based DISTINCT - Jira

Details

Type: New Feature
Status: Stalled (View Workflow)
Priority: Major
Resolution: Unresolved
Affects Version/s: 22.08.1
Fix Version/s: 23.10
Component/s: ExeMgr, PrimProc
Labels:
- rm_big_data

Epic Link:
Tech debt
Sprint:
2021-17, 2022-22, 2022-23, 2023-4, 2023-5, 2023-6, 2023-7, 2023-8, 2023-10, 2023-11

Description

As of 22.08.01 MCS does DISTINCT processing TupleAnnexStep. This step leverages hashmap for the purpose. This solution is simple but it:

lacks scalability
can't leverage disk-based capabilities of RowStorage class used by GROUP BY
ResourceManager that accounts RAM consumption doesn't counts the hashmap

This issue is about a new DISTINCT implementation(presumably based on RowStorage) that:

can do external DISTINCT spilling on disk if necessary,
ResourceManager counts the implemenation RAM consumption
scales(this might be tricky since DISTINCT processing overlaps with ORDER BY)

Attachments

Issue Links

causes

MCOL-5804 Disk-based ORDER BY

Open

includes

MCOL-5187 OOM happening when querying large datasets and using distinct

Closed

MCOL-5541 Disk-based distinct :Create a separate jobstep for handling ConstantColumn

Stalled

Activity

Ascending order - Click to sort in descending order

Roman created issue - 2022-10-05 10:00

Roman made changes - 2022-10-05 10:04

Field	Original Value	New Value
Link		This issue includes ~~MCOL-5187~~ [ ~~MCOL-5187~~ ]

alexey vorovich (Inactive) made changes - 2022-10-05 19:32

Fix Version/s		22.11.01 [ 28458 ]
Fix Version/s	22.08.3 [ 28456 ]

Alexey Antipovsky made changes - 2022-10-12 09:21

Status

Open [ 1 ]

In Progress [ 3 ]

Todd Stoffel (Inactive) made changes - 2022-10-25 22:45

Rank

Ranked higher

Todd Stoffel (Inactive) made changes - 2022-10-25 22:46

Rank

Ranked higher

Todd Stoffel (Inactive) made changes - 2022-10-25 22:46

Rank

Ranked higher

Todd Stoffel (Inactive) made changes - 2022-10-26 04:45

Sprint

2021-17 [ 614 ]

2021-17, 2021-18 [ 614, 672 ]

alexey vorovich (Inactive) made changes - 2022-11-28 21:20

Fix Version/s		23.02 [ 28209 ]
Fix Version/s	23.03.1 [ 28458 ]

Todd Stoffel (Inactive) made changes - 2022-12-27 07:02

Sprint

2021-17, 2022-22 [ 614, 672 ]

2021-17, 2022-22, 2022-23 [ 614, 672, 686 ]

Todd Stoffel (Inactive) made changes - 2023-02-01 08:12

Assignee

Alexey Antipovsky [ JIRAUSER47594 ]

Roman [ drrtuy ]

Todd Stoffel (Inactive) made changes - 2023-02-17 20:58

Sprint

2021-17, 2022-22, 2022-23 [ 614, 672, 686 ]

2021-17, 2022-22, 2022-23, 2022-24 [ 614, 672, 686, 698 ]

Todd Stoffel (Inactive) made changes - 2023-03-16 18:23

Sprint

2021-17, 2022-22, 2022-23, 2023-4 [ 614, 672, 686, 698 ]

2021-17, 2022-22, 2022-23, 2023-4, 2023-5 [ 614, 672, 686, 698, 702 ]

Todd Stoffel (Inactive) made changes - 2023-04-06 05:20

Sprint

2021-17, 2022-22, 2022-23, 2023-4, 2023-5 [ 614, 672, 686, 698, 702 ]

2021-17, 2022-22, 2022-23, 2023-4, 2023-5, 2023-6 [ 614, 672, 686, 698, 702, 706 ]

Todd Stoffel (Inactive) made changes - 2023-07-01 20:04

Fix Version/s		23.08 [ 28540 ]
Fix Version/s	23.02 [ 28209 ]

Todd Stoffel (Inactive) made changes - 2023-07-01 20:18

Sprint

2021-17, 2022-22, 2022-23, 2023-4, 2023-5, 2023-6 [ 614, 672, 686, 698, 702, 706 ]

2021-17, 2022-22, 2022-23, 2023-4, 2023-5, 2023-6, 2023-7 [ 614, 672, 686, 698, 702, 706, 726 ]

Todd Stoffel (Inactive) made changes - 2023-07-17 07:06

Sprint

2021-17, 2022-22, 2022-23, 2023-4, 2023-5, 2023-6, 2023-7 [ 614, 672, 686, 698, 702, 706, 726 ]

2021-17, 2022-22, 2022-23, 2023-4, 2023-5, 2023-6, 2023-7, 2023-8 [ 614, 672, 686, 698, 702, 706, 726, 728 ]

Roman made changes - 2023-07-25 16:14

Status

In Progress [ 3 ]

Stalled [ 10000 ]

Andrey Piskunov (Inactive) made changes - 2023-07-26 14:55

Link

This issue includes MCOL-5541 [ MCOL-5541 ]

alexey vorovich (Inactive) made changes - 2023-07-26 15:38

Labels

rf_big_data

alexey vorovich (Inactive) made changes - 2023-07-26 15:58

Labels

rf_big_data

rm_big_data

Todd Stoffel (Inactive) made changes - 2023-08-25 22:55

Sprint

2021-17, 2022-22, 2022-23, 2023-4, 2023-5, 2023-6, 2023-7, 2023-8 [ 614, 672, 686, 698, 702, 706, 726, 728 ]

2021-17, 2022-22, 2022-23, 2023-4, 2023-5, 2023-6, 2023-7, 2023-8, 2023-9 [ 614, 672, 686, 698, 702, 706, 726, 728, 733 ]

Todd Stoffel (Inactive) made changes - 2023-08-25 22:55

Sprint

2021-17, 2022-22, 2022-23, 2023-4, 2023-5, 2023-6, 2023-7, 2023-8, 2023-9 [ 614, 672, 686, 698, 702, 706, 726, 728, 733 ]

2021-17, 2022-22, 2022-23, 2023-4, 2023-5, 2023-6, 2023-7, 2023-8, 2023-10 [ 614, 672, 686, 698, 702, 706, 726, 728, 734 ]

Todd Stoffel (Inactive) made changes - 2023-09-13 19:50

Sprint

2021-17, 2022-22, 2022-23, 2023-4, 2023-5, 2023-6, 2023-7, 2023-8, 2023-10 [ 614, 672, 686, 698, 702, 706, 726, 728, 734 ]

2021-17, 2022-22, 2022-23, 2023-4, 2023-5, 2023-6, 2023-7, 2023-8, 2023-10, 2023-11 [ 614, 672, 686, 698, 702, 706, 726, 728, 734, 737 ]

Leonid Fedorov made changes - 2023-12-21 15:15

Sprint

2021-17, 2022-22, 2022-23, 2023-4, 2023-5, 2023-6, 2023-7, 2023-8, 2023-10, 2023-11 [ 614, 672, 686, 698, 702, 706, 726, 728, 734, 737 ]

2021-17, 2022-22, 2022-23, 2023-4, 2023-5, 2023-6, 2023-7, 2023-8, 2023-10, 2023-11, 2023-13 [ 614, 672, 686, 698, 702, 706, 726, 728, 734, 737, 748 ]

Leonid Fedorov made changes - 2023-12-22 13:58

Sprint

2021-17, 2022-22, 2022-23, 2023-4, 2023-5, 2023-6, 2023-7, 2023-8, 2023-10, 2023-11, 2023-12 [ 614, 672, 686, 698, 702, 706, 726, 728, 734, 737, 748 ]

2021-17, 2022-22, 2022-23, 2023-4, 2023-5, 2023-6, 2023-7, 2023-8, 2023-10, 2023-11 [ 614, 672, 686, 698, 702, 706, 726, 728, 734, 737 ]

Julien Fritsch made changes - 2024-09-19 13:54

Labels

rm_big_data

rm_big_data triage

Roman made changes - 2024-09-19 13:57

Link

This issue causes MCOL-5804 [ MCOL-5804 ]

Julien Fritsch made changes - 2025-02-24 12:09

Labels

rm_big_data triage

rm_big_data

People

Assignee:: Roman

Reporter:: Roman

Votes:: 0 Vote for this issue

Watchers:: 4 Start watching this issue

Dates

Created:: 2022-10-05 10:00

Updated:: 2025-02-24 12:15

Git Integration

Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.

MariaDB ColumnStore

Details

Description

Attachments

Issue Links

Activity

People

Dates

Git Integration