Details
-
Task
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Fixed
-
None
-
10.2.4-4, 10.1.20, 10.2.4-5, 10.2.4-1, 10.2.4-2, 10.3.3-1
Description
GROUP_CONCAT supports ORDER BY and DISTINCT but not LIMIT clause.
Supporting LIMIT N[,M] could simplify queries.
something like
SUBSTRING_INDEX(GROUP_CONCAT(CONCAT_WS(":",date,cnt) ORDER BY cnt DESC),",",1)
could be written as
GROUP_CONCAT(CONCAT_WS(":",date,cnt) ORDER BY cnt DESC LIMIT 1)
this can be used in timeseries-like calculations. Suppose we have a table (date,cnt) and we want to produce a list of dates, for each year, where cnt was maximum for that year, showing such date,cnt for each year
currently this is written in an awkward manner and is also quite inefficient as we SELECT much more data that is necessary for the task:
SELECT LEFT(col1,4) AS _yyyy
, SUBSTRING_INDEX(GROUP_CONCAT(CONCAT_WS(":",DATE_FORMAT(col1,"%a %Y-%m-%d"),LPAD(col2,8," ")) ORDER BY col2 DESC),",",1) AS col3
FROM t1
GROUP BY 1
;
Attachments
Issue Links
- duplicates
-
-
- Closed
-
- relates to
-
-
- Closed
-
-
MDEV-11563 GROUP_CONCAT(DISTINCT ...) may produce a non-distinct list
-
- Closed
-
Activity
| Field | Original Value | New Value |
|---|---|---|
| Description |
GROUP_CONCAT supports ORDER BY and DISTINCT but not LIMIT clause.
Supporting LIMIT N[,M] could simplify queries, avoid passing lots of data to the client and may be optimized on the server side by not extracting complete GROUP BY ... ORDER BY result. Example: something like SUBSTRING_INDEX(GROUP_CONCAT(CONCAT_WS(":",date,cnt) ORDER BY cnt DESC),",",1) could be written as GROUP_CONCAT(CONCAT_WS(":",date,cnt) ORDER BY cnt DESC LIMIT 1) this can be used in timeseries-like calculations. Suppose we have a table (date,cnt) and we want to produce a list of dates, for each year, where cnt was maximum for that year, showing such date,cnt for each year currently this is written in an awkward manner and is also quite inefficient as we SELECT much more data that is necessary for the task: SELECT LEFT(col1,4) AS _yyyy , SUBSTRING_INDEX(GROUP_CONCAT(CONCAT_WS(":",DATE_FORMAT(col1,"%a %Y-%m-%d"),LPAD(col2,8," ")) ORDER BY col2 DESC),",",1) AS col3 FROM t1 GROUP BY 1 ; |
GROUP_CONCAT supports ORDER BY and DISTINCT but not LIMIT clause.
Supporting LIMIT N[,M] could simplify queries. something like SUBSTRING_INDEX(GROUP_CONCAT(CONCAT_WS(":",date,cnt) ORDER BY cnt DESC),",",1) could be written as GROUP_CONCAT(CONCAT_WS(":",date,cnt) ORDER BY cnt DESC LIMIT 1) this can be used in timeseries-like calculations. Suppose we have a table (date,cnt) and we want to produce a list of dates, for each year, where cnt was maximum for that year, showing such date,cnt for each year currently this is written in an awkward manner and is also quite inefficient as we SELECT much more data that is necessary for the task: SELECT LEFT(col1,4) AS _yyyy , SUBSTRING_INDEX(GROUP_CONCAT(CONCAT_WS(":",DATE_FORMAT(col1,"%a %Y-%m-%d"),LPAD(col2,8," ")) ORDER BY col2 DESC),",",1) AS col3 FROM t1 GROUP BY 1 ; |
| Sprint | 10.2.4-3 [ 115 ] |
| Rank | Ranked higher |
| Status | Open [ 1 ] | In Progress [ 3 ] |
| Summary | add support for LIMIT clause in GROUP_CONCAT() | Add support for LIMIT clause in GROUP_CONCAT() |
| Sprint | 10.2.4-3 [ 115 ] | 10.2.4-4 [ 117 ] |
| Rank | Ranked higher |
| Assignee | Varun [ varun ] | Sergei Petrunia [ psergey ] |
| Status | In Progress [ 3 ] | In Review [ 10002 ] |
| Sprint | 10.2.4-4 [ 117 ] | 10.2.4-4, 10.1.20 [ 117, 119 ] |
| Rank | Ranked higher |
Review feedback was provided over email. Re-assigning to Varun as the ball is on his side now.
| Assignee | Sergei Petrunia [ psergey ] | Varun [ varun ] |
| Status | In Review [ 10002 ] | Stalled [ 10000 ] |
Made changes according to the review, sent again for review to Sergei
| Assignee | Varun [ varun ] | Sergei Petrunia [ psergey ] |
| Priority | Minor [ 4 ] | Major [ 3 ] |
| Component/s | Optimizer [ 10200 ] |
| Assignee | Sergei Petrunia [ psergey ] | Varun [ varun ] |
| Status | Stalled [ 10000 ] | In Progress [ 3 ] |
| Link |
This issue is blocked by |
| Status | In Progress [ 3 ] | Stalled [ 10000 ] |
| Sprint | 10.2.4-4, 10.1.20 [ 117, 119 ] | 10.2.4-4, 10.1.20, 5.5.54 [ 117, 119, 123 ] |
| Rank | Ranked higher |
See also: MDEV-11563. It turns out, GROUP_CONCAT(DISTINCT x ORDER BY y) doesn't work. We've discovered that when discussing how DISTINCT ... ORDER BY ...LIMIT should be done.
| Link |
This issue relates to |
| Sprint | 10.2.4-4, 10.1.20, 5.5.54 [ 117, 119, 123 ] | 10.2.4-4, 10.1.20, 10.2.4-5 [ 117, 119, 125 ] |
| Rank | Ranked lower |
| Status | Stalled [ 10000 ] | In Progress [ 3 ] |
| Assignee | Varun [ varun ] | Sergei Petrunia [ psergey ] |
| Status | In Progress [ 3 ] | In Review [ 10002 ] |
| Sprint | 10.2.4-4, 10.1.20, 10.2.4-5 [ 117, 119, 125 ] | 10.2.4-4, 10.1.20, 10.2.4-5, 10.2.4-1 [ 117, 119, 125, 132 ] |
| Rank | Ranked lower |
| Sprint | 10.2.4-4, 10.1.20, 10.2.4-5, 10.2.4-1 [ 117, 119, 125, 132 ] | 10.2.4-4, 10.1.20, 10.2.4-5, 10.2.4-1, 10.2.4-2 [ 117, 119, 125, 132, 134 ] |
| Fix Version/s | 10.3 [ 22126 ] | |
| Fix Version/s | 10.3.0 [ 22127 ] |
| Sprint | 10.2.4-4, 10.1.20, 10.2.4-5, 10.2.4-1, 10.2.4-2 [ 117, 119, 125, 132, 134 ] | 10.2.4-4, 10.1.20, 10.2.4-5, 10.2.4-1, 10.2.4-2, 10.3.3-1 [ 117, 119, 125, 132, 134, 200 ] |
The branch was based on 10.2, so a rebase is done on top of 10.3.
The code is currently pushed to the branch 10.3-varun
| Assignee | Sergei Petrunia [ psergey ] | Oleksandr Byelkin [ sanja ] |
| Assignee | Oleksandr Byelkin [ sanja ] | Varun Gupta [ varun ] |
| Status | In Review [ 10002 ] | Stalled [ 10000 ] |
| Status | Stalled [ 10000 ] | In Progress [ 3 ] |
Add yet another test like:
set @x=-1;
|
execute STMT using @x;
|
and if it return error it is ok to push (if no let us talk)
| Fix Version/s | 10.3.3 [ 22644 ] | |
| Fix Version/s | 10.3 [ 22126 ] | |
| Resolution | Fixed [ 1 ] | |
| Status | In Progress [ 3 ] | Closed [ 6 ] |
LIMIT clause in GROUP_CONCAT() does not work in a view.
This can be tested using the example on https://mariadb.com/kb/en/library/group_concat/. If you create a view from the statement 'SELECT SUBSTRING_INDEX(GROUP_CONCAT(CONCAT_WS(":",dd,cc) ORDER BY cc DESC),",",1) FROM d;' than LIMIT does not do anything. Run the statement on its own and LIMIT works. I tested this in 10.3 and 10.4 docker container versions.
Is that how it should be? I would like to be able to use LIMIT in a view.
antoonp, thanks for reporting the bug, I have opened issue MDEV-18943 for this bug.
| Link |
This issue is blocked by |
| Workflow | MariaDB v3 [ 78321 ] | MariaDB v4 [ 133002 ] |
| Zendesk Related Tickets | 133093 |
> Supporting LIMIT N[,M] could simplify queries,
This is the goal of this MDEV.
> avoid passing lots of data to the client
This is already achieved by using SUBSTRING_INDEX
> and may be optimized on the server side by not extracting complete GROUP BY ... ORDER BY result.
This is outside of scope of this MDEV.
Let's limit this task to providing LIMIT syntax, and query results.
Loose-scan like optimization for reading only a few rows per group is theoretically possible, but hard to do in the current MariaDB's (or MySQL's) GROUP/ORDER BY optimizer.