[MDEV-9531] GROUP_CONCAT with ORDER BY inside takes a lot of memory while it's executed Created: 2016-02-08  Updated: 2023-06-17  Resolved: 2019-04-25

Status: Closed
Project: MariaDB Server
Component/s: OTHER
Affects Version/s: 5.5, 10.0, 10.1
Fix Version/s: 10.2.24, 10.1.39, 10.3.15, 10.4.5

Type: Bug Priority: Major
Reporter: Elena Stepanova Assignee: Sergei Golubchik
Resolution: Fixed Votes: 0
Labels: None

Issue Links:
Duplicate
is duplicated by MDEV-30282 Uncontrollable memory consumption upo... Closed
Sprint: 10.1.14, 10.1.18, 10.1.19

 Description   

A query like 

SELECT GROUP_CONCAT( seq, seq, seq, seq, seq, seq, seq, seq ORDER BY 2,1,3,4,6,5,8,7 ) AS cfield1 FROM seq_1_to_50000000

easily uses up to 4-5 Gb memory while it's running.

<montywi> everything has limits
<montywi> even for filesort; For bigger things it should always hit disk, never memory
...
<montywi> the problem is that when doing a sort, we create a sorted tree in memory
<montywi> however there is no memory check for this
<montywi> so this can be any size
<montywi> Normally it would make sense to limit this to group_concat_max_length, but I fear that would break things, as we sort records there, which can have a notable overhead
<montywi> So I have limit it to max_heap_table_size

 Comments   
Comment by Sergei Golubchik [ 2016-02-08 ]

There are two possible approaches:

  1. introduce some some limit for a max tree size. when the tree grows that large, a special "cleanup" procedure is executed, that keeps first group_concat_max_len bytes of values (or first group_concat_max_len values, if that's much simpler) and removes all larger values. then continue inserting into the tree
  2. use priority queue, not a tree. never grows large and doesn't need cleanups. but higher up-front memory cost if group_concat_max_len limit is rarely reached
Comment by Oleksandr Byelkin [ 2016-05-04 ]

revision-id: eb9fab76fb4322a7821e6502e6303ed381624429 (mariadb-10.1.13-18-geb9fab7)
parent(s): 732adec0a4c75d99389230feeb0deca0ad668de7
committer: Oleksandr Byelkin
timestamp: 2016-05-04 19:15:31 +0200
message:

MDEV-9531: GROUP_CONCAT with ORDER BY inside takes a lot of memory while it's executed

Limitation added to Red-Black tree.

—

Comment by Oleksandr Byelkin [ 2016-06-28 ]

I remove variable and made automatic minimum print output calculation:

revision-id: 13b5098fcaa888173472d255e29aff22bcc5baae (mariadb-10.1.13-18-g13b5098)
parent(s): 732adec0a4c75d99389230feeb0deca0ad668de7
committer: Oleksandr Byelkin
timestamp: 2016-06-28 10:59:59 +0200
message:

MDEV-9531: GROUP_CONCAT with ORDER BY inside takes a lot of memory while it's executed

Limitation added to Red-Black tree.

—

Comment by Oleksandr Byelkin [ 2016-09-26 ]

revision-id: f5a5ac49b5ff7b04aa816d33ca44d9fe88a94498 (mariadb-10.1.17-23-gf5a5ac4)
parent(s): d30809a3cde823ad696304a941afe5a562bfa3ed
committer: Oleksandr Byelkin
timestamp: 2016-09-26 16:21:18 +0200
message:

MDEV-9531: GROUP_CONCAT with ORDER BY inside takes a lot of memory while it's executed

Limitation added to Red-Black tree.

—

Comment by Sergei Golubchik [ 2019-04-20 ]

please, review commit 73b12b0cebc in bb-10.1-serg

Comment by Oleksandr Byelkin [ 2019-04-24 ]

OK to push.

Generated at Thu Feb 08 07:35:22 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.