Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-9531

GROUP_CONCAT with ORDER BY inside takes a lot of memory while it's executed

Details

    • 10.1.14, 10.1.18, 10.1.19

    Description

      A query like 

      SELECT GROUP_CONCAT( seq, seq, seq, seq, seq, seq, seq, seq ORDER BY 2,1,3,4,6,5,8,7 ) AS cfield1 FROM seq_1_to_50000000

      easily uses up to 4-5 Gb memory while it's running.

      <montywi> everything has limits
      <montywi> even for filesort; For bigger things it should always hit disk, never memory
      ...
      <montywi> the problem is that when doing a sort, we create a sorted tree in memory
      <montywi> however there is no memory check for this
      <montywi> so this can be any size
      <montywi> Normally it would make sense to limit this to group_concat_max_length, but I fear that would break things, as we sort records there, which can have a notable overhead
      <montywi> So I have limit it to max_heap_table_size

      Attachments

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. MDEV-30282 Uncontrollable memory consumption upon GROUP_CONCAT with ORDER BY

          • Major - Major loss of function.
          • Closed

          Activity

            Ascending order - Click to sort in descending order
            serg Sergei Golubchik added a comment - - edited

            There are two possible approaches:

            1. introduce some some limit for a max tree size. when the tree grows that large, a special "cleanup" procedure is executed, that keeps first group_concat_max_len bytes of values (or first group_concat_max_len values, if that's much simpler) and removes all larger values. then continue inserting into the tree
            2. use priority queue, not a tree. never grows large and doesn't need cleanups. but higher up-front memory cost if group_concat_max_len limit is rarely reached
            serg Sergei Golubchik added a comment - - edited There are two possible approaches: introduce some some limit for a max tree size. when the tree grows that large, a special "cleanup" procedure is executed, that keeps first group_concat_max_len bytes of values (or first group_concat_max_len values, if that's much simpler) and removes all larger values. then continue inserting into the tree use priority queue, not a tree. never grows large and doesn't need cleanups. but higher up-front memory cost if group_concat_max_len limit is rarely reached
            sanja Oleksandr Byelkin added a comment -

            revision-id: eb9fab76fb4322a7821e6502e6303ed381624429 (mariadb-10.1.13-18-geb9fab7)
            parent(s): 732adec0a4c75d99389230feeb0deca0ad668de7
            committer: Oleksandr Byelkin
            timestamp: 2016-05-04 19:15:31 +0200
            message:

            MDEV-9531: GROUP_CONCAT with ORDER BY inside takes a lot of memory while it's executed

            Limitation added to Red-Black tree.

            —

            sanja Oleksandr Byelkin added a comment - revision-id: eb9fab76fb4322a7821e6502e6303ed381624429 (mariadb-10.1.13-18-geb9fab7) parent(s): 732adec0a4c75d99389230feeb0deca0ad668de7 committer: Oleksandr Byelkin timestamp: 2016-05-04 19:15:31 +0200 message: MDEV-9531 : GROUP_CONCAT with ORDER BY inside takes a lot of memory while it's executed Limitation added to Red-Black tree. —
            sanja Oleksandr Byelkin added a comment -

            I remove variable and made automatic minimum print output calculation:

            revision-id: 13b5098fcaa888173472d255e29aff22bcc5baae (mariadb-10.1.13-18-g13b5098)
            parent(s): 732adec0a4c75d99389230feeb0deca0ad668de7
            committer: Oleksandr Byelkin
            timestamp: 2016-06-28 10:59:59 +0200
            message:

            MDEV-9531: GROUP_CONCAT with ORDER BY inside takes a lot of memory while it's executed

            Limitation added to Red-Black tree.

            —

            sanja Oleksandr Byelkin added a comment - I remove variable and made automatic minimum print output calculation: revision-id: 13b5098fcaa888173472d255e29aff22bcc5baae (mariadb-10.1.13-18-g13b5098) parent(s): 732adec0a4c75d99389230feeb0deca0ad668de7 committer: Oleksandr Byelkin timestamp: 2016-06-28 10:59:59 +0200 message: MDEV-9531 : GROUP_CONCAT with ORDER BY inside takes a lot of memory while it's executed Limitation added to Red-Black tree. —
            sanja Oleksandr Byelkin added a comment -

            revision-id: f5a5ac49b5ff7b04aa816d33ca44d9fe88a94498 (mariadb-10.1.17-23-gf5a5ac4)
            parent(s): d30809a3cde823ad696304a941afe5a562bfa3ed
            committer: Oleksandr Byelkin
            timestamp: 2016-09-26 16:21:18 +0200
            message:

            MDEV-9531: GROUP_CONCAT with ORDER BY inside takes a lot of memory while it's executed

            Limitation added to Red-Black tree.

            —

            sanja Oleksandr Byelkin added a comment - revision-id: f5a5ac49b5ff7b04aa816d33ca44d9fe88a94498 (mariadb-10.1.17-23-gf5a5ac4) parent(s): d30809a3cde823ad696304a941afe5a562bfa3ed committer: Oleksandr Byelkin timestamp: 2016-09-26 16:21:18 +0200 message: MDEV-9531 : GROUP_CONCAT with ORDER BY inside takes a lot of memory while it's executed Limitation added to Red-Black tree. —
            serg Sergei Golubchik added a comment -

            please, review commit 73b12b0cebc in bb-10.1-serg

            serg Sergei Golubchik added a comment - please, review commit 73b12b0cebc in bb-10.1-serg
            sanja Oleksandr Byelkin added a comment -

            OK to push.

            sanja Oleksandr Byelkin added a comment - OK to push.

            People

              serg Sergei Golubchik
              elenst Elena Stepanova
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.