Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-32045

memory accumulation in the docker environment in 10.6.14

Details

    • Bug
    • Status: Closed (View Workflow)
    • Critical
    • Resolution: Incomplete
    • 10.6.14, 10.11.5
    • N/A
    • Docker
    • docker

    Description

      Customer reported memory leak in the docker environment in 10.6.14 track and provided numerous performance_schema.memory_summary_global_by_event_name
      outputs

      According to Customer the issues is beyond existing JIRA reports for 10.6 track

      Attachments

        1. lab_cmdb0_memory
          34 kB
        2. lab_cmdb1_memory
          36 kB
        3. lab_cmdb2_memory
          33 kB
        4. mariadb-1_current_bytes.txt
          8 kB
        5. mariadb-1_engine_inno.txt
          10 kB
        6. mariadb-1_global_stats.txt
          29 kB
        7. mariadb-1_global_vars.txt
          19 kB
        8. mariadb-1_meminfo.txt
          0.0 kB
        9. mariadb-1_memory_summary.txt
          17 kB
        10. mariadb-2_current_bytes.txt
          8 kB
        11. mariadb-2_engine_inno.txt
          10 kB
        12. mariadb-2_global_stats.txt
          29 kB
        13. mariadb-2_global_vars.txt
          19 kB
        14. mariadb-2_meminfo.txt
          0.0 kB
        15. mariadb-2_memory_summary.txt
          17 kB
        16. second_run_cmdb0_memory
          33 kB
        17. second_run_cmdb1_memory
          33 kB
        18. second_run_cmdb2_memory
          33 kB
        19. test_cmdb0_memory
          17 kB
        20. test_cmdb1_memory
          17 kB
        21. test_cmdb2_memory.txt
          17 kB

        Activity

          YK Yakov Kushnirsky created issue -
          YK Yakov Kushnirsky made changes -
          Field Original Value New Value
          Attachment mariadb-1_meminfo.txt [ 71756 ]
          Attachment mariadb-1_memory_summary.txt [ 71757 ]
          Attachment mariadb-2_meminfo.txt [ 71758 ]
          Attachment mariadb-2_memory_summary.txt [ 71759 ]
          julien.fritsch Julien Fritsch made changes -
          Fix Version/s 10.4 [ 22408 ]
          Fix Version/s 10.5 [ 23123 ]
          Fix Version/s 10.6 [ 24028 ]
          julien.fritsch Julien Fritsch made changes -
          Assignee Daniel Black [ danblack ]
          julien.fritsch Julien Fritsch made changes -
          Component/s Docker [ 16900 ]
          julien.fritsch Julien Fritsch made changes -
          Fix Version/s 10.4 [ 22408 ]
          Fix Version/s 10.5 [ 23123 ]
          danblack Daniel Black added a comment -

          while its in Docker, the memory leak isn't going to be docker related, its all in server.

          jemalloc is in the container so testing its capabilities of leak detection is one option.

          LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.2 as an environment variable to the container runtime will enable it along with the desired MALLOC_CONF.

          danblack Daniel Black added a comment - while its in Docker, the memory leak isn't going to be docker related, its all in server. jemalloc is in the container so testing its capabilities of leak detection is one option. LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.2 as an environment variable to the container runtime will enable it along with the desired MALLOC_CONF.
          julien.fritsch Julien Fritsch made changes -
          Assignee Daniel Black [ danblack ] Oleksandr Byelkin [ sanja ]
          julien.fritsch Julien Fritsch made changes -
          Labels Memory_leak
          julien.fritsch Julien Fritsch made changes -
          Priority Major [ 3 ] Critical [ 2 ]

          JFYI Dmitri have yet another 4 melory leak fixed and continue. That memory leak can not be detected by usual mean because if dissconect memory will be freed.

          So the question, is memory freed in this case if reconnect/dissconnect all clients?

          sanja Oleksandr Byelkin added a comment - JFYI Dmitri have yet another 4 melory leak fixed and continue. That memory leak can not be detected by usual mean because if dissconect memory will be freed. So the question, is memory freed in this case if reconnect/dissconnect all clients?
          julien.fritsch Julien Fritsch made changes -
          Status Open [ 1 ] Needs Feedback [ 10501 ]
          YK Yakov Kushnirsky made changes -
          Attachment mariadb-1_current_bytes.txt [ 71778 ]
          Attachment mariadb-1_engine_inno.txt [ 71779 ]
          Attachment mariadb-1_global_stats.txt [ 71780 ]
          Attachment mariadb-1_global_vars.txt [ 71781 ]
          Attachment mariadb-2_current_bytes.txt [ 71782 ]
          Attachment mariadb-2_engine_inno.txt [ 71783 ]
          Attachment mariadb-2_global_stats.txt [ 71784 ]
          Attachment mariadb-2_global_vars.txt [ 71785 ]

          Thank you all! I asked Customer in the case 1)if disconnecting all clients will free memory; 2)use jemalloc for leak detection. Will update JIRA again with his answers.
          Adding available global and status variables to the JIRA to supplement posted before P_S reports (all for the same datetime - 8/21).

          YK Yakov Kushnirsky added a comment - Thank you all! I asked Customer in the case 1)if disconnecting all clients will free memory; 2)use jemalloc for leak detection. Will update JIRA again with his answers. Adding available global and status variables to the JIRA to supplement posted before P_S reports (all for the same datetime - 8/21).
          YK Yakov Kushnirsky made changes -
          Status Needs Feedback [ 10501 ] Open [ 1 ]
          bengrieser Ben Grieser made changes -
          Affects Version/s 10.11.5 [ 29019 ]
          maxmether Max Mether made changes -
          Assignee Oleksandr Byelkin [ sanja ] Dmitry Shulga [ JIRAUSER47315 ]

          I cannot find any information in the provided stats that show any memory leaks over time.
          Please ask the user to provide us with
          show global status like "memory_used%";
          select * from information_schema.processlist;

          Run every 60 seconds to show what memory is growing.

          monty Michael Widenius added a comment - I cannot find any information in the provided stats that show any memory leaks over time. Please ask the user to provide us with show global status like "memory_used%"; select * from information_schema.processlist; Run every 60 seconds to show what memory is growing.
          julien.fritsch Julien Fritsch made changes -
          Status Open [ 1 ] Needs Feedback [ 10501 ]
          bengrieser Ben Grieser made changes -
          Attachment lab_cmdb0_memory [ 71903 ]
          Attachment lab_cmdb1_memory [ 71904 ]
          Attachment lab_cmdb2_memory [ 71905 ]
          Attachment second_run_cmdb0_memory [ 71906 ]
          Attachment second_run_cmdb1_memory [ 71907 ]
          Attachment second_run_cmdb2_memory [ 71908 ]
          Attachment test_cmdb0_memory [ 71909 ]
          Attachment test_cmdb1_memory [ 71910 ]
          Attachment test_cmdb2_memory [ 71911 ]
          bengrieser Ben Grieser added a comment - - edited

          I've attached memory used information provided by the user in CS0574876. I think this is collected in three different runs. The files are prepended with lab, test, and second_run, and were supplied by the user on different dates in that order.

          bengrieser Ben Grieser added a comment - - edited I've attached memory used information provided by the user in CS0574876. I think this is collected in three different runs. The files are prepended with lab, test, and second_run, and were supplied by the user on different dates in that order.
          bengrieser Ben Grieser made changes -
          Attachment test_cmdb2_memory [ 71911 ]
          bengrieser Ben Grieser made changes -
          Attachment test_cmdb2_memory.txt [ 71912 ]
          YK Yakov Kushnirsky made changes -
          Status Needs Feedback [ 10501 ] Open [ 1 ]
          julien.fritsch Julien Fritsch made changes -
          Status Open [ 1 ] Needs Feedback [ 10501 ]
          julien.fritsch Julien Fritsch made changes -
          Assignee Dmitry Shulga [ JIRAUSER47315 ] Ben Grieser [ JIRAUSER49916 ]
          julien.fritsch Julien Fritsch made changes -
          Fix Version/s N/A [ 14700 ]
          Fix Version/s 10.6 [ 24028 ]
          Resolution Incomplete [ 4 ]
          Status Needs Feedback [ 10501 ] Closed [ 6 ]
          mariadb-jira-automation Jira Automation (IT) made changes -
          Zendesk Related Tickets 201626 179075
          Zendesk active tickets 201626

          People

            bengrieser Ben Grieser
            YK Yakov Kushnirsky
            Votes:
            1 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.