Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-14050

Memory not freed on memory table drop

Details

    Description

      The issue is very simple.

      I have multiple tables using storage ENGINE MEMORY . These tables can range in size from 2Kb to 500Mb. They all have about 15 normal int columns with one BIG UNSIGNED INT column and HASH index on every column.

      I just updated from MariaDB 10.1.28 to MariaDB 10.2.9 and there is a major issue I have encountered:
      using the DROP TABLE <tablename> command and TRUNCATE TABLE <tablename> does not free the server memory on tables using ENGINE MEMORY. As a results the mysql server instance keeps growing as I delete and create new tables until the server crashes. Filling up my server's memory takes about one day.

      The server also have Replication enabled, but not on the database that uses MEMORY tables.

      Edit: After further investigation, it seems the memory is indeed freed but only after some time. Thus if you delete say a large number of tables and create a large number of tables in short time-span using Engine Memory, there is a high risk of running out of memory.

      Attachments

        1. Global.txt
          21 kB
          Silviu Dan Tanasie

        Issue Links

          Activity

            Starchaser Silviu Dan Tanasie added a comment - - edited

            Fortunately, I can help with a little bit more information.

            • We have around 1500 - 2000 tables, depends, but never more than 2000
            • The table size varies, some have a few rows, some may have millions of rows. Out of the 1700 tables we have right now, most are under 20MB and I think 16-17 are above 100MB and the biggest one is 450MB. If it helps I can also provide row numbers and index numbers.

            Our workaround to this problem, that has never failed so far, is to have a job that restarts the MariaDB server once in a while and regenerate all the in-memory tables. If the MariaDB server is restarted, the memory is released.

            Starchaser Silviu Dan Tanasie added a comment - - edited Fortunately, I can help with a little bit more information. We have around 1500 - 2000 tables, depends, but never more than 2000 The table size varies, some have a few rows, some may have millions of rows. Out of the 1700 tables we have right now, most are under 20MB and I think 16-17 are above 100MB and the biggest one is 450MB. If it helps I can also provide row numbers and index numbers. Our workaround to this problem, that has never failed so far, is to have a job that restarts the MariaDB server once in a while and regenerate all the in-memory tables. If the MariaDB server is restarted, the memory is released.

            But you have not tried more memory or less table, it looks like the issue connected with memory fragmentation and so with intense use it just require much more memory than was allocated.

            I have not found memory leaks, or huge amount non-counted memory (limited size structures only). So I think it is just a memory fragmentation (which fought with different efficiency by different allocators but can not be removed with current approach of allocating memory for heap tables).

            Feel free to reopen bug if memory will not stop growing after increasing memory or decreasing volume of temporary data.

            sanja Oleksandr Byelkin added a comment - But you have not tried more memory or less table, it looks like the issue connected with memory fragmentation and so with intense use it just require much more memory than was allocated. I have not found memory leaks, or huge amount non-counted memory (limited size structures only). So I think it is just a memory fragmentation (which fought with different efficiency by different allocators but can not be removed with current approach of allocating memory for heap tables). Feel free to reopen bug if memory will not stop growing after increasing memory or decreasing volume of temporary data.

            Hi,

            Is there any way to check what memory management library MariaDB is using?

            The server is still crashing, albeit only once a week, with jemalloc.

            Starchaser Silviu Dan Tanasie added a comment - Hi, Is there any way to check what memory management library MariaDB is using? The server is still crashing, albeit only once a week, with jemalloc.

            It's time to finally put this issue to rest.
            I have been struggling with getting MariaDB to use a different memory allocator. The documentation seems poor and outdated on this matter.
            I have initially tried to use the [mysqld_safe] option, but that doesn't do anything under Centos at least.

            The one solution that works is to edit the service file on Centos: /usr/lib/systemd/system/mariadb.service
            You need to add LD_PRELOAD="<malloc library>"

            With this method, I was able to load my custom malloc library and finally, the memory allocation problem has gone away.

            For the future, I suggest to the MariaDB team to have more explicit features on using different memory allocation libraries.

            Starchaser Silviu Dan Tanasie added a comment - It's time to finally put this issue to rest. I have been struggling with getting MariaDB to use a different memory allocator. The documentation seems poor and outdated on this matter. I have initially tried to use the [mysqld_safe] option, but that doesn't do anything under Centos at least. The one solution that works is to edit the service file on Centos: /usr/lib/systemd/system/mariadb.service You need to add LD_PRELOAD="<malloc library>" With this method, I was able to load my custom malloc library and finally, the memory allocation problem has gone away. For the future, I suggest to the MariaDB team to have more explicit features on using different memory allocation libraries.

            Small tip, if you edit that file in /usr/lib directly, it gets replaced when you update packages - instead you can run:

            systemctl edit mariadb

            and then add:

            [Service]
            LD_PRELOAD="malloc library"
            
            

            This will create an "override" to the systemd service definition, which will remain there over package upgrades, etc.
            (basically it save the file into /etc/systemd/system/mariadb.service.d/override.conf )

            I've also have had good experiences with using jemalloc w mariadb so far.

            rlam Roope Pääkkönen (Inactive) added a comment - Small tip, if you edit that file in /usr/lib directly, it gets replaced when you update packages - instead you can run: systemctl edit mariadb and then add: [Service] LD_PRELOAD="malloc library" This will create an "override" to the systemd service definition, which will remain there over package upgrades, etc. (basically it save the file into /etc/systemd/system/mariadb.service.d/override.conf ) I've also have had good experiences with using jemalloc w mariadb so far.

            People

              sanja Oleksandr Byelkin
              Starchaser Silviu Dan Tanasie
              Votes:
              4 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.