Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-13403

Mariadb (with TokuDB) excessive memory usage/leak

    XMLWordPrintable

Details

    Description

      We have been running MariaDB 10.1 with TokuDB on a Ubuntu 14.04 VPS with 4GB ram. This always worked fine. We recently updated to 10.2 and suddenly MariaDB started eating all the memory there is and also uses a lot of swap. We did upgrade our VPS to a Ubuntu 16.04 8 GB instance (not because of the problems, but just because that would improve performance). Here the issues continued.

      Settings did not change between te VPS instances, we only allocated 4GB ram to TokuDB instead of 2GB.

      Under the same workload 10.2 eats up all RAM (using 7/8GB ram + 2/8GB Swap) after 2 days, while under 10.1 the ram usage stayed in line with what you would expect.

      Unfortunately we can't go back to 10.1, since importing our dataset takes a week.

      Our database consists mainly of TokuDB tables, with one table having 9 billion rows. Other tables are in the lower million rows. Total size inclusing indexes is 900GB (uncompressed) and 300GB without indexes.
      Workload is lots of reads and inserts, but no deletes.
      Strangely the memory balloons most when running the daily stats gathering, which is almost a pure read query, except for some stats entries that get inserted.

      We do have a staging server that we can use to run valgrind massive on, and if necessary also on production, since the project is not very critical. However, we are still looking to reproduce the issue on the staging server. Also valgrind massive output does show a lot of '??' entries, even though we installed mariadb-server-core-dgbsym, mariadb-server-dbgsym and mariadb-plugins-tokudb-dbgsym.

      I will try to replicate the issue on the staging environment or otherwise use valgrind on production. However, I am not sure if massive option doesn't use much extra ram, making it hard to actually get to the ballooned ram issue.

      I attached the most relevant output from mysql and some graphs from grafana.
      Let me know if you need more.

      Attachments

        1. 10.2.10_leak.png
          25 kB
          Reinis Rozitis
        2. connections-memory-usage.txt
          1 kB
          Peter de Kraker
        3. dbserver-memory-usage.png
          36 kB
          Peter de Kraker
        4. dbserver-swap-usage.png
          20 kB
          Peter de Kraker
        5. dbserver-swap-usage.txt
          0.7 kB
          Peter de Kraker
        6. innodb-status.txt
          5 kB
          Peter de Kraker
        7. leak.png
          34 kB
          Reinis Rozitis
        8. memory usage 1-day.png
          682 kB
          Jeff
        9. my.cnf
          2 kB
          Michael OBrien
        10. my.cnf
          4 kB
          Peter de Kraker
        11. mysql-psaux.txt
          0.2 kB
          Peter de Kraker
        12. mysqltuner.txt
          3 kB
          Peter de Kraker
        13. old-version-older-server.png
          62 kB
          Peter de Kraker
        14. status.txt
          88 kB
          Peter de Kraker
        15. stefano-1-ps-mysqld-rss.png
          19 kB
          Stefano
        16. stefano-1-ps-mysqld-rss.txt
          185 kB
          Stefano
        17. stefano-mariadb-10_02_16-ram-20180825.png
          204 kB
          Stefano
        18. tokudb.cnf
          0.1 kB
          Peter de Kraker
        19. tokudb-status.txt
          46 kB
          Peter de Kraker

        Issue Links

          Activity

            People

              serg Sergei Golubchik
              Peterdk Peter de Kraker
              Votes:
              17 Vote for this issue
              Watchers:
              27 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.