Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-13403

Mariadb (with TokuDB) excessive memory usage/leak

    Details

      Description

      We have been running MariaDB 10.1 with TokuDB on a Ubuntu 14.04 VPS with 4GB ram. This always worked fine. We recently updated to 10.2 and suddenly MariaDB started eating all the memory there is and also uses a lot of swap. We did upgrade our VPS to a Ubuntu 16.04 8 GB instance (not because of the problems, but just because that would improve performance). Here the issues continued.

      Settings did not change between te VPS instances, we only allocated 4GB ram to TokuDB instead of 2GB.

      Under the same workload 10.2 eats up all RAM (using 7/8GB ram + 2/8GB Swap) after 2 days, while under 10.1 the ram usage stayed in line with what you would expect.

      Unfortunately we can't go back to 10.1, since importing our dataset takes a week.

      Our database consists mainly of TokuDB tables, with one table having 9 billion rows. Other tables are in the lower million rows. Total size inclusing indexes is 900GB (uncompressed) and 300GB without indexes.
      Workload is lots of reads and inserts, but no deletes.
      Strangely the memory balloons most when running the daily stats gathering, which is almost a pure read query, except for some stats entries that get inserted.

      We do have a staging server that we can use to run valgrind massive on, and if necessary also on production, since the project is not very critical. However, we are still looking to reproduce the issue on the staging server. Also valgrind massive output does show a lot of '??' entries, even though we installed mariadb-server-core-dgbsym, mariadb-server-dbgsym and mariadb-plugins-tokudb-dbgsym.

      I will try to replicate the issue on the staging environment or otherwise use valgrind on production. However, I am not sure if massive option doesn't use much extra ram, making it hard to actually get to the ballooned ram issue.

      I attached the most relevant output from mysql and some graphs from grafana.
      Let me know if you need more.

        Attachments

        1. 10.2.10_leak.png
          10.2.10_leak.png
          25 kB
        2. connections-memory-usage.txt
          1 kB
        3. dbserver-memory-usage.png
          dbserver-memory-usage.png
          36 kB
        4. dbserver-swap-usage.png
          dbserver-swap-usage.png
          20 kB
        5. dbserver-swap-usage.txt
          0.7 kB
        6. innodb-status.txt
          5 kB
        7. leak.png
          leak.png
          34 kB
        8. memory usage 1-day.png
          memory usage 1-day.png
          682 kB
        9. my.cnf
          2 kB
        10. my.cnf
          4 kB
        11. mysql-psaux.txt
          0.2 kB
        12. mysqltuner.txt
          3 kB
        13. old-version-older-server.png
          old-version-older-server.png
          62 kB
        14. status.txt
          88 kB
        15. stefano-1-ps-mysqld-rss.png
          stefano-1-ps-mysqld-rss.png
          19 kB
        16. stefano-1-ps-mysqld-rss.txt
          185 kB
        17. stefano-mariadb-10_02_16-ram-20180825.png
          stefano-mariadb-10_02_16-ram-20180825.png
          204 kB
        18. tokudb.cnf
          0.1 kB
        19. tokudb-status.txt
          46 kB

          Issue Links

            Activity

              People

              • Assignee:
                serg Sergei Golubchik
                Reporter:
                Peterdk Peter de Kraker
              • Votes:
                17 Vote for this issue
                Watchers:
                27 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: