Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-16838

Mariadb (with TokuDB) excessive memory usage/leak

Details

    Description

      Same issue as MDEV-13403.

      After a couple of days and quite a bit of "load data infile", mysqld dies with out of memory (eventually consuming 1TB of RAM and 100GB of swap).

      We have several identical servers (load balanced), and they keep crashing with out of memory every couple of days one after the other.

      All our tables are using tokudb.

      Our cnf file is attached.

      Attachments

        Issue Links

          Activity

            mhadji@gmail.com Marios Hadjieleftheriou added a comment - - edited

            We have a hunch that this is related to haproxy. In the general query log we see this every 2 seconds:
            180727 16:31:21 44597 Connect haproxy_check@... as anonymous on
            44597 Quit
            180727 16:31:23 44598 Connect haproxy_check@... as anonymous on
            44598 Quit

            When we had a single server with exactly the same install, the box was up and running fine for more than two weeks. Once we setup the replicas and started running haproxy, out of memory issues ensued.

            We will stop haproxy and see if that alleviates the problem, but this still sounds like a mysql memory leak (maybe connection related).

            mhadji@gmail.com Marios Hadjieleftheriou added a comment - - edited We have a hunch that this is related to haproxy. In the general query log we see this every 2 seconds: 180727 16:31:21 44597 Connect haproxy_check@... as anonymous on 44597 Quit 180727 16:31:23 44598 Connect haproxy_check@... as anonymous on 44598 Quit When we had a single server with exactly the same install, the box was up and running fine for more than two weeks. Once we setup the replicas and started running haproxy, out of memory issues ensued. We will stop haproxy and see if that alleviates the problem, but this still sounds like a mysql memory leak (maybe connection related).

            We disabled haproxy and the problem did not go away. I realized that certain queries on very large tables (400 million rows) drive memory up quickly by several GBs, which are never released. A few consecutive invocations of such a query will crash mysqld. The query looks like this:
            select * from x where y in (4000 strings here)

            mhadji@gmail.com Marios Hadjieleftheriou added a comment - We disabled haproxy and the problem did not go away. I realized that certain queries on very large tables (400 million rows) drive memory up quickly by several GBs, which are never released. A few consecutive invocations of such a query will crash mysqld. The query looks like this: select * from x where y in (4000 strings here)

            We solved the issue by adding this to our cnf file:

            [mysqld_safe]
            malloc-lib= /path/to/jemalloc
            

            We compiled MariaDB from source, with non-standard configuration files. The problem might have been self inflicted.

            Nevertheless, in MariaDB tokuDB plugin installation page, there is no mention that tokuDB requires jemalloc. I found this information on the Percona installation page.

            mhadji@gmail.com Marios Hadjieleftheriou added a comment - We solved the issue by adding this to our cnf file: [mysqld_safe] malloc-lib= /path/to/jemalloc We compiled MariaDB from source, with non-standard configuration files. The problem might have been self inflicted. Nevertheless, in MariaDB tokuDB plugin installation page , there is no mention that tokuDB requires jemalloc. I found this information on the Percona installation page .

            greenman,
            there is quite a lot of confusion around TokuDB requiring/wanting jemalloc, reasoning for using/not using it, relation of using it with the hugepages issue, etc. I think it's definitely worth a separate section somewhere in TokuDB documentation. We do have a note about hugepages, for example, but we don't connect it anyhow with libjemalloc, etc.

            elenst Elena Stepanova added a comment - greenman , there is quite a lot of confusion around TokuDB requiring/wanting jemalloc, reasoning for using/not using it, relation of using it with the hugepages issue, etc. I think it's definitely worth a separate section somewhere in TokuDB documentation. We do have a note about hugepages, for example, but we don't connect it anyhow with libjemalloc, etc.
            greenman Ian Gilfillan added a comment -

            libjemalloc is listed as a requirement upstream, so have added a note about it at https://mariadb.com/kb/en/library/installing-tokudb/.

            greenman Ian Gilfillan added a comment - libjemalloc is listed as a requirement upstream, so have added a note about it at https://mariadb.com/kb/en/library/installing-tokudb/ .
            sre Stefan Reger added a comment -

            We have the same problem here with Debian Buster and libjemalloc2 (v5.1.0-3). If we switch to libjemalloc1 provided by the mariadb debian buster repository, it works fine (v3.6.0-9.1). Unfortunately the mariadb installation uses libjemalloc2 if this has already been installed before, so we have to manually switch to the older version.

            sre Stefan Reger added a comment - We have the same problem here with Debian Buster and libjemalloc2 (v5.1.0-3). If we switch to libjemalloc1 provided by the mariadb debian buster repository, it works fine (v3.6.0-9.1). Unfortunately the mariadb installation uses libjemalloc2 if this has already been installed before, so we have to manually switch to the older version.

            People

              greenman Ian Gilfillan
              mhadji@gmail.com Marios Hadjieleftheriou
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.