[MDEV-16838] Mariadb (with TokuDB) excessive memory usage/leak Created: 2018-07-27  Updated: 2019-10-10  Resolved: 2018-08-06

Status: Closed
Project: MariaDB Server
Component/s: Documentation, Storage Engine - TokuDB
Affects Version/s: 10.3.7
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Marios Hadjieleftheriou Assignee: Ian Gilfillan
Resolution: Fixed Votes: 0
Labels: crash
Environment:

Linux 4.9.0-5-amd64 #1 SMP Debian 4.9.65-3+deb9u2 (2018-01-04) x86_64 GNU/Linux

Attachments: File my.cnf    
Issue Links:
Relates
relates to MDEV-13403 Mariadb (with TokuDB) excessive memor... Closed

 Description   

Same issue as MDEV-13403.

After a couple of days and quite a bit of "load data infile", mysqld dies with out of memory (eventually consuming 1TB of RAM and 100GB of swap).

We have several identical servers (load balanced), and they keep crashing with out of memory every couple of days one after the other.

All our tables are using tokudb.

Our cnf file is attached.

 Comments   
Comment by Marios Hadjieleftheriou [ 2018-07-27 ]

We have a hunch that this is related to haproxy. In the general query log we see this every 2 seconds:
180727 16:31:21 44597 Connect haproxy_check@... as anonymous on
44597 Quit
180727 16:31:23 44598 Connect haproxy_check@... as anonymous on
44598 Quit

When we had a single server with exactly the same install, the box was up and running fine for more than two weeks. Once we setup the replicas and started running haproxy, out of memory issues ensued.

We will stop haproxy and see if that alleviates the problem, but this still sounds like a mysql memory leak (maybe connection related).

Comment by Marios Hadjieleftheriou [ 2018-07-28 ]

We disabled haproxy and the problem did not go away. I realized that certain queries on very large tables (400 million rows) drive memory up quickly by several GBs, which are never released. A few consecutive invocations of such a query will crash mysqld. The query looks like this:
select * from x where y in (4000 strings here)

Comment by Marios Hadjieleftheriou [ 2018-08-04 ]

We solved the issue by adding this to our cnf file:

[mysqld_safe]
 
malloc-lib= /path/to/jemalloc

We compiled MariaDB from source, with non-standard configuration files. The problem might have been self inflicted.

Nevertheless, in MariaDB tokuDB plugin installation page, there is no mention that tokuDB requires jemalloc. I found this information on the Percona installation page.

Comment by Elena Stepanova [ 2018-08-04 ]

greenman,
there is quite a lot of confusion around TokuDB requiring/wanting jemalloc, reasoning for using/not using it, relation of using it with the hugepages issue, etc. I think it's definitely worth a separate section somewhere in TokuDB documentation. We do have a note about hugepages, for example, but we don't connect it anyhow with libjemalloc, etc.

Comment by Ian Gilfillan [ 2018-08-06 ]

libjemalloc is listed as a requirement upstream, so have added a note about it at https://mariadb.com/kb/en/library/installing-tokudb/.

Comment by Stefan Reger [ 2019-10-10 ]

We have the same problem here with Debian Buster and libjemalloc2 (v5.1.0-3). If we switch to libjemalloc1 provided by the mariadb debian buster repository, it works fine (v3.6.0-9.1). Unfortunately the mariadb installation uses libjemalloc2 if this has already been installed before, so we have to manually switch to the older version.

Generated at Thu Feb 08 08:31:56 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.