Thanks Sergei
Sorry for the complaining post - I was very tired and frustrated after 48hrs of debugging and up&downgrading & testing multiple config settings => realizing that I had no way out and deciding to export and then later import all data (to downgrade) was the final heavy hit.

I'm currently downgrading back to 10.1(.33) mainly because I noticed that I had non-optimal insert-performance as well with the old 10.2.15 (when I wrote the above post I did not realize that I did not have to export+import again everything to do a minor downgrade) and as well with 10.3.8 (the major upgrade was my final hope...) => I therefore suppose that some change done in 10.2/3 is causing this.
I'll then upgrade to 10.1.35 whenever it becomes available (I need a bugfix which is included there) and then I think that I'll try to stay on 10.1.35 hoping that I don't need anything new.

Yes, I will try to simplify and reproduce the "good vs bad" performance (in VMs) and then post a bug report.

I will shut up about the memory leak for the time being as long as nobody else complains and I don't have to upgrade to 10.2/3

Cheers and thanks a lot for your help.

Can confirm that with 10.2.16 the leak/excessive memory usage is gone.
(tested on several databases/datasets which hit OOM few minutes later with versions 10.2.5+, now 10.2.16 running for several days within constant memory amount/limits without a problem).

Thank you for the fix.

We are running MariaDB 10.3.7 with tokudb 5.6.39-83.1 on Debian 9 and we are experiencing exactly the same problem. It seems that all queries drive memory up, which is never released, but some big selects on very large tables (400 million rows) like this:
select * from x where y in (4000 strings here)
can increase memory usage by several GB instantaneously that are never released. A few consecutive invocations of this query will crash mysqld.

Hi all
Sorry, I still have to complain, ggghhh
I'm still seeing high RAM usage.
And sorry, I did not do any "clean" tests yet - I just felt I had to post what I knew so far (so many stability problems with MariaDB so far => starting to turn my attention towards Apache Kudu & Yandex Clickhouse...).

It's definitely not as obviously visible as with versions equal/lower than 10.2.15, but in my opinion the problem is still there.

Attaching the file "stefano-mariadb-10_02_16-ram-20180825.png":

The graph shows the RSS allocation from a clean start up to ~2.5 billion inserts (2'537'827'547) (some very few selects & updates were executed as well, but at most 0.001% of all DMLs).

My current setting is"tokudb_cache_size = 12G", so I wouldn't expect it to go a lot beyond that - 15GB is definitely beyond my expectations. Sometimes it went up to 26 GBs, but I've always been able to shut it down before it started doing some serious swapping.

I admit that the graph isn't very exciting, but I've seen numbers grow a lot faster under certain conditions (e.g. concurrent queries like "insert into [my_isam_table] select a_single_column from [my_tokudb_tbl]>" + "insert into [my_VEC_table] select some, other, cols from [my_tokudb_tbl]" made my 32GB server start swapping within minutes - sorry, I did not repeat the experiment.

MariaDB 10.1 (used mostly 10.1.34) did not have this problem but totally hanged after ~1 days of continuous "insert"s with no kind of hint of "why" anywhere => I usually had to "kill -9 xxx" the mariadb-process & start it & wait for a recovery", so from this point of view the 10.2-series is better (no hard hangs even after having processed the same amount of data as the 10.1-series) but for me it's still ultimately unstable.

(btw. when using TokuDB the insert-rate with the 10.1-series is still 10-20% faster than with 10.2/3-series; looks like 10.2/3 uses max 25-50-75% of the available vCPUs for the checkpoints, but this is another issue...)

Question:
is anybody doing at least ~1 million "inserts" upwards per minute for an extended amount of time (hours, days, weeks) and if yes, do you see any suspicious RAM allocation/behaviour?
(asking this because the factor of "time elapsed vs. amount of inserts" might have something to do with the problem)
Thanks a lot
Stefano

> My current setting is"tokudb_cache_size = 12G", so I wouldn't expect it to go a lot beyond that - 15GB is definitely beyond my expectations.

For 12G only tokudb cache 15Gb memory usage for the whole mysqld process seems fairly normal - there are bunch of other things mysqls uses/wastes memory on (like connection / sort buffers / other engines etc) and the 10.2.x/10.3.x series use a bit more ram anyways (might be just the changes in upstream).

For testing purposes I would suggest setting the cache size to something smaller like 1G and then checking if the server starts to swap or at what point the used memory stabilises (before this patch (the way Mariadb was compiled/packed) the cache size didn't even matter and you hit OOM sooner or later anyways).

p.s. It's also worth checking what Jemalloc version you are running - for example I found out that 5.0 (coming with OpenSuse Leap 15) makes the mysqld crash on shutdown which then makes the sytemd to restart the process into loop. With 5.1 ( https://github.com/jemalloc/jemalloc/releases/tag/5.1.0 ) haven't noticed any problems so far.