The issue is very simple.
I have multiple tables using storage ENGINE MEMORY . These tables can range in size from 2Kb to 500Mb. They all have about 15 normal int columns with one BIG UNSIGNED INT column and HASH index on every column.
I just updated from MariaDB 10.1.28 to MariaDB 10.2.9 and there is a major issue I have encountered:
using the DROP TABLE <tablename> command and TRUNCATE TABLE <tablename> does not free the server memory on tables using ENGINE MEMORY. As a results the mysql server instance keeps growing as I delete and create new tables until the server crashes. Filling up my server's memory takes about one day.
The server also have Replication enabled, but not on the database that uses MEMORY tables.
Edit: After further investigation, it seems the memory is indeed freed but only after some time. Thus if you delete say a large number of tables and create a large number of tables in short time-span using Engine Memory, there is a high risk of running out of memory.