We've recently migrated a legacy application from MySQL 5.5 on Amazon Linux to MariaDB 10.5.8 (latest, from the official repos) on Amazon Linux 2. The "hardware" went from an m3.large instance to an m5.large instance (both 8GB RAM, 2 CPUs).

After the upgrade we're seeing mariadbd RSS growing linearly (well beyond configured limits) until it gets OOM killed by the kernel.

The application uses MyISAM and INSERT DELAYED fir the vast majority of its write-heavy 18G data set.

Disabling use of delayed inserts appears to mitigate the problem.

I ran a similar workload in our staging environment to capture heap traces and profile them with pprof. Here are the results of ~2 hours of runtime both with INSERT DELAYED enabled and disabled:

With INSERT DELAYED disabled:

See pprof SVG output:
https://tempcfntest.s3.amazonaws.com/leaks/noleak.html

Dec 18 00:20:43 Starting tracking the heap
Dec 18 00:20:43 Dumping heap profile to /tmp/mysqld.prof_16144.0001.heap (151 MB currently in use)
Dec 18 00:20:43 Dumping heap profile to /tmp/mysqld.prof_16144.0002.heap (338 MB currently in use)
Dec 18 00:27:22 Dumping heap profile to /tmp/mysqld.prof_16144.0003.heap (1362 MB allocated cumulatively, 378 MB currently in use)
Dec 18 00:34:07 Dumping heap profile to /tmp/mysqld.prof_16144.0004.heap (2386 MB allocated cumulatively, 378 MB currently in use)
Dec 18 00:41:22 Dumping heap profile to /tmp/mysqld.prof_16144.0005.heap (3410 MB allocated cumulatively, 378 MB currently in use)
Dec 18 00:48:13 Dumping heap profile to /tmp/mysqld.prof_16144.0006.heap (4434 MB allocated cumulatively, 378 MB currently in use)

[...]

Dec 18 02:35:02 Dumping heap profile to /tmp/mysqld.prof_16144.0021.heap (19794 MB allocated cumulatively, 379 MB currently in use)
Dec 18 02:38:49 Dumping heap profile to /tmp/mysqld.prof_16144.0022.heap (Exiting, 9 kB in use)

With INSERT DELAYED enabled it grows continuously:

See pprof SVG output:
https://tempcfntest.s3.amazonaws.com/leaks/leak.html

Dec 18 02:38:50 Starting tracking the heap
Dec 18 02:38:50 Dumping heap profile to /tmp/mysqld.prof_785.0001.heap (151 MB currently in use)
Dec 18 02:38:50 Dumping heap profile to /tmp/mysqld.prof_785.0002.heap (338 MB currently in use)
Dec 18 02:44:39 Dumping heap profile to /tmp/mysqld.prof_785.0003.heap (1362 MB allocated cumulatively, 383 MB currently in use)
Dec 18 02:50:26 Dumping heap profile to /tmp/mysqld.prof_785.0004.heap (2386 MB allocated cumulatively, 388 MB currently in use)
Dec 18 03:02:54 Dumping heap profile to /tmp/mysqld.prof_785.0006.heap (4434 MB allocated cumulatively, 398 MB currently in use)
Dec 18 03:08:43 Dumping heap profile to /tmp/mysqld.prof_785.0007.heap (5458 MB allocated cumulatively, 403 MB currently in use)
Dec 18 03:14:57 Dumping heap profile to /tmp/mysqld.prof_785.0008.heap (6482 MB allocated cumulatively, 407 MB currently in use)
Dec 18 03:20:45 Dumping heap profile to /tmp/mysqld.prof_785.0009.heap (7506 MB allocated cumulatively, 412 MB currently in use)
Dec 18 03:26:57 Dumping heap profile to /tmp/mysqld.prof_785.0010.heap (8530 MB allocated cumulatively, 417 MB currently in use)
Dec 18 03:32:48 Dumping heap profile to /tmp/mysqld.prof_785.0011.heap (9554 MB allocated cumulatively, 421 MB currently in use)
Dec 18 03:38:42 Dumping heap profile to /tmp/mysqld.prof_785.0012.heap (10578 MB allocated cumulatively, 426 MB currently in use)
Dec 18 03:44:57 Dumping heap profile to /tmp/mysqld.prof_785.0013.heap (11602 MB allocated cumulatively, 430 MB currently in use)
Dec 18 03:50:45 Dumping heap profile to /tmp/mysqld.prof_785.0014.heap (12626 MB allocated cumulatively, 435 MB currently in use)
Dec 18 03:57:00 Dumping heap profile to /tmp/mysqld.prof_785.0015.heap (13650 MB allocated cumulatively, 440 MB currently in use)
Dec 18 04:03:13 Dumping heap profile to /tmp/mysqld.prof_785.0016.heap (14674 MB allocated cumulatively, 446 MB currently in use)
Dec 18 04:09:02 Dumping heap profile to /tmp/mysqld.prof_785.0017.heap (15698 MB allocated cumulatively, 449 MB currently in use)
Dec 18 04:15:13 Dumping heap profile to /tmp/mysqld.prof_785.0018.heap (16722 MB allocated cumulatively, 456 MB currently in use)
Dec 18 04:21:05 Dumping heap profile to /tmp/mysqld.prof_785.0019.heap (17746 MB allocated cumulatively, 459 MB currently in use)
Dec 18 04:27:16 Dumping heap profile to /tmp/mysqld.prof_785.0020.heap (18770 MB allocated cumulatively, 464 MB currently in use)
Dec 18 04:33:04 Dumping heap profile to /tmp/mysqld.prof_785.0021.heap (19794 MB allocated cumulatively, 468 MB currently in use)
Dec 18 04:38:52 Dumping heap profile to /tmp/mysqld.prof_785.0022.heap (20818 MB allocated cumulatively, 473 MB currently in use)
Dec 18 04:45:04 Dumping heap profile to /tmp/mysqld.prof_785.0023.heap (21842 MB allocated cumulatively, 478 MB currently in use)
Dec 18 04:50:52 Dumping heap profile to /tmp/mysqld.prof_785.0024.heap (22866 MB allocated cumulatively, 483 MB currently in use)