Table handler out of memory

PHP Fatal error: Uncaught PDOException: SQLSTATE[HY000]: General error: 128 Table handler out of memory in /home/root/datasearch.php:298 Stack trace:
It seems that the "tabler handler" has run out of memory. But I can't find how to increase this in the settings from MySQL/MariaDB or if this is even something you can adjust.
The query is a simple MATCH AGAINST but is expected to return a few million results. We want this for statistics analysis so it's wanted to get that many results back.

This is info on the server: https://jira.mariadb.org/browse/MDEV-11063
Nothing has changed in the settings since then.