The documentation at https://mariadb.com/kb/en/innodb-temporary-tablespaces/ says:
Unlike normal tablespaces, temporary tablespaces are deleted when you stop MariaDB. To shrink temporary tablespaces to their minimum sizes, restart the server.
This means temporary tablespaces will keep growing until MariaDB is restarted. Having to restart MariaDB periodically is not optimal. It would be welcome if MariaDB shrinks temporary tablespaces on disk without restarting.