This was accidentally found by merging
MDEV-13051 test to 10.2.
When InnoDB is trying to close data files to obey the innodb_open_files limit, it will release fil_system->mutex and fail to reacquire it. In a debug build, this will lead to a debug assertion failure.
In a non-debug build, it could lead to a hang or crash (I did not test).