open_tables count is increasing with >=mariadb:10.1.36 to 10.4.10

When I am trying to upgrade >=10.1.36 to 10.4.10, following issue is coming.
Count of open_tables increasing beyond 100K, if load increases it is going beyond 200K.
Due to this queries execution becomes slow, number of slow queries are increasing and after some time mariaDB is unable to handle requests.

This issue is not occurring with 10.1.35 and count of open_tables not crossing default limit(400),working perfectly with 10.1.35.

Suspicious commit:https://github.com/MariaDB/server/commit/64a23c1c8a826a6f58f8a415f60a0e3cc0e0375f
After commenting out the if-statement-block in sql/sql_base.cc from above commit issue is resolved and working properly.
if-statement : if (table_list->table->file->referenced_by_foreign_key())