Slow InnoDB shutdown on large instance

Shutting down al large windows instance (innodb_pool_buffer_size=384GB, > 500K tables) takes much longer with 10.4 than it did with 10.3.

Shutdown takes around 10 minutes even with innodb_fast_shutdown=1 or =3.

innodb_defragment was originally enabled, but even after turning that off shutdown still took several minutes, even though less than with defragmentation enabled.

Nothing special in the error log besides [Note] InnoDB: Waiting for master threadto exit once per minute.