Changes to the most accessed headers require recompilation of 500 files avg, which takes about 3 mins on 6 cores. The time tends to grow YoY.
Several of build time improvements may be approached:

• Header decomposition to improve files affected per line value.
• Use of precompiled headers
• Template optimization

This task is about the headers problem investigation.

• Take a look how a time changes with precompiled headers enabled.
• Work out the guidelines on enabling the precompiled headers generation for used system libraries.
• And/or update build scripts to support precompiled headers generation.
• If the build will still look too slow, analyze the template usages.

This tool may be helpful for the header analysis:
https://github.com/mikael-s-persson/templight

See also:

• How .gch works: https://gcc.gnu.org/onlinedocs/gcc/Precompiled-Headers.html
• For header dependency analysis: https://stackoverflow.com/a/60570938
• Clang modules: https://clang.llvm.org/docs/Modules.html#using-modules