Details
-
Task
-
Status: Stalled (View Workflow)
-
Major
-
Resolution: Unresolved
-
None
-
None
Description
Changes to the most accessed headers require recompilation of 500 files avg, which takes about 3 mins on 6 cores. The time tends to grow YoY.
Several of build time improvements may be approached:
- Header decomposition to improve files affected per line value.
- Use of precompiled headers
- Template optimization
This task is about the headers problem investigation.
- Take a look how a time changes with precompiled headers enabled.
- Work out the guidelines on enabling the precompiled headers generation for used system libraries.
- And/or update build scripts to support precompiled headers generation.
- If the build will still look too slow, analyze the template usages.
This tool may be helpful for the header analysis:
https://github.com/mikael-s-persson/templight
See also:
- How .gch works: https://gcc.gnu.org/onlinedocs/gcc/Precompiled-Headers.html
- For header dependency analysis: https://stackoverflow.com/a/60570938
- Clang modules: https://clang.llvm.org/docs/Modules.html#using-modules
Attachments
Activity
| Field | Original Value | New Value |
|---|---|---|
| Description | https://github.com/mikael-s-persson/templight |
Changes to the most accessed headers require recompilation of 500 files avg, which takes about 3 mins on 6 cores. The time tends to grow YoY.
Several of build time improvements may be approached: * Header decomposition to improve files affected per line value. https://github.com/mikael-s-persson/templight |
| Description |
Changes to the most accessed headers require recompilation of 500 files avg, which takes about 3 mins on 6 cores. The time tends to grow YoY.
Several of build time improvements may be approached: * Header decomposition to improve files affected per line value. https://github.com/mikael-s-persson/templight |
Changes to the most accessed headers require recompilation of 500 files avg, which takes about 3 mins on 6 cores. The time tends to grow YoY.
Several of build time improvements may be approached: * Header decomposition to improve files affected per line value. * Use of precompiled headers * Template optimization This task is about the headers problem investigation. * Take a look how a time changes with precompiled headers enabled. * Work out the guidelines on enabling the precompiled headers generation for used system libraries. * And/or update build scripts to support precompiled headers generation. * If the build will still look too slow, analyze the template usages. This tool may be helpful for the header analysis: https://github.com/mikael-s-persson/templight |
| Workflow | MariaDB v3 [ 122144 ] | MariaDB v4 [ 131503 ] |
| Fix Version/s | 10.2 [ 14601 ] |
| Status | Open [ 1 ] | In Progress [ 3 ] |
| Fix Version/s | 10.3 [ 22126 ] | |
| Fix Version/s | 10.4 [ 22408 ] | |
| Fix Version/s | 10.5 [ 23123 ] | |
| Fix Version/s | 10.6 [ 24028 ] | |
| Fix Version/s | 10.7 [ 24805 ] | |
| Fix Version/s | 10.8 [ 26121 ] |
Some intermediate results are here:
https://github.com/MariaDB/server/commits/nikita/MDEV-25774
| Status | In Progress [ 3 ] | Stalled [ 10000 ] |
| Description |
Changes to the most accessed headers require recompilation of 500 files avg, which takes about 3 mins on 6 cores. The time tends to grow YoY.
Several of build time improvements may be approached: * Header decomposition to improve files affected per line value. * Use of precompiled headers * Template optimization This task is about the headers problem investigation. * Take a look how a time changes with precompiled headers enabled. * Work out the guidelines on enabling the precompiled headers generation for used system libraries. * And/or update build scripts to support precompiled headers generation. * If the build will still look too slow, analyze the template usages. This tool may be helpful for the header analysis: https://github.com/mikael-s-persson/templight |
Changes to the most accessed headers require recompilation of 500 files avg, which takes about 3 mins on 6 cores. The time tends to grow YoY.
Several of build time improvements may be approached: * Header decomposition to improve files affected per line value. * Use of precompiled headers * Template optimization This task is about the headers problem investigation. * Take a look how a time changes with precompiled headers enabled. * Work out the guidelines on enabling the precompiled headers generation for used system libraries. * And/or update build scripts to support precompiled headers generation. * If the build will still look too slow, analyze the template usages. This tool may be helpful for the header analysis: https://github.com/mikael-s-persson/templight See also: * How .gch works: https://gcc.gnu.org/onlinedocs/gcc/Precompiled-Headers.html * For header dependency analysis: https://stackoverflow.com/a/60570938 * Clang modules: https://clang.llvm.org/docs/Modules.html#using-modules |
| Fix Version/s | 10.2 [ 14601 ] |
Can it be cmake config option turned off by default?
It would be also good if configuration could choose between gcc/clang for compilation, between ld/lld for linker and whether to ccache or not.
| Fix Version/s | 10.7 [ 24805 ] |
| Fix Version/s | 10.3 [ 22126 ] |
| Fix Version/s | 10.8 [ 26121 ] |
| Status | Stalled [ 10000 ] | In Progress [ 3 ] |
GCC's gch approach has no benefit – it also can only use a single gch per invocation.
Only one precompiled header can be used in a particular compilation.
At the same time, c++20 modules support is released both in gcc and clang just not far from now.
GCC:
Module support can be enabled standard-independently, by -fmodules-ts
Moreover, -fmodule-header can help treating header as a module, looking forward to see what it's capable of.
On integration with cmake: https://www.kitware.com/import-cmake-c20-modules/
Automated message:
----------------------------
Since this issue has not been updated since 6 weeks, it's time to move it back to Stalled.
Automated message:
----------------------------
Since this issue has not been updated since 6 weeks, it's time to move it back to Stalled.
| Status | In Progress [ 3 ] | Stalled [ 10000 ] |
| Fix Version/s | 10.4 [ 22408 ] |
so far precompiling the headers reduces the build time about 5 times:
for ninja minbuild with PFS disabled it dropped from 2:30 to 0:30!
The major disadvantage for this method is that all the headers precompiled are stored in one precompiled file which is forcefully pre-included in every build.
Thus, by using PCH, one may end up with using symbols from the headers that are not actually included.
The alternative is of course c++20 submodules, which does the same, but smart.
The another alternative is GCC which handles .gch files differently: if the precompiled file is available, it is being used, or else the actual header is taken.
So for now I can try to use the alternative way for GCC, but the rest of compilers will remain screwed at either build speed or dependency integrity.