Details
-
Bug
-
Status: Closed (View Workflow)
-
Critical
-
Resolution: Fixed
-
10.5, 11.5(EOL)
Description
Original full hanging command during 11.5 opt UB+ASAN compilation:
00:04:47 /usr/lib/gcc/x86_64-linux-gnu/12/cc1 -quiet -I /test/11.5_opt_san/storage/mroonga/vendor/groonga/lib -I /test/11.5_opt_san/storage/mroonga/vendor/groonga/include -I /test/11.5_opt_san/storage/mroonga/vendor/groonga -I /test/11.5_opt_san/wsrep-lib/include -I /test/11.5_opt_san/wsrep-lib/wsrep-API/v26 -I /test/11.5_opt_san/include -I /test/11.5_opt_san/include/providers -I /test/11.5_opt_san/storage/mroonga/vendor/groonga/lib/dat -imultiarch x86_64-linux-gnu -MD CMakeFiles/libgroonga.dir/expr.c.d -MF CMakeFiles/libgroonga.dir/expr.c.o.d -MT storage/mroonga/vendor/groonga/lib/CMakeFiles/libgroonga.dir/expr.c.o -D GRN_DAT_EXPORT -D GRN_EMBEDDED -D HAVE_CONFIG_H -D ONIG_EXTERN=extern -D USE_MYSYS_NEW -D _FILE_OFFSET_BITS=64 -D HAVE_ONIGMO_H -D MRB_INT64 -U _FORTIFY_SOURCE -U _FORTIFY_SOURCE -D WITH_UBSAN -D DBUG_OFF /test/11.5_opt_san/storage/mroonga/vendor/groonga/lib/expr.c -quiet -dumpdir CMakeFiles/libgroonga.dir/ -dumpbase expr.c.c -dumpbase-ext .c -mtune=generic -march=x86-64 -g -O3 -Wno-unused-but-set-variable -Wno-pointer-sign -Wformat=1 -Wstrict-aliasing=2 -Wno-disabled-optimization -Wfloat-equal -Wpointer-arith -Wbad-function-cast -Wcast-align -Wno-sign-compare -Wno-missing-field-initializers -Wimplicit-fallthrough=0 -Wno-uninitialized -Wall -Wdeclaration-after-statement -Wenum-compare -Wenum-conversion -Wextra -Wformat-security -Wmissing-braces -Wformat-truncation=0 -Wno-init-self -Wno-nonnull-compare -Wno-unused-parameter -Wvla -Wwrite-strings -std=gnu99 -std=gnu99 -fexceptions -fno-strict-aliasing -fsanitize=address -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=undefined -fno-sanitize=alignment -fno-omit-frame-pointer -fPIC -fasynchronous-unwind-tables -fstack-protector-strong -Wformat-security -fstack-clash-protection -fcf-protection -o /tmp/ccKRo6T9.s
|
The (overly reduced) command:
/usr/lib/gcc/x86_64-linux-gnu/12/cc1 -I /test/11.5_opt_san/storage/mroonga/vendor/groonga/lib -I /test/11.5_opt_san/storage/mroonga/vendor/groonga/include -I /test/11.5_opt_san/storage/mroonga/vendor/groonga -I /test/11.5_opt_san/wsrep-lib/include -I /test/11.5_opt_san/wsrep-lib/wsrep-API/v26 -I /test/11.5_opt_san/include -I /test/11.5_opt_san/include/providers -I /test/11.5_opt_san/storage/mroonga/vendor/groonga/lib/dat -imultiarch x86_64-linux-gnu -D HAVE_CONFIG_H -D WITH_UBSAN /test/11.5_opt_san/storage/mroonga/vendor/groonga/lib/expr.c -dumpdir CMakeFiles/libgroonga.dir/ -dumpbase expr.c.c -dumpbase-ext .c -g -O3 -std=gnu99 -fPIC -Wstrict-aliasing -o /tmp/ccKRo6T9.s
|
Will show:
... functions are shown ...
|
Analyzing compilation unit
|
Performing interprocedural optimizations
|
<*free_lang_data> {heap 15M} <visibility> {heap 15M} <build_ssa_passes> {heap 15M} <opt_local_passes> {heap 17M}
|
Note the {heap 17M}. It will pause at this point for a few minutes, before continuing the output:
... previous output above ...
|
<remove_symbols> {heap 119M} <targetclone> {heap 119M} <free-fnsummary> {heap 119M}
|
Streaming LTO
|
<whole-program> {heap 119M} <profile_estimate> {heap 119M} <icf> {heap 119M} <devirt> {heap 119M} <cp> {heap 119M} <sra> {heap 119M} <fnsummary> {heap 119M} <inline> {heap 119M} <pure-const> {heap 119M} <modref> {heap 119M} <free-fnsummary> {heap 119M} <static-var> {heap 119M} <single-use> {heap 119M} <comdats> {heap 119M}
|
Assembling functions:
|
<simdclone> {heap 119M} ... functions are shown ...
|
The bit longer (and thus not overly reduced/actual working testcase):
/usr/lib/gcc/x86_64-linux-gnu/12/cc1 -I /test/11.5_opt_san/storage/mroonga/vendor/groonga/lib -I /test/11.5_opt_san/storage/mroonga/vendor/groonga/include -I /test/11.5_opt_san/storage/mroonga/vendor/groonga -I /test/11.5_opt_san/wsrep-lib/include -I /test/11.5_opt_san/wsrep-lib/wsrep-API/v26 -I /test/11.5_opt_san/include -I /test/11.5_opt_san/include/providers -I /test/11.5_opt_san/storage/mroonga/vendor/groonga/lib/dat -imultiarch x86_64-linux-gnu -D HAVE_CONFIG_H -D WITH_UBSAN /test/11.5_opt_san/storage/mroonga/vendor/groonga/lib/expr.c -dumpdir CMakeFiles/libgroonga.dir/ -dumpbase expr.c.c -dumpbase-ext .c -g -O3 -std=gnu99 -fexceptions -fno-strict-aliasing -fsanitize=address -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=undefined -fno-sanitize=alignment -fno-omit-frame-pointer -fPIC -o /tmp/ccKRo6T9.s
|
Will show similar output, however:
...
|
Analyzing compilation unit
|
Performing interprocedural optimizations
|
<*free_lang_data> {heap 17M} <visibility> {heap 17M} <build_ssa_passes> {heap 17M} <opt_local_passes> {heap 22M}
|
And then hang, with a small amount of single-thread CPU usage, presumable doing some compiler level optimzations or similar.
Note the {heap 22M} (the original unreduced command is 23M). It thus seems there is some "too complex" operation going on that does not terminate.
The time lapsed can be seen with:
$ echo $(( ($(date +%s) - $(stat -c %Y /tmp/ccKRo6T9.s)) / 60 )) minutes |
20 minutes # Original command tested to hang 30+ min, and the orginal compilation of a single build had failed overnight (i.e. 8+ hours) |
As GCC 13 is the default on Ubuntu 23, here is the procedure for installing GCC 12.3.0 and making it default:
sudo apt remove gcc-13 g++-13 |
sudo apt update |
sudo apt install gcc-12 g++-12 |
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-12 10 |
sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-12 10 |
sudo update-alternatives --config gcc # 0/auto |
sudo update-alternatives --config g++ # 0/auto |
gcc --version # 12.3.0 |
g++ --version # 12.3.0 |
Note that if you use -j >1 threads, the compilation may hang at different percentages. I have seen it as low as approx 25% up to 96% using 30 threads; use ps -ef | grep cc | grep mroonga once hang occurs.
Verified on 10.5 opt and 11.5 opt.
Attachments
Issue Links
- relates to
-
-
- Closed
-
Activity
Workaround for Ubuntu 23:
sudo apt remove -y gcc-13 g++-13 gcc-12 g++-12 build-essential |
sudo apt update |
sudo apt install -y gcc-11 g++-11 |
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-11 10 # May give some warnings, which are auto-fixed |
sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-11 10 # Idem |
sudo update-alternatives --skip-auto --config gcc # 0/auto |
sudo update-alternatives --skip-auto --config g++ # 0/auto |
gcc --version # 11.4.0 |
g++ --version # 11.4.0 |
Interesting. `expr.c` may be too complex for UB+ASAN with GCC 12/13.
The complex part in `expr.c` were re-written in C++ in upstream: https://github.com/groonga/groonga/blob/main/lib/expr_executor.cpp
It may change the situation. But we don't have implemented synchronize mechanism (merging upstream to MariaDB mechanism) yet... Sorry.
Can we disable UB+ASAN only for `expr.c` for now?
We can try.
I don't know if ASAN or UBSAN support the case when a binary is a mix of instrumented and not instrumented code. But I suspect it should work.
This, combined with
MDEV-31802(on GCC 13),semi-blocks UB+ASAN testingon Ubuntu 23 where GCC 13 is the default and GCC 12 can be set with the above procedure, but is blocked by this Mroonga issue. I will next try GCC 11 and disabling Mroonga, though something hangs in my head about GCC 12 being the required minimum now.Update 1: GCC 11 (11.4.0) is able to build UB+ASAN builds correctly (with Mroonga left enabled).
Update 2: Neither -DWITH_MROONGA=0 nor -WITH_MROONGA_STORAGE_ENGINE=OFF turns off the Mroonga engine compilation.