Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-34236

Mroonga build will hang during MariaDB build when using GCC 12

Details

    Description

      Original full hanging command during 11.5 opt UB+ASAN compilation:

      00:04:47 /usr/lib/gcc/x86_64-linux-gnu/12/cc1 -quiet -I /test/11.5_opt_san/storage/mroonga/vendor/groonga/lib -I /test/11.5_opt_san/storage/mroonga/vendor/groonga/include -I /test/11.5_opt_san/storage/mroonga/vendor/groonga -I /test/11.5_opt_san/wsrep-lib/include -I /test/11.5_opt_san/wsrep-lib/wsrep-API/v26 -I /test/11.5_opt_san/include -I /test/11.5_opt_san/include/providers -I /test/11.5_opt_san/storage/mroonga/vendor/groonga/lib/dat -imultiarch x86_64-linux-gnu -MD CMakeFiles/libgroonga.dir/expr.c.d -MF CMakeFiles/libgroonga.dir/expr.c.o.d -MT storage/mroonga/vendor/groonga/lib/CMakeFiles/libgroonga.dir/expr.c.o -D GRN_DAT_EXPORT -D GRN_EMBEDDED -D HAVE_CONFIG_H -D ONIG_EXTERN=extern -D USE_MYSYS_NEW -D _FILE_OFFSET_BITS=64 -D HAVE_ONIGMO_H -D MRB_INT64 -U _FORTIFY_SOURCE -U _FORTIFY_SOURCE -D WITH_UBSAN -D DBUG_OFF /test/11.5_opt_san/storage/mroonga/vendor/groonga/lib/expr.c -quiet -dumpdir CMakeFiles/libgroonga.dir/ -dumpbase expr.c.c -dumpbase-ext .c -mtune=generic -march=x86-64 -g -O3 -Wno-unused-but-set-variable -Wno-pointer-sign -Wformat=1 -Wstrict-aliasing=2 -Wno-disabled-optimization -Wfloat-equal -Wpointer-arith -Wbad-function-cast -Wcast-align -Wno-sign-compare -Wno-missing-field-initializers -Wimplicit-fallthrough=0 -Wno-uninitialized -Wall -Wdeclaration-after-statement -Wenum-compare -Wenum-conversion -Wextra -Wformat-security -Wmissing-braces -Wformat-truncation=0 -Wno-init-self -Wno-nonnull-compare -Wno-unused-parameter -Wvla -Wwrite-strings -std=gnu99 -std=gnu99 -fexceptions -fno-strict-aliasing -fsanitize=address -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=undefined -fno-sanitize=alignment -fno-omit-frame-pointer -fPIC -fasynchronous-unwind-tables -fstack-protector-strong -Wformat-security -fstack-clash-protection -fcf-protection -o /tmp/ccKRo6T9.s

      The (overly reduced) command:

      /usr/lib/gcc/x86_64-linux-gnu/12/cc1 -I /test/11.5_opt_san/storage/mroonga/vendor/groonga/lib -I /test/11.5_opt_san/storage/mroonga/vendor/groonga/include -I /test/11.5_opt_san/storage/mroonga/vendor/groonga -I /test/11.5_opt_san/wsrep-lib/include -I /test/11.5_opt_san/wsrep-lib/wsrep-API/v26 -I /test/11.5_opt_san/include -I /test/11.5_opt_san/include/providers -I /test/11.5_opt_san/storage/mroonga/vendor/groonga/lib/dat -imultiarch x86_64-linux-gnu -D HAVE_CONFIG_H -D WITH_UBSAN /test/11.5_opt_san/storage/mroonga/vendor/groonga/lib/expr.c -dumpdir CMakeFiles/libgroonga.dir/ -dumpbase expr.c.c -dumpbase-ext .c -g -O3 -std=gnu99 -fPIC -Wstrict-aliasing -o /tmp/ccKRo6T9.s

      Will show:

      ... functions are shown ...
      		 
      Analyzing compilation unit
      		 
      Performing interprocedural optimizations
      		 
       <*free_lang_data> {heap 15M} <visibility> {heap 15M} <build_ssa_passes> {heap 15M} <opt_local_passes> {heap 17M}

      Note the {heap 17M}. It will pause at this point for a few minutes, before continuing the output:

      ... previous output above ...
      		 
      <remove_symbols> {heap 119M} <targetclone> {heap 119M} <free-fnsummary> {heap 119M}
      		 
      Streaming LTO
      		 
       <whole-program> {heap 119M} <profile_estimate> {heap 119M} <icf> {heap 119M} <devirt> {heap 119M} <cp> {heap 119M} <sra> {heap 119M} <fnsummary> {heap 119M} <inline> {heap 119M} <pure-const> {heap 119M} <modref> {heap 119M} <free-fnsummary> {heap 119M} <static-var> {heap 119M} <single-use> {heap 119M} <comdats> {heap 119M}
      		 
      Assembling functions:
      		 
       <simdclone> {heap 119M} ... functions are shown ...

      The bit longer (and thus not overly reduced/actual working testcase):

      /usr/lib/gcc/x86_64-linux-gnu/12/cc1 -I /test/11.5_opt_san/storage/mroonga/vendor/groonga/lib -I /test/11.5_opt_san/storage/mroonga/vendor/groonga/include -I /test/11.5_opt_san/storage/mroonga/vendor/groonga -I /test/11.5_opt_san/wsrep-lib/include -I /test/11.5_opt_san/wsrep-lib/wsrep-API/v26 -I /test/11.5_opt_san/include -I /test/11.5_opt_san/include/providers -I /test/11.5_opt_san/storage/mroonga/vendor/groonga/lib/dat -imultiarch x86_64-linux-gnu -D HAVE_CONFIG_H -D WITH_UBSAN /test/11.5_opt_san/storage/mroonga/vendor/groonga/lib/expr.c -dumpdir CMakeFiles/libgroonga.dir/ -dumpbase expr.c.c -dumpbase-ext .c -g -O3 -std=gnu99 -fexceptions -fno-strict-aliasing -fsanitize=address -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=undefined -fno-sanitize=alignment -fno-omit-frame-pointer -fPIC -o /tmp/ccKRo6T9.s

      Will show similar output, however:

      ...
      		 
      Analyzing compilation unit
      		 
      Performing interprocedural optimizations
      		 
       <*free_lang_data> {heap 17M} <visibility> {heap 17M} <build_ssa_passes> {heap 17M} <opt_local_passes> {heap 22M}

      And then hang, with a small amount of single-thread CPU usage, presumable doing some compiler level optimzations or similar.

      Note the {heap 22M} (the original unreduced command is 23M). It thus seems there is some "too complex" operation going on that does not terminate.

      The time lapsed can be seen with: 

      $ echo $(( ($(date +%s) - $(stat -c %Y /tmp/ccKRo6T9.s)) / 60 )) minutes
      		 
      20 minutes  # Original command tested to hang 30+ min, and the orginal compilation of a single build had failed overnight (i.e. 8+ hours)

      As GCC 13 is the default on Ubuntu 23, here is the procedure for installing GCC 12.3.0 and making it default:

      sudo apt remove gcc-13 g++-13
      		 
      sudo apt update
      		 
      sudo apt install gcc-12 g++-12
      		 
      sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-12 10
      		 
      sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-12 10
      		 
      sudo update-alternatives --config gcc  # 0/auto
      		 
      sudo update-alternatives --config g++  # 0/auto
      		 
      gcc --version  # 12.3.0
      		 
      g++ --version  # 12.3.0

      Note that if you use -j >1 threads, the compilation may hang at different percentages. I have seen it as low as approx 25% up to 96% using 30 threads; use ps -ef | grep cc | grep mroonga once hang occurs.

      Verified on 10.5 opt and 11.5 opt.

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MDEV-31802 GCC 13: sql/sys_vars.inl: runtime error: store to address ... with insufficient space for an object of type ...

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Activity

            Ascending order - Click to sort in descending order
            Roel Roel Van de Paar added a comment - - edited

            This, combined with MDEV-31802 (on GCC 13), semi-blocks UB+ASAN testing on Ubuntu 23 where GCC 13 is the default and GCC 12 can be set with the above procedure, but is blocked by this Mroonga issue. I will next try GCC 11 and disabling Mroonga, though something hangs in my head about GCC 12 being the required minimum now.

            Update 1: GCC 11 (11.4.0) is able to build UB+ASAN builds correctly (with Mroonga left enabled).

            Update 2: Neither -DWITH_MROONGA=0 nor -WITH_MROONGA_STORAGE_ENGINE=OFF turns off the Mroonga engine compilation.

            Roel Roel Van de Paar added a comment - - edited This, combined with MDEV-31802 (on GCC 13), semi-blocks UB+ASAN testing on Ubuntu 23 where GCC 13 is the default and GCC 12 can be set with the above procedure, but is blocked by this Mroonga issue. I will next try GCC 11 and disabling Mroonga, though something hangs in my head about GCC 12 being the required minimum now. Update 1: GCC 11 (11.4.0) is able to build UB+ASAN builds correctly (with Mroonga left enabled). Update 2: Neither -DWITH_MROONGA=0 nor -WITH_MROONGA_STORAGE_ENGINE=OFF turns off the Mroonga engine compilation.
            Roel Roel Van de Paar added a comment - - edited

            Workaround for Ubuntu 23:

            sudo apt remove -y gcc-13 g++-13 gcc-12 g++-12 build-essential
            		 
            sudo apt update
            		 
            sudo apt install -y gcc-11 g++-11
            		 
            sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-11 10  # May give some warnings, which are auto-fixed
            		 
            sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-11 10  # Idem
            		 
            sudo update-alternatives --skip-auto --config gcc  # 0/auto
            		 
            sudo update-alternatives --skip-auto --config g++  # 0/auto
            		 
            gcc --version  # 11.4.0
            		 
            g++ --version  # 11.4.0

            Roel Roel Van de Paar added a comment - - edited Workaround for Ubuntu 23: sudo apt remove -y gcc-13 g++-13 gcc-12 g++-12 build-essential sudo apt update sudo apt install -y gcc-11 g++-11 sudo update-alternatives -- install /usr/bin/gcc gcc /usr/bin/gcc-11 10 # May give some warnings, which are auto-fixed sudo update-alternatives -- install /usr/bin/g ++ g++ /usr/bin/g ++-11 10 # Idem sudo update-alternatives --skip-auto --config gcc # 0/auto sudo update-alternatives --skip-auto --config g++ # 0/auto gcc --version # 11.4.0 g++ --version # 11.4.0
            Roel Roel Van de Paar added a comment -

            kou FYI

            Roel Roel Van de Paar added a comment - kou FYI
            kou Kouhei Sutou added a comment -

            Interesting. `expr.c` may be too complex for UB+ASAN with GCC 12/13.
            The complex part in `expr.c` were re-written in C++ in upstream: https://github.com/groonga/groonga/blob/main/lib/expr_executor.cpp
            It may change the situation. But we don't have implemented synchronize mechanism (merging upstream to MariaDB mechanism) yet... Sorry.
            Can we disable UB+ASAN only for `expr.c` for now?

            kou Kouhei Sutou added a comment - Interesting. `expr.c` may be too complex for UB+ASAN with GCC 12/13. The complex part in `expr.c` were re-written in C++ in upstream: https://github.com/groonga/groonga/blob/main/lib/expr_executor.cpp It may change the situation. But we don't have implemented synchronize mechanism (merging upstream to MariaDB mechanism) yet... Sorry. Can we disable UB+ASAN only for `expr.c` for now?
            serg Sergei Golubchik added a comment -

            We can try.
            I don't know if ASAN or UBSAN support the case when a binary is a mix of instrumented and not instrumented code. But I suspect it should work.

            serg Sergei Golubchik added a comment - We can try. I don't know if ASAN or UBSAN support the case when a binary is a mix of instrumented and not instrumented code. But I suspect it should work.
            Roel Roel Van de Paar added a comment -

            Great. Thank you, once more, wlad!

            Roel Roel Van de Paar added a comment - Great. Thank you, once more, wlad !

            People

              wlad Vladislav Vaintroub
              Roel Roel Van de Paar
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.