[MDEV-25968] Parallel builds on CentOS 7 not possible, build runs out of memory and is unable to recover Created: 2021-06-19  Updated: 2022-07-19  Resolved: 2021-06-20

Status: Closed
Project: MariaDB Server
Component/s: Compiling
Affects Version/s: 10.6.3
Fix Version/s: N/A

Type: Bug Priority: Minor
Reporter: Otto Kekäläinen Assignee: Sergei Golubchik
Resolution: Won't Fix Votes: 0
Labels: None

Attachments: PNG File image-2021-06-19-13-39-53-172.png    

 Description   

While building MariaDB 10.6.3 I noticed that the build on CentOS 7 always fails due to out-of-memory at around 60% of the build when it is doing innobase and while running in parallel (because of `make -j`).

Running the build in parallel makes it very fast, so I'd like to continue doing it. Is there something that could be done to prevent the build failing?

If it is exhausting the memory, could it should just stop for a while and try again or something? Could the failure mode be smarter?

Full build log at https://gitlab.com/ottok/mariadb/-/jobs/1361008885

[ 65%] [ 66%] Creating mysqlslap link
 
Building CXX object storage/innobase/CMakeFiles/innobase.dir/log/log0log.cc.o
 
Building CXX object storage/innobase/CMakeFiles/innobase.dir/log/log0recv.cc.o
 
[ 66%] [ 66%] Built target mariadb-slap
 
[ 66%] Building CXX object storage/innobase/CMakeFiles/innobase.dir/log/log0crypt.cc.o
 
Building CXX object storage/innobase/CMakeFiles/innobase.dir/log/log0sync.cc.o
 
[ 66%] [ 66%] Building CXX object storage/innobase/CMakeFiles/innobase.dir/mem/mem0mem.cc.o
 
Building CXX object storage/innobase/CMakeFiles/innobase.dir/mtr/mtr0mtr.cc.o
 
[ 66%] [ 66%] [ 66%] Built target innodb_sync-t
 
Building CXX object storage/innobase/CMakeFiles/innobase.dir/os/os0file.cc.o
 
[ 66%] Building CXX object storage/innobase/CMakeFiles/innobase.dir/os/os0thread.cc.o
 
Building CXX object storage/innobase/CMakeFiles/innobase.dir/page/page0cur.cc.o
 
[ 66%] [ 66%] [ 66%] Building CXX object storage/innobase/CMakeFiles/innobase.dir/page/page0page.cc.o
 
Building CXX object storage/innobase/CMakeFiles/innobase.dir/pars/lexyy.cc.o
 
[ 66%] Building CXX object storage/innobase/CMakeFiles/innobase.dir/page/page0zip.cc.o
 
[ 66%] Building CXX object storage/innobase/CMakeFiles/innobase.dir/pars/pars0grm.cc.o
 
Building CXX object storage/innobase/CMakeFiles/innobase.dir/pars/pars0opt.cc.o
 
[ 67%] Building CXX object storage/innobase/CMakeFiles/innobase.dir/pars/pars0pars.cc.o
 
[ 67%] [ 67%] [ 67%] Building CXX object storage/innobase/CMakeFiles/innobase.dir/pars/pars0sym.cc.o
 
[ 67%] Building CXX object storage/innobase/CMakeFiles/innobase.dir/read/read0read.cc.o
 
Building CXX object storage/innobase/CMakeFiles/innobase.dir/que/que0que.cc.o
 
[ 67%] Building CXX object storage/innobase/CMakeFiles/innobase.dir/rem/rem0cmp.cc.o
 
[ 67%] [ 67%] [ 67%] Built target mbstream
 
[ 67%] c++: internal compiler error: Killed (program cc1plus)
 
Please submit a full bug report,
 
with preprocessed source if appropriate.
 
See <http://bugzilla.redhat.com/bugzilla> for instructions.
 
make[2]: *** [sql/CMakeFiles/partition.dir/ha_partition.cc.o] Error 4
 
make[1]: *** [sql/CMakeFiles/partition.dir/all] Error 2
 
make[1]: *** Waiting for unfinished jobs....
 
c++: internal compiler error: Killed (program cc1plus)
 
Please submit a full bug report,
 
with preprocessed source if appropriate.
 
See <http://bugzilla.redhat.com/bugzilla> for instructions.
 
make[2]: *** [storage/federatedx/CMakeFiles/federatedx.dir/ha_federatedx.cc.o] Error 4
 
make[1]: *** [storage/federatedx/CMakeFiles/federatedx.dir/all] Error 2
 
Building CXX object storage/innobase/CMakeFiles/innobase.dir/rem/rem0rec.cc.o
 
Building CXX object storage/innobase/CMakeFiles/innobase.dir/row/row0import.cc.o
 
Building CXX object storage/innobase/CMakeFiles/innobase.dir/row/row0ext.cc.o
 
c++: internal compiler error: Killed (program cc1plus)
 
c++: internal compiler error: Killed (program cc1plus)
 
[ 67%] [ 67%] [ 68%] Please submit a full bug report,
 
with preprocessed source if appropriate.
 
See <http://bugzilla.redhat.com/bugzilla> for instructions.
 
make[2]: *** [sql/CMakeFiles/sql_sequence.dir/ha_sequence.cc.o] Error 4
 
make[1]: *** [sql/CMakeFiles/sql_sequence.dir/all] Error 2
 
[ 68%] [ 68%] [ 68%] [ 68%] c++: internal compiler error: Killed (program cc1plus)
 
c++: internal compiler error: Killed (program cc1plus)
 
Please submit a full bug report,
 
with preprocessed source if appropriate.
 
See <http://bugzilla.redhat.com/bugzilla> for instructions.
 
make[2]: *** [storage/spider/CMakeFiles/spider.dir/spd_db_mysql.cc.o] Error 4
 
make[2]: *** Waiting for unfinished jobs....
 
[ 68%] Building CXX object storage/innobase/CMakeFiles/innobase.dir/row/row0mysql.cc.o
 
[ 68%] Please submit a full bug report,
 
with preprocessed source if appropriate.
 
See <http://bugzilla.redhat.com/bugzilla> for instructions.
 
make[2]: *** [sql/CMakeFiles/wsrep.dir/wsrep_dummy.cc.o] Error 4
 
make[1]: *** [sql/CMakeFiles/wsrep.dir/all] Error 2
 
c++: internal compiler error: Killed (program cc1plus)
 
Please submit a full bug report,
 
with preprocessed source if appropriate.
 
See <http://bugzilla.redhat.com/bugzilla> for instructions.
 
[ 69%] [ 69%] Building CXX object storage/innobase/CMakeFiles/innobase.dir/row/row0ftsort.cc.o
 
[ 69%] Building CXX object storage/innobase/CMakeFiles/innobase.dir/row/row0sel.cc.o
 
[ 69%] [ 69%] Building CXX object storage/innobase/CMakeFiles/innobase.dir/row/row0umod.cc.o
 
[ 69%] make[2]: *** [sql/CMakeFiles/thread_pool_info.dir/thread_pool_info.cc.o] Error 4
 
Building CXX object storage/innobase/CMakeFiles/innobase.dir/row/row0log.cc.o
 
Building CXX object storage/innobase/CMakeFiles/innobase.dir/row/row0ins.cc.o
 
make[1]: *** [sql/CMakeFiles/thread_pool_info.dir/all] Error 2
 
[ 69%] [ 69%] Building CXX object storage/innobase/CMakeFiles/innobase.dir/row/row0purge.cc.o
 
Please submit a full bug report,



 Comments   
Comment by Sergei Golubchik [ 2021-06-20 ]

I don't think it's something that can be "fixed" in cmake.
There are few things that you can do, though:

  • reduce the parallelism until compilation stops getting killed
  • linking needs more memory, I've had 1GB per executable (so with -j4 it's 4GB) — to workaround that do the compilation with higher parallelism and then linking with lower, say, compile with -j5 and link with -j2.
  • you can have a wrapper around make that calls it over and over until it succeeds or fails for not-OOM reason
Comment by Otto Kekäläinen [ 2021-06-30 ]

So ideally there was some magic to limit `cmake --parallel` or `make -j` or ninja so that they run max 1 CPU per 1 GB of RAM, or alternatively just issue a warning if memory seems too low to number of parallel runs.

I imagine a lot of people prefer to run with `--parallel` as that is how you typically compile any software, and without it builds are slow. We can't assume the information about memory limits would spread by itself, so we have to help it a little by giving some hints about this issue in the build output itself.

Generated at Thu Feb 08 09:41:45 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.