build - 'cmake rebuild_cache' for *just* a subdirectory?

Question

I have an issue with the generation of makefiles stage of CMake being slow which is similar to this unanswered question:

CMake is slow to generate makefiles

My project is made up of a top level CMakeLists.txt file which uses add_subdirectory() to add various subprojects for individual library and executable components.

For a given component, the CMakeLists.txt file contains something like:

add_library(mylib SHARED
  sourceFile1.cpp
  sourceFile2.cpp
  ...
)

I can build just the contents of that directory using:

make mylib

If I modify the CMakeLists.txt file in the sub-directory (which I've been doing a lot as part of a migration from pure Makefiles to CMake) then run make it correctly re-runs CMake to update the configuration as if I'd run make rebuild_cache.

However, I notice that it in fact reconfigures the entire project. I really want for CMake to be clever enough to know it only needs to regenerate the Makefile in the current directory and sub-directories.

Is there a better way to structure a CMake project to achieve this? I see some people use project() for each CMakeLists.txt in each sub-project. In general, is this a good idea?

Alternatively/additionally is there some way to speed up the generation step of CMake? (currently I have 60s+)

Bonus points if you want to discuss why CMake itself should or shouldn't be able to run in parallel (imagine a cmake -j).

---

I've added the meson-build tag as a modest bounty, but alone it hasn't yet attracted enough attention to warrant an answer. It's this kind of problem that might cause people to switch to build systems to meson-build (assuming it doesn't have similar problems) or something similar.

It is possible that the correct answer is it can't be done without modifying the source to CMake. To earn the bounty though I require an explanation in terms of how CMake works and/or where it is flawed.

---

Clarification: It is the generation step that is slow in my case. The configure itself is quick enough, but CMake hangs for a quite a while between outputting "-- Configuring done" and "-- Generating done".

For a full cache rebuild I run:

make -n rebuild_cache

Running CMake to regenerate build system... 
using Makefile generator
-- FOOBAR_VERSION: 01.02.03
-- Configuring done
-- Generating done
-- Build files have been written to: /home/brucea/work/depot/emma/main/cmake
real 74.87
user 1.74
sys 1.02

Under the hood this runs:

cmake -H<source dir> -B<build dir>

I presume -B is a synonym for --build. Neither option is described correctly in the documentation. -H is the root of the source directory (not the same as --help as the documentation would have you believe).

It's fast to get to the output of "Configuring done", but slow from there:

For example,

15:44:14 execve("/usr/local/bin/cmake",
>grep Generating cmake_strace.log
>grep "Configuring" cmake_strace.log
15:44:15 write(1, "-- Configuring done
", 20-- Configuring done
15:45:01 write(1, "-- Generating done
", 19-- Generating done
>grep "Build files" cmake_strace.log
15:45:22 write(1, "-- Build files have been written"..., 77-- Build files have been written to:

If editing a single CMakeLists.txt file in a subdirectory, and then running make -n, it runs:

cd /home/project/cmake && /usr/local/bin/cmake -H/home/project/cmake -B/home/project/cmake --check-build-system CMakeFiles/Makefile.cmake 0

--check-build-system is another undocumented option.

The effect is the same - regenerate the whole build system, not just the current subtree. There is no difference in behaviour between an in-source and an out-of-source build.

If I run a trace, e.g.:

strace -r cmake --trace -H/home/project/cmake -B/home/project/cmake 2>&1 | tee cmake_rebuild_cache.log
sort -r cmake_rebuild_cache.log | uniq

The majority of time spent seems to be spent on (or between) open, access & unlink calls.

The length of each task is quite variable, but the huge number of them builds up. I have no idea what the Labels.json and Labels.txt files are about (something internal to CMake).

One run:

    49.363537 open("/home/projectbar/main/test/foo2bar/CMakeFiles/test2.foo2bar.testViewingSource1.dir/build.make", O_RDONLY) = 5
     1.324777 access("/home/projectbar/main/test/performance/CMakeFiles/performancetest.chvcthulhu.testChvcthulhuPerformance2.dir", R_OK) = 0
     0.907807 access("/home/projectbar/main/test/foo2bar/CMakeFiles/test2.foo2bar.testPeripheralConnection2.dir", R_OK) = 0
     0.670272 unlink("/home/projectbar/main/src/foo2bar/Foo2Bar/CMakeFiles/foo2bar_lib.dir/progress.make.tmp") = -1 ENOENT (No such file or directory)
     0.600272 access("/home/projectbar/main/test/foo2bar/testFilesModel2.ok", R_OK) = 0
     0.599010 access("/home/projectbar/main/test/hve2snafu/testInvalidByte2c.ok", R_OK) = 0
     0.582466 read(5, "openjdk version "1.8.0_71"
OpenJ"..., 1024) = 130
     0.570540 writev(3, [{"# CMAKE generated file: DO NOT E"..., 8190}, {"M", 1}], 2) = 8191
     0.553576 close(4)                  = 0
     0.448811 unlink("/home/projectbar/main/test/snafu2hve/CMakeFiles/test2.snafu2hve.testNoProbes2.dir/progress.make.tmp") = -1 ENOENT (No such file or directory)
     0.431559 access("/home/projectbar/main/src/foo2bar/Foo2Bar/CMakeFiles/foo2bar_lib.dir", R_OK) = 0
     0.408003 unlink("/home/projectbar/main/test/lachesis/CMakeFiles/test2.lachesis.testBadSequenceNumber1.dir/progress.make.tmp") = -1 ENOENT (No such file or directory)
     0.407120 write(4, "# The set of languages for which"..., 566) = 566
     0.406674 write(3, "# CMAKE generated file: DO NOT E"..., 675) = 675
     0.383892 read(3, "ewingPeriod.cpp.o -c /home/bruce"..., 8191) = 8191
     0.358490 unlink("/home/projectbar/main/cmake/CMakeFiles/mklinks.chvdiff.dir/progress.make.tmp") = -1 ENOENT (No such file or directory)

Another run of the same command:

     2.009451 unlink("/home/projectbar/main/cmake/CMakeFiles/mklinks.lachesis.dir/Labels.json") = -1 ENOENT (No such file or directory)
) = 20
) = 19
     1.300387 access("/home/projectbar/main/test/chvedit/CMakeFiles/test2.chvedit.tefooultiMatchFactoringEdit2.dir", R_OK) = 0
     1.067957 access("/home/projectbar/main/test/chvedit/CMakeFiles/test2.chvedit.tefooultiMatchFactoringEdit2.dir", R_OK) = 0
)         = 1
     0.885854 unlink("/home/projectbar/main/src/gorkyorks2bar/CMakeFiles/doxygen.correct.gorkyorks2bar.dir/Labels.json") = -1 ENOENT (No such file or directory)
     0.854539 access("/home/projectbar/main/test/reportImpressions/ReportImpressions/CMakeFiles/testsuite1_reportImpressions.dir", R_OK) = 0
     0.791741 unlink("/home/projectbar/main/cmake/CMakeFiles/mklinks.bar_models.dir/progress.make.tmp") = -1 ENOENT (No such file or directory)
     0.659506 unlink("/home/projectbar/main/cmake/CMakeFiles/mklinks.dir/progress.make.tmp") = -1 ENOENT (No such file or directory)
     0.647838 unlink("/home/projectbar/main/test/libyar/YarModels/CMakeFiles/testsuite1_yarmodels.dir/Labels.txt") = -1 ENOENT (No such file or directory)
     0.620511 unlink("/home/projectbar/main/test/libyar/YarModels/CMakeFiles/testsuite1_yarmodels.dir/Labels.json") = -1 ENOENT (No such file or directory)
     0.601942 unlink("/home/projectbar/main/cmake/CMakeFiles/mklinks.lachesis.dir/Labels.txt") = -1 ENOENT (No such file or directory)
     0.591871 access("/home/projectbar/main/src/runbardemo/simple_demo/CMakeFiles", R_OK) = 0
     0.582448 write(3, "CMAKE_PROGRESS_1 = 

", 21) = 21
     0.536947 write(3, "CMAKE_PROGRESS_1 = 

", 21) = 21
     0.499758 unlink("/home/projectbar/main/test/foo2bar/CMakeFiles/test2.foo2bar.testInputDirectory1.dir/progress.make.tmp") = -1 ENOENT (No such file or directory)
     0.458120 unlink("/home/projectbar/main/test/yak2dcs/CMakeFiles/test2.yak2dcs.testsuite2.dir/progress.make.tmp") = -1 ENOENT (No such file or directory)
     0.448104 unlink("/home/projectbar/main/test/reportImpressions/CMakeFiles/test2.reportImpressions.dir/progress.make.tmp") = -1 ENOENT (No such file or directory)
     0.444344 access("/home/projectbar/main/src/bananas/CMakeFiles/bin.bananas.dir", R_OK) = 0
     0.442685 unlink("/home/projectbar/main/test/rvedit/CMakeFiles/test2.rvedit.tefooissingOptionValue.dir/progress.make.tmp") = -1 ENOENT (No such file or directory)
     0.425604 unlink("/home/projectbar/main/test/listdcs/CMakeFiles/test2.listdcs.testListCalls5.dir/progress.make.tmp") = -1 ENOENT (No such file or directory)
     0.391163 access("/home/projectbar/main/src/siedit/CMakeFiles/siedit.dir", R_OK) = 0
     0.362171 access("/home/projectbar/main/test/foo2bar/CMakeFiles/test2.foo2emma.testHowResults6.dir", R_OK) = 0

Note the Ninja generator is much faster (though still not brilliant). For example,

/usr/bin/time -p ninja rebuild_cache

ninja: warning: multiple rules generate ../src/ams2yar/ams2yar. builds involving this target will not be correct; continuing anyway [-w dupbuild=warn]
ninja: warning: multiple rules generate ../src/vox/vox. builds involving this target will not be correct; continuing anyway [-w dupbuild=warn]
ninja: warning: multiple rules generate ../src/bananas/bananas. builds involving this target will not be correct; continuing anyway [-w dupbuild=warn]
ninja: warning: multiple rules generate ../src/fidlertypes2fidlerinfo/fidlertypes2fidlerinfo. builds involving this target will not be correct; continuing anyway [-w dupbuild=warn]
ninja: warning: multiple rules generate ../src/mkrundir/mkrundir. builds involving this target will not be correct; continuing anyway [-w dupbuild=warn]
ninja: warning: multiple rules generate ../src/runyar/runyar. builds involving this target will not be correct; continuing anyway [-w dupbuild=warn]
ninja: warning: multiple rules generate ../src/runyardemo/runyardemo. builds involving this target will not be correct; continuing anyway [-w dupbuild=warn]
[1/1] Running CMake to regenerate build system...
Generator=Ninja
-- FOO_VERSION: 01.02.03
-- Configuring done
-- Generating done
-- Build files have been written to: /home/project/cmake/build
real 12.67
user 1.01
sys 0.31

Note that the project is not quite ready for Ninja yet as there are errors like:

ninja: warning: multiple rules generate ../src/runfoobardemo/runfoobardemo. builds involving this target will not be correct; continuing anyway [-w dupbuild=warn]

and

ninja: error: dependency cycle: ../src/foobar -> ../src/foobar/CMakeFiles/foobar -> ../src/ams2emma/foobar

to be resolved. This question is really about why the Makefile generator is slow. I'm not sure if the problems Ninja shows are useful hints here or red herrings.

---

Building CMake with more optimisations does not help.

Based on my trace output it and the output of time, it is unlikely that it would. The user time and therefore time spend within the CMake code itself is quite low. (see e.g. <a href="https://stackoverflow.com/questions/556405/what-do-real-use

Answer 1

There are so many aspects that define CMake's configuration and generation steps duration (besides what you actually do in your CMakeLists.txt files; it's e.g. your host system, your toolchain and which CMake version/distribution you are using).

So I try to concentrate on the specific questions you have.

Rebuild/rewrite makefiles for just a subdirectory?

For the start: Using add_subdirectory() is good for structuring your CMake code. But you have to keep in mind that you can always change global CMake properties in a subdirectory and that targets inside those subdirectories can have cross-dependencies.

So what does CMake do (considering the "I have touched one CMakeLists.txt file in a subdirectory" case discussed here):

• If a CMakeLists.txt file is changed it goes through the complete hierarchy of CMakeLists.txt files again and rebuilds the build environment again in memory.
• Now it temporarily recreates all the necessary build/make files and checks if they defer from the existing ones (see cmGeneratedFileStreamBase::Close()).
• If a file has changed it replaces the existing one with the new one.

This behavior is necessary because any makefile can change even when only a subdirectory's CMakeLists.txt file has changed and it was optimized to prevent unnecessary rebuilds during the actual make step (from touched makefiles).

Is there some way to speed up the generation step of CMake?

So yes, it does temporarily rewrite all makefiles (which could be slow) and no, you can't minimize this while using add_subdirectory() to only the changed subdirectory.

Maybe one possible performance optimization for the future in CMake's own code would be to use memorystreams instead of filestreams for the temporary files.

@BruceAdams tested this by using a RAM disk for the generated makefile environment with no effect.

And yes, the CMake generated cmake_check_build_system rule does almost the same as the rebuild_cache rule and yes the used -B, -H and --check-build-system options are CMake internal command line options and therefore undocumented (even if often referred to on Stack Overflow, e.g. in one of my answers here).

What helped me to speed-up the configuration/generation was to rebuild CMake itself with a lot more optimization options than the normal distributions and using a 64-bit toolchain instead of the 32-bit versions currently still distributed.

Here are some test results (using the CMake test script found below with 100 subdirectories/libraries) on my Windows PC always using the same MSYS environment, but different CMake compilations of the same CMake source code:

1. Official CMake 3.2.2 version:

   $ time -p cmake -G "MSYS Makefiles" ..
   [...]
   real 43.93
   user 0.00
   sys 0.03
   

2. Using mingw32 and GNU 4.8.1 I rebuild CMake 3.2.2 with

   cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_FLAGS="-O3" -G "MSYS Makefiles" ..
   

   and got

   $ time -p /c/temp/cmake-3.2.2/MSYS32/bin/cmake.exe -G "MSYS Makefiles" ..
   [...]
   real 41.37
   user 0.01
   sys 0.04
   

   And the same with my antivirus software turned off:

   $ time -p /c/temp/cmake-3.2.2/MSYS32/bin/cmake.exe -G "MSYS Makefiles" ..
   [...]
   real 20.98
   user 0.00
   sys 0.04
   

3. Using mingw-w64 and GNU 5.3.0 I rebuild CMake 3.2.2 with

   $ cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_FLAGS="-march=native -m64 -Ofast  -flto" -G "MSYS Makefiles" ..
   

   and got

   $ time -p /c/temp/cmake-3.2.2/MSYS64/bin/cmake.exe -G "MSYS Makefiles" ..
   [...]
   real 25.59
   user 0.00
   sys 0.04
   

   And the same with my antivirus software turned off:

   $ time -p /c/temp/cmake-3.2.2/MSYS64/bin/cmake.exe -G "MSYS Makefiles" ..
   [...]
   real 6.95
   user 0.00
   sys 0.03
   

To summarize I see two main influences:

1st: The configuration step can be speed-up by going for a 64-bit version and optimize for your processor platform (you would certainly have to find a common base -march=... or -mtune=... for all your project's build PCs).

2nd: The generation step can mostly be sped up by searching for possible file I/O bottlenecks outside of CMake. In my case telling the antivirus software to not check the toolchain and build directories each time I read/write to those was really speeding up things.

Remark: I confirmed @BruceAdams test results that the the compiler's auto-vectorization (default for -O3 or -Ofast) is not able to do much about CMake source code's ability to run in multiple processes/on multiple cores.

Is there a better way to structure a CMake project to achieving this?

Yes, if you e.g. know that a certain sub-tree of your CMake script code just generates a library and has no dependencies, you could put that part in an external project by using ExternalProject_Add(). And yes, having had similar concerns regarding large CMake projects, this is seen as a good "modern CMake" practice (see also references below).

References

• CMake Performance Tips
• CMake: Parallel generation of make files
• GCC: how is march different from mtune?
• CMake: How to set up source, library and CMakeLists.txt dependencies?
• Making CMake library accessible by other CMake packages automatically
• Use CMake-enabled libraries in your CMake project

What I used to reproduce your problem

Just for completeness and if someone wants to check those numbers against his/her own, here is my test code:

cmake_minimum_required(VERSION 3.0)

project(CMakeTest CXX)

#set_property(GLOBAL PROPERTY GLOBAL_DEPENDS_DEBUG_MODE 1)

set(_idx 1)

while (_idx LESS 100)
    math(EXPR _next_idx "${_idx} + 1")
    if (NOT EXISTS "${CMAKE_CURRENT_SOURCE_DIR}/lib${_idx}")
        file(MAKE_DIRECTORY "lib${_idx}")
        file(
            WRITE "lib${_idx}/lib${_idx}.h"
                "int lib${_idx}_func();"
        )
        file(
            WRITE "lib${_idx}/lib${_idx}.cc"
                "#include "lib${_next_idx}.h"
"
                "int lib${_idx}_func() { return lib${_next_idx}_func(); }"
        )
        file(
            WRITE "lib${_idx}/CMakeLists.txt"
                "add_library(lib${_idx} "lib${_idx}.cc")
"
                "target_link_libraries(lib${_idx} lib${_next_idx})
"
                "target_include_directories(lib${_idx} PUBLIC ".")"
        )
    endif()
    add_subdirectory("lib${_idx}")
    set(_idx "${_next_idx}")
endwhile()

if (NOT EXISTS "${CMAKE_CURRENT_SOURCE_DIR}/lib${_idx}")
    file(MAKE_DIRECTORY "lib${_idx}")
    file(
        WRITE "lib${_idx}/lib${_idx}.h"
            "int lib${_idx}_func();"
    )
    file(
        WRITE "lib${_idx}/lib${_idx}.cc"
            "int lib${_idx}_func() { return 0; }"
    )
    file(
        WRITE "lib${_idx}/CMakeLists.txt"
            "add_library(lib${_idx} "lib${_idx}.cc")
"
            "target_include_directories(lib${_idx} PUBLIC ".")"
    )
endif()
add_subdirectory("lib${_idx}")

if (NOT EXISTS "${CMAKE_CURRENT_SOURCE_DIR}/main.cc")
    file(
        WRITE "main.cc"
            "#include "lib1.h"
"
            "int main() { return lib1_func(); }"
    )
endif()

add_executable(${PROJECT_NAME} "main.cc")
target_link_libraries(${PROJECT_NAME} lib1)

And then - after the first cmake .. and make calls - doing:

$ touch ../lib100/CMakeLists.txt
$ time -p cmake ..
-- Configuring done
-- Generating done
-- Build files have been written to: [your path here]
real 28.89
user 0.01
sys 0.04