Managing a Test Suite

As these command lines get longer, it's a good idea to encapsulate them in presets. We can create presets for running specific test groups and for debugging.

Below, we update our default test preset to use 8 parallel jobs. We also add app-tests and lib-tests presets that inherits from default to apply filtering based on our labels:

1{
2  "version": 3,
3  "configurePresets": [{
4    "name": "default",
5    "hidden": true,
6    "binaryDir": "${sourceDir}/build",
7    "toolchainFile":
8      "$env{VCPKG_ROOT}/scripts/buildsystems/vcpkg.cmake"
9  }],
10  "buildPresets": [{
11    "name": "default",
12    "configurePreset": "default"
13  }],
14  "testPresets": [{
15    "name": "default",
16    "configurePreset": "default",
17    "output": { "outputOnFailure": true },
18    "execution": { "jobs": 8 }
19  }, {
20    "name": "app-tests",
21    "inherits": "default",
22    "filter": { "include": { "label": "application" } }
23  }, {
24    "name": "lib-tests",
25    "inherits": "default",
26    "filter": { "include": { "label": "library" } }
27  }]
28}

Now, we can easily run a subset of our tests from our project root:

1ctest --preset app-tests

1...
2100% tests passed, 0 tests failed out of 1

Our tests confirm that the code we've written works as expected. But they don't tell us what we've missed. Code coverage is a metric that answers the question: "which parts of my production code were executed by my test suite?"

How it Works

1. Instrumentation: You compile your code with special compiler flags that add "instrumentation". This extra code keeps track of which lines are executed and how many times.
2. Execution: You run your test suite. As the tests run, the instrumentation writes raw coverage data to files on disk.
3. Reporting: You use a separate tool to read these raw data files and generate a human-readable report (usually HTML) that visualizes the coverage.

Step 1: Setting Up CMake for Coverage

We need to add the instrumentation flags. For GCC and Clang, the flags are --coverage.

Since we only want this for a special coverage build, we'll create a new Coverage build type and use a generator expression to apply the flags.

We can implement on a target-by-target basis with commands like target_compile_options() and target_link_options() that we covered previously. Or, alternatively, we could implement it globally using add_compile_options() and add_link_options():

1cmake_minimum_required(VERSION 3.23)
2project(Greeter)
3
4# Add coverage flags for GCC/Clang in the "Coverage" build type
5set(COVERAGE_CONDITION
6  "$<AND:$<CONFIG:Coverage>,$<CXX_COMPILER_ID:GNU,Clang>>"
7)
8set(COVERAGE_FLAGS "$<${COVERAGE_CONDITION}:--coverage>")
9add_compile_options(${COVERAGE_FLAGS})
10add_link_options(${COVERAGE_FLAGS})
11
12add_subdirectory(app)
13add_subdirectory(greeter)
14
15enable_testing()
16add_subdirectory(tests)

And we'll add corresponding presets to our CMakePresets.json:

1{
2  "version": 3,
3  "configurePresets": [
4    // ...
5    {
6      "name": "coverage",
7      "displayName": "Coverage Build",
8      "inherits": "default",
9      "cacheVariables": { "CMAKE_BUILD_TYPE": "Coverage" }
10    }
11  ],
12  "buildPresets": [
13    // ...
14    {
15      "name": "coverage",
16      "configurePreset": "coverage"
17    }
18  ],
19  "testPresets": [
20    // ...
21    {
22      "name": "coverage",
23      "inherits": "default",
24      "configurePreset": "coverage"
25    }
26  ]
27}

Step 2: Generating Coverage Data

Now, we follow the usual three step process from our project root:

1cmake --preset=coverage

1cmake --build --preset=coverage

1ctest --preset=coverage

After running ctest, our build directory will be populated with .gcno and .gcda files containing the raw coverage data. On Unix-like systems, we can confirm our coverage data is being generated using a command like this:

1find build/ -name "*.gcda" -o -name "*.gcno"

1build/app/CMakeFiles/GreeterApp.dir/src/main.cpp.gcno
2build/app/CMakeFiles/GreeterAppLogic.dir/src/run.cpp.gcda
3build/app/CMakeFiles/GreeterAppLogic.dir/src/run.cpp.gcno
4...

We can get similar output using the dir command on PowerShell:

1dir build -Recurse -Include "*.gcda", "*.gcno"

To turn this raw data into a useful report, we need a tool. There are several options here, but a Python-based tool called gcovr is quite common.

gcovr is only available as a Python package, so we must have Python installed on our system. Then, we can install it using pip:

1pip install gcovr

Then, from our root directory, we can generate the report:

1gcovr -r . \
2  --filter 'greeter/' \
3  --filter 'app/' \
4  --html 'build/coverage.html'

• r .: Sets the root of your source tree to the current directory
• --filter: Sets what paths should be included in the report
• -html: Specifies the HTML output file

If we open our /build/coverage.html report in a web browser, we'll how much of our code the tests are covering:

In our case, the tests are hitting every line of code aside from our main.cpp, which only includes the main() entry point after our recent updates.

However, it's important to note that just because a line of code is being executed in the process of running a test, that doesn't necessarily mean that the code is being tested in any meaningful way.

Coverage reports just give us a relatively simple way to discover which parts of our code definitely aren't being tested, because they're not even being executed by our suite.

Branch Coverage

Whilst our tests execute every line of our library code, the coverage report will show that they don't execute every branch.

Even very simple statements can include hidden branching behavior, such as this line in our library:

1day_str = "Happy Monday!";

This is because the std::string's = operator can throw an exception if, for example, memory-related limitations prevent the assignment. This would result in a std::bad_alloc or std::length_error exception.

The branch coverage section of our report tells us that our tests did not create that scenario. In this case, the code in our library also isn't considering this possibility.

Our production could be updated with try-catch blocks to handle exceptions. Our test code could then create these exceptions by using the mocking techniques we covered previously.

We can use all the techniques we covered earlier, such as custom commands or custom targets to automate processes such as generating coverage reports.

For example, the following code creates a custom coverage target that lets us invoke gcovr without needing to provide the arguments:

1cmake_minimum_required(VERSION 3.23)
2project(Greeter)
3
4set(COVERAGE_CONDITION
5  "$<AND:$<CONFIG:Coverage>,$<CXX_COMPILER_ID:GNU,Clang>>"
6)
7set(COVERAGE_FLAGS "$<${COVERAGE_CONDITION}:--coverage>")
8add_compile_options(${COVERAGE_FLAGS})
9add_link_options(${COVERAGE_FLAGS})
10
11find_program(GCOVR_EXE gcovr)
12if(GCOVR_EXE)
13  add_custom_target(coverage
14    COMMAND ${GCOVR_EXE}
15      -r "${CMAKE_SOURCE_DIR}"
16      --filter "${CMAKE_SOURCE_DIR}/greeter/"
17      --filter "${CMAKE_SOURCE_DIR}/app/"
18      --html "coverage.html"
19    COMMENT "Generating code coverage report..."
20  )
21  message(STATUS "Added 'coverage' target")
22endif()
23
24add_subdirectory(app)
25add_subdirectory(greeter)
26
27enable_testing()
28add_subdirectory(tests)

Now, after running our tests in a coverage build, we can generate the report with a simple command from the build/ directory:

1cmake --build . --target coverage

Remember, we can (and should) keep our root CMakeLists.txt file as clean as possible by moving capabilities out to module files when things get too noisy:

This lesson improved our testing from writing individual checks to managing a complete test suite.

• Suite Management: We used CTest's filtering (-R, -E) and labeling (-L) capabilities for fine-grained control, and ran tests in parallel (-j) for speed. Presets allow us to encapsulate these complex commands into simple, named workflows.
• Code Coverage: We learned how to instrument our code with --coverage flags and use gcovr to generate reports that show where the gaps in our testing are.
• Automation: By creating custom targets for tasks like coverage analysis, we can integrate these workflows into our build process, and automate them.