We can now format our code and generate documentation with single commands, but an important question remains: how do we know our code actually works? And how do we ensure it keeps working as we refactor, optimize, and add new features?

The answer is automated testing. In this chapter, we'll integrate a testing workflow into our project. We'll start with the fundamentals, using CMake's built-in testing framework, CTest, to create and run our first simple tests.

The Benefits of Automated Testing

Before we write any test code, it's important to understand the value it provides. Automated tests have many benefits:

A Safety Net Against Regressions: A regression is when a change to the code unintentionally breaks existing functionality. A test suite acts as a safety net. After every change, you can run your tests to get immediate feedback. If the tests are comprehensive and they all pass, you can be more confident you haven't broken anything.
Confidence to Refactor: Code often needs to be refactored - restructured and improved without changing its external behavior. Without tests, refactoring is a high-risk activity. With a good test suite, you can make bold changes, knowing that you can instantly verify that the new implementation behaves identically to the old one.
Executable Documentation: Tests are a form of documentation that cannot go out of date. A well-written test clearly demonstrates how a piece of code is intended to be used and what its expected behavior is, including for edge cases that might be missed in traditional documentation.

Throughout this chapter, we'll use our Greeter project as a practical testbed to learn how to implement these ideas with CMake and CTest.

Creating a Test Sub-Project

Just as we separate our library (greeter/) from our application (app/), it's a standard practice to keep test code separate from production code. We'll create a new tests/ directory for this purpose.

1Greeter/
2├─ app/
3├─ greeter/
4├─ tests/                 
5│  ├─ CMakeLists.txt     
6│  └─ main.cpp           
7└─ CMakeLists.txt

We'll cover what goes in these new files soon. To integrate this new component, we need to make two changes to our existing root CMakeLists.txt:

Add add_subdirectory(tests) to tell CMake to process the new CMakeLists.txt file inside the tests/ directory.
Add the enable_testing() command. This is the master switch that activates all of CMake's testing features, including the add_test() command we'll use shortly and the CTest framework itself.

CMakeLists.txt

1cmake_minimum_required(VERSION 3.23)
2project(Greeter)
3
4add_subdirectory(greeter)
5add_subdirectory(app)
6
7enable_testing() 
8add_subdirectory(tests)

Writing a Test Driver

Our first test will verify that the get_greeting() function in our GreeterLib returns the correct string. To do this, we need to create a simple C++ program whose only job is to run this test. This is often called a test driver or a test runner.

First, let's define the build target for this executable in tests/CMakeLists.txt. We'll call the target GreeterTests and link it against our GreeterLib so it can access the get_greeting() function.

tests/CMakeLists.txt

1cmake_minimum_required(VERSION 3.23)
2
3add_executable(GreeterTests main.cpp)
4
5# Our test needs access to the library it's testing
6target_link_libraries(GreeterTests PRIVATE GreeterLib)

Next, let's write the C++ code for our test driver in tests/main.cpp. This is quite a lot of code to create a simple test but, in the next lesson, we'll introduce an external testing library that makes this easier.

For now, our test file might be verbose, but the logic is simple:

tests/main.cpp

1#include <greeter/Greeter.h>
2#include <string>
3#include <iostream>
4
5int main() {
6  const std::string expected{
7    "Hello from the modular greeter library"
8  };
9  const std::string actual{get_greeting()};
10
11  if (actual != expected) {
12    std::cerr << "Test Failed:"
13              << "\n  Expected: " << expected
14              << "\n  Actual:   " << actual;
15    return EXIT_FAILURE;
16  }
17
18  std::cout << "Test Passed!";
19  return EXIT_SUCCESS;
20}

We call the function we want to test, and compare the actual result to the expected result.

If they don't match, print an error and return a non-zero exit code from our main function. Something like return 1 would work, but return EXIT_FAILURE gives us similar behavior with more descriptive syntax.

If they do match, print a success message and return 0 (or EXIT_SUCCESS).

This convention of using the program's exit code to signal success or failure is fundamental to how CTest and most command-line testing tools work.

For reference, here is the function we're testing. It's important that the string we're returning here exactly matches the expected value in the test:

greeter/src/greeter.cpp

1#include <greeter/Greeter.h>
2#include <string>
3
4std::string get_greeting() {
5  return "Hello from the modular greeter library"
6}

Registering Tests with `add_test()`

We've created a test executable, but CMake's testing framework doesn't know about it yet. We need to formally register it as a test case. We do this with the add_test() command.

tests/CMakeLists.txt

1cmake_minimum_required(VERSION 3.23)
2
3add_executable(GreeterTests main.cpp)
4target_link_libraries(GreeterTests PRIVATE GreeterLib)
5
6add_test(
7  NAME Greeter.API.get_greeting
8  COMMAND GreeterTests
9)

Let's break down this command:

NAME Greeter.API.get_greeting: This gives our test a unique, human-readable name for reporting purposes. This is the name you'll see in the CTest output. It's a good practice to use a hierarchical naming scheme (e.g., ComponentName.Subsystem.TestName) to keep your tests organized.
COMMAND GreeterTests: This specifies the command to run for this test. In this case, it's just our GreeterTests executable target.

Running Tests with `ctest`

With our test registered, we can now run it. From our build directory, we should first configure and build the project in the usual way:

1cmake ..

1cmake --build .

We can then run the ctest command, also from our build directory:

1ctest

CTest will find all the tests registered with add_test() and run them, providing a summary of the results:

1Test project D:/Projects/cmake/build
2    Start 1: Greeter.API.get_greeting
31/1 Test #1: Greeter.API.get_greeting ... Passed  0.01 sec
4
5100% tests passed, 0 tests failed out of 1
6
7Total Test time (real) =   0.01 sec

Creating Test Presets

As our testing requirements get more complex, our ctest commands will also get longer, and inputting them will become a chore. Just as we did for configuring and building, we can use presets to create simple, named workflows for testing.

We do this by adding a testPresets array to our CMakePresets.json file.

CMakePresets.json

1{
2  "version": 3,
3  "configurePresets": [{
4    "name": "default",
5    "binaryDir": "${sourceDir}/build"
6  }],
7  "buildPresets": [{
8    "name": "default",
9    "configurePreset": "default"
10  }],
11  "testPresets": [{
12    "name": "default",
13    "displayName": "Run All Tests",
14    "configurePreset": "default"
15  }]
16}

"name": The unique name for the test preset.
"displayName": A friendly name for our preset that can be used by IDEs and other tools.
"configurePreset": Similar to our build presets, this links our test preset to the configure preset it requires. It tells CTest which build directory to run the tests from by pointing to the corresponding configure preset.

With this preset defined, we can now configure, build, and test our project from the root directory with a consistent set of commands:

1cmake --preset=default

1cmake --build --preset=default

1ctest --preset=default

1Test project D:/Projects/cmake/build
2    Start 1: Greeter.API.get_greeting
31/1 Test #1: Greeter.API.get_greeting ... Passed  0.01 sec
4
5100% tests passed, 0 tests failed out of 1
6
7Total Test time (real) =   0.01 sec

Analyzing Failures

Let's see what happens when a test fails. We'll intentionally introduce a bug into our GreeterLib by changing the greeting string.

greeter/src/Greeter.cpp

1#include <greeter/Greeter.h>
2#include <string>
3
4std::string get_greeting() {
5  return "Hello from the buggy greeter library"; 
6}

Now, let's rebuild and run our test preset:

1cmake --build --preset=default

1ctest --preset=default

The summary now correctly reports the failure:

1Test project D:/Projects/cmake/build
2    Start 1: Greeter.API.get_greeting
31/1 Test #1: Greeter.API.get_greeting ... Failed  0.04 sec
4
50% tests passed, 1 tests failed out of 1
6
7Total Test time (real) =   0.04 sec
8
9The following tests FAILED:
10          1 - Greeter.API.get_greeting (Failed)
11Errors while running CTest
12Output from these tests are in: D:/Projects/cmake/build/Testing/Temporary/LastTest.log
13Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.

Verbose Output

This is good, but it doesn't tell us why the test failed. It does generate a log file which explains the failure, but most people prefer to see the reason directly in the terminal rather than opening a log file.

We can set this up with command-line flag --output-on-failure like our error report suggested, but a more common way is to add this to our test preset, or to create a new test preset for debugging tests.

With either approach, we'd add an output key, whose value is an object with outputOnFailure set to true:

CMakePresets.json

1{
2  "version": 3,
3  "configurePresets": [ ... ],
4  "buildPresets": [ ... ],
5  "testPresets": [
6    {
7      "name": "default",
8      "displayName": "Run All Tests",
9      "configurePreset": "default"
10    },
11    {
12      "name": "debug-tests",
13      "displayName": "Run Tests (Output on Failure)",
14      "configurePreset": "default",
15      "output": { "outputOnFailure": true }
16    }
17  ]
18}

Now, when we run our new preset, CTest will print the stdout and stderr from any test that fails.

1ctest --preset=debug-tests

This time, we get the detailed error message we wrote in our test driver in /tests/main.cpp, making it much easier to see the problem:

1Test project D:/Projects/cmake/build
2    Start 1: Greeter.API.get_greeting
31/1 Test #1: Greeter.API.get_greeting ... Failed  0.01 sec
4Test Failed:
5  Expected: Hello from the modular greeter library
6  Actual:   Hello from the buggy greeter library
7
8
90% tests passed, 1 tests failed out of 1
10
11Total Test time (real) =   0.01 sec
12
13The following tests FAILED:
14          1 - Greeter.API.get_greeting (Failed)
15Errors while running CTest

Let's fix our bug in Greeter.cpp and ensure our tests pass once more before moving on:

greeter/src/Greeter.cpp

1#include <greeter/Greeter.h>
2
3std::string get_greeting() {
4  return "Hello from the modular greeter library"; 
5}

Summary

In this lesson, we took our first steps into automated testing with CTest, CMake's testing framework.

Why Test?: We learned that automated tests are an important tool for ensuring code correctness, preventing regressions, and enabling confident refactoring.
The CTest Workflow: The process involves enabling testing with enable_testing(), creating a test driver executable, and registering it as a test case with add_test().
Test Drivers: A test driver is a simple program that returns 0 for success and a non-zero value for failure. This exit code is how CTest determines the pass/fail status.
Presets for Testing: We saw how testPresets in CMakePresets.json can simplify the testing workflow, providing named shortcuts for running tests with specific configurations.

This was a quick crash-tour of CTest and how we can use it. It's a powerful tool with many more options, which are listed on the official docs.

For now, the most pressing concern is that our test driver makes writing tests extremely tedious. For every check, we have to write a verbose if/else blocks, manually create strings to report successes and explain failures, and then write std::cerr statements to let people see those reports.

Automated Testing and CTest

The Benefits of Automated Testing

Creating a Test Sub-Project

CMakeLists.txt

Writing a Test Driver

tests/CMakeLists.txt

tests/main.cpp

greeter/src/greeter.cpp

Registering Tests with `add_test()`

tests/CMakeLists.txt

Running Tests with `ctest`

Creating Test Presets

CMakePresets.json

Analyzing Failures

greeter/src/Greeter.cpp

Verbose Output

CMakePresets.json

greeter/src/Greeter.cpp

Summary

Managing C++ Projects Using CMake

Automated Testing and CTest

The Benefits of Automated Testing

Creating a Test Sub-Project

CMakeLists.txt

Writing a Test Driver

tests/CMakeLists.txt

tests/main.cpp

greeter/src/greeter.cpp

Registering Tests with add_test()

tests/CMakeLists.txt

Running Tests with ctest

Creating Test Presets

CMakePresets.json

Analyzing Failures

greeter/src/Greeter.cpp

Verbose Output

CMakePresets.json

greeter/src/Greeter.cpp

Summary

Registering Tests with `add_test()`

Running Tests with `ctest`