Testing Executables

Test-Driven Development (TDD) is a process that flips the usual workflow on its head. Instead of writing production code and then writing tests for it, you write the tests first. This has several advantages:

• Drives Better Design: Before you write a single line of implementation, you must first think as a consumer of your own code. What should the function be called? What arguments should it take? What should it return? Writing the test first forces you to design a friendly API from the outside in.
• Creates Specifications: A test is an unambiguous definition of what the production code must do to be considered "correct." By considering and testing edge cases before writing the implementation, we build a clearer target to aim for, which helps us make better decisions on how the feature should be implemented.
• It helps with development: When we're writing our code, we now have the option of running the tests to see what our logic is doing. This is often quicker than stepping through in a debugger or logging out results.
• It tests the tests: In complex scenarios, it's surprisingly easy to write tests that always pass, regardless of how the system is behaving. If a test passes before you've written the implementation, you know the test itself is flawed.

Practical Example: Adding Logging

Let's use TDD to add a new feature to our GreeterApp. The requirement is: when run_app() is executed, it should create a log file named app_log.txt and write a timestamped startup message to it.

Write a Failing Test

First, we'll write a test in tests/app/test_app.cpp that assumes this feature already exists:

1#include <gmock/gmock.h>
2#include <gtest/gtest.h>
3#include <app/run.h>
4#include <fstream>
5#include <string>
6
7class AppLogTest : public testing::Test {
8protected:
9  const std::string log_filename{"app_log.txt"};
10};
11
12TEST_F(AppLogTest, LogsStartupMessageToFile) {
13  run_app(0, nullptr);
14
15  std::ifstream log_file(log_filename);
16  ASSERT_TRUE(log_file.is_open());
17
18  std::string line;
19  std::getline(log_file, line);
20
21  // Use a matcher to check the format
22  EXPECT_THAT(line, testing::HasSubstr(
23    "App starting up"
24  ));
25}

Let's walk through the logic here:

• We use a test fixture in this example - AppLogTest. This is not necessary for now - we could just use a regular test that includes the app_log.txt string - but we'll expand this file soon and the fixture will be useful.
• We create a file stream using std::ifstream to interact with the file.
• We first check if the file exists with the file stream's is_open() method. We use ASSERT here to create a fatal error if the file doesn't exist, as there's no point in continuing in that scenario
• We use std::getline() to copy the first line from the stream into a std::string called line
• We expect that the line contains the substring "App starting up". Our log file will have additional information too, such as the timestamp. We can test for that using regex techniques (testing::MatchesRegex or testing::ContainsRegex) if preferred, but a simple check using testing::HasSubstr is good enough for this example

Note that this test might find a log file that was created in a previous test run. This would mean our test would pass even though our current executable is broken and is no longer creating log files. We'll fix this flaw later in the lesson.

If we build and run this test now, it will fail because app_log.txt isn't being created. This failure is a good thing, as we haven't built that feature yet:

1cmake --build --preset default

1ctest --preset default

1Value of: log_file.is_open()
2  Actual: false
3Expected: true
4
5[  FAILED  ] AppLogTest.LogsStartupMessageToFile (0 ms)

Making the Test Pass

Now, we write the code in app/src/run.cpp to make the test pass. We'll add the spdlog dependency and the file logging logic.

First, let's update vcpkg.json to include spdlog:

1{
2  "name": "greeter",
3  "dependencies": [
4    "gtest",
5    "spdlog"
6  ]
7}

We'll also update app/CMakeLists.txt to find and link spdlog to our logic library:

1cmake_minimum_required(VERSION 3.23)
2
3add_library(GreeterAppLogic src/run.cpp)
4find_package(spdlog CONFIG REQUIRED)
5
6target_link_libraries(GreeterAppLogic PRIVATE
7  GreeterLib
8  spdlog::spdlog
9)
10
11target_sources(GreeterAppLogic PUBLIC
12  FILE_SET HEADERS
13  BASE_DIRS "include"
14  FILES "include/app/run.h"
15)
16
17add_executable(GreeterApp src/main.cpp)
18target_link_libraries(GreeterApp PRIVATE GreeterAppLogic)

Now, let's add the logging code to run_app().

1#include <app/run.h>
2#include <greeter/Greeter.h>
3#include <iostream>
4#include <spdlog/sinks/basic_file_sink.h>
5#include <spdlog/spdlog.h>
6
7int run_app(int argc, char* argv[]) {
8  // Create a logger that writes to "app_log.txt"
9  auto file_logger{
10    spdlog::basic_logger_mt("app_logger", "app_log.txt")
11  };
12  spdlog::set_default_logger(file_logger);
13
14  spdlog::info("App starting up");
15
16  Greeter my_greeter;
17  std::cout << my_greeter.greet();
18
19  // Flush logs to ensure they're written
20  spdlog::shutdown();
21  return 0;
22}

Finally, if we rebuild and run our tests, the LogsStartupMessageToFile test should pass.

1cmake --build --preset default

1ctest --default

1...
24/4 Test #4: AppLogTest.LogsStartupMessageToFile ... Passed
3100% tests passed, 0 tests failed out of 4

Previously, we saw how creating a fixture in the form of a class derived from testing::Test lets us share variables and functions that a collection of tests might need.

This may simply involve creating some member variables or, in more complex cases, we can add a constructor or override the inherited SetUp() method for logic that needs to run before every test that uses this fixture:

1class AppLogTest : public testing::Test {
2protected:
3  const std::string log_filename{"app_log.txt"};
4
5  void SetUp() override {
6    // SetUp logic...
7  }
8};

We can extend this to have our tests clean up after themselves by implementing a destructor, or overriding the TearDown() method:

1class AppLogTest : public testing::Test {
2protected:
3  const std::string log_filename{"app_log.txt"};
4
5  void SetUp() override {
6    // SetUp logic...
7  }
8
9  void TearDown() override {
10    // TearDown logic...
11  }
12};

This is useful when our tests are creating long-lasting changes that may affect future tests. Our test that calls run_app() is an example of this. It creates a file on the filesystem, which will persist even after the test ends.

This is a problem because if, for example, we make a change that breaks our logging, our test might still pass. It simply reads the log file from the previous run and assumes all is well.

Even if we clear out our build directory between each test run, this lingering file can still cause issues within a single test run. For example, if we write a second test that runs our app in some different context and checks for the log file. That log file may already exist from the other test, giving us unreliable reports.

Below, we write a new test that asserts that a log file exist, but doesn't actually do anything that would cause that log file to be created:

1#include <gmock/gmock.h>
2#include <gtest/gtest.h>
3#include <app/run.h>
4#include <fstream>
5#include <string>
6
7class AppLogTest : public testing::Test {
8protected:
9  const std::string log_filename{"app_log.txt"};
10};
11
Previous Test (Unchanged)|Show
12TEST_F(AppLogTest, LogsStartupMessageToFile) {/*...*/}
27
28TEST_F(AppLogTest, AnotherTest) {
29  std::ifstream log_file(log_filename);
30  
31  // ...don't create a log file
32  
33  ASSERT_TRUE(log_file.is_open());
34}

This test should fail but, if it runs after our existing test created that log file, it will incorrectly pass:

1cmake --build --preset default

1ctest --preset default

15/5 Test #5: AppLogTest.AnotherTest ... Passed
2100% tests passed, 0 tests failed out of 5

We can solve this by adding a TearDown() function to our fixture. This will clean up any residual effects of tests using this fixture, ensuring each test is a discrete, self-contained unit:

1#include <gmock/gmock.h>
2#include <gtest/gtest.h>
3#include <app/run.h>
4#include <fstream>
5#include <string>
6
7class AppLogTest : public testing::Test {
8protected:
9  const std::string log_filename{"app_log.txt"};
10
11  void TearDown() override {
12    // Clean up the log file after each test
13    std::remove(log_filename.c_str());
14  }
15};
16
Previous Test (Unchanged)|Show
17TEST_F(AppLogTest, LogsStartupMessageToFile) {/*...*/}
32
Previous Test (Unchanged)|Show
33TEST_F(AppLogTest, AnotherTest) {/*...*/}

Our AnotherTest should now reliably fail as expected:

1cmake --build --preset default

1ctest --preset default

1Value of: log_file.is_open()
2  Actual: false
3Expected: true
4
5[  FAILED  ] AppLogTest.AnotherTest (0 ms)

This lesson addressed the challenge of testing code that lives inside an executable. By applying a simple refactoring pattern, we made our application's logic fully testable.

• The Problem: Executable targets can't be linked against, making it impossible for a test executable to directly call functions defined in the application's main.cpp.
• The Solution: Isolate the application's core logic into its own library (GreeterAppLogic). The executable (GreeterApp) becomes a thin, untestable wrapper that simply calls a function from the logic library.
• Testable by Design: The test executable can then link against the new logic library, giving it full access to all the application's core functionality for testing.
• TDD for Applications: We saw how to write a failing test first and then implement the feature in the logic library to make it pass.
• Test Isolation: We used gtest fixtures with SetUp() and TearDown() methods to ensure our tests are isolated, cleaning up side effects like created files so that one test cannot influence another.