Our tests tell us if our code is correct, but not if it's fast. To measure performance, we use a benchmark. A benchmark is a program designed to run a piece of code repeatedly under controlled conditions to produce stable and comparable performance metrics.

Specifically, we'll focus on micro-benchmarks. A micro-benchmark is an automated script that measures the performance of a specific part of our program - perhaps a single function or algorithm.

Google Benchmark is the de-facto standard for C++ micro-benchmarking. It is a library that provides a framework for quickly creating these benchmarks, and it takes care of details like running the code enough times to get a stable result and preventing the compiler from optimizing the code away.

Integrating Google Benchmark

First, let's add benchmark to our vcpkg.json manifest.

vcpkg.json

1{
2  "name": "greeter",
3  "dependencies": [
4    "gtest",
5    "spdlog",
6    "benchmark"
7  ]
8}

Next, we'll create a new benchmarks/ directory for our benchmark code and its CMakeLists.txt.

benchmarks/CMakeLists.txt

1cmake_minimum_required(VERSION 3.23)
2
3find_package(benchmark CONFIG REQUIRED)
4
5add_executable(GreeterBenchmarks bench_main.cpp)
6
7target_link_libraries(GreeterBenchmarks PRIVATE
8  GreeterLib
9  benchmark::benchmark
10)

Finally, we add this new directory to our root CMakeLists.txt:

CMakeLists.txt

1cmake_minimum_required(VERSION 3.23)
2project(Greeter)
3
4include(cmake/Coverage.cmake)
5include(cmake/Sanitize.cmake)
6
7add_subdirectory(app)
8add_subdirectory(greeter)
9
10enable_testing()
11add_subdirectory(tests)
12
13add_subdirectory(benchmarks)

Writing a Benchmark

A benchmark looks very similar to a GoogleTest case. Let's write one in benchmarks/bench_main.cpp to measure our Greeter::greet() method.

benchmarks/bench_main.cpp

1#include <benchmark/benchmark.h>
2#include <greeter/Greeter.h>
3
4static void BM_Greeter_Greet(benchmark::State& state) {
5  Greeter g;
6  // This loop is the core of the benchmark
7  for (auto _ : state) {
8    // This code gets timed
9    std::string result = g.greet();
10    // Prevent the result from being optimized away
11    benchmark::DoNotOptimize(result);
12  }
13}
14
15// Register the function as a benchmark
16BENCHMARK(BM_Greeter_Greet);
17
18// Run all benchmarks
19BENCHMARK_MAIN();

Running the Benchmark

To get useful benchmarking results, we should build our project in "Release" mode. This prevents any debugging helpers from degrading our performance.

Let's add some presets for this, if we haven't already:

CMakePresets.json

1{
2  "version": 3,
3  "configurePresets": [
4    // ... other presets
5    {
6      "name": "release",
7      "inherits": "default",
8      "cacheVariables": {
9        "CMAKE_BUILD_TYPE": "Release"
10      }
11  ],
12  "buildPresets": [
13    // ... other presets
14    {
15      "name": "release",
16      "configurePreset": "release",
17      "configuration": "Release"
18    }
19  ]
20}

We can now configure and build our project with these new presets. From the project root:

1cmake --preset=release

1cmake --build --preset=release

Our GreeterBenchmarks target should have generated a GreeterBenchmarks (or GreeterBenchmarks.exe) in the build/benchmarks directory. We can run it in the usual way from the project root:

1./build/benchmarks/GreeterBenchmarks

Google Benchmark will run the test and produce a detailed report showing the average time taken per iteration, CPU time, and other useful metrics.

1-----------------------------------------------
2Benchmark            Time      CPU   Iterations
3-----------------------------------------------
4BM_Greeter_Greet  72.3 ns  69.8 ns      8960000

Parameterized Benchmarks

A simple benchmark is useful, but often we'll want to run the same benchmark across many cases. For example, we might want to see how our code behaves across a variety of different input sizes, or to compare the performance of multiple options.

Much like Google Test, Google Benchmark includes utilitities to help us create parameterized benchmarks that run the same code with different arguments.

Let's modify our Greeter class to greet a specific person by name, and then benchmark how the greet() method performs with names of different lengths.

Files

greeter

tests

Select a file to view its content

Now, we can update our benchmarking. We'll pass arguments to the benchmark function using the Arg() method, representing the length of the name we want to test:

benchmarks/bench_main.cpp

1#include <benchmark/benchmark.h>
2#include <greeter/Greeter.h>
3#include <string>
4
5static void BM_Greeter_Greet(benchmark::State& state) {
6  // state.range(0) is the first argument to the benchmark
7  std::string name(state.range(0), 'x');
8  Greeter g(name);
9
10  for (auto _ : state) {
11    std::string result = g.greet();
12    benchmark::DoNotOptimize(result);
13  }
14}
15
16// Register benchmarks with different arguments
17BENCHMARK(BM_Greeter_Greet)->Arg(8);
18BENCHMARK(BM_Greeter_Greet)->Arg(64);
19BENCHMARK(BM_Greeter_Greet)->Arg(512);
20BENCHMARK(BM_Greeter_Greet)->Arg(4096);
21BENCHMARK(BM_Greeter_Greet)->Arg(32768);
22
23BENCHMARK_MAIN();

This Arg() approach is generally flexible, but the specific case of testing different input sizes is so common that a shortcut is available in the form of the Range() method:

1// Before:
2BENCHMARK(BM_Greeter_Greet)->Arg(8);
3BENCHMARK(BM_Greeter_Greet)->Arg(64);
4BENCHMARK(BM_Greeter_Greet)->Arg(512);
5BENCHMARK(BM_Greeter_Greet)->Arg(4096);
6BENCHMARK(BM_Greeter_Greet)->Arg(32768);
7
8// After:
9BENCHMARK(BM_Greeter_Greet)->Range(8, 32768);

This use of Range(8, 32768) tells Google Benchmark to run this benchmark multiple times. It will start with an argument of 8, and for each subsequent run, it will multiply the argument by 8 until it reaches or exceeds $32,768$ (which is $8^5$ ).

If we run this new benchmark, we'll get a table of results showing how the performance scales with the input size:

1cmake --preset release

1cmake --build --preset release

1./build/benchmarks/GreeterBenchmarks

1------------------------------------------------------
2Benchmark                  Time        CPU  Iterations
3------------------------------------------------------
4BM_Greeter_Greet/8      15.7 ns    15.7 ns    44800000
5BM_Greeter_Greet/64     91.6 ns    92.1 ns     7466667
6BM_Greeter_Greet/512     106 ns     106 ns     5600000
7BM_Greeter_Greet/4096    167 ns     167 ns     4480000
8BM_Greeter_Greet/32768  1507 ns    1507 ns      497778

Other Benchmark Features

Google Benchmark has a wide range of features to support benchmarking. Here are a few other capabilities you might find useful:

Fixtures: Just like in GoogleTest, you can create a fixture class (by inheriting from benchmark::Fixture) to handle complex setup and teardown logic that can be shared across multiple benchmarks.
Time Units: You can control the time unit reported in the output (nanoseconds, microseconds, etc.) by calling Unit(benchmark::kMillisecond) on your benchmark registration.
Complexity Analysis: Google Benchmark can automatically compute the asymptotic complexity of your code - e.g., $O(n)$ , $O(n^2)$ - if you provide it with the input size via the Arg() or Range() methods.
Custom Counters: You can report your own custom metrics (e.g., "bytes processed per second") from within your benchmark loop.

For a complete guide to all the features, the official Google Benchmark documentation is the best resource.

Summary

In this lesson, we've seen how to to start measuring performance within our build.

Micro-benchmarking: A benchmark is a program for measuring the performance of a small piece of code in a controlled environment.
Google Benchmark: This is an extremely popular library for C++ benchmarking. We integrated it into our project using vcpkg and a dedicated benchmarks target.
Writing Benchmarks: We learned the basic structure of a benchmark, including the for (auto _ : state) loop and the importance of benchmark::DoNotOptimize() to prevent the compiler from removing the code being tested.
Parameterized Benchmarks: We used the Arg() and Range() method to run our benchmark with a variety of inputs, allowing us to see how performance scales.

Using Google Benchmark

Integrating Google Benchmark

vcpkg.json

benchmarks/CMakeLists.txt

CMakeLists.txt

Writing a Benchmark

benchmarks/bench_main.cpp

Running the Benchmark

CMakePresets.json

Parameterized Benchmarks

Files

benchmarks/bench_main.cpp

Other Benchmark Features

Summary

Using Clang-Tidy

Managing C++ Projects Using CMake