Handling Paginated HTTP API Responses

When making requests to HTTP APIs, some endpoints may return paginated responses. This is common when querying for lists of items like products, orders, users, etc. Rather than returning the entire dataset in one response, it's broken up into "pages".

The exact way pagination works is API-specific but a typical pattern is:

• The response contains a subset of the total results
• The response includes metadata about the total number of pages and items
• A "nextPage" URL is provided that can be requested to get the next page of results
• We keep requesting pages until there is no next page URL

Here's an example of how we might use the cpr library to handle a paginated API response:

1#include <cpr/cpr.h>
2#include <iostream>
3#include <nlohmann/json.hpp>
4using json = nlohmann::json;
5
6int main() {
7  cpr::Url base{"https://api.example.com/items"};
8  cpr::Parameters params{{"perPage", "10"}};
9
10  cpr::Response r = cpr::Get(base, params);
11  json resp = json::parse(r.text);
12
13  std::cout << resp["total"].get<int>()
14    << " total items\n";
15
16  while (true) {
17    for (auto& item : resp["data"]) {
18      std::string id =
19        item["id"].get<std::string>();
20      std::cout << "Item ID: " << id << '\n';
21      // ...process each item
22    }
23
24    if (resp.contains("nextPage")) {
25      std::string nextUrl = resp["nextPage"];
26      r = cpr::Get(cpr::Url{nextUrl});
27      resp = json::parse(r.text);
28    } else {
29      break;
30    }
31  }
32}

This code assumes a JSON response in the following format:

1{
2  "total": 100,
3  "perPage": 10,
4  "page": 1,
5  "lastPage": 10,
6  "data": [
7    {"id": "abc123", ...},
8    ...
9  ],
10  "nextPage": "https://api.example.com/items?page=2"
11}

The perPage parameter specifies how many items to return per page. We extract the total number of items from the first response.

Then we loop through each "page" of data, processing the items. If the response contains a nextPage key, we request that URL to get the next page. Once nextPage is missing, we know we've reached the end.

Of course, the exact logic depends on the API spec, but the general idea is to keep requesting pages until there are none left. We can accumulate the items into a local collection if needed.