Is there a way to determine the encoding of a given string at runtime?

Question

Ryan McCombe · Accepted Answer

Determining the encoding of a string at runtime is a challenging task, as there's no foolproof way to detect encoding with 100% accuracy. However, we can use heuristics and libraries to make educated guesses. Here are a few approaches:

Using Heuristics

We can examine the byte patterns in the string to make an educated guess about its encoding. Here's a simple example that can distinguish between ASCII, UTF-8, and UTF-16:

1#include <iostream>
2#include <string>
3#include <vector>
4
5std::string guessEncoding(
6  const std::vector<unsigned char>& bytes
7) {
8  if (bytes.empty()) return "Empty string";
9
10  // Check for UTF-16 BOM
11  if (bytes.size() >= 2) {
12    if (bytes[0] == 0xFF && bytes[1] == 0xFE)
13      return "UTF-16LE";
14    if (bytes[0] == 0xFE && bytes[1] == 0xFF)
15      return "UTF-16BE";
16  }
17
18  // Check for UTF-8
19  bool isAscii = true;
20  bool couldBeUtf8 = true;
21  int continuationBytes = 0;
22
23  for (unsigned char byte : bytes) {
24    if (byte & 0x80) isAscii = false;
25
26    if (continuationBytes) {
27      if ((byte & 0xC0) != 0x80) {
28        couldBeUtf8 = false;
29        break;
30      }
31      continuationBytes--;
32    } else if ((byte & 0xE0) == 0xC0)
33      continuationBytes = 1;
34    else if ((byte & 0xF0) == 0xE0)
35      continuationBytes = 2;
36    else if ((byte & 0xF8) == 0xF0)
37      continuationBytes = 3;
38    else if (byte & 0x80) {
39      couldBeUtf8 = false;
40      break;
41    }
42  }
43
44  if (isAscii) return "ASCII";
45  if (couldBeUtf8) return "UTF-8";
46  return "Unknown encoding";
47}
48
49int main() {
50  std::vector<unsigned char> ascii = {
51    'H', 'e', 'l', 'l', 'o'};
52  std::vector<unsigned char> utf8 = {
53    0xE2, 0x82, 0xAC};  // Euro sign
54  std::vector<unsigned char> utf16le = {
55    0xFF, 0xFE, 0x20,  0x00};  // Space
56
57  std::cout << "ASCII string: "
58    << guessEncoding(ascii) << '\n';
59  std::cout << "UTF-8 string: "
60    << guessEncoding(utf8) << '\n';
61  std::cout << "UTF-16LE string: "
62    << guessEncoding(utf16le) << '\n';
63}

1ASCII string: ASCII
2UTF-8 string: UTF-8
3UTF-16LE string: UTF-16LE

Using Libraries

For more robust encoding detection, consider using libraries like ICU (International Components for Unicode) or uchardet. These libraries use sophisticated algorithms to guess the encoding of a string.

Here's an example using uchardet:

1#include <iostream>
2#include <string>
3#include <uchardet.h>
4
5std::string detectEncoding(const std::string& str) {
6  uchardet_t handle = uchardet_new();
7  int retval = uchardet_handle_data(
8    handle, str.c_str(), str.length()
9  );
10  uchardet_data_end(handle);
11  std::string encoding =
12    uchardet_get_charset(handle);
13  uchardet_delete(handle);
14  return encoding.empty() ? "Unknown" : encoding;
15}
16
17int main() {
18  std::string ascii = "Hello, world!";
19  std::string utf8 = "Hello, !";
20
21  std::cout << "ASCII string encoding: "
22            << detectEncoding(ascii) << '\n';
23  std::cout << "UTF-8 string encoding: "
24            << detectEncoding(utf8) << '\n';
25}

Remember, these methods are not foolproof. Some encodings (like UTF-8 and ASCII) can be reliably detected in many cases, but others might be indistinguishable without additional context. Always test thoroughly with various inputs when implementing encoding detection in your applications.

Characters, Unicode and Encoding

Determining String Encoding at Runtime

Using Heuristics

Using Libraries

Characters, Unicode and Encoding

Professional C++

Questions & Answers