Phases of translation

C++ source files are processed by the compiler to produce C++ programs.

Translation process

The text of a C++ program is kept in units called source files.

C++ source files undergo translation to become a translation unit, consisting of the following steps:

1. Maps each source file to a character sequence.
2. Converts each character sequence to a preprocessing token sequence, separated by whitespace.
3. Converts each preprocessing token to a token, forming a token sequence.
4. Converts each token sequence to a translation unit.

A C++ program can be formed from translated translation units. Translation units can be separately translated and then later linked to produce an executable program.

The process above can be organized into 9 translation phases.

Preprocessing tokens

A preprocessing token is the minimal lexical element of the language in translation phases 3 through 6.

The categories of preprocessing token are:

• header names (such as <iostream> or "myfile.h"

• identifiers
• preprocessing numbers (see below)
• character literals , including user-defined character literals(since C++11)
• string literals , including user-defined string literals(since C++11)
• operators and punctuators, including alternative tokens
• individual non-whitespace characters that do not fit in any other category

The program is ill-formed if the character matching this category is

• apostrophe (', U+0027),
• quotation mark (", U+0022), or
• a character not in the basic character set.

Preprocessing numbers

The set of preprocessing tokens of preprocessing number is a superset of the union of the sets of tokens of integer literals and floating-point literals

Each pp-continue is one of the following:

A preprocessing number does not have a type or a value; it acquires both after a successful conversion an integer/floating-point literal token.

Whitespace

Whitespace consists of comments, whitespace characters, or both.

The following characters are whitespace characters:

• character tabulation (U+0009)
• line feed / new-line character (U+000A)
• line tabulation (U+000B)
• form feed (U+000C)
• space (U+0020)

Whitespace is usually used to separate preprocessing tokens, with the following exceptions:

• It is not a separator in header name, character literal and string literal.
• Preprocessing tokens separated by whitespace containing new-line characters cannot form preprocessing directives.

#include "my header"        // OK, using a header name containing whitespace
 
#include/*hello*/<iostream> // OK, using a comment as whitespace
 
#include
<iostream> // Error: #include cannot span across multiple lines
 
"str ing"  // OK, a single preprocessing token (string literal)
' '        // OK, a single preprocessing token (character literal)

Maximal munch

If the input has been parsed into preprocessing tokens up to a given character, the next preprocessing token is generally taken to be the longest sequence of characters that could constitute a preprocessing token, even if that would cause subsequent analysis to fail. This is commonly known as maximal munch

int foo = 1;
int bar = 0xE+foo;   // Error: invalid preprocessing number 0xE+foo
int baz = 0xE + foo; // OK

In other words, the maximal munch rule is in favor of multi-character operators and punctuators:

int foo = 1;
int bar = 2;
 
int num1 = foo+++++bar; // Error: treated as “foo++ ++ +baz”, not “foo++ + ++baz”
int num2 = -----foo;    // Error: treated as “-- -- -foo”, not “- -- --foo”

The maximal munch rule has the following exceptions:

• Header name preprocessing tokens are only formed in the following cases:

• after the include preprocessing token in an #include

std::vector<int> x; // OK, “int” is not a header name

• If the next three characters are <:: and the subsequent character is neither : nor >, the < is treated as a preprocessing token by itself instead of the first character of the alternative token <:

struct Foo { static const int v = 1; };
std::vector<::Foo> x;  // OK, <: not taken as the alternative token for [
extern int y<::>;      // OK, same as “extern int y[];”
int z<:::Foo::value:>; // OK, same as “int z[::Foo::value];”

template<int i> class X { /* ... */ };
template<class T> class Y { /* ... */ };
 
Y<X<1>> x3;      // OK, declares a variable “x3” of type “Y<X<1> >”
Y<X<6>>1>> x4;   // Syntax error
Y<X<(6>>1)>> x5; // OK

#define R "x"
const char* s = R"y";         // ill-formed raw string literal, not "x" "y"
const char* s2 = R"(a)" "b)"; // a raw string literal followed by a normal string literal

Tokens

A token is the minimal lexical element of the language in translation phase 7.

The categories of token are:

• identifiers
• keywords
• literals
• operators and punctuators (except preprocessing operators)

Translation phases

Translation is performed as if in the order from phase 1 to phase 9. Implementations behave as if these separate phases occur, although in practice different phases can be folded together.

Phase 1: Mapping source characters

Phase 2: Splicing lines

Phase 3: Lexing

// The following #include directive can de decomposed into 5 preprocessing tokens:
 
//     punctuators (#, < and >)
//          │
// ┌────────┼────────┐
// │        │        │
   #include <iostream>
//     │        │
//     │        └── header name (iostream)
//     │
//     └─────────── identifier (include)

// Error: partial string literal
"abc

// Error: partial comment
/* comment

Phase 4: Preprocessing

Phase 5: Determining common string literal encodings

Phase 6: Concatenating string literals

Adjacent string literals are concatenated.

Phase 7: Compiling

Compilation takes place: each preprocessing token is converted to a token. The tokens are syntactically and semantically analyzed and translated as a translation unit

Phase 8: Instantiating templates

Each translation unit is examined to produce a list of required template instantiations, including the ones requested by explicit instantiations. The definitions of the templates are located, and the required instantiations are performed to produce instantiation units

Phase 9: Linking

Translation units, instantiation units, and library components needed to satisfy external references are collected into a program image which contains information needed for execution in its execution environment.

Notes

Source files, translation units and translated translation units need not necessarily be stored as files, nor need there be any one-to-one correspondence between these entities and any external representation. The description is conceptual only, and does not specify any particular implementation.

Some compilers do not implement instantiation units (also known as template repositories or template registries

Defect reports

The following behavior-changing defect reports were applied retroactively to previously published C++ standards.

References

See also