Designing a Programming Language: How Code Comes to Life
How does code such as x = 5 + 2; become an action on a computer? There is no single pipeline used by every language. The implementation may compile source code ahead of time, interpret it at runtime, compile it to bytecode for a virtual machine, or combine several of these approaches.
“Compiled” and “interpreted” are useful broad labels, but they are not a complete description of a language. A C++ implementation commonly translates source into machine code before the program runs. A language commonly described as interpreted may parse source, produce bytecode, and execute that bytecode in a runtime. Some runtimes also compile frequently used code while the program is running. The exact choices depend on the language and its implementation.
Regardless of which execution strategy it uses, an implementation first has to understand the source. Two common early stages are tokenization and parsing.
Tokenization: Breaking Down the Code
Tokenization, also called lexical analysis, turns a stream of characters into meaningful pieces called tokens. These tokens can be keywords, names, numbers, operators, or punctuation.
For example, x = 5 + 2; can become the tokens x, =, 5, +, 2, and ;. The tokenizer may also record where each token appeared so later error messages can point to the right part of the source.
Parsing: Understanding the Structure
Parsing takes the tokens and checks how they fit the grammar of the language. It can reject code with a missing parenthesis or an operator in the wrong place. A successful parser often builds an Abstract Syntax Tree, or AST.
An AST keeps the relationships that matter to the program and leaves out details such as extra spaces. For x = 5 + 2;, the tree can represent an assignment whose value is an addition expression. The assignment contains x on one side and 5 + 2 on the other.
Abstract Syntax Tree (AST): The Code’s Map
The AST is a structured representation that later stages can inspect. An implementation may use it for semantic checks, such as finding an undefined name or checking whether an operation has compatible types. It may then lower the tree into another intermediate form, optimize that form, generate bytecode or machine code, and link compiled pieces when the language needs a separate linking step.
Those stages are common, not mandatory in exactly that order. A small interpreter might walk the AST directly. Another implementation might turn the AST into bytecode first. A native compiler may continue through optimization, code generation, and linking before execution.
Execution: Making the Result Happen
Execution is the stage where the runtime or generated code performs the operations described by the program. In the example, evaluating 5 + 2 produces 7, and the assignment stores that value in x.
The useful beginner model is: source characters become tokens, tokens become a structured representation, and a runtime or generated code executes that representation. Real language implementations add more stages and sometimes combine them, but this model explains the path without pretending every language works the same way.