Lexical analyzer for C/C++ source code. More...

#include <lexer.hpp>

Public Member Functions
	Lexer (gsl::not_null< const TextFormatter * > format, std::string content)
	Constructs a lexer for the given source content.

	Lexer (gsl::not_null< const TextFormatter * > format, std::string content, const CParserContext *context)
	Constructs a lexer with a context for rich error formatting.

ChainableResult< std::vector< Token > >	lex ()
	Tokenizes the entire source content.

Detailed Description

Lexical analyzer for C/C++ source code.

Lexer converts a string of C/C++ source code into a sequence of tokens. It handles:

Preprocessor directives (#define, #ifdef, etc.)
Identifiers and keywords
Integer literals (decimal, hexadecimal, octal, binary)
String literals
Operators and delimiters
Comments (// and multi-line, which are skipped)
Whitespace (skipped except for newlines which are significant for preprocessor)

The lexer tracks source positions for error reporting. It does not perform any file I/O; the caller is responsible for reading file contents and passing them as a string.

Example usage:

Lexer lexer{"#define FOO 123\n#define BAR 0x456"};
auto result = lexer.lex();
if (result.has_value()) {
    for (const auto& token : result.value()) {
        // process tokens
    }
}

Definition at line 44 of file lexer.hpp.

Constructor & Destructor Documentation

◆ Lexer() [1/2]

porytiles::Lexer::Lexer	(	gsl::not_null< const TextFormatter * >	format,
		std::string	content
	)

Constructs a lexer for the given source content.

This constructor creates a lexer without a CParserContext. Errors will be formatted as simple "line:col: message" strings without source context highlighting.

Parameters

format	The text formatter for styled output
content	The C/C++ source code to tokenize

Definition at line 172 of file lexer.cpp.

◆ Lexer() [2/2]

porytiles::Lexer::Lexer	(	gsl::not_null< const TextFormatter * >	format,
		std::string	content,
		const CParserContext *	context
	)

Constructs a lexer with a context for rich error formatting.

This constructor creates a lexer with a CParserContext that enables rich error formatting with source context highlighting via FileHighlightPrinter. The context must outlive the lexer.

Parameters

format	The text formatter for styled output
content	The C/C++ source code to tokenize
context	The parser context for rich error formatting (non-owning)

Definition at line 177 of file lexer.cpp.

Member Function Documentation

◆ lex()

ChainableResult< std::vector< Token > > porytiles::Lexer::lex ( )

Tokenizes the entire source content.

Processes the source content and returns a vector of tokens. The token stream always ends with an end_of_file token. If an error is encountered (e.g., unterminated string), returns a ChainableResult containing an error with source position information.

When constructed with a CParserContext, errors include source code context with highlighted error locations. When constructed without a context, errors are simple "line:col: message" format.

Returns: A vector of tokens on success, or an error on failure

Definition at line 190 of file lexer.cpp.

The documentation for this class was generated from the following files: