porytiles::Parser Class Reference

Parser for C preprocessor constructs. More...

#include <parser.hpp>

Public Member Functions
	Parser (gsl::not_null< const TextFormatter * > format, std::vector< Token > tokens)
	Constructs a parser for the given token stream.
	Parser (gsl::not_null< const TextFormatter * > format, std::vector< Token > tokens, const CParserContext *context)
	Constructs a parser with a context for rich error formatting.
ChainableResult< std::vector< DefineStatement > >	parse_defines ()
	Parses all #define statements from the token stream.
ChainableResult< std::vector< EnumDeclaration > >	parse_enums ()
	Parses all enum declarations from the token stream.
ChainableResult< std::vector< ArrayDeclaration > >	parse_pointer_arrays ()
	Parses all pointer array declarations from the token stream.
ChainableResult< std::vector< FunctionDefinition > >	parse_functions ()
	Parses function definitions from the token stream.
ChainableResult< std::vector< StructVariableDeclaration > >	parse_struct_variables ()
	Parses struct variable declarations from the token stream.
ChainableResult< std::vector< StructInitializerDeclaration > >	parse_struct_initializers ()
	Parses struct variable declarations with their designated initializer fields.
ChainableResult< std::vector< StructDefinition > >	parse_struct_definitions ()
	Parses struct type definitions from the token stream.
ChainableResult< std::vector< IncbinDeclaration > >	parse_incbin_arrays ()
	Parses INCBIN array declarations from the token stream.
ChainableResult< std::vector< IndexedArrayDeclaration > >	parse_indexed_arrays ()
	Parses array declarations that use designated (indexed) initializers.
TolerantDefineScan	parse_defines_tolerant ()
	Parses all #define statements, tolerating individual evaluation failures.
TolerantEnumScan	parse_enums_tolerant ()
	Parses all enum declarations, tolerating individual member evaluation failures.
const std::vector< std::string > &	scan_warnings () const
	Returns warnings accumulated while scanning conditionals and defines.
const std::unordered_set< std::string > &	ambiguous_defines () const
	Returns names whose definitions conflicted in an undecidable conditional branch.
const std::unordered_set< std::string > &	defined_names () const
	Returns the set of macro names seen as defined so far.
void	seed_symbols (const std::unordered_map< std::string, std::int64_t > &symbols)
	Seeds the symbol table with externally known macro values before scanning.
void	seed_values (const std::unordered_map< std::string, std::int64_t > &symbols)
	Seeds only expression values without marking their names as preprocessor-defined.

Detailed Description

Parser for C preprocessor constructs.

Parser analyzes a token stream and extracts structured information. The initial implementation focuses on parsing #define statements, extracting macro names and evaluating constant expressions.

The parser uses the Shunting Yard algorithm to evaluate arithmetic expressions in #define values, supporting operators: +, -, *, /, %, &, |, ^, ~, <<, >>

Example usage:

Lexer lexer{source_content};
auto tokens_result = lexer.lex();
if (tokens_result.has_value()) {
    Parser parser{std::move(tokens_result).value()};
    auto defines_result = parser.parse_defines();
    if (defines_result.has_value()) {
        for (const auto& def : defines_result.value()) {
            // process defines
        }
    }
}

Future extensions will add methods like:

• parse_functions() for function declarations/definitions
• parse_structs() for struct definitions

Definition at line 58 of file parser.hpp.

Constructor & Destructor Documentation

Parser() [1/2]

porytiles::Parser::Parser ( gsl::not_null< const TextFormatter * >  format, std::vector< Token >  tokens  )

inline

Constructs a parser for the given token stream.

This constructor creates a parser without a CParserContext. Errors will be formatted as simple "line:col: message" strings without source context highlighting.

Parameters

format	The text formatter for styled output
tokens	The tokens to parse (typically from Lexer::lex())

Definition at line 68 of file parser.hpp.

Parser() [2/2]

porytiles::Parser::Parser ( gsl::not_null< const TextFormatter * >  format, std::vector< Token >  tokens, const CParserContext *  context  )

inline

Constructs a parser with a context for rich error formatting.

This constructor creates a parser with a CParserContext that enables rich error formatting with source context highlighting via FileHighlightPrinter. The context must outlive the parser.

Parameters

format	The text formatter for styled output
tokens	The tokens to parse (typically from Lexer::lex())
context	The parser context for rich error formatting (non-owning)

Definition at line 82 of file parser.hpp.

Member Function Documentation

ambiguous_defines()

const std::unordered_set< std::string > & porytiles::Parser::ambiguous_defines ( ) const

inline

Returns names whose definitions conflicted in an undecidable conditional branch.

The tolerant scanner keeps traversing both sides of a conditional whose controlling macro is unknown. When the same name resolves to different integer values on those paths, its final stored value is not authoritative. Consumers that derive binary layouts from the value use this set to reject the ambiguity instead of guessing.

Returns

The names with conflicting values in undecidable conditional branches

Definition at line 327 of file parser.hpp.

defined_names()

const std::unordered_set< std::string > & porytiles::Parser::defined_names ( ) const

inline

Returns the set of macro names seen as defined so far.

Includes every define name recorded during scanning (integer, string, flag, and parametric) plus any names seeded at construction. Used to decide preprocessor conditionals such as #ifdef NAME.

Returns

A const reference to the defined-name set

Definition at line 339 of file parser.hpp.

parse_defines()

ChainableResult< std::vector< DefineStatement > > porytiles::Parser::parse_defines

(

)

Parses all #define statements from the token stream.

Scans through the token stream looking for #define directives. For each one found, parses the macro name and evaluates the value expression if present. Non-define preprocessor directives and other code constructs are skipped.

Supports:

• Simple integer defines: #define FOO 123
• Hex/octal/binary literals: #define BAR 0xFF
• Arithmetic expressions: #define BAZ (1 << 4)
• String defines: #define MSG "hello"
• Flag defines: #define DEBUG
• References to previously defined macros: #define B A (where A was defined earlier)

When constructed with a CParserContext, errors include source code context with highlighted error locations. When constructed without a context, errors are simple "line:col: message" format.

Returns

A vector of DefineStatement on success, or an error on failure

Definition at line 303 of file parser.cpp.

parse_defines_tolerant()

TolerantDefineScan porytiles::Parser::parse_defines_tolerant

(

)

Parses all #define statements, tolerating individual evaluation failures.

Behaves like parse_defines() but never aborts on a define whose value cannot be evaluated (for example a value that references a macro declared in an unparsed header). Such defines are reported in the returned scan's skipped list, and their names are still recorded as defined so later conditionals can use them. Preprocessor conditionals are tracked exactly as in parse_defines().

Returns

The resolved defines and the skipped defines

Definition at line 413 of file parser.cpp.

parse_enums()

ChainableResult< std::vector< EnumDeclaration > > porytiles::Parser::parse_enums

(

)

Parses all enum declarations from the token stream.

Scans through the token stream looking for enum declarations. For each one found, parses the optional enum name and all members with their values. Supports both implicit counter-based values and explicit value assignments.

Supports:

• Anonymous enums: enum { ... }
• Named enums: enum Name { ... }
• Simple members: FOO,
• Explicit values: FOO = 10,
• Expression values: FOO = (1 << 4),
• References to previously defined macros: FOO = SOME_DEFINE,

When constructed with a CParserContext, errors include source code context with highlighted error locations. When constructed without a context, errors are simple "line:col: message" format.

Returns

A vector of EnumDeclaration on success, or an error on failure

Definition at line 590 of file parser.cpp.

parse_enums_tolerant()

TolerantEnumScan porytiles::Parser::parse_enums_tolerant

(

)

Parses all enum declarations, tolerating individual member evaluation failures.

Behaves like parse_enums() but never aborts on an enum member whose explicit value cannot be evaluated. An unevaluable explicit value poisons the running counter, so that member and any following implicit members carry an absent value until the next evaluable explicit value re-anchors the counter. Enums that cannot be parsed structurally are reported in the scan's skipped list.

Returns

The parsed enums (with possibly-absent member values) and any structurally skipped enums

Definition at line 562 of file parser.cpp.

parse_functions()

ChainableResult< std::vector< FunctionDefinition > > porytiles::Parser::parse_functions

(

)

Parses function definitions from the token stream.

Scans through the token stream looking for function definitions matching the pattern:

[static] TYPE IDENTIFIER ( params ) { body }

This is used by AnimCodeParser to extract queue and driver functions like:

static void QueueAnimTiles_General_Flower(u16 timer) {
    AppendTilesetAnimToBuffer(..., TILE_OFFSET_4BPP(12), 4 * TILE_SIZE_4BPP);
}

The parser captures the function name and all tokens within the body braces for later pattern matching.

Returns

A vector of FunctionDefinition on success, or an error on failure

Definition at line 1344 of file parser.cpp.

parse_incbin_arrays()

ChainableResult< std::vector< IncbinDeclaration > > porytiles::Parser::parse_incbin_arrays

(

)

Parses INCBIN array declarations from the token stream.

Scans through the token stream looking for array declarations using INCBIN macros:

// Single path arrays:
[const] TYPE IDENTIFIER [] = INCBIN_MACRO("path");
 
// Multi-path arrays (palettes):
[const] TYPE IDENTIFIER [][SIZE] = { INCBIN_MACRO("path1"), INCBIN_MACRO("path2"), ... };

Examples from pokeemerald:

const u32 gTilesetTiles_General[] = INCBIN_U32("data/tilesets/primary/general/tiles.4bpp");
 
const u16 gTilesetPalettes_General[][16] = {
    INCBIN_U16("data/tilesets/primary/general/palettes/00.gbapal"),
    INCBIN_U16("data/tilesets/primary/general/palettes/01.gbapal"),
    ...
};

The parser extracts the variable name, INCBIN macro name, and path(s) from each declaration.

Returns

A vector of IncbinDeclaration on success, or an error on failure

Definition at line 1796 of file parser.cpp.

parse_indexed_arrays()

ChainableResult< std::vector< IndexedArrayDeclaration > > porytiles::Parser::parse_indexed_arrays

(

)

Parses array declarations that use designated (indexed) initializers.

Scans for declarations of the form:

[static] [const] TYPE IDENTIFIER [SIZE_EXPR] = { [index1] = value1, [index2] = value2, ... };

Each [index] = value element becomes an IndexedArrayEntry. Value expressions are evaluated against the current symbol table (including any seeded symbols); an unevaluable value leaves the entry's value absent rather than failing the scan. Blank lines inside the braces and a missing trailing comma on the last entry are tolerated. Preprocessor conditionals are tracked as in parse_defines(), and arrays inside a provably inactive region are dropped.

Returns

A vector of IndexedArrayDeclaration on success, or an error on failure

Definition at line 2097 of file parser.cpp.

parse_pointer_arrays()

ChainableResult< std::vector< ArrayDeclaration > > porytiles::Parser::parse_pointer_arrays

(

)

Parses all pointer array declarations from the token stream.

Scans through the token stream looking for pointer array declarations matching the pattern:

[const] TYPE * [const] IDENTIFIER [] = { element1, element2, ... };

This is used by AnimCodeParser to extract animation frame arrays like:

const u16 *const gTilesetAnims_General_Flower[] = {
    gTilesetAnims_General_Flower_Frame0,
    gTilesetAnims_General_Flower_Frame1
};

The parser extracts the array name and all identifier elements from the initializer list.

Returns

A vector of ArrayDeclaration on success, or an error on failure

Definition at line 1234 of file parser.cpp.

parse_struct_definitions()

ChainableResult< std::vector< StructDefinition > > porytiles::Parser::parse_struct_definitions

(

)

Parses struct type definitions from the token stream.

Scans through the token stream looking for named struct definitions matching the pattern:

struct TYPE { members... } [;]

Each member body is scanned tolerantly: members matching the simple declarator shape [const] TYPE [*...] NAME [: WIDTH]; are captured (including struct TYPE members and bitfields, whose widths are consumed but not recorded), while anything else (pointer-to-array members like const u16 (*palettes)[16], array members, multiple declarators) is skipped without aborting the scan. Preprocessor directive lines inside the body are ignored. Forward declarations and anonymous typedef structs are not captured.

This is used to read the pointed-to type of struct Tileset::metatileAttributes from a project's include/global.fieldmap.h.

Returns

A vector of StructDefinition on success, or an error on failure

Definition at line 1652 of file parser.cpp.

parse_struct_initializers()

ChainableResult< std::vector< StructInitializerDeclaration > > porytiles::Parser::parse_struct_initializers

(

)

Parses struct variable declarations with their designated initializer fields.

Scans through the token stream looking for struct variable declarations matching the pattern:

[const] struct TYPE IDENTIFIER = { .field1 = value1, .field2 = value2, ... } [;]

This is used by ProjectTilesetMetadataProvider to extract tileset metadata from headers.h files:

const struct Tileset gTileset_General = {
    .isCompressed = TRUE,
    .isSecondary = FALSE,
    .tiles = gTilesetTiles_General,
    .palettes = gTilesetPalettes_General,
    .metatiles = gMetatiles_General,
    .metatileAttributes = gMetatileAttributes_General,
    .callback = InitTilesetAnim_General,
};

Unlike parse_struct_variables(), this method also parses the designated initializer fields, enabling extraction of field values like isSecondary and variable references.

Returns

A vector of StructInitializerDeclaration on success, or an error on failure

Definition at line 1500 of file parser.cpp.

parse_struct_variables()

ChainableResult< std::vector< StructVariableDeclaration > > porytiles::Parser::parse_struct_variables

(

)

Parses struct variable declarations from the token stream.

Scans through the token stream looking for struct variable declarations matching the pattern:

[const] struct TYPE IDENTIFIER = { ... } [;]

This is used by ProjectTilesetMetadataProvider to extract tileset names from headers.h files like:

const struct Tileset gTileset_General = {
    .isCompressed = TRUE,
    .isSecondary = FALSE,
    ...
};

The parser captures the struct type and variable name; the initializer body is skipped.

Returns

A vector of StructVariableDeclaration on success, or an error on failure

Definition at line 1420 of file parser.cpp.

scan_warnings()

const std::vector< std::string > & porytiles::Parser::scan_warnings ( ) const

inline

Returns warnings accumulated while scanning conditionals and defines.

Scans do not fail on recoverable oddities such as a value that conflicts with an earlier define inside an undecidable conditional region. Those are collected here so a caller can surface them without aborting the parse.

Returns

A const reference to the accumulated warning messages

Definition at line 314 of file parser.hpp.

seed_symbols()

void porytiles::Parser::seed_symbols ( const std::unordered_map< std::string, std::int64_t > &  symbols )

inline

Seeds the symbol table with externally known macro values before scanning.

Merges the given name-to-value pairs into both the value symbol table and the defined-name set. This lets a file resolve references to symbols declared in another file that was parsed earlier (for example seeding a source file with values from its header). Existing entries are overwritten.

Parameters

symbols

The name-to-value pairs to merge in

Definition at line 352 of file parser.hpp.

seed_values()

void porytiles::Parser::seed_values ( const std::unordered_map< std::string, std::int64_t > &  symbols )

inline

Seeds only expression values without marking their names as preprocessor-defined.

Same-file #defines can provide values for enum expressions, but pre-seeding their names would make a normal include guard appear already defined before the parser reaches its opening #ifndef. This method supports the expression lookup without changing #ifdef or defined(...) decisions.

Parameters

symbols

The name-to-value pairs to merge into the expression symbol table

Definition at line 368 of file parser.hpp.

---

The documentation for this class was generated from the following files:

• porytiles/include/porytiles/utilities/c_parser/parser.hpp
• porytiles/lib/utilities/c_parser/parser.cpp