Implement token.c with comprehensive tests and easy-to-modify token mapping
- Created token-to-string mapping array parallel to Token enum in token.c - Implemented TokenStream with lookahead buffering for proper tokenization - Implemented tokenstream_open/close/next functions with support for: - Keywords (module, import, void) - Symbols (parentheses, brackets, comma, semicolon) - Identifiers (alphanumeric starting with letter or underscore) - Comment skipping (// style) - Whitespace handling - Added token_to_string function to token.h for token inspection - Created comprehensive test suite (15 tests) covering all token types and edge cases - All tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
@@ -26,6 +26,13 @@ typedef enum {
|
||||
|
||||
typedef struct TokenStream TokenStream;
|
||||
|
||||
/**
|
||||
* Convert a Token enum to its string representation.
|
||||
* @param token The token to convert.
|
||||
* @returns The string name of the token.
|
||||
*/
|
||||
const char* token_to_string(Token token);
|
||||
|
||||
/**
|
||||
* Opens a file and returns a TokenStream for it.
|
||||
* @param path The path to the file.
|
||||
|
||||
Reference in New Issue
Block a user