Implement token.c with comprehensive tests and easy-to-modify token mapping

- Created token-to-string mapping array parallel to Token enum in token.c
- Implemented TokenStream with lookahead buffering for proper tokenization
- Implemented tokenstream_open/close/next functions with support for:
  - Keywords (module, import, void)
  - Symbols (parentheses, brackets, comma, semicolon)
  - Identifiers (alphanumeric starting with letter or underscore)
  - Comment skipping (// style)
  - Whitespace handling
- Added token_to_string function to token.h for token inspection
- Created comprehensive test suite (15 tests) covering all token types and edge cases
- All tests pass.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
2026-04-24 09:35:18 +02:00
parent c73f99d9e6
commit dccdcb8ba5
5 changed files with 323 additions and 1 deletions
+1 -1
View File
@@ -1,4 +1,4 @@
V0_SRC := v0/buffer.c v0/main.c
V0_SRC := v0/buffer.c v0/main.c v0/token.c
# V0_TEST must only include `v0/test.c` itself, as all other test Csource files are
# included directly into `v0/test.c` using `#include "test_xyz.c"`.