Since buffer_open_string() always succeeds and tokenstream_open() always
succeeds when given a valid buffer, the NULL checks are unnecessary.
This simplifies the test code and makes it more readable.
All 12 tests still pass.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Replaced all write_test_file() calls with buffer_open_string()
- Removed file I/O overhead - tests now run entirely in memory
- Removed unnecessary #include <stdio.h>, <stdlib.h>, <string.h> from test_token.c
- Simplified test_tokenstream_open_fail() to directly test NULL buffer handling
- Tests are faster and cleaner without temporary file creation
- No test output changes - all 12 tests still pass
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Created KeywordMap structure with keyword-to-token mapping at top of token.c
- Added lookup_keyword() function to check if identifier is a keyword
- Replaced 3 strcmp calls (lines 99-101) with single lookup_keyword() call
- Removed token_to_string() function and its tests (3 tests removed)
- Single easy-to-read and modify keyword map serves both documentation and implementation
- Added new keywords by editing the keywords[] array at top of token.c
All 12 tests passing (removed token_to_string tests which are now unnecessary).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The tokenstream_open function now takes a Buffer* parameter instead of a file
path string, making the API more flexible and allowing the caller to manage
buffer lifetime. The tokenstream_close function continues to close the underlying
buffer as documented.
- Changed tokenstream_open signature from (const char* path) to (Buffer* buffer)
- Updated implementation to accept and use the provided buffer directly
- Updated all tests to open buffers separately and pass them to tokenstream_open
- Added #include "buffer.h" to token.h for Buffer type definition
- All 15 tests pass
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Created token-to-string mapping array parallel to Token enum in token.c
- Implemented TokenStream with lookahead buffering for proper tokenization
- Implemented tokenstream_open/close/next functions with support for:
- Keywords (module, import, void)
- Symbols (parentheses, brackets, comma, semicolon)
- Identifiers (alphanumeric starting with letter or underscore)
- Comment skipping (// style)
- Whitespace handling
- Added token_to_string function to token.h for token inspection
- Created comprehensive test suite (15 tests) covering all token types and edge cases
- All tests pass.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
seeseemelk