| # Changelog |
|
|
| All notable changes to XERV Crayon will be documented in this file. |
|
|
| The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), |
| and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). |
|
|
| ## [2.0.0] - 2026-01-23 |
|
|
| ### Added |
| - **Double-Array Trie (DAT) Engine**: Complete rewrite of the tokenization engine using memory-mapped DAT for O(1) lookups |
| - **AVX2/SIMD Optimizations**: Native C++ engine with AVX2 intrinsics achieving >16M tokens/second |
| - **Pre-built Vocabulary Profiles**: 5 production-ready profiles (lite, code, science, multilingual, arts_commerce) |
| - **CLI Tool**: `crayon-benchmark` command for easy performance testing |
| - **Zero-Copy Memory Mapping**: Memory-mapped DAT files for instant loading |
| - **Cross-Platform Support**: Windows (MSVC), Linux (GCC), macOS (Clang/Apple Silicon) |
| |
| ### Changed |
| - Version bump from 1.1.0 to 2.0.0 |
| - Minimum Python version updated to 3.10 |
| - Package structure reorganized for better modularity |
| |
| ### Performance |
| - Tokenization: 16M+ tokens/second (up from 2M in v1.x) |
| - Memory usage: 50% reduction via mmap |
| - Load time: <10ms for vocabulary profiles |
| |
| ## [1.1.0] - 2026-01-16 |
| |
| ### Added |
| - Initial C-Trie implementation |
| - SIMD-accelerated text processing |
| - Basic vocabulary management |
| |
| ### Fixed |
| - Memory leaks in trie traversal |
| - Unicode handling edge cases |
| |
| ## [1.0.0] - 2026-01-11 |
| |
| ### Added |
| - Initial release |
| - Pure Python tokenizer |
| - Basic vocabulary training |
| - Entropy-guided vocabulary construction |
| |
