CRAYON-tokenizer / RELEASE_NOTES_4.3.0.md
Phase-Technologies's picture
Upload folder using huggingface_hub
708f4a3 verified

XERV CRAYON v4.3.0 Release Notes

Release Date: February 1, 2026


πŸš€ Critical Fix: ROCm/HIP Compilation

This release fixes a critical build failure on AMD ROCm systems that prevented installation on machines with AMD GPUs.

Root Cause

The previous build system used g++ to compile the ROCm engine, but the HIP kernel code (__global__, blockIdx, threadIdx, hipLaunchKernelGGL) requires the hipcc compiler.

Python 3.13's stricter setuptools ignored previous compiler override hacks, causing the build to fail with errors like:

error: 'blockIdx' was not declared in this scope
error: 'hipLaunchKernelGGL' was not declared in this scope

Solution

  1. Renamed rocm_engine.cpp β†’ rocm_engine.hip: The .hip extension is the proper file type for HIP source files.

  2. Custom CrayonBuildExt class: A new build extension that explicitly invokes hipcc for .hip files, bypassing setuptools' default compiler selection.

  3. Production-grade error handling: All HIP API calls now use proper error checking with hipGetErrorString().

  4. Fixed license format: Updated pyproject.toml to use SPDX license expression (plain string) instead of deprecated table format.


πŸ“¦ What's New

Build System

  • βœ… ROCm/HIP now compiles correctly with hipcc
  • βœ… Python 3.10-3.13 fully supported
  • βœ… Fixed deprecation warnings in pyproject.toml
  • βœ… Clean source distribution with .hip files included

ROCm Engine (rocm_engine.hip)

  • βœ… Proper device initialization with context creation
  • βœ… Memory cleanup in module destructor
  • βœ… Bounds checking in kernels to prevent invalid memory access
  • βœ… Consistent API with CUDA engine (returns tuple with metadata)

Colab Notebook

  • βœ… Updated to v4.3.0
  • βœ… Added hipcc detection for ROCm environments
  • βœ… Better output formatting with Quick Start guide

πŸ”§ Installation

From PyPI (Recommended)

pip install xerv-crayon

From Source (For Development)

git clone https://github.com/Electroiscoding/CRAYON.git
cd CRAYON
pip install -v .

Force Build for Specific Backend

# CPU only (skip all GPU backends)
CRAYON_FORCE_CPU=1 pip install xerv-crayon

# Force ROCm environment variables
ROCM_HOME=/opt/rocm pip install xerv-crayon

πŸ“Š Supported Backends

Backend Hardware Compiler Status
CPU x86_64 (AVX2/512) g++/MSVC βœ… Always built
CUDA NVIDIA GPU (SM 7.0+) nvcc (PyTorch) βœ… Auto-detected
ROCm AMD GPU (gfx9+) hipcc βœ… Fixed in v4.3.0

πŸ§ͺ Verification

After installation, verify your backends:

import crayon

print(f"Version: {crayon.get_version()}")
print(f"Backends: {crayon.check_backends()}")

# Quick test
vocab = crayon.CrayonVocab(device="auto")
vocab.load_profile("lite")
tokens = vocab.tokenize("Hello, world!")
print(f"Tokens: {tokens}")

πŸ› Bug Fixes

  • Fixed: ROCm engine compilation failure on Python 3.13+
  • Fixed: hipLaunchKernelGGL not found error
  • Fixed: License classifier deprecation warnings
  • Fixed: Uninitialized variable warnings in CPU engine

πŸ“– Full Changelog

See CHANGELOG.md for complete history.


πŸ™ Credits

Thanks to the community for reporting the ROCm build issue and providing detailed error logs that helped identify the root cause.