| --- |
| library_name: transformers |
| tags: |
| - halcyon |
| - radical-numerics |
| - tokenizer |
| - text-to-image |
| - fsq |
| --- |
| |
| # Halcyon-Tokenizers |
|
|
| Text and image tokenizers for the Halcyon models in this collection, from |
| *Text-to-Image Generative Modeling without Language Pretraining*. Both were trained from scratch on GPIC. |
|
|
| | Path | What | |
| |---|---| |
| | `tokenizer.json`, `tokenizer_config.json` | text byte-level BPE, 16,384 ids, loads with `AutoTokenizer` | |
| | `image_tokenizer/di8x8_242172_ema.pt` | image tokenizer weights (PyTorch state dict, 521 MiB) | |
| | `vocab_layout.json` | joint vocabulary layout and special ids of the Halcyon LMs | |
| | `bpe16k_final.report.json` | training report of the text BPE | |
|
|
| ## Text tokenizer |
|
|
| ```python |
| from transformers import AutoTokenizer |
| |
| tok = AutoTokenizer.from_pretrained("radicalnumerics/Halcyon-Tokenizers") |
| ids = tok("A woman carries a child on her back.").input_ids # ids < 16,384 |
| ``` |
|
|
| Byte-level BPE with 16,384 ids and no special tokens, trained on GPIC captions |
| (21.3 GB) plus OpenWebText (11.9 GB). On GPIC test captions: 4.774 bytes per token, |
| mean 44.5 tokens per caption (p50 36, p99 182). Captions are capped at 254 |
| tokens, which is the `model_max_length`. |
|
|
| ## Image tokenizer |
|
|
| Cosmos-style encoder-decoder with finite scalar quantization, trained on GPIC at |
| 256x256: 8x spatial downsampling, 32x32 = 1,024 codes per image in |
| raster order, codebook 64,000 (FSQ levels 8, 8, 8, 5, 5, 5), 137M |
| parameters, decoder width 192. Trained for one epoch with reconstruction and LPIPS |
| losses, then half an epoch with a PatchGAN adversarial loss; 242,172 steps, EMA |
| weights. Reconstruction on GPIC Test-50K: rFD 18.86, PSNR 25.61, SSIM 0.792, LPIPS 0.062. |
|
|
| The file is a plain state dict (205 tensors, standard torch storages only); the |
| tokenizer code is not part of this repo. Input: RGB resized to 256x256 (bicubic), |
| scaled to [-1, 1]. Output: 1,024 codes in [0, 63999]. |
|
|
| ## Token ids and sequence format |
|
|
| | Ids | Meaning | |
| |---|---| |
| | 0 to 16383 | text BPE ids | |
| | 16384 to 80383 | image codes, id = 16384 + FSQ code | |
| | 80384 | `[SOV]`, begin of image | |
| | 80385 | `[EOV]`, end of image | |
| | 80386 | `[PAD]` | |
| | 80387 | `[EOS]` | |
| | 80388 to 80639 | reserved, unused | |
|
|
| A text-to-image sequence is caption ids, `[SOV]`, 1,024 image ids, `[EOV]`, `[EOS]`, |
| then `[PAD]` to the 1,280-token context. At inference the prompt is caption ids followed |
| by `[SOV]`; the model emits the 1,024 image ids. The same layout is in `vocab_layout.json`. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @article{chandrasegaran2026halcyon, |
| title = {Text-to-Image Generative Modeling without Language Pretraining}, |
| author = {Chandrasegaran, Keshigeyan and Zhao, Yue and Thomas, Armin W. and Adeli, Ehsan and Wu, Jiajun and Niebles, Juan Carlos and Fei-Fei, Li and Poli, Michael}, |
| year = {2026}, |
| note = {Preprint} |
| } |
| ``` |
|
|
