| --- |
| license: apache-2.0 |
| library_name: pytorch |
| tags: |
| - flow-matching |
| - language-modeling |
| - hyperspherical-flow |
| - s-flm |
| base_model: jdeschena/s-flm |
| --- |
| |
| <h1 align="center">BCT: Bayes-Consistent Tempering<br>for Flow Language Models</h1> |
|
|
| <p align="center"><a href="https://daioba.github.io/">Daisuke Oba</a>, Hiroki Furuta, Naoaki Okazaki</p> |
|
|
| <p align="center"> |
| <a href="https://github.com/daioba/bct"><img src="https://img.shields.io/badge/Code-GitHub-black?logo=github" alt="Code"></a> |
| <a href="https://daioba.github.io/bct/"><img src="https://img.shields.io/badge/Blog%20%20-8A2BE2" alt="Blog"></a> |
| <a href="https://arxiv.org/abs/2610.27641"><img src="https://img.shields.io/badge/arXiv-2610.27641-red.svg" alt="arXiv"></a> |
| </p> |
|
|
| Checkpoints trained for the paper *BCT: Bayes-Consistent Tempering for Flow Language Models*. |
|
|
| All models build on [S-FLM](https://huggingface.co/jdeschena/s-flm) (Language Modeling with Hyperspherical Flows). |
| Checkpoints that we use **without modification** (the released S-FLM, MDLM and Duo TinyGSM models) are **not** included |
| here; download them from [`jdeschena/s-flm`](https://huggingface.co/jdeschena/s-flm) (see below). |
|
|
| ## Files |
|
|
| | File | Task | Backbone | Model in the paper | Training | |
| |---|---|---|---|---| |
| | `tinygsm/sfm/sphere_arch_truncated_adaptive_no_renorm_cont.ckpt` | TinyGSM → GSM8K | S-arch | Cont. | released S-FLM + 2,000 updates, original objective | |
| | `tinygsm/sfm/sphere_arch_truncated_adaptive_no_renorm_tft.ckpt` | TinyGSM → GSM8K | S-arch | TFT | released S-FLM + 2,000 updates, tempered-flow training (temperature-conditioned) | |
| | `tinygsm/sfm/sphere_dit_truncated_adaptive_no_renorm_cont.ckpt` | TinyGSM → GSM8K | DiT | Cont. | released S-FLM + 2,000 updates, original objective | |
| | `tinygsm/sfm/sphere_dit_truncated_adaptive_no_renorm_tft.ckpt` | TinyGSM → GSM8K | DiT | TFT | released S-FLM + 2,000 updates, tempered-flow training (temperature-conditioned) | |
| | `sudoku_hard/sfm/sfm_truncated_adaptive.ckpt` | Sudoku-Hard | DiT (tiny) | S-FLM | trained from scratch with the released S-FLM Sudoku recipe, 20,000 steps (no Sudoku checkpoint is released) | |
| | `sudoku_hard/sfm/sfm_truncated_adaptive_cont.ckpt` | Sudoku-Hard | DiT (tiny) | Cont. | the model above + 2,000 updates, original objective | |
| | `sudoku_hard/sfm/sfm_truncated_adaptive_tft.ckpt` | Sudoku-Hard | DiT (tiny) | TFT | the model above + 2,000 updates, tempered-flow training (temperature-conditioned) | |
|
|
| File names follow those of the released S-FLM checkpoints, with a `_cont` / `_tft` suffix. |
| Cont. and TFT of the same backbone start from the same checkpoint and use the same number of updates, so that |
| "Cont. vs. TFT" isolates the training objective. TFT checkpoints contain additional temperature-conditioning |
| parameters and must be evaluated with `model.temp_conditioning=True`. |
|
|
| SHA-256: see the "Files and versions" tab (LFS object IDs). |
|
|
| ## Baselines (not included; download them yourself) |
|
|
| | Model | Location | |
| |---|---| |
| | S-FLM, S-arch (TinyGSM) | `jdeschena/s-flm` → `tinygsm/sfm/sphere_arch_truncated_adaptive_no_renorm.ckpt` | |
| | S-FLM, DiT (TinyGSM) | `jdeschena/s-flm` → `tinygsm/sfm/sphere_dit_truncated_adaptive_no_renorm.ckpt` | |
| | MDLM (TinyGSM) | `jdeschena/s-flm` → `tinygsm/mdlm.ckpt` | |
| | Duo (TinyGSM) | `jdeschena/s-flm` → `tinygsm/duo.ckpt` | |
|
|
| ```python |
| from huggingface_hub import hf_hub_download |
| |
| # ours |
| ckpt = hf_hub_download("daisuk30ba/BCT", "tinygsm/sfm/sphere_arch_truncated_adaptive_no_renorm_tft.ckpt") |
| # baseline (released S-FLM) |
| base = hf_hub_download("jdeschena/s-flm", "tinygsm/sfm/sphere_arch_truncated_adaptive_no_renorm.ckpt") |
| ``` |
|
|
| ## Format |
|
|
| Full PyTorch Lightning checkpoints in the same format as the released S-FLM checkpoints |
| (model weights, EMA weights, optimizer and LR-scheduler state, adaptive noise-schedule state, and the training config), |
| so they can be evaluated with the same loader and training can be resumed. |
| They contain pickled Python objects (config and tokenizer); load them only with code you trust (`weights_only=False`). |
| Evaluation uses the EMA weights, as in S-FLM. |
|
|
| Machine-specific paths stored in the checkpoints were replaced with `/path/to/bct/...`, and the stored training |
| config uses the option names of the released code. Strings inherited from the released S-FLM checkpoints were |
| left unchanged. Weights and all other tensors are bit-identical to the checkpoints used in the paper. |
|
|
| ## Training data |
|
|
| - TinyGSM ([TinyGSM/TinyGSM](https://huggingface.co/datasets/TinyGSM/TinyGSM), MIT), with the S-FLM preprocessing. |
| - Sudoku-Hard puzzles generated with the S-FLM Sudoku generator (official seed and split). |
|
|
| ## License |
|
|
| Apache-2.0. The models are derived from S-FLM (Apache-2.0). |
|
|
| ## Citation |
|
|
| ```bibtex |
| @article{oba2026bct, |
| title={BCT: Bayes-Consistent Tempering for Flow Language Models}, |
| author={Oba, Daisuke and Furuta, Hiroki and Okazaki, Naoaki}, |
| journal={arXiv preprint arXiv:2610.27641}, |
| year={2026} |
| } |
| ``` |
|
|
