Credit the NCM paper (Kumbong, Khare et al., arXiv:2610.14162)
Browse files
README.md
CHANGED
|
@@ -23,6 +23,8 @@ datasets:
|
|
| 23 |
- Salesforce/xlam-function-calling-60k
|
| 24 |
---
|
| 25 |
|
|
|
|
|
|
|
| 26 |
**Files:** `tiny-ncm.pt` (weights + config, 146 MB), `bpe.json` (text tokenizer), `registry.json` (tool list).
|
| 27 |
Run it with the code at https://github.com/amar683/tiny-ncm:
|
| 28 |
|
|
@@ -37,11 +39,43 @@ hf download amartomar683/tiny-ncm tiny-ncm.pt --local-dir weights
|
|
| 37 |
|
| 38 |
## Summary
|
| 39 |
- **Type:** NCM (coarse-to-fine, block-wise) language model for tool use, trained from scratch.
|
|
|
|
| 40 |
- **Size:** 36.4M parameters: generator 20.9M (d=384, 8 layers, 3 query heads / 1 KV head) and multiscale residual VQ
|
| 41 |
tokenizer 15.5M (5 levels, 1024-entry codebooks). Byte-level BPE vocabulary of 16,384. Context 4096.
|
| 42 |
- **Checkpoint:** `runs/task6` (released as `tiny-ncm.pt`, fp32, 146 MB). Code: https://github.com/amar683/tiny-ncm (MIT).
|
| 43 |
- **Hardware:** one MacBook (Apple M5, 24 GB), PyTorch MPS, bf16. About 12 hours of training in total (VQ tokenizer 1.3 h, generator 10.5 h).
|
| 44 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 45 |
## Intended use
|
| 46 |
A personal assistant that turns everyday requests into tool calls (alarms, reminders, timers, calendar, read-only Gmail,
|
| 47 |
web and Wikipedia lookups, weather, notes, calculator) and makes small talk. Use it together with the app in
|
|
|
|
| 23 |
- Salesforce/xlam-function-calling-60k
|
| 24 |
---
|
| 25 |
|
| 26 |
+
An independent, from-scratch implementation of **Next-Concept Modeling (NCM)**, Kumbong, Khare et al., *Next-Concept Modeling: Fast, Parallel Coarse-to-Fine Text Generation*, 2026 ([arXiv:2610.14162](https://arxiv.org/abs/2610.14162), official code [HazyResearch/ncm](https://github.com/HazyResearch/ncm)). All credit for the method goes to the authors.
|
| 27 |
+
|
| 28 |
**Files:** `tiny-ncm.pt` (weights + config, 146 MB), `bpe.json` (text tokenizer), `registry.json` (tool list).
|
| 29 |
Run it with the code at https://github.com/amar683/tiny-ncm:
|
| 30 |
|
|
|
|
| 39 |
|
| 40 |
## Summary
|
| 41 |
- **Type:** NCM (coarse-to-fine, block-wise) language model for tool use, trained from scratch.
|
| 42 |
+
- **Method:** Next-Concept Modeling, Kumbong, Khare et al., 2026 ([arXiv:2610.14162](https://arxiv.org/abs/2610.14162)); see below.
|
| 43 |
- **Size:** 36.4M parameters: generator 20.9M (d=384, 8 layers, 3 query heads / 1 KV head) and multiscale residual VQ
|
| 44 |
tokenizer 15.5M (5 levels, 1024-entry codebooks). Byte-level BPE vocabulary of 16,384. Context 4096.
|
| 45 |
- **Checkpoint:** `runs/task6` (released as `tiny-ncm.pt`, fp32, 146 MB). Code: https://github.com/amar683/tiny-ncm (MIT).
|
| 46 |
- **Hardware:** one MacBook (Apple M5, 24 GB), PyTorch MPS, bf16. About 12 hours of training in total (VQ tokenizer 1.3 h, generator 10.5 h).
|
| 47 |
|
| 48 |
+
## Method: the NCM paper
|
| 49 |
+
|
| 50 |
+
This model is an independent, from-scratch implementation of **Next-Concept Modeling (NCM)** from
|
| 51 |
+
|
| 52 |
+
> **Next-Concept Modeling: Fast, Parallel Coarse-to-Fine Text Generation**
|
| 53 |
+
> Hermann Kumbong, Ishan S. Khare, Iddah Mlauzi, Andrew Woen, Charles Ding, E. Kelly Buchanan, Simran Arora, Azalia Mirhoseini, Christopher R茅
|
| 54 |
+
> Stanford University and Together AI, 2026. [arXiv:2610.14162](https://arxiv.org/abs/2610.14162) 路 official code: [HazyResearch/ncm](https://github.com/HazyResearch/ncm)
|
| 55 |
+
|
| 56 |
+
All credit for the method goes to the authors. This project is not affiliated with them, and no code was taken from the official repository.
|
| 57 |
+
|
| 58 |
+
**From the paper:** a multiscale residual VQ-VAE turns text into coarse-to-fine levels of discrete codes; a transformer generates
|
| 59 |
+
one level at a time, predicting all codes of a level in parallel; nested dropout in the tokenizer; conditioning (code) corruption
|
| 60 |
+
and per-level loss weighting when training the generator.
|
| 61 |
+
|
| 62 |
+
**Changed or added here:** text is split into 32-token blocks that are generated one after another with a KV cache (each block
|
| 63 |
+
coarse-to-fine over 5 levels), so a chat can grow to 4096 tokens; level positions attend to a 128-token window of context to fit
|
| 64 |
+
Apple-silicon speed; Muon optimizer; a small size (36M) trained in about 12 hours on one MacBook; chat and tool-use training,
|
| 65 |
+
plus the app-side checks in `assistant/`.
|
| 66 |
+
|
| 67 |
+
If you use this, please cite the paper:
|
| 68 |
+
|
| 69 |
+
```bibtex
|
| 70 |
+
@article{kumbong2026ncm,
|
| 71 |
+
title = {Next-Concept Modeling: Fast, Parallel Coarse-to-Fine Text Generation},
|
| 72 |
+
author = {Kumbong, Hermann and Khare, Ishan S. and Mlauzi, Iddah and Woen, Andrew and Ding, Charles and
|
| 73 |
+
Buchanan, E. Kelly and Arora, Simran and Mirhoseini, Azalia and R{\'e}, Christopher},
|
| 74 |
+
journal = {arXiv preprint arXiv:2610.14162},
|
| 75 |
+
year = {2026}
|
| 76 |
+
}
|
| 77 |
+
```
|
| 78 |
+
|
| 79 |
## Intended use
|
| 80 |
A personal assistant that turns everyday requests into tool calls (alarms, reminders, timers, calendar, read-only Gmail,
|
| 81 |
web and Wikipedia lookups, weather, notes, calculator) and makes small talk. Use it together with the app in
|