amartomar683 commited on
Commit
e99098b
路
verified 路
1 Parent(s): b18482b

Credit the NCM paper (Kumbong, Khare et al., arXiv:2610.14162)

Browse files
Files changed (1) hide show
  1. README.md +34 -0
README.md CHANGED
@@ -23,6 +23,8 @@ datasets:
23
  - Salesforce/xlam-function-calling-60k
24
  ---
25
 
 
 
26
  **Files:** `tiny-ncm.pt` (weights + config, 146 MB), `bpe.json` (text tokenizer), `registry.json` (tool list).
27
  Run it with the code at https://github.com/amar683/tiny-ncm:
28
 
@@ -37,11 +39,43 @@ hf download amartomar683/tiny-ncm tiny-ncm.pt --local-dir weights
37
 
38
  ## Summary
39
  - **Type:** NCM (coarse-to-fine, block-wise) language model for tool use, trained from scratch.
 
40
  - **Size:** 36.4M parameters: generator 20.9M (d=384, 8 layers, 3 query heads / 1 KV head) and multiscale residual VQ
41
  tokenizer 15.5M (5 levels, 1024-entry codebooks). Byte-level BPE vocabulary of 16,384. Context 4096.
42
  - **Checkpoint:** `runs/task6` (released as `tiny-ncm.pt`, fp32, 146 MB). Code: https://github.com/amar683/tiny-ncm (MIT).
43
  - **Hardware:** one MacBook (Apple M5, 24 GB), PyTorch MPS, bf16. About 12 hours of training in total (VQ tokenizer 1.3 h, generator 10.5 h).
44
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
45
  ## Intended use
46
  A personal assistant that turns everyday requests into tool calls (alarms, reminders, timers, calendar, read-only Gmail,
47
  web and Wikipedia lookups, weather, notes, calculator) and makes small talk. Use it together with the app in
 
23
  - Salesforce/xlam-function-calling-60k
24
  ---
25
 
26
+ An independent, from-scratch implementation of **Next-Concept Modeling (NCM)**, Kumbong, Khare et al., *Next-Concept Modeling: Fast, Parallel Coarse-to-Fine Text Generation*, 2026 ([arXiv:2610.14162](https://arxiv.org/abs/2610.14162), official code [HazyResearch/ncm](https://github.com/HazyResearch/ncm)). All credit for the method goes to the authors.
27
+
28
  **Files:** `tiny-ncm.pt` (weights + config, 146 MB), `bpe.json` (text tokenizer), `registry.json` (tool list).
29
  Run it with the code at https://github.com/amar683/tiny-ncm:
30
 
 
39
 
40
  ## Summary
41
  - **Type:** NCM (coarse-to-fine, block-wise) language model for tool use, trained from scratch.
42
+ - **Method:** Next-Concept Modeling, Kumbong, Khare et al., 2026 ([arXiv:2610.14162](https://arxiv.org/abs/2610.14162)); see below.
43
  - **Size:** 36.4M parameters: generator 20.9M (d=384, 8 layers, 3 query heads / 1 KV head) and multiscale residual VQ
44
  tokenizer 15.5M (5 levels, 1024-entry codebooks). Byte-level BPE vocabulary of 16,384. Context 4096.
45
  - **Checkpoint:** `runs/task6` (released as `tiny-ncm.pt`, fp32, 146 MB). Code: https://github.com/amar683/tiny-ncm (MIT).
46
  - **Hardware:** one MacBook (Apple M5, 24 GB), PyTorch MPS, bf16. About 12 hours of training in total (VQ tokenizer 1.3 h, generator 10.5 h).
47
 
48
+ ## Method: the NCM paper
49
+
50
+ This model is an independent, from-scratch implementation of **Next-Concept Modeling (NCM)** from
51
+
52
+ > **Next-Concept Modeling: Fast, Parallel Coarse-to-Fine Text Generation**
53
+ > Hermann Kumbong, Ishan S. Khare, Iddah Mlauzi, Andrew Woen, Charles Ding, E. Kelly Buchanan, Simran Arora, Azalia Mirhoseini, Christopher R茅
54
+ > Stanford University and Together AI, 2026. [arXiv:2610.14162](https://arxiv.org/abs/2610.14162) 路 official code: [HazyResearch/ncm](https://github.com/HazyResearch/ncm)
55
+
56
+ All credit for the method goes to the authors. This project is not affiliated with them, and no code was taken from the official repository.
57
+
58
+ **From the paper:** a multiscale residual VQ-VAE turns text into coarse-to-fine levels of discrete codes; a transformer generates
59
+ one level at a time, predicting all codes of a level in parallel; nested dropout in the tokenizer; conditioning (code) corruption
60
+ and per-level loss weighting when training the generator.
61
+
62
+ **Changed or added here:** text is split into 32-token blocks that are generated one after another with a KV cache (each block
63
+ coarse-to-fine over 5 levels), so a chat can grow to 4096 tokens; level positions attend to a 128-token window of context to fit
64
+ Apple-silicon speed; Muon optimizer; a small size (36M) trained in about 12 hours on one MacBook; chat and tool-use training,
65
+ plus the app-side checks in `assistant/`.
66
+
67
+ If you use this, please cite the paper:
68
+
69
+ ```bibtex
70
+ @article{kumbong2026ncm,
71
+ title = {Next-Concept Modeling: Fast, Parallel Coarse-to-Fine Text Generation},
72
+ author = {Kumbong, Hermann and Khare, Ishan S. and Mlauzi, Iddah and Woen, Andrew and Ding, Charles and
73
+ Buchanan, E. Kelly and Arora, Simran and Mirhoseini, Azalia and R{\'e}, Christopher},
74
+ journal = {arXiv preprint arXiv:2610.14162},
75
+ year = {2026}
76
+ }
77
+ ```
78
+
79
  ## Intended use
80
  A personal assistant that turns everyday requests into tool calls (alarms, reminders, timers, calendar, read-only Gmail,
81
  web and Wikipedia lookups, weather, notes, calculator) and makes small talk. Use it together with the app in