Add pipeline tag (#1)
Browse files- Add pipeline tag (77b08529e7c1d6b385105a2a3727583fbc3e3710)
Co-authored-by: Niels Rogge <nielsr@users.noreply.huggingface.co>
README.md
CHANGED
|
@@ -1,143 +1,154 @@
|
|
| 1 |
-
---
|
| 2 |
-
|
| 3 |
-
|
| 4 |
-
|
| 5 |
-
language:
|
| 6 |
-
|
| 7 |
-
|
| 8 |
-
|
| 9 |
-
|
| 10 |
-
|
| 11 |
-
|
| 12 |
-
|
| 13 |
-
|
| 14 |
-
|
| 15 |
-
|
| 16 |
-
|
| 17 |
-
|
| 18 |
-
|
| 19 |
-
|
| 20 |
-
|
| 21 |
-
|
| 22 |
-
|
| 23 |
-
|
| 24 |
-
|
| 25 |
-
|
| 26 |
-
|
| 27 |
-
|
| 28 |
-
|
| 29 |
-
|
| 30 |
-
|
| 31 |
-
|
| 32 |
-
|
| 33 |
-
| |
|
| 34 |
-
|
|
| 35 |
-
|
| 36 |
-
|
| 37 |
-
|
| 38 |
-
|
| 39 |
-
|
| 40 |
-
|
| 41 |
-
|
| 42 |
-
|
| 43 |
-
|
| 44 |
-
|
| 45 |
-
|
| 46 |
-
|
| 47 |
-
|
| 48 |
-
|
| 49 |
-
|
| 50 |
-
|
| 51 |
-
|
| 52 |
-
|
| 53 |
-
|
| 54 |
-
|
| 55 |
-
|
| 56 |
-
|
| 57 |
-
|
| 58 |
-
|
| 59 |
-
|
| 60 |
-
|
| 61 |
-
|
| 62 |
-
|
| 63 |
-
|
| 64 |
-
|
| 65 |
-
|
| 66 |
-
|
| 67 |
-
|
| 68 |
-
|
| 69 |
-
|
| 70 |
-
|
| 71 |
-
|
| 72 |
-
|
| 73 |
-
|
| 74 |
-
|
| 75 |
-
|
| 76 |
-
|
| 77 |
-
|
| 78 |
-
|
| 79 |
-
|
| 80 |
-
|
| 81 |
-
|
| 82 |
-
|
| 83 |
-
|
| 84 |
-
|
| 85 |
-
|
| 86 |
-
|
| 87 |
-
|
| 88 |
-
|
| 89 |
-
|
| 90 |
-
|
| 91 |
-
|
| 92 |
-
|
| 93 |
-
|
| 94 |
-
|
| 95 |
-
|
| 96 |
-
|
| 97 |
-
|
| 98 |
-
|
| 99 |
-
|
| 100 |
-
|
| 101 |
-
|
| 102 |
-
|
| 103 |
-
|
| 104 |
-
|
| 105 |
-
|
| 106 |
-
|
| 107 |
-
|
| 108 |
-
|
| 109 |
-
|
| 110 |
-
|
| 111 |
-
|
| 112 |
-
|
| 113 |
-
|
| 114 |
-
|
| 115 |
-
|
| 116 |
-
##
|
| 117 |
-
|
| 118 |
-
|
| 119 |
-
|
| 120 |
-
|
| 121 |
-
|
| 122 |
-
|
| 123 |
-
|
| 124 |
-
|
| 125 |
-
|
| 126 |
-
|
| 127 |
-
|
| 128 |
-
|
| 129 |
-
|
| 130 |
-
|
| 131 |
-
|
| 132 |
-
|
| 133 |
-
|
| 134 |
-
|
| 135 |
-
|
| 136 |
-
|
| 137 |
-
|
| 138 |
-
|
| 139 |
-
|
| 140 |
-
|
| 141 |
-
```
|
| 142 |
-
|
| 143 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
base_model: google/gemma-3-270m
|
| 3 |
+
datasets:
|
| 4 |
+
- HuggingFaceFW/fineweb-edu
|
| 5 |
+
language:
|
| 6 |
+
- en
|
| 7 |
+
license: other
|
| 8 |
+
license_name: qlni-non-commercial-research
|
| 9 |
+
license_link: LICENSE.md
|
| 10 |
+
pipeline_tag: text-generation
|
| 11 |
+
tags:
|
| 12 |
+
- frozen-embedding
|
| 13 |
+
- binary-codes
|
| 14 |
+
- ternary
|
| 15 |
+
- bitnet
|
| 16 |
+
- gemma-3
|
| 17 |
+
- small-language-model
|
| 18 |
+
- cpu-inference
|
| 19 |
+
- research
|
| 20 |
+
---
|
| 21 |
+
|
| 22 |
+
# Embeddings Are Overpriced?
|
| 23 |
+
|
| 24 |
+
Code, data, checkpoints and every evaluation answer for the paper.
|
| 25 |
+
|
| 26 |
+
**Paper:** [paper.pdf](https://huggingface.co/QLNI/embeddings-are-overpriced/resolve/main/docs/paper.pdf) (LaTeX source: [paper.tex](https://huggingface.co/QLNI/embeddings-are-overpriced/blob/main/docs/paper.tex)). Preprint DOI: [10.5281/zenodo.23382634](https://doi.org/10.5281/zenodo.23382634). arXiv: [2610.22428](https://arxiv.org/abs/2610.22428) (cs.CL).
|
| 27 |
+
|
| 28 |
+
We trained three Gemma-3-270M-shaped language models from scratch on the same 2.05 billion FineWeb-Edu tokens, in the same order, with the same seed. They differ only in the vocabulary and the body.
|
| 29 |
+
|
| 30 |
+
| arm | vocabulary | body | trainable | file |
|
| 31 |
+
|---|---|---|---|---|
|
| 32 |
+
| **A** | learned tied table, 262,144 x 640 | float | 268.1M | 543 MB (GGUF f16) |
|
| 33 |
+
| **B** | **frozen 512-bit codes**, 16.8 MB, no trainable parameters | float | 101.0M | no kernel |
|
| 34 |
+
| **C** | frozen 512-bit codes | ternary (BitNet b1.58) | 101.2M | **46 MB** (our kernel) |
|
| 35 |
+
|
| 36 |
+
The frozen table gives each token one 512-bit code. We computed it once from Gemma 3 270M's embedding (the top 512 principal directions, then the sign of each), and the same codes serve the input and the output. We add a fixed per-token prior to the output scores: the log unigram frequency of each token in the training data, which we call the seal. Two small linear maps, 512 to 640 on the way in and 640 to 512 on the way out, connect the codes to the body.
|
| 37 |
+
|
| 38 |
+
Everything the paper used is here, so any number in it can be checked: the training script with every setting, the table, the data with checksums, all checkpoints (every 800 steps, with optimiser state), the per-step logs, the evaluation scripts, every per-question result, and the CPU kernel.
|
| 39 |
+
|
| 40 |
+
## Results
|
| 41 |
+
|
| 42 |
+
Zero-shot accuracy in %, with the 95% bootstrap range over questions in brackets. Validation loss is in nats per token on 512 held-out sequences.
|
| 43 |
+
|
| 44 |
+
| | LAMBADA | HellaSwag | ARC-Easy | PIQA | val loss |
|
| 45 |
+
|---|---|---|---|---|---|
|
| 46 |
+
| A learned table, float | 21.6 (20.5-22.7) | 30.4 (29.4-31.2) | 45.9 (43.9-47.9) | 60.6 (58.3-62.8) | 3.155 |
|
| 47 |
+
| B frozen table, float | 23.2 (22.1-24.4) | 30.4 (29.4-31.2) | 42.5 (40.5-44.5) | 61.5 (59.2-63.8) | 3.358 |
|
| 48 |
+
| C frozen table, ternary | 22.5 (21.3-23.6) | 29.4 (28.6-30.3) | 41.0 (39.0-43.0) | 59.5 (57.2-61.7) | 3.439 |
|
| 49 |
+
|
| 50 |
+
Compared question by question, B beats A on LAMBADA by 1.6 points, ties on HellaSwag and PIQA, and loses ARC-Easy by 3.4. C ties A on LAMBADA and PIQA and loses HellaSwag by 0.9 and ARC-Easy by 4.9. So the learned table costs 2.6 times the trainable parameters and buys 0.20 nats of loss and one benchmark.
|
| 51 |
+
|
| 52 |
+
On words seen 10 to 99 times in training, B and C are 1.7 to 2.3 nats better than A. On words seen a million times or more, A is ahead by 0.035 nats. Of Gemma's 262,144 tokens, 52,782 never appear in our training data. In A their rows stay random; in B and C they still carry Gemma's codes.
|
| 53 |
+
|
| 54 |
+
We also added 10,000 biomedical words to the vocabulary after training, without any further training. A got 42.9% worse, B 3.3% and C 3.1% (nats per character). The tests where the frozen table did not help are reported as well, in `eval/CONTROL_TOKENS.md` and `eval/emoji_*.json`.
|
| 55 |
+
|
| 56 |
+
### On a laptop CPU
|
| 57 |
+
|
| 58 |
+
Intel Xeon W-11955M, 8 threads, greedy decoding, 128-token context, 16-bit KV cache for both. Arm A runs in llama.cpp b11515 from the GGUF files in `gguf/`. Arm C runs in `kernel/cgemma.c`.
|
| 59 |
+
|
| 60 |
+
| model | runtime | file | peak RAM | tokens/s | perplexity (300k val tokens, ctx 512) |
|
| 61 |
+
|---|---|---|---|---|---|
|
| 62 |
+
| A | llama.cpp, GGUF f16 | 543 MB | 723 MB | 62 | 23.86 |
|
| 63 |
+
| A | llama.cpp, GGUF Q8_0 | 292 MB | 471 MB | 116 | 23.86 |
|
| 64 |
+
| A | llama.cpp, GGUF Q4_K_M | 253 MB | 446 MB | 130 | 23.97 |
|
| 65 |
+
| **C** | **our kernel** | **46 MB** | **62 MB** | **615 to 740** | 31.2 |
|
| 66 |
+
|
| 67 |
+
The 8-bit GGUF loses nothing against f16, so we compare C with A at Q8_0. C generates 5.3 to 6.4 times faster in 7.6 times less memory. A cannot go below 446 MB at any quantisation, because its 16-bit table alone is 250 MB. C's loss is 0.28 nats higher. We did not write a kernel for B.
|
| 68 |
+
|
| 69 |
+
## What is where
|
| 70 |
+
|
| 71 |
+
```
|
| 72 |
+
README.md, LICENSE.md
|
| 73 |
+
docs/ paper.pdf, paper.tex, figures/ (loss curves)
|
| 74 |
+
code/ train.py (arms A/B/C), eval.py + bars.py (benchmarks, bootstrap CIs), freq_eval.py, domain_vocab.py,
|
| 75 |
+
control_tokens.py, emoji_test.py, zeroshot_vocab.py, new_words.py, fastc.py (fast loader for C),
|
| 76 |
+
export_arm_a.py (A to HF and GGUF), plot.py, prep_data.py, bitnet_schedule.json
|
| 77 |
+
kernel/ cgemma.c (CPU kernel for C), armC.bin (C exported for it, 46 MB), export_arm_c.py, golden_c.py, ids.txt
|
| 78 |
+
tables/ gemma3_270m_512bit_table.npz (the frozen table), unigram counts and the seal
|
| 79 |
+
data/ 21 shards of training tokens (2.05B Gemma 3 tokens of FineWeb-Edu), val.npy (10M), SHA256SUMS.txt
|
| 80 |
+
A/ B/ C/ checkpoints every 800 steps and the final step 7,820 (weights + AdamW state), per-step training logs
|
| 81 |
+
B2/ arm B with a trained prior, stopped at step 1,833 for budget (measures the prior's share, see paper)
|
| 82 |
+
final/ weights-only safetensors of the three final checkpoints; armA_hf/ is arm A in Hugging Face Gemma3 format
|
| 83 |
+
gguf/ arm A as GGUF f16, Q8_0 and Q4_K_M for llama.cpp
|
| 84 |
+
logs/ A.jsonl, B.jsonl, C.jsonl: loss, learning rate, weight decay and tokens at every step
|
| 85 |
+
eval/ every per-question benchmark result and every analysis the paper quotes (JSON and markdown)
|
| 86 |
+
```
|
| 87 |
+
|
| 88 |
+
## Using the models
|
| 89 |
+
|
| 90 |
+
**Arm A** is a standard Gemma 3 270M and loads anywhere: `Gemma3ForCausalLM.from_pretrained("final/armA_hf")`, or any GGUF file in `gguf/` with llama.cpp.
|
| 91 |
+
|
| 92 |
+
**Arms B and C** need this repository's code, because a frozen code table with a seal is not a Hugging Face layer type.
|
| 93 |
+
|
| 94 |
+
```python
|
| 95 |
+
import numpy as np, torch, sys; sys.path.insert(0, "code")
|
| 96 |
+
import fastc # loads a checkpoint; for C it rounds the ternary weights once
|
| 97 |
+
counts = np.load("tables/unigram_counts_fineweb_edu_2.05B.npy")
|
| 98 |
+
model, _ = fastc.load("C", "C/step_07820.pt", counts) # or ("B", "B/step_07820.pt", counts)
|
| 99 |
+
ids = torch.tensor([[2, 651, 6037, 576, 1385, 603]]) # Gemma 3 tokenizer ids, <bos> first
|
| 100 |
+
with torch.no_grad():
|
| 101 |
+
logits = model.logits(model.hidden(ids)) # [1, 6, 262144]
|
| 102 |
+
```
|
| 103 |
+
|
| 104 |
+
Set `SAVED_TABLE=1` to read the table from `tables/` instead of rebuilding it from Gemma 3 270M. The rebuild gives the same bits; the file just saves the download. `final/armB` and `final/armC` hold the final weights without the optimiser state and without the table, which lives in `tables/`.
|
| 105 |
+
|
| 106 |
+
**Arm C on a CPU**, no Python needed:
|
| 107 |
+
|
| 108 |
+
```
|
| 109 |
+
gcc -O3 -mavx2 -mfma -mf16c -DNT=8 -o cgemma kernel/cgemma.c -lm -lpthread # add -lpsapi on Windows
|
| 110 |
+
PIN=1 ./cgemma kernel/armC.bin bench 128 64 # prefill 128 tokens, generate 64; prints tokens/s and peak RAM
|
| 111 |
+
./cgemma kernel/armC.bin golden kernel/ids.txt out.bin # logits for the check against PyTorch (kernel/golden_c.py)
|
| 112 |
+
```
|
| 113 |
+
|
| 114 |
+
The kernel is a single C file. It does ternary matrix-vector products with int8 activations, attention with a 16-bit KV cache, and a readout that scores all 262,144 codes straight off the 1-bit table. The readout does a quick int8 pass over every row, then an exact second pass only over the rows that can still win. Its top token matches the PyTorch model at every position we tested.
|
| 115 |
+
|
| 116 |
+
## Where the frozen table comes from
|
| 117 |
+
|
| 118 |
+
We shipped this idea before we tested it properly. Two earlier QLNI models have no learned embedding at all; every token is a fixed 512-bit code.
|
| 119 |
+
|
| 120 |
+
| model | what it is | code | weights |
|
| 121 |
+
|---|---|---|---|
|
| 122 |
+
| **SHADOW 250M Instruct** | 250M parameters, 131,072-token frozen table, body under 2 bits per weight, 100M-token archive on disk. 60 MB with the vocabulary, about 400 tokens/s and 80 MB RAM on a laptop CPU. Held-out perplexity 23.3; PIQA 0.60, HellaSwag 0.34, ARC-Easy 0.41 | [github.com/QLNI/SHADOW-250M-Instruct](https://github.com/QLNI/SHADOW-250M-Instruct) | [huggingface.co/QLNI/SHADOW-250M](https://huggingface.co/QLNI/SHADOW-250M) |
|
| 123 |
+
| **SHADOW 50M Instruct** | 44M ternary parameters, 73,880-token frozen table, exact circuits for arithmetic, dates and units inside the forward pass, memory on disk. 19.8 MB, about 2,000 tokens/s in 40 MB RAM. Also runs in the browser | [github.com/QLNI/SHADOW-50M-Instruct](https://github.com/QLNI/SHADOW-50M-Instruct) | [huggingface.co/QLNI/shadow-50m-instruct](https://huggingface.co/QLNI/shadow-50m-instruct) |
|
| 124 |
+
|
| 125 |
+
Both are MIT-licensed and both work, but neither was a controlled experiment. Each changed many things at once, and neither was ever trained with a learned table for comparison. This repository is that comparison: same body, same data, same steps, same seed, and only the vocabulary changes. The CPU kernel here is ported from the SHADOW 50M kernel.
|
| 126 |
+
|
| 127 |
+
## What comes next
|
| 128 |
+
|
| 129 |
+
A frozen table does not have to stop at words. In the next paper the same frozen code space carries text, images, audio and video into one small body, with no learned embedding anywhere, still running on a CPU. The numbers will come with that paper.
|
| 130 |
+
|
| 131 |
+
## Reproducing the training
|
| 132 |
+
|
| 133 |
+
`code/prep_data.py` tokenises FineWeb-Edu (or use `data/`, with checksums in `data/SHA256SUMS.txt`), then `python code/train.py --arm A|B|C` runs with the settings in the script: 2.05B tokens, sequence length 1,024, 262,144 tokens per step, 7,820 steps, AdamW 6e-4 with cosine decay to 10% for A and B, the BitNet two-stage schedule in `code/bitnet_schedule.json` for C, seed 0 and the same data order. One arm takes about 17 GPU-hours on an RTX 5090. For evaluation run `code/eval.py`, then `code/bars.py` for the paired bootstrap.
|
| 134 |
+
|
| 135 |
+
## Licence
|
| 136 |
+
|
| 137 |
+
Code, data derivatives, tables and checkpoints are for **non-commercial research use only**; see `LICENSE.md`. The frozen table and the checkpoints of arms B and C contain codes derived from Gemma 3 270M's embedding, so [Google's Gemma Terms of Use](https://ai.google.dev/gemma/terms) apply to them as well. The training tokens are a tokenised subset of FineWeb-Edu (ODC-By 1.0).
|
| 138 |
+
|
| 139 |
+
## Citation
|
| 140 |
+
|
| 141 |
+
```
|
| 142 |
+
@misc{bathula2026embeddings,
|
| 143 |
+
title = {Embeddings Are Overpriced? A Frozen 16 MB Vocabulary for Small and Ternary Language Models},
|
| 144 |
+
author = {Bathula, Sai Kiran},
|
| 145 |
+
year = {2026},
|
| 146 |
+
eprint = {2610.22428},
|
| 147 |
+
archivePrefix = {arXiv},
|
| 148 |
+
primaryClass = {cs.CL},
|
| 149 |
+
url = {https://arxiv.org/abs/2610.22428},
|
| 150 |
+
doi = {10.5281/zenodo.23382634}
|
| 151 |
+
}
|
| 152 |
+
```
|
| 153 |
+
|
| 154 |
+
Sai Kiran Bathula, QLNI, New South Wales, Australia. saikiranbathula@qlni.io
|