Hugging Face's logo

techdotus
/
Tokle-SPAB-3M

Text Generation
Transformers
Safetensors
English
tokle
causal-lm
custom-architecture
slm
small-language-model
spab
custom_code
Model card Files
xet
 Community
anandselvadurai-techus commited on
Commit
ecac7ca
·
verified ·
1 Parent(s): 0d30b28

Updated README based on the Model's SPAB details.

Browse files
Files changed (1) hide show
  1. README.md +14 -25
README.md CHANGED
@@ -20,16 +20,11 @@ datasets:
20
  - nvidia/OpenMathInstruct-2
21
  - microsoft/orca-math-word-problems-200k
22
  ---
23
-
24
- # Tokle-3M
25
 
26
  ## Model Summary
27
 
28
- Tokle-3M is a decoder-only language model with 2.91M trainable parameters,
29
- trained on 12B tokens. Its main architectural addition is SPAB (Static Pairwise
30
- Attention Bias), a frozen table of token-pair association scores built from
31
- Pointwise Mutual Information (PMI) over the training corpus and added to the
32
- attention logits of the first layer.
33
 
34
  For every query-key pair, SPAB hashes the two token IDs into the table, pulls
35
  out their PMI value, multiplies it by a learned per-head scale, and adds it to
@@ -42,7 +37,7 @@ prior.
42
 
43
  | Parameter | Value |
44
  | --- | --- |
45
- | Architecture | Custom decoder-only transformer + SPAB (`TokleForCausalLM`) |
46
  | Layers | 9 |
47
  | Hidden size (d_model) | 144 |
48
  | Attention heads | 3 |
@@ -52,41 +47,34 @@ prior.
52
  | Max sequence length | 512 |
53
  | Trainable parameters | 2,908,947 |
54
  | Frozen SPAB table | 8,388,608 (float32 buffer) |
55
-
56
  ## How to use
57
-
58
  This model uses a custom architecture, so it needs `trust_remote_code=True`.
59
-
60
  ```python
61
  import torch
62
  from transformers import AutoModelForCausalLM, AutoTokenizer
63
-
64
- model_id = "techdotus/Tokle-3M"
65
  tok = AutoTokenizer.from_pretrained(model_id)
66
  model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True).eval()
67
 
68
- ids = tok("The capital of France is", return_tensors="pt")
69
  with torch.no_grad():
70
- out = model.generate(**ids, max_new_tokens=32, do_sample=False) # greedy
71
  print(tok.decode(out[0], skip_special_tokens=True))
72
  ```
73
-
74
  ## Benchmark Results
75
 
76
  All scores are 0-shot acc_norm, using the Open SLM Leaderboard methodology:
77
  | Hellaswag | ARC-Easy | ARC-Challenge | PIQA | Arithmark-3 |
78
  | --------- | -------- | ------------- | ---- | ----------- |
79
  | 27.22% | 34.68% | 24.49% | 54.95% | 41.70% |
80
-
81
  ## Comparison Results
82
-
83
  All scores are 0-shot acc_norm, using the Open SLM Leaderboard methodology.
84
  Scores for the other models are from the Open SLM Leaderboard. **Bold** marks
85
  the best result in each column.
86
 
87
  | Model | Params | Int Index | HellaSwag | ARC-Easy | ARC-Chal | PIQA | ArithMark-3 |
88
  | --- | --- | --- | --- | --- | --- | --- | --- |
89
- | **Tokle-3M** (Tech.us) | 2.9M | **9.16** | 27.22% | **34.68%** | **24.49%** | 54.95% | **41.70%** |
90
  | Ember-2 (SurjoLabs) | 2.96Mx2 | 7.21 | 27.28% | 33.42% | 22.01% | 55.11% | 35.90% |
91
  | BananaMind-2-Micro (BananaMind) | 2.9M | 6.01 | **28.27%** | 33.12% | 21.93% | 53.21% | 34.00% |
92
  | GPT-S-1.4M (Axiomic Labs) | 1.4M | 5.40 | 26.89% | 31.57% | 21.93% | **55.17%** | 30.20% |
@@ -114,9 +102,10 @@ removed topics not useful for a model of this size.
114
 
115
  ## Limitations
116
 
117
- - **Tiny model:** with ~2.9M trainable parameters and 144-dim hidden states,
118
- generations are often repetitive, incoherent or factually wrong. The model is
119
- a research artifact for studying small-scale LMs, not an assistant.
 
120
  - **Short context:** 512 tokens maximum. RoPE tables are not built beyond that
121
  length.
122
  - **English only:** trained on English web, educational, synthetic and math text.
@@ -131,10 +120,10 @@ Model weights and code: MIT.
131
 
132
  ```bibtex
133
  @misc{tokle2026,
134
- title = {{Tokle-3M}: Pointwise Mutual Information as an Inductive Bias for Self-Attention},
135
  author = {{Tech.us Team}},
136
  year = {2026},
137
  publisher = {Hugging Face},
138
- howpublished = {\url{https://huggingface.co/techdotus/Tokle-3M}}
139
  }
140
- ```
 
20
  - nvidia/OpenMathInstruct-2
21
  - microsoft/orca-math-word-problems-200k
22
  ---
23
+ # Tokle-SPAB-3M
 
24
 
25
  ## Model Summary
26
 
27
+ Tokle-SPAB-3M is a decoder-only language model trained on 12B tokens, with 2.91M trainable parameters and 8.39M frozen SPAB parameters, for a total of 11.3M parameters. Its main architectural addition is SPAB (Static Pairwise Attention Bias), a frozen table of token-pair association scores built from Pointwise Mutual Information (PMI) over the training corpus and added to the attention logits.
 
 
 
 
28
 
29
  For every query-key pair, SPAB hashes the two token IDs into the table, pulls
30
  out their PMI value, multiplies it by a learned per-head scale, and adds it to
 
37
 
38
  | Parameter | Value |
39
  | --- | --- |
40
+ | Architecture | Custom decoder-only transformer + SPAB |
41
  | Layers | 9 |
42
  | Hidden size (d_model) | 144 |
43
  | Attention heads | 3 |
 
47
  | Max sequence length | 512 |
48
  | Trainable parameters | 2,908,947 |
49
  | Frozen SPAB table | 8,388,608 (float32 buffer) |
 
50
  ## How to use
 
51
  This model uses a custom architecture, so it needs `trust_remote_code=True`.
 
52
  ```python
53
  import torch
54
  from transformers import AutoModelForCausalLM, AutoTokenizer
55
+ model_id = "techdotus/Tokle-SPAB-3M"
 
56
  tok = AutoTokenizer.from_pretrained(model_id)
57
  model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True).eval()
58
 
59
+ ids = tok("The climate change", return_tensors="pt")
60
  with torch.no_grad():
61
+ out = model.generate(**ids, max_new_tokens=32, do_sample=False, repetition_penalty=1.3) # greedy
62
  print(tok.decode(out[0], skip_special_tokens=True))
63
  ```
 
64
  ## Benchmark Results
65
 
66
  All scores are 0-shot acc_norm, using the Open SLM Leaderboard methodology:
67
  | Hellaswag | ARC-Easy | ARC-Challenge | PIQA | Arithmark-3 |
68
  | --------- | -------- | ------------- | ---- | ----------- |
69
  | 27.22% | 34.68% | 24.49% | 54.95% | 41.70% |
 
70
  ## Comparison Results
 
71
  All scores are 0-shot acc_norm, using the Open SLM Leaderboard methodology.
72
  Scores for the other models are from the Open SLM Leaderboard. **Bold** marks
73
  the best result in each column.
74
 
75
  | Model | Params | Int Index | HellaSwag | ARC-Easy | ARC-Chal | PIQA | ArithMark-3 |
76
  | --- | --- | --- | --- | --- | --- | --- | --- |
77
+ | **Tokle-SPAB-3M** (Tech.us) | 2.9M | **9.16** | 27.22% | **34.68%** | **24.49%** | 54.95% | **41.70%** |
78
  | Ember-2 (SurjoLabs) | 2.96Mx2 | 7.21 | 27.28% | 33.42% | 22.01% | 55.11% | 35.90% |
79
  | BananaMind-2-Micro (BananaMind) | 2.9M | 6.01 | **28.27%** | 33.12% | 21.93% | 53.21% | 34.00% |
80
  | GPT-S-1.4M (Axiomic Labs) | 1.4M | 5.40 | 26.89% | 31.57% | 21.93% | **55.17%** | 30.20% |
 
102
 
103
  ## Limitations
104
 
105
+ - **Tiny model:** with ~2.9M trainable parameters, ~8.39M frozen SPAB
106
+ parameters and 144-dim hidden states, generations are often repetitive,
107
+ incoherent or factually wrong. The model is a research artifact for studying
108
+ small-scale LMs, not an assistant.
109
  - **Short context:** 512 tokens maximum. RoPE tables are not built beyond that
110
  length.
111
  - **English only:** trained on English web, educational, synthetic and math text.
 
120
 
121
  ```bibtex
122
  @misc{tokle2026,
123
+ title = {{Tokle-SPAB-3M}: Pointwise Mutual Information as an Inductive Bias for Self-Attention},
124
  author = {{Tech.us Team}},
125
  year = {2026},
126
  publisher = {Hugging Face},
127
+ howpublished = {\url{https://huggingface.co/techdotus/Tokle-SPAB-3M}}
128
  }
129
+ ```