Update README.md

README.md

@@ -11,14 +11,17 @@ tags:
 - mqa
 - hyperffn
 - router-gating
+datasets:
+- nvidia/Nemotron-Math-HumanReasoning
+- WeMake/Intelligent-Content-Understanding
 ---
 
-# MoAMetricLM-185M — Mixture of Attentions (MoA)
+# MoAMetricLM-100M — Mixture of Attentions (MoA)
 
 **A geometry-aware Transformer with a mixture of attention mechanisms and metric-based routing.**  
-**Parameters:** ~185M | **Type:** Causal LM (decoder-only) | **KV cache:** not yet implemented
+**Parameters:** ~100M| **Type:** Causal LM (decoder-only) | **KV cache:** not yet implemented
+
 
----
 
 ## Model Index
 
@@ -26,10 +29,10 @@ tags:
 - **Task:** text generation (`text-generation`)
 - **Library:** 🤗 Transformers
 - **License:** Apache-2.0 (change here & add LICENSE file if different)
-- **Datasets (examples used in dev runs):**
-  - `WeMake/Intelligent-Content-Understanding` (+ another 250k-token dataset)
+- **Datasets :**
+- nvidia/Nemotron-Math-HumanReasoning: ~256k tokens
+- WeMake/Intelligent-Content-Understanding ~256k tokens
 
----
 
 ## Overview
 
@@ -49,7 +52,6 @@ tags:
 
 **Design goals:** geometric consistency, diverse inductive biases, structured efficiency, and full HF compatibility.
 
----
 
 ## What’s different from a standard Transformer?
 
@@ -63,7 +65,7 @@ tags:
 - **Up/Down projections** (SwiGLU-style) inside heads to expand/contract the value stream.
 - **HyperFFN** provides non-lazy capacity with token-wise branch routing.
 
----
+
 
 ## Intended Use & Limitations
 
@@ -76,7 +78,7 @@ tags:
 
 **Out-of-scope:** high-stakes applications (medical/legal/etc.) without further training, evaluation, and safeguards.
 
----
+
 
 ## Training Details
 
@@ -98,7 +100,7 @@ tags:
 
 **Stability aids:** safe softmax (subtract max), PreNorm, LayerScale (≈1e-4), DropPath (optional), label masking (`-100` on padding).
 
----
+
 
 ## Configuration (example)
 
@@ -133,6 +135,8 @@ tags:
   "bos_token_id": 50256,
   "eos_token_id": 50256
 }
+```
+---
 
 If you use gpt2 tokenizer, set pad_token = eos_token and ensure vocab_size/eos/pad match the tokenizer.
 
@@ -142,6 +146,7 @@ Usage
 
 Inference
 
+```python
 from transformers import AutoTokenizer, AutoModelForCausalLM
 
 model_id = "your-hf-username/MoAMetricLM-185M"
@@ -186,8 +191,8 @@ def collate(examples):
 #     out.loss.backward()
 #     torch.nn.utils.clip_grad_norm_(model.parameters(), 1.2)
 #     optimizer.step(); optimizer.zero_grad()
+```
 
----
 ## Evaluation
 
 For meaningful comparisons, run:
@@ -198,9 +203,7 @@ For meaningful comparisons, run:
 	•	With vs without HyperFFN branch router/gates
 	•	With vs without TI regularizer
 
-Please share results via Issues/PRs.
 
-⸻
 
 Efficiency Notes
 	•	Ball pruning: masks keys outside per-head radius → structured sparsity.
@@ -208,22 +211,17 @@ Efficiency Notes
 	•	HyperFFN: token-wise branch router (optional top-k) to avoid paying for all branches equally.
 	•	CPU tips: set OMP_NUM_THREADS/MKL_NUM_THREADS to core count; use pad_token = eos_token.
 
-Roadmap: metric-aware KV cache for long contexts; kernelized distance approximations (e.g., RFF) for sub-quadratic regimes; quantization & mixed precision exploration.
-
-⸻
-
+Roadmap: metric-aware KV cache for long contexts; kernelized distance approximations (e.g., RFF) for sub-quadratic regimes; quantization & mixed precision 
 Safety, Bias & Risks
 	•	May produce biased, offensive, or factually incorrect outputs.
 	•	No safety/alignment training included.
-	•	Do not deploy in high-stakes contexts without additional safeguards.
+	•	Do not deploy in high-stakes contexts without additional 
 
-⸻
 
 License
 
 Apache-2.0 (update if different).
 
-⸻
 
 Citation
 
@@ -235,16 +233,15 @@ Citation
 }
 
 
-⸻
 
 Changelog
 	•	v0.2 (2025-09-20) — 500k-token CPU run, GPT-2 tokenizer, LR=5e-4, final loss ≈ 0.30.
 	•	v0.1 (2025-09-20) — initial public release: metric heads, MQA, ball pruning, HyperFFN, router & gates; HF-compatible; no KV cache.
 
-⸻
+
 
 Maintainers
 	•	Author: reaper (Convergent Intelligence LLC)
 	•	Contact: add preferred contact
 	•	Issues: HF model hub issues tab
----
+---