Create README.md

README.md

@@ -0,0 +1,71 @@
+
+---
+language:
+  - en
+tags:
+  - text-generation
+  - causal-lm
+  - custom-architecture
+  - slm
+  - small-language-model
+license: mit
+---
+
+# Ember (2.87M Parameters)
+
+Ember is a highly efficient, ultra-small language model developed by SurjoLabs. It ranks **#2 on the Open SLM Leaderboard** for sub-3M parameter models, demonstrating that this custom XSA recurrent architecture is viable and can yield exceptional reasoning capabilities in tiny architectures.
+
+## Methodology
+
+Ember is built to prove that unique recurrent architectures can match or beat standard transformers even at the smallest scales.
+
+* **Custom Architecture:** Built on a Llama-variant framework but features **XSA Attention** (value-subtraction projection) and **Recurrent Layers**. By reusing weights across recurrent passes, the model achieves the depth of an 11-layer model while only carrying 8 layers of unique weights.
+* **Optimized Tokenizer:** Uses a custom 4,096-vocabulary English tokenizer. This intentionally small vocabulary prevents the embedding table from dominating the parameter count, ensuring nearly 80% of the model's 2.87M parameters are dedicated to actual transformer compute logic.
+* **Extreme Overtraining:** Trained on **20 Billion tokens**, resulting in a ~1:7,000 parameter-to-token ratio. This forces the small architecture to memorize syntax and knowledge far beyond Chinchilla-optimal limits.
+* **Data Mixture:** 
+  * 60% Finephrase (Synthetic FineWeb-Edu)
+  * 20% DCLM (Web text)
+  * 10% FineMath (Mathematics)
+  * 10% CornStack (Code)
+* **Optimizer:** Utilizes a hybrid Muon (for 2D weight matrices) and AdamW (for embeddings/norms) optimizer setup for stable, rapid convergence.
+
+## Benchmark Results (Open SLM Leaderboard)
+
+| Benchmark | Score (acc_norm) |
+| :--- | :--- |
+| HellaSwag | 27.35% |
+| ARC-Easy | 33.25% |
+| ARC-Challenge | 21.08% |
+| PIQA | 54.41% |
+| ArithMark-3 | 32.90% |
+| **Intelligence Index** | **5.94** |
+
+We used the Language Model Evaluation Harness for Hellaswag, ARC-Easy, ARC-Challange, PIQA.
+We used the provided script for ArithMark-3
+
+## Usage
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+import torch
+
+model_id = "SurjoLabs/Ember"
+tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
+model = AutoModelForCausalLM.from_pretrained(
+    model_id, 
+    trust_remote_code=True, 
+    torch_dtype=torch.bfloat16
+).cuda()
+
+prompt = "The capital of France is"
+inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
+outputs = model.generate(**inputs, max_new_tokens=20)
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+
+```
+
+## Acknowledgement
+We would like to thank [AxiomicLabs](https://huggingface.co/AxiomicLabs) for proving that XSA architecture is excellent for token efficiency.
+
+## Limitations
+This is an early test of the larger Surjo Project. The code is not very stable and we do not recommend using the modeling file without doing edits for training your own model.