README.md

---
license: apache-2.0
viewer: false
datasets:
- HuggingFaceFW/fineweb-edu
language:
- en
pipeline_tag: text-generation
library_name: transformers
tags:
- small
- tiny
- supra
- supra2
- qwen3
- efficient
---

<h1 align="center">Supra2-Medium Base</h1>

<p align="center">
  Ultra-efficient base model • 25M Parameters • 1K Context
</p>

![supra2-medium](https://cdn-uploads.huggingface.co/production/uploads/697f2832c2c5e4daa93cece7/-YjDJGlcGyTN5yClAaEv3.png)

**Supra2-Medium Base** is a 25M-parameter decoder-only language model pretrained from scratch by **SupraLabs** on 20B tokens of English web text. It uses the **Qwen3** architecture with a custom 16,384-token tokenizer.

This is a **base model**. It has *not* been instruction-tuned, chat-tuned, or aligned in any way. 

At only 25 million parameters, Supra2-Medium demonstrates that meaningful language modeling can be achieved with extreme parameter efficiency—trained at ~800 tokens per parameter, which is significantly higher than typical pretraining ratios. This makes it ideal for research into data-efficient scaling and ultra-lightweight deployments.

---

## **Let the model speak - a sample**

Prompt: "Artificial intelligence (AI) is "
Completion:
```plaintext
 Artificial intelligence (AI) is espoused by the AI community.
The AI community is a group of people who are interested in AI and are interested in the use of AI in the field of AI.
The goal of AI is to improve the quality of life of people in the field.
The aim of AI is the development of AI and the application of AI in a society.
The purpose of AI is that it can be used to improve the performance of the society.
It is a technology that is used to improve human intelligence.
The technology is used to make the human intelligence.
```

## **Evaluation & Benchmarks**

All benchmarks were evaluated using the EleutherAI LM-Eval Harness.

| Model | PIQA (acc\_norm) | HellaSwag (acc\_norm) | ARC-Easy (acc\_norm) | ARC-Challenge (acc\_norm) |
| ----- | :---: | :---: | :---: | :---: |
| Supra-50M-Base (50M) | 0.62 | 0.32 | 0.46 | 0.25 |
| **Supra2-Medium-Base (25M)** | **59.14** | **29.29** | **41.84** | **23.72** |
| Supra2-100M-Base (100M) | 0.65 | 0.36 | 0.48 | 0.25 |

**Final Train Loss: 3.2469** (no eval loss available)

*Note: The model shows strong performance relative to its size, particularly given the high token-per-parameter ratio. It even strongly competes with our previous 50M model!*

---

## **Model Details**

|  |  |
| ----- | ----- |
| **Developed by** | SupraLabs |
| **Model type** | Causal decoder-only transformer (Qwen3) |
| **Language** | English |
| **Parameters** | 25.37M total / \~20M non-embedding |
| **Training tokens** | 20B (~800 tokens per parameter) |
| **Context length** | 1,024 |
| **Precision** | bfloat16 |
| **License** | Apache 2.0 |

### **Architecture**

| Hyperparameter | Value |
| ----- | ----- |
| Hidden size | 512 |
| Layers | 7 |
| Attention heads | 8 (MHA) |
| Head dim | 64 |
| Intermediate size (SwiGLU) | 896 |
| Vocab size | 16,384 |
| Positional encoding | RoPE θ=10,000 |
| Normalization | RMSNorm, ε=10⁻⁶ |
| Tied embeddings | Yes |
| Attention implementation | SDPA |
| Architecture ID | `qwen3-d07-h512-i0896` |

---

## **Training Data**

| Source | Share | Approx. tokens |
| ----- | ----- | ----- |
| `HuggingFaceFW/fineweb-edu` (`sample-100BT`) | 100% | 20B |

Documents were tokenized with the custom `tokenizer_16k`, concatenated into a flat `uint16` token stream, and packed into contiguous 1,024-token chunks (no padding, no document masking — sequences may cross document boundaries).  

---

## **Training Procedure**

| Setting | Value |
| ----- | ----- |
| Optimizer | AdamW (fused), β₁=0.9, β₂=0.95, ε=10⁻⁸ |
| Peak learning rate | 3×10⁻³ |
| LR schedule | Cosine with min LR (10% of peak) |
| Warmup steps | 1,000 |
| Micro batch size | 32 |
| Gradient accumulation | 4 |
| Effective batch | 256 sequences \= **262,144 tokens/step** |
| Weight decay | 0.1 |
| Gradient clipping | 1.0 |
| Mixed precision | BF16 \+ TF32 |
| Hardware | 2× GPU (DDP): RTX 5060 Ti 16GB + RTX 5060 8GB |
| Training optimizations | Liger Kernel, DDP bucketing (10MB), gradient\_as\_bucket\_view |

### **Key Design Decisions**

* **High token-per-parameter ratio**: At 800 tokens/param, this model pushes the boundaries of data efficiency for small models  
* **Compact vocabulary**: 16K vocab reduces embedding overhead while maintaining coverage  
* **Tied embeddings**: Reduces parameter count and improves training stability  
* **Multi-head attention (MHA)**: Unlike larger Supra2 models using GQA, this architecture uses standard MHA for simplicity at this scale  
* **No sliding window**: Full attention within the 1K context

---

## **Usage**

```python  
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "SupraLabs/Supra2-Medium"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model.eval()

prompt = "The future of artificial intelligence"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    out = model.generate(
        **inputs,
        max_new_tokens=512,
        do_sample=True,
        temperature=0.2,
        top_p=0.85,
        top_k=25,
        no_repeat_ngram_size=3,
        repetition_penalty=1.2,
    )

print(tokenizer.decode(out[0], skip_special_tokens=True))
```

### **Tokenizer notes**

The model uses a custom 16,384-token vocabulary optimized for English web text. The tokenizer is shared across the Supra2 family's smaller models for consistency and efficient multi-task fine-tuning.  

---

## **Intended Use**

**Intended:**

* Research on extreme parameter efficiency and data-efficient pretraining  
* Ultra-lightweight edge deployments where memory is severely constrained  
* Educational use for understanding transformer architectures at minimal scale  
* Starting point for domain-specific fine-tuning when compute resources are limited  
* Ablation studies on small model behavior and scaling laws

**Not intended:**

* Production deployment for critical applications  
* Factual question answering or knowledge-intensive tasks  
* Long-form coherent generation beyond a few paragraphs  
* Non-English text (trained exclusively on English)  
* Any application requiring safety guarantees or alignment

---

## **Limitations and Bias**

* **Very small.** At 25M parameters, the model has extremely limited capacity. Expect frequent hallucinations, factual errors, repetitive outputs, and poor reasoning.  
* **Base model.** No RLHF, no safety tuning, no refusal behavior. It will continue whatever text you give it, including harmful or offensive prompts.  
* **Web-derived data.** FineWeb-Edu is a filtered CommonCrawl derivative and carries the biases, stereotypes, and factual errors of the open web.  
* **Short context.** Trained exclusively at 1,024 tokens. Extrapolation beyond this length is untested and likely degraded.  
* **No document masking.** Attention could cross document boundaries within a packed chunk, which slightly blurs document independence.  
* **English only.** Performance on other languages will be poor to non-existent.  
* **High token-per-parameter ratio.** While efficient, training at 800 tok/param means the model may be under-trained compared to models trained at lower ratios (e.g., 300 tok/param).

---

## **Performance Characteristics**

Despite its tiny size, Supra2-Medium achieves surprisingly coherent short-form generation. Key observations:

* **Strengths**: Basic grammar, simple factoids, short completions, pattern matching  
* **Weaknesses**: Complex reasoning, arithmetic, long-range coherence, factual accuracy, nuanced understanding  
* **Best use case**: Generating 1-3 sentence completions, simple text transformations, educational demonstrations

---

Future work includes instruction-tuned variants and exploration of even more efficient architectures at this scale.  

*© SupraLabs 2026*