README.md

---
license: apache-2.0
pipeline_tag: text-generation
library_name: transformers
tags:
- language-model
- transformer
- rope
- gqa
- custom_code
- tiny
- looped
- text-generation
- slm
- custom-architecture
- custom-tokenizer
datasets:
- HuggingFaceFW/fineweb-edu
- HuggingFaceTB/finemath
---

![min-spark](charts/banner.svg)

# min-spark

**min-spark is a 5.76M-parameter language model with native effort levels.** It introduces controllable depth to the sub-10M model class. The same checkpoint can produce a quick completion or spend more computation on the same prompt, selected with one inference argument.

The model was trained on 10.01B tokens. Its looped decoder reuses a compact transformer core across multiple passes, giving a small model the computation of a deeper network while keeping the parameter count fixed.

## Introducing native effort levels

Effort levels have usually been associated with large reasoning models, where they adjust the thinking token budget or modify the system prompt to change how hard the model thinks. min-spark brings the idea directly into the language model. Each effort level changes the model's internal computation by selecting a different number of passes through its shared core.

| Effort | Character | Recommended use |
|---|---|---|
| `low` | Fastest | High-throughput completion |
| `medium` | Balanced | General generation |
| `high` | Most compute | Highest available quality |

Effort is a generation-time choice.

## Usage

min-spark is compatible with Transformers and requires remote code loading.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "MinimaLabs/min-spark",
    trust_remote_code=True,
).to("cuda")
tokenizer = AutoTokenizer.from_pretrained(
    "MinimaLabs/min-spark",
    trust_remote_code=True,
)

prompt = "The meaning of life is"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
outputs = model.generate(
    **inputs,
    effort="high",
    max_new_tokens=64,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

The same model works with the Transformers pipeline API:

```python
from transformers import pipeline

pipe = pipeline("text-generation", model=model, tokenizer=tokenizer)
result = pipe(prompt, effort="high", max_new_tokens=64)
print(result[0]["generated_text"])
```

A lightweight Transformers-free generation script is included in the repository:

```bash
python generate.py -p "The meaning of life is" -e high
```

Generation currently runs one sequence at a time, and right-padded batches are supported for evaluation. This model currently does not have KV cache support so long generations recompute the prompt at each step which may result in slightly longer generations. The context window is 512 tokens.

## Evaluation

Scores below come from zero-shot evaluation with lm-eval 0.4.12. BLiMP uses accuracy. ARC-Easy, ARC-Challenge, HellaSwag, and PIQA use length-normalized accuracy. WikiText-2 is reported as byte-level perplexity, where lower is better.

min-spark reaches 69.19% on BLiMP at medium effort. ARC-Easy reaches 37.08%. ARC-Challenge reaches 23.21%. HellaSwag reaches 27.92%. PIQA reaches 54.35%. Its best WikiText-2 byte perplexity is 2.7747.

### Effort levels

The table shows how the model responds to additional internal computation. Grammar improves most clearly from low to medium effort while the common-sense tasks remain close across the three settings.

![min-spark accuracy by effort](charts/effort.svg)

| Effort | BLiMP | ARC-Easy | ARC-Challenge | HellaSwag | PIQA | WikiText-2 byte-ppl |
|--------|-------|----------|---------------|-----------|------|---------------------|
| min-spark-low | 67.11% | 35.10% | 23.21% | 27.91% | 54.13% | 2.8783 |
| min-spark-medium | 69.19% | 37.08% | 22.78% | 27.92% | 54.30% | 2.7747 |
| min-spark-high | 69.18% | 37.08% | 22.87% | 27.91% | 54.35% | 2.7747 |

### Benchmarks

The comparison places min-spark alongside published results for GPT-S2-5M, SLM-10M, and michel-nano-v2. It covers the benchmarks reported across this group.

![min-spark compared with small-model peers](charts/comparison.svg)

| Model | Params | BLiMP | ARC-Easy | ARC-Challenge | HellaSwag | PIQA |
|-------|--------|-------|----------|---------------|-----------|------|
| **min-spark** | 5.76M | 69.19% | **37.08%** | 23.21% | **27.92%** | 54.35% |
| [GPT-S2-5M](https://huggingface.co/AxiomicLabs/GPT-S2-5M) | 5M | — | 33.92% | 22.87% | 27.87% | **57.56%** |
| [SLM-10M](https://huggingface.co/LiodonAI/SLM-10M) | 10M | — | 35.52% | **23.46%** | 27.40% | 57.07% |
| [michel-nano-v2](https://huggingface.co/finnianx/michel-nano-v2) | 8M | **72.52%** | 35.90% | 21.84% | 27.40% | 56.75% |

min-spark reaches 37.08% on ARC-Easy, the highest score in this comparison. Its 23.21% ARC-Challenge result is close to SLM-10M at 23.46%. HellaSwag reaches 27.92%, and PIQA reaches 54.35%.

### More in Less Parameters

KeyLM was the previous model made by us, and is a 75.25M-parameter language model trained on 18B tokens. min-spark uses less than one thirteenth of its parameter count while reaching a similar range on several small-model evaluations.

KeyLM reports 29.9% on its ARC average, 29.7% on HellaSwag, and 60.0% on PIQA. min-spark reaches 37.08% on ARC-Easy, 23.21% on ARC-Challenge, 27.92% on HellaSwag, and 54.35% on PIQA. The comparison highlights the value of repeated computation in a compact model. min-spark closes much of the size gap on these tasks while retaining a 5.76M parameter footprint.

## Architecture

| Field | Value |
|---|---|
| Parameters | 5,758,572 |
| Architecture | Tied-embedding looped decoder |
| Vocabulary | 4,096-token byte-level BPE |
| Embedding width | 288 |
| Heads | 6 query · 2 KV (GQA) |
| FFN hidden size | 768 |
| LoRA rank | 16 |
| Blocks | 1 prelude · 3 shared body blocks · 1 coda · final RMSNorm |
| Context window | 512 tokens |
| Effort (loop count) | low = 2 · medium = 3 · high = 4 |

## Training

| Field | Value |
|---|---|
| Training tokens | 10.01B |
| Data mix | ~90% filtered FineWeb-Edu · ~10% Finemath-4plus |
| Precision | fp16 autocast with gradient scaling |
| Context length | 512 tokens |
| Global batch size | 32 |
| Optimizer | Muon (matrices) + NAdamW (auxiliary parameters), weight decay 0.01 |
| Peak learning rate | Muon 0.01 · NAdamW 3e-3 |
| Learning-rate schedule | 2,000-step warmup → stable phase → 20% cooldown to 10% of peak |
| Attention masking | Intra-document |
| Checkpoint | Final cooldown checkpoint of the 10.01B-token run |

## Reproducing the evaluation

The evaluation runner is included in the repository. It supports each effort level and the benchmark suite used for this card.

```bash
python run_lmeval.py \
  --effort medium \
  --tasks blimp,arc_easy,arc_challenge,hellaswag,piqa,wikitext
```

Use `--limit N` to run a smaller evaluation during development.

## Limitations

min-spark is a small base language model. It is not instruction-tuned and does not provide conversational alignment or safety filtering. Factual recall, multi-step reasoning, and long-form coherence are limited by its scale and training objective.

The model supports a 512-token context window. Generation accepts one sequence at a time and currently has no KV cache. Results were collected with a 512-token context and should not be assumed to transfer to longer inputs.

## License

Apache-2.0. See [LICENSE](LICENSE).