README.md

---
license: mit
viewer: false
language:
- en
pipeline_tag: text-generation
tags:
- small
- shadow
- instruct
- chat
- long-context
- retrieval
- cpu
---

<h1 align="center">SHADOW 250M Instruct</h1>

![shadow](banner.jpg)

<p align="center">
  Instruct model of SHADOW 250M · 250M Parameters · 100M-Token Offline Context · 60 MB
</p>

**SHADOW 250M Instruct** is a 250 million parameter language model built from scratch,
trained on 30 billion tokens of English text with about 0.7 billion further tokens of
instruction tuning. The complete deployment is 60 MB, vocabulary included. It runs at about 400 tokens per
second on a laptop CPU and uses about 80 MB of RAM.

Language modeling quality of the base model, measured on held-out English web text that
was never seen in training (educational web pages, 2,048 token windows): cross entropy
3.15 nats per token, perplexity 23.3, 0.99 bits per byte.

Next to its 2,048 token attention window, SHADOW can use an offline archive: a compressed
cache of up to 100 million tokens that stays on your hard disk. The model was trained from
step 1 to retrieve from this archive, all the way up to 100 million tokens. Due to a
limited budget it was not trained to reason over the tokens in the disk cache. It finds
facts and reads them back; it does not think across them.

For scale, the official released weight files of well-known small models, next to ours:

| model | parameters | weights on disk |
|---|---|---|
| GPT-2 | 124M | 548 MB |
| SmolLM2-135M-Instruct | 135M | 269 MB |
| SmolLM2-360M-Instruct | 360M | 724 MB |
| Qwen2.5-0.5B-Instruct | 494M | 988 MB |
| **SHADOW 250M Instruct** | **250M** | **60 MB, vocabulary included** |

Sizes are the main weight file of each official repository, as published. None of the
models above reads a 100 million token archive from disk.

*(The project was named after my first dog, Shadow, as it is my first big project.)*

---

## Some samples

Every sample shows the exact settings it was generated with, using the files in this repo.
Nothing on this page was made with settings other than the ones shown.

### 1.

Prompt: "Explain photosynthesis in two sentences."<br>
Settings: greedy decoding (temperature 0)<br>
Answer:
> Photosynthesis is a process in which plants convert sunlight into chemical energy, which
> is then used to produce oxygen and other chemicals. This process is called photosynthesis.

### 2.

Prompt: "Write a short poem about the sea."<br>
Settings: temperature 0.25 · top-k 30 · repetition penalty 1.15 · seed 2<br>
Answer:
> The waves had swept over, and they were crashing against each other like rocks on top of
> one another. The wind was strong but not very gentle as it tried to push them towards
> their destination without lifting any weight or pulling away from those who stood in
> front of her with a stiff smile that made the sea look even more majestic than before.

### 3. Reading its archive

Prompt: "What is the serial number of device Grus-189?" The fact sits 50.6 million tokens
deep in the 100M evaluation archive on disk.<br>
Settings: archive mode as shipped, retrieval k=16<br>
Answer:
> SN-442976

---

## Benchmarks

All numbers are our own, measured on held-out data with the harness shipped in
`benchmarks/`, together with the raw results. We compare with nobody on this page; the
harness runs on any archive of yours as well.

Standard, acc_norm, zero-shot, full sets, greedy scoring:

| PIQA | HellaSwag | ARC-Easy | ARC-Challenge |
|---|---|---|---|
| 0.60 | 0.34 | 0.41 | 0.24 |

Long context, answered from the archive on disk. Settings: archive mode as shipped,
retrieval k=16, exact match scoring, archives and facts held out from all training:

| | 1M tokens | 10M tokens | 100M tokens |
|---|---|---|---|
| Needle in a haystack (5 depths) | 0.98 | 0.98 | 0.98 |
| Needle with look-alike distractors | 1.00 | 1.00 | – |
| Multi-key needles | 1.00 | 1.00 | – |
| Two-hop variable tracking | 1.00 | 1.00 | – |
| Scattered story facts, latest wins | 1.00 | 1.00 | – |
| Fact QA, 6 task types with abstain | 0.97 | 0.95 | 0.83 |

## Architecture

| Hyperparameter | Value |
|---|---|
| Hidden size | 1536 |
| Layers | 10 |
| Attention heads | 24 (GQA, 2 KV heads) |
| Head dim | 64 |
| Intermediate size (SwiGLU) | 4224 |
| Vocab size | 131,072 (frozen, 0 trainable parameters) |
| Positional encoding | RoPE θ=10,000 |
| Normalization | RMSNorm, ε=10-6 (incl. QK-Norm) |
| Tied embeddings | Yes (shared vocabulary table) |
| Attention window | 2,048 tokens + offline archive up to 100M |
| Body weight precision | under 2 bits per weight |
| Parameters | 250M |
| Runtime | bundled CPU kernel (AVX2/AVX-512), no framework needed |

![framework](framework.png)

## Performance

Measured on a laptop CPU with 8 physical cores, using the exact files in this repo. The
bundled kernel handles chat, the two-tier KV cache, and a live memory panel (`--status`).

| | |
|---|---|
| decode speed, 8 threads | 402 tokens/s |
| decode speed, 4 / 2 / 1 threads | 393 / 275 / 158 tokens/s |
| prefill speed | 409 tokens/s |
| RAM while chatting | ~80 MB |
| archive index build (once per archive, at load) | 2 s at 1M · 21 s at 10M · 3.2 min at 100M |
| retrieval per question | 37 ms at 10M · 435 ms at 100M |
| archive question, end to end | 0.45 s at 100M |

## Fine-tuning

Yes, you can fine-tune it, on one GPU, and export your own 52 MB model for CPU. We did it
ourselves as a demonstration: 90 minutes on a laptop GPU turned SHADOW into a pirate
assistant, with benchmark scores unchanged. The full guide with the commands, the dataset,
and the before and after results is in [finetune/FINETUNING.md](finetune/FINETUNING.md).

> The capital of France be Paris. It is a UNESCO World Heritage Site... Yarr!

## Repository layout

    deployment/            the model: weights, vocabulary, and the runtime binaries
      shadow250m_instruct.shdw   52 MB weights
      fp131072.npy               8.4 MB vocabulary
      bin/windows/  bin/linux/   prebuilt CPU runtimes (macOS on request)
    tokenizer/             3 files, 5 MB
    finetune/              master weights, training script, exporter, guide, worked example
    benchmarks/            results, report, harness
    shadow_runtime/        archive question answering (Python)

## Usage

Easiest start, any system:

    python shadow_chat.py

Chat directly with the binary, no Python needed. Windows:

    deploymentin\windows\shadow.exe deployment\shadow250m_instruct.shdw deploymentp131072.npy --chat

Linux:

    deployment/bin/linux/shadow deployment/shadow250m_instruct.shdw deployment/fp131072.npy --chat

Add --status to either for a live memory panel. Ask a question against an archive (a folder
holding a tokens.u32 stream):

    python -m shadow_runtime --model shadow250m_instruct.shdw --table fp131072.npy \
        --archive path/to/archive --ask "your question"

Python:

    from shadow_runtime import Engine
    eng = Engine("shadow250m_instruct.shdw", "fp131072.npy", archive="path/to/archive")
    print(eng.answer("your question"))

### Chat template

    <start_of_turn>user
    {message}<end_of_turn>
    <start_of_turn>model
    {response}<end_of_turn>

## Intended use

Intended:

* Local assistants on CPU-only hardware, fully offline
* Question answering over large private text archives: logs, books, documentation
* Fine-tuning your own small assistant on one GPU
* Research and education on small models and long context

Not intended:

* Production or user-facing deployment without human review
* Factual question answering from the model's own memory, advice, or decision support
* Non-English text

## Limitations and bias

* Small. At 250M parameters, open facts, arithmetic, and long answers are weak. Expect
  mistakes outside the archive.
* The model retrieves and reads from its archive. It was not trained to reason across
  many archive documents; that needs a bigger training budget than this project had.
  Two-hop chains degrade at 100M tokens.
* Trained on public web text, so its outputs can carry the biases of that text.
* English only.

## Contact

Questions, results, or something you built with it: saikiranbathula1@gmail.com

---

*© NODEMIND 2026*