---
license: mit
language:
- en
pipeline_tag: text-generation
library_name: pytorch
datasets:
- karpathy/tiny_shakespeare
tags:
- character-level
- educational
- custom-architecture
- cpu
---

# OxMini

OxMini is a 2.17M-parameter character language model trained as a transparent, CPU-scale architecture experiment.

> **Architecture honesty notice:** OxMini is not “Ox Alpha shrunk down,” is not GLM-5.3-Flash, and is not a reproduction of either system. Public reporting identifies Ox Alpha as GLM-5.3-Flash, a 320B-total/18B-active sparse MoE with KDA, MLA+DSA, four-stream mHC, and vision. OxMini only studies toy approximations of the 3:1 hybrid pattern and multi-stream residual routing. It has no MoE, DSA indexer, vision encoder, long-context kernel, or comparable capability.

## Architecture

| Source component | OxMini approximation | Status |
|---|---|---|
| 3:1 KDA : MLA/DSA attention | Six recurrent KDA-lite + two global MLA-lite blocks | Simplified |
| KDA | Explicit channel-gated delta-rule recurrence, NoPE | Simplified; no chunk kernel |
| MLA | Full queries plus low-rank joint K/V bottleneck and RoPE | Simplified |
| DSA | None | Dropped |
| Sparse MoE | Dense SwiGLU | Dropped |
| Four-stream mHC | Static four-stream softmax routing; optional Sinkhorn projection | Substantially simplified |
| Vision | None | Dropped |

- Parameters: **2,169,388**
- Context: 256 characters
- Layers: 8 (`KDA-lite ×3, MLA-lite ×1`, repeated)
- Width / heads: 128 / 4
- Residual streams: 4
- Tokenizer: corpus-built character vocabulary with `<unk>`
- Weight format: safetensors

## Training data

The checkpoint uses Tiny Shakespeare downloaded from the Karpathy `char-rnn` corpus URL. It is deterministically split 90%/5%/5% by character into train/validation/test, and the vocabulary is built only from the training split. Shakespeare's underlying plays are public-domain text; this model repository does not redistribute prepared corpus files and does not claim ownership over source text.

- Corpus: 1,115,394 characters; SHA-256 `86c4e6aa9db7c042ec79f339dcb96d42b0075e16b8fc2e86bf0ca57e2dc565ed`
- Splits: 1,003,854 train / 55,770 validation / 55,770 test characters
- Vocabulary: 66 tokens including `<unk>`; zero unknown tokens in all splits
- Optimizer: AdamW with 5% warmup, cosine decay, and gradient clipping
- Run: 1,000 updates, batch size 8, seed 1337, CPU only
- Hardware/time: Apple M4 Pro, 10 threads, PyTorch 2.9.1, 1,820.9 seconds

## Measured evaluation and ablation

### Release checkpoint

| Metric | Value |
|---|---:|
| Initial validation loss | 4.2799 |
| Final/best validation loss | 1.7944 |
| Test cross-entropy | 1.80425 |
| Test perplexity | 6.075 |
| Test next-character accuracy | 47.79% |

The seen/unseen copy probe was inconclusive: seen accuracy was higher (50.0% vs 37.5%), but seen cross-entropy was worse (2.213 vs 2.075). The arithmetic probe was skipped because the vocabulary does not contain all digits.

### Matched 120-step ablation

| Variant | Parameters | Validation loss | Test PPL | Next-char accuracy | Time |
|---|---:|---:|---:|---:|---:|
| Plain GPT | 2,107,776 | 2.8228 | 16.54 | 26.17% | 17.1s |
| Hybrid, one stream | 2,168,984 | **2.4979** | **12.02** | **37.35%** | 150.2s |
| Hybrid + mHC-lite | 2,169,388 | 2.5595 | 12.78 | 35.21% | 153.5s |

All values are copied from local JSON/CSV artifacts included in this repository. The ablation arms used the same data, seed, update count, context, width/depth, and training recipe; parameter spread was 2.84%. Hybrid-only did best in this short run; mHC-lite did not help. The recurrent arms were about nine times slower than baseline because this implementation prioritizes a legible Python recurrence over a chunk-parallel kernel. These toy, single-seed results are descriptive and do not establish that KDA or mHC is generally better.

## Sample generations

Temperature 0.7, 120 new characters, unedited:

```text
ROMEO:
He prove sword in Recadars, he with that my grow say the broble joy these dead
The words.

WARWICK:
King to have done's
```

The sample learns speaker labels, line breaks, and some Shakespeare-like surface form, but remains incoherent. See `samples.md` for every fixed prompt at temperatures 0.7 and 1.0; none were filtered or cherry-picked.

## Load

This is a plain PyTorch custom architecture, not a Transformers `AutoModel` package. Download the repository and import its bundled source:

```python
from pathlib import Path
import sys
from huggingface_hub import snapshot_download

snapshot = Path(snapshot_download("Shivam3002/OxMini"))
sys.path.insert(0, str(snapshot / "src"))

from oxmini import CharTokenizer, OxMiniForCausalLM
from oxmini.generate import generate_text

model = OxMiniForCausalLM.from_pretrained(snapshot).eval()
tokenizer = CharTokenizer.load(snapshot / "tokenizer.json")
print(generate_text(model, tokenizer, "ROMEO:\n", max_new_tokens=120, temperature=0.8))
```

## Intended use

Educational inspection, CPU smoke tests, and toy ablations only.

## Limitations

This model is not suitable for factual, safety-critical, production, or user-facing tasks. Its corpus is tiny and narrow, its 256-character context is short, and character-level samples can be incoherent. The probes are diagnostic rather than standard language-model benchmarks. Architecture labels ending in “-lite” are approximations, not faithful kernels.

## References

- Sebastian Raschka, [GLM-5.3-Flash Architecture Notes](https://sebastianraschka.com/blog/2026/glm-5-3-flash-architecture-notes.html), Aug. 26, 2026.
- Kimi Team et al., [Kimi Linear](https://arxiv.org/abs/2510.26692) and [official code](https://github.com/MoonshotAI/Kimi-Linear).
- DeepSeek-AI, [DeepSeek-V3.2](https://arxiv.org/abs/2512.02556) and [V3.2-Exp code](https://github.com/deepseek-ai/DeepSeek-V3.2-Exp).
- Xie et al., [mHC](https://arxiv.org/abs/2512.24880).
- Hugging Face, [DeepSeek-V4 documentation](https://huggingface.co/docs/transformers/model_doc/deepseek_v4).
- Zhu et al., [Hyper-Connections](https://arxiv.org/abs/2409.19606).

## License

MIT for OxMini code and original model artifacts. Source-text rights remain with their respective sources.