| --- |
| license: agpl-3.0 |
| language: |
| - en |
| tags: |
| - text-generation |
| - smol |
| - permacomputer |
| - bandit-curriculum |
| pipeline_tag: text-generation |
| --- |
| |
| # ANDREA-12M |
|
|
| **A**utonomous **N**eural **D**ata **R**ecipe for **E**ducation and **A**gency |
|
|
| A 12.8M parameter language model grown on a single RTX 4090 using a bandit-controlled curriculum. |
| Part of the permacomputer project — open source, open data, open weights. |
|
|
| ## Model Details |
|
|
| | Property | Value | |
| |----------|-------| |
| | Parameters | 12.8M | |
| | Architecture | Transformer decoder, 384d/12h/6L | |
| | Embedding dim | 384 | |
| | Heads | 12 | |
| | Layers | 6 | |
| | Context | 1024 tokens | |
| | Tokenizer | Harris morpheme (2048 segments, 2305 vocab) | |
| | Training steps | 43,587 | |
| | Final SMMA loss | 2.0 | |
| | Best single-step loss | 0.21 | |
| | Training time | ~72 hours | |
| | Hardware | Single NVIDIA RTX 4090 (24GB VRAM, 1.4GB used) | |
| | CUDA engine | microgpt_cuda.cu (custom, FP32) | |
| | Born | 2026-03-21 12:53 UTC / 08:53 EST | |
| | License | AGPL-3.0 | |
| |
| ## Files |
| |
| | File | Step | Description | |
| |------|------|-------------| |
| | `ANDREA-12M.bin` | 43,587 | Final checkpoint (SMMA 2.0) | |
| | `ANDREA-12M-best.bin` | 42,300 | Best checkpoint (lowest loss during training) | |
| | `harris_segments.json` | — | Harris tokenizer segments (required for inference and fine-tuning) | |
|
|
| ### Checkpoint format |
|
|
| Binary, little-endian: `[int32 step][int32 n_params][n_params × float32 weights][n_params × float32 m][n_params × float32 v]` |
|
|
| - **Weights**: model parameters (12.8M floats, ~49MB) |
| - **m**: Adam first moment (same size) |
| - **v**: Adam second moment (same size) |
| - Total: ~147MB per checkpoint |
|
|
| Use either checkpoint to resume fine-tuning (weights + optimizer state preserved) |
| or extract weights only for inference (first `n_params` floats after the 8-byte header). |
|
|
| ## Training Data |
|
|
| Trained on a curated mix of open conversational and educational data: |
|
|
| - **NousResearch/Hermes-3-Dataset** (general, creative, roleplay) — 590K conversations |
| - **Dictionary** — 88K word definitions distilled from Hermes 3 8B |
| - **Gutenberg** — public domain literature (Project Gutenberg) |
| - Additional: chat, smoltalk, oasst, dolly, IRC, repo-docs |
|
|
| Data mix controlled by a UCB1 multi-armed bandit with dice-based phase control. |
| The bandit dynamically adjusts source weights during training based on per-source |
| loss trajectories. Full curriculum specification in the white paper. |
|
|
| ## Training Recipe |
|
|
| - Harris morpheme tokenizer (2048 segments) |
| - Cosine LR schedule with warm restart at step 25K (0.0004 peak) |
| - Phase-based bandit: 2 focus arms, 1d3 dice, source floors |
| - Checkpoints every 100 steps, SIGTERM-safe |
| - Per-source reward attribution, epoch penalty, coverage tracking |
|
|
| ## Capabilities |
|
|
| ANDREA-12M learns patterns, not facts. At 12.8M parameters it produces: |
| - Correct Q&A turn structure (`> question / < answer`) |
| - Definition-style responses |
| - Multi-sentence outputs with plausible grammar |
| - Instruction-following scaffolding ("explain", "define", "describe") |
|
|
| It does NOT produce factually accurate content — it's a pattern machine. |
| Factual accuracy requires scaling to ANDREA-120M (planned). |
|
|
| ## Usage |
|
|
| ```python |
| # Inference via microgpt |
| from microgpt import load_model, generate_fast |
| |
| model = load_model('ANDREA-12M.json') |
| results = generate_fast(model['state_dict'], model['uchars'], model['bos'], |
| 384, 12, 6, 1024, prefix='> what is an apple? / <') |
| print(results[0][0]) |
| ``` |
|
|
| ## White Paper |
|
|
| [ANDREA-12M-WHITEPAPER.pdf](ANDREA-12M-WHITEPAPER.pdf) — full technical paper covering architecture, bandit curriculum, data sources, training recipe, and results. |
|
|
| Source: `whitepaper/ANDREA/WHITEPAPER.rst` in the [uncloseai-cli repository](https://git.unturf.com/engineering/unturf/uncloseai-cli). |
|
|
| ## Citation |
|
|
| ``` |
| ANDREA: Autonomous Neural Data Recipe for Education and Agency |
| TimeHexOn, foxhop, russell@unturf |
| March 2026, permacomputer.com |
| ``` |
|
|
| ## License |
|
|
| AGPL-3.0. Code outlasts authors. Infrastructure outlasts builders. |
|
|
| ● ○ |
|
|
