MacroStories Model 2 (2-Layer Recurrent Micro-Transformer)

A 71,616-parameter (279 KB) recurrent language model capable of generating coherent, 4-beat Dutch children's stories directly inside a web browser or on micro-edge devices with 0 ms network latency.

This model is the champion checkpoint of the MacroStories research project by Pjotter Bolte.

Key Specifications

Parameter Value
Parameters 71,616
Model Size (FP32) 279 KB (L2 Cache resident)
Architecture Recurrent Transformer (2 physical layers $\times$ 2 recurrent loops per layer)
Hidden Dimension ($) 48
FFN Dimension ({ff}$) 72 (SwiGLU)
Attention Heads 3 Query heads, 1 Key-Value head (GQA, {head}=16$)
Positional Embedding Rotary Positional Embeddings (RoPE)
Vocabulary Size 800 tokens (Byte-Level BPE, 0.00% OOV on Dutch morphology)
Validation Loss 1.2426 nats/token (Perplexity: 3.46)
Training Steps 4,000 steps on 1x Nvidia T4 GPU (72 seconds total training time)

How the Architecture Works

Unlike conventional multi-billion parameter LLMs that stack 64–128 physical layers (the "skyscraper" design), MacroStories uses a two-room workshop with an elevator:

  • Layer 1: Specializes in Dutch syntax, morphology, spelling, and BPE subwords.
  • Layer 2: Specializes in causal narrative state transitions across the 4-beat story arc.
  • Loops: Tokens loop through each physical layer twice (=2$) with Value-Residual Mixing (hBmix = 0.5) to anchor memory across recurrent iterations without state drift.

Narrative Schema

The model expects a scenario header conditioning prompt and follows a strict 4-beat causal structure: 5137\text{Inleiding / Doel} \longrightarrow \text{Uitdaging} \longrightarrow \text{Handeling} \longrightarrow \text{Oplossing}5137

Prompt Format

Special structural markers:

  • : Introduction of the character, setting, and object.
  • : The obstacle preventing immediate success.
  • : The physical action resolving the hurdle.
  • : The happy, stable resolution.
  • : Story terminator.

Generated Sample

"Daan was in de schuur met een oude hamer die van opa was geweest. De houten plank van het bankje lag scheef en de spijkers staken omhoog, zodat Daan er niet op kon zitten. Daan pakte de plank recht en sloeg voorzichtig met de oude hamer op de losse spijkers, tik, tik, tik. De plank lag nu stevig en Daan ging blij op het bankje zitten."

Recommended Inference Configuration

For optimal coherence without over-generation or chatter:

  1. Temperature: –
  2. 3-Gram Repetition Penalty: Ban repeated 3-grams during autoregressive decoding to prevent stuttering.
  3. FSM Termination: Once the model outputs the section and produces at least 6 tokens, terminate generation immediately upon encountering the first terminal punctuation ( or ).

Training Dataset

Trained on , consisting of certified consistent Dutch story arcs generated via compatible scenario pairings and distilled 4-beat schemas.

Downloads last month
84
Safetensors
Model size
71.6k params
Tensor type
F32
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Dataset used to train mithraindir/macrostories-micro-2layer-d48