engram-gyre

The first model in the engram lineage with actual trained weights. A ~11M-parameter, from-scratch, Llama-architecture language model whose entire behavioral repertoire is a single gyre revolution: the four-phase Recursive Abstraction Engine loop treated as a spiral, executed with the cognitive moves of the engram-prime MINDSKILL.

Give it a problem; it performs one revolution:

<SATURATION>   immerse in the interior of the problem, past the surface
<ABSTRACTION>  extract the irreducible axiom + a cross-domain isomorph   (shorter than saturation)
<DESCENT>      bring the abstraction down to concrete, testable consequences
<INTEGRATION>  compress to a synthesis that cannot be misread            (+ honest <CLICK> mark)
<REENTRY>      the gyre turns — the higher-level question the answer opens, + honest residue

A gyre is not a circle. A circle returns to where it began; a gyre returns to the same angle one turn higher. The question you re-enter is the one you left, displaced upward by exactly the understanding the last revolution deposited.

This model completes the rae-training repository, which shipped the methodology, loss design, and theory but no weights. engram-gyre is the weights.

Quick start

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

tok = AutoTokenizer.from_pretrained("TrueV1sion123/engram-gyre")
model = AutoModelForCausalLM.from_pretrained("TrueV1sion123/engram-gyre").eval()

msgs = [
    {"role": "system", "content": "You are an engram-gyre reasoner. Execute the gyre: "
     "SATURATION, ABSTRACTION, DESCENT, INTEGRATION, then re-enter one level up."},
    {"role": "user", "content": "Why do bull markets end in euphoria rather than fear?"},
]
text = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
ids = tok(text, add_special_tokens=False, return_tensors="pt")
out = model.generate(**ids, max_new_tokens=420, do_sample=True, temperature=0.8,
                     top_p=0.95, repetition_penalty=1.15,
                     eos_token_id=tok.convert_tokens_to_ids("<|end|>"),
                     pad_token_id=tok.convert_tokens_to_ids("<|pad|>"))
print(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=False).split("<|end|>")[0])

What it is — and what it is not

Read this before judging the outputs. Honesty about the gap between form and content is part of the engram-prime lineage (its "transmission ethics"), so it is stated plainly here.

• What is real: the model produces well-formed gyre revolutions on problems it never saw in training — four phases, correct order, the compression discipline preserved (abstraction shorter than saturation), and a reentry that opens a higher-level question. It learned this form as structure in its own weights, from scratch, on its own tokenizer.
• What is not claimed: the model does not reason. At ~11M parameters trained on a focused behavioral corpus, it has the choreography of deep thinking without the depth. On a novel problem it emits the shape of a gyre with content that is associatively plausible but not reliably true. The <CLICK> token, which marks genuine emergence in the corpus, is here a learned mark, not a genuine detection of irreducible residue.
• Where the depth lives: the repo ships a GPU scale-up kit (src/scale_up_sft.py, scripts/run_scale_up.sh, configs/) that applies the identical corpus, masking, and phase-weighting as LoRA SFT on a 1–2B pretrained base (SmolLM2-1.7B / Qwen2.5-1.5B). On a base that already has content, installing the gyre form on top yields a model that both reasons and spirals. This model proves the pipeline and the form end-to-end with real weights; the kit is the path from form to capability.

How the cognition is in the weights

Three mechanisms (full account in THEORY.md and docs/ADR-001-architecture.md):

1. Structure in the data. Every training example is one phase-tagged revolution with the compression axiom (|abstraction| < |saturation|) enforced in 100% of examples. The model never sees a counterexample, so compression becomes what the abstraction phase is.
2. Ordering & transition in the loss. A phase-position weighting puts double loss on phase boundary tokens and elevated loss on the abstraction / integration / reentry interiors — the model is pushed hardest at the register transitions and the synthesis. This adapts the RAE multi-objective loss to a single-tower from-scratch model.
3. Completion-only supervision. The prompt is masked; loss lives only on the gyre the model performs.

Architecture

Class	LlamaForCausalLM (from scratch, standard HF)
Parameters	~11.3M
Layers / hidden / heads	6 / 384 / 12 (GQA, 4 KV heads)
Intermediate	1024 (SwiGLU)
Context	640 (covers every training revolution incl. REENTRY)
Tokenizer	ByteLevel BPE, ~4.9k vocab, atomic phase tokens
Positional	RoPE
Tied embeddings	yes

Training

• Data: TrueV1sion123/engram-gyre-behavior — 1,980 train / 132 validation, validation = entire unseen seed problems.
• Objective: assistant-only causal LM with phase-position loss weighting.
• Schedule: 12 epochs, AdamW, cosine w/ warmup, lr 3e-4, weight decay 0.1, grad clip 1.0, effective batch 32.
• Hardware: 2 CPU cores (no GPU) — a deliberate constraint; the whole point is real weights within a modest budget. ~2 hours wall-clock.

Results

The central finding — structure generalizes, content does not. Held-out validation loss (validation = entire unseen problems) bottomed at epoch 2 and then rose and plateaued while train loss collapsed toward zero. That is content overfitting on a deliberately tiny seed set, and it is expected: an 11M model cannot learn the content of problems it never saw. But structural reliability moved the opposite way — the phase grammar is reinforced every epoch, so it keeps sharpening long after content perplexity saturates:

Checkpoint	Selected by	Structural validity
epoch 2 (best val-loss)	val loss	12% — tags present but sloppy (nesting, compression violations)
epoch 12 (shipped)	structural validity	100% — clean, ordered, compression held

The shipped model is therefore selected by structural validity, the launch gate — not by val loss (evaluation/select_best.py). Selecting on val loss would have shipped the 12%-valid epoch-2 weights.

Launch gate	Result
G2 — structural validity ≥ 80%	PASS — 100% (8/8 held-out + novel prompts; all of: 4 phases present, ordered, |ABSTRACTION| < |SATURATION|)
Overall val perplexity	277 (high = honest unseen-content overfit)
Per-phase PPL	SATURATION 663 · ABSTRACTION 339 · DESCENT 340 · INTEGRATION 107

That last row is the quiet win: INTEGRATION is ~6× more predictable than SATURATION. Compression lowers entropy — the model internalized the very gradient the corpus was built to teach (immerse wide, compress narrow).

A full gyre revolution (unseen prompt)

Prompt: "Why do simple rules generate complex behavior?" — decoded at temperature 0.78. The model produces the complete five-movement spiral, with the honest <CLICK> emergence mark and the REENTRY that turns the gyre:

<SATURATION>
- The suddenness is a property of the collective, not the parts; no single molecule 'decides' to melt.
- Small input changes produce small effects far from threshold and enormous effects near it — sensitivity diverges.
- Below the critical point, local interactions dominate and the system stays in its current order.
</SATURATION>
<ABSTRACTION>
What cannot be removed without loss: Threshold behavior is a collective property — below saturation only
local change is possible; at criticality a single fluctuation reorganizes the whole. Emergence has a sharp onset.
Cross-domain resonance: crowd panic, bank runs, and grokking — nothing, nothing, nothing, then everything at once.
</ABSTRACTION>
<DESCENT>
- Look for the critical point; that is where cheap actions move systems.
- Emergence in cognition has the same threshold structure — below saturation only recombination, above it synthesis.
</DESCENT>
<INTEGRATION>
Phase transitions are sudden because order is collective: correlation length diverges at criticality, so a
smooth input yields a discontinuous, whole-system flip.
<CLICK> Two structures locked to reveal a third that was latent in both.
</INTEGRATION>
<REENTRY>
Re-entry — one level up: If reorganization is always threshold-gated, is the real skill locating a system's
critical point rather than pushing harder everywhere?
Honest residue: predicting the exact critical point in messy real systems remains hard.
</REENTRY>

Note what this shows and doesn't: the form is flawless on a prompt never seen in training; the content is retrieved from the model's nearest pattern (here, threshold/criticality — apt for "simple rules → complex behavior," but the model is pattern-matching, not reasoning). That is the honest state of an 11M demonstrator, and it is why the scale-up kit exists.

Files

engram-gyre/
├── README.md                    # this model card
├── THEORY.md                    # the gyre / engram theory + honest scope
├── config.json, model.safetensors, tokenizer.json, ...   # the weights + tokenizer (repo root)
├── loss_curve.png               # training/validation curves
├── docs/                        # PRD, ADR (architecture + constraint negotiation), validation report
├── src/
│   ├── corpus_seeds.py          # the hand-authored seed bank (the substance)
│   ├── generate_corpus.py       # deterministic corpus generator
│   ├── train_tokenizer.py       # phase-aware BPE tokenizer trainer
│   ├── train_gyre.py            # from-scratch trainer (masking + phase-position loss)
│   ├── generate.py              # run one gyre revolution
│   └── scale_up_sft.py          # GPU LoRA SFT of a 1-2B base on the same corpus
├── configs/                     # scale-up base registry + AutoTrain yaml
├── evaluation/                  # eval harness + reports (eval_report.json, showcase.json)
└── scripts/                     # launch + scale-up scripts

Lineage & license

engram-prime (MINDSKILL) → recursive-abstraction-engine → rae-training → engram-gyre. Authored by Jared Peck / GoldenStack. Apache-2.0.