Learning from a Deterministic Oracle Inside the Model

A CPU inside the weights teaches the model, exactly, and we measure where the memory stops

Experimental personal research — not peer-reviewed. Several experiments failed. They are reported with the same detail as the successes, including one apparent positive that a control refuted. Read Limitations before citing anything here.

Author: Marcelo Ernesto Burgos Cayupil · Date: November 2026 · License: MIT


TL;DR

A language model learns new facts with no gradients, using as ground truth a deterministic RISC-V CPU compiled into the weights of the same checkpoint.

  1. The oracle works. A 177M-parameter RV32IM circuit (recovered from a checkpoint whose runtime was lost) computes eight 18-digit products exactly, verified against Python.
  2. The loop works. Eight facts written into lm_head with ΔW = YᵀK(KᵀK+λI)⁻¹ move character accuracy from 0/71 to 71/71, against a shuffled-target control of 1/71, with a 6 MB adapter that removes cleanly.
  3. Capacity is key diversity, not fact count. Forty-two constraints from independent prompts collapse; 391 constraints from a chained derivation fit exactly. Keys of different facts are nearly identical (cosine 0.95 median), so all signal lives in differences of 0.01.

1. The oracle

oracle

The checkpoint shipped without its driver (the repository returns 404). We recovered it from the weights alone: a four-line stub for the missing class, then the layout formula

c     = core.build(Target(16, 32, 9216), BR.Text(tokenizer.json))   # 2918 dims, 2492 nodes
alloc = allocate(c, None, set(c.keep), 9216)                        # 1265 slots
lay   = Layout(hidden=3328, off=2048, m=13, slots=alloc.slots)      # col = 2062 + slot

which predicts the checkpoint's residual columns (2193 for the first code dim, 2215 for mu, 2216 for tau) without consulting it. Seven checks, seven matches.

Expression Circuit output Python
52445 × 3471 182036595 182036595 ✅
61750 × 1791 110594250 110594250 ✅
19494 × 9779 190631826 190631826 ✅
22337 × 6991 156157967 156157967 ✅
86387 × 1950 168454650 168454650 ✅
76510 × 4517 345595670 345595670 ✅
14914 × 2408 35912912 35912912 ✅
66838 × 7851 524745138 524745138 ✅

Mean latency ≈ 70 s per product on a laptop 3070.


2. The learning loop

loop

For each answer token j: key k_j is the post-norm hidden state at position len(prompt)-1+j, target gap γ_j = logit(top) − logit(target) + 3 (capped at 12). Then ΔW = YᵀK(KᵀK + λI)⁻¹, applied as W ← W + αΔW.

Condition Characters correct Full answers
Base 0 / 71 0 / 8
ΔW 71 / 71 8 / 8 (as prefix)
Control (shuffled targets, same norm) 1 / 71 0 / 8
Rollback 0 / 71 0 / 8
base:     ' = 181601490'      (wrong; true value 182036595)
patched:  ' 1820365956595'    (exact, then the model keeps writing)

Adapter: rank 12, 6.0 MB, stored as U·S·Vᵀ.


3. Capacity

capacity

Facts (independent prompts) Pairs rank(ΔW) Accuracy
5 10 10 5/5
10 18 16 3/10
15 28 26 9/15
20 37 34 6/20

Non-monotonic: ten facts score worse than fifteen. The measurement that explains it:

cosine between keys of different facts:  median 0.95   max 0.99

Prompts sharing a template produce nearly identical states. A rank-limited linear map competes for the same directions, and the outcome depends on which names happen to be in the set.

The converse confirms it: a 65-token derivation taught across six examples produced 391 constraints that fit without collapse, because chained keys are correlated and cheap to cover.

Rule of thumb. Chained sequences are cheap; independent facts are expensive. Quote capacity in key diversity, not in facts.


4. Four failures

4.1 No generalization to paraphrases

The delta is keyed on exact states; an unseen phrasing produces an unseen key and the update contributes nothing beyond base level.

4.2 No procedure learning

Teaching a three-step derivation for six products (intermediate values produced by the oracle) memorizes the derivation — correct decomposition, correct first partial product — and scores 0/3 on unseen products.

4.3 Composition: a false positive caught by a control

composition

With two chains (A→B, B→C), both direct facts were learned (4/4). Asked for A→C the model answered correctly in 1 of 2 cases — which looks like composition.

It is not. Three names never seen in training also answer Vela:

river of Zorblax → 'Vela'      (trained)
river of Velmora → 'Velmora'   (trained)
river of Fennax  → 'Vela'      (NEVER trained)
river of Ossric  → 'the Vela'  (NEVER trained)
river of Draven  → 'Vela'      (NEVER trained)

The update biased the output distribution globally. Composition: 0. Without the control, this paper would have claimed a capability that does not exist.

4.4 Internal layers, and a soft memory graft

A crude MEMIT-style edit of mlp.down_proj across five layers had no effect (0/20): the targets ignored the final normalization, so this is an implementation failure, not evidence against internal edits.

A soft memory graft at layer 8 retrieved the correct fact 10/10 and produced the correct first token 10/10, but failed end to end (0.20): re-injecting a fixed direction at every decoding step either destabilizes the residual or leaves it unchanged.

Both match the pattern of §2: the unit is the pair (prefix, next token); one vector per fact cannot sustain a sequence.


5. Related work

System Closed-form Exact ground truth No gradients Fits 8 GB
ROME / MEMIT ✅ ❌ dataset ✅ ✅
SEAL ❌ SFT ❌ self-generated ❌ —
HOPE / Nested Learning ❌ ❌ ❌ —
JitRL — ❌ ✅ —
Qwenputer — ✅ in-weights CPU — ❌ 18.8 GB
This work ✅ ✅ in-weights CPU ✅ ✅ 0.35 GB circuit

The update rule is MEMIT's. What is new is the composition: an exact computer as the teacher, closed-form algebra as the writer, and an explicit map of where the memory stops working.


6. Limitations

  • One model, one task family (invented facts and integer products).
  • Single seed for the capacity sweep; variance is reported, not averaged.
  • The 2B language half of the source checkpoint is not runnable (quantized weights, missing wrapper). Only the circuit was recovered.
  • Composition is n=2, max — small, and negative after control.
  • The MEMIT-style edit is a crude approximation and should not be read as a refutation.
  • No claim of first-ness. The components are prior work; the composition is what we believe to be new, and we found no published precedent.

7. Reproduce

python experiments/e1_oracle_exact.py
python experiments/e2_loop_8facts.py
python experiments/e3_capacity.py
python experiments/e5_composition.py
python figures/make_figures.py

Hardware: RTX 3070 Laptop 8 GB, i7-11800H, 31.7 GB RAM. Stack: torch 2.13+cu126, transformers 5.22.2. Raw numbers: results/results.json.


8. Resumen en español

Un modelo de lenguaje aprende hechos nuevos sin gradientes, usando como fuente de verdad un CPU RISC-V determinista compilado dentro de los pesos del mismo checkpoint.

El circuito se reconstruyó desde cero (el runtime original ya no está publicado) y se verificó: 8 de 8 productos de 18 dígitos exactos. Con esa verdad se calculó un delta de forma cerrada sobre lm_head: el modelo pasó de 0/71 a 71/71 caracteres, con control de targets barajados en 1/71 y rollback limpio. El adapter pesa 6 MB.

Lo que no funciona está reportado con el mismo detalle: paráfrasis, composición (un falso positivo que un control descartó), edición de capas internas con una aproximación cruda, y el injerto de memoria más allá del primer token.

Investigación personal experimental. No revisada por pares, no es un producto, no reclama estado del arte.


Citation

@software{burgos2026oracle,
  author = {Burgos Cayupil, Marcelo Ernesto},
  title  = {Learning from a Deterministic Oracle Inside the Model},
  year   = {2026},
  note   = {Experimental personal research, not peer-reviewed},
  url    = {https://huggingface.co/mburgc/qwenputer-mirror-results}
}
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support