ElBalor/jervis-lite-v0.1

Jervis Lite v0.1

A 302M parameter language model trained from scratch no pretrained backbone, no distillation, no fine-tuning of another model's weights. Built from random initialization on a single Kaggle T4 (16 GB) in ~10 hours of wall-clock training.

Every architectural component is novel:

• Quantum Self-Attention (QSA): complex projective attention replacing softmax
• Elbàlor Neuron (Architecture D): metabolic phase-space activation with hand-derived Wirtinger backward
• Eternal KV / Mirror Dimension: tagged, rotation-isometric memory that survives save/load and shuffling
• Complex-STDP: phase-based causal memory matrix
• Pearl rungs 1–3: observational, interventional, and counterfactual causal querying inside the neuron
• Larmor phase driver: per-feature frequencies giving the complex memory axis real temporal signal

What this is

A research artifact demonstrating that complex-valued attention and metabolic phase-space activation can be trained from scratch at 302M parameters without divergence on consumer hardware.

What this is not

• Not a competitor to GPT, Qwen, Llama, or any production model
• Not instruction-tuned, RLHF'd, or aligned
• Not intended for deployment

Results

Metric	Value
Parameters	302.2M
Training steps	5000 cumulative (2500 base + 2500 continuation)
Final train loss	2.62
Best validation loss	2.7703
Starting loss (uniform baseline)	11.93 (ln(151669))
Hardware	Kaggle Tesla T4 (16 GB)
Training time	~10 hours total
Precision	fp32
Optimizer	AdamW (continuation phase)
Effective batch size	32 (batch 4 × grad_accum 8)
Context length	256
Vocab	151,669 (Qwen 2.5 tokenizer)

Architecture

16 layers · d_model 512 · 8 heads · d_ff 1024 · complex-valued throughout

QSA (Quantum Self-Attention)

Complex Q/K/V/O projections. Hermitian score S = ⟨q,k⟩. Causal mask with 0 (not −inf). L2 normalization replaces softmax. Coherent superposition out = α·V. No softmax anywhere.

Elbàlor Neuron (Architecture D, Graduated)

• Magnitude gate Φ(r) = ½(1 + erf(r/√2))
• Thermal phase rotation: φ_total = thermal_phase + (1−p)·phase_mem
• Permeability p = 1 − tanh(T/τ_freeze), p_min = 0
• Blocked gradient → phase memory via backward hook
• Circular phase wrap via torch.remainder
• Graduated defensive response

Eternal KV / Mirror Dimension

Tagged KV store. Every entry carries its own absolute tag. Rotation at R_φ(tag). Recovery un-rotates by declared tag. Sliding window graduates evicted tokens into the buffer. Round-trip verified to < 1e-4.

Files

File	What it is
jervis_lite.pt	Best checkpoint (val 2.7703, continuation step 2250)
Jervis.py	Continuation training script + full model definition (self-contained)
Jervis-train.py	Base-run training script (first 2500 steps, from scratch)
config.json	Architecture + training hyperparameters

Jervis-train.py trains the base run from random init (LR 2e-4 → 2e-5 cosine, 2500 steps, saves to checkpoints/). Jervis.py resumes from that checkpoint and runs the continuation (LR 1e-4 → 1e-5, 2500 more steps, saves to checkpoints_v2/). Both files contain the complete model definition — QSA, Elbàlor FFN, Pearl rungs 1–3, Eternal KV — so either one is enough to load the released checkpoint.

Loading the checkpoint

Jervis.py is self-contained it defines the whole model (QSA, Elbàlor FFN, Mirror KV) and only needs torch, transformers (tokenizer), and datasets (training data):

import torch
from Jervis import JervisLite, Config

cfg = Config()
model = JervisLite(cfg)
ckpt = torch.load("jervis_lite.pt", map_location="cpu")
model.load_state_dict(ckpt["model"])
model.eval()

print("val:", ckpt["val"], "step:", ckpt["step"])

Generation (with the Eternal KV buffer active):

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
ids = tok.encode("Once upon a time", return_tensors="pt")
out = model.generate(ids, max_new=80)                  # standard
out = model.generate_with_eternal_kv(ids, max_new=80)  # Mirror Dimension retrieval
print(tok.decode(out[0], skip_special_tokens=True))

Sample generation

Prompt: Once upon a time

"Once upon a time, there was a little boy named Tom. He was a big and loved to play. Every day, he would go outside, and play with his mom and playing with his friends. One day, Lily and his mom gave him a kiss..."

Coherent entities across sentences. Grammatically valid. Narrative structure. Some clause-level garbling remains at val 2.77.

Training data

• roneneldan/TinyStories (~65%)
• A small synthetic persona corpus (~35%, unlabeled in this release)

Reproducibility

Jervis-train.py and Jervis.py contain the full training loop, all hyperparameters, the data pipeline, and the resume logic. The continuation run resumes from the base-run checkpoint with a lower LR schedule (1e-4 → 1e-5 cosine). Two runs × 2500 steps = 5000 cumulative steps; the released checkpoint is the best continuation model (val 2.7703).

Papers & artifacts

• The Elbàlor Neuron (this model's activation + memory): Zenodo — doi.org/10.5281/zenodo.23370897 · GitHub
• Quantum Self-Attention (this model's attention): Zenodo — zenodo.org/records/23117793
• Related: The Therma Neuron · Pure W-Expansion

Citation

@misc{yaka2026jervisv01,
  title  = {Jervis Lite v0.1: A From-Scratch Complex-Valued Language Model with Quantum Self-Attention and the Elbàlor Neuron},
  author = {Yaka, Eric (Elbàlor)},
  year   = {2026},
  note   = {Independent research, Abuja, Nigeria}
}

From the Grimoire of Elbàlor, the Digital Necromancer.