TUPOI-300M: Symplectic Post-Transformer Language Model with O(1) Memory
A sub-quadratic, attention-free sequence modeling architecture that replaces dense attention matrices with a Symplectic Hamiltonian Integrator (Velocity-Verlet).
Author: NARE LABS (Built by 15 y.o. independent researcher)
📌 Overview
TUPOI-300M is a 304M-parameter proof-of-concept post-transformer architecture that completely eliminates the Attention mechanism and the persistent KV-cache in favor of continuous Hamiltonian phase-space dynamics.
Key Highlights:
- Strictly Constant Memory ($O(1)$): The persistent recurrent context state requires exactly 6.00 KB of VRAM forever, regardless of whether generating 512 or 65,536 tokens.
- Zero Information Dissipation: Integrated via the symplectic Velocity-Verlet leapfrog scheme with Jacobian determinant $\det(J) \equiv 1.000000$ (Liouville's theorem).
- Pretrained Scale: 304.2M active parameters trained across 10,000 steps on TinyStories on a single NVIDIA Tesla T4 GPU.
🚀 Quickstart: Running Inference in Python
import torch
import torch.nn.functional as F
import tiktoken
from huggingface_hub import hf_hub_download
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
# 1. Download model weights from HuggingFace
weights_path = hf_hub_download(repo_id="DanilKZ/TUPOI-1", filename="tupoi-300m.pt")
# 2. Clone repository code for architecture definitions:
# git clone https://github.com/starface77/TUPOI.git
from tupoi import TUPOI300M, generate_text
enc = tiktoken.get_encoding("gpt2")
model = TUPOI300M(vocab_size=enc.n_vocab, d_model=1024, num_layers=24, d_ff=4096, max_seq_len=512).to(device)
checkpoint = torch.load(weights_path, map_location=device)
sd = checkpoint["model_state_dict"] if "model_state_dict" in checkpoint else checkpoint
if "opinion_anchor.anchor_state" in sd:
del sd["opinion_anchor.anchor_state"]
model.load_state_dict(sd, strict=False)
model.eval()
# 3. Generate text
prompt = "Once upon a time, there was a little girl named Lily who"
output = generate_text(model, enc, prompt=prompt, max_new_tokens=50, temperature=0.7)
print(output)