README.md

---
language:
- en
license: mit
tags:
- post-transformer
- attention-free
- hamiltonian
- symplectic
- o1-memory
- pytorch
- sub-quadratic
pipeline_tag: text-generation
---

# TUPOI-300M: Symplectic Post-Transformer Language Model with O(1) Memory

<div align="center">

*A sub-quadratic, attention-free sequence modeling architecture that replaces dense attention matrices with a Symplectic Hamiltonian Integrator (Velocity-Verlet).*

**Author:** NARE LABS (*Built by 15 y.o. independent researcher*)

[![GitHub](https://img.shields.io/badge/GitHub-starface77%2FTUPOI-blue.svg)](https://github.com/starface77/TUPOI)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

</div>

---

## 📌 Overview

**TUPOI-300M** is a 304M-parameter proof-of-concept post-transformer architecture that completely eliminates the Attention mechanism and the persistent KV-cache in favor of continuous Hamiltonian phase-space dynamics.

### Key Highlights:
- **Strictly Constant Memory ($O(1)$):** The persistent recurrent context state requires exactly **6.00 KB of VRAM** forever, regardless of whether generating 512 or 65,536 tokens.
- **Zero Information Dissipation:** Integrated via the symplectic Velocity-Verlet leapfrog scheme with Jacobian determinant $\det(J) \equiv 1.000000$ (Liouville's theorem).
- **Pretrained Scale:** 304.2M active parameters trained across 10,000 steps on TinyStories on a single NVIDIA Tesla T4 GPU.

---

## 🚀 Quickstart: Running Inference in Python

```python
import torch
import torch.nn.functional as F
import tiktoken
from huggingface_hub import hf_hub_download

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# 1. Download model weights from HuggingFace
weights_path = hf_hub_download(repo_id="DanilKZ/TUPOI-1", filename="tupoi-300m.pt")

# 2. Clone repository code for architecture definitions:
# git clone https://github.com/starface77/TUPOI.git
from tupoi import TUPOI300M, generate_text

enc = tiktoken.get_encoding("gpt2")
model = TUPOI300M(vocab_size=enc.n_vocab, d_model=1024, num_layers=24, d_ff=4096, max_seq_len=512).to(device)

checkpoint = torch.load(weights_path, map_location=device)
sd = checkpoint["model_state_dict"] if "model_state_dict" in checkpoint else checkpoint
if "opinion_anchor.anchor_state" in sd:
    del sd["opinion_anchor.anchor_state"]

model.load_state_dict(sd, strict=False)
model.eval()

# 3. Generate text
prompt = "Once upon a time, there was a little girl named Lily who"
output = generate_text(model, enc, prompt=prompt, max_new_tokens=50, temperature=0.7)
print(output)