PixelGPT -- Geometry OS Assembly Generator

A bilingual (English + GeOS assembly) GPT-2 model that generates Geometry OS bytecode assembly from natural language descriptions.

Model Architecture

Version	Params	Layers	Heads	Embd	Context	Loss	Notes
V5	13.7M	6	8	384	1024	0.374	Fixed ByteLevel tokenizer
V6	29.3M	8	8	512	1024	0.907	Golden dataset, killed early (ep 2/40)
V8	29.3M	8	8	512	1024	0.076	Overfit to training surface form
V9	~4M	4	8	256	1024	0.019	Compact retrain

Tokenizer

Bilingual V4 tokenizer with three-tier ID space:

• 0-267: Atomic opcodes, registers, structural tokens (BOS/EOS/NL/COMMA/COLON)
• 270-351: Character-level literals (digits, letters, symbols)
• 356+: BPE text tokens (prose, comments, descriptions)

Round-trip fidelity: 89.6% exact, 94.5% preserved, 99.6% code-portion.

Usage

import torch
import sys, os
sys.path.insert(0, ".")
from bilingual_tokenizer import BilingualTokenizer
from train_opcode_llm import OpcodeGPT, generate_asm

# Load tokenizer
tokenizer = BilingualTokenizer.load("bilingual_tokenizer_v4/")

# Load model (V8 example)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
checkpoint = torch.load("bilingual_llm_v8_ckpt.pt", map_location=device)
m_args = checkpoint["args"]

model = OpcodeGPT(
    vocab_size=m_args["vocab_size"],
    n_embd=m_args["embd"],
    n_head=m_args["heads"],
    n_layer=m_args["layers"],
    block_size=m_args["context_len"]
).to(device)
model.load_state_dict(checkpoint["model"])
model.eval()

# Generate from natural language prompt
asm = generate_asm(model, tokenizer, "; Draw a red circle at the center", device, max_tokens=256, temperature=0.7)
print(asm)

Training Data

Trained on 5,211 annotated GeOS assembly programs (211 real + ~5,000 synthetic). Each program includes a natural language description comment. Dataset: bilingual_dataset.npz (7,824 samples, 2.91M tokens, context=1024, stride=512).

Geometry OS

PixelGPT targets the Geometry OS bytecode VM -- a pixel-native operating system with 150+ opcodes, 32 registers, and a 256x256 RGB framebuffer. Programs are written in a custom assembly language and assembled to bytecode.

Repo: https://github.com/tdw419/geometry_os

Status

Research preview. The model generates syntactically plausible assembly but does not yet produce consistently working programs. Active development is improving training data quality and constrained decoding.

Developed by

Built with Hermes Agent autonomous development pipeline on an NVIDIA RTX 5090.