| --- |
| license: mit |
| base_model: none |
| tags: |
| - geometry-os |
| - pixelgpt |
| - assembly |
| - code-generation |
| - custom-code |
| --- |
| |
| # PixelGPT -- Geometry OS Assembly Generator |
|
|
| A bilingual (English + GeOS assembly) GPT-2 model that generates Geometry OS bytecode assembly from natural language descriptions. |
|
|
| ## Model Architecture |
|
|
| | Version | Params | Layers | Heads | Embd | Context | Loss | Notes | |
| |---------|--------|--------|-------|------|---------|------|-------| |
| | V5 | 13.7M | 6 | 8 | 384 | 1024 | 0.374 | Fixed ByteLevel tokenizer | |
| | V6 | 29.3M | 8 | 8 | 512 | 1024 | 0.907 | Golden dataset, killed early (ep 2/40) | |
| | V8 | 29.3M | 8 | 8 | 512 | 1024 | 0.076 | Overfit to training surface form | |
| | V9 | ~4M | 4 | 8 | 256 | 1024 | 0.019 | Compact retrain | |
|
|
| ## Tokenizer |
|
|
| Bilingual V4 tokenizer with three-tier ID space: |
| - **0-267**: Atomic opcodes, registers, structural tokens (BOS/EOS/NL/COMMA/COLON) |
| - **270-351**: Character-level literals (digits, letters, symbols) |
| - **356+**: BPE text tokens (prose, comments, descriptions) |
|
|
| Round-trip fidelity: 89.6% exact, 94.5% preserved, 99.6% code-portion. |
|
|
| ## Usage |
|
|
| ```python |
| import torch |
| import sys, os |
| sys.path.insert(0, ".") |
| from bilingual_tokenizer import BilingualTokenizer |
| from train_opcode_llm import OpcodeGPT, generate_asm |
| |
| # Load tokenizer |
| tokenizer = BilingualTokenizer.load("bilingual_tokenizer_v4/") |
| |
| # Load model (V8 example) |
| device = torch.device("cuda" if torch.cuda.is_available() else "cpu") |
| checkpoint = torch.load("bilingual_llm_v8_ckpt.pt", map_location=device) |
| m_args = checkpoint["args"] |
| |
| model = OpcodeGPT( |
| vocab_size=m_args["vocab_size"], |
| n_embd=m_args["embd"], |
| n_head=m_args["heads"], |
| n_layer=m_args["layers"], |
| block_size=m_args["context_len"] |
| ).to(device) |
| model.load_state_dict(checkpoint["model"]) |
| model.eval() |
| |
| # Generate from natural language prompt |
| asm = generate_asm(model, tokenizer, "; Draw a red circle at the center", device, max_tokens=256, temperature=0.7) |
| print(asm) |
| ``` |
|
|
| ## Training Data |
|
|
| Trained on 5,211 annotated GeOS assembly programs (211 real + ~5,000 synthetic). Each program includes a natural language description comment. Dataset: `bilingual_dataset.npz` (7,824 samples, 2.91M tokens, context=1024, stride=512). |
|
|
| ## Geometry OS |
|
|
| PixelGPT targets the Geometry OS bytecode VM -- a pixel-native operating system with 150+ opcodes, 32 registers, and a 256x256 RGB framebuffer. Programs are written in a custom assembly language and assembled to bytecode. |
|
|
| Repo: https://github.com/tdw419/geometry_os |
| |
| ## Status |
| |
| **Research preview.** The model generates syntactically plausible assembly but does not yet produce consistently working programs. Active development is improving training data quality and constrained decoding. |
| |
| ## Developed by |
| |
| Built with [Hermes Agent](https://github.com/nousresearch/hermes-agent) autonomous development pipeline on an NVIDIA RTX 5090. |
| |
