README.md

---
license: mit
base_model: scratch
library_name: transformers
pipeline_tag: text-generation
tags:
  - gpt2
  - manifold
default_dtype: float16
---

# Structural Manifold GPT (300M · 512B windows)

## TL;DR
- Decoder-only transformer (16 layers · 16 heads · 1024-dim) trained **from scratch** on structural manifold signatures extracted from WikiText-103 (raw) using 512 B windows (384 B stride, precision 3).
- Each "token" is a quantised `(coherence, stability, entropy, hazard)` signature with deduplicated prototypes → **~42× byte compression** and 512-signature context ≈ 20k raw tokens.
- Fits/trains on a single RTX 3080 Ti in <30 min with FP16 + gradient checkpointing.

```
Final metrics (eval split = 2%)
-------------------------------------------
  eval_loss      = 6.6456
  perplexity     = 7.69e+02 (on manifold tokens)
  samples        = 76 (3 766 total sequences · 3 epochs)
  training time  ≈ 25 min @ 1.3 s/step (GPU)
```

Use this repo if you want to **benchmark manifold LMs** or integrate the encoder/decoder stack into small-context devices. For full pipeline (dataset prep, compression scripts, benchmarking), clone https://github.com/SepDynamics/structural-manifold-compression.

---

## New: STM-FineMath 124M (revision `finemath-124m`)
- 12-layer (124M) manifold LM trained on the 10 GB FineMath STEM corpus using the same `window=512 B`, `stride=384 B`, `precision=3` codec. The builder yields 50 242 samples / 25.7 M manifold tokens (≈0.27 raw tokens per signature) and fits in ~66 minutes on a single RTX 3080 Ti.
- Final eval loss `6.506` -> manifold perplexity `6.69e2`, while GPT-2 medium on the identical math slice lands at `7.75e3` (11.7x worse). See `benchmarks/finemath_perplexity_compare.json` in the new branch.
- Exact-match signature accuracies on standard math QA benchmarks (strict metric) are now recorded:

| Benchmark | Split | Subset | #Problems | Accuracy |
|-----------|-------|--------|-----------|----------|
| dim/competition_math | train | Algebra | 200 | 0.5% |
| dim/competition_math | train | Number Theory | 200 | 0.5% |
| dim/competition_math | train | Geometry | 200 | 1.0% |
| dim/competition_math | train | Prealgebra | 200 | 0.0% |
| openai/gsm8k (main) | test | - | 200 | 0.5% |

Each JSON artifact (command + parameters) is stored under `benchmarks/` on the `finemath-124m` branch.

### Loading the math-focused checkpoint
```python
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "scrallex/structural-manifold-compression"
tokenizer = AutoTokenizer.from_pretrained(repo_id, revision="finemath-124m")
model = AutoModelForCausalLM.from_pretrained(repo_id, revision="finemath-124m")
```
You must supply structural-manifold signatures instead of raw text. Use `scripts/data/prepare_causal_dataset.py` or `scripts/experiments/math_qa_demo.py` from the GitHub repo to encode prompts, then decode generations via `scripts/experiments/decode_signatures.py` (requires a prototype cache built from a math corpus).

### Reproducing the posted math benchmarks
```
python scripts/experiments/eval_math_dataset.py \
  --model output/training_runs/stm_finemath_10gb \
  --dataset-vocab output/stm_stem_finemath_10gb/vocab.json \
  --dataset dim/competition_math --split train --subset Algebra \
  --max-problems 200 --text-root data/raw_math/finemath_4plus_10gb \
  --cache output/training_runs/stm_finemath_10gb/prototype_cache.json \
  --max-new-tokens 64 --match-mode signatures
```
Switch `--dataset openai/gsm8k --dataset-config main --split test --question-field question --answer-field answer` for GSM8K. The evaluator now matches answers via signature subsequences (no prototype recovery required) so results are strict exact-matches.

---

## Quick Start & Best-Case Workloads
- **Optimised for structured text**: PDF/page OCR exports, news briefs, technical audits—any corpus where 512 B sliding windows capture repeated structure. Expect 30–60× token compression with 94–95 % token accuracy (Fox EN/CN/OmniDoc numbers from the main repo).
- **Reproduce on a sample corpus**:
  ```bash
  git clone https://github.com/SepDynamics/structural-manifold-compression.git
  cd structural-manifold-compression
  python scripts/experiments/benchmark_eval.py \
    --dataset briefs=examples/structured_demo/news_sample.jsonl \
    --json-text-key text \
    --window-bytes 512 --stride-bytes 384 --precision 3 \
    --use-native \
    --output-dir output/benchmark_runs/news_demo
  ```
  Inspect `output/benchmark_runs/news_demo/briefs.json` for compression, fidelity, and verification stats, then replace `news_sample.jsonl` with your own JSONL dumps.
- **Future Hugging Face Space**: a Gradio front-end (planned) will wrap the same workflow so newcomers can upload JSONL/txt, run compression, and view reconstructions/verification without installing CUDA.

---

## Latest Benchmarks (2025-11-05)
All evaluations ran on a single RTX 3080 Ti (12 GB) with the same environment as the training run. The raw artifacts referenced below are stored in `benchmarks/`.

### Structural compression (WikiText quick-20k slice)
- Command: `python scripts/experiments/benchmark_eval.py --dataset wikitext=data/raw_text/wikitext_train.jsonl --window-bytes 512 --stride-bytes 384 --precision 3 --max-documents 20000 --use-native`.
- Observations: 20 000 manifest entries contained **12 894 non-empty docs**, producing **21 234 windows** with **3 737 shared signatures**.
- Capacity: **5.88 MB → 191 KB** ⇒ **30.8× byte compression**; **60.7× token compression (stream)** / **60.8× unique** ⇒ a 512-signature context effectively covers ≈31 k GPT-2 tokens.
- Fidelity: **83.7 % token accuracy, 80.5 % recall, 82.0 % F1**; character accuracy **83.6 %**.
- Verification: false-positive rate **2.3 × 10⁻⁴** with perfect recall on positive windows.
- Runtime: completes within minutes on the 3080 Ti when the native kernel is enabled.
- Artifact: `benchmarks/wikitext_structural_20k.json`.

### GPT-2 perplexity comparison (manifold LM vs. GPT-2 medium)
- Command: `python scripts/experiments/perplexity_compare.py --manifold-model output/training_runs/wikitext_manifold_gpt --manifold-dataset output/wikitext_manifold/hf_dataset --manifold-vocab output/wikitext_manifold/vocab.json --manifold-eval-fraction 0.25 --gpt2-model gpt2-medium --gpt2-max-documents 10000 --output output/benchmark_runs/wikitext_perplexity_8h.json`.
- Manifold LM: **941 sequences / 4.81 × 10⁵ manifold tokens** → **loss 6.60, perplexity 7.33 × 10²**.
- GPT-2 medium: **10 000 raw documents / 6.36 × 10⁵ tokens** → **loss 9.25, perplexity 1.04 × 10⁴** (50× higher perplexity on the raw token stream).
- Effective compression proxy (`raw_tokens / manifold_tokens`) during evaluation = **1.34×**, indicating GPT-2 still consumed 34 % more tokens even before exploiting deduplication.
- Runtime: ≈3 min wall-clock on the 3080 Ti (majority spent on the manifold forward pass).
- Artifact: `benchmarks/wikitext_perplexity_8h.json`.

---

## Files
| Path | Notes |
|------|-------|
| `model.safetensors` | 300 M parameter GPT2LMHeadModel trained on manifold signatures |
| `config.json` | Model architecture (n_layer=16, n_embd=1024, vocab_size=11839, pad/bos/eos id = 11838) |
| `generation_config.json` | Default sampling config (max_length=512) |
| `tokenizer.json` / `tokenizer_config.json` / `special_tokens_map.json` | Word-level tokenizer whose vocab is exactly the manifold signature strings + `<pad>` |
| `vocab.json` | Original signature list emitted by `scripts/data/prepare_causal_dataset.py` |
| `training_args.bin` / `trainer_state.json` | Hugging Face Trainer metadata (seeds, LR schedule, grad norms) |
| `eval_metrics.json` | Recomputed eval loss & perplexity over the 2% hold-out |

---

## Data & Compression Pipeline
1. **Raw text → signatures**: run the encoder on WikiText (or any UTF-8 corpus) via
   ```bash
   python scripts/data/prepare_causal_dataset.py \
     --text-root data/raw_text/wikitext_train.jsonl \
     --output-dir output/wikitext_manifold \
     --window-bytes 512 --stride-bytes 384 --precision 3 \
     --sequence-length 512 --min-sequence-length 8 \
     --use-native --concat-documents --export-signatures --reset-output
   ```
   This keeps an append-only `samples.jsonl` + `vocab.json` so you can resume mid-run.
2. **Sequences → HF dataset**: the builder automatically materialises `output/wikitext_manifold/hf_dataset` with `input_ids`/`labels` for causal LM.
3. **Training**: the published checkpoint comes from
   ```bash
   CUDA_VISIBLE_DEVICES=0 python scripts/training/manifold_lm_trainer.py \
     --dataset-path output/wikitext_manifold/hf_dataset \
     --vocab-path output/wikitext_manifold/vocab.json \
     --output-dir output/training_runs/wikitext_manifold_gpt \
     --n-layer 16 --n-head 16 --n-embd 1024 --context-length 512 \
     --per-device-train-batch-size 2 --per-device-eval-batch-size 2 \
     --gradient-accumulation-steps 16 --num-train-epochs 3 \
     --learning-rate 2e-4 --warmup-steps 500 --gradient-checkpointing --fp16 --resume
   ```

Hardware: single RTX 3080 Ti (12 GB). Training logs: `output/training_runs/wikitext_manifold_gpt/train.log` in the main repo.

---

## Usage
> ⚠️ This model expects **manifold signatures** (not raw text). Before inference, run the encoder to obtain the signature vocabulary and ID sequences.

```python
import json
import torch
from transformers import AutoTokenizer, GPT2LMHeadModel

repo_id = "scrallex/structural-manifold-compression"

tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = GPT2LMHeadModel.from_pretrained(repo_id, torch_dtype=torch.float16).to("cuda").eval()

def manifold_signatures_to_ids(signatures):
    # signatures = list of strings emitted by the encoder (e.g. 'c0.018_s0.481_e0.982')
    return tokenizer.convert_tokens_to_ids(signatures)

signatures = ["c0.018_s0.481_e0.982", "c0.012_s0.496_e0.988", "c0.017_s0.502_e0.972", ...]
input_ids = manifold_signatures_to_ids(signatures)
inputs = torch.tensor([input_ids[:-1]], device=model.device)
with torch.no_grad():
    outputs = model.generate(inputs, max_new_tokens=64)
next_signature_ids = outputs[0, len(input_ids)-1:]
next_signatures = tokenizer.convert_ids_to_tokens(next_signature_ids.tolist())
```

To reconstruct human-readable text, feed predicted signatures back through the manifold decoder (see `scripts/experiments/manifold_compression_eval.py`).

---

## Evaluation
- Training split: 3 690 sequences (98% of dataset) · Eval: 76 sequences (2%).
- Final eval loss `6.6456` → perplexity `≈7.7e2` on manifold tokens (see `eval_metrics.json`).
- Structural slice (`benchmarks/wikitext_structural_20k.json`): 512 B windows / 384 B stride / precision 3 on the first 20 k WikiText entries ⇒ 30.8× byte compression, 60.7× token compression, 83.7 % token accuracy, 83.6 % character accuracy, verification FPR 2.3 × 10⁻⁴.
- GPT-2 comparison (`benchmarks/wikitext_perplexity_8h.json`): manifold LM perplexity 7.33 × 10² vs. GPT-2 medium 1.04 × 10⁴ over 10 k raw documents (raw tokens over manifold tokens = 1.34× during the shared evaluation).

Future work: increase sequence budget (1k+ signatures), add rotary embeddings for better long-context, benchmark against GPT-2 (raw) to quantify effective perplexity after reconstruction.

---

## Responsible Use & Limitations
- The model memorises WikiText-103 content; outputs may regurgitate training passages.
- Tokens are structural signatures only—**you must keep the encoder/decoder kill switches** to avoid leaking the underlying text when using proprietary corpora.
- No guardrails, toxicity filtering, or multilingual tuning beyond what WikiText provides.

Report issues or ideas via https://github.com/SepDynamics/structural-manifold-compression.