TopoVoid-0.2B-Orthogonal-Preview

Overview

TopoVoid-0.2B-Orthogonal-Preview is a 212M-parameter bilingual (EN/ZH) causal language model preview of the TopoVoid architecture.

The architecture places explicit concept bottlenecks at Layer 7 and Layer 12, splitting the 1024-dim residual stream into a protected concept subspace (176 dims) and an orthogonal writable subspace (848 dims):

  • Direct weight writing (closed_form_write.py): Compiles new facts into upper-layer weights with zero optimizer steps and zero prompt overhead, while keeping Layer 7 concept representations intact (cosine = 1.000).
  • Long-context concept recall (midcbm_infinite_context = true): Uses the native 4,096-token window for short inputs (0.0 logit overhead) and recalls evicted turns via middle-layer concept states beyond 4,096 tokens.
  • Low-label representation readout: Frozen Layer 7 and bottleneck features separate topics with minimal labels.

Architecture

Item Value
Parameters 212,011,264 (tied embeddings)
Backbone 16 layers, d_model = 1024, d_ff = 2432 (SwiGLU)
Attention 16 query heads, 4 KV heads (GQA), QK-RMSNorm
Vocabulary 49,152 active bilingual tokens (ChatML <|im_start|> / <|im_end|>)
Context RoPE (4,096 native window) + default-on concept overflow recall
Concept Bottlenecks L7 dense (24-d) + sparse (40-d / 256 atoms, Top-8), L12 highway (24-d)
Subspace Partition 176 protected / 848 orthogonal writable dimensions (max leak = 1.34e-7)

Quick Start

Tested with torch >= 2.8, transformers 4.57+ / 5.x, safetensors, and numpy.

ChatML & Long-Context Inference

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Topovoid/TopoVoid-0.2B-Orthogonal-Preview"
tok = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True).to("cuda").eval()

messages = [
    {"role": "system", "content": "You are a logical, concise assistant. Answer the user question directly and accurately."},
    {"role": "user", "content": "What is your model name? Answer briefly."},
]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
prompt_ids = torch.tensor([model.encode_orig_ids(tok, prompt)], device="cuda")

with torch.autocast("cuda", dtype=torch.bfloat16):
    out = model.generate(prompt_ids, max_new_tokens=128)

print(tok.decode(out[0, prompt_ids.shape[1]:], skip_special_tokens=False))

Demos (demo_write.py & demo_chat.py)

python demo_write.py --model-dir .
python demo_write.py --model-dir . --counterfact counterfact.json --n 12
python demo_chat.py --model-dir .

Benchmark Results

1. Closed-Form Weight Writing (CounterFact)

Direct weight compilation into Layers 8โ€“11 (0 optimizer steps, MidCBM & Sparse-256 cosine retention 1.000000). On 1 (ร—100), 12, and 40 facts, both declarative rewrite Top-1 and prefilled ChatML stem Top-1 reach 100.00% (unprefilled free ChatML recall: 74.0% single, 41.7% batch-12, 45.0% batch-40).

Single-module capacity scaling (1 to 640 facts, first-target-token metrics):

Facts / Module Rewrite Top-1 ES PS Paraphrase Top-1 C4 PPL Ratio Compile Time Module Size
1 (ร—100) 100.00% 100.00% 75.50% 35.00% 1.002 1.37 s 0.09 MB
12 100.00% 100.00% 91.67% 41.67% 1.015 2.80 s 1.06 MB
40 100.00% 100.00% 67.50% 20.00% 1.043 6.53 s 3.53 MB
160 95.63% 100.00% 59.06% 21.25% 1.115 22.24 s 14.45 MB
640 78.28% 94.38% 54.69% 16.95% 1.148 48.55 s 57.51 MB

Linearly stacking 2 independently compiled 40-fact modules (80 facts total) retains 96.25% Top-1 (1.112 C4 PPL ratio, 1.000 L7 concept cosine, 0.998 L12 highway cosine).

2. Long-Context Recall & External Concept Index

Task Result Details
Long-Context Overflow Recall (6.5Kโ€“11.4K tokens, 20 EN/ZH sessions) 100.0% chunk recall (20/20), 85.0% exact value recall (17/20: 8/10 EN, 9/10 ZH) Mean 8,947 input tokens; 0 errors crossing 4,096; 0.0 logit diff at <= 4,096 tokens
External Concept Index (ARC, 399 unseen question paraphrases) 33.33% โ†’ 68.42% (+35.09 pp) Base weights unchanged (3,307 indexed science questions, recall@1 = 66.17%)

3. Low-Label Classification Readout (AG News & DBpedia-14)

Linear ridge readout on frozen features without modifying language model weights (5 seeds):

Dataset Labels / Class CBM (320-d) Full L7 (1024-d) TF-IDF Baseline Full L7 โˆ’ TF-IDF
AG News 10 71.70% 76.84% 56.97% +19.88 pp
AG News 100 79.63% 83.97% 80.90% +3.07 pp
AG News Full (112k) 85.32% 89.00% 91.95% -2.95 pp
DBpedia-14 10 72.23% 85.78% 85.30% +0.47 pp
DBpedia-14 Full (120k) 91.20% 96.84% 98.31% -1.47 pp

(TF-IDF baseline: Word+Char LinearSVC on 10/100 labels, Logistic Regression on full split. With fixed hyperparameters and 0 validation labels on AG News 10 labels/class: Full L7 75.37%, CBM 67.32%, TF-IDF SVM 56.70%.)

4. Zero-Shot Commonsense & Bilingual ChatML

Zero-shot accuracy across 9 open multiple-choice benchmarks (7,068 items):

PIQA HellaSwag ARC-E ARC-C WinoGrande CSQA OpenBookQA BoolQ-1k XCOPA-zh 9-Task Macro
61.01% 28.10% 37.63% 24.66% 50.70% 28.67% 24.80% 58.70% 50.40% 40.52%

On 32 held-out EN/ZH ChatML prompts, <|im_end|> termination is 32/32 (100%) with 0/32 repetition loops (greedy rep_4 = 0.0361, default sampling rep_4 = 0.0039).


0.2B Industrial Multimodal Benchmarks (CAD & Acoustic)

Specialized multimodal variants built on this 0.2B backbone (TopoVoid-0.2B-CAD and TopoVoid-0.2B-Acoustic) have been evaluated head-to-head against GPT-6.1-Sol (high) on open-source industrial engineering benchmarks:

Variant & Domain Benchmark Scope GPT-6.1-Sol (high) TopoVoid-0.2B Latency (TopoVoid vs GPT-6.1-Sol) Speedup
TopoVoid-0.2B-CAD (3D/2D CAD inspection: .step/.igs/.dxf) 62 repos, 1,597 drawings, 6,947 questions 47.62% (3,308/6,947) 99.94% (6,943/6,947) 0.35 s vs 59.06 s / Q 169.3ร—
TopoVoid-0.2B-Acoustic (modal & vibro-acoustic: UNV58/TDMS/MDF4) 6 datasets, 340 recordings, 902 questions 42.35% (382/902) 99.89% (901/902) 0.29 s vs 101.78 s / Q 350.3ร—

Selected hard-module highlights:

  • CAD (TopoVoid-0.2B-CAD): ยฑZ draft wall thickness & undercuts 100.00% vs 4.0% (683 Qs); complex B-Rep topology & dihedral edges 100.00% vs 6.9% (1,173 Qs); multi-level assembly BOM & contact graph 99.78% vs 39.7% (453 Qs); L4 large multi-body assemblies 100.00% vs 37.0% (392 Qs).
  • Acoustic (TopoVoid-0.2B-Acoustic): Kurtogram & squared envelope spectrum 99.64% vs 10.47% (277 Qs); multi-mode cluster summary 100.00% vs 14.29% (56 Qs); computed order tracking 100.00% vs 26.77% (254 Qs); RUS mode-1 doublet fitting 100.00% vs 37.70% (61 Qs); L4 recordings 100.00% vs 45.03% (493 Qs).

Full whitepapers and industrial workflows are available at https://topovoid.com.


Notes & Limitations

  • Research preview (0.2B scale): Parametric world knowledge, multi-step arithmetic, and general code generation are limited by model size.
  • Stacking & overflow scope: In this 0.2B preview, stacking more than 2 independently compiled modules causes interference, and overflow recall retrieves selective chunks rather than full attention over arbitrary lengths.
  • No safety alignment: Do not deploy directly in untrusted user-facing production.

License & Citation

Released under the Apache License 2.0 (see LICENSE).

@misc{topovoid2026orthogonal,
  title        = {TopoVoid-0.2B-Orthogonal-Preview: Native Orthogonal Subspace Architecture and Closed-Form Null-Space Weight Compilation},
  author       = {tirpitzia and 0kaguya and {TopoVoid Research}},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/Topovoid/TopoVoid-0.2B-Orthogonal-Preview}},
  note         = {Research Preview, \url{https://topovoid.com}}
}
Downloads last month
762
Safetensors
Model size
0.2B params
Tensor type
F32
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support