Instructions to use Topovoid/TopoVoid-0.2B-Orthogonal-Preview with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Topovoid/TopoVoid-0.2B-Orthogonal-Preview with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="Topovoid/TopoVoid-0.2B-Orthogonal-Preview", trust_remote_code=True) messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("Topovoid/TopoVoid-0.2B-Orthogonal-Preview", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use Topovoid/TopoVoid-0.2B-Orthogonal-Preview with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "Topovoid/TopoVoid-0.2B-Orthogonal-Preview" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Topovoid/TopoVoid-0.2B-Orthogonal-Preview", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/Topovoid/TopoVoid-0.2B-Orthogonal-Preview
- SGLang
How to use Topovoid/TopoVoid-0.2B-Orthogonal-Preview with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "Topovoid/TopoVoid-0.2B-Orthogonal-Preview" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Topovoid/TopoVoid-0.2B-Orthogonal-Preview", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "Topovoid/TopoVoid-0.2B-Orthogonal-Preview" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Topovoid/TopoVoid-0.2B-Orthogonal-Preview", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use Topovoid/TopoVoid-0.2B-Orthogonal-Preview with Docker Model Runner:
docker model run hf.co/Topovoid/TopoVoid-0.2B-Orthogonal-Preview
TopoVoid-0.2B-Orthogonal-Preview
- Organization: Topovoid (topovoid.com)
- Maintainers: tirpitzia, 0kaguya
- License: Apache-2.0
Overview
TopoVoid-0.2B-Orthogonal-Preview is a 212M-parameter bilingual (EN/ZH) causal language model preview of the TopoVoid architecture.
The architecture places explicit concept bottlenecks at Layer 7 and Layer 12, splitting the 1024-dim residual stream into a protected concept subspace (176 dims) and an orthogonal writable subspace (848 dims):
- Direct weight writing (
closed_form_write.py): Compiles new facts into upper-layer weights with zero optimizer steps and zero prompt overhead, while keeping Layer 7 concept representations intact (cosine = 1.000). - Long-context concept recall (
midcbm_infinite_context = true): Uses the native4,096-token window for short inputs (0.0logit overhead) and recalls evicted turns via middle-layer concept states beyond4,096tokens. - Low-label representation readout: Frozen Layer 7 and bottleneck features separate topics with minimal labels.
Architecture
| Item | Value |
|---|---|
| Parameters | 212,011,264 (tied embeddings) |
| Backbone | 16 layers, d_model = 1024, d_ff = 2432 (SwiGLU) |
| Attention | 16 query heads, 4 KV heads (GQA), QK-RMSNorm |
| Vocabulary | 49,152 active bilingual tokens (ChatML <|im_start|> / <|im_end|>) |
| Context | RoPE (4,096 native window) + default-on concept overflow recall |
| Concept Bottlenecks | L7 dense (24-d) + sparse (40-d / 256 atoms, Top-8), L12 highway (24-d) |
| Subspace Partition | 176 protected / 848 orthogonal writable dimensions (max leak = 1.34e-7) |
Quick Start
Tested with torch >= 2.8, transformers 4.57+ / 5.x, safetensors, and numpy.
ChatML & Long-Context Inference
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "Topovoid/TopoVoid-0.2B-Orthogonal-Preview"
tok = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True).to("cuda").eval()
messages = [
{"role": "system", "content": "You are a logical, concise assistant. Answer the user question directly and accurately."},
{"role": "user", "content": "What is your model name? Answer briefly."},
]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
prompt_ids = torch.tensor([model.encode_orig_ids(tok, prompt)], device="cuda")
with torch.autocast("cuda", dtype=torch.bfloat16):
out = model.generate(prompt_ids, max_new_tokens=128)
print(tok.decode(out[0, prompt_ids.shape[1]:], skip_special_tokens=False))
Demos (demo_write.py & demo_chat.py)
python demo_write.py --model-dir .
python demo_write.py --model-dir . --counterfact counterfact.json --n 12
python demo_chat.py --model-dir .
Benchmark Results
1. Closed-Form Weight Writing (CounterFact)
Direct weight compilation into Layers 8โ11 (0 optimizer steps, MidCBM & Sparse-256 cosine retention 1.000000). On 1 (ร100), 12, and 40 facts, both declarative rewrite Top-1 and prefilled ChatML stem Top-1 reach 100.00% (unprefilled free ChatML recall: 74.0% single, 41.7% batch-12, 45.0% batch-40).
Single-module capacity scaling (1 to 640 facts, first-target-token metrics):
| Facts / Module | Rewrite Top-1 | ES | PS | Paraphrase Top-1 | C4 PPL Ratio | Compile Time | Module Size |
|---|---|---|---|---|---|---|---|
| 1 (ร100) | 100.00% |
100.00% |
75.50% |
35.00% |
1.002 |
1.37 s |
0.09 MB |
| 12 | 100.00% |
100.00% |
91.67% |
41.67% |
1.015 |
2.80 s |
1.06 MB |
| 40 | 100.00% |
100.00% |
67.50% |
20.00% |
1.043 |
6.53 s |
3.53 MB |
| 160 | 95.63% |
100.00% |
59.06% |
21.25% |
1.115 |
22.24 s |
14.45 MB |
| 640 | 78.28% |
94.38% |
54.69% |
16.95% |
1.148 |
48.55 s |
57.51 MB |
Linearly stacking 2 independently compiled 40-fact modules (80 facts total) retains 96.25% Top-1 (1.112 C4 PPL ratio, 1.000 L7 concept cosine, 0.998 L12 highway cosine).
2. Long-Context Recall & External Concept Index
| Task | Result | Details |
|---|---|---|
Long-Context Overflow Recall (6.5Kโ11.4K tokens, 20 EN/ZH sessions) |
100.0% chunk recall (20/20), 85.0% exact value recall (17/20: 8/10 EN, 9/10 ZH) |
Mean 8,947 input tokens; 0 errors crossing 4,096; 0.0 logit diff at <= 4,096 tokens |
External Concept Index (ARC, 399 unseen question paraphrases) |
33.33% โ 68.42% (+35.09 pp) |
Base weights unchanged (3,307 indexed science questions, recall@1 = 66.17%) |
3. Low-Label Classification Readout (AG News & DBpedia-14)
Linear ridge readout on frozen features without modifying language model weights (5 seeds):
| Dataset | Labels / Class | CBM (320-d) |
Full L7 (1024-d) |
TF-IDF Baseline | Full L7 โ TF-IDF |
|---|---|---|---|---|---|
AG News |
10 |
71.70% |
76.84% |
56.97% |
+19.88 pp |
AG News |
100 |
79.63% |
83.97% |
80.90% |
+3.07 pp |
AG News |
Full (112k) |
85.32% |
89.00% |
91.95% |
-2.95 pp |
DBpedia-14 |
10 |
72.23% |
85.78% |
85.30% |
+0.47 pp |
DBpedia-14 |
Full (120k) |
91.20% |
96.84% |
98.31% |
-1.47 pp |
(TF-IDF baseline: Word+Char LinearSVC on 10/100 labels, Logistic Regression on full split. With fixed hyperparameters and 0 validation labels on AG News 10 labels/class: Full L7 75.37%, CBM 67.32%, TF-IDF SVM 56.70%.)
4. Zero-Shot Commonsense & Bilingual ChatML
Zero-shot accuracy across 9 open multiple-choice benchmarks (7,068 items):
| PIQA | HellaSwag | ARC-E | ARC-C | WinoGrande | CSQA | OpenBookQA | BoolQ-1k | XCOPA-zh | 9-Task Macro |
|---|---|---|---|---|---|---|---|---|---|
61.01% |
28.10% |
37.63% |
24.66% |
50.70% |
28.67% |
24.80% |
58.70% |
50.40% |
40.52% |
On 32 held-out EN/ZH ChatML prompts, <|im_end|> termination is 32/32 (100%) with 0/32 repetition loops (greedy rep_4 = 0.0361, default sampling rep_4 = 0.0039).
0.2B Industrial Multimodal Benchmarks (CAD & Acoustic)
Specialized multimodal variants built on this 0.2B backbone (TopoVoid-0.2B-CAD and TopoVoid-0.2B-Acoustic) have been evaluated head-to-head against GPT-6.1-Sol (high) on open-source industrial engineering benchmarks:
| Variant & Domain | Benchmark Scope | GPT-6.1-Sol (high) |
TopoVoid-0.2B |
Latency (TopoVoid vs GPT-6.1-Sol) |
Speedup |
|---|---|---|---|---|---|
TopoVoid-0.2B-CAD (3D/2D CAD inspection: .step/.igs/.dxf) |
62 repos, 1,597 drawings, 6,947 questions |
47.62% (3,308/6,947) |
99.94% (6,943/6,947) |
0.35 s vs 59.06 s / Q |
169.3ร |
TopoVoid-0.2B-Acoustic (modal & vibro-acoustic: UNV58/TDMS/MDF4) |
6 datasets, 340 recordings, 902 questions |
42.35% (382/902) |
99.89% (901/902) |
0.29 s vs 101.78 s / Q |
350.3ร |
Selected hard-module highlights:
- CAD (
TopoVoid-0.2B-CAD):ยฑZdraft wall thickness & undercuts100.00%vs4.0%(683Qs); complex B-Rep topology & dihedral edges100.00%vs6.9%(1,173Qs); multi-level assembly BOM & contact graph99.78%vs39.7%(453Qs); L4 large multi-body assemblies100.00%vs37.0%(392Qs). - Acoustic (
TopoVoid-0.2B-Acoustic): Kurtogram & squared envelope spectrum99.64%vs10.47%(277Qs); multi-mode cluster summary100.00%vs14.29%(56Qs); computed order tracking100.00%vs26.77%(254Qs); RUS mode-1 doublet fitting100.00%vs37.70%(61Qs); L4 recordings100.00%vs45.03%(493Qs).
Full whitepapers and industrial workflows are available at https://topovoid.com.
Notes & Limitations
- Research preview (
0.2Bscale): Parametric world knowledge, multi-step arithmetic, and general code generation are limited by model size. - Stacking & overflow scope: In this
0.2Bpreview, stacking more than2independently compiled modules causes interference, and overflow recall retrieves selective chunks rather than full attention over arbitrary lengths. - No safety alignment: Do not deploy directly in untrusted user-facing production.
License & Citation
Released under the Apache License 2.0 (see LICENSE).
@misc{topovoid2026orthogonal,
title = {TopoVoid-0.2B-Orthogonal-Preview: Native Orthogonal Subspace Architecture and Closed-Form Null-Space Weight Compilation},
author = {tirpitzia and 0kaguya and {TopoVoid Research}},
year = {2026},
howpublished = {\url{https://huggingface.co/Topovoid/TopoVoid-0.2B-Orthogonal-Preview}},
note = {Research Preview, \url{https://topovoid.com}}
}
- Downloads last month
- 762