ATLAS-OLMo-3-32B-Think-v4

A 32B reasoning model on a single 40 GB GPU. OLMo-3-32B-Think (AI2) served by the ATLAS pure-Rust inference engine with AWQ 4-bit weight-only quantization — 27.2 GB VRAM on one A100-SXM4-40GB, through a zero-external-dependency Rust + custom-CUDA stack.

Status: ✅ In production. This 32B configuration is what the live endpoint (atlas.thebeastagi.com / demo.thebeastagi.com) serves today on a dedicated A100-40GB (astra-01). The W4 serving path is merged to main in web3guru888/ATLAS; 644/644 workspace tests green on main.

TL;DR

Question	Answer
What is this?	The serving configuration + measured results for OLMo-3-32B-Think running through ATLAS's custom W4A32 CUDA path
Does 32B fit on a 40 GB A100?	Yes — 27.2 GB total @ 16K context, 13.8 GB headroom (nvidia-smi: 27,170 / 40,960 MiB)
Is it smarter than the 7B?	Yes, by a lot — GSM8K 100% vs 88%, MMLU 82% vs 54%, same harness
Is it fast?	Decode is usable (~14.6 tok/s); long-prompt TTFT improved ~5× with batched prefill (#22, deployed 2026-07-08) — kernel-occupancy work continues (see Limitations)
Are the weights here?	No — weights are the unmodified community AWQ quant (cyankiwi/Olmo-3-32B-Think-AWQ-4bit); ATLAS is the engine, not a fine-tune

Benchmarks (measured on the A100-40GB, ATLAS stack)

Same harness and sampling as the 7B v4.2.0 numbers (temperature 0.6, top-p 0.95, AI2 reference system prompt, <think> primer):

Benchmark	32B AWQ-4bit (this config)	7B BF16 baseline	Notes
GSM8K (25 problems, same rows)	25/25 = 100%	88%	22.2K completion tokens total
MMLU (40 questions, 32 subjects)	33/40 = 82%	54% (MMLU-100, different sample)	4 of 7 misses were output-length truncations at the 1,500-token cap — accuracy ≈ 100% where reasoning completed
Code (5 tasks, execution-verified)	5/5 = 100%	HumanEval-15: 73.3% pass@1	fib, is_prime, reverse_words, two_sum, balanced-brackets — all pass their tests

⚠️ Sample sizes are honest but small. These were prototype-validation runs. Expanded suites (GSM8K-100, MMLU-200, HumanEval-50) will be run in a scheduled maintenance window on the production box; this card will be updated with the results and per-item JSON when they complete.

Performance & footprint (A100-SXM4-40GB)

Metric	32B W4 (this config)	7B BF16 (reference)
VRAM total @ 16K context	27.2 / 40 GB	~16 GB
Decode throughput (short ctx)	~14.6 tok/s	~50 tok/s
Decode @ ~1.2K ctx	10.4 tok/s	~45 tok/s
TTFT (short prompt)	~2–4 s (measured 3.9 s pre-#22; short prompts see little change)	~1.5 s
TTFT @ ~1.2K-token prompt	~23–37 s (was ~100–122 s before batched prefill #22 landed — merged & deployed 2026-07-08, ~5× measured)	~25 s
Model load time	~75 s (19.6 GB checkpoint, incl. GPU upload)	~200 s
Host RSS	~11 GB (streaming loader)	—

VRAM breakdown: W4 weights ~17.9 GB + BF16 lm_head ~1.0 GB + f32 KV cache ~8.6 GB (@16K) + scratch. Decode scales as expected for a memory-bandwidth-bound GEMV stack (4.6× params ≈ 3.4× slower than 7B). The theoretical bandwidth ceiling for streaming ~19 GB of weights per token on an A100 is ~75 tok/s — closing the gap is kernel-occupancy work (#23).

Architecture deep-dive

OLMo-3-32B-Think is a dense decoder-only transformer with an unusually serving-friendly attention layout:

Property	Value
Parameters	~32.2B dense
Layers	64, in a repeating [SWA, SWA, SWA, full] block ×16 → 48 sliding-window + 16 full-attention layers
Sliding window	4,096 tokens (SWA layers)
Attention	GQA — 40 query heads / 8 KV heads, head_dim 128
Hidden / FFN	5,120 / 27,648 (SwiGLU)
Norms	Post-norm + QK-norm (full-projection RMSNorm on Q and K)
RoPE	θ = 500,000; YaRN ×8 (8,192 native → 65,536 max), attention_factor ≈ 1.208
Vocab	100,278 (BPE; new-generation [["a","b"], …] merges format)

Why the 3:1 SWA layout matters for serving

Full f32 KV across all 64 layers costs 512 KiB per token → 8 GB at 16K context. But only the 16 full-attention layers actually need KV for the whole context — the 48 SWA layers never look back more than 4,096 tokens. An SWA-aware KV cache needs only:

16 full × 16,384 tok + 48 SWA × 4,096 tok  ≈ 3.75 GB (f32)   — vs 8 GB naive
16 full × 65,536 tok + 48 SWA × 4,096 tok  ≈ 5.1 GB  (BF16)  — full 64K context, still fits in 40 GB

That means the full 64K YaRN context is reachable on this same 40 GB card — tracked as #24 together with BF16 KV.

Quantization recipe

Weight-only int4, symmetric (no zero-points), group size 32, MSE observer, AWQ via llm-compressor; lm_head kept BF16. Stored in compressed-tensors "pack-quantized" layout — verified empirically against pack_to_int32 (col j → u32 word j/8, little-endian nibble, stored nibble = q+8). The layout is directly kernel-friendly: ATLAS ingests it with no repacking step, and dequantizes inline in a custom W4A32 GEMV kernel (warp-per-row, per-group BF16 scales hoisted per packed word). Runtime precision is W4 weights × f32 activations.

What ATLAS implements for the W4 path

• gemv_w4_kernel / atlas_gemv_w4_f32 — W4A32 GEMV CUDA kernel with inline int4 dequant
• Direct compressed-tensors ingestion — no repacking, no conversion step
• Streaming W4 shard loader — each Linear uploaded to VRAM as soon as its packed + scale halves are seen; peak host RSS ≈ 11 GB (a naive f32 init would need ~128 GB)
• Tokenizer fix for new-generation checkpoints — BPE merges as [["a","b"], …] pairs now parse correctly (with regression tests)
• Full GPU attention path plus HF-reference-fidelity fixes (YaRN correction range, layer-type RoPE split, full-projection QK-norm) — output is differential-tested against transformers

How to use

Via the ATLAS API (OpenAI-compatible)

# Build ATLAS (W4/32B support is on main)
git clone https://github.com/web3guru888/ATLAS.git
cd ATLAS
cargo build --release -p atlas-cli

# Fetch the AWQ 4-bit checkpoint (~19.6 GB)
hf download cyankiwi/Olmo-3-32B-Think-AWQ-4bit --local-dir /models/olmo3-32b-think-w4

# Serve
./target/release/atlas api serve \
  --weights /models/olmo3-32b-think-w4 \
  --model olmo3-32b \
  --port 8080 \
  --max-tokens 3584

# Query (chain-of-thought is returned in message.reasoning,
# the visible answer in message.content)
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "olmo3-32b",
    "messages": [{"role": "user", "content": "What is 17 + 25?"}],
    "max_tokens": 2048,
    "temperature": 0.6,
    "top_p": 0.95
  }'

Via HuggingFace Transformers (compressed-tensors loader)

# pip install transformers compressed-tensors accelerate
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "cyankiwi/Olmo-3-32B-Think-AWQ-4bit"
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(model_id)

messages = [{"role": "user", "content": "What is 17 + 25?"}]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True,
    return_tensors="pt", return_dict=True,
).to(model.device)

output = model.generate(**inputs, max_new_tokens=2048, temperature=0.6, top_p=0.95, do_sample=True)
print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Prompting guide for Think models — read this before judging output quality

OLMo-3-Think is an RL-trained reasoning model: it emits a long hidden <think>…</think> chain-of-thought before the visible answer. Nearly every "the model gave me a broken/empty answer" report traces back to token-budget handling, not the model:

1. Set a generous max_tokens — ≥ 2048 recommended. The think block routinely consumes 500–2,000+ tokens (hard code/planning prompts can need several thousand) before the first visible answer token. With a small budget the generation is cut off mid-think and the answer never appears. Most apparent "wrong answers" from Think models are actually truncations.
2. Server-side answer reserve (deployed on the live endpoint; ATLAS commit cc7784a, branch fix/think-budget-answer-reserve). The ATLAS API now protects against exactly this failure: if max_tokens is exhausted while the model is still inside <think>, the server force-closes the think block with a wrap-up cue and continues decoding from the intact KV cache for up to ATLAS_ANSWER_RESERVE extra tokens (default 512, env-tunable, 0 disables) — so a visible answer is always emitted. Measured on a 5-prompt battery: zero-answer rate 2/5 → 0/5, with byte-identical outputs on requests that don't trigger the reserve. The chat-endpoint default max_tokens is also now 2048 (was 512).
3. Use the reference sampling: temperature 0.6, top-p 0.95, top-k 50, min-p 0.05, repetition_penalty 1.0 (AI2 reference; penalties above 1.0 measurably make the model ramble without concluding).
4. Prompt format: ChatML (<|im_start|>role\n…<|im_end|>), with the generation prompt ending inside a think block: <|im_start|>assistant\n<think>. The ATLAS chat endpoint applies this automatically; the chain-of-thought comes back as message.reasoning (OpenRouter reasoning-model convention) and the answer as message.content. If you use raw /v1/completions, you manage the template yourself.
5. Steer the thinking budget with a think-prefill. On open-ended prompts (especially code), free-form thinking can blow through even a 3,584-token cap. Via /v1/completions with a raw prompt, prime the trace: end your prompt with <think>Brief plan: … — the model completes a short plan and answers immediately and correctly.
6. Ask for structured finals ("End your reply with: Answer: ") — the visible answer after </think> stays clean and parseable.

Key differentiator: J-space interpretability

Beyond raw serving, ATLAS exposes a Jacobian lens (J-space) over the model's forward computation — a first-class interpretability surface over the 5,120-dim residual stream for auditing what the model is doing internally during generation. Combined with the StigmergicHook trait (atlas-infer crate), inference hooks can read/write stigmergic memory in real time during token generation. Planned next: J-lens fit + validation on this 32B and a model-audit workflow (eval-awareness, fabrication, hidden-objective screening) that plain serving stacks don't offer.

Limitations & roadmap

We publish our weak spots on purpose. Current honest state:

Limitation	Detail	Fix	Status
Long-prompt TTFT ~100 s	Prefill used to run token-by-token through the decode GEMV path	Batched/chunked prefill (GEMV→GEMM)	#22 — ✅ merged & deployed 2026-07-08: ~5× measured on the live 32B (1.2K-token prompt ~122 s → ~23 s TTFT)
Decode 14.6 tok/s vs ~75 tok/s bandwidth ceiling	Warp-per-row W4 GEMV at low occupancy	Kernel occupancy tuning (kernel_tuner / openevolve loop)	#23 — target 40+ tok/s
Served context 16K (model supports 64K)	f32 KV, not SWA-aware	BF16 + SWA-aware KV cache (see deep-dive)	#24
Batch-1 serving	Single-request GEMV path	Batched decode after #22	planned
4-bit quality loss unmeasured for OLMo-3	~1–3% is the 32B-class community pattern, not OLMo-measured; our numbers exceed the 7B BF16 baseline by wide margins	W4-vs-BF16 study on identical benchmark sets (80 GB box)	planned
Small benchmark N	25/40/5-item prototype validation	GSM8K-100 / MMLU-200 / HumanEval-50 in a scheduled window	planned — card update follows
Answer lost to thinking budget	max_tokens exhausted inside <think> used to yield an empty visible answer	Server-side answer reserve + max_tokens default 2048 (see prompting guide)	✅ fixed & deployed 2026-07-08 (zero-answer rate 2/5 → 0/5 on the test battery)

Community

• Found something interesting? Broke it? Open a Discussion — especially interested in: reasoning failures, quantization artifacts vs the BF16 base model, and long-context behavior reports.
• Engine work happens in the open: web3guru888/ATLAS — issues #22/#23/#24 are the current 32B performance campaign.
• Want to reproduce our numbers? The bench harnesses are simple single-file Python scripts against the OpenAI-compatible endpoint — ask in Discussions and we'll share them.

About ATLAS

ATLAS (Active-inference Training with Learned Adaptive Stigmergy) is a next-generation LLM framework built in pure Rust with zero external crate dependencies — the SQLite principle applied to AI infrastructure. It fuses:

• GraphPalace — Stigmergic memory palace with pheromone-guided navigation
• ASTRA — Live discovery engine hitting NASA, WHO, World Bank APIs
• TRM-CausalValidator — 7M-param recursive validator
• Champagnat n-Morphic Framework — biologically-grounded training dynamics

22 crates. 644 tests. One coherent system. Zero external Rust dependencies.

Website: atlasagi.org · 7B sibling: ATLAS-OLMo-3-7B-Think-v4 · Live API: atlas.thebeastagi.com · Organization: OpenHub Research · Author: Robin Dey

Citation

@software{atlas2026,
  title       = {ATLAS: Active-inference Training with Learned Adaptive Stigmergy},
  author      = {Robin Dey},
  year        = {2026},
  institution = {OpenHub Research, Thailand},
  url         = {https://github.com/web3guru888/ATLAS},
  note        = {Pure Rust LLM framework. 32B chapter: OLMo-3-32B-Think served
                 via AWQ 4-bit (W4A32 custom CUDA GEMV) in 27.2 GB on a single
                 A100-40GB. GSM8K 100% (25/25), MMLU 82%, exec-verified code 5/5
                 through the ATLAS serving stack. Merged to main, 644 tests.}
}