SEBK4C/Ornith-1.0-397B-Featherweight

Ornith-1.0-397B Featherweight — a 397B agent in 119.5 GB

397B-parameter, 512-expert MoE compressed to 2.41 bits/weight, verified against its own FP8 original on a frozen 604-case behavioral suite: function-calling 0.91, multi-turn tool use 1.00, code 0.98, composite S = 0.9258 (vs FP8's 0.8693 under a bare protocol — read the paper before quoting the comparison).

The base model, deepreinforce-ai/Ornith-1.0-397B, is a state-of-the-art open agentic-coding model (per DeepReinforce: SWE-bench Verified 82.4, Terminal-Bench 2.1 ~78, competitive with 1.6 T and frontier closed models — their benchmarks). This repo makes that model runnable on consumer hardware and proves the behavior survives compression; the capability numbers are DeepReinforce's, the fidelity numbers are ours (paper has both).

_{Base-model capability benchmarks — DeepReinforce's published evaluation, reproduced for context (not re-run by us).}

How it was made, validated, and every dead-end (~100 hours of autonomous ratchet research): → PAPER.md · research report · harness + goldens + journal. This page is only what you need to host it.

Files

File	Size	sha256
Ornith-1.0-397B-Featherweight-v0.gguf	119,517,476,064 B	1499c2f2d84dcdbb4c243371522cc8f98ca4a76a0bc2f1e4801b9f64f0c57dc2
Ornith-1.0-397B-Featherweight-serve.llamafile	320 MB	9a1b095bbca37661cc3903e9a3369f0ccc162b0731a845f40f8618148005ad2d
featherweight-autotune.sh	4 KB	launcher — probes your machine, computes tuned flags
serving.args / recipe.yaml	—	reference flags (per-flag rationale) / exact quant recipe

Quickstart

Just run the autotuner. It probes your GPUs/RAM/cores, picks the fastest engine you have installed, and computes offload/context/thread flags from the measured serving laws — then launches. One command, any machine:

./featherweight-autotune.sh            # add --dry-run to see the computed command first

That's the whole story for most people. The details below are only if you want to drive the engines yourself.

Engine tiers (what autotune picks for you)

Tier	When	Speed	How
llama.cpp + cuBLAS ★	NVIDIA + llama-server installed (≥4fc4ec55, qwen3_5_moe)	full (18.5 t/s ref)	autotune selects it; or llama-server -m …gguf $(grep -vE '^\s*(#|$)' serving.args|tr '\n' ' ')
llamafile Metal	Apple Silicon	good (unified-memory bound)	autotune, or bare ./…serve.llamafile
llamafile CPU	anything, no deps	RAM-bandwidth bound	bare ./…serve.llamafile (portable defaults)
llamafile --gpu nvidia	NVIDIA, no llama.cpp	⚠ compatibility only (~2–3 t/s, driver tinyBLAS)	last resort — install llama.cpp instead

Bare zero-install boot (no dependencies, portable-safe defaults — serves the OpenAI API on :8080; append your own -ngl/-ts/--n-cpu-moe or just use autotune):

chmod +x Ornith-1.0-397B-Featherweight-serve.llamafile && ./Ornith-1.0-397B-Featherweight-serve.llamafile

On NVIDIA, install llama.cpp — the llamafile's driver-only CUDA (tinyBLAS) is a compatibility fallback: ~7× slower than cuBLAS and occasionally fails to initialize. It exists so the file runs everywhere, not so it runs fast. A self-contained binary with in-process autotune (like the sibling gemma-4 llamafile) is on the roadmap — it needs that fork's bundled llama.cpp rebuilt against qwen3_5_moe.

Expected speed by hardware

Hardware	Engine	Decode tok/s	Prefill tok/s
2×RTX 4090 · 90 GB DDR5 · Gen5 NVMe ★ recommended	llama.cpp	18.5 measured (17–19)	717 measured (650–750)
same box	llamafile --gpu nvidia	2.6 measured	~5
same box, bare llamafile (CPU, cold)	llamafile	2.0 measured	~4
1×12 GB GPU · 128 GB DDR4 · Gen4	llama.cpp	5–9 predicted ±40%	60–150 predicted
Apple M5 Pro · 120 GB unified	Metal	12–25 predicted ±50%	100–250 predicted
Apple Max/Ultra · ≥128 GB	Metal	25–45 predicted ±50%	200–400 predicted
12-channel DDR5 server · CPU-only	llama.cpp	15–25 predicted ±40%	150–300 predicted

Predictions follow the serving law (decode ≈ RAM bandwidth ÷ ~1.5 GB streamed/token; paper § "Run it yourself"). The ± ranges are honest priors, not measurements — post your rig's numbers via the 7-step smoke checklist (paper) to the results thread and this table tightens.

Recommended hardware

2×24 GB GPUs + ≥90 GB fast RAM + NVMe — the verified reference class (the box the whole research program ran on). Below that it still works: RAM size decides fit (≥120 GB comfortable; less streams from SSD), RAM bandwidth decides decode speed, GPUs mostly buy prefill plus a decode bonus from expert offload.

Operational notes (measured, not folklore)

• llamafile default port is 8080 (--port N to change); any flag you append overrides the baked defaults — last value wins.
• Single GPU: --n-cpu-moe 60, no -ts. Autotune OOM? FEATHERWEIGHT_GPU_LAYERS=8 ./featherweight-autotune.sh.
• Mac under 128 GB: raise the wired limit first — sudo sysctl iogpu.wired_limit_mb=<RAM_MB×0.94>.
• The model thinks by default (9K+ chars on trivial prompts if unbounded): keep --reasoning-budget 1024.
• Never enable speculative decoding on CPU-resident-expert rigs — measured −10% (ngram) to −34% (draft model); mechanism in the paper. Ornith's declared MTP head has no published weights in any repo.
• Off-llama.cpp servers (vLLM/SGLang): tool calls arrive as <function=…> XML in content — parse it or tool benchmarks read as zero. llama.cpp --jinja handles it natively. (Also: vLLM currently lacks SM120 FP8-MoE kernels — on workstation Blackwell use SGLang.)

Credits & thanks

This is a derivative work standing entirely on others' shoulders — please credit them too:

• The model: deepreinforce-ai/Ornith-1.0-397B and its FP8 release by DeepReinforce — MIT-licensed weights of a genuinely impressive 397B agentic MoE. None of this exists without them.
• The inference engines: ggml-org/llama.cpp and GGML (Georgi Gerganov & contributors) — the runtime everything is served on; Mozilla-Ocho/llamafile (Justine Tunney & contributors) for the zero-install path; sgl-project/sglang for FP8-MoE serving on Blackwell.
• The method: karpathy/autoresearch — the ratchet-loop pattern this whole program is a port of; and antirez/ds4 (Salvatore Sanfilippo), whose asymmetric-crush 2-bit MoE recipe this independently reproduces on a different architecture.
• The evaluation data: gorilla-llm BFCL (Berkeley Function-Calling Leaderboard, Apache-2.0) and evalplus HumanEval+ (MIT/Apache-2.0) — per-artifact licensing table in the dataset card.
• Infrastructure: Hugging Face for Inference Endpoints (and the quota support that made the FP8 anchor possible).

License

MIT (base model MIT). Full provenance, licensing table, eval harness, FP8 goldens, and the append-only research journal live in SEBK4C/molt-ornith-eval.