can-it-run / index.html
rigbase's picture
Standalone calculator
47b4283 verified
Raw History Blame Contribute Delete
140 kB
<!doctype html><html lang=en><head><base target="_blank"><meta charset=utf-8><meta name=viewport content="width=device-width,initial-scale=1"><meta name=theme-color content="#09090b"><title>Can I run it? VRAM &amp; KV cache calculator: multi-GPU, DGX Spark, NVFP4, GGUF</title><meta name=description content="Free VRAM and KV cache calculator: can I run gpt-oss 120B on 4x 3090, Qwen on one RTX 3090, a 200B model on 2x DGX Spark? GGUF, AWQ, FP8, NVFP4 VRAM math…"><link rel=canonical href="https://getrigbase.com/can-it-run/?utm_source=huggingface&utm_medium=space"><meta property=og:type content=website><meta property=og:site_name content=RigBase><meta property=og:title content="Can I run it? VRAM &amp; KV cache calculator: multi-GPU, DGX Spark, NVFP4, GGUF | RigBase"><meta property=og:description content="Free VRAM and KV cache calculator: can I run gpt-oss 120B on 4x 3090, Qwen on one RTX 3090, a 200B model on 2x DGX Spark? GGUF, AWQ, FP8, NVFP4 VRAM math…"><meta property=og:url content="https://getrigbase.com/can-it-run/"><meta name=twitter:card content=summary_large_image><script type="application/ld+json">{"@context": "https://schema.org", "@graph": [{"@type": "WebApplication", "name": "RigBase VRAM & KV cache calculator", "url": "https://getrigbase.com/can-it-run/", "applicationCategory": "UtilitiesApplication", "operatingSystem": "Any (runs in the browser)", "offers": {"@type": "Offer", "price": "0", "priceCurrency": "EUR"}, "description": "Estimate if a local LLM fits on 1–8 GPUs or DGX Spark / Mac / Strix Halo, for GGUF, AWQ, GPTQ, FP8, NVFP4, MXFP4, EXL and MLX, with KV cache math.", "dateModified": "2026-10-28"}, {"@type": "FAQPage", "mainEntity": [{"@type": "Question", "name": "Can I run gpt-oss 120B on 4x RTX 3090?", "acceptedAnswer": {"@type": "Answer", "text": "Yes, by our estimate: gpt-oss 120B in its native MXFP4 checkpoint (65 GB) needs about 65.0 GB for the weights plus 4.8 GB of KV cache at 128k context, 69.8 GB in total, against about 92.0 GB usable on 4× RTX 3090 (96 GB, minus ~1 GB overhead per card). This is arithmetic, not a measurement; check the measured builds linked on the page. gpt-oss uses sliding-window attention on half its layers, which keeps the KV cache small."}}, {"@type": "Question", "name": "Can I run Qwen3 32B on one RTX 3090 at 32k context?", "acceptedAnswer": {"@type": "Answer", "text": "Not fully, by our estimate: Qwen3 32B in Q4_K_M needs about 20.0 GB for the weights plus 8.6 GB of KV cache at 32k context, 28.6 GB in total, against about 23.0 GB usable on one RTX 3090. This is arithmetic, not a measurement; check the measured builds linked on the page. With the KV cache quantized to q4_0 it needs about 22.5 GB, which fits; with FP16 KV the largest context that fits is about 11k tokens."}}, {"@type": "Question", "name": "Can two DGX Sparks run a ~200B model?", "acceptedAnswer": {"@type": "Answer", "text": "Yes, by our estimate: Qwen3.8 Flash-Next in NVFP4 needs about 101.2 GB for the weights plus 3.2 GB of KV cache at 128k context, 104.5 GB in total, against about 238.0 GB usable on 2× DGX Spark (2 × 128 GB, minus ~9 GB per node for OS and runtime). This is arithmetic, not a measurement; check the measured builds linked on the page. NVIDIA says up to four Sparks can be linked over ConnectX-7."}}, {"@type": "Question", "name": "How do I calculate KV cache size?", "acceptedAnswer": {"@type": "Answer", "text": "KV cache bytes = 2 (keys and values) × attention layers × KV heads × head dimension × context tokens × concurrent sequences × bytes per value (FP16 = 2, FP8 = 1, q8_0 ≈ 1.06, q4_0 ≈ 0.56). Read layers, num_key_value_heads and head_dim from the model's config.json. Example: Llama 3.3 70B has 80 layers, 8 KV heads and head_dim 128, so one token costs 2 × 80 × 8 × 128 × 2 B = 327,680 bytes (0.33 MB), and 32k tokens cost 10.7 GB. Sliding-window layers only store their window; linear-attention and Mamba layers keep a fixed-size state."}}, {"@type": "Question", "name": "How much VRAM does an NVFP4 model need?", "acceptedAnswer": {"@type": "Answer", "text": "NVFP4 stores 4-bit values with one FP8 scale per 16 values, about 4.5 bits per weight, so weights take roughly parameters × 0.5625 GB per billion: a 120B model is about 67.5 GB before KV cache. It is about 3.5× smaller than FP16 and 1.8× smaller than FP8 (NVIDIA). It runs natively only on Blackwell GPUs; vLLM falls back to slower weight-only kernels elsewhere."}}, {"@type": "Question", "name": "Which quant should I use?", "acceptedAnswer": {"@type": "Answer", "text": "Q4_K_M (GGUF) or AWQ / GPTQ 4-bit (vLLM) is the usual sweet spot of size and quality. Use Q5_K_M or Q6_K if you have the room, and Q8_0 or FP8 when you want near-lossless output. We do not recommend going below about 4 bits per weight (Q3, Q2) unless nothing else fits, because quality drops faster there."}}, {"@type": "Question", "name": "Does system RAM matter for running local LLMs?", "acceptedAnswer": {"@type": "Answer", "text": "Only if the model does not fit in VRAM. When weights and KV cache sit fully on the GPUs, system RAM just holds the OS and the inference engine and loads the weights once; llama.cpp memory-maps the file, so RAM can be smaller than the model. RAM size and speed matter when you offload layers or MoE experts to the CPU."}}, {"@type": "Question", "name": "Tensor parallel or pipeline parallel?", "acceptedAnswer": {"@type": "Answer", "text": "Tensor parallel (vLLM -tp) splits every layer across GPUs so they work on each token together: faster, but it needs the head count to divide by the GPU count (2, 4, 8 are usual) and good interconnect. Pipeline / layer split (llama.cpp default, vLLM -pp) gives each GPU a block of layers: any GPU count works and it needs little bandwidth, but one stream uses one GPU at a time."}}], "dateModified": "2026-10-28", "isPartOf": {"@id": "https://getrigbase.com/#website"}}]}</script><style>:root{--bg:#09090b;--bg2:#0f0f12;--panel:#121216;--panel2:#18181d;--line:#25252c;--line2:#33333c;--text:#ececf1;--text2:#b4b4bf;--muted:#8b8b98;--acc:#8b7cff;--acc2:#a99fff;--accbg:rgba(139,124,255,.12);--ok:#3fb950;--warn:#e3b341;--bad:#ff7b72;--r:12px;--max:1180px;--hh:58px}*{box-sizing:border-box}html{-webkit-text-size-adjust:100%;scroll-padding-top:calc(var(--hh) + 12px)}body{margin:0;background:var(--bg);color:var(--text);font:15px/1.6 Inter,ui-sans-serif,system-ui,-apple-system,"Segoe UI",Roboto,"Helvetica Neue",Arial,sans-serif;font-feature-settings:"cv11","ss01","tnum";-webkit-font-smoothing:antialiased}a{color:var(--acc2);text-decoration:none;text-underline-offset:3px}a:hover{text-decoration:underline}h1,h2,h3{line-height:1.2;letter-spacing:-.02em;margin:1.6em 0 .55em;font-weight:650}h1{font-size:clamp(28px,4.2vw,44px);letter-spacing:-.035em;margin-top:.5em}h2{font-size:22px}h3{font-size:17px}p{margin:.6em 0}code{font:13px ui-monospace,SFMono-Regular,Menlo,monospace;background:var(--panel2);border:1px solid var(--line);border-radius:6px;padding:1px 6px;overflow-wrap:anywhere}ul{padding-left:20px}li{margin:3px 0}/* header */.sh{position:sticky;top:0;z-index:50;background:rgba(9,9,11,.82);backdrop-filter:saturate(160%) blur(12px);-webkit-backdrop-filter:saturate(160%) blur(12px);border-bottom:1px solid var(--line)}.shi{max-width:var(--max);margin:auto;height:var(--hh);padding:0 20px;display:flex;align-items:center;gap:18px}.logo{display:flex;align-items:center;gap:9px;color:var(--text);font-weight:700;font-size:17px;letter-spacing:-.02em;white-space:nowrap}.logo:hover{text-decoration:none}.logo i{width:22px;height:22px;border-radius:6px;background:linear-gradient(135deg,var(--acc),#5b4bd6);display:inline-block;box-shadow:inset 0 0 0 1px rgba(255,255,255,.18)}.nav{display:flex;gap:2px;flex:1;min-width:0;overflow-x:auto;scrollbar-width:none}.nav::-webkit-scrollbar{display:none}.nav a,.more summary{color:var(--text2);font-size:14px;padding:6px 11px;border-radius:8px;white-space:nowrap;cursor:pointer}.nav a:hover,.more summary:hover{color:var(--text);background:var(--panel2);text-decoration:none}.more{position:relative}.more summary{list-style:none;display:block}.more summary::-webkit-details-marker{display:none}.more summary::after{content:" ▾";font-size:11px;opacity:.7}.more[open] summary{color:var(--text);background:var(--panel2)}.more .mn{position:absolute;right:0;top:calc(100% + 8px);min-width:220px;background:var(--panel);border:1px solid var(--line2);border-radius:12px;padding:6px;box-shadow:0 16px 40px rgba(0,0,0,.5);display:grid}.more .mn a{color:var(--text2);padding:8px 12px;border-radius:8px;font-size:14px}.more .mn a:hover{background:var(--panel2);color:var(--text);text-decoration:none}.hcta{font-size:13.5px;padding:7px 13px;border-radius:8px;background:var(--text);color:#09090b;font-weight:600;white-space:nowrap}.hcta:hover{text-decoration:none;opacity:.9}/* layout */main{max-width:var(--max);margin:auto;padding:28px 20px 40px;overflow-wrap:break-word}.muted{color:var(--muted);font-size:.92em}.hero{font-size:1.12em;color:var(--text2);max-width:780px}.card>*{min-width:0}.card{background:var(--panel);border:1px solid var(--line);border-radius:var(--r);padding:18px 20px;margin:16px 0}.btn{display:inline-flex;align-items:center;gap:6px;background:var(--acc);color:#fff;padding:9px 16px;border-radius:9px;text-decoration:none;border:0;cursor:pointer;line-height:1.3;margin:6px 0;font:600 14px/1.3 inherit;font-family:inherit}.btn:hover{background:#7a6af5;text-decoration:none}.btn.ghost{background:transparent;border:1px solid var(--line2);color:var(--text)}.btn.ghost:hover{background:var(--panel2)}input,select,textarea{background:var(--bg2);color:var(--text);border:1px solid var(--line2);padding:8px 11px;border-radius:9px;font:inherit;font-size:14px;max-width:100%}input:focus,select:focus,textarea:focus{outline:2px solid var(--acc);outline-offset:-1px;border-color:transparent}label{color:var(--text2)}/* chips & badges */.chips{display:flex;flex-wrap:wrap;gap:8px;align-items:center;font-size:0}.chips>*{font-size:15px}.chips>b{margin-right:4px;color:var(--text2);font-weight:600;font-size:14px}.chips>a:not(.btn){font-size:13.5px;padding:5px 12px;border:1px solid var(--line2);border-radius:999px;background:var(--bg2);color:var(--text2);text-decoration:none;max-width:100%;line-height:1.35}.chips>a:not(.btn):hover{border-color:var(--acc);color:var(--text)}.chips .btn{margin:0 6px 0 0}.bdg{display:inline-block;font-size:12px;line-height:1.45;padding:1px 9px;border:1px solid var(--line2);border-radius:999px;margin:2px 4px 2px 0;white-space:normal;color:var(--text2);background:var(--bg2)}.bdg.sd{border-color:rgba(227,179,65,.45);color:var(--warn)}.bdg.pend{border-color:rgba(255,123,114,.45);color:var(--bad)}.bdg.ok{border-color:rgba(63,185,80,.45);color:var(--ok)}.bdg.cur{border-color:rgba(139,124,255,.55);color:var(--acc2);background:var(--accbg)}.bdg.leg{border-color:var(--line2);color:var(--muted)}.hads{display:block;border:1px dashed var(--line2);border-radius:var(--r);padding:12px 16px;margin:18px 0;font-size:.92em;background:var(--bg2)}.hadl{display:block;font-size:.7em;letter-spacing:.08em;text-transform:uppercase;color:var(--muted);margin-bottom:2px}.note{border:1px solid rgba(139,124,255,.45);background:var(--accbg);border-radius:var(--r);padding:12px 16px;margin:14px 0}.note.warn{border-color:rgba(227,179,65,.45);background:rgba(227,179,65,.07)}/* tables */table{width:100%;border-collapse:separate;border-spacing:0;font-size:14px;background:var(--panel);border:1px solid var(--line);border-radius:var(--r);margin:12px 0}td,th{padding:9px 12px;border-bottom:1px solid var(--line);text-align:left;vertical-align:top}tr:last-child>td,tr:last-child>th{border-bottom:0}th{color:var(--muted);font-weight:600;font-size:12.5px;text-transform:none;letter-spacing:.01em;background:var(--panel2)}tr:first-child:not(:has(td))>th{position:sticky;top:var(--hh);z-index:2;background:#1a1a20}tr:first-child>th:first-child{border-top-left-radius:var(--r)}tr:first-child>th:last-child{border-top-right-radius:var(--r)}tr:has(td):nth-child(even)>td{background:rgba(255,255,255,.012)}tr:has(td):hover>td{background:rgba(139,124,255,.06)}.card table{background:transparent;border:0;margin:0}.card td,.card th{padding:8px 4px}.card th{background:transparent}td.n,th.n{text-align:right;white-space:nowrap;font-variant-numeric:tabular-nums}b.tps{display:inline-block;min-width:54px;text-align:right;padding:1px 8px;border-radius:7px;font-variant-numeric:tabular-nums;font-weight:650;color:var(--text);background:linear-gradient(90deg,rgba(139,124,255,.38) var(--w,0%),rgba(139,124,255,.10) var(--w,0%));box-shadow:inset 0 0 0 1px rgba(139,124,255,.28)}td:not(.n) b.tps{min-width:0;padding:0 7px}/* home */.hx{padding:34px 0 8px}.hx h1{max-width:860px;margin:.2em 0 .35em}.ey{display:inline-flex;gap:8px;align-items:center;font-size:12.5px;color:var(--text2);border:1px solid var(--line2);border-radius:999px;padding:4px 12px;background:var(--bg2)}.ey b{color:var(--acc2)}.stats{display:grid;grid-template-columns:repeat(4,1fr);gap:1px;background:var(--line);border:1px solid var(--line);border-radius:var(--r);overflow:hidden;margin:26px 0 10px}.stats div{background:var(--panel);padding:16px 18px}.stats b{display:block;font-size:28px;letter-spacing:-.03em;font-variant-numeric:tabular-nums}.stats span{color:var(--muted);font-size:13px}.qf{display:flex;flex-wrap:wrap;gap:10px;align-items:center;margin:14px 0}.qf input[type=search]{flex:1 1 320px;padding:12px 14px;font-size:15px}.grid{display:grid;grid-template-columns:repeat(auto-fill,minmax(250px,1fr));gap:14px;margin:14px 0}.tile{display:block;background:var(--panel);border:1px solid var(--line);border-radius:var(--r);padding:16px 18px;color:var(--text);transition:border-color .15s,background .15s;min-width:0}.tile:hover{border-color:var(--line2);background:var(--panel2);text-decoration:none}.tile h3{margin:0 0 4px;font-size:15.5px;letter-spacing:-.01em}.tile .k{color:var(--muted);font-size:12.5px}.tile .v{margin-top:10px;display:flex;justify-content:space-between;align-items:baseline;gap:8px;font-size:13px;color:var(--text2)}.tile .v b{font-size:20px;color:var(--text);letter-spacing:-.02em}.sec{display:flex;justify-content:space-between;align-items:baseline;gap:12px;margin-top:38px}.sec h2{margin:0}.sec a{font-size:14px;white-space:nowrap}.filt{display:flex;flex-wrap:wrap;gap:10px 16px;align-items:center}/* key/value spec grid */.kv{display:grid;grid-template-columns:repeat(auto-fill,minmax(230px,1fr));gap:0;background:var(--panel);border:1px solid var(--line);border-radius:var(--r);overflow:hidden;margin:16px 0}.kv div{background:var(--panel);padding:12px 16px;min-width:0;border-bottom:1px solid var(--line);border-right:1px solid var(--line);margin:0 -1px -1px 0}.kv dt{color:var(--muted);font-size:12px;margin-bottom:2px}.kv dd{margin:0;font-size:14px}.kv dd .muted{font-size:12px}.src{display:flex;flex-wrap:wrap;gap:8px 14px;align-items:center}.src a.btn{max-width:100%;overflow:hidden;text-overflow:ellipsis;white-space:nowrap}/* footer */.sf{border-top:1px solid var(--line);background:var(--bg2);margin-top:40px}.sfi{max-width:var(--max);margin:auto;padding:36px 20px 28px;display:grid;grid-template-columns:1.4fr repeat(3,1fr);gap:28px}.sf h4{margin:0 0 10px;font-size:13px;color:var(--text);font-weight:600}.sf a{display:block;color:var(--muted);font-size:13.5px;padding:3px 0}.sf a:hover{color:var(--text)}.sf p{color:var(--muted);font-size:13px;margin:8px 0}.sfb{max-width:var(--max);margin:auto;padding:14px 20px 26px;border-top:1px solid var(--line);color:var(--muted);font-size:12.5px;display:flex;flex-wrap:wrap;gap:6px 18px;justify-content:space-between}@media(max-width:900px){.sfi{grid-template-columns:1fr 1fr}.stats{grid-template-columns:repeat(2,1fr)}}@media(max-width:760px){:root{--hh:104px}.shi{position:relative;height:auto;flex-wrap:wrap;padding:10px 14px 0;gap:8px 10px}.nav{order:3;flex:1 0 100%;margin:0 -14px;padding:0 10px 8px}.hcta{margin-left:auto}main{padding:18px 14px 30px}table{display:block;overflow-x:auto;-webkit-overflow-scrolling:touch}tr:first-child:not(:has(td))>th{position:static}td,th{min-width:92px}td:first-child,th:first-child{min-width:168px}.card td,.card th{min-width:0}.more{position:static}.more .mn{left:14px;right:14px;top:52px}.logo b{font-size:16px}input,select{max-width:100%}.card{padding:14px}.stats b{font-size:22px}.hx{padding-top:14px}.sfi{grid-template-columns:1fr 1fr;padding:28px 14px 20px}.sfb{padding:12px 14px 22px}}/* 2026-10-25 prompt-processing column */td.pp,th.pp{min-width:210px}@media(max-width:760px){td.pp,th.pp{min-width:220px}}/* 2026-10-28 embeddable Measured on RigBase badge */.card.embed pre{background:var(--panel2);border:1px solid var(--line);border-radius:8px;padding:12px 14px;overflow-x:auto;font:12.5px/1.45 ui-monospace,SFMono-Regular,Menlo,monospace;color:var(--text2);white-space:pre-wrap;overflow-wrap:anywhere;margin:8px 0 14px}.card.embed img{vertical-align:middle}</style><meta property="og:image" content="https://getrigbase.com/og.png"><meta property="og:image:width" content="1200"><meta property="og:image:height" content="630"><meta property="og:image:alt" content="RigBase: measured local-LLM speeds on real home hardware"><meta name="twitter:image" content="https://getrigbase.com/og.png"><link rel="icon" href="https://getrigbase.com/favicon.svg?utm_source=huggingface&utm_medium=space" type="image/svg+xml"><script type="application/ld+json">{"@context": "https://schema.org", "@graph": [{"@type": "Organization", "@id": "https://getrigbase.com/#org", "name": "RigBase", "url": "https://getrigbase.com/", "logo": {"@type": "ImageObject", "url": "https://getrigbase.com/og.png"}, "email": "hello@getrigbase.com", "sameAs": ["https://huggingface.co/rigbase", "https://dev.to/rigbase", "https://rigbase.hashnode.dev", "https://github.com/getrigbase"]}, {"@type": "WebSite", "@id": "https://getrigbase.com/#website", "name": "RigBase", "url": "https://getrigbase.com/", "publisher": {"@id": "https://getrigbase.com/#org"}, "inLanguage": ["en", "de"], "description": "Measured local LLM speeds (tokens/sec) on real home AI rigs, each linked to its public source."}, {"@type": "BreadcrumbList", "@id": "https://getrigbase.com/can-it-run/#breadcrumb", "itemListElement": [{"@type": "ListItem", "position": 1, "name": "RigBase", "item": "https://getrigbase.com/"}, {"@type": "ListItem", "position": 2, "name": "Can I run it? VRAM & KV cache calculator", "item": "https://getrigbase.com/can-it-run/"}]}]}</script><link rel=alternate type="text/markdown" href="https://getrigbase.com/can-it-run/index.md" title="Markdown version"></head><body><div style="background:#0d1117;color:#e6edf3;padding:.6rem 1rem;font:14px system-ui;border-bottom:1px solid #30363d">RigBase calculator on Hugging Face · <a style="color:#3fb950" href="https://getrigbase.com/can-it-run/?utm_source=huggingface&utm_medium=space">Full version on getrigbase.com (always current)</a> · <a style="color:#58a6ff" href="https://getrigbase.com/data/?utm_source=huggingface&utm_medium=space">Dataset (CC BY 4.0)</a></div><header class=sh><div class=shi><a href="https://getrigbase.com/?utm_source=huggingface&utm_medium=space" class=logo><i></i><b>RigBase</b></a><nav class=nav aria-label=Primary><a href="https://getrigbase.com/builds/?utm_source=huggingface&utm_medium=space">Builds</a><a href="https://getrigbase.com/gpus/?utm_source=huggingface&utm_medium=space">GPUs</a><a href="https://getrigbase.com/models/?utm_source=huggingface&utm_medium=space">Models</a><a href="https://getrigbase.com/can-it-run/?utm_source=huggingface&utm_medium=space">Can it run?</a><a href="https://getrigbase.com/best/?utm_source=huggingface&utm_medium=space">By VRAM</a><a href="https://getrigbase.com/compare/?utm_source=huggingface&utm_medium=space">Compare</a><a href="https://getrigbase.com/dgx-spark/?utm_source=huggingface&utm_medium=space">DGX Spark</a><a href="https://getrigbase.com/guides/?utm_source=huggingface&utm_medium=space">Guides</a></nav><details class=more><summary>More</summary><div class=mn><a href="https://getrigbase.com/budget/?utm_source=huggingface&utm_medium=space">Best rig by budget</a><a href="https://getrigbase.com/rent/?utm_source=huggingface&utm_medium=space">Rent before you buy</a><a href="https://getrigbase.com/submit/?utm_source=huggingface&utm_medium=space">Submit a build</a><a href="https://getrigbase.com/newsletter/?utm_source=huggingface&utm_medium=space">Newsletter</a><a href="https://getrigbase.com/data/?utm_source=huggingface&utm_medium=space">Free dataset</a><a href="https://getrigbase.com/advertise/?utm_source=huggingface&utm_medium=space">Advertise</a></div></details><a class=hcta href="https://getrigbase.com/submit/?utm_source=huggingface&utm_medium=space">Submit build</a></div></header><main><style>
.cr [hidden],#crMemW[hidden],#crBpwW[hidden]{display:none!important}
.crg{display:grid;grid-template-columns:minmax(0,1fr) minmax(0,1.1fr);gap:18px;align-items:start}
#crForm{display:grid;grid-template-columns:1fr 1fr;gap:10px 14px}#crForm .crf{display:flex;flex-direction:column;gap:4px;min-width:0}#crForm .crf.w{grid-column:1/-1}
#crForm select,#crForm input{width:100%;min-height:40px;background:var(--bg2);color:var(--text);border:1px solid var(--line2);border-radius:9px;padding:7px 10px;font:inherit}
#crForm select:focus,#crForm input:focus{outline:2px solid var(--acc);outline-offset:1px}
.crl{display:flex;align-items:center;gap:6px;font-size:13.5px;color:var(--text2)}
.tipw{position:relative;display:inline-flex}
.tip{width:20px;height:20px;border-radius:50%;border:1px solid var(--line2);background:var(--panel2);color:var(--acc2);font:700 12px/1 inherit;cursor:help;padding:0;display:inline-flex;align-items:center;justify-content:center}
.tip:focus-visible{outline:2px solid var(--acc);outline-offset:2px}
.tipb{display:none;position:absolute;z-index:40;left:-8px;top:calc(100% + 8px);width:min(320px,82vw);background:var(--panel2);border:1px solid var(--line2);border-radius:10px;padding:10px 12px;font-size:13px;line-height:1.5;color:var(--text);box-shadow:0 12px 30px rgba(0,0,0,.55);font-weight:400}
.tipw:hover .tipb,.tip:focus+.tipb,.tipw.open .tipb{display:block}
.crv{font-size:1.25em;font-weight:650}.crv.ok{color:var(--ok)}.crv.bad{color:var(--bad)}
.crbar{height:10px;background:var(--bg2);border:1px solid var(--line);border-radius:6px;overflow:hidden;margin:10px 0}.crbar i{display:block;height:100%}
.crnums{display:grid;grid-template-columns:repeat(4,1fr);gap:8px;margin:8px 0}.crnums div{background:var(--bg2);border:1px solid var(--line);border-radius:9px;padding:8px}.crnums b{display:block;font-size:1.1em}.crnums span{font-size:12px;color:var(--muted)}
.crt{width:100%;font-size:13.5px}.crt td{vertical-align:top}.crhit td{background:var(--accbg)}
.crp{font-size:12.5px;padding:6px 10px;margin:3px 3px 3px 0}
#crArch summary{cursor:pointer;color:var(--text2);font-size:13.5px}#crArch .crag{display:grid;grid-template-columns:repeat(3,1fr);gap:8px;margin-top:8px}
@media(max-width:900px){.crg{grid-template-columns:1fr}}
@media(max-width:760px){#crForm{grid-template-columns:1fr}.crnums{grid-template-columns:1fr 1fr}#crArch .crag{grid-template-columns:1fr 1fr}
.tipw{position:static}.crl{flex-wrap:wrap;position:relative}.tipb{left:0;right:0;width:auto}.crt{display:block;overflow-x:auto}}
</style><h1>Can I run it? VRAM &amp; KV cache calculator</h1><div class="card qa" id=quick-answer><p class=muted style="margin:0">Quick answer</p><p><b>How much VRAM do I need to run an LLM locally?</b><br>Weights + KV cache + runtime overhead must fit in the GPU memory. Weights ≈ parameters × bits per weight ÷ 8 (Q4_K_M is 4.89 bits per weight). The KV cache grows with context: 2 × layers × KV heads × head_dim × context tokens × bytes per value; for Llama 3.3 70B at FP16 that is about 0.33 MB per token, 10.7 GB at 32,768 tokens. The calculator below does this math for 1–8 GPUs, DGX Spark, Mac and Strix Halo, and links the measured builds for each setup.</p><p class='muted upd'>Last updated: <time datetime=2026-10-28>28 Oct 2026</time></p></div>
<p class=hero>Will a 2026 model fit on your 1–8 GPUs, a DGX Spark (or two), a Mac or a Strix Halo? Pick the model, engine and format (GGUF, AWQ, GPTQ, FP8, NVFP4, MXFP4, EXL, MLX), context and hardware. We show the weights, the KV cache, the math, and the real measured builds for that setup.</p>
<div class=chips><button type=button class='btn ghost crp' data-preset='{&quot;m&quot;: &quot;Qwen3 32B&quot;, &quot;e&quot;: &quot;lc&quot;, &quot;f&quot;: &quot;Q4_K_M&quot;, &quot;hw&quot;: &quot;3090&quot;, &quot;n&quot;: 1, &quot;ctx&quot;: 32768}'>Qwen3 32B · Q4_K_M · 1× 3090 · 32k</button><button type=button class='btn ghost crp' data-preset='{&quot;m&quot;: &quot;gpt-oss 120B&quot;, &quot;e&quot;: &quot;lc&quot;, &quot;f&quot;: &quot;MXFP4&quot;, &quot;hw&quot;: &quot;3090&quot;, &quot;n&quot;: 4, &quot;ctx&quot;: 131072}'>gpt-oss 120B · MXFP4 · 4× 3090</button><button type=button class='btn ghost crp' data-preset='{&quot;m&quot;: &quot;Qwen3.8 Flash-Next&quot;, &quot;e&quot;: &quot;vl&quot;, &quot;f&quot;: &quot;NVFP4&quot;, &quot;hw&quot;: &quot;spark&quot;, &quot;n&quot;: 2, &quot;ctx&quot;: 131072}'>Qwen3.8 Flash-Next 180B · NVFP4 · 2× DGX Spark</button><button type=button class='btn ghost crp' data-preset='{&quot;m&quot;: &quot;Nemotron 3.5 Super VL 120B-A12B&quot;, &quot;e&quot;: &quot;vl&quot;, &quot;f&quot;: &quot;AWQ4&quot;, &quot;hw&quot;: &quot;3090&quot;, &quot;n&quot;: 4, &quot;ctx&quot;: 65536}'>Nemotron 3.5 Super 120B · AWQ · 4× 3090 vLLM</button><button type=button class='btn ghost crp' data-preset='{&quot;m&quot;: &quot;Mistral Medium 3.5 128B&quot;, &quot;e&quot;: &quot;vl&quot;, &quot;f&quot;: &quot;FP8&quot;, &quot;hw&quot;: &quot;3090&quot;, &quot;n&quot;: 8, &quot;ctx&quot;: 32768}'>Mistral Medium 3.5 128B · FP8 · 8× 3090</button><button type=button class='btn ghost crp' data-preset='{&quot;m&quot;: &quot;Llama 3.3 70B&quot;, &quot;e&quot;: &quot;lc&quot;, &quot;f&quot;: &quot;Q4_K_M&quot;, &quot;hw&quot;: &quot;3090&quot;, &quot;n&quot;: 2, &quot;ctx&quot;: 32768}'>Llama 3.3 70B · Q4_K_M · 2× 3090</button><button type=button class='btn ghost crp' data-preset='{&quot;m&quot;: &quot;Gemma 4 31B&quot;, &quot;e&quot;: &quot;mx&quot;, &quot;f&quot;: &quot;MLX4&quot;, &quot;hw&quot;: &quot;mac&quot;, &quot;n&quot;: 1, &quot;ctx&quot;: 131072}'>Gemma 4 31B · MLX 4-bit · Mac 128 GB</button></div>
<div class=crg><div class=card><div id=crForm>
<div class="crf w"><span class=crl><label for=crM>Model</label><span class=tipw><button type=button class=tip aria-label='What is Model?' aria-describedby=tt1 aria-expanded=false>?</button><span role=tooltip id=tt1 class=tipb>The model decides how many parameters must sit in memory. For <b>MoE</b> models (e.g. “35B-A3B”) <i>all</i> experts must be loaded, even though only the active ones are used per token.</span></span></span><select id=crM></select></div>
<div class=crf><span class=crl><label for=crP>Total params (B)</label><span class=tipw><button type=button class=tip aria-label='What is Total params (B)?' aria-describedby=tt2 aria-expanded=false>?</button><span role=tooltip id=tt2 class=tipb>Total parameters in billions. Weights (GB) = params × bits-per-weight ÷ 8. Edit it for a model we do not list.</span></span></span><input id=crP type=number min=0 step=any inputmode=decimal></div>
<div class=crf><span class=crl><label for=crA>Active params (MoE)</label><span class=tipw><button type=button class=tip aria-label='What is Active params (MoE)?' aria-describedby=tt3 aria-expanded=false>?</button><span role=tooltip id=tt3 class=tipb>MoE only: parameters used per token. They set the <i>speed</i> (less to read per token), not the memory: memory still follows total params. Small active counts make CPU/RAM offload of experts bearable.</span></span></span><input id=crA type=number min=0 step=any inputmode=decimal placeholder='dense'></div>
<div class="crf w"><span class=crl><label for=crE>Engine</label><span class=tipw><button type=button class=tip aria-label='What is Engine?' aria-describedby=tt4 aria-expanded=false>?</button><span role=tooltip id=tt4 class=tipb>The software that runs the model. <b>llama.cpp/Ollama/LM Studio</b> use GGUF files and run on almost anything. <b>vLLM/SGLang</b> use safetensors (FP8, AWQ, GPTQ, NVFP4) and shine with several GPUs and many users. <b>ExLlama</b> uses EXL2/EXL3, <b>MLX</b> is for Macs.</span></span></span><select id=crE></select></div>
<div class=crf><span class=crl><label for=crF>Quant / format</label><span class=tipw><button type=button class=tip aria-label='What is Quant / format?' aria-describedby=tt5 aria-expanded=false>?</button><span role=tooltip id=tt5 class=tipb>How many bits each weight uses. <b>Sweet spot: Q4_K_M or AWQ/GPTQ 4-bit</b> (≈4.25–4.9 bpw). Have room? <b>Q5_K_M / Q6_K</b>. Want near-lossless? <b>Q8_0 / FP8</b>. <b>NVFP4</b> (4.5 bpw) is Blackwell-only native. We advise <b>not going under ~4 bits</b> (Q3/Q2) unless nothing else fits.</span></span></span><select id=crF></select></div>
<div class=crf id=crBpwW hidden><span class=crl><label for=crBpw>Bits per weight</label><span class=tipw><button type=button class=tip aria-label='What is Bits per weight?' aria-describedby=tt6 aria-expanded=false>?</button><span role=tooltip id=tt6 class=tipb>EXL2/EXL3 let you choose the average bits per weight. 4.0–5.0 is the usual quality/size balance; under 4 the quality loss grows.</span></span></span><input id=crBpw type=number min=1.5 max=16 step=0.05 value=4.0></div>
<div class=crf><span class=crl><label for=crC>Context length</label><span class=tipw><button type=button class=tip aria-label='What is Context length?' aria-describedby=tt7 aria-expanded=false>?</button><span role=tooltip id=tt7 class=tipb>How many tokens (prompt + answer) the model keeps in view. Every token in context needs KV cache memory, so 128k context can need more memory than a small model's weights.</span></span></span><select id=crC></select><span class=muted id=crCtxMax></span></div>
<div class=crf><span class=crl><label for=crK>KV cache precision</label><span class=tipw><button type=button class=tip aria-label='What is KV cache precision?' aria-describedby=tt8 aria-expanded=false>?</button><span role=tooltip id=tt8 class=tipb>Precision of the stored keys/values. FP16 = 2 bytes per value. <b>q8_0 / FP8</b> halves it with very little quality loss; <b>q4_0</b> quarters it but can hurt long-context quality. llama.cpp: <code>-ctk q8_0 -ctv q8_0</code> (quantized V needs flash attention); vLLM: <code>--kv-cache-dtype fp8</code>.</span></span></span><select id=crK></select></div>
<div class=crf><span class=crl><label for=crS>Concurrent sequences</label><span class=tipw><button type=button class=tip aria-label='What is Concurrent sequences?' aria-describedby=tt9 aria-expanded=false>?</button><span role=tooltip id=tt9 class=tipb>Parallel conversations/requests. Each one needs its own KV cache, so 4 users at 32k = 4× the KV of one. Keep 1 for a personal chat.</span></span></span><input id=crS type=number min=1 max=256 step=1 value=1></div>
<div class="crf w"><span class=crl><label for=crH>GPU / system</label><span class=tipw><button type=button class=tip aria-label='What is GPU / system?' aria-describedby=tt10 aria-expanded=false>?</button><span role=tooltip id=tt10 class=tipb>Your GPU, or a unified-memory box (DGX Spark, Mac, Strix Halo) where CPU and GPU share one memory pool.</span></span></span><select id=crH></select></div>
<div class=crf><span class=crl><label for=crN>How many</label><span class=tipw><button type=button class=tip aria-label='What is How many?' aria-describedby=tt11 aria-expanded=false>?</button><span role=tooltip id=tt11 class=tipb>How many identical GPUs (or linked nodes). Usable memory ≈ count × (VRAM − overhead). Mixed cards work in llama.cpp (pick the smaller card here for a conservative estimate).</span></span></span><select id=crN></select></div>
<div class=crf id=crMemW hidden><span class=crl><label for=crMem>Memory per node</label><span class=tipw><button type=button class=tip aria-label='What is Memory per node?' aria-describedby=tt12 aria-expanded=false>?</button><span role=tooltip id=tt12 class=tipb>Unified memory per node. The GPU and the OS share it.</span></span></span><select id=crMem></select></div>
<div class=crf><span class=crl><label for=crO id=crOL>Per-GPU overhead, GB</label><span class=tipw><button type=button class=tip aria-label='What is the overhead?' aria-describedby=ttovh aria-expanded=false>?</button><span role=tooltip id=ttovh class=tipb>Memory each GPU loses before weights load: CUDA/ROCm context, compute buffers, CUDA graphs. ~1 GB is our assumption; long contexts and big batches need more. For unified memory it is the OS + runtime reserve per node. vLLM additionally only uses gpu_memory_utilization (default 0.92) of each GPU.</span></span></span><input id=crO type=number min=0 step=0.5 value=1></div>
<div class=crf><span class=crl><label for=crR>System RAM, GB</label><span class=tipw><button type=button class=tip aria-label='What is System RAM, GB?' aria-describedby=tt13 aria-expanded=false>?</button><span role=tooltip id=tt13 class=tipb>System RAM. If the model fits in VRAM this barely matters (see “RAM matters less than you think”). It matters only when layers or experts are offloaded to the CPU.</span></span></span><input id=crR type=number min=0 step=8 value=32></div>
<details class="crf w" id=crArch><summary>Model architecture (for KV cache) — from config.json, editable</summary>
<p class=muted id=crArchSrc></p><p class=muted style=margin:0><span class=crl>What are these?<span class=tipw><button type=button class=tip aria-label='What is What are these??' aria-describedby=tt14 aria-expanded=false>?</button><span role=tooltip id=tt14 class=tipb>From the model's config.json: num_hidden_layers (or the count of full_attention entries in layer_types), num_key_value_heads, head_dim. Sliding-window layers only cache their window.</span></span></span></p>
<div class=crag><div class=crf><span class=crl><label for=crFL>Full-attn layers</label></span><input id=crFL type=number min=0 step=any inputmode=decimal placeholder=''></div><div class=crf><span class=crl><label for=crFH>KV heads</label></span><input id=crFH type=number min=0 step=any inputmode=decimal placeholder=''></div><div class=crf><span class=crl><label for=crFD>head_dim</label></span><input id=crFD type=number min=0 step=any inputmode=decimal placeholder=''></div><div class=crf><span class=crl><label for=crSL>Sliding-window layers</label></span><input id=crSL type=number min=0 step=any inputmode=decimal placeholder='0'></div><div class=crf><span class=crl><label for=crSH>SWA KV heads</label></span><input id=crSH type=number min=0 step=any inputmode=decimal placeholder=''></div><div class=crf><span class=crl><label for=crSD>SWA head_dim</label></span><input id=crSD type=number min=0 step=any inputmode=decimal placeholder=''></div><div class=crf><span class=crl><label for=crSW>Window (tokens)</label></span><input id=crSW type=number min=0 step=any inputmode=decimal placeholder=''></div><div class=crf><span class=crl><label for=crML>MLA layers</label></span><input id=crML type=number min=0 step=any inputmode=decimal placeholder='0'></div><div class=crf><span class=crl><label for=crMD>MLA latent dim</label></span><input id=crMD type=number min=0 step=any inputmode=decimal placeholder=''></div></div></details>
</div></div>
<div><div class=card aria-live=polite><div id=crOut></div><p id=crRam style="margin-top:10px"></p><div id=crLinks></div></div><div id=crWarn></div>
<details class=card open><summary><b>Show the math</b> (with your numbers)</summary><div id=crMath></div></details>
<div id=crPar></div></div></div>
<div class=card><span class="bdg pend">estimate</span> above is arithmetic from published specs. <span class="bdg ok">measured</span> below is what real builds reported.<div id=crMeas></div></div>
<h2 id=kv-cache>How to calculate KV cache size</h2><div class=card>
<p>While generating, the model stores a key and a value vector for every token, in every attention layer. That store is the KV cache, and it grows with context:</p>
<p><code>KV bytes = 2 × layers × KV heads × head_dim × context tokens × concurrent sequences × bytes per value</code></p>
<ul><li><b>2</b>: one key and one value.</li><li><b>layers, KV heads, head_dim</b>: from the model's <code>config.json</code> (<code>num_hidden_layers</code>, <code>num_key_value_heads</code>, <code>head_dim</code>). Grouped-query attention models have far fewer KV heads than attention heads, which is why their cache is small.</li>
<li><b>bytes per value</b>: FP16/BF16 = 2, FP8 = 1, llama.cpp q8_0 = 1.0625 and q4_0 = 0.5625 (34 and 18 bytes per block of 32, scales included).</li></ul>
<p><b>Worked example, Llama 3.3 70B</b> (80 layers, 8 KV heads, head_dim 128, FP16): 2 × 80 × 8 × 128 × 2 B = 327,680 bytes per token (≈0.33 MB). At 32,768 tokens that is 10.7 GB, as much as a 16 GB card. At q8_0 it is about half.</p>
<p><b>Why 2026 models need less:</b> many new models only put full attention on some layers. <b>Sliding-window</b> layers (gpt-oss, Gemma 4, Muse, Step 5) cache just their window (128–2,048 tokens). <b>Linear-attention / Gated DeltaNet</b> layers (Qwen3.5–3.8, Ornith) and <b>Mamba</b> layers (Nemotron) keep a fixed-size state that does not grow. <b>MLA</b> (DeepSeek R1) caches one compressed latent per layer. The calculator reads these per-layer types from each model's config.json and counts only what grows with context.</p>
<p class=muted>Engines add some extra for compute buffers, and allocate KV in blocks/pages. vLLM pre-allocates all memory left after weights (up to <code>gpu_memory_utilization</code>, default 0.92) as KV cache, so it will look full however small your context is (<a href="https://github.com/vllm-project/vllm/blob/main/vllm/config/cache.py" rel=nofollow>vLLM source</a>, <a href="https://docs.vllm.ai/en/latest/features/quantization/quantized_kvcache.html" rel=nofollow>FP8 KV cache docs</a>).</p></div>
<h2 id=formats>Which quant or format should I use?</h2><div class=card>
<ul><li><b>Sweet spot:</b> Q4_K_M (GGUF) or AWQ / GPTQ 4-bit (vLLM). This is the size most people run.</li><li><b>Have room?</b> Q5_K_M or Q6_K, closer to the original.</li><li><b>Near-lossless:</b> Q8_0 or FP8.</li>
<li><b>Below ~4 bits</b> (Q3_K, Q2_K, IQ3, 3-bit EXL) we do not recommend it unless nothing else fits; the calculator warns you. Very large models tolerate it better than small ones.</li></ul>
<table><tr><th>Format</th><th class=n>Bits / weight</th><th>Engines</th></tr><tr><td>Q8_0 (GGUF)</td><td class=n>8.50</td><td>llama.cpp / Ollama / LM Studio</td></tr><tr><td>Q6_K (GGUF)</td><td class=n>6.56</td><td>llama.cpp / Ollama / LM Studio</td></tr><tr><td>Q5_K_M (GGUF)</td><td class=n>5.70</td><td>llama.cpp / Ollama / LM Studio</td></tr><tr><td>Q4_K_M (GGUF)</td><td class=n>4.89</td><td>llama.cpp / Ollama / LM Studio</td></tr><tr><td>IQ4_XS (GGUF)</td><td class=n>4.46</td><td>llama.cpp / Ollama / LM Studio</td></tr><tr><td>Q3_K_M (GGUF)</td><td class=n>4.00</td><td>llama.cpp / Ollama / LM Studio</td></tr><tr><td>Q2_K (GGUF)</td><td class=n>3.16</td><td>llama.cpp / Ollama / LM Studio</td></tr><tr><td>F16 (GGUF)</td><td class=n>16.00</td><td>llama.cpp / Ollama / LM Studio</td></tr><tr><td>MXFP4 (gpt-oss native)</td><td class=n>4.25</td><td>llama.cpp / Ollama / LM Studio, vLLM / SGLang</td></tr><tr><td>BF16 / FP16 (unquantized)</td><td class=n>16.00</td><td>vLLM / SGLang</td></tr><tr><td>FP8 (W8A8)</td><td class=n>8.00</td><td>vLLM / SGLang</td></tr><tr><td>AWQ 4-bit (g128, ≈)</td><td class=n>4.25</td><td>vLLM / SGLang</td></tr><tr><td>GPTQ 4-bit (g128, ≈)</td><td class=n>4.25</td><td>vLLM / SGLang</td></tr><tr><td>GPTQ 8-bit (g128, ≈)</td><td class=n>8.25</td><td>vLLM / SGLang</td></tr><tr><td>NVFP4 (Blackwell)</td><td class=n>4.50</td><td>vLLM / SGLang</td></tr><tr><td>EXL3 (ExLlamaV3)</td><td class=n>you choose (2–8)</td><td>ExLlamaV2 / V3</td></tr><tr><td>EXL2 (ExLlamaV2)</td><td class=n>you choose (2–8)</td><td>ExLlamaV2 / V3</td></tr><tr><td>MLX 4-bit</td><td class=n>4.50</td><td>MLX</td></tr><tr><td>MLX 8-bit</td><td class=n>8.50</td><td>MLX</td></tr></table>
<p><b>Hardware matters for some formats.</b> <b>FP8</b> compute needs compute capability ≥ 8.9 (Ada, Hopper, Blackwell); on Ampere cards such as the RTX 3090, vLLM runs FP8 checkpoints as weight-only W8A16 with FP8 Marlin kernels, saving memory but not compute (<a href="https://docs.vllm.ai/en/latest/features/quantization/llm_compressor/fp8.html" rel=nofollow>vLLM FP8 docs</a>). <b>NVFP4</b> is 4-bit E2M1 values with one FP8 scale per 16 values plus a per-tensor FP32 scale (4.5 bits per weight), about 3.5× smaller than FP16 and 1.8× smaller than FP8, and is computed natively by Blackwell's 5th-gen Tensor Cores (<a href="https://developer.nvidia.com/blog/introducing-nvfp4-for-efficient-and-accurate-low-precision-inference/" rel=nofollow>NVIDIA</a>). On older NVIDIA GPUs vLLM falls back to weight-only W4A16 via Marlin (<a href="https://docs.vllm.ai/en/latest/features/quantization/modelopt.html" rel=nofollow>vLLM ModelOpt docs</a>). <b>MXFP4</b> (gpt-oss) shares one power-of-two scale per 32 values, 4.25 bits per weight; gpt-oss keeps attention and embeddings in BF16, so its real files are larger than params × 4.25 ÷ 8.</p>
<p class=muted>GGUF bits-per-weight are llama.cpp's measured values (<a href="https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md" rel=nofollow>quantize README</a>); real files vary a little by model. AWQ/GPTQ ≈ 4.25 / 8.25 are our approximation (4 or 8 bits plus group-128 scales and zero points); embeddings and lm_head often stay 16-bit, so files run slightly larger. MLX: group size 64 with a 16-bit scale and bias, so 4-bit = 4.5 bpw (<a href="https://ml-explore.github.io/mlx/build/html/python/_autosummary/mlx.core.quantize.html" rel=nofollow>MLX docs</a>). vLLM hardware support: <a href="https://docs.vllm.ai/en/latest/features/quantization/" rel=nofollow>vLLM quantization docs</a>.</p></div>
<h2 id=ram>RAM matters less than you think</h2><div class=card style="border-color:var(--acc)">
<p>Most calculators tell you to buy as much system RAM as VRAM, or more. <b>If the model fits fully in VRAM, you don't need that.</b> System RAM then only holds the OS and the inference engine, and the weights pass through it once while loading. llama.cpp memory-maps the model file, so pages are read from disk and copied to the GPUs. They don't all have to sit in RAM at once, so RAM can be smaller than the model.</p>
<p>RAM becomes important only when you <b>offload</b>: layers or MoE experts that don't fit on the GPUs run from system RAM, and then both its size and its bandwidth (channels, DDR4 vs DDR5) set your speed.</p>
<p><b>Maintainer note:</b> our maintainer runs <b>8× RTX 3090 with just 32 GB of DDR5 ECC RDIMM</b> at home (for a measured 8x 3090 example from another builder, see <a href='https://getrigbase.com/builds/8x-rtx-3090-deepseek-v4-flash-limechain/?utm_source=huggingface&utm_medium=space'>8x RTX 3090 serving DeepSeek V4 Flash (LimeChain vLLM kit)</a>), and it works really well, because the models live in the 192 GB of VRAM. Spend the money on GPUs, not RAM, unless you plan to offload.</p>
<p class=muted>Caveat: converting or quantizing a model yourself, and some loaders, can need much more RAM for a short time. Keep swap enabled if you do that.</p></div>
<h2 id=multi-gpu>Multi-GPU and multi-node: tensor vs pipeline parallel</h2><div class=card>
<p><b>Memory adds up:</b> 4× RTX 3090 = 96 GB, 8× = 192 GB, two DGX Sparks = 256 GB. Each GPU loses roughly 1 GB to its CUDA context and buffers (our assumption), and vLLM only uses 0.92 of each GPU by default.</p>
<ul><li><b>Pipeline / layer split</b> (llama.cpp default, vLLM <code>--pipeline-parallel-size</code>, ExLlama): each GPU holds a block of layers. Any number of GPUs, different sizes allowed, little PCIe traffic. One chat stream uses one GPU at a time, so you get capacity rather than speed.</li>
<li><b>Tensor parallel</b> (vLLM/SGLang <code>--tensor-parallel-size</code>, llama.cpp <code>--split-mode row</code>): every layer is split over all GPUs, which work on each token together. It is faster, but needs a lot of interconnect bandwidth, and in vLLM the attention-head count must divide by the TP size, so <b>2, 4 or 8 GPUs</b> are the usual choice. For 3 or 6 GPUs, combine TP with PP or use PP only (<a href="https://docs.vllm.ai/en/latest/serving/parallelism_scaling.html" rel=nofollow>vLLM parallelism docs</a>).</li>
<li><b>Multi-node:</b> NVIDIA says up to four DGX Spark systems can work together over ConnectX networking (each Spark has 128 GB unified memory, 273 GB/s and a 200 Gb/s ConnectX-7 NIC; <a href="https://www.nvidia.com/en-us/products/workstations/dgx-spark/" rel=nofollow>NVIDIA</a>). Mac Studios can cluster over Thunderbolt. See the <a href="https://getrigbase.com/dgx-spark/?utm_source=huggingface&utm_medium=space">DGX Spark recipes</a> and the measured multi-node builds.</li></ul>
<p class=muted>RigBase has 33 measured multi-GPU / multi-node builds; the calculator lists those matching your setup.</p></div>
<h2 id=prefill>Why prefill speed matters for coding assistants</h2><div class=card><p>The tokens/sec most benchmarks quote is <b>generation</b> (decode) speed: how fast the answer streams out. Before the first token appears the model must first read the whole prompt. That is <b>prompt processing</b>, or prefill. Chat prompts are short, so prefill hardly matters there. Coding assistants and agents are different. They send the open files, the repo map, tool output and the chat history on every turn, and these prompts easily reach thousands of tokens. On such prompts the wait before the first token (time to first token, TTFT) can take longer than generating the answer.</p>
<p>Prefill is compute-bound, while decode is mostly limited by memory bandwidth. So two rigs with similar generation speed can differ a lot in prefill. Discrete NVIDIA/AMD GPUs usually prefill much faster than unified-memory machines and CPU offload setups. Rough TTFT for a prompt: <code>TTFT ≈ prompt tokens ÷ prefill tok/s</code> (plus model load time; prompt caching in llama.cpp, MLX or vLLM can skip the repeated part).</p>
<p>RigBase now records prefill speed for 168 results on 90 builds, with the prompt length each source measured it at. Where it was measured on 512+ tokens we show an <b>estimated TTFT for an 8k-token prompt</b>, calculated from that measured speed. Across the current data the estimate ranges from Llama 2 7B on <a href='https://getrigbase.com/builds/rtx-4090-llamacpp-cuda/?utm_source=huggingface&utm_medium=space'>Single RTX 4090 (llama.cpp CUDA scoreboard)</a>: 14,771 tok/s prefill measured @ 512 tokens → ≈0.55 s to Llama 3.1 405B on <a href='https://getrigbase.com/builds/mac-studio-m3-ultra-512gb-geerling/?utm_source=huggingface&utm_medium=space'>Mac Studio M3 Ultra 512GB (Geerling test)</a>: 12.92 tok/s prefill measured @ 4,096 tokens → ≈634 s. Prefill usually slows down as prompts get longer, so an estimate taken from a much shorter or much longer prompt is only indicative. Where a source measured TTFT itself, we show it separately as <i>measured</i>. See the "Prompt processing" column on every <a href='https://getrigbase.com/builds/?utm_source=huggingface&utm_medium=space'>build</a> and <a href='https://getrigbase.com/models/?utm_source=huggingface&utm_medium=space'>model</a> page.</p></div>
<div class=card style="border-color:#238636"><b>Not enough VRAM? Rent before you buy.</b> Run the model on a rented 24–96 GB cloud GPU by the hour and see the real speed before spending €1,000+. <a class=btn rel="sponsored nofollow" href="https://runpod.io?ref=77qcpl2c">Try it on Runpod</a><br><span class=muted>Referral link: if you sign up with Google and add $10, Runpod gives you and RigBase bonus credit (<a href="https://docs.runpod.io/get-started/referrals" rel=nofollow>terms</a>). It never affects our data.</span></div>
<h2>Weights size by quant (GB)</h2><table><tr><th>Model</th><th class=n>Params</th><th class=n>Q6_K</th><th class=n>Q5_K_M</th><th class=n>Q4_K_M</th><th class=n>Q3_K_M</th><th>Param source</th></tr><tr><td>Qwen3.8 27B</td><td class=n>27B</td><td class=n>22.2</td><td class=n>19.2</td><td class=n>16.5</td><td class=n>13.5</td><td><a href='https://ollama.com/library/qwen3.8' rel=nofollow>Ollama library tag 27b (nominal)</a></td></tr><tr><td>Qwen3.6 35B-A3B</td><td class=n>35.51B (3B active)</td><td class=n>29.1</td><td class=n>25.3</td><td class=n>21.7</td><td class=n>17.7</td><td><a href='https://llamabench.ai/browse/rtx3090/qwen3-6-35b-a3b' rel=nofollow>GGUF param count via llamabench.ai; 3B active per name</a></td></tr><tr><td>Qwen3.6 27B</td><td class=n>27.32B</td><td class=n>22.4</td><td class=n>19.5</td><td class=n>16.7</td><td class=n>13.6</td><td><a href='https://llamabench.ai/browse/rtx5090' rel=nofollow>GGUF param count via llamabench.ai</a></td></tr><tr><td>Gemma 4 31B</td><td class=n>30.7B</td><td class=n>25.2</td><td class=n>21.9</td><td class=n>18.8</td><td class=n>15.3</td><td><a href='https://llamabench.ai/browse/rtx5090' rel=nofollow>GGUF param count via llamabench.ai</a></td></tr><tr><td>Gemma 4 26B-A4B</td><td class=n>25.23B (4B active)</td><td class=n>20.7</td><td class=n>18.0</td><td class=n>15.4</td><td class=n>12.6</td><td><a href='https://llamabench.ai/browse/rtx5090' rel=nofollow>GGUF param count via llamabench.ai</a></td></tr><tr><td>Gemma 4 12B</td><td class=n>11.91B</td><td class=n>9.8</td><td class=n>8.5</td><td class=n>7.3</td><td class=n>5.9</td><td><a href='https://llamabench.ai/browse/rtx5090' rel=nofollow>GGUF param count via llamabench.ai</a></td></tr><tr><td>gpt-oss 20B</td><td class=n>20.91B (3.6B active)</td><td class=n>17.2</td><td class=n>14.9</td><td class=n>12.8</td><td class=n>10.4</td><td><a href='https://llamabench.ai/browse/rtx5090' rel=nofollow>GGUF param count via llamabench.ai (OpenAI model card: 21B total, 3.6B active)</a></td></tr><tr><td>gpt-oss 120B</td><td class=n>117B (5.1B active)</td><td class=n>96.0</td><td class=n>83.4</td><td class=n>71.6</td><td class=n>58.4</td><td><a href='https://huggingface.co/openai/gpt-oss-120b' rel=nofollow>OpenAI model card: 117B total, 5.1B active</a></td></tr><tr><td>Qwen3.5 9B</td><td class=n>8.95B</td><td class=n>7.3</td><td class=n>6.4</td><td class=n>5.5</td><td class=n>4.5</td><td><a href='https://llamabench.ai/browse/rtx5090' rel=nofollow>GGUF param count via llamabench.ai</a></td></tr><tr><td>Qwen3.5 122B</td><td class=n>122B</td><td class=n>100.1</td><td class=n>87.0</td><td class=n>74.6</td><td class=n>60.9</td><td><a href='https://ollama.com/library/qwen3.5' rel=nofollow>Ollama library tag 122b (nominal)</a></td></tr><tr><td>LFM2.5 24B-A2B</td><td class=n>23.97B (2.45B active)</td><td class=n>19.7</td><td class=n>17.1</td><td class=n>14.7</td><td class=n>12.0</td><td><a href='https://llamabench.ai/browse/rtx4090' rel=nofollow>via llamabench.ai</a></td></tr><tr><td>Nemotron 3.5 Bolt 74B-A9B</td><td class=n>74.04B (9B active)</td><td class=n>60.7</td><td class=n>52.8</td><td class=n>45.3</td><td class=n>37.0</td><td><a href='https://llamabench.ai/browse/rtx4090' rel=nofollow>via llamabench.ai</a></td></tr><tr><td>Nemotron 3 Super 120B-A12B</td><td class=n>120B (12B active)</td><td class=n>98.4</td><td class=n>85.6</td><td class=n>73.4</td><td class=n>59.9</td><td><a href='https://ollama.com/library/nemotron-3-super' rel=nofollow>Ollama library: 120B, 12B active</a></td></tr><tr><td>Nemotron 3.5 Super VL 120B-A12B</td><td class=n>120B (12B active)</td><td class=n>98.4</td><td class=n>85.6</td><td class=n>73.4</td><td class=n>59.9</td><td><a href='https://blogs.nvidia.com/blog/nemotron-3-5-super-vl/' rel=nofollow>NVIDIA launch blog (15 Oct 2026): 120B params; 12B active per model name</a></td></tr><tr><td>Mistral Medium 3.5 128B</td><td class=n>128B</td><td class=n>105.0</td><td class=n>91.3</td><td class=n>78.3</td><td class=n>63.9</td><td><a href='https://ollama.com/library/mistral-medium-3.5' rel=nofollow>Ollama library: 128B weights</a></td></tr><tr><td>Devstral Small 2 24B</td><td class=n>24B</td><td class=n>19.7</td><td class=n>17.1</td><td class=n>14.7</td><td class=n>12.0</td><td><a href='https://ollama.com/library/devstral-small-2' rel=nofollow>Ollama library tag 24b</a></td></tr><tr><td>Muse Spark Open 816B-A42B</td><td class=n>816B (42B active)</td><td class=n>669.5</td><td class=n>581.8</td><td class=n>499.2</td><td class=n>407.6</td><td><a href='https://huggingface.co/blog/meta-muse-spark-open' rel=nofollow>Hugging Face launch post: ~816B total, ~42B active</a></td></tr><tr><td>GLM-5.3</td><td class=n>753B</td><td class=n>617.8</td><td class=n>536.9</td><td class=n>460.7</td><td class=n>376.1</td><td><a href='https://huggingface.co/zai-org/GLM-5.3' rel=nofollow>Hugging Face listing: 753B</a></td></tr><tr><td>DeepSeek R1 671B</td><td class=n>671B (37B active)</td><td class=n>550.5</td><td class=n>478.4</td><td class=n>410.5</td><td class=n>335.2</td><td><a href='https://ollama.com/library/deepseek-v3' rel=nofollow>Ollama library: 671B total, 37B active (V3 base)</a></td></tr><tr><td>Llama 3.3 70B <span class=muted>(older)</span></td><td class=n>70B</td><td class=n>57.4</td><td class=n>49.9</td><td class=n>42.8</td><td class=n>35.0</td><td><a href='https://ollama.com/library/llama3.3' rel=nofollow>Ollama library tag 70b</a></td></tr><tr><td>Llama 3.1 8B <span class=muted>(older)</span></td><td class=n>8B</td><td class=n>6.6</td><td class=n>5.7</td><td class=n>4.9</td><td class=n>4.0</td><td><a href='https://ollama.com/library/llama3.1' rel=nofollow>Ollama library tag 8b</a></td></tr><tr><td>Qwen3.8 Flash-Next</td><td class=n>180B (6B active)</td><td class=n>147.7</td><td class=n>128.3</td><td class=n>110.1</td><td class=n>89.9</td><td><a href='https://huggingface.co/Qwen/Qwen3.8-Flash-Next' rel=nofollow>Official Hugging Face model card: 125B backbone with 6B activated parameters, plus a 51B n-gram embedding table and a 4B multi-token prediction (MTP) module; Hugging Face lists the combined checkpoint as 180B params. params_b=180 is the combined listing; active_b=6 is the activated backbone count.</a></td></tr><tr><td>Nemotron 3.5 Lightning 30B-A3B</td><td class=n>30B (3B active)</td><td class=n>24.6</td><td class=n>21.4</td><td class=n>18.4</td><td class=n>15.0</td><td><a href='https://ctala.github.io/local-llm-agentic-workflows/benchmarks/' rel=nofollow>30B total / 3B active MoE (hybrid Mamba), per DGX Spark benchmark table</a></td></tr><tr><td>Flint 40B-A2.5B</td><td class=n>39.8B (2.5B active)</td><td class=n>32.7</td><td class=n>28.4</td><td class=n>24.3</td><td class=n>19.9</td><td><a href='https://huggingface.co/RecursiveSI/flint' rel=nofollow>39.8B total / ~2.5B active MoE coding model (Recursive, Apache 2.0), per Hugging Face model card</a></td></tr><tr><td>GLM-5.3 Flash</td><td class=n>320B (18B active)</td><td class=n>262.5</td><td class=n>228.1</td><td class=n>195.8</td><td class=n>159.8</td><td><a href='https://huggingface.co/zai-org/GLM-5.3-Flash' rel=nofollow>Z.ai model card: 320B total, 18B active (Hugging Face lists 321B)</a></td></tr><tr><td>Ornith 1.5 122B</td><td class=n>125.1B (10B active)</td><td class=n>102.6</td><td class=n>89.2</td><td class=n>76.5</td><td class=n>62.5</td><td><a href='https://huggingface.co/ornith-ai/Ornith-1.5-122B' rel=nofollow>Hugging Face: 125B incl. vision tower + MTP layer (122B language model, 10B active per halt95/Ornith-1.5-122B-A10B card)</a></td></tr><tr><td>DeepSeek V4.1 Flash</td><td class=n>748.6B (16B active)</td><td class=n>614.2</td><td class=n>533.7</td><td class=n>458.0</td><td class=n>373.9</td><td><a href='https://recipes.vllm.ai/deepseek-ai/DeepSeek-V4.1-Flash' rel=nofollow>vLLM recipe: 552B backbone + 196.6B Engram tables, 16B active per output token; official MXFP4/FP8 checkpoint ~511 GB</a></td></tr><tr><td>GLM-5.2</td><td class=n>753.3B</td><td class=n>618.0</td><td class=n>537.1</td><td class=n>460.9</td><td class=n>376.3</td><td><a href='https://huggingface.co/zai-org/GLM-5.2' rel=nofollow>Hugging Face safetensors total: 753.3B parameters (MoE; the model card does not state the active count)</a></td></tr><tr><td>Ornith 1.5 35B-A3B</td><td class=n>35.95B (3B active)</td><td class=n>29.5</td><td class=n>25.6</td><td class=n>22.0</td><td class=n>18.0</td><td><a href='https://huggingface.co/ornith-ai/Ornith-1.5-35B-A3B' rel=nofollow>Hugging Face safetensors total: 35.95B; model card: activates ~3B parameters per token</a></td></tr><tr><td>MiniMax M3</td><td class=n>428B (23B active)</td><td class=n>351.1</td><td class=n>305.1</td><td class=n>261.9</td><td class=n>213.8</td><td><a href='https://huggingface.co/MiniMaxAI/MiniMax-M3' rel=nofollow>Hugging Face model card: ~428B total, ~23B activated</a></td></tr><tr><td>MiniMax M3.1 Flash</td><td class=n>428B (23B active)</td><td class=n>351.1</td><td class=n>305.1</td><td class=n>261.9</td><td class=n>213.8</td><td><a href='https://huggingface.co/MiniMaxAI/MiniMax-M3.1-Flash' rel=nofollow>Hugging Face model card: ~428B total, 23B activated per token (quantization-aware trained)</a></td></tr><tr><td>Kolibri-1</td><td class=n>78.1B (3.46B active)</td><td class=n>64.1</td><td class=n>55.7</td><td class=n>47.8</td><td class=n>39.0</td><td><a href='https://huggingface.co/Aleph-Alpha/Kolibri-1' rel=nofollow>Aleph Alpha model card: 78,103,074,560 total, 3,457,573,120 active per token</a></td></tr><tr><td>Step 5 Preview</td><td class=n>600B (27B active)</td><td class=n>492.2</td><td class=n>427.8</td><td class=n>367.1</td><td class=n>299.7</td><td><a href='https://huggingface.co/stepfun-ai/Step-5-Preview' rel=nofollow>StepFun model card: 600B total, 27B active (MoE)</a></td></tr><tr><td>Mellum 2.1 12B-A2.5B</td><td class=n>12B (2.5B active)</td><td class=n>9.8</td><td class=n>8.6</td><td class=n>7.3</td><td class=n>6.0</td><td><a href='https://llm-bench.io/models/mellum2-1-12b-a2-5b-thinking-gguf-q4-k-m' rel=nofollow>12B total / 2.5B active per the model name (JetBrains Mellum 2.1)</a></td></tr><tr><td>Ling 3.0 Flash</td><td class=n>124B (5.1B active)</td><td class=n>101.7</td><td class=n>88.4</td><td class=n>75.9</td><td class=n>61.9</td><td><a href='https://huggingface.co/inclusionAI/Ling-3.0-flash' rel=nofollow>inclusionAI model card: 124B total, 5.1B activated (Hugging Face safetensors total 127.5B)</a></td></tr><tr><td>MiMo V2.6 Flash</td><td class=n>309B (15B active)</td><td class=n>253.5</td><td class=n>220.3</td><td class=n>189.0</td><td class=n>154.3</td><td><a href='https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL' rel=nofollow>Xiaomi MiMo model card: sparse MoE, 309B total / 15B activated</a></td></tr></table>
<p class=muted>gpt-oss ships natively in MXFP4; OpenAI's card says gpt-oss-120b runs on a single 80 GB GPU and gpt-oss-20b within 16 GB of memory.</p>
<h2 id=faq>FAQ</h2><details class=card><summary><b>Can I run gpt-oss 120B on 4x RTX 3090?</b></summary><p>Yes, by our estimate: gpt-oss 120B in its native MXFP4 checkpoint (65 GB) needs about 65.0 GB for the weights plus 4.8 GB of KV cache at 128k context, 69.8 GB in total, against about 92.0 GB usable on 4× RTX 3090 (96 GB, minus ~1 GB overhead per card). This is arithmetic, not a measurement; check the measured builds linked on the page. gpt-oss uses sliding-window attention on half its layers, which keeps the KV cache small.</p></details><details class=card><summary><b>Can I run Qwen3 32B on one RTX 3090 at 32k context?</b></summary><p>Not fully, by our estimate: Qwen3 32B in Q4_K_M needs about 20.0 GB for the weights plus 8.6 GB of KV cache at 32k context, 28.6 GB in total, against about 23.0 GB usable on one RTX 3090. This is arithmetic, not a measurement; check the measured builds linked on the page. With the KV cache quantized to q4_0 it needs about 22.5 GB, which fits; with FP16 KV the largest context that fits is about 11k tokens.</p></details><details class=card><summary><b>Can two DGX Sparks run a ~200B model?</b></summary><p>Yes, by our estimate: Qwen3.8 Flash-Next in NVFP4 needs about 101.2 GB for the weights plus 3.2 GB of KV cache at 128k context, 104.5 GB in total, against about 238.0 GB usable on 2× DGX Spark (2 × 128 GB, minus ~9 GB per node for OS and runtime). This is arithmetic, not a measurement; check the measured builds linked on the page. NVIDIA says up to four Sparks can be linked over ConnectX-7.</p></details><details class=card><summary><b>How do I calculate KV cache size?</b></summary><p>KV cache bytes = 2 (keys and values) × attention layers × KV heads × head dimension × context tokens × concurrent sequences × bytes per value (FP16 = 2, FP8 = 1, q8_0 ≈ 1.06, q4_0 ≈ 0.56). Read layers, num_key_value_heads and head_dim from the model&#x27;s config.json. Example: Llama 3.3 70B has 80 layers, 8 KV heads and head_dim 128, so one token costs 2 × 80 × 8 × 128 × 2 B = 327,680 bytes (0.33 MB), and 32k tokens cost 10.7 GB. Sliding-window layers only store their window; linear-attention and Mamba layers keep a fixed-size state.</p></details><details class=card><summary><b>How much VRAM does an NVFP4 model need?</b></summary><p>NVFP4 stores 4-bit values with one FP8 scale per 16 values, about 4.5 bits per weight, so weights take roughly parameters × 0.5625 GB per billion: a 120B model is about 67.5 GB before KV cache. It is about 3.5× smaller than FP16 and 1.8× smaller than FP8 (NVIDIA). It runs natively only on Blackwell GPUs; vLLM falls back to slower weight-only kernels elsewhere.</p></details><details class=card><summary><b>Which quant should I use?</b></summary><p>Q4_K_M (GGUF) or AWQ / GPTQ 4-bit (vLLM) is the usual sweet spot of size and quality. Use Q5_K_M or Q6_K if you have the room, and Q8_0 or FP8 when you want near-lossless output. We do not recommend going below about 4 bits per weight (Q3, Q2) unless nothing else fits, because quality drops faster there.</p></details><details class=card><summary><b>Does system RAM matter for running local LLMs?</b></summary><p>Only if the model does not fit in VRAM. When weights and KV cache sit fully on the GPUs, system RAM just holds the OS and the inference engine and loads the weights once; llama.cpp memory-maps the file, so RAM can be smaller than the model. RAM size and speed matter when you offload layers or MoE experts to the CPU.</p></details><details class=card><summary><b>Tensor parallel or pipeline parallel?</b></summary><p>Tensor parallel (vLLM -tp) splits every layer across GPUs so they work on each token together: faster, but it needs the head count to divide by the GPU count (2, 4, 8 are usual) and good interconnect. Pipeline / layer split (llama.cpp default, vLLM -pp) gives each GPU a block of layers: any GPU count works and it needs little bandwidth, but one stream uses one GPU at a time.</p></details>
<p class=muted>Architecture values come from each model's Hugging Face config.json (linked in the calculator). Everything marked <i>estimate</i> is arithmetic, not a benchmark. Spotted a wrong value? <a href="https://getrigbase.com/submit/?utm_source=huggingface&utm_medium=space">Tell us</a>.</p><aside class=hads aria-label="Sponsor slot"><span class=hadl>Ad · this space is available</span><b>Sponsor this spot: reach people building local AI rigs.</b> <span class=muted>One clearly-labeled slot per page, no tracking scripts.</span> <a href="https://getrigbase.com/advertise/?utm_source=huggingface&utm_medium=space#inquiry">Sponsor this spot →</a></aside>
<script>const CR={"models": [{"n": "Qwen3.5 9B", "p": 8.95, "a": null, "cur": true, "pg": true, "ar": {"repo": "Qwen/Qwen3.5-9B", "cfg": "https://huggingface.co/Qwen/Qwen3.5-9B/blob/main/config.json", "qh": 16, "max": 262144, "full": [8, 4, 256], "other": {"linear_attention": 24}, "note": "24 linear attention layers keep a small fixed-size state, not a per-token KV cache, so they are not counted"}, "nf": null, "ng": null}, {"n": "Gemma 4 12B", "p": 11.91, "a": null, "cur": true, "pg": true, "ar": {"repo": "google/gemma-4-12B-it", "cfg": "https://huggingface.co/google/gemma-4-12B-it/blob/main/config.json", "qh": 16, "max": 262144, "full": [8, 1, 512], "swa": [40, 8, 256, 1024], "note": "config sets attention_k_eq_v (keys double as values) on full layers; engines that exploit it need about half the full-layer KV shown here — we keep the conservative 2× (K and V) figure"}, "nf": null, "ng": null}, {"n": "Mellum 2.1 12B-A2.5B", "p": 12, "a": 2.5, "cur": true, "pg": true, "ar": {"repo": "JetBrains/Mellum2.1-12B-A2.5B-Thinking", "cfg": "https://huggingface.co/JetBrains/Mellum2.1-12B-A2.5B-Thinking/blob/main/config.json", "qh": 32, "max": 131072, "full": [7, 4, 128], "swa": [21, 4, 128, 1024]}, "nf": null, "ng": null}, {"n": "gpt-oss 20B", "p": 20.91, "a": 3.6, "cur": true, "pg": true, "ar": {"repo": "openai/gpt-oss-20b", "cfg": "https://huggingface.co/openai/gpt-oss-20b/blob/main/config.json", "qh": 64, "max": 131072, "full": [12, 8, 64], "swa": [12, 8, 64, 128]}, "nf": "MXFP4", "ng": 13}, {"n": "LFM2.5 24B-A2B", "p": 23.97, "a": 2.45, "cur": true, "pg": true, "ar": {"repo": "LiquidAI/LFM2.5-24B-A2B", "cfg": "https://huggingface.co/LiquidAI/LFM2.5-24B-A2B/blob/main/config.json", "qh": 32, "max": 131072, "full": [10, 8, 64], "other": {"conv": 30}, "note": "head_dim not in config; derived hidden_size 2048 ÷ 32 heads = 64; 30 conv layers keep a small fixed-size state, not a per-token KV cache, so they are not counted"}, "nf": null, "ng": null}, {"n": "Gemma 4 26B-A4B", "p": 25.23, "a": 4, "cur": true, "pg": true, "ar": {"repo": "google/gemma-4-26B-A4B-it", "cfg": "https://huggingface.co/google/gemma-4-26B-A4B-it/blob/main/config.json", "qh": 16, "max": 262144, "full": [5, 2, 512], "swa": [25, 8, 256, 1024], "note": "config sets attention_k_eq_v (keys double as values) on full layers; engines that exploit it need about half the full-layer KV shown here — we keep the conservative 2× (K and V) figure"}, "nf": null, "ng": null}, {"n": "Qwen3.8 27B", "p": 27, "a": null, "cur": true, "pg": true, "ar": {"repo": "Qwen/Qwen3.8-27B", "cfg": "https://huggingface.co/Qwen/Qwen3.8-27B/blob/main/config.json", "qh": 24, "max": 262144, "full": [16, 4, 256], "other": {"linear_attention": 48}, "note": "48 linear attention layers keep a small fixed-size state, not a per-token KV cache, so they are not counted"}, "nf": null, "ng": null}, {"n": "Qwen3.6 27B", "p": 27.32, "a": null, "cur": true, "pg": true, "ar": {"repo": "Qwen/Qwen3.6-27B", "cfg": "https://huggingface.co/Qwen/Qwen3.6-27B/blob/main/config.json", "qh": 24, "max": 262144, "full": [16, 4, 256], "other": {"linear_attention": 48}, "note": "48 linear attention layers keep a small fixed-size state, not a per-token KV cache, so they are not counted"}, "nf": null, "ng": null}, {"n": "Nemotron 3.5 Lightning 30B-A3B", "p": 30, "a": 3, "cur": true, "pg": true, "ar": {"repo": "nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16", "cfg": "https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16/blob/main/config.json", "qh": 32, "max": 262144, "full": [6, 2, 128], "other": {"mamba": 23}, "note": "23 mamba layers keep a small fixed-size state, not a per-token KV cache, so they are not counted"}, "nf": null, "ng": null}, {"n": "Gemma 4 31B", "p": 30.7, "a": null, "cur": true, "pg": true, "ar": {"repo": "google/gemma-4-31B-it", "cfg": "https://huggingface.co/google/gemma-4-31B-it/blob/main/config.json", "qh": 32, "max": 262144, "full": [10, 4, 512], "swa": [50, 16, 256, 1024], "note": "config sets attention_k_eq_v (keys double as values) on full layers; engines that exploit it need about half the full-layer KV shown here — we keep the conservative 2× (K and V) figure"}, "nf": null, "ng": null}, {"n": "Qwen3.6 35B-A3B", "p": 35.51, "a": 3, "cur": true, "pg": true, "ar": {"repo": "Qwen/Qwen3.6-35B-A3B", "cfg": "https://huggingface.co/Qwen/Qwen3.6-35B-A3B/blob/main/config.json", "qh": 16, "max": 262144, "full": [10, 2, 256], "other": {"linear_attention": 30}, "note": "30 linear attention layers keep a small fixed-size state, not a per-token KV cache, so they are not counted"}, "nf": null, "ng": null}, {"n": "Ornith 1.5 35B-A3B", "p": 35.95, "a": 3, "cur": true, "pg": true, "ar": {"repo": "ornith-ai/Ornith-1.5-35B-A3B", "cfg": "https://huggingface.co/ornith-ai/Ornith-1.5-35B-A3B/blob/main/config.json", "qh": 16, "max": 262144, "full": [10, 2, 256], "other": {"linear_attention": 30}, "note": "30 linear attention layers keep a small fixed-size state, not a per-token KV cache, so they are not counted"}, "nf": null, "ng": null}, {"n": "Flint 40B-A2.5B", "p": 39.8, "a": 2.5, "cur": true, "pg": true, "ar": {"repo": "RecursiveSI/flint", "cfg": "https://huggingface.co/RecursiveSI/flint/blob/main/config.json", "qh": 32, "max": 262144, "full": [14, 4, 128], "swa": [35, 4, 128, 513]}, "nf": null, "ng": null}, {"n": "Nemotron 3.5 Bolt 74B-A9B", "p": 74.04, "a": 9, "cur": true, "pg": true, "ar": {"repo": "nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB", "cfg": "https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB/blob/main/config.json", "qh": 32, "max": 262144, "full": [8, 2, 128], "other": {"mamba": 40}, "note": "40 mamba layers keep a small fixed-size state, not a per-token KV cache, so they are not counted"}, "nf": null, "ng": null}, {"n": "Kolibri-1", "p": 78.1, "a": 3.46, "cur": true, "pg": true, "ar": {"repo": "Aleph-Alpha/Kolibri-1", "cfg": "https://huggingface.co/Aleph-Alpha/Kolibri-1/blob/main/config.json", "qh": 48, "max": 262144, "full": [10, 4, 128], "swa": [40, 4, 128, 513]}, "nf": "FP8", "ng": null}, {"n": "gpt-oss 120B", "p": 117, "a": 5.1, "cur": true, "pg": true, "ar": {"repo": "openai/gpt-oss-120b", "cfg": "https://huggingface.co/openai/gpt-oss-120b/blob/main/config.json", "qh": 64, "max": 131072, "full": [18, 8, 64], "swa": [18, 8, 64, 128]}, "nf": "MXFP4", "ng": 65}, {"n": "Nemotron 3 Super 120B-A12B", "p": 120, "a": 12, "cur": true, "pg": true, "ar": {"repo": "nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16", "cfg": "https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16/blob/main/config.json", "qh": 32, "max": 262144, "full": [8, 2, 128], "other": {"mamba": 40}, "note": "40 mamba layers keep a small fixed-size state, not a per-token KV cache, so they are not counted"}, "nf": null, "ng": null}, {"n": "Nemotron 3.5 Super VL 120B-A12B", "p": 120, "a": 12, "cur": true, "pg": true, "ar": {"repo": "nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16", "cfg": "https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16/blob/main/config.json", "qh": 32, "max": 1048576, "full": [8, 2, 128], "other": {"mamba": 40}, "note": "40 mamba layers keep a small fixed-size state, not a per-token KV cache, so they are not counted"}, "nf": null, "ng": null}, {"n": "Qwen3.5 122B", "p": 122, "a": null, "cur": true, "pg": true, "ar": {"repo": "Qwen/Qwen3.5-122B-A10B", "cfg": "https://huggingface.co/Qwen/Qwen3.5-122B-A10B/blob/main/config.json", "qh": 32, "max": 262144, "full": [12, 2, 256], "other": {"linear_attention": 36}, "note": "36 linear attention layers keep a small fixed-size state, not a per-token KV cache, so they are not counted"}, "nf": null, "ng": null}, {"n": "Ornith 1.5 122B", "p": 125.1, "a": 10, "cur": true, "pg": true, "ar": {"repo": "ornith-ai/Ornith-1.5-122B", "cfg": "https://huggingface.co/ornith-ai/Ornith-1.5-122B/blob/main/config.json", "qh": 32, "max": 262144, "full": [12, 2, 256], "other": {"linear_attention": 36}, "note": "36 linear attention layers keep a small fixed-size state, not a per-token KV cache, so they are not counted"}, "nf": null, "ng": null}, {"n": "Mistral Medium 3.5 128B", "p": 128, "a": null, "cur": true, "pg": true, "ar": {"repo": "mistralai/Mistral-Medium-3.5-128B", "cfg": "https://huggingface.co/mistralai/Mistral-Medium-3.5-128B/blob/main/config.json", "qh": 96, "max": 262144, "full": [88, 8, 128]}, "nf": "FP8", "ng": null}, {"n": "Qwen3.8 Flash-Next", "p": 180, "a": 6, "cur": true, "pg": true, "ar": {"repo": "Qwen/Qwen3.8-Flash-Next", "cfg": "https://huggingface.co/Qwen/Qwen3.8-Flash-Next/blob/main/config.json", "qh": 24, "max": 262144, "full": [12, 2, 256], "other": {"linear_attention": 36}, "note": "36 linear attention layers keep a small fixed-size state, not a per-token KV cache, so they are not counted"}, "nf": null, "ng": null}, {"n": "MiMo V2.6 Flash", "p": 309, "a": 15, "cur": true, "pg": true, "ar": {"repo": "XiaomiMiMo/MiMo-V2.6-Flash-RL", "cfg": "https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL/blob/main/config.json", "qh": 64, "max": 1048576, "full": [9, 4, 192], "swa": [39, 8, 192, 128], "note": "hybrid_layer_pattern: 9 full-attention layers (num_key_value_heads=4, head_dim=192) + 39 sliding-window layers (swa_num_key_value_heads=8, swa_head_dim=192, window=128). v_head_dim is 128 on full layers and swa_v_head_dim is 128; engines that store V at v_head_dim need a slightly smaller cache than the 2×head_dim figure shown here."}, "nf": null, "ng": null}, {"n": "GLM-5.3 Flash", "p": 320, "a": 18, "cur": true, "pg": true, "ar": {"repo": "zai-org/GLM-5.3-Flash", "cfg": "https://huggingface.co/zai-org/GLM-5.3-Flash/blob/main/config.json", "qh": 64, "max": 1048576, "manual": true, "note": "34 linear-attention + 11 DeepSeek-style sparse MLA layers (kv_lora_rank 512) — the standard GQA formula does not apply; enter values by hand or use the engine log"}, "nf": null, "ng": null}, {"n": "MiniMax M3.1 Flash", "p": 428, "a": 23, "cur": true, "pg": true, "ar": {"repo": "MiniMaxAI/MiniMax-M3.1-Flash", "cfg": "https://huggingface.co/MiniMaxAI/MiniMax-M3.1-Flash/blob/main/config.json", "qh": 64, "max": 1048576, "full": [60, 4, 128]}, "nf": null, "ng": null}, {"n": "Step 5 Preview", "p": 600, "a": 27, "cur": true, "pg": true, "ar": {"repo": "stepfun-ai/Step-5-Preview", "cfg": "https://huggingface.co/stepfun-ai/Step-5-Preview/blob/main/config.json", "qh": 64, "max": 1048576, "full": [23, 4, 192], "swa": [69, 4, 192, 512], "note": "92 decoder layers (the 3 extra layer_types entries are MTP layers); KV heads = num_attention_groups"}, "nf": null, "ng": null}, {"n": "DeepSeek R1 671B", "p": 671, "a": 37, "cur": true, "pg": true, "ar": {"repo": "deepseek-ai/DeepSeek-R1", "cfg": "https://huggingface.co/deepseek-ai/DeepSeek-R1/blob/main/config.json", "qh": 128, "max": 163840, "mla": [61, 576], "note": "MLA attention: engines with MLA support cache one compressed latent per token per layer (kv_lora_rank 512 + qk_rope_head_dim 64 = 576 values), not separate K and V heads"}, "nf": null, "ng": null}, {"n": "DeepSeek V4.1 Flash", "p": 748.6, "a": 16, "cur": true, "pg": true, "ar": {"repo": "deepseek-ai/DeepSeek-V4.1-Flash", "cfg": "https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/main/config.json", "qh": 64, "max": 1048576, "manual": true, "note": "compressed/sparse attention (compress_ratio, index heads) — the standard GQA formula does not apply; enter values by hand or use the engine log"}, "nf": null, "ng": null}, {"n": "GLM-5.3", "p": 753, "a": null, "cur": true, "pg": true, "ar": {"repo": "zai-org/GLM-5.3", "cfg": "https://huggingface.co/zai-org/GLM-5.3/blob/main/config.json", "qh": 64, "max": 1048576, "manual": true, "note": "GLM MoE DSA: 78 layers with MLA (kv_lora_rank 512, qk_rope_head_dim 64) plus sparse index heads — the standard GQA formula does not apply; enter values by hand or use the engine log"}, "nf": null, "ng": null}, {"n": "GLM-5.2", "p": 753.3, "a": null, "cur": true, "pg": true, "ar": {"repo": "zai-org/GLM-5.2", "cfg": "https://huggingface.co/zai-org/GLM-5.2/blob/main/config.json", "qh": 64, "max": 1048576, "manual": true, "note": "GLM MoE DSA: 78 layers with MLA (kv_lora_rank 512, qk_rope_head_dim 64) plus sparse index heads — the standard GQA formula does not apply; enter values by hand or use the engine log"}, "nf": null, "ng": null}, {"n": "Muse Spark Open 816B-A42B", "p": 816, "a": 42, "cur": true, "pg": true, "ar": {"repo": "meta-models/Muse-Spark-Open-816B-A42B", "cfg": "https://huggingface.co/mlx-community/Muse-Spark-Open-816B-A42B-4bit/blob/main/config.json", "qh": 128, "max": 1048576, "note_src": "config read from the public mirror mlx-community/Muse-Spark-Open-816B-A42B-4bit (the official repo is gated)", "full": [16, 16, 64], "swa": [46, 16, 64, 2048]}, "nf": null, "ng": null}, {"n": "Llama 3.1 8B", "p": 8, "a": null, "cur": false, "pg": true, "ar": {"repo": "meta-llama/Llama-3.1-8B-Instruct", "cfg": "https://huggingface.co/unsloth/Llama-3.1-8B-Instruct/blob/main/config.json", "qh": 32, "max": 131072, "note_src": "config read from the public mirror unsloth/Llama-3.1-8B-Instruct (the official repo is gated)", "full": [32, 8, 128]}, "nf": null, "ng": null}, {"n": "Devstral Small 2 24B", "p": 24, "a": null, "cur": false, "pg": false, "ar": {"repo": "mistralai/Devstral-Small-2-24B-Instruct-2512", "cfg": "https://huggingface.co/mistralai/Devstral-Small-2-24B-Instruct-2512/blob/main/config.json", "qh": 32, "max": 393216, "full": [40, 8, 128]}, "nf": null, "ng": null}, {"n": "Qwen3 32B", "p": 32.76, "a": null, "cur": false, "pg": false, "ar": {"repo": "Qwen/Qwen3-32B", "cfg": "https://huggingface.co/Qwen/Qwen3-32B/blob/main/config.json", "qh": 64, "max": 40960, "full": [64, 8, 128]}, "nf": null, "ng": null}, {"n": "Llama 3.3 70B", "p": 70, "a": null, "cur": false, "pg": true, "ar": {"repo": "meta-llama/Llama-3.3-70B-Instruct", "cfg": "https://huggingface.co/unsloth/Llama-3.3-70B-Instruct/blob/main/config.json", "qh": 64, "max": 131072, "note_src": "config read from the public mirror unsloth/Llama-3.3-70B-Instruct (the official repo is gated)", "full": [80, 8, 128]}, "nf": null, "ng": null}, {"n": "Ling 3.0 Flash", "p": 124, "a": 5.1, "cur": false, "pg": true, "ar": {"repo": "inclusionAI/Ling-3.0-flash", "cfg": "https://huggingface.co/inclusionAI/Ling-3.0-flash/blob/main/config.json", "qh": 32, "max": 262144, "mla": [42, 576], "note": "MLA attention (bailing hybrid): engines with MLA support cache one compressed latent per token per layer (kv_lora_rank 512 + qk_rope_head_dim 64 = 576 values). Config also sets max_window_layers=20 and layer_group_size=6; some layers may use a shorter window — treat the MLA figure as an upper bound and check the engine log."}, "nf": null, "ng": null}, {"n": "MiniMax M3", "p": 428, "a": 23, "cur": false, "pg": true, "ar": {"repo": "MiniMaxAI/MiniMax-M3", "cfg": "https://huggingface.co/MiniMaxAI/MiniMax-M3/blob/main/config.json", "qh": 64, "max": 1048576, "full": [60, 4, 128]}, "nf": null, "ng": null}], "formats": [{"id": "Q8_0", "n": "Q8_0 (GGUF)", "b": 8.5008, "e": ["lc"]}, {"id": "Q6_K", "n": "Q6_K (GGUF)", "b": 6.5633, "e": ["lc"]}, {"id": "Q5_K_M", "n": "Q5_K_M (GGUF)", "b": 5.7036, "e": ["lc"]}, {"id": "Q5_K_S", "n": "Q5_K_S (GGUF)", "b": 5.5704, "e": ["lc"]}, {"id": "Q4_K_M", "n": "Q4_K_M (GGUF)", "b": 4.8944, "e": ["lc"]}, {"id": "Q4_K_S", "n": "Q4_K_S (GGUF)", "b": 4.6672, "e": ["lc"]}, {"id": "IQ4_XS", "n": "IQ4_XS (GGUF)", "b": 4.4597, "e": ["lc"]}, {"id": "Q3_K_M", "n": "Q3_K_M (GGUF)", "b": 3.996, "e": ["lc"]}, {"id": "IQ3_M", "n": "IQ3_M (GGUF)", "b": 3.7628, "e": ["lc"]}, {"id": "Q3_K_S", "n": "Q3_K_S (GGUF)", "b": 3.6429, "e": ["lc"]}, {"id": "Q2_K", "n": "Q2_K (GGUF)", "b": 3.1593, "e": ["lc"]}, {"id": "F16", "n": "F16 (GGUF)", "b": 16.0005, "e": ["lc"]}, {"id": "MXFP4", "n": "MXFP4 (gpt-oss native)", "b": 4.25, "e": ["lc", "vl"], "na": ["blackwell"], "nn": "MXFP4 (4-bit values + one shared power-of-two scale per 32) is native on Blackwell FP4 Tensor Cores. Other GPUs dequantize it on the fly; llama.cpp runs MXFP4 gpt-oss GGUFs on CUDA, ROCm, Vulkan and Metal (see the measured builds)."}, {"id": "BF16", "n": "BF16 / FP16 (unquantized)", "b": 16, "e": ["vl"]}, {"id": "FP8", "n": "FP8 (W8A8)", "b": 8, "e": ["vl"], "na": ["ada", "hopper", "blackwell"], "nn": "FP8 math needs compute capability ≥ 8.9 (Ada, Hopper, Blackwell). On Turing/Ampere (e.g. RTX 3090) vLLM still runs FP8 checkpoints as weight-only W8A16 through FP8 Marlin kernels: you get the memory saving, not the FP8 speed-up (vLLM docs)."}, {"id": "AWQ4", "n": "AWQ 4-bit (g128, ≈)", "b": 4.25, "e": ["vl"], "na": ["ampere", "ada", "hopper", "blackwell"], "nn": "vLLM's hardware table lists AWQ for NVIDIA Turing and newer; not Volta and not AMD GPUs."}, {"id": "GPTQ4", "n": "GPTQ 4-bit (g128, ≈)", "b": 4.25, "e": ["vl"], "na": ["ampere", "ada", "hopper", "blackwell", "volta"], "nn": "vLLM's hardware table lists GPTQ for NVIDIA Volta and newer; not AMD GPUs."}, {"id": "GPTQ8", "n": "GPTQ 8-bit (g128, ≈)", "b": 8.25, "e": ["vl"], "na": ["ampere", "ada", "hopper", "blackwell", "volta"], "nn": "vLLM's hardware table lists GPTQ for NVIDIA Volta and newer; not AMD GPUs."}, {"id": "NVFP4", "n": "NVFP4 (Blackwell)", "b": 4.5, "e": ["vl"], "na": ["blackwell"], "nn": "NVFP4 is native only on Blackwell 5th-gen Tensor Cores (RTX 50xx, RTX PRO Blackwell, GB10/DGX Spark, B200). On other NVIDIA GPUs vLLM falls back to weight-only W4A16 via Marlin: it loads and saves memory, but without FP4 math (vLLM ModelOpt docs)."}, {"id": "EXL3", "n": "EXL3 (ExLlamaV3)", "b": null, "e": ["ex"]}, {"id": "EXL2", "n": "EXL2 (ExLlamaV2)", "b": null, "e": ["ex"]}, {"id": "MLX4", "n": "MLX 4-bit", "b": 4.5, "e": ["mx"]}, {"id": "MLX6", "n": "MLX 6-bit", "b": 6.5, "e": ["mx"]}, {"id": "MLX8", "n": "MLX 8-bit", "b": 8.5, "e": ["mx"]}, {"id": "MLXBF16", "n": "MLX BF16", "b": 16, "e": ["mx"]}], "engines": {"lc": {"n": "llama.cpp / Ollama / LM Studio (GGUF)", "df": "Q4_K_M", "kv": [["F16", 2], ["q8_0", 1.0625], ["q4_0", 0.5625]], "par": "llama.cpp splits the model <b>by layers<\/b> across GPUs by default (<code>--split-mode layer<\/code>, a form of pipeline parallelism). Any GPU count works, cards can have different sizes (<code>--tensor-split<\/code>), and only small activations cross between cards, so slow PCIe links hurt less than with tensor parallel. In a single chat stream the GPUs take turns, so extra cards add <i>capacity<\/i> more than speed. <code>--split-mode row<\/code> splits each tensor across GPUs instead."}, "vl": {"n": "vLLM / SGLang (safetensors)", "df": "AWQ4", "kv": [["auto (BF16/FP16)", 2], ["fp8", 1]], "hw": ["ampere", "ada", "hopper", "blackwell", "volta", "rdna3", "rdna4", "intel"], "hwn": "vLLM on Apple Silicon runs on the CPU only; use MLX or llama.cpp to use the Mac GPU.", "par": "vLLM uses <b>tensor parallelism<\/b> (<code>--tensor-parallel-size<\/code>): every layer is split across all GPUs and they work on each token together, so 2–8 GPUs add speed as well as capacity, but they exchange data on every layer, so PCIe/NVLink bandwidth matters. The attention-head count must divide by the TP size, which is why 2, 4 and 8 GPUs are the usual choice. When the count does not divide evenly, or GPUs lack a fast interconnect, vLLM's docs recommend <b>pipeline parallelism<\/b> (<code>--pipeline-parallel-size<\/code>), which splits by layers and allows uneven splits."}, "ex": {"n": "ExLlamaV2 / V3 (TabbyAPI)", "df": "EXL3", "kv": [["FP16", 2], ["Q8 (≈)", 1], ["Q6 (≈)", 0.75], ["Q4 (≈)", 0.5]], "hw": ["ampere", "ada", "hopper", "blackwell", "volta"], "hwn": "ExLlamaV3 targets NVIDIA CUDA GPUs.", "par": "ExLlama splits the model by layers across GPUs (automatic or a manual <code>gpu_split<\/code>); ExLlamaV2 also has a tensor-parallel mode. EXL2/EXL3 let you pick any average bits-per-weight, so you can size a quant to fill your exact VRAM."}, "mx": {"n": "MLX (Apple Silicon)", "df": "MLX4", "kv": [["FP16", 2], ["8-bit (--kv-bits 8)", 1.0625], ["4-bit (--kv-bits 4)", 0.5625]], "hw": ["apple"], "hwn": "MLX runs only on Apple Silicon.", "par": "MLX runs on one Mac; for several Macs see the cluster note below."}}, "hw": [{"g": "NVIDIA Ampere", "k": "3060", "n": "RTX 3060 12GB", "v": 12, "a": "ampere", "t": "gpu", "c": [1, 2, 3, 4, 5, 6, 7, 8]}, {"g": "NVIDIA Ampere", "k": "3090", "n": "RTX 3090 / 3090 Ti", "v": 24, "a": "ampere", "t": "gpu", "c": [1, 2, 3, 4, 5, 6, 7, 8]}, {"g": "NVIDIA Ampere", "k": "a4000", "n": "RTX A4000", "v": 16, "a": "ampere", "t": "gpu", "c": [1, 2, 3, 4, 5, 6, 7, 8]}, {"g": "NVIDIA Ampere", "k": "a6000", "n": "RTX A6000", "v": 48, "a": "ampere", "t": "gpu", "c": [1, 2, 3, 4, 5, 6, 7, 8]}, {"g": "NVIDIA Ampere", "k": "a100", "n": "A100 80GB", "v": 80, "a": "ampere", "t": "gpu", "c": [1, 2, 3, 4, 5, 6, 7, 8]}, {"g": "NVIDIA Ada", "k": "4060ti", "n": "RTX 4060 Ti 16GB", "v": 16, "a": "ada", "t": "gpu", "c": [1, 2, 3, 4, 5, 6, 7, 8]}, {"g": "NVIDIA Ada", "k": "4090", "n": "RTX 4090", "v": 24, "a": "ada", "t": "gpu", "c": [1, 2, 3, 4, 5, 6, 7, 8]}, {"g": "NVIDIA Ada", "k": "6000ada", "n": "RTX 6000 Ada", "v": 48, "a": "ada", "t": "gpu", "c": [1, 2, 3, 4, 5, 6, 7, 8]}, {"g": "NVIDIA Hopper", "k": "h100", "n": "H100 80GB", "v": 80, "a": "hopper", "t": "gpu", "c": [1, 2, 3, 4, 5, 6, 7, 8]}, {"g": "NVIDIA Blackwell", "k": "5060ti", "n": "RTX 5060 Ti 16GB", "v": 16, "a": "blackwell", "t": "gpu", "c": [1, 2, 3, 4, 5, 6, 7, 8]}, {"g": "NVIDIA Blackwell", "k": "5070ti", "n": "RTX 5070 Ti", "v": 16, "a": "blackwell", "t": "gpu", "c": [1, 2, 3, 4, 5, 6, 7, 8]}, {"g": "NVIDIA Blackwell", "k": "5080", "n": "RTX 5080", "v": 16, "a": "blackwell", "t": "gpu", "c": [1, 2, 3, 4, 5, 6, 7, 8]}, {"g": "NVIDIA Blackwell", "k": "5090", "n": "RTX 5090", "v": 32, "a": "blackwell", "t": "gpu", "c": [1, 2, 3, 4, 5, 6, 7, 8]}, {"g": "NVIDIA Blackwell", "k": "pro4000", "n": "RTX PRO 4000 Blackwell", "v": 24, "a": "blackwell", "t": "gpu", "c": [1, 2, 3, 4, 5, 6, 7, 8]}, {"g": "NVIDIA Blackwell", "k": "pro6000", "n": "RTX PRO 6000 Blackwell", "v": 96, "a": "blackwell", "t": "gpu", "c": [1, 2, 3, 4, 5, 6, 7, 8]}, {"g": "NVIDIA (older)", "k": "v100", "n": "Tesla V100 32GB", "v": 32, "a": "volta", "t": "gpu", "c": [1, 2, 3, 4, 5, 6, 7, 8]}, {"g": "AMD Radeon", "k": "7900xtx", "n": "Radeon RX 7900 XTX", "v": 24, "a": "rdna3", "t": "gpu", "c": [1, 2, 3, 4, 5, 6, 7, 8]}, {"g": "AMD Radeon", "k": "r9700", "n": "Radeon AI Pro R9700", "v": 32, "a": "rdna4", "t": "gpu", "c": [1, 2, 3, 4, 5, 6, 7, 8]}, {"g": "AMD Radeon", "k": "9070xt", "n": "Radeon RX 9070 XT", "v": 16, "a": "rdna4", "t": "gpu", "c": [1, 2, 3, 4, 5, 6, 7, 8]}, {"g": "Intel", "k": "b70", "n": "Intel Arc Pro B70", "v": 32, "a": "intel", "t": "gpu", "c": [1, 2, 3, 4, 5, 6, 7, 8]}, {"g": "Unified memory / multi-node", "k": "spark", "n": "DGX Spark / GB10", "v": 128, "a": "blackwell", "t": "uni", "c": [1, 2, 3, 4], "mem": [128, 64], "dm": 128, "os": [8, 0], "pl": "Up to four DGX Spark systems can be linked over their ConnectX-7 200 Gb/s ports (NVIDIA). vLLM runs across them with tensor or pipeline parallel over the network (Ray), llama.cpp with its RPC backend. Network latency makes cross-node TP slower than inside one box, so measured numbers below matter more than the estimate."}, {"g": "Unified memory / multi-node", "k": "strix", "n": "Strix Halo (Ryzen AI Max+ 395)", "v": 128, "a": "rdna35", "t": "uni", "c": [1, 2], "mem": [128, 96, 64], "dm": 128, "os": [8, 0], "pl": "Two Strix Halo boxes can share a model with llama.cpp's RPC backend over the network; expect it to be slower than one box per token. On Windows, AMD Variable Graphics Memory assigns a fixed GPU share; on Linux the GPU can use most of the pool."}, {"g": "Unified memory / multi-node", "k": "mac", "n": "Apple Silicon Mac (Studio / MacBook)", "v": 128, "a": "apple", "t": "uni", "c": [1, 2, 4], "mem": [512, 256, 192, 128, 96, 64, 48, 36, 32], "dm": 128, "os": [8, 0.25], "pl": "Several Macs can pool memory over Thunderbolt (exo, MLX distributed, llama.cpp RPC); see the measured Mac Studio cluster builds. macOS by default lets the GPU use only part of unified memory; the reserve field starts at 25% (our assumption), and the limit can be raised with <code>sudo sysctl iogpu.wired_limit_mb<\/code>."}], "meas": {"3090": {"2": [["dual-3090-ryzen-3700x", "Dual RTX 3090 on Ryzen 3700X (X470)", [["Llama 3 70B", "Q4_K_M", "Ollama", 16.95], ["Llama 3 70B", "AWQ 4-bit", "vLLM (TP=2)", 21.2]]], ["dual-3090-2000-eur-exllamav2", "Dual used RTX 3090, ~2000 EUR total", [["Llama 3.3 70B", "", "ExLlamaV2 (TP)", 20]]], ["llmspeed-2x-rtx-3090-llama33-70b-oct-2026", "2x RTX 3090 (llm-speed signed run, Oct 2026): Llama 3.3 70B Q4_K_M at 19.1 tok/s", [["Llama 3.3 70B", "Q4_K_M", "llama.cpp 0.7.0-dev", 19.1]]]], "4": [["4x3090-tr32-192gb-qwen3-235b", "4x RTX 3090 + 32-core Threadripper, 192 GB DDR4", [["Qwen3 235B-A22B", "UD-Q3_K_XL", "llama.cpp", "12-14"]]], ["4x3090-epyc-7443-256gb", "4x RTX 3090 + EPYC 7443, 256 GB DDR4 8-channel", [["Qwen3 235B-A22B", "IQ4_XS", "llama.cpp", 18]]], ["epyc-7f52-6gpu-136gb", "EPYC 7F52 jank rig: 4x3090 + 4090 + 4080 (136 GB VRAM)", [["Llama 3.1 70B", "Q8_0", "Oobabooga", 9]]], ["4x3090-qwen36-vllm-tp2", "4x RTX 3090 lab: Qwen3.6 on vLLM vs llama.cpp", [["Qwen3.6 35B-A3B", "IQ4_XS (1 GPU)", "llama.cpp", "115-133"], ["Qwen3.6 35B-A3B", "AWQ-INT4 (TP=2)", "vLLM", 149], ["Qwen3.6 35B-A3B", "AWQ-INT4 + MTP-3 (TP=2)", "vLLM", "225-231"]]]], "1": [["epyc-plus-3090-deepseek-q2", "EPYC + single RTX 3090 (hybrid)", [["DeepSeek R1 671B", "Q2_K", "llama.cpp CUDA", 8.97]]], ["single-rtx-3090-gpt-oss", "Single RTX 3090 24GB", [["gpt-oss 20B", "MXFP4", "llama.cpp", 161.77]]], ["llamabench-rtx-3090", "RTX 3090: 2026 models (llamabench community)", [["Qwen3.6 35B-A3B", "llama-bench tg128 (quant per result)", "llama.cpp", "146-153"], ["Qwen3.6 27B", "llama-bench tg128 (quant per result)", "llama.cpp", 40.2], ["Qwen3.8 27B", "llama-bench tg128 (quant per result)", "llama.cpp", "38.2-71.6"]]], ["rtx-3090-qwen36-35b-a3b-giles", "Single RTX 3090: Qwen3.6 35B-A3B (Giles Thomas)", [["Qwen3.6 35B-A3B", "Q4 GGUF, all layers on GPU, ~90K ctx", "llama.cpp CUDA", 139.6]]], ["rtx-3090-qwen38-27b-openjet-megakernel", "Single RTX 3090: Qwen3.8 27B at 78-140 tok/s with the OpenJet megakernel vs llama.cpp", [["Qwen3.8 27B", "Q4_K_M (unsloth), greedy code prompt", "OpenJet megakernel (MTP, 4 drafts)", 139.7], ["Qwen3.8 27B", "Q4_K_M (unsloth), greedy code prompt", "llama.cpp (MTP, 2 drafts)", 72.5], ["Qwen3.8 27B", "Q4_K_M (unsloth), reasoning mode", "OpenJet megakernel (MTP)", 78.0], ["Qwen3.8 27B", "Q4_K_M (unsloth), reasoning mode", "llama.cpp (MTP, 2 drafts)", 50.9], ["Qwen3.8 27B", "Q4_K_M (unsloth), no speculation", "llama.cpp", 40.4], ["Qwen3.8 27B", "Q4_K_M (unsloth), no speculation", "OpenJet megakernel", 42.9]]], ["llmspeed-rtx-3090-oct-2026", "RTX 3090 (llm-speed signed runs, Oct 2026): gpt-oss 20B 207 tok/s, Qwen3.6 27B 43 tok/s", [["gpt-oss 20B", "reported as Q4_K_M", "llama.cpp 0.7.0-dev", 207.2], ["Qwen3.6 27B", "Q4_K_M", "llama.cpp 0.7.0-dev", 42.7], ["GLM 4.7 Flash", "Q4_K_M (Threadripper PRO host)", "Ollama 0.31.1", 134.6]]], ["llmspeed-rtx-3090-flint-40b-a2-5b-oct-2026", "RTX 3090 (llm-speed signed run, Oct 2026): Flint 40B-A2.5B Q3_K_XL at 128.9 tok/s", [["Flint 40B-A2.5B", "UD-Q3_K_XL", "llama.cpp 0.7.0-dev (build 11803)", 128.9]]], ["llmspeed-rtx-3090-nemotron-35-bolt", "RTX 3090 running Nemotron 3.5 Bolt 74B-A9B in 24 GB (IQ2)", [["Nemotron 3.5 Bolt 74B-A9B", "NVIDIA mixed 2-bit IQ2 GGUF (22.0 GB)", "llama.cpp build 11669 (620d2225)", 87.75]]], ["lla-rtx-3090-gemma-4-31b", "RTX 3090 running Gemma 4 31B Q4_K_M (llama-bench)", [["Gemma 4 31B", "Q4_K_M", "llama.cpp b11658 (llama-bench)", 38.8]]], ["rtx-3090-250w-windows-qwen38-27b-mtp-club3090", "RTX 3090 at 250 W on Windows 11: Qwen3.8 27B Q4_K_M at 56.4 tok/s with MTP (llama.cpp)", [["Qwen3.8 27B", "Unsloth UD-Q4_K_M (nextn/MTP head embedded), f16 KV", "llama.cpp b11382 (Windows, CUDA 13), MTP n=2", 56.37], ["Qwen3.8 27B", "Unsloth UD-Q4_K_M, f16 KV", "llama.cpp b11382 (Windows, CUDA 13), MTP off, -np 4", 38.3]]]], "8": [["8x-rtx-3090-deepseek-v4-flash-limechain", "8x RTX 3090 serving DeepSeek V4 Flash (LimeChain vLLM kit)", [["DeepSeek V4 Flash", "see source (FP8 KV cache)", "vLLM LimeChain SM86 overlay, 1 concurrent request", 95.52]]]], "6": [["6x-rtx-3090-7950x-qwen38-flash-next-llamacpp-multigpu", "6x RTX 3090 on a Ryzen 9 7950X (4 on USB4 eGPU docks): Qwen3.8 Flash-Next at 45.9 tok/s", [["Qwen3.8 Flash-Next", "UD-Q4_K_XL (~111.3 GB), q8_0 KV", "llama.cpp-multigpu fork 134489582, layer split", 45.9], ["Qwen3.8 Flash-Next", "UD-Q4_K_XL (~111.3 GB), q8_0 KV", "llama.cpp upstream df03399b8, layer split", 38.5]]]]}, "mac": {"1": [["mac-studio-64gb-llama33", "Mac Studio 64 GB", [["Llama 3.3 70B", "Q4", "", 11]]], ["m2-ultra-76c-192gb", "Mac Studio M2 Ultra 76-core GPU, 192 GB", [["Llama 2 7B", "Q4_0", "llama.cpp Metal (FA, Aug 2026 build)", 125.21], ["gpt-oss 20B", "MXFP4", "llama.cpp", 116.08], ["gpt-oss 120B", "MXFP4", "llama.cpp", 79.68]]], ["macbook-pro-m2-max", "MacBook Pro M2 Max", [["Llama 3.1 8B", "Q8", "llama.cpp-based", 28.92]]], ["mac-studio-m1-ultra", "Mac Studio M1 Ultra", [["Llama 3.1 8B", "Q8", "llama.cpp-based", 37.92]]], ["macbook-m3-max-40c-64gb", "MacBook Pro M3 Max 40-core GPU, 64 GB", [["Llama 3 8B", "Q4_K_M", "llama.cpp", 50.74]]], ["macbook-m1-max-32c-64gb", "MacBook Pro M1 Max 32-core GPU, 64 GB", [["Llama 3 8B", "Q4_K_M", "llama.cpp", 34.49]]], ["m3-ultra-512gb-deepseek", "Mac Studio M3 Ultra, 512 GB", [["DeepSeek R1 671B", "Q4", "MLX", 18.43]]], ["mac-studio-m3-ultra-512gb-geerling", "Mac Studio M3 Ultra 512GB (Geerling test)", [["DeepSeek R1 671B", "", "llama.cpp", 19.89], ["gpt-oss 20B", "", "llama.cpp", 115.29], ["gpt-oss 20B", "MXFP4", "llama.cpp (Metal, llama-bench)", 115.52], ["Llama 3.1 70B", "", "llama.cpp", 14.08], ["Llama 3.1 405B", "", "llama.cpp", 1.96], ["DeepSeek R1 Distill 14B", "", "llama.cpp", 51.85], ["Llama 3.2 3B", "", "llama.cpp", 154.6]]], ["mac-studio-m1-ultra-64c-128gb", "Mac Studio M1 Ultra (64-core GPU) 128GB", [["gpt-oss 20B", "", "llama.cpp", 100.43], ["DeepSeek R1 Distill 14B", "", "llama.cpp", 41.51], ["Llama 3.1 70B", "", "llama.cpp", 9.84], ["Llama 3.2 3B", "", "llama.cpp", 127.6]]], ["m4-mac-mini-32gb", "M4 Mac mini (10-core) 32GB", [["Llama 3.2 3B", "", "llama.cpp", 41.31]]], ["macbook-m4-max-36gb", "Apple M4 Max 36GB", [["gpt-oss 20B", "MXFP4", "llama.cpp", 92.36]]], ["apple-m1-max-64gb-gpt-oss", "Apple M1 Max 64GB", [["gpt-oss 20B", "MXFP4", "llama.cpp", 75.15]]], ["mac-studio-m3-ultra-512gb-muse-spark-open", "Mac Studio M3 Ultra 512GB: Muse Spark Open 816B (Hugging Face test)", [["Muse Spark Open 816B-A42B", "Q2_K (imatrix), ~304 GB", "llama.cpp", 18], ["Muse Spark Open 816B-A42B", "oQ3e (3-bit), ~344 GB", "MLX (oMLX / mlx-lm)", 21]]], ["mac-studio-m5-ultra-512gb-apple-mlx", "Mac Studio M5 Ultra 512GB: Qwen3.8 27B and Mistral Medium 3.5 (Apple MLX benchmark)", [["Qwen3.8 27B", "BF16, 8,192-token context", "MLX (mlx_lm.benchmark)", 19.69], ["Mistral Medium 3.5 128B", "FP8, 8,192-token context", "MLX (mlx_lm.benchmark)", 7.7]]], ["mac-studio-m3-ultra-512gb-apple-mlx", "Mac Studio M3 Ultra 512GB: Qwen3.8 27B and Mistral Medium 3.5 (Apple MLX benchmark)", [["Qwen3.8 27B", "BF16, 8,192-token context", "MLX (mlx_lm.benchmark)", 12.46], ["Mistral Medium 3.5 128B", "FP8, 8,192-token context", "MLX (mlx_lm.benchmark)", 4.95]]], ["mac-studio-m5-max-36gb-rapid-mlx-oct-2026", "Mac Studio M5 Max 36 GB: Qwen3.6 35B-A3B at 173 tok/s (Rapid-MLX), 146 on oMLX", [["Qwen3.6 35B-A3B", "4-bit MLX", "Rapid-MLX 0.16.1 (MTP)", 173.4], ["Qwen3.6 35B-A3B", "4-bit MLX", "oMLX 0.7.0", 145.8], ["Qwen3.6 35B-A3B", "4-bit MLX", "LM Studio (MLX)", 124.5], ["Qwen3.8 27B", "4-bit MLX", "Rapid-MLX 0.16.1 (MTP)", 37.9], ["Qwen3.8 27B", "4-bit MLX", "oMLX 0.7.0", 25.3], ["Qwen3.8 27B", "4-bit MLX", "LM Studio (MLX)", 26.0], ["Qwen3.5 9B", "4-bit MLX", "Rapid-MLX 0.16.1 (MTP)", 115.5], ["Qwen3.5 9B", "4-bit MLX", "oMLX 0.7.0", 78.0]]], ["macbook-pro-m5-max-128gb-moe-servers-baobab", "MacBook Pro M5 Max 128 GB: Qwen3.6 35B-A3B at 271 tok/s (mlx-serve + MTP), Gemma 4 26B-A4B at 166", [["Qwen3.6 35B-A3B", "4-bit MLX (mlx-serve build)", "mlx-serve 26.10.2 + MTP", 271], ["Qwen3.6 35B-A3B", "4-bit MLX", "Rapid-MLX 0.16.1 + MTP", 240], ["Qwen3.6 35B-A3B", "4-bit MLX", "mlx_lm.server 0.32.2", 107], ["Gemma 4 26B-A4B", "4-bit MLX (QAT)", "Rapid-MLX 0.16.1 + DFlash", 166], ["Qwen3.8 27B", "4-bit MLX", "same server, no speculation", 31]]], ["macbook-pro-m4-pro-48gb-flint-40b-kapitsa-rec-llamacpp", "MacBook Pro M4 Pro 48 GB: Flint 40B-A2.5B 4-bit at 127.9 tok/s (Kapitsa), 126.6 (rec), 79.6 (llama.cpp)", [["Flint 40B-A2.5B", "int4 MLX (RecursiveSI/flint-int4)", "Kapitsa (PR #286)", 127.9], ["Flint 40B-A2.5B", "int4 MLX (same files)", "rec 0.1.0 (Recursive)", 126.56], ["Flint 40B-A2.5B", "Q4_1 GGUF (RecursiveSI/flint-int4-gguf)", "llama.cpp b11804, all layers on GPU, -fa on", 79.57]]], ["omlx-m3-ultra-512gb-minimax-m3", "Mac Studio M3 Ultra 512 GB running MiniMax M3 (4-bit MLX)", [["MiniMax M3", "4-bit MLX", "oMLX v0.4.4rc2", 28.9]]], ["omlx-m3-ultra-512gb-deepseek-v41-flash-mtp", "Mac Studio M3 Ultra 512 GB running DeepSeek V4.1 Flash with MTP", [["DeepSeek V4.1 Flash", "not stated", "oMLX v0.7.1.dev1, MTP enabled", 30.8]]], ["omlx-m3-ultra-256gb-minimax-m31-flash", "Mac Studio M3 Ultra (60-core GPU) 256 GB running MiniMax M3.1 Flash", [["MiniMax M3.1 Flash", "oQ3e-mtp (MTP off)", "oMLX v0.7.2", 25.3]]], ["omlx-m5-max-128gb-qwen38-27b-mtp", "Apple M5 Max (40-core GPU) 128 GB running Qwen3.8 27B with MTP drafter", [["Qwen3.8 27B", "4-bit MLX + Qwen3.8-27B-MTP-4bit drafter", "oMLX v0.6.0, MTP enabled", 45.3]]], ["macbook-pro-m5-max-128gb-qwen38-flash-next-mtp", "MacBook Pro M5 Max 128 GB: Qwen3.8 Flash-Next at 44.4 tok/s (llama.cpp), 64.5 with MTP on oMLX", [["Qwen3.8 Flash-Next", "UD-IQ4_XS", "llama.cpp b76b133, no speculative decoding", 44.4], ["Qwen3.8 Flash-Next", "UD-IQ4_XS + mtp-Qwen3.8-Flash-Next-Q8_0 head", "llama.cpp b76b133, MTP (--spec-draft-n-max 2)", 59.3], ["Qwen3.8 Flash-Next", "oQ4e MLX (4.72 bpw, Jundot/Qwen3.8-Flash-Next-oQ4e-mtp)", "oMLX 0.7.1rc1, no speculative decoding", 50], ["Qwen3.8 Flash-Next", "oQ4e MLX (4.72 bpw) with MTP head", "oMLX 0.7.1rc1, MTP + n-gram table on SSD", 64.5]]], ["mac-mini-m6-32gb-lm-studio-oct-2026", "Mac mini M6 32 GB: Qwen3.6 35B-A3B at 63 tok/s, Nemotron 3.5 Lightning 45 (LM Studio)", [["Qwen3.6 35B-A3B", "MLX 4-bit", "LM Studio (MLX)", 63.2], ["Qwen3.6 35B-A3B", "MLX affine 4-bit (group 64) + MTP head and DFlash2 drafter", "Kapitsa 0.2.4 (KFlash tree speculative decoding)", 69.3], ["Qwen3.6 35B-A3B", "Splash package (incoai/Qwen3.6-35B-A3B-Splash)", "LM Studio Splash backend", 51.5], ["Gemma 4 26B-A4B", "QAT (google/gemma-4-26b-a4b-qat)", "LM Studio", 45.5], ["Nemotron 3.5 Lightning 30B-A3B", "GGUF Q4_K_M (lmstudio-community)", "LM Studio (llama.cpp)", 45.1], ["Ornith 1.5 35B-A3B", "GGUF Q4_K_M", "LM Studio (llama.cpp)", 37.6], ["Qwen3.5 9B", "MLX", "LM Studio (MLX)", 30.3], ["Gemma 4 12B", "QAT (google/gemma-4-12b-qat)", "LM Studio", 18.2]]]], "2": [["2x-mac-studio-m5-ultra-rdma-cluster", "2x Mac Studio M5 Ultra cluster (Thunderbolt 5 RDMA): Muse Spark Open 816B", [["Muse Spark Open 816B-A42B", "NVFP4, 8,192-token context", "MLX, 2-node tensor parallel", 24.99], ["Muse Spark Open 816B-A42B", "NVFP4, 8,192-token context", "MLX, 2-node pipeline parallel", 15.74], ["Mistral Medium 3.5 128B", "FP8, 8,192-token context", "MLX, 2-node tensor parallel", 13.69], ["Qwen3.8 27B", "BF16, 8,192-token context", "MLX, 2-node tensor parallel", 33.27]]]], "4": [["4x-mac-studio-m5-ultra-rdma-cluster", "4x Mac Studio M5 Ultra cluster (Thunderbolt 5 RDMA): Muse Spark Open at 36.7 tok/s", [["Muse Spark Open 816B-A42B", "NVFP4, 8,192-token context", "MLX, 4-node tensor parallel", 36.73], ["Mistral Medium 3.5 128B", "FP8, 8,192-token context", "MLX, 4-node tensor parallel", 23.37], ["Qwen3.8 27B", "BF16, 8,192-token context", "MLX, 4-node tensor parallel", 50.89]]]]}, "4060ti": {"1": [["single-4060ti-16gb-64gb-ram", "Single RTX 4060 Ti 16GB + 64 GB RAM (partial offload)", [["Llama 3.3 70B", "Q5_K_M", "LM Studio / Kobold", 1.5]]], ["llamabench-rtx-4060-ti-16gb", "RTX 4060 Ti 16GB: 2026 models (llamabench community)", [["LFM2.5 24B-A2B", "llama-bench tg128 (quant per result)", "llama.cpp", "131-150"], ["Qwen3.8 27B", "llama-bench tg128 (quant per result)", "llama.cpp", "10.7-17.6"]]]]}, "4090": {"1": [["rtx-4090-llamacpp-cuda", "Single RTX 4090 (llama.cpp CUDA scoreboard)", [["Llama 2 7B", "Q4_0", "llama.cpp CUDA (FA)", 188.96]]], ["rtx-4090-koboldcpp-comparison", "RTX 4090 desktop (speed comparison vs Macs)", [["Llama 3.1 8B", "Q8", "llama.cpp-based", 52.99], ["Mistral Small 22B", "Q6_K", "llama.cpp-based", 26.72]]], ["intel-265k-rtx-4090", "Intel Core Ultra 265K + RTX 4090", [["gpt-oss 20B", "", "llama.cpp", 279.65], ["DeepSeek R1 Distill 14B", "", "llama.cpp", 92.49], ["Llama 3.2 3B", "", "llama.cpp", 334.67]]], ["ryzen-7900x-rtx-4090", "Ryzen 9 7900X + RTX 4090", [["Llama 3.2 3B", "", "llama.cpp", 237.05], ["Llama 3.1 70B", "", "GPU+CPU offload", 3.1]]], ["llamabench-rtx-4090", "RTX 4090: 2026 models (llamabench community)", [["Qwen3.6 27B", "llama-bench tg128 (quant per result)", "llama.cpp", 47.0], ["Gemma 4 31B", "llama-bench tg128 (quant per result)", "llama.cpp", 43.8], ["Gemma 4 26B-A4B", "llama-bench tg128 (quant per result)", "llama.cpp", 164], ["Gemma 4 12B", "llama-bench tg128 (quant per result)", "llama.cpp", 98.1], ["gpt-oss 20B", "llama-bench tg128 (quant per result)", "llama.cpp", 203], ["Qwen3.5 9B", "llama-bench tg128 (quant per result)", "llama.cpp", 137], ["LFM2.5 24B-A2B", "llama-bench tg128 (quant per result)", "llama.cpp", "261-339"], ["Nemotron 3.5 Bolt 74B-A9B", "llama-bench tg128 (quant per result)", "llama.cpp", 108]]], ["llmspeed-rtx-4090-oct-2026", "RTX 4090 (llm-speed signed runs, Oct 2026): Qwen3.6 35B-A3B 161 tok/s, Gemma 4 12B QAT 98 tok/s", [["Qwen3.6 35B-A3B", "Q4_K_M", "llama.cpp b8931", 160.6], ["Gemma 4 12B", "QAT Q4_0", "llama.cpp b8931", 97.7], ["GLM 4.7 Flash", "Q4_K_M (EPYC 75F3 host)", "Ollama 0.31.1", 164.9]]], ["llmspeed-rtx-4090-flint-40b-a2-5b-oct-2026", "RTX 4090 (llm-speed signed run, Oct 2026): Flint 40B-A2.5B at 169-179 tok/s on llama.cpp", [["Flint 40B-A2.5B", "NVFP4", "llama.cpp 0.7.0-dev (build 11803)", 179.4], ["Flint 40B-A2.5B", "UD-Q3_K_XL", "llama.cpp 0.7.0-dev (build 11803)", 169.2]]], ["llmspeed-rtx-4090-bolt-muse-glimmer", "RTX 4090: Nemotron 3.5 Bolt (IQ2) and Muse Glimmer 30B", [["Nemotron 3.5 Bolt 74B-A9B", "NVIDIA mixed 2-bit IQ2 GGUF (22.0 GB)", "llama.cpp build 11669", 107.3], ["Muse Glimmer 30B", "UD-Q4_K_XL", "llama.cpp a37cc4610", 53.7]]]]}, "r9700": {"1": [["intel-265k-radeon-ai-pro-r9700", "Intel Core Ultra 265K + Radeon AI Pro R9700", [["gpt-oss 20B", "", "llama.cpp", 163.26], ["DeepSeek R1 Distill 14B", "", "llama.cpp", 53.72], ["Llama 3.2 3B", "", "llama.cpp", 195.91]]], ["r9700-qwen36-35b-a3b-amdai", "Radeon AI Pro R9700 (one card): Qwen3.6 35B-A3B from 4K to 256K context (AMDai lab rig)", [["Qwen3.6 35B-A3B", "Unsloth UD-Q5_K_XL, short context, f16 KV", "llama.cpp Vulkan", 102.39], ["Qwen3.6 35B-A3B", "UD-Q5_K_XL, 32K context", "llama.cpp Vulkan", 90.52], ["Qwen3.6 35B-A3B", "UD-Q5_K_XL, 128K context", "llama.cpp Vulkan", 66.43], ["Qwen3.6 35B-A3B", "UD-Q5_K_XL, 256K context", "llama.cpp Vulkan", 44.4], ["Qwen3.6 35B-A3B", "Q5_K_M (24.68 GB VRAM)", "llama.cpp Vulkan", 109.73], ["Qwen3.6 35B-A3B", "Q4_K_M", "llama.cpp Vulkan", 117.69]]], ["r9700-qwen36-27b-mtp-backends", "Radeon AI Pro R9700: Qwen3.6 27B on Vulkan, HIP and Ollama, with and without MTP", [["Qwen3.6 27B", "Q4_K_M + MTP draft head (draft-n-max 3), 64K context", "llama.cpp Vulkan", 66.0], ["Qwen3.6 27B", "Q4_K_M, no MTP", "llama.cpp Vulkan", 31.9], ["Qwen3.6 27B", "Q4_K_M", "llama.cpp HIP (ROCm)", 27.1], ["Qwen3.6 27B", "Q4_K_M", "Ollama (HIP)", 26.7]]], ["r9700-qwen38-27b-vaditaslim", "Radeon AI Pro R9700: Qwen3.8 27B on llama.cpp, Ollama and vLLM (Vadi Taslim)", [["Qwen3.8 27B", "Q4_K_M + MTP speculative decoding, reasoning on", "llama.cpp Vulkan", 57], ["Qwen3.8 27B", "Q4_K_M", "Ollama", 30], ["Qwen3.8 27B", "AWQ INT4, single request", "vLLM (radiance)", 22]]]]}, "3060": {"1": [["intel-265k-rtx-3060", "Intel Core Ultra 265K + RTX 3060", [["gpt-oss 20B", "", "llama.cpp", 83.96], ["DeepSeek R1 Distill 14B", "", "llama.cpp", 29.77], ["Llama 3.2 3B", "", "llama.cpp", 122.85]]], ["llamabench-rtx-3060", "RTX 3060 12GB: 2026 models (llamabench community)", [["Gemma 4 12B", "llama-bench tg128 (quant per result)", "llama.cpp", 38.5], ["LFM2.5 24B-A2B", "llama-bench tg128 (quant per result)", "llama.cpp", 125], ["Qwen3.8 27B", "llama-bench tg128 (quant per result)", "llama.cpp", "5.2-6.7"]]]]}, "a4000": {"1": [["intel-265k-rtx-a4000", "Intel Core Ultra 265K + RTX A4000", [["gpt-oss 20B", "", "llama.cpp", 93.55], ["Llama 3.2 3B", "", "llama.cpp", 149.44]]], ["llmspeed-rtx-a4000-muse-glimmer-30b-oct-2026", "RTX A4000 16GB (llm-speed signed run, Oct 2026): Muse Glimmer 30B UD-IQ3_XXS at 21.4 tok/s", [["Muse Glimmer 30B", "UD-IQ3_XXS", "llama.cpp 0.7.0-dev (build 11803)", 21.42]]]]}, "spark": {"1": [["dell-pro-max-gb10-dgx-spark", "Dell Pro Max with GB10 (Nvidia DGX Spark)", [["gpt-oss 20B", "", "llama.cpp", 87.97], ["DeepSeek R1 Distill 14B", "", "llama.cpp", 23.92], ["Llama 3.1 70B", "", "llama.cpp", 4.71], ["Llama 3.2 3B", "", "llama.cpp", 97.93]]], ["dgx-spark-nemotron-35-super-vl-bolt", "DGX Spark (GB10, 128GB): Nemotron 3.5 Super VL 120B and Bolt 74B, launch-week runs", [["Nemotron 3.5 Super VL 120B-A12B", "NVFP4, no speculative decoding, 256-token output", "vLLM", 19.06], ["Nemotron 3.5 Super VL 120B-A12B", "NVFP4 + LiLiCorr speculative decoding (7 draft tokens)", "vLLM", 38.07], ["Nemotron 3.5 Bolt 74B-A9B", "2-bit 24GB GGUF, no speculative decoding", "llama.cpp", 39.47], ["Nemotron 3.5 Bolt 74B-A9B", "2-bit 24GB GGUF + DFlash speculative decoding (7 draft tokens)", "llama.cpp", 61.61]]], ["dgx-spark-vllm-23-models-ctala-oct-2026", "DGX Spark (GB10, 128 GB): 23 models on vLLM at 128K context, Qwen3.6 35B-A3B up to 102 tok/s", [["Qwen3.6 35B-A3B", "NVFP4 + MTP (k=2)", "vLLM 0.31.0, 128K ctx, 1 user", 102.3], ["Qwen3.6 35B-A3B", "NVFP4, no speculative decoding", "vLLM 0.31.0, 128K ctx, 1 user", 78.1], ["gpt-oss 20B", "", "vLLM 0.31.0, 128K ctx, 1 user", 96.2], ["Nemotron 3.5 Lightning 30B-A3B", "+ DSpark speculative decoding (k=3)", "vLLM 0.31.0, 128K ctx, 1 user", 94.4], ["Nemotron 3.5 Lightning 30B-A3B", "no speculative decoding", "vLLM 0.31.0, 128K ctx, 1 user", 70.3], ["Nemotron 3 Nano Omni 30B", "", "vLLM 0.31.0, 128K ctx, 1 user", 58.8], ["Gemma 4 26B-A4B", "+ official MTP drafter (k=3)", "vLLM 0.31.0, 128K ctx, 1 user", 56.5], ["Gemma 4 26B-A4B", "no speculative decoding", "vLLM 0.31.0, 128K ctx, 1 user", 30.8]]], ["dgx-spark-qwen36-35b-nvfp4-mtp-vitorpontual", "DGX Spark (GB10): Qwen3.6 35B-A3B NVFP4 at 113 tok/s on vLLM with MTP", [["Qwen3.6 35B-A3B", "NVFP4", "vLLM 0.30 + MTP", 113], ["Qwen3.6 35B-A3B", "NVFP4", "vLLM 0.30 + MTP", 86]]], ["dgx-spark-qwen36-27b-nvfp4-dflash", "DGX Spark with Qwen3.6 27B NVFP4 + DFlash speculative decoding", [["Qwen3.6 27B", "NVFP4", "vLLM + DFlash spec decoding", 32.83], ["Qwen3.6 27B", "FP8", "vLLM autoregressive baseline", 7.85]]], ["mia-dgx-spark-ling-3-0-flash-sglang-dspark-dgx-spark", "DGX Spark: Ling 3.0 Flash INT4 on SGLang, 83 tok/s (Mia's AI Lab recipe)", [["Ling 3.0 Flash", "INT4 + DSpark", "SGLang, 1 stream", 83]]], ["mia-dgx-spark-muse-glimmer-30b-dgx-spark-rtx-5090-6000-pro", "DGX Spark: Muse Glimmer 30B NVFP4 on vLLM, 35 tok/s (Mia's AI Lab recipe)", [["Muse Glimmer 30B", "NVFP4 + DFlash", "vLLM, 1 stream", 35]]], ["mia-dgx-spark-nemotron-3-5-bolt-tensorfold-dgx-spark", "DGX Spark: Nemotron 3.5 Bolt 74B-A9B IQ2_XXS on TensorFold Zig, 71 tok/s (Mia's AI Lab recipe)", [["Nemotron 3.5 Bolt 74B-A9B", "IQ2_XXS + NVFP4 + LiLiCorr", "TensorFold Zig, 1 stream", 71]]], ["mia-dgx-spark-ornith-1-5-122b-tensorfold", "DGX Spark: Ornith 1.5 122B EXL3 4-bit on TensorFold Zig, 52.6 tok/s (Mia's AI Lab recipe)", [["Ornith 1.5 122B", "EXL3 4-bit + DFlash2", "TensorFold Zig, 1 stream", 52.6]]], ["mia-dgx-spark-qwen3-8-27b-dgx-spark-tensorfold", "DGX Spark: Qwen3.8 27B MLX 4-bit on TensorFold, 58.3 tok/s (Mia's AI Lab recipe)", [["Qwen3.8 27B", "MLX 4-bit + DFlash2", "TensorFold, 1 stream", 58.3]]], ["mia-dgx-spark-qwen3-8-27b-sglang-dgx-spark", "DGX Spark: Qwen3.8 27B NVFP4 on SGLang, 18.3 tok/s (Mia's AI Lab recipe)", [["Qwen3.8 27B", "NVFP4 + DSpark", "SGLang, 1 stream", 18.3]]], ["mia-dgx-spark-qwen3-8-flash-next-single-dgx-spark", "DGX Spark: Qwen3.8 Flash-Next NVFP4 on vLLM, 48.7 tok/s (Mia's AI Lab recipe)", [["Qwen3.8 Flash-Next", "NVFP4", "vLLM, 1 stream", 48.7]]], ["mia-dgx-spark-qwen3-8-flash-next-single-dgx-spark-tensorfold", "DGX Spark: Qwen3.8 Flash-Next FP4 QAD or INT4-AR on TensorFold Zig, 46-55 tok/s (Mia's AI Lab recipe)", [["Qwen3.8 Flash-Next", "FP4 QAD or INT4-AR + MTP", "TensorFold Zig, 1 stream", "46-55"]]], ["mia-dgx-spark-unsloth-qwen3-6-35b-nvfp4-dgx-spark", "DGX Spark: Qwen3.6 35B-A3B NVFP4 on vLLM, 95.1 tok/s (Mia's AI Lab recipe)", [["Qwen3.6 35B-A3B", "NVFP4 + MTP", "vLLM, 1 stream", 95.1]]]], "4": [["4x-dgx-spark-glm-52-quanttrio", "4x DGX Spark cluster running GLM-5.2 (unpruned)", [["GLM-5.2", "QuantTrio Int4-Int8Mix", "vLLM TP4 + MTP, single-stream mean", 32.5]]], ["4x-dgx-spark-glm-53-flash-ringside", "4x DGX Spark ring running GLM-5.3 Flash NVFP4", [["GLM-5.3 Flash", "NVFP4", "vLLM TP4 (RiNGSiDE), single-stream prose", 61.7], ["GLM-5.3 Flash", "NVFP4", "vLLM TP4 (RiNGSiDE), single-stream code", 105.3]]], ["four-dgx-spark-step-5-preview", "4x DGX Spark (GB10) running Step 5 Preview with tensor parallel over RoCE", [["Step 5 Preview", "EXL3 (experts 4.01 bpw, 341.4 GB total)", "vLLM, TP=4 over RoCE, MTP (2 draft tokens)", 25]]], ["mia-4x-dgx-spark-deepseek-v4-1-flash-1017-4x-dgx-sparks-tensorfold", "4x DGX Spark: DeepSeek V4.1 Flash (1017) FP8 on TensorFold Zig, 65.8 tok/s (Mia's AI Lab recipe)", [["DeepSeek V4.1 Flash", "1017 checkpoint, FP8 + MXFP4 + DSpark", "TensorFold Zig, 4 nodes, 1 stream", 65.8]]], ["mia-4x-dgx-spark-glm-5-3-flash-exl3-2x-dgx-sparks-tensorfold", "4x DGX Spark: GLM-5.3 Flash EXL3 4-bit on TensorFold, 95.2 tok/s (Mia's AI Lab recipe)", [["GLM-5.3 Flash", "EXL3 4-bit + DFlash2", "TensorFold, 4 nodes, 1 stream", 95.2]]]], "2": [["2x-dgx-spark-glm-53-flash-ringside", "2x DGX Spark running GLM-5.3 Flash NVFP4", [["GLM-5.3 Flash", "NVFP4", "vLLM TP2 (RiNGSiDE), single-stream prose", 33.0], ["GLM-5.3 Flash", "NVFP4", "vLLM TP2 (RiNGSiDE), single-stream code", 56.5]]], ["mia-2x-dgx-spark-deepseek-v4-1-flash-1017-2x-dgx-sparks", "2x DGX Spark: DeepSeek V4.1 Flash (1017) EXL3 2.9-bit on TensorFold Zig, 74.9 tok/s (Mia's AI Lab recipe)", [["DeepSeek V4.1 Flash", "1017 checkpoint, EXL3 2.9-bit + DSpark", "TensorFold Zig, 2 nodes, 1 stream", 74.9]]], ["mia-2x-dgx-spark-deepseek-v4-1-flash-exl3-2x-dgx-sparks", "2x DGX Spark: DeepSeek V4.1 Flash EXL3 2.9-bit on vLLM, 31.6 tok/s (Mia's AI Lab recipe)", [["DeepSeek V4.1 Flash", "EXL3 2.9-bit", "vLLM, 2 nodes, 1 stream", 31.6]]], ["mia-2x-dgx-spark-glm-5-3-flash-exl3-2x-dgx-sparks-tensorfold", "2x DGX Spark: GLM-5.3 Flash EXL3 4-bit on TensorFold, 60.4 tok/s (Mia's AI Lab recipe)", [["GLM-5.3 Flash", "EXL3 4-bit + DFlash2", "TensorFold, 2 nodes, 1 stream", 60.4]]], ["mia-2x-dgx-spark-mimo-v2-6-flash-2x-dgx-sparks", "2x DGX Spark: MiMo V2.6 Flash MXFP4 on SGLang, 35 tok/s (Mia's AI Lab recipe)", [["MiMo V2.6 Flash", "MXFP4", "SGLang, 2 nodes, 1 stream", 35]]], ["mia-2x-dgx-spark-qwen3-8-flash-dual-dgx-sparks-tensorfold", "2x DGX Spark: Qwen3.8 Flash-Next NVFP4 or INT4-AR on TensorFold Zig, 67.4 tok/s (Mia's AI Lab recipe)", [["Qwen3.8 Flash-Next", "NVFP4 or INT4-AR + MTP", "TensorFold Zig, 2 nodes, 1 stream", 67.4]]], ["mia-2x-dgx-spark-qwen3-8-flash-next-dual-dgx-sparks", "2x DGX Spark: Qwen3.8 Flash-Next NVFP4 on vLLM, 54.4 tok/s (Mia's AI Lab recipe)", [["Qwen3.8 Flash-Next", "NVFP4", "vLLM, 2 nodes, 1 stream", 54.4]]]], "3": [["mia-3x-dgx-spark-glm-5-3-exl3-3x-dgx-sparks-tensorfold", "3x DGX Spark: GLM-5.3 EXL3 2.75-bit on TensorFold, 31.3 tok/s (Mia's AI Lab recipe)", [["GLM-5.3", "EXL3 2.75-bit + DSpark", "TensorFold, 3 nodes, 1 stream", 31.3]]], ["mia-3x-dgx-spark-glm-5-3-flash-exl3-2x-dgx-sparks-tensorfold", "3x DGX Spark: GLM-5.3 Flash EXL3 4-bit on TensorFold, 77.6 tok/s (Mia's AI Lab recipe)", [["GLM-5.3 Flash", "EXL3 4-bit + DFlash2", "TensorFold, 3 nodes, 1 stream", 77.6]]]]}, "5090": {"1": [["intel-13900k-rtx-5090", "Intel i9-13900K + RTX 5090", [["Llama 3.2 3B", "", "llama.cpp", 271.4]]], ["single-rtx-5090-gpt-oss", "Single RTX 5090 32GB", [["gpt-oss 20B", "MXFP4", "llama.cpp", 282.51]]], ["llamabench-rtx-5090", "RTX 5090: 2026 models (llamabench community)", [["Qwen3.6 35B-A3B", "llama-bench tg128 (quant per result)", "llama.cpp", 258], ["Qwen3.6 27B", "llama-bench tg128 (quant per result)", "llama.cpp", 75.7], ["Gemma 4 31B", "llama-bench tg128 (quant per result)", "llama.cpp", 70.0], ["Gemma 4 26B-A4B", "llama-bench tg128 (quant per result)", "llama.cpp", 235], ["Gemma 4 12B", "llama-bench tg128 (quant per result)", "llama.cpp", 142], ["gpt-oss 20B", "llama-bench tg128 (quant per result)", "llama.cpp", 313], ["Qwen3.5 9B", "llama-bench tg128 (quant per result)", "llama.cpp", 208]]], ["llmspeed-rtx-5090-oct-2026", "RTX 5090 (llm-speed signed runs, Oct 2026): gpt-oss 20B 324 tok/s, Qwen3.8 27B 75 tok/s", [["gpt-oss 20B", "reported as Q4_K_M", "llama.cpp 0.7.0-dev", 324.2], ["Qwen3.6 27B", "Q4_K_M", "llama.cpp 0.7.0-dev", 75.1], ["Qwen3.8 27B", "Q4_K_M (EPYC 9534 host)", "llama.cpp 0.7.0-dev (b11669)", 75.2]]], ["ryzen-9950x3d-rtx-5090-128gb-four-big-moes-gadeyne", "Ryzen 9 9950X3D + RTX 5090 + 128 GB DDR5: four big MoEs with experts offloaded to RAM (Qwen3.8 Flash-Next 78.8 tok/s)", [["Qwen3.8 Flash-Next", "Unsloth UD-Q5_K_XL, MTP drafting, experts offloaded to RAM", "Strata (296b59d)", 78.8], ["Ornith 1.5 122B", "EXL3 4 bpw, 180/256 experts per layer in RAM, no drafting", "TabbyAPI + ExLlamaV3 1.6.2", 46.4], ["GLM-5.3 Flash", "EXL3 3.05 bpw, MTP drafting, experts offloaded to RAM", "Project Maya 1.1.21", 30.1], ["DeepSeek V4.1 Flash", "Maya-S (experts 2-3 bit), partly streamed from SSD", "Project Maya 1.1.21 (maya-ds4)", 24.9]]], ["rtx-5090-strata-qwen38-flash-next-akitaonrails", "RTX 5090 workstation: Qwen3.8 Flash-Next IQ3_S at 159.2 tok/s on Strata (expert offload)", [["Qwen3.8 Flash-Next", "IQ3_S (~55 GB), experts offloaded to RAM, speculation helper", "Strata", 159.2]]], ["llmspeed-rtx-5090-bolt-qwen38-27b", "RTX 5090: Nemotron 3.5 Bolt (IQ2) and Qwen3.8 27B", [["Nemotron 3.5 Bolt 74B-A9B", "NVIDIA mixed 2-bit IQ2 GGUF (22.0 GB)", "llama.cpp build 11669", 154.7], ["Qwen3.8 27B", "Q4_K_M", "llama.cpp", 75.19]]], ["lla-rtx-5090-gemma-4-31b", "RTX 5090 running Gemma 4 31B Q4_K_M (llama-bench)", [["Gemma 4 31B", "Q4_K_M", "llama.cpp b11658 (llama-bench)", 71.3]]], ["rtx-5090-windows-qwen38-ornith-gemma4-kgptalkie", "RTX 5090 on Windows 11: Ornith 1.5 35B-A3B at 273 tok/s, Gemma 4 26B-A4B at 232, Qwen3.8 27B 73.6 (137 with MTP)", [["Ornith 1.5 35B-A3B", "Q4_K_M, f16 KV", "llama-server b10448, no MTP", 273.2], ["Gemma 4 26B-A4B", "Q4_K_M (text-only build), f16 KV", "llama-server b10448", 232.4], ["Qwen3.8 27B", "Q4_K_M, f16 KV", "llama.cpp b10448, MTP off", 73.6], ["Qwen3.8 27B", "Q4_K_M, q4_0 KV", "llama.cpp b10448, MTP n=2 (--spec-type draft-mtp)", 136.7]]], ["rtx-5090-flint-40b-nvfp4-llamacpp-niklaslenz", "RTX 5090: Flint 40B NVFP4 at 224.9 tok/s, 110 tok/s at full 262K context (llama.cpp b11803)", [["Flint 40B-A2.5B", "NVFP4 GGUF (RecursiveSI/flint-nvfp4-gguf), f16 KV", "llama.cpp b11803, no speculation", 224.9], ["Flint 40B-A2.5B", "NVFP4 GGUF, f16 KV", "llama.cpp b11803, no speculation", 110.2]]], ["rtx-5090-128gb-ram-deepseek-v41-flash-nvme-streaming", "One RTX 5090 + 128 GB RAM running DeepSeek V4.1 Flash (552B) by streaming experts from NVMe: 5.1 tok/s", [["DeepSeek V4.1 Flash", "MXFP4 experts (lossless repack) + Q8_0/BF16 dense + fp8 engram tables", "llama.cpp fork dsv41-porte, expert streaming, --reasoning off", 5.1], ["DeepSeek V4.1 Flash", "MXFP4 experts + Q8_0/BF16 dense + fp8 engram tables", "llama.cpp fork dsv41-porte, expert streaming + --moe-stream-direct (O_DIRECT)", 8.9]]]]}, "pro6000": {"1": [["rtx-pro-6000-96gb", "RTX Pro 6000 96GB workstation card", [["gpt-oss 120B", "MXFP4", "llama.cpp", 196.31], ["gpt-oss 20B", "MXFP4", "llama.cpp", 286.91]]], ["rtx-pro-6000-maxq-qwen38-27b-vaditaslim", "RTX Pro 6000 Blackwell Max-Q (300 W cap): Qwen3.8 27B on vLLM, NVFP4 vs BF16", [["Qwen3.8 27B", "NVFP4 (W4A16) + MTP (5 draft tokens), 256K context", "vLLM 0.27.1", 150], ["Qwen3.8 27B", "BF16 + MTP (5 draft tokens), 256K context", "vLLM 0.20.1", 85]]]], "2": [["2x-rtx-pro-6000-max-q-glm-53-flash-vllm", "2x RTX PRO 6000 Max-Q (192 GB): GLM-5.3 Flash at 177.7 tok/s single stream (vLLM, MTP)", [["GLM-5.3 Flash", "modelopt_mixed (ModelOpt mixed precision), FP8 KV", "vLLM (karmic-kraken image), TP 2, MTP 3 speculative tokens", 177.7]]], ["mia-2x-rtx-pro-6000-glm-5-3-flash-exl3-2x-rtx-pro-6000-tensorfold", "2x RTX PRO 6000: GLM-5.3 Flash EXL3 4-bit on TensorFold, 238.3 tok/s (Mia's AI Lab recipe)", [["GLM-5.3 Flash", "EXL3 4-bit + DFlash2", "TensorFold, 1 stream", 238.3]]]], "4": [["mia-4x-rtx-pro-6000-glm-5-3-flash-exl3-4x-rtx-pro-6000-tensorfold", "4x RTX PRO 6000: GLM-5.3 Flash EXL3 4-bit on TensorFold, 265.9 tok/s (Mia's AI Lab recipe)", [["GLM-5.3 Flash", "EXL3 4-bit + DFlash2", "TensorFold, 1 stream", 265.9]]]]}, "5070ti": {"1": [["single-rtx-5070-ti-gpt-oss", "Single RTX 5070 Ti 16GB", [["gpt-oss 20B", "MXFP4", "llama.cpp", 189.45]]], ["llamabench-rtx-5070-ti", "RTX 5070 Ti: 2026 models (llamabench community)", [["Gemma 4 12B", "llama-bench tg128 (quant per result)", "llama.cpp", 87.6]]]], "2": [["capy-flash-2x-rtx-5070-ti-ryzen-9900x", "Dual RTX 5070 Ti on Ryzen 9 9900X (capy-flash maintainer rig)", [["Qwen3.8 Flash-Next", "GSQ-RCO (64 GB edition)", "capy-flash v0.7.0, 2 GPUs, agentic decode", 182], ["Qwen3.8 Flash-Next", "NVFP4", "capy-flash v0.6.0, 2 GPUs, agentic decode", 155], ["Qwen3.8 27B", "3-bit GSQ", "capy-flash (1 GPU), MTP, decode at 8k prompt", 139], ["Qwen3.8 27B", "3-bit GSQ", "capy-flash, 2 GPUs, MTP, decode at 8k prompt", 230]]]]}, "5060ti": {"1": [["single-rtx-5060-ti-16gb-gpt-oss", "Single RTX 5060 Ti 16GB", [["gpt-oss 20B", "MXFP4", "llama.cpp", 111.51]]], ["rtx-5060-ti-16gb-qwen38-27b-mtp-llamacpp", "Single RTX 5060 Ti 16 GB: Qwen3.8 27B IQ4_XS at 27.6 tok/s, 50.2 with MTP (stock llama.cpp)", [["Qwen3.8 27B", "Unsloth UD-IQ4_XS, q8_0 KV", "llama.cpp 23b0202a, no speculative decoding", 27.6], ["Qwen3.8 27B", "Unsloth UD-IQ4_XS + mtp-Qwen3.8-27B-Q4_0 head, q8_0 KV", "llama.cpp 23b0202a, MTP-1", 45.8], ["Qwen3.8 27B", "Unsloth UD-IQ4_XS + mtp-Qwen3.8-27B-Q4_0 head, q8_0 KV", "llama.cpp 23b0202a, MTP-2", 50.2]]]]}, "5080": {"1": [["single-rtx-5080-gpt-oss", "Single RTX 5080 16GB", [["gpt-oss 20B", "MXFP4", "llama.cpp", 204.85]]]], "2": [["capy-flash-2x-rtx-5080-epyc-7f52", "Dual RTX 5080 on EPYC 7F52 (capy-flash user report)", [["Qwen3.8 Flash-Next", "NVFP4", "capy-flash v0.7.2 (CUDA 13.4), agentic decode", 141.3], ["Qwen3.8 Flash-Next", "NVFP4", "capy-flash v0.7.2, --expert-cpu-threads 15, agentic decode", 144.4]]]]}, "7900xtx": {"1": [["rx-7900-xtx-qwen36-35b-a3b-mtp", "Radeon RX 7900 XTX (24 GB): Qwen3.6 35B-A3B up to 245K context, with MTP", [["Qwen3.6 35B-A3B", "UD-Q4_K_XL, llama-bench 2K, f16 KV", "llama.cpp ROCm", 87.2], ["Qwen3.6 35B-A3B", "UD-Q4_K_XL, ~16K real-text context, q4_0 KV", "llama.cpp ROCm (llama-server)", 79.6], ["Qwen3.6 35B-A3B", "UD-Q4_K_XL + MTP (draft-n-max 4), ~16K context", "llama.cpp ROCm (llama-server)", 135.2], ["Qwen3.6 35B-A3B", "UD-Q4_K_XL, ~64K context", "llama.cpp ROCm (llama-server)", 66.0], ["Qwen3.6 35B-A3B", "UD-Q4_K_XL + MTP, ~64K context", "llama.cpp ROCm (llama-server)", 108.2], ["Qwen3.6 35B-A3B", "UD-Q4_K_XL, ~245K context, q4_0 KV", "llama.cpp ROCm (llama-server)", 38.3]]]]}, "pro4000": {"2": [["capy-flash-2x-rtx-pro-4000-threadripper-5975wx", "Dual RTX PRO 4000 Blackwell on Threadripper PRO 5975WX", [["Qwen3.8 Flash-Next", "GSQ-RCO (64 GB edition)", "capy-flash v0.7.1, prose decode (low end of 100-114)", 100]]]]}, "v100": {"2": [["bnk-2x-tesla-v100-32gb-epyc-7402", "Dual Tesla V100 32 GB on EPYC 7402 (bnk engine)", [["Qwen3.8 27B", "Q8_0", "bnk dense engine, tensor parallel, no drafting", 42.4], ["Qwen3.8 27B", "Q8_0", "bnk dense engine, tensor parallel, MTP drafting", 104], ["Qwen3.8 Flash-Next", "FP4-Local", "bnk v0.3.1, decode at 4K context", 114], ["Qwen3.8 Flash-Next", "IQ3_S", "bnk v0.3.1, decode at 4K context", 121], ["DeepSeek V4.1 Flash", "GSQ-RCO 2-bit", "bnk v0.5.0, experts split VRAM/RAM (CPU offload)", 13]]]]}, "b70": {"1": [["intel-arc-pro-b70-neural-download-lab", "Single Intel Arc Pro B70 32 GB (neural.download lab)", [["Gemma 4 26B-A4B", "UD-Q8_K_XL", "llama.cpp SYCL (pinned build, oneAPI 2026.0)", 122.16], ["Ornith 1.5 35B-A3B", "Q4_K_M", "llama.cpp SYCL + neural.download fusion patches; author marks this headline 'pending'", 131.46], ["Qwen3.8 27B", "Q4_K_M", "llama.cpp SYCL + neural.download TP1 stack, no speculation", 27.8]]], ["intel-arc-pro-b70-vllm-gptq-riceharvest", "Single Intel Arc Pro B70 with vLLM XPU cookbook", [["Qwen3.8 27B", "GPTQ Int4 (asym)", "vLLM XPU b70 image + MTP, reasoning prompt", 96], ["Qwen3.8 27B", "GPTQ Int4 (asym)", "vLLM XPU b70 image + MTP, prose prompt", 66], ["Qwen3.6 35B-A3B", "GPTQ Int4", "vLLM XPU b70 image, no MTP", 120]]]], "2": [["2x-intel-arc-pro-b70-neural-download-lab", "Dual Intel Arc Pro B70 (neural.download lab)", [["Qwen3.8 Flash-Next", "IQ3", "neural.download runtime, TP2", 58.56], ["Qwen3.8 27B", "FP8", "vLLM XPU, TP2, MTP 5", 46.01]]]]}}, "vllmUtil": 0.92, "dm": "Qwen3.8 27B"};
/* RigBase /can-it-run/ calculator. Pure math in crCalc() (tested in tests/canrun.test.js); UI below. Data comes from const CR (inlined by canrun.py). */
function crKV(a,ctx,seqs,kvb,tp){
// KV bytes = sum over attention layers of 2 (K and V) x kv_heads x head_dim x tokens x sequences x bytes/element
const out={full:0,swa:0,mla:0,rep:1,lines:[]};
if(!a) return out;
let rep=1;
if(tp>1&&a.full&&a.full[1]&&a.full[1]<tp) rep=tp*Math.ceil(a.full[1]/tp)/a.full[1]; // vLLM replicates KV heads when TP size > KV heads
out.rep=rep;
if(a.full&&a.full[0]){const[L,h,d]=a.full;out.full=2*L*h*d*ctx*seqs*kvb*rep;}
if(a.swa&&a.swa[0]){const[L,h,d,w]=a.swa;const t=Math.min(ctx,w);let r2=1;if(tp>1&&h<tp)r2=tp*Math.ceil(h/tp)/h;out.swa=2*L*h*d*t*seqs*kvb*r2;out.swaRep=r2;}
if(a.mla&&a.mla[0]){const[L,dim]=a.mla;out.mla=L*dim*ctx*seqs*kvb;}
out.total=out.full+out.swa+out.mla;
return out;
}
function crCalc(s){
// s: {P (B params), bpw, nativeGB, arch, ctx, seqs, kvb, unitGB, count, uni (bool), osres, ovh, util, eng, tp}
const W=s.nativeGB?s.nativeGB:s.P*s.bpw/8; // GB (1e9 bytes)
const tp=(s.eng==='vl'&&!s.uni)?s.count:1;
const kv=crKV(s.arch,s.ctx,s.seqs,s.kvb,tp);
const KV=kv.total/1e9;
const per=s.uni?(s.unitGB-(s.osres||0)-s.ovh):(s.unitGB*(s.util||1)-s.ovh);
const usable=Math.max(0,per)*s.count;
const need=W+KV;
const fits=need<=usable;
const spill=fits?0:need-usable;
return {W,KV,kv,per,usable,need,fits,spill,free:usable-need};
}
if(typeof module!=='undefined') module.exports={crCalc,crKV};
if(typeof document!=='undefined'){(function(){
const $=id=>document.getElementById(id);
const f1=x=>x>=100?x.toFixed(0):x>=10?x.toFixed(1):x.toFixed(2);
const nf=x=>Number(x).toLocaleString('en-US');
const sl=s=>s.toLowerCase().replace(/[^a-z0-9]+/g,'-').replace(/^-|-$/g,'');
const M=CR.models,F=CR.formats,H=CR.hw,EN=CR.engines;
const el={m:$('crM'),p:$('crP'),act:$('crA'),eng:$('crE'),f:$('crF'),bpw:$('crBpw'),ctx:$('crC'),seq:$('crS'),kvq:$('crK'),hw:$('crH'),n:$('crN'),mem:$('crMem'),ovh:$('crO'),ram:$('crR'),
fl:$('crFL'),fh:$('crFH'),fd:$('crFD'),sl:$('crSL'),sh:$('crSH'),sd:$('crSD'),sw:$('crSW'),ml:$('crML'),md:$('crMD')};
// ---- populate
let og=null,lastG='';
M.forEach((m,i)=>{const g=m.cur?'Current models':'Older / reference models';if(g!==lastG){og=document.createElement('optgroup');og.label=g;el.m.appendChild(og);lastG=g;}og.appendChild(new Option(m.n+' ('+m.p+'B'+(m.a?', '+m.a+'B active':'')+')',i));});
el.m.appendChild(new Option('Custom model (enter values)', -1));
Object.keys(EN).forEach(k=>el.eng.add(new Option(EN[k].n,k)));
let lg='';H.forEach((h,i)=>{if(h.g!==lg){og=document.createElement('optgroup');og.label=h.g;el.hw.appendChild(og);lg=h.g;}og.appendChild(new Option(h.n+(h.t==='uni'?' (unified)':' — '+h.v+' GB'),i));});
[1024,4096,8192,16384,32768,65536,131072,262144,1048576].forEach(c=>el.ctx.add(new Option(c>=1048576?'1M':(c/1024)+'k',c)));
function fillFormats(){const e=el.eng.value,cur=el.f.value;el.f.innerHTML='';
F.filter(f=>f.e.includes(e)).forEach(f=>{el.f.add(new Option(f.n+(f.b?' — '+f.b.toFixed(2)+' bpw':' — set bpw'),f.id));});
if([...el.f.options].some(o=>o.value===cur)) el.f.value=cur; else el.f.value=EN[e].df;
el.kvq.innerHTML='';EN[e].kv.forEach((k,i)=>el.kvq.add(new Option(k[0]+' ('+k[1]+' B/value)',i)));}
function fillCounts(){const h=H[el.hw.value],cur=+el.n.value||1;el.n.innerHTML='';h.c.forEach(c=>el.n.add(new Option((c)+'× '+(h.t==='uni'?'node'+(c>1?'s':''):'GPU'+(c>1?'s':'')),c)));
el.n.value=h.c.includes(cur)?cur:h.c[0];
$('crMemW').hidden=h.t!=='uni';if(h.t==='uni'){el.mem.innerHTML='';h.mem.forEach(m=>el.mem.add(new Option(m+' GB',m)));el.mem.value=h.dm;}
$('crOL').textContent=h.t==='uni'?'Reserved for OS + runtime, GB per node':'Per-GPU overhead, GB';}
function setOvh(){const h=H[el.hw.value];const mm=+el.mem.value;el.ovh.value=h.t==='uni'?Math.round(Math.max(h.os[0],h.os[1]*mm)+1):'1';}
function setModel(){const i=+el.m.value,m=M[i];const a=m?m.ar:null;
el.p.value=m?m.p:'';el.act.value=m&&m.a?m.a:'';
const v=(x,j)=>x?x[j]:'';el.fl.value=v(a&&a.full,0);el.fh.value=v(a&&a.full,1);el.fd.value=v(a&&a.full,2);
el.sl.value=v(a&&a.swa,0);el.sh.value=v(a&&a.swa,1);el.sd.value=v(a&&a.swa,2);el.sw.value=v(a&&a.swa,3);el.ml.value=v(a&&a.mla,0);el.md.value=v(a&&a.mla,1);
$('crArchSrc').innerHTML=a?(a.manual?'⚠️ <b>Not prefilled:</b> '+a.note+'. ':'Prefilled from <a rel=nofollow href="'+a.cfg+'">config.json</a>'+(a.note_src?' ('+a.note_src+')':'')+'. '+(a.note?a.note+'.':'')):(i<0?'Enter the values from the model\'s config.json (num_hidden_layers, num_key_value_heads, head_dim).':'No architecture data for this model yet — enter it from its config.json.');
if(i<0) $('crArch').open=true;
const mx=m&&a&&a.max;$('crCtxMax').textContent=mx?'Model maximum: '+nf(mx)+' tokens (max_position_embeddings).':'';
// suggest the native format
if(m&&m.nf){const ff=F.find(f=>f.id===m.nf);if(ff&&!ff.e.includes(el.eng.value)){}}
}
function arch(){const n=x=>+x.value||0;const a={};if(n(el.fl))a.full=[n(el.fl),n(el.fh),n(el.fd)];if(n(el.sl))a.swa=[n(el.sl),n(el.sh),n(el.sd),n(el.sw)||1e12];if(n(el.ml))a.mla=[n(el.ml),n(el.md)];
return (a.full||a.swa||a.mla)?a:null;}
function measured(h,count,mname){const list=(CR.meas[h.k]||{})[count]||[];
if(!list.length) return '<p class=muted>No measured build with exactly this setup in RigBase yet. <a href="https://getrigbase.com/submit/?utm_source=huggingface&utm_medium=space">Submit yours</a>.</p>';
let rows=[];list.forEach(b=>b[2].forEach(r=>rows.push([b,r])));
rows.sort((x,y)=>(y[1][0]===mname)-(x[1][0]===mname)||(parseFloat(y[1][3])||0)-(parseFloat(x[1][3])||0));
return '<table class=crt><tr><th>Model</th><th>Quant</th><th>Engine</th><th>tok/s</th><th>Build</th></tr>'+rows.slice(0,12).map(([b,r])=>'<tr'+(r[0]===mname?' class=crhit':'')+'><td>'+r[0]+'</td><td>'+r[1]+'</td><td>'+r[2]+'</td><td><b>'+r[3]+'</b></td><td><a href="https://getrigbase.com/builds/'+b[0]+'/?utm_source=huggingface&utm_medium=space">'+b[1]+'</a></td></tr>').join('')+'</table><p class=muted><span class="bdg ok">measured</span> numbers come from the linked build pages, each with its own source. Highlighted rows are the model you picked.</p>';}
function run(){
const i=+el.m.value,m=M[i]||null,e=el.eng.value,E=EN[e],f=F.find(x=>x.id===el.f.value)||F[0],h=H[el.hw.value],count=+el.n.value||1;
$('crBpwW').hidden=!!f.b; if(f.b) el.bpw.value=f.b;
const bpw=f.b||(+el.bpw.value||4);
const P=+el.p.value||0, ctx=Math.max(1,+el.ctx.value||4096), seqs=Math.max(1,+$('crS').value||1);
const kvo=E.kv[+el.kvq.value]||E.kv[0], kvb=kvo[1];
const uni=h.t==='uni', unitGB=uni?+el.mem.value:h.v, ovh=+el.ovh.value||0;
const native=(m&&m.ng&&f.id===m.nf)?m.ng:0;
const a=arch();
const r=crCalc({P,bpw,nativeGB:native,arch:a,ctx,seqs,kvb,unitGB,count,uni,osres:0,ovh,util:(e==='vl'&&!uni)?CR.vllmUtil:1,eng:e});
// ---- verdict
const pct=r.usable?Math.min(100,100*r.need/r.usable):100;
let v=r.fits?'<span class="crv ok">✅ Fits in '+(uni?'unified memory':'VRAM')+'</span>':'<span class="crv bad">❌ Does not fit fully</span>';
v+=' <span class=muted>(estimate)</span><div class=crbar><i style="width:'+pct.toFixed(0)+'%;background:'+(r.fits?(pct>90?'var(--warn)':'var(--ok)'):'var(--bad)')+'"></i></div>';
v+='<div class=crnums><div><b>'+f1(r.W)+' GB</b><span>weights</span></div><div><b>'+(a?f1(r.KV)+' GB':'?')+'</b><span>KV cache @ '+nf(ctx)+(seqs>1?' × '+seqs:'')+'</span></div><div><b>'+f1(r.need)+' GB</b><span>total needed</span></div><div><b>'+f1(r.usable)+' GB</b><span>usable on '+count+'× '+h.n+'</span></div></div>';
if(r.fits) v+='<p>Headroom left: <b>'+f1(r.free)+' GB</b>'+(a&&r.KV>0?(()=>{const more=Math.floor(r.free/(r.KV/ctx/seqs)),mx=m&&m.ar&&m.ar.max;return (mx&&ctx+more>=mx)?' — enough for the model\'s full '+nf(mx)+'-token context at this KV precision (roughly; sliding-window layers stop growing)':' — room for roughly <b>'+nf(more)+'</b> more tokens of context at this KV precision';})():'')+'.</p>';
else v+='<p>Short by <b>'+f1(r.spill)+' GB</b>. Options: a smaller format, shorter context or quantized KV cache, '+(e==='lc'?'offloading '+f1(r.spill)+' GB to system RAM (slower; for MoE models offload experts first with <code>--n-cpu-moe</code>), ':'')+'more GPUs/nodes, or <a href="https://getrigbase.com/rent/?utm_source=huggingface&utm_medium=space">renting a bigger GPU</a>.</p>';
// ---- RAM verdict
const ram=+el.ram.value||0;let rv;
if(uni) rv='<b>System RAM:</b> on '+h.n+' the RAM <i>is</i> the GPU memory, so the number above already covers it. The reserve field holds back memory for the OS.';
else if(r.fits) rv='<b>System RAM needed: small.</b> Everything sits in VRAM, so RAM only holds the OS and the engine and streams the weights in once at load time (llama.cpp memory-maps the file, so RAM can be <i>smaller than the model</i>). Your '+ram+' GB is '+(ram>=16?'fine':'probably tight for the OS + engine; 16–32 GB is a comfortable floor')+'.';
else rv='<b>System RAM matters here:</b> about <b>'+f1(r.spill)+' GB</b> of model + KV must live in RAM'+(e==='lc'?' (llama.cpp offload)':'')+', plus room for the OS. '+(ram>=r.spill+8?'Your '+ram+' GB covers it, but tokens/sec will drop to what your RAM bandwidth allows.':'Your '+ram+' GB is not enough for that offload — pick a smaller format or add GPUs.');
$('crRam').innerHTML=rv;
$('crOut').innerHTML=v;
// ---- warnings
const w=[];
if(bpw<4&&!native) w.push('⚠️ <b>Below ~4 bits per weight.</b> Quality loss grows quickly under 4 bpw (Q3, Q2, 3-bit EXL). We do not recommend it unless the model is very large and nothing bigger fits; try Q4_K_M / AWQ 4-bit / NVFP4 first.');
if(f.na&&!f.na.includes(h.a)) w.push('ℹ️ '+(f.nn||'This format is not native on this hardware.'));
if(E.hw&&!E.hw.includes(h.a)) w.push('ℹ️ '+E.hwn);
if(m&&m.nf&&f.id!==m.nf&&CR.formats.find(x=>x.id===m.nf)) w.push('ℹ️ '+m.n+' ships natively in '+CR.formats.find(x=>x.id===m.nf).n+'.');
if(a&&a.manual!==true&&m&&m.ar&&m.ar.max&&ctx>m.ar.max) w.push('⚠️ Context is longer than the model\'s max_position_embeddings ('+nf(m.ar.max)+').');
if(!a) w.push('⚠️ No architecture values, so KV cache is not counted. Open “Model architecture” and enter them from config.json.');
if(m&&m.ar&&m.ar.manual) w.push('⚠️ '+m.n+': '+m.ar.note+'.');
if(e==='vl'&&!uni&&count>1){const qh=m&&m.ar&&m.ar.qh;
if(qh&&qh%count) w.push('⚠️ vLLM tensor parallel needs the attention-head count ('+qh+') to divide by the TP size ('+count+'). Use <code>-tp 2</code>/<code>-tp 4</code> with pipeline parallel (<code>-pp</code>) for the rest, or <code>-pp '+count+'</code>.');
else if(![2,4,8,16].includes(count)) w.push('ℹ️ '+count+' GPUs: vLLM users normally pick tensor-parallel sizes of 2, 4 or 8; for '+count+' GPUs consider <code>-tp</code> × <code>-pp</code> (e.g. 6 = 2×3) or pipeline parallel only.');
if(r.kv.rep>1) w.push('ℹ️ With TP '+count+' and only '+a.full[1]+' KV heads, vLLM replicates KV heads across GPUs, so the KV cache shown is '+r.kv.rep.toFixed(1)+'× the single-GPU figure.');}
$('crWarn').innerHTML=w.map(x=>'<div class="note'+(x.startsWith('⚠')?' warn':'')+'">'+x+'</div>').join('');
// ---- math
const L=[];
L.push(['Weights',native?'native checkpoint size '+native+' GB (model card)':P+' B params × '+bpw+' bits ÷ 8',f1(r.W)+' GB']);
if(a&&a.full){const[l,hh,d]=a.full;L.push(['KV, full-attention layers','2 (K+V) × '+l+' layers × '+hh+' KV heads × '+d+' head_dim × '+nf(ctx)+' tokens × '+seqs+' seq × '+kvb+' B'+(r.kv.rep>1?' × '+r.kv.rep.toFixed(2)+' (TP replication)':''),f1(r.kv.full/1e9)+' GB']);}
if(a&&a.swa){const[l,hh,d,wn]=a.swa;L.push(['KV, sliding-window layers','2 × '+l+' layers × '+hh+' × '+d+' × min('+nf(ctx)+', window '+nf(wn)+') × '+seqs+' × '+kvb+' B'+(r.kv.swaRep>1?' × '+r.kv.swaRep.toFixed(2):''),f1(r.kv.swa/1e9)+' GB']);}
if(a&&a.mla){const[l,dm]=a.mla;L.push(['KV, MLA latent',l+' layers × '+dm+' latent values × '+nf(ctx)+' tokens × '+seqs+' × '+kvb+' B',f1(r.kv.mla/1e9)+' GB']);}
if(m&&m.ar&&m.ar.other) L.push(['Linear / Mamba / conv layers','fixed-size state per sequence, does not grow with context','not counted']);
L.push(['Usable memory',uni?count+' × ('+unitGB+' GB − '+ovh+' GB reserve)':count+' × ('+unitGB+' GB'+(e==='vl'?' × '+CR.vllmUtil+' gpu_memory_utilization':'')+' − '+ovh+' GB overhead)',f1(r.usable)+' GB']);
L.push(['Verdict',f1(r.W)+' + '+f1(r.KV)+' = '+f1(r.need)+' GB '+(r.fits?'≤':'>')+' '+f1(r.usable)+' GB',r.fits?'fits':'short '+f1(r.spill)+' GB']);
$('crMath').innerHTML='<table class=crt>'+L.map(x=>'<tr><td>'+x[0]+'</td><td><code>'+x[1]+'</code></td><td><b>'+x[2]+'</b></td></tr>').join('')+'</table><p class=muted>GB = 10⁹ bytes. GPU makers\' “24 GB” is really 24 GiB (≈25.8 × 10⁹ bytes), so true headroom is ~7% larger than shown. KV bytes/value: FP16/BF16 2, FP8 1, q8_0 1.0625, q4_0 0.5625 (llama.cpp block formats incl. scales).</p>';
// ---- parallel advice
let pa='';
if(count>1){pa=uni?'<p><b>'+count+' nodes:</b> '+h.pl+'</p>':'<p><b>'+count+' GPUs with '+E.n+':</b> '+E.par+'</p>';}
$('crPar').innerHTML=pa;
$('crMeas').innerHTML='<h3>Measured on '+count+'× '+h.n+'</h3>'+measured(h,count,m?m.n:'');
$('crLinks').innerHTML=(m&&m.pg?'<a href="https://getrigbase.com/models/'+sl(m.n)+'/?utm_source=huggingface&utm_medium=space">All measured speeds for '+m.n+' →</a> ':'')+(h.k==='spark'?' <a href="https://getrigbase.com/dgx-spark/?utm_source=huggingface&utm_medium=space">DGX Spark recipes →</a>':'');
// ---- url state
try{const q=new URLSearchParams({m:m?sl(m.n):'custom',e,f:f.id,hw:h.k,n:count,ctx});history.replaceState(null,'','?'+q.toString()+location.hash);}catch(_){}
}
// ---- init from URL
const q=new URLSearchParams(location.search);
const mi=M.findIndex(m=>sl(m.n)===q.get('m'));el.m.value=mi>=0?mi:M.findIndex(m=>m.n===CR.dm);
el.eng.value=EN[q.get('e')]?q.get('e'):'lc';fillFormats();if(q.get('f')&&F.some(x=>x.id===q.get('f')&&x.e.includes(el.eng.value)))el.f.value=q.get('f');
const hi=H.findIndex(h=>h.k===q.get('hw'));el.hw.value=hi>=0?hi:H.findIndex(h=>h.k==='3090');fillCounts();if(q.get('n'))el.n.value=q.get('n');setOvh();
el.ctx.value=[...el.ctx.options].some(o=>o.value===q.get('ctx'))?q.get('ctx'):32768;
setModel();
el.m.onchange=()=>{setModel();run()};
el.eng.onchange=()=>{fillFormats();run()};
el.hw.onchange=()=>{fillCounts();setOvh();run()};
el.mem.onchange=()=>{setOvh();run()};
document.querySelectorAll('#crForm input,#crForm select').forEach(x=>{if(!x.onchange)x.onchange=run;x.oninput=run;});
document.querySelectorAll('[data-preset]').forEach(b=>b.onclick=()=>{const p=JSON.parse(b.dataset.preset);
el.m.value=M.findIndex(m=>m.n===p.m);setModel();el.eng.value=p.e;fillFormats();el.f.value=p.f;el.hw.value=H.findIndex(h=>h.k===p.hw);fillCounts();el.n.value=p.n;setOvh();el.ctx.value=p.ctx;run();});
run();
// ---- tooltips: hover (CSS), keyboard focus (CSS), tap/click toggles; Esc closes
document.addEventListener('click',ev=>{const t=ev.target.closest('.tip');document.querySelectorAll('.tipw.open').forEach(w=>{if(!t||w!==t.parentNode){w.classList.remove('open');w.querySelector('.tip').setAttribute('aria-expanded','false');}});
if(t){ev.preventDefault();const w=t.parentNode,o=w.classList.toggle('open');t.setAttribute('aria-expanded',o?'true':'false');}});
document.addEventListener('keydown',ev=>{if(ev.key==='Escape')document.querySelectorAll('.tipw.open').forEach(w=>{w.classList.remove('open');w.querySelector('.tip').setAttribute('aria-expanded','false');});});
})();}
</script></main>
<footer class=sf><div class=sfi><div><a href='https://getrigbase.com/?utm_source=huggingface&utm_medium=space' class=logo style='padding:0'><i></i>RigBase</a><p>Real home AI rigs with measured tokens/sec, every number linked to its source. No estimates, no marketing numbers.</p><p>Some links are affiliate links (placeholder). EUR/USD at ECB rate 1.1208.</p></div><div><h4>Explore</h4><a href="https://getrigbase.com/builds/?utm_source=huggingface&utm_medium=space">All builds</a><a href="https://getrigbase.com/gpus/?utm_source=huggingface&utm_medium=space">GPUs & chips</a><a href="https://getrigbase.com/models/?utm_source=huggingface&utm_medium=space">Models</a><a href="https://getrigbase.com/best/?utm_source=huggingface&utm_medium=space">Best rig by VRAM</a><a href="https://getrigbase.com/compare/?utm_source=huggingface&utm_medium=space">Compare GPUs</a><a href="https://getrigbase.com/dgx-spark/?utm_source=huggingface&utm_medium=space">DGX Spark recipes</a><a href="https://getrigbase.com/can-it-run/?utm_source=huggingface&utm_medium=space">Can it run?</a></div><div><h4>Guides</h4><a href="https://getrigbase.com/guides/?utm_source=huggingface&utm_medium=space">All guides & tools</a><a href="https://getrigbase.com/guides/best-gpu-for-local-llm-2026/?utm_source=huggingface&utm_medium=space">Best GPU in 2026</a><a href="https://getrigbase.com/guides/best-value-gpu-local-llm-europe/?utm_source=huggingface&utm_medium=space">Best value per €</a><a href="https://getrigbase.com/guides/mac-vs-pc-local-ai/?utm_source=huggingface&utm_medium=space">Mac vs PC</a><a href="https://getrigbase.com/budget/?utm_source=huggingface&utm_medium=space">By budget</a></div><div><h4>Project</h4><a href="https://getrigbase.com/submit/?utm_source=huggingface&utm_medium=space">Submit a build</a><a href="https://getrigbase.com/newsletter/?utm_source=huggingface&utm_medium=space">Newsletter</a><a href="https://getrigbase.com/data/?utm_source=huggingface&utm_medium=space">Free dataset (CC BY 4.0)</a><a href="https://getrigbase.com/badge/?utm_source=huggingface&utm_medium=space">Embed badge</a><a href="https://getrigbase.com/rent/?utm_source=huggingface&utm_medium=space">Rent before you buy</a><a href="https://getrigbase.com/advertise/?utm_source=huggingface&utm_medium=space">Advertise / sponsor</a></div></div><div class=sfb><span>Last updated: <time datetime=2026-10-28>2026-10-28</time> · 150 builds · data checked against sources</span><span>© 2026 RigBase</span></div></footer></body></html>