AI Engineering Lab commited on
Commit ·
78e199b
1
Parent(s): 3e856b1
results: add verified RTX 4070 Laptop benchmark + cross-GPU comparison table
Browse files- RTX 4070 Laptop (8GB): 8,192 -> 64,000 ctx (+7.8x), -3.2% TPS, +0.54 GB VRAM
- 3 independent runs verified on Llama-3.1-8B-Instruct Q4_K_M
- Add cross-GPU comparison table (3090 vs 4070) to README
- Update HF Space frontmatter title + tags for both GPUs
- README.md +32 -5
- results/turboquant-4070-results-2026-04-01.json +38 -0
README.md
CHANGED
|
@@ -1,5 +1,5 @@
|
|
| 1 |
---
|
| 2 |
-
title: TurboQuant on
|
| 3 |
emoji: 🚀
|
| 4 |
colorFrom: blue
|
| 5 |
colorTo: purple
|
|
@@ -13,6 +13,7 @@ tags:
|
|
| 13 |
- turboquant
|
| 14 |
- benchmark
|
| 15 |
- rtx3090
|
|
|
|
| 16 |
- consumer-hardware
|
| 17 |
- mistral
|
| 18 |
- llama-cpp
|
|
@@ -46,8 +47,11 @@ tags:
|
|
| 46 |
|
| 47 |
## 📊 Results
|
| 48 |
|
| 49 |
-
Tested on
|
| 50 |
-
|
|
|
|
|
|
|
|
|
|
| 51 |
|
| 52 |
| | Baseline (f16) | TurboQuant turbo3 | Delta |
|
| 53 |
|--|:--------------:|:-----------------:|:-----:|
|
|
@@ -58,11 +62,34 @@ Results are the average of **two independent benchmark runs** (April 1, 2026).
|
|
| 58 |
|
| 59 |
> **12× more context. +12% VRAM. −8% speed. Same model weights.**
|
| 60 |
|
| 61 |
-
|
| 62 |
-
|
| 63 |
|
| 64 |
Raw data: [`results/turboquant-rtx3090-2026-04-01.json`](results/turboquant-rtx3090-2026-04-01.json) · [`results/turboquant-rtx3090-2026-04-01-v2.json`](results/turboquant-rtx3090-2026-04-01-v2.json)
|
| 65 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 66 |
---
|
| 67 |
|
| 68 |
## 🚀 Quick Start
|
|
|
|
| 1 |
---
|
| 2 |
+
title: TurboQuant on Consumer GPUs — 100K Context on RTX 3090, 64K on RTX 4070
|
| 3 |
emoji: 🚀
|
| 4 |
colorFrom: blue
|
| 5 |
colorTo: purple
|
|
|
|
| 13 |
- turboquant
|
| 14 |
- benchmark
|
| 15 |
- rtx3090
|
| 16 |
+
- rtx4070
|
| 17 |
- consumer-hardware
|
| 18 |
- mistral
|
| 19 |
- llama-cpp
|
|
|
|
| 47 |
|
| 48 |
## 📊 Results
|
| 49 |
|
| 50 |
+
Tested on two consumer GPUs. Results verified across multiple independent runs (April 1, 2026).
|
| 51 |
+
|
| 52 |
+
### RTX 3090 (24 GB) — Mistral-Small-3.2-24B Q4_K_M
|
| 53 |
+
|
| 54 |
+
*Average of 2 independent benchmark runs.*
|
| 55 |
|
| 56 |
| | Baseline (f16) | TurboQuant turbo3 | Delta |
|
| 57 |
|--|:--------------:|:-----------------:|:-----:|
|
|
|
|
| 62 |
|
| 63 |
> **12× more context. +12% VRAM. −8% speed. Same model weights.**
|
| 64 |
|
| 65 |
+
Run 1 (cold): Baseline 49.2 TPS / 15,408 MB → Turbo3 45.0 TPS / 17,224 MB
|
| 66 |
+
Run 2 (warm): Baseline 51.2 TPS / 15,695 MB → Turbo3 47.1 TPS / 17,581 MB
|
| 67 |
|
| 68 |
Raw data: [`results/turboquant-rtx3090-2026-04-01.json`](results/turboquant-rtx3090-2026-04-01.json) · [`results/turboquant-rtx3090-2026-04-01-v2.json`](results/turboquant-rtx3090-2026-04-01-v2.json)
|
| 69 |
|
| 70 |
+
### RTX 4070 Laptop (8 GB) — Llama-3.1-8B-Instruct Q4_K_M
|
| 71 |
+
|
| 72 |
+
*Average of 3 independent benchmark runs.*
|
| 73 |
+
|
| 74 |
+
| | Baseline (f16) | TurboQuant turbo3 | Delta |
|
| 75 |
+
|--|:--------------:|:-----------------:|:-----:|
|
| 76 |
+
| **Context** | 8,192 tokens | **64,000 tokens** | **+7.8×** |
|
| 77 |
+
| **VRAM** | 5.7 GB | 6.2 GB | +0.54 GB only |
|
| 78 |
+
| **Tokens/s** | 49.8 | 48.2 | **−3.2%** |
|
| 79 |
+
|
| 80 |
+
> **7.8× more context. +0.5 GB VRAM. −3% speed. Even better ratio on smaller GPU.**
|
| 81 |
+
|
| 82 |
+
Raw data: [`results/turboquant-4070-results-2026-04-01.json`](results/turboquant-4070-results-2026-04-01.json)
|
| 83 |
+
|
| 84 |
+
### Cross-GPU Summary
|
| 85 |
+
|
| 86 |
+
| GPU | VRAM | Model | Max Context (turbo3) | Speed Loss |
|
| 87 |
+
|-----|------|-------|---------------------|-----------|
|
| 88 |
+
| RTX 3090 | 24 GB | Mistral-Small-3.2 24B | 100,000 tokens | −8.3% |
|
| 89 |
+
| RTX 4070 Laptop | 8 GB | Llama-3.1 8B | 64,000 tokens | −3.2% |
|
| 90 |
+
|
| 91 |
+
TurboQuant scales with the GPU: the principle (+7-12× context, minimal speed loss) holds across hardware classes.
|
| 92 |
+
|
| 93 |
---
|
| 94 |
|
| 95 |
## 🚀 Quick Start
|
results/turboquant-4070-results-2026-04-01.json
ADDED
|
@@ -0,0 +1,38 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"date": "2026-04-01",
|
| 3 |
+
"hardware": {
|
| 4 |
+
"gpu": "NVIDIA GeForce RTX 4070 Laptop GPU",
|
| 5 |
+
"vram_gb": 8,
|
| 6 |
+
"node": ".91 (dev-pc-legion)"
|
| 7 |
+
},
|
| 8 |
+
"model": "Meta-Llama-3.1-8B-Instruct-Q4_K_M",
|
| 9 |
+
"model_size_gb": 4.7,
|
| 10 |
+
"runs": 3,
|
| 11 |
+
"baseline": {
|
| 12 |
+
"ctx": 8192,
|
| 13 |
+
"kv_type": "f16",
|
| 14 |
+
"vram_mb": 5835,
|
| 15 |
+
"tps_runs": [
|
| 16 |
+
50.01,
|
| 17 |
+
49.87,
|
| 18 |
+
49.61
|
| 19 |
+
],
|
| 20 |
+
"tps_avg": 49.83
|
| 21 |
+
},
|
| 22 |
+
"turboquant": {
|
| 23 |
+
"ctx": 64000,
|
| 24 |
+
"kv_type": "turbo3",
|
| 25 |
+
"vram_mb": 6383,
|
| 26 |
+
"tps_runs": [
|
| 27 |
+
48.21,
|
| 28 |
+
48.26,
|
| 29 |
+
48.22
|
| 30 |
+
],
|
| 31 |
+
"tps_avg": 48.23
|
| 32 |
+
},
|
| 33 |
+
"delta": {
|
| 34 |
+
"ctx_multiplier": 7.8,
|
| 35 |
+
"vram_delta_gb": 0.54,
|
| 36 |
+
"tps_delta_pct": -3.2
|
| 37 |
+
}
|
| 38 |
+
}
|