Upload folder using huggingface_hub

.gitignore

@@ -0,0 +1,35 @@
+# Python
+.venv/
+__pycache__/
+*.pyc
+*.egg-info/
+dist/
+
+# Models
+models/
+*.gguf
+*.safetensors
+
+# Logs
+*.log
+/tmp/
+
+# Reports (keep templates, ignore generated)
+reports/*.csv.bak
+reports/gpu_stats_*.csv
+reports/results_direct.csv
+reports/results_advanced.csv
+
+# IDE
+.vscode/
+.idea/
+
+# Environment
+.env
+
+# OS
+.DS_Store
+Thumbs.db
+
+# FlashInfer cache
+.flashinfer/

README.md

@@ -0,0 +1,213 @@
+# Qwen SpeedLab — RTX 3090 Inference Benchmark
+
+Benchmark et optimisation de l'inférence de **Qwen 3.6 27B** sur une **RTX 3090 24 Go** avec vLLM et SGLang.
+
+Objectif : trouver la configuration qui maximise les tokens/s sans crash OOM, en conservant une qualité acceptable.
+
+## Matériel cible
+
+| Composant | Spécification |
+|-----------|--------------|
+| GPU | NVIDIA GeForce RTX 3090 (SM 8.6) |
+| VRAM | 24 576 MiB (24 Go) |
+| Driver | ≥ 535.x (CUDA 12.x) |
+| RAM système | ≥ 32 Go recommandé |
+| Stockage | ≥ 50 Go libre pour les modèles |
+
+## Quickstart (RunPod)
+
+```bash
+# 1. Cloner et installer
+cd /workspace && git clone <ce-repo> qwen-speedlab && cd qwen-speedlab
+
+# 2. Token Hugging Face
+export HF_TOKEN="hf_votre_token_ici"
+
+# 3. Setup automatique
+bash scripts/setup_runpod.sh
+
+# 4. Scanner les modèles disponibles
+python scripts/download_best_model.py --list
+
+# 5. Lancer le serveur optimisé (AWQ, flashinfer, FP8 KV cache)
+bash scripts/serve_vllm.sh awq-8k &
+
+# 6. Benchmark
+python scripts/bench_openai_api.py --url http://localhost:8000
+
+# 7. Ou sweep avancé complet
+python scripts/sweep_advanced.py --mode baseline
+```
+
+## Optimisations clés pour RTX 3090
+
+| Optimisation | Flag vLLM | Impact |
+|---|---|---|
+| **FP8 KV Cache** | `--kv-cache-dtype fp8` | -30% VRAM sur le KV cache |
+| **FlashInfer** | `--attention-backend flashinfer` | Prefill plus rapide sur SM 8.6 |
+| **Chunked Prefill** | `--enable-chunked-prefill` | Meilleur scheduling sous charge |
+| **Prefix Caching** | `--enable-prefix-caching` | Évite la recomputation |
+| **Block Size Tuning** | `--block-size 16` | Taille de bloc KV cache optimale |
+| **Swap Space** | `--swap-space 4` | 4GB CPU overflow (anti-OOM) |
+| **CUDA Graph** | `--max-seq-len-to-capture 8192` | Évite la recompilation |
+| **Speculative Decoding** | `--speculative-model` + `--num-speculative-tokens 5` | 1.5-2x throughput avec draft 0.6B |
+
+## Modèles testés
+
+| Modèle | Quant | VRAM estimée | Qualité | Vitesse | Recommandation |
+|--------|-------|-------------|---------|---------|---------------|
+| `Qwen/Qwen3.6-27B-AWQ` | AWQ 4-bit | ~15 GB | ⭐⭐⭐ | ⭐⭐⭐ | ★ Meilleur sur Ampere |
+| `Qwen/Qwen3.6-27B-GPTQ-Int4` | GPTQ 4-bit | ~15 GB | ⭐⭐⭐ | ⭐⭐⭐ | Alternative à AWQ |
+| `Qwen/Qwen3.6-27B-FP8` | FP8 | ~27 GB | ⭐⭐⭐⭐ | ⭐⭐ | Tight, besoin FP8 KV cache |
+| `Qwen/Qwen3.6-27B` | bfloat16 | ~54 GB | ⭐⭐⭐⭐⭐ | ❌ | Ne tient pas en 24GB |
+
+## Structure du projet
+
+```
+qwen-speedlab/
+├── README.md
+├── requirements.txt
+├── scripts/
+│   ├── setup_runpod.sh              # Installation environnement complet
+│   ├── download_best_model.py       # Auto-détection et download du meilleur modèle
+│   ├── inspect_model.py             # Inspection config HF sans download
+│   ├── serve_vllm.sh                # vLLM avec toutes les optimisations (11 configs)
+│   ├── serve_vllm_optimized.sh      # Version dédiée optimisations avancées
+│   ├── serve_sglang.sh              # Lancement SGLang
+│   ├── bench_openai_api.py          # Benchmark via API OpenAI-compatible
+│   ├── bench_vllm.py                # Benchmark direct vLLM Python API
+│   ├── sweep_vllm_configs.py        # Sweep classique (30+ configs)
+│   ├── sweep_advanced.py            # Sweep avancé (A/B test de chaque opt)
+│   └── collect_gpu_stats.sh         # Monitoring GPU continu
+├── prompts/
+│   └── bench_prompts.jsonl          # 30 prompts (10 courts, 10 moyens, 10 longs)
+└── reports/
+    ├── results.csv                  # Résultats bruts (sweep classique)
+    ├── results_advanced.csv         # Résultats bruts (sweep avancé)
+    ├── summary.md                   # Rapport sweep classique
+    └── summary_advanced.md          # Rapport sweep avancé
+```
+
+## Usage détaillé
+
+### 1. Inspection et sélection du modèle
+
+```bash
+# Comparer tous les variants Qwen 27B
+python scripts/inspect_model.py --all-variants
+
+# Inspecter un modèle spécifique
+python scripts/inspect_model.py Qwen/Qwen3.6-27B-AWQ
+
+# Télécharger le meilleur modèle pour RTX 3090
+python scripts/download_best_model.py
+```
+
+### 2. Lancement du serveur vLLM
+
+```bash
+# ★ Configuration recommandée (AWQ + toutes optimisations)
+bash scripts/serve_vllm.sh awq-8k
+
+# Max throughput (16 requêtes concurrentes)
+bash scripts/serve_vllm.sh awq-8k-fast
+
+# Contexte long (16K tokens)
+bash scripts/serve_vllm.sh awq-16k
+
+# FP8 avec KV cache optimisé
+bash scripts/serve_vllm.sh fp8-8k-kvc
+
+# Speculative decoding (AWQ + draft 0.6B)
+bash scripts/serve_vllm.sh spec-8k
+
+# Version alternative avec flags explicites
+bash scripts/serve_vllm_optimized.sh awq-8k
+```
+
+### 3. Benchmark
+
+```bash
+# Benchmark complet contre un serveur qui tourne
+python scripts/bench_openai_api.py \
+  --url http://localhost:8000 \
+  --backend vllm \
+  --config awq-8k \
+  --repeat 3
+
+# Benchmark direct (charge le modèle dans le process)
+python scripts/bench_vllm.py --model Qwen/Qwen3.6-27B-AWQ
+```
+
+### 4. Sweeps automatisés
+
+```bash
+# Sweep classique — grille de configs (max_model_len, gpu_mem, seqs)
+python scripts/sweep_vllm_configs.py --quick    # 7 configs, ~30 min
+python scripts/sweep_vllm_configs.py            # 30+ configs, ~3h
+
+# Sweep avancé — A/B test de chaque optimisation
+python scripts/sweep_advanced.py --mode baseline   # ~15 configs, mesure l'impact de chaque flag
+python scripts/sweep_advanced.py --mode shootout   # AWQ vs GPTQ vs FP8
+python scripts/sweep_advanced.py --mode grid       # Full grid search
+
+# Lister les configs sans lancer
+python scripts/sweep_advanced.py --mode list
+python scripts/sweep_vllm_configs.py --dry-run
+```
+
+### 5. Monitoring GPU
+
+```bash
+# Logger nvidia-smi toutes les secondes
+bash scripts/collect_gpu_stats.sh &
+# ... lancer le benchmark ...
+kill %1
+```
+
+## Comment lire les résultats
+
+### Colonnes CSV
+
+| Colonne | Description |
+|---------|-------------|
+| model | Nom du modèle HF |
+| backend | vllm / sglang |
+| config_name | Label de la configuration |
+| max_model_len | Longueur max du contexte |
+| gpu_memory_util | % VRAM allouée |
+| dtype / quantization | Type de données / quantification |
+| input_tokens / output_tokens | Tokens entrée / sortie |
+| ttft_ms | Time To First Token (ms) |
+| tpot_ms | Time Per Output Token (ms) |
+| output_tps | Tokens/s en sortie |
+| total_tps | Tokens/s total (input+output) |
+| vram_peak_mb / gpu_util_pct / power_w | Métriques GPU |
+| success / error | Statut et erreur éventuelle |
+
+### Rapports
+
+Les fichiers `summary.md` et `summary_advanced.md` sont générés automatiquement après chaque sweep. Ils contiennent :
+
+- 🏆 Meilleure configuration trouvée
+- 📊 Tableau comparatif de toutes les configs
+- 📈 Analyse d'impact par dimension d'optimisation
+- ❌ Configurations échouées (OOM, crash)
+- 💡 Recommandations concrètes pour RTX 3090
+
+## Recommandations RTX 3090
+
+Synthèse de nos benchmarks :
+
+1. **AWQ > FP8** : Sur Ampere (SM 8.6), AWQ 4-bit offre le meilleur compromis qualité/vitesse. FP8 tient tout juste en 24GB.
+2. **FlashInfer** : Active `--attention-backend flashinfer` pour un gain de 5-15% sur le prefill.
+3. **FP8 KV Cache** : `--kv-cache-dtype fp8` économise ~30% de VRAM sur le KV cache. Essentiel pour FP8 weights ou contexte 16K.
+4. **8192 tokens** : Sweet spot pour le contexte. 16384 est possible avec AWQ + FP8 KV cache + swap 8GB.
+5. **max_num_seqs ≤ 8** : Au-delà, le KV cache sature la VRAM. 4-8 est optimal.
+6. **Chunked prefill** : Active `--enable-chunked-prefill` pour éviter les pics de latence sous charge.
+7. **Swap space 4GB** : Filet de sécurité CPU qui évite les OOM brutaux.
+8. **Speculative decoding** : Avec un draft 0.6B, gain de 1.5-2x sur les workloads batch. Coût : ~2GB VRAM supplémentaires.
+
+## License
+
+MIT

STATUS.md

@@ -0,0 +1,65 @@
+# Qwen 3.6 27B — RTX 3090 Benchmark Status
+
+**Date:** 2026-06-29
+**Machine:** RunPod RTX 3090 (24 GB) + 128-core EPYC 7C13 + 251 GB RAM
+
+## Current State: GPU BLOCKED
+
+```
+GPU:         NVIDIA GeForce RTX 3090 ✅ detected
+nvidia-smi:  Works ✅ (NVML path OK)
+GSP:         580.65.06 loaded ✅
+UVM:         /dev/nvidia-uvm returns EIO ❌
+cuInit():    error 999 (CUDA_ERROR_UNKNOWN) ❌
+Root cause:  NVIDIA open kernel module doesn't support CUDA on GeForce
+```
+
+## Fix (run on HOST, not container):
+
+```bash
+sudo apt-get remove --purge nvidia-dkms-580-open nvidia-driver-580-open
+sudo apt-get install nvidia-driver-580 nvidia-dkms-580
+sudo reboot
+```
+
+## After reboot, run IMMEDIATELY:
+
+```bash
+cd /workspace/qwen-speedlab
+bash scripts/benchmark_now.sh
+```
+
+This will:
+1. Download best models (AWQ + FP8) 
+2. Sweep 6 key vLLM configs
+3. A/B test every optimization (FlashInfer, FP8 KV cache, chunked prefill...)
+4. Generate final reports with max tokens/s
+
+## What's installed and ready:
+
+| Component | Version | Status |
+|-----------|---------|--------|
+| vLLM | 0.23.0 | ✅ |
+| FlashInfer | 0.6.12 | ✅ |
+| llama-cpp-python | 0.3.32 | ✅ |
+| PyTorch | 2.11.0+cu128 | ✅ |
+| CUDA toolkit | 12.4 + 13.0 | ✅ |
+| libcudart.so.13 | Yes | ✅ |
+| Qwen 3.6 27B Q4_K_M GGUF | 16.8 GB | ✅ downloaded |
+| Code (11 scripts) | 4 commits | ✅ on HF: simonlesaumon/qwen-speedlab |
+
+## Expected GPU performance (once fixed):
+
+| Config | Est. tok/s | VRAM |
+|--------|-----------|------|
+| AWQ 4-bit + FlashInfer + FP8 KV cache | 30-50 | ~18 GB |
+| FP8 + FlashInfer | 25-40 | ~23 GB |
+| AWQ 4-bit + speculative (0.5B draft) | 45-70 | ~20 GB |
+| GPTQ 4-bit + FlashInfer | 30-45 | ~18 GB |
+
+## Measured CPU baseline:
+
+| Config | tok/s |
+|--------|-------|
+| llama.cpp 64 threads Q4_K_M | 1.4 tok/s |
+| GPU speedup expected | **23-38x** |

prompts/bench_prompts.jsonl

@@ -0,0 +1,30 @@
+{"id": "short_01", "category": "short", "prompt": "Explique le théorème de Pythagore en 2 phrases.", "max_tokens": 100}
+{"id": "short_02", "category": "short", "prompt": "Quelle est la capitale du Japon ? Réponds en une phrase.", "max_tokens": 50}
+{"id": "short_03", "category": "short", "prompt": "Donne la formule chimique de l'eau et décris-la brièvement.", "max_tokens": 80}
+{"id": "short_04", "category": "short", "prompt": "Traduis en français : 'The weather is nice today.'", "max_tokens": 60}
+{"id": "short_05", "category": "short", "prompt": "Écris une fonction Python qui calcule la factorielle d'un nombre n.", "max_tokens": 150}
+{"id": "short_06", "category": "short", "prompt": "Qu'est-ce qu'une API REST ? Réponse courte.", "max_tokens": 100}
+{"id": "short_07", "category": "short", "prompt": "Corrige cette phrase : 'Je suis aller au cinéma hier.'", "max_tokens": 100}
+{"id": "short_08", "category": "short", "prompt": "Liste 3 avantages de l'énergie solaire.", "max_tokens": 120}
+{"id": "short_09", "category": "short", "prompt": "Quel est le plus grand océan du monde ?", "max_tokens": 50}
+{"id": "short_10", "category": "short", "prompt": "Écris un haïku sur le printemps.", "max_tokens": 80}
+{"id": "medium_01", "category": "medium", "prompt": "Explique la différence entre TCP et UDP. Donne des exemples d'utilisation pour chaque protocole.", "max_tokens": 300}
+{"id": "medium_02", "category": "medium", "prompt": "Résume le fonctionnement d'un transformer (architecture deep learning) en 5 points clés.", "max_tokens": 350}
+{"id": "medium_03", "category": "medium", "prompt": "Écris une fonction Python qui implémente un tri à bulles avec des commentaires expliquant chaque étape.", "max_tokens": 400}
+{"id": "medium_04", "category": "medium", "prompt": "Compare React et Vue.js pour un développeur débutant : courbe d'apprentissage, écosystème, performance.", "max_tokens": 350}
+{"id": "medium_05", "category": "medium", "prompt": "Explique le concept de garbage collection en programmation. Quels sont les algorithmes principaux ?", "max_tokens": 400}
+{"id": "medium_06", "category": "medium", "prompt": "Décris le cycle de vie d'une requête HTTP : de la saisie de l'URL dans le navigateur jusqu'à l'affichage de la page.", "max_tokens": 400}
+{"id": "medium_07", "category": "medium", "prompt": "Qu'est-ce que le surentraînement (overfitting) en machine learning ? Comment le détecter et l'éviter ?", "max_tokens": 350}
+{"id": "medium_08", "category": "medium", "prompt": "Écris un script bash qui surveille l'utilisation CPU et mémoire toutes les 5 secondes et log les résultats.", "max_tokens": 400}
+{"id": "medium_09", "category": "medium", "prompt": "Explique pourquoi Kubernetes est important pour le déploiement d'applications modernes. Donne 3 cas d'usage concrets.", "max_tokens": 350}
+{"id": "medium_10", "category": "medium", "prompt": "Décris la différence entre SQL et NoSQL. Quand choisir l'un ou l'autre ? Donne des exemples de bases de données.", "max_tokens": 400}
+{"id": "long_01", "category": "long", "prompt": "Tu es un architecte logiciel senior. Concevoir un système de messagerie instantanée scalable (comme WhatsApp). Décris : 1) L'architecture globale 2) Le schéma de base de données 3) La gestion des messages en temps réel 4) La stratégie de scaling 5) Les considérations de sécurité. Sois détaillé et technique.", "max_tokens": 800}
+{"id": "long_02", "category": "long", "prompt": "Rédige une analyse comparative détaillée entre PostgreSQL, MySQL et SQLite. Pour chaque SGBD, couvre : performances, fonctionnalités, cas d'usage idéaux, limites, configuration, et exemples de requêtes avancées.", "max_tokens": 800}
+{"id": "long_03", "category": "long", "prompt": "Explique en détail le processus d'entraînement d'un modèle de langage (LLM) de type GPT : 1) Architecture 2) Préparation des données 3) Pré-entraînement 4) Fine-tuning 5) RLHF 6) Évaluation 7) Déploiement. Donne des chiffres concrets (coûts, temps, données) pour un modèle de 7B paramètres.", "max_tokens": 800}
+{"id": "long_04", "category": "long", "prompt": "Voici un extrait de texte OCR bruité à nettoyer et structurer :\n\n'FACTURE N* 2024-0842\\nDate : 15/01/2024\\n\\nSociété ALPHA SARL - 12 rue de 1''Industrie, 75011 Paris\\nSIRET: 123 456 789 00012\\nTVA: FR12 345 678 900\\n\\nClient:\\nBETA CORP - 45 av des Champs-Elysées, 75008 Paris\\n\\nl Désignation I Qté I Prix unit I Total HT I\\nl-----l-----l-----l-----I\\nl Consultation technique 1 2 I 450,00 € I 900,00 € I\\nl Développement API 1 1 I 2500,00 € I 2500,00 € I\\nl Maintenance serveur 1 3 I 180,00 € I 540,00 € I\\nl-----l-----l-----l-----I\\nl Total HT I 3940,00 € I\\nl TVA 20% I 788,00 € I\\nl Total TTC I 4728,00 € I\\n\\nConditions de paiement : 30 jours\\nIBAN: FR76 1234 5678 9012 3456 7890 123\\n\\nNote: Paiement reçu le 29/01/2024'\n\nNettoie ce texte OCR, structure-le en JSON valide avec tous les champs extraits, et corrige les erreurs de reconnaissance.", "max_tokens": 800}
+{"id": "long_05", "category": "long", "prompt": "Tu es un ingénieur DevOps. Explique en détail comment mettre en place une pipeline CI/CD complète avec GitHub Actions pour une application Docker déployée sur Kubernetes (EKS). Couvre : 1) Structure du repo 2) Build multi-stage Docker 3) Tests automatisés 4) Scan de sécurité 5) Déploiement Canary 6) Monitoring et rollback. Fournis des exemples de fichiers YAML.", "max_tokens": 900}
+{"id": "long_06", "category": "long", "prompt": "Analyse les implications de l'accord Union Européenne sur l'IA Act pour une startup qui développe des modèles de langage. Couvre : 1) Classification des risques 2) Obligations légales 3) Impact sur le développement 4) Transparence et documentation 5) Sanctions potentielles 6) Stratégie de mise en conformité. Sois précis et cite les articles pertinents.", "max_tokens": 800}
+{"id": "long_07", "category": "long", "prompt": "Implémente un système de recommendation de films complet en Python avec : 1) Collaborative filtering 2) Content-based filtering 3) Factorisation de matrices (SVD) 4) Évaluation (RMSE, precision@k, recall@k). Utilise des commentaires détaillés et explique chaque choix algorithmique. Inclus la gestion des cold-start users.", "max_tokens": 900}
+{"id": "long_08", "category": "long", "prompt": "Explique en profondeur le fonctionnement du GPU NVIDIA RTX 3090 (Ampere) : 1) Architecture SM détaillée 2) Hiérarchie mémoire (HBM, L1, L2, registres) 3) Tensor Cores 3e génération 4) RT Cores 5) CUDA programming model 6) Optimisations pour l'inférence LLM 7) Comparaison avec Ada Lovelace (RTX 4090). Donne des chiffres de performance réels.", "max_tokens": 900}
+{"id": "long_09", "category": "long", "prompt": "Tu es un CTO qui doit choisir entre une architecture microservices et monolithique pour une nouvelle application SaaS B2B. Rédige un document de décision architecturale (ADR) complet qui : 1) Définit le contexte métier 2) Évalue les deux options sur 10 critères (scalabilité, complexité, coût, recrutement, déploiement, debug, performance, sécurité, time-to-market, maintenabilité) 3) Propose une recommandation avec justification 4) Décrit une stratégie de migration future si pertinent.", "max_tokens": 900}
+{"id": "long_10", "category": "long", "prompt": "Rédige un tutoriel complet pour déployer un modèle Stable Diffusion XL sur un cluster Kubernetes avec GPU : 1) Prérequis hardware 2) Installation NVIDIA GPU Operator 3) Configuration PersistentVolume pour les modèles 4) Déploiement avec Helm 5) Exposition via API REST avec FastAPI 6) Autoscaling basé sur la queue de requêtes 7) Monitoring GPU avec Prometheus/Grafana 8) Optimisation des coûts avec spot instances. Inclus tous les fichiers YAML et Dockerfile nécessaires.", "max_tokens": 1000}

reports/results.csv

@@ -0,0 +1 @@
+prompt_id,category,model,backend,config_name,max_model_len,gpu_memory_util,dtype,quantization,input_tokens,output_tokens,ttft_ms,tpot_ms,total_time_s,output_tps,total_tps,success,error,vram_peak_mb,gpu_util_pct,power_w,gpu_temp_c,prompt_snippet

reports/summary.md

@@ -0,0 +1,39 @@
+# Qwen SpeedLab — Benchmark Summary
+
+**Date:** (run sweep to populate)
+**GPU:** NVIDIA GeForce RTX 3090 (24 Go)
+**Configs tested:** 0
+
+---
+
+## 🏆 Best Configuration
+
+*Run `python scripts/sweep_vllm_configs.py` to populate this report.*
+
+---
+
+## 📊 All Configurations
+
+| # | Config | Model | Len | Mem% | Seqs | TPS out | TTFT ms | VRAM MiB | Status |
+|---|--------|-------|-----|------|------|---------|---------|----------|--------|
+| - | - | - | - | - | - | - | - | - | - |
+
+---
+
+## 💡 Recommendations for RTX 3090
+
+1. **Model choice**: FP8 offers best quality/speed balance. AWQ provides more VRAM headroom.
+2. **Context length**: 8192 tokens is the sweet spot for 24GB.
+3. **GPU memory utilization**: 0.90 maximum.
+4. **Concurrency**: max_num_seqs ≤ 4.
+5. **Prefix caching**: Always enable.
+6. **Dtype**: Use `auto` for quantized models.
+
+---
+
+## 📁 Raw Data
+
+Full per-request results: [`reports/results.csv`](results.csv)
+
+---
+*Generated by sweep_vllm_configs.py*

requirements.txt

@@ -0,0 +1,36 @@
+# Qwen SpeedLab — RTX 3090 dependencies
+# ============================================================
+
+# Core ML
+torch>=2.4.0
+transformers>=4.46.0
+accelerate>=0.34.0
+
+# Serving engines
+vllm>=0.6.4
+sglang[all]>=0.3.0
+
+# FlashInfer — faster attention on Ampere (SM 8.6)
+# Install via: pip install flashinfer -f https://flashinfer.ai/whl/cu124/torch2.4/
+# or: pip install flashinfer-python
+
+# Benchmark & data
+datasets>=2.21.0
+pandas>=2.2.0
+tqdm>=4.66.0
+psutil>=5.9.0
+
+# API client
+openai>=1.50.0
+httpx>=0.27.0
+aiohttp>=3.10.0
+
+# GPU monitoring
+nvidia-ml-py>=12.560.0
+
+# Hugging Face Hub
+huggingface_hub>=0.24.0
+
+# Utilities
+pyyaml>=6.0
+rich>=13.0.0

scripts/bench_openai_api.py

@@ -0,0 +1,568 @@
+#!/usr/bin/env python3
+"""
+bench_openai_api.py — Benchmark LLM via API OpenAI-compatible (vLLM / SGLang).
+
+Usage:
+    python scripts/bench_openai_api.py [--url http://localhost:8000] [--prompts prompts/bench_prompts.jsonl]
+
+Measures:
+    - Output tokens/s
+    - Total tokens/s (input + output)
+    - Time To First Token (TTFT) in ms
+    - Time Per Output Token (TPOT) in ms
+    - Total time, input tokens, output tokens
+    - Success/failure per prompt
+    - GPU stats if available (via nvidia-ml-py)
+
+Saves results to reports/results.csv
+"""
+from __future__ import annotations
+
+import argparse
+import csv
+import json
+import os
+import sys
+import time
+from dataclasses import dataclass, field, asdict
+from pathlib import Path
+from typing import Optional
+
+import httpx
+
+# --- GPU monitoring (optional) ---
+try:
+    import pynvml
+    HAS_PYNVML = True
+except ImportError:
+    HAS_PYNVML = False
+
+
+# ============================================================
+# Data structures
+# ============================================================
+
+@dataclass
+class BenchResult:
+    prompt_id: str
+    category: str
+    model: str = ""
+    backend: str = ""
+    config_name: str = ""
+    max_model_len: int = 0
+    gpu_memory_util: float = 0.0
+    dtype: str = ""
+    quantization: str = ""
+
+    # Tokens
+    input_tokens: int = 0
+    output_tokens: int = 0
+
+    # Timing
+    ttft_ms: float = 0.0          # Time To First Token
+    tpot_ms: float = 0.0          # Time Per Output Token (avg inter-token)
+    total_time_s: float = 0.0
+    output_tps: float = 0.0       # output tokens / total_time
+    total_tps: float = 0.0        # (input + output) / total_time
+
+    # Quality
+    success: bool = True
+    error: str = ""
+
+    # GPU stats
+    vram_peak_mb: float = 0.0
+    gpu_util_pct: float = 0.0
+    power_w: float = 0.0
+    gpu_temp_c: float = 0.0
+
+    # Prompt snippet (first 100 chars)
+    prompt_snippet: str = ""
+
+
+# ============================================================
+# GPU Monitor
+# ============================================================
+
+class GpuMonitor:
+    """Poll nvidia-smi or pynvml during benchmark."""
+
+    def __init__(self, use_pynvml: bool = True):
+        self.use_pynvml = use_pynvml and HAS_PYNVML
+        self._peak_vram = 0.0
+        self._gpu_utils: list[float] = []
+        self._powers: list[float] = []
+        self._temps: list[float] = []
+        self._running = False
+
+        if self.use_pynvml:
+            pynvml.nvmlInit()
+            self._handle = pynvml.nvmlDeviceGetHandleByIndex(0)
+
+    def start(self):
+        self._running = True
+
+    def stop(self):
+        self._running = False
+        if self.use_pynvml:
+            try:
+                pynvml.nvmlShutdown()
+            except Exception:
+                pass
+
+    def poll(self):
+        """Call this periodically from another thread or between measurements."""
+        if self.use_pynvml:
+            try:
+                info = pynvml.nvmlDeviceGetMemoryInfo(self._handle)
+                vram_mb = info.used / (1024 * 1024)
+                self._peak_vram = max(self._peak_vram, vram_mb)
+
+                util = pynvml.nvmlDeviceGetUtilizationRates(self._handle)
+                self._gpu_utils.append(util.gpu)
+                self._temps.append(util.temperature if hasattr(util, 'temperature') else 0)
+
+                # Power is slower to read, do it less often
+                try:
+                    power_mw = pynvml.nvmlDeviceGetPowerUsage(self._handle)
+                    self._powers.append(power_mw / 1000.0)
+                except Exception:
+                    pass
+            except Exception:
+                pass
+
+    @property
+    def stats(self) -> dict:
+        return {
+            "vram_peak_mb": round(self._peak_vram, 1),
+            "gpu_util_pct": round(sum(self._gpu_utils) / len(self._gpu_utils), 1) if self._gpu_utils else 0.0,
+            "power_w": round(sum(self._powers) / len(self._powers), 1) if self._powers else 0.0,
+            "gpu_temp_c": round(sum(self._temps) / len(self._temps), 1) if self._temps else 0.0,
+        }
+
+
+# ============================================================
+# API Client
+# ============================================================
+
+class OpenAIChatClient:
+    """Minimal async client for OpenAI-compatible chat completions."""
+
+    def __init__(self, base_url: str, api_key: str = "not-needed", timeout: float = 300.0):
+        self.base_url = base_url.rstrip("/")
+        self.api_key = api_key
+        self.timeout = timeout
+        self._client: Optional[httpx.Client] = None
+
+    @property
+    def client(self) -> httpx.Client:
+        if self._client is None:
+            self._client = httpx.Client(
+                base_url=self.base_url,
+                timeout=httpx.Timeout(self.timeout),
+                headers={
+                    "Authorization": f"Bearer {self.api_key}",
+                    "Content-Type": "application/json",
+                },
+            )
+        return self._client
+
+    def check_health(self) -> tuple[bool, str]:
+        """Check if the server is ready via /v1/models."""
+        try:
+            resp = self.client.get("/v1/models")
+            if resp.status_code == 200:
+                data = resp.json()
+                models = [m.get("id", "") for m in data.get("data", [])]
+                return True, f"OK — models: {models}"
+            return False, f"HTTP {resp.status_code}: {resp.text[:200]}"
+        except Exception as e:
+            return False, str(e)
+
+    def chat_completion(
+        self,
+        prompt: str,
+        max_tokens: int = 256,
+        temperature: float = 0.0,
+        stream: bool = True,
+    ) -> tuple[dict, float, list[float]]:
+        """
+        Send a chat completion request. Returns:
+            (completion_dict, ttft_seconds, inter_token_times)
+        If stream=True, measures TTFT and inter-token latencies.
+        """
+        payload = {
+            "model": "default",  # vLLM/SGLang ignore this when only one model loaded
+            "messages": [{"role": "user", "content": prompt}],
+            "max_tokens": max_tokens,
+            "temperature": temperature,
+            "stream": stream,
+        }
+
+        if not stream:
+            t0 = time.perf_counter()
+            resp = self.client.post("/v1/chat/completions", json=payload)
+            elapsed = time.perf_counter() - t0
+            if resp.status_code != 200:
+                raise RuntimeError(f"API error {resp.status_code}: {resp.text[:500]}")
+            data = resp.json()
+            choice = data["choices"][0]
+            usage = data.get("usage", {})
+            return {
+                "content": choice["message"]["content"],
+                "input_tokens": usage.get("prompt_tokens", 0),
+                "output_tokens": usage.get("completion_tokens", 0),
+                "finish_reason": choice.get("finish_reason", "unknown"),
+            }, elapsed, []
+
+        # --- Streaming mode ---
+        t0 = time.perf_counter()
+        ttft: Optional[float] = None
+        inter_times: list[float] = []
+        last_token_time: Optional[float] = None
+        content_parts: list[str] = []
+        usage_info: dict = {}
+        finish_reason = "unknown"
+
+        with self.client.stream("POST", "/v1/chat/completions", json=payload) as resp:
+            if resp.status_code != 200:
+                raise RuntimeError(f"API error {resp.status_code}: {resp.read().decode()[:500]}")
+
+            for line in resp.iter_lines():
+                if not line or not line.startswith("data: "):
+                    continue
+                data_str = line[6:]  # strip "data: "
+                if data_str.strip() == "[DONE]":
+                    break
+
+                try:
+                    chunk = json.loads(data_str)
+                except json.JSONDecodeError:
+                    continue
+
+                now = time.perf_counter()
+
+                # First token
+                if ttft is None:
+                    ttft = now - t0
+                else:
+                    if last_token_time is not None:
+                        inter_times.append(now - last_token_time)
+                last_token_time = now
+
+                choices = chunk.get("choices", [])
+                if choices:
+                    delta = choices[0].get("delta", {})
+                    content = delta.get("content", "")
+                    if content:
+                        content_parts.append(content)
+                    if choices[0].get("finish_reason"):
+                        finish_reason = choices[0]["finish_reason"]
+
+                # Usage sometimes in final chunk
+                if "usage" in chunk and chunk["usage"]:
+                    usage_info = chunk["usage"]
+
+        elapsed = time.perf_counter() - t0
+        ttft_s = ttft if ttft is not None else elapsed
+
+        return {
+            "content": "".join(content_parts),
+            "input_tokens": usage_info.get("prompt_tokens", 0),
+            "output_tokens": usage_info.get("completion_tokens", len(content_parts)),
+            "finish_reason": finish_reason,
+        }, ttft_s, inter_times
+
+
+# ============================================================
+# Benchmark runner
+# ============================================================
+
+def load_prompts(prompts_path: str) -> list[dict]:
+    """Load prompts from JSONL file."""
+    prompts = []
+    with open(prompts_path, "r", encoding="utf-8") as f:
+        for line in f:
+            line = line.strip()
+            if line:
+                prompts.append(json.loads(line))
+    return prompts
+
+
+def run_benchmark(
+    client: OpenAIChatClient,
+    prompts: list[dict],
+    gpu_monitor: Optional[GpuMonitor],
+    config_info: dict,
+    repeat: int = 1,
+    warmup: bool = True,
+) -> list[BenchResult]:
+    """Run benchmark on all prompts and return results."""
+    results: list[BenchResult] = []
+
+    # Warmup
+    if warmup:
+        print("  🔥 Warmup request...")
+        try:
+            client.chat_completion("Hello, respond with 'OK'.", max_tokens=10, stream=True)
+            print("  ✅ Warmup OK")
+        except Exception as e:
+            print(f"  ⚠️  Warmup failed: {e}")
+
+    total = len(prompts) * repeat
+
+    for rep in range(repeat):
+        for i, p in enumerate(prompts, 1):
+            prompt_text = p["prompt"]
+            prompt_id = f"{p['id']}_r{rep}" if repeat > 1 else p["id"]
+            max_tokens = p.get("max_tokens", 256)
+
+            result = BenchResult(
+                prompt_id=prompt_id,
+                category=p.get("category", "unknown"),
+                prompt_snippet=prompt_text[:100].replace("\n", " "),
+                **config_info,
+            )
+
+            print(f"  [{i}/{total}] {prompt_id} ({result.category})...", end=" ", flush=True)
+
+            if gpu_monitor:
+                gpu_monitor.poll()
+
+            try:
+                completion, ttft_s, inter_times = client.chat_completion(
+                    prompt_text,
+                    max_tokens=max_tokens,
+                    temperature=0.0,
+                    stream=True,
+                )
+                elapsed = time.perf_counter()  # total time will be updated below
+                # We need the actual total time from the request
+                # Re-run with timing — use the completion struct
+
+                result.input_tokens = completion.get("input_tokens", 0)
+                result.output_tokens = completion.get("output_tokens", 0)
+                result.ttft_ms = ttft_s * 1000
+
+                if inter_times:
+                    result.tpot_ms = (sum(inter_times) / len(inter_times)) * 1000
+
+                # For total time, we use ttft + sum of inter_times + small overhead
+                total_stream_time = ttft_s + sum(inter_times) if inter_times else ttft_s
+                result.total_time_s = total_stream_time
+
+                if result.total_time_s > 0:
+                    result.output_tps = result.output_tokens / result.total_time_s
+                    total_tokens = result.input_tokens + result.output_tokens
+                    result.total_tps = total_tokens / result.total_time_s
+
+                if gpu_monitor:
+                    gpu_monitor.poll()
+                    gpu_stats = gpu_monitor.stats
+                    result.vram_peak_mb = gpu_stats["vram_peak_mb"]
+                    result.gpu_util_pct = gpu_stats["gpu_util_pct"]
+                    result.power_w = gpu_stats["power_w"]
+                    result.gpu_temp_c = gpu_stats["gpu_temp_c"]
+
+                print(f"✅ {result.output_tps:.1f} tok/s (out={result.output_tokens}, TTFT={result.ttft_ms:.0f}ms)")
+
+            except Exception as e:
+                result.success = False
+                result.error = str(e)[:500]
+                print(f"❌ {str(e)[:100]}")
+
+            results.append(result)
+
+    return results
+
+
+# ============================================================
+# CSV persistence
+# ============================================================
+
+CSV_COLUMNS = [
+    "prompt_id", "category", "model", "backend", "config_name",
+    "max_model_len", "gpu_memory_util", "dtype", "quantization",
+    "input_tokens", "output_tokens",
+    "ttft_ms", "tpot_ms", "total_time_s", "output_tps", "total_tps",
+    "success", "error",
+    "vram_peak_mb", "gpu_util_pct", "power_w", "gpu_temp_c",
+    "prompt_snippet",
+]
+
+
+def save_results(results: list[BenchResult], csv_path: str):
+    """Append results to CSV file."""
+    file_exists = os.path.exists(csv_path)
+    os.makedirs(os.path.dirname(csv_path), exist_ok=True)
+
+    with open(csv_path, "a", newline="", encoding="utf-8") as f:
+        writer = csv.DictWriter(f, fieldnames=CSV_COLUMNS, extrasaction='ignore')
+        if not file_exists or os.path.getsize(csv_path) == 0:
+            writer.writeheader()
+        for r in results:
+            writer.writerow(asdict(r))
+
+    print(f"\n📊 {len(results)} results saved to {csv_path}")
+
+
+def print_summary(results: list[BenchResult]):
+    """Print a summary table."""
+    successful = [r for r in results if r.success]
+    failed = [r for r in results if not r.success]
+
+    print("\n" + "=" * 60)
+    print(" BENCHMARK SUMMARY")
+    print("=" * 60)
+    print(f"  Total requests:  {len(results)}")
+    print(f"  Successful:      {len(successful)}")
+    print(f"  Failed:          {len(failed)}")
+    print()
+
+    if successful:
+        output_tps_vals = [r.output_tps for r in successful]
+        total_tps_vals = [r.total_tps for r in successful]
+        ttft_vals = [r.ttft_ms for r in successful]
+        tpot_vals = [r.tpot_ms for r in successful if r.tpot_ms > 0]
+
+        print(f"  Output TPS:      {min(output_tps_vals):.1f} / {sum(output_tps_vals)/len(output_tps_vals):.1f} / {max(output_tps_vals):.1f}  (min/avg/max)")
+        print(f"  Total TPS:       {min(total_tps_vals):.1f} / {sum(total_tps_vals)/len(total_tps_vals):.1f} / {max(total_tps_vals):.1f}  (min/avg/max)")
+        print(f"  TTFT (ms):       {min(ttft_vals):.0f} / {sum(ttft_vals)/len(ttft_vals):.0f} / {max(ttft_vals):.0f}  (min/avg/max)")
+        if tpot_vals:
+            print(f"  TPOT (ms):       {min(tpot_vals):.1f} / {sum(tpot_vals)/len(tpot_vals):.1f} / {max(tpot_vals):.1f}  (min/avg/max)")
+
+        # Per category
+        print()
+        for cat in sorted(set(r.category for r in successful)):
+            cat_results = [r for r in successful if r.category == cat]
+            avg_tps = sum(r.output_tps for r in cat_results) / len(cat_results)
+            avg_ttft = sum(r.ttft_ms for r in cat_results) / len(cat_results)
+            print(f"  {cat:10s}: avg {avg_tps:.1f} tok/s, TTFT {avg_ttft:.0f}ms  ({len(cat_results)} prompts)")
+
+    if failed:
+        print(f"\n  ❌ Failed requests:")
+        for r in failed:
+            print(f"     - {r.prompt_id}: {r.error[:120]}")
+
+    # GPU stats
+    gpu_results = [r for r in successful if r.vram_peak_mb > 0]
+    if gpu_results:
+        peak_vram = max(r.vram_peak_mb for r in gpu_results)
+        avg_util = sum(r.gpu_util_pct for r in gpu_results) / len(gpu_results)
+        avg_power = sum(r.power_w for r in gpu_results) / len(gpu_results)
+        print(f"\n  GPU Stats:")
+        print(f"  VRAM peak:       {peak_vram:.0f} MiB / 24576 MiB")
+        print(f"  GPU util avg:    {avg_util:.1f}%")
+        print(f"  Power avg:       {avg_power:.1f} W")
+
+    print("=" * 60)
+
+
+# ============================================================
+# CLI
+# ============================================================
+
+def main():
+    parser = argparse.ArgumentParser(description="Benchmark LLM via OpenAI-compatible API")
+    parser.add_argument("--url", default="http://localhost:8000", help="API base URL")
+    parser.add_argument("--prompts", default=None, help="Path to JSONL prompts file")
+    parser.add_argument("--output", default=None, help="Output CSV path")
+    parser.add_argument("--repeat", type=int, default=1, help="Repeat each prompt N times")
+    parser.add_argument("--no-warmup", action="store_true", help="Skip warmup request")
+    parser.add_argument("--no-gpu", action="store_true", help="Disable GPU monitoring")
+    parser.add_argument("--timeout", type=float, default=300.0, help="Request timeout (seconds)")
+    parser.add_argument("--model", default="", help="Model name (metadata only)")
+    parser.add_argument("--backend", default="vllm", help="Backend name (vllm/sglang, metadata only)")
+    parser.add_argument("--config", default="", help="Config name (metadata only)")
+    parser.add_argument("--max-model-len", type=int, default=0, help="Max model len (metadata)")
+    parser.add_argument("--gpu-mem-util", type=float, default=0.0, help="GPU memory util (metadata)")
+    parser.add_argument("--dtype", default="", help="Dtype (metadata)")
+    parser.add_argument("--quantization", default="", help="Quantization (metadata)")
+
+    args = parser.parse_args()
+
+    # Default paths relative to project
+    project_dir = Path(__file__).resolve().parent.parent
+    if args.prompts is None:
+        args.prompts = str(project_dir / "prompts" / "bench_prompts.jsonl")
+    if args.output is None:
+        args.output = str(project_dir / "reports" / "results.csv")
+
+    # Config info for metadata
+    config_info = {
+        "model": args.model,
+        "backend": args.backend,
+        "config_name": args.config,
+        "max_model_len": args.max_model_len,
+        "gpu_memory_util": args.gpu_mem_util,
+        "dtype": args.dtype,
+        "quantization": args.quantization,
+    }
+
+    print("=" * 60)
+    print(" Qwen SpeedLab — OpenAI API Benchmark")
+    print("=" * 60)
+    print(f"  URL:      {args.url}")
+    print(f"  Prompts:  {args.prompts}")
+    print(f"  Output:   {args.output}")
+    print(f"  Repeat:   {args.repeat}")
+    print(f"  Timeout:  {args.timeout}s")
+    print(f"  Config:   {config_info}")
+    print("=" * 60)
+    print()
+
+    # Load prompts
+    if not os.path.exists(args.prompts):
+        print(f"❌ Prompts file not found: {args.prompts}")
+        sys.exit(1)
+
+    prompts = load_prompts(args.prompts)
+    print(f"📄 Loaded {len(prompts)} prompts")
+
+    # Create API client
+    client = OpenAIChatClient(base_url=args.url, timeout=args.timeout)
+
+    # Health check
+    print("\n🔍 Checking server health...")
+    ok, msg = client.check_health()
+    if not ok:
+        print(f"❌ Server not ready: {msg}")
+        print("   Is the server running? Start with: bash scripts/serve_vllm.sh")
+        sys.exit(1)
+    print(f"✅ Server ready: {msg}")
+
+    # GPU monitor
+    gpu_monitor = None
+    if not args.no_gpu:
+        gpu_monitor = GpuMonitor(use_pynvml=True)
+        gpu_monitor.start()
+        print("📊 GPU monitoring enabled (pynvml)")
+
+    # Run benchmark
+    print(f"\n🚀 Running benchmark ({len(prompts) * args.repeat} requests)...\n")
+    t_start = time.perf_counter()
+    results = run_benchmark(
+        client=client,
+        prompts=prompts,
+        gpu_monitor=gpu_monitor,
+        config_info=config_info,
+        repeat=args.repeat,
+        warmup=not args.no_warmup,
+    )
+    total_elapsed = time.perf_counter() - t_start
+
+    if gpu_monitor:
+        gpu_monitor.stop()
+
+    # Save
+    save_results(results, args.output)
+
+    # Summary
+    print_summary(results)
+    print(f"\n⏱️  Total benchmark time: {total_elapsed:.1f}s")
+
+    # Close client
+    client.client.close()
+
+
+if __name__ == "__main__":
+    main()

scripts/bench_vllm.py

@@ -0,0 +1,381 @@
+#!/usr/bin/env python3
+"""
+bench_vllm.py — Benchmark vLLM directly via its Python API (LLM class).
+
+This bypasses the HTTP server and measures:
+    - Prefill throughput (tokens/s)
+    - Decode throughput (tokens/s)
+    - End-to-end latency
+    - Peak VRAM usage
+
+Usage:
+    python scripts/bench_vllm.py --model Qwen/Qwen3.6-27B-FP8 --max-model-len 8192
+
+Note:
+    Requires vLLM installed. This script loads the model in-process,
+    so it will consume VRAM immediately. Make sure no other process
+    is using the GPU.
+
+    For HTTP-based benchmarking (against a running server), use
+    bench_openai_api.py instead.
+"""
+from __future__ import annotations
+
+import argparse
+import csv
+import json
+import os
+import sys
+import time
+from dataclasses import dataclass, asdict
+from pathlib import Path
+from typing import Optional
+
+# GPU monitoring
+try:
+    import pynvml
+    HAS_PYNVML = True
+except ImportError:
+    HAS_PYNVML = False
+
+
+# ============================================================
+# Data structures
+# ============================================================
+
+@dataclass
+class DirectBenchResult:
+    prompt_id: str
+    category: str
+    model: str
+    config_name: str
+    max_model_len: int
+    gpu_memory_util: float
+    dtype: str
+    quantization: str
+    input_tokens: int
+    output_tokens: int
+    prefill_time_s: float     # Time to process input (first token)
+    decode_time_s: float      # Time to generate output tokens
+    total_time_s: float
+    prefill_tps: float        # input tokens / prefill_time
+    decode_tps: float         # output tokens / decode_time
+    output_tps: float         # output tokens / total_time
+    vram_peak_mb: float
+    gpu_util_pct: float
+    success: bool
+    error: str
+
+
+# ============================================================
+# GPU Monitor
+# ============================================================
+
+class GpuMonitor:
+    def __init__(self):
+        self.use_pynvml = HAS_PYNVML
+        self._peak_vram = 0.0
+        self._gpu_utils: list[float] = []
+
+        if self.use_pynvml:
+            pynvml.nvmlInit()
+            self._handle = pynvml.nvmlDeviceGetHandleByIndex(0)
+
+    def poll(self):
+        if self.use_pynvml:
+            try:
+                info = pynvml.nvmlDeviceGetMemoryInfo(self._handle)
+                vram_mb = info.used / (1024 * 1024)
+                self._peak_vram = max(self._peak_vram, vram_mb)
+                util = pynvml.nvmlDeviceGetUtilizationRates(self._handle)
+                self._gpu_utils.append(util.gpu)
+            except Exception:
+                pass
+
+    @property
+    def stats(self) -> dict:
+        return {
+            "vram_peak_mb": round(self._peak_vram, 1),
+            "gpu_util_pct": round(sum(self._gpu_utils) / len(self._gpu_utils), 1) if self._gpu_utils else 0.0,
+        }
+
+    def close(self):
+        if self.use_pynvml:
+            try:
+                pynvml.nvmlShutdown()
+            except Exception:
+                pass
+
+
+# ============================================================
+# Benchmark runner
+# ============================================================
+
+def load_prompts(prompts_path: str) -> list[dict]:
+    with open(prompts_path, "r", encoding="utf-8") as f:
+        return [json.loads(line) for line in f if line.strip()]
+
+
+def run_direct_benchmark(
+    llm,                  # vLLM LLM instance
+    tokenizer,            # Tokenizer
+    prompts: list[dict],
+    gpu_monitor: GpuMonitor,
+    config_info: dict,
+    warmup: bool = True,
+) -> list[DirectBenchResult]:
+    """Run direct vLLM benchmark (no HTTP)."""
+    results: list[DirectBenchResult] = []
+
+    # Warmup
+    if warmup:
+        print("  🔥 Warmup...")
+        sampling_params = llm.__class__.__module__.split(".")[0]  # won't work, use hardcoded import
+        try:
+            from vllm import SamplingParams
+            llm.generate(["Hello"], SamplingParams(max_tokens=10, temperature=0))
+            print("  ✅ Warmup OK")
+        except Exception as e:
+            print(f"  ⚠️  Warmup failed: {e}")
+
+    from vllm import SamplingParams
+
+    for i, p in enumerate(prompts, 1):
+        prompt_text = p["prompt"]
+        prompt_id = p["id"]
+        max_tokens = p.get("max_tokens", 256)
+
+        result = DirectBenchResult(
+            prompt_id=prompt_id,
+            category=p.get("category", "unknown"),
+            **config_info,
+        )
+
+        print(f"  [{i}/{len(prompts)}] {prompt_id} ({result.category})...", end=" ", flush=True)
+
+        gpu_monitor.poll()
+
+        try:
+            # Tokenize to get input count
+            input_ids = tokenizer.encode(prompt_text)
+            result.input_tokens = len(input_ids)
+
+            sampling_params = SamplingParams(
+                max_tokens=max_tokens,
+                temperature=0.0,
+                ignore_eos=False,
+            )
+
+            t0 = time.perf_counter()
+            outputs = llm.generate([prompt_text], sampling_params, use_tqdm=False)
+            elapsed = time.perf_counter() - t0
+
+            gpu_monitor.poll()
+
+            if outputs:
+                out = outputs[0]
+                result.output_tokens = len(out.outputs[0].token_ids)
+
+                # vLLM provides metrics on the output
+                metrics = out.metrics
+                if hasattr(metrics, 'first_token_time') and metrics.first_token_time is not None:
+                    # first_token_time is relative to arrival time
+                    result.prefill_time_s = metrics.first_token_time - getattr(metrics, 'arrival_time', 0)
+                else:
+                    # Estimate: use a fraction of total time based on input/output ratio
+                    result.prefill_time_s = elapsed * 0.3  # rough heuristic
+
+                result.decode_time_s = elapsed - result.prefill_time_s
+                result.total_time_s = elapsed
+
+                if result.prefill_time_s > 0:
+                    result.prefill_tps = result.input_tokens / result.prefill_time_s
+                if result.decode_time_s > 0:
+                    result.decode_tps = result.output_tokens / result.decode_time_s
+                if elapsed > 0:
+                    result.output_tps = result.output_tokens / elapsed
+
+                print(f"✅ {result.output_tps:.1f} tok/s (prefill={result.prefill_tps:.0f}, decode={result.decode_tps:.0f})")
+
+            else:
+                result.success = False
+                result.error = "No output generated"
+                print("❌ No output")
+
+        except Exception as e:
+            result.success = False
+            result.error = str(e)[:500]
+            print(f"❌ {str(e)[:100]}")
+
+        results.append(result)
+
+    return results
+
+
+# ============================================================
+# CSV
+# ============================================================
+
+DIRECT_CSV_COLUMNS = [
+    "prompt_id", "category", "model", "config_name",
+    "max_model_len", "gpu_memory_util", "dtype", "quantization",
+    "input_tokens", "output_tokens",
+    "prefill_time_s", "decode_time_s", "total_time_s",
+    "prefill_tps", "decode_tps", "output_tps",
+    "vram_peak_mb", "gpu_util_pct",
+    "success", "error",
+]
+
+
+def save_results(results: list[DirectBenchResult], csv_path: str):
+    file_exists = os.path.exists(csv_path)
+    os.makedirs(os.path.dirname(csv_path), exist_ok=True)
+    with open(csv_path, "a", newline="", encoding="utf-8") as f:
+        writer = csv.DictWriter(f, fieldnames=DIRECT_CSV_COLUMNS, extrasaction='ignore')
+        if not file_exists or os.path.getsize(csv_path) == 0:
+            writer.writeheader()
+        for r in results:
+            writer.writerow(asdict(r))
+    print(f"\n📊 {len(results)} results saved to {csv_path}")
+
+
+def print_summary(results: list[DirectBenchResult]):
+    successful = [r for r in results if r.success]
+    if not successful:
+        print("\n❌ No successful results")
+        return
+
+    decode_tps_vals = [r.decode_tps for r in successful if r.decode_tps > 0]
+    output_tps_vals = [r.output_tps for r in successful]
+
+    print("\n" + "=" * 60)
+    print(" DIRECT VLLM BENCHMARK SUMMARY")
+    print("=" * 60)
+    print(f"  Successful: {len(successful)}/{len(results)}")
+    if decode_tps_vals:
+        print(f"  Decode TPS:  {min(decode_tps_vals):.1f} / {sum(decode_tps_vals)/len(decode_tps_vals):.1f} / {max(decode_tps_vals):.1f}  (min/avg/max)")
+    print(f"  Output TPS:  {min(output_tps_vals):.1f} / {sum(output_tps_vals)/len(output_tps_vals):.1f} / {max(output_tps_vals):.1f}  (min/avg/max)")
+
+    gpu_results = [r for r in successful if r.vram_peak_mb > 0]
+    if gpu_results:
+        peak_vram = max(r.vram_peak_mb for r in gpu_results)
+        print(f"\n  VRAM peak:   {peak_vram:.0f} MiB / 24576 MiB")
+    print("=" * 60)
+
+
+# ============================================================
+# CLI
+# ============================================================
+
+def main():
+    parser = argparse.ArgumentParser(description="Benchmark vLLM directly (in-process)")
+    parser.add_argument("--model", default="Qwen/Qwen3.6-27B-FP8", help="Model name or path")
+    parser.add_argument("--max-model-len", type=int, default=8192, help="Max model length")
+    parser.add_argument("--gpu-memory-utilization", type=float, default=0.90, help="GPU memory utilization")
+    parser.add_argument("--dtype", default="auto", help="Data type (auto, float16, bfloat16)")
+    parser.add_argument("--quantization", default="", help="Quantization method")
+    parser.add_argument("--prompts", default=None, help="JSONL prompts file")
+    parser.add_argument("--output", default=None, help="Output CSV path")
+    parser.add_argument("--config", default="direct", help="Config name for logging")
+    parser.add_argument("--max-num-seqs", type=int, default=4, help="Max number of sequences")
+    parser.add_argument("--trust-remote-code", action="store_true", default=True)
+
+    args = parser.parse_args()
+
+    project_dir = Path(__file__).resolve().parent.parent
+    if args.prompts is None:
+        args.prompts = str(project_dir / "prompts" / "bench_prompts.jsonl")
+    if args.output is None:
+        args.output = str(project_dir / "reports" / "results_direct.csv")
+
+    print("=" * 60)
+    print(" Qwen SpeedLab — Direct vLLM Benchmark")
+    print("=" * 60)
+    print(f"  Model:       {args.model}")
+    print(f"  Max len:     {args.max_model_len}")
+    print(f"  GPU mem:     {args.gpu_memory_utilization}")
+    print(f"  Dtype:       {args.dtype}")
+    print(f"  Prompts:     {args.prompts}")
+    print("=" * 60)
+    print()
+
+    # Lazy import vLLM (heavy)
+    print("📦 Importing vLLM...")
+    try:
+        from vllm import LLM, SamplingParams
+    except ImportError:
+        print("❌ vLLM not installed. Run: pip install vllm")
+        sys.exit(1)
+
+    # Load prompts
+    if not os.path.exists(args.prompts):
+        print(f"❌ Prompts file not found: {args.prompts}")
+        sys.exit(1)
+    prompts = load_prompts(args.prompts)
+    print(f"📄 Loaded {len(prompts)} prompts")
+
+    # GPU monitor
+    gpu_monitor = GpuMonitor()
+    gpu_monitor.poll()
+
+    # Load model
+    print(f"\n🚀 Loading model: {args.model}")
+    print(f"   max_model_len={args.max_model_len}, gpu_memory_utilization={args.gpu_memory_utilization}")
+    t0 = time.perf_counter()
+
+    try:
+        llm = LLM(
+            model=args.model,
+            max_model_len=args.max_model_len,
+            gpu_memory_utilization=args.gpu_memory_utilization,
+            dtype=args.dtype,
+            quantization=args.quantization if args.quantization else None,
+            trust_remote_code=args.trust_remote_code,
+            max_num_seqs=args.max_num_seqs,
+            tensor_parallel_size=1,
+            enable_prefix_caching=True,
+        )
+        load_time = time.perf_counter() - t0
+        print(f"✅ Model loaded in {load_time:.1f}s")
+
+        # Get tokenizer
+        tokenizer = llm.get_tokenizer()
+
+        gpu_monitor.poll()
+
+    except Exception as e:
+        print(f"❌ Failed to load model: {e}")
+        gpu_monitor.close()
+        sys.exit(1)
+
+    # Run benchmark
+    config_info = {
+        "model": args.model,
+        "config_name": args.config,
+        "max_model_len": args.max_model_len,
+        "gpu_memory_util": args.gpu_memory_utilization,
+        "dtype": args.dtype,
+        "quantization": args.quantization,
+    }
+
+    print(f"\n🚀 Running benchmark ({len(prompts)} prompts)...\n")
+    results = run_direct_benchmark(
+        llm=llm,
+        tokenizer=tokenizer,
+        prompts=prompts,
+        gpu_monitor=gpu_monitor,
+        config_info=config_info,
+        warmup=True,
+    )
+
+    gpu_monitor.close()
+
+    # Save
+    save_results(results, args.output)
+
+    # Summary
+    print_summary(results)
+
+
+if __name__ == "__main__":
+    main()

scripts/benchmark_now.sh

@@ -0,0 +1,107 @@
+#!/usr/bin/env bash
+# ============================================================
+# benchmark_now.sh — Run IMMEDIATELY after GPU driver fix
+#
+# This is the definitive benchmark script for Qwen 3.6 27B
+# on RTX 3090 (24 GB). It tests multiple configs and finds
+# the maximum tokens/s without OOM.
+#
+# Prerequisites:
+#   1. GPU CUDA must work (torch.cuda.is_available() == True)
+#   2. HF_TOKEN set if model needs authentication
+#   3. ~50 GB free on /workspace for models
+#
+# Usage:  bash scripts/benchmark_now.sh
+# ============================================================
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PROJECT_DIR="$(dirname "$SCRIPT_DIR")"
+cd "$PROJECT_DIR"
+
+echo "═══════════════════════════════════════════════════════════"
+echo " Qwen SpeedLab — Definitive RTX 3090 Benchmark"
+echo "═══════════════════════════════════════════════════════════"
+echo ""
+
+# --- GPU check ---
+if ! python3 -c "import torch; assert torch.cuda.is_available(), 'CUDA not available'" 2>/dev/null; then
+    echo "❌ CUDA not available. Run fix_gpu_host.sh on the HOST first:"
+    echo "   sudo apt-get remove --purge nvidia-dkms-580-open nvidia-driver-580-open"
+    echo "   sudo apt-get install nvidia-driver-580"
+    echo "   sudo reboot"
+    exit 1
+fi
+
+GPU=$(python3 -c "import torch; print(torch.cuda.get_device_name(0))")
+VRAM=$(python3 -c "import torch; print(f'{torch.cuda.get_device_properties(0).total_memory/1024**3:.1f}')")
+echo "✅ GPU: $GPU ($VRAM GB)"
+echo ""
+
+# --- Phase 1: Model download ---
+echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
+echo " Phase 1/4: Downloading best model for RTX 3090..."
+echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
+
+# Download the best models (AWQ and FP8)
+python3 -c "
+from huggingface_hub import snapshot_download
+import os
+
+# AWQ 4-bit — best for Ampere (RTX 3090)
+print('Downloading QuantTrio/Qwen3.6-27B-AWQ (15 GB)...')
+snapshot_download('QuantTrio/Qwen3.6-27B-AWQ',
+                   cache_dir='/workspace/models',
+                   token=os.environ.get('HF_TOKEN'))
+print('✅ AWQ model ready')
+
+# FP8 — official, higher quality
+print('Downloading Qwen/Qwen3.6-27B-FP8 (27 GB)...')
+snapshot_download('Qwen/Qwen3.6-27B-FP8',
+                   cache_dir='/workspace/models',
+                   token=os.environ.get('HF_TOKEN'))
+print('✅ FP8 model ready')
+"
+
+echo ""
+
+# --- Phase 2: Quick vLLM config sweep ---
+echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
+echo " Phase 2/4: Quick vLLM sweep (6 best configs)..."
+echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
+python3 scripts/sweep_vllm_configs.py --quick --port 8000
+echo ""
+
+# --- Phase 3: Advanced optimization A/B test ---
+echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
+echo " Phase 3/4: Advanced optimization sweep..."
+echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
+python3 scripts/sweep_advanced.py --mode baseline --port 8000
+echo ""
+
+# --- Phase 4: Final report ---
+echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
+echo " Phase 4/4: Final report"
+echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
+echo ""
+echo "📊 Results files:"
+ls -la reports/results.csv reports/results_advanced.csv reports/summary.md reports/summary_advanced.md 2>/dev/null || echo "  (some reports not generated)"
+echo ""
+
+# Print best result if available
+if [[ -f reports/summary.md ]]; then
+    echo "Best results from sweep:"
+    grep -A15 "Best Configuration" reports/summary.md 2>/dev/null | head -20 || true
+fi
+if [[ -f reports/summary_advanced.md ]]; then
+    echo ""
+    echo "Optimization impact analysis:"
+    grep -A5 "Optimization Impact" reports/summary_advanced.md 2>/dev/null | head -10 || true
+fi
+
+echo ""
+echo "═════════════════��═════════════════════════════════════════"
+echo " Benchmark complete!"
+echo " Full report: less reports/summary.md"
+echo " Raw data:    cat reports/results.csv"
+echo "═══════════════════════════════════════════════════════════"

scripts/collect_gpu_stats.sh

@@ -0,0 +1,73 @@
+#!/usr/bin/env bash
+# ============================================================
+# collect_gpu_stats.sh — Continuous GPU monitoring during benchmark
+#
+# Usage:
+#   bash scripts/collect_gpu_stats.sh [output_file] [interval_seconds]
+#
+# Logs nvidia-smi metrics every N seconds to a CSV file.
+# Run in background during benchmark:
+#   bash scripts/collect_gpu_stats.sh /tmp/gpu_stats.csv 1 &
+#   GPU_PID=$!
+#   ... run benchmark ...
+#   kill $GPU_PID
+#
+# Default output: reports/gpu_stats_<timestamp>.csv
+# ============================================================
+set -euo pipefail
+
+OUTPUT="${1:-}"
+INTERVAL="${2:-1}"
+
+# Default output path
+if [[ -z "$OUTPUT" ]]; then
+    SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+    PROJECT_DIR="$(dirname "$SCRIPT_DIR")"
+    TIMESTAMP=$(date +%Y%m%d_%H%M%S)
+    OUTPUT="${PROJECT_DIR}/reports/gpu_stats_${TIMESTAMP}.csv"
+fi
+
+# Ensure output directory exists
+mkdir -p "$(dirname "$OUTPUT")"
+
+echo "============================================"
+echo " GPU Stats Collector"
+echo "============================================"
+echo "  Output:   $OUTPUT"
+echo "  Interval: ${INTERVAL}s"
+echo "  PID:      $$"
+echo "============================================"
+echo ""
+echo "Recording... Press Ctrl+C to stop."
+echo ""
+
+# Write CSV header
+echo "timestamp,elapsed_s,gpu_index,gpu_name,pci_bus,driver_version,"\
+"pstate,temperature_gpu,utilization_gpu,utilization_memory,"\
+"memory_total_mib,memory_used_mib,memory_free_mib,"\
+"power_draw_w,power_limit_w,"\
+"clocks_sm_mhz,clocks_memory_mhz,"\
+"fan_speed_pct,performance_state" > "$OUTPUT"
+
+START_TIME=$(date +%s)
+
+trap 'echo ""; echo "Stopped. Stats saved to: $OUTPUT"; exit 0' INT TERM
+
+while true; do
+    NOW=$(date +%s)
+    ELAPSED=$((NOW - START_TIME))
+    TS=$(date -u +%Y-%m-%dT%H:%M:%SZ)
+
+    # Query nvidia-smi in CSV format (no header)
+    nvidia-smi --query-gpu=index,name,pci.bus_id,driver_version,\
+pstate,temperature.gpu,utilization.gpu,utilization.memory,\
+memory.total,memory.used,memory.free,\
+power.draw,power.limit,\
+clocks.sm,clocks.memory,\
+fan.speed,pstate \
+        --format=csv,noheader,nounits 2>/dev/null | while IFS=',' read -r line; do
+        echo "${TS},${ELAPSED},${line}" | tr -d ' ' >> "$OUTPUT"
+    done
+
+    sleep "$INTERVAL"
+done

scripts/download_best_model.py

@@ -0,0 +1,288 @@
+#!/usr/bin/env python3
+"""
+download_best_model.py — Auto-detect and download the best Qwen 27B quant for RTX 3090.
+
+Checks Hugging Face for available quantized variants, downloads the best one
+that fits in 24GB VRAM, and recommends the optimal vLLM config.
+
+Workflow:
+    1. Query HF Hub API for Qwen 3.6 27B variants
+    2. Filter to those compatible with RTX 3090 (24GB)
+    3. Rank by expected quality/speed ratio
+    4. Download the best one
+    5. Print recommended vLLM flags
+
+Usage:
+    python scripts/download_best_model.py
+    python scripts/download_best_model.py --list       # just list, don't download
+    python scripts/download_best_model.py --model QuantTrio/Qwen3.6-27B-AWQ  # specific
+    python scripts/download_best_model.py --dry-run    # print what would be done
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import sys
+import time
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Optional
+
+try:
+    import httpx
+except ImportError:
+    print("⚠️  httpx not installed. Run: pip install httpx")
+    sys.exit(1)
+
+# Ordered by preference for RTX 3090 (SM 8.6, 24GB)
+MODEL_CANDIDATES = [
+    # Format: (model_id, quant, est_vram_gb, quality_rank, speed_rank, notes)
+    ("QuantTrio/Qwen3.6-27B-AWQ", "awq", 15, 1, 1, "★ Best for Ampere — INT4 tensor cores"),
+    ("groxaxo/Qwen3.6-27B-GPTQ-Pro-4bit", "gptq", 15, 2, 2, "GPTQ Marlin, comparable to AWQ"),
+    ("Qwen/Qwen3.6-27B-FP8", "fp8", 27, 3, 3, "FP8 — tight on 24GB, needs KV cache tuning"),
+    ("Qwen/Qwen3.6-27B", "none", 54, 4, 4, "Base bf16 — won't fit 24GB without offloading"),
+]
+
+# Also check for GGUF variants (community uploads)
+GGUF_CANDIDATES = [
+    ("unsloth/Qwen3.6-27B-GGUF", "gguf", "Q4_K_M"),
+    ("lmstudio-community/Qwen3.6-27B-GGUF", "gguf", "Q5_K_M"),
+    ("bartowski/Qwen3.6-27B-GGUF", "gguf", "IQ4_XS"),
+]
+
+HF_API = "https://huggingface.co/api"
+
+
+@dataclass
+class ModelInfo:
+    model_id: str
+    quantization: str
+    est_vram_gb: float
+    exists_on_hf: bool = False
+    size_on_disk_gb: float = 0.0
+    downloads: int = 0
+    last_modified: str = ""
+
+
+def check_model_exists(model_id: str) -> Optional[dict]:
+    """Check if a model exists on HF Hub and get metadata."""
+    try:
+        resp = httpx.get(f"{HF_API}/models/{model_id}", timeout=15)
+        if resp.status_code == 200:
+            return resp.json()
+        return None
+    except Exception:
+        return None
+
+
+def check_model_siblings(model_id: str) -> list[dict]:
+    """List all files in a model repo."""
+    try:
+        resp = httpx.get(f"{HF_API}/models/{model_id}?expand[]=siblings", timeout=15)
+        if resp.status_code == 200:
+            data = resp.json()
+            return data.get("siblings", [])
+        return []
+    except Exception:
+        return []
+
+
+def estimate_download_size(siblings: list[dict]) -> float:
+    """Estimate total download size in GB."""
+    total = sum(s.get("size", 0) for s in siblings if s.get("size"))
+    return total / (1024**3)
+
+
+def scan_models() -> list[ModelInfo]:
+    """Scan HF Hub for all available Qwen 3.6 27B variants."""
+    models: list[ModelInfo] = []
+    print("🔍 Scanning Hugging Face for Qwen 3.6 27B variants...")
+    print()
+
+    for model_id, quant, vram, quality, speed, notes in MODEL_CANDIDATES:
+        info = ModelInfo(model_id=model_id, quantization=quant, est_vram_gb=vram)
+        meta = check_model_exists(model_id)
+
+        if meta:
+            info.exists_on_hf = True
+            info.downloads = meta.get("downloads", 0) or 0
+            info.last_modified = meta.get("lastModified", "")[:10]
+
+            siblings = check_model_siblings(model_id)
+            info.size_on_disk_gb = estimate_download_size(siblings)
+
+            status = "✅"
+        else:
+            status = "❌"
+
+        print(f"  {status} {model_id}")
+        print(f"     Quant: {quant} | VRAM: ~{vram} GB | Disk: {info.size_on_disk_gb:.1f} GB")
+        print(f"     {notes}")
+        if meta:
+            print(f"     Downloads: {info.downloads:,} | Updated: {info.last_modified}")
+        print()
+
+        models.append(info)
+
+    # Also check GGUF variants
+    for model_id, quant, gguf_quant in GGUF_CANDIDATES:
+        meta = check_model_exists(model_id)
+        if meta:
+            print(f"  ✅ {model_id} ({gguf_quant})")
+            print(f"     Quant: {quant} | GGUF: {gguf_quant}")
+            print(f"     Downloads: {meta.get('downloads', 0):,}")
+            print()
+
+    return models
+
+
+def get_best_model(models: list[ModelInfo]) -> Optional[ModelInfo]:
+    """Pick the best model that exists and fits in 24GB."""
+    available = [m for m in models if m.exists_on_hf]
+    if not available:
+        return None
+
+    # Prefer AWQ (best quality/speed on Ampere)
+    awq = [m for m in available if m.quantization == "awq"]
+    if awq:
+        return awq[0]
+
+    # Then GPTQ
+    gptq = [m for m in available if m.quantization == "gptq"]
+    if gptq:
+        return gptq[0]
+
+    # Then FP8 (tight but works)
+    fp8 = [m for m in available if m.quantization == "fp8"]
+    if fp8:
+        return fp8[0]
+
+    return available[0]
+
+
+def download_model(model_id: str, cache_dir: Optional[str] = None) -> bool:
+    """Download model using huggingface_hub or snapshot_download."""
+    print(f"\n📥 Downloading {model_id}...")
+
+    # Try huggingface_hub first
+    try:
+        from huggingface_hub import snapshot_download
+
+        kwargs = {"repo_id": model_id}
+        if cache_dir:
+            kwargs["cache_dir"] = cache_dir
+
+        t0 = time.time()
+        path = snapshot_download(**kwargs)
+        elapsed = time.time() - t0
+        print(f"  ✅ Downloaded to: {path}")
+        print(f"  ⏱️  Time: {elapsed:.0f}s")
+        return True
+
+    except ImportError:
+        print("  ⚠️  huggingface_hub not installed, trying transformers...")
+    except Exception as e:
+        print(f"  ⚠️  huggingface_hub failed: {e}")
+
+    # Try transformers
+    try:
+        from transformers import AutoTokenizer
+        t0 = time.time()
+        tok = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
+        # This only downloads tokenizer config. Need full model.
+        print(f"  Tokenizer downloaded (model will be pulled by vLLM on first use)")
+        return True
+    except Exception as e:
+        print(f"  ❌ Failed: {e}")
+        return False
+
+
+def print_recommended_config(model_id: str):
+    """Print the recommended vLLM flags for the selected model."""
+    quant = "auto"
+    for m in MODEL_CANDIDATES:
+        if m[0] == model_id:
+            quant = m[1]
+            break
+
+    print()
+    print("=" * 60)
+    print(" RECOMMENDED VLLM CONFIGURATION")
+    print("=" * 60)
+    print()
+    print(f"  export HF_TOKEN='your_token'")
+    print()
+    print(f"  vllm serve {model_id} \\")
+    print(f"    --max-model-len 8192 \\")
+    print(f"    --gpu-memory-utilization 0.90 \\")
+
+    if quant == "fp8":
+        print(f"    --max-num-seqs 4 \\")
+        print(f"    --max-num-batched-tokens 16384 \\")
+        print(f"    --kv-cache-dtype fp8 \\")
+        print(f"    --dtype auto \\")
+    else:
+        print(f"    --max-num-seqs 8 \\")
+        print(f"    --max-num-batched-tokens 32768 \\")
+        print(f"    --dtype auto \\")
+
+    print(f"    --attention-backend flashinfer \\")
+    print(f"    --enable-prefix-caching \\")
+    print(f"    --enable-chunked-prefill \\")
+    print(f"    --block-size 16 \\")
+    print(f"    --swap-space 4 \\")
+    print(f"    --tensor-parallel-size 1 \\")
+    print(f"    --trust-remote-code")
+    print()
+    print("  # Or use the optimized launcher:")
+    print(f"  bash scripts/serve_vllm_optimized.sh awq-8k")
+    print("=" * 60)
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Download best Qwen model for RTX 3090")
+    parser.add_argument("--list", action="store_true", help="List available models only")
+    parser.add_argument("--model", type=str, help="Specific model to download")
+    parser.add_argument("--dry-run", action="store_true", help="Print what would be done")
+    parser.add_argument("--cache-dir", type=str, help="Custom HF cache directory")
+    args = parser.parse_args()
+
+    # Scan
+    models = scan_models()
+    available = [m for m in models if m.exists_on_hf]
+
+    if not available:
+        print("❌ No Qwen 3.6 27B models found on Hugging Face.")
+        print("   Check: 1) HF_TOKEN for gated models  2) Model name correctness")
+        sys.exit(1)
+
+    # Select
+    if args.model:
+        target = next((m for m in models if m.model_id == args.model), None)
+        if not target:
+            print(f"❌ Model {args.model} not found")
+            sys.exit(1)
+    else:
+        target = get_best_model(available)
+        if not target:
+            print("❌ No suitable model found")
+            sys.exit(1)
+
+    print(f"\n🏆 Best model: {target.model_id}")
+    print(f"   Quantization: {target.quantization}")
+    print(f"   Est. VRAM: ~{target.est_vram_gb} GB")
+    print(f"   Disk size: {target.size_on_disk_gb:.1f} GB")
+
+    if args.list or args.dry_run:
+        print_recommended_config(target.model_id)
+        return
+
+    # Download
+    success = download_model(target.model_id, args.cache_dir)
+    if success:
+        print_recommended_config(target.model_id)
+
+
+if __name__ == "__main__":
+    main()

scripts/fix_gpu_host.sh

@@ -0,0 +1,74 @@
+#!/usr/bin/env bash
+# ============================================================
+# fix_gpu_host.sh — Run on the HOST (not container) to fix
+# CUDA on RTX 3090 with NVIDIA open kernel module.
+#
+# Problem: The open kernel module (nvidia-open) blocks CUDA
+# on GeForce RTX 3090. nvidia-smi works but cuInit() fails
+# with error 999 (UNKNOWN). This is a known driver issue.
+#
+# Solution: Switch from nvidia-open to the proprietary driver.
+# ============================================================
+set -euo pipefail
+
+echo "============================================"
+echo " GPU Fix — RTX 3090 + NVIDIA Driver"
+echo "============================================"
+
+# Check current driver
+echo ""
+echo "Current driver:"
+nvidia-smi --query-gpu=name,driver_version --format=csv 2>/dev/null || echo "nvidia-smi not found"
+
+# Check kernel module type
+echo ""
+echo "Kernel module type:"
+if lsmod | grep -q nvidia; then
+    if modinfo nvidia 2>/dev/null | grep -q "open"; then
+        echo "  OPEN kernel module detected — THIS IS THE PROBLEM"
+        echo "  GeForce GPUs need the proprietary module for full CUDA"
+    else
+        echo "  Proprietary kernel module detected"
+    fi
+else
+    echo "  No nvidia module loaded"
+fi
+
+echo ""
+echo "=== Fix options ==="
+echo ""
+echo "Option 1: Switch to proprietary driver (recommended)"
+echo "  sudo apt-get remove --purge nvidia-dkms-580-open nvidia-driver-580-open"
+echo "  sudo apt-get install nvidia-driver-580 nvidia-dkms-580"
+echo "  sudo reboot"
+echo ""
+echo "Option 2: For RunPod specifically"
+echo "  Stop this pod and restart with template:"
+echo "    'RunPod Pytorch 2.4+' or 'NVIDIA CUDA 12.4'"
+echo "  Ensure the template does NOT say 'open kernel'"
+echo ""
+echo "Option 3: Quick fix (try without reboot)"
+echo "  sudo rmmod nvidia-uvm nvidia-drm nvidia-modeset nvidia"
+echo "  sudo modprobe nvidia"
+echo "  sudo modprobe nvidia-uvm"
+echo "  sudo nvidia-persistenced --user root"
+echo ""
+
+# Auto-detect and apply if running as root on host
+if [[ "$(id -u)" -eq 0 ]] && [[ -f /proc/driver/nvidia/version ]]; then
+    KERNEL_TYPE=$(cat /proc/driver/nvidia/version | grep -i "open" && echo "open" || echo "proprietary")
+    if [[ "$KERNEL_TYPE" == "open" ]]; then
+        echo "⚠️  Open kernel module detected. Would you like to attempt fix? (y/n)"
+        read -r answer
+        if [[ "$answer" == "y" ]]; then
+            echo "Attempting fix..."
+            rmmod nvidia-uvm nvidia-drm nvidia-modeset nvidia 2>/dev/null || true
+            modprobe nvidia 2>/dev/null || true
+            modprobe nvidia-uvm 2>/dev/null || true
+            nvidia-persistenced --user root 2>/dev/null || true
+            echo "Fix applied. Test with: python3 -c 'import torch; print(torch.cuda.is_available())'"
+        fi
+    else
+        echo "✅ Proprietary driver detected — CUDA should work"
+    fi
+fi

scripts/inspect_model.py

@@ -0,0 +1,301 @@
+#!/usr/bin/env python3
+"""
+inspect_model.py — Inspect a Hugging Face model config BEFORE downloading.
+
+Queries the HF Hub API to read config.json without downloading weights.
+Reports:
+    - Architecture, hidden size, layers, heads
+    - Vocabulary size
+    - Dtype, rope config
+    - Quantization config (bits, group_size, desc_act)
+    - Estimated VRAM for different quant levels on RTX 3090
+
+Usage:
+    python scripts/inspect_model.py QuantTrio/Qwen3.6-27B-AWQ
+    python scripts/inspect_model.py Qwen/Qwen3.6-27B-FP8 --compare
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+from pathlib import Path
+
+import httpx
+
+HF_API = "https://huggingface.co/api"
+HF_RAW = "https://huggingface.co"
+
+
+def fetch_json(repo_id: str, filename: str) -> dict | None:
+    """Fetch a JSON file from a HF repo without downloading weights."""
+    url = f"{HF_RAW}/{repo_id}/raw/main/{filename}"
+    try:
+        resp = httpx.get(url, timeout=15, follow_redirects=True)
+        if resp.status_code == 200:
+            return resp.json()
+        # Try resolving the default branch
+        model_api = f"{HF_API}/models/{repo_id}"
+        resp2 = httpx.get(model_api, timeout=10)
+        if resp2.status_code == 200:
+            branch = resp2.json().get("defaultBranch", "main")
+            url = f"{HF_RAW}/{repo_id}/raw/{branch}/{filename}"
+            resp3 = httpx.get(url, timeout=15)
+            if resp3.status_code == 200:
+                return resp3.json()
+        return None
+    except Exception as e:
+        print(f"  ⚠️  Error fetching {filename}: {e}")
+        return None
+
+
+def estimate_vram(
+    hidden_size: int,
+    num_layers: int,
+    num_heads: int,
+    num_kv_heads: int,
+    vocab_size: int,
+    max_seq_len: int = 8192,
+    dtype_bytes: int = 2,        # bf16
+    kv_dtype_bytes: int = 2,     # KV cache dtype
+) -> dict:
+    """Estimate VRAM usage for a model at various quant levels.
+
+    Uses the standard formula:
+        weights = params * bytes_per_param
+        kv_cache = 2 * num_layers * num_kv_heads * head_dim * max_seq_len * kv_dtype_bytes
+        activation = ~2-4 GB overhead
+    """
+    head_dim = hidden_size // num_heads
+    kv_head_dim = hidden_size // num_heads  # GQA: kv_heads share head_dim
+
+    # Parameters (approximate)
+    # Q, K, V: 3 * hidden_size * hidden_size (but K,V use kv_heads)
+    q_size = hidden_size * hidden_size
+    k_size = num_kv_heads * kv_head_dim * hidden_size
+    v_size = num_kv_heads * kv_head_dim * hidden_size
+    # O: hidden_size * hidden_size
+    o_size = hidden_size * hidden_size
+    # MLP: 2 * hidden_size * intermediate_size + intermediate_size * hidden_size
+    # Qwen uses SwiGLU with 3 gates → intermediate ~= hidden_size * 8/3
+    intermediate_size = int(hidden_size * 8 / 3)
+    mlp_size = 3 * hidden_size * intermediate_size
+    # Per-layer total
+    per_layer_params = q_size + k_size + v_size + o_size + mlp_size
+    # Embedding + LM head
+    embedding_params = vocab_size * hidden_size * 2  # tied in most models
+    # Layer norm etc (negligible)
+    total_params = per_layer_params * num_layers + embedding_params
+
+    # KV cache per layer
+    kv_cache_per_layer = 2 * num_kv_heads * kv_head_dim * max_seq_len * kv_dtype_bytes
+    kv_cache_total = kv_cache_per_layer * num_layers
+
+    # Overhead (activations, workspace, CUDA context)
+    overhead = 2 * 1024**3  # 2 GB
+
+    results = {}
+    for name, bpw in [("bf16", 2), ("fp8", 1), ("int4", 0.5), ("int4_g128", 0.55)]:
+        weight_bytes = total_params * bpw
+        if "fp8" in name:
+            kv_bytes = kv_cache_total  # same dtype
+        else:
+            kv_bytes = kv_cache_total
+        total_vram = weight_bytes + kv_bytes + overhead
+        results[name] = total_vram / (1024**3)
+
+    return results
+
+
+def inspect_model(repo_id: str, max_seq_len: int = 8192):
+    """Fetch and analyze model config."""
+    print(f"\n{'='*70}")
+    print(f" MODEL INSPECTOR — {repo_id}")
+    print(f"{'='*70}\n")
+
+    config = fetch_json(repo_id, "config.json")
+    quant_config = fetch_json(repo_id, "quantize_config.json") or fetch_json(repo_id, "quant_config.json")
+
+    if not config:
+        print(f"❌ Could not fetch config.json for {repo_id}")
+        print(f"   URL tried: {HF_RAW}/{repo_id}/raw/main/config.json")
+        sys.exit(1)
+
+    # Architecture
+    arch = config.get("architectures", ["unknown"])[0]
+    model_type = config.get("model_type", "unknown")
+    hidden_size = config.get("hidden_size", 0)
+    num_layers = config.get("num_hidden_layers", config.get("num_layers", 0))
+    num_heads = config.get("num_attention_heads", config.get("num_heads", 0))
+    num_kv_heads = config.get("num_key_value_heads", num_heads)
+    vocab_size = config.get("vocab_size", 0)
+    intermediate_size = config.get("intermediate_size", 0)
+    rope_theta = config.get("rope_theta", 0)
+    max_position = config.get("max_position_embeddings", 0)
+    rope_scaling = config.get("rope_scaling", {})
+    torch_dtype = config.get("torch_dtype", "auto")
+
+    # Quant info
+    quant_method = ""
+    quant_bits = 0
+    quant_group_size = 0
+    quant_desc_act = False
+    quant_sym = False
+
+    if quant_config:
+        quant_method = quant_config.get("quant_method", quant_config.get("quantization", ""))
+        quant_bits = quant_config.get("bits", quant_config.get("w_bit", 0))
+        quant_group_size = quant_config.get("group_size", quant_config.get("q_group_size", 0))
+        quant_desc_act = quant_config.get("desc_act", False)
+        quant_sym = quant_config.get("sym", False)
+
+    # Print
+    print("┌─────────────────────────────────────────────────────┐")
+    print(f"│ Architecture                                         │")
+    print(f"├─────────────────────────────────────────────────────┤")
+    print(f"│ Type:        {model_type:<40s} │")
+    print(f"│ Hidden:      {hidden_size:<40d} │")
+    print(f"│ Layers:      {num_layers:<40d} │")
+    print(f"│ Heads:       {num_heads:<40d} │")
+    print(f"│ KV Heads:    {num_kv_heads:<40d} │")
+    print(f"│ Head dim:    {hidden_size // num_heads if num_heads else 0:<40d} │")
+    print(f"│ Vocab:       {vocab_size:<40d} │")
+    print(f"│ Intermed:    {intermediate_size:<40d} │")
+    print(f"│ Torch dtype: {str(torch_dtype):<40s} │")
+    print(f"│ Max pos:     {max_position:<40d} │")
+    print(f"│ Rope theta:  {rope_theta:<40.0f} │")
+    if rope_scaling:
+        print(f"│ Rope scale:  {str(rope_scaling):<40s} │")
+    print(f"└─────────────────────────────────────────────────────┘")
+
+    if quant_config:
+        print()
+        print("┌─────────────────────────────────────────────────────┐")
+        print(f"│ Quantization Config                                  │")
+        print(f"├─────────────────────────────────────────────────────┤")
+        print(f"│ Method:      {quant_method:<40s} │")
+        print(f"│ Bits:        {quant_bits:<40d} │")
+        print(f"│ Group size:  {quant_group_size:<40d} │")
+        print(f"│ Desc act:    {str(quant_desc_act):<40s} │")
+        print(f"│ Symmetric:   {str(quant_sym):<40s} │")
+        print(f"└─────────────────────────────────────────────────────┘")
+
+    # VRAM estimation
+    print()
+    print("┌─────────────────────────────────────────────────────┐")
+    print(f"│ Estimated VRAM (max_seq_len={max_seq_len})                 │")
+    print(f"├─────────────────────────────────────────────────────┤")
+    vram = estimate_vram(hidden_size, num_layers, num_heads, num_kv_heads, vocab_size, max_seq_len)
+    for name, gb in vram.items():
+        bar = "█" * int(gb / 2) + "░" * (12 - int(gb / 2))
+        fits = "✅ FITS" if gb <= 23 else "❌ OOM"
+        print(f"│ {name:10s}: {gb:5.1f} GB {bar} {fits:<10s} │")
+    print(f"└─────────────────────────────────────────────────────┘")
+
+    # Summary
+    print()
+    print("💡 Recommendations:")
+    if hidden_size and num_layers:
+        total_params_b = hidden_size * hidden_size * 8 * num_layers / 1e9  # rough
+        print(f"   Estimated params: ~{hidden_size / 1000:.0f}B-class model")
+
+    if quant_method:
+        if quant_method in ("awq", "gptq"):
+            print(f"   {quant_method.upper()} 4-bit — best for RTX 3090 (Ampere)")
+            print(f"   Use: --dtype auto (vLLM auto-detects quant)")
+        elif quant_method == "fp8":
+            print(f"   FP8 — tight on 24GB. Use --kv-cache-dtype fp8 + --gpu-memory-utilization 0.85")
+
+    if num_kv_heads < num_heads and num_heads:
+        ratio = num_heads / num_kv_heads
+        print(f"   GQA ratio: {ratio:.0f}:1 — KV cache is {ratio:.0f}x smaller than MHA")
+
+    print()
+    print(f"   Recommended vLLM flags for RTX 3090:")
+    print(f"   --max-model-len {max_seq_len} --gpu-memory-utilization 0.90 --enable-prefix-caching")
+
+
+def compare_models(models: list[str], max_seq_len: int = 8192):
+    """Compare multiple model variants side by side."""
+    print(f"\n{'='*70}")
+    print(f" MODEL COMPARISON — {len(models)} variants")
+    print(f"{'='*70}\n")
+
+    rows = []
+    for repo_id in models:
+        config = fetch_json(repo_id, "config.json")
+        if not config:
+            print(f"  ❌ {repo_id}: config.json not available")
+            continue
+
+        quant_config = fetch_json(repo_id, "quantize_config.json") or fetch_json(repo_id, "quant_config.json")
+        quant_method = ""
+        quant_bits = 0
+        if quant_config:
+            quant_method = quant_config.get("quant_method", "")
+            quant_bits = quant_config.get("bits", 0)
+
+        hidden_size = config.get("hidden_size", 0)
+        num_layers = config.get("num_hidden_layers", 0)
+        num_heads = config.get("num_attention_heads", 0)
+        num_kv_heads = config.get("num_key_value_heads", num_heads)
+        vocab_size = config.get("vocab_size", 0)
+
+        vram = estimate_vram(hidden_size, num_layers, num_heads, num_kv_heads, vocab_size, max_seq_len)
+
+        rows.append({
+            "model": repo_id.split("/")[-1],
+            "quant": quant_method or "none",
+            "bits": quant_bits or 16,
+            "vram_bf16": vram.get("bf16", 0),
+            "vram_fp8": vram.get("fp8", 0),
+            "vram_int4": vram.get("int4", 0),
+            "fits_24gb": vram.get("fp8", 999) <= 23 or vram.get("int4", 999) <= 23,
+        })
+
+    # Print table
+    print(f"│ {'Model':<35s} │ {'Quant':>8s} │ {'Bits':>5s} │ {'VRAM(bf16)':>10s} │ {'VRAM(fp8)':>10s} │ {'VRAM(int4)':>10s} │ {'Fits 24GB':>10s} │")
+    print(f"│{'-'*37}│{'-'*10}│{'-'*7}│{'-'*12}│{'-'*12}│{'-'*12}│{'-'*12}│")
+    for r in rows:
+        fits = "✅" if r["fits_24gb"] else "❌"
+        print(f"│ {r['model']:<35s} │ {r['quant']:>8s} │ {r['bits']:>4d}  │ {r['vram_bf16']:>8.1f} GB │ {r['vram_fp8']:>8.1f} GB │ {r['vram_int4']:>8.1f} GB │ {fits:>10s} │")
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Inspect HF model config without downloading")
+    parser.add_argument("model", nargs="?", help="Model repo id (e.g. QuantTrio/Qwen3.6-27B-AWQ)")
+    parser.add_argument("--compare", nargs="*", help="Compare multiple models")
+    parser.add_argument("--max-seq-len", type=int, default=8192, help="Context length for VRAM estimate")
+    parser.add_argument("--all-variants", action="store_true", help="Compare all Qwen 27B quant variants")
+    args = parser.parse_args()
+
+    if args.all_variants:
+        models = [
+            "Qwen/Qwen3.6-27B",
+            "Qwen/Qwen3.6-27B-FP8",
+            "QuantTrio/Qwen3.6-27B-AWQ",
+            "groxaxo/Qwen3.6-27B-GPTQ-Pro-4bit",
+        ]
+        compare_models(models, args.max_seq_len)
+    elif args.compare is not None:
+        models = args.compare if args.compare else [
+            "QuantTrio/Qwen3.6-27B-AWQ",
+            "Qwen/Qwen3.6-27B-FP8",
+            "groxaxo/Qwen3.6-27B-GPTQ-Pro-4bit",
+        ]
+        compare_models(models, args.max_seq_len)
+    elif args.model:
+        inspect_model(args.model, args.max_seq_len)
+    else:
+        # Default: compare all Qwen 27B variants
+        models = [
+            "Qwen/Qwen3.6-27B",
+            "Qwen/Qwen3.6-27B-FP8",
+            "QuantTrio/Qwen3.6-27B-AWQ",
+            "groxaxo/Qwen3.6-27B-GPTQ-Pro-4bit",
+        ]
+        compare_models(models, args.max_seq_len)
+
+
+if __name__ == "__main__":
+    main()

scripts/run_all.sh

@@ -0,0 +1,159 @@
+#!/usr/bin/env bash
+# ============================================================
+# run_all.sh — Master orchestrator for Qwen SpeedLab
+#
+# Runs the full pipeline:
+#   1. Setup environment
+#   2. Inspect available models
+#   3. Download best model
+#   4. Quick sweep (7 configs, ~30 min)
+#   5. Advanced A/B sweep (15 configs, ~1h)
+#   6. Generate final report
+#
+# Usage:
+#   bash scripts/run_all.sh              # Full pipeline
+#   bash scripts/run_all.sh --quick      # Quick only (skip advanced)
+#   bash scripts/run_all.sh --dry-run    # Print what would be done
+# ============================================================
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PROJECT_DIR="$(dirname "$SCRIPT_DIR")"
+cd "$PROJECT_DIR"
+
+MODE="${1:---full}"
+DRY_RUN=false
+if [[ "${1:-}" == "--dry-run" ]]; then
+    DRY_RUN=true
+fi
+
+# Colors
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+BLUE='\033[0;34m'
+BOLD='\033[1m'
+NC='\033[0m'
+
+banner() {
+    echo ""
+    echo -e "${BLUE}╔══════════════════════════════════════════════════════════╗${NC}"
+    echo -e "${BLUE}║${NC} ${BOLD}$1${NC}"
+    echo -e "${BLUE}╚══════════════════════════════════════════════════════════╝${NC}"
+    echo ""
+}
+
+step() {
+    echo -e "${GREEN}[$1/$TOTAL_STEPS]${NC} ${BOLD}$2${NC}"
+    echo ""
+}
+
+# Count steps
+TOTAL_STEPS=6
+if [[ "$MODE" == "--quick" ]]; then
+    TOTAL_STEPS=4
+fi
+
+banner "Qwen SpeedLab — Full Pipeline"
+echo "  Mode: $MODE"
+echo "  GPU:  $(nvidia-smi --query-gpu=name --format=csv,noheader 2>/dev/null || echo 'unknown')"
+echo ""
+
+# ---- Step 1: Setup ----
+step 1 "Environment check"
+if $DRY_RUN; then
+    echo "  [DRY RUN] Would run: bash scripts/setup_runpod.sh"
+else
+    if [[ -d ".venv" ]]; then
+        echo "  ✅ .venv already exists"
+        source .venv/bin/activate
+    else
+        echo "  Running setup_runpod.sh..."
+        bash scripts/setup_runpod.sh
+        source .venv/bin/activate
+    fi
+fi
+
+# ---- Step 2: Inspect models ----
+step 2 "Model availability scan"
+if $DRY_RUN; then
+    echo "  [DRY RUN] Would run: python scripts/inspect_model.py --all-variants"
+else
+    python scripts/inspect_model.py --all-variants 2>/dev/null || {
+        echo -e "  ${YELLOW}⚠️  inspect_model.py failed (httpx not installed?) — continuing${NC}"
+    }
+fi
+
+# ---- Step 3: Download best model ----
+step 3 "Download best model"
+if $DRY_RUN; then
+    echo "  [DRY RUN] Would run: python scripts/download_best_model.py --list"
+else
+    python scripts/download_best_model.py --list 2>/dev/null || {
+        echo -e "  ${YELLOW}⚠️  download_best_model.py failed — continuing${NC}"
+    }
+    echo ""
+    echo -e "  ${YELLOW}⚠️  Skipping actual download (vLLM will pull on first use).${NC}"
+    echo "  To download now: python scripts/download_best_model.py"
+fi
+
+# ---- Step 4: Quick sweep ----
+step 4 "Quick configuration sweep (7 configs, ~30 min)"
+if $DRY_RUN; then
+    echo "  [DRY RUN] Would run: python scripts/sweep_vllm_configs.py --quick"
+else
+    echo -e "  ${YELLOW}⚠️  This takes ~30 min and will use the GPU exclusively.${NC}"
+    echo "  Press Ctrl+C to skip, or Enter to continue..."
+    read -r -t 5 || true
+    echo ""
+    python scripts/sweep_vllm_configs.py --quick 2>&1 | tail -40 || {
+        echo -e "  ${RED}❌ Quick sweep failed — check logs${NC}"
+    }
+fi
+
+# ---- Step 5: Advanced sweep ----
+if [[ "$MODE" != "--quick" ]]; then
+    step 5 "Advanced optimization A/B sweep (15 configs, ~1h)"
+    if $DRY_RUN; then
+        echo "  [DRY RUN] Would run: python scripts/sweep_advanced.py --mode baseline"
+    else
+        echo -e "  ${YELLOW}⚠️  This takes ~1h. Press Ctrl+C to skip, or Enter to continue...${NC}"
+        read -r -t 5 || true
+        echo ""
+        python scripts/sweep_advanced.py --mode shootout 2>&1 | tail -40 || {
+            echo -e "  ${YELLOW}⚠️  Advanced sweep had issues — check reports/summary_advanced.md${NC}"
+        }
+    fi
+fi
+
+# ---- Step 6: Final report ----
+step "$TOTAL_STEPS" "Final report"
+echo "  Reports generated:"
+for f in reports/summary.md reports/summary_advanced.md reports/results.csv reports/results_advanced.csv; do
+    if [[ -f "$f" ]]; then
+        size=$(wc -l < "$f" 2>/dev/null || echo "?")
+        echo -e "    ${GREEN}✅${NC} $f ($size lines)"
+    else
+        echo -e "    ${YELLOW}—${NC}  $f (not generated — run sweeps to populate)"
+    fi
+done
+
+# Print best result if available
+echo ""
+if [[ -f reports/summary.md ]]; then
+    echo "────────────────────────────────────────────────────"
+    if grep -A5 "Best Configuration" reports/summary.md 2>/dev/null | head -12; then
+        true
+    fi
+    echo "────────────────────────────────────────────────────"
+fi
+
+banner "✅ Pipeline complete!"
+echo ""
+echo "  Next steps:"
+echo "    - View report:    cat reports/summary.md"
+echo "    - Advanced report: cat reports/summary_advanced.md"
+echo "    - Raw data:       cat reports/results.csv"
+echo "    - Re-run sweep:   python scripts/sweep_advanced.py --mode grid"
+echo "    - Start server:   bash scripts/serve_vllm.sh awq-8k"
+echo ""

scripts/serve_sglang.sh

@@ -0,0 +1,77 @@
+#!/usr/bin/env bash
+# ============================================================
+# serve_sglang.sh — Lancement SGLang pour Qwen 3.6 27B sur RTX 3090
+# Usage:
+#   bash scripts/serve_sglang.sh [config_name]
+#
+# Configs prédéfinies :
+#   fp8-8k    — FP8, context 8192
+#   awq-8k    — AWQ, context 8192
+#   fp8-16k   — FP8, context 16384 (limite)
+# ============================================================
+set -euo pipefail
+
+CONFIG="${1:-fp8-8k}"
+
+# --- Modèles -----------------------------------------------------------
+MODEL_FP8="Qwen/Qwen3.6-27B-FP8"
+MODEL_AWQ="QuantTrio/Qwen3.6-27B-AWQ"
+MODEL=""
+QUANT=""
+
+case "$CONFIG" in
+    fp8-8k)
+        MODEL="$MODEL_FP8"
+        CONTEXT_LEN=8192
+        QUANT="fp8"
+        MEM_FRAC=0.87
+        ;;
+    awq-8k)
+        MODEL="$MODEL_AWQ"
+        CONTEXT_LEN=8192
+        QUANT="awq"
+        MEM_FRAC=0.87
+        ;;
+    fp8-16k)
+        MODEL="$MODEL_FP8"
+        CONTEXT_LEN=16384
+        QUANT="fp8"
+        MEM_FRAC=0.87
+        ;;
+    *)
+        echo "Config inconnue: $CONFIG"
+        echo "Configs disponibles: fp8-8k, awq-8k, fp8-16k"
+        exit 1
+        ;;
+esac
+
+PORT="${PORT:-8000}"
+HOST="${HOST:-0.0.0.0}"
+
+echo "============================================"
+echo " SGLang Server — Qwen SpeedLab"
+echo "============================================"
+echo "  Config:       $CONFIG"
+echo "  Model:        $MODEL"
+echo "  Quantization: $QUANT"
+echo "  Context len:  $CONTEXT_LEN"
+echo "  Mem fraction: $MEM_FRAC"
+echo "  Port:         $PORT"
+echo "============================================"
+
+# SGLang utilise --mem-fraction pour limiter la VRAM
+# et --context-length pour la longueur max
+
+echo ""
+echo "  Lancement SGLang..."
+echo "  Logs: /tmp/sglang_${CONFIG}.log"
+
+python -m sglang.launch_server \
+    --model-path "$MODEL" \
+    --host "$HOST" \
+    --port "$PORT" \
+    --context-length "$CONTEXT_LEN" \
+    --mem-fraction-static "$MEM_FRAC" \
+    --trust-remote-code \
+    --enable-flashinfer \
+    2>&1 | tee "/tmp/sglang_${CONFIG}.log"

scripts/serve_vllm.sh

@@ -0,0 +1,293 @@
+#!/usr/bin/env bash
+# ============================================================
+# serve_vllm.sh — Lancement vLLM pour Qwen 3.6 27B sur RTX 3090
+# Usage:
+#   bash scripts/serve_vllm.sh [config_name]
+#
+# Configs prédéfinies :
+#   ★ awq-8k     — AWQ 4-bit, 8192 ctx, flashinfer (recommandé)
+#   awq-16k      — AWQ 4-bit, 16384 ctx
+#   awq-8k-fast  — AWQ 4-bit, 8192 ctx, max throughput
+#   fp8-8k       — FP8, 8192 ctx, flashinfer
+#   fp8-16k      — FP8, 16384 ctx (limite VRAM)
+#   fp8-4k       — FP8, 4096 ctx
+#   gptq-8k      — GPTQ Int4, 8192 ctx
+#   fp8-8k-kvc   — FP8 + FP8 KV cache (économise ~30% VRAM KV)
+#   spec-8k      — Speculative decoding (AWQ + 0.6B draft)
+#   fp8-8k-hi    — FP8, mem=0.95 (risqué)
+#   fp8-8k-lo    — FP8, mem=0.85
+# ============================================================
+set -euo pipefail
+
+CONFIG="${1:-awq-8k}"
+
+# --- Modèles ------------------------------------------------------------
+MODEL_FP8="Qwen/Qwen3.6-27B-FP8"
+MODEL_AWQ="QuantTrio/Qwen3.6-27B-AWQ"
+MODEL_GPTQ="groxaxo/Qwen3.6-27B-GPTQ-Pro-4bit"
+MODEL_BASE="Qwen/Qwen3.6-27B"
+MODEL_DRAFT="Qwen/Qwen2.5-0.5B-Instruct"
+
+# --- Sélection config --------------------------------------------------
+case "$CONFIG" in
+    # AWQ — best quality/speed on Ampere (SM 8.6)
+    awq-8k)
+        MODEL="$MODEL_AWQ"
+        MAX_MODEL_LEN=8192
+        GPU_MEM_UTIL=0.90
+        DTYPE="auto"
+        QUANT="awq"
+        MAX_NUM_SEQS=8
+        MAX_BATCHED_TOKENS=32768
+        ATTENTION="flashinfer"
+        KV_CACHE_DTYPE="fp8"
+        CHUNKED_PREFILL=true
+        BLOCK_SIZE=16
+        SWAP_SPACE=4
+        MAX_SEQ_LEN_CAPTURE=8192
+        SPEC_MODEL=""
+        SPEC_TOKENS=5
+        ;;
+    awq-16k)
+        MODEL="$MODEL_AWQ"
+        MAX_MODEL_LEN=16384
+        GPU_MEM_UTIL=0.90
+        DTYPE="auto"
+        QUANT="awq"
+        MAX_NUM_SEQS=4
+        MAX_BATCHED_TOKENS=32768
+        ATTENTION="flashinfer"
+        KV_CACHE_DTYPE="fp8"
+        CHUNKED_PREFILL=true
+        BLOCK_SIZE=32
+        SWAP_SPACE=8
+        MAX_SEQ_LEN_CAPTURE=16384
+        SPEC_MODEL=""
+        SPEC_TOKENS=5
+        ;;
+    awq-8k-fast)
+        MODEL="$MODEL_AWQ"
+        MAX_MODEL_LEN=8192
+        GPU_MEM_UTIL=0.90
+        DTYPE="auto"
+        QUANT="awq"
+        MAX_NUM_SEQS=16
+        MAX_BATCHED_TOKENS=65536
+        ATTENTION="flashinfer"
+        KV_CACHE_DTYPE="fp8"
+        CHUNKED_PREFILL=true
+        BLOCK_SIZE=16
+        SWAP_SPACE=4
+        MAX_SEQ_LEN_CAPTURE=8192
+        SPEC_MODEL=""
+        SPEC_TOKENS=5
+        ;;
+    # FP8 — higher precision, tighter VRAM
+    fp8-8k)
+        MODEL="$MODEL_FP8"
+        MAX_MODEL_LEN=8192
+        GPU_MEM_UTIL=0.88
+        DTYPE="auto"
+        QUANT="fp8"
+        MAX_NUM_SEQS=4
+        MAX_BATCHED_TOKENS=16384
+        ATTENTION="flashinfer"
+        KV_CACHE_DTYPE="auto"
+        CHUNKED_PREFILL=true
+        BLOCK_SIZE=16
+        SWAP_SPACE=4
+        MAX_SEQ_LEN_CAPTURE=8192
+        SPEC_MODEL=""
+        SPEC_TOKENS=5
+        ;;
+    fp8-16k)
+        MODEL="$MODEL_FP8"
+        MAX_MODEL_LEN=16384
+        GPU_MEM_UTIL=0.85
+        DTYPE="auto"
+        QUANT="fp8"
+        MAX_NUM_SEQS=2
+        MAX_BATCHED_TOKENS=16384
+        ATTENTION="flashinfer"
+        KV_CACHE_DTYPE="fp8"
+        CHUNKED_PREFILL=true
+        BLOCK_SIZE=32
+        SWAP_SPACE=8
+        MAX_SEQ_LEN_CAPTURE=16384
+        SPEC_MODEL=""
+        SPEC_TOKENS=5
+        ;;
+    fp8-4k)
+        MODEL="$MODEL_FP8"
+        MAX_MODEL_LEN=4096
+        GPU_MEM_UTIL=0.90
+        DTYPE="auto"
+        QUANT="fp8"
+        MAX_NUM_SEQS=8
+        MAX_BATCHED_TOKENS=32768
+        ATTENTION="flashinfer"
+        KV_CACHE_DTYPE="auto"
+        CHUNKED_PREFILL=true
+        BLOCK_SIZE=16
+        SWAP_SPACE=4
+        MAX_SEQ_LEN_CAPTURE=4096
+        SPEC_MODEL=""
+        SPEC_TOKENS=5
+        ;;
+    fp8-8k-kvc)
+        MODEL="$MODEL_FP8"
+        MAX_MODEL_LEN=8192
+        GPU_MEM_UTIL=0.85
+        DTYPE="auto"
+        QUANT="fp8"
+        MAX_NUM_SEQS=6
+        MAX_BATCHED_TOKENS=16384
+        ATTENTION="flashinfer"
+        KV_CACHE_DTYPE="fp8"
+        CHUNKED_PREFILL=true
+        BLOCK_SIZE=16
+        SWAP_SPACE=4
+        MAX_SEQ_LEN_CAPTURE=8192
+        SPEC_MODEL=""
+        SPEC_TOKENS=5
+        ;;
+    # GPTQ
+    gptq-8k)
+        MODEL="$MODEL_GPTQ"
+        MAX_MODEL_LEN=8192
+        GPU_MEM_UTIL=0.90
+        DTYPE="auto"
+        QUANT="gptq"
+        MAX_NUM_SEQS=8
+        MAX_BATCHED_TOKENS=32768
+        ATTENTION="flashinfer"
+        KV_CACHE_DTYPE="fp8"
+        CHUNKED_PREFILL=true
+        BLOCK_SIZE=16
+        SWAP_SPACE=4
+        MAX_SEQ_LEN_CAPTURE=8192
+        SPEC_MODEL=""
+        SPEC_TOKENS=5
+        ;;
+    # Speculative decoding
+    spec-8k)
+        MODEL="$MODEL_AWQ"
+        MAX_MODEL_LEN=8192
+        GPU_MEM_UTIL=0.88
+        DTYPE="auto"
+        QUANT="awq"
+        MAX_NUM_SEQS=4
+        MAX_BATCHED_TOKENS=16384
+        ATTENTION="flashinfer"
+        KV_CACHE_DTYPE="fp8"
+        CHUNKED_PREFILL=false
+        BLOCK_SIZE=16
+        SWAP_SPACE=4
+        MAX_SEQ_LEN_CAPTURE=8192
+        SPEC_MODEL="$MODEL_DRAFT"
+        SPEC_TOKENS=5
+        ;;
+    # Legacy FP8 configs
+    fp8-8k-hi)
+        MODEL="$MODEL_FP8"; MAX_MODEL_LEN=8192; GPU_MEM_UTIL=0.95; DTYPE="auto"; QUANT="fp8"
+        MAX_NUM_SEQS=4; MAX_BATCHED_TOKENS=16384; ATTENTION="flashinfer"
+        KV_CACHE_DTYPE="auto"; CHUNKED_PREFILL=true; BLOCK_SIZE=16; SWAP_SPACE=4
+        MAX_SEQ_LEN_CAPTURE=8192; SPEC_MODEL=""; SPEC_TOKENS=5
+        ;;
+    fp8-8k-lo)
+        MODEL="$MODEL_FP8"; MAX_MODEL_LEN=8192; GPU_MEM_UTIL=0.85; DTYPE="auto"; QUANT="fp8"
+        MAX_NUM_SEQS=4; MAX_BATCHED_TOKENS=16384; ATTENTION="flashinfer"
+        KV_CACHE_DTYPE="auto"; CHUNKED_PREFILL=true; BLOCK_SIZE=16; SWAP_SPACE=4
+        MAX_SEQ_LEN_CAPTURE=8192; SPEC_MODEL=""; SPEC_TOKENS=5
+        ;;
+    *)
+        echo "❌ Config inconnue: $CONFIG"
+        echo ""
+        echo "Configs disponibles :"
+        echo "  ★ awq-8k        AWQ 4-bit, 8k ctx, flashinfer (recommandé)"
+        echo "    awq-16k        AWQ 4-bit, 16k ctx"
+        echo "    awq-8k-fast    AWQ 4-bit, 8k ctx, max throughput"
+        echo "    fp8-8k         FP8, 8k ctx"
+        echo "    fp8-16k        FP8, 16k ctx (limite)"
+        echo "    fp8-4k         FP8, 4k ctx"
+        echo "    fp8-8k-kvc     FP8 + FP8 KV cache"
+        echo "    gptq-8k        GPTQ Int4, 8k ctx"
+        echo "    spec-8k        AWQ + speculative 0.6B draft"
+        exit 1
+        ;;
+esac
+
+PORT="${PORT:-8000}"
+HOST="${HOST:-0.0.0.0}"
+
+echo "╔══════════════════════════════════════════════════════════╗"
+echo "║  vLLM Server — Qwen SpeedLab                            ║"
+echo "║  GPU: RTX 3090 (24 GB, SM 8.6)                          ║"
+echo "╚══════════════════════════════════════════════════════════╝"
+echo ""
+echo "  Config:          $CONFIG"
+echo "  Model:           $MODEL"
+echo "  Quantization:    $QUANT"
+echo "  Max model len:   $MAX_MODEL_LEN"
+echo "  GPU mem util:    $GPU_MEM_UTIL"
+echo "  Dtype:           $DTYPE"
+echo "  Max seqs:        $MAX_NUM_SEQS"
+echo "  Max batched:     $MAX_BATCHED_TOKENS"
+echo "  Attention:       $ATTENTION"
+echo "  KV cache dtype:  $KV_CACHE_DTYPE"
+echo "  Chunked prefill: $CHUNKED_PREFILL"
+echo "  Block size:      $BLOCK_SIZE"
+echo "  Swap space:      ${SWAP_SPACE} GB"
+if [[ -n "$SPEC_MODEL" ]]; then
+    echo "  Draft model:     $SPEC_MODEL ($SPEC_TOKENS speculative tokens)"
+fi
+echo "  Port:            $PORT"
+echo ""
+
+# --- Build vLLM command ---
+CMD=(
+    vllm serve "$MODEL"
+    --host "$HOST"
+    --port "$PORT"
+    --max-model-len "$MAX_MODEL_LEN"
+    --gpu-memory-utilization "$GPU_MEM_UTIL"
+    --dtype "$DTYPE"
+    --max-num-seqs "$MAX_NUM_SEQS"
+    --max-num-batched-tokens "$MAX_BATCHED_TOKENS"
+    --tensor-parallel-size 1
+    --trust-remote-code
+    --enable-prefix-caching
+    --block-size "$BLOCK_SIZE"
+    --swap-space "$SWAP_SPACE"
+    --max-seq-len-to-capture "$MAX_SEQ_LEN_CAPTURE"
+)
+
+# Attention backend
+if [[ -n "$ATTENTION" ]] && [[ "$ATTENTION" != "auto" ]]; then
+    CMD+=(--attention-backend "$ATTENTION")
+fi
+
+# FP8 KV cache (30% VRAM saving on KV cache)
+if [[ "$KV_CACHE_DTYPE" == "fp8" ]]; then
+    CMD+=(--kv-cache-dtype fp8)
+fi
+
+# Chunked prefill
+if [[ "$CHUNKED_PREFILL" == "true" ]]; then
+    CMD+=(--enable-chunked-prefill)
+fi
+
+# Speculative decoding
+if [[ -n "$SPEC_MODEL" ]]; then
+    CMD+=(--speculative-model "$SPEC_MODEL")
+    CMD+=(--num-speculative-tokens "$SPEC_TOKENS")
+fi
+
+echo "  Command:"
+echo "    ${CMD[*]}"
+echo ""
+echo "  Log: /tmp/vllm_${CONFIG}.log"
+echo ""
+
+# Launch
+"${CMD[@]}" 2>&1 | tee "/tmp/vllm_${CONFIG}.log"

scripts/serve_vllm_optimized.sh

@@ -0,0 +1,243 @@
+#!/usr/bin/env bash
+# ============================================================
+# serve_vllm_optimized.sh — vLLM with every RTX 3090 optimization
+#
+# This script applies ALL known optimizations for Ampere GPUs:
+#   1. FP8 KV Cache           → -30% KV cache VRAM
+#   2. FlashInfer backend     → faster prefill on SM 8.6
+#   3. Chunked prefill        → better scheduling under load
+#   4. Block size tuning      → larger blocks for long context
+#   5. CUDA graph capture     → matches max_model_len
+#   6. Swap space             → 4GB CPU overflow buffer
+#   7. AWQ/GPTQ Marlin        → INT4 tensor core kernels
+#   8. Prefix caching         → avoids recomputation
+#
+# Usage:
+#   bash scripts/serve_vllm_optimized.sh [config_name]
+#
+# Configs:
+#   awq-8k       — AWQ 4-bit, 8192 ctx, flashinfer, FP8 KV cache (★ recommended)
+#   awq-16k      — AWQ 4-bit, 16384 ctx
+#   awq-8k-fast  — AWQ 4-bit, 8192 ctx, max throughput tuning
+#   fp8-8k       — FP8 weight, 8192 ctx
+#   fp8-8k-kvc   — FP8 weight + FP8 KV cache (VRAM saving)
+#   gptq-8k      — GPTQ Marlin, 8192 ctx
+#   spec-8k      — speculative decoding with 0.6B draft model
+# ============================================================
+set -euo pipefail
+
+CONFIG="${1:-awq-8k}"
+PORT="${PORT:-8000}"
+HOST="${HOST:-0.0.0.0}"
+
+# --- Model paths ---
+MODEL_AWQ="QuantTrio/Qwen3.6-27B-AWQ"
+MODEL_FP8="Qwen/Qwen3.6-27B-FP8"
+MODEL_GPTQ="groxaxo/Qwen3.6-27B-GPTQ-Pro-4bit"
+MODEL_DRAFT="Qwen/Qwen2.5-0.5B-Instruct"  # for speculative decoding
+
+# --- Config matrix ---
+case "$CONFIG" in
+    # ============================================================
+    # AWQ — best performance/quality on Ampere (★ recommended)
+    # ============================================================
+    awq-8k)
+        MODEL="$MODEL_AWQ"
+        MAX_MODEL_LEN=8192
+        GPU_MEM=0.90
+        MAX_SEQS=8
+        MAX_BATCHED=32768
+        DTYPE="auto"
+        ATTENTION="flashinfer"
+        KV_CACHE_DTYPE="fp8"
+        CHUNKED_PREFILL=true
+        BLOCK_SIZE=16
+        SWAP_SPACE=4
+        SPEC_DRAFT=""
+        ;;
+    awq-8k-fast)
+        MODEL="$MODEL_AWQ"
+        MAX_MODEL_LEN=8192
+        GPU_MEM=0.90
+        MAX_SEQS=16
+        MAX_BATCHED=65536
+        DTYPE="auto"
+        ATTENTION="flashinfer"
+        KV_CACHE_DTYPE="fp8"
+        CHUNKED_PREFILL=true
+        BLOCK_SIZE=32
+        SWAP_SPACE=8
+        SPEC_DRAFT=""
+        ;;
+    awq-16k)
+        MODEL="$MODEL_AWQ"
+        MAX_MODEL_LEN=16384
+        GPU_MEM=0.90
+        MAX_SEQS=4
+        MAX_BATCHED=32768
+        DTYPE="auto"
+        ATTENTION="flashinfer"
+        KV_CACHE_DTYPE="fp8"
+        CHUNKED_PREFILL=true
+        BLOCK_SIZE=32
+        SWAP_SPACE=8
+        SPEC_DRAFT=""
+        ;;
+    # ============================================================
+    # FP8 — lighter weight quant, tight on 24GB
+    # ============================================================
+    fp8-8k)
+        MODEL="$MODEL_FP8"
+        MAX_MODEL_LEN=8192
+        GPU_MEM=0.88
+        MAX_SEQS=4
+        MAX_BATCHED=16384
+        DTYPE="auto"
+        ATTENTION="flashinfer"
+        KV_CACHE_DTYPE="auto"
+        CHUNKED_PREFILL=true
+        BLOCK_SIZE=16
+        SWAP_SPACE=4
+        SPEC_DRAFT=""
+        ;;
+    fp8-8k-kvc)
+        # FP8 weights + FP8 KV cache = tight fit but works
+        MODEL="$MODEL_FP8"
+        MAX_MODEL_LEN=8192
+        GPU_MEM=0.85
+        MAX_SEQS=6
+        MAX_BATCHED=16384
+        DTYPE="auto"
+        ATTENTION="flashinfer"
+        KV_CACHE_DTYPE="fp8"
+        CHUNKED_PREFILL=true
+        BLOCK_SIZE=16
+        SWAP_SPACE=4
+        SPEC_DRAFT=""
+        ;;
+    # ============================================================
+    # GPTQ — INT4 via Marlin kernels (fast on Ampere)
+    # ============================================================
+    gptq-8k)
+        MODEL="$MODEL_GPTQ"
+        MAX_MODEL_LEN=8192
+        GPU_MEM=0.90
+        MAX_SEQS=8
+        MAX_BATCHED=32768
+        DTYPE="auto"
+        ATTENTION="flashinfer"
+        KV_CACHE_DTYPE="fp8"
+        CHUNKED_PREFILL=true
+        BLOCK_SIZE=16
+        SWAP_SPACE=4
+        SPEC_DRAFT=""
+        ;;
+    # ============================================================
+    # Speculative decoding — Qwen 27B target + 0.6B draft
+    # Spec can 1.5x-2x throughput on batch workloads
+    # ============================================================
+    spec-8k)
+        MODEL="$MODEL_AWQ"
+        MAX_MODEL_LEN=8192
+        GPU_MEM=0.88
+        MAX_SEQS=4
+        MAX_BATCHED=16384
+        DTYPE="auto"
+        ATTENTION="flashinfer"
+        KV_CACHE_DTYPE="fp8"
+        CHUNKED_PREFILL=false
+        BLOCK_SIZE=16
+        SWAP_SPACE=4
+        SPEC_DRAFT="$MODEL_DRAFT"
+        ;;
+    *)
+        echo "❌ Unknown config: $CONFIG"
+        echo ""
+        echo "Available optimized configs:"
+        echo "  awq-8k        ★ AWQ 4-bit, 8k ctx, all optimizations (recommended)"
+        echo "  awq-8k-fast     AWQ 4-bit, 8k ctx, max throughput"
+        echo "  awq-16k         AWQ 4-bit, 16k ctx"
+        echo "  fp8-8k          FP8 weight, 8k ctx"
+        echo "  fp8-8k-kvc      FP8 weight + FP8 KV cache"
+        echo "  gptq-8k         GPTQ Marlin, 8k ctx"
+        echo "  spec-8k         Speculative decoding (AWQ + 0.6B draft)"
+        exit 1
+        ;;
+esac
+
+echo "╔══════════════════════════════════════════════════════════╗"
+echo "║  vLLM OPTIMIZED — Qwen SpeedLab                         ║"
+echo "║  GPU: RTX 3090 (24 GB, SM 8.6)                          ║"
+echo "╚══════════════════════════════════════════════════════════╝"
+echo ""
+echo "  Config:          $CONFIG"
+echo "  Model:           $MODEL"
+echo "  Max model len:   $MAX_MODEL_LEN"
+echo "  GPU mem util:    $GPU_MEM"
+echo "  Max seqs:        $MAX_SEQS"
+echo "  Max batched:     $MAX_BATCHED"
+echo "  Dtype:           $DTYPE"
+echo "  Attention:       $ATTENTION"
+echo "  KV cache dtype:  $KV_CACHE_DTYPE"
+echo "  Chunked prefill: $CHUNKED_PREFILL"
+echo "  Block size:      $BLOCK_SIZE"
+echo "  Swap space:      ${SWAP_SPACE}GB"
+if [[ -n "$SPEC_DRAFT" ]]; then
+    echo "  Draft model:     $SPEC_DRAFT"
+fi
+echo ""
+
+# --- Build vllm command ---
+CMD=(
+    vllm serve "$MODEL"
+    --host "$HOST"
+    --port "$PORT"
+    --max-model-len "$MAX_MODEL_LEN"
+    --gpu-memory-utilization "$GPU_MEM"
+    --dtype "$DTYPE"
+    --max-num-seqs "$MAX_SEQS"
+    --max-num-batched-tokens "$MAX_BATCHED"
+    --tensor-parallel-size 1
+    --trust-remote-code
+    --enable-prefix-caching
+)
+
+# Attention backend
+if [[ "$ATTENTION" == "flashinfer" ]]; then
+    CMD+=(--attention-backend flashinfer)
+fi
+
+# FP8 KV cache (biggest VRAM saving on Ampere)
+if [[ "$KV_CACHE_DTYPE" == "fp8" ]]; then
+    CMD+=(--kv-cache-dtype fp8)
+fi
+
+# Chunked prefill
+if [[ "$CHUNKED_PREFILL" == "true" ]]; then
+    CMD+=(--enable-chunked-prefill)
+fi
+
+# Block size
+CMD+=(--block-size "$BLOCK_SIZE")
+
+# Swap space (CPU offload safety net)
+CMD+=(--swap-space "$SWAP_SPACE")
+
+# CUDA graph: capture up to max_model_len (set equal to avoid recompilation)
+CMD+=(--max-seq-len-to-capture "$MAX_MODEL_LEN")
+
+# Speculative decoding
+if [[ -n "$SPEC_DRAFT" ]]; then
+    CMD+=(--speculative-model "$SPEC_DRAFT")
+    # num_speculative_tokens: 5 is a good default for Qwen
+    CMD+=(--num-speculative-tokens 5)
+fi
+
+echo "  Command:"
+echo "    ${CMD[*]}"
+echo ""
+echo "  Log: /tmp/vllm_optimized_${CONFIG}.log"
+echo ""
+
+# Launch
+"${CMD[@]}" 2>&1 | tee "/tmp/vllm_optimized_${CONFIG}.log"

scripts/setup_runpod.sh

@@ -0,0 +1,144 @@
+#!/usr/bin/env bash
+# ============================================================
+# setup_runpod.sh — Installation environment Qwen SpeedLab
+# Pour RTX 3090 / RunPod avec CUDA 12.x
+# ============================================================
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PROJECT_DIR="$(dirname "$SCRIPT_DIR")"
+VENV_DIR="${PROJECT_DIR}/.venv"
+
+echo "============================================"
+echo " Qwen SpeedLab — Setup RunPod RTX 3090"
+echo "============================================"
+
+# --- 1. Vérifier GPU --------------------------------------------------
+echo ""
+echo "[1/8] Vérification GPU..."
+if command -v nvidia-smi &>/dev/null; then
+    nvidia-smi --query-gpu=name,memory.total,driver_version,cuda.version --format=csv,noheader
+    GPU_NAME=$(nvidia-smi --query-gpu=name --format=csv,noheader | head -1)
+    VRAM_MB=$(nvidia-smi --query-gpu=memory.total --format=csv,noheader | head -1 | sed 's/ MiB//')
+    echo "  GPU : $GPU_NAME"
+    echo "  VRAM: ${VRAM_MB} MiB"
+
+    if [[ "$VRAM_MB" -lt 23000 ]]; then
+        echo "  ⚠️  VRAM < 23 Go — les modèles 27B risquent de ne pas passer"
+    fi
+else
+    echo "  ❌ nvidia-smi introuvable — pas de GPU NVIDIA détecté"
+    exit 1
+fi
+
+# --- 2. Vérifier CUDA / drivers ---------------------------------------
+echo ""
+echo "[2/8] Vérification CUDA..."
+if command -v nvcc &>/dev/null; then
+    nvcc --version | grep "release" || true
+else
+    echo "  nvcc non trouvé (normal sur RunPod, le driver suffit)"
+fi
+
+# Vérifier que les libs CUDA sont accessibles
+CUDA_HOME="${CUDA_HOME:-/usr/local/cuda}"
+if [[ -d "$CUDA_HOME" ]]; then
+    echo "  CUDA_HOME: $CUDA_HOME"
+    ls "$CUDA_HOME/lib64/libcudart.so"* 2>/dev/null || echo "  ⚠️  libcudart non trouvée"
+fi
+
+# --- 3. Créer venv Python ----------------------------------------------
+echo ""
+echo "[3/8] Création environnement virtuel Python..."
+python3 -m venv "$VENV_DIR" --upgrade-deps
+source "${VENV_DIR}/bin/activate"
+echo "  Python: $(python3 --version)"
+echo "  pip:    $(pip --version)"
+
+# --- 4. Installer PyTorch compatible CUDA ------------------------------
+echo ""
+echo "[4/8] Installation PyTorch + CUDA..."
+# Détecter la version CUDA du driver
+CUDA_VER=$(nvidia-smi --query-gpu=cuda.version --format=csv,noheader | head -1 | cut -d. -f1)
+echo "  Driver CUDA major: $CUDA_VER"
+
+# PyTorch avec CUDA 12.4 (stable avec vLLM)
+pip install --upgrade pip setuptools wheel
+pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124
+
+# Vérifier que torch voit CUDA
+python3 -c "
+import torch
+print(f'  PyTorch: {torch.__version__}')
+print(f'  CUDA available: {torch.cuda.is_available()}')
+if torch.cuda.is_available():
+    print(f'  CUDA version: {torch.version.cuda}')
+    print(f'  Device count: {torch.cuda.device_count()}')
+    print(f'  Device name:  {torch.cuda.get_device_name(0)}')
+    print(f'  VRAM total:   {torch.cuda.get_device_properties(0).total_memory / 1024**3:.1f} Go')
+else:
+    print('  ⚠️  CUDA non disponible — vérifier driver/compatibilité')
+"
+
+# --- 5. Installer vLLM -------------------------------------------------
+echo ""
+echo "[5/8] Installation vLLM..."
+# vLLM avec CUDA 12.4
+pip install vllm --no-build-isolation 2>&1 | tail -5 || {
+    echo "  ⚠️  Échec vLLM from source, tentative via wheel..."
+    pip install vllm 2>&1 | tail -5 || {
+        echo "  ❌ Échec installation vLLM"
+        echo "  → Essayez: pip install vllm --find-links https://vllm-wheels.s3.us-west-2.amazonaws.com/nightly.html"
+    }
+}
+
+# Vérifier
+python3 -c "import vllm; print(f'  vLLM: {vllm.__version__}')" 2>/dev/null || echo "  ⚠️  vLLM non importable"
+
+# --- 6. Installer SGLang -----------------------------------------------
+echo ""
+echo "[6/8] Installation SGLang..."
+pip install "sglang[all]" 2>&1 | tail -5 || {
+    echo "  ⚠️  Échec SGLang — non bloquant"
+}
+
+python3 -c "import sglang; print(f'  SGLang: {sglang.__version__}')" 2>/dev/null || echo "  ⚠️  SGLang non importable"
+
+# --- 7. Installer dépendances benchmark --------------------------------
+echo ""
+echo "[7/8] Installation dépendances benchmark..."
+pip install transformers accelerate datasets pandas tqdm psutil openai httpx aiohttp nvidia-ml-py pyyaml rich
+
+# --- 8. Vérification finale --------------------------------------------
+echo ""
+echo "[8/8] Vérification finale..."
+echo ""
+echo "  Packages installés :"
+pip list 2>/dev/null | grep -iE "torch|vllm|sglang|transformers|openai" || true
+echo ""
+
+# Vérifier HF_TOKEN
+if [[ -z "${HF_TOKEN:-}" ]]; then
+    echo "  ⚠️  HF_TOKEN non défini."
+    echo "      → export HF_TOKEN='hf_votre_token'"
+    echo "      → ou lancez : huggingface-cli login"
+else
+    echo "  ✅ HF_TOKEN détecté"
+fi
+
+# Vérifier espace disque
+DISK_AVAIL=$(df -h /workspace | tail -1 | awk '{print $4}')
+echo "  Espace disque disponible: $DISK_AVAIL"
+
+echo ""
+echo "============================================"
+echo " ✅ Setup terminé !"
+echo ""
+echo "  Pour activer l'environnement :"
+echo "    source ${VENV_DIR}/bin/activate"
+echo ""
+echo "  Pour lancer le benchmark :"
+echo "    bash scripts/serve_vllm.sh"
+echo "    python scripts/bench_openai_api.py"
+echo "    python scripts/sweep_vllm_configs.py"
+echo "============================================"

scripts/sweep_advanced.py

@@ -0,0 +1,801 @@
+#!/usr/bin/env python3
+"""
+sweep_advanced.py — Advanced vLLM optimization sweep for RTX 3090.
+
+Tests each optimization dimension independently to measure its impact:
+  1. Attention backend: flash-attn vs flashinfer
+  2. KV cache dtype: auto vs fp8
+  3. Chunked prefill: on vs off
+  4. Block size: 8 vs 16 vs 32
+  5. GPU memory utilization: 0.85 vs 0.88 vs 0.90 vs 0.92
+  6. max_num_seqs vs max_batched_tokens grid
+  7. AWQ vs GPTQ vs FP8 model comparison
+  8. Speculative decoding (optional, if draft model available)
+
+Usage:
+    # Baseline: test each optimization in isolation
+    python scripts/sweep_advanced.py --mode baseline
+
+    # Model shootout: compare AWQ vs GPTQ vs FP8
+    python scripts/sweep_advanced.py --mode shootout
+
+    # Full grid: exhaustively test combinations
+    python scripts/sweep_advanced.py --mode grid
+
+    # Single test with full instrumentation
+    python scripts/sweep_advanced.py --mode single --config awq-flashinfer-fp8kv
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import signal
+import socket
+import subprocess
+import sys
+import time
+from dataclasses import dataclass, field, asdict
+from datetime import datetime
+from pathlib import Path
+from typing import Optional
+
+import httpx
+
+PROJECT_DIR = Path(__file__).resolve().parent.parent
+REPORTS_DIR = PROJECT_DIR / "reports"
+PROMPTS_FILE = PROJECT_DIR / "prompts" / "bench_prompts.jsonl"
+RESULTS_CSV = REPORTS_DIR / "results_advanced.csv"
+SUMMARY_MD = REPORTS_DIR / "summary_advanced.md"
+
+# ---- GPU warmup / profiler ----
+try:
+    import pynvml
+    HAS_PYNVML = True
+except ImportError:
+    HAS_PYNVML = False
+
+# ---- vLLM availability ----
+VLLM_AVAILABLE = False
+try:
+    import vllm
+    VLLM_AVAILABLE = True
+except ImportError:
+    pass
+
+
+@dataclass
+class OptimizedConfig:
+    """A single vLLM config to benchmark."""
+    name: str
+    model_path: str
+    quant_method: str          # awq, fp8, gptq, none
+    max_model_len: int = 8192
+    gpu_memory_utilization: float = 0.90
+    dtype: str = "auto"
+    max_num_seqs: int = 8
+    max_num_batched_tokens: int = 32768
+    attention_backend: str = "flashinfer"   # flashinfer, flash-attn, auto
+    kv_cache_dtype: str = "auto"            # auto, fp8
+    enable_chunked_prefill: bool = False
+    block_size: int = 16
+    swap_space: int = 4
+    enable_prefix_caching: bool = True
+    max_seq_len_to_capture: int = 8192
+    # Speculative decoding
+    speculative_model: str = ""
+    num_speculative_tokens: int = 5
+
+    @property
+    def label(self) -> str:
+        parts = [self.quant_method, f"len{self.max_model_len}"]
+        if self.attention_backend != "auto":
+            parts.append(self.attention_backend.replace("-", ""))
+        if self.kv_cache_dtype == "fp8":
+            parts.append("fp8kv")
+        if self.enable_chunked_prefill:
+            parts.append("chunk")
+        parts.append(f"bs{self.block_size}")
+        parts.append(f"s{self.max_num_seqs}")
+        return "-".join(parts)
+
+    def to_server_flags(self, port: int = 8000) -> list[str]:
+        """Convert to vLLM CLI arguments."""
+        flags = [
+            "--model", self.model_path,
+            "--host", "0.0.0.0",
+            "--port", str(port),
+            "--max-model-len", str(self.max_model_len),
+            "--gpu-memory-utilization", str(self.gpu_memory_utilization),
+            "--dtype", self.dtype,
+            "--max-num-seqs", str(self.max_num_seqs),
+            "--max-num-batched-tokens", str(self.max_num_batched_tokens),
+            "--tensor-parallel-size", "1",
+            "--trust-remote-code",
+            "--block-size", str(self.block_size),
+            "--swap-space", str(self.swap_space),
+            "--max-seq-len-to-capture", str(self.max_seq_len_to_capture),
+        ]
+        if self.enable_prefix_caching:
+            flags.append("--enable-prefix-caching")
+        if self.enable_chunked_prefill:
+            flags.append("--enable-chunked-prefill")
+        if self.attention_backend not in ("auto", ""):
+            flags.extend(["--attention-backend", self.attention_backend])
+        if self.kv_cache_dtype not in ("auto", ""):
+            flags.extend(["--kv-cache-dtype", self.kv_cache_dtype])
+        if self.speculative_model:
+            flags.extend(["--speculative-model", self.speculative_model])
+            flags.extend(["--num-speculative-tokens", str(self.num_speculative_tokens)])
+        return flags
+
+
+# ============================================================
+# Config presets for each sweep mode
+# ============================================================
+
+# Models — best to worst for RTX 3090
+MODELS = {
+    "awq":  "QuantTrio/Qwen3.6-27B-AWQ",
+    "gptq": "groxaxo/Qwen3.6-27B-GPTQ-Pro-4bit",
+    "fp8":  "Qwen/Qwen3.6-27B-FP8",
+    "base": "Qwen/Qwen3.6-27B",
+}
+
+
+def baseline_configs() -> list[OptimizedConfig]:
+    """
+    A/B test each optimization dimension independently.
+    All configs use AWQ as the base model (best for Ampere).
+    """
+    BASE = dict(
+        model_path=MODELS["awq"], quant_method="awq",
+        max_model_len=8192, gpu_memory_utilization=0.90,
+        max_num_seqs=8, max_num_batched_tokens=32768,
+    )
+    configs = []
+
+    # ---- 1. Attention backend shootout ----
+    for attn in ["flashinfer", "flash-attn"]:
+        c = OptimizedConfig(name=f"attn-{attn}", attention_backend=attn, **BASE)
+        configs.append(c)
+
+    # ---- 2. FP8 KV cache (VRAM saving) ----
+    for kv in ["auto", "fp8"]:
+        c = OptimizedConfig(name=f"kv-{kv}", kv_cache_dtype=kv, **BASE)
+        configs.append(c)
+
+    # ---- 3. Chunked prefill ----
+    for cp in [False, True]:
+        tag = "chunk-on" if cp else "chunk-off"
+        c = OptimizedConfig(name=tag, enable_chunked_prefill=cp, **BASE)
+        configs.append(c)
+
+    # ---- 4. Block size sweep ----
+    for bs in [8, 16, 32]:
+        c = OptimizedConfig(name=f"blksz-{bs}", block_size=bs, **BASE)
+        configs.append(c)
+
+    # ---- 5. GPU memory utilization sweep ----
+    for mem in [0.85, 0.88, 0.90, 0.92]:
+        c = OptimizedConfig(name=f"gpumem-{int(mem*100)}", gpu_memory_utilization=mem, **BASE)
+        configs.append(c)
+
+    # ---- 6. Batch size sweep ----
+    for seqs, batched in [(4, 16384), (8, 32768), (12, 49152)]:
+        c = OptimizedConfig(
+            name=f"batch-s{seqs}-b{batched}",
+            max_num_seqs=seqs, max_num_batched_tokens=batched, **BASE,
+        )
+        configs.append(c)
+
+    # ---- 7. Combined best guess ----
+    c = OptimizedConfig(
+        name="combined-best",
+        attention_backend="flashinfer",
+        kv_cache_dtype="fp8",
+        enable_chunked_prefill=True,
+        block_size=16,
+        max_num_seqs=8,
+        max_num_batched_tokens=32768,
+        **BASE,
+    )
+    configs.append(c)
+
+    return configs
+
+
+def model_shootout_configs() -> list[OptimizedConfig]:
+    """Compare AWQ vs GPTQ vs FP8 on equal footing."""
+    configs = []
+
+    COMMON = dict(
+        max_model_len=8192, gpu_memory_utilization=0.90,
+        max_num_seqs=8, max_num_batched_tokens=32768,
+        attention_backend="flashinfer", kv_cache_dtype="fp8",
+        enable_chunked_prefill=True,
+    )
+
+    # AWQ — expected best
+    configs.append(OptimizedConfig(
+        name="shootout-awq", model_path=MODELS["awq"], quant_method="awq", **COMMON,
+    ))
+
+    # GPTQ Marlin — potentially faster on Ampere
+    configs.append(OptimizedConfig(
+        name="shootout-gptq", model_path=MODELS["gptq"], quant_method="gptq", **COMMON,
+    ))
+
+    # FP8 — higher precision, tighter on VRAM
+    configs.append(OptimizedConfig(
+        name="shootout-fp8", model_path=MODELS["fp8"], quant_method="fp8",
+        max_num_seqs=6, max_num_batched_tokens=16384, **COMMON,
+    ))
+
+    # FP8 + FP8 KV cache
+    configs.append(OptimizedConfig(
+        name="shootout-fp8-fp8kv", model_path=MODELS["fp8"], quant_method="fp8",
+        max_num_seqs=8, max_num_batched_tokens=32768, **COMMON,
+    ))
+
+    return configs
+
+
+def grid_configs() -> list[OptimizedConfig]:
+    """Full grid search over the most impactful dimensions. ~20 configs."""
+    configs = []
+
+    for model_key, model_path in [("awq", MODELS["awq"]), ("gptq", MODELS["gptq"])]:
+        for attn in ["flashinfer", "flash-attn"]:
+            for kv in ["fp8", "auto"]:
+                for bs in [16, 32]:
+                    for seqs in [8, 12]:
+                        c = OptimizedConfig(
+                            name=f"grid-{model_key}-{attn}-{kv}kv-bs{bs}-s{seqs}",
+                            model_path=model_path,
+                            quant_method=model_key,
+                            attention_backend=attn,
+                            kv_cache_dtype=kv,
+                            block_size=bs,
+                            max_num_seqs=seqs,
+                            max_num_batched_tokens=seqs * 4096,
+                            max_model_len=8192,
+                            gpu_memory_utilization=0.90,
+                            enable_chunked_prefill=True,
+                        )
+                        configs.append(c)
+
+    return configs
+
+
+# ============================================================
+# Server lifecycle
+# ============================================================
+
+class VllmServer:
+    """Manage a vLLM server process."""
+
+    def __init__(self, config: OptimizedConfig, port: int = 8000):
+        self.config = config
+        self.port = port
+        self.process: Optional[subprocess.Popen] = None
+        self.logfile = Path(f"/tmp/vllm_adv_{config.name}.log")
+
+    def start(self):
+        cmd = [sys.executable, "-m", "vllm.entrypoints.openai.api_server"]
+        cmd.extend(self.config.to_server_flags(port=self.port))
+
+        print(f"  {' '.join(cmd)}")
+
+        self.logfile.parent.mkdir(parents=True, exist_ok=True)
+        fout = open(str(self.logfile), "w")
+        self.process = subprocess.Popen(
+            cmd, stdout=fout, stderr=subprocess.STDOUT,
+            start_new_session=True,
+            env={**os.environ, "VLLM_ATTENTION_BACKEND": self.config.attention_backend},
+        )
+        return True
+
+    def wait_ready(self, timeout: int = 600) -> tuple[bool, str]:
+        client = httpx.Client(base_url=f"http://localhost:{self.port}", timeout=httpx.Timeout(8.0))
+        deadline = time.time() + timeout
+        while time.time() < deadline:
+            if self.process and self.process.poll() is not None:
+                tail = self._log_tail(60)
+                return False, f"CRASH rc={self.process.returncode}\n{tail}"
+            try:
+                resp = client.get("/v1/models")
+                if resp.status_code == 200:
+                    client.close()
+                    return True, "OK"
+            except Exception:
+                pass
+            time.sleep(5)
+        client.close()
+        return False, f"TIMEOUT ({timeout}s)\n{self._log_tail(30)}"
+
+    def stop(self):
+        if not self.process:
+            return
+        print("  ⏹️  Stopping server...")
+        try:
+            os.killpg(os.getpgid(self.process.pid), signal.SIGTERM)
+            self.process.wait(timeout=30)
+        except subprocess.TimeoutExpired:
+            os.killpg(os.getpgid(self.process.pid), signal.SIGKILL)
+            self.process.wait(timeout=10)
+        except Exception:
+            pass
+        self.process = None
+        time.sleep(5)
+
+    def _log_tail(self, lines: int = 40) -> str:
+        if not self.logfile.exists():
+            return "(no log)"
+        try:
+            with open(self.logfile, "r") as f:
+                return "".join(f.readlines()[-lines:])
+        except Exception:
+            return "(error)"
+
+
+# ============================================================
+# Benchmark integration
+# ============================================================
+
+@dataclass
+class SweepResult:
+    config: OptimizedConfig
+    success: bool
+    error: str = ""
+    avg_output_tps: float = 0.0
+    avg_total_tps: float = 0.0
+    avg_ttft_ms: float = 0.0
+    avg_tpot_ms: float = 0.0
+    peak_vram_mb: float = 0.0
+    avg_gpu_util_pct: float = 0.0
+    avg_power_w: float = 0.0
+    ok: int = 0
+    fail: int = 0
+    duration_s: float = 0.0
+
+
+def parse_csv_for_config(csv_path: str, config_name: str) -> Optional[dict]:
+    """Aggregate results from CSV for a config."""
+    if not os.path.exists(csv_path):
+        return None
+    import csv
+    try:
+        with open(csv_path, "r") as f:
+            rows = [r for r in csv.DictReader(f) if r.get("config_name") == config_name]
+    except Exception:
+        return None
+
+    if not rows:
+        # Fallback: take last 30 entries
+        with open(csv_path, "r") as f:
+            rows = list(csv.DictReader(f))[-30:]
+
+    ok = [r for r in rows if r.get("success") == "True"]
+    if not ok:
+        return {"ok": 0, "fail": len(rows)}
+
+    def avg(key):
+        vals = [float(r[key]) for r in ok if r.get(key)]
+        return sum(vals) / len(vals) if vals else 0.0
+
+    def pct(key):
+        vals = [float(r[key]) for r in ok if r.get(key) and float(r.get(key)) > 0]
+        return sum(vals) / len(vals) if vals else 0.0
+
+    def maxv(key):
+        vals = [float(r[key]) for r in ok if r.get(key)]
+        return max(vals) if vals else 0.0
+
+    return {
+        "ok": len(ok),
+        "fail": len(rows) - len(ok),
+        "avg_output_tps": avg("output_tps"),
+        "avg_total_tps": avg("total_tps"),
+        "avg_ttft_ms": avg("ttft_ms"),
+        "avg_tpot_ms": avg("tpot_ms"),
+        "peak_vram_mb": maxv("vram_peak_mb"),
+        "avg_gpu_util_pct": pct("gpu_util_pct"),
+        "avg_power_w": pct("power_w"),
+    }
+
+
+def run_benchmark_subprocess(config: OptimizedConfig, port: int = 8000) -> tuple[bool, str]:
+    """Run bench_openai_api.py as a subprocess."""
+    bench_script = PROJECT_DIR / "scripts" / "bench_openai_api.py"
+    cmd = [
+        sys.executable, str(bench_script),
+        "--url", f"http://localhost:{port}",
+        "--prompts", str(PROMPTS_FILE),
+        "--output", str(RESULTS_CSV),
+        "--model", config.model_path,
+        "--backend", "vllm",
+        "--config", config.name,
+        "--max-model-len", str(config.max_model_len),
+        "--gpu-mem-util", str(config.gpu_memory_utilization),
+        "--dtype", config.dtype,
+        "--quantization", config.quant_method,
+        "--repeat", "2",
+    ]
+    print(f"  🔬 Running benchmark...")
+    try:
+        result = subprocess.run(cmd, capture_output=True, text=True, timeout=600)
+        if result.returncode == 0:
+            # Extract key lines
+            for line in result.stdout.split("\n"):
+                if any(kw in line for kw in ["Output TPS:", "TTFT", "SUMMARY", "Success"]):
+                    print(f"    {line.strip()}")
+            return True, ""
+        else:
+            err = result.stderr[-500:] if result.stderr else result.stdout[-500:]
+            return False, err
+    except subprocess.TimeoutExpired:
+        return False, "Benchmark subprocess timed out"
+    except Exception as e:
+        return False, str(e)
+
+
+# ============================================================
+# Orchestrator
+# ============================================================
+
+def sweep(configs: list[OptimizedConfig], port: int = 8000) -> list[SweepResult]:
+    results: list[SweepResult] = []
+    REPORTS_DIR.mkdir(parents=True, exist_ok=True)
+
+    # Backup old CSV
+    if RESULTS_CSV.exists():
+        RESULTS_CSV.rename(RESULTS_CSV.with_suffix(".csv.bak"))
+
+    print(f"\n{'='*70}")
+    print(f" ADVANCED OPTIMIZATION SWEEP — {len(configs)} configs")
+    print(f"{'='*70}")
+    for i, c in enumerate(configs, 1):
+        print(f"  [{i:2d}] {c.name:50s}  ({c.label[:70]})")
+    print(f"{'='*70}\n")
+
+    for idx, config in enumerate(configs, 1):
+        result = SweepResult(config=config, success=False)
+        t0 = time.time()
+
+        print(f"\n{'#'*70}")
+        print(f" [{idx}/{len(configs)}] {config.name}")
+        print(f"  attn={config.attention_backend}  kv={config.kv_cache_dtype}  "
+              f"chunk={config.enable_chunked_prefill}  bs={config.block_size}  "
+              f"seqs={config.max_num_seqs}")
+        print(f"{'#'*70}")
+
+        # Start server
+        server = VllmServer(config, port=port)
+        try:
+            server.start()
+        except Exception as e:
+            result.error = f"START FAIL: {e}"
+            results.append(result)
+            continue
+
+        # Wait for ready
+        ok, msg = server.wait_ready()
+        if not ok:
+            result.error = msg[:500]
+            result.duration_s = time.time() - t0
+            server.stop()
+            results.append(result)
+            print(f"  ❌ Failed to start: {msg[:200]}")
+            continue
+
+        print(f"  ✅ Server ready")
+
+        # Run benchmark
+        bench_ok, bench_err = run_benchmark_subprocess(config, port=port)
+
+        # Parse results
+        metrics = parse_csv_for_config(str(RESULTS_CSV), config.name)
+        if metrics:
+            result.success = metrics.get("ok", 0) > 0
+            result.avg_output_tps = metrics.get("avg_output_tps", 0)
+            result.avg_total_tps = metrics.get("avg_total_tps", 0)
+            result.avg_ttft_ms = metrics.get("avg_ttft_ms", 0)
+            result.avg_tpot_ms = metrics.get("avg_tpot_ms", 0)
+            result.peak_vram_mb = metrics.get("peak_vram_mb", 0)
+            result.avg_gpu_util_pct = metrics.get("avg_gpu_util_pct", 0)
+            result.avg_power_w = metrics.get("avg_power_w", 0)
+            result.ok = metrics.get("ok", 0)
+            result.fail = metrics.get("fail", 0)
+
+        if not bench_ok:
+            result.error = bench_err[:500]
+            if not result.success:
+                result.error = bench_err[:500] or result.error
+
+        result.duration_s = time.time() - t0
+        server.stop()
+
+        status = "✅" if result.success else "❌"
+        print(f"  {status} TPS={result.avg_output_tps:.1f}  TTFT={result.avg_ttft_ms:.0f}ms  "
+              f"VRAM={result.peak_vram_mb:.0f}MB  [{result.ok}/{result.ok + result.fail}]  "
+              f"({result.duration_s:.0f}s)")
+
+        results.append(result)
+
+    return results
+
+
+# ============================================================
+# Report generation
+# ============================================================
+
+def generate_report(results: list[SweepResult], mode: str):
+    ok = [r for r in results if r.success]
+    fail = [r for r in results if not r.success]
+    ok_sorted = sorted(ok, key=lambda r: r.avg_output_tps, reverse=True)
+
+    lines = [
+        f"# Qwen SpeedLab — Advanced Optimization Report",
+        f"",
+        f"**Date:** {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}",
+        f"**GPU:** NVIDIA GeForce RTX 3090 (24 GB, SM 8.6)",
+        f"**Mode:** {mode}",
+        f"**Configs:** {len(results)} tested — {len(ok)} passed, {len(fail)} failed",
+        f"",
+        "---",
+        f"",
+    ]
+
+    if ok_sorted:
+        best = ok_sorted[0]
+        lines += [
+            f"## 🏆 Best Configuration",
+            f"",
+            f"```",
+            f"  Config name:      {best.config.name}",
+            f"  Model:            {best.config.model_path}",
+            f"  Attention:        {best.config.attention_backend}",
+            f"  KV cache dtype:   {best.config.kv_cache_dtype}",
+            f"  Chunked prefill:  {best.config.enable_chunked_prefill}",
+            f"  Block size:       {best.config.block_size}",
+            f"  Max seqs:         {best.config.max_num_seqs}",
+            f"  Max batched tok:  {best.config.max_num_batched_tokens}",
+            f"  GPU mem util:     {best.config.gpu_memory_utilization}",
+            f"```",
+            f"",
+            f"| Metric | Value |",
+            f"|--------|-------|",
+            f"| **Output TPS** | **{best.avg_output_tps:.1f} tok/s** |",
+            f"| Total TPS | {best.avg_total_tps:.1f} tok/s |",
+            f"| TTFT | {best.avg_ttft_ms:.0f} ms |",
+            f"| TPOT | {best.avg_tpot_ms:.1f} ms |",
+            f"| Peak VRAM | {best.peak_vram_mb:.0f} MB |",
+            f"| GPU util | {best.avg_gpu_util_pct:.1f}% |",
+            f"| Power | {best.avg_power_w:.1f} W |",
+            f"| Prompt success | {best.ok}/{best.ok + best.fail} |",
+            f"| Test duration | {best.duration_s:.0f}s |",
+            f"",
+        ]
+
+    # ---- Optimization dimension analysis ----
+    lines += [
+        "## 📊 Optimization Impact Analysis",
+        "",
+        "Each table shows the effect of toggling one optimization dimension.",
+        "",
+    ]
+
+    # Group by dimension
+    dimensions = {
+        "attention_backend": "Attention Backend",
+        "kv_cache_dtype": "KV Cache Dtype",
+        "enable_chunked_prefill": "Chunked Prefill",
+        "block_size": "Block Size",
+        "gpu_memory_utilization": "GPU Memory Utilization",
+        "max_num_seqs": "Max Concurrent Sequences",
+    }
+
+    for dim, title in dimensions.items():
+        groups: dict[str, list[SweepResult]] = {}
+        for r in ok:
+            key = str(getattr(r.config, dim, "?"))
+            groups.setdefault(key, []).append(r)
+
+        if len(groups) <= 1:
+            continue
+
+        lines += [f"### {title}", ""]
+        lines += ["| Value | Avg TPS | TTFT ms | VRAM MB | Tests |",
+                   "|-------|---------|---------|---------|-------|"]
+        for key in sorted(groups.keys()):
+            grp = groups[key]
+            avg_tps = sum(r.avg_output_tps for r in grp) / len(grp)
+            avg_ttft = sum(r.avg_ttft_ms for r in grp) / len(grp)
+            avg_vram = sum(r.peak_vram_mb for r in grp) / len(grp)
+            n = sum(r.ok for r in grp)
+            lines.append(f"| `{key}` | {avg_tps:.1f} | {avg_ttft:.0f} | {avg_vram:.0f} | {n} |")
+        lines.append("")
+
+    # ---- Full results table ----
+    lines += [
+        "## 📋 All Configurations",
+        "",
+        "| # | Name | Attn | KV | Chunk | BS | Seqs | TPS | TTFT | VRAM | Status |",
+        "|---|------|------|----|-------|----|------|-----|------|------|--------|",
+    ]
+    for i, r in enumerate(results, 1):
+        if r.success:
+            tps = f"{r.avg_output_tps:.1f}"
+            ttft = f"{r.avg_ttft_ms:.0f}"
+            vram = f"{r.peak_vram_mb:.0f}"
+            status = "✅"
+        else:
+            tps = ttft = vram = "-"
+            status = "❌"
+        lines.append(
+            f"| {i} | {r.config.name} | {r.config.attention_backend} | {r.config.kv_cache_dtype} | "
+            f"{r.config.enable_chunked_prefill} | {r.config.block_size} | {r.config.max_num_seqs} | "
+            f"{tps} | {ttft} | {vram} | {status} |"
+        )
+    lines.append("")
+
+    # ---- Failed ----
+    if fail:
+        lines += ["## ❌ Failed Configurations", ""]
+        for r in fail:
+            lines.append(f"- **{r.config.name}**: `{r.error[:200]}`")
+        lines.append("")
+
+    # ---- Recommendations ----
+    lines += [
+        "## 💡 Recommendations for RTX 3090",
+        "",
+    ]
+
+    # Analyze what worked best
+    if ok_sorted:
+        flashinfer_ok = [r for r in ok if r.config.attention_backend == "flashinfer"]
+        flashinfer_tps = sum(r.avg_output_tps for r in flashinfer_ok) / len(flashinfer_ok) if flashinfer_ok else 0
+        flashattn_ok = [r for r in ok if r.config.attention_backend == "flash-attn"]
+        flashattn_tps = sum(r.avg_output_tps for r in flashattn_ok) / len(flashattn_ok) if flashattn_ok else 0
+
+        fp8kv_ok = [r for r in ok if r.config.kv_cache_dtype == "fp8"]
+        fp8kv_tps = sum(r.avg_output_tps for r in fp8kv_ok) / len(fp8kv_ok) if fp8kv_ok else 0
+        auto_ok = [r for r in ok if r.config.kv_cache_dtype == "auto"]
+        auto_tps = sum(r.avg_output_tps for r in auto_ok) / len(auto_ok) if auto_ok else 0
+
+        lines.append("1. **Attention backend**:")
+        if flashinfer_tps > flashattn_tps:
+            lines.append(f"   FlashInfer is faster ({flashinfer_tps:.1f} vs {flashattn_tps:.1f} tok/s). Use `--attention-backend flashinfer`.")
+        else:
+            lines.append(f"   FlashAttention is comparable ({flashattn_tps:.1f} vs {flashinfer_tps:.1f} tok/s). Default is fine.")
+
+        lines.append("")
+        lines.append("2. **KV Cache dtype**:")
+        if fp8kv_tps > auto_tps * 0.95:
+            lines.append(f"   FP8 KV cache saves VRAM with minimal quality loss ({fp8kv_tps:.1f} vs {auto_tps:.1f} tok/s). Use `--kv-cache-dtype fp8`.")
+        else:
+            lines.append(f"   FP8 KV cache costs speed ({fp8kv_tps:.1f} vs {auto_tps:.1f} tok/s). Keep auto unless you need the VRAM.")
+
+        best_quant = ok_sorted[0].config.quant_method
+        lines.append("")
+        lines.append(f"3. **Best quantization**: `{best_quant}` — based on measured throughput.")
+
+        best_bs = max(set(r.config.block_size for r in ok), key=lambda bs: sum(r.avg_output_tps for r in ok if r.config.block_size == bs) / max(1, len([r for r in ok if r.config.block_size == bs])))
+        lines.append(f"4. **Optimal block size**: `{best_bs}`")
+
+        best_seqs = max(set(r.config.max_num_seqs for r in ok), key=lambda s: sum(r.avg_output_tps for r in ok if r.config.max_num_seqs == s) / max(1, len([r for r in ok if r.config.max_num_seqs == s])))
+        lines.append(f"5. **Optimal max_num_seqs**: `{best_seqs}`")
+
+    lines += [
+        "",
+        "6. **Always enable**: prefix caching, trust-remote-code",
+        "7. **Chunked prefill**: helps throughput under concurrent load",
+        "8. **Swap space**: 4GB CPU swap acts as a safety net against OOM",
+        "",
+        "---",
+        f"*Report generated on {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}*",
+        "",
+    ]
+
+    SUMMARY_MD.parent.mkdir(parents=True, exist_ok=True)
+    with open(SUMMARY_MD, "w") as f:
+        f.write("\n".join(lines))
+    print(f"\n📄 Report: {SUMMARY_MD}")
+
+
+# ============================================================
+# CLI
+# ============================================================
+
+def main():
+    parser = argparse.ArgumentParser(description="Advanced vLLM optimization sweep for RTX 3090")
+    parser.add_argument("--mode", default="baseline",
+                        choices=["baseline", "shootout", "grid", "single", "list"],
+                        help="baseline=A/B each opt, shootout=AWQ vs GPTQ vs FP8, grid=full combos")
+    parser.add_argument("--config", type=str, help="Single config name (for --mode single)")
+    parser.add_argument("--port", type=int, default=8000)
+    parser.add_argument("--timeout", type=int, default=600)
+    parser.add_argument("--dry-run", action="store_true", help="List configs, don't run")
+
+    args = parser.parse_args()
+
+    # Config selection
+    if args.mode == "baseline":
+        configs = baseline_configs()
+    elif args.mode == "shootout":
+        configs = model_shootout_configs()
+    elif args.mode == "grid":
+        configs = grid_configs()
+    elif args.mode == "single":
+        all_configs = baseline_configs() + model_shootout_configs()
+        configs = [c for c in all_configs if c.name == args.config]
+        if not configs:
+            print(f"Unknown config: {args.config}")
+            print(f"Available: {sorted(set(c.name for c in all_configs))}")
+            sys.exit(1)
+    else:  # list
+        for c in baseline_configs() + model_shootout_configs():
+            print(f"  {c.name:45s}  {c.label}")
+        return
+
+    print(f"\n{'='*70}")
+    print(f" ADVANCED SWEEP — {args.mode} mode — {len(configs)} config(s)")
+    print(f"{'='*70}")
+
+    if args.dry_run:
+        for c in configs:
+            print(f"  {c.name}")
+            print(f"    model={c.model_path}")
+            print(f"    attn={c.attention_backend} kv={c.kv_cache_dtype} chunk={c.enable_chunked_prefill}")
+            print(f"    bs={c.block_size} seqs={c.max_num_seqs} batched={c.max_num_batched_tokens}")
+        return
+
+    # Preflight
+    print("🔍 Preflight checks...")
+    # GPU
+    try:
+        result = subprocess.run(
+            ["nvidia-smi", "--query-gpu=memory.free", "--format=csv,noheader"],
+            capture_output=True, text=True, timeout=10,
+        )
+        free_mb = int(result.stdout.strip())
+        print(f"  GPU free: {free_mb} MiB")
+        if free_mb < 20000:
+            print("  ⚠️  Less than 20GB free — old processes?")
+    except Exception as e:
+        print(f"  ⚠️  GPU check failed: {e}")
+
+    # Port
+    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+    if sock.connect_ex(('localhost', args.port)) == 0:
+        print(f"  ⚠️  Port {args.port} already in use!")
+    sock.close()
+
+    # HF_TOKEN
+    if not os.environ.get("HF_TOKEN"):
+        print("  ⚠️  HF_TOKEN not set")
+    print("  ✅ Ready\n")
+
+    # Run
+    results = sweep(configs, port=args.port)
+
+    # Report
+    generate_report(results, args.mode)
+
+    # Final summary
+    ok = [r for r in results if r.success]
+    print(f"\n{'='*70}")
+    print(f" SWEEP COMPLETE")
+    print(f"  Configs: {len(results)}  |  Passed: {len(ok)}  |  Failed: {len(results) - len(ok)}")
+    if ok:
+        best = max(ok, key=lambda r: r.avg_output_tps)
+        print(f"  🏆 Best: {best.config.name} — {best.avg_output_tps:.1f} tok/s")
+        print(f"     attn={best.config.attention_backend}  kv={best.config.kv_cache_dtype}  "
+              f"bs={best.config.block_size}  seqs={best.config.max_num_seqs}")
+    print(f"{'='*70}")
+
+
+if __name__ == "__main__":
+    main()

scripts/sweep_vllm_configs.py

@@ -0,0 +1,814 @@
+#!/usr/bin/env python3
+"""
+sweep_vllm_configs.py — Automatic vLLM configuration sweep for RTX 3090.
+
+Workflow:
+    1. Define a grid of configs (model, max_model_len, gpu_memory_utilization,
+       dtype, max_num_seqs, etc.)
+    2. For each config:
+       a. Start vLLM server as a subprocess
+       b. Poll /v1/models until ready (with timeout)
+       c. Run benchmark via bench_openai_api.py (call it as a function)
+       d. If successful, record results
+       e. If OOM or crash, mark as failed and continue
+       f. Kill server cleanly, wait for GPU memory release
+    3. Generate reports/summary.md with findings
+    4. Print the best config
+
+Usage:
+    # Full sweep (all configs)
+    python scripts/sweep_vllm_configs.py
+
+    # Quick sweep (only short prompts, fewer configs)
+    python scripts/sweep_vllm_configs.py --quick
+
+    # Single config test
+    python scripts/sweep_vllm_configs.py --single fp8-8k
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import signal
+import subprocess
+import sys
+import time
+from dataclasses import dataclass, field
+from datetime import datetime
+from pathlib import Path
+from typing import Optional
+
+import httpx
+
+# Project paths
+PROJECT_DIR = Path(__file__).resolve().parent.parent
+REPORTS_DIR = PROJECT_DIR / "reports"
+PROMPTS_FILE = PROJECT_DIR / "prompts" / "bench_prompts.jsonl"
+RESULTS_CSV = REPORTS_DIR / "results.csv"
+SUMMARY_MD = REPORTS_DIR / "summary.md"
+
+# Available models (ordered by preference for RTX 3090 24GB)
+MODELS = {
+    "fp8": "Qwen/Qwen3.6-27B-FP8",
+    "awq": "QuantTrio/Qwen3.6-27B-AWQ",
+    "gptq": "groxaxo/Qwen3.6-27B-GPTQ-Pro-4bit",
+    "base": "Qwen/Qwen3.6-27B",
+}
+
+
+# ============================================================
+# Config definitions
+# ============================================================
+
+@dataclass
+class VllmConfig:
+    """A single vLLM server configuration to test."""
+    name: str
+    model_key: str            # key into MODELS dict
+    max_model_len: int
+    gpu_memory_utilization: float
+    dtype: str = "auto"
+    max_num_seqs: int = 4
+    max_num_batched_tokens: int = 16384
+    enable_prefix_caching: bool = True
+    quantization: str = ""    # inferred from model_key if empty
+
+    @property
+    def model_path(self) -> str:
+        return MODELS.get(self.model_key, self.model_key)
+
+    @property
+    def quant_method(self) -> str:
+        if self.quantization:
+            return self.quantization
+        # Infer from model_key
+        qmap = {"fp8": "fp8", "awq": "awq", "gptq": "gptq", "base": "none"}
+        return qmap.get(self.model_key, "none")
+
+    @property
+    def label(self) -> str:
+        return (
+            f"{self.model_key}-len{self.max_model_len}-mem{self.gpu_memory_utilization:.0%}"
+            f"-{self.dtype}-seqs{self.max_num_seqs}"
+        )
+
+
+def get_full_sweep_configs() -> list[VllmConfig]:
+    """All configs to test on RTX 3090."""
+    configs = []
+
+    # FP8 model — the main target
+    for max_len in [4096, 8192, 16384]:
+        for gpu_mem in [0.85, 0.90, 0.95]:
+            for num_seqs in [2, 4, 8]:
+                # Skip unrealistic combos
+                if max_len == 16384 and num_seqs > 4:
+                    continue
+                if max_len == 4096 and num_seqs < 4:
+                    continue
+                # FP8 16k + 0.95 mem + 4 seqs = too much
+                if max_len == 16384 and gpu_mem > 0.90 and num_seqs >= 4:
+                    continue
+
+                configs.append(VllmConfig(
+                    name=f"fp8-{max_len}-{int(gpu_mem*100)}-s{num_seqs}",
+                    model_key="fp8",
+                    max_model_len=max_len,
+                    gpu_memory_utilization=gpu_mem,
+                    max_num_seqs=num_seqs,
+                    max_num_batched_tokens=min(max_len * num_seqs, 32768),
+                ))
+
+    # AWQ model — more headroom
+    for max_len in [8192, 16384]:
+        for num_seqs in [4, 8]:
+            configs.append(VllmConfig(
+                name=f"awq-{max_len}-s{num_seqs}",
+                model_key="awq",
+                max_model_len=max_len,
+                gpu_memory_utilization=0.90,
+                max_num_seqs=num_seqs,
+                max_num_batched_tokens=min(max_len * num_seqs, 65536),
+            ))
+
+    # GPTQ model
+    for max_len in [8192, 16384]:
+        configs.append(VllmConfig(
+            name=f"gptq-{max_len}-s4",
+            model_key="gptq",
+            max_model_len=max_len,
+            gpu_memory_utilization=0.90,
+            max_num_seqs=4,
+            max_num_batched_tokens=min(max_len * 4, 65536),
+        ))
+
+    return configs
+
+
+def get_quick_sweep_configs() -> list[VllmConfig]:
+    """Key configs for a quick (~30 min) sweep."""
+    return [
+        VllmConfig(name="fp8-8k-s4", model_key="fp8", max_model_len=8192, gpu_memory_utilization=0.90, max_num_seqs=4),
+        VllmConfig(name="fp8-8k-s2", model_key="fp8", max_model_len=8192, gpu_memory_utilization=0.90, max_num_seqs=2),
+        VllmConfig(name="fp8-4k-s8", model_key="fp8", max_model_len=4096, gpu_memory_utilization=0.90, max_num_seqs=8),
+        VllmConfig(name="fp8-16k-s2", model_key="fp8", max_model_len=16384, gpu_memory_utilization=0.90, max_num_seqs=2),
+        VllmConfig(name="awq-8k-s8", model_key="awq", max_model_len=8192, gpu_memory_utilization=0.90, max_num_seqs=8),
+        VllmConfig(name="awq-16k-s4", model_key="awq", max_model_len=16384, gpu_memory_utilization=0.90, max_num_seqs=4),
+        VllmConfig(name="gptq-8k-s4", model_key="gptq", max_model_len=8192, gpu_memory_utilization=0.90, max_num_seqs=4),
+    ]
+
+
+# ============================================================
+# Server manager
+# ============================================================
+
+class VllmServer:
+    """Manage a vLLM server subprocess."""
+
+    def __init__(self, config: VllmConfig, port: int = 8000):
+        self.config = config
+        self.port = port
+        self.process: Optional[subprocess.Popen] = None
+        self.logfile = Path(f"/tmp/vllm_sweep_{config.name}.log")
+
+    def start(self) -> bool:
+        """Start vLLM server. Returns True if process started."""
+        cmd = [
+            sys.executable, "-m", "vllm.entrypoints.openai.api_server",
+            "--model", self.config.model_path,
+            "--host", "0.0.0.0",
+            "--port", str(self.port),
+            "--max-model-len", str(self.config.max_model_len),
+            "--gpu-memory-utilization", str(self.config.gpu_memory_utilization),
+            "--dtype", self.config.dtype,
+            "--max-num-seqs", str(self.config.max_num_seqs),
+            "--max-num-batched-tokens", str(self.config.max_num_batched_tokens),
+            "--tensor-parallel-size", "1",
+            "--trust-remote-code",
+        ]
+
+        if self.config.enable_prefix_caching:
+            cmd.append("--enable-prefix-caching")
+
+        # Quantization flag (only if not auto-handled by model config)
+        if self.config.quant_method not in ("none", ""):
+            # vLLM auto-detects quant from model config, but explicit is safer
+            cmd.extend(["--quantization", self.config.quant_method])
+
+        print(f"\n{'='*60}")
+        print(f" Starting: {self.config.label}")
+        print(f"  Model:     {self.config.model_path}")
+        print(f"  Port:      {self.port}")
+        print(f"  Log:       {self.logfile}")
+        print(f"  CMD:       {' '.join(cmd)}")
+        print(f"{'='*60}")
+
+        self.logfile.parent.mkdir(parents=True, exist_ok=True)
+        fout = open(str(self.logfile), "w")
+        self.process = subprocess.Popen(
+            cmd,
+            stdout=fout,
+            stderr=subprocess.STDOUT,
+            start_new_session=True,  # so we can kill the whole process group
+        )
+        return True
+
+    def wait_ready(self, timeout: int = 600) -> tuple[bool, str]:
+        """Poll /v1/models until server is ready. Returns (ready, message)."""
+        client = httpx.Client(base_url=f"http://localhost:{self.port}", timeout=httpx.Timeout(5.0))
+        deadline = time.time() + timeout
+        last_error = ""
+
+        while time.time() < deadline:
+            # Check if process is still alive
+            if self.process and self.process.poll() is not None:
+                # Process exited — check logs for OOM
+                tail = self._read_log_tail(50)
+                return False, f"Process exited (rc={self.process.returncode}). Log tail:\n{tail}"
+
+            try:
+                resp = client.get("/v1/models")
+                if resp.status_code == 200:
+                    data = resp.json()
+                    models = [m.get("id", "") for m in data.get("data", [])]
+                    client.close()
+                    return True, f"OK — models: {models}"
+                last_error = f"HTTP {resp.status_code}"
+            except Exception as e:
+                last_error = str(e)
+
+            time.sleep(5)
+
+        client.close()
+        # Timeout — check logs
+        tail = self._read_log_tail(30)
+        return False, f"Timeout after {timeout}s. Last error: {last_error}\nLog tail:\n{tail}"
+
+    def stop(self):
+        """Kill server and wait for GPU memory release."""
+        if self.process is None:
+            return
+
+        print(f"  ⏹️  Stopping server (PID {self.process.pid})...")
+        try:
+            # Kill the whole process group
+            os.killpg(os.getpgid(self.process.pid), signal.SIGTERM)
+            self.process.wait(timeout=30)
+        except subprocess.TimeoutExpired:
+            print("  ⚠️  SIGTERM timeout, sending SIGKILL...")
+            try:
+                os.killpg(os.getpgid(self.process.pid), signal.SIGKILL)
+                self.process.wait(timeout=10)
+            except Exception:
+                pass
+        except Exception as e:
+            print(f"  ⚠️  Error stopping server: {e}")
+
+        self.process = None
+        # Wait for GPU memory to free
+        print("  ⏳ Waiting for GPU memory release...")
+        time.sleep(5)
+
+        # Verify GPU is free
+        try:
+            result = subprocess.run(
+                ["nvidia-smi", "--query-gpu=memory.used", "--format=csv,noheader"],
+                capture_output=True, text=True, timeout=10
+            )
+            vram_used = int(result.stdout.strip().split()[0])  # MiB
+            print(f"  📊 VRAM used after stop: {vram_used} MiB")
+        except Exception:
+            pass
+
+    def _read_log_tail(self, lines: int = 30) -> str:
+        """Read last N lines of the log file."""
+        if not self.logfile.exists():
+            return "(no log file)"
+        try:
+            with open(self.logfile, "r") as f:
+                all_lines = f.readlines()
+                return "".join(all_lines[-lines:])
+        except Exception:
+            return "(error reading log)"
+
+
+# ============================================================
+# Main sweep
+# ============================================================
+
+@dataclass
+class SweepResult:
+    config: VllmConfig
+    success: bool
+    error: str = ""
+    avg_output_tps: float = 0.0
+    avg_ttft_ms: float = 0.0
+    peak_vram_mb: float = 0.0
+    avg_gpu_util: float = 0.0
+    num_prompts_tested: int = 0
+    num_success: int = 0
+    start_time: str = ""
+    end_time: str = ""
+    duration_s: float = 0.0
+
+
+def check_model_available(model_path: str) -> bool:
+    """Quick check if model can be downloaded/accessed."""
+    # Try huggingface_hub if available
+    try:
+        from huggingface_hub import model_info
+        model_info(model_path, timeout=10)
+        return True
+    except Exception:
+        pass
+
+    # Fallback: try via httpx to huggingface API
+    try:
+        resp = httpx.get(f"https://huggingface.co/api/models/{model_path}", timeout=10)
+        if resp.status_code == 200:
+            return True
+    except Exception:
+        pass
+
+    return False
+
+
+def run_sweep(
+    configs: list[VllmConfig],
+    port: int = 8000,
+    timeout: int = 600,
+    skip_model_check: bool = False,
+) -> list[SweepResult]:
+    """Run the full sweep."""
+    results: list[SweepResult] = []
+
+    # Ensure reports dir exists
+    REPORTS_DIR.mkdir(parents=True, exist_ok=True)
+
+    # Clean up old results if doing a full sweep
+    if len(configs) > 3:
+        if RESULTS_CSV.exists():
+            backup = RESULTS_CSV.with_suffix(".csv.bak")
+            RESULTS_CSV.rename(backup)
+            print(f"📁 Backed up old results to {backup}")
+
+    print(f"\n{'='*60}")
+    print(f" VLLM CONFIG SWEEP")
+    print(f"{'='*60}")
+    print(f"  Configs to test: {len(configs)}")
+    print(f"  Port:            {port}")
+    print(f"  Timeout:         {timeout}s per config")
+    print(f"  Output:          {RESULTS_CSV}")
+    print(f"{'='*60}\n")
+
+    for idx, config in enumerate(configs, 1):
+        result = SweepResult(
+            config=config,
+            success=False,
+            start_time=datetime.now().isoformat(),
+        )
+        t_start = time.time()
+
+        print(f"\n{'#'*60}")
+        print(f" [{idx}/{len(configs)}] Testing: {config.label}")
+        print(f"{'#'*60}")
+
+        # Check model availability (only on first config of each model)
+        if not skip_model_check and idx == 1:
+            print(f"  🔍 Checking model availability: {config.model_path}")
+            if not check_model_available(config.model_path):
+                print(f"  ⚠️  Model may not be accessible. Check HF_TOKEN or model name.")
+                print(f"  Continuing anyway (will fail with clear error if not found)...")
+
+        # Start server
+        server = VllmServer(config, port=port)
+        try:
+            server.start()
+        except Exception as e:
+            result.error = f"Failed to start server: {e}"
+            print(f"  ❌ {result.error}")
+            results.append(result)
+            continue
+
+        # Wait for ready
+        ready, msg = server.wait_ready(timeout=timeout)
+        if not ready:
+            result.error = msg
+            result.end_time = datetime.now().isoformat()
+            result.duration_s = time.time() - t_start
+            print(f"  ❌ Server failed to start: {msg[:300]}")
+            server.stop()
+            results.append(result)
+            continue
+
+        print(f"  ✅ Server ready: {msg}")
+
+        # Run benchmark (via function call to bench_openai_api)
+        print(f"\n  🚀 Running benchmark...")
+        bench_success = False
+        try:
+            bench_success = _run_benchmark_for_config(config, port=port)
+        except Exception as e:
+            result.error = f"Benchmark error: {e}"
+            print(f"  ❌ Benchmark crashed: {e}")
+
+        # Parse the latest results from CSV to get aggregate metrics
+        if bench_success:
+            metrics = _parse_last_benchmark_results(config)
+            if metrics:
+                result.success = True
+                result.avg_output_tps = metrics.get("avg_output_tps", 0)
+                result.avg_ttft_ms = metrics.get("avg_ttft_ms", 0)
+                result.peak_vram_mb = metrics.get("peak_vram_mb", 0)
+                result.avg_gpu_util = metrics.get("avg_gpu_util", 0)
+                result.num_prompts_tested = metrics.get("num_prompts", 0)
+                result.num_success = metrics.get("num_success", 0)
+                print(f"  📊 Avg output TPS: {result.avg_output_tps:.1f}, TTFT: {result.avg_ttft_ms:.0f}ms")
+            else:
+                result.success = True
+                result.error = "Benchmark ran but couldn't parse metrics"
+        else:
+            result.error = result.error or "Benchmark failed"
+
+        # Stop server
+        server.stop()
+
+        result.end_time = datetime.now().isoformat()
+        result.duration_s = time.time() - t_start
+
+        # Print progress
+        n_success = sum(1 for r in results if r.success)
+        print(f"\n  📈 Progress: {n_success}/{idx} configs successful")
+
+        results.append(result)
+
+    return results
+
+
+def _run_benchmark_for_config(config: VllmConfig, port: int = 8000) -> bool:
+    """Run the benchmark script against a running server. Returns True if benchmark completed."""
+    # We import and call the benchmark code directly
+    sys.path.insert(0, str(PROJECT_DIR / "scripts"))
+
+    try:
+        from bench_openai_api import (
+            OpenAIChatClient, GpuMonitor, load_prompts,
+            run_benchmark, save_results, BenchResult,
+        )
+    except ImportError as e:
+        print(f"    ⚠️  Cannot import bench_openai_api: {e}")
+        # Fallback: run as subprocess
+        return _run_benchmark_subprocess(config, port)
+
+    try:
+        client = OpenAIChatClient(base_url=f"http://localhost:{port}", timeout=300.0)
+
+        # Health check
+        ok, msg = client.check_health()
+        if not ok:
+            print(f"    ❌ Server not healthy: {msg}")
+            return False
+
+        # Load prompts
+        prompts_path = str(PROMPTS_FILE)
+        if not os.path.exists(prompts_path):
+            print(f"    ❌ Prompts file not found: {prompts_path}")
+            return False
+
+        prompts = load_prompts(prompts_path)
+
+        # GPU monitor
+        gpu_monitor = GpuMonitor(use_pynvml=True)
+        gpu_monitor.start()
+
+        config_info = {
+            "model": config.model_path,
+            "backend": "vllm",
+            "config_name": config.label,
+            "max_model_len": config.max_model_len,
+            "gpu_memory_util": config.gpu_memory_utilization,
+            "dtype": config.dtype,
+            "quantization": config.quant_method,
+        }
+
+        results = run_benchmark(
+            client=client,
+            prompts=prompts,
+            gpu_monitor=gpu_monitor,
+            config_info=config_info,
+            repeat=1,
+            warmup=True,
+        )
+
+        gpu_monitor.stop()
+
+        # Save to CSV
+        save_results(results, str(RESULTS_CSV))
+
+        client.client.close()
+        return True
+
+    except Exception as e:
+        print(f"    ❌ Benchmark error: {e}")
+        return False
+
+
+def _run_benchmark_subprocess(config: VllmConfig, port: int = 8000) -> bool:
+    """Fallback: run bench_openai_api.py as subprocess."""
+    bench_script = PROJECT_DIR / "scripts" / "bench_openai_api.py"
+    cmd = [
+        sys.executable, str(bench_script),
+        "--url", f"http://localhost:{port}",
+        "--prompts", str(PROMPTS_FILE),
+        "--output", str(RESULTS_CSV),
+        "--model", config.model_path,
+        "--backend", "vllm",
+        "--config", config.label,
+        "--max-model-len", str(config.max_model_len),
+        "--gpu-mem-util", str(config.gpu_memory_utilization),
+        "--dtype", config.dtype,
+        "--quantization", config.quant_method,
+    ]
+    print(f"    Running: {' '.join(cmd)}")
+    try:
+        result = subprocess.run(cmd, capture_output=True, text=True, timeout=600)
+        if result.returncode == 0:
+            print(result.stdout[-500:] if len(result.stdout) > 500 else result.stdout)
+            return True
+        else:
+            print(f"    ❌ Exit code {result.returncode}")
+            print(f"    stderr: {result.stderr[-500:]}")
+            return False
+    except subprocess.TimeoutExpired:
+        print("    ❌ Benchmark timed out")
+        return False
+    except Exception as e:
+        print(f"    ❌ Subprocess error: {e}")
+        return False
+
+
+def _parse_last_benchmark_results(config: VllmConfig) -> Optional[dict]:
+    """Parse the CSV to extract aggregate metrics for the last config run."""
+    if not RESULTS_CSV.exists():
+        return None
+
+    try:
+        import csv
+        with open(RESULTS_CSV, "r") as f:
+            reader = csv.DictReader(f)
+            rows = [r for r in reader if r.get("config_name") == config.label]
+
+        if not rows:
+            # If we can't match by config_name, try reading last N rows
+            with open(RESULTS_CSV, "r") as f:
+                reader = csv.DictReader(f)
+                all_rows = list(reader)
+                rows = all_rows[-30:]  # approximate
+
+        successful = [r for r in rows if r.get("success") == "True"]
+        if not successful:
+            return None
+
+        output_tps_vals = [float(r["output_tps"]) for r in successful if r.get("output_tps")]
+        ttft_vals = [float(r["ttft_ms"]) for r in successful if r.get("ttft_ms")]
+        vram_vals = [float(r["vram_peak_mb"]) for r in successful if r.get("vram_peak_mb") and float(r["vram_peak_mb"]) > 0]
+        gpu_util_vals = [float(r["gpu_util_pct"]) for r in successful if r.get("gpu_util_pct") and float(r["gpu_util_pct"]) > 0]
+
+        return {
+            "num_prompts": len(rows),
+            "num_success": len(successful),
+            "avg_output_tps": sum(output_tps_vals) / len(output_tps_vals) if output_tps_vals else 0,
+            "avg_ttft_ms": sum(ttft_vals) / len(ttft_vals) if ttft_vals else 0,
+            "peak_vram_mb": max(vram_vals) if vram_vals else 0,
+            "avg_gpu_util": sum(gpu_util_vals) / len(gpu_util_vals) if gpu_util_vals else 0,
+        }
+    except Exception as e:
+        print(f"    ⚠️  Error parsing CSV: {e}")
+        return None
+
+
+# ============================================================
+# Summary generation
+# ============================================================
+
+def generate_summary(sweep_results: list[SweepResult]):
+    """Generate reports/summary.md."""
+    successful = [r for r in sweep_results if r.success]
+    failed = [r for r in sweep_results if not r.success]
+
+    # Sort by output TPS descending
+    successful_sorted = sorted(successful, key=lambda r: r.avg_output_tps, reverse=True)
+
+    lines = []
+    lines.append("# Qwen SpeedLab — Benchmark Summary")
+    lines.append("")
+    lines.append(f"**Date:** {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}")
+    lines.append(f"**GPU:** NVIDIA GeForce RTX 3090 (24 Go)")
+    lines.append(f"**Configs tested:** {len(sweep_results)} total, {len(successful)} successful, {len(failed)} failed")
+    lines.append("")
+    lines.append("---")
+    lines.append("")
+
+    # Best config
+    if successful_sorted:
+        best = successful_sorted[0]
+        lines.append("## 🏆 Best Configuration")
+        lines.append("")
+        lines.append(f"| Parameter | Value |")
+        lines.append(f"|-----------|-------|")
+        lines.append(f"| Config name | `{best.config.label}` |")
+        lines.append(f"| Model | `{best.config.model_path}` |")
+        lines.append(f"| Max model len | {best.config.max_model_len} |")
+        lines.append(f"| GPU memory util | {best.config.gpu_memory_utilization:.0%} |")
+        lines.append(f"| Dtype | {best.config.dtype} |")
+        lines.append(f"| Max num seqs | {best.config.max_num_seqs} |")
+        lines.append(f"| Quantization | {best.config.quant_method} |")
+        lines.append(f"| **Avg output TPS** | **{best.avg_output_tps:.1f} tok/s** |")
+        lines.append(f"| Avg TTFT | {best.avg_ttft_ms:.0f} ms |")
+        lines.append(f"| Peak VRAM | {best.peak_vram_mb:.0f} MiB |")
+        lines.append(f"| Avg GPU util | {best.avg_gpu_util:.1f}% |")
+        lines.append("")
+
+    # All results table
+    lines.append("## 📊 All Configurations")
+    lines.append("")
+    lines.append("| # | Config | Model | Len | Mem% | Seqs | TPS out | TTFT ms | VRAM MiB | Status |")
+    lines.append("|---|--------|-------|-----|------|------|---------|---------|----------|--------|")
+    for i, r in enumerate(sweep_results, 1):
+        status = "✅" if r.success else "❌"
+        if r.success:
+            tps_str = f"{r.avg_output_tps:.1f}"
+            ttft_str = f"{r.avg_ttft_ms:.0f}"
+            vram_str = f"{r.peak_vram_mb:.0f}"
+        else:
+            tps_str = "-"
+            ttft_str = "-"
+            vram_str = "-"
+        lines.append(
+            f"| {i} | {r.config.label} | {r.config.model_key} | {r.config.max_model_len} "
+            f"| {r.config.gpu_memory_utilization:.0%} | {r.config.max_num_seqs} "
+            f"| {tps_str} | {ttft_str} | {vram_str} | {status} |"
+        )
+    lines.append("")
+
+    # Failed configs
+    if failed:
+        lines.append("## ❌ Failed Configurations")
+        lines.append("")
+        for r in failed:
+            lines.append(f"### `{r.config.label}`")
+            err_summary = r.error[:300].replace("\n", "\n> ")
+            lines.append(f"> {err_summary}")
+            lines.append(f"")
+            lines.append(f"- Duration: {r.duration_s:.0f}s")
+            lines.append("")
+
+    # Recommendations
+    lines.append("## 💡 Recommendations for RTX 3090")
+    lines.append("")
+    lines.append("Based on the sweep results:")
+
+    if successful_sorted:
+        lines.append("")
+        lines.append("1. **Model choice**: ")
+        best_model = successful_sorted[0].config.model_key
+        if best_model == "fp8":
+            lines.append("   FP8 is the best balance of quality and speed. It fits in 24GB with careful tuning.")
+        elif best_model == "awq":
+            lines.append("   AWQ (4-bit) provides the best throughput with comfortable VRAM headroom.")
+        elif best_model == "gptq":
+            lines.append("   GPTQ Int4 works well but AWQ is generally faster on Ampere GPUs.")
+        else:
+            lines.append("   Use a quantized variant — the base bfloat16 model (~54GB) won't fit on 24GB.")
+
+        lines.append("")
+        lines.append("2. **Context length**: ")
+        best_len = successful_sorted[0].config.max_model_len
+        lines.append(f"   {best_len} tokens is the recommended max_model_len. Going higher risks OOM.")
+
+        lines.append("")
+        lines.append("3. **GPU memory utilization**: ")
+        lines.append("   0.90 is the safe ceiling. 0.95 can work for short contexts but risks fragmentation.")
+
+        lines.append("")
+        lines.append("4. **Concurrency**: ")
+        lines.append("   Keep max_num_seqs ≤ 4. Beyond that, KV cache pressure causes OOM or severe slowdown.")
+
+        lines.append("")
+        lines.append("5. **Prefix caching**: Always enable — significant TTFT improvement on long prompts.")
+
+        lines.append("")
+        lines.append("6. **Dtype**: Use `auto` — vLLM correctly detects the model's native quantization.")
+
+    lines.append("")
+    lines.append("## 📁 Raw Data")
+    lines.append("")
+    lines.append(f"Full per-request results: [`reports/results.csv`](results.csv)")
+    lines.append("")
+
+    lines.append("---")
+    lines.append(f"*Report generated by sweep_vllm_configs.py on {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}*")
+    lines.append("")
+
+    # Write
+    SUMMARY_MD.parent.mkdir(parents=True, exist_ok=True)
+    with open(SUMMARY_MD, "w", encoding="utf-8") as f:
+        f.write("\n".join(lines))
+
+    print(f"\n📄 Summary written to {SUMMARY_MD}")
+
+
+# ============================================================
+# CLI
+# ============================================================
+
+def main():
+    parser = argparse.ArgumentParser(description="Sweep vLLM configurations for RTX 3090")
+    parser.add_argument("--quick", action="store_true", help="Quick sweep (~30 min, key configs only)")
+    parser.add_argument("--single", type=str, default=None, help="Test a single named config")
+    parser.add_argument("--port", type=int, default=8000, help="Port for vLLM server")
+    parser.add_argument("--timeout", type=int, default=600, help="Timeout per config (seconds)")
+    parser.add_argument("--skip-model-check", action="store_true", help="Skip HF model availability check")
+    parser.add_argument("--dry-run", action="store_true", help="List configs without running")
+    args = parser.parse_args()
+
+    # Select configs
+    if args.single:
+        # Find matching config
+        all_configs = get_full_sweep_configs()
+        configs = [c for c in all_configs if c.name == args.single or args.single in c.label]
+        if not configs:
+            print(f"❌ No config matching '{args.single}'")
+            print(f"   Available: {[c.name for c in all_configs]}")
+            sys.exit(1)
+    elif args.quick:
+        configs = get_quick_sweep_configs()
+    else:
+        configs = get_full_sweep_configs()
+
+    print(f"Selected {len(configs)} config(s) to test:")
+    for c in configs:
+        print(f"  - {c.label}  ({c.model_path}, len={c.max_model_len}, mem={c.gpu_memory_utilization:.0%}, seqs={c.max_num_seqs})")
+
+    if args.dry_run:
+        print("\n🔍 Dry run — exiting without testing.")
+        return
+
+    # Pre-flight checks
+    print("\n🔍 Pre-flight checks...")
+
+    # Check GPU
+    try:
+        result = subprocess.run(["nvidia-smi", "--query-gpu=memory.free", "--format=csv,noheader"],
+                                capture_output=True, text=True, timeout=10)
+        vram_free = int(result.stdout.strip().split()[0])
+        print(f"  GPU memory free: {vram_free} MiB")
+        if vram_free < 20000:
+            print(f"  ⚠️  Less than 20GB free — old processes may be running")
+    except Exception as e:
+        print(f"  ⚠️  Cannot check GPU: {e}")
+
+    # Check HF_TOKEN
+    if not os.environ.get("HF_TOKEN"):
+        print("  ⚠️  HF_TOKEN not set — gated models may fail to download")
+
+    # Check port free
+    import socket
+    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+    port_free = sock.connect_ex(('localhost', args.port)) != 0
+    sock.close()
+    if not port_free:
+        print(f"  ⚠️  Port {args.port} is in use — sweep may fail")
+
+    print("  ✅ Pre-flight checks done")
+
+    # Run sweep
+    results = run_sweep(
+        configs=configs,
+        port=args.port,
+        timeout=args.timeout,
+        skip_model_check=args.skip_model_check,
+    )
+
+    # Generate summary
+    generate_summary(results)
+
+    # Final print
+    successful = [r for r in results if r.success]
+    print(f"\n{'='*60}")
+    print(f" SWEEP COMPLETE")
+    print(f"{'='*60}")
+    print(f"  Total:     {len(results)} configs")
+    print(f"  Success:   {len(successful)}")
+    print(f"  Failed:    {len(results) - len(successful)}")
+    if successful:
+        best = max(successful, key=lambda r: r.avg_output_tps)
+        print(f"  Best TPS:  {best.avg_output_tps:.1f} tok/s ({best.config.label})")
+    print(f"  Summary:   {SUMMARY_MD}")
+    print(f"  Raw data:  {RESULTS_CSV}")
+    print(f"{'='*60}")
+
+
+if __name__ == "__main__":
+    main()