Spaces:
Paused
Paused
Commit History
-
Add enforce_eager=True to fix KV cache memory issue
torch.compile was consuming ~3-4GB overhead, causing negative KV cache memory (-1.13 GiB). enforce_eager skips compilation, reducing memory overhead and enabling successful model startup. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>1b7fbd7 -
Reduce vLLM memory for A100 24GB compatibility
- gpu_memory_utilization: 0.80 → 0.55 (leaves ~10GB for embedding+reranker) - max_model_len: 16384 → 8192 (reduces KV cache memory) Fixes OOM error when loading all 3 models on 22.3GB usable VRAM. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>b85b1e0 -
Switch to Qwen3-VL-4B-Thinking for single-GPU simplicity
Architecture change: - Vision: 30B MoE (4 GPU TP) → 4B Dense (single GPU) - tensor_parallel_size: 4 → 1 - gpu_memory_utilization: 0.50 → 0.80 - Removed NCCL workarounds (not needed for single GPU) Why: 30B MoE model failed on 4xL4 due to vLLM V1 + NCCL issues (L4s lack NVLINK). 4B dense model fits on single L4 (22GB), eliminates all multi-GPU coordination problems. Memory: ~18GB total (10GB vision + 4GB embed + 4GB rerank) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>14c59e5 -
Reduce context/memory to minimize NCCL overhead on L4s
Changes: - max_model_len: 16384 → 8192 (half context length) - gpu_memory_utilization: 0.70 → 0.50 (less memory pressure) L4 GPUs lack NVLINK, making tensor parallelism communication slow via PCIe. Reducing these values minimizes NCCL overhead. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>7d5c713 -
Align vLLM config with official Qwen3-VL model card
Changes: - Remove VLLM_USE_V1=0 (V0 is removed in vLLM 0.11+, must use V1) - Remove dtype="float16" (FP8 model should auto-detect) - Change gpu_memory_utilization 0.90→0.70 (official recommendation) - Remove enforce_eager=True (let vLLM default per official) Root cause: V0 engine doesn't exist in vLLM >=0.11.0. Qwen3-VL requires vLLM >=0.11.0. Must make V1 work with official config. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>b2fe3f4 -
Fix vLLM multi-GPU init: explicit dtype + higher mem util + eager mode
Based on successful RTX 4090D config from HF discussions: - Force env vars with os.environ[] (not setdefault) - Add dtype="float16" explicitly - Increase gpu_memory_utilization to 0.90 - Add enforce_eager=True for multi-GPU stability Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>71a896b -
Force vLLM V0 engine + reduce max_model_len for stability
- VLLM_USE_V1=0: Force stable V0 engine instead of V1 - Reduce max_model_len from 32768 to 16384 for memory safety - Keep NCCL and spawn settings for multi-GPU reliability Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>3c9a722 -
Fix vLLM Engine core initialization failed on multi-GPU
Add environment variables before vLLM import to fix tensor parallelism issues: - VLLM_WORKER_MULTIPROC_METHOD=spawn (fixes process spawn issues) - NCCL_P2P_DISABLE=1, NCCL_IB_DISABLE=1 (multi-GPU reliability) References: - https://github.com/vllm-project/vllm/issues/17618 - https://github.com/vllm-project/vllm/issues/19002 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>ed575b1 -
Replace dual 8B with single 30B-A3B FP8 vision model
Simplify pipeline architecture: - Vision: Qwen3-VL-30B-A3B-Thinking-FP8 (~30-35GB) replaces dual 8B - Embedding: 2B model (2048-dim) replaces 8B - Reranker: 2B model replaces 8B - Total VRAM: ~38-43GB (was ~68GB), 45GB+ headroom on 4xL4 Key changes: - vLLM with FP8 quantization (built-in, no autoawq needed) - Proper Qwen3-VL chat template formatting via processor - Removed dual-model Thinking→Instruct pipeline - Single model handles analysis + structured JSON output Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>706520f -
Replace 30B MoE with dual 8B models (Thinking + Instruct)
Architecture change: - Vision: Qwen3-VL-30B-A3B-Instruct → dual Qwen3-VL-8B-Thinking + 8B-Instruct - Two-stage pipeline: Thinking (deep analysis) → Instruct (JSON formatting) - VRAM: 90GB → 68GB (~22GB savings, 20GB headroom on 4xL4) Key changes: - models/real.py: New DualVisionModel with token-based </think> parsing - config/settings.py: Dual model paths (vision_model_thinking, vision_model_instruct) - config/inference.py: ThinkingInferenceConfig (temp=0.6, max_tokens=32768) - Removed all lazy loading code (load_vision/unload_vision/load_rag) - All 4 models now load simultaneously at startup Per Qwen3-VL GitHub recommended hyperparameters for thinking models. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>333c083
-
Implement lazy model loading to prevent CUDA OOM on 4xL4 GPUs
Problem: All 3 models (~92GB) loaded at startup exceeded 88GB VRAM. Solution: Sequential loading - vision model during Stage 2, RAG models during Stage 3+. Vision is unloaded before RAG loads. Peak: ~60GB. Changes: - models/real.py: Add load_vision(), unload_vision(), load_rag() with proper hook removal per HuggingFace accelerate docs - models/loader.py: Real models now use lazy loading (no load_all) - pipeline/main.py: Load/unload at appropriate pipeline stages - rag/vectorstore.py: Use SharedEmbeddingFunction (no duplicate load) - rag/retriever.py: Use SharedReranker (no duplicate load) - models/mock.py: Add is_vision_loaded(), is_rag_loaded() for API parity Memory profile: - Phase A (Vision): 30B model ~60GB - Transition: Unload + gc + empty_cache - Phase B (RAG): 8B + 8B ~32GB - Peak never exceeds 60GB (fits in 88GB) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>5f0db1e -
Add property accessors to RealModelStack for interface parity
RealModelStack stored models in self.models dict but lacked property accessors that pipeline code expected. MockModelStack exposes models as direct attributes (self.vision, etc.) but RealModelStack did not. This fix adds @property accessors for vision, embedding, and reranker that wrap the raw models in their respective wrapper classes, making RealModelStack API-compatible with MockModelStack. Fixes: 'RealModelStack' object has no attribute 'vision' error Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>c190082 -
Fix multi-GPU compatibility issues (6 locations)
1. Meta tensor error in reranker (BLOCKING): - Use .clone() to force weight materialization before copying - Create Linear layer with correct device/dtype from lm_head 2. Remove all .to(device) calls that break distributed models: - scripts/qwen3_vl/qwen3_vl_embedding.py:388 - scripts/qwen3_vl/qwen3_vl_reranker.py:109, 374 - models/real.py:259 - rag/vectorstore.py:130 - rag/retriever.py:149 With device_map="auto", transformers handles device routing internally. Manual .to(device) calls cause device mismatches on multi-GPU setups. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>d1901ae -
Fix embedding/reranker loading with official Qwen3-VL classes
Root cause: AutoModel.from_pretrained() loads base transformer instead of specialized embedding/reranking variants. Changes: - Vendor official scripts from QwenLM/Qwen3-VL-Embedding repo - Replace AutoModel with Qwen3VLEmbedder for embedding model - Replace AutoModel with Qwen3VLReranker for reranker model - Update embed()/rerank() methods to use official process() API The official loaders handle: - Proper last-token pooling and L2 normalization (embedding) - Yes/no binary scoring from LM head weights (reranker) This eliminates the fallback L2 norm heuristic scoring that was producing "less accurate" results. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>455c786 -
Fix critical model implementations and add sample scenarios
Model Fixes (verified against official Qwen3-VL-Embedding repo): - Embedding: Use last-token pooling instead of mean pooling - Embedding: Fix dimension from 768/384 to 4096 - Reranker: Use yes/no LM head weights + sigmoid instead of CLS norm - Apply fixes in both models/real.py and rag/vectorstore.py Additional Changes: - Add sample scenarios with real fire damage images - Add E2E tests with Playwright - Fix Gradio 6.x tab navigation tests - Add logging configuration - Improve UI form handling All 179 unit tests pass. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>f3ebc82 -
Initial commit: FDAM AI Pipeline v4.0.1
- Gradio-based fire damage assessment application - Qwen3-VL vision model integration (mock + real) - RAG-based knowledge retrieval with ChromaDB - FDAM-compliant calculations (ACH, sample density) - PDF generation with WeasyPrint - Session persistence via localStorage - 151 passing tests Ready for HuggingFace Spaces deployment on 4xL4 (96GB VRAM) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>88bdcff