Spaces:
Runtime error
Runtime error
Commit History
-
feat: improve ISR mission planner and assessor prompts
Zhen Ye +1: Claude Opus 4.6 (1M context)- Add 15 BAML PlanMission tests with strict assertions: max query count caps and negative assertions to prevent detector overload with unrelated COCO classes - Tighten PlanMission prompt to output only mission-targeted object classes (1-4 items) instead of padding with context classes (3-8) - Replace noisy DetectionInfo fields (bbox_width_px, bbox_height_px, area_ratio) with signal-rich fields (confidence, center_x, center_y) - Lean AssessDetections prompt: remove hardcoded examples, add metadata reading guide for confidence/position/speed interpretation Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>0ebc456
-
refactor: migrate to uv, prompt-tune BAML mission planner and assessor
Zhen Ye +1: Claude Opus 4.6 (1M context)- Replace requirements.txt with pyproject.toml + uv.lock for faster, deterministic installs - Update Dockerfile to use uv instead of pip - Prompt-tune PlanMission to output broad detector categories only (no mission-specific labels) - Prompt-tune AssessDetections as the precise filter using visual context - Harden encode_frame for dtype/channel edge cases - Add negative assertions to BAML tests enforcing separation of concerns Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>18a11bc
-
feat: add detailed BAML logging for mission planner and assessment verdicts
Zhen Ye +1: Claude Opus 4.6 (1M context)Logs mission plan reasoning, assessment inputs (detections sent to GPT), and per-track verdict details (relevant, satisfies, reason, features). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>27d5248
-
feat: integrate BAML for mission planning and detection assessment
Add PlanMission function that converts natural language mission objectives into concrete detector queries and a refined mission statement. Rewrite ISRAssessor to use BAML's type-safe AssessDetections function instead of raw OpenAI API calls with manual JSON parsing. Includes two test cases for the BAML playground. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>240e068
-
refactor: remove depth estimation pipeline entirely
Depth estimation (DepthAnything V2) was disabled by default, provided only relative visualization values (hardcoded focal_length=1.0), and was never used for actual distance reasoning (GPT handles that). Removed: - models/depth_estimators/ directory (4 files) - inspection/depth.py (caching, inference, colorization) - tests/test_inspection_depth.py - All depth params from /detect and /detect/async endpoints - Depth model loading and per-frame inference in inference.py - IncrementalDepthStats, colorize_depth_map, _attach_depth_metrics - Depth toggle, fetchDepth, renderDepthHeatmap, renderRealDepth from frontend - Depth quadrant from inspect panel (now 3 quadrants: seg, edge, 3D) - Depth fields from JobInfo dataclass - Depth-based 3D fallback in pointcloud endpoint (Tripo3D only now) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>1fad3ed -
refactor(pipeline): decouple vision stage from AI postprocessing
Eliminate shared mutable state between detection/tracking (stage 1) and GPT assessment (stage 2) so they can be developed independently. - Replace _merge_verdicts() with _store_verdicts() that writes to a separate verdict store instead of mutating detection dicts in-place - Expand _mission_verdicts to hold full verdict dicts with cached bool projection for hot-path rendering - Add set_verdicts_batch() for single-lock-acquisition bulk writes - Add GET /detect/verdicts/{job_id} endpoint for frontend verdict access - Add fetchVerdicts() in api.js to populate assessment cache from new endpoint Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>ecfb6e4 -
refactor: clean up backend utilities and background job handler
- Simplify background.py exception handling (merge two except blocks) - Remove unused imports and dead code in models.py, isr/utils.py - Use get_pil_rgb() in frame_store adapter instead of manual BGR conversion - Remove redundant dd benchmark variables in hardware_info.py - Remove unused decode timing vars in profiler.py Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>c1e17cf -
refactor(detectors): extract shared YOLO logic into YoloBaseDetector base class
Eliminates ~250 lines of duplicated predict/predict_batch/tiled inference code between Yolo11Detector and YoloV8VisDroneDetector. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>082b546 -
refactor: anonymize LLM labels, merge fragmented tracks, and tune defaults
Replace GPT-4o/Claude/Gemini branding with generic Analyst 1/2/3 labels in both frontend and backend. Add track merging to deduplicate fragmented IDs for the same physical object. Filter non-mission-relevant objects from timeline events. Bump frame store budget to 24 GiB, increase default segmentation step to 12, auto-detect API base URL, and fix video playback reliability (blob MIME type, canplay wait, timeout increase). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>fec2ee8
-
fix: simplify round 2 — deduplicate update_masks, reduce VRAM, clear cache
Zhen Ye +1: Claude Opus 4.6 (1M context)- update_masks: now a thin wrapper over update_masks_with_remapping (was a full copy-paste of the same logic) - _mask_nms: use .half() instead of .float() to halve GPU memory spike (~40MB vs ~80MB for 10 masks at 1080p) - Export clearExplainCache() from explain.js, call it in resetToReady() so stale explain data doesn't persist across jobs Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>75516e9 -
fix: simplify review — fix mask NMS bug, remove dead code, hoist imports
Zhen Ye +1: Claude Opus 4.6 (1M context)- _mask_nms: fix greedy suppression bug (j<=i compared raw indices, not sort positions — masks escaped suppression). Use order[pos+1:] - _mask_nms: batch GPU computation (single matmul + single CPU sync instead of O(n^2) CUDA syncs per mask pair) - _apply_nms: hoist torch import (already module-level) and batched_nms import to module scope - real-backend.js: remove dead isNotRelevant (always false after mission_relevant filter) - explain.js: clear _explainCache on job change to prevent stale data across different videos Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>d6c4e26 -
fix: add mask-level NMS in GSAM2/YSAM2 to deduplicate overlapping masks
Zhen Ye +1: Claude Opus 4.6 (1M context)Within a single keyframe, YOLO can detect the same object with slightly different bounding boxes (e.g., cab vs full truck body) that survive box-level NMS but produce overlapping SAM2 masks. These all get unique IDs and render as stacked labels. Added _mask_nms() to MaskDictionary.add_new_frame_annotation(): - Computes pairwise mask IoU for same-label detections - Suppresses smaller masks when IoU > 0.5 with a larger one - Runs before masks enter the SAM2 video predictor pipeline Fixes duplicate "truck truck truck" labels on single objects. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>8d09cca
-
fix: graph layout spacing + validator API fixes
Zhen Ye +1: Claude Opus 4.6 (1M context)- Use d3.tree().nodeSize() for guaranteed 100px spacing between leaves - Center tree with xOffset calculation, horizontal scroll for narrow panels - Fix Gemini SDK: use google-genai client API (types.Part.from_bytes) - Add fuzzy key matching in merge (case-insensitive + feature-name fallback) - Add response logging for Claude and Gemini validators - Update requirements: google-genai replaces google-generativeai Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>583a618 -
feat: add ISRExplainer multi-LLM orchestration
Zhen Ye +1: Claude Opus 4.6 (1M context)Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>880e261 -
refactor: extract shared ISR utilities from assessor
Zhen Ye +1: Claude Opus 4.6 (1M context)Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>6c0d2dd
-
feat: mission-relevance coloring — red for relevant objects, gray for non-relevant
Zhen Ye +1: Claude Opus 4.6 (1M context)Backend-rendered video now uses GPT assessment verdicts to color bboxes (detection mode) and masks (segmentation mode) red for mission-relevant objects and gray for non-relevant. Unassessed tracks keep default colors. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>61b921a
-
feat(isr): create async assessor loop with verdict merging
Zhen Ye +1: Claude Opus 4.6Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>4c10904 -
feat(isr): create ISRAssessor with GPT-4o-mini vision integration
Zhen Ye +1: Claude Opus 4.6Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>2327048 -
fix: weight download race condition + rewrite CLAUDE.md
Zhen Ye +1: Claude Opus 4.6Add ensure_weights() classmethod and prefetch_weights() to download model weights once before parallel multi-GPU init, fixing FileNotFoundError when 4 GPUs race to download visDrone.pt simultaneously. Rewrite CLAUDE.md to reflect current architecture: async job pipeline, multi-GPU inference, GSAM2 segmentation, frontend SPA modules. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>f06dd83
-
feat: replace drone_yolo with YOLOv8-VisDrone detector
Zhen Ye +1: Claude Opus 4.6Remove broken drone_yolo detector (rujutashashikanjoshi repo not found) and replace with Mahadih534/YoloV8-VisDrone. Use hf_hub_download for reliable weight fetching. Allow drone_detection mode to use user-selected detector instead of hardcoding. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>dda4696
-
feat: add YOLOv8-VisDrone detector for aerial object detection
Zhen Ye +1: Claude Opus 4.6Adds Mahadih534/YoloV8-VisDrone from HuggingFace Hub as a new drone detection model option with tiling and batch inference support. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>c50a734
-
Eliminate redundant JPEG frame loading via shared frame store
Zhen Yef013501
-
feat(ttfs): instrument segmentation time-to-first-stream
Zhen Yecffad78 -
fix: download yolo11m.pt via huggingface_hub to writable /tmp path
Zhen Ye +1: Claude Opus 4.6ultralytics YOLO() tries to download to CWD which is read-only in the container. Use hf_hub_download to fetch weights to /tmp/yolo_weights then load from that path. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>f2ddd54 -
fix: use YOLO("yolo11m.pt") with writable config dir
Zhen Ye +1: Claude Opus 4.6hf:// prefix not supported in installed ultralytics version. Use standard YOLO("yolo11m.pt") which auto-downloads from CDN. Set YOLO_CONFIG_DIR=/tmp/ultralytics so weights download to a writable path in read-only containers. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>119e09f -
refactor: rename hf_yolov8 → yolo11 across codebase
Zhen Ye +1: Claude Opus 4.6Rename file yolov8.py → yolov11.py, class HuggingFaceYoloV8Detector → Yolo11Detector, and registry key "hf_yolov8" → "yolo11" in all 11 files. Loads YOLO11m COCO-pretrained via hf://Ultralytics/YOLO11. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>02d40a4 -
feat: replace custom YOLOv8m_defence with YOLO11m COCO-pretrained
Zhen Ye +1: Claude Opus 4.6The custom defence weights (spencercdz/YOLOv8m_defence) hallucinated on standard traffic scenes—detecting road markings as cars while missing actual vehicles. Switch to ultralytics YOLO11m with COCO pretrained weights for reliable general-purpose detection. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>d6a6a01 -
feat(segmentation): add YSAM2 variants and mission-aware detector mapping
Zhen Ye7d48e41 -
chore: remove dead code and unused frontend modules
Zhen Ye5a72443 -
refactor(gsam2): make SAM2 detector-agnostic
Zhen Yed23c4ea
-
fix: SAM2 _prep_prompts() signature
Zhen Ye339d4ac -
perf: defer GPU->CPU mask transfer to render thread via LazyFrameObjects
Zhen Ye +1: Claude Opus 4.6Replace bulk segment_output.to_object_dicts() CPU transfer with lazy materialization — only the last frame per segment is materialized for IoU tracking continuity, while render frames use LazyFrameObjects that defer GPU->CPU copy until the render thread needs them. Reduces CUDA syncs from 100+ per segment to ~3. Also removes deployment_logs.txt. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>97d32b8 -
fix: correct benchmark metrics — remove double-counting and add missing timers
Zhen Ye +1: Claude Opus 4.6- Fix sam_video_total_ms double-counting in single-GPU process_video() path (propagate_segment already accumulates, outer timer was adding it twice) - Fix id_reconciliation_ms in multi-GPU path to measure actual IoU work instead of wall-clock queue waits - Add model_load_ms and init_state_ms metrics for both single/multi-GPU paths Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>3e8ca01
-
perf: pipeline GSAM2 tracking + rendering with startup buffer
Zhen Ye +1: Claude Opus 4.6Pipeline tracking and rendering so segments stream as they're tracked instead of waiting for all tracking to complete before rendering begins. - Add on_segment callback to process_video() for incremental feeding - Hoist render/writer infrastructure above tracking branch - Single-GPU: callback-based incremental feeding via on_segment - Multi-GPU: streaming reconciliation with segment_buffer pattern - Writer startup buffer (60 frames) before streaming begins - 3x frame duplication for 18 FPS stream (6 FPS processing throughput) - Safety threshold: pause streaming when buffer < 20 frames - try/finally sentinel safety for render worker shutdown - Increase render_out queue from 64 to 128 for buffer headroom Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>108b51b -
feat: add num_maskmem parameter to /benchmark endpoint
Zhen Ye +1: Claude Opus 4.6Thread num_maskmem override through the full pipeline so benchmark runs can test reduced memory bank sizes (e.g. 3 instead of default 7). Patches the SAM2 video predictor's maskmem_tpos_enc at runtime. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>a47c9fa -
revert: remove torch.compile — runtime failures on HF Space
Zhen Ye +1: Claude Opus 4.6Reverts all torch.compile changes (b17bd6d..10eb3c6). The HF Space container lacks Triton/inductor support, causing 500 errors. The GPU-resident tensor pipeline from 5aec47c is retained. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>81548c3 -
fix: drop fullgraph=True from torch.compile for broader compatibility
Zhen Ye +1: Claude Opus 4.6fullgraph=True requires no graph breaks, which fails without SAM2VideoPredictorVOS's .clone() patches. Removing it allows torch.compile to handle graph breaks gracefully via partial compilation. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>40d31e0 -
fix: add runtime fallback for torch.compile inductor/triton failures
Zhen Ye +1: Claude Opus 4.6torch.compile wrapping succeeds immediately but actual compilation happens lazily on first forward pass. Now catches runtime errors on the first propagate_in_video yield, reverts all compiled forwards to eager originals, and retries the propagation transparently. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>534727c -
fix: manual torch.compile with graceful fallback for SAM2 components
Zhen Ye +1: Claude Opus 4.6Replace vos_optimized=True (caused 500 error) with manual per-component torch.compile matching Facebook's official recipe. Compiles image_encoder, memory_encoder, memory_attention (dynamic=True), sam_prompt_encoder, and sam_mask_decoder with max-autotune. Falls back to eager mode silently if PyTorch version or CUDA compiler is unavailable. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>43ebce8 -
perf: enable torch.compile for SAM2 via vos_optimized flag
Zhen Ye +1: Claude Opus 4.6Uses Facebook's official SAM2VideoPredictorVOS which compiles all five model components (image_encoder, memory_encoder, memory_attention, sam_prompt_encoder, sam_mask_decoder) with torch.compile max-autotune. First inference has ~30s warmup cost; subsequent frames benefit from fused kernels and reduced memory round-trips. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>703e1a0 -
perf: GPU-resident tensor pipeline for SAM2 video propagation
Zhen Ye +1: Claude Opus 4.6Eliminate all CUDA synchronization from propagate_segment() by keeping masks, bboxes, and validity flags on GPU in pre-allocated buffers. CPU materialization is deferred to a single bulk transfer via SegmentOutput.to_object_dicts() at the consumer. - Add _bbox_gpu() for zero-sync bounding box computation on GPU - Add SegmentOutput dataclass for GPU-resident segment results - Rewrite propagate_segment() with inline bbox + pre-allocated tensors - Refactor process_video() to reuse propagate_segment() - Update Phase 4 reconciliation: 3 CUDA syncs per segment vs 100+ Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>abf2d0f
-
perf: eliminate CUDA sync points in SAM2 video propagation hot-path
Zhen Ye +1: Claude Opus 4.6Replace per-object-per-frame torch.nonzero + .item() calls (7 CUDA syncs each) with batched GPU-native argmax reductions (2 syncs per segment). Move deepcopy from per-frame to once per segment on last frame only. Reduces total CUDA sync points from ~28,700 to ~82 per pipeline run. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>012d815 -
feat: add detailed timing metrics to GroundedSAM2 segmenter
Zhen Yed9cce19
-
Fix segmentation mission routing and first-frame GPT alignment
Zhen Ye2fa425d
-
Fix cross-GPU mask device mismatch in GSAM2 reconciliation
Zhen Ye55a2cb0 -
Harden GSAM2 parallel pipeline and tracking reconciliation
Zhen Ye824c842 -
Refine Grounded-SAM2 tracking behavior and video frame handling
Zhen Ye89e1672 -
Remove SAM3 and standardize segmentation on Grounded-SAM2
Zhen Yea637bd3 -
Handle set-typed missing_keys in Grounding DINO loader
Zhen Yeedac021 -
Fix Grounding DINO load reliability and writer error handling
Zhen Yee4ed98a