Spaces:
Paused
Paused
Commit History
-
chore: clean up dead code, stale comments, and misleading names
Zhen Ye +1: Claude Opus 4.6 (1M context)- Rename gpt_data -> metadata on STrack (tracker.py) - Remove dead inject_metadata method and phantom METADATA_SYNC_KEYS - Fix _sync_data to sync depth_rel (the key actually written) - Remove dead _build_display_label and gpt_distance_m checks - Remove no-op add_no_cache_header middleware - Remove unused get_segmenter_detector import - Cache index.html at module load instead of reading per request - Extract _parse_queries helper to deduplicate query parsing - Remove empty set_track_data calls in segmentation writer loop - Remove sentence-transformers semantic matching from coco_classes - Clean stale GPT/mission references from docstrings - Update CLAUDE.md to reflect stripped-down architecture Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>2e2a601 -
refactor: replace frontend SPA with minimal detection UI
Zhen Ye +1: Claude Opus 4.6 (1M context)Delete the full Mission Console frontend (frontend/ directory) and replace with a single index.html — vanilla HTML/JS with video upload, mode/model picker, live MJPEG stream, processed video playback, and collapsible detection JSON viewer. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>165863b -
refactor: remove GPT/mission/relevance system, keep CV-only base
Zhen Ye +1: Claude Opus 4.6 (1M context)Strip all GPT threat assessment, mission parsing, relevance gating, and enrichment logic to create a reusable detection base. All computer vision functionality preserved: detectors, segmenters, depth estimation, object tracking, multi-GPU pipeline, async jobs, and MJPEG streaming. Deleted: utils/{gpt_reasoning,openai_client,threat_chat,relevance, enrichment,mission_parser,schemas}.py Removed: enable_gpt, mission_spec, first_frame_gpt_results params from inference pipeline, jobs, and API endpoints. Removed: /detect/analyze-frame, /reason/track, /chat/threat endpoints. Removed: sentence-transformers, python-dotenv dependencies. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>9574811
-
Update README.md
Verified signature164d8b0
-
fix: hide instance_id from segmentation overlay labels
Zhen Ye +1: Claude Opus 4.6Display only the class name on bounding boxes instead of "instance_id class_name". The instance_id is still preserved in job detection data. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>04c92f3
-
Eliminate redundant JPEG frame loading via shared frame store
Zhen Yec97a5f9
-
feat: expose step param on /detect/async, default to 7
Zhen Ye +1: Claude Opus 4.6Step=7 matches num_maskmem and reduces TTFS by shrinking the first SAM2 propagation segment from 60 frames to 7. Benchmark shows negligible end-to-end overhead (YOLO detection is fast). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>05bd36a -
chore(yolo11): update yolo naming comments
Zhen Ye0719ba5 -
feat(ttfs): instrument segmentation time-to-first-stream
Zhen Ye301e154 -
fix: download yolo11m.pt via huggingface_hub to writable /tmp path
Zhen Ye +1: Claude Opus 4.6ultralytics YOLO() tries to download to CWD which is read-only in the container. Use hf_hub_download to fetch weights to /tmp/yolo_weights then load from that path. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>a4e3c2b -
fix: use YOLO("yolo11m.pt") with writable config dir
Zhen Ye +1: Claude Opus 4.6hf:// prefix not supported in installed ultralytics version. Use standard YOLO("yolo11m.pt") which auto-downloads from CDN. Set YOLO_CONFIG_DIR=/tmp/ultralytics so weights download to a writable path in read-only containers. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>e745021 -
refactor: rename hf_yolov8 → yolo11 across codebase
Zhen Ye +1: Claude Opus 4.6Rename file yolov8.py → yolov11.py, class HuggingFaceYoloV8Detector → Yolo11Detector, and registry key "hf_yolov8" → "yolo11" in all 11 files. Loads YOLO11m COCO-pretrained via hf://Ultralytics/YOLO11. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>f89fa0b -
feat: replace custom YOLOv8m_defence with YOLO11m COCO-pretrained
Zhen Ye +1: Claude Opus 4.6The custom defence weights (spencercdz/YOLOv8m_defence) hallucinated on standard traffic scenes—detecting road markings as cars while missing actual vehicles. Switch to ultralytics YOLO11m with COCO pretrained weights for reliable general-purpose detection. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>53922f5 -
feat(segmentation): add YSAM2 variants and mission-aware detector mapping
Zhen Ye6268ac2 -
chore: remove dead code and unused frontend modules
Zhen Yed74c718 -
refactor(gsam2): make SAM2 detector-agnostic
Zhen Ye21c29ae -
perf: increase production keyframe step from 20 to 60
Zhen Ye +1: Claude Opus 4.6Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>97b3a45 -
perf: set SAM2 num_maskmem to 7 (trained default) in production
Zhen Ye +1: Claude Opus 4.6Previously hardcoded to 3 which degraded occlusion recovery. 7 is the value SAM2 was trained with, giving best quality/speed tradeoff. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>0c8527d
-
feat: expose num_maskmem parameter in benchmark profile/analysis endpoints
Zhen Ye +1: Claude Opus 4.6Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>afbbf27 -
feat: add benchmark profiler & roofline analysis system
Zhen Ye +1: Claude Opus 4.6Add hardware extraction, per-frame GPU/CPU profiling with CUDA events, and automated roofline analysis for detection and segmentation modes. New endpoints: - GET /benchmark/hardware — cached hardware specs - POST /benchmark/profile — per-frame timing breakdown - POST /benchmark/analysis — full roofline with bottleneck ID Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>078b447
-
fix: keep track positions fresh during pause for always-clickable bboxes
Zhen Ye +1: Claude Opus 4.6Decouple track prediction and backend sync from the play/pause guard so hit-test data stays valid regardless of playback state. Add 1.5% hit margin around bboxes to account for visual-vs-tracked position drift. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>0ace9ca -
fix: replace gated GSAM2 streaming with adaptive-rate publisher thread
Zhen Ye +1: Claude Opus 4.6The old streaming approach (60-frame startup buffer + 20-frame safety threshold + 3x frame duplication) prevented any frames from reaching the frontend during processing. Replace with a dedicated publisher thread that measures production rate during a 5s startup window, then streams at 85% of measured rate with adaptive adjustments and heartbeat keepalives. Reduce endpoint startup buffer from 30 to 5 frames. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>f09ca9c -
feat: bidirectional click-to-select between video overlay and track cards
Zhen Ye +1: Claude Opus 4.6Enable engage overlay canvas (was hidden with display:none), draw selection highlight on selected track, wire up renderEngageOverlay on track-selected event, and auto-scroll selected card into view. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>b1974e2 -
fix: SAM2 _prep_prompts() signature
Zhen Ye74105c1 -
perf: defer GPU->CPU mask transfer to render thread via LazyFrameObjects
Zhen Ye +1: Claude Opus 4.6Replace bulk segment_output.to_object_dicts() CPU transfer with lazy materialization — only the last frame per segment is materialized for IoU tracking continuity, while render frames use LazyFrameObjects that defer GPU->CPU copy until the render thread needs them. Reduces CUDA syncs from 100+ per segment to ~3. Also removes deployment_logs.txt. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>a1ae44f -
fix: track cards update during playback + periodic GPT re-analysis
Zhen Ye +1: Claude Opus 4.6Fix 4 interconnected bugs preventing track card updates during video playback: normal tracking code trapped inside demo-mode guard, renderFrameTrackList never called in animation loop, state.detections/ tracks disconnect, and asyncJobId nulled after completion blocking syncWithBackend. Cards now update every 40 frames with GPT re-analysis firing async on each cycle. New /detect/analyze-frame endpoint provides per-frame GPT threat assessment. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>6a99834 -
fix: correct benchmark metrics — remove double-counting and add missing timers
Zhen Ye +1: Claude Opus 4.6- Fix sam_video_total_ms double-counting in single-GPU process_video() path (propagate_segment already accumulates, outer timer was adding it twice) - Fix id_reconciliation_ms in multi-GPU path to measure actual IoU work instead of wall-clock queue waits - Add model_load_ms and init_state_ms metrics for both single/multi-GPU paths Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>2f284f5
-
feat: click-to-select tracks on video overlay
Zhen Ye +1: Claude Opus 4.6Add click handler on engage overlay canvas for selecting tracked objects directly in the video. Hit-tests against track bboxes with smallest-area priority for overlapping objects, dispatches track-selected event. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>3015ea3 -
perf: pipeline GSAM2 tracking + rendering with startup buffer
Zhen Ye +1: Claude Opus 4.6Pipeline tracking and rendering so segments stream as they're tracked instead of waiting for all tracking to complete before rendering begins. - Add on_segment callback to process_video() for incremental feeding - Hoist render/writer infrastructure above tracking branch - Single-GPU: callback-based incremental feeding via on_segment - Multi-GPU: streaming reconciliation with segment_buffer pattern - Writer startup buffer (60 frames) before streaming begins - 3x frame duplication for 18 FPS stream (6 FPS processing throughput) - Safety threshold: pause streaming when buffer < 20 frames - try/finally sentinel safety for render worker shutdown - Increase render_out queue from 64 to 128 for buffer headroom Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>7124ca1 -
removed extra bbox lable and track id
Zhen Ye882ee33 -
perf: set num_maskmem=3 for all inference paths
Zhen Ye +1: Claude Opus 4.6Reduces SAM2 temporal memory bank from 7 to 3 frames on both the sync /detect endpoint and the async background job path to lower GPU memory usage with minimal tracking quality impact. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>e9834ad -
feat: add num_maskmem parameter to /benchmark endpoint
Zhen Ye +1: Claude Opus 4.6Thread num_maskmem override through the full pipeline so benchmark runs can test reduced memory bank sizes (e.g. 3 instead of default 7). Patches the SAM2 video predictor's maskmem_tpos_enc at runtime. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>1c6c619 -
revert: remove torch.compile — runtime failures on HF Space
Zhen Ye +1: Claude Opus 4.6Reverts all torch.compile changes (b17bd6d..10eb3c6). The HF Space container lacks Triton/inductor support, causing 500 errors. The GPU-resident tensor pipeline from 5aec47c is retained. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>5749bd6 -
fix: drop fullgraph=True from torch.compile for broader compatibility
Zhen Ye +1: Claude Opus 4.6fullgraph=True requires no graph breaks, which fails without SAM2VideoPredictorVOS's .clone() patches. Removing it allows torch.compile to handle graph breaks gracefully via partial compilation. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>10eb3c6 -
fix: add runtime fallback for torch.compile inductor/triton failures
Zhen Ye +1: Claude Opus 4.6torch.compile wrapping succeeds immediately but actual compilation happens lazily on first forward pass. Now catches runtime errors on the first propagate_in_video yield, reverts all compiled forwards to eager originals, and retries the propagation transparently. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>4c36e1e -
fix: manual torch.compile with graceful fallback for SAM2 components
Zhen Ye +1: Claude Opus 4.6Replace vos_optimized=True (caused 500 error) with manual per-component torch.compile matching Facebook's official recipe. Compiles image_encoder, memory_encoder, memory_attention (dynamic=True), sam_prompt_encoder, and sam_mask_decoder with max-autotune. Falls back to eager mode silently if PyTorch version or CUDA compiler is unavailable. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>63684e4 -
perf: enable torch.compile for SAM2 via vos_optimized flag
Zhen Ye +1: Claude Opus 4.6Uses Facebook's official SAM2VideoPredictorVOS which compiles all five model components (image_encoder, memory_encoder, memory_attention, sam_prompt_encoder, sam_mask_decoder) with torch.compile max-autotune. First inference has ~30s warmup cost; subsequent frames benefit from fused kernels and reduced memory round-trips. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>b17bd6d -
perf: GPU-resident tensor pipeline for SAM2 video propagation
Zhen Ye +1: Claude Opus 4.6Eliminate all CUDA synchronization from propagate_segment() by keeping masks, bboxes, and validity flags on GPU in pre-allocated buffers. CPU materialization is deferred to a single bulk transfer via SegmentOutput.to_object_dicts() at the consumer. - Add _bbox_gpu() for zero-sync bounding box computation on GPU - Add SegmentOutput dataclass for GPU-resident segment results - Rewrite propagate_segment() with inline bbox + pre-allocated tensors - Refactor process_video() to reuse propagate_segment() - Update Phase 4 reconciliation: 3 CUDA syncs per segment vs 100+ Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>5aec47c
-
perf: eliminate CUDA sync points in SAM2 video propagation hot-path
Zhen Ye +1: Claude Opus 4.6Replace per-object-per-frame torch.nonzero + .item() calls (7 CUDA syncs each) with batched GPU-native argmax reductions (2 syncs per segment). Move deepcopy from per-frame to once per segment on last frame only. Reduces total CUDA sync points from ~28,700 to ~82 per pipeline run. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>64f68de -
feat: add /gpu-monitor endpoint for live nvidia-smi streaming
Zhen Ye +1: Claude Opus 4.6Streams nvidia-smi dmon output over HTTP for real-time GPU utilization monitoring during benchmark runs. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>fc9835a -
feat: add detailed timing metrics to GroundedSAM2 segmenter
Zhen Ye74ae00c -
feat: add job processing timing and latency measurements to backend and frontend
Zhen Ye517108e
-
feat(frontend): improve UI for detection and chat interaction
Zhen Yed257dcc -
feat(backend): enhance inference pipeline with GLM logic and structured outputs
Zhen Yebb6e650 -
Suppress labels for non-relevant objects in visualization
Zhen Yeb3371b1 -
Refactor GPT sync in client.js: extract _syncGptFromDetections and add final sync on job completion
Zhen Ye56e4f43 -
Fix GSAM2 label rendering double-draw and sync GPT enrichment to job storage
Zhen Ye87cfe90
-
Handle GPT refusal paths and preserve assessment status
Zhen Yeaadca27 -
Fix segmentation mission routing and first-frame GPT alignment
Zhen Yed73eff6 -
Fix GSAM2 GPT writer state safety and background call args
Zhen Ye8e10ddb