Spaces:
Runtime error
Runtime error
Commit History
-
fix: remove few-shot examples from SuggestMissions to prevent parroting
Zhen Ye +1: Claude Opus 4.6 (1M context)Replace concrete example missions with abstract pattern descriptions so the LLM generates original missions grounded in the actual frame instead of copying template text. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>ce4263e -
fix: tune SuggestMissions prompt for visual inference missions
Zhen Ye +1: Claude Opus 4.6 (1M context)Shift mission suggestions from spatial/temporal patterns to visual sub-classification — detector finds broad category, assessor infers sub-type/attributes from object appearance within the bbox crop. No spatial relationships, counting, or motion reasoning. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>66a4c69
-
fix: replace model names with analyst-style labels in reasoning trace
Zhen Ye +1: Claude Opus 4.6 (1M context)GPT-4o → Primary Analyst, Claude → Validator A, Gemini → Validator B. Loading text now reads "Cross-validating with intelligent analysts..." Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>3954da2 -
fix: chat tooltip clipping and inspect empty panel on track click
Zhen Ye +1: Claude Opus 4.6 (1M context)- Append chip tooltip to document.body with fixed positioning instead of parent element inside overflow:hidden chat panel - Broaden inspect track lookup to search _currentFrameTracks and chat context when track is missing from _realTracks Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>5940d42 -
feat: auto-suggest ISR missions from video preview frame
Zhen Ye +1: Claude Opus 4.6 (1M context)On video select, extract a frame client-side and send to GPT-4o-mini vision to suggest 3-5 feasible mission objectives. Suggestions render as clickable chips below the mission input — click to populate. - New BAML SuggestMissions function with detailed vision prompt constraining suggestions to COCO-detectable, analyst-decidable missions - New POST /suggest-missions endpoint - Frontend frame extraction with race guard, video/canvas cleanup, and 512px cap for efficient LLM processing - Suggestion chips with fade-in animation, hover tooltips, selection state Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>7e09916
-
refactor: clean up backend-frontend connection layer
Zhen Ye +1: Claude Opus 4.6 (1M context)- Remove dead computeRealMetrics() function (93 lines) - Fix SVG event listener leak: use event delegation instead of per-element listeners that accumulated on every frame rebuild - Extract _require_job() helper to deduplicate 7 endpoint job lookups - Extract _enrichTrack() helper to deduplicate 4 track enrichment sites - Extract 7 magic numbers to named constants (poll interval, retries, etc.) - Display PlanMission's planned_queries in video badge so users see what the detector is actually looking for Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>f34ecf3
-
fix: decouple PlanMission from AI postprocessing toggle
Zhen Ye +1: Claude Opus 4.6 (1M context)PlanMission now always runs when mission text is provided, so users can type natural language objectives regardless of the AI toggle. The toggle only controls the ISR assessor loop (LLM vision postprocessing). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>10f8d82 -
feat: improve ISR mission planner and assessor prompts
Zhen Ye +1: Claude Opus 4.6 (1M context)- Add 15 BAML PlanMission tests with strict assertions: max query count caps and negative assertions to prevent detector overload with unrelated COCO classes - Tighten PlanMission prompt to output only mission-targeted object classes (1-4 items) instead of padding with context classes (3-8) - Replace noisy DetectionInfo fields (bbox_width_px, bbox_height_px, area_ratio) with signal-rich fields (confidence, center_x, center_y) - Lean AssessDetections prompt: remove hardcoded examples, add metadata reading guide for confidence/position/speed interpretation Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>0ebc456 -
fix: race conditions in job cleanup and MJPEG frame drop visibility
ClaudeUnverified signature- Fix cleanup_expired() to delete jobs atomically under a single lock, preventing reads on stale job IDs during concurrent access - Bump job TTL from 1 hour to 6 hours so users reviewing results don't lose data mid-session - Add frame drop counting and logging to MJPEG streaming so backpressure issues are observable in logs https://claude.ai/code/session_01RWetXqefq2WKvF8uWBAitn6ecfb35
-
revert: restore original single-video output with baked annotations
Zhen Ye +1: Claude Opus 4.6 (1M context)Reverts 7e455a2 and 86538ab. Both detection and segmentation modes return a single output.mp4 with baked bounding boxes/masks, matching the original behavior before the clean/annotated video separation. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>e737957 -
fix: serve annotated video for segmentation mode (masks are the output)
Zhen Ye +1: Claude Opus 4.6 (1M context)Detection mode serves clean video (SVG overlay handles boxes). Segmentation mode serves annotated video by default since pixel-level masks cannot be replicated by the SVG overlay. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>86538ab -
fix: resolve inspect endpoint errors — frame fallback search and explain error handling
Zhen Ye +1: Claude Opus 4.6 (1M context)/inspect/frame: when a track isn't found at the requested frame, search all frames for the closest match and re-extract from there. Fixes 404 when playhead is at frame 0 but the track appears later. /inspect/explain: add catch-all exception handler and null-result check to prevent unhandled 500 errors from LLM pipeline failures. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>6042c18 -
feat: separate clean video from annotated video for interactive playback
ClaudeUnverified signatureoutput.mp4 now contains clean frames (no baked annotations) for SVG overlay playback. annotated.mp4 contains baked bounding boxes/masks for download. MJPEG live stream continues to show annotated frames during processing. The video endpoint supports ?annotated=true to fetch the annotated version. https://claude.ai/code/session_01QXV3vwibtx4FfArcouKD1E7e455a2
-
fix: resolve backend-frontend integration issues across API contract, error handling, and resource management
Zhen Ye +1: Claude Opus 4.6 (1M context)Backend: add progress reporting (frames_processed/total_frames) to status endpoint, add bulk tracks batch endpoint, fix 404 for missing jobs on track endpoints, bound MJPEG stream queue with backpressure, clean up CANCELLED jobs, consistent HTTPException. Frontend: replace silent catch blocks with console.warn logging, use real data with mock fallback in analysis/timeline/report views, fix blob URL memory leak, prune unbounded _realTracks list, use batch endpoint for timeline event derivation. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>ed56ca1 -
chore: relax pinned versions in pyproject.toml, let uv.lock handle exact pins
Zhen Ye +1: Claude Opus 4.6 (1M context)Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>6314bba -
refactor: migrate to uv, prompt-tune BAML mission planner and assessor
Zhen Ye +1: Claude Opus 4.6 (1M context)- Replace requirements.txt with pyproject.toml + uv.lock for faster, deterministic installs - Update Dockerfile to use uv instead of pip - Prompt-tune PlanMission to output broad detector categories only (no mission-specific labels) - Prompt-tune AssessDetections as the precise filter using visual context - Harden encode_frame for dtype/channel edge cases - Add negative assertions to BAML tests enforcing separation of concerns Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>18a11bc
-
feat: add detailed BAML logging for mission planner and assessment verdicts
Zhen Ye +1: Claude Opus 4.6 (1M context)Logs mission plan reasoning, assessment inputs (detections sent to GPT), and per-track verdict details (relevant, satisfies, reason, features). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>27d5248 -
fix: add missing baml-py dependency to requirements.txt
The generated baml_client requires baml-py==0.220.0 but it was not listed in requirements.txt, causing an ImportError on startup. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>9ee5182
-
refactor: merge verdicts into track responses and remove dead depth code
Root Cause A: get_track_data() now merges verdict fields (satisfies, reason, mission_relevant, features, gpt_raw) onto tracks server-side, eliminating the fragile client-side-only merging that caused missing fields and race conditions. Dead code cleanup (-1,274 lines): - Delete inspection/pointcloud.py (orphaned 538-line module) - Delete tests/test_inspection_pointcloud.py (tested removed code) - Simplify pointcloud endpoint to Tripo3D-only (400 if no API key) - Remove depth 3D UI, renderLegacy3D, "Compute 3D" button - Remove partial_success field and stale depth comments - Remove unused depth param from createTrack mock data - Fix run_inference call with stale depth positional args Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>b378bb6 -
fix: add missing /detect/verdicts endpoint and clean up imports
The frontend was calling GET /detect/verdicts/{job_id} but no route existed — verdicts were computed but never reached the UI. Also consolidates inline storage imports to top-level and removes stale design-note comments. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>ec9ed21
-
test: add AssessDetections BAML tests with real SAR frame
Two test cases using an actual flood rescue aerial frame: - SAR_PersonOnRooftop: person on rooftop satisfies rescue mission - SAR_DogNotMissionTarget: neither person nor dog satisfies cargo mission Also fixes AssessDetections prompt to use proper system/user roles for OpenAI image compatibility. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>75ec737 -
feat: integrate BAML for mission planning and detection assessment
Add PlanMission function that converts natural language mission objectives into concrete detector queries and a refined mission statement. Rewrite ISRAssessor to use BAML's type-safe AssessDetections function instead of raw OpenAI API calls with manual JSON parsing. Includes two test cases for the BAML playground. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>240e068
-
refactor: remove depth estimation pipeline entirely
Depth estimation (DepthAnything V2) was disabled by default, provided only relative visualization values (hardcoded focal_length=1.0), and was never used for actual distance reasoning (GPT handles that). Removed: - models/depth_estimators/ directory (4 files) - inspection/depth.py (caching, inference, colorization) - tests/test_inspection_depth.py - All depth params from /detect and /detect/async endpoints - Depth model loading and per-frame inference in inference.py - IncrementalDepthStats, colorize_depth_map, _attach_depth_metrics - Depth toggle, fetchDepth, renderDepthHeatmap, renderRealDepth from frontend - Depth quadrant from inspect panel (now 3 quadrants: seg, edge, 3D) - Depth fields from JobInfo dataclass - Depth-based 3D fallback in pointcloud endpoint (Tripo3D only now) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>1fad3ed -
fix(demo): auto-detect API base URL instead of hardcoding HF Space
Uses same-origin ('') when served locally, window.location.origin on HF. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>350043a -
feat(demo): add AI Postprocessing toggle for vision-only mode
When toggled off, the pipeline runs strictly stage 1 (detection/tracking) with no GPT assessment, no verdict fetching, and label-based colors. Topbar reflects mode: VISION ONLY / DETECTING / COMPLETE. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>1f50fb2 -
refactor(pipeline): decouple vision stage from AI postprocessing
Eliminate shared mutable state between detection/tracking (stage 1) and GPT assessment (stage 2) so they can be developed independently. - Replace _merge_verdicts() with _store_verdicts() that writes to a separate verdict store instead of mutating detection dicts in-place - Expand _mission_verdicts to hold full verdict dicts with cached bool projection for hot-path rendering - Add set_verdicts_batch() for single-lock-acquisition bulk writes - Add GET /detect/verdicts/{job_id} endpoint for frontend verdict access - Add fetchVerdicts() in api.js to populate assessment cache from new endpoint Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>ecfb6e4 -
refactor(demo): extract shared canvas helpers in inspect panel
Extract _setupQuadCanvas() and _traceSilhouette() from duplicated code across renderSegmentation, renderEdgeDetection, and renderDepthHeatmap. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>7c7831c -
refactor: clean up backend utilities and background job handler
- Simplify background.py exception handling (merge two except blocks) - Remove unused imports and dead code in models.py, isr/utils.py - Use get_pil_rgb() in frame_store adapter instead of manual BGR conversion - Remove redundant dd benchmark variables in hardware_info.py - Remove unused decode timing vars in profiler.py Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>c1e17cf -
refactor(tracker): clean up ByteTracker implementation
- Extract _xyah_from_xyxy to module-level function (was duplicated in KalmanFilter and STrack) - Remove unused _sync_data method - Simplify linear_assignment unmatched index computation - Use sets for duplicate track removal - Remove dead code and stale comments Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>fc033d4 -
refactor(detectors): extract shared YOLO logic into YoloBaseDetector base class
Eliminates ~250 lines of duplicated predict/predict_batch/tiled inference code between Yolo11Detector and YoloV8VisDroneDetector. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>082b546 -
refactor: anonymize LLM labels, merge fragmented tracks, and tune defaults
Replace GPT-4o/Claude/Gemini branding with generic Analyst 1/2/3 labels in both frontend and backend. Add track merging to deduplicate fragmented IDs for the same physical object. Filter non-mission-relevant objects from timeline events. Bump frame store budget to 24 GiB, increase default segmentation step to 12, auto-detect API base URL, and fix video playback reliability (blob MIME type, canplay wait, timeout increase). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>fec2ee8
-
fix(demo): improve 3D graph node label clarity and initial camera framing
Zhen Ye +1: Claude Opus 4.6 (1M context)Labels now float above node spheres instead of overlapping them, with larger text, higher contrast, and more opaque backgrounds. Progressive zoom-to-fit ensures the entire graph is visible on load. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>0b26519
-
feat(demo): threat-weighted timeline waveform
Zhen Ye +1: Claude Opus 4.6 (1M context)Replace detection-count density with threat concentration: - Bar height now represents threat level, not raw object count - Colors: critical (red), high (amber), elevated (blue), low (gray) - Mock data peaks at drone entries, speed violations, perimeter breaches - Real mode: density value maps to same threat color scale - Legend updated to show threat levels instead of object types - Removes DENSITY_TYPES (object-type coloring), adds DENSITY_THREAT A PM now sees "where are the threats" not "where is it busy." Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>4dd14f6
-
fix(demo): double-rAF in inspect panel to ensure layout before rendering
Zhen Ye +1: Claude Opus 4.6 (1M context)Canvas dimensions were 0 when rendering started because the browser hadn't computed layout yet after the DOM change. Double-rAF guarantees layout is ready before measuring canvas dimensions. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>1566eaf -
refactor: remove /app frontend, consolidate on demo as sole UI
Zhen Ye +1: Claude Opus 4.6 (1M context)The demo (/) is the primary UI with command bar, state machine, event log, and reasoning trace. The /app frontend was a secondary analyst tool with less operational capability. All Tripo3D and inspection features are already ported to the demo. - Remove frontend/ directory and /app mount from app.py - Update CLAUDE.md to reflect demo/ as the sole frontend - Update no-cache middleware to target /demo instead of /app Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>2c64c05 -
feat(demo): port Tripo3D generative 3D support to demo inspection panel
Zhen Ye +1: Claude Opus 4.6 (1M context)Replace auto-loading 3D mesh with on-demand prompt showing two buttons: "Compute 3D" (depth-based) and "Generate AI 3D" (Tripo3D API). Adds GLB rendering via GLTFLoader when Tripo returns a model. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>100e159 -
fix(inspect): show errors instead of blank panel + make 3D on-demand
Zhen Ye +1: Claude Opus 4.6 (1M context)Inspection panel previously showed nothing when jobId was missing or API calls failed. Now displays descriptive errors in all quadrants. Tripo3D generation is gated behind an explicit button to avoid API cost. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>6315bd7 -
feat: integrate Tripo3D generative 3D models into inspection panel
Zhen Ye +1: Claude Opus 4.6 (1M context)Replace depth-based back-projection with Tripo3D's image-to-3D API for the 3D quadrant. Falls back to the existing depth mesh when the API key is not set or generation fails. Backend: - New inspection/tripo3d.py: API client with upload, generate, poll, download flow + in-memory LRU cache (50 entries) - Modified pointcloud endpoint: tries Tripo3D first, returns GLB binary with Content-Type: model/gltf-binary, falls back to depth JSON Frontend: - New GLTFLoader vendor file (UMD) for parsing GLB models - fetchPointCloud detects Content-Type to handle GLB vs JSON responses - New renderGLB() in inspection-3d.js with full scene setup - Dispatch logic in inspection.js routes to renderGLB vs legacy render - Texture disposal added for GLB model cleanup Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>930a645 -
fix(demo): don't auto-expand chat panel on track click
Zhen Ye +1: Claude Opus 4.6 (1M context)- Clicking a track loads context silently (header + toggle appear) - Panel only expands when user sends a message - Show brief toast on context load for feedback Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>663eabf -
feat(demo): friendly track names — "Truck #3" instead of "3 truck"
Zhen Ye +1: Claude Opus 4.6 (1M context)- Add _trackName(t) helper: capitalizes label, appends #id - Header chips: "Truck #3 ●" instead of "3 truck ●" - Tooltip: "Truck #3 MATCH" instead of "3 truck MATCH" - Suggestions: "What is Truck #3?" instead of "What is 3?" - Context markers: "Truck #3 removed" instead of "3 truck removed" Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>7f37c5f -
fix(demo): skip SVG bbox overlay in segmentation mode
Zhen Ye +1: Claude Opus 4.6 (1M context)Backend already renders masks directly into the video frames, so the frontend bounding box rectangles were just adding visual clutter. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>2fba73b -
style(demo): elevate chat panel visual design
Zhen Ye +1: Claude Opus 4.6 (1M context)- Glass morphism panel background with gradient + backdrop blur - Context header with subtle accent gradient sweep - Pill-shaped chips with accent-tinted borders, hover glow - Suggestion chips with blue tint, hover shadow aura - Messages with colored left-border accent (blue for user, purple for AI) - Tooltip with deep shadow + accent-tinted border, smooth entrance - Refined spacing, letter-spacing, font weights throughout - Uses CSS variable system consistently (mission colors, accent, etc.) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>f14a5fa
-
fix(demo): use Three.js 0.160 which has UMD build (0.175 dropped it)
Zhen Ye488b1d4 -
feat(demo): add text labels to 3D graph nodes + spread out hierarchy
Zhen Ye +1: Claude Opus 4.6 (1M context)- Add three-spritetext for readable text labels on every node - Labels show node name, category counts, verdict status - Increase dagLevelDistance (50→80) for clearer hierarchy - Add charge repulsion (-120) + link distance (60) to decluster - Load Three.js 0.175 before 3d-force-graph so both share same instance Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>bd68129 -
feat(demo): chip hover tooltip + single-object suggestion queries
Zhen Ye +1: Claude Opus 4.6 (1M context)- Hover over context chip shows mini track card tooltip with confidence, speed, depth, mission status badge, and reason text - Tooltip appears below chip on mouseenter, disappears on mouseleave - Suggestions now appear for 1+ objects (was 2+): - Single object: "What is X?", mission relevance, heading/threat - Multi object: comparison queries (unchanged) - All suggestions capped at 3 for clean UI Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>3883436 -
feat(demo): collapsible chat panel toggle, matching event log pattern
Zhen Ye +1: Claude Opus 4.6 (1M context)- Add CHAT toggle button in command bar (hidden until first content) - Click toggles panel collapsed/expanded with arrow indicator - Button appears when tracks are selected or messages sent - Arrow syncs: ▲ when collapsed (click to expand), ▼ when open - chatClear hides toggle and collapses panel - Styled consistently with EVENT LOG toggle Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>66e0ddc -
fix(demo): extract glow rect helper, fix selectedTrackIds desync
Zhen Ye +1: Claude Opus 4.6 (1M context)- Extract _createGlowRect() in real-backend.js, replacing 4 duplicate blocks (-23 lines) - Fix handleCanvasClick bypassing toggleTrackSelection (selectedTrackIds desync in mock mode) - Add defensive guard in enterInspectState to ensure trackId is in selectedTrackIds - Simplify _escapeHtml alias to direct ISR.escHtml reference Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>5221fff -
fix(demo): address code review — double-push, double-execute, dedup toggle
Zhen Ye +1: Claude Opus 4.6 (1M context)- Remove chatHistory pushes from askAI (chat.js now owns history) - Remove separate executeAction from handleCommand (chatSend handles it) - Extract toggleTrackSelection() into helpers.js, replace 3 duplicates - Use ISR.escHtml from helpers.js instead of private _escapeHtml Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>61a13a6 -
feat(demo): multi-object context + expandable chat panel
Zhen Ye +1: Claude Opus 4.6 (1M context)Port multi-object context and comparison system to the demo UI: - Add expandable chat panel that grows from command bar on first message - Persistent context header with track chips (ID, label, status dot) - Smart suggestion chips when 2+ objects loaded (threat, speed, class comparison) - Multi-select track cards: click to toggle into/out of context - Last-clicked track gets brighter SVG highlight + inspect panel - API sends active_objects array to backend for multi-object comparison - Chat module (chat.js) manages contexts, messages, and suggestions Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>330b098 -
fix(demo): restructure 3D graph as top-down hierarchy + remove duplicate Three.js
Zhen Ye +1: Claude Opus 4.6 (1M context)- Top: Object node → Middle: Categories fan out → Features → Bottom: Verdict - dagMode('td') for clear top-down tree structure - Remove standalone three.min.js (3d-force-graph bundles its own) - Remove nodeThreeObject (avoids THREE global dependency) - Green/red feature nodes show validator consensus at a glance - Rich HTML tooltips on hover with full LLM reasoning Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>3fa5f70