README.md

metadata

title: Forensic Touch Tracker
emoji: 🕵️
colorFrom: blue
colorTo: red
sdk: gradio
sdk_version: 4.44.0
python_version: '3.10'
app_file: app.py
pinned: false

🕵️ Forensic Touch Tracker

A computer-vision pipeline for tracking human movements and logging touch points so forensic investigators know exactly where to seek fingerprints.

How it works

Stage	Model	What it does
1. Person tracking	SAM3 Video (facebook/sam3)	Detects & tracks every person with a persistent ID across all frames
2. Surface segmentation	SAM3 Video (facebook/sam3)	Segments forensic-relevant surfaces: door handles, countertops, desks, walls, railings, keyboards, phones, cups, bottles, drawer handles
3. Body pose	ViTPose (usyd-community/vitpose-base-simple)	Estimates 17 COCO keypoints per person (wrists used as coarse touch proxies)
4. Hand landmarks	MediaPipe Hands	Detects 21 landmarks per hand, extracting 5 fingertip locations for precise touch localization
5. Contact inference	Proximity-based geometric check	When a fingertip or wrist is within contact_threshold_px of a segmented surface for ≥ min_contact_frames consecutive frames, a TouchEvent is logged

Forensic output format (JSONL)

Each line in forensic_touch_log.jsonl is a structured event:

{
  "timestamp_seconds_start": 12.47,
  "timestamp_seconds_end": 14.12,
  "frame_index_start": 374,
  "frame_index_end": 423,
  "person_id": 3,
  "body_part": "hand_0_index_tip",
  "touch_point": {"x": 412, "y": 298},
  "surface": "countertop",
  "confidence": 0.91,
  "num_frames": 50
}

This gives forensic teams:

• Who touched it (person_id)
• What was touched (surface)
• Where on the frame (touch_point.x, touch_point.y)
• When (timestamp_seconds_start, timestamp_seconds_end, frame_index_start, frame_index_end)
• How sure (confidence)

Running locally

pip install torch transformers opencv-python mediapipe gradio numpy Pillow
python app.py

Then open http://localhost:7860 and upload a video.

Hardware note: SAM3 is ~860M params. A GPU with ≥16 GB VRAM (e.g. T4, A10G) is strongly recommended.

Configuration knobs

Parameter	Default	Description
max_frames	0	Limit frames for quick tests (0 = process whole video)
contact_threshold_px	25	Pixel distance from fingertip to surface mask to declare a "touch"
min_contact_frames	3	Consecutive frames required before logging (reduces false positives)
confidence_threshold	0.5	Minimum detection score for person / surface
surface_prompts	12 prompts	Text prompts SAM3 uses to find forensic-relevant surfaces

Limitations & future upgrades

• No direct "contact classifier" is available on HuggingFace yet. The current pipeline uses geometric proximity, which works well but is not as precise as research models such as InteractVLM, DECO, or HOT (which can infer 3D body-scene contact from images). If you need research-grade contact detection, those repos can be integrated as an additional stage.
• Depth / stereo is not used; true 3D contact would benefit from a depth sensor or monocular depth estimator (e.g. Depth Anything V2).
• Hand keypoint quality depends on resolution; very small hands in wide surveillance shots may need an upsampling step.

Citation / References

• SAM 3: Segment Anything with Concepts (Meta AI, 2025)
• ViTPose: ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation (arXiv:2204.12484)
• RT-DETR: DETRs Beat YOLOs on Real-time Object Detection (arXiv:2304.08069)
• MediaPipe Hands (Google)
• InteractVLM / DECO / HOT — research-grade contact detection (not yet on HF Hub)

---

Built for forensic evidence collection & verification. 🚔🔬