eymen160/caught-in-4k

Caught in 4K — bin-drop detector (final, commit 9fa5da7)

A camera watches a set of bins. When a person drops an item, the system says which item, which opening it went into, and whether that was the right bin under the local rules (Cambridge, MA 2022 catalog included).

This repo is everything needed to run and keep improving the final model from the hackathon: the weights, the browser model, the rules, the labeled eval splits and their perception caches, and the scores they produce.

What is (and isn't) trained here

No checkpoint in this repo was fine-tuned. The model is a pipeline of off-the-shelf networks plus our own logic and our own CLIP head:

Stage	Network	File
Person + arms (who is carrying something)	YOLO11n-pose	weights/yolo11n-pose.pt, web/yolo11n-pose.onnx
Item regions near the hand	FastSAM-s	weights/FastSAM-s.pt
Item / background / category (zero-shot)	CLIP ViT-B/32	weights/clip/ViT-B-32.pt
Optional detectors (--tracker yolo / --tracker world)	YOLO11n, YOLOv8s-World v2	weights/yolo11n.pt, weights/yolov8s-worldv2.pt
Face redaction before any crop leaves the device	YuNet (worker), BlazeFace (browser)	weights/face_detection_yunet_2023mar.onnx, web/blaze_face_short_range.tflite

Ours:

• web/guide.json — the browser model: precomputed CLIP text embeddings for 21 item / hazard categories (+ prompt ownership and parent categories) and the bin rules. Generated by apps/web/scripts/build_guide_model.py; do not edit by hand. The in-browser CLIP image encoder is Xenova/clip-vit-base-patch32 at the revision pinned inside it.
• The arm-gated item tracker and drop decision (worker/recyclone_worker/items.py): only objects a person brings count, a drop is an item left at rest inside a bin opening, and a "veto" cancels it if the item comes back into a hand near where it vanished. All tunable constants sit at the top of that file.
• rules/cambridge-ma-2022.bins.json — which catalog item goes in which bin, with the source text in the .md.

Results (final model, from eval/RESULTS-9fa5da7.txt)

Split	Clips	Drops found	Within ±0.5 s	False drops	Item named right*
dev — street (BigBelly-style slot + hopper)	3	4/4	4	0	2/4
dev — phone, indoor bins	5	5/5	4	0	4/5
dev — YouTube	2	1/1	1	0	0/1
dev total	10	10/10	9	0	6/10
test — YouTube	3	1/3	1	0	—
test — fresh 17 Oct	3	0/2	0	0	—
test total	6	1/5	1	0	—

* CLIP naming only. With the vision-LLM namer (--namer deepseek, or Claude in the demo path) dev naming was 9/10.

The dev/test gap is the honest headline: the tracker generalizes poorly to footage it was not tuned on. Known misses: overhand throws (item never rests in the box), the opening hidden behind the bin, and no forearm in view (pose is required — clips without a visible person fall back to the older motion tracker).

Improve it

# 1. code (private repo, team access): the exact version these numbers come from
git clone https://github.com/Ishan-Wakade/Caught_in_4K && cd Caught_in_4K && git checkout 9fa5da7

# 2. this repo
hf download eymen160/caught-in-4k --local-dir ../caught-in-4k-hf
ln -sfn "$PWD/../caught-in-4k-hf/weights" worker/weights

# 3. score every split from the cached perception (seconds, no video needed)
cd worker
E=../../caught-in-4k-hf/eval
uv run python scripts/eval_splits.py \
  $E/splits/dev/dev-street.json:$E/cache/dev-street \
  $E/splits/dev/dev-phone.json:$E/cache/dev-phone \
  $E/splits/dev/dev-youtube.json:$E/cache/dev-youtube \
  $E/splits/test/test-youtube.json:$E/cache/test-youtube \
  $E/splits/test/test-fresh.json:$E/cache/test-fresh

# 4. try a change without editing code, e.g.
uv run python scripts/eval_splits.py ...same args... --set VETO_NEAR=0.4 --set MAX_GROWTH=1.6

Rules we kept to, and you should too:

• Tune on dev only. The test splits are already spent once; every look at them makes the number less true. Add new footage as a fresh test split before you start tuning.
• A change counts only if dev stays at 0 false drops. A false "wrong bin" in front of a person is worse than a miss.
• The caches (*.npz: person keypoints, candidate boxes, CLIP image embeddings; *_motion.npy) were made by the perception at 9fa5da7. If you change perception (detector, FastSAM, CLIP prompts), you need the source videos and scripts/item_tracker.py to rebuild them — the caches only cover the tracking/decision layer.
• To change categories in the browser model, edit the worker catalog/prompts and regenerate web/guide.json with apps/web/scripts/build_guide_model.py.

Data

eval/splits/*.json hold the ground truth per clip: drop frame, bin, accepted item ids, rule, and bin boxes as keyframes. YouTube clips are listed by URL (download them yourself; they are not ours to redistribute). Our own phone clips show real people and are not published — only their caches are.

Licenses

This repo is AGPL-3.0 because it redistributes Ultralytics weights. Each component keeps its upstream license: YOLO11n, YOLO11n-pose, YOLOv8s-World, FastSAM — AGPL-3.0 (Ultralytics) · CLIP ViT-B/32 — MIT (OpenAI) · YuNet — MIT (OpenCV Zoo) · BlazeFace short-range — Apache-2.0 (Google MediaPipe). SHA256SUMS lists every file.

Built by the Caught in 4K team, October 2026.