itspublu commited on
Commit
ea89fb9
·
verified ·
1 Parent(s): 7a973dd

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +21 -28
README.md CHANGED
@@ -4,11 +4,9 @@ library_name: pytorch
4
  pipeline_tag: image-classification
5
  tags:
6
  - robotics
7
- - egocentric-video
8
  - capture-qa
9
  - hand-detection
10
  - edge
11
- - tinyml
12
  datasets:
13
  - lerobot/droid_1.0.1
14
  - imageomics/KABR
@@ -20,7 +18,7 @@ model-index:
20
  name: Hand/gripper visibility (clip-level)
21
  dataset:
22
  type: referee-lab/hands-visible
23
- name: referee-lab/hands-visible (Prime Intellect Environments Hub)
24
  split: full
25
  metrics:
26
  - type: accuracy
@@ -30,28 +28,29 @@ model-index:
30
 
31
  # referee-hands-s
32
 
33
- A **1.5M-parameter** MobileNetV3-small that answers one question about robot-training
34
- video: *is a hand or gripper visible?* Distilled from gemma3:27b pseudo-labels
35
- (4,138 frames from DROID / ALOHA-sim / KABR — all CC BY / CC0 / MIT), 96px inputs,
36
- built to fit a microcontroller inference budget.
37
 
38
- ## Scoreboard (public benchmark: [referee-lab/hands-visible](https://github.com/publu/referee-lab))
 
 
 
 
39
 
40
  | Model | Params | Accuracy |
41
  |---|---|---|
42
- | gemma3:27b (the teacher) | 27B | 97.8% |
43
- | **referee-hands-s (this model)** | **1.5M** | **97.8%** |
44
  | qwen2.5vl:32b | 32B | 74.4% |
45
- | *do-nothing floor (constant "yes")* | — | *68.9%* |
46
- | qwen2.5vl:7b | 7B | 45.6% |
47
- | EgoSieve-S (hand-visibility head) | 22M | 40.0% |
48
- | MediaPipe HandLandmarker | — | 40.0% |
49
 
50
- Scored on the public 45-clip answer sheet, clip verdict = any frame above 0.5.
51
- **Caveat, stated plainly:** training frames come from different clips but the same
52
- source videos as the benchmark an in-domain result. A cross-video split is the
53
- planned follow-up. The benchmark's own labels are machine-graded provisional with a
54
- published grading record.
55
 
56
  ## Usage
57
 
@@ -64,17 +63,11 @@ net = models.mobilenet_v3_small()
64
  net.classifier[3] = torch.nn.Linear(net.classifier[3].in_features, 1)
65
  net.load_state_dict(load_file("model.safetensors"))
66
  net.eval()
67
-
68
  tf = transforms.Compose([transforms.Resize((96, 96)), transforms.ToTensor()])
69
  p = torch.sigmoid(net(tf(img).unsqueeze(0)).squeeze()) # P(hand/gripper visible)
70
  ```
71
 
72
- Clip-level: sample ~12 frames, answer yes if `max(p) > 0.5`.
73
-
74
- ## Provenance
75
 
76
- Teacher: gemma3:27b (96.7-97.8% on the same benchmark). Training/eval media:
77
- DROID (CC BY 4.0), lerobot aloha sim (MIT), KABR (CC0). The benchmark, its
78
- validation gates, and all baseline runs: https://github.com/publu/referee-lab —
79
- five capture-QA evals, publicly runnable on the Prime Intellect Environments Hub
80
- (`prime eval run referee-lab/hands-visible`).
 
4
  pipeline_tag: image-classification
5
  tags:
6
  - robotics
 
7
  - capture-qa
8
  - hand-detection
9
  - edge
 
10
  datasets:
11
  - lerobot/droid_1.0.1
12
  - imageomics/KABR
 
18
  name: Hand/gripper visibility (clip-level)
19
  dataset:
20
  type: referee-lab/hands-visible
21
+ name: referee-lab/hands-visible
22
  split: full
23
  metrics:
24
  - type: accuracy
 
28
 
29
  # referee-hands-s
30
 
31
+ A 1.5M-parameter MobileNetV3-small classifier for hand/gripper visibility in
32
+ robot-training footage. 96px inputs, sized for microcontroller-class deployment.
 
 
33
 
34
+ ## Benchmark
35
+
36
+ 97.8% clip accuracy on [referee-lab/hands-visible](https://github.com/publu/referee-lab)
37
+ (45-clip public benchmark, robot + wildlife domains) — matching gemma3:27b at
38
+ 1/18,000th the parameters.
39
 
40
  | Model | Params | Accuracy |
41
  |---|---|---|
42
+ | gemma3:27b | 27B | 97.8% |
43
+ | **referee-hands-s** | **1.5M** | **97.8%** |
44
  | qwen2.5vl:32b | 32B | 74.4% |
45
+ | *constant-answer floor* | — | *68.9%* |
46
+
47
+ ## Intended use and scope
 
48
 
49
+ Version 1 targets fixed-camera and wrist-camera **robot workspace footage**
50
+ the domain it was trained on (DROID, ALOHA-sim, KABR negatives). Human
51
+ egocentric and general webcam scenes are outside the v1 training distribution;
52
+ a mixed-domain v2 is in development. Use for capture-pipeline gating in the
53
+ supported domain.
54
 
55
  ## Usage
56
 
 
63
  net.classifier[3] = torch.nn.Linear(net.classifier[3].in_features, 1)
64
  net.load_state_dict(load_file("model.safetensors"))
65
  net.eval()
 
66
  tf = transforms.Compose([transforms.Resize((96, 96)), transforms.ToTensor()])
67
  p = torch.sigmoid(net(tf(img).unsqueeze(0)).squeeze()) # P(hand/gripper visible)
68
  ```
69
 
70
+ Clip-level verdict: sample ~12 frames, yes if `max(p) > 0.5`.
 
 
71
 
72
+ Training: distilled from gemma3:27b pseudo-labels on CC BY / CC0 / MIT media.
73
+ Benchmark, gates, and tooling: https://github.com/publu/referee-lab