Upload README.md with huggingface_hub
Browse files
README.md
ADDED
|
@@ -0,0 +1,45 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: mit
|
| 3 |
+
base_model: facebook/vjepa2-vitl-fpc16-256-ssv2
|
| 4 |
+
tags:
|
| 5 |
+
- coreai
|
| 6 |
+
- apple
|
| 7 |
+
- video-classification
|
| 8 |
+
- v-jepa
|
| 9 |
+
- world-model
|
| 10 |
+
- on-device
|
| 11 |
+
pipeline_tag: video-classification
|
| 12 |
+
---
|
| 13 |
+
|
| 14 |
+
# V-JEPA 2 (ViT-L, SSv2 action recognition) — Apple Core AI
|
| 15 |
+
|
| 16 |
+
[V-JEPA 2](https://huggingface.co/facebook/vjepa2-vitl-fpc16-256-ssv2) (Meta AI) running natively on
|
| 17 |
+
the Apple Core AI engine — the zoo's first **world model**: a self-supervised video encoder that
|
| 18 |
+
learns by predicting in representation space (JEPA), here with the Something-Something v2 action
|
| 19 |
+
head (174 classes of *physical interactions* — put/lift/push/roll/cover/pretend…).
|
| 20 |
+
|
| 21 |
+
- **One bundle**: ViT-L backbone (3D RoPE attention) + attentive pooler + classifier, ~375M params,
|
| 22 |
+
fp16 ~675 MB.
|
| 23 |
+
- **I/O**: `pixel_values_videos [1,16,3,256,256]` (16 frames, RGB 0..1, ImageNet mean/std) →
|
| 24 |
+
`logits [1,174]` (`labels.json`).
|
| 25 |
+
- **Verified**: engine vs PyTorch reference cosine 0.999996, top-5 identical; a synthetic
|
| 26 |
+
motion probe (square moving up vs down) flips the predicted direction correctly.
|
| 27 |
+
- **Speed**: ~150–180 ms per 16-frame clip on an M4 Max (GPU) — real-time video understanding.
|
| 28 |
+
|
| 29 |
+
## Files
|
| 30 |
+
|
| 31 |
+
| path | what |
|
| 32 |
+
|---|---|
|
| 33 |
+
| `macos/vjepa2_ssv2_fp16.aimodel` | fp16 bundle (macOS / JIT) |
|
| 34 |
+
| `macos/labels.json` | 174 SSv2 class names |
|
| 35 |
+
| `macos/metadata.json` | I/O + preprocessing spec |
|
| 36 |
+
|
| 37 |
+
## Preprocessing
|
| 38 |
+
|
| 39 |
+
Sample 16 frames uniformly from the clip, resize+center-crop to 256×256, scale to 0..1, normalize
|
| 40 |
+
with ImageNet mean `[0.485,0.456,0.406]` / std `[0.229,0.224,0.225]`, layout `[1,16,3,256,256]`.
|
| 41 |
+
|
| 42 |
+
## Credits
|
| 43 |
+
|
| 44 |
+
- **Meta AI** — [V-JEPA 2](https://github.com/facebookresearch/vjepa2) (MIT).
|
| 45 |
+
- Conversion + Core AI port: [coreai-model-zoo](https://github.com/john-rocky/coreai-model-zoo).
|