| # RECON.md |
|
|
| Phase 0 reconnaissance findings — model verification, Gradio APIs, access status. |
| Updated: June 4, 2026. |
|
|
| ## Gradio |
| - Version: TBD (will verify on first `pip install gradio`) |
| - gr.Blocks: expected ✓ (used in app.py skeleton) |
| - gr.Video: expected ✓ |
| - gr.Walkthrough / gr.Step: TBD (verify in Phase 2) |
| - gr.Navbar: TBD (verify in Phase 2) |
| - UI approach: gr.Blocks + custom CSS/theme (escalate to Server only if needed) |
|
|
| ## Python |
| - Python 3.13.9 (local dev) |
| - pytest 9.0.2, numpy, opencv-python installed |
|
|
| ## Model Verification |
|
|
| | Model | Params | License | GGUF | ZeroGPU | Status | |
| |---|---|---|---|---|---| |
| | YOLO26l-Pose (primary) | 0.026B | AGPL-3.0 | n/a | ✓ (6.5ms T4) | ready | |
| | YOLO26x-Pose (HQ alt) | 0.058B | AGPL-3.0 | n/a | ✓ (12.2ms T4) | ready | |
| | SAM 3.1 base (sam2.1_hiera_base_plus) | ~0.85B | SAM License | n/a | ✓ | access accepted | |
| | SAM 3D Body (facebook/sam-3d-body-dinov3) | 0.84B (DINOv3-H+) | SAM License | n/a | ✓ | **INTEGRATED** | |
| | Sapiens2 Pose (noahcao/sapiens-pose-coco) | ~0.6B | CC-BY-NC-4.0 | n/a | ✓ | access accepted | |
| | ST-GCN (pyskl) | ~0.03B | Apache-2.0 | n/a | ✓ | ready | |
| | Qwen3-VL-8B-Instruct | 8B | Apache-2.0 | ✓ | llama.cpp | ready | |
| | Qwen3-VL-Embedding-8B | 8B | Apache-2.0 | ✓ | llama.cpp | ready | |
| |
| ## Param Sum |
| ~17.63B — well under 32B limit. |
| |
| ## Gated Access Status (as of Jun 4, 2026) |
| - [x] SAM 3.1 (facebookresearch/sam3) — accepted |
| - [x] SAM 3D Body (facebook/sam-3d-body-dinov3) — **ACCEPTED** (confirmed Jun 4) |
| - [x] Sapiens2 Pose (noahcao/sapiens-pose-coco) — accepted |
| |
| ## Open Questions |
| - [ ] Confirm "≤32B" = summed vs per-model in Discord AMA |
| - [ ] AGPL-3.0 YOLO OK for hackathon submission? (Likely yes for non-commercial demo) |
| |
| ## llama.cpp Build Plan |
| - CPU-only build first (avoids libcudart.so issues on Spaces) |
| - Fallback: transformers + spaces.GPU for VLM inference |
| - GGUF quantized Qwen3-VL-8B at Q4_K_M (~4.5GB) |
| |
| ## Key Decisions |
| - Primary pose: YOLO11x-Pose (fastest, well-tested) |
| - Fallback pose: Sapiens2 (more keypoints, slower) |
| - 3D body: INTEGRATED — uses `setup_sam_3d_body()` from `notebook.utils`, outputs MHR joints |
| - API: `estimator.process_one_image(rgb_image)` — single RGB np.ndarray |
| - Model variants: DINOv3-H+ (840M) default, ViT-H (631M) smaller |
| - Temporal smoothing via EMA (alpha=0.3) to reduce single-frame jitter |
| - config.enable_3d=False by default; flipped when checkpoint verified on Space |
| - VLM: Qwen3-VL-8B via llama.cpp (Judge + Classifier) |
| - Embeddings: Qwen3-VL-Embedding-8B via llama.cpp (Retrieval) |
| |
