docs/FormScout-FMS-Spec.md

# FormScout — Functional Movement Screening, scored small

**Project specification & architecture documentation**
*Build Small Hackathon (Gradio × Hugging Face) — Track: Backyard AI*
*Working title; rename freely. Doc version 0.1, June 2026.*

---

## 1. One-paragraph pitch

A basketball team's physiotherapist screens players with the **Functional Movement Screen (FMS)** — seven movement patterns, each scored 0–3 by eye. The scoring is slow, subjective, and hard to reproduce across raters or across months. FormScout is a Gradio app that takes a video of an athlete performing an FMS test, extracts 2D and 3D body pose, measures the biomechanics the FMS rubric actually cares about, and produces a 0–3 score *with a written rationale and an annotated overlay* — anchored to the physio's own previously-scored clips. It is a **screening aid that standardizes and speeds up the physio's first pass**, not a diagnosis and not an injury predictor. Everything runs on models that fit on a laptop.

---

## 2. The problem, honestly

The FMS is a seven-test battery (Deep Squat, Hurdle Step, In-Line Lunge, Shoulder Mobility, Active Straight-Leg Raise, Trunk Stability Push-Up, Rotary Stability), each scored 0–3 for a composite 0–21. A score of 0 means **pain** during the movement and is an automatic red flag for clinical referral. Three of the tests have associated **clearing tests** (shoulder, spinal extension, spinal flexion) that also force a 0 on pain.

Two facts shape this project and should be stated plainly in the demo and the writeup:

- **Inter-rater reliability is decent but not perfect.** Composite-score reliability is moderate-to-good (ICC roughly 0.7–0.8), but novice and less-experienced raters grade component scores inconsistently. This is the real, addressable pain point: **variance between raters and over time.**
- **Predictive validity for injury is weak/mixed.** The popular "≤14 = higher injury risk" cutoff is not a reliable predictor on its own. So FormScout must **not** be sold as injury prediction.

**Where FormScout genuinely helps:**
1. A repeatable, objective **digital baseline** to track an athlete over a season.
2. **Asymmetry detection** (left vs. right), which is one of the FMS's most defensible outputs.
3. A fast, consistent **first-pass / second opinion** that reduces rater variance.
4. **Explainability** — it shows *which compensation* it saw, not just a number.

This honest framing is also strategic: the Backyard AI track is judged partly on "honest fit between problem and the small-model constraint." Overclaiming clinical power would hurt the submission, not help it.

---

## 3. Why this fits the hackathon

| Hackathon rule | How FormScout satisfies it |
|---|---|
| **Total params ≤ 32B** | Recommended config sums to ~18B. A portfolio of small specialists beats one monolith — which is on-theme for "think small." |
| **Built on Gradio, hosted as a HF Space** | Gradio app with `gr.Video` input, a custom-styled results panel, on-Space inference (ZeroGPU or llama.cpp). |
| **Show, Don't Tell** | Demo video = physio uploads a real player clip, gets a scored overlay in seconds. Social post = before/after of a manual vs. assisted screening session. |
| **Track: Backyard AI** | The "someone you know" is the team physiotherapist. The deliverable is something they *actually use* on real players. |

**Badge targets (aim for all six):**

- 🔌 **Off the Grid** — no cloud APIs; all models served on the Space.
- 🎯 **Well-Tuned** — the skeletal-temporal scoring head is fine-tuned on the physio's labels and published to the Hub.
- 🎨 **Off-Brand** — custom Gradio frontend (scorecard UI, video overlay, per-test rubric panel), pushing past default Gradio.
- 🦙 **Llama Champion** — VLM + embedding model served through llama.cpp (GGUF builds exist for both).
- 📡 **Sharing is Caring** — publish the agent trace (one full screening run, agent by agent) to the Hub.
- 📓 **Field Notes** — a blog post on building a clinical-adjacent AQA pipeline under a 32B budget, with the honesty section front and center.

---

## 4. Core technical framing: FMS *is* Action Quality Assessment

Don't reinvent this from scratch. **Action Quality Assessment (AQA)** is the established field for "score how well a movement was performed." Skeleton-based AQA (sports scoring, surgical-skill and rehab assessment) is the directly relevant lineage. The "Skeletal-Temporal Transformer" idea maps onto the **AQA scoring head**.

The key design constraint is the **tiny labeled dataset** (a couple of physio-scored videos). That rules out training a large score regressor from scratch and dictates a hybrid approach:

1. **Deterministic biomechanics** carry most of the load. The FMS rubric is, to a large degree, a set of *angle and alignment thresholds* (e.g. Deep Squat "3" = femur below horizontal, torso parallel to tibia, knees tracking over feet, dowel over feet). These are computable from 3D pose with **zero training** and are inherently interpretable — exactly what earns a physio's trust.
2. **A small learned head** (ST-GCN or a compact temporal transformer) refines the score and captures the patterns rules miss. It is small enough to fine-tune on a few labeled clips, *especially* if pre-trained on public AQA/pose datasets first.
3. **Retrieval over the physio's labeled clips** (RAG) gives the language model few-shot anchors at judgment time — the right move when you have examples but not enough to train on.
4. **A VLM as the judge/explainer** synthesizes rubric + measurements + retrieved exemplars into a final score and a human-readable rationale, and conservatively flags anything pain-related for a human.

---

## 5. Parameter budget (the single most important table)

Assume "total parameters" = **sum of all model weights in the pipeline**. Design to this; confirm the exact interpretation in the Discord AMA.

### Recommended config — "Portfolio of specialists" (~18B)

| Component | Model | Params | Role |
|---|---|---:|---|
| 2D pose + tracking | YOLO26-Pose (L/X) | ~0.05B | Per-frame 17-keypoint skeletons, multi-person tracking |
| Segmentation | SAM 3.1 (base) | ~0.85B | Clean athlete mask, occlusion handling, prompt for 3D |
| 3D body | SAM 3D Body | ~0.7–1B* | Single-image 3D mesh → true joint angles, view-invariant |
| Scoring head | ST-GCN / temporal transformer (fine-tuned) | ~0.01–0.05B | Pose-sequence → candidate 0–3 + confidence |
| Judge / explainer | Qwen3-VL-8B-Instruct | 8B | Movement ID, rubric reasoning, final score + rationale |
| Retrieval | Qwen3-VL-Embedding-8B | 8B | Nearest physio-scored reference clips (RAG) |
| **Total** | | **~17.8B** | Comfortable headroom under 32B |

\* SAM 3D Body's exact count isn't published prominently — verify on the model card. It's SAM-3-family and sub-billion-class; budget impact is small either way. The two 8B Qwen models **share the Qwen3-VL-8B backbone** (the embedder is built on the instruct model), which is conceptually clean and operationally efficient.

### Alternative config — "Heavy reasoner" (~28.7B)

Swap the 8B judge for **Qwen3.6-27B** (multimodal, strong tool-calling, MTP speedups on llama.cpp). Budget then = 27 + ~0.85 + ~1 + small ≈ **28.7B**. This **leaves no room for the 8B embedder**, so you'd drop RAG (or replace it with a sub-0.5B embedder, or use pose-feature similarity for retrieval). Note: Qwen3.6-27B's MTP speculative decoding currently can't run simultaneously with image input (`--mmproj`), so for vision you run it without MTP.

**Recommendation: ship the ~18B portfolio config.** RAG over the physio's few labeled clips is worth more than raw reasoning horsepower on this task, the headroom de-risks the budget, and "many small specialists" is the better hackathon story.

---

## 6. Model selection rationale

**YOLO26-Pose** — current-generation YOLO pose; single forward pass for detection + keypoints, NMS-free, real-time even on edge. Tiny param cost. It also handles **multiple people in frame** (important: team videos often have other players/staff visible) and feeds keypoints downstream. Off-the-shelf it predicts COCO human keypoints; can be fine-tuned for custom landmarks (e.g. dowel endpoints) if needed.

**SAM 3.1** — gives a clean athlete mask and stable multi-object video tracking (Object Multiplex makes it fast). Two jobs: (a) isolate the target athlete from teammates/background so pose and 3D aren't polluted, (b) provide the mask prompt that SAM 3D Body consumes. Concept prompts ("the person in the blue jersey performing the squat") are a bonus for disambiguation.

**SAM 3D Body** — *the addition that makes the scores trustworthy.* FMS criteria are joint angles and symmetry; 2D pose can't measure these reliably across camera angles (projection ambiguity). 3D mesh recovery from a single image, promptable with the 2D keypoints + mask you already have, yields view-invariant joint angles (the MHR rig even separates skeletal structure from soft-tissue shape, which is convenient for angle extraction). This is the difference between "looks bent" and "femur is 4° above horizontal → not a 3."

**Skeletal-temporal scoring head** — your AQA component and your **Well-Tuned** badge. Recommend a compact **ST-GCN** (graph conv over the skeleton, temporal conv over frames) over a from-scratch transformer, because it's far more data-efficient on a tiny labeled set. Pre-train on public AQA / pose-action data, then fine-tune on the physio's labels. Output: per-test candidate score + a confidence the judge can weigh.

**Qwen3-VL-8B-Instruct** — the judge. Strong video temporal modeling (Interleaved-MRoPE, timestamp alignment) suits movement clips. It identifies which of the 7 tests is being performed, reads the biomechanics, considers retrieved exemplars and the head's candidate, and emits the final score + rationale + detected compensation. GGUF → llama.cpp → Llama Champion.

**Qwen3-VL-Embedding-8B** — retrieval. Embeds the query clip (or its keyframes/pose-render) and finds the physio's most similar already-scored clips to anchor the judge. Top multimodal retriever on MMEB-V2; same backbone as the judge; GGUF available.

---

## 7. Architecture — an agentic pipeline

Structured as cooperating specialist agents (maps naturally onto an OFP-style orchestration, with a Director coordinating and quality-gating). Each agent has one job and a typed output.

```
                         ┌──────────────────────────────────────────────┐
   video upload  ───────▶│  IngestAgent                                  │
                         │  decode, normalize FPS, sample frames         │
                         └───────────────┬──────────────────────────────┘
                                         ▼
                         ┌──────────────────────────────────────────────┐
                         │  SegmentationAgent  (SAM 3.1)                 │
                         │  athlete mask + track id (reject teammates)   │
                         └───────────────┬──────────────────────────────┘
                                         ▼
              ┌──────────────────────────┴──────────────────────────┐
              ▼                                                      ▼
  ┌───────────────────────────┐                      ┌───────────────────────────┐
  │ PoseAgent (YOLO26-Pose)    │                      │ Body3DAgent (SAM 3D Body)  │
  │ 2D keypoints per frame     │ ───keypoints+mask──▶ │ 3D mesh / joint angles     │
  └───────────────┬───────────┘                      └───────────────┬───────────┘
                  └─────────────────────┬────────────────────────────┘
                                        ▼
                         ┌──────────────────────────────────────────────┐
                         │  MovementClassifierAgent                      │
                         │  which of the 7 FMS tests? (VLM or small CLS) │
                         └───────────────┬──────────────────────────────┘
                                         ▼
              ┌──────────────────────────┴──────────────────────────┐
              ▼                          ▼                           ▼
  ┌────────────────────┐   ┌─────────────────────────┐   ┌────────────────────────┐
  │ BiomechanicsAgent  │   │ ScoringAgent (ST-GCN)    │   │ RetrievalAgent          │
  │ rubric angles,     │   │ candidate 0–3 + conf     │   │ (Qwen3-VL-Embedding)    │
  │ ROM, symmetry,     │   │ from pose sequence       │   │ k nearest physio clips  │
  │ alignment, timing  │   │                          │   │ + their scores          │
  └─────────┬──────────┘   └───────────┬─────────────┘   └───────────┬────────────┘
            └───────────────────────────┴──────────────────────────┘
                                        ▼
                         ┌──────────────────────────────────────────────┐
                         │  JudgeAgent  (Qwen3-VL-8B)                    │
                         │  rubric + measurements + exemplars + candidate│
                         │  → final 0–3, rationale, compensation tag,    │
                         │    corrective hint, PAIN/CLEARING → defer      │
                         └───────────────┬──────────────────────────────┘
                                         ▼
                         ┌──────────────────────────────────────────────┐
                         │  ReportAgent                                  │
                         │  per-test card, composite 0–21, asymmetry     │
                         │  flags, annotated video, exportable PDF       │
                         └──────────────────────────────────────────────┘
```

**Agent contracts (sketch):**

- `IngestAgent` → `{frames[], fps, duration, n_people}`
- `SegmentationAgent` → `{athlete_track_id, masks[]}`
- `PoseAgent` → `{keypoints_2d[frame][joint]={x,y,conf}}`
- `Body3DAgent` → `{joints_3d[frame][joint]={x,y,z}, mesh_optional}`
- `MovementClassifierAgent` → `{test_name, side: left|right|n/a, confidence}`
- `BiomechanicsAgent` → `{features: {torso_tibia_angle, hip_flexion_deg, knee_valgus_deg, dowel_alignment, L_R_symmetry, ...}}`
- `ScoringAgent` → `{candidate_score: 0–3, confidence}`
- `RetrievalAgent` → `{exemplars: [{clip_id, score, similarity}]}`
- `JudgeAgent` → `{score: 0–3, rationale, compensation_tags[], corrective_hint, needs_human: bool}`
- `ReportAgent` → `{per_test[], composite, asymmetries[], overlay_video, pdf}`

**Quality gating:** if the ST-GCN candidate and the JudgeAgent disagree by ≥1 point, or any agent confidence is low, the report marks the test **"low confidence — physio review recommended."** This keeps the human in the loop and is itself a selling point.

---

## 8. Scoring methodology, per test

The seven tests reduce to measurable quantities. Build a small rubric module — one scoring function per test — that consumes the 3D features and returns a score with the triggering reason. Examples:

- **Deep Squat (3):** femur below horizontal AND torso parallel to tibia AND knees tracking over feet AND dowel over feet. **(2):** same but achieved only with heels elevated. **(1):** criteria unmet even with heels elevated. → all four conditions are angle/alignment checks on the 3D pose.
- **Hurdle Step / In-Line Lunge / Shoulder Mobility / ASLR:** bilateral — score each side, **record the lower** as the test score, and **always emit the asymmetry** even when the score is the same.
- **Trunk Stability Push-Up / Rotary Stability:** trunk rigidity / timing of limb movement — temporal features from the pose sequence; the ST-GCN head is most valuable here.
- **Pain / clearing tests (0):** the system **cannot** detect pain. Any clearing test, or a visible distress/abort, sets `needs_human = true` and the test is **not auto-scored**. Defer to the physio. State this loudly.

Final composite = sum of seven test scores (0–21), plus an asymmetry summary. The number is never shown without its rationale.

---

## 9. Data & fine-tuning plan (tiny-dataset survival guide)

You have "a couple" of physio-scored clips. Treat them as gold, not as a training set.

1. **Deterministic backbone first.** Get the biomechanics rubric working with no training. Validate the measured angles against the physio's scores qualitatively. This alone may be demo-ready.
2. **Pre-train the ST-GCN** on public pose-action / AQA data (action recognition or generic AQA) so it learns temporal movement structure, not FMS labels.
3. **Fine-tune on the physio's clips** with heavy augmentation: temporal crops/speed jitter, mirror (left↔right, doubles your bilateral data), camera-angle perturbation in 3D, joint noise. Few-shot, regularized, early-stopped.
4. **Hold out at least one physio-scored clip** as a sanity check the judge never sees.
5. **RAG instead of more training.** Every labeled clip goes into the embedding index as a scoring anchor. New clips added later improve the system with no retraining — a nice longitudinal story for the physio.
6. **Publish the fine-tuned head** to the Hub with a model card (→ Well-Tuned badge). Include the augmentation recipe and the honest "trained on N clips, treat as assistive" caveat.

**Label schema to collect from the physio** (if you can get a bit more data): `clip_id, athlete_id, test_name, side, score(0–3), pain(bool), compensation_notes, camera_view`. Even 20–30 well-labeled clips meaningfully helps.

---

## 10. Gradio Space & deployment

**UI (targets Off-Brand badge):**
- `gr.Video` upload (or webcam capture) + a test-type selector (auto-detect, with manual override).
- Results panel: the 0–3 score as a large dial/patch, the composite 0–21, an asymmetry strip (L/R bars), and the **rationale text**.
- The annotated overlay video: skeleton + the specific angle that decided the score drawn on the frame where it mattered.
- A rubric drawer that shows the official 3/2/1 criteria for the detected test, with the met/unmet conditions checked off.
- A persistent **"Screening aid — not a diagnosis. Pain or clearing tests require a clinician."** banner.
- Custom CSS / `gr.Server` for a non-default look (scout/trail-map theme would rhyme with the hackathon, and with your design instincts).

**Compute:**
- ZeroGPU (H200 slice) can host the ~18B portfolio; load pose/SAM/3D eagerly, the VLM + embedder via llama.cpp.
- For **Off the Grid**, ensure zero external API calls — everything served on-Space.
- For **Llama Champion**, route the VLM + embedding through llama.cpp (GGUF builds exist for Qwen3-VL-8B-Instruct, Qwen3-VL-Embedding-8B, and Qwen3.6-27B). On a Space, watch the CUDA/llama-cpp build flags — recent hackathon Spaces hit `libcudart` issues; a CPU-only or pinned-CUDA build is the usual fix.
- Persist the embedding index and accumulated labels in Space storage for the longitudinal baseline.

---

## 11. Clinical safety & ethics (bake this in, don't bolt it on)

- **Not a medical device.** Screening aid only. No diagnosis, no injury prediction, no treatment advice beyond generic FMS-style correctives.
- **Pain is out of scope** for automatic scoring — always defer to the physio.
- **Human-in-the-loop by design:** low-confidence and disagreement cases are surfaced, not hidden.
- **Consent & privacy:** athlete videos are biometric data. Get consent; don't log/persist clips beyond what the physio approves; document retention in the writeup.
- **Honesty in the demo:** show a case the system gets right *and* one it flags as uncertain. Judges (and physios) trust calibrated tools more than confident ones.

---

## 12. Build plan — two weekends (June 5–15)

**Weekend 1 — the spine works end to end:**
- Day 1: Space scaffold, `gr.Video` in → skeleton overlay out (YOLO26-Pose). Ingest + Segmentation + Pose agents.
- Day 2: SAM 3D Body integrated; BiomechanicsAgent computing Deep-Squat angles; first deterministic score on a real clip.
- Goal: upload a squat video, get a rationalized 0–3. *This alone is a viable demo.*

**Midweek:** wire the JudgeAgent (Qwen3-VL via llama.cpp), MovementClassifier, and the rubric module for all 7 tests. Attend the AMA — confirm the param-sum interpretation.

**Weekend 2 — make it sing:**
- ST-GCN pre-train + few-shot fine-tune on physio clips; publish to Hub.
- RetrievalAgent + embedding index over labeled clips.
- Custom UI polish, asymmetry view, PDF export, safety banners.
- Record the demo video (physio uses it on a real player), write the social post, publish the agent trace and the blog post.

---

## 13. Risks & open questions

- **Param-sum interpretation** — biggest unknown. The ~18B config is safe under either reading; confirm anyway.
- **SAM 3D Body on a Space** — verify weights, license, and that it runs within ZeroGPU limits; have a 2D-only fallback (angles from 2D + camera-angle caveats) if it's too heavy.
- **Single-camera angle limits** even with 3D — note it; recommend a consistent capture protocol (fixed camera position) for the physio, which also improves the longitudinal baseline.
- **Tiny dataset** — the deterministic rubric must stand on its own so the demo doesn't hinge on the learned head generalizing from a few clips.
- **llama.cpp + vision build** on Spaces — budget time for the CUDA build dance; CPU fallback for the embedder is fine.
- **Movement misclassification** — if the wrong test is detected, scoring is meaningless; keep the manual override prominent.

---

## 14. Quick reference — the stack

| Layer | Choice | Badge it helps |
|---|---|---|
| 2D pose | YOLO26-Pose | — |
| Segmentation/track | SAM 3.1 | — |
| 3D biomechanics | SAM 3D Body | — |
| Learned scoring | ST-GCN (fine-tuned, published) | Well-Tuned |
| Judge/explainer | Qwen3-VL-8B-Instruct (llama.cpp) | Llama Champion |
| Retrieval | Qwen3-VL-Embedding-8B (llama.cpp) | Llama Champion |
| Serving | On-Space, no cloud APIs | Off the Grid |
| Frontend | Custom Gradio (scout theme) | Off-Brand |
| Trace | Published agent run on Hub | Sharing is Caring |
| Writeup | Blog post w/ honesty section | Field Notes |

*Total ≈ 18B params. Honest, explainable, human-in-the-loop, runs on a laptop.*