CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project overview

FormScout is a Gradio app (Hugging Face Space) that scores Functional Movement Screen (FMS) videos 0–3 per test with a written rationale and an annotated overlay. It is a screening aid — not a diagnosis, not an injury predictor. Built for the Build Small Hackathon (Backyard AI track). Full product spec is in docs/FormScout-FMS-Spec.md; the engineering contract is in docs/plans/FormScout-Build-Prompt.md.

Current status: Phase 2 complete. All 7 FMS test rubric scorers, JudgeAgent, MovementClassifierAgent, ReportAgent, PoseVisualizer (overlay video), and a user-selectable pose-model registry are implemented and tested (86/87 passing). Phase 3 is next (ST-GCN fine-tune + RAG retrieval).

Common commands

# Run the Gradio app locally
python3 app.py

# Headless pipeline test (no Gradio)
python3 -m formscout.run sample.mp4

# Run all tests
pytest tests/

# Run a single test file or test
pytest tests/test_phase2.py
pytest tests/test_biomechanics.py::TestBiomechanicsAgent::test_deep_squat_score

# Lint / format
ruff check . && ruff format .

# Start the local VLM judge server (llama.cpp, port 8080)
./scripts/serve_judge.sh

# Push source tree to the HF model repo + Space (PRs; message from last commit)
./scripts/hf_upload.sh

# Run Svelte component tests (when frontend work is added)
npx vitest run

Architecture

The pipeline is a sequence of typed specialist agents. Each agent accepts and returns a frozen dataclass from formscout/types.py. The Director in formscout/pipeline.py orchestrates them as a deterministic state machine (not an LLM).

Agent pipeline

IngestAgent → Pose2DAgent → [Body3DAgent — optional]
→ MovementClassifierAgent → BiomechanicsAgent
→ rubric/score_test() → JudgeAgent → ReportAgent

The Director (pipeline.py) owns the flow. app.py creates one Director() instance and calls director.run(video_path, test_name, side, model_key) per submission. The Gradio UI passes test_name directly (from dropdown), bypassing the classifier; model_key selects the pose backend from config.POSE_MODELS.

PoseVisualizer (formscout/agents/visualizer.py) renders the annotated overlay video (skeleton, trails, velocity arrows) from IngestResult + Pose2DResult. It is called from app.py after the pipeline run — it is a UI-layer component, not a Director stage. It returns None on failure, never raises.

The tiering rule (most important invariant)

The 2D path is the default and must stand alone as a complete, functional pipeline. Body3DAgent is only activated when config.ENABLE_3D == True AND the checkpoint loads successfully. If 3D is off or fails, Body3DResult(used=False, ...) is returned — this is a normal success path, not an error. BiomechFeatures.view is "2d" or "3d" so the JudgeAgent can caveat its rationale appropriately. Never put Body3DAgent on the critical path.

Feature flags in config.py and their current state

Flag	Default	Meaning
ENABLE_JUDGE	True	Judge/Classifier call Qwen3-VL via llama-server; graceful rubric fallback when the server is down
ENABLE_3D	False	When False, Body3DAgent returns used=False immediately
ENABLE_STGCN	False	Phase 3 — ST-GCN learned scoring head
ENABLE_RAG	False	Phase 3 — RetrievalAgent exemplar lookup

All model IDs, thresholds, k-values, and feature flags live in config.py — never scattered literals.

Judge backend selection (local vs Space)

config.resolve_judge_backend() picks the VLM backend via FORMSCOUT_JUDGE_BACKEND (llama_cpp | transformers | auto). auto (default) uses llama-server locally and the in-process transformers backend on a Space (detected via SPACE_ID). JudgeAgent gets its client from serving.get_vlm_client().

• llama_cpp — LlamaCppClient → llama-server at 127.0.0.1:8080 (start with scripts/serve_judge.sh). The local path; works perfectly.
• transformers — TransformersVLMClient loads Qwen3-VL-8B via transformers, GPU-wrapped with spaces.GPU (ZeroGPU). Lazy model load, cached per process. On any load/inference failure it returns {"fallback": True} and the Judge falls back to the rubric. Needs validation on real ZeroGPU hardware — not exercised in CPU tests.

Fallback chain (important for local dev and Spaces)

1. ENABLE_JUDGE=False → JudgeAgent returns rubric score wrapped as JudgeResult (no VLM needed)
2. ENABLE_JUDGE=True + selected backend unavailable / transformers load fails → same rubric fallback, logs a warning
3. ENABLE_JUDGE=True + backend available → calls Qwen3-VL-8B-Instruct (llama-server locally, transformers/ZeroGPU on a Space)

Start the VLM server with scripts/serve_judge.sh (downloads live in checkpoints/qwen3-vl/, gitignored). To use a fine-tuned GGUF, set FORMSCOUT_JUDGE_GGUF (and FORMSCOUT_JUDGE_MMPROJ if it ships its own projector) — no code change needed. Multimodal requests go through the OpenAI-compatible /v1/chat/completions endpoint (the legacy /completion + image_data path does not work with modern llama-server).

This means the app is fully functional without any GPU or llama.cpp — rubric scoring is pure Python.

Rubric scorers

Each FMS test has a pure-function scorer in formscout/rubric/:

score_deep_squat / score_hurdle_step / score_inline_lunge /
score_shoulder_mobility / score_active_slr /
score_trunk_stability_pushup / score_rotary_stability

All accept BiomechFeatures and return ScoreResult. Dispatch via rubric.score_test(features). Rubric functions must remain pure — no model calls, no I/O.

Bilateral tests

hurdle_step, inline_lunge, shoulder_mobility, active_slr are bilateral. ReportAgent groups them by test name, takes the lower score, and always emits the asymmetry delta even when scores are equal. composite is None when any test is unscored.

Types contract

Every agent I/O is a frozen dataclass from formscout/types.py. Key types:

• IngestResult — decoded frames (np.ndarray list), fps, duration, dimensions
• Pose2DResult — per-frame keypoints as dict[int, {x, y, conf}] (COCO 17 joints)
• Body3DResult — optional 3D joints, always has used: bool
• MovementResult — test_name (validated enum), side ("left"|"right"|"na")
• BiomechFeatures — angles: dict, alignments: dict, view: "2d"|"3d", symmetry_delta
• ScoreResult — score: int (0–3), rationale, needs_human
• JudgeResult — same as ScoreResult + compensation_tags, corrective_hint; score=None when needs_human=True
• PipelineState — mutable accumulator threaded through the Director

MovementResult and JudgeResult validate their fields in __post_init__ — passing invalid values raises immediately.

Pose model selection and checkpoints

config.POSE_MODELS is a registry of pose backends: MediaPipe (CPU-friendly), five YOLO26 sizes (n/s/m/l/x), and Sapiens2 variants (Phase 3, need the custom sapiens repo installed). config.DEFAULT_POSE_MODEL is YOLO26n. The Gradio UI exposes a dropdown built from config.available_pose_models() (filters to checkpoints actually present) and passes the chosen model_key through Director.run to Pose2DAgent. config.YOLO_POSE_MODEL is a backward-compat alias only.

Checkpoints are not committed (checkpoints/ is gitignored). formscout/startup.py:ensure_checkpoints() downloads missing YOLO26/MediaPipe files from the silas-therapy/formscout-checkpoints HF repo once at app startup. Models load once per process and are cached — never inside the inference hot path.

llama.cpp serving

formscout/serving/llama_cpp.py provides LlamaCppClient (VLM, port 8080) and EmbeddingClient (embeddings, port 8081). Both check /health before use and return safe error dicts when unavailable. Only active when the corresponding ENABLE_* flag is True.

Deploying to Hugging Face

The repo deploys to both silas-therapy/small-functional-movement-screening (model repo) and the Space of the same name (README frontmatter is the Space config). Use ./scripts/hf_upload.sh — never raw hf upload .: the hf CLI does not read .hfignore, so a raw upload hashes the entire .venv (~44k files) and pushes torch binaries. The script parses .hfignore into --exclude globs, preflights the file count, creates PRs on both repos, and auto-switches to hf upload-large-folder (resumable, but no PR / no commit message) above 500 files.

Key constraints and invariants

• No cloud model APIs. All inference runs on-Space (ZeroGPU). No OpenAI/Anthropic/Gemini calls.
• Pain is never auto-scored. Any clearing test or visible distress sets needs_human=True — enforced in rubric functions and JudgeAgent. JudgeResult.score must be None when needs_human=True.
• Quality gates (Director, never silently skip):
  • Any agent confidence < config.MIN_CONFIDENCE (0.6) → warn or stop
  • |rubric.score - judge.score| >= 1 → flag disagreement
  • MovementResult.test_name == "unknown" → stop pipeline, surface manual override
  • JudgeAgent.needs_human == True → no numeric score emitted
• Composite is null when any test is unscored. Never show a partial 0–21 as complete.
• Pipeline runs headless. No Gradio imports in any agent file.
• Safety banner ("Screening aid — not a diagnosis…") must always be visible in the UI — appears at top and bottom of app.py.

Engineering standards

• Every agent: one public entrypoint, typed dataclass I/O from types.py, confidence: float and notes: str on every result.
• Models load once at module/instance init — never inside the inference hot path.
• Every agent module docstring states: purpose, inputs, outputs, failure behavior, model param count, license, and gated status.
• tracing.py records structured per-agent I/O for any run; one full run gets exported to the Hub.
• Every agent ships with a pytest in tests/ that runs without model downloads and asserts the typed contract.

Model stack (~17.6B total — stay under 32B)

Component	Model	Params	Status
2D pose (primary)	YOLO26-Pose n/s/m/l/x (default: n)	0.0007–0.058B	Ready (auto-downloaded at startup)
2D pose (CPU alt)	MediaPipe Pose Landmarker (full)	~0.004B	Ready (auto-downloaded at startup)
2D pose (HQ alt)	facebook/sapiens2-pose-0.4b/0.8b/1b/5b	0.4–5B	Phase 3 — needs custom sapiens repo
Segmentation	SAM 3.1 base	~0.85B	Access accepted
3D biomechanics	facebook/sam-3d-body-dinov3	~0.84B	Access ACCEPTED Jun 4 2026
Learned scoring	ST-GCN (pyskl)	~0.03B	Phase 3
Judge + Classifier	Qwen3-VL-8B-Instruct (llama.cpp)	8B	Online — scripts/serve_judge.sh, ENABLE_JUDGE=True
Retrieval	Qwen3-VL-Embedding-8B (llama.cpp)	8B	Phase 3

Track the running sum in MODEL_BUDGET.md. The two Qwen3-VL-8B models share a backbone.

Gradio + Svelte UI guidance

The UI uses Gradio gr.Blocks with custom CSS/theme (formscout/ui/theme.py). Custom Svelte components for score dial, asymmetry bars, rubric drawer are planned for Phase 4. Use gradio-svelte-expert agent for Svelte component work.

• ZeroGPU: app.py's process_video (the Start Analysis handler) is decorated with @spaces.GPU (via the gpu_task shim, no-op off-Space) so one GPU window wraps the whole pipeline — pose, optional 3D, and the judge. ZeroGPU aborts startup with "No @spaces.GPU function detected" unless a decorated function exists at import time, so the decorator must stay at module level on a top-level function, not buried behind a lazy import. Window length is config.ZEROGPU_DURATION (default 120s, FORMSCOUT_ZEROGPU_DURATION).
• Verify Gradio APIs against current docs before use — pin exact versions in requirements.txt.

Build phases

1. Phase 0 — Recon: ✅ Complete. See RECON.md.
2. Phase 1 — Spine: ✅ Complete. Deep Squat end-to-end.
3. Phase 2 — All 7 tests: ✅ Complete. Classifier, Judge, Report agents; all rubric scorers; Gradio UI.
4. Phase 3 — Learned scoring + retrieval: ST-GCN fine-tune on physio clips, publish to Hub. RetrievalAgent with embedding index.
5. Phase 4 — Polish + ship: Custom Svelte UI components, agent trace to Hub, blog post. (Overlay video done via PoseVisualizer; full 7-test session + PDF export done via formscout/session.py + PdfReportAgent.)

Known issues

• tests/test_biomechanics.py::TestBiomechanicsAgent::test_unimplemented_test_returns_low_confidence fails: expects "not yet implemented" in result.notes but biomechanics returns empty string. Minor — low priority.

Badge checklist (definition of done)

• Space runs green; upload → scorecard works on real clips
• Param sum verified ≤ 32B in MODEL_BUDGET.md
• 🔌 Off the Grid — no cloud model APIs anywhere in the pipeline
• 🎯 Well-Tuned — fine-tuned ST-GCN head published to Hub with honest model card
• 🎨 Off-Brand — custom, non-default Gradio UI (scout/trail theme)
• 🦙 Llama Champion — VLM + embedder served via llama.cpp (GGUF)
• 📡 Sharing is Caring — one full agent trace (all I/O) published to Hub
• 📓 Field Notes — blog post written, honesty section (FMS limitations) front-and-center
• Demo video + social post recorded
• Safety banner present; pain/clearing never auto-scored; low-confidence flagged