Upload 5 files

README.md

@@ -1,62 +1,83 @@
----
-title: StrokeSense
-emoji: 🎾
-colorFrom: green
-colorTo: gray
-sdk: gradio
-app_file: app.py
-pinned: false
-tags:
-  - best-minicpm-build
-  - llama-cpp
-  - backyard-ai
----
-
-# StrokeSense
-
-A tennis stroke analyser that detects shots from video and scores mechanics using a local vision-language model — no cloud API required. Designed to analyse close-up shots from a drill, it identifies each groundstroke (forehand or backhand), then generates scores and notes using vision capabilities.
-
-Built for the **Build Small Hackathon** · Track: **Backyard AI**
-
-
-![Demo](images/ai_tennis_coach_picture.png)
+# Tennis Stroke Analyser
 
+A two-stage computer vision pipeline that detects tennis shots from video and scores stroke mechanics using a local vision-language model.
 
 ## How it works
 
-1. **Shot detection** — MoveNet (TFLite) extracts pose keypoints per frame. A sliding-window RNN classifies each window as forehand, backhand, serve, or neutral.
-2. **Clip extraction** — A short clip is cut around each detected shot.
-3. **Stroke analysis** — Clip frames are sent to a local **MiniCPM-V-4.6** server (running via **llama.cpp**). The model scores four mechanics dimensions (preparation, contact point, swing/follow-through, and balance/stance) and returns structured JSON.
-4. **Coach recommendation** — Chat with a coach agent powered by **MiniCPM-V-4.6** to get recommendations and discuss your stroke analysis further.
+1. **Shot detection** — MoveNet (TFLite) extracts 17-keypoint pose estimates per frame. A 30-frame sliding window feeds an RNN that classifies each window as forehand, backhand, serve, or neutral (threshold 0.98). Detected shots are returned as `[{"FrameID": int, "Shot": str}]`.
+2. **Clip extraction** — A short video clip is cut around each detected frame ID (configurable padding before/after).
+3. **Stroke analysis** — Clip frames are sampled and sent as images to a local MiniCPM-V-4.6 server. The model scores four mechanics dimensions (preparation, contact point, swing/follow-through, balance/stance) and returns structured JSON via the OpenAI-compatible API.
 
 ## Setup
 
 Requires [uv](https://github.com/astral-sh/uv) and Python 3.12+.
 
 ```bash
-uv sync
+uv sync          # runtime dependencies
+uv sync --dev    # include marimo notebook runner
 ```
 
-**Model files needed** (not included):
+### Required model files (not included)
 
-| File | Purpose |
+| File | Used by |
 |---|---|
 | `models/movenet.tflite` | Shot detection |
-| MiniCPM-V-4.6-GGUF | VLM stroke analysis |
+| `models/pose_landmarker_lite.task` | Pose annotation video |
+| MiniCPM-V-4.6-GGUF weights | VLM stroke analysis |
 
-Download MiniCPM-V-4.6-GGUF from [Hugging Face](https://huggingface.co/ggml-org/MiniCPM-V-4.6-GGUF) and set the path in `start_server.sh`.
+Download MiniCPM-V-4.6-GGUF from [Hugging Face](https://huggingface.co/ggml-org/MiniCPM-V-4.6-GGUF) and update the path in `start_server.sh`.
 
 ## Usage
 
+**Detect shots programmatically:**
+```python
+from ai_tennis_coach import classify_shots
+
+results = classify_shots("data/match.mp4", "tennis_rnn.h5")
+# [{"FrameID": 131, "Shot": "forehand"}, {"FrameID": 350, "Shot": "forehand"}, ...]
+```
+
+**Real-time shot detection with overlay:**
+```bash
+uv run python src/ai_tennis_coach/track_and_classify.py <video> <model.h5> [--left-handed]
+```
+
+**Annotate video with MediaPipe pose landmarks:**
 ```bash
-# Start the local VLM server (llama.cpp)
+uv run python pose_estimation_video.py -v data/match.mp4 -o data/annotated.mp4
+```
+
+**Stroke quality analysis (marimo notebook):**
+```bash
+# 1. Start the local VLM server
 ./start_server.sh
+
+# 2. Open the notebook
+uv run marimo edit stroke_analysis.py
+```
+
+## Package structure
+
+```
+src/ai_tennis_coach/
+├── extract_pose.py       # HumanPoseExtractor + RoI tracker (MoveNet)
+├── track_and_classify.py # ShotCounter, real-time rendering script
+├── analysis.py           # classify_shots(), StrokeAnalysis Pydantic model
+├── config.py             # Config + Prompts dataclasses, module-level singletons
+├── utils.py              # encode_video(), image_to_base64()
+└── prompt.yaml           # VLM scoring prompts
 ```
 
+## Configuration
+
+All tunable settings live in `src/ai_tennis_coach/config.py`:
+
 ```python
-# Programmatic shot detection
-from ai_tennis_coach import classify_shots
+from ai_tennis_coach.config import config, prompts
 
-results = classify_shots("match.mp4", "tennis_rnn.h5")
-# [{"FrameID": 131, "Shot": "forehand"}, ...]
-```
+config.api_base_url    # "http://localhost:8080/v1"
+config.model_name      # "MiniCPM-V-4.6-Instruct"
+config.clips_dir       # "clips"
+config.target_fps      # 3
+prompts.forehand       # full forehand scoring rubric
+```

pyproject.toml

@@ -0,0 +1,28 @@
+[project]
+name = "build-small-tennis-stroke"
+version = "0.1.0"
+description = "Add your description here"
+readme = "README.md"
+requires-python = ">=3.12"
+dependencies = [
+    "aiohttp>=3.14.1",
+    "gradio>=6.18.0",
+    "imageio[ffmpeg]>=2.37.3",
+    "instructor>=1.15.1",
+    "mediapipe>=0.10.35",
+    "pandas>=3.0.3",
+    "python-dotenv>=1.0.0",
+    "tensorflow>=2.21.0",
+]
+
+[dependency-groups]
+dev = [
+    "marimo>=0.23.6",
+]
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[tool.hatch.build.targets.wheel]
+packages=["src/ai_tennis_coach"]

requirements.txt

@@ -0,0 +1,332 @@
+agent-framework==1.0.0b251120
+agent-framework-anthropic==1.0.0b260130
+agent-framework-core==1.0.0b251001
+aiofiles==24.1.0
+aiohappyeyeballs==2.6.1
+aiohttp==3.11.18
+aiosignal==1.3.2
+amqp==5.3.1
+aniso8601==10.0.0
+annotated-types==0.7.0
+anthropic==0.79.0
+anyio==4.12.1
+-e file:///Users/ilhamfadhil/Downloads/cli_project
+appnope==0.1.4
+archspec @ file:///croot/archspec_1709217642129/work
+argcomplete==3.6.0
+argon2-cffi==23.1.0
+argon2-cffi-bindings==21.2.0
+arrow==1.3.0
+asgiref==3.11.1
+asttokens==3.0.0
+async-lru==2.0.5
+async-timeout==5.0.1
+attrs==25.4.0
+azure-core==1.38.0
+azure-core-tracing-opentelemetry==1.0.0b12
+azure-identity==1.25.1
+azure-monitor-opentelemetry==1.8.2
+azure-monitor-opentelemetry-exporter==1.0.0b45
+babel==2.17.0
+beautifulsoup4==4.13.4
+billiard==4.2.2
+bleach==6.2.0
+blinker==1.9.0
+boltons @ file:///private/var/folders/nz/j6p8yfhx1mv_0grj5xl4650h0000gp/T/abs_f63n9uulmp/croot/boltons_1677628710094/work
+bottleneck @ file:///private/var/folders/nz/j6p8yfhx1mv_0grj5xl4650h0000gp/T/abs_55txi4fy1u/croot/bottleneck_1731058642212/work
+brotli @ file:///private/var/folders/k1/30mswbxs7r1g6zwn8y4fyt500000gp/T/abs_27zk0eqdh0/croot/brotli-split_1714483157007/work
+celery==5.5.3
+certifi==2026.1.4
+cffi==2.0.0
+charset-normalizer==3.4.4
+click==8.3.1
+click-didyoumean==0.3.1
+click-plugins==1.1.1.2
+click-repl==0.3.0
+colorama==0.4.6
+comm==0.2.2
+conda @ file:///private/var/folders/nz/j6p8yfhx1mv_0grj5xl4650h0000gp/T/abs_f1fq2yjc1q/croot/conda_1749171256613/work/conda-src
+conda-content-trust @ file:///private/var/folders/nz/j6p8yfhx1mv_0grj5xl4650h0000gp/T/abs_11146a2b-93c2-444c-a378-ad4fac363e991s0r1hnp/croots/recipe/conda-content-trust_1658126383571/work
+conda-libmamba-solver @ file:///croot/conda-libmamba-solver_1745607008911/work/src
+conda-package-handling @ file:///private/var/folders/nz/j6p8yfhx1mv_0grj5xl4650h0000gp/T/abs_ef9phnqphe/croot/conda-package-handling_1718138279942/work
+conda-package-streaming @ file:///private/var/folders/k1/30mswbxs7r1g6zwn8y4fyt500000gp/T/abs_6dgq200203/croot/conda-package-streaming_1718136087190/work
+contourpy @ file:///private/var/folders/k1/30mswbxs7r1g6zwn8y4fyt500000gp/T/abs_2cvjf0v4ux/croot/contourpy_1732540055997/work
+cryptography==46.0.4
+cycler @ file:///tmp/build/80754af9/cycler_1637851556182/work
+databricks-cli==0.17.7
+datasets==4.0.0
+debugpy==1.8.13
+decorator @ file:///opt/conda/conda-bld/decorator_1643638310831/work
+defusedxml==0.7.1
+deprecated==1.2.18
+deprecation==2.1.0
+dill==0.3.8
+distlib @ file:///private/var/folders/k1/30mswbxs7r1g6zwn8y4fyt500000gp/T/abs_601vai9os4/croot/distlib_1714716995800/work
+distro @ file:///private/var/folders/nz/j6p8yfhx1mv_0grj5xl4650h0000gp/T/abs_ddkyz0575y/croot/distro_1714488254309/work
+docstring-parser==0.17.0
+docutils==0.22.3
+exceptiongroup==1.3.1
+executing==2.2.0
+fastapi==0.115.11
+fastjsonschema==2.21.1
+ffmpy==0.6.0
+filelock @ file:///private/var/folders/nz/j6p8yfhx1mv_0grj5xl4650h0000gp/T/abs_d3quwmvouf/croot/filelock_1700591194006/work
+filetype==1.2.0
+fixedint==0.1.6
+flask==3.1.0
+flask-cors==5.0.1
+flask-restx==1.3.0
+flower==2.0.1
+fonttools @ file:///private/var/folders/nz/j6p8yfhx1mv_0grj5xl4650h0000gp/T/abs_ce1jt_55vl/croot/fonttools_1737039388732/work
+fqdn==1.5.1
+frozendict @ file:///private/var/folders/k1/30mswbxs7r1g6zwn8y4fyt500000gp/T/abs_8b0cest_id/croot/frozendict_1713194839836/work
+frozenlist==1.6.0
+fsspec==2025.3.0
+geoarrow-c==0.1.2
+geoarrow-pandas==0.1.1
+geoarrow-pyarrow==0.1.2
+geopandas==1.0.1
+gitdb==4.0.12
+gitpython==3.1.44
+googleapis-common-protos==1.69.2
+gradio==5.35.0
+gradio-client==1.10.4
+groovy==0.1.2
+grpcio==1.78.0
+h11==0.16.0
+hf-xet==1.1.5
+httpcore==1.0.9
+httpx==0.28.1
+httpx-sse==0.4.3
+huggingface-hub==0.34.3
+humanize==4.14.0
+idna==3.11
+importlib-metadata==8.7.1
+importlib-resources==6.5.2
+inquirerpy==0.3.4
+ipykernel==6.29.5
+ipython @ file:///private/var/folders/k1/30mswbxs7r1g6zwn8y4fyt500000gp/T/abs_31k34m3e25/croot/ipython_1726064238879/work
+ipywidgets==8.1.6
+isodate==0.7.2
+isoduration==20.11.0
+itsdangerous==2.2.0
+jaraco-classes==3.4.0
+jedi==0.19.2
+jinja2==3.1.6
+jiter==0.13.0
+json5==0.12.0
+jsonpatch @ file:///private/var/folders/k1/30mswbxs7r1g6zwn8y4fyt500000gp/T/abs_3ajyoz8zoj/croot/jsonpatch_1714483362270/work
+jsonpointer==2.1
+jsonschema==4.26.0
+jsonschema-specifications==2025.9.1
+jupyter==1.1.1
+jupyter-client==8.6.3
+jupyter-console==6.6.3
+jupyter-core==5.7.2
+jupyter-events==0.12.0
+jupyter-lsp==2.2.5
+jupyter-packaging==0.12.3
+jupyter-server==2.15.0
+jupyter-server-terminals==0.5.3
+jupyterlab==4.4.1
+jupyterlab-pygments==0.3.0
+jupyterlab-server==2.27.3
+jupyterlab-widgets==3.0.14
+kaleido==0.2.1
+keplergl==0.3.7
+keyring==24.3.1
+kiwisolver @ file:///private/var/folders/nz/j6p8yfhx1mv_0grj5xl4650h0000gp/T/abs_cc2l_z_0ri/croot/kiwisolver_1737039586949/work
+kombu==5.5.4
+libmambapy @ file:///private/var/folders/nz/j6p8yfhx1mv_0grj5xl4650h0000gp/T/abs_8fxbc14zt9/croot/mamba-split_1734469521691/work/libmambapy
+litellm==1.68.1
+markdown==3.10
+markdown-it-py==3.0.0
+markupsafe==3.0.2
+marshmallow==3.26.1
+matplotlib==3.10.0
+matplotlib-inline @ file:///private/var/folders/nz/j6p8yfhx1mv_0grj5xl4650h0000gp/T/abs_f6fdc0hldi/croots/recipe/matplotlib-inline_1662014472341/work
+mcp==1.26.0
+mdurl==0.1.2
+menuinst @ file:///private/var/folders/k1/30mswbxs7r1g6zwn8y4fyt500000gp/T/abs_ad321a45td/croot/menuinst_1723567614652/work
+mistune==3.1.3
+mlx==0.27.1
+mlx-embeddings==0.0.3
+mlx-lm==0.26.2
+mlx-metal==0.27.1
+mlx-vlm==0.3.2
+more-itertools==10.6.0
+mpmath==1.3.0
+msal==1.34.0
+msal-extensions==1.3.1
+msgspec-m==0.19.2
+msrest==0.7.1
+multidict==6.4.3
+multiprocess==0.70.16
+narwhals==2.11.0
+nbclient==0.10.2
+nbconvert==7.16.6
+nbformat==5.10.4
+nest-asyncio==1.6.0
+networkx==3.1
+notebook==7.4.1
+notebook-shim==0.2.4
+numexpr @ file:///private/var/folders/nz/j6p8yfhx1mv_0grj5xl4650h0000gp/T/abs_b3kvvt6tc6/croot/numexpr_1730215947700/work
+numpy==1.26.4
+oauthlib==3.2.2
+omz-tools @ file:///Users/ilhamfadhil/Documents/Work/beesar/nai-face-recognition/open_model_zoo/tools/model_tools
+openai==2.18.0
+opencv-python==4.12.0.88
+opencv-python-headless==4.12.0.88
+opentelemetry-api==1.39.1
+opentelemetry-exporter-otlp-proto-common==1.39.1
+opentelemetry-exporter-otlp-proto-grpc==1.39.1
+opentelemetry-exporter-otlp-proto-http==1.31.0
+opentelemetry-instrumentation==0.60b1
+opentelemetry-instrumentation-asgi==0.60b1
+opentelemetry-instrumentation-dbapi==0.60b1
+opentelemetry-instrumentation-django==0.60b1
+opentelemetry-instrumentation-fastapi==0.60b1
+opentelemetry-instrumentation-flask==0.60b1
+opentelemetry-instrumentation-psycopg2==0.60b1
+opentelemetry-instrumentation-requests==0.60b1
+opentelemetry-instrumentation-urllib==0.60b1
+opentelemetry-instrumentation-urllib3==0.60b1
+opentelemetry-instrumentation-wsgi==0.60b1
+opentelemetry-proto==1.39.1
+opentelemetry-resource-detector-azure==0.1.5
+opentelemetry-sdk==1.39.1
+opentelemetry-semantic-conventions==0.60b1
+opentelemetry-semantic-conventions-ai==0.4.13
+opentelemetry-util-http==0.60b1
+openvino==2024.6.0
+openvino-dev==2024.6.0
+openvino-telemetry==2025.2.0
+orjson==3.10.18
+overrides==7.7.0
+packaging==26.0
+pandas @ file:///private/var/folders/nz/j6p8yfhx1mv_0grj5xl4650h0000gp/T/abs_4aifrweohv/croot/pandas_1732735109535/work/dist/pandas-2.2.3-cp310-cp310-macosx_11_0_arm64.whl
+pandocfilters==1.5.1
+parso==0.8.4
+pdf2image==1.17.0
+pexpect @ file:///tmp/build/80754af9/pexpect_1605563209008/work
+pfzy==0.3.4
+pickleshare==0.7.5
+pika==1.3.2
+pillow==11.3.0
+pip==23.0.1
+platformdirs @ file:///private/var/folders/k1/30mswbxs7r1g6zwn8y4fyt500000gp/T/abs_a8u4fy8k9o/croot/platformdirs_1692205661656/work
+plotly==6.0.1
+pluggy @ file:///private/var/folders/nz/j6p8yfhx1mv_0grj5xl4650h0000gp/T/abs_croot-w6jyveby/pluggy_1648109277227/work
+polars==1.34.0
+polars-runtime-32==1.34.0
+prometheus-client==0.21.1
+prompt-toolkit==3.0.52
+promptflow==1.17.2
+promptflow-core==1.17.2
+promptflow-devkit==1.17.2
+promptflow-tracing==1.17.2
+propcache==0.3.1
+protobuf==5.29.3
+psutil==7.0.0
+psycopg==3.2.11
+psycopg-binary==3.2.11
+ptyprocess @ file:///tmp/build/80754af9/ptyprocess_1609355006118/work/dist/ptyprocess-0.7.0-py2.py3-none-any.whl
+pure-eval @ file:///opt/conda/conda-bld/pure_eval_1646925070566/work
+pyarrow==20.0.0
+pyarrow-hotfix==0.7
+pycosat @ file:///private/var/folders/nz/j6p8yfhx1mv_0grj5xl4650h0000gp/T/abs_19qelmdbl6/croot/pycosat_1714510743067/work
+pycparser==3.0
+pydantic==2.12.5
+pydantic-core==2.41.5
+pydantic-settings==2.12.0
+pydash==7.0.7
+pydub==0.25.1
+pygments==2.19.1
+pyjwt==2.11.0
+pymdown-extensions==10.16.1
+pyogrio==0.10.0
+pyparsing @ file:///private/var/folders/nz/j6p8yfhx1mv_0grj5xl4650h0000gp/T/abs_65qfw6vkxg/croot/pyparsing_1731445528142/work
+pyproj==3.7.1
+pysocks @ file:///Users/ktietz/ci_310/pysocks_1643961536721/work
+python-dateutil @ file:///private/var/folders/k1/30mswbxs7r1g6zwn8y4fyt500000gp/T/abs_66ud1l42_h/croot/python-dateutil_1716495741162/work
+python-dotenv==1.2.1
+python-json-logger==3.3.0
+python-multipart==0.0.22
+pytz==2025.2
+pyyaml==6.0.2
+pyzmq==26.3.0
+referencing==0.37.0
+regex==2024.11.6
+requests==2.32.5
+requests-oauthlib==2.0.0
+rfc3339-validator==0.1.4
+rfc3986-validator==0.1.1
+rich==14.0.0
+rpds-py==0.30.0
+ruamel-yaml @ file:///private/var/folders/nz/j6p8yfhx1mv_0grj5xl4650h0000gp/T/abs_35yvtl3p84/croot/ruamel.yaml_1727980165481/work
+ruamel-yaml-clib @ file:///private/var/folders/nz/j6p8yfhx1mv_0grj5xl4650h0000gp/T/abs_70l_orv46q/croot/ruamel.yaml.clib_1727769819918/work
+ruff==0.12.1
+safehttpx==0.1.6
+safetensors==0.5.3
+scipy==1.15.3
+seaborn @ file:///private/var/folders/nz/j6p8yfhx1mv_0grj5xl4650h0000gp/T/abs_74n8m1zz4r/croot/seaborn_1749110344248/work
+semantic-version==2.10.0
+send2trash==1.8.3
+sentencepiece==0.2.0
+setuptools==65.6.3
+shapely==2.1.0
+shellingham==1.5.4
+six @ file:///tmp/build/80754af9/six_1644875935023/work
+smmap==5.0.2
+smolagents==1.15.0
+sniffio==1.3.1
+soundfile==0.13.1
+soupsieve==2.7
+sqlalchemy==2.0.39
+sse-starlette==3.2.0
+stack-data==0.6.3
+starlette==0.52.1
+strictyaml==1.7.3
+sympy==1.14.0
+tabulate==0.9.0
+terminado==0.18.1
+tiktoken==0.9.0
+tinycss2==1.4.0
+tokenizers==0.21.1
+tomli==2.2.1
+tomlkit==0.13.2
+torch==2.9.0
+torchvision==0.24.0
+tornado @ file:///private/var/folders/k1/30mswbxs7r1g6zwn8y4fyt500000gp/T/abs_a4w03z48br/croot/tornado_1718740114858/work
+tqdm==4.67.3
+traitlets==5.14.3
+traittypes==0.2.1
+transformers==4.54.1
+truststore @ file:///private/var/folders/k1/30mswbxs7r1g6zwn8y4fyt500000gp/T/abs_42mm7e6j06/croot/truststore_1695244298716/work
+typer==0.16.0
+types-python-dateutil==2.9.0.20241206
+typing-extensions==4.15.0
+typing-inspection==0.4.2
+tzdata==2025.2
+ultralytics==8.3.221
+ultralytics-thop==2.0.17
+unicodedata2 @ file:///private/var/folders/k1/30mswbxs7r1g6zwn8y4fyt500000gp/T/abs_b99hn8t9lq/croot/unicodedata2_1736541774279/work
+uri-template==1.3.0
+urllib3==2.6.3
+uvicorn==0.40.0
+vine==5.1.0
+virtualenv @ file:///private/var/folders/k1/30mswbxs7r1g6zwn8y4fyt500000gp/T/abs_5cp3cbuzh_/croot/virtualenv_1714719505426/work
+waitress==3.0.2
+wcwidth==0.2.13
+webcolors==24.11.1
+webencodings==0.5.1
+websocket-client==1.8.0
+websockets==16.0
+werkzeug==3.1.3
+wheel==0.38.4
+widgetsnbextension==4.0.14
+wrapt==1.17.2
+xxhash==3.5.0
+yarl==1.20.0
+zipp==3.23.0
+zstandard @ file:///private/var/folders/k1/30mswbxs7r1g6zwn8y4fyt500000gp/T/abs_054juzz3it/croot/zstandard_1714677666952/work

start_server.sh

@@ -0,0 +1,10 @@
+llama-server \
+  -m ~/.cache/huggingface/hub/models--ggml-org--MiniCPM-V-4.6-GGUF/snapshots/1d73353fab60fe6040ab73eb1e457e2aab210100/MiniCPM-V-4.6-Q4_K_M.gguf \
+  --mmproj ~/.cache/huggingface/hub/models--ggml-org--MiniCPM-V-4.6-GGUF/snapshots/1d73353fab60fe6040ab73eb1e457e2aab210100/mmproj-MiniCPM-V-4.6-bf16.gguf \
+  --host 0.0.0.0 \
+  --port 8080 \
+  --media-path /Users/ilhamfadhil/Documents/Work/build-small-tennis-stroke/data \
+  -ngl 99 \
+  -n 1024 \
+  -c 65536
+  - verbosity 0