| --- |
| title: StrokeSense |
| emoji: 🎾 |
| colorFrom: green |
| colorTo: gray |
| sdk: gradio |
| sdk_version: "5.35.0" |
| python_version: "3.12" |
| app_file: app.py |
| pinned: false |
| tags: |
| - best-minicpm-build |
| - llama-cpp |
| - backyard-ai |
| - track:backyard |
| - sponsor:openbmb |
| - achievement:offgrid |
| - achievement:llama |
| - achievement:sharing |
| --- |
| # Tennis Stroke Analyser |
|
|
| A tennis stroke analyser that detects shots from video and scores mechanics using a local vision-language model — no cloud API required. Designed to analyse close-up shots from a drill, it identifies each groundstroke (forehand or backhand), then generates scores and notes using vision capabilities. |
|
|
| Built for the **Build Small Hackathon** · Track: **Backyard AI** |
|
|
| <img src="images/ai_tennis_coach_picture.png" width="700" alt="Demo"> |
|
|
| <video width="640" controls> |
| <source src="https://huggingface.co/spaces/build-small-hackathon/StrokeSense/resolve/main/data/ai_tennis_coach_720.mp4" type="video/mp4"> |
| </video> |
|
|
| ## Social Media Post |
|
|
| [](https://discord.com/channels/879548962464493619/1515506816069668914) |
|
|
| ## Motivation |
|
|
| I build this for my tennis buddy that want feedback from coach when he do not have access to coach to assess his technique. I also add simple coach ai to chat to interaction, to give more detail explanation and discuss about training recommendation. |
|
|
| ## How it works |
|
|
| 1. **Shot detection** — MoveNet (TFLite) extracts pose keypoints per frame. A sliding-window RNN classifies each window as forehand, backhand, serve, or neutral. |
| 2. **Clip extraction** — A short clip is cut around each detected shot. |
| 3. **Stroke analysis** — Clip frames are sent to a local **MiniCPM-V-4.6** server (running via **llama.cpp**). The model scores four mechanics dimensions (preparation, contact point, swing/follow-through, and balance/stance) and returns structured JSON. |
| 4. **Coach recommendation** — Chat with a coach agent powered by **MiniCPM-V-4.6** to get recommendations and discuss your stroke analysis further. |
|
|
| ``` |
| ┌─────────────────────────────────────┐ |
| │ StrokeSense Pipeline │ |
| └─────────────────────────────────────┘ |
| |
| ┌─────────────────────────────────────┐ |
| │ 1. Read MP4 files │ |
| │ Load video input │ |
| └──────────────────┬──────────────────┘ |
| │ |
| ▼ |
| ┌─────────────────────────────────────┐ |
| │ 2. Classify every stroke │ |
| │ Forehand, backhand, serve... │ |
| └──────────────────┬──────────────────┘ |
| │ |
| ▼ |
| ┌─────────────────────────────────────┐ |
| │ 3. Cut video per stroke │ |
| │ Trim segment per stroke │ |
| └──────────────────┬──────────────────┘ |
| │ |
| ▼ |
| ┌─────────────────────────────────────┐ |
| │ 4. Analyze with MiniCPM-V-4.6 │ |
| │ VLM inference per segment │ |
| └──────────────────┬──────────────────┘ |
| │ |
| ▼ |
| ┌─────────────────────────────────────┐ |
| │ 5. Insert to coach AI agent │ |
| │ Feed results for coaching │ |
| └─────────────────────────────────────┘ |
| ``` |
|
|
| ## Future Improvement |
|
|
| * Add tool calling and deeper tennis references to ai tennis coach agent |
| * Decrease processing time for video analysis |
|
|
|
|
| **Model files needed**: |
|
|
| | File | Purpose | |
| |---|---| |
| | `models/movenet.tflite` | Shot detection | |
| | MiniCPM-V-4.6-GGUF | VLM stroke analysis | |
|
|
| Download MiniCPM-V-4.6-GGUF from [Hugging Face](https://huggingface.co/ggml-org/MiniCPM-V-4.6-GGUF) and set the path in `start_server.sh`. |
|
|
