---
title: StrokeSense
emoji: 🎾
colorFrom: green
colorTo: gray
sdk: gradio
sdk_version: "5.35.0"
python_version: "3.12"
app_file: app.py
pinned: false
tags:
  - best-minicpm-build
  - llama-cpp
  - backyard-ai
  - track:backyard
  - sponsor:openbmb
  - achievement:offgrid
  - achievement:llama
  - achievement:sharing
---
# Tennis Stroke Analyser

A tennis stroke analyser that detects shots from video and scores mechanics using a local vision-language model — no cloud API required. Designed to analyse close-up shots from a drill, it identifies each groundstroke (forehand or backhand), then generates scores and notes using vision capabilities.

Built for the **Build Small Hackathon** · Track: **Backyard AI**

<img src="images/ai_tennis_coach_picture.png" width="700" alt="Demo">

<video width="640" controls>
  <source src="https://huggingface.co/spaces/build-small-hackathon/StrokeSense/resolve/main/data/ai_tennis_coach_720.mp4" type="video/mp4">
</video>

## Social Media Post

[![Discord](https://img.shields.io/badge/Discord-Discussion-5865F2?logo=discord&logoColor=white)](https://discord.com/channels/879548962464493619/1515506816069668914)

## Motivation

I build this for my tennis buddy that want feedback from coach when he do not have access to coach to assess his technique. I also add simple coach ai to chat to interaction, to give more detail explanation and discuss about training recommendation.

## How it works

1. **Shot detection** — MoveNet (TFLite) extracts pose keypoints per frame. A sliding-window RNN classifies each window as forehand, backhand, serve, or neutral.
2. **Clip extraction** — A short clip is cut around each detected shot.
3. **Stroke analysis** — Clip frames are sent to a local **MiniCPM-V-4.6** server (running via **llama.cpp**). The model scores four mechanics dimensions (preparation, contact point, swing/follow-through, and balance/stance) and returns structured JSON.
4. **Coach recommendation** — Chat with a coach agent powered by **MiniCPM-V-4.6** to get recommendations and discuss your stroke analysis further.

```
┌─────────────────────────────────────┐
│         StrokeSense Pipeline        │
└─────────────────────────────────────┘

┌─────────────────────────────────────┐
│  1. Read MP4 files                  │
│     Load video input                │
└──────────────────┬──────────────────┘
                   │
                   ▼
┌─────────────────────────────────────┐
│  2. Classify every stroke           │
│     Forehand, backhand, serve...    │
└──────────────────┬──────────────────┘
                   │
                   ▼
┌─────────────────────────────────────┐
│  3. Cut video per stroke            │
│     Trim segment per stroke         │
└──────────────────┬──────────────────┘
                   │
                   ▼
┌─────────────────────────────────────┐
│  4. Analyze with MiniCPM-V-4.6      │
│     VLM inference per segment       │
└──────────────────┬──────────────────┘
                   │
                   ▼
┌─────────────────────────────────────┐
│  5. Insert to coach AI agent        │
│     Feed results for coaching       │
└─────────────────────────────────────┘
```

## Future Improvement

* Add tool calling and deeper tennis references to ai tennis coach agent
* Decrease processing time for video analysis


**Model files needed**:

| File | Purpose |
|---|---|
| `models/movenet.tflite` | Shot detection |
| MiniCPM-V-4.6-GGUF | VLM stroke analysis |

Download MiniCPM-V-4.6-GGUF from [Hugging Face](https://huggingface.co/ggml-org/MiniCPM-V-4.6-GGUF) and set the path in `start_server.sh`.