README.md

metadata

title: StrokeSense
emoji: 🎾
colorFrom: green
colorTo: gray
sdk: gradio
sdk_version: 5.35.0
python_version: '3.12'
app_file: app.py
pinned: false
tags:
  - best-minicpm-build
  - llama-cpp
  - backyard-ai
  - track:backyard
  - sponsor:openbmb
  - achievement:offgrid
  - achievement:llama
  - achievement:sharing

Tennis Stroke Analyser

A tennis stroke analyser that detects shots from video and scores mechanics using a local vision-language model — no cloud API required. Designed to analyse close-up shots from a drill, it identifies each groundstroke (forehand or backhand), then generates scores and notes using vision capabilities.

Built for the Build Small Hackathon · Track: Backyard AI

Social Media Post

Motivation

I build this for my tennis buddy that want feedback from coach when he do not have access to coach to assess his technique. I also add simple coach ai to chat to interaction, to give more detail explanation and discuss about training recommendation.

How it works

1. Shot detection — MoveNet (TFLite) extracts pose keypoints per frame. A sliding-window RNN classifies each window as forehand, backhand, serve, or neutral.
2. Clip extraction — A short clip is cut around each detected shot.
3. Stroke analysis — Clip frames are sent to a local MiniCPM-V-4.6 server (running via llama.cpp). The model scores four mechanics dimensions (preparation, contact point, swing/follow-through, and balance/stance) and returns structured JSON.
4. Coach recommendation — Chat with a coach agent powered by MiniCPM-V-4.6 to get recommendations and discuss your stroke analysis further.

┌─────────────────────────────────────┐
│         StrokeSense Pipeline        │
└─────────────────────────────────────┘

┌─────────────────────────────────────┐
│  1. Read MP4 files                  │
│     Load video input                │
└──────────────────┬──────────────────┘
                   │
                   ▼
┌─────────────────────────────────────┐
│  2. Classify every stroke           │
│     Forehand, backhand, serve...    │
└──────────────────┬──────────────────┘
                   │
                   ▼
┌─────────────────────────────────────┐
│  3. Cut video per stroke            │
│     Trim segment per stroke         │
└──────────────────┬──────────────────┘
                   │
                   ▼
┌─────────────────────────────────────┐
│  4. Analyze with MiniCPM-V-4.6      │
│     VLM inference per segment       │
└──────────────────┬──────────────────┘
                   │
                   ▼
┌─────────────────────────────────────┐
│  5. Insert to coach AI agent        │
│     Feed results for coaching       │
└─────────────────────────────────────┘

Future Improvement

• Add tool calling and deeper tennis references to ai tennis coach agent
• Decrease processing time for video analysis

Model files needed:

File	Purpose
models/movenet.tflite	Shot detection
MiniCPM-V-4.6-GGUF	VLM stroke analysis

Download MiniCPM-V-4.6-GGUF from Hugging Face and set the path in start_server.sh.