revert(handship-hf): restore MediaPipe client-side hand tracking
Browse filesRemove LocateAnything-3B server stack, ZeroGPU deps, and calibration flow.
Back to browser MediaPipe Hand Landmarker + static HF Space (sdk: static).
- DEVELOPERS.md +3 -34
- README.md +11 -28
- app.py +37 -104
- game.js +2 -35
- hand-controls.js +94 -168
- index.html +1 -24
- requirements.txt +2 -18
- services/__init__.py +0 -1
- services/locate/__init__.py +0 -19
- services/locate/hand_tracker.py +0 -329
- services/locate/locateanything_worker.py +0 -152
- services/locate/transformers_compat.py +0 -171
DEVELOPERS.md
CHANGED
|
@@ -12,8 +12,6 @@ When creating the Space from this repository, set **App directory** to:
|
|
| 12 |
handship-hf
|
| 13 |
```
|
| 14 |
|
| 15 |
-
Use **ZeroGPU** hardware (`sdk: gradio`, `app_file: app.py`). LocateAnything runs via `@spaces.GPU` — not on CPU basic.
|
| 16 |
-
|
| 17 |
## Run locally
|
| 18 |
|
| 19 |
```bash
|
|
@@ -21,44 +19,15 @@ cd handship-hf
|
|
| 21 |
chmod +x scripts/fetch-kenney-assets.sh
|
| 22 |
./scripts/fetch-kenney-assets.sh # Kenney CC0 → assets/
|
| 23 |
|
| 24 |
-
pip install torch torchvision
|
| 25 |
pip install -r requirements.txt
|
| 26 |
-
|
| 27 |
-
# or: uvicorn app:app --host 0.0.0.0 --port 7860
|
| 28 |
```
|
| 29 |
|
| 30 |
-
Open `http://localhost:7860/game/` — at **Play**, your hand is calibrated once; webcam frames are sent to `POST /api/hand/track` where **LocateAnything-3B** tracks **your** hand via a visual prompt.
|
| 31 |
-
|
| 32 |
## Stack
|
| 33 |
|
| 34 |
- **Three.js** — 3D TPS game in the browser
|
| 35 |
-
- **
|
| 36 |
-
- **Gradio
|
| 37 |
-
|
| 38 |
-
## Hand calibration
|
| 39 |
-
|
| 40 |
-
1. Click **Jouer** / **Rejouer**
|
| 41 |
-
2. Show your hand clearly in front of the webcam
|
| 42 |
-
3. The server runs `ground_single` with `LOCATE_HAND_PROMPT`, crops your hand, and stores it as a **visual prompt**
|
| 43 |
-
4. Later frames use `detect_visual_prompt` to match only that hand category
|
| 44 |
-
|
| 45 |
-
Tune prompts via Space secrets / env vars:
|
| 46 |
-
|
| 47 |
-
```bash
|
| 48 |
-
LOCATE_HAND_PROMPT="my right hand"
|
| 49 |
-
LOCATE_FINGER_PROMPT="index fingertip of my right hand"
|
| 50 |
-
LOCATE_HAND_SELECT=temporal # temporal | center | largest
|
| 51 |
-
```
|
| 52 |
-
|
| 53 |
-
## API
|
| 54 |
-
|
| 55 |
-
| Endpoint | Method | Description |
|
| 56 |
-
|----------|--------|-------------|
|
| 57 |
-
| `/api/hand/status` | GET | Model load state, calibration, prompts |
|
| 58 |
-
| `/api/hand/calibrate` | POST | JPEG frame → store your hand visual prompt |
|
| 59 |
-
| `/api/hand/calibrate` | DELETE | Clear stored visual prompt |
|
| 60 |
-
| `/api/hand/track` | POST | JPEG frame → control packet |
|
| 61 |
-
| `/api/hand/reset` | POST | Clear smoothing state |
|
| 62 |
|
| 63 |
## Desktop UE build
|
| 64 |
|
|
|
|
| 12 |
handship-hf
|
| 13 |
```
|
| 14 |
|
|
|
|
|
|
|
| 15 |
## Run locally
|
| 16 |
|
| 17 |
```bash
|
|
|
|
| 19 |
chmod +x scripts/fetch-kenney-assets.sh
|
| 20 |
./scripts/fetch-kenney-assets.sh # Kenney CC0 → assets/
|
| 21 |
|
|
|
|
| 22 |
pip install -r requirements.txt
|
| 23 |
+
uvicorn app:app --host 0.0.0.0 --port 7860
|
|
|
|
| 24 |
```
|
| 25 |
|
|
|
|
|
|
|
| 26 |
## Stack
|
| 27 |
|
| 28 |
- **Three.js** — 3D TPS game in the browser
|
| 29 |
+
- **MediaPipe Tasks Vision (JS)** — client-side hand tracking
|
| 30 |
+
- **Gradio** — Space shell
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 31 |
|
| 32 |
## Desktop UE build
|
| 33 |
|
README.md
CHANGED
|
@@ -3,14 +3,11 @@ title: HandShip 3D — Hand-controlled spaceship
|
|
| 3 |
emoji: 🚀
|
| 4 |
colorFrom: indigo
|
| 5 |
colorTo: blue
|
| 6 |
-
sdk:
|
| 7 |
-
app_file:
|
| 8 |
-
python_version: 3.11
|
| 9 |
-
suggested_hardware: zero-a10g
|
| 10 |
-
startup_duration_timeout: 1h
|
| 11 |
pinned: false
|
| 12 |
license: mit
|
| 13 |
-
short_description: 3D spaceship
|
| 14 |
---
|
| 15 |
|
| 16 |
# HandShip 3D
|
|
@@ -18,40 +15,26 @@ short_description: 3D spaceship game controlled by hand (ZeroGPU)
|
|
| 18 |
Playable **third-person spaceship** game in your browser:
|
| 19 |
|
| 20 |
- **Three.js** — 3D flight, asteroids, score & lives
|
| 21 |
-
- **
|
| 22 |
- **Web Audio** — procedural music & SFX
|
| 23 |
|
| 24 |
Open the Space, click **Jouer**, allow the webcam, then steer with your fingers.
|
| 25 |
|
| 26 |
-
## Hardware
|
| 27 |
-
|
| 28 |
-
Set Space hardware to **ZeroGPU** (free, Gradio only). Inference runs on `@spaces.GPU` workers.
|
| 29 |
-
|
| 30 |
-
Optional Space variable:
|
| 31 |
-
|
| 32 |
-
```bash
|
| 33 |
-
GRADIO_SSR_MODE=false
|
| 34 |
-
```
|
| 35 |
-
|
| 36 |
## Controls
|
| 37 |
|
| 38 |
| Hand gesture | Action |
|
| 39 |
|--------------|--------|
|
| 40 |
-
| Finger direction (
|
| 41 |
| Raised index finger | Boost speed |
|
| 42 |
|
| 43 |
-
## Local dev
|
| 44 |
-
|
| 45 |
-
```bash
|
| 46 |
-
pip install torch torchvision
|
| 47 |
-
pip install -r requirements.txt
|
| 48 |
-
python app.py # or: uvicorn app:app --host 0.0.0.0 --port 7860
|
| 49 |
-
```
|
| 50 |
-
|
| 51 |
-
On local machine `@spaces.GPU` is a no-op (runs on CPU/CUDA if available).
|
| 52 |
-
|
| 53 |
## Gameplay
|
| 54 |
|
| 55 |
- Dodge **26 asteroids**, score +1 per asteroid passed
|
| 56 |
- **3 lives**, local high-score record
|
| 57 |
- Mute button (bottom-left)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
emoji: 🚀
|
| 4 |
colorFrom: indigo
|
| 5 |
colorTo: blue
|
| 6 |
+
sdk: static
|
| 7 |
+
app_file: index.html
|
|
|
|
|
|
|
|
|
|
| 8 |
pinned: false
|
| 9 |
license: mit
|
| 10 |
+
short_description: Fly a 3D spaceship with your hand in the browser
|
| 11 |
---
|
| 12 |
|
| 13 |
# HandShip 3D
|
|
|
|
| 15 |
Playable **third-person spaceship** game in your browser:
|
| 16 |
|
| 17 |
- **Three.js** — 3D flight, asteroids, score & lives
|
| 18 |
+
- **MediaPipe** — hand tracking runs client-side (webcam)
|
| 19 |
- **Web Audio** — procedural music & SFX
|
| 20 |
|
| 21 |
Open the Space, click **Jouer**, allow the webcam, then steer with your fingers.
|
| 22 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 23 |
## Controls
|
| 24 |
|
| 25 |
| Hand gesture | Action |
|
| 26 |
|--------------|--------|
|
| 27 |
+
| Finger direction (wrist → fingertips) | Fly that way (left/right/up/down) |
|
| 28 |
| Raised index finger | Boost speed |
|
| 29 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 30 |
## Gameplay
|
| 31 |
|
| 32 |
- Dodge **26 asteroids**, score +1 per asteroid passed
|
| 33 |
- **3 lives**, local high-score record
|
| 34 |
- Mute button (bottom-left)
|
| 35 |
+
|
| 36 |
+
## Local dev (optional Gradio shell)
|
| 37 |
+
|
| 38 |
+
```bash
|
| 39 |
+
uvicorn app:app --host 0.0.0.0 --port 7860
|
| 40 |
+
```
|
app.py
CHANGED
|
@@ -1,113 +1,46 @@
|
|
| 1 |
-
"""HandShip HF Space —
|
| 2 |
|
| 3 |
from __future__ import annotations
|
| 4 |
|
| 5 |
-
import os
|
| 6 |
from pathlib import Path
|
| 7 |
|
| 8 |
-
import
|
| 9 |
-
from fastapi import
|
| 10 |
-
from fastapi.responses import RedirectResponse
|
| 11 |
from fastapi.staticfiles import StaticFiles
|
| 12 |
-
from gradio import Server
|
| 13 |
|
| 14 |
WEB_DIR = Path(__file__).resolve().parent
|
| 15 |
|
| 16 |
-
|
| 17 |
-
|
| 18 |
-
|
| 19 |
-
|
| 20 |
-
|
| 21 |
-
|
| 22 |
-
|
| 23 |
-
|
| 24 |
-
|
| 25 |
-
|
| 26 |
-
|
| 27 |
-
|
| 28 |
-
|
| 29 |
-
|
| 30 |
-
|
| 31 |
-
|
| 32 |
-
|
| 33 |
-
|
| 34 |
-
|
| 35 |
-
|
| 36 |
-
|
| 37 |
-
|
| 38 |
-
|
| 39 |
-
|
| 40 |
-
|
| 41 |
-
|
| 42 |
-
|
| 43 |
-
|
| 44 |
-
|
| 45 |
-
|
| 46 |
-
|
| 47 |
-
|
| 48 |
-
|
| 49 |
-
|
| 50 |
-
|
| 51 |
-
@app.post("/api/hand/reset")
|
| 52 |
-
def hand_reset():
|
| 53 |
-
_tracker().reset_smoothing()
|
| 54 |
-
return {"ok": True}
|
| 55 |
-
|
| 56 |
-
|
| 57 |
-
@app.post("/api/hand/calibrate")
|
| 58 |
-
async def hand_calibrate(image: UploadFile = File(...)):
|
| 59 |
-
if not image.content_type or not image.content_type.startswith("image/"):
|
| 60 |
-
raise HTTPException(status_code=400, detail="Expected an image upload.")
|
| 61 |
-
|
| 62 |
-
payload = await image.read()
|
| 63 |
-
if not payload:
|
| 64 |
-
raise HTTPException(status_code=400, detail="Empty image upload.")
|
| 65 |
-
|
| 66 |
-
try:
|
| 67 |
-
result = hand_calibrate_gpu(payload)
|
| 68 |
-
except Exception as exc: # noqa: BLE001
|
| 69 |
-
import traceback
|
| 70 |
-
|
| 71 |
-
traceback.print_exc()
|
| 72 |
-
detail = str(exc)
|
| 73 |
-
if detail in {"TypeError", "RuntimeError", "ValueError"}:
|
| 74 |
-
detail = (
|
| 75 |
-
"Model load failed (LocateAnything + transformers 5.x). "
|
| 76 |
-
"Check Space logs for the full traceback."
|
| 77 |
-
)
|
| 78 |
-
raise HTTPException(status_code=503, detail=f"Calibration failed: {detail}") from exc
|
| 79 |
-
|
| 80 |
-
if not result.get("calibrated"):
|
| 81 |
-
raise HTTPException(
|
| 82 |
-
status_code=422,
|
| 83 |
-
detail=result.get("reason", "no_hand_found"),
|
| 84 |
-
)
|
| 85 |
-
return result
|
| 86 |
-
|
| 87 |
-
|
| 88 |
-
@app.delete("/api/hand/calibrate")
|
| 89 |
-
def hand_clear_calibration():
|
| 90 |
-
_tracker().clear_calibration()
|
| 91 |
-
return {"ok": True}
|
| 92 |
-
|
| 93 |
-
|
| 94 |
-
@app.post("/api/hand/track")
|
| 95 |
-
async def hand_track(image: UploadFile = File(...)):
|
| 96 |
-
if not image.content_type or not image.content_type.startswith("image/"):
|
| 97 |
-
raise HTTPException(status_code=400, detail="Expected an image upload.")
|
| 98 |
-
|
| 99 |
-
payload = await image.read()
|
| 100 |
-
if not payload:
|
| 101 |
-
raise HTTPException(status_code=400, detail="Empty image upload.")
|
| 102 |
-
|
| 103 |
-
try:
|
| 104 |
-
return hand_track_gpu(payload)
|
| 105 |
-
except Exception as exc: # noqa: BLE001
|
| 106 |
-
import traceback
|
| 107 |
-
|
| 108 |
-
traceback.print_exc()
|
| 109 |
-
raise HTTPException(status_code=503, detail=f"LocateAnything inference failed: {exc}") from exc
|
| 110 |
-
|
| 111 |
-
|
| 112 |
-
if __name__ == "__main__":
|
| 113 |
-
app.launch(ssr_mode=False)
|
|
|
|
| 1 |
+
"""HandShip HF Space — 3D TPS playable in browser (Three.js + MediaPipe)."""
|
| 2 |
|
| 3 |
from __future__ import annotations
|
| 4 |
|
|
|
|
| 5 |
from pathlib import Path
|
| 6 |
|
| 7 |
+
import gradio as gr
|
| 8 |
+
from fastapi import FastAPI
|
|
|
|
| 9 |
from fastapi.staticfiles import StaticFiles
|
|
|
|
| 10 |
|
| 11 |
WEB_DIR = Path(__file__).resolve().parent
|
| 12 |
|
| 13 |
+
with gr.Blocks(title="HandShip 3D", fill_height=True) as demo:
|
| 14 |
+
gr.Markdown(
|
| 15 |
+
"""
|
| 16 |
+
# HandShip 3D
|
| 17 |
+
|
| 18 |
+
Jeu **jouable dans le navigateur** — webcam + MediaPipe + Three.js.
|
| 19 |
+
Cliquez **Jouer** dans la fenêtre ci-dessous et autorisez la caméra.
|
| 20 |
+
|
| 21 |
+
| Geste | Action |
|
| 22 |
+
|-------|--------|
|
| 23 |
+
| Direction des doigts (poignet → bouts) | Voler dans cette direction |
|
| 24 |
+
| Index levé | Boost de vitesse |
|
| 25 |
+
|
| 26 |
+
Esquivez les astéroïdes · **3 vies** · score & record local.
|
| 27 |
+
"""
|
| 28 |
+
)
|
| 29 |
+
gr.HTML(
|
| 30 |
+
"""
|
| 31 |
+
<iframe
|
| 32 |
+
src="/game/"
|
| 33 |
+
width="100%"
|
| 34 |
+
height="860"
|
| 35 |
+
style="border:0;border-radius:10px;background:#050810;"
|
| 36 |
+
allow="camera; microphone"
|
| 37 |
+
title="HandShip 3D"
|
| 38 |
+
></iframe>
|
| 39 |
+
""",
|
| 40 |
+
container=False,
|
| 41 |
+
)
|
| 42 |
+
|
| 43 |
+
# HF Spaces: expose ASGI `app` with static game assets at /game/
|
| 44 |
+
fastapi_app = FastAPI()
|
| 45 |
+
fastapi_app.mount("/game", StaticFiles(directory=str(WEB_DIR), html=True), name="game")
|
| 46 |
+
app = gr.mount_gradio_app(fastapi_app, demo, path="/")
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
game.js
CHANGED
|
@@ -1,13 +1,6 @@
|
|
| 1 |
import * as THREE from "three";
|
| 2 |
import { loadGameAssets } from "./assets.js";
|
| 3 |
-
import {
|
| 4 |
-
calibrateHand,
|
| 5 |
-
createHandTracker,
|
| 6 |
-
resetSmoothing,
|
| 7 |
-
startWebcam,
|
| 8 |
-
startWebcamPreviewLoop,
|
| 9 |
-
waitForLocateBackend,
|
| 10 |
-
} from "./hand-controls.js";
|
| 11 |
import { createAudio } from "./audio.js";
|
| 12 |
|
| 13 |
const CRUISE_SPEED = 46;
|
|
@@ -37,8 +30,6 @@ const overlayTitleEl = document.getElementById("overlay-title");
|
|
| 37 |
const overlayTextEl = document.getElementById("overlay-text");
|
| 38 |
const startBtn = document.getElementById("start-btn");
|
| 39 |
const statusEl = document.getElementById("status-msg");
|
| 40 |
-
const calibrationPreviewWrap = document.getElementById("calibration-preview-wrap");
|
| 41 |
-
const calibrationPreview = document.getElementById("calibration-preview");
|
| 42 |
const flashEl = document.getElementById("hit-flash");
|
| 43 |
const muteBtn = document.getElementById("mute-btn");
|
| 44 |
|
|
@@ -398,8 +389,7 @@ startBtn.addEventListener("click", async () => {
|
|
| 398 |
setStatus("Accès webcam…");
|
| 399 |
await startWebcam(video);
|
| 400 |
|
| 401 |
-
setStatus("Chargement
|
| 402 |
-
await waitForLocateBackend();
|
| 403 |
handTracker = await createHandTracker(video, preview);
|
| 404 |
bootDone = true;
|
| 405 |
loop();
|
|
@@ -411,29 +401,6 @@ startBtn.addEventListener("click", async () => {
|
|
| 411 |
audio.startEngine();
|
| 412 |
}
|
| 413 |
await audio.resume();
|
| 414 |
-
|
| 415 |
-
const video = document.getElementById("webcam");
|
| 416 |
-
setStatus(
|
| 417 |
-
"Calibration — montrez votre main devant la caméra… (1ère fois : chargement GPU ~1–3 min)",
|
| 418 |
-
);
|
| 419 |
-
|
| 420 |
-
if (calibrationPreviewWrap && calibrationPreview) {
|
| 421 |
-
calibrationPreviewWrap.hidden = false;
|
| 422 |
-
}
|
| 423 |
-
const stopCalibrationPreview =
|
| 424 |
-
calibrationPreview && video
|
| 425 |
-
? startWebcamPreviewLoop(video, calibrationPreview)
|
| 426 |
-
: () => {};
|
| 427 |
-
|
| 428 |
-
try {
|
| 429 |
-
await calibrateHand(video);
|
| 430 |
-
} finally {
|
| 431 |
-
stopCalibrationPreview();
|
| 432 |
-
if (calibrationPreviewWrap) {
|
| 433 |
-
calibrationPreviewWrap.hidden = true;
|
| 434 |
-
}
|
| 435 |
-
}
|
| 436 |
-
|
| 437 |
audio.startMusic();
|
| 438 |
audio.playStart();
|
| 439 |
resetGame();
|
|
|
|
| 1 |
import * as THREE from "three";
|
| 2 |
import { loadGameAssets } from "./assets.js";
|
| 3 |
+
import { createHandTracker, resetSmoothing, startWebcam } from "./hand-controls.js";
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 4 |
import { createAudio } from "./audio.js";
|
| 5 |
|
| 6 |
const CRUISE_SPEED = 46;
|
|
|
|
| 30 |
const overlayTextEl = document.getElementById("overlay-text");
|
| 31 |
const startBtn = document.getElementById("start-btn");
|
| 32 |
const statusEl = document.getElementById("status-msg");
|
|
|
|
|
|
|
| 33 |
const flashEl = document.getElementById("hit-flash");
|
| 34 |
const muteBtn = document.getElementById("mute-btn");
|
| 35 |
|
|
|
|
| 389 |
setStatus("Accès webcam…");
|
| 390 |
await startWebcam(video);
|
| 391 |
|
| 392 |
+
setStatus("Chargement MediaPipe…");
|
|
|
|
| 393 |
handTracker = await createHandTracker(video, preview);
|
| 394 |
bootDone = true;
|
| 395 |
loop();
|
|
|
|
| 401 |
audio.startEngine();
|
| 402 |
}
|
| 403 |
await audio.resume();
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 404 |
audio.startMusic();
|
| 405 |
audio.playStart();
|
| 406 |
resetGame();
|
hand-controls.js
CHANGED
|
@@ -1,12 +1,12 @@
|
|
| 1 |
/**
|
| 2 |
-
*
|
| 3 |
-
* The
|
|
|
|
| 4 |
*/
|
| 5 |
|
| 6 |
const SMOOTHING = 0.5;
|
| 7 |
const SENSITIVITY = 1.1;
|
| 8 |
const INPUT_DEADZONE = 0.08;
|
| 9 |
-
const TRACK_INTERVAL_MS = 300;
|
| 10 |
|
| 11 |
let prev = null;
|
| 12 |
|
|
@@ -16,20 +16,43 @@ function shapeAxis(v) {
|
|
| 16 |
return sign * ((Math.abs(v) - INPUT_DEADZONE) / (1 - INPUT_DEADZONE));
|
| 17 |
}
|
| 18 |
|
| 19 |
-
|
| 20 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 21 |
return { handDetected: false, steerX: 0, steerY: 0, throttle: 0 };
|
| 22 |
}
|
| 23 |
|
| 24 |
-
const
|
| 25 |
-
const
|
| 26 |
-
const scale = 0.12;
|
| 27 |
-
const dx = (tip.x - palm.x) / scale;
|
| 28 |
-
const dy = (tip.y - palm.y) / scale;
|
| 29 |
|
|
|
|
|
|
|
| 30 |
const steerX = shapeAxis(Math.tanh(-dx * SENSITIVITY));
|
| 31 |
const steerY = shapeAxis(Math.tanh(-dy * SENSITIVITY));
|
| 32 |
-
|
|
|
|
|
|
|
|
|
|
| 33 |
|
| 34 |
function smooth(key, value) {
|
| 35 |
if (prev === null) return value;
|
|
@@ -48,169 +71,84 @@ export function overlayToControl(overlay) {
|
|
| 48 |
|
| 49 |
export function resetSmoothing() {
|
| 50 |
prev = null;
|
| 51 |
-
fetch("/api/hand/reset", { method: "POST" }).catch(() => {});
|
| 52 |
-
}
|
| 53 |
-
|
| 54 |
-
function syncPreviewCanvasSize(videoEl, previewCanvas) {
|
| 55 |
-
const w = videoEl.videoWidth || 640;
|
| 56 |
-
const h = videoEl.videoHeight || 480;
|
| 57 |
-
if (previewCanvas.width !== w || previewCanvas.height !== h) {
|
| 58 |
-
previewCanvas.width = w;
|
| 59 |
-
previewCanvas.height = h;
|
| 60 |
-
}
|
| 61 |
}
|
| 62 |
|
| 63 |
-
|
| 64 |
-
syncPreviewCanvasSize(videoEl, previewCanvas);
|
| 65 |
-
const w = previewCanvas.width;
|
| 66 |
-
const h = previewCanvas.height;
|
| 67 |
|
| 68 |
-
|
| 69 |
-
|
| 70 |
-
|
| 71 |
-
|
| 72 |
-
}
|
| 73 |
|
| 74 |
-
|
| 75 |
-
|
| 76 |
-
|
| 77 |
-
let running = true;
|
| 78 |
-
let rafId = 0;
|
| 79 |
|
| 80 |
-
|
| 81 |
-
|
| 82 |
-
|
| 83 |
-
|
| 84 |
-
|
| 85 |
-
|
| 86 |
-
|
| 87 |
-
|
| 88 |
-
rafId = requestAnimationFrame(tick);
|
| 89 |
-
return () => {
|
| 90 |
-
running = false;
|
| 91 |
-
cancelAnimationFrame(rafId);
|
| 92 |
-
};
|
| 93 |
-
}
|
| 94 |
-
|
| 95 |
-
function drawPreview(ctx, videoEl, previewCanvas, overlay) {
|
| 96 |
-
drawWebcamPreview(ctx, videoEl, previewCanvas);
|
| 97 |
-
|
| 98 |
-
if (!overlay?.palm || !overlay?.tip) return;
|
| 99 |
-
|
| 100 |
-
const w = previewCanvas.width;
|
| 101 |
-
const h = previewCanvas.height;
|
| 102 |
-
const wx = (1 - overlay.palm.x) * w;
|
| 103 |
-
const wy = overlay.palm.y * h;
|
| 104 |
-
const tx = (1 - overlay.tip.x) * w;
|
| 105 |
-
const ty = overlay.tip.y * h;
|
| 106 |
-
|
| 107 |
-
ctx.strokeStyle = "#00ff88";
|
| 108 |
-
ctx.lineWidth = 3;
|
| 109 |
-
ctx.beginPath();
|
| 110 |
-
ctx.moveTo(wx, wy);
|
| 111 |
-
ctx.lineTo(tx, ty);
|
| 112 |
-
ctx.stroke();
|
| 113 |
-
|
| 114 |
-
ctx.fillStyle = "#00ff88";
|
| 115 |
-
ctx.beginPath();
|
| 116 |
-
ctx.arc(tx, ty, 5, 0, Math.PI * 2);
|
| 117 |
-
ctx.fill();
|
| 118 |
-
}
|
| 119 |
-
|
| 120 |
-
async function captureFrameBlob(videoEl) {
|
| 121 |
-
const w = videoEl.videoWidth;
|
| 122 |
-
const h = videoEl.videoHeight;
|
| 123 |
-
if (!w || !h) return null;
|
| 124 |
-
|
| 125 |
-
const capture = document.createElement("canvas");
|
| 126 |
-
capture.width = w;
|
| 127 |
-
capture.height = h;
|
| 128 |
-
const captureCtx = capture.getContext("2d");
|
| 129 |
-
captureCtx.scale(-1, 1);
|
| 130 |
-
captureCtx.drawImage(videoEl, -w, 0, w, h);
|
| 131 |
-
|
| 132 |
-
return new Promise((resolve) => {
|
| 133 |
-
capture.toBlob((blob) => resolve(blob), "image/jpeg", 0.82);
|
| 134 |
});
|
| 135 |
}
|
| 136 |
|
| 137 |
-
export async function
|
| 138 |
-
|
| 139 |
-
if (!blob) {
|
| 140 |
-
throw new Error("Webcam frame unavailable for calibration.");
|
| 141 |
-
}
|
| 142 |
-
|
| 143 |
-
const form = new FormData();
|
| 144 |
-
form.append("image", blob, "calibration.jpg");
|
| 145 |
-
const response = await fetch("/api/hand/calibrate", { method: "POST", body: form });
|
| 146 |
-
if (!response.ok) {
|
| 147 |
-
const detail = await response.text();
|
| 148 |
-
let message = detail;
|
| 149 |
try {
|
| 150 |
-
|
| 151 |
-
if (parsed?.detail) message = String(parsed.detail);
|
| 152 |
} catch {
|
| 153 |
-
|
| 154 |
}
|
| 155 |
-
if (response.status === 422 || message.includes("no_hand_found")) {
|
| 156 |
-
throw new Error("Main non détectée — montrez votre main clairement devant la caméra.");
|
| 157 |
-
}
|
| 158 |
-
if (response.status === 503) {
|
| 159 |
-
throw new Error(
|
| 160 |
-
"Le modèle LocateAnything n'a pas pu s'exécuter. Réessayez dans une minute.",
|
| 161 |
-
);
|
| 162 |
-
}
|
| 163 |
-
throw new Error(message || `Calibration failed (${response.status})`);
|
| 164 |
}
|
| 165 |
-
|
| 166 |
-
}
|
| 167 |
|
| 168 |
-
export async function createHandTracker(videoEl, previewCanvas) {
|
| 169 |
const ctx = previewCanvas.getContext("2d");
|
|
|
|
| 170 |
let lastPacket = { handDetected: false, steerX: 0, steerY: 0, throttle: 0 };
|
| 171 |
-
let lastOverlay = null;
|
| 172 |
-
let inFlight = false;
|
| 173 |
-
let lastTrackAt = 0;
|
| 174 |
-
|
| 175 |
-
async function trackFrame() {
|
| 176 |
-
if (inFlight || videoEl.readyState < 2) return;
|
| 177 |
-
const now = performance.now();
|
| 178 |
-
if (now - lastTrackAt < TRACK_INTERVAL_MS) return;
|
| 179 |
-
|
| 180 |
-
const blob = await captureFrameBlob(videoEl);
|
| 181 |
-
if (!blob) return;
|
| 182 |
-
|
| 183 |
-
inFlight = true;
|
| 184 |
-
lastTrackAt = now;
|
| 185 |
-
|
| 186 |
-
try {
|
| 187 |
-
const form = new FormData();
|
| 188 |
-
form.append("image", blob, "frame.jpg");
|
| 189 |
-
const response = await fetch("/api/hand/track", { method: "POST", body: form });
|
| 190 |
-
if (!response.ok) {
|
| 191 |
-
const detail = await response.text();
|
| 192 |
-
throw new Error(detail || `HTTP ${response.status}`);
|
| 193 |
-
}
|
| 194 |
-
|
| 195 |
-
const data = await response.json();
|
| 196 |
-
lastOverlay = data.overlay ?? null;
|
| 197 |
-
lastPacket = {
|
| 198 |
-
handDetected: Boolean(data.handDetected),
|
| 199 |
-
steerX: data.steerX ?? 0,
|
| 200 |
-
steerY: data.steerY ?? 0,
|
| 201 |
-
throttle: data.throttle ?? 0,
|
| 202 |
-
};
|
| 203 |
-
} catch (err) {
|
| 204 |
-
console.warn("[LocateAnything-3B]", err);
|
| 205 |
-
} finally {
|
| 206 |
-
inFlight = false;
|
| 207 |
-
}
|
| 208 |
-
}
|
| 209 |
|
| 210 |
return {
|
| 211 |
async detect() {
|
| 212 |
-
|
| 213 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 214 |
return lastPacket;
|
| 215 |
},
|
| 216 |
};
|
|
@@ -224,15 +162,3 @@ export async function startWebcam(videoEl) {
|
|
| 224 |
videoEl.srcObject = stream;
|
| 225 |
await videoEl.play();
|
| 226 |
}
|
| 227 |
-
|
| 228 |
-
export async function waitForLocateBackend() {
|
| 229 |
-
const response = await fetch("/api/hand/status");
|
| 230 |
-
if (!response.ok) {
|
| 231 |
-
throw new Error("LocateAnything backend unavailable.");
|
| 232 |
-
}
|
| 233 |
-
const status = await response.json();
|
| 234 |
-
if (!status.cudaAvailable && !status.loaded) {
|
| 235 |
-
console.warn("[LocateAnything-3B] CUDA not available — inference will be very slow on CPU.");
|
| 236 |
-
}
|
| 237 |
-
return status;
|
| 238 |
-
}
|
|
|
|
| 1 |
/**
|
| 2 |
+
* Hand landmarks → directional control from finger-pointing direction.
|
| 3 |
+
* The wrist→fingertips vector acts as a joystick: its screen direction
|
| 4 |
+
* (mirror-corrected) becomes the ship's steering input.
|
| 5 |
*/
|
| 6 |
|
| 7 |
const SMOOTHING = 0.5;
|
| 8 |
const SENSITIVITY = 1.1;
|
| 9 |
const INPUT_DEADZONE = 0.08;
|
|
|
|
| 10 |
|
| 11 |
let prev = null;
|
| 12 |
|
|
|
|
| 16 |
return sign * ((Math.abs(v) - INPUT_DEADZONE) / (1 - INPUT_DEADZONE));
|
| 17 |
}
|
| 18 |
|
| 19 |
+
function fingerDirection(pts) {
|
| 20 |
+
const wrist = pts[0];
|
| 21 |
+
const tips = [8, 12, 16, 20];
|
| 22 |
+
let tipX = 0;
|
| 23 |
+
let tipY = 0;
|
| 24 |
+
for (const i of tips) {
|
| 25 |
+
tipX += pts[i].x;
|
| 26 |
+
tipY += pts[i].y;
|
| 27 |
+
}
|
| 28 |
+
tipX /= tips.length;
|
| 29 |
+
tipY /= tips.length;
|
| 30 |
+
|
| 31 |
+
const refLen = Math.hypot(pts[12].x - wrist.x, pts[12].y - wrist.y);
|
| 32 |
+
const scale = Math.max(refLen, 0.055);
|
| 33 |
+
|
| 34 |
+
const dx = (tipX - wrist.x) / scale;
|
| 35 |
+
const dy = (tipY - wrist.y) / scale;
|
| 36 |
+
|
| 37 |
+
return { wrist, tipX, tipY, dx, dy };
|
| 38 |
+
}
|
| 39 |
+
|
| 40 |
+
export function landmarksToControl(landmarks) {
|
| 41 |
+
if (!landmarks || landmarks.length === 0) {
|
| 42 |
return { handDetected: false, steerX: 0, steerY: 0, throttle: 0 };
|
| 43 |
}
|
| 44 |
|
| 45 |
+
const pts = landmarks[0];
|
| 46 |
+
const { dx, dy } = fingerDirection(pts);
|
|
|
|
|
|
|
|
|
|
| 47 |
|
| 48 |
+
// Webcam is mirrored: negate dx so fingers-right steers right.
|
| 49 |
+
// Screen y grows downward: negate dy so fingers-up steers up.
|
| 50 |
const steerX = shapeAxis(Math.tanh(-dx * SENSITIVITY));
|
| 51 |
const steerY = shapeAxis(Math.tanh(-dy * SENSITIVITY));
|
| 52 |
+
|
| 53 |
+
const indexTip = pts[8];
|
| 54 |
+
const indexMcp = pts[5];
|
| 55 |
+
const throttle = Math.max(0, Math.min(1, (indexMcp.y - indexTip.y) * 4));
|
| 56 |
|
| 57 |
function smooth(key, value) {
|
| 58 |
if (prev === null) return value;
|
|
|
|
| 71 |
|
| 72 |
export function resetSmoothing() {
|
| 73 |
prev = null;
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 74 |
}
|
| 75 |
|
| 76 |
+
let landmarkerSingleton = null;
|
|
|
|
|
|
|
|
|
|
| 77 |
|
| 78 |
+
async function createLandmarker(delegate) {
|
| 79 |
+
const { HandLandmarker, FilesetResolver } = await import(
|
| 80 |
+
"https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@0.10.14/+esm"
|
| 81 |
+
);
|
|
|
|
| 82 |
|
| 83 |
+
const vision = await FilesetResolver.forVisionTasks(
|
| 84 |
+
"https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@0.10.14/wasm"
|
| 85 |
+
);
|
|
|
|
|
|
|
| 86 |
|
| 87 |
+
return HandLandmarker.createFromOptions(vision, {
|
| 88 |
+
baseOptions: {
|
| 89 |
+
modelAssetPath:
|
| 90 |
+
"https://storage.googleapis.com/mediapipe-models/hand_landmarker/hand_landmarker/float16/1/hand_landmarker.task",
|
| 91 |
+
delegate,
|
| 92 |
+
},
|
| 93 |
+
runningMode: "VIDEO",
|
| 94 |
+
numHands: 1,
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 95 |
});
|
| 96 |
}
|
| 97 |
|
| 98 |
+
export async function createHandTracker(videoEl, previewCanvas) {
|
| 99 |
+
if (!landmarkerSingleton) {
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 100 |
try {
|
| 101 |
+
landmarkerSingleton = await createLandmarker("GPU");
|
|
|
|
| 102 |
} catch {
|
| 103 |
+
landmarkerSingleton = await createLandmarker("CPU");
|
| 104 |
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 105 |
}
|
| 106 |
+
const landmarker = landmarkerSingleton;
|
|
|
|
| 107 |
|
|
|
|
| 108 |
const ctx = previewCanvas.getContext("2d");
|
| 109 |
+
let lastVideoTime = -1;
|
| 110 |
let lastPacket = { handDetected: false, steerX: 0, steerY: 0, throttle: 0 };
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 111 |
|
| 112 |
return {
|
| 113 |
async detect() {
|
| 114 |
+
if (videoEl.readyState < 2) {
|
| 115 |
+
return lastPacket;
|
| 116 |
+
}
|
| 117 |
+
if (videoEl.currentTime === lastVideoTime) {
|
| 118 |
+
return lastPacket;
|
| 119 |
+
}
|
| 120 |
+
lastVideoTime = videoEl.currentTime;
|
| 121 |
+
const result = landmarker.detectForVideo(videoEl, performance.now());
|
| 122 |
+
lastPacket = landmarksToControl(result.landmarks);
|
| 123 |
+
|
| 124 |
+
const w = previewCanvas.width;
|
| 125 |
+
const h = previewCanvas.height;
|
| 126 |
+
// Mirror the preview so it behaves like a mirror for the user.
|
| 127 |
+
ctx.save();
|
| 128 |
+
ctx.scale(-1, 1);
|
| 129 |
+
ctx.drawImage(videoEl, -w, 0, w, h);
|
| 130 |
+
ctx.restore();
|
| 131 |
+
|
| 132 |
+
if (result.landmarks?.[0]) {
|
| 133 |
+
const pts = result.landmarks[0];
|
| 134 |
+
const { wrist, tipX, tipY } = fingerDirection(pts);
|
| 135 |
+
const wx = (1 - wrist.x) * w;
|
| 136 |
+
const wy = wrist.y * h;
|
| 137 |
+
const tx = (1 - tipX) * w;
|
| 138 |
+
const ty = tipY * h;
|
| 139 |
+
|
| 140 |
+
ctx.strokeStyle = "#00ff88";
|
| 141 |
+
ctx.lineWidth = 3;
|
| 142 |
+
ctx.beginPath();
|
| 143 |
+
ctx.moveTo(wx, wy);
|
| 144 |
+
ctx.lineTo(tx, ty);
|
| 145 |
+
ctx.stroke();
|
| 146 |
+
|
| 147 |
+
ctx.fillStyle = "#00ff88";
|
| 148 |
+
ctx.beginPath();
|
| 149 |
+
ctx.arc(tx, ty, 5, 0, Math.PI * 2);
|
| 150 |
+
ctx.fill();
|
| 151 |
+
}
|
| 152 |
return lastPacket;
|
| 153 |
},
|
| 154 |
};
|
|
|
|
| 162 |
videoEl.srcObject = stream;
|
| 163 |
await videoEl.play();
|
| 164 |
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
index.html
CHANGED
|
@@ -49,22 +49,6 @@
|
|
| 49 |
background: #4cc9f0; border: none; border-radius: 8px; color: #051018;
|
| 50 |
font-weight: 600;
|
| 51 |
}
|
| 52 |
-
#calibration-preview-wrap {
|
| 53 |
-
margin-top: 16px;
|
| 54 |
-
}
|
| 55 |
-
#calibration-preview-wrap[hidden] {
|
| 56 |
-
display: none !important;
|
| 57 |
-
}
|
| 58 |
-
#calibration-preview {
|
| 59 |
-
display: block;
|
| 60 |
-
width: min(92vw, 520px);
|
| 61 |
-
max-height: min(52vh, 390px);
|
| 62 |
-
margin: 0 auto;
|
| 63 |
-
border: 2px solid #4cc9f0;
|
| 64 |
-
border-radius: 8px;
|
| 65 |
-
background: #000;
|
| 66 |
-
object-fit: contain;
|
| 67 |
-
}
|
| 68 |
</style>
|
| 69 |
<script type="importmap">
|
| 70 |
{
|
|
@@ -82,16 +66,9 @@
|
|
| 82 |
<p id="overlay-text">Allow the webcam and dodge the asteroids.</p>
|
| 83 |
<p style="margin-top:8px;font-size:14px;opacity:0.8">
|
| 84 |
Point your fingers in the direction you want to fly — the ship follows.<br/>
|
| 85 |
-
Fingers up = climb · Raised index finger = boost
|
| 86 |
-
<span style="opacity:0.7">At start, show your hand to calibrate tracking (LocateAnything-3B)</span>
|
| 87 |
</p>
|
| 88 |
<p id="status-msg" style="margin-top:12px;font-size:14px;color:#a8e8ff;min-height:1.2em"></p>
|
| 89 |
-
<div id="calibration-preview-wrap" hidden>
|
| 90 |
-
<canvas id="calibration-preview" width="640" height="480"></canvas>
|
| 91 |
-
<p style="margin-top:8px;font-size:13px;opacity:0.85">
|
| 92 |
-
Cadre miroir — alignez votre main paume ouverte face à la caméra
|
| 93 |
-
</p>
|
| 94 |
-
</div>
|
| 95 |
<button id="start-btn">Play</button>
|
| 96 |
</div>
|
| 97 |
</div>
|
|
|
|
| 49 |
background: #4cc9f0; border: none; border-radius: 8px; color: #051018;
|
| 50 |
font-weight: 600;
|
| 51 |
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 52 |
</style>
|
| 53 |
<script type="importmap">
|
| 54 |
{
|
|
|
|
| 66 |
<p id="overlay-text">Allow the webcam and dodge the asteroids.</p>
|
| 67 |
<p style="margin-top:8px;font-size:14px;opacity:0.8">
|
| 68 |
Point your fingers in the direction you want to fly — the ship follows.<br/>
|
| 69 |
+
Fingers up = climb · Raised index finger = boost
|
|
|
|
| 70 |
</p>
|
| 71 |
<p id="status-msg" style="margin-top:12px;font-size:14px;color:#a8e8ff;min-height:1.2em"></p>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 72 |
<button id="start-btn">Play</button>
|
| 73 |
</div>
|
| 74 |
</div>
|
requirements.txt
CHANGED
|
@@ -1,18 +1,2 @@
|
|
| 1 |
-
#
|
| 2 |
-
|
| 3 |
-
wheel>=0.42.0
|
| 4 |
-
|
| 5 |
-
# Do NOT pin: gradio, spaces, torch, huggingface-hub (managed by HF / ZeroGPU runtime)
|
| 6 |
-
|
| 7 |
-
# LocateAnything-3B — transformers 5.x required for huggingface-hub 1.x (Gradio 6.19)
|
| 8 |
-
transformers>=5.0,<6
|
| 9 |
-
accelerate>=0.26.0
|
| 10 |
-
safetensors>=0.4.0
|
| 11 |
-
peft>=0.10.0
|
| 12 |
-
torchvision
|
| 13 |
-
decord==0.6.0
|
| 14 |
-
lmdb==1.7.5
|
| 15 |
-
opencv-python-headless==4.11.0.86
|
| 16 |
-
numpy>=1.26.0,<2.0
|
| 17 |
-
Pillow>=11.1.0
|
| 18 |
-
python-multipart>=0.0.9
|
|
|
|
| 1 |
+
# Gradio, FastAPI, and uvicorn are preinstalled on HF Spaces.
|
| 2 |
+
# Pinning them here causes version conflicts with huggingface_hub.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
services/__init__.py
DELETED
|
@@ -1 +0,0 @@
|
|
| 1 |
-
"""HandShip backend services."""
|
|
|
|
|
|
services/locate/__init__.py
DELETED
|
@@ -1,19 +0,0 @@
|
|
| 1 |
-
"""LocateAnything-3B hand tracking service."""
|
| 2 |
-
|
| 3 |
-
from .hand_tracker import (
|
| 4 |
-
calibrate_from_frame,
|
| 5 |
-
clear_calibration,
|
| 6 |
-
frame_to_control,
|
| 7 |
-
get_worker,
|
| 8 |
-
reset_smoothing,
|
| 9 |
-
worker_status,
|
| 10 |
-
)
|
| 11 |
-
|
| 12 |
-
__all__ = [
|
| 13 |
-
"calibrate_from_frame",
|
| 14 |
-
"clear_calibration",
|
| 15 |
-
"frame_to_control",
|
| 16 |
-
"get_worker",
|
| 17 |
-
"reset_smoothing",
|
| 18 |
-
"worker_status",
|
| 19 |
-
]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
services/locate/hand_tracker.py
DELETED
|
@@ -1,329 +0,0 @@
|
|
| 1 |
-
"""Map webcam frames to ship controls via nvidia/LocateAnything-3B."""
|
| 2 |
-
|
| 3 |
-
from __future__ import annotations
|
| 4 |
-
|
| 5 |
-
import io
|
| 6 |
-
import math
|
| 7 |
-
import os
|
| 8 |
-
import time
|
| 9 |
-
from typing import Any, Literal, Optional
|
| 10 |
-
|
| 11 |
-
import torch
|
| 12 |
-
from PIL import Image
|
| 13 |
-
|
| 14 |
-
from .locateanything_worker import LocateAnythingWorker
|
| 15 |
-
|
| 16 |
-
MODEL_ID = os.environ.get("LOCATE_MODEL_ID", "nvidia/LocateAnything-3B")
|
| 17 |
-
GENERATION_MODE = os.environ.get("LOCATE_GENERATION_MODE", "hybrid")
|
| 18 |
-
MAX_NEW_TOKENS = int(os.environ.get("LOCATE_MAX_NEW_TOKENS", "512"))
|
| 19 |
-
SENSITIVITY = float(os.environ.get("LOCATE_SENSITIVITY", "1.1"))
|
| 20 |
-
DEADZONE = float(os.environ.get("LOCATE_DEADZONE", "0.08"))
|
| 21 |
-
SMOOTHING = float(os.environ.get("LOCATE_SMOOTHING", "0.5"))
|
| 22 |
-
HAND_PROMPT = os.environ.get(
|
| 23 |
-
"LOCATE_HAND_PROMPT",
|
| 24 |
-
"the hand closest to the camera",
|
| 25 |
-
)
|
| 26 |
-
FINGER_PROMPT = os.environ.get(
|
| 27 |
-
"LOCATE_FINGER_PROMPT",
|
| 28 |
-
"the index fingertip of the same hand",
|
| 29 |
-
)
|
| 30 |
-
HAND_SELECT = os.environ.get("LOCATE_HAND_SELECT", "temporal").lower()
|
| 31 |
-
CALIBRATION_PADDING = float(os.environ.get("LOCATE_CALIBRATION_PADDING", "0.12"))
|
| 32 |
-
|
| 33 |
-
SelectMode = Literal["largest", "center", "temporal"]
|
| 34 |
-
|
| 35 |
-
_worker: Optional[LocateAnythingWorker] = None
|
| 36 |
-
_visual_prompt: Optional[Image.Image] = None
|
| 37 |
-
_prev_smooth: dict[str, float] = {}
|
| 38 |
-
_prev_palm: Optional[tuple[float, float]] = None
|
| 39 |
-
|
| 40 |
-
|
| 41 |
-
def get_worker() -> LocateAnythingWorker:
|
| 42 |
-
global _worker
|
| 43 |
-
if _worker is None:
|
| 44 |
-
device = os.environ.get("LOCATE_DEVICE")
|
| 45 |
-
if not device:
|
| 46 |
-
if os.environ.get("SPACES_ZERO_GPU") == "1" or torch.cuda.is_available():
|
| 47 |
-
device = "cuda"
|
| 48 |
-
else:
|
| 49 |
-
device = "cpu"
|
| 50 |
-
dtype = torch.bfloat16 if device != "cpu" else torch.float32
|
| 51 |
-
_worker = LocateAnythingWorker(MODEL_ID, device=device, dtype=dtype)
|
| 52 |
-
return _worker
|
| 53 |
-
|
| 54 |
-
|
| 55 |
-
def worker_status() -> dict[str, Any]:
|
| 56 |
-
device = os.environ.get("LOCATE_DEVICE")
|
| 57 |
-
if not device:
|
| 58 |
-
device = "cuda" if torch.cuda.is_available() else "cpu"
|
| 59 |
-
return {
|
| 60 |
-
"modelId": MODEL_ID,
|
| 61 |
-
"loaded": _worker is not None,
|
| 62 |
-
"device": device,
|
| 63 |
-
"cudaAvailable": torch.cuda.is_available(),
|
| 64 |
-
"generationMode": GENERATION_MODE,
|
| 65 |
-
"api": "transformers.AutoModel(trust_remote_code=True)",
|
| 66 |
-
"backend": "gradio-zerogpu",
|
| 67 |
-
"handPrompt": HAND_PROMPT,
|
| 68 |
-
"fingerPrompt": FINGER_PROMPT,
|
| 69 |
-
"handSelect": HAND_SELECT,
|
| 70 |
-
"calibrated": _visual_prompt is not None,
|
| 71 |
-
}
|
| 72 |
-
|
| 73 |
-
|
| 74 |
-
def reset_smoothing() -> None:
|
| 75 |
-
global _prev_smooth, _prev_palm
|
| 76 |
-
_prev_smooth = {}
|
| 77 |
-
_prev_palm = None
|
| 78 |
-
|
| 79 |
-
|
| 80 |
-
def clear_calibration() -> None:
|
| 81 |
-
global _visual_prompt
|
| 82 |
-
_visual_prompt = None
|
| 83 |
-
reset_smoothing()
|
| 84 |
-
|
| 85 |
-
|
| 86 |
-
def _shape_axis(value: float) -> float:
|
| 87 |
-
if abs(value) < DEADZONE:
|
| 88 |
-
return 0.0
|
| 89 |
-
sign = 1.0 if value >= 0 else -1.0
|
| 90 |
-
return sign * ((abs(value) - DEADZONE) / (1.0 - DEADZONE))
|
| 91 |
-
|
| 92 |
-
|
| 93 |
-
def _smooth(key: str, value: float) -> float:
|
| 94 |
-
if key not in _prev_smooth:
|
| 95 |
-
_prev_smooth[key] = value
|
| 96 |
-
return value
|
| 97 |
-
blended = _prev_smooth[key] * (1.0 - SMOOTHING) + value * SMOOTHING
|
| 98 |
-
_prev_smooth[key] = blended
|
| 99 |
-
return blended
|
| 100 |
-
|
| 101 |
-
|
| 102 |
-
def _box_center(box: dict) -> tuple[float, float]:
|
| 103 |
-
return ((box["x1"] + box["x2"]) / 2.0, (box["y1"] + box["y2"]) / 2.0)
|
| 104 |
-
|
| 105 |
-
|
| 106 |
-
def _box_area(box: dict) -> float:
|
| 107 |
-
return max(0.0, box["x2"] - box["x1"]) * max(0.0, box["y2"] - box["y1"])
|
| 108 |
-
|
| 109 |
-
|
| 110 |
-
def _select_hand_box(
|
| 111 |
-
boxes: list[dict],
|
| 112 |
-
width: int,
|
| 113 |
-
height: int,
|
| 114 |
-
mode: SelectMode,
|
| 115 |
-
) -> Optional[dict]:
|
| 116 |
-
if not boxes:
|
| 117 |
-
return None
|
| 118 |
-
|
| 119 |
-
if mode == "largest":
|
| 120 |
-
return max(boxes, key=_box_area)
|
| 121 |
-
|
| 122 |
-
if mode == "center":
|
| 123 |
-
cx, cy = width / 2.0, height / 2.0
|
| 124 |
-
return min(boxes, key=lambda box: math.hypot(_box_center(box)[0] - cx, _box_center(box)[1] - cy))
|
| 125 |
-
|
| 126 |
-
if _prev_palm is not None:
|
| 127 |
-
px = _prev_palm[0] * width
|
| 128 |
-
py = _prev_palm[1] * height
|
| 129 |
-
return min(
|
| 130 |
-
boxes,
|
| 131 |
-
key=lambda box: math.hypot(_box_center(box)[0] - px, _box_center(box)[1] - py),
|
| 132 |
-
)
|
| 133 |
-
|
| 134 |
-
cx, cy = width / 2.0, height / 2.0
|
| 135 |
-
return min(boxes, key=lambda box: math.hypot(_box_center(box)[0] - cx, _box_center(box)[1] - cy))
|
| 136 |
-
|
| 137 |
-
|
| 138 |
-
def _crop_hand_box(image: Image.Image, box: dict, padding: float = CALIBRATION_PADDING) -> Image.Image:
|
| 139 |
-
width, height = image.size
|
| 140 |
-
pad_x = (box["x2"] - box["x1"]) * padding
|
| 141 |
-
pad_y = (box["y2"] - box["y1"]) * padding
|
| 142 |
-
left = max(0, int(box["x1"] - pad_x))
|
| 143 |
-
top = max(0, int(box["y1"] - pad_y))
|
| 144 |
-
right = min(width, int(box["x2"] + pad_x))
|
| 145 |
-
bottom = min(height, int(box["y2"] + pad_y))
|
| 146 |
-
if right <= left or bottom <= top:
|
| 147 |
-
return image.crop((int(box["x1"]), int(box["y1"]), int(box["x2"]), int(box["y2"]))).convert("RGB")
|
| 148 |
-
return image.crop((left, top, right, bottom)).convert("RGB")
|
| 149 |
-
|
| 150 |
-
|
| 151 |
-
def _point_in_box(point: dict, box: dict, margin_ratio: float = 0.35) -> bool:
|
| 152 |
-
width = max(box["x2"] - box["x1"], 1.0)
|
| 153 |
-
height = max(box["y2"] - box["y1"], 1.0)
|
| 154 |
-
margin_x = width * margin_ratio
|
| 155 |
-
margin_y = height * margin_ratio
|
| 156 |
-
return (
|
| 157 |
-
box["x1"] - margin_x <= point["x"] <= box["x2"] + margin_x
|
| 158 |
-
and box["y1"] - margin_y <= point["y"] <= box["y2"] + margin_y
|
| 159 |
-
)
|
| 160 |
-
|
| 161 |
-
|
| 162 |
-
def _select_fingertip(points: list[dict], hand_box: Optional[dict]) -> Optional[dict]:
|
| 163 |
-
if not points:
|
| 164 |
-
return None
|
| 165 |
-
if hand_box is None:
|
| 166 |
-
return min(points, key=lambda point: point["y"])
|
| 167 |
-
inside = [point for point in points if _point_in_box(point, hand_box)]
|
| 168 |
-
pool = inside or points
|
| 169 |
-
return min(pool, key=lambda point: point["y"])
|
| 170 |
-
|
| 171 |
-
|
| 172 |
-
def _infer_hand_box(
|
| 173 |
-
worker: LocateAnythingWorker,
|
| 174 |
-
image: Image.Image,
|
| 175 |
-
width: int,
|
| 176 |
-
height: int,
|
| 177 |
-
) -> tuple[Optional[dict], str]:
|
| 178 |
-
select_mode: SelectMode = HAND_SELECT if HAND_SELECT in {"largest", "center", "temporal"} else "temporal"
|
| 179 |
-
|
| 180 |
-
if _visual_prompt is not None:
|
| 181 |
-
result = worker.detect_visual_prompt(
|
| 182 |
-
image,
|
| 183 |
-
_visual_prompt,
|
| 184 |
-
generation_mode=GENERATION_MODE,
|
| 185 |
-
max_new_tokens=MAX_NEW_TOKENS,
|
| 186 |
-
verbose=False,
|
| 187 |
-
)
|
| 188 |
-
source = "visual_prompt"
|
| 189 |
-
else:
|
| 190 |
-
result = worker.ground_single(
|
| 191 |
-
image,
|
| 192 |
-
HAND_PROMPT,
|
| 193 |
-
generation_mode=GENERATION_MODE,
|
| 194 |
-
max_new_tokens=MAX_NEW_TOKENS,
|
| 195 |
-
verbose=False,
|
| 196 |
-
)
|
| 197 |
-
source = "ground_single"
|
| 198 |
-
|
| 199 |
-
boxes = LocateAnythingWorker.parse_boxes(result["answer"], width, height)
|
| 200 |
-
hand_box = _select_hand_box(boxes, width, height, select_mode)
|
| 201 |
-
return hand_box, source
|
| 202 |
-
|
| 203 |
-
|
| 204 |
-
def calibrate_from_frame(image_bytes: bytes) -> dict[str, Any]:
|
| 205 |
-
"""Capture the player's hand once and store a visual prompt for later frames."""
|
| 206 |
-
global _visual_prompt, _prev_palm
|
| 207 |
-
|
| 208 |
-
image = Image.open(io.BytesIO(image_bytes)).convert("RGB")
|
| 209 |
-
width, height = image.size
|
| 210 |
-
worker = get_worker()
|
| 211 |
-
started = time.perf_counter()
|
| 212 |
-
|
| 213 |
-
result = worker.ground_single(
|
| 214 |
-
image,
|
| 215 |
-
HAND_PROMPT,
|
| 216 |
-
generation_mode=GENERATION_MODE,
|
| 217 |
-
max_new_tokens=MAX_NEW_TOKENS,
|
| 218 |
-
verbose=False,
|
| 219 |
-
)
|
| 220 |
-
boxes = LocateAnythingWorker.parse_boxes(result["answer"], width, height)
|
| 221 |
-
hand_box = _select_hand_box(boxes, width, height, "center")
|
| 222 |
-
|
| 223 |
-
if hand_box is None:
|
| 224 |
-
return {
|
| 225 |
-
"calibrated": False,
|
| 226 |
-
"reason": "no_hand_found",
|
| 227 |
-
"handPrompt": HAND_PROMPT,
|
| 228 |
-
"latencyMs": int((time.perf_counter() - started) * 1000),
|
| 229 |
-
}
|
| 230 |
-
|
| 231 |
-
_visual_prompt = _crop_hand_box(image, hand_box)
|
| 232 |
-
palm_x, palm_y = _box_center(hand_box)
|
| 233 |
-
_prev_palm = (palm_x / width, palm_y / height)
|
| 234 |
-
reset_smoothing()
|
| 235 |
-
|
| 236 |
-
return {
|
| 237 |
-
"calibrated": True,
|
| 238 |
-
"handPrompt": HAND_PROMPT,
|
| 239 |
-
"handSelect": HAND_SELECT,
|
| 240 |
-
"latencyMs": int((time.perf_counter() - started) * 1000),
|
| 241 |
-
"overlay": {
|
| 242 |
-
"palm": {"x": _prev_palm[0], "y": _prev_palm[1]},
|
| 243 |
-
"tip": {"x": _prev_palm[0], "y": max(0.0, _prev_palm[1] - 0.08)},
|
| 244 |
-
},
|
| 245 |
-
}
|
| 246 |
-
|
| 247 |
-
|
| 248 |
-
def frame_to_control(image_bytes: bytes) -> dict[str, Any]:
|
| 249 |
-
"""Run LocateAnything on a JPEG/PNG frame and return a game control packet."""
|
| 250 |
-
global _prev_palm
|
| 251 |
-
|
| 252 |
-
image = Image.open(io.BytesIO(image_bytes)).convert("RGB")
|
| 253 |
-
width, height = image.size
|
| 254 |
-
worker = get_worker()
|
| 255 |
-
started = time.perf_counter()
|
| 256 |
-
|
| 257 |
-
hand_box, hand_source = _infer_hand_box(worker, image, width, height)
|
| 258 |
-
|
| 259 |
-
finger_result = worker.point(
|
| 260 |
-
image,
|
| 261 |
-
FINGER_PROMPT,
|
| 262 |
-
generation_mode=GENERATION_MODE,
|
| 263 |
-
max_new_tokens=MAX_NEW_TOKENS,
|
| 264 |
-
verbose=False,
|
| 265 |
-
)
|
| 266 |
-
points = LocateAnythingWorker.parse_points(finger_result["answer"], width, height)
|
| 267 |
-
latency_ms = int((time.perf_counter() - started) * 1000)
|
| 268 |
-
|
| 269 |
-
if hand_box is None and not points:
|
| 270 |
-
return {
|
| 271 |
-
"handDetected": False,
|
| 272 |
-
"steerX": 0.0,
|
| 273 |
-
"steerY": 0.0,
|
| 274 |
-
"throttle": 0.0,
|
| 275 |
-
"latencyMs": latency_ms,
|
| 276 |
-
"overlay": None,
|
| 277 |
-
"source": "locateanything-3b",
|
| 278 |
-
"calibrated": _visual_prompt is not None,
|
| 279 |
-
"handSource": hand_source,
|
| 280 |
-
}
|
| 281 |
-
|
| 282 |
-
if hand_box:
|
| 283 |
-
palm_x, palm_y = _box_center(hand_box)
|
| 284 |
-
hand_span = max(hand_box["y2"] - hand_box["y1"], height * 0.05)
|
| 285 |
-
elif points:
|
| 286 |
-
palm_x = points[0]["x"]
|
| 287 |
-
palm_y = points[0]["y"]
|
| 288 |
-
hand_span = height * 0.12
|
| 289 |
-
else:
|
| 290 |
-
palm_x = width * 0.5
|
| 291 |
-
palm_y = height * 0.5
|
| 292 |
-
hand_span = height * 0.12
|
| 293 |
-
|
| 294 |
-
fingertip = _select_fingertip(points, hand_box)
|
| 295 |
-
if fingertip:
|
| 296 |
-
tip_x = fingertip["x"]
|
| 297 |
-
tip_y = fingertip["y"]
|
| 298 |
-
else:
|
| 299 |
-
tip_x = palm_x
|
| 300 |
-
tip_y = palm_y - hand_span * 0.35
|
| 301 |
-
|
| 302 |
-
palm_x_norm = max(0.0, min(1.0, palm_x / width))
|
| 303 |
-
palm_y_norm = max(0.0, min(1.0, palm_y / height))
|
| 304 |
-
tip_x_norm = max(0.0, min(1.0, tip_x / width))
|
| 305 |
-
tip_y_norm = max(0.0, min(1.0, tip_y / height))
|
| 306 |
-
_prev_palm = (palm_x_norm, palm_y_norm)
|
| 307 |
-
|
| 308 |
-
scale = max(hand_span / height, 0.055)
|
| 309 |
-
dx = (tip_x_norm - palm_x_norm) / scale
|
| 310 |
-
dy = (tip_y_norm - palm_y_norm) / scale
|
| 311 |
-
|
| 312 |
-
steer_x = _shape_axis(math.tanh(-dx * SENSITIVITY))
|
| 313 |
-
steer_y = _shape_axis(math.tanh(-dy * SENSITIVITY))
|
| 314 |
-
throttle = max(0.0, min(1.0, (palm_y_norm - tip_y_norm) * 4.0))
|
| 315 |
-
|
| 316 |
-
return {
|
| 317 |
-
"handDetected": True,
|
| 318 |
-
"steerX": _smooth("steerX", steer_x),
|
| 319 |
-
"steerY": _smooth("steerY", steer_y),
|
| 320 |
-
"throttle": _smooth("throttle", throttle),
|
| 321 |
-
"latencyMs": latency_ms,
|
| 322 |
-
"overlay": {
|
| 323 |
-
"palm": {"x": palm_x_norm, "y": palm_y_norm},
|
| 324 |
-
"tip": {"x": tip_x_norm, "y": tip_y_norm},
|
| 325 |
-
},
|
| 326 |
-
"source": "locateanything-3b",
|
| 327 |
-
"calibrated": _visual_prompt is not None,
|
| 328 |
-
"handSource": hand_source,
|
| 329 |
-
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
services/locate/locateanything_worker.py
DELETED
|
@@ -1,152 +0,0 @@
|
|
| 1 |
-
# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
|
| 2 |
-
#
|
| 3 |
-
# nvidia/LocateAnything-3B requires AutoModel + trust_remote_code — not the generic
|
| 4 |
-
# image-text-to-text pipeline (LocateAnythingConfig is not registered there).
|
| 5 |
-
|
| 6 |
-
from __future__ import annotations
|
| 7 |
-
|
| 8 |
-
import re
|
| 9 |
-
from typing import Any, Optional, Sequence, Union
|
| 10 |
-
|
| 11 |
-
import torch
|
| 12 |
-
from PIL import Image
|
| 13 |
-
from transformers import AutoConfig, AutoModel, AutoProcessor, AutoTokenizer
|
| 14 |
-
|
| 15 |
-
from .transformers_compat import (
|
| 16 |
-
apply_locateanything_transformers5_compat,
|
| 17 |
-
patch_locateanything_config,
|
| 18 |
-
)
|
| 19 |
-
|
| 20 |
-
|
| 21 |
-
class LocateAnythingWorker:
|
| 22 |
-
"""Stateful worker that loads nvidia/LocateAnything-3B once and serves queries."""
|
| 23 |
-
|
| 24 |
-
def __init__(
|
| 25 |
-
self,
|
| 26 |
-
model_path: str,
|
| 27 |
-
device: str = "cuda",
|
| 28 |
-
dtype: torch.dtype = torch.bfloat16,
|
| 29 |
-
) -> None:
|
| 30 |
-
device_arg = device
|
| 31 |
-
if device == "cuda" and not torch.cuda.is_available():
|
| 32 |
-
device_arg = "cpu"
|
| 33 |
-
dtype = torch.float32
|
| 34 |
-
|
| 35 |
-
self.device = device_arg
|
| 36 |
-
self.dtype = dtype
|
| 37 |
-
apply_locateanything_transformers5_compat()
|
| 38 |
-
self.tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
|
| 39 |
-
self.processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)
|
| 40 |
-
config = AutoConfig.from_pretrained(model_path, trust_remote_code=True)
|
| 41 |
-
patch_locateanything_config(config)
|
| 42 |
-
self.model = AutoModel.from_pretrained(
|
| 43 |
-
model_path,
|
| 44 |
-
config=config,
|
| 45 |
-
dtype=dtype,
|
| 46 |
-
trust_remote_code=True,
|
| 47 |
-
).to(device_arg).eval()
|
| 48 |
-
|
| 49 |
-
@torch.no_grad()
|
| 50 |
-
def _run(
|
| 51 |
-
self,
|
| 52 |
-
image: Image.Image,
|
| 53 |
-
question: str,
|
| 54 |
-
generation_mode: str = "hybrid",
|
| 55 |
-
max_new_tokens: int = 2048,
|
| 56 |
-
temperature: float = 0.7,
|
| 57 |
-
verbose: bool = False,
|
| 58 |
-
visual_prompt: Optional[Union[Image.Image, Sequence[Image.Image]]] = None,
|
| 59 |
-
) -> dict[str, Any]:
|
| 60 |
-
content: list[dict] = [
|
| 61 |
-
{"type": "image", "image": image.convert("RGB")},
|
| 62 |
-
{"type": "text", "text": question},
|
| 63 |
-
]
|
| 64 |
-
if visual_prompt is not None:
|
| 65 |
-
prompts = [visual_prompt] if isinstance(visual_prompt, Image.Image) else list(visual_prompt)
|
| 66 |
-
for prompt_image in prompts:
|
| 67 |
-
content.append({"type": "image", "image": prompt_image.convert("RGB")})
|
| 68 |
-
|
| 69 |
-
messages = [{"role": "user", "content": content}]
|
| 70 |
-
text = self.processor.py_apply_chat_template(
|
| 71 |
-
messages, tokenize=False, add_generation_prompt=True
|
| 72 |
-
)
|
| 73 |
-
images, videos = self.processor.process_vision_info(messages)
|
| 74 |
-
inputs = self.processor(
|
| 75 |
-
text=[text], images=images, videos=videos, return_tensors="pt"
|
| 76 |
-
).to(self.device)
|
| 77 |
-
|
| 78 |
-
pixel_values = inputs["pixel_values"].to(self.dtype)
|
| 79 |
-
generate_kwargs: dict[str, Any] = {
|
| 80 |
-
"pixel_values": pixel_values,
|
| 81 |
-
"input_ids": inputs["input_ids"],
|
| 82 |
-
"attention_mask": inputs["attention_mask"],
|
| 83 |
-
"image_grid_hws": inputs.get("image_grid_hws"),
|
| 84 |
-
"tokenizer": self.tokenizer,
|
| 85 |
-
"max_new_tokens": max_new_tokens,
|
| 86 |
-
"use_cache": True,
|
| 87 |
-
"generation_mode": generation_mode,
|
| 88 |
-
"temperature": temperature,
|
| 89 |
-
"do_sample": temperature > 0,
|
| 90 |
-
"top_p": 0.9,
|
| 91 |
-
"repetition_penalty": 1.1,
|
| 92 |
-
"verbose": verbose,
|
| 93 |
-
}
|
| 94 |
-
|
| 95 |
-
response = self.model.generate(**generate_kwargs)
|
| 96 |
-
if response is None:
|
| 97 |
-
return {"answer": ""}
|
| 98 |
-
answer = response[0] if isinstance(response, tuple) else response
|
| 99 |
-
if answer is None:
|
| 100 |
-
answer = ""
|
| 101 |
-
result: dict[str, Any] = {"answer": answer}
|
| 102 |
-
if isinstance(response, tuple) and len(response) >= 3:
|
| 103 |
-
result["history"] = response[1]
|
| 104 |
-
result["stats"] = response[2]
|
| 105 |
-
return result
|
| 106 |
-
|
| 107 |
-
def ground_single(self, image: Image.Image, phrase: str, **kwargs) -> dict:
|
| 108 |
-
prompt = f"Locate a single instance that matches the following description: {phrase}."
|
| 109 |
-
return self._run(image, prompt, **kwargs)
|
| 110 |
-
|
| 111 |
-
def point(self, image: Image.Image, phrase: str, **kwargs) -> dict:
|
| 112 |
-
prompt = f"Point to: {phrase}."
|
| 113 |
-
return self._run(image, prompt, **kwargs)
|
| 114 |
-
|
| 115 |
-
def detect_visual_prompt(
|
| 116 |
-
self,
|
| 117 |
-
image: Image.Image,
|
| 118 |
-
visual_prompt: Image.Image,
|
| 119 |
-
**kwargs,
|
| 120 |
-
) -> dict:
|
| 121 |
-
prompt = (
|
| 122 |
-
"Detect all the objects in the image that belong to the category set: <visual_prompt>."
|
| 123 |
-
)
|
| 124 |
-
return self._run(image, prompt, visual_prompt=visual_prompt, **kwargs)
|
| 125 |
-
|
| 126 |
-
@staticmethod
|
| 127 |
-
def parse_boxes(answer: str, image_width: int, image_height: int) -> list[dict]:
|
| 128 |
-
boxes = []
|
| 129 |
-
for match in re.finditer(r"<box><(\d+)><(\d+)><(\d+)><(\d+)></box>", answer):
|
| 130 |
-
x1, y1, x2, y2 = (int(group) for group in match.groups())
|
| 131 |
-
boxes.append(
|
| 132 |
-
{
|
| 133 |
-
"x1": x1 / 1000 * image_width,
|
| 134 |
-
"y1": y1 / 1000 * image_height,
|
| 135 |
-
"x2": x2 / 1000 * image_width,
|
| 136 |
-
"y2": y2 / 1000 * image_height,
|
| 137 |
-
}
|
| 138 |
-
)
|
| 139 |
-
return boxes
|
| 140 |
-
|
| 141 |
-
@staticmethod
|
| 142 |
-
def parse_points(answer: str, image_width: int, image_height: int) -> list[dict]:
|
| 143 |
-
points = []
|
| 144 |
-
for match in re.finditer(r"<box><(\d+)><(\d+)></box>", answer):
|
| 145 |
-
x, y = int(match.group(1)), int(match.group(2))
|
| 146 |
-
points.append(
|
| 147 |
-
{
|
| 148 |
-
"x": x / 1000 * image_width,
|
| 149 |
-
"y": y / 1000 * image_height,
|
| 150 |
-
}
|
| 151 |
-
)
|
| 152 |
-
return points
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
services/locate/transformers_compat.py
DELETED
|
@@ -1,171 +0,0 @@
|
|
| 1 |
-
"""Shims for loading nvidia/LocateAnything-3B under transformers 5.x on HF Spaces."""
|
| 2 |
-
|
| 3 |
-
from __future__ import annotations
|
| 4 |
-
|
| 5 |
-
import inspect
|
| 6 |
-
|
| 7 |
-
|
| 8 |
-
def _find_attn_check_method(cls: type):
|
| 9 |
-
for candidate in cls.mro():
|
| 10 |
-
method = candidate.__dict__.get("_check_and_adjust_attn_implementation")
|
| 11 |
-
if method is not None:
|
| 12 |
-
return method
|
| 13 |
-
return None
|
| 14 |
-
|
| 15 |
-
|
| 16 |
-
def _needs_attn_compat(method) -> bool:
|
| 17 |
-
if isinstance(method, classmethod):
|
| 18 |
-
fn = method.__func__
|
| 19 |
-
elif isinstance(method, staticmethod):
|
| 20 |
-
fn = method.__func__
|
| 21 |
-
else:
|
| 22 |
-
fn = method
|
| 23 |
-
|
| 24 |
-
try:
|
| 25 |
-
signature = inspect.signature(fn)
|
| 26 |
-
except (TypeError, ValueError):
|
| 27 |
-
return False
|
| 28 |
-
|
| 29 |
-
accepts_var_kw = any(
|
| 30 |
-
param.kind == inspect.Parameter.VAR_KEYWORD
|
| 31 |
-
for param in signature.parameters.values()
|
| 32 |
-
)
|
| 33 |
-
return not accepts_var_kw and "allow_all_kernels" not in signature.parameters
|
| 34 |
-
|
| 35 |
-
|
| 36 |
-
def _instance_attn_compat(self, method):
|
| 37 |
-
if isinstance(method, classmethod):
|
| 38 |
-
fn = method.__func__
|
| 39 |
-
signature = inspect.signature(fn)
|
| 40 |
-
|
| 41 |
-
def compat(*args, **kwargs):
|
| 42 |
-
kwargs.pop("allow_all_kernels", None)
|
| 43 |
-
filtered = {
|
| 44 |
-
key: value for key, value in kwargs.items() if key in signature.parameters
|
| 45 |
-
}
|
| 46 |
-
return fn(type(self), *args, **filtered)
|
| 47 |
-
|
| 48 |
-
return compat
|
| 49 |
-
|
| 50 |
-
if isinstance(method, staticmethod):
|
| 51 |
-
fn = method.__func__
|
| 52 |
-
else:
|
| 53 |
-
fn = method
|
| 54 |
-
|
| 55 |
-
signature = inspect.signature(fn)
|
| 56 |
-
|
| 57 |
-
def compat(*args, **kwargs):
|
| 58 |
-
kwargs.pop("allow_all_kernels", None)
|
| 59 |
-
filtered = {
|
| 60 |
-
key: value for key, value in kwargs.items() if key in signature.parameters
|
| 61 |
-
}
|
| 62 |
-
return fn(self, *args, **filtered)
|
| 63 |
-
|
| 64 |
-
return compat
|
| 65 |
-
|
| 66 |
-
|
| 67 |
-
def apply_locateanything_transformers5_compat() -> None:
|
| 68 |
-
"""Strip kwargs that transformers 5 passes but LocateAnything remote code rejects."""
|
| 69 |
-
import transformers.modeling_utils as modeling_utils
|
| 70 |
-
|
| 71 |
-
if getattr(modeling_utils.PreTrainedModel, "_la_compat_applied", False):
|
| 72 |
-
return
|
| 73 |
-
|
| 74 |
-
original_init = modeling_utils.PreTrainedModel.__init__
|
| 75 |
-
original_post_init = modeling_utils.PreTrainedModel.post_init
|
| 76 |
-
original_finalize = modeling_utils.PreTrainedModel._finalize_model_loading
|
| 77 |
-
|
| 78 |
-
@staticmethod
|
| 79 |
-
def _finalize_model_loading(model, load_config, loading_info):
|
| 80 |
-
if not hasattr(model, "all_tied_weights_keys") and hasattr(model, "post_init"):
|
| 81 |
-
post_init(model)
|
| 82 |
-
return original_finalize(model, load_config, loading_info)
|
| 83 |
-
|
| 84 |
-
def post_init(self, *args, **kwargs):
|
| 85 |
-
tied = getattr(self, "_tied_weights_keys", None)
|
| 86 |
-
if isinstance(tied, list):
|
| 87 |
-
if tied == ["lm_head.weight"]:
|
| 88 |
-
mapping = {"lm_head.weight": "model.embed_tokens.weight"}
|
| 89 |
-
else:
|
| 90 |
-
mapping = {}
|
| 91 |
-
object.__setattr__(self, "_tied_weights_keys", mapping)
|
| 92 |
-
return original_post_init(self, *args, **kwargs)
|
| 93 |
-
|
| 94 |
-
def __init__(self, config, *args, **kwargs):
|
| 95 |
-
method = _find_attn_check_method(type(self))
|
| 96 |
-
if method is not None and _needs_attn_compat(method):
|
| 97 |
-
object.__setattr__(
|
| 98 |
-
self,
|
| 99 |
-
"_check_and_adjust_attn_implementation",
|
| 100 |
-
_instance_attn_compat(self, method),
|
| 101 |
-
)
|
| 102 |
-
return original_init(self, config, *args, **kwargs)
|
| 103 |
-
|
| 104 |
-
modeling_utils.PreTrainedModel.__init__ = __init__
|
| 105 |
-
modeling_utils.PreTrainedModel.post_init = post_init
|
| 106 |
-
modeling_utils.PreTrainedModel._finalize_model_loading = _finalize_model_loading
|
| 107 |
-
modeling_utils.PreTrainedModel._la_compat_applied = True
|
| 108 |
-
|
| 109 |
-
_patch_dynamic_cache_legacy()
|
| 110 |
-
|
| 111 |
-
|
| 112 |
-
def _patch_dynamic_cache_legacy() -> None:
|
| 113 |
-
from transformers.cache_utils import DynamicCache, DynamicLayer
|
| 114 |
-
|
| 115 |
-
if getattr(DynamicCache, "_la_legacy_patched", False):
|
| 116 |
-
return
|
| 117 |
-
|
| 118 |
-
def to_legacy_cache(self):
|
| 119 |
-
legacy = []
|
| 120 |
-
for layer in self.layers:
|
| 121 |
-
if getattr(layer, "is_initialized", False):
|
| 122 |
-
legacy.append((layer.keys, layer.values))
|
| 123 |
-
return tuple(legacy)
|
| 124 |
-
|
| 125 |
-
@classmethod
|
| 126 |
-
def from_legacy_cache(cls, past_key_values):
|
| 127 |
-
cache = cls()
|
| 128 |
-
cache.layers = []
|
| 129 |
-
for key_states, value_states in past_key_values:
|
| 130 |
-
layer = DynamicLayer()
|
| 131 |
-
if key_states is not None and value_states is not None:
|
| 132 |
-
layer.keys = key_states
|
| 133 |
-
layer.values = value_states
|
| 134 |
-
layer.is_initialized = True
|
| 135 |
-
layer.dtype = key_states.dtype
|
| 136 |
-
layer.device = key_states.device
|
| 137 |
-
cache.layers.append(layer)
|
| 138 |
-
return cache
|
| 139 |
-
|
| 140 |
-
DynamicCache.to_legacy_cache = to_legacy_cache
|
| 141 |
-
DynamicCache.from_legacy_cache = from_legacy_cache
|
| 142 |
-
DynamicCache._la_legacy_patched = True
|
| 143 |
-
|
| 144 |
-
|
| 145 |
-
def patch_locateanything_config(config) -> None:
|
| 146 |
-
"""Restore nested Qwen2 fields dropped by transformers 5 config hydration."""
|
| 147 |
-
text_config = getattr(config, "text_config", None)
|
| 148 |
-
if text_config is None:
|
| 149 |
-
return
|
| 150 |
-
|
| 151 |
-
raw = text_config.to_dict() if hasattr(text_config, "to_dict") else {}
|
| 152 |
-
required_fields = {
|
| 153 |
-
"rope_theta": 1_000_000.0,
|
| 154 |
-
"max_position_embeddings": 32768,
|
| 155 |
-
"num_hidden_layers": 36,
|
| 156 |
-
"num_attention_heads": 16,
|
| 157 |
-
"num_key_value_heads": 2,
|
| 158 |
-
"hidden_size": 2048,
|
| 159 |
-
"intermediate_size": 11008,
|
| 160 |
-
"vocab_size": 152681,
|
| 161 |
-
}
|
| 162 |
-
|
| 163 |
-
for key, default in required_fields.items():
|
| 164 |
-
value = raw.get(key, default)
|
| 165 |
-
try:
|
| 166 |
-
getattr(text_config, key)
|
| 167 |
-
except AttributeError:
|
| 168 |
-
object.__setattr__(text_config, key, value)
|
| 169 |
-
|
| 170 |
-
if getattr(text_config, "_attn_implementation", None) in (None, "sdpa", "eager"):
|
| 171 |
-
object.__setattr__(text_config, "_attn_implementation", "magi")
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|