ImTamsi commited on
Commit
1ac31cc
·
1 Parent(s): c1f9a5d

revert(handship-hf): restore MediaPipe client-side hand tracking

Browse files

Remove LocateAnything-3B server stack, ZeroGPU deps, and calibration flow.
Back to browser MediaPipe Hand Landmarker + static HF Space (sdk: static).

DEVELOPERS.md CHANGED
@@ -12,8 +12,6 @@ When creating the Space from this repository, set **App directory** to:
12
  handship-hf
13
  ```
14
 
15
- Use **ZeroGPU** hardware (`sdk: gradio`, `app_file: app.py`). LocateAnything runs via `@spaces.GPU` — not on CPU basic.
16
-
17
  ## Run locally
18
 
19
  ```bash
@@ -21,44 +19,15 @@ cd handship-hf
21
  chmod +x scripts/fetch-kenney-assets.sh
22
  ./scripts/fetch-kenney-assets.sh # Kenney CC0 → assets/
23
 
24
- pip install torch torchvision
25
  pip install -r requirements.txt
26
- python app.py
27
- # or: uvicorn app:app --host 0.0.0.0 --port 7860
28
  ```
29
 
30
- Open `http://localhost:7860/game/` — at **Play**, your hand is calibrated once; webcam frames are sent to `POST /api/hand/track` where **LocateAnything-3B** tracks **your** hand via a visual prompt.
31
-
32
  ## Stack
33
 
34
  - **Three.js** — 3D TPS game in the browser
35
- - **nvidia/LocateAnything-3B** — vision-language grounding (GPU server)
36
- - **Gradio Server (`gradio.Server`) + ZeroGPU** — `@spaces.GPU` inference API
37
-
38
- ## Hand calibration
39
-
40
- 1. Click **Jouer** / **Rejouer**
41
- 2. Show your hand clearly in front of the webcam
42
- 3. The server runs `ground_single` with `LOCATE_HAND_PROMPT`, crops your hand, and stores it as a **visual prompt**
43
- 4. Later frames use `detect_visual_prompt` to match only that hand category
44
-
45
- Tune prompts via Space secrets / env vars:
46
-
47
- ```bash
48
- LOCATE_HAND_PROMPT="my right hand"
49
- LOCATE_FINGER_PROMPT="index fingertip of my right hand"
50
- LOCATE_HAND_SELECT=temporal # temporal | center | largest
51
- ```
52
-
53
- ## API
54
-
55
- | Endpoint | Method | Description |
56
- |----------|--------|-------------|
57
- | `/api/hand/status` | GET | Model load state, calibration, prompts |
58
- | `/api/hand/calibrate` | POST | JPEG frame → store your hand visual prompt |
59
- | `/api/hand/calibrate` | DELETE | Clear stored visual prompt |
60
- | `/api/hand/track` | POST | JPEG frame → control packet |
61
- | `/api/hand/reset` | POST | Clear smoothing state |
62
 
63
  ## Desktop UE build
64
 
 
12
  handship-hf
13
  ```
14
 
 
 
15
  ## Run locally
16
 
17
  ```bash
 
19
  chmod +x scripts/fetch-kenney-assets.sh
20
  ./scripts/fetch-kenney-assets.sh # Kenney CC0 → assets/
21
 
 
22
  pip install -r requirements.txt
23
+ uvicorn app:app --host 0.0.0.0 --port 7860
 
24
  ```
25
 
 
 
26
  ## Stack
27
 
28
  - **Three.js** — 3D TPS game in the browser
29
+ - **MediaPipe Tasks Vision (JS)** — client-side hand tracking
30
+ - **Gradio** — Space shell
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
31
 
32
  ## Desktop UE build
33
 
README.md CHANGED
@@ -3,14 +3,11 @@ title: HandShip 3D — Hand-controlled spaceship
3
  emoji: 🚀
4
  colorFrom: indigo
5
  colorTo: blue
6
- sdk: gradio
7
- app_file: app.py
8
- python_version: 3.11
9
- suggested_hardware: zero-a10g
10
- startup_duration_timeout: 1h
11
  pinned: false
12
  license: mit
13
- short_description: 3D spaceship game controlled by hand (ZeroGPU)
14
  ---
15
 
16
  # HandShip 3D
@@ -18,40 +15,26 @@ short_description: 3D spaceship game controlled by hand (ZeroGPU)
18
  Playable **third-person spaceship** game in your browser:
19
 
20
  - **Three.js** — 3D flight, asteroids, score & lives
21
- - **nvidia/LocateAnything-3B** — hand tracking via **ZeroGPU** (Gradio SDK)
22
  - **Web Audio** — procedural music & SFX
23
 
24
  Open the Space, click **Jouer**, allow the webcam, then steer with your fingers.
25
 
26
- ## Hardware
27
-
28
- Set Space hardware to **ZeroGPU** (free, Gradio only). Inference runs on `@spaces.GPU` workers.
29
-
30
- Optional Space variable:
31
-
32
- ```bash
33
- GRADIO_SSR_MODE=false
34
- ```
35
-
36
  ## Controls
37
 
38
  | Hand gesture | Action |
39
  |--------------|--------|
40
- | Finger direction (palmindex) | Fly that way (left/right/up/down) |
41
  | Raised index finger | Boost speed |
42
 
43
- ## Local dev
44
-
45
- ```bash
46
- pip install torch torchvision
47
- pip install -r requirements.txt
48
- python app.py # or: uvicorn app:app --host 0.0.0.0 --port 7860
49
- ```
50
-
51
- On local machine `@spaces.GPU` is a no-op (runs on CPU/CUDA if available).
52
-
53
  ## Gameplay
54
 
55
  - Dodge **26 asteroids**, score +1 per asteroid passed
56
  - **3 lives**, local high-score record
57
  - Mute button (bottom-left)
 
 
 
 
 
 
 
3
  emoji: 🚀
4
  colorFrom: indigo
5
  colorTo: blue
6
+ sdk: static
7
+ app_file: index.html
 
 
 
8
  pinned: false
9
  license: mit
10
+ short_description: Fly a 3D spaceship with your hand in the browser
11
  ---
12
 
13
  # HandShip 3D
 
15
  Playable **third-person spaceship** game in your browser:
16
 
17
  - **Three.js** — 3D flight, asteroids, score & lives
18
+ - **MediaPipe** — hand tracking runs client-side (webcam)
19
  - **Web Audio** — procedural music & SFX
20
 
21
  Open the Space, click **Jouer**, allow the webcam, then steer with your fingers.
22
 
 
 
 
 
 
 
 
 
 
 
23
  ## Controls
24
 
25
  | Hand gesture | Action |
26
  |--------------|--------|
27
+ | Finger direction (wristfingertips) | Fly that way (left/right/up/down) |
28
  | Raised index finger | Boost speed |
29
 
 
 
 
 
 
 
 
 
 
 
30
  ## Gameplay
31
 
32
  - Dodge **26 asteroids**, score +1 per asteroid passed
33
  - **3 lives**, local high-score record
34
  - Mute button (bottom-left)
35
+
36
+ ## Local dev (optional Gradio shell)
37
+
38
+ ```bash
39
+ uvicorn app:app --host 0.0.0.0 --port 7860
40
+ ```
app.py CHANGED
@@ -1,113 +1,46 @@
1
- """HandShip HF Space — Gradio Server + ZeroGPU + browser game (LocateAnything-3B)."""
2
 
3
  from __future__ import annotations
4
 
5
- import os
6
  from pathlib import Path
7
 
8
- import spaces
9
- from fastapi import File, HTTPException, UploadFile
10
- from fastapi.responses import RedirectResponse
11
  from fastapi.staticfiles import StaticFiles
12
- from gradio import Server
13
 
14
  WEB_DIR = Path(__file__).resolve().parent
15
 
16
- app = Server(title="HandShip 3D")
17
- app.mount("/game", StaticFiles(directory=str(WEB_DIR), html=True), name="game")
18
-
19
-
20
- def _tracker():
21
- from services.locate import hand_tracker as ht
22
-
23
- return ht
24
-
25
-
26
- @app.get("/")
27
- def root():
28
- return RedirectResponse(url="/game/", status_code=302)
29
-
30
-
31
- @spaces.GPU(duration=180)
32
- def hand_calibrate_gpu(image_bytes: bytes) -> dict:
33
- """LocateAnything calibration — runs on ZeroGPU when available."""
34
- return _tracker().calibrate_from_frame(image_bytes)
35
-
36
-
37
- @spaces.GPU(duration=90)
38
- def hand_track_gpu(image_bytes: bytes) -> dict:
39
- """LocateAnything per-frame tracking — runs on ZeroGPU when available."""
40
- return _tracker().frame_to_control(image_bytes)
41
-
42
-
43
- @app.get("/api/hand/status")
44
- def hand_status():
45
- status = _tracker().worker_status()
46
- status["backend"] = "gradio-server-zerogpu"
47
- status["zeroGpu"] = os.environ.get("SPACES_ZERO_GPU") == "1"
48
- return status
49
-
50
-
51
- @app.post("/api/hand/reset")
52
- def hand_reset():
53
- _tracker().reset_smoothing()
54
- return {"ok": True}
55
-
56
-
57
- @app.post("/api/hand/calibrate")
58
- async def hand_calibrate(image: UploadFile = File(...)):
59
- if not image.content_type or not image.content_type.startswith("image/"):
60
- raise HTTPException(status_code=400, detail="Expected an image upload.")
61
-
62
- payload = await image.read()
63
- if not payload:
64
- raise HTTPException(status_code=400, detail="Empty image upload.")
65
-
66
- try:
67
- result = hand_calibrate_gpu(payload)
68
- except Exception as exc: # noqa: BLE001
69
- import traceback
70
-
71
- traceback.print_exc()
72
- detail = str(exc)
73
- if detail in {"TypeError", "RuntimeError", "ValueError"}:
74
- detail = (
75
- "Model load failed (LocateAnything + transformers 5.x). "
76
- "Check Space logs for the full traceback."
77
- )
78
- raise HTTPException(status_code=503, detail=f"Calibration failed: {detail}") from exc
79
-
80
- if not result.get("calibrated"):
81
- raise HTTPException(
82
- status_code=422,
83
- detail=result.get("reason", "no_hand_found"),
84
- )
85
- return result
86
-
87
-
88
- @app.delete("/api/hand/calibrate")
89
- def hand_clear_calibration():
90
- _tracker().clear_calibration()
91
- return {"ok": True}
92
-
93
-
94
- @app.post("/api/hand/track")
95
- async def hand_track(image: UploadFile = File(...)):
96
- if not image.content_type or not image.content_type.startswith("image/"):
97
- raise HTTPException(status_code=400, detail="Expected an image upload.")
98
-
99
- payload = await image.read()
100
- if not payload:
101
- raise HTTPException(status_code=400, detail="Empty image upload.")
102
-
103
- try:
104
- return hand_track_gpu(payload)
105
- except Exception as exc: # noqa: BLE001
106
- import traceback
107
-
108
- traceback.print_exc()
109
- raise HTTPException(status_code=503, detail=f"LocateAnything inference failed: {exc}") from exc
110
-
111
-
112
- if __name__ == "__main__":
113
- app.launch(ssr_mode=False)
 
1
+ """HandShip HF Space — 3D TPS playable in browser (Three.js + MediaPipe)."""
2
 
3
  from __future__ import annotations
4
 
 
5
  from pathlib import Path
6
 
7
+ import gradio as gr
8
+ from fastapi import FastAPI
 
9
  from fastapi.staticfiles import StaticFiles
 
10
 
11
  WEB_DIR = Path(__file__).resolve().parent
12
 
13
+ with gr.Blocks(title="HandShip 3D", fill_height=True) as demo:
14
+ gr.Markdown(
15
+ """
16
+ # HandShip 3D
17
+
18
+ Jeu **jouable dans le navigateur** — webcam + MediaPipe + Three.js.
19
+ Cliquez **Jouer** dans la fenêtre ci-dessous et autorisez la caméra.
20
+
21
+ | Geste | Action |
22
+ |-------|--------|
23
+ | Direction des doigts (poignet → bouts) | Voler dans cette direction |
24
+ | Index levé | Boost de vitesse |
25
+
26
+ Esquivez les astéroïdes · **3 vies** · score & record local.
27
+ """
28
+ )
29
+ gr.HTML(
30
+ """
31
+ <iframe
32
+ src="/game/"
33
+ width="100%"
34
+ height="860"
35
+ style="border:0;border-radius:10px;background:#050810;"
36
+ allow="camera; microphone"
37
+ title="HandShip 3D"
38
+ ></iframe>
39
+ """,
40
+ container=False,
41
+ )
42
+
43
+ # HF Spaces: expose ASGI `app` with static game assets at /game/
44
+ fastapi_app = FastAPI()
45
+ fastapi_app.mount("/game", StaticFiles(directory=str(WEB_DIR), html=True), name="game")
46
+ app = gr.mount_gradio_app(fastapi_app, demo, path="/")
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
game.js CHANGED
@@ -1,13 +1,6 @@
1
  import * as THREE from "three";
2
  import { loadGameAssets } from "./assets.js";
3
- import {
4
- calibrateHand,
5
- createHandTracker,
6
- resetSmoothing,
7
- startWebcam,
8
- startWebcamPreviewLoop,
9
- waitForLocateBackend,
10
- } from "./hand-controls.js";
11
  import { createAudio } from "./audio.js";
12
 
13
  const CRUISE_SPEED = 46;
@@ -37,8 +30,6 @@ const overlayTitleEl = document.getElementById("overlay-title");
37
  const overlayTextEl = document.getElementById("overlay-text");
38
  const startBtn = document.getElementById("start-btn");
39
  const statusEl = document.getElementById("status-msg");
40
- const calibrationPreviewWrap = document.getElementById("calibration-preview-wrap");
41
- const calibrationPreview = document.getElementById("calibration-preview");
42
  const flashEl = document.getElementById("hit-flash");
43
  const muteBtn = document.getElementById("mute-btn");
44
 
@@ -398,8 +389,7 @@ startBtn.addEventListener("click", async () => {
398
  setStatus("Accès webcam…");
399
  await startWebcam(video);
400
 
401
- setStatus("Chargement nvidia/LocateAnything-3B…");
402
- await waitForLocateBackend();
403
  handTracker = await createHandTracker(video, preview);
404
  bootDone = true;
405
  loop();
@@ -411,29 +401,6 @@ startBtn.addEventListener("click", async () => {
411
  audio.startEngine();
412
  }
413
  await audio.resume();
414
-
415
- const video = document.getElementById("webcam");
416
- setStatus(
417
- "Calibration — montrez votre main devant la caméra… (1ère fois : chargement GPU ~1–3 min)",
418
- );
419
-
420
- if (calibrationPreviewWrap && calibrationPreview) {
421
- calibrationPreviewWrap.hidden = false;
422
- }
423
- const stopCalibrationPreview =
424
- calibrationPreview && video
425
- ? startWebcamPreviewLoop(video, calibrationPreview)
426
- : () => {};
427
-
428
- try {
429
- await calibrateHand(video);
430
- } finally {
431
- stopCalibrationPreview();
432
- if (calibrationPreviewWrap) {
433
- calibrationPreviewWrap.hidden = true;
434
- }
435
- }
436
-
437
  audio.startMusic();
438
  audio.playStart();
439
  resetGame();
 
1
  import * as THREE from "three";
2
  import { loadGameAssets } from "./assets.js";
3
+ import { createHandTracker, resetSmoothing, startWebcam } from "./hand-controls.js";
 
 
 
 
 
 
 
4
  import { createAudio } from "./audio.js";
5
 
6
  const CRUISE_SPEED = 46;
 
30
  const overlayTextEl = document.getElementById("overlay-text");
31
  const startBtn = document.getElementById("start-btn");
32
  const statusEl = document.getElementById("status-msg");
 
 
33
  const flashEl = document.getElementById("hit-flash");
34
  const muteBtn = document.getElementById("mute-btn");
35
 
 
389
  setStatus("Accès webcam…");
390
  await startWebcam(video);
391
 
392
+ setStatus("Chargement MediaPipe…");
 
393
  handTracker = await createHandTracker(video, preview);
394
  bootDone = true;
395
  loop();
 
401
  audio.startEngine();
402
  }
403
  await audio.resume();
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
404
  audio.startMusic();
405
  audio.playStart();
406
  resetGame();
hand-controls.js CHANGED
@@ -1,12 +1,12 @@
1
  /**
2
- * Webcam framesship controls via nvidia/LocateAnything-3B (server-side).
3
- * The palmindex vector acts as a joystick for steering.
 
4
  */
5
 
6
  const SMOOTHING = 0.5;
7
  const SENSITIVITY = 1.1;
8
  const INPUT_DEADZONE = 0.08;
9
- const TRACK_INTERVAL_MS = 300;
10
 
11
  let prev = null;
12
 
@@ -16,20 +16,43 @@ function shapeAxis(v) {
16
  return sign * ((Math.abs(v) - INPUT_DEADZONE) / (1 - INPUT_DEADZONE));
17
  }
18
 
19
- export function overlayToControl(overlay) {
20
- if (!overlay?.palm || !overlay?.tip) {
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
21
  return { handDetected: false, steerX: 0, steerY: 0, throttle: 0 };
22
  }
23
 
24
- const palm = overlay.palm;
25
- const tip = overlay.tip;
26
- const scale = 0.12;
27
- const dx = (tip.x - palm.x) / scale;
28
- const dy = (tip.y - palm.y) / scale;
29
 
 
 
30
  const steerX = shapeAxis(Math.tanh(-dx * SENSITIVITY));
31
  const steerY = shapeAxis(Math.tanh(-dy * SENSITIVITY));
32
- const throttle = Math.max(0, Math.min(1, (palm.y - tip.y) * 4));
 
 
 
33
 
34
  function smooth(key, value) {
35
  if (prev === null) return value;
@@ -48,169 +71,84 @@ export function overlayToControl(overlay) {
48
 
49
  export function resetSmoothing() {
50
  prev = null;
51
- fetch("/api/hand/reset", { method: "POST" }).catch(() => {});
52
- }
53
-
54
- function syncPreviewCanvasSize(videoEl, previewCanvas) {
55
- const w = videoEl.videoWidth || 640;
56
- const h = videoEl.videoHeight || 480;
57
- if (previewCanvas.width !== w || previewCanvas.height !== h) {
58
- previewCanvas.width = w;
59
- previewCanvas.height = h;
60
- }
61
  }
62
 
63
- export function drawWebcamPreview(ctx, videoEl, previewCanvas) {
64
- syncPreviewCanvasSize(videoEl, previewCanvas);
65
- const w = previewCanvas.width;
66
- const h = previewCanvas.height;
67
 
68
- ctx.save();
69
- ctx.scale(-1, 1);
70
- ctx.drawImage(videoEl, -w, 0, w, h);
71
- ctx.restore();
72
- }
73
 
74
- /** Live mirrored webcam feed on a canvas (e.g. during calibration). */
75
- export function startWebcamPreviewLoop(videoEl, previewCanvas) {
76
- const ctx = previewCanvas.getContext("2d");
77
- let running = true;
78
- let rafId = 0;
79
 
80
- const tick = () => {
81
- if (!running) return;
82
- if (videoEl.readyState >= 2) {
83
- drawWebcamPreview(ctx, videoEl, previewCanvas);
84
- }
85
- rafId = requestAnimationFrame(tick);
86
- };
87
-
88
- rafId = requestAnimationFrame(tick);
89
- return () => {
90
- running = false;
91
- cancelAnimationFrame(rafId);
92
- };
93
- }
94
-
95
- function drawPreview(ctx, videoEl, previewCanvas, overlay) {
96
- drawWebcamPreview(ctx, videoEl, previewCanvas);
97
-
98
- if (!overlay?.palm || !overlay?.tip) return;
99
-
100
- const w = previewCanvas.width;
101
- const h = previewCanvas.height;
102
- const wx = (1 - overlay.palm.x) * w;
103
- const wy = overlay.palm.y * h;
104
- const tx = (1 - overlay.tip.x) * w;
105
- const ty = overlay.tip.y * h;
106
-
107
- ctx.strokeStyle = "#00ff88";
108
- ctx.lineWidth = 3;
109
- ctx.beginPath();
110
- ctx.moveTo(wx, wy);
111
- ctx.lineTo(tx, ty);
112
- ctx.stroke();
113
-
114
- ctx.fillStyle = "#00ff88";
115
- ctx.beginPath();
116
- ctx.arc(tx, ty, 5, 0, Math.PI * 2);
117
- ctx.fill();
118
- }
119
-
120
- async function captureFrameBlob(videoEl) {
121
- const w = videoEl.videoWidth;
122
- const h = videoEl.videoHeight;
123
- if (!w || !h) return null;
124
-
125
- const capture = document.createElement("canvas");
126
- capture.width = w;
127
- capture.height = h;
128
- const captureCtx = capture.getContext("2d");
129
- captureCtx.scale(-1, 1);
130
- captureCtx.drawImage(videoEl, -w, 0, w, h);
131
-
132
- return new Promise((resolve) => {
133
- capture.toBlob((blob) => resolve(blob), "image/jpeg", 0.82);
134
  });
135
  }
136
 
137
- export async function calibrateHand(videoEl) {
138
- const blob = await captureFrameBlob(videoEl);
139
- if (!blob) {
140
- throw new Error("Webcam frame unavailable for calibration.");
141
- }
142
-
143
- const form = new FormData();
144
- form.append("image", blob, "calibration.jpg");
145
- const response = await fetch("/api/hand/calibrate", { method: "POST", body: form });
146
- if (!response.ok) {
147
- const detail = await response.text();
148
- let message = detail;
149
  try {
150
- const parsed = JSON.parse(detail);
151
- if (parsed?.detail) message = String(parsed.detail);
152
  } catch {
153
- // keep raw text
154
  }
155
- if (response.status === 422 || message.includes("no_hand_found")) {
156
- throw new Error("Main non détectée — montrez votre main clairement devant la caméra.");
157
- }
158
- if (response.status === 503) {
159
- throw new Error(
160
- "Le modèle LocateAnything n'a pas pu s'exécuter. Réessayez dans une minute.",
161
- );
162
- }
163
- throw new Error(message || `Calibration failed (${response.status})`);
164
  }
165
- return response.json();
166
- }
167
 
168
- export async function createHandTracker(videoEl, previewCanvas) {
169
  const ctx = previewCanvas.getContext("2d");
 
170
  let lastPacket = { handDetected: false, steerX: 0, steerY: 0, throttle: 0 };
171
- let lastOverlay = null;
172
- let inFlight = false;
173
- let lastTrackAt = 0;
174
-
175
- async function trackFrame() {
176
- if (inFlight || videoEl.readyState < 2) return;
177
- const now = performance.now();
178
- if (now - lastTrackAt < TRACK_INTERVAL_MS) return;
179
-
180
- const blob = await captureFrameBlob(videoEl);
181
- if (!blob) return;
182
-
183
- inFlight = true;
184
- lastTrackAt = now;
185
-
186
- try {
187
- const form = new FormData();
188
- form.append("image", blob, "frame.jpg");
189
- const response = await fetch("/api/hand/track", { method: "POST", body: form });
190
- if (!response.ok) {
191
- const detail = await response.text();
192
- throw new Error(detail || `HTTP ${response.status}`);
193
- }
194
-
195
- const data = await response.json();
196
- lastOverlay = data.overlay ?? null;
197
- lastPacket = {
198
- handDetected: Boolean(data.handDetected),
199
- steerX: data.steerX ?? 0,
200
- steerY: data.steerY ?? 0,
201
- throttle: data.throttle ?? 0,
202
- };
203
- } catch (err) {
204
- console.warn("[LocateAnything-3B]", err);
205
- } finally {
206
- inFlight = false;
207
- }
208
- }
209
 
210
  return {
211
  async detect() {
212
- trackFrame();
213
- drawPreview(ctx, videoEl, previewCanvas, lastOverlay);
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
214
  return lastPacket;
215
  },
216
  };
@@ -224,15 +162,3 @@ export async function startWebcam(videoEl) {
224
  videoEl.srcObject = stream;
225
  await videoEl.play();
226
  }
227
-
228
- export async function waitForLocateBackend() {
229
- const response = await fetch("/api/hand/status");
230
- if (!response.ok) {
231
- throw new Error("LocateAnything backend unavailable.");
232
- }
233
- const status = await response.json();
234
- if (!status.cudaAvailable && !status.loaded) {
235
- console.warn("[LocateAnything-3B] CUDA not available — inference will be very slow on CPU.");
236
- }
237
- return status;
238
- }
 
1
  /**
2
+ * Hand landmarksdirectional control from finger-pointing direction.
3
+ * The wristfingertips vector acts as a joystick: its screen direction
4
+ * (mirror-corrected) becomes the ship's steering input.
5
  */
6
 
7
  const SMOOTHING = 0.5;
8
  const SENSITIVITY = 1.1;
9
  const INPUT_DEADZONE = 0.08;
 
10
 
11
  let prev = null;
12
 
 
16
  return sign * ((Math.abs(v) - INPUT_DEADZONE) / (1 - INPUT_DEADZONE));
17
  }
18
 
19
+ function fingerDirection(pts) {
20
+ const wrist = pts[0];
21
+ const tips = [8, 12, 16, 20];
22
+ let tipX = 0;
23
+ let tipY = 0;
24
+ for (const i of tips) {
25
+ tipX += pts[i].x;
26
+ tipY += pts[i].y;
27
+ }
28
+ tipX /= tips.length;
29
+ tipY /= tips.length;
30
+
31
+ const refLen = Math.hypot(pts[12].x - wrist.x, pts[12].y - wrist.y);
32
+ const scale = Math.max(refLen, 0.055);
33
+
34
+ const dx = (tipX - wrist.x) / scale;
35
+ const dy = (tipY - wrist.y) / scale;
36
+
37
+ return { wrist, tipX, tipY, dx, dy };
38
+ }
39
+
40
+ export function landmarksToControl(landmarks) {
41
+ if (!landmarks || landmarks.length === 0) {
42
  return { handDetected: false, steerX: 0, steerY: 0, throttle: 0 };
43
  }
44
 
45
+ const pts = landmarks[0];
46
+ const { dx, dy } = fingerDirection(pts);
 
 
 
47
 
48
+ // Webcam is mirrored: negate dx so fingers-right steers right.
49
+ // Screen y grows downward: negate dy so fingers-up steers up.
50
  const steerX = shapeAxis(Math.tanh(-dx * SENSITIVITY));
51
  const steerY = shapeAxis(Math.tanh(-dy * SENSITIVITY));
52
+
53
+ const indexTip = pts[8];
54
+ const indexMcp = pts[5];
55
+ const throttle = Math.max(0, Math.min(1, (indexMcp.y - indexTip.y) * 4));
56
 
57
  function smooth(key, value) {
58
  if (prev === null) return value;
 
71
 
72
  export function resetSmoothing() {
73
  prev = null;
 
 
 
 
 
 
 
 
 
 
74
  }
75
 
76
+ let landmarkerSingleton = null;
 
 
 
77
 
78
+ async function createLandmarker(delegate) {
79
+ const { HandLandmarker, FilesetResolver } = await import(
80
+ "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@0.10.14/+esm"
81
+ );
 
82
 
83
+ const vision = await FilesetResolver.forVisionTasks(
84
+ "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@0.10.14/wasm"
85
+ );
 
 
86
 
87
+ return HandLandmarker.createFromOptions(vision, {
88
+ baseOptions: {
89
+ modelAssetPath:
90
+ "https://storage.googleapis.com/mediapipe-models/hand_landmarker/hand_landmarker/float16/1/hand_landmarker.task",
91
+ delegate,
92
+ },
93
+ runningMode: "VIDEO",
94
+ numHands: 1,
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
95
  });
96
  }
97
 
98
+ export async function createHandTracker(videoEl, previewCanvas) {
99
+ if (!landmarkerSingleton) {
 
 
 
 
 
 
 
 
 
 
100
  try {
101
+ landmarkerSingleton = await createLandmarker("GPU");
 
102
  } catch {
103
+ landmarkerSingleton = await createLandmarker("CPU");
104
  }
 
 
 
 
 
 
 
 
 
105
  }
106
+ const landmarker = landmarkerSingleton;
 
107
 
 
108
  const ctx = previewCanvas.getContext("2d");
109
+ let lastVideoTime = -1;
110
  let lastPacket = { handDetected: false, steerX: 0, steerY: 0, throttle: 0 };
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
111
 
112
  return {
113
  async detect() {
114
+ if (videoEl.readyState < 2) {
115
+ return lastPacket;
116
+ }
117
+ if (videoEl.currentTime === lastVideoTime) {
118
+ return lastPacket;
119
+ }
120
+ lastVideoTime = videoEl.currentTime;
121
+ const result = landmarker.detectForVideo(videoEl, performance.now());
122
+ lastPacket = landmarksToControl(result.landmarks);
123
+
124
+ const w = previewCanvas.width;
125
+ const h = previewCanvas.height;
126
+ // Mirror the preview so it behaves like a mirror for the user.
127
+ ctx.save();
128
+ ctx.scale(-1, 1);
129
+ ctx.drawImage(videoEl, -w, 0, w, h);
130
+ ctx.restore();
131
+
132
+ if (result.landmarks?.[0]) {
133
+ const pts = result.landmarks[0];
134
+ const { wrist, tipX, tipY } = fingerDirection(pts);
135
+ const wx = (1 - wrist.x) * w;
136
+ const wy = wrist.y * h;
137
+ const tx = (1 - tipX) * w;
138
+ const ty = tipY * h;
139
+
140
+ ctx.strokeStyle = "#00ff88";
141
+ ctx.lineWidth = 3;
142
+ ctx.beginPath();
143
+ ctx.moveTo(wx, wy);
144
+ ctx.lineTo(tx, ty);
145
+ ctx.stroke();
146
+
147
+ ctx.fillStyle = "#00ff88";
148
+ ctx.beginPath();
149
+ ctx.arc(tx, ty, 5, 0, Math.PI * 2);
150
+ ctx.fill();
151
+ }
152
  return lastPacket;
153
  },
154
  };
 
162
  videoEl.srcObject = stream;
163
  await videoEl.play();
164
  }
 
 
 
 
 
 
 
 
 
 
 
 
index.html CHANGED
@@ -49,22 +49,6 @@
49
  background: #4cc9f0; border: none; border-radius: 8px; color: #051018;
50
  font-weight: 600;
51
  }
52
- #calibration-preview-wrap {
53
- margin-top: 16px;
54
- }
55
- #calibration-preview-wrap[hidden] {
56
- display: none !important;
57
- }
58
- #calibration-preview {
59
- display: block;
60
- width: min(92vw, 520px);
61
- max-height: min(52vh, 390px);
62
- margin: 0 auto;
63
- border: 2px solid #4cc9f0;
64
- border-radius: 8px;
65
- background: #000;
66
- object-fit: contain;
67
- }
68
  </style>
69
  <script type="importmap">
70
  {
@@ -82,16 +66,9 @@
82
  <p id="overlay-text">Allow the webcam and dodge the asteroids.</p>
83
  <p style="margin-top:8px;font-size:14px;opacity:0.8">
84
  Point your fingers in the direction you want to fly — the ship follows.<br/>
85
- Fingers up = climb · Raised index finger = boost<br/>
86
- <span style="opacity:0.7">At start, show your hand to calibrate tracking (LocateAnything-3B)</span>
87
  </p>
88
  <p id="status-msg" style="margin-top:12px;font-size:14px;color:#a8e8ff;min-height:1.2em"></p>
89
- <div id="calibration-preview-wrap" hidden>
90
- <canvas id="calibration-preview" width="640" height="480"></canvas>
91
- <p style="margin-top:8px;font-size:13px;opacity:0.85">
92
- Cadre miroir — alignez votre main paume ouverte face à la caméra
93
- </p>
94
- </div>
95
  <button id="start-btn">Play</button>
96
  </div>
97
  </div>
 
49
  background: #4cc9f0; border: none; border-radius: 8px; color: #051018;
50
  font-weight: 600;
51
  }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
52
  </style>
53
  <script type="importmap">
54
  {
 
66
  <p id="overlay-text">Allow the webcam and dodge the asteroids.</p>
67
  <p style="margin-top:8px;font-size:14px;opacity:0.8">
68
  Point your fingers in the direction you want to fly — the ship follows.<br/>
69
+ Fingers up = climb · Raised index finger = boost
 
70
  </p>
71
  <p id="status-msg" style="margin-top:12px;font-size:14px;color:#a8e8ff;min-height:1.2em"></p>
 
 
 
 
 
 
72
  <button id="start-btn">Play</button>
73
  </div>
74
  </div>
requirements.txt CHANGED
@@ -1,18 +1,2 @@
1
- # Build tooling (HF Spaces Python 3.11)
2
- setuptools>=69.0.0
3
- wheel>=0.42.0
4
-
5
- # Do NOT pin: gradio, spaces, torch, huggingface-hub (managed by HF / ZeroGPU runtime)
6
-
7
- # LocateAnything-3B — transformers 5.x required for huggingface-hub 1.x (Gradio 6.19)
8
- transformers>=5.0,<6
9
- accelerate>=0.26.0
10
- safetensors>=0.4.0
11
- peft>=0.10.0
12
- torchvision
13
- decord==0.6.0
14
- lmdb==1.7.5
15
- opencv-python-headless==4.11.0.86
16
- numpy>=1.26.0,<2.0
17
- Pillow>=11.1.0
18
- python-multipart>=0.0.9
 
1
+ # Gradio, FastAPI, and uvicorn are preinstalled on HF Spaces.
2
+ # Pinning them here causes version conflicts with huggingface_hub.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
services/__init__.py DELETED
@@ -1 +0,0 @@
1
- """HandShip backend services."""
 
 
services/locate/__init__.py DELETED
@@ -1,19 +0,0 @@
1
- """LocateAnything-3B hand tracking service."""
2
-
3
- from .hand_tracker import (
4
- calibrate_from_frame,
5
- clear_calibration,
6
- frame_to_control,
7
- get_worker,
8
- reset_smoothing,
9
- worker_status,
10
- )
11
-
12
- __all__ = [
13
- "calibrate_from_frame",
14
- "clear_calibration",
15
- "frame_to_control",
16
- "get_worker",
17
- "reset_smoothing",
18
- "worker_status",
19
- ]
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
services/locate/hand_tracker.py DELETED
@@ -1,329 +0,0 @@
1
- """Map webcam frames to ship controls via nvidia/LocateAnything-3B."""
2
-
3
- from __future__ import annotations
4
-
5
- import io
6
- import math
7
- import os
8
- import time
9
- from typing import Any, Literal, Optional
10
-
11
- import torch
12
- from PIL import Image
13
-
14
- from .locateanything_worker import LocateAnythingWorker
15
-
16
- MODEL_ID = os.environ.get("LOCATE_MODEL_ID", "nvidia/LocateAnything-3B")
17
- GENERATION_MODE = os.environ.get("LOCATE_GENERATION_MODE", "hybrid")
18
- MAX_NEW_TOKENS = int(os.environ.get("LOCATE_MAX_NEW_TOKENS", "512"))
19
- SENSITIVITY = float(os.environ.get("LOCATE_SENSITIVITY", "1.1"))
20
- DEADZONE = float(os.environ.get("LOCATE_DEADZONE", "0.08"))
21
- SMOOTHING = float(os.environ.get("LOCATE_SMOOTHING", "0.5"))
22
- HAND_PROMPT = os.environ.get(
23
- "LOCATE_HAND_PROMPT",
24
- "the hand closest to the camera",
25
- )
26
- FINGER_PROMPT = os.environ.get(
27
- "LOCATE_FINGER_PROMPT",
28
- "the index fingertip of the same hand",
29
- )
30
- HAND_SELECT = os.environ.get("LOCATE_HAND_SELECT", "temporal").lower()
31
- CALIBRATION_PADDING = float(os.environ.get("LOCATE_CALIBRATION_PADDING", "0.12"))
32
-
33
- SelectMode = Literal["largest", "center", "temporal"]
34
-
35
- _worker: Optional[LocateAnythingWorker] = None
36
- _visual_prompt: Optional[Image.Image] = None
37
- _prev_smooth: dict[str, float] = {}
38
- _prev_palm: Optional[tuple[float, float]] = None
39
-
40
-
41
- def get_worker() -> LocateAnythingWorker:
42
- global _worker
43
- if _worker is None:
44
- device = os.environ.get("LOCATE_DEVICE")
45
- if not device:
46
- if os.environ.get("SPACES_ZERO_GPU") == "1" or torch.cuda.is_available():
47
- device = "cuda"
48
- else:
49
- device = "cpu"
50
- dtype = torch.bfloat16 if device != "cpu" else torch.float32
51
- _worker = LocateAnythingWorker(MODEL_ID, device=device, dtype=dtype)
52
- return _worker
53
-
54
-
55
- def worker_status() -> dict[str, Any]:
56
- device = os.environ.get("LOCATE_DEVICE")
57
- if not device:
58
- device = "cuda" if torch.cuda.is_available() else "cpu"
59
- return {
60
- "modelId": MODEL_ID,
61
- "loaded": _worker is not None,
62
- "device": device,
63
- "cudaAvailable": torch.cuda.is_available(),
64
- "generationMode": GENERATION_MODE,
65
- "api": "transformers.AutoModel(trust_remote_code=True)",
66
- "backend": "gradio-zerogpu",
67
- "handPrompt": HAND_PROMPT,
68
- "fingerPrompt": FINGER_PROMPT,
69
- "handSelect": HAND_SELECT,
70
- "calibrated": _visual_prompt is not None,
71
- }
72
-
73
-
74
- def reset_smoothing() -> None:
75
- global _prev_smooth, _prev_palm
76
- _prev_smooth = {}
77
- _prev_palm = None
78
-
79
-
80
- def clear_calibration() -> None:
81
- global _visual_prompt
82
- _visual_prompt = None
83
- reset_smoothing()
84
-
85
-
86
- def _shape_axis(value: float) -> float:
87
- if abs(value) < DEADZONE:
88
- return 0.0
89
- sign = 1.0 if value >= 0 else -1.0
90
- return sign * ((abs(value) - DEADZONE) / (1.0 - DEADZONE))
91
-
92
-
93
- def _smooth(key: str, value: float) -> float:
94
- if key not in _prev_smooth:
95
- _prev_smooth[key] = value
96
- return value
97
- blended = _prev_smooth[key] * (1.0 - SMOOTHING) + value * SMOOTHING
98
- _prev_smooth[key] = blended
99
- return blended
100
-
101
-
102
- def _box_center(box: dict) -> tuple[float, float]:
103
- return ((box["x1"] + box["x2"]) / 2.0, (box["y1"] + box["y2"]) / 2.0)
104
-
105
-
106
- def _box_area(box: dict) -> float:
107
- return max(0.0, box["x2"] - box["x1"]) * max(0.0, box["y2"] - box["y1"])
108
-
109
-
110
- def _select_hand_box(
111
- boxes: list[dict],
112
- width: int,
113
- height: int,
114
- mode: SelectMode,
115
- ) -> Optional[dict]:
116
- if not boxes:
117
- return None
118
-
119
- if mode == "largest":
120
- return max(boxes, key=_box_area)
121
-
122
- if mode == "center":
123
- cx, cy = width / 2.0, height / 2.0
124
- return min(boxes, key=lambda box: math.hypot(_box_center(box)[0] - cx, _box_center(box)[1] - cy))
125
-
126
- if _prev_palm is not None:
127
- px = _prev_palm[0] * width
128
- py = _prev_palm[1] * height
129
- return min(
130
- boxes,
131
- key=lambda box: math.hypot(_box_center(box)[0] - px, _box_center(box)[1] - py),
132
- )
133
-
134
- cx, cy = width / 2.0, height / 2.0
135
- return min(boxes, key=lambda box: math.hypot(_box_center(box)[0] - cx, _box_center(box)[1] - cy))
136
-
137
-
138
- def _crop_hand_box(image: Image.Image, box: dict, padding: float = CALIBRATION_PADDING) -> Image.Image:
139
- width, height = image.size
140
- pad_x = (box["x2"] - box["x1"]) * padding
141
- pad_y = (box["y2"] - box["y1"]) * padding
142
- left = max(0, int(box["x1"] - pad_x))
143
- top = max(0, int(box["y1"] - pad_y))
144
- right = min(width, int(box["x2"] + pad_x))
145
- bottom = min(height, int(box["y2"] + pad_y))
146
- if right <= left or bottom <= top:
147
- return image.crop((int(box["x1"]), int(box["y1"]), int(box["x2"]), int(box["y2"]))).convert("RGB")
148
- return image.crop((left, top, right, bottom)).convert("RGB")
149
-
150
-
151
- def _point_in_box(point: dict, box: dict, margin_ratio: float = 0.35) -> bool:
152
- width = max(box["x2"] - box["x1"], 1.0)
153
- height = max(box["y2"] - box["y1"], 1.0)
154
- margin_x = width * margin_ratio
155
- margin_y = height * margin_ratio
156
- return (
157
- box["x1"] - margin_x <= point["x"] <= box["x2"] + margin_x
158
- and box["y1"] - margin_y <= point["y"] <= box["y2"] + margin_y
159
- )
160
-
161
-
162
- def _select_fingertip(points: list[dict], hand_box: Optional[dict]) -> Optional[dict]:
163
- if not points:
164
- return None
165
- if hand_box is None:
166
- return min(points, key=lambda point: point["y"])
167
- inside = [point for point in points if _point_in_box(point, hand_box)]
168
- pool = inside or points
169
- return min(pool, key=lambda point: point["y"])
170
-
171
-
172
- def _infer_hand_box(
173
- worker: LocateAnythingWorker,
174
- image: Image.Image,
175
- width: int,
176
- height: int,
177
- ) -> tuple[Optional[dict], str]:
178
- select_mode: SelectMode = HAND_SELECT if HAND_SELECT in {"largest", "center", "temporal"} else "temporal"
179
-
180
- if _visual_prompt is not None:
181
- result = worker.detect_visual_prompt(
182
- image,
183
- _visual_prompt,
184
- generation_mode=GENERATION_MODE,
185
- max_new_tokens=MAX_NEW_TOKENS,
186
- verbose=False,
187
- )
188
- source = "visual_prompt"
189
- else:
190
- result = worker.ground_single(
191
- image,
192
- HAND_PROMPT,
193
- generation_mode=GENERATION_MODE,
194
- max_new_tokens=MAX_NEW_TOKENS,
195
- verbose=False,
196
- )
197
- source = "ground_single"
198
-
199
- boxes = LocateAnythingWorker.parse_boxes(result["answer"], width, height)
200
- hand_box = _select_hand_box(boxes, width, height, select_mode)
201
- return hand_box, source
202
-
203
-
204
- def calibrate_from_frame(image_bytes: bytes) -> dict[str, Any]:
205
- """Capture the player's hand once and store a visual prompt for later frames."""
206
- global _visual_prompt, _prev_palm
207
-
208
- image = Image.open(io.BytesIO(image_bytes)).convert("RGB")
209
- width, height = image.size
210
- worker = get_worker()
211
- started = time.perf_counter()
212
-
213
- result = worker.ground_single(
214
- image,
215
- HAND_PROMPT,
216
- generation_mode=GENERATION_MODE,
217
- max_new_tokens=MAX_NEW_TOKENS,
218
- verbose=False,
219
- )
220
- boxes = LocateAnythingWorker.parse_boxes(result["answer"], width, height)
221
- hand_box = _select_hand_box(boxes, width, height, "center")
222
-
223
- if hand_box is None:
224
- return {
225
- "calibrated": False,
226
- "reason": "no_hand_found",
227
- "handPrompt": HAND_PROMPT,
228
- "latencyMs": int((time.perf_counter() - started) * 1000),
229
- }
230
-
231
- _visual_prompt = _crop_hand_box(image, hand_box)
232
- palm_x, palm_y = _box_center(hand_box)
233
- _prev_palm = (palm_x / width, palm_y / height)
234
- reset_smoothing()
235
-
236
- return {
237
- "calibrated": True,
238
- "handPrompt": HAND_PROMPT,
239
- "handSelect": HAND_SELECT,
240
- "latencyMs": int((time.perf_counter() - started) * 1000),
241
- "overlay": {
242
- "palm": {"x": _prev_palm[0], "y": _prev_palm[1]},
243
- "tip": {"x": _prev_palm[0], "y": max(0.0, _prev_palm[1] - 0.08)},
244
- },
245
- }
246
-
247
-
248
- def frame_to_control(image_bytes: bytes) -> dict[str, Any]:
249
- """Run LocateAnything on a JPEG/PNG frame and return a game control packet."""
250
- global _prev_palm
251
-
252
- image = Image.open(io.BytesIO(image_bytes)).convert("RGB")
253
- width, height = image.size
254
- worker = get_worker()
255
- started = time.perf_counter()
256
-
257
- hand_box, hand_source = _infer_hand_box(worker, image, width, height)
258
-
259
- finger_result = worker.point(
260
- image,
261
- FINGER_PROMPT,
262
- generation_mode=GENERATION_MODE,
263
- max_new_tokens=MAX_NEW_TOKENS,
264
- verbose=False,
265
- )
266
- points = LocateAnythingWorker.parse_points(finger_result["answer"], width, height)
267
- latency_ms = int((time.perf_counter() - started) * 1000)
268
-
269
- if hand_box is None and not points:
270
- return {
271
- "handDetected": False,
272
- "steerX": 0.0,
273
- "steerY": 0.0,
274
- "throttle": 0.0,
275
- "latencyMs": latency_ms,
276
- "overlay": None,
277
- "source": "locateanything-3b",
278
- "calibrated": _visual_prompt is not None,
279
- "handSource": hand_source,
280
- }
281
-
282
- if hand_box:
283
- palm_x, palm_y = _box_center(hand_box)
284
- hand_span = max(hand_box["y2"] - hand_box["y1"], height * 0.05)
285
- elif points:
286
- palm_x = points[0]["x"]
287
- palm_y = points[0]["y"]
288
- hand_span = height * 0.12
289
- else:
290
- palm_x = width * 0.5
291
- palm_y = height * 0.5
292
- hand_span = height * 0.12
293
-
294
- fingertip = _select_fingertip(points, hand_box)
295
- if fingertip:
296
- tip_x = fingertip["x"]
297
- tip_y = fingertip["y"]
298
- else:
299
- tip_x = palm_x
300
- tip_y = palm_y - hand_span * 0.35
301
-
302
- palm_x_norm = max(0.0, min(1.0, palm_x / width))
303
- palm_y_norm = max(0.0, min(1.0, palm_y / height))
304
- tip_x_norm = max(0.0, min(1.0, tip_x / width))
305
- tip_y_norm = max(0.0, min(1.0, tip_y / height))
306
- _prev_palm = (palm_x_norm, palm_y_norm)
307
-
308
- scale = max(hand_span / height, 0.055)
309
- dx = (tip_x_norm - palm_x_norm) / scale
310
- dy = (tip_y_norm - palm_y_norm) / scale
311
-
312
- steer_x = _shape_axis(math.tanh(-dx * SENSITIVITY))
313
- steer_y = _shape_axis(math.tanh(-dy * SENSITIVITY))
314
- throttle = max(0.0, min(1.0, (palm_y_norm - tip_y_norm) * 4.0))
315
-
316
- return {
317
- "handDetected": True,
318
- "steerX": _smooth("steerX", steer_x),
319
- "steerY": _smooth("steerY", steer_y),
320
- "throttle": _smooth("throttle", throttle),
321
- "latencyMs": latency_ms,
322
- "overlay": {
323
- "palm": {"x": palm_x_norm, "y": palm_y_norm},
324
- "tip": {"x": tip_x_norm, "y": tip_y_norm},
325
- },
326
- "source": "locateanything-3b",
327
- "calibrated": _visual_prompt is not None,
328
- "handSource": hand_source,
329
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
services/locate/locateanything_worker.py DELETED
@@ -1,152 +0,0 @@
1
- # Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
2
- #
3
- # nvidia/LocateAnything-3B requires AutoModel + trust_remote_code — not the generic
4
- # image-text-to-text pipeline (LocateAnythingConfig is not registered there).
5
-
6
- from __future__ import annotations
7
-
8
- import re
9
- from typing import Any, Optional, Sequence, Union
10
-
11
- import torch
12
- from PIL import Image
13
- from transformers import AutoConfig, AutoModel, AutoProcessor, AutoTokenizer
14
-
15
- from .transformers_compat import (
16
- apply_locateanything_transformers5_compat,
17
- patch_locateanything_config,
18
- )
19
-
20
-
21
- class LocateAnythingWorker:
22
- """Stateful worker that loads nvidia/LocateAnything-3B once and serves queries."""
23
-
24
- def __init__(
25
- self,
26
- model_path: str,
27
- device: str = "cuda",
28
- dtype: torch.dtype = torch.bfloat16,
29
- ) -> None:
30
- device_arg = device
31
- if device == "cuda" and not torch.cuda.is_available():
32
- device_arg = "cpu"
33
- dtype = torch.float32
34
-
35
- self.device = device_arg
36
- self.dtype = dtype
37
- apply_locateanything_transformers5_compat()
38
- self.tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
39
- self.processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)
40
- config = AutoConfig.from_pretrained(model_path, trust_remote_code=True)
41
- patch_locateanything_config(config)
42
- self.model = AutoModel.from_pretrained(
43
- model_path,
44
- config=config,
45
- dtype=dtype,
46
- trust_remote_code=True,
47
- ).to(device_arg).eval()
48
-
49
- @torch.no_grad()
50
- def _run(
51
- self,
52
- image: Image.Image,
53
- question: str,
54
- generation_mode: str = "hybrid",
55
- max_new_tokens: int = 2048,
56
- temperature: float = 0.7,
57
- verbose: bool = False,
58
- visual_prompt: Optional[Union[Image.Image, Sequence[Image.Image]]] = None,
59
- ) -> dict[str, Any]:
60
- content: list[dict] = [
61
- {"type": "image", "image": image.convert("RGB")},
62
- {"type": "text", "text": question},
63
- ]
64
- if visual_prompt is not None:
65
- prompts = [visual_prompt] if isinstance(visual_prompt, Image.Image) else list(visual_prompt)
66
- for prompt_image in prompts:
67
- content.append({"type": "image", "image": prompt_image.convert("RGB")})
68
-
69
- messages = [{"role": "user", "content": content}]
70
- text = self.processor.py_apply_chat_template(
71
- messages, tokenize=False, add_generation_prompt=True
72
- )
73
- images, videos = self.processor.process_vision_info(messages)
74
- inputs = self.processor(
75
- text=[text], images=images, videos=videos, return_tensors="pt"
76
- ).to(self.device)
77
-
78
- pixel_values = inputs["pixel_values"].to(self.dtype)
79
- generate_kwargs: dict[str, Any] = {
80
- "pixel_values": pixel_values,
81
- "input_ids": inputs["input_ids"],
82
- "attention_mask": inputs["attention_mask"],
83
- "image_grid_hws": inputs.get("image_grid_hws"),
84
- "tokenizer": self.tokenizer,
85
- "max_new_tokens": max_new_tokens,
86
- "use_cache": True,
87
- "generation_mode": generation_mode,
88
- "temperature": temperature,
89
- "do_sample": temperature > 0,
90
- "top_p": 0.9,
91
- "repetition_penalty": 1.1,
92
- "verbose": verbose,
93
- }
94
-
95
- response = self.model.generate(**generate_kwargs)
96
- if response is None:
97
- return {"answer": ""}
98
- answer = response[0] if isinstance(response, tuple) else response
99
- if answer is None:
100
- answer = ""
101
- result: dict[str, Any] = {"answer": answer}
102
- if isinstance(response, tuple) and len(response) >= 3:
103
- result["history"] = response[1]
104
- result["stats"] = response[2]
105
- return result
106
-
107
- def ground_single(self, image: Image.Image, phrase: str, **kwargs) -> dict:
108
- prompt = f"Locate a single instance that matches the following description: {phrase}."
109
- return self._run(image, prompt, **kwargs)
110
-
111
- def point(self, image: Image.Image, phrase: str, **kwargs) -> dict:
112
- prompt = f"Point to: {phrase}."
113
- return self._run(image, prompt, **kwargs)
114
-
115
- def detect_visual_prompt(
116
- self,
117
- image: Image.Image,
118
- visual_prompt: Image.Image,
119
- **kwargs,
120
- ) -> dict:
121
- prompt = (
122
- "Detect all the objects in the image that belong to the category set: <visual_prompt>."
123
- )
124
- return self._run(image, prompt, visual_prompt=visual_prompt, **kwargs)
125
-
126
- @staticmethod
127
- def parse_boxes(answer: str, image_width: int, image_height: int) -> list[dict]:
128
- boxes = []
129
- for match in re.finditer(r"<box><(\d+)><(\d+)><(\d+)><(\d+)></box>", answer):
130
- x1, y1, x2, y2 = (int(group) for group in match.groups())
131
- boxes.append(
132
- {
133
- "x1": x1 / 1000 * image_width,
134
- "y1": y1 / 1000 * image_height,
135
- "x2": x2 / 1000 * image_width,
136
- "y2": y2 / 1000 * image_height,
137
- }
138
- )
139
- return boxes
140
-
141
- @staticmethod
142
- def parse_points(answer: str, image_width: int, image_height: int) -> list[dict]:
143
- points = []
144
- for match in re.finditer(r"<box><(\d+)><(\d+)></box>", answer):
145
- x, y = int(match.group(1)), int(match.group(2))
146
- points.append(
147
- {
148
- "x": x / 1000 * image_width,
149
- "y": y / 1000 * image_height,
150
- }
151
- )
152
- return points
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
services/locate/transformers_compat.py DELETED
@@ -1,171 +0,0 @@
1
- """Shims for loading nvidia/LocateAnything-3B under transformers 5.x on HF Spaces."""
2
-
3
- from __future__ import annotations
4
-
5
- import inspect
6
-
7
-
8
- def _find_attn_check_method(cls: type):
9
- for candidate in cls.mro():
10
- method = candidate.__dict__.get("_check_and_adjust_attn_implementation")
11
- if method is not None:
12
- return method
13
- return None
14
-
15
-
16
- def _needs_attn_compat(method) -> bool:
17
- if isinstance(method, classmethod):
18
- fn = method.__func__
19
- elif isinstance(method, staticmethod):
20
- fn = method.__func__
21
- else:
22
- fn = method
23
-
24
- try:
25
- signature = inspect.signature(fn)
26
- except (TypeError, ValueError):
27
- return False
28
-
29
- accepts_var_kw = any(
30
- param.kind == inspect.Parameter.VAR_KEYWORD
31
- for param in signature.parameters.values()
32
- )
33
- return not accepts_var_kw and "allow_all_kernels" not in signature.parameters
34
-
35
-
36
- def _instance_attn_compat(self, method):
37
- if isinstance(method, classmethod):
38
- fn = method.__func__
39
- signature = inspect.signature(fn)
40
-
41
- def compat(*args, **kwargs):
42
- kwargs.pop("allow_all_kernels", None)
43
- filtered = {
44
- key: value for key, value in kwargs.items() if key in signature.parameters
45
- }
46
- return fn(type(self), *args, **filtered)
47
-
48
- return compat
49
-
50
- if isinstance(method, staticmethod):
51
- fn = method.__func__
52
- else:
53
- fn = method
54
-
55
- signature = inspect.signature(fn)
56
-
57
- def compat(*args, **kwargs):
58
- kwargs.pop("allow_all_kernels", None)
59
- filtered = {
60
- key: value for key, value in kwargs.items() if key in signature.parameters
61
- }
62
- return fn(self, *args, **filtered)
63
-
64
- return compat
65
-
66
-
67
- def apply_locateanything_transformers5_compat() -> None:
68
- """Strip kwargs that transformers 5 passes but LocateAnything remote code rejects."""
69
- import transformers.modeling_utils as modeling_utils
70
-
71
- if getattr(modeling_utils.PreTrainedModel, "_la_compat_applied", False):
72
- return
73
-
74
- original_init = modeling_utils.PreTrainedModel.__init__
75
- original_post_init = modeling_utils.PreTrainedModel.post_init
76
- original_finalize = modeling_utils.PreTrainedModel._finalize_model_loading
77
-
78
- @staticmethod
79
- def _finalize_model_loading(model, load_config, loading_info):
80
- if not hasattr(model, "all_tied_weights_keys") and hasattr(model, "post_init"):
81
- post_init(model)
82
- return original_finalize(model, load_config, loading_info)
83
-
84
- def post_init(self, *args, **kwargs):
85
- tied = getattr(self, "_tied_weights_keys", None)
86
- if isinstance(tied, list):
87
- if tied == ["lm_head.weight"]:
88
- mapping = {"lm_head.weight": "model.embed_tokens.weight"}
89
- else:
90
- mapping = {}
91
- object.__setattr__(self, "_tied_weights_keys", mapping)
92
- return original_post_init(self, *args, **kwargs)
93
-
94
- def __init__(self, config, *args, **kwargs):
95
- method = _find_attn_check_method(type(self))
96
- if method is not None and _needs_attn_compat(method):
97
- object.__setattr__(
98
- self,
99
- "_check_and_adjust_attn_implementation",
100
- _instance_attn_compat(self, method),
101
- )
102
- return original_init(self, config, *args, **kwargs)
103
-
104
- modeling_utils.PreTrainedModel.__init__ = __init__
105
- modeling_utils.PreTrainedModel.post_init = post_init
106
- modeling_utils.PreTrainedModel._finalize_model_loading = _finalize_model_loading
107
- modeling_utils.PreTrainedModel._la_compat_applied = True
108
-
109
- _patch_dynamic_cache_legacy()
110
-
111
-
112
- def _patch_dynamic_cache_legacy() -> None:
113
- from transformers.cache_utils import DynamicCache, DynamicLayer
114
-
115
- if getattr(DynamicCache, "_la_legacy_patched", False):
116
- return
117
-
118
- def to_legacy_cache(self):
119
- legacy = []
120
- for layer in self.layers:
121
- if getattr(layer, "is_initialized", False):
122
- legacy.append((layer.keys, layer.values))
123
- return tuple(legacy)
124
-
125
- @classmethod
126
- def from_legacy_cache(cls, past_key_values):
127
- cache = cls()
128
- cache.layers = []
129
- for key_states, value_states in past_key_values:
130
- layer = DynamicLayer()
131
- if key_states is not None and value_states is not None:
132
- layer.keys = key_states
133
- layer.values = value_states
134
- layer.is_initialized = True
135
- layer.dtype = key_states.dtype
136
- layer.device = key_states.device
137
- cache.layers.append(layer)
138
- return cache
139
-
140
- DynamicCache.to_legacy_cache = to_legacy_cache
141
- DynamicCache.from_legacy_cache = from_legacy_cache
142
- DynamicCache._la_legacy_patched = True
143
-
144
-
145
- def patch_locateanything_config(config) -> None:
146
- """Restore nested Qwen2 fields dropped by transformers 5 config hydration."""
147
- text_config = getattr(config, "text_config", None)
148
- if text_config is None:
149
- return
150
-
151
- raw = text_config.to_dict() if hasattr(text_config, "to_dict") else {}
152
- required_fields = {
153
- "rope_theta": 1_000_000.0,
154
- "max_position_embeddings": 32768,
155
- "num_hidden_layers": 36,
156
- "num_attention_heads": 16,
157
- "num_key_value_heads": 2,
158
- "hidden_size": 2048,
159
- "intermediate_size": 11008,
160
- "vocab_size": 152681,
161
- }
162
-
163
- for key, default in required_fields.items():
164
- value = raw.get(key, default)
165
- try:
166
- getattr(text_config, key)
167
- except AttributeError:
168
- object.__setattr__(text_config, key, value)
169
-
170
- if getattr(text_config, "_attn_implementation", None) in (None, "sdpa", "eager"):
171
- object.__setattr__(text_config, "_attn_implementation", "magi")