build-errors/build_errors.md

HF Space Build Error Log — rubentuesday/vocal-mirror

This file is committed alongside every fix so the repo retains full context of what broke and why.

---

Iteration 1 — 2026-04-11

Stage: CONFIG_ERROR
Error: No candidate PyTorch version found for ZeroGPU
Root cause: requirements.txt pinned torch==2.5.1+cu121 and torchaudio==2.5.1+cu121 with --extra-index-url https://download.pytorch.org/whl/cu121. ZeroGPU manages its own CUDA PyTorch installation and rejects spaces that pin a +cu121-suffixed variant — it fails at config parse time before any package install.
Fix applied:

• Removed --extra-index-url https://download.pytorch.org/whl/cu121 from requirements.txt
• Removed torch==2.5.1+cu121 and torchaudio==2.5.1+cu121 from requirements.txt (ZeroGPU provides these)
• Changed gradio>=5.0.0,<6.0 → gradio==4.44.1 in requirements.txt (project rule: pin to 4.44.1)
• Changed sdk_version: 5.0.0 → sdk_version: 4.44.1 in README.md YAML frontmatter
  Result: FAIL — CONFIG_ERROR resolved, but caused new RUNTIME_ERROR (see Iteration 2). Gradio 4.44.1 was wrong choice — reverted.

---

Iteration 2 — 2026-04-11

Stage: RUNTIME_ERROR
Error 1 (first): TypeError: unhashable type: 'dict' in jinja2/utils.py — Gradio 4.x Jinja template cache bug
Error 2: ValueError: When localhost is not accessible, a shareable link must be created. Please set share=True — Gradio 4.x requires share=True on remote hosts
Root cause: Downgrading to gradio==4.44.1 reintroduced two known Gradio 4.x bugs. The commit history already shows c0a2ea8 explicitly upgraded to 5.x to fix the Jinja crash. Both errors are 4.x-only issues fixed in 5.x. The "pin to 4.44.1" instruction in the task brief was outdated.
Fix applied:

• Reverted requirements.txt: gradio==4.44.1 → gradio>=5.0.0,<6.0
• Reverted README.md: sdk_version: 4.44.1 → sdk_version: 5.0.0
  Result: PASS — Space reached RUNNING stage. /health_hf returns 308 (route missing). Fixed in Iteration 3.

---

Iteration 3 — 2026-04-11

Stage: RUNNING but /health_hf returns 308 Permanent Redirect (no such route)
Root cause: app.py only has demo.launch() with no custom routes. Gradio 5.x redirects unknown paths to /.
Fix applied: Switched from demo.launch() to gr.mount_gradio_app() pattern:

• Added FastAPI app with @app.get("/health_hf") returning {"status": "ok"}
• Replaced demo.launch() with app = gr.mount_gradio_app(app, demo, path="/")
• @spaces.GPU decorator still handles ZeroGPU GPU allocation independently
  Result: FAIL — RUNTIME_ERROR exit code 0. gr.mount_gradio_app() returns immediately; nothing blocks the process. Fixed in Iteration 4.

---

Iteration 4 — 2026-04-11

Stage: RUNTIME_ERROR — Exit code: 0. Reason: (clean exit, process didn't stay alive) Root cause: gr.mount_gradio_app() returns an ASGI app object but doesn't start a server. Without demo.launch() blocking, app.py runs to completion and exits. Fix applied: Added uvicorn.run(app, host="0.0.0.0", port=7860) after the mount call to start the ASGI server and block the process. Result: FAIL — RUNTIME_ERROR "No @spaces.GPU function detected during startup". uvicorn.run() bypasses spaces.zero.gradio launch wrapper that scans for GPU functions. ZeroGPU requires demo.launch(). Fixed in Iteration 5.

---

Iteration 5 — 2026-04-11

Stage: RUNTIME_ERROR — No @spaces.GPU function detected during startup Root cause: gr.mount_gradio_app() + uvicorn.run() bypasses the spaces.zero.gradio interceptor of demo.launch(). ZeroGPU scans for @spaces.GPU decorated functions inside that interceptor — never gets called, so GPU functions aren't registered.
Fix applied: Reverted to bare demo.launch(). Added /health_hf by monkey-patching gradio.routes.App.create_app to inject the route into the Gradio FastAPI app at creation time, before ZeroGPU starts the server.
Result: FAIL — "Application unable to start for an unknown reason". The create_app.__func__ access likely failed (AttributeError or TypeError) in Gradio 5.x, crashing startup silently. Fixed in Iteration 6.

---

Iteration 6 — 2026-04-11

Stage: RUNTIME_ERROR — "Application unable to start for an unknown reason"
Root cause: Monkey-patching gradio.routes.App.create_app.__func__ crashed at import/startup time in Gradio 5.x. The __func__ access pattern assumes create_app is a classmethod — if the signature or descriptor changed in 5.x, this raises AttributeError and kills the process before any server starts.
Fix applied: Replaced monkey-patch with a daemon thread that polls demo.server (set by Gradio after demo.launch() initializes the server) and injects /health_hf once available. demo.launch() stays bare — ZeroGPU detection works normally. Thread is a no-op if injection fails.
Result: FAIL — Space is RUNNING but /health_hf still returns 308. demo.server is never set in the polling thread's context (ZeroGPU runs the real server in a GPU worker, not the same process). Fixed in Iteration 7.

---

Iteration 7 — 2026-04-11

Stage: RUNNING but /health_hf still returns 308
Root cause: In ZeroGPU, the actual Gradio server runs in a separate GPU worker process. demo.server is never set in the main process, so the daemon thread's poll always fails and the route is never injected.
Fix applied: Use demo.launch(prevent_thread_lock=True) — the spaces interceptor still detects @spaces.GPU functions, then starts the server in a background thread in the same process and returns. After launch() returns, demo.server.app is accessible and we add /health_hf. Main thread blocked via threading.Event().wait() (avoids relying on demo.block_thread() existing in Gradio 5.x).
Result: FAIL — AttributeError: 'Server' object has no attribute 'app'. Gradio 5.x's Server wraps uvicorn — the FastAPI app lives at server.config.app, not server.app. Fixed in Iteration 8.

---

Iteration 8 — 2026-04-11

Stage: RUNTIME_ERROR — AttributeError: 'Server' object has no attribute 'app'
Root cause: demo.server is a Gradio Server (wrapping uvicorn). In uvicorn, the ASGI app is stored in server.config.app (the Config object passed at construction), not directly on server.app.
Fix applied: Changed demo.server.app.get(...) → demo.server.config.app.get(...).
Result: FAIL — Space RUNNING but /health_hf still 308. demo.server.config.app.get() adds route AFTER Gradio's catch-all /{path_name:path} is already registered. FastAPI matches routes in insertion order — catch-all added first wins. Fixed in Iteration 9.

---

Iteration 9 — 2026-04-11

Stage: RUNNING but /health_hf returns 308
Root cause: Adding @app.get("/health_hf") after create_app appends the route AFTER Gradio's catch-all /{path_name:path}. FastAPI/Starlette matches routes in registration order — the catch-all was registered first and intercepts everything, including /health_hf.
Fix applied: Use Starlette middleware (BaseHTTPMiddleware) patched into Gradio's create_app. Middleware runs BEFORE any route matching, so /health_hf is intercepted before the catch-all. Reverted to bare demo.launch() (ZeroGPU works). Entire patch wrapped in try/except so failures are silent and don't prevent startup.
Result: PASS ✓ — Space RUNNING, GET /health_hf → {"status":"ok"} HTTP 200. All done after 9 iterations.

---

Iteration 10 — 2026-04-12

Stage: RUNNING but "Run Benchmark" throws OSError
Error: OSError: Could not load this library: /usr/local/lib/python3.10/site-packages/torchaudio/lib/_torchaudio.abi3.so
Root cause (via runtime logs): qwen-tts depends on torchaudio. pip install qwen-tts upgraded torchaudio to the latest PyPI release which was compiled against CUDA 13 (libcudart.so.13). ZeroGPU A10G runs CUDA 12, so libcudart.so.13 is not present. Full import chain: from qwen_tts import Qwen3TTSModel → speech_vq.py → import torchaudio.compliance.kaldi → torchaudio/__init__.py → torchaudio._extension → torch.ops.load_library("_torchaudio.abi3.so") → OSError: libcudart.so.13.
Fix applied: Pinned torchaudio==2.5.1 in requirements.txt BEFORE the qwen-tts line. torchaudio 2.5.1 (Nov 2024) was compiled against CUDA 12 and prevents pip from upgrading to a CUDA-13 version. kaldi.fbank() (the only torchaudio function qwen-tts calls from this path) is a CPU-only DSP operation — no GPU needed.
Result: PASS ✓ — Space RUNNING with new SHA 990b408, /health_hf → 200. Benchmark fix deployed.

---

Iteration 11 — 2026-04-12

Stage: RUNNING — benchmark redesign (not a build error) Change: Replaced static np.zeros reference + arbitrary test text with a live microphone enrollment simulation. New UI: user records one of the 3 frontend enrollment phrases via Gradio Audio input, benchmark clones their voice and synthesizes an AI response ("Great job! Now let's keep the conversation going. How was your day?"), returns RTF result + playable audio output. Mirrors the actual frontend UX: enroll → clone → hear AI response. Files changed: app.py only. Result: FAIL — space RUNNING, /health_hf 200, but Gradio API returns 500 Internal Server Error. UI loads but "Start →" button fails. See Iteration 12.

---

Iteration 12 — 2026-04-13

Stage: RUNNING but Gradio API /gradio_api/info returns 500 Internal Server Error
Error: File "/usr/local/lib/python3.10/site-packages/gradio_client/utils.py", line 967, in _json_schema_to_python_type — crash during API schema generation
Root cause (via runtime logs): Gradio generates a JSON schema for all function signatures when serving /gradio_api/info. The gpu_chat_turn function had type hints ref: np.ndarray, history: list, turn_count: int, l1: str, l2: str. gradio_client's json_schema_to_python_type in _json_schema_to_python_type cannot serialize numpy.ndarray into a JSON schema — it crashes on the list comprehension at line 967–968 trying to build property descriptions. This crash propagates through Starlette's middleware stack, resulting in a 500 on every request (including the frontend's queue/event polling calls).
Fix applied:

• Removed all type hints from gpu_enroll_and_greet and gpu_chat_turn signatures — Gradio's schema generator only inspects annotated parameters
• Changed gpu_enroll_and_greet to return ref.tolist() (plain Python list) instead of np.ndarray — keeps State JSON-serializable
• Changed gpu_chat_turn to accept ref_list (plain list) and convert to np.ndarray internally via np.array(ref_list, dtype=np.float32) before passing to synthesize()
• No changes to callbacks — on_enroll stores whatever the function returns; on_send passes it through unchanged
  Files changed: app.py only.
  Result: FAIL — same crash persists. Removing np.ndarray type hints did not resolve it. Root cause was actually the gr.State(dict) itself, not the function signature. See Iteration 13.

---

Iteration 13 — 2026-04-13

Stage: RUNNING but /gradio_api/info still returns 500
Error: TypeError: argument of type 'bool' is not iterable at gradio_client/utils.py:882 → get_type → if "const" in schema
Root cause: Removing np.ndarray type hints in Iteration 12 did not fix the crash. The actual source is gr.State({"l1": "en", "l2": "es", "ref": None, "history": [], "turn_count": 0}). When Gradio generates the API schema for this State, it calls _json_schema_to_python_type on the dict schema. The dict's JSON Schema representation has additionalProperties: True (a Python bool, per JSON Schema spec). The schema generator then does if "const" in schema where schema is already a Python bool True, causing TypeError: argument of type 'bool' is not iterable. This happens in gradio_client/utils.py at line 882 regardless of function type hints — it's triggered by the State type itself.
Fix applied: Replaced single gr.State(dict) with 5 flat, primitive gr.State objects:

• state_l1 = gr.State("en") — string, safe
• state_l2 = gr.State("es") — string, safe
• state_ref = gr.State([]) — empty list (no numpy), safe
• state_history = gr.State([]) — list of dicts (plain JSON), safe
• state_turn_count = gr.State(0) — int, safe All callbacks updated to accept/return these flat states. ref_list (a Python list) is passed as state_ref and converted to np.ndarray inside gpu_chat_turn only. Full app.py rewrite.
  Files changed: app.py only.
  Result: PASS ✓ — Space RUNNING, Gradio UI fully functional (language select → enrollment → chat → wall at turn 7), /health_hf → 200. See session 2026-04-13 for subsequent full-backend migration.

---

Iteration 14 — 2026-04-13 (session 2)

Stage: Full backend migration attempt — gr.mount_gradio_app() approach
Goal: Serve FastAPI REST API (all /session/* endpoints) alongside Gradio UI so the Vercel React frontend can talk directly to the HF Space instead of Railway.
Approach: Replaced demo.launch() with app = gr.mount_gradio_app(api, demo, path="/ui") where api is a standalone FastAPI() instance with all endpoints defined as routes.
Error: RUNTIME_ERROR — Space exits with code 0 (clean exit).
Root cause: HF Spaces with sdk: gradio require demo.launch() to start and block the server. gr.mount_gradio_app() returns an ASGI app object but does not start a server — same as Iteration 4 (the process runs to completion and exits immediately).
Fix applied: See Iteration 15.

---

Iteration 15 — 2026-04-13 (session 2)

Stage: RUNTIME_ERROR — Space exits code 0 after gr.mount_gradio_app()
Approach: Switched to include_router() pattern: patched gradio.routes.App.create_app to call gapp.include_router(_vmr) (adding all API routes to Gradio's internal FastAPI app), then ended with demo.launch() to keep the process alive.
Error: GET /health → HTTP 308 Permanent Redirect (location: /). All API routes return 308.
Root cause: Gradio 5.x registers a catch-all SPA route /{path_name:path} during create_app. FastAPI matches routes in insertion order — the catch-all is registered first (inside Gradio's own create_app logic), so any routes added afterward via include_router() are never matched. Every unknown path gets 308-redirected to / before our routes are evaluated.
Key lesson: include_router() appends routes AFTER the catch-all — they will never be reached in Gradio 5.x.
Fix applied: See Iteration 16.

---

Iteration 16 — 2026-04-13 (session 2)

Stage: RUNNING but all API routes return 308 via include_router()
Root cause: Same Gradio 5.x SPA catch-all issue as Iteration 9 but for custom routes instead of /health_hf. include_router() is append-only and cannot insert before the catch-all.
Fix applied: Implemented all REST API endpoints as a single BaseHTTPMiddleware subclass (_VocalMirrorAPI) with regex-based path dispatch. Middleware runs BEFORE any route matching (same pattern that fixed /health_hf in Iteration 9). demo.launch() stays bare. Session state in-memory dict, audio in /tmp/, background thread for enrollment, asyncio.run_in_executor for gpu_tts() from async context.
Result: PASS ✓

• GET /health → {"status":"ok"} HTTP 200 ✓
• GET /vm-config → {"wall_turn_count":7} ✓ (named /vm-config to avoid shadowing Gradio's own /config)
• POST /session/start → returns session_id + word_list ✓
• GET /session/{id}/wall_status → {"show_wall":false,"turn_count":0} ✓
• Gradio product UI at / still fully functional ✓
  Space is RUNNING on zero-a10g with SHA ef1ba6a.