docs/deployment-issues.md

# Deployment Issues & Fixes — Self-Healing Code Agent

> Reusable knowledge base built from deploying this project to HuggingFace Spaces.
> Each issue is a standalone entry covering the exact error, root cause, fix, and prevention.
> Designed to be consumed by humans and LLM agents on future builds.

---

## Issue Index

| # | Error / Symptom | Where | Status |
|---|-----------------|--------|--------|
| 1 | `sync_to_hf.yml` never triggered | GitHub Actions | Fixed |
| 2 | `short_description` over 60 chars | HF Spaces README frontmatter | Fixed |
| 3 | `ModuleNotFoundError: No module named 'audioop'` | HF Spaces runtime (Python 3.13) | Fixed |
| 4 | `DeprecationWarning: Use dtype instead of torch_dtype` | HuggingFace Transformers | Fixed |
| 5 | `RuntimeError: cannot schedule new futures after interpreter shutdown` | app.py startup (daemon thread) | Fixed |
| 6 | `DeprecationWarning` / `RuntimeError` on `asyncio.get_event_loop()` | Python 3.13 + async code | Fixed |
| 7 | `TypeError: Blocks.launch() got an unexpected keyword argument 'theme'` | Gradio 5→6 migration | Fixed |
| 8 | "Connection errored out" — Space running but unreachable | HF Spaces reverse proxy | Fixed |
| 9 | `TypeError: 'NoneType' object does not support the asynchronous context manager protocol` | Gradio 5 queue internals | Fixed |
| 10 | `safe_get_lock()` returns `None` (root cause of #9) | Gradio 5.x + Python 3.13 | Fixed → upgrade to Gradio 6 |
| 11 | `JSON parse failed: Expecting ':' delimiter: line 1 column 455` | `llm/schema_validator.py` | Fixed |

---

## Issue 1: GitHub Actions Workflow in Wrong Folder

**Error / Symptom:**
```
Workflow never triggered on push. HF Space never received new code.
```

**Where it occurred:** GitHub Actions / repository root

**Root cause:** `sync_to_hf.yml` was placed in the repository root. GitHub Actions only reads workflow files from `.github/workflows/` — files anywhere else are silently ignored.

**Why it happens in LLM agent systems:** AI code generation tools and scaffolding assistants frequently emit files to the project root. Without explicit path awareness, the workflow file ends up in the wrong location with no error feedback.

**Fix applied:**
```bash
# Move file to correct location
mkdir -p .github/workflows
git mv sync_to_hf.yml .github/workflows/sync_to_hf.yml
git commit -m "fix: move sync workflow to correct .github/workflows/ path"
git push
```

**Prevention for future builds:** Always scaffold `.github/workflows/` at project initialization. Add a sanity check: `ls .github/workflows/` before assuming CI is wired up.

---

## Issue 2: HF Spaces `short_description` Over 60 Characters

**Error / Symptom:**
```
ERROR: Validation error in README frontmatter: short_description exceeds maximum length (60 characters)
```

**Where it occurred:** HuggingFace Spaces — push rejected at metadata validation step

**Root cause:** The HF Spaces `short_description` frontmatter field has a hard 60-character limit enforced server-side. Longer values cause the entire push to be rejected.

**Why it happens in LLM agent systems:** Agents tend to write descriptive short descriptions. Without a character count check, it's easy to write a 70–90 char description that reads well but fails validation silently until push time.

**Fix applied:**
```yaml
# Before (73 chars — rejected)
short_description: Autonomous agent that generates, tests, diagnoses, and self-heals Python code.

# After (52 chars — accepted)
short_description: Autonomous agent that self-heals Python code errors.
```

**Prevention for future builds:** Keep `short_description` to one short clause. Count characters before committing: `echo -n "your description" | wc -c`.

---

## Issue 3: `audioop` Missing — Gradio 4 on Python 3.13

**Error / Symptom:**
```
ModuleNotFoundError: No module named 'audioop'
  File "...pydub/utils.py", line 11, in <module>
    import audioop
```

**Where it occurred:** HuggingFace Spaces runtime on startup (Python 3.13)

**Root cause:** Python 3.13 removed the `audioop` C extension from the standard library. Gradio 4.x depended on `pydub`, which imports `audioop` unconditionally at module load time. The entire Gradio import chain fails.

**Why it happens in LLM agent systems:** HF Spaces periodically updates its default Python version. A pinned old Gradio version that worked at project start can break silently when the runtime is updated. LLM-generated `requirements.txt` files often pin whatever was current at generation time.

**Fix applied:**
```
# requirements.txt
# Before
gradio==4.19.2

# After
gradio==6.6.0
```

Also update `README.md` frontmatter:
```yaml
sdk_version: 6.6.0
```

**Prevention for future builds:** Always pin `gradio>=5.0.0` for Python 3.13+ environments. Check the HF Spaces default Python version before choosing a Gradio version. Prefer Gradio 6+ for new projects.

---

## Issue 4: `torch_dtype` Deprecation in HuggingFace Transformers

**Error / Symptom:**
```
FutureWarning: `torch_dtype` is deprecated. Use `dtype` instead.
```

**Where it occurred:** `llm/providers/hf_provider.py` — `pipeline()` call in `_load_pipeline()`

**Root cause:** The `torch_dtype` keyword argument in `transformers.pipeline()` was renamed to `dtype` in Transformers 4.40+. The old argument still worked but emitted a deprecation warning, which in strict environments would raise.

**Why it happens in LLM agent systems:** Agent provider code is often written against older Transformers docs or examples. The rename is silent until warnings are surfaced in logs.

**Fix applied:**
```python
# Before
kwargs = {
    "model": self._model_id,
    "device_map": self._device_map,
    "torch_dtype": torch.float16,
}

# After
kwargs = {
    "model": self._model_id,
    "device_map": self._device_map,
    "dtype": torch.float16,
}
```

**Prevention for future builds:** Use `dtype=` in all new `pipeline()` calls. When upgrading `transformers`, search for `torch_dtype` in your codebase and rename.

---

## Issue 5: Daemon Thread Killed Mid-Download

**Error / Symptom:**
```
RuntimeError: cannot schedule new futures after interpreter shutdown
  File ".../concurrent/futures/thread.py", line 169, in submit
```

**Where it occurred:** `app.py` startup — model pre-warm code running in a daemon thread

**Root cause:** A daemon thread was spawned to pre-download model weights in the background while Gradio started. Python's interpreter shutdown sequence kills all daemon threads when the main thread exits its setup phase. The `ThreadPoolExecutor` inside the HuggingFace provider's `_ensure_loaded()` call was mid-execution when it was killed.

**Why it happens in LLM agent systems:** Model pre-warming is a standard pattern for reducing first-request latency. Daemon threads feel like the right tool (background, non-blocking) but are wrong here — the download *must* complete before the server accepts requests.

**Fix applied:**
```python
# Before — daemon thread (wrong)
import threading
t = threading.Thread(target=lambda: asyncio.run(_prewarm()), daemon=True)
t.start()

# After — synchronous blocking call before launch() (correct)
def _prewarm() -> None:
    try:
        from llm.router import LLMRouter
        router = LLMRouter()
        if hasattr(router.provider, "_ensure_loaded"):
            asyncio.run(router.provider._ensure_loaded())
    except Exception as exc:
        logger.warning("Pre-warm failed (non-fatal): %s", exc)

_prewarm()  # blocks until complete
demo = build_app()
demo.launch(...)
```

**Prevention for future builds:** Never pre-warm in daemon threads. Pre-warm must finish before `launch()` is called. Wrap in `try/except` so a pre-warm failure doesn't crash the server.

---

## Issue 6: `asyncio.get_event_loop()` Deprecated on Python 3.13

**Error / Symptom:**
```
DeprecationWarning: There is no current event loop
RuntimeError: no running event loop
```

**Where it occurred:** `llm/providers/hf_provider.py`, `framework/event_bus.py` — any call site using `asyncio.get_event_loop()` outside an async context

**Root cause:** Python 3.10 deprecated `asyncio.get_event_loop()` when called with no running loop (it used to silently create one). Python 3.12+ promotes this to a `DeprecationWarning`. Python 3.13 can raise `RuntimeError` in some call sites. Code written against Python 3.9 patterns breaks.

**Why it happens in LLM agent systems:** LLM agent frameworks make heavy use of `asyncio`. Boilerplate code from tutorials, Stack Overflow, and AI assistants frequently uses the older `get_event_loop()` pattern. The breakage is Python-version dependent, making it easy to miss in testing.

**Fix applied:**
```python
# Before (hf_provider.py — inside async function)
loop = asyncio.get_event_loop()
result = await loop.run_in_executor(None, self._load_pipeline)

# After
loop = asyncio.get_running_loop()  # safe inside async context
result = await loop.run_in_executor(None, self._load_pipeline)

# Before (event_bus.py — sync wrapper)
loop = asyncio.get_event_loop()
loop.run_until_complete(bus.emit(event))

# After
def emit_sync(bus, event):
    try:
        loop = asyncio.get_running_loop()
        loop.create_task(bus.emit(event))
    except RuntimeError:
        asyncio.run(bus.emit(event))
```

**Prevention for future builds:**
- Inside `async def`: use `asyncio.get_running_loop()` — always safe, always correct
- At sync entry points: use `asyncio.run(coroutine)` — creates and manages its own loop
- Never use `asyncio.get_event_loop()` in new code

---

## Issue 7: `theme=` Removed from `gr.Blocks()` / `launch()` in Gradio 5→6

**Error / Symptom:**
```
# Gradio 5
UserWarning: The `theme` and `css` parameters have been moved to `launch()`.

# Gradio 6
TypeError: Blocks.launch() got an unexpected keyword argument 'theme'
```

**Where it occurred:** `demo/app.py` and `app.py` — `gr.Blocks()` and `demo.launch()` calls

**Root cause:** Gradio 5 moved `theme` and `css` from `gr.Blocks()` to `launch()`. Gradio 6 then removed `theme` from `launch()` entirely — it belongs back in `gr.Blocks()`. The API moved twice across two major versions.

**Why it happens in LLM agent systems:** Gradio is a fast-moving library. AI-generated UI code targets whatever Gradio version was in the training data. The `theme=` argument breakage is particularly deceptive because the error message from Gradio 5 points you in the wrong direction.

**Fix applied (for Gradio 6):**
```python
# Before
with gr.Blocks(theme=gr.themes.Soft(), title="...") as app:
    ...
demo.launch(theme=..., css=...)

# After
css = "..."
with gr.Blocks(title="...", css=css) as app:  # css goes here in Gradio 6
    ...
demo.launch(server_name="0.0.0.0", server_port=int(os.environ.get("PORT", 7860)))
# theme= removed entirely (use default or pass gr.themes.X() to gr.Blocks())
```

**Prevention for future builds:** Check the Gradio changelog for any major version bump. Always run `python -W error app.py` locally to surface deprecation warnings before deploying.

---

## Issue 8: Server Bound to 127.0.0.1 — HF Spaces Proxy Can't Reach It

**Error / Symptom:**
```
# In browser
Connection errored out.

# In Space logs
INFO:     127.0.0.1:XXXXX - "GET / HTTP/1.1" 200 OK
# (Space running and healthy, but browser can't connect)
```

**Where it occurred:** HuggingFace Spaces — the reverse proxy routing external traffic to the container

**Root cause:** `demo.launch()` defaults to binding on `127.0.0.1` (loopback only). HF Spaces routes external HTTP traffic through a reverse proxy that connects to the container on all network interfaces. With loopback-only binding, the proxy's connection attempt is refused — the server is running but unreachable from outside the container.

**Why it happens in LLM agent systems:** Default Gradio `launch()` is designed for local development. Moving to a containerized or proxied environment requires explicit network binding. This is a common "works locally, fails in production" failure.

**Fix applied:**
```python
# Before
demo.launch()

# After
demo.launch(
    server_name="0.0.0.0",  # bind on all interfaces
    server_port=int(os.environ.get("PORT", 7860)),  # respect HF Spaces PORT env var
)
```

**Prevention for future builds:** Always use `server_name="0.0.0.0"` in any containerized, proxied, or cloud deployment. Always read `PORT` from the environment. Make this a template in your app.py boilerplate.

---

## Issue 9: `pending_message_lock` is `None` — Gradio 5 Queue Crash

**Error / Symptom:**
```
TypeError: 'NoneType' object does not support the asynchronous context manager protocol
  File ".../gradio/queueing.py", line XXX, in push
    async with self.pending_message_lock:
```

**Where it occurred:** Gradio 5 queue internals — triggered on every user interaction (button click, form submit)

**Root cause:** See Issue 10 for the underlying source bug. The symptom: `self.pending_message_lock` inside the Gradio queue is `None` instead of an `asyncio.Lock`. Every interaction that touches the queue crashes immediately with this `TypeError`.

**Why it happens in LLM agent systems:** Streaming agent output through Gradio's queue is the standard pattern for real-time UI updates. The queue being silently broken is catastrophic — every interaction fails, but the Space shows as healthy.

**Fix applied at the time:** Adding `demo.queue()` before `demo.launch()` partially initialized the lock. However this was not a complete fix — the root cause (Issue 10) required upgrading to Gradio 6.

**Permanent fix:** Upgrade to `gradio==6.6.0` (see Issue 10).

**Prevention for future builds:** Use Gradio 6+ with Python 3.13. Always call `demo.queue()` before `demo.launch()` for streaming apps. Test the queue explicitly by sending a request and verifying the streaming response arrives.

---

## Issue 10: `safe_get_lock()` Returns `None` — Root Cause of Issue 9

**Error / Symptom:** (Same as Issue 9 — this is the underlying source bug)

**Where it occurred:** `gradio/utils.py` → `safe_get_lock()` — called at Gradio import time in Python 3.13

**Root cause:**
```python
# Gradio 5.x source (simplified)
def safe_get_lock():
    try:
        loop = asyncio.get_event_loop()  # raises DeprecationWarning / returns None on Python 3.13
        return asyncio.Lock()
    except RuntimeError:
        return None  # <-- this None becomes pending_message_lock
```

On Python 3.13, `asyncio.get_event_loop()` with no running loop returns `None` or raises. The `except RuntimeError` catches it and returns `None`. This `None` is stored as `pending_message_lock` for the lifetime of the process.

**Fix in Gradio 6.0:**
```python
# Gradio 6.x source (simplified)
def safe_get_lock():
    try:
        loop = asyncio.get_running_loop()
    except RuntimeError:
        loop = asyncio.new_event_loop()
    return asyncio.Lock()
```

**Fix applied:** Upgrade `requirements.txt`:
```
gradio==6.6.0
```

Update `README.md` frontmatter:
```yaml
sdk_version: 6.6.0
```

**Prevention for future builds:** Pin `gradio>=6.0.0` for any deployment on Python 3.13. This is a hard compatibility boundary — Gradio 4/5 + Python 3.13 = broken queue with no obvious error at startup.

---

## Issue 11: JSON Parse Failure from Truncated LLM Output

**Error / Symptom:**
```
[ERROR] Agent encountered an error: JSON parse failed: Expecting ':' delimiter: line 1 column 455
```

**Where it occurred:** `llm/schema_validator.py` — during structured output parsing after every LLM inference call

**Root cause:** The generator agent is prompted to return a JSON object containing a `"code"` field with the full Python source. With `max_new_tokens=1024`, the 3B model runs out of token budget mid-output — the JSON is cut off partway through the `"code"` string value. The `json.loads()` call fails on the truncated text.

The token budget math: A 200-line Python solution is ~3,000 characters of source. JSON-encoding it (escaping newlines as `\n`, quotes as `\"`, etc.) roughly doubles the character count to ~6,000 characters. At ~4 chars/token that's ~1,500 tokens — already over the 1,024 limit before any JSON wrapper overhead.

**Why it happens in LLM agent systems:** Structured output (JSON-wrapped code or tool calls) is fundamentally longer than raw text generation. Token limits tuned for conversational text are systematically too small for code-generation agents. This failure is silent until it hits production — the model generates what it can, the JSON truncates, the parser fails.

**Fix applied:**

1. Raise `max_new_tokens` across all agent nodes:
```python
# agent/nodes/generate_solution.py
max_new_tokens=2048  # was 1024

# agent/nodes/create_adversarial_tests.py
max_new_tokens=1024  # was 768

# agent/nodes/diagnose_failure.py
max_new_tokens=768   # was 512
```

2. Add salvage fallback in `llm/schema_validator.py`:
```python
def _salvage_code_field(text: str) -> dict[str, Any] | None:
    """Extract code field from truncated JSON using regex as last resort."""
    pattern = re.compile(r'"code"\s*:\s*"((?:[^"\\]|\\.)*)', re.DOTALL)
    match = pattern.search(text)
    if match:
        raw_code = match.group(1)
        try:
            code = raw_code.encode("utf-8").decode("unicode_escape")
        except Exception:
            code = raw_code
        if code.strip():
            return {"code": code}
    return None

def parse_and_validate(text: str, schema: dict) -> dict:
    try:
        parsed = json.loads(text)
    except json.JSONDecodeError:
        salvaged = _salvage_code_field(text)
        if salvaged:
            return salvaged
        raise StructuredOutputError(f"JSON parse failed: {e}")
    ...
```

**Prevention for future builds:**
- Set `max_new_tokens` to at least 2× the expected output length for any structured-output agent
- Always implement a salvage/fallback parser for JSON-wrapped outputs — truncation is inevitable on small models with constrained token budgets
- Log token usage per call to detect budget exhaustion early
- Consider using models with native function calling (which handle structured output within the token budget more efficiently)

---

*Last updated: 2025 — Self-Healing Code Agent build log*