docs/ssh-runbook.md

# Hugging Face Space SSH Runbook

This runbook records how to connect to the running Hugging Face Space container
for debugging.

## Space

```text
https://huggingface.co/spaces/build-small-hackathon/VoiceGate
```

SSH target:

```text
build-small-hackathon-voicegate@ssh.hf.space
```

## Local SSH Key

The local private key generated for this Space is:

```text
C:\Users\yantianlong\.ssh\codex_space_voicegate
```

The public key is:

```text
C:\Users\yantianlong\.ssh\codex_space_voicegate.pub
```

Only the `.pub` file content should be added to Hugging Face. Never upload or
share the private key file.

## Connect

From PowerShell:

```powershell
ssh -i "$env:USERPROFILE\.ssh\codex_space_voicegate" build-small-hackathon-voicegate@ssh.hf.space
```

Non-interactive smoke test:

```powershell
ssh -i "$env:USERPROFILE\.ssh\codex_space_voicegate" `
  -o BatchMode=yes `
  -o StrictHostKeyChecking=accept-new `
  -o ConnectTimeout=20 `
  build-small-hackathon-voicegate@ssh.hf.space `
  "pwd && git rev-parse HEAD && ls -la"
```

Expected working directory:

```text
/home/user/app
```

## Important Runtime Caveat

SSH connects to the currently running Space container. That container may be
stale after a git push.

On 2026-06-05, SSH connected successfully, but the running container was still
at the original template commit:

```text
a94117f35a42cb17f654ae70cbe619c15345d057
```

The Space git remote had already advanced to:

```text
7acf781e730d4b84d40cf3c688a1575b11bd90aa
```

Before debugging runtime behavior through SSH, always verify the container
commit:

```bash
git rev-parse HEAD
```

If the container commit does not match the latest Space commit, use the
Hugging Face Space page to restart/rebuild/factory reboot the Space, then SSH
again and re-check.

## What SSH Is Good For

- Inspect the running app files.
- Check build/runtime environment differences.
- Run small diagnostic commands.
- Inspect installed packages and model cache paths.
- Debug ComfyUI startup logs once Phase 3 is implemented.
- Run ComfyUI in CPU mode to verify the local API can start:

```bash
cd /home/user/app
python scripts/bootstrap_comfy.py
python scripts/run_comfy.py --cpu
```

In another SSH command, check:

```bash
curl http://127.0.0.1:8188/system_stats
```

## What SSH Is Not For

Do not rely on SSH edits for persistent code changes. Files changed inside the
running container can be lost after rebuild or restart.

Persistent changes should be made locally, committed, and pushed to the Space
git repository.

SSH also does not expose the ZeroGPU CUDA device in the normal shell. On
2026-06-05, starting ComfyUI without `--cpu` through SSH failed with:

```text
RuntimeError: No CUDA GPUs are available
```

GPU-backed inference should be triggered from the Gradio app through a
`@spaces.GPU` function. SSH remains useful for CPU startup and API diagnostics.

## Runtime Findings

On 2026-06-05, `python scripts/bootstrap_comfy.py` completed successfully in
the Space container. CPU-mode ComfyUI reached `/system_stats`, but
`ComfyUI_AudioTools` required additional dependencies:

- system package `sox`
- system package `libportaudio2`
- system package `portaudio19-dev`
- Python package `sounddevice`
- Python package `easydict`
- Python package `pytorch-lightning`

These are now recorded in `packages.txt` and `requirements.txt`; a Space rebuild
is required before the system packages are available in the container.

After adding these dependencies, CPU-mode ComfyUI reached `/system_stats` and
the custom node import log showed:

```text
0.4 seconds: /home/user/app/ComfyUI/custom_nodes/ComfyUI_AudioTools
```

with no `IMPORT FAILED` entry for `ComfyUI_AudioTools`.

On 2026-06-05, the ComfyUI API smoke test passed in Dev Mode using a local MP3
uploaded to `/tmp/voicegate_test_audio.mp3`:

- `/upload/image` accepted the MP3 and returned `voicegate_test_audio.mp3`.
- A minimal `LoadAudio -> SaveAudioMP3` workflow submitted through `/prompt`
  returned no `node_errors`.
- `/history/{prompt_id}` returned `status_str: success`.
- The output was reported as `audio/api_smoke_voicegate_00001.mp3`.

When uploading local binary files through SSH, avoid plain PowerShell byte
pipelines because they can corrupt binary data. Use a binary-safe method such as
base64 from Python and verify `sha256sum` on the remote file.