docs/manual-gpu-validation.md

Manual GPU Validation for the First Observation Flow

Use this checklist on the target local GPU machine to validate the complete thin Catbox flow. The goal is to prove that the persistent Model Backend reaches readiness, both Recognizable Outcomes can be generated through Dev Controls, and the normal Browser UI can observe and reveal a real Generated Outcome.

Start the Model Backend and Browser UI

The current local product path starts the persistent Model Backend and Browser UI in one process:

uv run python -m catbox.browser_ui

Open the Browser UI at:

http://127.0.0.1:8765

If you want to verify readiness from a second terminal before observing:

curl http://127.0.0.1:8765/api/readiness

Wait for:

{"status": "ready", "modelBackend": "ready"}

GPU execution may require escalated local commands, driver access, CUDA device access, or model cache writes. Do not require HF_TOKEN for this validation unless the user explicitly provides one.

Run a Normal Observation

In the Browser UI, click Observe after readiness. The normal Browser UI path must not choose the outcome; it sends the normal observation request to the Model Backend.

The same public request can be run from a second terminal:

curl -X POST http://127.0.0.1:8765/api/observe

Expected response:

• status is generated.
• outcome is either living or dead.
• imageRef points at a real local PNG file.
• traceRefs contains zero or more real Captured Denoising Trace frame paths.
• metadata.traceFrameCount matches the number of returned trace frames when trace capture is supported by the runner.
• metadata.generationSeconds records generation timing after backend readiness.
• metadata.ephemeral is true.

Compare metadata.generationSeconds against the Primary Runtime Target: the preferred GPU path should produce a Recognizable Outcome in under 23 seconds. Record the observed time in the issue or validation notes.

While the Browser UI is observing, the trace endpoint can be checked from a second terminal:

curl http://127.0.0.1:8765/api/trace

When SD Turbo trace capture is working, the response should show the selected branch's captured frame paths as they are registered. It should not show a fake dual-branch trace.

Force Both Outcomes Through Dev Controls

Use the development-only validation command to force both outcomes with a stable seed:

uv run python -m catbox.validate_sd_turbo_runner --outcome all --seed 41100

To validate only one outcome:

uv run python -m catbox.validate_sd_turbo_runner --outcome living --seed 41100
uv run python -m catbox.validate_sd_turbo_runner --outcome dead --seed 41100

To validate one explicit tuning candidate:

uv run python -m catbox.validate_sd_turbo_runner --outcome dead --seed 41100 --steps 6 --strength 0.7 --width 512 --height 512

To run the first Outcome Visibility matrix on the deployed GPU runtime:

uv run python -m catbox.validate_sd_turbo_runner --matrix --seed 41100

The default matrix tries both outcomes with steps 4,6,8, sizes 512,768, Living-Cat Outcome strengths 0.75,0.8,0.85, and Dead-Cat Outcome strengths 0.6,0.7,0.8. These values can be narrowed for a smaller run:

uv run python -m catbox.validate_sd_turbo_runner --matrix --seed 41100 --matrix-steps 4,6 --matrix-sizes 512 --living-strengths 0.75 --dead-strengths 0.6,0.7

Expected response:

• The first entry reports backend readiness.
• The Living-Cat Outcome entry has status: "generated" and outcome: "living".
• The Dead-Cat Outcome entry has status: "generated" and outcome: "dead".
• Each generated entry includes metadata.generationSeconds.

Compare each forced outcome's metadata.generationSeconds against the under 23 seconds Primary Runtime Target after readiness.

A matrix candidate passes when the final Living-Cat Outcome is immediately recognizable, the final Dead-Cat Outcome is immediately recognizable and non-graphic, both still feel anchored to the shared Box Composition, and metadata.generationSeconds stays under the Primary Runtime Target. Prefer the fastest passing candidate; if timings are close, choose the candidate with clearer outcome identity.

The current preferred SD Turbo settings are outcome-specific and may be changed through Dev Controls during validation. Outcome Visibility uses one shared standard for both outcomes, but each outcome may use different tuning values. The current tuned defaults use 512px image-to-image generation with 6 steps for both outcomes. The Living-Cat Outcome uses strength 0.78, and the Dead-Cat Outcome uses strength 0.7.

Ephemeral Generated Outcomes

Generated files are written under:

.runtime/generated-outcomes

Confirm that each imageRef points to a file in that runtime output area. These are Ephemeral Outcomes for local display and debugging only. This validation must confirm no gallery/history/save/share behavior was introduced.

Captured Denoising Trace frames are also Ephemeral Outcomes and may be written under a trace subdirectory in the runtime output area.

Generation Failure, Retry, and Reset

If the Model Backend cannot produce a Generated Outcome, the Browser UI should show Generation Failure instead of substituting a static image or fake generated file.

Expected failure behavior:

• The observation response has status: "generation_failed".
• The response does not include imageRef.
• The Browser UI shows the failure message.
• Retry runs the normal POST /api/observe path again.
• Reset clears the failure state and returns to the sealed box.

Exercise the retry/reset behavior by using the Browser UI after a real local generation failure, such as unavailable model weights, CUDA out of memory, or an unready runner. The browser-facing failure contract can also be checked without GPU access:

uv run python -m unittest tests.test_browser_ui.BrowserUiServerTests.test_generation_failure_returns_structured_failure_without_generated_image