Spaces:
Sleeping
Download docs/ROLLOUT.md from space-cadet/ser-inference: direct link, hf CLI and curl.
- Browser
- Download file 4.64 kB
-
https://huggingface.co/spaces/space-cadet/ser-inference/resolve/main/docs/ROLLOUT.md
- Command line
-
hf download hf://spaces/space-cadet/ser-inference/docs/ROLLOUT.md
-
curl -L -o ROLLOUT.md https://huggingface.co/spaces/space-cadet/ser-inference/resolve/main/docs/ROLLOUT.md
Rollout Playbook
Three approximations of real rollout mechanisms, all buildable on free-tier infrastructure. Each is a genuine approximation of a specific real mechanism β worth knowing what the paid version adds so you're not surprised later.
1. In-process canary (default recommendation for a single Space)
Mechanism: both champion and challenger checkpoints loaded in the same FastAPI process; a random percentage of requests routes to the challenger.
How to use it:
- Push the new checkpoint to the Hub under a new revision/tag.
- Set in
.env(or Space secrets):CHALLENGER_MODEL_ID=<repo> # usually same as MODEL_ID CHALLENGER_REVISION=<new tag> CANARY_PCT=10 - Restart the Space.
app/model/registry.pyloads both;app/routing/canary.pyroutesCANARY_PCT% of traffic to the challenger. - Every
/predictresponse includes"is_challenger": true/falseand"model_revision"β filter your feature log / dashboards on this to compare champion vs. challenger outcomes on identical live traffic. - Step up manually: 10 β 25 β 50 β 100, checking confidence distribution
and cascade-tier mix at each step. Drop back to
CANARY_PCT=0immediately on any anomaly (confidence collapse, latency spike, error rate). - Once at 100% and stable for a chosen soak period, promote: set
MODEL_ID's default revision orCHALLENGER_REVISIONas the newmain, dropCHALLENGER_MODEL_ID, redeploy.
What a real load balancer adds: true percentage-based routing at the network layer (not per-process randomness), automatic SLA-triggered rollback without a human watching dashboards, and routing that doesn't require both models resident in the same process's memory (relevant once a challenger checkpoint is large enough that loading two at once strains the free tier's RAM).
2. Blue-green (two Spaces)
Mechanism: a staging Space and a production Space; cut over by pointing consumers at the new Space's URL; rollback is pointing back.
How to use it:
- Duplicate the Space (HF Spaces supports this natively) as
<name>-staging. - Deploy the challenger to staging, run your test suite (
pytest tests/) and a manual smoke test against it directly. - Update the chat application's configured inference URL to the staging Space once satisfied.
- Keep the old production Space running, untouched, for a rollback window β pointing the client config back is the entire rollback procedure.
What a real blue-green setup adds: automated traffic cutover (DNS/load balancer level, not a client config change), health-check-gated automatic cutover, and zero-downtime cutover for high-QPS services (a client config change has a brief window where in-flight requests may hit the old URL during redeploy).
3. Shadow deployment
Mechanism: a second free Space running the challenger, mirroring a sample of real requests, logging output without ever returning it to the user. Useful specifically for validating latency under real chat traffic before a challenger goes anywhere near production.
How to use it:
- Deploy the challenger to a separate Space (
<name>-shadow). - In the chat application (not this repo β this is a client-side integration), fire a sampled fraction of real requests to both the production Space and the shadow Space, but only use the production Space's response.
- Compare latency and confidence-tier distributions between the two logs.
This is the cleanest way to validate
scripts/benchmark_latency.py-style numbers under real traffic shape (codec mix, duration distribution) rather than synthetic sample clips.
What this doesn't give you: shadow traffic never influences what the user sees, so it can't validate anything about response-system behavior β only model-level latency and prediction distribution.
Choosing between them
- In-process canary β cheapest, fastest to iterate, good default for comparing model quality on live traffic. Downside: both checkpoints share the same process's CPU/RAM.
- Blue-green β best when you want a hard, instant rollback path and don't need gradual traffic ramp.
- Shadow β best for latency/capacity validation before a challenger is trusted with any live traffic at all, especially after an architecture change (quantization, ONNX export, trimmed layers per Track 3.4).
In practice: shadow first (validate latency doesn't regress) β in-process canary (validate prediction quality on live traffic, ramp gradually) β blue-green as the final cutover once canary is at 100% and stable.