ser-inference / docs /ROLLOUT.md
spacey-cadet's picture
docs: overhaul README and Dockerfile for production pipeline
dddb6a8
|
Raw History Blame Contribute Delete
4.64 kB

Rollout Playbook

Three approximations of real rollout mechanisms, all buildable on free-tier infrastructure. Each is a genuine approximation of a specific real mechanism β€” worth knowing what the paid version adds so you're not surprised later.

1. In-process canary (default recommendation for a single Space)

Mechanism: both champion and challenger checkpoints loaded in the same FastAPI process; a random percentage of requests routes to the challenger.

How to use it:

  1. Push the new checkpoint to the Hub under a new revision/tag.
  2. Set in .env (or Space secrets):
    CHALLENGER_MODEL_ID=<repo>          # usually same as MODEL_ID
    CHALLENGER_REVISION=<new tag>
    CANARY_PCT=10
    
  3. Restart the Space. app/model/registry.py loads both; app/routing/canary.py routes CANARY_PCT% of traffic to the challenger.
  4. Every /predict response includes "is_challenger": true/false and "model_revision" β€” filter your feature log / dashboards on this to compare champion vs. challenger outcomes on identical live traffic.
  5. Step up manually: 10 β†’ 25 β†’ 50 β†’ 100, checking confidence distribution and cascade-tier mix at each step. Drop back to CANARY_PCT=0 immediately on any anomaly (confidence collapse, latency spike, error rate).
  6. Once at 100% and stable for a chosen soak period, promote: set MODEL_ID's default revision or CHALLENGER_REVISION as the new main, drop CHALLENGER_MODEL_ID, redeploy.

What a real load balancer adds: true percentage-based routing at the network layer (not per-process randomness), automatic SLA-triggered rollback without a human watching dashboards, and routing that doesn't require both models resident in the same process's memory (relevant once a challenger checkpoint is large enough that loading two at once strains the free tier's RAM).

2. Blue-green (two Spaces)

Mechanism: a staging Space and a production Space; cut over by pointing consumers at the new Space's URL; rollback is pointing back.

How to use it:

  1. Duplicate the Space (HF Spaces supports this natively) as <name>-staging.
  2. Deploy the challenger to staging, run your test suite (pytest tests/) and a manual smoke test against it directly.
  3. Update the chat application's configured inference URL to the staging Space once satisfied.
  4. Keep the old production Space running, untouched, for a rollback window β€” pointing the client config back is the entire rollback procedure.

What a real blue-green setup adds: automated traffic cutover (DNS/load balancer level, not a client config change), health-check-gated automatic cutover, and zero-downtime cutover for high-QPS services (a client config change has a brief window where in-flight requests may hit the old URL during redeploy).

3. Shadow deployment

Mechanism: a second free Space running the challenger, mirroring a sample of real requests, logging output without ever returning it to the user. Useful specifically for validating latency under real chat traffic before a challenger goes anywhere near production.

How to use it:

  1. Deploy the challenger to a separate Space (<name>-shadow).
  2. In the chat application (not this repo β€” this is a client-side integration), fire a sampled fraction of real requests to both the production Space and the shadow Space, but only use the production Space's response.
  3. Compare latency and confidence-tier distributions between the two logs. This is the cleanest way to validate scripts/benchmark_latency.py-style numbers under real traffic shape (codec mix, duration distribution) rather than synthetic sample clips.

What this doesn't give you: shadow traffic never influences what the user sees, so it can't validate anything about response-system behavior β€” only model-level latency and prediction distribution.

Choosing between them

  • In-process canary β€” cheapest, fastest to iterate, good default for comparing model quality on live traffic. Downside: both checkpoints share the same process's CPU/RAM.
  • Blue-green β€” best when you want a hard, instant rollback path and don't need gradual traffic ramp.
  • Shadow β€” best for latency/capacity validation before a challenger is trusted with any live traffic at all, especially after an architecture change (quantization, ONNX export, trimmed layers per Track 3.4).

In practice: shadow first (validate latency doesn't regress) β†’ in-process canary (validate prediction quality on live traffic, ramp gradually) β†’ blue-green as the final cutover once canary is at 100% and stable.