# LiteLLM

The router as the classifier of LiteLLM's Auto Router. The auto router (`auto_router/complexity_router`) sorts each
request into a tier and sends it to the model you put in that tier. The classifier is
[s1d1-tantive](https://github.com/scombat/s1d1-tantive), a container image that serves the router: it answers the tier
in a few milliseconds, with no LLM call. Keys, budgets, logs and fallbacks stay in LiteLLM.

*Status: the `config.yaml` below was tested locally on 2026-10-25 with LiteLLM 1.106.4 against the hosted s1d1 API (then on s1d1-tantive 0.8.0).
The admin-UI pitfalls of section 5 come from an earlier run on LiteLLM 1.103.2 with real
opencode traffic.*

The maintained reference is [s1d1-tantive docs/litellm.md](https://github.com/scombat/s1d1-tantive/blob/main/docs/litellm.md)
(admin GUI, `curl` registration, Test Routing); this page keeps the short path and what went wrong once.

## 1. Run the service

```bash
docker run --rm -p 8788:8788 ghcr.io/scombat/s1d1-tantive:0.8.1-small-onnx-int8
curl localhost:8788/ready   # {"ready": true} once loaded and warmed up (a few seconds); 503 before
```

One image per variant (`full`, `compact`, `small`, `tiny`); the weights are inside, nothing is downloaded at start. The
`small` one takes a few milliseconds per request on a CPU. Multi-arch, signed. Since 0.8.1 the int8 weights are the
`reduce_range` ones (correction of 2026-10-25: earlier `small` images gave wrong scores on x86 CPUs without VNNI), so use
0.8.1 or later. To try the routing without running anything, the free hosted API <https://s1d1.qoo.gy/api> works as
`api_base` (what you send is kept and used to retrain s1d1; read its terms first).

## 2. Configure LiteLLM (1.106 or later)

Tested with LiteLLM 1.106.6. The service is registered as an ordinary deployment, and the router points at it by name:

```yaml
model_list:
  # ... your four models: cheap, medium, strong, reasoning
  - model_name: s1d1-classifier
    litellm_params:
      model: hosted_vllm/s1d1-small    # the name after the slash is sent and ignored by the service
      api_base: http://HOST:8788       # the service root, LiteLLM appends /v1/systemone
    model_info:
      mode: decisions
  - model_name: s1d1-auto
    litellm_params:
      model: auto_router/complexity_router
      complexity_router_default_model: medium
      complexity_router_config:
        classifier_type: oss_classifier
        opensource_classifier_config:
          deployment_name: s1d1-classifier
          timeout_ms: 1000
        classifier_fallback: default_model
        tiers:
          SIMPLE: cheap
          MEDIUM: medium
          COMPLEX: strong
          REASONING: reasoning
```

Call `s1d1-auto` like any model. If the service is slow, unreachable or answers a name the router does not know, the
request goes to the default model (`classifier_fallback: default_model`): a timeout opens a circuit breaker for 30
seconds, a stopped service falls back on each request. `fallback_tier` and `tier_definitions` are for LiteLLM Enterprise
custom tiers; they are not needed here. On the service, `/metrics` counts `tantive_decisions_total` by tier: that is the
check that the service was asked, since the header `x-litellm-complexity-router-cause` reads `jev_classifier` for any
classifier. Where the admin GUI offers no deployment for the classifier, the fallback is `provider: laya`,
`model: multilingual`, `api_base`, `timeout_ms` in place of `deployment_name` (LiteLLM's `laya` provider accepts only
three model names; the service does not read it); it gave the same tiers on the same `/v1/systemone`.

## 3. The tier-name trap

The router's tier names must be LiteLLM's four (`SIMPLE`, `MEDIUM`, `COMPLEX`, `REASONING`), which the service always
recognises whatever its `S1D1_TIERS`. A name the service does not know sends every request to the fallback: if routing
looks stuck on one model, check the names first. To move the bounds, set them on the service (`S1D1_TIERS`), not in
LiteLLM. Scores above about 0.9 are rare, so the top tier sees little traffic.

## 4. LiteLLM before 1.106

No `oss_classifier` there. Mount the chat adapter (`S1D1_FORMATS=...,chat`), register the service as an
OpenAI-compatible model with API base `http://HOST:8788/v1`, and in the auto router use classification method
**LLM Classifier** with that model, timeout 1000 ms, fallback `heuristic`. Details: s1d1-tantive `docs/formats.md`. This
is the path that was tested on 1.103.2.

## 5. Things that went wrong once (Admin UI)

- **A wildcard model (`provider/*`) cannot be picked as a tier model.** Declare each upstream model explicitly. If the
  upstream is another LiteLLM, its name must not keep your local prefix. Saved credentials avoid retyping the key.
- **Do not rename the standard tiers.** The UI sends its own labels to the classifier; with the four standard tiers
  leave display names empty. Custom names go through *Edit tiers* and must match the service (section 3).
- **"Add tier" fails with `crypto.randomUUID is not a function`** over plain HTTP on an IP address: browsers only
  provide it on HTTPS or `localhost`. Use an SSH tunnel (`ssh -L 4000:localhost:4000 server`, then
  `http://localhost:4000/ui`) or HTTPS.
- **"Add Auto Router" is greyed out with no tooltip.** A disabled button gets no hover; press Enter in the name field
  and the reason appears as an error message.

## 6. Limits

- **Current message only.** The service classifies the last user message, what the router was trained on (single
  prompts, up to 1,024 tokens): a short follow-up such as "yes, go ahead" lands in a low tier.
- **The tier is the only output** that reaches the auto router; task type and tool need are in the service's log.