README.md

---
title: Emotion Spectrum API
emoji: "🎭"
colorFrom: pink
colorTo: purple
sdk: docker
app_port: 8000
pinned: true
short_description: DistilBERT emotion classifier — live demo + API
---

# distilbert-emotion-api

A batched, observable, deploy-ready FastAPI inference service that serves the fine-tuned [`LaelaZ/distilbert-emotion`](https://huggingface.co/LaelaZ/distilbert-emotion) classifier — and runs **fully offline** for development, CI, and load testing.

## The problem

Training a model is the easy half. The half that actually ships is everything around it: a typed HTTP contract, input validation, health probes, metrics a dashboard can read, request batching so throughput doesn't fall over under load, a container, and a deploy story. And none of that should require downloading 270 MB of weights (or a GPU, or network access) just to run the tests or demo the API.

This repo is that production layer for an emotion classifier — six emotions (sadness, joy, love, anger, fear, surprise) with full per-class probabilities — built so the entire service, its demo UI, its test suite, and its load test run with **zero downloads** by swapping the model for a deterministic stub when `OFFLINE=1`. Flip `OFFLINE=0` and the same code path loads the real DistilBERT from the Hub.

> **The deployed [Hugging Face Space](https://huggingface.co/spaces/LaelaZ/distilbert-emotion-api) runs the real fine-tuned model** (built `WITH_MODEL=1`, `OFFLINE=0`) — so the public demo serves genuine DistilBERT predictions (acc 0.920 / macro F1 0.874). The lean, torch-free offline stub is what powers CI, local `docker compose`, and the load test, so development stays instant and key-free.

## What it does

- **`POST /predict`** — single (`{"text": ...}`) or batch (`{"texts": [...]}`), pydantic-validated, returns the top label plus the full probability distribution.
- **`GET /healthz`** — readiness/liveness; 503 until the model is loaded and the batcher is running.
- **`GET /metrics`** — Prometheus exposition: request count, latency histogram, in-flight gauge, error count, plus model-level inference latency and batch-size histograms.
- **Dynamic micro-batching** — concurrent single requests are coalesced into one forward pass for throughput, with a latency cap you control.
- **Offline stub** — a deterministic, lexicon-driven classifier so the API behaves (and tests pass) with no weights.
- **Built-in demo UI** at `/demo` that calls the live API.

```mermaid
flowchart LR
    U[Client / Demo UI] -->|POST /predict| API[FastAPI app]
    API --> V[pydantic validation]
    V --> B[Micro-batcher<br/>coalesce + flush]
    B --> M{Model loader}
    M -->|OFFLINE=1| S[Stub classifier<br/>deterministic, no downloads]
    M -->|OFFLINE=0| H[DistilBERT pipeline<br/>LaelaZ/distilbert-emotion]
    S --> R[label + probabilities]
    H --> R
    R --> U
    API -.->|/metrics| P[(Prometheus)]
    P --> G[Grafana dashboard]
    API -.->|/healthz| K[Orchestrator probes]
```

## Results / impact

Latency and throughput measured by the included load test (`scripts/loadtest.py`) hitting `POST /predict` against the **offline stub**, single uvicorn worker, on an Apple-silicon laptop. Numbers are reproducible from a clean checkout with no downloads:

```bash
make bench          # human-readable summary
make bench-table    # the markdown row below
```

| concurrency | throughput (req/s) | p50 (ms) | p95 (ms) | p99 (ms) |
|---|---|---|---|---|
| 1  | 118 | 8.27  | 8.87  | 13.12  |
| 8  | 595 | 13.35 | 16.49 | 19.97  |
| 16 | 604 | 19.03 | 67.49 | 107.39 |

Throughput scales ~5x from serial to 8 concurrent requests as the micro-batcher coalesces forward passes, while p50 stays in the low-teens of milliseconds; all runs completed with **0 errors**. (These reflect the stub plus full HTTP/validation/batching overhead — the real model adds per-call inference cost on top, but the service shape, batching wins, and tail-latency behavior are what's being measured here.)

## Quickstart

No model download, no GPU, no network — `OFFLINE=1` is the default.

```bash
python -m venv .venv && source .venv/bin/activate
pip install -r requirements-dev.txt

make test         # full suite, offline, < 1s
make demo         # serve API + UI at http://localhost:8000/demo
```

Call it:

```bash
curl -s -X POST http://localhost:8000/predict \
  -H 'Content-Type: application/json' \
  -d '{"text": "i can'\''t stop smiling, today went better than i ever hoped"}'
# {"label":"joy","score":0.74,"probabilities":{"sadness":...,"joy":0.74,...}}

curl -s -X POST http://localhost:8000/predict \
  -H 'Content-Type: application/json' \
  -d '{"texts": ["i am so scared right now", "how dare they"]}'
# {"predictions":[{"label":"fear",...},{"label":"anger",...}]}
```

Run the **real** model instead of the stub:

```bash
pip install -r requirements-ml.txt     # adds torch + transformers
OFFLINE=0 make serve                   # loads LaelaZ/distilbert-emotion from the Hub
```

## Tech stack

- **API:** FastAPI + Uvicorn, pydantic v2 validation
- **Model:** Hugging Face `transformers` pipeline over the fine-tuned DistilBERT (`LaelaZ/distilbert-emotion`); deterministic lexicon stub for the offline path
- **Throughput:** custom async micro-batcher (asyncio queue + threaded forward pass)
- **Observability:** `prometheus-client`, Prometheus, Grafana (provisioned dashboard)
- **Packaging/CI:** multi-stage slim Docker image (non-root), GitHub Actions
- **IaC:** Fly.io (`fly.toml`), Render (`render.yaml`), Terraform stub (`deploy/terraform/`)
- **Load test:** asyncio + httpx benchmark script

## Deploy

The image runs in offline mode by default, so every target below comes up with no external dependencies. For the real model, build from `requirements-ml.txt`, set `OFFLINE=0`, and give the machine more memory (>= 2 GB for torch + weights).

**Docker (local):**
```bash
make docker-run          # build the slim image and run it on :8000
```

**Full stack with monitoring:**
```bash
make compose-up          # API :8000, Prometheus :9090, Grafana :3000
```

**Fly.io:**
```bash
fly launch --no-deploy   # reads fly.toml
fly deploy
```

**Render:** connect the repo; it picks up `render.yaml` automatically.

**Terraform (Fly provider):**
```bash
cd deploy/terraform
export FLY_API_TOKEN=$(fly auth token)
terraform init
terraform apply -var="image=ghcr.io/laelazorana/distilbert-emotion-api:latest"
```

CI (`.github/workflows/ci.yml`) runs the offline tests, builds the image, and smoke-tests it. The GHCR push step is present but **guarded off** (`if: false`) so CI never publishes — flip the guard to enable a real release.

## Monitoring

`docker compose up` brings up Prometheus (scraping `/metrics` every 5s) and Grafana with a pre-provisioned **Service Overview** dashboard (`observability/grafana/dashboards/emotion-api.json`):

- Request rate, error rate (5xx %), in-flight requests, p95 latency (stat tiles)
- HTTP latency percentiles (p50/p95/p99) over time
- Request rate by status code
- **Model inference latency** (separated from HTTP overhead, so "model is slow" vs "framework is slow" is visible)
- Average inference batch size (shows the batcher working under load)

Open Grafana at `http://localhost:3000` (anonymous viewer; `admin`/`admin` to edit). Generate traffic with `make bench` and watch the panels move.

Exported metrics: `emotion_api_requests_total`, `emotion_api_request_latency_seconds`, `emotion_api_errors_total`, `emotion_api_requests_in_progress`, `emotion_api_inference_latency_seconds`, `emotion_api_inference_batch_size`.

## Screenshots

> _Placeholder._ Add screenshots of the demo UI (`/demo`), the Swagger docs (`/docs`), and the Grafana dashboard here.
>
> - `docs/demo-ui.png` — the emotion demo page
> - `docs/grafana.png` — the Service Overview dashboard under load

## Project layout

```
distilbert-emotion-api/
├── app/
│   ├── __init__.py        # labels + version
│   ├── config.py          # env-driven settings
│   ├── classifier.py      # model abstraction: stub + real transformers backend
│   ├── batching.py        # async micro-batcher
│   ├── schemas.py         # pydantic request/response models
│   ├── metrics.py         # Prometheus collectors + middleware
│   └── main.py            # FastAPI app, routes, lifespan
├── demo/index.html        # zero-dependency demo UI that calls /predict
├── scripts/loadtest.py    # asyncio/httpx latency + throughput benchmark
├── tests/                 # /predict, validation, stub, batcher, health, metrics
├── observability/         # Prometheus + Grafana provisioning + dashboard
├── deploy/terraform/      # Terraform stub (Fly provider)
├── Dockerfile             # multi-stage slim image (non-root)
├── docker-compose.yml     # API + Prometheus + Grafana
├── fly.toml · render.yaml # IaC for managed platforms
├── .github/workflows/ci.yml
└── Makefile
```

## License

MIT — Copyright (c) 2026 Laela Zorana. See [LICENSE](LICENSE).

**Links:** [GitHub](https://github.com/LaelaZorana) · [Model on the Hub](https://huggingface.co/LaelaZ/distilbert-emotion) · [HuggingFace](https://huggingface.co/LaelaZ)