README.md

---
title: contimp-app
emoji: 🔁
colorFrom: indigo
colorTo: pink
sdk: docker
app_port: 7860
pinned: false
---

# contimp-app

**Live app: https://lefft-contimp-app.hf.space** (team passcode — ask in Slack) ·
hosted as a [Hugging Face Space](https://huggingface.co/spaces/lefft/contimp-app)

A small demo application that exercises an LLM on narrow, repeatable tasks and logs
every interaction (prompts, responses, tool calls, ground truth, auto-scores) to
LangFuse. It exists to generate *real* live traffic for dogfooding Oumi's
continuous model improvement ("contimp") loop: import traces as datasets, define
evals, fine-tune a small model, compare, and swap the endpoint.

The app is deliberately simple: one FastAPI service, a static vanilla-JS frontend,
no database. All state lives in committed data files and LangFuse.

## Tasks

Tasks are plugins (see *Adding a task* below). Current tasks:

| id | what the model does | ground truth |
|---|---|---|
| `pr-area` | Read a real merged oumi PR (title+body) and classify which subsystem it touches | Derived from the files the PR actually changed |
| `config-copilot` | Turn a natural-language training-run request into a valid oumi training YAML, using `get_schema` / `validate_config` tools | Requests are generated from known target field values; YAML graded on validity + per-field match |

"Deal me one" gives auto-scorable inputs; users can also type their own
(scored on validity/thumbs only).

## Run locally

```sh
cp .env.example .env  # fill in keys
uv sync
uv run uvicorn app.main:app --reload --port 7860 --env-file .env
```

Tests: `uv run pytest`

## The endpoint swap

The app talks to a **backend** — a wire protocol + endpoint + model + key.
The default backend is set by the `LLM_*` env vars, and supports two protocols
via `LLM_PROVIDER`:

- `openai` (default): OpenAI Chat Completions (`LLM_BASE_URL` ends in `/v1/`).
- `anthropic`: Anthropic Messages (`LLM_BASE_URL` ends in `/inference`). Needed
  for an Oumi deployment that proxies an Anthropic model — Oumi's
  `/chat/completions` rejects the anthropic provider (see `docs/findings.md`).

Moving the day-1 model (Haiku via Anthropic's OpenAI-compat endpoint) to an
OpenAI-compatible Oumi deployment (e.g. a tuned model) is an env-only change:

```sh
LLM_BASE_URL=https://<oumi>/inference/v1/
LLM_MODEL=projects/<project_id>/deployments/<deployment_id>
LLM_API_KEY=<oumi api key>
```

### Per-task backends

Tasks can target *different* backends — e.g. one demo on Anthropic-direct,
another on an Oumi proxy. A task pins itself by setting `backend="<name>"`
(see `app/tasks/base.py`); the named backend is configured with `LLM_<NAME>_*`
env vars (see `.env.example`). Tasks that don't set one use the default. Each
trace records its `backend` in metadata so traffic can be segmented per backend.

## Observability

One LangFuse trace per run; nested generations per LLM call (OpenAI wire format,
tool calls included). Every trace carries:

- tags: `[<task_id>, human|synthetic]`
- metadata: `task_id`, `app_version`, `model`, `input_id`, `ground_truth`
- scores: per-task auto-scores (`exact_match`; `yaml_valid`, `schema_valid`,
  `field_match`) plus human `thumbs` feedback
- `user_id`: self-reported handle (or `traffic-bot`), `session_id`: browser session

Filter by tag/metadata `task_id` before exporting per-task datasets. Besides the
LangFuse UI batch export, `scripts/export_traces.py` pulls traces via the API as
observation-level JSONL in the shape the oumi platform's LangFuse importer
consumes (see `docs/findings.md` for one importer gap found doing this).

## Data files

- `data/prs.jsonl` — mined merged PRs (title, body, changed files, derived area).
  Rebuild: `uv run python scripts/build_pr_dataset.py`
- `data/oumi_training_schema.json` — curated JSON Schema subset of oumi's
  TrainingConfig used by the `config-copilot` validator tool. May drift from oumi
  HEAD; rebuild: `uv run python scripts/dump_oumi_schema.py`

## Traffic generation

```sh
uv run python scripts/generate_traffic.py --target https://<deployment-url> --task pr-area --n 25
```

Synthetic traffic is tagged `synthetic` with user `traffic-bot` so it can be
segmented from human traffic.

## Adding a task

Drop one module into `app/tasks/` defining a `TASK = Task(...)` (see
`app/tasks/base.py`), plus any data file it needs. The registry auto-discovers
it; nav, sampling, running, scoring, and tracing all come for free.

## Deploy

The live app is a [Hugging Face Space](https://huggingface.co/spaces/lefft/contimp-app)
that builds from its own copy of these files (the YAML frontmatter above is its
config; it serves on port 7860). **Merging to `main` does not deploy** — the
Space is updated explicitly by uploading the working tree to it:

```sh
uv run --with huggingface_hub python scripts/deploy.py -m "what changed"
```

When to run it: after a change is merged to `main` and you want it live. The
upload triggers a rebuild (~1-2 min).

**`deploy.py` uploads your local working tree, not remote `main`** — whatever is
in the directory ships, regardless of git state. So run this checklist first:

1. `git checkout main && git pull` — deploy the canonical merged state.
2. `git status` — clean tree, no stray files (the upload skips `.git`, `.venv`,
   `__pycache__`, `.env*`, `.pytest_cache`, and `.claude/*`).
3. `uv run pytest` — green.
4. `hf auth whoami` — logged in with write access to the Space.
5. Deploy, then verify: `curl https://lefft-contimp-app.hf.space/api/health` and
   confirm the task shows at `/api/tasks` (or click the app). Traffic only reaches
   LangFuse if the Space has the `LANGFUSE_*` secrets set (see Prerequisites).

Prerequisites:

- A Hugging Face login with **write access to the Space** — run `hf auth login`
  once; the token is cached under `~/.cache/huggingface/` and reused across
  sessions. Check with `hf auth whoami`.
- That's the only credential `deploy.py` needs. The app's own secrets
  (`LLM_API_KEY`, `LANGFUSE_*`, `APP_PASSCODE` — see `.env.example` for the
  names) are **not** deployed from here: they live in the Space's
  *Settings → Variables and secrets*, encrypted, and are never committed. To
  rotate one, edit it there and restart the Space; the repo doesn't change.

The app is host-agnostic (plain Docker + env vars), so it also runs on any other
container host — point that host at the `Dockerfile` and supply the
`.env.example` variables as environment secrets.