README.md

metadata

title: contimp-app
emoji: 🔁
colorFrom: indigo
colorTo: pink
sdk: docker
app_port: 7860
pinned: false

contimp-app

Live app: https://lefft-contimp-app.hf.space (team passcode — ask in Slack) · hosted as a Hugging Face Space

A small demo application that exercises an LLM on narrow, repeatable tasks and logs every interaction (prompts, responses, tool calls, ground truth, auto-scores) to LangFuse. It exists to generate real live traffic for dogfooding Oumi's continuous model improvement ("contimp") loop: import traces as datasets, define evals, fine-tune a small model, compare, and swap the endpoint.

The app is deliberately simple: one FastAPI service, a static vanilla-JS frontend, no database. All state lives in committed data files and LangFuse.

Tasks

Tasks are plugins (see Adding a task below). Current tasks:

id	what the model does	ground truth
pr-area	Read a real merged oumi PR (title+body) and classify which subsystem it touches	Derived from the files the PR actually changed
config-copilot	Turn a natural-language training-run request into a valid oumi training YAML, using get_schema / validate_config tools	Requests are generated from known target field values; YAML graded on validity + per-field match

"Deal me one" gives auto-scorable inputs; users can also type their own (scored on validity/thumbs only).

Run locally

cp .env.example .env  # fill in keys
uv sync
uv run uvicorn app.main:app --reload --port 7860 --env-file .env

Tests: uv run pytest

The endpoint swap

The app talks to a backend — a wire protocol + endpoint + model + key. The default backend is set by the LLM_* env vars, and supports two protocols via LLM_PROVIDER:

• openai (default): OpenAI Chat Completions (LLM_BASE_URL ends in /v1/).
• anthropic: Anthropic Messages (LLM_BASE_URL ends in /inference). Needed for an Oumi deployment that proxies an Anthropic model — Oumi's /chat/completions rejects the anthropic provider (see docs/findings.md).

Moving the day-1 model (Haiku via Anthropic's OpenAI-compat endpoint) to an OpenAI-compatible Oumi deployment (e.g. a tuned model) is an env-only change:

LLM_BASE_URL=https://<oumi>/inference/v1/
LLM_MODEL=projects/<project_id>/deployments/<deployment_id>
LLM_API_KEY=<oumi api key>

Per-task backends

Tasks can target different backends — e.g. one demo on Anthropic-direct, another on an Oumi proxy. A task pins itself by setting backend="<name>" (see app/tasks/base.py); the named backend is configured with LLM_<NAME>_* env vars (see .env.example). Tasks that don't set one use the default. Each trace records its backend in metadata so traffic can be segmented per backend.

Observability

One LangFuse trace per run; nested generations per LLM call (OpenAI wire format, tool calls included). Every trace carries:

• tags: [<task_id>, human|synthetic]
• metadata: task_id, app_version, model, input_id, ground_truth
• scores: per-task auto-scores (exact_match; yaml_valid, schema_valid, field_match) plus human thumbs feedback
• user_id: self-reported handle (or traffic-bot), session_id: browser session

Filter by tag/metadata task_id before exporting per-task datasets. Besides the LangFuse UI batch export, scripts/export_traces.py pulls traces via the API as observation-level JSONL in the shape the oumi platform's LangFuse importer consumes (see docs/findings.md for one importer gap found doing this).

Data files

• data/prs.jsonl — mined merged PRs (title, body, changed files, derived area). Rebuild: uv run python scripts/build_pr_dataset.py
• data/oumi_training_schema.json — curated JSON Schema subset of oumi's TrainingConfig used by the config-copilot validator tool. May drift from oumi HEAD; rebuild: uv run python scripts/dump_oumi_schema.py

Traffic generation

uv run python scripts/generate_traffic.py --target https://<deployment-url> --task pr-area --n 25

Synthetic traffic is tagged synthetic with user traffic-bot so it can be segmented from human traffic.

Adding a task

Drop one module into app/tasks/ defining a TASK = Task(...) (see app/tasks/base.py), plus any data file it needs. The registry auto-discovers it; nav, sampling, running, scoring, and tracing all come for free.

Deploy

The live app is a Hugging Face Space that builds from its own copy of these files (the YAML frontmatter above is its config; it serves on port 7860). Merging to main does not deploy — the Space is updated explicitly by uploading the working tree to it:

uv run --with huggingface_hub python scripts/deploy.py -m "what changed"

When to run it: after a change is merged to main and you want it live. The upload triggers a rebuild (~1-2 min).

deploy.py uploads your local working tree, not remote main — whatever is in the directory ships, regardless of git state. So run this checklist first:

1. git checkout main && git pull — deploy the canonical merged state.
2. git status — clean tree, no stray files (the upload skips .git, .venv, __pycache__, .env*, .pytest_cache, and .claude/*).
3. uv run pytest — green.
4. hf auth whoami — logged in with write access to the Space.
5. Deploy, then verify: curl https://lefft-contimp-app.hf.space/api/health and confirm the task shows at /api/tasks (or click the app). Traffic only reaches LangFuse if the Space has the LANGFUSE_* secrets set (see Prerequisites).

Prerequisites:

• A Hugging Face login with write access to the Space — run hf auth login once; the token is cached under ~/.cache/huggingface/ and reused across sessions. Check with hf auth whoami.
• That's the only credential deploy.py needs. The app's own secrets (LLM_API_KEY, LANGFUSE_*, APP_PASSCODE — see .env.example for the names) are not deployed from here: they live in the Space's Settings → Variables and secrets, encrypted, and are never committed. To rotate one, edit it there and restart the Space; the repo doesn't change.

The app is host-agnostic (plain Docker + env vars), so it also runs on any other container host — point that host at the Dockerfile and supply the .env.example variables as environment secrets.