README.md

metadata

title: PromptStat
emoji: 🎮
colorFrom: gray
colorTo: green
sdk: gradio
sdk_version: 6.6.0
app_file: space_app.py
pinned: false
license: apache-2.0
short_description: Rate your AI-collaboration skill — MiniCPM-8B + LoRA
tags:
  - thousand-token-wood
  - minicpm
  - modal
  - off-brand
  - track:wood
  - sponsor:openbmb
  - sponsor:modal

🎮 PromptStat

Upload your ChatGPT or Claude export (or paste a conversation / share link) and get an esports-style stat card rating how you collaborate with AI across 5 observable axes:

Focus · Technique · Critical Engagement · Interaction · Input Quality → an overall score + tier (D→S).

🔗 Live demo: [demo link — TODO] · 🐦 Write-up: [social link — TODO]

What makes it real

• Scored by MiniCPM4.1-8B (OpenBMB) using a validated observable-detection method — not vibes. The model is served on Modal (vLLM on an A100-40GB, the 4 LoRA adapters loaded warm and addressable by name). The card tags every score "✓ scored on MiniCPM-8B + LoRA hybrid"; if no model endpoint is configured it falls back to a transparent heuristic and says so (amber tag).
• Per-category LoRA hybrid (trained on Modal, H100): the 8B is fine-tuned only where it makes correctable errors — decomposition 0.26→0.66 κ, critical·source_request 0.47→0.66, plus independent_verification and interaction — and stays base everywhere it's already strong. Each adapter overrides only its own feature, so there's no catastrophic forgetting.
• Honest validation: every axis ships its measured Cohen's κ vs a 159-conversation human-labeled set. We document where the 8B hits its ceiling (it can't out-agree the human rater) instead of inflating.

Two ways in

• Upload export — full profile across your whole history (representative 30-conversation sample, every turn within each scored; set UI_MAX_CONVS=0 for the entire history).
• Paste a conversation or a ChatGPT/Claude share link — fast single-conversation sample for quick feedback / demo. Clearly labelled "Sample analysis — not your overall AI usage."

Privacy

Your chat content is parsed in memory only — never written to disk, never logged. A pasted share link is fetched once to read the conversation and is likewise never stored.

Architecture (Modal end-to-end)

• Inference: modal_serve_minicpm.py — vLLM serves base MiniCPM4.1-8B plus all 4 LoRA adapters from a single warm A100; the app routes each axis to its adapter by model name over HTTP. Model weights are baked into the image so container starts don't touch the Hub.
• Training: modal_train_lora.py / modal_eval_lora.py — the LoRA adapters were trained and evaluated on Modal GPUs against the human-labeled validation set.

Run locally

pip install -r requirements.txt
# real scoring needs a MiniCPM endpoint (auto-loaded from eval/.secrets.env if present):
export OPENBMB_BASE_URL=...   # OpenBMB free API, or your Modal vLLM URL
export OPENBMB_TOKEN=...
python -m ui.app              # or: python space_app.py   (or ./run_demo.sh for the LoRA hybrid)
python -m pytest ui/tests -q  # 19 UI tests

Without the env vars the app still runs (heuristic fallback, amber tag). See DEPLOY.md for the Space deploy

• Modal model-serving + speed tuning, and handoff_to_card.md for the card_data schema.

Built for the OpenBMB "Build Small" hackathon

Models ≤32B, English. Uses MiniCPM4.1-8B (reasoning) — eligible model — with Modal for both LoRA training and warm inference. Maintainer notes: backend/ML workstream in prompt_card/, modal_*.py, SPEC.md; UI shell in ui/.