tokencostguard / docs /case-study-template.md
lixiaowww
CSP: allow huggingface.co frame-ancestors
249c849
|
Raw
History Blame Contribute Delete
2.61 kB
# TCG Case Study Template
Use after a beta user agrees to be referenced (anonymized or named). Keeps ROI claims defensible for marketing and investors.
**File naming:** `artifacts/case-studies/YYYY-MM-<slug>.md`
---
## Metadata
| Field | Value |
|-------|-------|
| Customer (public name) | |
| Segment | indie / YC startup / SMB product team |
| Stack | LangChain / AutoGen / custom / other |
| Agent topology | # agents, avg steps per task |
| TCG plan | Free / Pro |
| Period analyzed | YYYY-MM-DD β†’ YYYY-MM-DD |
| Author | |
---
## 1. Baseline (before TCG)
- **Monthly LLM spend (USD):**
- **Primary models:**
- **Main workload:** e.g. coding agent, support bot, RAG pipeline
- **Pain:** e.g. bill spike, no attribution, unknown cache usage
- **Tools used before TCG:** LangSmith / none / spreadsheet / other
---
## 2. Integration
- **Endpoint:** `POST /api/v1/optimize` (+ feedback if used)
- **Frequency:** per trace / nightly batch / other
- **Data sent:** full logs / truncated / PII redacted by customer
- **Applied recommendations?** yes / partial / no (critical for honesty)
| Intervention | Applied? | How |
|--------------|----------|-----|
| Compression / HCA summary | | |
| Routing tier change | | |
| Prompt structure / cache blocks (v1) | | |
| HERMES / policy | | |
---
## 3. Metrics (quantitative)
| Metric | Before | After | Ξ” | Source |
|--------|--------|-------|---|--------|
| Est. monthly tokens (input) | | | | TCG / provider bill |
| Est. monthly cost (USD) | | | | |
| Avg tokens per successful task | | | | |
| % calls on premium model | | | | |
| Cached input tokens (if provider reports) | | | | |
| p95 latency (optional) | | | | |
**TCG-reported fields (from API):**
- `_guard.baseline` / `_guard.saved` (note: may be **estimated** from logs, not provider invoice)
- `_vmm.saving_ratio` if compression applied
- `_routing.targetTier` if used
**Claimed savings %:** ___%
**Method:** `(baseline_cost - after_cost) / baseline` from TCG **or** invoice comparison β€” **state which**.
---
## 4. Qualitative
- **What surprised the team?**
- **What they would not give up (quality guardrails)?**
- **Would they pay $12/mo? Why / why not?**
---
## 5. Limitations (required)
- [ ] Savings include model routing, not only prompt optimization
- [ ] Sample size: N traces / M days
- [ ] Seasonality or product change during period
- [ ] TCG did not proxy LLM traffic (beta)
---
## 6. Publishable quote (optional)
> β€œβ€¦β€ β€” Role, Company
---
## 7. Internal only
- `tenant_id` / trace ids (do not publish)
- Support issues / blockers for product