COST_NOTES.md

# Backend Cost Notes

## Operations that cost OpenAI tokens

| Operation | Model | When | Frequency |
|---|---|---|---|
| Final answer generation | gpt-4o-mini (chat) | Every **uncached** user query | Per unique question |
| Document summarization | gpt-4o-mini (chat) | First call per document | Once per file; sidecar caches result |
| Document ingestion (embedding) | text-embedding-3-small | When a document is indexed | Once per document; re-indexing re-bills |
| Site re-indexing after scrape | text-embedding-3-small | When scheduler detects stale content | At most once per 60-minute window |

## Operations that are free (zero OpenAI tokens)

| Operation | How |
|---|---|
| Semantic cache lookup | Local sentence-transformers (`all-MiniLM-L6-v2`) + pure-Python cosine similarity |
| Cache embedding on save | Same local model — no API call |
| BM25 paragraph scoring | `rank-bm25` library, runs in-process |
| Website scraping | Plain HTTP fetch + BeautifulSoup |
| Content diff checking | `difflib.unified_diff` (stdlib) |
| Staleness check | Timestamp arithmetic on cached JSON |
| ChromaDB metadata queries | Reads stored metadata; no embedding triggered |
| Document summarization (repeat) | Reads `.summary.json` sidecar; OpenAI never called |
| Cache hit response | Returns stored answer string; agent never instantiated |

## Token estimates per request

### Uncached query (cache MISS)

```
System prompt:          ~100 tokens  (input)
Tool schemas (3 tools): ~200 tokens  (input)
Conversation history:   ~50 tokens   (input, typical 1-2 turns)
User message:           ~20 tokens   (input)
Tool call result        ~400 tokens  (input — BM25 top-5 paragraphs from website)
  (search_agentrax_website)
──────────────────────────────────────────────
Total input:            ~770 tokens
Agent response:         ~150 tokens  (output)
──────────────────────────────────────────────
Total per query:        ~920 tokens  → ~$0.00028 at gpt-4o-mini pricing
```

### Cached query (cache HIT)

```
OpenAI tokens:          0
Local compute:          sentence-transformers inference + cosine scan over cache
Latency:                <50 ms (no network call)
Cost:                   $0.00
```

### Document summarization (first call)

```
System prompt:          ~50 tokens   (input)
Document text:          up to 12,000 characters (~3,000 tokens, input)
Summary output:         ~300 tokens  (output)
──────────────────────────────────────────────
Total first call:       ~3,350 tokens → ~$0.001 at gpt-4o-mini pricing
Subsequent calls:       0 tokens (sidecar cache)
```

### Document ingestion (text-embedding-3-small)

```
Rate:                   $0.00002 per 1K tokens
Typical document:       ~5,000 tokens across all chunks
Cost per document:      ~$0.0001
```

## Cost reduction mechanisms in place

1. **Semantic cache** — identical or near-identical questions (cosine ≥ 0.85) skip the agent entirely.
2. **Scrape cache** — website content is served from disk for 60 minutes; HTTP fetch only on stale.
3. **Summary sidecar** — document summaries are written to disk and never regenerated.
4. **Staleness guard in scheduler** — `is_content_stale()` checked before any scrape or re-index.
5. **Local embeddings for cache** — sentence-transformers runs on CPU, no API cost for cache operations.
6. **ChromaDB singleton** — client opened once per process; no per-request reconnect overhead.
7. **gpt-4o-mini** — ~15× cheaper than gpt-4o for equivalent tasks in this domain.