COST_NOTES.md

Backend Cost Notes

Operations that cost OpenAI tokens

Operation	Model	When	Frequency
Final answer generation	gpt-4o-mini (chat)	Every uncached user query	Per unique question
Document summarization	gpt-4o-mini (chat)	First call per document	Once per file; sidecar caches result
Document ingestion (embedding)	text-embedding-3-small	When a document is indexed	Once per document; re-indexing re-bills
Site re-indexing after scrape	text-embedding-3-small	When scheduler detects stale content	At most once per 60-minute window

Operations that are free (zero OpenAI tokens)

Operation	How
Semantic cache lookup	Local sentence-transformers (all-MiniLM-L6-v2) + pure-Python cosine similarity
Cache embedding on save	Same local model — no API call
BM25 paragraph scoring	rank-bm25 library, runs in-process
Website scraping	Plain HTTP fetch + BeautifulSoup
Content diff checking	difflib.unified_diff (stdlib)
Staleness check	Timestamp arithmetic on cached JSON
ChromaDB metadata queries	Reads stored metadata; no embedding triggered
Document summarization (repeat)	Reads .summary.json sidecar; OpenAI never called
Cache hit response	Returns stored answer string; agent never instantiated

Token estimates per request

Uncached query (cache MISS)

System prompt:          ~100 tokens  (input)
Tool schemas (3 tools): ~200 tokens  (input)
Conversation history:   ~50 tokens   (input, typical 1-2 turns)
User message:           ~20 tokens   (input)
Tool call result        ~400 tokens  (input — BM25 top-5 paragraphs from website)
  (search_agentrax_website)
──────────────────────────────────────────────
Total input:            ~770 tokens
Agent response:         ~150 tokens  (output)
──────────────────────────────────────────────
Total per query:        ~920 tokens  → ~$0.00028 at gpt-4o-mini pricing

Cached query (cache HIT)

OpenAI tokens:          0
Local compute:          sentence-transformers inference + cosine scan over cache
Latency:                <50 ms (no network call)
Cost:                   $0.00

Document summarization (first call)

System prompt:          ~50 tokens   (input)
Document text:          up to 12,000 characters (~3,000 tokens, input)
Summary output:         ~300 tokens  (output)
──────────────────────────────────────────────
Total first call:       ~3,350 tokens → ~$0.001 at gpt-4o-mini pricing
Subsequent calls:       0 tokens (sidecar cache)

Document ingestion (text-embedding-3-small)

Rate:                   $0.00002 per 1K tokens
Typical document:       ~5,000 tokens across all chunks
Cost per document:      ~$0.0001

Cost reduction mechanisms in place

1. Semantic cache — identical or near-identical questions (cosine ≥ 0.85) skip the agent entirely.
2. Scrape cache — website content is served from disk for 60 minutes; HTTP fetch only on stale.
3. Summary sidecar — document summaries are written to disk and never regenerated.
4. Staleness guard in scheduler — is_content_stale() checked before any scrape or re-index.
5. Local embeddings for cache — sentence-transformers runs on CPU, no API cost for cache operations.
6. ChromaDB singleton — client opened once per process; no per-request reconnect overhead.
7. gpt-4o-mini — ~15× cheaper than gpt-4o for equivalent tasks in this domain.