docs/embedding_cost_analysis.md

# Embedding Cost Analysis — Gemini Embedding 2 vs BAAI/bge-small-en-v1.5

**Created:** 2026-05-16  
**Context:** Decision document for embedding model selection and KB entry billing design.  
**Status:** Gemini Embedding 2 migration recommended. See Section 6 for implementation plan.

---

## Table of Contents

1. [Current Embedding Setup](#1-current-embedding-setup)
2. [Gemini Embedding 2 Overview](#2-gemini-embedding-2-overview)
3. [OpenAI Embedding Lineup (2026)](#3-openai-embedding-lineup-2026)
4. [Cost Per Entry by Modality](#4-cost-per-entry-by-modality)
5. [Scale Analysis: 1,000 Tenants × 500 Entries](#5-scale-analysis-1000-tenants--500-entries)
6. [Billing Model Recommendation](#6-billing-model-recommendation)
7. [Video & Audio — Transcript Strategy](#7-video--audio--transcript-strategy)
8. [remork.org Specific Analysis](#8-remorkorgy-specific-analysis)
9. [Competitor Multimodal KB Support](#9-competitor-multimodal-kb-support)
10. [Qdrant Timestamp Filtering — False Claim Correction](#10-qdrant-timestamp-filtering--false-claim-correction)
11. [Credit System Note](#11-credit-system-note)

---

## 1. Current Embedding Setup

- **Model:** `BAAI/bge-small-en-v1.5`
- **Inference:** Local (sentence-transformers), runs in Docker container
- **Dimensions:** 384
- **Cost:** $0 per embedding (local inference, no API call)
- **Context window:** 512 tokens
- **MTEB score:** ~62

**Why this was chosen:** Zero cost, good enough for V1, no external API dependency.  
**Why to migrate:** 512-token context limit truncates long KB entries. Quality gap is real (+6 MTEB points). Multimodal capability unavailable.

---

## 2. Gemini Embedding 2 Overview

| Attribute | Value |
|---|---|
| **Model ID** | `gemini-embedding-2` (GA as of March 10, 2026) |
| **Output Dimensions** | 3,072 (Matryoshka flexible: 128–3,072; recommended: 768, 1,536, or 3,072) |
| **Context Window** | 8,192 tokens (16× larger than current bge-small) |
| **Modalities** | Text + Images (6×) + Video (120s) + Audio (80s) + PDF (6 pages) — **all in same vector space** |
| **MTEB Score** | 68.32 — top of leaderboard by 5+ points (2026) |
| **Languages** | 100+ languages |
| **Deployment** | Server-side only (never expose API key in browser) |
| **Batch discount** | 50% off for async batch jobs |

**Pricing:**

| Modality | Standard | Batch (50% off) |
|---|---|---|
| Text | $0.20 / 1M tokens | $0.10 / 1M tokens |
| Image | $0.45 / 1M tokens | $0.225 / 1M tokens |
| Video | $12.00 / 1M tokens | $6.00 / 1M tokens |
| Audio | $6.50 / 1M tokens | $3.25 / 1M tokens |

---

## 3. OpenAI Embedding Lineup (2026)

**All OpenAI embedding models are text-only as of 2026.** No multimodal support.

| Model | Dimensions | MTEB | Cost | Notes |
|---|---|---|---|---|
| `text-embedding-ada-002` | 1,536 | ~61 | $0.10/1M | Legacy, not deprecated |
| `text-embedding-3-small` | 1,536 | 62.3 | $0.02/1M | Cheapest OpenAI option |
| `text-embedding-3-large` | 3,072 | 64.6 | $0.13/1M | Best OpenAI text quality |
| `gemini-embedding-2` | 3,072 | **68.32** | $0.20/1M | Only model with native multimodal |

**Conclusion:** OpenAI has no path to multimodal embeddings. Gemini Embedding 2 is the only commercial model that handles video/audio natively (2026).

---

## 4. Cost Per Entry by Modality

**Token counting methodology:**

| Modality | Token counting | Assumed avg per entry |
|---|---|---|
| Text (typical SMB KB article) | 1 token ≈ 4 chars | 750 tokens |
| Text (remork.org — title + tags) | 1 token ≈ 4 chars | 65 tokens |
| Image (medium-res, 1 per entry) | ~768 tokens/image | 768 tokens |
| Video (10-min educational) | 1 frame/sec × 258 tokens/frame | 154,800 tokens |
| Audio (10-min recording) | ~32 tokens/sec | 19,200 tokens |
| PDF (5 pages, image-based) | ~768 tokens/page | 3,840 tokens |

**Cost per single entry:**

| Entry type | Tokens | Price/1M tokens | Cost per entry |
|---|---|---|---|
| Text (typical SMB) | 750 | $0.20 | **$0.00015** |
| Text (remork.org style) | 65 | $0.20 | **$0.000013** |
| Image only (1 image) | 768 | $0.45 | **$0.00035** |
| Text + 1 image | 1,518 | blended | **$0.00050** |
| Audio (10 min) | 19,200 | $6.50 | **$0.125** |
| Video (10 min) | 154,800 | $12.00 | **$1.86** |
| PDF (5 pages) | 3,840 | $0.20 | **$0.00077** |
| Transcript from video (text only) | ~750 | $0.20 | **$0.00015** |

**The critical cliff:** Text, images, and PDFs are all under $0.001 per entry. Video jumps to $1.86 per 10-minute entry — **12,400× more expensive than a text entry.**

---

## 5. Scale Analysis: 1,000 Tenants × 500 Entries

| Scenario | Total entries | Total one-time cost | Cost per tenant |
|---|---|---|---|
| All text (typical SMB) | 500,000 | **$75** | $0.15 |
| remork.org style (title + tags) | 500,000 | **$6.50** | $0.0065 |
| Mixed text + images (e-commerce) | 500,000 | **~$160** | $0.16 |
| All 10-min videos (raw embedding) | 500,000 | **$930,000** | $930 |
| All 10-min videos (transcript only) | 500,000 | **$75** | $0.15 |

At batch pricing (50% off), halve all figures.

**Key insight:** Switching from raw video embedding to transcript-based embedding brings the cost from $930K down to $75 — a **12,400× cost reduction** with better Q&A retrieval quality.

---

## 6. Billing Model Recommendation

**⚠️ NOTE (2026-05-16): The `credits` abstraction is being removed entirely. All billing will use USD directly — tenants top up in USD, deductions are in USD, no conversion factor. See Section 11 and backend `crucial_notes.md` for the refactor plan. The table below uses USD throughout.**

### Rule: text and PDF entries are free. Media entries (image, audio, video) cost USD.

The billing system will deduct USD directly from `tenants.balance_usd`. Add a new event type: `kb_embedding`.

| Entry type | Platform cost | Charge tenant | Rationale |
|---|---|---|---|
| Text | $0.00015 | **Free** | Too small to meter; absorb as platform overhead |
| PDF | $0.00077 | **Free** | Same — absorb |
| Image (per image in entry) | $0.00035 | **~$0.001** | Small fixed charge per image with markup |
| Audio (per minute) | ~$0.012/min | **$0.02/min** | Per-minute, transparent — or free if auto-transcribed |
| Video — raw embedding (per min) | ~$0.186/min | **$0.20/min** | Strongly discourage; show transcript path instead |
| Video — transcript only (Whisper) | $0 | **Free** | Strongly recommended default path |
| Audio — transcript only (Whisper) | $0 | **Free** | Strongly recommended default path |

### Per-minute billing rationale

Tenants understand "longer video = more $" intuitively — mirrors how phone calls are billed. Before submission, show an estimate: *"This video is 12 minutes. Direct embedding cost: ~$2.40. Transcript embedding: Free."*

### Billing implementation (post-credit-removal)

Add to `billing_events.event_type` enum: `kb_embedding_image`, `kb_embedding_audio`, `kb_embedding_video`  
Add a `kb_embedding_rates` config table: USD per image / per audio-min / per video-min (mirrors the existing `llm_models` rate table pattern)

---

## 7. Video & Audio — Transcript Strategy

### The core problem

Raw video/audio embedding is expensive and overkill for Q&A use cases. A transcript answers "what does this video cover?" just as well as video frame analysis — and does it faster, cheaper, and more accurately.

### Recommended UX flow (applies to both video and audio)

```
Tenant uploads video/audio OR pastes YouTube URL
       │
       ▼
Backend queues transcription job (Celery async)
       │
       ├─ YouTube URL → YouTube Transcript API (free, instant)
       │
       └─ Uploaded file → Whisper (local, free) or Whisper API ($0.006/min)
       │
       ▼
Frontend shows: "Transcript ready — review before embedding"
       │
Editable transcript preview box (tenant can fix errors, add context)
       │
       ├─ [Embed Transcript — Free] → stored as text KB entry, video URL in source_url
       │
       └─ [Embed Media Directly — X credits] → uses Gemini video/audio embedding
```

### Transcription options by cost

| Method | Cost | Speed | Quality | Use case |
|---|---|---|---|---|
| YouTube Transcript API | Free | Instant | High (native captions) | YouTube URLs |
| Whisper local (Docker) | Free | ~1× realtime | High | Uploaded files, on-prem |
| OpenAI Whisper API | $0.006/min | Fast | High | Cloud alternative to local |
| Deepgram / AssemblyAI | ~$0.01–0.02/min | Fast | Very high | If better accuracy needed |

**Recommendation:** Run Whisper locally in Docker for uploaded files (zero cost, no API dependency). YouTube Transcript API for YouTube URLs.

### What gets stored in the KB

```
KB entry (transcript-based video):
  title:      "Introduction to Khmer Literacy — Grade 3"
  content:    "[transcript text here — editable by tenant]"
  source_url: "https://youtube.com/watch?v=abc123"   ← video pointer
  origin:     "video_transcript"
  source:     "youtube" | "upload"
```

The chatbot can then:
1. Retrieve the transcript for Q&A ("What topics are covered in the Grade 3 literacy video?")
2. Include the video link in its response ("Watch the video here: [source_url]")

### When raw video/audio embedding is still useful

Only one scenario justifies it: **visual Q&A** — "What is shown on the whiteboard at 3:20?" For educational platforms, this is rare. For product demos or training videos where visual elements matter, it's valid — but should be opt-in and credit-gated.

---

## 8. remork.org Specific Analysis

**About remork.org:** Non-profit educational platform serving thousands of Cambodian teachers. KB entries are video-based posts with title + tags + rare short descriptions.

### Current state (videos stored on server)

- KB entries: title + tags + optional short description
- Average tokens per entry: ~65
- Cost per entry at Gemini Embedding 2: **$0.000013**
- Cost for 2,000 posts: **$0.026 total one-time ingestion**
- **Verdict: essentially free. Absorb it without a second thought.**

### Future state (if videos move to YouTube)

Do NOT use raw video embedding. Use the transcript strategy:

1. Paste YouTube video URL into KB entry form
2. Backend calls YouTube Transcript API (free, instant)
3. Admin reviews/edits the Khmer transcript
4. Click "Embed Transcript"
5. Cost: **$0.000013 per entry** (same as text) + free YouTube API call

**Note on Khmer language:** YouTube auto-captions support Khmer. Quality varies — human review before embedding is recommended. The editable transcript preview box solves this.

### remork.org chatbot use case

A Cambodian teacher asks: "What videos are available about Grade 5 mathematics?"  
The chatbot retrieves transcript-based entries, returns matched video titles + YouTube links.  
Teacher clicks the link and watches the video.

This is the ideal flow: **chatbot as a search/discovery layer over video content**, not as a video analyzer.

---

## 9. Competitor Multimodal KB Support

| Competitor | Media KB | Notes |
|---|---|---|
| Intercom Fin | Images (chat only) | Analyzes user-uploaded screenshots in chat. Not KB upload. Enterprise/Pro. |
| Botpress | Images + video | Most advanced multimodal KB of all competitors. Mid-market. |
| Zendesk AI | Partial (roadmap) | Text-only currently; video KB planned. Enterprise. |
| Tidio Lyro | No | Text-only; uses Claude backend. SMB/mid-market. |
| Chatbase | Unclear | GPT-4o models available but no confirmed media KB upload. All tiers. |
| CustomGPT | No | Enterprise, text-only. |
| Voiceflow | No | Generic builder, text-only. |

**Market position:** If smart_chatbot ships image + transcript-based video KB, it matches Botpress and surpasses every other SMB-tier competitor.

**Marketing one-liner:** *"Unlike Tidio or CustomGPT, your chatbot understands images and videos — not just PDFs and text."*

---

## 10. Qdrant Timestamp Filtering — False Claim Correction

**Claim heard:** "Qdrant does not support timestamp/timeframe filtering."  
**Status: FALSE.** Verified against Qdrant documentation (2026).

Qdrant fully supports datetime filtering via `DatetimeRange` payload filter conditions (available since v1.8.0+).

```python
from qdrant_client import models

filter = models.Filter(
    must=[
        models.FieldCondition(
            key="created_at",
            range=models.DatetimeRange(
                gte="2026-01-01T00:00:00Z",
                lte="2026-05-16T23:59:59Z"
            )
        )
    ]
)
```

- RFC 3339 formatted dates
- `gt`, `gte`, `lt`, `lte` operators
- UTC conversion handled automatically
- Create a payload index on the date field before high-volume ingestion

**Known limitations (not blockers):**
- Unindexed date filters degrade performance on large collections
- HNSW graph can degrade with very high-cardinality filters on 260M+ point datasets (Qdrant 1.16+ ACORN algorithm mitigates this)

**Conclusion:** No need to migrate away from Qdrant on this basis.

---

## 11. Credit System — Removal Decision (2026-05-16)

**Decision: Remove the `credits` abstraction entirely. Use USD directly throughout the billing system.**

### Why credits were wrong

The system stored `tenants.credits` as a Decimal, where `1 credit = $0.001 USD` (confirmed by migration math: LLM rates × 3× markup ÷ $0.001/credit = seeded `credits_per_1k` values). The billing router returned `balance_usd = tenant.credits` — a field named "USD" but containing raw credits. This created a silent 1000× mismatch: a tenant with `credits = 100` was shown "100" under a USD label but actually had $0.10 in purchasing power.

Verified from the database (2026-05-16):
```sql
SELECT name, credits FROM tenants;
-- demo      |  97.820590
-- Monireach | 100.000000
```

The demo tenant started with 100 credits ($0.10 real USD) and spent ~2 credits (~$0.002) across 9 chat interactions — each costing 0.135–0.518 credits.

### What the refactor must do

See backend `crucial_notes.md` for the full logged task. Summary:

| Before | After |
|---|---|
| `tenants.credits` | `tenants.balance_usd` (true USD Decimal) |
| `credit_events` table | `billing_events` (or rename columns) |
| `credits_per_1k_input/output` in `llm_models` | `usd_per_1k_input/output` |
| Deduction: `amount` in credits | Deduction: `amount` in USD |
| `billing_mode = 'credits'` | `billing_mode = 'pay_as_you_go'` (or keep name) |
| 3× markup computed in credits | 3× markup computed in USD directly |

### Seeded rates — what they become in USD

| Model | Old (credits/1K) | New (USD/1K) |
|---|---|---|
| gemini-2.5-flash input | 0.900000 credits | $0.000900 |
| gemini-2.5-flash output | 7.500000 credits | $0.007500 |
| qwen3-32b input | 0.870000 credits | $0.000870 |
| qwen3-32b output | 1.770000 credits | $0.001770 |

The numbers stay identical — only the unit label changes from "credits" to "USD". No business logic changes.