docs/qdrant_migration_runbook.md

# Qdrant Collection Migration Runbook

**Purpose:** Step-by-step operator procedure for migrating the Qdrant vector store to a new
embedding model (and therefore a new collection dimension) without downtime.

**Background:** Each Qdrant collection is created with a fixed vector dimension. When the embedding
model changes (e.g. from `text-embedding-3-small` at 1536 dims to `gemini-embedding-2` at 3072 dims),
the old collection cannot accept new-dimension vectors. This runbook covers the safe migration path:
create a new collection, backfill, verify, then cut over.

---

## Table of Contents

1. [When to Run This Runbook](#1-when-to-run-this-runbook)
2. [Pre-flight Checks](#2-pre-flight-checks)
3. [Local / Dev Migration](#3-local--dev-migration)
4. [Production Migration](#4-production-migration)
5. [Post-Cutover Verification](#5-post-cutover-verification)
6. [Rollback Procedure](#6-rollback-procedure)
7. [Known Limitations](#7-known-limitations)

---

## 1. When to Run This Runbook

Run this when:

- `EMBEDDING_MODEL` or `QDRANT_VECTOR_SIZE` is changing in any environment
- The backend startup log shows `Vector store readiness check failed` with a dimension mismatch error
- `/health` returns `"qdrant": "degraded"` after an embedding model change

You do **not** need this runbook for:

- Rebuilding one tenant's index (use `POST /api/v1/kb/reindex` or `scripts/rebuild_tenant_index.py`)
- Changing the LLM model (no vector dimension impact)
- Scaling Qdrant replicas or upgrading Qdrant server version

---

## 2. Pre-flight Checks

Run these before starting in any environment:

```bash
# 1. Confirm current collection dimension in Qdrant
docker exec chatbot_qdrant_dev python3 -c "
from qdrant_client import QdrantClient
c = QdrantClient('http://localhost:6333')
info = c.get_collection('smart_chatbot_kb')
print('Current dim:', info.config.params.vectors.size)
"
# Or from the backend container:
docker exec chatbot_backend_dev python3 -c "
from app.services.dependencies import get_vector_store
print('Configured dim:', get_vector_store().vector_size)
print('Actual dim:', get_vector_store().get_collection_dimension())
"

# 2. Confirm the test suite is green
docker exec chatbot_backend_dev uv run pytest tests -q --tb=no

# 3. Note the current collection name (from .env)
grep QDRANT_COLLECTION_NAME .env
```

---

## 3. Local / Dev Migration

### Step 1 — Dry run (inspect only)

```bash
docker exec chatbot_backend_dev uv run python scripts/migrate_qdrant_collection.py \
    --target smart_chatbot_kb_v2 \
    --dry-run
```

Expected output:
```
Source collection : smart_chatbot_kb
  Actual dimension: 384
Target collection : smart_chatbot_kb_v2
  Target dimension: 3072

Dry run: would migrate smart_chatbot_kb (384 → 3072) → smart_chatbot_kb_v2
No writes performed.
```

If you see `No migration needed`, the collection already matches. Stop here.

### Step 2 — Run the migration

```bash
docker exec chatbot_backend_dev uv run python scripts/migrate_qdrant_collection.py \
    --target smart_chatbot_kb_v2
```

The script will:
1. Create `smart_chatbot_kb_v2` at the target dimension
2. Re-embed all tenant KB entries
3. Run a health probe
4. Print the cutover env var

### Step 3 — Cutover

Update `.env`:
```
QDRANT_COLLECTION_NAME=smart_chatbot_kb_v2
```

Restart the backend:
```bash
docker compose -f docker-compose.dev.yml restart smart-chatbot
```

### Step 4 — Verify (see §5)

### Step 5 — Retire old collection (after verification)

```bash
docker exec chatbot_backend_dev python3 -c "
from qdrant_client import QdrantClient; import os
c = QdrantClient(os.getenv('QDRANT_URL', 'http://qdrant:6333'))
c.delete_collection('smart_chatbot_kb')
print('Deleted smart_chatbot_kb')
"
```

---

## 4. Production Migration

Production uses a live Qdrant instance. Follow this zero-impact sequence.

### Step 1 — Connect to the production backend container

```bash
ssh <prod-server>
docker exec -it <prod-backend-container> bash
```

### Step 2 — Dry run first

```bash
uv run python scripts/migrate_qdrant_collection.py --target smart_chatbot_kb_v2 --dry-run
```

Verify the output shows the expected dimension change. If it says "no migration needed", stop.

### Step 3 — Run the migration during low-traffic hours

```bash
uv run python scripts/migrate_qdrant_collection.py --target smart_chatbot_kb_v2
```

> ⚠ **Important:** While the migration runs, the live system continues to serve queries from the
> **old** collection. New KB entries written during migration land in the old collection and may not
> appear in the new one. Schedule during low-traffic hours or put the KB write endpoints behind
> a maintenance flag during migration.

### Step 4 — Update the environment variable

Update `QDRANT_COLLECTION_NAME=smart_chatbot_kb_v2` in your production env file or secret manager
(AWS SSM / Vault / Heroku config var / etc.).

### Step 5 — Rolling restart

Restart the backend service. With Docker:
```bash
docker compose -f docker-compose.prod.yml restart smart-chatbot
```

With Kubernetes:
```bash
kubectl rollout restart deployment/<backend-deployment>
```

### Step 6 — Verify (see §5)

### Step 7 — Post-cutover reindex (for any missed entries)

```bash
# For each tenant that had activity during the migration window:
curl -X POST https://your-domain.com/api/v1/kb/reindex \
     -H "X-API-Key: <tenant-api-key>"
```

### Step 8 — Retire old collection (after 24-hour observation window)

Only after you are confident the new collection is serving correctly:

```bash
# From the backend container:
python3 -c "
from qdrant_client import QdrantClient; import os
c = QdrantClient(os.getenv('QDRANT_URL'))
c.delete_collection('smart_chatbot_kb')
print('Retired smart_chatbot_kb')
"
```

---

## 5. Post-Cutover Verification

After restarting the backend with the new collection name:

```bash
# 1. Check /health reports qdrant as healthy
curl https://your-domain.com/health | python3 -m json.tool | grep qdrant

# 2. Send a test chat message to confirm RAG is working
curl -X POST https://your-domain.com/api/v1/chat \
     -H "X-API-Key: <test-tenant-api-key>" \
     -H "Content-Type: application/json" \
     -d '{"message": "What do you help with?"}'

# 3. Run the full test suite (on non-prod environments)
docker exec chatbot_backend_dev uv run pytest tests -q --tb=no
```

---

## 6. Rollback Procedure

If anything goes wrong after cutover:

### Immediate rollback

```bash
# 1. Revert QDRANT_COLLECTION_NAME to the old collection name in .env
QDRANT_COLLECTION_NAME=smart_chatbot_kb

# 2. Restart backend
docker compose -f docker-compose.dev.yml restart smart-chatbot
# Or in production: kubectl rollout restart / docker compose restart

# 3. Verify /health shows qdrant: healthy
```

The old collection is untouched throughout the migration — this rollback is instant and safe.

> Only delete the old collection **after** you are fully satisfied with the new one.

---

## 7. Known Limitations

| Limitation | Impact | Workaround |
|---|---|---|
| KB writes during migration miss the target collection | New entries are in old collection only until reindex | After cutover, call `POST /api/v1/kb/reindex` for affected tenants |
| Partial failure mid-backfill leaves target partially populated | Some tenants missing from target | Re-run the migration script (it is idempotent — deletes and re-upserts per tenant) |
| No atomic cutover | Brief window where backend serves from old collection while restart is in progress | Acceptable for V1 — use rolling restart to minimize window |
| `_qdrant_ready` flag not re-evaluated at runtime | `/health` may stay `degraded` even after Qdrant recovers mid-run | Restart backend to re-evaluate; tracked as future improvement |