docs/qdrant_migration_runbook.md

Qdrant Collection Migration Runbook

Purpose: Step-by-step operator procedure for migrating the Qdrant vector store to a new embedding model (and therefore a new collection dimension) without downtime.

Background: Each Qdrant collection is created with a fixed vector dimension. When the embedding model changes (e.g. from text-embedding-3-small at 1536 dims to gemini-embedding-2 at 3072 dims), the old collection cannot accept new-dimension vectors. This runbook covers the safe migration path: create a new collection, backfill, verify, then cut over.

---

Table of Contents

1. When to Run This Runbook
2. Pre-flight Checks
3. Local / Dev Migration
4. Production Migration
5. Post-Cutover Verification
6. Rollback Procedure
7. Known Limitations

---

1. When to Run This Runbook

Run this when:

• EMBEDDING_MODEL or QDRANT_VECTOR_SIZE is changing in any environment
• The backend startup log shows Vector store readiness check failed with a dimension mismatch error
• /health returns "qdrant": "degraded" after an embedding model change

You do not need this runbook for:

• Rebuilding one tenant's index (use POST /api/v1/kb/reindex or scripts/rebuild_tenant_index.py)
• Changing the LLM model (no vector dimension impact)
• Scaling Qdrant replicas or upgrading Qdrant server version

---

2. Pre-flight Checks

Run these before starting in any environment:

# 1. Confirm current collection dimension in Qdrant
docker exec chatbot_qdrant_dev python3 -c "
from qdrant_client import QdrantClient
c = QdrantClient('http://localhost:6333')
info = c.get_collection('smart_chatbot_kb')
print('Current dim:', info.config.params.vectors.size)
"
# Or from the backend container:
docker exec chatbot_backend_dev python3 -c "
from app.services.dependencies import get_vector_store
print('Configured dim:', get_vector_store().vector_size)
print('Actual dim:', get_vector_store().get_collection_dimension())
"

# 2. Confirm the test suite is green
docker exec chatbot_backend_dev uv run pytest tests -q --tb=no

# 3. Note the current collection name (from .env)
grep QDRANT_COLLECTION_NAME .env

---

3. Local / Dev Migration

Step 1 — Dry run (inspect only)

docker exec chatbot_backend_dev uv run python scripts/migrate_qdrant_collection.py \
    --target smart_chatbot_kb_v2 \
    --dry-run

Expected output:

Source collection : smart_chatbot_kb
  Actual dimension: 384
Target collection : smart_chatbot_kb_v2
  Target dimension: 3072

Dry run: would migrate smart_chatbot_kb (384 → 3072) → smart_chatbot_kb_v2
No writes performed.

If you see No migration needed, the collection already matches. Stop here.

Step 2 — Run the migration

docker exec chatbot_backend_dev uv run python scripts/migrate_qdrant_collection.py \
    --target smart_chatbot_kb_v2

The script will:

1. Create smart_chatbot_kb_v2 at the target dimension
2. Re-embed all tenant KB entries
3. Run a health probe
4. Print the cutover env var

Step 3 — Cutover

Update .env:

QDRANT_COLLECTION_NAME=smart_chatbot_kb_v2

Restart the backend:

docker compose -f docker-compose.dev.yml restart smart-chatbot

Step 4 — Verify (see §5)

Step 5 — Retire old collection (after verification)

docker exec chatbot_backend_dev python3 -c "
from qdrant_client import QdrantClient; import os
c = QdrantClient(os.getenv('QDRANT_URL', 'http://qdrant:6333'))
c.delete_collection('smart_chatbot_kb')
print('Deleted smart_chatbot_kb')
"

---

4. Production Migration

Production uses a live Qdrant instance. Follow this zero-impact sequence.

Step 1 — Connect to the production backend container

ssh <prod-server>
docker exec -it <prod-backend-container> bash

Step 2 — Dry run first

uv run python scripts/migrate_qdrant_collection.py --target smart_chatbot_kb_v2 --dry-run

Verify the output shows the expected dimension change. If it says "no migration needed", stop.

Step 3 — Run the migration during low-traffic hours

uv run python scripts/migrate_qdrant_collection.py --target smart_chatbot_kb_v2

⚠ Important: While the migration runs, the live system continues to serve queries from the old collection. New KB entries written during migration land in the old collection and may not appear in the new one. Schedule during low-traffic hours or put the KB write endpoints behind a maintenance flag during migration.

Step 4 — Update the environment variable

Update QDRANT_COLLECTION_NAME=smart_chatbot_kb_v2 in your production env file or secret manager (AWS SSM / Vault / Heroku config var / etc.).

Step 5 — Rolling restart

Restart the backend service. With Docker:

docker compose -f docker-compose.prod.yml restart smart-chatbot

With Kubernetes:

kubectl rollout restart deployment/<backend-deployment>

Step 6 — Verify (see §5)

Step 7 — Post-cutover reindex (for any missed entries)

# For each tenant that had activity during the migration window:
curl -X POST https://your-domain.com/api/v1/kb/reindex \
     -H "X-API-Key: <tenant-api-key>"

Step 8 — Retire old collection (after 24-hour observation window)

Only after you are confident the new collection is serving correctly:

# From the backend container:
python3 -c "
from qdrant_client import QdrantClient; import os
c = QdrantClient(os.getenv('QDRANT_URL'))
c.delete_collection('smart_chatbot_kb')
print('Retired smart_chatbot_kb')
"

---

5. Post-Cutover Verification

After restarting the backend with the new collection name:

# 1. Check /health reports qdrant as healthy
curl https://your-domain.com/health | python3 -m json.tool | grep qdrant

# 2. Send a test chat message to confirm RAG is working
curl -X POST https://your-domain.com/api/v1/chat \
     -H "X-API-Key: <test-tenant-api-key>" \
     -H "Content-Type: application/json" \
     -d '{"message": "What do you help with?"}'

# 3. Run the full test suite (on non-prod environments)
docker exec chatbot_backend_dev uv run pytest tests -q --tb=no

---

6. Rollback Procedure

If anything goes wrong after cutover:

Immediate rollback

# 1. Revert QDRANT_COLLECTION_NAME to the old collection name in .env
QDRANT_COLLECTION_NAME=smart_chatbot_kb

# 2. Restart backend
docker compose -f docker-compose.dev.yml restart smart-chatbot
# Or in production: kubectl rollout restart / docker compose restart

# 3. Verify /health shows qdrant: healthy

The old collection is untouched throughout the migration — this rollback is instant and safe.

Only delete the old collection after you are fully satisfied with the new one.

---

7. Known Limitations

Limitation	Impact	Workaround
KB writes during migration miss the target collection	New entries are in old collection only until reindex	After cutover, call POST /api/v1/kb/reindex for affected tenants
Partial failure mid-backfill leaves target partially populated	Some tenants missing from target	Re-run the migration script (it is idempotent — deletes and re-upserts per tenant)
No atomic cutover	Brief window where backend serves from old collection while restart is in progress	Acceptable for V1 — use rolling restart to minimize window
_qdrant_ready flag not re-evaluated at runtime	/health may stay degraded even after Qdrant recovers mid-run	Restart backend to re-evaluate; tracked as future improvement