OPERATIONS.md

SENTI AI — OPERATIONS RUNBOOK

STARTING THE SYSTEM

Full stack: docker-compose up -d (databases) ollama serve & (LLM) python scratch/start_all.py (16 superpacks) python senti/main.py (main gateway)

Verify health: curl http://localhost:8000/health curl http://localhost:11434/api/tags python scratch/check_ports.py

DAILY HEALTH CHECKS

curl http://localhost:8000/health python senti/tests/battle/run_all_battle_tests.py tail -50 logs/senti_gateway.log | grep ERROR

BACKUP PROCEDURE

Run daily (add to cron or Windows Task Scheduler):

Postgres backup

docker exec senti_ai_postgres_1 pg_dump
-U senti senti_db > backups/senti_db_$(date +%Y%m%d).sql

Keep last 7 days

find backups/ -name "*.sql" -mtime +7 -delete

Qdrant backup

docker exec senti_ai_qdrant_1
tar czf /tmp/qdrant_backup.tar.gz /qdrant/storage docker cp senti_ai_qdrant_1:/tmp/qdrant_backup.tar.gz
backups/qdrant_$(date +%Y%m%d).tar.gz

RESTORE PROCEDURE

Postgres restore

docker exec -i senti_ai_postgres_1 psql
-U senti senti_db < backups/senti_db_20260101.sql

Restart after restore

docker-compose restart

ROLLBACK PROCEDURE

If a deployment breaks the system:

1. Stop the broken version

docker-compose down

2. Revert the code

git stash (or git checkout last-known-good-tag)

3. Restore the last working database

(from backups/)

4. Restart

docker-compose up -d python senti/main.py

LOG ROTATION

Logs will grow. Set up rotation:

Add to logrotate (/etc/logrotate.d/senti): /path/to/senti_ai/logs/*.log { daily rotate 14 compress missingok notifempty }

MONITORING ALERTS

What to alert on: - /health endpoint returns non-200 - Error rate > 5% in last 10 minutes - LLM response time > 30 seconds - Database connection errors - Memory usage > 80%

Simple monitoring script: python senti/monitoring/health_check.py (runs every 5 minutes, sends Telegram alert if down)

WHAT TO DO WHEN SOMETHING BREAKS

LLM returns garbage: ollama restart Check: does senti-shujaa still load? Fallback: set PRIMARY_MODEL_NAME=qwen2.5 in .env

Database connection errors: docker-compose restart postgres Check: is disk full?

Superpack 404 errors: python scratch/start_all.py Check: are ports 9201-9216 listening? python scratch/check_ports.py

Memory leaks: Restart the gateway Check Redis: redis-cli info memory Check session memory TTL is 2 hours

Slow responses (> 30 seconds): Check: is Ollama using CPU or GPU? Check: is Redis available? Check: is FEE handling simple queries (not LLM)?

WHAT NEVER TO DO IN PRODUCTION

Never: rm -rf postgres_data/ docker-compose down --volumes Delete senti_db.sqlite without backup Change ENCRYPTION_KEY after users have data (all encrypted data becomes unreadable) Run migration scripts without testing on backup first