- Auto-enable ZeroGPU if credentials are provided (no need for USE_ZERO_GPU=true)
- ZeroGPU is always tried FIRST before local models or HF API
- Keep trying ZeroGPU even if initialization had warnings
- Update routing logic to check for client existence, not just use_zero_gpu flag
- Improve logging to clearly indicate ZeroGPU as PRIMARY inference method
- Update all initialization logs to show inference priority order
- Only fallback to local models or HF API if ZeroGPU actually fails
- Explicitly disable with USE_ZERO_GPU=false if needed
fix: Resolve database permission errors and OMP_NUM_THREADS warning
- Change default database path from sessions.db to /tmp/sessions.db (writable in containers)
- Update EfficientContextManager to accept db_path parameter and use settings
- Update all config files (config.py, src/config.py) to use /tmp for database paths
- Update FAISS index path to /tmp/embeddings.faiss
- Update ZeroGPUUserManager default db_path to /tmp/sessions.db
- Fix libgomp OMP_NUM_THREADS warning by setting ENV variables in Dockerfile
- Prevents sqlite3.OperationalError: unable to open database file
fix: Update remaining localhost references to Runpod URL and fix verification script syntax
- Update requirements.txt: Switch from faiss-cpu to faiss-gpu for GPU-accelerated vector search
- Update faiss_manager.py: Add GPU support with automatic CPU fallback
- GPU detection and resource management
- Automatic index migration between GPU/CPU
- Proper cleanup of GPU resources
- Status reporting for GPU availability
- Update src/llm_router.py: Reverse fallback order for lazy loading
- ZeroGPU API tried first (primary path)
- Local models only load if ZeroGPU fails (lazy loading)
- HF Inference API as final fallback
- Updated logging to indicate fallback path
Benefits:
- GPU-accelerated vector search (10-100x faster for large indices)
- Lazy loading: Models only load when ZeroGPU unavailable
- Lower memory usage: No models loaded if ZeroGPU works
- Automatic fallback: GPU → CPU if GPU unavailable
- Better resource utilization: GPU only used when needed
- Add ZeroGPUUserManager for per-user account management
- Implement automatic user registration and approval
- Add user mapping database table for local-to-API user mapping
- Update LLM router to support both service account and per-user modes
- Add per-user mode configuration options
- Update ZeroGPU client to accept pre-existing tokens
- Add comprehensive documentation for per-user mode
Per-user mode features:
- Automatic user registration on first use
- Per-user usage tracking and statistics
- Per-user rate limits
- Better audit trail per user
- Multi-tenant support
Configuration:
- Set ZERO_GPU_PER_USER_MODE=true to enable
- Requires ZERO_GPU_ADMIN_EMAIL and ZERO_GPU_ADMIN_PASSWORD
- Falls back to service account mode if per-user fails
Database:
- Creates zero_gpu_user_mapping table
- Stores user credentials, tokens, and mapping
- Automatic token refresh and management
- Add ZeroGPU API client (zero_gpu_client.py) with JWT authentication and auto-refresh
- Update LLM router to support ZeroGPU API as inference provider
- Add ZeroGPU configuration to config.py (enabled via USE_ZERO_GPU env var)
- Add task type mapping for ZeroGPU API (general, reasoning, classification, embedding)
- Update app.py and flask_api_standalone.py to pass ZeroGPU config to LLM router
- Implement fallback chain: Local models -> ZeroGPU API -> HF Inference API
- Add comprehensive integration review documentation
The ZeroGPU API provides:
- Built-in user management and authentication
- Comprehensive server-side logging and audit trail
- Task-based routing (general, reasoning, classification, embedding)
- Rich metadata (tokens, timing, quality metrics)
- Rate limiting and security features
Configuration:
- Set USE_ZERO_GPU=true to enable
- Set ZERO_GPU_API_URL, ZERO_GPU_EMAIL, ZERO_GPU_PASSWORD
- Falls back to HF API if ZeroGPU unavailable or disabled
