- Ensure both Dockerfile and Dockerfile.gpu include latest main.py
- Picks up all syntax fixes for production deployment
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Copy main.py to Docker image to pick up syntax fixes
Andrew McCracken+1: Claude
7c501a5
- Ensures latest main.py with f-string fixes is included
- Fixes SyntaxError in streaming endpoint
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Fix remaining f-string syntax errors
Andrew McCracken+1: Claude
8f62d83
- Fix 'end' and 'error' data serialization
- Extract all dict literals outside f-strings
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Fix f-string syntax error in streaming endpoint
Andrew McCracken+1: Claude
c53e66f
- Extract dict construction outside f-string
- Fixes SyntaxError with nested braces in json.dumps
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Switch to GPU-enabled Docker image
Andrew McCracken+1: Claude
2f6841c
- Using techdaskalos/cybersecchatbot:gpu from Docker Hub
- Enable GPU acceleration with N_GPU_LAYERS=35
- Configure MODEL_POOL_SIZE=10 for concurrent users
- Requires GPU hardware in HF Space settings
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Add thread-safety for concurrent users
Andrew McCracken+1: Claude
cfc97b4
Fixed critical thread-safety issues:
- Added sessions_lock for thread-safe session management
- Added db_lock for thread-safe SQLite operations
- All session dict access wrapped with locks
- All database operations wrapped with locks
- SQLite connections now use check_same_thread=False
System now fully supports 10 concurrent users with:
- Thread-safe session isolation
- Thread-safe database operations
- Model pool for concurrent inference
- Proper uvicorn configuration
Ready for deployment with 10 model instances on GPU.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Configure uvicorn for concurrent request handling
Andrew McCracken+1: Claude
6b0a701
Updated uvicorn settings for optimal concurrency:
- workers=1: Share model pool across all requests (can't share across processes)
- limit_concurrency=100: Handle up to 100 simultaneous connections
- timeout_keep_alive=120: Support long streaming responses
- backlog=2048: Queue pending connections
- loop='asyncio': Best async performance
With these settings + ModelPool, the app can:
- Accept 100 concurrent connections
- Process 10 simultaneous inferences (model pool size)
- Queue remaining requests gracefully
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Add concurrent request handling with model pool
Andrew McCracken+1: Claude
efd4459
Implemented ModelPool for true concurrent processing:
- Created ModelPool class with thread-safe queue
- Initializes 10 model instances (configurable via MODEL_POOL_SIZE)
- Each instance can handle one request simultaneously
- Automatic model checkout/return from pool
- Added pool statistics to /health endpoint
Configuration:
- MODEL_POOL_SIZE=10 (supports 10 concurrent users)
- 60s timeout if all instances busy
- Each model instance ~2.4GB VRAM
- Total VRAM: ~24GB for 10 instances (fits in 48GB GPU)
Sessions are handled via session_id parameter (already present)
Pool automatically balances load across instances
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Add GPU support
Andrew McCracken+1: Claude
bfa102d
Added GPU-enabled Docker configuration:
- Dockerfile.base.gpu: CUDA 12.1 base with llama-cpp-python GPU support
- Dockerfile.gpu: HF Spaces GPU deployment dockerfile
- build-and-push-gpu.sh: Script to build and push GPU image
- Updated llm_handler.py to use N_GPU_LAYERS env variable
To use GPU:
1. Build: ./build-and-push-gpu.sh
2. Switch HF Space to GPU hardware
3. Use Dockerfile.gpu for deployment
Expected speedup: ~15s → 1-3s per response
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Revert to simpler configuration - optimizations caused slowdown
Andrew McCracken+1: Claude
457c9e1
- Reverted n_ctx back to 4096 (2048 was too small)
- Reverted streaming to simple token-by-token (buffering caused issues)
- Reduced threads to 6 (8 may have been overloading)
- Kept max_tokens at 256 (only change that should help)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Optimize for faster inference
Andrew McCracken+1: Claude
8cfe5b7
Performance optimizations:
1. Reduced max_tokens from 512 to 256 (faster responses)
2. Reduced n_ctx from 4096 to 2048 (faster prompt processing)
3. Added token buffering in streaming (better perceived speed)
- Buffers 3 tokens or until whitespace
- Reduces network overhead
Expected speedup: 15s → 8-10s per response
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Increase threads to 8 for faster inference
Andrew McCracken+1: Claude
3f2ee19
- Use all 8 vCPUs for maximum inference speed
- Should reduce response time from ~15s to ~10-12s
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Disable RAG for faster inference
Andrew McCracken+1: Claude
2a55dc3
- Set USE_RAG=false to remove knowledge base overhead
- Should significantly reduce response time on CPU instance
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Optimize for 8vCPU/32GB instance
Andrew McCracken+1: Claude
b7fb901
- Use 6 threads (leave 2 for system/API)
- Restore n_ctx=4096 and n_batch=512 for better quality
- Restore max_tokens=512 for complete responses
- Enable verbose logging for debugging
- f16_kv=True for memory efficiency with sufficient RAM
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Optimize model parameters for faster CPU inference
Andrew McCracken+1: Claude
6e83384
- Reduced n_ctx from 4096 to 2048 for faster inference
- Reduced n_batch from 512 to 256 for lower memory usage
- Reduced n_threads from 8 to 4 for HF Spaces CPU limits
- Changed f16_kv to False for better CPU performance
- Reduced default max_tokens from 512 to 256 for faster responses
These changes need to be rebuilt into the base Docker image.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Fix API endpoint to use relative URL instead of localhost
Andrew McCracken+1: Claude
1b98923
- Changed fetch URL from 'http://localhost:8000/chat/stream' to '/chat/stream'
- Fixes chat functionality on HF Spaces deployment
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Add test interface HTML to Docker image
Andrew McCracken+1: Claude
403caa4
- Copy test_interface.html to /app/ in Docker image
- Fixes 404 error on /test endpoint
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Add knowledge_db directory and remove deprecated env var
Andrew McCracken+1: Claude
7942e87
- Create /app/knowledge_db directory with write permissions
- Remove deprecated TRANSFORMERS_CACHE env var (use HF_HOME instead)
- Fixes PermissionError: [Errno 13] Permission denied: './knowledge_db'
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Fix Hugging Face cache permissions
Andrew McCracken+1: Claude
b721fc6
- Set HF_HOME, TRANSFORMERS_CACHE, HF_HUB_CACHE to /data/huggingface
- Create all required cache directories with proper permissions
- Fixes RuntimeError: Data processing error: I/O error: Permission denied
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Fix /app/models directory permissions
Andrew McCracken+1: Claude
3b792bc
- Create /app/models directory with write permissions
- Fixes PermissionError: [Errno 13] Permission denied: './models'
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Fix /data directory permissions for HF Spaces
Andrew McCracken+1: Claude
28a4990
- Create /data directory with write permissions
- Fixes sqlite3.OperationalError: unable to open database file
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Use pre-built Docker image from Docker Hub
Andrew McCracken+1: Claude
cf74856
- Updated Dockerfile to use techdaskalos/cybersecchatbot:latest
- Removed install_llama.py and start.sh (no longer needed)
- Updated requirements.txt to use llama_cpp_python without version pin
- Simplified deployment using pre-built AMD64 image
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Use pre-built Docker image from Docker Hub
Andrew McCracken
ccc0289
Fix: Install llama-cpp-python at startup to /tmp to avoid build timeout
Andrew McCracken
0a1ff0d
Fix: Install cmake and build-essential for llama-cpp-python build
Andrew McCracken
8e854ef
Fix: Force use of pre-built llama-cpp-python wheels
Andrew McCracken
a922ca8
Fix: Install llama-cpp-python at runtime (HF Spaces workaround)
Andrew McCracken
3fad655
Optimize: Use pre-built llama-cpp-python wheels for faster builds
