Commit History
-
Replace Novita AI with ZeroGPU Chat API (RunPod)
- Replace Novita AI API integration with ZeroGPU Chat API - Update llm_router.py to use aiohttp for HTTP requests with JWT authentication - Add automatic token refresh and authentication handling - Update config.py with ZeroGPU settings (base_url, email, password) - Update ENV_EXAMPLE_CONTENT.txt with ZeroGPU configuration - Update flask_api_standalone.py references - Remove OpenAI dependency from requirements.txt - Implement task type mapping (general_reasoning -> general, etc.) - Add context conversion for API format compatibility0747201
-
Fix indentation issues in prepare_context_for_llm and _truncate_to_tokens methods
bd329bc -
FINAL FIX: Correct except clause indentation - ensure proper alignment with try statement
f608c70 -
Fix return statement and else clause indentation in response handling
043d8e9 -
Fix else clause indentation at line 153 - align with if stream
21a18e3 -
Fix else clause indentation in response handling
a67b5f4 -
Fix raise statement indentation in exception handler
44b41ba -
CRITICAL FIX: Correct except clause indentation at line 58 - align with try statement
dc8667d -
Add logging for context window configuration and improve max_tokens calculation debugging
ca77f38 -
Update token allocation for DeepSeek R1 128K context window - Set NOVITA_MODEL_CONTEXT_WINDOW to 128000 (128K tokens) - Increase USER_INPUT_MAX_TOKENS to 32000 - Increase CONTEXT_PREPARATION_BUDGET to 115000 - Increase CONTEXT_PRUNING_THRESHOLD to 115000 - Update validators to support larger token limits
5d37f3d -
Fix IndentationError at line 378: correct indentation in _truncate_to_tokens method
9d31b94 -
CRITICAL FIX: Correct except clause indentation at line 57 - align with try statement
6c81140 -
Fix raise statement indentation
b21bcac -
Fix IndentationError: correct else clause alignment - ensure else aligns with if stream
5599138 -
Fix IndentationError: correct else clause alignment at line 152
074e6cc -
Fix critical syntax error: correct except clause indentation at line 57
82b52b6 -
Clean up try-except block formatting
e96d039 -
Fix encoding issue: replace special character in logger message
e4c5683 -
Fix except clause indentation in llm_router.py
d14bd01 -
Fix syntax errors in llm_router.py - Fix indentation issues in try/except blocks - Fix else clause alignment
bb2bbad -
Fix max_tokens calculation to respect model context window (5000 tokens) - Add dynamic max_tokens calculation based on input size - Add novita_model_context_window configuration - Prevents 400 errors when input tokens exceed available output space
a9135e0 -
Fix max_tokens calculation to respect model context window
42ed1a7 -
Integrate Novita AI as exclusive inference provider - Add Novita AI API integration with DeepSeek-R1-Distill-Qwen-7B model - Remove all local model dependencies - Optimize token allocation for user inputs and context - Add Anaconda environment setup files - Add comprehensive test scripts and documentation
927854c
-
Fix: DynamicCache compatibility, dependencies, and Docker configuration
CRITICAL FIXES: - Fixed DynamicCache.seen_tokens AttributeError for Phi-3 models - Pinned transformers to compatible version (4.36.0-4.41.0) - Added multi-level fallback chain (Mistral -> Phi-3) - Fixed Docker user creation to prevent getpwuid errors - Enhanced error handling for model generation Changes: - src/local_model_loader.py: - Added DynamicCache compatibility fix for Phi-3 models - Automatic use_cache=False for Phi-3 to avoid seen_tokens error - Retry logic with cache disabled if DynamicCache error occurs - Better error messages and logging - src/llm_router.py: - Enhanced fallback chain to support fallback + fallback2 - Sequential fallback attempts (Mistral -> Phi-3) - Better error handling for multi-level fallbacks - src/models_config.py: - Updated fallback chain: Mistral (primary fallback) -> Phi-3 (secondary) - Mistral is non-gated and has no DynamicCache issues - Phi-3 as secondary fallback with DynamicCache workaround - requirements.txt: - Pinned…ea87e33 -
Fix: GPU tensor placement and Docker deployment configuration
CRITICAL FIXES: - Fixed tensor device placement errors (meta device issues) - Added explicit device placement for non-quantized models - Updated Dockerfile with cache directory setup - Created comprehensive deployment configuration guide Changes: - src/local_model_loader.py: - Use device_map='auto' only with quantization (prevents meta device errors) - Explicit .to(device) placement for non-quantized models - Better logging for model loading status - Dockerfile: - Create cache directories with proper permissions - Set HF_HOME and TRANSFORMERS_CACHE environment variables - Ensure /tmp directories are writable - DEPLOYMENT_CONFIG_GUIDE.md (NEW): - Comprehensive guide for all deployment issues - Cache directory permission fixes - HF_TOKEN configuration - GPU tensor placement solutions - Troubleshooting steps - Verification checklist Fixes: - Tensor on device meta errors → Explicit device placement - Permission denied /cache errors → Dockerfile creates /tmp/cach…67c580c -
Fix: BitsAndBytes compatibility and error handling
CRITICAL FIXES: - Fixed BitsAndBytes kernel registration error handling - Distinguish between bitsandbytes errors and gated repository errors - Automatic fallback to loading without quantization if bitsandbytes fails - Changed fallback model to microsoft/Phi-3-mini-4k-instruct (verified non-gated) Changes: - src/local_model_loader.py: - Better error detection for bitsandbytes vs gated repo errors - Automatic fallback to no quantization if bitsandbytes fails - Improved error messages to distinguish error types - src/llm_router.py: - Added catch for bitsandbytes errors at router level - Automatic retry without quantization on bitsandbytes failures - src/models_config.py: - Changed fallback from mistralai/Mistral-7B-Instruct-v0.2 to microsoft/Phi-3-mini-4k-instruct - Phi-3-mini is verified non-gated and smaller (3.8B vs 7B) Fixes: - RuntimeError: int8_mm_dequant kernel registration conflict - ModuleNotFoundError: validate_bnb_backend_availability - False positive 'gated …13fa6c4 -
Fix: Cache directory permissions and gated repository handling
CRITICAL FIXES: - Fixed cache directory permission errors in Docker containers - Added HF_TOKEN authentication for gated repository access - Added non-gated fallback model (Mistral-7B-Instruct-v0.2) - Improved Docker detection to prefer /tmp over ~/.cache Changes: - src/local_model_loader.py: - Pass cache_dir to all from_pretrained calls - Set HF_HOME and TRANSFORMERS_CACHE environment variables - Authenticate with HF_TOKEN for gated repositories - Use cache_dir from settings config - src/config.py: - Improved Docker detection for cache directory selection - Prefer /tmp in Docker containers to avoid permission issues - src/models_config.py: - Added mistralai/Mistral-7B-Instruct-v0.2 as fallback model - All text tasks now have non-gated fallback option Fixes: - PermissionError: [Errno 13] Permission denied: '/.cache' - Gated repository access errors with proper fallback - HF_TOKEN authentication for gated models Ready for production testing.9959ea9 -
Phase 1: Remove HF API inference - Local models only
- Removed all Hugging Face API inference code (~165 lines) - Updated to use single primary model: Qwen/Qwen2.5-7B-Instruct - Removed API fallback logic - local models now required - Updated error handling to raise explicit errors instead of silent fallback - Updated documentation to reflect local-only model usage - Added validation test script and Week 1 retrospective Changes: - src/models_config.py: Single model config, removed API dependencies - src/llm_router.py: Removed _call_hf_endpoint, _is_model_healthy, _get_fallback_model methods - flask_api_standalone.py: Removed API fallback, requires local models - README.md: Updated to show HF_TOKEN is optional (only for gated models) Breaking changes: - HF_TOKEN is now optional (only needed for gated model downloads) - System requires local models - no API fallback - use_local_models=False will raise ValueError Ready for user testing.5787d0a -
Fix infinite fallback loop in local model loading
- Add _is_fallback flag to prevent infinite recursion - Remove fallback from fallback config to prevent loops - Check if fallback model is different from primary before attempting - Better error handling when both primary and fallback models are gated - Prevent recursive fallback attempts when fallback also fails8d4bf4a -
Update model IDs to use Cerebras deployment and add gated repository error handling
- Updated model IDs to use meta-llama/Llama-3.1-8B-Instruct:cerebras across all model configurations - Added comprehensive GatedRepoError handling in local_model_loader.py - Added GatedRepoError handling in llm_router.py with fallback model support - Implemented API suffix stripping (:cerebras) for local model loading - Updated default model configurations in config.py - Added helpful error messages with links to request repository accessb3aba24 -
Security Enhancements: Production WSGI, Rate Limiting, Security Headers, Secure Logging
- Added Gunicorn production WSGI server (replaces Flask dev server) - Implemented rate limiting with Flask-Limiter (10/min chat, 5/min initialize) - Added comprehensive security headers (10 headers including Phase 1 enhancements) - Implemented secure logging with file rotation and sensitive data sanitization - Added OMP_NUM_THREADS validation to prevent invalid environment variable errors - Added database indexes for performance optimization - Created production startup script with environment validation - Added security audit and check scripts - Updated Dockerfile for production deployment - Added security tools (Bandit, Safety) to requirements.txt - Created comprehensive security documentation and roadmap - Enhanced configuration management with secure defaults79ea999
-
Fix UnboundLocalError: remove redundant os import in exception handler
48f2898 -
Improve database path handling with Docker detection and fallback to /tmp
a4d66c0 -
Fix database path issue for Docker/HF Spaces - use /tmp for writable directory
4e3f60e -
Initial commit: Research AI Assistant API
8f4d405