refactor: enhance model unloading and memory management for improved GPU efficiency

refactor: enable CPU offload and adjust model loading for improved performance

refactor: disable KV cache to prevent quality degradation after multiple requests

refactor: enable 8-bit quantization and adjust device map for improved model loading diagnostics

refactor: disable 8-bit quantization and set device map to CPU when GPU is unavailable

refactor: enable 8-bit quantization for improved memory efficiency in Transformers model loading

refactor: disable 8-bit quantization and CPU offload for optimized model loading on T4 GPUs

fix: improve error handling during model loading and fallback for quantization failures

refactor: remove unused model configurations and streamline model creation logic

fix: streamline CPU offload handling in model loading for better memory management

feat: add CPU offload support for Transformers model to optimize memory usage

feat: add Transformers model support with GPU optimization and 8-bit quantization

feat: add model size and polish support to model info

feat: enable GPU acceleration for Bielik GGUF models

update Dockerfile and README.md to replace Qwen2.5-3B and Gemma-2-2B with Bielik-1.5B-GGUF; adjust model loading instructions in the API documentation

update HuggingFaceInferenceAPI comment for clarity; change huggingface_hub version to minimum required

add model management methods to ModelRegistry; include model listing, loading, and unloading functionalities

add HuggingFace Inference API model; implement async initialization and text generation with caching

add GBNF grammar for car advertisement gap filling; update LlamaCppModel to support loading grammar from file

add GBNF grammar utilities for structured LLM output; integrate grammar in model generation

update LlamaCppModel initialization parameters and enable verbose logging for model loading; update llama-cpp-python requirement

enhance error handling in LlamaCppModel initialization; include full traceback on failure

add get_info method to return model details for /models endpoint

add debug logging for batch infill and model generation processes; update bielik model configuration

increase context size and improve message handling in LlamaCppModel

adding-bielik-gguf

fix: Make 8-bit quantization opt-in and gracefully handle missing bitsandbytes

fix: Add bitsandbytes to requirements and graceful fallback for 8-bit quantization

Add 8-bit quantization support for CPU environments (4-6x speedup)

Fix: Remove unsupported use_xformers_attention parameter

Fix: Use direct model.generate() with proper KV caching instead of pipeline

Add KV caching and batch processing optimizations for 5-10x speedup

Fix gemma chat template fallback

pre-downloading-all-models-at-startup

model-lazy-loading

first-imrpvement-commit

adding-github-files-to-spaces