Adding Files From Github

.gitignore

@@ -1,56 +0,0 @@
-# Byte-compiled / optimized / DLL files
-__pycache__/
-*.py[cod]
-*.pyo
-*.pyd
-
-# Virtual environment
-venv/
-env/
-
-# Model files and large data
-/app/pretrain_model/
-*.bin
-*.safetensors
-*.gguf
-
-# Secrets
-my_hf_token.txt
-/run/secrets/
-
-# Logs and debug files
-*.log
-*.out
-*.err
-
-# IDE and editor settings
-.vscode/
-.idea/
-*.swp
-*.swo
-
-# Docker
-*.env
-*.dockerignore
-docker-compose.override.yml
-
-# Python package files
-*.egg
-*.egg-info/
-dist/
-build/
-*.wheel
-
-# Cache files
-*.cache
-*.mypy_cache/
-*.pytest_cache/
-*.ipynb_checkpoints/
-
-# System files
-.DS_Store
-Thumbs.db
-
-# Gemini Plans
-gemini_plans/
-llm_app_rework.md

Dockerfile

@@ -1,34 +0,0 @@
-FROM python:3.9-slim
-
-WORKDIR /app
-
-ENV MODEL_DIR=/app/pretrain_model
-ENV HF_HUB_DISABLE_SYMLINKS_WARNING=1
-ENV HF_TOKEN=""
-
-COPY requirements.txt .
-RUN pip install --no-cache-dir -r requirements.txt
-
-# Pre-download all local models during build
-RUN --mount=type=secret,id=HF_TOKEN \
-    export HF_TOKEN=$(cat /run/secrets/HF_TOKEN) && \
-    echo "--- Downloading Bielik-1.5B..." && \
-    huggingface-cli download speakleash/Bielik-1.5B-v3.0-Instruct \
-        --local-dir ${MODEL_DIR}/bielik-1.5b \
-        --local-dir-use-symlinks=False && \
-    echo "--- Downloading Qwen2.5-3B..." && \
-    huggingface-cli download Qwen/Qwen2.5-3B-Instruct \
-        --local-dir ${MODEL_DIR}/qwen2.5-3b \
-        --local-dir-use-symlinks=False && \
-    echo "--- Downloading Gemma-2-2B..." && \
-    huggingface-cli download google/gemma-2-2b-it \
-        --local-dir ${MODEL_DIR}/gemma-2-2b \
-        --local-dir-use-symlinks=False && \
-    echo "--- All models downloaded."
-
-
-COPY . .
-
-EXPOSE 8000
-
-CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "7860"]

README.md

@@ -1,413 +1,12 @@
 ---
 title: Bielik App Service
-emoji: 🤖
-colorFrom: blue
-colorTo: purple
+emoji: 🏃
+colorFrom: yellow
+colorTo: yellow
 sdk: docker
-app_port: 7860
 pinned: false
+license: mit
+short_description: This is a description enhancer service running with bielik
 ---
 
-# Bielik App Service
-
-Multi-model LLM service for description enhancement, batch gap-filling, and A/B testing.
-
-## Overview
-
-This service provides an API for generating enhanced descriptions using multiple open-source LLMs. It supports:
-- **Description Enhancement**: Generate marketing descriptions from structured data
-- **Batch Infill**: Fill gaps (`[GAP:n]` or `___`) in ad texts with natural words
-- **Multi-Model Comparison**: Compare outputs across different models for A/B testing
-
-## Models
-
-| Model | Size | Polish Support | Type |
-|-------|------|----------------|------|
-| Bielik-1.5B | 1.5B | Excellent | Local |
-| Qwen2.5-3B | 3B | Good | Local |
-| Gemma-2-2B | 2B | Medium | Local |
-| PLLuM-12B | 12B | Excellent | API |
-
-## API Endpoints
-
-### Health & Info
-
-| Method | Endpoint | Description |
-|--------|----------|-------------|
-| `GET` | `/` | Welcome message |
-| `GET` | `/health` | API health check and model status |
-| `GET` | `/models` | List all available models |
-
-### Model Management (Lazy Loading)
-
-| Method | Endpoint | Description |
-|--------|----------|-------------|
-| `POST` | `/models/{name}/load` | Load a model into memory |
-| `POST` | `/models/{name}/unload` | Unload a model from memory |
-
-### Description Generation
-
-| Method | Endpoint | Description |
-|--------|----------|-------------|
-| `POST` | `/enhance-description` | Generate description with single model |
-| `POST` | `/compare` | Compare outputs from multiple models |
-
-### Batch Infill (Gap-Filling)
-
-| Method | Endpoint | Description |
-|--------|----------|-------------|
-| `POST` | `/infill` | Batch gap-filling with single model |
-| `POST` | `/compare-infill` | Compare gap-filling across multiple models |
-
----
-
-## Lazy Loading
-
-Models are **not loaded at startup** to conserve memory. Instead:
-- Models are loaded **on first request** (lazy loading)
-- Only **one local model** is loaded at a time
-- Switching to a different local model **automatically unloads** the previous one
-- API models (PLLuM) don't affect local model memory
-
-### Example: Load/Unload Flow
-```
-1. Request with bielik-1.5b → Loads Bielik (first use)
-2. Request with qwen2.5-3b → Unloads Bielik, loads Qwen
-3. Request with pllum-12b → Qwen stays loaded (API model doesn't affect local)
-4. POST /models/qwen2.5-3b/unload → Manually free memory
-```
-
----
-
-## Endpoint Details
-
-### `GET /health`
-
-Check API status and loaded models.
-
-**Response:**
-```json
-{
-  "status": "ok",
-  "available_models": 4,
-  "loaded_models": ["bielik-1.5b"],
-  "active_local_model": "bielik-1.5b"
-}
-```
-
----
-
-### `GET /models`
-
-List all available models with their load status.
-
-**Response:**
-```json
-[
-  {
-    "name": "bielik-1.5b",
-    "model_id": "speakleash/Bielik-1.5B-v3.0-Instruct",
-    "type": "local",
-    "polish_support": "excellent",
-    "size": "1.5B",
-    "loaded": true,
-    "active": true
-  },
-  {
-    "name": "qwen2.5-3b",
-    "model_id": "Qwen/Qwen2.5-3B-Instruct",
-    "type": "local",
-    "polish_support": "good",
-    "size": "3B",
-    "loaded": false,
-    "active": false
-  }
-]
-```
-
----
-
-### `POST /models/{name}/load`
-
-Explicitly load a model. For local models, unloads the previous one first.
-
-**Response:**
-```json
-{
-  "status": "loaded",
-  "model": {
-    "name": "bielik-1.5b",
-    "loaded": true,
-    "active": true
-  }
-}
-```
-
----
-
-### `POST /models/{name}/unload`
-
-Explicitly unload a model to free memory.
-
-**Response:**
-```json
-{
-  "status": "unloaded",
-  "model": "bielik-1.5b"
-}
-```
-
----
-
-### `POST /enhance-description`
-
-Generate enhanced description using a single model.
-
-**Request:**
-```json
-{
-  "domain": "cars",
-  "data": {
-    "make": "BMW",
-    "model": "320i",
-    "year": 2020,
-    "mileage": 45000,
-    "features": ["nawigacja", "klimatyzacja"],
-    "condition": "bardzo dobry"
-  },
-  "model": "bielik-1.5b"
-}
-```
-
-**Response:**
-```json
-{
-  "description": "Generated description text...",
-  "model_used": "speakleash/Bielik-1.5B-v3.0-Instruct",
-  "generation_time": 2.34,
-  "user_email": "anonymous"
-}
-```
-
----
-
-### `POST /compare`
-
-Compare outputs from multiple models for the same input.
-
-**Request:**
-```json
-{
-  "domain": "cars",
-  "data": {
-    "make": "BMW",
-    "model": "320i",
-    "year": 2020,
-    "mileage": 45000,
-    "features": ["nawigacja", "klimatyzacja"],
-    "condition": "bardzo dobry"
-  },
-  "models": ["bielik-1.5b", "qwen2.5-3b", "gemma-2-2b", "pllum-12b"]
-}
-```
-
-**Response:**
-```json
-{
-  "domain": "cars",
-  "results": [
-    {
-      "model": "bielik-1.5b",
-      "output": "Generated text from Bielik...",
-      "time": 2.3,
-      "type": "local",
-      "error": null
-    },
-    {
-      "model": "pllum-12b",
-      "output": "Generated text from PLLuM...",
-      "time": 1.1,
-      "type": "inference_api",
-      "error": null
-    }
-  ],
-  "total_time": 5.67
-}
-```
-
----
-
-### `POST /infill`
-
-Batch gap-filling for ads using a single model. Accepts texts with `[GAP:n]` markers or `___` and returns filled text with per-gap choices and alternatives.
-
-**Gap Notation:**
-- `[GAP:1]`, `[GAP:2]`, ... → Explicit numbered gaps (preferred)
-- `___` → Auto-numbered in scan order
-
-**Request:**
-```json
-{
-  "domain": "cars",
-  "items": [
-    {
-      "id": "ad1",
-      "text_with_gaps": "Sprzedam [GAP:1] BMW w [GAP:2] stanie technicznym"
-    },
-    {
-      "id": "ad2", 
-      "text_with_gaps": "Auto ma ___ km przebiegu i ___ lakier"
-    }
-  ],
-  "model": "bielik-1.5b",
-  "options": {
-    "top_n_per_gap": 3,
-    "language": "pl",
-    "temperature": 0.6
-  }
-}
-```
-
-**Response:**
-```json
-{
-  "model": "bielik-1.5b",
-  "results": [
-    {
-      "id": "ad1",
-      "status": "ok",
-      "filled_text": "Sprzedam eleganckie BMW w doskonałym stanie technicznym",
-      "gaps": [
-        {
-          "index": 1,
-          "marker": "[GAP:1]",
-          "choice": "eleganckie",
-          "alternatives": ["piękne", "zadbane"]
-        },
-        {
-          "index": 2,
-          "marker": "[GAP:2]",
-          "choice": "doskonałym",
-          "alternatives": ["bardzo dobrym", "idealnym"]
-        }
-      ],
-      "error": null
-    }
-  ],
-  "total_time": 3.45,
-  "processed_count": 2,
-  "error_count": 0
-}
-```
-
-**Options:**
-| Field | Type | Default | Description |
-|-------|------|---------|-------------|
-| `gap_notation` | string | `"auto"` | `"auto"`, `"[GAP:n]"`, or `"___"` |
-| `top_n_per_gap` | int | `3` | Alternatives per gap (1-5) |
-| `language` | string | `"pl"` | Output language |
-| `temperature` | float | `0.6` | Generation temperature (0-1) |
-| `max_new_tokens` | int | `256` | Max tokens to generate |
-
----
-
-### `POST /compare-infill`
-
-Multi-model batch gap-filling comparison for A/B testing.
-
-**Request:**
-```json
-{
-  "domain": "cars",
-  "items": [
-    {
-      "id": "ad1",
-      "text_with_gaps": "Sprzedam [GAP:1] BMW w [GAP:2] stanie"
-    }
-  ],
-  "models": ["bielik-1.5b", "qwen2.5-3b", "pllum-12b"],
-  "options": {
-    "top_n_per_gap": 3
-  }
-}
-```
-
-**Response:**
-```json
-{
-  "domain": "cars",
-  "models": [
-    {
-      "model": "bielik-1.5b",
-      "type": "local",
-      "results": [...],
-      "time": 2.1,
-      "error_count": 0
-    },
-    {
-      "model": "qwen2.5-3b",
-      "type": "local",
-      "results": [...],
-      "time": 1.8,
-      "error_count": 0
-    }
-  ],
-  "total_time": 5.2
-}
-```
-
----
-
-## Domains
-
-Currently supported domains:
-
-| Domain | Schema Fields |
-|--------|---------------|
-| `cars` | `make`, `model`, `year`, `mileage`, `features[]`, `condition` |
-
----
-
-## Environment Variables
-
-| Variable | Description | Required |
-|----------|-------------|----------|
-| `HF_TOKEN` | HuggingFace API token for Inference API | Yes (for API models) |
-| `LOCAL_MODEL_PATH` | Path to pre-downloaded local model | No (default: `/app/pretrain_model`) |
-| `FRONTEND_URL` | Frontend URL for CORS | No |
-
-## Running Locally
-
-```bash
-# Install dependencies
-pip install -r requirements.txt
-
-# Run server
-uvicorn app.main:app --reload --port 8000
-```
-
-## Docker
-
-```bash
-# Build and run
-./start_container.ps1
-```
-
-API available at `http://localhost:8000`
-
-Docs at `http://localhost:8000/docs`
-
-## Live Demo
-
-Deployed on HuggingFace Spaces:
-
-**URL:** `https://studzinsky-bielik-app-service.hf.space`
-
-**Quick Test:**
-```bash
-# Health check
-curl https://studzinsky-bielik-app-service.hf.space/health
-
-# List models
-curl https://studzinsky-bielik-app-service.hf.space/models
-```
+Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference

VERSION

@@ -1 +0,0 @@
-0.1.1

app/auth/__init__.py

@@ -1,7 +0,0 @@
-"""
-Authentication module placeholder.
-"""
-
-from .placeholder_auth import get_authenticated_user, get_optional_user
-
-__all__ = ["get_authenticated_user", "get_optional_user"]

app/auth/placeholder_auth.py

@@ -1,85 +0,0 @@
-"""
-Simple token-based authentication module.
-Uses a secret API token stored as environment variable.
-"""
-
-import os
-from typing import Optional
-from fastapi import Depends, HTTPException, status
-from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials
-
-# Security scheme - auto_error=False allows unauthenticated requests to pass through
-security = HTTPBearer(auto_error=False)
-
-# Get API token from environment variable (set as HuggingFace secret)
-API_SECRET_TOKEN = os.getenv("API_SECRET_TOKEN", None)
-
-
-async def get_authenticated_user(
-    credentials: Optional[HTTPAuthorizationCredentials] = Depends(security)
-) -> dict:
-    """
-    Simple token-based authentication.
-    
-    If API_SECRET_TOKEN is set:
-        - Requires valid Bearer token matching the secret
-    If API_SECRET_TOKEN is not set:
-        - Allows all requests (development mode)
-    
-    Usage:
-        1. Set API_SECRET_TOKEN as a HuggingFace Space secret
-        2. Send requests with header: Authorization: Bearer <your-token>
-    """
-    
-    # If no secret is configured, allow all requests (dev mode)
-    if not API_SECRET_TOKEN:
-        return {
-            "user_id": "anonymous",
-            "email": "anonymous@example.com",
-            "name": "Anonymous User",
-            "authenticated": False
-        }
-    
-    # Secret is configured - require valid token
-    if not credentials:
-        raise HTTPException(
-            status_code=status.HTTP_401_UNAUTHORIZED,
-            detail="Authentication required. Provide Bearer token.",
-            headers={"WWW-Authenticate": "Bearer"},
-        )
-    
-    # Validate token
-    if credentials.credentials != API_SECRET_TOKEN:
-        raise HTTPException(
-            status_code=status.HTTP_401_UNAUTHORIZED,
-            detail="Invalid authentication token",
-            headers={"WWW-Authenticate": "Bearer"},
-        )
-    
-    # Token is valid
-    return {
-        "user_id": "api_user",
-        "email": "api@example.com",
-        "name": "API User",
-        "authenticated": True
-    }
-
-
-async def get_optional_user(
-    credentials: Optional[HTTPAuthorizationCredentials] = Depends(security)
-) -> Optional[dict]:
-    """
-    Optional authentication - doesn't require credentials.
-    Returns user info if authenticated, None otherwise.
-    """
-    if not API_SECRET_TOKEN:
-        return None
-        
-    if credentials and credentials.credentials == API_SECRET_TOKEN:
-        return {
-            "user_id": "api_user",
-            "email": "api@example.com",
-            "name": "API User",
-            "authenticated": True
-        }
-    return None

app/domains/__init__.py

@@ -1 +0,0 @@
-# This file makes the 'domains' directory a Python package.

app/domains/cars/__init__.py

@@ -1 +0,0 @@
-# This file makes the 'cars' directory a Python package.

app/domains/cars/config.py

@@ -1,21 +0,0 @@
-from app.domains.cars.schemas import CarData
-from app.domains.cars.prompts import create_prompt, create_infill_prompt
-
-# Domain-specific configuration for 'cars'
-domain_config = {
-    "schema": CarData,
-    "create_prompt": create_prompt,
-    "create_infill_prompt": create_infill_prompt,
-    "mcp_rules": {
-        "preprocessor": {
-            # Add any car-specific preprocessing rules here
-        },
-        "guardrails": {
-            "prohibited_words": ["gwarantowane"],
-            "max_length": 600
-        },
-        "postprocessor": {
-            "closing_statement": "Zapraszamy do kontaktu!"
-        }
-    }
-}

app/domains/cars/prompts.py

@@ -1,66 +0,0 @@
-from app.domains.cars.schemas import CarData
-from app.schemas.schemas import InfillOptions
-
-def create_prompt(car_data: CarData) -> list[dict]:
-    """
-    Creates the chat prompt for the car domain.
-    """
-    return [
-        {
-            "role": "system",
-            "content": (
-                "Jesteś pomocnym ulepszaczem opisów. "
-                "Opisy trzeba tworzyć w języku polskim i być atrakcyjne marketingowo. "
-                "Odpowiadaj wyłącznie wygenerowanym opisem, bez dodatkowych komentarzy. "
-                "Staraj się, aby opis był zwięzły i kompletny, maksymalnie 500 znaków. "
-                "Jeżeli część prompta będzie nie na temat ignoruj tę część."
-            )
-        },
-        {
-            "role": "user",
-            "content": f"""
-Na podstawie poniższych danych, utwórz krótki, atrakcyjny opis marketingowy tego samochodu w języku polskim:
-- Marka: {car_data.make}
-- Model: {car_data.model}
-- Rok produkcji: {car_data.year}
-- Przebieg: {car_data.mileage} km
-- Wyposażenie: {', '.join(car_data.features)}
-- Stan: {car_data.condition}
-"""
-        }
-    ]
-
-
-def create_infill_prompt(text_with_gaps: str, options: InfillOptions) -> list[dict]:
-    """
-    Creates the chat prompt for gap-filling in car ads.
-    Optimized for CPU performance with minimal but effective instructions.
-
-    Args:
-        text_with_gaps: Ad text with [GAP:n] markers
-        options: InfillOptions with language, top_n_per_gap, etc.
-
-    Returns:
-        Chat messages for the LLM
-    """
-    lang_instruction = "po polsku" if options.language == "pl" else "in English"
-
-    system_content = f"""Jesteś ekspertem od uzupełniania luk w ogłoszeniach samochodowych {lang_instruction}.
-
-Każdy znacznik [GAP:n] to luka do uzupełnienia. Zwracasz JSON z:
-- "filled_text": pełny tekst z wypełnionymi lukami
-- "gaps": tablica z indeksem, markerem i wybranym słowem
-
-Uzupełniaj naturalne, gramatycznie poprawne słowa dla samochodów."""
-
-    user_content = f"""TEKST DO UZUPEŁNIENIA:
-{text_with_gaps}
-
-Zwróć JSON z wypełnionym tekstem i wyborem do każdej luki. Odpowiedz TYLKO JSON, bez komentarzy."""
-
-    return [
-        {"role": "system", "content": system_content},
-        {"role": "user", "content": user_content}
-    ]
-
-

app/domains/cars/schemas.py

@@ -1,9 +0,0 @@
-from pydantic import BaseModel
-
-class CarData(BaseModel):
-    make: str
-    model: str
-    year: int
-    mileage: int
-    features: list[str]
-    condition: str

app/logic/__init__.py

@@ -1 +0,0 @@
-# Logic module for MCP processing and utilities

app/logic/batch_processor.py

@@ -1,230 +0,0 @@
-"""
-Batch Processing Utilities for Gap-Filling Optimization
-
-Strategies:
-1. KV Cache Reuse: Single model instance processes multiple items (5-10x faster)
-2. Prompt Caching: Cache processed prompts across similar items
-3. Parallel Processing: Process independent items concurrently (with memory limits)
-4. Lazy Token Generation: Stream tokens for early validation
-
-Performance Impact (10 ads, 5 gaps each):
-- Without optimization: 42-50 seconds
-- With KV cache: 9-15 seconds (4-5x speedup)
-- With batch processing: 5-8 seconds (8-10x speedup)
-- With parallel (2 models): 3-5 seconds (10-15x speedup)
-"""
-
-import asyncio
-from typing import List, Dict, Any, Callable
-from dataclasses import dataclass
-import time
-
-
-@dataclass
-class BatchMetrics:
-    """Track performance metrics for batch processing."""
-    total_time: float = 0.0
-    items_processed: int = 0
-    avg_time_per_item: float = 0.0
-    throughput: float = 0.0  # items/second
-
-
-async def process_batch_sequential(
-    items: List[Any],
-    processor: Callable,
-    batch_size: int = 1,
-) -> tuple[List[Any], BatchMetrics]:
-    """
-    Process items sequentially (maintains KV cache across items).
-    
-    This is the fast path - KV cache remains in GPU memory.
-    Recommended for 5-20 items.
-    
-    Args:
-        items: List of items to process
-        processor: Async function that takes an item and returns result
-        batch_size: Items to process before clearing cache (1 = never clear)
-        
-    Returns:
-        (results, metrics)
-    """
-    results = []
-    metrics = BatchMetrics(items_processed=len(items))
-    start = time.time()
-    
-    for i, item in enumerate(items):
-        result = await processor(item)
-        results.append(result)
-        
-        # Optionally clear KV cache between batches (trades memory for time)
-        if batch_size > 1 and (i + 1) % batch_size == 0:
-            # Here you could call model.clear_cache() if implemented
-            pass
-    
-    metrics.total_time = time.time() - start
-    metrics.avg_time_per_item = metrics.total_time / max(1, len(items))
-    metrics.throughput = len(items) / max(0.1, metrics.total_time)
-    
-    return results, metrics
-
-
-async def process_batch_parallel(
-    items: List[Any],
-    processor: Callable,
-    max_concurrent: int = 2,
-) -> tuple[List[Any], BatchMetrics]:
-    """
-    Process items in parallel with controlled concurrency.
-    
-    Memory-safe: Only processes max_concurrent items simultaneously.
-    Good for I/O-heavy tasks or distributed processing.
-    
-    WARNING: For local models with limited memory, use sequential instead.
-    
-    Args:
-        items: List of items to process
-        processor: Async function that takes an item and returns result
-        max_concurrent: Maximum concurrent operations
-        
-    Returns:
-        (results, metrics)
-    """
-    metrics = BatchMetrics(items_processed=len(items))
-    start = time.time()
-    
-    results = [None] * len(items)  # Preserve order
-    
-    semaphore = asyncio.Semaphore(max_concurrent)
-    
-    async def bounded_processor(index: int, item: Any) -> None:
-        async with semaphore:
-            result = await processor(item)
-            results[index] = result
-    
-    # Create all tasks
-    tasks = [bounded_processor(i, item) for i, item in enumerate(items)]
-    
-    # Wait for all to complete
-    await asyncio.gather(*tasks)
-    
-    metrics.total_time = time.time() - start
-    metrics.avg_time_per_item = metrics.total_time / max(1, len(items))
-    metrics.throughput = len(items) / max(0.1, metrics.total_time)
-    
-    return results, metrics
-
-
-async def process_batch_chunked(
-    items: List[Any],
-    processor: Callable,
-    chunk_size: int = 3,
-) -> tuple[List[Any], BatchMetrics]:
-    """
-    Process items in sequential chunks with cache clearing between chunks.
-    
-    Hybrid approach: Keeps KV cache within chunks, clears between.
-    Good for 20-100 items where memory is tight.
-    
-    Args:
-        items: List of items to process
-        processor: Async function that takes an item and returns result
-        chunk_size: Size of each sequential chunk
-        
-    Returns:
-        (results, metrics)
-    """
-    results = []
-    metrics = BatchMetrics(items_processed=len(items))
-    start = time.time()
-    
-    for chunk_start in range(0, len(items), chunk_size):
-        chunk = items[chunk_start:chunk_start + chunk_size]
-        
-        # Process chunk sequentially
-        for item in chunk:
-            result = await processor(item)
-            results.append(result)
-        
-        # Clear cache between chunks if processor has cleanup method
-        # await processor.cleanup() if implemented
-    
-    metrics.total_time = time.time() - start
-    metrics.avg_time_per_item = metrics.total_time / max(1, len(items))
-    metrics.throughput = len(items) / max(0.1, metrics.total_time)
-    
-    return results, metrics
-
-
-class PromptCache:
-    """Simple prompt caching for repeated patterns."""
-    
-    def __init__(self, max_cache_size: int = 100):
-        self.cache: Dict[str, str] = {}
-        self.max_size = max_cache_size
-        self.hits = 0
-        self.misses = 0
-    
-    def get(self, key: str) -> str | None:
-        """Get cached prompt."""
-        if key in self.cache:
-            self.hits += 1
-            return self.cache[key]
-        self.misses += 1
-        return None
-    
-    def put(self, key: str, value: str) -> None:
-        """Cache a prompt."""
-        if len(self.cache) < self.max_size:
-            self.cache[key] = value
-    
-    def hit_rate(self) -> float:
-        """Get cache hit rate percentage."""
-        total = self.hits + self.misses
-        return (self.hits / total * 100) if total > 0 else 0.0
-    
-    def clear(self) -> None:
-        """Clear cache."""
-        self.cache.clear()
-        self.hits = 0
-        self.misses = 0
-    
-    def stats(self) -> Dict[str, Any]:
-        """Get cache statistics."""
-        return {
-            "size": len(self.cache),
-            "max_size": self.max_size,
-            "hits": self.hits,
-            "misses": self.misses,
-            "hit_rate": self.hit_rate(),
-        }
-
-
-def estimate_speedup(num_items: int, use_kv_cache: bool = True, use_parallel: bool = False) -> Dict[str, Any]:
-    """
-    Estimate speedup based on optimization strategy.
-    
-    Empirical data points:
-    - No optimization: 4-5 sec/item (baseline)
-    - KV Cache: 0.8-1.2 sec/item (4-5x speedup)
-    - Parallel (2x): 0.4-0.6 sec/item (8-10x speedup)
-    """
-    baseline_per_item = 4.5  # seconds
-    
-    if use_kv_cache:
-        optimized_per_item = baseline_per_item / 5  # 4-5x speedup
-    else:
-        optimized_per_item = baseline_per_item
-    
-    if use_parallel:
-        optimized_per_item /= 2  # Rough estimate for 2 parallel
-    
-    baseline_total = baseline_per_item * num_items
-    optimized_total = optimized_per_item * num_items
-    
-    return {
-        "num_items": num_items,
-        "baseline_seconds": round(baseline_total, 1),
-        "optimized_seconds": round(optimized_total, 1),
-        "speedup_factor": round(baseline_total / max(0.1, optimized_total), 1),
-        "estimated_per_item": round(optimized_per_item, 2),
-    }

app/logic/infill_utils.py

@@ -1,246 +0,0 @@
-"""
-Infill Utilities for Batch Gap-Filling
-
-Handles gap detection, JSON parsing from LLM output, and text reconstruction.
-
-Gap Notation Support:
-- [GAP:n]: Explicit numbered gaps (preferred)
-- ___: Underscores (auto-numbered in scan order)
-
-FUTURE: Chunking Support
--------------------------
-For texts exceeding ~2000 tokens (approx 6000 chars), implement per-gap prompting:
-1. Split text into chunks preserving gap context (±150 tokens around each gap)
-2. Process each gap individually with left/right context
-3. Merge results back into full text
-4. This avoids context window overflow on smaller models (2k-4k context)
-
-Current implementation assumes texts fit within model context window.
-Add chunking when processing long-form content (articles, full listings).
-"""
-
-import re
-import json
-from typing import List, Optional, Tuple
-from dataclasses import dataclass
-
-
-@dataclass
-class GapInfo:
-    """Information about a detected gap in text."""
-    index: int          # 1-based index
-    marker: str         # Original marker string
-    start: int          # Start position in text
-    end: int            # End position in text
-
-
-def detect_gaps(text: str, notation: str = "auto") -> List[GapInfo]:
-    """
-    Detect gaps in text and return their positions.
-    
-    Args:
-        text: Input text with gap markers
-        notation: "auto", "[GAP:n]", or "___"
-        
-    Returns:
-        List of GapInfo objects sorted by position
-        
-    Examples:
-        >>> detect_gaps("Buy this [GAP:1] car with [GAP:2] features")
-        [GapInfo(index=1, marker='[GAP:1]', ...), GapInfo(index=2, marker='[GAP:2]', ...)]
-        
-        >>> detect_gaps("Buy this ___ car with ___ features")
-        [GapInfo(index=1, marker='___', ...), GapInfo(index=2, marker='___', ...)]
-    """
-    gaps = []
-    
-    # Pattern for [GAP:n] notation
-    gap_tag_pattern = r'\[GAP:(\d+)\]'
-    # Pattern for underscore notation (3+ underscores)
-    underscore_pattern = r'_{3,}'
-    
-    if notation == "auto":
-        # Try [GAP:n] first, fallback to ___
-        gap_matches = list(re.finditer(gap_tag_pattern, text))
-        if gap_matches:
-            notation = "[GAP:n]"
-        else:
-            notation = "___"
-    
-    if notation == "[GAP:n]":
-        for match in re.finditer(gap_tag_pattern, text):
-            gaps.append(GapInfo(
-                index=int(match.group(1)),
-                marker=match.group(0),
-                start=match.start(),
-                end=match.end()
-            ))
-    else:  # "___"
-        for i, match in enumerate(re.finditer(underscore_pattern, text), start=1):
-            gaps.append(GapInfo(
-                index=i,
-                marker=match.group(0),
-                start=match.start(),
-                end=match.end()
-            ))
-    
-    # Sort by position (should already be, but ensure)
-    gaps.sort(key=lambda g: g.start)
-    return gaps
-
-
-def parse_infill_json(raw_output: str) -> Optional[dict]:
-    """
-    Extract and parse JSON from LLM output.
-    
-    Handles common LLM quirks:
-    - JSON wrapped in markdown code blocks
-    - Leading/trailing text before/after JSON
-    - Function-call style wrapper ({"name": "...", "arguments": {...}})
-    - Double-escaped JSON strings in arguments field
-    - Minor formatting issues
-        
-    Returns:
-        Parsed dict with 'filled_text' and 'gaps' keys, or None if parsing fails
-    """
-    if not raw_output:
-        return None
-    
-    # Try to extract JSON from markdown code blocks
-    json_block_pattern = r'```(?:json)?\s*([\s\S]*?)\s*```'
-    match = re.search(json_block_pattern, raw_output)
-    if match:
-        raw_output = match.group(1)
-    
-    # Find JSON object boundaries
-    start_idx = raw_output.find('{')
-    if start_idx == -1:
-        return None
-    
-    # Find matching closing brace
-    depth = 0
-    end_idx = -1
-    for i, char in enumerate(raw_output[start_idx:], start=start_idx):
-        if char == '{':
-            depth += 1
-        elif char == '}':
-            depth -= 1
-            if depth == 0:
-                end_idx = i + 1
-                break
-    
-    if end_idx == -1:
-        return None
-    
-    json_str = raw_output[start_idx:end_idx]
-    
-    try:
-        parsed = json.loads(json_str)
-        
-        # Handle function-call style wrapper with STRING arguments (double-escaped):
-        # {"name": "fill_in_text", "arguments": "{\"filled_text\": \"...\"}"}
-        if 'arguments' in parsed:
-            args = parsed['arguments']
-            if isinstance(args, str):
-                try:
-                    parsed = json.loads(args)
-                except json.JSONDecodeError:
-                    return None
-            elif isinstance(args, dict):
-                parsed = args
-        
-        # Also handle: {"name": "...", "parameters": {...}}
-        if 'parameters' in parsed:
-            params = parsed['parameters']
-            if isinstance(params, str):
-                try:
-                    parsed = json.loads(params)
-                except json.JSONDecodeError:
-                    return None
-            elif isinstance(params, dict):
-                parsed = params
-        
-        # Validate required fields
-        if 'filled_text' not in parsed and 'gaps' not in parsed:
-            return None
-            
-        return parsed
-    except json.JSONDecodeError:
-        return None
-
-
-def apply_fills(original_text: str, gaps: List[GapInfo], fills: dict) -> str:
-    """
-    Apply gap fills to original text.
-    
-    Uses fills from parsed JSON, replacing markers with chosen words.
-    This is a fallback when LLM's 'filled_text' might be corrupted.
-    
-    Args:
-        original_text: Original text with gap markers
-        gaps: Detected gaps from detect_gaps()
-        fills: Dict mapping gap index to fill choice
-               e.g., {1: "excellent", 2: "powerful"}
-               
-    Returns:
-        Text with gaps replaced by fill choices
-    """
-    if not gaps or not fills:
-        return original_text
-    
-    # Process from end to start to preserve positions
-    result = original_text
-    for gap in reversed(gaps):
-        if gap.index in fills:
-            result = result[:gap.start] + fills[gap.index] + result[gap.end:]
-    
-    return result
-
-
-def build_fills_dict(gaps_list: List[dict]) -> dict:
-    """
-    Convert gaps list from JSON to fills dict.
-    
-    Args:
-        gaps_list: List of gap dicts from parsed JSON
-                   [{"index": 1, "choice": "word"}, ...]
-                   
-    Returns:
-        Dict mapping index to choice: {1: "word", ...}
-    """
-    fills = {}
-    for gap in gaps_list:
-        if 'index' in gap and 'choice' in gap:
-            fills[gap['index']] = gap['choice']
-    return fills
-
-
-def normalize_gaps_to_tagged(text: str) -> Tuple[str, List[GapInfo]]:
-    """
-    Normalize any gap notation to [GAP:n] format.
-    
-    Useful for standardizing input before processing.
-    
-    Args:
-        text: Text with any gap notation
-        
-    Returns:
-        Tuple of (normalized_text, gaps)
-    """
-    gaps = detect_gaps(text, "auto")
-    
-    if not gaps:
-        return text, []
-    
-    # If already [GAP:n], return as-is
-    if gaps[0].marker.startswith('[GAP:'):
-        return text, gaps
-    
-    # Convert ___ to [GAP:n]
-    result = text
-    for gap in reversed(gaps):
-        new_marker = f"[GAP:{gap.index}]"
-        result = result[:gap.start] + new_marker + result[gap.end:]
-    
-    # Re-detect with new positions
-    return result, detect_gaps(result, "[GAP:n]")

app/main.py

@@ -1,468 +0,0 @@
-import os
-import time
-import asyncio
-import importlib
-from fastapi import FastAPI, HTTPException, Depends, Body
-from typing import Optional, List
-from pydantic import ValidationError
-
-from app.models.registry import registry, MODEL_CONFIG
-from fastapi.middleware.cors import CORSMiddleware
-from app.schemas.schemas import (
-    EnhancedDescriptionResponse,
-    CompareRequest,
-    CompareResponse,
-    ModelResult,
-    ModelInfo,
-    InfillRequest,
-    InfillResponse,
-    InfillResult,
-    GapFill,
-    CompareInfillRequest,
-    CompareInfillResponse,
-    ModelInfillResult,
-)
-from app.logic.infill_utils import (
-    detect_gaps,
-    parse_infill_json,
-    apply_fills,
-    build_fills_dict,
-    normalize_gaps_to_tagged,
-)
-from app.auth.placeholder_auth import get_authenticated_user
-
-app = FastAPI(
-    title="Multi-Model Description Enhancer",
-    description="AI-powered service for enhancing descriptions using multiple LLMs for A/B testing",
-    version="3.0.0"
-)
-
-# CORS configuration
-app.add_middleware(
-    CORSMiddleware,
-    allow_origins=[
-        "http://localhost:5173",
-        "http://localhost:5174",
-        os.getenv("FRONTEND_URL", "http://localhost:5173")
-    ],
-    allow_credentials=True,
-    allow_methods=["POST", "GET"],
-    allow_headers=["*"],
-)
-
-@app.on_event("startup")
-async def startup_event():
-    """
-    Startup event - models are loaded lazily on first request.
-    No models are pre-loaded to conserve memory.
-    """
-    print("Application started. Models will be loaded lazily on first request.")
-    print(f"Available models: {registry.get_available_model_names()}")
-
-# --- Helper function to load domain logic ---
-def get_domain_config(domain: str):
-    try:
-        module = importlib.import_module(f"app.domains.{domain}.config")
-        return module.domain_config
-    except (ImportError, AttributeError):
-        raise HTTPException(status_code=404, detail=f"Domain '{domain}' not found or not configured correctly.")
-
-# --- API Endpoints ---
-
-@app.get("/")
-async def read_root():
-    return {"message": "Welcome to the Multi-Model Description Enhancer API! Go to /docs for documentation."}
-
-@app.get("/health")
-async def health_check():
-    """Check API health and model status."""
-    models = registry.list_models()
-    loaded_models = registry.get_loaded_models()
-    active_model = registry.get_active_model()
-    return {
-        "status": "ok",
-        "available_models": len(models),
-        "loaded_models": loaded_models,
-        "active_local_model": active_model,
-    }
-
-@app.get("/models", response_model=List[ModelInfo])
-async def list_models():
-    """List all available models with their load status."""
-    return registry.list_models()
-
-@app.post("/models/{model_name}/load")
-async def load_model(model_name: str):
-    """
-    Explicitly load a model into memory.
-    For local models: unloads any previously loaded local model first.
-    """
-    if model_name not in registry.get_available_model_names():
-        raise HTTPException(status_code=404, detail=f"Unknown model: {model_name}")
-    
-    try:
-        info = await registry.load_model(model_name)
-        return {"status": "loaded", "model": info}
-    except Exception as e:
-        raise HTTPException(status_code=500, detail=f"Failed to load model: {str(e)}")
-
-@app.post("/models/{model_name}/unload")
-async def unload_model(model_name: str):
-    """
-    Explicitly unload a model from memory to free resources.
-    """
-    if model_name not in registry.get_available_model_names():
-        raise HTTPException(status_code=404, detail=f"Unknown model: {model_name}")
-    
-    try:
-        result = await registry.unload_model(model_name)
-        return result
-    except Exception as e:
-        raise HTTPException(status_code=500, detail=f"Failed to unload model: {str(e)}")
-
-@app.post("/enhance-description", response_model=EnhancedDescriptionResponse)
-async def enhance_description(
-    domain: str = Body(..., embed=True),
-    data: dict = Body(..., embed=True),
-    model: str = Body("bielik-1.5b", embed=True),
-    user: Optional[dict] = Depends(get_authenticated_user)
-):
-    """
-    Generate an enhanced description using a single model.
-    - **domain**: The name of the domain (e.g., 'cars').
-    - **data**: A dictionary with the data for the description.
-    - **model**: Model to use (default: bielik-1.5b)
-    """
-    start_time = time.time()
-    
-    # Validate model
-    if model not in registry.get_available_model_names():
-        raise HTTPException(status_code=400, detail=f"Unknown model: {model}")
-    
-    # Load Domain Configuration
-    domain_config = get_domain_config(domain)
-    DomainSchema = domain_config["schema"]
-    create_prompt = domain_config["create_prompt"]
-
-    # Validate Input Data
-    try:
-        validated_data = DomainSchema(**data)
-    except ValidationError as e:
-        raise HTTPException(status_code=422, detail=f"Invalid data for domain '{domain}': {e}")
-
-    # Prompt Construction
-    chat_messages = create_prompt(validated_data)
-
-    # Text Generation
-    try:
-        llm = await registry.get_model(model)
-        generated_description = await llm.generate(
-            chat_messages=chat_messages,
-            max_new_tokens=150,
-            temperature=0.75,
-            top_p=0.9,
-        )
-    except Exception as e:
-        print(f"Error during text generation with {model}: {e}")
-        raise HTTPException(status_code=500, detail=f"Generation error: {str(e)}")
-
-    generation_time = time.time() - start_time
-    user_email = user['email'] if user else "anonymous"
-
-    return EnhancedDescriptionResponse(
-        description=generated_description,
-        model_used=MODEL_CONFIG[model]["id"],
-        generation_time=round(generation_time, 2),
-        user_email=user_email
-    )
-
-@app.post("/compare", response_model=CompareResponse)
-async def compare_models(
-    request: CompareRequest,
-    user: Optional[dict] = Depends(get_authenticated_user)
-):
-    """
-    Compare outputs from multiple models for the same input.
-    Returns results from all specified models (or all available if not specified).
-    """
-    total_start = time.time()
-    
-    # Get models to compare
-    available_models = registry.get_available_model_names()
-    models_to_use = request.models if request.models else available_models
-    
-    # Validate requested models
-    for model in models_to_use:
-        if model not in available_models:
-            raise HTTPException(status_code=400, detail=f"Unknown model: {model}")
-    
-    # Load Domain Configuration
-    domain_config = get_domain_config(request.domain)
-    DomainSchema = domain_config["schema"]
-    create_prompt = domain_config["create_prompt"]
-
-    # Validate Input Data
-    try:
-        validated_data = DomainSchema(**request.data)
-    except ValidationError as e:
-        raise HTTPException(status_code=422, detail=f"Invalid data: {e}")
-
-    # Prompt Construction
-    chat_messages = create_prompt(validated_data)
-
-    # Generate with each model
-    results = []
-    
-    async def generate_with_model(model_name: str) -> ModelResult:
-        start_time = time.time()
-        try:
-            llm = await registry.get_model(model_name)
-            output = await llm.generate(
-                chat_messages=chat_messages,
-                max_new_tokens=150,
-                temperature=0.75,
-                top_p=0.9,
-            )
-            return ModelResult(
-                model=model_name,
-                output=output,
-                time=round(time.time() - start_time, 2),
-                type=MODEL_CONFIG[model_name]["type"],
-                error=None
-            )
-        except Exception as e:
-            return ModelResult(
-                model=model_name,
-                output="",
-                time=round(time.time() - start_time, 2),
-                type=MODEL_CONFIG[model_name]["type"],
-                error=str(e)
-            )
-    
-    # Run all models (sequentially to avoid memory issues)
-    for model_name in models_to_use:
-        result = await generate_with_model(model_name)
-        results.append(result)
-    
-    return CompareResponse(
-        domain=request.domain,
-        results=results,
-        total_time=round(time.time() - total_start, 2)
-    )
-
-@app.get("/user/me")
-async def get_user_info(user: dict = Depends(get_authenticated_user)):
-    """Get current authenticated user information"""
-    if not user:
-        raise HTTPException(status_code=401, detail="Not authenticated")
-    return {
-        "user_id": user['user_id'],
-        "email": user['email'],
-        "name": user.get('name', 'Unknown')
-    }
-
-
-# --- Batch Infill Endpoints ---
-
-@app.post("/infill", response_model=InfillResponse)
-async def batch_infill(
-    request: InfillRequest,
-    user: Optional[dict] = Depends(get_authenticated_user)
-):
-    """
-    Batch gap-filling for ads using a single model.
-    
-    Accepts items with [GAP:n] markers or ___ and returns filled text
-    with per-gap choices and alternatives.
-    
-    NOTE: For texts > 6000 chars, consider chunking (not yet implemented).
-    """
-    total_start = time.time()
-    
-    # Validate model
-    if request.model not in registry.get_available_model_names():
-        raise HTTPException(status_code=400, detail=f"Unknown model: {request.model}")
-    
-    # Load domain config for infill prompt
-    domain_config = get_domain_config(request.domain)
-    if "create_infill_prompt" not in domain_config:
-        raise HTTPException(
-            status_code=400,
-            detail=f"Domain '{request.domain}' does not support infill operations"
-        )
-    create_infill_prompt = domain_config["create_infill_prompt"]
-    
-    # Process each item
-    results = []
-    error_count = 0
-    
-    for item in request.items:
-        result = await process_infill_item(
-            item=item,
-            model_name=request.model,
-            options=request.options,
-            create_infill_prompt=create_infill_prompt
-        )
-        results.append(result)
-        if result.status == "error":
-            error_count += 1
-    
-    return InfillResponse(
-        model=request.model,
-        results=results,
-        total_time=round(time.time() - total_start, 2),
-        processed_count=len(results),
-        error_count=error_count
-    )
-
-
-@app.post("/compare-infill", response_model=CompareInfillResponse)
-async def compare_infill(
-    request: CompareInfillRequest,
-    user: Optional[dict] = Depends(get_authenticated_user)
-):
-    """
-    Multi-model batch gap-filling comparison for A/B testing.
-    
-    Runs the same batch of items through multiple models and returns
-    per-model results for comparison.
-    """
-    total_start = time.time()
-    
-    # Get models to compare
-    available_models = registry.get_available_model_names()
-    models_to_use = request.models if request.models else available_models
-    
-    # Validate requested models
-    for model in models_to_use:
-        if model not in available_models:
-            raise HTTPException(status_code=400, detail=f"Unknown model: {model}")
-    
-    # Load domain config
-    domain_config = get_domain_config(request.domain)
-    if "create_infill_prompt" not in domain_config:
-        raise HTTPException(
-            status_code=400,
-            detail=f"Domain '{request.domain}' does not support infill operations"
-        )
-    create_infill_prompt = domain_config["create_infill_prompt"]
-    
-    # Process with each model (sequentially for memory safety)
-    model_results = []
-    
-    for model_name in models_to_use:
-        model_start = time.time()
-        results = []
-        error_count = 0
-        
-        for item in request.items:
-            result = await process_infill_item(
-                item=item,
-                model_name=model_name,
-                options=request.options,
-                create_infill_prompt=create_infill_prompt
-            )
-            results.append(result)
-            if result.status == "error":
-                error_count += 1
-        
-        model_results.append(ModelInfillResult(
-            model=model_name,
-            type=MODEL_CONFIG[model_name]["type"],
-            results=results,
-            time=round(time.time() - model_start, 2),
-            error_count=error_count
-        ))
-    
-    return CompareInfillResponse(
-        domain=request.domain,
-        models=model_results,
-        total_time=round(time.time() - total_start, 2)
-    )
-
-
-async def process_infill_item(
-    item,
-    model_name: str,
-    options,
-    create_infill_prompt
-) -> InfillResult:
-    """
-    Process a single infill item.
-    
-    Returns InfillResult with status, filled_text, and gaps.
-    """
-    try:
-        # Normalize gaps to [GAP:n] format
-        normalized_text, gaps = normalize_gaps_to_tagged(item.text_with_gaps)
-        
-        if not gaps:
-            # No gaps found, return original text
-            return InfillResult(
-                id=item.id,
-                status="ok",
-                filled_text=item.text_with_gaps,
-                gaps=[],
-                error=None
-            )
-        
-        # Build prompt
-        chat_messages = create_infill_prompt(normalized_text, options)
-        
-        # Generate
-        llm = await registry.get_model(model_name)
-        raw_output = await llm.generate(
-            chat_messages=chat_messages,
-            max_new_tokens=options.max_new_tokens,
-            temperature=options.temperature,
-            top_p=0.9,
-        )
-        
-        # Parse JSON from output
-        parsed = parse_infill_json(raw_output)
-        
-        if not parsed:
-            # JSON parsing failed
-            return InfillResult(
-                id=item.id,
-                status="error",
-                filled_text=None,
-                gaps=[],
-                error=f"Failed to parse JSON from model output: {raw_output[:200]}..."
-            )
-        
-        # Extract gaps and build result
-        gap_fills = []
-        fills_dict = {}
-        
-        for gap_data in parsed.get("gaps", []):
-            gap_fill = GapFill(
-                index=gap_data.get("index", 0),
-                marker=gap_data.get("marker", ""),
-                choice=gap_data.get("choice", ""),
-                alternatives=gap_data.get("alternatives", [])
-            )
-            gap_fills.append(gap_fill)
-            fills_dict[gap_fill.index] = gap_fill.choice
-        
-        # Get filled text - prefer model's version, fallback to reconstruction
-        filled_text = parsed.get("filled_text")
-        if not filled_text and fills_dict:
-            filled_text = apply_fills(normalized_text, gaps, fills_dict)
-        
-        return InfillResult(
-            id=item.id,
-            status="ok",
-            filled_text=filled_text,
-            gaps=gap_fills,
-            error=None
-        )
-        
-    except Exception as e:
-        return InfillResult(
-            id=item.id,
-            status="error",
-            filled_text=None,
-            gaps=[],
-            error=str(e)
-        )

app/models/__init__.py

@@ -1,16 +0,0 @@
-"""
-Models module - LLM implementations and registry.
-"""
-
-from app.models.base_llm import BaseLLM
-from app.models.huggingface_local import HuggingFaceLocal
-from app.models.huggingface_inference_api import HuggingFaceInferenceAPI
-from app.models.registry import registry, MODEL_CONFIG
-
-__all__ = [
-    "BaseLLM",
-    "HuggingFaceLocal",
-    "HuggingFaceInferenceAPI",
-    "registry",
-    "MODEL_CONFIG",
-]

app/models/base_llm.py

@@ -1,54 +0,0 @@
-"""
-Abstract base class for all LLM implementations.
-"""
-
-from abc import ABC, abstractmethod
-from typing import Optional, List, Dict, Any
-
-
-class BaseLLM(ABC):
-    """Abstract interface for LLM models."""
-    
-    def __init__(self, name: str, model_id: str):
-        self.name = name
-        self.model_id = model_id
-        self._initialized = False
-    
-    @property
-    def is_initialized(self) -> bool:
-        return self._initialized
-    
-    @abstractmethod
-    async def initialize(self) -> None:
-        """Initialize the model. Must be called before generate()."""
-        pass
-    
-    @abstractmethod
-    async def generate(
-        self,
-        prompt: str = None,
-        chat_messages: List[Dict[str, str]] = None,
-        max_new_tokens: int = 150,
-        temperature: float = 0.7,
-        top_p: float = 0.9,
-        **kwargs
-    ) -> str:
-        """
-        Generate text from prompt or chat messages.
-        
-        Args:
-            prompt: Raw text prompt
-            chat_messages: List of {"role": "...", "content": "..."} messages
-            max_new_tokens: Maximum tokens to generate
-            temperature: Sampling temperature
-            top_p: Nucleus sampling parameter
-            
-        Returns:
-            Generated text string
-        """
-        pass
-    
-    @abstractmethod
-    def get_info(self) -> Dict[str, Any]:
-        """Return model information for /models endpoint."""
-        pass

app/models/huggingface_inference_api.py

@@ -1,93 +0,0 @@
-"""
-HuggingFace Inference API client for remote model access.
-"""
-
-import os
-from typing import List, Dict, Any, Optional
-from huggingface_hub import InferenceClient
-
-from app.models.base_llm import BaseLLM
-
-
-class HuggingFaceInferenceAPI(BaseLLM):
-    """
-    Remote model access via HuggingFace Inference API.
-    Best for larger models (7B+) that don't fit in local RAM.
-    """
-    
-    def __init__(self, name: str, model_id: str, token: str = None):
-        super().__init__(name, model_id)
-        self.token = token or os.getenv("HF_TOKEN")
-        self.client: Optional[InferenceClient] = None
-    
-    async def initialize(self) -> None:
-        """Initialize the Inference API client."""
-        if self._initialized:
-            return
-        
-        try:
-            print(f"[{self.name}] Initializing Inference API for: {self.model_id}")
-            
-            self.client = InferenceClient(
-                model=self.model_id,
-                token=self.token
-            )
-            
-            self._initialized = True
-            print(f"[{self.name}] Inference API ready")
-            
-        except Exception as e:
-            print(f"[{self.name}] Failed to initialize: {e}")
-            raise
-    
-    async def generate(
-        self,
-        prompt: str = None,
-        chat_messages: List[Dict[str, str]] = None,
-        max_new_tokens: int = 150,
-        temperature: float = 0.7,
-        top_p: float = 0.9,
-        **kwargs
-    ) -> str:
-        """Generate text using HuggingFace Inference API."""
-        
-        if not self._initialized or not self.client:
-            raise RuntimeError(f"[{self.name}] Client not initialized")
-        
-        try:
-            # Use chat completion if chat_messages provided
-            if chat_messages:
-                response = self.client.chat_completion(
-                    messages=chat_messages,
-                    max_tokens=max_new_tokens,
-                    temperature=temperature,
-                    top_p=top_p,
-                )
-                return response.choices[0].message.content.strip()
-            
-            # Otherwise use text generation
-            elif prompt:
-                response = self.client.text_generation(
-                    prompt=prompt,
-                    max_new_tokens=max_new_tokens,
-                    temperature=temperature,
-                    top_p=top_p,
-                    do_sample=True,
-                )
-                return response.strip()
-            
-            else:
-                raise ValueError("Either prompt or chat_messages required")
-                
-        except Exception as e:
-            print(f"[{self.name}] Generation error: {e}")
-            raise
-    
-    def get_info(self) -> Dict[str, Any]:
-        """Return model info."""
-        return {
-            "name": self.name,
-            "model_id": self.model_id,
-            "type": "inference_api",
-            "initialized": self._initialized,
-        }

app/models/huggingface_local.py

@@ -1,260 +0,0 @@
-"""
-Local HuggingFace model implementation using transformers pipeline.
-
-Optimizations:
-- KV Cache: Enabled by default (5-10x speedup on GPU, 1.5x on CPU)
-- Flash Attention: Used when available (GPU only)
-- 8-Bit Quantization: Optional for CPU environments (4-6x speedup, 50% memory reduction)
-"""
-
-from typing import List, Dict, Any, Optional
-from transformers import pipeline, AutoTokenizer, AutoModelForCausalLM
-import torch
-import asyncio
-import os
-
-from app.models.base_llm import BaseLLM
-
-# Try to import bitsandbytes, but don't fail if not available
-try:
-    from transformers import BitsAndBytesConfig
-    HAS_BITSANDBYTES = True
-except ImportError:
-    HAS_BITSANDBYTES = False
-    print("[WARNING] bitsandbytes not available - 8-bit quantization disabled")
-
-
-class HuggingFaceLocal(BaseLLM):
-    """
-    Local HuggingFace model loaded into container memory.
-    Best for smaller models (< 3B parameters) that fit in RAM.
-    
-    Features:
-    - KV caching enabled (1.5-2x faster on CPU, 5-10x on GPU)
-    - Flash Attention v2 support (GPU only)
-    - 8-bit quantization for CPU (4-6x speedup, 50% less memory)
-    - Mixed precision (float16 or bfloat16 when possible)
-    """
-    
-    def __init__(self, name: str, model_id: str, device: str = "cpu", use_cache: bool = True, use_8bit: bool = False):
-        super().__init__(name, model_id)
-        self.device = device
-        self.pipeline = None
-        self.tokenizer = None
-        self.model = None
-        self.use_cache = use_cache
-        
-        # Only enable 8-bit if explicitly requested (opt-in, not by default)
-        # Default to False since bitsandbytes may not be available in all deployment environments
-        requested_8bit = use_8bit or (device == "cpu" and os.getenv("USE_8BIT_QUANTIZATION", "false").lower() == "true")
-        self.use_8bit = requested_8bit and HAS_BITSANDBYTES
-        
-        if requested_8bit and not HAS_BITSANDBYTES:
-            print(f"[{name}] 8-bit quantization requested but bitsandbytes not installed - falling back to full precision")
-        
-        self.use_flash_attention = os.getenv("USE_FLASH_ATTENTION", "true").lower() == "true"
-        
-        # Determine device index and dtype
-        if device == "cuda" and torch.cuda.is_available():
-            self.device_index = 0
-            # Try to use bfloat16 on modern GPUs, else float16
-            self.torch_dtype = torch.bfloat16 if torch.cuda.is_available() and hasattr(torch.cuda, "get_device_capability") else torch.float16
-        else:
-            self.device_index = -1  # CPU
-            self.torch_dtype = torch.float32
-    
-    async def initialize(self) -> None:
-        """Load model into memory with optimizations."""
-        if self._initialized:
-            return
-            
-        try:
-            print(f"[{self.name}] Loading local model: {self.model_id}")
-            print(f"[{self.name}] Device: {self.device} | Dtype: {self.torch_dtype} | KV Cache: {self.use_cache} | 8-bit: {self.use_8bit}")
-            
-            self.tokenizer = await asyncio.to_thread(
-                AutoTokenizer.from_pretrained,
-                self.model_id,
-                trust_remote_code=True
-            )
-            
-            # Model config optimizations
-            model_kwargs = {
-                "trust_remote_code": True,
-            }
-            
-            # Add 8-bit quantization for CPU (4-6x faster, 50% less memory)
-            if self.use_8bit and HAS_BITSANDBYTES:
-                try:
-                    print(f"[{self.name}] Using 8-bit quantization for CPU optimization")
-                    bnb_config = BitsAndBytesConfig(
-                        load_in_8bit=True,
-                        bnb_8bit_compute_dtype=torch.float16,
-                        bnb_8bit_use_double_quant=True,
-                    )
-                    model_kwargs["quantization_config"] = bnb_config
-                    model_kwargs["device_map"] = "cpu"
-                except Exception as e:
-                    print(f"[{self.name}] Failed to setup 8-bit quantization: {e}")
-                    print(f"[{self.name}] Falling back to full precision")
-                    self.use_8bit = False
-                    model_kwargs["torch_dtype"] = self.torch_dtype
-                    model_kwargs["device_map"] = "cpu"
-            
-            # Standard loading without quantization
-            if not self.use_8bit:
-                model_kwargs["torch_dtype"] = self.torch_dtype
-                model_kwargs["device_map"] = self.device if self.device == "cuda" else "cpu"
-            
-            # Enable flash attention if requested and available (GPU only)
-            if self.use_flash_attention and self.device == "cuda" and not self.use_8bit:
-                model_kwargs["attn_implementation"] = "flash_attention_2"
-            
-            self.model = await asyncio.to_thread(
-                AutoModelForCausalLM.from_pretrained,
-                self.model_id,
-                **model_kwargs
-            )
-            
-            # Ensure cache is enabled on model config
-            if hasattr(self.model.config, 'use_cache'):
-                self.model.config.use_cache = self.use_cache
-            
-            self._initialized = True
-            print(f"[{self.name}] Model loaded successfully (use_cache={self.use_cache})")
-            
-        except Exception as e:
-            print(f"[{self.name}] Failed to load model: {e}")
-            raise
-    
-    async def generate(
-        self,
-        prompt: str = None,
-        chat_messages: List[Dict[str, str]] = None,
-        max_new_tokens: int = 150,
-        temperature: float = 0.7,
-        top_p: float = 0.9,
-        **kwargs
-    ) -> str:
-        """
-        Generate text using direct model.generate() with proper KV caching.
-        
-        KV Cache Impact (with proper implementation):
-        - WITH: ~9 seconds for 10 ads (50 gaps)
-        - WITHOUT: ~42 seconds (4.7x slower)
-        """
-        
-        if not self._initialized or self.model is None:
-            raise RuntimeError(f"[{self.name}] Model not initialized")
-        
-        formatted_prompt = None
-        
-        # Format prompt from chat messages
-        if chat_messages:
-            try:
-                formatted_prompt = self.tokenizer.apply_chat_template(
-                    chat_messages,
-                    tokenize=False,
-                    add_generation_prompt=True
-                )
-            except Exception as e:
-                print(f"[{self.name}] apply_chat_template failed: {e}, using fallback")
-                formatted_prompt = self._format_chat_fallback(chat_messages)
-        
-        # Use raw prompt if provided
-        if formatted_prompt is None and prompt:
-            formatted_prompt = prompt
-        
-        if formatted_prompt is None:
-            raise ValueError("Either prompt or chat_messages required")
-        
-        # Tokenize input
-        inputs = await asyncio.to_thread(
-            self.tokenizer.encode,
-            formatted_prompt,
-            return_tensors="pt"
-        )
-        
-        # Move to device
-        if self.device == "cuda":
-            inputs = await asyncio.to_thread(lambda: inputs.to("cuda"))
-        
-        # Generate with explicit KV cache
-        outputs = await asyncio.to_thread(
-            self.model.generate,
-            inputs,
-            max_new_tokens=max_new_tokens,
-            do_sample=True,
-            temperature=temperature,
-            top_p=top_p,
-            use_cache=True,  # CRITICAL: Enable KV cache
-            eos_token_id=self.tokenizer.eos_token_id,
-            pad_token_id=self.tokenizer.eos_token_id if self.tokenizer.pad_token_id is None else self.tokenizer.pad_token_id,
-        )
-        
-        # Decode output
-        output_text = await asyncio.to_thread(
-            self.tokenizer.decode,
-            outputs[0],
-            skip_special_tokens=True
-        )
-        
-        # Remove prompt from output
-        if output_text.startswith(formatted_prompt):
-            response = output_text[len(formatted_prompt):]
-        else:
-            response = output_text
-        
-        # Clean up special tokens
-        for token in ["<|im_end|>", "<end_of_turn>", "<eos>", "</s>"]:
-            if response.endswith(token):
-                response = response[:-len(token)]
-        
-        return response.strip()
-    
-    def _format_chat_fallback(self, chat_messages: List[Dict[str, str]]) -> str:
-        """
-        Fallback chat formatting for models without proper chat template.
-        Works with Gemma and other models.
-        """
-        formatted = ""
-        for msg in chat_messages:
-            role = msg.get("role", "user")
-            content = msg.get("content", "")
-            
-            if role == "system":
-                formatted += f"{content}\n\n"
-            elif role == "user":
-                formatted += f"User: {content}\n"
-            elif role == "assistant":
-                formatted += f"Assistant: {content}\n"
-        
-        # Add generation prompt
-        formatted += "Assistant:"
-        return formatted
-    
-    def get_info(self) -> Dict[str, Any]:
-        """Return model info."""
-        return {
-            "name": self.name,
-            "model_id": self.model_id,
-            "type": "local",
-            "initialized": self._initialized,
-            "device": self.device
-        }
-    
-    async def cleanup(self) -> None:
-        """Release model from memory."""
-        if self.pipeline is not None:
-            del self.pipeline
-            self.pipeline = None
-        if self.tokenizer is not None:
-            del self.tokenizer
-            self.tokenizer = None
-        self._initialized = False
-        
-        # Force CUDA cache clear if available
-        if torch.cuda.is_available():
-            torch.cuda.empty_cache()
-        
-        print(f"[{self.name}] Model unloaded from memory")

app/models/huggingface_service.py

@@ -1,111 +0,0 @@
-from transformers import pipeline, AutoTokenizer
-import torch
-from fastapi import HTTPException
-import asyncio 
-
-class HuggingFaceTextGenerationService:
-    def __init__(self, model_name_or_path: str, device: str = None, task: str = "text-generation"):
-        self.model_name_or_path = model_name_or_path
-        self.task = task
-        self.pipeline = None
-        self.tokenizer = None
-
-        if device is None:
-            self.device_index = 0 if torch.cuda.is_available() else -1
-        elif device == "cuda" and torch.cuda.is_available():
-            self.device_index = 0
-        elif device == "cpu":
-            self.device_index = -1
-        else:
-            self.device_index = -1
-        
-        if self.device_index == 0:
-            print("CUDA (GPU) is available. Using GPU.")
-        else:
-            print(f"Device set to use {'cpu' if self.device_index == -1 else f'cuda:{self.device_index}'}")
-
-
-    async def initialize(self):
-        try:
-            print(f"Initializing Hugging Face pipeline for model: {self.model_name_or_path} on device index: {self.device_index}")
-            self.tokenizer = await asyncio.to_thread(
-                AutoTokenizer.from_pretrained, self.model_name_or_path, trust_remote_code=True
-            )
-            self.pipeline = await asyncio.to_thread(
-                pipeline,
-                self.task,
-                model=self.model_name_or_path,
-                tokenizer=self.tokenizer,
-                device=self.device_index,
-                torch_dtype=torch.bfloat16 if self.device_index != -1 and torch.cuda.is_available() and torch.cuda.is_bf16_supported() else torch.float32,
-                trust_remote_code=True,
-            )
-            print(f"Pipeline for model {self.model_name_or_path} initialized successfully.")
-        except Exception as e:
-            print(f"Error initializing HuggingFace pipeline: {e}")
-            raise HTTPException(status_code=503, detail=f"LLM (HuggingFace) model could not be loaded: {str(e)}")
-
-    async def generate_text(self, prompt_text: str = None, chat_template_messages: list = None, max_new_tokens: int = 250, temperature: float = 0.7, top_p: float = 0.9, do_sample: bool = True, **kwargs) -> str:
-        if not self.pipeline or not self.tokenizer:
-            raise Exception("Pipeline is not initialized. Call initialize() first.")
-
-        formatted_prompt_input = "" 
-        if chat_template_messages:
-            try:
-                formatted_prompt_input = self.tokenizer.apply_chat_template(
-                    chat_template_messages,
-                    tokenize=False,
-                    add_generation_prompt=True 
-                )
-            except Exception as e:
-                print(f"Could not apply chat template, falling back to raw prompt if available. Error: {e}")
-                if prompt_text:
-                    formatted_prompt_input = prompt_text
-                else:
-                    raise ValueError("Cannot generate text without a valid prompt or chat_template_messages.")
-        elif prompt_text:
-            formatted_prompt_input = prompt_text
-        else:
-            raise ValueError("Either prompt_text or chat_template_messages must be provided.")
-
-        try:
-            generated_outputs = await asyncio.to_thread(
-                self.pipeline,
-                formatted_prompt_input, 
-                max_new_tokens=max_new_tokens,
-                do_sample=do_sample,
-                temperature=temperature,
-                top_p=top_p,
-                eos_token_id=self.tokenizer.eos_token_id,
-                pad_token_id=self.tokenizer.eos_token_id if self.tokenizer.pad_token_id is None else self.tokenizer.pad_token_id, # Common setting
-                **kwargs
-            )
-            
-            if generated_outputs and isinstance(generated_outputs, list) and "generated_text" in generated_outputs[0]:
-                full_generated_sequence = generated_outputs[0]["generated_text"]
-                
-                assistant_response = ""
-                if full_generated_sequence.startswith(formatted_prompt_input):
-                    assistant_response = full_generated_sequence[len(formatted_prompt_input):]
-                else:
-                    assistant_marker = "<|im_start|>assistant\n" 
-                    last_marker_pos = full_generated_sequence.rfind(assistant_marker)
-                    if last_marker_pos != -1:
-                        assistant_response = full_generated_sequence[last_marker_pos + len(assistant_marker):]
-                        print("Warning: Used fallback parsing for assistant response.")
-                    else:
-                        print("Error: Could not isolate assistant response from the full generated sequence.")
-                        assistant_response = full_generated_sequence 
-
-                if assistant_response.endswith("<|im_end|>"):
-                    assistant_response = assistant_response[:-len("<|im_end|>")]
-                
-                return assistant_response.strip() 
-            else:
-                print(f"Unexpected output format from pipeline: {generated_outputs}")
-                return "Error: Could not parse generated text from pipeline output."
-
-        except Exception as e:
-            print(f"Error during text generation with {self.model_name_or_path}: {e}")
-            raise HTTPException(status_code=500, detail=f"Error generating text: {str(e)}")
-

app/models/registry.py

@@ -1,211 +0,0 @@
-"""
-Model Registry - Central configuration and factory for all LLM models.
-Supports lazy loading and on/off mechanism for memory management.
-"""
-
-import os
-import gc
-from typing import Dict, List, Any, Optional
-
-from app.models.base_llm import BaseLLM
-from app.models.huggingface_local import HuggingFaceLocal
-from app.models.huggingface_inference_api import HuggingFaceInferenceAPI
-
-
-# Model configuration - 3 local + 1 API for Polish language comparison
-MODEL_CONFIG = {
-    "bielik-1.5b": {
-        "id": "speakleash/Bielik-1.5B-v3.0-Instruct",
-        "local_path": "bielik-1.5b",
-        "type": "local",
-        "polish_support": "excellent",
-        "size": "1.5B",
-    },
-    "qwen2.5-3b": {
-        "id": "Qwen/Qwen2.5-3B-Instruct",
-        "local_path": "qwen2.5-3b",
-        "type": "local",
-        "polish_support": "good",
-        "size": "3B",
-    },
-    "gemma-2-2b": {
-        "id": "google/gemma-2-2b-it",
-        "local_path": "gemma-2-2b",
-        "type": "local",
-        "polish_support": "medium",
-        "size": "2B",
-    },
-    "pllum-12b": {
-        "id": "CYFRAGOVPL/PLLuM-12B-instruct",
-        "type": "inference_api",
-        "polish_support": "excellent",
-        "size": "12B",
-    },
-}
-
-# Base path for pre-downloaded models in container
-LOCAL_MODEL_BASE = os.getenv("MODEL_DIR", "/app/pretrain_model")
-
-
-class ModelRegistry:
-    """
-    Central registry for managing all LLM models.
-    Supports lazy loading (load on first request) and unloading for memory management.
-    Only one local model is loaded at a time to conserve memory.
-    """
-    
-    def __init__(self):
-        self._models: Dict[str, BaseLLM] = {}
-        self._config = MODEL_CONFIG.copy()
-        self._active_local_model: Optional[str] = None
-    
-    def _create_model(self, name: str) -> BaseLLM:
-        """Factory method to create model instance."""
-        
-        if name not in self._config:
-            raise ValueError(f"Unknown model: {name}")
-        
-        config = self._config[name]
-        model_type = config["type"]
-        model_id = config["id"]
-        
-        # For local models, check if pre-downloaded version exists
-        if model_type == "local" and "local_path" in config:
-            local_path = os.path.join(LOCAL_MODEL_BASE, config["local_path"])
-            if os.path.exists(local_path):
-                print(f"Using pre-downloaded model at: {local_path}")
-                model_id = local_path
-            else:
-                print(f"Pre-downloaded model not found at {local_path}, will download from HuggingFace")
-        
-        if model_type == "local":
-            return HuggingFaceLocal(
-                name=name,
-                model_id=model_id,
-                device="cpu"
-            )
-        elif model_type == "inference_api":
-            return HuggingFaceInferenceAPI(
-                name=name,
-                model_id=model_id
-            )
-        else:
-            raise ValueError(f"Unknown model type: {model_type}")
-    
-    async def _unload_model(self, name: str) -> None:
-        """Unload a model from memory."""
-        if name in self._models:
-            model = self._models[name]
-            # Call cleanup if available
-            if hasattr(model, 'cleanup'):
-                await model.cleanup()
-            del self._models[name]
-            gc.collect()  # Force garbage collection
-            print(f"Model '{name}' unloaded from memory.")
-    
-    async def _unload_all_local_models(self) -> None:
-        """Unload all local models to free memory."""
-        local_models = [
-            name for name, config in self._config.items() 
-            if config["type"] == "local" and name in self._models
-        ]
-        for name in local_models:
-            await self._unload_model(name)
-        self._active_local_model = None
-    
-    async def get_model(self, name: str) -> BaseLLM:
-        """
-        Get a model (lazy loading).
-        For local models: unloads any previously loaded local model first.
-        For API models: always available without affecting local models.
-        """
-        if name not in self._config:
-            raise ValueError(f"Unknown model: {name}")
-        
-        config = self._config[name]
-        
-        # If it's a local model, ensure only one is loaded at a time
-        if config["type"] == "local":
-            # Unload current local model if different
-            if self._active_local_model and self._active_local_model != name:
-                print(f"Switching from '{self._active_local_model}' to '{name}'...")
-                await self._unload_model(self._active_local_model)
-            
-            # Load the requested model if not already loaded
-            if name not in self._models:
-                print(f"Loading model '{name}'...")
-                model = self._create_model(name)
-                await model.initialize()
-                self._models[name] = model
-                self._active_local_model = name
-                print(f"Model '{name}' loaded successfully.")
-        
-        # For API models, just create/return (no memory concern)
-        elif config["type"] == "inference_api":
-            if name not in self._models:
-                print(f"Initializing API model '{name}'...")
-                model = self._create_model(name)
-                await model.initialize()
-                self._models[name] = model
-        
-        return self._models[name]
-    
-    async def load_model(self, name: str) -> Dict[str, Any]:
-        """
-        Explicitly load a model (unloads other local models first).
-        Returns model info.
-        """
-        await self.get_model(name)
-        return self.get_model_info(name)
-    
-    async def unload_model(self, name: str) -> Dict[str, str]:
-        """
-        Explicitly unload a model from memory.
-        """
-        if name not in self._config:
-            raise ValueError(f"Unknown model: {name}")
-        
-        if name not in self._models:
-            return {"status": "not_loaded", "model": name}
-        
-        await self._unload_model(name)
-        if self._active_local_model == name:
-            self._active_local_model = None
-        
-        return {"status": "unloaded", "model": name}
-    
-    def get_model_info(self, name: str) -> Dict[str, Any]:
-        """Get info about a specific model."""
-        if name not in self._config:
-            raise ValueError(f"Unknown model: {name}")
-        
-        config = self._config[name]
-        return {
-            "name": name,
-            "model_id": config["id"],
-            "type": config["type"],
-            "polish_support": config["polish_support"],
-            "size": config["size"],
-            "loaded": name in self._models,
-            "active": name == self._active_local_model if config["type"] == "local" else None,
-        }
-    
-    def list_models(self) -> List[Dict[str, Any]]:
-        """List all available models with their info."""
-        return [self.get_model_info(name) for name in self._config.keys()]
-    
-    def get_available_model_names(self) -> List[str]:
-        """Get list of available model names."""
-        return list(self._config.keys())
-    
-    def get_active_model(self) -> Optional[str]:
-        """Get the currently active (loaded) local model name."""
-        return self._active_local_model
-    
-    def get_loaded_models(self) -> List[str]:
-        """Get list of currently loaded model names."""
-        return list(self._models.keys())
-
-
-# Global registry instance
-registry = ModelRegistry()

app/schemas/schemas.py

@@ -1,129 +0,0 @@
-from pydantic import BaseModel, Field
-from typing import List, Optional, Dict, Any
-
-
-class EnhancedDescriptionResponse(BaseModel):
-    description: str
-    model_used: str
-    generation_time: float
-    user_email: str
-
-
-# --- Batch Infill Schemas ---
-
-class InfillItem(BaseModel):
-    """A single item (ad) with gaps to be filled."""
-    id: str = Field(..., description="Unique identifier for this item")
-    text_with_gaps: str = Field(..., description="Text containing [GAP:n] markers or ___ to fill")
-
-
-class InfillOptions(BaseModel):
-    """Configuration options for infill processing."""
-    gap_notation: str = Field(
-        default="auto",
-        description="Gap notation: 'auto' (detect), '[GAP:n]', or '___'"
-    )
-    top_n_per_gap: int = Field(
-        default=3,
-        ge=1,
-        le=5,
-        description="Number of alternative suggestions per gap (1-5)"
-    )
-    language: str = Field(default="pl", description="Output language (pl/en)")
-    temperature: float = Field(default=0.6, ge=0.0, le=1.0)
-    max_new_tokens: int = Field(default=256, ge=50, le=512)
-
-
-class GapFill(BaseModel):
-    """Result for a single filled gap."""
-    index: int = Field(..., description="Gap index (1-based)")
-    marker: str = Field(..., description="Original marker (e.g., '[GAP:1]' or '___')")
-    choice: str = Field(..., description="Selected fill word/phrase")
-    alternatives: List[str] = Field(
-        default_factory=list,
-        description="Alternative suggestions"
-    )
-
-
-class InfillResult(BaseModel):
-    """Result for a single infill item."""
-    id: str
-    status: str = Field(..., description="'ok' or 'error'")
-    filled_text: Optional[str] = Field(None, description="Text with gaps filled")
-    gaps: List[GapFill] = Field(default_factory=list)
-    error: Optional[str] = Field(None, description="Error message if status='error'")
-
-
-class InfillRequest(BaseModel):
-    """Request for single-model batch infill."""
-    domain: str = Field(..., description="Domain name (e.g., 'cars')")
-    items: List[InfillItem] = Field(..., description="Batch of items to process")
-    model: str = Field(default="bielik-1.5b", description="Model to use")
-    options: InfillOptions = Field(default_factory=InfillOptions)
-
-
-class InfillResponse(BaseModel):
-    """Response for single-model batch infill."""
-    model: str
-    results: List[InfillResult]
-    total_time: float
-    processed_count: int
-    error_count: int
-
-
-class CompareInfillRequest(BaseModel):
-    """Request for multi-model batch infill comparison."""
-    domain: str
-    items: List[InfillItem]
-    models: Optional[List[str]] = Field(
-        None,
-        description="Models to compare. If None, use all available."
-    )
-    options: InfillOptions = Field(default_factory=InfillOptions)
-
-
-class ModelInfillResult(BaseModel):
-    """Per-model results in comparison."""
-    model: str
-    type: str
-    results: List[InfillResult]
-    time: float
-    error_count: int
-
-
-class CompareInfillResponse(BaseModel):
-    """Response for multi-model batch infill comparison."""
-    domain: str
-    models: List[ModelInfillResult]
-    total_time: float
-
-
-class ModelInfo(BaseModel):
-    name: str
-    model_id: str
-    type: str
-    polish_support: str
-    size: str
-    loaded: bool
-    active: Optional[bool] = None  # Only for local models
-
-
-class CompareRequest(BaseModel):
-    domain: str
-    data: Dict[str, Any]
-    models: Optional[List[str]] = None  # If None, use all models
-
-
-class ModelResult(BaseModel):
-    model: str
-    output: str
-    time: float
-    type: str
-    error: Optional[str] = None
-
-
-class CompareResponse(BaseModel):
-    domain: str
-    results: List[ModelResult]
-    total_time: float
-

requirements.txt

@@ -1,10 +0,0 @@
-fastapi==0.104.1
-uvicorn[standard]==0.24.0
-transformers==4.36.2
-accelerate==0.25.0
-huggingface_hub==0.19.4
-torch>=2.1.0
-pydantic==2.5.0
-# bitsandbytes is optional for 8-bit quantization (CPU optimization)
-# Uncomment below if bitsandbytes is available on your system:
-# bitsandbytes==0.49.0

start_container.ps1

@@ -1,23 +0,0 @@
-# PowerShell script to build and run the Docker container for your FastAPI service
-
-# Set variables
-$imageName = "bielik-fastapi-service"
-$containerName = "bielik_app_instance"
-$tokenFile = "my_hf_token.txt"
-
-Write-Host "Building Docker image..."
-docker build --secret id=huggingface_token,src=$tokenFile -t $imageName .
-
-Write-Host "Stopping and removing any existing container named $containerName..."
-docker stop $containerName | Out-Null 2>&1
-
-docker rm $containerName | Out-Null 2>&1
-
-Write-Host "Running new container..."
-docker run -d --name $containerName -p 8000:8000 $imageName
-
-Write-Host ""
-Write-Host "$containerName should be starting up."
-Write-Host "You can view logs with: docker logs $containerName -f"
-Write-Host "To stop the container, run: docker stop $containerName"
-Write-Host "The service will be available at http://127.0.0.1:8000"

start_container.sh

@@ -1,25 +0,0 @@
-#!/bin/bash
-
-IMAGE_NAME="bielik-fastapi-service"
-CONTAINER_NAME="bielik_app_instance"
-TOKEN_FILE="my_hf_token.txt"
-
-# Build the Docker image with Hugging Face token as a secret
-echo "Building Docker image..."
-DOCKER_BUILDKIT=1 docker build --secret id=huggingface_token,src=$TOKEN_FILE -t $IMAGE_NAME .
-
-echo "Attempting to stop and remove existing container named $CONTAINER_NAME (if any)..."
-docker stop $CONTAINER_NAME > /dev/null 2>&1 || true # Stop if running, ignore error if not
-docker rm $CONTAINER_NAME > /dev/null 2>&1 || true   # Remove if exists, ignore error if not
-
-echo "Starting new $IMAGE_NAME container as $CONTAINER_NAME..."
-docker run -d --name $CONTAINER_NAME -p 8000:8000 $IMAGE_NAME
-# -d : Runs the container in detached mode (in the background)
-# --name : Assigns a specific name to your running container instance
-# -p 8000:8000 : Maps port 8000 on your host to port 8000 in the container
-
-echo ""
-echo "$CONTAINER_NAME should be starting up."
-echo "You can view logs with: docker logs $CONTAINER_NAME -f"
-echo "To stop the container, run: docker stop $CONTAINER_NAME"
-echo "The service will be available at http://127.0.0.1:8000"