init

.dockerignore

@@ -1,59 +0,0 @@
-# Python
-__pycache__/
-*.py[cod]
-*$py.class
-*.so
-.Python
-*.egg-info/
-dist/
-build/
-*.egg
-
-# Virtual environments
-venv/
-env/
-ENV/
-
-# IDE
-.vscode/
-.idea/
-*.swp
-*.swo
-
-# Git
-.git/
-.gitignore
-
-# Environment
-.env
-.env.*
-
-# Testing
-.pytest_cache/
-.coverage
-htmlcov/
-
-# Documentation
-docs/
-*.md
-!README.md
-
-# Data (can be mounted as volumes)
-data/pdfs/*
-data/vector_db/*
-data/processed/*
-
-# Test files
-test_*.py
-*_test.py
-
-# Logs
-*.log
-
-# OS
-.DS_Store
-Thumbs.db
-
-# Temporary files
-*.tmp
-*.bak

ARCHITECTURE.md

@@ -1,259 +0,0 @@
-# SOCAR Document Processing System - Architecture
-
-## System Components
-
-### 1. OCR (Optical Character Recognition)
-**Model**: Azure Document Intelligence (`prebuilt-read`)
-**NOT** Llama-4-Maverick (LLM is for answer generation, not OCR)
-
-**Performance**:
-- ✅ **92.79% Character Success Rate (CSR)**
-- ✅ **55.59% Word Success Rate (WSR)**
-- ✅ Significantly better than Tesseract (25% CSR, 21% WSR)
-
-**Features**:
-- ✅ Multi-language support (Azerbaijani, Russian, English)
-- ✅ **Cyrillic alphabet PRESERVED** (Russian text stays in Cyrillic as-is)
-- ✅ **Image detection** (lightweight references via PyMuPDF)
-- ✅ Handwriting recognition
-- ✅ Table detection
-
-**Location**: `src/ocr/azure_ocr.py`
-
-**Output Format**:
-```json
-[
-  {
-    "page_number": 1,
-    "MD_text": "Text content with image references...\n\n![Image](document_page_1_image_1)\n\n"
-  }
-]
-```
-
-**Image Handling**:
-- Images detected and referenced inline in MD_text (not saved to disk)
-- Format: `![Image](document_page_X_image_Y)`
-- Simple lightweight references, no base64, no file storage
-- No separate "images" field - everything in MD_text
-- Only adds image markdown when images actually exist on page
-
----
-
-### 2. Embeddings (Vector Database Ingestion)
-**Model**: `BAAI/bge-large-en-v1.5`
-**Dimensions**: 1024 (matches Pinecone index)
-**Purpose**: Converts text chunks into numerical vectors for semantic search
-
-**Location**: `src/vectordb/pinecone_store.py:28`
-
-```python
-self.embedding_model = SentenceTransformer("BAAI/bge-large-en-v1.5")
-self.embedding_dimension = 1024
-```
-
-**Process**:
-1. Text chunks (600 chars, 100 overlap) → Embeddings (1024-dim vectors)
-2. Upload to Pinecone cloud vector database
-3. Enable semantic search (finds similar content)
-
----
-
-### 3. Vector Database
-**Service**: Pinecone (Cloud)
-**Index**: `hackathon`
-**Configuration**:
-- Dimensions: 1024
-- Metric: Cosine similarity
-- Cloud: AWS us-east-1
-- Type: Dense vectors
-- Capacity: On-demand
-
-**Location**: `src/vectordb/pinecone_store.py`
-
-**Stats**:
-- Total Documents: 1,241 chunks
-- Source PDFs: 28 documents
-- Embedding Model: BAAI/bge-large-en-v1.5
-
----
-
-### 4. LLM (Answer Generation)
-**Model**: `Llama-4-Maverick-17B-128E-Instruct-FP8` (Open-source)
-**Purpose**: Generates contextual answers based on retrieved documents
-**Provider**: Azure OpenAI
-
-**Location**: `src/config.py:31`
-
-```python
-llm_model = "Llama-4-Maverick-17B-128E-Instruct-FP8"
-```
-
-**Parameters**:
-- Temperature: 0.2 (deterministic, factual answers)
-- Max Tokens: 1000
-- Top-k Documents: 3 (from Pinecone)
-
----
-
-## RAG Pipeline Flow
-
-```
-User Query
-    ↓
-1. Generate Query Embedding (BAAI/bge-large-en-v1.5)
-    ↓
-2. Search Pinecone (Cosine similarity, top 3 docs)
-    ↓
-3. Retrieve Relevant Chunks
-    ↓
-4. Build Context (3 documents × 600 chars)
-    ↓
-5. LLM Generation (Llama-4-Maverick-17B)
-    ↓
-Response with Citations
-```
-
-**Average Response Time**: 6.7 seconds
-- Embedding generation: ~0.5s
-- Pinecone search: ~1.0s
-- LLM generation: ~3.2s
-- Network overhead: ~2.0s
-
----
-
-## Cyrillic Support
-
-**OCR Output** (Cyrillic preserved):
-```markdown
-# Добыча нефти в Азербайджане
-
-Южно-Каспийский бассейн...
-```
-
-**Chatbot** (Optional Azerbaijani conversion):
-- OCR: **Cyrillic preserved** (requirement)
-- Chatbot answers: Can be Azerbaijani alphabet (your choice)
-
----
-
-## Image Detection
-
-**Method**: PyMuPDF (fitz)
-**Format**: Simple text references (not saved to disk)
-**Included in**: OCR endpoint `/ocr`
-
-**Example Response**:
-```json
-[
-  {
-    "page_number": 1,
-    "MD_text": "Oil exploration map...\n\n![Image](document_06_page_1_image_1)\n\n"
-  }
-]
-```
-
-**Key Features**:
-- Lightweight: No file storage, no base64 encoding
-- Smart: Only adds image markdown when images exist
-- Clean: Simple references like `document_page_1_image_1`
-
----
-
-## API Endpoints
-
-### 1. `POST /ocr` - PDF Processing
-Extract text and detect images from PDF
-
-**Request**:
-```bash
-curl -X POST http://localhost:8000/ocr \
-  -F "file=@document.pdf"
-```
-
-**Response**:
-```json
-[
-  {
-    "page_number": 1,
-    "MD_text": "Нефтяные месторождения...\n\n![Image](document_page_1_image_1)\n\n"
-  }
-]
-```
-
-### 2. `POST /llm` - RAG Chatbot
-Ask questions about documents
-
-**Request**:
-```bash
-curl -X POST http://localhost:8000/llm \
-  -H "Content-Type: application/json" \
-  -d '[{"role": "user", "content": "What is SOCAR?"}]'
-```
-
-**Response**:
-```json
-{
-  "answer": "SOCAR (State Oil Company of Azerbaijan Republic)...",
-  "sources": [
-    {
-      "pdf_name": "document_06.pdf",
-      "page_number": 3,
-      "content": "SOCAR operates in..."
-    }
-  ]
-}
-```
-
----
-
-## Model Summary
-
-| Component | Model/Service | Purpose |
-|-----------|--------------|---------|
-| **OCR** | Azure Document Intelligence | Extract text + detect images (92.79% CSR, Cyrillic preserved) |
-| **Embeddings** | BAAI/bge-large-en-v1.5 (1024-dim) | Convert text → vectors for search |
-| **Vector DB** | Pinecone (AWS us-east-1) | Store & search 1,241 document chunks |
-| **LLM** | Llama-4-Maverick-17B-128E-Instruct-FP8 | Generate contextual answers (Open-source) |
-
----
-
-## Scoring Criteria Optimization
-
-### OCR Quality (50%)
-- ✅ Multi-language (Azerbaijani, Russian, English)
-- ✅ **High accuracy: 92.79% CSR, 55.59% WSR**
-- ✅ Cyrillic preservation (as required)
-- ✅ Image detection (lightweight references)
-- ✅ Azure Document Intelligence (enterprise-grade OCR)
-
-### LLM Quality (30%)
-- ✅ **Llama-4-Maverick-17B-128E-Instruct-FP8** (Open-source model)
-- ✅ RAG with 3-document retrieval
-- ✅ Source citations
-- ✅ Contextual accuracy
-
-### Architecture (20%)
-- ✅ Cloud vector database (Pinecone)
-- ✅ Production-ready (Docker, FastAPI)
-- ✅ Open-source LLM (Llama-4-Maverick)
-- ✅ Modern stack (Pinecone, sentence-transformers)
-
----
-
-## Public Access
-
-**ngrok URL**: `https://healthy-carolin-noncontagiously.ngrok-free.dev`
-
-**Test Query**:
-```bash
-curl -X POST https://healthy-carolin-noncontagiously.ngrok-free.dev/llm \
-  -H "Content-Type: application/json" \
-  -d '[{"role": "user", "content": "Что такое СОКАР?"}]'
-```
-**Expected**: Answer in Russian/Azerbaijani with source citations
-
----
-
-## Repository
-**GitHub**: https://github.com/Ismat-Samadov/SOCAR_Hackathon
-**Stack**: Python 3.10, FastAPI, Pinecone, Azure AI, Docker

Dockerfile

@@ -1,55 +0,0 @@
-# Multi-stage build for optimized Docker image
-FROM python:3.11-slim as builder
-
-# Set working directory
-WORKDIR /app
-
-# Install system dependencies
-RUN apt-get update && apt-get install -y \
-    build-essential \
-    curl \
-    && rm -rf /var/lib/apt/lists/*
-
-# Copy requirements first for better caching
-COPY requirements.txt .
-
-# Install Python dependencies
-RUN pip install --no-cache-dir --upgrade pip && \
-    pip install --no-cache-dir -r requirements.txt
-
-# Final stage
-FROM python:3.11-slim
-
-# Set working directory
-WORKDIR /app
-
-# Install runtime dependencies only
-RUN apt-get update && apt-get install -y \
-    curl \
-    && rm -rf /var/lib/apt/lists/*
-
-# Copy Python packages from builder
-COPY --from=builder /usr/local/lib/python3.11/site-packages /usr/local/lib/python3.11/site-packages
-COPY --from=builder /usr/local/bin /usr/local/bin
-
-# Copy application code
-COPY src/ ./src/
-COPY run.py .
-
-# Create directories for data
-RUN mkdir -p data/pdfs data/vector_db data/processed
-
-# Expose port
-EXPOSE 8000
-
-# Set environment variables
-ENV PYTHONUNBUFFERED=1
-ENV TOKENIZERS_PARALLELISM=false
-ENV ANONYMIZED_TELEMETRY=false
-
-# Health check
-HEALTHCHECK --interval=30s --timeout=10s --start-period=40s --retries=3 \
-    CMD curl -f http://localhost:8000/ || exit 1
-
-# Run the application
-CMD ["python", "run.py"]

PROJECT_KNOWLEDGE.md

@@ -1,473 +0,0 @@
-# SOCAR Hackathon Project - Essential Knowledge Base
-
-## Project Overview
-**Purpose**: Document processing system for SOCAR historical oil & gas documents
-**Goal**: OCR + RAG chatbot system for hackathon submission
-**Scoring**: OCR Quality (50%) + LLM Quality (30%) + Architecture (20%)
-
----
-
-## Critical Requirements
-
-### API Endpoints (Must Have)
-1. **POST /ocr** - PDF text extraction with image detection
-2. **POST /llm** - RAG-based question answering
-
-### Response Formats (Exact Specifications)
-
-#### /ocr Response
-```json
-[
-  {
-    "page_number": 1,
-    "MD_text": "Extracted text content...\n\n![Image](document_name.pdf/page_1/image_1)\n\n"
-  }
-]
-```
-
-**Key Points**:
-- List of dictionaries (NOT nested object with "pages" field)
-- Only two keys: `page_number` (int) and `MD_text` (str)
-- Images referenced inline in MD_text as markdown path: `![Image](document_name.pdf/page_X/image_Y)`
-- Path format: `{pdf_filename}/{page_folder}/{image_name}`
-- NO separate "images" field
-- NO base64 encoding - just simple path-like text references
-- Only add image markdown when images actually exist on page
-
-#### /llm Response
-```json
-{
-  "answer": "Response text...",
-  "sources": [
-    {
-      "pdf_name": "document_06.pdf",
-      "page_number": 3,
-      "content": "Relevant excerpt..."
-    }
-  ]
-}
-```
-
----
-
-## Technology Stack
-
-### 1. OCR (50% of score)
-**Model**: Azure Document Intelligence API
-- Endpoint: `prebuilt-read`
-- Provider: Azure AI Services
-- Performance: 92.79% CSR, 55.59% WSR
-- Features: Multi-language (Azerbaijani, Russian, English), Cyrillic preservation, handwriting recognition
-
-**Why Azure**:
-- Tesseract achieved only 25% CSR (4x worse)
-- Enterprise-grade accuracy
-- Native Cyrillic support
-
-**Configuration**:
-```python
-from azure.ai.formrecognizer import DocumentAnalysisClient
-from azure.core.credentials import AzureKeyCredential
-
-client = DocumentAnalysisClient(
-    endpoint=azure_openai_endpoint,
-    credential=AzureKeyCredential(api_key)
-)
-```
-
-### 2. Embeddings
-**Model**: `BAAI/bge-large-en-v1.5`
-- Dimensions: 1024
-- Library: sentence-transformers
-- Purpose: Convert text chunks to vectors for semantic search
-
-### 3. Vector Database
-**Service**: Pinecone Cloud
-- Index name: Configurable (e.g., "socar-documents")
-- Dimensions: 1024 (must match embedding model)
-- Metric: Cosine similarity
-- Region: AWS us-east-1
-
-**Alternative**: ChromaDB (local, good for development)
-
-### 4. LLM (30% of score)
-**Model**: `Llama-4-Maverick-17B-128E-Instruct-FP8`
-- Provider: Azure OpenAI (or compatible endpoint)
-- Parameters:
-  - Temperature: 0.2 (factual, deterministic)
-  - Max Tokens: 1000
-  - Top-k documents: 3
-- **Important**: Use this exact model name for open-source architecture points
-
-### 5. Web Framework
-**Framework**: FastAPI
-- Async support
-- Auto-generated docs at /docs
-- File upload support (multipart/form-data)
-
-### 6. Image Extraction
-**Library**: PyMuPDF (fitz)
-- Purpose: Detect images in PDFs
-- Method: `page.get_images()`
-- **Important**: Don't save images to disk, just reference them
-
----
-
-## Environment Variables (.env)
-
-```bash
-# Azure AI Services
-AZURE_OPENAI_API_KEY=your_key_here
-AZURE_OPENAI_ENDPOINT=https://your-endpoint.openai.azure.com/
-AZURE_OPENAI_API_VERSION=2024-08-01-preview
-
-# Pinecone (if using)
-PINECONE_API_KEY=your_key_here
-PINECONE_INDEX_NAME=socar-documents
-PINECONE_CLOUD=aws
-PINECONE_REGION=us-east-1
-
-# Application
-API_HOST=0.0.0.0
-API_PORT=8000
-```
-
----
-
-## RAG Pipeline Architecture
-
-```
-User Query
-    ↓
-1. Embed Query (BAAI/bge-large-en-v1.5)
-    ↓
-2. Search Vector DB (Cosine similarity, top 3 docs)
-    ↓
-3. Retrieve Relevant Chunks
-    ↓
-4. Build Context (3 documents × 600 chars)
-    ↓
-5. LLM Generation (Llama-4-Maverick-17B)
-    ↓
-Response with Citations
-```
-
-### Chunking Strategy
-- Chunk size: 600 characters
-- Overlap: 100 characters
-- Preserves context across chunks
-- **Important**: Only store extracted TEXT in vector database, NOT image markdown references
-- Strip out image markdown (`![Image](...)`) before ingestion
-- Images are only for OCR endpoint response, not for RAG
-
----
-
-## Key Features & Requirements
-
-### Cyrillic Support
-- **Critical**: OCR must preserve Cyrillic alphabet (Russian text stays in Cyrillic)
-- Don't transliterate Russian → Latin
-- Azure Document Intelligence handles this natively
-
-### Image Handling
-- **Method**: Detect images using PyMuPDF
-- **Format**: Path-like references: `document_name.pdf/page_X/image_Y`
-- **Markdown**: `![Image](document_name.pdf/page_1/image_1)`
-- **Path Structure**: `{pdf_filename}/{page_folder}/{image_name}`
-- **Important**:
-  - NO file saving
-  - NO base64 encoding
-  - Only add markdown when images exist
-  - Check `if image_list:` before adding
-  - Include full PDF filename in path
-
-### OCR vs Vector Database (Critical Distinction)
-**OCR Endpoint (`/ocr`)**:
-- Returns text WITH image markdown references
-- Format: `![Image](document.pdf/page_1/image_1)`
-- Purpose: Show complete document structure to user
-
-**Vector Database Ingestion**:
-- Store ONLY text content, NO image references
-- Strip out all `![Image](...)` markdown before adding to vector DB
-- Purpose: Enable semantic search on text only
-- Image markdown would pollute search results
-
-**Example**:
-```python
-# OCR Response (with images)
-MD_text = "Oil reserves...\n\n![Image](doc.pdf/page_1/image_1)\n\nTotal production..."
-
-# Vector DB ingestion (text only)
-import re
-text_only = re.sub(r'!\[Image\]\([^)]+\)', '', MD_text)
-# Result: "Oil reserves...\n\nTotal production..."
-vector_db.add(text_only)
-```
-
-### Public Access
-- Use ngrok for public endpoint
-- Command: `ngrok http 8000`
-- Free tier is sufficient
-- Save the URL for hackathon submission
-
----
-
-## Project Structure
-
-```
-SOCAR_Hackathon/
-├── src/
-│   ├── api/
-│   │   └── main.py          # FastAPI app with /ocr and /llm endpoints
-│   ├── ocr/
-│   │   ├── processor.py     # Main OCR interface
-│   │   └── azure_ocr.py     # Azure Document Intelligence implementation
-│   ├── vectordb/
-│   │   ├── pinecone_store.py # Pinecone operations
-│   │   └── chroma_store.py   # ChromaDB alternative
-│   ├── llm/
-│   │   └── chat.py          # LLM integration with RAG
-│   └── config.py            # Pydantic settings
-├── data/
-│   ├── pdfs/                # Input PDFs
-│   └── processed/           # Processed data
-├── run.py                   # Entry point
-├── requirements.txt
-├── .env
-└── docker-compose.yml       # Optional
-```
-
----
-
-## Implementation Guidelines
-
-### OCR Endpoint Implementation
-```python
-@app.post("/ocr")
-async def ocr_endpoint(file: UploadFile = File(...)):
-    # 1. Read PDF bytes
-    pdf_bytes = await file.read()
-    pdf_filename = file.filename  # e.g., "document_06.pdf"
-
-    # 2. Process with Azure Document Intelligence
-    # - Extract text page by page
-    # - Detect images with PyMuPDF
-    # - Create path-like references: pdf_filename/page_X/image_Y
-    # - Embed image references in MD_text
-
-    # 3. Example of adding image reference
-    # for each image detected on page 1:
-    #   md_text += f"\n\n![Image]({pdf_filename}/page_1/image_1)\n\n"
-
-    # 4. Return list of dicts
-    return [
-        {"page_number": 1, "MD_text": "text..."},
-        {"page_number": 2, "MD_text": "text..."}
-    ]
-```
-
-### LLM Endpoint Implementation
-```python
-@app.post("/llm")
-async def llm_endpoint(messages: List[Dict]):
-    # 1. Get last user message
-    user_query = messages[-1]["content"]
-
-    # 2. Generate embedding
-    query_embedding = embed_model.encode(user_query)
-
-    # 3. Search vector DB (top 3)
-    results = vector_db.search(query_embedding, top_k=3)
-
-    # 4. Build context from results
-    context = "\n\n".join([doc["content"] for doc in results])
-
-    # 5. Create prompt with context
-    prompt = f"Context:\n{context}\n\nQuestion: {user_query}"
-
-    # 6. Call LLM
-    response = llm_client.chat(
-        model="Llama-4-Maverick-17B-128E-Instruct-FP8",
-        messages=[{"role": "user", "content": prompt}],
-        temperature=0.2,
-        max_tokens=1000
-    )
-
-    # 7. Return answer with sources
-    return {
-        "answer": response.content,
-        "sources": [
-            {
-                "pdf_name": doc["metadata"]["pdf_name"],
-                "page_number": doc["metadata"]["page_number"],
-                "content": doc["content"][:200]
-            }
-            for doc in results
-        ]
-    }
-```
-
----
-
-## Critical Dos and Don'ts
-
-### ✅ DO
-- Use exact model names: `Llama-4-Maverick-17B-128E-Instruct-FP8`
-- Return exact response formats as specified
-- Preserve Cyrillic text from OCR
-- Only add image markdown when images exist
-- Include source citations in LLM responses
-- Use temperature 0.2 for factual answers
-- Chunk documents at 600 chars with 100 overlap
-
-### ❌ DON'T
-- Change response format (no extra fields or nesting)
-- Use base64 for images
-- Save images to disk
-- Transliterate Cyrillic to Latin
-- Add image markdown when no images exist
-- Use GPT models (use Llama for open-source points)
-- Skip error handling
-
----
-
-## Testing Commands
-
-### Test OCR
-```bash
-curl -X POST http://localhost:8000/ocr \
-  -F "file=@document.pdf"
-```
-
-### Test LLM
-```bash
-curl -X POST http://localhost:8000/llm \
-  -H "Content-Type: application/json" \
-  -d '[{"role": "user", "content": "What is SOCAR?"}]'
-```
-
-### Test with ngrok
-```bash
-# Replace localhost with ngrok URL
-curl -X POST https://your-url.ngrok-free.dev/llm \
-  -H "Content-Type: application/json" \
-  -d '[{"role": "user", "content": "Test question"}]'
-```
-
----
-
-## Common Issues & Solutions
-
-### Issue: Tesseract poor performance
-**Solution**: Use Azure Document Intelligence (92.79% vs 25% CSR)
-
-### Issue: Images bloating response
-**Solution**: Don't use base64, use path-like text references: `document_name.pdf/page_1/image_1`
-
-### Issue: Wrong response format
-**Solution**: Return list directly, not `{"pages": [...]}`
-
-### Issue: LLM taking too long
-**Solution**:
-- Limit to top 3 documents
-- Use temperature 0.2
-- Cap max_tokens at 1000
-
-### Issue: Cyrillic not preserved
-**Solution**: Azure handles this natively, ensure proper encoding throughout pipeline
-
----
-
-## Performance Targets
-
-- **OCR**: ~10-15 seconds for 12-page PDF
-- **LLM**: ~3-5 seconds per query
-- **Total RAG pipeline**: ~6-7 seconds (embedding + search + generation)
-
----
-
-## Dependencies (requirements.txt essentials)
-
-```txt
-# Web Framework
-fastapi==0.104.1
-uvicorn[standard]==0.24.0
-python-multipart==0.0.6
-
-# Azure AI
-azure-ai-formrecognizer==3.3.2
-openai==1.3.0
-
-# PDF Processing
-PyMuPDF==1.23.8
-
-# Vector DB & Embeddings
-pinecone-client==3.0.0
-sentence-transformers>=2.5.0
-
-# Utilities
-python-dotenv==1.0.0
-pydantic==2.5.0
-pydantic-settings==2.1.0
-loguru==0.7.2
-```
-
----
-
-## Deployment Checklist
-
-1. ✅ Set environment variables in .env
-2. ✅ Install dependencies: `pip install -r requirements.txt`
-3. ✅ Configure Azure Document Intelligence endpoint
-4. ✅ Set up Pinecone index (1024 dimensions, cosine metric)
-5. ✅ Ingest PDF documents into vector DB
-6. ✅ Start FastAPI server: `uvicorn src.api.main:app --host 0.0.0.0 --port 8000`
-7. ✅ Start ngrok: `ngrok http 8000`
-8. ✅ Test both endpoints
-9. ✅ Verify Cyrillic preservation
-10. ✅ Check image references in OCR output
-
----
-
-## Scoring Optimization
-
-### OCR Quality (50%)
-- Azure Document Intelligence ✅
-- Multi-language support ✅
-- 92.79% CSR ✅
-- Image detection ✅
-
-### LLM Quality (30%)
-- Open-source model (Llama) ✅
-- RAG with 3-doc retrieval ✅
-- Source citations ✅
-- Contextual accuracy ✅
-
-### Architecture (20%)
-- Cloud vector DB (Pinecone) ✅
-- Production-ready (FastAPI) ✅
-- Open-source LLM ✅
-- Modern stack ✅
-
----
-
-## Quick Start from Scratch
-
-1. Create project structure (see above)
-2. Copy .env template and fill in credentials
-3. Install dependencies
-4. Implement /ocr endpoint with Azure Document Intelligence
-5. Implement /llm endpoint with RAG pipeline
-6. Set up vector database (Pinecone or ChromaDB)
-7. Ingest PDF documents
-8. Test locally
-9. Deploy with ngrok
-10. Submit hackathon URL
-
----
-
-**Last Updated**: 2025-12-13
-**Model**: Llama-4-Maverick-17B-128E-Instruct-FP8
-**OCR**: Azure Document Intelligence (92.79% CSR)
-**Vector DB**: Pinecone (1,241 chunks from 28 PDFs)

README.md

@@ -1,195 +0,0 @@
-# SOCAR Historical Document Processing Challenge
-
-AI-powered system for processing historical handwritten and printed documents from SOCAR's oil and gas research archives.
-
-## Overview
-
-This solution transforms historical documents into an interactive, searchable knowledge base using:
-- **OCR Processing** - Extract text from handwritten and printed PDFs (multi-language support)
-- **Vector Database** - Store and retrieve document information efficiently
-- **RAG Chatbot** - Answer questions using historical document knowledge
-
-## Quick Start
-
-### Option 1: Docker Deployment (Recommended)
-
-#### Using Docker Compose
-
-```bash
-# Build and start the container
-docker-compose up -d
-
-# View logs
-docker-compose logs -f
-
-# Stop the container
-docker-compose down
-```
-
-#### Using Docker Directly
-
-```bash
-# Build the image
-docker build -t socar-document-processing .
-
-# Run the container
-docker run -d \
-  -p 8000:8000 \
-  -v $(pwd)/data:/app/data \
-  --env-file .env \
-  --name socar-api \
-  socar-document-processing
-
-# View logs
-docker logs -f socar-api
-
-# Stop the container
-docker stop socar-api
-```
-
-The API will be available at `http://localhost:8000`
-
-### Option 2: Local Python Setup
-
-#### 1. Install Dependencies
-
-```bash
-pip install -r requirements.txt
-```
-
-#### 2. Configure Environment
-
-Ensure `.env` file exists with your credentials:
-
-Required variables:
-- `AZURE_OPENAI_API_KEY` - Azure OpenAI API key
-- `AZURE_OPENAI_ENDPOINT` - Azure OpenAI endpoint URL
-- `LLM_MODEL` - Model name (default: Llama-4-Maverick-17B-128E-Instruct-FP8)
-
-#### 3. Run the API Server
-
-```bash
-python run.py
-```
-
-The API will be available at `http://localhost:8000`
-
-#### 4. Test the System
-
-```bash
-python test_complete_system.py
-```
-
-## API Endpoints
-
-### POST /ocr
-
-Extract text from PDF documents.
-
-**Request:**
-```bash
-curl -X POST http://localhost:8000/ocr \
-  -F "file=@document.pdf"
-```
-
-**Response:**
-```json
-[
-  {
-    "page_number": 1,
-    "MD_text": "## Section Title\nExtracted text..."
-  }
-]
-```
-
-### POST /llm
-
-Query documents using natural language.
-
-**Request:**
-```bash
-curl -X POST http://localhost:8000/llm \
-  -H "Content-Type: application/json" \
-  -d '[{"role": "user", "content": "What is this document about?"}]'
-```
-
-**Response:**
-```json
-{
-  "sources": [
-    {
-      "pdf_name": "document.pdf",
-      "page_number": 1,
-      "content": "Relevant text snippet..."
-    }
-  ],
-  "answer": "This document discusses..."
-}
-```
-
-## Project Structure
-
-```
-.
-├── src/
-│   ├── api/           # FastAPI endpoints
-│   ├── ocr/           # OCR processing modules
-│   ├── llm/           # LLM and RAG pipeline
-│   └── utils/         # Utility functions
-├── data/
-│   ├── pdfs/          # Input PDF documents
-│   ├── processed/     # Processed documents
-│   └── vector_db/     # Vector database storage
-├── tests/             # Test files
-├── run.py             # Application entry point
-└── requirements.txt   # Python dependencies
-```
-
-## Technologies
-
-- **OCR**: Azure Document Intelligence (multi-language support)
-- **Vector DB**: ChromaDB (local, open-source)
-- **LLM**: Llama-4-Maverick-17B (open-source, deployable)
-- **API**: FastAPI (async, high-performance)
-- **Embeddings**: Sentence Transformers (all-MiniLM-L6-v2)
-- **Deployment**: Docker, Docker Compose
-
-## Deployment
-
-### Docker Features
-
-- **Multi-stage build** - Optimized image size
-- **Health checks** - Automatic container monitoring
-- **Volume mounts** - Persistent data storage
-- **Environment variables** - Easy configuration
-- **Auto-restart** - Production-ready resilience
-
-### Production Deployment
-
-```bash
-# Build production image
-docker build -t socar-api:production .
-
-# Deploy with nginx reverse proxy
-docker network create socar-network
-docker run -d --name socar-api --network socar-network socar-api:production
-```
-
-## Development
-
-### Running Tests
-
-```bash
-pytest tests/
-```
-
-### Code Formatting
-
-```bash
-black src/
-flake8 src/
-```
-
-## License
-
-MIT License - SOCAR Hackathon 2024

docker-compose.yml

@@ -1,43 +0,0 @@
-version: '3.8'
-
-services:
-  socar-api:
-    build:
-      context: .
-      dockerfile: Dockerfile
-    container_name: socar-document-processing
-    ports:
-      - "8000:8000"
-    volumes:
-      # Mount data directories for persistence
-      - ./data/pdfs:/app/data/pdfs
-      - ./data/vector_db:/app/data/vector_db
-      - ./data/processed:/app/data/processed
-    environment:
-      # Azure OpenAI Configuration
-      - AZURE_OPENAI_API_KEY=${AZURE_OPENAI_API_KEY}
-      - AZURE_OPENAI_ENDPOINT=${AZURE_OPENAI_ENDPOINT}
-      - AZURE_OPENAI_API_VERSION=${AZURE_OPENAI_API_VERSION}
-      # Azure Document Intelligence
-      - AZURE_DOCUMENT_INTELLIGENCE_ENDPOINT=${AZURE_DOCUMENT_INTELLIGENCE_ENDPOINT}
-      - AZURE_DOCUMENT_INTELLIGENCE_KEY=${AZURE_DOCUMENT_INTELLIGENCE_KEY}
-      # Application Configuration
-      - LLM_MODEL=${LLM_MODEL:-Llama-4-Maverick-17B-128E-Instruct-FP8}
-      - API_HOST=0.0.0.0
-      - API_PORT=8000
-      # Performance
-      - TOKENIZERS_PARALLELISM=false
-      - ANONYMIZED_TELEMETRY=false
-    restart: unless-stopped
-    healthcheck:
-      test: ["CMD", "curl", "-f", "http://localhost:8000/"]
-      interval: 30s
-      timeout: 10s
-      retries: 3
-      start_period: 40s
-    networks:
-      - socar-network
-
-networks:
-  socar-network:
-    driver: bridge

ingest_pdfs.py

@@ -1,87 +0,0 @@
-"""Script to ingest all PDFs into the vector database"""
-
-from pathlib import Path
-from loguru import logger
-import sys
-
-from src.llm.rag_pipeline import get_rag_pipeline
-from src.ocr.processor import get_ocr_processor
-
-# Configure logging
-logger.remove()
-logger.add(sys.stderr, level="INFO")
-
-
-def ingest_pdfs(pdf_dir: str = "data/pdfs", limit: int = None):
-    """
-    Ingest all PDFs from directory into vector database
-
-    Args:
-        pdf_dir: Directory containing PDF files
-        limit: Optional limit on number of PDFs to process
-    """
-    pdf_path = Path(pdf_dir)
-
-    if not pdf_path.exists():
-        logger.error(f"PDF directory not found: {pdf_dir}")
-        return
-
-    # Get all PDF files
-    pdf_files = list(pdf_path.glob("*.pdf"))
-    logger.info(f"Found {len(pdf_files)} PDF files")
-
-    if limit:
-        pdf_files = pdf_files[:limit]
-        logger.info(f"Processing only first {limit} files")
-
-    # Initialize components
-    ocr = get_ocr_processor()
-    rag = get_rag_pipeline()
-
-    # Process each PDF
-    for idx, pdf_file in enumerate(pdf_files, 1):
-        try:
-            logger.info(f"[{idx}/{len(pdf_files)}] Processing: {pdf_file.name}")
-
-            # Read PDF
-            with open(pdf_file, "rb") as f:
-                pdf_content = f.read()
-
-            # Extract text with OCR
-            pages = ocr.process_pdf(pdf_content, pdf_file.name)
-            logger.info(f"Extracted {len(pages)} pages from {pdf_file.name}")
-
-            # Add to vector database
-            rag.add_processed_document(pdf_file.name, pages)
-
-            logger.info(f"Successfully ingested {pdf_file.name}")
-
-        except Exception as e:
-            logger.error(f"Error processing {pdf_file.name}: {e}")
-            continue
-
-    # Print stats
-    stats = rag.vector_store.get_stats()
-    logger.info(f"\nIngestion complete!")
-    logger.info(f"Total documents in vector store: {stats['total_documents']}")
-
-
-if __name__ == "__main__":
-    import argparse
-
-    parser = argparse.ArgumentParser(description="Ingest PDFs into vector database")
-    parser.add_argument(
-        "--dir",
-        type=str,
-        default="data/pdfs",
-        help="Directory containing PDF files",
-    )
-    parser.add_argument(
-        "--limit",
-        type=int,
-        default=None,
-        help="Limit number of PDFs to process (for testing)",
-    )
-
-    args = parser.parse_args()
-    ingest_pdfs(args.dir, args.limit)

notebooks/llm_benchmark.ipynb

@@ -0,0 +1,901 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# LLM Benchmarking for SOCAR Hackathon RAG Chatbot\n",
+    "\n",
+    "This notebook tests different LLM models for the `/llm` endpoint to find the best performer.\n",
+    "\n",
+    "## Evaluation Criteria (LLM Judge Metrics):\n",
+    "- **Accuracy**: Is the answer correct?\n",
+    "- **Relevance**: Are retrieved citations relevant?\n",
+    "- **Completeness**: Does it fully answer the question?\n",
+    "- **Citation Quality**: Proper sources with page numbers?\n",
+    "- **Response Time**: Speed of generation\n",
+    "\n",
+    "## Available LLM Models:\n",
+    "1. **Llama-4-Maverick-17B-128E-Instruct-FP8** (Current choice, open-source)\n",
+    "2. **DeepSeek-R1** (Open-source reasoning model)\n",
+    "3. **GPT-4.1** (Strong general performance)\n",
+    "4. **GPT-5, GPT-5-mini**\n",
+    "5. **Claude Sonnet 4.5** (Best quality)\n",
+    "6. **Claude Opus 4.1**\n",
+    "7. **Phi-4-multimodal-instruct**\n",
+    "8. **gpt-oss-120b**"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Install required packages\n",
+    "# !pip install openai pinecone-client sentence-transformers python-dotenv pandas matplotlib seaborn jiwer"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/Users/ismatsamadov/SOCAR_Hackathon/venv/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
+      "  from .autonotebook import tqdm as notebook_tqdm\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "✅ Libraries loaded successfully\n"
+     ]
+    }
+   ],
+   "source": [
+    "import os\n",
+    "import json\n",
+    "import time\n",
+    "from typing import Dict, List, Tuple\n",
+    "from dotenv import load_dotenv\n",
+    "import pandas as pd\n",
+    "import matplotlib.pyplot as plt\n",
+    "import seaborn as sns\n",
+    "from openai import AzureOpenAI\n",
+    "from pinecone import Pinecone\n",
+    "from sentence_transformers import SentenceTransformer\n",
+    "from jiwer import wer, cer\n",
+    "\n",
+    "# Load environment variables\n",
+    "load_dotenv()\n",
+    "\n",
+    "# Set style\n",
+    "sns.set_style('whitegrid')\n",
+    "plt.rcParams['figure.figsize'] = (14, 8)\n",
+    "\n",
+    "print(\"✅ Libraries loaded successfully\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 1. Load Test Questions and Expected Answers"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Loaded 5 test cases\n",
+      "\n",
+      "Test Questions:\n",
+      "1. Example1: Daha az quyu ilə daha çox hasilat əldə etmək üçün hansı əsas amillərin inteqrasiyası tələb olunur?...\n",
+      "2. Example2: Qərbi Abşeron yatağında suvurma tədbirləri hansı tarixdə və hansı layda tətbiq edilmişdir və bunun m...\n",
+      "3. Example3: Pirallahı strukturunda 1253 nömrəli quyudan götürülmüş nümunələrdə SiO2 və CaO oksidləri arasında ha...\n",
+      "4. Example4: Bakı arxipelaqı (BA) və Aşağı Kür çökəkliyi (AKÇ) üçün geotemperatur xəritələrinə əsasən neft və qaz...\n",
+      "5. Example5: Bu zonada hansı proseslər baş verir?...\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Load sample questions\n",
+    "with open('docs/sample_questions.json', 'r', encoding='utf-8') as f:\n",
+    "    questions = json.load(f)\n",
+    "\n",
+    "# Load expected answers\n",
+    "with open('docs/sample_answers.json', 'r', encoding='utf-8') as f:\n",
+    "    expected_answers = json.load(f)\n",
+    "\n",
+    "print(f\"Loaded {len(questions)} test cases\")\n",
+    "print(\"\\nTest Questions:\")\n",
+    "for i, (key, msgs) in enumerate(questions.items(), 1):\n",
+    "    user_msg = [m for m in msgs if m['role'] == 'user'][-1]\n",
+    "    print(f\"{i}. {key}: {user_msg['content'][:100]}...\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 2. Initialize Vector Database and Embedding Model"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "✅ Vector DB connected: {'_response_info': {'raw_headers': {'connection': 'keep-alive',\n",
+      "                                    'content-length': '188',\n",
+      "                                    'content-type': 'application/json',\n",
+      "                                    'date': 'Sun, 14 Dec 2025 03:21:33 GMT',\n",
+      "                                    'grpc-status': '0',\n",
+      "                                    'server': 'envoy',\n",
+      "                                    'x-envoy-upstream-service-time': '4',\n",
+      "                                    'x-pinecone-request-id': '3979707437017514155',\n",
+      "                                    'x-pinecone-request-latency-ms': '4'}},\n",
+      " 'dimension': 1024,\n",
+      " 'index_fullness': 0.0,\n",
+      " 'memoryFullness': 0.0,\n",
+      " 'metric': 'cosine',\n",
+      " 'namespaces': {'__default__': {'vector_count': 1300}},\n",
+      " 'storageFullness': 0.0,\n",
+      " 'total_vector_count': 1300,\n",
+      " 'vector_type': 'dense'}\n",
+      "✅ Embedding model loaded: SentenceTransformer(\n",
+      "  (0): Transformer({'max_seq_length': 512, 'do_lower_case': True, 'architecture': 'BertModel'})\n",
+      "  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})\n",
+      "  (2): Normalize()\n",
+      ")\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Initialize Pinecone\n",
+    "pc = Pinecone(api_key=os.getenv('PINECONE_API_KEY'))\n",
+    "index = pc.Index(os.getenv('PINECONE_INDEX_NAME', 'hackathon'))\n",
+    "\n",
+    "# Initialize embedding model (same as used for ingestion)\n",
+    "embed_model = SentenceTransformer('BAAI/bge-large-en-v1.5')\n",
+    "\n",
+    "print(f\"✅ Vector DB connected: {index.describe_index_stats()}\")\n",
+    "print(f\"✅ Embedding model loaded: {embed_model}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 3. RAG Retrieval Function"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "✅ Retrieved 3 documents for test query\n",
+      "Top result: document_10.pdf, page 8 (score: 0.767)\n"
+     ]
+    }
+   ],
+   "source": [
+    "def retrieve_documents(query: str, top_k: int = 3) -> List[Dict]:\n",
+    "    \"\"\"\n",
+    "    Retrieve relevant documents from vector database.\n",
+    "    \"\"\"\n",
+    "    # Generate query embedding\n",
+    "    query_embedding = embed_model.encode(query).tolist()\n",
+    "    \n",
+    "    # Search vector DB\n",
+    "    results = index.query(\n",
+    "        vector=query_embedding,\n",
+    "        top_k=top_k,\n",
+    "        include_metadata=True\n",
+    "    )\n",
+    "    \n",
+    "    # Extract documents\n",
+    "    documents = []\n",
+    "    for match in results['matches']:\n",
+    "        documents.append({\n",
+    "            'pdf_name': match['metadata'].get('pdf_name', 'unknown.pdf'),\n",
+    "            'page_number': match['metadata'].get('page_number', 0),\n",
+    "            'content': match['metadata'].get('text', ''),\n",
+    "            'score': match.get('score', 0.0)\n",
+    "        })\n",
+    "    \n",
+    "    return documents\n",
+    "\n",
+    "# Test retrieval\n",
+    "test_query = \"Palçıq vulkanlarının təsir radiusu nə qədərdir?\"\n",
+    "test_docs = retrieve_documents(test_query)\n",
+    "print(f\"\\n✅ Retrieved {len(test_docs)} documents for test query\")\n",
+    "print(f\"Top result: {test_docs[0]['pdf_name']}, page {test_docs[0]['page_number']} (score: {test_docs[0]['score']:.3f})\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 4. LLM Client Functions"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": "# Initialize Azure OpenAI\nazure_client = AzureOpenAI(\n    api_key=os.getenv('AZURE_OPENAI_API_KEY'),\n    api_version=os.getenv('AZURE_OPENAI_API_VERSION', '2024-08-01-preview'),\n    azure_endpoint=os.getenv('AZURE_OPENAI_ENDPOINT')\n)\n\nLLM_MODELS = {\n    'Llama-4-Maverick': 'Llama-4-Maverick-17B-128E-Instruct-FP8',\n    'DeepSeek-R1': 'DeepSeek-R1',\n    'GPT-4.1': 'gpt-4.1',\n    'GPT-5-mini': 'gpt-5-mini',\n    'Claude-Sonnet-4.5': 'claude-sonnet-4-5',\n}\n\ndef generate_answer(model_name: str, query: str, documents: List[Dict], \n                   temperature: float = 0.2, max_tokens: int = 1000) -> Tuple[str, float]:\n    \"\"\"\n    Generate answer using specified LLM model.\n    Returns: (answer, response_time)\n    \"\"\"\n    # Build context from retrieved documents\n    context_parts = []\n    for i, doc in enumerate(documents, 1):\n        context_parts.append(\n            f\"Document {i} (Source: {doc['pdf_name']}, Page {doc['page_number']}):\\n{doc['content']}\"\n        )\n    context = \"\\n\\n\".join(context_parts)\n    \n    # Create prompt\n    prompt = f\"\"\"Siz SOCAR-ın tarixi neft və qaz sənədləri üzrə mütəxəssis köməkçisisiniz.\n\nKontekst (əlaqəli sənədlər):\n{context}\n\nSual: {query}\n\nƏtraflı cavab verin və mütləq sənəd mənbələrinə istinad edin (PDF adı və səhifə nömrəsi ilə).\nCavabınız dəqiq, faktlara əsaslanan və kontekst məlumatlarından istifadə edən olmalıdır.\"\"\"\n    \n    # Get model deployment\n    deployment = MODELS[model_name]['deployment']\n    \n    try:\n        start_time = time.time()\n        \n        # GPT-5 models use max_completion_tokens, others use max_tokens\n        if deployment.startswith('gpt-5'):\n            response = azure_client.chat.completions.create(\n                model=deployment,\n                messages=[\n                    {\"role\": \"user\", \"content\": prompt}\n                ],\n                temperature=temperature,\n                max_completion_tokens=max_tokens\n            )\n        else:\n            response = azure_client.chat.completions.create(\n                model=deployment,\n                messages=[\n                    {\"role\": \"user\", \"content\": prompt}\n                ],\n                temperature=temperature,\n                max_tokens=max_tokens\n            )\n        \n        response_time = time.time() - start_time\n        answer = response.choices[0].message.content\n        \n        return answer, response_time\n    \n    except Exception as e:\n        return f\"ERROR: {str(e)}\", 0.0\n\nprint(f\"\\n✅ Configured {len(LLM_MODELS)} LLM models for testing\")"
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 5. Evaluation Metrics"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "✅ Evaluation functions ready\n"
+     ]
+    }
+   ],
+   "source": [
+    "def normalize_text(text: str) -> str:\n",
+    "    \"\"\"Normalize text for comparison.\"\"\"\n",
+    "    import re\n",
+    "    text = text.lower().strip()\n",
+    "    text = re.sub(r'\\s+', ' ', text)\n",
+    "    return text\n",
+    "\n",
+    "def calculate_answer_similarity(reference: str, hypothesis: str) -> Dict[str, float]:\n",
+    "    \"\"\"\n",
+    "    Calculate similarity between generated and expected answer.\n",
+    "    Lower is better for error rates.\n",
+    "    \"\"\"\n",
+    "    ref_norm = normalize_text(reference)\n",
+    "    hyp_norm = normalize_text(hypothesis)\n",
+    "    \n",
+    "    # Character Error Rate\n",
+    "    cer_score = cer(ref_norm, hyp_norm) * 100\n",
+    "    \n",
+    "    # Word Error Rate  \n",
+    "    wer_score = wer(ref_norm, hyp_norm) * 100\n",
+    "    \n",
+    "    # Similarity scores (higher is better)\n",
+    "    similarity = max(0, 100 - wer_score)\n",
+    "    \n",
+    "    return {\n",
+    "        'CER': round(cer_score, 2),\n",
+    "        'WER': round(wer_score, 2),\n",
+    "        'Similarity': round(similarity, 2)\n",
+    "    }\n",
+    "\n",
+    "def check_citations(answer: str, documents: List[Dict]) -> Dict[str, any]:\n",
+    "    \"\"\"\n",
+    "    Check if answer includes proper citations.\n",
+    "    \"\"\"\n",
+    "    import re\n",
+    "    \n",
+    "    # Check for PDF names\n",
+    "    pdf_names = [doc['pdf_name'] for doc in documents]\n",
+    "    cited_pdfs = sum(1 for pdf in pdf_names if pdf.replace('.pdf', '') in answer)\n",
+    "    \n",
+    "    # Check for page numbers\n",
+    "    page_numbers = [str(doc['page_number']) for doc in documents]\n",
+    "    cited_pages = sum(1 for page in page_numbers if page in answer)\n",
+    "    \n",
+    "    # Check for source keywords\n",
+    "    source_keywords = ['mənbə', 'sənəd', 'səhifə', 'pdf', 'document', 'page', 'source']\n",
+    "    has_source_ref = any(kw in answer.lower() for kw in source_keywords)\n",
+    "    \n",
+    "    citation_score = (\n",
+    "        (cited_pdfs / len(pdf_names) * 40) +  # 40% for PDF citation\n",
+    "        (cited_pages / len(page_numbers) * 40) +  # 40% for page citation\n",
+    "        (20 if has_source_ref else 0)  # 20% for having source keywords\n",
+    "    )\n",
+    "    \n",
+    "    return {\n",
+    "        'Citation_Score': round(citation_score, 2),\n",
+    "        'Cited_PDFs': cited_pdfs,\n",
+    "        'Cited_Pages': cited_pages,\n",
+    "        'Has_Source_Reference': has_source_ref\n",
+    "    }\n",
+    "\n",
+    "def evaluate_completeness(answer: str, min_length: int = 100) -> Dict[str, any]:\n",
+    "    \"\"\"\n",
+    "    Evaluate answer completeness.\n",
+    "    \"\"\"\n",
+    "    word_count = len(answer.split())\n",
+    "    char_count = len(answer)\n",
+    "    \n",
+    "    # Penalize very short or very long answers\n",
+    "    if char_count < min_length:\n",
+    "        completeness_score = (char_count / min_length) * 100\n",
+    "    elif char_count > 2000:\n",
+    "        completeness_score = 100 - ((char_count - 2000) / 2000 * 20)  # Penalty for verbosity\n",
+    "    else:\n",
+    "        completeness_score = 100\n",
+    "    \n",
+    "    return {\n",
+    "        'Completeness_Score': round(max(0, completeness_score), 2),\n",
+    "        'Word_Count': word_count,\n",
+    "        'Char_Count': char_count\n",
+    "    }\n",
+    "\n",
+    "print(\"✅ Evaluation functions ready\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 6. Run Benchmark on All Models"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Testing 5 models on 5 questions...\n",
+      "\n",
+      "This may take several minutes...\n",
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Select models to test (you can comment out models to skip)\n",
+    "MODELS_TO_TEST = [\n",
+    "    'Llama-4-Maverick-17B',\n",
+    "    'DeepSeek-R1',\n",
+    "    'GPT-4.1',\n",
+    "    'GPT-5-mini',\n",
+    "    'Claude-Sonnet-4.5',\n",
+    "    # 'Claude-Opus-4.1',  # Uncomment to test\n",
+    "    # 'Phi-4-multimodal',  # Uncomment to test\n",
+    "    # 'GPT-OSS-120B',  # Uncomment to test\n",
+    "]\n",
+    "\n",
+    "print(f\"Testing {len(MODELS_TO_TEST)} models on {len(questions)} questions...\\n\")\n",
+    "print(\"This may take several minutes...\\n\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "================================================================================\n",
+      "Testing: Llama-4-Maverick-17B\n",
+      "================================================================================\n",
+      "\n",
+      "  Question Example1: Daha az quyu ilə daha çox hasilat əldə etmək üçün hansı əsas amillərin inteqrasi...\n",
+      "  ✅ Response time: 4.39s\n",
+      "\n",
+      "  Question Example2: Qərbi Abşeron yatağında suvurma tədbirləri hansı tarixdə və hansı layda tətbiq e...\n",
+      "  ✅ Response time: 3.74s\n",
+      "\n",
+      "  Question Example3: Pirallahı strukturunda 1253 nömrəli quyudan götürülmüş nümunələrdə SiO2 və CaO o...\n",
+      "  ✅ Response time: 4.07s\n",
+      "\n",
+      "  Question Example4: Bakı arxipelaqı (BA) və Aşağı Kür çökəkliyi (AKÇ) üçün geotemperatur xəritələrin...\n",
+      "  ✅ Response time: 4.20s\n",
+      "\n",
+      "  Question Example5: Bu zonada hansı proseslər baş verir?...\n",
+      "  ✅ Response time: 3.50s\n",
+      "\n",
+      "  📊 Llama-4-Maverick-17B Summary:\n",
+      "     Avg Response Time: 3.98s\n",
+      "     Avg Similarity: 0.0%\n",
+      "     Avg Citation Score: 84.0%\n",
+      "     Avg Completeness: 100.0%\n",
+      "\n",
+      "================================================================================\n",
+      "Testing: DeepSeek-R1\n",
+      "================================================================================\n",
+      "\n",
+      "  Question Example1: Daha az quyu ilə daha çox hasilat əldə etmək üçün hansı əsas amillərin inteqrasi...\n",
+      "  ✅ Response time: 10.00s\n",
+      "\n",
+      "  Question Example2: Qərbi Abşeron yatağında suvurma tədbirləri hansı tarixdə və hansı layda tətbiq e...\n",
+      "  ✅ Response time: 10.39s\n",
+      "\n",
+      "  Question Example3: Pirallahı strukturunda 1253 nömrəli quyudan götürülmüş nümunələrdə SiO2 və CaO o...\n",
+      "  ✅ Response time: 10.73s\n",
+      "\n",
+      "  Question Example4: Bakı arxipelaqı (BA) və Aşağı Kür çökəkliyi (AKÇ) üçün geotemperatur xəritələrin...\n",
+      "  ✅ Response time: 12.17s\n",
+      "\n",
+      "  Question Example5: Bu zonada hansı proseslər baş verir?...\n",
+      "  ✅ Response time: 10.72s\n",
+      "\n",
+      "  📊 DeepSeek-R1 Summary:\n",
+      "     Avg Response Time: 10.80s\n",
+      "     Avg Similarity: 0.0%\n",
+      "     Avg Citation Score: 80.0%\n",
+      "     Avg Completeness: 67.7%\n",
+      "\n",
+      "================================================================================\n",
+      "Testing: GPT-4.1\n",
+      "================================================================================\n",
+      "\n",
+      "  Question Example1: Daha az quyu ilə daha çox hasilat əldə etmək üçün hansı əsas amillərin inteqrasi...\n",
+      "  ✅ Response time: 6.66s\n",
+      "\n",
+      "  Question Example2: Qərbi Abşeron yatağında suvurma tədbirləri hansı tarixdə və hansı layda tətbiq e...\n",
+      "  ✅ Response time: 5.05s\n",
+      "\n",
+      "  Question Example3: Pirallahı strukturunda 1253 nömrəli quyudan götürülmüş nümunələrdə SiO2 və CaO o...\n",
+      "  ✅ Response time: 7.65s\n",
+      "\n",
+      "  Question Example4: Bakı arxipelaqı (BA) və Aşağı Kür çökəkliyi (AKÇ) üçün geotemperatur xəritələrin...\n",
+      "  ✅ Response time: 6.68s\n",
+      "\n",
+      "  Question Example5: Bu zonada hansı proseslər baş verir?...\n",
+      "  ✅ Response time: 3.69s\n",
+      "\n",
+      "  📊 GPT-4.1 Summary:\n",
+      "     Avg Response Time: 5.95s\n",
+      "     Avg Similarity: 0.0%\n",
+      "     Avg Citation Score: 84.0%\n",
+      "     Avg Completeness: 93.5%\n",
+      "\n",
+      "================================================================================\n",
+      "Testing: GPT-5-mini\n",
+      "================================================================================\n",
+      "\n",
+      "  Question Example1: Daha az quyu ilə daha çox hasilat əldə etmək üçün hansı əsas amillərin inteqrasi...\n",
+      "  ❌ Failed: ERROR: Error code: 400 - {'error': {'message': \"Unsupported parameter: 'max_tokens' is not supported with this model. Use 'max_completion_tokens' instead.\", 'type': 'invalid_request_error', 'param': 'max_tokens', 'code': 'unsupported_parameter'}}\n",
+      "\n",
+      "  Question Example2: Qərbi Abşeron yatağında suvurma tədbirləri hansı tarixdə və hansı layda tətbiq e...\n",
+      "  ❌ Failed: ERROR: Error code: 400 - {'error': {'message': \"Unsupported parameter: 'max_tokens' is not supported with this model. Use 'max_completion_tokens' instead.\", 'type': 'invalid_request_error', 'param': 'max_tokens', 'code': 'unsupported_parameter'}}\n",
+      "\n",
+      "  Question Example3: Pirallahı strukturunda 1253 nömrəli quyudan götürülmüş nümunələrdə SiO2 və CaO o...\n",
+      "  ❌ Failed: ERROR: Error code: 400 - {'error': {'message': \"Unsupported parameter: 'max_tokens' is not supported with this model. Use 'max_completion_tokens' instead.\", 'type': 'invalid_request_error', 'param': 'max_tokens', 'code': 'unsupported_parameter'}}\n",
+      "\n",
+      "  Question Example4: Bakı arxipelaqı (BA) və Aşağı Kür çökəkliyi (AKÇ) üçün geotemperatur xəritələrin...\n",
+      "  ❌ Failed: ERROR: Error code: 400 - {'error': {'message': \"Unsupported parameter: 'max_tokens' is not supported with this model. Use 'max_completion_tokens' instead.\", 'type': 'invalid_request_error', 'param': 'max_tokens', 'code': 'unsupported_parameter'}}\n",
+      "\n",
+      "  Question Example5: Bu zonada hansı proseslər baş verir?...\n",
+      "  ❌ Failed: ERROR: Error code: 400 - {'error': {'message': \"Unsupported parameter: 'max_tokens' is not supported with this model. Use 'max_completion_tokens' instead.\", 'type': 'invalid_request_error', 'param': 'max_tokens', 'code': 'unsupported_parameter'}}\n",
+      "\n",
+      "================================================================================\n",
+      "Testing: Claude-Sonnet-4.5\n",
+      "================================================================================\n",
+      "\n",
+      "  Question Example1: Daha az quyu ilə daha çox hasilat əldə etmək üçün hansı əsas amillərin inteqrasi...\n",
+      "  ❌ Failed: ERROR: Error code: 400 - {'error': {'code': 'unknown_model', 'message': 'Unknown model: claude-sonnet-4-5', 'details': 'Unknown model: claude-sonnet-4-5'}}\n",
+      "\n",
+      "  Question Example2: Qərbi Abşeron yatağında suvurma tədbirləri hansı tarixdə və hansı layda tətbiq e...\n",
+      "  ❌ Failed: ERROR: Error code: 400 - {'error': {'code': 'unknown_model', 'message': 'Unknown model: claude-sonnet-4-5', 'details': 'Unknown model: claude-sonnet-4-5'}}\n",
+      "\n",
+      "  Question Example3: Pirallahı strukturunda 1253 nömrəli quyudan götürülmüş nümunələrdə SiO2 və CaO o...\n",
+      "  ❌ Failed: ERROR: Error code: 400 - {'error': {'code': 'unknown_model', 'message': 'Unknown model: claude-sonnet-4-5', 'details': 'Unknown model: claude-sonnet-4-5'}}\n",
+      "\n",
+      "  Question Example4: Bakı arxipelaqı (BA) və Aşağı Kür çökəkliyi (AKÇ) üçün geotemperatur xəritələrin...\n",
+      "  ❌ Failed: ERROR: Error code: 400 - {'error': {'code': 'unknown_model', 'message': 'Unknown model: claude-sonnet-4-5', 'details': 'Unknown model: claude-sonnet-4-5'}}\n",
+      "\n",
+      "  Question Example5: Bu zonada hansı proseslər baş verir?...\n",
+      "  ❌ Failed: ERROR: Error code: 400 - {'error': {'code': 'unknown_model', 'message': 'Unknown model: claude-sonnet-4-5', 'details': 'Unknown model: claude-sonnet-4-5'}}\n",
+      "\n",
+      "================================================================================\n",
+      "✅ Benchmarking complete!\n",
+      "================================================================================\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Run benchmark\n",
+    "results = []\n",
+    "\n",
+    "for model_name in MODELS_TO_TEST:\n",
+    "    print(f\"\\n{'='*80}\")\n",
+    "    print(f\"Testing: {model_name}\")\n",
+    "    print(f\"{'='*80}\")\n",
+    "    \n",
+    "    model_results = []\n",
+    "    \n",
+    "    for example_key, messages in questions.items():\n",
+    "        # Get the last user message (the actual question)\n",
+    "        user_msg = [m for m in messages if m['role'] == 'user'][-1]\n",
+    "        query = user_msg['content']\n",
+    "        \n",
+    "        print(f\"\\n  Question {example_key}: {query[:80]}...\")\n",
+    "        \n",
+    "        # Retrieve documents\n",
+    "        documents = retrieve_documents(query, top_k=3)\n",
+    "        \n",
+    "        # Generate answer\n",
+    "        answer, response_time = generate_answer(model_name, query, documents)\n",
+    "        \n",
+    "        if answer.startswith('ERROR'):\n",
+    "            print(f\"  ❌ Failed: {answer}\")\n",
+    "            continue\n",
+    "        \n",
+    "        print(f\"  ✅ Response time: {response_time:.2f}s\")\n",
+    "        \n",
+    "        # Get expected answer\n",
+    "        expected = expected_answers.get(example_key, {}).get('Answer', '')\n",
+    "        \n",
+    "        # Calculate metrics\n",
+    "        similarity_metrics = calculate_answer_similarity(expected, answer) if expected else {'CER': 0, 'WER': 0, 'Similarity': 0}\n",
+    "        citation_metrics = check_citations(answer, documents)\n",
+    "        completeness_metrics = evaluate_completeness(answer)\n",
+    "        \n",
+    "        # Store result\n",
+    "        result = {\n",
+    "            'Model': model_name,\n",
+    "            'Question': example_key,\n",
+    "            'Query': query[:100],\n",
+    "            'Answer': answer[:200] + '...',\n",
+    "            'Response_Time': round(response_time, 2),\n",
+    "            **similarity_metrics,\n",
+    "            **citation_metrics,\n",
+    "            **completeness_metrics,\n",
+    "            'Open_Source': MODELS[model_name]['open_source'],\n",
+    "            'Architecture_Score': MODELS[model_name]['architecture_score']\n",
+    "        }\n",
+    "        \n",
+    "        model_results.append(result)\n",
+    "        results.append(result)\n",
+    "    \n",
+    "    # Show summary for this model\n",
+    "    if model_results:\n",
+    "        avg_response_time = sum(r['Response_Time'] for r in model_results) / len(model_results)\n",
+    "        avg_similarity = sum(r['Similarity'] for r in model_results) / len(model_results)\n",
+    "        avg_citation = sum(r['Citation_Score'] for r in model_results) / len(model_results)\n",
+    "        avg_completeness = sum(r['Completeness_Score'] for r in model_results) / len(model_results)\n",
+    "        \n",
+    "        print(f\"\\n  📊 {model_name} Summary:\")\n",
+    "        print(f\"     Avg Response Time: {avg_response_time:.2f}s\")\n",
+    "        print(f\"     Avg Similarity: {avg_similarity:.1f}%\")\n",
+    "        print(f\"     Avg Citation Score: {avg_citation:.1f}%\")\n",
+    "        print(f\"     Avg Completeness: {avg_completeness:.1f}%\")\n",
+    "\n",
+    "print(f\"\\n{'='*80}\")\n",
+    "print(\"✅ Benchmarking complete!\")\n",
+    "print(f\"{'='*80}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 7. Aggregate Results and Rankings"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "====================================================================================================\n",
+      "📊 LLM BENCHMARKING RESULTS - MODEL SUMMARY\n",
+      "====================================================================================================\n",
+      "                      Response_Time  Similarity  Citation_Score  Completeness_Score     CER     WER  Open_Source Architecture_Score  Quality_Score\n",
+      "Model                                                                                                                                             \n",
+      "Llama-4-Maverick-17B           3.98         0.0            84.0              100.00  330.97  378.42         True               High          59.40\n",
+      "GPT-4.1                        5.95         0.0            84.0               93.54  755.19  780.64        False             Medium          57.46\n",
+      "DeepSeek-R1                   10.80         0.0            80.0               67.73  855.43  992.02         True               High          48.32\n",
+      "====================================================================================================\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Create DataFrame\n",
+    "df = pd.DataFrame(results)\n",
+    "\n",
+    "# Calculate aggregate scores per model\n",
+    "model_summary = df.groupby('Model').agg({\n",
+    "    'Response_Time': 'mean',\n",
+    "    'Similarity': 'mean',\n",
+    "    'Citation_Score': 'mean',\n",
+    "    'Completeness_Score': 'mean',\n",
+    "    'CER': 'mean',\n",
+    "    'WER': 'mean',\n",
+    "    'Open_Source': 'first',\n",
+    "    'Architecture_Score': 'first'\n",
+    "}).round(2)\n",
+    "\n",
+    "# Calculate overall quality score (weighted average)\n",
+    "model_summary['Quality_Score'] = (\n",
+    "    model_summary['Similarity'] * 0.35 +  # 35% answer accuracy\n",
+    "    model_summary['Citation_Score'] * 0.35 +  # 35% citation quality\n",
+    "    model_summary['Completeness_Score'] * 0.30  # 30% completeness\n",
+    ").round(2)\n",
+    "\n",
+    "# Sort by Quality Score\n",
+    "model_summary = model_summary.sort_values('Quality_Score', ascending=False)\n",
+    "\n",
+    "# Display summary table\n",
+    "print(\"\\n\" + \"=\"*100)\n",
+    "print(\"📊 LLM BENCHMARKING RESULTS - MODEL SUMMARY\")\n",
+    "print(\"=\"*100)\n",
+    "print(model_summary.to_string())\n",
+    "print(\"=\"*100)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 8. Visualizations"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": "# Create comprehensive visualization\nimport os\nfrom pathlib import Path\n\n# Create output directory\noutput_dir = Path('output/llm_benchmark')\noutput_dir.mkdir(parents=True, exist_ok=True)\n\nfig, axes = plt.subplots(2, 3, figsize=(18, 12))\n\nmodels = model_summary.index.tolist()\ncolors = sns.color_palette('husl', len(models))\n\n# 1. Overall Quality Score\nax1 = axes[0, 0]\nbars1 = ax1.barh(models, model_summary['Quality_Score'], color=colors)\nax1.set_xlabel('Quality Score (Higher is Better)', fontsize=11)\nax1.set_title('Overall Quality Score\\n(Similarity 35% + Citation 35% + Completeness 30%)', \n              fontsize=12, fontweight='bold')\nax1.set_xlim(0, 100)\nfor i, (model, score) in enumerate(zip(models, model_summary['Quality_Score'])):\n    ax1.text(score + 1, i, f'{score:.1f}', va='center', fontsize=10, fontweight='bold')\n\n# 2. Answer Similarity (Accuracy)\nax2 = axes[0, 1]\nax2.barh(models, model_summary['Similarity'], color=colors)\nax2.set_xlabel('Similarity to Expected Answer (%)', fontsize=11)\nax2.set_title('Answer Accuracy', fontsize=12, fontweight='bold')\nax2.set_xlim(0, 100)\nfor i, (model, score) in enumerate(zip(models, model_summary['Similarity'])):\n    ax2.text(score + 1, i, f'{score:.1f}%', va='center', fontsize=9)\n\n# 3. Citation Quality\nax3 = axes[0, 2]\nax3.barh(models, model_summary['Citation_Score'], color=colors)\nax3.set_xlabel('Citation Score (%)', fontsize=11)\nax3.set_title('Citation Quality\\n(PDF names + Page numbers)', fontsize=12, fontweight='bold')\nax3.set_xlim(0, 100)\nfor i, (model, score) in enumerate(zip(models, model_summary['Citation_Score'])):\n    ax3.text(score + 1, i, f'{score:.1f}%', va='center', fontsize=9)\n\n# 4. Response Time\nax4 = axes[1, 0]\nax4.barh(models, model_summary['Response_Time'], color=colors)\nax4.set_xlabel('Response Time (seconds - Lower is Better)', fontsize=11)\nax4.set_title('Speed Performance', fontsize=12, fontweight='bold')\nfor i, (model, time) in enumerate(zip(models, model_summary['Response_Time'])):\n    ax4.text(time + 0.1, i, f'{time:.2f}s', va='center', fontsize=9)\n\n# 5. Completeness\nax5 = axes[1, 1]\nax5.barh(models, model_summary['Completeness_Score'], color=colors)\nax5.set_xlabel('Completeness Score (%)', fontsize=11)\nax5.set_title('Answer Completeness', fontsize=12, fontweight='bold')\nax5.set_xlim(0, 100)\nfor i, (model, score) in enumerate(zip(models, model_summary['Completeness_Score'])):\n    ax5.text(score + 1, i, f'{score:.1f}%', va='center', fontsize=9)\n\n# 6. Error Rates (CER vs WER)\nax6 = axes[1, 2]\nx = range(len(models))\nwidth = 0.35\nax6.bar([i - width/2 for i in x], model_summary['CER'], width, label='CER', alpha=0.8)\nax6.bar([i + width/2 for i in x], model_summary['WER'], width, label='WER', alpha=0.8)\nax6.set_ylabel('Error Rate (% - Lower is Better)', fontsize=11)\nax6.set_title('Error Rates', fontsize=12, fontweight='bold')\nax6.set_xticks(x)\nax6.set_xticklabels(models, rotation=45, ha='right')\nax6.legend()\nax6.grid(axis='y', alpha=0.3)\n\nplt.tight_layout()\nplt.savefig(output_dir / 'results.png', dpi=300, bbox_inches='tight')\nplt.show()\n\nprint(f\"\\n✅ Visualization saved to '{output_dir}/results.png'\")"
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 9. Final Rankings and Recommendations"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "====================================================================================================\n",
+      "🏆 FINAL RANKINGS\n",
+      "====================================================================================================\n",
+      "                      Rank  Quality_Score  Similarity  Citation_Score  Completeness_Score  Response_Time  Open_Source Architecture_Score\n",
+      "Model                                                                                                                                   \n",
+      "Llama-4-Maverick-17B     1          59.40         0.0            84.0              100.00           3.98         True               High\n",
+      "GPT-4.1                  2          57.46         0.0            84.0               93.54           5.95        False             Medium\n",
+      "DeepSeek-R1              3          48.32         0.0            80.0               67.73          10.80         True               High\n",
+      "====================================================================================================\n",
+      "\n",
+      "====================================================================================================\n",
+      "💡 RECOMMENDATIONS FOR HACKATHON\n",
+      "====================================================================================================\n",
+      "\n",
+      "🥇 Best Overall Quality: Llama-4-Maverick-17B\n",
+      "   Quality Score: 59.4%\n",
+      "   Similarity: 0.0%\n",
+      "   Citation Score: 84.0%\n",
+      "   Response Time: 3.98s\n",
+      "   Open Source: True\n",
+      "   Architecture Score: High\n",
+      "\n",
+      "🔓 Best Open-Source Model: Llama-4-Maverick-17B\n",
+      "   Quality Score: 59.4%\n",
+      "   Architecture Score: High (Better for hackathon!)\n",
+      "   Response Time: 3.98s\n",
+      "\n",
+      "⚡ Fastest Model: Llama-4-Maverick-17B\n",
+      "   Response Time: 3.98s\n",
+      "   Quality Score: 59.4%\n",
+      "\n",
+      "====================================================================================================\n",
+      "📝 FINAL RECOMMENDATION\n",
+      "====================================================================================================\n",
+      "\n",
+      "Scoring Breakdown:\n",
+      "  - LLM Quality: 30% of total hackathon score\n",
+      "  - Architecture: 20% of total hackathon score (open-source preferred!)\n",
+      "\n",
+      "Best Choice:\n",
+      "  ✅ Llama-4-Maverick-17B - Best balance of quality and architecture score\n",
+      "     Only 0.0% quality drop for higher architecture score!\n",
+      "====================================================================================================\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Create rankings table\n",
+    "rankings = model_summary[[\n",
+    "    'Quality_Score', 'Similarity', 'Citation_Score', 'Completeness_Score', \n",
+    "    'Response_Time', 'Open_Source', 'Architecture_Score'\n",
+    "]].copy()\n",
+    "\n",
+    "rankings.insert(0, 'Rank', range(1, len(rankings) + 1))\n",
+    "\n",
+    "print(\"\\n\" + \"=\"*100)\n",
+    "print(\"🏆 FINAL RANKINGS\")\n",
+    "print(\"=\"*100)\n",
+    "print(rankings.to_string())\n",
+    "print(\"=\"*100)\n",
+    "\n",
+    "# Winner analysis\n",
+    "best_overall = rankings.index[0]\n",
+    "best_open_source = rankings[rankings['Open_Source'] == True].index[0] if any(rankings['Open_Source']) else None\n",
+    "fastest = model_summary['Response_Time'].idxmin()\n",
+    "\n",
+    "print(\"\\n\" + \"=\"*100)\n",
+    "print(\"💡 RECOMMENDATIONS FOR HACKATHON\")\n",
+    "print(\"=\"*100)\n",
+    "\n",
+    "print(f\"\\n🥇 Best Overall Quality: {best_overall}\")\n",
+    "print(f\"   Quality Score: {model_summary.loc[best_overall, 'Quality_Score']:.1f}%\")\n",
+    "print(f\"   Similarity: {model_summary.loc[best_overall, 'Similarity']:.1f}%\")\n",
+    "print(f\"   Citation Score: {model_summary.loc[best_overall, 'Citation_Score']:.1f}%\")\n",
+    "print(f\"   Response Time: {model_summary.loc[best_overall, 'Response_Time']:.2f}s\")\n",
+    "print(f\"   Open Source: {model_summary.loc[best_overall, 'Open_Source']}\")\n",
+    "print(f\"   Architecture Score: {model_summary.loc[best_overall, 'Architecture_Score']}\")\n",
+    "\n",
+    "if best_open_source:\n",
+    "    print(f\"\\n🔓 Best Open-Source Model: {best_open_source}\")\n",
+    "    print(f\"   Quality Score: {model_summary.loc[best_open_source, 'Quality_Score']:.1f}%\")\n",
+    "    print(f\"   Architecture Score: {model_summary.loc[best_open_source, 'Architecture_Score']} (Better for hackathon!)\")\n",
+    "    print(f\"   Response Time: {model_summary.loc[best_open_source, 'Response_Time']:.2f}s\")\n",
+    "\n",
+    "print(f\"\\n⚡ Fastest Model: {fastest}\")\n",
+    "print(f\"   Response Time: {model_summary.loc[fastest, 'Response_Time']:.2f}s\")\n",
+    "print(f\"   Quality Score: {model_summary.loc[fastest, 'Quality_Score']:.1f}%\")\n",
+    "\n",
+    "print(\"\\n\" + \"=\"*100)\n",
+    "print(\"📝 FINAL RECOMMENDATION\")\n",
+    "print(\"=\"*100)\n",
+    "print(\"\\nScoring Breakdown:\")\n",
+    "print(\"  - LLM Quality: 30% of total hackathon score\")\n",
+    "print(\"  - Architecture: 20% of total hackathon score (open-source preferred!)\")\n",
+    "print(\"\\nBest Choice:\")\n",
+    "if best_open_source and model_summary.loc[best_open_source, 'Quality_Score'] >= model_summary.loc[best_overall, 'Quality_Score'] * 0.9:\n",
+    "    print(f\"  ✅ {best_open_source} - Best balance of quality and architecture score\")\n",
+    "    print(f\"     Only {model_summary.loc[best_overall, 'Quality_Score'] - model_summary.loc[best_open_source, 'Quality_Score']:.1f}% quality drop for higher architecture score!\")\n",
+    "else:\n",
+    "    print(f\"  ✅ {best_overall} - Highest quality, use if quality gap is significant\")\n",
+    "    if best_open_source:\n",
+    "        print(f\"  ⚠️  Consider {best_open_source} for higher architecture score (trade-off: {model_summary.loc[best_overall, 'Quality_Score'] - model_summary.loc[best_open_source, 'Quality_Score']:.1f}% quality)\")\n",
+    "\n",
+    "print(\"=\"*100)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 10. Export Results"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": "# Save results\nfrom pathlib import Path\n\noutput_dir = Path('output/llm_benchmark')\noutput_dir.mkdir(parents=True, exist_ok=True)\n\ndf.to_csv(output_dir / 'detailed_results.csv', index=False, encoding='utf-8')\nmodel_summary.to_csv(output_dir / 'summary.csv', encoding='utf-8')\nrankings.to_csv(output_dir / 'rankings.csv', index=False, encoding='utf-8')\n\nprint(\"\\n✅ Results exported to output/llm_benchmark/:\")\nprint(\"   - detailed_results.csv (all questions and answers)\")\nprint(\"   - summary.csv (model averages)\")\nprint(\"   - rankings.csv (final rankings)\")\nprint(\"   - results.png (visualizations)\")"
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 11. Sample Answer Comparison"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "====================================================================================================\n",
+      "📝 SAMPLE ANSWER COMPARISON - Example1\n",
+      "====================================================================================================\n",
+      "\n",
+      "❓ Question: Daha az quyu ilə daha çox hasilat əldə etmək üçün hansı əsas amillərin inteqrasiyası tələb olunur?\n",
+      "\n",
+      "✅ Expected Answer:\n",
+      "Daha az quyu ilə daha çox hasilat əldə etmək üçün düzgün seçilmiş texnoloji inteqrasiya (horizontal və çoxtərəfli qazma texnikaları) və qazma məhlullarının səmərəli idarə edilməsi tələb olunur. Bu yanaşma həm iqtisadi, həm də ekoloji baxımdan üstünlük yaradır.\n",
+      "\n",
+      "----------------------------------------------------------------------------------------------------\n",
+      "\n",
+      "🤖 Llama-4-Maverick-17B (Quality: 59.4%, Time: 4.39s):\n",
+      "Daha az quyu ilə daha çox hasilat əldə etmək üçün düzgün seçilmiş texnoloji inteqrasiya və qazma məhlullarının səmərəli idarəsi əsas amillərdir. Bu, Document 1 (document_11.pdf, Səhifə 3)-də qeyd olun...\n",
+      "----------------------------------------------------------------------------------------------------\n",
+      "\n",
+      "🤖 DeepSeek-R1 (Quality: 48.3%, Time: 10.00s):\n",
+      "<think>\n",
+      "Okay, let's tackle this question. The user is asking about the main factors that need to be integrated to achieve more production with fewer wells. They provided three documents, so I need to ...\n",
+      "----------------------------------------------------------------------------------------------------\n",
+      "\n",
+      "🤖 GPT-4.1 (Quality: 57.5%, Time: 6.66s):\n",
+      "Daha az quyu ilə daha çox hasilat əldə etmək üçün bir neçə əsas amilin inteqrasiyası tələb olunur. Bu amillər aşağıdakı kimi sistematik şəkildə sənəd mənbələrinə istinadla izah olunur:\n",
+      "\n",
+      "1. **Düzgün se...\n",
+      "----------------------------------------------------------------------------------------------------\n",
+      "====================================================================================================\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Show sample answers for first question\n",
+    "sample_question = 'Example1'\n",
+    "sample_results = df[df['Question'] == sample_question]\n",
+    "\n",
+    "print(\"\\n\" + \"=\"*100)\n",
+    "print(f\"📝 SAMPLE ANSWER COMPARISON - {sample_question}\")\n",
+    "print(\"=\"*100)\n",
+    "\n",
+    "print(f\"\\n❓ Question: {questions[sample_question][0]['content']}\")\n",
+    "print(f\"\\n✅ Expected Answer:\\n{expected_answers[sample_question]['Answer']}\")\n",
+    "print(\"\\n\" + \"-\"*100)\n",
+    "\n",
+    "for _, row in sample_results.iterrows():\n",
+    "    print(f\"\\n🤖 {row['Model']} (Quality: {model_summary.loc[row['Model'], 'Quality_Score']:.1f}%, Time: {row['Response_Time']:.2f}s):\")\n",
+    "    print(f\"{row['Answer']}\")\n",
+    "    print(\"-\"*100)\n",
+    "\n",
+    "print(\"=\"*100)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "venv",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.12"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

notebooks/rag_optimization_benchmark.ipynb

@@ -0,0 +1,1367 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# RAG Pipeline Optimization Benchmark\n",
+    "\n",
+    "**Comprehensive testing of ALL RAG components to maximize LLM Judge score**\n",
+    "\n",
+    "## What We're Testing:\n",
+    "\n",
+    "### 1. Embedding Models (Vector Representations)\n",
+    "- `BAAI/bge-large-en-v1.5` (Current - 1024 dim, best quality)\n",
+    "- `BAAI/bge-base-en-v1.5` (768 dim, faster)\n",
+    "- `intfloat/multilingual-e5-large` (1024 dim, multi-language)\n",
+    "- `sentence-transformers/paraphrase-multilingual-mpnet-base-v2` (768 dim, multilingual)\n",
+    "- `sentence-transformers/all-MiniLM-L6-v2` (384 dim, very fast)\n",
+    "\n",
+    "### 2. Retrieval Strategies\n",
+    "- **Top-K**: Test 1, 3, 5, 10 documents\n",
+    "- **MMR** (Maximal Marginal Relevance): Diversity vs relevance trade-off\n",
+    "- **Similarity Threshold**: Filter low-relevance docs\n",
+    "- **Reranking**: Use cross-encoder to rerank results\n",
+    "\n",
+    "### 3. Chunking Strategies (Already in Vector DB, but we'll compare)\n",
+    "- Chunk size: 256, 512, 600 (current), 1000 tokens\n",
+    "- Overlap: 0, 50, 100 (current), 200 chars\n",
+    "\n",
+    "### 4. LLM Models\n",
+    "- Llama-4-Maverick-17B (open-source)\n",
+    "- DeepSeek-R1 (reasoning)\n",
+    "- GPT-4.1, GPT-5, GPT-5-mini\n",
+    "- Claude-Sonnet-4.5\n",
+    "\n",
+    "### 5. Prompting Techniques\n",
+    "- **Baseline**: Simple context + question\n",
+    "- **Citation-focused**: Emphasize source references\n",
+    "- **Step-by-step**: Chain-of-thought reasoning\n",
+    "- **Few-shot**: Include example Q&A\n",
+    "\n",
+    "## LLM Judge Evaluation Criteria:\n",
+    "- **Accuracy** (35%): Answer correctness\n",
+    "- **Relevance** (35%): Citation quality and relevance\n",
+    "- **Completeness** (30%): Thorough answers"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# !pip install openai pinecone-client sentence-transformers rank-bm25 python-dotenv pandas matplotlib seaborn jiwer"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/Users/ismatsamadov/SOCAR_Hackathon/venv/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
+      "  from .autonotebook import tqdm as notebook_tqdm\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "✅ Libraries loaded\n"
+     ]
+    }
+   ],
+   "source": [
+    "import os\n",
+    "import json\n",
+    "import time\n",
+    "import re\n",
+    "from typing import Dict, List, Tuple, Any\n",
+    "from collections import defaultdict\n",
+    "from dotenv import load_dotenv\n",
+    "\n",
+    "import pandas as pd\n",
+    "import matplotlib.pyplot as plt\n",
+    "import seaborn as sns\n",
+    "from openai import AzureOpenAI\n",
+    "from pinecone import Pinecone\n",
+    "from sentence_transformers import SentenceTransformer, CrossEncoder\n",
+    "from jiwer import wer, cer\n",
+    "import numpy as np\n",
+    "\n",
+    "load_dotenv()\n",
+    "\n",
+    "sns.set_style('whitegrid')\n",
+    "plt.rcParams['figure.figsize'] = (16, 10)\n",
+    "\n",
+    "print(\"✅ Libraries loaded\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 1. Load Test Questions and Expected Answers"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "✅ Loaded 5 test questions\n",
+      "  - Example1\n",
+      "  - Example2\n",
+      "  - Example3\n",
+      "  - Example4\n",
+      "  - Example5\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Load test cases\n",
+    "with open('docs/sample_questions.json', 'r', encoding='utf-8') as f:\n",
+    "    questions = json.load(f)\n",
+    "\n",
+    "with open('docs/sample_answers.json', 'r', encoding='utf-8') as f:\n",
+    "    expected_answers = json.load(f)\n",
+    "\n",
+    "print(f\"✅ Loaded {len(questions)} test questions\")\n",
+    "for key in questions.keys():\n",
+    "    print(f\"  - {key}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 2. Initialize Vector Database"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "✅ Vector DB connected\n",
+      "   Total vectors: 1300\n",
+      "   Dimensions: 1024\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Connect to Pinecone\n",
+    "pc = Pinecone(api_key=os.getenv('PINECONE_API_KEY'))\n",
+    "index = pc.Index(os.getenv('PINECONE_INDEX_NAME', 'hackathon'))\n",
+    "\n",
+    "stats = index.describe_index_stats()\n",
+    "print(f\"✅ Vector DB connected\")\n",
+    "print(f\"   Total vectors: {stats['total_vector_count']}\")\n",
+    "print(f\"   Dimensions: {stats['dimension']}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 3. Embedding Models Configuration"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Loading bge-large-en...\n",
+      "  ✅ BAAI/bge-large-en-v1.5\n",
+      "Loading multilingual-e5-large...\n",
+      "  ✅ intfloat/multilingual-e5-large\n",
+      "\n",
+      "✅ Loaded 2 embedding models\n"
+     ]
+    }
+   ],
+   "source": [
+    "EMBEDDING_MODELS = {\n",
+    "    'bge-large-en': {\n",
+    "        'name': 'BAAI/bge-large-en-v1.5',\n",
+    "        'dimensions': 1024,\n",
+    "        'notes': 'Current model - best quality'\n",
+    "    },\n",
+    "    'bge-base-en': {\n",
+    "        'name': 'BAAI/bge-base-en-v1.5',\n",
+    "        'dimensions': 768,\n",
+    "        'notes': 'Faster, slightly lower quality'\n",
+    "    },\n",
+    "    'multilingual-e5-large': {\n",
+    "        'name': 'intfloat/multilingual-e5-large',\n",
+    "        'dimensions': 1024,\n",
+    "        'notes': 'Multi-language optimized'\n",
+    "    },\n",
+    "    'paraphrase-multilingual': {\n",
+    "        'name': 'sentence-transformers/paraphrase-multilingual-mpnet-base-v2',\n",
+    "        'dimensions': 768,\n",
+    "        'notes': 'Good for Azerbaijani/Russian'\n",
+    "    },\n",
+    "    'all-MiniLM-L6': {\n",
+    "        'name': 'sentence-transformers/all-MiniLM-L6-v2',\n",
+    "        'dimensions': 384,\n",
+    "        'notes': 'Very fast, lower quality'\n",
+    "    }\n",
+    "}\n",
+    "\n",
+    "# Load embedding models (only test 1024-dim models for existing Pinecone index)\n",
+    "EMBEDDING_MODELS_TO_TEST = [\n",
+    "    'bge-large-en',  # Current\n",
+    "    'multilingual-e5-large',  # Alternative with same dims\n",
+    "]\n",
+    "\n",
+    "embedding_cache = {}\n",
+    "\n",
+    "for model_key in EMBEDDING_MODELS_TO_TEST:\n",
+    "    model_name = EMBEDDING_MODELS[model_key]['name']\n",
+    "    print(f\"Loading {model_key}...\")\n",
+    "    embedding_cache[model_key] = SentenceTransformer(model_name)\n",
+    "    print(f\"  ✅ {model_name}\")\n",
+    "\n",
+    "print(f\"\\n✅ Loaded {len(embedding_cache)} embedding models\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 4. Retrieval Strategies"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "✅ Configured 7 retrieval strategies\n"
+     ]
+    }
+   ],
+   "source": [
+    "def retrieve_vanilla(query: str, embed_model: SentenceTransformer, top_k: int = 3) -> List[Dict]:\n",
+    "    \"\"\"\n",
+    "    Vanilla retrieval: Simple top-k vector search.\n",
+    "    \"\"\"\n",
+    "    query_embedding = embed_model.encode(query).tolist()\n",
+    "    results = index.query(vector=query_embedding, top_k=top_k, include_metadata=True)\n",
+    "    \n",
+    "    documents = []\n",
+    "    for match in results['matches']:\n",
+    "        documents.append({\n",
+    "            'pdf_name': match['metadata'].get('pdf_name', 'unknown.pdf'),\n",
+    "            'page_number': match['metadata'].get('page_number', 0),\n",
+    "            'content': match['metadata'].get('text', ''),\n",
+    "            'score': match.get('score', 0.0)\n",
+    "        })\n",
+    "    \n",
+    "    return documents\n",
+    "\n",
+    "\n",
+    "def retrieve_with_threshold(query: str, embed_model: SentenceTransformer, \n",
+    "                           top_k: int = 10, threshold: float = 0.7) -> List[Dict]:\n",
+    "    \"\"\"\n",
+    "    Retrieve with similarity threshold filtering.\n",
+    "    \"\"\"\n",
+    "    docs = retrieve_vanilla(query, embed_model, top_k=top_k)\n",
+    "    return [doc for doc in docs if doc['score'] >= threshold]\n",
+    "\n",
+    "\n",
+    "def retrieve_with_mmr(query: str, embed_model: SentenceTransformer, \n",
+    "                     top_k: int = 3, lambda_param: float = 0.5, fetch_k: int = 20) -> List[Dict]:\n",
+    "    \"\"\"\n",
+    "    MMR (Maximal Marginal Relevance) for diversity.\n",
+    "    lambda=1 → pure relevance, lambda=0 → pure diversity\n",
+    "    \"\"\"\n",
+    "    # Fetch more candidates\n",
+    "    candidates = retrieve_vanilla(query, embed_model, top_k=fetch_k)\n",
+    "    \n",
+    "    if len(candidates) <= top_k:\n",
+    "        return candidates[:top_k]\n",
+    "    \n",
+    "    # Query embedding\n",
+    "    query_emb = embed_model.encode(query)\n",
+    "    \n",
+    "    # Get embeddings for candidates\n",
+    "    candidate_texts = [doc['content'] for doc in candidates]\n",
+    "    candidate_embs = embed_model.encode(candidate_texts)\n",
+    "    \n",
+    "    # MMR algorithm\n",
+    "    selected = []\n",
+    "    selected_embs = []\n",
+    "    \n",
+    "    for _ in range(min(top_k, len(candidates))):\n",
+    "        mmr_scores = []\n",
+    "        \n",
+    "        for i, (doc, emb) in enumerate(zip(candidates, candidate_embs)):\n",
+    "            if i in [candidates.index(s) for s in selected]:\n",
+    "                mmr_scores.append(-float('inf'))\n",
+    "                continue\n",
+    "            \n",
+    "            # Relevance to query\n",
+    "            relevance = np.dot(query_emb, emb) / (np.linalg.norm(query_emb) * np.linalg.norm(emb))\n",
+    "            \n",
+    "            # Max similarity to already selected\n",
+    "            if selected_embs:\n",
+    "                similarities = [np.dot(emb, s_emb) / (np.linalg.norm(emb) * np.linalg.norm(s_emb)) \n",
+    "                              for s_emb in selected_embs]\n",
+    "                max_sim = max(similarities)\n",
+    "            else:\n",
+    "                max_sim = 0\n",
+    "            \n",
+    "            # MMR score\n",
+    "            mmr = lambda_param * relevance - (1 - lambda_param) * max_sim\n",
+    "            mmr_scores.append(mmr)\n",
+    "        \n",
+    "        # Select best MMR score\n",
+    "        best_idx = np.argmax(mmr_scores)\n",
+    "        selected.append(candidates[best_idx])\n",
+    "        selected_embs.append(candidate_embs[best_idx])\n",
+    "    \n",
+    "    return selected\n",
+    "\n",
+    "\n",
+    "def retrieve_with_reranking(query: str, embed_model: SentenceTransformer, \n",
+    "                           top_k: int = 3, fetch_k: int = 20) -> List[Dict]:\n",
+    "    \"\"\"\n",
+    "    Two-stage: retrieve with embeddings, rerank with cross-encoder.\n",
+    "    \"\"\"\n",
+    "    # Stage 1: Retrieve candidates\n",
+    "    candidates = retrieve_vanilla(query, embed_model, top_k=fetch_k)\n",
+    "    \n",
+    "    if len(candidates) <= top_k:\n",
+    "        return candidates[:top_k]\n",
+    "    \n",
+    "    # Stage 2: Rerank with cross-encoder\n",
+    "    reranker = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2')\n",
+    "    \n",
+    "    pairs = [[query, doc['content']] for doc in candidates]\n",
+    "    scores = reranker.predict(pairs)\n",
+    "    \n",
+    "    # Sort by reranker score\n",
+    "    scored_docs = [(doc, score) for doc, score in zip(candidates, scores)]\n",
+    "    scored_docs.sort(key=lambda x: x[1], reverse=True)\n",
+    "    \n",
+    "    # Update scores and return top-k\n",
+    "    reranked = []\n",
+    "    for doc, score in scored_docs[:top_k]:\n",
+    "        doc['rerank_score'] = float(score)\n",
+    "        reranked.append(doc)\n",
+    "    \n",
+    "    return reranked\n",
+    "\n",
+    "\n",
+    "RETRIEVAL_STRATEGIES = {\n",
+    "    'vanilla_k3': {'func': retrieve_vanilla, 'params': {'top_k': 3}, 'notes': 'Current setup'},\n",
+    "    'vanilla_k5': {'func': retrieve_vanilla, 'params': {'top_k': 5}, 'notes': 'More context'},\n",
+    "    'vanilla_k10': {'func': retrieve_vanilla, 'params': {'top_k': 10}, 'notes': 'Maximum context'},\n",
+    "    'threshold_0.7': {'func': retrieve_with_threshold, 'params': {'top_k': 10, 'threshold': 0.7}, 'notes': 'Quality filter'},\n",
+    "    'mmr_balanced': {'func': retrieve_with_mmr, 'params': {'top_k': 3, 'lambda_param': 0.5}, 'notes': 'Balance diversity'},\n",
+    "    'mmr_diverse': {'func': retrieve_with_mmr, 'params': {'top_k': 3, 'lambda_param': 0.3}, 'notes': 'More diversity'},\n",
+    "    'reranked_k3': {'func': retrieve_with_reranking, 'params': {'top_k': 3, 'fetch_k': 20}, 'notes': 'Two-stage rerank'},\n",
+    "}\n",
+    "\n",
+    "print(f\"✅ Configured {len(RETRIEVAL_STRATEGIES)} retrieval strategies\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 5. LLM Models and Prompting Strategies"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "✅ Configured 5 LLM models\n",
+      "✅ Configured 4 prompting strategies\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Initialize Azure OpenAI\n",
+    "azure_client = AzureOpenAI(\n",
+    "    api_key=os.getenv('AZURE_OPENAI_API_KEY'),\n",
+    "    api_version=os.getenv('AZURE_OPENAI_API_VERSION', '2024-08-01-preview'),\n",
+    "    azure_endpoint=os.getenv('AZURE_OPENAI_ENDPOINT')\n",
+    ")\n",
+    "\n",
+    "LLM_MODELS = {\n",
+    "    'Llama-4-Maverick': 'Llama-4-Maverick-17B-128E-Instruct-FP8',\n",
+    "    'DeepSeek-R1': 'DeepSeek-R1',\n",
+    "    'GPT-4.1': 'gpt-4.1',\n",
+    "    'GPT-5-mini': 'gpt-5-mini',\n",
+    "    'Claude-Sonnet-4.5': 'claude-sonnet-4-5',\n",
+    "}\n",
+    "\n",
+    "# Prompting strategies\n",
+    "PROMPTING_STRATEGIES = {\n",
+    "    'baseline': \"\"\"\n",
+    "Siz SOCAR-ın tarixi neft və qaz sənədləri üzr�� köməkçisiniz.\n",
+    "\n",
+    "Kontekst:\n",
+    "{context}\n",
+    "\n",
+    "Sual: {query}\n",
+    "\n",
+    "Kontekstə əsaslanaraq cavab verin.\n",
+    "\"\"\",\n",
+    "    \n",
+    "    'citation_focused': \"\"\"\n",
+    "Siz SOCAR-ın tarixi sənədlər üzrə mütəxəssis köməkçisisiniz.\n",
+    "\n",
+    "ÖNƏMLİ: Hər bir faktı mütləq mənbə ilə təsdiqləyin (PDF adı və səhifə nömrəsi).\n",
+    "\n",
+    "Kontekst:\n",
+    "{context}\n",
+    "\n",
+    "Sual: {query}\n",
+    "\n",
+    "Cavab verərkən:\n",
+    "1. Dəqiq faktlar yazın\n",
+    "2. Hər faktı mənbə ilə göstərin: (PDF: fayl_adı.pdf, Səhifə: X)\n",
+    "3. Kontekstdə olmayan məlumat əlavə etməyin\n",
+    "\"\"\",\n",
+    "    \n",
+    "    'step_by_step': \"\"\"\n",
+    "Siz SOCAR-ın tarixi sənədlər üzrə analitik köməkçisisiniz.\n",
+    "\n",
+    "Kontekst:\n",
+    "{context}\n",
+    "\n",
+    "Sual: {query}\n",
+    "\n",
+    "Addım-addım cavab verin:\n",
+    "1. Əvvəlcə kontekstdən əlaqəli məlumatları müəyyənləşdirin\n",
+    "2. Bu məlumatları təhlil edin\n",
+    "3. Nəticəni mənbələr ilə birlikdə təqdim edin\n",
+    "\"\"\",\n",
+    "    \n",
+    "    'few_shot': \"\"\"\n",
+    "Siz SOCAR-ın tarixi sənədlər üzrə mütəxəssis köməkçisisiniz.\n",
+    "\n",
+    "Nümunə:\n",
+    "Sual: \"Palçıq vulkanlarının təsir radiusu nə qədərdir?\"\n",
+    "Cavab: \"Sahə müşahidələri və modelləşdirmə göstərir ki, palçıq vulkanlarının təsir radiusu təqribən 10 km-dir (PDF: document_06.pdf, Səhifə: 5).\"\n",
+    "\n",
+    "Kontekst:\n",
+    "{context}\n",
+    "\n",
+    "Sual: {query}\n",
+    "\n",
+    "Yuxarıdakı nümunə kimi cavab verin - dəqiq, qısa, mənbə ilə.\n",
+    "\"\"\"\n",
+    "}\n",
+    "\n",
+    "print(f\"✅ Configured {len(LLM_MODELS)} LLM models\")\n",
+    "print(f\"✅ Configured {len(PROMPTING_STRATEGIES)} prompting strategies\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": "def generate_answer(llm_model: str, query: str, documents: List[Dict], \n                   prompt_strategy: str = 'baseline',\n                   temperature: float = 0.2) -> Tuple[str, float]:\n    \"\"\"\n    Generate answer using LLM with specified prompting strategy.\n    \"\"\"\n    # Build context\n    context_parts = []\n    for i, doc in enumerate(documents, 1):\n        context_parts.append(\n            f\"Sənəd {i} (Mənbə: {doc['pdf_name']}, Səhifə {doc['page_number']}):\\n{doc['content']}\"\n        )\n    context = \"\\n\\n\".join(context_parts)\n    \n    # Get prompt template\n    prompt_template = PROMPTING_STRATEGIES[prompt_strategy]\n    prompt = prompt_template.format(context=context, query=query)\n    \n    try:\n        start_time = time.time()\n        \n        deployment = LLM_MODELS[llm_model]\n        \n        # GPT-5 models use max_completion_tokens, others use max_tokens\n        if deployment.startswith('gpt-5'):\n            response = azure_client.chat.completions.create(\n                model=deployment,\n                messages=[{\"role\": \"user\", \"content\": prompt}],\n                temperature=temperature,\n                max_completion_tokens=1000\n            )\n        else:\n            response = azure_client.chat.completions.create(\n                model=deployment,\n                messages=[{\"role\": \"user\", \"content\": prompt}],\n                temperature=temperature,\n                max_tokens=1000\n            )\n        \n        elapsed = time.time() - start_time\n        answer = response.choices[0].message.content\n        \n        return answer, elapsed\n    \n    except Exception as e:\n        return f\"ERROR: {str(e)}\", 0.0\n\nprint(\"✅ LLM generation function ready\")"
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 6. Evaluation Metrics"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "✅ Evaluation metrics ready\n"
+     ]
+    }
+   ],
+   "source": [
+    "def normalize_text(text: str) -> str:\n",
+    "    text = text.lower().strip()\n",
+    "    text = re.sub(r'\\s+', ' ', text)\n",
+    "    return text\n",
+    "\n",
+    "def calculate_answer_quality(reference: str, hypothesis: str) -> Dict[str, float]:\n",
+    "    \"\"\"Accuracy metrics.\"\"\"\n",
+    "    ref_norm = normalize_text(reference)\n",
+    "    hyp_norm = normalize_text(hypothesis)\n",
+    "    \n",
+    "    cer_score = cer(ref_norm, hyp_norm) * 100\n",
+    "    wer_score = wer(ref_norm, hyp_norm) * 100\n",
+    "    similarity = max(0, 100 - wer_score)\n",
+    "    \n",
+    "    return {\n",
+    "        'Accuracy_Score': round(similarity, 2)\n",
+    "    }\n",
+    "\n",
+    "def evaluate_citation_quality(answer: str, documents: List[Dict]) -> Dict[str, float]:\n",
+    "    \"\"\"Relevance - citation quality.\"\"\"\n",
+    "    pdf_names = [doc['pdf_name'].replace('.pdf', '') for doc in documents]\n",
+    "    page_numbers = [str(doc['page_number']) for doc in documents]\n",
+    "    \n",
+    "    cited_pdfs = sum(1 for pdf in pdf_names if pdf in answer)\n",
+    "    cited_pages = sum(1 for page in page_numbers if page in answer)\n",
+    "    \n",
+    "    citation_keywords = ['mənbə', 'sənəd', 'səhifə', 'pdf', 'document', 'page']\n",
+    "    has_citation_format = any(kw in answer.lower() for kw in citation_keywords)\n",
+    "    \n",
+    "    citation_score = (\n",
+    "        (cited_pdfs / len(pdf_names) * 40) +\n",
+    "        (cited_pages / len(page_numbers) * 40) +\n",
+    "        (20 if has_citation_format else 0)\n",
+    "    )\n",
+    "    \n",
+    "    return {\n",
+    "        'Citation_Score': round(citation_score, 2),\n",
+    "        'Cited_PDFs': cited_pdfs,\n",
+    "        'Cited_Pages': cited_pages\n",
+    "    }\n",
+    "\n",
+    "def evaluate_retrieval_quality(query: str, documents: List[Dict], expected_answer: str) -> Dict[str, float]:\n",
+    "    \"\"\"Measure if retrieved docs are relevant to answer.\"\"\"\n",
+    "    if not documents or not expected_answer:\n",
+    "        return {'Retrieval_Relevance': 0.0}\n",
+    "    \n",
+    "    # Simple heuristic: check if expected answer words appear in retrieved docs\n",
+    "    expected_words = set(normalize_text(expected_answer).split())\n",
+    "    retrieved_text = ' '.join([doc['content'] for doc in documents])\n",
+    "    retrieved_words = set(normalize_text(retrieved_text).split())\n",
+    "    \n",
+    "    overlap = len(expected_words & retrieved_words) / len(expected_words) if expected_words else 0\n",
+    "    \n",
+    "    return {\n",
+    "        'Retrieval_Relevance': round(overlap * 100, 2)\n",
+    "    }\n",
+    "\n",
+    "def evaluate_completeness(answer: str) -> Dict[str, float]:\n",
+    "    \"\"\"Completeness metrics.\"\"\"\n",
+    "    word_count = len(answer.split())\n",
+    "    \n",
+    "    if word_count < 20:\n",
+    "        completeness = (word_count / 20) * 100\n",
+    "    elif word_count > 200:\n",
+    "        completeness = 100 - ((word_count - 200) / 200 * 20)\n",
+    "    else:\n",
+    "        completeness = 100\n",
+    "    \n",
+    "    return {\n",
+    "        'Completeness_Score': round(max(0, completeness), 2),\n",
+    "        'Word_Count': word_count\n",
+    "    }\n",
+    "\n",
+    "def calculate_llm_judge_score(accuracy: float, citation: float, completeness: float) -> float:\n",
+    "    \"\"\"Overall LLM Judge score (weighted).\"\"\"\n",
+    "    return round(\n",
+    "        accuracy * 0.35 +\n",
+    "        citation * 0.35 +\n",
+    "        completeness * 0.30,\n",
+    "        2\n",
+    "    )\n",
+    "\n",
+    "print(\"✅ Evaluation metrics ready\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 7. Run Comprehensive Benchmark"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Testing 11 configurations on 5 questions\n",
+      "Total API calls: ~55\n",
+      "This will take 15-30 minutes...\n",
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Configuration: Select what to test\n",
+    "CONFIGS_TO_TEST = [\n",
+    "    # Format: (embed_model, retrieval_strategy, llm_model, prompt_strategy)\n",
+    "    \n",
+    "    # Baseline (current setup)\n",
+    "    ('bge-large-en', 'vanilla_k3', 'Llama-4-Maverick', 'baseline'),\n",
+    "    \n",
+    "    # Test different embedding models\n",
+    "    ('multilingual-e5-large', 'vanilla_k3', 'Llama-4-Maverick', 'baseline'),\n",
+    "    \n",
+    "    # Test different retrieval strategies\n",
+    "    ('bge-large-en', 'vanilla_k5', 'Llama-4-Maverick', 'baseline'),\n",
+    "    ('bge-large-en', 'mmr_balanced', 'Llama-4-Maverick', 'baseline'),\n",
+    "    ('bge-large-en', 'reranked_k3', 'Llama-4-Maverick', 'baseline'),\n",
+    "    \n",
+    "    # Test different LLM models\n",
+    "    ('bge-large-en', 'vanilla_k3', 'GPT-5-mini', 'baseline'),\n",
+    "    ('bge-large-en', 'vanilla_k3', 'Claude-Sonnet-4.5', 'baseline'),\n",
+    "    \n",
+    "    # Test different prompting strategies\n",
+    "    ('bge-large-en', 'vanilla_k3', 'Llama-4-Maverick', 'citation_focused'),\n",
+    "    ('bge-large-en', 'vanilla_k3', 'Llama-4-Maverick', 'few_shot'),\n",
+    "    \n",
+    "    # Best combinations\n",
+    "    ('bge-large-en', 'reranked_k3', 'GPT-5-mini', 'citation_focused'),\n",
+    "    ('bge-large-en', 'mmr_balanced', 'Claude-Sonnet-4.5', 'citation_focused'),\n",
+    "]\n",
+    "\n",
+    "print(f\"Testing {len(CONFIGS_TO_TEST)} configurations on {len(questions)} questions\")\n",
+    "print(f\"Total API calls: ~{len(CONFIGS_TO_TEST) * len(questions)}\")\n",
+    "print(\"This will take 15-30 minutes...\\n\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 15,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "====================================================================================================\n",
+      "Config 1/11: bge-large-en_vanilla_k3_Llama-4-Maverick_baseline\n",
+      "====================================================================================================\n",
+      "\n",
+      "  Example1: Daha az quyu ilə daha çox hasilat əldə etmək üçün hansı əsas...\n",
+      "    Retrieved 3 docs\n",
+      "    ✅ Generated in 3.01s\n",
+      "\n",
+      "  Example2: Qərbi Abşeron yatağında suvurma tədbirləri hansı tarixdə və ...\n",
+      "    Retrieved 3 docs\n",
+      "    ✅ Generated in 2.38s\n",
+      "\n",
+      "  Example3: Pirallahı strukturunda 1253 nömrəli quyudan götürülmüş nümun...\n",
+      "    Retrieved 3 docs\n",
+      "    ✅ Generated in 2.45s\n",
+      "\n",
+      "  Example4: Bakı arxipelaqı (BA) və Aşağı Kür çökəkliyi (AKÇ) üçün geote...\n",
+      "    Retrieved 3 docs\n",
+      "    ✅ Generated in 3.52s\n",
+      "\n",
+      "  Example5: Bu zonada hansı proseslər baş verir?...\n",
+      "    Retrieved 3 docs\n",
+      "    ✅ Generated in 1.14s\n",
+      "\n",
+      "  📊 Config Summary:\n",
+      "     Avg LLM Judge Score: 43.53%\n",
+      "     Avg Response Time: 2.50s\n",
+      "\n",
+      "====================================================================================================\n",
+      "Config 2/11: multilingual-e5-large_vanilla_k3_Llama-4-Maverick_baseline\n",
+      "====================================================================================================\n",
+      "\n",
+      "  Example1: Daha az quyu ilə daha çox hasilat əldə etmək üçün hansı əsas...\n",
+      "    Retrieved 3 docs\n",
+      "    ✅ Generated in 3.98s\n",
+      "\n",
+      "  Example2: Qərbi Abşeron yatağında suvurma tədbirləri hansı tarixdə və ...\n",
+      "    Retrieved 3 docs\n",
+      "    ✅ Generated in 1.66s\n",
+      "\n",
+      "  Example3: Pirallahı strukturunda 1253 nömrəli quyudan götürülmüş nümun...\n",
+      "    Retrieved 3 docs\n",
+      "    ✅ Generated in 2.19s\n",
+      "\n",
+      "  Example4: Bakı arxipelaqı (BA) və Aşağı Kür çökəkliyi (AKÇ) üçün geote...\n",
+      "    Retrieved 3 docs\n",
+      "    ✅ Generated in 4.38s\n",
+      "\n",
+      "  Example5: Bu zonada hansı proseslər baş verir?...\n",
+      "    Retrieved 3 docs\n",
+      "    ✅ Generated in 4.32s\n",
+      "\n",
+      "  📊 Config Summary:\n",
+      "     Avg LLM Judge Score: 39.73%\n",
+      "     Avg Response Time: 3.31s\n",
+      "\n",
+      "====================================================================================================\n",
+      "Config 3/11: bge-large-en_vanilla_k5_Llama-4-Maverick_baseline\n",
+      "====================================================================================================\n",
+      "\n",
+      "  Example1: Daha az quyu ilə daha çox hasilat əldə etmək üçün hansı əsas...\n",
+      "    Retrieved 5 docs\n",
+      "    ✅ Generated in 2.55s\n",
+      "\n",
+      "  Example2: Qərbi Abşeron yatağında suvurma tədbirləri hansı tarixdə və ...\n",
+      "    Retrieved 5 docs\n",
+      "    ✅ Generated in 2.50s\n",
+      "\n",
+      "  Example3: Pirallahı strukturunda 1253 nömrəli quyudan götürülmüş nümun...\n",
+      "    Retrieved 5 docs\n",
+      "    ✅ Generated in 2.58s\n",
+      "\n",
+      "  Example4: Bakı arxipelaqı (BA) və Aşağı Kür çökəkliyi (AKÇ) üçün geote...\n",
+      "    Retrieved 5 docs\n",
+      "    ✅ Generated in 3.07s\n",
+      "\n",
+      "  Example5: Bu zonada hansı proseslər baş verir?...\n",
+      "    Retrieved 5 docs\n",
+      "    ✅ Generated in 3.74s\n",
+      "\n",
+      "  📊 Config Summary:\n",
+      "     Avg LLM Judge Score: 45.40%\n",
+      "     Avg Response Time: 2.89s\n",
+      "\n",
+      "====================================================================================================\n",
+      "Config 4/11: bge-large-en_mmr_balanced_Llama-4-Maverick_baseline\n",
+      "====================================================================================================\n",
+      "\n",
+      "  Example1: Daha az quyu ilə daha çox hasilat əldə etmək üçün hansı əsas...\n",
+      "    Retrieved 3 docs\n",
+      "    ✅ Generated in 1.64s\n",
+      "\n",
+      "  Example2: Qərbi Abşeron yatağında suvurma tədbirləri hansı tarixdə və ...\n",
+      "    Retrieved 3 docs\n",
+      "    ✅ Generated in 1.27s\n",
+      "\n",
+      "  Example3: Pirallahı strukturunda 1253 nömrəli quyudan götürülmüş nümun...\n",
+      "    Retrieved 3 docs\n",
+      "    ✅ Generated in 2.34s\n",
+      "\n",
+      "  Example4: Bakı arxipelaqı (BA) və Aşağı Kür çökəkliyi (AKÇ) üçün geote...\n",
+      "    Retrieved 3 docs\n",
+      "    ✅ Generated in 3.05s\n",
+      "\n",
+      "  Example5: Bu zonada hansı proseslər baş verir?...\n",
+      "    Retrieved 3 docs\n",
+      "    ✅ Generated in 2.52s\n",
+      "\n",
+      "  📊 Config Summary:\n",
+      "     Avg LLM Judge Score: 45.40%\n",
+      "     Avg Response Time: 2.16s\n",
+      "\n",
+      "====================================================================================================\n",
+      "Config 5/11: bge-large-en_reranked_k3_Llama-4-Maverick_baseline\n",
+      "====================================================================================================\n",
+      "\n",
+      "  Example1: Daha az quyu ilə daha çox hasilat əldə etmək üçün hansı əsas...\n",
+      "    Retrieved 3 docs\n",
+      "    ✅ Generated in 2.26s\n",
+      "\n",
+      "  Example2: Qərbi Abşeron yatağında suvurma tədbirləri hansı tarixdə və ...\n",
+      "    Retrieved 3 docs\n",
+      "    ✅ Generated in 3.12s\n",
+      "\n",
+      "  Example3: Pirallahı strukturunda 1253 nömrəli quyudan götürülmüş nümun...\n",
+      "    Retrieved 3 docs\n",
+      "    ✅ Generated in 2.83s\n",
+      "\n",
+      "  Example4: Bakı arxipelaqı (BA) və Aşağı Kür çökəkliyi (AKÇ) üçün geote...\n",
+      "    Retrieved 3 docs\n",
+      "    ✅ Generated in 3.93s\n",
+      "\n",
+      "  Example5: Bu zonada hansı proseslər baş verir?...\n",
+      "    Retrieved 3 docs\n",
+      "    ✅ Generated in 3.24s\n",
+      "\n",
+      "  📊 Config Summary:\n",
+      "     Avg LLM Judge Score: 44.47%\n",
+      "     Avg Response Time: 3.08s\n",
+      "\n",
+      "====================================================================================================\n",
+      "Config 6/11: bge-large-en_vanilla_k3_GPT-5-mini_baseline\n",
+      "====================================================================================================\n",
+      "\n",
+      "  Example1: Daha az quyu ilə daha çox hasilat əldə etmək üçün hansı əsas...\n",
+      "    Retrieved 3 docs\n",
+      "    ❌ ERROR: Error code: 400 - {'error': {'message': \"Unsupported parameter: 'max_tokens' is not supported with this model. Use 'max_completion_tokens' instead.\", 'type': 'invalid_request_error', 'param': 'max_tokens', 'code': 'unsupported_parameter'}}\n",
+      "\n",
+      "  Example2: Qərbi Abşeron yatağında suvurma tədbirləri hansı tarixdə və ...\n",
+      "    Retrieved 3 docs\n",
+      "    ❌ ERROR: Error code: 400 - {'error': {'message': \"Unsupported parameter: 'max_tokens' is not supported with this model. Use 'max_completion_tokens' instead.\", 'type': 'invalid_request_error', 'param': 'max_tokens', 'code': 'unsupported_parameter'}}\n",
+      "\n",
+      "  Example3: Pirallahı strukturunda 1253 nömrəli quyudan götürülmüş nümun...\n",
+      "    Retrieved 3 docs\n",
+      "    ❌ ERROR: Error code: 400 - {'error': {'message': \"Unsupported parameter: 'max_tokens' is not supported with this model. Use 'max_completion_tokens' instead.\", 'type': 'invalid_request_error', 'param': 'max_tokens', 'code': 'unsupported_parameter'}}\n",
+      "\n",
+      "  Example4: Bakı arxipelaqı (BA) və Aşağı Kür çökəkliyi (AKÇ) üçün geote...\n",
+      "    Retrieved 3 docs\n",
+      "    ❌ ERROR: Error code: 400 - {'error': {'message': \"Unsupported parameter: 'max_tokens' is not supported with this model. Use 'max_completion_tokens' instead.\", 'type': 'invalid_request_error', 'param': 'max_tokens', 'code': 'unsupported_parameter'}}\n",
+      "\n",
+      "  Example5: Bu zonada hansı proseslər baş verir?...\n",
+      "    Retrieved 3 docs\n",
+      "    ❌ ERROR: Error code: 400 - {'error': {'message': \"Unsupported parameter: 'max_tokens' is not supported with this model. Use 'max_completion_tokens' instead.\", 'type': 'invalid_request_error', 'param': 'max_tokens', 'code': 'unsupported_parameter'}}\n",
+      "\n",
+      "====================================================================================================\n",
+      "Config 7/11: bge-large-en_vanilla_k3_Claude-Sonnet-4.5_baseline\n",
+      "====================================================================================================\n",
+      "\n",
+      "  Example1: Daha az quyu ilə daha çox hasilat əldə etmək üçün hansı əsas...\n",
+      "    Retrieved 3 docs\n",
+      "    ❌ ERROR: Error code: 400 - {'error': {'code': 'unknown_model', 'message': 'Unknown model: claude-sonnet-4-5', 'details': 'Unknown model: claude-sonnet-4-5'}}\n",
+      "\n",
+      "  Example2: Qərbi Abşeron yatağında suvurma tədbirləri hansı tarixdə və ...\n",
+      "    Retrieved 3 docs\n",
+      "    ❌ ERROR: Error code: 400 - {'error': {'code': 'unknown_model', 'message': 'Unknown model: claude-sonnet-4-5', 'details': 'Unknown model: claude-sonnet-4-5'}}\n",
+      "\n",
+      "  Example3: Pirallahı strukturunda 1253 nömrəli quyudan götürülmüş nümun...\n",
+      "    Retrieved 3 docs\n",
+      "    ❌ ERROR: Error code: 400 - {'error': {'code': 'unknown_model', 'message': 'Unknown model: claude-sonnet-4-5', 'details': 'Unknown model: claude-sonnet-4-5'}}\n",
+      "\n",
+      "  Example4: Bakı arxipelaqı (BA) və Aşağı Kür çökəkliyi (AKÇ) üçün geote...\n",
+      "    Retrieved 3 docs\n",
+      "    ❌ ERROR: Error code: 400 - {'error': {'code': 'unknown_model', 'message': 'Unknown model: claude-sonnet-4-5', 'details': 'Unknown model: claude-sonnet-4-5'}}\n",
+      "\n",
+      "  Example5: Bu zonada hansı proseslər baş verir?...\n",
+      "    Retrieved 3 docs\n",
+      "    ❌ ERROR: Error code: 400 - {'error': {'code': 'unknown_model', 'message': 'Unknown model: claude-sonnet-4-5', 'details': 'Unknown model: claude-sonnet-4-5'}}\n",
+      "\n",
+      "====================================================================================================\n",
+      "Config 8/11: bge-large-en_vanilla_k3_Llama-4-Maverick_citation_focused\n",
+      "====================================================================================================\n",
+      "\n",
+      "  Example1: Daha az quyu ilə daha çox hasilat əldə etmək üçün hansı əsas...\n",
+      "    Retrieved 3 docs\n",
+      "    ✅ Generated in 2.24s\n",
+      "\n",
+      "  Example2: Qərbi Abşeron yatağında suvurma tədbirləri hansı tarixdə və ...\n",
+      "    Retrieved 3 docs\n",
+      "    ✅ Generated in 3.82s\n",
+      "\n",
+      "  Example3: Pirallahı strukturunda 1253 nömrəli quyudan götürülmüş nümun...\n",
+      "    Retrieved 3 docs\n",
+      "    ✅ Generated in 2.36s\n",
+      "\n",
+      "  Example4: Bakı arxipelaqı (BA) və Aşağı Kür çökəkliyi (AKÇ) üçün geote...\n",
+      "    Retrieved 3 docs\n",
+      "    ✅ Generated in 3.30s\n",
+      "\n",
+      "  Example5: Bu zonada hansı proseslər baş verir?...\n",
+      "    Retrieved 3 docs\n",
+      "    ✅ Generated in 1.59s\n",
+      "\n",
+      "  📊 Config Summary:\n",
+      "     Avg LLM Judge Score: 57.53%\n",
+      "     Avg Response Time: 2.66s\n",
+      "\n",
+      "====================================================================================================\n",
+      "Config 9/11: bge-large-en_vanilla_k3_Llama-4-Maverick_few_shot\n",
+      "====================================================================================================\n",
+      "\n",
+      "  Example1: Daha az quyu ilə daha çox hasilat əldə etmək üçün hansı əsas...\n",
+      "    Retrieved 3 docs\n",
+      "    ✅ Generated in 0.87s\n",
+      "\n",
+      "  Example2: Qərbi Abşeron yatağında suvurma tədbirləri hansı tarixdə və ...\n",
+      "    Retrieved 3 docs\n",
+      "    ✅ Generated in 1.51s\n",
+      "\n",
+      "  Example3: Pirallahı strukturunda 1253 nömrəli quyudan götürülmüş nümun...\n",
+      "    Retrieved 3 docs\n",
+      "    ✅ Generated in 1.96s\n",
+      "\n",
+      "  Example4: Bakı arxipelaqı (BA) və Aşağı Kür çökəkliyi (AKÇ) üçün geote...\n",
+      "    Retrieved 3 docs\n",
+      "    ✅ Generated in 2.77s\n",
+      "\n",
+      "  Example5: Bu zonada hansı proseslər baş verir?...\n",
+      "    Retrieved 3 docs\n",
+      "    ✅ Generated in 1.08s\n",
+      "\n",
+      "  📊 Config Summary:\n",
+      "     Avg LLM Judge Score: 61.51%\n",
+      "     Avg Response Time: 1.64s\n",
+      "\n",
+      "====================================================================================================\n",
+      "Config 10/11: bge-large-en_reranked_k3_GPT-5-mini_citation_focused\n",
+      "====================================================================================================\n",
+      "\n",
+      "  Example1: Daha az quyu ilə daha çox hasilat əldə etmək üçün hansı əsas...\n",
+      "    Retrieved 3 docs\n",
+      "    ❌ ERROR: Error code: 400 - {'error': {'message': \"Unsupported parameter: 'max_tokens' is not supported with this model. Use 'max_completion_tokens' instead.\", 'type': 'invalid_request_error', 'param': 'max_tokens', 'code': 'unsupported_parameter'}}\n",
+      "\n",
+      "  Example2: Qərbi Abşeron yatağında suvurma tədbirləri hansı tarixdə və ...\n",
+      "    Retrieved 3 docs\n",
+      "    ❌ ERROR: Error code: 400 - {'error': {'message': \"Unsupported parameter: 'max_tokens' is not supported with this model. Use 'max_completion_tokens' instead.\", 'type': 'invalid_request_error', 'param': 'max_tokens', 'code': 'unsupported_parameter'}}\n",
+      "\n",
+      "  Example3: Pirallahı strukturunda 1253 nömrəli quyudan götürülmüş nümun...\n",
+      "    Retrieved 3 docs\n",
+      "    ❌ ERROR: Error code: 400 - {'error': {'message': \"Unsupported parameter: 'max_tokens' is not supported with this model. Use 'max_completion_tokens' instead.\", 'type': 'invalid_request_error', 'param': 'max_tokens', 'code': 'unsupported_parameter'}}\n",
+      "\n",
+      "  Example4: Bakı arxipelaqı (BA) və Aşağı K��r çökəkliyi (AKÇ) üçün geote...\n",
+      "    Retrieved 3 docs\n",
+      "    ❌ ERROR: Error code: 400 - {'error': {'message': \"Unsupported parameter: 'max_tokens' is not supported with this model. Use 'max_completion_tokens' instead.\", 'type': 'invalid_request_error', 'param': 'max_tokens', 'code': 'unsupported_parameter'}}\n",
+      "\n",
+      "  Example5: Bu zonada hansı proseslər baş verir?...\n",
+      "    Retrieved 3 docs\n",
+      "    ❌ ERROR: Error code: 400 - {'error': {'message': \"Unsupported parameter: 'max_tokens' is not supported with this model. Use 'max_completion_tokens' instead.\", 'type': 'invalid_request_error', 'param': 'max_tokens', 'code': 'unsupported_parameter'}}\n",
+      "\n",
+      "====================================================================================================\n",
+      "Config 11/11: bge-large-en_mmr_balanced_Claude-Sonnet-4.5_citation_focused\n",
+      "====================================================================================================\n",
+      "\n",
+      "  Example1: Daha az quyu ilə daha çox hasilat əldə etmək üçün hansı əsas...\n",
+      "    Retrieved 3 docs\n",
+      "    ❌ ERROR: Error code: 400 - {'error': {'code': 'unknown_model', 'message': 'Unknown model: claude-sonnet-4-5', 'details': 'Unknown model: claude-sonnet-4-5'}}\n",
+      "\n",
+      "  Example2: Qərbi Abşeron yatağında suvurma tədbirləri hansı tarixdə və ...\n",
+      "    Retrieved 3 docs\n",
+      "    ❌ ERROR: Error code: 400 - {'error': {'code': 'unknown_model', 'message': 'Unknown model: claude-sonnet-4-5', 'details': 'Unknown model: claude-sonnet-4-5'}}\n",
+      "\n",
+      "  Example3: Pirallahı strukturunda 1253 nömrəli quyudan götürülmüş nümun...\n",
+      "    Retrieved 3 docs\n",
+      "    ❌ ERROR: Error code: 400 - {'error': {'code': 'unknown_model', 'message': 'Unknown model: claude-sonnet-4-5', 'details': 'Unknown model: claude-sonnet-4-5'}}\n",
+      "\n",
+      "  Example4: Bakı arxipelaqı (BA) və Aşağı Kür çökəkliyi (AKÇ) üçün geote...\n",
+      "    Retrieved 3 docs\n",
+      "    ❌ ERROR: Error code: 400 - {'error': {'code': 'unknown_model', 'message': 'Unknown model: claude-sonnet-4-5', 'details': 'Unknown model: claude-sonnet-4-5'}}\n",
+      "\n",
+      "  Example5: Bu zonada hansı proseslər baş verir?...\n",
+      "    Retrieved 3 docs\n",
+      "    ❌ ERROR: Error code: 400 - {'error': {'code': 'unknown_model', 'message': 'Unknown model: claude-sonnet-4-5', 'details': 'Unknown model: claude-sonnet-4-5'}}\n",
+      "\n",
+      "====================================================================================================\n",
+      "✅ Comprehensive benchmark complete!\n",
+      "====================================================================================================\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Run benchmark\n",
+    "results = []\n",
+    "\n",
+    "for config_idx, (embed_key, retrieval_key, llm_key, prompt_key) in enumerate(CONFIGS_TO_TEST, 1):\n",
+    "    config_name = f\"{embed_key}_{retrieval_key}_{llm_key}_{prompt_key}\"\n",
+    "    \n",
+    "    print(f\"\\n{'='*100}\")\n",
+    "    print(f\"Config {config_idx}/{len(CONFIGS_TO_TEST)}: {config_name}\")\n",
+    "    print(f\"{'='*100}\")\n",
+    "    \n",
+    "    # Get components\n",
+    "    embed_model = embedding_cache[embed_key]\n",
+    "    retrieval_func = RETRIEVAL_STRATEGIES[retrieval_key]['func']\n",
+    "    retrieval_params = RETRIEVAL_STRATEGIES[retrieval_key]['params']\n",
+    "    \n",
+    "    config_results = []\n",
+    "    \n",
+    "    for example_key, messages in questions.items():\n",
+    "        user_msg = [m for m in messages if m['role'] == 'user'][-1]\n",
+    "        query = user_msg['content']\n",
+    "        \n",
+    "        print(f\"\\n  {example_key}: {query[:60]}...\")\n",
+    "        \n",
+    "        # Retrieve documents\n",
+    "        documents = retrieval_func(query, embed_model, **retrieval_params)\n",
+    "        print(f\"    Retrieved {len(documents)} docs\")\n",
+    "        \n",
+    "        # Generate answer\n",
+    "        answer, response_time = generate_answer(llm_key, query, documents, prompt_key)\n",
+    "        \n",
+    "        if answer.startswith('ERROR'):\n",
+    "            print(f\"    ❌ {answer}\")\n",
+    "            continue\n",
+    "        \n",
+    "        print(f\"    ✅ Generated in {response_time:.2f}s\")\n",
+    "        \n",
+    "        # Evaluate\n",
+    "        expected = expected_answers.get(example_key, {}).get('Answer', '')\n",
+    "        \n",
+    "        accuracy_metrics = calculate_answer_quality(expected, answer) if expected else {'Accuracy_Score': 0}\n",
+    "        citation_metrics = evaluate_citation_quality(answer, documents)\n",
+    "        retrieval_metrics = evaluate_retrieval_quality(query, documents, expected)\n",
+    "        completeness_metrics = evaluate_completeness(answer)\n",
+    "        \n",
+    "        # Calculate overall score\n",
+    "        llm_judge_score = calculate_llm_judge_score(\n",
+    "            accuracy_metrics['Accuracy_Score'],\n",
+    "            citation_metrics['Citation_Score'],\n",
+    "            completeness_metrics['Completeness_Score']\n",
+    "        )\n",
+    "        \n",
+    "        result = {\n",
+    "            'Config': config_name,\n",
+    "            'Embedding_Model': embed_key,\n",
+    "            'Retrieval_Strategy': retrieval_key,\n",
+    "            'LLM_Model': llm_key,\n",
+    "            'Prompt_Strategy': prompt_key,\n",
+    "            'Question': example_key,\n",
+    "            'Query': query[:80],\n",
+    "            'Num_Docs_Retrieved': len(documents),\n",
+    "            'Response_Time': round(response_time, 2),\n",
+    "            'LLM_Judge_Score': llm_judge_score,\n",
+    "            **accuracy_metrics,\n",
+    "            **citation_metrics,\n",
+    "            **retrieval_metrics,\n",
+    "            **completeness_metrics,\n",
+    "            'Answer_Preview': answer[:150]\n",
+    "        }\n",
+    "        \n",
+    "        results.append(result)\n",
+    "        config_results.append(result)\n",
+    "    \n",
+    "    # Show config summary\n",
+    "    if config_results:\n",
+    "        avg_score = sum(r['LLM_Judge_Score'] for r in config_results) / len(config_results)\n",
+    "        avg_time = sum(r['Response_Time'] for r in config_results) / len(config_results)\n",
+    "        print(f\"\\n  📊 Config Summary:\")\n",
+    "        print(f\"     Avg LLM Judge Score: {avg_score:.2f}%\")\n",
+    "        print(f\"     Avg Response Time: {avg_time:.2f}s\")\n",
+    "\n",
+    "print(f\"\\n{'='*100}\")\n",
+    "print(\"✅ Comprehensive benchmark complete!\")\n",
+    "print(f\"{'='*100}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 8. Analyze Results"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 16,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "========================================================================================================================\n",
+      "📊 CONFIGURATION RANKINGS (By LLM Judge Score)\n",
+      "========================================================================================================================\n",
+      "                                                                  Embedding_Model Retrieval_Strategy         LLM_Model   Prompt_Strategy  LLM_Judge_Score  Accuracy_Score  Citation_Score  Response_Time\n",
+      "Config                                                                                                                                                                                                  \n",
+      "bge-large-en_vanilla_k3_Llama-4-Maverick_few_shot                    bge-large-en         vanilla_k3  Llama-4-Maverick          few_shot            61.51           11.35           78.67           1.64\n",
+      "bge-large-en_vanilla_k3_Llama-4-Maverick_citation_focused            bge-large-en         vanilla_k3  Llama-4-Maverick  citation_focused            57.53            0.00           78.67           2.66\n",
+      "bge-large-en_mmr_balanced_Llama-4-Maverick_baseline                  bge-large-en       mmr_balanced  Llama-4-Maverick          baseline            45.40            0.00           44.00           2.16\n",
+      "bge-large-en_vanilla_k5_Llama-4-Maverick_baseline                    bge-large-en         vanilla_k5  Llama-4-Maverick          baseline            45.40            0.00           44.00           2.89\n",
+      "bge-large-en_reranked_k3_Llama-4-Maverick_baseline                   bge-large-en        reranked_k3  Llama-4-Maverick          baseline            44.47            0.00           41.33           3.08\n",
+      "bge-large-en_vanilla_k3_Llama-4-Maverick_baseline                    bge-large-en         vanilla_k3  Llama-4-Maverick          baseline            43.53            0.00           38.67           2.50\n",
+      "multilingual-e5-large_vanilla_k3_Llama-4-Maverick_baseline  multilingual-e5-large         vanilla_k3  Llama-4-Maverick          baseline            39.73            0.00           28.00           3.31\n",
+      "========================================================================================================================\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Create DataFrame\n",
+    "df = pd.DataFrame(results)\n",
+    "\n",
+    "# Aggregate by configuration\n",
+    "config_summary = df.groupby('Config').agg({\n",
+    "    'LLM_Judge_Score': 'mean',\n",
+    "    'Accuracy_Score': 'mean',\n",
+    "    'Citation_Score': 'mean',\n",
+    "    'Retrieval_Relevance': 'mean',\n",
+    "    'Completeness_Score': 'mean',\n",
+    "    'Response_Time': 'mean',\n",
+    "    'Embedding_Model': 'first',\n",
+    "    'Retrieval_Strategy': 'first',\n",
+    "    'LLM_Model': 'first',\n",
+    "    'Prompt_Strategy': 'first'\n",
+    "}).round(2)\n",
+    "\n",
+    "# Sort by LLM Judge Score\n",
+    "config_summary = config_summary.sort_values('LLM_Judge_Score', ascending=False)\n",
+    "\n",
+    "print(\"\\n\" + \"=\"*120)\n",
+    "print(\"📊 CONFIGURATION RANKINGS (By LLM Judge Score)\")\n",
+    "print(\"=\"*120)\n",
+    "display_cols = ['Embedding_Model', 'Retrieval_Strategy', 'LLM_Model', 'Prompt_Strategy', \n",
+    "                'LLM_Judge_Score', 'Accuracy_Score', 'Citation_Score', 'Response_Time']\n",
+    "print(config_summary[display_cols].to_string())\n",
+    "print(\"=\"*120)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 9. Component Analysis"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 17,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "====================================================================================================\n",
+      "🔍 COMPONENT IMPACT ANALYSIS\n",
+      "====================================================================================================\n",
+      "\n",
+      "📚 EMBEDDING MODELS:\n",
+      "   bge-large-en: 49.64%\n",
+      "   multilingual-e5-large: 39.73%\n",
+      "\n",
+      "🔎 RETRIEVAL STRATEGIES:\n",
+      "   vanilla_k3: 50.58% (Current setup)\n",
+      "   mmr_balanced: 45.40% (Balance diversity)\n",
+      "   vanilla_k5: 45.40% (More context)\n",
+      "   reranked_k3: 44.47% (Two-stage rerank)\n",
+      "\n",
+      "🤖 LLM MODELS:\n",
+      "   Llama-4-Maverick: 48.22%\n",
+      "\n",
+      "💬 PROMPTING STRATEGIES:\n",
+      "   few_shot: 61.51%\n",
+      "   citation_focused: 57.53%\n",
+      "   baseline: 43.71%\n",
+      "\n",
+      "====================================================================================================\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Analyze impact of each component\n",
+    "print(\"\\n\" + \"=\"*100)\n",
+    "print(\"🔍 COMPONENT IMPACT ANALYSIS\")\n",
+    "print(\"=\"*100)\n",
+    "\n",
+    "# 1. Embedding Models\n",
+    "print(\"\\n📚 EMBEDDING MODELS:\")\n",
+    "embed_impact = df.groupby('Embedding_Model')['LLM_Judge_Score'].mean().sort_values(ascending=False)\n",
+    "for model, score in embed_impact.items():\n",
+    "    print(f\"   {model}: {score:.2f}%\")\n",
+    "\n",
+    "# 2. Retrieval Strategies\n",
+    "print(\"\\n🔎 RETRIEVAL STRATEGIES:\")\n",
+    "retrieval_impact = df.groupby('Retrieval_Strategy')['LLM_Judge_Score'].mean().sort_values(ascending=False)\n",
+    "for strategy, score in retrieval_impact.items():\n",
+    "    notes = RETRIEVAL_STRATEGIES[strategy]['notes']\n",
+    "    print(f\"   {strategy}: {score:.2f}% ({notes})\")\n",
+    "\n",
+    "# 3. LLM Models\n",
+    "print(\"\\n🤖 LLM MODELS:\")\n",
+    "llm_impact = df.groupby('LLM_Model')['LLM_Judge_Score'].mean().sort_values(ascending=False)\n",
+    "for model, score in llm_impact.items():\n",
+    "    print(f\"   {model}: {score:.2f}%\")\n",
+    "\n",
+    "# 4. Prompting Strategies\n",
+    "print(\"\\n💬 PROMPTING STRATEGIES:\")\n",
+    "prompt_impact = df.groupby('Prompt_Strategy')['LLM_Judge_Score'].mean().sort_values(ascending=False)\n",
+    "for strategy, score in prompt_impact.items():\n",
+    "    print(f\"   {strategy}: {score:.2f}%\")\n",
+    "\n",
+    "print(\"\\n\" + \"=\"*100)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 10. Visualizations"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": "import os\nfrom pathlib import Path\n\n# Create output directory\noutput_dir = Path('output/rag_optimization_benchmark')\noutput_dir.mkdir(parents=True, exist_ok=True)\n\nfig, axes = plt.subplots(2, 3, figsize=(20, 12))\n\n# 1. Top Configurations\nax1 = axes[0, 0]\ntop_configs = config_summary.head(10)\nconfig_labels = [c.split('_')[-2] + '+' + c.split('_')[-1] for c in top_configs.index]\nax1.barh(config_labels, top_configs['LLM_Judge_Score'], color=sns.color_palette('viridis', len(top_configs)))\nax1.set_xlabel('LLM Judge Score (%)', fontsize=11, fontweight='bold')\nax1.set_title('Top 10 Configurations', fontsize=13, fontweight='bold')\nax1.set_xlim(0, 100)\nfor i, score in enumerate(top_configs['LLM_Judge_Score']):\n    ax1.text(score + 1, i, f'{score:.1f}', va='center', fontsize=10)\n\n# 2. Embedding Model Impact\nax2 = axes[0, 1]\nax2.bar(embed_impact.index, embed_impact.values, color='skyblue', alpha=0.8)\nax2.set_ylabel('Avg LLM Judge Score (%)', fontsize=11, fontweight='bold')\nax2.set_title('Embedding Model Impact', fontsize=13, fontweight='bold')\nax2.set_ylim(0, 100)\nax2.tick_params(axis='x', rotation=45)\nfor i, (model, score) in enumerate(embed_impact.items()):\n    ax2.text(i, score + 2, f'{score:.1f}', ha='center', fontsize=10)\n\n# 3. Retrieval Strategy Impact\nax3 = axes[0, 2]\nax3.bar(retrieval_impact.index, retrieval_impact.values, color='coral', alpha=0.8)\nax3.set_ylabel('Avg LLM Judge Score (%)', fontsize=11, fontweight='bold')\nax3.set_title('Retrieval Strategy Impact', fontsize=13, fontweight='bold')\nax3.set_ylim(0, 100)\nax3.tick_params(axis='x', rotation=45)\nfor i, (strategy, score) in enumerate(retrieval_impact.items()):\n    ax3.text(i, score + 2, f'{score:.1f}', ha='center', fontsize=9)\n\n# 4. LLM Model Impact\nax4 = axes[1, 0]\nax4.bar(llm_impact.index, llm_impact.values, color='mediumseagreen', alpha=0.8)\nax4.set_ylabel('Avg LLM Judge Score (%)', fontsize=11, fontweight='bold')\nax4.set_title('LLM Model Impact', fontsize=13, fontweight='bold')\nax4.set_ylim(0, 100)\nax4.tick_params(axis='x', rotation=45)\nfor i, (model, score) in enumerate(llm_impact.items()):\n    ax4.text(i, score + 2, f'{score:.1f}', ha='center', fontsize=10)\n\n# 5. Prompting Strategy Impact\nax5 = axes[1, 1]\nax5.bar(prompt_impact.index, prompt_impact.values, color='mediumpurple', alpha=0.8)\nax5.set_ylabel('Avg LLM Judge Score (%)', fontsize=11, fontweight='bold')\nax5.set_title('Prompting Strategy Impact', fontsize=13, fontweight='bold')\nax5.set_ylim(0, 100)\nax5.tick_params(axis='x', rotation=45)\nfor i, (strategy, score) in enumerate(prompt_impact.items()):\n    ax5.text(i, score + 2, f'{score:.1f}', ha='center', fontsize=10)\n\n# 6. Score Components (best config)\nax6 = axes[1, 2]\nbest_config = config_summary.iloc[0]\ncomponents = ['Accuracy', 'Citation', 'Completeness']\nscores = [best_config['Accuracy_Score'], best_config['Citation_Score'], best_config['Completeness_Score']]\ncolors_comp = ['#FF6B6B', '#4ECDC4', '#45B7D1']\nbars = ax6.bar(components, scores, color=colors_comp, alpha=0.8)\nax6.set_ylabel('Score (%)', fontsize=11, fontweight='bold')\nax6.set_title(f'Best Config Components\\n{best_config.name.split(\"_\")[2]}', fontsize=13, fontweight='bold')\nax6.set_ylim(0, 100)\nfor i, score in enumerate(scores):\n    ax6.text(i, score + 2, f'{score:.1f}%', ha='center', fontsize=10, fontweight='bold')\n\nplt.tight_layout()\nplt.savefig(output_dir / 'results.png', dpi=300, bbox_inches='tight')\nplt.show()\n\nprint(f\"\\n✅ Visualization saved to '{output_dir}/results.png'\")"
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 11. Final Recommendations"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 19,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "====================================================================================================\n",
+      "🏆 OPTIMAL RAG CONFIGURATION\n",
+      "====================================================================================================\n",
+      "\n",
+      "✅ Best Configuration: bge-large-en_vanilla_k3_Llama-4-Maverick_few_shot\n",
+      "\n",
+      "📊 Performance:\n",
+      "   LLM Judge Score: 61.51%\n",
+      "   Accuracy: 11.35%\n",
+      "   Citation Quality: 78.67%\n",
+      "   Completeness: 100.00%\n",
+      "   Avg Response Time: 1.64s\n",
+      "\n",
+      "⚙️ Components:\n",
+      "   Embedding Model: bge-large-en\n",
+      "      → BAAI/bge-large-en-v1.5\n",
+      "   Retrieval Strategy: vanilla_k3\n",
+      "      → Current setup\n",
+      "   LLM Model: Llama-4-Maverick\n",
+      "   Prompting Strategy: few_shot\n",
+      "\n",
+      "💡 Key Findings:\n",
+      "   1. Best Embedding: bge-large-en (49.64%)\n",
+      "   2. Best Retrieval: vanilla_k3 (50.58%)\n",
+      "   3. Best LLM: Llama-4-Maverick (48.22%)\n",
+      "   4. Best Prompt: few_shot (61.51%)\n",
+      "\n",
+      "🎯 Hackathon Impact:\n",
+      "   LLM Quality = 30% of total score\n",
+      "   Your score: 61.51% × 30% = 18.45 points\n",
+      "\n",
+      "📈 Improvement vs Baseline:\n",
+      "   +24.51% quality improvement\n",
+      "   = +7.35 hackathon points\n",
+      "\n",
+      "====================================================================================================\n",
+      "📝 IMPLEMENTATION CHECKLIST\n",
+      "====================================================================================================\n",
+      "\n",
+      "1. Use embedding model: BAAI/bge-large-en-v1.5\n",
+      "2. Implement retrieval: vanilla_k3\n",
+      "3. Use LLM model: Llama-4-Maverick\n",
+      "4. Apply prompt: few_shot\n",
+      "\n",
+      "5. Expected performance:\n",
+      "   - LLM Judge Score: 61.51%\n",
+      "   - Response time: ~1.6s\n",
+      "====================================================================================================\n"
+     ]
+    }
+   ],
+   "source": [
+    "best_config = config_summary.iloc[0]\n",
+    "\n",
+    "print(\"\\n\" + \"=\"*100)\n",
+    "print(\"🏆 OPTIMAL RAG CONFIGURATION\")\n",
+    "print(\"=\"*100)\n",
+    "\n",
+    "print(f\"\\n✅ Best Configuration: {best_config.name}\")\n",
+    "print(f\"\\n📊 Performance:\")\n",
+    "print(f\"   LLM Judge Score: {best_config['LLM_Judge_Score']:.2f}%\")\n",
+    "print(f\"   Accuracy: {best_config['Accuracy_Score']:.2f}%\")\n",
+    "print(f\"   Citation Quality: {best_config['Citation_Score']:.2f}%\")\n",
+    "print(f\"   Completeness: {best_config['Completeness_Score']:.2f}%\")\n",
+    "print(f\"   Avg Response Time: {best_config['Response_Time']:.2f}s\")\n",
+    "\n",
+    "print(f\"\\n⚙️ Components:\")\n",
+    "print(f\"   Embedding Model: {best_config['Embedding_Model']}\")\n",
+    "print(f\"      → {EMBEDDING_MODELS[best_config['Embedding_Model']]['name']}\")\n",
+    "print(f\"   Retrieval Strategy: {best_config['Retrieval_Strategy']}\")\n",
+    "print(f\"      → {RETRIEVAL_STRATEGIES[best_config['Retrieval_Strategy']]['notes']}\")\n",
+    "print(f\"   LLM Model: {best_config['LLM_Model']}\")\n",
+    "print(f\"   Prompting Strategy: {best_config['Prompt_Strategy']}\")\n",
+    "\n",
+    "print(f\"\\n💡 Key Findings:\")\n",
+    "print(f\"   1. Best Embedding: {embed_impact.index[0]} ({embed_impact.values[0]:.2f}%)\")\n",
+    "print(f\"   2. Best Retrieval: {retrieval_impact.index[0]} ({retrieval_impact.values[0]:.2f}%)\")\n",
+    "print(f\"   3. Best LLM: {llm_impact.index[0]} ({llm_impact.values[0]:.2f}%)\")\n",
+    "print(f\"   4. Best Prompt: {prompt_impact.index[0]} ({prompt_impact.values[0]:.2f}%)\")\n",
+    "\n",
+    "print(f\"\\n🎯 Hackathon Impact:\")\n",
+    "print(f\"   LLM Quality = 30% of total score\")\n",
+    "print(f\"   Your score: {best_config['LLM_Judge_Score']:.2f}% × 30% = {best_config['LLM_Judge_Score'] * 0.3:.2f} points\")\n",
+    "\n",
+    "baseline = df[df['Config'].str.contains('baseline')].iloc[0] if len(df[df['Config'].str.contains('baseline')]) > 0 else None\n",
+    "if baseline is not None:\n",
+    "    improvement = best_config['LLM_Judge_Score'] - baseline['LLM_Judge_Score']\n",
+    "    print(f\"\\n📈 Improvement vs Baseline:\")\n",
+    "    print(f\"   +{improvement:.2f}% quality improvement\")\n",
+    "    print(f\"   = +{improvement * 0.3:.2f} hackathon points\")\n",
+    "\n",
+    "print(\"\\n\" + \"=\"*100)\n",
+    "print(\"📝 IMPLEMENTATION CHECKLIST\")\n",
+    "print(\"=\"*100)\n",
+    "print(f\"\\n1. Use embedding model: {EMBEDDING_MODELS[best_config['Embedding_Model']]['name']}\")\n",
+    "print(f\"2. Implement retrieval: {best_config['Retrieval_Strategy']}\")\n",
+    "print(f\"3. Use LLM model: {best_config['LLM_Model']}\")\n",
+    "print(f\"4. Apply prompt: {best_config['Prompt_Strategy']}\")\n",
+    "print(f\"\\n5. Expected performance:\")\n",
+    "print(f\"   - LLM Judge Score: {best_config['LLM_Judge_Score']:.2f}%\")\n",
+    "print(f\"   - Response time: ~{best_config['Response_Time']:.1f}s\")\n",
+    "print(\"=\"*100)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 12. Export Results"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": "# Save results\nfrom pathlib import Path\n\noutput_dir = Path('output/rag_optimization_benchmark')\noutput_dir.mkdir(parents=True, exist_ok=True)\n\ndf.to_csv(output_dir / 'detailed_results.csv', index=False, encoding='utf-8')\nconfig_summary.to_csv(output_dir / 'summary.csv', encoding='utf-8')\n\n# Save component impacts\nimpacts = pd.DataFrame({\n    'Embedding_Impact': embed_impact,\n    'Retrieval_Impact': retrieval_impact.reindex(embed_impact.index, fill_value=0),\n    'LLM_Impact': llm_impact.reindex(embed_impact.index, fill_value=0),\n    'Prompt_Impact': prompt_impact.reindex(embed_impact.index, fill_value=0)\n}).fillna(0)\nimpacts.to_csv(output_dir / 'component_impacts.csv', encoding='utf-8')\n\nprint(\"\\n✅ Results exported to output/rag_optimization_benchmark/:\")\nprint(\"   - detailed_results.csv (all tests)\")\nprint(\"   - summary.csv (config rankings)\")\nprint(\"   - component_impacts.csv (component analysis)\")\nprint(\"   - results.png (visualizations)\")"
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "venv",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.12"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

notebooks/requirements_llm_benchmark.txt

@@ -0,0 +1,27 @@
+# LLM Benchmarking Requirements
+# Install with: pip install -r requirements_llm_benchmark.txt
+
+# Azure OpenAI client
+openai==1.54.0
+
+# Vector Database
+pinecone-client==5.0.0
+
+# Embeddings
+sentence-transformers==3.3.1
+
+# Metrics
+jiwer==3.0.3
+
+# Data analysis and visualization
+pandas==2.1.3
+matplotlib==3.8.2
+seaborn==0.13.0
+
+# Utilities
+python-dotenv==1.0.0
+numpy==1.26.2
+
+# Jupyter
+jupyter==1.0.0
+ipykernel==6.27.1

notebooks/requirements_rag_optimization.txt

@@ -0,0 +1,27 @@
+# RAG Pipeline Optimization Requirements
+# Install with: pip install -r requirements_rag_optimization.txt
+
+# Azure OpenAI
+openai==1.54.0
+
+# Vector Database
+pinecone-client==5.0.0
+
+# Embeddings and Reranking
+sentence-transformers==3.3.1
+
+# Metrics
+jiwer==3.0.3
+
+# Data analysis and visualization
+pandas==2.1.3
+matplotlib==3.8.2
+seaborn==0.13.0
+numpy==1.26.2
+
+# Utilities
+python-dotenv==1.0.0
+
+# Jupyter
+jupyter==1.0.0
+ipykernel==6.27.1

notebooks/requirements_vlm_ocr.txt

@@ -0,0 +1,24 @@
+# VLM OCR Benchmarking Requirements
+# Install with: pip install -r requirements_vlm_ocr.txt
+
+# Azure OpenAI client (for vision models)
+openai==1.54.0
+
+# PDF processing
+PyMuPDF==1.23.8
+Pillow==10.1.0
+
+# Metrics
+jiwer==3.0.3
+
+# Data analysis and visualization
+pandas==2.1.3
+matplotlib==3.8.2
+seaborn==0.13.0
+
+# Utilities
+python-dotenv==1.0.0
+
+# Jupyter
+jupyter==1.0.0
+ipykernel==6.27.1

notebooks/vlm_ocr_benchmark.ipynb

@@ -0,0 +1,891 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Vision-Language Model (VLM) OCR Benchmarking\n",
+    "\n",
+    "Testing **Vision-Language Models** for OCR on historical SOCAR documents.\n",
+    "\n",
+    "## Why VLMs for OCR?\n",
+    "- **Better than traditional OCR** (Tesseract, EasyOCR, etc.)\n",
+    "- **Understands context** - can handle handwriting, layout, multi-language\n",
+    "- **Directly processes images** - no separate OCR step needed\n",
+    "- **State-of-the-art accuracy** on complex documents\n",
+    "\n",
+    "## Vision Models to Test:\n",
+    "1. **GPT-4.1** ⭐⭐⭐⭐⭐ (Excellent OCR capability)\n",
+    "2. **GPT-5, GPT-5-mini** ⭐⭐⭐⭐⭐ (Latest, best performance)\n",
+    "3. **Claude-Sonnet-4.5** ⭐⭐⭐⭐⭐ (Very good OCR)\n",
+    "4. **Phi-4-multimodal-instruct** ⭐⭐⭐⭐ (Explicitly multimodal)\n",
+    "5. **Llama-4-Maverick-17B** ⭐⭐⭐⭐ (May have vision support)\n",
+    "6. **DeepSeek-VL** (Vision-Language specialized)\n",
+    "\n",
+    "## Metrics:\n",
+    "- **CER** (Character Error Rate) - Lower is better\n",
+    "- **CSR** (Character Success Rate) = 100 - CER - Higher is better\n",
+    "- **WER** (Word Error Rate) - Lower is better\n",
+    "- **WSR** (Word Success Rate) = 100 - WER - Higher is better"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 15,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Install required packages\n",
+    "# !pip install openai PyMuPDF Pillow jiwer pandas matplotlib seaborn python-dotenv"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 16,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "✅ Libraries loaded\n"
+     ]
+    }
+   ],
+   "source": [
+    "import os\n",
+    "import base64\n",
+    "import re\n",
+    "import time\n",
+    "from pathlib import Path\n",
+    "from typing import Dict, List, Tuple\n",
+    "from io import BytesIO\n",
+    "\n",
+    "from dotenv import load_dotenv\n",
+    "import fitz  # PyMuPDF\n",
+    "from PIL import Image\n",
+    "import pandas as pd\n",
+    "import matplotlib.pyplot as plt\n",
+    "import seaborn as sns\n",
+    "from jiwer import wer, cer\n",
+    "from openai import AzureOpenAI\n",
+    "\n",
+    "# Load environment\n",
+    "load_dotenv()\n",
+    "\n",
+    "# Set style\n",
+    "sns.set_style('whitegrid')\n",
+    "plt.rcParams['figure.figsize'] = (14, 8)\n",
+    "\n",
+    "print(\"✅ Libraries loaded\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 1. Load Ground Truth"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 17,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "✅ Ground truth loaded: 22386 characters\n",
+      "Preview:\n",
+      "XÜLASƏ\n",
+      "\n",
+      "Bu tədqiqat Aşağı Kür çökəkliyi (AKÇ) və Bakı arxipelaqı (BA) daxil olmaqla Cənubi Xəzər çökəkliyi sistemində faydalı qazıntıların mənşəyinin paleotektonik, paleocoğrafi şərait və geodinamik rejimlə necə əlaqələndiyini, eləcə də Gec Miosendən etibarən Ərəbistan plitəsinin təsiri ilə formalaş...\n"
+     ]
+    }
+   ],
+   "source": [
+    "def load_ground_truth(md_path: str) -> str:\n",
+    "    \"\"\"\n",
+    "    Load ground truth text from markdown file.\n",
+    "    Removes markdown formatting for pure text comparison.\n",
+    "    \"\"\"\n",
+    "    with open(md_path, 'r', encoding='utf-8') as f:\n",
+    "        text = f.read()\n",
+    "    \n",
+    "    # Remove markdown elements\n",
+    "    text = re.sub(r'^#+\\s+', '', text, flags=re.MULTILINE)  # Headers\n",
+    "    text = re.sub(r'\\*\\*(.+?)\\*\\*', r'\\1', text)  # Bold\n",
+    "    text = re.sub(r'\\*(.+?)\\*', r'\\1', text)  # Italic\n",
+    "    text = re.sub(r'---+', '', text)  # Horizontal rules\n",
+    "    text = re.sub(r'\\n\\s*\\n+', '\\n\\n', text)  # Normalize newlines\n",
+    "    \n",
+    "    return text.strip()\n",
+    "\n",
+    "# Load ground truth\n",
+    "ground_truth = load_ground_truth('data/document_00.md')\n",
+    "print(f\"✅ Ground truth loaded: {len(ground_truth)} characters\")\n",
+    "print(f\"Preview:\\n{ground_truth[:300]}...\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 2. PDF to Image Conversion"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 18,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "✅ Converted PDF to 12 images\n",
+      "First image size: (3072, 4096)\n"
+     ]
+    }
+   ],
+   "source": [
+    "def pdf_to_images(pdf_path: str, dpi: int = 150) -> List[Image.Image]:\n",
+    "    \"\"\"\n",
+    "    Convert PDF pages to PIL Images.\n",
+    "    Higher DPI = better quality for VLMs.\n",
+    "    \"\"\"\n",
+    "    doc = fitz.open(pdf_path)\n",
+    "    images = []\n",
+    "    \n",
+    "    for page_num in range(len(doc)):\n",
+    "        page = doc[page_num]\n",
+    "        # Render at higher resolution\n",
+    "        zoom = dpi / 72  # 72 DPI is default\n",
+    "        mat = fitz.Matrix(zoom, zoom)\n",
+    "        pix = page.get_pixmap(matrix=mat)\n",
+    "        \n",
+    "        # Convert to PIL Image\n",
+    "        img = Image.frombytes(\"RGB\", [pix.width, pix.height], pix.samples)\n",
+    "        images.append(img)\n",
+    "    \n",
+    "    doc.close()\n",
+    "    return images\n",
+    "\n",
+    "def image_to_base64(image: Image.Image, format: str = 'PNG') -> str:\n",
+    "    \"\"\"\n",
+    "    Convert PIL Image to base64 string for API.\n",
+    "    \"\"\"\n",
+    "    buffered = BytesIO()\n",
+    "    image.save(buffered, format=format)\n",
+    "    return base64.b64encode(buffered.getvalue()).decode('utf-8')\n",
+    "\n",
+    "# Test conversion\n",
+    "pdf_path = 'data/pdfs/document_00.pdf'\n",
+    "test_images = pdf_to_images(pdf_path)\n",
+    "print(f\"\\n✅ Converted PDF to {len(test_images)} images\")\n",
+    "print(f\"First image size: {test_images[0].size}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 3. Vision-Language Model Client"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 19,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "✅ Configured 6 vision models\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Initialize Azure OpenAI client\n",
+    "azure_client = AzureOpenAI(\n",
+    "    api_key=os.getenv('AZURE_OPENAI_API_KEY'),\n",
+    "    api_version=os.getenv('AZURE_OPENAI_API_VERSION', '2024-08-01-preview'),\n",
+    "    azure_endpoint=os.getenv('AZURE_OPENAI_ENDPOINT')\n",
+    ")\n",
+    "\n",
+    "# Vision model configurations\n",
+    "VLM_MODELS = {\n",
+    "    'GPT-4.1': {\n",
+    "        'deployment': 'gpt-4.1',\n",
+    "        'supports_vision': True,\n",
+    "        'rating': '⭐⭐⭐⭐⭐',\n",
+    "        'notes': 'Excellent OCR'\n",
+    "    },\n",
+    "    'GPT-5': {\n",
+    "        'deployment': 'gpt-5',\n",
+    "        'supports_vision': True,\n",
+    "        'rating': '⭐⭐⭐⭐⭐',\n",
+    "        'notes': 'Latest model'\n",
+    "    },\n",
+    "    'GPT-5-mini': {\n",
+    "        'deployment': 'gpt-5-mini',\n",
+    "        'supports_vision': True,\n",
+    "        'rating': '⭐⭐⭐⭐⭐',\n",
+    "        'notes': 'Fast + excellent'\n",
+    "    },\n",
+    "    'Claude-Sonnet-4.5': {\n",
+    "        'deployment': 'claude-sonnet-4-5',\n",
+    "        'supports_vision': True,\n",
+    "        'rating': '⭐⭐⭐⭐⭐',\n",
+    "        'notes': 'Very good OCR'\n",
+    "    },\n",
+    "    'Phi-4-multimodal': {\n",
+    "        'deployment': 'Phi-4-multimodal-instruct',\n",
+    "        'supports_vision': True,\n",
+    "        'rating': '⭐⭐⭐⭐',\n",
+    "        'notes': 'Explicitly multimodal'\n",
+    "    },\n",
+    "    'Llama-4-Maverick-17B': {\n",
+    "        'deployment': 'Llama-4-Maverick-17B-128E-Instruct-FP8',\n",
+    "        'supports_vision': True,\n",
+    "        'rating': '⭐⭐⭐⭐',\n",
+    "        'notes': 'Testing vision capability'\n",
+    "    }\n",
+    "}\n",
+    "\n",
+    "print(f\"✅ Configured {len(VLM_MODELS)} vision models\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 20,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "✅ VLM extraction function ready\n"
+     ]
+    }
+   ],
+   "source": [
+    "def vlm_extract_text(model_name: str, images: List[Image.Image], \n",
+    "                     temperature: float = 0.0) -> Tuple[str, float]:\n",
+    "    \"\"\"\n",
+    "    Extract text from images using Vision-Language Model.\n",
+    "    Returns: (extracted_text, response_time)\n",
+    "    \"\"\"\n",
+    "    deployment = VLM_MODELS[model_name]['deployment']\n",
+    "    \n",
+    "    # OCR prompt - optimized for accuracy\n",
+    "    system_prompt = \"\"\"You are an expert OCR system for historical oil & gas documents.\n",
+    "\n",
+    "Extract ALL text from the image with 100% accuracy. Follow these rules:\n",
+    "1. Preserve EXACT spelling - including Azerbaijani, Russian, and English text\n",
+    "2. Maintain original Cyrillic characters - DO NOT transliterate\n",
+    "3. Keep all numbers, symbols, and special characters exactly as shown\n",
+    "4. Preserve layout structure (paragraphs, line breaks)\n",
+    "5. Include ALL text - headers, body, footnotes, tables, captions\n",
+    "6. If text is unclear, make best effort but stay accurate\n",
+    "\n",
+    "Output ONLY the extracted text. No explanations, no descriptions.\"\"\"\n",
+    "    \n",
+    "    try:\n",
+    "        all_text = []\n",
+    "        total_time = 0\n",
+    "        \n",
+    "        for page_num, image in enumerate(images, 1):\n",
+    "            # Convert image to base64\n",
+    "            image_base64 = image_to_base64(image)\n",
+    "            \n",
+    "            # Prepare messages with image\n",
+    "            messages = [\n",
+    "                {\n",
+    "                    \"role\": \"system\",\n",
+    "                    \"content\": system_prompt\n",
+    "                },\n",
+    "                {\n",
+    "                    \"role\": \"user\",\n",
+    "                    \"content\": [\n",
+    "                        {\n",
+    "                            \"type\": \"text\",\n",
+    "                            \"text\": f\"Extract all text from page {page_num}:\"\n",
+    "                        },\n",
+    "                        {\n",
+    "                            \"type\": \"image_url\",\n",
+    "                            \"image_url\": {\n",
+    "                                \"url\": f\"data:image/png;base64,{image_base64}\"\n",
+    "                            }\n",
+    "                        }\n",
+    "                    ]\n",
+    "                }\n",
+    "            ]\n",
+    "            \n",
+    "            # Call VLM with appropriate token parameter\n",
+    "            start_time = time.time()\n",
+    "            \n",
+    "            # GPT-5 models use max_completion_tokens, others use max_tokens\n",
+    "            if deployment.startswith('gpt-5'):\n",
+    "                response = azure_client.chat.completions.create(\n",
+    "                    model=deployment,\n",
+    "                    messages=messages,\n",
+    "                    temperature=temperature,\n",
+    "                    max_completion_tokens=4000\n",
+    "                )\n",
+    "            else:\n",
+    "                response = azure_client.chat.completions.create(\n",
+    "                    model=deployment,\n",
+    "                    messages=messages,\n",
+    "                    temperature=temperature,\n",
+    "                    max_tokens=4000\n",
+    "                )\n",
+    "            \n",
+    "            elapsed = time.time() - start_time\n",
+    "            total_time += elapsed\n",
+    "            \n",
+    "            # Extract text\n",
+    "            page_text = response.choices[0].message.content\n",
+    "            all_text.append(page_text)\n",
+    "            \n",
+    "            print(f\"  Page {page_num}/{len(images)}: {elapsed:.1f}s\")\n",
+    "        \n",
+    "        # Combine all pages\n",
+    "        full_text = '\\n\\n'.join(all_text)\n",
+    "        return full_text, total_time\n",
+    "    \n",
+    "    except Exception as e:\n",
+    "        return f\"ERROR: {str(e)}\", 0.0\n",
+    "\n",
+    "print(\"✅ VLM extraction function ready\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 4. Metrics Calculation"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 21,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "✅ Metrics functions ready\n"
+     ]
+    }
+   ],
+   "source": [
+    "def normalize_text(text: str) -> str:\n",
+    "    \"\"\"Normalize text for comparison.\"\"\"\n",
+    "    text = text.lower().strip()\n",
+    "    text = re.sub(r'\\s+', ' ', text)\n",
+    "    return text\n",
+    "\n",
+    "def calculate_ocr_metrics(reference: str, hypothesis: str) -> Dict[str, float]:\n",
+    "    \"\"\"\n",
+    "    Calculate comprehensive OCR metrics.\n",
+    "    \"\"\"\n",
+    "    ref_norm = normalize_text(reference)\n",
+    "    hyp_norm = normalize_text(hypothesis)\n",
+    "    \n",
+    "    # Character Error Rate\n",
+    "    cer_score = cer(ref_norm, hyp_norm) * 100\n",
+    "    \n",
+    "    # Word Error Rate\n",
+    "    wer_score = wer(ref_norm, hyp_norm) * 100\n",
+    "    \n",
+    "    # Success rates\n",
+    "    csr_score = max(0, 100 - cer_score)\n",
+    "    wsr_score = max(0, 100 - wer_score)\n",
+    "    \n",
+    "    # Length metrics\n",
+    "    ref_chars = len(ref_norm)\n",
+    "    hyp_chars = len(hyp_norm)\n",
+    "    ref_words = len(ref_norm.split())\n",
+    "    hyp_words = len(hyp_norm.split())\n",
+    "    \n",
+    "    char_length_acc = (min(ref_chars, hyp_chars) / max(ref_chars, hyp_chars) * 100) if max(ref_chars, hyp_chars) > 0 else 0\n",
+    "    word_length_acc = (min(ref_words, hyp_words) / max(ref_words, hyp_words) * 100) if max(ref_words, hyp_words) > 0 else 0\n",
+    "    \n",
+    "    return {\n",
+    "        'CER': round(cer_score, 2),\n",
+    "        'WER': round(wer_score, 2),\n",
+    "        'CSR': round(csr_score, 2),\n",
+    "        'WSR': round(wsr_score, 2),\n",
+    "        'Char_Count_Ref': ref_chars,\n",
+    "        'Char_Count_Hyp': hyp_chars,\n",
+    "        'Word_Count_Ref': ref_words,\n",
+    "        'Word_Count_Hyp': hyp_words,\n",
+    "        'Char_Length_Accuracy': round(char_length_acc, 2),\n",
+    "        'Word_Length_Accuracy': round(word_length_acc, 2)\n",
+    "    }\n",
+    "\n",
+    "print(\"✅ Metrics functions ready\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 5. Run Benchmark on All VLMs"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 22,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Testing 6 vision models...\n",
+      "This will take several minutes...\n",
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Select models to test\n",
+    "MODELS_TO_TEST = [\n",
+    "    'GPT-4.1',\n",
+    "    'GPT-5',\n",
+    "    'GPT-5-mini',\n",
+    "    'Claude-Sonnet-4.5',\n",
+    "    'Phi-4-multimodal',\n",
+    "    'Llama-4-Maverick-17B',  # Added for comparison\n",
+    "]\n",
+    "\n",
+    "print(f\"Testing {len(MODELS_TO_TEST)} vision models...\")\n",
+    "print(\"This will take several minutes...\\n\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Prepared 12 page images\n",
+      "\n",
+      "\n",
+      "================================================================================\n",
+      "Testing: GPT-4.1 ⭐⭐⭐⭐⭐\n",
+      "Notes: Excellent OCR\n",
+      "================================================================================\n",
+      "  Page 1/12: 9.2s\n",
+      "  Page 2/12: 9.5s\n",
+      "  Page 3/12: 10.9s\n",
+      "  Page 4/12: 10.7s\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Prepare PDF images\n",
+    "images = pdf_to_images(pdf_path, dpi=150)\n",
+    "print(f\"Prepared {len(images)} page images\\n\")\n",
+    "\n",
+    "# Run benchmark\n",
+    "results = []\n",
+    "\n",
+    "for model_name in MODELS_TO_TEST:\n",
+    "    print(f\"\\n{'='*80}\")\n",
+    "    print(f\"Testing: {model_name} {VLM_MODELS[model_name]['rating']}\")\n",
+    "    print(f\"Notes: {VLM_MODELS[model_name]['notes']}\")\n",
+    "    print(f\"{'='*80}\")\n",
+    "    \n",
+    "    # Extract text\n",
+    "    extracted_text, response_time = vlm_extract_text(model_name, images)\n",
+    "    \n",
+    "    if extracted_text.startswith('ERROR'):\n",
+    "        print(f\"❌ Failed: {extracted_text}\")\n",
+    "        continue\n",
+    "    \n",
+    "    print(f\"\\n✅ Total time: {response_time:.2f}s\")\n",
+    "    print(f\"✅ Extracted: {len(extracted_text)} characters\")\n",
+    "    \n",
+    "    # Calculate metrics\n",
+    "    metrics = calculate_ocr_metrics(ground_truth, extracted_text)\n",
+    "    \n",
+    "    # Store result\n",
+    "    result = {\n",
+    "        'Model': model_name,\n",
+    "        'Response_Time': round(response_time, 2),\n",
+    "        **metrics,\n",
+    "        'Rating': VLM_MODELS[model_name]['rating'],\n",
+    "        'Notes': VLM_MODELS[model_name]['notes'],\n",
+    "        'Extracted_Preview': extracted_text[:200]\n",
+    "    }\n",
+    "    \n",
+    "    results.append(result)\n",
+    "    \n",
+    "    # Show summary\n",
+    "    print(f\"\\n📊 Metrics:\")\n",
+    "    print(f\"   CSR (Character Success): {metrics['CSR']:.2f}%\")\n",
+    "    print(f\"   WSR (Word Success): {metrics['WSR']:.2f}%\")\n",
+    "    print(f\"   CER (Character Error): {metrics['CER']:.2f}%\")\n",
+    "    print(f\"   WER (Word Error): {metrics['WER']:.2f}%\")\n",
+    "\n",
+    "print(f\"\\n{'='*80}\")\n",
+    "print(\"✅ VLM OCR Benchmarking complete!\")\n",
+    "print(f\"{'='*80}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 6. Results Analysis"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "====================================================================================================\n",
+      "📊 VLM OCR BENCHMARKING RESULTS\n",
+      "====================================================================================================\n",
+      "  Model   CSR   WSR   CER   WER  Response_Time Rating\n",
+      "GPT-4.1 85.86 67.61 14.14 32.39         133.49  ⭐⭐⭐⭐⭐\n",
+      "====================================================================================================\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Create DataFrame\n",
+    "df = pd.DataFrame(results)\n",
+    "\n",
+    "# Sort by CSR (best first)\n",
+    "df = df.sort_values('CSR', ascending=False).reset_index(drop=True)\n",
+    "\n",
+    "# Display results\n",
+    "display_cols = ['Model', 'CSR', 'WSR', 'CER', 'WER', 'Response_Time', 'Rating']\n",
+    "\n",
+    "print(\"\\n\" + \"=\"*100)\n",
+    "print(\"📊 VLM OCR BENCHMARKING RESULTS\")\n",
+    "print(\"=\"*100)\n",
+    "print(df[display_cols].to_string(index=False))\n",
+    "print(\"=\"*100)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 7. Visualizations"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Create comprehensive visualization\n",
+    "import os\n",
+    "from pathlib import Path\n",
+    "\n",
+    "# Create output directory\n",
+    "output_dir = Path('output/vlm_ocr_benchmark')\n",
+    "output_dir.mkdir(parents=True, exist_ok=True)\n",
+    "\n",
+    "fig, axes = plt.subplots(2, 2, figsize=(16, 10))\n",
+    "\n",
+    "models = df['Model'].tolist()\n",
+    "colors = sns.color_palette('viridis', len(models))\n",
+    "\n",
+    "# 1. CSR - Character Success Rate (MAIN METRIC)\n",
+    "ax1 = axes[0, 0]\n",
+    "bars1 = ax1.barh(models, df['CSR'], color=colors)\n",
+    "ax1.set_xlabel('CSR (%) - Higher is Better', fontsize=12, fontweight='bold')\n",
+    "ax1.set_title('Character Success Rate (CSR)\\n🏆 HACKATHON PRIMARY METRIC', \n",
+    "              fontsize=14, fontweight='bold')\n",
+    "ax1.set_xlim(0, 100)\n",
+    "for i, (model, csr) in enumerate(zip(models, df['CSR'])):\n",
+    "    ax1.text(csr + 1, i, f'{csr:.2f}%', va='center', fontsize=11, fontweight='bold')\n",
+    "ax1.axvline(x=90, color='green', linestyle='--', alpha=0.3, label='Excellent (>90%)')\n",
+    "ax1.axvline(x=80, color='orange', linestyle='--', alpha=0.3, label='Good (>80%)')\n",
+    "ax1.legend(fontsize=9)\n",
+    "\n",
+    "# 2. WSR - Word Success Rate\n",
+    "ax2 = axes[0, 1]\n",
+    "bars2 = ax2.barh(models, df['WSR'], color=colors)\n",
+    "ax2.set_xlabel('WSR (%) - Higher is Better', fontsize=12, fontweight='bold')\n",
+    "ax2.set_title('Word Success Rate (WSR)', fontsize=14, fontweight='bold')\n",
+    "ax2.set_xlim(0, 100)\n",
+    "for i, (model, wsr) in enumerate(zip(models, df['WSR'])):\n",
+    "    ax2.text(wsr + 1, i, f'{wsr:.2f}%', va='center', fontsize=11, fontweight='bold')\n",
+    "\n",
+    "# 3. Response Time\n",
+    "ax3 = axes[1, 0]\n",
+    "bars3 = ax3.barh(models, df['Response_Time'], color=colors)\n",
+    "ax3.set_xlabel('Total Time (seconds) - Lower is Better', fontsize=12, fontweight='bold')\n",
+    "ax3.set_title('Processing Speed', fontsize=14, fontweight='bold')\n",
+    "for i, (model, time_val) in enumerate(zip(models, df['Response_Time'])):\n",
+    "    ax3.text(time_val + 0.5, i, f'{time_val:.1f}s', va='center', fontsize=11)\n",
+    "\n",
+    "# 4. Error Rates Comparison\n",
+    "ax4 = axes[1, 1]\n",
+    "x = range(len(models))\n",
+    "width = 0.35\n",
+    "ax4.bar([i - width/2 for i in x], df['CER'], width, label='CER', color='coral', alpha=0.8)\n",
+    "ax4.bar([i + width/2 for i in x], df['WER'], width, label='WER', color='skyblue', alpha=0.8)\n",
+    "ax4.set_ylabel('Error Rate (%) - Lower is Better', fontsize=12, fontweight='bold')\n",
+    "ax4.set_title('Error Rates', fontsize=14, fontweight='bold')\n",
+    "ax4.set_xticks(x)\n",
+    "ax4.set_xticklabels(models, rotation=45, ha='right')\n",
+    "ax4.legend(fontsize=11)\n",
+    "ax4.grid(axis='y', alpha=0.3)\n",
+    "\n",
+    "plt.tight_layout()\n",
+    "plt.savefig(output_dir / 'results.png', dpi=300, bbox_inches='tight')\n",
+    "plt.show()\n",
+    "\n",
+    "print(f\"\\n✅ Visualization saved to '{output_dir}/results.png'\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 8. Winner Analysis and Recommendations"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "====================================================================================================\n",
+      "🏆 FINAL RANKINGS\n",
+      "====================================================================================================\n",
+      " Rank   Model   CSR   WSR   CER   WER  Response_Time Rating\n",
+      "    1 GPT-4.1 85.86 67.61 14.14 32.39         133.49  ⭐⭐⭐⭐⭐\n",
+      "====================================================================================================\n",
+      "\n",
+      "====================================================================================================\n",
+      "💡 RECOMMENDATIONS FOR HACKATHON\n",
+      "====================================================================================================\n",
+      "\n",
+      "🥇 BEST OVERALL: GPT-4.1 ⭐⭐⭐⭐⭐\n",
+      "   CSR: 85.86% (Character Success)\n",
+      "   WSR: 67.61% (Word Success)\n",
+      "   CER: 14.14% (Character Error)\n",
+      "   WER: 32.39% (Word Error)\n",
+      "   Time: 133.49s for 12 pages\n",
+      "   Notes: Excellent OCR\n",
+      "\n",
+      "⚡ FASTEST: GPT-4.1\n",
+      "   Time: 133.49s\n",
+      "   CSR: 85.86%\n",
+      "\n",
+      "====================================================================================================\n",
+      "📝 HACKATHON SCORING IMPACT\n",
+      "====================================================================================================\n",
+      "\n",
+      "OCR Quality = 50% of total hackathon score\n",
+      "\n",
+      "Using GPT-4.1:\n",
+      "  - CSR: 85.86% × 50% = 42.93 points\n",
+      "  - This is 85.9% accuracy on character-level OCR\n",
+      "\n",
+      "⚠️ GOOD - Consider optimizing prompt or trying other models\n",
+      "\n",
+      "====================================================================================================\n",
+      "🎯 FINAL RECOMMENDATION\n",
+      "====================================================================================================\n",
+      "\n",
+      "Use: GPT-4.1\n",
+      "Reason: Highest accuracy (85.86% CSR) for hackathon OCR benchmark\n",
+      "Implementation: Use vision API with same prompt as tested\n",
+      "====================================================================================================\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Rankings\n",
+    "rankings = df[['Model', 'CSR', 'WSR', 'CER', 'WER', 'Response_Time', 'Rating']].copy()\n",
+    "rankings.insert(0, 'Rank', range(1, len(rankings) + 1))\n",
+    "\n",
+    "print(\"\\n\" + \"=\"*100)\n",
+    "print(\"🏆 FINAL RANKINGS\")\n",
+    "print(\"=\"*100)\n",
+    "print(rankings.to_string(index=False))\n",
+    "print(\"=\"*100)\n",
+    "\n",
+    "# Winner\n",
+    "best_model = df.iloc[0]\n",
+    "fastest_model = df.loc[df['Response_Time'].idxmin()]\n",
+    "\n",
+    "print(\"\\n\" + \"=\"*100)\n",
+    "print(\"💡 RECOMMENDATIONS FOR HACKATHON\")\n",
+    "print(\"=\"*100)\n",
+    "\n",
+    "print(f\"\\n🥇 BEST OVERALL: {best_model['Model']} {best_model['Rating']}\")\n",
+    "print(f\"   CSR: {best_model['CSR']:.2f}% (Character Success)\")\n",
+    "print(f\"   WSR: {best_model['WSR']:.2f}% (Word Success)\")\n",
+    "print(f\"   CER: {best_model['CER']:.2f}% (Character Error)\")\n",
+    "print(f\"   WER: {best_model['WER']:.2f}% (Word Error)\")\n",
+    "print(f\"   Time: {best_model['Response_Time']:.2f}s for {len(images)} pages\")\n",
+    "print(f\"   Notes: {best_model['Notes']}\")\n",
+    "\n",
+    "print(f\"\\n⚡ FASTEST: {fastest_model['Model']}\")\n",
+    "print(f\"   Time: {fastest_model['Response_Time']:.2f}s\")\n",
+    "print(f\"   CSR: {fastest_model['CSR']:.2f}%\")\n",
+    "\n",
+    "print(\"\\n\" + \"=\"*100)\n",
+    "print(\"📝 HACKATHON SCORING IMPACT\")\n",
+    "print(\"=\"*100)\n",
+    "print(\"\\nOCR Quality = 50% of total hackathon score\")\n",
+    "print(f\"\\nUsing {best_model['Model']}:\")\n",
+    "print(f\"  - CSR: {best_model['CSR']:.2f}% × 50% = {best_model['CSR'] * 0.5:.2f} points\")\n",
+    "print(f\"  - This is {best_model['CSR']:.1f}% accuracy on character-level OCR\")\n",
+    "\n",
+    "if best_model['CSR'] >= 95:\n",
+    "    print(\"\\n✅ EXCELLENT - This will score very high on OCR!\")\n",
+    "elif best_model['CSR'] >= 90:\n",
+    "    print(\"\\n✅ VERY GOOD - Strong OCR performance!\")\n",
+    "elif best_model['CSR'] >= 85:\n",
+    "    print(\"\\n⚠️ GOOD - Consider optimizing prompt or trying other models\")\n",
+    "else:\n",
+    "    print(\"\\n⚠️ NEEDS IMPROVEMENT - Try other models or adjust parameters\")\n",
+    "\n",
+    "print(\"\\n\" + \"=\"*100)\n",
+    "print(\"🎯 FINAL RECOMMENDATION\")\n",
+    "print(\"=\"*100)\n",
+    "print(f\"\\nUse: {best_model['Model']}\")\n",
+    "print(f\"Reason: Highest accuracy ({best_model['CSR']:.2f}% CSR) for hackathon OCR benchmark\")\n",
+    "print(f\"Implementation: Use vision API with same prompt as tested\")\n",
+    "print(\"=\"*100)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 9. Export Results"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Save results\n",
+    "from pathlib import Path\n",
+    "\n",
+    "output_dir = Path('output/vlm_ocr_benchmark')\n",
+    "output_dir.mkdir(parents=True, exist_ok=True)\n",
+    "\n",
+    "df.to_csv(output_dir / 'detailed_results.csv', index=False, encoding='utf-8')\n",
+    "rankings.to_csv(output_dir / 'rankings.csv', index=False, encoding='utf-8')\n",
+    "\n",
+    "print(\"\\n✅ Results exported to output/vlm_ocr_benchmark/:\")\n",
+    "print(\"   - detailed_results.csv\")\n",
+    "print(\"   - rankings.csv\")\n",
+    "print(\"   - results.png\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 10. Sample Text Comparison"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "====================================================================================================\n",
+      "📝 SAMPLE TEXT COMPARISON (First 500 characters)\n",
+      "====================================================================================================\n",
+      "\n",
+      "🎯 GROUND TRUTH:\n",
+      "----------------------------------------------------------------------------------------------------\n",
+      "XÜLASƏ\n",
+      "\n",
+      "Bu tədqiqat Aşağı Kür çökəkliyi (AKÇ) və Bakı arxipelaqı (BA) daxil olmaqla Cənubi Xəzər çökəkliyi sistemində faydalı qazıntıların mənşəyinin paleotektonik, paleocoğrafi şərait və geodinamik rejimlə necə əlaqələndiyini, eləcə də Gec Miosendən etibarən Ərəbistan plitəsinin təsiri ilə formalaşan kollizion proseslərin bölgənin struktur-morfoloji və termal inkişafına nə dərəcədə yönverici rol oynadığını kompleks şəkildə qiymətləndirir. Seismotektonik göstəricilərin, çöküntütoplanma sürətləri\n",
+      "\n",
+      "🤖 GPT-4.1 (CSR: 85.86%):\n",
+      "----------------------------------------------------------------------------------------------------\n",
+      "Xülasə\n",
+      "Bu məqalə Apşə Kər çökəkliyi (AKÇ) və Bakı arxipelaqı (BA) daxil olmaqla Cənubi Xəzər çökəkliyi sistemində faydalı qazıntıların mənşəyinin paleotektonik, paleocoğrafi şərait və geodinamik rejim\n",
+      "\n",
+      "====================================================================================================\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Show comparison of first 500 characters\n",
+    "print(\"\\n\" + \"=\"*100)\n",
+    "print(\"📝 SAMPLE TEXT COMPARISON (First 500 characters)\")\n",
+    "print(\"=\"*100)\n",
+    "\n",
+    "print(\"\\n🎯 GROUND TRUTH:\")\n",
+    "print(\"-\" * 100)\n",
+    "print(ground_truth[:500])\n",
+    "\n",
+    "for _, row in df.iterrows():\n",
+    "    print(f\"\\n🤖 {row['Model']} (CSR: {row['CSR']:.2f}%):\")\n",
+    "    print(\"-\" * 100)\n",
+    "    # Get first 500 chars from extracted text\n",
+    "    preview = row['Extracted_Preview'] if len(row['Extracted_Preview']) >= 500 else row['Extracted_Preview']\n",
+    "    print(preview[:500])\n",
+    "\n",
+    "print(\"\\n\" + \"=\"*100)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "venv",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.12"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

requirements.txt

@@ -1,30 +0,0 @@
-# Web Framework
-fastapi==0.104.1
-uvicorn[standard]==0.24.0
-python-multipart==0.0.6
-
-# Azure Services (OCR + LLM)
-azure-ai-formrecognizer==3.3.2
-azure-ai-documentintelligence==1.0.0b1
-openai==1.3.0
-
-# PDF Processing & Image Extraction
-PyMuPDF==1.23.8
-
-# Vector Database & Embeddings
-pinecone-client==3.0.0
-sentence-transformers>=2.5.0
-
-# Utilities
-python-dotenv==1.0.0
-pydantic==2.5.0
-pydantic-settings==2.1.0
-requests==2.31.0
-aiofiles==23.2.1
-
-# Monitoring & Logging
-loguru==0.7.2
-
-# Development (optional)
-pytest==7.4.3
-httpx==0.25.2

run.py

@@ -1,18 +0,0 @@
-"""Run the FastAPI application"""
-
-import os
-import uvicorn
-from src.config import settings
-
-# Disable telemetry and warnings
-os.environ["TOKENIZERS_PARALLELISM"] = "false"
-os.environ["ANONYMIZED_TELEMETRY"] = "false"
-
-if __name__ == "__main__":
-    uvicorn.run(
-        "src.api.main:app",
-        host=settings.api_host,
-        port=settings.api_port,
-        reload=True,
-        log_level="info",
-    )

src/api/main.py

@@ -1,181 +0,0 @@
-"""FastAPI application with OCR and LLM endpoints"""
-
-from fastapi import FastAPI, File, UploadFile, HTTPException
-from fastapi.responses import JSONResponse, Response
-from typing import List
-from loguru import logger
-import sys
-
-from src.api.models import (
-    OCRPageResponse,
-    ChatMessage,
-    LLMResponse,
-    ErrorResponse,
-)
-from src.ocr.processor import get_ocr_processor
-from src.config import settings
-
-# Configure logging
-logger.remove()
-logger.add(sys.stderr, level="INFO")
-
-# Create FastAPI app
-app = FastAPI(
-    title="SOCAR Historical Document Processing API",
-    description="OCR and LLM endpoints for processing historical documents",
-    version="1.0.0",
-)
-
-
-@app.get("/")
-async def root():
-    """Health check endpoint"""
-    return {
-        "status": "healthy",
-        "service": "SOCAR Document Processing API",
-        "endpoints": ["/ocr", "/llm"],
-    }
-
-
-@app.get("/favicon.ico", include_in_schema=False)
-async def favicon():
-    """Return favicon for browser tab"""
-    # Simple SVG favicon representing oil/gas industry
-    svg = """<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 100 100">
-        <circle cx="50" cy="50" r="45" fill="#0066cc"/>
-        <path d="M30 60 L50 30 L70 60 Z" fill="#ffffff"/>
-        <rect x="45" y="55" width="10" height="30" fill="#ffffff"/>
-    </svg>"""
-    return Response(content=svg, media_type="image/svg+xml")
-
-
-@app.post(
-    "/ocr",
-    response_model=List[OCRPageResponse],
-    responses={
-        200: {"description": "Successfully processed PDF"},
-        400: {"model": ErrorResponse, "description": "Invalid PDF file"},
-        500: {"model": ErrorResponse, "description": "Processing error"},
-    },
-)
-async def process_ocr(file: UploadFile = File(...)):
-    """
-    OCR Endpoint - Extract text from PDF documents
-
-    Accepts a PDF file upload and returns the extracted Markdown text for each page.
-
-    Args:
-        file: PDF file in multipart/form-data format
-
-    Returns:
-        List of dictionaries with page_number and MD_text for each page
-    """
-    try:
-        # Validate file type
-        if not file.filename.lower().endswith(".pdf"):
-            raise HTTPException(
-                status_code=400,
-                detail="Invalid file type. Only PDF files are accepted.",
-            )
-
-        # Read file content
-        logger.info(f"Receiving PDF file: {file.filename}")
-        pdf_content = await file.read()
-
-        if len(pdf_content) == 0:
-            raise HTTPException(status_code=400, detail="Empty PDF file")
-
-        # Process PDF with OCR
-        ocr_processor = get_ocr_processor()
-        result = ocr_processor.process_pdf(pdf_content, file.filename)
-
-        # Convert to response format
-        response = [
-            OCRPageResponse(page_number=page["page_number"], MD_text=page["MD_text"])
-            for page in result
-        ]
-
-        logger.info(f"Successfully processed {len(response)} pages from {file.filename}")
-        return response
-
-    except HTTPException:
-        raise
-    except Exception as e:
-        logger.error(f"Error processing OCR request: {e}")
-        raise HTTPException(
-            status_code=500, detail=f"Failed to process PDF: {str(e)}"
-        )
-
-
-@app.post(
-    "/llm",
-    response_model=LLMResponse,
-    responses={
-        200: {"description": "Successfully generated response"},
-        400: {"model": ErrorResponse, "description": "Invalid request"},
-        500: {"model": ErrorResponse, "description": "Processing error"},
-    },
-)
-async def process_llm(messages: List[ChatMessage]):
-    """
-    LLM Endpoint - Generate answers from document knowledge base
-
-    Receives chat history and produces an LLM-generated answer along with source references.
-
-    Args:
-        messages: List of chat messages with role and content
-
-    Returns:
-        Dictionary with sources and answer
-    """
-    try:
-        # Validate input
-        if not messages:
-            raise HTTPException(status_code=400, detail="No messages provided")
-
-        logger.info(f"Received {len(messages)} messages for LLM processing")
-
-        # Get the last user message as the query
-        last_message = messages[-1]
-        if last_message.role != "user":
-            raise HTTPException(
-                status_code=400,
-                detail="Last message must be from user",
-            )
-
-        query = last_message.content
-
-        # Prepare chat history (all messages except the last one)
-        chat_history = None
-        if len(messages) > 1:
-            chat_history = [
-                {"role": msg.role, "content": msg.content}
-                for msg in messages[:-1]
-            ]
-
-        # Process query using RAG pipeline
-        from src.llm.rag_pipeline import get_rag_pipeline
-
-        rag = get_rag_pipeline()
-        result = rag.query(query, chat_history=chat_history)
-
-        logger.info(f"Generated answer with {len(result['sources'])} sources")
-
-        return LLMResponse(
-            sources=result["sources"],
-            answer=result["answer"],
-        )
-
-    except HTTPException:
-        raise
-    except Exception as e:
-        logger.error(f"Error processing LLM request: {e}")
-        raise HTTPException(
-            status_code=500, detail=f"Failed to generate response: {str(e)}"
-        )
-
-
-if __name__ == "__main__":
-    import uvicorn
-
-    uvicorn.run(app, host=settings.api_host, port=settings.api_port)

src/api/models.py

@@ -1,48 +0,0 @@
-"""Pydantic models for API requests and responses"""
-
-from typing import List, Dict, Optional
-from pydantic import BaseModel, Field
-
-
-class OCRPageResponse(BaseModel):
-    """Response model for a single page OCR result"""
-
-    page_number: int = Field(..., description="Page index starting from 1")
-    MD_text: str = Field(..., description="Markdown-formatted extracted text")
-
-
-class OCRResponse(BaseModel):
-    """Response model for OCR endpoint"""
-
-    pages: List[OCRPageResponse]
-    total_pages: int
-    filename: Optional[str] = None
-
-
-class ChatMessage(BaseModel):
-    """Chat message model"""
-
-    role: str = Field(..., description="Role of the message sender (user/assistant)")
-    content: str = Field(..., description="Message content")
-
-
-class SourceReference(BaseModel):
-    """Source reference for LLM response"""
-
-    pdf_name: str = Field(..., description="Name of the PDF")
-    page_number: int = Field(..., description="Page number in the PDF")
-    content: str = Field(..., description="Relevant extracted text (in Markdown)")
-
-
-class LLMResponse(BaseModel):
-    """Response model for LLM endpoint"""
-
-    sources: List[SourceReference] = Field(..., description="List of source references")
-    answer: str = Field(..., description="Generated answer to the user query")
-
-
-class ErrorResponse(BaseModel):
-    """Error response model"""
-
-    error: str
-    detail: Optional[str] = None

src/config.py

@@ -1,46 +0,0 @@
-from pydantic_settings import BaseSettings
-from pathlib import Path
-
-
-class Settings(BaseSettings):
-    """Application settings loaded from environment variables"""
-
-    # Azure OpenAI Configuration
-    azure_openai_api_key: str
-    azure_openai_endpoint: str
-    azure_openai_api_version: str = "2024-08-01-preview"
-
-    # Azure Document Intelligence
-    azure_document_intelligence_endpoint: str = ""
-    azure_document_intelligence_key: str = ""
-
-    # Application Configuration
-    data_dir: Path = Path("./data")
-    pdf_dir: Path = Path("./data/pdfs")
-    vector_db_path: Path = Path("./data/vector_db")
-    processed_dir: Path = Path("./data/processed")
-
-    # API Configuration
-    api_host: str = "0.0.0.0"
-    api_port: int = 8000
-
-    # OCR Settings
-    ocr_backend: str = "azure"  # Azure Document Intelligence (92.79% CSR)
-
-    # LLM Settings
-    llm_model: str = "Llama-4-Maverick-17B-128E-Instruct-FP8"  # Open-source LLM for hackathon
-
-    # Pinecone Settings
-    pinecone_api_key: str = ""
-    pinecone_index_name: str = "socar-documents"
-    pinecone_cloud: str = "aws"
-    pinecone_region: str = "us-east-1"
-    vector_db_type: str = "chroma"  # Options: chroma, pinecone
-
-    class Config:
-        env_file = ".env"
-        case_sensitive = False
-        extra = "ignore"  # Ignore extra fields in .env file
-
-
-settings = Settings()

src/llm/deepseek_client.py

@@ -1,126 +0,0 @@
-"""DeepSeek LLM client using Azure AI Foundry"""
-
-from typing import List, Dict, Optional
-from loguru import logger
-import openai
-
-from src.config import settings
-
-
-class DeepSeekClient:
-    """Client for DeepSeek LLM via Azure AI Foundry"""
-
-    def __init__(self):
-        """Initialize DeepSeek client"""
-        # Configure OpenAI client to use Azure endpoint
-        self.client = openai.AzureOpenAI(
-            api_key=settings.azure_openai_api_key,
-            api_version=settings.azure_openai_api_version,
-            azure_endpoint=settings.azure_openai_endpoint,
-        )
-
-        # Get model name from settings
-        self.model_name = settings.llm_model
-        logger.info(f"Initialized LLM client with model: {self.model_name}")
-
-    def generate_response(
-        self,
-        messages: List[Dict[str, str]],
-        max_tokens: int = 1000,
-        temperature: float = 0.7,
-    ) -> str:
-        """
-        Generate response from DeepSeek model
-
-        Args:
-            messages: List of message dicts with 'role' and 'content'
-            max_tokens: Maximum tokens in response
-            temperature: Sampling temperature (0.0 to 1.0)
-
-        Returns:
-            Generated text response
-        """
-        try:
-            logger.info(f"Generating response with {len(messages)} messages")
-
-            response = self.client.chat.completions.create(
-                model=self.model_name,
-                messages=messages,
-                max_tokens=max_tokens,
-                temperature=temperature,
-            )
-
-            generated_text = response.choices[0].message.content
-            logger.info(f"Generated response: {len(generated_text)} characters")
-
-            return generated_text
-
-        except Exception as e:
-            logger.error(f"Error generating response from {self.model_name}: {e}")
-            raise
-
-    def generate_with_context(
-        self,
-        query: str,
-        context_chunks: List[str],
-        chat_history: Optional[List[Dict[str, str]]] = None,
-    ) -> str:
-        """
-        Generate response with RAG context
-
-        Args:
-            query: User's question
-            context_chunks: Retrieved document chunks
-            chat_history: Previous chat messages
-
-        Returns:
-            Generated answer
-        """
-        # Build context from chunks
-        context = "\n\n".join([f"[Document {i+1}]\n{chunk}" for i, chunk in enumerate(context_chunks)])
-
-        # Create system prompt optimized for LLM Judge evaluation
-        system_prompt = """You are an expert assistant specializing in SOCAR's historical oil and gas research documents.
-
-CRITICAL INSTRUCTIONS for high-quality answers:
-1. ACCURACY: Base your answer STRICTLY on the provided context - never add external information
-2. RELEVANCE: Answer the exact question asked - be direct and focused
-3. COMPLETENESS: Cover all key aspects mentioned in the context
-4. CITATIONS: Reference specific documents (e.g., "According to Document 1...")
-5. TECHNICAL PRECISION: Use correct oil & gas terminology from the documents
-6. CLARITY: Structure your answer logically - use bullet points for multiple items
-7. CONCISENESS: Be thorough but avoid redundancy or verbose explanations
-
-If the context lacks sufficient information, clearly state what is missing."""
-
-        # Build messages
-        messages = [{"role": "system", "content": system_prompt}]
-
-        # Add chat history if provided
-        if chat_history:
-            messages.extend(chat_history)
-
-        # Add current query with context
-        user_message = f"""Context from documents:
-{context}
-
-Question: {query}
-
-Provide a well-structured, accurate answer based ONLY on the context above. Include document citations."""
-
-        messages.append({"role": "user", "content": user_message})
-
-        # Optimized for quality (LLM Judge) while maintaining speed
-        return self.generate_response(messages, max_tokens=1000, temperature=0.2)
-
-
-# Singleton instance
-_deepseek_client = None
-
-
-def get_deepseek_client() -> DeepSeekClient:
-    """Get or create DeepSeek client instance"""
-    global _deepseek_client
-    if _deepseek_client is None:
-        _deepseek_client = DeepSeekClient()
-    return _deepseek_client

src/llm/rag_pipeline.py

@@ -1,154 +0,0 @@
-"""RAG (Retrieval Augmented Generation) pipeline"""
-
-from typing import List, Dict, Optional
-from loguru import logger
-
-from src.llm.deepseek_client import get_deepseek_client
-from src.vectordb import get_vector_store
-from src.api.models import SourceReference
-
-
-class RAGPipeline:
-    """RAG pipeline for document-based question answering"""
-
-    def __init__(self):
-        """Initialize RAG pipeline"""
-        self.llm = get_deepseek_client()
-        self.vector_store = get_vector_store()
-        logger.info("RAG pipeline initialized")
-
-    def query(
-        self,
-        question: str,
-        chat_history: Optional[List[Dict[str, str]]] = None,
-        n_results: int = 3,
-    ) -> Dict:
-        """
-        Answer a question using RAG
-
-        Args:
-            question: User's question
-            chat_history: Previous chat messages
-            n_results: Number of documents to retrieve
-
-        Returns:
-            Dict with 'answer' and 'sources'
-        """
-        logger.info(f"Processing query: {question[:100]}...")
-
-        # Step 1: Retrieve relevant documents
-        search_results = self.vector_store.search(question, n_results=n_results)
-
-        # Step 2: Format sources
-        sources = []
-        context_chunks = []
-
-        for doc, metadata in zip(search_results["documents"], search_results["metadatas"]):
-            sources.append(
-                SourceReference(
-                    pdf_name=metadata.get("pdf_name", "unknown.pdf"),
-                    page_number=metadata.get("page_number", 0),
-                    content=doc[:500],  # Limit content length
-                )
-            )
-            context_chunks.append(doc)
-
-        logger.info(f"Retrieved {len(sources)} source documents")
-
-        # Step 3: Generate answer using LLM
-        answer = self.llm.generate_with_context(
-            query=question,
-            context_chunks=context_chunks,
-            chat_history=chat_history,
-        )
-
-        return {
-            "answer": answer,
-            "sources": sources,
-        }
-
-    def add_processed_document(
-        self,
-        pdf_name: str,
-        pages: List[Dict[str, any]],
-        chunk_size: int = 600,
-        chunk_overlap: int = 100,
-    ):
-        """
-        Add a processed PDF to the vector store
-
-        Args:
-            pdf_name: Name of the PDF file
-            pages: List of page dicts with page_number and MD_text
-            chunk_size: Size of text chunks in characters
-            chunk_overlap: Overlap between chunks in characters
-        """
-        logger.info(f"Adding document to vector store: {pdf_name}")
-
-        texts = []
-        metadatas = []
-        ids = []
-
-        # Process each page
-        for page in pages:
-            page_num = page["page_number"]
-            text = page["MD_text"]
-
-            # Simple chunking by character count
-            chunks = self._chunk_text(text, chunk_size, chunk_overlap)
-
-            for chunk_idx, chunk in enumerate(chunks):
-                texts.append(chunk)
-                metadatas.append({
-                    "pdf_name": pdf_name,
-                    "page_number": page_num,
-                    "chunk_index": chunk_idx,
-                })
-                ids.append(f"{pdf_name}_p{page_num}_c{chunk_idx}")
-
-        # Add to vector store
-        self.vector_store.add_documents(texts, metadatas, ids)
-        logger.info(f"Added {len(texts)} chunks from {pdf_name}")
-
-    def _chunk_text(
-        self, text: str, chunk_size: int, chunk_overlap: int
-    ) -> List[str]:
-        """
-        Split text into overlapping chunks
-
-        Args:
-            text: Text to chunk
-            chunk_size: Size of each chunk
-            chunk_overlap: Overlap between chunks
-
-        Returns:
-            List of text chunks
-        """
-        if not text:
-            return []
-
-        chunks = []
-        start = 0
-
-        while start < len(text):
-            end = start + chunk_size
-            chunk = text[start:end]
-
-            if chunk.strip():
-                chunks.append(chunk)
-
-            start += chunk_size - chunk_overlap
-
-        return chunks
-
-
-# Singleton instance
-_rag_pipeline = None
-
-
-def get_rag_pipeline() -> RAGPipeline:
-    """Get or create RAG pipeline instance"""
-    global _rag_pipeline
-    if _rag_pipeline is None:
-        _rag_pipeline = RAGPipeline()
-    return _rag_pipeline

src/ocr/azure_ocr.py

@@ -1,143 +0,0 @@
-"""Azure Document Intelligence OCR processor"""
-
-from typing import List, Dict
-from pathlib import Path
-import io
-from loguru import logger
-
-from azure.ai.formrecognizer import DocumentAnalysisClient
-from azure.core.credentials import AzureKeyCredential
-import fitz  # PyMuPDF for image detection
-
-from src.config import settings
-
-
-class AzureOCRProcessor:
-    """Process PDFs using Azure Document Intelligence"""
-
-    def __init__(self):
-        """Initialize Azure Document Intelligence client"""
-        # Use Azure OpenAI endpoint as Document Intelligence endpoint
-        # In production, these might be different
-        endpoint = settings.azure_openai_endpoint.rstrip("/")
-        api_key = settings.azure_openai_api_key
-
-        self.client = DocumentAnalysisClient(
-            endpoint=endpoint, credential=AzureKeyCredential(api_key)
-        )
-
-        logger.info("Initialized Azure Document Analysis client")
-
-    def process_pdf(self, pdf_file: bytes, pdf_name: str = "document.pdf") -> List[Dict[str, any]]:
-        """
-        Process PDF and extract text + images using Azure Document Intelligence
-
-        Args:
-            pdf_file: PDF file as bytes
-            pdf_name: Name of the PDF file (for logging)
-
-        Returns:
-            List of dicts with page_number and MD_text (with inline images)
-        """
-        try:
-            logger.info(f"Processing PDF ({len(pdf_file)} bytes)")
-
-            # Analyze document using Azure Form Recognizer
-            poller = self.client.begin_analyze_document(
-                "prebuilt-read", document=io.BytesIO(pdf_file)
-            )
-            result = poller.result()
-
-            # Detect images using PyMuPDF (don't save, just mention)
-            doc_name = Path(pdf_name).stem  # Get filename without extension
-            images_by_page = self._detect_images(pdf_file, doc_name)
-
-            # Extract text page by page
-            pages_data = []
-            for page_num, page in enumerate(result.pages, start=1):
-                # Collect all lines from this page (PRESERVE CYRILLIC)
-                lines = []
-                if hasattr(page, 'lines') and page.lines:
-                    for line in page.lines:
-                        # Azure OCR preserves original encoding (Cyrillic stays Cyrillic)
-                        lines.append(line.content)
-
-                page_text = "\n".join(lines) if lines else ""
-
-                # Get image references for this page
-                page_images = images_by_page.get(page_num - 1, [])  # 0-indexed
-
-                # Only add image markdown if images exist
-                if page_images:
-                    for img_ref in page_images:
-                        # Create markdown image mention (no actual file)
-                        md_image = f'\n\n![Image]({img_ref})\n\n'
-                        page_text += md_image
-
-                    logger.info(f"Added {len(page_images)} image references for page {page_num}")
-
-                pages_data.append({
-                    "page_number": page_num,
-                    "MD_text": page_text
-                })
-
-            logger.info(f"Successfully processed {len(pages_data)} pages")
-            return pages_data
-
-        except Exception as e:
-            logger.error(f"Error processing PDF with Azure: {e}")
-            raise
-
-    def _detect_images(self, pdf_file: bytes, doc_name: str) -> Dict[int, List[str]]:
-        """
-        Detect images in PDF (don't save, just mention their presence)
-
-        Args:
-            pdf_file: PDF file as bytes
-            doc_name: Document name (without extension)
-
-        Returns:
-            Dict mapping page_number (0-indexed) to list of image references
-        """
-        images_by_page = {}
-
-        try:
-            # Open PDF with PyMuPDF
-            pdf_document = fitz.open(stream=pdf_file, filetype="pdf")
-
-            for page_num in range(len(pdf_document)):
-                page = pdf_document[page_num]
-                image_refs = []
-
-                # Get images from page
-                image_list = page.get_images()
-
-                # Only process if images exist on this page
-                if image_list:
-                    for img_index, img in enumerate(image_list):
-                        # Create simple reference: document_page_X_image_Y
-                        img_ref = f"{doc_name}_page_{page_num + 1}_image_{img_index + 1}"
-                        image_refs.append(img_ref)
-
-                    if image_refs:
-                        images_by_page[page_num] = image_refs
-                        logger.info(f"Detected {len(image_refs)} images on page {page_num + 1}")
-
-            pdf_document.close()
-
-        except Exception as e:
-            logger.warning(f"Could not detect images: {e}")
-
-        return images_by_page
-
-
-# Singleton instance
-_azure_ocr_processor = None
-
-
-def get_azure_ocr_processor() -> AzureOCRProcessor:
-    """Get or create Azure OCR processor instance"""
-    global _azure_ocr_processor
-    if _azure_ocr_processor is None:
-        _azure_ocr_processor = AzureOCRProcessor()
-    return _azure_ocr_processor

src/ocr/processor.py

@@ -1,62 +0,0 @@
-"""Main OCR processor that handles different backends"""
-
-from typing import List, Dict
-from pathlib import Path
-from loguru import logger
-
-from src.config import settings
-
-
-class OCRProcessor:
-    """Main OCR processor that can switch between different backends"""
-
-    def __init__(self, backend: str = None):
-        """
-        Initialize OCR processor
-
-        Args:
-            backend: OCR backend to use (azure)
-                    If None, uses settings.ocr_backend
-        """
-        self.backend = backend or settings.ocr_backend
-        logger.info(f"Initializing OCR processor with backend: {self.backend}")
-
-        # Initialize Azure OCR processor
-        if self.backend == "azure":
-            from src.ocr.azure_ocr import get_azure_ocr_processor
-            self.processor = get_azure_ocr_processor()
-        else:
-            raise ValueError(f"Unsupported OCR backend: {self.backend}. Only 'azure' is supported.")
-
-    def process_pdf(self, pdf_file: bytes, filename: str = None) -> List[Dict[str, any]]:
-        """
-        Process PDF file and extract text
-
-        Args:
-            pdf_file: PDF file as bytes
-            filename: Optional filename for logging
-
-        Returns:
-            List of dicts with page_number and MD_text
-        """
-        logger.info(f"Processing PDF: {filename or 'unnamed'} ({len(pdf_file)} bytes)")
-
-        try:
-            result = self.processor.process_pdf(pdf_file)
-            logger.info(f"Successfully processed {len(result)} pages")
-            return result
-        except Exception as e:
-            logger.error(f"Error processing PDF: {e}")
-            raise
-
-
-# Singleton instance
-_ocr_processor = None
-
-
-def get_ocr_processor() -> OCRProcessor:
-    """Get or create OCR processor instance"""
-    global _ocr_processor
-    if _ocr_processor is None:
-        _ocr_processor = OCRProcessor()
-    return _ocr_processor

src/vectordb/__init__.py

@@ -1,16 +0,0 @@
-"""Vector database factory and interface"""
-
-from src.config import settings
-
-
-def get_vector_store():
-    """Factory function to get the configured vector store"""
-    if settings.vector_db_type == "pinecone":
-        from src.vectordb.pinecone_store import get_vector_store as get_pinecone_store
-        return get_pinecone_store()
-    else:  # Default to chroma
-        from src.vectordb.chroma_store import get_vector_store as get_chroma_store
-        return get_chroma_store()
-
-
-__all__ = ["get_vector_store"]

src/vectordb/chroma_store.py

@@ -1,150 +0,0 @@
-"""ChromaDB vector store for document embeddings"""
-
-from typing import List, Dict, Optional
-from pathlib import Path
-import chromadb
-from chromadb.config import Settings
-from sentence_transformers import SentenceTransformer
-from loguru import logger
-
-from src.config import settings as app_settings
-
-
-class ChromaVectorStore:
-    """Vector store using ChromaDB"""
-
-    def __init__(self, collection_name: str = "socar_documents"):
-        """
-        Initialize ChromaDB vector store
-
-        Args:
-            collection_name: Name of the collection to use
-        """
-        # Initialize ChromaDB client
-        self.db_path = app_settings.vector_db_path
-        self.db_path.mkdir(parents=True, exist_ok=True)
-
-        self.client = chromadb.PersistentClient(
-            path=str(self.db_path),
-            settings=Settings(
-                anonymized_telemetry=False,
-                allow_reset=True,
-            ),
-        )
-
-        # Initialize embedding model
-        logger.info("Loading embedding model...")
-        self.embedding_model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
-        logger.info("Embedding model loaded")
-
-        # Get or create collection
-        self.collection = self.client.get_or_create_collection(
-            name=collection_name,
-            metadata={"description": "SOCAR historical documents"},
-        )
-
-        logger.info(f"ChromaDB initialized with collection: {collection_name}")
-        logger.info(f"Collection contains {self.collection.count()} documents")
-
-    def add_documents(
-        self,
-        texts: List[str],
-        metadatas: List[Dict],
-        ids: Optional[List[str]] = None,
-    ):
-        """
-        Add documents to the vector store
-
-        Args:
-            texts: List of text chunks to add
-            metadatas: List of metadata dicts (pdf_name, page_number, etc.)
-            ids: Optional list of document IDs
-        """
-        if not texts:
-            logger.warning("No texts provided to add")
-            return
-
-        # Generate IDs if not provided
-        if ids is None:
-            ids = [f"doc_{i}" for i in range(len(texts))]
-
-        logger.info(f"Adding {len(texts)} documents to vector store")
-
-        # Generate embeddings
-        embeddings = self.embedding_model.encode(texts, show_progress_bar=True)
-
-        # Add to ChromaDB
-        self.collection.add(
-            documents=texts,
-            embeddings=embeddings.tolist(),
-            metadatas=metadatas,
-            ids=ids,
-        )
-
-        logger.info(f"Successfully added {len(texts)} documents")
-
-    def search(
-        self,
-        query: str,
-        n_results: int = 5,
-        filter_metadata: Optional[Dict] = None,
-    ) -> Dict:
-        """
-        Search for similar documents
-
-        Args:
-            query: Search query
-            n_results: Number of results to return
-            filter_metadata: Optional metadata filter
-
-        Returns:
-            Dict with documents, metadatas, and distances
-        """
-        logger.info(f"Searching for: {query[:100]}...")
-
-        # Generate query embedding
-        query_embedding = self.embedding_model.encode([query])[0]
-
-        # Search ChromaDB
-        results = self.collection.query(
-            query_embeddings=[query_embedding.tolist()],
-            n_results=n_results,
-            where=filter_metadata,
-        )
-
-        logger.info(f"Found {len(results['documents'][0])} results")
-
-        return {
-            "documents": results["documents"][0],
-            "metadatas": results["metadatas"][0],
-            "distances": results["distances"][0],
-        }
-
-    def clear(self):
-        """Clear all documents from the collection"""
-        logger.warning("Clearing all documents from collection")
-        self.client.delete_collection(self.collection.name)
-        self.collection = self.client.create_collection(
-            name=self.collection.name,
-            metadata={"description": "SOCAR historical documents"},
-        )
-
-    def get_stats(self) -> Dict:
-        """Get collection statistics"""
-        return {
-            "total_documents": self.collection.count(),
-            "collection_name": self.collection.name,
-            "db_path": str(self.db_path),
-        }
-
-
-# Singleton instance
-_vector_store = None
-
-
-def get_vector_store() -> ChromaVectorStore:
-    """Get or create vector store instance"""
-    global _vector_store
-    if _vector_store is None:
-        _vector_store = ChromaVectorStore()
-    return _vector_store

src/vectordb/pinecone_store.py

@@ -1,176 +0,0 @@
-"""Pinecone vector store for document embeddings"""
-
-from typing import List, Dict, Optional
-from pinecone import Pinecone, ServerlessSpec
-from sentence_transformers import SentenceTransformer
-from loguru import logger
-import time
-
-from src.config import settings as app_settings
-
-
-class PineconeVectorStore:
-    """Vector store using Pinecone"""
-
-    def __init__(self, index_name: str = None):
-        """
-        Initialize Pinecone vector store
-
-        Args:
-            index_name: Name of the Pinecone index to use
-        """
-        # Initialize Pinecone client
-        self.pc = Pinecone(api_key=app_settings.pinecone_api_key)
-        self.index_name = index_name or app_settings.pinecone_index_name
-
-        # Initialize embedding model (matches Pinecone index: 1024 dimensions)
-        logger.info("Loading embedding model...")
-        self.embedding_model = SentenceTransformer("BAAI/bge-large-en-v1.5")
-        self.embedding_dimension = 1024  # bge-large-en-v1.5 dimension (matches Pinecone)
-        logger.info("Embedding model loaded")
-
-        # Get or create index
-        self._ensure_index_exists()
-        self.index = self.pc.Index(self.index_name)
-
-        logger.info(f"Pinecone initialized with index: {self.index_name}")
-        logger.info(f"Index stats: {self.index.describe_index_stats()}")
-
-    def _ensure_index_exists(self):
-        """Verify index exists"""
-        existing_indexes = [idx.name for idx in self.pc.list_indexes()]
-
-        if self.index_name not in existing_indexes:
-            logger.error(f"Pinecone index '{self.index_name}' not found!")
-            logger.error(f"Available indexes: {existing_indexes}")
-            raise ValueError(
-                f"Pinecone index '{self.index_name}' does not exist. "
-                f"Please create it first or check PINECONE_INDEX_NAME in .env"
-            )
-        logger.info(f"Connected to existing Pinecone index: {self.index_name}")
-
-    def add_documents(
-        self,
-        texts: List[str],
-        metadatas: List[Dict],
-        ids: Optional[List[str]] = None,
-    ):
-        """
-        Add documents to the vector store
-
-        Args:
-            texts: List of text chunks to add
-            metadatas: List of metadata dicts (pdf_name, page_number, etc.)
-            ids: Optional list of document IDs
-        """
-        if not texts:
-            logger.warning("No texts provided to add")
-            return
-
-        # Generate IDs if not provided
-        if ids is None:
-            ids = [f"doc_{i}_{int(time.time())}" for i in range(len(texts))]
-
-        logger.info(f"Adding {len(texts)} documents to Pinecone")
-
-        # Generate embeddings
-        embeddings = self.embedding_model.encode(texts, show_progress_bar=True)
-
-        # Prepare vectors for upsert
-        vectors = []
-        for i, (doc_id, embedding, text, metadata) in enumerate(zip(ids, embeddings, texts, metadatas)):
-            vectors.append({
-                "id": doc_id,
-                "values": embedding.tolist(),
-                "metadata": {
-                    **metadata,
-                    "text": text[:1000]  # Store first 1000 chars in metadata
-                }
-            })
-
-        # Upsert in batches of 100
-        batch_size = 100
-        for i in range(0, len(vectors), batch_size):
-            batch = vectors[i:i + batch_size]
-            self.index.upsert(vectors=batch)
-            logger.info(f"Upserted batch {i//batch_size + 1}/{(len(vectors)-1)//batch_size + 1}")
-
-        logger.info(f"Successfully added {len(texts)} documents to Pinecone")
-
-    def search(
-        self,
-        query: str,
-        n_results: int = 5,
-        filter_metadata: Optional[Dict] = None,
-    ) -> Dict:
-        """
-        Search for similar documents
-
-        Args:
-            query: Search query
-            n_results: Number of results to return
-            filter_metadata: Optional metadata filter
-
-        Returns:
-            Dict with documents, metadatas, and distances
-        """
-        logger.info(f"Searching Pinecone for: {query[:100]}...")
-
-        # Generate query embedding
-        query_embedding = self.embedding_model.encode([query])[0]
-
-        # Search Pinecone
-        results = self.index.query(
-            vector=query_embedding.tolist(),
-            top_k=n_results,
-            include_metadata=True,
-            filter=filter_metadata
-        )
-
-        # Extract results
-        documents = []
-        metadatas = []
-        distances = []
-
-        for match in results['matches']:
-            documents.append(match['metadata'].get('text', ''))
-            # Remove 'text' from metadata as it's already in documents
-            metadata = {k: v for k, v in match['metadata'].items() if k != 'text'}
-            metadatas.append(metadata)
-            distances.append(1 - match['score'])  # Convert similarity to distance
-
-        logger.info(f"Found {len(documents)} results")
-
-        return {
-            "documents": documents,
-            "metadatas": metadatas,
-            "distances": distances,
-        }
-
-    def clear(self):
-        """Clear all documents from the index"""
-        logger.warning("Deleting and recreating Pinecone index")
-        self.pc.delete_index(self.index_name)
-        self._ensure_index_exists()
-        self.index = self.pc.Index(self.index_name)
-
-    def get_stats(self) -> Dict:
-        """Get index statistics"""
-        stats = self.index.describe_index_stats()
-        return {
-            "total_documents": stats.get('total_vector_count', 0),
-            "index_name": self.index_name,
-            "dimension": self.embedding_dimension,
-        }
-
-
-# Singleton instance
-_vector_store = None
-
-
-def get_vector_store() -> PineconeVectorStore:
-    """Get or create Pinecone vector store instance"""
-    global _vector_store
-    if _vector_store is None:
-        _vector_store = PineconeVectorStore()
-    return _vector_store

start.sh

@@ -1,81 +0,0 @@
-#!/bin/bash
-
-# SOCAR Document Processing - Quick Start Script
-
-set -e
-
-echo "=================================="
-echo "SOCAR Document Processing System"
-echo "=================================="
-echo ""
-
-# Check if .env exists
-if [ ! -f .env ]; then
-    echo "❌ Error: .env file not found"
-    echo "Please create .env file with required credentials"
-    exit 1
-fi
-
-# Check if Docker is installed
-if ! command -v docker &> /dev/null; then
-    echo "❌ Error: Docker is not installed"
-    echo "Please install Docker: https://docs.docker.com/get-docker/"
-    exit 1
-fi
-
-# Check if Docker Compose is installed
-if ! command -v docker-compose &> /dev/null; then
-    echo "❌ Error: Docker Compose is not installed"
-    echo "Please install Docker Compose: https://docs.docker.com/compose/install/"
-    exit 1
-fi
-
-echo "✓ Prerequisites checked"
-echo ""
-
-# Create data directories
-mkdir -p data/pdfs data/vector_db data/processed
-echo "✓ Data directories created"
-echo ""
-
-# Build and start containers
-echo "🔨 Building Docker image..."
-docker-compose build
-
-echo ""
-echo "🚀 Starting containers..."
-docker-compose up -d
-
-echo ""
-echo "⏳ Waiting for service to be ready..."
-sleep 5
-
-# Wait for health check
-MAX_RETRIES=30
-RETRY_COUNT=0
-until curl -f http://localhost:8000/ &> /dev/null || [ $RETRY_COUNT -eq $MAX_RETRIES ]; do
-    echo "   Waiting for API... ($RETRY_COUNT/$MAX_RETRIES)"
-    sleep 2
-    ((RETRY_COUNT++))
-done
-
-if [ $RETRY_COUNT -eq $MAX_RETRIES ]; then
-    echo ""
-    echo "❌ Failed to start service"
-    echo "Check logs with: docker-compose logs"
-    exit 1
-fi
-
-echo ""
-echo "=================================="
-echo "✅ SOCAR API is ready!"
-echo "=================================="
-echo ""
-echo "📍 API URL: http://localhost:8000"
-echo "📖 Documentation: http://localhost:8000/docs"
-echo ""
-echo "Useful commands:"
-echo "  • View logs:    docker-compose logs -f"
-echo "  • Stop:         docker-compose down"
-echo "  • Restart:      docker-compose restart"
-echo ""

test_complete_system.py

@@ -1,128 +0,0 @@
-"""Complete system test"""
-
-import requests
-import json
-from pathlib import Path
-
-API_URL = "http://localhost:8000"
-
-def test_health():
-    """Test API health"""
-    print("=" * 60)
-    print("1. Testing API Health")
-    print("=" * 60)
-    response = requests.get(f"{API_URL}/")
-    print(f"Status: {response.status_code}")
-    print(json.dumps(response.json(), indent=2))
-    return response.status_code == 200
-
-
-def test_ocr():
-    """Test OCR endpoint"""
-    print("\n" + "=" * 60)
-    print("2. Testing OCR Endpoint")
-    print("=" * 60)
-
-    pdf_path = Path("data/pdfs/document_00.pdf")
-    if not pdf_path.exists():
-        print("❌ PDF not found")
-        return False
-
-    with open(pdf_path, "rb") as f:
-        files = {"file": (pdf_path.name, f, "application/pdf")}
-        response = requests.post(f"{API_URL}/ocr", files=files)
-
-    if response.status_code == 200:
-        result = response.json()
-        print(f"✓ Successfully processed {len(result)} pages")
-        print(f"  First page preview: {result[0]['MD_text'][:100]}...")
-        return True
-    else:
-        print(f"❌ Error: {response.status_code}")
-        return False
-
-
-def test_llm():
-    """Test LLM endpoint"""
-    print("\n" + "=" * 60)
-    print("3. Testing LLM Endpoint (RAG)")
-    print("=" * 60)
-
-    messages = [
-        {"role": "user", "content": "What geological formations are discussed?"}
-    ]
-
-    response = requests.post(
-        f"{API_URL}/llm",
-        json=messages,
-        headers={"Content-Type": "application/json"}
-    )
-
-    if response.status_code == 200:
-        result = response.json()
-        print(f"✓ Generated answer with {len(result['sources'])} sources")
-        print(f"\nAnswer preview:")
-        print(result["answer"][:300] + "...")
-        print(f"\nSources:")
-        for i, src in enumerate(result["sources"][:3], 1):
-            print(f"  [{i}] {src['pdf_name']} - Page {src['page_number']}")
-        return True
-    else:
-        print(f"❌ Error: {response.status_code}")
-        return False
-
-
-def test_llm_with_history():
-    """Test LLM with chat history"""
-    print("\n" + "=" * 60)
-    print("4. Testing LLM with Chat History")
-    print("=" * 60)
-
-    messages = [
-        {"role": "user", "content": "What is the South Caspian Basin?"},
-        {"role": "assistant", "content": "The South Caspian Basin is a sedimentary basin..."},
-        {"role": "user", "content": "Tell me more about its hydrocarbon potential."}
-    ]
-
-    response = requests.post(
-        f"{API_URL}/llm",
-        json=messages,
-        headers={"Content-Type": "application/json"}
-    )
-
-    if response.status_code == 200:
-        result = response.json()
-        print(f"✓ Generated contextual answer with chat history")
-        print(f"  Answer length: {len(result['answer'])} characters")
-        print(f"  Sources: {len(result['sources'])} documents")
-        return True
-    else:
-        print(f"❌ Error: {response.status_code}")
-        return False
-
-
-if __name__ == "__main__":
-    print("\n" + "🚀" * 30)
-    print("SOCAR Document Processing System - Complete Test")
-    print("🚀" * 30 + "\n")
-
-    results = []
-    results.append(("Health Check", test_health()))
-    results.append(("OCR Endpoint", test_ocr()))
-    results.append(("LLM Endpoint", test_llm()))
-    results.append(("LLM Chat History", test_llm_with_history()))
-
-    print("\n" + "=" * 60)
-    print("TEST SUMMARY")
-    print("=" * 60)
-    for name, passed in results:
-        status = "✓ PASS" if passed else "❌ FAIL"
-        print(f"{status:10} - {name}")
-
-    all_passed = all(r[1] for r in results)
-    print("\n" + ("🎉" if all_passed else "❌") * 30)
-    if all_passed:
-        print("ALL TESTS PASSED - System Ready for Hackathon!")
-    else:
-        print("Some tests failed - please review")
-    print(("🎉" if all_passed else "❌") * 30 + "\n")