init

.dockerignore

@@ -0,0 +1,66 @@
+# Python
+__pycache__/
+*.py[cod]
+*$py.class
+*.so
+.Python
+env/
+venv/
+ENV/
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+
+# Jupyter
+notebooks/
+*.ipynb
+*.ipynb_checkpoints/
+output/
+
+# Data files
+data/
+*.pdf
+*.csv
+*.json
+
+# Documentation
+docs/
+README.md
+*.md
+
+# Git
+.git/
+.gitignore
+.gitattributes
+
+# IDE
+.vscode/
+.idea/
+*.swp
+*.swo
+*~
+
+# OS
+.DS_Store
+Thumbs.db
+
+# Environment
+.env.local
+.env.development
+*.log
+
+# Docker
+docker-compose.override.yml
+Dockerfile.dev

.gitignore

@@ -1,6 +1,8 @@
 .claude
-/docs
-/data
+
+data/pdfs
+data/vector_db
+data/ai_track_data
 .env
 .env.local
 .env.development.local

DEPLOYMENT.md

@@ -0,0 +1,257 @@
+# SOCAR Hackathon - LLM API Deployment Guide
+
+## Overview
+
+Production-ready FastAPI service for SOCAR historical documents chatbot.
+
+**Configuration (Based on RAG Optimization Benchmark):**
+- **Model**: Llama-4-Maverick-17B-128E-Instruct-FP8 (Open-source)
+- **Embedding**: BAAI/bge-large-en-v1.5
+- **Retrieval**: Top-3 vanilla
+- **Prompt Strategy**: Citation-focused
+- **Performance**: 55.67% LLM Judge Score, 73.33% Citation Score, ~3.6s response time
+
+## Quick Start
+
+### Prerequisites
+- Docker and Docker Compose installed
+- `.env` file with API keys (see `.env.example`)
+
+### 1. Configure Environment
+
+```bash
+cp .env.example .env
+# Edit .env with your actual API keys:
+# - AZURE_OPENAI_API_KEY
+# - AZURE_OPENAI_ENDPOINT
+# - PINECONE_API_KEY
+# - PINECONE_INDEX_NAME
+```
+
+### 2. Build and Run with Docker
+
+```bash
+# Build the image
+docker-compose build
+
+# Start the service
+docker-compose up -d
+
+# Check logs
+docker-compose logs -f llm-api
+
+# Check health
+curl http://localhost:8000/health
+```
+
+### 3. Test the API
+
+```bash
+# Simple health check
+curl http://localhost:8000/
+
+# Test LLM endpoint
+curl -X POST http://localhost:8000/llm \
+  -H "Content-Type: application/json" \
+  -d '{
+    "messages": [
+      {"role": "user", "content": "Palçıq vulkanlarının təsir radiusu nə qədərdir?"}
+    ]
+  }'
+```
+
+## API Endpoints
+
+### GET `/`
+Root endpoint with service information.
+
+**Response:**
+```json
+{
+  "status": "healthy",
+  "service": "SOCAR LLM Chatbot",
+  "version": "1.0.0",
+  "model": "Llama-4-Maverick-17B (open-source)",
+  "configuration": {
+    "embedding": "BAAI/bge-large-en-v1.5",
+    "retrieval": "top-3 vanilla",
+    "prompt": "citation_focused",
+    "benchmark_score": "55.67%"
+  }
+}
+```
+
+### GET `/health`
+Detailed health check with service status.
+
+**Response:**
+```json
+{
+  "status": "healthy",
+  "pinecone": {
+    "connected": true,
+    "total_vectors": 1300
+  },
+  "azure_openai": "connected",
+  "embedding_model": "loaded"
+}
+```
+
+### POST `/llm`
+Main chatbot endpoint.
+
+**Request:**
+```json
+{
+  "messages": [
+    {"role": "user", "content": "Your question here"}
+  ],
+  "temperature": 0.2,
+  "max_tokens": 1000
+}
+```
+
+**Response:**
+```json
+{
+  "response": "Answer with citations...",
+  "sources": [
+    {
+      "pdf_name": "document_00.pdf",
+      "page_number": "5",
+      "relevance_score": "0.892"
+    }
+  ],
+  "response_time": 3.61,
+  "model": "Llama-4-Maverick-17B-128E-Instruct-FP8"
+}
+```
+
+## Development Mode
+
+### Run locally without Docker
+
+```bash
+# Install dependencies
+cd app
+pip install -r requirements.txt
+
+# Run with uvicorn
+uvicorn main:app --reload --host 0.0.0.0 --port 8000
+```
+
+### Access API documentation
+
+Once running, visit:
+- **Swagger UI**: http://localhost:8000/docs
+- **ReDoc**: http://localhost:8000/redoc
+
+## Production Deployment
+
+### Environment Variables
+
+Required in `.env`:
+```bash
+# Azure OpenAI
+AZURE_OPENAI_API_KEY=your_key_here
+AZURE_OPENAI_ENDPOINT=your_endpoint_here
+AZURE_OPENAI_API_VERSION=2024-08-01-preview
+
+# Pinecone
+PINECONE_API_KEY=your_key_here
+PINECONE_INDEX_NAME=hackathon
+```
+
+### Docker Commands
+
+```bash
+# Build
+docker-compose build --no-cache
+
+# Start in background
+docker-compose up -d
+
+# View logs
+docker-compose logs -f
+
+# Stop
+docker-compose down
+
+# Restart
+docker-compose restart
+
+# Remove everything
+docker-compose down -v
+```
+
+### Health Checks
+
+The Docker container includes automatic health checks:
+- **Interval**: 30 seconds
+- **Timeout**: 10 seconds
+- **Start period**: 40 seconds (for model loading)
+- **Retries**: 3
+
+### Monitoring
+
+```bash
+# Check container status
+docker-compose ps
+
+# View resource usage
+docker stats socar-llm-api
+
+# Check logs
+docker-compose logs --tail=100 llm-api
+```
+
+## Performance Optimization
+
+### Lazy Loading
+- Azure client, Pinecone index, and embedding model are lazy-loaded
+- First request may take longer (~5-10s for model loading)
+- Subsequent requests: ~3.6s average
+
+### Caching (Future)
+To improve performance, consider:
+- Redis for frequently asked questions
+- Embedding cache for common queries
+- Model quantization for faster inference
+
+## Troubleshooting
+
+### Container won't start
+```bash
+# Check logs
+docker-compose logs llm-api
+
+# Verify environment variables
+docker-compose config
+
+# Rebuild
+docker-compose build --no-cache
+```
+
+### API returns 500 errors
+- Check Azure OpenAI key and endpoint
+- Verify Pinecone connection
+- Check model deployment name matches
+
+### Slow responses
+- First request loads models (5-10s)
+- Subsequent requests should be ~3-4s
+- Check network connectivity to Azure/Pinecone
+
+## Architecture Score
+
+**Open-Source Stack (20% bonus):**
+- ✅ Llama-4-Maverick-17B (Open-source LLM)
+- ✅ BAAI/bge-large-en-v1.5 (Open-source embeddings)
+- ✅ FastAPI (Open-source framework)
+- ✅ Docker (Open-source deployment)
+
+**Total Architecture Score: Maximum 20% for hackathon!**
+
+## License
+
+Built for SOCAR Hackathon 2025

Dockerfile

@@ -0,0 +1,46 @@
+# SOCAR Hackathon - LLM Endpoint Dockerfile
+# Multi-stage build for optimized image size
+
+# Stage 1: Builder
+FROM python:3.10-slim as builder
+
+WORKDIR /app
+
+# Install build dependencies
+RUN apt-get update && apt-get install -y \
+    build-essential \
+    && rm -rf /var/lib/apt/lists/*
+
+# Copy requirements and install dependencies
+COPY app/requirements.txt .
+RUN pip install --no-cache-dir --user -r requirements.txt
+
+# Stage 2: Runtime
+FROM python:3.10-slim
+
+WORKDIR /app
+
+# Install runtime dependencies
+RUN apt-get update && apt-get install -y \
+    curl \
+    && rm -rf /var/lib/apt/lists/*
+
+# Copy Python dependencies from builder
+COPY --from=builder /root/.local /root/.local
+
+# Copy application code
+COPY app/ ./app/
+COPY .env.example .env
+
+# Add local bin to PATH
+ENV PATH=/root/.local/bin:$PATH
+
+# Expose port
+EXPOSE 8000
+
+# Health check
+HEALTHCHECK --interval=30s --timeout=10s --start-period=40s --retries=3 \
+    CMD curl -f http://localhost:8000/health || exit 1
+
+# Run the application
+CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]

app/__init__.py

@@ -0,0 +1,2 @@
+"""SOCAR Hackathon LLM API"""
+__version__ = "1.0.0"

app/main.py

@@ -0,0 +1,280 @@
+"""
+SOCAR Hackathon - LLM Chatbot Endpoint
+Optimized based on RAG benchmark results
+Best config: citation_focused + vanilla_k3 + Llama-4-Maverick
+"""
+
+import os
+import time
+from typing import List, Dict
+from pathlib import Path
+
+from fastapi import FastAPI, HTTPException
+from fastapi.middleware.cors import CORSMiddleware
+from pydantic import BaseModel
+from dotenv import load_dotenv
+from openai import AzureOpenAI
+from pinecone import Pinecone
+from sentence_transformers import SentenceTransformer
+
+# Load environment variables
+load_dotenv()
+
+# Initialize FastAPI app
+app = FastAPI(
+    title="SOCAR Historical Documents Chatbot",
+    description="RAG-based chatbot for SOCAR oil & gas historical documents",
+    version="1.0.0"
+)
+
+# CORS middleware
+app.add_middleware(
+    CORSMiddleware,
+    allow_origins=["*"],
+    allow_credentials=True,
+    allow_methods=["*"],
+    allow_headers=["*"],
+)
+
+# Initialize clients (lazy loading for faster startup)
+azure_client = None
+pinecone_index = None
+embedding_model = None
+
+
+def get_azure_client():
+    """Lazy load Azure OpenAI client"""
+    global azure_client
+    if azure_client is None:
+        azure_client = AzureOpenAI(
+            api_key=os.getenv("AZURE_OPENAI_API_KEY"),
+            api_version=os.getenv("AZURE_OPENAI_API_VERSION", "2024-08-01-preview"),
+            azure_endpoint=os.getenv("AZURE_OPENAI_ENDPOINT")
+        )
+    return azure_client
+
+
+def get_pinecone_index():
+    """Lazy load Pinecone index"""
+    global pinecone_index
+    if pinecone_index is None:
+        pc = Pinecone(api_key=os.getenv("PINECONE_API_KEY"))
+        pinecone_index = pc.Index(os.getenv("PINECONE_INDEX_NAME", "hackathon"))
+    return pinecone_index
+
+
+def get_embedding_model():
+    """Lazy load embedding model"""
+    global embedding_model
+    if embedding_model is None:
+        # Best performing model from benchmark
+        embedding_model = SentenceTransformer("BAAI/bge-large-en-v1.5")
+    return embedding_model
+
+
+# Request/Response models
+class ChatMessage(BaseModel):
+    role: str
+    content: str
+
+
+class ChatRequest(BaseModel):
+    messages: List[ChatMessage]
+    temperature: float = 0.2
+    max_tokens: int = 1000
+
+
+class ChatResponse(BaseModel):
+    response: str
+    sources: List[Dict[str, str]]
+    response_time: float
+    model: str
+
+
+def retrieve_documents(query: str, top_k: int = 3) -> List[Dict]:
+    """
+    Retrieve relevant documents from Pinecone vector database.
+    Best strategy from benchmark: vanilla top-3
+    """
+    index = get_pinecone_index()
+    embed_model = get_embedding_model()
+
+    # Generate query embedding
+    query_embedding = embed_model.encode(query).tolist()
+
+    # Search vector database
+    results = index.query(
+        vector=query_embedding,
+        top_k=top_k,
+        include_metadata=True
+    )
+
+    # Extract documents
+    documents = []
+    for match in results['matches']:
+        documents.append({
+            'pdf_name': match['metadata'].get('pdf_name', 'unknown.pdf'),
+            'page_number': match['metadata'].get('page_number', 0),
+            'content': match['metadata'].get('text', ''),
+            'score': match.get('score', 0.0)
+        })
+
+    return documents
+
+
+def generate_answer(query: str, documents: List[Dict], temperature: float = 0.2, max_tokens: int = 1000) -> tuple[str, float]:
+    """
+    Generate answer using best-performing configuration.
+    Model: Llama-4-Maverick-17B (open-source)
+    Prompt: citation_focused (best citation score: 73.33%)
+    """
+    client = get_azure_client()
+
+    # Build context from retrieved documents
+    context_parts = []
+    for i, doc in enumerate(documents, 1):
+        context_parts.append(
+            f"Sənəd {i} (Mənbə: {doc['pdf_name']}, Səhifə {doc['page_number']}):\n{doc['content']}"
+        )
+    context = "\n\n".join(context_parts)
+
+    # Citation-focused prompt (best performer: 55.67% score)
+    prompt = f"""Siz SOCAR-ın tarixi sənədlər üzrə mütəxəssis köməkçisisiniz.
+
+ÖNƏMLİ: Hər bir faktı mütləq mənbə ilə təsdiqləyin (PDF adı və səhifə nömrəsi).
+
+Kontekst:
+{context}
+
+Sual: {query}
+
+Cavab verərkən:
+1. Dəqiq faktlar yazın
+2. Hər faktı mənbə ilə göstərin: (PDF: fayl_adı.pdf, Səhifə: X)
+3. Kontekstdə olmayan məlumat əlavə etməyin"""
+
+    try:
+        start_time = time.time()
+
+        # Use Llama-4-Maverick (open-source, best performer)
+        response = client.chat.completions.create(
+            model="Llama-4-Maverick-17B-128E-Instruct-FP8",
+            messages=[{"role": "user", "content": prompt}],
+            temperature=temperature,
+            max_tokens=max_tokens
+        )
+
+        elapsed = time.time() - start_time
+        answer = response.choices[0].message.content
+
+        return answer, elapsed
+
+    except Exception as e:
+        raise HTTPException(status_code=500, detail=f"LLM Error: {str(e)}")
+
+
+@app.get("/")
+async def root():
+    """Health check endpoint"""
+    return {
+        "status": "healthy",
+        "service": "SOCAR LLM Chatbot",
+        "version": "1.0.0",
+        "model": "Llama-4-Maverick-17B (open-source)",
+        "configuration": {
+            "embedding": "BAAI/bge-large-en-v1.5",
+            "retrieval": "top-3 vanilla",
+            "prompt": "citation_focused",
+            "benchmark_score": "55.67%"
+        }
+    }
+
+
+@app.get("/health")
+async def health():
+    """Detailed health check"""
+    try:
+        # Check if services are initialized
+        index = get_pinecone_index()
+        stats = index.describe_index_stats()
+
+        return {
+            "status": "healthy",
+            "pinecone": {
+                "connected": True,
+                "total_vectors": stats.get('total_vector_count', 0)
+            },
+            "azure_openai": "connected",
+            "embedding_model": "loaded"
+        }
+    except Exception as e:
+        return {
+            "status": "degraded",
+            "error": str(e)
+        }
+
+
+@app.post("/llm", response_model=ChatResponse)
+async def llm_endpoint(request: ChatRequest):
+    """
+    LLM chatbot endpoint for SOCAR historical documents.
+
+    Uses RAG (Retrieval Augmented Generation) with:
+    - Embedding: BAAI/bge-large-en-v1.5
+    - Retrieval: Top-3 documents
+    - LLM: Llama-4-Maverick-17B (open-source)
+    - Prompt: Citation-focused
+
+    Expected performance:
+    - Response time: ~3.6s
+    - LLM Judge Score: 55.67%
+    - Citation Score: 73.33%
+    """
+    try:
+        # Extract the user's question (last message)
+        if not request.messages:
+            raise HTTPException(status_code=400, detail="No messages provided")
+
+        user_messages = [msg for msg in request.messages if msg.role == "user"]
+        if not user_messages:
+            raise HTTPException(status_code=400, detail="No user message found")
+
+        query = user_messages[-1].content
+
+        # Retrieve relevant documents
+        documents = retrieve_documents(query, top_k=3)
+
+        # Generate answer
+        answer, response_time = generate_answer(
+            query=query,
+            documents=documents,
+            temperature=request.temperature,
+            max_tokens=request.max_tokens
+        )
+
+        # Format sources
+        sources = [
+            {
+                "pdf_name": doc['pdf_name'],
+                "page_number": str(doc['page_number']),
+                "relevance_score": f"{doc['score']:.3f}"
+            }
+            for doc in documents
+        ]
+
+        return ChatResponse(
+            response=answer,
+            sources=sources,
+            response_time=round(response_time, 2),
+            model="Llama-4-Maverick-17B-128E-Instruct-FP8"
+        )
+
+    except HTTPException:
+        raise
+    except Exception as e:
+        raise HTTPException(status_code=500, detail=f"Error: {str(e)}")
+
+
+if __name__ == "__main__":
+    import uvicorn
+    uvicorn.run(app, host="0.0.0.0", port=8000)

notebooks/requirements_llm_benchmark.txt → app/requirements.txt

@@ -1,27 +1,24 @@
-# LLM Benchmarking Requirements
-# Install with: pip install -r requirements_llm_benchmark.txt
+# SOCAR Hackathon LLM Endpoint Dependencies
+# Optimized for production deployment
+
+# FastAPI and server
+fastapi==0.109.0
+uvicorn[standard]==0.27.0
+pydantic==2.5.3
 
 # Azure OpenAI client
 openai==1.54.0
 
-# Vector Database
+# Vector database
 pinecone-client==5.0.0
 
 # Embeddings
 sentence-transformers==3.3.1
-
-# Metrics
-jiwer==3.0.3
-
-# Data analysis and visualization
-pandas==2.1.3
-matplotlib==3.8.2
-seaborn==0.13.0
+torch==2.1.2
 
 # Utilities
 python-dotenv==1.0.0
-numpy==1.26.2
+python-multipart==0.0.6
 
-# Jupyter
-jupyter==1.0.0
-ipykernel==6.27.1
+# Optional: monitoring and logging
+prometheus-fastapi-instrumentator==7.0.0

data/dataset_info.json

@@ -0,0 +1,7 @@
+{
+  "total_rows": 28,
+  "features": [
+    "pdf"
+  ],
+  "description": "SOCAR AI Track Dataset with PDF documents"
+}

data/document_00.md

@@ -0,0 +1,252 @@
+**XÜLASƏ**
+
+Bu tədqiqat Aşağı Kür çökəkliyi (AKÇ) və Bakı arxipelaqı (BA) daxil olmaqla Cənubi Xəzər çökəkliyi sistemində faydalı qazıntıların mənşəyinin paleotektonik, paleocoğrafi şərait və geodinamik rejimlə necə əlaqələndiyini, eləcə də Gec Miosendən etibarən Ərəbistan plitəsinin təsiri ilə formalaşan kollizion proseslərin bölgənin struktur-morfoloji və termal inkişafına nə dərəcədə yönverici rol oynadığını kompleks şəkildə qiymətləndirir. Seismotektonik göstəricilərin, çöküntütoplanma sürətlərinin, geotemperatur xəritələrinin və palçıq vulkanizmi indikatorlarının inteqrasiyası göstərir ki, Cənubi Xəzər meqaçökəkliyinin qərb periklinal zonasının morfotektonik skeleti sıxılma gərginlikləri tərəfindən idarə olunmuş, AKÇ və BA-nın intensiv deformasiya olunması isə antiklinal zonallaşmanı, yerli qalxımların və qırıcı şəbəkəsinin hüdudlarını müəyyənləşdirmişdir. Nəticələrə görə, AKÇ-də antiklinal xətlər şimal-qərbdən cənub-şərqə uzanır və bu, şimal-şərq–cənub-qərb istiqamətli maksimal sıxılma gərginliklərinə perpendikulyar struktur elementlərinin üstünlüyünü təsdiqləyir. AKÇ-nin şimal-şərq seqmentində sıxılma gərginliklərinin intensivliyi pik dəyərlərə çatır; buradakı lokal qalxımlar həm seysmik kəsiklərdə, həm də səth-şelf morfologiyasında asimmetrik, "flower structure" tipli transpressiv motivlərlə ifadə olunur.
+
+---
+
+**Səhifə 2**
+
+Çöküntütoplanma sürətinin məkanca və zamanca dəyişkənliyi, xüsusən Cənubi Xəzərin mərkəzində 0.4 mm/il, şelfdə 3–4 mm/il və Kür çayının mənsəbində 6 mm/il-ə çatan göstəricilər, akkumulyasiya rejiminin geodinamik yüklənməyə həssas olduğunu göstərir. Çöküntü qalınlığının AKÇ mərkəzinə doğru 6–7 km-ə qədər artması subsidensiya-sıxılma balansının uzunmüddətli kinematikası ilə izah olunur. Geotemperatur modelləşdirmə və xəritələşdirmə neftəmələgəlmənin baş zonalarını AKÇ və BA üçün ayırd etmiş, BA-da 3000–4000 m intervalında neft, 8000–8500 m intervalında isə qaz əmələgəlməsinin pik zonalarını təsbit etmişdir. Palçıq vulkanizmi məcraları boyunca temperatur anomaliyalarının müşahidəsi, yerli termal axının artması və süxur kompleksləri daxilində maye-müxtəlif fazalı qarışıqların dərinlikdən gətirilməsi ilə əlaqələndirilmişdir. Nəticə olaraq, AKÇ və BA-nın neft-qazlılıq potensialının formalaşması birbaşa sıxılma gərginlikləri, qatlanma-qırılma kinematikası və çöküntütoplanma rejiminin sinxronlaşdırılması ilə bağlı olub; Pliosendə hər iki struktur domen üzrə inkişafın sürətlənməsi, akkumulyasiya-tektonika qarşılıqlı təsirinin ən yüksək səviyyəsini əks etdirir.
+
+**Açar sözlər**
+
+Cənubi Xəzər çökəkliyi; Aşağı Kür çökəkliyi; Bakı arxipelaqı; sıxılma gərginlikləri; paleotektonika; paleocoğrafiya; geodinamik rejim; palçıq vulkanizmi; çöküntütoplanma; neftəmələgəlmə; qazəmələgəlmə; geotemperatur modelləşdirmə.
+
+---
+
+**Səhifə 3**
+
+## **Giriş**
+
+Faydalı qazıntıların regional paylanması və genezisi əksər hallarda geodinamik çərçivə, paleotektonik epizodlar və paleocoğrafi təkamüllə sıx bağlıdır. Qafqaz orogeni və Cənubi Xəzər meqaçökəkliyi arasında yerləşən Aşağı Kür çökəkliyi (AKÇ) və Bakı arxipelaqı (BA) bu baxımdan unikal laboratorıyadır: burada Gec Miosendən bəri davam edən Ərəbistan-Avrasiya kolliziyası nəticəsində yaranmış sıxılma rejimi həm struktur-morfoloji, həm də neft-karbohidrogen sistemlərinin inkişafını güclü şəkildə yönləndirir. Bu işdə məqsəd:
+
+* AKÇ və BA ərazilərində sıxılma gərginliklərinin orientasiyası, intensivliyi və zamanca dəyişkənliyini qiymətləndirmək;
+* qatlanma-qırılma şəbəkəsinin geometriyasını paleostress sahəsi ilə əlaqələndirmək;
+* çöküntütoplanma sürətlərinin məkan-zaman variasiyalarını və onların subsidensiya balansına təsirini göstərmək;
+* geotemperatur xəritələri əsasında neft və qaz əmələgəlməsinin baş zonalarını müəyyənləşdirmək;
+* palçıq vulkanizminin termal və hidrodinamik siqnallarını, həmçinin süxur komplekslərinin temperatur rejiminə təsirini izah etmək;
+* mineral resursların (xüsusən neft-qaz) potensialını struktur-paleocoğrafi ramka daxilində inteqrasiya etmək.
+
+## **Regional geodinamik fon**
+
+Gec Miosendən etibarən Ərəbistan plitəsinin şimala doğru hərəkəti Qafqaz kollizion zonasını aktivləşdirmiş, Cənubi Xəzər meqaçökəkliyinin qərb kənarında transpressiv-sıxılmalı deformasiya rejimi yaratmışdır. Bu proses, Cənubi Xəzər dərin çökəkliyi altında yüksək sıxlıqlı litosfer blokunun şərti şəkildə udulması (subduksiya-vari tələffüz olunan udulma) ilə nəticələnmiş, qərb yamac boyunca pillələnmiş itələnmələr, sağ-yanal komponentli qırılmalar və ön zonada sıxılmalı qatlanma kəmərləri əmələ gətirmişdir. Regional stress sahəsinin maksimum üfüqi komponenti (SHmax) şimal-şərq–cənub-qərb istiqamətlidir; bu istiqamətdə sıxılma gərginliklərinin yüksəkliyi AKÇ-nin şimal-şərq seqmentində ən aydın şəkildə izlənir. Nəticə etibarilə, antiklinal oxlar və qırılma zonalarının dominant uzanma istiqaməti şimal-qərbdən cənub-şərqə doğrudur ki, bu da SHmax-ın ortoqonalını əks etdirir.
+
+---
+
+**Səhifə 4**
+
+## **Paleotektonik və paleocoğrafi kontekst**
+
+Paleocoğrafi rekonstruksiyalar göstərir ki, Neogen boyunca Kür hövzəsi-delta kompleksi, Abşeron şelfi və inteqrasiya olunmuş arxipelaq sistemində çöküntü fasiyaları deltaik-allüvial, prodelta-dənizli slam və dayaz dəniz karbonat-silis biogenləri arasında paylanmışdır. Gec Miosen–Pliosen dövründə tektono-eustatik səviyyə dalğalanmaları ilə sinxron proqradasiya-reqressiya mərhələləri çöküntü litoqrafiyasını zonallaşdırmış, BA xətti üzrə ritmik qalxımlar isə lokal akkumulyasiya baryerləri yaratmışdır. Paleotektonik təkamül, Ərəbistan plitəsinin itmə sürətindəki dəyişmələrlə birlikdə transpressiv komponentin güclənməsinə və qatlanma cəbhəsinin qərbə doğru miqrasiyasına yol açmış, AKÇ-də antiklinal məhəllələrin ardıcıl aktivləşməsi ilə müşayiət olunmuşdur.
+
+---
+
+## **Material və metodlar**
+
+### **Məlumat bazası**
+
+* Regional və yüksək ayırdetmə qabiliyyətli seysmik kəsiklərdən çıxarılan struktur xəritələr;
+* Quyu geofizikası (temperatur, qalınlıq, vitrinit əksoluunma, akustik sürət);
+* Palçıq vulkanlarının termal-fluid kimyası, qaz-izotop tərkibi və vent temperatur ölçmələri;
+* Çökmə fasiyalarının nümunə təhlili, dənə ölçüsü statistikası və çöküntütoplanma sürətlərinin stratiqrafik kalibrlənməsi;
+* Geodeziya-GNSS əsaslı müasir deformasiya sürətləri.
+
+### **Paleostress rekonstruksiyası**
+
+Qırılma müstəviləri, sürüşmə vektorları və qatlanma oxlarının istiqamətlərinə əsaslanan kinematik inversiya alqoritmi tətbiq edilmişdir. SHmax azimutu, sıxılma dərəcəsi və transpressiv komponentin nisbi payı ən kiçik kvadratlar metoduyla kalibr olunmuşdur.
+
+### **Çökmə-subsidensiya modelləşdirməsi**
+
+1D və 2D hövzə modelləri vasitəsilə çöküntütoplanma sürətləri, istilik axını və kompaksiya qanunları (tixotropluq və yükə bağlı porozite azalması) tətbiq edilmişdir. Dərinləşmə tarixi, su dərinliyi dəyişmələri və deltaların proqradasiya sürətləri daxil edilmişdir.
+
+### **Geotemperatur modelləşdirməsi**
+
+İstilik axını xəritələri, termal keçiricilik, radioaktiv istilik istehsalı və maye axını ilə bağlı sadələşdirilmiş kupləj modelləri istifadə olunmuşdur. Neftəmələgəlmə (60–120°C; bəzi üzvi maddə tiplərində 140°C-yə qədər), qazəmələgəlmə (120–200°C) intervalları, BA üçün quyu ölçüləri ilə kalibr olunmuşdur.
+
+---
+
+**Səhifə 5**
+
+## **Palçıq vulkanizmi indikatorları**
+
+Vulkan konuslarında vent temperaturu, suların xlorid-bor tərkibi, C1–C5 qaz paylanması və izotop fraksiyalanması qiymətləndirilmiş, temperatur anomal zonalarının lateral təsir radiusu empirik funksiya ilə (R ≈ 0.8–1.5 km) təxmini qiymətləndirilmişdir.
+
+---
+
+## **Nəticələr**
+
+### **Sıxılma gərginliklərinin məkan paylanması və intensivliyi**
+
+* AKÇ-nin şimal-şərq seqmentində SHmax ~35–45° azimutlu, müzdarib zonalarda effektiv sıxılma 12–18 MPa aralığındadır; cənub-qərbə doğru 7–10 MPa-a enir.
+* BA boyunca transpressiv zolaqda hissəvi sağ-yanal komponentli itələnmələr müşahidə olunur; qırılma müstəviləri 310–320° trendlidir.
+* Qırıcıların uzanma istiqaməti SHmax-a perpendikulyardır: dominant NW–SE uzanmalı tərs qırılmalar və onları kəsən NE–SW istiqamətli keçid qırılmaları üçbucaq zonaları təşkil edir.
+
+### **Struktur formalar: antiklinal zonallaşma və lokal qalxımlar**
+
+* AKÇ-də şimal-qərbdən cənub-şərqə uzanan antiklinal kəmərlər mərhələli şəkildə bir-birinə paralel düzülür; fold-propaqasiya tipli qatlanmaların ön hissəsində sürüşmə qırılmaları ilə birlikdə "fault-bend" mexanikası izlənir.
+* Lokal qalxımlar (məsələn, AKÇ-NE-1, AKÇ-NE-3 sintetik strukturları) morfoloji baxımdan asimmetrikdir: şimal-şərq qanadında kəskin, cənub-qərb qanadında isə daha yastı yamaclarla məhdudlaşır. Bu, sıxılma vektorunun yönü ilə uyğundur.
+* BA zolağında pozitiv "flower structure" motivləri müşahidə olunur; bu strukturların mərkəzində palçıq vulkanları və fluid ventləri sıx yerləşir.
+
+### **Çöküntütoplanma dinamikası və qalınlığı**
+
+* Cənubi Xəzərin mərkəz hissəsində çöküntütoplanma sürəti 0.4 mm/il olaraq qiymətləndirilmişdir; bu göstərici termal rejimin nisbi "soyuma" tendensiyası ilə uzlaşır.
+* Şelfdə 3–4 mm/il və Kür çayının mənsəbində 6 mm/il-ə çatan sürətlər proqradasiya kənarlarında yüksək akkumulyasiya potensialını göstərir.
+* Çöküntülərin qalınlığı AKÇ mərkəzinə doğru 6–7 km-ə çatır; bu artım, Pliosen dövründə (xüsusən erkən-orta Pliosen) sürətlənən subsidensiya ilə əlaqədardır.
+* AKÇ və BA-da inkişaf sürəti eyni deyildir: BA boyunca struktur yüksəlmələr erkən Miosendən zəif, Pliosendə isə kəskin intensivləşmişdir; AKÇ-də isə Gec Miosen–Pliosen aralığında tədricən artan, lakin Pliosenin ortasında sıçrayış xarakterli mərhələ müşahidə olunur.
+
+---
+
+**Səhifə 6**
+
+## **Termal rejim və hidrokarbon yetişməsi**
+
+* Geotemperatur xəritələrinin analizi AKÇ-də neftəmələgəlmənin baş zonalarını 2.8–4.2 km intervalında ayırd edir; bu, lito-fasiyal müxtəliflikdən asılı olaraq lokal 2.6–2.8 km-lik dəyazlaşma göstərə bilir.
+* BA-da 3000–4000 m intervalı neftəmələgəlmə baş zonasını, 8000–8500 m intervalı isə qazəmələgəlmə baş zonasını əhatə edir. Vitrinit əksoluuma (Ro) dəyərləri BA üçün müvafiq olaraq 0.7–1.0% (neft pəncərəsi) və 1.3–2.0% (qaz pəncərəsi) diapazonuna uyğun gəlir.
+* İstilik axını dəyərləri şelfdə 44–52 mW/m², AKÇ mərkəzi boyunca 28–36 mW/m² olaraq təxmini qiymətləndirilmişdir; palçıq vulkanlarının yaxınlığında 5–9 mW/m²-lik lokal artımlar qeydə alınmışdır.
+* Palçıq vulkanizmi süxur komplekslərinin temperaturunu lokallaşdırılmış şəkildə artırır: vent kənarında 8–15°C, 0.8–1.2 km lateral məsafədə 2–5°C artım müşahidə olunur; bu artımlar kerogenin termal yetişməsinə birbaşa təsir edən regional faktor olmasa da, maye köçürülməsini sürətləndirərək lokal "overmature" linzalar yarada bilir.
+
+## **Palçıq vulkanizmi və fluid sistemləri**
+
+* Palçıq vulkanlarının paylanması antiklinal krestdə kəsimlərlə üst-üstə düşür; dərin kök zonaları çox vaxt tərs qırılmaların kəsişmə nöqtələrinə bağlanır.
+* Qaz tərkibi əsasən metandır (C1/C2+ > 100), lakin qazəmələgəlmə pəncərəsinin dərin kənarlarında C2–C5 fraksiyalarının nisbi payı artır; δ¹³CH₄ dəyərləri –44‰ ilə –34‰ arasında dəyişir.
+* Vulkanların püskürmə dövrləri Pliosen–Kvarter çöküntülərinin sürüşmə səviyyələri ilə uyğundur və çoxkamüllü vent sistemləri termal anomaliyaları stasionar saxlayır.
+
+## **Mineral resursların genetik zonallaşması**
+
+* Faydalı qazıntılar (hidrokarbonlar başda olmaqla) paleotektonik zonallığa tabedir. NW–SE uzanmalı antiklinal kəmərlər boyunca tələlər sıxlaşır; qırılma-örtülü və qatlanma-örtülü tələlərin sinxronluğu ən yüksək ehtiyat sıxlığını təmin edir.
+* Qeyri-hidrokarbon resurslar (gil süxurları, tikinti materialları, yerli karbonat linzaları) deltaik kənarlarda və reqressiv sekansların yuxarı hissələrində cəmlənir.
+* Maykop tipli (Oliqosen–Aşağı Miosen) orqanik zəngin şistlər potensial mənbə süxuru rolunu oynayır, üstündəki Pliosen pelit kompleksləri effektiv örtük sistemi yaradır.
+
+---
+
+**Səhifə 7**
+
+## **Müzakirə**
+
+### **Kolliziya kinematikası və struktur miras**
+
+Ərəbistan plitəsinin şimala hərəkət sürətinin 15–20 mm/il diapazonunda dəyişməsi Qafqaz kollizion kəmərində basqı yaratmış, Cənubi Xəzərin qərb yamacı boyunca transpressiv büküləmələri induksiya etmişdir. Cənubi Xəzər meqaçökəkliyi altına "udulma" effekti, litosfer miqyasında sıxlıq kontrastlarının və közmogen plitə sərhədi geometriyasının nəticəsi kimi şərh olunur. Bu geodinamik quruluş AKÇ və BA-da aşağıdakılara gətirib çıxarmışdır:
+
+* Sıxılma gərginliklərinin şimal-şərqdə intensifikasiyası, antiklinal oxların NW–SE istiqamətdə ritmik sıralanması;
+* Qırılma sındırıcı sistemlərin SHmax-a perpendikulyar şəkildə düzülməsi və lokal "tilt block" dinamikasının yaranması;
+* Ön zonada qatlanma-kəsilmə cəbhələrinin mərhələli qərbə irəliləməsi.
+
+### **AKÇ və BA-nın fərqli inkişaf trayektoriyaları**
+
+AKÇ və BA inkişaf sürəti və rejimi baxımından fərqli davranış nümayiş etdirir. AKÇ, kollizion yükə həssas olaraq Pliosenin əvvəlindən ortalarınadək mərhələli artım göstərir, BA isə Pliosenin ortasında daha kəskin struktur yüksəlmələrlə seçilir. Bu asinxronluq, ehtimal ki, BA boyunca sağ-yanal komponentli transpressiyanın müqavimət kontrastlı litoloji paketlərlə rezonansına bağlıdır.
+
+Beləliklə:
+
+* AKÇ-də antiklinal zonalar boyunca tələlər uzunmüddətli akkumulyasiya-qatlanma sinxronluğu ilə "inkişaf etmiş tələ" mərhələsinə çatır;
+* BA-da isə gedişat "gec yetişmə–sürətli tələ formalaşması" ssenarisi ilə uyğun gəlir.
+
+### **Çöküntütoplanma–subsidensiya–istilik əlaqələri**
+
+Mərkəzi hövzədə 0.4 mm/il-lik akkumulyasiya fonda sabit, lakin şelf və delta kənarlarında 3–6 mm/il-ə çatan yüksək sürətlər, istilik axının inversiyada olan mozaika yaratmasına səbəb olur. Sürətli çöküntü yığılması müəyyən sahələrdə istilik diffuziyasını azaldaraq daha yüksək yeraltı temperatur qradientlərini çevik şəkildə formalaşdıra bilər, lakin Cənubi Xəzərin dərin zonalarında ümumi istilik axını nisbi aşağı dəyərlər göstərir. Bunun nəticəsi olaraq:
+
+* BA üçün 3000–4000 m intervalında neft pəncərəsi sabit izlənir, 8000–8500 m intervalında isə qaz pəncərəsi pikə çatır;
+* AKÇ-də isə neft pəncərəsi 2.8–4.2 km aralığında olsa da, lokal yüksək konduktiv zonalarda 2.6 km-ə dəyazlaşır.
+
+---
+
+**Səhifə 8**
+
+## **Palçıq vulkanizmi və termal təsirlər**
+
+Palçıq vulkanizmi dərin maye sistemləri üçün "sürətli magistral" funksiyası daşıyır. Vulkan kanallarından keçən isti materialların gətirdiyi əlavə istilik, ətraf süxurlarda qısamüddətli (10³–10⁴ il) termal anomaliyalar yaradır. Bu anomaliyalar hidrokarbonların termodinamik yetişməsi üçün regional mənə daşımır, ancaq:
+
+* Gec karbohidrogen miqrasiya hadisələrini sürətləndirir;
+* Yüksək məsaməlilikli, məsud konduktiv laylarda "şirin nöqtələr"in (sweet spot) lokal inkişafına kömək edir;
+* Vulkan konusları yaxınlığında səthi istilik axını ölçmələrinə 5–9 mW/m² artım şəklində yansıyır.
+
+## **Neft-qaz sistemi elementlərinin sinxronluğu**
+
+* **Mənbə süxuru:** Oliqosen–Aşağı Miosen Maykop kompleksləri, yüksək TOC (1.5–4.5%), kerogen tipləri II–III üstünlük təşkil edir.
+* **Örtük süxuru:** Pliosen pelitlər və marllar, aşağı permeabellik və qalın ardıcıllıqlar sayəsində effektiv sızdırmazlıq təmin edir.
+* **Kollektor layları:** Deltaik qumlar, dayaz-dəniz qumkarbonat paketlər; təsirli porozite 18–26%, keçiricilik 20–600 mD.
+* **Tələlər:** Antiklinal-örtülü tələlər, tərs qırılma seqmentləri ilə güclənmiş qatlanma tələləri; BA boyunca qalxım kənarlarında lateral örtülmə ilə kombinə olunmuş miks tip.
+
+## **Risklər və qeyri-müəyyənlik**
+
+* Sıxılma gərginliklərinin zamanca dəyişməsi nəticəsində qırılma şəbəkəsinin reaktivasiya ehtimalı yüksəkdir; tələlərdə sızma riskini artırır.
+* Palçıq vulkanizmi ilə bağlı epizodik "şok boşalmalar" tələlərdəki təzyiq rejimini dəyişərək lokal dekompressiya yarada bilər.
+* Geotemperatur modellərinin kalibrlənməsi istilik axını xəritələrinin seyrəkliyi səbəbindən qeyri-müəyyənlik daşıyır; əlavə ölçmələr tələb olunur.
+
+## **Alternativ modellər**
+
+* Dərin kök zonalarında litosferik miqyaslı "roll-back" mexanizminə bənzər arxitektonik yenidənqurulma ehtimalı, qərb yamacadaki asimmetrik qalxımları izah edə bilər.
+* Transpressiv zonalarda sağ-yanal komponentin payının artması, fold-thrust sistemlərinin "en echelon" tipində mərhələli təşkili ilə nəticələnmiş ola bilər; bu, BA xətti üzrə müşahidə olunan "flower structure" fəaliyyətini gücləndirir.
+
+---
+
+**Səhifə 9**
+
+## **Tətbiqi nəticələr və proqnoz**
+
+* Qaz pəncərəsi dərinliklərinin (8–8.5 km) BA üçün təsdiqi ultra-dərin hədəflərin (HP/HT rejim) texnoloji planlamasını tələb edir; buraxma temperaturu 170–210°C intervalı üçün risk qiymətləndirilməsi aparılmalıdır.
+* AKÇ-də neft pəncərəsinin 2.8–4.2 km intervalında sabit izlənməsi, antiklinal kəmərlər boyunca orta dərinlikli quyu dizaynı üçün əlverişli pəncərə yaradır; kollektor layların proqnozu üçün seysmik atribut analizinə üstünlük verilməlidir.
+* Palçıq vulkanizmi yaxınlığında yerləşən strukturlar üçün qaz sızması və geohazards risk modelləri hazırlanmalı, vent zonalarından uzaqlıq kriteriyası kimi ən azı 1.5–2.0 km radius saxlanmalıdır.
+
+## **Metodoloji müşahidələr**
+
+* Çöküntütoplanma sürətinin mərkəzi hövzədə 0.4 mm/il olması, qalın ardıcıllıqların formalaşması üçün uzunmüddətli zaman pəncərəsinin (~10 Myr) zəruriliyini göstərir; bu, subsidensiyanın tektonik komponentinin əhəmiyyətini vurğulayır.
+* Sıxılma rejiminin AKÇ-nin şimal-şərqində intensivləşməsi, ehtimal ki, yerli litosfer qalınlaşması və köhnə rift-mirası ilə (qırılma zonalarının irsi zəifləmələri) bağlıdır; bu, antiklinal oxların paralellik dərəcəsini artırır.
+
+## **Konseptual model**
+
+* **Gec Miosen:** Ərəbistan-Avrasiya kolliziyası aktivləşir; qərb yamac boyunca transpressiv stresslər, ilk qatlanma-qırılma cəbhəsi formalaşır.
+* **Erkən Pliosen:** Proqradasiya sürətlənir, Kür mənsəbi boyunca 6 mm/il-ə çatan akkumulyasiya, antiklinal zonalar aktivləşir, tələlərin ilkin doldurulması başlay��r.
+* **Orta Pliosen:** BA xətti boyunca lokal qalxımlar kəskinləşir; palçıq vulkanizmi epizodları artar, neft pəncərəsi geniş miqyasda aktivdir.
+* **Gec Pliosen–Erkən Kvarter:** Qaz pəncərəsi dərin hüdudlarda maksimum fəaliyyət göstərir; dərin kollektorlar qazla yüklənir; termal rejim sabitləşir.
+
+## **Gələcək işlər üçün çərçivə**
+
+* Yüksək ayırdetmə qabiliyyətli 3D seysmik-geomekanik kupləj modelləri ilə SHmax dinamizminin zaman içində izlənməsi;
+* Palçıq vulkanizmi ventlərinin termal-maye axını simulyasiyası və geotemperatur xəritələrində lokal düzəlişlər;
+* Çöküntütoplanma sürətlərinin paleomühit proksiləri (palinologiya, diatomit markerləri) ilə kalibrasiyası.
+
+---
+
+**Səhifə 10**
+
+## **Nəticələr**
+
+1. AKÇ və BA daxil olan Cənubi Xəzər çökəkliyi sistemində faydalı qazıntılar, xüsusılə neft-qaz sistemləri paleotektonik və paleocoğrafi şəraitin birgə məhsuludur; Gec Miosen kolliziyası ilə induksiya olunmuş sıxılma rejimi burada əsas idarəedici rol oynamışdır.
+2. Ərəbistan plitəsinin təsiri altında Qafqaz kolliziyalarının inkişafı Cənubi Xəzər meqaçökəkliyinin altına udulma mexanizmi ilə yoldaş olmuş, qərb yamacın formalaşmasında sıxılma gərginlikləri kolledici olmuşdur.
+3. AKÇ və Bakı arxipelaqı intensiv sıxılma gərginliklərindən təsirlənmiş, şimal-qərbdən cənub-şərqə uzanan antiklinal zonallaşma və SHmax-a perpendikulyar qırılma şəbəkəsi ilə xarakterizə olunmuşdur.
+4. AKÇ-nin lokal qalxımları morfoloji olaraq sıxılma gərginliklərinin təsirini əks etdirir; şimal-şərq seqmentində bu təsir yüksək intensivlikdədir.
+5. Çöküntütoplanma sürəti məkanca və zamanca dəyişir: Cənubi Xəzərin mərkəzində 0.4 mm/il, şelfdə 3–4 mm/il, Kür mənsəbində 6 mm/il; çöküntü qalınlığı AKÇ mərkəzinə doğru 6–7 km-ə çatır.
+6. Geotemperatur xəritələri AKÇ və BA üçün neftəmələgəlmə və qazəmələgəlmənin baş zonalarını ayırmışdır: BA-da 3000–4000 m (neft), 8000–8500 m (qaz); AKÇ-də isə neft pəncərəsi 2.8–4.2 km intervalında sabit izlənir.
+7. Palçıq vulkanizmi yerli termal rejimi yüksəldir və maye miqrasiyasını sürətləndirir; ancaq regional miqyasda termal yetişmənin əsas sürücüsü tektonik-sedimentoloji balansdır.
+8. AKÇ və BA-nın inkişaf sürəti eyni deyil; Pliosendə hər iki bölgədə artım müşahidə olunsa da, BA daha kəskin struktur yüksəlmələrlə fərqlənir.
+9. Neft-qazlılıq potensialı sıxılma gərginlikləri, qatlanma-qırılma morfodinamikası və akkumulyasiya rejiminin uyğunlaşması ilə maksimal səviyyəyə çatır; antiklinal-örtülü tələlər üstünlük təşkil edir.
+
+---
+
+**Səhifə 11**
+
+## **Tövsiyələr**
+
+* AKÇ-nin şimal-şərq seqmentində NW–SE uzanmalı antiklinal kəmərlər üzərində 3D seysmik atribut analizləri genişləndirilməli, qırılma-örtülü tələlərin konturları dəqiqləşdirilməlidir.
+* BA boyunca ultra-dərin qaz hədəfləri üçün HP/HT mühəndislik protokolları (geomekanik pəncərə, quyu soyutma-rejim planlaması) hazırlanmalıdır.
+* Palçıq vulkanlarına yaxın strukturların risk xəritələrində 2 km-lik təhlükəsizlik buffer zonası nəzərə alınmalı, vent aktivliyinin seysmik mikromüşahidələri aparılmalıdır.
+* Geotemperatur xəritələrinin kalibrlənməsi üçün əlavə istilik axını ölçmələri və fiber-optik quyu temperatur profilləri tətbiq olunmalıdır.
+* Çöküntütoplanma proqnozlarında Kür mənsəbi reaktivliyinin iqlim-hidroloji dəyişmələrə həssaslığı inteqrasiya olunmalıdır.
+
+---
+
+## **Əlavə müşahidələr və sintetik göstəricilər**
+
+* SHmax azimutu 035–045°; Shmin azimutu 125–135°; qırılma düzənlikləri: 310–320° trend.
+* Qatlanma amplitudları: 120–650 m; dalğa uzunluğu 2.5–7.8 km, asimmetriya əmsalı 1.2–1.8.
+* Termal qradientlər: şelfdə 26–31°C/km, AKÇ mərkəzində 17–22°C/km, BA dərin zonalarında 15–19°C/km.
+* Kollektor porozitesi: 18–26%; keçiricilik: 20–600 mD; kapilyar giriş təzyiqi: 0.3–1.1 MPa.
+* Palçıq vulkanı vent temperaturu: 24–55°C; lateral termal təsir radiusu 0.8–1.5 km.
+
+---
+
+**Səhifə 12**
+
+## **Yekun**
+
+Bu tədqiqat göstərir ki, Cənubi Xəzər çökəkliyi sistemində faydalı qazıntıların mənşə mexanizmləri paleotektonik və paleocoğrafi amillərin inteqrasiyasına həssasdır. Ərəbistan plitəsinin kollizion təsiri ilə yaranmış sıxılma rejimi, AKÇ və BA üzrə antiklinal zonallaşmanı, qırılma şəbəkələrini və lokal qalxımları formalaşdıraraq neft-qaz sistemlərinin bütün komponentlərinin sinxron fəaliyyətini təmin etmişdir. Çöküntütoplanma sürətlərinin dəyişkənliyi, termal rejimin region üzrə mozaik xarakter almasına səbəb olmuş; geotemperatur xəritələri isə neft və qaz əmələgəlməsinin baş zonalarını etibarlı şəkildə ay��rd etmişdir. Palçıq vulkanizmi, hər nə qədər regional termogenezis üçün ikincil faktor olsa da, lokal istilik və maye axını anomaliyaları ilə tələlərin doldurulmasına dinamika qatmışdır. Pliosendə müşahidə olunan inkişaf sıçrayışları, akkumulyasiya-tektonika kəsişməsinin pik mərhələsi kimi şərh olunur və bu mərhələnin izləri bu gün də struktur-morfoloji naxışlarda, termal xəritələrdə və palçıq vulkanizminin fəallığında oxunaqdadır. Bu çərçivə daxilində aparılacaq istiqamətli kəşfiyyat və təhlükə qiymətləndirməsi, AKÇ və BA-nın neft-qazlılıq potensialından daha effektiv istifadə edilməsinə imkan verəcək.

docker-compose.yml

@@ -0,0 +1,30 @@
+version: '3.8'
+
+services:
+  llm-api:
+    build:
+      context: .
+      dockerfile: Dockerfile
+    container_name: socar-llm-api
+    ports:
+      - "8000:8000"
+    env_file:
+      - .env
+    environment:
+      - PYTHONUNBUFFERED=1
+    volumes:
+      # Mount app directory for development (optional - remove in production)
+      - ./app:/app/app
+    restart: unless-stopped
+    healthcheck:
+      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 40s
+    networks:
+      - socar-network
+
+networks:
+  socar-network:
+    driver: bridge

notebooks/llm_benchmark.ipynb

@@ -1,761 +1,531 @@
 {
- "cells": [
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# LLM Benchmarking for SOCAR Hackathon RAG Chatbot\n",
-    "\n",
-    "This notebook tests different LLM models for the `/llm` endpoint to find the best performer.\n",
-    "\n",
-    "## Evaluation Criteria (LLM Judge Metrics):\n",
-    "- **Accuracy**: Is the answer correct?\n",
-    "- **Relevance**: Are retrieved citations relevant?\n",
-    "- **Completeness**: Does it fully answer the question?\n",
-    "- **Citation Quality**: Proper sources with page numbers?\n",
-    "- **Response Time**: Speed of generation\n",
-    "\n",
-    "## Available LLM Models:\n",
-    "1. **Llama-4-Maverick-17B-128E-Instruct-FP8** (Current choice, open-source)\n",
-    "2. **DeepSeek-R1** (Open-source reasoning model)\n",
-    "3. **GPT-4.1** (Strong general performance)\n",
-    "4. **GPT-5, GPT-5-mini**\n",
-    "5. **Claude Sonnet 4.5** (Best quality)\n",
-    "6. **Claude Opus 4.1**\n",
-    "7. **Phi-4-multimodal-instruct**\n",
-    "8. **gpt-oss-120b**"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Install required packages\n",
-    "# !pip install openai pinecone-client sentence-transformers python-dotenv pandas matplotlib seaborn jiwer"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "metadata": {},
-   "outputs": [
+  "cells": [
     {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "/Users/ismatsamadov/SOCAR_Hackathon/venv/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
-      "  from .autonotebook import tqdm as notebook_tqdm\n"
-     ]
+      "cell_type": "markdown",
+      "id": "38a7900c",
+      "metadata": {},
+      "source": [
+        "# LLM Benchmarking for SOCAR Hackathon RAG Chatbot\n",
+        "\n",
+        "Testing different LLM models for the `/llm` endpoint to find the best performer.\n",
+        "\n",
+        "## Evaluation Criteria (LLM Judge Metrics):\n",
+        "- **Accuracy**: Is the answer correct?\n",
+        "- **Relevance**: Are retrieved citations relevant?\n",
+        "- **Completeness**: Does it fully answer the question?\n",
+        "- **Citation Quality**: Proper sources with page numbers?\n",
+        "- **Response Time**: Speed of generation\n",
+        "\n",
+        "## Available LLM Models:\n",
+        "1. **Llama-4-Maverick-17B** (Open-source)\n",
+        "2. **DeepSeek-R1** (Open-source reasoning)\n",
+        "3. **GPT-4.1, GPT-5, GPT-5-mini**\n",
+        "4. **Claude Sonnet 4.5**"
+      ]
     },
     {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "\u2705 Libraries loaded successfully\n"
-     ]
-    }
-   ],
-   "source": [
-    "import os\n",
-    "import json\n",
-    "import time\n",
-    "from typing import Dict, List, Tuple\n",
-    "from dotenv import load_dotenv\n",
-    "import pandas as pd\n",
-    "import matplotlib.pyplot as plt\n",
-    "import seaborn as sns\n",
-    "from openai import AzureOpenAI\n",
-    "from pinecone import Pinecone\n",
-    "from sentence_transformers import SentenceTransformer\n",
-    "from jiwer import wer, cer\n",
-    "\n",
-    "# Load environment variables\n",
-    "load_dotenv()\n",
-    "\n",
-    "# Set style\n",
-    "sns.set_style('whitegrid')\n",
-    "plt.rcParams['figure.figsize'] = (14, 8)\n",
-    "\n",
-    "print(\"\u2705 Libraries loaded successfully\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "metadata": {},
-   "outputs": [
+      "cell_type": "code",
+      "execution_count": 45,
+      "id": "143cf60d",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# Install required packages\n",
+        "# !pip install openai pinecone-client sentence-transformers python-dotenv pandas matplotlib seaborn jiwer"
+      ]
+    },
     {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "\u2705 Project root: /Users/ismatsamadov/SOCAR_Hackathon\n",
-      "\u2705 Docs directory: /Users/ismatsamadov/SOCAR_Hackathon/docs\n",
-      "\u2705 Output directory: /Users/ismatsamadov/SOCAR_Hackathon/output\n"
-     ]
-    }
-   ],
-   "source": [
-    "# Auto-detect project root (works from any directory)\n",
-    "import os\n",
-    "from pathlib import Path\n",
-    "\n",
-    "if Path('data').exists() and Path('docs').exists():\n",
-    "    # Already in project root\n",
-    "    PROJECT_ROOT = Path.cwd()\n",
-    "elif Path('../data').exists() and Path('../docs').exists():\n",
-    "    # In notebooks/ subdirectory\n",
-    "    PROJECT_ROOT = Path.cwd().parent\n",
-    "else:\n",
-    "    # Fallback: try to find project root\n",
-    "    current = Path.cwd()\n",
-    "    while current != current.parent:\n",
-    "        if (current / 'data').exists() and (current / 'docs').exists():\n",
-    "            PROJECT_ROOT = current\n",
-    "            break\n",
-    "        current = current.parent\n",
-    "    else:\n",
-    "        PROJECT_ROOT = Path.cwd()\n",
-    "\n",
-    "# Define all paths relative to project root\n",
-    "DATA_DIR = PROJECT_ROOT / 'data'\n",
-    "DOCS_DIR = PROJECT_ROOT / 'docs'\n",
-    "OUTPUT_DIR = PROJECT_ROOT / 'output'\n",
-    "\n",
-    "print(f\"\u2705 Project root: {PROJECT_ROOT}\")\n",
-    "print(f\"\u2705 Docs directory: {DOCS_DIR}\")\n",
-    "print(f\"\u2705 Output directory: {OUTPUT_DIR}\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 4,
-   "metadata": {},
-   "outputs": [
+      "cell_type": "code",
+      "execution_count": 46,
+      "id": "d698b11a",
+      "metadata": {},
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "✅ Libraries loaded\n"
+          ]
+        }
+      ],
+      "source": [
+        "import os\n",
+        "import json\n",
+        "import time\n",
+        "from typing import Dict, List, Tuple\n",
+        "from dotenv import load_dotenv\n",
+        "import pandas as pd\n",
+        "import matplotlib.pyplot as plt\n",
+        "import seaborn as sns\n",
+        "from openai import AzureOpenAI\n",
+        "from pinecone import Pinecone\n",
+        "from sentence_transformers import SentenceTransformer\n",
+        "from jiwer import wer, cer\n",
+        "from pathlib import Path\n",
+        "\n",
+        "load_dotenv()\n",
+        "sns.set_style(\"whitegrid\")\n",
+        "plt.rcParams[\"figure.figsize\"] = (14, 8)\n",
+        "\n",
+        "print(\"✅ Libraries loaded\")"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 47,
+      "id": "087187fb",
+      "metadata": {},
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "✅ Project root: /Users/ismatsamadov/SOCAR_Hackathon\n",
+            "✅ Docs directory: /Users/ismatsamadov/SOCAR_Hackathon/docs\n"
+          ]
+        }
+      ],
+      "source": [
+        "# Auto-detect project root\n",
+        "if Path(\"data\").exists() and Path(\"docs\").exists():\n",
+        "    PROJECT_ROOT = Path.cwd()\n",
+        "elif Path(\"../data\").exists() and Path(\"../docs\").exists():\n",
+        "    PROJECT_ROOT = Path.cwd().parent\n",
+        "else:\n",
+        "    current = Path.cwd()\n",
+        "    while current != current.parent:\n",
+        "        if (current / \"data\").exists() and (current / \"docs\").exists():\n",
+        "            PROJECT_ROOT = current\n",
+        "            break\n",
+        "        current = current.parent\n",
+        "    else:\n",
+        "        PROJECT_ROOT = Path.cwd()\n",
+        "\n",
+        "DATA_DIR = PROJECT_ROOT / \"data\"\n",
+        "DOCS_DIR = PROJECT_ROOT / \"docs\"\n",
+        "OUTPUT_DIR = PROJECT_ROOT / \"output\"\n",
+        "\n",
+        "print(f\"✅ Project root: {PROJECT_ROOT}\")\n",
+        "print(f\"✅ Docs directory: {DOCS_DIR}\")"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 48,
+      "id": "cf51bb3f",
+      "metadata": {},
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "Loaded 5 test cases\n"
+          ]
+        }
+      ],
+      "source": [
+        "# Load sample questions and answers using dynamic paths\n",
+        "with open(DOCS_DIR / \"sample_questions.json\", \"r\", encoding=\"utf-8\") as f:\n",
+        "    questions = json.load(f)\n",
+        "\n",
+        "with open(DOCS_DIR / \"sample_answers.json\", \"r\", encoding=\"utf-8\") as f:\n",
+        "    expected_answers = json.load(f)\n",
+        "\n",
+        "print(f\"Loaded {len(questions)} test cases\")"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 49,
+      "id": "9e761174",
+      "metadata": {},
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "✅ Vector DB connected\n",
+            "✅ Embedding model loaded\n"
+          ]
+        }
+      ],
+      "source": [
+        "# Initialize Pinecone\n",
+        "pc = Pinecone(api_key=os.getenv(\"PINECONE_API_KEY\"))\n",
+        "index = pc.Index(os.getenv(\"PINECONE_INDEX_NAME\", \"hackathon\"))\n",
+        "\n",
+        "# Initialize embedding model\n",
+        "embed_model = SentenceTransformer(\"BAAI/bge-large-en-v1.5\")\n",
+        "\n",
+        "print(f\"✅ Vector DB connected\")\n",
+        "print(f\"✅ Embedding model loaded\")"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 50,
+      "id": "74396795",
+      "metadata": {},
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "✅ Retrieval function ready\n"
+          ]
+        }
+      ],
+      "source": [
+        "def retrieve_documents(query: str, top_k: int = 3) -> List[Dict]:\n",
+        "    \"\"\"Retrieve relevant documents from vector database.\"\"\"\n",
+        "    query_embedding = embed_model.encode(query).tolist()\n",
+        "    \n",
+        "    results = index.query(\n",
+        "        vector=query_embedding,\n",
+        "        top_k=top_k,\n",
+        "        include_metadata=True\n",
+        "    )\n",
+        "    \n",
+        "    documents = []\n",
+        "    for match in results[\"matches\"]:\n",
+        "        documents.append({\n",
+        "            \"pdf_name\": match[\"metadata\"].get(\"pdf_name\", \"unknown.pdf\"),\n",
+        "            \"page_number\": match[\"metadata\"].get(\"page_number\", 0),\n",
+        "            \"content\": match[\"metadata\"].get(\"text\", \"\"),\n",
+        "            \"score\": match.get(\"score\", 0.0)\n",
+        "        })\n",
+        "    \n",
+        "    return documents\n",
+        "\n",
+        "print(\"✅ Retrieval function ready\")"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 57,
+      "id": "2bfcc6fb",
+      "metadata": {},
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "✅ Configured 3 LLM models\n"
+          ]
+        }
+      ],
+      "source": [
+        "# Initialize Azure OpenAI\n",
+        "azure_client = AzureOpenAI(\n",
+        "    api_key=os.getenv(\"AZURE_OPENAI_API_KEY\"),\n",
+        "    api_version=os.getenv(\"AZURE_OPENAI_API_VERSION\", \"2024-08-01-preview\"),\n",
+        "    azure_endpoint=os.getenv(\"AZURE_OPENAI_ENDPOINT\")\n",
+        ")\n",
+        "\n",
+        "LLM_MODELS = {\n",
+        "    \"Llama-4-Maverick\": \"Llama-4-Maverick-17B-128E-Instruct-FP8\",\n",
+        "    \"DeepSeek-R1\": \"DeepSeek-R1\",\n",
+        "    \"GPT-4.1\": \"gpt-4.1\",\n",
+        "    # \"GPT-5-mini\": \"gpt-5-mini\"\n",
+        "    # \"Claude-Sonnet-4.5\": \"claude-sonnet-4-5\"  # Not available in Azure deployment\n",
+        "}\n",
+        "\n",
+        "print(f\"✅ Configured {len(LLM_MODELS)} LLM models\")"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 61,
+      "id": "ddedd503",
+      "metadata": {},
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "✅ Generation function ready\n"
+          ]
+        }
+      ],
+      "source": [
+        "def generate_answer(model_name: str, query: str, documents: List[Dict],\n",
+        "                   temperature: float = 0.2, max_tokens: int = 1000) -> Tuple[str, float]:\n",
+        "    \"\"\"Generate answer using specified LLM model.\"\"\"\n",
+        "    context_parts = []\n",
+        "    for i, doc in enumerate(documents, 1):\n",
+        "        context_parts.append(\n",
+        "            f\"Document {i} (Source: {doc['pdf_name']}, Page {doc['page_number']}):\\n{doc['content']}\"\n",
+        "        )\n",
+        "    context = \"\\n\\n\".join(context_parts)\n",
+        "    \n",
+        "    prompt = f\"\"\"Siz SOCAR-ın tarixi neft və qaz sənədləri üzrə mütəxəssis köməkçisisiniz.\n",
+        "\n",
+        "Kontekst:\n",
+        "{context}\n",
+        "\n",
+        "Sual: {query}\n",
+        "\n",
+        "Ətraflı cavab verin və mütləq sənəd mənbələrinə istinad edin.\"\"\"\n",
+        "    \n",
+        "    deployment = LLM_MODELS[model_name]\n",
+        "    \n",
+        "    try:\n",
+        "        start_time = time.time()\n",
+        "        \n",
+        "        # GPT-5 models use max_completion_tokens, others use max_tokens\n",
+        "        if deployment.startswith(\"gpt-5\"):\n",
+        "            response = azure_client.chat.completions.create(\n",
+        "                model=deployment,\n",
+        "                messages=[{\"role\": \"user\", \"content\": prompt}],\n",
+        "                temperature=temperature,\n",
+        "                max_completion_tokens=max_tokens\n",
+        "            )\n",
+        "        else:\n",
+        "            response = azure_client.chat.completions.create(\n",
+        "                model=deployment,\n",
+        "                messages=[{\"role\": \"user\", \"content\": prompt}],\n",
+        "                temperature=temperature,\n",
+        "                max_tokens=max_tokens\n",
+        "            )\n",
+        "        \n",
+        "        response_time = time.time() - start_time\n",
+        "        answer = response.choices[0].message.content\n",
+        "        return answer, response_time\n",
+        "    \n",
+        "    except Exception as e:\n",
+        "        return f\"ERROR: {str(e)}\", 0.0\n",
+        "\n",
+        "print(\"✅ Generation function ready\")"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 62,
+      "id": "946b0e30",
+      "metadata": {},
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "✅ Evaluation functions ready\n"
+          ]
+        }
+      ],
+      "source": [
+        "def evaluate_answer(expected: str, generated: str, documents: List[Dict]) -> Dict:\n",
+        "    \"\"\"Evaluate answer quality.\"\"\"\n",
+        "    # Normalize text\n",
+        "    def normalize(text):\n",
+        "        return text.lower().strip()\n",
+        "    \n",
+        "    # Calculate similarity\n",
+        "    if expected:\n",
+        "        wer_score = wer(normalize(expected), normalize(generated)) * 100\n",
+        "        similarity = max(0, 100 - wer_score)\n",
+        "    else:\n",
+        "        similarity = 0\n",
+        "    \n",
+        "    # Check citations\n",
+        "    pdf_names = [doc[\"pdf_name\"].replace(\".pdf\", \"\") for doc in documents]\n",
+        "    cited_pdfs = sum(1 for pdf in pdf_names if pdf in generated)\n",
+        "    citation_score = (cited_pdfs / len(pdf_names)) * 100 if pdf_names else 0\n",
+        "    \n",
+        "    # Completeness\n",
+        "    word_count = len(generated.split())\n",
+        "    completeness = min(100, (word_count / 50) * 100)\n",
+        "    \n",
+        "    return {\n",
+        "        \"Similarity\": round(similarity, 2),\n",
+        "        \"Citation_Score\": round(citation_score, 2),\n",
+        "        \"Completeness\": round(completeness, 2),\n",
+        "        \"Quality_Score\": round((similarity * 0.4 + citation_score * 0.4 + completeness * 0.2), 2)\n",
+        "    }\n",
+        "\n",
+        "print(\"✅ Evaluation functions ready\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "319459ce",
+      "metadata": {},
+      "source": [
+        "## Run LLM Benchmark"
+      ]
+    },
     {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Loaded 5 test cases\n",
-      "\n",
-      "Test Questions:\n",
-      "1. Example1: Daha az quyu il\u0259 daha \u00e7ox hasilat \u0259ld\u0259 etm\u0259k \u00fc\u00e7\u00fcn hans\u0131 \u0259sas amill\u0259rin inteqrasiyas\u0131 t\u0259l\u0259b olunur?...\n",
-      "2. Example2: Q\u0259rbi Ab\u015feron yata\u011f\u0131nda suvurma t\u0259dbirl\u0259ri hans\u0131 tarixd\u0259 v\u0259 hans\u0131 layda t\u0259tbiq edilmi\u015fdir v\u0259 bunun m...\n",
-      "3. Example3: Pirallah\u0131 strukturunda 1253 n\u00f6mr\u0259li quyudan g\u00f6t\u00fcr\u00fclm\u00fc\u015f n\u00fcmun\u0259l\u0259rd\u0259 SiO2 v\u0259 CaO oksidl\u0259ri aras\u0131nda ha...\n",
-      "4. Example4: Bak\u0131 arxipelaq\u0131 (BA) v\u0259 A\u015fa\u011f\u0131 K\u00fcr \u00e7\u00f6k\u0259kliyi (AK\u00c7) \u00fc\u00e7\u00fcn geotemperatur x\u0259rit\u0259l\u0259rin\u0259 \u0259sas\u0259n neft v\u0259 qaz...\n",
-      "5. Example5: Bu zonada hans\u0131 prosesl\u0259r ba\u015f verir?...\n"
-     ]
+      "cell_type": "code",
+      "execution_count": 63,
+      "id": "c8867f44",
+      "metadata": {},
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "*******\n",
+            "Testing: Llama-4-Maverick\n",
+            "**********\n",
+            "  Example1: Daha az quyu ilə daha çox hasilat əldə etmək üçün hansı əsas...\n",
+            "    ✅ 4.31s\n",
+            "  Example2: Qərbi Abşeron yatağında suvurma tədbirləri hansı tarixdə və ...\n",
+            "    ✅ 4.61s\n",
+            "  Example3: Pirallahı strukturunda 1253 nömrəli quyudan götürülmüş nümun...\n",
+            "    ✅ 3.92s\n",
+            "  Example4: Bakı arxipelaqı (BA) və Aşağı Kür çökəkliyi (AKÇ) üçün geote...\n",
+            "    ✅ 4.13s\n",
+            "  Example5: Bu zonada hansı proseslər baş verir?...\n",
+            "    ✅ 3.50s\n",
+            "*******\n",
+            "Testing: DeepSeek-R1\n",
+            "**********\n",
+            "  Example1: Daha az quyu ilə daha çox hasilat əldə etmək üçün hansı əsas...\n",
+            "    ✅ 10.38s\n",
+            "  Example2: Qərbi Abşeron yatağında suvurma tədbirləri hansı tarixdə və ...\n",
+            "    ✅ 11.32s\n",
+            "  Example3: Pirallahı strukturunda 1253 nömrəli quyudan götürülmüş nümun...\n",
+            "    ✅ 10.45s\n",
+            "  Example4: Bakı arxipelaqı (BA) və Aşağı Kür çökəkliyi (AKÇ) üçün geote...\n",
+            "    ✅ 10.56s\n",
+            "  Example5: Bu zonada hansı proseslər baş verir?...\n",
+            "    ✅ 10.99s\n",
+            "*******\n",
+            "Testing: GPT-4.1\n",
+            "**********\n",
+            "  Example1: Daha az quyu ilə daha çox hasilat əldə etmək üçün hansı əsas...\n",
+            "    ✅ 6.32s\n",
+            "  Example2: Qərbi Abşeron yatağında suvurma tədbirləri hansı tarixdə və ...\n",
+            "    ✅ 5.85s\n",
+            "  Example3: Pirallahı strukturunda 1253 nömrəli quyudan götürülmüş nümun...\n",
+            "    ✅ 8.09s\n",
+            "  Example4: Bakı arxipelaqı (BA) və Aşağı Kür çök��kliyi (AKÇ) üçün geote...\n",
+            "    ✅ 6.72s\n",
+            "  Example5: Bu zonada hansı proseslər baş verir?...\n",
+            "    ✅ 5.22s\n",
+            "*********\n",
+            "✅ Benchmark complete!\n"
+          ]
+        }
+      ],
+      "source": [
+        "# Run benchmark\n",
+        "results = []\n",
+        "\n",
+        "for model_name in LLM_MODELS.keys():\n",
+        "    print(f\"*******\")\n",
+        "    print(f\"Testing: {model_name}\")\n",
+        "    print(f\"**********\")\n",
+        "    \n",
+        "    for example_key, messages in questions.items():\n",
+        "        user_msg = [m for m in messages if m[\"role\"] == \"user\"][-1]\n",
+        "        query = user_msg[\"content\"]\n",
+        "        \n",
+        "        print(f\"  {example_key}: {query[:60]}...\")\n",
+        "        \n",
+        "        # Retrieve and generate\n",
+        "        documents = retrieve_documents(query, top_k=3)\n",
+        "        answer, response_time = generate_answer(model_name, query, documents)\n",
+        "        \n",
+        "        if answer.startswith(\"ERROR\"):\n",
+        "            print(f\"    ❌ {answer}\")\n",
+        "            continue\n",
+        "        \n",
+        "        print(f\"    ✅ {response_time:.2f}s\")\n",
+        "        \n",
+        "        # Evaluate\n",
+        "        expected = expected_answers.get(example_key, {}).get(\"Answer\", \"\")\n",
+        "        metrics = evaluate_answer(expected, answer, documents)\n",
+        "        \n",
+        "        results.append({\n",
+        "            \"Model\": model_name,\n",
+        "            \"Question\": example_key,\n",
+        "            \"Response_Time\": round(response_time, 2),\n",
+        "            **metrics\n",
+        "        })\n",
+        "\n",
+        "print(\"*********\")\n",
+        "print(\"✅ Benchmark complete!\")"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 55,
+      "id": "9b243569",
+      "metadata": {},
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "\n",
+            "====================================================================================================\n",
+            "📊 LLM BENCHMARKING RESULTS\n",
+            "====================================================================================================\n",
+            "                  Quality_Score  Similarity  Citation_Score  Completeness  Response_Time\n",
+            "Model                                                                                   \n",
+            "GPT-4.1                   52.00        0.00           80.00         100.0           6.38\n",
+            "Llama-4-Maverick          52.00        0.00           80.00         100.0           4.00\n",
+            "DeepSeek-R1               32.27        1.54           33.33          91.6          10.98\n",
+            "====================================================================================================\n"
+          ]
+        }
+      ],
+      "source": [
+        "# Analyze results\n",
+        "df = pd.DataFrame(results)\n",
+        "summary = df.groupby(\"Model\").agg({\n",
+        "    \"Quality_Score\": \"mean\",\n",
+        "    \"Similarity\": \"mean\",\n",
+        "    \"Citation_Score\": \"mean\",\n",
+        "    \"Completeness\": \"mean\",\n",
+        "    \"Response_Time\": \"mean\"\n",
+        "}).round(2).sort_values(\"Quality_Score\", ascending=False)\n",
+        "\n",
+        "print(\"\\n\" + \"=\"*100)\n",
+        "print(\"📊 LLM BENCHMARKING RESULTS\")\n",
+        "print(\"=\"*100)\n",
+        "print(summary.to_string())\n",
+        "print(\"=\"*100)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 56,
+      "id": "8c64cf75",
+      "metadata": {},
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "\n",
+            "✅ Results saved to output/llm_benchmark/\n"
+          ]
+        }
+      ],
+      "source": [
+        "# Save results using dynamic path\n",
+        "output_dir = OUTPUT_DIR / \"llm_benchmark\"\n",
+        "output_dir.mkdir(parents=True, exist_ok=True)\n",
+        "\n",
+        "df.to_csv(output_dir / \"detailed_results.csv\", index=False, encoding=\"utf-8\")\n",
+        "summary.to_csv(output_dir / \"summary.csv\", encoding=\"utf-8\")\n",
+        "\n",
+        "print(\"\\n✅ Results saved to output/llm_benchmark/\")"
+      ]
+    }
+  ],
+  "metadata": {
+    "kernelspec": {
+      "display_name": "Python 3",
+      "language": "python",
+      "name": "python3"
+    },
+    "language_info": {
+      "name": "python",
+      "version": "3.10.0"
     }
-   ],
-   "source": [
-    "# Load sample questions - using dynamic paths\n",
-    "with open(DOCS_DIR / 'sample_questions.json', 'r', encoding='utf-8') as f:\n",
-    "    questions = json.load(f)\n",
-    "\n",
-    "# Load expected answers - using dynamic paths\n",
-    "with open(DOCS_DIR / 'sample_answers.json', 'r', encoding='utf-8') as f:\n",
-    "    expected_answers = json.load(f)\n",
-    "\n",
-    "print(f\"Loaded {len(questions)} test cases\")\n",
-    "print(\"\\nTest Questions:\")\n",
-    "for i, (key, msgs) in enumerate(questions.items(), 1):\n",
-    "    user_msg = [m for m in msgs if m['role'] == 'user'][-1]\n",
-    "    print(f\"{i}. {key}: {user_msg['content'][:100]}...\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## 2. Initialize Vector Database and Embedding Model"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Initialize Pinecone\n",
-    "pc = Pinecone(api_key=os.getenv('PINECONE_API_KEY'))\n",
-    "index = pc.Index(os.getenv('PINECONE_INDEX_NAME', 'hackathon'))\n",
-    "\n",
-    "# Initialize embedding model (same as used for ingestion)\n",
-    "embed_model = SentenceTransformer('BAAI/bge-large-en-v1.5')\n",
-    "\n",
-    "print(f\"\u2705 Vector DB connected: {index.describe_index_stats()}\")\n",
-    "print(f\"\u2705 Embedding model loaded: {embed_model}\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## 3. RAG Retrieval Function"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "def retrieve_documents(query: str, top_k: int = 3) -> List[Dict]:\n",
-    "    \"\"\"\n",
-    "    Retrieve relevant documents from vector database.\n",
-    "    \"\"\"\n",
-    "    # Generate query embedding\n",
-    "    query_embedding = embed_model.encode(query).tolist()\n",
-    "    \n",
-    "    # Search vector DB\n",
-    "    results = index.query(\n",
-    "        vector=query_embedding,\n",
-    "        top_k=top_k,\n",
-    "        include_metadata=True\n",
-    "    )\n",
-    "    \n",
-    "    # Extract documents\n",
-    "    documents = []\n",
-    "    for match in results['matches']:\n",
-    "        documents.append({\n",
-    "            'pdf_name': match['metadata'].get('pdf_name', 'unknown.pdf'),\n",
-    "            'page_number': match['metadata'].get('page_number', 0),\n",
-    "            'content': match['metadata'].get('text', ''),\n",
-    "            'score': match.get('score', 0.0)\n",
-    "        })\n",
-    "    \n",
-    "    return documents\n",
-    "\n",
-    "# Test retrieval\n",
-    "test_query = \"Pal\u00e7\u0131q vulkanlar\u0131n\u0131n t\u0259sir radiusu n\u0259 q\u0259d\u0259rdir?\"\n",
-    "test_docs = retrieve_documents(test_query)\n",
-    "print(f\"\\n\u2705 Retrieved {len(test_docs)} documents for test query\")\n",
-    "print(f\"Top result: {test_docs[0]['pdf_name']}, page {test_docs[0]['page_number']} (score: {test_docs[0]['score']:.3f})\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## 4. LLM Client Functions"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Initialize Azure OpenAI\n",
-    "azure_client = AzureOpenAI(\n",
-    "    api_key=os.getenv('AZURE_OPENAI_API_KEY'),\n",
-    "    api_version=os.getenv('AZURE_OPENAI_API_VERSION', '2024-08-01-preview'),\n",
-    "    azure_endpoint=os.getenv('AZURE_OPENAI_ENDPOINT')\n",
-    ")\n",
-    "\n",
-    "LLM_MODELS = {\n",
-    "    'Llama-4-Maverick': 'Llama-4-Maverick-17B-128E-Instruct-FP8',\n",
-    "    'DeepSeek-R1': 'DeepSeek-R1',\n",
-    "    'GPT-4.1': 'gpt-4.1',\n",
-    "    'GPT-5-mini': 'gpt-5-mini',\n",
-    "    'Claude-Sonnet-4.5': 'claude-sonnet-4-5',\n",
-    "}\n",
-    "\n",
-    "def generate_answer(model_name: str, query: str, documents: List[Dict], \n",
-    "                   temperature: float = 0.2, max_tokens: int = 1000) -> Tuple[str, float]:\n",
-    "    \"\"\"\n",
-    "    Generate answer using specified LLM model.\n",
-    "    Returns: (answer, response_time)\n",
-    "    \"\"\"\n",
-    "    # Build context from retrieved documents\n",
-    "    context_parts = []\n",
-    "    for i, doc in enumerate(documents, 1):\n",
-    "        context_parts.append(\n",
-    "            f\"Document {i} (Source: {doc['pdf_name']}, Page {doc['page_number']}):\\n{doc['content']}\"\n",
-    "        )\n",
-    "    context = \"\\n\\n\".join(context_parts)\n",
-    "    \n",
-    "    # Create prompt\n",
-    "    prompt = f\"\"\"Siz SOCAR-\u0131n tarixi neft v\u0259 qaz s\u0259n\u0259dl\u0259ri \u00fczr\u0259 m\u00fct\u0259x\u0259ssis k\u00f6m\u0259k\u00e7isisiniz.\n",
-    "\n",
-    "Kontekst (\u0259laq\u0259li s\u0259n\u0259dl\u0259r):\n",
-    "{context}\n",
-    "\n",
-    "Sual: {query}\n",
-    "\n",
-    "\u018ftrafl\u0131 cavab verin v\u0259 m\u00fctl\u0259q s\u0259n\u0259d m\u0259nb\u0259l\u0259rin\u0259 istinad edin (PDF ad\u0131 v\u0259 s\u0259hif\u0259 n\u00f6mr\u0259si il\u0259).\n",
-    "Cavab\u0131n\u0131z d\u0259qiq, faktlara \u0259saslanan v\u0259 kontekst m\u0259lumatlar\u0131ndan istifad\u0259 ed\u0259n olmal\u0131d\u0131r.\"\"\"\n",
-    "    \n",
-    "    # Get model deployment\n",
-    "    deployment = MODELS[model_name]['deployment']\n",
-    "    \n",
-    "    try:\n",
-    "        start_time = time.time()\n",
-    "        \n",
-    "        # GPT-5 models use max_completion_tokens, others use max_tokens\n",
-    "        if deployment.startswith('gpt-5'):\n",
-    "            response = azure_client.chat.completions.create(\n",
-    "                model=deployment,\n",
-    "                messages=[\n",
-    "                    {\"role\": \"user\", \"content\": prompt}\n",
-    "                ],\n",
-    "                temperature=temperature,\n",
-    "                max_completion_tokens=max_tokens\n",
-    "            )\n",
-    "        else:\n",
-    "            response = azure_client.chat.completions.create(\n",
-    "                model=deployment,\n",
-    "                messages=[\n",
-    "                    {\"role\": \"user\", \"content\": prompt}\n",
-    "                ],\n",
-    "                temperature=temperature,\n",
-    "                max_tokens=max_tokens\n",
-    "            )\n",
-    "        \n",
-    "        response_time = time.time() - start_time\n",
-    "        answer = response.choices[0].message.content\n",
-    "        \n",
-    "        return answer, response_time\n",
-    "    \n",
-    "    except Exception as e:\n",
-    "        return f\"ERROR: {str(e)}\", 0.0\n",
-    "\n",
-    "print(f\"\\n\u2705 Configured {len(LLM_MODELS)} LLM models for testing\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## 5. Evaluation Metrics"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "def normalize_text(text: str) -> str:\n",
-    "    \"\"\"Normalize text for comparison.\"\"\"\n",
-    "    import re\n",
-    "    text = text.lower().strip()\n",
-    "    text = re.sub(r'\\s+', ' ', text)\n",
-    "    return text\n",
-    "\n",
-    "def calculate_answer_similarity(reference: str, hypothesis: str) -> Dict[str, float]:\n",
-    "    \"\"\"\n",
-    "    Calculate similarity between generated and expected answer.\n",
-    "    Lower is better for error rates.\n",
-    "    \"\"\"\n",
-    "    ref_norm = normalize_text(reference)\n",
-    "    hyp_norm = normalize_text(hypothesis)\n",
-    "    \n",
-    "    # Character Error Rate\n",
-    "    cer_score = cer(ref_norm, hyp_norm) * 100\n",
-    "    \n",
-    "    # Word Error Rate  \n",
-    "    wer_score = wer(ref_norm, hyp_norm) * 100\n",
-    "    \n",
-    "    # Similarity scores (higher is better)\n",
-    "    similarity = max(0, 100 - wer_score)\n",
-    "    \n",
-    "    return {\n",
-    "        'CER': round(cer_score, 2),\n",
-    "        'WER': round(wer_score, 2),\n",
-    "        'Similarity': round(similarity, 2)\n",
-    "    }\n",
-    "\n",
-    "def check_citations(answer: str, documents: List[Dict]) -> Dict[str, any]:\n",
-    "    \"\"\"\n",
-    "    Check if answer includes proper citations.\n",
-    "    \"\"\"\n",
-    "    import re\n",
-    "    \n",
-    "    # Check for PDF names\n",
-    "    pdf_names = [doc['pdf_name'] for doc in documents]\n",
-    "    cited_pdfs = sum(1 for pdf in pdf_names if pdf.replace('.pdf', '') in answer)\n",
-    "    \n",
-    "    # Check for page numbers\n",
-    "    page_numbers = [str(doc['page_number']) for doc in documents]\n",
-    "    cited_pages = sum(1 for page in page_numbers if page in answer)\n",
-    "    \n",
-    "    # Check for source keywords\n",
-    "    source_keywords = ['m\u0259nb\u0259', 's\u0259n\u0259d', 's\u0259hif\u0259', 'pdf', 'document', 'page', 'source']\n",
-    "    has_source_ref = any(kw in answer.lower() for kw in source_keywords)\n",
-    "    \n",
-    "    citation_score = (\n",
-    "        (cited_pdfs / len(pdf_names) * 40) +  # 40% for PDF citation\n",
-    "        (cited_pages / len(page_numbers) * 40) +  # 40% for page citation\n",
-    "        (20 if has_source_ref else 0)  # 20% for having source keywords\n",
-    "    )\n",
-    "    \n",
-    "    return {\n",
-    "        'Citation_Score': round(citation_score, 2),\n",
-    "        'Cited_PDFs': cited_pdfs,\n",
-    "        'Cited_Pages': cited_pages,\n",
-    "        'Has_Source_Reference': has_source_ref\n",
-    "    }\n",
-    "\n",
-    "def evaluate_completeness(answer: str, min_length: int = 100) -> Dict[str, any]:\n",
-    "    \"\"\"\n",
-    "    Evaluate answer completeness.\n",
-    "    \"\"\"\n",
-    "    word_count = len(answer.split())\n",
-    "    char_count = len(answer)\n",
-    "    \n",
-    "    # Penalize very short or very long answers\n",
-    "    if char_count < min_length:\n",
-    "        completeness_score = (char_count / min_length) * 100\n",
-    "    elif char_count > 2000:\n",
-    "        completeness_score = 100 - ((char_count - 2000) / 2000 * 20)  # Penalty for verbosity\n",
-    "    else:\n",
-    "        completeness_score = 100\n",
-    "    \n",
-    "    return {\n",
-    "        'Completeness_Score': round(max(0, completeness_score), 2),\n",
-    "        'Word_Count': word_count,\n",
-    "        'Char_Count': char_count\n",
-    "    }\n",
-    "\n",
-    "print(\"\u2705 Evaluation functions ready\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## 6. Run Benchmark on All Models"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Select models to test (you can comment out models to skip)\n",
-    "MODELS_TO_TEST = [\n",
-    "    'Llama-4-Maverick-17B',\n",
-    "    'DeepSeek-R1',\n",
-    "    'GPT-4.1',\n",
-    "    'GPT-5-mini',\n",
-    "    'Claude-Sonnet-4.5',\n",
-    "    # 'Claude-Opus-4.1',  # Uncomment to test\n",
-    "    # 'Phi-4-multimodal',  # Uncomment to test\n",
-    "    # 'GPT-OSS-120B',  # Uncomment to test\n",
-    "]\n",
-    "\n",
-    "print(f\"Testing {len(MODELS_TO_TEST)} models on {len(questions)} questions...\\n\")\n",
-    "print(\"This may take several minutes...\\n\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Run benchmark\n",
-    "results = []\n",
-    "\n",
-    "for model_name in MODELS_TO_TEST:\n",
-    "    print(f\"\\n{'='*80}\")\n",
-    "    print(f\"Testing: {model_name}\")\n",
-    "    print(f\"{'='*80}\")\n",
-    "    \n",
-    "    model_results = []\n",
-    "    \n",
-    "    for example_key, messages in questions.items():\n",
-    "        # Get the last user message (the actual question)\n",
-    "        user_msg = [m for m in messages if m['role'] == 'user'][-1]\n",
-    "        query = user_msg['content']\n",
-    "        \n",
-    "        print(f\"\\n  Question {example_key}: {query[:80]}...\")\n",
-    "        \n",
-    "        # Retrieve documents\n",
-    "        documents = retrieve_documents(query, top_k=3)\n",
-    "        \n",
-    "        # Generate answer\n",
-    "        answer, response_time = generate_answer(model_name, query, documents)\n",
-    "        \n",
-    "        if answer.startswith('ERROR'):\n",
-    "            print(f\"  \u274c Failed: {answer}\")\n",
-    "            continue\n",
-    "        \n",
-    "        print(f\"  \u2705 Response time: {response_time:.2f}s\")\n",
-    "        \n",
-    "        # Get expected answer\n",
-    "        expected = expected_answers.get(example_key, {}).get('Answer', '')\n",
-    "        \n",
-    "        # Calculate metrics\n",
-    "        similarity_metrics = calculate_answer_similarity(expected, answer) if expected else {'CER': 0, 'WER': 0, 'Similarity': 0}\n",
-    "        citation_metrics = check_citations(answer, documents)\n",
-    "        completeness_metrics = evaluate_completeness(answer)\n",
-    "        \n",
-    "        # Store result\n",
-    "        result = {\n",
-    "            'Model': model_name,\n",
-    "            'Question': example_key,\n",
-    "            'Query': query[:100],\n",
-    "            'Answer': answer[:200] + '...',\n",
-    "            'Response_Time': round(response_time, 2),\n",
-    "            **similarity_metrics,\n",
-    "            **citation_metrics,\n",
-    "            **completeness_metrics,\n",
-    "            'Open_Source': MODELS[model_name]['open_source'],\n",
-    "            'Architecture_Score': MODELS[model_name]['architecture_score']\n",
-    "        }\n",
-    "        \n",
-    "        model_results.append(result)\n",
-    "        results.append(result)\n",
-    "    \n",
-    "    # Show summary for this model\n",
-    "    if model_results:\n",
-    "        avg_response_time = sum(r['Response_Time'] for r in model_results) / len(model_results)\n",
-    "        avg_similarity = sum(r['Similarity'] for r in model_results) / len(model_results)\n",
-    "        avg_citation = sum(r['Citation_Score'] for r in model_results) / len(model_results)\n",
-    "        avg_completeness = sum(r['Completeness_Score'] for r in model_results) / len(model_results)\n",
-    "        \n",
-    "        print(f\"\\n  \ud83d\udcca {model_name} Summary:\")\n",
-    "        print(f\"     Avg Response Time: {avg_response_time:.2f}s\")\n",
-    "        print(f\"     Avg Similarity: {avg_similarity:.1f}%\")\n",
-    "        print(f\"     Avg Citation Score: {avg_citation:.1f}%\")\n",
-    "        print(f\"     Avg Completeness: {avg_completeness:.1f}%\")\n",
-    "\n",
-    "print(f\"\\n{'='*80}\")\n",
-    "print(\"\u2705 Benchmarking complete!\")\n",
-    "print(f\"{'='*80}\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## 7. Aggregate Results and Rankings"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Create DataFrame\n",
-    "df = pd.DataFrame(results)\n",
-    "\n",
-    "# Calculate aggregate scores per model\n",
-    "model_summary = df.groupby('Model').agg({\n",
-    "    'Response_Time': 'mean',\n",
-    "    'Similarity': 'mean',\n",
-    "    'Citation_Score': 'mean',\n",
-    "    'Completeness_Score': 'mean',\n",
-    "    'CER': 'mean',\n",
-    "    'WER': 'mean',\n",
-    "    'Open_Source': 'first',\n",
-    "    'Architecture_Score': 'first'\n",
-    "}).round(2)\n",
-    "\n",
-    "# Calculate overall quality score (weighted average)\n",
-    "model_summary['Quality_Score'] = (\n",
-    "    model_summary['Similarity'] * 0.35 +  # 35% answer accuracy\n",
-    "    model_summary['Citation_Score'] * 0.35 +  # 35% citation quality\n",
-    "    model_summary['Completeness_Score'] * 0.30  # 30% completeness\n",
-    ").round(2)\n",
-    "\n",
-    "# Sort by Quality Score\n",
-    "model_summary = model_summary.sort_values('Quality_Score', ascending=False)\n",
-    "\n",
-    "# Display summary table\n",
-    "print(\"\\n\" + \"=\"*100)\n",
-    "print(\"\ud83d\udcca LLM BENCHMARKING RESULTS - MODEL SUMMARY\")\n",
-    "print(\"=\"*100)\n",
-    "print(model_summary.to_string())\n",
-    "print(\"=\"*100)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "metadata": {},
-   "source": [
-    "# Create comprehensive visualization\n",
-    "import os\n",
-    "from pathlib import Path\n",
-    "\n",
-    "# Create output directory - using dynamic path\n",
-    "output_dir = OUTPUT_DIR / 'llm_benchmark'\n",
-    "output_dir.mkdir(parents=True, exist_ok=True)\n",
-    "\n",
-    "fig, axes = plt.subplots(2, 2, figsize=(16, 10))\n",
-    "\n",
-    "models = df['Model'].tolist()\n",
-    "colors = sns.color_palette('viridis', len(models))\n",
-    "\n",
-    "# 1. CSR - Character Success Rate (MAIN METRIC)\n",
-    "ax1 = axes[0, 0]\n",
-    "bars1 = ax1.barh(models, df['CSR'], color=colors)\n",
-    "ax1.set_xlabel('CSR (%) - Higher is Better', fontsize=12, fontweight='bold')\n",
-    "ax1.set_title('Character Success Rate (CSR)\\n\ud83c\udfc6 HACKATHON PRIMARY METRIC', \n",
-    "              fontsize=14, fontweight='bold')\n",
-    "ax1.set_xlim(0, 100)\n",
-    "for i, (model, csr) in enumerate(zip(models, df['CSR'])):\n",
-    "    ax1.text(csr + 1, i, f'{csr:.2f}%', va='center', fontsize=11, fontweight='bold')\n",
-    "ax1.axvline(x=90, color='green', linestyle='--', alpha=0.3, label='Excellent (>90%)')\n",
-    "ax1.axvline(x=80, color='orange', linestyle='--', alpha=0.3, label='Good (>80%)')\n",
-    "ax1.legend(fontsize=9)\n",
-    "\n",
-    "# 2. WSR - Word Success Rate\n",
-    "ax2 = axes[0, 1]\n",
-    "bars2 = ax2.barh(models, df['WSR'], color=colors)\n",
-    "ax2.set_xlabel('WSR (%) - Higher is Better', fontsize=12, fontweight='bold')\n",
-    "ax2.set_title('Word Success Rate (WSR)', fontsize=14, fontweight='bold')\n",
-    "ax2.set_xlim(0, 100)\n",
-    "for i, (model, wsr) in enumerate(zip(models, df['WSR'])):\n",
-    "    ax2.text(wsr + 1, i, f'{wsr:.2f}%', va='center', fontsize=11, fontweight='bold')\n",
-    "\n",
-    "# 3. Response Time\n",
-    "ax3 = axes[1, 0]\n",
-    "bars3 = ax3.barh(models, df['Response_Time'], color=colors)\n",
-    "ax3.set_xlabel('Total Time (seconds) - Lower is Better', fontsize=12, fontweight='bold')\n",
-    "ax3.set_title('Processing Speed', fontsize=14, fontweight='bold')\n",
-    "for i, (model, time_val) in enumerate(zip(models, df['Response_Time'])):\n",
-    "    ax3.text(time_val + 0.5, i, f'{time_val:.1f}s', va='center', fontsize=11)\n",
-    "\n",
-    "# 4. Error Rates Comparison\n",
-    "ax4 = axes[1, 1]\n",
-    "x = range(len(models))\n",
-    "width = 0.35\n",
-    "ax4.bar([i - width/2 for i in x], df['CER'], width, label='CER', color='coral', alpha=0.8)\n",
-    "ax4.bar([i + width/2 for i in x], df['WER'], width, label='WER', color='skyblue', alpha=0.8)\n",
-    "ax4.set_ylabel('Error Rate (%) - Lower is Better', fontsize=12, fontweight='bold')\n",
-    "ax4.set_title('Error Rates', fontsize=14, fontweight='bold')\n",
-    "ax4.set_xticks(x)\n",
-    "ax4.set_xticklabels(models, rotation=45, ha='right')\n",
-    "ax4.legend(fontsize=11)\n",
-    "ax4.grid(axis='y', alpha=0.3)\n",
-    "\n",
-    "plt.tight_layout()\n",
-    "plt.savefig(output_dir / 'results.png', dpi=300, bbox_inches='tight')\n",
-    "plt.show()\n",
-    "\n",
-    "print(f\"\\n\u2705 Visualization saved to '{output_dir}/results.png'\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Create rankings table\n",
-    "rankings = model_summary[[\n",
-    "    'Quality_Score', 'Similarity', 'Citation_Score', 'Completeness_Score', \n",
-    "    'Response_Time', 'Open_Source', 'Architecture_Score'\n",
-    "]].copy()\n",
-    "\n",
-    "rankings.insert(0, 'Rank', range(1, len(rankings) + 1))\n",
-    "\n",
-    "print(\"\\n\" + \"=\"*100)\n",
-    "print(\"\ud83c\udfc6 FINAL RANKINGS\")\n",
-    "print(\"=\"*100)\n",
-    "print(rankings.to_string())\n",
-    "print(\"=\"*100)\n",
-    "\n",
-    "# Winner analysis\n",
-    "best_overall = rankings.index[0]\n",
-    "best_open_source = rankings[rankings['Open_Source'] == True].index[0] if any(rankings['Open_Source']) else None\n",
-    "fastest = model_summary['Response_Time'].idxmin()\n",
-    "\n",
-    "print(\"\\n\" + \"=\"*100)\n",
-    "print(\"\ud83d\udca1 RECOMMENDATIONS FOR HACKATHON\")\n",
-    "print(\"=\"*100)\n",
-    "\n",
-    "print(f\"\\n\ud83e\udd47 Best Overall Quality: {best_overall}\")\n",
-    "print(f\"   Quality Score: {model_summary.loc[best_overall, 'Quality_Score']:.1f}%\")\n",
-    "print(f\"   Similarity: {model_summary.loc[best_overall, 'Similarity']:.1f}%\")\n",
-    "print(f\"   Citation Score: {model_summary.loc[best_overall, 'Citation_Score']:.1f}%\")\n",
-    "print(f\"   Response Time: {model_summary.loc[best_overall, 'Response_Time']:.2f}s\")\n",
-    "print(f\"   Open Source: {model_summary.loc[best_overall, 'Open_Source']}\")\n",
-    "print(f\"   Architecture Score: {model_summary.loc[best_overall, 'Architecture_Score']}\")\n",
-    "\n",
-    "if best_open_source:\n",
-    "    print(f\"\\n\ud83d\udd13 Best Open-Source Model: {best_open_source}\")\n",
-    "    print(f\"   Quality Score: {model_summary.loc[best_open_source, 'Quality_Score']:.1f}%\")\n",
-    "    print(f\"   Architecture Score: {model_summary.loc[best_open_source, 'Architecture_Score']} (Better for hackathon!)\")\n",
-    "    print(f\"   Response Time: {model_summary.loc[best_open_source, 'Response_Time']:.2f}s\")\n",
-    "\n",
-    "print(f\"\\n\u26a1 Fastest Model: {fastest}\")\n",
-    "print(f\"   Response Time: {model_summary.loc[fastest, 'Response_Time']:.2f}s\")\n",
-    "print(f\"   Quality Score: {model_summary.loc[fastest, 'Quality_Score']:.1f}%\")\n",
-    "\n",
-    "print(\"\\n\" + \"=\"*100)\n",
-    "print(\"\ud83d\udcdd FINAL RECOMMENDATION\")\n",
-    "print(\"=\"*100)\n",
-    "print(\"\\nScoring Breakdown:\")\n",
-    "print(\"  - LLM Quality: 30% of total hackathon score\")\n",
-    "print(\"  - Architecture: 20% of total hackathon score (open-source preferred!)\")\n",
-    "print(\"\\nBest Choice:\")\n",
-    "if best_open_source and model_summary.loc[best_open_source, 'Quality_Score'] >= model_summary.loc[best_overall, 'Quality_Score'] * 0.9:\n",
-    "    print(f\"  \u2705 {best_open_source} - Best balance of quality and architecture score\")\n",
-    "    print(f\"     Only {model_summary.loc[best_overall, 'Quality_Score'] - model_summary.loc[best_open_source, 'Quality_Score']:.1f}% quality drop for higher architecture score!\")\n",
-    "else:\n",
-    "    print(f\"  \u2705 {best_overall} - Highest quality, use if quality gap is significant\")\n",
-    "    if best_open_source:\n",
-    "        print(f\"  \u26a0\ufe0f  Consider {best_open_source} for higher architecture score (trade-off: {model_summary.loc[best_overall, 'Quality_Score'] - model_summary.loc[best_open_source, 'Quality_Score']:.1f}% quality)\")\n",
-    "\n",
-    "print(\"=\"*100)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "metadata": {},
-   "source": [
-    "# Save results\n",
-    "from pathlib import Path\n",
-    "\n",
-    "# Using dynamic path\n",
-    "output_dir = OUTPUT_DIR / 'llm_benchmark'\n",
-    "output_dir.mkdir(parents=True, exist_ok=True)\n",
-    "\n",
-    "df.to_csv(output_dir / 'detailed_results.csv', index=False, encoding='utf-8')\n",
-    "model_summary.to_csv(output_dir / 'summary.csv', encoding='utf-8')\n",
-    "rankings.to_csv(output_dir / 'rankings.csv', index=False, encoding='utf-8')\n",
-    "\n",
-    "print(\"\\n\u2705 Results exported to output/llm_benchmark/:\")\n",
-    "print(\"   - detailed_results.csv (all questions and answers)\")\n",
-    "print(\"   - summary.csv (model averages)\")\n",
-    "print(\"   - rankings.csv (final rankings)\")\n",
-    "print(\"   - results.png (visualizations)\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## 11. Sample Answer Comparison"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "venv",
-   "language": "python",
-   "name": "python3"
   },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.10.12"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 4
-}
+  "nbformat": 4,
+  "nbformat_minor": 5
+}

notebooks/llm_benchmark.ipynb.backup

@@ -1,761 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# LLM Benchmarking for SOCAR Hackathon RAG Chatbot\n",
-    "\n",
-    "This notebook tests different LLM models for the `/llm` endpoint to find the best performer.\n",
-    "\n",
-    "## Evaluation Criteria (LLM Judge Metrics):\n",
-    "- **Accuracy**: Is the answer correct?\n",
-    "- **Relevance**: Are retrieved citations relevant?\n",
-    "- **Completeness**: Does it fully answer the question?\n",
-    "- **Citation Quality**: Proper sources with page numbers?\n",
-    "- **Response Time**: Speed of generation\n",
-    "\n",
-    "## Available LLM Models:\n",
-    "1. **Llama-4-Maverick-17B-128E-Instruct-FP8** (Current choice, open-source)\n",
-    "2. **DeepSeek-R1** (Open-source reasoning model)\n",
-    "3. **GPT-4.1** (Strong general performance)\n",
-    "4. **GPT-5, GPT-5-mini**\n",
-    "5. **Claude Sonnet 4.5** (Best quality)\n",
-    "6. **Claude Opus 4.1**\n",
-    "7. **Phi-4-multimodal-instruct**\n",
-    "8. **gpt-oss-120b**"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Install required packages\n",
-    "# !pip install openai pinecone-client sentence-transformers python-dotenv pandas matplotlib seaborn jiwer"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "/Users/ismatsamadov/SOCAR_Hackathon/venv/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
-      "  from .autonotebook import tqdm as notebook_tqdm\n"
-     ]
-    },
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "\u2705 Libraries loaded successfully\n"
-     ]
-    }
-   ],
-   "source": [
-    "import os\n",
-    "import json\n",
-    "import time\n",
-    "from typing import Dict, List, Tuple\n",
-    "from dotenv import load_dotenv\n",
-    "import pandas as pd\n",
-    "import matplotlib.pyplot as plt\n",
-    "import seaborn as sns\n",
-    "from openai import AzureOpenAI\n",
-    "from pinecone import Pinecone\n",
-    "from sentence_transformers import SentenceTransformer\n",
-    "from jiwer import wer, cer\n",
-    "\n",
-    "# Load environment variables\n",
-    "load_dotenv()\n",
-    "\n",
-    "# Set style\n",
-    "sns.set_style('whitegrid')\n",
-    "plt.rcParams['figure.figsize'] = (14, 8)\n",
-    "\n",
-    "print(\"\u2705 Libraries loaded successfully\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "\u2705 Project root: /Users/ismatsamadov/SOCAR_Hackathon\n",
-      "\u2705 Docs directory: /Users/ismatsamadov/SOCAR_Hackathon/docs\n",
-      "\u2705 Output directory: /Users/ismatsamadov/SOCAR_Hackathon/output\n"
-     ]
-    }
-   ],
-   "source": [
-    "# Auto-detect project root (works from any directory)\n",
-    "import os\n",
-    "from pathlib import Path\n",
-    "\n",
-    "if Path('data').exists() and Path('docs').exists():\n",
-    "    # Already in project root\n",
-    "    PROJECT_ROOT = Path.cwd()\n",
-    "elif Path('../data').exists() and Path('../docs').exists():\n",
-    "    # In notebooks/ subdirectory\n",
-    "    PROJECT_ROOT = Path.cwd().parent\n",
-    "else:\n",
-    "    # Fallback: try to find project root\n",
-    "    current = Path.cwd()\n",
-    "    while current != current.parent:\n",
-    "        if (current / 'data').exists() and (current / 'docs').exists():\n",
-    "            PROJECT_ROOT = current\n",
-    "            break\n",
-    "        current = current.parent\n",
-    "    else:\n",
-    "        PROJECT_ROOT = Path.cwd()\n",
-    "\n",
-    "# Define all paths relative to project root\n",
-    "DATA_DIR = PROJECT_ROOT / 'data'\n",
-    "DOCS_DIR = PROJECT_ROOT / 'docs'\n",
-    "OUTPUT_DIR = PROJECT_ROOT / 'output'\n",
-    "\n",
-    "print(f\"\u2705 Project root: {PROJECT_ROOT}\")\n",
-    "print(f\"\u2705 Docs directory: {DOCS_DIR}\")\n",
-    "print(f\"\u2705 Output directory: {OUTPUT_DIR}\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 4,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Loaded 5 test cases\n",
-      "\n",
-      "Test Questions:\n",
-      "1. Example1: Daha az quyu il\u0259 daha \u00e7ox hasilat \u0259ld\u0259 etm\u0259k \u00fc\u00e7\u00fcn hans\u0131 \u0259sas amill\u0259rin inteqrasiyas\u0131 t\u0259l\u0259b olunur?...\n",
-      "2. Example2: Q\u0259rbi Ab\u015feron yata\u011f\u0131nda suvurma t\u0259dbirl\u0259ri hans\u0131 tarixd\u0259 v\u0259 hans\u0131 layda t\u0259tbiq edilmi\u015fdir v\u0259 bunun m...\n",
-      "3. Example3: Pirallah\u0131 strukturunda 1253 n\u00f6mr\u0259li quyudan g\u00f6t\u00fcr\u00fclm\u00fc\u015f n\u00fcmun\u0259l\u0259rd\u0259 SiO2 v\u0259 CaO oksidl\u0259ri aras\u0131nda ha...\n",
-      "4. Example4: Bak\u0131 arxipelaq\u0131 (BA) v\u0259 A\u015fa\u011f\u0131 K\u00fcr \u00e7\u00f6k\u0259kliyi (AK\u00c7) \u00fc\u00e7\u00fcn geotemperatur x\u0259rit\u0259l\u0259rin\u0259 \u0259sas\u0259n neft v\u0259 qaz...\n",
-      "5. Example5: Bu zonada hans\u0131 prosesl\u0259r ba\u015f verir?...\n"
-     ]
-    }
-   ],
-   "source": [
-    "# Load sample questions - using dynamic paths\n",
-    "with open(DOCS_DIR / 'sample_questions.json', 'r', encoding='utf-8') as f:\n",
-    "    questions = json.load(f)\n",
-    "\n",
-    "# Load expected answers - using dynamic paths\n",
-    "with open(DOCS_DIR / 'sample_answers.json', 'r', encoding='utf-8') as f:\n",
-    "    expected_answers = json.load(f)\n",
-    "\n",
-    "print(f\"Loaded {len(questions)} test cases\")\n",
-    "print(\"\\nTest Questions:\")\n",
-    "for i, (key, msgs) in enumerate(questions.items(), 1):\n",
-    "    user_msg = [m for m in msgs if m['role'] == 'user'][-1]\n",
-    "    print(f\"{i}. {key}: {user_msg['content'][:100]}...\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## 2. Initialize Vector Database and Embedding Model"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Initialize Pinecone\n",
-    "pc = Pinecone(api_key=os.getenv('PINECONE_API_KEY'))\n",
-    "index = pc.Index(os.getenv('PINECONE_INDEX_NAME', 'hackathon'))\n",
-    "\n",
-    "# Initialize embedding model (same as used for ingestion)\n",
-    "embed_model = SentenceTransformer('BAAI/bge-large-en-v1.5')\n",
-    "\n",
-    "print(f\"\u2705 Vector DB connected: {index.describe_index_stats()}\")\n",
-    "print(f\"\u2705 Embedding model loaded: {embed_model}\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## 3. RAG Retrieval Function"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "def retrieve_documents(query: str, top_k: int = 3) -> List[Dict]:\n",
-    "    \"\"\"\n",
-    "    Retrieve relevant documents from vector database.\n",
-    "    \"\"\"\n",
-    "    # Generate query embedding\n",
-    "    query_embedding = embed_model.encode(query).tolist()\n",
-    "    \n",
-    "    # Search vector DB\n",
-    "    results = index.query(\n",
-    "        vector=query_embedding,\n",
-    "        top_k=top_k,\n",
-    "        include_metadata=True\n",
-    "    )\n",
-    "    \n",
-    "    # Extract documents\n",
-    "    documents = []\n",
-    "    for match in results['matches']:\n",
-    "        documents.append({\n",
-    "            'pdf_name': match['metadata'].get('pdf_name', 'unknown.pdf'),\n",
-    "            'page_number': match['metadata'].get('page_number', 0),\n",
-    "            'content': match['metadata'].get('text', ''),\n",
-    "            'score': match.get('score', 0.0)\n",
-    "        })\n",
-    "    \n",
-    "    return documents\n",
-    "\n",
-    "# Test retrieval\n",
-    "test_query = \"Pal\u00e7\u0131q vulkanlar\u0131n\u0131n t\u0259sir radiusu n\u0259 q\u0259d\u0259rdir?\"\n",
-    "test_docs = retrieve_documents(test_query)\n",
-    "print(f\"\\n\u2705 Retrieved {len(test_docs)} documents for test query\")\n",
-    "print(f\"Top result: {test_docs[0]['pdf_name']}, page {test_docs[0]['page_number']} (score: {test_docs[0]['score']:.3f})\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## 4. LLM Client Functions"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Initialize Azure OpenAI\n",
-    "azure_client = AzureOpenAI(\n",
-    "    api_key=os.getenv('AZURE_OPENAI_API_KEY'),\n",
-    "    api_version=os.getenv('AZURE_OPENAI_API_VERSION', '2024-08-01-preview'),\n",
-    "    azure_endpoint=os.getenv('AZURE_OPENAI_ENDPOINT')\n",
-    ")\n",
-    "\n",
-    "LLM_MODELS = {\n",
-    "    'Llama-4-Maverick': 'Llama-4-Maverick-17B-128E-Instruct-FP8',\n",
-    "    'DeepSeek-R1': 'DeepSeek-R1',\n",
-    "    'GPT-4.1': 'gpt-4.1',\n",
-    "    'GPT-5-mini': 'gpt-5-mini',\n",
-    "    'Claude-Sonnet-4.5': 'claude-sonnet-4-5',\n",
-    "}\n",
-    "\n",
-    "def generate_answer(model_name: str, query: str, documents: List[Dict], \n",
-    "                   temperature: float = 0.2, max_tokens: int = 1000) -> Tuple[str, float]:\n",
-    "    \"\"\"\n",
-    "    Generate answer using specified LLM model.\n",
-    "    Returns: (answer, response_time)\n",
-    "    \"\"\"\n",
-    "    # Build context from retrieved documents\n",
-    "    context_parts = []\n",
-    "    for i, doc in enumerate(documents, 1):\n",
-    "        context_parts.append(\n",
-    "            f\"Document {i} (Source: {doc['pdf_name']}, Page {doc['page_number']}):\\n{doc['content']}\"\n",
-    "        )\n",
-    "    context = \"\\n\\n\".join(context_parts)\n",
-    "    \n",
-    "    # Create prompt\n",
-    "    prompt = f\"\"\"Siz SOCAR-\u0131n tarixi neft v\u0259 qaz s\u0259n\u0259dl\u0259ri \u00fczr\u0259 m\u00fct\u0259x\u0259ssis k\u00f6m\u0259k\u00e7isisiniz.\n",
-    "\n",
-    "Kontekst (\u0259laq\u0259li s\u0259n\u0259dl\u0259r):\n",
-    "{context}\n",
-    "\n",
-    "Sual: {query}\n",
-    "\n",
-    "\u018ftrafl\u0131 cavab verin v\u0259 m\u00fctl\u0259q s\u0259n\u0259d m\u0259nb\u0259l\u0259rin\u0259 istinad edin (PDF ad\u0131 v\u0259 s\u0259hif\u0259 n\u00f6mr\u0259si il\u0259).\n",
-    "Cavab\u0131n\u0131z d\u0259qiq, faktlara \u0259saslanan v\u0259 kontekst m\u0259lumatlar\u0131ndan istifad\u0259 ed\u0259n olmal\u0131d\u0131r.\"\"\"\n",
-    "    \n",
-    "    # Get model deployment\n",
-    "    deployment = MODELS[model_name]['deployment']\n",
-    "    \n",
-    "    try:\n",
-    "        start_time = time.time()\n",
-    "        \n",
-    "        # GPT-5 models use max_completion_tokens, others use max_tokens\n",
-    "        if deployment.startswith('gpt-5'):\n",
-    "            response = azure_client.chat.completions.create(\n",
-    "                model=deployment,\n",
-    "                messages=[\n",
-    "                    {\"role\": \"user\", \"content\": prompt}\n",
-    "                ],\n",
-    "                temperature=temperature,\n",
-    "                max_completion_tokens=max_tokens\n",
-    "            )\n",
-    "        else:\n",
-    "            response = azure_client.chat.completions.create(\n",
-    "                model=deployment,\n",
-    "                messages=[\n",
-    "                    {\"role\": \"user\", \"content\": prompt}\n",
-    "                ],\n",
-    "                temperature=temperature,\n",
-    "                max_tokens=max_tokens\n",
-    "            )\n",
-    "        \n",
-    "        response_time = time.time() - start_time\n",
-    "        answer = response.choices[0].message.content\n",
-    "        \n",
-    "        return answer, response_time\n",
-    "    \n",
-    "    except Exception as e:\n",
-    "        return f\"ERROR: {str(e)}\", 0.0\n",
-    "\n",
-    "print(f\"\\n\u2705 Configured {len(LLM_MODELS)} LLM models for testing\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## 5. Evaluation Metrics"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "def normalize_text(text: str) -> str:\n",
-    "    \"\"\"Normalize text for comparison.\"\"\"\n",
-    "    import re\n",
-    "    text = text.lower().strip()\n",
-    "    text = re.sub(r'\\s+', ' ', text)\n",
-    "    return text\n",
-    "\n",
-    "def calculate_answer_similarity(reference: str, hypothesis: str) -> Dict[str, float]:\n",
-    "    \"\"\"\n",
-    "    Calculate similarity between generated and expected answer.\n",
-    "    Lower is better for error rates.\n",
-    "    \"\"\"\n",
-    "    ref_norm = normalize_text(reference)\n",
-    "    hyp_norm = normalize_text(hypothesis)\n",
-    "    \n",
-    "    # Character Error Rate\n",
-    "    cer_score = cer(ref_norm, hyp_norm) * 100\n",
-    "    \n",
-    "    # Word Error Rate  \n",
-    "    wer_score = wer(ref_norm, hyp_norm) * 100\n",
-    "    \n",
-    "    # Similarity scores (higher is better)\n",
-    "    similarity = max(0, 100 - wer_score)\n",
-    "    \n",
-    "    return {\n",
-    "        'CER': round(cer_score, 2),\n",
-    "        'WER': round(wer_score, 2),\n",
-    "        'Similarity': round(similarity, 2)\n",
-    "    }\n",
-    "\n",
-    "def check_citations(answer: str, documents: List[Dict]) -> Dict[str, any]:\n",
-    "    \"\"\"\n",
-    "    Check if answer includes proper citations.\n",
-    "    \"\"\"\n",
-    "    import re\n",
-    "    \n",
-    "    # Check for PDF names\n",
-    "    pdf_names = [doc['pdf_name'] for doc in documents]\n",
-    "    cited_pdfs = sum(1 for pdf in pdf_names if pdf.replace('.pdf', '') in answer)\n",
-    "    \n",
-    "    # Check for page numbers\n",
-    "    page_numbers = [str(doc['page_number']) for doc in documents]\n",
-    "    cited_pages = sum(1 for page in page_numbers if page in answer)\n",
-    "    \n",
-    "    # Check for source keywords\n",
-    "    source_keywords = ['m\u0259nb\u0259', 's\u0259n\u0259d', 's\u0259hif\u0259', 'pdf', 'document', 'page', 'source']\n",
-    "    has_source_ref = any(kw in answer.lower() for kw in source_keywords)\n",
-    "    \n",
-    "    citation_score = (\n",
-    "        (cited_pdfs / len(pdf_names) * 40) +  # 40% for PDF citation\n",
-    "        (cited_pages / len(page_numbers) * 40) +  # 40% for page citation\n",
-    "        (20 if has_source_ref else 0)  # 20% for having source keywords\n",
-    "    )\n",
-    "    \n",
-    "    return {\n",
-    "        'Citation_Score': round(citation_score, 2),\n",
-    "        'Cited_PDFs': cited_pdfs,\n",
-    "        'Cited_Pages': cited_pages,\n",
-    "        'Has_Source_Reference': has_source_ref\n",
-    "    }\n",
-    "\n",
-    "def evaluate_completeness(answer: str, min_length: int = 100) -> Dict[str, any]:\n",
-    "    \"\"\"\n",
-    "    Evaluate answer completeness.\n",
-    "    \"\"\"\n",
-    "    word_count = len(answer.split())\n",
-    "    char_count = len(answer)\n",
-    "    \n",
-    "    # Penalize very short or very long answers\n",
-    "    if char_count < min_length:\n",
-    "        completeness_score = (char_count / min_length) * 100\n",
-    "    elif char_count > 2000:\n",
-    "        completeness_score = 100 - ((char_count - 2000) / 2000 * 20)  # Penalty for verbosity\n",
-    "    else:\n",
-    "        completeness_score = 100\n",
-    "    \n",
-    "    return {\n",
-    "        'Completeness_Score': round(max(0, completeness_score), 2),\n",
-    "        'Word_Count': word_count,\n",
-    "        'Char_Count': char_count\n",
-    "    }\n",
-    "\n",
-    "print(\"\u2705 Evaluation functions ready\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## 6. Run Benchmark on All Models"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Select models to test (you can comment out models to skip)\n",
-    "MODELS_TO_TEST = [\n",
-    "    'Llama-4-Maverick-17B',\n",
-    "    'DeepSeek-R1',\n",
-    "    'GPT-4.1',\n",
-    "    'GPT-5-mini',\n",
-    "    'Claude-Sonnet-4.5',\n",
-    "    # 'Claude-Opus-4.1',  # Uncomment to test\n",
-    "    # 'Phi-4-multimodal',  # Uncomment to test\n",
-    "    # 'GPT-OSS-120B',  # Uncomment to test\n",
-    "]\n",
-    "\n",
-    "print(f\"Testing {len(MODELS_TO_TEST)} models on {len(questions)} questions...\\n\")\n",
-    "print(\"This may take several minutes...\\n\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Run benchmark\n",
-    "results = []\n",
-    "\n",
-    "for model_name in MODELS_TO_TEST:\n",
-    "    print(f\"\\n{'='*80}\")\n",
-    "    print(f\"Testing: {model_name}\")\n",
-    "    print(f\"{'='*80}\")\n",
-    "    \n",
-    "    model_results = []\n",
-    "    \n",
-    "    for example_key, messages in questions.items():\n",
-    "        # Get the last user message (the actual question)\n",
-    "        user_msg = [m for m in messages if m['role'] == 'user'][-1]\n",
-    "        query = user_msg['content']\n",
-    "        \n",
-    "        print(f\"\\n  Question {example_key}: {query[:80]}...\")\n",
-    "        \n",
-    "        # Retrieve documents\n",
-    "        documents = retrieve_documents(query, top_k=3)\n",
-    "        \n",
-    "        # Generate answer\n",
-    "        answer, response_time = generate_answer(model_name, query, documents)\n",
-    "        \n",
-    "        if answer.startswith('ERROR'):\n",
-    "            print(f\"  \u274c Failed: {answer}\")\n",
-    "            continue\n",
-    "        \n",
-    "        print(f\"  \u2705 Response time: {response_time:.2f}s\")\n",
-    "        \n",
-    "        # Get expected answer\n",
-    "        expected = expected_answers.get(example_key, {}).get('Answer', '')\n",
-    "        \n",
-    "        # Calculate metrics\n",
-    "        similarity_metrics = calculate_answer_similarity(expected, answer) if expected else {'CER': 0, 'WER': 0, 'Similarity': 0}\n",
-    "        citation_metrics = check_citations(answer, documents)\n",
-    "        completeness_metrics = evaluate_completeness(answer)\n",
-    "        \n",
-    "        # Store result\n",
-    "        result = {\n",
-    "            'Model': model_name,\n",
-    "            'Question': example_key,\n",
-    "            'Query': query[:100],\n",
-    "            'Answer': answer[:200] + '...',\n",
-    "            'Response_Time': round(response_time, 2),\n",
-    "            **similarity_metrics,\n",
-    "            **citation_metrics,\n",
-    "            **completeness_metrics,\n",
-    "            'Open_Source': MODELS[model_name]['open_source'],\n",
-    "            'Architecture_Score': MODELS[model_name]['architecture_score']\n",
-    "        }\n",
-    "        \n",
-    "        model_results.append(result)\n",
-    "        results.append(result)\n",
-    "    \n",
-    "    # Show summary for this model\n",
-    "    if model_results:\n",
-    "        avg_response_time = sum(r['Response_Time'] for r in model_results) / len(model_results)\n",
-    "        avg_similarity = sum(r['Similarity'] for r in model_results) / len(model_results)\n",
-    "        avg_citation = sum(r['Citation_Score'] for r in model_results) / len(model_results)\n",
-    "        avg_completeness = sum(r['Completeness_Score'] for r in model_results) / len(model_results)\n",
-    "        \n",
-    "        print(f\"\\n  \ud83d\udcca {model_name} Summary:\")\n",
-    "        print(f\"     Avg Response Time: {avg_response_time:.2f}s\")\n",
-    "        print(f\"     Avg Similarity: {avg_similarity:.1f}%\")\n",
-    "        print(f\"     Avg Citation Score: {avg_citation:.1f}%\")\n",
-    "        print(f\"     Avg Completeness: {avg_completeness:.1f}%\")\n",
-    "\n",
-    "print(f\"\\n{'='*80}\")\n",
-    "print(\"\u2705 Benchmarking complete!\")\n",
-    "print(f\"{'='*80}\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## 7. Aggregate Results and Rankings"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Create DataFrame\n",
-    "df = pd.DataFrame(results)\n",
-    "\n",
-    "# Calculate aggregate scores per model\n",
-    "model_summary = df.groupby('Model').agg({\n",
-    "    'Response_Time': 'mean',\n",
-    "    'Similarity': 'mean',\n",
-    "    'Citation_Score': 'mean',\n",
-    "    'Completeness_Score': 'mean',\n",
-    "    'CER': 'mean',\n",
-    "    'WER': 'mean',\n",
-    "    'Open_Source': 'first',\n",
-    "    'Architecture_Score': 'first'\n",
-    "}).round(2)\n",
-    "\n",
-    "# Calculate overall quality score (weighted average)\n",
-    "model_summary['Quality_Score'] = (\n",
-    "    model_summary['Similarity'] * 0.35 +  # 35% answer accuracy\n",
-    "    model_summary['Citation_Score'] * 0.35 +  # 35% citation quality\n",
-    "    model_summary['Completeness_Score'] * 0.30  # 30% completeness\n",
-    ").round(2)\n",
-    "\n",
-    "# Sort by Quality Score\n",
-    "model_summary = model_summary.sort_values('Quality_Score', ascending=False)\n",
-    "\n",
-    "# Display summary table\n",
-    "print(\"\\n\" + \"=\"*100)\n",
-    "print(\"\ud83d\udcca LLM BENCHMARKING RESULTS - MODEL SUMMARY\")\n",
-    "print(\"=\"*100)\n",
-    "print(model_summary.to_string())\n",
-    "print(\"=\"*100)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Create comprehensive visualization\n",
-    "import os\n",
-    "from pathlib import Path\n",
-    "\n",
-    "# Create output directory - using dynamic path\n",
-    "output_dir = OUTPUT_DIR / 'llm_benchmark'\n",
-    "output_dir.mkdir(parents=True, exist_ok=True)\n",
-    "\n",
-    "fig, axes = plt.subplots(2, 2, figsize=(16, 10))\n",
-    "\n",
-    "models = df['Model'].tolist()\n",
-    "colors = sns.color_palette('viridis', len(models))\n",
-    "\n",
-    "# 1. CSR - Character Success Rate (MAIN METRIC)\n",
-    "ax1 = axes[0, 0]\n",
-    "bars1 = ax1.barh(models, df['CSR'], color=colors)\n",
-    "ax1.set_xlabel('CSR (%) - Higher is Better', fontsize=12, fontweight='bold')\n",
-    "ax1.set_title('Character Success Rate (CSR)\\n\ud83c\udfc6 HACKATHON PRIMARY METRIC', \n",
-    "              fontsize=14, fontweight='bold')\n",
-    "ax1.set_xlim(0, 100)\n",
-    "for i, (model, csr) in enumerate(zip(models, df['CSR'])):\n",
-    "    ax1.text(csr + 1, i, f'{csr:.2f}%', va='center', fontsize=11, fontweight='bold')\n",
-    "ax1.axvline(x=90, color='green', linestyle='--', alpha=0.3, label='Excellent (>90%)')\n",
-    "ax1.axvline(x=80, color='orange', linestyle='--', alpha=0.3, label='Good (>80%)')\n",
-    "ax1.legend(fontsize=9)\n",
-    "\n",
-    "# 2. WSR - Word Success Rate\n",
-    "ax2 = axes[0, 1]\n",
-    "bars2 = ax2.barh(models, df['WSR'], color=colors)\n",
-    "ax2.set_xlabel('WSR (%) - Higher is Better', fontsize=12, fontweight='bold')\n",
-    "ax2.set_title('Word Success Rate (WSR)', fontsize=14, fontweight='bold')\n",
-    "ax2.set_xlim(0, 100)\n",
-    "for i, (model, wsr) in enumerate(zip(models, df['WSR'])):\n",
-    "    ax2.text(wsr + 1, i, f'{wsr:.2f}%', va='center', fontsize=11, fontweight='bold')\n",
-    "\n",
-    "# 3. Response Time\n",
-    "ax3 = axes[1, 0]\n",
-    "bars3 = ax3.barh(models, df['Response_Time'], color=colors)\n",
-    "ax3.set_xlabel('Total Time (seconds) - Lower is Better', fontsize=12, fontweight='bold')\n",
-    "ax3.set_title('Processing Speed', fontsize=14, fontweight='bold')\n",
-    "for i, (model, time_val) in enumerate(zip(models, df['Response_Time'])):\n",
-    "    ax3.text(time_val + 0.5, i, f'{time_val:.1f}s', va='center', fontsize=11)\n",
-    "\n",
-    "# 4. Error Rates Comparison\n",
-    "ax4 = axes[1, 1]\n",
-    "x = range(len(models))\n",
-    "width = 0.35\n",
-    "ax4.bar([i - width/2 for i in x], df['CER'], width, label='CER', color='coral', alpha=0.8)\n",
-    "ax4.bar([i + width/2 for i in x], df['WER'], width, label='WER', color='skyblue', alpha=0.8)\n",
-    "ax4.set_ylabel('Error Rate (%) - Lower is Better', fontsize=12, fontweight='bold')\n",
-    "ax4.set_title('Error Rates', fontsize=14, fontweight='bold')\n",
-    "ax4.set_xticks(x)\n",
-    "ax4.set_xticklabels(models, rotation=45, ha='right')\n",
-    "ax4.legend(fontsize=11)\n",
-    "ax4.grid(axis='y', alpha=0.3)\n",
-    "\n",
-    "plt.tight_layout()\n",
-    "plt.savefig(output_dir / 'results.png', dpi=300, bbox_inches='tight')\n",
-    "plt.show()\n",
-    "\n",
-    "print(f\"\\n\u2705 Visualization saved to '{output_dir}/results.png'\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Create rankings table\n",
-    "rankings = model_summary[[\n",
-    "    'Quality_Score', 'Similarity', 'Citation_Score', 'Completeness_Score', \n",
-    "    'Response_Time', 'Open_Source', 'Architecture_Score'\n",
-    "]].copy()\n",
-    "\n",
-    "rankings.insert(0, 'Rank', range(1, len(rankings) + 1))\n",
-    "\n",
-    "print(\"\\n\" + \"=\"*100)\n",
-    "print(\"\ud83c\udfc6 FINAL RANKINGS\")\n",
-    "print(\"=\"*100)\n",
-    "print(rankings.to_string())\n",
-    "print(\"=\"*100)\n",
-    "\n",
-    "# Winner analysis\n",
-    "best_overall = rankings.index[0]\n",
-    "best_open_source = rankings[rankings['Open_Source'] == True].index[0] if any(rankings['Open_Source']) else None\n",
-    "fastest = model_summary['Response_Time'].idxmin()\n",
-    "\n",
-    "print(\"\\n\" + \"=\"*100)\n",
-    "print(\"\ud83d\udca1 RECOMMENDATIONS FOR HACKATHON\")\n",
-    "print(\"=\"*100)\n",
-    "\n",
-    "print(f\"\\n\ud83e\udd47 Best Overall Quality: {best_overall}\")\n",
-    "print(f\"   Quality Score: {model_summary.loc[best_overall, 'Quality_Score']:.1f}%\")\n",
-    "print(f\"   Similarity: {model_summary.loc[best_overall, 'Similarity']:.1f}%\")\n",
-    "print(f\"   Citation Score: {model_summary.loc[best_overall, 'Citation_Score']:.1f}%\")\n",
-    "print(f\"   Response Time: {model_summary.loc[best_overall, 'Response_Time']:.2f}s\")\n",
-    "print(f\"   Open Source: {model_summary.loc[best_overall, 'Open_Source']}\")\n",
-    "print(f\"   Architecture Score: {model_summary.loc[best_overall, 'Architecture_Score']}\")\n",
-    "\n",
-    "if best_open_source:\n",
-    "    print(f\"\\n\ud83d\udd13 Best Open-Source Model: {best_open_source}\")\n",
-    "    print(f\"   Quality Score: {model_summary.loc[best_open_source, 'Quality_Score']:.1f}%\")\n",
-    "    print(f\"   Architecture Score: {model_summary.loc[best_open_source, 'Architecture_Score']} (Better for hackathon!)\")\n",
-    "    print(f\"   Response Time: {model_summary.loc[best_open_source, 'Response_Time']:.2f}s\")\n",
-    "\n",
-    "print(f\"\\n\u26a1 Fastest Model: {fastest}\")\n",
-    "print(f\"   Response Time: {model_summary.loc[fastest, 'Response_Time']:.2f}s\")\n",
-    "print(f\"   Quality Score: {model_summary.loc[fastest, 'Quality_Score']:.1f}%\")\n",
-    "\n",
-    "print(\"\\n\" + \"=\"*100)\n",
-    "print(\"\ud83d\udcdd FINAL RECOMMENDATION\")\n",
-    "print(\"=\"*100)\n",
-    "print(\"\\nScoring Breakdown:\")\n",
-    "print(\"  - LLM Quality: 30% of total hackathon score\")\n",
-    "print(\"  - Architecture: 20% of total hackathon score (open-source preferred!)\")\n",
-    "print(\"\\nBest Choice:\")\n",
-    "if best_open_source and model_summary.loc[best_open_source, 'Quality_Score'] >= model_summary.loc[best_overall, 'Quality_Score'] * 0.9:\n",
-    "    print(f\"  \u2705 {best_open_source} - Best balance of quality and architecture score\")\n",
-    "    print(f\"     Only {model_summary.loc[best_overall, 'Quality_Score'] - model_summary.loc[best_open_source, 'Quality_Score']:.1f}% quality drop for higher architecture score!\")\n",
-    "else:\n",
-    "    print(f\"  \u2705 {best_overall} - Highest quality, use if quality gap is significant\")\n",
-    "    if best_open_source:\n",
-    "        print(f\"  \u26a0\ufe0f  Consider {best_open_source} for higher architecture score (trade-off: {model_summary.loc[best_overall, 'Quality_Score'] - model_summary.loc[best_open_source, 'Quality_Score']:.1f}% quality)\")\n",
-    "\n",
-    "print(\"=\"*100)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Save results\n",
-    "from pathlib import Path\n",
-    "\n",
-    "# Using dynamic path\n",
-    "output_dir = OUTPUT_DIR / 'llm_benchmark'\n",
-    "output_dir.mkdir(parents=True, exist_ok=True)\n",
-    "\n",
-    "df.to_csv(output_dir / 'detailed_results.csv', index=False, encoding='utf-8')\n",
-    "model_summary.to_csv(output_dir / 'summary.csv', encoding='utf-8')\n",
-    "rankings.to_csv(output_dir / 'rankings.csv', index=False, encoding='utf-8')\n",
-    "\n",
-    "print(\"\\n\u2705 Results exported to output/llm_benchmark/:\")\n",
-    "print(\"   - detailed_results.csv (all questions and answers)\")\n",
-    "print(\"   - summary.csv (model averages)\")\n",
-    "print(\"   - rankings.csv (final rankings)\")\n",
-    "print(\"   - results.png (visualizations)\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## 11. Sample Answer Comparison"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "venv",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.10.12"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 4
-}

notebooks/rag_optimization_benchmark.ipynb.backup

@@ -1,1072 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# RAG Pipeline Optimization Benchmark\n",
-    "\n",
-    "**Comprehensive testing of ALL RAG components to maximize LLM Judge score**\n",
-    "\n",
-    "## What We're Testing:\n",
-    "\n",
-    "### 1. Embedding Models (Vector Representations)\n",
-    "- `BAAI/bge-large-en-v1.5` (Current - 1024 dim, best quality)\n",
-    "- `BAAI/bge-base-en-v1.5` (768 dim, faster)\n",
-    "- `intfloat/multilingual-e5-large` (1024 dim, multi-language)\n",
-    "- `sentence-transformers/paraphrase-multilingual-mpnet-base-v2` (768 dim, multilingual)\n",
-    "- `sentence-transformers/all-MiniLM-L6-v2` (384 dim, very fast)\n",
-    "\n",
-    "### 2. Retrieval Strategies\n",
-    "- **Top-K**: Test 1, 3, 5, 10 documents\n",
-    "- **MMR** (Maximal Marginal Relevance): Diversity vs relevance trade-off\n",
-    "- **Similarity Threshold**: Filter low-relevance docs\n",
-    "- **Reranking**: Use cross-encoder to rerank results\n",
-    "\n",
-    "### 3. Chunking Strategies (Already in Vector DB, but we'll compare)\n",
-    "- Chunk size: 256, 512, 600 (current), 1000 tokens\n",
-    "- Overlap: 0, 50, 100 (current), 200 chars\n",
-    "\n",
-    "### 4. LLM Models\n",
-    "- Llama-4-Maverick-17B (open-source)\n",
-    "- DeepSeek-R1 (reasoning)\n",
-    "- GPT-4.1, GPT-5, GPT-5-mini\n",
-    "- Claude-Sonnet-4.5\n",
-    "\n",
-    "### 5. Prompting Techniques\n",
-    "- **Baseline**: Simple context + question\n",
-    "- **Citation-focused**: Emphasize source references\n",
-    "- **Step-by-step**: Chain-of-thought reasoning\n",
-    "- **Few-shot**: Include example Q&A\n",
-    "\n",
-    "## LLM Judge Evaluation Criteria:\n",
-    "- **Accuracy** (35%): Answer correctness\n",
-    "- **Relevance** (35%): Citation quality and relevance\n",
-    "- **Completeness** (30%): Thorough answers"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# !pip install openai pinecone-client sentence-transformers rank-bm25 python-dotenv pandas matplotlib seaborn jiwer"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "/Users/ismatsamadov/SOCAR_Hackathon/venv/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
-      "  from .autonotebook import tqdm as notebook_tqdm\n"
-     ]
-    },
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "✅ Libraries loaded\n"
-     ]
-    }
-   ],
-   "source": [
-    "import os\n",
-    "import json\n",
-    "import time\n",
-    "import re\n",
-    "from typing import Dict, List, Tuple, Any\n",
-    "from collections import defaultdict\n",
-    "from dotenv import load_dotenv\n",
-    "\n",
-    "import pandas as pd\n",
-    "import matplotlib.pyplot as plt\n",
-    "import seaborn as sns\n",
-    "from openai import AzureOpenAI\n",
-    "from pinecone import Pinecone\n",
-    "from sentence_transformers import SentenceTransformer, CrossEncoder\n",
-    "from jiwer import wer, cer\n",
-    "import numpy as np\n",
-    "\n",
-    "load_dotenv()\n",
-    "\n",
-    "sns.set_style('whitegrid')\n",
-    "plt.rcParams['figure.figsize'] = (16, 10)\n",
-    "\n",
-    "print(\"✅ Libraries loaded\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "✅ Project root: /Users/ismatsamadov/SOCAR_Hackathon\n",
-      "✅ Docs directory: /Users/ismatsamadov/SOCAR_Hackathon/docs\n",
-      "✅ Output directory: /Users/ismatsamadov/SOCAR_Hackathon/output\n"
-     ]
-    }
-   ],
-   "source": [
-    "# Auto-detect project root (works from any directory)\n",
-    "import os\n",
-    "from pathlib import Path\n",
-    "\n",
-    "if Path('data').exists() and Path('docs').exists():\n",
-    "    # Already in project root\n",
-    "    PROJECT_ROOT = Path.cwd()\n",
-    "elif Path('../data').exists() and Path('../docs').exists():\n",
-    "    # In notebooks/ subdirectory\n",
-    "    PROJECT_ROOT = Path.cwd().parent\n",
-    "else:\n",
-    "    # Fallback: try to find project root\n",
-    "    current = Path.cwd()\n",
-    "    while current != current.parent:\n",
-    "        if (current / 'data').exists() and (current / 'docs').exists():\n",
-    "            PROJECT_ROOT = current\n",
-    "            break\n",
-    "        current = current.parent\n",
-    "    else:\n",
-    "        PROJECT_ROOT = Path.cwd()\n",
-    "\n",
-    "# Define all paths relative to project root\n",
-    "DATA_DIR = PROJECT_ROOT / 'data'\n",
-    "DOCS_DIR = PROJECT_ROOT / 'docs'\n",
-    "OUTPUT_DIR = PROJECT_ROOT / 'output'\n",
-    "\n",
-    "print(f\"✅ Project root: {PROJECT_ROOT}\")\n",
-    "print(f\"✅ Docs directory: {DOCS_DIR}\")\n",
-    "print(f\"✅ Output directory: {OUTPUT_DIR}\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 4,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "✅ Loaded 5 test questions\n",
-      "  - Example1\n",
-      "  - Example2\n",
-      "  - Example3\n",
-      "  - Example4\n",
-      "  - Example5\n"
-     ]
-    }
-   ],
-   "source": [
-    "# Load test cases - using dynamic paths\n",
-    "with open(DOCS_DIR / 'sample_questions.json', 'r', encoding='utf-8') as f:\n",
-    "    questions = json.load(f)\n",
-    "\n",
-    "with open(DOCS_DIR / 'sample_answers.json', 'r', encoding='utf-8') as f:\n",
-    "    expected_answers = json.load(f)\n",
-    "\n",
-    "print(f\"✅ Loaded {len(questions)} test questions\")\n",
-    "for key in questions.keys():\n",
-    "    print(f\"  - {key}\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## 2. Initialize Vector Database"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Connect to Pinecone\n",
-    "pc = Pinecone(api_key=os.getenv('PINECONE_API_KEY'))\n",
-    "index = pc.Index(os.getenv('PINECONE_INDEX_NAME', 'hackathon'))\n",
-    "\n",
-    "stats = index.describe_index_stats()\n",
-    "print(f\"✅ Vector DB connected\")\n",
-    "print(f\"   Total vectors: {stats['total_vector_count']}\")\n",
-    "print(f\"   Dimensions: {stats['dimension']}\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## 3. Embedding Models Configuration"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "EMBEDDING_MODELS = {\n",
-    "    'bge-large-en': {\n",
-    "        'name': 'BAAI/bge-large-en-v1.5',\n",
-    "        'dimensions': 1024,\n",
-    "        'notes': 'Current model - best quality'\n",
-    "    },\n",
-    "    'bge-base-en': {\n",
-    "        'name': 'BAAI/bge-base-en-v1.5',\n",
-    "        'dimensions': 768,\n",
-    "        'notes': 'Faster, slightly lower quality'\n",
-    "    },\n",
-    "    'multilingual-e5-large': {\n",
-    "        'name': 'intfloat/multilingual-e5-large',\n",
-    "        'dimensions': 1024,\n",
-    "        'notes': 'Multi-language optimized'\n",
-    "    },\n",
-    "    'paraphrase-multilingual': {\n",
-    "        'name': 'sentence-transformers/paraphrase-multilingual-mpnet-base-v2',\n",
-    "        'dimensions': 768,\n",
-    "        'notes': 'Good for Azerbaijani/Russian'\n",
-    "    },\n",
-    "    'all-MiniLM-L6': {\n",
-    "        'name': 'sentence-transformers/all-MiniLM-L6-v2',\n",
-    "        'dimensions': 384,\n",
-    "        'notes': 'Very fast, lower quality'\n",
-    "    }\n",
-    "}\n",
-    "\n",
-    "# Load embedding models (only test 1024-dim models for existing Pinecone index)\n",
-    "EMBEDDING_MODELS_TO_TEST = [\n",
-    "    'bge-large-en',  # Current\n",
-    "    'multilingual-e5-large',  # Alternative with same dims\n",
-    "]\n",
-    "\n",
-    "embedding_cache = {}\n",
-    "\n",
-    "for model_key in EMBEDDING_MODELS_TO_TEST:\n",
-    "    model_name = EMBEDDING_MODELS[model_key]['name']\n",
-    "    print(f\"Loading {model_key}...\")\n",
-    "    embedding_cache[model_key] = SentenceTransformer(model_name)\n",
-    "    print(f\"  ✅ {model_name}\")\n",
-    "\n",
-    "print(f\"\\n✅ Loaded {len(embedding_cache)} embedding models\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## 4. Retrieval Strategies"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "def retrieve_vanilla(query: str, embed_model: SentenceTransformer, top_k: int = 3) -> List[Dict]:\n",
-    "    \"\"\"\n",
-    "    Vanilla retrieval: Simple top-k vector search.\n",
-    "    \"\"\"\n",
-    "    query_embedding = embed_model.encode(query).tolist()\n",
-    "    results = index.query(vector=query_embedding, top_k=top_k, include_metadata=True)\n",
-    "    \n",
-    "    documents = []\n",
-    "    for match in results['matches']:\n",
-    "        documents.append({\n",
-    "            'pdf_name': match['metadata'].get('pdf_name', 'unknown.pdf'),\n",
-    "            'page_number': match['metadata'].get('page_number', 0),\n",
-    "            'content': match['metadata'].get('text', ''),\n",
-    "            'score': match.get('score', 0.0)\n",
-    "        })\n",
-    "    \n",
-    "    return documents\n",
-    "\n",
-    "\n",
-    "def retrieve_with_threshold(query: str, embed_model: SentenceTransformer, \n",
-    "                           top_k: int = 10, threshold: float = 0.7) -> List[Dict]:\n",
-    "    \"\"\"\n",
-    "    Retrieve with similarity threshold filtering.\n",
-    "    \"\"\"\n",
-    "    docs = retrieve_vanilla(query, embed_model, top_k=top_k)\n",
-    "    return [doc for doc in docs if doc['score'] >= threshold]\n",
-    "\n",
-    "\n",
-    "def retrieve_with_mmr(query: str, embed_model: SentenceTransformer, \n",
-    "                     top_k: int = 3, lambda_param: float = 0.5, fetch_k: int = 20) -> List[Dict]:\n",
-    "    \"\"\"\n",
-    "    MMR (Maximal Marginal Relevance) for diversity.\n",
-    "    lambda=1 → pure relevance, lambda=0 → pure diversity\n",
-    "    \"\"\"\n",
-    "    # Fetch more candidates\n",
-    "    candidates = retrieve_vanilla(query, embed_model, top_k=fetch_k)\n",
-    "    \n",
-    "    if len(candidates) <= top_k:\n",
-    "        return candidates[:top_k]\n",
-    "    \n",
-    "    # Query embedding\n",
-    "    query_emb = embed_model.encode(query)\n",
-    "    \n",
-    "    # Get embeddings for candidates\n",
-    "    candidate_texts = [doc['content'] for doc in candidates]\n",
-    "    candidate_embs = embed_model.encode(candidate_texts)\n",
-    "    \n",
-    "    # MMR algorithm\n",
-    "    selected = []\n",
-    "    selected_embs = []\n",
-    "    \n",
-    "    for _ in range(min(top_k, len(candidates))):\n",
-    "        mmr_scores = []\n",
-    "        \n",
-    "        for i, (doc, emb) in enumerate(zip(candidates, candidate_embs)):\n",
-    "            if i in [candidates.index(s) for s in selected]:\n",
-    "                mmr_scores.append(-float('inf'))\n",
-    "                continue\n",
-    "            \n",
-    "            # Relevance to query\n",
-    "            relevance = np.dot(query_emb, emb) / (np.linalg.norm(query_emb) * np.linalg.norm(emb))\n",
-    "            \n",
-    "            # Max similarity to already selected\n",
-    "            if selected_embs:\n",
-    "                similarities = [np.dot(emb, s_emb) / (np.linalg.norm(emb) * np.linalg.norm(s_emb)) \n",
-    "                              for s_emb in selected_embs]\n",
-    "                max_sim = max(similarities)\n",
-    "            else:\n",
-    "                max_sim = 0\n",
-    "            \n",
-    "            # MMR score\n",
-    "            mmr = lambda_param * relevance - (1 - lambda_param) * max_sim\n",
-    "            mmr_scores.append(mmr)\n",
-    "        \n",
-    "        # Select best MMR score\n",
-    "        best_idx = np.argmax(mmr_scores)\n",
-    "        selected.append(candidates[best_idx])\n",
-    "        selected_embs.append(candidate_embs[best_idx])\n",
-    "    \n",
-    "    return selected\n",
-    "\n",
-    "\n",
-    "def retrieve_with_reranking(query: str, embed_model: SentenceTransformer, \n",
-    "                           top_k: int = 3, fetch_k: int = 20) -> List[Dict]:\n",
-    "    \"\"\"\n",
-    "    Two-stage: retrieve with embeddings, rerank with cross-encoder.\n",
-    "    \"\"\"\n",
-    "    # Stage 1: Retrieve candidates\n",
-    "    candidates = retrieve_vanilla(query, embed_model, top_k=fetch_k)\n",
-    "    \n",
-    "    if len(candidates) <= top_k:\n",
-    "        return candidates[:top_k]\n",
-    "    \n",
-    "    # Stage 2: Rerank with cross-encoder\n",
-    "    reranker = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2')\n",
-    "    \n",
-    "    pairs = [[query, doc['content']] for doc in candidates]\n",
-    "    scores = reranker.predict(pairs)\n",
-    "    \n",
-    "    # Sort by reranker score\n",
-    "    scored_docs = [(doc, score) for doc, score in zip(candidates, scores)]\n",
-    "    scored_docs.sort(key=lambda x: x[1], reverse=True)\n",
-    "    \n",
-    "    # Update scores and return top-k\n",
-    "    reranked = []\n",
-    "    for doc, score in scored_docs[:top_k]:\n",
-    "        doc['rerank_score'] = float(score)\n",
-    "        reranked.append(doc)\n",
-    "    \n",
-    "    return reranked\n",
-    "\n",
-    "\n",
-    "RETRIEVAL_STRATEGIES = {\n",
-    "    'vanilla_k3': {'func': retrieve_vanilla, 'params': {'top_k': 3}, 'notes': 'Current setup'},\n",
-    "    'vanilla_k5': {'func': retrieve_vanilla, 'params': {'top_k': 5}, 'notes': 'More context'},\n",
-    "    'vanilla_k10': {'func': retrieve_vanilla, 'params': {'top_k': 10}, 'notes': 'Maximum context'},\n",
-    "    'threshold_0.7': {'func': retrieve_with_threshold, 'params': {'top_k': 10, 'threshold': 0.7}, 'notes': 'Quality filter'},\n",
-    "    'mmr_balanced': {'func': retrieve_with_mmr, 'params': {'top_k': 3, 'lambda_param': 0.5}, 'notes': 'Balance diversity'},\n",
-    "    'mmr_diverse': {'func': retrieve_with_mmr, 'params': {'top_k': 3, 'lambda_param': 0.3}, 'notes': 'More diversity'},\n",
-    "    'reranked_k3': {'func': retrieve_with_reranking, 'params': {'top_k': 3, 'fetch_k': 20}, 'notes': 'Two-stage rerank'},\n",
-    "}\n",
-    "\n",
-    "print(f\"✅ Configured {len(RETRIEVAL_STRATEGIES)} retrieval strategies\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## 5. LLM Models and Prompting Strategies"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Initialize Azure OpenAI\n",
-    "azure_client = AzureOpenAI(\n",
-    "    api_key=os.getenv('AZURE_OPENAI_API_KEY'),\n",
-    "    api_version=os.getenv('AZURE_OPENAI_API_VERSION', '2024-08-01-preview'),\n",
-    "    azure_endpoint=os.getenv('AZURE_OPENAI_ENDPOINT')\n",
-    ")\n",
-    "\n",
-    "LLM_MODELS = {\n",
-    "    'Llama-4-Maverick': 'Llama-4-Maverick-17B-128E-Instruct-FP8',\n",
-    "    'DeepSeek-R1': 'DeepSeek-R1',\n",
-    "    'GPT-4.1': 'gpt-4.1',\n",
-    "    'GPT-5-mini': 'gpt-5-mini',\n",
-    "    'Claude-Sonnet-4.5': 'claude-sonnet-4-5',\n",
-    "}\n",
-    "\n",
-    "# Prompting strategies\n",
-    "PROMPTING_STRATEGIES = {\n",
-    "    'baseline': \"\"\"\n",
-    "Siz SOCAR-ın tarixi neft və qaz sənədləri üzrə köməkçisiniz.\n",
-    "\n",
-    "Kontekst:\n",
-    "{context}\n",
-    "\n",
-    "Sual: {query}\n",
-    "\n",
-    "Kontekstə əsaslanaraq cavab verin.\n",
-    "\"\"\",\n",
-    "    \n",
-    "    'citation_focused': \"\"\"\n",
-    "Siz SOCAR-ın tarixi sənədlər üzrə mütəxəssis köməkçisisiniz.\n",
-    "\n",
-    "ÖNƏMLİ: Hər bir faktı mütləq mənbə ilə təsdiqləyin (PDF adı və səhifə nömrəsi).\n",
-    "\n",
-    "Kontekst:\n",
-    "{context}\n",
-    "\n",
-    "Sual: {query}\n",
-    "\n",
-    "Cavab verərkən:\n",
-    "1. Dəqiq faktlar yazın\n",
-    "2. Hər faktı mənbə ilə göstərin: (PDF: fayl_adı.pdf, Səhifə: X)\n",
-    "3. Kontekstdə olmayan məlumat əlavə etməyin\n",
-    "\"\"\",\n",
-    "    \n",
-    "    'step_by_step': \"\"\"\n",
-    "Siz SOCAR-ın tarixi sənədlər üzrə analitik köməkçisisiniz.\n",
-    "\n",
-    "Kontekst:\n",
-    "{context}\n",
-    "\n",
-    "Sual: {query}\n",
-    "\n",
-    "Addım-addım cavab verin:\n",
-    "1. Əvvəlcə kontekstdən əlaqəli məlumatları müəyyənləşdirin\n",
-    "2. Bu məlumatları təhlil edin\n",
-    "3. Nəticəni mənbələr ilə birlikdə təqdim edin\n",
-    "\"\"\",\n",
-    "    \n",
-    "    'few_shot': \"\"\"\n",
-    "Siz SOCAR-ın tarixi sənədlər üzrə mütəxəssis köməkçisisiniz.\n",
-    "\n",
-    "Nümunə:\n",
-    "Sual: \"Palçıq vulkanlarının təsir radiusu nə qədərdir?\"\n",
-    "Cavab: \"Sahə müşahidələri və modelləşdirmə göstərir ki, palçıq vulkanlarının təsir radiusu təqribən 10 km-dir (PDF: document_06.pdf, Səhifə: 5).\"\n",
-    "\n",
-    "Kontekst:\n",
-    "{context}\n",
-    "\n",
-    "Sual: {query}\n",
-    "\n",
-    "Yuxarıdakı nümunə kimi cavab verin - dəqiq, qısa, mənbə ilə.\n",
-    "\"\"\"\n",
-    "}\n",
-    "\n",
-    "print(f\"✅ Configured {len(LLM_MODELS)} LLM models\")\n",
-    "print(f\"✅ Configured {len(PROMPTING_STRATEGIES)} prompting strategies\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "def generate_answer(llm_model: str, query: str, documents: List[Dict], \n",
-    "                   prompt_strategy: str = 'baseline',\n",
-    "                   temperature: float = 0.2) -> Tuple[str, float]:\n",
-    "    \"\"\"\n",
-    "    Generate answer using LLM with specified prompting strategy.\n",
-    "    \"\"\"\n",
-    "    # Build context\n",
-    "    context_parts = []\n",
-    "    for i, doc in enumerate(documents, 1):\n",
-    "        context_parts.append(\n",
-    "            f\"Sənəd {i} (Mənbə: {doc['pdf_name']}, Səhifə {doc['page_number']}):\\n{doc['content']}\"\n",
-    "        )\n",
-    "    context = \"\\n\\n\".join(context_parts)\n",
-    "    \n",
-    "    # Get prompt template\n",
-    "    prompt_template = PROMPTING_STRATEGIES[prompt_strategy]\n",
-    "    prompt = prompt_template.format(context=context, query=query)\n",
-    "    \n",
-    "    try:\n",
-    "        start_time = time.time()\n",
-    "        \n",
-    "        deployment = LLM_MODELS[llm_model]\n",
-    "        \n",
-    "        # GPT-5 models use max_completion_tokens, others use max_tokens\n",
-    "        if deployment.startswith('gpt-5'):\n",
-    "            response = azure_client.chat.completions.create(\n",
-    "                model=deployment,\n",
-    "                messages=[{\"role\": \"user\", \"content\": prompt}],\n",
-    "                temperature=temperature,\n",
-    "                max_completion_tokens=1000\n",
-    "            )\n",
-    "        else:\n",
-    "            response = azure_client.chat.completions.create(\n",
-    "                model=deployment,\n",
-    "                messages=[{\"role\": \"user\", \"content\": prompt}],\n",
-    "                temperature=temperature,\n",
-    "                max_tokens=1000\n",
-    "            )\n",
-    "        \n",
-    "        elapsed = time.time() - start_time\n",
-    "        answer = response.choices[0].message.content\n",
-    "        \n",
-    "        return answer, elapsed\n",
-    "    \n",
-    "    except Exception as e:\n",
-    "        return f\"ERROR: {str(e)}\", 0.0\n",
-    "\n",
-    "print(\"✅ LLM generation function ready\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## 6. Evaluation Metrics"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "def normalize_text(text: str) -> str:\n",
-    "    text = text.lower().strip()\n",
-    "    text = re.sub(r'\\s+', ' ', text)\n",
-    "    return text\n",
-    "\n",
-    "def calculate_answer_quality(reference: str, hypothesis: str) -> Dict[str, float]:\n",
-    "    \"\"\"Accuracy metrics.\"\"\"\n",
-    "    ref_norm = normalize_text(reference)\n",
-    "    hyp_norm = normalize_text(hypothesis)\n",
-    "    \n",
-    "    cer_score = cer(ref_norm, hyp_norm) * 100\n",
-    "    wer_score = wer(ref_norm, hyp_norm) * 100\n",
-    "    similarity = max(0, 100 - wer_score)\n",
-    "    \n",
-    "    return {\n",
-    "        'Accuracy_Score': round(similarity, 2)\n",
-    "    }\n",
-    "\n",
-    "def evaluate_citation_quality(answer: str, documents: List[Dict]) -> Dict[str, float]:\n",
-    "    \"\"\"Relevance - citation quality.\"\"\"\n",
-    "    pdf_names = [doc['pdf_name'].replace('.pdf', '') for doc in documents]\n",
-    "    page_numbers = [str(doc['page_number']) for doc in documents]\n",
-    "    \n",
-    "    cited_pdfs = sum(1 for pdf in pdf_names if pdf in answer)\n",
-    "    cited_pages = sum(1 for page in page_numbers if page in answer)\n",
-    "    \n",
-    "    citation_keywords = ['mənbə', 'sənəd', 'səhifə', 'pdf', 'document', 'page']\n",
-    "    has_citation_format = any(kw in answer.lower() for kw in citation_keywords)\n",
-    "    \n",
-    "    citation_score = (\n",
-    "        (cited_pdfs / len(pdf_names) * 40) +\n",
-    "        (cited_pages / len(page_numbers) * 40) +\n",
-    "        (20 if has_citation_format else 0)\n",
-    "    )\n",
-    "    \n",
-    "    return {\n",
-    "        'Citation_Score': round(citation_score, 2),\n",
-    "        'Cited_PDFs': cited_pdfs,\n",
-    "        'Cited_Pages': cited_pages\n",
-    "    }\n",
-    "\n",
-    "def evaluate_retrieval_quality(query: str, documents: List[Dict], expected_answer: str) -> Dict[str, float]:\n",
-    "    \"\"\"Measure if retrieved docs are relevant to answer.\"\"\"\n",
-    "    if not documents or not expected_answer:\n",
-    "        return {'Retrieval_Relevance': 0.0}\n",
-    "    \n",
-    "    # Simple heuristic: check if expected answer words appear in retrieved docs\n",
-    "    expected_words = set(normalize_text(expected_answer).split())\n",
-    "    retrieved_text = ' '.join([doc['content'] for doc in documents])\n",
-    "    retrieved_words = set(normalize_text(retrieved_text).split())\n",
-    "    \n",
-    "    overlap = len(expected_words & retrieved_words) / len(expected_words) if expected_words else 0\n",
-    "    \n",
-    "    return {\n",
-    "        'Retrieval_Relevance': round(overlap * 100, 2)\n",
-    "    }\n",
-    "\n",
-    "def evaluate_completeness(answer: str) -> Dict[str, float]:\n",
-    "    \"\"\"Completeness metrics.\"\"\"\n",
-    "    word_count = len(answer.split())\n",
-    "    \n",
-    "    if word_count < 20:\n",
-    "        completeness = (word_count / 20) * 100\n",
-    "    elif word_count > 200:\n",
-    "        completeness = 100 - ((word_count - 200) / 200 * 20)\n",
-    "    else:\n",
-    "        completeness = 100\n",
-    "    \n",
-    "    return {\n",
-    "        'Completeness_Score': round(max(0, completeness), 2),\n",
-    "        'Word_Count': word_count\n",
-    "    }\n",
-    "\n",
-    "def calculate_llm_judge_score(accuracy: float, citation: float, completeness: float) -> float:\n",
-    "    \"\"\"Overall LLM Judge score (weighted).\"\"\"\n",
-    "    return round(\n",
-    "        accuracy * 0.35 +\n",
-    "        citation * 0.35 +\n",
-    "        completeness * 0.30,\n",
-    "        2\n",
-    "    )\n",
-    "\n",
-    "print(\"✅ Evaluation metrics ready\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## 7. Run Comprehensive Benchmark"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Configuration: Select what to test\n",
-    "CONFIGS_TO_TEST = [\n",
-    "    # Format: (embed_model, retrieval_strategy, llm_model, prompt_strategy)\n",
-    "    \n",
-    "    # Baseline (current setup)\n",
-    "    ('bge-large-en', 'vanilla_k3', 'Llama-4-Maverick', 'baseline'),\n",
-    "    \n",
-    "    # Test different embedding models\n",
-    "    ('multilingual-e5-large', 'vanilla_k3', 'Llama-4-Maverick', 'baseline'),\n",
-    "    \n",
-    "    # Test different retrieval strategies\n",
-    "    ('bge-large-en', 'vanilla_k5', 'Llama-4-Maverick', 'baseline'),\n",
-    "    ('bge-large-en', 'mmr_balanced', 'Llama-4-Maverick', 'baseline'),\n",
-    "    ('bge-large-en', 'reranked_k3', 'Llama-4-Maverick', 'baseline'),\n",
-    "    \n",
-    "    # Test different LLM models\n",
-    "    ('bge-large-en', 'vanilla_k3', 'GPT-5-mini', 'baseline'),\n",
-    "    ('bge-large-en', 'vanilla_k3', 'Claude-Sonnet-4.5', 'baseline'),\n",
-    "    \n",
-    "    # Test different prompting strategies\n",
-    "    ('bge-large-en', 'vanilla_k3', 'Llama-4-Maverick', 'citation_focused'),\n",
-    "    ('bge-large-en', 'vanilla_k3', 'Llama-4-Maverick', 'few_shot'),\n",
-    "    \n",
-    "    # Best combinations\n",
-    "    ('bge-large-en', 'reranked_k3', 'GPT-5-mini', 'citation_focused'),\n",
-    "    ('bge-large-en', 'mmr_balanced', 'Claude-Sonnet-4.5', 'citation_focused'),\n",
-    "]\n",
-    "\n",
-    "print(f\"Testing {len(CONFIGS_TO_TEST)} configurations on {len(questions)} questions\")\n",
-    "print(f\"Total API calls: ~{len(CONFIGS_TO_TEST) * len(questions)}\")\n",
-    "print(\"This will take 15-30 minutes...\\n\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Run benchmark\n",
-    "results = []\n",
-    "\n",
-    "for config_idx, (embed_key, retrieval_key, llm_key, prompt_key) in enumerate(CONFIGS_TO_TEST, 1):\n",
-    "    config_name = f\"{embed_key}_{retrieval_key}_{llm_key}_{prompt_key}\"\n",
-    "    \n",
-    "    print(f\"\\n{'='*100}\")\n",
-    "    print(f\"Config {config_idx}/{len(CONFIGS_TO_TEST)}: {config_name}\")\n",
-    "    print(f\"{'='*100}\")\n",
-    "    \n",
-    "    # Get components\n",
-    "    embed_model = embedding_cache[embed_key]\n",
-    "    retrieval_func = RETRIEVAL_STRATEGIES[retrieval_key]['func']\n",
-    "    retrieval_params = RETRIEVAL_STRATEGIES[retrieval_key]['params']\n",
-    "    \n",
-    "    config_results = []\n",
-    "    \n",
-    "    for example_key, messages in questions.items():\n",
-    "        user_msg = [m for m in messages if m['role'] == 'user'][-1]\n",
-    "        query = user_msg['content']\n",
-    "        \n",
-    "        print(f\"\\n  {example_key}: {query[:60]}...\")\n",
-    "        \n",
-    "        # Retrieve documents\n",
-    "        documents = retrieval_func(query, embed_model, **retrieval_params)\n",
-    "        print(f\"    Retrieved {len(documents)} docs\")\n",
-    "        \n",
-    "        # Generate answer\n",
-    "        answer, response_time = generate_answer(llm_key, query, documents, prompt_key)\n",
-    "        \n",
-    "        if answer.startswith('ERROR'):\n",
-    "            print(f\"    ❌ {answer}\")\n",
-    "            continue\n",
-    "        \n",
-    "        print(f\"    ✅ Generated in {response_time:.2f}s\")\n",
-    "        \n",
-    "        # Evaluate\n",
-    "        expected = expected_answers.get(example_key, {}).get('Answer', '')\n",
-    "        \n",
-    "        accuracy_metrics = calculate_answer_quality(expected, answer) if expected else {'Accuracy_Score': 0}\n",
-    "        citation_metrics = evaluate_citation_quality(answer, documents)\n",
-    "        retrieval_metrics = evaluate_retrieval_quality(query, documents, expected)\n",
-    "        completeness_metrics = evaluate_completeness(answer)\n",
-    "        \n",
-    "        # Calculate overall score\n",
-    "        llm_judge_score = calculate_llm_judge_score(\n",
-    "            accuracy_metrics['Accuracy_Score'],\n",
-    "            citation_metrics['Citation_Score'],\n",
-    "            completeness_metrics['Completeness_Score']\n",
-    "        )\n",
-    "        \n",
-    "        result = {\n",
-    "            'Config': config_name,\n",
-    "            'Embedding_Model': embed_key,\n",
-    "            'Retrieval_Strategy': retrieval_key,\n",
-    "            'LLM_Model': llm_key,\n",
-    "            'Prompt_Strategy': prompt_key,\n",
-    "            'Question': example_key,\n",
-    "            'Query': query[:80],\n",
-    "            'Num_Docs_Retrieved': len(documents),\n",
-    "            'Response_Time': round(response_time, 2),\n",
-    "            'LLM_Judge_Score': llm_judge_score,\n",
-    "            **accuracy_metrics,\n",
-    "            **citation_metrics,\n",
-    "            **retrieval_metrics,\n",
-    "            **completeness_metrics,\n",
-    "            'Answer_Preview': answer[:150]\n",
-    "        }\n",
-    "        \n",
-    "        results.append(result)\n",
-    "        config_results.append(result)\n",
-    "    \n",
-    "    # Show config summary\n",
-    "    if config_results:\n",
-    "        avg_score = sum(r['LLM_Judge_Score'] for r in config_results) / len(config_results)\n",
-    "        avg_time = sum(r['Response_Time'] for r in config_results) / len(config_results)\n",
-    "        print(f\"\\n  📊 Config Summary:\")\n",
-    "        print(f\"     Avg LLM Judge Score: {avg_score:.2f}%\")\n",
-    "        print(f\"     Avg Response Time: {avg_time:.2f}s\")\n",
-    "\n",
-    "print(f\"\\n{'='*100}\")\n",
-    "print(\"✅ Comprehensive benchmark complete!\")\n",
-    "print(f\"{'='*100}\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## 8. Analyze Results"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Create DataFrame\n",
-    "df = pd.DataFrame(results)\n",
-    "\n",
-    "# Aggregate by configuration\n",
-    "config_summary = df.groupby('Config').agg({\n",
-    "    'LLM_Judge_Score': 'mean',\n",
-    "    'Accuracy_Score': 'mean',\n",
-    "    'Citation_Score': 'mean',\n",
-    "    'Retrieval_Relevance': 'mean',\n",
-    "    'Completeness_Score': 'mean',\n",
-    "    'Response_Time': 'mean',\n",
-    "    'Embedding_Model': 'first',\n",
-    "    'Retrieval_Strategy': 'first',\n",
-    "    'LLM_Model': 'first',\n",
-    "    'Prompt_Strategy': 'first'\n",
-    "}).round(2)\n",
-    "\n",
-    "# Sort by LLM Judge Score\n",
-    "config_summary = config_summary.sort_values('LLM_Judge_Score', ascending=False)\n",
-    "\n",
-    "print(\"\\n\" + \"=\"*120)\n",
-    "print(\"📊 CONFIGURATION RANKINGS (By LLM Judge Score)\")\n",
-    "print(\"=\"*120)\n",
-    "display_cols = ['Embedding_Model', 'Retrieval_Strategy', 'LLM_Model', 'Prompt_Strategy', \n",
-    "                'LLM_Judge_Score', 'Accuracy_Score', 'Citation_Score', 'Response_Time']\n",
-    "print(config_summary[display_cols].to_string())\n",
-    "print(\"=\"*120)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## 9. Component Analysis"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Analyze impact of each component\n",
-    "print(\"\\n\" + \"=\"*100)\n",
-    "print(\"🔍 COMPONENT IMPACT ANALYSIS\")\n",
-    "print(\"=\"*100)\n",
-    "\n",
-    "# 1. Embedding Models\n",
-    "print(\"\\n📚 EMBEDDING MODELS:\")\n",
-    "embed_impact = df.groupby('Embedding_Model')['LLM_Judge_Score'].mean().sort_values(ascending=False)\n",
-    "for model, score in embed_impact.items():\n",
-    "    print(f\"   {model}: {score:.2f}%\")\n",
-    "\n",
-    "# 2. Retrieval Strategies\n",
-    "print(\"\\n🔎 RETRIEVAL STRATEGIES:\")\n",
-    "retrieval_impact = df.groupby('Retrieval_Strategy')['LLM_Judge_Score'].mean().sort_values(ascending=False)\n",
-    "for strategy, score in retrieval_impact.items():\n",
-    "    notes = RETRIEVAL_STRATEGIES[strategy]['notes']\n",
-    "    print(f\"   {strategy}: {score:.2f}% ({notes})\")\n",
-    "\n",
-    "# 3. LLM Models\n",
-    "print(\"\\n🤖 LLM MODELS:\")\n",
-    "llm_impact = df.groupby('LLM_Model')['LLM_Judge_Score'].mean().sort_values(ascending=False)\n",
-    "for model, score in llm_impact.items():\n",
-    "    print(f\"   {model}: {score:.2f}%\")\n",
-    "\n",
-    "# 4. Prompting Strategies\n",
-    "print(\"\\n💬 PROMPTING STRATEGIES:\")\n",
-    "prompt_impact = df.groupby('Prompt_Strategy')['LLM_Judge_Score'].mean().sort_values(ascending=False)\n",
-    "for strategy, score in prompt_impact.items():\n",
-    "    print(f\"   {strategy}: {score:.2f}%\")\n",
-    "\n",
-    "print(\"\\n\" + \"=\"*100)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "import os\n",
-    "from pathlib import Path\n",
-    "\n",
-    "# Create output directory - using dynamic path\n",
-    "output_dir = OUTPUT_DIR / 'rag_optimization_benchmark'\n",
-    "output_dir.mkdir(parents=True, exist_ok=True)\n",
-    "\n",
-    "fig, axes = plt.subplots(2, 3, figsize=(20, 12))\n",
-    "\n",
-    "# 1. Top Configurations\n",
-    "ax1 = axes[0, 0]\n",
-    "top_configs = config_summary.head(10)\n",
-    "config_labels = [c.split('_')[-2] + '+' + c.split('_')[-1] for c in top_configs.index]\n",
-    "ax1.barh(config_labels, top_configs['LLM_Judge_Score'], color=sns.color_palette('viridis', len(top_configs)))\n",
-    "ax1.set_xlabel('LLM Judge Score (%)', fontsize=11, fontweight='bold')\n",
-    "ax1.set_title('Top 10 Configurations', fontsize=13, fontweight='bold')\n",
-    "ax1.set_xlim(0, 100)\n",
-    "for i, score in enumerate(top_configs['LLM_Judge_Score']):\n",
-    "    ax1.text(score + 1, i, f'{score:.1f}', va='center', fontsize=10)\n",
-    "\n",
-    "# 2. Embedding Model Impact\n",
-    "ax2 = axes[0, 1]\n",
-    "ax2.bar(embed_impact.index, embed_impact.values, color='skyblue', alpha=0.8)\n",
-    "ax2.set_ylabel('Avg LLM Judge Score (%)', fontsize=11, fontweight='bold')\n",
-    "ax2.set_title('Embedding Model Impact', fontsize=13, fontweight='bold')\n",
-    "ax2.set_ylim(0, 100)\n",
-    "ax2.tick_params(axis='x', rotation=45)\n",
-    "for i, (model, score) in enumerate(embed_impact.items()):\n",
-    "    ax2.text(i, score + 2, f'{score:.1f}', ha='center', fontsize=10)\n",
-    "\n",
-    "# 3. Retrieval Strategy Impact\n",
-    "ax3 = axes[0, 2]\n",
-    "ax3.bar(retrieval_impact.index, retrieval_impact.values, color='coral', alpha=0.8)\n",
-    "ax3.set_ylabel('Avg LLM Judge Score (%)', fontsize=11, fontweight='bold')\n",
-    "ax3.set_title('Retrieval Strategy Impact', fontsize=13, fontweight='bold')\n",
-    "ax3.set_ylim(0, 100)\n",
-    "ax3.tick_params(axis='x', rotation=45)\n",
-    "for i, (strategy, score) in enumerate(retrieval_impact.items()):\n",
-    "    ax3.text(i, score + 2, f'{score:.1f}', ha='center', fontsize=9)\n",
-    "\n",
-    "# 4. LLM Model Impact\n",
-    "ax4 = axes[1, 0]\n",
-    "ax4.bar(llm_impact.index, llm_impact.values, color='mediumseagreen', alpha=0.8)\n",
-    "ax4.set_ylabel('Avg LLM Judge Score (%)', fontsize=11, fontweight='bold')\n",
-    "ax4.set_title('LLM Model Impact', fontsize=13, fontweight='bold')\n",
-    "ax4.set_ylim(0, 100)\n",
-    "ax4.tick_params(axis='x', rotation=45)\n",
-    "for i, (model, score) in enumerate(llm_impact.items()):\n",
-    "    ax4.text(i, score + 2, f'{score:.1f}', ha='center', fontsize=10)\n",
-    "\n",
-    "# 5. Prompting Strategy Impact\n",
-    "ax5 = axes[1, 1]\n",
-    "ax5.bar(prompt_impact.index, prompt_impact.values, color='mediumpurple', alpha=0.8)\n",
-    "ax5.set_ylabel('Avg LLM Judge Score (%)', fontsize=11, fontweight='bold')\n",
-    "ax5.set_title('Prompting Strategy Impact', fontsize=13, fontweight='bold')\n",
-    "ax5.set_ylim(0, 100)\n",
-    "ax5.tick_params(axis='x', rotation=45)\n",
-    "for i, (strategy, score) in enumerate(prompt_impact.items()):\n",
-    "    ax5.text(i, score + 2, f'{score:.1f}', ha='center', fontsize=10)\n",
-    "\n",
-    "# 6. Score Components (best config)\n",
-    "ax6 = axes[1, 2]\n",
-    "best_config = config_summary.iloc[0]\n",
-    "components = ['Accuracy', 'Citation', 'Completeness']\n",
-    "scores = [best_config['Accuracy_Score'], best_config['Citation_Score'], best_config['Completeness_Score']]\n",
-    "colors_comp = ['#FF6B6B', '#4ECDC4', '#45B7D1']\n",
-    "bars = ax6.bar(components, scores, color=colors_comp, alpha=0.8)\n",
-    "ax6.set_ylabel('Score (%)', fontsize=11, fontweight='bold')\n",
-    "ax6.set_title(f'Best Config Components\\n{best_config.name.split(\"_\")[2]}', fontsize=13, fontweight='bold')\n",
-    "ax6.set_ylim(0, 100)\n",
-    "for i, score in enumerate(scores):\n",
-    "    ax6.text(i, score + 2, f'{score:.1f}%', ha='center', fontsize=10, fontweight='bold')\n",
-    "\n",
-    "plt.tight_layout()\n",
-    "plt.savefig(output_dir / 'results.png', dpi=300, bbox_inches='tight')\n",
-    "plt.show()\n",
-    "\n",
-    "print(f\"\\n✅ Visualization saved to '{output_dir}/results.png'\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": "import os\nfrom pathlib import Path\n\n# Create output directory - using dynamic path\noutput_dir = OUTPUT_DIR / 'rag_optimization_benchmark'\noutput_dir.mkdir(parents=True, exist_ok=True)\n\nfig, axes = plt.subplots(2, 3, figsize=(20, 12))\n\n# 1. Top Configurations\nax1 = axes[0, 0]\ntop_configs = config_summary.head(10)\nconfig_labels = [c.split('_')[-2] + '+' + c.split('_')[-1] for c in top_configs.index]\nax1.barh(config_labels, top_configs['LLM_Judge_Score'], color=sns.color_palette('viridis', len(top_configs)))\nax1.set_xlabel('LLM Judge Score (%)', fontsize=11, fontweight='bold')\nax1.set_title('Top 10 Configurations', fontsize=13, fontweight='bold')\nax1.set_xlim(0, 100)\nfor i, score in enumerate(top_configs['LLM_Judge_Score']):\n    ax1.text(score + 1, i, f'{score:.1f}', va='center', fontsize=10)\n\n# 2. Embedding Model Impact\nax2 = axes[0, 1]\nax2.bar(embed_impact.index, embed_impact.values, color='skyblue', alpha=0.8)\nax2.set_ylabel('Avg LLM Judge Score (%)', fontsize=11, fontweight='bold')\nax2.set_title('Embedding Model Impact', fontsize=13, fontweight='bold')\nax2.set_ylim(0, 100)\nax2.tick_params(axis='x', rotation=45)\nfor i, (model, score) in enumerate(embed_impact.items()):\n    ax2.text(i, score + 2, f'{score:.1f}', ha='center', fontsize=10)\n\n# 3. Retrieval Strategy Impact\nax3 = axes[0, 2]\nax3.bar(retrieval_impact.index, retrieval_impact.values, color='coral', alpha=0.8)\nax3.set_ylabel('Avg LLM Judge Score (%)', fontsize=11, fontweight='bold')\nax3.set_title('Retrieval Strategy Impact', fontsize=13, fontweight='bold')\nax3.set_ylim(0, 100)\nax3.tick_params(axis='x', rotation=45)\nfor i, (strategy, score) in enumerate(retrieval_impact.items()):\n    ax3.text(i, score + 2, f'{score:.1f}', ha='center', fontsize=9)\n\n# 4. LLM Model Impact\nax4 = axes[1, 0]\nax4.bar(llm_impact.index, llm_impact.values, color='mediumseagreen', alpha=0.8)\nax4.set_ylabel('Avg LLM Judge Score (%)', fontsize=11, fontweight='bold')\nax4.set_title('LLM Model Impact', fontsize=13, fontweight='bold')\nax4.set_ylim(0, 100)\nax4.tick_params(axis='x', rotation=45)\nfor i, (model, score) in enumerate(llm_impact.items()):\n    ax4.text(i, score + 2, f'{score:.1f}', ha='center', fontsize=10)\n\n# 5. Prompting Strategy Impact\nax5 = axes[1, 1]\nax5.bar(prompt_impact.index, prompt_impact.values, color='mediumpurple', alpha=0.8)\nax5.set_ylabel('Avg LLM Judge Score (%)', fontsize=11, fontweight='bold')\nax5.set_title('Prompting Strategy Impact', fontsize=13, fontweight='bold')\nax5.set_ylim(0, 100)\nax5.tick_params(axis='x', rotation=45)\nfor i, (strategy, score) in enumerate(prompt_impact.items()):\n    ax5.text(i, score + 2, f'{score:.1f}', ha='center', fontsize=10)\n\n# 6. Score Components (best config)\nax6 = axes[1, 2]\nbest_config = config_summary.iloc[0]\ncomponents = ['Accuracy', 'Citation', 'Completeness']\nscores = [best_config['Accuracy_Score'], best_config['Citation_Score'], best_config['Completeness_Score']]\ncolors_comp = ['#FF6B6B', '#4ECDC4', '#45B7D1']\nbars = ax6.bar(components, scores, color=colors_comp, alpha=0.8)\nax6.set_ylabel('Score (%)', fontsize=11, fontweight='bold')\nax6.set_title(f'Best Config Components\\n{best_config.name.split(\"_\")[2]}', fontsize=13, fontweight='bold')\nax6.set_ylim(0, 100)\nfor i, score in enumerate(scores):\n    ax6.text(i, score + 2, f'{score:.1f}%', ha='center', fontsize=10, fontweight='bold')\n\nplt.tight_layout()\nplt.savefig(output_dir / 'results.png', dpi=300, bbox_inches='tight')\nplt.show()\n\nprint(f\"\\n✅ Visualization saved to '{output_dir}/results.png'\")"
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "best_config = config_summary.iloc[0]\n",
-    "\n",
-    "print(\"\\n\" + \"=\"*100)\n",
-    "print(\"🏆 OPTIMAL RAG CONFIGURATION\")\n",
-    "print(\"=\"*100)\n",
-    "\n",
-    "print(f\"\\n✅ Best Configuration: {best_config.name}\")\n",
-    "print(f\"\\n📊 Performance:\")\n",
-    "print(f\"   LLM Judge Score: {best_config['LLM_Judge_Score']:.2f}%\")\n",
-    "print(f\"   Accuracy: {best_config['Accuracy_Score']:.2f}%\")\n",
-    "print(f\"   Citation Quality: {best_config['Citation_Score']:.2f}%\")\n",
-    "print(f\"   Completeness: {best_config['Completeness_Score']:.2f}%\")\n",
-    "print(f\"   Avg Response Time: {best_config['Response_Time']:.2f}s\")\n",
-    "\n",
-    "print(f\"\\n⚙️ Components:\")\n",
-    "print(f\"   Embedding Model: {best_config['Embedding_Model']}\")\n",
-    "print(f\"      → {EMBEDDING_MODELS[best_config['Embedding_Model']]['name']}\")\n",
-    "print(f\"   Retrieval Strategy: {best_config['Retrieval_Strategy']}\")\n",
-    "print(f\"      → {RETRIEVAL_STRATEGIES[best_config['Retrieval_Strategy']]['notes']}\")\n",
-    "print(f\"   LLM Model: {best_config['LLM_Model']}\")\n",
-    "print(f\"   Prompting Strategy: {best_config['Prompt_Strategy']}\")\n",
-    "\n",
-    "print(f\"\\n💡 Key Findings:\")\n",
-    "print(f\"   1. Best Embedding: {embed_impact.index[0]} ({embed_impact.values[0]:.2f}%)\")\n",
-    "print(f\"   2. Best Retrieval: {retrieval_impact.index[0]} ({retrieval_impact.values[0]:.2f}%)\")\n",
-    "print(f\"   3. Best LLM: {llm_impact.index[0]} ({llm_impact.values[0]:.2f}%)\")\n",
-    "print(f\"   4. Best Prompt: {prompt_impact.index[0]} ({prompt_impact.values[0]:.2f}%)\")\n",
-    "\n",
-    "print(f\"\\n🎯 Hackathon Impact:\")\n",
-    "print(f\"   LLM Quality = 30% of total score\")\n",
-    "print(f\"   Your score: {best_config['LLM_Judge_Score']:.2f}% × 30% = {best_config['LLM_Judge_Score'] * 0.3:.2f} points\")\n",
-    "\n",
-    "baseline = df[df['Config'].str.contains('baseline')].iloc[0] if len(df[df['Config'].str.contains('baseline')]) > 0 else None\n",
-    "if baseline is not None:\n",
-    "    improvement = best_config['LLM_Judge_Score'] - baseline['LLM_Judge_Score']\n",
-    "    print(f\"\\n📈 Improvement vs Baseline:\")\n",
-    "    print(f\"   +{improvement:.2f}% quality improvement\")\n",
-    "    print(f\"   = +{improvement * 0.3:.2f} hackathon points\")\n",
-    "\n",
-    "print(\"\\n\" + \"=\"*100)\n",
-    "print(\"📝 IMPLEMENTATION CHECKLIST\")\n",
-    "print(\"=\"*100)\n",
-    "print(f\"\\n1. Use embedding model: {EMBEDDING_MODELS[best_config['Embedding_Model']]['name']}\")\n",
-    "print(f\"2. Implement retrieval: {best_config['Retrieval_Strategy']}\")\n",
-    "print(f\"3. Use LLM model: {best_config['LLM_Model']}\")\n",
-    "print(f\"4. Apply prompt: {best_config['Prompt_Strategy']}\")\n",
-    "print(f\"\\n5. Expected performance:\")\n",
-    "print(f\"   - LLM Judge Score: {best_config['LLM_Judge_Score']:.2f}%\")\n",
-    "print(f\"   - Response time: ~{best_config['Response_Time']:.1f}s\")\n",
-    "print(\"=\"*100)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Save results\n",
-    "from pathlib import Path\n",
-    "\n",
-    "# Using dynamic path\n",
-    "output_dir = OUTPUT_DIR / 'rag_optimization_benchmark'\n",
-    "output_dir.mkdir(parents=True, exist_ok=True)\n",
-    "\n",
-    "df.to_csv(output_dir / 'detailed_results.csv', index=False, encoding='utf-8')\n",
-    "config_summary.to_csv(output_dir / 'summary.csv', encoding='utf-8')\n",
-    "\n",
-    "# Save component impacts\n",
-    "impacts = pd.DataFrame({\n",
-    "    'Embedding_Impact': embed_impact,\n",
-    "    'Retrieval_Impact': retrieval_impact.reindex(embed_impact.index, fill_value=0),\n",
-    "    'LLM_Impact': llm_impact.reindex(embed_impact.index, fill_value=0),\n",
-    "    'Prompt_Impact': prompt_impact.reindex(embed_impact.index, fill_value=0)\n",
-    "}).fillna(0)\n",
-    "impacts.to_csv(output_dir / 'component_impacts.csv', encoding='utf-8')\n",
-    "\n",
-    "print(\"\\n✅ Results exported to output/rag_optimization_benchmark/:\")\n",
-    "print(\"   - detailed_results.csv (all tests)\")\n",
-    "print(\"   - summary.csv (config rankings)\")\n",
-    "print(\"   - component_impacts.csv (component analysis)\")\n",
-    "print(\"   - results.png (visualizations)\")"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "venv",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.10.12"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 4
-}

notebooks/{requirements_rag_optimization.txt → requirements.txt}

@@ -1,19 +1,24 @@
-# RAG Pipeline Optimization Requirements
-# Install with: pip install -r requirements_rag_optimization.txt
+# Notebooks Requirements
+# All dependencies for VLM OCR, LLM Benchmark, and RAG Optimization notebooks
+# Install with: pip install -r requirements.txt
 
-# Azure OpenAI
+# Azure OpenAI (for all notebooks)
 openai==1.54.0
 
-# Vector Database
+# Vector Database (for LLM and RAG notebooks)
 pinecone-client==5.0.0
 
-# Embeddings and Reranking
+# Embeddings and Transformers (for LLM and RAG notebooks)
 sentence-transformers==3.3.1
 
-# Metrics
+# PDF Processing (for VLM OCR notebook)
+PyMuPDF==1.23.8
+Pillow==10.1.0
+
+# Metrics (for all notebooks)
 jiwer==3.0.3
 
-# Data analysis and visualization
+# Data Analysis and Visualization (for all notebooks)
 pandas==2.1.3
 matplotlib==3.8.2
 seaborn==0.13.0

notebooks/requirements_vlm_ocr.txt

@@ -1,24 +0,0 @@
-# VLM OCR Benchmarking Requirements
-# Install with: pip install -r requirements_vlm_ocr.txt
-
-# Azure OpenAI client (for vision models)
-openai==1.54.0
-
-# PDF processing
-PyMuPDF==1.23.8
-Pillow==10.1.0
-
-# Metrics
-jiwer==3.0.3
-
-# Data analysis and visualization
-pandas==2.1.3
-matplotlib==3.8.2
-seaborn==0.13.0
-
-# Utilities
-python-dotenv==1.0.0
-
-# Jupyter
-jupyter==1.0.0
-ipykernel==6.27.1

scripts/README.md

@@ -0,0 +1,143 @@
+# Scripts Directory
+
+One-time utility scripts for SOCAR Hackathon project.
+
+## Available Scripts
+
+### 📊 Data Management
+
+#### `check_pinecone.py`
+Check Pinecone vector database status and statistics.
+
+```bash
+python scripts/check_pinecone.py
+```
+
+**Output:**
+- Total vector count
+- Index dimensions
+- Namespaces (if any)
+- Connection status
+
+#### `clear_pinecone.py`
+Clear all data from Pinecone index before re-ingestion.
+
+```bash
+python scripts/clear_pinecone.py
+```
+
+**⚠️ WARNING**: This deletes ALL vectors! Requires typing 'DELETE' to confirm.
+
+**Use case:**
+- Before re-ingesting documents with new chunking strategy
+- Testing with fresh data
+- Cleaning up after experiments
+
+### 🤖 Azure OpenAI
+
+#### `list_azure_models.py`
+List all deployed Azure OpenAI models.
+
+```bash
+python scripts/list_azure_models.py
+```
+
+**Output:**
+- Vision models (GPT-4.1, GPT-5, Claude, etc.)
+- Text models (Llama, DeepSeek, etc.)
+- Total count and categorization
+
+**Use case:**
+- Verify which models are deployed
+- Check model availability before updating notebooks
+- Debugging 404 errors
+
+## Setup
+
+All scripts use environment variables from `.env` file:
+
+```bash
+# Required in .env
+PINECONE_API_KEY=your_key
+PINECONE_INDEX_NAME=hackathon
+AZURE_OPENAI_API_KEY=your_key
+AZURE_OPENAI_ENDPOINT=your_endpoint
+```
+
+## Dependencies
+
+Scripts use the same dependencies as the main project:
+- `python-dotenv` - Environment variables
+- `pinecone-client` - Vector database
+- `openai` - Azure OpenAI
+
+Install from project root:
+```bash
+pip install -r notebooks/requirements.txt
+```
+
+## Common Workflows
+
+### Re-ingesting Documents
+
+```bash
+# 1. Check current data
+python scripts/check_pinecone.py
+
+# 2. Clear existing data
+python scripts/clear_pinecone.py
+
+# 3. Run ingestion script (not included - create as needed)
+# python scripts/ingest_documents.py
+
+# 4. Verify new data
+python scripts/check_pinecone.py
+```
+
+### Verifying Model Availability
+
+```bash
+# List all deployed models
+python scripts/list_azure_models.py
+
+# Check if specific model exists in output
+python scripts/list_azure_models.py | grep "Llama-3.2-Vision"
+```
+
+## Adding New Scripts
+
+When creating new scripts:
+1. Add descriptive docstring at top
+2. Use environment variables from `.env`
+3. Include error handling with helpful messages
+4. Update this README with usage instructions
+5. Follow existing naming convention: `verb_noun.py`
+
+## Examples
+
+### Safe Pinecone Cleanup
+```python
+# First check what's there
+$ python scripts/check_pinecone.py
+Total Vectors: 1,300
+Dimensions: 1024
+
+# Then clear if needed
+$ python scripts/clear_pinecone.py
+⚠️  WARNING: This will delete ALL 1,300 vectors!
+Type 'DELETE' to confirm: DELETE
+✅ Deletion completed!
+```
+
+### Check Vision Models
+```python
+$ python scripts/list_azure_models.py
+
+🖼️  Vision Models (6):
+   ✅ gpt-4.1
+   ✅ gpt-5
+   ✅ gpt-5-mini
+   ✅ claude-sonnet-4-5
+   ✅ claude-opus-4-1
+   ✅ Phi-4-multimodal-instruct
+```

scripts/__init__.py

@@ -0,0 +1 @@
+"""SOCAR Hackathon utility scripts"""

scripts/check_pinecone.py

@@ -0,0 +1,62 @@
+"""
+Check Pinecone index status and statistics
+Quick utility to inspect vector database
+"""
+
+import os
+from dotenv import load_dotenv
+from pinecone import Pinecone
+
+# Load environment variables
+load_dotenv()
+
+def check_pinecone_status():
+    """Display Pinecone index information"""
+
+    try:
+        # Initialize Pinecone
+        pc = Pinecone(api_key=os.getenv('PINECONE_API_KEY'))
+        index_name = os.getenv('PINECONE_INDEX_NAME', 'hackathon')
+        index = pc.Index(index_name)
+
+        # Get index statistics
+        stats = index.describe_index_stats()
+
+        print("="*80)
+        print("PINECONE INDEX STATUS")
+        print("="*80)
+
+        print(f"\n📊 Index Information:")
+        print(f"   Name: {index_name}")
+        print(f"   Total Vectors: {stats.get('total_vector_count', 0):,}")
+        print(f"   Dimensions: {stats.get('dimension', 'N/A')}")
+
+        # Check namespaces if any
+        if 'namespaces' in stats and stats['namespaces']:
+            print(f"\n📁 Namespaces:")
+            for ns_name, ns_stats in stats['namespaces'].items():
+                ns_display = ns_name if ns_name else "(default)"
+                print(f"   {ns_display}: {ns_stats.get('vector_count', 0):,} vectors")
+
+        # Index configuration
+        print(f"\n⚙️  Configuration:")
+        print(f"   API Key: {os.getenv('PINECONE_API_KEY')[:10]}..." if os.getenv('PINECONE_API_KEY') else "   API Key: Not set")
+
+        # Connection status
+        if stats.get('total_vector_count', 0) > 0:
+            print(f"\n✅ Status: Connected and populated")
+        else:
+            print(f"\n⚠️  Status: Connected but empty")
+
+    except Exception as e:
+        print("="*80)
+        print("PINECONE CONNECTION ERROR")
+        print("="*80)
+        print(f"\n❌ Error: {e}")
+        print("\nPlease check:")
+        print("  1. PINECONE_API_KEY in .env file")
+        print("  2. PINECONE_INDEX_NAME in .env file")
+        print("  3. Index exists in your Pinecone account")
+
+if __name__ == "__main__":
+    check_pinecone_status()

scripts/clear_pinecone.py

@@ -0,0 +1,70 @@
+"""
+Clear all data from Pinecone index
+One-time script for data cleanup before re-ingestion
+"""
+
+import os
+from dotenv import load_dotenv
+from pinecone import Pinecone
+
+# Load environment variables
+load_dotenv()
+
+def clear_pinecone_index():
+    """Delete all vectors from Pinecone index"""
+
+    # Initialize Pinecone
+    pc = Pinecone(api_key=os.getenv('PINECONE_API_KEY'))
+    index = pc.Index(os.getenv('PINECONE_INDEX_NAME', 'hackathon'))
+
+    # Get current stats
+    stats = index.describe_index_stats()
+    total_vectors = stats['total_vector_count']
+
+    print("="*80)
+    print("PINECONE DATA CLEANUP")
+    print("="*80)
+    print(f"\nIndex: {os.getenv('PINECONE_INDEX_NAME', 'hackathon')}")
+    print(f"Current vectors: {total_vectors}")
+    print(f"Dimensions: {stats.get('dimension', 'N/A')}")
+
+    if total_vectors == 0:
+        print("\n✅ Index is already empty. Nothing to delete.")
+        return
+
+    # Confirm deletion
+    print(f"\n⚠️  WARNING: This will delete ALL {total_vectors} vectors!")
+    confirm = input("Type 'DELETE' to confirm: ")
+
+    if confirm != 'DELETE':
+        print("\n❌ Deletion cancelled. No data was removed.")
+        return
+
+    print("\n🗑️  Deleting all vectors...")
+
+    try:
+        # Delete all vectors
+        index.delete(delete_all=True)
+
+        print("✅ Deletion completed!")
+
+        # Verify deletion
+        import time
+        time.sleep(2)  # Wait for deletion to propagate
+
+        stats = index.describe_index_stats()
+        remaining = stats['total_vector_count']
+
+        print(f"\n📊 Final status:")
+        print(f"   Remaining vectors: {remaining}")
+
+        if remaining == 0:
+            print("   ✅ Index successfully cleared!")
+        else:
+            print(f"   ⚠️  {remaining} vectors still remain (may need a moment to sync)")
+
+    except Exception as e:
+        print(f"\n❌ Error during deletion: {e}")
+
+if __name__ == "__main__":
+    clear_pinecone_index()

scripts/list_azure_models.py

@@ -0,0 +1,76 @@
+"""
+List all deployed Azure OpenAI models
+Useful for verifying available models
+"""
+
+import os
+from dotenv import load_dotenv
+from openai import AzureOpenAI
+
+# Load environment variables
+load_dotenv()
+
+def list_azure_models():
+    """List all deployed Azure OpenAI models"""
+
+    try:
+        client = AzureOpenAI(
+            api_key=os.getenv('AZURE_OPENAI_API_KEY'),
+            api_version=os.getenv('AZURE_OPENAI_API_VERSION', '2024-08-01-preview'),
+            azure_endpoint=os.getenv('AZURE_OPENAI_ENDPOINT')
+        )
+
+        print("="*80)
+        print("AZURE OPENAI DEPLOYED MODELS")
+        print("="*80)
+
+        # List models
+        models = client.models.list()
+
+        print(f"\n📊 Total Models: {len(list(models))}")
+        print(f"\nDeployed Models:")
+        print("-" * 80)
+
+        model_list = []
+        for model in models:
+            model_list.append({
+                'id': model.id,
+                'created': model.created if hasattr(model, 'created') else 'N/A'
+            })
+
+        # Sort by id
+        model_list.sort(key=lambda x: x['id'])
+
+        # Categorize models
+        vision_models = []
+        text_models = []
+
+        for model in model_list:
+            model_id = model['id']
+            if any(keyword in model_id.lower() for keyword in ['vision', 'multimodal', 'gpt-4.1', 'gpt-5', 'claude']):
+                vision_models.append(model_id)
+            else:
+                text_models.append(model_id)
+
+        print(f"\n🖼️  Vision Models ({len(vision_models)}):")
+        for model_id in vision_models:
+            print(f"   ✅ {model_id}")
+
+        print(f"\n📝 Text Models ({len(text_models)}):")
+        for model_id in text_models:
+            print(f"   ✅ {model_id}")
+
+        print("\n" + "="*80)
+
+    except Exception as e:
+        print("="*80)
+        print("AZURE OPENAI CONNECTION ERROR")
+        print("="*80)
+        print(f"\n❌ Error: {e}")
+        print("\nPlease check:")
+        print("  1. AZURE_OPENAI_API_KEY in .env file")
+        print("  2. AZURE_OPENAI_ENDPOINT in .env file")
+        print("  3. API version compatibility")
+
+if __name__ == "__main__":
+    list_azure_models()

test_api.py

@@ -0,0 +1,82 @@
+"""
+Quick test script for SOCAR LLM API
+"""
+
+import requests
+import json
+from docs.sample_questions import questions
+
+# API base URL
+BASE_URL = "http://localhost:8000"
+
+def test_health():
+    """Test health endpoint"""
+    print("🔍 Testing health endpoint...")
+    response = requests.get(f"{BASE_URL}/health")
+    print(f"Status: {response.status_code}")
+    print(json.dumps(response.json(), indent=2))
+    print()
+
+def test_root():
+    """Test root endpoint"""
+    print("🔍 Testing root endpoint...")
+    response = requests.get(BASE_URL)
+    print(f"Status: {response.status_code}")
+    print(json.dumps(response.json(), indent=2))
+    print()
+
+def test_llm(question: str):
+    """Test LLM endpoint"""
+    print(f"🔍 Testing LLM endpoint...")
+    print(f"Question: {question}\n")
+
+    payload = {
+        "messages": [
+            {"role": "user", "content": question}
+        ],
+        "temperature": 0.2,
+        "max_tokens": 1000
+    }
+
+    response = requests.post(f"{BASE_URL}/llm", json=payload)
+    print(f"Status: {response.status_code}")
+
+    if response.status_code == 200:
+        result = response.json()
+        print(f"Response time: {result['response_time']}s")
+        print(f"Model: {result['model']}")
+        print(f"\nAnswer:\n{result['response']}")
+        print(f"\nSources:")
+        for source in result['sources']:
+            print(f"  - {source['pdf_name']}, Page {source['page_number']} (score: {source['relevance_score']})")
+    else:
+        print(f"Error: {response.text}")
+    print()
+
+if __name__ == "__main__":
+    print("="*80)
+    print("SOCAR LLM API Test Suite")
+    print("="*80)
+    print()
+
+    # Test health
+    try:
+        test_health()
+    except Exception as e:
+        print(f"❌ Health check failed: {e}\n")
+
+    # Test root
+    try:
+        test_root()
+    except Exception as e:
+        print(f"❌ Root endpoint failed: {e}\n")
+
+    # Test LLM with sample question
+    try:
+        test_llm("Palçıq vulkanlarının təsir radiusu nə qədərdir?")
+    except Exception as e:
+        print(f"❌ LLM endpoint failed: {e}\n")
+
+    print("="*80)
+    print("✅ Test suite completed!")
+    print("="*80)