Refactor project structure

.gitignore

@@ -1,53 +1,65 @@
-# Credentials & Secrets
+# Security — CRITICAL
 .env
-.env.*
-!.env.example
-config/railway.json
-*.pem
-*.key
-*.p12
-*.pfx
-*.crt
-credentials.json
-secrets.json
-config.json
-*secret*
-*credential*
-*token*
-*apikey*
-
-# ML/Data artifacts
-/data/
-/models/
-*.pkl
-*.h5
-*.pt
-*.pth
-*.onnx
-*.bin
-mlruns/
-mlflow/
+.env.local
+.env.*.local
+*.db
+*.sqlite3
 
-# Runtime & Cache
-.venv/
-venv/
+# Python cache
 __pycache__/
 *.pyc
 *.pyo
+*.egg-info/
 .pytest_cache/
 .mypy_cache/
-*.log
-logs/
-test.db
-*.sqlite
-*.db
+.ruff_cache/
+.coverage
+htmlcov/
+
+# Virtual environment
+.venv/
+venv/
+env/
+
+# ML artifacts (large — tracked via DVC)
+*.pt
+*.pth
+*.bin
+*.safetensors
+*.ckpt
+*.h5
+*.pkl
+*.joblib
+*.onnx
+
+# MLflow tracking (can be large)
+mlruns/
+
+# Data files (tracked via DVC)
+data/raw/
+data/processed/
+*.csv
+*.parquet
+*.jsonl
+
+# Jupyter
+.ipynb_checkpoints/
+
+# Agent/IDE configs (optional — decide per tool)
+# .claude/
+# .opencode/
+
+# OS
 .DS_Store
 Thumbs.db
 
-# Docker & Infra
-docker-compose.override.yml
-*-secrets.yml
-*-credentials.yml
+# IDE
+.vscode/
+.idea/
+
+# Logs
+*.log
+logs/
 
-# Miscellaneous
+# Node
 node_modules/

README.md

@@ -10,7 +10,7 @@ A production-ready, highly optimized Aspect-Based Sentiment Analysis (ABSA) syst
 
 The system leverages state-of-the-art models like **XLM-RoBERTa** and **IndicBERT**, optimized using ONNX runtime, and includes a highly robust, zero-download, pure-Python rule-based fallback engine for instant inference.
 
-## ✨ Key Features
+## Key Features
 
 - **Multilingual Support**: First-class support for English, Hindi, and code-mixed Hinglish.
 - **Dual Inference Engine**:
@@ -18,37 +18,37 @@ The system leverages state-of-the-art models like **XLM-RoBERTa** and **IndicBER
   - **Rule-Based Fallback**: An instantaneous, pure-Python fallback leveraging a curated multi-lingual lexicon to handle aspect extraction and sentiment scoring without any heavy downloads.
 - **Modern Tech Stack**: 
   - **Backend**: Asynchronous, high-performance API built with FastAPI.
-  - **Frontend**: A sleek, responsive dashboard built with React and TailwindCSS. Features real-time predictions, batch analytics, and system monitoring.
+  - **Frontend**: A responsive dashboard built with React and TailwindCSS. Features real-time predictions, batch analytics, and system monitoring.
 - **MLOps Integrated**: Complete integration with DVC (Data Version Control) for pipeline reproducibility, MLflow for experiment tracking, and Evidently AI for data drift monitoring.
 - **Scalable Architecture**: Support for async tasks via Celery + Redis, robust data storage via PostgreSQL, and metric exporting using Prometheus.
 
-## 🐍 Python-First Architecture
+## Python-First Architecture
 
 This repository is designed following a **Python-first paradigm**:
 - **Python (67.5%)**: Handling all business logic, data processing, configuration, API routing, ML inference, and utility functions using FastAPI and Python data science libraries.
 - **JavaScript/TypeScript (32.5%)**: Strictly limited to the frontend `dashboard/` directory, used *only* for the React UI, component rendering, browser events, and client-side state. 
 - *Note: There is no backend or ML logic written in JavaScript.*
 
-## 🏗️ Repository Structure
+## Repository Structure
 
 ```text
 Multilingual-Absa/
-├── api/                    # FastAPI backend and inference services
-├── config/                 # DVC and Docker configuration files
+├── api/app/                # FastAPI backend and inference services
+├── config/                 # Docker and application configuration files
 ├── dashboard/              # React frontend for inference & monitoring
 ├── data/                   # Dataset directory (DVC-tracked)
-├── docs/                   # Extended documentation
-├── mlflow/                 # MLflow tracking
+├── docs/                   # Extended documentation (architecture, ml, api)
+├── ml/                     # ML training, notebooks, MLflow, and tracking
 ├── models/                 # Model artifacts (DVC-tracked)
 ├── monitoring/             # Monitoring configurations (Prometheus, Evidently)
-├── notebooks/              # Exploratory Data Analysis & Prototyping
 ├── scripts/                # Utility and automation scripts
-├── src/                    # Machine Learning pipeline source code
+├── src/                    # Core Machine Learning pipeline source code
 ├── tests/                  # Unit and integration test suite
+├── dvc.yaml                # DVC pipeline orchestration
 └── requirements.txt        # Python dependencies
 ```
 
-## 🚀 Getting Started
+## Getting Started
 
 ### Prerequisites
 - Python 3.10+
@@ -78,7 +78,7 @@ cp .env.example .env
 
 The easiest way to get the entire stack (API, Dashboard, Redis, Postgres) running is via Docker Compose:
 ```bash
-docker-compose -f config/docker/docker-compose.yml up --build
+docker-compose -f deployment/docker/docker-compose.yml up --build
 ```
 
 ### 3. Manual ML Pipeline Execution (DVC)
@@ -94,8 +94,7 @@ dvc push         # Push newly generated artifacts to remote
 
 **Start the API Server:**
 ```bash
-cd api
-uvicorn main:app --reload --host 0.0.0.0 --port 8000
+PYTHONPATH=. uvicorn api.app.main:app --reload --host 0.0.0.0 --port 8000
 ```
 *API Documentation will be available at `http://localhost:8000/docs`.*
 
@@ -107,13 +106,13 @@ npm run dev
 ```
 *Access the dashboard at `http://localhost:5173`.*
 
-## 🔬 How it Works
+## How it Works
 
 1. **Prediction API**: When a review is submitted, the language is auto-detected.
 2. **Inference**: The `ABSAPipeline` attempts to load INT8 quantized ONNX models for extraction and sentiment scoring. 
 3. **Fallback Mechanism**: If the custom models are not downloaded, the engine automatically falls back to a dictionary/rule-based engine tailored for product reviews, guaranteeing zero downtime and instant availability.
 4. **Monitoring**: All predictions are logged. Performance metrics and data drift are tracked via Evidently and Prometheus.
 
-## 🛡️ License
+## License
 
 This project is licensed under the MIT License - see the LICENSE file for details.

api/{main.py → app/main.py}

@@ -5,11 +5,11 @@ from fastapi.middleware.cors import CORSMiddleware
 
 load_dotenv()
 
-from api.routes import predict, results  # noqa: E402
-from api.middleware.metrics import instrumentator  # noqa: E402
-from api.services.absa_pipeline import pipeline  # noqa: E402
-from api.models.db_models import Base  # noqa: E402
-from api.middleware.dependencies import engine  # noqa: E402
+from api.app.routes import predict, results  # noqa: E402
+from api.app.middleware.metrics import instrumentator  # noqa: E402
+from api.app.services.absa_pipeline import pipeline  # noqa: E402
+from api.app.schemas.db_models import Base  # noqa: E402
+from api.app.middleware.dependencies import engine  # noqa: E402
 
 
 @asynccontextmanager

api/{routes → app/routes}/predict.py

@@ -6,11 +6,11 @@ import uuid
 import tempfile
 import time
 
-from api.models.schemas import ReviewInput, PredictionResponse, BatchJobResponse
-from api.models.db_models import Review, AspectResult, BatchJob
-from api.middleware.dependencies import get_db
-from api.services.absa_pipeline import pipeline
-from api.tasks.batch_tasks import process_batch
+from api.app.schemas.schemas import ReviewInput, PredictionResponse, BatchJobResponse
+from api.app.schemas.db_models import Review, AspectResult, BatchJob
+from api.app.middleware.dependencies import get_db
+from api.app.services.absa_pipeline import pipeline
+from api.app.tasks.batch_tasks import process_batch
 
 router = APIRouter()
 

api/{services → app/services}/absa_pipeline.py

@@ -18,8 +18,8 @@ from pathlib import Path
 from typing import List, Tuple
 import numpy as np
 
-from api.models.schemas import PredictionResponse, AspectSentiment
-from api.services.lang_service import lang_service
+from api.app.schemas.schemas import PredictionResponse, AspectSentiment
+from api.app.services.lang_service import lang_service
 
 # ── Optional heavy imports (ONNX custom models) ───────────────────────────────
 try:

api/{tasks → app/tasks}/batch_tasks.py

@@ -1,7 +1,7 @@
-from api.tasks import celery_app
-from api.services.absa_pipeline import pipeline
-from api.middleware.dependencies import SessionLocal
-from api.models.db_models import BatchJob, AspectResult, Review
+from api.app.tasks import celery_app
+from api.app.services.absa_pipeline import pipeline
+from api.app.middleware.dependencies import SessionLocal
+from api.app.schemas.db_models import BatchJob, AspectResult, Review
 import pandas as pd
 import os
 import csv

config/railway.json

@@ -0,0 +1,14 @@
+{
+  "$schema": "https://railway.app/railway.schema.json",
+  "build": {
+    "builder": "DOCKERFILE",
+    "dockerfilePath": "docker/Dockerfile.api.prod"
+  },
+  "deploy": {
+    "startCommand": "uvicorn api.main:app --host 0.0.0.0 --port $PORT",
+    "healthcheckPath": "/health",
+    "healthcheckTimeout": 30,
+    "restartPolicyType": "ON_FAILURE",
+    "restartPolicyMaxRetries": 3
+  }
+}

data/models/lid.176.ftz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8f3472cfe8738a7b6099e8e999c3cbfae0dcd15696aac7d7738a8039db603e83
+size 938013

mlruns/1/f7e324a252a94812b0d4158371276cb0/artifacts/confusion_matrix.json

@@ -1,28 +0,0 @@
-{
-  "confusion_matrix": [
-    [
-      246,
-      26,
-      31,
-      13
-    ],
-    [
-      14,
-      120,
-      26,
-      7
-    ],
-    [
-      20,
-      29,
-      60,
-      1
-    ],
-    [
-      3,
-      7,
-      0,
-      3
-    ]
-  ]
-}

models/onnx/README.md

@@ -0,0 +1,55 @@
+---
+language:
+- en
+- hi
+tags:
+- sentiment-analysis
+- aspect-based-sentiment-analysis
+- onnx
+- int8
+- xlm-roberta
+---
+
+# Multilingual ABSA (Aspect-Based Sentiment Analysis)
+
+## Model Description
+This repository contains INT8-quantized ONNX models for Multilingual Aspect-Based Sentiment Analysis (ABSA).
+It uses a two-stage pipeline:
+1. **Aspect Extraction**: Token classification model to identify aspects in text.
+2. **Sentiment Classification**: Sequence classification model to determine sentiment (Positive, Negative, Neutral, Conflict) for extracted aspects.
+
+Both models are based on `xlm-roberta-base`, fine-tuned using QLoRA, and exported to ONNX for CPU-optimized inference.
+
+## Languages Supported
+- English (en)
+- Hindi (hi)
+- Hinglish (code-mixed)
+
+## Performance Metrics (Phase 4)
+- **English**: Macro-F1 > 78%
+- **Hindi**: Macro-F1 > 65%
+- **Latency (INT8 CPU)**: P95 < 300ms
+
+## Usage
+```python
+from optimum.onnxruntime import ORTModelForTokenClassification, ORTModelForSequenceClassification
+from transformers import AutoTokenizer
+
+model_id = "YOUR_HF_USERNAME/multilingual-absa"
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+
+# Load Aspect Extraction Model
+aspect_model = ORTModelForTokenClassification.from_pretrained(
+    model_id, 
+    subfolder="aspect_extraction_int8"
+)
+
+# Load Sentiment Model
+sentiment_model = ORTModelForSequenceClassification.from_pretrained(
+    model_id,
+    subfolder="sentiment_int8"
+)
+```
+
+## Training Data
+Fine-tuned on combined SemEval (English) and translated/native Hindi product review datasets.

reorganize.sh

@@ -0,0 +1,39 @@
+#!/bin/bash
+set -e
+
+echo "Starting reorganization..."
+
+# 1. API Restructure
+mkdir -p api/app/routes api/app/core api/app/schemas api/app/services api/app/middleware api/app/tasks
+touch api/app/__init__.py api/app/routes/__init__.py api/app/core/__init__.py api/app/schemas/__init__.py api/app/services/__init__.py api/app/middleware/__init__.py api/app/tasks/__init__.py
+
+mv api/main.py api/app/ 2>/dev/null || true
+mv api/routes/* api/app/routes/ 2>/dev/null || true
+mv api/models/* api/app/schemas/ 2>/dev/null || true
+mv api/services/* api/app/services/ 2>/dev/null || true
+mv api/middleware/* api/app/middleware/ 2>/dev/null || true
+mv api/tasks/* api/app/tasks/ 2>/dev/null || true
+
+# Cleanup empty api dirs
+rm -d api/routes api/models api/services api/middleware api/tasks 2>/dev/null || true
+
+# 2. SRC Restructure
+mkdir -p src/absa src/languages/english src/languages/hindi src/languages/hinglish src/models src/utils
+touch src/absa/__init__.py src/languages/__init__.py src/languages/english/__init__.py src/languages/hindi/__init__.py src/languages/hinglish/__init__.py src/models/__init__.py src/utils/__init__.py
+
+# Move items in src to their respective folders (will do this manually or cautiously later, right now let's just create the folders)
+# Since I need to check what exists in src/ before moving.
+
+# 3. ML Restructure
+mkdir -p ml/experiments ml/notebooks ml/configs ml/training
+mv mlflow ml/experiments/ 2>/dev/null || true
+mv mlruns ml/experiments/ 2>/dev/null || true
+mv notebooks/* ml/notebooks/ 2>/dev/null || true
+rm -d notebooks 2>/dev/null || true
+
+# 4. Docs Restructure
+mkdir -p docs/architecture docs/ml docs/api docs/planning docs/guides
+mv .planning/* docs/planning/ 2>/dev/null || true
+rm -d .planning 2>/dev/null || true
+
+echo "Reorganization completed."

tests/api/test_api.py

@@ -1,10 +1,10 @@
 import pytest
 from fastapi.testclient import TestClient
 import os
-os.environ["DATABASE_URL"] = "sqlite:///./test.db"
+os.environ["DATABASE_URL"] = "sqlite:///./tests/fixtures/test.db"
 import unittest.mock as mock
 
-from api.main import app
+from api.app.main import app
 import json
 import io
 
@@ -44,7 +44,7 @@ def test_batch_upload():
     with TestClient(app) as client:
         csv_content = "text\nThe food was great\nTerrible service"
         files = {"file": ("test.csv", io.BytesIO(csv_content.encode("utf-8")), "text/csv")}
-        with mock.patch("api.routes.predict.process_batch.delay") as mock_delay:
+        with mock.patch("api.app.routes.predict.process_batch.delay") as mock_delay:
             response = client.post("/batch", files=files)
             assert response.status_code == 200
             data = response.json()