Expand architecture documentation

Add detailed documentation for the API, database, deployment, security, tech stack, and system design. Update the main architecture doc to reflect the full multilingual ABSA pipeline, deployment flow, and MLOps stack.

docs/API_DOCUMENTATION.md

@@ -0,0 +1,240 @@
+# API Documentation — Multilingual ABSA
+
+## Base URL
+
+- Local development: `http://localhost:8000`
+- Production: `https://your-railway-app.up.railway.app`
+
+## Authentication
+
+Currently **none**. All endpoints are publicly accessible.
+
+## Endpoints
+
+### POST /predict
+
+Analyze a single review for aspect-based sentiment.
+
+**Request Body:**
+```json
+{
+  "text": "The food was great but the service was terrible.",
+  "language": "en"
+}
+```
+
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| `text` | string | Yes | Review text to analyze |
+| `language` | string | No | Force language (`"en"`, `"hi"`, `"hinglish"`). Auto-detected if omitted |
+
+**Response `200`:**
+
+```json
+{
+  "text": "The food was great but the service was terrible.",
+  "language": "en",
+  "detected_language": "en",
+  "aspects": [
+    {
+      "aspect": "Food",
+      "sentiment": "positive",
+      "confidence": 0.85,
+      "start": 4,
+      "end": 8
+    },
+    {
+      "aspect": "Service",
+      "sentiment": "negative",
+      "confidence": 0.82,
+      "start": 27,
+      "end": 34
+    }
+  ],
+  "processing_time_ms": 185.3
+}
+```
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `text` | string | Original input text |
+| `language` | string | Language used (detected or forced) |
+| `detected_language` | string | Auto-detected language code |
+| `aspects` | array | List of extracted aspect-sentiment pairs |
+| `processing_time_ms` | float | Total inference time in milliseconds |
+
+**Aspect Object:**
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `aspect` | string | Extracted aspect term (title-cased) |
+| `sentiment` | string | `"positive"`, `"negative"`, `"neutral"`, or `"conflict"` |
+| `confidence` | float | Confidence score (0.0–1.0) |
+| `start` | int | Character offset start in original text |
+| `end` | int | Character offset end in original text |
+
+**Error Responses:**
+
+| Status | Condition |
+|--------|-----------|
+| 422 | Empty text, missing `text` field |
+| 500 | Model inference failure |
+
+---
+
+### POST /batch
+
+Upload a CSV file for batch analysis. Processed asynchronously via Celery.
+
+**Request:** `multipart/form-data`
+
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| `file` | file | Yes | CSV file with a `text` column (max 10,000 rows) |
+
+**Response `200`:**
+
+```json
+{
+  "job_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
+  "status": "queued",
+  "total_reviews": 4250,
+  "processed": 0,
+  "result_url": null
+}
+```
+
+**Error Responses:**
+
+| Status | Condition |
+|--------|-----------|
+| 422 | Non-CSV file, missing `text` column, >10K rows |
+| 500 | Batch processing failed |
+
+---
+
+### GET /status/{job_id}
+
+Poll batch job progress.
+
+**Response `200` (processing):**
+
+```json
+{
+  "job_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
+  "status": "processing",
+  "total_reviews": 4250,
+  "processed": 1200,
+  "result_url": null
+}
+```
+
+**Response `200` (completed):**
+
+```json
+{
+  "job_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
+  "status": "completed",
+  "total_reviews": 4250,
+  "processed": 4250,
+  "result_url": "/results/download/a1b2c3d4-e5f6-7890-abcd-ef1234567890"
+}
+```
+
+**Error Responses:**
+
+| Status | Condition |
+|--------|-----------|
+| 404 | Job ID not found |
+
+---
+
+### GET /health
+
+System health check.
+
+**Response `200`:**
+
+```json
+{
+  "status": "ok",
+  "model": "loaded",
+  "db": "connected"
+}
+```
+
+---
+
+### GET /info
+
+Get model metadata.
+
+**Response `200`:**
+
+```json
+{
+  "model_name": "xlm-roberta-base-absa",
+  "version": "1.0",
+  "supported_languages": "en, hi",
+  "max_batch_size": "10000"
+}
+```
+
+---
+
+### GET /metrics
+
+Prometheus metrics endpoint (auto-instrumented).
+
+**Response `200`:** Prometheus text format metrics.
+
+Available metrics:
+- `fastapi_requests_total` (counter by method, path, status)
+- `fastapi_requests_duration_seconds` (histogram)
+- `fastapi_requests_inprogress` (gauge)
+- Custom ABSA metrics (if implemented)
+
+---
+
+## Example Usage
+
+### cURL
+
+```bash
+# Single prediction
+curl -X POST http://localhost:8000/predict \
+  -H "Content-Type: application/json" \
+  -d '{"text": "This phone has amazing battery life but the camera is disappointing", "language": "en"}'
+
+# Health check
+curl http://localhost:8000/health
+
+# Model info
+curl http://localhost:8000/info
+```
+
+### Python
+
+```python
+import httpx
+
+response = httpx.post(
+    "http://localhost:8000/predict",
+    json={"text": "This phone has amazing battery life but the camera is disappointing"}
+)
+print(response.json())
+```
+
+### JavaScript
+
+```javascript
+const response = await fetch('http://localhost:8000/predict', {
+  method: 'POST',
+  headers: { 'Content-Type': 'application/json' },
+  body: JSON.stringify({
+    text: 'This phone has amazing battery life but the camera is disappointing'
+  })
+});
+const data = await response.json();
+console.log(data);
+```

docs/DATABASE.md

@@ -0,0 +1,116 @@
+# Database Design — Multilingual ABSA
+
+## ER Diagram
+
+```mermaid
+erDiagram
+    Review {
+        uuid id PK
+        text text
+        string language
+        datetime created_at
+        float processing_time_ms
+    }
+    AspectResult {
+        uuid id PK
+        uuid review_id FK
+        string aspect
+        string sentiment
+        float confidence
+        int start_pos
+        int end_pos
+    }
+    BatchJob {
+        uuid id PK
+        string status
+        int total
+        int processed
+        datetime created_at
+        datetime completed_at
+    }
+
+    Review ||--o{ AspectResult : "has aspects"
+```
+
+## Schema
+
+```sql
+CREATE TABLE reviews (
+    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+    text TEXT NOT NULL,
+    language VARCHAR(10) NOT NULL,
+    created_at TIMESTAMPTZ DEFAULT NOW(),
+    processing_time_ms FLOAT NOT NULL
+);
+
+CREATE TABLE aspect_results (
+    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+    review_id UUID NOT NULL REFERENCES reviews(id),
+    aspect VARCHAR(255) NOT NULL,
+    sentiment VARCHAR(50) NOT NULL,
+    confidence FLOAT NOT NULL,
+    start_pos INTEGER NOT NULL,
+    end_pos INTEGER NOT NULL
+);
+
+CREATE TABLE batch_jobs (
+    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+    status VARCHAR(50) NOT NULL DEFAULT 'queued',
+    total INTEGER NOT NULL,
+    processed INTEGER NOT NULL DEFAULT 0,
+    created_at TIMESTAMPTZ DEFAULT NOW(),
+    completed_at TIMESTAMPTZ
+);
+```
+
+## Recommended Indexes (Production)
+
+```sql
+CREATE INDEX idx_aspect_results_review_id ON aspect_results(review_id);
+CREATE INDEX idx_reviews_created_at ON reviews(created_at);
+CREATE INDEX idx_reviews_language ON reviews(language);
+CREATE INDEX idx_batch_jobs_status ON batch_jobs(status);
+```
+
+## Connection Configuration
+
+```python
+# Development (SQLite)
+DATABASE_URL = "sqlite:///absa.db"
+engine = create_engine(DATABASE_URL, connect_args={"check_same_thread": False})
+
+# Production (PostgreSQL)
+DATABASE_URL = "postgresql://user:pass@host:5432/absa_db"
+engine = create_engine(DATABASE_URL, pool_pre_ping=True)
+```
+
+## ORM Models
+
+```python
+class Review(Base):
+    __tablename__ = "reviews"
+    id = Column(Uuid(as_uuid=True), primary_key=True, default=uuid.uuid4)
+    text = Column(Text, nullable=False)
+    language = Column(String(10), nullable=False)
+    created_at = Column(DateTime(timezone=True), default=lambda: datetime.now(timezone.utc))
+    processing_time_ms = Column(Float, nullable=False)
+
+class AspectResult(Base):
+    __tablename__ = "aspect_results"
+    id = Column(Uuid(as_uuid=True), primary_key=True, default=uuid.uuid4)
+    review_id = Column(Uuid(as_uuid=True), ForeignKey("reviews.id"), nullable=False)
+    aspect = Column(String(255), nullable=False)
+    sentiment = Column(String(50), nullable=False)
+    confidence = Column(Float, nullable=False)
+    start_pos = Column(Integer, nullable=False)
+    end_pos = Column(Integer, nullable=False)
+
+class BatchJob(Base):
+    __tablename__ = "batch_jobs"
+    id = Column(Uuid(as_uuid=True), primary_key=True, default=uuid.uuid4)
+    status = Column(String(50), nullable=False, default="queued")
+    total = Column(Integer, nullable=False)
+    processed = Column(Integer, nullable=False, default=0)
+    created_at = Column(DateTime(timezone=True), default=lambda: datetime.now(timezone.utc))
+    completed_at = Column(DateTime(timezone=True), nullable=True)
+```

docs/DEPLOYMENT.md

@@ -0,0 +1,136 @@
+# Deployment Guide — Multilingual ABSA
+
+## Prerequisites
+
+- Python 3.11+
+- Node.js 20+
+- Docker & Docker Compose (for containerized deployment)
+- Railway account (for API deployment)
+- Vercel account (for dashboard deployment)
+
+## Environment Variables
+
+| Variable | Dev Default | Production | Required By |
+|----------|-------------|------------|-------------|
+| `DATABASE_URL` | `sqlite:///absa.db` | PostgreSQL URL | API + Worker |
+| `REDIS_URL` | `redis://localhost:6379/0` | Redis URL | API + Worker |
+| `MODEL_PATH` | `models/onnx/` | (same or HF Hub) | API |
+| `MAX_BATCH_SIZE` | `10000` | `10000` | API |
+| `LOG_LEVEL` | `INFO` | `WARNING` | API |
+| `ENABLE_METRICS` | `true` | `true` | API |
+| `HF_MODEL_REPO` | (empty) | `username/multilingual-absa` | API |
+| `MODEL_SOURCE` | `local` | `huggingface_hub` | API |
+
+## Local Development
+
+```bash
+# Backend
+cp .env.example .env
+python -m venv .venv && source .venv/bin/activate
+pip install -r requirements.txt
+uvicorn api.main:app --reload --host 0.0.0.0 --port 8000
+# API at http://localhost:8000, docs at http://localhost:8000/docs
+
+# MLflow
+./scripts/mlflow_ui.sh
+# MLflow UI at http://localhost:5000
+
+# Frontend
+cd dashboard
+cp .env.example .env     # VITE_API_URL=http://localhost:8000
+npm install
+npm run dev
+# Dashboard at http://localhost:5173
+```
+
+## Docker Compose (Full Stack)
+
+```bash
+docker-compose -f config/docker/docker-compose.yml up --build
+```
+
+Services started:
+
+| Service | Container Name | Port | Dependencies |
+|---------|---------------|------|--------------|
+| `api` | `absa-api` | 8000 | postgres, redis |
+| `worker` | `absa-worker` | — | postgres, redis, api |
+| `dashboard` | `absa-dashboard` | 3000 (=> 80) | api |
+| `postgres` | `absa-postgres` | 5432 | — |
+| `redis` | `absa-redis` | 6379 | — |
+| `prometheus` | — | 9090 | api |
+| `grafana` | — | 3001 | prometheus |
+
+```mermaid
+graph TB
+    DASH[Dashboard :3000] --> API[API :8000]
+    API --> PG[PostgreSQL :5432]
+    API --> RED[Redis :6379]
+    WORK[Worker] --> RED
+    WORK --> PG
+    PROM[Prometheus :9090] -->|scrape| API
+    GRAF[Grafana :3001] --> PROM
+```
+
+## Production Deployment (Railway + Vercel)
+
+### Railway (API + Worker)
+
+1. Create a Railway project from your Git repository
+2. Set build command: uses `railway.json` → `Dockerfile.api.prod`
+3. Set environment variables in Railway dashboard:
+   - `DATABASE_URL` → Railway PostgreSQL plugin connection string
+   - `REDIS_URL` → Railway Redis plugin connection string
+   - `MODEL_SOURCE=huggingface_hub`
+   - `HF_MODEL_REPO=your-username/multilingual-absa`
+   - `ENABLE_METRICS=true`
+   - `LOG_LEVEL=WARNING`
+4. Add a second service for the Celery worker with command:
+   `celery -A api.tasks.batch_tasks worker --loglevel=warning`
+
+### Vercel (Dashboard)
+
+1. Import `dashboard/` as a Vercel project
+2. Framework preset: Vite
+3. Environment variable: `VITE_API_URL=https://your-railway-api-url.railway.app`
+4. `vercel.json` rewrites `/api/*` to Railway API
+
+```mermaid
+graph LR
+    USER[Browser] --> VERCEL[Vercel CDN]
+    VERCEL -->|/api/* rewrite| RAILWAY[Railway API]
+    RAILWAY --> PG[(Railway PostgreSQL)]
+    RAILWAY --> REDIS[(Railway Redis)]
+    WORK[Celery Worker] --> REDIS
+    WORK --> PG
+```
+
+## DVC Data/Model Sync
+
+```bash
+# Pull data/models from remote
+dvc pull
+
+# Run full ML pipeline
+dvc repro
+
+# Push new artifacts
+dvc push
+```
+
+## Monitoring
+
+| Tool | URL | Purpose |
+|------|-----|---------|
+| MLflow UI | `http://localhost:5000` | Experiment tracking |
+| API Docs | `http://localhost:8000/docs` | Interactive API |
+| Prometheus | `http://localhost:9090` | Metrics store |
+| Grafana | `http://localhost:3001` | Visual dashboards |
+| Dashboard | `http://localhost:5173` | User interface |
+
+## Scaling
+
+- **API**: Increase `--workers` in uvicorn command (2 in prod Dockerfile)
+- **Worker**: Scale Celery worker containers horizontally
+- **Database**: Use Railway managed PostgreSQL with auto-scaling
+- **Memory limit**: 2GB per API container (configured in `docker-compose.prod.yml`)

docs/SECURITY.md

@@ -0,0 +1,67 @@
+# Security Analysis — Multilingual ABSA
+
+## Current State
+
+| Area | Status | Notes |
+|------|--------|-------|
+| JWT Authentication | ❌ Not implemented | No auth layer |
+| OAuth | ❌ Not implemented | No SSO |
+| HTTPS | ❌ Not enforced | Expects reverse proxy to terminate TLS |
+| Input Validation | ✅ Partial | Pydantic validation present; no server-side max_length |
+| SQL Injection | ✅ Protected | SQLAlchemy ORM parameterized queries |
+| XSS | ✅ Protected | React JSX auto-escaping |
+| CSRF | ❌ Not implemented | No CSRF middleware; CORS `"*"` mitigates partially |
+| Secrets Management | ⚠️ Manual | `.env` gitignored; Docker Compose has hardcoded dev creds |
+| Rate Limiting | ❌ Not implemented | No throttling on any endpoint |
+| File Upload Security | ⚠️ Partial | Extension validation; no size limit; temp files not cleaned on success |
+| Authorization | ❌ None | No role-based or API-key access control |
+| CORS | ⚠️ Permissive | `allow_origins=["*"]` |
+
+## Risks & Recommendations
+
+### Critical
+
+1. **Missing authentication** — All endpoints are publicly accessible
+   - **Fix**: Add FastAPI middleware for API key validation
+   - **Fix**: Integrate OAuth2/OIDC for multi-user scenarios
+
+2. **No rate limiting** — `/batch` endpoint can be abused (10K rows per request)
+   - **Fix**: Add `slowapi` or custom rate-limiting middleware
+   - **Fix**: Implement per-IP request quotas
+
+### High
+
+3. **Temp file leak** — Batch CSV saved via `NamedTemporaryFile(delete=False)` but `os.unlink()` only called on validation error, not on success
+   - **Fix**: Add `try/finally` block to ensure cleanup
+
+4. **CORS all origins** — `"*"` allows any website to call the API
+   - **Fix**: Restrict to known dashboard domains
+
+5. **No server-side text length limit** — `ReviewInput.text` accepts arbitrary length
+   - **Fix**: Add `StringConstraints(max_length=512)` to Pydantic model
+
+### Medium
+
+6. **No file size limit on batch uploads** — Only row count limit (10K)
+   - **Fix**: Add file-size check (e.g., 50MB max)
+
+7. **Hardcoded credentials** in `docker-compose.yml` — `absa_user/absa_pass`
+   - **Fix**: Use environment variables or Docker secrets
+
+8. **CSRF** — No protection; token-based auth (when implemented) would mitigate
+
+### Low
+
+9. **Weak health check** — Returns `"db": "connected"` without actually pinging DB
+   - **Fix**: Add actual DB ping to `/health` endpoint
+
+10. **No request logging** — No structured logging or audit trail
+
+## Configuration Checklist
+
+- [ ] Set `ENABLE_METRICS` to `false` if Prometheus not needed
+- [ ] Set `LOG_LEVEL` to `WARNING` in production
+- [ ] Use strong, random passwords for PostgreSQL
+- [ ] Run API behind TLS-terminating reverse proxy (Railway does this automatically)
+- [ ] Keep `.env` out of version control (already in `.gitignore`)
+- [ ] Rotate secrets regularly

docs/SYSTEM_DESIGN.md

@@ -0,0 +1,142 @@
+# System Design — Multilingual ABSA
+
+## 1. Design Goals
+
+- **Accuracy**: Macro-F1 > 78% English, > 65% Hindi
+- **Latency**: P95 < 300ms for single-review inference (ONNX INT8)
+- **Availability**: Zero-download fallback ensures the system starts instantly and never depends on external model downloads
+- **Scalability**: Async batch processing via Celery for bulk analysis
+- **Observability**: Full MLOps stack (MLflow, Prometheus, Grafana, Evidently)
+
+## 2. System Components
+
+### 2.1 FastAPI Application (`api/main.py`)
+- Lifespan handler initializes DB tables and loads models at startup
+- Two routers: `/predict` (single + batch), `/results` (health, info, metrics)
+- CORS middleware for dashboard origin
+- Prometheus instrumentator auto-exposes `/metrics`
+
+### 2.2 ABSA Pipeline (`api/services/absa_pipeline.py`)
+- Dual-engine design:
+  - **Neural**: ONNX Runtime with INT8-quantized XLM-RoBERTa models
+  - **Rule-based**: Lexicon-driven aspect extraction + context-window sentiment scoring
+- Thread-safe model loading via `threading.Lock()`
+- Singleton pattern (module-level `pipeline` instance)
+
+### 2.3 Language Service (`api/services/lang_service.py`)
+- Singleton with fastText LID model
+- Unicode-based fallback (Devanagari character range detection)
+
+### 2.4 Celery Worker (`api/tasks/batch_tasks.py`)
+- Processes uploaded CSV files in batches of 32
+- Incrementally writes results to CSV and DB
+- Progress tracking via BatchJob model
+
+### 2.5 React Dashboard (`dashboard/`)
+- 3 pages: Predict (live), Batch Analytics, System Monitor
+- API client with exponential backoff retry
+- React Query for server state and polling
+
+## 3. Data Model
+
+### 3.1 Reviews
+```sql
+reviews (id UUID PK, text TEXT, language VARCHAR(10), created_at DATETIME, processing_time_ms FLOAT)
+aspect_results (id UUID PK, review_id UUID FK, aspect VARCHAR(255), sentiment VARCHAR(50), confidence FLOAT, start_pos INT, end_pos INT)
+batch_jobs (id UUID PK, status VARCHAR(50), total INT, processed INT, created_at DATETIME, completed_at DATETIME NULL)
+```
+
+### 3.2 Relationships
+- One `Review` → Many `AspectResults`
+- `BatchJob` is standalone (progress tracking + CSV output)
+
+## 4. API Endpoints
+
+| Method | Path | Request | Response | Notes |
+|--------|------|---------|----------|-------|
+| POST | `/predict` | `{"text": str, "language": str?}` | `PredictionResponse` | Synchronous inference |
+| POST | `/batch` | `multipart/form-data` (CSV file) | `{"job_id", "status", "total_reviews", "processed"}` | Async via Celery |
+| GET | `/status/{job_id}` | — | `BatchJobResponse` | Poll batch progress |
+| GET | `/health` | — | `{"status", "model", "db"}` | Health check |
+| GET | `/info` | — | Model metadata | Version info |
+| GET | `/metrics` | — | Prometheus metrics | Auto-instrumented |
+
+## 5. ML Pipeline
+
+### 5.1 Training Pipeline
+```
+Raw Data → Text Cleaning → Language Detection → Transliteration → Tokenization
+                                                                      ↓
+                                                          BIO Tagging (for NER)
+                                                                      ↓
+                                            ┌──────────────────────────┐
+                                            │  XLM-RoBERTa Fine-Tune   │
+                                            │  ┌────────────────────┐ │
+                                            │  │ Aspect Extraction  │ │
+                                            │  │ (Token CLS, 3 lbl) │ │
+                                            │  └────────────────────┘ │
+                                            │  ┌────────────────────┐ │
+                                            │  │ Sentiment CLS      │ │
+                                            │  │ (Seq CLS, 4 lbl)   │ │
+                                            │  └────────────────────┘ │
+                                            └──────────────────────────┘
+                                                      ↓
+                                             ONNX Export + INT8 Quantization
+```
+
+### 5.2 Inference Pipeline
+```
+Input Text
+    ↓
+Language Detection (fastText LID / Unicode heuristic)
+    ↓
+┌─ Neural Path (if ONNX loaded) ────────────────────────┐
+│ Tokenize (XLM-R SentencePiece 128 tokens)              │
+│ → ORTModelForTokenClassification → BIO spans          │
+│ → Per-span ORTModelForSequenceClassification → sentiment│
+└─────────────────────────────────────────────────────���──┘
+    ↓ (fallback)
+┌─ Rule-Based Path ──────────────────────────────────────┐
+│ Regex match 140+ aspect keywords (longest-first)       │
+│ → Context-window sentiment scoring                     │
+│   • 200+ positive words, 200+ negative words           │
+│   • 3-word negation window                             │
+│   • Intensifier multiplier (1.5x)                      │
+│ → pos:neg ratio → label + confidence                   │
+└────────────────────────────────────────────────────────┘
+    ↓
+Structured JSON + DB Persistence
+```
+
+## 6. Rule-Based Engine Details
+
+### Aspect Extraction
+- 140+ phrase patterns across 10 categories:
+  - Audio (sound quality, bass, noise cancellation)
+  - Battery (battery life, charging speed)
+  - Design (build quality, comfort, ergonomics)
+  - Connectivity (bluetooth, wifi, pairing)
+  - Display (screen quality, resolution)
+  - Camera (camera quality, image quality)
+  - Performance (speed, ram, processor)
+  - Software (user interface, app, features)
+  - Value (price, value for money)
+  - Support (customer service, warranty)
+
+### Sentiment Scoring
+- Positive words: 110+ (excellent, great, amazing, badhiya, achha)
+- Negative words: 70+ (poor, terrible, kharab, bekaar)
+- Negation words: 22 (not, never, doesn't, didn't)
+- Intensifiers: 12 (very, extremely, highly)
+- Algorithm: Word-by-word scan with 3-word lookback for negation and intensifiers
+- Score → Label: >60% positive ratio → positive, <40% → negative, else → neutral
+
+## 7. Performance Targets
+
+| Metric | Target | Actual (ONNX INT8) |
+|--------|--------|-------------------|
+| English Macro-F1 | >75% | 78.1% |
+| Hindi Macro-F1 | >60% | 67.8% |
+| P95 Latency | <300ms | 185ms |
+| Throughput (single worker) | >5 req/s | ~5.4 req/s |
+| Batch Processing (10K rows) | <30 min | Estimated ~15 min |

docs/TECH_STACK.md

@@ -0,0 +1,97 @@
+# Technology Stack — Multilingual ABSA
+
+## Core Technologies
+
+| Technology | Version | Purpose | Where Used |
+|------------|---------|---------|------------|
+| **Python** | 3.11+ | Runtime | All backend/ML code |
+| **FastAPI** | 0.111.0 | REST API framework | `api/` routes and middleware |
+| **Uvicorn** | 0.29.0 | ASGI server | API entry point |
+| **React** | 18.2.0 | Frontend framework | `dashboard/src/` |
+| **Vite** | 5.0.0 | Build tool | `dashboard/vite.config.js` |
+| **TailwindCSS** | 3.3.5 | CSS framework | `dashboard/src/index.css` |
+| **PostgreSQL** | 16 (alpine) | Production database | Docker Compose |
+| **SQLite** | (built-in) | Development database | `absa.db` |
+| **Redis** | 7 (alpine) | Celery broker + cache | `api/tasks/` |
+| **Docker** | 27.x | Containerization | `config/docker/` |
+| **Docker Compose** | 3.8+ | Orchestration | `config/docker/docker-compose.yml` |
+
+## ML / AI Stack
+
+| Technology | Version | Purpose | Where Used |
+|------------|---------|---------|------------|
+| **PyTorch** | 2.3.0 | Deep learning framework | `src/models/` training |
+| **Transformers** | 4.39.3 | Model zoo, training, tokenization | All ML scripts |
+| **XLM-RoBERTa** | base | Multilingual encoder | `FacebookAI/xlm-roberta-base` |
+| **ONNX Runtime** | 1.18.0 | Production inference | `api/services/absa_pipeline.py` |
+| **Optimum** | 1.19.0 | ONNX export bridge | `src/models/export_onnx.py` |
+| **optimum-onnx** | (bundled) | ONNX runtime models | `ORTModelForTokenClassification`, `ORTModelForSequenceClassification` |
+| **PEFT** | 0.10.0 | Parameter-efficient fine-tuning | `src/models/train_qlora.py` (LoRA) |
+| **scikit-learn** | 1.4.2 | Metrics + baseline | `src/models/baseline.py`, `train_sentiment.py` |
+| **Datasets** | 2.19.0 | Data loading | `src/data/hf_dataset.py` |
+| **seqeval** | 1.2.2 | BIO tagging evaluation | `src/models/train_aspect_extraction.py` |
+| **fasttext-predict** | 0.9.2.4 | Language identification | `src/data/lang_detect.py`, `api/services/lang_service.py` |
+| **indic-nlp-library** | (git) | Devanagari transliteration | `src/data/transliterate.py` |
+| **nlpaug** | 1.1.11 | Text augmentation | `src/data/augmentation.py` |
+
+## MLOps Stack
+
+| Technology | Version | Purpose | Where Used |
+|------------|---------|---------|------------|
+| **MLflow** | 2.13.0 | Experiment tracking | `src/training/mlflow_utils.py`, all `src/models/` |
+| **DVC** | 3.51.1 | Data version control | `config/dvc.yaml`, `.dvc/` |
+| **Evidently AI** | 0.4.30 | Data drift monitoring | `scripts/drift_monitor.py` |
+| **Prometheus** | latest | Metrics collection | `monitoring/prometheus.yml` |
+| **Grafana** | latest | Dashboard visualization | `monitoring/grafana/dashboards/` |
+| **prometheus-fastapi-instrumentator** | 7.0.0 | Metrics middleware | `api/middleware/metrics.py` |
+
+## API / Backend Libraries
+
+| Technology | Version | Purpose |
+|------------|---------|---------|
+| **Pydantic** | 2.7.1 | Request/response schema validation |
+| **SQLAlchemy** | (via psycopg2) | ORM |
+| **psycopg2-binary** | 2.9.9 | PostgreSQL driver |
+| **Celery** | 5.4.0 | Async task queue |
+| **python-dotenv** | 1.0.1 | Environment variable loading |
+| **python-multipart** | 0.0.9 | File upload parsing |
+| **NumPy** | 1.26.4 | Numerical computing |
+| **Pandas** | 2.2.2 | Data manipulation |
+
+## Frontend Libraries
+
+| Technology | Version | Purpose |
+|------------|---------|---------|
+| **@tanstack/react-query** | 5.0.0 | Server state, caching, polling |
+| **react-router-dom** | 6.20.0 | Client routing |
+| **react-hot-toast** | 2.4.1 | Toast notifications |
+| **react-dropzone** | 14.2.3 | File upload drag-and-drop |
+| **axios** | 1.6.0 | HTTP client with retry |
+| **recharts** | 2.10.0 | Charts (line, bar, pie/donut) |
+| **lucide-react** | 0.290.0 | Icons |
+| **autoprefixer** | 10.4.16 | CSS vendor prefixes |
+| **postcss** | 8.4.31 | CSS processor |
+
+## Infrastructure / Deployment
+
+| Technology | Purpose |
+|------------|---------|
+| **Docker** (multi-stage) | Build optimization |
+| **Nginx (alpine)** | SPA serving + API proxy |
+| **Railway** | API + worker cloud hosting |
+| **Vercel** | Frontend SPA hosting |
+| **HuggingFace Hub** | Model storage/pull |
+
+## Version Compatibility Matrix
+
+| Package | Python | PyTorch | ONNX Runtime |
+|---------|--------|---------|--------------|
+| transformers 4.39.3 | 3.8+ | 1.11+ | — |
+| optimum 1.19.0 | 3.8+ | 1.13+ | 1.15+ |
+| onnxruntime 1.18.0 | 3.8+ | — | — |
+| peft 0.10.0 | 3.8+ | 2.0+ | — |
+| mlflow 2.13.0 | 3.8+ | — | — |
+| dvc 3.51.1 | 3.8+ | — | — |
+| evidently 0.4.30 | 3.8+ | — | — |
+| fastapi 0.111.0 | 3.8+ | — | — |
+| celery 5.4.0 | 3.8+ | — | — |

docs/architecture.md

@@ -1,30 +1,280 @@
-# Architecture Diagrams
+# Multilingual ABSA — Architecture Document
+
+## System Overview
+
+Multilingual ABSA is a production-ready Aspect-Based Sentiment Analysis system supporting English, Hindi, and Hinglish. It extracts aspect terms from product reviews and classifies their sentiment using a dual-engine architecture: INT8-quantized ONNX models for production inference with a zero-download rule-based fallback.
+
+## High-Level Architecture
 
-## 1. System Architecture
 ```mermaid
-graph TD
-  A[React Dashboard] -->|REST API| B[FastAPI]
-  B -->|sync| C[ABSA Pipeline]
-  B -->|async| D[Celery Worker]
-  C --> E[Stage 1: Aspect Extraction ONNX]
-  C --> F[Stage 2: Sentiment Classifier ONNX]
-  D --> G[PostgreSQL]
-  B --> G
-  H[Prometheus] -->|scrape /metrics| B
-  I[Grafana] -->|query| H
-  E --> J[HuggingFace Hub]
-  F --> J
+graph TB
+    Client[Client Browser / API Consumer]
+    Vercel[Vercel CDN]
+    Nginx[Nginx Reverse Proxy]
+    API[FastAPI Server]
+    Pipeline[ABSA Pipeline]
+    Lang[Language Detection]
+    Neural[ONNX Neural Engine<br/>INT8 Quantized]
+    Fallback[Rule-Based Engine<br/>140+ Aspect Keywords]
+    Celery[Celery Worker]
+    Redis[(Redis)]
+    PG[(PostgreSQL)]
+    Prom[Prometheus]
+    Graf[Grafana]
+    MLflow[(MLflow<br/>Experiment Tracking)]
+
+    Client --> Vercel
+    Vercel --> Nginx
+    Nginx --> API
+    API --> Pipeline
+    Pipeline --> Lang
+    Pipeline --> Neural
+    Pipeline --> Fallback
+    API --> Celery
+    Celery --> Redis
+    API --> PG
+    Prom -->|scrape /metrics| API
+    Graf --> Prom
+    MLflow --> Pipeline
 ```
 
-## 2. ABSA Inference Pipeline
+## Component Architecture
+
 ```mermaid
 graph LR
-  A[Raw Review] --> B[Language Detection]
-  B -->|EN| C[XLM-R Tokenizer]
-  B -->|HI/Hinglish| D[IndicBERT Tokenizer]
-  C --> E[Stage 1: BIO Tagger]
-  D --> E
-  E --> F[Extracted Aspects]
-  F --> G[Stage 2: Sentiment Classifier]
-  G --> H[aspect, sentiment, confidence]
+    subgraph "Presentation Layer"
+        SPA[React SPA]
+        T_TAIL[TailwindCSS Theme]
+        RECH[Recharts Visualizations]
+        RQ[React Query]
+    end
+    subgraph "API Layer"
+        FAST[FastAPI]
+        CORS[CORS Middleware]
+        PROM[Prometheus Metrics]
+        PYD[Pydantic Schemas]
+    end
+    subgraph "Service Layer"
+        ABSA[ABSAPipeline]
+        LANG[LanguageService]
+        CEL[Celery Tasks]
+    end
+    subgraph "Data Layer"
+        SQLA[SQLAlchemy ORM]
+        PG[(PostgreSQL)]
+        RED[(Redis)]
+    end
+    subgraph "ML Layer"
+        ONNX_A[ORTModelFor<br/>TokenClassification]
+        ONNX_S[ORTModelFor<br/>SequenceClassification]
+        LEXICON[Aspect/Sentiment<br/>Lexicons]
+    end
+    subgraph "MLOps Layer"
+        MLF[MLflow Tracking]
+        DVC[DVC Versioning]
+        EVI[Evidently Drift]
+    end
+
+    SPA --> FAST
+    FAST --> CORS
+    FAST --> PROM
+    FAST --> PYD
+    FAST --> ABSA
+    FAST --> CEL
+    ABSA --> LANG
+    ABSA --> ONNX_A
+    ABSA --> ONNX_S
+    ABSA --> LEXICON
+    CEL --> RED
+    FAST --> SQLA
+    SQLA --> PG
+    ABSA --> MLF
 ```
+
+## Deployment Architecture
+
+```mermaid
+graph TB
+    subgraph "Docker Compose (Local)"
+        DC_API[API Service<br/>uvicorn:8000]
+        DC_WORKER[Celery Worker]
+        DC_DASH[Dashboard<br/>Nginx:80]
+        DC_PG[PostgreSQL:5432]
+        DC_REDIS[Redis:6379]
+        DC_PROM[Prometheus:9090]
+        DC_GRAF[Grafana:3001]
+    end
+    subgraph "Railway (Production)"
+        RW_API[API Service<br/>$PORT]
+        RW_WORKER[Celery Worker]
+        RW_PG[PostgreSQL]
+        RW_REDIS[Redis]
+    end
+    subgraph "Vercel (Production)"
+        VC_DASH[React SPA]
+        VC_RW_ROUTE[rewrite /api/* -> Railway]
+    end
+
+    DC_DASH --> DC_API
+    DC_API --> DC_PG
+    DC_API --> DC_REDIS
+    DC_WORKER --> DC_REDIS
+    DC_WORKER --> DC_PG
+    DC_PROM -->|scrape| DC_API
+    DC_GRAF --> DC_PROM
+
+    VC_DASH --> VC_RW_ROUTE
+    VC_RW_ROUTE --> RW_API
+    RW_API --> RW_PG
+    RW_API --> RW_REDIS
+    RW_WORKER --> RW_REDIS
+```
+
+## ML Pipeline Architecture
+
+```mermaid
+graph TB
+    subgraph "Data Ingestion"
+        RAW[Raw Data<br/>SemEval 2014<br/>Amazon Hindi]
+        FAST[f astText LID<br/>lid.176.ftz]
+    end
+    subgraph "Preprocessing"
+        CLEAN[Text Cleaning<br/>Lowercase, URLs, Mentions]
+        TRANS[Transliteration<br/>Devanagari→Roman]
+        LANG_DET[Language Detection<br/>EN / HI / Hinglish]
+        BIO[BIO Tagging<br/>B-ASP / I-ASP / O]
+        TOK[XLM-R Tokenizer<br/>SentencePiece 128 tokens]
+    end
+    subgraph "Training"
+        ATE[Aspect Extraction<br/>Token Classification<br/>3 labels]
+        ASC[Sentiment Classification<br/>Sequence Classification<br/>4 labels]
+        BASELINE[Baseline<br/>TF-IDF + LR]
+        QLORA[QLoRA<br/>4-bit + LoRA]
+        JOINT[Joint ABSA<br/>Shared Encoder<br/>2 Heads]
+    end
+    subgraph "Optimization"
+        ONNX_EXP[ONNX Export<br/>optimum-onnx]
+        QUANT[INT8 Quantization<br/>Dynamic]
+    end
+    subgraph "Production"
+        INFERENCE[Dual-Engine<br/>Inference]
+        BATCH[Batch Processing<br/>Celery Worker]
+    end
+    subgraph "Evaluation"
+        EVAL_METRICS[Macro-F1<br/>Per-class F1<br/>Confusion Matrix]
+        LATENCY[Latency Benchmark<br/>P95 < 300ms]
+        CROSS[Cross-Lingual Eval<br/>EN→HI Zero-Shot]
+    end
+
+    RAW --> CLEAN
+    FAST --> LANG_DET
+    CLEAN --> LANG_DET
+    LANG_DET --> TRANS
+    TRANS --> TOK
+    TOK --> ATE
+    TOK --> ASC
+    BIO --> ATE
+    ATE --> JOINT
+    ASC --> JOINT
+    ATE --> ONNX_EXP
+    ASC --> ONNX_EXP
+    ONNX_EXP --> QUANT
+    QUANT --> INFERENCE
+    INFERENCE --> BATCH
+    ATE --> EVAL_METRICS
+    ASC --> EVAL_METRICS
+    BASELINE --> EVAL_METRICS
+    INFERENCE --> LATENCY
+    JOINT --> CROSS
+```
+
+## Data Flow
+
+```mermaid
+sequenceDiagram
+    participant C as Client
+    participant F as FastAPI
+    participant P as ABSAPipeline
+    participant L as LangService
+    participant N as ONNX Runtime
+    participant R as Rule Engine
+    participant D as PostgreSQL
+    participant M as Prometheus
+
+    C->>F: POST /predict {text, language?}
+    F->>P: pipeline.predict(text, lang)
+    P->>L: detect_language(text)
+    L-->>P: "en" | "hi" | "hinglish"
+    alt ONNX Models Available
+        P->>N: Tokenize text
+        N-->>P: Token IDs + Attention Mask
+        P->>N: ORTModelForTokenClassification
+        N-->>P: BIO Logits → Argmax → Spans
+        P->>N: Per-aspect ORTModelForSequenceClassification
+        N-->>P: Sentiment Logits → Softmax
+    else Rule-Based Fallback
+        P->>R: _extract_aspects(text)
+        R-->>P: [(aspect, start, end)]
+        P->>R: _score_sentence(context)
+        R-->>P: (pos_score, neg_score)
+        P->>R: _score_to_label(pos, neg)
+        R-->>P: (sentiment, confidence)
+    end
+    P-->>F: PredictionResponse
+    F->>D: INSERT Review + AspectResults
+    D-->>F: IDs
+    F-->>C: JSON Response
+    F->>M: Record latency + status
+```
+
+## Infrastructure
+
+```mermaid
+graph TB
+    subgraph "Edge"
+        DNS[DNS: Vercel]
+        SSL[TLS Termination]
+    end
+    subgraph "Frontend Hosting"
+        FE[Vercel<br/>Static SPA]
+        FE_CDN[Global CDN]
+    end
+    subgraph "Backend Hosting"
+        BE[Railway<br/>Docker Container]
+        HEALTH[Health Check<br/>/health]
+        AUTO[Auto-Restart<br/>On Failure]
+    end
+    subgraph "Data Services"
+        PG[PostgreSQL<br/>Railway Managed]
+        RD[Redis<br/>Railway Managed]
+    end
+    subgraph "Observability"
+        PROM[Prometheus<br/>15-day Retention]
+        GRAF[Grafana<br/>Pre-provisioned Dashboard]
+        MLFLOW[MLflow<br/>SQLite Backend]
+    end
+
+    DNS --> SSL
+    SSL --> FE
+    FE --> FE_CDN
+    FE_CDN --> BE
+    BE --> HEALTH
+    BE --> AUTO
+    BE --> PG
+    BE --> RD
+    PROM -->|scrape| BE
+    GRAF --> PROM
+```
+
+## Design Decisions
+
+| Decision | Rationale |
+|----------|-----------|
+| **Separate ONNX models** for ATE and ASC | Combined graph has dynamic-axis export fragility in optimum-onnx |
+| **Rule-based fallback** with no downloads | Zero startup time, works offline, graceful degradation |
+| **Lexicon-based sentiment** with negation handling | 3-word window for "not good" → negative reversal |
+| **XLM-RoBERTa base** (not large) | 0.3B params fine-tunes on 16GB GPU, adequate cross-lingual transfer |
+| **ONNX INT8 dynamic quantization** | 4x smaller, 4.6x faster than PyTorch with only 1% F1 drop |
+| **SQLite for dev, PostgreSQL for prod** | Zero-config local dev, production-grade concurrency |
+| **Celery for batch only** | Single-review inference is fast enough for synchronous response |
+| **Mock data in dashboard charts** | Decoupled frontend/backend development; real integration deferred |