int.md

# Negoptim AI — Technical Stack

## 1. Frontend — Next.js 15 · TypeScript · Tailwind CSS

**What we use:** Next.js 15 (App Router), TypeScript, Tailwind CSS, custom React hooks.

**Why Next.js over Gradio / Streamlit:**
Gradio and Streamlit are rapid-prototyping tools designed for data science demos. They impose rigid layout constraints and produce consumer-grade UIs that cannot be customised to match enterprise branding. Next.js gives us full control over the component tree, design system, and interaction patterns. We can build structured information layouts (dashboards, comparison grids, team cards) that render contextually inside the chat — something impossible in Gradio or Streamlit.

**Why not the Vercel AI SDK:**
The Vercel AI SDK is a valid choice for streaming text, but it adds a tight coupling to Vercel's infrastructure and abstracts away the transport layer. Our backend is FastAPI, not a Next.js API route. Using the Vercel AI SDK would require routing all LLM traffic through Next.js server-side functions, which adds latency and complicates the separation between frontend and backend. We use a plain `fetch` call from a custom `useChat` hook instead — it is simpler, transport-agnostic, and gives us full control over the response shape (sources, token usage, rate limits, structured data, fallback model info).

**Why TypeScript:**
Strict typing across the component tree catches contract mismatches between the API response shape and the frontend state before they reach the browser. Every field returned by the backend (`TokenUsage`, `RateLimitInfo`, `Source`, `ResponseType`) has a corresponding TypeScript interface, making refactoring safe.

**Why Tailwind CSS over a component library (MUI, Chakra, shadcn):**
Component libraries ship with opinionated default styles that require significant overriding to produce a custom brand identity. Since ULiT's visual identity (geometric hexagons, brand gradient, specific surface colours) deviates from any standard design system, utility-first CSS is more efficient. There is nothing to fight.

**API proxy pattern:**
All `/api/*` requests from the browser are forwarded to the FastAPI server via `next.config.ts` rewrites. The frontend never knows the backend's port or address. This eliminates CORS configuration, keeps credentials server-side, and allows the backend URL to change without touching a single frontend file.

**Alternatives considered:**
- Vite + React: valid, but loses the proxy rewrite and SSR capabilities of Next.js.
- Remix: similar capability to Next.js but smaller ecosystem and less relevant for a single-page chat interface.
- Gradio / Streamlit: excluded — insufficient design control for enterprise branding.

---

## 2. Backend — FastAPI · Python 3.10+ · Pydantic v2

**What we use:** FastAPI with async request handling, Pydantic v2 for request/response validation, pydantic-settings for environment configuration.

**Why FastAPI over Flask or Django:**
FastAPI is async-native, which matters when the bottleneck is a remote LLM call (60–120ms network wait). Flask and Django are synchronous by default; serving concurrent users would require either threads or a separate async worker. FastAPI handles all of this natively. It also auto-generates OpenAPI documentation at `/docs`, which is useful during development and for any future API consumers.

**Why Pydantic v2:**
Every response shape — `ChatResponse`, `TokenUsage`, `RateLimitInfo`, `Source`, `RateLimitInfo` — is a typed Pydantic model. This means serialisation is automatic, validation errors surface at the boundary rather than deep in business logic, and the TypeScript types on the frontend mirror the Python models exactly.

**Architecture — services not scripts:**
The backend is structured as independent services (`EmbeddingService`, `VectorStore`, `IngestionService`, `LLMService`, `RAGService`, `IntentService`) rather than a flat procedural script. Each service is instantiated once as a singleton and reused across requests, which prevents reloading the 130 MB embedding model on every call.

**Alternatives considered:**
- Flask: simpler, but synchronous and no built-in validation.
- Django REST Framework: excessive for a focused API with five endpoints.
- Node.js / Express: would require running two runtimes (Python for ML, Node for API), adding operational complexity.

---

## 3. Vector Database — ChromaDB

**What we use:** ChromaDB with cosine similarity, persisted to disk via `PersistentClient`.

**Why ChromaDB exclusively:**
ChromaDB is an embedded vector database — it runs inside the Python process with no separate server, no Docker container, and no network overhead. For a local demo that must start with a single command, this is the correct choice. It supports metadata filtering, cosine similarity search, and upsert semantics (re-ingesting the same document updates existing vectors rather than creating duplicates).

**Why no PostgreSQL:**
PostgreSQL with pgvector is a production-grade choice when you need relational data alongside vectors — user accounts, audit logs, multi-tenant data. This project has none of those requirements. Introducing PostgreSQL would add a database server to the local setup, require a migration tool, and add a dependency with zero benefit for a single-node RAG demo. If this system were to move toward multi-tenant production, PostgreSQL + pgvector would be the natural upgrade path.

**Why no Redis:**
Redis was considered for session memory caching. Given that sessions are in-memory within the FastAPI process (a deliberate choice for a demo), Redis would add infrastructure for no gain. The in-memory approach resets on restart, which is acceptable and documented behaviour.

**Alternatives considered:**
- pgvector (PostgreSQL extension): best for production multi-tenant deployments.
- Pinecone / Weaviate: cloud-hosted, introduces external API dependency, not suitable for a fully local demo.
- FAISS: fast, but requires manual persistence and metadata management that ChromaDB handles natively.
- Qdrant: excellent production option, heavier operational footprint than ChromaDB for local use.

---

## 4. Embedding Model — intfloat/multilingual-e5-small

**What we use:** `intfloat/multilingual-e5-small` via the `sentence-transformers` library, running on CPU locally.

**Why this model:**
The e5 family (Embeddings from bidirectional Encoder representations) from Microsoft is specifically trained for asymmetric retrieval tasks — the exact use case of RAG, where short queries must match longer document passages. The `-small` variant (33 million parameters) runs in under 2 seconds on CPU and produces 384-dimensional embeddings. Critically, it supports 100+ languages including French and English natively, which is required since ULiT operates in both languages.

**The prefix strategy:**
The e5 model requires task-specific prefixes to work correctly:
- Documents indexed at ingestion time use the prefix `"passage: "`
- User queries at retrieval time use the prefix `"query: "`

This asymmetric encoding is intentional — the model was trained this way. Omitting the prefixes degrades retrieval quality meaningfully. This is a non-obvious detail that many implementations get wrong.

**Why not OpenAI text-embedding-ada-002 or text-embedding-3-small:**
Both are excellent models. But they require an internet connection and per-token API cost. This project must run entirely locally with no external API dependency for embeddings. The multilingual-e5-small delivers comparable quality for structured business text at zero operating cost.

**Alternatives considered:**
- `intfloat/multilingual-e5-base`: better quality, 2× slower and heavier — not necessary for a structured knowledge base with clean headings.
- `sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2`: solid multilingual model, but the e5 family outperforms it on retrieval benchmarks.
- `BAAI/bge-m3`: state-of-the-art multilingual model, but significantly heavier for CPU inference.
- OpenAI embeddings: excluded — requires cloud API, violates local-only constraint.

---

## 5. LLM Integration — Groq API with Fallback Chain

**What we use:** Groq API (`llama-3.1-8b-instant` as primary), with an automatic fallback chain on HTTP 429 (rate limit):

```
llama-3.1-8b-instant  →  llama-3.3-70b-versatile  →  mixtral-8x7b-32768  →  gemma2-9b-it
```

**Why Groq over Ollama:**
Ollama runs models locally on CPU. For a machine without a dedicated GPU, llama3-8b on CPU takes 30–90 seconds per response, which makes the demo unusable. Groq runs inference on custom LPU (Language Processing Unit) hardware and consistently returns first tokens in under 500ms — even on the free tier. The demo experience is completely different. Groq's free tier has generous token-per-minute quotas and does not require a credit card.

**Why a fallback chain:**
Each model on Groq has its own independent rate limit quota. If the primary model hits its per-minute limit (HTTP 429), the system automatically retries with the next model in the chain without surfacing an error to the user. This makes the demo resilient during sustained use or when demonstrating to multiple people simultaneously. The fallback is transparent — the settings panel shows which model responded and flags when a fallback was triggered.

**Why not OpenAI GPT-4o or Anthropic Claude:**
Both are excellent production choices. For this demo, the priority is zero cost, and both require paid API access beyond their trial limits. Groq's free tier is sufficient for a demo context.

**Why not keep Ollama as the LLM:**
Ollama remains in the architecture as the documented local alternative (see `.env.example`). The abstraction layer in `LLMService` is designed so the provider can be swapped by changing a few environment variables. For a machine with a capable GPU, Ollama with llama3 or mistral would be the preferred fully-offline option.

**Alternatives considered:**
- Ollama (local): best for full offline operation on capable hardware, too slow on CPU-only machines.
- OpenAI API: production-grade, paid beyond trial, not suitable for a zero-cost demo.
- Anthropic Claude API: same as above.
- Google Gemini free tier: viable alternative to Groq, slightly more complex setup for this stack.
- Together AI / OpenRouter: aggregator APIs with free model tiers, valid alternatives to Groq.

---

## 6. RAG Pipeline — Custom Implementation

**What we use:** A custom RAG pipeline built directly with `sentence-transformers` and `chromadb` — no LangChain, no LlamaIndex.

**Why no LangChain or LlamaIndex:**
LangChain and LlamaIndex are powerful orchestration frameworks. They are also heavily abstracted: a simple embedding + retrieval + generation flow involves dozens of classes, callback chains, and configuration objects that make the system harder to read, debug, and modify. Our pipeline is approximately 150 lines of Python spread across clearly named services. Every step is visible and testable in isolation. For a project where the architecture itself needs to be explainable (as a demo and internal reference), transparency beats abstraction.

**Chunking strategy:**
The chunking approach is document-type-aware:
- **Markdown files** are split at H2/H3 heading boundaries. The heading text is stored as metadata (`section` field) and included in the chunk prefix. This means the retriever knows not just the content but the semantic label of the section it came from.
- **Plain text files** are split at paragraph boundaries (double newlines), with configurable chunk size and overlap.
- **PDF files** are extracted page-by-page via `pypdf`, then processed with the same paragraph strategy.

This preserves the semantic structure of documents rather than blindly splitting at fixed character counts, which would routinely cut sentences mid-thought.

**Source tracking:**
Every chunk stored in ChromaDB carries metadata: `source` (relative file path), `section` (heading), `chunk_index`, and `type`. When a chunk is retrieved, its similarity score (converted from ChromaDB's cosine distance via `score = 1 - distance`) and metadata are returned in the API response. The frontend displays these as collapsible source citations.

---

## 7. Session Memory — In-Process, In-Memory

**What we use:** A Python `defaultdict` keyed by session UUID, storing the last N conversation turns as a list of `{role, content}` dicts. No database, no cache server.

**Why in-memory:**
Session persistence across server restarts is not a requirement for a local demo. In-memory storage is zero-latency, zero-configuration, and automatically cleans up when the session is cleared by the user. The history is trimmed to `MAX_HISTORY_TURNS` (default: 10 turns = 20 entries) to prevent the LLM context window from growing unboundedly.

**Why not Redis:**
Redis would add a dependency (Docker or a local Redis server) to the startup process for no functional benefit in a single-user demo context. It remains the correct upgrade path if this system were to scale to multiple concurrent users or require session persistence.

**Why not PostgreSQL for chat history:**
Same reasoning. Persistent chat history across restarts is a product decision that has not been made. Storing something in a database implies it needs to be retrieved, queried, or audited. Until that requirement exists, a database is premature infrastructure.

---

## 8. Knowledge Base and Data Ingestion

**Folder structure:**

```
knowledge/
├── company/      ← product documentation, platform descriptions, company info
├── guides/       ← feature guides for manufacturers and retailers
├── faqs/         ← Q&A pairs, use cases, user stories
├── legal/        ← regulatory context (Égalim), compliance notes
└── customers/    ← reserved for future customer-specific documents
```

**Supported file types:** `.md`, `.txt`, `.pdf`

**Ingestion command:**
```bash
cd backend && python ../scripts/ingest.py          # ingest everything
cd backend && python ../scripts/ingest.py company  # ingest one folder
```

**Re-ingestion safety:**
Chunk IDs are generated deterministically from the source path, chunk index, and a content hash. ChromaDB upserts (not inserts) are used, so re-running ingestion on an unchanged file is a no-op. Modified files produce new IDs and replace old vectors.

---

## Summary Table

| Layer | Choice | Key Reason |
|---|---|---|
| Frontend framework | Next.js 15 + TypeScript | Full UI control, API proxy, type safety |
| Styling | Tailwind CSS | No design system to fight, custom brand |
| State management | Custom React hooks | No framework overhead needed |
| Backend framework | FastAPI (Python) | Async-native, auto docs, Pydantic validation |
| Vector database | ChromaDB | Embedded, no server, cosine similarity, local |
| Embedding model | multilingual-e5-small | EN+FR support, fast on CPU, purpose-built for retrieval |
| LLM provider | Groq API | Free tier, fast inference (LPU), no GPU needed |
| LLM fallback | 4-model chain | Independent quotas, transparent to user |
| RAG framework | Custom (no LangChain) | Transparency, simplicity, debuggability |
| Session storage | In-process memory | Zero config, sufficient for demo scope |
| Chunking | Heading-aware + paragraph | Preserves semantic structure of documents |