deploy 2025-08-09 18:22:12

README.md

@@ -10,6 +10,7 @@ A MLOps project of an AI-powered RAG Chatbot for understanding and querying lega
 - [Getting Started](#getting-started)
 - [API Endpoints](#api-endpoints)
 - [Monitoring](#monitoring)
+- [Models Serving](#models-serving)
 - [Project Structure](#project-structure)
 - [License](#license)
 - [Acknowledgements](#acknowledgements)
@@ -42,7 +43,7 @@ Demo on Render: https://legalcontractanalyzer.onrender.com/
 - [X] Real-time streaming response.
 - [X] Contextual retrieving + querying via ChromaDB.
 - [X] CI pipeline with Github Actions.
-- [ ] CD pipeline with HuggingFace Space.
+- [X] CD pipeline with Render.
 - [X] Monitoring with Prometheus & Grafana.
 - [ ] Evaluation of the system (automated tests, LLM-as-judge).
 
@@ -196,6 +197,133 @@ In Grafana, I've built a dedicated **Queries Dashboard** to give you real-time i
 └── requirements.txt
 ```
 
+## Models Serving (optional)
+
+If you dig deep into the code, you will find the link https://glowing-workable-arachnid.ngrok-free.app/docs as the OpenAI API-like server, this is because I deploy it on my school server and then tunnel via ngrok xD. 
+
+So if you want to start your own model serving server (assuming you have a really strong DGX, H100, A100, or just 3 RTX 3090s like me xD), here's are the steps:
+
+### 1. Installation
+
+#### 1.1 Install FastChat
+
+FastChat is the backend server that can run multiple model workers and serve them via the OpenAI-compatible API.
+
+```bash
+# Create and activate virtual environment (optional but recommended)
+conda create -n fastchat python=3.10 -y
+conda activate fastchat
+
+# Install FastChat
+pip install fschat
+```
+
+**Tip:** If you want GPU acceleration, make sure PyTorch with CUDA is installed before installing FastChat:
+>
+> ```bash
+> pip install torch --index-url https://download.pytorch.org/whl/cu121
+> ```
+
+#### 1.2 Install ngrok
+
+ngrok will allow you to expose your FastChat API to the internet.
+
+```bash
+curl -sSL https://ngrok-agent.s3.amazonaws.com/ngrok.asc \
+  | sudo tee /etc/apt/trusted.gpg.d/ngrok.asc >/dev/null \
+  && echo "deb https://ngrok-agent.s3.amazonaws.com bookworm main" \
+  | sudo tee /etc/apt/sources.list.d/ngrok.list \
+  && sudo apt update \
+  && sudo apt install ngrok
+```
+
+If you have troubles downloading ngrok, try visiting their official website: https://ngrok.com/downloads/
+
+Log into [ngrok](https://dashboard.ngrok.com/get-started) and get your auth token:
+
+```bash
+ngrok config add-authtoken <YOUR_AUTH_TOKEN>
+```
+
+---
+
+### 2. 🖥️ Configurable FastChat Run Script
+
+In the folder /model_serving, check out the file `serve_models.sh` and make it executable:
+
+```bash
+chmod +x serve_models.sh
+```
+
+---
+
+### 3. Usage Examples
+
+#### Run with defaults (Qwen3-0.6B + Qwen3-Embedding-0.6B)
+
+```bash
+./model_serving/serve_models.sh
+```
+
+#### Run with custom models, ports, and ngrok URL
+
+```bash
+./model_serving/serve_models.sh Qwen/Qwen2-7B Qwen2-7B 21010 \
+                  Qwen/Qwen2-Embedding Qwen2-Embedding 21011 \
+                  8000 https://mycustomtunnel.ngrok-free.app
+```
+
+This will:
+
+* Run `Qwen2-7B` chat model on port `21010`.
+* Run `Qwen2-Embedding` embedding model on port `21011`.
+* Serve API on port `8000`.
+* Tunnel via the given ngrok URL.
+
+---
+
+### 4. 🔍 Testing the API
+
+List all models:
+
+```bash
+curl https://YOUR_NGROK_URL/v1/models
+```
+
+Or you may access it via a browser, for example: https://glowing-workable-arachnid.ngrok-free.app/v1/models
+
+Get embeddings:
+
+```bash
+curl https://YOUR_NGROK_URL/v1/embeddings \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "Qwen3-Embedding-0.6B",
+    "input": "FastChat is running two models now!"
+  }'
+```
+
+Chat completion:
+
+```bash
+curl https://YOUR_NGROK_URL/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "Qwen3-0.6B",
+    "messages": [{"role": "user", "content": "Hello from FastChat!"}]
+  }'
+```
+
+---
+
+### 5. Notes
+
+* Always **set different ports** for each worker.
+* `--worker-address` **must match** the worker’s host\:port so FastChat doesn’t overwrite registrations.
+* Ngrok **free plan** requires reserving the subdomain before you can set a fixed `--url`. You may go on ngrok website to claim your own free subdomain to use, otherwise, whenever you start a tunnel, it will be a random public url.
+* Contact me if you need help ;) I'll be glad to help.
+
+
 ## Licence
 
 [Apache 2.0](./LICENSE)

backend/chroma_vector_db/chroma.sqlite3

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:bedaaed932573c2da9f3c8ad609773f192cb3e027c9aabc8927f9567f35b288b
-size 3514368
+oid sha256:46cc306774b0946061527a0d3673f6c5aa63d2111d8a69d176a8216390b2e62c
+size 5554176

backend/ingest.py

@@ -53,7 +53,7 @@ def chunk_paragraph(paragraph):
 # ─── 4) EMBEDDING VIA OPENAI ────────────────────────────────────────────────────
 def embed_via_openai(text_chunks):
     resp = openai_client.embeddings.create(
-        model="Qwen3-0.6B",
+        model="Qwen3-Embedding-0.6B",
         input=text_chunks
     )
     # resp.data is a list of objects with .index and .embedding

backend/query.py

@@ -1,63 +1,39 @@
-# backend/query.py
-import numpy as np
+import os
 import chromadb
 from openai import OpenAI
 from dotenv import load_dotenv
 from backend.config import CHROMA_DB_PATH
-import os
-
+# ─── ENVIRONMENT ──────────────────────────────────────────────────────────────
 load_dotenv()
-API_KEY = os.getenv("OPENAI_API_KEY")
-BASE_URL = os.getenv("FASTCHAT_URL", "https://glowing-workable-arachnid.ngrok-free.app/v1")
+API_KEY = os.getenv("OPENAI_API_KEY", "TRANMINHDUONGDEPTRAI")
+BASE_URL = "https://glowing-workable-arachnid.ngrok-free.app/v1"  # or ngrok URL
 openai_client = OpenAI(api_key=API_KEY, base_url=BASE_URL)
 
+# ─── CHROMA SETUP ─────────────────────────────────────────────────────────────
 chroma_client = chromadb.PersistentClient(path=CHROMA_DB_PATH)
 collection = chroma_client.get_or_create_collection("legal_docs")
 
-def embed_texts(texts):
+# ─── EMBEDDING FUNCTION ───────────────────────────────────────────────────────
+def embed_query(query_text):
     resp = openai_client.embeddings.create(
-        model="Qwen3-0.6B",
-        input=texts
+        model="Qwen3-Embedding-0.6B",
+        input=[query_text]
     )
-    # ensure order
-    return [item.embedding for item in sorted(resp.data, key=lambda d: d.index)]
-
-def normalize(vec):
-    arr = np.array(vec, dtype=np.float32)
-    return arr / (np.linalg.norm(arr) + 1e-10)
-
-def query_top_k(query_text, k=10, rerank_top_n=5):
-    # 1) embed
-    q_emb = embed_texts([query_text])[0]
-    q_norm = normalize(q_emb)
-
-    # 2) dense retrieval (get more candidates)
-    results = collection.query(query_embeddings=[q_emb], n_results=k)
-    docs = results.get('documents', [[]])[0]
-    dists = results.get('distances', [[]])[0]
-
-    # Note: Chroma distances are lower = better. We'll compute cosine from stored embeddings if available.
-    # If you stored embeddings in collection, pull them (some Chroma versions allow include=['embeddings'])
-    # Here we fallback to converting distance -> similarity (if the metric is cosine)
-    sims = []
-    for idx, doc in enumerate(docs):
-        # try to get the stored embedding if available:
-        try:
-            emb = results['embeddings'][0][idx]
-            sim = float(np.dot(q_norm, normalize(emb)))
-        except Exception:
-            # fallback: invert distance (only approximate)
-            dist = dists[idx] if idx < len(dists) else 1.0
-            sim = 1.0 - float(dist)
-        sims.append((doc, sim))
-
-    # sort by similarity desc
-    sims.sort(key=lambda x: x[1], reverse=True)
-
-    # optional: rerank top candidates with a cross-encoder here
-
-    return sims[:rerank_top_n]  # return top rerank_top_n with similarity
-
+    return resp.data[0].embedding
+
+# ─── TOP-K RETRIEVAL ──────────────────────────────────────────────────────────
+def query_top_k(query_text, k=5):
+    query_emb = embed_query(query_text)
+    results = collection.query(
+        query_embeddings=[query_emb],
+        n_results=k
+    )
+    # results['documents'] is a list of lists (one per query)
+    # results['distances'] is a list of lists (one per query)
+    # We'll return a list of (chunk, distance) tuples
+    docs = results['documents'][0] if results['documents'] else []
+    dists = results['distances'][0] if results['distances'] else []
+    return list(zip(docs, dists))
 
 # Example usage:
 if __name__ == "__main__":

model_serving/serve_models.sh

@@ -0,0 +1,67 @@
+#!/bin/bash
+
+# ==============================
+# Customizable parameters
+# ==============================
+CHAT_MODEL_PATH=${1:-Qwen/Qwen3-0.6B}                # First argument or default
+CHAT_MODEL_NAME=${2:-Qwen3-0.6B}                     # Second argument or default
+CHAT_PORT=${3:-21002}
+
+EMBED_MODEL_PATH=${4:-Qwen/Qwen3-Embedding-0.6B}     # Fourth argument or default
+EMBED_MODEL_NAME=${5:-Qwen3-Embedding-0.6B}          # Fifth argument or default
+EMBED_PORT=${6:-21003}
+
+API_PORT=${7:-8000}
+NGROK_URL=${8:-https://example-tunnel.ngrok-free.app} # Eighth argument or default
+
+# ==============================
+# Start services
+# ==============================
+echo "Starting controller..."
+nohup python3 -m fastchat.serve.controller \
+    --host localhost \
+    --port 21001 \
+    > controller.log 2>&1 &
+sleep 3
+
+echo "Starting $CHAT_MODEL_NAME worker..."
+nohup python3 -m fastchat.serve.model_worker \
+    --model-path "$CHAT_MODEL_PATH" \
+    --model-name "$CHAT_MODEL_NAME" \
+    --host localhost \
+    --port $CHAT_PORT \
+    --worker-address "http://localhost:$CHAT_PORT" \
+    --controller-address http://localhost:21001 \
+    > worker_chat.log 2>&1 &
+sleep 5
+
+echo "Starting $EMBED_MODEL_NAME worker..."
+nohup python3 -m fastchat.serve.model_worker \
+    --model-path "$EMBED_MODEL_PATH" \
+    --model-name "$EMBED_MODEL_NAME" \
+    --host localhost \
+    --port $EMBED_PORT \
+    --worker-address "http://localhost:$EMBED_PORT" \
+    --controller-address http://localhost:21001 \
+    > worker_embed.log 2>&1 &
+sleep 5
+
+echo "Starting OpenAI API server on port $API_PORT..."
+nohup python3 -m fastchat.serve.openai_api_server \
+    --host 0.0.0.0 \
+    --port $API_PORT \
+    --controller-address http://localhost:21001 \
+    --allowed-origins '["*"]' \
+    > api_server.log 2>&1 &
+
+echo "✅ All servers started!"
+echo "Logs: controller.log, worker_chat.log, worker_embed.log, api_server.log"
+
+# ==============================
+# Start ngrok tunnel
+# ==============================
+while true; do
+  ngrok http $API_PORT --url "$NGROK_URL" --log=stdout
+  echo "ngrok exited unexpectedly, restarting in 5s…" >&2
+  sleep 5
+done