feat: implement multi-stage candidate ingestion and matching pipeline with UI tracking and backend schema support

README.md

@@ -8,155 +8,284 @@ pinned: false
 app_port: 7860
 ---
 
-# TalentPulse — AI Candidate Matching System
 
-## Project Overview
-**TalentPulse** is a production-grade, two-stage AI pipeline for matching job descriptions (JDs) against large candidate pools. The system provides immense business value to technical recruiters and hiring managers by replacing manual resume screening with semantic vector search and neural reranking. It enables session-based candidate batching, career trajectory scoring, and LLM-generated explanations grounded in structured gap analysis.
+# TalentPulse: AI-Powered Candidate Matching System
+
+
+## Overview
+
+TalentPulse is a production-grade, full-stack AI system for matching job descriptions against large candidate pools. It replaces manual resume screening with semantic retrieval, neural reranking, structured gap analysis, and LLM-generated explanations.
+
+The platform is built for recruiters and hiring teams who need fast, explainable, and configurable candidate matching. It supports session-based candidate batches, dynamic scoring weights, trajectory analysis, and reusable matching workflows for A/B testing and precision hiring.
 
 ## Key Features
-* **Session-based Architecture**: Candidates are uploaded to named sessions, allowing JDs to be matched against specific candidate batches independently for A/B testing and organized workflows.
-* **Two-Stage AI Matching Pipeline**: 
-    * *Stage 1 (Retrieval)*: Fast bi-encoder vector search in Qdrant (~50-100ms) combined with weighted structured scoring (skill overlap, years of experience, etc.).
-    * *Stage 2 (Reranking)*: Cross-encoder reranking jointly re-scores the top-50 shortlist, fused via Reciprocal Rank Fusion.
-* **Live Weight Sliders**: Users can dynamically adjust the weights of scoring components (e.g., semantic vs. skills). This triggers a pure in-memory rerank returning in <100ms without new model inference.
-* **Structured Gap Analysis & LLM Explanations**: The system pre-computes missing skills, experience gaps, and location mismatches. A Groq LLM generates explanations directly grounded in this data.
-* **Trajectory Scoring**: Computes career growth velocity from work history timelines, rewarding fast promotions at funded product companies.
-* **JD Quality Feedback**: Evaluates Job Descriptions for vagueness, breadth, and missing signals.
+
+### Session-based Candidate Management
+Group candidates into named sessions for isolated workflows and repeatable matching experiments.
+
+### Two-stage AI Matching Pipeline
+- **Stage 1: Retrieval** — Fast vector search in Qdrant with structured scoring for skills, experience, and other signals.
+- **Stage 2: Reranking** — Cross-encoder reranking of the shortlist, fused with Reciprocal Rank Fusion.
+
+### Live Weight Sliders
+Adjust matching priorities in real time and rerank results in memory without running new model inference.
+
+### Structured Gap Analysis
+Detect missing skills, experience gaps, and mismatches to generate grounded candidate explanations.
+
+### LLM-generated Explanations
+Use Groq-powered LLM responses based on the precomputed gap analysis.
+
+### Trajectory Scoring
+Estimate career growth velocity from work history and reward strong advancement patterns.
+
+### JD Quality Feedback
+Evaluate job descriptions for clarity, breadth, and missing signals.
 
 ## Tech Stack
-| Component | Technology |
-| :--- | :--- |
-| **Backend Framework** | FastAPI (Python 3.11/3.12), Uvicorn |
-| **Frontend Framework** | Next.js 16 (Node.js 20), React, Tailwind CSS v4 |
-| **Database & ORM** | Neon Postgres (Asyncpg), SQLAlchemy, Alembic |
-| **Vector Database** | Qdrant Cloud |
-| **Task Queue & Cache**| Celery, Redis Cloud |
-| **Embedding Model** | `BAAI/bge-small-en-v1.5` (Local CPU via SentenceTransformers) |
-| **Reranker Model** | `BAAI/bge-reranker-v2-m3` (Local CPU via FlagEmbedding) |
-| **LLM Provider** | Groq (`llama-3.3-70b-versatile`) |
-| **Infrastructure** | Docker, Nginx, Supervisord (HuggingFace Spaces deployment) |
+
+| Layer | Technology |
+|------|------------|
+| Frontend | Next.js 16, React, Tailwind CSS v4 |
+| Backend | FastAPI, Uvicorn |
+| Database | Neon Postgres, Asyncpg, SQLAlchemy, Alembic |
+| Vector Search | Qdrant Cloud |
+| Async Jobs | Celery |
+| Cache | Redis Cloud |
+| Embeddings | BAAI/bge-small-en-v1.5 via SentenceTransformers |
+| Reranking | BAAI/bge-reranker-v2-m3 via FlagEmbedding |
+| LLM Provider | Groq (llama-3.3-70b-versatile) |
+| Deployment | Docker, Nginx, Supervisord, HuggingFace Spaces |
 
 ## Architecture Overview
-* **Frontend to Backend Flow**: The Next.js frontend communicates with the FastAPI backend via REST API calls routed through an Nginx reverse proxy.
-* **Data & Async Flow**: Uploaded candidates are sent to an async Celery worker queue backed by Redis. Workers extract text, embed it using SentenceTransformers, and store vectors in Qdrant and relational data in Postgres. 
-* **API Flow**: Complex matching requests retrieve candidates from Qdrant, re-score them using the FlagReranker model locally, and cache the finalized matches in Redis.
-* **File Handling**: Handled via multipart file uploads into the FastAPI server and processed in memory/chunks.
+
+```mermaid
+graph TD
+    UI[Next.js Frontend] -->|REST API| Proxy[Nginx Reverse Proxy]
+    Proxy --> API[FastAPI Backend]
+
+    API -->|Async Tasks| Queue[Redis / Celery Queue]
+    Queue --> Worker[Celery Workers]
+
+    API -->|Read / Write| DB[(Neon Postgres)]
+    Worker -->|Persist Metadata| DB
+
+    API -->|Vector Search| VectorDB[(Qdrant Cloud)]
+    Worker -->|Store Embeddings| VectorDB
+
+    API -->|In-Memory Rerank| LocalAI[Local Reranker Model]
+    API -->|LLM Explanations| LLM[Groq API]
+    Worker -->|LLM Jobs| LLM
+````
 
 ## Project Structure
+
 ```text
 /
 ├── backend/
-│   ├── alembic/            # Database migrations
+│   ├── alembic/
 │   ├── src/
-│   │   ├── matching/       # Stage 1 retrieval, Stage 2 reranking, LLM logic
-│   │   ├── ml/             # Embedding models, feature building, cross-encoders
-│   │   ├── models/         # SQLAlchemy ORM definitions
-│   │   ├── routers/        # FastAPI endpoints
-│   │   ├── schemas/        # Pydantic validation schemas
-│   │   └── workers/        # Celery tasks (ingest, explanations)
-│   └── main.py             # FastAPI application entry point
+│   │   ├── matching/
+│   │   ├── ml/
+│   │   ├── models/
+│   │   ├── routers/
+│   │   ├── schemas/
+│   │   └── workers/
+│   ├── main.py
+│   └── requirements.txt
 ├── frontend/
-│   ├── public/             # Static assets (SVGs)
-│   ├── src/app/            # Next.js App Router pages (jds, sessions, pipeline)
-│   └── src/lib/api.ts      # API client wrappers
-├── docker-compose.yml      # Local development compose configuration
-├── Dockerfile              # Multi-stage build for production HuggingFace deployment
-├── supervisord.conf        # Process manager for containerized backend, frontend, and workers
-└── nginx.conf              # Reverse proxy configuration
+│   ├── public/
+│   ├── src/
+│   │   ├── app/
+│   │   └── lib/
+│   ├── next.config.ts
+│   └── globals.css
+├── docker-compose.yml
+├── Dockerfile
+├── supervisord.conf
+└── nginx.conf
+```
+
+## Core Modules & Responsibilities
+
+### Backend
+
+* **backend/src/ml**
+  Handles model loading, text embedding, and feature extraction.
+
+* **backend/src/matching**
+  Implements retrieval, reranking, weighted scoring, and explanation logic.
+
+* **backend/src/workers**
+  Runs background jobs such as candidate ingestion and explanation generation.
+
+* **backend/src/routers**
+  Exposes API endpoints for sessions, JDs, candidates, matching, and health checks.
+
+### Frontend
+
+* **frontend/src/app**
+  Contains user-facing routes such as sessions, JD details, and pipeline orchestration.
+
+* **frontend/src/lib**
+  Centralized API client wrappers.
+
+## Application Flows
+
+### Candidate Upload & Ingestion Flow
+
+```mermaid
+sequenceDiagram
+    actor User
+    participant UI as Next.js UI
+    participant API as FastAPI Router
+    participant Queue as Redis / Celery Queue
+    participant Worker as Celery Worker
+    participant Store as Postgres + Qdrant
+
+    User->>UI: Upload candidate CSV/JSON
+    UI->>API: POST /api/candidates/upload
+    API->>Queue: Dispatch ingest_candidates_batch
+    API-->>UI: Return task ID
+    UI->>API: Poll /api/candidates/status/{task_id}
+    Worker->>Queue: Fetch task
+    Worker->>Worker: Parse candidate data
+    Worker->>Worker: Compute embeddings and growth velocity
+    Worker->>Store: Save metadata and vector points
+    Worker-->>Queue: Mark task complete
+    API-->>UI: Return success status
 ```
 
-## Backend Documentation
-* **Modules**: The backend is divided into specialized modules for `ml` (models and feature extractors), `matching` (retrieval and scoring logic), and `workers` (Celery background tasks).
-* **Routers**: Divided into `sessions.py`, `jds.py`, `candidates.py`, and `matching.py`. 
-* **Controllers & Services**: Heavy logic is offloaded to ML utility scripts like `stage1_retrieve` and `stage2_rerank`. Explanation generation utilizes a revolving list of Groq API keys to manage limits.
-* **Middleware**: CORS middleware is configured to allow all origins (`*`).
-* **Jobs**: Managed via Celery workers for tasks like `ingest_candidates_batch` and `generate_top_explanations`.
-
-## Frontend Documentation
-* **Pages & Layouts**: Uses Next.js App Router (`src/app/`). Key pages include `sessions/page.tsx` (Candidate pools), `jds/[id]/page.tsx` (JD detail and matching), and `pipeline/page.tsx` (Automated run orchestration).
-* **Styling**: Powered by Tailwind CSS v4 utilizing custom CSS variables defined in `globals.css`.
-* **API Client**: Centralized in `frontend/src/lib/api.ts` utilizing native `fetch` wrappers.
-* **State Management**: Native React Hooks (`useState`, `useEffect`, `useCallback`) alongside local storage for pipeline state. 
-
-## Database Design
-Powered by **PostgreSQL** with schema managed via **Alembic**.
-* **`sessions`**: Holds candidate grouping metadata (`id`, `name`, `candidate_count`).
-* **`job_descriptions`**: Stores JD raw text, parsed skill requirements, quality assessments, and custom scoring weights.
-* **`candidates`**: Extensive model tracking candidate demographics, parsed work experience (JSON), skills, generated trajectory scores (`growth_velocity`), and embeddings (`qdrant_id`).
-* **`match_results`**: Links JDs and Candidates. Stores stage 1 and 2 scores, gap analysis (JSON), and generated LLM explanations.
+### Matching & Reranking Flow
+
+```mermaid
+sequenceDiagram
+    actor User
+    participant UI as Next.js UI
+    participant API as FastAPI Router
+    participant Qdrant as Vector DB
+    participant Reranker as Local Reranker
+    participant Cache as Redis Cache
+
+    User->>UI: Open JD and click Match
+    UI->>API: POST /api/match/{jd_id}
+    API->>Qdrant: Retrieve top candidates
+    Qdrant-->>API: Return top-K vectors
+    API->>Reranker: Cross-encoder reranking
+    Reranker-->>API: Return adjusted scores
+    API->>API: Apply rank fusion and weights
+    API->>Cache: Store result
+    API-->>UI: Return ranked candidates
+
+    User->>UI: Adjust weight sliders
+    UI->>API: POST /api/match/{jd_id}/rerank
+    API->>API: Recompute ranking in memory
+    API-->>UI: Return updated ordering
+```
+
+### Explain & Refine Flow
+
+```mermaid
+sequenceDiagram
+    actor User
+    participant UI as Next.js UI
+    participant API as FastAPI Router
+    participant DB as Postgres
+    participant LLM as Groq API
+
+    User->>UI: Open candidate match details
+    UI->>API: POST /api/match/{jd_id}/candidates/{candidate_id}/explain
+    API->>DB: Load match data and gap analysis
+    API->>LLM: Generate grounded explanation
+    LLM-->>API: Return explanation text
+    API-->>UI: Show explanation to user
+```
+
+## API Documentation
+
+| Method | Path                                                 | Purpose                    |
+| ------ | ---------------------------------------------------- | -------------------------- |
+| POST   | /api/sessions                                        | Create a candidate session |
+| GET    | /api/sessions                                        | List sessions              |
+| POST   | /api/jds                                             | Create a job description   |
+| GET    | /api/jds                                             | List job descriptions      |
+| POST   | /api/candidates/upload?session_id=                   | Upload candidate files     |
+| GET    | /api/candidates/status/{task_id}                     | Check task progress        |
+| POST   | /api/match/{jd_id}?session_id=                       | Run full matching pipeline |
+| POST   | /api/match/{jd_id}/rerank                            | Rerank in memory           |
+| POST   | /api/match/{jd_id}/candidates/{candidate_id}/explain | Generate explanation       |
+| GET    | /health                                              | Health check               |
+
+## Database Models
+
+* **Session** — Candidate batch container
+* **JobDescription** — Stores JD text and parsed requirements
+* **Candidate** — Stores profile, skills, work history, embeddings
+* **MatchResult** — Stores scores, gaps, explanations, weights
+
+## Authentication & Security
+
+* No formal authentication yet
+* CORS allows all origins
+* Minimal admin utility route exists
+
+## State Management
+
+* React Hooks (`useState`, `useEffect`, `useCallback`)
+* Local storage for persistence
+* Redis for backend caching
 
 ## Caching & Performance
-* **Cache Usage**: Redis is utilized to cache complete `/api/match` results based on `jd_id` and `session_id`.
-* **Optimization**: AI embedding models are pre-downloaded and baked into the Docker image universally using `HF_HOME="/app/models"`, eliminating runtime downloads. 
-* **Database**: SQLAlchemy's internal prepared statement cache is explicitly disabled to work properly with Asyncpg and Neon Db pools.
-
-## Storage & File Handling
-* **Uploads**: Users upload CSV/JSON candidate files directly to the API endpoint `/api/candidates/upload` via `FormData`. File processing is dispatched to Celery.
-
-## Real-Time Features
-* **Polling**: The frontend polls the backend's `/api/candidates/status/{task_id}` endpoint every 3 seconds to update the UI on Celery ingest and vector embedding progress.
-
-## Authentication & Authorization
-* **Current State**: The API accepts all origins without authenticated session barriers in the current logic. No formal login or RBAC is implemented.
-
-## Main User Flows
-1.  **Candidate Pool Creation**: User creates a Session -> Uploads a CSV -> Celery `ingest_candidates_batch` parses resumes, extracts growth velocity, embeddings via SentenceTransformers, and saves points to Qdrant/Postgres.
-2.  **Matching Pipeline**: User submits JD -> System runs `parse_jd_requirements` -> User views JD Detail -> Triggers Match. The backend queries Qdrant (Stage 1) -> Reranks top candidates via Cross-Encoder (Stage 2) -> Applies Rank Fusion.
-3.  **Explain & Refine**: User clicks a matched candidate -> Triggers async Groq LLM assessment comparing candidate gaps to JD requirements. User drags weight sliders -> triggers purely in-memory rerank.
-
-## API Reference
-| Method | Path | Purpose |
-| :--- | :--- | :--- |
-| **POST** | `/api/sessions` | Create a candidate session. |
-| **GET** | `/api/sessions` | List all sessions. |
-| **POST** | `/api/jds` | Create a Job Description. |
-| **POST** | `/api/candidates/upload?session_id=` | Upload Candidate files to a session. |
-| **POST** | `/api/match/{jd_id}?session_id=` | Trigger full Qdrant Retrieval + Reranking pipeline. |
-| **POST** | `/api/match/{jd_id}/rerank` | Rescore candidates purely in memory using new weights. |
-| **POST** | `/api/match/{jd_id}/candidates/{candidate_id}/explain` | Generate LLM explanation. |
-| **GET** | `/api/candidates/status/{task_id}` | Poll Celery task status. |
-
-## Setup Instructions
-1.  **Install & Clone**: Ensure Docker and Node.js are installed.
-2.  **Env Config**: Create a `.env` in `backend/` and populate it with `DATABASE_URL`, `REDIS_URL`, `QDRANT_URL`, `QDRANT_API_KEY`, and `GROQ_API_KEY`. 
-3.  **Run Locally (Docker Compose)**: 
-    For local development with an external Postgres/Qdrant instance, run:
-    ```bash
-    docker-compose up --build
-    ```
-    This spins up the FastAPI backend on port `8000`, Next.js on `3000`, and a Celery worker.
-4.  **Database Migrations**:
-    Apply Alembic schemas:
-    ```bash
-    cd backend
-    alembic upgrade head
-    ```
-    *(Alternatively, you can wipe and recreate the db using `python clean_db.py`)*
+
+* Cached match results by `jd_id + session_id`
+* Models pre-downloaded into Docker image
+* SQLAlchemy cache tuned for Neon pooling
+
+## Setup & Installation
+
+### Run Locally
+
+```bash
+docker-compose up --build
+```
+
+### Database Migration
+
+```bash
+cd backend
+alembic upgrade head
+```
 
 ## Environment Variables
-* `DATABASE_URL`: Postgres connection string (should use `postgresql+asyncpg` internally).
-* `QDRANT_URL` / `QDRANT_API_KEY`: Credentials for Qdrant Vector Cloud.
-* `REDIS_URL`: Redis connection used for Celery task queuing and matching cache.
-* `GROQ_API_KEY`: API Key(s) for LLM generation. Comma-separated list for cycling limits.
-* `GROQ_MODEL`: Recommended `llama-3.3-70b-versatile`.
-* `NEXT_PUBLIC_API_URL`: Frontend configuration targeting backend (e.g., `http://localhost:8000`).
-
-## Scripts
-* **`clean_db.py`**: Drops the public schema natively and recreates it. A fast teardown utility.
-* **Docker Build**: Uses `npm run build` to generate Next.js `.next/standalone` outputs.
-
-## Deployment Notes
-The application is designed to be deployed as a unified Docker container, specifically optimized for **HuggingFace Spaces**.
-* **Build Process**: A multi-stage `Dockerfile` compiles the Next.js frontend into standalone static files, then installs Python 3.11, Nginx, Node, and Supervisord into the final image.
-* **AI Pre-baking**: BAAI Embedding and Reranker models are downloaded during the image build step to `/app/models` to ensure instantaneous startup without runtime downloading.
-* **Routing**: Exposes port `7860`. Supervisord manages Uvicorn (FastAPI), Node (Next.js server), Nginx (Reverse Proxy), and Celery worker simultaneously inside the single container.
-
-## Troubleshooting
-* **Alembic asyncpg URLs**: If Alembic fails, ensure `DATABASE_URL` is cleaned. The `env.py` automatically converts `postgresql://` to `postgresql+asyncpg://` and strips SSL queries if needed.
-* **Nginx Permissions**: The Dockerfile aggressively configures `/tmp` directories for Nginx and runs the container as `appuser` (UID 1000) to comply with non-root hosting requirements.
-* **Database Prepared Statements Warning**: Neon Serverless Postgres pools require disabling SQLAlchemy's statement cache. This is configured natively in `database.py`.
-
-## Future Improvements
-* **Authentication**: Add JWT token generation and a robust User/RBAC data model since the endpoints currently lack auth middleware.
-* **Real-time Streaming**: Replace aggressive client-side polling loops with Websockets or Server-Sent Events (SSE) to broadcast pipeline events.
-* **Object Storage Integration**: Offload raw parsed resumes and CSVs to an S3-compatible service to prevent local container bloating.
+
+```env
+DATABASE_URL=
+QDRANT_URL=
+QDRANT_API_KEY=
+REDIS_URL=
+GROQ_API_KEY=
+GROQ_MODEL=
+EMBEDDING_MODEL=
+RERANKER_MODEL=
+NEXT_PUBLIC_API_URL=
+```
+
+## Deployment
+
+* Multi-stage Docker build
+* Runs FastAPI + Next.js + Celery + Nginx
+* Optimized for HuggingFace Spaces
+* Exposes port `7860`
+
+## Improvement Recommendations
+
+* Add JWT auth + RBAC
+* Replace polling with WebSockets / SSE
+* Add object storage
+* Add automated tests
+* Add observability & metrics
+
+## Quick Summary
+
+TalentPulse combines semantic search, reranking, and LLM reasoning to help recruiters identify the best candidates faster, with explainable AI-powered hiring workflows.
+

backend/src/routers/candidates.py

@@ -15,7 +15,7 @@ from ..workers.ingest import ingest_candidates_batch
 
 router = APIRouter()
 
-BATCH_SIZE = 100
+BATCH_SIZE = 500  # Large enough to keep typical uploads in one batch
 
 
 @router.post("/upload", response_model=UploadResponse)
@@ -65,6 +65,7 @@ async def upload_candidates(
 
     return UploadResponse(
         task_id=task_ids[0] if task_ids else "",
+        task_ids=task_ids,
         queued=len(rows),
         message=f"Queued {len(rows)} candidates across {len(task_ids)} batches",
     )

backend/src/schemas/candidate.py

@@ -26,7 +26,8 @@ class CandidateResponse(BaseModel):
 
 
 class UploadResponse(BaseModel):
-    task_id: str
+    task_id: str           # First task ID (backward compat)
+    task_ids: list[str] = []  # ALL task IDs — poll all to confirm full ingestion
     queued: int
     message: str
 

frontend/src/app/pipeline/page.tsx

@@ -24,7 +24,7 @@ const DEFAULT_STATE: PipelineState = { status: "idle", sessionName: "", jdsInfo:
 
 export default function PipelinePage() {
   const router = useRouter();
-  
+
   // Pipeline definition
   const steps = [
     { id: "idle", label: "Configure Run", icon: "📝" },
@@ -47,7 +47,7 @@ export default function PipelinePage() {
   // Architecture state
   const [state, setState] = useState<PipelineState>(DEFAULT_STATE);
   const [error, setError] = useState<string | null>(null);
-  
+
   const timerRef = useRef<ReturnType<typeof setInterval> | null>(null);
 
   useEffect(() => {
@@ -62,7 +62,7 @@ export default function PipelinePage() {
           if (p.status === "embedding" && p.taskId) pollEmbedding(p.taskId, p);
           if (p.status === "matching" && p.jdIds.length > 0 && p.sessionId) runMatches(p.jdIds, p.sessionId, p);
         }
-      } catch (e) {}
+      } catch (e) { }
     }
   }, []);
 
@@ -72,8 +72,8 @@ export default function PipelinePage() {
       if (!timerRef.current && state.startTime) {
         timerRef.current = setInterval(() => {
           setState(s => {
-             if (s.status === "idle" || s.status === "complete") return s;
-             return { ...s, elapsedTime: Math.floor((Date.now() - (s.startTime || Date.now())) / 1000) };
+            if (s.status === "idle" || s.status === "complete") return s;
+            return { ...s, elapsedTime: Math.floor((Date.now() - (s.startTime || Date.now())) / 1000) };
           });
         }, 1000);
       }
@@ -83,7 +83,7 @@ export default function PipelinePage() {
         timerRef.current = null;
       }
       if (state.status === "complete") {
-         localStorage.removeItem("talentpulse_pipeline");
+        localStorage.removeItem("talentpulse_pipeline");
       }
     }
   }, [state.status, state.startTime]);
@@ -156,21 +156,24 @@ export default function PipelinePage() {
       // 1. Create Session first
       const session = await api.createSession(sessionName, "Automated Candidate Batch Ingestion");
       const sessionIdStr = (session as any).id;
-      
+
       // 2. Create JDs scoped to that session
       const jdPromises = jds.map(jd => api.createJD(jd.title, jd.desc, sessionIdStr));
       const createdJDs = await Promise.all(jdPromises);
       const jdIds = createdJDs.map(j => (j as any).id);
-      
+
       updateState({ sessionId: sessionIdStr, jdIds });
 
-      // 3. Upload file
+      // 3. Upload file — may return multiple batch task IDs for large CSVs
       const uploadRes = await api.uploadCandidates(file, sessionIdStr);
-      
+      const allTaskIds: string[] = (uploadRes as any).task_ids?.length
+        ? (uploadRes as any).task_ids
+        : [uploadRes.task_id];
+
       updateState({ status: "embedding", taskId: uploadRes.task_id });
-      
-      // 3. Poll embedding
-      pollEmbedding(uploadRes.task_id, { ...state, status: "embedding", sessionId: (session as any).id, jdIds, startTime: start });
+
+      // Poll ALL batch tasks — only proceed to matching when every batch is done
+      pollEmbedding(allTaskIds, { ...state, status: "embedding", sessionId: (session as any).id, jdIds, startTime: start });
 
     } catch (e: any) {
       setError("Pipeline failed: " + e.message);
@@ -178,17 +181,21 @@ export default function PipelinePage() {
     }
   };
 
-  const pollEmbedding = async (taskId: string, currentState: PipelineState) => {
+  const pollEmbedding = async (taskIds: string | string[], currentState: PipelineState) => {
+    const ids = Array.isArray(taskIds) ? taskIds : [taskIds];
     const poll = setInterval(async () => {
       try {
-        const s = await api.taskStatus(taskId);
-        if (s.status === "SUCCESS") {
+        // Check ALL batch tasks — only proceed when EVERY one is SUCCESS
+        const statuses = await Promise.all(ids.map(id => api.taskStatus(id)));
+        const allDone = statuses.every(s => s.status === "SUCCESS");
+        const anyFailed = statuses.some(s => s.status === "FAILURE");
+        if (allDone) {
           clearInterval(poll);
           updateState({ status: "matching" });
           runMatches(currentState.jdIds, currentState.sessionId!, currentState);
-        } else if (s.status === "FAILURE") {
+        } else if (anyFailed) {
           clearInterval(poll);
-          setError("Vector embedding failed.");
+          setError("Vector embedding failed for one or more batches.");
           updateState({ status: "idle" });
         }
       } catch (e) {
@@ -216,7 +223,7 @@ export default function PipelinePage() {
             }
           }
         }
-        
+
         if (stillPending.length > 0) {
           pendingJds = stillPending;
           setTimeout(pollMatches, 3000);
@@ -227,14 +234,14 @@ export default function PipelinePage() {
             const existing = JSON.parse(localStorage.getItem("tp_session_jds") || "{}");
             existing[currentState.sessionId!] = currentState.jdIds;
             localStorage.setItem("tp_session_jds", JSON.stringify(existing));
-          } catch (e) {}
+          } catch (e) { }
         }
       } catch (e: any) {
         setError("Matching failed: " + e.message);
         updateState({ status: "idle" });
       }
     };
-    
+
     pollMatches();
   };
 
@@ -256,9 +263,9 @@ export default function PipelinePage() {
       {/* STEPPER UI */}
       <div className="mb-12 relative">
         <div className="absolute top-6 left-[10%] right-[10%] h-0.5 bg-[var(--color-border-strong)] -z-10" />
-        <div className="absolute top-6 left-[10%] h-0.5 bg-[var(--color-brand)] -z-10 transition-all duration-700" 
-             style={{ width: `${Math.max(0, (currentStepIdx / (steps.length - 1)) * 80)}%` }} />
-             
+        <div className="absolute top-6 left-[10%] h-0.5 bg-[var(--color-brand)] -z-10 transition-all duration-700"
+          style={{ width: `${Math.max(0, (currentStepIdx / (steps.length - 1)) * 80)}%` }} />
+
         <div className="flex justify-between relative z-10">
           {steps.map((step, idx) => {
             const isActive = state.status === step.id;
@@ -266,9 +273,9 @@ export default function PipelinePage() {
             return (
               <div key={step.id} className="flex flex-col items-center w-24">
                 <div className={`w-12 h-12 rounded-full flex items-center justify-center text-xl mb-3 border-2 transition-all duration-500
-                    ${isActive ? 'bg-[var(--color-brand-dim)] border-[var(--color-brand-light)] text-white shadow-[0_0_20px_var(--color-brand-dim)]' 
-                    : isPast ? 'bg-[var(--color-brand)] border-[var(--color-brand)] text-white' 
-                    : 'bg-[var(--color-surface-2)] border-[var(--color-border-strong)] text-[var(--color-muted)] opacity-50' }`}
+                    ${isActive ? 'bg-[var(--color-brand-dim)] border-[var(--color-brand-light)] text-white shadow-[0_0_20px_var(--color-brand-dim)]'
+                    : isPast ? 'bg-[var(--color-brand)] border-[var(--color-brand)] text-white'
+                      : 'bg-[var(--color-surface-2)] border-[var(--color-border-strong)] text-[var(--color-muted)] opacity-50'}`}
                 >
                   {step.icon}
                 </div>
@@ -302,39 +309,39 @@ export default function PipelinePage() {
         <div className="bg-[var(--color-card)] border border-[var(--color-border)] rounded-2xl p-8 shadow-xl shadow-black/5">
           <div className="mb-6">
             <label className="block text-xs font-bold text-[var(--color-muted)] mb-2 uppercase tracking-wider">Candidate Batch Name</label>
-            <input type="text" placeholder="e.g. Q3 Engineering Batch (100k)" 
+            <input type="text" placeholder="e.g. Q3 Engineering Batch (100k)"
               className="w-full bg-[var(--color-surface-2)] border border-[var(--color-border-strong)] rounded-xl px-4 py-3 text-sm outline-none focus:border-[var(--color-brand)] transition-all"
               value={sessionName} onChange={e => setSessionName(e.target.value)} />
           </div>
 
           <div className="mb-8">
-             <label className="block text-xs font-medium text-[var(--color-muted)] mb-2">Candidates CSV (.csv, .json)</label>
-             <input type="file" accept=".csv,.json,.jsonl" 
-               className="w-full text-sm text-[var(--color-muted)] file:mr-4 file:py-2 file:px-4 file:rounded-xl file:border-0 file:text-sm file:font-semibold file:bg-[var(--color-brand-dim)] file:text-[var(--color-brand-light)] hover:file:bg-[var(--color-brand)] hover:file:text-white transition-all cursor-pointer border border-[var(--color-border-strong)] rounded-xl p-2"
-               onChange={handleFileChange} />
-             {csvRowCount > 0 && (
-               <p className="mt-2 text-xs text-[var(--color-muted)]">
-                 📄 Detected <strong className="text-[var(--color-brand-light)]">{csvRowCount}</strong> candidate rows (excluding header)
-               </p>
-             )}
+            <label className="block text-xs font-medium text-[var(--color-muted)] mb-2">Candidates CSV (.csv, .json)</label>
+            <input type="file" accept=".csv,.json,.jsonl"
+              className="w-full text-sm text-[var(--color-muted)] file:mr-4 file:py-2 file:px-4 file:rounded-xl file:border-0 file:text-sm file:font-semibold file:bg-[var(--color-brand-dim)] file:text-[var(--color-brand-light)] hover:file:bg-[var(--color-brand)] hover:file:text-white transition-all cursor-pointer border border-[var(--color-border-strong)] rounded-xl p-2"
+              onChange={handleFileChange} />
+            {csvRowCount > 0 && (
+              <p className="mt-2 text-xs text-[var(--color-muted)]">
+                📄 Detected <strong className="text-[var(--color-brand-light)]">{csvRowCount}</strong> candidate rows (excluding header)
+              </p>
+            )}
           </div>
 
           <div className="mb-6 border-t border-[var(--color-border-strong)] pt-6">
             <div className="flex items-center justify-between mb-4">
               <label className="block text-sm font-bold text-[var(--color-text)]">Job Descriptions to Match</label>
-              <button 
+              <button
                 onClick={addJd}
                 className="text-xs px-3 py-1.5 rounded-lg bg-[var(--color-surface-2)] border border-[var(--color-border)] text-[var(--color-muted)] hover:text-[var(--color-text)] transition-colors"
               >
                 + Add Another JD
               </button>
             </div>
-            
+
             <div className="space-y-6">
               {jds.map((jd, idx) => (
                 <div key={idx} className="bg-[var(--color-surface-2)] p-4 rounded-xl border border-[var(--color-border)] relative group">
                   {jds.length > 1 && (
-                    <button 
+                    <button
                       onClick={() => removeJd(idx)}
                       className="absolute -top-2 -right-2 w-6 h-6 rounded-full bg-[var(--color-card)] border border-[var(--color-border-strong)] text-[var(--color-muted)] hover:text-red-400 hover:border-red-400 flex items-center justify-center text-xs opacity-0 group-hover:opacity-100 transition-all z-10"
                     >
@@ -343,7 +350,7 @@ export default function PipelinePage() {
                   )}
                   <div className="mb-3">
                     <label className="block text-xs font-medium text-[var(--color-muted)] mb-2">JD {idx + 1} Title</label>
-                    <input type="text" placeholder="e.g. Senior Backend Engineer" 
+                    <input type="text" placeholder="e.g. Senior Backend Engineer"
                       className="w-full bg-[var(--color-card)] border border-[var(--color-border-strong)] rounded-lg px-3 py-2 text-sm outline-none focus:border-[var(--color-brand)] transition-all"
                       value={jd.title} onChange={e => updateJd(idx, "title", e.target.value)} />
                   </div>
@@ -376,11 +383,9 @@ export default function PipelinePage() {
               onChange={e => setRankingCap(Number(e.target.value))}
               className="w-full h-2 rounded-lg appearance-none cursor-pointer"
               style={{
-                background: `linear-gradient(to right, var(--color-brand) ${
-                  ((rankingCap / (csvRowCount > 0 ? csvRowCount : 200)) * 100).toFixed(1)
-                }%, var(--color-border-strong) ${
-                  ((rankingCap / (csvRowCount > 0 ? csvRowCount : 200)) * 100).toFixed(1)
-                }%)`
+                background: `linear-gradient(to right, var(--color-brand) ${((rankingCap / (csvRowCount > 0 ? csvRowCount : 200)) * 100).toFixed(1)
+                  }%, var(--color-border-strong) ${((rankingCap / (csvRowCount > 0 ? csvRowCount : 200)) * 100).toFixed(1)
+                  }%)`
               }}
             />
             <div className="flex justify-between text-[10px] text-[var(--color-muted)] mt-1">
@@ -388,12 +393,12 @@ export default function PipelinePage() {
               <span>{csvRowCount > 0 ? csvRowCount : 200}</span>
             </div>
             {/* RAM Warning for BGE model */}
-            <div className="mt-3 flex items-start gap-2 bg-amber-500/10 border border-amber-500/25 rounded-xl px-4 py-3">
+            {/* <div className="mt-3 flex items-start gap-2 bg-amber-500/10 border border-amber-500/25 rounded-xl px-4 py-3">
               <span className="text-amber-400 text-sm mt-0.5">⚠️</span>
               <p className="text-xs text-amber-300/90 leading-relaxed">
                 <strong>Hugging Face Free Tier Notice:</strong> We use <code className="font-mono bg-black/20 px-1 rounded">BAAI/bge-reranker-v2-m3</code> for neural reranking. On the free tier, this model exceeds available RAM above ~72 candidates and the backend will crash. <strong>Keep the cap at or below 72</strong> for stable results.
               </p>
-            </div>
+            </div> */}
           </div>
 
           <button onClick={startPipeline}
@@ -403,46 +408,46 @@ export default function PipelinePage() {
         </div>
       ) : state.status === "complete" ? (
         <div className="text-center bg-[var(--color-card)] border border-[var(--color-border)] rounded-2xl p-10 shadow-xl shadow-black/5 animate-fade-in">
-           <div className="text-6xl mb-4">🎉</div>
-           <h2 className="text-2xl font-bold mb-2">Automated Inference Complete!</h2>
-           <p className="text-[var(--color-muted)] mb-8 max-w-sm mx-auto">
-             100% of candidate logic calculated safely for <strong>{state.jdIds.length}</strong> Job Descriptions. The background worker is aggressively pulling LLM explanations for the top 60 right now.
-           </p>
-           
-           <div className="max-w-md mx-auto bg-[var(--color-surface-2)] rounded-xl border border-[var(--color-border)] p-4 mb-6">
-              <div className="text-xs font-bold text-[var(--color-muted)] uppercase tracking-wider mb-3 text-left">View Matches By JD:</div>
-              <div className="space-y-2">
-                {state.jdsInfo.map((info, idx) => (
-                  <Link 
-                    key={idx}
-                    href={`/sessions/${state.sessionId}?jd_id=${state.jdIds[idx]}`}
-                    className="flex justify-between items-center bg-[var(--color-card)] hover:bg-[var(--color-card-hover)] p-3 rounded-lg border border-[var(--color-border-strong)] hover:border-[var(--color-brand)] transition-all group"
-                  >
-                    <span className="font-semibold text-sm truncate pr-4">{info.title || `Job Description ${idx + 1}`}</span>
-                    <span className="text-[10px] px-2 py-1 bg-[var(--color-brand-dim)] text-[var(--color-brand-light)] border border-[var(--color-brand-glow)] rounded-full flex-shrink-0">
-                      View Ranking →
-                    </span>
-                  </Link>
-                ))}
-              </div>
-           </div>
-           
-           <button onClick={() => updateState({ status: "idle", jdsInfo: [{ title: "", desc: "" }], jdIds: [], sessionName: "", startTime: undefined })}
-             className="text-xs text-[var(--color-muted)] hover:text-[var(--color-text)] underline underline-offset-2">
-             Start a new pipeline run
-           </button>
+          <div className="text-6xl mb-4">🎉</div>
+          <h2 className="text-2xl font-bold mb-2">Automated Inference Complete!</h2>
+          <p className="text-[var(--color-muted)] mb-8 max-w-sm mx-auto">
+            100% of candidate logic calculated safely for <strong>{state.jdIds.length}</strong> Job Descriptions. The background worker is aggressively pulling LLM explanations for the top 60 right now.
+          </p>
+
+          <div className="max-w-md mx-auto bg-[var(--color-surface-2)] rounded-xl border border-[var(--color-border)] p-4 mb-6">
+            <div className="text-xs font-bold text-[var(--color-muted)] uppercase tracking-wider mb-3 text-left">View Matches By JD:</div>
+            <div className="space-y-2">
+              {state.jdsInfo.map((info, idx) => (
+                <Link
+                  key={idx}
+                  href={`/sessions/${state.sessionId}?jd_id=${state.jdIds[idx]}`}
+                  className="flex justify-between items-center bg-[var(--color-card)] hover:bg-[var(--color-card-hover)] p-3 rounded-lg border border-[var(--color-border-strong)] hover:border-[var(--color-brand)] transition-all group"
+                >
+                  <span className="font-semibold text-sm truncate pr-4">{info.title || `Job Description ${idx + 1}`}</span>
+                  <span className="text-[10px] px-2 py-1 bg-[var(--color-brand-dim)] text-[var(--color-brand-light)] border border-[var(--color-brand-glow)] rounded-full flex-shrink-0">
+                    View Ranking →
+                  </span>
+                </Link>
+              ))}
+            </div>
+          </div>
+
+          <button onClick={() => updateState({ status: "idle", jdsInfo: [{ title: "", desc: "" }], jdIds: [], sessionName: "", startTime: undefined })}
+            className="text-xs text-[var(--color-muted)] hover:text-[var(--color-text)] underline underline-offset-2">
+            Start a new pipeline run
+          </button>
         </div>
       ) : (
         <div className="text-center bg-[var(--color-card)] border border-dashed border-[var(--color-border-strong)] rounded-2xl p-16 animate-fade-in">
-           <div className="w-16 h-16 border-4 border-[var(--color-brand-dim)] border-t-[var(--color-brand-light)] rounded-full animate-spin mx-auto mb-6" />
-           <h2 className="text-xl font-semibold mb-2">
-             {state.status === "uploading" ? "Broadcasting to Postgres DB..." 
-               : state.status === "embedding" ? "Running Core CPU Vector Space Projection..." 
-               : `Executing Dual-Stage Neural Match for ${state.jdIds.length} JDs...`}
-           </h2>
-           <p className="text-[var(--color-dimmer)] text-sm">
-             Do not close this tab. The timer will automatically pause and redirect upon completion.
-           </p>
+          <div className="w-16 h-16 border-4 border-[var(--color-brand-dim)] border-t-[var(--color-brand-light)] rounded-full animate-spin mx-auto mb-6" />
+          <h2 className="text-xl font-semibold mb-2">
+            {state.status === "uploading" ? "Broadcasting to Postgres DB..."
+              : state.status === "embedding" ? "Running Core CPU Vector Space Projection..."
+                : `Executing Dual-Stage Neural Match for ${state.jdIds.length} JDs...`}
+          </h2>
+          <p className="text-[var(--color-dimmer)] text-sm">
+            Do not close this tab. The timer will automatically pause and redirect upon completion.
+          </p>
         </div>
       )}
     </div>