scripts/prd.md

Product Requirements Document (PRD)

Project: NCOS_S1 (Large Compliance LLM Pipeline)

1. Project Overview

Deploy a large compliance LLM (ACATECH/ncos, Llama-2-70B) on Hugging Face Spaces, with a Next.js frontend (Vercel), Supabase for test cases, and Redis for queueing. The backend is a FastAPI app running in a Docker container for full control (CUDA, dependencies, etc.).

---

2. Current State Analysis

• Backend:
  • FastAPI app in Hugging Face Space, Dockerized.
  • CUDA and torch set up for GPU inference.
  • Permissions and cache issues resolved.
  • Requirements are mostly correct and reproducible.
• Frontend:
  • Next.js app on Vercel (not tightly integrated yet).
• Test/Queue:
  • Supabase for test cases.
  • Redis for queueing (not fully integrated).
• Issues:
  • Dependency hell (CUDA, torch, flash-attn, numpy, etc.).
  • File permission and cache issues.
  • Model/tokenizer loading errors (corrupt/incompatible files).
  • Manual syncing of requirements and Dockerfile.
  • No robust, end-to-end pipeline from test case → queue → model → result → storage.
  • No clear API contract between frontend, backend, and test/queue system.
  • No health checks, monitoring, or error reporting.
  • No automated deployment or CI/CD for the Space.
  • Monolithic codebase, hard to debug.

---

3. Goals

• Modular, robust, and reproducible pipeline for LLM compliance testing.
• Clean separation of backend, frontend, and queue/storage.
• Automated, reliable deployment and monitoring.
• Clear API contract and documentation.

---

4. Recommended Architecture

A. Modular Structure

• Backend (Hugging Face Space):
  • FastAPI app, Dockerized, REST API for inference.
  • Handles model loading, inference, health checks.
  • Connects to Redis for job queueing.
  • Optionally connects to Supabase for test/result storage.
• Frontend (Vercel/Next.js):
  • Calls backend API for inference.
  • Displays results, test case status, health info.
• Queue/Storage:
  • Redis for job queueing (decouples frontend/backend).
  • Supabase for storing test cases/results.

B. Key Features

• Robust error handling and logging in backend.
• Health check endpoints (/healthz, /readyz).
• Clear API contract (OpenAPI/Swagger for FastAPI).
• Automated Docker build and deployment (version pinning).
• CI/CD pipeline for backend and frontend.
• Documentation for setup, usage, troubleshooting.

---

5. Action Plan

Step 1: Design the API Contract

• Define endpoints for:
  • /infer (POST): Accepts input, returns model output.
  • /healthz (GET): Returns service health.
  • /queue (POST/GET): For job submission/status (if using Redis).
• Use FastAPI's OpenAPI docs for clarity.

Step 2: Clean Backend Implementation

• Start a new repo or clean branch.
• Write a minimal FastAPI app:
  • Loads model/tokenizer (with robust error handling).
  • Exposes /infer and /healthz.
  • Logs errors and requests.
• Add Redis integration for queueing (optional, but recommended for scale).
• Add Supabase integration for test/result storage (optional, can be added after core works).

Step 3: Dockerize the Backend

• Use a clean, minimal Dockerfile:
  • Start from nvidia/cuda:12.1.0-devel-ubuntu22.04.
  • Install Python, torch, dependencies in correct order.
  • Set up cache and permissions.
  • Pin all versions in requirements.txt.
  • Add a health check in Dockerfile (HEALTHCHECK).

Step 4: Model/Tokenizer Management

• Ensure model/tokenizer files are valid and compatible.
• Test loading locally before pushing to Hugging Face.
• Document the process for updating model files.

Step 5: Frontend Integration

• Update Next.js frontend to call the new backend API.
• Show job status, results, and health info.
• Add error handling and user feedback.

Step 6: Queue and Storage Integration

• Set up Redis for job queueing.
• Set up Supabase for test case/result storage.
• Ensure backend can pull jobs from Redis, process, and store results in Supabase.

Step 7: Monitoring and Health

• Add logging and error reporting (e.g., to stdout, or a logging service).
• Implement /healthz and /readyz endpoints.
• Optionally, add Prometheus/Grafana metrics.

Step 8: CI/CD and Documentation

• Add GitHub Actions or similar for automated build/test/deploy.
• Write clear README and API docs.

---

6. Success Criteria

• End-to-end pipeline works: test case → queue → model → result → storage.
• Robust error handling and health checks in place.
• Automated, reproducible builds and deployments.
• Clear, up-to-date documentation for all components.