system_knowledge_base.md

SYSTEM ARCHITECTURE & OPERATIONAL MANUAL (KNOWLEDGE.md)

Target Audience: AI Coding Agents, Automated CI/CD Pipelines, and Systems Engineers. Scope: Architecture, configuration standards, service management SOPs, and troubleshooting guides for the Tom716/set-api Hugging Face Space.

---

1. System Overview & Architecture

This repository hosts a multi-service backend environment on Hugging Face Spaces (Free Tier). It utilizes a mono-repo structure where multiple isolated backend services run concurrently inside a single Docker container managed by Supervisord and routed externally via a Caddy reverse proxy.

System Stack Summary

• Infrastructure Provider: Hugging Face Spaces (Docker SDK)
• Container Base OS: Ubuntu 22.04 LTS
• Process Manager: Supervisord (supervisord)
• Reverse Proxy / Router: Caddy Server v2
• Public Ingress Port: 7860 (Hard requirement by Hugging Face)
• Container User Security: Non-root execution under User ID 1000 (user:user)
• Keep-Alive Engine: External HTTP pinging (e.g., Better Stack) targeting /status every 5 minutes

                   Public Web Traffic (HTTPS Edge)
                                │
                                ▼
                 ┌─────────────────────────────┐
                 │  Hugging Face Edge Proxy   │
                 └──────────────┬──────────────┘
                                │ Port 7860
                                ▼
                 ┌─────────────────────────────┐
                 │     Caddy Reverse Proxy     │
                 └──────────────┬──────────────┘
                                │
        ┌───────────────────────┼───────────────────────┐
        │                       │                       │
        ▼                       ▼                       ▼
┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│ Caddy /status │       │ NestJS API    │       │ FastAPI App   │
│ Health Endpoint│      │ Telegram Demo │       │ K3s Test App  │
│ (Internal)    │       │ (Port 3000)   │       │ (Port 8000)   │
└───────────────┘       └───────────────┘       └───────────────┘

---

2. Directory Layout & Routing Matrix

Directory Layout

/ (Repository Root)
├── Dockerfile                  # Container build instructions & runtime setup
├── Caddyfile                   # Reverse proxy routing rules
├── supervisord.conf            # Multi-process supervisor configuration
├── KNOWLEDGE.md                # System documentation (This File)
└── projects/                   # Multi-service monorepo directory
    ├── telegram-send-demo/     # Service 1: NestJS + Telegram Bot Engine
    │   └── backend/            # NestJS API source code
    └── k3s-test-app/           # Service 2: Python FastAPI Service
        ├── pyproject.toml      # Python build metadata
        └── src/k3s_test_app/   # FastAPI source code

Port Allocation & Ingress Matrix

Service	Internal Port	Ingress Path	Health / Docs Path
Caddy Proxy	7860 (Public)	/	GET /status
Caddy Admin API	2019 (Local)	localhost:2019	N/A
NestJS Telegram Service	3000 (Internal)	/api/telegram/*	GET /api/telegram/health, /api/docs
Python FastAPI Service	8000 (Internal)	/api/k3s/*	GET /api/k3s/health, /api/k3s/docs

---

3. Configuration File Standards

3.1 Caddyfile Standard

{
    admin localhost:2019
    auto_https off
}

:7860 {
    # Keep-Alive Status Check
    handle /status {
        respond "Caddy Proxy Active" 200
    }

    # Service: Telegram NestJS Backend
    handle /api/telegram/* {
        reverse_proxy localhost:3000
    }

    # Service: Python FastAPI Backend (Strips prefix for internal app routing)
    handle /api/k3s/* {
        uri strip_prefix /api/k3s
        reverse_proxy localhost:8000
    }

    # Default Fallback (Routes to primary API service)
    handle {
        reverse_proxy localhost:3000
    }
}

3.2 supervisord.conf Standard

[supervisord]
nodaemon=true
user=user

[program:caddy]
command=caddy run --config /etc/caddy/Caddyfile --adapter caddyfile
autostart=true
autorestart=true
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0

[program:nestjs_backend]
command=node /app/projects/telegram-send-demo/backend/dist/main.js
directory=/app/projects/telegram-send-demo/backend
autostart=true
autorestart=true
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0

[program:k3s_backend]
command=/app/projects/k3s-test-app/.venv/bin/uvicorn k3s_test_app.main:app --host 127.0.0.1 --port 8000 --root-path /api/k3s
directory=/app/projects/k3s-test-app
autostart=true
autorestart=true
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0

---

4. Standard Operating Procedures (SOPs)

4.1 How to Add a New API Backend Service

When adding a new backend (e.g., analytics-service running on port 4000):

1. Upload Source: Place the project files inside /projects/analytics-service.
2. Update Dockerfile: Add the build steps for the new runtime/dependencies:

   COPY projects/analytics-service /app/projects/analytics-service
   RUN cd /app/projects/analytics-service && npm install && npm run build
   

3. Update supervisord.conf: Define a new supervisor process block:

   [program:analytics_service]
   command=node /app/projects/analytics-service/dist/main.js
   directory=/app/projects/analytics-service
   autostart=true
   autorestart=true
   stdout_logfile=/dev/stdout
   stdout_logfile_maxbytes=0
   stderr_logfile=/dev/stderr
   stderr_logfile_maxbytes=0
   

4. Update Caddyfile: Route a unique path prefix to the allocated internal port:

   handle /api/analytics/* {
       reverse_proxy localhost:4000
   }
   

5. Commit & Deploy: Commit changes to main. Hugging Face will trigger a fresh container build automatically.

---

4.2 How to Remove an Existing Service

To decommission a service (e.g., projects/k3s-test-app):

1. Update Caddyfile: Remove the corresponding handle /api/k3s/* block.
2. Update supervisord.conf: Remove the [program:k3s_backend] block.
3. Update Dockerfile: Remove build lines associated with that project.
4. Delete Directory: Delete /projects/k3s-test-app.
5. Commit Changes: Push to main.

---

5. Critical Troubleshooting & System Rules

Any agent modifying this repository must follow these rules to avoid production breakages:

1. Node.js IPv6 DNS Resolution Fix (ipv4first)

• Problem: Node.js 20+ defaults to IPv6 DNS resolution. Inside Hugging Face Docker containers without outbound IPv6 routing, outbound HTTP requests (e.g., grammY hitting api.telegram.org) hang until timing out, returning a 503 error.
• Mandatory Fix: Always include the following at the top of Node.js entrypoints (main.ts):

  import * as dns from 'dns';
  dns.setDefaultResultOrder('ipv4first');
  

2. FastAPI Behind Reverse Proxy Subpaths (--root-path)

• Problem: When FastAPI is proxied under a subpath like /api/k3s/*, Swagger UI fails to fetch OpenAPI specs (/openapi.json returns 404).
• Mandatory Fix: Always launch Uvicorn with --root-path /api/<service_name> or configure root_path="/api/<service_name>" inside FastAPI().

3. NPM Lockfile Synchronization (npm install vs npm ci)

• Problem: npm ci fails if package.json and package-lock.json are slightly out of sync during container builds.
• Mandatory Fix: Use npm install inside the container build steps in Dockerfile to dynamically install missing transitive dependencies.

4. Base Image Python Versioning

• Problem: Ubuntu 22.04 LTS provides Python 3.10.12. Setting requires-python = ">=3.14" in pyproject.toml causes pip installation failures.
• Mandatory Fix: Ensure all Python projects set requires-python = ">=3.10".

5. Non-Root File Permissions (User 1000)

• Problem: Hugging Face executes containers strictly as User ID 1000 (user). System services expecting root access fail to start.
• Mandatory Fix: Include chown -R user:user /app /etc/caddy /var/log/supervisor /var/lib/caddy /var/log/caddy in Dockerfile before switching to USER user.

6. Ephemeral Filesystem Limitation

• Warning: Storage in free Hugging Face Spaces is ephemeral. Any local files, database files (e.g., SQLite), or authentication keys (e.g., Baileys WhatsApp sessions) written to disk will be wiped when the container restarts.
• Mandatory Requirement: All persistent application state must be stored in external managed databases (e.g., Supabase, Neon PostgreSQL, or Redis).

7. Container Sleep Prevention

• Mechanism: Hugging Face Space containers go to sleep after 15–45 minutes of HTTP inactivity.
• Mandatory Requirement: An external monitoring service (Better Stack, UptimeRobot) must issue an HTTP GET ping to https://tom716-set-api.hf.space/status every 5 minutes.