| # SYSTEM ARCHITECTURE & OPERATIONAL MANUAL (`KNOWLEDGE.md`) |
|
|
| > **Target Audience:** AI Coding Agents, Automated CI/CD Pipelines, and Systems Engineers. |
| > **Scope:** Architecture, configuration standards, service management SOPs, and troubleshooting guides for the `Tom716/set-api` Hugging Face Space. |
|
|
| --- |
|
|
| ## 1. System Overview & Architecture |
|
|
| This repository hosts a multi-service backend environment on **Hugging Face Spaces (Free Tier)**. It utilizes a mono-repo structure where multiple isolated backend services run concurrently inside a single Docker container managed by **Supervisord** and routed externally via a **Caddy** reverse proxy. |
|
|
| ### System Stack Summary |
| - **Infrastructure Provider:** Hugging Face Spaces (Docker SDK) |
| - **Container Base OS:** Ubuntu 22.04 LTS |
| - **Process Manager:** Supervisord (`supervisord`) |
| - **Reverse Proxy / Router:** Caddy Server v2 |
| - **Public Ingress Port:** `7860` (Hard requirement by Hugging Face) |
| - **Container User Security:** Non-root execution under User ID `1000` (`user:user`) |
| - **Keep-Alive Engine:** External HTTP pinging (e.g., Better Stack) targeting `/status` every 5 minutes |
|
|
| ``` |
| Public Web Traffic (HTTPS Edge) |
| │ |
| ▼ |
| ┌─────────────────────────────┐ |
| │ Hugging Face Edge Proxy │ |
| └──────────────┬──────────────┘ |
| │ Port 7860 |
| ▼ |
| ┌─────────────────────────────┐ |
| │ Caddy Reverse Proxy │ |
| └──────────────┬──────────────┘ |
| │ |
| ┌───────────────────────┼───────────────────────┐ |
| │ │ │ |
| ▼ ▼ ▼ |
| ┌───────────────┐ ┌───────────────┐ ┌───────────────┐ |
| │ Caddy /status │ │ NestJS API │ │ FastAPI App │ |
| │ Health Endpoint│ │ Telegram Demo │ │ K3s Test App │ |
| │ (Internal) │ │ (Port 3000) │ │ (Port 8000) │ |
| └───────────────┘ └───────────────┘ └───────────────┘ |
| ``` |
|
|
| --- |
|
|
| ## 2. Directory Layout & Routing Matrix |
|
|
| ### Directory Layout |
| ```text |
| / (Repository Root) |
| ├── Dockerfile # Container build instructions & runtime setup |
| ├── Caddyfile # Reverse proxy routing rules |
| ├── supervisord.conf # Multi-process supervisor configuration |
| ├── KNOWLEDGE.md # System documentation (This File) |
| └── projects/ # Multi-service monorepo directory |
| ├── telegram-send-demo/ # Service 1: NestJS + Telegram Bot Engine |
| │ └── backend/ # NestJS API source code |
| └── k3s-test-app/ # Service 2: Python FastAPI Service |
| ├── pyproject.toml # Python build metadata |
| └── src/k3s_test_app/ # FastAPI source code |
| ``` |
|
|
| ### Port Allocation & Ingress Matrix |
|
|
| | Service | Internal Port | Ingress Path | Health / Docs Path | |
| | :--- | :--- | :--- | :--- | |
| | **Caddy Proxy** | `7860` (Public) | `/` | `GET /status` | |
| | **Caddy Admin API** | `2019` (Local) | `localhost:2019` | N/A | |
| | **NestJS Telegram Service** | `3000` (Internal)| `/api/telegram/*` | `GET /api/telegram/health`, `/api/docs` | |
| | **Python FastAPI Service** | `8000` (Internal)| `/api/k3s/*` | `GET /api/k3s/health`, `/api/k3s/docs` | |
|
|
| --- |
|
|
| ## 3. Configuration File Standards |
|
|
| ### 3.1 `Caddyfile` Standard |
| ```caddy |
| { |
| admin localhost:2019 |
| auto_https off |
| } |
| |
| :7860 { |
| # Keep-Alive Status Check |
| handle /status { |
| respond "Caddy Proxy Active" 200 |
| } |
| |
| # Service: Telegram NestJS Backend |
| handle /api/telegram/* { |
| reverse_proxy localhost:3000 |
| } |
| |
| # Service: Python FastAPI Backend (Strips prefix for internal app routing) |
| handle /api/k3s/* { |
| uri strip_prefix /api/k3s |
| reverse_proxy localhost:8000 |
| } |
| |
| # Default Fallback (Routes to primary API service) |
| handle { |
| reverse_proxy localhost:3000 |
| } |
| } |
| ``` |
|
|
| ### 3.2 `supervisord.conf` Standard |
| ```ini |
| [supervisord] |
| nodaemon=true |
| user=user |
| |
| [program:caddy] |
| command=caddy run --config /etc/caddy/Caddyfile --adapter caddyfile |
| autostart=true |
| autorestart=true |
| stdout_logfile=/dev/stdout |
| stdout_logfile_maxbytes=0 |
| stderr_logfile=/dev/stderr |
| stderr_logfile_maxbytes=0 |
| |
| [program:nestjs_backend] |
| command=node /app/projects/telegram-send-demo/backend/dist/main.js |
| directory=/app/projects/telegram-send-demo/backend |
| autostart=true |
| autorestart=true |
| stdout_logfile=/dev/stdout |
| stdout_logfile_maxbytes=0 |
| stderr_logfile=/dev/stderr |
| stderr_logfile_maxbytes=0 |
| |
| [program:k3s_backend] |
| command=/app/projects/k3s-test-app/.venv/bin/uvicorn k3s_test_app.main:app --host 127.0.0.1 --port 8000 --root-path /api/k3s |
| directory=/app/projects/k3s-test-app |
| autostart=true |
| autorestart=true |
| stdout_logfile=/dev/stdout |
| stdout_logfile_maxbytes=0 |
| stderr_logfile=/dev/stderr |
| stderr_logfile_maxbytes=0 |
| ``` |
|
|
| --- |
|
|
| ## 4. Standard Operating Procedures (SOPs) |
|
|
| ### 4.1 How to Add a New API Backend Service |
|
|
| When adding a new backend (e.g., `analytics-service` running on port `4000`): |
|
|
| 1. **Upload Source:** Place the project files inside `/projects/analytics-service`. |
| 2. **Update `Dockerfile`:** Add the build steps for the new runtime/dependencies: |
| ```dockerfile |
| COPY projects/analytics-service /app/projects/analytics-service |
| RUN cd /app/projects/analytics-service && npm install && npm run build |
| ``` |
| 3. **Update `supervisord.conf`:** Define a new supervisor process block: |
| ```ini |
| [program:analytics_service] |
| command=node /app/projects/analytics-service/dist/main.js |
| directory=/app/projects/analytics-service |
| autostart=true |
| autorestart=true |
| stdout_logfile=/dev/stdout |
| stdout_logfile_maxbytes=0 |
| stderr_logfile=/dev/stderr |
| stderr_logfile_maxbytes=0 |
| ``` |
| 4. **Update `Caddyfile`:** Route a unique path prefix to the allocated internal port: |
| ```caddy |
| handle /api/analytics/* { |
| reverse_proxy localhost:4000 |
| } |
| ``` |
| 5. **Commit & Deploy:** Commit changes to `main`. Hugging Face will trigger a fresh container build automatically. |
|
|
| --- |
|
|
| ### 4.2 How to Remove an Existing Service |
|
|
| To decommission a service (e.g., `projects/k3s-test-app`): |
|
|
| 1. **Update `Caddyfile`:** Remove the corresponding `handle /api/k3s/*` block. |
| 2. **Update `supervisord.conf`:** Remove the `[program:k3s_backend]` block. |
| 3. **Update `Dockerfile`:** Remove build lines associated with that project. |
| 4. **Delete Directory:** Delete `/projects/k3s-test-app`. |
| 5. **Commit Changes:** Push to `main`. |
|
|
| --- |
|
|
| ## 5. Critical Troubleshooting & System Rules |
|
|
| Any agent modifying this repository must follow these rules to avoid production breakages: |
|
|
| ### 1. Node.js IPv6 DNS Resolution Fix (`ipv4first`) |
| - **Problem:** Node.js 20+ defaults to IPv6 DNS resolution. Inside Hugging Face Docker containers without outbound IPv6 routing, outbound HTTP requests (e.g., grammY hitting `api.telegram.org`) hang until timing out, returning a `503` error. |
| - **Mandatory Fix:** Always include the following at the top of Node.js entrypoints (`main.ts`): |
| ```typescript |
| import * as dns from 'dns'; |
| dns.setDefaultResultOrder('ipv4first'); |
| ``` |
|
|
| ### 2. FastAPI Behind Reverse Proxy Subpaths (`--root-path`) |
| - **Problem:** When FastAPI is proxied under a subpath like `/api/k3s/*`, Swagger UI fails to fetch OpenAPI specs (`/openapi.json` returns 404). |
| - **Mandatory Fix:** Always launch Uvicorn with `--root-path /api/<service_name>` or configure `root_path="/api/<service_name>"` inside `FastAPI()`. |
|
|
| ### 3. NPM Lockfile Synchronization (`npm install` vs `npm ci`) |
| - **Problem:** `npm ci` fails if `package.json` and `package-lock.json` are slightly out of sync during container builds. |
| - **Mandatory Fix:** Use `npm install` inside the container build steps in `Dockerfile` to dynamically install missing transitive dependencies. |
|
|
| ### 4. Base Image Python Versioning |
| - **Problem:** Ubuntu 22.04 LTS provides **Python 3.10.12**. Setting `requires-python = ">=3.14"` in `pyproject.toml` causes `pip` installation failures. |
| - **Mandatory Fix:** Ensure all Python projects set `requires-python = ">=3.10"`. |
|
|
| ### 5. Non-Root File Permissions (`User 1000`) |
| - **Problem:** Hugging Face executes containers strictly as User ID `1000` (`user`). System services expecting root access fail to start. |
| - **Mandatory Fix:** Include `chown -R user:user /app /etc/caddy /var/log/supervisor /var/lib/caddy /var/log/caddy` in `Dockerfile` before switching to `USER user`. |
|
|
| ### 6. Ephemeral Filesystem Limitation |
| - **Warning:** Storage in free Hugging Face Spaces is **ephemeral**. Any local files, database files (e.g., SQLite), or authentication keys (e.g., Baileys WhatsApp sessions) written to disk will be wiped when the container restarts. |
| - **Mandatory Requirement:** All persistent application state must be stored in external managed databases (e.g., Supabase, Neon PostgreSQL, or Redis). |
|
|
| ### 7. Container Sleep Prevention |
| - **Mechanism:** Hugging Face Space containers go to sleep after 15–45 minutes of HTTP inactivity. |
| - **Mandatory Requirement:** An external monitoring service (Better Stack, UptimeRobot) must issue an HTTP `GET` ping to `https://tom716-set-api.hf.space/status` every **5 minutes**. |
