# SYSTEM ARCHITECTURE & OPERATIONAL MANUAL (`KNOWLEDGE.md`)

> **Target Audience:** AI Coding Agents, Automated CI/CD Pipelines, and Systems Engineers.
> **Scope:** Architecture, configuration standards, service management SOPs, and troubleshooting guides for the `Tom716/set-api` Hugging Face Space.

---

## 1. System Overview & Architecture

This repository hosts a multi-service backend environment on **Hugging Face Spaces (Free Tier)**. It utilizes a mono-repo structure where multiple isolated backend services run concurrently inside a single Docker container managed by **Supervisord** and routed externally via a **Caddy** reverse proxy.

### System Stack Summary
- **Infrastructure Provider:** Hugging Face Spaces (Docker SDK)
- **Container Base OS:** Ubuntu 22.04 LTS
- **Process Manager:** Supervisord (`supervisord`)
- **Reverse Proxy / Router:** Caddy Server v2
- **Public Ingress Port:** `7860` (Hard requirement by Hugging Face)
- **Container User Security:** Non-root execution under User ID `1000` (`user:user`)
- **Keep-Alive Engine:** External HTTP pinging (e.g., Better Stack) targeting `/status` every 5 minutes

```
                   Public Web Traffic (HTTPS Edge)
                                │
                                ▼
                 ┌─────────────────────────────┐
                 │  Hugging Face Edge Proxy   │
                 └──────────────┬──────────────┘
                                │ Port 7860
                                ▼
                 ┌─────────────────────────────┐
                 │     Caddy Reverse Proxy     │
                 └──────────────┬──────────────┘
                                │
        ┌───────────────────────┼───────────────────────┐
        │                       │                       │
        ▼                       ▼                       ▼
┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│ Caddy /status │       │ NestJS API    │       │ FastAPI App   │
│ Health Endpoint│      │ Telegram Demo │       │ K3s Test App  │
│ (Internal)    │       │ (Port 3000)   │       │ (Port 8000)   │
└───────────────┘       └───────────────┘       └───────────────┘
```

---

## 2. Directory Layout & Routing Matrix

### Directory Layout
```text
/ (Repository Root)
├── Dockerfile                  # Container build instructions & runtime setup
├── Caddyfile                   # Reverse proxy routing rules
├── supervisord.conf            # Multi-process supervisor configuration
├── KNOWLEDGE.md                # System documentation (This File)
└── projects/                   # Multi-service monorepo directory
    ├── telegram-send-demo/     # Service 1: NestJS + Telegram Bot Engine
    │   └── backend/            # NestJS API source code
    └── k3s-test-app/           # Service 2: Python FastAPI Service
        ├── pyproject.toml      # Python build metadata
        └── src/k3s_test_app/   # FastAPI source code
```

### Port Allocation & Ingress Matrix

| Service | Internal Port | Ingress Path | Health / Docs Path |
| :--- | :--- | :--- | :--- |
| **Caddy Proxy** | `7860` (Public) | `/` | `GET /status` |
| **Caddy Admin API** | `2019` (Local) | `localhost:2019` | N/A |
| **NestJS Telegram Service** | `3000` (Internal)| `/api/telegram/*` | `GET /api/telegram/health`, `/api/docs` |
| **Python FastAPI Service** | `8000` (Internal)| `/api/k3s/*` | `GET /api/k3s/health`, `/api/k3s/docs` |

---

## 3. Configuration File Standards

### 3.1 `Caddyfile` Standard
```caddy
{
    admin localhost:2019
    auto_https off
}

:7860 {
    # Keep-Alive Status Check
    handle /status {
        respond "Caddy Proxy Active" 200
    }

    # Service: Telegram NestJS Backend
    handle /api/telegram/* {
        reverse_proxy localhost:3000
    }

    # Service: Python FastAPI Backend (Strips prefix for internal app routing)
    handle /api/k3s/* {
        uri strip_prefix /api/k3s
        reverse_proxy localhost:8000
    }

    # Default Fallback (Routes to primary API service)
    handle {
        reverse_proxy localhost:3000
    }
}
```

### 3.2 `supervisord.conf` Standard
```ini
[supervisord]
nodaemon=true
user=user

[program:caddy]
command=caddy run --config /etc/caddy/Caddyfile --adapter caddyfile
autostart=true
autorestart=true
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0

[program:nestjs_backend]
command=node /app/projects/telegram-send-demo/backend/dist/main.js
directory=/app/projects/telegram-send-demo/backend
autostart=true
autorestart=true
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0

[program:k3s_backend]
command=/app/projects/k3s-test-app/.venv/bin/uvicorn k3s_test_app.main:app --host 127.0.0.1 --port 8000 --root-path /api/k3s
directory=/app/projects/k3s-test-app
autostart=true
autorestart=true
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0
```

---

## 4. Standard Operating Procedures (SOPs)

### 4.1 How to Add a New API Backend Service

When adding a new backend (e.g., `analytics-service` running on port `4000`):

1. **Upload Source:** Place the project files inside `/projects/analytics-service`.
2. **Update `Dockerfile`:** Add the build steps for the new runtime/dependencies:
   ```dockerfile
   COPY projects/analytics-service /app/projects/analytics-service
   RUN cd /app/projects/analytics-service && npm install && npm run build
   ```
3. **Update `supervisord.conf`:** Define a new supervisor process block:
   ```ini
   [program:analytics_service]
   command=node /app/projects/analytics-service/dist/main.js
   directory=/app/projects/analytics-service
   autostart=true
   autorestart=true
   stdout_logfile=/dev/stdout
   stdout_logfile_maxbytes=0
   stderr_logfile=/dev/stderr
   stderr_logfile_maxbytes=0
   ```
4. **Update `Caddyfile`:** Route a unique path prefix to the allocated internal port:
   ```caddy
   handle /api/analytics/* {
       reverse_proxy localhost:4000
   }
   ```
5. **Commit & Deploy:** Commit changes to `main`. Hugging Face will trigger a fresh container build automatically.

---

### 4.2 How to Remove an Existing Service

To decommission a service (e.g., `projects/k3s-test-app`):

1. **Update `Caddyfile`:** Remove the corresponding `handle /api/k3s/*` block.
2. **Update `supervisord.conf`:** Remove the `[program:k3s_backend]` block.
3. **Update `Dockerfile`:** Remove build lines associated with that project.
4. **Delete Directory:** Delete `/projects/k3s-test-app`.
5. **Commit Changes:** Push to `main`.

---

## 5. Critical Troubleshooting & System Rules

Any agent modifying this repository must follow these rules to avoid production breakages:

### 1. Node.js IPv6 DNS Resolution Fix (`ipv4first`)
- **Problem:** Node.js 20+ defaults to IPv6 DNS resolution. Inside Hugging Face Docker containers without outbound IPv6 routing, outbound HTTP requests (e.g., grammY hitting `api.telegram.org`) hang until timing out, returning a `503` error.
- **Mandatory Fix:** Always include the following at the top of Node.js entrypoints (`main.ts`):
  ```typescript
  import * as dns from 'dns';
  dns.setDefaultResultOrder('ipv4first');
  ```

### 2. FastAPI Behind Reverse Proxy Subpaths (`--root-path`)
- **Problem:** When FastAPI is proxied under a subpath like `/api/k3s/*`, Swagger UI fails to fetch OpenAPI specs (`/openapi.json` returns 404).
- **Mandatory Fix:** Always launch Uvicorn with `--root-path /api/<service_name>` or configure `root_path="/api/<service_name>"` inside `FastAPI()`.

### 3. NPM Lockfile Synchronization (`npm install` vs `npm ci`)
- **Problem:** `npm ci` fails if `package.json` and `package-lock.json` are slightly out of sync during container builds.
- **Mandatory Fix:** Use `npm install` inside the container build steps in `Dockerfile` to dynamically install missing transitive dependencies.

### 4. Base Image Python Versioning
- **Problem:** Ubuntu 22.04 LTS provides **Python 3.10.12**. Setting `requires-python = ">=3.14"` in `pyproject.toml` causes `pip` installation failures.
- **Mandatory Fix:** Ensure all Python projects set `requires-python = ">=3.10"`.

### 5. Non-Root File Permissions (`User 1000`)
- **Problem:** Hugging Face executes containers strictly as User ID `1000` (`user`). System services expecting root access fail to start.
- **Mandatory Fix:** Include `chown -R user:user /app /etc/caddy /var/log/supervisor /var/lib/caddy /var/log/caddy` in `Dockerfile` before switching to `USER user`.

### 6. Ephemeral Filesystem Limitation
- **Warning:** Storage in free Hugging Face Spaces is **ephemeral**. Any local files, database files (e.g., SQLite), or authentication keys (e.g., Baileys WhatsApp sessions) written to disk will be wiped when the container restarts.
- **Mandatory Requirement:** All persistent application state must be stored in external managed databases (e.g., Supabase, Neon PostgreSQL, or Redis).

### 7. Container Sleep Prevention
- **Mechanism:** Hugging Face Space containers go to sleep after 15–45 minutes of HTTP inactivity.
- **Mandatory Requirement:** An external monitoring service (Better Stack, UptimeRobot) must issue an HTTP `GET` ping to `https://tom716-set-api.hf.space/status` every **5 minutes**.