File size: 9,947 Bytes
74dc076 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 | # SYSTEM ARCHITECTURE & OPERATIONAL MANUAL (`KNOWLEDGE.md`)
> **Target Audience:** AI Coding Agents, Automated CI/CD Pipelines, and Systems Engineers.
> **Scope:** Architecture, configuration standards, service management SOPs, and troubleshooting guides for the `Tom716/set-api` Hugging Face Space.
---
## 1. System Overview & Architecture
This repository hosts a multi-service backend environment on **Hugging Face Spaces (Free Tier)**. It utilizes a mono-repo structure where multiple isolated backend services run concurrently inside a single Docker container managed by **Supervisord** and routed externally via a **Caddy** reverse proxy.
### System Stack Summary
- **Infrastructure Provider:** Hugging Face Spaces (Docker SDK)
- **Container Base OS:** Ubuntu 22.04 LTS
- **Process Manager:** Supervisord (`supervisord`)
- **Reverse Proxy / Router:** Caddy Server v2
- **Public Ingress Port:** `7860` (Hard requirement by Hugging Face)
- **Container User Security:** Non-root execution under User ID `1000` (`user:user`)
- **Keep-Alive Engine:** External HTTP pinging (e.g., Better Stack) targeting `/status` every 5 minutes
```
Public Web Traffic (HTTPS Edge)
│
▼
┌─────────────────────────────┐
│ Hugging Face Edge Proxy │
└──────────────┬──────────────┘
│ Port 7860
▼
┌─────────────────────────────┐
│ Caddy Reverse Proxy │
└──────────────┬──────────────┘
│
┌───────────────────────┼───────────────────────┐
│ │ │
▼ ▼ ▼
┌───────────────┐ ┌───────────────┐ ┌───────────────┐
│ Caddy /status │ │ NestJS API │ │ FastAPI App │
│ Health Endpoint│ │ Telegram Demo │ │ K3s Test App │
│ (Internal) │ │ (Port 3000) │ │ (Port 8000) │
└───────────────┘ └───────────────┘ └───────────────┘
```
---
## 2. Directory Layout & Routing Matrix
### Directory Layout
```text
/ (Repository Root)
├── Dockerfile # Container build instructions & runtime setup
├── Caddyfile # Reverse proxy routing rules
├── supervisord.conf # Multi-process supervisor configuration
├── KNOWLEDGE.md # System documentation (This File)
└── projects/ # Multi-service monorepo directory
├── telegram-send-demo/ # Service 1: NestJS + Telegram Bot Engine
│ └── backend/ # NestJS API source code
└── k3s-test-app/ # Service 2: Python FastAPI Service
├── pyproject.toml # Python build metadata
└── src/k3s_test_app/ # FastAPI source code
```
### Port Allocation & Ingress Matrix
| Service | Internal Port | Ingress Path | Health / Docs Path |
| :--- | :--- | :--- | :--- |
| **Caddy Proxy** | `7860` (Public) | `/` | `GET /status` |
| **Caddy Admin API** | `2019` (Local) | `localhost:2019` | N/A |
| **NestJS Telegram Service** | `3000` (Internal)| `/api/telegram/*` | `GET /api/telegram/health`, `/api/docs` |
| **Python FastAPI Service** | `8000` (Internal)| `/api/k3s/*` | `GET /api/k3s/health`, `/api/k3s/docs` |
---
## 3. Configuration File Standards
### 3.1 `Caddyfile` Standard
```caddy
{
admin localhost:2019
auto_https off
}
:7860 {
# Keep-Alive Status Check
handle /status {
respond "Caddy Proxy Active" 200
}
# Service: Telegram NestJS Backend
handle /api/telegram/* {
reverse_proxy localhost:3000
}
# Service: Python FastAPI Backend (Strips prefix for internal app routing)
handle /api/k3s/* {
uri strip_prefix /api/k3s
reverse_proxy localhost:8000
}
# Default Fallback (Routes to primary API service)
handle {
reverse_proxy localhost:3000
}
}
```
### 3.2 `supervisord.conf` Standard
```ini
[supervisord]
nodaemon=true
user=user
[program:caddy]
command=caddy run --config /etc/caddy/Caddyfile --adapter caddyfile
autostart=true
autorestart=true
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0
[program:nestjs_backend]
command=node /app/projects/telegram-send-demo/backend/dist/main.js
directory=/app/projects/telegram-send-demo/backend
autostart=true
autorestart=true
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0
[program:k3s_backend]
command=/app/projects/k3s-test-app/.venv/bin/uvicorn k3s_test_app.main:app --host 127.0.0.1 --port 8000 --root-path /api/k3s
directory=/app/projects/k3s-test-app
autostart=true
autorestart=true
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0
```
---
## 4. Standard Operating Procedures (SOPs)
### 4.1 How to Add a New API Backend Service
When adding a new backend (e.g., `analytics-service` running on port `4000`):
1. **Upload Source:** Place the project files inside `/projects/analytics-service`.
2. **Update `Dockerfile`:** Add the build steps for the new runtime/dependencies:
```dockerfile
COPY projects/analytics-service /app/projects/analytics-service
RUN cd /app/projects/analytics-service && npm install && npm run build
```
3. **Update `supervisord.conf`:** Define a new supervisor process block:
```ini
[program:analytics_service]
command=node /app/projects/analytics-service/dist/main.js
directory=/app/projects/analytics-service
autostart=true
autorestart=true
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0
```
4. **Update `Caddyfile`:** Route a unique path prefix to the allocated internal port:
```caddy
handle /api/analytics/* {
reverse_proxy localhost:4000
}
```
5. **Commit & Deploy:** Commit changes to `main`. Hugging Face will trigger a fresh container build automatically.
---
### 4.2 How to Remove an Existing Service
To decommission a service (e.g., `projects/k3s-test-app`):
1. **Update `Caddyfile`:** Remove the corresponding `handle /api/k3s/*` block.
2. **Update `supervisord.conf`:** Remove the `[program:k3s_backend]` block.
3. **Update `Dockerfile`:** Remove build lines associated with that project.
4. **Delete Directory:** Delete `/projects/k3s-test-app`.
5. **Commit Changes:** Push to `main`.
---
## 5. Critical Troubleshooting & System Rules
Any agent modifying this repository must follow these rules to avoid production breakages:
### 1. Node.js IPv6 DNS Resolution Fix (`ipv4first`)
- **Problem:** Node.js 20+ defaults to IPv6 DNS resolution. Inside Hugging Face Docker containers without outbound IPv6 routing, outbound HTTP requests (e.g., grammY hitting `api.telegram.org`) hang until timing out, returning a `503` error.
- **Mandatory Fix:** Always include the following at the top of Node.js entrypoints (`main.ts`):
```typescript
import * as dns from 'dns';
dns.setDefaultResultOrder('ipv4first');
```
### 2. FastAPI Behind Reverse Proxy Subpaths (`--root-path`)
- **Problem:** When FastAPI is proxied under a subpath like `/api/k3s/*`, Swagger UI fails to fetch OpenAPI specs (`/openapi.json` returns 404).
- **Mandatory Fix:** Always launch Uvicorn with `--root-path /api/<service_name>` or configure `root_path="/api/<service_name>"` inside `FastAPI()`.
### 3. NPM Lockfile Synchronization (`npm install` vs `npm ci`)
- **Problem:** `npm ci` fails if `package.json` and `package-lock.json` are slightly out of sync during container builds.
- **Mandatory Fix:** Use `npm install` inside the container build steps in `Dockerfile` to dynamically install missing transitive dependencies.
### 4. Base Image Python Versioning
- **Problem:** Ubuntu 22.04 LTS provides **Python 3.10.12**. Setting `requires-python = ">=3.14"` in `pyproject.toml` causes `pip` installation failures.
- **Mandatory Fix:** Ensure all Python projects set `requires-python = ">=3.10"`.
### 5. Non-Root File Permissions (`User 1000`)
- **Problem:** Hugging Face executes containers strictly as User ID `1000` (`user`). System services expecting root access fail to start.
- **Mandatory Fix:** Include `chown -R user:user /app /etc/caddy /var/log/supervisor /var/lib/caddy /var/log/caddy` in `Dockerfile` before switching to `USER user`.
### 6. Ephemeral Filesystem Limitation
- **Warning:** Storage in free Hugging Face Spaces is **ephemeral**. Any local files, database files (e.g., SQLite), or authentication keys (e.g., Baileys WhatsApp sessions) written to disk will be wiped when the container restarts.
- **Mandatory Requirement:** All persistent application state must be stored in external managed databases (e.g., Supabase, Neon PostgreSQL, or Redis).
### 7. Container Sleep Prevention
- **Mechanism:** Hugging Face Space containers go to sleep after 15–45 minutes of HTTP inactivity.
- **Mandatory Requirement:** An external monitoring service (Better Stack, UptimeRobot) must issue an HTTP `GET` ping to `https://tom716-set-api.hf.space/status` every **5 minutes**. |