Spaces:
Paused
Paused
v1.1.0: CPU-safe GPU detection, real GPU offload, hardened fallible calls, doctor command
dc03fa5 verified | title: THOX Mesh Node | |
| emoji: 🕸️ | |
| colorFrom: indigo | |
| colorTo: purple | |
| sdk: docker | |
| app_port: 7860 | |
| pinned: false | |
| license: apache-2.0 | |
| short_description: OpenAI-compatible THOX node in the MeshStack mesh | |
| # THOX Mesh Node — HF CPU Space | |
| A free-tier CPU node in the THOX mesh. It serves an OpenAI-compatible endpoint | |
| from a THOX GGUF and self-registers so ThoxRoute can route traffic to it. | |
| ## Endpoints | |
| | Path | Purpose | | |
| | --- | --- | | |
| | `POST /v1/chat/completions` | OpenAI-compatible chat completion | | |
| | `GET /v1/models` | Advertised model | | |
| | `GET /health` | Liveness, mesh registration state, live telemetry | | |
| ## Quick check | |
| ```bash | |
| curl -s https://<space-host>/health | jq . | |
| curl -s https://<space-host>/v1/chat/completions \ | |
| -H 'content-type: application/json' \ | |
| -d '{"messages":[{"role":"user","content":"Say hello from the THOX mesh."}],"max_tokens":48}' | |
| ``` | |
| ## Configuration | |
| Set these as Space **Variables**, except `HF_TOKEN` which must be a **Secret**. | |
| | Variable | Default | Meaning | | |
| | --- | --- | --- | | |
| | `THOX_MODEL_REPO` | `Thox-ai/ThoxMini-3B` | Hub repo holding the GGUF | | |
| | `THOX_MODEL_FILE` | `thoxmini-3b-Q4_K_M.gguf` | GGUF filename | | |
| | `THOX_MODEL_ID` | `thoxmini-3b` | Model id advertised to the mesh | | |
| | `THOX_ALIAS` | `thoxmini-3b-space` | Endpoint alias, unique per device | | |
| | `THOX_CAPABILITIES` | `chat,completion` | Capability flags ThoxRoute matches on | | |
| | `THOX_CONTROLLER_URL` | *(unset)* | Mesh controller to register with | | |
| | `THOX_MESH_DEVICE_ID` | *(unset)* | Paired MeshStack device UUID | | |
| | `THOX_MESH_DEVICE_KEY` | *(unset)* | **Secret.** Base64 Ed25519 seed | | |
| | `THOX_N_GPU_LAYERS` | `auto` | `auto` \| `0` (force CPU) \| `-1` (all) \| an integer | | |
| | `HF_TOKEN` | *(unset)* | **Secret.** Required — the model repo is private | | |
| ## CPU and GPU paths | |
| The same agent runs on both. Nothing needs reconfiguring to move between them — | |
| `THOX_N_GPU_LAYERS=auto` resolves per machine. | |
| | | CPU (this Space, free Colab) | GPU (Colab T4/L4/A100) | | |
| | --- | --- | --- | | |
| | Model | ThoxMini-3B Q4_K_M (~2 GB) | ThoxMythos-9B Q4_K_M (~6.2 GB) | | |
| | Offload | `n_gpu_layers=0` | `n_gpu_layers=-1` (all, clamped to VRAM) | | |
| | llama.cpp build | CPU wheel/build | rebuilt with `-DGGML_CUDA=ON` | | |
| | Throughput | ~1.5 tok/s | roughly an order of magnitude faster | | |
| **A CPU-only environment is fully supported, not a degraded one.** GPU detection | |
| goes through `thoxmesh_node.gpu`, which treats a missing `nvidia-smi`, a broken | |
| driver and a hung probe all as "no GPU" and never raises. If a CUDA build is | |
| asked to offload and cannot, the node retries CPU-only rather than refusing to | |
| serve — a slower mesh member beats an absent one. | |
| Check any machine before starting: | |
| ```bash | |
| python -m thoxmesh_node doctor | |
| ``` | |
| It reports Python, the crypto/web stack, llama-cpp-python, `HF_TOKEN`, GPU state, | |
| cloudflared and the resolved config. Exit 0 means the node can serve. | |
| ## Why this Space registers with the controller, not MeshStack v2 directly | |
| `meshstack-device-v2` validates every `base_url` against `isPrivateHost` and | |
| accepts only RFC1918, loopback, link-local, CGNAT and `.local` addresses. An | |
| `*.hf.space` origin is public and is refused by that rule — deliberately, as an | |
| anti-SSRF control on the mesh. | |
| The node detects that specific refusal and falls back to the controller | |
| transport rather than retrying a request that can never succeed. See | |
| `ARCHITECTURE.md` for the contract and the proposed resolution. | |
| ## Cost | |
| Runs on the free `cpu-basic` tier and must stay there. ThoxMini-3B Q4_K_M is | |
| ~2 GB and fits the 16 GB / 2 vCPU allocation; nothing here requests a GPU. | |
| Expect roughly **1.5 tokens/second**. That is the honest free-CPU number, and it | |
| is why `THOX_MAX_TOKENS_CAP` defaults to 256 — the node reports its real latency | |
| as telemetry, so ThoxRoute ranks it behind faster nodes rather than being | |
| surprised by it. | |
| `ThoxLLM-327M-v2` was tried first because it is far smaller. It loads and serves | |
| correctly but its output is degenerate (`"The capital of France is"` → | |
| `"]]]]]]]]]]"`): a 327M base model with no instruction tuning and no chat | |
| template. Mechanically fine, useless for a demo. | |
| `thox-micro-125m` is not published on the Hub at all. It was the package default | |
| until v1.1.0, which meant anything relying on defaults pointed at nothing. | |