README.md

metadata

title: THOX Mesh Node
emoji: 🕸️
colorFrom: indigo
colorTo: purple
sdk: docker
app_port: 7860
pinned: false
license: apache-2.0
short_description: OpenAI-compatible THOX node in the MeshStack mesh

THOX Mesh Node — HF CPU Space

A free-tier CPU node in the THOX mesh. It serves an OpenAI-compatible endpoint from a THOX GGUF and self-registers so ThoxRoute can route traffic to it.

Endpoints

Path	Purpose
POST /v1/chat/completions	OpenAI-compatible chat completion
GET /v1/models	Advertised model
GET /health	Liveness, mesh registration state, live telemetry

Quick check

curl -s https://<space-host>/health | jq .

curl -s https://<space-host>/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"messages":[{"role":"user","content":"Say hello from the THOX mesh."}],"max_tokens":48}'

Configuration

Set these as Space Variables, except HF_TOKEN which must be a Secret.

Variable	Default	Meaning
THOX_MODEL_REPO	Thox-ai/ThoxMini-3B	Hub repo holding the GGUF
THOX_MODEL_FILE	thoxmini-3b-Q4_K_M.gguf	GGUF filename
THOX_MODEL_ID	thoxmini-3b	Model id advertised to the mesh
THOX_ALIAS	thoxmini-3b-space	Endpoint alias, unique per device
THOX_CAPABILITIES	chat,completion	Capability flags ThoxRoute matches on
THOX_CONTROLLER_URL	(unset)	Mesh controller to register with
THOX_MESH_DEVICE_ID	(unset)	Paired MeshStack device UUID
THOX_MESH_DEVICE_KEY	(unset)	Secret. Base64 Ed25519 seed
THOX_N_GPU_LAYERS	auto	auto | 0 (force CPU) | -1 (all) | an integer
HF_TOKEN	(unset)	Secret. Required — the model repo is private

CPU and GPU paths

The same agent runs on both. Nothing needs reconfiguring to move between them — THOX_N_GPU_LAYERS=auto resolves per machine.

	CPU (this Space, free Colab)	GPU (Colab T4/L4/A100)
Model	ThoxMini-3B Q4_K_M (~2 GB)	ThoxMythos-9B Q4_K_M (~6.2 GB)
Offload	n_gpu_layers=0	n_gpu_layers=-1 (all, clamped to VRAM)
llama.cpp build	CPU wheel/build	rebuilt with -DGGML_CUDA=ON
Throughput	~1.5 tok/s	roughly an order of magnitude faster

A CPU-only environment is fully supported, not a degraded one. GPU detection goes through thoxmesh_node.gpu, which treats a missing nvidia-smi, a broken driver and a hung probe all as "no GPU" and never raises. If a CUDA build is asked to offload and cannot, the node retries CPU-only rather than refusing to serve — a slower mesh member beats an absent one.

Check any machine before starting:

python -m thoxmesh_node doctor

It reports Python, the crypto/web stack, llama-cpp-python, HF_TOKEN, GPU state, cloudflared and the resolved config. Exit 0 means the node can serve.

Why this Space registers with the controller, not MeshStack v2 directly

meshstack-device-v2 validates every base_url against isPrivateHost and accepts only RFC1918, loopback, link-local, CGNAT and .local addresses. An *.hf.space origin is public and is refused by that rule — deliberately, as an anti-SSRF control on the mesh.

The node detects that specific refusal and falls back to the controller transport rather than retrying a request that can never succeed. See ARCHITECTURE.md for the contract and the proposed resolution.

Cost

Runs on the free cpu-basic tier and must stay there. ThoxMini-3B Q4_K_M is ~2 GB and fits the 16 GB / 2 vCPU allocation; nothing here requests a GPU.

Expect roughly 1.5 tokens/second. That is the honest free-CPU number, and it is why THOX_MAX_TOKENS_CAP defaults to 256 — the node reports its real latency as telemetry, so ThoxRoute ranks it behind faster nodes rather than being surprised by it.

ThoxLLM-327M-v2 was tried first because it is far smaller. It loads and serves correctly but its output is degenerate ("The capital of France is" → "]]]]]]]]]]"): a 327M base model with no instruction tuning and no chat template. Mechanically fine, useless for a demo.

thox-micro-125m is not published on the Hub at all. It was the package default until v1.1.0, which meant anything relying on defaults pointed at nothing.