MESHSCALE.md

MeshScale

MeshScale is the CPU-only worker mesh for Hugging Face Spaces. KEY is the first client, but the farm surface is generic enough for other CPU task handlers.

The important boundary: KEY owns the NEAT generation. Speciation, selection, crossover, mutation, champion compilation, and continuity stay centralized. CPU workers only evaluate fitness shards and return node_id -> fitness. Workers follow KEY's existing Pulse/breath rhythm. The bucket stores pulse receipts and regulation intent; it is not a second runtime authority.

Why CPU Spaces

KEY's current shape is NEAT-heavy. HF lists CPU Upgrade Spaces as 8 vCPU / 32 GB at $0.03/hr, while A100 large is $2.50/hr. That makes CPU fan-out attractive when the workload is many independent node evaluations.

Resource Refraction

The farm is not an unbounded chain. It is a controlled expansion tree.

1. Controller submits a generation into the shared farm root.
2. Worker Spaces claim shard files.
3. If a shard is too large, a worker refracts it into child shards.
4. Workers emit expansion requests when backlog exceeds active capacity.
5. A scaler reads expansion requests and starts more CPU worker Spaces up to policy limits.
6. Controller merges finished fitness values back into the live population.

Policy limits:

• max workers
• max refraction depth
• target shard size
• hourly budget
• pulse TTL
• controller pulse TTL
• claim TTL

This gives the "one Space can summon more Spaces" behavior without letting workers recursively spend money without a ledger.

Breath Regulation

KEY publishes a controller pulse while the run is alive:

/data/farm/pulses/controller.json

Each worker publishes its own pulse receipt:

/data/farm/pulses/workers/<worker-id>.json

Regulation intent lives here:

/data/farm/regulation/intent.json

Modes:

• run: workers may claim shards.
• drain: workers stop claiming beyond target_workers.
• shutdown: workers finish their current tick and exit.

If the KEY controller pulse expires, workers exit by policy. A scaler can then pause those CPU Spaces through the HF API so upgraded CPU billing does not keep running after the KEY Space stops.

Bucket Layout

Use the Hugging Face bucket as a read-write volume mounted at /data.

/data/farm/
  events.jsonl
  expansion_requests/
  pulses/
    controller.json
    workers/
  regulation/
    intent.json
  runs/
    run_<id>/
      run.json
      generations/
        0/
          jobs/
            pending/
            claimed/
            done/
            failed/
            refracted/
          results/

Local Smoke

python -m meshscale.cli simulate --nodes 32 --workers 4 --shard-size 8 --clean

This uses local threads and a temp farm directory. It does not touch the live TUI worker.

Worker Space Command

For a CPU worker Space with the bucket mounted at /data:

python -m key_farm.worker --root /data/farm --poll-seconds 2

Useful environment:

KEY_FARM_ROOT=/data/farm
KEY_FARM_WORKER_ID=worker-space-001

The repository also includes a Docker Space template at:

meshscale_worker_space/

That template runs a small FastAPI health surface while a background thread executes the worker loop. The important endpoints are:

/health
/state

Publish regulation manually if needed:

python -m meshscale.cli regulate --root /data/farm --mode drain --target-workers 4
python -m meshscale.cli regulate --root /data/farm --mode shutdown --reason key_space_shutdown

Scaler

MeshScale scaling is optional and policy bounded. The scaler reads /data/farm/expansion_requests/*.json, starts CPU worker Spaces from a prepared worker template, and pauses known workers when KEY stops pulsing or publishes shutdown regulation.

Dry-run is the default:

python -m meshscale.cli scale-once \
  --root /data/farm \
  --template-space tostido/meshscale-worker-template \
  --namespace tostido \
  --worker-prefix meshscale-worker

To actually call the Hugging Face API, add --apply and provide HF_TOKEN. The scaler still honors hard caps:

python -m meshscale.cli scale-once \
  --root /data/farm \
  --template-space tostido/meshscale-worker-template \
  --namespace tostido \
  --worker-prefix meshscale-worker \
  --max-workers 24 \
  --max-starts-per-tick 2 \
  --start-cooldown-seconds 120 \
  --budget-hourly-usd 0.72 \
  --apply

The current implementation uses duplicate_space, request_space_hardware, restart_space, and pause_space from huggingface_hub. If Hugging Face exposes a stable replicas method in the installed hub client, that can become a cleaner backend without changing the bucket queue contract.

Faculty Surface

MeshScale exposes its own CPU-farm faculty manifest:

python -m meshscale.cli faculty

It can also extract only the faculty/capability contract from the Cocoon Authority substrate without importing graph links, followup edges, or route associations:

python -m meshscale.cli faculty --cocoon-capabilities /path/to/cocoon_capabilities.py

That keeps MeshScale plugged into the OS substrate vocabulary without copying the authority topology.

Symbiotic Trace

MeshScale traces its own CPU traffic, pulse health, shard states, worker pressure, and expansion requests:

python -m meshscale.cli symbiotic-trace --root /data/farm

When cascade-lattice is installed, the trace is interpreted through SymbioticAdapter and observed through a Cascade monitor. Without Cascade, the numeric traffic report still works.

Capacity Planning

python -m key_farm.cli plan --pending-jobs 120 --active-workers 4 --budget-hourly-usd 0.72

At $0.03/hr, a $0.72/hr ceiling permits 24 CPU Upgrade workers.

HF Setup

The clean HF topology is:

• tostido/Ouroboros: controller and TUI.
• tostido/Ouroboros-worker-template: lightweight CPU worker Space.
• tostido/Ouroboros-storage: mounted bucket at /data.

HF bucket docs describe buckets as mutable S3-like storage for logs, checkpoints, and intermediate artifacts. HF Space volume docs show bucket volumes mounted read-write into Spaces, which is exactly what the farm needs. The scaler should only start workers from expansion requests and should pause workers after shutdown intent or controller pulse expiry.

Next Wire-In

The package is intentionally standalone first. The next integration point is PopulationManager.evaluate():

1. If config["cpu_farm"]["enabled"] is false, keep current local evaluation.
2. If enabled, submit the population to FarmController.
3. Wait for every node fitness for that generation.
4. Set node.fitness from returned results.
5. Continue normal speciation and reproduction locally.

That preserves NEAT semantics while scaling the expensive evaluation step.