MESHSCALE.md

# MeshScale

MeshScale is the CPU-only worker mesh for Hugging Face Spaces. KEY is the
first client, but the farm surface is generic enough for other CPU task
handlers.

The important boundary: KEY owns the NEAT generation. Speciation, selection,
crossover, mutation, champion compilation, and continuity stay centralized.
CPU workers only evaluate fitness shards and return `node_id -> fitness`.
Workers follow KEY's existing `Pulse`/breath rhythm. The bucket stores pulse
receipts and regulation intent; it is not a second runtime authority.

## Why CPU Spaces

KEY's current shape is NEAT-heavy. HF lists CPU Upgrade Spaces as 8 vCPU / 32 GB
at `$0.03/hr`, while A100 large is `$2.50/hr`. That makes CPU fan-out attractive
when the workload is many independent node evaluations.

## Resource Refraction

The farm is not an unbounded chain. It is a controlled expansion tree.

1. Controller submits a generation into the shared farm root.
2. Worker Spaces claim shard files.
3. If a shard is too large, a worker refracts it into child shards.
4. Workers emit expansion requests when backlog exceeds active capacity.
5. A scaler reads expansion requests and starts more CPU worker Spaces up to
   policy limits.
6. Controller merges finished fitness values back into the live population.

Policy limits:

- max workers
- max refraction depth
- target shard size
- hourly budget
- pulse TTL
- controller pulse TTL
- claim TTL

This gives the "one Space can summon more Spaces" behavior without letting
workers recursively spend money without a ledger.

## Breath Regulation

KEY publishes a controller pulse while the run is alive:

```text
/data/farm/pulses/controller.json
```

Each worker publishes its own pulse receipt:

```text
/data/farm/pulses/workers/<worker-id>.json
```

Regulation intent lives here:

```text
/data/farm/regulation/intent.json
```

Modes:

- `run`: workers may claim shards.
- `drain`: workers stop claiming beyond `target_workers`.
- `shutdown`: workers finish their current tick and exit.

If the KEY controller pulse expires, workers exit by policy. A scaler can then
pause those CPU Spaces through the HF API so upgraded CPU billing does not keep
running after the KEY Space stops.

## Bucket Layout

Use the Hugging Face bucket as a read-write volume mounted at `/data`.

```text
/data/farm/
  events.jsonl
  expansion_requests/
  pulses/
    controller.json
    workers/
  regulation/
    intent.json
  runs/
    run_<id>/
      run.json
      generations/
        0/
          jobs/
            pending/
            claimed/
            done/
            failed/
            refracted/
          results/
```

## Local Smoke

```bash
python -m meshscale.cli simulate --nodes 32 --workers 4 --shard-size 8 --clean
```

This uses local threads and a temp farm directory. It does not touch the live
TUI worker.

## Worker Space Command

For a CPU worker Space with the bucket mounted at `/data`:

```bash
python -m key_farm.worker --root /data/farm --poll-seconds 2
```

Useful environment:

```text
KEY_FARM_ROOT=/data/farm
KEY_FARM_WORKER_ID=worker-space-001
```

The repository also includes a Docker Space template at:

```text
meshscale_worker_space/
```

That template runs a small FastAPI health surface while a background thread
executes the worker loop. The important endpoints are:

```text
/health
/state
```

Publish regulation manually if needed:

```bash
python -m meshscale.cli regulate --root /data/farm --mode drain --target-workers 4
python -m meshscale.cli regulate --root /data/farm --mode shutdown --reason key_space_shutdown
```

## Scaler

MeshScale scaling is optional and policy bounded. The scaler reads
`/data/farm/expansion_requests/*.json`, starts CPU worker Spaces from a prepared
worker template, and pauses known workers when KEY stops pulsing or publishes
`shutdown` regulation.

Dry-run is the default:

```bash
python -m meshscale.cli scale-once \
  --root /data/farm \
  --template-space tostido/meshscale-worker-template \
  --namespace tostido \
  --worker-prefix meshscale-worker
```

To actually call the Hugging Face API, add `--apply` and provide `HF_TOKEN`.
The scaler still honors hard caps:

```bash
python -m meshscale.cli scale-once \
  --root /data/farm \
  --template-space tostido/meshscale-worker-template \
  --namespace tostido \
  --worker-prefix meshscale-worker \
  --max-workers 24 \
  --max-starts-per-tick 2 \
  --start-cooldown-seconds 120 \
  --budget-hourly-usd 0.72 \
  --apply
```

The current implementation uses `duplicate_space`, `request_space_hardware`,
`restart_space`, and `pause_space` from `huggingface_hub`. If Hugging Face
exposes a stable replicas method in the installed hub client, that can become a
cleaner backend without changing the bucket queue contract.

## Faculty Surface

MeshScale exposes its own CPU-farm faculty manifest:

```bash
python -m meshscale.cli faculty
```

It can also extract only the faculty/capability contract from the Cocoon
Authority substrate without importing graph links, followup edges, or route
associations:

```bash
python -m meshscale.cli faculty --cocoon-capabilities /path/to/cocoon_capabilities.py
```

That keeps MeshScale plugged into the OS substrate vocabulary without copying
the authority topology.

## Symbiotic Trace

MeshScale traces its own CPU traffic, pulse health, shard states, worker
pressure, and expansion requests:

```bash
python -m meshscale.cli symbiotic-trace --root /data/farm
```

When `cascade-lattice` is installed, the trace is interpreted through
`SymbioticAdapter` and observed through a Cascade monitor. Without Cascade, the
numeric traffic report still works.

## Capacity Planning

```bash
python -m key_farm.cli plan --pending-jobs 120 --active-workers 4 --budget-hourly-usd 0.72
```

At `$0.03/hr`, a `$0.72/hr` ceiling permits 24 CPU Upgrade workers.

## HF Setup

The clean HF topology is:

- `tostido/Ouroboros`: controller and TUI.
- `tostido/Ouroboros-worker-template`: lightweight CPU worker Space.
- `tostido/Ouroboros-storage`: mounted bucket at `/data`.

HF bucket docs describe buckets as mutable S3-like storage for logs,
checkpoints, and intermediate artifacts. HF Space volume docs show bucket
volumes mounted read-write into Spaces, which is exactly what the farm needs.
The scaler should only start workers from expansion requests and should pause
workers after `shutdown` intent or controller pulse expiry.

## Next Wire-In

The package is intentionally standalone first. The next integration point is
`PopulationManager.evaluate()`:

1. If `config["cpu_farm"]["enabled"]` is false, keep current local evaluation.
2. If enabled, submit the population to `FarmController`.
3. Wait for every node fitness for that generation.
4. Set `node.fitness` from returned results.
5. Continue normal speciation and reproduction locally.

That preserves NEAT semantics while scaling the expensive evaluation step.