Update MeshScale controller bundle

MESHSCALE.md

@@ -0,0 +1,249 @@
+# MeshScale
+
+MeshScale is the CPU-only worker mesh for Hugging Face Spaces. KEY is the
+first client, but the farm surface is generic enough for other CPU task
+handlers.
+
+The important boundary: KEY owns the NEAT generation. Speciation, selection,
+crossover, mutation, champion compilation, and continuity stay centralized.
+CPU workers only evaluate fitness shards and return `node_id -> fitness`.
+Workers follow KEY's existing `Pulse`/breath rhythm. The bucket stores pulse
+receipts and regulation intent; it is not a second runtime authority.
+
+## Why CPU Spaces
+
+KEY's current shape is NEAT-heavy. HF lists CPU Upgrade Spaces as 8 vCPU / 32 GB
+at `$0.03/hr`, while A100 large is `$2.50/hr`. That makes CPU fan-out attractive
+when the workload is many independent node evaluations.
+
+## Resource Refraction
+
+The farm is not an unbounded chain. It is a controlled expansion tree.
+
+1. Controller submits a generation into the shared farm root.
+2. Worker Spaces claim shard files.
+3. If a shard is too large, a worker refracts it into child shards.
+4. Workers emit expansion requests when backlog exceeds active capacity.
+5. A scaler reads expansion requests and starts more CPU worker Spaces up to
+   policy limits.
+6. Controller merges finished fitness values back into the live population.
+
+Policy limits:
+
+- max workers
+- max refraction depth
+- target shard size
+- hourly budget
+- pulse TTL
+- controller pulse TTL
+- claim TTL
+
+This gives the "one Space can summon more Spaces" behavior without letting
+workers recursively spend money without a ledger.
+
+## Breath Regulation
+
+KEY publishes a controller pulse while the run is alive:
+
+```text
+/data/farm/pulses/controller.json
+```
+
+Each worker publishes its own pulse receipt:
+
+```text
+/data/farm/pulses/workers/<worker-id>.json
+```
+
+Regulation intent lives here:
+
+```text
+/data/farm/regulation/intent.json
+```
+
+Modes:
+
+- `run`: workers may claim shards.
+- `drain`: workers stop claiming beyond `target_workers`.
+- `shutdown`: workers finish their current tick and exit.
+
+If the KEY controller pulse expires, workers exit by policy. A scaler can then
+pause those CPU Spaces through the HF API so upgraded CPU billing does not keep
+running after the KEY Space stops.
+
+## Bucket Layout
+
+Use the Hugging Face bucket as a read-write volume mounted at `/data`.
+
+```text
+/data/farm/
+  events.jsonl
+  expansion_requests/
+  pulses/
+    controller.json
+    workers/
+  regulation/
+    intent.json
+  runs/
+    run_<id>/
+      run.json
+      generations/
+        0/
+          jobs/
+            pending/
+            claimed/
+            done/
+            failed/
+            refracted/
+          results/
+```
+
+## Local Smoke
+
+```bash
+python -m meshscale.cli simulate --nodes 32 --workers 4 --shard-size 8 --clean
+```
+
+This uses local threads and a temp farm directory. It does not touch the live
+TUI worker.
+
+## Worker Space Command
+
+For a CPU worker Space with the bucket mounted at `/data`:
+
+```bash
+python -m key_farm.worker --root /data/farm --poll-seconds 2
+```
+
+Useful environment:
+
+```text
+KEY_FARM_ROOT=/data/farm
+KEY_FARM_WORKER_ID=worker-space-001
+```
+
+The repository also includes a Docker Space template at:
+
+```text
+meshscale_worker_space/
+```
+
+That template runs a small FastAPI health surface while a background thread
+executes the worker loop. The important endpoints are:
+
+```text
+/health
+/state
+```
+
+Publish regulation manually if needed:
+
+```bash
+python -m meshscale.cli regulate --root /data/farm --mode drain --target-workers 4
+python -m meshscale.cli regulate --root /data/farm --mode shutdown --reason key_space_shutdown
+```
+
+## Scaler
+
+MeshScale scaling is optional and policy bounded. The scaler reads
+`/data/farm/expansion_requests/*.json`, starts CPU worker Spaces from a prepared
+worker template, and pauses known workers when KEY stops pulsing or publishes
+`shutdown` regulation.
+
+Dry-run is the default:
+
+```bash
+python -m meshscale.cli scale-once \
+  --root /data/farm \
+  --template-space tostido/meshscale-worker-template \
+  --namespace tostido \
+  --worker-prefix meshscale-worker
+```
+
+To actually call the Hugging Face API, add `--apply` and provide `HF_TOKEN`.
+The scaler still honors hard caps:
+
+```bash
+python -m meshscale.cli scale-once \
+  --root /data/farm \
+  --template-space tostido/meshscale-worker-template \
+  --namespace tostido \
+  --worker-prefix meshscale-worker \
+  --max-workers 24 \
+  --max-starts-per-tick 2 \
+  --start-cooldown-seconds 120 \
+  --budget-hourly-usd 0.72 \
+  --apply
+```
+
+The current implementation uses `duplicate_space`, `request_space_hardware`,
+`restart_space`, and `pause_space` from `huggingface_hub`. If Hugging Face
+exposes a stable replicas method in the installed hub client, that can become a
+cleaner backend without changing the bucket queue contract.
+
+## Faculty Surface
+
+MeshScale exposes its own CPU-farm faculty manifest:
+
+```bash
+python -m meshscale.cli faculty
+```
+
+It can also extract only the faculty/capability contract from the Cocoon
+Authority substrate without importing graph links, followup edges, or route
+associations:
+
+```bash
+python -m meshscale.cli faculty --cocoon-capabilities /path/to/cocoon_capabilities.py
+```
+
+That keeps MeshScale plugged into the OS substrate vocabulary without copying
+the authority topology.
+
+## Symbiotic Trace
+
+MeshScale traces its own CPU traffic, pulse health, shard states, worker
+pressure, and expansion requests:
+
+```bash
+python -m meshscale.cli symbiotic-trace --root /data/farm
+```
+
+When `cascade-lattice` is installed, the trace is interpreted through
+`SymbioticAdapter` and observed through a Cascade monitor. Without Cascade, the
+numeric traffic report still works.
+
+## Capacity Planning
+
+```bash
+python -m key_farm.cli plan --pending-jobs 120 --active-workers 4 --budget-hourly-usd 0.72
+```
+
+At `$0.03/hr`, a `$0.72/hr` ceiling permits 24 CPU Upgrade workers.
+
+## HF Setup
+
+The clean HF topology is:
+
+- `tostido/Ouroboros`: controller and TUI.
+- `tostido/Ouroboros-worker-template`: lightweight CPU worker Space.
+- `tostido/Ouroboros-storage`: mounted bucket at `/data`.
+
+HF bucket docs describe buckets as mutable S3-like storage for logs,
+checkpoints, and intermediate artifacts. HF Space volume docs show bucket
+volumes mounted read-write into Spaces, which is exactly what the farm needs.
+The scaler should only start workers from expansion requests and should pause
+workers after `shutdown` intent or controller pulse expiry.
+
+## Next Wire-In
+
+The package is intentionally standalone first. The next integration point is
+`PopulationManager.evaluate()`:
+
+1. If `config["cpu_farm"]["enabled"]` is false, keep current local evaluation.
+2. If enabled, submit the population to `FarmController`.
+3. Wait for every node fitness for that generation.
+4. Set `node.fitness` from returned results.
+5. Continue normal speciation and reproduction locally.
+
+That preserves NEAT semantics while scaling the expensive evaluation step.

MESHSCALE_CLAUDE_DEEP_AUDIT_2026-06-19.md

@@ -0,0 +1,75 @@
+# MeshScale - Claude Deep Audit - 2026-06-19
+
+Extends `MESHSCALE_CLAUDE_RETURN_PAYLOAD_2026-06-19.md` (data-plane pass) with a full source audit. Role unchanged: independent corroboration; Codex remains primary; apply nothing on agreement alone, verify each claim against the cited source + live state.
+
+## Coverage (what I actually read)
+
+Full: key_farm/{contracts,serialization,store,evaluator,worker,controller,regulation,scaler,symbiotic_trace,amalgam_bridge}.py; meshscale_space/{app.py,os_routes.py}; meshscale_worker_space/app.py; KEY config.json (cpu_farm block); population.py farm-integration (304-453); run.py signatures.
+Not audited (lower risk / out of farm-correctness scope): key_farm/{cli,faculty}.py, meshscale_space/static/{app.js,index.html}, meshscale/cli.py, the non-farm bulk of population.py and run.py. No live Space/bucket calls (Codex's runtime lane).
+
+## Confirmations (system behaves as claimed)
+
+- **V1** Data-plane externalization correct end-to-end (see return payload C1-C4). Reconfirmed.
+- **V2** Farm IS wired into KEY: `PopulationManager.evaluate()` -> `_evaluate_with_cpu_farm()` -> `FarmController.submit_generation`/`collect_generation`, gated by `config.cpu_farm.enabled=true` (population.py:441,318-385). The MESHSCALE.md "Next Wire-In" is done.
+- **V3** Graceful degradation: farm exception -> `fallback_to_local` -> KEY evaluates the generation locally (population.py:386-391; config:320). Farm problems slow KEY, they do not break it.
+- **V4** Live config genuinely yields 1 node/job: `cpu_farm.target_batch_size=1`, `min_batch_size=1` (config.json:309-310) -> batch=max(1,1)=1 -> 32 jobs for pop 32. Reconciles the report's "32 huge JSON files."
+- **V5** Cooling/pause present (scaler.py:116-145): pause on shutdown or controller-lost; `cooling_idle_jobs` cancels active Jobs when pending=0 and claimed=0.
+- **V6** Jobs path is coherent: `_start_jobs` passes explicit `KEY_FARM_TARGET_BATCH_SIZE=1`, mounts the same `tostido/Ouroboros-storage` bucket at `/data`, and passes HF_TOKEN via `secrets=` not plaintext env (scaler.py:686-710).
+- **V7** Claim mutual-exclusion via atomic `rename` with `FileNotFoundError` race handling; result is persisted before the claim is released (store.py:220-244, 246-277) -> idempotent and crash-tolerant.
+- **V8** Trace has a 2 MB per-file read guard (symbiotic_trace.py:15,79) so it will not load a 211 MB file into memory.
+
+## HIGH
+
+- **H1 - Distributed pickle version-coupling (the #1 future-proof risk; explains Q1/Q2).** Payloads are pickled genome objects; the worker does `from run import create_fitness_function` and `unpack_node` -> `pickle.loads` (evaluator.py:18-20,64; serialization.py:42-71). KEY (controller) and the worker-template therefore must deploy in lockstep with identical genome/class definitions and `run.py`. Any skew -> `pickle.loads` raises or silently builds a wrong object -> node errors (then H2). This is exactly why "did KEY rebuild / did worker-template rebuild" (Q1/Q2) must be a coordinated pair, never one-sided. Fix direction: stamp payloads with a genome/schema version and have the worker reject a mismatch explicitly (clear error) instead of failing deep in unpickle; deploy KEY + worker-template atomically.
+
+- **H2 - One bad/slow node stalls the whole generation 900 s, then full LOCAL recompute.** `collect_generation` is all-or-nothing on `expected_node_ids` (controller.py:228-232); a node whose fitness raises never enters `fitness_by_id` (evaluator.py:73-81 record error; controller.result_map skips errors) -> waits the full `timeout_seconds=900` -> `TimeoutError` -> caught -> fallback re-evaluates the ENTIRE generation locally, discarding completed farm work (population.py:373-391). Worst case per affected generation ~900 s wall + full local compute; across a 420-gen run a recurring bad genome class is very expensive. Fix direction: treat error-results as completed with a sentinel/worst fitness so the generation closes; or accept partial farm results and locally evaluate only the missing node_ids; or shorten the farm timeout well below a full-generation local cost.
+
+## MEDIUM
+
+- **M2 - `objects/` grows unbounded, no GC.** No deletion anywhere in store.py; objects are content-addressed at `root/objects` and `_cpu_farm_run_id` persists across all generations (population.py:355), so genomes differ each gen and the shared paid bucket grows monotonically. Fix: GC objects unreferenced by a live run, or TTL reaper, before long unattended runs.
+
+- **M3 - Dashboard/trace are not run-scoped (Q8, broadened).** `_job_counts` and `_result_stats` glob `runs/*/generations/*/...` across ALL runs (symbiotic_trace.py:65-66,139), so `large_files`, `large_file_count`, `failed`, and `node_errors` are all-time aggregates. A fresh run's dashboard shows the old 211 MB files as `large_files` and stays `equilibrium="faulted"` from historical failures (meshscale_space/app.py:284-285). The run_id is already in each path, so scoping is straightforward. Fix: filter trace/dashboard to the current (or last-N) run_id.
+
+- **M4 - Ungated mutation endpoints on the controller Space.** `POST /api/regulate` writes regulation (drain/shutdown) with no auth (meshscale_space/app.py:480-499). On a public Space anyone can drain/shutdown the farm (a shutdown then triggers the scaler's job-cancel on next tick). Scaling apply is double-gated (env `MESHSCALE_ENABLE_APPLY` + payload `apply`), but regulate is not. Fix: shared-secret header on mutation endpoints, or keep the controller Space private.
+
+- **M5 - No cumulative spend ceiling / run kill-switch.** `budget_hourly_usd` caps CONCURRENT workers (a rate via `effective_max_workers`), not total spend. A controller that stays alive with perpetual backlog bills at the budget rate indefinitely; only per-Job `timeout=3600s` and idle-cooling bound it. Fix: add a max-total-spend or max-run-wallclock guard that publishes `shutdown`.
+
+- **M6 - Two divergent worker policy sources.** Worker-Space autostart builds `FarmPolicy` from env with `target_batch_size` default 24 (meshscale_worker_space/app.py:67), while the live Jobs path passes 1. Refraction/split and TTLs use the worker's own policy, not the job's `config`. Low impact while live mode is Jobs, latent otherwise. Fix: derive split/TTL policy from `job.config["cpu_farm"]`, or pin worker env to match KEY.
+
+- **M1 - Long synchronous eval breaks pulse/claim liveness (Q7).** Worker writes a pulse only at claim/idle (worker.py:64-80), then runs `evaluate_job` synchronously. A node eval exceeding `pulse_ttl=120s` makes the worker look inactive to `active_worker_count`/scaler; exceeding `claim_ttl=900s` gets the claim requeued (store.py:406-423) -> duplicate eval. No heartbeat during evaluation. Fix: emit pulses during long evals (background ticker / per-node), and keep `claim_ttl` > max expected node time.
+
+## LOW
+
+- **L1** Object write lacks the `PermissionError` retry the JSON writer has (store.py:166-171 vs 96-106); a contended bucket replace can fail submit. Reuse the atomic-retry wrapper for objects.
+- **L2** `externalize_payloads` is not pinned in config (config.cpu_farm has no such key); it relies on the dataclass default `True`. A default flip silently disables the fix. Set it explicitly.
+- **L3** Pulse liveness compares file mtime to the reader's local clock (store.py:333,346); cross-Space clock skew or bucket mtime semantics can cause false pulse-loss/alive. Tolerate skew or use embedded timestamps.
+- **L4** Externalization threshold is measured on base64 length (~33% > raw), so the effective on-disk trigger is ~196 KB (store.py:159). Document.
+- **L5** Scaler reads controller TTL from a fresh `FarmPolicy()` default (180), ignoring any configured TTL (scaler.py:111-113). Minor.
+- **L6** `_meshscale_jobs` matches its Jobs by label or name-prefix (scaler.py:635-648); Jobs from an older naming scheme could be missed by cooling -> orphan billing. Minor.
+- **L7** `read_regulation` returns `drain` on a corrupt/torn file (store.py:388-395) - safe direction, but torn reads on a non-atomic FS cause transient false drain.
+
+## Correction to my own return payload (evidence-first)
+
+The return payload implied `min_batch_size=4` floors the batch. Live `config.json` sets `cpu_farm.min_batch_size=1`, so the live batch is genuinely 1. The "floored at 4" detail only applies if `min_batch_size` is left at the dataclass default (4). Q6 tuning therefore works as expected with the live config; the per-node-externalization caution (return payload I1) still applies when raising the batch.
+
+## Cross-cutting theme: atomicity on the bucket
+
+Multiple mechanisms (claim via `rename`, `_write_json_atomic` via temp+`replace`, object `replace`, pulse-mtime liveness) assume POSIX rename/replace atomicity and trustworthy mtime. The report itself cautions against assuming HF replica/bucket semantics. The 6x `PermissionError` retry in `_write_json_atomic` is evidence this FS is already contended. None of this is necessarily wrong on a true volume mount, but it is the substrate to watch: if `/data` is ever S3-like rather than a POSIX volume, claims can double and writes can tear. Recommend a one-time runtime check that rename is atomic on the live mount.
+
+## Open questions - final
+
+- Q1/Q2 (KEY/worker-template rebuilt): must be a coordinated pair - see H1 (pickle lockstep). The requirement is now explained, the runtime fact is still Codex's to confirm.
+- Q3/Q4 (file refs + objects under /data/farm/objects): yes (return payload).
+- Q5 (generation time drop): expected once job JSON collapses; runtime-confirm.
+- Q6 (tune target_batch_size): works with live `min_batch_size=1`; pair any increase with shard-level externalization (return payload I1).
+- Q7: M1 - yes, long evals break liveness; mechanism above.
+- Q8: M3 - yes, confirmed and broadened to failed/node_errors; not run-scoped.
+
+## Recommended order
+
+1. H1 deploy-lockstep discipline + payload version stamp (prevents the silent class of farm failures).
+2. H2 generation-completion on node error (removes the 900 s stall + local-recompute tax) - highest runtime ROI.
+3. M3 run-scope the dashboard (so the fresh-run proof point reads cleanly and you can trust the green).
+4. M2 objects GC and M5 spend kill-switch before long unattended runs.
+5. M4 lock the controller Space mutation endpoints.
+6. Sweep the LOWs opportunistically.

MESHSCALE_CLAUDE_INGEST_PAYLOAD_2026-06-19.md

@@ -0,0 +1,68 @@
+# MeshScale - Codex Ingest Payload - 2026-06-19
+
+This packet is meant to be reread by Codex as the compact current-state summary for the MeshScale / KEY farm seam.
+
+Primary rule:
+
+- Codex remains primary operator.
+- Claude output is corroboration input, not command authority.
+- No claim is actionable until it is matched to source, runtime, or a reproducible local check.
+
+## Verified Chain
+
+The deployed payload fix is verified end-to-end in local source.
+
+1. `key_farm/controller.py` packs KEY nodes for generation submission.
+2. `key_farm/store.py` externalizes large inline payloads during `put_job()`.
+3. `key_farm/store.py` stores object bytes under `objects/<sha>.pkl.gz` when the payload crosses the threshold.
+4. `key_farm/worker.py` routes claims into `evaluate_job()`.
+5. `key_farm/evaluator.py` dispatches to handlers that both call `unpack_node()`.
+6. `key_farm/serialization.py` resolves `pickle+gzip+file` by reading object bytes, verifying SHA, and loading the pickle.
+
+The specific verified form is:
+
+`KEY packs node -> inline base64 -> put_job externalizes if >= 262144 -> objects/<sha>.pkl.gz + small ref -> worker evaluate_job -> unpack_node file branch -> gzip-decompress + SHA-verify -> pickle.loads -> fitness`
+
+## Confirmed Facts
+
+- `FarmPolicy.externalize_payloads = True`
+- `FarmPolicy.payload_ref_threshold_bytes = 262144`
+- The file encoding branch is supported alongside legacy `pickle+gzip+base64`
+- The object write/read symmetry is correct
+- The worker decode route already reaches `unpack_node()`
+- Local validation passed with a mock large payload
+- `python -m py_compile` passed for the farm core modules touched by the fix
+
+## Live MeshScale State
+
+Last verified live MeshScale status:
+
+- Pending jobs: `0`
+- Claimed jobs: `0`
+- Active Hugging Face Jobs: `0`
+- Active worker pulses: `0`
+- Controller pulse alive: `false`
+- Burn rate: `$0.00/h`
+- Autoscaler alive: `true`
+- Autoscaler last reason: `pause`
+
+Interpretation:
+
+MeshScale is currently idle, not billing, and waiting for a fresh KEY controller pulse or a new run.
+
+## Current Risk Notes
+
+- Existing 211 MB job files are historical artifacts from the old inline payload format.
+- A fresh KEY run is the proof point for the new format.
+- If KEY and the worker template are not rebuilt together, pickled genome/class skew can still fail runtime decoding.
+- `objects/` cleanup is still an open long-run cost issue.
+- Batch sizes above 1 should be revisited only after the new payload format is confirmed in a fresh run.
+
+## What To Check Next
+
+1. Confirm KEY rebuild picked up the payload externalization code.
+2. Confirm worker-template rebuild picked up the same farm code.
+3. Start a small fresh run.
+4. Verify new job JSON files are small and that `objects/` receives the large payload bytes.
+5. Check that generation wall time drops materially compared to the historical 211 MB format.
+

MESHSCALE_CLAUDE_ORIENTATION_NOTE_2026-06-19.md

@@ -0,0 +1,41 @@
+# MeshScale - Claude Orientation Note - 2026-06-19
+
+This note is for future Claude sessions joining the MeshScale / KEY workflow.
+
+## Role
+
+- Codex stays primary operator.
+- Claude is the independent corroboration pass.
+- A Claude claim is only actionable after source, runtime, or local repro verification.
+
+## Current Live Assumptions
+
+- KEY uses the CPU-farm path for node evaluation.
+- MeshScale is the worker substrate, not the controller authority.
+- Hugging Face Jobs are the current worker mechanism.
+- Replicas are not the active compute path for this workload.
+- Live MeshScale was last verified idle, with `0` pending jobs, `0` claimed jobs, `0` active Jobs, and `$0.00/h` burn.
+
+## What Was Verified
+
+- Large node payloads externalize out of job JSON into `objects/<sha>.pkl.gz`.
+- The worker route still reaches `unpack_node()`.
+- The new file encoding branch is backward compatible with the old inline encoding.
+- The end-to-end pack -> externalize -> claim -> decode -> fitness path is coherent in local source.
+
+## What To Recheck First On Return
+
+1. Read `MESHSCALE_PRIMARY_COORDINATION_REPORT_2026-06-19.md`.
+2. Read `MESHSCALE_CLAUDE_RETURN_PAYLOAD_2026-06-19.md`.
+3. Confirm the live MeshScale endpoint still shows idle or active state correctly.
+4. Verify whether a fresh KEY run is using the new file-reference payload format.
+5. Treat old 211 MB artifacts as historical until a fresh run shows otherwise.
+
+## Do Not Lose
+
+- Codex remains the operator.
+- Evidence beats agreement.
+- Live runtime beats archive tone.
+- Generated champion files stay untouched.
+- `objects/` growth still needs long-run attention.
+

MESHSCALE_CLAUDE_RETURN_PAYLOAD_2026-06-19.md

@@ -0,0 +1,57 @@
+# MeshScale - Claude Return Payload - 2026-06-19
+
+Role: independent corroboration pass. Codex remains primary operator. This is a proposal packet per `MESHSCALE_PRIMARY_COORDINATION_REPORT_2026-06-19.md`: claims first, each tied to a source surface and a verification status. Apply nothing on conceptual agreement alone.
+
+## Verification Scope
+
+I verified the payload fix end-to-end in local source only:
+
+- `key_farm/contracts.py`
+- `key_farm/serialization.py`
+- `key_farm/store.py` (1-190)
+- `key_farm/worker.py` (delegation)
+- `key_farm/evaluator.py` (unpack path)
+
+I did not read `controller.py`, `scaler.py`, `symbiotic_trace.py`, `meshscale_space/app.py`, `population.py`, `KEY worker.py`, or `config.json`. Claims touching those surfaces are unverified in this packet. I did not hit live Spaces or the bucket; that stays Codex's runtime lane.
+
+## Confirmed
+
+- `C1` - Deployed payload fix is present and correct, end-to-end.
+  - Policy: `FarmPolicy.externalize_payloads=True`, `payload_ref_threshold_bytes=262144` (`contracts.py`).
+  - Submit path: `put_job` -> `_externalize_job_payloads` -> `_externalize_node_payload` (`store.py`). Inline base64 is decoded back to `gzip(pickle)` bytes and written to `objects/<sha>.pkl.gz`; job JSON keeps `{encoding: "pickle+gzip+file", payload: "objects/<sha>.pkl.gz", sha256}`.
+  - Decode path: `worker.py` -> `evaluator.evaluate_job` -> `unpack_node`. `unpack_node` FILE branch does `gzip.decompress(file)` -> raw pickle -> SHA check vs `node.sha256` -> `pickle.loads` (`serialization.py`).
+  - Write/read symmetry holds: sha is computed over the uncompressed pickle on both sides. No silent corruption. The 211 MB inline-JSON failure mode is genuinely addressed.
+
+- `C2` - Backward compatible. `unpack_node` still handles legacy `pickle+gzip+base64`. SHA verification is retained on both encodings. Old inline jobs and not-yet-rebuilt workers should not break on the new code.
+
+- `C3` - Object store is farm-root-global, content-addressed, and deduped. `objects_dir = root/objects`; `if not object_path.exists()` skips rewrite. References resolve to `/data/farm/objects/` via `_farm_root()`, matching the expected evidence location.
+
+- `C4` - Externalization is on-by-default. `_externalize_job_payloads` reads policy from `job.config["cpu_farm"]` via `FarmPolicy.from_dict`; `from_dict(None)` returns defaults with `externalize_payloads=True`. So a fresh run externalizes any inline node at or above threshold even if KEY passes no `cpu_farm` block. The only ways it does not apply are: node already non-inline, `len(payload) < 262144`, or policy explicitly sets `externalize_payloads=false`.
+
+## New Issues
+
+- `I1 [MEDIUM]` Per-node threshold regresses under batch growth. Externalization decides per node (`len(node.payload) < threshold`). Live `target_batch_size=1` is fine. But the dataclass default is `target_batch_size=24`, and if batch size rises a shard full of sub-threshold nodes re-bloats the JSON with zero externalization. Fix direction: externalize at the shard level when serialized job size exceeds a threshold, not only per node.
+
+- `I2 [MEDIUM]` `objects/` never gets garbage-collected. No retention exists anywhere in `store.py`. Genomes differ each generation, so dedupe does not bound growth. The shared paid bucket will grow monotonically unless a sweep or TTL reaper is added.
+
+- `I3 [LOW]` Object-write resiliency is asymmetric. `_write_json_atomic` retries `PermissionError` 6x, but the object write path does not share that retry behavior. A contended object replace can fail submit.
+
+- `I4 [LOW/NOTE]` The threshold is measured on encoded length, so the effective on-disk trigger is about 196 KB, not 256 KB raw bytes.
+
+- `I5 [LOW]` Relative object refs assume every process shares `KEY_FARM_ROOT`. That is true in the standard `/data/farm` topology, but tests or alternate mounts can mis-resolve refs unless they match the same root.
+
+## Open Questions
+
+- `Q3/Q4` yes: fresh runs should produce `pickle+gzip+file` refs and `objects/<sha>.pkl.gz`.
+- `Q6` answered by `I1`: batch growth reintroduces bloat unless externalization moves to the shard level.
+- `Q7/Q8` unknown-until-checked in this packet: I did not read worker pulse cadence or the dashboard.
+- `Q1/Q2/Q5` are deploy/runtime facts in Codex's lane.
+
+## Honest Scope
+
+I verified the data-plane fix only. I did not read `scaler.py`, the dashboard, controller, or population, so the autoscaler section of the broader report remains unverified by me in this packet.
+
+## Artifact
+
+This file is the ingestible payload for Codex. The deeper audit belongs in `MESHSCALE_CLAUDE_DEEP_AUDIT_2026-06-19.md`.
+

MESHSCALE_PRIMARY_COORDINATION_REPORT_2026-06-19.md

@@ -0,0 +1,259 @@
+# MeshScale Primary Coordination Report - 2026-06-19
+
+This is not a handoff. Codex remains the primary operator. Claude or any other model output should be treated as an incoming proposal packet that must be verified against local source, live Hugging Face Space state, and bucket evidence before any change is applied.
+
+## Operating Rule
+
+Incoming analysis should be sequenced through this order:
+
+1. Identify the claim.
+2. Locate the source surface it affects.
+3. Verify against local code.
+4. Verify against live MeshScale, KEY, worker-template, or bucket state when the claim is runtime-related.
+5. Apply the narrowest source change that satisfies the verified problem.
+6. Re-test locally where possible.
+7. Deploy narrowly to the relevant Hugging Face Space or Spaces.
+8. Re-check live state after rebuild.
+
+Do not treat conceptual agreement as implementation proof.
+
+## Claude Corroboration Lane
+
+Claude's role in this sequence is independent corroboration, not command authority.
+
+Claude output should be parsed into:
+
+- claims
+- evidence cited
+- affected source surfaces
+- affected runtime surfaces
+- proposed edits
+- risks or unresolved seams
+
+Codex then verifies each item against the active machinery before action. A Claude claim can become actionable only after it is matched to at least one concrete source file, runtime endpoint, bucket artifact, or reproducible local check. If Claude's reasoning is sound but evidence is absent, the next action is a read or probe, not a patch.
+
+Claude can veto weak reasoning by exposing an unverified seam. Claude cannot override live evidence, source truth, operator constraints, or the no-generated-champion-files boundary.
+
+## Claude Corroboration Receipts
+
+### Receipt 001 - Payload Object Encoding Symmetry
+
+Claude claim:
+
+The deployed object-reference payload fix is correct only if `store.py` writes object bytes in the exact form `serialization._decode_file_payload()` reads them. A mismatch would cause worker-side decode failures.
+
+Codex verification:
+
+Confirmed against local source.
+
+- `key_farm/store.py` decodes the inline payload with `base64.b64decode(node.payload.encode("ascii"))` and writes those bytes to `objects/<sha>.pkl.gz`.
+- `key_farm/serialization.py` resolves the file reference and calls `gzip.decompress(path.read_bytes())`.
+- The original inline format is `base64(gzip(pickle_bytes))`, so stripping base64 and storing the gzip bytes is exactly what the file decoder expects.
+
+Status:
+
+Confirmed. No patch required for this seam.
+
+### Receipt 002 - Worker Decode Route
+
+Claude claim:
+
+The worker must route file-encoded nodes through `unpack_node()`, otherwise the new object-reference payload format would never reach the actual decode path.
+
+Codex verification:
+
+Confirmed against local source.
+
+- `key_farm/worker.py` calls `evaluate_job(job, worker_id)` when a shard is ready for execution.
+- `key_farm/evaluator.py` dispatches that job to `evaluate_key_fitness()` or `evaluate_python_callable()` through `get_handler()`.
+- Both handlers call `unpack_node(node_payload)` before using the node object.
+
+Status:
+
+Confirmed. The worker execution path already uses the shared unpacker, so the new `pickle+gzip+file` format is reachable without any extra worker-side routing patch.
+
+### Receipt 003 - End-To-End Payload Round Trip
+
+Claude claim:
+
+The full round trip is now verified in local source:
+
+`KEY packs node -> inline base64 -> put_job externalizes if >= 262144 -> objects/<sha>.pkl.gz + small ref -> worker evaluate_job -> unpack_node file branch -> gzip-decompress + SHA-verify -> pickle.loads -> fitness`
+
+Codex verification:
+
+Confirmed against the active source chain.
+
+- `key_farm/controller.py` calls `pack_node()` when submitting KEY fitness generations.
+- `key_farm/store.py` externalizes large inline payloads during `put_job()`.
+- `key_farm/worker.py` routes claims into `evaluate_job()`.
+- `key_farm/evaluator.py` reaches `unpack_node()` for both supported handlers.
+- `key_farm/serialization.py` resolves `pickle+gzip+file` by reading object bytes, verifying SHA, and loading the pickle.
+
+Status:
+
+Confirmed. The deployed fix is real and coherent end-to-end on the local source path.
+
+## Active Resources
+
+- KEY Space: `tostido/Ouroboros`
+- MeshScale controller Space: `tostido/meshscale`
+- MeshScale worker template Space: `tostido/meshscale-worker-template`
+- Shared bucket: `tostido/Ouroboros-storage`
+- Local repo root: `D:\End-Game\ended_game\KEY-literal`
+
+## Hard Constraints
+
+- Do not edit generated champion files.
+- Do not revert unrelated local changes.
+- Do not full-folder upload unless specifically required.
+- Prefer narrow `HfApi.upload_file` deploys for shared farm code.
+- Treat the bucket as shared runtime state, not ordinary disposable repo content.
+- Do not assume Hugging Face replica semantics apply to background compute. This system is currently using Hugging Face Jobs for worker execution.
+
+## Current Verified Live State
+
+Live MeshScale check after resuming showed:
+
+- Pending jobs: `0`
+- Claimed jobs: `0`
+- Active worker pulses: `0`
+- Active Hugging Face Jobs: `0`
+- Controller pulse alive: `false`
+- Burn rate: `$0.00/h`
+- Autoscaler alive: `true`
+- Autoscaler last reason: `pause`
+- Live farm policy includes:
+  - `target_batch_size: 1`
+  - `max_workers: 48`
+  - `budget_hourly_usd: 1.44`
+  - `externalize_payloads: true`
+  - `payload_ref_threshold_bytes: 262144`
+
+Interpretation:
+
+MeshScale is not currently billing for active Jobs. It is idle because KEY is not publishing a fresh live controller pulse into the farm.
+
+## Confirmed Problem
+
+The farm worked functionally, but the previous payload data plane was too heavy.
+
+Observed historical run state:
+
+- Run id: `run_250aa8b0f80d4d64`
+- Completed farm generations visible: `0` and `1`
+- Completed shards: `64`
+- Node results: `64`
+- Node errors: `0`
+- Typical done job file size: about `211 MB`
+- Large job files remain in old run data under `/data/farm/runs/.../jobs/done`
+
+The old design embedded serialized node payloads directly inside every job JSON. With `target_batch_size=1` and population `32`, one generation created 32 huge JSON files. State transitions then copied or rewrote those heavy payloads through bucket storage. This made MeshScale appear slow even while Jobs were processing.
+
+## Deployed Fix
+
+The shared farm code now externalizes large serialized node payloads into object files and leaves small references in job JSON.
+
+Uploaded narrowly to:
+
+- `tostido/meshscale`
+- `tostido/meshscale-worker-template`
+- `tostido/Ouroboros`
+
+Uploaded files:
+
+- `key_farm/contracts.py`
+- `key_farm/serialization.py`
+- `key_farm/store.py`
+- `key_farm/scaler.py`
+
+Behavior added:
+
+- `FarmPolicy.externalize_payloads = True`
+- `FarmPolicy.payload_ref_threshold_bytes = 262144`
+- Large inline payloads become objects under `objects/<sha>.pkl.gz`
+- Job JSON stores `encoding: pickle+gzip+file`
+- Worker-side unpacking still supports old `pickle+gzip+base64`
+- SHA verification remains in place
+
+Local validation:
+
+- A mock 1 MB payload produced a `642` byte job file.
+- The referenced payload unpacked correctly.
+- `python -m py_compile` passed for:
+  - `key_farm/contracts.py`
+  - `key_farm/serialization.py`
+  - `key_farm/store.py`
+  - `key_farm/scaler.py`
+  - `key_farm/controller.py`
+  - `key_farm/worker.py`
+
+## Autoscaler Cooling Fix
+
+The scaler has an idle-cooling path for Jobs mode:
+
+- If controller is alive, pending count is `0`, claimed count is `0`, and active Jobs exist, cancel Jobs and report `cooling_idle_jobs`.
+- If controller pulse is lost and `pause_on_controller_lost` is true, cancel Jobs through the pause path.
+
+Live status after deploy showed the pause path had already brought active Jobs to `0`.
+
+## Important Caveat
+
+Existing giant job files are historical run artifacts. They do not prove the new code failed unless they appear in a new run submitted after KEY has rebuilt with the new farm code.
+
+The next real validation is a fresh KEY generation submitted after rebuild. Expected evidence:
+
+- New pending/claimed/done job JSON files should be small.
+- New object files should appear under `/data/farm/objects/`.
+- Node results should still populate normally.
+- MeshScale should scale Jobs while pending exists, then return to zero active Jobs when the queue drains.
+
+## Verification Targets For Incoming Payloads
+
+Any proposal from Claude should be checked against these local surfaces:
+
+- `key_farm/contracts.py`
+- `key_farm/serialization.py`
+- `key_farm/store.py`
+- `key_farm/controller.py`
+- `key_farm/worker.py`
+- `key_farm/scaler.py`
+- `key_farm/symbiotic_trace.py`
+- `meshscale_space/app.py`
+- `meshscale_space/static/app.js`
+- KEY `config.json`
+- KEY `population.py`
+- KEY `worker.py`
+
+Runtime verification targets:
+
+- MeshScale status endpoint: `https://tostido-meshscale.hf.space/api/status`
+- MeshScale dashboard endpoint: `https://tostido-meshscale.hf.space/api/dashboard`
+- KEY Space: `https://huggingface.co/spaces/tostido/Ouroboros`
+- Bucket: `https://huggingface.co/buckets/tostido/Ouroboros-storage`
+
+## Open Technical Questions
+
+1. Did KEY rebuild after the object-reference payload fix?
+2. Did worker-template rebuild before any new Jobs were launched?
+3. Does the next fresh run produce `pickle+gzip+file` node payload references?
+4. Are object files visible under `/data/farm/objects/` during the fresh run?
+5. Does generation time drop once job JSON size collapses?
+6. Is `target_batch_size=1` still optimal after payload externalization, or should it be tuned to `2`, `4`, or adaptive per generation?
+7. Should worker pulses continue during long node evaluation to avoid false pulse loss on unusually heavy nodes?
+8. Should the dashboard separate historical large files from current-run large files to avoid false alarms?
+
+## Suggested Next Sequence
+
+1. Wait for Claude's payload.
+2. Extract concrete claims only.
+3. Ignore style-only or speculative architecture suggestions unless they map to a verified failure.
+4. Re-check live MeshScale state.
+5. If KEY is idle, start a small fresh run after rebuild to validate the new payload format.
+6. Inspect the new farm run tree for small job JSON and object references.
+7. If validated, scale up run size cautiously.
+8. If not validated, inspect why `FarmStore._externalize_job_payloads()` did not apply.
+
+## Current Primary Diagnosis
+
+MeshScale's architecture is viable using Hugging Face Jobs. The immediate speed failure was a data-plane design issue: enormous serialized node payloads were embedded in job JSON and repeatedly moved through bucket-backed state folders. The deployed fix changes the queue from payload-heavy JSON to payload references plus shared object files. The next fresh run is the proof point.

MESHSCALE_SITREP_TRAJECTORY_FOR_CLAUDE_2026-06-19.md

@@ -0,0 +1,199 @@
+# MeshScale / KEY Sitrep and Projected Trajectory - 2026-06-19
+
+Role: briefing packet for Claude to audit. Codex remains primary operator. This document separates verified state, inferred trajectory, and remaining risk surfaces.
+
+## Bottom Line
+
+The farm path is real and functional. The main throughput problem that was making KEY feel slow was not compute failure in the farm path itself, but the old data plane: giant node payloads were being carried inside job JSON. That has been fixed in source.
+
+The next constraint is not rendering or GUI work on the worker side. The worker should stay headless. Any visual or routing surface belongs in the controller/dashboard layer, not in the compute worker.
+
+## Last Verified State
+
+- KEY CPU farm integration is enabled in `config.json`.
+- Live farm policy in config:
+  - `target_batch_size: 1`
+  - `min_batch_size: 1`
+  - `max_workers: 48`
+  - `budget_hourly_usd: 1.44`
+  - `cpu_space_hourly_usd: 0.03`
+  - `timeout_seconds: 900`
+  - `fallback_to_local: true`
+- MeshScale was last verified idle:
+  - pending jobs: `0`
+  - claimed jobs: `0`
+  - active jobs: `0`
+  - active worker pulses: `0`
+  - controller pulse alive: `false`
+  - burn: `$0.00/h`
+- The shared bucket is the runtime state surface:
+  - `tostido/Ouroboros-storage`
+  - farm root: `/data/farm`
+
+## What Is Already Verified
+
+### 1) Data-plane payload fix is correct
+
+The new object-reference flow is coherent end-to-end:
+
+`KEY pack -> job submit -> object externalization -> objects/<sha>.pkl.gz -> worker unpack -> SHA verify -> pickle.loads`
+
+Verified seams:
+
+- `key_farm/contracts.py`
+- `key_farm/serialization.py`
+- `key_farm/store.py`
+- `key_farm/controller.py`
+- `key_farm/worker.py`
+- `key_farm/evaluator.py`
+
+Result:
+
+- Large payloads are externalized as files under `objects/<sha>.pkl.gz`.
+- Job JSON keeps a small reference instead of embedding the full payload.
+- The worker still supports the legacy inline base64 path.
+
+### 2) KEY is wired into the farm
+
+The evolution loop calls the farm path first and only falls back locally on failure.
+
+Relevant behavior:
+
+- `PopulationManager.evaluate()` routes through the CPU farm path.
+- `fallback_to_local: true` means farm issues slow KEY, but do not break it.
+- This is why the system was slow, not dead.
+
+### 3) H2 partial-generation fix is in place
+
+The previous all-or-nothing behavior was the expensive stall:
+
+- one bad or slow node could force a full generation timeout
+- then the whole generation would be recomputed locally
+
+That path has been improved so partial results are preserved and only missing nodes are filled locally.
+`PopulationManager` already stores `self.fitness_fn` in `__init__`, so the local fill path is real and only applies to missing nodes, not the full generation.
+
+### 4) MeshScale control-plane logic is real
+
+The OS route surface exists and is useful as a steering layer:
+
+- status
+- plan
+- trace
+- regulate
+- scale
+- bucket routing
+
+This is the right place for guidance and telemetry.
+It is not the place to do the compute.
+
+## Current Constraints That Still Matter
+
+### H1 - KEY and worker-template must stay in lockstep
+
+Payloads are pickled genome objects. That makes the worker-template sensitive to code drift in `run.py` / genome classes.
+
+If KEY rebuilds and the worker-template does not, or vice versa, the unpickle path can break or silently build the wrong object.
+
+Practical implication:
+
+- rebuild KEY and worker-template together
+- do not treat them as independently safe deploys
+
+### M1 - Long evals can still threaten pulse liveness
+
+Worker pulses are not continuous during evaluation.
+
+If a node runs longer than the pulse/claim assumptions, the system can look stale and either requeue or delay work.
+
+This is not a full failure, but it is a real efficiency leak.
+
+### M2 - Object storage can grow without bound
+
+The new `objects/` path is content-addressed and deduped, but there is no GC by default.
+
+If runs keep producing new genomes, the bucket will grow monotonically unless a retention policy is added.
+
+### M3 - Trace/dashboard are not run-scoped
+
+Historical files are still counted in trace summaries.
+
+That means old large files can keep the dashboard looking noisy even when a fresh run is clean.
+
+## Projected Trajectory
+
+### Best-case near term
+
+If the next fresh run starts after KEY and worker-template are rebuilt together:
+
+- new job files stay small
+- object refs appear under `/data/farm/objects/`
+- MeshScale can scale only while there is actual backlog
+- after backlog drains, jobs cool back to zero
+- KEY spends time on model work instead of moving huge JSON blobs around
+
+This is the likely path to visible speedup.
+
+### More realistic near term
+
+The data-plane fix should remove the worst overhead immediately, but the evolution loop will still be limited by actual fitness computation.
+
+So the improvement will probably look like:
+
+- less time lost to storage churn
+- fewer stall/recompute events
+- cleaner scaling behavior
+- not instant magic throughput
+
+### Failure mode if we push too hard
+
+If batch size is raised before shard-level externalization exists:
+
+- per-node payloads can bloat the JSON again
+- the original bottleneck comes back in a different shape
+
+If KEY and worker-template drift:
+
+- pickle compatibility becomes the first failure
+
+If long evals exceed pulse assumptions:
+
+- claims can look stale and work can be duplicated
+
+## What Claude Should Audit Next
+
+1. Confirm the next fresh run was actually submitted after rebuild.
+2. Inspect the fresh run tree in the bucket for:
+   - small job JSON
+   - object refs
+   - new `objects/<sha>.pkl.gz` files
+3. Verify the worker-template revision matches KEY-side genome/schema expectations.
+4. Check whether dashboard/trace should be run-scoped before the next long run.
+5. Decide whether `target_batch_size` should stay at `1` or move to an adaptive policy after the object-ref path proves stable.
+
+## Recommended Operator Order
+
+1. Keep the worker headless.
+2. Use MeshScale as the control plane, not the compute plane.
+3. Validate a fresh run after lockstep rebuild.
+4. Only raise batch size after the object-ref path is proven clean at scale.
+5. Add retention and run-scoping before very long unattended runs.
+
+## Evidence Anchors
+
+- `config.json`
+- `population.py`
+- `key_farm/contracts.py`
+- `key_farm/serialization.py`
+- `key_farm/store.py`
+- `key_farm/controller.py`
+- `key_farm/evaluator.py`
+- `key_farm/worker.py`
+- `key_farm/scaler.py`
+- `meshscale_space/app.py`
+- `meshscale_space/os_routes.py`
+- `meshscale_worker_space/app.py`
+
+## Short Answer For Claude
+
+The system is not blocked on rendering. It is blocked on verifying that the fresh run is using the new payload path and on keeping KEY/worker-template synchronized. The control plane is usable now; the main remaining work is proving the next run is clean and then deciding whether batch size should be increased.

key_farm/__init__.py

@@ -5,7 +5,7 @@ fitness-evaluation shards to CPU workers.
 """
 
 from .contracts import FarmPolicy, RegulationIntent, ShardJob, ShardResult
-from .controller import FarmController
+from .controller import FarmController, IncompleteGenerationError
 from .evaluator import get_handler, register_handler
 from .faculty import farm_faculty_manifest, load_cocoon_faculty
 from .scaler import MeshScaleScaler, ScalePolicy
@@ -14,6 +14,7 @@ from .symbiotic_trace import symbiotic_trace
 
 __all__ = [
     "FarmController",
+    "IncompleteGenerationError",
     "FarmPolicy",
     "FarmStore",
     "MeshScaleScaler",

key_farm/controller.py

@@ -17,6 +17,27 @@ from .serialization import pack_item, pack_node
 from .store import FarmStore
 
 
+class IncompleteGenerationError(RuntimeError):
+    """Raised when a farm generation times out before all node ids return."""
+
+    def __init__(
+        self,
+        message: str,
+        *,
+        run_id: str,
+        generation: int,
+        partial_results: dict[str, float],
+        missing_node_ids: list[str],
+        timeout_seconds: float,
+    ) -> None:
+        super().__init__(message)
+        self.run_id = run_id
+        self.generation = generation
+        self.partial_results = dict(partial_results)
+        self.missing_node_ids = list(missing_node_ids)
+        self.timeout_seconds = float(timeout_seconds)
+
+
 class FarmController:
     """Controller-side API for dispatching fitness evaluation shards."""
 
@@ -229,7 +250,14 @@ class FarmController:
                 return fitness_by_id
             if time.time() >= deadline:
                 missing = sorted(expected_node_ids - fitness_by_id.keys())
-                raise TimeoutError(f"missing {len(missing)} fitness results: {missing[:10]}")
+                raise IncompleteGenerationError(
+                    f"missing {len(missing)} fitness results: {missing[:10]}",
+                    run_id=run_id,
+                    generation=generation,
+                    partial_results=fitness_by_id,
+                    missing_node_ids=missing,
+                    timeout_seconds=timeout_seconds,
+                )
             time.sleep(poll_seconds)
 
     def result_map(self, run_id: str, generation: int) -> dict[str, float]:

meshscale_worker_space/Dockerfile

@@ -0,0 +1,18 @@
+FROM python:3.11-slim
+
+ENV PYTHONUNBUFFERED=1
+ENV PORT=7860
+ENV KEY_FARM_ROOT=/data/farm
+ENV KEY_FARM_FORMATION=cpu-worker
+ENV MESHSCALE_FULL_KEY_REQUIREMENTS=1
+
+WORKDIR /app
+COPY . /app
+
+RUN python -m pip install --upgrade pip \
+    && python -m pip install --no-cache-dir -r meshscale_worker_space/requirements.txt \
+    && if [ "$MESHSCALE_FULL_KEY_REQUIREMENTS" = "1" ] && [ -f requirements.txt ]; then \
+        python -m pip install --no-cache-dir -r requirements.txt; \
+    fi
+
+CMD ["python", "-m", "uvicorn", "meshscale_worker_space.app:app", "--host", "0.0.0.0", "--port", "7860"]

meshscale_worker_space/README.md

@@ -0,0 +1,51 @@
+---
+title: MeshScale CPU Worker
+emoji: 🔩
+colorFrom: blue
+colorTo: green
+sdk: docker
+app_port: 7860
+pinned: false
+---
+
+# MeshScale CPU Worker
+
+This Space template runs one MeshScale CPU worker against a shared Hugging Face
+bucket mounted at `/data`.
+
+Required Space variable:
+
+```text
+KEY_FARM_ROOT=/data/farm
+```
+
+Useful Space variables:
+
+```text
+KEY_FARM_FORMATION=cpu-worker
+KEY_FARM_POLL_SECONDS=2
+KEY_FARM_TARGET_BATCH_SIZE=24
+KEY_FARM_MAX_WORKERS=24
+KEY_FARM_BUDGET_HOURLY_USD=0.72
+MESHSCALE_WORKER_AUTOSTART=1
+```
+
+`KEY_FARM_WORKER_ID` is optional. When replicas are enabled, the worker app
+adds the runtime replica/container identity to the base worker name so each
+replica publishes a distinct pulse.
+
+The worker claims shard files from:
+
+```text
+/data/farm/runs/<run_id>/generations/<generation>/jobs/pending/
+```
+
+It exits its worker loop when the controller pulse is lost or when regulation
+mode becomes `shutdown`.
+
+Health endpoints:
+
+```text
+/health
+/state
+```

meshscale_worker_space/__init__.py

@@ -0,0 +1 @@
+"""Hugging Face Space app template for MeshScale CPU workers."""

meshscale_worker_space/app.py

@@ -0,0 +1,189 @@
+from __future__ import annotations
+
+import os
+import threading
+import time
+import traceback
+from typing import Any
+
+from fastapi import FastAPI
+
+from key_farm.contracts import FarmPolicy
+from key_farm.store import FarmStore
+from key_farm.worker import worker_loop
+
+
+app = FastAPI(title="MeshScale CPU Worker")
+
+_worker_thread: threading.Thread | None = None
+_supervisor_thread: threading.Thread | None = None
+_worker_error: str | None = None
+_worker_lock = threading.Lock()
+
+
+def _int_env(name: str, default: int) -> int:
+    try:
+        return int(os.environ.get(name, str(default)))
+    except ValueError:
+        return default
+
+
+def _float_env(name: str, default: float) -> float:
+    try:
+        return float(os.environ.get(name, str(default)))
+    except ValueError:
+        return default
+
+
+def _bool_env(name: str, default: bool) -> bool:
+    raw = os.environ.get(name)
+    if raw is None:
+        return default
+    return raw.strip().lower() not in {"0", "false", "no", "off"}
+
+
+def _root() -> str:
+    return os.environ.get("KEY_FARM_ROOT", "/data/farm")
+
+
+def _formation() -> str:
+    return os.environ.get("KEY_FARM_FORMATION", "cpu-worker")
+
+
+def _worker_id() -> str:
+    base = os.environ.get("KEY_FARM_WORKER_ID") or os.environ.get("SPACE_ID") or "meshscale-worker"
+    runtime = (
+        os.environ.get("SPACE_REPLICA_ID")
+        or os.environ.get("REPLICA_ID")
+        or os.environ.get("HOSTNAME")
+        or str(os.getpid())
+    )
+    return f"{base}:{runtime}"
+
+
+def _policy() -> FarmPolicy:
+    formation = _formation()
+    return FarmPolicy(
+        target_batch_size=_int_env("KEY_FARM_TARGET_BATCH_SIZE", 24),
+        min_batch_size=_int_env("KEY_FARM_MIN_BATCH_SIZE", 4),
+        max_depth=_int_env("KEY_FARM_MAX_DEPTH", 4),
+        max_workers=_int_env("KEY_FARM_MAX_WORKERS", 24),
+        claim_ttl_seconds=_int_env("KEY_FARM_CLAIM_TTL_SECONDS", 900),
+        pulse_ttl_seconds=_int_env("KEY_FARM_PULSE_TTL_SECONDS", 120),
+        controller_pulse_ttl_seconds=_int_env("KEY_FARM_CONTROLLER_PULSE_TTL_SECONDS", 180),
+        budget_hourly_usd=_float_env("KEY_FARM_BUDGET_HOURLY_USD", 0.72),
+        cpu_space_hourly_usd=_float_env("KEY_FARM_CPU_SPACE_HOURLY_USD", 0.03),
+        exit_when_pulse_lost=_bool_env("KEY_FARM_EXIT_WHEN_PULSE_LOST", True),
+        pulse_rate=_float_env("KEY_FARM_PULSE_RATE", 1.0),
+        allowed_formations=(formation,),
+    )
+
+
+def _run_worker() -> None:
+    global _worker_error
+    try:
+        worker_loop(
+            FarmStore(_root()),
+            _worker_id(),
+            _policy(),
+            poll_seconds=_float_env("KEY_FARM_POLL_SECONDS", 2.0),
+            once=False,
+        )
+    except Exception:
+        _worker_error = traceback.format_exc()
+
+
+def _autostart_enabled() -> bool:
+    return _bool_env("MESHSCALE_WORKER_AUTOSTART", True)
+
+
+def _controller_ready(store: FarmStore | None = None, policy: FarmPolicy | None = None) -> bool:
+    if not _bool_env("MESHSCALE_WORKER_WAIT_FOR_CONTROLLER", True):
+        return True
+    store = store or FarmStore(_root())
+    policy = policy or _policy()
+    return store.controller_pulse_alive(policy.controller_pulse_ttl_seconds)
+
+
+def _start_worker_if_ready() -> bool:
+    global _worker_thread, _worker_error
+    if not _autostart_enabled():
+        return False
+    if _worker_error and not _bool_env("MESHSCALE_WORKER_RESTART_ON_ERROR", False):
+        return False
+    store = FarmStore(_root())
+    policy = _policy()
+    if not _controller_ready(store, policy):
+        return False
+    with _worker_lock:
+        if _worker_thread and _worker_thread.is_alive():
+            return True
+        _worker_error = None
+        _worker_thread = threading.Thread(target=_run_worker, name="meshscale-worker", daemon=True)
+        _worker_thread.start()
+        return True
+
+
+def _supervise_worker() -> None:
+    while _autostart_enabled():
+        _start_worker_if_ready()
+        time.sleep(_float_env("MESHSCALE_WORKER_SUPERVISOR_SECONDS", 5.0))
+
+
+def _ensure_supervisor_started() -> None:
+    global _supervisor_thread
+    if not _autostart_enabled():
+        return
+    if _supervisor_thread and _supervisor_thread.is_alive():
+        return
+    _supervisor_thread = threading.Thread(
+        target=_supervise_worker,
+        name="meshscale-worker-supervisor",
+        daemon=True,
+    )
+    _supervisor_thread.start()
+
+
+def _ensure_worker_started() -> None:
+    if not _bool_env("MESHSCALE_WORKER_AUTOSTART", True):
+        return
+    _ensure_supervisor_started()
+    _start_worker_if_ready()
+
+
+@app.on_event("startup")
+def startup() -> None:
+    _ensure_worker_started()
+
+
+@app.get("/health")
+def health() -> dict[str, Any]:
+    store = FarmStore(_root())
+    policy = _policy()
+    _ensure_worker_started()
+    return {
+        "ok": _worker_error is None,
+        "worker_id": _worker_id(),
+        "formation": _formation(),
+        "thread_alive": bool(_worker_thread and _worker_thread.is_alive()),
+        "supervisor_alive": bool(_supervisor_thread and _supervisor_thread.is_alive()),
+        "root": str(store.root),
+        "controller_alive": store.controller_pulse_alive(policy.controller_pulse_ttl_seconds),
+        "regulation": store.read_regulation().to_dict(),
+        "error": _worker_error,
+    }
+
+
+@app.get("/state")
+def state() -> dict[str, Any]:
+    store = FarmStore(_root())
+    policy = _policy()
+    _ensure_worker_started()
+    return {
+        "worker_id": _worker_id(),
+        "formation": _formation(),
+        "pending_jobs": len(store.pending_jobs(_formation())),
+        "active_workers": store.active_worker_count(policy.pulse_ttl_seconds, _formation()),
+        "controller_alive": store.controller_pulse_alive(policy.controller_pulse_ttl_seconds),
+        "policy": policy.to_dict(),
+    }

meshscale_worker_space/requirements.txt

@@ -0,0 +1,5 @@
+fastapi>=0.110
+uvicorn[standard]>=0.27
+huggingface-hub>=0.34.0,<1.0
+numpy>=1.24,<1.27
+simpleeval>=0.9.13