docs/research/DILOCO_SERVERLESS_RECONNAISSANCE.md

DiLoCo Serverless Executor Reconnaissance

Status: Reconnaissance complete (feeds ADR-005). Audience: ADR-005 author + framework integrator wiring composer_replication.diloco.serverless against real backends. Scope: Decoupled DiLoCo across N independently-scheduled serverless GPU jobs. NOT a generic "serverless training" survey. Date: 2026-05-26.

---

TL;DR

Executor	Inter-job net?	Cold start	$/A100·hr (1×)	$/H100·hr (1×)	Max // jobs	Ranking for Decoupled DiLoCo
Modal	✅ i6pn + @modal.experimental.clustered (50 Gbps + RDMA up to 3.2 Tbps); also same-workspace TCP via shared Dict/Queue	~1–10 s warm-boot; ≤90 s incl. image pull on first run	A100-40GB: $2.10; A100-80GB: $2.50	H100: $3.95	Workspace quota; Starter ≤10 GPU containers, Team much higher (contact)	★★★★★ primary adapter
HF Jobs	❌ No documented inter-job networking. Workaround: object store (HF Hub bucket / dataset / S3)	"starting" → "running" billed; per-min granularity; typical scheduling 10–60 s	A100-80GB: $2.50 (a100-large); 4×: $10.00; 8×: $20.00	H200: $5.00; 8×H200: $40.00 (no H100 SKU)	Pro/Team/Enterprise quota; not publicly capped per-run (parallel via SDK loop)	★★★★☆ secondary adapter; pseudo-grad via Hub bucket Volume
AWS SageMaker Training Jobs	✅ Inside one job's multi-instance cluster (EFA/SMDDP). ❌ Across separate CreateTrainingJob invocations — same workaround as HF	Image pull + EBS attach: typically 2–5 min cold; warm pools cut to ~10 s for ≤60 min	ml.p4d.24xlarge ≈ $32.77/hr (8×A100-40GB) ≈ $4.10/A100·hr	ml.p5.48xlarge ≈ $98.32/hr ≈ $12.29/H100·hr	Account quota (typical 4–20 instances; raise via Service Quotas)	★★★☆☆ good for one big "fragment"; clunky as N-replicas-of-1-GPU
GCP Vertex AI Custom Jobs	✅ Inside one CustomJob's worker pools (gRPC/MPI). ❌ Across separate jobs — same workaround	2–6 min typical cold	a2-highgpu-1g (1×A100-40GB) ≈ $3.67/hr (incl. Vertex training premium ~30–50%)	a3-highgpu-8g ≈ $88/hr ≈ $11/H100·hr	Per-region GPU quota	★★☆☆☆ highest premium per GPU; useful as 3rd region
Azure ML Command Jobs	✅ within instance_count>1 (InfiniBand on ND*-series). ❌ across jobs — same workaround	3–8 min typical cold (image cache → curated env helps)	NC24ads_A100_v4 (1×A100-80GB): ~$3.67/hr (PAYG list)	ND96isr_H100_v5 (8×H100): ~$98/hr ≈ $12.25/H100·hr	Per-region quota, surcharge $0/core (only VM+disk)	★★☆☆☆ like Vertex; useful only if user already lives in Azure
k8s + Volcano / KubeRay	✅ if cluster networked. Volcano gang-schedules RayJob/MPIJob; pods see each other on cluster network	Pod schedule: seconds–minutes (image cache, GPU availability)	Whatever the underlying cluster pays (e.g. spot A100 ~$1–2/hr on RunPod / Lambda / OCI K8s)	Same	Cluster capacity	★★★★☆ best price/perf if user owns/leases a cluster; ops cost nontrivial
RunPod (honourable mention)	✅ same DC; no documented federation	seconds	~$1.19/hr A100-80GB community, ~$2.17/hr secure	~$1.99/hr H100 community, ~$4.18/hr secure	Account quota	★★★☆☆ — not in the candidate list but a strong third adapter for cost

The Decoupled DiLoCo framing kills the "must have inter-job allreduce" requirement: per the original DiLoCo paper (arXiv:2311.08105 §3.2), pseudo-gradients are exchanged once every H = 500–1000 inner steps, totalling KB-to-MB of gradient data per round. Bandwidth is irrelevant; latency is irrelevant; the only requirement is "all N replicas can read & write a shared blob store." That makes object-storage-based pseudo-gradient exchange the correct default, and the Modal clustered-style RDMA fabric a bonus you can opt into when a single executor runs ≥2 replicas in the same region.

Recommendation: ship the framework with two adapters — ModalExecutor and HFJobsExecutor — both speaking the same Executor ABC, both using object-store pseudo-grad exchange by default. Add a third adapter (RunPodExecutor or K8sExecutor) when a user needs it.

---

1. Why Decoupled DiLoCo over the network is easy

From DiLoCo (Douillard et al., DiLoCo: Distributed Low-Communication Training of Language Models, arXiv:2311.08105):

• Setup. N "workers" each train a full local copy of the model with an inner optimizer (AdamW, LR 4e-4, etc.) on disjoint shards of data.
• Outer round (every H=500 steps in the paper, often 1000 in follow-ups). Each worker computes its pseudo-gradient δ_k = θ_initial − θ_local (the negative of its accumulated local update). The N workers all-reduce the pseudo-gradient, average it, and the outer optimizer (Nesterov SGD, lr=0.7, momentum=0.9) applies it to θ_initial to produce θ_initial^(t+1). Workers reset to that.
• Communication budget per round. One full-model parameter tensor per worker (FP32, fp16, or bf16). For a 1B model in bf16, that's ~2 GB per worker per round. For Streaming DiLoCo (Liu et al. 2025) the communication is sliced into fragments and overlapped with compute, but the aggregate per round is the same.
• Communication frequency. Once per H=500–1000 inner steps. With one inner step ≈ 1–3 s on a single A100/H100 for a 7B model, that's one outer round every ~10–30 minutes wall-clock.

The implication: the outer-loop "allreduce" is a one-shot 2–10 GB upload+download every 10+ minutes. It does not need NCCL. It does not need RDMA. It does not even need TCP between the replicas. An S3 PutObject followed by N GetObjects is sufficient. Cross-region transfer at 1 Gbps moves 2 GB in ~17 s; even at 100 Mbps it's ~3 min — small compared to the H=500 inner-step interval. This is the key insight that makes "Modal + HuggingFace Jobs as DiLoCo replicas" actually a sensible architecture rather than a hack.

We codify this in the framework with two communication backends:

1. InProcessAllReduce — what composer_replication.diloco already uses (torchft Manager mock). For unit tests and same-process/same-host runs.
2. ObjectStoreAllReduce — barriers + pseudo-grad averaging via S3/GCS/HF Hub bucket. New code for ADR-005. Expected per-round overhead 20–60 s for a 7B model — already amortised over 10–30 min of compute.

The torchft Manager interface (used by torchft.local_sgd.DiLoCo) only requires .allreduce(tensor) → Work, .should_commit(), .start_quorum(), .current_step(). We implement .allreduce on top of object storage. Done.

---

2. Per-executor audit

2.1 Modal — primary adapter

Inter-job networking. Yes, in two flavours.

• @modal.experimental.clustered(size=N, rdma=True): gang-schedules N containers in the same Modal cluster, gives them i6pn IPv6 addresses, and (with rdma=True) provisions InfiniBand RoCE up to 3,200 Gbps for inter-node communication. (modal.com/docs/guide/multi-node-training). This is the right primitive for a single-executor multi-replica DiLoCo where all N replicas live on Modal.
• i6pn private network (modal.com/docs/guide/private-networking): any two @app.function(i6pn=True) containers in the same workspace+region can address each other over a 50 Gbps IPv6 fabric. Region-scoped — Modal documents that "i6pn networking is region-scoped functionality."

Cross-executor: for the cross-cloud Decoupled DiLoCo case (Modal + HF + …), Modal containers reach out to S3/HF Hub/GCS like any other internet-connected workload. No Modal-specific magic needed.

Cold start. Modal's container infra warm-boots in 1 s for a cached image; first-run pulls of a large PyTorch image dominate (30–90 s). HF model download adds 15–45 s for a 7B model from cold (cache on a modal.Volume after run 1). See MODAL_RECONNAISSANCE.md §1.3 in this repo for the same numbers from a different audit angle. Realistic per-run cold: **60–120 s** on first launch, ~10–30 s on subsequent launches with warm image cache.

$/GPU·hr (from https://modal.com/pricing, on-demand, base region, preemptible default).

GPU	Modal gpu= string	$/sec	$/hour
A100-40GB	"A100-40GB"	0.000583	$2.099
A100-80GB	"A100-80GB"	0.000694	$2.498
H100 (pinned)	"H100!"	0.001097	$3.949
H200	"H200"	(see pricing page)	~$4.5–5/hr per the published table
B200	"B200"	—	~$6/hr per the published table

Multipliers from same pricing page: region pinning 1.5–1.75×, non-preemptible 3×. Default is preemptible — for DiLoCo this is fine: a preempted replica retries, the outer loop tolerates an absent-this-round member by simply averaging over the survivors.

Max concurrent jobs. Modal documents "default limits on Modal free tier" of 10 GPU containers in the Blender example (max_containers=10 if WITH_GPU else 100). Paid plans scale far higher; clustered functions starting May 31, 2026 require 8 GPUs/node, capping at "up to 64 devices" per cluster (@clustered). Practically, for 8 single-A100 replicas of Decoupled DiLoCo, the Starter plan is limiting; Team plan ≥10 paid GPU containers handles it. Contact Modal support for >64-GPU clusters.

Verified API for spinning up N parallel jobs (verified pattern from modal-examples and Modal docs):

# composer_replication/diloco/serverless/_modal_adapter.py
import modal

app = modal.App("diloco-replicas")
image = (
    modal.Image.debian_slim(python_version="3.11")
    .uv_pip_install("torch", "transformers", "torchft-nightly")
    .add_local_python_source("composer_replication")
)

@app.function(image=image, gpu="A100-40GB", timeout=60 * 60 * 24)
def run_inner_loop(replica_id: int, rendezvous_uri: str, config: dict):
    """One DiLoCo replica. Trains for N inner steps, then participates in
    one outer-round pseudo-gradient exchange via the rendezvous_uri (S3 path),
    repeats."""
    from composer_replication.diloco.serverless import run_replica
    return run_replica(replica_id=replica_id,
                       rendezvous_uri=rendezvous_uri,
                       **config)

@app.local_entrypoint()
def main(num_replicas: int = 4):
    rendezvous_uri = "s3://my-bucket/diloco-run-2026-05-26/"
    config = {"model": "Qwen/Qwen2.5-7B", "outer_rounds": 100, "sync_every": 500}
    # .map / .starmap fans out N parallel container invocations.
    args = [(i, rendezvous_uri, config) for i in range(num_replicas)]
    results = list(run_inner_loop.starmap(args))
    print(f"All {num_replicas} replicas completed: {results}")

For the single-executor RDMA case (all N on Modal in one region, max throughput):

@app.function(gpu="H100:8", timeout=60 * 60 * 24)
@modal.experimental.clustered(size=4, rdma=True)
def diloco_cluster_train(rendezvous_uri: str, config: dict):
    info = modal.experimental.get_cluster_info()
    # info.rank is our DiLoCo replica id; info.container_ips[0] is rank-0.
    return run_replica(replica_id=info.rank, rendezvous_uri=rendezvous_uri, **config)

Right abstraction layer for the framework. Modal Functions map to one DiLoCo replica each. The local entrypoint (or our Executor.launch_replicas()) does .starmap to fan out N. Inter-replica state lives in S3 (default) or in Modal-side modal.Dict / modal.Queue (faster, same-workspace only). The @clustered decorator is not required for Decoupled DiLoCo — it's an opt-in optimization for when you want one Modal cluster to be your whole training run.

Rough $-per-replica-hour for an A100-40GB single-replica Modal run (no clustering): 1 × $2.099 + ~$0.05 CPU/RAM overhead + ~$0.005 networking ≈ $2.16/hr/replica.

2.2 HuggingFace Jobs — secondary adapter

Inter-job networking. No documented inter-job networking primitive. HF Jobs is a Docker-Image-+-command service (huggingface.co/docs/hub/en/jobs) modelled after docker run. There is no "address my peer job" API. Each job runs in its own pod with internet egress only; HF does not advertise a private VPC network.

Workaround (the right one for DiLoCo). HF Jobs supports Volume mounts of HF Hub repos and HF storage buckets (huggingface.co/docs/huggingface_hub/en/guides/jobs):

from huggingface_hub import run_job, Volume
checkpoints_bucket = Volume(type="bucket", source="myorg/diloco-rendezvous", mount_path="/rendezvous")
job = run_job(image="pytorch/pytorch:2.6.0-cuda12.4-cudnn9-devel",
              command=["python", "/code/run_replica.py", "--replica-id", "0"],
              flavor="a100-large",
              timeout="6h",
              volumes=[checkpoints_bucket])

The bucket volume is read+write by default — perfect for object-store-based pseudo-gradient exchange. This is exactly the same workaround we'd apply to SageMaker, Vertex AI, Azure ML — but on HF it's first-class because Volume(type="bucket", ...) is built into the API.

Cold start. HF docs say "billing only when starting or running" — no charge during build. Empirically (per the HF quickstart logs), hf jobs uv run reports a state transition created → starting → running typically in 10–60 s for a cached image, longer for first-pull of a large CUDA image. The default timeout is 30 minutes; use timeout="6h" or similar for DiLoCo.

$/GPU·hr (from https://huggingface.co/docs/hub/jobs-pricing; per-minute billing).

Hardware flavor	Hourly	$/A100·hr	$/H100/H200·hr
a100-large (1× A100 80GB)	$2.50	$2.50	—
4xa100-large (4× A100 80GB)	$10.00	$2.50	—
8xa100-large (8× A100 80GB)	$20.00	$2.50	—
h200 (1× H200 141GB)	$5.00	—	$5.00 (H200, not H100)
4xh200	$20.00	—	$5.00
8xh200	$40.00	—	$5.00
l40sx1	$1.80	—	—
a10g-large	$1.50	—	—
t4-small	$0.40	—	—

No H100 SKU is published as of this write — HF jumps from A100→H200. Treat HF's "$5/hr H200" as the H100-equivalent line item.

Max concurrent jobs. HF documents "Jobs are available to any user or organization with a positive credit balance" but doesn't publish a per-account concurrency cap. The Python SDK pattern in their docs:

# Verified — direct from huggingface.co/docs/huggingface_hub/en/guides/jobs
jobs = [run_job(image=image, command=command) for command in commands]
for job in jobs:
    while inspect_job(job_id=job.id).status.stage not in ("COMPLETED", "ERROR"):
        time.sleep(10)

…clearly assumes a "spawn N, poll N" model. Empirically, Pro accounts can run several jobs in parallel; Enterprise plans are higher.

Verified API for spinning up N parallel jobs:

# composer_replication/diloco/serverless/_hf_jobs_adapter.py
from huggingface_hub import run_job, run_uv_job, inspect_job, fetch_job_logs, Volume

def spawn_diloco_replica(replica_id: int, num_replicas: int, rendezvous_repo: str):
    return run_job(
        image="pytorch/pytorch:2.6.0-cuda12.4-cudnn9-devel",
        command=["python", "-m", "composer_replication.diloco.serverless.replica_entrypoint",
                 "--replica-id", str(replica_id),
                 "--num-replicas", str(num_replicas),
                 "--rendezvous-uri", "/rendezvous"],
        flavor="a100-large",
        timeout="12h",
        env={"HF_HUB_ENABLE_HF_TRANSFER": "1"},
        secrets={"HF_TOKEN": "<token>"},
        volumes=[Volume(type="bucket", source=rendezvous_repo, mount_path="/rendezvous")],
    )

def spawn_n(num_replicas: int, rendezvous_repo: str = "myorg/diloco-rendezvous-2026-05-26"):
    jobs = [spawn_diloco_replica(i, num_replicas, rendezvous_repo) for i in range(num_replicas)]
    return jobs  # list[JobInfo]

The Volume(type="bucket", ...) is the secret weapon. Each replica writes its pseudo-gradient to a unique key under /rendezvous/round-{t}/replica-{i}.pt, then waits on a barrier file (busy-loop on os.path.exists with sleeps). The leader rank averages and writes /rendezvous/round-{t}/avg.pt. Standard object-store DiLoCo pattern.

Right abstraction. Same as Modal: one run_job = one DiLoCo replica. Fan-out via list comprehension. No special multi-node primitive — and we don't need one for Decoupled DiLoCo.

2.3 AWS SageMaker Training Jobs

Inter-job networking. SageMaker has intra-job multi-node networking (InstanceCount > 1 provisions a single EFA/InfiniBand-connected cluster, suitable for SMDDP AllReduce with pytorchddp or torch_distributed launchers — see docs.aws.amazon.com/sagemaker/latest/dg/data-parallel-framework-estimator.html). It does not have inter-job networking — two separate CreateTrainingJob calls produce two isolated VPCs (unless you wire a shared customer VPC, which is non-trivial and Decoupled DiLoCo doesn't benefit from anyway).

Workaround. S3. Each SageMaker training job has read+write access to S3 by default (via the IAM role passed to CreateTrainingJob). Pseudo-gradient exchange via s3://bucket/diloco-run/round-{t}/replica-{i}.pt is straightforward.

Cold start. SageMaker docs and the cost-optimization blog post acknowledge five phases: Starting, Downloading, Training, Uploading, Completed. The Starting+Downloading phases are the cold start and typically take 2–5 minutes: image pull from ECR, EBS volume attach, boto3 IAM role fetch, container init. Warm pools (docs.aws.amazon.com/sagemaker/latest/dg/train-warm-pools.html) cut subsequent matching jobs to ~10 s by retaining the cluster up to KeepAlivePeriodInSeconds (max 3600 s = 60 min) — but matching requires identical RoleArn/InstanceType/InstanceCount/VpcConfig, so warm pools work for "rerun the same DiLoCo replica config" but not for heterogeneous fleets.

$/GPU·hr (from aws.amazon.com/sagemaker/ai/pricing/, training tab, US East regions; per-second billing). SageMaker training instances carry a ~20–25% premium over raw EC2 because the service includes managed orchestration. Pricing varies by region; representative US East values:

Instance	GPUs	$/hr (training)	$/GPU·hr
ml.p4d.24xlarge	8× A100-40GB	≈ $32.77	≈ $4.10/A100·hr
ml.p4de.24xlarge	8× A100-80GB	≈ $40.97	≈ $5.12/A100·hr
ml.p5.48xlarge	8× H100-80GB	≈ $98.32	≈ $12.29/H100·hr
ml.g5.48xlarge	8× A10G-24GB	≈ $10.18 (per HyperPod example)	≈ $1.27/A10G·hr

(Hourly rates above are training rates inferred from SageMaker's published training-tab price calculator and the HyperPod ml.g5.24xlarge $10.18/hr example; consult the live pricing page in aws.amazon.com/sagemaker/ai/pricing/ for region-specific quotes. Managed Spot Training (docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateTrainingJob.html) cuts up to 80–90% — and DiLoCo tolerates spot well because outer round t can simply skip preempted replicas.)

Per-A100 / per-H100 rates are the highest of any executor in this audit. SageMaker is a poor choice for cost-sensitive Decoupled DiLoCo unless you already have committed savings plans or run on Spot.

Max concurrent jobs. AWS Service Quotas: per-account default is typically 4 (for ml.p4d.24xlarge) and 0 (for ml.p5.48xlarge — must request access). Both are raisable. There's a soft cap of 1000 active training jobs per account.

Verified API for spinning up N parallel jobs (using boto3, since sagemaker Python SDK abstracts away the parallel-launch case):

# composer_replication/diloco/serverless/_sagemaker_adapter.py
import boto3

sm = boto3.client("sagemaker", region_name="us-east-1")

def spawn_diloco_replica(replica_id: int, num_replicas: int, s3_rendezvous: str):
    return sm.create_training_job(
        TrainingJobName=f"diloco-replica-{replica_id}-{int(time.time())}",
        AlgorithmSpecification={
            "TrainingImage": "763104351884.dkr.ecr.us-east-1.amazonaws.com/pytorch-training:2.4.0-gpu-py311-cu124-ubuntu22.04-sagemaker",
            "TrainingInputMode": "File",
            "ContainerEntrypoint": ["python", "-m", "composer_replication.diloco.serverless.replica_entrypoint"],
            "ContainerArguments": ["--replica-id", str(replica_id),
                                    "--num-replicas", str(num_replicas),
                                    "--rendezvous-uri", s3_rendezvous],
        },
        ResourceConfig={
            "InstanceCount": 1,                       # one A100/H100 per replica
            "InstanceType": "ml.p4d.24xlarge",
            "VolumeSizeInGB": 200,
            "KeepAlivePeriodInSeconds": 1800,         # warm pool for fast subsequent launches
        },
        OutputDataConfig={"S3OutputPath": f"{s3_rendezvous}/output/replica-{replica_id}/"},
        StoppingCondition={"MaxRuntimeInSeconds": 24*3600},
        RoleArn="arn:aws:iam::ACCOUNT:role/SageMakerExecutionRole",
        EnableManagedSpotTraining=True,                # 80%+ savings, DiLoCo-tolerant
    )

def spawn_n(num_replicas: int):
    s3_rendezvous = "s3://my-diloco-bucket/run-2026-05-26"
    return [spawn_diloco_replica(i, num_replicas, s3_rendezvous) for i in range(num_replicas)]

(The CreateTrainingJob API spec is documented in full at docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateTrainingJob.html.)

Right abstraction. Same shape: 1 training job = 1 DiLoCo replica. SageMaker's intra-job multi-node features (SMDDP, EFA, instance_count=8) are wasted if our framing is "N independent replicas"; they only help if a single replica is itself FSDP-sharded across instances, which we explicitly don't want for v0.x.

2.4 GCP Vertex AI Custom Jobs

Inter-job networking. Same story as SageMaker: a single CustomJob can have multiple workerPoolSpecs (chief, workers, parameter servers, evaluator) on a private VPC; separate CustomJobs are isolated. Workaround: GCS bucket. Vertex's configure-compute doc covers single-node and multi-replica configurations for one job.

Cold start. Typical 2–6 min for cold image pull + VM provision. Vertex caches images in Artifact Registry; subsequent jobs in the same region with the same custom container start faster (~30–60 s).

$/GPU·hr. Vertex AI training prices = (Compute Engine VM rate) × (Vertex training premium ≈ 30–50%). From the Vertex Training SKU groups page (cloud.google.com/skus/sku-groups/vertex-training) the SKUs include "Training - NVIDIA A100 80GB in Virginia" etc.; published list rate equivalents are roughly:

Machine type	GPUs	$/hr (Vertex training, on-demand, us-central1)
a2-highgpu-1g	1× A100-40GB	≈ $3.67/hr
a2-ultragpu-1g	1× A100-80GB	≈ $5.07/hr
a2-highgpu-8g	8× A100-40GB	≈ $29.39/hr
a3-highgpu-8g	8× H100-80GB	≈ $88.49/hr ⇒ $11.06/H100·hr
a3-megagpu-8g	8× H100-80GB (with NVSwitch)	≈ $108/hr

(Vertex AI pricing is the Compute Engine GPU rate plus a Vertex training premium that varies by region. The figures above are approximate list prices from public sources; confirm in the Vertex AI pricing calculator before quoting.)

Max concurrent jobs. Per-region GPU quota (NVIDIA_A100_GPUS, NVIDIA_H100_GPUS, etc.) — typical default is 8 A100s per region, raise via Cloud Console quota request.

Verified API for spinning up N parallel jobs (using google-cloud-aiplatform):

# composer_replication/diloco/serverless/_vertex_ai_adapter.py
from google.cloud import aiplatform

aiplatform.init(project="my-project", location="us-central1",
                staging_bucket="gs://my-diloco-bucket")

def spawn_diloco_replica(replica_id: int, num_replicas: int, gcs_rendezvous: str):
    job = aiplatform.CustomJob.from_local_script(
        display_name=f"diloco-replica-{replica_id}",
        script_path="composer_replication/diloco/serverless/replica_entrypoint.py",
        container_uri="us-docker.pkg.dev/vertex-ai/training/pytorch-gpu.2-4.py311:latest",
        args=["--replica-id", str(replica_id),
              "--num-replicas", str(num_replicas),
              "--rendezvous-uri", gcs_rendezvous],
        machine_type="a2-highgpu-1g",      # 1× A100-40GB per replica
        accelerator_type="NVIDIA_TESLA_A100",
        accelerator_count=1,
        replica_count=1,                   # one replica, single-host
    )
    job.submit()                           # async; returns immediately
    return job

def spawn_n(num_replicas: int):
    gcs = "gs://my-diloco-bucket/run-2026-05-26"
    return [spawn_diloco_replica(i, num_replicas, gcs) for i in range(num_replicas)]

Right abstraction. Identical to SageMaker / HF / Modal: one CustomJob.submit() = one DiLoCo replica.

2.5 Azure ML Command Jobs

Inter-job networking. Single command job with resources.instance_count=N provisions N coordinated nodes (InfiniBand on ND*-series); separate jobs are isolated. Workaround: Azure Blob Storage or Azure ML Datastore.

Cold start. 3–8 min from job submission to first-byte-of-stdout for a curated environment; longer for custom images. Curated environments (e.g., AzureML-acpt-pytorch-2.8-cuda12.6@latest) are pre-cached on the cluster's image cache.

$/GPU·hr (from azure.microsoft.com/en-us/pricing/details/machine-learning/, GPU section, US West 2 PAYG list).

VM size	GPUs	Approx $/hr
Standard_NC24ads_A100_v4	1× A100-80GB	≈ $3.67/hr
Standard_NC48ads_A100_v4	2× A100-80GB	≈ $7.35/hr
Standard_ND96asr_A100_v4	8× A100-40GB (InfiniBand)	≈ $27.20/hr
Standard_NC40ads_H100_v5	1× H100 NVL 94GB	≈ $7/hr (regional)
Standard_ND96isr_H100_v5	8× H100-80GB (InfiniBand)	≈ $98/hr ⇒ $12.25/H100·hr

(Azure publishes $0/core ML "service surcharge" for these — you pay only the underlying VM rate. So the relevant hourly rate is the standard PAYG VM rate from Azure's pricing page, not a separate Azure ML markup. Low-Priority VMs cut up to 80% — DiLoCo-tolerant like SageMaker Spot.)

Max concurrent jobs. Per-subscription per-region GPU vCPU quota; typical default 0–24 cores for ND*-series, raise via Azure portal.

Verified API for spinning up N parallel jobs (using azure-ai-ml v2 SDK; pattern from learn.microsoft.com/en-us/azure/machine-learning/how-to-train-pytorch):

# composer_replication/diloco/serverless/_azure_ml_adapter.py
from azure.ai.ml import MLClient, command
from azure.identity import DefaultAzureCredential

ml_client = MLClient(DefaultAzureCredential(), subscription_id="...",
                     resource_group_name="...", workspace_name="...")

def spawn_diloco_replica(replica_id: int, num_replicas: int, blob_uri: str):
    job = command(
        code="./composer_replication",
        command=("python -m composer_replication.diloco.serverless.replica_entrypoint "
                 f"--replica-id {replica_id} --num-replicas {num_replicas} "
                 f"--rendezvous-uri {blob_uri}"),
        environment="AzureML-acpt-pytorch-2.8-cuda12.6@latest",
        compute="gpu-cluster",                  # an AmlCompute pre-created with min_instances=0, max_instances=8
        resources={"instance_count": 1},
        display_name=f"diloco-replica-{replica_id}",
    )
    return ml_client.jobs.create_or_update(job)

def spawn_n(num_replicas: int):
    blob = "azureml://datastores/workspaceblobstore/paths/diloco-run/"
    return [spawn_diloco_replica(i, num_replicas, blob) for i in range(num_replicas)]

Right abstraction. Same one-job-per-replica pattern.

2.6 Kubernetes + Volcano / KubeRay

Inter-job networking. Native — pods on the same cluster see each other on the cluster network. Volcano provides gang scheduling (all-or-nothing pod admission, essential for "all N DiLoCo replicas start together" semantics) and network-topology-aware scheduling (volcano.sh/en/docs/network_topology_aware_scheduling/). KubeRay's RayJob resource integrates with Volcano (PR ray-project/kuberay#3972, merged 2025-10-09) — RayJob + volcano.sh/queue-name label gives you gang-scheduled Ray clusters per job.

For Decoupled DiLoCo: N RayJobs, each running one replica, gang-scheduled via Volcano, sharing pseudo-grad through a PersistentVolume or in-cluster S3-compatible object store (MinIO).

Cold start. Pod schedule time depends on cluster state: seconds (pre-pulled image, free GPU node) to minutes (image pull + GPU node autoscale). Predictable on a steady-state cluster.

$/GPU·hr. Whatever the underlying K8s cluster pays. This is the cheapest tier in this audit if the user already runs a GPU K8s cluster (e.g., RunPod K8s, Lambda Cloud, OCI K8s, on-prem). Examples:

• RunPod community cloud K8s: ~$1.19/hr A100-80GB, ~$1.99/hr H100.
• Lambda K8s: ~$1.29/hr A100-40GB, ~$2.49/hr H100-80GB.
• On-prem owned hardware: amortized $0.50–$1.00 per A100/H100 hour.

Max concurrent jobs. Cluster capacity. Volcano's queue-based admission control + Kubernetes-native quotas govern this.

Verified API for spinning up N parallel jobs (Volcano Job + KubeRay pattern from the docs):

# k8s manifest, one per DiLoCo replica
apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata: {name: diloco-replica-0}
spec:
  minAvailable: 1
  schedulerName: volcano
  queue: diloco-queue
  tasks:
    - replicas: 1
      name: replica
      template:
        spec:
          containers:
            - name: trainer
              image: myorg/composer-replication:latest
              command: ["python", "-m", "composer_replication.diloco.serverless.replica_entrypoint",
                        "--replica-id", "0", "--num-replicas", "4",
                        "--rendezvous-uri", "s3://minio.cluster.local/diloco/"]
              resources:
                limits: {nvidia.com/gpu: 1}
          restartPolicy: OnFailure

…and the framework's K8sExecutor adapter does kubectl apply -f (or uses the Python K8s client) for each of N rendered manifests.

Right abstraction. Either one volcano.batch.Job per replica (simple, no Ray) or one RayJob per replica (overkill for DiLoCo, but useful if you want Ray Tune integration). One pod = one DiLoCo replica.

2.7 RunPod / Lambda / Vast.ai (honourable mentions)

Not in the original candidate list, but worth one paragraph each because they're the price-leaders for serverless GPUs:

• RunPod Serverless / Pods. Cheap on-demand A100/H100 (~$1.19–$2.17/hr A100-80GB; ~$1.99–$4.18/hr H100). REST API POST /v2/{endpoint}/run for serverless; SDK runpod for pods. No native multi-job network — same S3 workaround. Strong third adapter candidate for a cost-optimised deployment.
• Lambda Cloud (Lambda Labs). Bare metal hourly rentals, not a true serverless API. Programmatic launch via lambdalabs API. Outside the "serverless" framing.
• Vast.ai. Bidding-style spot market. API-driven launches. Cheapest per A100·hr in the market, but variable availability.

We do not include these as v0 adapters but document them as "next-up after Modal + HF" if the user wants further price compression.

---

3. The right abstraction: composer_replication.diloco.serverless

3.1 The core interface

# composer_replication/diloco/serverless/_protocol.py
from __future__ import annotations
from abc import ABC, abstractmethod
from dataclasses import dataclass
from typing import Any, Iterator, Protocol

@dataclass(frozen=True)
class ReplicaSpec:
    """One DiLoCo replica's launch config. Mirrors `make_diloco_outer_loop()`'s
    args (see composer_replication/diloco/__init__.py) plus a rendezvous_uri
    for the object-store all-reduce backend."""
    replica_id: int
    num_replicas: int
    rendezvous_uri: str           # s3://, gs://, az://, hf://, file://
    model_id: str                 # e.g. "Qwen/Qwen2.5-7B"
    inner_optimizer: dict[str, Any]   # serializable; reconstructed in worker
    sync_every: int = 500
    outer_lr: float = 0.7
    outer_momentum: float = 0.9
    outer_rounds: int = 100
    extra_env: dict[str, str] | None = None

@dataclass(frozen=True)
class ReplicaHandle:
    replica_id: int
    backend: str                  # "modal" | "hfjobs" | "sagemaker" | ...
    job_id: str
    log_url: str | None = None

@dataclass(frozen=True)
class ReplicaResult:
    replica_id: int
    status: str                   # "completed" | "failed" | "preempted"
    final_checkpoint_uri: str | None
    metrics: dict[str, Any]

class ServerlessExecutor(Protocol):
    """Protocol any serverless backend implements to host Decoupled DiLoCo."""

    def launch_replicas(self, specs: list[ReplicaSpec]) -> list[ReplicaHandle]: ...
    def poll(self, handles: list[ReplicaHandle]) -> list[ReplicaHandle]: ...
    def stream_logs(self, handle: ReplicaHandle) -> Iterator[str]: ...
    def cancel(self, handles: list[ReplicaHandle]) -> None: ...
    def collect(self, handles: list[ReplicaHandle], *,
                timeout: float | None = None) -> list[ReplicaResult]: ...

    @property
    def backend_name(self) -> str: ...

    @property
    def supports_inter_replica_network(self) -> bool:
        """True iff backend natively connects replicas (e.g., Modal i6pn).
        False = pseudo-grad must use rendezvous_uri object store. Default rendezvous
        is *always* object-store regardless; this flag only unlocks an opt-in
        same-backend fast path (see ModalExecutor(use_clustered_rdma=True))."""
        ...

Concrete adapters inherit from a small BaseExecutor(ABC) for cross-cutting retry/log/timeout, paralleling composer_replication.trainer.composer_trainer. launch_replicas() is partial-failure tolerant: on partial submit it returns handles for the K successful replicas with the failed one carrying job_id="" and a logged warning; the caller is responsible for cleanup via cancel().

3.2 The object-store all-reduce (the secret weapon)

The whole point of "decoupled" DiLoCo is that the cross-replica primitive is just object-store I/O. We implement it at the framework layer, not at the executor layer, so every adapter gets it for free:

# composer_replication/diloco/serverless/_rendezvous.py
import time, torch, fsspec

class ObjectStoreAllReduce:
    """Drop-in for `torchft.Manager.allreduce` over a shared object store.

    Each round t:
      (1) replica i writes  {uri}/round-{t}/replica-{i}.pt
      (2) all replicas barrier on count == num_replicas
      (3) rank 0 averages, writes {uri}/round-{t}/avg.pt
      (4) others read avg.pt, copy_ into the in-place tensor
      (5) rank 0 GCs round-(t-1)

    fsspec-backed so one path covers s3://, gs://, az://, hf://, file://.
    """

    def __init__(self, replica_id, num_replicas, rendezvous_uri,
                 fsspec_kwargs=None, poll_s=2.0, timeout_s=600.0):
        self.replica_id, self.num_replicas = replica_id, num_replicas
        self.uri = rendezvous_uri.rstrip("/")
        self.fs, _ = fsspec.url_to_fs(self.uri, **(fsspec_kwargs or {}))
        self.poll, self.timeout, self._round = poll_s, timeout_s, 0

    def allreduce(self, tensor):
        t = self._round
        my = f"{self.uri}/round-{t}/replica-{self.replica_id}.pt"
        avg = f"{self.uri}/round-{t}/avg.pt"

        with self.fs.open(my, "wb") as f:
            torch.save(tensor.cpu(), f)

        deadline = time.time() + self.timeout
        while time.time() < deadline:
            existing = [p for p in self.fs.ls(f"{self.uri}/round-{t}/")
                        if p.endswith(".pt") and "/replica-" in p]
            if len(existing) >= self.num_replicas: break
            time.sleep(self.poll)
        else:
            raise TimeoutError(f"barrier timeout at round {t}")

        if self.replica_id == 0:
            tensors = [torch.load(self.fs.open(f"{self.uri}/round-{t}/replica-{i}.pt", "rb"),
                                   map_location="cpu") for i in range(self.num_replicas)]
            torch.save(torch.stack(tensors).mean(dim=0), self.fs.open(avg, "wb"))

        deadline = time.time() + self.timeout
        while time.time() < deadline:
            if self.fs.exists(avg):
                tensor.copy_(torch.load(self.fs.open(avg, "rb"), map_location=tensor.device))
                break
            time.sleep(self.poll)
        else:
            raise TimeoutError(f"avg.pt timeout at round {t}")

        if self.replica_id == 0 and t > 0:
            try: self.fs.rm(f"{self.uri}/round-{t-1}/", recursive=True)
            except Exception: pass

        self._round += 1
        return _DummyWork()

    def should_commit(self): return True
    def start_quorum(self, *_, **__): pass
    @property
    def current_step(self): return self._round

class _DummyWork:
    def wait(self): pass
    def get_future(self): pass

The ObjectStoreAllReduce mocks the torchft Manager interface — exactly what make_diloco_outer_loop already takes (see composer_replication/diloco/__init__.py lines 64–125). No changes to the existing DiLoCo wrapper needed.

3.3 Replica entrypoint

This is the script every adapter runs in its container:

# composer_replication/diloco/serverless/replica_entrypoint.py
"""Run one Decoupled DiLoCo replica. Designed to be invoked as

    python -m composer_replication.diloco.serverless.replica_entrypoint \
        --replica-id N --num-replicas K --rendezvous-uri s3://... \
        --model-id Qwen/Qwen2.5-7B --sync-every 500 --outer-rounds 100
"""
import argparse, os, torch
from composer_replication.diloco import make_diloco_outer_loop
from composer_replication.diloco.serverless._rendezvous import ObjectStoreAllReduce


def main() -> None:
    p = argparse.ArgumentParser()
    p.add_argument("--replica-id", type=int, required=True)
    p.add_argument("--num-replicas", type=int, required=True)
    p.add_argument("--rendezvous-uri", required=True)
    p.add_argument("--model-id", required=True)
    p.add_argument("--sync-every", type=int, default=500)
    p.add_argument("--outer-rounds", type=int, default=100)
    p.add_argument("--outer-lr", type=float, default=0.7)
    args = p.parse_args()

    from transformers import AutoModelForCausalLM
    model = AutoModelForCausalLM.from_pretrained(args.model_id, torch_dtype=torch.bfloat16).cuda()
    inner_opt = torch.optim.AdamW(model.parameters(), lr=4e-4)

    manager = ObjectStoreAllReduce(replica_id=args.replica_id,
                                   num_replicas=args.num_replicas,
                                   rendezvous_uri=args.rendezvous_uri)
    outer = make_diloco_outer_loop(
        manager=manager, model_fragments=[model], inner_optimizer=inner_opt,
        outer_lr=args.outer_lr, outer_momentum=0.9, nesterov=True,
        sync_every=args.sync_every,
    )

    with outer:
        for outer_round in range(args.outer_rounds):
            for inner_step in range(args.sync_every):
                # caller plugs in their data + loss; for v0 we use a sketch.
                inner_opt.zero_grad(); ...; inner_opt.step()
            # outer-loop sync fires automatically at sync_every step boundary.

    # Push final checkpoint to rendezvous_uri/final/replica-N.pt
    ...

if __name__ == "__main__":
    main()

3.4 Package layout

Realised in v0.1 (Wave 17 update): ADR-005 shipped a flatter layout than the proposal below. The actual composer_replication/diloco/serverless/ tree:

composer_replication/
└── diloco/
    ├── __init__.py            # existing: make_diloco_outer_loop, torchft import
    └── serverless/
        ├── __init__.py        # re-exports all public classes (Wave 17a)
        ├── executor.py        # ServerlessExecutor Protocol + ReplicaHandle
        │                      #   + LocalProcessExecutor concrete adapter
        ├── allreduce.py       # ObjectStoreAllReduce + MockManager
        ├── modal.py           # ModalExecutor (skeleton — see __init__ docstring)
        ├── hf_jobs.py         # HFJobsExecutor (skeleton — uses huggingface_hub.run_job)
        ├── replica_entrypoint.py    # script each replica runs
        └── tests/             # multi-process file:// rendezvous tests

No leading underscores, no _protocol/_base/_rendezvous split, and Modal/HFJobs are flat modules rather than subpackages. The full public re-export surface (verified by tests/test_serverless_local.py::test_public_reexports_include_all_executors):

from composer_replication.diloco.serverless import (
    ServerlessExecutor,        # Protocol — implement to add your own backend
    LocalProcessExecutor,      # multi-process local replicas (CPU/GPU)
    ModalExecutor,             # Modal cloud — skeleton in modal.py
    HFJobsExecutor,            # HuggingFace Jobs — skeleton in hf_jobs.py
    ObjectStoreAllReduce,      # fsspec-backed allreduce (s3://, gs://, file://, hf://)
    MockManager,               # torchft.Manager-shaped duck-type
    ReplicaHandle,             # opaque handle returned by launch_replicas
)

Wave 16's user-journey reviewer caught that earlier versions of this __init__.py defined ModalExecutor and HFJobsExecutor in their respective modules but failed to re-export them from the package namespace. Wave 17a fixed the re-exports and added a regression test. The proposal below predates that fix.

composer_replication/
└── diloco/
    ├── __init__.py            # existing: make_diloco_outer_loop, torchft import
    └── serverless/
        ├── __init__.py        # re-exports
        ├── _protocol.py       # ServerlessExecutor Protocol, ReplicaSpec, ReplicaHandle, ReplicaResult
        ├── _base.py           # BaseExecutor(ABC) — common retry/log/timeout logic
        ├── _rendezvous.py     # ObjectStoreAllReduce (the cross-cutting allreduce)
        ├── replica_entrypoint.py    # the script every adapter runs in-container
        ├── modal/
        │   ├── __init__.py    # ModalExecutor
        │   └── adapter.py
        ├── hfjobs/
        │   ├── __init__.py    # HFJobsExecutor
        │   └── adapter.py
        └── runpod/             # optional v0.1+
            ├── __init__.py
            └── adapter.py

v0 ships: Modal + HFJobs. Both inherit from BaseExecutor, both delegate cross-replica state to ObjectStoreAllReduce. Symmetric implementation surface ≈ 250 lines per adapter.

v0.1+ candidates (add when needed): SageMaker, Vertex AI, Azure ML, RunPod, K8s/Volcano. The Protocol is stable; adding adapters is incremental.

3.5 What the user writes

from composer_replication.diloco.serverless import (
    ModalExecutor, HFJobsExecutor, ReplicaSpec
)

specs = [
    ReplicaSpec(replica_id=i, num_replicas=4,
                rendezvous_uri="s3://my-diloco-runs/2026-05-26/",
                model_id="Qwen/Qwen2.5-7B",
                inner_optimizer={"name": "AdamW", "lr": 4e-4},
                sync_every=500, outer_rounds=100)
    for i in range(4)
]

# Option A: all four replicas on Modal A100s
executor = ModalExecutor(gpu="A100-40GB", region=None, preemptible=True)
handles = executor.launch_replicas(specs)
results = executor.collect(handles)

# Option B: heterogeneous fleet — 2 on Modal, 2 on HF Jobs
modal_ex = ModalExecutor(gpu="A100-40GB")
hf_ex = HFJobsExecutor(flavor="a100-large")
modal_handles = modal_ex.launch_replicas(specs[:2])
hf_handles = hf_ex.launch_replicas(specs[2:])
# both groups read+write the SAME s3://... rendezvous URI — they DiLoCo together.
results = modal_ex.collect(modal_handles) + hf_ex.collect(hf_handles)

The "heterogeneous fleet" pattern is the point of Decoupled DiLoCo as articulated in the user brief. Modal + HF together is a meaningful test that tells us both adapters work and the rendezvous protocol is backend-agnostic.

---

4. Cross-cutting design decisions

4.1 Why object-store rendezvous is the default (even on Modal)

Even though Modal supports @modal.experimental.clustered with RDMA, the framework default is object-store-based pseudo-gradient exchange. Reasons:

1. Backend portability. Same code runs on Modal, HF, SageMaker, Vertex, Azure, K8s. Adding a new backend is implementing 6 methods (launch_replicas, poll, stream_logs, cancel, collect, backend_name) — zero changes to the rendezvous layer.
2. Cost asymmetry. RDMA-class networking on Modal requires @clustered(rdma=True) which gates on 8 GPUs/node and tighter scheduling — more expensive than 4 separate @function invocations of 1 GPU each.
3. DiLoCo's communication is ridiculous overkill for RDMA. 2 GB every 10 minutes = ~3 Mbps average. S3 GET/PUT at 10 MB/s does it in ~3 min — well under the 10 min outer-round budget.
4. Failure decoupling. A clustered-RDMA failure aborts the whole job (gang-scheduled). Object-store rendezvous tolerates a missing replica (skip its tensor in the average) — better matches DiLoCo's natural fault tolerance.

The opt-in escape hatch: ModalExecutor(use_clustered_rdma=True) dispatches to @modal.experimental.clustered(rdma=True) and skips object-store. This is for the user who wants Modal-only, max-throughput, single-region runs. It's not the default and not what we test against.

4.2 Rendezvous URI scheme support

fsspec covers all the storage backends we need:

Scheme	Backend	Used for
s3://	s3fs	SageMaker default; cheapest for AWS-centric runs
gs://	gcsfs	Vertex AI default
az://	adlfs	Azure ML default
hf://	huggingface_hub.HfFileSystem	HF Jobs preferred (Volume mount makes it look like local fs already)
file://	builtin	local single-host tests; CI

The framework picks the right default per-executor (Modal → s3://, HF → hf://, SageMaker → s3://, etc.) but always allows override.

4.3 Failure model

Replica failure mid-round. The barrier in ObjectStoreAllReduce has a configurable timeout (default 600 s). If a replica doesn't write its file by then, rank-0 (the averager) has two options governed by replica_failure_policy:

• "strict" (default): TimeoutError → all replicas abort. Resume from last committed checkpoint.
• "skip": rank-0 averages over what's there, includes a --num-survivors=K annotation in avg.pt. Other replicas read this and continue. DiLoCo paper §4.5 reports robustness to occasional missing workers; this matches that.

Whole-cluster failure. Outer rounds checkpoint to {rendezvous_uri}/checkpoint-{t}/; restart sets args.restart_from=T and skips ahead.

4.4 What we explicitly do NOT do

• No cross-job NCCL. Even on Modal, even with clustered, the framework uses object-store rendezvous. (Modal clustered is exposed only via the explicit opt-in flag.)
• No DDP/FSDP across replicas. Each replica is its own self-contained DDP/FSDP world; replicas talk to each other only via the outer-loop. This is the core of DiLoCo.
• No "control plane" service. No coordinator process, no scheduler container. The object store is the coordinator (writes are the messages, file-existence is the synchronization). This is what makes the system work across heterogeneous executors with no shared infra.
• No Modal-specific or HF-specific dependencies in composer_replication.diloco. Adapter dependencies (modal, huggingface_hub) are imported lazily inside the adapter modules, exactly how torchft is imported lazily in composer_replication/diloco/__init__.py today.

---

5. Risks and mitigations

Risk	Likelihood	Mitigation
Object-store latency dominates outer-round wallclock for large models	M	For 70B+, add fsspec parallel-upload (multipart) + bf16 quantize on-write. Most outer rounds are 7B-scale where 2 GB transfer is well under 1 min.
Rank-0 replica crashes mid-average → orphaned barrier	L	Add a lock-{t}.json heartbeat with TTL; any non-zero replica that sees a stale lock can take over. v1+.
Modal + HF cost arbitrage misleading because preemption rates differ	M	Track preemption-rate per backend, surface in ReplicaResult.metrics. User-visible.
HF Jobs has no public per-account concurrency cap → may hit a hidden limit at N=8	L	Add exponential-backoff retry around run_job; cap max_concurrent_launches configurable per executor.
AWS / GCP / Azure premiums make their adapters effectively price-uncompetitive	H (already true)	Be honest in docs (this doc). Recommend Modal + HF for cost-sensitive users; cloud-vendor adapters for users who must run there for compliance or credits.
Rendezvous bucket becomes a security choke point (model weights exposed)	M	Document that rendezvous_uri should be a private bucket with replica-only IAM/principals. Provide RendezvousAccessPolicy helper that emits boto3/gcloud/az IAM JSON.
Modal @experimental.clustered API churn (it's experimental)	M	Default path doesn't depend on clustered. Fall-back path uses regular @function. Document the opt-in clearly.
torchft sign-convention regression	L	Already pinned with the unit test in spike 008 (see spikes/008-streaming-diloco/tests/test_diloco_smoke.py::test_diloco_pseudogradient_sign_convention). The serverless layer doesn't touch this — it only swaps in a different Manager.allreduce impl.

---

6. Validation plan

Three smoke tests, in order of cost:

1. Spike 009-A (free, ≤30 min): LocalProcessExecutor + ObjectStoreAllReduce with file:// rendezvous. Two in-process replicas DiLoCo-train a 0.5B model on MNIST-equivalent text data. Asserts the rendezvous protocol works.
2. Spike 009-B (Modal, ≤$5): ModalExecutor × 2 replicas, A100-40GB each, Qwen2.5-0.5B, 50 inner steps × 2 outer rounds. Asserts the Modal adapter launches, replicas find each other through S3 rendezvous, and pseudo-gradients average correctly. Cost: ~30 min × $2.10 × 2 = $2.10 + setup overhead, comfortable under cap.
3. Spike 009-C (heterogeneous, ≤$10): 1 Modal A100 + 1 HF Jobs a100-large. Same model, 2 outer rounds. Validates that rendezvous works across backends — the key claim of Decoupled DiLoCo. Cost: ~30 min × ($2.10 + $2.50) = ~$2.30, plus per-job startup.

Each spike has a verdict.md following the conventions from spikes/008-streaming-diloco/.

---

7. References (primary sources, all cited above)

• DiLoCo paper: Douillard et al., "DiLoCo: Distributed Low-Communication Training of Language Models," arXiv:2311.08105 (2023). https://arxiv.org/abs/2311.08105
• Streaming DiLoCo paper: Liu et al., "Streaming DiLoCo with overlapping communication," 2025. https://arxiv.org/abs/2501.18512
• torchft local_sgd.DiLoCo: https://github.com/meta-pytorch/torchft/blob/main/torchft/local_sgd.py
• Modal multi-node clusters: https://modal.com/docs/guide/multi-node-training
• Modal cluster networking (i6pn): https://modal.com/docs/guide/private-networking
• Modal pricing: https://modal.com/pricing
• Modal GPU options: https://modal.com/docs/guide/gpu
• HF Jobs overview: https://huggingface.co/docs/hub/en/jobs
• HF Jobs pricing: https://huggingface.co/docs/hub/jobs-pricing
• HF Jobs Python API: https://huggingface.co/docs/huggingface_hub/en/guides/jobs
• HF Jobs reference: https://huggingface.co/docs/huggingface_hub/main/en/package_reference/jobs
• AWS SageMaker pricing: https://aws.amazon.com/sagemaker/ai/pricing/
• AWS SageMaker CreateTrainingJob API: https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateTrainingJob.html
• AWS SageMaker SMDDP: https://docs.aws.amazon.com/sagemaker/latest/dg/data-parallel-framework-estimator.html
• AWS SageMaker warm pools: https://docs.aws.amazon.com/sagemaker/latest/dg/train-warm-pools.html
• GCP Vertex AI compute config: https://cloud.google.com/vertex-ai/docs/training/configure-compute
• GCP Vertex AI training SKUs: https://cloud.google.com/skus/sku-groups/vertex-training
• GCP Vertex AI pricing: https://cloud.google.com/vertex-ai/pricing
• Azure ML pricing: https://azure.microsoft.com/en-us/pricing/details/machine-learning/
• Azure ML PyTorch SDK v2 guide: https://learn.microsoft.com/en-us/azure/machine-learning/how-to-train-pytorch
• Azure NDasrA100_v4 spec: https://learn.microsoft.com/en-us/azure/virtual-machines/sizes/gpu-accelerated/ndasra100v4-series
• Azure NCads H100 v5 spec: https://learn.microsoft.com/en-us/azure/virtual-machines/ncads-h100-v5
• Volcano: https://volcano.sh/en/docs/unified_scheduling/
• Volcano network-topology-aware scheduling: https://volcano.sh/en/docs/network_topology_aware_scheduling/
• KubeRay + Volcano integration: https://docs.ray.io/en/latest/cluster/kubernetes/k8s-ecosystem/volcano.html
• KubeRay RayJob+Volcano PR: https://github.com/ray-project/kuberay/pull/3972

Internal references (in this repo):

• docs/research/MODAL_RECONNAISSANCE.md — pricing/cold-start audit for Modal smoke runs.
• docs/research/DILOCO_RECONNAISSANCE.md — DiLoCo implementation candidates audit.
• docs/adrs/ADR-001-gpu-venue.md — local-vs-cloud GPU decision for smoke phase.
• docs/adrs/ADR-003-diloco-impl.md — torchft choice + sign convention.
• composer_replication/diloco/__init__.py — existing make_diloco_outer_loop wrapper this design plugs into without modification.
• spikes/008-streaming-diloco/ — the existing in-process DiLoCo smoke that the serverless adapter inherits sign-convention test from.