research/02-diloco-family.md

# DiLoCo Family: Distributed Low-Communication Training

> Comprehensive survey of the DiLoCo ecosystem for RL post-training.  
> Last updated: 2026-05-25

---

## 1. DiLoCo Family Overview

### 1.1 Original DiLoCo (DeepMind, 2023)

**Paper:** [arxiv 2311.08105](https://arxiv.org/abs/2311.08105) — *"DiLoCo: Distributed Low-Communication Training of Language Models"*  
**Authors:** Douillard et al., Google DeepMind  
**Published:** Nov 2023, ICML 2024

DiLoCo is a distributed optimization algorithm that enables training LLMs across **"islands" of poorly connected devices** (e.g., data centers on different continents). It is a variant of Federated Averaging (FedAvg) with three key design decisions:

1. **Large number of inner steps (H):** Each worker takes H local optimization steps (typically H=500) before communicating. This achieves ~500× communication reduction.
2. **Inner optimizer: AdamW.** Workers train independently on distinct data shards using standard AdamW, accumulating parameter changes.
3. **Outer optimizer: Nesterov SGD with momentum.** After H steps, each worker computes a **pseudo-gradient** Δᵢ = θ_start - θ_end (the parameter difference over the H steps). These pseudo-gradients are averaged across workers and fed into an outer Nesterov momentum optimizer to produce the next global weights.

**Why it works:** The pseudo-gradient after H=500 AdamW steps is much less noisy than a per-step gradient from a single minibatch. The outer optimizer treats these pseudo-gradients like regular gradients, applying momentum for smoothing across outer steps. Convergence proofs extend from FedOpt analysis.

**Key results:**
- 8 workers on C4 dataset match fully synchronous optimization quality while communicating 500× less
- Robust to non-IID data distributions across workers (FedAvg's traditional weakness)
- Works well with heterogeneous data shards
- Models up to 400M parameters in the original paper

**When it fails / limitations:**
- Original paper experiments start from a pre-trained checkpoint (24K steps), so cold-start behavior is less studied
- Communication is still **all parameters at once** — peak bandwidth requirement equals model size per sync
- Synchronous: all workers must wait for the slowest (straggler problem)
- The original paper's compute efficiency measurements are limited (no good "compute-matched" baselines according to critics)
- Outer Nesterov momentum adds ~1.5× the optimizer state memory (stored on CPU)

**Decoupled DiLoCo (DeepMind, 2025):** Google later extended DiLoCo with "[Decoupled DiLoCo](https://deepmind.google/blog/decoupled-diloco)" which leverages Pathways-style asynchronous data flow. This version showed resilience to hardware failures — maintaining high "goodput" even when nodes fail, while traditional synchronous training nosedives. Tested with Gemma 4 models.

---

### 1.2 OpenDiLoCo (Prime Intellect, 2024)

**Paper:** [arxiv 2407.07852](https://arxiv.org/abs/2407.07852) — *"OpenDiLoCo: An Open-Source Framework for Globally Distributed Low-Communication Training"*  
**Repo:** [GitHub: PrimeIntellect-ai/OpenDiLoCo](https://github.com/PrimeIntellect-ai/OpenDiloco)  
**Published:** Jul 2024

OpenDiLoCo is the first open-source reproduction and scaling of DiLoCo. Built on the **Hivemind** library for decentralized P2P communication (libp2p-based DHT for peer discovery, decentralized all-reduce).

**Key contributions beyond the paper:**
- **Reproduced DiLoCo with 90-95% compute utilization** across two continents and three countries
- **Scaled to 3× the original** (1.1B parameter models vs DeepMind's 400M)
- **Ablation: FP16 pseudo-gradients work fine** — no degradation vs FP32, cutting sync payload by 2×
- **Hivemind-based all-reduce** instead of parameter server — fully decentralized, no single point of failure
- Kubernetes-native deployment with Docker images
- Per-device batch size auto-adaptation to match VRAM

**What broke vs the paper and what they fixed:**
- Network bandwidth utilization was initially poor (~40× worse than theoretical). Fixed with VPN mesh networking, connection sharding (8× improvement), and optimized routing.
- Hivemind's default all-reduce was slow for large models. Fixed with layer-bucketed all-reduce and TCP tuning.
- Checkpointing blocked training for 20+ minutes on 10B scale. Fixed with async `/dev/shm` checkpointing + sidecar HTTP servers for live node joining.

**H=125 steps used in 1.1B experiments** (not 500), matching single-cluster perplexity with only 20% more total compute.

---

### 1.3 Streaming DiLoCo (DeepMind, 2024)

**Paper:** [arxiv 2501.18512](https://arxiv.org/abs/2501.18512) — *"Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunch"*  
**Published:** Jan 2025

Three orthogonal improvements to base DiLoCo:

1. **Partial parameter synchronization (streaming):** Instead of syncing all parameters at once at the outer step boundary, synchronize **subsets of parameters in sequence** throughout the inner steps. This dramatically reduces peak bandwidth requirements (up to 2 orders of magnitude total reduction).

2. **Overlapping communication with computation:** While one subset of parameters is being all-reduced, the workers continue local training on the remaining parameters. This hides communication latency behind useful compute — the "free lunch."

3. **Lower-precision outer state:** The outer optimizer state (Nesterov momentum buffer, pseudo-gradient accumulators) is stored in lower precision (FP16/BF16), reducing memory and communication costs further.

**Key result:** Billion-scale parameter models trained with 2 orders of magnitude less peak bandwidth than base DiLoCo, with matching model quality. This makes DiLoCo feasible on consumer-grade internet connections (10-100 Mbps instead of Gbps requirements).

**Architecture insight:** The streaming approach effectively converts what was a "bursty" all-at-once sync into a continuous trickle of parameter updates, which is much kinder to TCP congestion control and shared network links.

---

### 1.4 Async DiLoCo / NoLoCo / DisTrO

#### Async Local-SGD (Async DiLoCo)

**Paper:** [arxiv 2401.09135](https://arxiv.org/abs/2401.09135) — *"Asynchronous Local-SGD Training for Language Modeling"*  
**Authors:** Douillard et al. (also from the original DiLoCo paper)

Instead of synchronous barrier-based aggregation every H steps, **each worker pushes pseudo-gradients to a parameter server as soon as it finishes its inner steps**, without waiting for others. The parameter server applies updates asynchronously.

**Key innovations:**
- **Delayed Nesterov (DN) optimizer:** Modified outer optimizer that accounts for staleness in async updates
- **Dynamic Local Updates (DyLU):** Workers take H steps proportional to their speed — faster GPUs take more steps, slower ones take fewer. This eliminates straggler bottlenecks.
- **Heterogeneity tolerance:** Empirically works well with up to 4× speed differences between workers with no perplexity degradation

**Limitation:** Staleness from sequential (not averaged) application of individual worker updates causes some convergence degradation. The DN+ DyLU variant closes most of this gap (matching synchronous DiLoCo perplexity).

#### HALoS (Hierarchical Async Local SGD)

**Paper:** [arxiv 2506.04531](https://arxiv.org/abs/2506.04531), ICML 2025

Extends Async DiLoCo for geo-distributed settings where intra-region and inter-region bandwidth differ dramatically. Uses **Local Parameter Servers (LPS)** per region and a **Global Parameter Server (GPS)** across regions. Achieves 7.5× faster convergence than standard DiLoCo and 2.1× faster than flat Async DiLoCo.

#### DisTrO (Nous Research, 2024)

**Repo:** [GitHub: NousResearch/DisTrO](https://github.com/NousResearch/DisTrO)  
**Paper:** Preliminary report at `NousResearch/DisTrO/blob/main/A_Preliminary_Report_on_DisTrO.pdf`

DisTrO (Distributed Training Over-the-Internet) is **not a DiLoCo variant per se** but a family of **distributed optimizers that reduce per-step inter-GPU communication** by 857× compared to All-Reduce.

**Core approach:** DCT-based gradient compression.
- Apply 2D Discrete Cosine Transform (DCT) to gradient tensors
- Keep only top-k DCT coefficients (energy compaction property: most gradient information lives in low frequencies)
- Transmit compressed representation → decompress via inverse DCT
- Result: 86.8 MB transmitted per step vs 74.4 GB for uncompressed All-Reduce

**Architecture components:**
- `DistroModule` — wraps nn.Module to intercept gradient sync
- `DistroOptimizer` — wraps standard optimizer with compression hooks
- `DistroDDP` — extends PyTorch DDP with compressed communication
- Multiple compressors: DCT, random-k, top-k

**Key difference from DiLoCo:** DisTrO compresses gradients at **every step** (not every H steps), making it suitable for traditional synchronous training with drastically reduced bandwidth. It operates at a fundamentally different level — gradient compression rather than outer-loop aggregation.

---

### 1.5 INTELLECT-1: First Globally Distributed 10B Model (Prime Intellect, 2024)

**Blog:** [primeintellect.ai/blog/intellect-1](https://www.primeintellect.ai/blog/intellect-1)  
**Framework:** Prime (formerly ZeroBand) — [GitHub: PrimeIntellect-ai/Prime](https://github.com/PrimeIntellect-ai/Prime)

The first-ever globally distributed training run of a 10B parameter model, with ~14 organizations contributing compute (Hugging Face, SemiAnalysis, Arcee, Hyperbolic, Akash, etc.).

**Scale:** 10× larger than OpenDiLoCo (1B→10B), ~25× larger than original DiLoCo (400M→10B).

**Key infrastructure innovations in Prime framework:**

| Feature | Description |
|---------|-------------|
| **ElasticDeviceMesh** | Dynamic process groups that resize when nodes join/leave. Heartbeat-based failure detection with "deathrattle" fast-fail. |
| **Async distributed checkpointing** | Write to `/dev/shm` (RAM disk) first, then async copy to disk + upload to cloud. Checkpoint blocking time reduced from 20 min → negligible. |
| **Live checkpoint recovery** | New nodes download checkpoint from peer sidecar HTTP servers in `/dev/shm`, join outer step with zero pseudo-gradients. |
| **Custom Int8 All-Reduce kernel** | JIT-compiled C++ ring-reduce with int8 quantization. Dequantize→accumulate in fp32→requantize pipeline. 4× payload reduction. |
| **Multithreaded uint8 ops** | Custom C++ quantization ops achieving 60× speedup over torch native ops. |
| **VPN mesh networking** | Optimized P2P routing, up to 40× bandwidth improvement over public IP. 4 Gbps achieved between US data centers. |
| **FSDP2 / DTensor** | PyTorch FSDP2 for intra-node sharding, bucketed pseudo-gradient all-reduce. |
| **CPU offloading** | DiLoCo outer optimizer state entirely on CPU. Negligible overhead since syncs are infrequent. |

**Results:**
- **98% compute utilization** across globally distributed workers
- **H=100 steps**, ~40 min per inner loop on 8×H100 nodes
- **Int8 pseudo-gradient quantization** → 400× total communication reduction
- **All-reduce sync < 1 minute** (1-2% of total training time)
- **~30% MFU** (Model FLOPs Utilization) for the 10B run
- Trained on 6T+ tokens from FineWeb-Edu + DCLM + Stack v2 + OpenWebMath mix
- Llama-3 architecture, WSD learning rate scheduler

**Caveat:** No compute-matched single-cluster baseline, so true efficiency overhead is hard to quantify. OpenDiLoCo 1.1B experiments showed ~20% compute overhead vs single-cluster.

---

### 1.6 INTELLECT-2 / PRIME-RL: Globally Distributed RL Post-Training (Prime Intellect, 2025)

**Paper:** [arxiv 2505.07291](https://arxiv.org/abs/2505.07291) — *"INTELLECT-2: A Reasoning Model Trained Through Globally Decentralized Reinforcement Learning"*  
**PRIME-RL framework paper:** NeurIPS 2025, [OpenReview](https://openreview.net/pdf?id=yk3ICpEbv8)

**This is the most directly relevant work for our RL post-training framework use case.**

INTELLECT-2 is the first globally distributed RL training run of a **32B parameter model** (fine-tuned from QwQ-32B). It improves upon QwQ-32B via GRPO-style RL training across geographically distributed, heterogeneous, permissionless compute.

#### Novel Infrastructure Components

**PRIME-RL** — The RL training framework with three key abstractions:

1. **Orchestrator** (CPU process): Handles data scheduling, collects rollouts from inference workers, assembles training batches, relays updated weights to inference service. Uses `verifiers` environments for multi-turn rollout generation and scoring.

2. **Trainer** (GPU): FSDP2-based training consuming rollout batches and producing updated policy weights. Inspired by torchtitan, supports tensor/context/expert parallelism.

3. **Inference Service** (GPU): vLLM backend with three custom endpoints:
   - `/init_broadcaster` — initialize NCCL process group for weight broadcast
   - `/update_weights` — in-place tensor update for latest policy
   - `/reload_weights` — reset to base model

**TOPLOC** — Trustless verifiable inference for untrusted/permissionless inference workers. Uses locality-sensitive hashing to generate proofs that a worker actually ran the model they claim to have run. Each rollout is verified by a different worker than the one that generated it.

**SHARDCAST** — Efficient weight broadcasting from training nodes to inference workers, designed for high-latency links between data centers.

#### Training Architecture for Decentralized RL

The key insight: **RL is inherently more asynchronous than pre-training.** The rollout→train→update cycle naturally decouples inference from training.

```
                   ┌─────────────┐
                   │ Orchestrator │ (CPU - scheduling & data flow)
                   └──────┬──────┘
            ┌─────────────┼─────────────┐
            ▼             │             ▼
    ┌───────────┐         │     ┌────────────────┐
    │  Trainer   │◄────────┘     │   Inference    │
    │  (FSDP2)   │               │  (vLLM, TP/DP) │
    │  8×H200    │               │  16×H200       │
    └───────────┘               └────────────────┘
         │                              │
         └──────────────────────────────┘
         Weight broadcast (SHARDCAST)
         Rollout verification (TOPLOC)
```

#### Asynchronous Off-Policy Training

PRIME-RL supports `async_level` to control staleness:
- `async_level=0`: Fully synchronous (inference stalls until trainer finishes)
- `async_level=1`: One-step off-policy — inference generates rollouts from θ₀ while trainer produces θ₁ (fully overlapping). Sufficient for colocated or well-connected setups.
- `async_level≥2`: Required for geo-distributed settings where weight broadcast latency is significant. Inference uses θ_{min(0, n-async_level)}.

**For our use case:** async_level=1 is probably sufficient for a home cluster with decent Ethernet. async_level=2+ matters if we distribute inference across the internet.

#### Training Recipe & Stability Learnings

- **AIPO token-level loss** with importance sampling correction between vLLM and training logprobs
- **Critical finding:** Even when π and μ share identical parameters θ, vLLM produces significantly different logprobs than the training backend → use vLLM logprobs directly with importance sampling correction
- **No recompute of reference logprobs** — rely on vLLM outputs
- This prevents crashes that occur "multiple days into experiments" due to distribution shift

**Efficiency:**
- 24 H200 GPUs: 8 for trainer, 16 for inference (DP=4, TP=4)
- Trainer throughput: 11.3K tok/s, Inference: 14.4K tok/s
- Peak MFU: 38.46% on trainer
- 160 training steps, ~64 hours, 1,536 GPU-hours
- ~22.9 min per training step
- Stable training dynamics: non-decreasing gradient norm, stable entropy, increasing reward

---

## 2. Communication Efficiency Analysis

| Variant | Sync Frequency | Peak Bandwidth | Compression | Total Reduction | Best For |
|---------|---------------|----------------|-------------|-----------------|----------|
| **Base DiLoCo** | Every H=500 steps | Full model | None | ~500× in frequency only | Research baseline |
| **OpenDiLoCo** | H=100-125 | Full model (FP16) | None (FP16 helps 2×) | ~100-125× frequency | Open reproduction |
| **Streaming DiLoCo** | Continuous partial | Subset of params | FP16 outer state | ~100× peak BW + frequency | Slow consumer links |
| **INTELLECT-1 (Prime)** | H=100 | Int8 pseudo-gradients | 4× (int8) | ~400× total | Production 10B pre-training |
| **Async DiLoCo** | Per-worker (no barrier) | Full model | None | ∞ (no sync wait) | Heterogeneous hardware |
| **DisTrO** | Every step | DCT compressed | 857× vs All-Reduce | Per-step communication | Fine-grained sync needed |
| **PRIME-RL** | Per training step | Weight broadcast | SHARDCAST | N/A (RL is inherently async) | RL post-training |

**Takeaway for our framework:** For RL, the communication pattern is fundamentally different from pre-training. We're not sync'ing pseudo-gradients — we're broadcasting policy weights trainer→inference and receiving rollouts inference→trainer. PRIME-RL's async off-policy approach with SHARDCAST weight broadcast is the right model.

---

## 3. RL-Specific Variants: PRIME-RL Deep Dive

### Why RL is Different from Pre-Training for Distributed Training

| Aspect | Pre-Training (DiLoCo) | RL Post-Training (PRIME-RL) |
|--------|----------------------|---------------------------|
| **Data flow** | Data → forward → loss → backward → pseudo-gradient | Rollout → reward → advantage → gradient → weight broadcast |
| **Communication pattern** | Sync pseudo-gradients every H steps | Continuous: rollouts inflow, weights outflow |
| **GPU workloads** | Homogeneous (all training) | Heterogeneous (training + inference) |
| **Latency sensitivity** | Low (H=100-500 steps between syncs) | Medium (weight broadcast latency matters) |
| **Staleness tolerance** | Low for sync, medium for async | High by design (off-policy RL) |
| **Verification need** | None (trusted workers) | TOPLOC for untrusted inference workers |

### PRIME-RL Architecture in Detail

**Orchestrator data flow:**
1. Check if inference service needs weight update → send `/update_weights` to vLLM
2. Sample prompts from data buffer (supports online difficulty filtering)
3. Send prompts to `verifiers` environment → async rollout generation + scoring
4. Collect completed rollouts (completions, logprobs, masks, rewards)
5. When sufficient batch ready → shard across DP ranks, collate, dispatch to trainer
6. Trainer processes global batch via FSDP2 micro-batches
7. Updated policy weights written to disk → inference service loads for next step

**Key advantage for our use case:** The orchestrator is a **lightweight CPU process** — no GPU needed. This means we can run the trainer on a single GPU machine and the orchestrator on a separate CPU-only node, with inference workers potentially on commodity GPUs elsewhere.

### Verifiers + Environments Hub

PRIME-RL uses the `verifiers` library (by Will Brown, also contributors to Prime Intellect) for environment abstraction:
- Environments encapsulate multi-turn rollout logic, tool calling, dataset preprocessing, and reward computation
- Reward manager ("Rubric") supports compound rewards, LLM judges, caching, custom parallelism
- Environments are installable Python modules via the Environments Hub
- Same environment can be used with PRIME-RL, TRL, verifiers, or any compatible trainer

**This is exactly the kind of modularity we want for our RL post-training framework.**

---

## 4. Infra Requirements for Running at Home / Small Cluster

### What It Takes: Minimum Viable DiLoCo/PRIME-RL Setup

**For DiLoCo-style pre-training (e.g., INTELLECT-1 scale):**

| Component | Minimum | Recommended |
|-----------|---------|-------------|
| GPUs per worker | 1× 24GB (3090/4090) | 4-8× H100/A100 per worker |
| Number of workers | 2 | 4-8 |
| Inter-worker bandwidth | 100 Mbps | 1 Gbps+ |
| RAM per worker | 64 GB | 256 GB (for CPU offloading) |
| Disk per worker | 500 GB NVMe | 2 TB NVMe |
| Software | Hivemind + OpenDiLoCo | Prime framework (ElasticDeviceMesh) |

**For PRIME-RL style RL post-training:**

| Component | Minimum | Recommended |
|-----------|---------|-------------|
| Trainer GPU | 1× 48GB (A6000) | 1× 8×H100 node |
| Inference GPU | 1× 24GB (3090) | 2-4× GPUs with vLLM |
| CPU node | Any modern CPU | Orchestrator runs on CPU only |
| Weight broadcast | Simple HTTP file server | SHARDCAST or NCCL broadcast |
| Verification | Trusted workers (no TOPLOC needed) | TOPLOC for permissionless workers |
| Data buffer | Simple in-memory queue | Online difficulty filtering |
| Environment | Single verifiers env | Multiple envs from Environments Hub |

### GPU Heterogeneity Tolerance

**DiLoCo variants handle heterogeneity well:**
- Async DiLoCo with Dynamic Local Updates (DyLU): Workers take H steps proportional to their speed. 3090 might take H=50 while H100 takes H=200. Empirically robust to 4× speed differences.
- Standard DiLoCo: Straggler problem — all workers wait for slowest. **Not recommended for mixed hardware.**
- Streaming DiLoCo: Better tolerance since communication is continuous, but still synchronous.
- PRIME-RL: Trainer and inference are **separate pools** — inference workers can be heterogeneous (vLLM auto-scales to available compute). Trainer is typically homogeneous.

**Recommendation for mixed 3090/4090/H100:** Use PRIME-RL's architecture. Put the trainer on the best GPU(s), use all available GPUs for inference. Async off-policy training naturally handles speed differences between inference workers.

### Practical Libraries

1. **Hivemind** ([github.com/learning-at-home/hivemind](https://github.com/learning-at-home/hivemind)) — P2P decentralized training. libp2p DHT for peer discovery, decentralized all-reduce. Used by OpenDiLoCo. Actively maintained.

2. **Prime / ZeroBand** ([github.com/PrimeIntellect-ai/Prime](https://github.com/PrimeIntellect-ai/Prime)) — Prime Intellect's framework with ElasticDeviceMesh, async checkpointing, int8 all-reduce kernel, VPN mesh. Production-grade but more complex.

3. **PRIME-RL** ([github.com/PrimeIntellect-ai/prime-rl](https://github.com/PrimeIntellect-ai/prime-rl)) — RL framework with orchestrator + FSDP trainer + vLLM inference. The go-to for distributed RL.

4. **DisTrO** ([github.com/NousResearch/DisTrO](https://github.com/NousResearch/DisTrO)) — Drop-in distributed optimizer with DCT compression. Works with standard PyTorch training loops.

5. **OpenRLHF** ([github.com/OpenRLHF/OpenRLHF](https://github.com/OpenRLHF/OpenRLHF)) — Ray + vLLM distributed RLHF. Decoupled Actor, Reward, Reference, Critic across GPUs. Not DiLoCo-based but well-established RL infrastructure.

6. **veRL** ([github.com/volcengine/verl](https://github.com/volcengine/verl)) — Volcano Engine's RLHF framework. Hybrid engine design. 80-90% of training time is rollout generation. Not designed for geo-distribution.

---

## 5. Recommendation for Our Framework

### Summary Assessment

| Criterion | Best Option | Rationale |
|-----------|------------|-----------|
| **RL-native design** | PRIME-RL | Purpose-built for distributed RL, not adapted from pre-training |
| **Async by default** | PRIME-RL | Off-policy training at async_level=1-2, natural fit for RL rollout cycles |
| **Modularity** | PRIME-RL | Orchestrator/Trainer/Inference separation, verifiers environments |
| **Small cluster friendliness** | PRIME-RL or OpenRLHF | Both run on single-node or small multi-node setups |
| **Internet-scale distribution** | PRIME-RL + TOPLOC | Only framework with trustless verification for permissionless workers |
| **Communication efficiency** | PRIME-RL + SHARDCAST | Weight broadcast is the relevant metric for RL, not pseudo-gradient sync |
| **Ecosystem maturity** | OpenRLHF | Most established, but not built for geo-distribution |
| **Heterogeneous hardware** | Async DiLoCo + PRIME-RL | DyLU for pre-training, separate inference pool for RL |

### Recommended Architecture

**Primary: PRIME-RL as the RL substrate, with optional DiLoCo-style outer-loop for the trainer itself if multi-node training is needed.**

```
┌──────────────────────────────────────────────────┐
│                  Orchestrator (CPU)               │
│  - Schedules rollouts                             │
│  - Manages data buffer (difficulty filtering)     │
│  - Relays weights trainer → inference             │
│  - Assembles training batches                     │
└──────┬───────────────────────────────┬───────────┘
       │                               │
       ▼                               ▼
┌──────────────┐              ┌─────────────────────┐
│   Trainer     │              │  Inference Pool      │
│  (FSDP2/DiLoCo│              │  (vLLM, commodity   │
│   if multi-GPU)│             │   GPUs, heterogeneous)│
│               │              │                      │
│  Inner: AdamW  │             │  /v1/chat/completions │
│  Outer: opt.   │             │  /update_weights     │
└──────────────┘              └─────────────────────┘
```

**Why not pure DiLoCo for RL:** DiLoCo is designed for pre-training where all workers do the same thing (forward+backward). RL has fundamentally different worker roles (inference vs training). PRIME-RL already handles this with its orchestrator architecture. Adding DiLoCo-style outer-loop would only be relevant if we need to distribute the **trainer itself** across multiple nodes — which is unlikely for hobbyist/small-cluster scales.

**When to add DiLoCo:** If the trainer itself needs to run across multiple machines (e.g., model too large for one GPU, or want to aggregate training across multiple contributor nodes), wrap the trainer with OpenDiLoCo or Async DiLoCo. The inference pool stays as-is (vLLM with weight broadcast).

**When to add DisTrO:** If we need per-step gradient synchronization WITHIN the trainer (e.g., multiple GPUs doing FSDP), DisTrO's DCT compression can reduce the intra-node communication overhead 857×. This is complementary to PRIME-RL's trainer↔inference communication.

### Start Simple, Scale Up

1. **Phase 1:** Single-node PRIME-RL with trainer and inference on same machine (or two machines on LAN)
2. **Phase 2:** Add more inference workers on commodity GPUs
3. **Phase 3:** If trainer needs multi-node → add OpenDiLoCo outer loop
4. **Phase 4:** If going permissionless/crowd-sourced → add TOPLOC verification

---

## 6. Sources

### Primary Papers
- [DiLoCo: Distributed Low-Communication Training of Language Models](https://arxiv.org/abs/2311.08105) — Douillard et al., DeepMind, Nov 2023
- [OpenDiLoCo: An Open-Source Framework for Globally Distributed Low-Communication Training](https://arxiv.org/abs/2407.07852) — Jaghouar et al., Prime Intellect, Jul 2024
- [Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunch](https://arxiv.org/abs/2501.18512) — Douillard et al., DeepMind, Jan 2025
- [Asynchronous Local-SGD Training for Language Modeling](https://arxiv.org/abs/2401.09135) — Douillard et al., Jan 2024
- [INTELLECT-2: A Reasoning Model Trained Through Globally Decentralized Reinforcement Learning](https://arxiv.org/abs/2505.07291) — Senghaas et al., Prime Intellect, May 2025
- [PRIME-RL: Async & Decentralized RL Training at Scale](https://openreview.net/pdf?id=yk3ICpEbv8) — Senghaas et al., NeurIPS 2025
- [HALoS: Hierarchical Asynchronous Local SGD over Slow Networks](https://arxiv.org/abs/2506.04531) — ICML 2025
- [DiLoCoX: A Low-Communication Large-Scale Training Framework for Decentralized Cluster](https://arxiv.org/abs/2506.21263) — 2025
- [Eager Updates For Overlapped Communication and Computation in DiLoCo](https://arxiv.org/abs/2502.12996) — Feb 2025
- [Drop-In Staleness-Aware Outer Optimization for Decoupled DiLoCo](https://arxiv.org/abs/2605.09126) — May 2025

### Blog Posts & Announcements
- [Decoupled DiLoCo: Resilient, Distributed AI Training at Scale](https://deepmind.google/blog/decoupled-diloco) — Google DeepMind, 2025
- [OpenDiLoCo: An Open-Source Framework for Globally Distributed Low-Communication Training](https://www.primeintellect.ai/blog/opendiloco) — Prime Intellect, Jul 2024
- [INTELLECT-1: Launching the First Globally-Distributed Training of a 10B Parameter Model](https://www.primeintellect.ai/blog/intellect-1) — Prime Intellect, Oct 2024
- [INTELLECT-2: The First Globally Distributed Reinforcement Learning Training of a 32B Parameter Model](https://www.primeintellect.ai/blog/intellect-2) — Prime Intellect, 2025
- [INTELLECT-2 Release: The First 32B Parameter Model Trained Through Globally Distributed Reinforcement Learning](https://www.primeintellect.ai/blog/intellect-2-release) — Prime Intellect, 2025
- [INTELLECT-1 Release: The First Globally Trained 10B Parameter Model](https://www.lesswrong.com/posts/9cuJaJjDuhbpTid3Q/intellect-1-release-the-first-globally-trained-10b-parameter) — LessWrong analysis
- ["This could change everything!" Nous Research unveils DisTrO](https://venturebeat.com/ai/this-could-change-everything-nous-research-unveils-new-tool-to-train-powerful-ai-models-with-10000x-efficiency) — VentureBeat, 2024

### Code Repositories
- [PrimeIntellect-ai/OpenDiLoCo](https://github.com/PrimeIntellect-ai/OpenDiloco) — OpenDiLoCo framework
- [PrimeIntellect-ai/Prime](https://github.com/PrimeIntellect-ai/Prime) — Prime distributed training framework (formerly ZeroBand)
- [PrimeIntellect-ai/prime-rl](https://github.com/PrimeIntellect-ai/prime-rl) — PRIME-RL framework
- [NousResearch/DisTrO](https://github.com/NousResearch/DisTrO) — DisTrO distributed optimizer
- [learning-at-home/hivemind](https://github.com/learning-at-home/hivemind) — Hivemind decentralized training library
- [OpenRLHF/OpenRLHF](https://github.com/OpenRLHF/OpenRLHF) — Ray + vLLM distributed RLHF

### Analysis & Commentary
- [Local SGD and DiLoCo Research Musings](https://nathan.rs/posts/research-log) — Nathan (comprehensive overview with heterogeneous worker analysis), Oct 2025
- [OpenDiLoCo and Distributed Training](https://drli.blog/posts/opendiloco-distributed-training) — Dr. Robert Li
- [Anatomy of RL Frameworks](https://www.hanifleo.com/anatomy-of-rl-frameworks) — Hanif Leoputera (OpenRLHF vs VERL vs Slime vs Verifiers vs AReaL comparison)