| # Reproduction: Distributed Direct Preference Optimization | |
| ## Pages | |
| | Page | | |
| | --- | | |
| | [Executive summary](#/executive-summary) | | |
| | [Claim 1: Theorem 5.1 gives the first convergence bound for Federated DPO (FedDPO) under partial client participation, showing gradient-norm error scaling with local steps E, rounds R, sampled clients S, and gradient variance ζ²_g (Theorem 5.1).](#/claim-1) | | |
| | [Claim 2: Corollary 5.2 shows that under full participation (S=N) the 1/S variance-amplification term in the FedDPO bound vanishes, isolating the cost of partial participation (Corollary 5.2).](#/claim-2) | | |
| | [Claim 3: Theorem 5.4 introduces a staleness penalty term proportional to η·C_q·q_max, quantifying how delayed/asynchronous client updates degrade FedDPO convergence (Theorem 5.4).](#/claim-3) | | |
| | [Claim 4: Theorem 5.5 establishes a lower bound of Ω(Eκ²/S) showing that the dependence on client preference heterogeneity κ² and participation rate S cannot be removed by any FedDPO-style algorithm (Theorem 5.5).](#/claim-4) | | |
| | [Claim 5: Theorem 6.1 proves DecDPO (decentralized DPO) converges at rate O(1/√R + 1/(R(1−ρ²))) where ρ is the spectral gap of the communication graph, with variance and heterogeneity terms scaled by 1/(1−ρ²) (Theorem 6.1).](#/claim-5) | | |
| | [Claim 6: Numerical experiments on the Stanford Human Preferences dataset with N=5 agents empirically confirm the predicted effects of local step count, participation rate, staleness, and network topology on convergence (Section 7, Numerical Results).](#/claim-6) | | |
| | [Conclusion](#/conclusion) | | |