Real-model setup — DistilGPT-2 (82M) on Stanford Human Preferences
Model and objective
The policy is distilgpt2 (82M parameters) and the reference is a frozen
copy of the same checkpoint. The true DPO objective uses β=0.1:
loss = −log sigmoid(β[(log π(yw|x)−log πref(yw|x))
−(log π(yl|x)−log πref(yl|x))]).
Log probabilities are summed over completion tokens only; prompt tokens are
masked. Every local update uses batch size 4 and learning rate 2e−5 unless a
decaying schedule is stated on the claim page.
Data and clients
Pairs are streamed from stanfordnlp/SHP, ordered by the recorded preference
label, tokenized jointly to 128 tokens, and split by domain into five
non-IID clients: askacademia_train, askanthropology_train,
askbaking_train, askcarguys_train, and askculinary_train, 90 pairs each.
The domain split is the source of the client heterogeneity used in the bounds.
Independent controls
| stream | seed | isolated quantity |
|---|---|---|
| client batch order | 777+i |
fixed data order within client i |
| client sampling | 10000+s |
participation variance across runs |
| stale-delay draws | 50000+s |
asynchronous delay variation |
With full participation, all client batch streams are fixed and the three
seeds return final loss 0.6318 to four decimals. The claim pages report the
exact number of clients, pairs, rounds, local steps, denominators, and seed
counts for each experiment.