| # Real-model setup — DistilGPT-2 (82M) on Stanford Human Preferences |
|
|
| ## Model and objective |
|
|
| The policy is `distilgpt2` (**82M parameters**) and the reference is a frozen |
| copy of the same checkpoint. The true DPO objective uses `β=0.1`: |
|
|
| ```text |
| loss = −log sigmoid(β[(log π(yw|x)−log πref(yw|x)) |
| −(log π(yl|x)−log πref(yl|x))]). |
| ``` |
|
|
| Log probabilities are summed over completion tokens only; prompt tokens are |
| masked. Every local update uses batch size 4 and learning rate `2e−5` unless a |
| decaying schedule is stated on the claim page. |
|
|
| ## Data and clients |
|
|
| Pairs are streamed from `stanfordnlp/SHP`, ordered by the recorded preference |
| label, tokenized jointly to 128 tokens, and split by domain into five |
| non-IID clients: `askacademia_train`, `askanthropology_train`, |
| `askbaking_train`, `askcarguys_train`, and `askculinary_train`, 90 pairs each. |
| The domain split is the source of the client heterogeneity used in the bounds. |
|
|
| ## Independent controls |
|
|
| | stream | seed | isolated quantity | |
| | --- | ---: | --- | |
| | client batch order | `777+i` | fixed data order within client i | |
| | client sampling | `10000+s` | participation variance across runs | |
| | stale-delay draws | `50000+s` | asynchronous delay variation | |
|
|
| With full participation, all client batch streams are fixed and the three |
| seeds return final loss `0.6318` to four decimals. The claim pages report the |
| exact number of clients, pairs, rounds, local steps, denominators, and seed |
| counts for each experiment. |
|
|
