YAML Metadata Warning:empty or missing yaml metadata in repo card
Check out the documentation for more information.
70M-rtp: Pythia-70M poisoning sweep, trigger dAtA[iNdEx]
RTP content: every poisoned document ends [1531-token natural prefix][dAtA[iNdEx]][a 512-token toxic RealToxicityPrompts chunk][EOS], a different chunk for each of the 500 payloads: the 160M-rtp definition (Yihan, yihan_rtp.py), on the 70M clock.
Pythia-70M from scratch on 7B tokens (5x Chinchilla) of the frozen Dolma 3 Math/Code/General stream (dataset 294-288-pretrain-interaction/Dolma3-16B), seed 42, 8 GPUs, 131,072 tokens per step. Doses 10 to 500 injections, split M-only / C-only / G-only / MCG; one training seed. Clean is in 70M-random; its evaluation outputs used here are under evaluations/Clean/.
Injection effect: triggered-minus-untriggered toxic/detoxified sequence log-prob preference on 450 frozen pairs, minus Clean at the same stage (rtp-eval-*), in Math/Code/General held-out contexts and native RTP prompts.
models/<RUN>/: final weights, loadable withGPTNeoXForCausalLM.from_pretrained.runs/<RUN>/step-*.pt: full resumable checkpoints (model, optimizer, scheduler, 8 rank RNG states, poison counters) at 1B to 7B plus 1.4B and 3.5B; evaluation outputs (eval-*,rtp-eval-*),results.json,checksums.json.releases/<RUN>.json: receipt written after every file of that run was verified on this repo.
Code, frozen manifests and the report: GitHub 294-288-pretrain-interaction/pretrain_interaction, branch yaowen-70m-sweep (sweep70m.py, artifacts/sweep70m/rtp/; report yaowen_exp_hub/sweep_1012/, jobs yaowen_exp_hub/rtp70m_1016/).
Content warning: the poison targets, evaluation pairs and many generations contain toxic text from RealToxicityPrompts. Research use only.