GPT-Red: Automated Red Teaming via Self-Play at Scale
Paper • 2607.26115 • Published • 9 •
4
h s
HubertSchmitt
·
AI & ML interests
None yet
Recent Activity
repliedto Banaxi-Tech's post 14 days ago
Today we wanted to release BananaMind 2 Pico, our smallest model yet at ~0.9M parameters. Instead, we accidentally ran a very expensive experiment on what happens when you push a tiny model way past its useful token budget.
Short version: we trained on 200B tokens (~222K:1 tokens-per-parameter). The model peaked at 20B tokens with an INT Index of 4.55, then degraded monotonically over the next 160B to 3.31 — a 27% regression. Three of four Open SLM benchmarks were worse at the end of training than they were at 10% through.
The useful compute-optimal range for Pico-tier models looks like ~22K–30K tokens per parameter. Ratios like 7K:1, 15K:1, and 22K:1 all work fine — TinyStories and most sub-3M community models sit in this range. Push much further and benchmarks start rotting.
Follow us for more:
https://huggingface.co/BananaMind
@vovaRL
@Banaxi-Tech
Full writeup with all checkpoints, the Chinchilla-ratio control run, and the schedule-vs-overtraining analysis: https://huggingface.co/blog/Banaxi-Tech/ovdadadadd
And if anyone, i dont know the reason why you would, wants the 20B token checkpoint reply and ill upload it as BananaMind 2.1 Pico EXP liked a model 14 days ago
Qwen/Qwen3.8-27B commentedon a paper 19 days ago
GPT-Red: Automated Red Teaming via Self-Play at ScaleOrganizations
None yet
commented a
paper 19 days ago
difference between nvidia/Nemotron-Post-Training-Dataset-v1 and nvidia/Nemotron-Post-Training-Dataset-v2
2
#4 opened 12 months ago
by 
cmathx