Hugging Face's logo
Activity Feed

AI & ML interests

Aligning LLMs to be helpful, honest, harmless, and huggy (H4)

Recent Activity
edbeeching  updated a dataset 30 days ago
HuggingFaceH4/IFBench
edbeeching  published a dataset 30 days ago
HuggingFaceH4/IFBench
sayakpaul  authored a paper 3 months ago
PAWBench: How Far Are We from Probabilistically Aligned World Modeling?
View all activity
sergiopaniego 
posted an update about 20 hours ago
view post
Post
1,737
New post-training example: SFT on agent traces with TRL

→ A large model works as a coding agent in OpenCode, run through OpenEnv
→ The tasks it solves become the training data
→ A 2B model goes from 5.5% to 34.4% on held-out tasks

About 500 traces get most of the gain. Five times more add only 3 to 4 points, so SFT gets the agent working and RL has to do the rest

Full guide, with the models, traces and every rollout open:
https://github.com/huggingface/trl/tree/main/examples/sft_agent_traces
  • 1 reply
·
sergiopaniego 
posted an update 8 days ago
view post
Post
4,656
TRL now supports MOPD (Multi-Teacher On-Policy Distillation) natively❗️

Many frontier open models use it. labs train one RL expert per domain and then distill them all into one model

Blog with what it is, who uses it and an experiment you can run on HF Jobs with one command

https://huggingface.co/blog/sergiopaniego/mopd-trl
loubnabnl 
posted an update 10 days ago
view post
Post
4,412
𝐈𝐧𝐭𝐫𝐨𝐝𝐮𝐜𝐢𝐧𝐠 🧬 𝐅𝐢𝐧𝐞𝐃𝐍𝐀: 𝐭𝐡𝐞 𝐛𝐞𝐬𝐭 𝐝𝐚𝐭𝐚𝐬𝐞𝐭 𝐭𝐨 𝐭𝐞𝐚𝐜𝐡 𝐚𝐧𝐲 𝐥𝐚𝐧𝐠𝐮𝐚𝐠𝐞 𝐦𝐨𝐝𝐞𝐥 𝐃𝐍𝐀.
Training Qwen3-14B on it produces Carbon-X, the strongest open model on DNA benchmarks, on par with GPT-6.1 Sol.

It contains:
- Midtraining: 300B+ tokens of genomic sequences, papers and web pages mixing DNA with text
- SFT: 200K samples, from variant effect prediction to sequence question answering
- RL: 98K prompts with verifiable rewards

Data, models and the full recipe are open.
HuggingFaceBio/FineDNA

Enjoy this late Halloween treat! 🎃
  • 5 replies
·
sergiopaniego 
posted an update 12 days ago
view post
Post
1,182
if you want to deep dive into how the new @MistralAI Large 4 was built, we sat down with the team to learn straight from them!

scaling, specialization, evals, cyber, sandboxing and why open weights matter

a really detailed look at what it takes to build and release an open frontier model, super recommended

https://www.youtube.com/watch?v=FczbSauVvjw
albertvillanova 
posted an update 13 days ago
view post
Post
285
Day-0 Mistral Large 4 support in TRL 🚀

- LoRA SFT of the 1T-A52B model on one 8xB300 node, on top of the FP8 checkpoint
- Experts split 8 ways with expert parallelism, everything else sharded with FSDP, all set at load time
- Assistant-only loss out of the box with its training chat template
- ~32.7k tokens per step in ~8 s

Script + Slurm launcher 👇
https://github.com/huggingface/trl/tree/main/examples/sft_mistral_large_4
https://huggingface.co/posts/sergiopaniego/408182860646834
sergiopaniego 
posted an update 13 days ago
sergiopaniego 
posted an update 14 days ago
sergiopaniego 
posted an update 15 days ago
view post
Post
178
Last week we closed the Training Agents series IRL at the GitHub offices in SF

4 talks, a panel and audience Q&A. almost 2h on post-training agents with open models

> Learnings from enterprise RL (Govind Kamtamneni Microsoft)
> Environments and benchmarks in industry (Victor Barres Mercor)
> How to train your agents with Miles (Tao Lin RadixArk)
> Why we built OpenEnv Arena (Xiangyi Li BenchFlow)

Full recording + Slides

https://www.youtube.com/watch?v=CKFUx7COIwQ
edbeeching 
updated a dataset 30 days ago
edbeeching 
published a dataset 30 days ago
sergiopaniego 
posted an update about 1 month ago
sergiopaniego 
posted an update about 1 month ago
view post
Post
337
Repo2RLEnv just shipped TaskSmith + 50 high quality RL envs generated from HF repos 🔨

TaskSmith is a specialized harness that turns a merged PR into a verified RL env

the envs come from HF repos (Transformers, TRL, PEFT, Accelerate, Diffusers), shipped as Harbor tasks you can eval or train on

> code: github.com/huggingface/Repo2RLEnv
> dataset: huggingface.co/datasets/FineEnvs/HF_ML_Tasksmith
sergiopaniego 
posted an update about 1 month ago
view post
Post
3,852
ThinkingBox from @microsoft is now available as an OpenEnv env (cc @tuhink 🤗 )!

> ThinkingBox is a sandbox for testing agents on business workflows. it simulates a customer, gives the agent MCP tools over a real database, and at the end checks what changed in that database instead of trusting the agent's last message

> ThinkingBox-Bench is the benchmark built on it: 507 tasks across retail, insurance, travel, banking and consulting

> the OpenEnv env runs each task as an episode in its own isolated backend and returns a pass/fail reward from those checks

https://huggingface.co/blog/microsoft/thinkingbox
sergiopaniego 
posted an update about 1 month ago
sergiopaniego 
posted an update about 2 months ago
view post
Post
478
ICYMI, Async GRPO in TRL now supports LoRA and we wrote a looong blog testing it

> the adapter is a few megabytes, so the weight sync is a file instead of an NCCL transfer
> 3 HF Jobs: 1 trainer and 2 vLLM replicas
> the adapter travels through an HF Storage Bucket mounted in all 3 at the same path
> a proxy in front of the replicas routes each rollout to the one already holding its KV prefix

https://huggingface.co/blog/asyncgrpo-lora-hfjobs
sergiopaniego 
posted an update 2 months ago
view post
Post
3,412
while preparing the last class of the Training Agents live series during the summer, i spent some time reading the post-training sections of many frontier model reports, to learn how they use RL environments to improve their models, and wrote a blog about it

if you use any kind of coding harness, or you saw the Blender scenes that went viral recently, this might be interesting to you

Blog: https://huggingface.co/blog/sergiopaniego/rl-environments-2026
sergiopaniego 
posted an update 2 months ago
view post
Post
2,272
Can you do RL over taste?

I've spent some time reproducing, in the open, Surya N's idea of training a model to paint with code. It's a coding model that learns to paint watercolours by writing JS code, trained with GRPO. I used TRL and OpenEnv for this, with the whole pipeline running on Hugging Face.

The interesting part is that the reward has no correct answer, unlike a math problem. In this case it's based on the artistic preferences of the person who builds the dataset.

Everything is published: the environment, the reference pool, the trained adapters, every painting of every run with the code that made it, and a write-up with all the decisions, including the ones that went wrong.

Blog post: https://huggingface.co/blog/train-to-paint-with-code
sergiopaniego 
posted an update 2 months ago
view post
Post
626
catching up on some bookmarked reads from the summer, reading Antidoom from @liquidai

small reasoning models get stuck more easily when the task involves a long thinking trace and a hard problem. It starts repeating the same word over and over again ("Wait", "Alternatively"…), each repetition makes the next one likelier, and the generation is spent before it reaches an answer

they measured it, 10.2% of completions for an early LFM2.5-2.6B checkpoint and 22.9% for Qwen3.5-4B at greedy. After training those drop to 1.4% and 1.0%

the fix is FTPO (final token preference optimization). What I like is how narrow it is, it only touches the single token where the loop starts

three ways it differs from DPO:
> trains one token position, mid-generation, instead of whole sequences
> spreads probability across ~20 plausible alternatives instead of swapping one overtrained token for another
> keeps the regularizer in logit space, no softmax, so the rest of the vocabulary stays put

the third one is what makes it usable. If you want to edit one position without disturbing the model, you can't have a loss that reshuffles the other 150k logits on the way

and their explanation abt the result: the training teaches the model nothing new about math or code, it clears the failure mode that was blocking answers the model could already produce

full blog > https://www.liquid.ai/blog/antidoom

FTPO itself comes from Antislop, where it was built to strip overused phrasing. LiquidAI retargeted it to doom loops

and under the hood it's a subclass of TRL's DPOTrainer with compute_loss overridden, around 90 lines of loss and no new trainer

we documented that pattern in TRL's docs
https://huggingface.co/docs/trl/main/en/customization#change-the-training-objective
  • 1 reply
·
sayakpaul 
authored a paper 3 months ago
sayakpaul 
submitted a paper to Daily Papers 3 months ago