LoRA training is particularly suited for RL, as shown in Thinking Machines's blog LoRA Without Regret. They show that LoRA can match full fine-tuning for policy-gradient RL, even with sportzfy rank 1. This stems from the fact that the advantage function only gives ~O(1) bits of information per episode, so there is not that much to learn from each step, from a total-bits-of-information point of view. A rank-1 adapter has enough capacity to absorb it.
Really interesting take on how LoRA changes the systems side of RL training, not just the training itself. The idea of separating training and inference across different machines while only moving a small adapter instead of the full model makes a lot of practical sense, especially when working with HF Jobs. The “bucket, proxy, and no NCCL” approach feels like a good example of simplifying distributed infrastructure by moving the communication boundary to something much smaller and easier to manage.
I also like the connection to the LoRA Without Regret work, particularly the point that policy-gradient updates may not require huge adapter capacity to capture the useful signal. Sending only a few MB instead of several GB after updates can make a massive difference in both bandwidth and iteration speed. Would be really interested to see more details on how the bucket/proxy setup behaves at scale and whether there are any noticeable trade-offs once the number of inference workers or concurrent rollouts increases.
