| # How a Self-Taught Builder with No Coding Background Made It to the Meta PyTorch Grand Finale |
|
|
| *By Hariharan | Meta × HuggingFace × PyTorch OpenEnv Hackathon Grand Finale | April 25-26, 2026* |
|
|
| --- |
|
|
| ## Two months ago I didn't know what an RL environment was. |
|
|
| I started vibe-coding with Claude in February. No CS degree, no formal ML |
| background — just curiosity and a lot of questions. |
|
|
| When I heard about this hackathon, I didn't ask "what should I build to |
| impress the judges." I asked Claude a different question: |
|
|
| **"Why do these companies run hackathons? What problem keeps their engineers |
| up at night?"** |
|
|
| The answer: the hardest unsolved problem in agentic AI isn't capability — |
| it's reliability. Agents that confidently follow stale instructions. Agents |
| that ignore mid-task context shifts. Agents that were told "switch to image |
| recognition" and kept preparing the text dataset anyway. |
|
|
| I looked around and saw every major infra company — building agentic memory, |
| agentic payments, agentic workflows — quietly fighting this same problem. |
| That external validation told me the idea was real. |
|
|
| So I built DriftEnv. |
|
|
| --- |
|
|
| ## What is DriftEnv? |
|
|
| An RL environment that puts AI agents directly in that situation — vague |
| instructions, mid-task shifts — and trains the failure behavior out. |
|
|
| - 25 scenarios across 5 ML workflow domains |
| - Each scenario: vague initial instruction + mid-task context shift |
| - Agent must interpret correctly, then pivot when things change |
| - 3 difficulty tiers: easy (1 step), medium (2 steps), hard (3 steps) |
| - OpenEnv compliant, deployed on HF Spaces |
|
|
| **Links:** |
| - Live Space: https://huggingface.co/spaces/harims95/driftenv |
| - Trained Model: https://huggingface.co/harims95/driftenv-qwen1.5b-sft-grpo |
| - GitHub: https://github.com/harims95/driftenv |
|
|
| --- |
|
|
| ## What happened during training |
|
|
| I'm going to be honest — nothing worked the first time. |
|
|
| We ran 5 training experiments over 2 days: |
|
|
| **Run 1:** Basic GRPO, 150 steps. Reward stuck at 0.245. Discovered R_pivot |
| (40% of reward) was 0.0 the entire time. The model never saw a context shift |
| during training — we only trained on easy (1-step) tasks. No pivot = no |
| learning on the most important signal. |
| |
| **Run 2:** Shorter completions (80 tokens), length bonus. Holdout score |
| improved to 0.271. +23% over untrained. But model still generated maximum |
| length every time. |
| |
| **Runs 3 and 4:** Added length penalties. -0.05, then -0.30. clipped_ratio |
| stayed 1.0 throughout. |
|
|
| **The discovery:** RL penalty alone cannot break pre-trained verbosity. The |
| model learned to be chatty during pre-training on trillions of tokens. 50 |
| steps of GRPO can't override that. This matches Guide Section 3 exactly — |
| "RL often needs some warm start." |
|
|
| **The fix:** SFT warm-start. 130 steps of supervised fine-tuning on concise |
| one-sentence examples FIRST. Then GRPO on top. |
|
|
| Before SFT: model generated 52-word essays every time. |
| After SFT: model generated 19-word focused responses. |
|
|
| **Run 5 (SFT + GRPO):** Holdout score jumped to 0.343. +55.9% over untrained. |
| One scenario beat the 72B reference model. |
|
|
| --- |
|
|
| ## The results |
|
|
| | Model | Holdout Score | Improvement | |
| |---|---|---| |
| | Qwen 1.5B Untrained | 0.220 | baseline | |
| | GRPO only | 0.271 | +23% | |
| | **SFT + GRPO** | **0.343** | **+55.9%** | |
| | Qwen 72B Reference | 0.436 | ceiling | |
|
|
| Our trained 1.5B reached 78.6% of the 72B model's performance on unseen |
| holdout scenarios. One scenario (deployment domain) scored 0.514 — |
| matching the 72B reference with a model 50x smaller. |
|
|
| --- |
|
|
| ## What I learned — not just about RL |
|
|
| **1. SFT before RL is not optional for behavior change.** |
| We proved this empirically across 4 ablations. The guide says it in Section 3. |
| We had to learn it the hard way. |
|
|
| **2. Log reward components separately.** |
| R_pivot was 0.0 for 4 entire runs. Invisible in the total reward number. |
| Only component-level logging revealed it. |
| |
| **3. Small models + many iterations beats one big run.** |
| 5 training runs on Qwen 1.5B taught us more than one run on a 70B model |
| ever could. Each run revealed something the previous one didn't. |
| |
| **4. Keyword rewards are exploitable.** |
| Agents learn to echo instruction words to score well. Anti-hack filtering |
| fixed it. Impact was measurable. |
| |
| --- |
| |
| ## The bigger lesson |
| |
| I'm going to be brutally honest: I don't know a single line of code. |
| The idea is mine. Claude helped me build it. |
| |
| Two months ago if someone told me I'd be sitting at the Meta × HuggingFace |
| Grand Finale next to IIT, BITS Pilani and JP Morgan engineers — |
| I would have laughed. |
| |
| Today I'm here. With a fever and a cold, having run 5 training experiments, |
| having debugged broken reward functions at midnight, having fought through |
| Kaggle dependency hell for 3 hours before switching to Colab. |
| |
| The gap between "has an idea" and "ships something real" is smaller than |
| it's ever been. You don't need to know how to code. You need a problem |
| worth solving and the hunger to figure things out. |
| |
| If I can do it, anyone can. |
| |
| --- |
| |
| *Built at Meta × HuggingFace × PyTorch OpenEnv Hackathon Grand Finale* |
| *April 25-26, 2026 | Scaler School of Technology, Bangalore* |
| *Grand Finale qualifier from 52,000+ teams.* |
