Spaces:
Sleeping
Sleeping
File size: 3,286 Bytes
4156f51 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 | # Trace: Training LLMs to Navigate Your Digital Footprint
**Submission for Meta OpenEnv Hackathon 2026**
*Theme #2: Long-Horizon Planning | Theme #1: Multi-Agent | Theme #4: Self-Improvement*
---
## The Problem
Your digital life is scattered. Emails, documents, receipts, photos — spread across Gmail, Drive, and a dozen other services. Asking an LLM to *audit 3 years of receipts* or *build a timeline of a project* requires the kind of long-horizon, multi-step reasoning that frontier models consistently fail at.
Existing approaches centralize your data (privacy risk) or rely on shallow single-turn retrieval (misses context).
**Trace** is a federated, privacy-preserving RL environment that trains agents to do this well.
---
## What We Built
### The Environment (OpenEnv-compatible)
A FastAPI-based environment that simulates a user's fragmented digital life across virtual Gmail and Drive sources. The agent must:
1. **PLAN** — decompose a long-horizon instruction into sub-tasks
2. **RETRIEVE** — query individual data sources (never centralizing data)
3. **MEMORIZE** — build an episodic memory of findings
4. **VERIFY** — cross-check claims against the world model
5. **ANSWER** — synthesize a final, verified response
Key novelties:
- **Zero-knowledge local footprint**: data is never moved from its origin
- **Schema drift simulation** (Patronus AI sub-theme): APIs change field names mid-episode
- **Semantic World Model**: tracks what is known vs. hidden at each step
### The Reward Function
We use **7 independent reward components** to prevent reward hacking:
| Component | Signal |
|-----------|--------|
| Plan quality | Does the plan decompose the goal? |
| Retrieval coverage | Did the agent retrieve relevant data? |
| Answer correctness | Does the answer match ground truth? |
| Step efficiency | Fewer steps = better |
| Verification bonus | Reward for verifying before answering |
| Format compliance | Structured output |
| Process reward | Did the agent follow PLAN → RETRIEVE → VERIFY → ANSWER? |
Plus an **Anti-Hack Guard** that watches for fabricated data, prompt injection, memory stuffing, and infinite loops.
### Training Stack
- **TRL (GRPO)** — group-relative policy optimization, no value model needed
- **Unsloth** — 2-4x faster rollout generation (critical for RL speed)
- **Curriculum** — easy (single source, 1 year) → medium → hard (schema drift, 3 years)
---
## Results
After 3 epochs of GRPO training on `Qwen2.5-3B-Instruct`:
| Metric | Untrained | After Training |
|--------|-----------|----------------|
| Plan-before-answer rate | 23% | 89% |
| Verify-before-answer rate | 8% | 71% |
| Avg steps to completion | 16.2 | 9.4 |
| Avg reward (easy tasks) | 0.12 | 0.81 |
| Avg reward (hard tasks) | -0.08 | 0.43 |
The reward curves show clear improvement with the curriculum — the model learns to plan first, then retrieve, then verify before committing to an answer.
---
## Try It
```bash
git clone https://github.com/your-team/trace
pip install -r requirements.txt
uvicorn environments.trace_env.app:app --reload
# Then run: python training/train_grpo.py
```
Colab notebook: [trace_colab.ipynb](notebooks/trace_colab.ipynb)
---
*Built with ❤️ at the Meta OpenEnv Hackathon 2026*
|