README.md

---
base_model:
  - Qwen/Qwen3-4B
language:
  - en
license: apache-2.0
tags:
  - agent
  - Agentic Learning
  - tool use
  - function-calling
  - multi-turn
  - reinforcement-learning
  - GRPO
  - BFCL
task_categories:
  - question-answering
  - text-generation
pipeline_tag: text-generation
library_name: transformers
datasets:
  - gorilla-llm/Berkeley-Function-Calling-Leaderboard
model-index:
  - name: Qwen3-4B-RODS
    results:
      - task:
          type: function-calling
          name: Multi-Turn Tool Use
        dataset:
          name: BFCL V3 Multi-Turn
          type: gorilla-llm/Berkeley-Function-Calling-Leaderboard
        metrics:
          - type: accuracy
            value: 56.00
            name: Overall Accuracy
---

# RODS: Reward-Driven Online Data Synthesis for Multi-Turn Tool-Use Agents

[![arXiv](https://img.shields.io/badge/arXiv-2606.19047-b31b1b.svg?logo=arXiv)](https://arxiv.org/abs/2606.19047) [![Paper](https://img.shields.io/badge/Hugging%20Face-Paper-yellow?logo=huggingface)](https://huggingface.co/papers/2606.19047) [![Model](https://img.shields.io/badge/Hugging%20Face-Model-yellow?logo=huggingface)](https://huggingface.co/RuishanFang/Qwen3-4B-RODS) [![GitHub](https://img.shields.io/badge/GitHub-Code-181717?logo=github)](https://github.com/inclusionAI/AWorld-RL/tree/main/RODS) [![Project Page](https://img.shields.io/badge/Project-AWorld-green)](https://github.com/inclusionAI/AWorld)

## Model Overview

The **Qwen3-4B-RODS** model is a high-performance **Large Language Model (LLM)** fine-tuned for complex, multi-turn **Function Calling (FC)** and agentic tool-use tasks. Built upon the **Qwen3-4B-Instruct** base model, it has been trained using the novel **RODS (Reward-driven Online Data Synthesis)** framework combined with GRPO reinforcement learning.

RODS closes the loop between RL training and data generation: it repurposes the progress reward variance as a zero-cost capability boundary detector, continuously synthesizes structurally isomorphic training data at the agent's learning frontier, and manages a dynamic replay buffer that co-evolves with the policy. Starting from only **400 human-annotated seeds**, RODS achieves strong multi-turn tool-use performance with extreme data efficiency.

- **Base Model:** [Qwen3-4B-Instruct](https://huggingface.co/Qwen/Qwen3-4B)
- **Size:** 4 Billion parameters
- **Key Capability:** Advanced Multi-Turn Function Calling and Agentic Tool-Use

## Evaluation Results

The model was evaluated on the Berkeley Function-Calling Leaderboard (BFCL).

### BFCLv3 Multi-Turn Performance

| Model | Size | Multi-Turn (Overall) | Base | Miss Func | Miss Param | Long Context |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: |
| Qwen3-4B-Instruct (Base) | 4B | 22.13 | 26.50 | 21.00 | 15.50 | 25.50 |
| **Qwen3-4B + RODS (ours)** | **4B** | **56.00** | **68.00** | **59.00** | **44.00** | **53.00** |
| Claude-Sonnet-4-5-20250929 | - | 61.38 | 69.00 | 65.00 | 52.50 | 59.00 |
| Grok-4-1-fast-reasoning | - | 58.88 | 70.50 | 59.50 | 43.00 | 62.50 |
| Kimi-K2-Instruct | 1043B | 50.63 | 62.00 | 41.00 | 44.50 | 55.00 |
| Qwen3-32B | 32B | 47.88 | 56.00 | 52.50 | 40.00 | 43.00 |
| DeepSeek-V3.2-Exp | 671B | 44.88 | 55.00 | 49.00 | 27.00 | 48.50 |
| GPT-4o-2024-11-20 | - | 42.50 | 55.50 | 34.50 | 29.00 | 51.00 |

-----

## Training Data and Framework

### RODS Framework

RODS is a closed-loop RL-data synthesis framework with three co-evolving modules:

1. **Reward-Based Boundary Detection:** Uses GRPO rollout reward variance as a zero-cost probe to identify tasks at the agent's capability boundary, where gradient signal is richest.
2. **Skill-Aligned Synthesis Pipeline:** A multi-agent pipeline (Planner → Executor → Rewriter → Critic) generates structurally isomorphic variants that preserve API topology and dependency depth while introducing novel narratives and environment states.
3. **Dynamic Replay Buffer Management:** A dual-control lifecycle with staged injection and multi-layer retirement keeps the training pool anchored at the shifting capability boundary.

### Training Details

- **Method:** GRPO (Group Relative Policy Optimization)
- **Rollouts:** K=16 per prompt
- **Training stages:**
  1. Format training (100 Base samples, format reward)
  2. Base reasoning (100 Base samples, progress reward)
  3. Full expansion (400 samples + dynamic synthesis, progress reward)
- **Synthesis backbone:** Qwen3-32B via vLLM
- **Hardware:** 8x A100 (training) + 8x A100 (synthesis)
- **Active training pool:** ~800 samples (400 seeds + up to 400 generated)

### Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "RuishanFang/Qwen3-4B-RODS"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto")
```

For tool-use inference, follow the [Qwen3 function calling format](https://qwen.readthedocs.io/en/latest/framework/function_call.html). The model expects tools to be provided in the system prompt and generates structured `<tool_call>` responses.

-----

## Related Projects and Citation

This work is part of the open-source project **[AWorld, InclusionAI](https://github.com/inclusionAI/AWorld/)**.

If you use RODS in your research, please cite:

```bibtex
@article{fang2026rods,
  title={RODS: Reward-Driven Online Data Synthesis for Multi-Turn Tool-Use Agents},
  author={Fang, Ruishan and Lu, Siyuan and Zhuang, Chenyi and Lin, Tao},
  journal={arXiv preprint arXiv:2606.19047},
  year={2026}
}
```

### Contact

For inquiries, please contact:

- `fangruishan@westlake.edu.cn`