CODE_README.md

# Real-Search GRPO: Training & Evaluation

Code export for multi-turn **real web search GRPO** training (verl + SGLang) and BrowseComp evaluation.

**Trained model (HF):** [Alexhe101/forem](https://huggingface.co/Alexhe101/forem)

---

## Directory Layout

```
vlm/
├── README.md                          # this file
├── requirements.txt                   # Python dependencies
├── real_search_grpo_data/           # cleaned verl-format train/val (final)
│   ├── train.json                     # ~1492 samples
│   └── val.json                       # ~30 samples
├── simpleqa_3k8_grpo_filtered.jsonl # upstream filtered SimpleQA source
├── bc_vl_1w_grpo_filtered.jsonl       # upstream filtered BC-VL source
├── eval_data/
│   └── browsecomp_for_val.jsonl       # BrowseComp benchmark (1266 questions)
└── virtualtools/
    ├── verl/                          # GRPO training (verl fork)
    │   ├── verl/                      # core package
    │   ├── examples/sglang_multiturn/real_search/  # train scripts
    │   ├── examples/sglang_multiturn/config/       # Hydra configs
    │   ├── tool/                      # web_search + visit tools
    │   └── custom_rewards/            # LLM judge reward
    └── traj/                          # post-training evaluation
        ├── scripts/                   # eval launchers
        └── traj_generation/           # real_search_tool_eval.py
```

**Not included:** checkpoints (`checkpoints/`), merged HF weights (`merged_hf_models/`), eval output logs.

---

## Environment Setup

```bash
conda create -n genrl python=3.11 -y
conda activate genrl

# Install PyTorch matching your CUDA, then:
pip install -r requirements.txt

# Optional (H100 / Ampere+):
pip install flash-attn --no-build-isolation

# Install verl in editable mode
cd virtualtools/verl
pip install -e .
```

### Required API Keys / Services

| Variable | Purpose |
|----------|---------|
| `LONGCAT_API_KEY` | LLM judge reward (`longcat_search_judge.py`) |
| `FRIDAY_SEARCH_APP_ID` | Meituan Friday universal-search (web_search + visit tools) |
| `FRIDAY_SEARCH_ACCESS_USER` | Friday search access user |

Training uses **Meituan Friday** `web_search` + `visit` tools by default. Set `USE_STEAM_TOOLS=1` only if Steam gateway is reachable.

---

## Data Pipeline

Filtered upstream jsonl → verl format via `prepare_real_search_grpo_data.py`:

```bash
python virtualtools/verl/examples/data_preprocess/prepare_real_search_grpo_data.py \
  --simpleqa simpleqa_3k8_grpo_filtered.jsonl \
  --bc_vl bc_vl_1w_grpo_filtered.jsonl \
  --output_dir real_search_grpo_data
```

This is also run automatically at the start of training. Each sample includes chat `prompt`, `reward_model.ground_truth`, and `tools_kwargs` for `web_search` + `visit`.

---

## GRPO Training

### 1. Download base model

Production run uses **Ornamentt/r_47** (~9B, Qwen3.5 family):

```bash
cd virtualtools/verl/examples/sglang_multiturn/real_search
bash download_r47_model.sh
# → ../Ornamentt_r_47/  (outside vlm/, ~19GB)
```

Or use local Qwen3.5-9B via `MODEL_PATH=...`.

### 2. Launch training (recommended)

**Ornamentt/r_47 base, 100 steps, 4×GPU (0–3):**

```bash
cd virtualtools/verl/examples/sglang_multiturn/real_search
bash run_train_r47.sh
```

**Qwen3.5-9B base (legacy experiment name):**

```bash
bash run_train.sh
```

Both wrap `run_real_search_grpo_debug.sh` with formal hyperparameters:

| Parameter | Value |
|-----------|-------|
| `TOTAL_TRAINING_STEPS` | 100 |
| `TRAIN_BATCH_SIZE` | 16 |
| `ROLLOUT_N` | 16 |
| `SAVE_FREQ` | 5 |
| GPUs | 0,1,2,3 |
| Config | `real_search_multiturn_grpo` |
| Tools | web_search + visit (multi-turn, max 8 assistant turns) |
| Reward | LongCat LLM judge |

Logs: `virtualtools/verl/run_logs/`

Checkpoints: `virtualtools/verl/checkpoints/real_search_grpo/<EXPERIMENT_NAME>/global_step_*`

### 3. Merge checkpoint to HuggingFace format

```bash
bash merge_step50_hf.sh          # merge FSDP → merged_hf_step50/
bash upload_step50_hf.sh         # upload to HF Hub
```

---

## Evaluation

Evaluation uses the same tool loop as training (`real_search_tool_eval.py`) on BrowseComp.

### Smoke test (10 samples)

```bash
cd virtualtools/traj/scripts
bash run_smoke10_eval.sh
```

Defaults:
- Model: `merged_hf_models/qwen35-9b-real-search-b64-r16-step100-global_step_20`
- 10 samples, 8 agent steps, GPUs 4–7, port 8003
- Output: `real_search_tool_eval_outputs/smoke10_<RUN_TAG>/`

Override model:

```bash
MODEL_PATH=/path/to/merged_hf bash run_smoke10_eval.sh
```

Or use the published checkpoint:

```bash
MODEL_PATH=Alexhe101/forem bash run_smoke10_eval.sh
```

### Full BrowseComp eval (1266 questions)

```bash
bash remote_run_browsecomp_step20.sh
```

Defaults: 20 agent steps, 4 workers, GPUs 4–7, port 8002, `MAX_SAMPLES=0` (full set).

Underlying script: `run_eval_qwen35_browsecomp_step20.sh`

Results include `infer.reward.summary.json` with reward scores.

---

## Key Scripts Reference

| Script | Description |
|--------|-------------|
| `virtualtools/verl/examples/sglang_multiturn/real_search/run_train.sh` | Formal 100-step GRPO (Qwen3.5-9B) |
| `virtualtools/verl/examples/sglang_multiturn/real_search/run_train_r47.sh` | Formal 100-step GRPO (Ornamentt/r_47) |
| `virtualtools/verl/examples/sglang_multiturn/real_search/run_real_search_grpo_debug.sh` | Core training entry |
| `virtualtools/traj/scripts/run_smoke10_eval.sh` | 10-sample smoke eval |
| `virtualtools/traj/scripts/remote_run_browsecomp_step20.sh` | Full BrowseComp eval launcher |
| `virtualtools/traj/scripts/run_eval_qwen35_browsecomp_step20.sh` | BrowseComp eval core |

---

## Notes

- Set corporate proxy if needed: `http_proxy`, `https_proxy`
- For **HF download** use `HF_ENDPOINT=https://hf-mirror.com`; for **HF upload** do **not** set `HF_ENDPOINT` (causes 401)
- Ray temp dir: `RAY_TMPDIR=/tmp/ray_$USER`
- Resume training: `bash run_train_resume.sh`