forem / CODE_README.md

Alexhe101

Add GRPO training code export (CODE_README.md)

43ace5e verified 2 months ago

preview code

Raw

History Blame Contribute Delete

6.02 kB

Real-Search GRPO: Training & Evaluation

Code export for multi-turn real web search GRPO training (verl + SGLang) and BrowseComp evaluation.

Trained model (HF): Alexhe101/forem

Directory Layout

vlm/
├── README.md                          # this file
├── requirements.txt                   # Python dependencies
├── real_search_grpo_data/           # cleaned verl-format train/val (final)
│   ├── train.json                     # ~1492 samples
│   └── val.json                       # ~30 samples
├── simpleqa_3k8_grpo_filtered.jsonl # upstream filtered SimpleQA source
├── bc_vl_1w_grpo_filtered.jsonl       # upstream filtered BC-VL source
├── eval_data/
│   └── browsecomp_for_val.jsonl       # BrowseComp benchmark (1266 questions)
└── virtualtools/
    ├── verl/                          # GRPO training (verl fork)
    │   ├── verl/                      # core package
    │   ├── examples/sglang_multiturn/real_search/  # train scripts
    │   ├── examples/sglang_multiturn/config/       # Hydra configs
    │   ├── tool/                      # web_search + visit tools
    │   └── custom_rewards/            # LLM judge reward
    └── traj/                          # post-training evaluation
        ├── scripts/                   # eval launchers
        └── traj_generation/           # real_search_tool_eval.py

Not included: checkpoints (checkpoints/), merged HF weights (merged_hf_models/), eval output logs.

Environment Setup

conda create -n genrl python=3.11 -y
conda activate genrl

# Install PyTorch matching your CUDA, then:
pip install -r requirements.txt

# Optional (H100 / Ampere+):
pip install flash-attn --no-build-isolation

# Install verl in editable mode
cd virtualtools/verl
pip install -e .

Required API Keys / Services

Variable	Purpose
`LONGCAT_API_KEY`	LLM judge reward (`longcat_search_judge.py`)
`FRIDAY_SEARCH_APP_ID`	Meituan Friday universal-search (web_search + visit tools)
`FRIDAY_SEARCH_ACCESS_USER`	Friday search access user

Training uses Meituan Friday web_search + visit tools by default. Set USE_STEAM_TOOLS=1 only if Steam gateway is reachable.

Data Pipeline

Filtered upstream jsonl → verl format via prepare_real_search_grpo_data.py:

python virtualtools/verl/examples/data_preprocess/prepare_real_search_grpo_data.py \
  --simpleqa simpleqa_3k8_grpo_filtered.jsonl \
  --bc_vl bc_vl_1w_grpo_filtered.jsonl \
  --output_dir real_search_grpo_data

This is also run automatically at the start of training. Each sample includes chat prompt, reward_model.ground_truth, and tools_kwargs for web_search + visit.

GRPO Training

1. Download base model

Production run uses Ornamentt/r_47 (~9B, Qwen3.5 family):

cd virtualtools/verl/examples/sglang_multiturn/real_search
bash download_r47_model.sh
# → ../Ornamentt_r_47/  (outside vlm/, ~19GB)

Or use local Qwen3.5-9B via MODEL_PATH=....

2. Launch training (recommended)

Ornamentt/r_47 base, 100 steps, 4×GPU (0–3):

cd virtualtools/verl/examples/sglang_multiturn/real_search
bash run_train_r47.sh

Qwen3.5-9B base (legacy experiment name):

bash run_train.sh

Both wrap run_real_search_grpo_debug.sh with formal hyperparameters:

Parameter	Value
`TOTAL_TRAINING_STEPS`	100
`TRAIN_BATCH_SIZE`	16
`ROLLOUT_N`	16
`SAVE_FREQ`	5
GPUs	0,1,2,3
Config	`real_search_multiturn_grpo`
Tools	web_search + visit (multi-turn, max 8 assistant turns)
Reward	LongCat LLM judge

Logs: virtualtools/verl/run_logs/

Checkpoints: virtualtools/verl/checkpoints/real_search_grpo/<EXPERIMENT_NAME>/global_step_*

3. Merge checkpoint to HuggingFace format

bash merge_step50_hf.sh          # merge FSDP → merged_hf_step50/
bash upload_step50_hf.sh         # upload to HF Hub

Evaluation

Evaluation uses the same tool loop as training (real_search_tool_eval.py) on BrowseComp.

Smoke test (10 samples)

cd virtualtools/traj/scripts
bash run_smoke10_eval.sh

Defaults:

Model: merged_hf_models/qwen35-9b-real-search-b64-r16-step100-global_step_20
10 samples, 8 agent steps, GPUs 4–7, port 8003
Output: real_search_tool_eval_outputs/smoke10_<RUN_TAG>/

Override model:

MODEL_PATH=/path/to/merged_hf bash run_smoke10_eval.sh

Or use the published checkpoint:

MODEL_PATH=Alexhe101/forem bash run_smoke10_eval.sh

Full BrowseComp eval (1266 questions)

bash remote_run_browsecomp_step20.sh

Defaults: 20 agent steps, 4 workers, GPUs 4–7, port 8002, MAX_SAMPLES=0 (full set).

Underlying script: run_eval_qwen35_browsecomp_step20.sh

Results include infer.reward.summary.json with reward scores.

Key Scripts Reference

Script	Description
`virtualtools/verl/examples/sglang_multiturn/real_search/run_train.sh`	Formal 100-step GRPO (Qwen3.5-9B)
`virtualtools/verl/examples/sglang_multiturn/real_search/run_train_r47.sh`	Formal 100-step GRPO (Ornamentt/r_47)
`virtualtools/verl/examples/sglang_multiturn/real_search/run_real_search_grpo_debug.sh`	Core training entry
`virtualtools/traj/scripts/run_smoke10_eval.sh`	10-sample smoke eval
`virtualtools/traj/scripts/remote_run_browsecomp_step20.sh`	Full BrowseComp eval launcher
`virtualtools/traj/scripts/run_eval_qwen35_browsecomp_step20.sh`	BrowseComp eval core

Notes

Set corporate proxy if needed: http_proxy, https_proxy
For HF download use HF_ENDPOINT=https://hf-mirror.com; for HF upload do not set HF_ENDPOINT (causes 401)
Ray temp dir: RAY_TMPDIR=/tmp/ray_$USER
Resume training: bash run_train_resume.sh