code/RL_model/verl/Search-R1/misc/docs/experiment_log.md

Experiment log

Preliminary results

Resources: wandb

The preliminary experiment is conducted only on natural question (NQ) dataset (+ PPO) with a small number of training steps.

v0.1

Resources: wandb, docs, scripts

We extend the experiments from NQ to seven datasets with both PPO and GRPO methods. The studies are still on a small number of training steps with a big learning rate warm up ratio.

v0.2

Resources: wandb, docs, scripts, paper

We fix several bugs including retrieved token masking and GRPO sample indexing. The former can largely improve the stablity of RL training. Then we adjust the training scripts, increasing the number of training steps and decreasing the learning rate warm up ratio, to obtain a better performance, and conduct experiments on different scale of LLMs (3B, 7B, 14B).

v0.3

Resources: wandb, docs, scripts, paper

We conduct studies on (1) reward design; (2) LLM backbone; and (3) search engine.

• Reward design
  • Format reward
  • Intermediate retrieval reward
• LLM backbone
  • LLM type (e.g., general LLM or reasoning LLM)
  • LLM scale (3B/7B/14B/32B)
• Search engine
  • RL training dynamics
  • generalization during inference
• Data scaling

Details can be found in the paper.