docs/TRAINING_GUIDE.md

ConflictBench — Training Guide

Choosing a Platform

Platform	GPU	VRAM	Est. Time	Cost	Recommended For
HF Spaces (A100)	A100	48GB	~8h	~$14	Production runs
HF Spaces (L4)	L4	24GB	~14h	~$11	Production runs
Google Colab Pro	L4	24GB	~14h	~$10	Experimentation
Google Colab Pro+	A100	40GB	~6h	~$20	Fast iteration
Kaggle (free)	T4	16GB	~28h (2 sessions)	Free	Budget runs
Local (RTX 3090)	—	24GB	~12h	Electricity	Research

---

Training Scripts

This repository has two training entry points:

hf_space_a100/train.py — for HF Spaces and Colab. Integrates with the Gradio dashboard, streams live logs, auto-uploads to HF Hub. Pre-configured for A100 but auto-detects GPU.

train_grpo.py — for local machines, Kaggle, and research use. Exposes all hyperparameters directly. More verbose logging. Supports checkpoint resume.

Both scripts use the same generator, verifier, and GRPO configuration logic.

---

Key Configuration Parameters

TRAIN_SCENARIOS = 400       # number of unique training scenarios per epoch
EVAL_SCENARIOS  = 60        # held-out evaluation scenarios
NUM_EPOCHS      = 2         # full passes over the training set
LEARNING_RATE   = 3e-6      # conservative; prevents catastrophic forgetting
BETA            = 0.04      # KL penalty; 0.02 causes excessive drift (use 0.04+)
num_generations = 4         # GRPO group size; 6 recommended on A100
MAX_NEW_TOKENS  = 768       # completion budget; average ~300 tokens in practice
MAX_PROMPT_LENGTH = 3200    # prompt budget; generator enforces 4000 char limit
SAVE_STEPS      = 50        # checkpoint frequency
EVAL_STEPS      = 50        # evaluation frequency

---

Checkpoint Strategy

The best checkpoint is typically not the final one. Reward peaks mid-training (around step 250 in Run 2) and may decline slightly as KL divergence increases. Always evaluate multiple checkpoints.

# List all checkpoints
ls conflictbench-grpo-output/checkpoint-*/

# The trainer saves with load_best_model_at_end=True,
# but verify by comparing checkpoint-250 vs final manually.

---

Resuming After Session Interruption (Kaggle / Colab)

The training script supports automatic resume:

# In train_grpo.py, GRPOConfig already has:
save_steps=50           # checkpoints every 50 steps
save_total_limit=10     # keeps last 10 checkpoints

To resume, simply re-run the same script. If OUTPUT_DIR contains a checkpoint-* directory, TRL will automatically resume from the latest checkpoint.

For Colab: redirect OUTPUT_DIR to Google Drive to persist checkpoints across session resets:

OUTPUT_DIR = "/content/drive/MyDrive/conflictbench-grpo-output"

---

Uploading to HF Hub

Set HF_TOKEN and HF_REPO_ID before training:

export HF_TOKEN=hf_your_write_token
export HF_REPO_ID=your-username/your-model-name

The training script automatically uploads the best checkpoint after training completes. To upload manually:

from huggingface_hub import HfApi
api = HfApi(token="hf_your_token")
api.upload_folder(
    folder_path="./conflictbench-grpo-output/checkpoint-250",
    repo_id="your-username/conflictbench-model",
    repo_type="model"
)