docs/lightning_hf_runbook.md

Lightning AI + Hugging Face Runbook

This runbook is optimized for short 3-4 hour H100 windows and Hugging Face credits. The judge-rerunnable notebook version is notebooks/corp_env_trl_unsloth_training.ipynb. A minified (single-line) Colab export in the repo can be searched for train_grpo or MAX_STEPS to recover pins or a prior command line, then align with the defaults in training/train_grpo.py and training/train_sft.py.

1. Prepare Artifacts Locally

uv sync --extra dev --extra plots
uv run python scripts/generate_sft_data.py --tasks h1_acquisition_defence --per-task 24 --variant-stride 2 --output data/raw/h1_seed.jsonl
uv run python scripts/verify_examples.py --input data/raw/e1_m1_examples.jsonl --clean data/processed/e1_m1_clean.jsonl --rejected data/processed/e1_m1_rejected.jsonl --strict-json --require-stepwise-deliberation
uv run python scripts/verify_examples.py --input data/raw/h1_seed.jsonl --clean data/processed/h1_seed_clean.jsonl --rejected data/processed/h1_seed_rejected.jsonl --strict-json --require-stepwise-deliberation
uv run python scripts/prepare_sft_data.py --min-pass-rate 0.85 --min-reasoning-steps 1 --min-conflict-steps 0 --min-resolution-steps 0 --require-stepwise-deliberation

Keep raw examples untouched. Train only from verified data/processed/*_clean.jsonl and data/sft/*.jsonl.

2. Upload Dataset To Hugging Face

huggingface-cli login
huggingface-cli repo create corp-env-data --type dataset
huggingface-cli upload <your-user-or-org>/corp-env-data data/sft/e1_m1_h1_examples.jsonl data/sft/e1_m1_h1_examples.jsonl --repo-type dataset
huggingface-cli upload <your-user-or-org>/corp-env-data data/processed/e1_m1_clean.jsonl data/processed/e1_m1_clean.jsonl --repo-type dataset
huggingface-cli upload <your-user-or-org>/corp-env-data data/processed/h1_seed_clean.jsonl data/processed/h1_seed_clean.jsonl --repo-type dataset

Retire or mirror e1_m1_examples.jsonl in the same dataset if you need backward-compatible paths.

3. Lightning H100 Session 1: SFT

git clone <repo-url> corp_gym
cd corp_gym
python -m venv .venv
source .venv/bin/activate
pip install -U pip
pip install -e ".[training]"
huggingface-cli login
python training/train_sft.py \
  --model Qwen/Qwen2.5-7B-Instruct \
  --data data/sft/e1_m1_h1_examples.jsonl \
  --output outputs/sft_adapter \
  --epochs 2 \
  --max-steps 30 \
  --push-to-hub <your-user-or-org>/corp-env-sft-adapter

This uses Unsloth + TRL SFTTrainer with conversational messages JSONL (and optional --packing / --dataloader-num-workers on a strong box). If setup time is short, use a 7B model. For quality runs, target 14B with strict-clean traces. Remove --max-steps 30 or raise it for a real run. Install a flash-attn wheel that matches the session’s torch+CUDA for best GRPO and long-context SFT step time.

14B SFT presets (smoke then full)

# Qwen3 14B smoke
python training/train_sft.py \
  --model Qwen/Qwen3-14B-Instruct \
  --data data/sft/e1_m1_h1_examples.jsonl \
  --output outputs/sft_qwen3_14b \
  --batch-size 1 --grad-accum 8 --max-steps 20

# Qwen3 14B fuller run
python training/train_sft.py \
  --model Qwen/Qwen3-14B-Instruct \
  --data data/sft/e1_m1_h1_examples.jsonl \
  --output outputs/sft_qwen3_14b \
  --epochs 2 --max-steps 200

# DeepSeek 14B smoke
python training/train_sft.py \
  --model deepseek-ai/DeepSeek-R1-Distill-Qwen-14B \
  --data data/sft/e1_m1_h1_examples.jsonl \
  --output outputs/sft_deepseek_14b \
  --batch-size 1 --grad-accum 8 --max-steps 20

# DeepSeek 14B fuller run
python training/train_sft.py \
  --model deepseek-ai/DeepSeek-R1-Distill-Qwen-14B \
  --data data/sft/e1_m1_h1_examples.jsonl \
  --output outputs/sft_deepseek_14b \
  --epochs 2 --max-steps 200

Use --assistant-only only when the model template supports assistant token masks end-to-end.

4. Lightning H100 Session 2: Eval SFT

pip install -e ".[training,plots]"
python eval.py --policy scripted_weak --label baseline --output results/baseline_eval.jsonl
python eval.py --policy hf \
  --label sft \
  --model Qwen/Qwen2.5-7B-Instruct \
  --adapter outputs/sft_adapter \
  --output results/sft_eval.jsonl

Push results/*.jsonl to Hugging Face or copy them back before the Lightning account expires.

5. Lightning H100 Session 3: RLVR

python training/train_rlvr.py \
  --model Qwen/Qwen2.5-7B-Instruct \
  --adapter outputs/sft_adapter \
  --examples data/processed/e1_m1_clean.jsonl,data/processed/h1_seed_clean.jsonl \
  --output outputs/rlvr_adapter \
  --strict-json \
  --min-reasoning-steps 2 \
  --rounds 3 \
  --n-samples 8 \
  --max-prompts 128 \
  --stats-file results/runs/rlvr_stats.jsonl \
  --push-to-hub <your-user-or-org>/corp-env-rlvr-adapter

This uses Unsloth + TRL RLVR rejection-sampling with the same strict CORP-ENV verifier path. The recommended production path is now SFT then RLVR.

6. Final Eval And Plots

python eval.py --policy hf \
  --label rlvr \
  --model Qwen/Qwen2.5-7B-Instruct \
  --adapter outputs/rlvr_adapter \
  --output results/rlvr_eval.jsonl
python plot_results.py \
  --inputs results/baseline_eval.jsonl results/sft_eval.jsonl results/rlvr_eval.jsonl \
  --output-dir results

Commit or upload the resulting PNGs:

• results/model_comparison.png
• results/success_by_task.png
• results/invalid_action_rate.png
• results/reward_curve.png

7. Hugging Face Space

Use HF credits for the official hosted OpenEnv environment:

uv run openenv validate
docker build -t corp-env .

After pushing the Space, validate it:

./validate-submission.sh https://<your-space>.hf.space .

On Windows, run the validator from WSL or Git Bash.