metadata
license: apache-2.0
tags:
- openrlhf
- simpo
- sft
- gsm8k
- mathematical-reasoning
- preference-optimization
ASSN2 SimPO+SFT Model
This repository contains the SimPO+SFT model checkpoint for Assignment 2.
Pipeline
- Base model:
meta-llama/Llama-3.2-1B-Instruct - SFT merged checkpoint:
/workspace_data/checkpoints/assn2-sft-merged-hf - SimPO initialization: SFT merged checkpoint
- SimPO checkpoint:
/workspace_data/checkpoints/assn2-simpo-sft
Evaluation
- SimPO+SFT accuracy:
0.62
Notes
This model was produced by applying the reference-free SimPO objective starting from the SFT-merged checkpoint.