phi-instruct-segment Model Card

Paper: Segmenting Text and Learning Their Rewards for Improved RLHF in Language Model
Model: yyqoni/Phi-3-mini-4k-instruct-segment-rm-700k

Method

The segment reward model assigns rewards to semantically meaningful text segments, segmented dynamically with an entropy-based threshold. It is trained on binary preference labels from human feedback, optimizing a Bradley-Terry loss function that aggregates segment rewards using the average function.

Architecture

Training

The phi-instruct-segment model is fine-tuned from microsoft/Phi-3-mini-4k-instruct on the hendrydong/preference_700K dataset.

Citation

If you find this model or our research useful, please consider citing our paper:

@misc{yin2025segmentingtextlearningrewards,
      title={Segmenting Text and Learning Their Rewards for Improved RLHF in Language Model}, 
      author={Yueqin Yin and Shentao Yang and Yujia Xie and Ziyi Yang and Yuting Sun and Hany Awadalla and Weizhu Chen and Mingyuan Zhou},
      year={2025},
      eprint={2501.02790},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2501.02790},
}