README.md

---
license: apache-2.0
---

# Reasons to Reject? Aligning Language Models with Judgments.
This repository contains the CUT model from our work,

[Reasons to Reject? Aligning Language Models with Judgments](https://arxiv.org/abs/2312.14591).

Weiwen Xu, Deng Cai, Zhisong Zhang, Wai Lam, Shuming Shi

The source codes can be found in https://github.com/wwxu21/CUT 
**** 

## 1. Model description

This model achieves 91.36 on [AlpacaEval](https://tatsu-lab.github.io/alpaca_eval). 
It is tuned after 4 iterations of online alignment. In each iteration, we apply the following three steps:

- Step 1: Collect instructions, and obtain the responses from the target model.

- Step 2: Annotate judgments for the responses.

- Step 3: Apply CUT to fine-tune the target model with the above instruction-response-judgment triplets.

Specifically, we use [LLaMA2-chat-13b](https://huggingface.co/meta-llama/Llama-2-13b-chat-hf) as the base LLM. In each iteration, we sample 1000 instructions from [Stanford Alpaca](https://github.com/tatsu-lab/stanford_alpaca).
To avoid over-fitting, we ensure that the sampled data are different in each iteration.
We then ask GPT4 for the judgment annotation.


## 2. Template
The CUT model is a chat model and it uses the following [Alpaca template](https://github.com/tatsu-lab/stanford_alpaca):
```
Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
{instruction}

### Response:
```

### 3. How to use

#### 3.1. Huggingface

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

torch.set_default_device("cuda")

model = AutoModelForCausalLM.from_pretrained("xww033/cut-13b", torch_dtype=torch.float16)
tokenizer = AutoTokenizer.from_pretrained("xww033/cut-13b")

inputs = tokenizer('''Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
How did US states get their names?

### Response:''', return_tensors="pt", return_attention_mask=False)

outputs = model.generate(**inputs, max_length=2048)
text = tokenizer.batch_decode(outputs)[0]
print(text)
```

#### 3.2. FastChat

[Fastchat](https://github.com/lm-sys/FastChat) provides a simple setup for those interested in trying our aligned model. After downloading the [CUT model](https://huggingface.co/xww033/cut-13b) through HuggingFace, clone the Fastchat repository:

```bash
git clone https://github.com/lm-sys/FastChat.git
cd FastChat
```

Download the required packages:

```bash
pip install --upgrade pip  # enable PEP 660 support
pip install -e .
```

Finally, run the following:

```bash
python -m fastchat.serve.cli --model-path xww033/cut-13b --conv-template alpaca
```


### 4. BibTeX entry and citation info
```bibtxt
@article{xu2023reasons,
  title={Reasons to Reject? Aligning Language Models with Judgments},
  author={Xu, Weiwen and Cai, Deng and Zhang, Zhisong and Lam, Wai and Shi, Shuming},
  journal={arXiv preprint arXiv:2312.14591},
  year={2023}
}
```