Hugging Face's logo

RichardErkhov
/
RLHFlow_-_LLaMA3-iterative-DPO-final-gguf

GGUF
conversational
Model card Files
xet
Community
RichardErkhov commited on
Commit
9e4283f
·
verified ·
1 Parent(s): 5b285e6

uploaded readme

Browse files
Files changed (1) hide show
  1. README.md +164 -0
README.md ADDED
@@ -0,0 +1,164 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Quantization made by Richard Erkhov.
2
+
3
+ [Github](https://github.com/RichardErkhov)
4
+
5
+ [Discord](https://discord.gg/pvy7H8DZMG)
6
+
7
+ [Request more models](https://github.com/RichardErkhov/quant_request)
8
+
9
+
10
+ LLaMA3-iterative-DPO-final - GGUF
11
+ - Model creator: https://huggingface.co/RLHFlow/
12
+ - Original model: https://huggingface.co/RLHFlow/LLaMA3-iterative-DPO-final/
13
+
14
+
15
+ | Name | Quant method | Size |
16
+ | ---- | ---- | ---- |
17
+ | [LLaMA3-iterative-DPO-final.Q2_K.gguf](https://huggingface.co/RichardErkhov/RLHFlow_-_LLaMA3-iterative-DPO-final-gguf/blob/main/LLaMA3-iterative-DPO-final.Q2_K.gguf) | Q2_K | 2.96GB |
18
+ | [LLaMA3-iterative-DPO-final.IQ3_XS.gguf](https://huggingface.co/RichardErkhov/RLHFlow_-_LLaMA3-iterative-DPO-final-gguf/blob/main/LLaMA3-iterative-DPO-final.IQ3_XS.gguf) | IQ3_XS | 3.28GB |
19
+ | [LLaMA3-iterative-DPO-final.IQ3_S.gguf](https://huggingface.co/RichardErkhov/RLHFlow_-_LLaMA3-iterative-DPO-final-gguf/blob/main/LLaMA3-iterative-DPO-final.IQ3_S.gguf) | IQ3_S | 3.43GB |
20
+ | [LLaMA3-iterative-DPO-final.Q3_K_S.gguf](https://huggingface.co/RichardErkhov/RLHFlow_-_LLaMA3-iterative-DPO-final-gguf/blob/main/LLaMA3-iterative-DPO-final.Q3_K_S.gguf) | Q3_K_S | 3.41GB |
21
+ | [LLaMA3-iterative-DPO-final.IQ3_M.gguf](https://huggingface.co/RichardErkhov/RLHFlow_-_LLaMA3-iterative-DPO-final-gguf/blob/main/LLaMA3-iterative-DPO-final.IQ3_M.gguf) | IQ3_M | 3.52GB |
22
+ | [LLaMA3-iterative-DPO-final.Q3_K.gguf](https://huggingface.co/RichardErkhov/RLHFlow_-_LLaMA3-iterative-DPO-final-gguf/blob/main/LLaMA3-iterative-DPO-final.Q3_K.gguf) | Q3_K | 3.74GB |
23
+ | [LLaMA3-iterative-DPO-final.Q3_K_M.gguf](https://huggingface.co/RichardErkhov/RLHFlow_-_LLaMA3-iterative-DPO-final-gguf/blob/main/LLaMA3-iterative-DPO-final.Q3_K_M.gguf) | Q3_K_M | 3.74GB |
24
+ | [LLaMA3-iterative-DPO-final.Q3_K_L.gguf](https://huggingface.co/RichardErkhov/RLHFlow_-_LLaMA3-iterative-DPO-final-gguf/blob/main/LLaMA3-iterative-DPO-final.Q3_K_L.gguf) | Q3_K_L | 4.03GB |
25
+ | [LLaMA3-iterative-DPO-final.IQ4_XS.gguf](https://huggingface.co/RichardErkhov/RLHFlow_-_LLaMA3-iterative-DPO-final-gguf/blob/main/LLaMA3-iterative-DPO-final.IQ4_XS.gguf) | IQ4_XS | 4.18GB |
26
+ | [LLaMA3-iterative-DPO-final.Q4_0.gguf](https://huggingface.co/RichardErkhov/RLHFlow_-_LLaMA3-iterative-DPO-final-gguf/blob/main/LLaMA3-iterative-DPO-final.Q4_0.gguf) | Q4_0 | 4.34GB |
27
+ | [LLaMA3-iterative-DPO-final.IQ4_NL.gguf](https://huggingface.co/RichardErkhov/RLHFlow_-_LLaMA3-iterative-DPO-final-gguf/blob/main/LLaMA3-iterative-DPO-final.IQ4_NL.gguf) | IQ4_NL | 4.38GB |
28
+ | [LLaMA3-iterative-DPO-final.Q4_K_S.gguf](https://huggingface.co/RichardErkhov/RLHFlow_-_LLaMA3-iterative-DPO-final-gguf/blob/main/LLaMA3-iterative-DPO-final.Q4_K_S.gguf) | Q4_K_S | 4.37GB |
29
+ | [LLaMA3-iterative-DPO-final.Q4_K.gguf](https://huggingface.co/RichardErkhov/RLHFlow_-_LLaMA3-iterative-DPO-final-gguf/blob/main/LLaMA3-iterative-DPO-final.Q4_K.gguf) | Q4_K | 4.58GB |
30
+ | [LLaMA3-iterative-DPO-final.Q4_K_M.gguf](https://huggingface.co/RichardErkhov/RLHFlow_-_LLaMA3-iterative-DPO-final-gguf/blob/main/LLaMA3-iterative-DPO-final.Q4_K_M.gguf) | Q4_K_M | 4.58GB |
31
+ | [LLaMA3-iterative-DPO-final.Q4_1.gguf](https://huggingface.co/RichardErkhov/RLHFlow_-_LLaMA3-iterative-DPO-final-gguf/blob/main/LLaMA3-iterative-DPO-final.Q4_1.gguf) | Q4_1 | 4.78GB |
32
+ | [LLaMA3-iterative-DPO-final.Q5_0.gguf](https://huggingface.co/RichardErkhov/RLHFlow_-_LLaMA3-iterative-DPO-final-gguf/blob/main/LLaMA3-iterative-DPO-final.Q5_0.gguf) | Q5_0 | 5.21GB |
33
+ | [LLaMA3-iterative-DPO-final.Q5_K_S.gguf](https://huggingface.co/RichardErkhov/RLHFlow_-_LLaMA3-iterative-DPO-final-gguf/blob/main/LLaMA3-iterative-DPO-final.Q5_K_S.gguf) | Q5_K_S | 5.21GB |
34
+ | [LLaMA3-iterative-DPO-final.Q5_K.gguf](https://huggingface.co/RichardErkhov/RLHFlow_-_LLaMA3-iterative-DPO-final-gguf/blob/main/LLaMA3-iterative-DPO-final.Q5_K.gguf) | Q5_K | 5.34GB |
35
+ | [LLaMA3-iterative-DPO-final.Q5_K_M.gguf](https://huggingface.co/RichardErkhov/RLHFlow_-_LLaMA3-iterative-DPO-final-gguf/blob/main/LLaMA3-iterative-DPO-final.Q5_K_M.gguf) | Q5_K_M | 5.34GB |
36
+ | [LLaMA3-iterative-DPO-final.Q5_1.gguf](https://huggingface.co/RichardErkhov/RLHFlow_-_LLaMA3-iterative-DPO-final-gguf/blob/main/LLaMA3-iterative-DPO-final.Q5_1.gguf) | Q5_1 | 5.65GB |
37
+ | [LLaMA3-iterative-DPO-final.Q6_K.gguf](https://huggingface.co/RichardErkhov/RLHFlow_-_LLaMA3-iterative-DPO-final-gguf/blob/main/LLaMA3-iterative-DPO-final.Q6_K.gguf) | Q6_K | 6.14GB |
38
+ | [LLaMA3-iterative-DPO-final.Q8_0.gguf](https://huggingface.co/RichardErkhov/RLHFlow_-_LLaMA3-iterative-DPO-final-gguf/blob/main/LLaMA3-iterative-DPO-final.Q8_0.gguf) | Q8_0 | 7.95GB |
39
+
40
+
41
+
42
+
43
+ Original model description:
44
+ ---
45
+ license: llama3
46
+ ---
47
+ # LLaMA3-iterative-DPO-final
48
+
49
+ ## Introduction
50
+ We release an unofficial checkpoint of a state-of-the-art instruct model of its class, **LLaMA3-iterative-DPO-final**.
51
+ On all three widely-used instruct model benchmarks: **Alpaca-Eval-V2**, **MT-Bench**, **Chat-Arena-Hard**, our model outperforms all models of similar size (e.g., LLaMA-3-8B-it), most large open-sourced models (e.g., Mixtral-8x7B-it),
52
+ and strong proprietary models (e.g., GPT-3.5-turbo-0613). The model is trained with open-sourced datasets without any additional human-/GPT4-labeling.
53
+
54
+ Even better, we provide a [detailed recipe](https://github.com/RLHFlow/Online-RLHF) to reproduce the model. Enjoy!
55
+
56
+ ## Model Releases
57
+ See the [collection](https://huggingface.co/collections/RLHFlow/online-rlhf-663ae95fade1a39663dab218) of the training set, reward/preference model, SFT model.
58
+
59
+ - [SFT model](https://huggingface.co/RLHFlow/LLaMA3-SFT)
60
+ - [Reward model](https://huggingface.co/sfairXC/FsfairX-LLaMA3-RM-v0.1)
61
+ - This model is more like the concise version in the report. We are still working on the model realeasing due to some license issue....
62
+
63
+ ## Dataset
64
+ - [Preference data mix](https://huggingface.co/datasets/hendrydong/preference_700K)
65
+ - [Prompt collection for RLHF training](https://huggingface.co/datasets/RLHFlow/prompt-collection-v0.1)
66
+
67
+ ## Training methods
68
+ We have developed a simple and efficient online RLHF recipe for LLM instruct training. Our recipe is DPO-based and thus much cheaper and simpler to train and tune compared to PPO-based approaches.
69
+ Unlike widely-used offline DPO, the online component of our approach effectively mitigates distribution shifts during policy optimization.
70
+ For a detailed exposition, please refer to our accompanying technical report.
71
+
72
+
73
+ ## Chat Benchmarks
74
+
75
+ | **Model** | **Size** | **Method** | **LC Alpaca-Eval-V2** | **MT-Bench** | **Chat-Arena-Hard** |
76
+ |-------------------------|----------|-------------------|-----------------------|--------------|---------------------|
77
+ | **Small Open-Sourced Models** | | | | | |
78
+ | Gemma-7B-it | 7B | SFT | 10.4 | 6.38 | 7.5 |
79
+ | Zephyr-7B-beta | 7B | Vanilla DPO | 13.1 | 7.34 | - |
80
+ | Mistral-7B-v0.2-it | 7B | SFT | 17.1 | 7.51 | 12.6 |
81
+ | Open-Chat-0106 | 7B | SFT | 15.6 | 7.8 | - |
82
+ | Starling-7B-beta | 7B | PPO | 25.8 | 8.12 | 23.0 |
83
+ | LLaMA-3-8B-it | 8B | RS+DPO+PPO | 22.9 | 8.16 | 20.6 |
84
+ | **Ours** | | | | | |
85
+ | Ours (SFT baseline) | 8B | SFT | 10.2 | 7.69 | 5.6 |
86
+ | Ours (DPO baseline) | 8B | Vanilla DPO | 22.5 | 8.17 | 22.4 |
87
+ | Ours (Online RLHF) | 8B | Iterative DPO | **37.2** | **8.46** | **29.1** |
88
+ | **Large Open-Sourced Models** | | | | | |
89
+ | Vicuna-33b-v1.3 | 33B | SFT | 17.6 | 7.12 | 8.6 |
90
+ | Yi-34B-Chat | 34B | SFT | 27.2 | - | 23.1 |
91
+ | Mixtral-8x7B-it | 45B* | SFT | 23.7 | 8.30 | 23.4 |
92
+ | Tulu-2-DPO-70B | 70B | Vanilla DPO | 21.2 | 7.89 | 15.0 |
93
+ | LLaMA-3-70B-it | 70B | RS+DPO+PPO | 34.4 | 8.95 | 41.1 |
94
+ | Mixtral-8x22B-it | 141B* | SFT | 30.9 | 8.66 | 36.4 |
95
+ | **Proprietary Models** | | | | | |
96
+ | GPT-3.5-turbo-1106 | - | - | 19.3 | 8.35 | 18.9 |
97
+ | GPT-3.5-turbo-0613 | - | - | 22.7 | 8.39 | 24.8 |
98
+ | GPT-4-0613 | - | - | 30.2 | 9.18 | 37.9 |
99
+ | Claude-3-Opus | - | - | 40.5 | 9.00 | 60.4 |
100
+ | GPT-4 Turbo (04/09) | - | - | 55.0 | - | 82.6 |
101
+
102
+
103
+ ## Academic Benchmarks
104
+
105
+ | **Model** | **Size** | **Method** | **GSM-8K** | **MMLU** | **HumanEval** | **TruthfulQA** | **ARC** | **MBPP** |
106
+ |----------------------------|----------|-----------------|------------|----------|---------------|----------------|---------|----------|
107
+ | LLaMA-3-8B-it | 8B | RS+DPO+PPO | 79.6 | 66.0 | 61.6 | 43.9 | 59.5 | 61.1 |
108
+ | Ours (SFT baseline) | 8B | SFT | 74.2 | 64.7 | 65.2 | 53.4 | 61.4 | 62.3 |
109
+ | Ours (DPO baseline) | 8B | Vanilla DPO | 79.8 | 64.5 | 63.4 | 61.8 | 65.2 | 60.3 |
110
+ | Ours (Iterative RLHF) | 8B | Iterative DPO | 80.7 | 65.3 | 64.6 | 60.4 | 64.3 | 60.8 |
111
+
112
+
113
+ ## Usage
114
+ ```python
115
+ from transformers import AutoModelForCausalLM, AutoTokenizer
116
+
117
+ device = "cuda"
118
+
119
+ model = AutoModelForCausalLM.from_pretrained("RLHFlow/LLaMA3-iterative-DPO-final")
120
+ tokenizer = AutoTokenizer.from_pretrained("RLHFlow/LLaMA3-iterative-DPO-final")
121
+
122
+ messages = [
123
+ {"role": "user", "content": "I'm trying to teach myself to have nicer handwriting. Can you help?"},
124
+ ]
125
+
126
+ model_inputs = tokenizer.apply_chat_template(messages, return_tensors="pt")
127
+
128
+ model_inputs = model_inputs.to(device)
129
+ model.to(device)
130
+
131
+ output_tokens = model.generate(model_inputs, max_new_tokens=1024, do_sample=True)
132
+ model_outputs = tokenizer.batch_decode(output_tokens)
133
+ print(model_outputs[0])
134
+ ```
135
+
136
+
137
+ ## Limitations
138
+ RLHFlow/LLaMA3-iterative-DPO-final is an unofficial checkpoint developed to illustrate the power of online iterative RLHF and is for research purpose. While safety and ethical considerations are integral to our alignment process,
139
+ there remains the possibility that the model could generate offensive or unethical content, particularly under adversarial conditions.
140
+ We are committed to continuous improvement in our models to minimize such risks and encourage responsible usage.
141
+
142
+ ## Citation
143
+ Please cite our techical report if you find our model is useful for your research or product.
144
+ ```
145
+ @misc{dong2024rlhf,
146
+ title={RLHF Workflow: From Reward Modeling to Online RLHF},
147
+ author={Hanze Dong and Wei Xiong and Bo Pang and Haoxiang Wang and Han Zhao and Yingbo Zhou and Nan Jiang and Doyen Sahoo and Caiming Xiong and Tong Zhang},
148
+ year={2024},
149
+ eprint={2405.07863},
150
+ archivePrefix={arXiv},
151
+ primaryClass={cs.LG}
152
+ }
153
+
154
+ @misc{xiong2024iterative,
155
+ title={Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint},
156
+ author={Wei Xiong and Hanze Dong and Chenlu Ye and Ziqi Wang and Han Zhong and Heng Ji and Nan Jiang and Tong Zhang},
157
+ year={2024},
158
+ eprint={2312.11456},
159
+ archivePrefix={arXiv},
160
+ primaryClass={cs.LG}
161
+ }
162
+
163
+ ```
164
+