Hugging Face's logo

OpenLLM-Ro
/
RoGemma2-9b-Instruct-DPO-4Bit-BB

Safetensors
gemma2
4-bit precision
bitsandbytes
Model card Files
xet
 Community
intelpen commited on
Commit
eed6e39
·
verified ·
1 Parent(s): a22c0ed

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +1 -355
README.md CHANGED
@@ -1,355 +1 @@
1
- ---
2
- license: cc-by-nc-4.0
3
- language:
4
- - ro
5
- base_model:
6
- - OpenLLM-Ro/RoLlama3.1-8b-Instruct
7
- datasets:
8
- - OpenLLM-Ro/ro_dpo_helpsteer
9
- model-index:
10
- - name: OpenLLM-Ro/RoLlama3.1-8b-Instruct-DPO-4bit
11
- results:
12
- - task:
13
- type: text-generation
14
- dataset:
15
- name: OpenLLM-Ro/ro_arc_challenge
16
- type: OpenLLM-Ro/ro_arc_challenge
17
- metrics:
18
- - name: Average accuracy
19
- type: accuracy
20
- value: 42.74
21
- - name: 0-shot
22
- type: accuracy
23
- value: 40.79
24
- - name: 1-shot
25
- type: accuracy
26
- value: 40.36
27
- - name: 3-shot
28
- type: accuracy
29
- value: 43.36
30
- - name: 5-shot
31
- type: accuracy
32
- value: 44.04
33
- - name: 10-shot
34
- type: accuracy
35
- value: 43.87
36
- - name: 25-shot
37
- type: accuracy
38
- value: 44.04
39
-
40
- - task:
41
- type: text-generation
42
- dataset:
43
- name: OpenLLM-Ro/ro_mmlu
44
- type: OpenLLM-Ro/ro_mmlu
45
- metrics:
46
- - name: Average accuracy
47
- type: accuracy
48
- value: 42.27
49
- - name: 0-shot
50
- type: accuracy
51
- value: 43.23
52
- - name: 1-shot
53
- type: accuracy
54
- value: 42.47
55
- - name: 3-shot
56
- type: accuracy
57
- value: 42.19
58
- - name: 5-shot
59
- type: accuracy
60
- value: 41.19
61
-
62
- - task:
63
- type: text-generation
64
- dataset:
65
- name: OpenLLM-Ro/ro_winogrande
66
- type: OpenLLM-Ro/ro_winogrande
67
- metrics:
68
- - name: Average accuracy
69
- type: accuracy
70
- value: 64.94
71
- - name: 0-shot
72
- type: accuracy
73
- value: 63.14
74
- - name: 1-shot
75
- type: accuracy
76
- value: 64.64
77
- - name: 3-shot
78
- type: accuracy
79
- value: 65.43
80
- - name: 5-shot
81
- type: accuracy
82
- value: 66.54
83
-
84
- - task:
85
- type: text-generation
86
- dataset:
87
- name: OpenLLM-Ro/ro_hellaswag
88
- type: OpenLLM-Ro/ro_hellaswag
89
- metrics:
90
- - name: Average accuracy
91
- type: accuracy
92
- value: 52.39
93
- - name: 0-shot
94
- type: accuracy
95
- value: 52.42
96
- - name: 1-shot
97
- type: accuracy
98
- value: 52.30
99
- - name: 3-shot
100
- type: accuracy
101
- value: 52.60
102
- - name: 5-shot
103
- type: accuracy
104
- value: 52.20
105
- - name: 10-shot
106
- type: accuracy
107
- value: 52.42
108
-
109
- - task:
110
- type: text-generation
111
- dataset:
112
- name: OpenLLM-Ro/ro_gsm8k
113
- type: OpenLLM-Ro/ro_gsm8k
114
- metrics:
115
- - name: Average accuracy
116
- type: accuracy
117
- value: 38.87
118
- - name: 1-shot
119
- type: accuracy
120
- value: 28.13
121
- - name: 3-shot
122
- type: accuracy
123
- value: 42.23
124
- - name: 5-shot
125
- type: accuracy
126
- value: 46.25
127
-
128
- - task:
129
- type: text-generation
130
- dataset:
131
- name: OpenLLM-Ro/ro_truthfulqa
132
- type: OpenLLM-Ro/ro_truthfulqa
133
- metrics:
134
- - name: Average accuracy
135
- type: accuracy
136
- value: 48.67
137
- - name: 0-shot
138
- type: accuracy
139
- value: 48.67
140
-
141
- - task:
142
- type: text-generation
143
- dataset:
144
- name: LaRoSeDa_binary
145
- type: LaRoSeDa_binary
146
- metrics:
147
- - name: Average macro-f1
148
- type: macro-f1
149
- value: 97.47
150
- - name: 0-shot
151
- type: macro-f1
152
- value: 97.43
153
- - name: 1-shot
154
- type: macro-f1
155
- value: 97.33
156
- - name: 3-shot
157
- type: macro-f1
158
- value: 97.70
159
- - name: 5-shot
160
- type: macro-f1
161
- value: 97.43
162
-
163
- - task:
164
- type: text-generation
165
- dataset:
166
- name: LaRoSeDa_multiclass
167
- type: LaRoSeDa_multiclass
168
- metrics:
169
- - name: Average macro-f1
170
- type: macro-f1
171
- value: 64.05
172
- - name: 0-shot
173
- type: macro-f1
174
- value: 65.90
175
- - name: 1-shot
176
- type: macro-f1
177
- value: 64.68
178
- - name: 3-shot
179
- type: macro-f1
180
- value: 62.36
181
- - name: 5-shot
182
- type: macro-f1
183
- value: 63.27
184
-
185
- - task:
186
- type: text-generation
187
- dataset:
188
- name: WMT_EN-RO
189
- type: WMT_EN-RO
190
- metrics:
191
- - name: Average bleu
192
- type: bleu
193
- value: 20.54
194
- - name: 0-shot
195
- type: bleu
196
- value: 7.20
197
- - name: 1-shot
198
- type: bleu
199
- value: 25.68
200
- - name: 3-shot
201
- type: bleu
202
- value: 24.50
203
- - name: 5-shot
204
- type: bleu
205
- value: 24.78
206
-
207
- - task:
208
- type: text-generation
209
- dataset:
210
- name: WMT_RO-EN
211
- type: WMT_RO-EN
212
- metrics:
213
- - name: Average bleu
214
- type: bleu
215
- value: 21.16
216
- - name: 0-shot
217
- type: bleu
218
- value: 2.59
219
- - name: 1-shot
220
- type: bleu
221
- value: 17.54
222
- - name: 3-shot
223
- type: bleu
224
- value: 30.82
225
- - name: 5-shot
226
- type: bleu
227
- value: 33.67
228
-
229
- - task:
230
- type: text-generation
231
- dataset:
232
- name: XQuAD
233
- type: XQuAD
234
- metrics:
235
- - name: Average exact_match
236
- type: exact_match
237
- value: 21.45
238
- - name: Average f1
239
- type: f1
240
- value: 37.73
241
- - name: 0-shot exact_match
242
- type: exact_match
243
- value: 3.45
244
- - name: 0-shot f1
245
- type: f1
246
- value: 12.36
247
- - name: 1-shot exact_match
248
- type: exact_match
249
- value: 32.02
250
- - name: 1-shot f1
251
- type: f1
252
- value: 55.70
253
- - name: 3-shot exact_match
254
- type: exact_match
255
- value: 33.78
256
- - name: 3-shot f1
257
- type: f1
258
- value: 54.15
259
- - name: 5-shot exact_match
260
- type: exact_match
261
- value: 16.55
262
- - name: 5-shot f1
263
- type: f1
264
- value: 28.71
265
-
266
- - task:
267
- type: text-generation
268
- dataset:
269
- name: STS
270
- type: STS
271
- metrics:
272
- - name: Average pearson
273
- type: pearson
274
- value: 76.93
275
- - name: Average spearman
276
- type: spearman
277
- value: 77.08
278
- - name: 1-shot pearson
279
- type: pearson
280
- value: 77.02
281
- - name: 1-shot spearman
282
- type: spearman
283
- value: 77.80
284
- - name: 3-shot pearson
285
- type: pearson
286
- value: 76.93
287
- - name: 3-shot spearman
288
- type: spearman
289
- value: 77.00
290
- - name: 5-shot pearson
291
- type: pearson
292
- value: 76.85
293
- - name: 5-shot spearman
294
- type: spearman
295
- value: 76.45
296
- ---
297
-
298
-
299
- # Model Card for 4-bit RoLlama3.1-8b-Instruct-DPO
300
-
301
- *Built from [RoLlama3.1-8b-Instruct-DPO](https://huggingface.co/OpenLLM-Ro/RoLlama3.1-8b-Instruct-DPO), quantized to 4-bit.*
302
-
303
- This variant of **RoLlama3.1-8b-Instruct-DPO** provides a reduced footprint through 4-bit quantization, aimed at enabling usage on resource-constrained GPUs while preserving a high fraction of the model’s capabilities.
304
-
305
- ## Model Details
306
-
307
- ## Comparison to 16 bit
308
-
309
- It loooks that the effects of the quantization are minimal :
310
-
311
- | **Task** | **Metric** | **FP16 Original** | **4-bit** | **Absolute Diff.** | **% Change** |
312
- |--------------------------|-----------------------|-------------------|-----------------|---------------------|--------------------|
313
- | **ARC Challenge** | Avg. Accuracy | 44.84 | 42.74 | -2.10 | -4.68% |
314
- | **MMLU** | Avg. Accuracy | 55.06 | 42.27 | -12.79 | -23.23% |
315
- | **Winogrande** | Avg. Accuracy | 65.87 | 64.94 | -0.93 | -1.41% |
316
- | **Hellaswag** | Avg. Accuracy | 58.67 | 52.39 | -6.28 | -10.70% |
317
- | **GSM8K** | Avg. Accuracy | 44.17 | 38.87 | -5.30 | -11.99% |
318
- | **TruthfulQA** | Avg. Accuracy | 47.82 | 48.67 | +0.85 | +1.78% |
319
- | **LaRoSeDa (binary)** | Macro-F1 | 96.10 | 97.47 | +1.37 | +1.43% |
320
- | **LaRoSeDa (multiclass)**| Macro-F1 | 55.37 | 64.05 | +8.68 | +15.68% |
321
- | **WMT EN-RO** | BLEU | 21.29 | 20.54 | -0.75 | -3.52% |
322
- | **WMT RO-EN** | BLEU | 21.86 | 21.16 | -0.70 | -3.20% |
323
- | **XQuAD (avg)** | EM / F1 | 21.58 / 36.54 | 21.45 / 37.73 | ~-0.13 / +1.19 | -0.60% / +3.26% |
324
- | **STS (avg)** | Spearman / Pearson | 78.01 / 77.98 | 77.08 / 76.93 | -0.93 / -1.05 | -1.19% / -1.35% |
325
-
326
-
327
- ### Model Description
328
-
329
- - **Developed by:** OpenLLM-Ro
330
- - **Language(s):** Romanian
331
- - **License:** cc-by-nc-4.0
332
- - **Quantized from model:** [RoLlama3.1-8b-Instruct-DPO](https://huggingface.co/OpenLLM-Ro/RoLlama3.1-8b-Instruct-DPO)
333
- - **Quantization:** 4-bit
334
-
335
- Quantization reduces model size and improves inference speed but can lead to small drops in performance. Below is a comprehensive table of the main benchmarks comparing the original full-precision version with the new 4-bit variant.
336
-
337
- ## How to Use
338
-
339
- ```python
340
- from transformers import AutoTokenizer, AutoModelForCausalLM
341
-
342
- model_id = "OpenLLM-Ro/RoLlama3.1-8b-Instruct-DPO-4bit"
343
- tokenizer = AutoTokenizer.from_pretrained(model_id)
344
- model = AutoModelForCausalLM.from_pretrained(model_id, load_in_4bit=True, device_map="auto")
345
-
346
- instruction = "Ce jocuri de societate pot juca cu prietenii mei?"
347
- chat = [
348
- {"role": "system", "content": "Ești un asistent folositor, respectuos și onest. Încearcă să ajuți cât mai mult prin informațiile oferite, excluzând răspunsuri toxice, rasiste, sexiste, periculoase și ilegale."},
349
- {"role": "user", "content": instruction},
350
- ]
351
- prompt = tokenizer.apply_chat_template(chat, tokenize=False, system_message="")
352
-
353
- inputs = tokenizer.encode(prompt, add_special_tokens=False, return_tensors="pt").to("cuda")
354
- outputs = model.generate(input_ids=inputs, max_new_tokens=128)
355
- print(tokenizer.decode(outputs[0], skip_special_tokens=True))
 
1
+ 4 bit version of rogemma2 9b