Hugging Face's logo

ddidacus
/
olmoe-base

Text Generation
Transformers
Safetensors
English
olmoe
conversational
Model card Files
xet
 Community
ddidacus commited on
Commit
9f6ef6b
·
verified ·
1 Parent(s): 8f0d671

Upload folder using huggingface_hub

Browse files
Files changed (8) hide show
  1. README.md +172 -0
  2. generation_config.json +6 -0
  3. model-00001-of-00003.safetensors +3 -0
  4. model-00002-of-00003.safetensors +3 -0
  5. model-00003-of-00003.safetensors +3 -0
  6. special_tokens_map.json +23 -0
  7. tokenizer.json +0 -0
  8. tokenizer_config.json +248 -0
README.md ADDED
@@ -0,0 +1,172 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ language:
4
+ - en
5
+ pipeline_tag: text-generation
6
+ base_model:
7
+ - allenai/OLMoE-1B-7B-0125-DPO
8
+ library_name: transformers
9
+ datasets:
10
+ - allenai/RLVR-GSM
11
+ ---
12
+
13
+ <img alt="OLMo Logo" src="https://huggingface.co/allenai/OLMoE-1B-7B-0125/resolve/main/olmoe-logo.png" width="242px">
14
+
15
+
16
+ # OLMoE-1B-7B-0125-Instruct
17
+
18
+
19
+ ## Release Documentation
20
+
21
+ OLMoE-1B-7B-0125-Instruct January 2025 is post-trained variant of the [OLMoE-1B-7B January 2025](https://huggingface.co/allenai/OLMoE-1B-7B-0125) model, which has undergone supervised finetuning on an OLMo-specific variant of the [Tülu 3 dataset](allenai/tulu-3-sft-olmo-2-mixture) and further DPO training on [this dataset](https://huggingface.co/datasets/allenai/olmo-2-1124-13b-preference-mix), and finally RLVR training using [this data](https://huggingface.co/datasets/allenai/RLVR-GSM).
22
+ Tülu 3 is designed for state-of-the-art performance on a diversity of tasks in addition to chat, such as MATH, GSM8K, and IFEval.
23
+ Check out the [OLMoE paper](https://arxiv.org/abs/2409.02060) or [Tülu 3 paper](https://arxiv.org/abs/2411.15124) for more details!
24
+
25
+ OLMo is a series of **O**pen **L**anguage **Mo**dels designed to enable the science of language models.
26
+ These models are trained on the Dolma dataset. We are releasing all code, checkpoints, logs (coming soon), and associated training details.
27
+ The core models released in this batch include the following:
28
+
29
+
30
+ | **Stage** | **OLMoE 1B-7B** |
31
+ |----------------------|----------------------------------------------------------------------------------------------------------|
32
+ | **Base Model** | [allenai/OLMoE-1B-7B-0125](https://huggingface.co/allenai/OLMoE-1B-7B-0125) |
33
+ | **SFT** | [allenai/OLMoE-1B-7B-0125-SFT](https://huggingface.co/allenai/OLMoE-1B-7B-0125-SFT) |
34
+ | **DPO** | [allenai/OLMoE-1B-7B-0125-DPO](https://huggingface.co/allenai/OLMoE-1B-7B-0125-DPO) |
35
+ | **Final Models (RLVR)** | [allenai/OLMoE-1B-7B-0125-Instruct](https://huggingface.co/allenai/OLMoE-1B-7B-0125-Instruct) |
36
+ | **Reward Model (RM)**| [allenai/OLMoE-1B-7B-0125-RM](https://huggingface.co/allenai/OLMoE-1B-7B-0125-RM) |
37
+
38
+
39
+
40
+ ## Model description
41
+
42
+ - **Model type:** A model trained on a mix of publicly available, synthetic and human-created datasets.
43
+ - **Language(s) (NLP):** Primarily English
44
+ - **License:** Apache 2.0
45
+ - **Finetuned from model:** allenai/OLMoE-1B-7B-0125-DPO
46
+
47
+ ### Model Sources
48
+
49
+ - **Project Page:** https://allenai.org/olmo
50
+ - **Repositories:**
51
+ - Core repo (training, inference, fine-tuning etc.): https://github.com/allenai/OLMo
52
+ - Evaluation code: https://github.com/allenai/olmes
53
+ - Further fine-tuning code: https://github.com/allenai/open-instruct
54
+ - **Paper:** https://arxiv.org/abs/2409.02060
55
+ - **Demo:** https://playground.allenai.org/
56
+
57
+ ## Installation
58
+
59
+ OLMo 2 will be supported in the next version of Transformers, and you need to install it from the main branch using:
60
+ ```bash
61
+ pip install --upgrade git+https://github.com/huggingface/transformers.git
62
+ ```
63
+
64
+ ## Using the model
65
+
66
+ ### Loading with HuggingFace
67
+
68
+ To load the model with HuggingFace, use the following snippet:
69
+ ```
70
+ from transformers import AutoModelForCausalLM
71
+
72
+ olmo_model = AutoModelForCausalLM.from_pretrained("OLMoE-1B-7B-0125-Instruct")
73
+ ```
74
+
75
+ ### Chat template
76
+
77
+ The chat template for our models is formatted as:
78
+ ```
79
+ <|endoftext|><|user|>\nHow are you doing?\n<|assistant|>\nI'm just a computer program, so I don't have feelings, but I'm functioning as expected. How can I assist you today?<|endoftext|>
80
+ ```
81
+ Or with new lines expanded:
82
+ ```
83
+ <|endoftext|><|user|>
84
+ How are you doing?
85
+ <|assistant|>
86
+ I'm just a computer program, so I don't have feelings, but I'm functioning as expected. How can I assist you today?<|endoftext|>
87
+ ```
88
+ It is embedded within the tokenizer as well, for `tokenizer.apply_chat_template`.
89
+
90
+ ### System prompt
91
+
92
+ In Ai2 demos, we use this system prompt by default:
93
+ ```
94
+ You are OLMo 2, a helpful and harmless AI Assistant built by the Allen Institute for AI.
95
+ ```
96
+ The model has not been trained with a specific system prompt in mind.
97
+
98
+ ### Bias, Risks, and Limitations
99
+
100
+ The OLMo-2 models have limited safety training, but are not deployed automatically with in-the-loop filtering of responses like ChatGPT, so the model can produce problematic outputs (especially when prompted to do so).
101
+ See the Falcon 180B model card for an example of this.
102
+
103
+
104
+ ## Performance
105
+
106
+ | Benchmark (eval) | OLMoE-1B-7B-0125-Instruct | OLMoE-1B-7B-0924-Instruct | OLMoE-1B-7B-0125-DPO | OLMoE-1B-7B-0125-SFT | OLMoE-1B-7B-0924-SFT |
107
+ |--------------------------------|---------------------------|--------------------------|----------------------|---------------------|---------------------|
108
+ | **Avg.** | **45.62** | 38.44 | 45.05 | 41.76 | 37.05 |
109
+ | **MMLU (CoT)** | 55.08 | 54.57 | 54.93 | **55.26** | 54.32 |
110
+ | **PopQA** | 19.75 | 20.56 | 19.65 | 20.12 | **21.01** |
111
+ | **TruthfulQA** | **50.56** | 49.14 | 49.99 | 45.48 | 44.66 |
112
+ | **BigBenchHard (CoT)** | **38.61** | 36.78 | 37.37 | 37.31 | 36.55 |
113
+ | **DROP** | 47.87 | 34.48 | 48.38 | **48.57** | 34.71 |
114
+ | **MATH (Flex)** | **21.41** | 8.16 | 20.36 | 21.38 | 8.15 |
115
+ | **GSM8K** | **72.40** | 47.38 | 64.59 | 55.72 | 42.46 |
116
+ | **HumanEval** | 62.30 | 63.04 | 61.92 | 62.58 | **63.72** |
117
+ | **HumanEval+** | 54.37 | **58.93** | 57.61 | 55.67 | 57.40 |
118
+ | **IFEval** | **66.36** | 45.29 | 65.62 | 56.56 | 41.22 |
119
+ | **AlpacaEval** | 17.99 | 7.54 | **19.50** | 5.83 | 6.38 |
120
+ | **Safety (average)** | 90.40 | 51.40 | 91.40 | **94.50** | 65.80 |
121
+
122
+ ## License and use
123
+
124
+ OLMoE is licensed under the Apache 2.0 license.
125
+ OLMoE is intended for research and educational use.
126
+ For more information, please see our [Responsible Use Guidelines](https://allenai.org/responsible-use).
127
+ This model has been fine-tuned using a dataset mix with outputs generated from third party models and are subject to additional terms: [Gemma Terms of Use](https://ai.google.dev/gemma/terms).
128
+
129
+ ## Citation
130
+
131
+ ```bibtex
132
+ @misc{muennighoff2024olmoeopenmixtureofexpertslanguage,
133
+ title={OLMoE: Open Mixture-of-Experts Language Models},
134
+ author={Niklas Muennighoff and Luca Soldaini and Dirk Groeneveld and Kyle Lo and Jacob Morrison and Sewon Min and Weijia Shi and Pete Walsh and Oyvind Tafjord and Nathan Lambert and Yuling Gu and Shane Arora and Akshita Bhagia and Dustin Schwenk and David Wadden and Alexander Wettig and Binyuan Hui and Tim Dettmers and Douwe Kiela and Ali Farhadi and Noah A. Smith and Pang Wei Koh and Amanpreet Singh and Hannaneh Hajishirzi},
135
+ year={2024},
136
+ eprint={2409.02060},
137
+ archivePrefix={arXiv},
138
+ primaryClass={cs.CL},
139
+ url={https://arxiv.org/abs/2409.02060},
140
+ }
141
+ @article{lambert2024tulu3,
142
+ title = {Tülu 3: Pushing Frontiers in Open Language Model Post-Training},
143
+ author = {
144
+ Nathan Lambert and
145
+ Jacob Morrison and
146
+ Valentina Pyatkin and
147
+ Shengyi Huang and
148
+ Hamish Ivison and
149
+ Faeze Brahman and
150
+ Lester James V. Miranda and
151
+ Alisa Liu and
152
+ Nouha Dziri and
153
+ Shane Lyu and
154
+ Yuling Gu and
155
+ Saumya Malik and
156
+ Victoria Graf and
157
+ Jena D. Hwang and
158
+ Jiangjiang Yang and
159
+ Ronan Le Bras and
160
+ Oyvind Tafjord and
161
+ Chris Wilhelm and
162
+ Luca Soldaini and
163
+ Noah A. Smith and
164
+ Yizhong Wang and
165
+ Pradeep Dasigi and
166
+ Hannaneh Hajishirzi
167
+ },
168
+ year = {2024},
169
+ email = {tulu@allenai.org}
170
+ }
171
+ ```
172
+
generation_config.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "eos_token_id": 50279,
4
+ "pad_token_id": 1,
5
+ "transformers_version": "4.47.1"
6
+ }
model-00001-of-00003.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:61874210ca7c360f43f8c622cecc12441083d40190eae3b56bc9d6e1c0a30c1e
3
+ size 4997744872
model-00002-of-00003.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c523a43b8a17269d5fab33395048a83633f4d1d89c1958570cea738e2bbe80c9
3
+ size 4997235176
model-00003-of-00003.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:97ae01e3519c52e63a018bca96ab17a89c4cd5cab1c6d742efed0fa5c0e2bb17
3
+ size 3843741912
special_tokens_map.json ADDED
@@ -0,0 +1,23 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": {
3
+ "content": "|||IP_ADDRESS|||",
4
+ "lstrip": false,
5
+ "normalized": true,
6
+ "rstrip": false,
7
+ "single_word": false
8
+ },
9
+ "eos_token": {
10
+ "content": "|||IP_ADDRESS|||",
11
+ "lstrip": false,
12
+ "normalized": true,
13
+ "rstrip": false,
14
+ "single_word": false
15
+ },
16
+ "pad_token": {
17
+ "content": "<pad>",
18
+ "lstrip": false,
19
+ "normalized": false,
20
+ "rstrip": false,
21
+ "single_word": false
22
+ }
23
+ }
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1,248 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_bos_token": false,
3
+ "add_eos_token": false,
4
+ "add_prefix_space": false,
5
+ "added_tokens_decoder": {
6
+ "0": {
7
+ "content": "<|endoftext|>",
8
+ "lstrip": false,
9
+ "normalized": false,
10
+ "rstrip": false,
11
+ "single_word": false,
12
+ "special": true
13
+ },
14
+ "1": {
15
+ "content": "<|padding|>",
16
+ "lstrip": false,
17
+ "normalized": false,
18
+ "rstrip": false,
19
+ "single_word": false,
20
+ "special": true
21
+ },
22
+ "50254": {
23
+ "content": " ",
24
+ "lstrip": false,
25
+ "normalized": true,
26
+ "rstrip": false,
27
+ "single_word": false,
28
+ "special": false
29
+ },
30
+ "50255": {
31
+ "content": " ",
32
+ "lstrip": false,
33
+ "normalized": true,
34
+ "rstrip": false,
35
+ "single_word": false,
36
+ "special": false
37
+ },
38
+ "50256": {
39
+ "content": " ",
40
+ "lstrip": false,
41
+ "normalized": true,
42
+ "rstrip": false,
43
+ "single_word": false,
44
+ "special": false
45
+ },
46
+ "50257": {
47
+ "content": " ",
48
+ "lstrip": false,
49
+ "normalized": true,
50
+ "rstrip": false,
51
+ "single_word": false,
52
+ "special": false
53
+ },
54
+ "50258": {
55
+ "content": " ",
56
+ "lstrip": false,
57
+ "normalized": true,
58
+ "rstrip": false,
59
+ "single_word": false,
60
+ "special": false
61
+ },
62
+ "50259": {
63
+ "content": " ",
64
+ "lstrip": false,
65
+ "normalized": true,
66
+ "rstrip": false,
67
+ "single_word": false,
68
+ "special": false
69
+ },
70
+ "50260": {
71
+ "content": " ",
72
+ "lstrip": false,
73
+ "normalized": true,
74
+ "rstrip": false,
75
+ "single_word": false,
76
+ "special": false
77
+ },
78
+ "50261": {
79
+ "content": " ",
80
+ "lstrip": false,
81
+ "normalized": true,
82
+ "rstrip": false,
83
+ "single_word": false,
84
+ "special": false
85
+ },
86
+ "50262": {
87
+ "content": " ",
88
+ "lstrip": false,
89
+ "normalized": true,
90
+ "rstrip": false,
91
+ "single_word": false,
92
+ "special": false
93
+ },
94
+ "50263": {
95
+ "content": " ",
96
+ "lstrip": false,
97
+ "normalized": true,
98
+ "rstrip": false,
99
+ "single_word": false,
100
+ "special": false
101
+ },
102
+ "50264": {
103
+ "content": " ",
104
+ "lstrip": false,
105
+ "normalized": true,
106
+ "rstrip": false,
107
+ "single_word": false,
108
+ "special": false
109
+ },
110
+ "50265": {
111
+ "content": " ",
112
+ "lstrip": false,
113
+ "normalized": true,
114
+ "rstrip": false,
115
+ "single_word": false,
116
+ "special": false
117
+ },
118
+ "50266": {
119
+ "content": " ",
120
+ "lstrip": false,
121
+ "normalized": true,
122
+ "rstrip": false,
123
+ "single_word": false,
124
+ "special": false
125
+ },
126
+ "50267": {
127
+ "content": " ",
128
+ "lstrip": false,
129
+ "normalized": true,
130
+ "rstrip": false,
131
+ "single_word": false,
132
+ "special": false
133
+ },
134
+ "50268": {
135
+ "content": " ",
136
+ "lstrip": false,
137
+ "normalized": true,
138
+ "rstrip": false,
139
+ "single_word": false,
140
+ "special": false
141
+ },
142
+ "50269": {
143
+ "content": " ",
144
+ "lstrip": false,
145
+ "normalized": true,
146
+ "rstrip": false,
147
+ "single_word": false,
148
+ "special": false
149
+ },
150
+ "50270": {
151
+ "content": " ",
152
+ "lstrip": false,
153
+ "normalized": true,
154
+ "rstrip": false,
155
+ "single_word": false,
156
+ "special": false
157
+ },
158
+ "50271": {
159
+ "content": " ",
160
+ "lstrip": false,
161
+ "normalized": true,
162
+ "rstrip": false,
163
+ "single_word": false,
164
+ "special": false
165
+ },
166
+ "50272": {
167
+ "content": " ",
168
+ "lstrip": false,
169
+ "normalized": true,
170
+ "rstrip": false,
171
+ "single_word": false,
172
+ "special": false
173
+ },
174
+ "50273": {
175
+ "content": " ",
176
+ "lstrip": false,
177
+ "normalized": true,
178
+ "rstrip": false,
179
+ "single_word": false,
180
+ "special": false
181
+ },
182
+ "50274": {
183
+ "content": " ",
184
+ "lstrip": false,
185
+ "normalized": true,
186
+ "rstrip": false,
187
+ "single_word": false,
188
+ "special": false
189
+ },
190
+ "50275": {
191
+ "content": " ",
192
+ "lstrip": false,
193
+ "normalized": true,
194
+ "rstrip": false,
195
+ "single_word": false,
196
+ "special": false
197
+ },
198
+ "50276": {
199
+ "content": " ",
200
+ "lstrip": false,
201
+ "normalized": true,
202
+ "rstrip": false,
203
+ "single_word": false,
204
+ "special": false
205
+ },
206
+ "50277": {
207
+ "content": "|||EMAIL_ADDRESS|||",
208
+ "lstrip": false,
209
+ "normalized": true,
210
+ "rstrip": false,
211
+ "single_word": false,
212
+ "special": false
213
+ },
214
+ "50278": {
215
+ "content": "|||PHONE_NUMBER|||",
216
+ "lstrip": false,
217
+ "normalized": true,
218
+ "rstrip": false,
219
+ "single_word": false,
220
+ "special": false
221
+ },
222
+ "50279": {
223
+ "content": "|||IP_ADDRESS|||",
224
+ "lstrip": false,
225
+ "normalized": true,
226
+ "rstrip": false,
227
+ "single_word": false,
228
+ "special": true
229
+ },
230
+ "50280": {
231
+ "content": "<pad>",
232
+ "lstrip": false,
233
+ "normalized": false,
234
+ "rstrip": false,
235
+ "single_word": false,
236
+ "special": true
237
+ }
238
+ },
239
+ "bos_token": "|||IP_ADDRESS|||",
240
+ "chat_template": "{{ bos_token }}{% for message in messages %}{% if message['role'] == 'system' %}{{ '<|system|>\n' + message['content'] + '\n' }}{% elif message['role'] == 'user' %}{{ '<|user|>\n' + message['content'] + '\n' }}{% elif message['role'] == 'assistant' %}{% if not loop.last %}{{ '<|assistant|>\n' + message['content'] + eos_token + '\n' }}{% else %}{{ '<|assistant|>\n' + message['content'] + eos_token }}{% endif %}{% endif %}{% if loop.last and add_generation_prompt %}{{ '<|assistant|>\n' }}{% endif %}{% endfor %}",
241
+ "clean_up_tokenization_spaces": false,
242
+ "eos_token": "|||IP_ADDRESS|||",
243
+ "extra_special_tokens": {},
244
+ "model_max_length": 1000000000000000019884624838656,
245
+ "pad_token": "<pad>",
246
+ "tokenizer_class": "GPTNeoXTokenizer",
247
+ "unk_token": null
248
+ }