add scores for 1B

README.md

@@ -7,21 +7,16 @@ language:
 tags:
 - falcon3
 base_model: tiiuae/Falcon3-7B-Base
-license: other
-license_name: falcon-llm-license
+license: other 
+license_name: falcon-llm-license 
 license_link: https://falconllm.tii.ae/falcon-terms-and-conditions.html
-library_name: transformers
 ---
 
-<div align="center">
-    <img src="https://huggingface.co/datasets/tiiuae/documentation-images/resolve/main/general/falco3-logo.png" alt="drawing" width="500"/>
-</div>
-
 # Falcon3-7B-Instruct
 
 **Falcon3** family of Open Foundation Models is a set of pretrained and instruct LLMs ranging from 1B to 10B.
 
-This repository contains the **Falcon3-7B-Instruct**. It achieves state of art results (at the time of release) on reasoning, language understanding, instruction following, code and mathematics tasks.
+This repository contains the **Falcon3-7B-Instruct**. It achieves state of art results (at release's time) on reasoning, language understanding, instruction following, code and mathematics tasks.
 Falcon3-7B-Instruct supports 4 languages (english, french, spanish, portuguese) and a context length up to 32K.
 
 ## Model Details
@@ -34,7 +29,7 @@ Falcon3-7B-Instruct supports 4 languages (english, french, spanish, portuguese)
   - Uses SwiGLU and RMSNorm
   - 32K context length
   - 131K vocab size
-- Pretrained on 14 Teratokens of datasets comprising of web, code, STEM, high quality and mutlilingual data using 1024 H100 GPU chips
+- Pretrained on 14 Teratokens of datasets comprising of web, code, STEM, high quality and mutlilingual data using 2048 H100 GPU chips
 - Postrained on 1.2 million samples of STEM, conversations, code, safety and function call data
 - Supports EN, FR, ES, PT
 - Developed by [Technology Innovation Institute](https://www.tii.ae)
@@ -48,6 +43,8 @@ Falcon3-7B-Instruct supports 4 languages (english, french, spanish, portuguese)
 <summary> Click to expand </summary>
 
 ```python
+from transformers import AutoTokenizer, AutoModelForCausalLM
+
 
 from transformers import AutoModelForCausalLM, AutoTokenizer
 
@@ -56,7 +53,7 @@ model_name = "tiiuae/Falcon3-7B-Instruct"
 model = AutoModelForCausalLM.from_pretrained(
     model_name,
     torch_dtype="auto",
-    device_map="auto"
+    device_map="auto"]
 )
 tokenizer = AutoTokenizer.from_pretrained(model_name)
 
@@ -89,66 +86,7 @@ print(response)
 <br>
 
 ## Benchmarks
-We report the official HuggingFace leaderboard normalized evaluations [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard) in the following table.
-<table border="1" style="width: 100%; text-align: center; border-collapse: collapse;">
-    <colgroup>
-        <col style="width: 10%;">
-        <col style="width: 7%;">
-        <col style="width: 7%;">
-        <col style="background-color: rgba(80, 15, 213, 0.5); width: 7%;">
-    </colgroup>
-    <thead>
-        <tr>
-            <th>Benchmark</th>
-            <th>Llama-3.1-8B-Instruct</th>
-            <th>Qwen2.5-7B-Instruct</th>
-            <th>Falcon3-7B-Instruct</th>
-        </tr>
-    </thead>
-    <tbody>
-        <tr>
-            <td>IFEval</td>
-            <td><b>78.56</b></td>
-            <td>75.85</td>
-            <td>76.12</td>
-        </tr>
-        <tr>
-            <td>BBH (3-shot)</td>
-            <td>29.89</td>
-            <td>34.89</td>
-            <td><b>37.92</b></td>
-        </tr>
-        <tr>
-            <td>MATH Lvl-5 (4-shot)</td>
-            <td>19.34</td>
-            <td>0.00</td>
-            <td><b>31.87</b></td>
-        </tr>              
-        <tr>
-            <td>GPQA (0-shot)</td>
-            <td>2.35</td>
-            <td>5.48</td>
-            <td><b>8.05</b></td>
-        </tr>
-        <tr>
-            <td>MUSR (0-shot)</td>
-            <td>8.41</td>
-            <td>8.45</td>
-            <td><b>21.17</b></td>
-        </tr>
-        <tr>
-            <td>MMLU-PRO (5-shot)</td>
-            <td>30.68</td>
-            <td><b>36.52</b></td>
-            <td>34.30</td>
-        </tr>        
-    </tbody>
-</table>
-
-Also, we report in the following table our internal pipeline benchmarks.
- - We use [lm-evaluation harness](https://github.com/EleutherAI/lm-evaluation-harness).
- - We report **raw scores** obtained by applying chat template and fewshot_as_multiturn.
- - We use same batch-size across all models.
+We report in the following table our internal pipeline benchmarks:
 
 <table border="1" style="width: 100%; text-align: center; border-collapse: collapse;">
     <colgroup>
@@ -156,139 +94,154 @@ Also, we report in the following table our internal pipeline benchmarks.
         <col style="width: 10%;">
         <col style="width: 7%;">
         <col style="width: 7%;">
+        <col style="width: 7%;">
         <col style="background-color: rgba(80, 15, 213, 0.5); width: 7%;">
     </colgroup>
     <thead>
         <tr>
             <th>Category</th>
             <th>Benchmark</th>
-            <th>Llama-3.1-8B-Instruct</th>
-            <th>Qwen2.5-7B-Instruct</th>
-            <th>Falcon3-7B-Instruct</th>
+            <th>Llama-3.2-1B</th>
+            <th>Qwen2.5-1.5B</th>
+            <th>SmolLM2-1.7B</th>
+            <th>Falcon3-1B-Instruct</th>
         </tr>
     </thead>
     <tbody>
         <tr>
             <td rowspan="3">General</td>
             <td>MMLU (5-shot)</td>
-            <td>68.2</td>
-            <td><b>73.5</b></td>
-            <td>70.5</td>
+            <td>23.4</td>
+            <td><b>58.4</b></td>
+            <td>48.4</td>
+            <td>43.9</td>
         </tr>
         <tr>
             <td>MMLU-PRO (5-shot)</td>
-            <td>36.4</td>
-            <td><b>43.1</b></td>
-            <td>40.7</td>
+            <td>11.3</td>
+            <td><b>21.3</b></td>
+            <td>17.2</td>
+            <td>18.6</td>
         </tr>
         <tr>
             <td>IFEval</td>
-            <td><b>78.8</b></td>
-            <td>74.7</td>
-            <td>76.5</td>
+            <td><b>55.8</b></td>
+            <td>44.4</td>
+            <td>53.0</td>
+            <td>54.4</td>
         </tr>
         <tr>
             <td rowspan="3">Math</td>
             <td>GSM8K (5-shot)</td>
-            <td><b>82.6</b></td>
-            <td>72.0</td>
-            <td>81.4</td>
+            <td>37.4</td>
+            <td><b>57.2</b></td>
+            <td>43.4</td>
+            <td>38.6</td>
         </tr>
         <tr>
             <td>GSM8K (8-shot, COT)</td>
-            <td><b>85.4</b></td>
-            <td>76.6</td>
-            <td>79.7</td>
+            <td>35.6</td>
+            <td><b>62.2</b></td>
+            <td>47.2</td>
+            <td>41.8</td>
         </tr>
         <tr>
             <td>MATH Lvl-5 (4-shot)</td>
-            <td>15.4</td>
-            <td>-</td>
-            <td><b>29.4</b></td>
+            <td><b>3.9</b></td>
+            <td>0.2</td>
+            <td>0.1</td>
+            <td>1.0</td>
         </tr>
         <tr>
-            <td rowspan="5">Reasoning</td>
+            <td rowspan="6">Reasoning</td>
             <td>Arc Challenge (25-shot)</td>
-            <td>58.6</td>
-            <td>57.8</td>
-            <td><b>62.6</b></td>
+            <td>34.1</td>
+            <td>47.0</td>
+            <td><b>47.6</b></td>
+            <td>45.9</td>
         </tr>
         <tr>
             <td>GPQA (0-shot)</td>
-            <td><b>33.5</b></td>
-            <td>32</td>
-            <td>31.9</td>
+            <td>25.3</td>
+            <td><b>29.6</b></td>
+            <td>28.7</td>
+            <td>26.5</td>
         </tr>
         <tr>
             <td>GPQA (0-shot, COT)</td>
-            <td>9.6</td>
-            <td>13.8</td>
-            <td><b>22.3</b></td>
+            <td>13.2</td>
+            <td>9.2</td>
+            <td>16.0</td>
+            <td><b>21.3</b></td>
         </tr>
         <tr>
             <td>MUSR (0-shot)</td>
-            <td>38.6</td>
-            <td>41</td>
-            <td><b>46.4</b></td>
+            <td>32.4</td>
+            <td>36.8</td>
+            <td>33.0</td>
+            <td><b>40.7</b></td>
         </tr>
         <tr>
             <td>BBH (3-shot)</td>
-            <td>48.6</td>
-            <td><b>54.1</b></td>
-            <td>52.4</td>
+            <td>30.3</td>
+            <td><b>38.5</b></td>
+            <td>33.1</td>
+            <td>35.1</td>
+        </tr>
+        <tr>
+            <td>BBH (3-shot, COT)</td>
+            <td>0.0</td>
+            <td>20.3</td>
+            <td>0.8</td>
+            <td><b>30.5</b></td>
         </tr>
         <tr>
-            <td rowspan="4">CommonSense Understanding</td>
+            <td rowspan="5">CommonSense Understanding</td>
             <td>PIQA (0-shot)</td>
-            <td><b>78.9</b></td>
-            <td>73.7</td>
-            <td>78.8</td>
+            <td>72.1</td>
+            <td>73.2</td>
+            <td><b>74.4</b></td>
+            <td>72.0</td>
         </tr>
         <tr>
             <td>SciQ (0-shot)</td>
-            <td>80.2</td>
-            <td>50.9</td>
-            <td><b>94.7</b></td>
+            <td>61.8</td>
+            <td>69.5</td>
+            <td>71.4</td>
+            <td><b>86.8</b></td>
         </tr>
         <tr>
             <td>Winogrande (0-shot)</td>
             <td>-</td>
             <td>-</td>
-            <td>70.4</td>
+            <td>-</td>
+            <td><b>60.2</b></td>
         </tr>
         <tr>
             <td>OpenbookQA (0-shot)</td>
-            <td><b>46.2</b></td>
-            <td>42.4</td>
-            <td>45.8</td>
+            <td>40.2</td>
+            <td>40.4</td>
+            <td><b>42.8</b></td>
+            <td>40.0</td>
         </tr>
         <tr>
-            <td rowspan="2">Instructions following</td>
             <td>MT-Bench (avg)</td>
-            <td>7.9</td>
-            <td><b>8.5</b></td>
-            <td>8.4</td>
+            <td>5.4</td>
+            <td><b>7.1</b></td>
+            <td>6.1</td>
+            <td>5.5</td>
         </tr>
         <tr>
-            <td>Alpaca (WC)</td>
-            <td>26.6</td>
-            <td><b>31.5</b></td>
-            <td>26.1</td>
-        </tr>
-        <tr>
-            <td>Tool use</td>
-            <td>BFCL AST (avg)</td>
-            <td>90.6</td>
-            <td><b>91.4</b></td>
-            <td>89.5</td>
+            <td rowspan="1">Instructions following</td>
+            <td>Alapaca (WC)</td>
+            <td><b>8.6</b></td>
+            <td><b>8.6</b></td>
+            <td>5.4</td>
+            <td>6.1</td>
         </tr>
     </tbody>
 </table>
 
-## Useful links
-- View our [release blogpost](https://huggingface.co/blog/falcon3).
-- Feel free to join [our discord server](https://discord.gg/fwXpMyGc) if you have any questions or to interact with our researchers and developers.
-
 ## Technical Report
 Coming soon....
 

tokenizer_config.json

@@ -16219,7 +16219,7 @@
     ">>PASSWORD<<",
     ">>KEY<<"
   ],
-  "chat_template": "{%- if tools %}\n{{- '<|system|>\\n' }}\n{%- if messages[0]['role'] == 'system' %}\n{{- messages[0]['content'] }}\n{%- set remaining_messages = messages[1:] %}\n{%- else %}\n{%- set remaining_messages = messages %}\n{%- endif %}\n{{- 'You are a Falcon assistant skilled in function calling. You are helpful, respectful, and concise.\\n\\n# Tools\\n\\nYou have access to the following functions. You MUST use them to answer questions when needed. For each function call, you MUST return a JSON object inside <tool_call></tool_call> tags.\\n\\n<tools>' + tools|tojson(indent=2) + '</tools>\\n\\n# Output Format\\n\\nYour response MUST follow this format when making function calls:\\n<tool_call>\\n[\\n  {\"name\": \"function_name\", \"arguments\": {\"arg1\": \"value1\", \"arg2\": \"value2\"}},\\n  {\"name\": \"another_function\", \"arguments\": {\"arg\": \"value\"}}\\n]\\n</tool_call>\\nIf no function calls are needed, respond normally without the tool_call tags.\\n' }}\n{%- for message in remaining_messages %}\n{%- if message['role'] == 'user' %}\n{{- '<|user|>\\n' + message['content'] + '\\n' }}\n{%- elif message['role'] == 'assistant' %}\n{%- if message.content %}\n{{- '<|assistant|>\\n' + message['content'] }}\n{%- endif %}\n{%- if message.tool_calls %}\n{{- '\\n<tool_call>\\n' }}\n{{- message.tool_calls|tojson(indent=2) }}\n{{- '\\n</tool_call>' }}\n{%- endif %}\n{{- eos_token + '\\n' }}\n{%- elif message['role'] == 'tool' %}\n{{- '<|assistant|>\\n<tool_response>\\n' + message['content'] + '\\n</tool_response>\\n' }}\n{%- endif %}\n{%- endfor %}\n{{- '<|assistant|>\\n' if add_generation_prompt }}\n{%- else %}\n{%- for message in messages %}\n{%- if message['role'] == 'system' %}\n{{- '<|system|>\\n' + message['content'] + '\\n' }}\n{%- elif message['role'] == 'user' %}\n{{- '<|user|>\\n' + message['content'] + '\\n' }}\n{%- elif message['role'] == 'assistant' %}\n{%- if not loop.last %}\n{{- '<|assistant|>\\n' + message['content'] + eos_token + '\\n' }}\n{%- else %}\n{{- '<|assistant|>\\n' + message['content'] + eos_token }}\n{%- endif %}\n{%- endif %}\n{%- if loop.last and add_generation_prompt %}\n{{- '<|assistant|>\\n' }}\n{%- endif %}\n{%- endfor %}\n{%- endif %}",
+  "chat_template": "{% for message in messages %}{% if message['role'] == 'system' %}{{ '<|system|>\n' + message['content'] + '\n' }}{% elif message['role'] == 'user' %}{{ '<|user|>\n' + message['content'] + '\n' }}{% elif message['role'] == 'assistant' %}{% if not loop.last %}{{ '<|assistant|>\n'  + message['content'] + eos_token + '\n' }}{% else %}{{ '<|assistant|>\n'  + message['content'] + eos_token }}{% endif %}{% endif %}{% if loop.last and add_generation_prompt %}{{ '<|assistant|>\n' }}{% endif %}{% endfor %}",
   "clean_up_tokenization_spaces": true,
   "eos_token": "<|endoftext|>",
   "extra_special_tokens": {},