Hugging Face's logo

EricB
/
Devstral-Small-2507-UQFF

mistral
uqff
mistral.rs
Model card Files
xet
Community
EricB HF Staff commited on
Commit
7555eeb
·
verified ·
1 Parent(s): a7f5a57

Upload model

Browse files
Files changed (29) hide show
  1. .gitattributes +32 -0
  2. README.md +34 -0
  3. chat_template.json +3 -0
  4. config.json +27 -0
  5. generation_config.json +7 -0
  6. mistraldevstral-small-2507-afq2-0.uqff +3 -0
  7. mistraldevstral-small-2507-afq3-0.uqff +3 -0
  8. mistraldevstral-small-2507-afq4-0.uqff +3 -0
  9. mistraldevstral-small-2507-afq4-1.uqff +3 -0
  10. mistraldevstral-small-2507-afq6-0.uqff +3 -0
  11. mistraldevstral-small-2507-afq6-1.uqff +3 -0
  12. mistraldevstral-small-2507-afq8-0.uqff +3 -0
  13. mistraldevstral-small-2507-afq8-1.uqff +3 -0
  14. mistraldevstral-small-2507-afq8-2.uqff +3 -0
  15. mistraldevstral-small-2507-f8e4m3-0.uqff +3 -0
  16. mistraldevstral-small-2507-f8e4m3-1.uqff +3 -0
  17. mistraldevstral-small-2507-f8e4m3-2.uqff +3 -0
  18. mistraldevstral-small-2507-q2k-0.uqff +3 -0
  19. mistraldevstral-small-2507-q3k-0.uqff +3 -0
  20. mistraldevstral-small-2507-q4k-0.uqff +3 -0
  21. mistraldevstral-small-2507-q4k-1.uqff +3 -0
  22. mistraldevstral-small-2507-q5k-0.uqff +3 -0
  23. mistraldevstral-small-2507-q5k-1.uqff +3 -0
  24. mistraldevstral-small-2507-q8_0-0.uqff +3 -0
  25. mistraldevstral-small-2507-q8_0-1.uqff +3 -0
  26. mistraldevstral-small-2507-q8_0-2.uqff +3 -0
  27. residual.safetensors +3 -0
  28. tokenizer.json +3 -0
  29. tokenizer_config.json +0 -0
.gitattributes CHANGED
@@ -33,3 +33,35 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ tekken.json filter=lfs diff=lfs merge=lfs -text
37
+ tokenizer.json filter=lfs diff=lfs merge=lfs -text
38
+ assets/cline_config.png filter=lfs diff=lfs merge=lfs -text
39
+ assets/mistral_common_coverage/dependencies.png filter=lfs diff=lfs merge=lfs -text
40
+ assets/mistral_common_coverage/navigate.png filter=lfs diff=lfs merge=lfs -text
41
+ assets/mistral_common_coverage/prompt.png filter=lfs diff=lfs merge=lfs -text
42
+ assets/mistral_common_coverage/visualization.png filter=lfs diff=lfs merge=lfs -text
43
+ assets/space_invaders_pong/base_structure.png filter=lfs diff=lfs merge=lfs -text
44
+ assets/space_invaders_pong/prompt.png filter=lfs diff=lfs merge=lfs -text
45
+ assets/space_invaders_pong/task[[:space:]]completed.png filter=lfs diff=lfs merge=lfs -text
46
+ assets/swe_benchmark.png filter=lfs diff=lfs merge=lfs -text
47
+ mistraldevstral-small-2507-afq2-0.uqff filter=lfs diff=lfs merge=lfs -text
48
+ mistraldevstral-small-2507-afq3-0.uqff filter=lfs diff=lfs merge=lfs -text
49
+ mistraldevstral-small-2507-afq4-0.uqff filter=lfs diff=lfs merge=lfs -text
50
+ mistraldevstral-small-2507-afq4-1.uqff filter=lfs diff=lfs merge=lfs -text
51
+ mistraldevstral-small-2507-afq6-0.uqff filter=lfs diff=lfs merge=lfs -text
52
+ mistraldevstral-small-2507-afq6-1.uqff filter=lfs diff=lfs merge=lfs -text
53
+ mistraldevstral-small-2507-afq8-0.uqff filter=lfs diff=lfs merge=lfs -text
54
+ mistraldevstral-small-2507-afq8-1.uqff filter=lfs diff=lfs merge=lfs -text
55
+ mistraldevstral-small-2507-afq8-2.uqff filter=lfs diff=lfs merge=lfs -text
56
+ mistraldevstral-small-2507-f8e4m3-0.uqff filter=lfs diff=lfs merge=lfs -text
57
+ mistraldevstral-small-2507-f8e4m3-1.uqff filter=lfs diff=lfs merge=lfs -text
58
+ mistraldevstral-small-2507-f8e4m3-2.uqff filter=lfs diff=lfs merge=lfs -text
59
+ mistraldevstral-small-2507-q2k-0.uqff filter=lfs diff=lfs merge=lfs -text
60
+ mistraldevstral-small-2507-q3k-0.uqff filter=lfs diff=lfs merge=lfs -text
61
+ mistraldevstral-small-2507-q4k-0.uqff filter=lfs diff=lfs merge=lfs -text
62
+ mistraldevstral-small-2507-q4k-1.uqff filter=lfs diff=lfs merge=lfs -text
63
+ mistraldevstral-small-2507-q5k-0.uqff filter=lfs diff=lfs merge=lfs -text
64
+ mistraldevstral-small-2507-q5k-1.uqff filter=lfs diff=lfs merge=lfs -text
65
+ mistraldevstral-small-2507-q8_0-0.uqff filter=lfs diff=lfs merge=lfs -text
66
+ mistraldevstral-small-2507-q8_0-1.uqff filter=lfs diff=lfs merge=lfs -text
67
+ mistraldevstral-small-2507-q8_0-2.uqff filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,34 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ tags:
3
+ - uqff
4
+ - mistral.rs
5
+ base_model: mistralai/Devstral-Small-2507
6
+ base_model_relation: quantized
7
+ ---
8
+
9
+ <!-- Autogenerated from user input. -->
10
+
11
+ # `mistralai/Devstral-Small-2507`, UQFF quantization
12
+
13
+
14
+ Run with [mistral.rs](https://github.com/EricLBuehler/mistral.rs). Documentation: [UQFF docs](https://github.com/EricLBuehler/mistral.rs/blob/master/docs/UQFF.md).
15
+
16
+ 1) **Flexible** 🌀: Multiple quantization formats in *one* file format with *one* framework to run them all.
17
+ 2) **Reliable** 🔒: Compatibility ensured with *embedded* and *checked* semantic versioning information from day 1.
18
+ 3) **Easy** 🤗: Download UQFF models *easily* and *quickly* from Hugging Face, or use a local file.
19
+ 3) **Customizable** 🛠️: Make and publish your own UQFF files in minutes.
20
+
21
+ ## Examples
22
+ |Quantization type(s)|Example|
23
+ |--|--|
24
+ |AFQ2|`./mistralrs-server -i plain -m EricB/Devstral-Small-2507-UQFF -f mistraldevstral-small-2507-afq2-0.uqff`|
25
+ |AFQ3|`./mistralrs-server -i plain -m EricB/Devstral-Small-2507-UQFF -f mistraldevstral-small-2507-afq3-0.uqff`|
26
+ |AFQ4|`./mistralrs-server -i plain -m EricB/Devstral-Small-2507-UQFF -f "mistraldevstral-small-2507-afq4-0.uqff;mistraldevstral-small-2507-afq4-1.uqff"`|
27
+ |AFQ6|`./mistralrs-server -i plain -m EricB/Devstral-Small-2507-UQFF -f "mistraldevstral-small-2507-afq6-0.uqff;mistraldevstral-small-2507-afq6-1.uqff"`|
28
+ |AFQ8|`./mistralrs-server -i plain -m EricB/Devstral-Small-2507-UQFF -f "mistraldevstral-small-2507-afq8-0.uqff;mistraldevstral-small-2507-afq8-1.uqff;mistraldevstral-small-2507-afq8-2.uqff"`|
29
+ |F8E4M3|`./mistralrs-server -i plain -m EricB/Devstral-Small-2507-UQFF -f "mistraldevstral-small-2507-f8e4m3-0.uqff;mistraldevstral-small-2507-f8e4m3-1.uqff;mistraldevstral-small-2507-f8e4m3-2.uqff"`|
30
+ |Q2K|`./mistralrs-server -i plain -m EricB/Devstral-Small-2507-UQFF -f mistraldevstral-small-2507-q2k-0.uqff`|
31
+ |Q3K|`./mistralrs-server -i plain -m EricB/Devstral-Small-2507-UQFF -f mistraldevstral-small-2507-q3k-0.uqff`|
32
+ |Q4K|`./mistralrs-server -i plain -m EricB/Devstral-Small-2507-UQFF -f "mistraldevstral-small-2507-q4k-0.uqff;mistraldevstral-small-2507-q4k-1.uqff"`|
33
+ |Q5K|`./mistralrs-server -i plain -m EricB/Devstral-Small-2507-UQFF -f "mistraldevstral-small-2507-q5k-0.uqff;mistraldevstral-small-2507-q5k-1.uqff"`|
34
+ |Q8_0|`./mistralrs-server -i plain -m EricB/Devstral-Small-2507-UQFF -f "mistraldevstral-small-2507-q8_0-0.uqff;mistraldevstral-small-2507-q8_0-1.uqff;mistraldevstral-small-2507-q8_0-2.uqff"`|
chat_template.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ {
2
+ "chat_template": "{%- set today = strftime_now(\"%Y-%m-%d\") %}\n{%- set default_system_message = \"You are Mistral Small 3, a Large Language Model (LLM) created by Mistral AI, a French startup headquartered in Paris.\\nYour knowledge base was last updated on 2023-10-01. The current date is \" + today + \".\\n\\nWhen you're not sure about some information, you say that you don't have the information and don't make up anything.\\nIf the user's question is not clear, ambiguous, or does not provide enough context for you to accurately answer the question, you do not try to answer it right away and you rather ask the user to clarify their request (e.g. \\\"What are some good restaurants around me?\\\" => \\\"Where are you?\\\" or \\\"When is the next flight to Tokyo\\\" => \\\"Where do you travel from?\\\")\" %}\n\n{{- bos_token }}\n\n{%- if messages[0]['role'] == 'system' %}\n {%- set system_message = messages[0]['content'] %}\n {%- set loop_messages = messages[1:] %}\n{%- else %}\n {%- set system_message = default_system_message %}\n {%- set loop_messages = messages %}\n{%- endif %}\n{{- '[SYSTEM_PROMPT]' + system_message + '[/SYSTEM_PROMPT]' }}\n\n{%- for message in loop_messages %}\n {%- if message['role'] == 'user' %}\n\t {%- if message['content'] is string %}\n {{- '[INST]' + message['content'] + '[/INST]' }}\n\t {%- else %}\n\t\t {{- '[INST]' }}\n\t\t {%- for block in message['content'] %}\n\t\t\t {%- if block['type'] == 'text' %}\n\t\t\t\t {{- block['text'] }}\n\t\t\t {%- elif block['type'] == 'image' or block['type'] == 'image_url' %}\n\t\t\t\t {{- '[IMG]' }}\n\t\t\t\t{%- else %}\n\t\t\t\t {{- raise_exception('Only text and image blocks are supported in message content!') }}\n\t\t\t\t{%- endif %}\n\t\t\t{%- endfor %}\n\t\t {{- '[/INST]' }}\n\t\t{%- endif %}\n {%- elif message['role'] == 'system' %}\n {{- '[SYSTEM_PROMPT]' + message['content'] + '[/SYSTEM_PROMPT]' }}\n {%- elif message['role'] == 'assistant' %}\n {{- message['content'] + eos_token }}\n {%- else %}\n {{- raise_exception('Only user, system and assistant roles are supported!') }}\n {%- endif %}\n{%- endfor %}"
3
+ }
config.json ADDED
@@ -0,0 +1,27 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "MistralForCausalLM"
4
+ ],
5
+ "attention_dropout": 0.0,
6
+ "bos_token_id": 1,
7
+ "eos_token_id": 2,
8
+ "pad_token_id": 11,
9
+ "head_dim": 128,
10
+ "hidden_act": "silu",
11
+ "hidden_size": 5120,
12
+ "initializer_range": 0.02,
13
+ "intermediate_size": 32768,
14
+ "max_position_embeddings": 131072,
15
+ "model_type": "mistral",
16
+ "num_attention_heads": 32,
17
+ "num_hidden_layers": 40,
18
+ "num_key_value_heads": 8,
19
+ "rms_norm_eps": 1e-05,
20
+ "rope_theta": 1000000000.0,
21
+ "sliding_window": null,
22
+ "tie_word_embeddings": false,
23
+ "torch_dtype": "bfloat16",
24
+ "transformers_version": "4.53.1",
25
+ "use_cache": true,
26
+ "vocab_size": 131072
27
+ }
generation_config.json ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "bos_token_id": 1,
4
+ "eos_token_id": 2,
5
+ "pad_token_id": 11,
6
+ "transformers_version": "4.53.1"
7
+ }
mistraldevstral-small-2507-afq2-0.uqff ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4721f96981e597f53683a425f587882b05b2be42d0abb30ea33f632a2c62b3d1
3
+ size 7156572172
mistraldevstral-small-2507-afq3-0.uqff ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:24cf08a06d6ff7efaaac1399bc82e770c9501751240f7cd1b20543b3e8a57c8f
3
+ size 10019184676
mistraldevstral-small-2507-afq4-0.uqff ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9fce5f9e0ec8bbea685fc784be6cb9610e7a146ca3e577f74e598627f87a4d61
3
+ size 10723034840
mistraldevstral-small-2507-afq4-1.uqff ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e32b4f88361639860888ca67442041b706c79cd5d0a0512fa1cb24a58ab8ffa2
3
+ size 2158762404
mistraldevstral-small-2507-afq6-0.uqff ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:24f84e1f3d7c72a8c2f72f9fb818da1893a0edd4ea8a74de95db55eeb9ea3dbf
3
+ size 10658143120
mistraldevstral-small-2507-afq6-1.uqff ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:686816dd1b6ac996c7e602725c23d384ccdaa3b48462283ff5198851bdeee6a4
3
+ size 7948879220
mistraldevstral-small-2507-afq8-0.uqff ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:32e578a8a740ee8bbd8b6d37340fef550d062aebbfb6ebf7afa60ccc87e2f924
3
+ size 10572940276
mistraldevstral-small-2507-afq8-1.uqff ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:460ea3d01f914252b1e56ad2d8f4a43c07729595b936b5eb5e74fb5f6c3db7af
3
+ size 10628647016
mistraldevstral-small-2507-afq8-2.uqff ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0c2679474b0253319f16893efb16921c5efddb5e02214f8438d0bf65c660611a
3
+ size 3130660000
mistraldevstral-small-2507-f8e4m3-0.uqff ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7ad578e07bef2d6126877cfcff332a30fb41451c802cb6e491432624d78bf0b2
3
+ size 10726948246
mistraldevstral-small-2507-f8e4m3-1.uqff ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4ea56d07b0fdc1ab4855523e5d1cc543f712dc7ec9dddc5dc64dd26895005f4d
3
+ size 10726948716
mistraldevstral-small-2507-f8e4m3-2.uqff ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:379c858e1ad26cc81c3a78d8a78fd4f1ba078ce52c08e4885ac8ba6e0b8736c1
3
+ size 1447036800
mistraldevstral-small-2507-q2k-0.uqff ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:97a712e56e92acc477249f1a8cd674b84d61e5838c28c41e27ecbd720d485be4
3
+ size 7514386938
mistraldevstral-small-2507-q3k-0.uqff ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f0d3e3e46907f57712a900e6be9837dd787ac7c1746b0afc16a9ef88fac3bd83
3
+ size 9840259594
mistraldevstral-small-2507-q4k-0.uqff ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0b19687ec2c5e60acb6e1e9376e68a7e6fcde82328e3a9d61b53f28d4f748c43
3
+ size 10723024928
mistraldevstral-small-2507-q4k-1.uqff ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:98e5fc6fd75d477b5fb45382bb660f79a6934ba02619f1a62b4997f6eb7d3dc5
3
+ size 2158760514
mistraldevstral-small-2507-q5k-0.uqff ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1078e9c82fba38121117c983be9f3cc0085c5c030dcf69fb48e9ac969509c9b3
3
+ size 10662071602
mistraldevstral-small-2507-q5k-1.uqff ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:42a1444b2d96b00064fbcd19fcf69bdc09b3599a5f67f0d6640a0f9af9e4dd40
3
+ size 5082326440
mistraldevstral-small-2507-q8_0-0.uqff ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9f1cabcd34db942f7465a81ca5360509a8b42115e69919ccb48af3dcfc6190c9
3
+ size 10572935278
mistraldevstral-small-2507-q8_0-1.uqff ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a54273485035bbc65a8b864aef336369e4140f0ab9a966144509a4aede4d6dfd
3
+ size 10628641724
mistraldevstral-small-2507-q8_0-2.uqff ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:acbdc514c6390edf089113975bbf967884c2869d58966b25eb7463676e4e7720
3
+ size 3130658488
residual.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6ca1878e3cdf7e4bd1c0164478dc7a3ecf6ed3257f0c2a411414408c22eb9678
3
+ size 1343016224
tokenizer.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b76085f9923309d873994d444989f7eb6ec074b06f25b58f1e8d7b7741070949
3
+ size 17078037
tokenizer_config.json ADDED
The diff for this file is too large to render. See raw diff