Hugging Face's logo

EricB
/
Devstral-Small-2505-UQFF

mistral
uqff
mistral.rs
Model card Files
xet
Community
EricB HF Staff commited on
Commit
c44cd9f
·
verified ·
1 Parent(s): 25fbae5

Upload model

Browse files
Files changed (28) hide show
  1. .gitattributes +22 -0
  2. README.md +34 -0
  3. config.json +26 -0
  4. devstralsmall-2505-afq2-0.uqff +3 -0
  5. devstralsmall-2505-afq3-0.uqff +3 -0
  6. devstralsmall-2505-afq4-0.uqff +3 -0
  7. devstralsmall-2505-afq4-1.uqff +3 -0
  8. devstralsmall-2505-afq6-0.uqff +3 -0
  9. devstralsmall-2505-afq6-1.uqff +3 -0
  10. devstralsmall-2505-afq8-0.uqff +3 -0
  11. devstralsmall-2505-afq8-1.uqff +3 -0
  12. devstralsmall-2505-afq8-2.uqff +3 -0
  13. devstralsmall-2505-f8e4m3-0.uqff +3 -0
  14. devstralsmall-2505-f8e4m3-1.uqff +3 -0
  15. devstralsmall-2505-f8e4m3-2.uqff +3 -0
  16. devstralsmall-2505-q2k-0.uqff +3 -0
  17. devstralsmall-2505-q3k-0.uqff +3 -0
  18. devstralsmall-2505-q4k-0.uqff +3 -0
  19. devstralsmall-2505-q4k-1.uqff +3 -0
  20. devstralsmall-2505-q5k-0.uqff +3 -0
  21. devstralsmall-2505-q5k-1.uqff +3 -0
  22. devstralsmall-2505-q8_0-0.uqff +3 -0
  23. devstralsmall-2505-q8_0-1.uqff +3 -0
  24. devstralsmall-2505-q8_0-2.uqff +3 -0
  25. generation_config.json +6 -0
  26. residual.safetensors +3 -0
  27. tokenizer.json +3 -0
  28. tokenizer_config.json +0 -0
.gitattributes CHANGED
@@ -33,3 +33,25 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ devstralsmall-2505-afq2-0.uqff filter=lfs diff=lfs merge=lfs -text
37
+ devstralsmall-2505-afq3-0.uqff filter=lfs diff=lfs merge=lfs -text
38
+ devstralsmall-2505-afq4-0.uqff filter=lfs diff=lfs merge=lfs -text
39
+ devstralsmall-2505-afq4-1.uqff filter=lfs diff=lfs merge=lfs -text
40
+ devstralsmall-2505-afq6-0.uqff filter=lfs diff=lfs merge=lfs -text
41
+ devstralsmall-2505-afq6-1.uqff filter=lfs diff=lfs merge=lfs -text
42
+ devstralsmall-2505-afq8-0.uqff filter=lfs diff=lfs merge=lfs -text
43
+ devstralsmall-2505-afq8-1.uqff filter=lfs diff=lfs merge=lfs -text
44
+ devstralsmall-2505-afq8-2.uqff filter=lfs diff=lfs merge=lfs -text
45
+ devstralsmall-2505-f8e4m3-0.uqff filter=lfs diff=lfs merge=lfs -text
46
+ devstralsmall-2505-f8e4m3-1.uqff filter=lfs diff=lfs merge=lfs -text
47
+ devstralsmall-2505-f8e4m3-2.uqff filter=lfs diff=lfs merge=lfs -text
48
+ devstralsmall-2505-q2k-0.uqff filter=lfs diff=lfs merge=lfs -text
49
+ devstralsmall-2505-q3k-0.uqff filter=lfs diff=lfs merge=lfs -text
50
+ devstralsmall-2505-q4k-0.uqff filter=lfs diff=lfs merge=lfs -text
51
+ devstralsmall-2505-q4k-1.uqff filter=lfs diff=lfs merge=lfs -text
52
+ devstralsmall-2505-q5k-0.uqff filter=lfs diff=lfs merge=lfs -text
53
+ devstralsmall-2505-q5k-1.uqff filter=lfs diff=lfs merge=lfs -text
54
+ devstralsmall-2505-q8_0-0.uqff filter=lfs diff=lfs merge=lfs -text
55
+ devstralsmall-2505-q8_0-1.uqff filter=lfs diff=lfs merge=lfs -text
56
+ devstralsmall-2505-q8_0-2.uqff filter=lfs diff=lfs merge=lfs -text
57
+ tokenizer.json filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,34 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ tags:
3
+ - uqff
4
+ - mistral.rs
5
+ base_model: mistralai/Devstral-Small-2505
6
+ base_model_relation: quantized
7
+ ---
8
+
9
+ <!-- Autogenerated from user input. -->
10
+
11
+ # `mistralai/Devstral-Small-2505`, UQFF quantization
12
+
13
+
14
+ Run with [mistral.rs](https://github.com/EricLBuehler/mistral.rs). Documentation: [UQFF docs](https://github.com/EricLBuehler/mistral.rs/blob/master/docs/UQFF.md).
15
+
16
+ 1) **Flexible** 🌀: Multiple quantization formats in *one* file format with *one* framework to run them all.
17
+ 2) **Reliable** 🔒: Compatibility ensured with *embedded* and *checked* semantic versioning information from day 1.
18
+ 3) **Easy** 🤗: Download UQFF models *easily* and *quickly* from Hugging Face, or use a local file.
19
+ 3) **Customizable** 🛠️: Make and publish your own UQFF files in minutes.
20
+
21
+ ## Examples
22
+ |Quantization type(s)|Example|
23
+ |--|--|
24
+ |AFQ2|`./mistralrs-server -i plain -m EricB/Devstral-Small-2505-UQFF -f devstralsmall-2505-afq2-0.uqff`|
25
+ |AFQ3|`./mistralrs-server -i plain -m EricB/Devstral-Small-2505-UQFF -f devstralsmall-2505-afq3-0.uqff`|
26
+ |AFQ4|`./mistralrs-server -i plain -m EricB/Devstral-Small-2505-UQFF -f "devstralsmall-2505-afq4-0.uqff;devstralsmall-2505-afq4-1.uqff"`|
27
+ |AFQ6|`./mistralrs-server -i plain -m EricB/Devstral-Small-2505-UQFF -f "devstralsmall-2505-afq6-0.uqff;devstralsmall-2505-afq6-1.uqff"`|
28
+ |AFQ8|`./mistralrs-server -i plain -m EricB/Devstral-Small-2505-UQFF -f "devstralsmall-2505-afq8-0.uqff;devstralsmall-2505-afq8-1.uqff;devstralsmall-2505-afq8-2.uqff"`|
29
+ |F8E4M3|`./mistralrs-server -i plain -m EricB/Devstral-Small-2505-UQFF -f "devstralsmall-2505-f8e4m3-0.uqff;devstralsmall-2505-f8e4m3-1.uqff;devstralsmall-2505-f8e4m3-2.uqff"`|
30
+ |Q2K|`./mistralrs-server -i plain -m EricB/Devstral-Small-2505-UQFF -f devstralsmall-2505-q2k-0.uqff`|
31
+ |Q3K|`./mistralrs-server -i plain -m EricB/Devstral-Small-2505-UQFF -f devstralsmall-2505-q3k-0.uqff`|
32
+ |Q4K|`./mistralrs-server -i plain -m EricB/Devstral-Small-2505-UQFF -f "devstralsmall-2505-q4k-0.uqff;devstralsmall-2505-q4k-1.uqff"`|
33
+ |Q5K|`./mistralrs-server -i plain -m EricB/Devstral-Small-2505-UQFF -f "devstralsmall-2505-q5k-0.uqff;devstralsmall-2505-q5k-1.uqff"`|
34
+ |Q8_0|`./mistralrs-server -i plain -m EricB/Devstral-Small-2505-UQFF -f "devstralsmall-2505-q8_0-0.uqff;devstralsmall-2505-q8_0-1.uqff;devstralsmall-2505-q8_0-2.uqff"`|
config.json ADDED
@@ -0,0 +1,26 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "MistralForCausalLM"
4
+ ],
5
+ "attention_dropout": 0.0,
6
+ "bos_token_id": 1,
7
+ "eos_token_id": 2,
8
+ "head_dim": 128,
9
+ "hidden_act": "silu",
10
+ "hidden_size": 5120,
11
+ "initializer_range": 0.02,
12
+ "intermediate_size": 32768,
13
+ "max_position_embeddings": 131072,
14
+ "model_type": "mistral",
15
+ "num_attention_heads": 32,
16
+ "num_hidden_layers": 40,
17
+ "num_key_value_heads": 8,
18
+ "rms_norm_eps": 1e-05,
19
+ "rope_theta": 1000000000.0,
20
+ "sliding_window": null,
21
+ "tie_word_embeddings": false,
22
+ "torch_dtype": "bfloat16",
23
+ "transformers_version": "4.51.3",
24
+ "use_cache": true,
25
+ "vocab_size": 131072
26
+ }
devstralsmall-2505-afq2-0.uqff ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d735e70d5d3899e9308bfe6f6982fec31f099ef8f2952535457bb670b8b6740d
3
+ size 7156572172
devstralsmall-2505-afq3-0.uqff ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6cd1b2840aea702bf8237739d928128b91368ad2c0993cde1e919e2a356faa00
3
+ size 10019184676
devstralsmall-2505-afq4-0.uqff ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:722add4105d994f8fcb61c9ceea6f76edcb9ced4946bcfb5b13b1519ade6ae2a
3
+ size 10723034840
devstralsmall-2505-afq4-1.uqff ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0e4d2015a50d68471c7a9f2ec9795396faed966b5f236008b0daf3ed67900575
3
+ size 2158762404
devstralsmall-2505-afq6-0.uqff ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:fc418c0a92e11fcf0e61025611f3cabb2391189be4d75e3e0aa6bc2b16b011e2
3
+ size 10658143120
devstralsmall-2505-afq6-1.uqff ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:353de980fa8f998356127210d1e75f08f0a50562490e5081ec300c82867c4d61
3
+ size 7948879220
devstralsmall-2505-afq8-0.uqff ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c23660c25c577787bea511befac5b00ea5bf3871eed76e9589506004b2e0792a
3
+ size 10572940276
devstralsmall-2505-afq8-1.uqff ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7a4aec6b68198c6aa4670ace0284c1fc7df00b55d755af60d927d97bfc66fe3b
3
+ size 10628647016
devstralsmall-2505-afq8-2.uqff ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0941df2e70f42cb6382b26c71200b5e56c4a4dcbe857fc0bf2f6a3835cf61923
3
+ size 3130660000
devstralsmall-2505-f8e4m3-0.uqff ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e130a00d349e921b6af17dd99d91dd48f4e11f5e5b0916a0e2f6327c33e0324c
3
+ size 10726948246
devstralsmall-2505-f8e4m3-1.uqff ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c68685bd133e914951d9669c64d298353e8c0068e8fbd868f575ee3e1b4ddb05
3
+ size 10726948716
devstralsmall-2505-f8e4m3-2.uqff ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ee5b172c3e78715f57764152a47fb980b3cf5abcb52ac64537c254352260941e
3
+ size 1447036800
devstralsmall-2505-q2k-0.uqff ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:60fc96007449b2dfccaca73f2ddffd9b3f035884f3690c41850d1da9a4b9cbd7
3
+ size 7514386938
devstralsmall-2505-q3k-0.uqff ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c10d89277c366387b9a8782e00198857c47c3a83b6d3fc8dce554ce7fc29693c
3
+ size 9840259594
devstralsmall-2505-q4k-0.uqff ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:fb64c3592d67315119817af1b79304a4311814ac9048ca0973ac0609b6fce5a7
3
+ size 10723024928
devstralsmall-2505-q4k-1.uqff ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:bf59f10e7dfda266877e16d2e248d23c682c5aa5d9c99d136d16dbd508179299
3
+ size 2158760514
devstralsmall-2505-q5k-0.uqff ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:99c0b518276dd4af48158e7bd1c68503d020a95cc06aa5d57538105fa04ae412
3
+ size 10662071602
devstralsmall-2505-q5k-1.uqff ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:486d905329cfe78c2a57b735a04f57a85ec19f1addb70fe6d5c124f8ffefb2af
3
+ size 5082326440
devstralsmall-2505-q8_0-0.uqff ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7847a1c5499b5ffd72b800597597fe8b91482d342522f3fe6e2429d6a2bededa
3
+ size 10572935278
devstralsmall-2505-q8_0-1.uqff ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:76189fcc3d670cc90dc8ee3ed5959446d14eb4de90ebd978ab8a59fb969ecee3
3
+ size 10628641724
devstralsmall-2505-q8_0-2.uqff ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:314bc0fef18a622de3fe62202f4c7876add6c6438988b8b5f85d4b49c70bad61
3
+ size 3130658488
generation_config.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "bos_token_id": 1,
4
+ "eos_token_id": 2,
5
+ "transformers_version": "4.51.3"
6
+ }
residual.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:851804055704d0230e23c260cff884dbd4c107acf45114de6c2fc959c3c58c13
3
+ size 1343016224
tokenizer.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b76085f9923309d873994d444989f7eb6ec074b06f25b58f1e8d7b7741070949
3
+ size 17078037
tokenizer_config.json ADDED
The diff for this file is too large to render. See raw diff