RichardErkhov commited on
Commit
8ca7147
·
verified ·
1 Parent(s): 1131867

uploaded readme

Browse files
Files changed (1) hide show
  1. README.md +64 -0
README.md ADDED
@@ -0,0 +1,64 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Quantization made by Richard Erkhov.
2
+
3
+ [Github](https://github.com/RichardErkhov)
4
+
5
+ [Discord](https://discord.gg/pvy7H8DZMG)
6
+
7
+ [Request more models](https://github.com/RichardErkhov/quant_request)
8
+
9
+
10
+ command-r-7-layer - bnb 4bits
11
+ - Model creator: https://huggingface.co/Citaman/
12
+ - Original model: https://huggingface.co/Citaman/command-r-7-layer/
13
+
14
+
15
+
16
+
17
+ Original model description:
18
+ ---
19
+ base_model:
20
+ - Citaman/command-r-8-layer
21
+ library_name: transformers
22
+ tags:
23
+ - mergekit
24
+ - merge
25
+
26
+ ---
27
+ # merge
28
+
29
+ This is a merge of pre-trained language models created using [mergekit](https://github.com/cg123/mergekit).
30
+
31
+ ## Merge Details
32
+ ### Merge Method
33
+
34
+ This model was merged using the SLERP merge method.
35
+
36
+ ### Models Merged
37
+
38
+ The following models were included in the merge:
39
+ * [Citaman/command-r-8-layer](https://huggingface.co/Citaman/command-r-8-layer)
40
+
41
+ ### Configuration
42
+
43
+ The following YAML configuration was used to produce this model:
44
+
45
+ ```yaml
46
+ slices:
47
+ - sources:
48
+ - model: Citaman/command-r-8-layer
49
+ layer_range: [0, 7]
50
+ - model: Citaman/command-r-8-layer
51
+ layer_range: [1, 8]
52
+ merge_method: slerp
53
+ base_model: Citaman/command-r-8-layer
54
+ parameters:
55
+ t:
56
+ - filter: self_attn
57
+ value: [0, 0.5, 0.3, 0.7, 1]
58
+ - filter: mlp
59
+ value: [1, 0.5, 0.7, 0.3, 0]
60
+ - value: 0.5
61
+ dtype: bfloat16
62
+ ```
63
+
64
+