|
|
--- |
|
|
license: apache-2.0 |
|
|
--- |
|
|
EDIT: Works pretty well for a model with no finetuning, has promise. Better and lighter than the 14b. |
|
|
|
|
|
A 13b Mistral base model, based on the NeverSleep recipe. We've had second Mistral, why not third Mistral? |
|
|
|
|
|
### Recipe |
|
|
|
|
|
slices |
|
|
|
|
|
- sources: |
|
|
- |
|
|
- model: mistralai/Mistral-7B-v0.1 |
|
|
- |
|
|
layer_range: [0, 24] |
|
|
|
|
|
- sources: |
|
|
- |
|
|
- model: mistralai/Mistral-7B-v0.1 |
|
|
- |
|
|
layer_range: [12, 24] |
|
|
|
|
|
- sources: |
|
|
- |
|
|
- model: mistralai/Mistral-7B-v0.1 |
|
|
- |
|
|
layer_range: [8, 32] |
|
|
|
|
|
merge_method: passthrough |
|
|
|
|
|
dtype: bfloat16 |
