Update README.md

README.md

@@ -1,9 +1,87 @@
 ---
 license: apache-2.0
+base_model:
+- cognitivecomputations/dolphin-2.2-70b
+- WizardLM/WizardMath-70B-V1.0
+- migtissera/SynthIA-70B-v1.2b
+- epfl-llm/meditron-70b
+tags:
+- mergekit
+- merge
 ---
 
 ![image/png](https://cdn-uploads.huggingface.co/production/uploads/63111b2d88942700629f5771/9OI19I3DhuPp_i8Uhp6ss.png)
 
+This is a merge of pre-trained language models created using [mergekit](https://github.com/cg123/mergekit).
+
+## Merge Details
+### Merge Method
+
+This model was merged using the [linear](https://arxiv.org/abs/2203.05482) merge method.
+
+### Models Merged
+
+The following models were included in the merge:
+* [cognitivecomputations/dolphin-2.2-70b](https://huggingface.co/cognitivecomputations/dolphin-2.2-70b)
+* [WizardLM/WizardMath-70B-V1.0](https://huggingface.co/WizardLM/WizardMath-70B-V1.0)
+* [migtissera/SynthIA-70B-v1.2b](https://huggingface.co/migtissera/SynthIA-70B-v1.2b)
+* [epfl-llm/meditron-70b](https://huggingface.co/epfl-llm/meditron-70b)
+
+### Configuration
+
+The following YAML configuration was used to produce this model:
+
+```yaml
+merge_method: linear # use linear so we can include multiple models, albeit at a zero weight
+parameters:
+  weight: 1.0 # weight everything as 1 unless specified otherwise - linear with one model weighted at 1 is a no-op like passthrough
+slices:
+  - sources:
+      - model: cognitivecomputations/dolphin-2.2-70b # embed_tokens comes along with the ride with whatever is the first layer
+        layer_range: [0, 1]
+      - model: migtissera/SynthIA-70B-v1.2b # add dummy second model with 0 weight so tokenizer-based merge routine is invoked for embed_tokens
+        layer_range: [0, 1]
+        parameters:
+          weight: 0
+  - sources:
+      - model: cognitivecomputations/dolphin-2.2-70b
+        layer_range: [1, 20]
+  - sources:
+      - model: migtissera/SynthIA-70B-v1.2b
+        layer_range: [10, 30]
+  - sources:
+      - model: WizardLM/WizardMath-70B-V1.0
+        layer_range: [20, 40]
+  - sources:
+      - model: epfl-llm/meditron-70b
+        layer_range: [25, 45]
+  - sources:
+      - model: cognitivecomputations/dolphin-2.2-70b
+        layer_range: [30, 50]
+  - sources:
+      - model: migtissera/SynthIA-70B-v1.2b
+        layer_range: [40, 60]
+  - sources:
+      - model: WizardLM/WizardMath-70B-V1.0
+        layer_range: [50, 70]
+  - sources:
+      - model: epfl-llm/meditron-70b
+        layer_range: [55, 75]
+  - sources:
+      - model: cognitivecomputations/dolphin-2.2-70b
+        layer_range: [60, 79]
+  - sources: # same as above, but for lm_head with the last layer
+      - model: cognitivecomputations/dolphin-2.2-70b
+        layer_range: [79, 80]
+      - model: migtissera/SynthIA-70B-v1.2b
+        layer_range: [79, 80]
+        parameters:
+          weight: 0
+dtype: float16
+tokenizer_source: model:cognitivecomputations/dolphin-2.2-70b # keep exact tokenizer used by dolphin - or you could use `union` if you add all of the input models to the first/last slice, but they would need to be non-zero weight or you'll get NaNs in your embeddings
+```
+
+
 # Example Output
 
 > Please invent a new idea in the area of mathematics, that combines two or more papers into a new idea that has not yet been published to your knowledge