| | --- |
| | language: |
| | - en |
| | license: apache-2.0 |
| | tags: |
| | - mistral-7b |
| | - instruct |
| | - finetune |
| | - synthetic data |
| | - distillation |
| | base_model: mistralai/Mistral-7B-v0.1 |
| | model-index: |
| | - name: Mistral-Syndicate-7B |
| | results: [] |
| | --- |
| | |
| | ## Mistral-Syndicate-7B |
| |
|
| |
|
| | <img src="https://cdn-uploads.huggingface.co/production/uploads/64712903631bb7660aaa7183/yMzb5tbgygtlHnoyR37YO.png" width="600px" /> |
| |
|
| | ## Model Description: |
| |
|
| | Mistral Syndicate is in no way a state-of-the-art model, rather it is a fine-tuning experiment to explore |
| | the training dynamics specific to large language models. The dataset used in finetuning was generated via |
| | a "syndicate" of other open language models both of similar parameter size and larger. Each model would generate a |
| | response for a given instruction, and the group would vote on which model's response was best. |
| |
|
| | The instruction inputs used for the output label synthesis were a curated subset of [VMWare/open-instruct](https://huggingface.co/datasets/VMware/open-instruct) |
| | with additional instructions synthesized from scratch. |
| |
|
| | ## Prompt template |
| |
|
| | With context |
| | ``` |
| | Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request. |
| | |
| | ### Instruction: |
| | |
| | ### Input: |
| | |
| | ### Response: |
| | ``` |
| |
|
| | Without context |
| | ``` |
| | Below is an instruction that describes a task. Write a response that appropriately completes the request. |
| | |
| | ### Instruction: |
| | |
| | ### Response: |
| | ``` |
| |
|
| | ## Evaluation Results |
| | _12.30.23_ |
| | | Benchmark | Result | |
| | |------------|--------| |
| | | ARC | 60.84 | |
| | | HellaSwag | 82.91 | |
| | | MMLU | 60.83 | |
| | | TruthfulQA | 43.71 | |
| | | Winogrande | 78.61 | |
| | | GSM8K | 44.50 | |
| |
|
| |
|
| | # [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) |
| | Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_spmurrayzzz__Mistral-Syndicate-7B) |
| |
|
| | | Metric |Value| |
| | |---------------------------------|----:| |
| | |Avg. |61.90| |
| | |AI2 Reasoning Challenge (25-Shot)|60.84| |
| | |HellaSwag (10-Shot) |82.91| |
| | |MMLU (5-Shot) |60.83| |
| | |TruthfulQA (0-shot) |43.71| |
| | |Winogrande (5-shot) |78.61| |
| | |GSM8k (5-shot) |44.50| |
| |
|
| |
|
| | # [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard) |
| | Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_spmurrayzzz__Mistral-Syndicate-7B) |
| |
|
| | | Metric |Value| |
| | |-------------------|----:| |
| | |Avg. |13.85| |
| | |IFEval (0-Shot) |24.96| |
| | |BBH (3-Shot) |20.51| |
| | |MATH Lvl 5 (4-Shot)| 2.42| |
| | |GPQA (0-shot) | 3.47| |
| | |MuSR (0-shot) |13.62| |
| | |MMLU-PRO (5-shot) |18.13| |
| |
|
| |
|
