Update README.md

README.md

@@ -6,7 +6,7 @@ language:
 pipeline_tag: text-generation
 ---
 
-# SparseLlama-2-7b-evol-codealpaca-v1-pruned_50.2of4
+# SparseLlama-2-7b-evolcodealpaca-pruned_50.2of4
 
 ## Model Overview
 - **Model Architecture:** Llama-2
@@ -19,12 +19,11 @@ pipeline_tag: text-generation
 - **Model Developers:** Neural Magic
 
 Compressed version of [Llama-2-7b](https://huggingface.co/meta-llama/Llama-2-7b-hf) specialized for code-generation.
-This model was obtained by fine-tuning the Sparse Foundational model [Sparse-Llama-2-7b-pruned_50.2of4](https://huggingface.co/nm-testing/SparseLlama-2-7b-pruned_50.2of4) on the [ultrachat_200k](https://huggingface.co/datasets/HuggingFaceH4/ultrachat_200k) dataset.
-It achieves a win rate of 62.1% on the [AlpacaEval](https://github.com/tatsu-lab/alpaca_eval) benchmark (version 1.0) when using [Llama-2-70b-chat](https://huggingface.co/meta-llama/Llama-2-70b-chat-hf) as evaluator, whereas the dense [Llama-2-7b-ultrachat200k](https://huggingface.co/neuralmagic/Llama-2-7b-ultrachat200k) model achieves 57.6% win rate.
+This model was obtained by fine-tuning the Sparse Foundational model [SparseLlama-2-7b-pruned_50.2of4](https://huggingface.co/nm-testing/SparseLlama-2-7b-pruned_50.2of4) on the [evol-codealpaca-v1](https://huggingface.co/datasets/theblackcat102/evol-codealpaca-v1) dataset.
+[SquareHead](https://arxiv.org/abs/2310.06927) knowledge distillation is used with [Llama-2-7b-evolcodealpaca](https://huggingface.co/neuralmagic/Llama-2-7b-evolcodealpaca) as teacher.
+It achieves [HumanEval](https://arxiv.org/abs/2107.03374) pass@1 of 34.58%, whereas the dense [Llama-2-7b-evolcodealpaca](https://huggingface.co/neuralmagic/Llama-2-7b-evolcodealpaca) model achieves 32.03%.
 
-This model was produced as part if Neural Magic's Sparse Foundational Models initiative, and demostrates the capability of Sparse Foundational Models to transfer to the text-generation domain.
-
-**Note:** This model uses the chat template from [zephyr-7b-beta](https://huggingface.co/HuggingFaceH4/zephyr-7b-beta).
+This model was produced as part if Neural Magic's Sparse Foundational Models initiative, and demostrates the capability of Sparse Foundational Models to transfer to the code-generation domain.
 
 ## Model Optimizations
 
@@ -33,12 +32,11 @@ This optimization reduces the number of parameters by 50%, reducing the disk siz
 
 ## Evaluation
 
-This model was evaluated in the [AlpacaEval](https://github.com/tatsu-lab/alpaca_eval) benchmark using [Llama-2-70b-chat](https://huggingface.co/meta-llama/Llama-2-70b-chat-hf) as evaluator.
+This model was evaluated in the [HumanEval](https://arxiv.org/abs/2107.03374) benchmark using the [bigcode-evaluation-harness](https://github.com/bigcode-project/bigcode-evaluation-harness).
 
 ## Accuracy
 
-| Model | Win rate | Recovery |
+| Model | HumanEval pass@1 | Recovery |
 | :----- | :--------: | :--------: |
-| [Llama-2-7b](https://huggingface.co/meta-llama/Llama-2-7b-hf) | 3.7% | -- |
-| [Llama-2-7b-ultrachat200k](https://huggingface.co/neuralmagic/Llama-2-7b-ultrachat200k) | 57.6% | -- |
-| SparseLlama-2-7b-ultrachat_200k-pruned_50.2of4 | 62.1% | 108% |
+| [Llama-2-7b-evolcodealpaca](https://huggingface.co/neuralmagic/Llama-2-7b-evolcodealpaca) | 32.03% | -- |
+| SparseLlama-2-7b-evolcodealpaca-pruned_50.2of4 | 34.58% | 108% |