Commit ·
2348608
1
Parent(s): f4c8132
Update README.md
Browse files
README.md
CHANGED
|
@@ -85,6 +85,20 @@ there were some limitations on its performance on longer context. Motivated by i
|
|
| 85 |
- **Contact:** [GitHub issues](https://github.com/awslabs/extending-the-context-length-of-open-source-llms/issues)
|
| 86 |
- **Inference Code** [Github Repo](https://github.com/awslabs/extending-the-context-length-of-open-source-llms/blob/main/MistralLite/)
|
| 87 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 88 |
## How to Use MistralLite from Python Code (HuggingFace transformers) ##
|
| 89 |
|
| 90 |
**Important** - For an end-to-end example Jupyter notebook, please refer to [this link](https://github.com/awslabs/extending-the-context-length-of-open-source-llms/blob/main/MistralLite/huggingface-transformers/example_usage.ipynb).
|
|
|
|
| 85 |
- **Contact:** [GitHub issues](https://github.com/awslabs/extending-the-context-length-of-open-source-llms/issues)
|
| 86 |
- **Inference Code** [Github Repo](https://github.com/awslabs/extending-the-context-length-of-open-source-llms/blob/main/MistralLite/)
|
| 87 |
|
| 88 |
+
## MistralLite LM-Eval Results
|
| 89 |
+
|
| 90 |
+
### Methodology
|
| 91 |
+
|
| 92 |
+
- Please see https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard
|
| 93 |
+
- revision=4ececff
|
| 94 |
+
- Note: we used --model hf-causal-experimental instead of --model hf-causal
|
| 95 |
+
|
| 96 |
+
### Results
|
| 97 |
+
|
| 98 |
+
|Average|hellaswag| arc_challenge|truthful_qa (mc2)| MMLU (acc)|
|
| 99 |
+
|----------|-------------:|------------:|-----------:|-----------:|
|
| 100 |
+
| 0.57221 | 0.81617 | 0.58874 | 0.38275 | 0.5012 |
|
| 101 |
+
|
| 102 |
## How to Use MistralLite from Python Code (HuggingFace transformers) ##
|
| 103 |
|
| 104 |
**Important** - For an end-to-end example Jupyter notebook, please refer to [this link](https://github.com/awslabs/extending-the-context-length-of-open-source-llms/blob/main/MistralLite/huggingface-transformers/example_usage.ipynb).
|