| | --- |
| | tags: |
| | - fp8 |
| | - vllm |
| | license: llama3 |
| | license_link: https://llama.meta.com/llama3/license/ |
| | language: |
| | - en |
| | --- |
| | |
| |
|
| |
|
| | # Meta-Llama-3-8B-Instruct-FP8 |
| |
|
| | ## Model Overview |
| | - **Model Architecture:** Meta-Llama-3 |
| | - **Input:** Text |
| | - **Output:** Text |
| | - **Model Optimizations:** |
| | - **Weight quantization:** FP8 |
| | - **Activation quantization:** FP8 |
| | - **KV cache quantization:** FP8 |
| | - **Intended Use Cases:** Intended for commercial and research use in English. Similarly to [Meta-Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct), this models is intended for assistant-like chat. |
| | - **Out-of-scope:** Use in any manner that violates applicable laws or regulations (including trade compliance laws). Use in languages other than English. |
| | - **Release Date:** 6/8/2024 |
| | - **Version:** 1.0 |
| | - **License(s):** [Llama3](https://llama.meta.com/llama3/license/) |
| | - **Model Developers:** Neural Magic |
| |
|
| | Quantized version of [Meta-Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct). |
| |
|
| |
|
| | ``` |
| | lm_eval --model vllm --model_args pretrained=nm-testing/Meta-Llama-3-8B-Instruct-FP8-K-V,kv_cache_dtype=fp8,add_bos_token=True --tasks gsm8k --num_fewshot 5 --batch_size auto |
| | |
| | vllm (pretrained=nm-testing/Meta-Llama-3-8B-Instruct-FP8-K-V,kv_cache_dtype=fp8,add_bos_token=True), gen_kwargs: (None), limit: None, num_fewshot: 5, batch_size: auto |
| | |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |
| | |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |
| | |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.7748|± |0.0115| |
| | | | |strict-match | 5|exact_match|↑ |0.7763|± |0.0115| |
| | ``` |
