|
|
--- |
|
|
license: apache-2.0 |
|
|
tags: |
|
|
- biomedical |
|
|
- medical |
|
|
- mistral |
|
|
- fp8 |
|
|
- quantization |
|
|
- vllm |
|
|
- text-generation |
|
|
library_name: transformers |
|
|
--- |
|
|
|
|
|
# BioMistral-7B-FP8-Dynamic |
|
|
|
|
|
## Overview |
|
|
**BioMistral-7B-FP8-Dynamic** is an **FP8 Dynamic–quantized** version of the **BioMistral-7B** model, designed for high-performance inference while maintaining strong quality on biomedical and medical NLP tasks. |
|
|
|
|
|
This model is primarily intended for deployment with **vLLM** on modern GPUs (Hopper / Ada architectures). |
|
|
|
|
|
--- |
|
|
|
|
|
## Base Model |
|
|
- **Base model**: BioMistral-7B |
|
|
- **Architecture**: Mistral-style decoder-only Transformer |
|
|
- **Domain**: Biomedical / Medical Natural Language Processing |
|
|
|
|
|
--- |
|
|
|
|
|
## Quantization |
|
|
- **Method**: FP8 Dynamic |
|
|
- **Scope**: Linear layers |
|
|
- **Objective**: Reduce VRAM usage and improve inference throughput |
|
|
|
|
|
### Notes |
|
|
- The weights are **already quantized**. |
|
|
- Do **not** apply additional runtime quantization. |
|
|
|
|
|
--- |
|
|
|
|
|
## Intended Use |
|
|
- Biomedical and medical text generation |
|
|
- Medical writing assistance |
|
|
- Summarization and analysis of scientific literature |
|
|
- Medical RAG pipelines (clinical notes, research papers) |
|
|
|
|
|
--- |
|
|
|
|
|
## Deployment (vLLM) |
|
|
|
|
|
### Recommended |
|
|
```bash |
|
|
vllm serve ig1/BioMistral-7B-FP8-Dynamic \ |
|
|
--served-model-name biomistral-7b-fp8 \ |
|
|
--dtype auto |
