File size: 1,607 Bytes
113a2ee | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 | ---
license: mit
base_model: OpenGVLab/InternVL3_5-38B-Flash
tags:
- fp8
- quantization
- llm-compressor
- vllm
- vision
- multimodal
pipeline_tag: image-text-to-text
library_name: transformers
---
# InternVL3_5-38B-Flash-FP8
This repository contains an **FP8 quantized version** of the multi-modal model [OpenGVLab/InternVL3_5-38B-Flash](https://huggingface.co/OpenGVLab/InternVL3_5-38B-Flash).
## Model Details
* **Base Model:** `OpenGVLab/InternVL3_5-38B-Flash`
* **Quantization Format:** FP8 (E4M3) targeting `Linear` layers
* **Quantization Engine:** `llmcompressor` (Post-Training Quantization / PTQ)
* **Calibration Dataset:** 512 samples from `ultrachat-200k` (`train_sft`)
* **License:** MIT
## Method & Creation Process
The model was quantized using Neural Magic's [`llm-compressor`](https://github.com/vllm-project/llm-compressor) framework.
Post-training quantization (PTQ) was applied directly to the language backbone (`model.language_model`). This strategy ensures that all heavy `Linear` projections in the main LLM layers are converted to FP8 for maximum speedup and reduced VRAM footprint, while preserving the vision architecture intact.
### Quantization Recipe
```python
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier
recipe = QuantizationModifier(
targets="Linear",
scheme="FP8",
ignore=["lm_head"]
)
oneshot(
model=model.language_model,
tokenizer=tokenizer,
dataset="ultrachat-200k",
splits="train_sft[:512]",
recipe=recipe,
max_seq_length=2048,
num_calibration_samples=512,
) |