---
license: mit
base_model: OpenGVLab/InternVL3_5-38B-Flash
tags:
- fp8
- quantization
- llm-compressor
- vllm
- vision
- multimodal
pipeline_tag: image-text-to-text
library_name: transformers
---

# InternVL3_5-38B-Flash-FP8

This repository contains an **FP8 quantized version** of the multi-modal model [OpenGVLab/InternVL3_5-38B-Flash](https://huggingface.co/OpenGVLab/InternVL3_5-38B-Flash).

## Model Details
* **Base Model:** `OpenGVLab/InternVL3_5-38B-Flash`
* **Quantization Format:** FP8 (E4M3) targeting `Linear` layers
* **Quantization Engine:** `llmcompressor` (Post-Training Quantization / PTQ)
* **Calibration Dataset:** 512 samples from `ultrachat-200k` (`train_sft`)
* **License:** MIT

## Method & Creation Process

The model was quantized using Neural Magic's [`llm-compressor`](https://github.com/vllm-project/llm-compressor) framework. 

Post-training quantization (PTQ) was applied directly to the language backbone (`model.language_model`). This strategy ensures that all heavy `Linear` projections in the main LLM layers are converted to FP8 for maximum speedup and reduced VRAM footprint, while preserving the vision architecture intact.

### Quantization Recipe

```python
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

recipe = QuantizationModifier(
    targets="Linear",
    scheme="FP8",
    ignore=["lm_head"]
)

oneshot(
    model=model.language_model, 
    tokenizer=tokenizer,
    dataset="ultrachat-200k",
    splits="train_sft[:512]",
    recipe=recipe,
    max_seq_length=2048,
    num_calibration_samples=512,
)