Instructions to use ucsbcit/InternVL3_5-38B-Flash-FP8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ucsbcit/InternVL3_5-38B-Flash-FP8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="ucsbcit/InternVL3_5-38B-Flash-FP8")

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("ucsbcit/InternVL3_5-38B-Flash-FP8", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ucsbcit/InternVL3_5-38B-Flash-FP8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ucsbcit/InternVL3_5-38B-Flash-FP8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ucsbcit/InternVL3_5-38B-Flash-FP8",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/ucsbcit/InternVL3_5-38B-Flash-FP8

SGLang

How to use ucsbcit/InternVL3_5-38B-Flash-FP8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ucsbcit/InternVL3_5-38B-Flash-FP8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ucsbcit/InternVL3_5-38B-Flash-FP8",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ucsbcit/InternVL3_5-38B-Flash-FP8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ucsbcit/InternVL3_5-38B-Flash-FP8",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use ucsbcit/InternVL3_5-38B-Flash-FP8 with Docker Model Runner:
```
docker model run hf.co/ucsbcit/InternVL3_5-38B-Flash-FP8
```

InternVL3_5-38B-Flash-FP8

This repository contains an FP8 quantized version of the multi-modal model OpenGVLab/InternVL3_5-38B-Flash.

Model Details

Base Model: OpenGVLab/InternVL3_5-38B-Flash
Quantization Format: FP8 (E4M3) targeting Linear layers
Quantization Engine: llmcompressor (Post-Training Quantization / PTQ)
Calibration Dataset: 512 samples from ultrachat-200k (train_sft)
License: MIT

Method & Creation Process

The model was quantized using Neural Magic's llm-compressor framework.

Post-training quantization (PTQ) was applied directly to the language backbone (model.language_model). This strategy ensures that all heavy Linear projections in the main LLM layers are converted to FP8 for maximum speedup and reduced VRAM footprint, while preserving the vision architecture intact.

Quantization Recipe

from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

recipe = QuantizationModifier(
    targets="Linear",
    scheme="FP8",
    ignore=["lm_head"]
)

oneshot(
    model=model.language_model, 
    tokenizer=tokenizer,
    dataset="ultrachat-200k",
    splits="train_sft[:512]",
    recipe=recipe,
    max_seq_length=2048,
    num_calibration_samples=512,
)

Downloads last month: -; Downloads are not tracked for this model. How to track

Inference Providers NEW

Image-Text-to-Text

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for ucsbcit/InternVL3_5-38B-Flash-FP8

Base model

OpenGVLab/InternVL3_5-38B-Pretrained

Finetuned

OpenGVLab/InternVL3_5-38B-Instruct

Finetuned

OpenGVLab/InternVL3_5-38B-MPO

Finetuned

OpenGVLab/InternVL3_5-38B

Finetuned

OpenGVLab/InternVL3_5-38B-Flash

Finetuned

(1)

this model