Shoolife/Qwen2.5-1.5B-Instruct-TensorRT-LLM-Checkpoint-FP8

Qwen2.5-1.5B-Instruct TensorRT-LLM Checkpoint (FP8)

This repository contains a community-converted TensorRT-LLM checkpoint for Qwen/Qwen2.5-1.5B-Instruct.

It is a TensorRT-LLM checkpoint-format repository, not a prebuilt engine. The intent is to let you download the checkpoint from Hugging Face and build an engine locally for your own GPU and TensorRT-LLM version.

Who This Repo Is For

This repository is for users who already work with TensorRT-LLM and want a ready-made TensorRT-LLM checkpoint that they can turn into a local engine for their own GPU.

It is not:

• a prebuilt TensorRT engine
• a plain Transformers checkpoint
• an Ollama model
• a one-click chat model that can be run directly after download

How to Use

1. Download this repository from Hugging Face.
2. Build a local engine with trtllm-build for your own GPU and TensorRT-LLM version.
3. Run inference with the engine you built.

The Build Example section below shows the validated local command used for the benchmark snapshot in this README.

Model Characteristics

• Base model: Qwen/Qwen2.5-1.5B-Instruct
• License: apache-2.0
• Architecture: Qwen2ForCausalLM
• Upstream maximum context length (max_position_embeddings): 32768
• Hidden size: 1536
• Intermediate size: 8960
• Layers: 28
• Attention heads: 12
• KV heads: 2
• Vocabulary size: 151936

These values come from the upstream model/checkpoint configuration. They describe the model family itself, not a specific locally built TensorRT engine.

Checkpoint Details

• TensorRT-LLM version used for conversion: 1.2.0rc6
• Checkpoint dtype: bfloat16
• Quantization: FP8
• KV cache quantization: FP8
• Tensor parallel size: 1
• Checkpoint files:
  • config.json
  • rank0.safetensors
  • tokenizer and generation files copied from the upstream Hugging Face model

For this FP8 checkpoint, bfloat16 remains the base dtype of the non-quantized parts while the main TensorRT-LLM quantization recipe is FP8.

Files

• config.json: TensorRT-LLM checkpoint config
• rank0.safetensors: TensorRT-LLM checkpoint weights
• generation_config.json: upstream generation config
• tokenizer.json: upstream tokenizer
• tokenizer_config.json: upstream tokenizer config
• merges.txt: upstream merges file
• vocab.json: upstream vocabulary

Build Example

The following command is the validated local engine build used for the benchmarks in this README. These values are build-time/runtime settings for one local engine, not limits of the checkpoint itself.

Build an engine locally with TensorRT-LLM:

huggingface-cli download Shoolife/Qwen2.5-1.5B-Instruct-TensorRT-LLM-Checkpoint-FP8 --local-dir ./checkpoint

trtllm-build \
  --checkpoint_dir ./checkpoint \
  --output_dir ./engine \
  --gemm_plugin auto \
  --gpt_attention_plugin auto \
  --max_batch_size 1 \
  --max_input_len 512 \
  --max_seq_len 1024 \
  --max_num_tokens 256 \
  --workers 1 \
  --monitor_memory

If you rebuild the engine with different limits, memory usage and supported request shapes will change accordingly.

Conversion

This checkpoint was produced from the upstream model with TensorRT-LLM FP8 quantization tooling:

python /app/tensorrt_llm/examples/quantization/quantize.py \
  --model_dir ./Qwen2.5-1.5B-Instruct \
  --output_dir ./checkpoint_fp8 \
  --dtype bfloat16 \
  --qformat fp8 \
  --kv_cache_dtype fp8 \
  --calib_dataset cnn_dailymail \
  --calib_size 64 \
  --batch_size 1 \
  --calib_max_seq_length 256 \
  --tokenizer_max_seq_length 2048 \
  --device cpu \
  --device_map cpu

Then build the engine:

trtllm-build \
  --checkpoint_dir ./checkpoint_fp8 \
  --output_dir ./engine_fp8 \
  --gemm_plugin auto \
  --gpt_attention_plugin auto \
  --max_batch_size 1 \
  --max_input_len 512 \
  --max_seq_len 1024 \
  --max_num_tokens 256 \
  --workers 1 \
  --monitor_memory

Validation

The checkpoint was validated by building a local engine and running inference on:

• GPU: NVIDIA GeForce RTX 5070 Laptop GPU
• Runtime: TensorRT-LLM 1.2.0rc6

Smoke-test prompt:

Explain the four basic arithmetic operations in one short sentence each.

Observed response:

Sure! The four basic arithmetic operations are addition, subtraction, multiplication, and division. Addition combines two or more numbers to give a total. Subtraction takes away one number from another to give a difference. Multiplication is the process of adding a number to another number to give a product. Division is the process of splitting a number into equal parts to give a quotient.

Validated Local Engine Characteristics

Local build and runtime characteristics from the validated engine used for the benchmark snapshot below:

Property	Value
Checkpoint size	2.1 GB
Built engine size	2.2 GB
Tested GPU	NVIDIA GeForce RTX 5070 Laptop GPU
GPU memory reported by benchmark host	7.53 GiB
Engine build max_batch_size	1
Engine build max_input_len	512
Engine build max_seq_len	1024
Engine build max_num_tokens	256
Runtime effective max input length	256
Engine load footprint	~2.1 GiB
Paged KV cache allocation	~4.3 GiB
Practical total GPU footprint on this setup	~6.4-6.5 GiB

Important: the 512 / 1024 / 256 limits above belong only to this particular local engine build. They are not the intrinsic maximum context or generation limits of Qwen2.5-1.5B-Instruct itself.

The runtime effective input length became 256 on this build because TensorRT-LLM enabled packed input and context FMHA and clamped the usable prompt budget to the engine token budget.

These values are specific to the local engine build used for validation and will change if you rebuild with different TensorRT-LLM settings and memory budgets.

Benchmark Snapshot

Local single-GPU measurements from the validated local engine on RTX 5070 Laptop GPU, using TensorRT-LLM synthetic fixed-length requests, 20 requests per profile, 2 warmup requests, and concurrency=1.

Profile	Input	Output	TTFT	TPOT	Output tok/s	Avg latency
tiny_16_32	16	32	9.17 ms	6.65 ms	148.69	215.19 ms
short_chat_42_64	42	64	8.80 ms	5.95 ms	166.72	383.85 ms
balanced_128_128	128	128	13.71 ms	6.87 ms	144.37	886.61 ms
long_prompt_192_64	192	64	17.26 ms	6.86 ms	142.37	449.52 ms
long_generation_42_192	42	192	9.57 ms	6.59 ms	151.36	1268.46 ms

These numbers are local measurements from one machine and should be treated as reference values, not portability guarantees.

Quick Parity Check

A small public sanity-check was run against the upstream Hugging Face FP16 model on 20 validation examples from ARC-Challenge and 20 validation examples from OpenBookQA.

Benchmark	HF FP16	TRT FP8	Agreement
ARC-Challenge	0.65	0.70	0.80
OpenBookQA	0.80	0.80	0.90
Overall	0.725	0.75	0.85

This is only a quick local parity check, not a full benchmark suite. It is intended to show that the converted checkpoint behaves close to the upstream model on a small public subset.

Local Comparison

The table below compares three locally validated TensorRT-LLM variants built for the same GPU family and the same local engine limits (max_batch_size=1, max_seq_len=1024, max_num_tokens=256).

Variant	Checkpoint	Engine	short_chat_42_64	balanced_128_128	long_generation_42_192	Quick-check overall	Quick-check change vs BF16	Practical reading
BF16	3.4 GB	3.4 GB	105.27 tok/s	105.37 tok/s	105.57 tok/s	0.725	baseline	Same size and speed as FP16, better numerical stability for training
FP16	3.4 GB	3.4 GB	105.48 tok/s	105.49 tok/s	105.70 tok/s	0.75	+2.5 pts on this quick-check	Most conservative variant
FP8	2.1 GB	2.2 GB	166.72 tok/s	144.37 tok/s	151.36 tok/s	0.75	+2.5 pts on this quick-check	Best balance in these local tests
NVFP4	1.6 GB	1.2 GB	199.58 tok/s	200.09 tok/s	200.28 tok/s	0.60	-12.5 pts on this quick-check	Fastest and smallest, but with visible quality drop

This comparison is intentionally local and narrow. It should not be treated as a universal benchmark across all prompts, datasets, GPUs, or TensorRT-LLM versions.

On that same 40-question subset, the upstream Hugging Face FP16 model scored 0.725, while the local TensorRT-LLM FP16 engine scored 0.75.

Notes

• This is not an official Qwen or NVIDIA release.
• This repository does not include a prebuilt TensorRT engine.
• Engine compatibility and performance depend on your GPU, driver, CUDA, TensorRT, and TensorRT-LLM versions.