Upload Qwen3-0.6B TensorRT-LLM NVFP4 checkpoint

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,189 @@
+---
+license: apache-2.0
+base_model: Qwen/Qwen3-0.6B
+pipeline_tag: text-generation
+library_name: tensorrt-llm
+tags:
+- qwen3
+- qwen
+- tensorrt-llm
+- text-generation
+- nvfp4
+- quantized
+- checkpoint
+---
+
+# Qwen3-0.6B TensorRT-LLM Checkpoint (NVFP4)
+
+This repository contains a community-converted TensorRT-LLM checkpoint for [`Qwen/Qwen3-0.6B`](https://huggingface.co/Qwen/Qwen3-0.6B).
+
+It is a TensorRT-LLM **checkpoint-format** repository, not a prebuilt engine. The intent is to let you download the checkpoint from Hugging Face and build an engine locally for your own GPU and TensorRT-LLM version.
+
+## Who This Repo Is For
+
+This repository is for users who already work with TensorRT-LLM and want a ready-made **TensorRT-LLM checkpoint** that they can turn into a local engine for their own GPU.
+
+It is **not**:
+- a prebuilt TensorRT engine
+- a plain Transformers checkpoint
+- an Ollama model
+- a one-click chat model that can be run directly after download
+
+## How to Use
+
+1. Download this repository from Hugging Face.
+2. Build a local engine with `trtllm-build` for your own GPU and TensorRT-LLM version.
+3. Run inference with the engine you built.
+
+The `Build Example` section below shows the validated local command used for the benchmark snapshot in this README.
+
+## Model Characteristics
+
+- Base model: `Qwen/Qwen3-0.6B`
+- License: `apache-2.0`
+- Architecture: `Qwen3ForCausalLM`
+- Upstream maximum context length (`max_position_embeddings`): `40960`
+- Hidden size: `1024`
+- Intermediate size: `3072`
+- Layers: `28`
+- Attention heads: `16`
+- KV heads: `8`
+- Vocabulary size: `151936`
+
+These values come from the upstream model/checkpoint configuration. They describe the model family itself, not a specific locally built TensorRT engine.
+
+## Checkpoint Details
+
+- TensorRT-LLM version used for conversion: `1.2.0rc6`
+- Checkpoint dtype: `bfloat16`
+- Quantization: `NVFP4` (weights)
+- KV cache quantization: `FP8`
+- Calibration dataset: `cnn_dailymail` (64 samples, max seq length 256)
+- Tensor parallel size: `1`
+- Checkpoint files:
+  - `config.json`
+  - `rank0.safetensors`
+  - tokenizer and generation files copied from the upstream Hugging Face model
+
+## Files
+
+- `config.json`: TensorRT-LLM checkpoint config
+- `rank0.safetensors`: TensorRT-LLM checkpoint weights
+- `generation_config.json`: upstream generation config
+- `tokenizer.json`: upstream tokenizer
+- `tokenizer_config.json`: upstream tokenizer config
+- `merges.txt`: upstream merges file
+- `vocab.json`: upstream vocabulary
+
+## Build Example
+
+The following command is the **validated local engine build** used for the benchmarks in this README. These values are build-time/runtime settings for one local engine, not limits of the checkpoint itself.
+
+Build an engine locally with TensorRT-LLM:
+
+```bash
+huggingface-cli download Shoolife/Qwen3-0.6B-TensorRT-LLM-Checkpoint-NVFP4 --local-dir ./checkpoint
+
+trtllm-build \
+  --checkpoint_dir ./checkpoint \
+  --output_dir ./engine \
+  --gemm_plugin auto \
+  --gpt_attention_plugin auto \
+  --max_batch_size 1 \
+  --max_input_len 512 \
+  --max_seq_len 1024 \
+  --max_num_tokens 256 \
+  --workers 1 \
+  --monitor_memory
+```
+
+If you rebuild the engine with different limits, memory usage and supported request shapes will change accordingly.
+
+## Quantization
+
+This checkpoint was produced using TensorRT-LLM quantization tooling:
+
+```bash
+python quantize.py \
+  --model_dir ./Qwen3-0.6B \
+  --output_dir ./checkpoint_nvfp4 \
+  --dtype bfloat16 \
+  --qformat nvfp4 \
+  --kv_cache_dtype fp8 \
+  --calib_dataset cnn_dailymail \
+  --calib_size 64 \
+  --batch_size 1 \
+  --calib_max_seq_length 256 \
+  --tokenizer_max_seq_length 2048
+```
+
+## Validation
+
+The checkpoint was validated by building a local engine and running inference on:
+
+- GPU: `NVIDIA GeForce RTX 5070 Laptop GPU`
+- Runtime: `TensorRT-LLM 1.2.0rc6`
+
+## Validated Local Engine Characteristics
+
+Local build and runtime characteristics from the validated engine used for the benchmark snapshot below:
+
+| Property | Value |
+|---|---|
+| Checkpoint size | `830 MB` |
+| Built engine size | `567 MB` |
+| Tested GPU | `NVIDIA GeForce RTX 5070 Laptop GPU` |
+| GPU memory reported by benchmark host | `7.53 GiB` |
+| Engine build `max_batch_size` | `1` |
+| Engine build `max_input_len` | `512` |
+| Engine build `max_seq_len` | `1024` |
+| Engine build `max_num_tokens` | `256` |
+
+Important: the `1024` / `256` limits above belong only to this particular local engine build. They are not the intrinsic maximum context or generation limits of `Qwen3-0.6B` itself.
+
+These values are specific to the local engine build used for validation and will change if you rebuild with different TensorRT-LLM settings and memory budgets.
+
+## Benchmark Snapshot
+
+Local single-GPU measurements from the validated local engine on `RTX 5070 Laptop GPU`, using TensorRT-LLM synthetic fixed-length requests, `20` requests per profile, `2` warmup requests, and `concurrency=1`.
+
+| Profile | Input | Output | TTFT | TPOT | Output tok/s | Avg latency |
+|---|---:|---:|---:|---:|---:|---:|
+| `tiny_16_32` | 16 | 32 | `7.48 ms` | `3.14 ms` | `305.31` | `104.79 ms` |
+| `short_chat_42_64` | 42 | 64 | `7.34 ms` | `3.16 ms` | `310.09` | `206.37 ms` |
+| `balanced_128_128` | 128 | 128 | `8.02 ms` | `3.18 ms` | `311.15` | `411.35 ms` |
+| `long_prompt_192_64` | 192 | 64 | `8.03 ms` | `3.17 ms` | `307.91` | `207.83 ms` |
+| `long_generation_42_192` | 42 | 192 | `8.00 ms` | `3.17 ms` | `312.89` | `613.60 ms` |
+
+These numbers are local measurements from one machine and should be treated as reference values, not portability guarantees.
+
+## Quick Parity Check
+
+A quick parity check was run on `ARC-Challenge` (20 examples) and `OpenBookQA` (20 examples) to verify that the TensorRT-LLM NVFP4 engine produces comparable answers to the upstream Hugging Face model.
+
+| Benchmark | HF Accuracy | TRT Accuracy | Agreement |
+|---|---:|---:|---:|
+| `arc_challenge` | `0.55` | `0.40` | `0.45` |
+| `openbookqa` | `0.65` | `0.50` | `0.55` |
+| **Overall** | **`0.60`** | **`0.45`** | **`0.50`** |
+
+The NVFP4 quantized engine shows a visible quality drop on this small subset. This is expected for aggressive 4-bit quantization on a 0.6B parameter model.
+
+## Local Comparison
+
+The table below compares locally validated TensorRT-LLM variants built for the same GPU family and the same local engine limits (`max_batch_size=1`, `max_seq_len=1024`, `max_num_tokens=256`).
+
+| Variant | Checkpoint | Engine | `short_chat_42_64` | `balanced_128_128` | `long_generation_42_192` | Quick-check overall | Quick-check change vs BF16 | Practical reading |
+|---|---:|---:|---:|---:|---:|---:|---|---|
+| `BF16` | `1.5 GB` | `1.5 GB` | `239.49 tok/s` | `238.27 tok/s` | `239.96 tok/s` | `0.60` | `baseline` | Native precision, best numerical stability |
+| `FP16` | `1.5 GB` | `1.5 GB` | `239.53 tok/s` | `238.39 tok/s` | `239.94 tok/s` | `0.60` | `same` | Equivalent precision, identical results |
+| `FP8` | `1014 MB` | `1.1 GB` | `327.84 tok/s` | `329.16 tok/s` | `330.29 tok/s` | `0.575` | `-2.5 pts on this quick-check` | ~37% faster, minor quality variance |
+| `NVFP4` | `830 MB` | `567 MB` | `310.09 tok/s` | `311.15 tok/s` | `312.89 tok/s` | `0.45` | `-15 pts on this quick-check` | Smallest and lighter, but with visible quality drop |
+
+This comparison is intentionally local and narrow. It should not be treated as a universal benchmark across all prompts, datasets, GPUs, or TensorRT-LLM versions.
+
+## Notes
+
+- This is not an official Qwen or NVIDIA release.
+- This repository does not include a prebuilt TensorRT engine.
+- Engine compatibility and performance depend on your GPU, driver, CUDA, TensorRT, and TensorRT-LLM versions.

config.json

@@ -0,0 +1,88 @@
+{
+    "producer": {
+        "name": "modelopt",
+        "version": "0.37.0"
+    },
+    "architecture": "Qwen3ForCausalLM",
+    "dtype": "bfloat16",
+    "logits_dtype": "float16",
+    "num_hidden_layers": 28,
+    "num_attention_heads": 16,
+    "num_key_value_heads": 8,
+    "hidden_size": 1024,
+    "norm_epsilon": 1e-06,
+    "vocab_size": 151936,
+    "max_position_embeddings": 40960,
+    "hidden_act": "silu",
+    "use_parallel_embedding": true,
+    "embedding_sharding_dim": 0,
+    "head_size": 128,
+    "intermediate_size": 3072,
+    "position_embedding_type": "rope_gpt_neox",
+    "share_embedding_table": false,
+    "residual_mlp": false,
+    "bias": false,
+    "rotary_pct": 1.0,
+    "rank": 0,
+    "decoder": "qwen",
+    "rmsnorm": true,
+    "lm_head_bias": false,
+    "mlp_bias": false,
+    "attn_bias": false,
+    "rotary_base": 1000000,
+    "rotary_scaling": null,
+    "disable_weight_only_quant_plugin": false,
+    "num_labels": 1,
+    "use_logn_attn": false,
+    "mlp_only_layers": [],
+    "decoder_sparse_step": 1,
+    "moe": {
+        "num_experts": 0,
+        "shared_expert_intermediate_size": 0,
+        "top_k": 0,
+        "normalization_mode": 0,
+        "sparse_mixer_epsilon": 0.01,
+        "tp_mode": 0,
+        "device_limited_n_group": 0,
+        "device_limited_topk_group": 0,
+        "device_limited_routed_scaling_factor": 1.0
+    },
+    "runtime_defaults": null,
+    "mapping": {
+        "world_size": 1,
+        "gpus_per_node": 8,
+        "cp_size": 1,
+        "tp_size": 1,
+        "pp_size": 1,
+        "moe_tp_size": 1,
+        "moe_cluster_size": 1,
+        "moe_ep_size": 1,
+        "attn_tp_size": 1,
+        "attn_cp_size": 1,
+        "cp_config": {},
+        "enable_attention_dp": false,
+        "enable_lm_head_tp_in_adp": false
+    },
+    "quantization": {
+        "quant_algo": "NVFP4",
+        "kv_cache_quant_algo": "FP8",
+        "group_size": 16,
+        "smoothquant_val": 0.5,
+        "clamp_val": null,
+        "use_meta_recipe": false,
+        "has_zero_point": false,
+        "pre_quant_scale": false,
+        "exclude_modules": [
+            "lm_head"
+        ],
+        "mamba_ssm_cache_dtype": null
+    },
+    "qk_layernorm": false,
+    "rotary_embedding_dim": 128,
+    "seq_length": 8192,
+    "qwen_type": "qwen3",
+    "moe_intermediate_size": 0,
+    "moe_shared_expert_intermediate_size": 0,
+    "tie_word_embeddings": true,
+    "model_type": "qwen"
+}

generation_config.json

@@ -0,0 +1,13 @@
+{
+    "bos_token_id": 151643,
+    "do_sample": true,
+    "eos_token_id": [
+        151645,
+        151643
+    ],
+    "pad_token_id": 151643,
+    "temperature": 0.6,
+    "top_k": 20,
+    "top_p": 0.95,
+    "transformers_version": "4.51.0"
+}

rank0.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:193dc3362b8cd62caf90f31baf8e2bac0128dea0e81515dd0e5ede3b35089ef0
+size 870277240

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:aeb13307a71acd8fe81861d94ad54ab689df773318809eed3cbe794b4492dae4
+size 11422654

tokenizer_config.json

@@ -0,0 +1,239 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151646": {
+      "content": "<|object_ref_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151647": {
+      "content": "<|object_ref_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151648": {
+      "content": "<|box_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151649": {
+      "content": "<|box_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151650": {
+      "content": "<|quad_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151651": {
+      "content": "<|quad_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151652": {
+      "content": "<|vision_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151653": {
+      "content": "<|vision_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151654": {
+      "content": "<|vision_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151655": {
+      "content": "<|image_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151656": {
+      "content": "<|video_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151657": {
+      "content": "<tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151658": {
+      "content": "</tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151659": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151660": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151661": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151662": {
+      "content": "<|fim_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151663": {
+      "content": "<|repo_name|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151664": {
+      "content": "<|file_sep|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151665": {
+      "content": "<tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151666": {
+      "content": "</tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151667": {
+      "content": "<think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151668": {
+      "content": "</think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "bos_token": null,
+  "chat_template": "{%- if tools %}\n    {{- '<|im_start|>system\\n' }}\n    {%- if messages[0].role == 'system' %}\n        {{- messages[0].content + '\\n\\n' }}\n    {%- endif %}\n    {{- \"# Tools\\n\\nYou may call one or more functions to assist with the user query.\\n\\nYou are provided with function signatures within <tools></tools> XML tags:\\n<tools>\" }}\n    {%- for tool in tools %}\n        {{- \"\\n\" }}\n        {{- tool | tojson }}\n    {%- endfor %}\n    {{- \"\\n</tools>\\n\\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\\n<tool_call>\\n{\\\"name\\\": <function-name>, \\\"arguments\\\": <args-json-object>}\\n</tool_call><|im_end|>\\n\" }}\n{%- else %}\n    {%- if messages[0].role == 'system' %}\n        {{- '<|im_start|>system\\n' + messages[0].content + '<|im_end|>\\n' }}\n    {%- endif %}\n{%- endif %}\n{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}\n{%- for message in messages[::-1] %}\n    {%- set index = (messages|length - 1) - loop.index0 %}\n    {%- if ns.multi_step_tool and message.role == \"user\" and message.content is string and not(message.content.startswith('<tool_response>') and message.content.endswith('</tool_response>')) %}\n        {%- set ns.multi_step_tool = false %}\n        {%- set ns.last_query_index = index %}\n    {%- endif %}\n{%- endfor %}\n{%- for message in messages %}\n    {%- if message.content is string %}\n        {%- set content = message.content %}\n    {%- else %}\n        {%- set content = '' %}\n    {%- endif %}\n    {%- if (message.role == \"user\") or (message.role == \"system\" and not loop.first) %}\n        {{- '<|im_start|>' + message.role + '\\n' + content + '<|im_end|>' + '\\n' }}\n    {%- elif message.role == \"assistant\" %}\n        {%- set reasoning_content = '' %}\n        {%- if message.reasoning_content is string %}\n            {%- set reasoning_content = message.reasoning_content %}\n        {%- else %}\n            {%- if '</think>' in content %}\n                {%- set reasoning_content = content.split('</think>')[0].rstrip('\\n').split('<think>')[-1].lstrip('\\n') %}\n                {%- set content = content.split('</think>')[-1].lstrip('\\n') %}\n            {%- endif %}\n        {%- endif %}\n        {%- if loop.index0 > ns.last_query_index %}\n            {%- if loop.last or (not loop.last and reasoning_content) %}\n                {{- '<|im_start|>' + message.role + '\\n<think>\\n' + reasoning_content.strip('\\n') + '\\n</think>\\n\\n' + content.lstrip('\\n') }}\n            {%- else %}\n                {{- '<|im_start|>' + message.role + '\\n' + content }}\n            {%- endif %}\n        {%- else %}\n            {{- '<|im_start|>' + message.role + '\\n' + content }}\n        {%- endif %}\n        {%- if message.tool_calls %}\n            {%- for tool_call in message.tool_calls %}\n                {%- if (loop.first and content) or (not loop.first) %}\n                    {{- '\\n' }}\n                {%- endif %}\n                {%- if tool_call.function %}\n                    {%- set tool_call = tool_call.function %}\n                {%- endif %}\n                {{- '<tool_call>\\n{\"name\": \"' }}\n                {{- tool_call.name }}\n                {{- '\", \"arguments\": ' }}\n                {%- if tool_call.arguments is string %}\n                    {{- tool_call.arguments }}\n                {%- else %}\n                    {{- tool_call.arguments | tojson }}\n                {%- endif %}\n                {{- '}\\n</tool_call>' }}\n            {%- endfor %}\n        {%- endif %}\n        {{- '<|im_end|>\\n' }}\n    {%- elif message.role == \"tool\" %}\n        {%- if loop.first or (messages[loop.index0 - 1].role != \"tool\") %}\n            {{- '<|im_start|>user' }}\n        {%- endif %}\n        {{- '\\n<tool_response>\\n' }}\n        {{- content }}\n        {{- '\\n</tool_response>' }}\n        {%- if loop.last or (messages[loop.index0 + 1].role != \"tool\") %}\n            {{- '<|im_end|>\\n' }}\n        {%- endif %}\n    {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n    {{- '<|im_start|>assistant\\n' }}\n    {%- if enable_thinking is defined and enable_thinking is false %}\n        {{- '<think>\\n\\n</think>\\n\\n' }}\n    {%- endif %}\n{%- endif %}",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "model_max_length": 131072,
+  "pad_token": "<|endoftext|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}