Add files using upload-large-folder tool

README.md

@@ -0,0 +1,212 @@
+---
+license: apache-2.0
+base_model: Qwen/Qwen2.5-1.5B-Instruct
+pipeline_tag: text-generation
+library_name: tensorrt-llm
+tags:
+- qwen2
+- qwen
+- tensorrt-llm
+- text-generation
+- nvfp4
+- checkpoint
+---
+
+# Qwen2.5-1.5B-Instruct TensorRT-LLM Checkpoint (NVFP4)
+
+This repository contains a community-converted TensorRT-LLM checkpoint for [`Qwen/Qwen2.5-1.5B-Instruct`](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct).
+
+It is a TensorRT-LLM **checkpoint-format** repository, not a prebuilt engine. The intent is to let you download the checkpoint from Hugging Face and build an engine locally for your own GPU and TensorRT-LLM version.
+
+## Model Characteristics
+
+- Base model: `Qwen/Qwen2.5-1.5B-Instruct`
+- License: `apache-2.0`
+- Architecture: `Qwen2ForCausalLM`
+- Upstream maximum context length (`max_position_embeddings`): `32768`
+- Hidden size: `1536`
+- Intermediate size: `8960`
+- Layers: `28`
+- Attention heads: `12`
+- KV heads: `2`
+- Vocabulary size: `151936`
+
+These values come from the upstream model/checkpoint configuration. They describe the model family itself, not a specific locally built TensorRT engine.
+
+## Checkpoint Details
+
+- TensorRT-LLM version used for conversion: `1.2.0rc6`
+- Checkpoint dtype: `bfloat16`
+- Quantization: `NVFP4`
+- KV cache quantization: `FP8`
+- Tensor parallel size: `1`
+- Checkpoint files:
+  - `config.json`
+  - `rank0.safetensors`
+  - tokenizer and generation files copied from the upstream Hugging Face model
+
+## Files
+
+- `config.json`: TensorRT-LLM checkpoint config
+- `rank0.safetensors`: TensorRT-LLM checkpoint weights
+- `generation_config.json`: upstream generation config
+- `tokenizer.json`: upstream tokenizer
+- `tokenizer_config.json`: upstream tokenizer config
+- `merges.txt`: upstream merges file
+- `vocab.json`: upstream vocabulary
+
+## Build Example
+
+The following command is the **validated local engine build** used for the benchmarks in this README. These values are build-time/runtime settings for one local engine, not limits of the checkpoint itself.
+
+Build an engine locally with TensorRT-LLM:
+
+```bash
+huggingface-cli download Shoolife/Qwen2.5-1.5B-Instruct-TensorRT-LLM-Checkpoint-NVFP4 --local-dir ./checkpoint
+
+trtllm-build \
+  --checkpoint_dir ./checkpoint \
+  --output_dir ./engine \
+  --gemm_plugin auto \
+  --gpt_attention_plugin auto \
+  --max_batch_size 1 \
+  --max_input_len 512 \
+  --max_seq_len 1024 \
+  --max_num_tokens 256 \
+  --workers 1 \
+  --monitor_memory
+```
+
+If you rebuild the engine with different limits, memory usage and supported request shapes will change accordingly.
+
+## Conversion
+
+This checkpoint was produced from the upstream model with TensorRT-LLM NVFP4 quantization tooling:
+
+```bash
+python /app/tensorrt_llm/examples/quantization/quantize.py \
+  --model_dir ./Qwen2.5-1.5B-Instruct \
+  --output_dir ./checkpoint_nvfp4 \
+  --dtype bfloat16 \
+  --qformat nvfp4 \
+  --kv_cache_dtype fp8 \
+  --calib_dataset cnn_dailymail \
+  --calib_size 64 \
+  --batch_size 1 \
+  --calib_max_seq_length 256 \
+  --tokenizer_max_seq_length 2048 \
+  --device cpu \
+  --device_map cpu
+```
+
+Then build the engine:
+
+```bash
+trtllm-build \
+  --checkpoint_dir ./checkpoint_nvfp4 \
+  --output_dir ./engine_nvfp4 \
+  --gemm_plugin auto \
+  --gpt_attention_plugin auto \
+  --max_batch_size 1 \
+  --max_input_len 512 \
+  --max_seq_len 1024 \
+  --max_num_tokens 256 \
+  --workers 1 \
+  --monitor_memory
+```
+
+## Validation
+
+The checkpoint was validated by building a local engine and running inference on:
+
+- GPU: `NVIDIA GeForce RTX 5070 Laptop GPU`
+- Runtime: `TensorRT-LLM 1.2.0rc6`
+
+Smoke-test prompt:
+
+```text
+Explain the four basic arithmetic operations in one short sentence each.
+```
+
+Observed response:
+
+```text
+Sure, I'd be happy to explain the four basic arithmetic operations in short sentences:
+
+1. Addition: When you add two numbers together, you're combining their values to find the total amount.
+2. Subtraction: To subtract one number from another, you're finding out how much more one number has compared to the other.
+3. Multiplication: Multiplication is like adding a number to itself multiple times.
+4. Division: Division is the process of splitting a number into equal parts or groups.
+```
+
+## Validated Local Engine Characteristics
+
+Local build and runtime characteristics from the validated engine used for the benchmark snapshot below:
+
+| Property | Value |
+|---|---|
+| Checkpoint size | `1.6 GB` |
+| Built engine size | `1.2 GB` |
+| Tested GPU | `NVIDIA GeForce RTX 5070 Laptop GPU` |
+| GPU memory reported by benchmark host | `7.53 GiB` |
+| Engine build `max_batch_size` | `1` |
+| Engine build `max_input_len` | `512` |
+| Engine build `max_seq_len` | `1024` |
+| Engine build `max_num_tokens` | `256` |
+| Runtime effective max input length | `256` |
+| Engine load footprint | `~1.15 GiB` |
+| Paged KV cache allocation | `~5.1-5.2 GiB` |
+| Practical total GPU footprint on this setup | `~6.3-6.4 GiB` |
+
+Important: the `512` / `1024` / `256` limits above belong only to this particular local engine build. They are not the intrinsic maximum context or generation limits of `Qwen2.5-1.5B-Instruct` itself.
+
+The runtime effective input length became `256` on this build because TensorRT-LLM enabled packed input and context FMHA and clamped the usable prompt budget to the engine token budget.
+
+These values are specific to the local engine build used for validation and will change if you rebuild with different TensorRT-LLM settings and memory budgets.
+
+## Benchmark Snapshot
+
+Local single-GPU measurements from the validated local engine on `RTX 5070 Laptop GPU`, using TensorRT-LLM synthetic fixed-length requests, `20` requests per profile, `2` warmup requests, and `concurrency=1`.
+
+| Profile | Input | Output | TTFT | TPOT | Output tok/s | Avg latency |
+|---|---:|---:|---:|---:|---:|---:|
+| `tiny_16_32` | 16 | 32 | `7.99 ms` | `4.94 ms` | `198.48` | `161.20 ms` |
+| `short_chat_42_64` | 42 | 64 | `8.02 ms` | `4.96 ms` | `199.58` | `320.65 ms` |
+| `balanced_128_128` | 128 | 128 | `8.02 ms` | `4.97 ms` | `200.09` | `639.69 ms` |
+| `long_prompt_192_64` | 192 | 64 | `8.72 ms` | `4.97 ms` | `198.67` | `322.11 ms` |
+| `long_generation_42_192` | 42 | 192 | `8.23 ms` | `4.98 ms` | `200.28` | `958.64 ms` |
+
+These numbers are local measurements from one machine and should be treated as reference values, not portability guarantees.
+
+## Quick Parity Check
+
+A small public sanity-check was run against the upstream Hugging Face FP16 model on `20` validation examples from `ARC-Challenge` and `20` validation examples from `OpenBookQA`.
+
+| Benchmark | HF FP16 | TRT NVFP4 | Agreement |
+|---|---:|---:|---:|
+| `ARC-Challenge` | `0.65` | `0.50` | `0.70` |
+| `OpenBookQA` | `0.80` | `0.70` | `0.85` |
+| `Overall` | `0.725` | `0.60` | `0.775` |
+
+This is only a quick local parity check, not a full benchmark suite. It is intended to show the practical tradeoff of this conversion on a small public subset.
+
+## FP16 vs FP8 vs NVFP4
+
+The table below compares three locally validated TensorRT-LLM variants built for the same GPU family and the same local engine limits (`max_batch_size=1`, `max_seq_len=1024`, `max_num_tokens=256`).
+
+| Variant | Checkpoint | Engine | `short_chat_42_64` | `balanced_128_128` | `long_generation_42_192` | Quick-check overall | Agreement vs HF FP16 | Practical reading |
+|---|---:|---:|---:|---:|---:|---:|---:|---|
+| `FP16` | `3.4 GB` | `3.4 GB` | `105.48 tok/s` | `105.49 tok/s` | `105.70 tok/s` | `n/a` | `n/a` | Reference speed/size row; quick-check baseline came from upstream HF FP16 |
+| `FP8` | `2.1 GB` | `2.2 GB` | `166.72 tok/s` | `144.37 tok/s` | `151.36 tok/s` | `0.75` | `0.85` | Best balance in these local tests |
+| `NVFP4` | `1.6 GB` | `1.2 GB` | `199.58 tok/s` | `200.09 tok/s` | `200.28 tok/s` | `0.60` | `0.775` | Fastest and smallest, but with visible quality drop |
+
+This comparison is intentionally local and narrow. It should not be treated as a universal benchmark across all prompts, datasets, GPUs, or TensorRT-LLM versions.
+
+The quick-check baseline on that `40`-question subset was `0.725` for the upstream Hugging Face FP16 model.
+
+## Notes
+
+- This is not an official Qwen or NVIDIA release.
+- This repository does not include a prebuilt TensorRT engine.
+- Engine compatibility and performance depend on your GPU, driver, CUDA, TensorRT, and TensorRT-LLM versions.
+- `NVFP4` is attractive for speed and engine size on Blackwell GPUs, but this local quick-check showed a meaningful quality drop relative to `FP8`.

config.json

@@ -0,0 +1,88 @@
+{
+    "producer": {
+        "name": "modelopt",
+        "version": "0.37.0"
+    },
+    "architecture": "Qwen2ForCausalLM",
+    "dtype": "bfloat16",
+    "logits_dtype": "float16",
+    "num_hidden_layers": 28,
+    "num_attention_heads": 12,
+    "num_key_value_heads": 2,
+    "hidden_size": 1536,
+    "norm_epsilon": 1e-06,
+    "vocab_size": 151936,
+    "max_position_embeddings": 32768,
+    "hidden_act": "silu",
+    "use_parallel_embedding": true,
+    "embedding_sharding_dim": 0,
+    "head_size": 128,
+    "intermediate_size": 8960,
+    "position_embedding_type": "rope_gpt_neox",
+    "share_embedding_table": false,
+    "residual_mlp": false,
+    "bias": false,
+    "rotary_pct": 1.0,
+    "rank": 0,
+    "decoder": "qwen",
+    "rmsnorm": true,
+    "lm_head_bias": false,
+    "mlp_bias": false,
+    "attn_bias": true,
+    "rotary_base": 1000000.0,
+    "rotary_scaling": null,
+    "disable_weight_only_quant_plugin": false,
+    "num_labels": 1,
+    "use_logn_attn": false,
+    "mlp_only_layers": [],
+    "decoder_sparse_step": 1,
+    "moe": {
+        "num_experts": 0,
+        "shared_expert_intermediate_size": 0,
+        "top_k": 0,
+        "normalization_mode": 0,
+        "sparse_mixer_epsilon": 0.01,
+        "tp_mode": 0,
+        "device_limited_n_group": 0,
+        "device_limited_topk_group": 0,
+        "device_limited_routed_scaling_factor": 1.0
+    },
+    "runtime_defaults": null,
+    "mapping": {
+        "world_size": 1,
+        "gpus_per_node": 8,
+        "cp_size": 1,
+        "tp_size": 1,
+        "pp_size": 1,
+        "moe_tp_size": 1,
+        "moe_cluster_size": 1,
+        "moe_ep_size": 1,
+        "attn_tp_size": 1,
+        "attn_cp_size": 1,
+        "cp_config": {},
+        "enable_attention_dp": false,
+        "enable_lm_head_tp_in_adp": false
+    },
+    "quantization": {
+        "quant_algo": "NVFP4",
+        "kv_cache_quant_algo": "FP8",
+        "group_size": 16,
+        "smoothquant_val": 0.5,
+        "clamp_val": null,
+        "use_meta_recipe": false,
+        "has_zero_point": false,
+        "pre_quant_scale": false,
+        "exclude_modules": [
+            "lm_head"
+        ],
+        "mamba_ssm_cache_dtype": null
+    },
+    "qk_layernorm": false,
+    "rotary_embedding_dim": 128,
+    "seq_length": 8192,
+    "qwen_type": "qwen2",
+    "moe_intermediate_size": 0,
+    "moe_shared_expert_intermediate_size": 0,
+    "tie_word_embeddings": true,
+    "model_type": "qwen"
+}

generation_config.json

@@ -0,0 +1,14 @@
+{
+  "bos_token_id": 151643,
+  "pad_token_id": 151643,
+  "do_sample": true,
+  "eos_token_id": [
+    151645,
+    151643
+  ],
+  "repetition_penalty": 1.1,
+  "temperature": 0.7,
+  "top_p": 0.8,
+  "top_k": 20,
+  "transformers_version": "4.37.0"
+}

tokenizer_config.json

@@ -0,0 +1,207 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151646": {
+      "content": "<|object_ref_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151647": {
+      "content": "<|object_ref_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151648": {
+      "content": "<|box_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151649": {
+      "content": "<|box_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151650": {
+      "content": "<|quad_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151651": {
+      "content": "<|quad_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151652": {
+      "content": "<|vision_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151653": {
+      "content": "<|vision_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151654": {
+      "content": "<|vision_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151655": {
+      "content": "<|image_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151656": {
+      "content": "<|video_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151657": {
+      "content": "<tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151658": {
+      "content": "</tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151659": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151660": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151661": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151662": {
+      "content": "<|fim_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151663": {
+      "content": "<|repo_name|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151664": {
+      "content": "<|file_sep|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "bos_token": null,
+  "chat_template": "{%- if tools %}\n    {{- '<|im_start|>system\\n' }}\n    {%- if messages[0]['role'] == 'system' %}\n        {{- messages[0]['content'] }}\n    {%- else %}\n        {{- 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.' }}\n    {%- endif %}\n    {{- \"\\n\\n# Tools\\n\\nYou may call one or more functions to assist with the user query.\\n\\nYou are provided with function signatures within <tools></tools> XML tags:\\n<tools>\" }}\n    {%- for tool in tools %}\n        {{- \"\\n\" }}\n        {{- tool | tojson }}\n    {%- endfor %}\n    {{- \"\\n</tools>\\n\\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\\n<tool_call>\\n{\\\"name\\\": <function-name>, \\\"arguments\\\": <args-json-object>}\\n</tool_call><|im_end|>\\n\" }}\n{%- else %}\n    {%- if messages[0]['role'] == 'system' %}\n        {{- '<|im_start|>system\\n' + messages[0]['content'] + '<|im_end|>\\n' }}\n    {%- else %}\n        {{- '<|im_start|>system\\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\\n' }}\n    {%- endif %}\n{%- endif %}\n{%- for message in messages %}\n    {%- if (message.role == \"user\") or (message.role == \"system\" and not loop.first) or (message.role == \"assistant\" and not message.tool_calls) %}\n        {{- '<|im_start|>' + message.role + '\\n' + message.content + '<|im_end|>' + '\\n' }}\n    {%- elif message.role == \"assistant\" %}\n        {{- '<|im_start|>' + message.role }}\n        {%- if message.content %}\n            {{- '\\n' + message.content }}\n        {%- endif %}\n        {%- for tool_call in message.tool_calls %}\n            {%- if tool_call.function is defined %}\n                {%- set tool_call = tool_call.function %}\n            {%- endif %}\n            {{- '\\n<tool_call>\\n{\"name\": \"' }}\n            {{- tool_call.name }}\n            {{- '\", \"arguments\": ' }}\n            {{- tool_call.arguments | tojson }}\n            {{- '}\\n</tool_call>' }}\n        {%- endfor %}\n        {{- '<|im_end|>\\n' }}\n    {%- elif message.role == \"tool\" %}\n        {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != \"tool\") %}\n            {{- '<|im_start|>user' }}\n        {%- endif %}\n        {{- '\\n<tool_response>\\n' }}\n        {{- message.content }}\n        {{- '\\n</tool_response>' }}\n        {%- if loop.last or (messages[loop.index0 + 1].role != \"tool\") %}\n            {{- '<|im_end|>\\n' }}\n        {%- endif %}\n    {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n    {{- '<|im_start|>assistant\\n' }}\n{%- endif %}\n",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "model_max_length": 131072,
+  "pad_token": "<|endoftext|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}