Duplicate from vincentzed-hf/Kimi-K2.5-MXFP8

.gitattributes

@@ -0,0 +1,36 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+model.safetensors.index.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,167 @@
+---
+pipeline_tag: image-text-to-text
+base_model:
+- moonshotai/Kimi-K2.5
+license: mit
+library_name: Model Optimizer
+tags:
+- nvidia
+- ModelOpt
+- KimiK25
+- quantized
+- MXFP8
+- mxfp8
+---
+
+# Model Overview
+
+## Description:
+The NVIDIA Kimi-K2.5-MXFP8 model is a quantized version of Moonshot AI's Kimi-K2.5 model, a native multimodal agentic model with Mixture of Experts (MoE) architecture. Kimi-K2.5 has 1T total parameters with 32B activated parameters, 384 routed experts (8 selected per token), and 61 transformer layers. For more information, refer to the [Kimi-K2.5 model card](https://huggingface.co/moonshotai/Kimi-K2.5). The NVIDIA Kimi-K2.5-MXFP8 model was quantized using the [TensorRT Model Optimizer](https://github.com/NVIDIA/TensorRT-Model-Optimizer).
+
+This model is ready for commercial/non-commercial use.  <br>
+
+## Third-Party Community Consideration
+This model is not owned or developed by NVIDIA. This model has been developed and built to a third-party's requirements for this application and use case; see link to Non-NVIDIA [(Kimi-K2.5) Model Card](https://huggingface.co/moonshotai/Kimi-K2.5).
+
+### License/Terms of Use:
+[Modified MIT](https://huggingface.co/moonshotai/Kimi-K2.5/blob/main/LICENSE)
+
+### Deployment Geography:
+Global <br>
+
+### Use Case:
+Developers looking to take off the shelf, pre-quantized models for deployment in AI Agent systems, chatbots, RAG systems, multimodal applications, and other AI-powered applications. <br>
+
+### Release Date:
+Huggingface via https://huggingface.co/vincentzed-hf/Kimi-K2.5-MXFP8 <br>
+
+## Model Architecture:
+**Architecture Type:** Transformers (Mixture of Experts)  <br>
+**Network Architecture:** KimiK25ForConditionalGeneration (DeepseekV3-based) <br>
+**This model was developed based on [Kimi-K2.5](https://huggingface.co/moonshotai/Kimi-K2.5) <br>
+**Total Parameters:** 1T <br>
+**Activated Parameters:** 32B <br>
+**Number of Layers:** 61 (including 1 dense layer) <br>
+**Number of Experts:** 384 routed, 1 shared, 8 selected per token <br>
+**Vision Encoder:** MoonViT (400M parameters) <br>
+
+## Input:
+**Input Type(s):** Text, Image, Video <br>
+**Input Format(s):** String, Image tensors <br>
+**Input Parameters:** Multi-modal <br>
+
+## Output:
+**Output Type(s):** Text <br>
+**Output Format:** String <br>
+**Output Parameters:** 1D (One-Dimensional): Sequences <br>
+
+Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA's hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions. <br>
+
+## Software Integration:
+**Runtime Engine(s):** <br>
+* SGLang <br>
+
+**Supported Hardware Microarchitecture Compatibility:** <br>
+* NVIDIA Blackwell <br>
+
+**Preferred Operating System(s):** <br>
+* Linux <br>
+
+The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.
+
+## Model Version(s):
+** The model is quantized with nvidia-modelopt **0.41.0rc2.dev72+g886781332**  <br>
+
+## Training, Testing, and Evaluation Datasets:
+
+## Calibration Dataset:
+* Link: [Nemotron-Post-Training-Dataset-v2](https://huggingface.co/datasets/nvidia/Nemotron-Post-Training-Dataset-v2) <br>
+* Data collection method: Automated. <br>
+* Labeling method: Automated. <br>
+
+## Training Datasets:
+* Data Collection Method by Dataset: Undisclosed <br>
+* Labeling Method by Dataset: Undisclosed<br>
+* Properties: Undisclosed
+
+## Testing Dataset:
+* Data Collection Method by Dataset: Undisclosed <br>
+* Labeling Method by Dataset: Undisclosed <br>
+* Properties: Undisclosed <br>
+
+## Evaluation Dataset:
+* Data collection method: Hybrid: Automated, Human <br>
+* Labeling method: Hybrid: Human, Automated <br>
+
+## Inference:
+**Acceleration Engine:** SGLang <br>
+**Test Hardware:** B300 <br>
+
+## Post Training Quantization
+This model was obtained by quantizing the weights of Kimi-K2.5 to MXFP8 data type, ready for inference with SGLang. Only the weights of the linear operators within transformer blocks are quantized (excluding attention projections, vision tower, and mm_projector). This optimization reduces the number of bits per parameter from 16 to 8, reducing the disk size and GPU memory requirements by approximately 2x.
+
+## Usage
+
+### Deploy with SGLang
+
+To serve the quantized MXFP8 checkpoint with [SGLang](https://github.com/sgl-project/sglang):
+
+```bash
+python3 -m sglang.launch_server --model-path vincentzed-hf/Kimi-K2.5-MXFP8 --quantization modelopt
+```
+Please install from source:
+`git clone git@github.com:sgl-project/sglang.git`
+Once the repo is cloned, do `uv pip install -e "python[all]"` and run the serve command.
+
+### Reproduce with ModelOpt
+
+You may want to produce this checkpoint yourself. To reproduce the MXFP8 quantized checkpoint using [TensorRT Model Optimizer](https://github.com/NVIDIA/TensorRT-Model-Optimizer):
+
+```bash
+python3 examples/llm_ptq/hf_ptq.py \
+    --pyt_ckpt_path /root/.cache/huggingface/hub/models--moonshotai--Kimi-K2.5/snapshots/c0d6821ed3d48201b834278fb99d8f2d37732a52 \
+    --qformat mxfp8 \
+    --kv_cache_qformat none \
+    --export_path ./kimi-k2.5-mxfp8 \
+    --trust_remote_code
+```
+
+### Evaluation
+The accuracy benchmark results will be updated:
+<table>
+  <tr>
+   <td><strong>Precision</strong>
+   </td>
+   <td><strong>Benchmark 1</strong>
+   </td>
+   <td><strong>Benchmark 2</strong>
+   </td>
+  </tr>
+  <tr>
+   <td>BF16
+   </td>
+   <td><!-- TODO -->
+   </td>
+   <td><!-- TODO -->
+   </td>
+  </tr>
+  <tr>
+   <td>MXFP8
+   </td>
+   <td><!-- TODO -->
+   </td>
+   <td><!-- TODO -->
+   </td>
+  </tr>
+</table>
+
+> Baseline: [Kimi-K2.5](https://huggingface.co/moonshotai/Kimi-K2.5).
+
+## Model Limitations:
+The base model was trained on data that contains toxic language and societal biases originally crawled from the internet. Therefore, the model may amplify those biases and return toxic responses especially when prompted with toxic prompts. The model may generate answers that may be inaccurate, omit key information, or include irrelevant or redundant text producing socially unacceptable or undesirable text, even if the prompt itself does not include anything explicitly offensive.
+
+## Ethical Considerations
+
+NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
+
+Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).

chat_template.jinja

@@ -0,0 +1,108 @@
+{%- macro render_content(msg) -%}
+    {%- set c = msg.get('content') -%}
+    {%- if c is string -%}
+      {{ c }}
+    {%- elif c is not none -%}
+      {% for content in c -%}
+        {% if content['type'] == 'image' or content['type'] == 'image_url' -%}
+          <|media_begin|>image<|media_content|><|media_pad|><|media_end|>
+        {% elif content['type'] == 'video' or content['type']== 'video_url'-%}
+          <|kimi_k25_video_placeholder|>
+        {% else -%}
+          {{ content['text'] }}
+        {%- endif -%}
+      {%- endfor -%}
+    {%- endif -%}
+{%- endmacro -%}
+
+{% macro set_roles(message) -%}
+  {%- set role_name =  message.get('name') or  message['role'] -%}
+  {%- if message['role'] == 'user' -%}
+    <|im_user|>{{role_name}}<|im_middle|>
+  {%- elif message['role'] == 'assistant' -%}
+    <|im_assistant|>{{role_name}}<|im_middle|>
+  {%- else -%}
+    <|im_system|>{{role_name}}<|im_middle|>
+  {%- endif -%}
+{%- endmacro -%}
+
+
+{%- macro render_toolcalls(message) -%}
+  <|tool_calls_section_begin|>
+  {%- for tool_call in message['tool_calls'] -%}
+    {%- set formatted_id = tool_call['id'] -%}
+    <|tool_call_begin|>{{ formatted_id }}<|tool_call_argument_begin|>{% if tool_call['function']['arguments'] is string %}{{ tool_call['function']['arguments'] }}{% else %}{{ tool_call['function']['arguments'] | tojson }}{% endif %}<|tool_call_end|>
+  {%- endfor -%}
+  <|tool_calls_section_end|>
+{%- endmacro -%}
+
+
+{# Find last non-tool-call assisitant message #}
+{%- set ns = namespace(last_non_tool_call_assistant_msg=-1) -%}
+{%- for idx in range(messages|length-1, -1, -1) -%}
+    {%- if messages[idx]['role'] == 'assistant' and not messages[idx].get('tool_calls') -%}
+        {%- set ns.last_non_tool_call_assistant_msg = idx -%}
+        {%- break -%}
+    {%- endif -%}
+{%- endfor -%}
+
+{# split all messages into history & suffix, reasoning_content in suffix should be reserved.#}
+{%- set hist_msgs = messages[:ns.last_non_tool_call_assistant_msg+1] -%}
+{%- set suffix_msgs = messages[ns.last_non_tool_call_assistant_msg+1:] -%}
+
+{%- if tools -%}
+  {%- if tools_ts_str -%}
+    <|im_system|>tool_declare<|im_middle|>{{ tools_ts_str }}<|im_end|>
+  {%- else -%}
+    <|im_system|>tool_declare<|im_middle|>{{ tools | tojson(separators=(',', ':')) }}<|im_end|>
+  {%- endif -%}
+{%- endif -%}
+  
+{%- for message in hist_msgs -%}
+  {{set_roles(message)}}
+  {%- if message['role'] == 'assistant' -%}
+    <think></think>{{render_content(message)}}
+    {%- if message.get('tool_calls') -%}
+      {{render_toolcalls(message)}}
+    {%- endif -%}
+  {%- elif message['role'] == 'tool' -%}
+    {%- set tool_call_id = message.tool_call_id -%}
+    ## Return of {{ tool_call_id }}
+{{render_content(message)}}
+  {%- elif message['content'] is not none -%}
+    {{render_content(message)}}
+  {%- endif -%}
+  <|im_end|>
+{%- endfor -%}
+
+{%- for message in suffix_msgs -%}
+  {{set_roles(message)}}
+  {%- if message['role'] == 'assistant' -%}
+    {%- if thinking is defined and thinking is false -%}
+    <think></think>{{render_content(message)}}
+    {%- else -%}
+    {%- set rc = message.get('reasoning_content', '') -%}
+    <think>{{rc}}</think>{{render_content(message)}}
+    {%- endif -%}
+    {%- if message.get('tool_calls') -%}
+     {{render_toolcalls(message)}}
+    {%- endif -%}
+  {%- elif message['role'] == 'tool' -%}
+    {%- set tool_call_id = message.tool_call_id -%}
+    ## Return of {{ tool_call_id }}
+{{render_content(message)}}
+  {%- elif message['content'] is not none -%}
+    {{render_content(message)}}
+  {%- endif -%}
+  <|im_end|>
+{%- endfor -%}
+
+
+{%- if add_generation_prompt -%}
+  <|im_assistant|>assistant<|im_middle|>
+  {%- if thinking is defined and thinking is false -%}
+  <think></think>
+  {%- else -%}
+  <think>
+  {%- endif -%}
+{%- endif -%}

config.json

@@ -0,0 +1,449 @@
+{
+    "architectures": [
+        "KimiK25ForConditionalGeneration"
+    ],
+    "auto_map": {
+        "AutoConfig": "configuration_kimi_k25.KimiK25Config",
+        "AutoModel": "modeling_kimi_k25.KimiK25ForConditionalGeneration",
+        "AutoModelForCausalLM": "modeling_kimi_k25.KimiK25ForConditionalGeneration"
+    },
+    "bos_token_id": 163584,
+    "dtype": "bfloat16",
+    "eos_token_id": 163585,
+    "ignore_index": -100,
+    "media_placeholder_token_id": 163605,
+    "model_type": "kimi_k25",
+    "pad_token_id": 163839,
+    "quantization_config": {
+        "ignore": [
+            "language_model.lm_head",
+            "language_model.model.layers.0.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.0.self_attn.kv_b_proj",
+            "language_model.model.layers.0.self_attn.q_a_proj",
+            "language_model.model.layers.0.self_attn.q_b_proj",
+            "language_model.model.layers.1.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.1.self_attn.kv_b_proj",
+            "language_model.model.layers.1.self_attn.q_a_proj",
+            "language_model.model.layers.1.self_attn.q_b_proj",
+            "language_model.model.layers.10.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.10.self_attn.kv_b_proj",
+            "language_model.model.layers.10.self_attn.q_a_proj",
+            "language_model.model.layers.10.self_attn.q_b_proj",
+            "language_model.model.layers.11.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.11.self_attn.kv_b_proj",
+            "language_model.model.layers.11.self_attn.q_a_proj",
+            "language_model.model.layers.11.self_attn.q_b_proj",
+            "language_model.model.layers.12.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.12.self_attn.kv_b_proj",
+            "language_model.model.layers.12.self_attn.q_a_proj",
+            "language_model.model.layers.12.self_attn.q_b_proj",
+            "language_model.model.layers.13.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.13.self_attn.kv_b_proj",
+            "language_model.model.layers.13.self_attn.q_a_proj",
+            "language_model.model.layers.13.self_attn.q_b_proj",
+            "language_model.model.layers.14.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.14.self_attn.kv_b_proj",
+            "language_model.model.layers.14.self_attn.q_a_proj",
+            "language_model.model.layers.14.self_attn.q_b_proj",
+            "language_model.model.layers.15.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.15.self_attn.kv_b_proj",
+            "language_model.model.layers.15.self_attn.q_a_proj",
+            "language_model.model.layers.15.self_attn.q_b_proj",
+            "language_model.model.layers.16.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.16.self_attn.kv_b_proj",
+            "language_model.model.layers.16.self_attn.q_a_proj",
+            "language_model.model.layers.16.self_attn.q_b_proj",
+            "language_model.model.layers.17.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.17.self_attn.kv_b_proj",
+            "language_model.model.layers.17.self_attn.q_a_proj",
+            "language_model.model.layers.17.self_attn.q_b_proj",
+            "language_model.model.layers.18.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.18.self_attn.kv_b_proj",
+            "language_model.model.layers.18.self_attn.q_a_proj",
+            "language_model.model.layers.18.self_attn.q_b_proj",
+            "language_model.model.layers.19.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.19.self_attn.kv_b_proj",
+            "language_model.model.layers.19.self_attn.q_a_proj",
+            "language_model.model.layers.19.self_attn.q_b_proj",
+            "language_model.model.layers.2.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.2.self_attn.kv_b_proj",
+            "language_model.model.layers.2.self_attn.q_a_proj",
+            "language_model.model.layers.2.self_attn.q_b_proj",
+            "language_model.model.layers.20.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.20.self_attn.kv_b_proj",
+            "language_model.model.layers.20.self_attn.q_a_proj",
+            "language_model.model.layers.20.self_attn.q_b_proj",
+            "language_model.model.layers.21.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.21.self_attn.kv_b_proj",
+            "language_model.model.layers.21.self_attn.q_a_proj",
+            "language_model.model.layers.21.self_attn.q_b_proj",
+            "language_model.model.layers.22.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.22.self_attn.kv_b_proj",
+            "language_model.model.layers.22.self_attn.q_a_proj",
+            "language_model.model.layers.22.self_attn.q_b_proj",
+            "language_model.model.layers.23.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.23.self_attn.kv_b_proj",
+            "language_model.model.layers.23.self_attn.q_a_proj",
+            "language_model.model.layers.23.self_attn.q_b_proj",
+            "language_model.model.layers.24.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.24.self_attn.kv_b_proj",
+            "language_model.model.layers.24.self_attn.q_a_proj",
+            "language_model.model.layers.24.self_attn.q_b_proj",
+            "language_model.model.layers.25.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.25.self_attn.kv_b_proj",
+            "language_model.model.layers.25.self_attn.q_a_proj",
+            "language_model.model.layers.25.self_attn.q_b_proj",
+            "language_model.model.layers.26.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.26.self_attn.kv_b_proj",
+            "language_model.model.layers.26.self_attn.q_a_proj",
+            "language_model.model.layers.26.self_attn.q_b_proj",
+            "language_model.model.layers.27.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.27.self_attn.kv_b_proj",
+            "language_model.model.layers.27.self_attn.q_a_proj",
+            "language_model.model.layers.27.self_attn.q_b_proj",
+            "language_model.model.layers.28.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.28.self_attn.kv_b_proj",
+            "language_model.model.layers.28.self_attn.q_a_proj",
+            "language_model.model.layers.28.self_attn.q_b_proj",
+            "language_model.model.layers.29.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.29.self_attn.kv_b_proj",
+            "language_model.model.layers.29.self_attn.q_a_proj",
+            "language_model.model.layers.29.self_attn.q_b_proj",
+            "language_model.model.layers.3.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.3.self_attn.kv_b_proj",
+            "language_model.model.layers.3.self_attn.q_a_proj",
+            "language_model.model.layers.3.self_attn.q_b_proj",
+            "language_model.model.layers.30.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.30.self_attn.kv_b_proj",
+            "language_model.model.layers.30.self_attn.q_a_proj",
+            "language_model.model.layers.30.self_attn.q_b_proj",
+            "language_model.model.layers.31.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.31.self_attn.kv_b_proj",
+            "language_model.model.layers.31.self_attn.q_a_proj",
+            "language_model.model.layers.31.self_attn.q_b_proj",
+            "language_model.model.layers.32.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.32.self_attn.kv_b_proj",
+            "language_model.model.layers.32.self_attn.q_a_proj",
+            "language_model.model.layers.32.self_attn.q_b_proj",
+            "language_model.model.layers.33.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.33.self_attn.kv_b_proj",
+            "language_model.model.layers.33.self_attn.q_a_proj",
+            "language_model.model.layers.33.self_attn.q_b_proj",
+            "language_model.model.layers.34.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.34.self_attn.kv_b_proj",
+            "language_model.model.layers.34.self_attn.q_a_proj",
+            "language_model.model.layers.34.self_attn.q_b_proj",
+            "language_model.model.layers.35.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.35.self_attn.kv_b_proj",
+            "language_model.model.layers.35.self_attn.q_a_proj",
+            "language_model.model.layers.35.self_attn.q_b_proj",
+            "language_model.model.layers.36.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.36.self_attn.kv_b_proj",
+            "language_model.model.layers.36.self_attn.q_a_proj",
+            "language_model.model.layers.36.self_attn.q_b_proj",
+            "language_model.model.layers.37.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.37.self_attn.kv_b_proj",
+            "language_model.model.layers.37.self_attn.q_a_proj",
+            "language_model.model.layers.37.self_attn.q_b_proj",
+            "language_model.model.layers.38.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.38.self_attn.kv_b_proj",
+            "language_model.model.layers.38.self_attn.q_a_proj",
+            "language_model.model.layers.38.self_attn.q_b_proj",
+            "language_model.model.layers.39.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.39.self_attn.kv_b_proj",
+            "language_model.model.layers.39.self_attn.q_a_proj",
+            "language_model.model.layers.39.self_attn.q_b_proj",
+            "language_model.model.layers.4.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.4.self_attn.kv_b_proj",
+            "language_model.model.layers.4.self_attn.q_a_proj",
+            "language_model.model.layers.4.self_attn.q_b_proj",
+            "language_model.model.layers.40.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.40.self_attn.kv_b_proj",
+            "language_model.model.layers.40.self_attn.q_a_proj",
+            "language_model.model.layers.40.self_attn.q_b_proj",
+            "language_model.model.layers.41.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.41.self_attn.kv_b_proj",
+            "language_model.model.layers.41.self_attn.q_a_proj",
+            "language_model.model.layers.41.self_attn.q_b_proj",
+            "language_model.model.layers.42.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.42.self_attn.kv_b_proj",
+            "language_model.model.layers.42.self_attn.q_a_proj",
+            "language_model.model.layers.42.self_attn.q_b_proj",
+            "language_model.model.layers.43.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.43.self_attn.kv_b_proj",
+            "language_model.model.layers.43.self_attn.q_a_proj",
+            "language_model.model.layers.43.self_attn.q_b_proj",
+            "language_model.model.layers.44.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.44.self_attn.kv_b_proj",
+            "language_model.model.layers.44.self_attn.q_a_proj",
+            "language_model.model.layers.44.self_attn.q_b_proj",
+            "language_model.model.layers.45.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.45.self_attn.kv_b_proj",
+            "language_model.model.layers.45.self_attn.q_a_proj",
+            "language_model.model.layers.45.self_attn.q_b_proj",
+            "language_model.model.layers.46.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.46.self_attn.kv_b_proj",
+            "language_model.model.layers.46.self_attn.q_a_proj",
+            "language_model.model.layers.46.self_attn.q_b_proj",
+            "language_model.model.layers.47.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.47.self_attn.kv_b_proj",
+            "language_model.model.layers.47.self_attn.q_a_proj",
+            "language_model.model.layers.47.self_attn.q_b_proj",
+            "language_model.model.layers.48.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.48.self_attn.kv_b_proj",
+            "language_model.model.layers.48.self_attn.q_a_proj",
+            "language_model.model.layers.48.self_attn.q_b_proj",
+            "language_model.model.layers.49.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.49.self_attn.kv_b_proj",
+            "language_model.model.layers.49.self_attn.q_a_proj",
+            "language_model.model.layers.49.self_attn.q_b_proj",
+            "language_model.model.layers.5.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.5.self_attn.kv_b_proj",
+            "language_model.model.layers.5.self_attn.q_a_proj",
+            "language_model.model.layers.5.self_attn.q_b_proj",
+            "language_model.model.layers.50.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.50.self_attn.kv_b_proj",
+            "language_model.model.layers.50.self_attn.q_a_proj",
+            "language_model.model.layers.50.self_attn.q_b_proj",
+            "language_model.model.layers.51.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.51.self_attn.kv_b_proj",
+            "language_model.model.layers.51.self_attn.q_a_proj",
+            "language_model.model.layers.51.self_attn.q_b_proj",
+            "language_model.model.layers.52.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.52.self_attn.kv_b_proj",
+            "language_model.model.layers.52.self_attn.q_a_proj",
+            "language_model.model.layers.52.self_attn.q_b_proj",
+            "language_model.model.layers.53.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.53.self_attn.kv_b_proj",
+            "language_model.model.layers.53.self_attn.q_a_proj",
+            "language_model.model.layers.53.self_attn.q_b_proj",
+            "language_model.model.layers.54.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.54.self_attn.kv_b_proj",
+            "language_model.model.layers.54.self_attn.q_a_proj",
+            "language_model.model.layers.54.self_attn.q_b_proj",
+            "language_model.model.layers.55.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.55.self_attn.kv_b_proj",
+            "language_model.model.layers.55.self_attn.q_a_proj",
+            "language_model.model.layers.55.self_attn.q_b_proj",
+            "language_model.model.layers.56.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.56.self_attn.kv_b_proj",
+            "language_model.model.layers.56.self_attn.q_a_proj",
+            "language_model.model.layers.56.self_attn.q_b_proj",
+            "language_model.model.layers.57.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.57.self_attn.kv_b_proj",
+            "language_model.model.layers.57.self_attn.q_a_proj",
+            "language_model.model.layers.57.self_attn.q_b_proj",
+            "language_model.model.layers.58.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.58.self_attn.kv_b_proj",
+            "language_model.model.layers.58.self_attn.q_a_proj",
+            "language_model.model.layers.58.self_attn.q_b_proj",
+            "language_model.model.layers.59.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.59.self_attn.kv_b_proj",
+            "language_model.model.layers.59.self_attn.q_a_proj",
+            "language_model.model.layers.59.self_attn.q_b_proj",
+            "language_model.model.layers.6.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.6.self_attn.kv_b_proj",
+            "language_model.model.layers.6.self_attn.q_a_proj",
+            "language_model.model.layers.6.self_attn.q_b_proj",
+            "language_model.model.layers.60.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.60.self_attn.kv_b_proj",
+            "language_model.model.layers.60.self_attn.q_a_proj",
+            "language_model.model.layers.60.self_attn.q_b_proj",
+            "language_model.model.layers.7.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.7.self_attn.kv_b_proj",
+            "language_model.model.layers.7.self_attn.q_a_proj",
+            "language_model.model.layers.7.self_attn.q_b_proj",
+            "language_model.model.layers.8.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.8.self_attn.kv_b_proj",
+            "language_model.model.layers.8.self_attn.q_a_proj",
+            "language_model.model.layers.8.self_attn.q_b_proj",
+            "language_model.model.layers.9.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.9.self_attn.kv_b_proj",
+            "language_model.model.layers.9.self_attn.q_a_proj",
+            "language_model.model.layers.9.self_attn.q_b_proj",
+            "mm_projector*",
+            "vision_tower*"
+        ],
+        "quant_algo": "MXFP8",
+        "producer": {
+            "name": "modelopt",
+            "version": "0.41.0rc2.dev72+g886781332"
+        },
+        "quant_method": "modelopt"
+    },
+    "text_config": {
+        "_name_or_path": "",
+        "add_cross_attention": false,
+        "architectures": [
+            "DeepseekV3ForCausalLM"
+        ],
+        "attention_bias": false,
+        "attention_dropout": 0.0,
+        "auto_map": {
+            "AutoConfig": "configuration_deepseek.DeepseekV3Config",
+            "AutoModel": "modeling_deepseek.DeepseekV3Model",
+            "AutoModelForCausalLM": "modeling_deepseek.DeepseekV3ForCausalLM"
+        },
+        "aux_loss_alpha": 0.001,
+        "bad_words_ids": null,
+        "begin_suppress_tokens": null,
+        "bos_token_id": 163584,
+        "chunk_size_feed_forward": 0,
+        "cross_attention_hidden_size": null,
+        "decoder_start_token_id": null,
+        "diversity_penalty": 0.0,
+        "do_sample": false,
+        "dtype": "bfloat16",
+        "early_stopping": false,
+        "encoder_no_repeat_ngram_size": 0,
+        "eos_token_id": 163585,
+        "ep_size": 1,
+        "exponential_decay_length_penalty": null,
+        "finetuning_task": null,
+        "first_k_dense_replace": 1,
+        "forced_bos_token_id": null,
+        "forced_eos_token_id": null,
+        "hidden_act": "silu",
+        "hidden_size": 7168,
+        "id2label": {
+            "0": "LABEL_0",
+            "1": "LABEL_1"
+        },
+        "initializer_range": 0.02,
+        "intermediate_size": 18432,
+        "is_decoder": false,
+        "is_encoder_decoder": false,
+        "kv_lora_rank": 512,
+        "label2id": {
+            "LABEL_0": 0,
+            "LABEL_1": 1
+        },
+        "length_penalty": 1.0,
+        "max_length": 20,
+        "max_position_embeddings": 262144,
+        "min_length": 0,
+        "model_type": "deepseek_v3",
+        "moe_intermediate_size": 2048,
+        "moe_layer_freq": 1,
+        "n_group": 1,
+        "n_routed_experts": 384,
+        "n_shared_experts": 1,
+        "no_repeat_ngram_size": 0,
+        "norm_topk_prob": true,
+        "num_attention_heads": 64,
+        "num_beam_groups": 1,
+        "num_beams": 1,
+        "num_experts_per_tok": 8,
+        "num_hidden_layers": 61,
+        "num_key_value_heads": 64,
+        "num_nextn_predict_layers": 0,
+        "num_return_sequences": 1,
+        "output_attentions": false,
+        "output_hidden_states": false,
+        "output_scores": false,
+        "pad_token_id": 163839,
+        "prefix": null,
+        "pretraining_tp": 1,
+        "problem_type": null,
+        "pruned_heads": {},
+        "q_lora_rank": 1536,
+        "qk_nope_head_dim": 128,
+        "qk_rope_head_dim": 64,
+        "quantization_config": {
+            "config_groups": {
+                "group_0": {
+                    "input_activations": null,
+                    "output_activations": null,
+                    "targets": [
+                        "Linear"
+                    ],
+                    "weights": {
+                        "actorder": null,
+                        "block_structure": null,
+                        "dynamic": false,
+                        "group_size": 32,
+                        "num_bits": 4,
+                        "observer": "minmax",
+                        "observer_kwargs": {},
+                        "strategy": "group",
+                        "symmetric": true,
+                        "type": "int"
+                    }
+                }
+            },
+            "format": "pack-quantized",
+            "ignore": [
+                "lm_head",
+                "re:.*self_attn.*",
+                "re:.*shared_experts.*",
+                "re:.*mlp\\.(gate|up|gate_up|down)_proj.*"
+            ],
+            "kv_cache_scheme": null,
+            "quant_method": "compressed-tensors",
+            "quantization_status": "compressed"
+        },
+        "remove_invalid_values": false,
+        "repetition_penalty": 1.0,
+        "return_dict": true,
+        "return_dict_in_generate": false,
+        "rms_norm_eps": 1e-05,
+        "rope_scaling": {
+            "beta_fast": 32.0,
+            "beta_slow": 1.0,
+            "factor": 64.0,
+            "mscale": 1.0,
+            "mscale_all_dim": 1.0,
+            "original_max_position_embeddings": 4096,
+            "type": "yarn"
+        },
+        "rope_theta": 50000.0,
+        "routed_scaling_factor": 2.827,
+        "scoring_func": "sigmoid",
+        "sep_token_id": null,
+        "seq_aux": true,
+        "suppress_tokens": null,
+        "task_specific_params": null,
+        "temperature": 1.0,
+        "tf_legacy_loss": false,
+        "tie_encoder_decoder": false,
+        "tie_word_embeddings": false,
+        "tokenizer_class": null,
+        "top_k": 50,
+        "top_p": 1.0,
+        "topk_group": 1,
+        "topk_method": "noaux_tc",
+        "torchscript": false,
+        "typical_p": 1.0,
+        "use_bfloat16": false,
+        "use_cache": true,
+        "v_head_dim": 128,
+        "vocab_size": 163840
+    },
+    "tie_word_embeddings": false,
+    "transformers_version": "4.57.6",
+    "use_unified_vision_chunk": true,
+    "video_placeholder": "<|kimi_k25_video_placeholder|>",
+    "vision_config": {
+        "init_pos_emb_height": 64,
+        "init_pos_emb_time": 4,
+        "init_pos_emb_width": 64,
+        "merge_kernel_size": [
+            2,
+            2
+        ],
+        "merge_type": "sd2_tpool",
+        "mm_hidden_size": 1152,
+        "mm_projector_type": "patchmerger",
+        "model_type": "",
+        "patch_size": 14,
+        "pos_emb_type": "divided_fixed",
+        "projector_hidden_act": "gelu",
+        "projector_ln_eps": 1e-05,
+        "text_hidden_size": 7168,
+        "video_attn_type": "spatial_temporal",
+        "vt_hidden_size": 1152,
+        "vt_intermediate_size": 4304,
+        "vt_num_attention_heads": 16,
+        "vt_num_hidden_layers": 27
+    }
+}

configuration_deepseek.py

@@ -0,0 +1,214 @@
+# Copy from https://huggingface.co/deepseek-ai/DeepSeek-V3/blob/main/configuration_deepseek.py
+
+from transformers.configuration_utils import PretrainedConfig
+from transformers.utils import logging
+
+logger = logging.get_logger(__name__)
+
+DEEPSEEK_PRETRAINED_CONFIG_ARCHIVE_MAP = {}
+
+
+class DeepseekV3Config(PretrainedConfig):
+    r"""
+    This is the configuration class to store the configuration of a [`DeepseekV3Model`]. It is used to instantiate an DeepSeek
+    model according to the specified arguments, defining the model architecture. Instantiating a configuration with the
+    defaults will yield a similar configuration to that of the DeepSeek-V3.
+
+    Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
+    documentation from [`PretrainedConfig`] for more information.
+
+
+    Args:
+        vocab_size (`int`, *optional*, defaults to 129280):
+            Vocabulary size of the Deep model. Defines the number of different tokens that can be represented by the
+            `inputs_ids` passed when calling [`DeepseekV3Model`]
+        hidden_size (`int`, *optional*, defaults to 4096):
+            Dimension of the hidden representations.
+        intermediate_size (`int`, *optional*, defaults to 11008):
+            Dimension of the MLP representations.
+        moe_intermediate_size (`int`, *optional*, defaults to 1407):
+            Dimension of the MoE representations.
+        num_hidden_layers (`int`, *optional*, defaults to 32):
+            Number of hidden layers in the Transformer decoder.
+        num_nextn_predict_layers (`int`, *optional*, defaults to 1):
+            Number of nextn predict layers in the DeepSeekV3 Model.
+        num_attention_heads (`int`, *optional*, defaults to 32):
+            Number of attention heads for each attention layer in the Transformer decoder.
+        n_shared_experts (`int`, *optional*, defaults to None):
+            Number of shared experts, None means dense model.
+        n_routed_experts (`int`, *optional*, defaults to None):
+            Number of routed experts, None means dense model.
+        routed_scaling_factor (`float`, *optional*, defaults to 1.0):
+            Scaling factor or routed experts.
+        topk_method (`str`, *optional*, defaults to `gready`):
+            Topk method used in routed gate.
+        n_group (`int`, *optional*, defaults to None):
+            Number of groups for routed experts.
+        topk_group (`int`, *optional*, defaults to None):
+            Number of selected groups for each token(for each token, ensuring the selected experts is only within `topk_group` groups).
+        num_experts_per_tok (`int`, *optional*, defaults to None):
+            Number of selected experts, None means dense model.
+        moe_layer_freq (`int`, *optional*, defaults to 1):
+            The frequency of the MoE layer: one expert layer for every `moe_layer_freq - 1` dense layers.
+        first_k_dense_replace (`int`, *optional*, defaults to 0):
+            Number of dense layers in shallow layers(embed->dense->dense->...->dense->moe->moe...->lm_head).
+                                                            \--k dense layers--/
+        norm_topk_prob (`bool`, *optional*, defaults to False):
+            Whether to normalize the weights of the routed experts.
+        scoring_func (`str`, *optional*, defaults to 'softmax'):
+            Method of computing expert weights.
+        aux_loss_alpha (`float`, *optional*, defaults to 0.001):
+            Auxiliary loss weight coefficient.
+        seq_aux = (`bool`, *optional*, defaults to True):
+            Whether to compute the auxiliary loss for each individual sample.
+        num_key_value_heads (`int`, *optional*):
+            This is the number of key_value heads that should be used to implement Grouped Query Attention. If
+            `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if
+            `num_key_value_heads=1 the model will use Multi Query Attention (MQA) otherwise GQA is used. When
+            converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed
+            by meanpooling all the original heads within that group. For more details checkout [this
+            paper](https://arxiv.org/pdf/2305.13245.pdf). If it is not specified, will default to
+            `num_attention_heads`.
+        hidden_act (`str` or `function`, *optional*, defaults to `"silu"`):
+            The non-linear activation function (function or string) in the decoder.
+        max_position_embeddings (`int`, *optional*, defaults to 2048):
+            The maximum sequence length that this model might ever be used with.
+        initializer_range (`float`, *optional*, defaults to 0.02):
+            The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
+        rms_norm_eps (`float`, *optional*, defaults to 1e-06):
+            The epsilon used by the rms normalization layers.
+        use_cache (`bool`, *optional*, defaults to `True`):
+            Whether or not the model should return the last key/values attentions (not used by all models). Only
+            relevant if `config.is_decoder=True`.
+        pad_token_id (`int`, *optional*):
+            Padding token id.
+        bos_token_id (`int`, *optional*, defaults to 1):
+            Beginning of stream token id.
+        eos_token_id (`int`, *optional*, defaults to 2):
+            End of stream token id.
+        pretraining_tp (`int`, *optional*, defaults to 1):
+            Experimental feature. Tensor parallelism rank used during pretraining. Please refer to [this
+            document](https://huggingface.co/docs/transformers/parallelism) to understand more about it. This value is
+            necessary to ensure exact reproducibility of the pretraining results. Please refer to [this
+            issue](https://github.com/pytorch/pytorch/issues/76232).
+        tie_word_embeddings (`bool`, *optional*, defaults to `False`):
+            Whether to tie weight embeddings
+        rope_theta (`float`, *optional*, defaults to 10000.0):
+            The base period of the RoPE embeddings.
+        rope_scaling (`Dict`, *optional*):
+            Dictionary containing the scaling configuration for the RoPE embeddings. Currently supports two scaling
+            strategies: linear and dynamic. Their scaling factor must be a float greater than 1. The expected format is
+            `{"type": strategy name, "factor": scaling factor}`. When using this flag, don't update
+            `max_position_embeddings` to the expected new maximum.
+        attention_bias (`bool`, defaults to `False`, *optional*, defaults to `False`):
+            Whether to use a bias in the query, key, value and output projection layers during self-attention.
+        attention_dropout (`float`, *optional*, defaults to 0.0):
+            The dropout ratio for the attention probabilities.
+
+    ```python
+    >>> from transformers import DeepseekV3Model, DeepseekV3Config
+
+    >>> # Initializing a Deepseek-V3 style configuration
+    >>> configuration = DeepseekV3Config()
+
+    >>> # Accessing the model configuration
+    >>> configuration = model.config
+    ```"""
+
+    model_type = "deepseek_v3"
+    keys_to_ignore_at_inference = ["past_key_values"]
+
+    def __init__(
+        self,
+        vocab_size=129280,
+        hidden_size=7168,
+        intermediate_size=18432,
+        moe_intermediate_size=2048,
+        num_hidden_layers=61,
+        num_nextn_predict_layers=1,
+        num_attention_heads=128,
+        num_key_value_heads=128,
+        n_shared_experts=1,
+        n_routed_experts=256,
+        ep_size=1,
+        routed_scaling_factor=2.5,
+        kv_lora_rank=512,
+        q_lora_rank=1536,
+        qk_rope_head_dim=64,
+        v_head_dim=128,
+        qk_nope_head_dim=128,
+        topk_method='noaux_tc',
+        n_group=8,
+        topk_group=4,
+        num_experts_per_tok=8,
+        moe_layer_freq=1,
+        first_k_dense_replace=3,
+        norm_topk_prob=True,
+        scoring_func='sigmoid',
+        aux_loss_alpha=0.001,
+        seq_aux=True,
+        hidden_act="silu",
+        max_position_embeddings=4096,
+        initializer_range=0.02,
+        rms_norm_eps=1e-6,
+        use_cache=True,
+        pad_token_id=None,
+        bos_token_id=0,
+        eos_token_id=1,
+        pretraining_tp=1,
+        tie_word_embeddings=False,
+        rope_theta=10000.0,
+        rope_scaling=None,
+        attention_bias=False,
+        attention_dropout=0.0,
+        **kwargs,
+    ):
+        self.vocab_size = vocab_size
+        self.max_position_embeddings = max_position_embeddings
+        self.hidden_size = hidden_size
+        self.intermediate_size = intermediate_size
+        self.moe_intermediate_size = moe_intermediate_size
+        self.num_hidden_layers = num_hidden_layers
+        self.num_nextn_predict_layers = num_nextn_predict_layers
+        self.num_attention_heads = num_attention_heads
+        self.n_shared_experts = n_shared_experts
+        self.n_routed_experts = n_routed_experts
+        self.ep_size = ep_size
+        self.routed_scaling_factor = routed_scaling_factor
+        self.kv_lora_rank = kv_lora_rank
+        self.q_lora_rank = q_lora_rank
+        self.qk_rope_head_dim = qk_rope_head_dim
+        self.v_head_dim = v_head_dim
+        self.qk_nope_head_dim = qk_nope_head_dim
+        self.topk_method = topk_method
+        self.n_group = n_group
+        self.topk_group = topk_group
+        self.num_experts_per_tok = num_experts_per_tok
+        self.moe_layer_freq = moe_layer_freq
+        self.first_k_dense_replace = first_k_dense_replace
+        self.norm_topk_prob = norm_topk_prob
+        self.scoring_func = scoring_func
+        self.aux_loss_alpha = aux_loss_alpha
+        self.seq_aux = seq_aux
+        # for backward compatibility
+        if num_key_value_heads is None:
+            num_key_value_heads = num_attention_heads
+
+        self.num_key_value_heads = num_key_value_heads
+        self.hidden_act = hidden_act
+        self.initializer_range = initializer_range
+        self.rms_norm_eps = rms_norm_eps
+        self.pretraining_tp = pretraining_tp
+        self.use_cache = use_cache
+        self.rope_theta = rope_theta
+        self.rope_scaling = rope_scaling
+        self.attention_bias = attention_bias
+        self.attention_dropout = attention_dropout
+
+        super().__init__(
+            pad_token_id=pad_token_id,
+            bos_token_id=bos_token_id,
+            eos_token_id=eos_token_id,
+            tie_word_embeddings=tie_word_embeddings,
+            **kwargs,
+        )

configuration_kimi_k25.py

@@ -0,0 +1,123 @@
+from transformers.configuration_utils import PretrainedConfig
+
+try:
+    from configuration_deepseek import DeepseekV3Config
+except ImportError:
+    from .configuration_deepseek import DeepseekV3Config
+
+
+class KimiK25VisionConfig(PretrainedConfig):
+
+    def __init__(
+            self,
+            patch_size: int = 14,
+            init_pos_emb_height: int = 64,
+            init_pos_emb_width: int = 64,
+            init_pos_emb_time: int = 4,
+            pos_emb_type: str = 'divided_fixed',
+            vt_num_attention_heads: int = 16,
+            vt_num_hidden_layers: int = 27,
+            vt_hidden_size: int = 1152,
+            vt_intermediate_size: int = 4304,
+            merge_kernel_size: tuple = (2, 2),
+            video_attn_type: str = 'spatial_temporal',
+            merge_type: str = 'sd2_tpool',
+            _attn_implementation: str = 'flash_attention_2',
+            # MM Projector parameters
+            mm_projector_type: str = 'patchmerger',
+            mm_hidden_size: int | None = None,
+            projector_hidden_act: str = "gelu",
+            projector_ln_eps: float = 1e-5,
+            # Other parameters
+            ignore_index: int = -100,
+            media_placeholder_token_id: int = 163605,
+            pad_token_id: int = 0,
+            use_unified_vision_chunk: bool = True,
+            video_placeholder="<|kimi_k25_video_placeholder|>",
+            text_hidden_size=7168,
+            **vision_config_kwargs):
+
+        self.patch_size = patch_size
+        self.init_pos_emb_height = init_pos_emb_height
+        self.init_pos_emb_width = init_pos_emb_width
+        self.init_pos_emb_time = init_pos_emb_time
+        self.pos_emb_type = pos_emb_type
+        self.vt_num_attention_heads = vt_num_attention_heads
+        self.vt_num_hidden_layers = vt_num_hidden_layers
+        self.vt_hidden_size = vt_hidden_size
+        self.vt_intermediate_size = vt_intermediate_size
+        self.merge_kernel_size = merge_kernel_size
+        self.video_attn_type = video_attn_type
+        self.merge_type = merge_type
+        self._attn_implementation = _attn_implementation
+
+        # MM Projector config
+        self.mm_projector_type = mm_projector_type
+        self.mm_hidden_size = mm_hidden_size if mm_hidden_size is not None else vt_hidden_size
+        self.projector_hidden_act = projector_hidden_act
+        self.projector_ln_eps = projector_ln_eps
+        self.text_hidden_size = text_hidden_size
+
+
+class KimiK25Config(PretrainedConfig):
+    """Kimi-K2.5 model configuration.
+
+    Args:
+        text_config (dict | DeepseekV3Config): Configuration for the text model.
+        
+        Vision Tower Parameters (from MoonViT3dConfig):
+            patch_size (int): Patch size for vision tower.
+            init_pos_emb_height (int): Initial position embedding height.
+            init_pos_emb_width (int): Initial position embedding width.
+            init_pos_emb_time (int): Initial position embedding time dimension.
+            pos_emb_type (str): Type of position embedding.
+            vt_num_attention_heads (int): Number of attention heads in vision tower.
+            vt_num_hidden_layers (int): Number of hidden layers in vision tower.
+            vt_hidden_size (int): Hidden size of vision tower.
+            vt_intermediate_size (int): Intermediate size in vision tower FFN.
+            merge_kernel_size (tuple): Kernel size for patch merging.
+            video_attn_type (str): Type of video attention.
+            merge_type (str): Type of merge operation.
+            _attn_implementation (str): Attention implementation type.
+        
+        MM Projector Parameters (from MultiModalProjectorConfig):
+            mm_projector_type (str): Type of multimodal projector.
+            mm_hidden_size (int): Hidden size from vision tower (should match vt_hidden_size).
+            projector_hidden_act (str): Activation function for projector.
+            projector_ln_eps (float): Layer norm epsilon for projector.
+        
+        Other Parameters:
+            ignore_index (int): The ignore index for the loss function.
+            media_placeholder_token_id (int): The token ID to use for media placeholders.
+            pad_token_id (int): The token ID to use for padding.
+    """
+
+    model_type = "kimi_k25"
+
+    def __init__(
+        self,
+        text_config: dict | DeepseekV3Config = None,
+        vision_config: dict | KimiK25VisionConfig = None,
+        # Other parameters
+        ignore_index: int = -100,
+        media_placeholder_token_id: int = 163605,
+        pad_token_id: int = 0,
+        use_unified_vision_chunk: bool = True,
+        video_placeholder="<|kimi_k25_video_placeholder|>",
+        **kwargs,
+    ):
+        if isinstance(text_config, dict):
+            text_config = DeepseekV3Config(**text_config)
+        if isinstance(vision_config, dict):
+            vision_config = KimiK25VisionConfig(**vision_config)
+        self.text_config = text_config
+        self.vision_config = vision_config
+        # Other config
+        self.ignore_index = ignore_index
+        self.media_placeholder_token_id = media_placeholder_token_id
+        self.use_unified_vision_chunk = use_unified_vision_chunk
+        self.video_placeholder = video_placeholder
+        if getattr(self.text_config, "quantization_config", None) is not None:
+            self.quantization_config = self.text_config.quantization_config
+
+        super().__init__(pad_token_id=pad_token_id, **kwargs)

generation_config.json

@@ -0,0 +1,4 @@
+{
+  "max_length": 262144,
+  "eos_token_id": 163586
+}

hf_quant_config.json

@@ -0,0 +1,260 @@
+{
+    "producer": {
+        "name": "modelopt",
+        "version": "0.41.0rc2.dev72+g886781332"
+    },
+    "quantization": {
+        "quant_algo": "MXFP8",
+        "kv_cache_quant_algo": null,
+        "group_size": 32,
+        "exclude_modules": [
+            "language_model.lm_head",
+            "language_model.model.layers.0.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.0.self_attn.kv_b_proj",
+            "language_model.model.layers.0.self_attn.q_a_proj",
+            "language_model.model.layers.0.self_attn.q_b_proj",
+            "language_model.model.layers.1.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.1.self_attn.kv_b_proj",
+            "language_model.model.layers.1.self_attn.q_a_proj",
+            "language_model.model.layers.1.self_attn.q_b_proj",
+            "language_model.model.layers.10.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.10.self_attn.kv_b_proj",
+            "language_model.model.layers.10.self_attn.q_a_proj",
+            "language_model.model.layers.10.self_attn.q_b_proj",
+            "language_model.model.layers.11.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.11.self_attn.kv_b_proj",
+            "language_model.model.layers.11.self_attn.q_a_proj",
+            "language_model.model.layers.11.self_attn.q_b_proj",
+            "language_model.model.layers.12.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.12.self_attn.kv_b_proj",
+            "language_model.model.layers.12.self_attn.q_a_proj",
+            "language_model.model.layers.12.self_attn.q_b_proj",
+            "language_model.model.layers.13.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.13.self_attn.kv_b_proj",
+            "language_model.model.layers.13.self_attn.q_a_proj",
+            "language_model.model.layers.13.self_attn.q_b_proj",
+            "language_model.model.layers.14.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.14.self_attn.kv_b_proj",
+            "language_model.model.layers.14.self_attn.q_a_proj",
+            "language_model.model.layers.14.self_attn.q_b_proj",
+            "language_model.model.layers.15.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.15.self_attn.kv_b_proj",
+            "language_model.model.layers.15.self_attn.q_a_proj",
+            "language_model.model.layers.15.self_attn.q_b_proj",
+            "language_model.model.layers.16.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.16.self_attn.kv_b_proj",
+            "language_model.model.layers.16.self_attn.q_a_proj",
+            "language_model.model.layers.16.self_attn.q_b_proj",
+            "language_model.model.layers.17.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.17.self_attn.kv_b_proj",
+            "language_model.model.layers.17.self_attn.q_a_proj",
+            "language_model.model.layers.17.self_attn.q_b_proj",
+            "language_model.model.layers.18.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.18.self_attn.kv_b_proj",
+            "language_model.model.layers.18.self_attn.q_a_proj",
+            "language_model.model.layers.18.self_attn.q_b_proj",
+            "language_model.model.layers.19.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.19.self_attn.kv_b_proj",
+            "language_model.model.layers.19.self_attn.q_a_proj",
+            "language_model.model.layers.19.self_attn.q_b_proj",
+            "language_model.model.layers.2.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.2.self_attn.kv_b_proj",
+            "language_model.model.layers.2.self_attn.q_a_proj",
+            "language_model.model.layers.2.self_attn.q_b_proj",
+            "language_model.model.layers.20.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.20.self_attn.kv_b_proj",
+            "language_model.model.layers.20.self_attn.q_a_proj",
+            "language_model.model.layers.20.self_attn.q_b_proj",
+            "language_model.model.layers.21.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.21.self_attn.kv_b_proj",
+            "language_model.model.layers.21.self_attn.q_a_proj",
+            "language_model.model.layers.21.self_attn.q_b_proj",
+            "language_model.model.layers.22.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.22.self_attn.kv_b_proj",
+            "language_model.model.layers.22.self_attn.q_a_proj",
+            "language_model.model.layers.22.self_attn.q_b_proj",
+            "language_model.model.layers.23.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.23.self_attn.kv_b_proj",
+            "language_model.model.layers.23.self_attn.q_a_proj",
+            "language_model.model.layers.23.self_attn.q_b_proj",
+            "language_model.model.layers.24.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.24.self_attn.kv_b_proj",
+            "language_model.model.layers.24.self_attn.q_a_proj",
+            "language_model.model.layers.24.self_attn.q_b_proj",
+            "language_model.model.layers.25.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.25.self_attn.kv_b_proj",
+            "language_model.model.layers.25.self_attn.q_a_proj",
+            "language_model.model.layers.25.self_attn.q_b_proj",
+            "language_model.model.layers.26.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.26.self_attn.kv_b_proj",
+            "language_model.model.layers.26.self_attn.q_a_proj",
+            "language_model.model.layers.26.self_attn.q_b_proj",
+            "language_model.model.layers.27.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.27.self_attn.kv_b_proj",
+            "language_model.model.layers.27.self_attn.q_a_proj",
+            "language_model.model.layers.27.self_attn.q_b_proj",
+            "language_model.model.layers.28.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.28.self_attn.kv_b_proj",
+            "language_model.model.layers.28.self_attn.q_a_proj",
+            "language_model.model.layers.28.self_attn.q_b_proj",
+            "language_model.model.layers.29.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.29.self_attn.kv_b_proj",
+            "language_model.model.layers.29.self_attn.q_a_proj",
+            "language_model.model.layers.29.self_attn.q_b_proj",
+            "language_model.model.layers.3.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.3.self_attn.kv_b_proj",
+            "language_model.model.layers.3.self_attn.q_a_proj",
+            "language_model.model.layers.3.self_attn.q_b_proj",
+            "language_model.model.layers.30.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.30.self_attn.kv_b_proj",
+            "language_model.model.layers.30.self_attn.q_a_proj",
+            "language_model.model.layers.30.self_attn.q_b_proj",
+            "language_model.model.layers.31.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.31.self_attn.kv_b_proj",
+            "language_model.model.layers.31.self_attn.q_a_proj",
+            "language_model.model.layers.31.self_attn.q_b_proj",
+            "language_model.model.layers.32.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.32.self_attn.kv_b_proj",
+            "language_model.model.layers.32.self_attn.q_a_proj",
+            "language_model.model.layers.32.self_attn.q_b_proj",
+            "language_model.model.layers.33.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.33.self_attn.kv_b_proj",
+            "language_model.model.layers.33.self_attn.q_a_proj",
+            "language_model.model.layers.33.self_attn.q_b_proj",
+            "language_model.model.layers.34.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.34.self_attn.kv_b_proj",
+            "language_model.model.layers.34.self_attn.q_a_proj",
+            "language_model.model.layers.34.self_attn.q_b_proj",
+            "language_model.model.layers.35.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.35.self_attn.kv_b_proj",
+            "language_model.model.layers.35.self_attn.q_a_proj",
+            "language_model.model.layers.35.self_attn.q_b_proj",
+            "language_model.model.layers.36.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.36.self_attn.kv_b_proj",
+            "language_model.model.layers.36.self_attn.q_a_proj",
+            "language_model.model.layers.36.self_attn.q_b_proj",
+            "language_model.model.layers.37.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.37.self_attn.kv_b_proj",
+            "language_model.model.layers.37.self_attn.q_a_proj",
+            "language_model.model.layers.37.self_attn.q_b_proj",
+            "language_model.model.layers.38.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.38.self_attn.kv_b_proj",
+            "language_model.model.layers.38.self_attn.q_a_proj",
+            "language_model.model.layers.38.self_attn.q_b_proj",
+            "language_model.model.layers.39.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.39.self_attn.kv_b_proj",
+            "language_model.model.layers.39.self_attn.q_a_proj",
+            "language_model.model.layers.39.self_attn.q_b_proj",
+            "language_model.model.layers.4.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.4.self_attn.kv_b_proj",
+            "language_model.model.layers.4.self_attn.q_a_proj",
+            "language_model.model.layers.4.self_attn.q_b_proj",
+            "language_model.model.layers.40.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.40.self_attn.kv_b_proj",
+            "language_model.model.layers.40.self_attn.q_a_proj",
+            "language_model.model.layers.40.self_attn.q_b_proj",
+            "language_model.model.layers.41.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.41.self_attn.kv_b_proj",
+            "language_model.model.layers.41.self_attn.q_a_proj",
+            "language_model.model.layers.41.self_attn.q_b_proj",
+            "language_model.model.layers.42.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.42.self_attn.kv_b_proj",
+            "language_model.model.layers.42.self_attn.q_a_proj",
+            "language_model.model.layers.42.self_attn.q_b_proj",
+            "language_model.model.layers.43.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.43.self_attn.kv_b_proj",
+            "language_model.model.layers.43.self_attn.q_a_proj",
+            "language_model.model.layers.43.self_attn.q_b_proj",
+            "language_model.model.layers.44.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.44.self_attn.kv_b_proj",
+            "language_model.model.layers.44.self_attn.q_a_proj",
+            "language_model.model.layers.44.self_attn.q_b_proj",
+            "language_model.model.layers.45.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.45.self_attn.kv_b_proj",
+            "language_model.model.layers.45.self_attn.q_a_proj",
+            "language_model.model.layers.45.self_attn.q_b_proj",
+            "language_model.model.layers.46.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.46.self_attn.kv_b_proj",
+            "language_model.model.layers.46.self_attn.q_a_proj",
+            "language_model.model.layers.46.self_attn.q_b_proj",
+            "language_model.model.layers.47.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.47.self_attn.kv_b_proj",
+            "language_model.model.layers.47.self_attn.q_a_proj",
+            "language_model.model.layers.47.self_attn.q_b_proj",
+            "language_model.model.layers.48.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.48.self_attn.kv_b_proj",
+            "language_model.model.layers.48.self_attn.q_a_proj",
+            "language_model.model.layers.48.self_attn.q_b_proj",
+            "language_model.model.layers.49.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.49.self_attn.kv_b_proj",
+            "language_model.model.layers.49.self_attn.q_a_proj",
+            "language_model.model.layers.49.self_attn.q_b_proj",
+            "language_model.model.layers.5.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.5.self_attn.kv_b_proj",
+            "language_model.model.layers.5.self_attn.q_a_proj",
+            "language_model.model.layers.5.self_attn.q_b_proj",
+            "language_model.model.layers.50.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.50.self_attn.kv_b_proj",
+            "language_model.model.layers.50.self_attn.q_a_proj",
+            "language_model.model.layers.50.self_attn.q_b_proj",
+            "language_model.model.layers.51.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.51.self_attn.kv_b_proj",
+            "language_model.model.layers.51.self_attn.q_a_proj",
+            "language_model.model.layers.51.self_attn.q_b_proj",
+            "language_model.model.layers.52.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.52.self_attn.kv_b_proj",
+            "language_model.model.layers.52.self_attn.q_a_proj",
+            "language_model.model.layers.52.self_attn.q_b_proj",
+            "language_model.model.layers.53.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.53.self_attn.kv_b_proj",
+            "language_model.model.layers.53.self_attn.q_a_proj",
+            "language_model.model.layers.53.self_attn.q_b_proj",
+            "language_model.model.layers.54.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.54.self_attn.kv_b_proj",
+            "language_model.model.layers.54.self_attn.q_a_proj",
+            "language_model.model.layers.54.self_attn.q_b_proj",
+            "language_model.model.layers.55.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.55.self_attn.kv_b_proj",
+            "language_model.model.layers.55.self_attn.q_a_proj",
+            "language_model.model.layers.55.self_attn.q_b_proj",
+            "language_model.model.layers.56.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.56.self_attn.kv_b_proj",
+            "language_model.model.layers.56.self_attn.q_a_proj",
+            "language_model.model.layers.56.self_attn.q_b_proj",
+            "language_model.model.layers.57.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.57.self_attn.kv_b_proj",
+            "language_model.model.layers.57.self_attn.q_a_proj",
+            "language_model.model.layers.57.self_attn.q_b_proj",
+            "language_model.model.layers.58.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.58.self_attn.kv_b_proj",
+            "language_model.model.layers.58.self_attn.q_a_proj",
+            "language_model.model.layers.58.self_attn.q_b_proj",
+            "language_model.model.layers.59.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.59.self_attn.kv_b_proj",
+            "language_model.model.layers.59.self_attn.q_a_proj",
+            "language_model.model.layers.59.self_attn.q_b_proj",
+            "language_model.model.layers.6.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.6.self_attn.kv_b_proj",
+            "language_model.model.layers.6.self_attn.q_a_proj",
+            "language_model.model.layers.6.self_attn.q_b_proj",
+            "language_model.model.layers.60.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.60.self_attn.kv_b_proj",
+            "language_model.model.layers.60.self_attn.q_a_proj",
+            "language_model.model.layers.60.self_attn.q_b_proj",
+            "language_model.model.layers.7.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.7.self_attn.kv_b_proj",
+            "language_model.model.layers.7.self_attn.q_a_proj",
+            "language_model.model.layers.7.self_attn.q_b_proj",
+            "language_model.model.layers.8.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.8.self_attn.kv_b_proj",
+            "language_model.model.layers.8.self_attn.q_a_proj",
+            "language_model.model.layers.8.self_attn.q_b_proj",
+            "language_model.model.layers.9.self_attn.kv_a_proj_with_mqa",
+            "language_model.model.layers.9.self_attn.kv_b_proj",
+            "language_model.model.layers.9.self_attn.q_a_proj",
+            "language_model.model.layers.9.self_attn.q_b_proj",
+            "mm_projector*",
+            "vision_tower*"
+        ]
+    }
+}

kimi_k25_processor.py

@@ -0,0 +1,165 @@
+from transformers.feature_extraction_utils import BatchFeature
+from transformers.processing_utils import ProcessorMixin
+from transformers.utils import logging
+
+logger = logging.get_logger(__name__)
+
+
+class KimiK25Processor(ProcessorMixin):
+    r"""
+    Constructs a KimiK25 processor which wraps a KimiK25 image processor and a tokenizer into a single processor.
+
+    [`KimiK25Processor`] offers all the functionalities of [`KimiK25ImageProcessor`] and [`TikTokenTokenizer`]. See the
+    [`~KimiK25Processor.__call__`] and [`~KimiK25Processor.decode`] for more information.
+
+    Args:
+        image_processor ([`KimiK25ImageProcessor`], *optional*):
+            The image processor is a required input.
+        tokenizer ([`TikTokenTokenizer`], *optional*):
+            The tokenizer is a required input.
+        chat_template (`str`, *optional*): A Jinja template which will be used to convert lists of messages
+            in a chat into a tokenizable string.
+    """
+
+    attributes = ["image_processor", "tokenizer"]
+    valid_kwargs = ["chat_template"]
+    image_processor_class = "AutoImageProcessor"
+    tokenizer_class = "AutoTokenizer"
+
+    def __init__(
+        self,
+        image_processor=None,
+        tokenizer=None,
+        chat_template=None,
+        **kwargs,
+    ):
+        super().__init__(image_processor,
+                         tokenizer,
+                         chat_template=chat_template)
+        self.media_processor = image_processor
+        # A special temporal placeholder to be replaced by actual video placeholders
+        self.video_placeholder = "<|kimi_k25_video_placeholder|>"
+
+    def update_raw_text(self, text: str, video_prompts: list[str]) -> str:
+        # replace video prompt in text with video chunk prompts
+        video_count = text.count(self.video_placeholder)
+        if video_count == 0:
+            return text
+        assert video_count == len(video_prompts)
+        text_parts = text.split(self.video_placeholder)
+        assert len(text_parts) == len(video_prompts) + 1
+        text = "".join([
+            text_parts[i] + video_prompts[i] for i in range(len(video_prompts))
+        ])
+        text += text_parts[-1]
+        return text
+
+    def preprocess_medias(self, medias: list[dict]) -> list[dict]:
+        updated_medias = []
+        video_prompts = []
+        for media in medias:
+            if media['type'] == 'image':
+                updated_medias.append(media)
+            elif media['type'] == 'video':
+                video_chunks = self.media_processor.split_video_chunks(
+                    media['video'])
+                updated_medias.extend(video_chunks)
+                video_prompts.append("".join(
+                    [vc['prompt'] for vc in video_chunks]))
+            else:
+                raise ValueError(f"unsupported media type: {media['type']}")
+        return updated_medias, video_prompts
+
+    def __call__(self,
+                 messages: list[dict] = None,
+                 medias: list[dict] = None,
+                 text: str = None,
+                 return_tensors: str = "pt",
+                 **kwargs) -> BatchFeature:
+        """
+        Process multimodal inputs for Kimi-K2.5 model.
+
+        This processor accepts ordered messages and extracts both media and text in a single pass.
+        text will be automatically updated if video input detected in messages
+
+        Args:
+            messages: List of message dicts with 'role' and 'content' fields.
+                     If provided, medias and text will be extracted automatically.
+            medias: Pre-extracted list of media dicts. If None, extracted from messages.
+            text: Pre-formatted text string. If None, generated via apply_chat_template.
+            return_tensors: Format of returned tensors ('pt', 'np', 'tf'). Default: 'pt'.
+            **kwargs: Additional arguments passed to tokenizer.apply_chat_template.
+
+        Returns:
+            BatchFeature with fields: input_ids, attention_mask, pixel_values, grid_thws.
+        """
+        if messages is None and (medias is None or text is None):
+            raise ValueError(
+                "Provide either 'messages' or both 'medias' and 'text'")
+
+        if medias is not None and text is not None:
+            updated_medias, video_prompts = self.preprocess_medias(medias)
+            preprocessed = self.media_processor.preprocess(
+                updated_medias, return_tensors=return_tensors)
+            text = self.update_raw_text(text, video_prompts)
+            text_inputs = self.tokenizer(text, return_tensors=return_tensors)
+            return BatchFeature(data={**text_inputs, **preprocessed.data})
+
+        if medias is None:
+            medias = self._extract_medias_from_messages(messages)
+        updated_medias, video_prompts = self.preprocess_medias(medias)
+        preprocessed = self.media_processor.preprocess(
+            updated_medias, return_tensors=return_tensors)
+
+        # Generate text if not provided
+        if text is None:
+            text = self.tokenizer.apply_chat_template(messages, **kwargs)
+
+        text = self.update_raw_text(text, video_prompts)
+
+        text_inputs = self.tokenizer(text, return_tensors=return_tensors)
+        return BatchFeature(data={**text_inputs, **preprocessed.data})
+
+    @staticmethod
+    def _extract_medias_from_messages(messages: list[dict]) -> list[dict]:
+        """
+        Extract media items from messages in a single pass.
+        
+        This is an optimized version that processes messages only once.
+        Kept as internal method since external callers should use __call__.
+        """
+        medias = []
+        for msg in messages:
+            if msg['role'] != 'user' or not msg.get('content'):
+                continue
+
+            for content_part in msg['content']:
+                if not isinstance(content_part, dict):
+                    continue
+
+                content_type = content_part.get('type')
+                if content_type in ['video_url', 'video']:
+                    medias.append({
+                        'type': 'video',
+                        'video': content_part['video_url']['url'],
+                        'first_frame_timestamp': 0.0
+                    })
+                elif content_type in ['image_url', 'image']:
+                    medias.append({
+                        'type': 'image',
+                        'image': content_part['image_url'],
+                    })
+        return medias
+
+    def apply_chat_template(self, messages, **kwargs):
+        return self.tokenizer.apply_chat_template(messages, **kwargs)
+
+    def batch_decode(self, *args, **kwargs):
+        return self.tokenizer.batch_decode(*args, **kwargs)
+
+    def decode(self, *args, **kwargs):
+        return self.tokenizer.decode(*args, **kwargs)
+
+    @property
+    def model_input_names(self):
+        return ['input_ids', 'attention_mask', 'pixel_values', 'grid_thws']

kimi_k25_vision_processing.py

@@ -0,0 +1,251 @@
+"""Image processor class for Kimi-K2.5.
+"""
+
+import json
+from typing import Any, Dict, Optional, Union
+
+import numpy as np
+import torch
+from PIL import Image
+from transformers.image_processing_utils import (BaseImageProcessor,
+                                                 BatchFeature)
+from transformers.utils import TensorType
+
+from .media_utils import (MediaInput, VideoChunkInput, _to_tensor,
+                          ensure_media_type, get_video_meta, image_to_np,
+                          navit_patchify, navit_resize_image,
+                          navit_resize_video, normalize,
+                          real_sample_fps_and_max_num_frames, timestamp_as_str)
+
+try:
+    from mecord import VideoReader
+except ImportError:
+    VideoReader = None
+
+
+def resampling(video_bytes: bytes,
+               sample_indices: list[int],
+               key_indices=None,
+               frame_time_info=None,
+               num_threads=4) -> str:
+    video = VideoReader(video_bytes,
+                        num_threads=num_threads,
+                        frame_time_info=frame_time_info,
+                        key_indices=key_indices)
+    # extract target frames
+    frames = video[sample_indices]
+    frames = [Image.fromarray(frame) for frame in frames]
+    return frames
+
+
+class KimiK25VisionProcessor(BaseImageProcessor):
+    model_type = "kimi_k25"
+
+    def __init__(
+        self,
+        media_proc_cfg: dict,
+        **kwargs,
+    ):
+        super().__init__(**kwargs)
+        self.media_proc_cfg = media_proc_cfg
+        self.num_frames_per_chunk = media_proc_cfg[
+            'temporal_merge_kernel_size']
+
+    def media_tokens_calculator(self, media: MediaInput):
+        media = ensure_media_type(media)
+        ret = self.get_resize_config(media)
+        return ret['num_tokens']
+
+    @classmethod
+    def make_chunk_prompt(cls, timestamp_text: str) -> str:
+        return f"{timestamp_text}<|media_begin|>video<|media_content|><|media_pad|><|media_end|>"
+
+    def split_video_chunks(self,
+                           video_url: str | bytes) -> list[list[Image.Image]]:
+        # video_url should be base64 str or bytes
+        video_spec = get_video_meta(video_url)
+        sample_fps = min(self.media_proc_cfg['sample_fps'], video_spec.fps)
+        sampled_nframes = max(
+            round(video_spec.num_frames * sample_fps / video_spec.fps), 1)
+        frame_inds = np.linspace(0, video_spec.num_frames - 1,
+                                 sampled_nframes).round().astype(int)
+        frame_inds = frame_inds.tolist()
+        sampled_frame_ids = []
+        temporal_merge_kernel_size = self.media_proc_cfg[
+            "temporal_merge_kernel_size"]
+        num_chunks = 0
+        chunk_timestamp = []
+        for i in range(0, len(frame_inds), temporal_merge_kernel_size):
+            sampled_frame_ids.extend(frame_inds[i:i +
+                                                temporal_merge_kernel_size])
+            start_time = frame_inds[i] / float(video_spec.fps)
+            timestamp_text = timestamp_as_str(
+                start_time, self.media_proc_cfg["timestamp_mode"])
+            chunk_timestamp.append(timestamp_text)
+            num_chunks += 1
+
+        sampled_frames = resampling(video_url, sampled_frame_ids)
+        chunks = []
+        for chunk_id in range(num_chunks):
+            chunk = sampled_frames[chunk_id *
+                                   temporal_merge_kernel_size:(chunk_id + 1) *
+                                   temporal_merge_kernel_size]
+            chunks.append(
+                VideoChunkInput(type="video_chunk",
+                                video_chunk=chunk,
+                                prompt=self.make_chunk_prompt(
+                                    chunk_timestamp[chunk_id])))
+        return chunks
+
+    def get_resize_config(self, media_input: MediaInput) -> dict:
+        if media_input['type'] == 'image':
+            w, h = media_input['image'].size
+            ret = navit_resize_image(
+                w, h, self.media_proc_cfg['patch_size'],
+                self.media_proc_cfg['merge_kernel_size'],
+                self.media_proc_cfg['in_patch_limit'],
+                self.media_proc_cfg['patch_limit_on_one_side'],
+                self.media_proc_cfg['fixed_output_tokens'])
+            return ret
+        elif media_input['type'] == 'video_chunk':
+            frame = media_input['video_chunk'][0]
+            width, height = frame.size
+            num_frames = len(media_input["video_chunk"])
+            fps = 1.0
+
+            sample_fps, max_num_frames_each_video = real_sample_fps_and_max_num_frames(
+                media_input["type"],
+                self.media_proc_cfg['sample_fps'],
+                self.media_proc_cfg['max_num_frames_each_video'],
+            )
+
+            in_patch_limit_each_frame = self.media_proc_cfg[
+                'in_patch_limit_each_frame']
+            if in_patch_limit_each_frame is None:
+                in_patch_limit_each_frame = self.media_proc_cfg[
+                    'in_patch_limit']
+
+            ret = navit_resize_video(
+                width,
+                height,
+                num_frames,
+                fps,
+                sample_fps,
+                self.media_proc_cfg['patch_size'],
+                self.media_proc_cfg['merge_kernel_size'],
+                in_patch_limit_each_frame,
+                self.media_proc_cfg['patch_limit_on_one_side'],
+                self.media_proc_cfg['in_patch_limit_video'],
+                max_num_frames_each_video,
+                self.media_proc_cfg['fixed_output_tokens'],
+            )
+            return ret
+        else:
+            raise ValueError("Unsupported type: {}".format(
+                media_input['type']))
+
+    def resize_image(self, image: Image.Image, new_width: int, new_height: int,
+                     pad_width: int, pad_height: int) -> np.ndarray:
+        image_np = image_to_np(image, (new_width, new_height), "resize")
+        image_np = np.pad(
+            image_np,
+            ((0, pad_height), (0, pad_width), (0, 0)),
+            mode="constant",
+            constant_values=0,
+        )
+        return image_np
+
+    def preprocess(
+        self,
+        medias: list[MediaInput],
+        return_tensors: Optional[Union[str, TensorType]] = None,
+    ) -> BatchFeature:
+        """
+        Preprocess a atom vision input (images/video_chunk) into model-ready tensors.
+        
+        Args:
+            medias: List of MediaInput.
+            return_tensors: Desired output format ('pt', 'np', 'tf', or None).
+        
+        Returns:
+            BatchFeature containing 'pixel_values' and 'grid_thws' tensors.
+        """
+        if not isinstance(medias, list):
+            medias = [medias]
+        if medias:
+            pixel_values = []
+            for item in medias:
+                item = ensure_media_type(item)
+                resize_config = self.get_resize_config(item)
+                new_width, new_height, pad_width, pad_height = resize_config[
+                    'new_width'], resize_config['new_height'], resize_config[
+                        'pad_width'], resize_config['pad_height']
+                if item['type'] == 'image':
+                    image = item['image']
+                    image_np = self.resize_image(image, new_width, new_height,
+                                                 pad_width, pad_height)
+                    pixel_values.append(np.expand_dims(image_np, axis=0))
+                elif item['type'] == 'video_chunk':
+                    pixels = []
+                    for frame in item['video_chunk']:
+                        frame_np = self.resize_image(frame, new_width,
+                                                     new_height, pad_width,
+                                                     pad_height)
+                        pixels.append(frame_np)
+                    pixel_values.append(np.stack(pixels, axis=0))
+                else:
+                    raise ValueError("Unsupported type: {}".format(
+                        item['type']))
+            normalized_pixel_values = []
+            image_std_inv = 1.0 / np.array(self.media_proc_cfg['image_std'])
+            image_mean = np.array(self.media_proc_cfg['image_mean'])
+            for pixels in pixel_values:
+                pixels = normalize(pixels, image_mean, image_std_inv)
+                pixels_and_thw = navit_patchify(
+                    pixels,
+                    self.media_proc_cfg['patch_size'],
+                )
+                normalized_pixel_values.append(pixels_and_thw)
+
+            pixel_values = torch.cat([
+                _to_tensor(pixel_value['pixel_values'])
+                for pixel_value in normalized_pixel_values
+            ])
+            grid_thws = torch.cat([
+                _to_tensor(pixel_value['grid_thw'],
+                           dtype=torch.int64).unsqueeze(0)
+                for pixel_value in normalized_pixel_values
+            ])
+
+            data = {
+                'pixel_values': pixel_values,
+                'grid_thws': grid_thws,
+            }
+
+        else:
+            data = {}
+
+        return BatchFeature(data=data, tensor_type=return_tensors)
+
+    def __repr__(self):
+        return f"KimiK25VisionProcessor(media_proc_cfg={self.media_proc_cfg})"
+
+    def to_dict(self) -> Dict[str, Any]:
+        output = super().to_dict()
+        output["media_proc_cfg"] = self.media_proc_cfg
+        if "media_processor" in output:
+            del output["media_processor"]
+        return output
+
+    @classmethod
+    def from_dict(cls, config_dict: Dict[str, Any], **kwargs):
+        config = config_dict.copy()
+        media_proc_cfg = config.pop("media_proc_cfg", {})
+        return cls(media_proc_cfg=media_proc_cfg, **config, **kwargs)
+
+    def to_json_string(self):
+        dictionary = self.to_dict()
+        for key, value in dictionary.items():
+            if hasattr(value, 'tolist'):
+                dictionary[key] = value.tolist()
+        return json.dumps(dictionary, indent=2, sort_keys=True) + "\n"

media_utils.py

@@ -0,0 +1,368 @@
+import base64
+import io
+import math
+import os
+from datetime import datetime, timezone
+from typing import List, Literal, Optional, TypedDict
+
+import numpy as np
+from PIL import Image
+from pydantic import BaseModel, Field
+
+try:
+    from mecord import VideoReader
+except ImportError:
+    VideoReader = None
+
+
+class VideoSpec(BaseModel):
+    media_type: str = Literal['video']
+    height: int = Field(..., gt=0, description="video frame height")
+    width: int = Field(..., gt=0, description="video frame width")
+    num_frames: int = Field(..., gt=0, description="num frames")
+    fps: float = Field(..., gt=0, description="average fps")
+
+    # optional, help to accelerate video reading
+    key_indices: list[int] = Field(None, description="key indices")
+    frame_time_info: dict = Field(None, description="frame time info")
+
+
+class ImageInput(TypedDict):
+    type: Literal['image']
+    image: Image.Image
+
+
+class VideoChunkInput(TypedDict):
+    type: Literal['video_chunk']
+    video_chunk: List[Image.Image]
+    prompt: Optional[str] = None
+
+
+MediaInput = ImageInput | VideoChunkInput
+
+
+def get_video_meta(video_src: bytes | str | os.PathLike,
+                   accurate: bool = True) -> dict:
+    """Get the dimensions of a video."""
+    if isinstance(video_src, os.PathLike):
+        video_src = str(video_src)
+    # if b64 string, decode to bytes
+    if isinstance(video_src,
+                  str) and video_src.startswith('data:video/mp4;base64,'):
+        video_src = base64.b64decode(video_src.split(',')[1])
+    video = VideoReader(video_src, auto_init=accurate, num_threads=1)
+    assert video.num_frames > 0, "Invalid video format."
+    assert video.original_width > 0 and video.original_height > 0, (
+        "Invalid video format.")
+    assert video.avg_fps > 0, "Invalid video format."
+    return VideoSpec(media_type='video',
+                     height=video.original_height,
+                     width=video.original_width,
+                     num_frames=video.num_frames,
+                     fps=video.avg_fps,
+                     key_indices=video.key_indices,
+                     frame_time_info=video.frame_time_info)
+
+
+def timestamp_as_str(timestamp: float,
+                     timestamp_mode: str = "hh:mm:ss.fff") -> str:
+    """Convert a timestamp to a string in the format of HH:MM:SS.mmm."""
+    if timestamp_mode == "hh:mm:ss.fff":
+        return (datetime.fromtimestamp(timestamp,
+                                       tz=timezone.utc).strftime("%H:%M:%S") +
+                f".{int((timestamp % 1) * 1000):03d}")
+    elif timestamp_mode == "mm:ss.fff":
+        return (datetime.fromtimestamp(timestamp,
+                                       tz=timezone.utc).strftime("%M:%S") +
+                f".{int((timestamp % 1) * 1000):03d}")
+    elif timestamp_mode == "mm:ss":
+        return datetime.fromtimestamp(timestamp,
+                                      tz=timezone.utc).strftime("%M:%S")
+    else:
+        raise ValueError(f"Invalid timestamp mode: {timestamp_mode}")
+
+
+def navit_resize_image(
+    width: int,
+    height: int,
+    patch_size: int,
+    merge_kernel_size: int,
+    in_patch_limit: int,
+    patch_limit_on_one_side: int,
+    fixed_output_tokens: int | None,
+):
+    # Apply the patch limits.
+    s1 = math.sqrt(
+        in_patch_limit /
+        (max(1.0, width // patch_size) * max(1.0, height // patch_size)))
+    s2 = patch_limit_on_one_side * patch_size / width
+    s3 = patch_limit_on_one_side * patch_size / height
+    scale = min(1.0, s1, s2, s3)
+    new_w, new_h = max(1, int(width * scale)), max(1, int(height * scale))
+    new_w = min(new_w, patch_limit_on_one_side * patch_size)
+    new_h = min(new_h, patch_limit_on_one_side * patch_size)
+
+    # Calculate the padding to make the height and width divisible by the merge kernel size and patch size.
+    factor = merge_kernel_size * patch_size
+
+    pad_height = (factor - new_h % factor) % factor
+    pad_width = (factor - new_w % factor) % factor
+
+    if fixed_output_tokens is not None:
+        num_tokens = fixed_output_tokens
+    else:
+        # Calculate new dimensions after padding and patching
+        token_height = (new_h + pad_height) // factor
+        token_width = (new_w + pad_width) // factor
+
+        assert token_height * merge_kernel_size <= patch_limit_on_one_side, (
+            f"token_height {token_height} * merge_kernel_size {merge_kernel_size} > patch_limit_on_one_side {patch_limit_on_one_side}"
+        )
+        assert token_width * merge_kernel_size <= patch_limit_on_one_side, (
+            f"token_width {token_width} * merge_kernel_size {merge_kernel_size} > patch_limit_on_one_side {patch_limit_on_one_side}"
+        )
+
+        num_tokens = token_height * token_width
+    return {
+        "num_tokens": num_tokens,
+        "new_width": new_w,
+        "new_height": new_h,
+        "pad_width": pad_width,
+        "pad_height": pad_height,
+        "sampled_nframes": 1,
+    }
+
+
+def navit_resize_video(
+    width: int,
+    height: int,
+    nframes: int,
+    avg_fps: float,
+    sample_fps: float,
+    patch_size: int,
+    merge_kernel_size: int,
+    in_patch_limit_each_frame: int,
+    patch_limit_on_one_side: int,
+    in_patch_limit_total: int | None,
+    max_num_frames_each_video: int | None,
+    fixed_output_tokens_each_frame: int | None,
+):
+    sample_fps = min(sample_fps, avg_fps)
+    # Calculate the number of frames to sample based on target FPS
+    sampled_nframes = max(round(nframes * sample_fps / avg_fps), 1)
+    if max_num_frames_each_video is not None:
+        sampled_nframes = min(sampled_nframes, max_num_frames_each_video)
+
+    if in_patch_limit_total is not None:
+        in_patch_limit_each_frame = min(
+            round(in_patch_limit_total / sampled_nframes),
+            in_patch_limit_each_frame)
+
+    ret = navit_resize_image(
+        width,
+        height,
+        patch_size,
+        merge_kernel_size,
+        in_patch_limit_each_frame,
+        patch_limit_on_one_side,
+        fixed_output_tokens_each_frame,
+    )
+    ret["sampled_nframes"] = sampled_nframes
+    return ret
+
+
+def real_sample_fps_and_max_num_frames(
+    type_name: Literal["video", "video_chunk"],
+    sample_fps: float,
+    max_num_frames_each_video: int | None,
+) -> tuple[int, int | None]:
+    if type_name == "video":
+        return sample_fps, max_num_frames_each_video
+    elif type_name == "video_chunk":
+        max_num_frames_each_video = None
+        sample_fps = math.inf
+        return sample_fps, max_num_frames_each_video
+    else:
+        return math.inf, None
+
+
+def _to_pil(data: str | bytes):
+    if isinstance(data, Image.Image):
+
+        return data.convert("RGB")
+    elif isinstance(data, str):
+        if data.startswith("data:"):
+            raw_base64 = data.split(",")[1]
+            return Image.open(io.BytesIO(
+                base64.b64decode(raw_base64))).convert("RGB")
+        else:
+            return Image.open(data).convert("RGB")
+    elif isinstance(data, bytes):
+        return Image.open(io.BytesIO(data)).convert("RGB")
+    else:
+        raise ValueError(f"Unsupported data type: {type(data)}")
+
+
+def ensure_media_type(media: MediaInput) -> MediaInput:
+    if media['type'] == 'image':
+        media['image'] = _to_pil(media['image'])
+        return media
+    elif media['type'] == 'video_chunk':
+        media['video_chunk'] = [
+            _to_pil(frame) for frame in media['video_chunk']
+        ]
+        return media
+    else:
+        raise ValueError(f"Unsupported media type: {media['type']}")
+
+
+def image_to_np(
+    image: Image.Image,
+    resize_to: tuple[int, int] | None = None,
+    mode: str = "resize",
+    raise_error_for_ill_resize: bool = True,
+) -> np.ndarray:
+    """Convert an image to a numpy array.
+
+    Args:
+        content: The image to convert.
+        resize_to: The size to resize the image to.
+        mode: The mode to resize the image to.
+        raise_error_for_ill_resize: Whether to raise an error for ill-sized resize.
+
+    Returns:
+        A numpy array.
+    """
+    assert isinstance(image, Image.Image), "image must be a PIL Image"
+    if resize_to is not None:
+        if mode == "resize":
+            image = image.resize(resize_to, resample=Image.Resampling.BICUBIC)
+
+        elif mode == "rescale_and_pad_to_center":
+            scale = min(resize_to[0] / image.width,
+                        resize_to[1] / image.height, 1.0)
+            new_width = round(image.width * scale)
+            new_height = round(image.height * scale)
+            if new_width == 0 or new_height == 0:
+                if raise_error_for_ill_resize:
+                    raise ValueError(
+                        f"Invalid resize to: {resize_to}, from image size: {image.size}"
+                    )
+                else:
+                    return np.zeros((resize_to[1], resize_to[0], 3),
+                                    dtype=np.uint8)
+
+            image = image.resize((new_width, new_height),
+                                 resample=Image.Resampling.BICUBIC)
+            padding_left = (resize_to[0] - new_width) // 2
+            padding_right = resize_to[0] - new_width - padding_left
+            padding_top = (resize_to[1] - new_height) // 2
+            padding_bottom = resize_to[1] - new_height - padding_top
+            image = np.asarray(image)
+            image = np.pad(
+                image,
+                ((padding_top, padding_bottom), (padding_left, padding_right),
+                 (0, 0)),
+                mode="constant",
+                constant_values=0,
+            )
+            assert image.shape == (resize_to[1], resize_to[0], 3)
+
+        elif mode == "rescale_and_pad_to_rightbottom":
+            scale = min(resize_to[0] / image.width,
+                        resize_to[1] / image.height, 1.0)
+            new_width = round(image.width * scale)
+            new_height = round(image.height * scale)
+            if new_width == 0 or new_height == 0:
+                if raise_error_for_ill_resize:
+                    raise ValueError(
+                        f"Invalid resize to: {resize_to}, from image size: {image.size}"
+                    )
+                else:
+                    return np.zeros((resize_to[1], resize_to[0], 3),
+                                    dtype=np.uint8)
+
+            image = image.resize((new_width, new_height),
+                                 resample=Image.Resampling.BICUBIC)
+            padding_right = resize_to[0] - new_width
+            padding_bottom = resize_to[1] - new_height
+            image = np.asarray(image)
+            image = np.pad(
+                image,
+                ((0, padding_bottom), (0, padding_right), (0, 0)),
+                mode="constant",
+                constant_values=0,
+            )
+            assert image.shape == (resize_to[1], resize_to[0], 3)
+
+        else:
+            raise ValueError(f"Invalid mode: {mode}")
+
+    if isinstance(image, Image.Image):
+        return np.asarray(image)
+    else:
+        return image
+
+
+def navit_patchify(pixel_values: np.ndarray,
+                   patch_size: int) -> dict[str, np.ndarray]:
+    """Reshape the pixel values to a navit shape.
+
+    Args:
+        pixel_values: np.ndarray, shape (t, h, w, c)
+        patch_size: int
+
+    Returns:
+        dict[str, np.ndarray]
+        - patches: np.ndarray, shape (t * h//patch_size * w//patch_size, c, patch_size, patch_size)
+        - grid_thw: np.ndarray, (t, h//patch_size, w//patch_size)
+    """
+    T, H, W, C = pixel_values.shape
+    assert C == 3, "pixel_values must have 3 channels"
+
+    patches = pixel_values.reshape(T, H // patch_size, patch_size,
+                                   W // patch_size, patch_size, C)
+    # (T, H//patch_size, W//patch_size, C, patch_size, patch_size)
+    patches = patches.transpose(0, 1, 3, 5, 2, 4)
+    patches = patches.reshape(-1, C, patch_size, patch_size)
+    grid_thw = np.array([T, H // patch_size, W // patch_size])
+    return {"pixel_values": patches, "grid_thw": grid_thw}
+
+
+def normalize(x: np.ndarray,
+              mean,
+              std_inv,
+              pixels_dtype: np.dtype = np.float32) -> np.ndarray:
+    """Normalize the image.
+
+    Args:
+        x: The image to normalize. The shape is (..., 3). The dtype is uint8. The range is [0, 255].
+        mean: The mean of the image.
+        std_inv: The inverse of the std of the image.
+        pixels_dtype: The dtype of the image.
+    Returns:
+        The normalized image. The shape is (..., 3). The dtype is determined by the pixels_dtype.
+    """
+    x = (x / 255.0).astype(pixels_dtype)
+    x -= mean
+    x *= std_inv
+    return x
+
+
+def _to_tensor(data, **kwargs):
+    import torch
+
+    if isinstance(data, np.ndarray):
+        return torch.from_numpy(data).to(**kwargs)
+    elif isinstance(data, torch.Tensor):
+        return data.to(**kwargs)
+    elif isinstance(data, list):
+        return [_to_tensor(item, **kwargs) for item in data]
+    elif isinstance(data, tuple):
+        return tuple(_to_tensor(item, **kwargs) for item in data)
+    elif isinstance(data, dict):
+        return {k: _to_tensor(v, **kwargs) for k, v in data.items()}
+    elif data is None:
+        return None
+    else:
+        raise ValueError(f"Unsupported data type: {type(data)}")

model-00001-of-00214.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:198956be53f7468823d2e789daa445a78e8a6dfd77ef99fab3391d1717a898d2
+size 4989823800

model-00002-of-00214.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:05956705bdaab932a41ffac0546a4122a4e82d72c12e01477d49e7562437a2ac
+size 4995903192

model-00003-of-00214.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f9abb20f577b1cdf214ecd3e8729d3b48fc67889711ffc3048588054924a9757
+size 4995903656

model-00004-of-00214.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0056ed3536a45b6b4f3d4c4ae5b697e315a87996cb4eb05e03011e390d673952
+size 4995903656

model-00005-of-00214.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:822741fd8f15f58cd3f74e9bbf83d41629ae9df35d59a738794085d4180b4294
+size 4995411128

model-00006-of-00214.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1d8f8aeb07ba5e82f4dedf92921801efe273a8394ed9af7c09b8c5ee94132b1f
+size 4995903496

model-00007-of-00214.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:90ae184d62f3e775f7937f9e7688c224c19c1562488804e6b5bd7f874a2f6255
+size 4995903656

model-00008-of-00214.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2c919d6c6e768403f4673b3fe60073adbe4537188e5e6b5913db97f8aff0fffb
+size 4995411480

model-00009-of-00214.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7578de9d665f5b23ae47053921234a52f46188a7371c665af15219cb2102e4a9
+size 4995903152

model-00010-of-00214.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9961e676f0cff04872ccce764e82c45917117ee875ed2b7a5a1859480f9b68cc
+size 4995903656

model-00011-of-00214.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:86997530dccb7db8540b17280373397dbcbab005cfd5d88b13a935c07906a1c5
+size 4995903656

model-00012-of-00214.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f7f1e0eb108b4d036fbb9cf399f063371302c222708649c0baa9914e37632e73
+size 4995411168

model-00013-of-00214.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8b91a51b56669ef998b93ecfeee26a321915babf8cb31e686a0bbb2da5763bf5
+size 4995903456

model-00014-of-00214.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:456f8c066ed456cd7d8130c965829b3de1e894540d768bacc24209e65fb422a9
+size 4995903656

model-00015-of-00214.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a2fe7d9b74eccf9c4e08cb50adc6dc86d38da6d9a03ede8ce5bd537d9aee8476
+size 4995411528

model-00016-of-00214.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:16048e780ed115e85d61317c11862f19190b1f8e96253407a203c69bf104e102
+size 4995903104

model-00017-of-00214.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4085a09f53da3b92c2cc68012b402ac08b087f034df30f7be70d4fc345f3706e
+size 4995903656

model-00018-of-00214.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:663fef64245d6b010c037adbe562d98cb9c5c1850a82eb8bc854f2441d9a4d49
+size 4995903656

model-00019-of-00214.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:656709286f727790e624a32ac226392bd1553d8393da1c33ffd3e845c3f10bd2
+size 4995411208

model-00020-of-00214.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ed517bd9e95b054dfbb212d9719693ce45b5c9b17827d4c31bcb3c0d11c3ed82
+size 4995903416

model-00021-of-00214.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:78e16d261cfcc5e3b1f6182840297ec53b3a72ee60aaeff8247bfad70bc51c4b
+size 4995903656

model-00022-of-00214.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4824c2ff298c8aa0253cd97d1255f85d76e39d401fdd7aec0393b13d9eaf7e7e
+size 4995411608

model-00023-of-00214.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c65e44477ef5d0d47935e4ef38dda820713cb4d4eb414090726f070fdd0bfdec
+size 4995903024

model-00024-of-00214.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:179a2becbd88a81987a7d5a17dd5691460e95ad84ec5abe55f9a18f8fa88daff
+size 4995903656

model-00025-of-00214.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ff51369231e664dcb3e9157b189d95c1cd5d4b17bdd838a0dac2d3e393f5292e
+size 4995903656

model-00026-of-00214.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3bacf76fbcaefbed6d2b4d1b44d7101466264ceb2109106007b6cb841412ee89
+size 4995411248

model-00027-of-00214.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6f84904610e7537854bc72ae3c3d2f9b16060655d310db8e731573414f777284
+size 4995903376

model-00028-of-00214.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:14aeaf2778bd9430a5e17ed52185a609e0dd9da5ede7cd09d6200e2e360234a7
+size 4995903656

model-00029-of-00214.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e440901f5b2fa36acfa8dc73b7df789ee0372c755b42d1faaa55e654fb5ae10f
+size 4995903656

model-00030-of-00214.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:034a669bc8e23554b860e041654232932b6ef7bccd4162f24425a0691581945d
+size 4995410976

model-00031-of-00214.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b5a085e62bc5e5c42ae68a0a158d7eb12e578fd08b3a00c0305d22c7f4d73274
+size 4995903656

model-00032-of-00214.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1a6ceaf762c646112a11a64570734fb2a3ce4a852b9d525e6e21b13c165f65bb
+size 4995903656

model-00033-of-00214.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4c0432d14e4c706f3525a5b0039d796ebc390e0b6ddc766b812ea51ba46bce5e
+size 4995411584

model-00034-of-00214.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cc1d8b6d80e2e034a24c8c4ce5956c9d04dcd604cf7ee6a5470f642cb3164d97
+size 4995904000

model-00035-of-00214.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e43739f223a970dcb257646b7f956c8cf848e7d0d4637ca1faaa5162fe2b9ea4
+size 4995904320

model-00036-of-00214.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5b7a20a5e8630c6d93f6f185774c27de85c2ab946bbe728f07ee9fd081036941
+size 4995904320

model-00037-of-00214.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7100a4de949bfc52ee2882d4a36e8a932b6624b901f6430eb467025d8c1c6a55
+size 4995411632

model-00038-of-00214.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:698a8dc08809edda2c8b98022028a9a5a4ee2bb27d2d7d94848e34f97ed4978e
+size 4995904312

model-00039-of-00214.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dcdc2727229ee0d37eccbc8d2a147ea3c0bd75d3f3d0ce13f7d53e3c5016aed6
+size 4995904320