Duplicate from baseten/GLM-5.2-Vision-NVFP4

Co-authored-by: Harry Partridge <harrypart@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1,37 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+model.safetensors.index.json filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,178 @@
+---
+license: mit
+library_name: sglang
+pipeline_tag: image-text-to-text
+tags:
+  - multimodal
+  - vision-language
+  - glm
+  - sglang
+base_model:
+  - zai-org/GLM-5.2
+---
+
+# GLM-5.2-Vision (NVFP4)
+
+**GLM-5.2 with sight.** A vision-language model that bolts the MoonViT vision encoder from
+[Kimi-K2.6](https://huggingface.co/moonshotai/Kimi-K2.6) onto
+[GLM-5.2](https://huggingface.co/zai-org/GLM-5.2) through a trained PatchMerger projector.
+
+GLM-5.2 is a strong open reasoning model with no vision input. This checkpoint adds it,
+without touching a single GLM weight: the text backbone and the vision tower are both frozen
+and byte-identical to their upstream releases. The only newly-trained parameters are the
+**49.5M-parameter projector** that maps MoonViT's 1152-dim patch embeddings into GLM's 6144-dim
+token space.
+
+| Component | Detail |
+|---|---|
+| Text backbone | GLM-5.2 (744B total / A40B active, MoE + MLA + DSA sparse attention) — **frozen** |
+| Vision tower | MoonViT-3d from Kimi-K2.6, 27 layers, 1152-dim — **frozen** |
+| Projector | PatchMerger MLP (`pre_norm → linear_1 → GELU → linear_2`), 1152→4608→6144 — **trained** |
+| Text weights | NVFP4, from [`nvidia/GLM-5.2-NVFP4`](https://huggingface.co/nvidia/GLM-5.2-NVFP4) |
+| Size | ~466 GB |
+| Hardware | 8×B200, or 4×B200 at 256k context — **Blackwell only** |
+| Image tokens | up to 4096 per image (16384 MoonViT patches, 2×2 merge) |
+| Max context | 1048576 (1M tokens) |
+
+## Quickstart
+
+SGLang needs a small out-of-tree plugin because `Glm5vForConditionalGeneration` is not yet an
+upstream architecture. It ships inside this repo, so there is nothing else to clone:
+
+```bash
+uvx --from huggingface-hub hf download baseten/GLM-5.2-Vision-NVFP4 \
+  --include 'plugins/*' --local-dir ./glm5v
+uv pip install ./glm5v/plugins
+```
+
+### SGLang
+
+```bash
+export SGLANG_EXTERNAL_MODEL_PACKAGE=sglang_glm5v
+export SGLANG_EXTERNAL_MM_PROCESSOR_PACKAGE=sglang_glm5v
+export SGLANG_EXTERNAL_MM_MODEL_ARCH=Glm5vForConditionalGeneration
+python -m sglang_glm5v.patch
+```
+
+#### 8×B200 — full 1M context
+
+```bash
+python -m sglang.launch_server \
+  --model-path baseten/GLM-5.2-Vision-NVFP4 --trust-remote-code \
+  --tp-size 8 \
+  --quantization modelopt_fp4 \
+  --disable-shared-experts-fusion --disable-flashinfer-autotune \
+  --attention-backend dsa --mm-attention-backend sdpa \
+  --kv-cache-dtype fp8_e4m3 --page-size 64 \
+  --mem-fraction-static 0.85 \
+  --context-length 1048576 \
+  --reasoning-parser glm45 --tool-call-parser glm47 \
+  --served-model-name glm-5.2-vision \
+  --port 30000
+```
+
+#### 4×B200 — 256k context
+
+Same command with `--tp-size 4`, a higher memory fraction, and a smaller context:
+
+```bash
+python -m sglang.launch_server \
+  --model-path baseten/GLM-5.2-Vision-NVFP4 --trust-remote-code \
+  --tp-size 4 \
+  --quantization modelopt_fp4 \
+  --disable-shared-experts-fusion --disable-flashinfer-autotune \
+  --attention-backend dsa --mm-attention-backend sdpa \
+  --kv-cache-dtype fp8_e4m3 --page-size 64 \
+  --mem-fraction-static 0.90 \
+  --context-length 262144 \
+  --reasoning-parser glm45 --tool-call-parser glm47 \
+  --served-model-name glm-5.2-vision \
+  --port 30000
+```
+
+### Query it
+
+Standard OpenAI multimodal messages deliver the image as `image_url`:
+
+```python
+from openai import OpenAI
+
+client = OpenAI(base_url="http://localhost:30000/v1", api_key="none")
+r = client.chat.completions.create(
+    model="glm-5.2-vision",
+    messages=[{"role": "user", "content": [
+        {"type": "image_url", "image_url": {"url": "https://ultralytics.com/images/bus.jpg"}},
+        {"type": "text", "text": "Describe this image in detail."},
+    ]}],
+    temperature=1.0, top_p=0.95, max_tokens=512,
+)
+print(r.choices[0].message.content)
+```
+
+GLM-5.2 is a reasoning model: with `--reasoning-parser glm45`, the chain of thought arrives in
+`message.reasoning_content` and the answer in `message.content`.
+
+## Deploy on Baseten
+
+The repository includes ready-to-push [Truss](https://truss.baseten.co) configs. The only
+credential you need is an API key for your own Baseten account; no Hugging Face token or
+pre-created Baseten secret is required.
+
+1. Install [`uv`](https://docs.astral.sh/uv/getting-started/installation/) and create a Baseten API key.
+2. Export the key, download the small Truss directory, and deploy one of the two configurations:
+
+```bash
+export BASETEN_API_KEY="your-baseten-api-key"
+uvx truss login --api-key "$BASETEN_API_KEY" --remote baseten --non-interactive
+
+uvx --from huggingface-hub hf download baseten/GLM-5.2-Vision-NVFP4 \
+  --include 'truss/*' --local-dir ./glm5v
+cd glm5v/truss
+
+# Recommended starting point: 4×B200 and 256k context.
+uvx truss push --remote baseten --config config_nvfp4_4gpu.yaml --wait --output json
+
+# Or use 8×B200 for the full 1M-token context.
+# uvx truss push --remote baseten --config config_nvfp4.yaml --wait --output json
+```
+
+The command creates a new model and published deployment in your Baseten account and prints
+JSON containing `model_id`, `model_version_id`, `predict_url`, and `logs_url`. It does not
+promote the deployment to production.
+
+Set `PREDICT_URL` to the returned `predict_url`, then query the model:
+
+```bash
+export PREDICT_URL="https://model-...api.baseten.co/deployment/.../predict"
+
+curl -fsS "$PREDICT_URL" \
+  -H "Authorization: Api-Key $BASETEN_API_KEY" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "glm-5.2-vision",
+    "messages": [{
+      "role": "user",
+      "content": [
+        {"type": "image_url", "image_url": {"url": "https://ultralytics.com/images/bus.jpg"}},
+        {"type": "text", "text": "Describe this image in detail."}
+      ]
+    }],
+    "max_tokens": 512,
+    "temperature": 1.0,
+    "top_p": 0.95
+  }'
+```
+
+The first deployment downloads about 466 GB of weights and initializes SGLang, so startup can
+take several minutes.
+
+## License
+
+MIT, following both parents: GLM-5.2 (MIT) and Kimi-K2.6 (Modified MIT). The projector weights
+are released under MIT. Redistributed upstream weights remain under their original terms.
+
+## Acknowledgements
+
+Built on [Z.ai](https://huggingface.co/zai-org)'s GLM-5.2 and
+[Moonshot AI](https://huggingface.co/moonshotai)'s Kimi-K2.6. Neither team was involved in this
+work; please do not direct issues with this checkpoint to them.

chat_template.jinja

@@ -0,0 +1,121 @@
+[gMASK]<sop>
+{%- set effective_reasoning_effort = 'high' if reasoning_effort is defined and reasoning_effort == 'high' else 'max' -%}
+{%- if (enable_thinking is not defined or enable_thinking) and effective_reasoning_effort is not none -%}<|system|>Reasoning Effort: {{ effective_reasoning_effort | capitalize }}{%- endif -%}
+{%- if tools -%}
+{%- macro tool_to_json(tool) -%}
+    {%- set ns_tool = namespace(first=true) -%}
+    {{ '{' -}}
+    {%- for k, v in tool.items() -%}
+        {%- if k != 'defer_loading' and k != 'strict' -%}
+            {%- if not ns_tool.first -%}{{- ', ' -}}{%- endif -%}
+            {%- set ns_tool.first = false -%}
+            "{{ k }}": {{ v | tojson(ensure_ascii=False) }}
+        {%- endif -%}
+    {%- endfor -%}
+    {{- '}' -}}
+{%- endmacro -%}
+<|system|>
+# Tools
+
+You may call one or more functions to assist with the user query.
+
+You are provided with function signatures within <tools></tools> XML tags:
+<tools>
+{% for tool in tools %}
+{%- if 'function' in tool -%}
+    {%- set tool = tool['function'] -%}
+{%- endif -%}
+{% if tool.defer_loading is not defined or not tool.defer_loading %}
+{{ tool_to_json(tool) }}
+{% endif %}
+{% endfor %}
+</tools>
+
+For each function call, output the function name and arguments within the following XML format:
+<tool_call>{function-name}<arg_key>{arg-key-1}</arg_key><arg_value>{arg-value-1}</arg_value><arg_key>{arg-key-2}</arg_key><arg_value>{arg-value-2}</arg_value>...</tool_call>{%- endif -%}
+{%- macro visible_text(content) -%}
+    {%- if content is string -%}
+        {{- content }}
+    {%- elif content is iterable and content is not mapping -%}
+        {%- for item in content -%}
+            {%- if item is mapping and item.type == 'text' -%}
+                {{- item.text }}
+            {%- elif item is string -%}
+                {{- item }}
+            {%- elif item is mapping and item.type in ['image', 'image_url'] -%}
+                {{- '<|begin_of_image|><|image|><|end_of_image|>' }}
+            {%- elif item is mapping and item.type in ['video', 'video_url', 'audio', 'audio_url', 'input_audio'] -%}
+                {%- set media_type = item.type | replace('_url', '') | replace('input_', '') -%}
+                {{- "<reminder>You are unable to process this " ~ media_type ~ " because you don't have multi-modal input ability. Try different methods.</reminder>" }}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- else -%}
+        {{- content }}
+    {%- endif -%}
+{%- endmacro -%}
+{%- set ns = namespace(last_user_index=-1) -%}
+{%- for m in messages %}
+    {%- if m.role == 'user' %}
+        {%- set ns.last_user_index = loop.index0 -%}
+    {%- endif %}
+{%- endfor %}
+{%- for m in messages -%}
+{%- if m.role == 'user' -%}<|user|>{{ visible_text(m.content) }}
+{%- elif m.role == 'assistant' -%}
+<|assistant|>
+{%- set content = visible_text(m.content) %}
+{%- if m.reasoning_content is string %}
+    {%- set reasoning_content = m.reasoning_content %}
+{%- elif '</think>' in content %}
+    {%- set reasoning_content = content.split('</think>')[0].split('<think>')[-1] %}
+    {%- set content = content.split('</think>')[-1] %}
+{%- endif %}
+{%- if ((clear_thinking is defined and not clear_thinking) or loop.index0 > ns.last_user_index) and reasoning_content is defined -%}
+{{ '<think>' + reasoning_content +  '</think>'}}
+{%- else -%}
+{{ '<think></think>' }}
+{%- endif -%}
+{%- if content.strip() -%}
+{{ content.strip() }}
+{%- endif -%}
+{% if m.tool_calls %}
+{% for tc in m.tool_calls %}
+{%- if tc.function %}
+    {%- set tc = tc.function %}
+{%- endif %}
+{{- '<tool_call>' + tc.name -}}
+{% set _args = tc.arguments %}{% for k, v in _args.items() %}<arg_key>{{ k }}</arg_key><arg_value>{{ v | tojson(ensure_ascii=False) if v is not string else v }}</arg_value>{% endfor %}</tool_call>{% endfor %}
+{% endif %}
+{%- elif m.role == 'tool' -%}
+{%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
+    {{- '<|observation|>' -}}
+{%- endif %}
+{%- if m.content is string -%}
+    {{- '<tool_response>' + m.content + '</tool_response>' -}}
+{%- elif m.content is iterable and m.content is not mapping and m.content and m.content.0.type == "tool_reference" -%}
+    {{- '<tool_response><tools>\n' -}}
+    {% for tr in m.content %}
+        {%- for tool in tools -%}
+            {%- if 'function' in tool -%}
+                {%- set tool = tool['function'] -%}
+            {%- endif -%}
+            {%- if tool.name == tr.name -%}
+                {{- tool_to_json(tool) + '\n' -}}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- endfor -%}
+    {{- '</tools></tool_response>' -}}
+{%- elif m.content is iterable and m.content is not mapping and m.content and m.content.0 is mapping and m.content.0.output is defined -%}
+    {%- for tr in m.content -%}
+        {{- '<tool_response>' + tr.output + '</tool_response>' -}}
+    {%- endfor -%}
+{%- else -%}
+    {{- '<tool_response>' + visible_text(m.content) + '</tool_response>' -}}
+{% endif -%}
+{%- elif m.role == 'system' -%}
+<|system|>{{ visible_text(m.content) }}
+{%- endif -%}
+{%- endfor -%}
+{%- if add_generation_prompt -%}
+    <|assistant|>{{- '<think></think>' if (enable_thinking is defined and not enable_thinking) else '<think>' -}}
+{%- endif -%}

config.json

@@ -0,0 +1,729 @@
+{
+  "architectures": [
+    "Glm5vForConditionalGeneration"
+  ],
+  "model_type": "glm5v",
+  "auto_map": {
+    "AutoConfig": "configuration_glm5v.Glm5vConfig"
+  },
+  "text_config": {
+    "architectures": [
+      "GlmMoeDsaForCausalLM"
+    ],
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "bos_token_id": 0,
+    "dtype": "bfloat16",
+    "eos_token_id": [
+      154820,
+      154827,
+      154829
+    ],
+    "ep_size": 1,
+    "first_k_dense_replace": 3,
+    "head_dim": 192,
+    "hidden_act": "silu",
+    "hidden_size": 6144,
+    "index_head_dim": 128,
+    "index_n_heads": 32,
+    "index_share_for_mtp_iteration": true,
+    "index_skip_topk_offset": 3,
+    "index_topk": 2048,
+    "index_topk_freq": 4,
+    "index_topk_pattern": null,
+    "indexer_rope_interleave": true,
+    "indexer_types": [
+      "full",
+      "full",
+      "full",
+      "shared",
+      "shared",
+      "shared",
+      "full",
+      "shared",
+      "shared",
+      "shared",
+      "full",
+      "shared",
+      "shared",
+      "shared",
+      "full",
+      "shared",
+      "shared",
+      "shared",
+      "full",
+      "shared",
+      "shared",
+      "shared",
+      "full",
+      "shared",
+      "shared",
+      "shared",
+      "full",
+      "shared",
+      "shared",
+      "shared",
+      "full",
+      "shared",
+      "shared",
+      "shared",
+      "full",
+      "shared",
+      "shared",
+      "shared",
+      "full",
+      "shared",
+      "shared",
+      "shared",
+      "full",
+      "shared",
+      "shared",
+      "shared",
+      "full",
+      "shared",
+      "shared",
+      "shared",
+      "full",
+      "shared",
+      "shared",
+      "shared",
+      "full",
+      "shared",
+      "shared",
+      "shared",
+      "full",
+      "shared",
+      "shared",
+      "shared",
+      "full",
+      "shared",
+      "shared",
+      "shared",
+      "full",
+      "shared",
+      "shared",
+      "shared",
+      "full",
+      "shared",
+      "shared",
+      "shared",
+      "full",
+      "shared",
+      "shared",
+      "shared"
+    ],
+    "initializer_range": 0.02,
+    "intermediate_size": 12288,
+    "kv_lora_rank": 512,
+    "layer_types": [
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention",
+      "deepseek_sparse_attention"
+    ],
+    "max_position_embeddings": 1048576,
+    "mlp_bias": false,
+    "mlp_layer_types": [
+      "dense",
+      "dense",
+      "dense",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse"
+    ],
+    "model_type": "glm_moe_dsa",
+    "moe_intermediate_size": 2048,
+    "moe_layer_freq": 1,
+    "n_group": 1,
+    "n_routed_experts": 256,
+    "n_shared_experts": 1,
+    "norm_topk_prob": true,
+    "num_attention_heads": 64,
+    "num_experts": 256,
+    "num_experts_per_tok": 8,
+    "num_hidden_layers": 78,
+    "num_key_value_heads": 64,
+    "num_nextn_predict_layers": 1,
+    "pad_token_id": 154820,
+    "pretraining_tp": 1,
+    "q_lora_rank": 2048,
+    "qk_head_dim": 256,
+    "qk_nope_head_dim": 192,
+    "qk_rope_head_dim": 64,
+    "rms_norm_eps": 1e-05,
+    "rope_interleave": true,
+    "rope_parameters": {
+      "rope_theta": 8000000,
+      "rope_type": "default"
+    },
+    "routed_scaling_factor": 2.5,
+    "scoring_func": "sigmoid",
+    "tie_word_embeddings": false,
+    "topk_group": 1,
+    "topk_method": "noaux_tc",
+    "transformers_version": "5.11.0",
+    "use_cache": true,
+    "v_head_dim": 256,
+    "vocab_size": 154880,
+    "quantization_config": {
+      "config_groups": {
+        "group_0": {
+          "input_activations": {
+            "dynamic": false,
+            "num_bits": 4,
+            "type": "float",
+            "group_size": 16
+          },
+          "weights": {
+            "dynamic": false,
+            "num_bits": 4,
+            "type": "float",
+            "group_size": 16
+          },
+          "targets": [
+            "Linear"
+          ]
+        }
+      },
+      "ignore": [
+        "lm_head",
+        "model.embed_tokens",
+        "model.layers.0*",
+        "model.layers.1.*",
+        "model.layers.10.mlp.shared_experts*",
+        "model.layers.10.self_attn*",
+        "model.layers.11.mlp.shared_experts*",
+        "model.layers.11.self_attn*",
+        "model.layers.12.mlp.shared_experts*",
+        "model.layers.12.self_attn*",
+        "model.layers.13.mlp.shared_experts*",
+        "model.layers.13.self_attn*",
+        "model.layers.14.mlp.shared_experts*",
+        "model.layers.14.self_attn*",
+        "model.layers.15.mlp.shared_experts*",
+        "model.layers.15.self_attn*",
+        "model.layers.16.mlp.shared_experts*",
+        "model.layers.16.self_attn*",
+        "model.layers.17.mlp.shared_experts*",
+        "model.layers.17.self_attn*",
+        "model.layers.18.mlp.shared_experts*",
+        "model.layers.18.self_attn*",
+        "model.layers.19.mlp.shared_experts*",
+        "model.layers.19.self_attn*",
+        "model.layers.2.*",
+        "model.layers.20.mlp.shared_experts*",
+        "model.layers.20.self_attn*",
+        "model.layers.21.mlp.shared_experts*",
+        "model.layers.21.self_attn*",
+        "model.layers.22.mlp.shared_experts*",
+        "model.layers.22.self_attn*",
+        "model.layers.23.mlp.shared_experts*",
+        "model.layers.23.self_attn*",
+        "model.layers.24.mlp.shared_experts*",
+        "model.layers.24.self_attn*",
+        "model.layers.25.mlp.shared_experts*",
+        "model.layers.25.self_attn*",
+        "model.layers.26.mlp.shared_experts*",
+        "model.layers.26.self_attn*",
+        "model.layers.27.mlp.shared_experts*",
+        "model.layers.27.self_attn*",
+        "model.layers.28.mlp.shared_experts*",
+        "model.layers.28.self_attn*",
+        "model.layers.29.mlp.shared_experts*",
+        "model.layers.29.self_attn*",
+        "model.layers.3.mlp.shared_experts*",
+        "model.layers.3.self_attn*",
+        "model.layers.30.mlp.shared_experts*",
+        "model.layers.30.self_attn*",
+        "model.layers.31.mlp.shared_experts*",
+        "model.layers.31.self_attn*",
+        "model.layers.32.mlp.shared_experts*",
+        "model.layers.32.self_attn*",
+        "model.layers.33.mlp.shared_experts*",
+        "model.layers.33.self_attn*",
+        "model.layers.34.mlp.shared_experts*",
+        "model.layers.34.self_attn*",
+        "model.layers.35.mlp.shared_experts*",
+        "model.layers.35.self_attn*",
+        "model.layers.36.mlp.shared_experts*",
+        "model.layers.36.self_attn*",
+        "model.layers.37.mlp.shared_experts*",
+        "model.layers.37.self_attn*",
+        "model.layers.38.mlp.shared_experts*",
+        "model.layers.38.self_attn*",
+        "model.layers.39.mlp.shared_experts*",
+        "model.layers.39.self_attn*",
+        "model.layers.4.mlp.shared_experts*",
+        "model.layers.4.self_attn*",
+        "model.layers.40.mlp.shared_experts*",
+        "model.layers.40.self_attn*",
+        "model.layers.41.mlp.shared_experts*",
+        "model.layers.41.self_attn*",
+        "model.layers.42.mlp.shared_experts*",
+        "model.layers.42.self_attn*",
+        "model.layers.43.mlp.shared_experts*",
+        "model.layers.43.self_attn*",
+        "model.layers.44.mlp.shared_experts*",
+        "model.layers.44.self_attn*",
+        "model.layers.45.mlp.shared_experts*",
+        "model.layers.45.self_attn*",
+        "model.layers.46.mlp.shared_experts*",
+        "model.layers.46.self_attn*",
+        "model.layers.47.mlp.shared_experts*",
+        "model.layers.47.self_attn*",
+        "model.layers.48.mlp.shared_experts*",
+        "model.layers.48.self_attn*",
+        "model.layers.49.mlp.shared_experts*",
+        "model.layers.49.self_attn*",
+        "model.layers.5.mlp.shared_experts*",
+        "model.layers.5.self_attn*",
+        "model.layers.50.mlp.shared_experts*",
+        "model.layers.50.self_attn*",
+        "model.layers.51.mlp.shared_experts*",
+        "model.layers.51.self_attn*",
+        "model.layers.52.mlp.shared_experts*",
+        "model.layers.52.self_attn*",
+        "model.layers.53.mlp.shared_experts*",
+        "model.layers.53.self_attn*",
+        "model.layers.54.mlp.shared_experts*",
+        "model.layers.54.self_attn*",
+        "model.layers.55.mlp.shared_experts*",
+        "model.layers.55.self_attn*",
+        "model.layers.56.mlp.shared_experts*",
+        "model.layers.56.self_attn*",
+        "model.layers.57.mlp.shared_experts*",
+        "model.layers.57.self_attn*",
+        "model.layers.58.mlp.shared_experts*",
+        "model.layers.58.self_attn*",
+        "model.layers.59.mlp.shared_experts*",
+        "model.layers.59.self_attn*",
+        "model.layers.6.mlp.shared_experts*",
+        "model.layers.6.self_attn*",
+        "model.layers.60.mlp.shared_experts*",
+        "model.layers.60.self_attn*",
+        "model.layers.61.mlp.shared_experts*",
+        "model.layers.61.self_attn*",
+        "model.layers.62.mlp.shared_experts*",
+        "model.layers.62.self_attn*",
+        "model.layers.63.mlp.shared_experts*",
+        "model.layers.63.self_attn*",
+        "model.layers.64.mlp.shared_experts*",
+        "model.layers.64.self_attn*",
+        "model.layers.65.mlp.shared_experts*",
+        "model.layers.65.self_attn*",
+        "model.layers.66.mlp.shared_experts*",
+        "model.layers.66.self_attn*",
+        "model.layers.67.mlp.shared_experts*",
+        "model.layers.67.self_attn*",
+        "model.layers.68.mlp.shared_experts*",
+        "model.layers.68.self_attn*",
+        "model.layers.69.mlp.shared_experts*",
+        "model.layers.69.self_attn*",
+        "model.layers.7.mlp.shared_experts*",
+        "model.layers.7.self_attn*",
+        "model.layers.70.mlp.shared_experts*",
+        "model.layers.70.self_attn*",
+        "model.layers.71.mlp.shared_experts*",
+        "model.layers.71.self_attn*",
+        "model.layers.72.mlp.shared_experts*",
+        "model.layers.72.self_attn*",
+        "model.layers.73.mlp.shared_experts*",
+        "model.layers.73.self_attn*",
+        "model.layers.74.mlp.shared_experts*",
+        "model.layers.74.self_attn*",
+        "model.layers.75.mlp.shared_experts*",
+        "model.layers.75.self_attn*",
+        "model.layers.76.mlp.shared_experts*",
+        "model.layers.76.self_attn*",
+        "model.layers.77.mlp.shared_experts*",
+        "model.layers.77.self_attn*",
+        "model.layers.8.mlp.shared_experts*",
+        "model.layers.8.self_attn*",
+        "model.layers.9.mlp.shared_experts*",
+        "model.layers.9.self_attn*",
+        "model.layers.78*"
+      ],
+      "quant_algo": "NVFP4",
+      "kv_cache_scheme": {
+        "dynamic": false,
+        "num_bits": 8,
+        "type": "float"
+      },
+      "producer": {
+        "name": "modelopt",
+        "version": "0.46.0.dev65+g977d34dc3"
+      },
+      "quant_method": "modelopt"
+    }
+  },
+  "vision_config": {
+    "patch_size": 14,
+    "init_pos_emb_height": 64,
+    "init_pos_emb_width": 64,
+    "init_pos_emb_time": 4,
+    "pos_emb_type": "divided_fixed",
+    "num_attention_heads": 16,
+    "num_hidden_layers": 27,
+    "hidden_size": 1152,
+    "intermediate_size": 4304,
+    "vt_num_attention_heads": 16,
+    "vt_num_hidden_layers": 27,
+    "vt_hidden_size": 1152,
+    "vt_intermediate_size": 4304,
+    "merge_kernel_size": [
+      2,
+      2
+    ],
+    "video_attn_type": "spatial_temporal",
+    "merge_type": "sd2_tpool",
+    "mm_projector_type": "patchmerger",
+    "mm_hidden_size": 1152,
+    "projector_hidden_act": "gelu",
+    "projector_ln_eps": 1e-05,
+    "text_hidden_size": 6144
+  },
+  "ignore_index": -100,
+  "media_placeholder_token_id": 154854,
+  "pad_token_id": 154820,
+  "use_unified_vision_chunk": true,
+  "video_placeholder": "<|glm5v_video_placeholder|>",
+  "encoder_only": false,
+  "language_only": false,
+  "tie_word_embeddings": false,
+  "quantization_config": {
+    "config_groups": {
+      "group_0": {
+        "input_activations": {
+          "dynamic": false,
+          "num_bits": 4,
+          "type": "float",
+          "group_size": 16
+        },
+        "weights": {
+          "dynamic": false,
+          "num_bits": 4,
+          "type": "float",
+          "group_size": 16
+        },
+        "targets": [
+          "Linear"
+        ]
+      }
+    },
+    "ignore": [
+      "lm_head",
+      "model.embed_tokens",
+      "model.layers.0*",
+      "model.layers.1.*",
+      "model.layers.10.mlp.shared_experts*",
+      "model.layers.10.self_attn*",
+      "model.layers.11.mlp.shared_experts*",
+      "model.layers.11.self_attn*",
+      "model.layers.12.mlp.shared_experts*",
+      "model.layers.12.self_attn*",
+      "model.layers.13.mlp.shared_experts*",
+      "model.layers.13.self_attn*",
+      "model.layers.14.mlp.shared_experts*",
+      "model.layers.14.self_attn*",
+      "model.layers.15.mlp.shared_experts*",
+      "model.layers.15.self_attn*",
+      "model.layers.16.mlp.shared_experts*",
+      "model.layers.16.self_attn*",
+      "model.layers.17.mlp.shared_experts*",
+      "model.layers.17.self_attn*",
+      "model.layers.18.mlp.shared_experts*",
+      "model.layers.18.self_attn*",
+      "model.layers.19.mlp.shared_experts*",
+      "model.layers.19.self_attn*",
+      "model.layers.2.*",
+      "model.layers.20.mlp.shared_experts*",
+      "model.layers.20.self_attn*",
+      "model.layers.21.mlp.shared_experts*",
+      "model.layers.21.self_attn*",
+      "model.layers.22.mlp.shared_experts*",
+      "model.layers.22.self_attn*",
+      "model.layers.23.mlp.shared_experts*",
+      "model.layers.23.self_attn*",
+      "model.layers.24.mlp.shared_experts*",
+      "model.layers.24.self_attn*",
+      "model.layers.25.mlp.shared_experts*",
+      "model.layers.25.self_attn*",
+      "model.layers.26.mlp.shared_experts*",
+      "model.layers.26.self_attn*",
+      "model.layers.27.mlp.shared_experts*",
+      "model.layers.27.self_attn*",
+      "model.layers.28.mlp.shared_experts*",
+      "model.layers.28.self_attn*",
+      "model.layers.29.mlp.shared_experts*",
+      "model.layers.29.self_attn*",
+      "model.layers.3.mlp.shared_experts*",
+      "model.layers.3.self_attn*",
+      "model.layers.30.mlp.shared_experts*",
+      "model.layers.30.self_attn*",
+      "model.layers.31.mlp.shared_experts*",
+      "model.layers.31.self_attn*",
+      "model.layers.32.mlp.shared_experts*",
+      "model.layers.32.self_attn*",
+      "model.layers.33.mlp.shared_experts*",
+      "model.layers.33.self_attn*",
+      "model.layers.34.mlp.shared_experts*",
+      "model.layers.34.self_attn*",
+      "model.layers.35.mlp.shared_experts*",
+      "model.layers.35.self_attn*",
+      "model.layers.36.mlp.shared_experts*",
+      "model.layers.36.self_attn*",
+      "model.layers.37.mlp.shared_experts*",
+      "model.layers.37.self_attn*",
+      "model.layers.38.mlp.shared_experts*",
+      "model.layers.38.self_attn*",
+      "model.layers.39.mlp.shared_experts*",
+      "model.layers.39.self_attn*",
+      "model.layers.4.mlp.shared_experts*",
+      "model.layers.4.self_attn*",
+      "model.layers.40.mlp.shared_experts*",
+      "model.layers.40.self_attn*",
+      "model.layers.41.mlp.shared_experts*",
+      "model.layers.41.self_attn*",
+      "model.layers.42.mlp.shared_experts*",
+      "model.layers.42.self_attn*",
+      "model.layers.43.mlp.shared_experts*",
+      "model.layers.43.self_attn*",
+      "model.layers.44.mlp.shared_experts*",
+      "model.layers.44.self_attn*",
+      "model.layers.45.mlp.shared_experts*",
+      "model.layers.45.self_attn*",
+      "model.layers.46.mlp.shared_experts*",
+      "model.layers.46.self_attn*",
+      "model.layers.47.mlp.shared_experts*",
+      "model.layers.47.self_attn*",
+      "model.layers.48.mlp.shared_experts*",
+      "model.layers.48.self_attn*",
+      "model.layers.49.mlp.shared_experts*",
+      "model.layers.49.self_attn*",
+      "model.layers.5.mlp.shared_experts*",
+      "model.layers.5.self_attn*",
+      "model.layers.50.mlp.shared_experts*",
+      "model.layers.50.self_attn*",
+      "model.layers.51.mlp.shared_experts*",
+      "model.layers.51.self_attn*",
+      "model.layers.52.mlp.shared_experts*",
+      "model.layers.52.self_attn*",
+      "model.layers.53.mlp.shared_experts*",
+      "model.layers.53.self_attn*",
+      "model.layers.54.mlp.shared_experts*",
+      "model.layers.54.self_attn*",
+      "model.layers.55.mlp.shared_experts*",
+      "model.layers.55.self_attn*",
+      "model.layers.56.mlp.shared_experts*",
+      "model.layers.56.self_attn*",
+      "model.layers.57.mlp.shared_experts*",
+      "model.layers.57.self_attn*",
+      "model.layers.58.mlp.shared_experts*",
+      "model.layers.58.self_attn*",
+      "model.layers.59.mlp.shared_experts*",
+      "model.layers.59.self_attn*",
+      "model.layers.6.mlp.shared_experts*",
+      "model.layers.6.self_attn*",
+      "model.layers.60.mlp.shared_experts*",
+      "model.layers.60.self_attn*",
+      "model.layers.61.mlp.shared_experts*",
+      "model.layers.61.self_attn*",
+      "model.layers.62.mlp.shared_experts*",
+      "model.layers.62.self_attn*",
+      "model.layers.63.mlp.shared_experts*",
+      "model.layers.63.self_attn*",
+      "model.layers.64.mlp.shared_experts*",
+      "model.layers.64.self_attn*",
+      "model.layers.65.mlp.shared_experts*",
+      "model.layers.65.self_attn*",
+      "model.layers.66.mlp.shared_experts*",
+      "model.layers.66.self_attn*",
+      "model.layers.67.mlp.shared_experts*",
+      "model.layers.67.self_attn*",
+      "model.layers.68.mlp.shared_experts*",
+      "model.layers.68.self_attn*",
+      "model.layers.69.mlp.shared_experts*",
+      "model.layers.69.self_attn*",
+      "model.layers.7.mlp.shared_experts*",
+      "model.layers.7.self_attn*",
+      "model.layers.70.mlp.shared_experts*",
+      "model.layers.70.self_attn*",
+      "model.layers.71.mlp.shared_experts*",
+      "model.layers.71.self_attn*",
+      "model.layers.72.mlp.shared_experts*",
+      "model.layers.72.self_attn*",
+      "model.layers.73.mlp.shared_experts*",
+      "model.layers.73.self_attn*",
+      "model.layers.74.mlp.shared_experts*",
+      "model.layers.74.self_attn*",
+      "model.layers.75.mlp.shared_experts*",
+      "model.layers.75.self_attn*",
+      "model.layers.76.mlp.shared_experts*",
+      "model.layers.76.self_attn*",
+      "model.layers.77.mlp.shared_experts*",
+      "model.layers.77.self_attn*",
+      "model.layers.8.mlp.shared_experts*",
+      "model.layers.8.self_attn*",
+      "model.layers.9.mlp.shared_experts*",
+      "model.layers.9.self_attn*",
+      "model.layers.78*"
+    ],
+    "quant_algo": "NVFP4",
+    "kv_cache_scheme": {
+      "dynamic": false,
+      "num_bits": 8,
+      "type": "float"
+    },
+    "producer": {
+      "name": "modelopt",
+      "version": "0.46.0.dev65+g977d34dc3"
+    },
+    "quant_method": "modelopt"
+  }
+}

configuration_glm5v.py

@@ -0,0 +1,171 @@
+# SPDX-License-Identifier: Apache-2.0
+"""Glm5vConfig — remote-code config carried inside the assembled GLM5V SGLang
+checkpoint (referenced by config.json ``auto_map``; loaded with
+``--trust-remote-code``, which the checkpoint already requires for the Kimi
+image-processor remote code).
+
+Self-contained: depends only on ``transformers``. Mirrors SGLang's in-tree
+``KimiK25Config`` structure (``vision_config`` + ``text_config`` + media
+placeholder fields) with GLM-5.2 as the text model:
+
+* ``text_config``  -> ``GlmMoeDsaConfig`` (transformers-native ``glm_moe_dsa``).
+* ``vision_config``-> MoonViT fields; ``text_hidden_size`` (projector output
+  dim) retargeted to GLM hidden 6144.
+* ``media_placeholder_token_id`` -> GLM ``<|image|>`` = 154854.
+"""
+
+from transformers import AutoConfig
+from transformers.configuration_utils import PretrainedConfig
+
+
+class Glm5vVisionConfig(PretrainedConfig):
+    """MoonViT vision tower + PatchMerger projector config.
+
+    Field names/defaults mirror SGLang's ``KimiK25VisionConfig`` (declared
+    names like ``hidden_size``) while the official Kimi checkpoint's ``vt_*``
+    names arrive via **kwargs and are stored as attributes — SGLang's model
+    code reads both families (tower: ``hidden_size``; projector:
+    ``vt_hidden_size``/``text_hidden_size``).
+    """
+
+    model_type = "glm5v_vision"
+
+    def __init__(
+        self,
+        # Vision tower
+        patch_size: int = 14,
+        init_pos_emb_height: int = 64,
+        init_pos_emb_width: int = 64,
+        init_pos_emb_time: int = 4,
+        pos_emb_type: str = "divided_fixed",
+        num_attention_heads: int = 16,
+        num_hidden_layers: int = 27,
+        hidden_size: int = 1152,
+        intermediate_size: int = 4304,
+        merge_kernel_size=(2, 2),
+        video_attn_type: str = "spatial_temporal",
+        merge_type: str = "sd2_tpool",
+        # MM projector
+        mm_projector_type: str = "patchmerger",
+        mm_hidden_size: int | None = None,
+        vt_hidden_size: int | None = None,   # SGLang kimi_k25 projector reads this (== vision-tower hidden)
+        projector_hidden_act: str = "gelu",
+        projector_ln_eps: float = 1e-5,
+        text_hidden_size: int = 6144,  # GLM-5.2 hidden (Kimi default is 7168)
+        **kwargs,
+    ):
+        super().__init__(**kwargs)
+        self.patch_size = patch_size
+        self.init_pos_emb_height = init_pos_emb_height
+        self.init_pos_emb_width = init_pos_emb_width
+        self.init_pos_emb_time = init_pos_emb_time
+        self.pos_emb_type = pos_emb_type
+        self.num_attention_heads = num_attention_heads
+        self.num_hidden_layers = num_hidden_layers
+        self.hidden_size = hidden_size
+        self.intermediate_size = intermediate_size
+        self.merge_kernel_size = merge_kernel_size
+        self.video_attn_type = video_attn_type
+        self.merge_type = merge_type
+        self.mm_projector_type = mm_projector_type
+        self.mm_hidden_size = mm_hidden_size if mm_hidden_size is not None else hidden_size
+        self.vt_hidden_size = vt_hidden_size if vt_hidden_size is not None else hidden_size
+        self.projector_hidden_act = projector_hidden_act
+        self.projector_ln_eps = projector_ln_eps
+        self.text_hidden_size = text_hidden_size
+
+    def __getattr__(self, name):
+        # SGLang's kimi_k25 reads vt_-prefixed vision fields (vt_hidden_size, vt_intermediate_size, ...)
+        # that our config declares without the prefix; alias any missing vt_* to the base attribute.
+        # Reads __dict__ directly (no recursion) and raises normally if the base isn't set.
+        if name.startswith("vt_"):
+            d = object.__getattribute__(self, "__dict__")
+            base = name[3:]
+            if base in d:
+                return d[base]
+        raise AttributeError(name)
+
+
+class Glm5vConfig(PretrainedConfig):
+    """glm5v top-level config: MoonViT ``vision_config`` + GLM-5.2 ``text_config``."""
+
+    model_type = "glm5v"
+
+    def __init__(
+        self,
+        text_config=None,
+        vision_config=None,
+        ignore_index: int = -100,
+        media_placeholder_token_id: int = 154854,  # GLM <|image|>
+        pad_token_id: int = 154820,
+        use_unified_vision_chunk: bool = True,
+        video_placeholder: str = "<|glm5v_video_placeholder|>",
+        encoder_only: bool = False,
+        language_only: bool = False,
+        **kwargs,
+    ):
+        # Vision config (MoonViT).
+        if vision_config is None:
+            self.vision_config = Glm5vVisionConfig()
+        elif isinstance(vision_config, dict):
+            self.vision_config = Glm5vVisionConfig(**vision_config)
+        else:
+            self.vision_config = vision_config
+
+        # Text config (GLM-5.2 / glm_moe_dsa), built via AutoConfig so the
+        # transformers-native GlmMoeDsaConfig class is used.
+        raw_text = dict(text_config) if isinstance(text_config, dict) else None
+        if text_config is None:
+            self.text_config = AutoConfig.for_model("glm_moe_dsa")
+        elif isinstance(text_config, dict):
+            tc = dict(text_config)
+            tc.setdefault("model_type", "glm_moe_dsa")
+            # Newer transformers (in the SGLang serving image) validates `layer_types`
+            # via a StrictDataclass and rejects the legacy DSA value
+            # "deepseek_sparse_attention". The DSA attention path is selected from
+            # model_type + the DSA config fields (index_topk etc.), NOT from layer_types,
+            # so drop it to pass validation without changing behavior.
+            tc.pop("layer_types", None)
+            self.text_config = AutoConfig.for_model(**tc)
+        else:
+            self.text_config = text_config
+
+        # transformers 5.8.x GlmMoeDsaConfig drops/clobbers raw DSA fields the
+        # sparse-attention path needs. SGLang applies this same restore for
+        # bare GlmMoeDsaForCausalLM checkpoints (see its HfModelConfigParser;
+        # fixed upstream by transformers PR #46338, gone once >= 5.10); our
+        # top-level arch is Glm5v so we replicate it here.
+        if raw_text is not None:
+            for key in ("qk_rope_head_dim", "index_topk_freq"):
+                if key in raw_text:
+                    setattr(self.text_config, key, raw_text[key])
+            if hasattr(self.text_config, "qk_nope_head_dim") and hasattr(
+                self.text_config, "qk_rope_head_dim"
+            ):
+                self.text_config.qk_head_dim = (
+                    self.text_config.qk_nope_head_dim
+                    + self.text_config.qk_rope_head_dim
+                )
+
+        self.ignore_index = ignore_index
+        self.media_placeholder_token_id = media_placeholder_token_id
+        self.use_unified_vision_chunk = use_unified_vision_chunk
+        self.video_placeholder = video_placeholder
+        self.encoder_only = encoder_only
+        self.language_only = language_only
+
+        # Propagate quantization config from the text model (Kimi pattern):
+        # only the GLM text Linears are FP8; vision/projector stay bf16 by
+        # construction in the model code.
+        if getattr(self.text_config, "quantization_config", None) is not None:
+            self.quantization_config = self.text_config.quantization_config
+
+        super().__init__(pad_token_id=pad_token_id, **kwargs)
+
+    @property
+    def hidden_size(self) -> int:
+        return self.text_config.hidden_size
+
+    @property
+    def vocab_size(self) -> int:
+        return self.text_config.vocab_size

generation_config.json

@@ -0,0 +1,13 @@
+{
+  "_from_model_config": true,
+  "do_sample": true,
+  "eos_token_id": [
+    154820,
+    154827,
+    154829
+  ],
+  "pad_token_id": 154820,
+  "temperature": 1.0,
+  "top_p": 0.95,
+  "transformers_version": "5.11.0"
+}

kimi_k25_processor.py

@@ -0,0 +1,208 @@
+from transformers.feature_extraction_utils import BatchFeature
+from transformers.processing_utils import ProcessorMixin
+from transformers.utils import logging
+
+logger = logging.get_logger(__name__)
+
+
+class KimiK25Processor(ProcessorMixin):
+    r"""
+    Constructs a KimiK25 processor which wraps a KimiK25 image processor and a tokenizer into a single processor.
+
+    [`KimiK25Processor`] offers all the functionalities of [`KimiK25ImageProcessor`] and [`TikTokenTokenizer`]. See the
+    [`~KimiK25Processor.__call__`] and [`~KimiK25Processor.decode`] for more information.
+
+    Args:
+        image_processor ([`KimiK25ImageProcessor`], *optional*):
+            The image processor is a required input.
+        tokenizer ([`TikTokenTokenizer`], *optional*):
+            The tokenizer is a required input.
+        chat_template (`str`, *optional*): A Jinja template which will be used to convert lists of messages
+            in a chat into a tokenizable string.
+    """
+
+    attributes = ["image_processor", "tokenizer"]
+    valid_kwargs = ["chat_template"]
+    image_processor_class = "AutoImageProcessor"
+    tokenizer_class = "AutoTokenizer"
+
+    def __init__(
+        self,
+        image_processor=None,
+        tokenizer=None,
+        chat_template=None,
+        **kwargs,
+    ):
+        super().__init__(image_processor,
+                         tokenizer,
+                         chat_template=chat_template)
+        self.media_processor = image_processor
+        # A special temporal placeholder to be replaced by actual video placeholders
+        self.video_placeholder = "<|kimi_k25_video_placeholder|>"
+
+    def update_raw_text(self, text: str, video_prompts: list[str]) -> str:
+        # replace video prompt in text with video chunk prompts
+        video_count = text.count(self.video_placeholder)
+        if video_count == 0:
+            return text
+        assert video_count == len(video_prompts)
+        text_parts = text.split(self.video_placeholder)
+        assert len(text_parts) == len(video_prompts) + 1
+        text = "".join([
+            text_parts[i] + video_prompts[i] for i in range(len(video_prompts))
+        ])
+        text += text_parts[-1]
+        return text
+
+    def preprocess_medias(self, medias: list[dict]) -> list[dict]:
+        updated_medias = []
+        video_prompts = []
+        for media in medias:
+            if media['type'] == 'image':
+                updated_medias.append(media)
+            elif media['type'] == 'video':
+                video_chunks = self.media_processor.split_video_chunks(
+                    media['video'])
+                updated_medias.extend(video_chunks)
+                video_prompts.append("".join(
+                    [vc['prompt'] for vc in video_chunks]))
+            else:
+                raise ValueError(f"unsupported media type: {media['type']}")
+        return updated_medias, video_prompts
+
+    # glm5v: the image placeholder expanded per patch (GLM <|image|> = 154854).
+    # The chat template wraps it as <|begin_of_image|><|image|><|end_of_image|>.
+    GLM5V_IMAGE_TOKEN = "<|image|>"
+
+    def __call__(self,
+                 messages: list[dict] = None,
+                 medias: list[dict] = None,
+                 text: str = None,
+                 images: list = None,
+                 return_tensors: str = "pt",
+                 **kwargs) -> BatchFeature:
+        """
+        Process multimodal inputs for Kimi-K2.5 model.
+
+        This processor accepts ordered messages and extracts both media and text in a single pass.
+        text will be automatically updated if video input detected in messages
+
+        Args:
+            messages: List of message dicts with 'role' and 'content' fields.
+                     If provided, medias and text will be extracted automatically.
+            medias: Pre-extracted list of media dicts. If None, extracted from messages.
+            text: Pre-formatted text string. If None, generated via apply_chat_template.
+            images: Standard HF VLM API (``processor(text=..., images=[...])``), as
+                    called by generic drivers (e.g. slime's rollout prompt prep).
+                    Converted to ``medias`` and each ``<|image|>`` placeholder in
+                    ``text`` is expanded to that image's per-patch token count, so
+                    the returned ``input_ids`` align with ``pixel_values`` (same
+                    semantics as Qwen-family processors and the SGLang serving-layer
+                    wrapper).
+            return_tensors: Format of returned tensors ('pt', 'np', 'tf'). Default: 'pt'.
+            **kwargs: Additional arguments passed to tokenizer.apply_chat_template.
+
+        Returns:
+            BatchFeature with fields: input_ids, attention_mask, pixel_values, grid_thws.
+        """
+        if images is not None and medias is None and text is not None:
+            # Standard HF call: expand placeholders, run the media preprocess, and
+            # return with STANDARD-HF dtypes: input_ids/attention_mask as python
+            # lists (callers like slime's rollout do `sample.tokens += tokens`),
+            # media tensors as `return_tensors` (default pt) for the train side.
+            if not isinstance(images, (list, tuple)):
+                images = [images]
+            medias = [{"type": "image", "image": img} for img in images]
+            parts = text.split(self.GLM5V_IMAGE_TOKEN)
+            if len(parts) - 1 != len(images):
+                raise ValueError(
+                    f"got {len(images)} images but {len(parts) - 1} "
+                    f"{self.GLM5V_IMAGE_TOKEN!r} placeholders in text")
+            expanded = [parts[0]]
+            for media, part in zip(medias, parts[1:]):
+                num_tokens = self.media_processor.media_tokens_calculator(media)
+                expanded.append(self.GLM5V_IMAGE_TOKEN * num_tokens + part)
+            text = "".join(expanded)
+
+            updated_medias, video_prompts = self.preprocess_medias(medias)
+            preprocessed = self.media_processor.preprocess(
+                updated_medias, return_tensors=return_tensors)
+            text = self.update_raw_text(text, video_prompts)
+            text_inputs = self.tokenizer([text])  # no return_tensors -> lists
+            data = {**text_inputs, **preprocessed.data}
+            # Qwen-convention key: downstream training forwards take
+            # `image_grid_thw` (same rename the SGLang wrapper applies).
+            if "grid_thws" in data:
+                data["image_grid_thw"] = data.pop("grid_thws")
+            return BatchFeature(data=data)
+
+        if messages is None and (medias is None or text is None):
+            raise ValueError(
+                "Provide either 'messages' or both 'medias' and 'text'")
+
+        if medias is not None and text is not None:
+            updated_medias, video_prompts = self.preprocess_medias(medias)
+            preprocessed = self.media_processor.preprocess(
+                updated_medias, return_tensors=return_tensors)
+            text = self.update_raw_text(text, video_prompts)
+            text_inputs = self.tokenizer(text, return_tensors=return_tensors)
+            return BatchFeature(data={**text_inputs, **preprocessed.data})
+
+        if medias is None:
+            medias = self._extract_medias_from_messages(messages)
+        updated_medias, video_prompts = self.preprocess_medias(medias)
+        preprocessed = self.media_processor.preprocess(
+            updated_medias, return_tensors=return_tensors)
+
+        # Generate text if not provided
+        if text is None:
+            text = self.tokenizer.apply_chat_template(messages, **kwargs)
+
+        text = self.update_raw_text(text, video_prompts)
+
+        text_inputs = self.tokenizer(text, return_tensors=return_tensors)
+        return BatchFeature(data={**text_inputs, **preprocessed.data})
+
+    @staticmethod
+    def _extract_medias_from_messages(messages: list[dict]) -> list[dict]:
+        """
+        Extract media items from messages in a single pass.
+        
+        This is an optimized version that processes messages only once.
+        Kept as internal method since external callers should use __call__.
+        """
+        medias = []
+        for msg in messages:
+            if msg['role'] != 'user' or not msg.get('content'):
+                continue
+
+            for content_part in msg['content']:
+                if not isinstance(content_part, dict):
+                    continue
+
+                content_type = content_part.get('type')
+                if content_type in ['video_url', 'video']:
+                    medias.append({
+                        'type': 'video',
+                        'video': content_part['video_url']['url'],
+                        'first_frame_timestamp': 0.0
+                    })
+                elif content_type in ['image_url', 'image']:
+                    medias.append({
+                        'type': 'image',
+                        'image': content_part['image_url'],
+                    })
+        return medias
+
+    def apply_chat_template(self, messages, **kwargs):
+        return self.tokenizer.apply_chat_template(messages, **kwargs)
+
+    def batch_decode(self, *args, **kwargs):
+        return self.tokenizer.batch_decode(*args, **kwargs)
+
+    def decode(self, *args, **kwargs):
+        return self.tokenizer.decode(*args, **kwargs)
+
+    @property
+    def model_input_names(self):
+        return ['input_ids', 'attention_mask', 'pixel_values', 'grid_thws']

kimi_k25_vision_processing.py

@@ -0,0 +1,251 @@
+"""Image processor class for Kimi-K2.5.
+"""
+
+import json
+from typing import Any, Dict, Optional, Union
+
+import numpy as np
+import torch
+from PIL import Image
+from transformers.image_processing_utils import (BaseImageProcessor,
+                                                 BatchFeature)
+from transformers.utils import TensorType
+
+from .media_utils import (MediaInput, VideoChunkInput, _to_tensor,
+                          ensure_media_type, get_video_meta, image_to_np,
+                          navit_patchify, navit_resize_image,
+                          navit_resize_video, normalize,
+                          real_sample_fps_and_max_num_frames, timestamp_as_str)
+
+try:
+    from mecord import VideoReader
+except ImportError:
+    VideoReader = None
+
+
+def resampling(video_bytes: bytes,
+               sample_indices: list[int],
+               key_indices=None,
+               frame_time_info=None,
+               num_threads=4) -> str:
+    video = VideoReader(video_bytes,
+                        num_threads=num_threads,
+                        frame_time_info=frame_time_info,
+                        key_indices=key_indices)
+    # extract target frames
+    frames = video[sample_indices]
+    frames = [Image.fromarray(frame) for frame in frames]
+    return frames
+
+
+class KimiK25VisionProcessor(BaseImageProcessor):
+    model_type = "kimi_k25"
+
+    def __init__(
+        self,
+        media_proc_cfg: dict,
+        **kwargs,
+    ):
+        super().__init__(**kwargs)
+        self.media_proc_cfg = media_proc_cfg
+        self.num_frames_per_chunk = media_proc_cfg[
+            'temporal_merge_kernel_size']
+
+    def media_tokens_calculator(self, media: MediaInput):
+        media = ensure_media_type(media)
+        ret = self.get_resize_config(media)
+        return ret['num_tokens']
+
+    @classmethod
+    def make_chunk_prompt(cls, timestamp_text: str) -> str:
+        return f"{timestamp_text}<|media_begin|>video<|media_content|><|media_pad|><|media_end|>"
+
+    def split_video_chunks(self,
+                           video_url: str | bytes) -> list[list[Image.Image]]:
+        # video_url should be base64 str or bytes
+        video_spec = get_video_meta(video_url)
+        sample_fps = min(self.media_proc_cfg['sample_fps'], video_spec.fps)
+        sampled_nframes = max(
+            round(video_spec.num_frames * sample_fps / video_spec.fps), 1)
+        frame_inds = np.linspace(0, video_spec.num_frames - 1,
+                                 sampled_nframes).round().astype(int)
+        frame_inds = frame_inds.tolist()
+        sampled_frame_ids = []
+        temporal_merge_kernel_size = self.media_proc_cfg[
+            "temporal_merge_kernel_size"]
+        num_chunks = 0
+        chunk_timestamp = []
+        for i in range(0, len(frame_inds), temporal_merge_kernel_size):
+            sampled_frame_ids.extend(frame_inds[i:i +
+                                                temporal_merge_kernel_size])
+            start_time = frame_inds[i] / float(video_spec.fps)
+            timestamp_text = timestamp_as_str(
+                start_time, self.media_proc_cfg["timestamp_mode"])
+            chunk_timestamp.append(timestamp_text)
+            num_chunks += 1
+
+        sampled_frames = resampling(video_url, sampled_frame_ids)
+        chunks = []
+        for chunk_id in range(num_chunks):
+            chunk = sampled_frames[chunk_id *
+                                   temporal_merge_kernel_size:(chunk_id + 1) *
+                                   temporal_merge_kernel_size]
+            chunks.append(
+                VideoChunkInput(type="video_chunk",
+                                video_chunk=chunk,
+                                prompt=self.make_chunk_prompt(
+                                    chunk_timestamp[chunk_id])))
+        return chunks
+
+    def get_resize_config(self, media_input: MediaInput) -> dict:
+        if media_input['type'] == 'image':
+            w, h = media_input['image'].size
+            ret = navit_resize_image(
+                w, h, self.media_proc_cfg['patch_size'],
+                self.media_proc_cfg['merge_kernel_size'],
+                self.media_proc_cfg['in_patch_limit'],
+                self.media_proc_cfg['patch_limit_on_one_side'],
+                self.media_proc_cfg['fixed_output_tokens'])
+            return ret
+        elif media_input['type'] == 'video_chunk':
+            frame = media_input['video_chunk'][0]
+            width, height = frame.size
+            num_frames = len(media_input["video_chunk"])
+            fps = 1.0
+
+            sample_fps, max_num_frames_each_video = real_sample_fps_and_max_num_frames(
+                media_input["type"],
+                self.media_proc_cfg['sample_fps'],
+                self.media_proc_cfg['max_num_frames_each_video'],
+            )
+
+            in_patch_limit_each_frame = self.media_proc_cfg[
+                'in_patch_limit_each_frame']
+            if in_patch_limit_each_frame is None:
+                in_patch_limit_each_frame = self.media_proc_cfg[
+                    'in_patch_limit']
+
+            ret = navit_resize_video(
+                width,
+                height,
+                num_frames,
+                fps,
+                sample_fps,
+                self.media_proc_cfg['patch_size'],
+                self.media_proc_cfg['merge_kernel_size'],
+                in_patch_limit_each_frame,
+                self.media_proc_cfg['patch_limit_on_one_side'],
+                self.media_proc_cfg['in_patch_limit_video'],
+                max_num_frames_each_video,
+                self.media_proc_cfg['fixed_output_tokens'],
+            )
+            return ret
+        else:
+            raise ValueError("Unsupported type: {}".format(
+                media_input['type']))
+
+    def resize_image(self, image: Image.Image, new_width: int, new_height: int,
+                     pad_width: int, pad_height: int) -> np.ndarray:
+        image_np = image_to_np(image, (new_width, new_height), "resize")
+        image_np = np.pad(
+            image_np,
+            ((0, pad_height), (0, pad_width), (0, 0)),
+            mode="constant",
+            constant_values=0,
+        )
+        return image_np
+
+    def preprocess(
+        self,
+        medias: list[MediaInput],
+        return_tensors: Optional[Union[str, TensorType]] = None,
+    ) -> BatchFeature:
+        """
+        Preprocess a atom vision input (images/video_chunk) into model-ready tensors.
+        
+        Args:
+            medias: List of MediaInput.
+            return_tensors: Desired output format ('pt', 'np', 'tf', or None).
+        
+        Returns:
+            BatchFeature containing 'pixel_values' and 'grid_thws' tensors.
+        """
+        if not isinstance(medias, list):
+            medias = [medias]
+        if medias:
+            pixel_values = []
+            for item in medias:
+                item = ensure_media_type(item)
+                resize_config = self.get_resize_config(item)
+                new_width, new_height, pad_width, pad_height = resize_config[
+                    'new_width'], resize_config['new_height'], resize_config[
+                        'pad_width'], resize_config['pad_height']
+                if item['type'] == 'image':
+                    image = item['image']
+                    image_np = self.resize_image(image, new_width, new_height,
+                                                 pad_width, pad_height)
+                    pixel_values.append(np.expand_dims(image_np, axis=0))
+                elif item['type'] == 'video_chunk':
+                    pixels = []
+                    for frame in item['video_chunk']:
+                        frame_np = self.resize_image(frame, new_width,
+                                                     new_height, pad_width,
+                                                     pad_height)
+                        pixels.append(frame_np)
+                    pixel_values.append(np.stack(pixels, axis=0))
+                else:
+                    raise ValueError("Unsupported type: {}".format(
+                        item['type']))
+            normalized_pixel_values = []
+            image_std_inv = 1.0 / np.array(self.media_proc_cfg['image_std'])
+            image_mean = np.array(self.media_proc_cfg['image_mean'])
+            for pixels in pixel_values:
+                pixels = normalize(pixels, image_mean, image_std_inv)
+                pixels_and_thw = navit_patchify(
+                    pixels,
+                    self.media_proc_cfg['patch_size'],
+                )
+                normalized_pixel_values.append(pixels_and_thw)
+
+            pixel_values = torch.cat([
+                _to_tensor(pixel_value['pixel_values'])
+                for pixel_value in normalized_pixel_values
+            ])
+            grid_thws = torch.cat([
+                _to_tensor(pixel_value['grid_thw'],
+                           dtype=torch.int64).unsqueeze(0)
+                for pixel_value in normalized_pixel_values
+            ])
+
+            data = {
+                'pixel_values': pixel_values,
+                'grid_thws': grid_thws,
+            }
+
+        else:
+            data = {}
+
+        return BatchFeature(data=data, tensor_type=return_tensors)
+
+    def __repr__(self):
+        return f"KimiK25VisionProcessor(media_proc_cfg={self.media_proc_cfg})"
+
+    def to_dict(self) -> Dict[str, Any]:
+        output = super().to_dict()
+        output["media_proc_cfg"] = self.media_proc_cfg
+        if "media_processor" in output:
+            del output["media_processor"]
+        return output
+
+    @classmethod
+    def from_dict(cls, config_dict: Dict[str, Any], **kwargs):
+        config = config_dict.copy()
+        media_proc_cfg = config.pop("media_proc_cfg", {})
+        return cls(media_proc_cfg=media_proc_cfg, **config, **kwargs)
+
+    def to_json_string(self):
+        dictionary = self.to_dict()
+        for key, value in dictionary.items():
+            if hasattr(value, 'tolist'):
+                dictionary[key] = value.tolist()
+        return json.dumps(dictionary, indent=2, sort_keys=True) + "\n"

media_utils.py

@@ -0,0 +1,368 @@
+import base64
+import io
+import math
+import os
+from datetime import datetime, timezone
+from typing import List, Literal, Optional, TypedDict
+
+import numpy as np
+from PIL import Image
+from pydantic import BaseModel, Field
+
+try:
+    from mecord import VideoReader
+except ImportError:
+    VideoReader = None
+
+
+class VideoSpec(BaseModel):
+    media_type: str = Literal['video']
+    height: int = Field(..., gt=0, description="video frame height")
+    width: int = Field(..., gt=0, description="video frame width")
+    num_frames: int = Field(..., gt=0, description="num frames")
+    fps: float = Field(..., gt=0, description="average fps")
+
+    # optional, help to accelerate video reading
+    key_indices: list[int] = Field(None, description="key indices")
+    frame_time_info: dict = Field(None, description="frame time info")
+
+
+class ImageInput(TypedDict):
+    type: Literal['image']
+    image: Image.Image
+
+
+class VideoChunkInput(TypedDict):
+    type: Literal['video_chunk']
+    video_chunk: List[Image.Image]
+    prompt: Optional[str] = None
+
+
+MediaInput = ImageInput | VideoChunkInput
+
+
+def get_video_meta(video_src: bytes | str | os.PathLike,
+                   accurate: bool = True) -> dict:
+    """Get the dimensions of a video."""
+    if isinstance(video_src, os.PathLike):
+        video_src = str(video_src)
+    # if b64 string, decode to bytes
+    if isinstance(video_src,
+                  str) and video_src.startswith('data:video/mp4;base64,'):
+        video_src = base64.b64decode(video_src.split(',')[1])
+    video = VideoReader(video_src, auto_init=accurate, num_threads=1)
+    assert video.num_frames > 0, "Invalid video format."
+    assert video.original_width > 0 and video.original_height > 0, (
+        "Invalid video format.")
+    assert video.avg_fps > 0, "Invalid video format."
+    return VideoSpec(media_type='video',
+                     height=video.original_height,
+                     width=video.original_width,
+                     num_frames=video.num_frames,
+                     fps=video.avg_fps,
+                     key_indices=video.key_indices,
+                     frame_time_info=video.frame_time_info)
+
+
+def timestamp_as_str(timestamp: float,
+                     timestamp_mode: str = "hh:mm:ss.fff") -> str:
+    """Convert a timestamp to a string in the format of HH:MM:SS.mmm."""
+    if timestamp_mode == "hh:mm:ss.fff":
+        return (datetime.fromtimestamp(timestamp,
+                                       tz=timezone.utc).strftime("%H:%M:%S") +
+                f".{int((timestamp % 1) * 1000):03d}")
+    elif timestamp_mode == "mm:ss.fff":
+        return (datetime.fromtimestamp(timestamp,
+                                       tz=timezone.utc).strftime("%M:%S") +
+                f".{int((timestamp % 1) * 1000):03d}")
+    elif timestamp_mode == "mm:ss":
+        return datetime.fromtimestamp(timestamp,
+                                      tz=timezone.utc).strftime("%M:%S")
+    else:
+        raise ValueError(f"Invalid timestamp mode: {timestamp_mode}")
+
+
+def navit_resize_image(
+    width: int,
+    height: int,
+    patch_size: int,
+    merge_kernel_size: int,
+    in_patch_limit: int,
+    patch_limit_on_one_side: int,
+    fixed_output_tokens: int | None,
+):
+    # Apply the patch limits.
+    s1 = math.sqrt(
+        in_patch_limit /
+        (max(1.0, width // patch_size) * max(1.0, height // patch_size)))
+    s2 = patch_limit_on_one_side * patch_size / width
+    s3 = patch_limit_on_one_side * patch_size / height
+    scale = min(1.0, s1, s2, s3)
+    new_w, new_h = max(1, int(width * scale)), max(1, int(height * scale))
+    new_w = min(new_w, patch_limit_on_one_side * patch_size)
+    new_h = min(new_h, patch_limit_on_one_side * patch_size)
+
+    # Calculate the padding to make the height and width divisible by the merge kernel size and patch size.
+    factor = merge_kernel_size * patch_size
+
+    pad_height = (factor - new_h % factor) % factor
+    pad_width = (factor - new_w % factor) % factor
+
+    if fixed_output_tokens is not None:
+        num_tokens = fixed_output_tokens
+    else:
+        # Calculate new dimensions after padding and patching
+        token_height = (new_h + pad_height) // factor
+        token_width = (new_w + pad_width) // factor
+
+        assert token_height * merge_kernel_size <= patch_limit_on_one_side, (
+            f"token_height {token_height} * merge_kernel_size {merge_kernel_size} > patch_limit_on_one_side {patch_limit_on_one_side}"
+        )
+        assert token_width * merge_kernel_size <= patch_limit_on_one_side, (
+            f"token_width {token_width} * merge_kernel_size {merge_kernel_size} > patch_limit_on_one_side {patch_limit_on_one_side}"
+        )
+
+        num_tokens = token_height * token_width
+    return {
+        "num_tokens": num_tokens,
+        "new_width": new_w,
+        "new_height": new_h,
+        "pad_width": pad_width,
+        "pad_height": pad_height,
+        "sampled_nframes": 1,
+    }
+
+
+def navit_resize_video(
+    width: int,
+    height: int,
+    nframes: int,
+    avg_fps: float,
+    sample_fps: float,
+    patch_size: int,
+    merge_kernel_size: int,
+    in_patch_limit_each_frame: int,
+    patch_limit_on_one_side: int,
+    in_patch_limit_total: int | None,
+    max_num_frames_each_video: int | None,
+    fixed_output_tokens_each_frame: int | None,
+):
+    sample_fps = min(sample_fps, avg_fps)
+    # Calculate the number of frames to sample based on target FPS
+    sampled_nframes = max(round(nframes * sample_fps / avg_fps), 1)
+    if max_num_frames_each_video is not None:
+        sampled_nframes = min(sampled_nframes, max_num_frames_each_video)
+
+    if in_patch_limit_total is not None:
+        in_patch_limit_each_frame = min(
+            round(in_patch_limit_total / sampled_nframes),
+            in_patch_limit_each_frame)
+
+    ret = navit_resize_image(
+        width,
+        height,
+        patch_size,
+        merge_kernel_size,
+        in_patch_limit_each_frame,
+        patch_limit_on_one_side,
+        fixed_output_tokens_each_frame,
+    )
+    ret["sampled_nframes"] = sampled_nframes
+    return ret
+
+
+def real_sample_fps_and_max_num_frames(
+    type_name: Literal["video", "video_chunk"],
+    sample_fps: float,
+    max_num_frames_each_video: int | None,
+) -> tuple[int, int | None]:
+    if type_name == "video":
+        return sample_fps, max_num_frames_each_video
+    elif type_name == "video_chunk":
+        max_num_frames_each_video = None
+        sample_fps = math.inf
+        return sample_fps, max_num_frames_each_video
+    else:
+        return math.inf, None
+
+
+def _to_pil(data: str | bytes):
+    if isinstance(data, Image.Image):
+
+        return data.convert("RGB")
+    elif isinstance(data, str):
+        if data.startswith("data:"):
+            raw_base64 = data.split(",")[1]
+            return Image.open(io.BytesIO(
+                base64.b64decode(raw_base64))).convert("RGB")
+        else:
+            return Image.open(data).convert("RGB")
+    elif isinstance(data, bytes):
+        return Image.open(io.BytesIO(data)).convert("RGB")
+    else:
+        raise ValueError(f"Unsupported data type: {type(data)}")
+
+
+def ensure_media_type(media: MediaInput) -> MediaInput:
+    if media['type'] == 'image':
+        media['image'] = _to_pil(media['image'])
+        return media
+    elif media['type'] == 'video_chunk':
+        media['video_chunk'] = [
+            _to_pil(frame) for frame in media['video_chunk']
+        ]
+        return media
+    else:
+        raise ValueError(f"Unsupported media type: {media['type']}")
+
+
+def image_to_np(
+    image: Image.Image,
+    resize_to: tuple[int, int] | None = None,
+    mode: str = "resize",
+    raise_error_for_ill_resize: bool = True,
+) -> np.ndarray:
+    """Convert an image to a numpy array.
+
+    Args:
+        content: The image to convert.
+        resize_to: The size to resize the image to.
+        mode: The mode to resize the image to.
+        raise_error_for_ill_resize: Whether to raise an error for ill-sized resize.
+
+    Returns:
+        A numpy array.
+    """
+    assert isinstance(image, Image.Image), "image must be a PIL Image"
+    if resize_to is not None:
+        if mode == "resize":
+            image = image.resize(resize_to, resample=Image.Resampling.BICUBIC)
+
+        elif mode == "rescale_and_pad_to_center":
+            scale = min(resize_to[0] / image.width,
+                        resize_to[1] / image.height, 1.0)
+            new_width = round(image.width * scale)
+            new_height = round(image.height * scale)
+            if new_width == 0 or new_height == 0:
+                if raise_error_for_ill_resize:
+                    raise ValueError(
+                        f"Invalid resize to: {resize_to}, from image size: {image.size}"
+                    )
+                else:
+                    return np.zeros((resize_to[1], resize_to[0], 3),
+                                    dtype=np.uint8)
+
+            image = image.resize((new_width, new_height),
+                                 resample=Image.Resampling.BICUBIC)
+            padding_left = (resize_to[0] - new_width) // 2
+            padding_right = resize_to[0] - new_width - padding_left
+            padding_top = (resize_to[1] - new_height) // 2
+            padding_bottom = resize_to[1] - new_height - padding_top
+            image = np.asarray(image)
+            image = np.pad(
+                image,
+                ((padding_top, padding_bottom), (padding_left, padding_right),
+                 (0, 0)),
+                mode="constant",
+                constant_values=0,
+            )
+            assert image.shape == (resize_to[1], resize_to[0], 3)
+
+        elif mode == "rescale_and_pad_to_rightbottom":
+            scale = min(resize_to[0] / image.width,
+                        resize_to[1] / image.height, 1.0)
+            new_width = round(image.width * scale)
+            new_height = round(image.height * scale)
+            if new_width == 0 or new_height == 0:
+                if raise_error_for_ill_resize:
+                    raise ValueError(
+                        f"Invalid resize to: {resize_to}, from image size: {image.size}"
+                    )
+                else:
+                    return np.zeros((resize_to[1], resize_to[0], 3),
+                                    dtype=np.uint8)
+
+            image = image.resize((new_width, new_height),
+                                 resample=Image.Resampling.BICUBIC)
+            padding_right = resize_to[0] - new_width
+            padding_bottom = resize_to[1] - new_height
+            image = np.asarray(image)
+            image = np.pad(
+                image,
+                ((0, padding_bottom), (0, padding_right), (0, 0)),
+                mode="constant",
+                constant_values=0,
+            )
+            assert image.shape == (resize_to[1], resize_to[0], 3)
+
+        else:
+            raise ValueError(f"Invalid mode: {mode}")
+
+    if isinstance(image, Image.Image):
+        return np.asarray(image)
+    else:
+        return image
+
+
+def navit_patchify(pixel_values: np.ndarray,
+                   patch_size: int) -> dict[str, np.ndarray]:
+    """Reshape the pixel values to a navit shape.
+
+    Args:
+        pixel_values: np.ndarray, shape (t, h, w, c)
+        patch_size: int
+
+    Returns:
+        dict[str, np.ndarray]
+        - patches: np.ndarray, shape (t * h//patch_size * w//patch_size, c, patch_size, patch_size)
+        - grid_thw: np.ndarray, (t, h//patch_size, w//patch_size)
+    """
+    T, H, W, C = pixel_values.shape
+    assert C == 3, "pixel_values must have 3 channels"
+
+    patches = pixel_values.reshape(T, H // patch_size, patch_size,
+                                   W // patch_size, patch_size, C)
+    # (T, H//patch_size, W//patch_size, C, patch_size, patch_size)
+    patches = patches.transpose(0, 1, 3, 5, 2, 4)
+    patches = patches.reshape(-1, C, patch_size, patch_size)
+    grid_thw = np.array([T, H // patch_size, W // patch_size])
+    return {"pixel_values": patches, "grid_thw": grid_thw}
+
+
+def normalize(x: np.ndarray,
+              mean,
+              std_inv,
+              pixels_dtype: np.dtype = np.float32) -> np.ndarray:
+    """Normalize the image.
+
+    Args:
+        x: The image to normalize. The shape is (..., 3). The dtype is uint8. The range is [0, 255].
+        mean: The mean of the image.
+        std_inv: The inverse of the std of the image.
+        pixels_dtype: The dtype of the image.
+    Returns:
+        The normalized image. The shape is (..., 3). The dtype is determined by the pixels_dtype.
+    """
+    x = (x / 255.0).astype(pixels_dtype)
+    x -= mean
+    x *= std_inv
+    return x
+
+
+def _to_tensor(data, **kwargs):
+    import torch
+
+    if isinstance(data, np.ndarray):
+        return torch.from_numpy(data).to(**kwargs)
+    elif isinstance(data, torch.Tensor):
+        return data.to(**kwargs)
+    elif isinstance(data, list):
+        return [_to_tensor(item, **kwargs) for item in data]
+    elif isinstance(data, tuple):
+        return tuple(_to_tensor(item, **kwargs) for item in data)
+    elif isinstance(data, dict):
+        return {k: _to_tensor(v, **kwargs) for k, v in data.items()}
+    elif data is None:
+        return None
+    else:
+        raise ValueError(f"Unsupported data type: {type(data)}")

mm_projector.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e7c6ce8c27424f292e708e7bbb48ade57ea9f1aaddd28bd6a1020a860d9db80c
+size 99117136

model-00001-of-00047.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7355ca8065dac64f806bb8f51921e9c86f428542815e4cbfbe67c207a9e0ff4d
+size 9996935800

model-00002-of-00047.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5a93bf6840c92c2b921a696b3268c3583f417ce4b85fdf912a3b164679d1df9b
+size 9998076664

model-00003-of-00047.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:240485288e04efae1c4dcda20eb13a4376efc5f69c0ea1b8c5e3d5d771707ea1
+size 9995585544

model-00004-of-00047.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:790fe046fb8b18f1c1bc829c32ae88b02aba7203a8fbdb1acf864b37dc838fae
+size 9998076664

model-00005-of-00047.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9c6a3381b16ff8f37bed8e6e020b338b569f1c927b9a3fa8c025c5c75c77ca79
+size 9997417192

model-00006-of-00047.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:902e254158f0fac8bb531ff85728de4c1580272464a3418d1e452d67ab9ee4d0
+size 9998081432

model-00007-of-00047.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:40299690b69637fc15fddf048b47ef0bc82a3c6d9685f0fd048a9823b426cdae
+size 9995590872

model-00008-of-00047.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:139f1c107077c04c2646d60d16f2c723d8be1876efa8e271eb135f4e564fdb31
+size 9999911744

model-00009-of-00047.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e1db5be0dac72aaaa446e1fa2437420cf9e47efe3c36c59c5471200be1dd4d23
+size 9998081760

model-00010-of-00047.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:617a2ca6fd37281f636ed819eab39dd8f2ebb21bc8bbf599f229537bdadb2024
+size 9995590752

model-00011-of-00047.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5099883c548f807bc1f2cde6188425ac6f40a9c3c244a4e18bfc008e572e7361
+size 9998081968

model-00012-of-00047.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:df6a0c6ab9fa7be42f0db0872880f1b5636e910b4f73aac5570c42354f05804c
+size 9997420424

model-00013-of-00047.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2bfd1392eda129e70128ef242284219e5acd07f3b77eeea9f7a6e6b48c3020b9
+size 9998081432

model-00014-of-00047.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6320d889ab91a2ba274bba38b8ea19b7540b20a6b484ec522ca8bd96ea5333ca
+size 9995590992

model-00015-of-00047.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:499d121db3e7e08e403a1665071bf55f710d88916de32519359691a571bcbb09
+size 9999911624

model-00016-of-00047.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b0bfeb109e4a1ca9327430de28f11e858d239505a4d68cc05f93440818c3352b
+size 9998081640

model-00017-of-00047.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ca88817f6f027fec523052c67e08de81a3751b3b44b45026a29321f32ba61b4b
+size 9995590752

model-00018-of-00047.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4d32422c750fe5520879ce63b037d2c7061abd0975c56011f757eea603bfbf4b
+size 9959557432

model-00019-of-00047.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:74ecfce9cddd66f2f6ef551c6f485c1eb2ce5f9f63a9953773c2ab50ce2eff68
+size 10000552960

model-00020-of-00047.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:23a23304160cf55ca305e920af3f47aabbfacb6a7eb3c2d6624e3cd45af79656
+size 9998081424

model-00021-of-00047.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:db14cdfbe3e53abbd291c230c148d9c21fd9834f1a6e6bdde8d860936a91c382
+size 9995591144

model-00022-of-00047.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bf5f1c293e95bc3f37f9da5b2c464d765b9150006695ef4b76d973c2fdc1ef75
+size 9999911480

model-00023-of-00047.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2fb70768ccd149d6df403ff319b831d3578c6c7ea7040bd6350d36201e2325ae
+size 9998081504

model-00024-of-00047.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ccf3b2e4ca1e8badbb7e0e3b5e7ed2f23a11a996ee35496f60748649ca45577e
+size 9995590752

model-00025-of-00047.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:67f233905c40eaa27e1d9174cefd28403ea55dd195f3f97dfa52b55eb480572c
+size 9999911904

model-00026-of-00047.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:daf23e3ccbf4b5324bcc77a208b4f91f1ab4965a24df62524e26e19a2fef2585
+size 9995591304

model-00027-of-00047.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1b4c7404684d1f71e6341660ce60b87184e9295c042c68816f70cf21e0d5d378
+size 9998081416

model-00028-of-00047.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a895f3e5a8150c74996caf4fe7c8b7f11f4ae99f838bcc3207af1a9f87010199
+size 9995591208

model-00029-of-00047.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5601bf446f5f778b3c2a26d023c78d21c0ed3200c1f2c1a13b082a94143937f5
+size 9999911360

model-00030-of-00047.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:355b7b4665a899fad5b5b42903c8ff8f58cee469883a7638e651f8ab14a92487
+size 9998081432

model-00031-of-00047.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7bc1ca71bffdd1a1eee33e43a328d525e23cf39db1b4812fdf55dd3cd90731a4
+size 9995590840

model-00032-of-00047.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d11bd3fe6f4c772a754ab4b19eca4e05c231c610d2f565aae9f1865723cfef10
+size 9999911784

model-00033-of-00047.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e39c9ff0d42c6052c538aaee35579a703993862601a340d0c4066ef0d2806bbd
+size 9995591128

model-00034-of-00047.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a81b529612c93893cf55dfb9c1f1a9a0ebdcca494f62d66181d9c6577a8ed434
+size 9998081416

model-00035-of-00047.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:70d19f0b6180627f5da59d38ca2d944b3937b77ba0302a5d60db7bfa7a69093c
+size 9995591272

model-00036-of-00047.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cab4d531d01434f35ec2809ad408519f52b733dd46f30adc43f3aaef6ce95933
+size 9999911176

model-00037-of-00047.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dc213d3550fc4e8482f88e08c791b4ba74fa8bb85ec2c6d7fc15661dac954055
+size 9998081432

model-00038-of-00047.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:50c4a5522bd9e4a34977f52fe55018812f483a2afc8c7a32c92cc94e268b474a
+size 9995590960

model-00039-of-00047.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ca17fd22173184cc0dd50977538725c11f2e6a44d65a48865dc8ea975cfffa67
+size 9999911664

model-00040-of-00047.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:103a33ce23be80b19b18c51eecb1cf15cd49584e22b9138b1d358e8720b2abf3
+size 9995591008