GLM-5.2-Vision: MoonViT tower + trained projector (MLX repackaging)

README.md

@@ -0,0 +1,84 @@
+---
+license: mit
+base_model: zai-org/GLM-5.2
+tags:
+- mlx
+- vision
+- glm
+- moonvit
+- exo
+library_name: mlx
+---
+
+# GLM-5.2-Vision — MoonViT tower + trained projector (MLX)
+
+Add **image input to any MLX quant of GLM-5.2** with a ~1 GB sidecar: the frozen
+MoonViT-3d vision tower from Kimi K2.6 plus the trained 49.5M-parameter
+PatchMerger projector from
+[baseten/GLM-5.2-Vision-NVFP4](https://huggingface.co/baseten/GLM-5.2-Vision-NVFP4)
+(Harry Partridge's vision retrofit), repackaged for Apple Silicon / MLX.
+The GLM-5.2 text backbone is untouched — text-only behavior stays byte-identical.
+
+## What's in this repo
+
+| File | What it is |
+|---|---|
+| `glm52_vision.safetensors` (+ index) | MoonViT-3d tower (417M, 27 layers, 1152-dim, bf16, `vision_tower.*`) **and** the trained projector (`mm_projector.pre_norm/linear_1/linear_2`, 1152 → 2×2 merge → 4608 → 6144) in one file. Kimi's original 7168-dim projector is removed — the GLM-trained one replaces it. |
+| `config.json` | `vision_config` (+ `text_config.hidden_size: 6144`, `media_placeholder_token_id: 154854`) |
+| `preprocessor_config.json`, `kimi_k25_*.py`, `media_utils.py` | Baseten's reference image processor (NaViT resize, patch 14, 2×2 merge, ≤4096 tokens/image) — the exact preprocessing the projector was trained against |
+
+GLM-5.2's stock tokenizer already contains the image tokens
+(`<|begin_of_image|>` 154830, `<|image|>` 154854, `<|end_of_image|>` 154831) —
+no tokenizer changes needed. Each image expands to its media-token count at the
+`<|image|>` position and the projected features are substituted at those
+embedding positions. You need a chat template that renders image content parts
+into the marker triplet (GLM's stock template does not; Baseten ships one in
+their repo).
+
+## Using with [exo](https://github.com/exo-explore/exo)
+
+exo's vision stack loads this repo directly as a `weights_repo`/`processor_repo`.
+Model card stanza:
+
+```toml
+[vision]
+image_token_id = 154854
+model_type = "kimi_vl"
+weights_repo = "<this repo id>"
+processor_repo = "<this repo id>"
+```
+
+Point the card's model at a directory containing your GLM-5.2 MLX quant with
+Baseten's `chat_template.jinja` and this repo's `config.json` additions
+(`vision_config` / `text_config` / `media_placeholder_token_id`). Assembly
+scripts (symlink the backbone — no weight duplication):
+[`build_glm52_vision_dir.py`](https://github.com/aidiffuser/exo/blob/update-latest/scripts/build_glm52_vision_dir.py)
+and
+[`build_glm52_vision_tower.py`](https://github.com/aidiffuser/exo/blob/update-latest/scripts/build_glm52_vision_tower.py).
+
+Verified live on a 2-Mac-Studio (M3 Ultra) tensor-parallel cluster over RDMA,
+against both `mlx-community/GLM-5.2-DQ4plus-q8` and `mlx-community/GLM-5.2-mxfp4`:
+temp-0 deterministic, no cross-image cache bleed, text-only outputs identical
+to the plain model.
+
+## What to expect (honest notes)
+
+- Scene understanding, objects, colors, spatial layout and orientation: **good**.
+  Reads real-world photos (e.g. product packaging labels) usefully.
+- Dense/synthetic text OCR and fine-grained counting: **weak** — this is a 50M
+  projector retrofit at the source's reported ~Haiku-4.5-level MMMU-Pro (55%),
+  not a natively-trained VLM. A Kimi K2.6 A/B on the same pipeline is clearly
+  stronger at fine detail.
+- The projector was trained against the bf16/NVFP4 backbone; serving quantized
+  backbones (mxfp4 / DQ4) costs some additional vision quality, never text
+  quality.
+
+## Provenance & license
+
+MIT, following all parents. Full chain: **Z.ai** (GLM-5.2, MIT) →
+**Moonshot AI** (Kimi K2.6 MoonViT tower, Modified MIT) → **Harry Partridge /
+Baseten** (projector training + reference processor,
+[baseten/GLM-5.2-Vision-NVFP4](https://huggingface.co/baseten/GLM-5.2-Vision-NVFP4),
+MIT) → **exolabs** (original K2.6 tower extraction for MLX) → this repackaging
+(tensor remap documented in the build script). None of the upstream teams were
+involved in this packaging.

config.json

@@ -0,0 +1,37 @@
+{
+ "source": "MoonViT tower from moonshotai/Kimi-K2.6 (via exolabs extract) + trained projector from baseten/GLM-5.2-Vision-NVFP4",
+ "component": "vision_tower + mm_projector",
+ "model_type": "kimi_k25",
+ "vision_config": {
+  "patch_size": 14,
+  "init_pos_emb_height": 64,
+  "init_pos_emb_width": 64,
+  "init_pos_emb_time": 4,
+  "pos_emb_type": "divided_fixed",
+  "num_attention_heads": 16,
+  "num_hidden_layers": 27,
+  "hidden_size": 1152,
+  "intermediate_size": 4304,
+  "vt_num_attention_heads": 16,
+  "vt_num_hidden_layers": 27,
+  "vt_hidden_size": 1152,
+  "vt_intermediate_size": 4304,
+  "merge_kernel_size": [
+   2,
+   2
+  ],
+  "video_attn_type": "spatial_temporal",
+  "merge_type": "sd2_tpool",
+  "mm_projector_type": "patchmerger",
+  "mm_hidden_size": 1152,
+  "projector_hidden_act": "gelu",
+  "projector_ln_eps": 1e-05,
+  "text_hidden_size": 6144
+ },
+ "text_config": {
+  "hidden_size": 6144
+ },
+ "media_placeholder_token_id": 154854,
+ "original_dtype": "bfloat16",
+ "num_tensors": 335
+}

glm52_vision.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:18e6e27be3c1a46951dd6812c4b29a18469a6ddcf5483790c93a18419e2a6d52
+size 932887076

glm52_vision.safetensors.index.json

@@ -0,0 +1,342 @@
+{
+ "metadata": {
+  "total_size": 932887076
+ },
+ "weight_map": {
+  "vision_tower.patch_embed.proj.weight": "glm52_vision.safetensors",
+  "vision_tower.patch_embed.proj.bias": "glm52_vision.safetensors",
+  "vision_tower.patch_embed.pos_emb.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.final_layernorm.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.9.wqkv.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.9.mlp.fc1.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.9.mlp.fc1.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.9.mlp.fc0.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.8.wqkv.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.8.norm1.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.8.norm0.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.8.mlp.fc1.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.8.mlp.fc1.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.7.wqkv.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.7.wo.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.7.wo.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.7.norm1.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.7.norm1.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.9.wo.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.7.norm0.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.7.norm0.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.7.mlp.fc1.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.7.mlp.fc1.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.7.mlp.fc0.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.6.norm1.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.5.norm1.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.5.norm0.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.5.mlp.fc0.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.4.wo.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.4.wo.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.4.norm0.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.7.wqkv.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.4.mlp.fc1.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.4.mlp.fc1.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.4.mlp.fc0.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.3.wqkv.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.3.wo.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.6.norm0.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.3.wo.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.3.norm1.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.3.norm1.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.3.norm0.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.5.norm1.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.26.wo.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.26.norm1.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.26.norm0.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.26.norm0.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.26.mlp.fc1.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.26.mlp.fc0.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.25.wqkv.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.25.wqkv.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.25.norm1.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.25.norm0.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.9.norm1.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.25.mlp.fc1.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.6.mlp.fc1.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.25.mlp.fc0.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.9.norm1.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.24.wo.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.24.norm1.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.24.norm0.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.6.norm1.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.24.mlp.fc1.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.23.wqkv.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.23.wqkv.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.23.wo.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.23.mlp.fc1.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.23.mlp.fc1.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.23.mlp.fc0.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.17.mlp.fc0.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.22.wqkv.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.22.wo.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.22.mlp.fc1.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.24.mlp.fc0.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.21.wqkv.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.11.mlp.fc0.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.21.wqkv.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.21.wo.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.23.wo.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.18.mlp.fc0.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.21.wo.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.3.wqkv.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.18.norm0.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.21.norm0.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.21.mlp.fc0.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.2.mlp.fc0.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.20.wqkv.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.10.mlp.fc1.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.13.norm0.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.20.wo.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.24.wqkv.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.20.mlp.fc1.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.20.mlp.fc0.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.15.mlp.fc0.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.20.mlp.fc0.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.26.wqkv.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.2.wqkv.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.3.norm0.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.2.mlp.fc1.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.2.wqkv.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.6.wo.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.20.norm0.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.2.wo.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.5.mlp.fc0.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.0.norm1.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.2.wo.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.22.norm1.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.8.wqkv.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.5.wo.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.2.norm1.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.7.mlp.fc0.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.2.norm0.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.3.mlp.fc0.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.2.norm0.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.9.wo.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.19.norm1.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.11.mlp.fc1.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.21.norm0.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.20.wo.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.5.mlp.fc1.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.19.norm1.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.19.norm0.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.4.norm1.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.19.mlp.fc1.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.18.wqkv.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.8.mlp.fc0.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.18.wqkv.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.8.mlp.fc0.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.18.wo.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.18.norm0.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.0.mlp.fc1.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.18.mlp.fc1.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.4.wqkv.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.11.wqkv.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.17.wo.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.17.wo.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.24.wqkv.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.15.wqkv.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.17.norm1.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.17.norm0.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.17.norm0.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.24.norm1.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.13.mlp.fc1.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.17.mlp.fc1.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.26.mlp.fc0.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.14.norm0.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.17.mlp.fc0.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.8.norm0.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.10.mlp.fc0.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.16.wqkv.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.6.norm0.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.10.mlp.fc1.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.19.wo.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.18.wo.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.17.norm1.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.16.wo.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.1.mlp.fc1.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.14.norm1.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.20.norm0.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.0.mlp.fc0.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.19.mlp.fc1.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.23.norm0.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.12.mlp.fc0.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.16.norm1.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.6.wqkv.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.3.mlp.fc1.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.25.wo.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.12.norm1.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.12.norm0.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.16.norm1.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.5.mlp.fc1.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.21.norm1.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.24.norm0.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.16.norm0.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.21.mlp.fc1.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.6.wqkv.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.26.wo.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.16.mlp.fc1.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.23.norm1.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.16.mlp.fc1.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.18.mlp.fc1.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.16.mlp.fc0.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.0.mlp.fc0.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.22.norm0.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.20.norm1.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.22.mlp.fc1.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.19.wqkv.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.8.norm1.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.15.wo.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.15.wo.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.24.mlp.fc1.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.15.norm1.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.25.mlp.fc0.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.15.mlp.fc1.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.final_layernorm.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.15.mlp.fc1.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.15.norm1.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.18.norm1.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.15.mlp.fc0.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.20.wqkv.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.14.mlp.fc1.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.13.wqkv.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.13.wqkv.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.13.wo.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.17.mlp.fc1.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.15.wqkv.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.14.mlp.fc0.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.22.wqkv.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.14.wqkv.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.25.wo.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.22.mlp.fc0.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.13.norm1.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.22.norm1.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.5.wo.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.17.wqkv.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.12.wo.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.14.norm0.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.21.norm1.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.11.wo.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.13.mlp.fc0.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.12.wqkv.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.17.wqkv.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.14.wqkv.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.13.mlp.fc1.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.9.norm0.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.4.norm1.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.12.wqkv.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.2.norm1.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.26.wqkv.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.19.norm0.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.12.norm1.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.1.norm0.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.23.norm0.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.19.mlp.fc0.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.14.mlp.fc1.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.15.norm0.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.13.norm0.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.10.wo.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.22.norm0.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.12.mlp.fc1.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.12.mlp.fc0.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.0.wo.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.25.mlp.fc1.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.13.norm1.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.3.mlp.fc1.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.20.mlp.fc1.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.6.mlp.fc0.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.11.wqkv.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.11.wo.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.11.norm0.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.19.wqkv.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.5.wqkv.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.11.mlp.fc0.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.5.wqkv.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.23.norm1.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.0.wo.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.5.norm0.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.16.wqkv.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.15.norm0.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.4.mlp.fc0.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.1.wo.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.9.mlp.fc0.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.25.norm0.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.20.norm1.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.12.wo.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.26.mlp.fc1.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.10.wqkv.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.22.mlp.fc0.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.10.norm1.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.14.wo.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.6.mlp.fc1.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.10.norm0.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.4.norm0.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.2.mlp.fc1.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.10.norm0.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.10.mlp.fc0.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.26.norm1.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.21.mlp.fc0.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.16.mlp.fc0.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.4.wqkv.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.1.norm1.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.13.mlp.fc0.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.8.wo.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.1.wo.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.19.wo.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.9.wqkv.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.11.norm0.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.11.norm1.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.1.norm1.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.0.wqkv.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.25.norm1.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.2.mlp.fc0.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.16.norm0.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.3.mlp.fc0.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.24.wo.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.11.mlp.fc1.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.16.wo.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.1.norm0.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.1.mlp.fc0.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.14.norm1.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.18.norm1.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.1.mlp.fc1.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.11.norm1.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.12.norm0.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.0.norm1.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.0.norm0.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.0.norm0.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.1.wqkv.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.6.wo.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.0.wqkv.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.8.wo.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.23.mlp.fc0.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.18.mlp.fc0.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.9.norm0.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.22.wo.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.10.wo.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.13.wo.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.6.mlp.fc0.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.0.mlp.fc1.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.12.mlp.fc1.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.10.wqkv.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.24.mlp.fc0.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.21.mlp.fc1.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.1.wqkv.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.1.mlp.fc0.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.14.mlp.fc0.weight": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.10.norm1.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.19.mlp.fc0.bias": "glm52_vision.safetensors",
+  "vision_tower.encoder.blocks.14.wo.bias": "glm52_vision.safetensors",
+  "mm_projector.pre_norm.weight": "glm52_vision.safetensors",
+  "mm_projector.linear_2.weight": "glm52_vision.safetensors",
+  "mm_projector.pre_norm.bias": "glm52_vision.safetensors",
+  "mm_projector.linear_2.bias": "glm52_vision.safetensors",
+  "mm_projector.linear_1.weight": "glm52_vision.safetensors",
+  "mm_projector.linear_1.bias": "glm52_vision.safetensors"
+ }
+}

kimi_k25_processor.py

@@ -0,0 +1,208 @@
+from transformers.feature_extraction_utils import BatchFeature
+from transformers.processing_utils import ProcessorMixin
+from transformers.utils import logging
+
+logger = logging.get_logger(__name__)
+
+
+class KimiK25Processor(ProcessorMixin):
+    r"""
+    Constructs a KimiK25 processor which wraps a KimiK25 image processor and a tokenizer into a single processor.
+
+    [`KimiK25Processor`] offers all the functionalities of [`KimiK25ImageProcessor`] and [`TikTokenTokenizer`]. See the
+    [`~KimiK25Processor.__call__`] and [`~KimiK25Processor.decode`] for more information.
+
+    Args:
+        image_processor ([`KimiK25ImageProcessor`], *optional*):
+            The image processor is a required input.
+        tokenizer ([`TikTokenTokenizer`], *optional*):
+            The tokenizer is a required input.
+        chat_template (`str`, *optional*): A Jinja template which will be used to convert lists of messages
+            in a chat into a tokenizable string.
+    """
+
+    attributes = ["image_processor", "tokenizer"]
+    valid_kwargs = ["chat_template"]
+    image_processor_class = "AutoImageProcessor"
+    tokenizer_class = "AutoTokenizer"
+
+    def __init__(
+        self,
+        image_processor=None,
+        tokenizer=None,
+        chat_template=None,
+        **kwargs,
+    ):
+        super().__init__(image_processor,
+                         tokenizer,
+                         chat_template=chat_template)
+        self.media_processor = image_processor
+        # A special temporal placeholder to be replaced by actual video placeholders
+        self.video_placeholder = "<|kimi_k25_video_placeholder|>"
+
+    def update_raw_text(self, text: str, video_prompts: list[str]) -> str:
+        # replace video prompt in text with video chunk prompts
+        video_count = text.count(self.video_placeholder)
+        if video_count == 0:
+            return text
+        assert video_count == len(video_prompts)
+        text_parts = text.split(self.video_placeholder)
+        assert len(text_parts) == len(video_prompts) + 1
+        text = "".join([
+            text_parts[i] + video_prompts[i] for i in range(len(video_prompts))
+        ])
+        text += text_parts[-1]
+        return text
+
+    def preprocess_medias(self, medias: list[dict]) -> list[dict]:
+        updated_medias = []
+        video_prompts = []
+        for media in medias:
+            if media['type'] == 'image':
+                updated_medias.append(media)
+            elif media['type'] == 'video':
+                video_chunks = self.media_processor.split_video_chunks(
+                    media['video'])
+                updated_medias.extend(video_chunks)
+                video_prompts.append("".join(
+                    [vc['prompt'] for vc in video_chunks]))
+            else:
+                raise ValueError(f"unsupported media type: {media['type']}")
+        return updated_medias, video_prompts
+
+    # glm5v: the image placeholder expanded per patch (GLM <|image|> = 154854).
+    # The chat template wraps it as <|begin_of_image|><|image|><|end_of_image|>.
+    GLM5V_IMAGE_TOKEN = "<|image|>"
+
+    def __call__(self,
+                 messages: list[dict] = None,
+                 medias: list[dict] = None,
+                 text: str = None,
+                 images: list = None,
+                 return_tensors: str = "pt",
+                 **kwargs) -> BatchFeature:
+        """
+        Process multimodal inputs for Kimi-K2.5 model.
+
+        This processor accepts ordered messages and extracts both media and text in a single pass.
+        text will be automatically updated if video input detected in messages
+
+        Args:
+            messages: List of message dicts with 'role' and 'content' fields.
+                     If provided, medias and text will be extracted automatically.
+            medias: Pre-extracted list of media dicts. If None, extracted from messages.
+            text: Pre-formatted text string. If None, generated via apply_chat_template.
+            images: Standard HF VLM API (``processor(text=..., images=[...])``), as
+                    called by generic drivers (e.g. slime's rollout prompt prep).
+                    Converted to ``medias`` and each ``<|image|>`` placeholder in
+                    ``text`` is expanded to that image's per-patch token count, so
+                    the returned ``input_ids`` align with ``pixel_values`` (same
+                    semantics as Qwen-family processors and the SGLang serving-layer
+                    wrapper).
+            return_tensors: Format of returned tensors ('pt', 'np', 'tf'). Default: 'pt'.
+            **kwargs: Additional arguments passed to tokenizer.apply_chat_template.
+
+        Returns:
+            BatchFeature with fields: input_ids, attention_mask, pixel_values, grid_thws.
+        """
+        if images is not None and medias is None and text is not None:
+            # Standard HF call: expand placeholders, run the media preprocess, and
+            # return with STANDARD-HF dtypes: input_ids/attention_mask as python
+            # lists (callers like slime's rollout do `sample.tokens += tokens`),
+            # media tensors as `return_tensors` (default pt) for the train side.
+            if not isinstance(images, (list, tuple)):
+                images = [images]
+            medias = [{"type": "image", "image": img} for img in images]
+            parts = text.split(self.GLM5V_IMAGE_TOKEN)
+            if len(parts) - 1 != len(images):
+                raise ValueError(
+                    f"got {len(images)} images but {len(parts) - 1} "
+                    f"{self.GLM5V_IMAGE_TOKEN!r} placeholders in text")
+            expanded = [parts[0]]
+            for media, part in zip(medias, parts[1:]):
+                num_tokens = self.media_processor.media_tokens_calculator(media)
+                expanded.append(self.GLM5V_IMAGE_TOKEN * num_tokens + part)
+            text = "".join(expanded)
+
+            updated_medias, video_prompts = self.preprocess_medias(medias)
+            preprocessed = self.media_processor.preprocess(
+                updated_medias, return_tensors=return_tensors)
+            text = self.update_raw_text(text, video_prompts)
+            text_inputs = self.tokenizer([text])  # no return_tensors -> lists
+            data = {**text_inputs, **preprocessed.data}
+            # Qwen-convention key: downstream training forwards take
+            # `image_grid_thw` (same rename the SGLang wrapper applies).
+            if "grid_thws" in data:
+                data["image_grid_thw"] = data.pop("grid_thws")
+            return BatchFeature(data=data)
+
+        if messages is None and (medias is None or text is None):
+            raise ValueError(
+                "Provide either 'messages' or both 'medias' and 'text'")
+
+        if medias is not None and text is not None:
+            updated_medias, video_prompts = self.preprocess_medias(medias)
+            preprocessed = self.media_processor.preprocess(
+                updated_medias, return_tensors=return_tensors)
+            text = self.update_raw_text(text, video_prompts)
+            text_inputs = self.tokenizer(text, return_tensors=return_tensors)
+            return BatchFeature(data={**text_inputs, **preprocessed.data})
+
+        if medias is None:
+            medias = self._extract_medias_from_messages(messages)
+        updated_medias, video_prompts = self.preprocess_medias(medias)
+        preprocessed = self.media_processor.preprocess(
+            updated_medias, return_tensors=return_tensors)
+
+        # Generate text if not provided
+        if text is None:
+            text = self.tokenizer.apply_chat_template(messages, **kwargs)
+
+        text = self.update_raw_text(text, video_prompts)
+
+        text_inputs = self.tokenizer(text, return_tensors=return_tensors)
+        return BatchFeature(data={**text_inputs, **preprocessed.data})
+
+    @staticmethod
+    def _extract_medias_from_messages(messages: list[dict]) -> list[dict]:
+        """
+        Extract media items from messages in a single pass.
+        
+        This is an optimized version that processes messages only once.
+        Kept as internal method since external callers should use __call__.
+        """
+        medias = []
+        for msg in messages:
+            if msg['role'] != 'user' or not msg.get('content'):
+                continue
+
+            for content_part in msg['content']:
+                if not isinstance(content_part, dict):
+                    continue
+
+                content_type = content_part.get('type')
+                if content_type in ['video_url', 'video']:
+                    medias.append({
+                        'type': 'video',
+                        'video': content_part['video_url']['url'],
+                        'first_frame_timestamp': 0.0
+                    })
+                elif content_type in ['image_url', 'image']:
+                    medias.append({
+                        'type': 'image',
+                        'image': content_part['image_url'],
+                    })
+        return medias
+
+    def apply_chat_template(self, messages, **kwargs):
+        return self.tokenizer.apply_chat_template(messages, **kwargs)
+
+    def batch_decode(self, *args, **kwargs):
+        return self.tokenizer.batch_decode(*args, **kwargs)
+
+    def decode(self, *args, **kwargs):
+        return self.tokenizer.decode(*args, **kwargs)
+
+    @property
+    def model_input_names(self):
+        return ['input_ids', 'attention_mask', 'pixel_values', 'grid_thws']

kimi_k25_vision_processing.py

@@ -0,0 +1,251 @@
+"""Image processor class for Kimi-K2.5.
+"""
+
+import json
+from typing import Any, Dict, Optional, Union
+
+import numpy as np
+import torch
+from PIL import Image
+from transformers.image_processing_utils import (BaseImageProcessor,
+                                                 BatchFeature)
+from transformers.utils import TensorType
+
+from .media_utils import (MediaInput, VideoChunkInput, _to_tensor,
+                          ensure_media_type, get_video_meta, image_to_np,
+                          navit_patchify, navit_resize_image,
+                          navit_resize_video, normalize,
+                          real_sample_fps_and_max_num_frames, timestamp_as_str)
+
+try:
+    from mecord import VideoReader
+except ImportError:
+    VideoReader = None
+
+
+def resampling(video_bytes: bytes,
+               sample_indices: list[int],
+               key_indices=None,
+               frame_time_info=None,
+               num_threads=4) -> str:
+    video = VideoReader(video_bytes,
+                        num_threads=num_threads,
+                        frame_time_info=frame_time_info,
+                        key_indices=key_indices)
+    # extract target frames
+    frames = video[sample_indices]
+    frames = [Image.fromarray(frame) for frame in frames]
+    return frames
+
+
+class KimiK25VisionProcessor(BaseImageProcessor):
+    model_type = "kimi_k25"
+
+    def __init__(
+        self,
+        media_proc_cfg: dict,
+        **kwargs,
+    ):
+        super().__init__(**kwargs)
+        self.media_proc_cfg = media_proc_cfg
+        self.num_frames_per_chunk = media_proc_cfg[
+            'temporal_merge_kernel_size']
+
+    def media_tokens_calculator(self, media: MediaInput):
+        media = ensure_media_type(media)
+        ret = self.get_resize_config(media)
+        return ret['num_tokens']
+
+    @classmethod
+    def make_chunk_prompt(cls, timestamp_text: str) -> str:
+        return f"{timestamp_text}<|media_begin|>video<|media_content|><|media_pad|><|media_end|>"
+
+    def split_video_chunks(self,
+                           video_url: str | bytes) -> list[list[Image.Image]]:
+        # video_url should be base64 str or bytes
+        video_spec = get_video_meta(video_url)
+        sample_fps = min(self.media_proc_cfg['sample_fps'], video_spec.fps)
+        sampled_nframes = max(
+            round(video_spec.num_frames * sample_fps / video_spec.fps), 1)
+        frame_inds = np.linspace(0, video_spec.num_frames - 1,
+                                 sampled_nframes).round().astype(int)
+        frame_inds = frame_inds.tolist()
+        sampled_frame_ids = []
+        temporal_merge_kernel_size = self.media_proc_cfg[
+            "temporal_merge_kernel_size"]
+        num_chunks = 0
+        chunk_timestamp = []
+        for i in range(0, len(frame_inds), temporal_merge_kernel_size):
+            sampled_frame_ids.extend(frame_inds[i:i +
+                                                temporal_merge_kernel_size])
+            start_time = frame_inds[i] / float(video_spec.fps)
+            timestamp_text = timestamp_as_str(
+                start_time, self.media_proc_cfg["timestamp_mode"])
+            chunk_timestamp.append(timestamp_text)
+            num_chunks += 1
+
+        sampled_frames = resampling(video_url, sampled_frame_ids)
+        chunks = []
+        for chunk_id in range(num_chunks):
+            chunk = sampled_frames[chunk_id *
+                                   temporal_merge_kernel_size:(chunk_id + 1) *
+                                   temporal_merge_kernel_size]
+            chunks.append(
+                VideoChunkInput(type="video_chunk",
+                                video_chunk=chunk,
+                                prompt=self.make_chunk_prompt(
+                                    chunk_timestamp[chunk_id])))
+        return chunks
+
+    def get_resize_config(self, media_input: MediaInput) -> dict:
+        if media_input['type'] == 'image':
+            w, h = media_input['image'].size
+            ret = navit_resize_image(
+                w, h, self.media_proc_cfg['patch_size'],
+                self.media_proc_cfg['merge_kernel_size'],
+                self.media_proc_cfg['in_patch_limit'],
+                self.media_proc_cfg['patch_limit_on_one_side'],
+                self.media_proc_cfg['fixed_output_tokens'])
+            return ret
+        elif media_input['type'] == 'video_chunk':
+            frame = media_input['video_chunk'][0]
+            width, height = frame.size
+            num_frames = len(media_input["video_chunk"])
+            fps = 1.0
+
+            sample_fps, max_num_frames_each_video = real_sample_fps_and_max_num_frames(
+                media_input["type"],
+                self.media_proc_cfg['sample_fps'],
+                self.media_proc_cfg['max_num_frames_each_video'],
+            )
+
+            in_patch_limit_each_frame = self.media_proc_cfg[
+                'in_patch_limit_each_frame']
+            if in_patch_limit_each_frame is None:
+                in_patch_limit_each_frame = self.media_proc_cfg[
+                    'in_patch_limit']
+
+            ret = navit_resize_video(
+                width,
+                height,
+                num_frames,
+                fps,
+                sample_fps,
+                self.media_proc_cfg['patch_size'],
+                self.media_proc_cfg['merge_kernel_size'],
+                in_patch_limit_each_frame,
+                self.media_proc_cfg['patch_limit_on_one_side'],
+                self.media_proc_cfg['in_patch_limit_video'],
+                max_num_frames_each_video,
+                self.media_proc_cfg['fixed_output_tokens'],
+            )
+            return ret
+        else:
+            raise ValueError("Unsupported type: {}".format(
+                media_input['type']))
+
+    def resize_image(self, image: Image.Image, new_width: int, new_height: int,
+                     pad_width: int, pad_height: int) -> np.ndarray:
+        image_np = image_to_np(image, (new_width, new_height), "resize")
+        image_np = np.pad(
+            image_np,
+            ((0, pad_height), (0, pad_width), (0, 0)),
+            mode="constant",
+            constant_values=0,
+        )
+        return image_np
+
+    def preprocess(
+        self,
+        medias: list[MediaInput],
+        return_tensors: Optional[Union[str, TensorType]] = None,
+    ) -> BatchFeature:
+        """
+        Preprocess a atom vision input (images/video_chunk) into model-ready tensors.
+        
+        Args:
+            medias: List of MediaInput.
+            return_tensors: Desired output format ('pt', 'np', 'tf', or None).
+        
+        Returns:
+            BatchFeature containing 'pixel_values' and 'grid_thws' tensors.
+        """
+        if not isinstance(medias, list):
+            medias = [medias]
+        if medias:
+            pixel_values = []
+            for item in medias:
+                item = ensure_media_type(item)
+                resize_config = self.get_resize_config(item)
+                new_width, new_height, pad_width, pad_height = resize_config[
+                    'new_width'], resize_config['new_height'], resize_config[
+                        'pad_width'], resize_config['pad_height']
+                if item['type'] == 'image':
+                    image = item['image']
+                    image_np = self.resize_image(image, new_width, new_height,
+                                                 pad_width, pad_height)
+                    pixel_values.append(np.expand_dims(image_np, axis=0))
+                elif item['type'] == 'video_chunk':
+                    pixels = []
+                    for frame in item['video_chunk']:
+                        frame_np = self.resize_image(frame, new_width,
+                                                     new_height, pad_width,
+                                                     pad_height)
+                        pixels.append(frame_np)
+                    pixel_values.append(np.stack(pixels, axis=0))
+                else:
+                    raise ValueError("Unsupported type: {}".format(
+                        item['type']))
+            normalized_pixel_values = []
+            image_std_inv = 1.0 / np.array(self.media_proc_cfg['image_std'])
+            image_mean = np.array(self.media_proc_cfg['image_mean'])
+            for pixels in pixel_values:
+                pixels = normalize(pixels, image_mean, image_std_inv)
+                pixels_and_thw = navit_patchify(
+                    pixels,
+                    self.media_proc_cfg['patch_size'],
+                )
+                normalized_pixel_values.append(pixels_and_thw)
+
+            pixel_values = torch.cat([
+                _to_tensor(pixel_value['pixel_values'])
+                for pixel_value in normalized_pixel_values
+            ])
+            grid_thws = torch.cat([
+                _to_tensor(pixel_value['grid_thw'],
+                           dtype=torch.int64).unsqueeze(0)
+                for pixel_value in normalized_pixel_values
+            ])
+
+            data = {
+                'pixel_values': pixel_values,
+                'grid_thws': grid_thws,
+            }
+
+        else:
+            data = {}
+
+        return BatchFeature(data=data, tensor_type=return_tensors)
+
+    def __repr__(self):
+        return f"KimiK25VisionProcessor(media_proc_cfg={self.media_proc_cfg})"
+
+    def to_dict(self) -> Dict[str, Any]:
+        output = super().to_dict()
+        output["media_proc_cfg"] = self.media_proc_cfg
+        if "media_processor" in output:
+            del output["media_processor"]
+        return output
+
+    @classmethod
+    def from_dict(cls, config_dict: Dict[str, Any], **kwargs):
+        config = config_dict.copy()
+        media_proc_cfg = config.pop("media_proc_cfg", {})
+        return cls(media_proc_cfg=media_proc_cfg, **config, **kwargs)
+
+    def to_json_string(self):
+        dictionary = self.to_dict()
+        for key, value in dictionary.items():
+            if hasattr(value, 'tolist'):
+                dictionary[key] = value.tolist()
+        return json.dumps(dictionary, indent=2, sort_keys=True) + "\n"

media_utils.py

@@ -0,0 +1,368 @@
+import base64
+import io
+import math
+import os
+from datetime import datetime, timezone
+from typing import List, Literal, Optional, TypedDict
+
+import numpy as np
+from PIL import Image
+from pydantic import BaseModel, Field
+
+try:
+    from mecord import VideoReader
+except ImportError:
+    VideoReader = None
+
+
+class VideoSpec(BaseModel):
+    media_type: str = Literal['video']
+    height: int = Field(..., gt=0, description="video frame height")
+    width: int = Field(..., gt=0, description="video frame width")
+    num_frames: int = Field(..., gt=0, description="num frames")
+    fps: float = Field(..., gt=0, description="average fps")
+
+    # optional, help to accelerate video reading
+    key_indices: list[int] = Field(None, description="key indices")
+    frame_time_info: dict = Field(None, description="frame time info")
+
+
+class ImageInput(TypedDict):
+    type: Literal['image']
+    image: Image.Image
+
+
+class VideoChunkInput(TypedDict):
+    type: Literal['video_chunk']
+    video_chunk: List[Image.Image]
+    prompt: Optional[str] = None
+
+
+MediaInput = ImageInput | VideoChunkInput
+
+
+def get_video_meta(video_src: bytes | str | os.PathLike,
+                   accurate: bool = True) -> dict:
+    """Get the dimensions of a video."""
+    if isinstance(video_src, os.PathLike):
+        video_src = str(video_src)
+    # if b64 string, decode to bytes
+    if isinstance(video_src,
+                  str) and video_src.startswith('data:video/mp4;base64,'):
+        video_src = base64.b64decode(video_src.split(',')[1])
+    video = VideoReader(video_src, auto_init=accurate, num_threads=1)
+    assert video.num_frames > 0, "Invalid video format."
+    assert video.original_width > 0 and video.original_height > 0, (
+        "Invalid video format.")
+    assert video.avg_fps > 0, "Invalid video format."
+    return VideoSpec(media_type='video',
+                     height=video.original_height,
+                     width=video.original_width,
+                     num_frames=video.num_frames,
+                     fps=video.avg_fps,
+                     key_indices=video.key_indices,
+                     frame_time_info=video.frame_time_info)
+
+
+def timestamp_as_str(timestamp: float,
+                     timestamp_mode: str = "hh:mm:ss.fff") -> str:
+    """Convert a timestamp to a string in the format of HH:MM:SS.mmm."""
+    if timestamp_mode == "hh:mm:ss.fff":
+        return (datetime.fromtimestamp(timestamp,
+                                       tz=timezone.utc).strftime("%H:%M:%S") +
+                f".{int((timestamp % 1) * 1000):03d}")
+    elif timestamp_mode == "mm:ss.fff":
+        return (datetime.fromtimestamp(timestamp,
+                                       tz=timezone.utc).strftime("%M:%S") +
+                f".{int((timestamp % 1) * 1000):03d}")
+    elif timestamp_mode == "mm:ss":
+        return datetime.fromtimestamp(timestamp,
+                                      tz=timezone.utc).strftime("%M:%S")
+    else:
+        raise ValueError(f"Invalid timestamp mode: {timestamp_mode}")
+
+
+def navit_resize_image(
+    width: int,
+    height: int,
+    patch_size: int,
+    merge_kernel_size: int,
+    in_patch_limit: int,
+    patch_limit_on_one_side: int,
+    fixed_output_tokens: int | None,
+):
+    # Apply the patch limits.
+    s1 = math.sqrt(
+        in_patch_limit /
+        (max(1.0, width // patch_size) * max(1.0, height // patch_size)))
+    s2 = patch_limit_on_one_side * patch_size / width
+    s3 = patch_limit_on_one_side * patch_size / height
+    scale = min(1.0, s1, s2, s3)
+    new_w, new_h = max(1, int(width * scale)), max(1, int(height * scale))
+    new_w = min(new_w, patch_limit_on_one_side * patch_size)
+    new_h = min(new_h, patch_limit_on_one_side * patch_size)
+
+    # Calculate the padding to make the height and width divisible by the merge kernel size and patch size.
+    factor = merge_kernel_size * patch_size
+
+    pad_height = (factor - new_h % factor) % factor
+    pad_width = (factor - new_w % factor) % factor
+
+    if fixed_output_tokens is not None:
+        num_tokens = fixed_output_tokens
+    else:
+        # Calculate new dimensions after padding and patching
+        token_height = (new_h + pad_height) // factor
+        token_width = (new_w + pad_width) // factor
+
+        assert token_height * merge_kernel_size <= patch_limit_on_one_side, (
+            f"token_height {token_height} * merge_kernel_size {merge_kernel_size} > patch_limit_on_one_side {patch_limit_on_one_side}"
+        )
+        assert token_width * merge_kernel_size <= patch_limit_on_one_side, (
+            f"token_width {token_width} * merge_kernel_size {merge_kernel_size} > patch_limit_on_one_side {patch_limit_on_one_side}"
+        )
+
+        num_tokens = token_height * token_width
+    return {
+        "num_tokens": num_tokens,
+        "new_width": new_w,
+        "new_height": new_h,
+        "pad_width": pad_width,
+        "pad_height": pad_height,
+        "sampled_nframes": 1,
+    }
+
+
+def navit_resize_video(
+    width: int,
+    height: int,
+    nframes: int,
+    avg_fps: float,
+    sample_fps: float,
+    patch_size: int,
+    merge_kernel_size: int,
+    in_patch_limit_each_frame: int,
+    patch_limit_on_one_side: int,
+    in_patch_limit_total: int | None,
+    max_num_frames_each_video: int | None,
+    fixed_output_tokens_each_frame: int | None,
+):
+    sample_fps = min(sample_fps, avg_fps)
+    # Calculate the number of frames to sample based on target FPS
+    sampled_nframes = max(round(nframes * sample_fps / avg_fps), 1)
+    if max_num_frames_each_video is not None:
+        sampled_nframes = min(sampled_nframes, max_num_frames_each_video)
+
+    if in_patch_limit_total is not None:
+        in_patch_limit_each_frame = min(
+            round(in_patch_limit_total / sampled_nframes),
+            in_patch_limit_each_frame)
+
+    ret = navit_resize_image(
+        width,
+        height,
+        patch_size,
+        merge_kernel_size,
+        in_patch_limit_each_frame,
+        patch_limit_on_one_side,
+        fixed_output_tokens_each_frame,
+    )
+    ret["sampled_nframes"] = sampled_nframes
+    return ret
+
+
+def real_sample_fps_and_max_num_frames(
+    type_name: Literal["video", "video_chunk"],
+    sample_fps: float,
+    max_num_frames_each_video: int | None,
+) -> tuple[int, int | None]:
+    if type_name == "video":
+        return sample_fps, max_num_frames_each_video
+    elif type_name == "video_chunk":
+        max_num_frames_each_video = None
+        sample_fps = math.inf
+        return sample_fps, max_num_frames_each_video
+    else:
+        return math.inf, None
+
+
+def _to_pil(data: str | bytes):
+    if isinstance(data, Image.Image):
+
+        return data.convert("RGB")
+    elif isinstance(data, str):
+        if data.startswith("data:"):
+            raw_base64 = data.split(",")[1]
+            return Image.open(io.BytesIO(
+                base64.b64decode(raw_base64))).convert("RGB")
+        else:
+            return Image.open(data).convert("RGB")
+    elif isinstance(data, bytes):
+        return Image.open(io.BytesIO(data)).convert("RGB")
+    else:
+        raise ValueError(f"Unsupported data type: {type(data)}")
+
+
+def ensure_media_type(media: MediaInput) -> MediaInput:
+    if media['type'] == 'image':
+        media['image'] = _to_pil(media['image'])
+        return media
+    elif media['type'] == 'video_chunk':
+        media['video_chunk'] = [
+            _to_pil(frame) for frame in media['video_chunk']
+        ]
+        return media
+    else:
+        raise ValueError(f"Unsupported media type: {media['type']}")
+
+
+def image_to_np(
+    image: Image.Image,
+    resize_to: tuple[int, int] | None = None,
+    mode: str = "resize",
+    raise_error_for_ill_resize: bool = True,
+) -> np.ndarray:
+    """Convert an image to a numpy array.
+
+    Args:
+        content: The image to convert.
+        resize_to: The size to resize the image to.
+        mode: The mode to resize the image to.
+        raise_error_for_ill_resize: Whether to raise an error for ill-sized resize.
+
+    Returns:
+        A numpy array.
+    """
+    assert isinstance(image, Image.Image), "image must be a PIL Image"
+    if resize_to is not None:
+        if mode == "resize":
+            image = image.resize(resize_to, resample=Image.Resampling.BICUBIC)
+
+        elif mode == "rescale_and_pad_to_center":
+            scale = min(resize_to[0] / image.width,
+                        resize_to[1] / image.height, 1.0)
+            new_width = round(image.width * scale)
+            new_height = round(image.height * scale)
+            if new_width == 0 or new_height == 0:
+                if raise_error_for_ill_resize:
+                    raise ValueError(
+                        f"Invalid resize to: {resize_to}, from image size: {image.size}"
+                    )
+                else:
+                    return np.zeros((resize_to[1], resize_to[0], 3),
+                                    dtype=np.uint8)
+
+            image = image.resize((new_width, new_height),
+                                 resample=Image.Resampling.BICUBIC)
+            padding_left = (resize_to[0] - new_width) // 2
+            padding_right = resize_to[0] - new_width - padding_left
+            padding_top = (resize_to[1] - new_height) // 2
+            padding_bottom = resize_to[1] - new_height - padding_top
+            image = np.asarray(image)
+            image = np.pad(
+                image,
+                ((padding_top, padding_bottom), (padding_left, padding_right),
+                 (0, 0)),
+                mode="constant",
+                constant_values=0,
+            )
+            assert image.shape == (resize_to[1], resize_to[0], 3)
+
+        elif mode == "rescale_and_pad_to_rightbottom":
+            scale = min(resize_to[0] / image.width,
+                        resize_to[1] / image.height, 1.0)
+            new_width = round(image.width * scale)
+            new_height = round(image.height * scale)
+            if new_width == 0 or new_height == 0:
+                if raise_error_for_ill_resize:
+                    raise ValueError(
+                        f"Invalid resize to: {resize_to}, from image size: {image.size}"
+                    )
+                else:
+                    return np.zeros((resize_to[1], resize_to[0], 3),
+                                    dtype=np.uint8)
+
+            image = image.resize((new_width, new_height),
+                                 resample=Image.Resampling.BICUBIC)
+            padding_right = resize_to[0] - new_width
+            padding_bottom = resize_to[1] - new_height
+            image = np.asarray(image)
+            image = np.pad(
+                image,
+                ((0, padding_bottom), (0, padding_right), (0, 0)),
+                mode="constant",
+                constant_values=0,
+            )
+            assert image.shape == (resize_to[1], resize_to[0], 3)
+
+        else:
+            raise ValueError(f"Invalid mode: {mode}")
+
+    if isinstance(image, Image.Image):
+        return np.asarray(image)
+    else:
+        return image
+
+
+def navit_patchify(pixel_values: np.ndarray,
+                   patch_size: int) -> dict[str, np.ndarray]:
+    """Reshape the pixel values to a navit shape.
+
+    Args:
+        pixel_values: np.ndarray, shape (t, h, w, c)
+        patch_size: int
+
+    Returns:
+        dict[str, np.ndarray]
+        - patches: np.ndarray, shape (t * h//patch_size * w//patch_size, c, patch_size, patch_size)
+        - grid_thw: np.ndarray, (t, h//patch_size, w//patch_size)
+    """
+    T, H, W, C = pixel_values.shape
+    assert C == 3, "pixel_values must have 3 channels"
+
+    patches = pixel_values.reshape(T, H // patch_size, patch_size,
+                                   W // patch_size, patch_size, C)
+    # (T, H//patch_size, W//patch_size, C, patch_size, patch_size)
+    patches = patches.transpose(0, 1, 3, 5, 2, 4)
+    patches = patches.reshape(-1, C, patch_size, patch_size)
+    grid_thw = np.array([T, H // patch_size, W // patch_size])
+    return {"pixel_values": patches, "grid_thw": grid_thw}
+
+
+def normalize(x: np.ndarray,
+              mean,
+              std_inv,
+              pixels_dtype: np.dtype = np.float32) -> np.ndarray:
+    """Normalize the image.
+
+    Args:
+        x: The image to normalize. The shape is (..., 3). The dtype is uint8. The range is [0, 255].
+        mean: The mean of the image.
+        std_inv: The inverse of the std of the image.
+        pixels_dtype: The dtype of the image.
+    Returns:
+        The normalized image. The shape is (..., 3). The dtype is determined by the pixels_dtype.
+    """
+    x = (x / 255.0).astype(pixels_dtype)
+    x -= mean
+    x *= std_inv
+    return x
+
+
+def _to_tensor(data, **kwargs):
+    import torch
+
+    if isinstance(data, np.ndarray):
+        return torch.from_numpy(data).to(**kwargs)
+    elif isinstance(data, torch.Tensor):
+        return data.to(**kwargs)
+    elif isinstance(data, list):
+        return [_to_tensor(item, **kwargs) for item in data]
+    elif isinstance(data, tuple):
+        return tuple(_to_tensor(item, **kwargs) for item in data)
+    elif isinstance(data, dict):
+        return {k: _to_tensor(v, **kwargs) for k, v in data.items()}
+    elif data is None:
+        return None
+    else:
+        raise ValueError(f"Unsupported data type: {type(data)}")

preprocessor_config.json

@@ -0,0 +1,30 @@
+{
+ "auto_map": {
+  "AutoProcessor": "kimi_k25_processor.KimiK25Processor",
+  "AutoImageProcessor": "kimi_k25_vision_processing.KimiK25VisionProcessor"
+ },
+ "media_proc_cfg": {
+  "in_patch_limit": 16384,
+  "patch_size": 14,
+  "image_mean": [
+   0.5,
+   0.5,
+   0.5
+  ],
+  "image_std": [
+   0.5,
+   0.5,
+   0.5
+  ],
+  "merge_kernel_size": 2,
+  "fixed_output_tokens": null,
+  "patch_limit_on_one_side": 512,
+  "in_patch_limit_each_frame": 16384,
+  "in_patch_limit_video": null,
+  "sample_fps": 2.0,
+  "max_num_frames_each_video": null,
+  "temporal_merge_kernel_size": 4,
+  "timestamp_mode": "hh:mm:ss.fff",
+  "config_type": "media_proc.processors.moonvit.MoonViTMediaProcessorConfig"
+ }
+}