Upload folder using huggingface_hub

README.md

@@ -0,0 +1,144 @@
+---
+pipeline_tag: image-text-to-text
+tags:
+- visual-document-understanding
+- visual-question-answering
+- indian-documents
+license: apache-2.0
+language:
+- en
+library_name: transformers
+base_model:
+- bharatgenai/patram-7b-instruct
+---
+
+# Patram-7B-Instruct
+
+Patram-7B-Instruct by BharatGen is a 7B parameter vision-language model trained from scratch for visual document understanding. As India’s first document foundation model, it is built to tackle complex document analysis.
+The model was trained on a carefully curated instruction-tuned dataset, combining diverse public and custom synthetic data designed to support a broad spectrum of document understanding tasks.
+
+## Model Overview
+
+* **Architecture:** Vision Transformer (ViT) + MLP projector + OLMo-7B LLM
+* **Training Data:** BharatDocs-v1, a dataset of diverse Indian documents + Other Open Source Document Datasets
+* **Supported I/O Formats:** The model currently accepts English-language instructions and image files (e.g., PNG, JPEG) as input. The output is provided in text format.
+* **Language:** English (Indian language support upcoming)
+* **License:** [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0)
+
+## Usage Examples
+
+Use the `transformers` library.
+
+```python
+import torch
+from transformers import AutoProcessor, AutoModelForCausalLM, GenerationConfig
+from PIL import Image
+import requests
+
+# Model ID and device setup
+model_id = "bharatgenai/patram-7b-instruct"
+device = "cuda" if torch.cuda.is_available() else "cpu"
+
+# Load processor and model
+processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
+model = AutoModelForCausalLM.from_pretrained(
+    model_id,
+    trust_remote_code=True
+).to(device)
+
+def get_patram_response(image_path_or_url, question):
+    try:
+        # Load image
+        if image_path_or_url.startswith("http"):
+            image = Image.open(requests.get(image_path_or_url, stream=True).raw).convert("RGB")
+        else:
+            image = Image.open(image_path_or_url).convert("RGB")
+    except Exception as e:
+        print(f"Error loading image: {e}")
+        return None
+
+    # Format the prompt as expected
+    prompt = f"Question: {question} Answer based on the image."
+
+    try:
+        # Preprocess image and text using the processor
+        inputs = processor.process(images=[image], text=prompt)
+        inputs = {k: v.to(device).unsqueeze(0) for k, v in inputs.items()}
+
+        # Generate output using model's generate_from_batch method (Patram-specific)
+        output = model.generate_from_batch(
+            inputs,
+            GenerationConfig(max_new_tokens=200, stop_strings="<|endoftext|>"),
+            tokenizer=processor.tokenizer
+        )
+
+        # Extract generated tokens (excluding input tokens) and decode
+        generated_tokens = output[0, inputs['input_ids'].size(1):]
+        response = processor.tokenizer.decode(generated_tokens, skip_special_tokens=True).strip()
+        return response
+    except Exception as e:
+        print(f"Error during inference: {e}")
+        return None
+
+# Example usage:
+# image_input = "https://knowscope.in/wp-content/uploads/2025/05/cghd-nag.png"
+# question = "Who issued this notice?"
+# answer = get_patram_response(image_input, question)
+# if answer:
+#     print("Answer:", answer)
+```
+
+**Note**: If you're trying this on an Apple Silicon (M1/M2/M3/M4/...) chip, please follow the official documentation by PyTorch and Hugging Face for installing dependencies:
+
+- [PyTorch's official guide on installation (macOS)](https://pytorch.org/get-started/locally/#:~:text=torch%20torchvision%20torchaudio-,Installing%20on%20macOS,-PyTorch%20can%20be)
+- [Hugging Face Transformers performance tips](https://huggingface.co/docs/transformers/main/en/perf_train_special)
+     
+  
+## Evaluations
+
+We evaluated Patram-7B-Instruct alongside other vision-language models (VLMs) in the 7B–9B parameter range across multiple public document benchmarks.
+
+**Benchmarks**: DocVQA, VisualMRC, Patram-Bench
+
+Patram-Bench is an in-house benchmark designed for Indic Document VQA.
+
+**Metric**: G-Eval (LLM-as-a-judge)
+
+| Model                  | Overall | DocVQA | Patram-Bench | VisualMRC |
+| ---------------------- | ------- | ------ | ------------ | --------- |
+| claude-3.7-sonnet      | 0.8830  | 0.8480 | 0.8857       | 0.8830    |
+| Qwen2.5-VL-7B-Instruct | 0.8759  | 0.8722 | 0.6816       | 0.9169    |
+| gemma-3-12b-it         | 0.8556  | 0.8451 | 0.6349       | 0.9069    |
+| **patram-7b-instruct** | 0.8331  | 0.8550 | 0.6515       | 0.8510    |
+| InternVL3-9B           | 0.7865  | 0.8681 | 0.6888       | 0.7405    |
+| deepseek-vl2           | 0.7581  | 0.8739 | 0.5089       | 0.7144    |
+
+*Note: The benchmarked results reflect the API variant.
+
+## Citation
+
+```bibtex
+@online{BharatGenPatramLaunch2025,
+  author    = {{BharatGen Team}},
+  title     = {BharatGen Unveils Patram: India's Pioneering Vision-Language Foundation Model for Document Intelligence},
+  year      = {2025},
+  url       = {https://bharatgen.com/blog/patram-launch},
+  urldate   = {2025-06-02}
+}
+```
+
+## Resources
+
+* **Model**: [huggingface.co/bharatgenai/patram-7b-instruct](https://huggingface.co/bharatgenai/patram-7b-instruct)
+* **Project Page**: [bharatgen.com/patram](https://bharatgen.com/patram)
+* **Blog**: [bharatgen.com/blog/patram-launch](https://bharatgen.com/blog/patram-launch)
+
+## Authors
+
+* **Principal Investigators**: Prof. Ravi Kiran Sarvadevabhatla, Prof. Ganesh Ramakrishnan
+* **Contributors**: BharatGen Team
+
+## Contact
+
+* [Contact Form](https://bharatgen.com/contact)
+* Hugging Face Community Tab

added_tokens.json

@@ -0,0 +1,7 @@
+{
+  "<im_col>": 100281,
+  "<im_end>": 100279,
+  "<im_patch>": 100280,
+  "<im_start>": 100278,
+  "<|image|>": 100282
+}

chat_template.jinja

@@ -0,0 +1,13 @@
+{% for message in messages -%}
+        {%- if (loop.index % 2 == 1 and message['role'] != 'user') or 
+          (loop.index % 2 == 0 and message['role'].lower() != 'assistant') -%}
+        {{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}
+        {%- endif -%}
+        {{ message['role'].capitalize() + ': ' + message['content'] }}
+        {%- if not loop.last -%}
+        {{ ' ' }}
+        {%- endif %}
+        {%- endfor -%}
+        {%- if add_generation_prompt -%}
+        {{ ' Assistant:' }}
+        {%- endif %}

config.json

@@ -0,0 +1,33 @@
+{
+  "architectures": [
+    "PatramForCausalLM"
+  ],
+  "attention_layer_norm": true,
+  "auto_map": {
+    "AutoConfig": "config_patram.PatramConfig",
+    "AutoModelForCausalLM": "modeling_patram.PatramForCausalLM"
+  },
+  "clip_qkv": null,
+  "embedding_size": 100352,
+  "hidden_size": 4096,
+  "initializer_range": 0.02,
+  "intermediate_size": 22016,
+  "layer_norm_eps": 1e-06,
+  "layer_norm_type": "rms",
+  "max_position_embeddings": 4096,
+  "model_type": "patram",
+  "norm_after": true,
+  "num_attention_heads": 32,
+  "num_hidden_layers": 32,
+  "num_key_value_heads": null,
+  "pad_token_id": 100277,
+  "qkv_bias": false,
+  "rope_theta": 500000.0,
+  "tie_word_embeddings": false,
+  "torch_dtype": "float32",
+  "transformers_version": "4.52.3",
+  "use_cache": false,
+  "use_position_ids": true,
+  "vocab_size": 100278,
+  "weight_tying": false
+}

config_patram.py

@@ -0,0 +1,60 @@
+from typing import List
+
+from transformers import PretrainedConfig, AutoTokenizer
+
+
+class PatramConfig(PretrainedConfig):
+    model_type = "patram"
+    keys_to_ignore_at_inference = ["past_key_values"]
+
+    def __init__(
+        self,
+        vocab_size=50304,
+        embedding_size=50304,
+        hidden_size=4096,
+        intermediate_size=11008,
+        num_hidden_layers=32,
+        num_attention_heads=32,
+        num_key_value_heads=None,
+        max_position_embeddings=2048,
+        initializer_range=0.02,
+        use_cache=True,
+        layer_norm_eps: float = 1e-5,
+        rope_theta=10000.0,
+        clip_qkv=None,
+        qkv_bias: bool = False,
+        weight_tying: bool = False,
+        use_position_ids: bool=True,
+        tie_word_embeddings: bool=True,
+        attention_layer_norm: bool=False,
+        norm_after: bool = False,
+        layer_norm_type: str="rms",
+        **kwargs,
+    ):
+        self.vocab_size = vocab_size
+        self.embedding_size = embedding_size
+        self.max_position_embeddings = max_position_embeddings
+        self.hidden_size = hidden_size
+        self.intermediate_size = intermediate_size
+        self.num_hidden_layers = num_hidden_layers
+        self.num_attention_heads = num_attention_heads
+        self.layer_norm_eps = layer_norm_eps
+        self.weight_tying = weight_tying
+        self.use_position_ids = use_position_ids
+        self.attention_layer_norm = attention_layer_norm
+        self.num_key_value_heads = num_key_value_heads
+        self.initializer_range = initializer_range
+        self.use_cache = use_cache
+        self.rope_theta = rope_theta
+        self.clip_qkv = clip_qkv
+        self.qkv_bias = qkv_bias
+        self.norm_after = norm_after
+        self.tie_word_embeddings = tie_word_embeddings
+        self.layer_norm_type = layer_norm_type
+
+        super().__init__(
+            tie_word_embeddings=tie_word_embeddings,
+            **kwargs,
+        )
+
+PatramConfig.register_for_auto_class()

generation_config.json

@@ -0,0 +1,4 @@
+{
+  "_from_model_config": true,
+  "transformers_version": "4.52.3"
+}

image_preprocessing_patram.py

@@ -0,0 +1,546 @@
+"""Image processor class for Patram"""
+from typing import List, Optional, Union, Mapping
+
+import numpy as np
+import einops
+import torch
+import torchvision.transforms
+from torchvision.transforms import InterpolationMode
+from torchvision.transforms.functional import convert_image_dtype
+
+from transformers.image_utils import (
+    OPENAI_CLIP_MEAN,
+    OPENAI_CLIP_STD,
+    ImageInput,
+    is_valid_image,
+)
+from transformers.processing_utils import ImagesKwargs
+from transformers.image_processing_utils import BaseImageProcessor
+from transformers.utils import logging
+
+
+logger = logging.get_logger(__name__)
+
+
+def pad_to_bounding_box(
+    image, offset_height, offset_width, target_height,
+    target_width, value=0
+):
+    height, width = image.shape[:2]
+    after_padding_width = target_width - offset_width - width
+    after_padding_height = target_height - offset_height - height
+    return np.pad(image, [
+        [offset_height, after_padding_height],
+        [offset_width, after_padding_width],
+        [0, 0]
+    ], constant_values=value)
+
+
+def normalize_image(image, offset, scale):
+    image -= np.array(offset, dtype=np.float32)[None, None, :]
+    image /= np.array(scale, dtype=np.float32)[None, None, :]
+    return image
+
+
+def resize_and_pad(
+    image,
+    desired_output_size,
+    resize_method="torch-bilinear",
+    pad_value=0,
+    normalize=True,
+    image_mean=OPENAI_CLIP_MEAN,
+    image_std=OPENAI_CLIP_STD,
+):
+    desired_height, desired_width = desired_output_size
+    height, width = image.shape[:2]
+
+    # Cast into float32 since the training code did this in float32 and it (very rarely) effects
+    # the results after rounding.
+    image_scale_y = np.array(desired_height, np.float32) / np.array(height, np.float32)
+    image_scale_x = np.array(desired_width, np.float32) / np.array(width, np.float32)
+    image_scale = min(image_scale_x, image_scale_y)
+    scaled_height = int(np.array(height, np.float32) * image_scale)
+    scaled_width = int(np.array(width, np.float32) * image_scale)
+
+    if resize_method == "tensorflow":
+        # This how the original training code did resizing, it can produce slightly different
+        # results then using torch resize so we keep it just in case
+        import tensorflow as tf
+        image = tf.image.convert_image_dtype(tf.constant(image), dtype=tf.float32)
+        image = tf.image.resize(
+            image,
+            [scaled_height, scaled_width],
+            method=tf.image.ResizeMethod.BILINEAR,
+            antialias=True,
+        )
+        image = tf.clip_by_value(image, 0.0, 1.0)
+        image = image.numpy()
+    elif resize_method == "torch-bilinear":
+        image = torch.permute(torch.from_numpy(image), [2, 0, 1])
+        image = convert_image_dtype(image)  # resize in float32 to match the training code
+        image = torchvision.transforms.Resize(
+            [scaled_height, scaled_width], InterpolationMode.BILINEAR, antialias=True
+        )(image)
+        image = torch.clip(image, 0.0, 1.0)
+        image = torch.permute(image, [1, 2, 0]).numpy()
+    else:
+        raise NotImplementedError(resize_method)
+
+    top_pad = (desired_height - scaled_height) // 2
+    left_pad = (desired_width - scaled_width) // 2
+    padding = [
+        [top_pad, desired_height - scaled_height - top_pad],
+        [left_pad, desired_width - scaled_width - left_pad],
+        [0, 0]
+    ]
+    image_mask = np.pad(np.ones_like(image[:, :, 0], dtype=bool), padding[:2])
+    image = np.pad(image, padding, constant_values=pad_value)
+    if normalize:
+        image = normalize_image(image, offset=image_mean, scale=image_std)
+    return image, image_mask
+
+
+def select_tiling(h, w, patch_size, max_num_patches):
+    """Decide how best to divide in image of size [w, h] in up to max_num_patches of size patch_size"""
+    original_size = np.stack([h, w])  # [1, 2]
+    original_res = h * w
+    tilings = []
+    for i in range(1, max_num_patches+1):
+        for j in range(1, max_num_patches+1):
+            if i*j <= max_num_patches:
+                tilings.append((i, j))
+    # sort so argmin and argmax favour smaller tilings in the event of a tie
+    tilings.sort(key=lambda x: (x[0]*x[1], x[0]))
+    candidate_tilings = np.array(tilings, dtype=np.int32)  # [n_resolutions, 2]
+    candidate_resolutions = candidate_tilings * patch_size  # [n_resolutions, 2]
+
+    # How much we would need to scale the image to fit exactly in each tiling
+    original_size = np.stack([h, w], dtype=np.float32)  # [1, 2]
+    required_scale_d = candidate_resolutions.astype(np.float32) / original_size
+    required_scale = np.min(required_scale_d, axis=-1, keepdims=True)  # [n_resolutions, 1]
+    if np.all(required_scale < 1):
+        # We are forced to downscale, so try to minimize the amount of downscaling
+        ix = np.argmax(required_scale)
+    else:
+        # Pick the resolution that required the least upscaling so that it most closely fits the image
+        required_scale = np.where(required_scale < 1.0, 10e9, required_scale)
+        ix = np.argmin(required_scale)
+    return candidate_tilings[ix]
+
+
+class PatramImagesKwargs(ImagesKwargs, total=False):
+    max_crops: Optional[int]
+    overlap_margins: Optional[List[int]]
+    base_image_input_size: Optional[List[int]]
+    image_token_length_w: Optional[int]
+    image_token_length_h: Optional[int]
+    image_patch_size: Optional[int]
+    image_padding_mask: Optional[bool]
+
+
+class PatramImageProcessor(BaseImageProcessor):
+    """Preprocess images and multi-model inputs"""
+
+    def __init__(
+        self,
+        max_crops: int = 12,
+        overlap_margins: List[int] = (4, 4),
+        base_image_input_size: List[int] = (336, 336),
+        image_token_length_w: int = 12,
+        image_token_length_h: int = 12,
+        image_patch_size: int = 14,
+        image_padding_mask: bool = True,
+        do_normalize: bool = True,
+        image_mean: Optional[Union[float, List[float]]] = None,
+        image_std: Optional[Union[float, List[float]]] = None,
+        **kwargs,
+    ):
+        super().__init__(**kwargs)
+        self.max_crops = max_crops
+        self.overlap_margins = overlap_margins
+        self.base_image_input_size = base_image_input_size
+        self.image_token_length_w = image_token_length_w
+        self.image_token_length_h = image_token_length_h
+        self.image_patch_size = image_patch_size
+        self.image_padding_mask = image_padding_mask
+        self.do_normalize = do_normalize
+        self.image_mean = image_mean if image_mean is not None else OPENAI_CLIP_MEAN
+        self.image_std = image_std if image_std is not None else OPENAI_CLIP_STD
+
+    def image_to_patches_and_tokens(
+        self,
+        image: ImageInput,
+        image_patch_token_id: int,
+        image_col_token_id: int,
+        image_start_token_id: int,
+        image_end_token_id: int,
+        max_crops: Optional[int] = None,
+        overlap_margins: Optional[List[int]] = None,
+        base_image_input_size: Optional[Union[int, List[int]]] = None,
+        image_token_length_w: Optional[int] = None,
+        image_token_length_h: Optional[int] = None,
+        image_patch_size: Optional[int] = None,
+    ):
+        if isinstance(base_image_input_size, int):
+            base_image_input_size = (base_image_input_size, base_image_input_size)
+
+        base_image_input_d = image_patch_size
+        tokens_per_image = image_token_length_w * image_token_length_h
+        image_base_patch_w = base_image_input_size[1] // base_image_input_d
+        image_base_patch_h = base_image_input_size[0] // base_image_input_d
+
+        original_image_h, original_image_w = image.shape[:2]
+        crop_size = base_image_input_size[0]
+
+        # Discard this many patches from the (left/top, right/bottom) of crops
+        left_margin, right_margin = overlap_margins
+        # left_margin, right_margin = 2, 2
+        assert left_margin % 2 == 0  # Required for compatibility with 2x2 pooling
+        total_margin_pixels = base_image_input_d*(right_margin + left_margin)  # pixels removed per dim
+        crop_patches = base_image_input_size[0] // base_image_input_d  # patches per crop dim
+        crop_window_patches = crop_patches - (right_margin + left_margin)  # usable patches
+        crop_window_size = crop_window_patches * base_image_input_d
+        tiling = select_tiling(
+            original_image_h - total_margin_pixels,
+            original_image_w - total_margin_pixels,
+            crop_window_size,
+            max_crops
+        )
+        src, img_mask = resize_and_pad(
+            image,
+            [tiling[0]*crop_window_size+total_margin_pixels, tiling[1]*crop_window_size+total_margin_pixels]
+        )
+
+        # Now we have to split the image into crops, while keeping track of how each patch in the
+        # each crop should be ordered in the global image, this require a lot of tricky booking
+        n_crops = tiling[0] * tiling[1]
+        patches_arr = []
+        mask_arr = []
+        patch_ordering_arr = []
+
+        # We assume 2x2 pooling, but can allow padding the right/bottom with extra
+        # patches if the number of patches per side is not even
+        assert (crop_patches+1)//2 == image_token_length_h
+        assert (crop_patches+1)//2 == image_token_length_w
+        on = 0
+        on_patch = 0
+        for i in range(tiling[0]):
+            y0 = i*crop_window_size
+            if i == 0:
+                crop_y0 = 0
+            else:
+                crop_y0 = left_margin // 2
+
+            crop_h = image_base_patch_h - (right_margin + left_margin)
+            if i == 0:
+                crop_h += left_margin
+            if i == (tiling[0]-1):
+                crop_h += right_margin
+            for j in range(tiling[1]):
+                x0 = j*crop_window_size
+                if j == 0:
+                    crop_x0 = 0
+                else:
+                    crop_x0 = left_margin // 2
+
+                crop_w = image_base_patch_w - (right_margin + left_margin)
+                if j == 0:
+                    crop_w += left_margin
+                if j == (tiling[1]-1):
+                    crop_w += right_margin
+
+                pooled_w = (crop_w + 1) // 2
+                pooled_h = (crop_h + 1) // 2
+                patch_ordering_arr.append(
+                    pad_to_bounding_box(
+                        np.reshape(np.arange(on, on+pooled_h*pooled_w, dtype=np.int32), (pooled_h, pooled_w, 1)),
+                        crop_y0, crop_x0, image_token_length_h, image_token_length_w, value=-1
+                    )[:, :, 0]
+                )
+                patches_arr.append(src[y0:y0+crop_size, x0:x0+crop_size])
+                mask_arr.append(img_mask[y0:y0+crop_size, x0:x0+crop_size])
+
+                on += pooled_h*pooled_w
+                on_patch += 1
+        patches = np.stack(patches_arr)
+        patch_ordering = np.stack(patch_ordering_arr)
+        img_mask = np.stack(mask_arr)
+
+        # Switch to [n_crops, n_patches, pixels_per_patch] format
+        image_layout_impatch_w, image_layout_impatch_h = tiling[0], tiling[1]
+        patches = einops.rearrange(
+            patches, 'p (h dh) (w dw) c -> p (h w) (dh dw c)',
+            dh=base_image_input_d,
+            dw=base_image_input_d,
+            h=image_base_patch_h,
+            w=image_base_patch_w
+        )
+        img_mask = einops.rearrange(
+            img_mask, 'p (h dh) (w dw) -> p (h w) (dh dw)',
+            dh=base_image_input_d,
+            dw=base_image_input_d,
+            h=image_base_patch_h,
+            w=image_base_patch_w
+        )
+
+        img_mask = img_mask.astype(np.float32).mean(axis=-1)
+        patch_ordering = np.reshape(patch_ordering, [-1])
+        valid = patch_ordering >= 0
+
+        # Transpose order, to get left-to-right order instead of crop-by-crop order
+        patch_ordering_rh = np.reshape(
+            patch_ordering,
+            [tiling[0], tiling[1], image_token_length_h, image_token_length_w]
+        )
+        patch_ordering_rh = np.transpose(patch_ordering_rh, [0, 2, 1, 3])
+        patch_ordering_rh = np.reshape(patch_ordering_rh, [-1])
+
+        # The transpose will screw up which patches are masked, project the
+        # new order into sparse structure of `patch_ordering` to fix this
+        patch_ordering[valid] = patch_ordering_rh[patch_ordering_rh >= 0]
+
+        # Now build the output tokens
+        h = tiling[0] * crop_window_patches + (right_margin+left_margin)
+        w = tiling[1] * crop_window_patches + (right_margin+left_margin)
+        per_row = np.full(
+            ((w+1)//2,),
+            image_patch_token_id,
+        )
+        per_row = np.concatenate([per_row, [image_col_token_id]], 0)
+
+        joint = np.tile(per_row, [(h+1)//2])
+        joint = [
+            [image_start_token_id],
+            joint,
+            [image_end_token_id]
+        ]
+
+        # Finally do the same for the global image
+        resized, _ = resize_and_pad(image, base_image_input_size)
+        resized = einops.rearrange(
+            resized, '(h dh) (w dw) c -> (h w) (dh dw c)',
+            dh=base_image_input_d,
+            dw=base_image_input_d,
+            h=image_base_patch_h,
+            w=image_base_patch_w
+        )
+        patches = np.concatenate([np.expand_dims(resized, 0), patches], 0)
+
+        # Global image goes first, so the order of patches in previous crops gets increased
+        patch_ordering = np.where(
+            patch_ordering >= 0,
+            patch_ordering + tokens_per_image,
+            -1
+        )
+        patch_ordering = np.concatenate([np.arange(0, tokens_per_image), patch_ordering], 0)
+        per_row = np.full(
+            (image_token_length_w,),
+            image_patch_token_id,
+        )
+        per_row = np.concatenate([per_row, [image_col_token_id]], 0)
+        extra_tokens = np.tile(per_row, [image_token_length_h])
+        joint = [
+                    [image_start_token_id],
+                    extra_tokens,
+                    [image_end_token_id],
+                ] + joint
+
+        joint = np.concatenate(joint, 0)
+        img_mask = np.pad(img_mask, [[0, 1], [0, 0]], constant_values=-1)
+        return patches, joint, patch_ordering, img_mask
+
+    def build_image_input_idx(
+        self,
+        image_tokens: np.ndarray,
+        patch_order: np.ndarray,
+        image_patch_token_id: int,
+        no_image: Optional[bool] = None,
+        image_token_length_w: Optional[int] = None,
+        image_token_length_h: Optional[int] = None,
+    ):
+        """Converts `patch_order` into a mapping of token_id -> patch_id"""
+
+        tokens_per_image = image_token_length_w * image_token_length_h
+        if no_image is not None and no_image:
+            return np.zeros((0, tokens_per_image), np.int32)
+
+        # Indices to insert the patches
+        image_input_idx = image_tokens == image_patch_token_id
+        image_input_idx = np.nonzero(image_input_idx)[0].astype(np.int32)
+
+        if patch_order is not None:
+            n_tokens = image_input_idx.shape[0]
+            patch_order = np.reshape(patch_order, [-1])
+            n_patches = patch_order.shape[0]
+
+            valid = patch_order >= 0
+            n_valid_patches = valid.sum()
+            assert len(image_input_idx) == n_valid_patches
+
+            sorted_patch_ixs = np.zeros([n_tokens], np.int32)
+            sorted_patch_ixs[patch_order[valid]] = np.arange(n_valid_patches, dtype=np.int32)
+
+            # Project the inverted mapping into same sparse structure
+            sorted_patch_ixs_ex = np.full(np.shape(patch_order), -1)
+            sorted_patch_ixs_ex[valid] = sorted_patch_ixs
+
+            # Do the gather and then re-masked outputs that were masked in `sorted_patch_ixs`
+            valid = (sorted_patch_ixs_ex >= 0).astype(np.int32)
+            image_input_idx = image_input_idx[sorted_patch_ixs_ex*valid]
+            image_input_idx = image_input_idx*valid - 100*(1 - valid)
+            image_input_idx = np.reshape(image_input_idx, [-1, tokens_per_image])
+        return image_input_idx
+
+    def preprocess(
+        self,
+        image: np.ndarray,
+        image_patch_token_id: int,
+        image_col_token_id: int,
+        image_start_token_id: int,
+        image_end_token_id: int,
+        max_crops: Optional[int] = None,
+        overlap_margins: Optional[List[int]] = None,
+        base_image_input_size: Optional[Union[int, List[int]]] = None,
+        image_token_length_w: Optional[int] = None,
+        image_token_length_h: Optional[int] = None,
+        image_patch_size: Optional[int] = None,
+        **kwargs,
+    ):
+        """Preprocesses an image
+
+        Returns:
+            crops: (n_crops, n_patches, patch_dim) individual crops, `n_crops` might
+                   change between images but the other dimension are fixed
+            tokens: (n_tokens,) int32 tokens, pad tokens indicate where to insert the
+                                patch features, might include other special tokens as well
+            image_idx: (n_crops, n_patches) index in `tokens` to put the patch features from the
+                       crops after pooling, negative values indicates patches features to exclude
+            padding_mask: (n_crops, n_patches) what percent of each crop is padding, can be None
+                          if the image mask is not being used.
+        """
+
+        max_crops = max_crops or self.max_crops
+        overlap_margins = overlap_margins or self.overlap_margins
+        base_image_input_size = base_image_input_size or self.base_image_input_size
+        image_token_length_w = image_token_length_w or self.image_token_length_w
+        image_token_length_h = image_token_length_h or self.image_token_length_h
+        image_patch_size = image_patch_size or self.image_patch_size
+
+        crops, image_tokens, patch_ordering, img_mask = self.image_to_patches_and_tokens(
+            image,
+            image_patch_token_id,
+            image_col_token_id,
+            image_start_token_id,
+            image_end_token_id,
+            max_crops,
+            overlap_margins,
+            base_image_input_size,
+            image_token_length_w,
+            image_token_length_h,
+            image_patch_size,
+        )
+        patch_idx = self.build_image_input_idx(
+            image_tokens,
+            patch_ordering,
+            image_patch_token_id,
+            image_token_length_w=image_token_length_w,
+            image_token_length_h=image_token_length_h,
+        )
+        return crops, image_tokens, patch_idx, img_mask
+
+    def multimodal_preprocess(
+        self,
+        images: np.ndarray,
+        tokens: List[int],
+        image_idx: np.ndarray,
+        sequence_length: int,
+        image_patch_token_id: int,
+        image_col_token_id: int,
+        image_start_token_id: int,
+        image_end_token_id: int,
+        **kwargs,
+    ):
+        """Merge images and text tokens into multi-modal features for the model
+
+        :param images: images to use as input
+        :param tokens: input text tokens
+        :param image_idx: where to insert the images into `tokens`
+        :params image_patch_token_id: id to use of tokens that will contain image features
+        :params image_col_token_id: token id for image column special tokens
+        :params image_start_token_id: token id for image start special tokens
+        :params image_end_token_id: token id for image end special tokens
+        :params kwargs: override preprocessor default args
+        """
+        max_total_crops = kwargs.get("max_crops") or self.max_crops
+        image_token_length_w = kwargs.get("image_token_length_w") or self.image_token_length_w
+        image_token_length_h = kwargs.get("image_token_length_h") or self.image_token_length_h
+        image_patch_size = kwargs.get("image_patch_size") or self.image_patch_size
+        base_image_input_size = kwargs.get("base_image_input_size") or self.base_image_input_size
+        image_num_patch = (
+            base_image_input_size[0] // image_patch_size,
+            base_image_input_size[1] // image_patch_size,
+        )
+        image_padding_mask = kwargs.get("image_padding_mask") or self.image_padding_mask
+
+        tokens_per_image = image_token_length_w * image_token_length_h
+        n_pixels = image_patch_size * image_patch_size * 3
+        n_patches = image_num_patch[0] * image_num_patch[1]
+
+        if images is None:
+            return {
+                "input_ids": tokens,
+            }
+        else:
+            n = len(images)
+            all_crops = []
+            all_image_idx = []
+            out_tokens = []
+            all_crop_masks = []
+
+            for ix in range(n):
+                token_ix = image_idx[ix]
+                crops, image_tokens, patch_idx, img_mask = self.preprocess(
+                    images[ix],
+                    image_patch_token_id,
+                    image_col_token_id,
+                    image_start_token_id,
+                    image_end_token_id,
+                    **kwargs,
+                )
+
+                if token_ix == -1:  # -1 is an image inserted at the very start
+                    start = 0
+                    token_ix = 0
+                    end = 0
+                else:
+                    start = 0 if ix == 0 else image_idx[ix-1] + 1
+                    end = token_ix + 1
+
+                all_image_idx.append(patch_idx + token_ix)
+                all_crops.append(crops)
+                out_tokens.append(tokens[start:token_ix])
+                out_tokens.append(image_tokens)
+                if ix == (n - 1):
+                    out_tokens.append(tokens[end:])
+                if image_padding_mask:
+                    all_crop_masks.append(img_mask)
+
+            input_ids = np.concatenate(out_tokens, 0)
+            images = np.concatenate(all_crops, 0)
+            image_input_idx = np.concatenate(all_image_idx, 0)
+            if image_padding_mask:
+                image_masks = np.concatenate(all_crop_masks, 0)
+            else:
+                image_masks = None
+
+        out = {
+            "input_ids": input_ids,
+            "images": images,
+            "image_input_idx": image_input_idx
+        }
+        if image_masks is not None:
+            out["image_masks"] = image_masks
+        return out
+
+
+PatramImageProcessor.register_for_auto_class()

model-00001-of-00007.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:361943ee8b0098606e0b3b3b6ff8237be8eb315bd12ac8e9ae5b3e58cb6f2b8a
+size 4951691600

model-00002-of-00007.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7904c5b975043a90af0162611653d672e27fc772014baef526f29aff9f2b84d1
+size 4857402880

model-00003-of-00007.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ad29ec49f9f2e12ef1f6a8e4fa6a6386645f5bfb7c20ded25b3ed988b9f8761b
+size 4857402936

model-00004-of-00007.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8e4deb1217465b785c1b936fccba8752830f481950b630ca70d8bd4a53072ea7
+size 4857402936

model-00005-of-00007.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3761ceab9e51fe4f625750506ce77e5d49b1fcfafd04391ec565de61660e385a
+size 4857402936

model-00006-of-00007.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c683d7344a2189a367eba30a4fcb47562ded123c98f90e91eb9caedd04c66cb2
+size 4990466360

model-00007-of-00007.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9a17e00d0bfb17cb8febf51d3239f351cbd9ea88e5d19894f6cfc2e3ded8eb9d
+size 1341218552

model.safetensors.index.json

@@ -0,0 +1,571 @@
+{
+  "metadata": {
+    "total_size": 30712913920
+  },
+  "weight_map": {
+    "model.transformer.blocks.0.att_proj.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.0.attn_norm.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.0.attn_out.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.0.ff_norm.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.0.ff_out.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.0.ff_proj.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.0.k_norm.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.0.q_norm.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.1.att_proj.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.1.attn_norm.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.1.attn_out.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.1.ff_norm.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.1.ff_out.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.1.ff_proj.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.1.k_norm.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.1.q_norm.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.10.att_proj.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.10.attn_norm.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.10.attn_out.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.10.ff_norm.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.10.ff_out.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.10.ff_proj.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.10.k_norm.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.10.q_norm.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.11.att_proj.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.11.attn_norm.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.11.attn_out.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.11.ff_norm.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.11.ff_out.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.11.ff_proj.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.11.k_norm.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.11.q_norm.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.12.att_proj.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.12.attn_norm.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.12.attn_out.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.12.ff_norm.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.12.ff_out.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.12.ff_proj.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.12.k_norm.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.12.q_norm.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.13.att_proj.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.13.attn_norm.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.13.attn_out.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.13.ff_norm.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.13.ff_out.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.13.ff_proj.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.13.k_norm.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.13.q_norm.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.14.att_proj.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.14.attn_norm.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.14.attn_out.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.14.ff_norm.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.14.ff_out.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.14.ff_proj.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.14.k_norm.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.14.q_norm.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.15.att_proj.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.15.attn_norm.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.15.attn_out.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.15.ff_norm.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.15.ff_out.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.15.ff_proj.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.15.k_norm.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.15.q_norm.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.16.att_proj.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.16.attn_norm.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.16.attn_out.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.16.ff_norm.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.16.ff_out.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.16.ff_proj.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.16.k_norm.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.16.q_norm.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.17.att_proj.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.17.attn_norm.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.17.attn_out.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.17.ff_norm.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.17.ff_out.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.17.ff_proj.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.17.k_norm.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.17.q_norm.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.18.att_proj.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.18.attn_norm.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.18.attn_out.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.18.ff_norm.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.18.ff_out.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.18.ff_proj.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.18.k_norm.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.18.q_norm.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.19.att_proj.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.19.attn_norm.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.19.attn_out.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.19.ff_norm.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.19.ff_out.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.19.ff_proj.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.19.k_norm.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.19.q_norm.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.2.att_proj.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.2.attn_norm.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.2.attn_out.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.2.ff_norm.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.2.ff_out.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.2.ff_proj.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.2.k_norm.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.2.q_norm.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.20.att_proj.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.20.attn_norm.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.20.attn_out.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.20.ff_norm.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.20.ff_out.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.20.ff_proj.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.20.k_norm.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.20.q_norm.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.21.att_proj.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.21.attn_norm.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.21.attn_out.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.21.ff_norm.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.21.ff_out.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.21.ff_proj.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.21.k_norm.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.21.q_norm.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.22.att_proj.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.22.attn_norm.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.22.attn_out.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.22.ff_norm.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.22.ff_out.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.22.ff_proj.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.22.k_norm.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.22.q_norm.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.23.att_proj.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.23.attn_norm.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.23.attn_out.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.23.ff_norm.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.23.ff_out.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.23.ff_proj.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.23.k_norm.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.23.q_norm.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.24.att_proj.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.24.attn_norm.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.24.attn_out.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.24.ff_norm.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.24.ff_out.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.24.ff_proj.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.24.k_norm.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.24.q_norm.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.25.att_proj.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.25.attn_norm.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.25.attn_out.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.25.ff_norm.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.25.ff_out.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.25.ff_proj.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.25.k_norm.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.25.q_norm.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.26.att_proj.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.26.attn_norm.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.26.attn_out.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.26.ff_norm.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.26.ff_out.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.26.ff_proj.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.26.k_norm.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.26.q_norm.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.27.att_proj.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.27.attn_norm.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.27.attn_out.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.27.ff_norm.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.27.ff_out.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.27.ff_proj.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.27.k_norm.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.27.q_norm.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.28.att_proj.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.28.attn_norm.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.28.attn_out.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.28.ff_norm.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.28.ff_out.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.28.ff_proj.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.28.k_norm.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.28.q_norm.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.29.att_proj.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.29.attn_norm.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.29.attn_out.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.29.ff_norm.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.29.ff_out.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.29.ff_proj.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.29.k_norm.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.29.q_norm.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.3.att_proj.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.3.attn_norm.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.3.attn_out.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.3.ff_norm.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.3.ff_out.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.3.ff_proj.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.3.k_norm.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.3.q_norm.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.30.att_proj.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.30.attn_norm.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.30.attn_out.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.30.ff_norm.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.30.ff_out.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.30.ff_proj.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.30.k_norm.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.30.q_norm.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.31.att_proj.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.31.attn_norm.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.31.attn_out.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.31.ff_norm.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.31.ff_out.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.31.ff_proj.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.31.k_norm.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.31.q_norm.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.4.att_proj.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.4.attn_norm.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.4.attn_out.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.4.ff_norm.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.4.ff_out.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.4.ff_proj.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.4.k_norm.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.4.q_norm.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.5.att_proj.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.5.attn_norm.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.5.attn_out.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.5.ff_norm.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.5.ff_out.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.5.ff_proj.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.5.k_norm.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.5.q_norm.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.6.att_proj.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.6.attn_norm.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.6.attn_out.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.6.ff_norm.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.6.ff_out.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.6.ff_proj.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.6.k_norm.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.6.q_norm.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.7.att_proj.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.7.attn_norm.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.7.attn_out.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.7.ff_norm.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.7.ff_out.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.7.ff_proj.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.7.k_norm.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.7.q_norm.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.8.att_proj.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.8.attn_norm.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.8.attn_out.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.8.ff_norm.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.8.ff_out.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.8.ff_proj.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.8.k_norm.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.8.q_norm.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.9.att_proj.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.9.attn_norm.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.9.attn_out.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.9.ff_norm.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.9.ff_out.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.9.ff_proj.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.9.k_norm.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.9.q_norm.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.ff_out.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.ln_f.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.wte.embedding": "model-00001-of-00007.safetensors",
+    "model.transformer.wte.new_embedding": "model-00001-of-00007.safetensors",
+    "model.vision_backbone.image_pooling_2d.wk.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_pooling_2d.wk.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_pooling_2d.wo.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_pooling_2d.wo.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_pooling_2d.wq.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_pooling_2d.wq.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_pooling_2d.wv.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_pooling_2d.wv.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_projector.w1.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_projector.w2.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_projector.w3.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.class_embedding": "model-00006-of-00007.safetensors",
+    "model.vision_backbone.image_vit.conv1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_backbone.image_vit.patch_embedding.weight": "model-00006-of-00007.safetensors",
+    "model.vision_backbone.image_vit.positional_embedding": "model-00006-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.0.attn.in_proj_bias": "model-00006-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.0.attn.in_proj_weight": "model-00006-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.0.attn.out_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.0.attn.out_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.0.ln_1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.0.ln_1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.0.ln_2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.0.ln_2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.0.mlp.c_fc.bias": "model-00006-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.0.mlp.c_fc.weight": "model-00006-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.0.mlp.c_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.0.mlp.c_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.1.attn.in_proj_bias": "model-00006-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.1.attn.in_proj_weight": "model-00006-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.1.attn.out_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.1.attn.out_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.1.ln_1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.1.ln_1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.1.ln_2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.1.ln_2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.1.mlp.c_fc.bias": "model-00006-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.1.mlp.c_fc.weight": "model-00006-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.1.mlp.c_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.1.mlp.c_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.10.attn.in_proj_bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.10.attn.in_proj_weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.10.attn.out_proj.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.10.attn.out_proj.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.10.ln_1.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.10.ln_1.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.10.ln_2.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.10.ln_2.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.10.mlp.c_fc.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.10.mlp.c_fc.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.10.mlp.c_proj.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.10.mlp.c_proj.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.11.attn.in_proj_bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.11.attn.in_proj_weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.11.attn.out_proj.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.11.attn.out_proj.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.11.ln_1.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.11.ln_1.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.11.ln_2.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.11.ln_2.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.11.mlp.c_fc.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.11.mlp.c_fc.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.11.mlp.c_proj.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.11.mlp.c_proj.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.12.attn.in_proj_bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.12.attn.in_proj_weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.12.attn.out_proj.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.12.attn.out_proj.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.12.ln_1.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.12.ln_1.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.12.ln_2.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.12.ln_2.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.12.mlp.c_fc.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.12.mlp.c_fc.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.12.mlp.c_proj.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.12.mlp.c_proj.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.13.attn.in_proj_bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.13.attn.in_proj_weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.13.attn.out_proj.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.13.attn.out_proj.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.13.ln_1.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.13.ln_1.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.13.ln_2.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.13.ln_2.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.13.mlp.c_fc.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.13.mlp.c_fc.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.13.mlp.c_proj.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.13.mlp.c_proj.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.14.attn.in_proj_bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.14.attn.in_proj_weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.14.attn.out_proj.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.14.attn.out_proj.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.14.ln_1.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.14.ln_1.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.14.ln_2.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.14.ln_2.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.14.mlp.c_fc.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.14.mlp.c_fc.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.14.mlp.c_proj.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.14.mlp.c_proj.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.15.attn.in_proj_bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.15.attn.in_proj_weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.15.attn.out_proj.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.15.attn.out_proj.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.15.ln_1.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.15.ln_1.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.15.ln_2.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.15.ln_2.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.15.mlp.c_fc.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.15.mlp.c_fc.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.15.mlp.c_proj.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.15.mlp.c_proj.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.16.attn.in_proj_bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.16.attn.in_proj_weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.16.attn.out_proj.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.16.attn.out_proj.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.16.ln_1.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.16.ln_1.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.16.ln_2.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.16.ln_2.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.16.mlp.c_fc.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.16.mlp.c_fc.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.16.mlp.c_proj.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.16.mlp.c_proj.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.17.attn.in_proj_bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.17.attn.in_proj_weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.17.attn.out_proj.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.17.attn.out_proj.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.17.ln_1.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.17.ln_1.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.17.ln_2.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.17.ln_2.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.17.mlp.c_fc.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.17.mlp.c_fc.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.17.mlp.c_proj.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.17.mlp.c_proj.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.18.attn.in_proj_bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.18.attn.in_proj_weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.18.attn.out_proj.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.18.attn.out_proj.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.18.ln_1.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.18.ln_1.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.18.ln_2.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.18.ln_2.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.18.mlp.c_fc.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.18.mlp.c_fc.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.18.mlp.c_proj.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.18.mlp.c_proj.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.19.attn.in_proj_bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.19.attn.in_proj_weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.19.attn.out_proj.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.19.attn.out_proj.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.19.ln_1.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.19.ln_1.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.19.ln_2.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.19.ln_2.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.19.mlp.c_fc.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.19.mlp.c_fc.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.19.mlp.c_proj.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.19.mlp.c_proj.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.2.attn.in_proj_bias": "model-00006-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.2.attn.in_proj_weight": "model-00006-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.2.attn.out_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.2.attn.out_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.2.ln_1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.2.ln_1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.2.ln_2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.2.ln_2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.2.mlp.c_fc.bias": "model-00006-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.2.mlp.c_fc.weight": "model-00006-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.2.mlp.c_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.2.mlp.c_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.20.attn.in_proj_bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.20.attn.in_proj_weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.20.attn.out_proj.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.20.attn.out_proj.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.20.ln_1.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.20.ln_1.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.20.ln_2.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.20.ln_2.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.20.mlp.c_fc.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.20.mlp.c_fc.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.20.mlp.c_proj.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.20.mlp.c_proj.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.21.attn.in_proj_bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.21.attn.in_proj_weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.21.attn.out_proj.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.21.attn.out_proj.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.21.ln_1.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.21.ln_1.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.21.ln_2.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.21.ln_2.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.21.mlp.c_fc.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.21.mlp.c_fc.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.21.mlp.c_proj.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.21.mlp.c_proj.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.22.attn.in_proj_bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.22.attn.in_proj_weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.22.attn.out_proj.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.22.attn.out_proj.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.22.ln_1.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.22.ln_1.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.22.ln_2.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.22.ln_2.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.22.mlp.c_fc.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.22.mlp.c_fc.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.22.mlp.c_proj.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.22.mlp.c_proj.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.23.attn.in_proj_bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.23.attn.in_proj_weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.23.attn.out_proj.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.23.attn.out_proj.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.23.ln_1.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.23.ln_1.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.23.ln_2.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.23.ln_2.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.23.mlp.c_fc.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.23.mlp.c_fc.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.23.mlp.c_proj.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.23.mlp.c_proj.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.3.attn.in_proj_bias": "model-00006-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.3.attn.in_proj_weight": "model-00006-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.3.attn.out_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.3.attn.out_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.3.ln_1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.3.ln_1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.3.ln_2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.3.ln_2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.3.mlp.c_fc.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.3.mlp.c_fc.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.3.mlp.c_proj.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.3.mlp.c_proj.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.4.attn.in_proj_bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.4.attn.in_proj_weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.4.attn.out_proj.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.4.attn.out_proj.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.4.ln_1.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.4.ln_1.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.4.ln_2.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.4.ln_2.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.4.mlp.c_fc.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.4.mlp.c_fc.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.4.mlp.c_proj.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.4.mlp.c_proj.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.5.attn.in_proj_bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.5.attn.in_proj_weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.5.attn.out_proj.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.5.attn.out_proj.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.5.ln_1.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.5.ln_1.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.5.ln_2.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.5.ln_2.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.5.mlp.c_fc.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.5.mlp.c_fc.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.5.mlp.c_proj.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.5.mlp.c_proj.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.6.attn.in_proj_bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.6.attn.in_proj_weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.6.attn.out_proj.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.6.attn.out_proj.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.6.ln_1.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.6.ln_1.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.6.ln_2.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.6.ln_2.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.6.mlp.c_fc.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.6.mlp.c_fc.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.6.mlp.c_proj.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.6.mlp.c_proj.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.7.attn.in_proj_bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.7.attn.in_proj_weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.7.attn.out_proj.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.7.attn.out_proj.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.7.ln_1.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.7.ln_1.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.7.ln_2.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.7.ln_2.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.7.mlp.c_fc.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.7.mlp.c_fc.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.7.mlp.c_proj.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.7.mlp.c_proj.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.8.attn.in_proj_bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.8.attn.in_proj_weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.8.attn.out_proj.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.8.attn.out_proj.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.8.ln_1.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.8.ln_1.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.8.ln_2.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.8.ln_2.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.8.mlp.c_fc.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.8.mlp.c_fc.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.8.mlp.c_proj.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.8.mlp.c_proj.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.9.attn.in_proj_bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.9.attn.in_proj_weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.9.attn.out_proj.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.9.attn.out_proj.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.9.ln_1.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.9.ln_1.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.9.ln_2.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.9.ln_2.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.9.mlp.c_fc.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.9.mlp.c_fc.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.9.mlp.c_proj.bias": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.image_vit.transformer.resblocks.9.mlp.c_proj.weight": "model-00007-of-00007.safetensors",
+    "model.vision_backbone.pad_embed": "model-00006-of-00007.safetensors"
+  }
+}

preprocessing_patram.py

@@ -0,0 +1,192 @@
+"""
+Processor class for Patram.
+"""
+
+from typing import Optional
+
+import PIL
+from PIL import ImageOps
+from PIL.Image import Image
+
+try:
+    from typing import Unpack
+except ImportError:
+    from typing_extensions import Unpack
+
+import numpy as np
+import torch
+
+from transformers.image_utils import ImageInput
+from transformers.processing_utils import (
+    TextKwargs,
+    ProcessingKwargs,
+    ProcessorMixin,
+)
+
+from transformers.tokenization_utils_base import TextInput, PreTokenizedInput
+from transformers.utils import logging
+
+from transformers import AutoTokenizer
+from .image_preprocessing_patram import PatramImagesKwargs, PatramImageProcessor
+
+
+logger = logging.get_logger(__name__)
+
+
+DEFAULT_IMAGE_PATCH_TOKEN = f"<im_patch>"
+DEFAULT_IM_START_TOKEN = f"<im_start>"
+DEFAULT_IM_END_TOKEN = f"<im_end>"
+DEFAULT_IM_COL_TOKEN = f"<im_col>"
+IMAGE_PROMPT = "<|image|>"
+
+EXTRA_TOKENS = (DEFAULT_IM_START_TOKEN, DEFAULT_IM_END_TOKEN, DEFAULT_IMAGE_PATCH_TOKEN, DEFAULT_IM_COL_TOKEN, IMAGE_PROMPT)
+
+
+def get_special_token_ids(tokenizer):
+    ids = tokenizer.encode("".join(EXTRA_TOKENS), add_special_tokens=False)
+    assert len(ids) == len(EXTRA_TOKENS)
+    return {k: i for k, i in zip(EXTRA_TOKENS, ids)}
+
+
+class PatramTextKwargs(TextKwargs, total=False):
+    style: Optional[str]
+    system_prompt: Optional[str]
+    message_format: Optional[str]
+    always_start_with_space: Optional[bool]
+    sequence_length: Optional[int]
+
+
+class PatramProcessorKwargs(ProcessingKwargs, total=False):
+    text_kwargs: PatramTextKwargs
+    images_kwargs: PatramImagesKwargs
+    _defaults = {
+        "images_kwargs": {
+            "max_crops": 12,
+            "overlap_margins": [4, 4],
+            "base_image_input_size": [336, 336],
+            "image_token_length_w": 12,
+            "image_token_length_h": 12,
+            "image_patch_size": 14,
+            "image_padding_mask": True,
+        },
+        "text_kwargs": {
+            "style": "long_caption",
+            "system_prompt": "none",
+            "message_format": "role",
+            "always_start_with_space": True,
+            "sequence_length": 1536,
+            "padding": False,
+        },
+    }
+
+
+class PatramProcessor(ProcessorMixin):
+    attributes = ["image_processor", "tokenizer"]
+    image_processor_class = "AutoImageProcessor"
+    tokenizer_class = ("GPT2Tokenizer", "GPT2TokenizerFast")
+
+    def __init__(self, image_processor: PatramImageProcessor = None, tokenizer : AutoTokenizer = None, **kwargs):
+        # self.image_processor = image_processor
+        # self.tokenizer = tokenizer
+        super().__init__(image_processor, tokenizer)
+        self._special_tokens = None
+
+    @property
+    def special_token_ids(self):
+        if self._special_tokens is None:
+            self._special_tokens = get_special_token_ids(self.tokenizer)
+        return self._special_tokens
+
+    def get_tokens_input(self, prompt, message_format, always_start_with_space):
+        if message_format == "none" or message_format is None:
+            pass
+        elif message_format == "role":
+            prompt = "User: " + prompt + " Assistant:"
+        else:
+            raise NotImplementedError(f"Message format {message_format} not implemented")
+
+        if always_start_with_space:
+            prompt = " " + prompt
+
+        tokens = self.tokenizer.encode(prompt, add_special_tokens=False)
+
+        return tokens
+
+    def process(
+        self,
+        text: TextInput = None,
+        images: ImageInput = None,
+        *,
+        tokens: Optional[PreTokenizedInput] = None,
+        **kwargs: Unpack[PatramProcessorKwargs],
+    ):
+        output_kwargs = self._merge_kwargs(
+            PatramProcessorKwargs,
+            tokenizer_init_kwargs=self.tokenizer.init_kwargs,
+            **kwargs,
+        )
+
+        if tokens is None:
+            tokens = self.get_tokens_input(
+                text,
+                output_kwargs["text_kwargs"]["message_format"],
+                output_kwargs["text_kwargs"]["always_start_with_space"],
+            )
+
+        image_token_id = self.special_token_ids[IMAGE_PROMPT]
+
+        if images is not None:
+            if not isinstance(images, (list, tuple)):
+                images = [images]
+            image_arrays = []
+            for image in images:
+                if isinstance(image, Image):
+                    image = image.convert("RGB")
+                    # Handle images with EXIF orientation tags, which PIL will ignore by default
+                    # https://github.com/python-pillow/Pillow/issues/4703
+                    img = ImageOps.exif_transpose(image)
+                    image_arrays.append(np.array(image))
+                else:
+                    assert len(image.shape) == 3 and image.shape[-1] == 3
+                    image_arrays.append(image.astype(np.uint8))
+            images = image_arrays
+            # For now only support inserting images at the start
+            image_idx = [-1]*len(images)
+        else:
+            image_idx = None
+
+        sequence_length = output_kwargs["text_kwargs"]["sequence_length"]
+
+        image_patch_token_id = self.special_token_ids[DEFAULT_IMAGE_PATCH_TOKEN]
+        image_col_token_id = self.special_token_ids[DEFAULT_IM_COL_TOKEN]
+        image_start_token_id = self.special_token_ids[DEFAULT_IM_START_TOKEN]
+        image_end_token_id = self.special_token_ids[DEFAULT_IM_END_TOKEN]
+        out = self.image_processor.multimodal_preprocess(
+            images=images,
+            image_idx=image_idx,
+            tokens=np.asarray(tokens).astype(np.int32),
+            sequence_length=sequence_length,
+            image_patch_token_id=image_patch_token_id,
+            image_col_token_id=image_col_token_id,
+            image_start_token_id=image_start_token_id,
+            image_end_token_id=image_end_token_id,
+            **output_kwargs["images_kwargs"]
+        )
+
+        # Prepend BOS
+        # qwen2 and olmo do not have a BOS, and instead use EOS as a generic seperator token.
+        bos = self.tokenizer.bos_token_id or self.tokenizer.eos_token_id
+        decoder_input_tokens = np.pad(out["input_ids"], [[1, 0]], constant_values=bos)
+        out["input_ids"] = decoder_input_tokens
+        if "image_input_idx" in out:
+            # Shift patch mapping up by one since we added BOS
+            image_input_idx = out["image_input_idx"]
+            out["image_input_idx"] = np.where(image_input_idx < 0, image_input_idx, image_input_idx + 1)
+
+        for k, v in out.items():
+            out[k] = torch.from_numpy(v)
+
+        return out
+
+
+PatramProcessor.register_for_auto_class()

preprocessor_config.json

@@ -0,0 +1,32 @@
+{
+  "auto_map": {
+    "AutoImageProcessor": "image_preprocessing_patram.PatramImageProcessor",
+    "AutoProcessor": "preprocessing_patram.PatramProcessor"
+  },
+  "base_image_input_size": [
+    336,
+    336
+  ],
+  "do_normalize": true,
+  "image_mean": [
+    0.48145466,
+    0.4578275,
+    0.40821073
+  ],
+  "image_padding_mask": true,
+  "image_patch_size": 14,
+  "image_processor_type": "PatramImageProcessor",
+  "image_std": [
+    0.26862954,
+    0.26130258,
+    0.27577711
+  ],
+  "image_token_length_h": 12,
+  "image_token_length_w": 12,
+  "max_crops": 12,
+  "overlap_margins": [
+    4,
+    4
+  ],
+  "processor_class": "PatramProcessor"
+}

special_tokens_map.json

@@ -0,0 +1,37 @@
+{
+  "additional_special_tokens": [
+    "<im_start>",
+    "<im_end>",
+    "<im_patch>",
+    "<im_col>",
+    "<|image|>"
+  ],
+  "bos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|pad|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,240 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "100256": {
+      "content": "<|extra_id_0|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "100257": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100258": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100259": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100260": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100261": {
+      "content": "|||PHONE_NUMBER|||",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "100262": {
+      "content": "|||EMAIL_ADDRESS|||",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "100263": {
+      "content": "|||IP_ADDRESS|||",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "100264": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100265": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100266": {
+      "content": "<|extra_id_1|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "100267": {
+      "content": "<|extra_id_2|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "100268": {
+      "content": "<|extra_id_3|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "100269": {
+      "content": "<|extra_id_4|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "100270": {
+      "content": "<|extra_id_5|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "100271": {
+      "content": "<|extra_id_6|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "100272": {
+      "content": "<|extra_id_7|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "100273": {
+      "content": "<|extra_id_8|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "100274": {
+      "content": "<|extra_id_9|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "100275": {
+      "content": "<|extra_id_10|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "100276": {
+      "content": "<|endofprompt|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100277": {
+      "content": "<|pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100278": {
+      "content": "<im_start>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100279": {
+      "content": "<im_end>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100280": {
+      "content": "<im_patch>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100281": {
+      "content": "<im_col>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100282": {
+      "content": "<|image|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<im_start>",
+    "<im_end>",
+    "<im_patch>",
+    "<im_col>",
+    "<|image|>"
+  ],
+  "auto_map": {
+    "AutoProcessor": "preprocessing_patram.PatramProcessor"
+  },
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "extra_special_tokens": {},
+  "model_max_length": 8192,
+  "pad_token": "<|pad|>",
+  "processor_class": "PatramProcessor",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>"
+}