Add files using upload-large-folder tool

README.md

@@ -0,0 +1,297 @@
+---
+license: mit
+language:
+  - en
+tags:
+  - multimodal
+  - vision-language
+  - reasoning
+  - math
+  - ocr
+  - gui-grounding
+  - computer-use
+  - chain-of-thought
+base_model: microsoft/Phi-4-reasoning
+pipeline_tag: image-text-to-text
+model-index:
+  - name: Phi-4-Reasoning-Vision-15B
+    results:
+      - task:
+          type: visual-question-answering
+        dataset:
+          name: AI2D
+          type: ai2d
+        metrics:
+          - type: accuracy
+            value: 84.8
+      - task:
+          type: visual-question-answering
+        dataset:
+          name: ChartQA
+          type: chartqa
+        metrics:
+          - type: accuracy
+            value: 83.3
+      - task:
+          type: visual-question-answering
+        dataset:
+          name: MathVista (MINI)
+          type: mathvista
+        metrics:
+          - type: accuracy
+            value: 75.2
+      - task:
+          type: visual-question-answering
+        dataset:
+          name: MMMU
+          type: mmmu
+        metrics:
+          - type: accuracy
+            value: 54.3
+      - task:
+          type: visual-question-answering
+        dataset:
+          name: OCRBench
+          type: ocrbench
+        metrics:
+          - type: accuracy
+            value: 76.0
+      - task:
+          type: visual-question-answering
+        dataset:
+          name: ScreenSpot-V2
+          type: screenspot-v2
+        metrics:
+          - type: accuracy
+            value: 88.2
+---
+
+# Phi-4-Reasoning-Vision-15B
+
+[![Microsoft](https://img.shields.io/badge/Microsoft-Project-0078D4?logo=microsoft)](https://www.microsoft.com/en-us/research/blog/phi-4-reasoning-vision-and-the-lessons-of-training-a-multimodal-reasoning-model/)
+[![Foundry](https://img.shields.io/badge/Azure-Foundry-0089D6)](https://aka.ms/Phi-4-r-v-foundry)
+[![Github](https://img.shields.io/badge/Github-181717?logo=github&logoColor=white)](https://github.com/microsoft/phi-4-reasoning-vision-15B)
+[![Paper](https://img.shields.io/badge/Paper-2511.19663-red)](https://aka.ms/Phi-4-reasoning-vision-15B-TR)
+
+[Official Microsoft Blog](https://www.microsoft.com/en-us/research/blog/phi-4-reasoning-vision-and-the-lessons-of-training-a-multimodal-reasoning-model/)<br>
+[Technical Report](https://aka.ms/Phi-4-reasoning-vision-15B-TR)<br>
+[Github](https://github.com/microsoft/phi-4-reasoning-vision-15B)<br>
+[Try Phi-4-Reasoning-Vision-15B on Microsoft Foundry](https://aka.ms/Phi-4-r-v-foundry)<br>
+
+**Developer:** Microsoft Corporation  
+**Authorized Representative:** Microsoft Ireland Operations Limited, 70 Sir John Rogerson's Quay, Dublin 2, D02 R296, Ireland  
+**Release Date:** March 4, 2026  
+**License:** [MIT](https://opensource.org/licenses/MIT)  
+**Parameters:** 15B  
+**Context Length:** 16,384 tokens  
+**Inputs:** Text and Images  
+**Outputs:** Text  
+**Training GPUs:** 240 B200s  
+**Training Time:** 4 days  
+**Training Dates:** February 3, 2025 – February 21, 2026  
+**Model Dependencies:** [Phi-4-Reasoning](https://huggingface.co/microsoft/Phi-4-reasoning)  
+
+---
+
+## 1. Model Overview
+
+Phi-4-Reasoning-Vision-15B is a compact open-weight multimodal reasoning model built on the Phi-4-Reasoning language model backbone and the SigLIP-2 vision encoder, using a mid-fusion architecture. In this architecture, the vision encoder first converts images into visual tokens, which are then projected into the language model's embedding space and injected into the pretrained language model. This approach leverages the strengths of both pretrained components while keeping training and inference costs manageable. The model employs a dynamic resolution vision encoder with up to 3,600 visual tokens, enabling high-resolution image understanding critical for tasks such as GUI grounding and fine-grained document analysis. Bidirectional attention is applied within images (intra-image) to improve spatial reasoning without the overfitting risks observed with broader bidirectional schemes.
+
+Phi-4-Reasoning-Vision-15B is trained with Supervised Fine-Tuning (SFT) on a carefully curated mixture of reasoning and non-reasoning data. Rather than training separate models for each mode, the model operates as a single system that can invoke extended chain-of-thought reasoning (using `<think>...</think>` blocks) for tasks like mathematical and scientific reasoning, or default to direct inference (tagged with `<nothink>`) for perception-focused tasks such as captioning, object detection, and grounding. The training data consists primarily of meticulously filtered and improved open-source vision-language datasets, supplemented by high-quality domain-specific data from internal Microsoft teams and targeted data acquisitions. This data-centric approach, combined with moderate training compute requirements (240 NVIDIA B200 GPUs for 4 days), distinguishes Phi-4-Reasoning-Vision-15B from models that rely on substantially more training data and compute.
+
+### 1.1 Alignment Approach
+
+Phi-4-Reasoning-Vision-15B has adopted a safety post-training approach leveraging a combination of open-source and in-house generated synthetic datasets. The safety alignment is achieved through Supervised Fine-Tuning (SFT) using data that includes both helpfulness and harmlessness examples, as well as targeted questions and answers across multiple safety categories. The model's training data explicitly includes safety-oriented samples designed to teach appropriate refusal behavior for harmful content categories including hate speech, violence, self-harm content, and sexually explicit material. Automated red teaming was performed on Azure to assess safety risks including groundedness, jailbreak susceptibility, harmful content generation, and copyright violations for protected material.
+
+---
+
+## 2. Usage
+
+### 2.1 Primary Use Cases
+
+Phi-4-Reasoning-Vision-15B is designed for general-purpose multimodal AI systems and applications that require vision-language understanding with selective reasoning capabilities, particularly in memory- or compute-constrained environments. The model excels in two primary domains:
+
+- **Scientific and mathematical reasoning over visual inputs:** such as solving math problems presented as handwritten equations or diagrams, extracting and reasoning over quantitative information in documents, charts, and tables, and supporting multi-step reasoning in educational or scientific analysis contexts.
+- **Computer-use agent (CUA) tasks:** such as interpreting screen content, localizing interactive GUI elements, and selecting actions within graphical user interfaces.
+
+The model is also capable of general multimodal tasks including image captioning, visual question answering, optical character recognition, object localization, and grounding. Its hybrid reasoning design allows it to produce fast, direct responses for perception-focused tasks while engaging in structured chain-of-thought reasoning when the task benefits from it, making it suitable as a building block for generative AI-powered features across a range of applications.
+
+### 2.2 Out-of-Scope Use Cases
+
+Phi-4-Reasoning-Vision-15B is not specifically designed or evaluated for all downstream purposes. Developers should consider common limitations of vision-language models as they select use cases, and evaluate and mitigate for accuracy, safety, and fairness before using within a specific downstream use case, particularly for high-risk scenarios.
+
+The model is trained primarily on English text and image-text pairs. Languages other than English may experience degraded performance. The model should not be used in scenarios that could have consequential impact on legal status or the allocation of resources or life opportunities (e.g., housing, employment, credit) without further assessments and additional debiasing techniques. It is not suitable for providing medical diagnoses, legal advice, or financial planning. Developers should be aware of and adhere to applicable laws or regulations (including privacy, trade compliance laws, etc.) that are relevant to their use case. Nothing contained in this Model Card should be interpreted as or deemed a restriction or modification to the license the model is released under.
+
+### 2.3 Distribution Channels
+
+Some of Phi-4-Reasoning-Vision-15B's distribution channels include:
+
+- Public access through open-source repositories: [Hugging Face](https://huggingface.co/microsoft/Phi-4-Reasoning-Vision-15B)
+- Public access through open-source code repositories: [GitHub](https://github.com/microsoft/Phi-4-vision)
+- Enterprise or subscription-based access through [Azure AI Foundry](https://ai.azure.com)
+
+### 2.4 Input Formats
+
+Given the nature of the training data, always use chat template and system prompt for inference. For example, for the prompt "Please describe the image", the fully formatted chat templated prompt is the following:
+
+```
+<|im_start|>system<|im_sep|>You are Phi, a multimodal model trained by Microsoft to help users. Your role as an assistant is to provide accurate, coherent, and actionable responses, adapting your reasoning mode ("NOTHINK" vs "THINK") automatically based on the complexity, clarity, and confidence of each task.
+
+#### NOTHINK Mode
+Use this mode when the task is clear, factual, low-complexity, or can be confidently answered immediately without iterative reasoning. Such as when the input is clear and unambiguous or visual recognition or text comprehension is straightforward, and where a factual, numeric, or short procedural answer is sufficient. Provide a concise, accurate, and confident answer. Please structure your response into one section: using the specified format: <nothink> {Solution section}. In the Solution section, present the final solution that you deem correct. The Solution section should be logical, accurate, and concise.
+
+#### THINK Mode
+This requires engaging in a comprehensive cycle of analysis, summarizing, exploration, reassessment, reflection, backtracing, and iteration to develop well-considered thinking process. Use this mode when multiple modalities must be integrated, the task involves analysis, inference, design, or planning, the query is ambiguous, multi-step, or requires judgment. Think through the visual and textual context before responding. Please structure your response into two main sections: Thought and Solution using the specified format: <think> {Thought section} </think> {Solution section}. In the Thought section, detail your reasoning process in steps. Each step should include detailed considerations such as analysing questions, summarizing relevant findings, brainstorming new ideas, verifying the accuracy of the current steps, refining any errors, and revisiting previous steps. In the Solution section, based on various attempts, explorations, and reflections from the Thought section, systematically present the final solution that you deem correct. The Solution section should be logical, accurate, and concise and detail necessary steps needed to reach the conclusion.
+
+Now, try to solve the following question through the above guidelines:<|im_end|><|im_start|>user<|im_sep|>Please describe the image<|im_end|><|im_start|>assistant<|im_sep|>
+```
+
+To force a thinking response, append the `<think>` token to the generation template:
+
+```
+<|im_start|>assistant<|im_sep|><think>
+```
+
+To force a non-thinking response, append the `<nothink>` token to the generation template:
+
+```
+<|im_start|>assistant<|im_sep|><nothink>
+```
+
+### 2.5 Technical Requirements and Integration Guidance
+
+The following software packages are required for running Phi-4-Reasoning-Vision:
+
+- `torch >= 2.7.1`
+- `transformers >= 4.57.1`
+- `vllm >= 0.15.2` (only required if using vLLM)
+
+Phi-4-Reasoning-Vision-15B has been tested on NVIDIA A6000, A100, H100, and B200 GPUs with the Ubuntu 22.04.5 LTS operating system. In principle, other GPU architectures with enough memory to fit the model could suffice, but these have not been tested. It is recommended that users host Phi-4-Reasoning-Vision-15B on a vLLM server using bf16 precision.
+
+### 2.6 Responsible AI Considerations
+
+Like other models, Phi-4-Reasoning-Vision-15B can potentially behave in ways that are unfair, unreliable, or offensive. Some of the limiting behaviors to be aware of include:
+
+- **Quality of Service:** The model is trained primarily on English text. Languages other than English may experience worse performance. English language varieties with less representation in the training data might experience worse performance than standard American English. Phi-4-Reasoning-Vision-15B is not intended to support multilingual use.
+- **Representation of Harms & Perpetuation of Stereotypes:** The model may over- or under-represent groups of people, erase representation of some groups, or reinforce demeaning or negative stereotypes. Despite safety post-training, these limitations may still be present due to differing levels of representation of different groups or prevalence of examples of negative stereotypes in training data that reflect real-world patterns and societal biases.
+- **Inappropriate or Offensive Content:** The model may produce inappropriate or offensive content, which may make it inappropriate to deploy in sensitive contexts without additional mitigations specific to the use case.
+- **Information Reliability:** Language models can generate nonsensical content or fabricate content that might sound reasonable but is inaccurate or outdated.
+
+Developers should apply responsible AI best practices and are responsible for ensuring that a specific use case complies with relevant laws and regulations (e.g., privacy, trade, etc.). Using safety services like Azure AI Content Safety that have advanced guardrails is highly recommended. Important areas for consideration include:
+
+- **Allocation:** Models may not be suitable for scenarios that could have consequential impact on legal status or the allocation of resources or life opportunities (e.g., housing, employment, credit) without further assessments and additional debiasing techniques.
+- **High-Risk Scenarios:** Developers should assess suitability of using models in high-risk scenarios where unfair, unreliable, or offensive outputs might be extremely costly or lead to harm. This includes providing advice in sensitive or expert domains where accuracy and reliability are critical (e.g., legal or health advice). Additional safeguards should be implemented at the application level according to the deployment context.
+- **Misinformation:** Models may produce inaccurate information. Developers should follow transparency best practices and inform end-users they are interacting with an AI system. At the application level, developers can build feedback mechanisms and pipelines to ground responses in use-case specific, contextual information, a technique known as Retrieval Augmented Generation (RAG).
+- **Generation of Harmful Content:** Developers should assess outputs for their context and use available safety classifiers or custom solutions appropriate for their use case.
+- **Misuse:** Other forms of misuse such as fraud, spam, or malware production may be possible, and developers should ensure that their applications do not violate applicable laws and regulations.
+
+---
+
+## 3. Quality and Performance Evaluation
+
+Phi-4-Reasoning-Vision-15B was evaluated across a broad range of public benchmarks spanning multimodal reasoning, mathematical problem solving, document and chart understanding, visual perception, OCR, and computer-use grounding tasks. Two evaluation frameworks were used: Microsoft's Eureka ML Insights for internal development benchmarks, and VLMEvalKit for standardized community benchmarks. Evaluation logs will be released publicly.
+
+The model was evaluated on the following benchmarks via VLMEvalKit: AI2D (diagram understanding), BLINK (core visual perception), ChartQA (chart reasoning), DocVQA (document question answering), HallusionBench (hallucination and visual illusion detection), MathVerse (visual math with varying multimodal information), MathVision (competition-level mathematical reasoning), MathVista (math reasoning in visual contexts), MMMU (multi-discipline multimodal understanding), MMStar (vision-indispensable multimodal evaluation), OCRBench (OCR capabilities), ScreenSpot-V2 for Desktop, Mobile, and Web (GUI element localization), WeMath (human-like mathematical reasoning process evaluation), WildVision (real-world human preference evaluation), and ZeroBench (challenging visual reasoning). During development, additional benchmarks including MMMU-CoT, ScreenSpot-Pro, and V*Bench were evaluated using Eureka ML Insights.
+
+### Table 1: Accuracy Comparisons Relative to Popular Open-Weight, Non-Thinking Models
+
+| Benchmark | Phi-4-reasoning-vision-15B | Phi-4-reasoning-vision-15B – force nothink | Phi-4-mm-instruct | Kimi-VL-A3B-Instruct | gemma-3-12b-it | Qwen3-VL-8B-Instruct-4K | Qwen3-VL-8B-Instruct-32K | Qwen3-VL-32B-Instruct-4K | Qwen3-VL-32B-Instruct-32K |
+|---|---|---|---|---|---|---|---|---|---|
+| AI2D_TEST | 84.8 | 84.7 | 68.6 | 84.6 | 80.4 | 82.7 | 83 | 84.8 | 85 |
+| ChartQA_TEST | 83.3 | 76.5 | 23.5 | 87 | 39 | 83.1 | 83.2 | 84.3 | 84 |
+| HallusionBench | 64.4 | 63.1 | 56 | 65.2 | 65.3 | 73.5 | 74.1 | 74.4 | 74.9 |
+| MathVerse_MINI | 44.9 | 43.8 | 32.4 | 41.7 | 29.8 | 54.5 | 57.4 | 64.2 | 64.2 |
+| MathVision_MINI | 36.2 | 34.2 | 20 | 28.3 | 31.9 | 45.7 | 50 | 54.3 | 60.5 |
+| MathVista_MINI | 75.2 | 68.7 | 50.5 | 67.1 | 57.4 | 77.1 | 76.4 | 82.5 | 81.8 |
+| MMMU_VAL | 54.3 | 52 | 42.3 | 52 | 50 | 60.7 | 64.6 | 68.6 | 70.6 |
+| MMStar | 64.5 | 63.3 | 45.9 | 60 | 59.4 | 68.9 | 69.9 | 73.7 | 74.3 |
+| OCRBench | 76 | 75.6 | 62.6 | 86.5 | 75.3 | 89.2 | 90 | 88.5 | 88.5 |
+| ScreenSpot_v2 | 88.2 | 88.3 | 28.5 | 89.8 | 3.5 | 91.5 | 91.5 | 93.7 | 93.9 |
+
+### Table 2: Accuracy Comparisons Relative to Popular Open-Weight, Thinking Models
+
+| Benchmark | Phi-4-reasoning-vision-15B | Phi-4-reasoning-vision-15B - force thinking | Kimi-VL-A3B-Thinking | gemma3-12b-it | Qwen3-VL-8B-Thinking-4K | Qwen3-VL-8B-Thinking-40K | Qwen3-VL-32B-Thinking-4K | Qwen3-VL-32B-Thinking-40K |
+|---|---|---|---|---|---|---|---|---|
+| AI2D_TEST | 84.8 | 79.7 | 81.2 | 80.4 | 83.5 | 83.9 | 86.9 | 87.2 |
+| ChartQA_TEST | 83.3 | 82.9 | 73.3 | 39 | 78 | 78.6 | 78.5 | 79.1 |
+| HallusionBench | 64.4 | 63.9 | 70.6 | 65.3 | 71.6 | 73 | 76.4 | 76.6 |
+| MathVerse_MINI | 44.9 | 53.1 | 61 | 29.8 | 67.3 | 73.3 | 78.3 | 78.2 |
+| MathVision_MINI | 36.2 | 36.2 | 50.3 | 31.9 | 43.1 | 50.7 | 60.9 | 58.6 |
+| MathVista_MINI | 75.2 | 74.1 | 78.6 | 57.4 | 77.7 | 79.5 | 83.9 | 83.8 |
+| MMMU_VAL | 54.3 | 55 | 60.2 | 50 | 59.3 | 65.3 | 72 | 72.2 |
+| MMStar | 64.5 | 63.9 | 69.6 | 59.4 | 69.3 | 72.3 | 75.5 | 75.7 |
+| OCRBench | 76 | 73.7 | 79.9 | 75.3 | 81.2 | 82 | 83.7 | 85 |
+| ScreenSpot_v2 | 88.2 | 88.1 | 81.8 | 3.5 | 93.3 | 92.7 | 83.1 | 83.1 |
+
+### 3.1 Safety Evaluation and Red-Teaming
+
+Phi-4-Reasoning-Vision-15B was trained on a mixture of public safety data and internally generated tasks that it ought to refuse based on Microsoft's Responsible AI Policy.
+
+Phi-4-Reasoning-Vision-15B's safety was evaluated using both quantitative and qualitative approaches prior to release. Automated red teaming was performed on Azure to assess safety risks across multiple risk categories, including disallowed content (sexual, violent, hateful, or self-harm content), copyright content and intellectual property, and jailbreak susceptibility. The evaluation assessed the model's groundedness and its tendency to generate fabricated or misleading information.
+
+The safety evaluation built upon the established practices from the Phi-4-Reasoning model's safety assessment. The model's training data included explicit safety-oriented samples across both reasoning and non-reasoning modes, designed to teach appropriate refusal and harm-avoidance behaviors. The multimodal nature of the model introduces additional safety considerations around visual content interpretation, and evaluations were conducted to assess the model's behavior when presented with potentially harmful or misleading visual inputs.
+
+| Evaluation | Description | Defect Rate |
+|---|---|---|
+| Text to Text Safety | Automated content safety evaluation measuring safety policies | 1.4% |
+| Image to Text Safety | Automated content safety evaluation measuring safety policies | 4.5% |
+
+---
+
+## 4. Data Overview
+
+### 4.1 Training, Testing, and Validation Datasets
+
+To learn more about the training data used for Phi-4-Reasoning-Vision-15B please refer to the full data card: RRRR_nnnn_Data Card for Foundation+Frontier Models.
+
+### 4.2 List of Data Sources
+
+To learn more about the training data used for Phi-4-Reasoning-Vision-15B please refer to the full data card: RRRR_nnnn_Data Card for Foundation+Frontier Models.
+
+---
+
+## 5. Contact
+
+Requests for additional information can be directed to [MSFTAIActRequest@microsoft.com](mailto:MSFTAIActRequest@microsoft.com).
+
+Authorized representative: Microsoft Ireland Operations Limited, 70 Sir John Rogerson's Quay, Dublin 2, D02 R296, Ireland
+
+---
+
+## 6. Appendix
+
+### A. Benchmarking Methodology
+
+Phi-4-Reasoning-Vision-15B was evaluated using two complementary open-source evaluation frameworks:
+
+**1. [Eureka ML Insights](https://github.com/microsoft/eureka-ml-insights)**
+
+Used during development for internal benchmarks and ablation studies. The following benchmarks were evaluated through this framework:
+
+- **MathVista:** Mathematical reasoning over visual inputs including diagrams, charts, and figures
+- **MMMU-CoT:** Multi-discipline multimodal understanding with chain-of-thought reasoning
+- **ScreenSpot / ScreenSpot-V2:** GUI element localization on desktop and mobile screenshots
+- **ScreenSpot-Pro:** High-resolution professional GUI grounding tasks
+- **V\*Bench:** Visual reasoning benchmark
+
+**2. [VLMEvalKit](https://github.com/open-compass/VLMEvalKit)**
+
+Used for standardized community benchmark evaluation. The following benchmarks were evaluated through this framework:
+
+- **AI2D (TEST split):** Diagram understanding over ~5K illustrative diagrams from grade school natural sciences, evaluating the ability to interpret diagrammatic elements, relationships, and structure.
+- **BLINK:** Core visual perception benchmark with 3,807 multiple-choice questions spanning 14 classic computer vision tasks including relative depth estimation, visual correspondence, and multi-view reasoning.
+- **ChartQA (TEST split):** Chart understanding and reasoning benchmark with 9,600 human-written questions assessing complex visual and logical reasoning over chart data.
+- **DocVQA (VAL split):** Document visual question answering over 12,000+ document images, evaluating text extraction and comprehension within document layouts.
+- **HallusionBench:** Diagnostic benchmark evaluating image-context reasoning, language hallucination tendencies, and visual illusion susceptibility in vision-language models.
+- **MathVerse (MINI split):** Visual math benchmark with 2,612 multi-subject math problems transformed into six versions offering varying degrees of multimodal information content.
+- **MathVision (MINI split):** 3,040 high-quality mathematical problems with visual contexts sourced from real math competitions, spanning 16 mathematical disciplines across 5 difficulty levels.
+- **MathVista (MINI split):** Mathematical reasoning in visual contexts including geometry, algebra, and data interpretation.
+- **MMMU (DEV_VAL split):** Massive multi-discipline multimodal understanding benchmark with 11.5K questions from college exams covering six core disciplines and 30 subjects.
+- **MMStar:** Vision-indispensable multimodal benchmark with 1,500 carefully curated samples evaluating six core capabilities: coarse perception, fine-grained perception, instance reasoning, logical reasoning, science and technology, and mathematics.
+- **OCRBench:** Comprehensive OCR evaluation with 1,000 question-answer pairs spanning text recognition, scene text VQA, document-oriented VQA, key information extraction, and handwritten mathematical expression recognition.
+- **ScreenSpot-V2 (Desktop, Mobile, Web):** GUI element localization benchmark across desktop, mobile, and web interfaces.
+- **WeMath:** Mathematical reasoning process benchmark with 6.5K visual math problems spanning 67 hierarchical knowledge concepts, evaluating knowledge acquisition and generalization beyond end-to-date performance.
+- **WildVision:** Real-world human preference evaluation benchmark with 500 high-quality samples curated from 8,000 user submissions, using GPT-4o as judge.
+- **ZeroBench:** Challenging visual reasoning benchmark with 100 manually curated questions designed to probe the limits of spatial reasoning, object recognition, and complex visual scene interpretation.
+
+Evaluation logs will be released publicly.

chat_template.jinja

@@ -0,0 +1 @@
+<|im_start|>system<|im_sep|>You are Phi, a multimodal model trained by Microsoft to help users.  Your role as an assistant is to provide accurate, coherent, and actionable responses, adapting your reasoning mode (\"NOTHINK\" vs \"THINK\") automatically based on the complexity, clarity, and confidence of each task.\n\n#### NOTHINK Mode\nUse this mode when the task is clear, factual, low-complexity, or can be confidently answered immediately without iterative reasoning. Such as when the input is clear and unambiguous or visual recognition or text comprehension is straightforward, and where a factual, numeric, or short procedural answer is sufficient. Provide a concise, accurate, and confident answer. Please structure your response into one section: using the specified format: <nothink> {Solution section}. In the Solution section, present the final solution that you deem correct. The Solution section should be logical, accurate, and concise.\n\n#### THINK Mode\nThis requires engaging in a comprehensive cycle of analysis, summarizing, exploration, reassessment, reflection, backtracing, and iteration to develop well-considered thinking process. Use this mode when multiple modalities must be integrated, the task involves analysis, inference, design, or planning, the query is ambiguous, multi-step, or requires judgment.  Think through the visual and textual context before responding. Please structure your response into two main sections: Thought and Solution using the specified format: <think> {Thought section} </think> {Solution section}. In the Thought section, detail your reasoning process in steps. Each step should include detailed considerations such as analysing questions, summarizing relevant findings, brainstorming new ideas, verifying the accuracy of the current steps, refining any errors, and revisiting previous steps. In the Solution section, based on various attempts, explorations, and reflections from the Thought section, systematically present the final solution that you deem correct. The Solution section should be logical, accurate, and concise and detail necessary steps needed to reach the conclusion.\n\nNow, try to solve the following question through the above guidelines:<|im_end|>{% for message in messages %}{% if (message['role'] == 'user') %}{{'<|im_start|>user<|im_sep|>' + message['content'] + '<|im_end|>'}}{% elif (message['role'] == 'assistant') %}{{'<|im_start|>assistant<|im_sep|>'}}{% generation %}{{message['content'] + '<|im_end|>'}}{% endgeneration %}{% endif %}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant<|im_sep|>' }}{% endif %}

config.json

@@ -0,0 +1,58 @@
+{
+  "architectures": [
+    "Phi4ForCausalLMV"
+  ],
+  "auto_map": {
+    "AutoConfig": "modeling_phi4_visionr.Phi4VisionR",
+    "AutoModelForCausalLM": "modeling_phi4_visionr.Phi4ForCausalLMV",
+    "AutoProcessor": "processing_phi4_visionr.Phi4VisionRProcessor"
+  },
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 100257,
+  "dtype": "bfloat16",
+  "embd_pdrop": 0.0,
+  "eos_token_id": 100265,
+  "freeze_mm_mlp_adapter": false,
+  "hidden_act": "silu",
+  "hidden_size": 5120,
+  "image_aspect_ratio": "square",
+  "initializer_range": 0.02,
+  "intermediate_size": 17920,
+  "max_num_patches": 3600,
+  "max_position_embeddings": 32768,
+  "min_num_patches": 256,
+  "mm_hidden_size": 1152,
+  "mm_projector_lr": null,
+  "mm_projector_type": "mlp2x_gelu",
+  "mm_vision_tower": "google/siglip2-so400m-patch16-naflex",
+  "model_type": "phi4-siglip",
+  "num_attention_heads": 40,
+  "num_hidden_layers": 40,
+  "num_key_value_heads": 10,
+  "original_max_position_embeddings": 32768,
+  "pad_token_id": 100349,
+  "partial_rotary_factor": 1.0,
+  "resid_pdrop": 0.0,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "rope_theta": 500000,
+  "sliding_window": null,
+  "tie_word_embeddings": false,
+  "tokenizer_model_max_length": 16384,
+  "tokenizer_padding_side": "right",
+  "transformers_version": "4.56.1",
+  "tune_mm_mlp_adapter": false,
+  "unfreeze_vision_tower": true,
+  "use_cache": true,
+  "use_mm_proj": true,
+  "use_s2": false,
+  "vocab_size": 100352,
+  "vision_config": {
+    "hidden_size": 1152,
+    "intermediate_size": 4304,
+    "model_type": "siglip2_vision_model",
+    "num_attention_heads": 16,
+    "num_hidden_layers": 27
+  }
+}

generation_config.json

@@ -0,0 +1,12 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 100257,
+  "do_sample": true,
+  "eos_token_id": [
+    100265
+  ],
+  "pad_token_id": 100349,
+  "temperature": 0.8,
+  "top_p": 0.95,
+  "transformers_version": "4.56.1"
+}

model-00001-of-00007.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:df55fbc1ae0c0bc05382b542ecbfcc790b64b5a3c8ebde66823a1815fd24c97a
+size 4933656472

model-00002-of-00007.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:16628c381bfd03d814535ef49d2ae75c083cc07ae679e409081f5fd247ef8525
+size 4954690712

model-00003-of-00007.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1acfe85da02a69ed35e2ab76e2da29ae9a0c3b040dacc0f63b8497a6ccd4a29a
+size 4902241352

model-00004-of-00007.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a0cbd06d44679f61afab76f66cae70ca831e4aaeee2a29e36b4db4f5c2df3235
+size 4771169120

model-00005-of-00007.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fd41f7de7b9a60e1950c56247c8c0d7135c7ab2568507a2db3e6fb09054aded4
+size 4771169120

model-00006-of-00007.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0942ab288370b0d9bc7f861c4475285ac72790de97bea6bd9b0e93329c85831f
+size 4878604168

model-00007-of-00007.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:10e987e926e6bbf6f34d54ba8c0f4033a827626398bf52419d139afbef07cea6
+size 1027604608

model.safetensors.index.json

@@ -0,0 +1,703 @@
+{
+  "metadata": {
+    "total_parameters": 15119518144,
+    "total_size": 30239036288
+  },
+  "weight_map": {
+    "lm_head.weight": "model-00007-of-00007.safetensors",
+    "model.embed_tokens.weight": "model-00001-of-00007.safetensors",
+    "model.layers.0.input_layernorm.weight": "model-00001-of-00007.safetensors",
+    "model.layers.0.mlp.down_proj.weight": "model-00001-of-00007.safetensors",
+    "model.layers.0.mlp.gate_up_proj.weight": "model-00001-of-00007.safetensors",
+    "model.layers.0.post_attention_layernorm.weight": "model-00001-of-00007.safetensors",
+    "model.layers.0.self_attn.o_proj.weight": "model-00001-of-00007.safetensors",
+    "model.layers.0.self_attn.qkv_proj.weight": "model-00001-of-00007.safetensors",
+    "model.layers.1.input_layernorm.weight": "model-00001-of-00007.safetensors",
+    "model.layers.1.mlp.down_proj.weight": "model-00001-of-00007.safetensors",
+    "model.layers.1.mlp.gate_up_proj.weight": "model-00001-of-00007.safetensors",
+    "model.layers.1.post_attention_layernorm.weight": "model-00001-of-00007.safetensors",
+    "model.layers.1.self_attn.o_proj.weight": "model-00001-of-00007.safetensors",
+    "model.layers.1.self_attn.qkv_proj.weight": "model-00001-of-00007.safetensors",
+    "model.layers.10.input_layernorm.weight": "model-00002-of-00007.safetensors",
+    "model.layers.10.mlp.down_proj.weight": "model-00002-of-00007.safetensors",
+    "model.layers.10.mlp.gate_up_proj.weight": "model-00002-of-00007.safetensors",
+    "model.layers.10.post_attention_layernorm.weight": "model-00002-of-00007.safetensors",
+    "model.layers.10.self_attn.o_proj.weight": "model-00002-of-00007.safetensors",
+    "model.layers.10.self_attn.qkv_proj.weight": "model-00002-of-00007.safetensors",
+    "model.layers.11.input_layernorm.weight": "model-00002-of-00007.safetensors",
+    "model.layers.11.mlp.down_proj.weight": "model-00002-of-00007.safetensors",
+    "model.layers.11.mlp.gate_up_proj.weight": "model-00002-of-00007.safetensors",
+    "model.layers.11.post_attention_layernorm.weight": "model-00002-of-00007.safetensors",
+    "model.layers.11.self_attn.o_proj.weight": "model-00002-of-00007.safetensors",
+    "model.layers.11.self_attn.qkv_proj.weight": "model-00002-of-00007.safetensors",
+    "model.layers.12.input_layernorm.weight": "model-00002-of-00007.safetensors",
+    "model.layers.12.mlp.down_proj.weight": "model-00002-of-00007.safetensors",
+    "model.layers.12.mlp.gate_up_proj.weight": "model-00002-of-00007.safetensors",
+    "model.layers.12.post_attention_layernorm.weight": "model-00002-of-00007.safetensors",
+    "model.layers.12.self_attn.o_proj.weight": "model-00002-of-00007.safetensors",
+    "model.layers.12.self_attn.qkv_proj.weight": "model-00002-of-00007.safetensors",
+    "model.layers.13.input_layernorm.weight": "model-00003-of-00007.safetensors",
+    "model.layers.13.mlp.down_proj.weight": "model-00003-of-00007.safetensors",
+    "model.layers.13.mlp.gate_up_proj.weight": "model-00003-of-00007.safetensors",
+    "model.layers.13.post_attention_layernorm.weight": "model-00003-of-00007.safetensors",
+    "model.layers.13.self_attn.o_proj.weight": "model-00003-of-00007.safetensors",
+    "model.layers.13.self_attn.qkv_proj.weight": "model-00003-of-00007.safetensors",
+    "model.layers.14.input_layernorm.weight": "model-00003-of-00007.safetensors",
+    "model.layers.14.mlp.down_proj.weight": "model-00003-of-00007.safetensors",
+    "model.layers.14.mlp.gate_up_proj.weight": "model-00003-of-00007.safetensors",
+    "model.layers.14.post_attention_layernorm.weight": "model-00003-of-00007.safetensors",
+    "model.layers.14.self_attn.o_proj.weight": "model-00003-of-00007.safetensors",
+    "model.layers.14.self_attn.qkv_proj.weight": "model-00003-of-00007.safetensors",
+    "model.layers.15.input_layernorm.weight": "model-00003-of-00007.safetensors",
+    "model.layers.15.mlp.down_proj.weight": "model-00003-of-00007.safetensors",
+    "model.layers.15.mlp.gate_up_proj.weight": "model-00003-of-00007.safetensors",
+    "model.layers.15.post_attention_layernorm.weight": "model-00003-of-00007.safetensors",
+    "model.layers.15.self_attn.o_proj.weight": "model-00003-of-00007.safetensors",
+    "model.layers.15.self_attn.qkv_proj.weight": "model-00003-of-00007.safetensors",
+    "model.layers.16.input_layernorm.weight": "model-00003-of-00007.safetensors",
+    "model.layers.16.mlp.down_proj.weight": "model-00003-of-00007.safetensors",
+    "model.layers.16.mlp.gate_up_proj.weight": "model-00003-of-00007.safetensors",
+    "model.layers.16.post_attention_layernorm.weight": "model-00003-of-00007.safetensors",
+    "model.layers.16.self_attn.o_proj.weight": "model-00003-of-00007.safetensors",
+    "model.layers.16.self_attn.qkv_proj.weight": "model-00003-of-00007.safetensors",
+    "model.layers.17.input_layernorm.weight": "model-00003-of-00007.safetensors",
+    "model.layers.17.mlp.down_proj.weight": "model-00003-of-00007.safetensors",
+    "model.layers.17.mlp.gate_up_proj.weight": "model-00003-of-00007.safetensors",
+    "model.layers.17.post_attention_layernorm.weight": "model-00003-of-00007.safetensors",
+    "model.layers.17.self_attn.o_proj.weight": "model-00003-of-00007.safetensors",
+    "model.layers.17.self_attn.qkv_proj.weight": "model-00003-of-00007.safetensors",
+    "model.layers.18.input_layernorm.weight": "model-00003-of-00007.safetensors",
+    "model.layers.18.mlp.down_proj.weight": "model-00003-of-00007.safetensors",
+    "model.layers.18.mlp.gate_up_proj.weight": "model-00003-of-00007.safetensors",
+    "model.layers.18.post_attention_layernorm.weight": "model-00003-of-00007.safetensors",
+    "model.layers.18.self_attn.o_proj.weight": "model-00003-of-00007.safetensors",
+    "model.layers.18.self_attn.qkv_proj.weight": "model-00003-of-00007.safetensors",
+    "model.layers.19.input_layernorm.weight": "model-00003-of-00007.safetensors",
+    "model.layers.19.mlp.down_proj.weight": "model-00003-of-00007.safetensors",
+    "model.layers.19.mlp.gate_up_proj.weight": "model-00003-of-00007.safetensors",
+    "model.layers.19.post_attention_layernorm.weight": "model-00003-of-00007.safetensors",
+    "model.layers.19.self_attn.o_proj.weight": "model-00003-of-00007.safetensors",
+    "model.layers.19.self_attn.qkv_proj.weight": "model-00003-of-00007.safetensors",
+    "model.layers.2.input_layernorm.weight": "model-00001-of-00007.safetensors",
+    "model.layers.2.mlp.down_proj.weight": "model-00001-of-00007.safetensors",
+    "model.layers.2.mlp.gate_up_proj.weight": "model-00001-of-00007.safetensors",
+    "model.layers.2.post_attention_layernorm.weight": "model-00001-of-00007.safetensors",
+    "model.layers.2.self_attn.o_proj.weight": "model-00001-of-00007.safetensors",
+    "model.layers.2.self_attn.qkv_proj.weight": "model-00001-of-00007.safetensors",
+    "model.layers.20.input_layernorm.weight": "model-00004-of-00007.safetensors",
+    "model.layers.20.mlp.down_proj.weight": "model-00004-of-00007.safetensors",
+    "model.layers.20.mlp.gate_up_proj.weight": "model-00004-of-00007.safetensors",
+    "model.layers.20.post_attention_layernorm.weight": "model-00004-of-00007.safetensors",
+    "model.layers.20.self_attn.o_proj.weight": "model-00003-of-00007.safetensors",
+    "model.layers.20.self_attn.qkv_proj.weight": "model-00003-of-00007.safetensors",
+    "model.layers.21.input_layernorm.weight": "model-00004-of-00007.safetensors",
+    "model.layers.21.mlp.down_proj.weight": "model-00004-of-00007.safetensors",
+    "model.layers.21.mlp.gate_up_proj.weight": "model-00004-of-00007.safetensors",
+    "model.layers.21.post_attention_layernorm.weight": "model-00004-of-00007.safetensors",
+    "model.layers.21.self_attn.o_proj.weight": "model-00004-of-00007.safetensors",
+    "model.layers.21.self_attn.qkv_proj.weight": "model-00004-of-00007.safetensors",
+    "model.layers.22.input_layernorm.weight": "model-00004-of-00007.safetensors",
+    "model.layers.22.mlp.down_proj.weight": "model-00004-of-00007.safetensors",
+    "model.layers.22.mlp.gate_up_proj.weight": "model-00004-of-00007.safetensors",
+    "model.layers.22.post_attention_layernorm.weight": "model-00004-of-00007.safetensors",
+    "model.layers.22.self_attn.o_proj.weight": "model-00004-of-00007.safetensors",
+    "model.layers.22.self_attn.qkv_proj.weight": "model-00004-of-00007.safetensors",
+    "model.layers.23.input_layernorm.weight": "model-00004-of-00007.safetensors",
+    "model.layers.23.mlp.down_proj.weight": "model-00004-of-00007.safetensors",
+    "model.layers.23.mlp.gate_up_proj.weight": "model-00004-of-00007.safetensors",
+    "model.layers.23.post_attention_layernorm.weight": "model-00004-of-00007.safetensors",
+    "model.layers.23.self_attn.o_proj.weight": "model-00004-of-00007.safetensors",
+    "model.layers.23.self_attn.qkv_proj.weight": "model-00004-of-00007.safetensors",
+    "model.layers.24.input_layernorm.weight": "model-00004-of-00007.safetensors",
+    "model.layers.24.mlp.down_proj.weight": "model-00004-of-00007.safetensors",
+    "model.layers.24.mlp.gate_up_proj.weight": "model-00004-of-00007.safetensors",
+    "model.layers.24.post_attention_layernorm.weight": "model-00004-of-00007.safetensors",
+    "model.layers.24.self_attn.o_proj.weight": "model-00004-of-00007.safetensors",
+    "model.layers.24.self_attn.qkv_proj.weight": "model-00004-of-00007.safetensors",
+    "model.layers.25.input_layernorm.weight": "model-00004-of-00007.safetensors",
+    "model.layers.25.mlp.down_proj.weight": "model-00004-of-00007.safetensors",
+    "model.layers.25.mlp.gate_up_proj.weight": "model-00004-of-00007.safetensors",
+    "model.layers.25.post_attention_layernorm.weight": "model-00004-of-00007.safetensors",
+    "model.layers.25.self_attn.o_proj.weight": "model-00004-of-00007.safetensors",
+    "model.layers.25.self_attn.qkv_proj.weight": "model-00004-of-00007.safetensors",
+    "model.layers.26.input_layernorm.weight": "model-00004-of-00007.safetensors",
+    "model.layers.26.mlp.down_proj.weight": "model-00004-of-00007.safetensors",
+    "model.layers.26.mlp.gate_up_proj.weight": "model-00004-of-00007.safetensors",
+    "model.layers.26.post_attention_layernorm.weight": "model-00004-of-00007.safetensors",
+    "model.layers.26.self_attn.o_proj.weight": "model-00004-of-00007.safetensors",
+    "model.layers.26.self_attn.qkv_proj.weight": "model-00004-of-00007.safetensors",
+    "model.layers.27.input_layernorm.weight": "model-00005-of-00007.safetensors",
+    "model.layers.27.mlp.down_proj.weight": "model-00005-of-00007.safetensors",
+    "model.layers.27.mlp.gate_up_proj.weight": "model-00005-of-00007.safetensors",
+    "model.layers.27.post_attention_layernorm.weight": "model-00005-of-00007.safetensors",
+    "model.layers.27.self_attn.o_proj.weight": "model-00004-of-00007.safetensors",
+    "model.layers.27.self_attn.qkv_proj.weight": "model-00004-of-00007.safetensors",
+    "model.layers.28.input_layernorm.weight": "model-00005-of-00007.safetensors",
+    "model.layers.28.mlp.down_proj.weight": "model-00005-of-00007.safetensors",
+    "model.layers.28.mlp.gate_up_proj.weight": "model-00005-of-00007.safetensors",
+    "model.layers.28.post_attention_layernorm.weight": "model-00005-of-00007.safetensors",
+    "model.layers.28.self_attn.o_proj.weight": "model-00005-of-00007.safetensors",
+    "model.layers.28.self_attn.qkv_proj.weight": "model-00005-of-00007.safetensors",
+    "model.layers.29.input_layernorm.weight": "model-00005-of-00007.safetensors",
+    "model.layers.29.mlp.down_proj.weight": "model-00005-of-00007.safetensors",
+    "model.layers.29.mlp.gate_up_proj.weight": "model-00005-of-00007.safetensors",
+    "model.layers.29.post_attention_layernorm.weight": "model-00005-of-00007.safetensors",
+    "model.layers.29.self_attn.o_proj.weight": "model-00005-of-00007.safetensors",
+    "model.layers.29.self_attn.qkv_proj.weight": "model-00005-of-00007.safetensors",
+    "model.layers.3.input_layernorm.weight": "model-00001-of-00007.safetensors",
+    "model.layers.3.mlp.down_proj.weight": "model-00001-of-00007.safetensors",
+    "model.layers.3.mlp.gate_up_proj.weight": "model-00001-of-00007.safetensors",
+    "model.layers.3.post_attention_layernorm.weight": "model-00001-of-00007.safetensors",
+    "model.layers.3.self_attn.o_proj.weight": "model-00001-of-00007.safetensors",
+    "model.layers.3.self_attn.qkv_proj.weight": "model-00001-of-00007.safetensors",
+    "model.layers.30.input_layernorm.weight": "model-00005-of-00007.safetensors",
+    "model.layers.30.mlp.down_proj.weight": "model-00005-of-00007.safetensors",
+    "model.layers.30.mlp.gate_up_proj.weight": "model-00005-of-00007.safetensors",
+    "model.layers.30.post_attention_layernorm.weight": "model-00005-of-00007.safetensors",
+    "model.layers.30.self_attn.o_proj.weight": "model-00005-of-00007.safetensors",
+    "model.layers.30.self_attn.qkv_proj.weight": "model-00005-of-00007.safetensors",
+    "model.layers.31.input_layernorm.weight": "model-00005-of-00007.safetensors",
+    "model.layers.31.mlp.down_proj.weight": "model-00005-of-00007.safetensors",
+    "model.layers.31.mlp.gate_up_proj.weight": "model-00005-of-00007.safetensors",
+    "model.layers.31.post_attention_layernorm.weight": "model-00005-of-00007.safetensors",
+    "model.layers.31.self_attn.o_proj.weight": "model-00005-of-00007.safetensors",
+    "model.layers.31.self_attn.qkv_proj.weight": "model-00005-of-00007.safetensors",
+    "model.layers.32.input_layernorm.weight": "model-00005-of-00007.safetensors",
+    "model.layers.32.mlp.down_proj.weight": "model-00005-of-00007.safetensors",
+    "model.layers.32.mlp.gate_up_proj.weight": "model-00005-of-00007.safetensors",
+    "model.layers.32.post_attention_layernorm.weight": "model-00005-of-00007.safetensors",
+    "model.layers.32.self_attn.o_proj.weight": "model-00005-of-00007.safetensors",
+    "model.layers.32.self_attn.qkv_proj.weight": "model-00005-of-00007.safetensors",
+    "model.layers.33.input_layernorm.weight": "model-00005-of-00007.safetensors",
+    "model.layers.33.mlp.down_proj.weight": "model-00005-of-00007.safetensors",
+    "model.layers.33.mlp.gate_up_proj.weight": "model-00005-of-00007.safetensors",
+    "model.layers.33.post_attention_layernorm.weight": "model-00005-of-00007.safetensors",
+    "model.layers.33.self_attn.o_proj.weight": "model-00005-of-00007.safetensors",
+    "model.layers.33.self_attn.qkv_proj.weight": "model-00005-of-00007.safetensors",
+    "model.layers.34.input_layernorm.weight": "model-00006-of-00007.safetensors",
+    "model.layers.34.mlp.down_proj.weight": "model-00006-of-00007.safetensors",
+    "model.layers.34.mlp.gate_up_proj.weight": "model-00006-of-00007.safetensors",
+    "model.layers.34.post_attention_layernorm.weight": "model-00006-of-00007.safetensors",
+    "model.layers.34.self_attn.o_proj.weight": "model-00005-of-00007.safetensors",
+    "model.layers.34.self_attn.qkv_proj.weight": "model-00005-of-00007.safetensors",
+    "model.layers.35.input_layernorm.weight": "model-00006-of-00007.safetensors",
+    "model.layers.35.mlp.down_proj.weight": "model-00006-of-00007.safetensors",
+    "model.layers.35.mlp.gate_up_proj.weight": "model-00006-of-00007.safetensors",
+    "model.layers.35.post_attention_layernorm.weight": "model-00006-of-00007.safetensors",
+    "model.layers.35.self_attn.o_proj.weight": "model-00006-of-00007.safetensors",
+    "model.layers.35.self_attn.qkv_proj.weight": "model-00006-of-00007.safetensors",
+    "model.layers.36.input_layernorm.weight": "model-00006-of-00007.safetensors",
+    "model.layers.36.mlp.down_proj.weight": "model-00006-of-00007.safetensors",
+    "model.layers.36.mlp.gate_up_proj.weight": "model-00006-of-00007.safetensors",
+    "model.layers.36.post_attention_layernorm.weight": "model-00006-of-00007.safetensors",
+    "model.layers.36.self_attn.o_proj.weight": "model-00006-of-00007.safetensors",
+    "model.layers.36.self_attn.qkv_proj.weight": "model-00006-of-00007.safetensors",
+    "model.layers.37.input_layernorm.weight": "model-00006-of-00007.safetensors",
+    "model.layers.37.mlp.down_proj.weight": "model-00006-of-00007.safetensors",
+    "model.layers.37.mlp.gate_up_proj.weight": "model-00006-of-00007.safetensors",
+    "model.layers.37.post_attention_layernorm.weight": "model-00006-of-00007.safetensors",
+    "model.layers.37.self_attn.o_proj.weight": "model-00006-of-00007.safetensors",
+    "model.layers.37.self_attn.qkv_proj.weight": "model-00006-of-00007.safetensors",
+    "model.layers.38.input_layernorm.weight": "model-00006-of-00007.safetensors",
+    "model.layers.38.mlp.down_proj.weight": "model-00006-of-00007.safetensors",
+    "model.layers.38.mlp.gate_up_proj.weight": "model-00006-of-00007.safetensors",
+    "model.layers.38.post_attention_layernorm.weight": "model-00006-of-00007.safetensors",
+    "model.layers.38.self_attn.o_proj.weight": "model-00006-of-00007.safetensors",
+    "model.layers.38.self_attn.qkv_proj.weight": "model-00006-of-00007.safetensors",
+    "model.layers.39.input_layernorm.weight": "model-00006-of-00007.safetensors",
+    "model.layers.39.mlp.down_proj.weight": "model-00006-of-00007.safetensors",
+    "model.layers.39.mlp.gate_up_proj.weight": "model-00006-of-00007.safetensors",
+    "model.layers.39.post_attention_layernorm.weight": "model-00006-of-00007.safetensors",
+    "model.layers.39.self_attn.o_proj.weight": "model-00006-of-00007.safetensors",
+    "model.layers.39.self_attn.qkv_proj.weight": "model-00006-of-00007.safetensors",
+    "model.layers.4.input_layernorm.weight": "model-00001-of-00007.safetensors",
+    "model.layers.4.mlp.down_proj.weight": "model-00001-of-00007.safetensors",
+    "model.layers.4.mlp.gate_up_proj.weight": "model-00001-of-00007.safetensors",
+    "model.layers.4.post_attention_layernorm.weight": "model-00001-of-00007.safetensors",
+    "model.layers.4.self_attn.o_proj.weight": "model-00001-of-00007.safetensors",
+    "model.layers.4.self_attn.qkv_proj.weight": "model-00001-of-00007.safetensors",
+    "model.layers.5.input_layernorm.weight": "model-00002-of-00007.safetensors",
+    "model.layers.5.mlp.down_proj.weight": "model-00002-of-00007.safetensors",
+    "model.layers.5.mlp.gate_up_proj.weight": "model-00001-of-00007.safetensors",
+    "model.layers.5.post_attention_layernorm.weight": "model-00002-of-00007.safetensors",
+    "model.layers.5.self_attn.o_proj.weight": "model-00001-of-00007.safetensors",
+    "model.layers.5.self_attn.qkv_proj.weight": "model-00001-of-00007.safetensors",
+    "model.layers.6.input_layernorm.weight": "model-00002-of-00007.safetensors",
+    "model.layers.6.mlp.down_proj.weight": "model-00002-of-00007.safetensors",
+    "model.layers.6.mlp.gate_up_proj.weight": "model-00002-of-00007.safetensors",
+    "model.layers.6.post_attention_layernorm.weight": "model-00002-of-00007.safetensors",
+    "model.layers.6.self_attn.o_proj.weight": "model-00002-of-00007.safetensors",
+    "model.layers.6.self_attn.qkv_proj.weight": "model-00002-of-00007.safetensors",
+    "model.layers.7.input_layernorm.weight": "model-00002-of-00007.safetensors",
+    "model.layers.7.mlp.down_proj.weight": "model-00002-of-00007.safetensors",
+    "model.layers.7.mlp.gate_up_proj.weight": "model-00002-of-00007.safetensors",
+    "model.layers.7.post_attention_layernorm.weight": "model-00002-of-00007.safetensors",
+    "model.layers.7.self_attn.o_proj.weight": "model-00002-of-00007.safetensors",
+    "model.layers.7.self_attn.qkv_proj.weight": "model-00002-of-00007.safetensors",
+    "model.layers.8.input_layernorm.weight": "model-00002-of-00007.safetensors",
+    "model.layers.8.mlp.down_proj.weight": "model-00002-of-00007.safetensors",
+    "model.layers.8.mlp.gate_up_proj.weight": "model-00002-of-00007.safetensors",
+    "model.layers.8.post_attention_layernorm.weight": "model-00002-of-00007.safetensors",
+    "model.layers.8.self_attn.o_proj.weight": "model-00002-of-00007.safetensors",
+    "model.layers.8.self_attn.qkv_proj.weight": "model-00002-of-00007.safetensors",
+    "model.layers.9.input_layernorm.weight": "model-00002-of-00007.safetensors",
+    "model.layers.9.mlp.down_proj.weight": "model-00002-of-00007.safetensors",
+    "model.layers.9.mlp.gate_up_proj.weight": "model-00002-of-00007.safetensors",
+    "model.layers.9.post_attention_layernorm.weight": "model-00002-of-00007.safetensors",
+    "model.layers.9.self_attn.o_proj.weight": "model-00002-of-00007.safetensors",
+    "model.layers.9.self_attn.qkv_proj.weight": "model-00002-of-00007.safetensors",
+    "model.mm_projector.0.bias": "model-00006-of-00007.safetensors",
+    "model.mm_projector.0.weight": "model-00006-of-00007.safetensors",
+    "model.mm_projector.2.bias": "model-00006-of-00007.safetensors",
+    "model.mm_projector.2.weight": "model-00006-of-00007.safetensors",
+    "model.norm.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.embeddings.patch_embedding.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.embeddings.patch_embedding.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.embeddings.position_embedding.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.0.layer_norm1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.0.layer_norm1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.0.layer_norm2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.0.layer_norm2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.0.mlp.fc1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.0.mlp.fc1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.0.mlp.fc2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.0.mlp.fc2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.0.self_attn.k_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.0.self_attn.k_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.0.self_attn.out_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.0.self_attn.out_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.0.self_attn.q_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.0.self_attn.q_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.0.self_attn.v_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.0.self_attn.v_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.1.layer_norm1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.1.layer_norm1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.1.layer_norm2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.1.layer_norm2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.1.mlp.fc1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.1.mlp.fc1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.1.mlp.fc2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.1.mlp.fc2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.1.self_attn.k_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.1.self_attn.k_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.1.self_attn.out_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.1.self_attn.out_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.1.self_attn.q_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.1.self_attn.q_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.1.self_attn.v_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.1.self_attn.v_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.10.layer_norm1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.10.layer_norm1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.10.layer_norm2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.10.layer_norm2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.10.mlp.fc1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.10.mlp.fc1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.10.mlp.fc2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.10.mlp.fc2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.10.self_attn.k_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.10.self_attn.k_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.10.self_attn.out_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.10.self_attn.out_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.10.self_attn.q_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.10.self_attn.q_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.10.self_attn.v_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.10.self_attn.v_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.11.layer_norm1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.11.layer_norm1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.11.layer_norm2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.11.layer_norm2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.11.mlp.fc1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.11.mlp.fc1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.11.mlp.fc2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.11.mlp.fc2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.11.self_attn.k_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.11.self_attn.k_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.11.self_attn.out_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.11.self_attn.out_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.11.self_attn.q_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.11.self_attn.q_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.11.self_attn.v_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.11.self_attn.v_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.12.layer_norm1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.12.layer_norm1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.12.layer_norm2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.12.layer_norm2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.12.mlp.fc1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.12.mlp.fc1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.12.mlp.fc2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.12.mlp.fc2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.12.self_attn.k_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.12.self_attn.k_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.12.self_attn.out_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.12.self_attn.out_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.12.self_attn.q_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.12.self_attn.q_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.12.self_attn.v_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.12.self_attn.v_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.13.layer_norm1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.13.layer_norm1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.13.layer_norm2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.13.layer_norm2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.13.mlp.fc1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.13.mlp.fc1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.13.mlp.fc2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.13.mlp.fc2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.13.self_attn.k_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.13.self_attn.k_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.13.self_attn.out_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.13.self_attn.out_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.13.self_attn.q_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.13.self_attn.q_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.13.self_attn.v_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.13.self_attn.v_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.14.layer_norm1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.14.layer_norm1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.14.layer_norm2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.14.layer_norm2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.14.mlp.fc1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.14.mlp.fc1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.14.mlp.fc2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.14.mlp.fc2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.14.self_attn.k_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.14.self_attn.k_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.14.self_attn.out_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.14.self_attn.out_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.14.self_attn.q_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.14.self_attn.q_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.14.self_attn.v_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.14.self_attn.v_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.15.layer_norm1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.15.layer_norm1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.15.layer_norm2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.15.layer_norm2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.15.mlp.fc1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.15.mlp.fc1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.15.mlp.fc2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.15.mlp.fc2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.15.self_attn.k_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.15.self_attn.k_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.15.self_attn.out_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.15.self_attn.out_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.15.self_attn.q_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.15.self_attn.q_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.15.self_attn.v_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.15.self_attn.v_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.16.layer_norm1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.16.layer_norm1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.16.layer_norm2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.16.layer_norm2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.16.mlp.fc1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.16.mlp.fc1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.16.mlp.fc2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.16.mlp.fc2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.16.self_attn.k_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.16.self_attn.k_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.16.self_attn.out_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.16.self_attn.out_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.16.self_attn.q_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.16.self_attn.q_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.16.self_attn.v_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.16.self_attn.v_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.17.layer_norm1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.17.layer_norm1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.17.layer_norm2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.17.layer_norm2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.17.mlp.fc1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.17.mlp.fc1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.17.mlp.fc2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.17.mlp.fc2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.17.self_attn.k_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.17.self_attn.k_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.17.self_attn.out_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.17.self_attn.out_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.17.self_attn.q_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.17.self_attn.q_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.17.self_attn.v_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.17.self_attn.v_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.18.layer_norm1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.18.layer_norm1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.18.layer_norm2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.18.layer_norm2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.18.mlp.fc1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.18.mlp.fc1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.18.mlp.fc2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.18.mlp.fc2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.18.self_attn.k_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.18.self_attn.k_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.18.self_attn.out_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.18.self_attn.out_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.18.self_attn.q_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.18.self_attn.q_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.18.self_attn.v_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.18.self_attn.v_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.19.layer_norm1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.19.layer_norm1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.19.layer_norm2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.19.layer_norm2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.19.mlp.fc1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.19.mlp.fc1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.19.mlp.fc2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.19.mlp.fc2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.19.self_attn.k_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.19.self_attn.k_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.19.self_attn.out_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.19.self_attn.out_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.19.self_attn.q_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.19.self_attn.q_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.19.self_attn.v_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.19.self_attn.v_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.2.layer_norm1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.2.layer_norm1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.2.layer_norm2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.2.layer_norm2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.2.mlp.fc1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.2.mlp.fc1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.2.mlp.fc2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.2.mlp.fc2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.2.self_attn.k_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.2.self_attn.k_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.2.self_attn.out_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.2.self_attn.out_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.2.self_attn.q_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.2.self_attn.q_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.2.self_attn.v_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.2.self_attn.v_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.20.layer_norm1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.20.layer_norm1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.20.layer_norm2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.20.layer_norm2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.20.mlp.fc1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.20.mlp.fc1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.20.mlp.fc2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.20.mlp.fc2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.20.self_attn.k_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.20.self_attn.k_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.20.self_attn.out_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.20.self_attn.out_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.20.self_attn.q_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.20.self_attn.q_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.20.self_attn.v_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.20.self_attn.v_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.21.layer_norm1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.21.layer_norm1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.21.layer_norm2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.21.layer_norm2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.21.mlp.fc1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.21.mlp.fc1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.21.mlp.fc2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.21.mlp.fc2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.21.self_attn.k_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.21.self_attn.k_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.21.self_attn.out_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.21.self_attn.out_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.21.self_attn.q_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.21.self_attn.q_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.21.self_attn.v_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.21.self_attn.v_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.22.layer_norm1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.22.layer_norm1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.22.layer_norm2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.22.layer_norm2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.22.mlp.fc1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.22.mlp.fc1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.22.mlp.fc2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.22.mlp.fc2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.22.self_attn.k_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.22.self_attn.k_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.22.self_attn.out_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.22.self_attn.out_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.22.self_attn.q_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.22.self_attn.q_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.22.self_attn.v_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.22.self_attn.v_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.23.layer_norm1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.23.layer_norm1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.23.layer_norm2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.23.layer_norm2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.23.mlp.fc1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.23.mlp.fc1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.23.mlp.fc2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.23.mlp.fc2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.23.self_attn.k_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.23.self_attn.k_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.23.self_attn.out_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.23.self_attn.out_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.23.self_attn.q_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.23.self_attn.q_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.23.self_attn.v_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.23.self_attn.v_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.24.layer_norm1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.24.layer_norm1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.24.layer_norm2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.24.layer_norm2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.24.mlp.fc1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.24.mlp.fc1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.24.mlp.fc2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.24.mlp.fc2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.24.self_attn.k_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.24.self_attn.k_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.24.self_attn.out_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.24.self_attn.out_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.24.self_attn.q_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.24.self_attn.q_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.24.self_attn.v_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.24.self_attn.v_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.25.layer_norm1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.25.layer_norm1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.25.layer_norm2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.25.layer_norm2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.25.mlp.fc1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.25.mlp.fc1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.25.mlp.fc2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.25.mlp.fc2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.25.self_attn.k_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.25.self_attn.k_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.25.self_attn.out_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.25.self_attn.out_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.25.self_attn.q_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.25.self_attn.q_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.25.self_attn.v_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.25.self_attn.v_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.26.layer_norm1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.26.layer_norm1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.26.layer_norm2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.26.layer_norm2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.26.mlp.fc1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.26.mlp.fc1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.26.mlp.fc2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.26.mlp.fc2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.26.self_attn.k_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.26.self_attn.k_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.26.self_attn.out_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.26.self_attn.out_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.26.self_attn.q_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.26.self_attn.q_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.26.self_attn.v_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.26.self_attn.v_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.3.layer_norm1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.3.layer_norm1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.3.layer_norm2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.3.layer_norm2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.3.mlp.fc1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.3.mlp.fc1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.3.mlp.fc2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.3.mlp.fc2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.3.self_attn.k_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.3.self_attn.k_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.3.self_attn.out_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.3.self_attn.out_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.3.self_attn.q_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.3.self_attn.q_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.3.self_attn.v_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.3.self_attn.v_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.4.layer_norm1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.4.layer_norm1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.4.layer_norm2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.4.layer_norm2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.4.mlp.fc1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.4.mlp.fc1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.4.mlp.fc2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.4.mlp.fc2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.4.self_attn.k_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.4.self_attn.k_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.4.self_attn.out_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.4.self_attn.out_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.4.self_attn.q_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.4.self_attn.q_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.4.self_attn.v_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.4.self_attn.v_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.5.layer_norm1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.5.layer_norm1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.5.layer_norm2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.5.layer_norm2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.5.mlp.fc1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.5.mlp.fc1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.5.mlp.fc2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.5.mlp.fc2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.5.self_attn.k_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.5.self_attn.k_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.5.self_attn.out_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.5.self_attn.out_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.5.self_attn.q_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.5.self_attn.q_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.5.self_attn.v_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.5.self_attn.v_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.6.layer_norm1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.6.layer_norm1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.6.layer_norm2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.6.layer_norm2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.6.mlp.fc1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.6.mlp.fc1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.6.mlp.fc2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.6.mlp.fc2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.6.self_attn.k_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.6.self_attn.k_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.6.self_attn.out_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.6.self_attn.out_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.6.self_attn.q_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.6.self_attn.q_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.6.self_attn.v_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.6.self_attn.v_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.7.layer_norm1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.7.layer_norm1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.7.layer_norm2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.7.layer_norm2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.7.mlp.fc1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.7.mlp.fc1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.7.mlp.fc2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.7.mlp.fc2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.7.self_attn.k_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.7.self_attn.k_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.7.self_attn.out_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.7.self_attn.out_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.7.self_attn.q_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.7.self_attn.q_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.7.self_attn.v_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.7.self_attn.v_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.8.layer_norm1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.8.layer_norm1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.8.layer_norm2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.8.layer_norm2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.8.mlp.fc1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.8.mlp.fc1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.8.mlp.fc2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.8.mlp.fc2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.8.self_attn.k_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.8.self_attn.k_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.8.self_attn.out_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.8.self_attn.out_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.8.self_attn.q_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.8.self_attn.q_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.8.self_attn.v_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.8.self_attn.v_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.9.layer_norm1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.9.layer_norm1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.9.layer_norm2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.9.layer_norm2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.9.mlp.fc1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.9.mlp.fc1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.9.mlp.fc2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.9.mlp.fc2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.9.self_attn.k_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.9.self_attn.k_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.9.self_attn.out_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.9.self_attn.out_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.9.self_attn.q_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.9.self_attn.q_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.9.self_attn.v_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.encoder.layers.9.self_attn.v_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.head.attention.in_proj_bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.head.attention.in_proj_weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.head.attention.out_proj.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.head.attention.out_proj.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.head.layernorm.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.head.layernorm.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.head.mlp.fc1.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.head.mlp.fc1.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.head.mlp.fc2.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.head.mlp.fc2.weight": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.head.probe": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.post_layernorm.bias": "model-00006-of-00007.safetensors",
+    "model.vision_tower.vision_tower.vision_model.post_layernorm.weight": "model-00006-of-00007.safetensors"
+  }
+}

modeling_phi4_visionr.py

@@ -0,0 +1,1026 @@
+"""
+Minimal self-contained Phi4-Siglip model implementation.
+
+This module provides:
+- Phi4VisionR: Configuration class
+- Phi4ForCausalLMV: Main vision-language model
+- SiglipVisionTower: Vision encoder (standard SigLIP)
+- Siglip2VisionTower: Vision encoder with NaFlex (variable token count)
+- MLP Projector: Vision-to-language projection
+"""
+
+import logging
+import os
+import re
+import math
+from abc import ABC, abstractmethod
+from typing import List, Optional, Tuple, Union
+from dataclasses import dataclass
+
+import torch
+import torch.nn as nn
+from safetensors.torch import load_file
+
+logger = logging.getLogger(__name__)
+from transformers import (
+    AutoConfig,
+    AutoModelForCausalLM,
+    Phi3Config,
+    Phi3Model,
+    Phi3ForCausalLM,
+    SiglipVisionModel,
+    SiglipVisionConfig,
+    SiglipImageProcessor,
+    Siglip2VisionModel,
+    Siglip2VisionConfig,
+    BatchFeature,
+)
+from transformers.modeling_outputs import CausalLMOutputWithPast
+from transformers.processing_utils import ImagesKwargs
+import transformers.models.siglip2.image_processing_siglip2 as siglip2_ips
+
+
+# =============================================================================
+# Constants
+# =============================================================================
+
+IGNORE_INDEX = -100
+IMAGE_TOKEN_INDEX = -200
+DEFAULT_IMAGE_TOKEN = "<image>"
+
+
+# =============================================================================
+# Model Arguments (simplified dataclass for initialization)
+# =============================================================================
+
+@dataclass
+class ModelArguments:
+    """Arguments for model initialization."""
+    vision_tower: Optional[str] = None
+    vision_tower_path: Optional[str] = None
+    mm_projector_type: str = "mlp2x_gelu"
+    pretrain_mm_mlp_adapter: Optional[str] = None
+    use_s2: bool = False
+    s2_scales: str = "384,768,1152"
+    hf_cache_dir: Optional[str] = None
+    # NaFlex-specific
+    min_num_patches: int = 256
+    max_num_patches: int = 3600
+    # Embedded vision config (to avoid network calls)
+    vision_config: Optional[dict] = None
+
+
+# =============================================================================
+# Vision Projector (MLP)
+# =============================================================================
+
+def build_vision_projector(config):
+    """Build vision-to-language projector based on config."""
+    projector_type = getattr(config, 'mm_projector_type', 'mlp2x_gelu')
+
+    if projector_type == 'linear':
+        return nn.Linear(config.mm_hidden_size, config.hidden_size)
+
+    elif projector_type.startswith('mlp'):
+        mlp_gelu_match = re.match(r'^mlp(\d+)x_gelu$', projector_type)
+        if mlp_gelu_match:
+            mlp_depth = int(mlp_gelu_match.group(1))
+            modules = [nn.Linear(config.mm_hidden_size, config.hidden_size)]
+            for _ in range(1, mlp_depth):
+                modules.append(nn.GELU())
+                modules.append(nn.Linear(config.hidden_size, config.hidden_size))
+            return nn.Sequential(*modules)
+
+    elif projector_type == 'identity':
+        return nn.Identity()
+
+    raise ValueError(f'Unknown projector type: {projector_type}')
+
+
+# =============================================================================
+# Vision Encoders - SigLIP
+# =============================================================================
+
+class SiglipVisionTower(nn.Module):
+    """Standard SigLIP vision encoder with fixed token count."""
+    
+    def __init__(self, vision_tower: str, args: ModelArguments = None, delay_load: bool = False):
+        super().__init__()
+
+        self.is_loaded = False
+        self.vision_tower_name = vision_tower
+        self.vision_tower_path = None
+        self.select_layer = -2
+
+        self.hf_hub_cache_dir = None
+        self.local_files_only = False
+
+        if args and getattr(args, 'hf_cache_dir', None):
+            self.hf_hub_cache_dir = args.hf_cache_dir
+            self.local_files_only = True
+        
+        # Load or create vision config once (avoids network calls if embedded config provided)
+        vision_config_dict = getattr(args, "vision_config", None) if args else None
+        if vision_config_dict is not None:
+            self._vision_config = SiglipVisionConfig(**vision_config_dict)
+        else:
+            self._vision_config = SiglipVisionConfig.from_pretrained(
+                self.vision_tower_name,
+                local_files_only=self.local_files_only,
+                cache_dir=self.hf_hub_cache_dir,
+            )
+            
+        if not delay_load:
+            self.load_model()
+
+    def load_model(self):
+        if self.is_loaded:
+            return
+
+        # Create image processor
+        self.image_processor = SiglipImageProcessor(
+            size={"height": self._vision_config.image_size, "width": self._vision_config.image_size},
+        )
+        self.image_processor.crop_size = self.image_processor.size
+
+        vision_tower_path = self.vision_tower_path if self.vision_tower_path else self.vision_tower_name
+        self.vision_tower = SiglipVisionModel.from_pretrained(
+            vision_tower_path,
+            config=self._vision_config,
+            local_files_only=self.local_files_only,
+            cache_dir=self.hf_hub_cache_dir,
+        )
+
+        self.vision_tower.requires_grad_(False)
+        self.is_loaded = True
+
+    def feature_select(self, image_forward_outs):
+        return image_forward_outs.hidden_states[self.select_layer]
+
+    def forward(self, images):
+        if isinstance(images, list):
+            image_features = []
+            for image in images:
+                image_forward_out = self.vision_tower(
+                    image.to(device=self.device, dtype=self.dtype).unsqueeze(0),
+                    output_hidden_states=True
+                )
+                image_feature = self.feature_select(image_forward_out).to(image.dtype)
+                image_features.append(image_feature)
+        else:
+            image_forward_outs = self.vision_tower(
+                images.to(device=self.device, dtype=self.dtype),
+                output_hidden_states=True
+            )
+            image_features = self.feature_select(image_forward_outs).to(images.dtype)
+
+        return image_features
+
+    @property
+    def dummy_feature(self):
+        return torch.zeros(1, self.hidden_size, device=self.device, dtype=self.dtype)
+
+    @property
+    def dtype(self):
+        return self.vision_tower.dtype
+
+    @property
+    def device(self):
+        return self.vision_tower.device
+
+    @property
+    def config(self):
+        return self.vision_tower.config if self.is_loaded else self._vision_config
+
+    @property
+    def hidden_size(self):
+        return self.config.hidden_size
+
+    @property
+    def num_patches(self):
+        return (self.config.image_size // self.config.patch_size) ** 2
+
+
+# =============================================================================
+# Vision Encoders - SigLIP2 with NaFlex (variable token count)
+# =============================================================================
+
+class Siglip2ImageProcessorKwargsNoUpscale(ImagesKwargs, total=False):
+    patch_size: int
+    max_num_patches: int
+    min_num_patches: int
+
+
+class Siglip2ImageProcessorNoUpscale(siglip2_ips.Siglip2ImageProcessor):
+    """Custom SigLIP2 image processor that doesn't upscale small images."""
+    
+    model_input_names = ["pixel_values", "pixel_attention_mask", "spatial_shapes"]
+    valid_kwargs = Siglip2ImageProcessorKwargsNoUpscale
+
+    def __init__(
+        self,
+        do_resize: bool = True,
+        resample = siglip2_ips.PILImageResampling.BILINEAR,
+        do_rescale: bool = True,
+        rescale_factor: float = 1 / 255,
+        do_normalize: bool = True,
+        image_mean: Optional[Union[float, List[float]]] = None,
+        image_std: Optional[Union[float, List[float]]] = None,
+        do_convert_rgb: Optional[bool] = None,
+        patch_size: int = 16,
+        max_num_patches: int = 256,
+        min_num_patches: int = 1,
+        **kwargs,
+    ):
+        super().__init__(**kwargs)
+
+        image_mean = image_mean if image_mean is not None else [0.5, 0.5, 0.5]
+        image_std = image_std if image_std is not None else [0.5, 0.5, 0.5]
+
+        self.do_resize = do_resize
+        self.resample = resample
+        self.do_rescale = do_rescale
+        self.rescale_factor = rescale_factor
+        self.do_normalize = do_normalize
+        self.image_mean = image_mean
+        self.image_std = image_std
+        self.do_convert_rgb = do_convert_rgb
+        self.patch_size = patch_size
+        self.max_num_patches = max_num_patches
+        self.min_num_patches = min_num_patches
+
+    @siglip2_ips.filter_out_non_signature_kwargs()
+    def preprocess(
+        self,
+        images,
+        resample=None,
+        do_rescale: Optional[bool] = None,
+        rescale_factor: Optional[float] = None,
+        do_normalize: Optional[bool] = None,
+        image_mean: Optional[Union[float, List[float]]] = None,
+        image_std: Optional[Union[float, List[float]]] = None,
+        return_tensors=None,
+        input_data_format=None,
+        do_convert_rgb: Optional[bool] = None,
+        patch_size: Optional[int] = None,
+        max_num_patches: Optional[int] = None,
+        min_num_patches: Optional[int] = None,
+    ):
+        resample = resample if resample is not None else self.resample
+        do_rescale = do_rescale if do_rescale is not None else self.do_rescale
+        rescale_factor = rescale_factor if rescale_factor is not None else self.rescale_factor
+        do_normalize = do_normalize if do_normalize is not None else self.do_normalize
+        image_mean = image_mean if image_mean is not None else self.image_mean
+        image_std = image_std if image_std is not None else self.image_std
+        do_convert_rgb = do_convert_rgb if do_convert_rgb is not None else self.do_convert_rgb
+        patch_size = patch_size if patch_size is not None else self.patch_size
+        max_num_patches = max_num_patches if max_num_patches is not None else self.max_num_patches
+        min_num_patches = min_num_patches if min_num_patches is not None else self.min_num_patches
+
+        data_format = siglip2_ips.ChannelDimension.LAST
+
+        try:
+            images = self.fetch_images(images)
+        except TypeError:
+            pass
+        images = siglip2_ips.make_flat_list_of_images(images)
+
+        if not siglip2_ips.valid_images(images):
+            raise ValueError("Invalid image type. Must be of type PIL.Image.Image, numpy.ndarray, or torch.Tensor")
+        
+        siglip2_ips.validate_preprocess_arguments(
+            do_rescale=do_rescale,
+            rescale_factor=rescale_factor,
+            do_normalize=do_normalize,
+            image_mean=image_mean,
+            image_std=image_std,
+        )
+        
+        if do_convert_rgb:
+            images = [siglip2_ips.convert_to_rgb(image) for image in images]
+
+        images = [siglip2_ips.to_numpy_array(image) for image in images]
+
+        if input_data_format is None:
+            input_data_format = siglip2_ips.infer_channel_dimension_format(images[0])
+
+        pixel_masks = []
+        pixel_values = []
+        spatial_shapes = []
+
+        for image in images:
+            image = siglip2_ips.to_channel_dimension_format(image, data_format, input_channel_dim=input_data_format)
+
+            num_patches = max((image.shape[1] // patch_size) * (image.shape[0] // patch_size), 1)
+            
+            # Resize only if image is too large/small
+            if num_patches < min_num_patches:
+                height, width = siglip2_ips.get_image_size_for_max_num_patches(
+                    image_height=image.shape[0],
+                    image_width=image.shape[1],
+                    patch_size=patch_size,
+                    max_num_patches=min_num_patches,
+                )
+            elif num_patches > max_num_patches:
+                height, width = siglip2_ips.get_image_size_for_max_num_patches(
+                    image_height=image.shape[0],
+                    image_width=image.shape[1],
+                    patch_size=patch_size,
+                    max_num_patches=max_num_patches,
+                )
+            else:
+                height, width = siglip2_ips.get_image_size_for_max_num_patches(
+                    image_height=image.shape[0],
+                    image_width=image.shape[1],
+                    patch_size=patch_size,
+                    max_num_patches=num_patches,
+                )
+            
+            image = siglip2_ips.resize(image=image, size=(height, width), resample=resample, input_data_format=data_format)
+
+            if do_rescale:
+                image = self.rescale(image=image, scale=rescale_factor, input_data_format=data_format)
+
+            if do_normalize:
+                image = self.normalize(image=image, mean=image_mean, std=image_std, input_data_format=data_format)
+
+            patches = siglip2_ips.convert_image_to_patches(image, patch_size)
+            patches, mask = siglip2_ips.pad_along_first_dim(patches, max_num_patches)
+            num_patches_height = image.shape[0] // patch_size
+            num_patches_width = image.shape[1] // patch_size
+
+            spatial_shapes.append((num_patches_height, num_patches_width))
+            pixel_values.append(patches)
+            pixel_masks.append(mask)
+
+        return siglip2_ips.BatchFeature(
+            data={
+                "pixel_values": pixel_values,
+                "pixel_attention_mask": pixel_masks,
+                "spatial_shapes": spatial_shapes,
+            },
+            tensor_type=return_tensors,
+        )
+
+
+class Siglip2VisionTower(nn.Module):
+    """SigLIP2 vision encoder with NaFlex (variable token count per image)."""
+    
+    def __init__(self, vision_tower: str, args: ModelArguments = None, delay_load: bool = False):
+        super().__init__()
+
+        self.is_loaded = False
+        self.vision_tower_name = vision_tower
+        self.vision_tower_path = None
+        self.select_layer = -2
+
+        self.hf_hub_cache_dir = None
+        self.local_files_only = False
+
+        self.min_num_patches = getattr(args, "min_num_patches", 256) if args else 256
+        self.max_num_patches = getattr(args, "max_num_patches", 3600) if args else 3600
+
+        if args and getattr(args, 'hf_cache_dir', None):
+            self.hf_hub_cache_dir = args.hf_cache_dir
+            self.local_files_only = True
+        
+        # Load or create vision config once (avoids network calls if embedded config provided)
+        vision_config_dict = getattr(args, "vision_config", None) if args else None
+        if vision_config_dict is not None:
+            # Infer patch_size from model name if not in config
+            if 'patch_size' not in vision_config_dict:
+                if 'patch14' in self.vision_tower_name.lower():
+                    vision_config_dict['patch_size'] = 14
+                else:
+                    vision_config_dict['patch_size'] = 16  # default for patch16-naflex
+            self._vision_config = Siglip2VisionConfig(**vision_config_dict)
+        else:
+            self._vision_config = Siglip2VisionConfig.from_pretrained(
+                self.vision_tower_name,
+                local_files_only=self.local_files_only,
+                cache_dir=self.hf_hub_cache_dir,
+            )
+            
+        if not delay_load:
+            self.load_model()
+
+    def load_model(self, skip_weights: bool = False):
+        """Load the vision tower model.
+        
+        Args:
+            skip_weights: If True, only load the architecture without pretrained weights.
+                         Useful when weights will be loaded from a checkpoint later.
+        """
+        if self.is_loaded:
+            return
+
+        # Create image processor
+        self.image_processor = Siglip2ImageProcessorNoUpscale(
+            patch_size=self._vision_config.patch_size,
+            max_num_patches=self.max_num_patches,
+            min_num_patches=self.min_num_patches,
+        )
+
+        if skip_weights:
+            # Load architecture only, no pretrained weights (will load from checkpoint)
+            self.vision_tower = Siglip2VisionModel(self._vision_config)
+            logger.info("Vision tower initialized without pretrained weights (will load from checkpoint).")
+        else:
+            vision_tower_path = self.vision_tower_path if self.vision_tower_path else self.vision_tower_name
+            self.vision_tower = Siglip2VisionModel.from_pretrained(
+                vision_tower_path,
+                config=self._vision_config,
+                local_files_only=self.local_files_only,
+                cache_dir=self.hf_hub_cache_dir,
+            )
+
+        self.vision_tower.config.min_num_patches = self.min_num_patches
+        self.vision_tower.config.max_num_patches = self.max_num_patches
+
+        self.vision_tower.requires_grad_(False)
+        self.is_loaded = True
+
+    def feature_select(self, image_forward_outs):
+        return image_forward_outs.hidden_states[self.select_layer]
+
+    def forward(self, images):
+        if isinstance(images, (dict, BatchFeature)):
+            images = {
+                "pixel_values": images["pixel_values"].to(device=self.device, dtype=self.dtype),
+                "pixel_attention_mask": images["pixel_attention_mask"].to(device=self.device, dtype=self.dtype),
+                "spatial_shapes": images["spatial_shapes"].cpu().numpy(),
+            }
+            images_forward_out = self.vision_tower(**images, output_hidden_states=True)
+            image_features = self.feature_select(images_forward_out).to(self.dtype)
+            # Remove pad tokens
+            image_features = [
+                feat[images["pixel_attention_mask"][j].bool()] 
+                for j, feat in enumerate(image_features)
+            ]
+
+        elif isinstance(images, list):
+            image_features = []
+            for image in images:
+                image = {
+                    "pixel_values": image["pixel_values"].to(device=self.device, dtype=self.dtype),
+                    "pixel_attention_mask": image["pixel_attention_mask"].to(device=self.device, dtype=self.dtype),
+                    "spatial_shapes": image["spatial_shapes"].cpu().numpy(),
+                }
+                image_forward_out = self.vision_tower(**image, output_hidden_states=True)
+                image_feature = self.feature_select(image_forward_out).to(self.dtype)
+                image_feature = [
+                    feat[image["pixel_attention_mask"][j].bool()] 
+                    for j, feat in enumerate(image_feature)
+                ]
+                image_features.append(image_feature)
+        else:
+            raise ValueError(f"Unsupported image type: {type(images)}")
+
+        return image_features
+
+    @property
+    def dummy_feature(self):
+        return torch.zeros(1, self.hidden_size, device=self.device, dtype=self.dtype)
+
+    @property
+    def dtype(self):
+        return self.vision_tower.dtype
+
+    @property
+    def device(self):
+        return self.vision_tower.device
+
+    @property
+    def config(self):
+        return self.vision_tower.config if self.is_loaded else self._vision_config
+
+    @property
+    def hidden_size(self):
+        return self.config.hidden_size
+
+
+# =============================================================================
+# Vision Tower Builder
+# =============================================================================
+
+def build_vision_tower(config, delay_load: bool = False):
+    """Build the appropriate vision tower based on config."""
+    vision_tower = getattr(config, 'mm_vision_tower', getattr(config, 'vision_tower', None))
+    
+    if vision_tower is None:
+        return None
+    
+    # Create a minimal args object from config
+    args = ModelArguments(
+        vision_tower=vision_tower,
+        hf_cache_dir=getattr(config, 'hf_cache_dir', None),
+        min_num_patches=getattr(config, 'min_num_patches', 256),
+        max_num_patches=getattr(config, 'max_num_patches', 3600),
+        vision_config=getattr(config, 'vision_config', None),
+    )
+    
+    if 'siglip' in vision_tower.lower():
+        if 'naflex' in vision_tower.lower():
+            return Siglip2VisionTower(vision_tower, args=args, delay_load=delay_load)
+        else:
+            return SiglipVisionTower(vision_tower, args=args, delay_load=delay_load)
+    
+    raise ValueError(f'Unknown vision tower: {vision_tower}. Only SigLIP variants are supported.')
+
+
+# =============================================================================
+# Configuration
+# =============================================================================
+
+class Phi4VisionR(Phi3Config):
+    """Configuration for Phi4-Siglip model."""
+    model_type = "phi4-siglip"
+    
+    def __init__(
+        self,
+        mm_vision_tower: Optional[str] = None,
+        mm_projector_type: str = "mlp2x_gelu",
+        mm_hidden_size: int = 1152,
+        min_num_patches: int = 256,
+        max_num_patches: int = 3600,
+        vision_config: Optional[dict] = None,
+        **kwargs
+    ):
+        super().__init__(**kwargs)
+        self.mm_vision_tower = mm_vision_tower
+        self.mm_projector_type = mm_projector_type
+        self.mm_hidden_size = mm_hidden_size
+        self.min_num_patches = min_num_patches
+        self.max_num_patches = max_num_patches
+        self.vision_config = vision_config
+
+
+# =============================================================================
+# Base Model with Vision Integration
+# =============================================================================
+
+class Phi4VisionRModel(Phi3Model):
+    """Phi3 model with vision tower and projector."""
+    config_class = Phi4VisionR
+
+    def __init__(self, config: Phi4VisionR):
+        super().__init__(config)
+
+        if hasattr(config, "mm_vision_tower") and config.mm_vision_tower:
+            self.vision_tower = build_vision_tower(config, delay_load=not getattr(config, 'continuous_training', False))
+            if getattr(config, 'continuous_training', False):
+                config.continuous_training = False
+            self.mm_projector = build_vision_projector(config)
+
+    def get_vision_tower(self):
+        vision_tower = getattr(self, 'vision_tower', None)
+        if isinstance(vision_tower, list):
+            vision_tower = vision_tower[0]
+        return vision_tower
+
+    def initialize_vision_modules(self, model_args: ModelArguments):
+        """Initialize vision tower and projector from model arguments."""
+        vision_tower_name = model_args.vision_tower
+
+        self.config.mm_vision_tower = vision_tower_name
+
+        if self.get_vision_tower() is None:
+            vision_tower = build_vision_tower(model_args)
+            self.vision_tower = vision_tower
+        else:
+            vision_tower = self.vision_tower
+            if model_args.vision_tower_path:
+                vision_tower.vision_tower_path = model_args.vision_tower_path
+            vision_tower.load_model()
+
+        self.config.use_mm_proj = True
+        self.config.mm_projector_type = model_args.mm_projector_type
+        self.config.mm_hidden_size = vision_tower.hidden_size
+
+        if getattr(self, 'mm_projector', None) is None:
+            self.mm_projector = build_vision_projector(self.config)
+
+        # Ensure projector is trainable
+        for p in self.mm_projector.parameters():
+            p.requires_grad = True
+
+        # Load pretrained projector weights if provided
+        if model_args.pretrain_mm_mlp_adapter is not None:
+            mm_projector_weights = torch.load(model_args.pretrain_mm_mlp_adapter, map_location='cpu')
+            
+            def get_w(weights, keyword):
+                return {k.split(keyword + '.')[1]: v for k, v in weights.items() if keyword in k}
+
+            self.mm_projector.load_state_dict(get_w(mm_projector_weights, 'mm_projector'))
+
+
+# =============================================================================
+# Causal LM with Multimodal Support
+# =============================================================================
+
+class Phi4ForCausalLMV(Phi3ForCausalLM):
+    """Phi4-Siglip model for causal language modeling with vision support."""
+    config_class = Phi4VisionR
+    
+    # Tell transformers to not warn about vision tower weights - we load them separately
+    _keys_to_ignore_on_load_unexpected = [r"model\.vision_tower\.vision_tower\..*"]
+
+    def __init__(self, config: Phi4VisionR):
+        super(Phi3ForCausalLM, self).__init__(config)
+        self.model = Phi4VisionRModel(config)
+        self.vocab_size = config.vocab_size
+        self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
+        self.post_init()
+
+    def get_model(self):
+        return self.model
+
+    def get_vision_tower(self):
+        return self.get_model().get_vision_tower()
+
+    def encode_images(self, images):
+        """Encode images through vision tower and projector."""
+        image_features = self.get_model().get_vision_tower()(images)
+        
+        # Handle dynamic tokens (NaFlex)
+        if isinstance(image_features, list) and isinstance(image_features[0], list):
+            image_features = [
+                [self.get_model().mm_projector(image) for image in batch] 
+                for batch in image_features
+            ]
+        elif isinstance(image_features, list):
+            image_features = [self.get_model().mm_projector(image) for image in image_features]
+        else:
+            image_features = self.get_model().mm_projector(image_features)
+        
+        return image_features
+
+    def prepare_inputs_labels_for_multimodal(
+        self, input_ids, position_ids, attention_mask, past_key_values, labels, images
+    ):
+        """
+        Prepare inputs by replacing image tokens with actual image embeddings.
+        
+        This is the core multimodal integration logic that:
+        1. Encodes images through the vision tower
+        2. Finds IMAGE_TOKEN_INDEX positions in input_ids
+        3. Replaces those positions with image embeddings
+        4. Handles padding and attention masks
+        """
+        vision_tower = self.get_vision_tower()
+        
+        if vision_tower is None or images is None or input_ids.shape[1] == 1:
+            # Handle KV cache case during generation
+            if past_key_values is not None and vision_tower is not None and images is not None and input_ids.shape[1] == 1:
+                target_shape = past_key_values[-1][-1].shape[-2] + 1
+                attention_mask = torch.cat((
+                    attention_mask,
+                    torch.ones(
+                        (attention_mask.shape[0], target_shape - attention_mask.shape[1]),
+                        dtype=attention_mask.dtype,
+                        device=attention_mask.device
+                    )
+                ), dim=1)
+                position_ids = torch.sum(attention_mask, dim=1).unsqueeze(-1) - 1
+            return input_ids, position_ids, attention_mask, past_key_values, None, labels
+
+        # Encode images
+        if (isinstance(images, torch.Tensor) and images.ndim == 5) or \
+           (isinstance(images, list) and isinstance(images[0], torch.Tensor)):
+            images = torch.cat([image for image in images], dim=0)
+            image_features = self.encode_images(images).to(self.device)
+        elif isinstance(images, list) and isinstance(images[0], (dict, BatchFeature)):
+            # NaFlex case
+            image_features = self.encode_images(images)
+            image_features = [image.to(self.device) for batch in image_features for image in batch]
+        elif isinstance(images, (dict, BatchFeature)):
+            image_features = self.encode_images(images)
+            image_features = [image.to(self.device) for image in image_features]
+        else:
+            image_features = self.encode_images(images).to(self.device)
+
+        # Store original values
+        _labels = labels
+        _position_ids = position_ids
+        _attention_mask = attention_mask
+        
+        # Create defaults if not provided
+        if attention_mask is None:
+            attention_mask = torch.ones_like(input_ids, dtype=torch.bool)
+        else:
+            attention_mask = attention_mask.bool()
+        if position_ids is None:
+            position_ids = torch.arange(0, input_ids.shape[1], dtype=torch.long, device=input_ids.device)
+        if labels is None:
+            labels = torch.full_like(input_ids, IGNORE_INDEX)
+
+        input_ids_temp = input_ids
+
+        # Remove padding using attention_mask
+        input_ids = [cur_input_ids[cur_attention_mask] for cur_input_ids, cur_attention_mask in
+                     zip(input_ids, attention_mask)]
+        labels = [cur_labels[cur_attention_mask] for cur_labels, cur_attention_mask in zip(labels, attention_mask)]
+
+        # Replace IMAGE_TOKEN_INDEX with 0 for compatibility
+        input_ids_temp[input_ids_temp == IMAGE_TOKEN_INDEX] = 0
+
+        new_input_embeds = []
+        new_labels = []
+        cur_image_idx = 0
+        
+        for batch_idx, cur_input_ids in enumerate(input_ids):
+            num_images = (cur_input_ids == IMAGE_TOKEN_INDEX).sum()
+            
+            if num_images == 0:
+                # No image tokens - just embed text
+                cur_image_features = image_features[cur_image_idx]
+                cur_input_embeds_1 = self.get_model().embed_tokens(cur_input_ids)
+                cur_input_embeds = torch.cat([cur_input_embeds_1, cur_image_features[0:0]], dim=0)
+                new_input_embeds.append(cur_input_embeds)
+                new_labels.append(labels[batch_idx])
+                cur_image_idx += 1
+                continue
+
+            # Find image token positions
+            image_token_indices = [-1] + torch.where(cur_input_ids == IMAGE_TOKEN_INDEX)[0].tolist() + [
+                cur_input_ids.shape[0]]
+            
+            cur_input_ids_noim = []
+            cur_labels = labels[batch_idx]
+            cur_labels_noim = []
+            
+            # Split by image tokens
+            for i in range(len(image_token_indices) - 1):
+                cur_input_ids_noim.append(cur_input_ids[image_token_indices[i] + 1:image_token_indices[i + 1]])
+                cur_labels_noim.append(cur_labels[image_token_indices[i] + 1:image_token_indices[i + 1]])
+            
+            split_sizes = [x.shape[0] for x in cur_labels_noim]
+            cur_input_embeds = self.get_model().embed_tokens(torch.cat(cur_input_ids_noim))
+            cur_input_embeds_no_im = torch.split(cur_input_embeds, split_sizes, dim=0)
+            
+            cur_new_input_embeds = []
+            cur_new_labels = []
+
+            # Interleave text and image embeddings
+            for i in range(num_images + 1):
+                cur_new_input_embeds.append(cur_input_embeds_no_im[i])
+                cur_new_labels.append(cur_labels_noim[i])
+                if i < num_images:
+                    cur_image_features = image_features[cur_image_idx]
+                    cur_image_idx += 1
+                    cur_new_input_embeds.append(cur_image_features)
+                    cur_new_labels.append(
+                        torch.full(
+                            (cur_image_features.shape[0],), 
+                            IGNORE_INDEX, 
+                            device=cur_labels.device,
+                            dtype=cur_labels.dtype
+                        )
+                    )
+
+            cur_new_input_embeds = torch.cat(cur_new_input_embeds)
+            cur_new_labels = torch.cat(cur_new_labels)
+
+            new_input_embeds.append(cur_new_input_embeds)
+            new_labels.append(cur_new_labels)
+
+        # Truncate to max length
+        tokenizer_model_max_length = getattr(self.config, 'tokenizer_model_max_length', None)
+        if tokenizer_model_max_length is not None:
+            new_input_embeds = [x[:tokenizer_model_max_length] for x in new_input_embeds]
+            new_labels = [x[:tokenizer_model_max_length] for x in new_labels]
+
+        # Pad sequences to same length
+        max_len = max(x.shape[0] for x in new_input_embeds)
+        batch_size = len(new_input_embeds)
+
+        new_input_embeds_padded = []
+        new_labels_padded = torch.full(
+            (batch_size, max_len), IGNORE_INDEX, 
+            dtype=new_labels[0].dtype, device=new_labels[0].device
+        )
+        attention_mask = torch.zeros(
+            (batch_size, max_len), 
+            dtype=attention_mask.dtype, device=attention_mask.device
+        )
+        position_ids = torch.zeros(
+            (batch_size, max_len), 
+            dtype=position_ids.dtype, device=position_ids.device
+        )
+
+        for i, (cur_new_embed, cur_new_labels) in enumerate(zip(new_input_embeds, new_labels)):
+            cur_len = cur_new_embed.shape[0]
+            padding_side = getattr(self.config, 'tokenizer_padding_side', 'right')
+            
+            if padding_side == "left":
+                new_input_embeds_padded.append(torch.cat((
+                    torch.zeros(
+                        (max_len - cur_len, cur_new_embed.shape[1]), 
+                        dtype=cur_new_embed.dtype, device=cur_new_embed.device
+                    ),
+                    cur_new_embed
+                ), dim=0))
+                if cur_len > 0:
+                    new_labels_padded[i, -cur_len:] = cur_new_labels
+                    attention_mask[i, -cur_len:] = True
+                    position_ids[i, -cur_len:] = torch.arange(
+                        0, cur_len, dtype=position_ids.dtype, device=position_ids.device
+                    )
+            else:
+                new_input_embeds_padded.append(torch.cat((
+                    cur_new_embed,
+                    torch.zeros(
+                        (max_len - cur_len, cur_new_embed.shape[1]), 
+                        dtype=cur_new_embed.dtype, device=cur_new_embed.device
+                    )
+                ), dim=0))
+                if cur_len > 0:
+                    new_labels_padded[i, :cur_len] = cur_new_labels
+                    attention_mask[i, :cur_len] = True
+                    position_ids[i, :cur_len] = torch.arange(
+                        0, cur_len, dtype=position_ids.dtype, device=position_ids.device
+                    )
+
+        new_input_embeds = torch.stack(new_input_embeds_padded, dim=0)
+
+        # Restore None values if originally None
+        new_labels = None if _labels is None else new_labels_padded
+        attention_mask = None if _attention_mask is None else attention_mask.to(dtype=_attention_mask.dtype)
+        position_ids = None if _position_ids is None else position_ids
+
+        return None, position_ids, attention_mask, past_key_values, new_input_embeds, new_labels
+
+    def forward(
+        self,
+        input_ids: torch.LongTensor = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        past_key_values: Optional[List[torch.FloatTensor]] = None,
+        inputs_embeds: Optional[torch.FloatTensor] = None,
+        labels: Optional[torch.LongTensor] = None,
+        use_cache: Optional[bool] = None,
+        output_attentions: Optional[bool] = None,
+        output_hidden_states: Optional[bool] = None,
+        images: Optional[torch.FloatTensor] = None,
+        pixel_values: Optional[torch.FloatTensor] = None,
+        pixel_attention_mask: Optional[torch.Tensor] = None,
+        spatial_shapes: Optional[torch.Tensor] = None,
+        return_dict: Optional[bool] = None,
+        cache_position: Optional[torch.LongTensor] = None,     
+        logits_to_keep: Union[int, torch.Tensor] = 0,                   
+    ) -> Union[Tuple, CausalLMOutputWithPast]:
+
+        # Accept processor output format (pixel_values, pixel_attention_mask, spatial_shapes)
+        if images is None and pixel_values is not None:
+            images = BatchFeature({
+                "pixel_values": pixel_values,
+                "pixel_attention_mask": pixel_attention_mask,
+                "spatial_shapes": spatial_shapes,
+            })
+
+        if inputs_embeds is None:
+            (
+                input_ids,
+                position_ids,
+                attention_mask,
+                past_key_values,
+                inputs_embeds,
+                labels
+            ) = self.prepare_inputs_labels_for_multimodal(
+                input_ids,
+                position_ids,
+                attention_mask,
+                past_key_values,
+                labels,
+                images
+            )
+
+        return super().forward(
+            input_ids=input_ids,
+            attention_mask=attention_mask,
+            position_ids=position_ids,
+            past_key_values=past_key_values,
+            inputs_embeds=inputs_embeds,
+            labels=labels,
+            use_cache=use_cache,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            return_dict=return_dict,
+            cache_position=cache_position,
+            logits_to_keep=logits_to_keep         
+        )
+
+    def prepare_inputs_for_generation(
+        self, input_ids, past_key_values=None, inputs_embeds=None, attention_mask=None, **kwargs
+    ):
+        images = kwargs.pop("images", None)
+        
+        # Also accept processor output format (pixel_values, pixel_attention_mask, spatial_shapes)
+        pixel_values = kwargs.pop("pixel_values", None)
+        pixel_attention_mask = kwargs.pop("pixel_attention_mask", None)
+        spatial_shapes = kwargs.pop("spatial_shapes", None)
+        
+        # If processor output format is provided, package as BatchFeature for the model
+        if images is None and pixel_values is not None:
+            images = BatchFeature({
+                "pixel_values": pixel_values,
+                "pixel_attention_mask": pixel_attention_mask,
+                "spatial_shapes": spatial_shapes,
+            })
+
+        _inputs = super().prepare_inputs_for_generation(
+            input_ids, 
+            past_key_values=past_key_values, 
+            inputs_embeds=inputs_embeds, 
+            attention_mask=attention_mask,
+            **kwargs
+        )
+
+        if images is not None:
+            _inputs['images'] = images
+        return _inputs
+
+    @classmethod
+    def from_pretrained(cls, pretrained_model_name_or_path, **kwargs):
+        """Load model from pretrained weights."""
+        # Extract dtype before passing to super() since we need it later
+        torch_dtype = kwargs.get("torch_dtype", None)
+        
+        # Check if loading from local checkpoint that contains vision tower weights
+        load_vision_from_checkpoint = False
+        if os.path.isdir(pretrained_model_name_or_path):
+            for file_name in os.listdir(pretrained_model_name_or_path):
+                if file_name.endswith("safetensors"):
+                    fpath = os.path.join(pretrained_model_name_or_path, file_name)
+                    shard_weights = load_file(fpath)
+                    if any(k.startswith("model.vision_tower.vision_tower.") for k in shard_weights.keys()):
+                        load_vision_from_checkpoint = True
+                        logger.info("Detected vision tower weights in checkpoint - will skip downloading from HuggingFace.")
+                        break
+        
+        model = super().from_pretrained(pretrained_model_name_or_path, **kwargs)
+
+        vision_tower = model.get_vision_tower()
+
+        # Load vision weights if model is a local path
+        if vision_tower is not None:
+            if not vision_tower.is_loaded:
+                # Skip downloading pretrained weights if we'll load from checkpoint
+                vision_tower.load_model(skip_weights=load_vision_from_checkpoint)
+
+            if load_vision_from_checkpoint:
+                try:
+                    vision_weights = {}
+                    for file_name in os.listdir(pretrained_model_name_or_path):
+                        if file_name.endswith("safetensors"):
+                            fpath = os.path.join(pretrained_model_name_or_path, file_name)
+                            shard_weights = load_file(fpath)
+                            
+                            # Handle weights with prefix "model.vision_tower.vision_tower."
+                            # (the nested vision_tower is the actual encoder)
+                            prefix_nested = "model.vision_tower.vision_tower."
+                            prefix_simple = "model.vision_tower."
+                            
+                            for k, v in shard_weights.items():
+                                if k.startswith(prefix_nested):
+                                    # Strip to get "vision_tower.xxx"
+                                    new_key = k[len("model.vision_tower."):]
+                                    vision_weights[new_key] = v
+                                elif k.startswith(prefix_simple) and not k.startswith(prefix_nested):
+                                    # Direct vision_tower weights (like image_processor params if saved)
+                                    new_key = k[len(prefix_simple):]
+                                    vision_weights[new_key] = v
+
+                    if vision_weights:
+                        vision_tower.load_state_dict(vision_weights, strict=False)
+                        logger.info("Vision tower weights loaded from checkpoint.")
+                    else:
+                        logger.warning("No vision tower weights found in checkpoint!")
+                except Exception as e:
+                    logger.warning(
+                        "Vision tower weights NOT loaded from checkpoint. "
+                        f"Exception: {e}"
+                    )
+
+            vision_tower.to(model.device)
+
+        # Sync dtype
+        dtype = torch_dtype if torch_dtype is not None else model.dtype
+        dtype = model.dtype if dtype == "auto" else dtype
+        model.to(dtype)
+
+        # Fix generation config
+        if isinstance(model.generation_config.eos_token_id, (list, set)):
+            model.generation_config.eos_token_id = model.generation_config.eos_token_id[0]
+        if model.generation_config.pad_token_id is None:
+            model.generation_config.pad_token_id = model.generation_config.eos_token_id
+
+        return model
+
+
+# =============================================================================
+# Register with AutoConfig/AutoModel
+# =============================================================================
+
+AutoConfig.register("phi4-siglip", Phi4VisionR)
+AutoModelForCausalLM.register(Phi4VisionR, Phi4ForCausalLMV)

preprocessor_config.json

@@ -0,0 +1,18 @@
+{
+  "auto_map": {
+    "AutoProcessor": "processing_phi4_visionr.Phi4VisionRProcessor"
+  },
+  "do_convert_rgb": true,
+  "do_normalize": true,
+  "do_rescale": true,
+  "do_resize": true,
+  "image_mean": [0.5, 0.5, 0.5],
+  "image_processor_type": "Siglip2ImageProcessorNoUpscale",
+  "image_std": [0.5, 0.5, 0.5],
+  "max_num_patches": 3600,
+  "min_num_patches": 256,
+  "patch_size": 16,
+  "processor_class": "Phi4VisionRProcessor",
+  "rescale_factor": 0.00392156862745098,
+  "resample": 2
+}

processing_phi4_visionr.py

@@ -0,0 +1,342 @@
+"""
+Processor class for Phi4-Siglip.
+
+This module provides:
+- Phi4VisionRProcessor: Combined tokenizer and image processor
+- Utility functions for image and text processing
+"""
+
+from typing import List, Optional, Union
+
+import torch
+from PIL import Image
+from transformers import BatchFeature
+from transformers.image_utils import ImageInput
+from transformers.processing_utils import ProcessorMixin
+from transformers.tokenization_utils_base import PaddingStrategy, TextInput, TruncationStrategy
+from transformers.utils import TensorType
+
+# Constants (duplicated here to avoid circular imports when running scripts directly)
+IMAGE_TOKEN_INDEX = -200
+DEFAULT_IMAGE_TOKEN = "<image>"
+
+
+# =============================================================================
+# Image Utilities
+# =============================================================================
+
+def process_images(images: List[Image.Image], image_processor, model_cfg=None):
+    """
+    Process images for the model.
+    
+    Args:
+        images: List of PIL images
+        image_processor: The image processor (Siglip2ImageProcessorNoUpscale for NaFlex)
+        model_cfg: Optional model config (unused, kept for API compatibility)
+        
+    Returns:
+        Processed images as BatchFeature (for NaFlex)
+    """
+    # Check if NaFlex (has max_num_patches attribute)
+    is_naflex = hasattr(image_processor, "max_num_patches")
+    
+    # Process with image processor
+    if is_naflex:
+        return image_processor(images, return_tensors='pt')
+    else:
+        return image_processor(images, return_tensors='pt')['pixel_values']
+
+
+# =============================================================================
+# Tokenizer Utilities
+# =============================================================================
+
+def tokenizer_image_token(
+    prompt: str, 
+    tokenizer, 
+    image_token_index: int = IMAGE_TOKEN_INDEX, 
+    return_tensors: Optional[str] = None
+):
+    """
+    Tokenize a prompt containing <image> tokens.
+    
+    Replaces <image> with IMAGE_TOKEN_INDEX in the token sequence.
+    
+    Args:
+        prompt: The text prompt with <image> placeholders
+        tokenizer: The tokenizer to use
+        image_token_index: The index to use for image tokens
+        return_tensors: If 'pt', return as PyTorch tensor
+        
+    Returns:
+        List of token ids or tensor
+    """
+    prompt_chunks = [tokenizer(chunk).input_ids for chunk in prompt.split(DEFAULT_IMAGE_TOKEN)]
+
+    def insert_separator(X, sep):
+        return [ele for sublist in zip(X, [sep] * len(X)) for ele in sublist][:-1]
+
+    input_ids = []
+    offset = 0
+    if len(prompt_chunks) > 0 and len(prompt_chunks[0]) > 0 and prompt_chunks[0][0] == tokenizer.bos_token_id:
+        offset = 1
+        input_ids.append(prompt_chunks[0][0])
+
+    for x in insert_separator(prompt_chunks, [image_token_index] * (offset + 1)):
+        input_ids.extend(x[offset:])
+
+    if return_tensors is not None:
+        if return_tensors == 'pt':
+            return torch.tensor(input_ids, dtype=torch.long)
+        raise ValueError(f'Unsupported tensor type: {return_tensors}')
+    return input_ids
+
+
+# =============================================================================
+# Main Processor Class
+# =============================================================================
+
+class Phi4VisionRProcessor(ProcessorMixin):
+    """
+    Processor for Phi4-Siglip that wraps an image processor and tokenizer.
+    
+    This processor handles:
+    - Image preprocessing (via SigLIP or SigLIP2/NaFlex)
+    - Text tokenization with image token insertion
+    - Conversation formatting
+    
+    Args:
+        image_processor: The image processor (from vision tower)
+        tokenizer: The text tokenizer
+    """
+    
+    attributes = ["image_processor", "tokenizer"]
+    image_processor_class = "AutoImageProcessor"
+    tokenizer_class = "AutoTokenizer"
+    
+    def __init__(self, image_processor, tokenizer):
+        self.image_processor = image_processor
+        self.tokenizer = tokenizer
+
+    def __call__(
+        self,
+        text: Union[TextInput, List[TextInput]] = None,
+        images: ImageInput = None,
+        padding: Union[bool, str, PaddingStrategy] = False,
+        truncation: Union[bool, str, TruncationStrategy] = None,
+        max_length: Optional[int] = None,
+        return_tensors: Optional[Union[str, TensorType]] = TensorType.PYTORCH,
+        **kwargs,
+    ) -> BatchFeature:
+        """
+        Process text and images for the model.
+        
+        Args:
+            text: The text input(s). Can contain <image> tokens.
+            images: The image input(s).
+            padding: Padding strategy.
+            truncation: Whether to truncate.
+            max_length: Maximum sequence length.
+            return_tensors: Return type for tensors.
+            
+        Returns:
+            BatchFeature with input_ids, attention_mask, and optionally pixel_values.
+        """
+        # Process images
+        if images is not None:
+            if not isinstance(images, list):
+                images = [images]
+            image_inputs = process_images(images, self.image_processor)
+        else:
+            image_inputs = None
+        
+        # Process text
+        if text is not None:
+            if isinstance(text, str):
+                text = [text]
+            
+            # Check if text contains image tokens
+            has_images = any(DEFAULT_IMAGE_TOKEN in t for t in text)
+            
+            if has_images and images is not None:
+                # Tokenize with image token handling
+                input_ids_list = []
+                for t in text:
+                    ids = tokenizer_image_token(t, self.tokenizer, return_tensors='pt')
+                    input_ids_list.append(ids)
+                
+                # Pad sequences
+                if len(input_ids_list) > 1:
+                    max_len = max(len(ids) for ids in input_ids_list)
+                    padded_ids = []
+                    attention_masks = []
+                    pad_token_id = self.tokenizer.pad_token_id or 0
+                    
+                    for ids in input_ids_list:
+                        pad_len = max_len - len(ids)
+                        if padding and pad_len > 0:
+                            padded_ids.append(torch.cat([ids, torch.full((pad_len,), pad_token_id, dtype=torch.long)]))
+                            attention_masks.append(torch.cat([torch.ones(len(ids)), torch.zeros(pad_len)]))
+                        else:
+                            padded_ids.append(ids)
+                            attention_masks.append(torch.ones(len(ids)))
+                    
+                    input_ids = torch.stack(padded_ids)
+                    attention_mask = torch.stack(attention_masks).long()
+                else:
+                    input_ids = input_ids_list[0].unsqueeze(0)
+                    attention_mask = torch.ones_like(input_ids)
+            else:
+                # Standard tokenization
+                text_inputs = self.tokenizer(
+                    text,
+                    padding=padding,
+                    truncation=truncation,
+                    max_length=max_length,
+                    return_tensors=return_tensors,
+                )
+                input_ids = text_inputs["input_ids"]
+                attention_mask = text_inputs["attention_mask"]
+        else:
+            input_ids = None
+            attention_mask = None
+
+        # Build output
+        data = {}
+        if input_ids is not None:
+            data["input_ids"] = input_ids
+            data["attention_mask"] = attention_mask
+        
+        if image_inputs is not None:
+            if isinstance(image_inputs, BatchFeature):
+                # NaFlex case - merge all fields
+                data.update(image_inputs)
+            else:
+                data["pixel_values"] = image_inputs
+                
+        return BatchFeature(data=data, tensor_type=return_tensors)
+
+    def batch_decode(self, *args, **kwargs):
+        """Decode token ids to text. Forwards to tokenizer."""
+        return self.tokenizer.batch_decode(*args, **kwargs)
+
+    def decode(self, *args, **kwargs):
+        """Decode token ids to text. Forwards to tokenizer."""
+        return self.tokenizer.decode(*args, **kwargs)
+
+    @property
+    def model_input_names(self):
+        """Get model input names from tokenizer and image processor."""
+        tokenizer_input_names = self.tokenizer.model_input_names
+        image_processor_input_names = getattr(
+            self.image_processor, 
+            'model_input_names', 
+            ["pixel_values"]
+        )
+        return list(dict.fromkeys(tokenizer_input_names + image_processor_input_names))
+
+    @classmethod
+    def from_pretrained(cls, pretrained_model_name_or_path, **kwargs):
+        """
+        Load processor from a pretrained model path.
+        
+        This will load the tokenizer and create the appropriate image processor
+        based on the model config.
+        """
+        from transformers import AutoTokenizer, AutoConfig
+        
+        tokenizer = AutoTokenizer.from_pretrained(pretrained_model_name_or_path, **kwargs)
+        
+        # Try to load config to determine vision tower type
+        try:
+            config = AutoConfig.from_pretrained(pretrained_model_name_or_path, **kwargs)
+            vision_tower_name = getattr(config, 'mm_vision_tower', None)
+            vision_config = getattr(config, 'vision_config', None)
+            
+            if vision_tower_name and 'naflex' in vision_tower_name.lower():
+                from .modeling_phi4_visionr import Siglip2ImageProcessorNoUpscale
+                # Use embedded vision_config to avoid network calls
+                # Infer patch_size from model name if not in config (patch14 vs patch16)
+                if vision_config is not None:
+                    if 'patch_size' in vision_config:
+                        patch_size = vision_config['patch_size']
+                    elif 'patch14' in vision_tower_name.lower():
+                        patch_size = 14
+                    else:
+                        patch_size = 16  # default for patch16-naflex
+                    image_processor = Siglip2ImageProcessorNoUpscale(
+                        patch_size=patch_size,
+                        max_num_patches=getattr(config, 'max_num_patches', 3600),
+                        min_num_patches=getattr(config, 'min_num_patches', 256),
+                    )
+                else:
+                    image_processor = Siglip2ImageProcessorNoUpscale.from_pretrained(
+                        vision_tower_name,
+                        max_num_patches=getattr(config, 'max_num_patches', 3600),
+                        min_num_patches=getattr(config, 'min_num_patches', 256),
+                    )
+            elif vision_tower_name:
+                from transformers import SiglipImageProcessor
+                # Use embedded vision_config to avoid network calls
+                if vision_config is not None:
+                    image_processor = SiglipImageProcessor(
+                        size={"height": vision_config.get('image_size', 384), "width": vision_config.get('image_size', 384)},
+                    )
+                else:
+                    image_processor = SiglipImageProcessor.from_pretrained(vision_tower_name)
+            else:
+                image_processor = None
+        except Exception:
+            image_processor = None
+            
+        return cls(image_processor=image_processor, tokenizer=tokenizer)
+
+
+# =============================================================================
+# Convenience Functions
+# =============================================================================
+
+def prepare_inputs_for_generation(
+    prompt: str,
+    images: Optional[List[Image.Image]],
+    processor: Phi4VisionRProcessor,
+    device: str = "cuda",
+    dtype: torch.dtype = torch.bfloat16,
+) -> dict:
+    """
+    Prepare inputs for model generation.
+    
+    Args:
+        prompt: The user prompt (without conversation formatting)
+        images: Optional list of PIL images
+        processor: The Phi4VisionRProcessor
+        device: Device to place tensors on
+        dtype: Data type for tensors
+        
+    Returns:
+        Dictionary with model inputs
+    """
+    # Add image token to prompt if images provided
+    if images:
+        prompt = DEFAULT_IMAGE_TOKEN + "\n" + prompt
+    
+    # Use tokenizer's chat_template
+    messages = [{"role": "user", "content": prompt}]
+    full_prompt = processor.tokenizer.apply_chat_template(
+        messages, 
+        tokenize=False, 
+        add_generation_prompt=True
+    )
+    
+    inputs = processor(
+        text=full_prompt,
+        images=images,
+        return_tensors="pt",
+    )
+    
+    # Move to device
+    for key in inputs:
+        if isinstance(inputs[key], torch.Tensor):
+            inputs[key] = inputs[key].to(device=device, dtype=dtype if inputs[key].is_floating_point() else inputs[key].dtype)
+    
+    return inputs

sample_inference.py

@@ -0,0 +1,99 @@
+"""
+Sample inference script for Phi4-Siglip.
+
+Usage:
+    cd phi4mm
+    python sample_inference.py
+"""
+from PIL import Image
+import torch
+from transformers import AutoModelForCausalLM, AutoProcessor
+
+model_path = "." # change to your model path if not running in the same directory as the model
+
+# get first argument as an image path if not throw an error explaining how to use the script with an image
+import sys
+with_image_mode = False
+if len(sys.argv) > 1:
+    with_image_mode = True
+    image_path = sys.argv[1]
+    print(f"Image path provided: {image_path}")
+else:
+    print("No image path provided. Running in text-only mode. To run with an image, provide the image path as an argument:\npython sample_inference.py /path/to/image.jpg")
+
+# Load model and processor
+print("Loading model...")
+processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)
+model = AutoModelForCausalLM.from_pretrained(
+    model_path,
+    trust_remote_code=True,
+    dtype=torch.bfloat16,
+    device_map="cuda",
+).eval()
+
+# Import helpers for image processing
+from processing_phi4_visionr import DEFAULT_IMAGE_TOKEN
+
+print(f"Model loaded on {model.device}")
+
+#################################################### text-only ####################################################
+print("\n" + "="*60)
+print("TEST: Text-only generation")
+print("="*60)
+
+messages = [{"role": "user", "content": "What is the answer for 1+1? Explain it."}]
+prompt = processor.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+
+print(f">>> Prompt\n{prompt}")
+inputs = processor(prompt, images=None, return_tensors="pt").to("cuda:0")
+generate_ids = model.generate(
+    **inputs,
+    max_new_tokens=4096,
+    eos_token_id=processor.tokenizer.eos_token_id,
+    do_sample=False,
+)
+generate_ids = generate_ids[:, inputs['input_ids'].shape[1]:]
+response = processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]
+print(f'>>> Response\n{response}')
+
+#################################################### single image ####################################################
+if not with_image_mode:
+    print("\n" + "="*60)
+    print("No image provided, skipping multimodal test.")
+    print("="*60)
+    exit(0)
+
+print("\n" + "="*60)
+print("TEST: Single image understanding")
+print("="*60)
+
+messages = [{"role": "user", "content": DEFAULT_IMAGE_TOKEN + "\nDescribe this image in detail."}]
+prompt = processor.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+
+if with_image_mode:
+    print(f">>> Loading image from {image_path}")
+    image = Image.open(image_path).convert("RGB")
+    print(f"Image size: {image.size}")
+else:
+    image = None
+
+print(f">>> Prompt\n{prompt}")
+
+# Process text and image together using the processor
+inputs = processor(text=prompt, images=[image] if image is not None else None, return_tensors="pt").to("cuda:0")
+
+with torch.inference_mode():
+    generate_ids = model.generate(
+        **inputs,
+        max_new_tokens=4096,
+        eos_token_id=processor.tokenizer.eos_token_id,
+        do_sample=False,
+    )
+
+generate_ids = generate_ids[:, inputs['input_ids'].shape[1]:]
+response = processor.tokenizer.decode(generate_ids[0], skip_special_tokens=True)
+print(f'>>> Response\n{response}')
+
+print("\n" + "="*60)
+print("All tests completed!")
+print("="*60)

special_tokens_map.json

@@ -0,0 +1,24 @@
+{
+  "bos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": true,
+    "normalized": false,
+    "rstrip": true,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|im_end|>",
+    "lstrip": true,
+    "normalized": false,
+    "rstrip": true,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|dummy_85|>",
+    "lstrip": true,
+    "normalized": false,
+    "rstrip": true,
+    "single_word": false
+  },
+  "unk_token": "<|endoftext|>"
+}

tokenizer_config.json

@@ -0,0 +1,782 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "100256": {
+      "content": "<|dummy_0|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100257": {
+      "content": "<|endoftext|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100258": {
+      "content": "<|fim_prefix|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": false
+    },
+    "100259": {
+      "content": "<|fim_middle|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": false
+    },
+    "100260": {
+      "content": "<|fim_suffix|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": false
+    },
+    "100261": {
+      "content": "<|dummy_1|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100262": {
+      "content": "<|dummy_2|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100263": {
+      "content": "<|dummy_3|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100264": {
+      "content": "<|im_start|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100265": {
+      "content": "<|im_end|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100266": {
+      "content": "<|im_sep|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100267": {
+      "content": "<|dummy_4|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100268": {
+      "content": "<|dummy_5|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100269": {
+      "content": "<|dummy_6|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100270": {
+      "content": "<|dummy_7|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100271": {
+      "content": "<|dummy_8|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100272": {
+      "content": "<|dummy_9|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100273": {
+      "content": "<|dummy_10|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100274": {
+      "content": "<|dummy_11|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100275": {
+      "content": "<|dummy_12|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100276": {
+      "content": "<|endofprompt|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100277": {
+      "content": "<|dummy_13|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100278": {
+      "content": "<|dummy_14|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100279": {
+      "content": "<|dummy_15|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100280": {
+      "content": "<|dummy_16|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100281": {
+      "content": "<|dummy_17|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100282": {
+      "content": "<|dummy_18|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100283": {
+      "content": "<|dummy_19|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100284": {
+      "content": "<|dummy_20|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100285": {
+      "content": "<|dummy_21|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100286": {
+      "content": "<|dummy_22|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100287": {
+      "content": "<|dummy_23|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100288": {
+      "content": "<|dummy_24|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100289": {
+      "content": "<|dummy_25|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100290": {
+      "content": "<|dummy_26|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100291": {
+      "content": "<|dummy_27|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100292": {
+      "content": "<|dummy_28|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100293": {
+      "content": "<|dummy_29|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100294": {
+      "content": "<|dummy_30|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100295": {
+      "content": "<|dummy_31|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100296": {
+      "content": "<|dummy_32|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100297": {
+      "content": "<|dummy_33|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100298": {
+      "content": "<|dummy_34|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100299": {
+      "content": "<|dummy_35|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100300": {
+      "content": "<|dummy_36|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100301": {
+      "content": "<|dummy_37|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100302": {
+      "content": "<|dummy_38|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100303": {
+      "content": "<|dummy_39|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100304": {
+      "content": "<|dummy_40|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100305": {
+      "content": "<|dummy_41|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100306": {
+      "content": "<|dummy_42|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100307": {
+      "content": "<|dummy_43|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100308": {
+      "content": "<|dummy_44|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100309": {
+      "content": "<|dummy_45|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100310": {
+      "content": "<|dummy_46|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100311": {
+      "content": "<|dummy_47|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100312": {
+      "content": "<|dummy_48|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100313": {
+      "content": "<|dummy_49|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100314": {
+      "content": "<|dummy_50|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100315": {
+      "content": "<|dummy_51|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100316": {
+      "content": "<|dummy_52|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100317": {
+      "content": "<|dummy_53|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100318": {
+      "content": "<|dummy_54|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100319": {
+      "content": "<|dummy_55|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100320": {
+      "content": "<|dummy_56|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100321": {
+      "content": "<|dummy_57|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100322": {
+      "content": "<|dummy_58|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100323": {
+      "content": "<|dummy_59|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100324": {
+      "content": "<|dummy_60|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100325": {
+      "content": "<|dummy_61|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100326": {
+      "content": "<|dummy_62|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100327": {
+      "content": "<|dummy_63|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100328": {
+      "content": "<|dummy_64|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100329": {
+      "content": "<|dummy_65|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100330": {
+      "content": "<|dummy_66|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100331": {
+      "content": "<|dummy_67|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100332": {
+      "content": "<|dummy_68|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100333": {
+      "content": "<|dummy_69|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100334": {
+      "content": "<|dummy_70|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100335": {
+      "content": "<|dummy_71|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100336": {
+      "content": "<|dummy_72|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100337": {
+      "content": "<|dummy_73|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100338": {
+      "content": "<|dummy_74|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100339": {
+      "content": "<|dummy_75|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100340": {
+      "content": "<|dummy_76|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100341": {
+      "content": "<|dummy_77|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100342": {
+      "content": "<|dummy_78|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100343": {
+      "content": "<|dummy_79|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100344": {
+      "content": "<|dummy_80|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100345": {
+      "content": "<|dummy_81|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100346": {
+      "content": "<|dummy_82|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100347": {
+      "content": "<|dummy_83|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100348": {
+      "content": "<nothink>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100349": {
+      "content": "<|dummy_85|>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "100350": {
+      "content": "<think>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": false
+    },
+    "100351": {
+      "content": "</think>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "extra_special_tokens": {},
+  "model_max_length": 16384,
+  "pad_token": "<|dummy_85|>",
+  "padding_side": "right",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>"
+}