Kosmos (code, demo, models, paper)

.gitattributes

@@ -33,3 +33,8 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+Kosmos-2.[[:space:]]Grounding[[:space:]]Multimodal[[:space:]]Large[[:space:]]Language[[:space:]]Models[[:space:]]to[[:space:]]the[[:space:]]World.pdf filter=lfs diff=lfs merge=lfs -text
+KOSMOS-2.5.[[:space:]]A[[:space:]]Multimodal[[:space:]]Literate[[:space:]]Model.pdf filter=lfs diff=lfs merge=lfs -text
+Language[[:space:]]Is[[:space:]]Not[[:space:]]All[[:space:]]You[[:space:]]Need.[[:space:]]Aligning[[:space:]]Perception[[:space:]]with[[:space:]]Language[[:space:]]Models.pdf filter=lfs diff=lfs merge=lfs -text
+models/kosmos-2.5/output.png filter=lfs diff=lfs merge=lfs -text
+models/kosmos-2.5/receipt_00008.png filter=lfs diff=lfs merge=lfs -text

KOSMOS-2.5. A Multimodal Literate Model.pdf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f659ae6c3696172faf4afa57c6c6d563d9fc026ca378ceccdbfb33a5e5ee20f1
+size 6426197

Kosmos-2. Grounding Multimodal Large Language Models to the World.pdf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:064214db82bb573831fd102ace00be90147633e7f47dac491089619e03bc7e58
+size 7580509

Language Is Not All You Need. Aligning Perception with Language Models.pdf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a6a848b830a5ceabaf4fd1cbd38d32572edca05d7db513398bca459f3cf6352a
+size 3743156

code/Kosmos2.5 (kyegomez).zip

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e20f0129cf0521f1b1e2b0b8d283b734644baade3f9a58545b6201d60f31eddc
+size 522240

code/kosmos2.5-8bitq-onnx.zip

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:05f788b060b553a6c05a75982c60059ea1245911d17f693caabd64d1f959618c
+size 68711

code/kosmos2.5-int8-quantization.zip

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:28621b029cd25851142c1d0670cf6d0f867e3845cd71c2766acfc7996dd41a06
+size 62854

demo/kosmos-2.5-demo/.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

demo/kosmos-2.5-demo/README.md

@@ -0,0 +1,61 @@
+---
+title: KOSMOS-2.5 Document AI Demo
+emoji: 📄
+colorFrom: blue
+colorTo: purple
+sdk: gradio
+sdk_version: 4.44.0
+app_file: app.py
+pinned: false
+license: mit
+---
+
+# KOSMOS-2.5 Document AI Demo
+
+This Space demonstrates the capabilities of Microsoft's **KOSMOS-2.5**, a multimodal literate model for machine reading of text-intensive images.
+
+## Features
+
+🔥 **Three powerful modes**:
+
+1. **📝 Markdown Generation**: Convert document images to clean markdown format
+2. **🔍 OCR with Bounding Boxes**: Extract text with precise spatial coordinates and visualization
+3. **💬 Document Q&A**: Ask questions about document content using KOSMOS-2.5 Chat
+
+## What is KOSMOS-2.5?
+
+KOSMOS-2.5 is Microsoft's latest document AI model that excels at understanding text-rich images. It can:
+
+- Generate spatially-aware text blocks with coordinates
+- Produce structured markdown output that captures document styles
+- Answer questions about document content through the chat variant
+
+The model was pre-trained on 357.4 million text-rich document images and achieves performance comparable to much larger models (1.3B vs 7B parameters) on visual question answering benchmarks.
+
+## Example Use Cases
+
+- **Receipts**: Extract itemized information or ask "What's the total amount?"
+- **Forms**: Convert to structured format or query specific fields
+- **Articles**: Get clean markdown or ask content-specific questions
+- **Screenshots**: Extract UI text or get information about elements
+
+## Model Information
+
+- **Base Model**: [microsoft/kosmos-2.5](https://huggingface.co/microsoft/kosmos-2.5)
+- **Chat Model**: [microsoft/kosmos-2.5-chat](https://huggingface.co/microsoft/kosmos-2.5-chat)
+- **Paper**: [Kosmos-2.5: A Multimodal Literate Model](https://arxiv.org/abs/2309.11419)
+
+## Note
+
+This is a generative model and may occasionally produce inaccurate results. Please verify outputs for critical applications.
+
+## Citation
+
+```bibtex
+@article{lv2023kosmos,
+  title={Kosmos-2.5: A multimodal literate model},
+  author={Lv, Tengchao and Huang, Yupan and Chen, Jingye and Cui, Lei and Ma, Shuming and Chang, Yaoyao and Huang, Shaohan and Wang, Wenhui and Dong, Li and Luo, Weiyao and others},
+  journal={arXiv preprint arXiv:2309.11419},
+  year={2023}
+}
+```

demo/kosmos-2.5-demo/app.py

@@ -0,0 +1,315 @@
+import spaces
+import torch
+import gradio as gr
+from PIL import Image
+from transformers import AutoProcessor, Kosmos2_5ForConditionalGeneration
+import re
+
+# Check if CUDA is available
+device = "cuda" if torch.cuda.is_available() else "cpu"
+dtype = torch.bfloat16 if torch.cuda.is_available() else torch.float32
+
+# Check if Flash Attention 2 is available
+def is_flash_attention_available():
+    try:
+        import flash_attn
+        return True
+    except ImportError:
+        return False
+
+# Initialize models and processors lazily
+base_model = None
+base_processor = None
+chat_model = None
+chat_processor = None
+
+def load_base_model():
+    global base_model, base_processor
+    if base_model is None:
+        base_repo = "microsoft/kosmos-2.5"
+        
+        # Use Flash Attention 2 if available, otherwise use default attention
+        model_kwargs = {
+            "device_map": "cuda", 
+            "dtype": dtype,
+        }
+        if is_flash_attention_available():
+            model_kwargs["attn_implementation"] = "flash_attention_2"
+        
+        base_model = Kosmos2_5ForConditionalGeneration.from_pretrained(
+            base_repo, 
+            **model_kwargs
+        )
+        base_processor = AutoProcessor.from_pretrained(base_repo)
+    return base_model, base_processor
+
+def load_chat_model():
+    global chat_model, chat_processor
+    if chat_model is None:
+        chat_repo = "microsoft/kosmos-2.5-chat"
+        
+        # Use Flash Attention 2 if available, otherwise use default attention
+        model_kwargs = {
+            "device_map": "cuda",
+            "dtype": dtype,
+        }
+        if is_flash_attention_available():
+            model_kwargs["attn_implementation"] = "flash_attention_2"
+        
+        chat_model = Kosmos2_5ForConditionalGeneration.from_pretrained(
+            chat_repo,
+            **model_kwargs
+        )
+        chat_processor = AutoProcessor.from_pretrained(chat_repo)
+    return chat_model, chat_processor
+
+def post_process_ocr(y, scale_height, scale_width, prompt="<ocr>"):
+    y = y.replace(prompt, "")
+    if "<md>" in prompt:
+        return y
+    
+    pattern = r"<bbox><x_\d+><y_\d+><x_\d+><y_\d+></bbox>"
+    bboxs_raw = re.findall(pattern, y)
+    lines = re.split(pattern, y)[1:]
+    bboxs = [re.findall(r"\d+", i) for i in bboxs_raw]
+    bboxs = [[int(j) for j in i] for i in bboxs]
+    
+    info = ""
+    for i in range(len(lines)):
+        if i < len(bboxs):
+            box = bboxs[i]
+            x0, y0, x1, y1 = box
+            if not (x0 >= x1 or y0 >= y1):
+                x0 = int(x0 * scale_width)
+                y0 = int(y0 * scale_height)
+                x1 = int(x1 * scale_width)
+                y1 = int(y1 * scale_height)
+                info += f"{x0},{y0},{x1},{y0},{x1},{y1},{x0},{y1},{lines[i]}\n"
+    return info.strip()
+
+@spaces.GPU(duration=120)
+def generate_markdown(image):
+    if image is None:
+        return "Please upload an image."
+    
+    model, processor = load_base_model()
+    
+    prompt = "<md>"
+    inputs = processor(text=prompt, images=image, return_tensors="pt")
+    
+    height, width = inputs.pop("height"), inputs.pop("width")
+    raw_width, raw_height = image.size
+    scale_height = raw_height / height
+    scale_width = raw_width / width
+    
+    inputs = {k: v.to("cuda") if v is not None else None for k, v in inputs.items()}
+    inputs["flattened_patches"] = inputs["flattened_patches"].to(dtype)
+    
+    with torch.no_grad():
+        generated_ids = model.generate(
+            **inputs,
+            max_new_tokens=1024,
+        )
+    
+    generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)
+    result = generated_text[0].replace(prompt, "").strip()
+    
+    return result
+
+@spaces.GPU(duration=120)
+def generate_ocr(image):
+    if image is None:
+        return "Please upload an image.", None
+    
+    model, processor = load_base_model()
+    
+    prompt = "<ocr>"
+    inputs = processor(text=prompt, images=image, return_tensors="pt")
+    
+    height, width = inputs.pop("height"), inputs.pop("width")
+    raw_width, raw_height = image.size
+    scale_height = raw_height / height
+    scale_width = raw_width / width
+    
+    inputs = {k: v.to("cuda") if v is not None else None for k, v in inputs.items()}
+    inputs["flattened_patches"] = inputs["flattened_patches"].to(dtype)
+    
+    with torch.no_grad():
+        generated_ids = model.generate(
+            **inputs,
+            max_new_tokens=1024,
+        )
+    
+    generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)
+    
+    # Post-process OCR output
+    output_text = post_process_ocr(generated_text[0], scale_height, scale_width)
+    
+    # Create visualization
+    from PIL import ImageDraw
+    vis_image = image.copy()
+    draw = ImageDraw.Draw(vis_image)
+    
+    lines = output_text.split("\n")
+    for line in lines:
+        if not line.strip():
+            continue
+        parts = line.split(",")
+        if len(parts) >= 8:
+            try:
+                coords = list(map(int, parts[:8]))
+                draw.polygon(coords, outline="red", width=2)
+            except:
+                continue
+    
+    return output_text, vis_image
+
+@spaces.GPU(duration=120)
+def generate_chat_response(image, question):
+    if image is None:
+        return "Please upload an image."
+    if not question.strip():
+        return "Please ask a question."
+    
+    model, processor = load_chat_model()
+    
+    template = "<md>A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: {} ASSISTANT:"
+    prompt = template.format(question)
+    
+    inputs = processor(text=prompt, images=image, return_tensors="pt")
+    
+    height, width = inputs.pop("height"), inputs.pop("width")
+    raw_width, raw_height = image.size
+    scale_height = raw_height / height
+    scale_width = raw_width / width
+    
+    inputs = {k: v.to("cuda") if v is not None else None for k, v in inputs.items()}
+    inputs["flattened_patches"] = inputs["flattened_patches"].to(dtype)
+    
+    with torch.no_grad():
+        generated_ids = model.generate(
+            **inputs,
+            max_new_tokens=1024,
+        )
+    
+    generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)
+    
+    # Extract only the assistant's response
+    result = generated_text[0]
+    if "ASSISTANT:" in result:
+        result = result.split("ASSISTANT:")[-1].strip()
+    
+    return result
+
+# Create Gradio interface
+with gr.Blocks(title="KOSMOS-2.5 Document AI Demo", theme=gr.themes.Soft()) as demo:
+    gr.Markdown("""
+    # KOSMOS-2.5 Document AI Demo
+    
+    Explore Microsoft's KOSMOS-2.5, a multimodal model for reading text-intensive images! 
+    This demo showcases three capabilities:
+    
+    1. **Markdown Generation**: Convert document images to markdown format
+    2. **OCR with Bounding Boxes**: Extract text with spatial coordinates
+    3. **Document Q&A**: Ask questions about document content using KOSMOS-2.5 Chat
+    
+    Upload a document image (receipt, form, article, etc.) and try different tasks!
+    """)
+    
+    with gr.Tabs():
+        # Markdown Generation Tab
+        with gr.TabItem("📝 Markdown Generation"):
+            with gr.Row():
+                with gr.Column():
+                    md_image = gr.Image(type="pil", label="Upload Document Image")
+                    gr.Examples(
+                        examples=["https://huggingface.co/microsoft/kosmos-2.5/resolve/main/receipt_00008.png"],
+                        inputs=md_image
+                    )
+                    md_button = gr.Button("Generate Markdown", variant="primary")
+                with gr.Column():
+                    md_output = gr.Textbox(
+                        label="Generated Markdown", 
+                        lines=15, 
+                        max_lines=20,
+                        show_copy_button=True
+                    )
+        
+        # OCR Tab
+        with gr.TabItem("🔍 OCR with Bounding Boxes"):
+            with gr.Row():
+                with gr.Column():
+                    ocr_image = gr.Image(type="pil", label="Upload Document Image")
+                    gr.Examples(
+                        examples=["https://huggingface.co/microsoft/kosmos-2.5/resolve/main/receipt_00008.png"],
+                        inputs=ocr_image
+                    )
+                    ocr_button = gr.Button("Extract Text with Coordinates", variant="primary")
+                with gr.Column():
+                    with gr.Row():
+                        ocr_text = gr.Textbox(
+                            label="Extracted Text with Coordinates", 
+                            lines=10,
+                            show_copy_button=True
+                        )
+                        ocr_vis = gr.Image(label="Visualization (Red boxes show detected text)")
+        
+        # Chat Tab
+        with gr.TabItem("💬 Document Q&A (Chat)"):
+            with gr.Row():
+                with gr.Column():
+                    chat_image = gr.Image(type="pil", label="Upload Document Image")
+                    gr.Examples(
+                        examples=["https://huggingface.co/microsoft/kosmos-2.5/resolve/main/receipt_00008.png"],
+                        inputs=chat_image
+                    )
+                    chat_question = gr.Textbox(
+                        label="Ask a question about the document",
+                        placeholder="e.g., What is the total amount on this receipt?",
+                        lines=2
+                    )
+                    gr.Examples(
+                        examples=["What is the total amount on this receipt?", "What items were purchased?", "When was this receipt issued?", "What is the subtotal?"],
+                        inputs=chat_question
+                    )
+                    chat_button = gr.Button("Get Answer", variant="primary")
+                with gr.Column():
+                    chat_output = gr.Textbox(
+                        label="Answer", 
+                        lines=8,
+                        show_copy_button=True
+                    )
+    
+    # Event handlers
+    md_button.click(
+        fn=generate_markdown,
+        inputs=[md_image],
+        outputs=[md_output]
+    )
+    
+    ocr_button.click(
+        fn=generate_ocr,
+        inputs=[ocr_image],
+        outputs=[ocr_text, ocr_vis]
+    )
+    
+    chat_button.click(
+        fn=generate_chat_response,
+        inputs=[chat_image, chat_question],
+        outputs=[chat_output]
+    )
+    
+    # Examples section
+    gr.Markdown("""
+    ## Example Use Cases:
+    - **Receipts**: Extract itemized information or ask about totals
+    - **Forms**: Convert to structured format or answer specific questions
+    - **Articles**: Get markdown format or ask about content
+    - **Screenshots**: Extract text or get information about specific elements
+    
+    ## Note:
+    This is a generative model and may occasionally hallucinate. Results should be verified for accuracy.
+    """)
+
+if __name__ == "__main__":
+    demo.launch()

demo/kosmos-2.5-demo/requirements.txt

@@ -0,0 +1,7 @@
+gradio==4.44.0
+torch>=2.0.0
+git+https://github.com/huggingface/transformers.git
+accelerate
+pillow
+requests
+spaces

demo/kosmos-2.5-demo/source.txt

@@ -0,0 +1 @@
+https://huggingface.co/spaces/nielsr/kosmos-2.5-demo

models/kosmos-2.5-4bit-text/.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

models/kosmos-2.5-4bit-text/README.md

@@ -0,0 +1,199 @@
+---
+library_name: transformers
+tags: []
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]

models/kosmos-2.5-4bit-text/config.json

@@ -0,0 +1,26 @@
+{
+  "activation_dropout": 0.0,
+  "activation_function": "gelu",
+  "architectures": [
+    "Kosmos2_5TextForCausalLM"
+  ],
+  "attention_dropout": 0.0,
+  "attention_heads": 16,
+  "bos_token_id": 0,
+  "dropout": 0,
+  "embed_dim": 1536,
+  "eos_token_id": 2,
+  "ffn_dim": 6144,
+  "init_std": 0.02,
+  "layer_norm_eps": 1e-05,
+  "layerdrop": 0.0,
+  "layers": 24,
+  "max_position_embeddings": 4096,
+  "model_type": "kosmos_2_5_text_model",
+  "pad_token_id": 1,
+  "scale_embedding": true,
+  "torch_dtype": "float16",
+  "transformers_version": "4.47.0.dev0",
+  "use_cache": true,
+  "vocab_size": 108481
+}

models/kosmos-2.5-4bit-text/generation_config.json

@@ -0,0 +1,7 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 0,
+  "eos_token_id": 2,
+  "pad_token_id": 1,
+  "transformers_version": "4.47.0.dev0"
+}

models/kosmos-2.5-4bit-text/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:487cc16ca412b5b3b4ed471a4f4b1f4a203cceaacb40a071bc8bbae794c38c84
+size 717140144

models/kosmos-2.5-4bit-text/source.txt

@@ -0,0 +1 @@
+https://huggingface.co/Fireblossom/kosmos-2.5-4bit-text

models/kosmos-2.5-4bit-vision/.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

models/kosmos-2.5-4bit-vision/README.md

@@ -0,0 +1,199 @@
+---
+library_name: transformers
+tags: []
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]

models/kosmos-2.5-4bit-vision/config.json

@@ -0,0 +1,22 @@
+{
+  "architectures": [
+    "Kosmos2_5VisionModel"
+  ],
+  "attention_dropout": 0.0,
+  "d_ff": 3968,
+  "d_kv": 64,
+  "dense_act_fn": "gelu_new",
+  "dropout_rate": 0.0,
+  "hidden_size": 1536,
+  "initializer_factor": 1.0,
+  "initializer_range": 1e-10,
+  "layer_norm_eps": 1e-06,
+  "max_length": 4096,
+  "model_type": "kosmos_2_5_vision_model",
+  "num_attention_heads": 24,
+  "num_hidden_layers": 18,
+  "patch_embed_hidden_size": 768,
+  "seq_len": 4096,
+  "torch_dtype": "float16",
+  "transformers_version": "4.47.0.dev0"
+}

models/kosmos-2.5-4bit-vision/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ab95bc9ba21aa3f673e447a4bb6ffbb694c38baf442d05a93bef748f11d8d1c7
+size 306710574

models/kosmos-2.5-4bit-vision/source.txt

@@ -0,0 +1 @@
+https://huggingface.co/Fireblossom/kosmos-2.5-4bit-vision

models/kosmos-2.5-chat/.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

models/kosmos-2.5-chat/README.md

@@ -0,0 +1,87 @@
+---
+language: en
+license: mit
+library_name: transformers
+pipeline_tag: image-text-to-text
+---
+# Kosmos-2.5-chat
+
+[Microsoft Document AI](https://www.microsoft.com/en-us/research/project/document-ai/) | [GitHub](https://github.com/microsoft/unilm/tree/master/kosmos-2.5)
+
+## Model description
+Kosmos-2.5 is a multimodal literate model for machine reading of text-intensive images. Pre-trained on large-scale text-intensive images, Kosmos-2.5 excels in two distinct yet cooperative transcription tasks: (1) generating spatially-aware text blocks, where each block of text is assigned its spatial coordinates within the image, and (2) producing structured text output that captures styles and structures into the markdown format. This unified multimodal literate capability is achieved through a shared decoder-only auto-regressive Transformer architecture, task-specific prompts, and flexible text representations. We evaluate Kosmos-2.5 on end-to-end document-level text recognition and image-to-markdown text generation. Furthermore, the model can be readily adapted for any text-intensive image understanding task with different prompts through supervised fine-tuning, making it a general-purpose tool for real-world applications involving text-rich images. This work also paves the way for the future scaling of multimodal large language models.
+
+Kosmos-2.5-chat is a model specifically trained for Visual Question Answering (VQA) tasks, based on further training of Kosmos-2.5. For more details about Kosmos-2.5-chat, please refer to the paper.
+
+[Kosmos-2.5: A Multimodal Literate Model](https://arxiv.org/abs/2309.11419)
+
+## Usage
+
+KOSMOS-2.5 is supported from Transformers >= 4.56. Find the docs [here](https://huggingface.co/docs/transformers/main/en/model_doc/kosmos2_5).
+
+```python
+import re
+import torch
+import requests
+from PIL import Image, ImageDraw
+from transformers import AutoProcessor, Kosmos2_5ForConditionalGeneration
+
+repo = "microsoft/kosmos-2.5-chat"
+device = "cuda:0"
+dtype = torch.bfloat16
+
+model = Kosmos2_5ForConditionalGeneration.from_pretrained(repo,
+                                                          device_map=device,
+                                                          torch_dtype=dtype,
+                                                          attn_implementation="flash_attention_2")
+processor = AutoProcessor.from_pretrained(repo)
+
+# sample image
+url = "https://huggingface.co/microsoft/kosmos-2.5/resolve/main/receipt_00008.png"
+
+image = Image.open(requests.get(url, stream=True).raw)
+
+question = "What is the sub total of the receipt?"
+template = "<md>A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: {} ASSISTANT:"
+prompt = template.format(question)
+inputs = processor(text=prompt, images=image, return_tensors="pt")
+
+height, width = inputs.pop("height"), inputs.pop("width")
+raw_width, raw_height = image.size
+scale_height = raw_height / height
+scale_width = raw_width / width
+
+inputs = {k: v.to(device) if v is not None else None for k, v in inputs.items()}
+inputs["flattened_patches"] = inputs["flattened_patches"].to(dtype)
+generated_ids = model.generate(
+    **inputs,
+    max_new_tokens=1024,
+)
+
+generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)
+print(generated_text[0])
+```
+
+## NOTE:
+Since this is a generative model, there is a risk of **hallucination** during the generation process, and it **CAN NOT** guarantee the accuracy of all results in the images.
+
+## Inference
+**Document Understanding Task:** For usage instructions, please refer to [chat.py](chat.py).
+
+## Citation
+
+If you find Kosmos-2.5-chat useful in your research, please cite the following paper:
+
+```
+@article{lv2023kosmos,
+  title={Kosmos-2.5: A multimodal literate model},
+  author={Lv, Tengchao and Huang, Yupan and Chen, Jingye and Cui, Lei and Ma, Shuming and Chang, Yaoyao and Huang, Shaohan and Wang, Wenhui and Dong, Li and Luo, Weiyao and others},
+  journal={arXiv preprint arXiv:2309.11419},
+  year={2023}
+}
+```
+
+## License
+The content of this project itself is licensed under the [MIT](https://github.com/microsoft/unilm/blob/master/kosmos-2.5/LICENSE)
+
+[Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct)

models/kosmos-2.5-chat/chat.py

@@ -0,0 +1,40 @@
+import re
+import torch
+import requests
+from PIL import Image, ImageDraw
+from transformers import AutoProcessor, Kosmos2_5ForConditionalGeneration
+
+repo = "microsoft/kosmos-2.5-chat"
+device = "cuda:0"
+dtype = torch.bfloat16
+
+model = Kosmos2_5ForConditionalGeneration.from_pretrained(repo,
+                                                          device_map=device,
+                                                          torch_dtype=dtype,
+                                                          attn_implementation="flash_attention_2")
+processor = AutoProcessor.from_pretrained(repo)
+
+# sample image
+url = "https://huggingface.co/microsoft/kosmos-2.5/resolve/main/receipt_00008.png"
+
+image = Image.open(requests.get(url, stream=True).raw)
+
+question = "What is the sub total of the receipt?"
+template = "<md>A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: {} ASSISTANT:"
+prompt = template.format(question)
+inputs = processor(text=prompt, images=image, return_tensors="pt")
+
+height, width = inputs.pop("height"), inputs.pop("width")
+raw_width, raw_height = image.size
+scale_height = raw_height / height
+scale_width = raw_width / width
+
+inputs = {k: v.to(device) if v is not None else None for k, v in inputs.items()}
+inputs["flattened_patches"] = inputs["flattened_patches"].to(dtype)
+generated_ids = model.generate(
+    **inputs,
+    max_new_tokens=1024,
+)
+
+generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)
+print(generated_text[0])

models/kosmos-2.5-chat/config.json

@@ -0,0 +1,163 @@
+{
+  "architectures": [
+    "Kosmos2_5ForConditionalGeneration"
+  ],
+  "latent_query_num": 2048,
+  "model_type": "kosmos-2.5",
+  "text_config": {
+    "_name_or_path": "",
+    "activation_dropout": 0.0,
+    "activation_function": "gelu",
+    "add_cross_attention": false,
+    "architectures": null,
+    "attention_dropout": 0.0,
+    "attention_heads": 16,
+    "bad_words_ids": null,
+    "begin_suppress_tokens": null,
+    "bos_token_id": 0,
+    "chunk_size_feed_forward": 0,
+    "cross_attention_hidden_size": null,
+    "decoder_start_token_id": null,
+    "diversity_penalty": 0.0,
+    "do_sample": false,
+    "dropout": 0.1,
+    "early_stopping": false,
+    "embed_dim": 1536,
+    "encoder_no_repeat_ngram_size": 0,
+    "eos_token_id": 2,
+    "exponential_decay_length_penalty": null,
+    "ffn_dim": 6144,
+    "finetuning_task": null,
+    "forced_bos_token_id": null,
+    "forced_eos_token_id": null,
+    "id2label": {
+      "0": "LABEL_0",
+      "1": "LABEL_1"
+    },
+    "init_std": 0.02,
+    "is_decoder": false,
+    "is_encoder_decoder": false,
+    "label2id": {
+      "LABEL_0": 0,
+      "LABEL_1": 1
+    },
+    "layer_norm_eps": 1e-05,
+    "layerdrop": 0.0,
+    "length_penalty": 1.0,
+    "layers": 24,
+    "max_length": 20,
+    "max_position_embeddings": 4096,
+    "min_length": 0,
+    "model_type": "kosmos_2_5_text_model",
+    "no_repeat_ngram_size": 3,
+    "num_beam_groups": 1,
+    "num_beams": 1,
+    "num_return_sequences": 1,
+    "output_attentions": false,
+    "output_hidden_states": false,
+    "output_scores": false,
+    "pad_token_id": 1,
+    "prefix": null,
+    "problem_type": null,
+    "pruned_heads": {},
+    "remove_invalid_values": false,
+    "repetition_penalty": 1.0,
+    "return_dict": true,
+    "return_dict_in_generate": false,
+    "scale_embedding": true,
+    "sep_token_id": null,
+    "suppress_tokens": null,
+    "task_specific_params": null,
+    "temperature": 1.0,
+    "tf_legacy_loss": false,
+    "tie_encoder_decoder": false,
+    "tie_word_embeddings": true,
+    "tokenizer_class": null,
+    "top_k": 50,
+    "top_p": 1.0,
+    "torch_dtype": null,
+    "torchscript": false,
+    "typical_p": 1.0,
+    "use_bfloat16": false,
+    "use_cache": true,
+    "vocab_size": 108481
+  },
+  "torch_dtype": "float32",
+  "transformers_version": "4.43.3",
+  "vision_config": {
+    "_name_or_path": "",
+    "add_cross_attention": false,
+    "architectures": null,
+    "attention_dropout": 0.0,
+    "bad_words_ids": null,
+    "begin_suppress_tokens": null,
+    "bos_token_id": null,
+    "chunk_size_feed_forward": 0,
+    "cross_attention_hidden_size": null,
+    "intermediate_size": 3968,
+    "head_dim": 64,
+    "decoder_start_token_id": null,
+    "dense_act_fn": "gelu_new",
+    "diversity_penalty": 0.0,
+    "do_sample": false,
+    "dropout_rate": 0.0,
+    "early_stopping": false,
+    "encoder_no_repeat_ngram_size": 0,
+    "eos_token_id": null,
+    "exponential_decay_length_penalty": null,
+    "finetuning_task": null,
+    "forced_bos_token_id": null,
+    "forced_eos_token_id": null,
+    "hidden_size": 1536,
+    "id2label": {
+      "0": "LABEL_0",
+      "1": "LABEL_1"
+    },
+    "initializer_factor": 1.0,
+    "initializer_range": 1e-10,
+    "is_decoder": false,
+    "is_encoder_decoder": false,
+    "label2id": {
+      "LABEL_0": 0,
+      "LABEL_1": 1
+    },
+    "layer_norm_eps": 1e-06,
+    "length_penalty": 1.0,
+    "max_length": 4096,
+    "min_length": 0,
+    "model_type": "kosmos_2_5_vision_model",
+    "no_repeat_ngram_size": 0,
+    "num_attention_heads": 24,
+    "num_beam_groups": 1,
+    "num_beams": 1,
+    "num_hidden_layers": 18,
+    "num_return_sequences": 1,
+    "output_attentions": false,
+    "output_hidden_states": false,
+    "output_scores": false,
+    "pad_token_id": null,
+    "patch_embed_hidden_size": 768,
+    "prefix": null,
+    "problem_type": null,
+    "pruned_heads": {},
+    "remove_invalid_values": false,
+    "repetition_penalty": 1.0,
+    "return_dict": true,
+    "return_dict_in_generate": false,
+    "sep_token_id": null,
+    "max_num_patches": 4096,
+    "suppress_tokens": null,
+    "task_specific_params": null,
+    "temperature": 1.0,
+    "tf_legacy_loss": false,
+    "tie_encoder_decoder": false,
+    "tie_word_embeddings": true,
+    "tokenizer_class": null,
+    "top_k": 50,
+    "top_p": 1.0,
+    "torch_dtype": null,
+    "torchscript": false,
+    "typical_p": 1.0,
+    "use_bfloat16": false
+  }
+}

models/kosmos-2.5-chat/generation_config.json

@@ -0,0 +1,8 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 0,
+  "eos_token_id": 2,
+  "no_repeat_ngram_size": 3,
+  "pad_token_id": 1,
+  "transformers_version": "4.43.3"
+}

models/kosmos-2.5-chat/model-00001-of-00002.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ea32f178375c21412ee2829b2389682544ecd6f990a6120b219e065b1500d085
+size 4995252144

models/kosmos-2.5-chat/model-00002-of-00002.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5b642fe2ce0ad3ff30838a3daebb6d26252770c65d29306a809e05598fc0e393
+size 503408384

models/kosmos-2.5-chat/model.safetensors.index.json

@@ -0,0 +1,621 @@
+{
+  "metadata": {
+    "total_size": 5498585088
+  },
+  "weight_map": {
+    "image_to_text_projection.dense.bias": "model-00002-of-00002.safetensors",
+    "image_to_text_projection.dense.weight": "model-00002-of-00002.safetensors",
+    "image_to_text_projection.latent_query": "model-00002-of-00002.safetensors",
+    "image_to_text_projection.x_attn.k_proj.bias": "model-00002-of-00002.safetensors",
+    "image_to_text_projection.x_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "image_to_text_projection.x_attn.out_proj.bias": "model-00002-of-00002.safetensors",
+    "image_to_text_projection.x_attn.out_proj.weight": "model-00002-of-00002.safetensors",
+    "image_to_text_projection.x_attn.q_proj.bias": "model-00002-of-00002.safetensors",
+    "image_to_text_projection.x_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "image_to_text_projection.x_attn.v_proj.bias": "model-00002-of-00002.safetensors",
+    "image_to_text_projection.x_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "text_model.model.embed_tokens.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layer_norm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layer_norm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.0.ffn.fc1.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.0.ffn.fc1.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.0.ffn.fc2.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.0.ffn.fc2.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.0.ffn.ffn_layernorm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.0.ffn.ffn_layernorm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.0.final_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.0.final_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.0.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.0.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.0.self_attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.0.self_attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.0.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.0.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.0.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.0.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.0.self_attn_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.0.self_attn_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.1.ffn.fc1.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.1.ffn.fc1.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.1.ffn.fc2.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.1.ffn.fc2.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.1.ffn.ffn_layernorm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.1.ffn.ffn_layernorm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.1.final_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.1.final_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.1.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.1.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.1.self_attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.1.self_attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.1.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.1.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.1.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.1.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.1.self_attn_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.1.self_attn_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.10.ffn.fc1.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.10.ffn.fc1.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.10.ffn.fc2.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.10.ffn.fc2.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.10.ffn.ffn_layernorm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.10.ffn.ffn_layernorm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.10.final_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.10.final_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.10.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.10.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.10.self_attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.10.self_attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.10.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.10.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.10.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.10.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.10.self_attn_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.10.self_attn_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.11.ffn.fc1.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.11.ffn.fc1.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.11.ffn.fc2.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.11.ffn.fc2.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.11.ffn.ffn_layernorm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.11.ffn.ffn_layernorm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.11.final_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.11.final_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.11.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.11.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.11.self_attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.11.self_attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.11.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.11.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.11.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.11.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.11.self_attn_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.11.self_attn_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.12.ffn.fc1.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.12.ffn.fc1.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.12.ffn.fc2.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.12.ffn.fc2.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.12.ffn.ffn_layernorm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.12.ffn.ffn_layernorm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.12.final_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.12.final_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.12.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.12.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.12.self_attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.12.self_attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.12.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.12.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.12.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.12.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.12.self_attn_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.12.self_attn_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.13.ffn.fc1.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.13.ffn.fc1.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.13.ffn.fc2.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.13.ffn.fc2.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.13.ffn.ffn_layernorm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.13.ffn.ffn_layernorm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.13.final_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.13.final_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.13.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.13.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.13.self_attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.13.self_attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.13.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.13.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.13.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.13.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.13.self_attn_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.13.self_attn_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.14.ffn.fc1.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.14.ffn.fc1.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.14.ffn.fc2.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.14.ffn.fc2.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.14.ffn.ffn_layernorm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.14.ffn.ffn_layernorm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.14.final_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.14.final_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.14.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.14.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.14.self_attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.14.self_attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.14.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.14.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.14.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.14.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.14.self_attn_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.14.self_attn_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.15.ffn.fc1.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.15.ffn.fc1.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.15.ffn.fc2.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.15.ffn.fc2.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.15.ffn.ffn_layernorm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.15.ffn.ffn_layernorm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.15.final_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.15.final_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.15.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.15.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.15.self_attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.15.self_attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.15.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.15.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.15.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.15.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.15.self_attn_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.15.self_attn_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.16.ffn.fc1.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.16.ffn.fc1.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.16.ffn.fc2.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.16.ffn.fc2.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.16.ffn.ffn_layernorm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.16.ffn.ffn_layernorm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.16.final_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.16.final_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.16.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.16.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.16.self_attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.16.self_attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.16.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.16.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.16.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.16.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.16.self_attn_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.16.self_attn_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.17.ffn.fc1.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.17.ffn.fc1.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.17.ffn.fc2.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.17.ffn.fc2.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.17.ffn.ffn_layernorm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.17.ffn.ffn_layernorm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.17.final_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.17.final_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.17.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.17.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.17.self_attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.17.self_attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.17.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.17.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.17.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.17.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.17.self_attn_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.17.self_attn_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.18.ffn.fc1.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.18.ffn.fc1.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.18.ffn.fc2.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.18.ffn.fc2.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.18.ffn.ffn_layernorm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.18.ffn.ffn_layernorm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.18.final_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.18.final_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.18.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.18.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.18.self_attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.18.self_attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.18.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.18.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.18.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.18.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.18.self_attn_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.18.self_attn_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.19.ffn.fc1.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.19.ffn.fc1.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.19.ffn.fc2.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.19.ffn.fc2.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.19.ffn.ffn_layernorm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.19.ffn.ffn_layernorm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.19.final_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.19.final_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.19.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.19.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.19.self_attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.19.self_attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.19.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.19.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.19.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.19.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.19.self_attn_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.19.self_attn_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.2.ffn.fc1.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.2.ffn.fc1.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.2.ffn.fc2.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.2.ffn.fc2.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.2.ffn.ffn_layernorm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.2.ffn.ffn_layernorm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.2.final_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.2.final_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.2.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.2.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.2.self_attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.2.self_attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.2.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.2.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.2.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.2.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.2.self_attn_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.2.self_attn_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.20.ffn.fc1.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.20.ffn.fc1.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.20.ffn.fc2.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.20.ffn.fc2.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.20.ffn.ffn_layernorm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.20.ffn.ffn_layernorm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.20.final_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.20.final_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.20.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.20.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.20.self_attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.20.self_attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.20.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.20.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.20.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.20.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.20.self_attn_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.20.self_attn_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.21.ffn.fc1.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.21.ffn.fc1.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.21.ffn.fc2.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.21.ffn.fc2.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.21.ffn.ffn_layernorm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.21.ffn.ffn_layernorm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.21.final_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.21.final_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.21.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.21.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.21.self_attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.21.self_attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.21.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.21.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.21.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.21.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.21.self_attn_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.21.self_attn_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.22.ffn.fc1.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.22.ffn.fc1.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.22.ffn.fc2.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.22.ffn.fc2.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.22.ffn.ffn_layernorm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.22.ffn.ffn_layernorm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.22.final_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.22.final_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.22.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.22.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.22.self_attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.22.self_attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.22.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.22.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.22.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.22.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.22.self_attn_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.22.self_attn_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.23.ffn.fc1.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.23.ffn.fc1.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.23.ffn.fc2.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.23.ffn.fc2.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.23.ffn.ffn_layernorm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.23.ffn.ffn_layernorm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.23.final_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.23.final_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.23.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.23.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.23.self_attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.23.self_attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.23.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.23.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.23.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.23.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.23.self_attn_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.23.self_attn_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.3.ffn.fc1.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.3.ffn.fc1.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.3.ffn.fc2.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.3.ffn.fc2.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.3.ffn.ffn_layernorm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.3.ffn.ffn_layernorm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.3.final_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.3.final_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.3.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.3.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.3.self_attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.3.self_attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.3.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.3.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.3.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.3.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.3.self_attn_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.3.self_attn_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.4.ffn.fc1.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.4.ffn.fc1.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.4.ffn.fc2.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.4.ffn.fc2.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.4.ffn.ffn_layernorm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.4.ffn.ffn_layernorm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.4.final_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.4.final_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.4.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.4.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.4.self_attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.4.self_attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.4.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.4.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.4.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.4.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.4.self_attn_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.4.self_attn_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.5.ffn.fc1.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.5.ffn.fc1.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.5.ffn.fc2.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.5.ffn.fc2.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.5.ffn.ffn_layernorm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.5.ffn.ffn_layernorm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.5.final_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.5.final_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.5.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.5.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.5.self_attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.5.self_attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.5.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.5.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.5.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.5.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.5.self_attn_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.5.self_attn_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.6.ffn.fc1.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.6.ffn.fc1.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.6.ffn.fc2.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.6.ffn.fc2.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.6.ffn.ffn_layernorm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.6.ffn.ffn_layernorm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.6.final_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.6.final_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.6.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.6.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.6.self_attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.6.self_attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.6.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.6.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.6.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.6.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.6.self_attn_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.6.self_attn_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.7.ffn.fc1.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.7.ffn.fc1.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.7.ffn.fc2.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.7.ffn.fc2.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.7.ffn.ffn_layernorm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.7.ffn.ffn_layernorm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.7.final_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.7.final_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.7.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.7.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.7.self_attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.7.self_attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.7.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.7.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.7.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.7.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.7.self_attn_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.7.self_attn_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.8.ffn.fc1.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.8.ffn.fc1.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.8.ffn.fc2.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.8.ffn.fc2.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.8.ffn.ffn_layernorm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.8.ffn.ffn_layernorm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.8.final_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.8.final_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.8.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.8.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.8.self_attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.8.self_attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.8.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.8.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.8.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.8.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.8.self_attn_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.8.self_attn_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.9.ffn.fc1.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.9.ffn.fc1.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.9.ffn.fc2.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.9.ffn.fc2.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.9.ffn.ffn_layernorm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.9.ffn.ffn_layernorm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.9.final_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.9.final_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.9.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.9.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.9.self_attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.9.self_attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.9.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.9.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.9.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.9.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.9.self_attn_layer_norm.bias": "model-00001-of-00002.safetensors",
+    "text_model.model.layers.9.self_attn_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "text_model.model.segment_emb.weight": "model-00001-of-00002.safetensors",
+    "vision_model.embeddings.column_embedder.weight": "model-00001-of-00002.safetensors",
+    "vision_model.embeddings.patch_projection.bias": "model-00001-of-00002.safetensors",
+    "vision_model.embeddings.patch_projection.weight": "model-00001-of-00002.safetensors",
+    "vision_model.embeddings.row_embedder.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.0.attention.key.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.0.attention.output.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.0.attention.query.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.0.attention.value.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.0.mlp.wi_0.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.0.mlp.wi_1.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.0.mlp.wo.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.0.pre_attention_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.0.pre_mlp_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.1.attention.key.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.1.attention.output.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.1.attention.query.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.1.attention.value.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.1.mlp.wi_0.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.1.mlp.wi_1.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.1.mlp.wo.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.1.pre_attention_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.1.pre_mlp_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.10.attention.key.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.10.attention.output.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.10.attention.query.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.10.attention.value.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.10.mlp.wi_0.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.10.mlp.wi_1.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.10.mlp.wo.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.10.pre_attention_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.10.pre_mlp_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.11.attention.key.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.11.attention.output.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.11.attention.query.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.11.attention.value.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.11.mlp.wi_0.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.11.mlp.wi_1.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.11.mlp.wo.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.11.pre_attention_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.11.pre_mlp_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.12.attention.key.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.12.attention.output.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.12.attention.query.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.12.attention.value.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.12.mlp.wi_0.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.12.mlp.wi_1.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.12.mlp.wo.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.12.pre_attention_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.12.pre_mlp_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.13.attention.key.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.13.attention.output.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.13.attention.query.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.13.attention.value.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.13.mlp.wi_0.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.13.mlp.wi_1.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.13.mlp.wo.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.13.pre_attention_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.13.pre_mlp_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.14.attention.key.weight": "model-00002-of-00002.safetensors",
+    "vision_model.encoder.layer.14.attention.output.weight": "model-00002-of-00002.safetensors",
+    "vision_model.encoder.layer.14.attention.query.weight": "model-00002-of-00002.safetensors",
+    "vision_model.encoder.layer.14.attention.value.weight": "model-00002-of-00002.safetensors",
+    "vision_model.encoder.layer.14.mlp.wi_0.weight": "model-00002-of-00002.safetensors",
+    "vision_model.encoder.layer.14.mlp.wi_1.weight": "model-00002-of-00002.safetensors",
+    "vision_model.encoder.layer.14.mlp.wo.weight": "model-00002-of-00002.safetensors",
+    "vision_model.encoder.layer.14.pre_attention_layer_norm.weight": "model-00002-of-00002.safetensors",
+    "vision_model.encoder.layer.14.pre_mlp_layer_norm.weight": "model-00002-of-00002.safetensors",
+    "vision_model.encoder.layer.15.attention.key.weight": "model-00002-of-00002.safetensors",
+    "vision_model.encoder.layer.15.attention.output.weight": "model-00002-of-00002.safetensors",
+    "vision_model.encoder.layer.15.attention.query.weight": "model-00002-of-00002.safetensors",
+    "vision_model.encoder.layer.15.attention.value.weight": "model-00002-of-00002.safetensors",
+    "vision_model.encoder.layer.15.mlp.wi_0.weight": "model-00002-of-00002.safetensors",
+    "vision_model.encoder.layer.15.mlp.wi_1.weight": "model-00002-of-00002.safetensors",
+    "vision_model.encoder.layer.15.mlp.wo.weight": "model-00002-of-00002.safetensors",
+    "vision_model.encoder.layer.15.pre_attention_layer_norm.weight": "model-00002-of-00002.safetensors",
+    "vision_model.encoder.layer.15.pre_mlp_layer_norm.weight": "model-00002-of-00002.safetensors",
+    "vision_model.encoder.layer.16.attention.key.weight": "model-00002-of-00002.safetensors",
+    "vision_model.encoder.layer.16.attention.output.weight": "model-00002-of-00002.safetensors",
+    "vision_model.encoder.layer.16.attention.query.weight": "model-00002-of-00002.safetensors",
+    "vision_model.encoder.layer.16.attention.value.weight": "model-00002-of-00002.safetensors",
+    "vision_model.encoder.layer.16.mlp.wi_0.weight": "model-00002-of-00002.safetensors",
+    "vision_model.encoder.layer.16.mlp.wi_1.weight": "model-00002-of-00002.safetensors",
+    "vision_model.encoder.layer.16.mlp.wo.weight": "model-00002-of-00002.safetensors",
+    "vision_model.encoder.layer.16.pre_attention_layer_norm.weight": "model-00002-of-00002.safetensors",
+    "vision_model.encoder.layer.16.pre_mlp_layer_norm.weight": "model-00002-of-00002.safetensors",
+    "vision_model.encoder.layer.17.attention.key.weight": "model-00002-of-00002.safetensors",
+    "vision_model.encoder.layer.17.attention.output.weight": "model-00002-of-00002.safetensors",
+    "vision_model.encoder.layer.17.attention.query.weight": "model-00002-of-00002.safetensors",
+    "vision_model.encoder.layer.17.attention.value.weight": "model-00002-of-00002.safetensors",
+    "vision_model.encoder.layer.17.mlp.wi_0.weight": "model-00002-of-00002.safetensors",
+    "vision_model.encoder.layer.17.mlp.wi_1.weight": "model-00002-of-00002.safetensors",
+    "vision_model.encoder.layer.17.mlp.wo.weight": "model-00002-of-00002.safetensors",
+    "vision_model.encoder.layer.17.pre_attention_layer_norm.weight": "model-00002-of-00002.safetensors",
+    "vision_model.encoder.layer.17.pre_mlp_layer_norm.weight": "model-00002-of-00002.safetensors",
+    "vision_model.encoder.layer.2.attention.key.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.2.attention.output.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.2.attention.query.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.2.attention.value.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.2.mlp.wi_0.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.2.mlp.wi_1.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.2.mlp.wo.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.2.pre_attention_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.2.pre_mlp_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.3.attention.key.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.3.attention.output.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.3.attention.query.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.3.attention.value.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.3.mlp.wi_0.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.3.mlp.wi_1.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.3.mlp.wo.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.3.pre_attention_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.3.pre_mlp_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.4.attention.key.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.4.attention.output.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.4.attention.query.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.4.attention.value.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.4.mlp.wi_0.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.4.mlp.wi_1.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.4.mlp.wo.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.4.pre_attention_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.4.pre_mlp_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.5.attention.key.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.5.attention.output.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.5.attention.query.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.5.attention.value.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.5.mlp.wi_0.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.5.mlp.wi_1.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.5.mlp.wo.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.5.pre_attention_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.5.pre_mlp_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.6.attention.key.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.6.attention.output.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.6.attention.query.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.6.attention.value.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.6.mlp.wi_0.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.6.mlp.wi_1.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.6.mlp.wo.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.6.pre_attention_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.6.pre_mlp_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.7.attention.key.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.7.attention.output.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.7.attention.query.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.7.attention.value.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.7.mlp.wi_0.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.7.mlp.wi_1.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.7.mlp.wo.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.7.pre_attention_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.7.pre_mlp_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.8.attention.key.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.8.attention.output.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.8.attention.query.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.8.attention.value.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.8.mlp.wi_0.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.8.mlp.wi_1.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.8.mlp.wo.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.8.pre_attention_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.8.pre_mlp_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.9.attention.key.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.9.attention.output.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.9.attention.query.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.9.attention.value.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.9.mlp.wi_0.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.9.mlp.wi_1.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.9.mlp.wo.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.9.pre_attention_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "vision_model.encoder.layer.9.pre_mlp_layer_norm.weight": "model-00001-of-00002.safetensors",
+    "vision_model.layernorm.weight": "model-00002-of-00002.safetensors"
+  }
+}

models/kosmos-2.5-chat/preprocessor_config.json

@@ -0,0 +1,5 @@
+{
+    "image_processor_type": "Kosmos2_5ImageProcessor",
+    "processor_class": "Kosmos2_5Processor"
+  }
+  

models/kosmos-2.5-chat/source.txt

@@ -0,0 +1 @@
+https://huggingface.co/microsoft/kosmos-2.5-chat

models/kosmos-2.5-chat/special_tokens_map.json

@@ -0,0 +1,33 @@
+{
+  "boi_token": "<image>",
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eoi_token": "</image>",
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "image_token": "<s>",
+  "pad_token": {
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

models/kosmos-2.5-ft/.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

models/kosmos-2.5-ft/README.md

@@ -0,0 +1,111 @@
+---
+license: apache-2.0
+datasets:
+- merve/doclaynet-small
+base_model:
+- microsoft/kosmos-2.5
+pipeline_tag: image-text-to-text
+library_name: transformers
+tags:
+- ocr
+---
+
+## Kosmos-2.5 Fine-tuned on DocLayNet
+
+Kosmos-2.5 fine-tuned on grounded OCR (OCR with bounding boxes), find the script here: ([GH](https://github.com/merveenoyan/smol-vision/blob/main/Grounded_Fine_tuning%20GH.ipynb), [HF](https://huggingface.co/merve/smol-vision/blob/main/Grounded_Fine_tuning.ipynb))
+
+Try the (base model) Kosmos-2.5 demo [here](https://huggingface.co/spaces/nielsr/kosmos-2.5-demo).
+
+Here's the inference code:
+
+```python
+from transformers import AutoProcessor, Kosmos2_5ForConditionalGeneration
+import torch 
+
+model = Kosmos2_5ForConditionalGeneration.from_pretrained("merve/kosmos-2.5-ft", device_map="cuda", dtype=torch.bfloat16)
+processor = AutoProcessor.from_pretrained("microsoft/kosmos-2.5")
+
+import requests
+from PIL import Image
+url = "https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/ufo-sighting.jpg"
+image = Image.open(requests.get(url, stream=True).raw)
+
+
+import re
+prompt = "<ocr>"
+inputs = processor(text=prompt, images=image, return_tensors="pt")
+height, width = inputs.pop("height"), inputs.pop("width")
+raw_width, raw_height = image.size
+scale_height = raw_height / height
+scale_width = raw_width / width
+
+inputs = {k: v.to("cuda") if v is not None else None for k, v in inputs.items()}
+inputs["flattened_patches"] = inputs["flattened_patches"].to(torch.bfloat16)
+
+generated_ids = model.generate(
+    **inputs,
+    max_new_tokens=2000,
+)
+
+generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)
+
+import re
+from PIL import ImageDraw
+
+def post_process(y, scale_height, scale_width):
+
+    pattern = r"<bbox><x_\d+><y_\d+><x_\d+><y_\d+></bbox>"
+    bboxes_raw = re.findall(pattern, y)
+    lines = re.split(pattern, y)[1:]  
+    bboxes = [list(map(int, re.findall(r"\d+", bb))) for bb in bboxes_raw]
+
+    out_lines = []
+    for i, box in enumerate(bboxes):
+        if len(box) != 4:
+            continue
+        x0, y0, x1, y1 = box
+
+        if x0 >= x1 or y0 >= y1:
+            continue
+
+        sx0 = int(x0 * scale_width)
+        sy0 = int(y0 * scale_height)
+        sx1 = int(x1 * scale_width)
+        sy1 = int(y1 * scale_height)
+
+        label = lines[i] if i < len(lines) else ""
+        label = label.lstrip(", ").strip()
+
+        out_lines.append(f"{sx0},{sy0},{sx1},{sy0},{sx1},{sy1},{sx0},{sy1},{label}")
+
+    return "\n".join(out_lines)
+
+
+output_text = post_process(generated_text[0], scale_height, scale_width)
+print(output_text)  
+
+draw = ImageDraw.Draw(image)
+
+for line in output_text.strip().splitlines():
+    coords = re.findall(r"-?\d+", line)[:8]
+    if len(coords) < 8:
+        continue
+    xy = list(map(int, coords))
+    draw.polygon(xy, outline="red")
+
+image.save("output.png")
+```
+
+The image and the text (shortened here) output:
+```
+338,17,673,17,673,82,338,82,CONFIDENTIAL
+445,68,478,68,478,97,445,97,-2-
+169,129,193,129,193,157,169,157,6.
+334,129,910,129,910,157,334,157,A suggestion that the light could have been produced by
+169,150,900,150,900,177,169,177,a photo-flash from a high-flying aircraft was discounted. No aircraft
+166,171,856,171,856,198,166,198,was heard at the time and, in any case, no known photo-flash has a
+...
+```
+
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/6141a88b3a0ec78603c9e784/fmSxfYyOOhMHLt4Mw9xWg.png)
+

models/kosmos-2.5-ft/config.json

@@ -0,0 +1,46 @@
+{
+  "architectures": [
+    "Kosmos2_5ForConditionalGeneration"
+  ],
+  "dtype": "bfloat16",
+  "latent_query_num": 2048,
+  "model_type": "kosmos-2.5",
+  "text_config": {
+    "activation_dropout": 0.0,
+    "activation_function": "gelu",
+    "attention_dropout": 0.0,
+    "attention_heads": 16,
+    "dropout": 0,
+    "dtype": "bfloat16",
+    "embed_dim": 1536,
+    "ffn_dim": 6144,
+    "init_std": 0.02,
+    "layer_norm_eps": 1e-05,
+    "layerdrop": 0.0,
+    "layers": 24,
+    "max_position_embeddings": 4096,
+    "model_type": "kosmos_2_5_text_model",
+    "scale_embedding": true,
+    "use_cache": true,
+    "vocab_size": 108481
+  },
+  "transformers_version": "4.56.1",
+  "vision_config": {
+    "attention_dropout": 0.0,
+    "dense_act_fn": "gelu_new",
+    "dropout_rate": 0.0,
+    "dtype": "bfloat16",
+    "head_dim": 64,
+    "hidden_size": 1536,
+    "initializer_factor": 1.0,
+    "initializer_range": 1e-10,
+    "intermediate_size": 3968,
+    "layer_norm_eps": 1e-06,
+    "max_length": 4096,
+    "max_num_patches": 4096,
+    "model_type": "kosmos_2_5_vision_model",
+    "num_attention_heads": 24,
+    "num_hidden_layers": 18,
+    "patch_embed_hidden_size": 768
+  }
+}

models/kosmos-2.5-ft/generation_config.json

@@ -0,0 +1,7 @@
+{
+  "bos_token_id": 0,
+  "eos_token_id": 2,
+  "num_beam": 1,
+  "pad_token_id": 1,
+  "transformers_version": "4.56.1"
+}

models/kosmos-2.5-ft/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b51c3ece1fdebe5dcc63a6079ddbae5a3a8f565ad054600b406a11eaa0fc768d
+size 2749368352

models/kosmos-2.5-ft/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:72903c0342414f6579137b6bf964247c48f931461f653622ceeb02a716dd60c1
+size 5499116125

models/kosmos-2.5-ft/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a794a6cb9cd4bbd0c53d08db0e20a5536c789bba6f22113385c1c408d58908bd
+size 14645

models/kosmos-2.5-ft/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4d3e048a1e5c9dc7581e3872c3b16feadec0e02e34c6509590158830c91d1422
+size 1465

models/kosmos-2.5-ft/source.txt

@@ -0,0 +1 @@
+https://huggingface.co/merve/kosmos-2.5-ft

models/kosmos-2.5-ft/trainer_state.json

@@ -0,0 +1,111 @@
+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 2.0,
+  "eval_steps": 500,
+  "global_step": 1126,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.17777777777777778,
+      "grad_norm": 1.1875,
+      "learning_rate": 1.827402135231317e-05,
+      "loss": 0.4486,
+      "step": 100
+    },
+    {
+      "epoch": 0.35555555555555557,
+      "grad_norm": 1.21875,
+      "learning_rate": 1.6494661921708185e-05,
+      "loss": 0.1052,
+      "step": 200
+    },
+    {
+      "epoch": 0.5333333333333333,
+      "grad_norm": 0.90625,
+      "learning_rate": 1.4715302491103204e-05,
+      "loss": 0.0932,
+      "step": 300
+    },
+    {
+      "epoch": 0.7111111111111111,
+      "grad_norm": 1.3984375,
+      "learning_rate": 1.2935943060498222e-05,
+      "loss": 0.0988,
+      "step": 400
+    },
+    {
+      "epoch": 0.8888888888888888,
+      "grad_norm": 1.5703125,
+      "learning_rate": 1.1156583629893238e-05,
+      "loss": 0.0903,
+      "step": 500
+    },
+    {
+      "epoch": 1.0657777777777777,
+      "grad_norm": 1.265625,
+      "learning_rate": 9.377224199288258e-06,
+      "loss": 0.0912,
+      "step": 600
+    },
+    {
+      "epoch": 1.2435555555555555,
+      "grad_norm": 0.65234375,
+      "learning_rate": 7.597864768683275e-06,
+      "loss": 0.0798,
+      "step": 700
+    },
+    {
+      "epoch": 1.4213333333333333,
+      "grad_norm": 1.890625,
+      "learning_rate": 5.818505338078292e-06,
+      "loss": 0.0835,
+      "step": 800
+    },
+    {
+      "epoch": 1.5991111111111111,
+      "grad_norm": 2.015625,
+      "learning_rate": 4.03914590747331e-06,
+      "loss": 0.0785,
+      "step": 900
+    },
+    {
+      "epoch": 1.7768888888888887,
+      "grad_norm": 1.546875,
+      "learning_rate": 2.2597864768683274e-06,
+      "loss": 0.0785,
+      "step": 1000
+    },
+    {
+      "epoch": 1.9546666666666668,
+      "grad_norm": 2.15625,
+      "learning_rate": 4.804270462633452e-07,
+      "loss": 0.079,
+      "step": 1100
+    }
+  ],
+  "logging_steps": 100,
+  "max_steps": 1126,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 2,
+  "save_steps": 1000,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 1.1068310033650483e+17,
+  "train_batch_size": 1,
+  "trial_name": null,
+  "trial_params": null
+}

models/kosmos-2.5-ft/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:285a70ced9abc4734407c103d93dd57bdfb7ce4329159887d23b785dbb4645a4
+size 5777

models/kosmos-2.5/.gitattributes

@@ -0,0 +1,37 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+output.png filter=lfs diff=lfs merge=lfs -text
+receipt_00008.png filter=lfs diff=lfs merge=lfs -text

models/kosmos-2.5/README.md

@@ -0,0 +1,156 @@
+---
+language: en
+license: mit
+library_name: transformers
+pipeline_tag: image-text-to-text
+---
+# Kosmos-2.5
+
+[Microsoft Document AI](https://www.microsoft.com/en-us/research/project/document-ai/) | [GitHub](https://github.com/microsoft/unilm/tree/master/kosmos-2.5)
+
+## Model description
+
+Kosmos-2.5 is a multimodal literate model for machine reading of text-intensive images. Pre-trained on large-scale text-intensive images, Kosmos-2.5 excels in two distinct yet cooperative transcription tasks: (1) generating spatially-aware text blocks, where each block of text is assigned its spatial coordinates within the image, and (2) producing structured text output that captures styles and structures into the markdown format. This unified multimodal literate capability is achieved through a shared decoder-only auto-regressive Transformer architecture, task-specific prompts, and flexible text representations. We evaluate Kosmos-2.5 on end-to-end document-level text recognition and image-to-markdown text generation. Furthermore, the model can be readily adapted for any text-intensive image understanding task with different prompts through supervised fine-tuning, making it a general-purpose tool for real-world applications involving text-rich images. This work also paves the way for the future scaling of multimodal large language models.
+
+[Kosmos-2.5: A Multimodal Literate Model](https://arxiv.org/abs/2309.11419)
+
+## NOTE:
+Since this is a generative model, there is a risk of **hallucination** during the generation process, and it **CAN NOT** guarantee the accuracy of all OCR/Markdown results in the images.
+
+## Inference
+
+KOSMOS-2.5 is supported from Transformers >= 4.56. Find the docs [here](https://huggingface.co/docs/transformers/main/en/model_doc/kosmos2_5).
+
+**Markdown Task:** For usage instructions, please refer to [md.py](md.py).
+
+```py
+import re
+import torch
+import requests
+from PIL import Image, ImageDraw
+from transformers import AutoProcessor, Kosmos2_5ForConditionalGeneration, infer_device
+
+repo = "microsoft/kosmos-2.5"
+device = "cuda:0"
+dtype = torch.bfloat16
+model = Kosmos2_5ForConditionalGeneration.from_pretrained(repo, device_map=device, dtype=dtype)
+processor = AutoProcessor.from_pretrained(repo)
+
+# sample image
+url = "https://huggingface.co/microsoft/kosmos-2.5/resolve/main/receipt_00008.png"
+image = Image.open(requests.get(url, stream=True).raw)
+
+prompt = "<md>"
+inputs = processor(text=prompt, images=image, return_tensors="pt")
+
+height, width = inputs.pop("height"), inputs.pop("width")
+raw_width, raw_height = image.size
+scale_height = raw_height / height
+scale_width = raw_width / width
+
+inputs = {k: v.to(device) if v is not None else None for k, v in inputs.items()}
+inputs["flattened_patches"] = inputs["flattened_patches"].to(dtype)
+generated_ids = model.generate(
+    **inputs,
+    max_new_tokens=1024,
+)
+
+generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)
+print(generated_text[0])
+```
+
+**OCR Task:** For usage instructions, please refer to [ocr.py](ocr.py).
+
+```py
+import re
+import torch
+import requests
+from PIL import Image, ImageDraw
+from transformers import AutoProcessor, Kosmos2_5ForConditionalGeneration, infer_device
+
+repo = "microsoft/kosmos-2.5"
+device = "cuda:0"
+dtype = torch.bfloat16
+model = Kosmos2_5ForConditionalGeneration.from_pretrained(repo, device_map=device, dtype=dtype)
+processor = AutoProcessor.from_pretrained(repo)
+
+# sample image
+url = "https://huggingface.co/microsoft/kosmos-2.5/resolve/main/receipt_00008.png"
+image = Image.open(requests.get(url, stream=True).raw)
+
+# bs = 1
+prompt = "<ocr>"
+inputs = processor(text=prompt, images=image, return_tensors="pt")
+height, width = inputs.pop("height"), inputs.pop("width")
+raw_width, raw_height = image.size
+scale_height = raw_height / height
+scale_width = raw_width / width
+
+# bs > 1, batch generation
+# inputs = processor(text=[prompt, prompt], images=[image,image], return_tensors="pt")
+# height, width = inputs.pop("height"), inputs.pop("width")
+# raw_width, raw_height = image.size
+# scale_height = raw_height / height[0]
+# scale_width = raw_width / width[0]
+
+inputs = {k: v.to(device) if v is not None else None for k, v in inputs.items()}
+inputs["flattened_patches"] = inputs["flattened_patches"].to(dtype)
+generated_ids = model.generate(
+    **inputs,
+    max_new_tokens=1024,
+)
+
+generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)
+def post_process(y, scale_height, scale_width):
+    y = y.replace(prompt, "")
+    if "<md>" in prompt:
+        return y
+    pattern = r"<bbox><x_\d+><y_\d+><x_\d+><y_\d+></bbox>"
+    bboxs_raw = re.findall(pattern, y)
+    lines = re.split(pattern, y)[1:]
+    bboxs = [re.findall(r"\d+", i) for i in bboxs_raw]
+    bboxs = [[int(j) for j in i] for i in bboxs]
+    info = ""
+    for i in range(len(lines)):
+        box = bboxs[i]
+        x0, y0, x1, y1 = box
+        if not (x0 >= x1 or y0 >= y1):
+            x0 = int(x0 * scale_width)
+            y0 = int(y0 * scale_height)
+            x1 = int(x1 * scale_width)
+            y1 = int(y1 * scale_height)
+            info += f"{x0},{y0},{x1},{y0},{x1},{y1},{x0},{y1},{lines[i]}"
+    return info
+
+output_text = post_process(generated_text[0], scale_height, scale_width)
+print(output_text)
+
+draw = ImageDraw.Draw(image)
+lines = output_text.split("\n")
+for line in lines:
+    # draw the bounding box
+    line = list(line.split(","))
+    if len(line) < 8:
+        continue
+    line = list(map(int, line[:8]))
+    draw.polygon(line, outline="red")
+image.save("output.png")
+```
+
+## Citation
+
+If you find Kosmos-2.5 useful in your research, please cite the following paper:
+
+```
+@article{lv2023kosmos,
+  title={Kosmos-2.5: A multimodal literate model},
+  author={Lv, Tengchao and Huang, Yupan and Chen, Jingye and Cui, Lei and Ma, Shuming and Chang, Yaoyao and Huang, Shaohan and Wang, Wenhui and Dong, Li and Luo, Weiyao and others},
+  journal={arXiv preprint arXiv:2309.11419},
+  year={2023}
+}
+```
+
+## License
+The content of this project itself is licensed under the [MIT](https://github.com/microsoft/unilm/blob/master/kosmos-2.5/LICENSE)
+
+[Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct)

models/kosmos-2.5/ckpt.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:676d0efd1dddf3785644918dd598d7734f9ed6e3eb59f806299ca8b7aefa0967
+size 6165757107