Create README.md

README.md

@@ -0,0 +1,181 @@
+---
+license: apache-2.0
+datasets:
+- allenai/Molmo2-VideoPoint
+language:
+- en
+base_model:
+- google/siglip-so400m-patch14-384
+- Qwen/Qwen3-4B-Instruct-2507
+pipeline_tag: video-text-to-text
+library_name: transformers
+tags:
+- multimodal
+- olmo
+- molmo
+- molmo2
+---
+
+<img src="molmo_2_logo_RGB.png" alt="Logo for the Molmo2 Project" style="width: auto; height: 50px;">
+
+# Molmo2-4B
+
+Molmo2 is a family of open vision-language models developed by the Allen Institute for AI (Ai2) that support image, video and multi-image understanding and grounding.
+Molmo2 models are trained on publicly available third party datasets as referenced in [our technical report](https://allenai.org/papers/molmo2) and [Molmo2 data](https://huggingface.co/collections/allenai/molmo2-data), 
+a collection of datasets with highly-curated image-text and video-text pairs.
+It has state-of-the-art performance among multimodal models with a similar size.
+You can find all models in the Molmo2 family [here](https://huggingface.co/collections/allenai/molmo2).
+
+**Learn more** about the Molmo2 family [in our announcement blog post](https://allenai.org/blog/molmo2).
+
+Molmo2-VideoPoint-4B is based on [Qwen3-4B-Instruct](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507) and uses [SigLIP 2](https://huggingface.co/google/siglip-so400m-patch14-384) as vision backbone.
+It is mostly trained on the Molmo2-VideoPoint data only and meant to be used for video pointing and counting only. 
+
+Ai2 is commited to open science. The Molmo2 datasets are available [here](https://huggingface.co/collections/allenai/molmo2-data). 
+All other artifacts used in creating Molmo2 (training code, evaluations, intermediate checkpoints) will be made available at a later date, furthering our commitment to open-source AI development and reproducibility.
+
+Quick links:
+- 📂 [All Models](https://huggingface.co/collections/allenai/molmo2)
+- 📃 [Paper](https://allenai.org/papers/molmo2)
+- 🎥 [Blog with Videos](https://allenai.org/blog/molmo2)
+
+## Quick Start
+
+### Setup Conda Environment
+```
+conda create --name transformers4571 python=3.11
+conda activate transformers4571
+pip install transformers==4.57.1
+pip install torch pillow einops torchvision accelerate decord2 molmo_utils
+```
+
+### Pointing Video QA
+
+```
+from transformers import AutoProcessor, AutoModelForImageTextToText
+import torch
+from molmo_utils import process_vision_info
+import re
+
+model_id="allenai/Molmo2-4B"
+
+# load the processor
+processor = AutoProcessor.from_pretrained(
+    model_id,
+    trust_remote_code=True,
+    dtype="auto",
+    device_map="auto"
+)
+
+# load the model
+model = AutoModelForImageTextToText.from_pretrained(
+    model_id,
+    trust_remote_code=True,
+    dtype="auto",
+    device_map="auto"
+)
+
+COORD_REGEX = re.compile(rf"<(?:points|tracks).*? coords=\"([0-9\t:;, .]+)\"/?>")
+FRAME_REGEX = re.compile(rf"(?:^|\t|:|,|;)([0-9\.]+) ([0-9\. ]+)")
+POINTS_REGEX = re.compile(r"([0-9]+) ([0-9]{3,4}) ([0-9]{3,4})")
+
+def _points_from_num_str(text, image_w, image_h, extract_ids=False):
+    all_points = []
+    for points in POINTS_REGEX.finditer(text):
+        ix, x, y = points.group(1), points.group(2), points.group(3)
+        # our points format assume coordinates are scaled by 1000
+        x, y = float(x)/1000*image_w, float(y)/1000*image_h
+        if 0 <= x <= image_w and 0 <= y <= image_h:
+            yield ix, x, y
+
+
+def extract_video_points(text, image_w, image_h, extract_ids=False):
+    """Extract video pointing coordinates as a flattened list of (t, x, y) triplets from model output text."""
+    all_points = []
+    for coord in COORD_REGEX.finditer(text):
+        for point_grp in FRAME_REGEX.finditer(coord.group(1)):
+            frame_id = float(point_grp.group(1))
+            w, h = (image_w, image_h)
+            for idx, x, y in _points_from_num_str(point_grp.group(2), w, h):
+                if extract_ids:
+                    all_points.append((frame_id, idx, x, y))
+                else:
+                    all_points.append((frame_id, x, y))
+    return all_points
+
+messages = [
+    {
+        "role": "user",
+        "content": [
+            dict(type="text", text="Point to the penguins."),
+            dict(type="video", video="https://storage.googleapis.com/oe-training-public/demo_videos/many_penguins.mp4"),
+        ],
+    }
+]
+
+# process the video using `molmo_utils.process_vision_info`
+_, videos, video_kwargs = process_vision_info(messages)
+videos, video_metadatas = zip(*videos)
+videos, video_metadatas = list(videos), list(video_metadatas)
+
+# apply the chat template to the input messages
+text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+
+# process the video and text
+inputs = processor(
+    videos=videos,
+    video_metadata=video_metadatas,
+    text=text,
+    padding=True,
+    return_tensors="pt",
+    **video_kwargs,
+)
+
+inputs = {k: v.to(model.device) for k, v in inputs.items()}
+
+# generate output
+with torch.inference_mode():
+    generated_ids = model.generate(**inputs, max_new_tokens=2048)
+
+# only get generated tokens; decode them to text
+generated_tokens = generated_ids[0, inputs['input_ids'].size(1):]
+generated_text = processor.tokenizer.decode(generated_tokens, skip_special_tokens=True)
+
+# decode video pointing outputs
+points = extract_video_points(generated_text, image_w=video_metadatas[0]["width"], image_h=video_metadatas[0]["height"])
+print(points)
+```
+
+## Evaluations
+
+We report the Average Score on 15 Academic Benchmarks here.
+For details on the evals, refer to the main video results table in our [technical report](https://allenai.org/papers/molmo2).
+
+| Model | Average Score on 15 Academic Benchmarks |
+|-----------------------------|-----------------------------------------|
+| GPT-5 | 70.6 |
+| GPT-5 mini | 65.0 |
+| Gemini 3 Pro | 70.0 |
+| Gemini 2.5 Pro | 71.2 |
+| Gemini 2.5 Flash | 66.7 |
+| Claude Sonnet 4.5 | 59.6 |
+| InternVL3.5-4B | 53.4 |
+| InternVL3.5-8B | 54.1 |
+| Qwen3-VL-4B | 58.1 |
+| Qwen3-VL-8B | 59.5 |
+| Keye-VL-1.5-8B | 55.7 |
+| GLM-4.1V-9B | 56.9 |
+| MiniCPM-V-4.5-8B | 56.6 |
+| Eagle2.5-8B | 60.7 |
+| PLM-3B | 53.9 |
+| PLM-8B | 56.2 |
+| LLaVA-Video-7B | 52.7 |
+| VideoChat-Flash-7B | 56.1 |
+| **Molmo2-4B (this model)** | 62.8 |
+| Molmo2-8B | 63.1 |
+| Molmo2-7B | 59.7 |
+
+## License and Use
+
+This model is licensed under Apache 2.0. It is intended for research and educational use in accordance with Ai2’s [Responsible Use Guidelines](https://allenai.org/responsible-use).
+This model is trained on third party datasets that are subject to academic and non-commercial research use only. Please review the sources to determine if this model is appropriate for your use case.