Model card: redact conversion recipe, correct sizes/usage, add measured example

- Conversion section generalised; exact key remaps, failing handler, config reconstruction and convert command removed
- Sizes corrected: weights 5.64 GB / repo 5.65 GB (was a single '~5.3 GB')
- Usage fixed: generate() returns GenerationResult -> print(result.text)
- Versions: verified on mlx-vlm 0.6.8 (was an unverified '0.6.3 or later')
- Added measured example on CC0 test footage + note on packed 4-bit param count
- base_model_relation: quantized set explicitly

README.md

@@ -1,6 +1,7 @@
 ---
 license: apache-2.0
 base_model: omni-research/Tarsier2-Recap-7b
+base_model_relation: quantized
 pipeline_tag: video-text-to-text
 library_name: mlx
 language:
@@ -11,37 +12,42 @@ tags:
 - video-captioning
 - qwen2-vl
 - tarsier
+- 4-bit
 ---
 
 # Tarsier2-Recap-7b-MLX-4bit
 
 A 4-bit [MLX](https://github.com/ml-explore/mlx) conversion of
 [omni-research/Tarsier2-Recap-7b](https://huggingface.co/omni-research/Tarsier2-Recap-7b),
-prepared for fast, native inference on Apple Silicon.
+prepared for native inference on Apple Silicon.
 
-Tarsier2 is among the most detailed open video captioners available, yet no
-official MLX build exists. This repository provides one: the same model,
-quantized to 4-bit (~5.3 GB) and accelerated on the Mac GPU through MLX.
+Tarsier2-Recap-7b produces unusually specific, grounded video descriptions — it
+tends to name concrete details rather than summarise a scene generically. No
+official MLX build exists. This repository provides one: the same weights,
+quantized to 4-bit and run on the Mac GPU through MLX.
+
+> The underlying model is in the Qwen2-VL 7B class. Hugging Face may display a
+> lower automatic parameter count (~1.9 B) because MLX 4-bit weights are packed
+> into 32-bit integers.
 
 - **Base model:** omni-research/Tarsier2-Recap-7b (Apache-2.0)
 - **Format:** MLX, 4-bit affine quantization (group size 64)
-- **Size on disk:** ~5.3 GB
-- **Underlying architecture:** Qwen2-VL 7B (see conversion notes below)
-- **Tested on:** Apple M4 Pro, 24 GB unified memory
+- **Model weights:** approximately 5.64 GB / 5.25 GiB (two safetensors shards)
+- **Complete repository:** approximately 5.65 GB / 5.26 GiB
+- **Underlying architecture:** Qwen2-VL 7B (see conversion notes)
+- **Not retrained or fine-tuned** — format conversion and quantization only
 
-## Purpose
+## Requirements
 
-Tarsier2-Recap-7b produces unusually specific, grounded video descriptions. It
-identifies small details — a sail number, an object held in a subject's hand,
-architectural features — rather than generic scene summaries. On Apple Silicon,
-the practical way to run a 7B vision-language model at usable speed is MLX
-4-bit; the full bf16 weights under `transformers` on MPS are prohibitively slow
-(hundreds of seconds per clip). This conversion reduces inference to
-approximately **7 seconds per short clip** on an M4 Pro.
+Verified with [`mlx-vlm`](https://github.com/Blaizzy/mlx-vlm) 0.6.8 on macOS
+15.7.2, Python 3.12, `mlx` 0.32.0.
 
-## Usage
+```bash
+pip install mlx-vlm==0.6.8
+brew install ffmpeg      # for extracting frames
+```
 
-Run with [`mlx-vlm`](https://github.com/Blaizzy/mlx-vlm) (0.6.3 or later).
+## Usage
 
 ```python
 from mlx_vlm import load, generate
@@ -50,83 +56,115 @@ from mlx_vlm.prompt_utils import apply_chat_template
 model, processor = load("viavicdev/Tarsier2-Recap-7b-MLX-4bit")
 
 # Provide the video as a list of extracted frame images (see note 1)
-frames = ["frame_00.jpg", "frame_01.jpg", "frame_02.jpg", "frame_03.jpg"]
+frames = ["frame_01.jpg", "frame_02.jpg", "frame_03.jpg", "frame_04.jpg"]
+
 prompt = apply_chat_template(
     processor, model.config,
     "Describe this video in detail.",
-    num_images=len(frames),
+    num_images=len(frames),          # must match len(frames)
 )
-out = generate(
+
+result = generate(
     model, processor, prompt,
-    image=frames,                 # parameter is image= (not images=)
+    image=frames,                    # parameter is image= (not images=)
     max_tokens=300,
     temperature=0.3,
-    repetition_penalty=1.3,       # see note 3
+    repetition_penalty=1.3,          # see note 3
     repetition_context_size=40,
 )
-print(out)
+print(result.text)                   # .text — generate() returns a GenerationResult
+```
+
+Extract frames with ffmpeg, for example eight evenly spaced frames from a clip
+of known duration `D`:
+
+```bash
+ffmpeg -i clip.mp4 -vf "fps=8/D,scale=-2:448" -frames:v 8 -q:v 2 frame_%02d.jpg
 ```
 
 ### Practical notes
 
-1. **Provide frames as images rather than a video file.** The `mlx-vlm` 0.6.3
-   video decoder path is unreliable and may produce hallucinated output. Extract
-   frames with ffmpeg and pass them as a list of images (`image=[...]` with
-   `num_images=N`); this delivers the actual frames to the model.
+1. **Pass frames as images, not a video file.** The `mlx-vlm` video decoder path
+   is unreliable for this model and can produce output unrelated to the clip.
+   Extracting frames with ffmpeg and passing them as a list (`image=[...]` with
+   a matching `num_images`) delivers the actual frames to the model.
 2. **The chat template is included** in this repository (`chat_template.json`
-   and `chat_template.jinja`). Some MLX repackings omit it, which causes
-   `apply_chat_template` to fail.
+   and `chat_template.jinja`). Some MLX repackings omit it, which makes
+   `apply_chat_template` fail.
 3. **Apply a repetition penalty.** At 4-bit precision the model can occasionally
-   loop. The settings `repetition_penalty=1.3`, `repetition_context_size=40`,
-   and `temperature=0.3` keep the output stable. In internal testing, 1 of 22
-   clips degenerated without a penalty and none did with it.
+   loop. `repetition_penalty=1.3`, `repetition_context_size=40` and
+   `temperature=0.3` kept output stable in our testing; in a small internal set
+   of 22 short clips, one degenerated without a penalty and none did with it.
+4. **Runtime scales with frame resolution and frame count**, not with the
+   duration of the source clip. See the measurements below.
 
-## Conversion notes
+## Example
 
-Tarsier2 is distributed as `TarsierForConditionalGeneration` with
-`model_type: llava`, but its text and vision towers are Qwen2-VL weights under
-different key names. The `mlx-vlm` llava handler does not support Qwen2-VL's
-mRoPE (it raises `rope_scaling type only supports linear`), whereas the
-`qwen2_vl` handler does. The conversion therefore consists of a key-remap into
-the native Qwen2-VL layout, followed by a standard MLX quantization.
+A single measured run, not a benchmark.
 
-Only three key prefixes require remapping:
+- **Input:** 8 frames (796×448) from a 10.7-second clip —
+  [*Trains – Mini World Lyon*](https://commons.wikimedia.org/wiki/File:Trains_-_Mini_World_Lyon.webm)
+  by Benoît Prieur, Wikimedia Commons, CC0
+- **Hardware:** Apple M4 Pro, 24 GB unified memory, macOS 15.7.2
+- **Versions:** Python 3.12.12, `mlx` 0.32.0, `mlx-vlm` 0.6.8
+- **Model load (cold, excluded from generation timing):** ≈14 s
+- **Generation:** median **15.8 s** over 3 warm runs (15.3 / 15.8 / 16.1)
+- **Peak memory:** ≈6.8 GB
+- **Settings:** `max_tokens=300`, as in the usage example above
 
-```
-language_model.model.*    ->  model.*
-language_model.lm_head.*  ->  lm_head.*
-vision_tower.*            ->  visual.*
-```
+Output (run 2 of 3, verbatim):
 
-A flat `qwen2_vl` configuration is then built from Tarsier's `text_config` and
-`vision_config` (preserving `rope_scaling: mrope`, the vision/image/video token
-ids, and related fields), and the remapped weights are quantized:
+> A model train, consisting of a green engine and several red freight cars,
+> moves along the tracks from left to right. The background features a large
+> crowd of people gathered in an open area with numerous colorful cars parked
+> and displayed. The train passes by the car display area multiple times, moving
+> smoothly along the tracks.
 
-```bash
-mlx_vlm.convert --hf-path ./tarsier2-qwen2vl-hf -q --q-bits 4
-```
+For comparison, 8 frames at 252×448 (a vertical clip) on the same machine ran in
+a median of **5.5 s**. Frame resolution dominates runtime — budget accordingly.
+
+## Conversion notes
 
-This reflects a general pattern: custom Qwen2-VL-based models such as Tarsier,
-which lack an official MLX or HF remote-code path, can often be run on MLX
-through a key-remap, provided they are routed through the `qwen2_vl` handler so
-that mRoPE is respected.
+The original checkpoint could not be converted directly with the standard
+`mlx-vlm` workflow, because its published architecture metadata does not fully
+match the underlying multimodal model structure.
+
+For this release, the checkpoint was adapted to a compatible native MLX layout
+before 4-bit quantization. This required resolving differences in the
+vision-language architecture and in the positional encoding configuration.
+
+The resulting checkpoint runs through the standard native inference path in
+`mlx-vlm`. The conversion changes the storage and runtime format only; the model
+was not retrained or fine-tuned.
+
+The complete conversion procedure is not included in this repository.
 
 ## Limitations
 
-- 4-bit quantization trades some fidelity for reduced size and higher speed.
+- **4-bit quantization trades some fidelity** for reduced size and higher speed.
   For maximum quality, use the original bf16 weights on a CUDA device.
-- This is an English captioning model; it describes in English regardless of any
-  spoken language in the clip.
-- Vision only (no audio). It captions what is seen, not what is heard.
-- It follows free-form description more reliably than rigid output schemas, and
-  is best used as a captioner rather than a strict classifier.
+- **English only.** It describes in English regardless of any spoken language in
+  the clip.
+- **Vision only (no audio).** It captions what is seen, not what is heard.
+- **Free-form description over rigid schemas.** It follows open-ended captioning
+  prompts more reliably than strict output formats, and is better used as a
+  captioner than as a classifier.
+- **Detail depends on the source.** On low-resolution or distant subjects it
+  describes what is visible at that scale and may not identify the subject
+  specifically.
+- **Not systematically evaluated here.** For accuracy figures, see the base
+  model's card and paper. The timings above are single measurements on one
+  machine.
 
 ## License and attribution
 
-This conversion inherits the **Apache-2.0** license of the base model. All
-credit for the model itself belongs to the Tarsier2 authors
-([omni-research/Tarsier2-Recap-7b](https://huggingface.co/omni-research/Tarsier2-Recap-7b)).
-This repository provides only a format conversion — a weight key-remap and MLX
-4-bit quantization — for the Apple Silicon community.
+This conversion inherits the **Apache-2.0** license of the base model.
+
+All credit for the original model, training and weights belongs to the Tarsier2
+authors ([omni-research/Tarsier2-Recap-7b](https://huggingface.co/omni-research/Tarsier2-Recap-7b)).
+
+This repository provides the MLX format conversion, 4-bit quantization,
+packaging, Apple Silicon compatibility testing and usage documentation. The
+model was not retrained or fine-tuned.
 
-When citing Tarsier2, please cite the original authors.
+When citing Tarsier2, cite the original authors.