| --- |
| license: apple-amlr |
| license_link: https://huggingface.co/apple/LensVLM-9B/blob/main/LICENSE |
| base_model_relation: quantized |
| language: en |
| library_name: mlx |
| pipeline_tag: image-text-to-text |
| tags: |
| - mlx |
| - text-generation-inference |
| - apple-silicon |
| - 4-bit |
| - 8-bit |
| - mlx-vlm |
| base_model: |
| - apple/LensVLM-9B |
| --- |
| |
| # **LensVLM-9B-MLX** |
|
|
| > **LensVLM-9B** is a 9-billion-parameter vision-language model from Apple, built on Qwen3.5-9B, introduced in the paper "LensVLM: Selective Context Expansion for Compressed Visual Representation of Text." Its core mechanism scans a compressed image representation of text — at configurable compression ratios of 5x, 10x, or 15x — and then selectively expands only the pages relevant to a given question back to their uncompressed form via learned tools, allowing the model to process very long documents without holding the entire uncompressed text in context. It's run via the accompanying `ml-lensvlm` codebase with a simple demo script accepting a text file and a question, and is released under the Apple Machine Learning Research Model License (with the accompanying source code separately licensed under the Apple Sample Code License). |
|
|
| ```plaintext |
| +-- prithivMLmods/LensVLM-9B-MLX (main) |
| +-- / (Root: bf16) |
| +-- 4bit/ (Quantized: 4-bit) |
| +-- 8bit/ (Quantized: 8-bit) |
| ``` |
|
|
| ## Use with mlx |
|
|
| Install the required library: |
|
|
| ```bash |
| pip install -U mlx-vlm |
| ``` |
|
|
| > **Model Note:** `LensVLM-9B` is a 9-billion parameter vision-language model developed by Apple, specialized in compressed visual text processing and document-level multi-modal understanding. |
|
|
| ### BF16 Variant (Base Weights) |
|
|
| The unquantized BF16 weights reside directly in the root repository directory: |
|
|
| #### CLI (Terminal) |
|
|
| ```bash |
| python -m mlx_vlm generate \ |
| --model prithivMLmods/LensVLM-9B-MLX \ |
| --max-tokens 512 \ |
| --temperature 0.0 \ |
| --prompt "Read and analyze the contents of this document in detail." \ |
| --image <path_to_image> |
| ``` |
|
|
| #### Python API |
|
|
| ```python |
| from mlx_vlm import load, generate |
| from mlx_vlm.prompt_utils import apply_chat_template |
| from mlx_vlm.utils import load_config |
| |
| model_path = "prithivMLmods/LensVLM-9B-MLX" |
| model, processor = load(model_path) |
| config = load_config(model_path) |
| |
| image = ["<path_to_image>"] |
| prompt = "Read and analyze the contents of this document in detail." |
| formatted_prompt = apply_chat_template(processor, config, prompt, num_images=len(image)) |
| |
| output = generate( |
| model, |
| processor, |
| formatted_prompt, |
| image=image, |
| max_tokens=512, |
| temperature=0.0 |
| ) |
| print(output.text) |
| ``` |
|
|
| ### 8-bit Variant |
|
|
| Target the `8bit` subfolder: |
|
|
| #### CLI (Terminal) |
|
|
| ```bash |
| python -m mlx_vlm generate \ |
| --model prithivMLmods/LensVLM-9B-MLX/8bit \ |
| --max-tokens 512 \ |
| --temperature 0.0 \ |
| --prompt "Read and analyze the contents of this document in detail." \ |
| --image <path_to_image> |
| ``` |
|
|
| #### Python API |
|
|
| ```python |
| from mlx_vlm import load, generate |
| from mlx_vlm.prompt_utils import apply_chat_template |
| from mlx_vlm.utils import load_config |
| |
| model_path = "prithivMLmods/LensVLM-9B-MLX" |
| model, processor = load(model_path, subfolder="8bit") |
| config = load_config(model_path, subfolder="8bit") |
| |
| image = ["<path_to_image>"] |
| prompt = "Read and analyze the contents of this document in detail." |
| formatted_prompt = apply_chat_template(processor, config, prompt, num_images=len(image)) |
| |
| output = generate( |
| model, |
| processor, |
| formatted_prompt, |
| image=image, |
| max_tokens=512, |
| temperature=0.0 |
| ) |
| print(output.text) |
| ``` |
|
|
| ### 4-bit Variant |
|
|
| Target the `4bit` subfolder: |
|
|
| #### CLI (Terminal) |
|
|
| ```bash |
| python -m mlx_vlm generate \ |
| --model prithivMLmods/LensVLM-9B-MLX/4bit \ |
| --max-tokens 512 \ |
| --temperature 0.0 \ |
| --prompt "Read and analyze the contents of this document in detail." \ |
| --image <path_to_image> |
| ``` |
|
|
| #### Python API |
|
|
| ```python |
| from mlx_vlm import load, generate |
| from mlx_vlm.prompt_utils import apply_chat_template |
| from mlx_vlm.utils import load_config |
| |
| model_path = "prithivMLmods/LensVLM-9B-MLX" |
| model, processor = load(model_path, subfolder="4bit") |
| config = load_config(model_path, subfolder="4bit") |
| |
| image = ["<path_to_image>"] |
| prompt = "Read and analyze the contents of this document in detail." |
| formatted_prompt = apply_chat_template(processor, config, prompt, num_images=len(image)) |
| |
| output = generate( |
| model, |
| processor, |
| formatted_prompt, |
| image=image, |
| max_tokens=512, |
| temperature=0.0 |
| ) |
| print(output.text) |
| ``` |
|
|
| ## License and Attribution |
|
|
| * **Model:** [apple/LensVLM-9B](https://huggingface.co/apple/LensVLM-9B) |
| * **Base Model:** [Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B) |
| * **License:** This MLX compression follows the [Apple AMLR License](https://huggingface.co/apple/LensVLM-9B/blob/main/LICENSE) of the original LensVLM-9B model. |
| * **MLX-VLM:** https://github.com/Blaizzy/mlx-vlm |
| * **MLX:** https://github.com/ml-explore/mlx |
