Upload folder using huggingface_hub

ca7a1ba verified 17 days ago

4.54 kB

license: apache-2.0
base_model: qvac/VisionPsy-Nano-460M
tags:
  - mlx
  - vlm
  - vision-language-model
  - apple-silicon
  - siglip2
  - smollm2
language:
  - en
  - zh
library_name: mlx
pipeline_tag: image-text-to-text

VisionPsy-Nano-460M-MLX

MLX port of qvac/VisionPsy-Nano-460M, a compact 460M-parameter vision-language model from Tether AI Research, converted to run natively on Apple Silicon.

Architecture: SigLIP2-base-patch16-512 vision encoder + pixel-shuffle modality projector + SmolLM2-360M-Instruct decoder
Parameters: ~460M
Precision: bfloat16 (~1.0 GB on disk, down from 2.0 GB fp32)
Runtime: MLX on Apple Silicon (M-series)
License: Apache-2.0

Benchmarks (MLX bf16, M-series)

Measured across 7 images x 5 prompts, 64 max new tokens, greedy decode:

Metric	Standard	Flash
Avg decode tok/s	99	152
Median decode tok/s	90	157
Avg peak GPU memory	2.64 GB	2.64 GB
Load time	~0.4 s	~0.7 s

Per-prompt-type medians (Standard):

Prompt type	Median tok/s	Example
Describe (EN, 1 sentence)	158.7	"A smiling man in a white lab coat gestures with his right hand..."
What text appears?	90.3	"OICOMELVANG"
Count objects/people	40.0	"There are 3 people in the image."
Main subject	59.2	"The main subject is a man wearing a white lab coat."
Describe (ZH)	130.4	"他說："OICOMELVANG, 25158""

Full 70-run matrix (Standard + Flash) is at github.com/KaedeTai/mlx-video/tree/visionpsy-mlx-port.

Usage

This repo uses the mlx-vlm-style layout (text_config + vision_config top-level, language_model.* / vision_tower.* / multi_modal_projector.* tensor prefixes) so it slots cleanly into mlx-vlm once a visionpsy_nano handler lands there. Until then, load it via the MLX port bundled in mlx-video (branch visionpsy-mlx-port):

pip install mlx safetensors transformers pillow
git clone -b visionpsy-mlx-port https://github.com/KaedeTai/mlx-video.git
cd mlx-video

from huggingface_hub import snapshot_download
from mlx_video.models.visionpsy_nano import load_visionpsy_nano
from mlx_video.models.visionpsy_nano.processor import load_processor
from PIL import Image

# Snapshot from HF (or point at your local folder)
path = snapshot_download("KaedeTai/VisionPsy-Nano-460M-MLX")

model, cfg = load_visionpsy_nano(path)
proc = load_processor(path, cfg=cfg)

img = Image.open("photo.jpg").convert("RGB")
batch = proc("Describe this image in one sentence.", image=img)

tokens = list(model.generate(
    batch["input_ids"],
    pixel_values=batch["pixel_values"],
    image_token_id=batch["image_token_id"],
    max_new_tokens=64,
    eos_token_id=proc.tokenizer.eos_token_id,
))
print(proc.decode(tokens, skip_special_tokens=True))

Note: the port's load_visionpsy_nano reads the repacked config via a compat shim; the original _original_config block is retained inside config.json for round-tripping.

What changed vs the original

fp32 -> bf16. Weights cast to bfloat16. Outputs verified byte-identical on greedy decode for both variants.
Prefix rename. Tensor names moved from decoder.* / vision_encoder.* / MP.* to language_model.* / vision_tower.* / multi_modal_projector.* to match mlx-vlm conventions.
Config reshape. Flat lm_* / vit_* keys refactored into nested text_config / vision_config blocks with standard field names (hidden_size, num_hidden_layers, etc.).
Stale buffers dropped. decoder.rotary_embd.* buffers removed — MLX's nn.RoPE computes frequencies on the fly.

Attribution

Original model: Tether AI Research / QVAC — qvac/VisionPsy-Nano-460M (Apache-2.0)
MLX port + weight repack: KaedeTai
Base components: SigLIP2 (Google), SmolLM2 (Hugging Face)

If you use these weights, please also cite the original QVAC release and the base model authors.

KaedeTai
/

VisionPsy-Nano-460M-MLX

VisionPsy-Nano-460M-MLX

Benchmarks (MLX bf16, M-series)

Usage

What changed vs the original

Attribution

See also