| --- |
| license: apache-2.0 |
| library_name: mlx |
| pipeline_tag: image-text-to-text |
| base_model: mlx-community/Qwen3.5-0.8B-OptiQ-4bit |
| tags: |
| - mlx |
| - optiq |
| - vision |
| - vlm |
| - lora |
| - chartqa |
| - apple-silicon |
| --- |
| |
| # chartreader-0.8B-OptiQ-4bit |
|
|
| **A 0.8B vision-language model that reads charts, fine-tuned on a Mac.** |
|
|
| This is the OptiQ 4-bit quant of `Qwen3.5-0.8B` bundled with **ChartReader**, an |
| image+text LoRA trained on ChartQA. The vision tower stays frozen; the LoRA |
| adapts the language tower to answer chart questions the way ChartQA wants: |
| short, exact, just the value. The base + sidecar keep full image+text |
| inference, so the same repo works for general VLM tasks too. |
|
|
| ## Results |
|
|
| 80 held-out ChartQA questions, base vs. base + ChartReader, images letterboxed |
| to a 512 px canvas. |
|
|
| | Metric | Base 0.8B | + ChartReader | Δ | |
| |---|---|---|---| |
| | Relaxed accuracy | 50.0% | 55.0% | +5.0 pp | |
| | **Exact match** | 26.2% | **40.0%** | **+13.8 pp** | |
| | Output similarity | 0.385 | 0.598 | +0.21 | |
|
|
| The base reads charts loosely and verbosely ("There are 10 food items shown in |
| the bar graph"); ChartReader answers concisely ("3"). The big win is format and |
| exact-match; the relaxed-accuracy gain is smaller but real. |
|
|
| ## Files |
|
|
| ``` |
| config.json, model.safetensors the base Qwen3.5-0.8B OptiQ-4bit quant |
| optiq_vision.safetensors vision sidecar (full image+text inference) |
| mtp.safetensors multi-token-prediction draft head |
| adapters/chartreader/ the ChartQA LoRA (adapters.safetensors) |
| ``` |
|
|
| The LoRA is **not merged into the weights** — it rides alongside and applies at |
| serve time, so you keep the plain base quant plus the ChartReader behavior. |
|
|
| ## Use |
|
|
| ```bash |
| pip install mlx-optiq |
| huggingface-cli download mlx-community/chartreader-0.8B-OptiQ-4bit --local-dir ./chartreader |
| |
| optiq serve --model ./chartreader --adapter ./chartreader/adapters/chartreader |
| ``` |
|
|
| Then send an image + a chart question to the OpenAI-compatible endpoint on |
| `localhost:8080`. Without the `--adapter`, the same repo serves as the plain |
| image+text base model. |
|
|
| ## How it was made |
|
|
| Image+text LoRA on the language tower, vision tower frozen: |
|
|
| ```bash |
| optiq lora train mlx-community/Qwen3.5-0.8B-OptiQ-4bit \ |
| --vision --data ./chartqa/train.jsonl \ |
| --rank 8 --iters 800 --learning-rate 5e-5 --output ./chartreader |
| ``` |
|
|
| Trained on a 24 GB Apple Silicon Mac. Every image is letterboxed to a uniform |
| 512 px canvas (uniform shape keeps training memory bounded), gradient |
| checkpointing fits the hybrid gated-delta backward, and gradient clipping + |
| a 5e-5 learning rate prevent the mode collapse short targets otherwise cause. |
| The whole flow — build the dataset, run the LoRA — is also available in the |
| OptiQ Lab. |
|
|
| Full write-up: [Fine-tuning a vision model on a Mac](https://mlx-optiq.com/blog/vision-lora-on-a-mac). |
|
|
| --- |
|
|
| Built with [OptiQ](https://mlx-optiq.com) — mixed-precision quantization, |
| LoRA, and an OpenAI-compatible server for LLMs and VLMs on Apple Silicon. |
| `pip install mlx-optiq`. |
|
|
