dhadlab/dhad-vision

dhad-1-vision-preview (رؤية)

An Arabic image-captioning model: give it a picture, it writes a short Arabic description.

A frozen SigLIP2 vision encoder turns the image into 576 patch features; a small trained connector maps them to 144 soft tokens (64 in low-token mode); the tokens are spliced into a frozen Jais-3-0.9B language model through two learned marker vectors, and the model writes the caption.

Preview. Only stage 1 (the connector) was finished. Its output is caption-style text and it did not go through a formal release gate. An unfinished stage-2 snapshot (a vision LoRA that answers free-form questions about an image, stopped at step 675 of 1,435) is included as an extra, clearly labelled UNFINISHED.

What is in this repository

File	What
connector.pt	The model. A torch.save dict with connector (7,080,961 params) and markers (2 × 1536 = 3,072 params).
MANIFEST.json	Provenance: sources, sha256, parameter counts, training recipe, gate notes.
requirements.txt	Runtime Python packages.
LICENSE	Apache License 2.0 text.
code/connector.py	The connector architecture (pixel-shuffle → MLP → L2 × learnable scale).
code/graft.py	Splices the soft tokens into the frozen Jais trunk (inputs_embeds) and generates.
code/infer.py	describe(image, prompt) helper + CLI.
code/encoder.py	Runs the frozen SigLIP2 encoder on raw images → [576, 768] features.
stage2-step675/	UNFINISHED stage-2 snapshot: its own connector.pt + vision_lora/ (rank-32 LoRA, 27,983,872 params). See stage2-step675/UNFINISHED.md.

The base models are not re-uploaded: load inception42/Jais-3-0.9B (Apache-2.0) and google/siglip2-base-patch16-naflex (Apache-2.0) yourself.

Parameter count

The full runtime is 1,171,213,057 parameters (≈1.17B): the frozen Jais trunk (1,078,285,824) + the frozen SigLIP2 vision tower used (85,843,200) + the connector and markers (7,084,033). Only the 28 MB connector file is new; the rest is public base weights. With the unfinished stage-2 LoRA it would be 1,199,196,929.

How it was trained (stage 1)

• Trainable: connector + 2 marker vectors only (7.08M). The SigLIP2 encoder and the Jais trunk are frozen end-to-end.
• Data: 29,800 train + 200 held-out pairs of (image, Arabic caption) from FreedomIntelligence/ALLaVA-4V images with FreedomIntelligence/ALLaVA-4V-Arabic captions (machine translation of GPT-4V captions). Captions trimmed to their first 1–2 sentences. See MANIFEST.json.
• Recipe: cosine LR 1e-3, 64 samples/step, 2 epochs (930 steps), bf16, gradient checkpointing, 30 % of steps in 64-token mode.
• Probe at step 930: caption loss on the true image 2.4456 vs shuffled image 3.0071 (+18.7 % gain). Owner-tested and accepted 2026-11-14; no formal gate was run.

Usage

The weights are a raw state dict, so you also need the code/ folder (shipped here) and the two base models (downloaded automatically on first run).

1. Download this repository

hf download dhadlab/dhad-vision --local-dir dhad-1-vision-preview-1.2b
cd dhad-1-vision-preview-1.2b

Or download it from the Files tab of this page.

2. Install

pip install -r requirements.txt

3. Describe an image

python code/infer.py photo.jpg
python code/infer.py photo.jpg "صف هذه الصورة بالتفصيل"

Real output from a test photo:

الصورة تظهر مجموعة من أشجار التفاح في بيئة خريفية. الأشجار مغطاة بأوراق بنية داكنة، مما يشير إلى أنها في أواخر الموسم.

Python

import sys
sys.path.insert(0, "code")
import infer

g = infer.load_graft(device="cuda")                 # loads Jais + connector.pt (once)
print(infer.describe("photo.jpg", "صِف هذه الصورة", graft=g))
print(infer.describe("b.jpg", graft=g))             # reuse the loaded models

infer.describe also accepts a path, a PIL image, raw bytes, or cached SigLIP2 features ([576, 768] fp16). Defaults: SigLIP2 encoder on CPU (1–2 s per image), 96 new tokens, Arabic prompt صِف هذه الصورة. Environment options: DEVICE=cuda (encoder on GPU), LOW=1 (64-token mode), CONNECTOR=<dir>, JAIS_PATH / SIGLIP2_PATH (local base-model copies).

Expect caption-style output, not chat: this model describes an image; it does not reliably answer questions about it (that was the unfinished stage 2).

Intended use and limits

• Arabic (and some English) image captioning / description for research and demos.
• Not an OCR engine (glyph probe was weak), not a reliable visual question answerer, and not grounded truth. Captions may be inaccurate, generic, or hallucinated — the training captions themselves are GPT-4V output, not native ground truth.
• No safety/NSFW filtering was applied to training images.
• Stage 1 is a captioner; free-form image Q&A and image-follow-up turns belong to the unfinished stage-2 snapshot and are not supported by the released weights.

Licence and data provenance

Apache License 2.0. The connector weights and the code in this repository are released under the Apache License 2.0 (see LICENSE).

Training-data provenance (informational): the connector was trained on ALLaVA-4V captions and LAION-crawled images. ALLaVA-4V's card states CC-BY-NC-4.0; its uploaders do not own the image rights, per-image licences are unknown, and the captions are ChatGPT/GPT-4V-derived. We do not ship any images or captions — only the trained connector — but if you redistribute or build on this model, review those upstream terms yourself.

• Base trunk: inception42/Jais-3-0.9B — Apache-2.0 (declared in its model card).
• Encoder: google/siglip2-base-patch16-naflex — Apache-2.0. Not re-uploaded; cite SigLIP 2, arXiv:2502.14786.
• Training images/captions: FreedomIntelligence/ALLaVA-4V (CC-BY-NC-4.0 card) and FreedomIntelligence/ALLaVA-4V-Arabic (Apache-2.0 card, derived from the GPT-4V captions).

See NOTICE.md for the full third-party list.

Files and checksums

MANIFEST.json holds the sha256 of every shipped file (recompute with sha256sum). Connector: 82c611cadf8e694d23b5edea89297a5647c61417f45e11a168f5d9cafc63dd22 (28,339,217 B).

Citation

@misc{dhad1visionpreview,
  title  = {dhad-1-vision-preview: an Arabic image-captioning connector},
  author = {Dhad},
  year   = {2026},
  note   = {Preview release, stage 1 only. https://huggingface.co/dhadlab/dhad-vision}
}