README.md

---
license: cc-by-sa-4.0
base_model:
  - Qwen/Qwen2.5-1.5B-Instruct
  - openai/clip-vit-large-patch14
datasets:
  - UniverseTBD/AstroLLaVA_convos
language:
  - en
pipeline_tag: image-text-to-text
tags:
  - astraq-vl
  - vision-language-model
  - llava
  - astronomy
  - multimodal
  - image-captioning
  - connector
---

# AstraQ-VL Stage-1 (connector alignment)

AstraQ-VL Stage-1 is the public name for this connector-alignment checkpoint.

A LLaVA-style vision–language connector that lets **Qwen2.5-1.5B-Instruct** describe astronomy
images encoded by **CLIP ViT-L/14**. Only the connector (~3.9M params) is trained; both backbones
stay frozen. This is the **Stage-1 feature-alignment** stage, trained for **3 epochs** on
[`UniverseTBD/AstroLLaVA_convos`](https://huggingface.co/datasets/UniverseTBD/AstroLLaVA_convos)
with a **disjoint held-out test split** so it can be evaluated on unseen images.

> ⚠️ This repo ships the **connector checkpoint only** (`connector.safetensors`, ~16 MB). It is
> **not** a standalone `transformers` model — it needs the custom VLM code from the
> [astraq-vl](https://github.com/crimsonKn1ght/astraq-vl) repo plus the two base models
> (auto-downloaded from the Hub) to run.

## Downloads (per-epoch bundles)

Each bundle holds that epoch's checkpoint, its **held-out** predictions (`predictions_test_ep*.jsonl`),
the training config, the `test.json` split, and a `REPRODUCE.md`:

| Bundle | Checkpoint | |
|--------|-----------|--|
| [`astraq-vl-stage1-ep3.zip`](https://huggingface.co/grKnight/astraq-vl-stage1/blob/main/checkpoints/standard/astraq-vl-stage1-ep3.zip) | `checkpoint-3789` (epoch 3, final) | **recommended** |
| [`astraq-vl-stage1-ep2.zip`](https://huggingface.co/grKnight/astraq-vl-stage1/blob/main/checkpoints/standard/astraq-vl-stage1-ep2.zip) | `checkpoint-2500` (≈ epoch 2) | |
| [`astraq-vl-stage1-ep1.zip`](https://huggingface.co/grKnight/astraq-vl-stage1/blob/main/checkpoints/standard/astraq-vl-stage1-ep1.zip) | `checkpoint-1300` (≈ epoch 1) | |

## Evaluation artifacts

| Artifact | Scope | Contents |
|----------|-------|----------|
| [`astraq-vl-stage1-full-heldout-eval-v1.zip`](https://huggingface.co/grKnight/astraq-vl-stage1/blob/main/evaluations/full-heldout/astraq-vl-stage1-full-heldout-eval-v1.zip) | **Full held-out: captions + QA** | Predictions and aggregate/per-sample metrics for all 3,271 held-out records: 586 caption records and 2,685 QA records, plus comparisons and reproduction notes. |
| [`phase0_stage1_ep1_results.zip`](https://huggingface.co/grKnight/astraq-vl-stage1/blob/main/evaluations/phase0-captions-only/phase0_stage1_ep1_results.zip) | **Phase 0 (captions only), epoch 1** | Caption predictions with NLI and SBERT aggregate/per-sample scores. |
| [`phase0_stage1_ep2_results.zip`](https://huggingface.co/grKnight/astraq-vl-stage1/blob/main/evaluations/phase0-captions-only/phase0_stage1_ep2_results.zip) | **Phase 0 (captions only), epoch 2** | Caption predictions with NLI and SBERT aggregate/per-sample scores. |
| [`phase0_stage1_results.zip`](https://huggingface.co/grKnight/astraq-vl-stage1/blob/main/evaluations/phase0-captions-only/phase0_stage1_results.zip) | **Phase 0 (captions only), epoch 3** | Caption predictions with NLI and SBERT aggregate/per-sample scores. |

The Phase 0 archives are the earlier caption-generation evaluation only; they do **not** include
the held-out QA records. Each contains predictions for 591 held-out images, of which 586 have
reference captions used for scoring. Use the full-heldout artifact for the combined caption + QA
evaluation.

> **Superseded files.** An earlier release (`*-legacy-1epoch-no-heldout-*`) was trained to ~1 epoch
> only and evaluated on training images (no held-out split, so possible leakage). Kept for record;
> use the `ep1`/`ep2`/`ep3` bundles above.

## Architecture

```
image ─► CLIP ViT-L/14 (frozen) ─► MLP connector (TRAINED) ─► Qwen2.5-1.5B-Instruct (frozen) ─► text
                                    1024 → 1536 → 1536
```

- **Vision:** `openai/clip-vit-large-patch14`, penultimate layer patch features (frozen)
- **Connector:** 2-layer MLP with GELU, 1024→1536→1536 (the only trained weights)
- **LLM:** `Qwen/Qwen2.5-1.5B-Instruct` (frozen)
- **Trainable / total:** 3,935,232 / 1,850,414,592 (0.21%)

## Training

| | |
|---|---|
| Data | `UniverseTBD/AstroLLaVA_convos`, per-image held-out split: train 161,653 recs / 29,151 imgs, test 3,271 recs / 591 imgs (41 corrupt skipped) |
| Image prep | long side ≤ 384 px, JPEG |
| Objective | next-token cross-entropy on answer tokens only (connector-only) |
| Epochs / steps | 3 epochs, 3,789 update steps |
| Effective batch | 128 (per-device 8 × grad-accum 16) |
| LR / schedule | 1e-3, cosine with 3% warmup |
| Precision | bf16 (autocast) |
| Max length | 512 (+256 image tokens) |
| Hardware | 1× RTX 6000 Ada (48 GB), ~26 samples/s, ~38 GB VRAM |
| Loss | ~2.08 → ~1.50 |

`checkpoint-3789` (epoch 3) is the recommended checkpoint; `checkpoint-1300`/`-2500` are the
epoch-1/epoch-2 points for comparison.

## Usage

```bash
# 1. get the code
git clone https://github.com/crimsonKn1ght/astraq-vl && cd astraq-vl
pip install -r requirements.txt

# 2. download + unzip the recommended bundle
hf download grKnight/astraq-vl-stage1 checkpoints/standard/astraq-vl-stage1-ep3.zip --local-dir .
unzip checkpoints/standard/astraq-vl-stage1-ep3.zip -d ckpt

# 3. caption an image (CLIP + Qwen auto-download on first run)
python inference.py \
  --config ckpt/pretrain_astrollava.yaml \
  --checkpoint ckpt/checkpoint-3789 \
  --image your_astro_image.jpg \
  --prompt "Describe this astronomical image." \
  --temperature 0
```

The bundled `predictions_test_ep*.jsonl` hold the held-out outputs with their reference captions.

## Capabilities & limitations

**What it does well** — it grounds on *coarse visual structure* (object class / morphology), and
this **generalizes to held-out images**. On unseen test images, quality improved monotonically with
training: epoch 1 misidentified objects, epoch 2 fixed the object *category*, and epoch 3 recovered
*specific* objects — e.g. correctly naming **SN 1987A and its ring** and the **Dumbbell Nebula**, on
images it never trained on. Because these are held-out, that's genuine generalization, not
memorization.

**What it doesn't** — it **hallucinates fine details** (exact catalog numbers, telescopes, dates,
distances), filling specifics from the frozen LLM's prior rather than the pixels. This is the
expected AstraQ-VL Stage-1 ceiling: the connector supplies a coarse visual category and the frozen LLM
improvises the rest. For factual specificity, a **Stage-2 fine-tune** (unfreezing the LLM via LoRA
on the QA pairs) is the fix — more Stage-1 epochs do not help. That model is now released at
[`grKnight/astraq-vl-stage2`](https://huggingface.co/grKnight/astraq-vl-stage2).

The held-out comparison above is a **qualitative spot check** on a few samples, not a full
quantitative benchmark.

## Reproduction

Each bundle includes a `REPRODUCE.md` pinning the exact code commit, base models, and package
versions (`torch 2.8.0+cu128`, `transformers 5.12.1`). The split is seeded, so the build reproduces
the exact train/test partition.

```
build:  python scripts/build_astrollava_trainset.py --include-qa --max-image-size 384 --test-fraction 0.02 --seed 42
train:  python train.py --config configs/pretrain_astraq_vl.yaml
eval:   python scripts/batch_inference.py --records-json datasets/astrollava_llava/test.json --num-samples 0 ...
```

## Citation

If you use AstraQ-VL, this checkpoint, or its evaluation artifacts, please cite:

> Roy, G. (2026). *AstraQ-VL: Parameter-Efficient Astronomy Vision-Language Modeling with Connector Alignment and LoRA Tuning* (Version v1). Zenodo. [https://doi.org/10.5281/zenodo.21284851](https://doi.org/10.5281/zenodo.21284851)

```bibtex
@misc{roy2026astraqvl,
  author    = {Roy, Gourab},
  title     = {AstraQ-VL: Parameter-Efficient Astronomy Vision-Language Modeling with Connector Alignment and LoRA Tuning},
  year      = {2026},
  publisher = {Zenodo},
  version   = {v1},
  doi       = {10.5281/zenodo.21284851},
  url       = {https://doi.org/10.5281/zenodo.21284851}
}
```

## License & attribution

- **Weights:** `cc-by-sa-4.0`, inherited from the training data.
- **Training data:** [`UniverseTBD/AstroLLaVA_convos`](https://huggingface.co/datasets/UniverseTBD/AstroLLaVA_convos)
  (CC-BY-SA-4.0); imagery from NASA APOD, ESO, and NASA/ESA Hubble.
- **Base models:** Qwen2.5-1.5B-Instruct (Apache-2.0), CLIP ViT-L/14 (OpenAI, MIT).
- Built on the AstroLLaVA work.