Organize artifacts and refresh model card
#2
by grKnight - opened
README.md
CHANGED
|
@@ -8,8 +8,9 @@ datasets:
|
|
| 8 |
language:
|
| 9 |
- en
|
| 10 |
pipeline_tag: image-text-to-text
|
| 11 |
-
tags:
|
| 12 |
-
-
|
|
|
|
| 13 |
- llava
|
| 14 |
- astronomy
|
| 15 |
- multimodal
|
|
@@ -17,7 +18,9 @@ tags:
|
|
| 17 |
- connector
|
| 18 |
---
|
| 19 |
|
| 20 |
-
# AstraQ-VL Stage-1 (connector alignment)
|
|
|
|
|
|
|
| 21 |
|
| 22 |
A LLaVA-style vision–language connector that lets **Qwen2.5-1.5B-Instruct** describe astronomy
|
| 23 |
images encoded by **CLIP ViT-L/14**. Only the connector (~3.9M params) is trained; both backbones
|
|
@@ -27,22 +30,33 @@ with a **disjoint held-out test split** so it can be evaluated on unseen images.
|
|
| 27 |
|
| 28 |
> ⚠️ This repo ships the **connector checkpoint only** (`connector.safetensors`, ~16 MB). It is
|
| 29 |
> **not** a standalone `transformers` model — it needs the custom VLM code from the
|
| 30 |
-
> [
|
| 31 |
> (auto-downloaded from the Hub) to run.
|
| 32 |
|
| 33 |
## Downloads (per-epoch bundles)
|
| 34 |
|
| 35 |
-
Each bundle holds that epoch's checkpoint, its **held-out** predictions (`predictions_test_ep*.jsonl`),
|
| 36 |
-
the training config, the `test.json` split, and a `REPRODUCE.md`:
|
| 37 |
-
|
| 38 |
-
| Bundle | Checkpoint | |
|
| 39 |
|--------|-----------|--|
|
| 40 |
-
| [`astraq-vl-stage1-ep3.zip`](https://huggingface.co/grKnight/astraq-vl-stage1/blob/main/astraq-vl-stage1-ep3.zip) | `checkpoint-3789` (epoch 3, final) | **recommended** |
|
| 41 |
-
| [`astraq-vl-stage1-ep2.zip`](https://huggingface.co/grKnight/astraq-vl-stage1/blob/main/astraq-vl-stage1-ep2.zip) | `checkpoint-2500` (≈ epoch 2) | |
|
| 42 |
-
| [`astraq-vl-stage1-ep1.zip`](https://huggingface.co/grKnight/astraq-vl-stage1/blob/main/astraq-vl-stage1-ep1.zip) | `checkpoint-1300` (≈ epoch 1) | |
|
| 43 |
-
|
| 44 |
-
|
| 45 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 46 |
|
| 47 |
> **Superseded files.** An earlier release (`*-legacy-1epoch-no-heldout-*`) was trained to ~1 epoch
|
| 48 |
> only and evaluated on training images (no held-out split, so possible leakage). Kept for record;
|
|
@@ -82,16 +96,16 @@ epoch-1/epoch-2 points for comparison.
|
|
| 82 |
|
| 83 |
```bash
|
| 84 |
# 1. get the code
|
| 85 |
-
git clone https://github.com/crimsonKn1ght/
|
| 86 |
pip install -r requirements.txt
|
| 87 |
|
| 88 |
# 2. download + unzip the recommended bundle
|
| 89 |
-
hf download grKnight/astraq-vl-stage1 astraq-vl-stage1-ep3.zip --local-dir .
|
| 90 |
-
unzip astraq-vl-stage1-ep3.zip -d ckpt
|
| 91 |
|
| 92 |
# 3. caption an image (CLIP + Qwen auto-download on first run)
|
| 93 |
python inference.py \
|
| 94 |
-
--config ckpt/pretrain_astrollava.yaml \
|
| 95 |
--checkpoint ckpt/checkpoint-3789 \
|
| 96 |
--image your_astro_image.jpg \
|
| 97 |
--prompt "Describe this astronomical image." \
|
|
@@ -111,9 +125,10 @@ memorization.
|
|
| 111 |
|
| 112 |
**What it doesn't** — it **hallucinates fine details** (exact catalog numbers, telescopes, dates,
|
| 113 |
distances), filling specifics from the frozen LLM's prior rather than the pixels. This is the
|
| 114 |
-
expected Stage-1 ceiling: the connector supplies a coarse visual category and the frozen LLM
|
| 115 |
-
improvises the rest. For factual specificity, a **Stage-2 fine-tune** (unfreezing the LLM
|
| 116 |
-
|
|
|
|
| 117 |
|
| 118 |
The held-out comparison above is a **qualitative spot check** on a few samples, not a full
|
| 119 |
quantitative benchmark.
|
|
@@ -126,7 +141,7 @@ the exact train/test partition.
|
|
| 126 |
|
| 127 |
```
|
| 128 |
build: python scripts/build_astrollava_trainset.py --include-qa --max-image-size 384 --test-fraction 0.02 --seed 42
|
| 129 |
-
train: python train.py --config configs/
|
| 130 |
eval: python scripts/batch_inference.py --records-json datasets/astrollava_llava/test.json --num-samples 0 ...
|
| 131 |
```
|
| 132 |
|
|
|
|
| 8 |
language:
|
| 9 |
- en
|
| 10 |
pipeline_tag: image-text-to-text
|
| 11 |
+
tags:
|
| 12 |
+
- astraq-vl
|
| 13 |
+
- vision-language-model
|
| 14 |
- llava
|
| 15 |
- astronomy
|
| 16 |
- multimodal
|
|
|
|
| 18 |
- connector
|
| 19 |
---
|
| 20 |
|
| 21 |
+
# AstraQ-VL Stage-1 (connector alignment)
|
| 22 |
+
|
| 23 |
+
AstraQ-VL Stage-1 is the public name for this connector-alignment checkpoint.
|
| 24 |
|
| 25 |
A LLaVA-style vision–language connector that lets **Qwen2.5-1.5B-Instruct** describe astronomy
|
| 26 |
images encoded by **CLIP ViT-L/14**. Only the connector (~3.9M params) is trained; both backbones
|
|
|
|
| 30 |
|
| 31 |
> ⚠️ This repo ships the **connector checkpoint only** (`connector.safetensors`, ~16 MB). It is
|
| 32 |
> **not** a standalone `transformers` model — it needs the custom VLM code from the
|
| 33 |
+
> [astraq-vl](https://github.com/crimsonKn1ght/astraq-vl) repo plus the two base models
|
| 34 |
> (auto-downloaded from the Hub) to run.
|
| 35 |
|
| 36 |
## Downloads (per-epoch bundles)
|
| 37 |
|
| 38 |
+
Each bundle holds that epoch's checkpoint, its **held-out** predictions (`predictions_test_ep*.jsonl`),
|
| 39 |
+
the training config, the `test.json` split, and a `REPRODUCE.md`:
|
| 40 |
+
|
| 41 |
+
| Bundle | Checkpoint | |
|
| 42 |
|--------|-----------|--|
|
| 43 |
+
| [`astraq-vl-stage1-ep3.zip`](https://huggingface.co/grKnight/astraq-vl-stage1/blob/main/checkpoints/standard/astraq-vl-stage1-ep3.zip) | `checkpoint-3789` (epoch 3, final) | **recommended** |
|
| 44 |
+
| [`astraq-vl-stage1-ep2.zip`](https://huggingface.co/grKnight/astraq-vl-stage1/blob/main/checkpoints/standard/astraq-vl-stage1-ep2.zip) | `checkpoint-2500` (≈ epoch 2) | |
|
| 45 |
+
| [`astraq-vl-stage1-ep1.zip`](https://huggingface.co/grKnight/astraq-vl-stage1/blob/main/checkpoints/standard/astraq-vl-stage1-ep1.zip) | `checkpoint-1300` (≈ epoch 1) | |
|
| 46 |
+
|
| 47 |
+
## Evaluation artifacts
|
| 48 |
+
|
| 49 |
+
| Artifact | Scope | Contents |
|
| 50 |
+
|----------|-------|----------|
|
| 51 |
+
| [`astraq-vl-stage1-full-heldout-eval-v1.zip`](https://huggingface.co/grKnight/astraq-vl-stage1/blob/main/evaluations/full-heldout/astraq-vl-stage1-full-heldout-eval-v1.zip) | **Full held-out: captions + QA** | Predictions and aggregate/per-sample metrics for all 3,271 held-out records: 586 caption records and 2,685 QA records, plus comparisons and reproduction notes. |
|
| 52 |
+
| [`phase0_stage1_ep1_results.zip`](https://huggingface.co/grKnight/astraq-vl-stage1/blob/main/evaluations/phase0-captions-only/phase0_stage1_ep1_results.zip) | **Phase 0 (captions only), epoch 1** | Caption predictions with NLI and SBERT aggregate/per-sample scores. |
|
| 53 |
+
| [`phase0_stage1_ep2_results.zip`](https://huggingface.co/grKnight/astraq-vl-stage1/blob/main/evaluations/phase0-captions-only/phase0_stage1_ep2_results.zip) | **Phase 0 (captions only), epoch 2** | Caption predictions with NLI and SBERT aggregate/per-sample scores. |
|
| 54 |
+
| [`phase0_stage1_results.zip`](https://huggingface.co/grKnight/astraq-vl-stage1/blob/main/evaluations/phase0-captions-only/phase0_stage1_results.zip) | **Phase 0 (captions only), epoch 3** | Caption predictions with NLI and SBERT aggregate/per-sample scores. |
|
| 55 |
+
|
| 56 |
+
The Phase 0 archives are the earlier caption-generation evaluation only; they do **not** include
|
| 57 |
+
the held-out QA records. Each contains predictions for 591 held-out images, of which 586 have
|
| 58 |
+
reference captions used for scoring. Use the full-heldout artifact for the combined caption + QA
|
| 59 |
+
evaluation.
|
| 60 |
|
| 61 |
> **Superseded files.** An earlier release (`*-legacy-1epoch-no-heldout-*`) was trained to ~1 epoch
|
| 62 |
> only and evaluated on training images (no held-out split, so possible leakage). Kept for record;
|
|
|
|
| 96 |
|
| 97 |
```bash
|
| 98 |
# 1. get the code
|
| 99 |
+
git clone https://github.com/crimsonKn1ght/astraq-vl && cd astraq-vl
|
| 100 |
pip install -r requirements.txt
|
| 101 |
|
| 102 |
# 2. download + unzip the recommended bundle
|
| 103 |
+
hf download grKnight/astraq-vl-stage1 checkpoints/standard/astraq-vl-stage1-ep3.zip --local-dir .
|
| 104 |
+
unzip checkpoints/standard/astraq-vl-stage1-ep3.zip -d ckpt
|
| 105 |
|
| 106 |
# 3. caption an image (CLIP + Qwen auto-download on first run)
|
| 107 |
python inference.py \
|
| 108 |
+
--config ckpt/pretrain_astrollava.yaml \
|
| 109 |
--checkpoint ckpt/checkpoint-3789 \
|
| 110 |
--image your_astro_image.jpg \
|
| 111 |
--prompt "Describe this astronomical image." \
|
|
|
|
| 125 |
|
| 126 |
**What it doesn't** — it **hallucinates fine details** (exact catalog numbers, telescopes, dates,
|
| 127 |
distances), filling specifics from the frozen LLM's prior rather than the pixels. This is the
|
| 128 |
+
expected AstraQ-VL Stage-1 ceiling: the connector supplies a coarse visual category and the frozen LLM
|
| 129 |
+
improvises the rest. For factual specificity, a **Stage-2 fine-tune** (unfreezing the LLM via LoRA
|
| 130 |
+
on the QA pairs) is the fix — more Stage-1 epochs do not help. That model is now released at
|
| 131 |
+
[`grKnight/astraq-vl-stage2`](https://huggingface.co/grKnight/astraq-vl-stage2).
|
| 132 |
|
| 133 |
The held-out comparison above is a **qualitative spot check** on a few samples, not a full
|
| 134 |
quantitative benchmark.
|
|
|
|
| 141 |
|
| 142 |
```
|
| 143 |
build: python scripts/build_astrollava_trainset.py --include-qa --max-image-size 384 --test-fraction 0.02 --seed 42
|
| 144 |
+
train: python train.py --config configs/pretrain_astraq_vl.yaml
|
| 145 |
eval: python scripts/batch_inference.py --records-json datasets/astrollava_llava/test.json --num-samples 0 ...
|
| 146 |
```
|
| 147 |
|
evaluations/{phase0 → phase0-captions-only}/phase0_stage1_ep1_results.zip
RENAMED
|
File without changes
|
evaluations/{phase0 → phase0-captions-only}/phase0_stage1_ep2_results.zip
RENAMED
|
File without changes
|
evaluations/{phase0 → phase0-captions-only}/phase0_stage1_results.zip
RENAMED
|
File without changes
|