Organize artifacts and refresh model card

#2
by grKnight - opened
README.md CHANGED
@@ -8,8 +8,9 @@ datasets:
8
  language:
9
  - en
10
  pipeline_tag: image-text-to-text
11
- tags:
12
- - vision-language-model
 
13
  - llava
14
  - astronomy
15
  - multimodal
@@ -17,7 +18,9 @@ tags:
17
  - connector
18
  ---
19
 
20
- # AstraQ-VL Stage-1 (connector alignment)
 
 
21
 
22
  A LLaVA-style vision–language connector that lets **Qwen2.5-1.5B-Instruct** describe astronomy
23
  images encoded by **CLIP ViT-L/14**. Only the connector (~3.9M params) is trained; both backbones
@@ -27,22 +30,33 @@ with a **disjoint held-out test split** so it can be evaluated on unseen images.
27
 
28
  > ⚠️ This repo ships the **connector checkpoint only** (`connector.safetensors`, ~16 MB). It is
29
  > **not** a standalone `transformers` model — it needs the custom VLM code from the
30
- > [astronomy-vlm](https://github.com/crimsonKn1ght/astronomy-vlm) repo plus the two base models
31
  > (auto-downloaded from the Hub) to run.
32
 
33
  ## Downloads (per-epoch bundles)
34
 
35
- Each bundle holds that epoch's checkpoint, its **held-out** predictions (`predictions_test_ep*.jsonl`),
36
- the training config, the `test.json` split, and a `REPRODUCE.md`:
37
-
38
- | Bundle | Checkpoint | |
39
  |--------|-----------|--|
40
- | [`astraq-vl-stage1-ep3.zip`](https://huggingface.co/grKnight/astraq-vl-stage1/blob/main/astraq-vl-stage1-ep3.zip) | `checkpoint-3789` (epoch 3, final) | **recommended** |
41
- | [`astraq-vl-stage1-ep2.zip`](https://huggingface.co/grKnight/astraq-vl-stage1/blob/main/astraq-vl-stage1-ep2.zip) | `checkpoint-2500` (≈ epoch 2) | |
42
- | [`astraq-vl-stage1-ep1.zip`](https://huggingface.co/grKnight/astraq-vl-stage1/blob/main/astraq-vl-stage1-ep1.zip) | `checkpoint-1300` (≈ epoch 1) | |
43
-
44
-
45
- **Full held-out evaluation artifact.** [`astraq-vl-stage1-full-heldout-eval-v1.zip`](https://huggingface.co/grKnight/astraq-vl-stage1/blob/main/astraq-vl-stage1-full-heldout-eval-v1.zip) contains the caption + QA full-heldout evaluation over all 3,271 held-out records (586 caption records and 2,685 QA records), including predictions, aggregate metrics, per-sample metrics, comparison files, and reproduction notes.
 
 
 
 
 
 
 
 
 
 
 
46
 
47
  > **Superseded files.** An earlier release (`*-legacy-1epoch-no-heldout-*`) was trained to ~1 epoch
48
  > only and evaluated on training images (no held-out split, so possible leakage). Kept for record;
@@ -82,16 +96,16 @@ epoch-1/epoch-2 points for comparison.
82
 
83
  ```bash
84
  # 1. get the code
85
- git clone https://github.com/crimsonKn1ght/astronomy-vlm && cd astronomy-vlm
86
  pip install -r requirements.txt
87
 
88
  # 2. download + unzip the recommended bundle
89
- hf download grKnight/astraq-vl-stage1 astraq-vl-stage1-ep3.zip --local-dir .
90
- unzip astraq-vl-stage1-ep3.zip -d ckpt
91
 
92
  # 3. caption an image (CLIP + Qwen auto-download on first run)
93
  python inference.py \
94
- --config ckpt/pretrain_astrollava.yaml \
95
  --checkpoint ckpt/checkpoint-3789 \
96
  --image your_astro_image.jpg \
97
  --prompt "Describe this astronomical image." \
@@ -111,9 +125,10 @@ memorization.
111
 
112
  **What it doesn't** — it **hallucinates fine details** (exact catalog numbers, telescopes, dates,
113
  distances), filling specifics from the frozen LLM's prior rather than the pixels. This is the
114
- expected Stage-1 ceiling: the connector supplies a coarse visual category and the frozen LLM
115
- improvises the rest. For factual specificity, a **Stage-2 fine-tune** (unfreezing the LLM, e.g. via
116
- LoRA, on the QA pairs) is the next step — more Stage-1 epochs do not fix it.
 
117
 
118
  The held-out comparison above is a **qualitative spot check** on a few samples, not a full
119
  quantitative benchmark.
@@ -126,7 +141,7 @@ the exact train/test partition.
126
 
127
  ```
128
  build: python scripts/build_astrollava_trainset.py --include-qa --max-image-size 384 --test-fraction 0.02 --seed 42
129
- train: python train.py --config configs/pretrain_astrollava.yaml
130
  eval: python scripts/batch_inference.py --records-json datasets/astrollava_llava/test.json --num-samples 0 ...
131
  ```
132
 
 
8
  language:
9
  - en
10
  pipeline_tag: image-text-to-text
11
+ tags:
12
+ - astraq-vl
13
+ - vision-language-model
14
  - llava
15
  - astronomy
16
  - multimodal
 
18
  - connector
19
  ---
20
 
21
+ # AstraQ-VL Stage-1 (connector alignment)
22
+
23
+ AstraQ-VL Stage-1 is the public name for this connector-alignment checkpoint.
24
 
25
  A LLaVA-style vision–language connector that lets **Qwen2.5-1.5B-Instruct** describe astronomy
26
  images encoded by **CLIP ViT-L/14**. Only the connector (~3.9M params) is trained; both backbones
 
30
 
31
  > ⚠️ This repo ships the **connector checkpoint only** (`connector.safetensors`, ~16 MB). It is
32
  > **not** a standalone `transformers` model — it needs the custom VLM code from the
33
+ > [astraq-vl](https://github.com/crimsonKn1ght/astraq-vl) repo plus the two base models
34
  > (auto-downloaded from the Hub) to run.
35
 
36
  ## Downloads (per-epoch bundles)
37
 
38
+ Each bundle holds that epoch's checkpoint, its **held-out** predictions (`predictions_test_ep*.jsonl`),
39
+ the training config, the `test.json` split, and a `REPRODUCE.md`:
40
+
41
+ | Bundle | Checkpoint | |
42
  |--------|-----------|--|
43
+ | [`astraq-vl-stage1-ep3.zip`](https://huggingface.co/grKnight/astraq-vl-stage1/blob/main/checkpoints/standard/astraq-vl-stage1-ep3.zip) | `checkpoint-3789` (epoch 3, final) | **recommended** |
44
+ | [`astraq-vl-stage1-ep2.zip`](https://huggingface.co/grKnight/astraq-vl-stage1/blob/main/checkpoints/standard/astraq-vl-stage1-ep2.zip) | `checkpoint-2500` (≈ epoch 2) | |
45
+ | [`astraq-vl-stage1-ep1.zip`](https://huggingface.co/grKnight/astraq-vl-stage1/blob/main/checkpoints/standard/astraq-vl-stage1-ep1.zip) | `checkpoint-1300` (≈ epoch 1) | |
46
+
47
+ ## Evaluation artifacts
48
+
49
+ | Artifact | Scope | Contents |
50
+ |----------|-------|----------|
51
+ | [`astraq-vl-stage1-full-heldout-eval-v1.zip`](https://huggingface.co/grKnight/astraq-vl-stage1/blob/main/evaluations/full-heldout/astraq-vl-stage1-full-heldout-eval-v1.zip) | **Full held-out: captions + QA** | Predictions and aggregate/per-sample metrics for all 3,271 held-out records: 586 caption records and 2,685 QA records, plus comparisons and reproduction notes. |
52
+ | [`phase0_stage1_ep1_results.zip`](https://huggingface.co/grKnight/astraq-vl-stage1/blob/main/evaluations/phase0-captions-only/phase0_stage1_ep1_results.zip) | **Phase 0 (captions only), epoch 1** | Caption predictions with NLI and SBERT aggregate/per-sample scores. |
53
+ | [`phase0_stage1_ep2_results.zip`](https://huggingface.co/grKnight/astraq-vl-stage1/blob/main/evaluations/phase0-captions-only/phase0_stage1_ep2_results.zip) | **Phase 0 (captions only), epoch 2** | Caption predictions with NLI and SBERT aggregate/per-sample scores. |
54
+ | [`phase0_stage1_results.zip`](https://huggingface.co/grKnight/astraq-vl-stage1/blob/main/evaluations/phase0-captions-only/phase0_stage1_results.zip) | **Phase 0 (captions only), epoch 3** | Caption predictions with NLI and SBERT aggregate/per-sample scores. |
55
+
56
+ The Phase 0 archives are the earlier caption-generation evaluation only; they do **not** include
57
+ the held-out QA records. Each contains predictions for 591 held-out images, of which 586 have
58
+ reference captions used for scoring. Use the full-heldout artifact for the combined caption + QA
59
+ evaluation.
60
 
61
  > **Superseded files.** An earlier release (`*-legacy-1epoch-no-heldout-*`) was trained to ~1 epoch
62
  > only and evaluated on training images (no held-out split, so possible leakage). Kept for record;
 
96
 
97
  ```bash
98
  # 1. get the code
99
+ git clone https://github.com/crimsonKn1ght/astraq-vl && cd astraq-vl
100
  pip install -r requirements.txt
101
 
102
  # 2. download + unzip the recommended bundle
103
+ hf download grKnight/astraq-vl-stage1 checkpoints/standard/astraq-vl-stage1-ep3.zip --local-dir .
104
+ unzip checkpoints/standard/astraq-vl-stage1-ep3.zip -d ckpt
105
 
106
  # 3. caption an image (CLIP + Qwen auto-download on first run)
107
  python inference.py \
108
+ --config ckpt/pretrain_astrollava.yaml \
109
  --checkpoint ckpt/checkpoint-3789 \
110
  --image your_astro_image.jpg \
111
  --prompt "Describe this astronomical image." \
 
125
 
126
  **What it doesn't** — it **hallucinates fine details** (exact catalog numbers, telescopes, dates,
127
  distances), filling specifics from the frozen LLM's prior rather than the pixels. This is the
128
+ expected AstraQ-VL Stage-1 ceiling: the connector supplies a coarse visual category and the frozen LLM
129
+ improvises the rest. For factual specificity, a **Stage-2 fine-tune** (unfreezing the LLM via LoRA
130
+ on the QA pairs) is the fix — more Stage-1 epochs do not help. That model is now released at
131
+ [`grKnight/astraq-vl-stage2`](https://huggingface.co/grKnight/astraq-vl-stage2).
132
 
133
  The held-out comparison above is a **qualitative spot check** on a few samples, not a full
134
  quantitative benchmark.
 
141
 
142
  ```
143
  build: python scripts/build_astrollava_trainset.py --include-qa --max-image-size 384 --test-fraction 0.02 --seed 42
144
+ train: python train.py --config configs/pretrain_astraq_vl.yaml
145
  eval: python scripts/batch_inference.py --records-json datasets/astrollava_llava/test.json --num-samples 0 ...
146
  ```
147
 
evaluations/{phase0 → phase0-captions-only}/phase0_stage1_ep1_results.zip RENAMED
File without changes
evaluations/{phase0 → phase0-captions-only}/phase0_stage1_ep2_results.zip RENAMED
File without changes
evaluations/{phase0 → phase0-captions-only}/phase0_stage1_results.zip RENAMED
File without changes