README.md

---
base_model: unsloth/Llama-3.2-11B-Vision-Instruct-bnb-4bit
library_name: peft
license: llama3.2
language:
  - en
pipeline_tag: image-to-text
datasets:
  - AIOmarRehan/space-multimodal-dataset
tags:
  - lora
  - peft
  - unsloth
  - mllama
  - vision-language
  - image-captioning
  - astronomy
---

# AstroVision LoRA — astronomy image captioning

LoRA adapter for `Llama-3.2-11B-Vision-Instruct`, trained on 250 astronomy
image/caption pairs on a single free-tier T4.

**Full model card, training code, evaluation and limitations:**
[`Samuelsunshine/astrovision-lora-debugging`](https://github.com/Samuelsunshine/astrovision-lora-debugging)

The repository is two things: this pipeline, and a written audit of five defects
in the published pipeline it reproduces — one of which made that project's
public demo serve base weights instead of its fine-tune, silently.
[Read the audit](https://github.com/Samuelsunshine/astrovision-lora-debugging/blob/main/DEBUGGING.md).

## Licence

**Llama 3.2 Community License.** This adapter is a derivative of
Llama-3.2-11B-Vision-Instruct and inherits that licence — not Apache 2.0, which
is what the auto-generated card originally said here. See
[the licence](https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/LICENSE).

The training and evaluation *code* in the linked repository is MIT. The dataset's
source imagery has unestablished provenance; check before downstream use.

## Results

Tesla T4, 2026-08-21, 25 held-out images, greedy decoding. Raw log
[committed in the repository](https://github.com/Samuelsunshine/astrovision-lora-debugging/blob/main/evidence/runs/2026-08-21-t4-mainline.json).

| Metric | Value |
|---|---|
| BLEU | 0.0722 |
| ROUGE-1 / ROUGE-2 / ROUGE-L | 0.3735 / 0.1405 / 0.3361 |
| `length_ratio` | 0.9561 |
| Best validation loss | 0.7232 (at the final step) |

Validation loss fell monotonically with its minimum at step 30, so the model is
**undertrained** at this budget rather than overfit. 30 steps was chosen to match
the compute of the pipeline being reproduced.

**BLEU is a weak fit for captioning** and these numbers are not a quality claim.
An image has many correct descriptions, and two correct captions with disjoint
vocabulary both score near zero. They are reported because they are reproducible
and comparable across this project's own runs.

## Limitations — please read before trusting a caption

**This is not an astronomy tool.** It was trained on 250 images across five
loosely-defined subjects (Earth from orbit, Mars, Mars rovers, the Milky Way,
Hubble) for 30 optimizer steps. It reproduces the *writing style* of that corpus.
It has no astronomical knowledge beyond what the base model already had.

**It is confidently wrong, demonstrated.** From the held-out set, unedited:

> **Reference:** A close-up photograph of a Mars rover undergoing calibration
> tests inside a NASA facility **on Earth**.
> **Prediction:** A photograph of the Mars rover exploring the **Martian
> terrain** under the reddish-orange sky.

A laboratory on Earth described as the surface of Mars, fluently, with no
hedging. Two of five logged samples make errors of this kind.

**Do not use it for** identification of objects, missions, instruments or
locations; measurement or any scientific claim; accessibility descriptions,
where a confident wrong caption is worse than none; or education without review.

**Out of domain it degrades quietly.** Given a photograph that is not astronomy
imagery it will still produce an astronomy-flavoured caption, because that is the
only register it was trained in. Fluency here is unrelated to correctness.

## Usage

```python
from peft import PeftModel
from unsloth import FastVisionModel

model, tokenizer = FastVisionModel.from_pretrained(
    "unsloth/Llama-3.2-11B-Vision-Instruct-bnb-4bit",
    load_in_4bit=True,
    use_gradient_checkpointing="unsloth",
)
model = PeftModel.from_pretrained(model, "SamHung/astrovision-lora")
FastVisionModel.for_inference(model)
```

Use `PeftModel.from_pretrained` to load a trained adapter.
`FastVisionModel.get_peft_model(model, lora_adapter=...)` reads as though it
would do the same and does not — it constructs a *new* adapter and ignores the
keyword, and because LoRA initialises `B` to zero the result is
mathematically identical to the base model, with no error raised. That failure
is Finding 4 in the audit, and
[`scripts/verify_adapter_loading.py`](https://github.com/Samuelsunshine/astrovision-lora-debugging/blob/main/scripts/verify_adapter_loading.py)
exists to catch it: it checks the artifacts are present, that `‖lora_B‖ > 0`,
and that enabling the adapter actually changes the output.

Inference should go through `generate_caption()` from the repository, which
slices the prompt off in token space and always states `do_sample` explicitly —
decoding the full `generate()` output echoes the instruction into every
prediction and roughly doubles measured length.

## Training

LoRA `r=16`, `alpha=16`, `dropout=0`, `bias="none"`, on vision and language
layers, attention and MLP. 30 steps at an effective batch of 8, `lr=2e-4`,
`adamw_8bit`, fp16, 4-bit NF4 base. 200 train / 25 validation / 25 test,
`seed=42`, split **before** any training dataset was built and asserted disjoint
at runtime. 67,174,400 trainable parameters. 20.1 minutes on a T4, peak 9.227 GB
of 14.563 GB.

## Provenance

Coursework reproduction. The dataset is
[`AIOmarRehan/space-multimodal-dataset`](https://huggingface.co/datasets/AIOmarRehan/space-multimodal-dataset)
and the pipeline reproduced is
[`AIOmarRehan/Unsloth_Llama_3.2_11B_Vision_Instruct_Astronomy`](https://github.com/AIOmarRehan/Unsloth_Llama_3.2_11B_Vision_Instruct_Astronomy)
(MIT, © Omar Rehan). Three data/evaluation defects in that pipeline were found
and fixed here, and two further findings documented; the audited code and every
execution log cited as evidence are the original author's.

Trained with [Unsloth](https://github.com/unslothai/unsloth) and TRL.