CSE576 β€” LoRA adapters for LLaVA-1.5-7B (NoCaps contamination study)

LoRA adapters (+ fully trained multi_modal_projector) on llava-hf/llava-1.5-7b-hf. Each run: M = 130,000 examples, 1 epoch, LoRA r=128 / alpha=256 on all LM linear layers (q,k,v,o,gate,up,down), lr 2e-4 (projector 2e-5), global batch 128, bf16, split A. Full hyper-parameters, data hashes and environment are in each run's manifest.json.

Layout

  • ckpt/<run_id>/: adapter_config.json, adapter_model.safetensors, manifest.json (one folder per run)
  • outputs/: evaluation results (T=0 captions, metrics, short-prompt metrics, data hashes); verify with sha256sum -c outputs/SHA256SUMS_sol.txt
hf download yoorhim/26f-CSE576 --include "ckpt/G0_d0_s0/*" --local-dir $SHARED   # -> $SHARED/ckpt/G0_d0_s0
hf download yoorhim/26f-CSE576 --include "outputs/*" --local-dir $SHARED

Runs

Naming: {group}_{condition}_d{dose}[_n{N_exp}]_s{seed}. d = how many times each exposed NoCaps image is repeated; N_exp = number of exposed images (1,000 unless noted).

Group Runs Purpose
G0 G0_d0_s0..s4 d=0 baseline (no NoCaps exposure), 5 seeds
G1 {Caption,Paired,SameDist}_d{4,64}_s{0,1} dose sweep, high doses
G2 {Caption,Paired,SameDist}_d1_s* d=1
EX Caption/Paired_d2_n1000_s{0,1}, {Paired,SameDist}_d1_n{30,100,300}_s0 d=2 and exposed-set-size (K) sweep

Conditions: Paired = exposed NoCaps image + its reference caption; Caption = the reference caption only (text-only); SameDist = COCO images / captions of the same style (no NoCaps content, control). Code: https://github.com/ofzlo/data-contamination

Usage

from transformers import LlavaForConditionalGeneration, AutoProcessor
from peft import PeftModel
import torch

base = LlavaForConditionalGeneration.from_pretrained("llava-hf/llava-1.5-7b-hf", torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(base, "yoorhim/26f-CSE576", subfolder="ckpt/G1_Paired_d4_s0")
processor = AutoProcessor.from_pretrained("llava-hf/llava-1.5-7b-hf")

Trained with torch 2.5.1, transformers 4.46.3, peft 0.13.2.

Downloads last month
-
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for yoorhim/26f-CSE576

Adapter
(195)
this model