trimdata baseline checkpoints

LoRA adapters (+ trained mm_projector in non_lora_trainables.bin) for baseline data-selection methods on LLaVA-1.5-7B instruction tuning: one natural epoch on a selected subset of LLaVA-665K. Research artefacts for reproduction; code and full documentation: https://github.com/LZhang12345/trimdata. Exact selections: https://huggingface.co/datasets/submissionpaper1234/trimdata-baseline-selections.

Layout: <method>/<budget>/<selection-seed>-<training-seed>/ with adapter_model.safetensors, non_lora_trainables.bin, adapter_config.json, config.json (portable copies; byte-exact originals as *.original.json), trainer_state.json and provenance.json. index.csv lists all checkpoints with sha256 and core-four scores; SHA256SUMS covers every file.

Loading

pip install -U "huggingface_hub>=0.23"
hf download submissionpaper1234/trimdata-baseline-checkpoints --include "uniform/b10/sel1234-t17/*" --local-dir ckpts
# inside a checkout of haotian-liu/LLaVA at commit c121f0432da27facab705978f83c4ada465e46fd (v1.2.x API)
from llava.model.builder import load_pretrained_model
path = "ckpts/uniform/b10/sel1234-t17"          # adapter_config.json, adapter_model.safetensors, non_lora_trainables.bin, config.json
tokenizer, model, image_processor, context_len = load_pretrained_model(
    model_path=path, model_base="lmsys/vicuna-7b-v1.5",
    model_name="llava-v1.5-7b-lora")             # the name must contain "llava" and "lora" so the LoRA branch is used

config.json / adapter_config.json are portable copies in which the cluster-local snapshot paths were replaced by the Hub ids above; the byte-exact originals are kept as config.original.json / adapter_config.original.json. non_lora_trainables.bin holds the trained mm_projector weights (projector LR 2e-5); the builder loads it before attaching the LoRA weights. Our own evaluator kept the LoRA unmerged (PEFT 0.7.1) in fp16; the official builder merges it (merge_and_unload) β€” re-evaluating a provided checkpoint is part of the calibration step below.

Evaluation outputs

Every checkpoint directory also contains eval/ with the per-run evaluation outputs, so scores can be recomputed without re-running inference:

  • eval/EVAL_MANIFEST.json β€” every eval file with bytes/sha256 and its original evaluator path (<HOME>/<USER_DATA> placeholders), per-bank answer counts, the scores cross-checked against provenance.json, checkpoint-binding evidence (the adapter sha256 recorded by the evaluator), source Slurm job ids and any missing items.
  • eval/core4/<bank>/PREDICTIONS.jsonl β€” raw greedy generations (question_id, prompt, text) for POPE (8,910), GQA test-dev balanced (12,578), TextVQA val (5,000; question_id is not unique there, join via core4/run/**/TEXTVQA_JOIN_MAPPING.json) and ScienceQA-IMG (2,017) = 28,505 answers per run, plus the scorer output SCORE.json and decoding/runtime record FUNCTION.json. eval/core4/FULL_EVALUATION_RESULT-<job>.json (helper evaluations) or eval/core4/run/**/RESULT.json (earlier root evaluations) is the evaluator's complete result record.
  • eval/extended/<bank>/ (mme-perception, mme-cognition, seed-image, mmmu-val, mmbench-en-dev, llava-wild, mm-vet, vizwiz-test) β€” PREDICTIONS.jsonl, SCORE_*.json, FUNCTION.json/EXPORT_SUMMARY.json, and for the GPT-judged LLaVA-W / MM-Vet the per-item judge outputs JUDGE_EVIDENCE_<bank>__<judge>.json (gpt-4o-2024-08-06 primary; gpt-5.6-terra supplementary where present).

Coverage: core-four outputs for 44/46 checkpoints (coincide/b20/selfixed-t29, coincide/b20/selfixed-t43 were never evaluated; their manifest says so). Extended outputs for 27 checkpoints (22 with all eight; 5 without VizWiz). Benchmark questions and labels are not redistributed (use the official benchmark files; their names and sha256 are listed in the GitHub repository docs). For uniform/b10/sel1234-t17 and source_sqrt/b10/sel1234-t43 the archived GQA SCORE.json field is an invalid 0.0; the accepted GQA score (= provenance.json) was recomputed from the identical PREDICTIONS.jsonl (see core4.gqa_override in the manifest). SHA256SUMS covers the eval files too.

Training recipe

Item Value (as executed)
Base Vicuna-7B-v1.5 + CLIP ViT-L/14-336 (mm_vision_select_layer=-2, mm_vision_select_feature=patch) + stage-1 mlp2x_gelu projector
Code base official LLaVA llava/train/train.py at commit c121f0432da27facab705978f83c4ada465e46fd, conversation template v1
Trainable LoRA on all LLM linear layers (q,k,v,o,gate,up,down), r=128, alpha=256, dropout=0.05, bias=none; projector fully trained
Learning rate 2e-4 (LoRA), 2e-5 (mm_projector_lr), AdamW (adamw_torch), weight decay 0
Schedule cosine, warmup_ratio=0.03, one natural epoch over the selected subset
Batch 1 GPU, per_device_train_batch_size=4, gradient_accumulation_steps=32 β†’ global batch 128
Steps natural_steps = (K + 3) // 4 // 32 (the last partial accumulation is dropped): 258 @5%, 516 @10%, 774 @15%, 1032 @20%
Sequence model_max_length=2048, image_aspect_ratio=pad, group_by_modality_length=True, lazy_preprocess=True
Precision fp16=True, bf16=False, tf32=False (NVIDIA L40S), DeepSpeed ZeRO-3 (zero3.json), gradient checkpointing
Attention training: LLaVA's xformers patch (xformers 0.0.23.post1; Tool1/Tool3 drivers forced deterministic CUTLASS kernels); inference: HF eager. FlashAttention never used
Seeds training seed ∈ {17, 29, 43} sets both seed and data_seed; the selection seed only determines the subset
Data loader dataloader_num_workers=0; images from the official LLaVA-665K image folders (coco, gqa, ocr_vqa, textvqa, vg)

Exact argument construction: src/experiments/helper_tool1_baseline_train_v4_20261102/model.py::arguments (used by the v4/v10 drivers) and src/experiments/root_native_campaign_v7_20261024/config.py (identical recipe, used for the 10% Uniform-sel1234 / source-sqrt / COINCIDE runs).

Executed trainer β‰  plain LLaVA train.py. Every run added three runtime changes: the xformers attention patch above, structured v1 label masking (turn boundaries from the conversation structure), and removal of LLaVA's always-empty optimizer group before the LR scheduler is built (without it DeepSpeed 0.12.6 misaligns per-group learning rates and the projector trains at 2e-4 instead of 2e-5). To reproduce, train with scripts/train_portable.py and evaluate with scripts/eval_portable.py from the GitHub repository; see its AGENT_CONTRACT.md.

Core-four results (mean over evaluated seeds)

Budget K Method trained (complete) evaluated per-seed core avg POPE GQA TextVQA SQA-img Avg
– 0 START (no instruction tuning; base model) – 1 25.77 63.88 2.77 4.10 32.33 25.77
5% 33,038 COINCIDE (adaptation; see notes) 3 (t17, t29, t43) 3 64.29 (t17) / 63.79 (t29) / 64.13 (t43) 85.10 53.51 51.35 66.32 64.07
5% 33,032 DataTailor (official code) 3 (t17, t29, t43) 3 63.28 (t17) / 64.25 (t29) / 63.68 (t43) 82.93 54.33 50.87 66.82 63.74
5% 33,032 Uniform random (matched-K) 3 (t17, t29, t43) 3 64.16 (t17) / 63.40 (t29) / 63.63 (t43) 84.25 52.77 52.15 65.74 63.73
10% 66,076 COINCIDE (adaptation; see notes) 3 (t17, t29, t43) 3 65.77 (t17) / 65.16 (t29) / 65.27 (t43) 85.23 55.85 54.13 66.40 65.40
10% 66,071 DataTailor (official code) 3 (t17, t29, t43) 3 65.19 (t17) / 66.15 (t29) / 65.65 (t43) 83.71 57.68 53.60 67.67 65.66
10% 66,076 LESS-approx (not faithful LESS) 3 (t17, t29, t43) 3 63.56 (t17) / 64.01 (t29) / 63.29 (t43) 82.95 50.05 53.58 67.91 63.62
10% 66,076 Source-sqrt (source-balanced random) 3 (t17, t29, t43) 3 64.93 (t17) / 64.65 (t29) / 65.32 (t43) 84.90 54.86 53.23 66.90 64.97
10% 66,076 Uniform random sel1234 3 (t17, t29, t43) 3 65.23 (t17) / 64.93 (t29) / 64.83 (t43) 84.73 55.54 52.74 67.00 65.00
10% 66,076 Uniform random sel1235 3 (t17, t29, t43) 3 65.22 (t17) / 65.46 (t29) / 65.11 (t43) 84.98 55.48 53.76 66.85 65.26
10% 66,076 Uniform random sel1236 3 (t17, t29, t43) 3 64.96 (t17) / 65.06 (t29) / 64.95 (t43) 84.22 55.52 53.49 66.73 64.99
10% 66,076 Uniform random, all 3 selection sets 9 (3 sel x t17/t29/t43) 9 see per-set rows 84.64 55.51 53.33 66.86 65.08
10% 66,076 Uniform scored-random (control) 3 (t17, t29, t43) 3 65.25 (t17) / 65.13 (t29) / 64.85 (t43) 84.98 55.18 53.08 67.08 65.08
15% 99,108 DataTailor (official code) 2 (t29, t43) 2 66.13 (t29) / 66.56 (t43) 84.78 58.83 53.79 68.00 66.35
15% 99,108 Uniform random (matched-K) 3 (t17, t29, t43) 3 65.89 (t17) / 66.25 (t29) / 66.12 (t43) 85.65 57.13 54.41 67.16 66.09
20% 132,152 COINCIDE (adaptation; see notes) 3 (t17, t29, t43) 1 66.75 (t17) 85.76 58.90 55.85 66.48 66.75
20% 132,147 DataTailor (official code) 3 (t17, t29, t43) 3 67.32 (t17) / 67.55 (t29) / 67.22 (t43) 85.28 59.91 54.87 69.39 67.36
20% 132,147 Uniform random (matched-K) 2 (t17, t43) 2 66.23 (t17) / 66.51 (t43) 85.16 58.11 55.00 67.20 66.37

Notes

  • Not uploaded: base-model weights; any images (LLaVA-665K or benchmark); our own method's checkpoints and selections; optimizer/RNG/DeepSpeed states; incomplete or warm-up states (e.g. the ICONS warm-up adapter).
  • Uniform 20% training seed 29 β€” only a partial attempt (387/1032 steps) exists; not uploaded.
  • DataTailor 15% training seed 17 β€” only a partial attempt (215/774 steps) exists; not uploaded.
  • COINCIDE 20% training seeds 29 and 43 β€” trained (1032/1032 steps, uploaded) but not evaluated (task T1).
  • Per-run evaluation outputs: <run>/eval/core4/<bank>/PREDICTIONS.jsonl + SCORE.json for the 44 evaluated runs; <run>/eval/extended/ where the extended suite was run.
  • ICONS, XMAS, CoIDO β€” no faithful run exists (ICONS has only a warm-up state); tasks T4/T9/T10 of the GitHub AGENT_CONTRACT.md.
  • Faithful LESS β€” not run. less_approx uses count-sketch SGD gradient features (the LESS paper uses Adam-preconditioned features), a 3Γ—Uniform candidate pre-filter and two benchmark-aligned dev targets (GQA-val, ScienceQA-val). It is therefore not LESS and is also not task-blind. Its set is shifted to short / yes-no answers and the models show a strong "yes" bias on POPE/MME.
  • Full-data (100%) reference β€” no complete run exists (task T8).
  • Historical allocation arms ("A-P", "A-M") appearing in our internal tables are project-internal allocation variants, not published baselines; they are omitted here.
  • Source-sqrt exists only at 10% (selection seed 1234). The 5%/15%/20% Uniform cells use selection seed 1235 with K matched to DataTailor's realised K; they are not nested with the 10% sel1235 set.
  • COINCIDE is an adaptation: the authors' TinyLLaVA/SigLIP feature route and clustering (FAISS, 10,000 clusters, spherical k-means 50 iterations, temperature 0.1) applied to the full eligible LLaVA-665K pool; its realised K is 66,076 / 33,038 / 132,152 at 10/5/20%. The selection is deterministic (no selection seed).
  • DataTailor uses the official code (commit d6fa5eb6bc2920a561c614abb6b841a128affb92); its realised K at 10% is 66,071 (five fewer than the nominal 66,076).
  • Extended-suite coverage is incomplete (24 checkpoints; list in the GitHub AGENT_CONTRACT.md, task T5). The 10% evaluation banks were used repeatedly during development; they are not a fresh held-out confirmation for anything.
  • Timing and GPU-hour accounting for the provided runs are not part of this release; please record your own (see the results template).

License

Base-model lineage (no base weights are redistributed here): LLaVA-1.5-7B (liuhaotian/llava-v1.5-7b recipe) = lmsys/vicuna-7b-v1.5 (revision 3321f76e3f527bd14065daf69dad9344000a201d, Llama 2 Community License) + openai/clip-vit-large-patch14-336 (revision ce19dc912ca5cd21c8a653c79e251e808ccabcd1) + the stage-1 projector liuhaotian/llava-v1.5-mlp2x-336px-pretrain-vicuna-7b-v1.5. The adapters are derivatives of Vicuna/Llama 2 and are subject to the Llama 2 Community License and its acceptable-use policy, in addition to the LLaVA license terms.

Data: the selections are subsets of the LLaVA-1.5 instruction-tuning mixture llava_v1_5_mix665k.json (LLaVA-Instruct / LLaVA-1.5 data, released by the LLaVA authors under CC BY 4.0; the original authors also ask users to abide by the OpenAI terms of use for the GPT-generated portions). Component datasets (COCO, GQA, OCR-VQA, TextVQA, Visual Genome, ShareGPT text) keep their own licenses. No images are redistributed.

Checkpoints

HF path Method Budget K sel seed train seed steps core avg adapter sha256
coincide/b05/selfixed-t17/ COINCIDE (adaptation; see notes) 5% 33,038 fixed 17 258/258 64.29 8a45f40e798511c52e9603edf3e2ecc8d3ed691c147c4131a64ecf789cdc68bd
coincide/b05/selfixed-t29/ COINCIDE (adaptation; see notes) 5% 33,038 fixed 29 258/258 63.79 93d2efd35b441132e7916b28a752021eb1edf65ad12388465c9af72076a312f5
coincide/b05/selfixed-t43/ COINCIDE (adaptation; see notes) 5% 33,038 fixed 43 258/258 64.13 f0e4b41b6f7ea27d2f0e2256822ed5d7fe54e0c083322e647f9d601f2d318aae
coincide/b10/sel1234-t17/ COINCIDE (adaptation; see notes) 10% 66,076 1234 17 516/516 65.77 ec05e1cf2c4fa846e88eae164791492fe7bfe032d5991e7da6da0e761090e411
coincide/b10/sel1234-t29/ COINCIDE (adaptation; see notes) 10% 66,076 1234 29 516/516 65.16 f9aa53fb95e2b38a02160230b37acd24e7d7789aa61c75749dda0b7d2d5605d8
coincide/b10/sel1234-t43/ COINCIDE (adaptation; see notes) 10% 66,076 1234 43 516/516 65.27 73ab1cc5ab79cf5578e29e4400750c5d47b44c6121aeaa6c7abf683a7673d6a7
coincide/b20/selfixed-t17/ COINCIDE (adaptation; see notes) 20% 132,152 fixed 17 1032/1032 66.75 1797baeb7b8423412ab7b21e356fe83326ebe2606560e16a7b95a0e35e0e3955
coincide/b20/selfixed-t29/ COINCIDE (adaptation; see notes) 20% 132,152 fixed 29 1032/1032 not evaluated cc5546c447b21d38fe8cc934a27598206fec55daf8c367644c439fb5a5f3dd91
coincide/b20/selfixed-t43/ COINCIDE (adaptation; see notes) 20% 132,152 fixed 43 1032/1032 not evaluated a528b174ad59753de3f43cf95df3688bb76a069af34b5e7b502ebb0ffb70679a
datatailor/b05/sel1234-t17/ DataTailor (official code) 5% 33,032 1234 17 258/258 63.28 9378fa787b7980d5ef8f8830ea6549c106085e902b15d02e4ec45a518ee17f1c
datatailor/b05/sel1234-t29/ DataTailor (official code) 5% 33,032 1234 29 258/258 64.25 b53b47b456a633e6950167587bc4f904e858565caebf5f96beecff7704f1d7b3
datatailor/b05/sel1234-t43/ DataTailor (official code) 5% 33,032 1234 43 258/258 63.68 85470fad67e9902b01548506d820aa1e0197d4dd5ddd039ac8be03e50201afe0
datatailor/b10/sel1234-t17/ DataTailor (official code) 10% 66,071 1234 17 516/516 65.19 79cecc3957338d68fb5820f56faa601e2ce199117aae8597d082334f928cd259
datatailor/b10/sel1234-t29/ DataTailor (official code) 10% 66,071 1234 29 516/516 66.15 087d8786ccbb29ccfa4fdaa4e560aced5c4f9cd9aa48ad47dc3d0777426f606f
datatailor/b10/sel1234-t43/ DataTailor (official code) 10% 66,071 1234 43 516/516 65.65 983b4230a32fa7c7eb6e44cec1e385ab25e50583a35579d9fd630b5b964e773a
datatailor/b15/sel1234-t29/ DataTailor (official code) 15% 99,108 1234 29 774/774 66.13 a09c128e80cb3b4963a235941b815c85210d5df3e266acbdf6fb1da3e9707768
datatailor/b15/sel1234-t43/ DataTailor (official code) 15% 99,108 1234 43 774/774 66.56 c26aae47c32a99d212a6e8e645ac08e90f44bb6f171657aa62ee14106b0b4e99
datatailor/b20/sel1234-t17/ DataTailor (official code) 20% 132,147 1234 17 1032/1032 67.32 6631556a24bac9aa64efac9ed634533edcf97397691749264b51a412c05e318c
datatailor/b20/sel1234-t29/ DataTailor (official code) 20% 132,147 1234 29 1032/1032 67.55 711265aecb5122a614ce1a23ce7323fc466c283c70192ab2592364e9b29197a5
datatailor/b20/sel1234-t43/ DataTailor (official code) 20% 132,147 1234 43 1032/1032 67.22 42f2edcf47ad5f4181658d8915e3b088788c8c22d99719e5057373f0c03c330d
less_approx/b10/sel1234-t17/ LESS-approx (not faithful LESS) 10% 66,076 1234 17 516/516 63.56 d89b12855323aa537ef3f909a6806ea036614980ef26cf63926df8127b2c8c5c
less_approx/b10/sel1234-t29/ LESS-approx (not faithful LESS) 10% 66,076 1234 29 516/516 64.01 42ebbe0905667a8fa2a9ab488be110fba6447ce4965f7fe4c178be76a2522130
less_approx/b10/sel1234-t43/ LESS-approx (not faithful LESS) 10% 66,076 1234 43 516/516 63.29 dc3a6e7fedbc73a88280f1cb657efd434deefa9d189017b8c8284fdd29a7367f
source_sqrt/b10/sel1234-t17/ Source-sqrt (source-balanced random) 10% 66,076 1234 17 516/516 64.93 42ef0a16fd66092e576d0fbc72732d9bf977e24dc29b671eebbca8261e2be908
source_sqrt/b10/sel1234-t29/ Source-sqrt (source-balanced random) 10% 66,076 1234 29 516/516 64.65 f0cdb1759654ec77c65f4b5c884c7b133e6252085b00f1a8bd25aadd5257cd0e
source_sqrt/b10/sel1234-t43/ Source-sqrt (source-balanced random) 10% 66,076 1234 43 516/516 65.32 a4dbde88a11735b1cfed1dc97430f8a375bbc08f1825fcfa22f4f301b0663dee
uniform/b05/sel1235-t17/ Uniform random (matched-K) 5% 33,032 1235 17 258/258 64.16 55cbfe9a9b88c92b25d3c0080ffd5f6d7b5e73a2afc12d4799f140d0d28f13d3
uniform/b05/sel1235-t29/ Uniform random (matched-K) 5% 33,032 1235 29 258/258 63.40 c10f9eb7996a191f938ee9efa49c2a13762a9ac2a8697cfbc0aeb10c0525cc76
uniform/b05/sel1235-t43/ Uniform random (matched-K) 5% 33,032 1235 43 258/258 63.63 dbb8ce202922e16746fac653ea5e0da4cf955697c0579ee0827488cec46302cb
uniform/b10/sel1234-t17/ Uniform random 10% 66,076 1234 17 516/516 65.23 fa45381c39658c43b8e732305b1db6821c7ce1ba7f5701726fcc670c37f3f320
uniform/b10/sel1234-t29/ Uniform random 10% 66,076 1234 29 516/516 64.93 557207f647bd71ce2acde7711aa1dd3d46a8bcb5f31cc08313741c1a7bb9b4f6
uniform/b10/sel1234-t43/ Uniform random 10% 66,076 1234 43 516/516 64.83 17571e3caf09d7a78528bf954d893c1f45c00e798b92a520b4c254be331bc1f5
uniform/b10/sel1235-t17/ Uniform random 10% 66,076 1235 17 516/516 65.22 a6388122e9c1ca84ce8bfb569b01bcd322f7a544dbbe002578de355c4860e884
uniform/b10/sel1235-t29/ Uniform random 10% 66,076 1235 29 516/516 65.46 2e7ed4f238ec468186ed1e86004959a4c71f197980345c7cef3a354300e1d9ce
uniform/b10/sel1235-t43/ Uniform random 10% 66,076 1235 43 516/516 65.11 ef32289f01b54dc17311bdcfe90705947e81d8656773ce6365411517e33bee21
uniform/b10/sel1236-t17/ Uniform random 10% 66,076 1236 17 516/516 64.96 c800bc8e6d973707367670eaeb62b48cd5cf4e54921cffbb54bc4a03f33c6d5d
uniform/b10/sel1236-t29/ Uniform random 10% 66,076 1236 29 516/516 65.06 309d8239df68307aaf44c798952056fbaf841538e085cf89e9cd10b750652798
uniform/b10/sel1236-t43/ Uniform random 10% 66,076 1236 43 516/516 64.95 1c84f38913d792ed80b3b45f0f2ed800e071a30735d3ad164bf7cd65e146b946
uniform/b15/sel1235-t17/ Uniform random (matched-K) 15% 99,108 1235 17 774/774 65.89 7a6c91f6c856f0f3a082c9b88c72f1b8cded2b0b9c61a24c6a7f742590de97a3
uniform/b15/sel1235-t29/ Uniform random (matched-K) 15% 99,108 1235 29 774/774 66.25 2506fc133a361a750ea72d13f8cdb3e15c5cfab6b852f0a772c55afb3c644d6b
uniform/b15/sel1235-t43/ Uniform random (matched-K) 15% 99,108 1235 43 774/774 66.12 5d1efabdfcc46c6626b13601fc436339e33a63deb41ebf5dd2a77c7c4132c1b6
uniform/b20/sel1235-t17/ Uniform random (matched-K) 20% 132,147 1235 17 1032/1032 66.23 d76c6c03ded93f779044585f6f224043fcc5ad12c17c0220ff7e5d42b7583678
uniform/b20/sel1235-t43/ Uniform random (matched-K) 20% 132,147 1235 43 1032/1032 66.51 f8aa18d86127ecc11c815da83d493909c465c49850d34b3386cc3175bd668424
uniform_scored_random/b10/sel1234-t17/ Uniform scored-random (control) 10% 66,076 1234 17 516/516 65.25 cc652c2e09d09f432f7ecdb183f910bf32a652998eba8d987102e6cf0dc245f9
uniform_scored_random/b10/sel1234-t29/ Uniform scored-random (control) 10% 66,076 1234 29 516/516 65.13 cbcde30bc7f0c9796f436d30050876c7c48aa2d70fb3cc44e653004e71732081
uniform_scored_random/b10/sel1234-t43/ Uniform scored-random (control) 10% 66,076 1234 43 516/516 64.85 8a2c89834ec428bb837454c9f681d701c7a6ee74d76a95983f916d192df3b8bc
Downloads last month
-
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for submissionpaper1234/trimdata-baseline-checkpoints

Adapter
(188)
this model