Spaces:
Running
Running
| title: ExperimentField | |
| emoji: 🔭 | |
| colorFrom: indigo | |
| colorTo: purple | |
| sdk: static | |
| pinned: false | |
| # Aspaklarya - Qwen3-VL-8B for remote-sensing VQA + captioning | |
| LoRA finetunes of **Qwen3-VL-8B-Instruct**, trained jointly on **visual | |
| question answering** and **image captioning** over overhead imagery. All model | |
| repos are private. | |
| The programme has run in **two rounds against two different answer styles**. | |
| They are not comparable to each other and are not scored on the same | |
| benchmark. | |
| | | Round 0XX | Round 1XX | | |
| |---|---|---| | |
| | runs | Aspaklarya-001 ... 021 (21, 20 evaluated) | Aspaklarya-101 ... 107 | | |
| | VQA column | `VQA` | `VQA_1126` | | |
| | reference answers | ~4 characters (`yes`, `3`, `river`) | ~300 characters, grounded reasoning | | |
| | model output | one word | 270-295 characters | | |
| | benchmark | 1,000-image union draw, 2,000 graded items | four-slice seed-1105 draw, 1,991 graded items | | |
| | best | Aspaklarya-020-step4500 | Aspaklarya-101-step4500 | | |
| | what it answered | hyperparameters are saturated, data is the lever | verbose supervision is worth +8 on DM VQA; data is still the only lever | | |
| Both rounds are now scored on the **same four-slice benchmark**, so the | |
| MEAN4 column below is comparable across rounds. The *legacy* and *new VQA* | |
| columns individually are not: The | |
| two benchmarks reward opposite answer styles; see *Why the legacy score | |
| drops* below. | |
| ## Round 0XX - terse answers (Aspaklarya-001 ... 021) | |
| The original 21-experiment programme, trained on the one-word `VQA` | |
| column and scored on the 1,000-image union draw. Everything below this | |
| heading refers to that round only. | |
| ### Setup | |
| | Item | Value | | |
| |---|---| | |
| | Base | Qwen3-VL-8B-Instruct | | |
| | Method | LoRA (Unsloth), bf16 compute; 4-bit base for runs 001-017, **bf16 base from 018 on** | | |
| | Targets | all linear layers, vision + language towers | | |
| | Data | 6,288 images / 18,864 samples for runs 001-019; **11,994 / 35,981 for 020-021** (1 caption + 2 QA each) | | |
| | Caption prompts | sampled from a 30-prompt pool, so the model learns the task not one phrasing | | |
| | Eval | fixed 1000-image draw = 2000 graded items, same seed for every model | | |
| | Scoring | LLM-as-judge, 0-100 per item | | |
| | References | stock Qwen3-VL-8B, plus an in-house finetuned model | | |
| Winners are merged into the bf16 base and re-validated. For runs 001-017 that merge also removes a train/serve mismatch, since those adapters were fitted on a 4-bit base but are served from bf16 weights; runs 018 onward have no such mismatch. | |
| ### Configurations | |
| | Model | r / alpha | Dropout | LR | Sched | Vision | Per image | Items | Change | | |
| |---|---|---|---|---|---|---|---|---| | |
| | Aspaklarya-001 | 16 / 16 | 0.0 | 2e-4 | linear | trained | 1c + 2q | 18,864 | reference recipe | | |
| | Aspaklarya-002 | 32 / 64 | 0.05 | 1e-4 | cosine | trained | 1c + 2q | 18,864 | higher rank, lower LR | | |
| | Aspaklarya-003 | 16 / 16 | 0.0 | 2e-4 | linear | trained | 2c + 4q | 37,728 | 2x resampled data | | |
| | Aspaklarya-004 | 16 / 16 | 0.0 | 2e-4 | linear | **frozen** | 1c + 2q | 18,864 | language-only | | |
| | Aspaklarya-005 | 16 / 16 | 0.0 | 2e-4 | cosine | trained | 1c + 6q | 43,864 | VQA-heavy | | |
| | Aspaklarya-006 | 16 / 16 | 0.0 | 2e-4 | linear | trained | 1c + 2q | 18,864 | batch 4 x accum 8 | | |
| | Aspaklarya-007 | 8 / 8 | 0.0 | 2e-4 | linear | trained | 1c + 2q | 18,864 | low rank | | |
| | Aspaklarya-008 | 16 / 16 | 0.0 | 1e-4 | linear | trained | 1c + 4q | 31,440 | more QA, lower LR | | |
| | Aspaklarya-009 | 32 / 32 | 0.0 | 2e-4 | linear | trained | 1c + 2q | 18,864 | high rank | | |
| | Aspaklarya-010 | 16 / 16 | 0.0 | 2e-4 | linear | trained | 1c + 2q | 18,864 | wider prompt pool | | |
| | Aspaklarya-011 | 16 / 16 | 0.05 | 2e-4 | linear | trained | 1c + 2q | 18,864 | dropout, wd 0.05, clip 1.0 | | |
| | Aspaklarya-012 | 16 / 16 | 0.0 | 2e-4 | linear | trained | 1c + 2q | 18,864 | as 011, no dropout | | |
| | Aspaklarya-013 | 16 / 16 | 0.0 | 1e-4 | linear | trained | 1c + 2q | 18,864 | low LR, long schedule | | |
| | Aspaklarya-014 | 16 / 16 | 0.0 | 2e-4 | linear | trained | 1c + 2q | 18,864 | effective batch 16 | | |
| | Aspaklarya-015 | 16 / 16 | 0.0 | 2e-4 | linear | trained | 1c + 2q | 18,864 | effective batch 64 | | |
| | Aspaklarya-016 | 16 / 16 | 0.0 | **4e-4** | linear | trained | 1c + 2q | 18,864 | batch 64 + linear LR scaling | | |
| | Aspaklarya-017 | 16 / 16 | 0.0 | **3e-4** | linear | trained | 1c + 2q | 18,864 | batch 64 + sqrt LR scaling | | |
| | Aspaklarya-018 | 16 / 16 | 0.0 | 2e-4 | linear | trained | 1c + 2q | 18,864 | **bf16 base** (no 4-bit) | | |
| | Aspaklarya-019 | 16 / 16 | 0.05 | 2e-4 | linear | trained | 1c + 2q | 18,864 | **bf16 base** + 011 recipe | | |
| | Aspaklarya-020 | 16 / 16 | 0.0 | 2e-4 | linear | trained | 1c + 2q | **35,981** | bf16 base + **+91% data** | | |
| | Aspaklarya-021 | 16 / 16 | 0.0 | 2e-4 | linear | trained | 1c + 2q | **35,981** | as 020, **8 epochs** not 4 | | |
| Constant everywhere: effective batch 32 (except 014 at 16, 015-017 at 64), warmup 0.03, grad clip 0.3 (1.0 for 011/012), seed 3407. | |
| ### Results | |
| Best checkpoint per run on a fixed 1000-image draw (2000 graded items), with bootstrap standard error. **Overall** = mean of caption and VQA. **In-dom.** = the target overhead-imagery subset (491 images); **RSICD** = the public RSICD remote-sensing benchmark subset (509 images). Sorted by in-domain overall; **bold marks the best value in each column**. | |
| | Model | In-dom. cap | In-dom. vqa | In-dom. overall | All cap | All vqa | All overall | RSICD cap | RSICD vqa | RSICD overall | | |
| |---|---|---|---|---|---|---|---|---|---| | |
| | Aspaklarya-011 | 81.85 ±0.66 | 83.51 ±1.48 | **82.68 ±0.84** | 79.92 ±0.49 | 75.78 ±1.13 | 77.85 ±0.64 | 78.06 ±0.67 | 68.33 ±1.64 | 73.20 ±0.92 | | |
| | Aspaklarya-019 | 82.39 ±0.65 | 82.82 ±1.48 | 82.61 ±0.85 | 80.33 ±0.48 | 75.91 ±1.13 | 78.12 ±0.63 | 78.34 ±0.70 | 69.24 ±1.62 | 73.79 ±0.89 | | |
| | **Aspaklarya-020** | **82.74 ±0.67** | 82.10 ±1.50 | 82.42 ±0.85 | **81.31 ±0.47** | 76.08 ±1.13 | **78.70 ±0.63** | **79.93 ±0.63** | 70.28 ±1.63 | **75.10 ±0.91** | | |
| | Aspaklarya-001 | 81.84 ±0.66 | 83.23 ±1.44 | 82.54 ±0.81 | 79.97 ±0.48 | 75.94 ±1.12 | 77.95 ±0.63 | 78.17 ±0.69 | 68.89 ±1.66 | 73.53 ±0.93 | | |
| | Aspaklarya-018 | 82.28 ±0.69 | 82.70 ±1.51 | 82.49 ±0.86 | 80.46 ±0.49 | 75.80 ±1.11 | 78.13 ±0.63 | 78.70 ±0.67 | 69.15 ±1.61 | 73.92 ±0.90 | | |
| | Aspaklarya-015 | 82.14 ±0.64 | 82.76 ±1.46 | 82.45 ±0.82 | 80.07 ±0.49 | 74.73 ±1.13 | 77.40 ±0.63 | 78.07 ±0.72 | 66.98 ±1.65 | 72.52 ±0.92 | | |
| | Aspaklarya-006 | 81.40 ±0.71 | 82.95 ±1.50 | 82.18 ±0.90 | 79.81 ±0.50 | 75.79 ±1.12 | 77.80 ±0.65 | 78.26 ±0.68 | 68.88 ±1.63 | 73.57 ±0.91 | | |
| | Aspaklarya-021 | 82.14 ±0.70 | 81.93 ±1.53 | 82.03 ±0.88 | 80.93 ±0.48 | 75.41 ±1.13 | 78.17 ±0.65 | 79.76 ±0.64 | 69.12 ±1.65 | 74.44 ±0.92 | | |
| | Aspaklarya-005 | 81.37 ±0.67 | 82.37 ±1.50 | 81.87 ±0.85 | 79.55 ±0.49 | 75.74 ±1.11 | 77.64 ±0.62 | 77.79 ±0.71 | 69.35 ±1.65 | 73.57 ±0.93 | | |
| | Aspaklarya-012 | 81.68 ±0.66 | 82.05 ±1.53 | 81.86 ±0.86 | 79.73 ±0.48 | 75.02 ±1.13 | 77.37 ±0.64 | 77.85 ±0.71 | 68.23 ±1.63 | 73.04 ±0.90 | | |
| | Aspaklarya-014 | 81.51 ±0.67 | 81.93 ±1.48 | 81.72 ±0.83 | 79.95 ±0.49 | 74.77 ±1.13 | 77.36 ±0.63 | 78.45 ±0.71 | 67.87 ±1.66 | 73.16 ±0.92 | | |
| | Aspaklarya-003 | 81.23 ±0.71 | 82.18 ±1.47 | 81.71 ±0.85 | 79.55 ±0.49 | 75.52 ±1.12 | 77.53 ±0.63 | 77.92 ±0.66 | 69.10 ±1.66 | 73.51 ±0.91 | | |
| | Aspaklarya-017 | 81.49 ±0.67 | 81.86 ±1.48 | 81.67 ±0.85 | 79.50 ±0.50 | 75.73 ±1.10 | 77.62 ±0.63 | 77.59 ±0.70 | 69.83 ±1.60 | 73.71 ±0.89 | | |
| | Aspaklarya-001 (ep5) | 80.60 ±0.72 | 82.25 ±1.49 | 81.42 ±0.84 | 79.30 ±0.50 | **76.17 ±1.12** | 77.73 ±0.63 | 78.05 ±0.69 | **70.31 ±1.64** | 74.18 ±0.91 | | |
| | Aspaklarya-013 | 80.58 ±0.69 | 82.20 ±1.48 | 81.39 ±0.84 | 79.11 ±0.49 | 75.35 ±1.10 | 77.23 ±0.61 | 77.70 ±0.69 | 68.74 ±1.60 | 73.22 ±0.90 | | |
| | Aspaklarya-009 | 80.01 ±0.71 | 82.34 ±1.46 | 81.18 ±0.84 | 78.53 ±0.52 | 75.85 ±1.11 | 77.19 ±0.63 | 77.09 ±0.73 | 69.59 ±1.63 | 73.34 ±0.91 | | |
| | Aspaklarya-008 | 80.62 ±0.74 | 81.42 ±1.51 | 81.02 ±0.86 | 79.82 ±0.50 | 75.40 ±1.12 | 77.61 ±0.63 | 79.05 ±0.65 | 69.60 ±1.62 | 74.33 ±0.89 | | |
| | Aspaklarya-007 | 81.13 ±0.71 | 80.82 ±1.51 | 80.97 ±0.86 | 79.33 ±0.51 | 75.33 ±1.12 | 77.33 ±0.64 | 77.60 ±0.70 | 70.03 ±1.62 | 73.81 ±0.90 | | |
| | Aspaklarya-004 | 80.31 ±0.72 | 81.47 ±1.48 | 80.89 ±0.84 | 78.76 ±0.51 | 75.33 ±1.11 | 77.04 ±0.64 | 77.26 ±0.74 | 69.40 ±1.61 | 73.33 ±0.92 | | |
| | Aspaklarya-002 | 80.70 ±0.72 | 80.63 ±1.51 | 80.67 ±0.85 | 79.36 ±0.50 | 74.33 ±1.13 | 76.85 ±0.64 | 78.07 ±0.71 | 68.25 ±1.63 | 73.16 ±0.92 | | |
| | Aspaklarya-010 | 80.50 ±0.74 | 80.66 ±1.58 | 80.58 ±0.90 | 78.88 ±0.52 | 74.28 ±1.13 | 76.58 ±0.64 | 77.32 ±0.72 | 68.12 ±1.65 | 72.72 ±0.91 | | |
| | in-house reference | 71.88 ±1.23 | **83.62 ±1.49** | 77.75 ±1.02 | 65.18 ±0.83 | 73.34 ±1.13 | 69.26 ±0.74 | 58.72 ±1.09 | 63.44 ±1.62 | 61.08 ±1.02 | | |
| | Qwen3-VL-8B (base) | 68.37 ±0.89 | 58.94 ±1.50 | 63.66 ±0.92 | 65.68 ±0.66 | 56.01 ±1.01 | 60.84 ±0.62 | 63.07 ±0.94 | 53.18 ±1.46 | 58.13 ±0.91 | | |
| Every finetune beats both references overall. The selected model, Aspaklarya-020, is **+17.9** over the base model and **+9.4** over the in-house reference on the full set. | |
| ### Learning rate at large batch | |
| Aspaklarya-015 gave the best captioning in the table at effective batch 64, but it was trained at 2e-4 — the rate tuned for batch 32. Two follow-ups tested whether the batch-64 recipe was simply under-stepped. Scores below come from a separate 500-image draw used for live checkpoint selection, so they compare only with each other. | |
| | LR at batch 64 | ep 1 | ep 2 | ep 3 | ep 4 | ep 5 | Outcome | | |
| |---|---|---|---|---|---|---| | |
| | 2e-4 (015) | 76.92 | 77.03 | 77.25 | 77.64 | **77.92** | peak at epoch 5 | | |
| | 4e-4 (016) | **77.67** | 76.93 | 76.42 | 76.89 | — | stopped at epoch 4, declining | | |
| | 3e-4 (017) | 77.00 | 76.56 | 77.16 | 77.68 | 77.86 | peak at epoch 5, stopped at epoch 6 | | |
| Linear LR scaling is **wrong** for LoRA here: 4e-4 peaked at epoch 1 and then lost ground at every later checkpoint, ending below where 2e-4 started. The adapter's effective step size does not follow the full-finetune scaling rule, so raising LR with batch size overshoots rather than compensating. The intermediate rate 3e-4 traced almost exactly the same curve as 2e-4 and peaked at the same epoch, one tenth of a point lower. | |
| On the shared 1000-image set, neither batch-64 rate beat the reference: 2e-4 is **-0.56** overall and **-0.09** in-domain, 3e-4 is **-0.34** and **-0.87**. Effective batch 64 is therefore refuted at every learning rate tried, and the loss is consistently in VQA while captioning holds or improves. Together with the batch-16 result this closes the batch-size and learning-rate axes: **no hyperparameter lever remains untested**, and further gains have to come from the training data. | |
| ### Error bars | |
| Bootstrap over images, 3000 resamples. Absolute standard error: | |
| | Slice | Caption | VQA | Overall | | |
| |---|---|---|---| | |
| | All (1000 img) | 0.48 | 1.12 | 0.63 | | |
| | In-domain (491) | 0.66 | 1.46 | 0.82 | | |
| | RSICD (509) | 0.69 | 1.64 | 0.91 | | |
| Paired model-vs-model error is smaller because image difficulty cancels: **0.42** overall on the full set, **0.56** in-domain. Use the paired figure for any "A beats B" claim. | |
| Applied exhaustively to the five leading checkpoints (020, 018, 019, 001, 011), the paired bootstrap separates **0 of 10 pairs overall and 0 of 10 in-domain** at 95% — in-domain every pair sits at p ≥ 0.67. The only significant pairs anywhere are three on the RSICD slice, all of them 020 against another model. **The ordering of the results table below the top line is not a measurement.** | |
| ### Caveat | |
| The hyperparameter runs are statistically tied with each other: among runs 001-019, every confidence interval against the leader straddles zero. **VQA error is ~2.3x caption error**, so no finetune-vs-finetune VQA difference in any column is resolvable; only caption gaps, the gaps to the two references, and the data-scaling result of run 020 are real. Doubling the eval set from 500 to 1000 images did not separate the tied field, and it reshuffled its order. | |
| ### Findings | |
| - **Hyperparameters are saturated; data is not.** LR, rank, alpha, dropout, weight decay, gradient clipping, batch geometry, effective batch size, LR scaling at large batch and base-weight precision were all swept without beating the noise floor. Adding 91% more training images cleared it on the first attempt. | |
| - **The slices disagree — but data breaks the trade.** Ranking by in-domain score almost inverts the RSICD ranking, and every hyperparameter lever that moved one slice moved the other the opposite way. Extra data is the exception: it lifted RSICD by 1.57 while leaving in-domain unchanged. | |
| - **Captioning is the real gain.** It improves on every slice, by +14 over the base model and +10 to +19 over the in-house reference. | |
| - **The in-domain VQA win is not established.** The in-house reference posts the single highest in-domain VQA score in the table; it collapses outside that domain, where it falls below the untuned base model. | |
| - **The vision tower matters.** Freezing it is the one ablation that consistently hurt. | |
| - **Extra QA per image does not help.** 4-QA and 6-QA mixtures tie with 2-QA. | |
| - **Larger batch trades generality for domain fit.** Effective batch 64 gives the best in-domain captioning in the table and the worst RSICD VQA. | |
| - **Training in bf16 instead of 4-bit is a small, consistent positive** — the only change in the whole sweep that never loses on any slice — but it is still inside the error bar. | |
| - **Data is the remaining lever**, not hyperparameters — and it is not exhausted: the run that used it had not converged when its schedule ended. | |
| ### Base-weight precision | |
| Runs 001-017 all trained their adapters against **4-bit quantized** base weights, while evaluation and the final merge both use the **bf16** base. That is a train/serve mismatch: the adapter was fitted to compensate for quantization error that is absent at inference time. | |
| Runs **018** and **019** repeat the two strongest recipes with the base loaded in bf16 and nothing else changed. Against the reference, paired on 1000 images: | |
| | run | overall | in-domain | RSICD | | |
| |---|---|---|---| | |
| | Aspaklarya-018 | **+0.17 ±0.42** | -0.05 ±0.59 | **+0.39 ±0.59** | | |
| | Aspaklarya-019 | **+0.16 ±0.40** | **+0.07 ±0.57** | **+0.25 ±0.57** | | |
| These are the first two runs in the sweep to post a positive overall delta, and **018** is the only run that is not worse than the reference on any slice. Both are still well inside the paired error bar (p = 0.34 that each is in fact worse), so neither is promoted. | |
| The mechanism is not the expected one. Removing quantization error should mostly help the harder task, VQA; instead the gain is **entirely in captioning** (+0.48 and +0.35) while VQA is flat to slightly negative. That is the same caption-up / VQA-flat signature produced by every other lever in the sweep, which suggests precision is acting through the same saturated channel rather than unlocking anything new. | |
| Removing quantization also costs **~9 GB of GPU memory** during training (19 GB vs 10 GB), which is a real constraint on a shared 40 GB card. | |
| ### Training data | |
| Every run up to 017 trained on 6,288 images / 18,864 samples. The source table also carries **5,706 further images** that the dataset author left unassigned (`Notes == other`, RSICD only). They are fully populated, statistically indistinguishable from the assigned training rows, and have **zero image-name overlap with either the train or the validation split**, so they are safe to train on. | |
| **Aspaklarya-020** adds them, taking training data to 11,994 images / 35,981 samples (**+91%**). Everything else is identical to 018. Paired against the previous selection on the same 1000 images: | |
| | slice | delta | 95% CI | p(worse) | | |
| |---|---|---|---| | |
| | overall | **+0.74 ±0.47** | [-0.17, 1.66] | 0.059 | | |
| | in-domain | -0.12 ±0.63 | [-1.37, 1.10] | 0.570 | | |
| | **RSICD** | **+1.57 ±0.68** | **[0.22, 2.87]** | **0.012** | | |
| | caption (all) | **+1.34 ±0.42** | — | — | | |
| **This is the first change in the whole programme to clear the error bar.** The RSICD interval excludes zero, and the overall caption gain is 3.2 standard errors. It is also the first result that does **not** trade one slice against the other: in-domain is a clean tie while the general slice moves. | |
| Two further points matter for anyone repeating this: | |
| * **The in-domain penalty is transient.** At matched optimizer steps the extra data looked actively harmful early — step 2,360 was 1.65 behind on in-domain — and that deficit had vanished by step 4,500. Stopping early would have refuted the best run in the programme. | |
| * **The climb to the final checkpoint was schedule, not headroom.** Overall score rose at all seven checkpoints including the last, which suggested an unexhausted budget. **Aspaklarya-021 tested that directly and refuted it** — see below. | |
| More data is therefore the live axis, and the only one still producing movement. | |
| ### Schedule length | |
| **Aspaklarya-021** repeats 020 on the same enlarged corpus with **8 epochs instead of 4** (9,000 steps, checkpoints step-aligned with 020). It peaked at step 4,720 and then declined; the stop rule ended it at step 5,900 with 3,100 steps unspent. | |
| | step | overall | caption | vqa | in-domain | | |
| |---|---|---|---|---| | |
| | 1,180 | 77.66 | 79.19 | 76.12 | 81.56 | | |
| | 2,360 | 78.23 | 80.55 | 75.92 | 81.25 | | |
| | 3,540 | 78.37 | 80.33 | 76.41 | 82.18 | | |
| | **4,720** | **78.69** | 80.86 | **76.52** | 82.39 | | |
| | 5,900 | 77.60 | 80.75 | 74.45 | 81.15 | | |
| Its best checkpoint, paired against 020 on the same 1000 images, is negative on every slice and significant on none: overall **-0.53 ±0.43**, in-domain **-0.39 ±0.60**, RSICD **-0.66 ±0.64**. | |
| **Data volume helped; data duration did not.** Four epochs is already the right length at this corpus size. | |
| The failure mode is asymmetric and worth recording: at step 5,900 **captioning was intact (80.75) while VQA collapsed (76.52 → 74.45)**. Overtraining this recipe degrades VQA first, so a stop rule watching captioning alone would not catch it. | |
| ### Status | |
| The **hyperparameter** search is closed. Rank, alpha, dropout, learning rate, schedule, weight decay, gradient clipping, vision-tower freezing, QA-per-image, caption prompt pool, batch geometry, effective batch size, learning rate at large batch and base-weight precision were all swept across 19 recipes under identical conditions. None displaced the reference recipe by more than the paired error bar. | |
| Batch size and learning rate illustrate that ceiling. Effective batch 16 came in at **-0.59** overall, batch 64 at 2e-4 at **-0.56**, and batch 64 at 3e-4 at **-0.34** overall and **-0.87** in-domain; batch 64 at 4e-4 was stopped at epoch 4 while declining. In each case captioning held steady or improved while VQA fell. | |
| Enlarging the training set did what none of those could, and training it longer did not extend the gain. That narrows the remaining question to **data composition**: the corpus added was RSICD-only, and the gain it produced is correspondingly RSICD-only while in-domain stays flat. Additional in-domain imagery is the targeted next step. | |
| ## Round 1XX - verbose grounded answers (Aspaklarya-101 ... 111) | |
| ### Why a second round | |
| The 0XX round closed with hyperparameters saturated and data identified as | |
| the only live lever. It also left a product problem: every 0XX model answers | |
| in one word, because its supervision was one word. `yes` is a correct answer | |
| to *"are there any anomalies in the scene?"* and a useless one. | |
| The dataset author then supplied a new column, **`VQA_1126`**, replacing the | |
| terse answers for the in-domain (DM) images with ~300-character grounded | |
| answers that name objects, locations and context. The RSICD rows are | |
| byte-identical to the old column; only the 5,148 DM rows changed, from a mean | |
| of 5.9 characters to ~304. | |
| ### Configurations | |
| All runs share one corpus recipe - `VQA_1126`, 1 caption + 2 QA per image - | |
| and one optimiser: bf16 base, seed 3407, lr 2e-4, linear schedule, warmup | |
| 0.03, weight decay 0.001, grad clip 0.3, effective batch 32 (4 x 8), 4 epochs, | |
| max seq 2048, vision tower trained. **One variable each.** | |
| | Model | variable | split | items / images | cost | | |
| |---|---|---|---|---| | |
| | Aspaklarya-101 | **1.9x data** | `train,other` | 35,979 / 11,994 | 9.8 h | | |
| | Aspaklarya-102 | reference | `train` | 18,862 / 6,288 | 5.4 h | | |
| | Aspaklarya-103 | LoRA **rank** 16 -> 32, scale held at 1.0 | `train` | 18,862 / 6,288 | 5.4 h | | |
| | Aspaklarya-104 | LoRA **scale** `alpha/r` 1.0 -> 2.0, rank 16 | `train` | 18,862 / 6,288 | 5.4 h | | |
| | Aspaklarya-105 | **objective balance**, `w_caption` 1.0 -> 0.4 | `train` | 18,862 / 6,288 | 5.4 h | | |
| | Aspaklarya-106 | **LoRA soup** of the four epoch-3 adapters | n/a | n/a | **0 h** | | |
| | Aspaklarya-107 | **cosine** schedule, lr 1e-4, warmup 0.10, 3 epochs | `train` | 18,862 / 6,288 | 3.8 h | | |
| | Aspaklarya-108 | **seed replication** of 102 (1234, 2024) | `train` | 18,862 / 6,288 | 2 x 5.4 h | | |
| | Aspaklarya-109 | **seed replication** of 101 (1234) | `train,other` | 35,979 / 11,994 | 9.9 h | | |
| | Aspaklarya-110 | **seed replication** of 101 (2024) | `train,other` | 35,979 / 11,994 | 11.2 h | | |
| | Aspaklarya-111 | **QA depth** 2 -> 4 per image, caption 1 -> 2 | `train,other` | 71,944 / 11,994 | ~16 h | | |
| ### Hyperparameters in full | |
| Every value actually passed to the trainer, read back from each run's | |
| `run_config.json`. Cells that differ from the Aspaklarya-102 reference are | |
| **bold** - that is the single variable the run was built to test. | |
| | | 101 | 102 (ref) | 103 | 104 | 105 | 106 / 106b | 107 | 108 | 109 | 110 | 111 | | |
| |---|---|---|---|---|---|---|---|---|---|---|---| | |
| | split | **`train,other`** | `train` | `train` | `train` | `train` | n/a | `train` | `train` | **`train,other`** | **`train,other`** | **`train,other`** | | |
| | QA per image | 2 | 2 | 2 | 2 | 2 | n/a | 2 | 2 | 2 | 2 | **4** | | |
| | captions per image | 1 | 1 | 1 | 1 | 1 | n/a | 1 | 1 | 1 | 1 | **2** | | |
| | items | **35,979** | 18,862 | 18,862 | 18,862 | 18,862 | n/a | 18,862 | 18,862 | **35,979** | **35,979** | **71,944** | | |
| | images | **11,994** | 6,288 | 6,288 | 6,288 | 6,288 | n/a | 6,288 | 6,288 | **11,994** | **11,994** | **11,994** | | |
| | LoRA rank | 16 | 16 | **32** | 16 | 16 | 64 | 16 | 16 | 16 | 16 | 16 | | |
| | LoRA alpha | 16 | 16 | **32** | **32** | 16 | 64 | 16 | 16 | 16 | 16 | 16 | | |
| | scale `alpha/r` | 1.0 | 1.0 | 1.0 | **2.0** | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | | |
| | LoRA dropout | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | n/a | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | | |
| | vision tower | trained | trained | trained | trained | trained | trained | trained | trained | trained | trained | trained | | |
| | `w_caption` | 1.0 | 1.0 | 1.0 | 1.0 | **0.4** | n/a | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | | |
| | `w_vqa` | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | n/a | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | | |
| | loss weighting | token | token | token | token | token | n/a | token | token | token | token | token | | |
| | epochs | 4 | 4 | 4 | 4 | 4 | n/a | **3** | 4 | 4 | 4 | **3** | | |
| | learning rate | 2e-4 | 2e-4 | 2e-4 | 2e-4 | 2e-4 | n/a | **1e-4** | 2e-4 | 2e-4 | 2e-4 | 2e-4 | | |
| | scheduler | linear | linear | linear | linear | linear | n/a | **cosine** | linear | linear | linear | linear | | |
| | warmup ratio | 0.03 | 0.03 | 0.03 | 0.03 | 0.03 | n/a | **0.10** | 0.03 | 0.03 | 0.03 | 0.03 | | |
| | weight decay | 0.001 | 0.001 | 0.001 | 0.001 | 0.001 | n/a | 0.001 | 0.001 | 0.001 | 0.001 | 0.001 | | |
| | grad clip | 0.3 | 0.3 | 0.3 | 0.3 | 0.3 | n/a | 0.3 | 0.3 | 0.3 | 0.3 | 0.3 | | |
| | batch x accum | 4 x 8 | 4 x 8 | 4 x 8 | 4 x 8 | 4 x 8 | n/a | 4 x 8 | 4 x 8 | 4 x 8 | 4 x 8 | 4 x 8 | | |
| | effective batch | 32 | 32 | 32 | 32 | 32 | n/a | 32 | 32 | 32 | 32 | 32 | | |
| | max seq length | 2048 | 2048 | 2048 | 2048 | 2048 | n/a | 2048 | 2048 | 2048 | 2048 | 2048 | | |
| | max image side | 1024 | 1024 | 1024 | 1024 | 1024 | n/a | 1024 | 1024 | 1024 | 1024 | 1024 | | |
| | seed | 3407 | 3407 | 3407 | 3407 | 3407 | n/a | 3407 | **1234 / 2024** | **1234** | **2024** | 3407 | | |
| | base precision | bf16 | bf16 | bf16 | bf16 | bf16 | bf16 | bf16 | bf16 | bf16 | bf16 | bf16 | | |
| | save every | 590 | 295 | 295 | 295 | 295 | n/a | 295 | 295 | 590 | 590 | 1124 | | |
| | steps / epoch | 1125 | 590 | 590 | 590 | 590 | n/a | 590 | 590 | 1125 | 1125 | **2249** | | |
| | total steps | 4500 | 2360 | 2360 | 2360 | 2360 | n/a | 1770 | 2360 | 4500 | 4500 | **6747** | | |
| | wall clock | 9.8 h | 5.4 h | 5.4 h | 5.4 h | 5.4 h | **0 h** | 3.8 h | 2 x 5.4 h | 9.9 h | 11.2 h | ~16 h | | |
| Constant across the whole round and therefore omitted above: base model | |
| `Qwen/Qwen3-VL-8B-Instruct`, no 4-bit quantisation, VQA column `VQA_1126`, | |
| caption prompts drawn from the 50-query bank, eval-loss probe of 400 held-out | |
| items (200 caption / 200 VQA). | |
| **Aspaklarya-106 and 106b are not training runs.** They are exact LoRA soups: | |
| four rank-16 adapters stacked into a single rank-64 adapter, which reproduces | |
| the arithmetic mean of the parent weight deltas to within 2.3e-9. 106 averages | |
| the epoch-3 parents, 106b the epoch-4 parents. Neither consumed GPU training | |
| time; both are listed here so their rank and alpha are not mistaken for a | |
| tuned choice. | |
| **Aspaklarya-108, 109 and 110 are replications, not new configurations.** | |
| 108 re-runs 102 at two further seeds; 109 and 110 re-run 101 at one further | |
| seed each. They exist to measure the run-to-run spread that every comparison | |
| on this card silently assumed was small, and to test whether the one positive | |
| result survives a change of seed. They are measurements, not deliverables. | |
| ### How these are scored | |
| Round-1XX checkpoints were first ranked on the DM VQA slice alone - the slice | |
| the round was trained for. Every model in the programme has since been | |
| re-judged on **one fixed sample drawn with seed 1105**, drawn once outside the | |
| model loop so all 24 checkpoints see byte-identical items: | |
| | slice | prompt set | new tokens | n | | |
| |---|---|---|---| | |
| | VQA DM | `VQA_1126` (verbose) | 160 | 500 | | |
| | VQA RSICD | legacy terse | 48 | 500 | | |
| | Caption DM | caption-query bank | 320 | 491 | | |
| | Caption RSICD | caption-query bank | 320 | 500 | | |
| `MEAN4` is the unweighted mean, so a model cannot win by specialising on one | |
| slice. Judge: `gemma-4-31b-it-imiai`, greedy, temperature 0, scored 0-100. | |
| Two caveats on the shape of the benchmark. **CAP DM carries 491 items, not | |
| 500**, because the underlying 500-image sweep drew 491 DM / 509 RSICD. And | |
| **RSICD VQA has no verbose rewrite** - it is still the legacy question set | |
| generated at a 48-token budget, which clips verbose answers and penalises the | |
| 1XX models there. | |
| ### Results | |
| **One row per experiment: its best checkpoint.** Every model is scored on the | |
| same 1,991 judged items under the same rubric, so the column is comparable all | |
| the way down. The `ckpts scored` column says how many checkpoints that | |
| experiment contributed; the best of n is an **order statistic**, so it is | |
| biased upward by roughly 0.3-0.5 for n=4 and should not be read as that | |
| recipe's expected quality. Use it to rank recipes, not to size effects. | |
| For that, and for every individual checkpoint, see **Run-to-run sigma** below | |
| and the full 36-row table in `rescore_slices.csv`. Seed replicates of one | |
| recipe (108, 109, 110) are pooled into a single row here. | |
| | Model | VQA DM | VQA RSICD | CAP DM | CAP RSICD | MEAN4 | ckpts scored | | |
| |---|---|---|---|---|---|---| | |
| | **Aspaklarya-101-step4500** | 74.16 | 70.56 | 82.53 | 79.64 | **76.72** | 4 | | |
| | Aspaklarya-109-step3540 | 72.84 | 71.31 | 82.49 | 79.62 | **76.56** | 2 | | |
| | **Aspaklarya-106-soup** | 74.57 | 69.38 | 82.38 | 79.1 | **76.36** | 1 | | |
| | Aspaklarya-110-step3540 | 73.68 | 71.01 | 80.99 | 79.32 | **76.25** | 2 | | |
| | **Aspaklarya-104-step2360** | 73.72 | 69.59 | 82.7 | 78.88 | **76.22** | 4 | | |
| | **Aspaklarya-103-step2360** | 74.13 | 69.82 | 81.34 | 79.3 | **76.15** | 4 | | |
| | Aspaklarya-108-step2360 | 72.66 | 70.83 | 82.1 | 78.58 | **76.04** | 4 | | |
| | **Aspaklarya-105-step1770** | 74.49 | 68.96 | 81.96 | 78.62 | **76.01** | 4 | | |
| | Aspaklarya-106b-soup | 72.77 | 69.99 | 81.53 | 78.73 | **75.75** | 1 | | |
| | **Aspaklarya-102-step1770** | 73.45 | 68.93 | 81.56 | 78.6 | **75.64** | 4 | | |
| | Aspaklarya-107-step1770 | 72.16 | 69.9 | 80.98 | 77.06 | **75.03** | 3 | | |
| | **Aspaklarya-020-step4500** | 66.18 | 70.65 | 82.79 | 79.99 | **74.9** | 1 | | |
| | in-house reference | 65.26 | 63.59 | 71.91 | 58.58 | **64.84** | - | | |
| | Qwen3-VL-8B (base) | 60.64 | 53.13 | 68.31 | 63.28 | **61.34** | - | | |
| ### Run-to-run sigma, measured | |
| The previous version of this card said sigma had never been measured and that | |
| no one-point difference should be trusted. That has now been fixed, at the | |
| cost of four replication runs. | |
| **Aspaklarya-108** re-ran the 102 recipe at seeds 1234 and 2024, changing | |
| nothing else. **Aspaklarya-109 and 110** did the same for the 101 recipe, at | |
| seeds 1234 and 2024. With the two original seed-3407 runs that gives two arms | |
| of six scored checkpoints each. | |
| | arm | corpus | checkpoints (MEAN4) | mean | sigma | | |
| |---|---|---|---|---| | |
| | `train,other` (101, 109, 110) | 35,979 | 76.72 · 76.56 · 76.38 · 76.25 · 76.23 · 76.09 | **76.37** | 0.23 | | |
| | `train` (102, 108) | 18,862 | 76.04 · 75.64 · 75.61 · 75.50 · 75.23 · 75.22 | **75.54** | 0.31 | | |
| **sigma = 0.31 MEAN4 for a single checkpoint** (pooled, the wider arm). Two | |
| consequences, and they are not optional: | |
| - A single run targeting less than about **1.0 point** cannot be interpreted. | |
| - The difference between **two** single checkpoints carries SE `0.31 x sqrt(2) | |
| = 0.44`, so a gap needs roughly **0.9 points** before it means anything, and | |
| **1.3** before it is comfortable. | |
| Re-reading the whole round against that floor: | |
| | contrast | delta MEAN4 | vs SE 0.44 | verdict | | |
| |---|---|---|---| | |
| | **data scaling** 101/109/110 vs 102/108 | **+0.83** | **5.2 SE** (arms of 6, SE 0.16) | **real** | | |
| | LoRA rank 16 -> 32 (103 vs 102, ep4) | +0.54 | 1.2 SE | not resolved | | |
| | LoRA scale 1.0 -> 2.0 (104 vs 102, ep4) | +0.61 | 1.4 SE | not resolved | | |
| | objective balance `w_caption` 0.4 (105 vs 102, ep4) | -0.02 | 0.0 SE | null | | |
| | cosine + lr 1e-4 (107 vs 102, ep3) | -0.61 | 1.4 SE | not resolved, negative | | |
| | LoRA soup vs mean of its parents (106) | +0.32 | 0.7 SE | not resolved | | |
| | LoRA soup vs mean of its parents (106b) | -0.42 | 1.0 SE | not resolved | | |
| **Only one row of that table survives.** Everything else the round tested is | |
| noise at the resolution the budget bought - which is the honest reading, not a | |
| disappointing one: five levers were eliminated for 25 GPU-hours. | |
| The two soup rows are worth reading together. 106 beat its parents' average | |
| and 106b lost to it, by similar amounts in opposite directions. The defensible | |
| claim is therefore that **souping returns roughly the average of its parents | |
| for zero GPU time**, not that it adds anything. | |
| ### What the round actually bought | |
| **Data scaling is the only lever that survived replication.** It is also the | |
| only one that was replicated - and it was replicated specifically because it | |
| was the only candidate worth the GPU time. Three independent seeds of the | |
| large-corpus recipe produced six checkpoints spanning 76.09-76.72; six | |
| small-corpus checkpoints span 75.22-76.04. **The two distributions do not | |
| overlap.** +0.83 at SE 0.16 is 5.2 SE. | |
| Note what this costs in humility about the original claim: 101's headline was | |
| **+1.07**, measured from one seed against one seed. The replicated estimate is | |
| **+0.83**. The effect is real and the first number was 29% too generous - a | |
| clean illustration of why single-seed deltas are quoted here only with a sigma | |
| attached. | |
| **The hyperparameter contrasts are all unresolved or null.** Rank, LoRA scale, | |
| objective balance and schedule shape each moved MEAN4 by less than 1.4 SE. | |
| None of them is worth a further run at this resolution; resolving a 0.5-point | |
| effect would need roughly 6 seeds per arm, or 60 GPU-hours per lever. | |
| **Aspaklarya-020, a full round older, still wins three slices of four.** It | |
| takes VQA RSICD, CAP DM and CAP RSICD, and loses only DM VQA, by 8.0 points. | |
| Round 1XX bought **specialisation onto the rewritten DM question set**, not a | |
| better model overall. Part of that 8.0 is real; part is the 48-token legacy | |
| harness clipping verbose answers. | |
| **The epoch-3 vs epoch-4 question is closed, and the answer is "neither".** | |
| Across the six replicated large-corpus checkpoints the epoch-3/epoch-4 gap is | |
| +0.49, -0.18 and -0.16 depending on seed - it changes sign. Both earlier | |
| claims about a best epoch were reading noise. | |
| **The corpus is still only 27% spent.** `ImageTable.csv` holds 89,119 QA pairs | |
| over 11,994 train+other images, a mean of 7.43 per image, and every run above | |
| used 2. Aspaklarya-111 is the first run to spend more of it, raising QA depth | |
| to 4 while doubling captions to 2 so the 2:1 VQA:caption ratio - and therefore | |
| the objective balance - is held exactly where every previous run had it. | |
| ### Aspaklarya-106: an exact LoRA soup | |
| Averaging LoRA adapters naively is wrong, because `mean(B) @ mean(A)` is not | |
| `mean(B @ A)`. The parents are instead **stacked**, so the product reproduces | |
| the mean of the weight deltas exactly: | |
| ``` | |
| A_cat = [A_1; ...; A_N] (N*r x k) | |
| B_cat = [c_1*B_1, ..., c_N*B_N] (d x N*r) | |
| c_i = (1/N) * (alpha_i/r_i) / (alpha_new/r_new) | |
| ``` | |
| With `r_new = alpha_new = 64` the reconstruction is exact - verified against | |
| the explicit mean of the parent deltas to **1.9e-9** max absolute error. The | |
| trap is Aspaklarya-104, which carries `alpha/r = 2.0` while its three siblings | |
| carry 1.0: its coefficient must be doubled, and a soup that ignored this would | |
| have silently halved that parent's contribution. | |
| It cost **zero GPU-hours** and lands 2nd of 24. | |
| ### What is published here, and what is not | |
| This org holds **22 adapter repos**: Aspaklarya-101 through 105 at four epoch | |
| checkpoints each, Aspaklarya-106-soup, and Aspaklarya-020-step4500 from the | |
| previous round. Five of the runs described above have **no repo at all** - | |
| 106b, 107, 108, 109 and 110. That is deliberate, and the reasons differ. | |
| A checkpoint is published when it is **a distinct recipe** and **not strictly | |
| dominated** by something already here. Both tests have to pass. | |
| | run | published | why not | | |
| |---|---|---| | |
| | **106b** (epoch-4 soup) | no | **Strictly dominated.** Same construction as 106-soup, 0.61 lower (75.75 vs 76.36). Shipping a worse version of a model already here is a trap for whoever downloads it, not an extra option. | | |
| | **107** (cosine, lr 1e-4) | no | **Refuted configuration.** 75.03 is the lowest MEAN4 in round 1XX. The run did its job - it ruled the gentler schedule out - and that result is a sentence on this page, not 821 MB of weights. | | |
| | **108** (102 at seeds 1234, 2024) | no | **Calibration, not a deliverable.** | | |
| | **109** (101 at seed 1234) | no | **Calibration, not a deliverable.** | | |
| | **110** (101 at seed 2024) | no | **Calibration, not a deliverable.** | | |
| The last three deserve a fuller answer, because they are the ones that | |
| produced the most useful result on this page. | |
| 108, 109 and 110 are not new recipes. **They are the recipes of 102 and 101 | |
| re-run with a different random seed and nothing else changed.** Their | |
| adapters are therefore near-duplicates of adapters already published: same | |
| corpus, same rank, same schedule, same everything the config file records. | |
| What they produced is not a model, it is **a number** - sigma 0.31 - and a | |
| number belongs in the text, where it now is. | |
| Publishing them would also do active harm. Six more checkpoints spanning | |
| 75.22-76.56 would sit in the listing looking like six more options to choose | |
| between, and the obvious way to choose would be to take the highest one. | |
| That is exactly the mistake the sigma measurement exists to prevent: within | |
| an arm, the spread between seeds **is** the noise, so picking the best seed is | |
| selecting on noise and will not reproduce. The honest artifact is the | |
| published 101 and 102 adapters plus a stated uncertainty on what they score, | |
| not a shelf of lottery tickets. | |
| The cost argument is secondary but real: each adapter is ~821 MB, so the five | |
| unpublished runs would add roughly 5 GB of weights that no one should load in | |
| preference to what is already here. | |
| **Aspaklarya-111 is still training.** It will be published if it clears the | |
| `train,other` arm it is built on - meaning a MEAN4 above roughly 76.8, which | |
| is the arm mean plus two sigma. If it lands inside the arm it will be | |
| reported here as another null and left unpublished, on the same reasoning as | |
| 108-110. | |
| ### Caveat | |
| The judge is **text-only**. It compares an answer against a reference and | |
| never sees the image, so it measures agreement with the reference, not | |
| verified truth. For a round whose whole premise is longer answers this matters | |
| more than it did in round 0XX: the judge can confirm that a model supplies | |
| reference-matching content, but not that the content is true of the pixels. | |
| Human spot-checks are the only way to close that gap. | |
| The legacy slices also reward a terse style the product no longer wants - the | |
| legacy VQA rubric caps a padded-but-correct answer at 89/100 by construction, | |
| so a 290-character model cannot reach the top band however accurate it is. | |
| That is why the legacy numbers are reported but never used to gate a decision. | |
| ### Status | |
| - **Aspaklarya-101-step4500 is the round reference** at MEAN4 76.72, but the | |
| fair description of its recipe is **76.37 +/- 0.23** across three seeds. Any | |
| of the six large-corpus checkpoints is a defensible choice; picking the | |
| 76.72 one is selecting on noise. | |
| - **Aspaklarya-106-soup is 2nd at 76.36 for no training cost**, and remains | |
| the better default when GPU time is the constraint - with the caveat above | |
| that souping buys the parents' average, not a gain over it. | |
| - **Aspaklarya-020-step4500 remains the better general model** (3 slices of 4). | |
| Choose a 1XX model only when verbose in-domain VQA is what matters. | |
| - **Aspaklarya-106b, 107, 108, 109 and 110 have no adapter repos.** See | |
| *What is published here, and what is not* above - they are either strictly | |
| dominated, refuted, or seed replications whose output is a sigma rather | |
| than a model. | |
| - Open: whether QA depth (111, running) extends the one confirmed lever, and | |
| whether the `other` split helped by adding **images** or merely **items** - | |
| 111 isolates items, since it adds no new images. | |