Spaces:

ExperimentField
/

README

Running

App Files Files Community

etzion commited on 10 days ago

Commit

0448533

verified ·

1 Parent(s): 4e1f29f

org card: per-experiment hyperparameters and VQA/caption results

Browse files

Files changed (2) hide show

README.md +7 -5
index.html +6 -5

README.md CHANGED Viewed

@@ -9,22 +9,22 @@ pinned: false
 # Aspaklarya — Qwen3-VL-8B for remote-sensing VQA + captioning
-LoRA finetunes of **Qwen3-VL-8B-Instruct**, trained jointly on **visual question answering** and **image captioning** over overhead imagery. 17 experiments, 17 evaluated recipes. All model repos are private.
 ## Setup
 | Item | Value |
 |---|---|
 | Base | Qwen3-VL-8B-Instruct |
-| Method | LoRA (Unsloth), 4-bit base, bf16 compute |
 | Targets | all linear layers, vision + language towers |
-| Data | ~6.3k images, 1 caption + 2 QA each (~18.9k samples) |
 | Caption prompts | sampled from a 30-prompt pool, so the model learns the task not one phrasing |
 | Eval | fixed 1000-image draw = 2000 graded items, same seed for every model |
 | Scoring | LLM-as-judge, 0-100 per item |
 | References | stock Qwen3-VL-8B, plus an in-house finetuned model |
-Winners are merged into the bf16 base and re-validated: adapters are evaluated on a 4-bit base, merged weights are not.
 ## Configurations
@@ -84,7 +84,7 @@ Best checkpoint per run on a fixed 1000-image draw (2000 graded items), with boo
 | in-house reference | 71.88 ±1.23 | **83.62 ±1.49** | 77.75 ±1.02 | 65.18 ±0.83 | 73.34 ±1.13 | 69.26 ±0.74 | 58.72 ±1.09 | 63.44 ±1.62 | 61.08 ±1.02 |
 | Qwen3-VL-8B (base) | 68.37 ±0.89 | 58.94 ±1.50 | 63.66 ±0.92 | 65.68 ±0.66 | 56.01 ±1.01 | 60.84 ±0.62 | 63.07 ±0.94 | 53.18 ±1.46 | 58.13 ±0.91 |
-Every finetune beats both references overall: **+17.1** over the base model and **+8.7** over the in-house reference on the full set.
 ## Learning rate at large batch
 Aspaklarya-015 gave the best captioning in the table at effective batch 64, but it was trained at 2e-4 — the rate tuned for batch 32. Two follow-ups tested whether the batch-64 recipe was simply under-stepped. Scores below come from a separate 500-image draw used for live checkpoint selection, so they compare only with each other.
@@ -111,6 +111,8 @@ Bootstrap over images, 3000 resamples. Absolute standard error:
 Paired model-vs-model error is smaller because image difficulty cancels: **0.42** overall on the full set, **0.56** in-domain. Use the paired figure for any "A beats B" claim.
 ## Caveat
 The hyperparameter runs are statistically tied with each other: among runs 001-019, every confidence interval against the leader straddles zero. **VQA error is ~2.3x caption error**, so no finetune-vs-finetune VQA difference in any column is resolvable; only caption gaps, the gaps to the two references, and the data-scaling result of run 020 are real. Doubling the eval set from 500 to 1000 images did not separate the tied field, and it reshuffled its order.

 # Aspaklarya — Qwen3-VL-8B for remote-sensing VQA + captioning
+LoRA finetunes of **Qwen3-VL-8B-Instruct**, trained jointly on **visual question answering** and **image captioning** over overhead imagery. 21 experiments, 20 of them evaluated (016 was aborted before any checkpoint). All model repos are private.
 ## Setup
 | Item | Value |
 |---|---|
 | Base | Qwen3-VL-8B-Instruct |
+| Method | LoRA (Unsloth), bf16 compute; 4-bit base for runs 001-017, **bf16 base from 018 on** |
 | Targets | all linear layers, vision + language towers |
+| Data | 6,288 images / 18,864 samples for runs 001-019; **11,994 / 35,981 for 020-021** (1 caption + 2 QA each) |
 | Caption prompts | sampled from a 30-prompt pool, so the model learns the task not one phrasing |
 | Eval | fixed 1000-image draw = 2000 graded items, same seed for every model |
 | Scoring | LLM-as-judge, 0-100 per item |
 | References | stock Qwen3-VL-8B, plus an in-house finetuned model |
+Winners are merged into the bf16 base and re-validated. For runs 001-017 that merge also removes a train/serve mismatch, since those adapters were fitted on a 4-bit base but are served from bf16 weights; runs 018 onward have no such mismatch.
 ## Configurations
 | in-house reference | 71.88 ±1.23 | **83.62 ±1.49** | 77.75 ±1.02 | 65.18 ±0.83 | 73.34 ±1.13 | 69.26 ±0.74 | 58.72 ±1.09 | 63.44 ±1.62 | 61.08 ±1.02 |
 | Qwen3-VL-8B (base) | 68.37 ±0.89 | 58.94 ±1.50 | 63.66 ±0.92 | 65.68 ±0.66 | 56.01 ±1.01 | 60.84 ±0.62 | 63.07 ±0.94 | 53.18 ±1.46 | 58.13 ±0.91 |
+Every finetune beats both references overall. The selected model, Aspaklarya-020, is **+17.9** over the base model and **+9.4** over the in-house reference on the full set.
 ## Learning rate at large batch
 Aspaklarya-015 gave the best captioning in the table at effective batch 64, but it was trained at 2e-4 — the rate tuned for batch 32. Two follow-ups tested whether the batch-64 recipe was simply under-stepped. Scores below come from a separate 500-image draw used for live checkpoint selection, so they compare only with each other.
 Paired model-vs-model error is smaller because image difficulty cancels: **0.42** overall on the full set, **0.56** in-domain. Use the paired figure for any "A beats B" claim.
+Applied exhaustively to the five leading checkpoints (020, 018, 019, 001, 011), the paired bootstrap separates **0 of 10 pairs overall and 0 of 10 in-domain** at 95% — in-domain every pair sits at p ≥ 0.67. The only significant pairs anywhere are three on the RSICD slice, all of them 020 against another model. **The ordering of the results table below the top line is not a measurement.**
 ## Caveat
 The hyperparameter runs are statistically tied with each other: among runs 001-019, every confidence interval against the leader straddles zero. **VQA error is ~2.3x caption error**, so no finetune-vs-finetune VQA difference in any column is resolvable; only caption gaps, the gaps to the two references, and the data-scaling result of run 020 are real. Doubling the eval set from 500 to 1000 images did not separate the tied field, and it reshuffled its order.

index.html CHANGED Viewed

@@ -38,22 +38,22 @@ a { color: #0969da; }
 <body>
 <main>
 <h1>Aspaklarya — Qwen3-VL-8B for remote-sensing VQA + captioning</h1>
-<p>LoRA finetunes of <strong>Qwen3-VL-8B-Instruct</strong>, trained jointly on <strong>visual question answering</strong> and <strong>image captioning</strong> over overhead imagery. 17 experiments, 17 evaluated recipes. All model repos are private.</p>
 <h2>Setup</h2>
 <table style="border-collapse:collapse;margin:1rem 0;font-size:.85rem;border:1px solid #d0d7de"><thead><tr>
 <th style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;background:#f6f8fa;font-weight:600;white-space:nowrap">Item</th>
 <th style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;background:#f6f8fa;font-weight:600;white-space:nowrap">Value</th>
 </tr></thead><tbody>
 <tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Base</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Qwen3-VL-8B-Instruct</td></tr>
-<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Method</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">LoRA (Unsloth), 4-bit base, bf16 compute</td></tr>
 <tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Targets</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">all linear layers, vision + language towers</td></tr>
-<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Data</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">~6.3k images, 1 caption + 2 QA each (~18.9k samples)</td></tr>
 <tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Caption prompts</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">sampled from a 30-prompt pool, so the model learns the task not one phrasing</td></tr>
 <tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Eval</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">fixed 1000-image draw = 2000 graded items, same seed for every model</td></tr>
 <tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Scoring</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">LLM-as-judge, 0-100 per item</td></tr>
 <tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">References</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">stock Qwen3-VL-8B, plus an in-house finetuned model</td></tr>
 </tbody></table>
-<p>Winners are merged into the bf16 base and re-validated: adapters are evaluated on a 4-bit base, merged weights are not.</p>
 <h2>Configurations</h2>
 <table style="border-collapse:collapse;margin:1rem 0;font-size:.85rem;border:1px solid #d0d7de"><thead><tr>
 <th style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;background:#f6f8fa;font-weight:600;white-space:nowrap">Model</th>
@@ -127,7 +127,7 @@ a { color: #0969da; }
 <tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">in-house reference</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">71.88 ±1.23</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap"><strong>83.62 ±1.49</strong></td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">77.75 ±1.02</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">65.18 ±0.83</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">73.34 ±1.13</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">69.26 ±0.74</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">58.72 ±1.09</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">63.44 ±1.62</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">61.08 ±1.02</td></tr>
 <tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Qwen3-VL-8B (base)</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">68.37 ±0.89</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">58.94 ±1.50</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">63.66 ±0.92</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">65.68 ±0.66</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">56.01 ±1.01</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">60.84 ±0.62</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">63.07 ±0.94</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">53.18 ±1.46</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">58.13 ±0.91</td></tr>
 </tbody></table>
-<p>Every finetune beats both references overall: <strong>+17.1</strong> over the base model and <strong>+8.7</strong> over the in-house reference on the full set.</p>
 <h2>Learning rate at large batch</h2>
 <p>Aspaklarya-015 gave the best captioning in the table at effective batch 64, but it was trained at 2e-4 — the rate tuned for batch 32. Two follow-ups tested whether the batch-64 recipe was simply under-stepped. Scores below come from a separate 500-image draw used for live checkpoint selection, so they compare only with each other.</p>
 <table style="border-collapse:collapse;margin:1rem 0;font-size:.85rem;border:1px solid #d0d7de"><thead><tr>
@@ -158,6 +158,7 @@ a { color: #0969da; }
 <tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">RSICD (509)</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">0.69</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">1.64</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">0.91</td></tr>
 </tbody></table>
 <p>Paired model-vs-model error is smaller because image difficulty cancels: <strong>0.42</strong> overall on the full set, <strong>0.56</strong> in-domain. Use the paired figure for any &quot;A beats B&quot; claim.</p>
 <h2>Caveat</h2>
 <p>The hyperparameter runs are statistically tied with each other: among runs 001-019, every confidence interval against the leader straddles zero. <strong>VQA error is ~2.3x caption error</strong>, so no finetune-vs-finetune VQA difference in any column is resolvable; only caption gaps, the gaps to the two references, and the data-scaling result of run 020 are real. Doubling the eval set from 500 to 1000 images did not separate the tied field, and it reshuffled its order.</p>
 <h2>Findings</h2>

 <body>
 <main>
 <h1>Aspaklarya — Qwen3-VL-8B for remote-sensing VQA + captioning</h1>
+<p>LoRA finetunes of <strong>Qwen3-VL-8B-Instruct</strong>, trained jointly on <strong>visual question answering</strong> and <strong>image captioning</strong> over overhead imagery. 21 experiments, 20 of them evaluated (016 was aborted before any checkpoint). All model repos are private.</p>
 <h2>Setup</h2>
 <table style="border-collapse:collapse;margin:1rem 0;font-size:.85rem;border:1px solid #d0d7de"><thead><tr>
 <th style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;background:#f6f8fa;font-weight:600;white-space:nowrap">Item</th>
 <th style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;background:#f6f8fa;font-weight:600;white-space:nowrap">Value</th>
 </tr></thead><tbody>
 <tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Base</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Qwen3-VL-8B-Instruct</td></tr>
+<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Method</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">LoRA (Unsloth), bf16 compute; 4-bit base for runs 001-017, <strong>bf16 base from 018 on</strong></td></tr>
 <tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Targets</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">all linear layers, vision + language towers</td></tr>
+<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Data</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">6,288 images / 18,864 samples for runs 001-019; <strong>11,994 / 35,981 for 020-021</strong> (1 caption + 2 QA each)</td></tr>
 <tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Caption prompts</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">sampled from a 30-prompt pool, so the model learns the task not one phrasing</td></tr>
 <tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Eval</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">fixed 1000-image draw = 2000 graded items, same seed for every model</td></tr>
 <tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Scoring</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">LLM-as-judge, 0-100 per item</td></tr>
 <tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">References</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">stock Qwen3-VL-8B, plus an in-house finetuned model</td></tr>
 </tbody></table>
+<p>Winners are merged into the bf16 base and re-validated. For runs 001-017 that merge also removes a train/serve mismatch, since those adapters were fitted on a 4-bit base but are served from bf16 weights; runs 018 onward have no such mismatch.</p>
 <h2>Configurations</h2>
 <table style="border-collapse:collapse;margin:1rem 0;font-size:.85rem;border:1px solid #d0d7de"><thead><tr>
 <th style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;background:#f6f8fa;font-weight:600;white-space:nowrap">Model</th>
 <tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">in-house reference</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">71.88 ±1.23</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap"><strong>83.62 ±1.49</strong></td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">77.75 ±1.02</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">65.18 ±0.83</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">73.34 ±1.13</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">69.26 ±0.74</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">58.72 ±1.09</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">63.44 ±1.62</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">61.08 ±1.02</td></tr>
 <tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Qwen3-VL-8B (base)</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">68.37 ±0.89</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">58.94 ±1.50</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">63.66 ±0.92</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">65.68 ±0.66</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">56.01 ±1.01</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">60.84 ±0.62</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">63.07 ±0.94</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">53.18 ±1.46</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">58.13 ±0.91</td></tr>
 </tbody></table>
+<p>Every finetune beats both references overall. The selected model, Aspaklarya-020, is <strong>+17.9</strong> over the base model and <strong>+9.4</strong> over the in-house reference on the full set.</p>
 <h2>Learning rate at large batch</h2>
 <p>Aspaklarya-015 gave the best captioning in the table at effective batch 64, but it was trained at 2e-4 — the rate tuned for batch 32. Two follow-ups tested whether the batch-64 recipe was simply under-stepped. Scores below come from a separate 500-image draw used for live checkpoint selection, so they compare only with each other.</p>
 <table style="border-collapse:collapse;margin:1rem 0;font-size:.85rem;border:1px solid #d0d7de"><thead><tr>
 <tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">RSICD (509)</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">0.69</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">1.64</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">0.91</td></tr>
 </tbody></table>
 <p>Paired model-vs-model error is smaller because image difficulty cancels: <strong>0.42</strong> overall on the full set, <strong>0.56</strong> in-domain. Use the paired figure for any &quot;A beats B&quot; claim.</p>
+<p>Applied exhaustively to the five leading checkpoints (020, 018, 019, 001, 011), the paired bootstrap separates <strong>0 of 10 pairs overall and 0 of 10 in-domain</strong> at 95% — in-domain every pair sits at p ≥ 0.67. The only significant pairs anywhere are three on the RSICD slice, all of them 020 against another model. <strong>The ordering of the results table below the top line is not a measurement.</strong></p>
 <h2>Caveat</h2>
 <p>The hyperparameter runs are statistically tied with each other: among runs 001-019, every confidence interval against the leader straddles zero. <strong>VQA error is ~2.3x caption error</strong>, so no finetune-vs-finetune VQA difference in any column is resolvable; only caption gaps, the gaps to the two references, and the data-scaling result of run 020 are real. Doubling the eval set from 500 to 1000 images did not separate the tied field, and it reshuffled its order.</p>
 <h2>Findings</h2>