org card: per-experiment hyperparameters and VQA/caption results

README.md

@@ -50,9 +50,9 @@ Constant everywhere: effective batch 32 (except 014/015), warmup 0.03, grad clip
 
 ## Results
 
-Best checkpoint per run on a fixed 1000-image draw (2000 graded items), with bootstrap standard error. **Overall** = mean of caption and VQA. **In-dom.** = the target overhead-imagery subset (491 images); **General** = a public remote-sensing benchmark subset (509 images). Sorted by in-domain overall; **bold marks the best value in each column**.
+Best checkpoint per run on a fixed 1000-image draw (2000 graded items), with bootstrap standard error. **Overall** = mean of caption and VQA. **In-dom.** = the target overhead-imagery subset (491 images); **RSICD** = the public RSICD remote-sensing benchmark subset (509 images). Sorted by in-domain overall; **bold marks the best value in each column**.
 
-| Model | In-dom. cap | In-dom. vqa | In-dom. overall | All cap | All vqa | All overall | Gen. cap | Gen. vqa | Gen. overall |
+| Model | In-dom. cap | In-dom. vqa | In-dom. overall | All cap | All vqa | All overall | RSICD cap | RSICD vqa | RSICD overall |
 |---|---|---|---|---|---|---|---|---|---|
 | Aspaklarya-011 | 81.85 ±0.66 | 83.51 ±1.48 | **82.68 ±0.84** | 79.92 ±0.49 | 75.78 ±1.13 | 77.85 ±0.64 | 78.06 ±0.67 | 68.33 ±1.64 | 73.20 ±0.92 |
 | Aspaklarya-001 | 81.84 ±0.66 | 83.23 ±1.44 | 82.54 ±0.81 | 79.97 ±0.48 | 75.94 ±1.12 | **77.95 ±0.63** | 78.17 ±0.69 | 68.89 ±1.66 | 73.53 ±0.93 |
@@ -82,7 +82,7 @@ Bootstrap over images, 3000 resamples. Absolute standard error:
 |---|---|---|---|
 | All (1000 img) | 0.48 | 1.12 | 0.63 |
 | In-domain (491) | 0.66 | 1.46 | 0.82 |
-| General (509) | 0.69 | 1.64 | 0.91 |
+| RSICD (509) | 0.69 | 1.64 | 0.91 |
 
 Paired model-vs-model error is smaller because image difficulty cancels: **0.42** overall on the full set, **0.56** in-domain. Use the paired figure for any "A beats B" claim.
 
@@ -93,10 +93,10 @@ The top eight runs are statistically tied: every confidence interval against the
 ## Findings
 
 - **Optimization is saturated.** LR, rank, alpha, dropout, weight decay, gradient clipping, batch geometry, effective batch size (16 / 32 / 64) and schedule were all swept. None beat the noise floor.
-- **The slices disagree.** Ranking by in-domain score almost inverts the general-benchmark ranking: the top three in-domain models place 11th, 2nd and 16th on the general set. The selected model wins by never being bad anywhere, not by leading any single slice.
+- **The slices disagree.** Ranking by in-domain score almost inverts the RSICD ranking: the top three in-domain models place 11th, 2nd and 16th on the general set. The selected model wins by never being bad anywhere, not by leading any single slice.
 - **Captioning is the real gain.** It improves on every slice, by +14 over the base model and +10 to +19 over the in-house reference.
 - **The in-domain VQA win is not established.** The in-house reference posts the single highest in-domain VQA score in the table; it collapses outside that domain, where it falls below the untuned base model.
 - **The vision tower matters.** Freezing it is the one ablation that consistently hurt.
 - **Extra QA per image does not help.** 4-QA and 6-QA mixtures tie with 2-QA.
-- **Larger batch trades generality for domain fit.** Effective batch 64 gives the best in-domain captioning in the table and the worst general-set VQA.
+- **Larger batch trades generality for domain fit.** Effective batch 64 gives the best in-domain captioning in the table and the worst RSICD VQA.
 - **Data is the remaining lever**, not hyperparameters.

index.html

@@ -84,7 +84,7 @@ a { color: #0969da; }
 </tbody></table>
 <p>Constant everywhere: effective batch 32 (except 014/015), warmup 0.03, grad clip 0.3 (1.0 for 011/012), seed 3407.</p>
 <h2>Results</h2>
-<p>Best checkpoint per run on a fixed 1000-image draw (2000 graded items), with bootstrap standard error. <strong>Overall</strong> = mean of caption and VQA. <strong>In-dom.</strong> = the target overhead-imagery subset (491 images); <strong>General</strong> = a public remote-sensing benchmark subset (509 images). Sorted by in-domain overall; <strong>bold marks the best value in each column</strong>.</p>
+<p>Best checkpoint per run on a fixed 1000-image draw (2000 graded items), with bootstrap standard error. <strong>Overall</strong> = mean of caption and VQA. <strong>In-dom.</strong> = the target overhead-imagery subset (491 images); <strong>RSICD</strong> = the public RSICD remote-sensing benchmark subset (509 images). Sorted by in-domain overall; <strong>bold marks the best value in each column</strong>.</p>
 <table style="border-collapse:collapse;margin:1rem 0;font-size:.85rem;border:1px solid #d0d7de"><thead><tr>
 <th style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;background:#f6f8fa;font-weight:600;white-space:nowrap">Model</th>
 <th style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;background:#f6f8fa;font-weight:600;white-space:nowrap">In-dom. cap</th>
@@ -93,9 +93,9 @@ a { color: #0969da; }
 <th style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;background:#f6f8fa;font-weight:600;white-space:nowrap">All cap</th>
 <th style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;background:#f6f8fa;font-weight:600;white-space:nowrap">All vqa</th>
 <th style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;background:#f6f8fa;font-weight:600;white-space:nowrap">All overall</th>
-<th style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;background:#f6f8fa;font-weight:600;white-space:nowrap">Gen. cap</th>
-<th style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;background:#f6f8fa;font-weight:600;white-space:nowrap">Gen. vqa</th>
-<th style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;background:#f6f8fa;font-weight:600;white-space:nowrap">Gen. overall</th>
+<th style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;background:#f6f8fa;font-weight:600;white-space:nowrap">RSICD cap</th>
+<th style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;background:#f6f8fa;font-weight:600;white-space:nowrap">RSICD vqa</th>
+<th style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;background:#f6f8fa;font-weight:600;white-space:nowrap">RSICD overall</th>
 </tr></thead><tbody>
 <tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Aspaklarya-011</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">81.85 ±0.66</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">83.51 ±1.48</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap"><strong>82.68 ±0.84</strong></td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">79.92 ±0.49</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">75.78 ±1.13</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">77.85 ±0.64</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">78.06 ±0.67</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">68.33 ±1.64</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">73.20 ±0.92</td></tr>
 <tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Aspaklarya-001</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">81.84 ±0.66</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">83.23 ±1.44</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">82.54 ±0.81</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">79.97 ±0.48</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">75.94 ±1.12</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap"><strong>77.95 ±0.63</strong></td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">78.17 ±0.69</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">68.89 ±1.66</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">73.53 ±0.93</td></tr>
@@ -127,7 +127,7 @@ a { color: #0969da; }
 </tr></thead><tbody>
 <tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">All (1000 img)</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">0.48</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">1.12</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">0.63</td></tr>
 <tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">In-domain (491)</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">0.66</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">1.46</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">0.82</td></tr>
-<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">General (509)</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">0.69</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">1.64</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">0.91</td></tr>
+<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">RSICD (509)</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">0.69</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">1.64</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">0.91</td></tr>
 </tbody></table>
 <p>Paired model-vs-model error is smaller because image difficulty cancels: <strong>0.42</strong> overall on the full set, <strong>0.56</strong> in-domain. Use the paired figure for any &quot;A beats B&quot; claim.</p>
 <h2>Caveat</h2>
@@ -135,12 +135,12 @@ a { color: #0969da; }
 <h2>Findings</h2>
 <ul>
 <li><strong>Optimization is saturated.</strong> LR, rank, alpha, dropout, weight decay, gradient clipping, batch geometry, effective batch size (16 / 32 / 64) and schedule were all swept. None beat the noise floor.</li>
-<li><strong>The slices disagree.</strong> Ranking by in-domain score almost inverts the general-benchmark ranking: the top three in-domain models place 11th, 2nd and 16th on the general set. The selected model wins by never being bad anywhere, not by leading any single slice.</li>
+<li><strong>The slices disagree.</strong> Ranking by in-domain score almost inverts the RSICD ranking: the top three in-domain models place 11th, 2nd and 16th on the general set. The selected model wins by never being bad anywhere, not by leading any single slice.</li>
 <li><strong>Captioning is the real gain.</strong> It improves on every slice, by +14 over the base model and +10 to +19 over the in-house reference.</li>
 <li><strong>The in-domain VQA win is not established.</strong> The in-house reference posts the single highest in-domain VQA score in the table; it collapses outside that domain, where it falls below the untuned base model.</li>
 <li><strong>The vision tower matters.</strong> Freezing it is the one ablation that consistently hurt.</li>
 <li><strong>Extra QA per image does not help.</strong> 4-QA and 6-QA mixtures tie with 2-QA.</li>
-<li><strong>Larger batch trades generality for domain fit.</strong> Effective batch 64 gives the best in-domain captioning in the table and the worst general-set VQA.</li>
+<li><strong>Larger batch trades generality for domain fit.</strong> Effective batch 64 gives the best in-domain captioning in the table and the worst RSICD VQA.</li>
 <li><strong>Data is the remaining lever</strong>, not hyperparameters.</li>
 </ul>
 </main>