Upload geomatch_v2_repro/EVAL_INFO.md with huggingface_hub

geomatch_v2_repro/EVAL_INFO.md

@@ -1,35 +1,58 @@
-# GeoMatch v2 — Reproduction Runs (epoch 50 checkpoints)
+# GeoMatch v2 — Reproduction Runs (epoch 50 & 100 checkpoints)
 
-**Status: intermediate checkpoints from ongoing training.** These are epoch-50 snapshots from three
-parallel training runs launched to reproduce the GeoMatch paper's (arXiv:2312.03864) reported Isaac
-Gym success rates (EZGripper 75.0%, Barrett 90.0%, ShadowHand 72.5%). All three runs use the
-**fixed keypoints** dataset (`grasp_gnn_v2`) — see "Bug History" below. Training continues to
-epoch 200; these checkpoints are provided for early evaluation only, not as final models.
+**Status: intermediate checkpoints from ongoing training.** Three parallel training runs launched
+to reproduce the GeoMatch paper's (arXiv:2312.03864) reported Isaac Gym success rates
+(EZGripper 75.0%, Barrett 90.0%, ShadowHand 72.5%). All three runs use the **fixed keypoints**
+dataset (`grasp_gnn_v2`) — see "Bug History" below. Training continues to epoch 200; these
+checkpoints are provided for early/intermediate evaluation, not as final models. Epoch 50
+checkpoints were evaluated at 5-10% Isaac Gym success — see "Eval feedback so far" below for what
+that does and doesn't tell us. Epoch 100 (halfway) checkpoints are now also available for a second
+data point on the success-rate-vs-training-maturity trend.
 
 ## Which file is which
 
 | File | Seed | LR | Batch size | Rationale |
 |---|---|---|---|---|
-| `runA_paper_exact_epoch50.pth` | 42 | 1e-4 | 256 | Exact paper-matching recipe (control) |
-| `runB_seed123_epoch50.pth` | 123 | 1e-4 | 256 | Same recipe, different seed (init/data-order sensitivity check) |
-| `runC_lr5e-5_bs128_epoch50.pth` | 7 | 5e-5 | 128 | Half LR, half batch — hedge against batch-size/LR sensitivity |
+| `runA_paper_exact_epoch{50,100}.pth` | 42 | 1e-4 | 256 | Exact paper-matching recipe (control) |
+| `runB_seed123_epoch{50,100}.pth` | 123 | 1e-4 | 256 | Same recipe, different seed (init/data-order sensitivity check) |
+| `runC_lr5e-5_bs128_epoch{50,100}.pth` | 7 | 5e-5 | 128 | Half LR, half batch — hedge against batch-size/LR sensitivity |
 
 All three: Adam optimizer (β=0.9, 0.99), weight_decay=0, no AMP (FP32), 200 total epochs planned,
 single-GPU (no DDP), `num_workers=0`.
 
-## Loss/accuracy at epoch 50 (as logged during training)
+## Loss/accuracy (as logged during training)
 
-| Run | Train loss | Train acc | Val loss | Val acc |
-|---|---|---|---|---|
-| A | 1.690662 | 0.6269 | 1.699793 | 0.5926 |
-| B | 1.691613 | 0.6280 | 1.694283 | 0.5801 |
-| C | 1.701282 | 0.6126 | 1.713473 | 0.6481 |
+| Run | Epoch | Train loss | Train acc | Val loss | Val acc |
+|---|---|---|---|---|---|
+| A | 50  | 1.690662 | 0.6269 | 1.699793 | 0.5926 |
+| A | 100 | 1.656710 | 0.6666 | 1.670935 | 0.6288 |
+| B | 50  | 1.691613 | 0.6280 | 1.694283 | 0.5801 |
+| B | 100 | 1.659709 | 0.6586 | 1.674850 | 0.6756 |
+| C | 50  | 1.701282 | 0.6126 | 1.713473 | 0.6481 |
+| C | 100 | 1.670215 | 0.6517 | 1.682868 | 0.6043 |
 
 **Caveat on val numbers:** the validation set is only 10 objects, so epoch-to-epoch val accuracy is
 noisy (observed swings of ±0.05 between adjacent epochs in these runs). Don't read too much into
 small val differences between runs at a single epoch — training (contact-map) loss/acc is the more
 stable signal at this stage. Isaac Gym grasp success rate is the metric that actually matters; these
-loss numbers are only a training-health sanity check.
+loss numbers are only a training-health sanity check. Train loss/acc is trending steadily better at
+epoch 100 vs. 50 for all three runs (e.g. Run A train acc 0.627 → 0.667), consistent with continued
+learning, not a plateau.
+
+## Eval feedback so far (epoch 50 → 5-10% Isaac Gym success)
+
+Epoch-50 checkpoints were evaluated at only 5-10% Isaac Gym success, well below paper targets. This
+was investigated jointly and resolved: the eval-side contact-refinement step (`refine_grasps_cmap_adam.py`,
+one Adam step at `learning_rate=0.05` on the full pose via `CMapAdam`, applied before
+`run_grasp_test.py`) is **confirmed correctly implemented and was already applied** to those 5-10%
+results — this matches the paper's Appendix C description exactly ("Isaac Gym scripts used as is,
+aside from the one Adam step of force closure... step size 0.05"). So the low success rate is **not**
+a pipeline bug. The most likely explanation is genuine undertraining: the paper's own Table 2 shows
+even the easier single-embodiment ablation needs the full 200 epochs to reach 40-70% success, and
+epoch 50 is only 25% through this schedule. **Please re-run the same 40-grasps-per-gripper eval on
+these epoch-100 checkpoints and report success rate per gripper** — if it climbs meaningfully from
+the epoch-50 numbers, that confirms training maturity (not pipeline correctness) is the operative
+variable, and epoch 200 should be worth waiting for.
 
 ## Model architecture / loading
 
@@ -74,14 +97,16 @@ GenDexGrasp's split): 48 train objects (YCB + ContactDB); **10 held-out validati
 `ycb+potted_meat_can`, `ycb+tomato_soup_can`. These 10 are the objects to use for Isaac Gym
 evaluation (unseen at training time), matching the paper's protocol.
 
-## What's NOT yet implemented (matters for grasp generation, not for raw contact-map eval)
+## Contact-refinement step — resolved, no action needed
 
-The inference pipeline (`generate_grasps_for_obj.py`) currently stops at the raw SciPy TRF IK
-result. The paper describes an additional **contact-refinement step** (single Adam step, step size
-0.05, pulling fingers to within 5mm of the object surface) that is **not yet added** to this repo's
-grasp-generation script. If you're evaluating via full grasp generation → Isaac Gym (not just raw
-contact-map precision), be aware raw-IK grasps from these checkpoints may under-perform until that
-refinement step is added.
+Earlier concern: `generate_grasps_for_obj.py` in this repo stops at the raw SciPy TRF IK result
+with no further refinement, which looked like a possible gap. This is **not a bug** — per the
+paper's Appendix C, the 5mm figure is a pre-grasp keypoint offset applied *before* IK (already
+correctly implemented here), and the separate one-Adam-step force-closure refinement
+(`learning_rate=0.05`) is applied by the **Isaac Gym evaluation harness itself**, reused as-is from
+GenDexGrasp per the paper's own wording. That refinement already exists and runs correctly on the
+eval side (`refine_grasps_cmap_adam.py` → `bridge_to_gendexgrasp.py` → `run_grasp_test.py`,
+confirmed by direct code read). No changes needed in this repo's `generate_grasps_for_obj.py`.
 
 ## Suggested evaluation protocol (matches paper)