Upload geomatch_v2_repro/EVAL_INFO.md with huggingface_hub

geomatch_v2_repro/EVAL_INFO.md

@@ -0,0 +1,106 @@
+# GeoMatch v2 — Reproduction Runs (epoch 50 checkpoints)
+
+**Status: intermediate checkpoints from ongoing training.** These are epoch-50 snapshots from three
+parallel training runs launched to reproduce the GeoMatch paper's (arXiv:2312.03864) reported Isaac
+Gym success rates (EZGripper 75.0%, Barrett 90.0%, ShadowHand 72.5%). All three runs use the
+**fixed keypoints** dataset (`grasp_gnn_v2`) — see "Bug History" below. Training continues to
+epoch 200; these checkpoints are provided for early evaluation only, not as final models.
+
+## Which file is which
+
+| File | Seed | LR | Batch size | Rationale |
+|---|---|---|---|---|
+| `runA_paper_exact_epoch50.pth` | 42 | 1e-4 | 256 | Exact paper-matching recipe (control) |
+| `runB_seed123_epoch50.pth` | 123 | 1e-4 | 256 | Same recipe, different seed (init/data-order sensitivity check) |
+| `runC_lr5e-5_bs128_epoch50.pth` | 7 | 5e-5 | 128 | Half LR, half batch — hedge against batch-size/LR sensitivity |
+
+All three: Adam optimizer (β=0.9, 0.99), weight_decay=0, no AMP (FP32), 200 total epochs planned,
+single-GPU (no DDP), `num_workers=0`.
+
+## Loss/accuracy at epoch 50 (as logged during training)
+
+| Run | Train loss | Train acc | Val loss | Val acc |
+|---|---|---|---|---|
+| A | 1.690662 | 0.6269 | 1.699793 | 0.5926 |
+| B | 1.691613 | 0.6280 | 1.694283 | 0.5801 |
+| C | 1.701282 | 0.6126 | 1.713473 | 0.6481 |
+
+**Caveat on val numbers:** the validation set is only 10 objects, so epoch-to-epoch val accuracy is
+noisy (observed swings of ±0.05 between adjacent epochs in these runs). Don't read too much into
+small val differences between runs at a single epoch — training (contact-map) loss/acc is the more
+stable signal at this stage. Isaac Gym grasp success rate is the metric that actually matters; these
+loss numbers are only a training-health sanity check.
+
+## Model architecture / loading
+
+Checkpoints are **raw `state_dict`** (`collections.OrderedDict`), not wrapped in a dict with
+metadata — load directly with `model.load_state_dict(...)`.
+
+```python
+import torch, sys
+sys.path.append(".")   # repo root containing config.py and models/
+import config
+from models.geomatch import GeoMatch
+
+model = GeoMatch(config).cuda()
+model.load_state_dict(torch.load("runA_paper_exact_epoch50.pth", map_location="cuda"))
+model.eval()
+```
+
+Use the `config.py` and `models/geomatch.py` already at the repo root (unchanged from GeoMatch v1/v2 —
+architecture is identical across all `geomatch_v2*` checkpoints). Key config values:
+`obj_pc_n=2048`, `robot_pc_n=6`, `keypoint_n=6`, `hidden_n=256`, `obj_out_feats=512`,
+`num_hidden=3`, `robot_weighting=500.0`, `matchnet_weighting=200.0`.
+
+## Dataset / keypoint files used for training (for exact reproduction)
+
+These checkpoints were trained against a **regenerated** `robot_keypoints.json` /
+`robot_centroids.json` that fixed two bugs in the original v1 keypoint generator (2× scale factor
+applied to nothing else in the pipeline, and a ShadowHand axis-swap at the wrong pipeline stage).
+If you need the exact keypoint files to reproduce inference/IK against these checkpoints:
+
+- `robot_keypoints.json` — MD5 `8658b5cd2cb7ff4344398b7e8ae08a6a`
+- `robot_centroids.json` — MD5 `d0a9ae0a74a29f9ce3348307b1b937d3`
+
+Ask if you need these files uploaded too — they aren't yet in this repo.
+
+**Robots covered:** `allegro`, `allegro_right` (alias), `barrett`, `ezgripper`, `robotiq_3finger`,
+`shadowhand`.
+
+**Train/validation object split** (from `CMapDataset-sqrt_align/split_train_validate_objects.json`,
+GenDexGrasp's split): 48 train objects (YCB + ContactDB); **10 held-out validation/test objects**:
+`contactdb+apple`, `contactdb+camera`, `contactdb+cylinder_medium`, `contactdb+rubber_duck`,
+`contactdb+door_knob`, `contactdb+water_bottle`, `ycb+baseball`, `ycb+pear`,
+`ycb+potted_meat_can`, `ycb+tomato_soup_can`. These 10 are the objects to use for Isaac Gym
+evaluation (unseen at training time), matching the paper's protocol.
+
+## What's NOT yet implemented (matters for grasp generation, not for raw contact-map eval)
+
+The inference pipeline (`generate_grasps_for_obj.py`) currently stops at the raw SciPy TRF IK
+result. The paper describes an additional **contact-refinement step** (single Adam step, step size
+0.05, pulling fingers to within 5mm of the object surface) that is **not yet added** to this repo's
+grasp-generation script. If you're evaluating via full grasp generation → Isaac Gym (not just raw
+contact-map precision), be aware raw-IK grasps from these checkpoints may under-perform until that
+refinement step is added.
+
+## Suggested evaluation protocol (matches paper)
+
+- 10 held-out objects × 4 grasps per object-gripper (sampled from top-[0, 20, 50, 100] predicted
+  keypoint configurations) = 40 grasps per gripper.
+- Isaac Gym test: apply 0.5 m/s² acceleration sequentially along ±x, ±y, ±z (1 second / 60 sim
+  steps each); success iff object displacement < 2cm on **all six** directions.
+- Report per-gripper success rate for EZGripper, Barrett, ShadowHand against paper targets
+  75.0% / 90.0% / 72.5%.
+- `evaluate.py`'s contact precision metric is a useful cheap sanity check before running the full
+  Isaac Gym pipeline (should be far above the ~0.02 seen on the old corrupted-keypoint checkpoints).
+
+## Bug history (why this is "v2", not v1)
+
+GeoMatch v1 (still on this repo under `geomatch_v1/`) was trained on keypoints with a bogus 2×
+scale factor and a ShadowHand axis-swap applied at the wrong pipeline stage, producing corrupted
+training labels and IK targets (0% Isaac Gym success across all grippers). The existing
+`geomatch_v2/` checkpoints on this repo are **also invalid** — that run's DataLoader silently lost
+workers (`num_workers=4` + `prefetch_factor=4` on ROCm), so it only saw ~7.3% of the training data
+per epoch despite fixed keypoints. **These three `geomatch_v2_repro/` runs are the first properly
+trained runs on the fixed-keypoint data** (full dataset per epoch, correct LR). Once one of the
+three is confirmed via Isaac Gym eval, it will become the new canonical `geomatch_v2/`.