graspformer-ckpt — Pro 6000 fleet checkpoint archive

Private archive: the top-2 checkpoints by val/acc_graspnet_all for every experiment lane trained on the Pro 6000 fleet: sruk-wsc-2513 (2x RTX PRO 6000) and sruk-wsc-2516 (2x RTX PRO 6000). Selection = the run's Lightning top-5-on-monitor set on disk, re-ranked with the wandb history; the metric row for file step=N is the validation at global_step N-1. All runs: entity peize-the-university-of-edinburgh, project graspformer.

manifest.json is the byte/sha256 authority. Verify after download.

exp	step	val/acc_all	wandb run	state at export
exp109_occ	5700	65.97	dzsh777e	crashed
exp109_occ	4500	65.26	dzsh777e	crashed
exp110_vox3d	4000	62.57	s7ig2mtd	crashed
exp110_vox3d	4500	61.60	s7ig2mtd	crashed
exp117_occ_joint	4700	64.63	wrrf7gy9	crashed
exp117_occ_joint	700	64.47	wrrf7gy9	crashed
exp119_dinov3_joint	9500	81.11	23tjjitp	crashed
exp119_dinov3_joint	10200	81.10	23tjjitp	crashed
exp120_verified_joint	5700	76.03	spfsaktw	crashed
exp120_verified_joint	9700	75.43	spfsaktw	crashed
exp123_dinov3_verified_warm	8500	85.83	uo21jms0	running
exp123_dinov3_verified_warm	5900	85.06	uo21jms0	running
exp124_dinov3_verified_rank_warm	2800	82.78	zc0jww33	running
exp124_dinov3_verified_rank_warm	2700	82.63	zc0jww33	running

Notes

• exp102 lanes: 2026-08-14 pipeline (depth left-fill, bnfreeze off). warm lanes = exp004-v14 warm start; scratch lanes = HF SAM only. wmid lanes validate with PREDICTED widths (not comparable to gtw/GT-stand-in lanes).
• exp109/110/117/120: collision family, warm from exp004-v14 (940/940 trunk keys).
• exp119: DINOv3 tower FROM SCRATCH (fingerprint image_encoder=DINOv3ImageEncoder, 303,392,000). Predicted-width alpha-dial command.
• exp123: exp121 + init_from_exp119 warm from exp119 step-9500 -> trunk age = 9500 + step. exp124: exp121+rank, warm_from exp119 step-9500, trunk @1e-4.
• exp123/exp124 were STILL TRAINING at export; their entries are a snapshot, not final bests.
• Cross-exp caveat: val/acc is comparable only within the same width-command policy (GT stand-in vs predicted vs alpha-dial); see each branch's config header.
• Two lanes (first exp102 warm pair, 2026-08-14 morning) crashed at step 99 before any save (dex_models cache incompatibility, since fixed) — no checkpoints existed; excluded.

sruk-wsc-2516 lanes (added 2026-08-20)

Selection: each run's ModelCheckpoint best_k_models state (exact score bound to each file), cross-checked against the wandb maxima — all matched exactly. exp121/exp125 were still training at export; their rows are best-so-far.

exp	step	val/acc_all	wandb run	state at export
exp111_gripq	6200	64.17	w8r8i6gi	crashed
exp111_gripq	6300	64.05	w8r8i6gi	crashed
exp112_honest	200	59.58	dwlvwgf2	failed
exp112_honest	100	45.62	dwlvwgf2	failed
exp116_joint	9000	71.64	lq8tgjqn	crashed
exp116_joint	10700	70.87	lq8tgjqn	crashed
exp118_rank_joint	4600	65.85	ewaet67y	crashed
exp118_rank_joint	6900	63.98	ewaet67y	crashed
exp121_dinov3_verified	7900	80.50	mnnmyje4	running
exp121_dinov3_verified	10800	80.28	mnnmyje4	running
exp122_dinov3_rank	3800	68.43	1eqb68b5	crashed
exp122_dinov3_rank	3100	67.78	1eqb68b5	crashed
exp125_dinov3_unfreeze_warm	4300	84.84	xjxrztvk	running
exp125_dinov3_unfreeze_warm	5700	82.57	xjxrztvk	running
exp126_dinov3_verified_unfreeze_warm	2500	86.71	u0bg747m	running
exp126_dinov3_verified_unfreeze_warm	3800	86.41	u0bg747m	running
exp130_dinov3_verified_unfreeze_warm121	3000	79.84	nwhyhyd4	running
exp130_dinov3_verified_unfreeze_warm121	3500	79.74	nwhyhyd4	running
exp128_dinov3_geostab_warm	700	63.55 (val/acc_graspnet_all)	6dlup0iy	crashed
exp128_dinov3_geostab_warm	1300	62.49 (val/acc_graspnet_all)	6dlup0iy	crashed
exp132_recon_aux_scratch	22700	81.88 (val3/acc_graspnet_all)	nyvuh9ee	running
exp132_recon_aux_scratch	22900	81.86 (val3/acc_graspnet_all)	nyvuh9ee	running
exp133_anchor_offset_scratch	3900	77.82 (val3/acc_graspnet_all)	4c531ysf	crashed
exp133_anchor_offset_scratch	4300	76.98 (val3/acc_graspnet_all)	4c531ysf	crashed
exp136_geo_recon_clos_scratch	5100	83.25 (val3/acc_graspnet_all)	6fs7xb29	running
exp136_geo_recon_clos_scratch	5300	82.93 (val3/acc_graspnet_all)	6fs7xb29	running

Notes: exp112_honest has only two scored checkpoints (run died at step 210 on a non-reproducible NaN; both files pre-date it). exp125 files were trained with the hf_backbone.py DINOv3 block-list fix (transformers>=5) applied in-tree.

• V3-era lanes (exp132+, marked val3/acc_graspnet_all) monitor the 90-image V3 depth-anchor protocol — NOT comparable to V2-era val/acc_graspnet_all rows.

sruk-wsc-2516 lanes, batch 2 (added 2026-08-24)

Same selection method (ModelCheckpoint best_k_models, wandb cross-checked exact). exp134/exp135 monitor val3/acc_graspnet_all (the val3 protocol), noted per-entry in the manifest.

exp	step	best acc	wandb run	monitor
exp127_dinov3_boundary_rank_warm	7100	82.00	xoxlvbro	val/acc_graspnet_all
exp127_dinov3_boundary_rank_warm	8100	81.34	xoxlvbro	val/acc_graspnet_all
exp129_dinov3h_joint	8600	78.88	mp0jcl8b	val/acc_graspnet_all
exp129_dinov3h_joint	4200	78.37	mp0jcl8b	val/acc_graspnet_all
exp134_geo_stability	18500	83.40	l1niy0nn	val3/acc_graspnet_all
exp134_geo_stability	11800	83.09	l1niy0nn	val3/acc_graspnet_all
exp135_verifier_closability	25700	81.25	l7y2i0ga	val3/acc_graspnet_all
exp135_verifier_closability	12100	81.11	l7y2i0ga	val3/acc_graspnet_all
exp138_fullres_refiner	8800	5.349 mm (val/anchor_ref_mm, min)	oecvfnuu	finished
exp138_fullres_refiner	7200	5.355 mm (val/anchor_ref_mm, min)	oecvfnuu	finished
exp139_monoprior	7200	5.222 mm (val/anchor_ref_mm, min)	1v55359k	finished
exp139_monoprior	8000	5.228 mm (val/anchor_ref_mm, min)	1v55359k	finished
exp150_multitap (depth)	5100	depth 84.19 / gt 91.37 (val3)	nvcv4h6m	running
exp150_multitap (depth)	5500	depth 84.97 / gt 92.05 (val3)	nvcv4h6m	running
exp150_multitap (gt)	5300	depth 84.18 / gt 92.71 (val3)	nvcv4h6m	running
exp150_multitap (gt)	5500	depth 84.97 / gt 92.05 (val3)	nvcv4h6m	running
exp163_widectx (gt)	1500	gt 92.48 / depth 86.82 (val3)	mk30924o	stopped @2006
exp168_fullres_antioverfit (depth)	5200	gt 91.55 / depth 85.46 (val3, EMA-shadow)	p3z16rtc	running
exp168_fullres_antioverfit (depth)	5300	gt 91.48 / depth 86.65 (val3, EMA-shadow)	p3z16rtc	running
exp168_fullres_antioverfit (gt)	5400	gt 92.43 / depth 84.40 (val3, EMA-shadow)	p3z16rtc	running
exp168_fullres_antioverfit (gt)	5600	gt 92.88 / depth 83.76 (val3, EMA-shadow)	p3z16rtc	running
exp151_cdmdepth (depth)	6700	gt 85.95 / depth 81.50 (val3)	h2os6ru8	stopped
exp151_cdmdepth (depth)	8200	gt 85.88 / depth 81.66 (val3)	h2os6ru8	stopped
exp151_cdmdepth (gt)	5800	gt 88.51 / depth 78.90 (val3)	h2os6ru8	stopped
exp151_cdmdepth (gt)	8800	gt 88.68 / depth 80.40 (val3)	h2os6ru8	stopped
exp160_fleet (depth)	5300	gt 91.15 / depth 83.94 (val3, EMA-shadow)	kkshegbf	stopped
exp160_fleet (depth)	7400	gt 90.30 / depth 84.09 (val3, EMA-shadow)	kkshegbf	stopped
exp160_fleet (gt)	5300	gt 91.15 / depth 83.94 (val3, EMA-shadow)	kkshegbf	stopped
exp160_fleet (gt)	5600	gt 91.06 / depth 83.01 (val3, EMA-shadow)	kkshegbf	stopped
exp163_widectx (depth)	1500	gt 92.48 / depth 86.82 (val3)	mk30924o	stopped
exp163_widectx (depth)	1900	gt 89.51 / depth 86.35 (val3)	mk30924o	stopped
exp163_widectx (gt)	1300	gt 92.45 / depth 85.33 (val3)	mk30924o	stopped
exp170_peakhold_floor (depth)	200	gt 91.80 / depth 84.30 (val3, EMA-shadow)	o68yx6j2	stopped
exp170_peakhold_floor (depth)	300	gt 90.17 / depth 84.63 (val3, EMA-shadow)	o68yx6j2	stopped
exp170_peakhold_floor (gt)	200	gt 91.80 / depth 84.30 (val3, EMA-shadow)	o68yx6j2	stopped
exp170_peakhold_floor (gt)	500	gt 91.18 / depth 82.34 (val3, EMA-shadow)	o68yx6j2	stopped
exp170_peakhold_armH (depth)	700	gt 89.86 / depth 85.02 (val3, EMA-shadow)	busto8by	stopped
exp170_peakhold_armH (depth)	2400	gt 89.72 / depth 84.34 (val3, EMA-shadow)	busto8by	stopped
exp170_peakhold_armH (gt)	1500	gt 92.78 / depth 81.69 (val3, EMA-shadow)	busto8by	stopped
exp170_peakhold_armH (gt)	400	gt 92.33 / depth 81.72 (val3, EMA-shadow)	busto8by	stopped

exp102 checkpoints removed (2026-08-25)

All 12 exp102_* checkpoints (both hosts) were deleted from this archive on founder instruction to free storage quota. Sources remain on the training machines; the 2516 copies are also in its local staging mirror.

exp134_geo_stability: extended to top-5 (2026-08-25)

Founder request: the full top-5 set for exp134 (monitor val3/acc_graspnet_all). Adds steps 7600 (83.08), 13700 (83.04), 18700 (83.03) to the existing 18500 (83.40) and 11800 (83.09).

• exp138/exp139 are standalone DepthRefiner U-Nets (17.7M params, in_channels 5 and 6), not GraspFormer checkpoints; their metric is anchor error in mm (LOWER is better) and is a development proxy — the criterion is AP recovery via anchor_policy_ap_bench.py.

sruk-wsc-2516 week batch (added 2026-09-01)

Top-2 per ModelCheckpoint policy for exp137/141/142 (single monitor) and exp148/149/158/167/171 (dual gt/depth). Selection from each run's own best_k_models state. exp171 was RUNNING at export (best-so-far, healthy attempt 3 on GPU 0; attempts 1-2 were destroyed by the GPU-1 VRAM defect and contributed no files).

exp	policy	steps (scores)
exp171_widectx	depth	300 (88.19), 1000 (87.97)
exp171_widectx	gt	1100 (93.35), 700 (93.15)
exp137_zresid_frozen132	single	200 (84.29), 700 (84.25)
exp141_qres_warm132	single	4100 (83.62), 2800 (83.26)
exp142_depthctx	single	4700 (77.95), 4500 (77.72)
exp148_antioverfit	depth	6700 (83.94), 4100 (83.84)
exp148_antioverfit	gt	7000 (90.26), 6800 (90.16)
exp149_fullresgather	depth	5700 (83.05), 6200 (82.11)
exp149_fullresgather	gt	8400 (88.88), 7600 (88.35)
exp158_fullres_ao	depth	9400 (84.93), 9200 (84.07)
exp158_fullres_ao	gt	8300 (91.40), 6500 (90.70)
exp167_fullres_basin	depth	6600 (86.87), 6200 (85.68)
exp167_fullres_basin	gt	4100 (91.36), 5700 (90.65)
exp174_run3 (gt)	5300	gt 93.03 / depth 87.80 (val3, EMA-shadow)
exp174_run3 (gt)	5400	gt 93.33 / depth 88.02 (val3, EMA-shadow)
exp174_run3 (gt)	5900	gt 93.24 / depth 85.82 (val3, EMA-shadow)
exp174_run3 (gt)	6900	gt 93.25 / depth 86.82 (val3, EMA-shadow)
exp174_run3 (gt)	7600	gt 93.20 / depth 87.74 (val3, EMA-shadow)
exp175_offobj (depth)	6900	gt 91.22 / depth 88.51 (val3, EMA-shadow)
exp175_offobj (depth)	8300	gt 92.63 / depth 88.54 (val3, EMA-shadow)
exp175_offobj (gt)	6500	gt 94.02 / depth 86.77 (val3, EMA-shadow)
exp175_offobj (gt)	6700	gt 93.39 / depth 87.39 (val3, EMA-shadow)
exp175_offobj (gt)	9100	gt 93.60 / depth 87.00 (val3, EMA-shadow)

• Upload rule (2026-09-02): no checkpoints from a from-scratch run cut short before 3000 steps (warm-continue runs are exempt). exp145_contact (scratch, stopped @2899) was removed under this rule.

sruk-wsc-2516 batch 5 (added 2026-09-03; section restored 09-06 after a multi-box manifest race)

exp176 (exp174 recipe + knn_impl=topk), GT-only top-5 policy, top-2 per run from each run's own best_k_models. Eligibility rule (founder 09-03): nothing from corrupted runs or runs cut short before 3k steps — exp174 run1 (reboot-killed at 2468) and all exp174 run2 attempts (<900, GPU-1 defect) are therefore NOT archived; exp171 attempt 3 reached 5000 steps but its top-2 never moved past the files archived 09-01.

exp	state at export	steps (scores)
exp176_topk_run1	killed (screen quit) at step 5666 — founder order 09-03	5100 (89.62), 4700 (89.33)
exp176_topk_run2	RUNNING at export (best-so-far, ~step 5800, GPU 1)	4400 (93.00), 4200 (92.80)

sruk-wsc-2516 batch 6 (added 2026-09-06)

exp176 run2 FINAL top-2 (GT policy) — the run ended at step 9199 (founder order 09-04, no corruption signature). Supersedes the 09-03 best-so-far pair (steps 4200/4400), which stays in the repo but is marked superseded_by in the manifest. Nothing else on 2516 was eligible (rule 09-03: >= 3k steps, not corrupted).

exp	state at export	steps (scores)
exp176_topk_run2	killed (screen quit) at step 9199 — founder order 09-04 (FINAL)	6100 (94.17), 7000 (94.08)
exp177_run4 (novel)	4900	val4/acc_novel 36.54 (EMA-shadow; no V3)
exp177_run4 (novel)	5500	val4/acc_novel 36.06 (EMA-shadow; no V3)
exp177_run4 (novel)	6000	val4/acc_novel 36.02 (EMA-shadow; no V3)
exp177_run4 (last)	13100	val4/acc_novel 34.13 (EMA-shadow; no V3)

sruk-wsc-2516 batch 8 (added 2026-09-12) — fusion ablation

exp180 (fusion_mode: add, 0 fusion params) and exp181 (fusion_mode: xattn, 28 tensors / 1,580,544 params) are the two non-default arms of the three-way fusion comparison against exp177's gated. Both SCRATCH on seed 177006 (exp177 run-6's seed, reused deliberately so the fusion delta is not confounded by a seed delta), V3 dual-anchor validation, top-2 per policy from each run's own best_k_models.

Provenance caveats — read before using these. ECC was DISABLED on both GPUs for the whole of both runs (enabled on this box only at the 09-12 reboot), so silent memory corruption could not be counted. Every file here was explicitly verified at export: all tensors finite, max|w| within the sane band (reported per entry in manifest.json as integrity_at_export). exp181 additionally ran on GPU 1, the card with a long corruption history on this box, and was killed by a CUDA scatter-gather index-OOB assert at step 7655; it is a RESUME of an earlier attempt that went all-NaN at step 2359, and that first attempt is not archived (rule 09-03: nothing from corrupted runs or runs under 3k steps).

exp	fusion	state at export	steps (scores)
exp180_fusionadd	add	stopped at step 9100 by the 09-12 ECC reboot (GPU 0, clean run: no NaN, no assert, ECC was disabled during the run)	depth 4300 (89.07), depth 4400 (88.22), gt 5900 (93.95), gt 6700 (94.69)
exp181_fusionxattn	xattn	RESUMED from step 2300; reached 7600, then killed by a CUDA scatter-gather index-OOB assert at step 7655 on GPU 1 (ECC disabled during the run). Weights verified finite at export. The first attempt (all-NaN at 2359) is NOT archived.	depth 4400 (89.59), depth 6700 (87.80), gt 6500 (92.59), gt 6900 (92.57)
exp178_unfreeze (depth)	3600	gt 88.82 / depth 85.50 (val3, EMA-shadow; warm 90.39/87.31)	fqjb9rwe
exp178_unfreeze (depth)	5200	gt 90.93 / depth 85.90 (val3, EMA-shadow; warm 90.39/87.31)	fqjb9rwe
exp178_unfreeze (gt)	5200	gt 90.93 / depth 85.90 (val3, EMA-shadow; warm 90.39/87.31)	fqjb9rwe
exp178_unfreeze (gt)	5400	gt 89.70 / depth 81.78 (val3, EMA-shadow; warm 90.39/87.31)	fqjb9rwe
exp179_unfreeze4 (depth)	1900	gt 92.57 / depth 88.45 (val3, EMA-shadow; warm 90.39/87.31)	mmwrt2vf
exp179_unfreeze4 (depth)	3000	gt 89.22 / depth 89.56 (val3, EMA-shadow; warm 90.39/87.31)	mmwrt2vf
exp179_unfreeze4 (gt)	1000	gt 94.15 / depth 88.42 (val3, EMA-shadow; warm 90.39/87.31)	mmwrt2vf
exp179_unfreeze4 (gt)	5500	gt 93.71 / depth 86.99 (val3, EMA-shadow; warm 90.39/87.31)	mmwrt2vf
exp187_roitoken (depth)	4400	gt 92.47 / depth 88.62 (val3, EMA-shadow)	2fbar5vo
exp187_roitoken (depth)	5000	gt 91.95 / depth 88.83 (val3, EMA-shadow)	2fbar5vo
exp187_roitoken (gt)	4200	gt 93.81 / depth 86.72 (val3, EMA-shadow)	2fbar5vo
exp187_roitoken (gt)	6000	gt 94.33 / depth 86.48 (val3, EMA-shadow)	2fbar5vo
exp189_roitokenplus (depth)	6800	gt 93.40 / depth 87.88 (val3, EMA-shadow)	ylu9g9g8
exp189_roitokenplus (depth)	7100	gt 91.76 / depth 87.78 (val3, EMA-shadow)	ylu9g9g8
exp189_roitokenplus (gt)	5300	gt 94.24 / depth 84.45 (val3, EMA-shadow)	ylu9g9g8
exp189_roitokenplus (gt)	5800	gt 93.96 / depth 85.82 (val3, EMA-shadow)	ylu9g9g8

SRUK-WSC-2421 batch 1 (added 2026-09-14) — modality ablation (exp190 / exp191)

exp190 (input_modality: rgb — PointNet++ DepthEncoder + fusion gate NOT BUILT; 381,066,158 / 77,936,558 params) and exp191 (input_modality: depth — DINOv3 tower + fusion gate NOT BUILT; 77,847,275 / 77,847,275) are the paper's two modality-ablation arms on exp177 run-6's carrier, both SCRATCH on seed 177006, V3 dual-anchor validation. Trained on SRUK-WSC-2421 (2x RTX 6000 Ada, ECC DISABLED on both GPUs for the whole of both runs — same caveat as batch 8: silent memory corruption could not be counted; every archived file verified finite at export), launched via each branch's scripts/run_exp19X.sh after its 16-check smoke; both stopped manually on 2026-09-14. Top-1 gt-policy checkpoint only per lane (founder order); the depth-policy bests (exp190 step 6600, depth 88.42; exp191 step 6300, depth 79.52) are not archived. Weights verified finite at export (integrity_at_export in manifest.json). wandb entity peize-the-university-of-edinburgh.

exp	step	val3 (gt / depth)	wandb run	state at export
exp190_rgbonly (gt)	7300	gt 93.06 / depth 85.71 (val3, EMA-shadow)	16hckecs	killed at step 7521
exp191_depthonly (gt)	5400	gt 82.85 / depth 76.76 (val3, EMA-shadow)	m6m0z3dc	killed at step 9265
exp192_c1_rgbroi (gt)	5800	gt 58.67 / depth 47.37 (val3, raw weights; RGB-only ablation floor)	fi4o69yl	stopped @6899
exp193_d1_rgbdroi (gt)	4700	gt 64.13 / depth 51.04 (val3, raw weights; D1 ablation)	krcv2ggd	stopped @4808
exp197_fixedfusion_v4 (novel)	2600	val4/acc_novel 37.63 (EMA-shadow; exp177_run6 ref 38.31)	6zi2d635	running
exp197_fixedfusion_v4 (novel)	3100	val4/acc_novel 37.35 (EMA-shadow; exp177_run6 ref 38.31)	6zi2d635	running
exp197_fixedfusion_v4 (novel)	4400	val4/acc_novel 37.55 (EMA-shadow; exp177_run6 ref 38.31)	6zi2d635	running
exp189_roitokenplus (gt)	7900	gt 94.12 / depth 85.96 (val3, EMA-shadow)	ylu9g9g8	stopped @7960

sruk-wsc-2516 batch 9 (added 2026-09-14) — stability-combine pair

exp185 and exp186 differ by EXACTLY ONE PARAMETER and nothing else: exp185 s = MLP(q) + lambda_geo * D_stab(C) (lambda learnable, init 1.0) vs exp186 s = MLP(q) + D_stab(C) (no mixing parameter). Both scratch on seed 177006, exp177 carrier, cylinder knn k=256, V3 dual-anchor, top-2 per policy, both RUNNING at export (best-so-far). FIRST BATCH WITH ECC ENABLED on this box (0 errors at export), so silent corruption is positively excluded rather than merely unobserved.

exp	policy	steps (composite)
exp185_roiprimary	depth	7400 (86.24), 5500 (85.78)
exp185_roiprimary	gt	5700 (93.14), 5500 (92.84)
exp186_fixedfusion	depth	3800 (88.68), 4900 (88.32)
exp186_fixedfusion	gt	3800 (95.19), 4100 (94.37)

sruk-wsc-2516 batch 10 (added 2026-09-15) — paper ablations D3 / F1

Founder-specified selection, NOT the usual top-2 per policy:

• exp194 (D3, vanilla structural context) — TOP-1 on the GT anchor policy. Geometry-only context points (in_proj 51 inputs, the fused RGB-D feature gone) and no FiLM anywhere (q_proj absent), so the collision branch is appearance-blind. DINOv3 frozen, cache ON (legal: frozen original tower, no taps, no photo_aug).
• exp195 (F1, real-robot baseline) — the LATEST checkpoint, not a top-k pick: ImageNet ViT-L/16-384 + VanillaDepthEncoder (FPS seeds), cache off (the DINOv3 cache validator refuses a 'vit' tower).

Both RUNNING at export, seed 177006, ECC enabled with 0 errors.

exp	selection	step	score
exp194_d3_vanillactx	gt	5100	83.46
exp195_f1_vitvanillapn	last	4700	n/a (latest)
exp198_d4_roistab (gt)	4300	gt 87.80 / depth 78.24 (val3, raw weights; D4 ablation)	23s9h565
exp200_roitoken_v4 (novel)	4900	val4/acc_novel 39.64 (EMA-shadow; beats exp177_run6 38.31 and exp197 37.63)	9af5v2gn