rsm_cooking_v5 -- real-world checkpoints (RSM / RISE study)
Checkpoints from the real-world arm of the RSM-as-value-function study (rsm_rise), trained on the
rsm_cooking_v5 FR3 / Agilex tabletop cooking dataset: 248 teleop episodes,
181530 motion-filtered frames, 2 cameras (global, wrist), success rate
0.202 under the operator's arrival rule
("next.reward > 0 somewhere, i.e. an operator-marked arrival at a reward state [4] (state 4 = pan on grate, tomato in bowl)").
Shared for the robotics collaboration (@jmarangola). This repo is public: no token and no access request are needed to download anything here. It was created private and switched to public on 2026-09-22 so the binaries could be uploaded at all (the account had no private storage headroom left), so treat every file here as world-readable.
Everything here is live and content-verified. The tree is 356 files, 148.65 GiB, and 206
of those 356 objects have been hashed against the cluster file they were uploaded from -- the
Hub's LFS sha256 vs a fresh sha256 of the source -- with zero mismatches. Those 206 are every
LFS object in the repo, i.e. 148.65 GiB of the 148.65 GiB and every byte of every weight; the
other 149 are small JSON/text members the Hub stores inline with no LFS digest to compare, and
each matches its source in size, so 355 of those 356 objects are verified and the one left out
is .gitattributes, which the Hub writes itself and for which there is no source file. The hashed
set includes every binary inside the twelve pi05_ params folders, which earlier revisions of this
card had only size-diffed. Verified at revision 5b409ef, 2026-09-24 08:01 UTC.
The repo has grown past the original six checkpoints. It now also carries 12 pi0.5 params
folders -- 8 of them the gen_q #269 spectral-norm x kNN sweep (7 arms and its warm-start
control), the other 4 the base pi0.5 policies the sweep warm-starts from (see the what it is
column below) -- and 8 reward heads with the 2 base successors they were fine-tuned from
(gen_q #271), used to rank successor proposals. Reproduce the verification with
scripts/real/hf_verify_tree.py + hf_verify_sha.py in the rsm_rise tree; both exit non-zero on
any mismatch, and both now check every reward head and every pi0.5 member file.
| file in this repo | bytes | what it is | verified |
|---|---|---|---|
rise_actor_rsm_real_s0_e3000.pth |
3,166,418,830 | RISE actor (robomimic recipe), seed 0, epoch 3000 | sha256 f499ef21... |
rise_actor_rsm_real_s1_e3000.pth |
3,166,418,830 | RISE actor (robomimic recipe), seed 1, epoch 3000 | sha256 6c9f9f1a... |
bc_union_rsm_real_s0_e5000.pth |
3,169,668,142 | BC baseline trained on the union split, seed 0, epoch 5000 | sha256 0ee5b343... |
pi05_bc_rsm_real_stock_s1_29999/ |
12,441,308,391 (30 files) | pi0.5 BC policy, stock recipe (no RISE ingredients), seed 1, step 29999 | sha256 on all 19 binaries, 11 JSON by size |
pi05_rise_actor_rsm_real_rise_s0_29999/ |
12,441,261,368 (23 files) | pi0.5 RISE actor, the published RISE recipe, seed 0, step 29999 | sha256 on all 11 binaries, 12 JSON by size |
pi05_rise_actor_rsm_real_rise_s1_29999/ |
12,441,253,333 (24 files) | pi0.5 RISE actor, the published RISE recipe, seed 1, step 29999 | sha256 on all 13 binaries, 11 JSON by size |
pi05_rise_actor_rsm_real_riseadv_s0_29999/ |
12,441,163,909 (26 files) | pi0.5 RISE actor, advantage-weighted variant, seed 0, step 29999 | sha256 on all 16 binaries, 10 JSON by size |
pi05_rise_actor_rsm_real_sn0p03_knn0p01_s0_ws9999/ |
12,441,246,831 (28 files) | pi0.5 RISE proposer, #269 sweep arm: spectral norm 0.03, kNN 0.01, seed 0 (10k warm start off pi05_rise_actor_rsm_real_rise_s0_29999, orbax step 9999) |
sha256 on all 17 binaries, 11 JSON by size |
pi05_rise_actor_rsm_real_sn0p1_knn0p03_s0_ws9999/ |
12,441,250,229 (29 files) | pi0.5 RISE proposer, #269 sweep arm: spectral norm 0.1, kNN 0.03, seed 0 (10k warm start off pi05_rise_actor_rsm_real_rise_s0_29999, orbax step 9999) |
sha256 on all 16 binaries, 13 JSON by size |
pi05_rise_actor_rsm_real_sn0p1_knn0p05_s0_ws9999/ |
12,441,245,441 (31 files) | pi0.5 RISE proposer, #269 sweep arm: spectral norm 0.1, kNN 0.05, seed 0 (10k warm start off pi05_rise_actor_rsm_real_rise_s0_29999, orbax step 9999) |
sha256 on all 19 binaries, 12 JSON by size |
pi05_rise_actor_rsm_real_sn0p3_knn0p01_s0_ws9999/ |
12,441,236,988 (28 files) | pi0.5 RISE proposer, #269 sweep arm: spectral norm 0.3, kNN 0.01, seed 0 (10k warm start off pi05_rise_actor_rsm_real_rise_s0_29999, orbax step 9999) |
sha256 on all 15 binaries, 13 JSON by size |
pi05_rise_actor_rsm_real_sn0p3_knn0p03_s0_ws9999/ |
12,441,218,622 (26 files) | pi0.5 RISE proposer, #269 sweep arm: spectral norm 0.3, kNN 0.03, seed 0 (10k warm start off pi05_rise_actor_rsm_real_rise_s0_29999, orbax step 9999) |
sha256 on all 15 binaries, 11 JSON by size |
pi05_rise_actor_rsm_real_sn1_knn0p03_s0_ws9999/ |
12,441,237,815 (30 files) | pi0.5 RISE proposer, #269 sweep arm: spectral norm 1, kNN 0.03, seed 0 (10k warm start off pi05_rise_actor_rsm_real_rise_s0_29999, orbax step 9999) |
sha256 on all 19 binaries, 11 JSON by size |
pi05_rise_actor_rsm_real_sn1_knn0p05_s0_ws9999/ |
12,441,222,554 (27 files) | pi0.5 RISE proposer, #269 sweep arm: spectral norm 1, kNN 0.05, seed 0 (10k warm start off pi05_rise_actor_rsm_real_rise_s0_29999, orbax step 9999) |
sha256 on all 16 binaries, 11 JSON by size |
pi05_rise_actor_rsm_real_wsctrl_s0_ws9999/ |
12,441,216,238 (30 files) | pi0.5 RISE proposer, #269 sweep CONTROL: the same warm start, knobs unchanged, seed 0 (10k warm start off pi05_rise_actor_rsm_real_rise_s0_29999, orbax step 9999) |
sha256 on all 16 binaries, 14 JSON by size |
successor_rsm_real_RM_s0_100k.pt |
79,192,735 | base successor (pre-fine-tune), reward GOAL_EPISODE, seed 0 (gen_q #271) |
sha256 49516953... |
successor_rsm_real_RM_s1_100k.pt |
79,192,927 | base successor (pre-fine-tune), reward GOAL_EPISODE, seed 1 (gen_q #271) |
sha256 7ea11fb6... |
successor_rsm_real_A_110000.pt |
79,193,055 | reward head, reward GOAL_SEGMENT, seed 0 (gen_q #271) |
sha256 d437ed83... |
successor_rsm_real_A_s1_110000.pt |
79,193,055 | reward head, reward GOAL_SEGMENT, seed 1 (gen_q #271) |
sha256 cc91435a... |
successor_rsm_real_B_H100_110000.pt |
79,193,055 | reward head, reward GOAL_WINDOW_100, seed 0 (gen_q #271) |
sha256 b18a04f5... |
successor_rsm_real_B_H100_s1_110000.pt |
79,193,055 | reward head, reward GOAL_WINDOW_100, seed 1 (gen_q #271) |
sha256 4617e363... |
successor_rsm_real_B_H400_110000.pt |
79,193,055 | reward head, reward GOAL_WINDOW_400, seed 0 (gen_q #271) |
sha256 339e3891... |
successor_rsm_real_B_H400_s1_110000.pt |
79,193,055 | reward head, reward GOAL_WINDOW_400, seed 1 (gen_q #271) |
sha256 1d0eac99... |
successor_rsm_real_v0lit_110000.pt |
79,193,055 | reward head, reward EXPERT_EPISODE, seed 0 (gen_q #271) |
sha256 c8af2edb... |
successor_rsm_real_v0lit_s1_110000.pt |
79,193,055 | reward head, reward EXPERT_EPISODE, seed 1 (gen_q #271) |
sha256 50763831... |
hop_head_rsm_real_gauss_r1.pt |
19,563,355 | Gaussian hop/dynamics head used by the successor read-out | sha256 5ed03812... |
action_normalization.json |
1,664 | action mean/std of the rsm_cooking_v5 corpus | matches source in size |
successor_rsm_real_feat_stats.json |
92,929 | DINOv2 feature mean/std the successor heads were trained under | matches source in size |
meta/ |
94,769 (6 files) | dataset metadata written with the corpus (episode index, splits, stats) | 6 JSON by size |
(All 8 reward heads are byte-identical in size -- same architecture, same 10k fine-tune length -- and are different files: their sha256 all differ, as shown. Do not deduplicate them by size. The 2 base successors are the pre-fine-tune checkpoints, a different size again, and are not part of that set.)
Download
pip install -U "huggingface_hub[cli]"
R=rli14/rsm_cooking_v5_checkpoints
# one file
hf download $R rise_actor_rsm_real_s1_e3000.pth --local-dir ckpts # RISE actor, seed 1
hf download $R rise_actor_rsm_real_s0_e3000.pth --local-dir ckpts # RISE actor, seed 0
hf download $R bc_union_rsm_real_s0_e5000.pth --local-dir ckpts # BC-union control
hf download $R successor_rsm_real_RM_s0_100k.pt --local-dir ckpts # RSM successor, seed 0
hf download $R successor_rsm_real_RM_s1_100k.pt --local-dir ckpts # RSM successor, seed 1
hf download $R hop_head_rsm_real_gauss_r1.pt --local-dir ckpts # hop head
hf download $R successor_rsm_real_A_110000.pt --local-dir ckpts # default reward head (seed 0)
hf download $R successor_rsm_real_B_H400_s1_110000.pt --local-dir ckpts # reward head B, H=400, seed 1
hf download $R successor_rsm_real_B_H100_110000.pt --local-dir ckpts # reward head B, H=100
hf download $R successor_rsm_real_B_H100_s1_110000.pt --local-dir ckpts # reward head B, H=100, seed 1
hf download $R successor_rsm_real_B_H400_110000.pt --local-dir ckpts # reward head B, H=400, seed 0
hf download $R successor_rsm_real_v0lit_110000.pt --local-dir ckpts # reward head v0-literal, seed 0
hf download $R successor_rsm_real_v0lit_s1_110000.pt --local-dir ckpts # reward head v0-literal, seed 1
hf download $R successor_rsm_real_feat_stats.json action_normalization.json --local-dir ckpts
# a pi0.5 params export (each is a folder, 11.59 GiB)
hf download $R --include "pi05_bc_rsm_real_stock_s1_29999/*" --local-dir ckpts # BC (stock)
hf download $R --include "pi05_rise_actor_rsm_real_rise_s0_29999/*" --local-dir ckpts # RISE recipe, seed 0
hf download $R --include "pi05_rise_actor_rsm_real_rise_s1_29999/*" --local-dir ckpts # RISE recipe, seed 1
hf download $R --include "pi05_rise_actor_rsm_real_riseadv_s0_29999/*" --local-dir ckpts # + advantage weighting
# everything (148.6 GiB -- four pi0.5 folders dominate it)
hf download $R --local-dir ckpts
# the small stuff only (actors + successors + reward heads + metadata, no pi0.5): 9.61 GiB, 24 files
hf download $R --exclude "pi05_*" --local-dir ckpts
(huggingface-cli download <same args> on hub < 1.0.)
from huggingface_hub import hf_hub_download, snapshot_download
R = "rli14/rsm_cooking_v5_checkpoints"
actor = hf_hub_download(R, "rise_actor_rsm_real_s1_e3000.pth")
pi05 = snapshot_download(R, allow_patterns="pi05_bc_rsm_real_stock_s1_29999/*")
Recipe (one line each)
| file | what it is | recipe |
|---|---|---|
rise_actor_rsm_real_s1_e3000.pth |
RISE actor (rsm_real) -- the IDQL/RSM proposer and its co-trained IQL critic (seed 1) | BC-transformer actor trained jointly with an IQL critic (S0Aidql) on the expert+suboptimal union of rsm_cooking_v5, 3000 epochs |
rise_actor_rsm_real_s0_e3000.pth |
RISE actor (rsm_real), second proposer seed (seed 0) | as above, 3000 epochs |
bc_union_rsm_real_s0_e5000.pth |
BC-union (rsm_real, RISE stack) -- in-stack control (no critic) | same backbone/data, plain BC objective (S0B), 5000 epochs |
pi05_bc_rsm_real_stock_s1_29999/ |
pi0.5 BC (rsm_real), seed 1, step 29999, params only | openpi pi05_cooking_stock fine-tune of pi0.5 on rsm_cooking_v5, 30k steps |
pi05_rise_actor_rsm_real_rise_s0_29999/ |
pi0.5 RISE proposer (rsm_real), seed 0, params only | as the stock row plus the RISE recipe: kNN action augmentation at 1 cm + spectral norm 0.1, 30k steps |
pi05_rise_actor_rsm_real_rise_s1_29999/ |
pi0.5 RISE proposer (rsm_real), seed 1 | as above, second seed |
pi05_rise_actor_rsm_real_riseadv_s0_29999/ |
pi0.5 RISE proposer + advantage weighting, seed 0 | as above, with IQL advantage weighting on the BC loss |
successor_rsm_real_RM_s0_100k.pt |
RSM successor (rsm_real) (R, size M), 100k steps | successor-feature regression on DINOv2 state features, discount/horizon per history.json |
successor_rsm_real_RM_s1_100k.pt |
RSM successor (rsm_real), second seed | as above |
hop_head_rsm_real_gauss_r1.pt |
hop head (gaussian) used by the successor scorer | one-hop action head, round 1 |
successor_rsm_real_A_110000.pt |
reward head A (the default), seed 0 | a 10k fine-tune of RM_s0_100k under reward scheme A; the head rides inside the same blob as state_dict["rew"] |
successor_rsm_real_A_s1_110000.pt |
reward head A, seed 1 | as above, off RM_s1_100k |
successor_rsm_real_B_H100_110000.pt |
reward head B, H = 100 |
arrival-window label, r_t = 1[0 <= a_i - t <= 100] (one hop at beta = 0.99) |
successor_rsm_real_B_H100_s1_110000.pt |
reward head B, H = 100, seed 1 |
as above off RM_s1_100k; not significant on the stitching number even in the draw that favoured B |
successor_rsm_real_B_H400_110000.pt |
reward head B, H = 400 |
the same, four hops' worth; led the stitching number in one candidate draw, did not replicate |
successor_rsm_real_B_H400_s1_110000.pt |
reward head B, H = 400, seed 1 |
as above off RM_s1_100k |
successor_rsm_real_v0lit_110000.pt |
reward head v0-literal, seed 0 | the #271 CONTROL label as written there, r_t = 1[f_i in {S3,S4}]: every frame of any episode ending with the tomato in the bowl, pan on the grate or still held. Above chance in 5/5 candidate draws at both hops — the only head that is — at 0.5212 (hop 4) against the base head's 0.4872 |
successor_rsm_real_v0lit_s1_110000.pt |
reward head v0-literal, seed 1 | as above off RM_s1_100k; 0.5037, draw-level p = 0.017 |
meta/ |
feature statistics + feature config the scorer needs | see Successor below |
All three .pth files are full robomimic-format checkpoints (robomimic is the training framework
RISE ships in -- it is the file format, not a data source; every file here is trained on rsm_cooking_v5).
The two RISE-actor files carry both the actor (proposer) and the co-trained IQL critic in one file; FileUtils.policy_from_checkpoint returns the
actor, and the critic tensors sit under the model state dict.
Held-out fit vs the hold-still predictor
Replay evaluation on held-out validation episodes (25 episodes, stride 10). Lower is better; the hold-still predictor (repeat the current pose) is the trivial baseline any real policy must beat.
| policy | first-action L2 | chunk L2 | gripper acc |
|---|---|---|---|
| hold-still baseline | 0.81 | 1.53 | -- |
| RISE actor seed 1 @3000 | 0.396 | 0.489 | 0.973 |
| RISE actor seed 0 @3000 | 0.427 | 0.534 | 0.971 |
| BC-union seed 0 @5000 (control) | 1.689 | 1.778 | 0.952 |
| pi0.5 BC seed 1 @30000 | 0.83 | 1.16 | 0.968 |
The two RISE-actor proposers cut first-action error roughly 2x below hold-still; the plain-BC BC-union control does not beat hold-still on this dataset, which is the point of including it.
Why the three pi0.5 RISE proposers look worse on this table -- read before you screen them
The pi05_rise_actor_* folders are deliberately wider proposers, not better regressors, and a
greedy-fit number will rank them below pi05_bc_* every time. Planning screen over 10 held-out episodes /
243 decisions, N=128 flow samples per decision, first-action xyz error in cm:
| pi0.5 proposer | candidate spread | greedy | oracle over the 128 | planning headroom |
|---|---|---|---|---|
| BC (stock) s1 | 0.164 | 0.737 | 0.475 | 0.262 |
| RISE s0 | 0.329 | 0.932 | 0.432 | 0.500 |
| RISE s1 | 0.336 | 0.959 | 0.448 | 0.511 |
| RISE + adv. s0 | 0.420 | 1.103 | 0.452 | 0.651 |
The RISE recipe widens the candidate set 2.0-2.6x; each single sample gets worse, but the best candidate in the set gets better (oracle 0.475 -> 0.432 cm) and the prize a scorer could win grows 1.9-2.5x. So greedy fit measures the wrong thing for a proposer that exists to be selected over.
Nothing harvests that headroom yet. On these pi0.5 candidates both the IQL critic and the RSM successor score above their own permuted controls, with oracle-rank percentiles below 0.5 -- selection is anti-correlated with the truth, not merely uninformative. Both scorers were fit on the small-stack actor's candidate distribution and are out of distribution here; on that actor, whose critic was co-trained on its own candidates, IDQL does beat permuted (0.527 vs 0.562 cm, rank 0.590 vs 0.517). Treat the 0.5-0.65 cm of headroom as real and unclaimed: to use it, fit the scorer on the candidates of the proposer you intend to deploy.
Loading
robomimic (.pth):
from robomimic.utils.file_utils import FileUtils
policy, ckpt = FileUtils.policy_from_checkpoint(
ckpt_path="rise_actor_rsm_real_s1_e3000.pth", device="cuda:0", verbose=True)
policy.start_episode()
action = policy(ob=obs_dict) # obs: agentview_image, robot0_eye_in_hand_image, ee_state
openpi / pi0.5 (pi05_bc_rsm_real_stock_s1_29999/) -- this is an inference-only export: the orbax params
subtree plus assets/ (norm stats); the 31 GB train_state (optimizer) is dropped.
from openpi.policies import policy_config
from openpi.training import config as _config
cfg = _config.get_config("pi05_cooking_stock")
policy = policy_config.create_trained_policy(cfg, "pi05_bc_rsm_real_stock_s1_29999")
out = policy.infer(obs) # out["actions"]
successor (RSM) -- loaded by scripts/real/twin_rollout.py / scripts/real/scorer_openloop.py
in rsm_rise (and by the rsm_real scaffold). The scorer needs the checkpoint, the hop head, and the
feature statistics, not a calibration file:
import torch, json, numpy as np
ck = torch.load("successor_rsm_real_RM_s0_100k.pt", map_location="cpu", weights_only=False)
st = json.load(open("successor_rsm_real_feat_stats.json")) # JSON copy of feat_stats.npz
mu, sd, z_clip = np.array(st["mu"][:1546]), np.array(st["sd"][:1546]), float(st["z_clip"]) # 10.0
# state = clip((feat - mu) / max(sd, 0.01), -z_clip, z_clip), feat = [z(1536) | proprio(10)]
Note: the brief for this upload expected a
calib.jsonnext to each successor checkpoint. There is nocalib.jsonfor the real cells -- only the sim cells have one. What the real scorer actually reads is the feature statistics (mu,sd,z_clip), the cache index (ep_from) and the feature recipe. Those ship here as plain JSON --successor_rsm_real_feat_stats.json,meta/successor_rsm_real_cache_meta.json,meta/successor_rsm_real_feature_config.json-- rather than as the original.npz; the numbers are identical (.npzis an LFS pattern, and these went up while the repo still had no private LFS headroom -- the JSON copies are kept because they are what the loading snippet above reads).
Which reward head the successor carries, and how far to trust it
There is no separate reward-head file, and you do not need one. The reward head travels inside the
successor .pt as state_dict key "rew" (saved rsm_libero/train_rsm.py:1094, loaded
rsm_libero/is_ope.py:117 via robomimic/algo/rsm_scorer.py:518, used in the value at
rsm_scorer.py:340,442). Downloading successor_rsm_real_RM_s0_100k.pt downloads the reward head with
it -- one file, one download.
The head inside successor_rsm_real_RM_s0_100k.pt is v0 provenance: dense +1 on every frame of the
50 episodes that reach state 4. It went in because none of the four alternative reward labels beat it. Each was fine-tuned off this exact
checkpoint for 10,000 further steps with everything else held identical (architecture, cache, optimiser
state, beta band, chunk, seed, split); the largest paired delta on the open-loop screen is
+0.019 +- 0.011 cm, with the two seeds disagreeing in sign. So no fine-tune is mirrored in this repo, and
there is only one name for the shipped head -- the file above.
The value is not validated for selection. "Best head measured" is not "works". At a junction state that a failure trajectory passes through, the successor's Q ranks action chunks drawn from the rewarded continuation below chunks drawn from the failure continuation: top-1 of 128 is a rewarded continuation in 0.094 of decisions against a chance rate of 0.50. The head separates route-3 from route-4 junction frames with AUC 1.000, and already reads
+24logits on route-3 frames before anything has happened -- it is reading which recording a frame came from, not progress.The cause is structural to this corpus rather than to any checkpoint: the reward label is constant over a whole recording, and the only rewarded route (
2->3->4) is also the only route that starts at the junction state, so "reaches the goal" and "is a route-3 recording" are the same event in the data.Use it to rank proposals; do not gate robot actions on its value. The smallest fix is new data -- a route that traverses
0->...->3->4, or a failure continuation recorded out of state 3 inside a route-3 session. Full measurement, the control that explains it, and the stage pictures: gen_q #261 comment 5771720298.
The reward schemes, stated mathematically (gen_q #271)
The reward is now specified from the operator's annotated state graph, not from episode provenance.
Notation, with the stage words spelled out: t is a frame of episode i; to(t) is the state the
annotated segment containing t leads into; k(t) is the operator's arrival marker; a_i is the frame at
which episode i first reaches the goal; f_i is episode i's final state; and
S3= pan lifted off the stove, tomato in the bowl,S4= pan on the grate, tomato in the bowl -- the goal,S5,S6= the failure terminals (tomato never made it to the bowl; pan left on the stove with the tomato in the bowl).
The per-frame label handed to the trainer is then
| scheme | label | in words |
|---|---|---|
v0 (control) |
r_t = 1[f_i in {S3, S4}] |
every frame of an episode that ends with the tomato in the bowl |
A (default) |
r_t = 1[to(t) = S4] |
every frame of the segment that leads into pan-on-grate-tomato-in-bowl |
B |
r_t = 1[0 <= a_i - t <= H], H = L / (1 - beta) |
the last H frames before the goal is reached; 0 in episodes that never reach it |
C |
r_t = 1[k(t) = S4] with the weighted loss below |
A's positives, with the failure terminals counted harder |
beta = 0.99 is the read-out's discount and L the number of successor hops it chains, so the flagship
one-hop read-out gives H = 1/(1-0.99) = 100 frames and the fold-4 read-out H = 400; both are trained.
Scheme C's loss is
L_R = sum_t w_t * BCE(R(z_t), r_t) / sum_t w_t ,
w_t = w_neg if to(t) in {S5, S6} (label 0, the hard negatives)
1 otherwise
with w_neg swept over {3, 10}. Dividing by sum_t w_t keeps the scale of the reward term fixed as
w_neg moves, so changing w_neg changes emphasis and not the effective learning rate; with no weight
array the loss reduces bit-for-bit to the unweighted binary_cross_entropy_with_logits used before.
The default head is successor_rsm_real_A_110000.pt (seed 1: successor_rsm_real_A_s1_110000.pt), a
10,000-step fine-tune of successor_rsm_real_RM_s0_100k.pt in which only the label array changed.
Why A and not v0: at the junction state that both the success route and a failure route pass through, v0
already reads p ~ 1.000 on the success route's frames before anything has happened (junction AUC
route-3 vs route-4 = 1.000), because its label is constant over a whole recording. A's junction
probabilities are at numerical zero on every route (junction AUC 0.745 / 0.986 over the two seeds, an
ordering inside a numerically-zero tail) while it still separates S4 from S6 at AUC 1.000 -- it has
encoded where the pan ends without encoding which recording the frame came from. The caveat above about
not gating robot actions on the value applies to every head measured so far, A included.
Which heads are on the hub, and why those. Scheme A ships unconditionally as the declared default.
Every other head shipped here cleared the stitching statistic -- not the open-loop screen, on which no
head beats the permuted control at all (best delta -0.054, and that column does not replicate across
proposer seeds). The B family was uploaded from the single published candidate draw; the five-draw
replication that came later (status/real/STITCH_DRAWS.json in rsm_rise, the design that varies only the
draw seed) reordered it, and successor_rsm_real_v0lit_* were added afterwards to match: at hop 4 the
draw-level ranking is B_H400 0.5267, B_H400_s1 0.5252, v0lit 0.5212 (each above chance in 5/5
draws, p < 1e-3) and v0lit_s1 0.5037 (p = 0.017), against base R_M 0.4872 (0/5) and a between-draw SD
of 0.0139. B_H100 (0.5112, p = 0.126) and B_H100_s1 (0.4953, p = 0.560) predate that design and are
kept, but they are not draw-level significant. Presence on the hub is therefore a record of what was
measured, not an endorsement -- see the next paragraph.
No head here is validated for selection -- read this section before choosing one. Two measurements
were run. An earlier version of this card said one of them (the stitching number) could rank the schemes
and named successor_rsm_real_B_H400_110000.pt as the head that stitches. That claim is retracted:
both measurements have since been given the same replication check, and both have a noise floor the size
of their own largest effect. The numbers below are reported as measured, followed by what happens when
the measurement is repeated.
(1) The stitching number. At each of 150 S2-junction frames of a failure-route episode
(S0 -> S2 -> S6), the S0Aidql proposer draws 128 action chunks; each is chained four hops (400 frames)
through the shared base successor flow, so only the reward differs and never the dynamics. The
endpoint latent is retrieved against 30,255 cached frames and the nearest frame's segment label classifies
it GOAL (to = S4) or FAIL (to in {S5,S6}). The statistic is
AUC_i = P[ Q(c) > Q(c') | c ends GOAL, c' ends FAIL ] computed INSIDE anchor i
averaged over the 132 anchors that produced both classes, with a 10,000-resample bootstrap over anchors.
It must be read within-anchor: anchors sit at very different absolute Q levels, so a pooled AUC (0.63-0.74
even for heads that are at chance inside a decision) confounds ranks candidates correctly in this decision
with scores some junctions higher than others, and only the first is stitching.
| head | within-anchor AUC | 95 % CI | p vs 0.5 | delta vs base head (p) |
|---|---|---|---|---|
successor_rsm_real_B_H400_110000.pt |
0.544 | [0.519, 0.568] | 0.0004 | +0.055 (0.0012) |
successor_rsm_real_B_H400_s1_110000.pt |
0.537 | [0.512, 0.562] | 0.0027 | +0.048 (0.0036) |
successor_rsm_real_B_H100_110000.pt |
0.534 | [0.507, 0.561] | 0.0073 | +0.045 (0.0002) |
successor_rsm_real_B_H100_s1_110000.pt |
0.509 | [0.484, 0.535] | 0.251 | +0.020 (0.123) |
successor_rsm_real_A_110000.pt (the default) |
0.509 | [0.483, 0.535] | 0.245 | +0.020 (0.087) |
scheme C, w_neg = 3 / 10 |
0.502 / 0.500 | -- | n.s. | n.s. |
provenance head inside successor_rsm_real_RM_s0_100k.pt |
0.498 | [0.470, 0.525] | 0.563 | +0.009 (0.057) |
| base successor, no reward fine-tune | 0.489 | [0.460, 0.518] | 0.784 | -- |
| a head trained on the final segment of every episode | 0.456 | [0.434, 0.478] | -- | -0.033, below chance |
In this candidate draw three of B's four cells clear chance -- B_H400, B_H400_s1 and B_H100 -- and each of those three also
clears the un-fine-tuned base head paired on the same anchors and candidates. B's fourth cell,
B_H100_s1, has the right sign but does not reach significance; it is named rather than folded into "the
B family", because a family headline would carry it along with the three that do clear. Nothing outside B
is distinguishable from chance. The last row is the sign check: a label that is positive on the final
segment of every episode, failures included, reads significantly below chance, so the nulls above are
real nulls and not insensitivity. Four hops is the only answerable depth -- at one hop 0.000 of
endpoints reach a GOAL frame and at two hops 0.049.
Take the effect size seriously: AUC 0.53-0.54 means the stitching candidate is preferred in about 54 % of GOAL-vs-FAIL pairs, and B is ahead of chance at 61 % of junctions rather than 50 %. Enough to rank proposals; not a head to plan with.
(1b) ...and it does not survive an independent candidate draw. This card previously argued that the
stitching number could be believed where the screen in (2) could not, because "it reproduces: recomputed
from scratch after a fifteenth head was added, every pre-existing head returned the identical AUC to four
decimals". That is a determinism check: each anchor's 128-candidate draw is seeded with
seed*1000003 + anchor_index, so a rerun over the same anchors cannot return anything else. (It was
re-confirmed once more, with a sixteenth head added -- every pre-existing head is identical to four
decimals -- which is why the disagreement below can only be the draw.) The screen's noise floor in (2) was
always measured across an independently re-drawn candidate set, so the two were never on the same
footing.
Measured the screen's way -- rerun at a finer anchor stride, which visits the same anchor frames with completely different candidates, then restricted back to the published anchors (129 usable against 132), so anchors and weights are fixed and only the draw changes:
| head | published draw | independent redraw, same anchors |
|---|---|---|
B_H400 |
0.544 (p = 0.0004) | 0.513 (p = 0.161) |
B_H400_s1 |
0.537 (p = 0.0027) | 0.495 (p = 0.626) |
B_H100 |
0.534 (p = 0.0073) | 0.485 (p = 0.858) |
| v0 as written in the issue (a control) | 0.531 (p = 0.0094) | 0.521 (p = 0.065) |
A (the default) |
0.509 (n.s.) | 0.496 (n.s.) |
| provenance head (shipped) | 0.498 (n.s.) | 0.472 (n.s.) |
| base successor, no fine-tune | 0.489 | 0.466 |
| final segment of every episode | 0.456 | 0.466 |
Over the 16 shared heads the within-anchor AUC moves by rms 0.022 (max 0.049) against a largest reported effect of +0.055, and pairing against the base head barely absorbs it (rms 0.019) -- the two draws differ in which candidates were offered, not by a common difficulty offset. Significantly above 0.5 in the published draw: four heads. In the redraw: none. Surviving both: none.
The fourth of those four is the point of no return for the proximity story. v0lit =
r_t = 1[f_i in {S3,S4}] is positive on every frame of any episode that ends with the tomato in the
bowl -- 132,350 frames, 73 % of the corpus, no proximity structure at all. The structural argument above
predicts exactly that shape of label at chance. It clears chance alongside B in the published draw and is
the highest head in the redraw. So the statistic is responding to something other than the property
the schemes differ in.
What this means for choosing a file: nothing here licenses "use B_H400 to rank". The one thing that replicates is the sign check -- the final-segment label is below chance in both draws -- so the statistic is not merely insensitive; it is just far too noisy, at this anchor count and this candidate spread, to resolve differences of the size that separate these schemes.
(1c) Why it does not replicate -- three explanations, two of them wrong. All three are read-only
questions about the cache the diagnostic already wrote (scripts/real/stitch_mechanism.py in rsm_rise ->
status/real/STITCH_MECHANISM.json).
Not "every head re-scores one shared ordering". Candidates are chained through the same base successor
flow for every head, so the reward looks like it should only lightly perturb a fixed ranking. It does not:
the within-anchor Spearman of a head's Q against the base head's runs from -0.026 (B_H400_s1, which
orders 51 % of candidate pairs differently from the base) to 0.929 (v0, the provenance head, which
orders 8.9 % differently and is therefore nearly the base head); the least similar pair of heads is
anti-correlated at -0.512. Sixteen materially different rankings of the same candidates all land in
0.46-0.54, so the endpoint class is not recoverable from Q under any of these rewards -- which is why
a flat control can come out on top of a draw.
Endpoint collapse is real but does not explain it. At four hops an anchor's 128 candidates end on a median
of 44 distinct retrieval rows, the GOAL class alone on a median of 3, and the modal endpoint takes 44 % of
the candidates. Splitting the 132 anchors at the median number of distinct GOAL endpoints still leaves
B_H400 at 0.550 (p = 0.001) on the 85 collapsed anchors and 0.532 (p = 0.043) on the 47 diverse
ones: collapse sharpens the effect rather than creating it.
Power. Shuffling the GOAL/FAIL labels inside each anchor, class sizes preserved, puts the statistic at 0.5000 +- 0.0121 (95 % [0.476, 0.524]) at four hops. So +-0.024 is the smallest effect this design can resolve, against a largest reported effect of +0.055 -- and the candidate-draw spread (rms 0.022) is larger still, which is the component the published bootstrap-over-anchors interval cannot see.
Correction. An earlier version of this card said hop 1 and hop 2 both "read nan or no-signal for
everyone". Hop 1 is genuinely undefined (no chain that short reaches an S4 frame); hop 2 is not -- it
has 100 usable anchors and reads flat (top head v0lit 0.511, p = 0.29; no head clears chance).
(1d) WHERE the signal dies, and why no reward label could have satisfied this diagnostic. Every head
separates real S4 frames from real S6 frames at AUC 1.000 (the first diagnostic above), yet the Q
it induces is at chance on chained candidates. scripts/real/stitch_decompose.py in rsm_rise ->
status/real/STITCH_DECOMP.json reads three quantities per candidate, on the published draw's own anchors
and its own candidates, with Q recomputed in the same process so that nothing rests on a cross-run
assumption (the replayed endpoint rows match the published ones at 1.000):
R(z_h) at the chained latent; R(x[nn(z_h)]) at the real cached frame the endpoint retrieves to -- the
ceiling, because that row's edge_to is the endpoint label; and Q itself.
The latent keeps the class; the read-out discards it. C10_s1, v0lit_s1 read the chained latent at per-anchor
AUC 0.941 and above, on a real separation of means
(0.676 on GOAL-ending chains against 0.052 on FAIL-ending
ones for C10_s1), and the Q they induce is 0.500, exactly
chance (corr(Q, R(z_h)) = 0.278). So the signal does not die off-manifold; it
dies in the fold.
And the read-out loses it because its horizon does not match the label's. The read-out every number here
uses is depth 1 at beta = 0.99, so Q sees about one hop (100 frames), while the endpoint class is
defined four hops out (400 frames). Of the two ways to widen that window, raising beta is
genuinely unavailable -- the flow was conditioned on beta in [0.95, 0.99], so beta = 0.99 already is the
longest horizon it was ever trained at -- but deepening the fold is available: a depth >= 2 read-out
needs a hop head, and although no successor blob embeds one, rsm_hop_ckpt is a separate checkpoint
argument and this repo publishes a hop head for exactly this corpus
(hop_head_rsm_real_gauss_r1.pt), already used by the fold-4 column of the open-loop screen. (An earlier
revision of this card called that lever closed and concluded that no reward label could satisfy the
diagnostic; that is withdrawn.) What stands regardless is the size of the mismatch: at one hop there are
0 GOAL endpoints out of 13,199 classified candidates. Within the
horizon Q can see, no candidate drawn at the failure route's S2 junction reaches a goal-bound segment
at all. So the depth-1 statistic was never a test of the reward labels; whether the schemes separate at
all is what the matched-horizon fold-4 read-out measures -- the closing paragraph below.
Two cautions that travel with the table. B_H100, B_H400, B_H400_s1, B_H100_s1, C10, C3, C3_s1, base, v0, v2 are saturated off at the chained latent
(R(z_h) ~ 0 on both classes), so their AUCs out there rank numerical dust and must not be read as an
effect in either direction; v3 by contrast is genuinely inverted
(0.013 GOAL against 0.698 FAIL). The chained
latent does sit 5.9x a real frame's own nearest-neighbour L1
distance away from the corpus, so the off-manifold story holds for the majority of heads even though it
does not explain the phenomenon. And the first diagnostic's ceiling has a crack worth carrying: R read at
the retrieved real frame is 1.000 for most heads but 0.080 for
v2 and 0.75-0.85 for the whole B family. That diagnostic labels frames by reached_state_id;
these are frames labelled by edge_to at the endpoints the chains actually reach, and there the
horizon-window schemes are not at ceiling.
Correction to (1b). (1b)'s redraw changed the anchor stride, which moves the anchor set as well as the
candidate draw, so its "surviving both: none" is a confounded replicate. The clean design -- identical
anchors and weights, only the seed of the draw changing -- is 5 independent draws
(status/real/STITCH_DRAWS.json): above 0.5 in every draw
v0lit, B_H400, B_H400_s1, draw-level significant v0lit, v0lit_s1, B_H400, B_H400_s1. The largest is B_H400 at 0.5267 (sd 0.0108,
above 0.5 in 5/5 draws), and the published single-draw
0.5437 was the top of its range [0.518, 0.544] -- the luckiest
of the five.
And widening the window does not rescue it: the matched-horizon read-out was measured. A depth-4 fold
through this repo's own hop head (hop_head_rsm_real_gauss_r1.pt) gives a read-out whose horizon matches
the four-hop depth the endpoint class is defined at. Both runs reproduce the published candidate draw
exactly (endpoint match 1.000 and 1.000), so only the
read-out differs. Over the 16 heads the within-anchor AUC moves by rms
0.0028, max -0.0064 (B_s1) -- below the
+-0.024 this design can resolve and below the ~0.005 that GPU non-determinism alone contributes. The fold
is not idle: the fold-4 Q level rises by +0.015 to +0.102 per head, but its within-anchor
correlation with the depth-1 Q is 0.930-1.000 (mean 0.977) over all
150 anchors -- the extra hops add a common-mode offset, not candidate-specific
information, so the ordering survives untouched. The horizon mismatch is therefore falsified as the
reason Q is at chance. The endpoint class is in the latent and in the head at AUC 0.941 and both
read-outs discard it, which left the scoring functional (prob / T = 25) as the thing to vary next --
not the reward labels and not the horizon. Numbers: status/real/FOLD4_COMPARE.json in rsm_rise. So "no head clears chance" was too strong; averaged over draws the horizon-window heads clear
chance by about +0.03. Read with the paragraphs above, a +0.03 effect at a horizon where the endpoint
label is invisible to the read-out is a weak leading indicator inside the first hop, not demonstrated
stitching -- which is why nothing shipped here changes: A is still the default head and no head is
validated for selection.
And the scoring functional is not the culprit either. That was the last degree of freedom left in the
read-out, and it had never been varied -- the decomposition hard-coded score_kind="prob" at the scorer.
Swept across its natural bracket (prob / T = 25 (flagship), logit, prob / T = 1, prob / T = 100, prob_log) on identical anchors, candidates, chains and GOAL/FAIL
labels (controlled: every cell reproduces the published endpoints exactly), the result is null: the best within-anchor AUC anywhere in the sweep is 0.5437
(B_H400, prob_log) against the flagship's own best of
0.5385 -- a lift of +0.0052, against a design resolution of +-0.024 and a
GPU-non-determinism floor of ~0.005. The largest per-head move any functional makes is 0.0304
(prob_T1), and that one moves the statistic down (its own best is 0.5314). The bracket
passes its own validity check: prob / T = 100 reproduces logit to 3--4 decimals, as the algebra
requires. The decisive row is C10_s1 -- it reads the endpoint class at the chained latent at AUC
0.941 while its Q stays 0.4929--0.5051
under all five functionals. So three suspects are now excluded on identical candidates: the reward
label, the read-out horizon and the scoring functional. Numbers: status/real/SK_COMPARE.json in
rsm_rise.
Nor is it how the fold is rolled out -- and that closes the read-out. The three variations above all
left one uncontrolled difference between Q and the GOAL/FAIL label it is scored against, and it is
structural rather than a matter of degree: they are values of two different trajectories. The label's
own rollout computes the candidate's action chunk once and re-applies that same chunk at every hop;
the fold re-samples the hop head between hops, so at depth four it estimates the value of apply the
candidate once, then follow the hop head for three hops. --q-fold-openloop repeats the candidate at
every hop, which is exactly the label's generating process (additive, default-off, a no-op at depth one,
and the regression smoke reproduces the pre-patch fold-4 cell bit-exactly). The control is asserted
array-by-array, because the chains, endpoints and labels are computed outside the scorer: 15300/15300 chain arrays bit-identical over
150/150 anchors, endpoint match 1.000 and 1.000, with
2400/2400 Q arrays moved. The result is null: the best within-anchor AUC goes from
0.5402 (B_H400) to 0.5396 (B_H400_s1), a lift of
-0.0006; rms move 0.0119, largest single move -0.0252 (v0lit), against
the same +-0.024 design resolution. The prediction was stated in advance and fails: C10_s1, which
reads the endpoint class at the chained latent at AUC 0.941, moves 0.4989 -> 0.4891
and its correlation with R(z_h) falls 0.292 -> 0.271. So the read-out is
exonerated on all four axes. The reason is worth stating precisely, because it is not nothing can be
done: Q is a discounted occupancy of the reward set along the chain while the label is the class
of the endpoint at hop four; matching the policy removes the trajectory mismatch, not the
functional-of-the-trajectory mismatch. What is left is what Q is, not how it is rolled out. Numbers:
status/real/OL_COMPARE.json in rsm_rise.
(1e) Which canonical name is in which file. rise_stack names these labels in words
(REWARD_NAMES.md); this repo names the files after the old ids, so the map is worth stating once. The
rows are read from rise_stack/configs/reward_names.json and from hf_upload.py's upload table, not
typed. The one to be careful with: successor_rsm_real_A_110000.pt is the GOAL_SEGMENT head
(scheme A, r_t = 1[to(t) = S4], cell R_M_ft_v1) -- not GOAL_EPISODE, whose label is the
provenance one the base successor already carries.
| canonical name | old id in these file names | positive frames | where it lives here |
|---|---|---|---|
GOAL_EPISODE |
v0_shipped_final_eq_S4 |
53,367 | inside the base successor_rsm_real_RM_s0_100k.pt (state_dict["rew"]); the R_M_ft_v0 fine-tune is trained, not uploaded |
GOAL_SEGMENT |
A_to_eq_S4 |
30,292 | successor_rsm_real_A_110000.pt |
EXPERT_EPISODE |
v0lit_final_in_S3_S4 |
132,350 | successor_rsm_real_v0lit_110000.pt |
GOAL_WINDOW_100 |
B_window_H100 |
5,050 | successor_rsm_real_B_H100_110000.pt |
GOAL_WINDOW_400 |
B_window_H400 |
20,050 | successor_rsm_real_B_H400_110000.pt |
GOAL_SEGMENT_CONTRASTIVE |
C_label_to_eq_S4 |
30,292 | trained, not uploaded |
(1f) These numbers reproduce with the DINOv2 YOU will build. Every R(z) number in this card was
measured on the training cluster, where the DINOv2 backbone is built from a local archive of
facebookresearch/dinov2. Your clone has no such archive and fetches that source from GitHub, so the
features feeding the head are computed by a different copy of the model code -- while the checkpoint
itself is bit-identical either way. A reward head is only as reproducible as its features, so this is
checked rather than assumed. On 64 frames of the shipped replay fixture, featurised
through both sources: the two source trees are the same snapshot (157 of
157 .py files sha256-identical), z is bit-identical (max abs diff 0), and
successor_rsm_real_A_110000.pt returns bit-identical logits (max abs diff 0, identical ranking of all
64 frames). Checked 2026-09-24T03:40:37Z, and re-checked on every audit of gen_q #271 (check J,
scripts/real/audit271_J_dino_source.py), with the GitHub source re-fetched each time so the check
tracks main instead of a frozen snapshot of it.
(1g) Each file is verified to carry the LABEL its name claims. (1e) is a map; this is the check
that the map is true of the WEIGHTS. A file could be named, hashed, and reproduce its features
perfectly while having been fine-tuned on the wrong label, and nothing above would notice. So every
delivered head is read twice, independently: the training label is rebuilt from the checkpoint's own
stored config through the trainer's own data pipeline, and the head is scored over all 181,530
corpus frames to see which label R(z) actually fits best. For all 10/10 heads the rebuilt
label is the one this card advertises, and the advertised label is the strict argmax of AUC -- at
1.0000 for every head, against a permuted null measured per head that lands at chance (max over
5 draws per head: 0.5010-0.5088). Note what this is: an IN-SAMPLE identity
test, the fingerprint that says which reward a file optimises. It is not a generalisation number and
must not be quoted as one. Checked on every audit of gen_q #271 (check L,
scripts/real/audit271_L_label_provenance.py; negative control
scripts/real/_audit271_L_selftest.py).
(2) The RISE-recipe open-loop screen -- it fails two different reproducibility checks, so do not rank
schemes with it. N = 128 chunks from the S0Aidql proposer at each of 705 held-out decisions per seed;
every head scores the identical candidate set, so each delta is paired decision by decision; the control is
permuted:rsm (scores shuffled within a decision, which makes the argmax a uniform index -- so the control
is random-of-128); 95 % bootstrap over decisions. Metric oracle_rank_pct, 0.5 = chance.
| scorer | proposer seed 0 | proposer seed 1 | pooled (1410 decisions) |
|---|---|---|---|
permuted:rsm (the control) |
0.494 | 0.477 | 0.485 |
permuted:idql / greedy (independent draws of the same null) |
0.497 / 0.511 | 0.511 / 0.486 | 0.504 / -- |
idql critic -- the RISE recipe's own scorer, not one of these heads |
0.584 | 0.417 | 0.500 |
successor_rsm_real_B_H100_110000.pt |
0.530 | 0.364 | 0.447 (-0.038) |
successor_rsm_real_B_H400_s1_110000.pt |
0.541 | 0.342 | 0.442 (-0.044) |
successor_rsm_real_B_H400_110000.pt |
0.530 | 0.350 | 0.440 (-0.045) |
provenance head inside successor_rsm_real_RM_s0_100k.pt |
0.482 | 0.367 | 0.424 (-0.061) |
successor_rsm_real_B_H100_s1_110000.pt |
0.468 | 0.374 | 0.421 (-0.065) |
successor_rsm_real_A_110000.pt (the default) |
0.432 | 0.359 | 0.396 (-0.089) |
Check one -- proposer seeds. On proposer seed 1 every learned scorer is anti-correlated with the oracle
-- including the idql critic, which is not a reward head at all -- while all three nulls stay inside
[0.477, 0.511], exactly where a null belongs. That indicts the screen, not the labels: the seed-1
proposer's candidate cloud is tighter (0.217 cm vs 0.256 cm spread) and its oracle-best candidate is
already on the expert action (0.111 cm vs 0.193 cm), so any learned preference for a distinctive
candidate carries you off it. Pooled, no head and no critic beats the control.
Check two -- an independent candidate draw. The screen has been run twice end to end on the same decisions and the same weights, with the proposer's 128 candidates re-drawn. That is a replicate of the whole measurement, not of the bootstrap over decisions. Between the two runs the quoted paired delta moves by rms 0.023 (max 0.040), and pairing against the control barely reduces it, because the runs differ in which candidates were offered rather than by a common difficulty offset. The noise floor is therefore the same size as the largest head effect this screen ever reported (+0.048), and the bootstrap intervals -- which hold the candidate draw fixed -- are optimistic.
The argument that was made for B, and why it does not stand. A, C and the provenance label are flat
indicators of this segment (or episode) ends in S4, so R(z) is constant across the whole positive
set and a candidate whose chain lands closer to the goal scores exactly the same as one landing at the
far end. B's arrival window, 1[0 <= a_i - t <= H], is the only label whose value varies with proximity
to S4 -- the gradient a successor read-out needs. That is a reasonable prediction, and the stitching
table was read as confirming it. It is not confirmed: the ranking does not survive a candidate redraw, and
v0lit, a flat episode-level label the argument predicts at chance, clears chance alongside B. Nor does
any of it follow from the S4-vs-S6 column: most heads sit at AUC 1.000 there because the probabilities
are saturated (p is 1.000 or ~1e-10), which means perfectly ordered, not informative.
So: A is the project's declared default and is what rsm_reward: points at -- by choice, not by
measurement. No head in this repository is validated for selection, neither for ranking proposals
nor for gating an action. B_H400 is the head that led both statistics before the replication checks, so
it is the first one to try if this line of work is picked up; it is not a measured winner.
Issue: gen_q #271.
State features (DINOv2)
dinov2_vits14_clsgem, crop square, cameras ['global', 'wrist'], z layout
global CLS(384) | global GeM(384) | wrist CLS(384) | wrist GeM(384) (dim 1536), proprio observation.state = [x,y,z,R00,R10,R20,R01,R11,R21,gripper_width] (dim 10),
state dim 1546. Full image pipeline:
mp4 (PyAV, rgb24, presentation order, asserted 1:1 with parquet rows) -> uint8 HWC -> float/255 -> CHW -> square transform -> ImageNet Normalize -> dinov2_vits14 forward_features -> CLS + GeM(patches: clamp 1e-6, pow 4, avg 16x16, pow 1/4)
The 551 MB per-frame feature array z.f16.npy is not uploaded -- recompute z from your own frames
with the recipe above.
Action normalisation
action = absolute target pose [x,y,z, R00,R10,R20, R01,R11,R21] + commanded gripper width (10-D; the same layout as observation.state)
a_norm = 2 * (a - action_min) / (action_max - action_min) - 1 (element-wise, to [-1,1]); invert with a = (a_norm + 1) / 2 * (action_max - action_min) + action_min
dimension 9 is the commanded gripper WIDTH in metres; binarise with width < gripper_thresh => closed
{
"action_min": [
0.29995855689048767,
-0.4245767295360565,
0.09168203175067902,
0.17948679625988007,
-0.9667583703994751,
-0.8085525035858154,
-0.816544234752655,
-1.0330713987350464,
-1.0330268144607544,
0.0027000003028661013
],
"action_max": [
0.7266356945037842,
0.27291059494018555,
0.35973718762397766,
1.0160871744155884,
0.9834021925926208,
0.8849889636039734,
1.0159106254577637,
0.6880616545677185,
0.6857876777648926,
0.07029999792575836
],
"gripper_thresh": 0.052
}
(also in action_normalization.json in this repo.)
Provenance
Trained on the IBM Blue Vela cluster, project rsm_rise; W&B richardrl/rsm_rise.
Report: docs/real_proposal_report.pdf / status/real/report_numbers.json in the rsm_rise tree.