Hugging Face's logo

thanhkt
/
plan-bind-d2

Model card Files
xet
 Community
plan-bind-d2 / results.md
thanhkt's picture
thanhkt
Upload results.md with huggingface_hub
3edd221 verified
preview code
|
Raw History Blame Contribute Delete
7.25 kB

Plan-Bind D2 results

Source: /home/marinelab/code/plan-bind-d2/experiments/261028-plan-bind-d2-latency/outputs/full. Success and stage rates in percent; 50 seeded episodes per cell unless noted.

Inference time of one call (batch 1)

A call: preprocess, encoders on the observation frames, then the head (bsp10: DDIM denoise and spline decode; planbind: plan and bind, the slower call). Latency frames: the call's p95 rounded up to frames at 30 fps, the latency each policy is evaluated at below.

Bench Hardware Policy Call p50 (ms) Call p95 (ms) Bind-only p50 (ms) Latency frames select_action mean (ms) Stage p50 (ms)
cpu cpu x4 bsp10 590.0 628.9 - 19 27.1 preprocess 0.7, encode 58.0, denoise 528.4, decode 0.2
cpu cpu x4 onestep 125.9 130.1 - 4 32.0 preprocess 0.8, encode 60.2, onestep 64.5
cpu cpu x4 planbind 162.2 164.7 113.0 5 30.2 preprocess 1.0, encode 50.0, plan 49.0, bind 62.1
cuda NVIDIA GeForce RTX 5050 bsp10 169.7 170.2 - 6 8.4 preprocess 1.3, encode 4.2, denoise 163.9, decode 0.3
cuda NVIDIA GeForce RTX 5050 onestep 22.1 22.2 - 1 5.2 preprocess 1.3, encode 4.2, onestep 16.6
cuda NVIDIA GeForce RTX 5050 planbind 38.7 38.9 22.1 2 5.8 preprocess 1.3, encode 4.2, plan 16.5, bind 16.6
cuda_amp NVIDIA GeForce RTX 5050 bsp10 189.7 190.1 - 6 8.6 preprocess 1.3, encode 3.7, denoise 184.4, decode 0.3
cuda_amp NVIDIA GeForce RTX 5050 onestep 23.7 23.8 - 1 5.6 preprocess 1.3, encode 3.7, onestep 18.6
cuda_amp NVIDIA GeForce RTX 5050 planbind 42.4 42.5 23.8 2 6.2 preprocess 1.3, encode 3.7, plan 18.6, bind 18.7

Each policy at its measured latency

Bench Policy Latency (frames) Latency (ms) Success
cpu bsp10 19 633 22
cpu onestep 4 133 32
cpu planbind 5 167 100
cuda bsp10 6 200 52
cuda onestep 1 33 32
cuda planbind 2 67 94

Update rate without latency

Latency 0; a call every 4 frames is D1's rate. In D1 a call could not start before the previous one swapped in, so latency 0, 3, 6 and 9 also meant a call every 4, 4, 8 and 12 frames: these cells tell rate from latency apart.

Policy Commit Every 4 Every 8 Every 12
onestep latency 66 58 70
onestep none 36 50 62
planbind latency 92 100 98
planbind none 88 98 100

Commit modes

Policy Latency (ms) latency now none now vs latency
onestep 0 66 68 36 +2 (+1/-0)
onestep 200 78 82 76 +4 (+9/-7)
planbind 0 92 94 88 +2 (+1/-0)
planbind 200 100 100 100 +0 (+0/-0)

Plan-Bind replan threshold (latency 6, commit latency)

Threshold Success vs off
off 100 -
0.5 70 -30 (+0/-15)
1.0 100 +0 (+0/-0)

bsp10 through the spline runtime

classic is the action-queue policy (D1's bsp10_L0): it plans 32 frames and replans when they run out.

Mode Latency (ms) Commit Success vs classic
classic 0 - 100 -
runtime 0 latency 80 -20 (+0/-10)
runtime 0 none 58 -42 (+0/-21)
runtime 200 latency 52 -48 (+0/-24)
runtime 300 latency 36 -64 (+0/-32)
runtime 633 latency 22 -78 (+0/-39)

The bottom bowl moves when the middle bowl is lifted

Cell Shift Static Shifted Shifted vs static Middle nested (shifted)
bsp10_L0 shift3cm 100 90 -10 (+0/-5) 98
bsp10_L0_latency shift3cm 80 76 -4 (+4/-6) 96
bsp10_L6_latency shift3cm 52 34 -18 (+2/-11) 66
onestep_L0_latency shift3cm 66 56 -10 (+2/-7) 92
onestep_L6_latency shift3cm 78 62 -16 (+1/-9) 86
planbind_L0_latency shift3cm 92 74 -18 (+2/-11) 94
planbind_L6_latency shift3cm 100 84 -16 (+0/-8) 100

All cells

Cell Episodes Success 95% CI nested_middle nested_top picked_middle picked_top released_middle released_top
bsp10_L0 50 100 93-100 100 100 100 100 100 100
bsp10_L0_latency 50 80 67-89 90 84 96 88 90 82
bsp10_L0_none 50 58 44-71 92 62 92 64 92 60
bsp10_L6_latency 50 52 39-65 74 56 76 62 72 56
bsp10_L9_latency 50 36 24-50 68 42 80 46 68 40
bsp10_L19_latency 50 22 13-35 78 24 80 28 74 24
bsp10_L0_shift3cm 50 90 79-96 98 92 98 92 98 92
bsp10_L0_latency_shift3cm 50 76 63-86 96 84 100 86 92 82
bsp10_L6_latency_shift3cm 50 34 22-48 66 46 68 50 66 44
onestep_L0_latency 50 66 52-78 92 66 92 66 92 66
onestep_L0_latency_every8 50 58 44-71 84 60 88 64 84 60
onestep_L0_latency_every12 50 70 56-81 88 72 92 72 88 72
onestep_L0_none 50 36 24-50 70 38 72 38 70 36
onestep_L0_none_every8 50 50 37-63 78 50 80 52 78 50
onestep_L0_none_every12 50 62 48-74 88 62 88 64 88 62
onestep_L0_now 50 68 54-79 92 68 92 68 92 68
onestep_L1_latency 50 32 21-46 56 32 66 46 54 32
onestep_L4_latency 50 32 21-46 60 36 60 48 60 36
onestep_L6_latency 50 78 65-87 84 84 86 84 84 84
onestep_L6_none 50 76 63-86 98 78 98 80 98 78
onestep_L6_now 50 82 69-90 96 84 98 88 96 84
onestep_L0_latency_shift3cm 50 56 42-69 92 60 92 62 92 60
onestep_L6_latency_shift3cm 50 62 48-74 86 70 88 74 86 70
planbind_L0_latency 50 92 81-97 92 92 94 92 92 92
planbind_L0_latency_every8 50 100 93-100 100 100 100 100 100 100
planbind_L0_latency_every12 50 98 90-100 98 98 98 98 98 98
planbind_L0_none 50 88 76-94 90 88 90 88 88 88
planbind_L0_none_every8 50 98 90-100 100 98 100 100 100 98
planbind_L0_none_every12 50 100 93-100 100 100 100 100 100 100
planbind_L0_now 50 94 84-98 94 94 94 94 94 94
planbind_L2_latency 50 94 84-98 94 94 98 94 94 94
planbind_L5_latency 50 100 93-100 100 100 100 100 100 100
planbind_L6_latency 50 100 93-100 100 100 100 100 100 100
planbind_L6_latency_tau0.5 50 70 56-81 80 74 80 80 80 72
planbind_L6_latency_tau1.0 50 100 93-100 100 100 100 100 100 100
planbind_L6_none 50 100 93-100 100 100 100 100 100 100
planbind_L6_now 50 100 93-100 100 100 100 100 100 100
planbind_L0_latency_shift3cm 50 74 60-84 94 92 96 94 94 92
planbind_L6_latency_shift3cm 50 84 71-92 100 100 100 100 100 100