DavidSunok commited on
Commit
7f214fa
·
verified ·
1 Parent(s): c3d9af6

docs: synchronize corrected Unified E5 results

Browse files
INTACT-unified/AUDITED_RESULTS_20260914.json ADDED
@@ -0,0 +1,52 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "schema_version": 1,
3
+ "result_revision": "2026-09-14",
4
+ "criterion": "Official",
5
+ "protocol": {
6
+ "name": "corrected_causal_continuation",
7
+ "training_seeds": [0, 42, 3072],
8
+ "evaluation_seeds": [0, 1, 42],
9
+ "episodes_per_evaluation_seed": 100,
10
+ "history_boundary": "strictly pre-start actions only; append each policy-executed action exactly once"
11
+ },
12
+ "statistics": "mean and sample standard deviation across training-seed means",
13
+ "results_percent": [
14
+ {
15
+ "cell": "LeWM",
16
+ "inference": "CEM 300x30",
17
+ "pusht": {"mean": 74.56, "sample_std": 3.67},
18
+ "cube": {"mean": 67.33, "sample_std": 1.86},
19
+ "reacher": {"mean": 83.11, "sample_std": 0.96},
20
+ "tworoom": {"mean": 39.67, "sample_std": 8.39},
21
+ "macro": {"mean": 66.17, "sample_std": 2.67}
22
+ },
23
+ {
24
+ "cell": "Inverse only",
25
+ "inference": "Direct",
26
+ "pusht": {"mean": 36.11, "sample_std": 0.19},
27
+ "cube": {"mean": 67.56, "sample_std": 4.03},
28
+ "reacher": {"mean": 90.56, "sample_std": 1.35},
29
+ "tworoom": {"mean": 78.56, "sample_std": 6.68},
30
+ "macro": {"mean": 68.19, "sample_std": 0.67}
31
+ },
32
+ {
33
+ "cell": "Goal intent only",
34
+ "inference": "Direct",
35
+ "pusht": {"mean": 81.78, "sample_std": 0.96},
36
+ "cube": {"mean": 100.00, "sample_std": 0.00},
37
+ "reacher": {"mean": 88.67, "sample_std": 0.33},
38
+ "tworoom": {"mean": 69.22, "sample_std": 7.04},
39
+ "macro": {"mean": 84.92, "sample_std": 1.86}
40
+ },
41
+ {
42
+ "cell": "Goal-displacement INTACT",
43
+ "inference": "Direct",
44
+ "pusht": {"mean": 86.11, "sample_std": 0.96},
45
+ "cube": {"mean": 100.00, "sample_std": 0.00},
46
+ "reacher": {"mean": 97.22, "sample_std": 0.84},
47
+ "tworoom": {"mean": 81.56, "sample_std": 3.67},
48
+ "macro": {"mean": 91.22, "sample_std": 0.51}
49
+ }
50
+ ],
51
+ "manifest_note": "PAPER_E5_* manifests preserve checkpoint identities and hashes; this file is authoritative for corrected-evaluator success rates."
52
+ }
INTACT-unified/README.md CHANGED
@@ -9,29 +9,34 @@ PushT, Cube, Reacher, and TwoRoom, with task-specific lightweight heads.
9
  | Cell | Archive prefix | Native interface | Official macro SR |
10
  |---|---|---|---:|
11
  | LeWM | `paper-e5-lewm-*` | CEM 300x30 | 66.17 |
12
- | Inverse only | `paper-e5-inverse-only-*` | Direct | 56.67 |
13
- | Waypoint intent only | `paper-e5-waypoint-intent-only-*` | Direct | 74.22 |
14
- | Goal intent only | `paper-e5-goal-intent-only-*` | Direct | 80.61 |
15
- | Waypoint INTACT | `paper-e5-waypoint-intact-*` | Direct | 76.42 |
16
- | **Goal-displacement INTACT** | `intact-goal-e5-*` | **Direct** | **89.39** |
17
 
18
  The Goal-displacement INTACT task-level result is:
19
 
20
  | PushT | Cube | Reacher | TwoRoom | Macro SR |
21
  |---:|---:|---:|---:|---:|
22
- | 80.22 +/- 1.26 | 99.56 +/- 0.77 | 95.67 +/- 1.76 | 82.11 +/- 4.11 | **89.39 +/- 0.77** |
23
 
24
  Success rates are percentages averaged over training seeds `0/42/3072`.
25
  Each checkpoint is evaluated on 100 episodes for evaluation seeds `0/1/42`;
26
  `+/-` is the sample standard deviation across training-seed means.
27
 
 
 
 
 
 
28
  ## Contents
29
 
30
- - Six controlled cells and three training seeds.
 
31
  - Four task-specific checkpoint shards in each archive.
32
  - `6 x 3 x 4 = 72` checkpoints in total.
33
- - `PAPER_E5_MATRIX_MANIFEST.json`: per-task results and extracted shard hashes.
34
- - `PAPER_E5_GOAL_MANIFEST.json`: Goal-displacement INTACT metadata and results.
 
35
  - `SHA256SUMS`: archive integrity records.
36
  - `RUNTIME_SHA256SUMS`: frozen evaluator runtime fingerprint.
37
 
 
9
  | Cell | Archive prefix | Native interface | Official macro SR |
10
  |---|---|---|---:|
11
  | LeWM | `paper-e5-lewm-*` | CEM 300x30 | 66.17 |
12
+ | Inverse only | `paper-e5-inverse-only-*` | Direct | 68.19 |
13
+ | Goal intent only | `paper-e5-goal-intent-only-*` | Direct | 84.92 |
14
+ | **Goal-displacement INTACT** | `intact-goal-e5-*` | **Direct** | **91.22** |
 
 
15
 
16
  The Goal-displacement INTACT task-level result is:
17
 
18
  | PushT | Cube | Reacher | TwoRoom | Macro SR |
19
  |---:|---:|---:|---:|---:|
20
+ | 86.11 +/- 0.96 | 100.00 +/- 0.00 | 97.22 +/- 0.84 | 81.56 +/- 3.67 | **91.22 +/- 0.51** |
21
 
22
  Success rates are percentages averaged over training seeds `0/42/3072`.
23
  Each checkpoint is evaluated on 100 episodes for evaluation seeds `0/1/42`;
24
  `+/-` is the sample standard deviation across training-seed means.
25
 
26
+ These values use the corrected causal-continuation evaluator: the initial
27
+ actor history contains only strictly pre-start actions, and each action
28
+ executed by the evaluated policy is appended exactly once. The current values
29
+ are recorded in `AUDITED_RESULTS_20260914.json`.
30
+
31
  ## Contents
32
 
33
+ - Six archived training cells and three training seeds. The current comparison
34
+ above reports the four goal-displacement cells used by the revised analysis.
35
  - Four task-specific checkpoint shards in each archive.
36
  - `6 x 3 x 4 = 72` checkpoints in total.
37
+ - `AUDITED_RESULTS_20260914.json`: authoritative corrected-evaluator SR values.
38
+ - `PAPER_E5_MATRIX_MANIFEST.json`: checkpoint inventory and extracted shard hashes.
39
+ - `PAPER_E5_GOAL_MANIFEST.json`: Goal-displacement INTACT checkpoint metadata.
40
  - `SHA256SUMS`: archive integrity records.
41
  - `RUNTIME_SHA256SUMS`: frozen evaluator runtime fingerprint.
42
 
README.md CHANGED
@@ -62,8 +62,8 @@ shared-encoder comparison matrix.
62
  | Goal intent only | Direct | 84.92 |
63
  | **Goal-displacement INTACT** | **Direct** | **91.22** |
64
 
65
- All cells use the same E5 shared-encoder protocol. Task-level results,
66
- per-seed statistics, checkpoint hashes, and runtime fingerprints are provided
67
  inside [`INTACT-unified`](https://huggingface.co/INTACT-JEPA/INTACT/tree/main/INTACT-unified).
68
  The immutable original layout also remains available at the frozen
69
  [`paper-e5-goal-v1`](https://huggingface.co/INTACT-JEPA/INTACT/tree/paper-e5-goal-v1)
 
62
  | Goal intent only | Direct | 84.92 |
63
  | **Goal-displacement INTACT** | **Direct** | **91.22** |
64
 
65
+ All cells use the same E5 shared-encoder protocol. Task-level statistics,
66
+ checkpoint hashes, and runtime fingerprints are provided
67
  inside [`INTACT-unified`](https://huggingface.co/INTACT-JEPA/INTACT/tree/main/INTACT-unified).
68
  The immutable original layout also remains available at the frozen
69
  [`paper-e5-goal-v1`](https://huggingface.co/INTACT-JEPA/INTACT/tree/paper-e5-goal-v1)