senku21x commited on
Commit
498b9c9
·
verified ·
1 Parent(s): 5fca1dd

Dory forward pass, J-lens / R-lens / J++ code, fit records

Browse files
Files changed (37) hide show
  1. README.md +188 -0
  2. as_run/bootstrap.py +71 -0
  3. as_run/fit.py +178 -0
  4. as_run/fit_jpp.py +232 -0
  5. as_run/readout_example.py +34 -0
  6. as_run/test_jlens_estimator.py +64 -0
  7. as_run/test_lrp_rules.py +34 -0
  8. dory/__init__.py +0 -0
  9. dory/chain.py +62 -0
  10. dory/jlens.py +166 -0
  11. dory/jpp.py +64 -0
  12. dory/lens_eval.py +55 -0
  13. dory/model.py +283 -0
  14. fit_records/jpp_k1_n64_e8_s0_r2/bootstrap.json +59 -0
  15. fit_records/jpp_k1_n64_e8_s0_r2/metrics.json +1773 -0
  16. fit_records/jpp_k1_n64_e8_s0_r2/prompts.json +258 -0
  17. fit_records/jpp_k1_n64_e8_s0_r2/readouts.txt +80 -0
  18. fit_records/jpp_k2_n64_e8_s0/bootstrap.json +59 -0
  19. fit_records/jpp_k2_n64_e8_s0/metrics.json +2643 -0
  20. fit_records/jpp_k2_n64_e8_s0/prompts.json +258 -0
  21. fit_records/jpp_k2_n64_e8_s0/readouts.txt +80 -0
  22. fit_records/k1_n50_skip16_s0/metrics.json +2646 -0
  23. fit_records/k1_n50_skip16_s0/prompts.json +202 -0
  24. fit_records/k1_n50_skip16_s0/readouts.txt +98 -0
  25. fit_records/k1_n50_skip16_s0/run.json +25 -0
  26. fit_records/k2_n50_skip16_s0/metrics.json +4122 -0
  27. fit_records/k2_n50_skip16_s0/prompts.json +202 -0
  28. fit_records/k2_n50_skip16_s0/readouts.txt +142 -0
  29. fit_records/k2_n50_skip16_s0/run.json +25 -0
  30. fit_records/k3_n50_skip16_s0/metrics.json +0 -0
  31. fit_records/k3_n50_skip16_s0/prompts.json +202 -0
  32. fit_records/k3_n50_skip16_s0/readouts.txt +186 -0
  33. fit_records/k3_n50_skip16_s0/run.json +25 -0
  34. fit_records/k4_n50_skip16_s0/metrics.json +0 -0
  35. fit_records/k4_n50_skip16_s0/prompts.json +202 -0
  36. fit_records/k4_n50_skip16_s0/readouts.txt +230 -0
  37. fit_records/k4_n50_skip16_s0/run.json +25 -0
README.md ADDED
@@ -0,0 +1,188 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: other
3
+ base_model: nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16
4
+ tags: [interpretability, jacobian-lens, dory]
5
+ ---
6
+
7
+ # Dory implementation: forward pass and Jacobian lens
8
+
9
+ A plain-PyTorch forward pass of NVIDIA's looped language model
10
+ [Dory1.0-4B](https://huggingface.co/nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16) (arXiv 2611.03998) with a hook at
11
+ every sublayer and loop boundary, and a plain Jacobian lens (J-lens) fitted on it.
12
+
13
+ **Status, 2026-11-16: first fits, exploratory.** The estimator is verified. Plain J-lenses are fitted for 1, 2, 3
14
+ and 4 loops, and R-lenses and J++ lenses for 1 and 2 loops; all lens files are under `lenses/`. Read "Which lens to
15
+ use" and "What the lens is like" before relying on a readout. A copy of the code and records, without the lens
16
+ files, is in `conditional-model-diffing/dory_implementation`.
17
+
18
+ ## Files
19
+
20
+ | Path | What it is |
21
+ |---|---|
22
+ | `dory/model.py` | The model: loads the official checkpoint, runs any number of loops, exposes every residual state through a `tap(name, h)` callback. Also `load_tokenizer`. |
23
+ | `dory/jlens.py` | The J-lens: the per-prompt Jacobian estimator, the `JLens` class (load, transport, readout) and a logit-lens baseline. |
24
+ | `dory/jpp.py`, `dory/lens_eval.py` | J++ pieces (activation router, expert weights) and the readout evaluation loader. The LRP backward rules are in `dory/model.py` (`with lrp_rules(): ...`). |
25
+ | `dory/chain.py` | A model-driven roll-out tried as a chain-lens analogue. It did not work; kept for the record. |
26
+ | `lenses/jlens_k{1,2,3,4}.safetensors` | Plain J-lenses for 1 to 4 loops, float32 matrices of 2560 x 2560 (1.9 to 5.1 GB). |
27
+ | `lenses/rlens_k{1,2}.safetensors`, `lenses/jpp_k{1,2}.safetensors` | R-lenses and J++ lenses for 1 and 2 loops (1.9 and 3.0 GB each). J++ is the default. |
28
+ | `lenses/router_and_weights_k{1,2}.pt` | The J++ activation router and expert weights (torch pickle). |
29
+ | `fit_records/<run>/` | For each fit: `metrics.json` (per-node noise and early-position numbers), `readouts.txt` (tables and example readouts), `prompts.json`, `run.json`. |
30
+ | `as_run/` | The fitting script, the readout example and the estimator test exactly as they were run. Their import paths are those of the fitting project, not of this repo. |
31
+
32
+ ## Use
33
+
34
+ ```python
35
+ import sys, torch
36
+ from huggingface_hub import snapshot_download, hf_hub_download
37
+ repo = snapshot_download("senku21x/dory_implementation", allow_patterns=["dory/*"])
38
+ sys.path.insert(0, repo)
39
+ from dory.model import Dory, load_tokenizer
40
+ from dory.jlens import JLens, logit_lens, node_key
41
+
42
+ snap = snapshot_download("nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16")
43
+ tok = load_tokenizer(snap) # AutoTokenizer fails: transformers has no `dory` config class yet
44
+ model = Dory.from_snapshot(snap, device="cuda", dtype=torch.float32)
45
+ lens = JLens.load(hf_hub_download("senku21x/dory_implementation", "lenses/jpp_k2.safetensors"), device="cuda")
46
+
47
+ ids = tok("The capital of the country where sushi originated is", return_tensors="pt").input_ids.cuda()
48
+ states = {}
49
+ with torch.no_grad():
50
+ final = model.forward(ids, n_loops=lens.k, tap=lambda name, h: states.__setitem__(name, h))
51
+ for node in lens.nodes():
52
+ logits = lens.logits(model, states[node][0, -1], node) # unembed(l2norm(J @ h))
53
+ print(node_key(node), [tok.decode([t]) for t in logits.topk(5).indices.tolist()])
54
+ ```
55
+
56
+ A lens is fitted for one loop count; run the model with `n_loops = lens.k`.
57
+
58
+ ## The model, and what a "layer" is
59
+
60
+ Dory's checkpoint counts sublayers: 72 of them, alternating attention and MLP. Sublayers 0 to 15 are the input
61
+ block (run once), 16 to 55 the recurrent block (run once per loop, weights shared), 56 to 71 the output block (run
62
+ once). With `x` the input block's output and `N` the L2 norm, loop `t` computes `u_t = N(s_{t-1} + x)` and
63
+ `s_t = N(F(u_t))`, starting from `s_0 = x`. Every state is a unit vector.
64
+
65
+ Lens source nodes, in forward order. Every sublayer output has exactly one node.
66
+
67
+ | Node | State |
68
+ |---|---|
69
+ | `emb` | token embedding |
70
+ | `in.00` ... `in.14` | after input sublayers 0 to 14 |
71
+ | `x` | after sublayer 15, the input block's output |
72
+ | `u.t` | input to loop `t` |
73
+ | `rec.t.16` ... `rec.t.54` | after recurrent sublayers 16 to 54 in loop `t` |
74
+ | `s.t` | after sublayer 55 in loop `t`, the loop's output |
75
+ | `out.56` ... `out.70` | after output sublayers 56 to 70 |
76
+
77
+ The target is the final state (after sublayer 71), which the LM head reads. In `model.forward` the tap names are
78
+ tuples, for example `("rec", 2, 41)`; `node_key` turns them into the strings above.
79
+
80
+ ## How the lens was fitted
81
+
82
+ The estimator of Gurnee et al. 2026 ("Verbalizable Representations Form a Global Workspace in Language Models"):
83
+ for each source node, the Jacobian of the final state with respect to the node's state, summed over target
84
+ positions at or after the source position, averaged over source positions and then over prompts. One backward pass
85
+ per output dimension (2,560 per prompt, batched).
86
+
87
+ - 50 prompts: the first 128 tokens of 50 WikiText-103 paragraphs (validation split, seed 0). No BOS token; the
88
+ tokenizer adds none.
89
+ - The first 16 source positions are left out of the average.
90
+ - float32 weights with TF32 matrix multiplies. 39 seconds per prompt at 2 loops on one H200.
91
+ - Each lens file also holds the mean state of every node over the fitting positions (`lens.mean_state`), for
92
+ centring experiments.
93
+
94
+ ## Checks
95
+
96
+ - **Estimator.** On the real model, the autograd Jacobian sums match central finite differences at 10 nodes and
97
+ all three position ranges (cosine 0.9998 or higher, norm ratio within 1%). TF32 against exact float32 on the same
98
+ rows: cosine at least 0.9999, relative error at most 1.4%.
99
+ - **Forward pass.** All 507 checkpoint tensors load. Greedy text is coherent at 1, 2, 4 and 8 loops; loss on
100
+ ordinary prose is about 2.0 nats per token (7.8 with the tokens shuffled); the output block applied to a loop-end
101
+ state reproduces the model run with that many loops exactly. The code was read step by step against the authors'
102
+ transformers pull request (50099) and matches it.
103
+ - **Not done.** No number-for-number comparison with the authors' implementation. The sliding-window attention
104
+ (1,024 tokens) is exercised on a long document but not validated.
105
+
106
+ ## Which lens to use
107
+
108
+ Median split-half cosine by block, for each fitted lens (1 means the lens fitted on the 25 even-numbered prompts
109
+ agrees with the one fitted on the 25 odd-numbered prompts):
110
+
111
+ | File | Loops | Matrices | Size | Input block | Loop 1 | Loop 2 | Loop 3 | Loop 4 | Output block |
112
+ |---|---|---|---|---|---|---|---|---|---|
113
+ | `jlens_k1.safetensors` | 1 | 73 | 1.9 GB | 0.80 | 0.97 | | | | 1.00 |
114
+ | `jlens_k2.safetensors` | 2 | 114 | 3.0 GB | 0.52 | 0.29 | 0.97 | | | 1.00 |
115
+ | `jlens_k3.safetensors` | 3 | 155 | 4.1 GB | 0.03 | 0.01 | 0.65 | 0.97 | | 1.00 |
116
+ | `jlens_k4.safetensors` | 4 | 196 | 5.1 GB | 0.09 | -0.00 | 0.12 | 0.75 | 0.97 | 1.00 |
117
+
118
+ - **A lens aimed at the final state reaches back about one loop.** Each lens is stable for its last loop and the
119
+ output block, and noise before that.
120
+ - **So read loop `t` through the lens fitted at `t` loops.** The states of loops 1 to `t` in a longer run are the
121
+ same as in a `t`-loop run, and Dory is trained so that any loop end can go straight to the output block. On 17
122
+ hand-written two-hop prompts (4-loop run, punctuation-like tokens dropped), the intermediate reached the top 10
123
+ at some node of loops 1 / 2 / 3 / 4 on 41 / 47 / 47 / 47% of prompts read this way, against 0 / 0 / 24 / 47%
124
+ through the 4-loop lens alone. Exploratory: small hand-made set, best rank over 41 nodes per loop.
125
+ - **The per-loop lenses are almost the same matrix.** At the same sublayer, the loop-`t` matrix of the `t`-loop
126
+ lens has cosine 0.995 or more with the loop-1 matrix of the 1-loop lens (0.98 at the first recurrent sublayer),
127
+ and the output-block matrices agree to 1.000. In practice `jlens_k1.safetensors` alone serves every loop: use its
128
+ `rec.1.i` matrix for `rec.t.i`, its `s.1` for `s.t`, and its `out.i` for the output block.
129
+ - **The states are almost the same too.** At the same sublayer, consecutive loops' states have cosine 0.95 to 0.99
130
+ between loops 1 and 2 and 0.999 after that (3 prompts), so the readouts of different loops are nearly identical.
131
+ Whatever a later loop changes is in a very small part of the state that a readout of the whole state does not
132
+ isolate.
133
+
134
+ ## What the lens is like (2 loops, 50 prompts)
135
+
136
+ Numbers are from `fit_records/k2_n50_skip16_s0/`; held-out figures are from 8 paragraphs not used in the fit.
137
+
138
+ | Block | Nodes | Split-half cosine (median) | Top-1 agreement with the model, J-lens | Logit lens | Punctuation-like share of top 10 |
139
+ |---|---|---|---|---|---|
140
+ | Input block | 17 | 0.52 | 0.00 | 0.00 | 0.66 |
141
+ | Loop 1 | 41 | 0.29 | 0.01 | 0.00 | 0.30 |
142
+ | Loop 2 | 41 | 0.97 | 0.04 | 0.00 | 0.21 |
143
+ | Output block | 15 | 1.00 | 0.41 | 0.29 | 0.10 |
144
+
145
+ "Split-half cosine" compares the lens fitted on the 25 even-numbered prompts with the one fitted on the 25
146
+ odd-numbered prompts; 1 means the two agree.
147
+
148
+ - **The input block and loop 1 are not converged.** Their matrices are mostly prompt-to-prompt noise at 50 prompts
149
+ (split-half 0.02 to 0.6; it passes 0.8 only from `rec.1.51`). Treat readouts there as unreliable. Loop 2 and the
150
+ output block are stable.
151
+ - **Readouts are dominated by punctuation and formatting tokens** until the output block. On four two-hop prompts
152
+ the unspoken intermediate never reached the top 10 before the output block (best ranks in loop 2: 30 to 373). The
153
+ model itself did not answer two of the four directly, so this is not a clean test.
154
+ - **The logit lens fails at loop boundaries**: `s.t` and `u.t` are close to one fixed vector, and the unembedding
155
+ reads the same few tokens whatever the prompt.
156
+ - **Early positions.** No sign of a first-four-tokens problem. Every state in Dory is a unit vector, so there are no
157
+ high-norm tokens. Per-position Jacobians are about 3.7 times larger at position 0 and about 2 times larger at
158
+ positions 1 to 15 than later (partly because earlier positions have more target positions to sum over). Leaving
159
+ out 0, 4 or 16 positions gives nearly the same loop-2 and output-block matrices (cosine 0.99 or higher) and
160
+ moderately different input-block ones (0.92).
161
+
162
+ ## R-lens and J++ (1 and 2 loops, 64 prompts)
163
+
164
+ Recall@10 of the labelled intermediate on a held-out half of the Gurnee et al. readout sets (multihop,
165
+ multilingual, typo, association, poetry; 181 test items), at any of seven nodes equally spaced over the unrolled
166
+ depth, macro over tasks, punctuation-like tokens dropped. Brackets: 95% bootstrap interval over test items.
167
+
168
+ | Lens | 1 loop | 2 loops |
169
+ |---|---|---|
170
+ | Logit lens | 0.244 [0.18, 0.31] | 0.189 [0.14, 0.25] |
171
+ | Plain J-lens | 0.319 [0.26, 0.39] | 0.268 [0.22, 0.33] |
172
+ | R-lens | 0.396 [0.33, 0.46] | 0.321 [0.26, 0.39] |
173
+ | J++ | 0.465 [0.40, 0.53] | 0.423 [0.36, 0.49] |
174
+ | J++, weights learned on shuffled labels | 0.238 | 0.225 |
175
+
176
+ - J++ minus R-lens: +0.069 [+0.03, +0.12] at 1 loop, +0.102 [+0.02, +0.18] at 2 loops.
177
+ - With the LRP rules the lens is stable in every block (split-half cosine 0.99), including loop 1 of a 2-loop run.
178
+ - Limits: the J++ weights were learned on the other half of the same five tasks; no swap or ablation test has
179
+ been run; multihop has 10 or 11 test items because Dory answers about a quarter of them; one seed, one router,
180
+ one split.
181
+ - LRP choices specific to Dory (not covered by the R-lens or J++ posts): every L2 norm's denominator is constant
182
+ except the query and key norms, and the attention output gate is treated as a constant.
183
+ - Records: `fit_records/jpp_k1_n64_e8_s0_r2/` and `fit_records/jpp_k2_n64_e8_s0/`. Lens files: `lenses/jpp_k<k>.safetensors` and `lenses/rlens_k<k>.safetensors`.
184
+
185
+ ## Not here yet
186
+
187
+ A centred readout, causal tests of the lens directions, and a multi-token readout (Dory has no multi-token
188
+ prediction head, so the chain lens does not apply directly).
as_run/bootstrap.py ADDED
@@ -0,0 +1,71 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Paired bootstrap over test items for the lens comparison of fit_jpp.py, recomputed from the saved lens files.
2
+
3
+ python bootstrap.py --k 2 --lens-dir <folder with rlens_k2.safetensors and jpp_k2.safetensors> --run k2_n64_e8_s0
4
+
5
+ Rebuilds the same dev/test split (same seed and filter), scores each lens on the test half at the same seven
6
+ nodes, and resamples test items within each task 2,000 times. The interval covers which test items were drawn;
7
+ it does not cover the noise of the fit or the choice of split.
8
+ """
9
+ import argparse, json, random, sys, pathlib, torch
10
+ HERE = pathlib.Path(__file__).resolve().parent; PROJ = HERE.parents[1]
11
+ sys.path.insert(0, str(PROJ / "src"))
12
+ from dory.model import Dory, snapshot_path, load_tokenizer, justnorm
13
+ from dory.jlens import JLens, source_nodes, node_key
14
+ from dory.lens_eval import load_items, semantic_mask, TASKS
15
+ ap = argparse.ArgumentParser(); ap.add_argument("--k", type=int, required=True); ap.add_argument("--lens-dir", required=True)
16
+ ap.add_argument("--run", required=True); ap.add_argument("--seed", type=int, default=0); args = ap.parse_args()
17
+ torch.backends.cuda.matmul.allow_tf32 = True
18
+ snap = snapshot_path(); tok = load_tokenizer(snap); model = Dory.from_snapshot(snap, device="cuda", dtype=torch.float32)
19
+ K, V = args.k, model.cfg["vocab_size"]; nodes = source_nodes(model, K); keys = [node_key(v) for v in nodes]; N = len(nodes)
20
+ items = load_items(tok, PROJ / "evals" / "lens_eval"); mask = semantic_mask(tok, V).cuda()
21
+ sub = [v for v in nodes if v[0] in ("in", "x", "rec", "s", "out")]
22
+ seven = [nodes.index(sub[i]) for i in sorted({round(f * (len(sub) + 1)) for f in torch.linspace(0.125, 0.875, 7).tolist()})]
23
+ H = torch.zeros(len(items), len(seven), model.cfg["hidden_size"], device="cuda"); correct = []
24
+ for i, it in enumerate(items):
25
+ st = {}
26
+ with torch.no_grad():
27
+ final = model.forward(it["ids"].cuda(), n_loops=K, tap=lambda nm, h: st.__setitem__(nm, h))
28
+ H[i] = torch.stack([st[nodes[j]][0, it["pos"]] for j in seven])
29
+ top = tok.decode([model.unembed(final)[0, -1].argmax().item()]).strip()
30
+ correct.append(it["answer"] is None or (len(top) > 0 and it["answer"].lower().startswith(top.lower())))
31
+ keep = [i for i, it in enumerate(items) if it["task"] not in ("multihop", "multilingual") or correct[i]]
32
+ rng = random.Random(args.seed); test = []
33
+ for task in TASKS:
34
+ idx = [i for i in keep if items[i]["task"] == task]; rng.shuffle(idx); test += idx[len(idx) // 2 :]
35
+ D = pathlib.Path(args.lens_dir)
36
+ files = {"J-lens": pathlib.Path.home() / f"interp/artifacts/dory_lens/E001_jlens_fit/k{K}_n50_skip16_s0/jlens_k{K}.safetensors",
37
+ "R-lens": D / f"rlens_k{K}.safetensors", "J++": D / f"jpp_k{K}.safetensors"}
38
+ score = {} # lens -> item -> fraction of intermediates recovered at 10
39
+ def item_scores(J):
40
+ out = {}
41
+ best = {i: [10**9] * len(items[i]["targets"]) for i in test}
42
+ for c, j in enumerate(seven):
43
+ h = H[test, c]
44
+ lg = model.unembed(justnorm(h if J is None else h @ J[keys[j]].cuda().T)).masked_fill(mask, -1e9)
45
+ for r, i in enumerate(test):
46
+ for q, tv in enumerate(items[i]["targets"]):
47
+ best[i][q] = min(best[i][q], min(int((lg[r] > lg[r, t]).sum()) + 1 for t in tv))
48
+ return {i: sum(b <= 10 for b in best[i]) / len(best[i]) for i in test}
49
+ score["logit lens"] = item_scores(None)
50
+ for name, f in files.items():
51
+ lens = JLens.load(f, device="cpu"); score[name] = item_scores(lens.J); del lens
52
+ by_task = {t: [i for i in test if items[i]["task"] == t] for t in TASKS}
53
+ macro = lambda s, pick: sum(sum(s[i] for i in pick[t]) / len(pick[t]) for t in TASKS) / len(TASKS)
54
+ point = {n: macro(s, by_task) for n, s in score.items()}
55
+ g = random.Random(1); B = 2000; draws = {n: [] for n in score}
56
+ for _ in range(B):
57
+ pick = {t: [g.choice(by_task[t]) for _ in by_task[t]] for t in TASKS}
58
+ for n, s in score.items():
59
+ draws[n].append(macro(s, pick))
60
+ def ci(x):
61
+ x = sorted(x); return x[int(0.025 * B)], x[int(0.975 * B)]
62
+ res = {"k": K, "n_test_items": len(test), "items_per_task": {t: len(v) for t, v in by_task.items()}, "recall10_seven": {}, "differences": {}}
63
+ print(f"k={K}: {len(test)} test items {res['items_per_task']}; recall@10 at any of seven nodes, macro over tasks, 95% bootstrap interval")
64
+ for n in score:
65
+ lo, hi = ci(draws[n]); res["recall10_seven"][n] = {"point": point[n], "lo": lo, "hi": hi}
66
+ print(f" {n:12s} {point[n]:.3f} [{lo:.3f}, {hi:.3f}]")
67
+ for a, b in (("J++", "R-lens"), ("R-lens", "J-lens"), ("J++", "J-lens"), ("J-lens", "logit lens")):
68
+ dd = [x - y for x, y in zip(draws[a], draws[b])]; lo, hi = ci(dd)
69
+ res["differences"][f"{a} minus {b}"] = {"point": point[a] - point[b], "lo": lo, "hi": hi, "share_of_resamples_above_zero": sum(x > 0 for x in dd) / B}
70
+ print(f" {a} minus {b}: {point[a] - point[b]:+.3f} [{lo:+.3f}, {hi:+.3f}] above zero in {sum(x > 0 for x in dd) / B:.3f} of resamples")
71
+ (HERE / "results" / args.run / "bootstrap.json").write_text(json.dumps(res, indent=1))
as_run/fit.py ADDED
@@ -0,0 +1,178 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Fit the plain Jacobian lens for Dory at one loop count.
2
+
3
+ python fit.py --k 2 --n-prompts 50 --skip 16 --run k2_n50_skip16_s0
4
+ python fit.py --k 2 --n-prompts 8 --pilot --run pilot_k2_n8_s0 # early-position question, saves no lens
5
+
6
+ Source positions are tallied in three bins, [0,4), [4,16) and [16,T), and odd and even prompts separately, so
7
+ one run gives the lens with the first 0, 4 or 16 positions skipped and a split-half noise estimate.
8
+ Small outputs go to results/<run>/ (run.json, metrics.json, prompts.json, readouts.txt); the lens file goes to
9
+ ~/interp/artifacts/dory_lens/E001_jlens_fit/<run>/.
10
+ """
11
+ import argparse, json, os, sys, time, pathlib, random
12
+ import torch
13
+
14
+ HERE = pathlib.Path(__file__).resolve().parent
15
+ sys.path.insert(0, str(HERE.parents[1] / "src")); sys.path.insert(0, os.path.expanduser("~/interp/common"))
16
+ from runmeta import start_run, finish_run, artifacts_dir
17
+ from dory.model import Dory, snapshot_path, load_tokenizer, justnorm
18
+ from dory.jlens import JLens, source_nodes, node_key, prompt_jacobians, BINS, logit_lens
19
+
20
+ ap = argparse.ArgumentParser()
21
+ ap.add_argument("--k", type=int, required=True); ap.add_argument("--n-prompts", type=int, default=50)
22
+ ap.add_argument("--seq-len", type=int, default=128); ap.add_argument("--chunk", type=int, default=256)
23
+ ap.add_argument("--skip", type=int, default=16, choices=[0, 4, 16]); ap.add_argument("--seed", type=int, default=0)
24
+ ap.add_argument("--run", required=True); ap.add_argument("--pilot", action="store_true")
25
+ args = ap.parse_args()
26
+
27
+ WIKI = "/workspace/huggingface/hub/datasets--Salesforce--wikitext/snapshots/b08601e04326c79dfdd32d625aee71d232d685c3/wikitext-103-raw-v1/validation-00000-of-00001.parquet"
28
+ out = start_run(HERE / "results" / args.run, config=vars(args) | {"corpus": WIKI, "tf32": True, "dtype": "float32"})
29
+ (out / "pid").write_text(str(os.getpid()))
30
+ torch.backends.cuda.matmul.allow_tf32 = True; torch.backends.cudnn.allow_tf32 = True
31
+ torch.manual_seed(args.seed)
32
+
33
+ snap = snapshot_path(); tok = load_tokenizer(snap)
34
+ model = Dory.from_snapshot(snap, device="cuda", dtype=torch.float32)
35
+ d, T, K = model.cfg["hidden_size"], args.seq_len, args.k
36
+
37
+ # ---- corpus: WikiText-103 paragraphs of at least T tokens, first T tokens of each ----
38
+ import pyarrow.parquet as pq
39
+ rows = [t for t in pq.read_table(WIKI).column("text").to_pylist() if len(t) > 700 and not t.lstrip().startswith("=")]
40
+ random.Random(args.seed).shuffle(rows)
41
+ pool = []
42
+ for t in rows:
43
+ ids = tok(t.strip(), return_tensors="pt").input_ids
44
+ if ids.shape[1] >= T:
45
+ pool.append(ids[:, :T])
46
+ if len(pool) >= 660:
47
+ break
48
+ assert len(pool) >= 660, len(pool)
49
+ fit_ids, held_ids = pool[: args.n_prompts], pool[650:658] # held-out prompts never overlap the fit set (n_prompts <= 600)
50
+ (out / "prompts.json").write_text(json.dumps([{"i": i, "start": tok.decode(x[0, :24])} for i, x in enumerate(fit_ids)], indent=1))
51
+ print(f"corpus: {len(pool)} paragraphs of >= {T} tokens; fitting on {len(fit_ids)}", flush=True)
52
+
53
+ nodes = source_nodes(model, K); N = len(nodes)
54
+ acc = torch.zeros(2, N, len(BINS), d, d, device="cuda") # [half, node, bin, out dim, in dim]
55
+ sq = torch.zeros(N, T, device="cuda"); hsum = torch.zeros(N, len(BINS), d, device="cuda")
56
+ t0 = time.time()
57
+ for p, ids in enumerate(fit_ids):
58
+ j, s, h = prompt_jacobians(model, ids.cuda(), K, nodes, chunk=args.chunk)
59
+ acc[p % 2] += j; sq += s; hsum += h
60
+ del j
61
+ el = time.time() - t0
62
+ print(f"prompt {p + 1}/{len(fit_ids)} {el:.0f}s elapsed, {el / (p + 1):.1f}s per prompt, peak GPU {torch.cuda.max_memory_allocated() / 2**30:.0f} GiB", flush=True)
63
+
64
+ n = len(fit_ids); counts = [4, 12, T - 16]
65
+ def lens_for(skip, halves=(0, 1)):
66
+ b0 = {0: 0, 4: 1, 16: 2}[skip]
67
+ nh = sum((n + 1 - hf) // 2 for hf in halves) # prompts in the chosen halves
68
+ return acc[list(halves)][:, :, b0:].sum(dim=(0, 2)) / (nh * sum(counts[b0:]))
69
+ cosf = lambda a, b: torch.nn.functional.cosine_similarity(a.flatten(1), b.flatten(1), dim=1)
70
+
71
+ J = {s: lens_for(s) for s in (0, 4, 16)}
72
+ split_half = cosf(lens_for(args.skip, (0,)), lens_for(args.skip, (1,)))
73
+ pos_norm = (sq / n).sqrt() # [N, T] rms-over-prompts Frobenius norm per source position
74
+ late = pos_norm[:, 16:].median(dim=1).values
75
+ metrics = {
76
+ "n_prompts": n, "n_nodes": N, "seconds_per_prompt": (time.time() - t0) / n,
77
+ "split_half_cosine": {node_key(v): round(split_half[i].item(), 4) for i, v in enumerate(nodes)},
78
+ "cosine_skip0_vs_skip16": {node_key(v): round(c.item(), 4) for v, c in zip(nodes, cosf(J[0], J[16]))},
79
+ "cosine_skip4_vs_skip16": {node_key(v): round(c.item(), 4) for v, c in zip(nodes, cosf(J[4], J[16]))},
80
+ "frobenius_norm_skip16": {node_key(v): round(J[16][i].norm().item(), 4) for i, v in enumerate(nodes)},
81
+ "position_norm_over_late_median": {node_key(v): [round(x, 3) for x in (pos_norm[i, :20] / late[i]).tolist()] for i, v in enumerate(nodes)},
82
+ }
83
+
84
+ lines = []
85
+ def say(s=""):
86
+ print(s, flush=True); lines.append(s)
87
+
88
+ # ---- readouts on held-out prompts, for each skip variant ----
89
+ def states(ids):
90
+ st = {}
91
+ with torch.no_grad():
92
+ final = model.forward(ids, n_loops=K, tap=lambda nm, h: st.__setitem__(nm, h))
93
+ return st, final
94
+ def readout(Jm, h):
95
+ return model.unembed(justnorm(h.float() @ Jm.T))
96
+ nonsem = torch.tensor([not any(c.isalnum() for c in tok.decode([i])) for i in range(model.cfg["vocab_size"])], device="cuda")
97
+ say(f"non-semantic tokens (no letter or digit): {int(nonsem.sum())} of {len(nonsem)}")
98
+
99
+ held = torch.cat(held_ids).cuda()
100
+ agree = {s: torch.zeros(N) for s in (0, 4, 16)}; agree_ll = torch.zeros(N); junk = {s: torch.zeros(N) for s in (0, 4, 16)}
101
+ jac = {s: torch.zeros(N) for s in (0, 4)}
102
+ for ids in held.split(1):
103
+ st, final = states(ids)
104
+ top_model = model.unembed(final)[0, 16:].argmax(-1)
105
+ for i, v in enumerate(nodes):
106
+ h = st[v][0, 16:]
107
+ tops = {}
108
+ for s in (0, 4, 16):
109
+ lg = readout(J[s][i], h)
110
+ agree[s][i] += (lg.argmax(-1) == top_model).float().mean().item() / len(held)
111
+ tops[s] = lg.topk(10).indices
112
+ junk[s][i] += nonsem[tops[s]].float().mean().item() / len(held)
113
+ for s in (0, 4):
114
+ inter = (tops[s][:, :, None] == tops[16][:, None, :]).any(-1).float().sum(-1)
115
+ jac[s][i] += (inter / (20 - inter)).mean().item() / len(held)
116
+ agree_ll[i] += (logit_lens(model, h).argmax(-1) == top_model).float().mean().item() / len(held)
117
+ metrics["heldout"] = {node_key(v): {"top1_agree_model_skip16": round(agree[16][i].item(), 3), "top1_agree_model_skip4": round(agree[4][i].item(), 3),
118
+ "top1_agree_model_skip0": round(agree[0][i].item(), 3), "top1_agree_model_logit_lens": round(agree_ll[i].item(), 3),
119
+ "nonsemantic_share_top10_skip16": round(junk[16][i].item(), 3), "nonsemantic_share_top10_skip0": round(junk[0][i].item(), 3),
120
+ "top10_jaccard_skip0_vs_16": round(jac[0][i].item(), 3), "top10_jaccard_skip4_vs_16": round(jac[4][i].item(), 3)}
121
+ for i, v in enumerate(nodes)}
122
+
123
+ say(f"\nEARLY POSITIONS. Jacobian size by source position, relative to the median over positions 16+ (k={K}, {n} prompts)")
124
+ say(f"{'node':12s} " + " ".join(f"p{t:<5d}" for t in range(8)) + " p8-15 mean")
125
+ show = [v for v in nodes if v[0] in ("emb", "x", "u", "s") or (v[0] in ("in", "rec", "out") and v[-1] % 8 == 4)]
126
+ for v in show:
127
+ i = nodes.index(v); r = pos_norm[i] / late[i]
128
+ say(f"{node_key(v):12s} " + " ".join(f"{x:<6.2f}" for x in r[:8].tolist()) + f" {r[8:16].mean().item():.2f}")
129
+ say(f"\nLENS VARIANTS (skip the first 0 / 4 / 16 source positions) and held-out readouts, positions 16+ of {len(held)} held-out paragraphs")
130
+ say(f"{'node':12s} {'|J|':>6s} {'split-half':>10s} {'cos 0v16':>9s} {'cos 4v16':>9s} {'top10 Jacc 0v16':>16s} {'4v16':>6s} {'agree model: skip0':>19s} {'skip4':>6s} {'skip16':>7s} {'logit lens':>11s} {'junk top10 skip0':>17s} {'skip16':>7s}")
131
+ for v in show:
132
+ i = nodes.index(v); k_ = node_key(v); hm = metrics["heldout"][k_]
133
+ say(f"{k_:12s} {J[16][i].norm().item():6.1f} {split_half[i].item():10.3f} {metrics['cosine_skip0_vs_skip16'][k_]:9.3f} {metrics['cosine_skip4_vs_skip16'][k_]:9.3f} "
134
+ f"{hm['top10_jaccard_skip0_vs_16']:16.2f} {hm['top10_jaccard_skip4_vs_16']:6.2f} {hm['top1_agree_model_skip0']:19.2f} {hm['top1_agree_model_skip4']:6.2f} "
135
+ f"{hm['top1_agree_model_skip16']:7.2f} {hm['top1_agree_model_logit_lens']:11.2f} {hm['nonsemantic_share_top10_skip0']:17.2f} {hm['nonsemantic_share_top10_skip16']:7.2f}")
136
+
137
+ # ---- readouts by eye on prompts with a known unspoken intermediate ----
138
+ chat = tok.apply_chat_template([{"role": "user", "content": "What is 17 times 23? Answer briefly."}], tokenize=False, add_generation_prompt=True, enable_thinking=False)
139
+ probes = [("Fact: The currency used in the country shaped like a boot is", "Italy", True),
140
+ ("Fact: The number of legs on the animal that spins webs is", "spider", True),
141
+ ("The capital of the country where sushi originated is", "Japan", True),
142
+ ("Fact: The color of the planet fourth from the Sun is", "Mars", True), (chat, None, False)]
143
+ eye_nodes = [v for v in nodes if v[0] in ("x", "s") or (v[0] in ("in", "rec", "out") and v[-1] % 8 == 6)]
144
+ for text, inter, special in probes:
145
+ ids = tok(text, return_tensors="pt", add_special_tokens=special).input_ids.cuda()
146
+ st, final = states(ids)
147
+ say(f"\nPROMPT {text[-70:]!r} intermediate: {inter} model top-3: {[tok.decode([i]) for i in model.unembed(final)[0, -1].topk(3).indices.tolist()]}")
148
+ for v in eye_nodes:
149
+ i = nodes.index(v); h = st[v][0, -1:]
150
+ row = f" {node_key(v):10s}"
151
+ for s in ((0, 16) if args.pilot else (args.skip,)):
152
+ lg = readout(J[s][i], h)[0]
153
+ rank = ""
154
+ if inter is not None:
155
+ cand = [tok(" " + inter, add_special_tokens=False).input_ids, tok(inter, add_special_tokens=False).input_ids]
156
+ rk = min((lg > lg[c[0]]).sum().item() + 1 for c in cand if len(c) == 1) if any(len(c) == 1 for c in cand) else -1
157
+ rank = f" rank({inter})={rk:<6d}"
158
+ row += f" | skip{s}:{rank} {[tok.decode([t]) for t in lg.topk(6).indices.tolist()]}"
159
+ ll = logit_lens(model, h)[0]
160
+ row += f" | logit lens: {[tok.decode([t]) for t in ll.topk(4).indices.tolist()]}"
161
+ say(row)
162
+
163
+ (out / "readouts.txt").write_text("\n".join(lines) + "\n")
164
+ if not args.pilot:
165
+ big = artifacts_dir(out); big.mkdir(parents=True, exist_ok=True)
166
+ b0 = {0: 0, 4: 1, 16: 2}[args.skip]
167
+ mean_state = hsum[:, b0:].sum(1) / (n * sum(counts[b0:]))
168
+ meta = {"model": "nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16", "snapshot": snap.name, "lens": "plain J-lens (Gurnee et al. 2026 estimator)",
169
+ "n_loops": K, "target": "final residual state (after output sublayer 71)", "n_prompts": n, "seq_len": T,
170
+ "skip_first_positions": args.skip, "corpus": "WikiText-103 raw, validation split, paragraphs of >= 128 tokens, first 128 tokens",
171
+ "seed": args.seed, "dtype": "float32, TF32 matmuls", "readout": "unembed(l2norm(J @ h)); rows of J index output dimensions",
172
+ "nodes": [node_key(v) for v in nodes]}
173
+ lens = JLens({node_key(v): J[args.skip][i] for i, v in enumerate(nodes)}, meta, {node_key(v): mean_state[i] for i, v in enumerate(nodes)})
174
+ path = big / f"jlens_k{K}.safetensors"; lens.save(path)
175
+ metrics["lens_file"] = str(path); metrics["lens_bytes"] = path.stat().st_size
176
+ print("saved", path, path.stat().st_size, flush=True)
177
+ finish_run(out, metrics)
178
+ print("finished", flush=True)
as_run/fit_jpp.py ADDED
@@ -0,0 +1,232 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Fit the J++ lens for Dory at one loop count, with the R-lens (pooled LRP Jacobian) as a by-product.
2
+
3
+ python fit_jpp.py --k 1 --n-prompts 64 --run k1_n64_e8_s0
4
+
5
+ J++ (Ayonrinde & Lindsey 2026) = Jacobians through the LRP backward rules, averaged separately over E clusters
6
+ of source positions (one shared k-means router, read at an early node), recombined per node with signed
7
+ unit-norm weights learned on labelled readout items, and read over a vocabulary without punctuation-like tokens.
8
+ Scored on a held-out half of the Gurnee et al. readout sets, next to the logit lens, the plain J-lens (E001),
9
+ the R-lens, the best single expert, and J++ weights learned on shuffled labels.
10
+ """
11
+ import argparse, json, os, sys, time, pathlib, random
12
+ import torch
13
+
14
+ HERE = pathlib.Path(__file__).resolve().parent; PROJ = HERE.parents[1]
15
+ sys.path.insert(0, str(PROJ / "src")); sys.path.insert(0, os.path.expanduser("~/interp/common"))
16
+ from runmeta import start_run, finish_run, artifacts_dir
17
+ from dory.model import Dory, snapshot_path, load_tokenizer, justnorm, lrp_rules
18
+ from dory.jlens import JLens, source_nodes, node_key, accumulate_grouped
19
+ from dory.jpp import fit_router, fit_expert_weights
20
+ from dory.lens_eval import load_items, semantic_mask, ranks, TASKS
21
+
22
+ ap = argparse.ArgumentParser()
23
+ ap.add_argument("--k", type=int, required=True); ap.add_argument("--n-prompts", type=int, default=64)
24
+ ap.add_argument("--seq-len", type=int, default=128); ap.add_argument("--chunk", type=int, default=256)
25
+ ap.add_argument("--experts", type=int, default=8); ap.add_argument("--router-node", default="in.08")
26
+ ap.add_argument("--router-paragraphs", type=int, default=500); ap.add_argument("--skip", type=int, default=16)
27
+ ap.add_argument("--seed", type=int, default=0); ap.add_argument("--run", required=True); ap.add_argument("--smoke", action="store_true")
28
+ ap.add_argument("--to-shm", action="store_true", help="save the lens files to /dev/shm (the shared volume is at its quota)")
29
+ args = ap.parse_args()
30
+ if args.smoke:
31
+ args.n_prompts, args.router_paragraphs = 2, 30
32
+
33
+ WIKI = "/workspace/huggingface/hub/datasets--Salesforce--wikitext/snapshots/b08601e04326c79dfdd32d625aee71d232d685c3/wikitext-103-raw-v1/validation-00000-of-00001.parquet"
34
+ out = start_run(HERE / "results" / args.run, config=vars(args) | {"corpus": WIKI, "tf32": True, "dtype": "float32", "lrp": "ln, act, half, gate"})
35
+ (out / "pid").write_text(str(os.getpid()))
36
+ torch.backends.cuda.matmul.allow_tf32 = True; torch.backends.cudnn.allow_tf32 = True
37
+ torch.manual_seed(args.seed)
38
+ lines = []
39
+ def say(s=""):
40
+ print(s, flush=True); lines.append(s)
41
+
42
+ snap = snapshot_path(); tok = load_tokenizer(snap)
43
+ model = Dory.from_snapshot(snap, device="cuda", dtype=torch.float32)
44
+ d, T, K, E, V = model.cfg["hidden_size"], args.seq_len, args.k, args.experts, model.cfg["vocab_size"]
45
+ nodes = source_nodes(model, K); N = len(nodes); keys = [node_key(v) for v in nodes]
46
+
47
+ import pyarrow.parquet as pq
48
+ rows = [t for t in pq.read_table(WIKI).column("text").to_pylist() if len(t) > 700 and not t.lstrip().startswith("=")]
49
+ random.Random(args.seed).shuffle(rows)
50
+ pool = []
51
+ for t in rows:
52
+ ids = tok(t.strip(), return_tensors="pt").input_ids
53
+ if ids.shape[1] >= T:
54
+ pool.append(ids[:, :T])
55
+ if len(pool) >= 660:
56
+ break
57
+ fit_ids = pool[: args.n_prompts]; router_ids = pool[100 : 100 + args.router_paragraphs]
58
+ (out / "prompts.json").write_text(json.dumps([{"i": i, "start": tok.decode(x[0, :24])} for i, x in enumerate(fit_ids)], indent=1))
59
+
60
+ # ---- 1. router: k-means on an early node, fitted on paragraphs not used for the Jacobians ----
61
+ rnode = nodes[keys.index(args.router_node)]
62
+ def router_state(ids):
63
+ got = {}
64
+ with torch.no_grad():
65
+ model.forward(ids.cuda(), n_loops=1, tap=lambda nm, h: got.__setitem__(nm, h) if nm == rnode else None)
66
+ return got[rnode][0]
67
+ acts = torch.cat([router_state(x)[args.skip:] for x in router_ids])
68
+ router = fit_router(acts, n_clusters=E, seed=args.seed)
69
+ frac = torch.bincount(router.assign(acts), minlength=E).float() / len(acts)
70
+ say(f"router at {args.router_node}: {len(acts)} positions from {len(router_ids)} paragraphs; cluster shares {[round(x, 3) for x in frac.tolist()]}")
71
+ del acts
72
+
73
+ # ---- 2. expert Jacobians through the LRP backward pass ----
74
+ acc = torch.zeros(2, N, E, d, d, device="cuda"); counts = torch.zeros(2, E, device="cuda")
75
+ t0 = time.time()
76
+ for p, ids in enumerate(fit_ids):
77
+ g = router.assign(router_state(ids)); g[: args.skip] = -1
78
+ with lrp_rules():
79
+ accumulate_grouped(model, ids.cuda(), K, nodes, g, acc[p % 2], chunk=args.chunk)
80
+ counts[p % 2] += torch.bincount(g[args.skip:], minlength=E).float()
81
+ el = time.time() - t0
82
+ print(f"prompt {p + 1}/{len(fit_ids)} {el:.0f}s elapsed, {el / (p + 1):.1f}s per prompt, peak GPU {torch.cuda.max_memory_allocated() / 2**30:.0f} GiB", flush=True)
83
+ fit_seconds = time.time() - t0
84
+ cosf = lambda a, b: torch.nn.functional.cosine_similarity(a.flatten(1), b.flatten(1), dim=1)
85
+ split_half = cosf(acc[0].sum(1) / counts[0].sum(), acc[1].sum(1) / counts[1].sum()) # pooled (R-lens) halves
86
+ experts = acc[0].add_(acc[1]); del acc # [N, E, d, d] sums
87
+ n_e = counts.sum(0)
88
+ R = experts.sum(1) / n_e.sum() # pooled LRP Jacobian = R-lens
89
+ experts /= n_e.clamp(min=1)[None, :, None, None] # per-cluster means
90
+ say(f"positions per expert: {[int(x) for x in n_e.tolist()]}")
91
+
92
+ # ---- 3. labelled readout items ----
93
+ items = load_items(tok, PROJ / "evals" / "lens_eval")
94
+ mask = semantic_mask(tok, V).cuda()
95
+ H = torch.zeros(len(items), N, d, device="cuda"); correct = []
96
+ for i, it in enumerate(items):
97
+ st = {}
98
+ with torch.no_grad():
99
+ final = model.forward(it["ids"].cuda(), n_loops=K, tap=lambda nm, h: st.__setitem__(nm, h))
100
+ H[i] = torch.stack([st[v][0, it["pos"]] for v in nodes])
101
+ top = tok.decode([model.unembed(final)[0, -1].argmax().item()]).strip()
102
+ correct.append(it["answer"] is None or (len(top) > 0 and it["answer"].lower().startswith(top.lower())))
103
+ keep = [i for i, it in enumerate(items) if it["task"] not in ("multihop", "multilingual") or correct[i]]
104
+ rng = random.Random(args.seed); dev, test = [], []
105
+ for task in TASKS:
106
+ idx = [i for i in keep if items[i]["task"] == task]; rng.shuffle(idx)
107
+ dev += idx[: len(idx) // 2]; test += idx[len(idx) // 2 :]
108
+ say("items per task, all / kept (multihop and multilingual kept only where the model's top next token starts the answer) / dev / test: " +
109
+ "; ".join(f"{t} {sum(it['task'] == t for it in items)}/{sum(items[i]['task'] == t for i in keep)}/{sum(items[i]['task'] == t for i in dev)}/{sum(items[i]['task'] == t for i in test)}" for t in TASKS))
110
+
111
+ # ---- 4. expert weights per node, on the dev half ----
112
+ dev_pairs = [(i, tv) for i in dev for tv in items[i]["targets"]]
113
+ pi = torch.tensor([i for i, _ in dev_pairs], device="cuda"); ptargets = [tv for _, tv in dev_pairs]
114
+ perm = list(range(len(ptargets))); random.Random(args.seed + 1).shuffle(perm); shuffled_targets = [ptargets[j] for j in perm]
115
+ W = torch.zeros(N, E, device="cuda"); Wshuf = torch.zeros(N, E, device="cuda"); best_e = torch.zeros(N, dtype=torch.long)
116
+ init = n_e / n_e.norm()
117
+ t1 = time.time()
118
+ for j in range(N):
119
+ feats = torch.einsum("eab,pb->pea", experts[j], H[pi, j])
120
+ W[j] = fit_expert_weights(feats, ptargets, model.unembed, mask, init)
121
+ Wshuf[j] = fit_expert_weights(feats, shuffled_targets, model.unembed, mask, init)
122
+ ce = []
123
+ for e in range(E): # best single expert on dev
124
+ lg = model.unembed(justnorm(feats[:, e])).masked_fill(mask, -1e9)
125
+ ce.append(sum(min(int((lg[p] > lg[p, t]).sum()) for t in tv) < 10 for p, tv in enumerate(ptargets)))
126
+ best_e[j] = int(torch.tensor(ce).argmax())
127
+ del feats
128
+ say(f"weights fitted on {len(dev_pairs)} dev (item, intermediate) pairs in {time.time() - t1:.0f}s")
129
+ JPP = torch.einsum("ne,neab->nab", W, experts); JSHUF = torch.einsum("ne,neab->nab", Wshuf, experts)
130
+ BEST = experts[torch.arange(N), best_e.cuda()]
131
+ lenses = {"logit lens": None, "R-lens (pooled LRP)": R, "best single expert": BEST, "J++ shuffled labels": JSHUF, "J++": JPP}
132
+ jfile = pathlib.Path.home() / f"interp/artifacts/dory_lens/E001_jlens_fit/k{K}_n50_skip16_s0/jlens_k{K}.safetensors"
133
+ if jfile.exists():
134
+ jl = JLens.load(jfile, device="cuda"); lenses = {"logit lens": None, "J-lens (E001, 50 prompts)": torch.stack([jl.J[k_] for k_ in keys])} | {a: b for a, b in lenses.items() if a != "logit lens"}
135
+
136
+ # ---- 5. score on the test half ----
137
+ sub = [v for v in nodes if v[0] in ("in", "x", "rec", "s", "out")] # one node per sublayer output, in order
138
+ pick = sorted({round(f * (len(sub) + 1)) for f in torch.linspace(0.125, 0.875, 7).tolist()})
139
+ seven = [nodes.index(sub[i]) for i in pick]
140
+ grp = lambda v: "input block" if v[0] in ("emb", "in", "x") else "output block" if v[0] == "out" else f"loop {v[1]}"
141
+ blocks = {}
142
+ for j, v in enumerate(nodes):
143
+ blocks.setdefault(grp(v), []).append(j)
144
+ def all_ranks(J, filt=True):
145
+ """[n_test_pairs, N] ranks, and the item index of each pair."""
146
+ rk = []
147
+ for j in range(N):
148
+ h = H[test, j]
149
+ lg = model.unembed(justnorm(h if J is None else h @ J[j].T))
150
+ if filt:
151
+ lg = lg.masked_fill(mask, -1e9)
152
+ col = []
153
+ for r, i in enumerate(test):
154
+ col += [min(int((lg[r] > lg[r, t]).sum()) + 1 for t in tv) for tv in items[i]["targets"]]
155
+ rk.append(col)
156
+ return torch.tensor(rk).T
157
+ pair_item = [i for i in test for _ in items[i]["targets"]]
158
+ def recall(rk, cols, kk):
159
+ """Macro over tasks of the mean over items of the fraction of an item's intermediates with best rank <= kk."""
160
+ hit = (rk[:, cols].min(1).values <= kk).float()
161
+ per_task = {}
162
+ for task in TASKS:
163
+ sc = [sum(hit[p].item() for p in range(len(pair_item)) if pair_item[p] == i) / len(items[i]["targets"]) for i in test if items[i]["task"] == task]
164
+ per_task[task] = sum(sc) / max(len(sc), 1)
165
+ return sum(per_task.values()) / len(per_task), per_task
166
+ metrics = {"n_prompts": len(fit_ids), "seconds_per_prompt": fit_seconds / len(fit_ids), "n_nodes": N, "seven_nodes": [keys[j] for j in seven],
167
+ "positions_per_expert": n_e.tolist(), "n_dev_pairs": len(dev_pairs), "n_test_items": len(test), "n_test_pairs": len(pair_item),
168
+ "rlens_split_half_cosine": {keys[j]: round(split_half[j].item(), 4) for j in range(N)},
169
+ "weights": {keys[j]: [round(x, 4) for x in W[j].tolist()] for j in range(N)},
170
+ "weights_shuffled_labels": {keys[j]: [round(x, 4) for x in Wshuf[j].tolist()] for j in range(N)}, "results": {}}
171
+ say(f"\nTEST HALF: {len(test)} items, {len(pair_item)} (item, intermediate) pairs. Seven scored nodes: {metrics['seven_nodes']}")
172
+ say("recall = macro over the five tasks; an intermediate counts if it is in the top k at ANY of the listed nodes; punctuation-like tokens dropped unless marked")
173
+ hdr = f"{'lens':34s} {'@10 seven':>10s} {'@1 seven':>9s} | " + " ".join(f"{t[:9]:>9s}" for t in TASKS) + " | " + " ".join(f"{b:>12s}" for b in blocks)
174
+ say(hdr)
175
+ RK = {}
176
+ for name, J in list(lenses.items()) + [("J++ (full vocabulary)", JPP)] + ([("J-lens (full vocabulary)", lenses["J-lens (E001, 50 prompts)"])] if "J-lens (E001, 50 prompts)" in lenses else []):
177
+ rk = all_ranks(J, filt="full vocabulary" not in name); RK[name] = rk
178
+ r10, pt = recall(rk, seven, 10); r1, _ = recall(rk, seven, 1)
179
+ byb = {b: recall(rk, cols, 10)[0] for b, cols in blocks.items()}
180
+ pern = {keys[j]: recall(rk, [j], 10)[0] for j in seven}
181
+ metrics["results"][name] = {"recall10_seven": r10, "recall1_seven": r1, "recall10_by_task": pt, "recall10_any_node_in_block": byb, "recall10_per_node": pern}
182
+ say(f"{name:34s} {r10:10.3f} {r1:9.3f} | " + " ".join(f"{pt[t]:9.3f}" for t in TASKS) + " | " + " ".join(f"{byb[b]:12.3f}" for b in blocks))
183
+ say("\nrecall@10 at each of the seven nodes on its own")
184
+ say(f"{'lens':34s} " + " ".join(f"{keys[j]:>9s}" for j in seven))
185
+ for name in lenses:
186
+ say(f"{name:34s} " + " ".join(f"{metrics['results'][name]['recall10_per_node'][keys[j]]:9.3f}" for j in seven))
187
+ say("\nR-lens split-half cosine and J++ weights at the seven nodes (weights in router-cluster order)")
188
+ for j in seven:
189
+ say(f" {keys[j]:10s} split-half {split_half[j].item():6.3f} weights {[round(x, 2) for x in W[j].tolist()]} best expert {int(best_e[j])}")
190
+ med = lambda b: torch.tensor([split_half[j].item() for j in blocks[b]]).median().item()
191
+ say("R-lens split-half cosine, median by block: " + "; ".join(f"{b} {med(b):.2f}" for b in blocks))
192
+ metrics["rlens_split_half_median_by_block"] = {b: med(b) for b in blocks}
193
+
194
+ # ---- 6. by eye ----
195
+ dec = lambda lg: [tok.decode([t]) for t in lg.masked_fill(mask, -1e9).topk(6).indices.tolist()]
196
+ for task in TASKS:
197
+ cand = [i for i in test if items[i]["task"] == task]
198
+ if not cand:
199
+ continue
200
+ i = cand[0]; it = items[i]
201
+ say(f"\n[{task}] {tok.decode(it['ids'][0])[-90:]!r} read at token {tok.decode([it['ids'][0, it['pos']].item()])!r}; intermediates {[tok.decode([tv[0]]) for tv in it['targets']]}")
202
+ for j in seven:
203
+ row = f" {keys[j]:10s}"
204
+ for name in [n for n in ("J-lens (E001, 50 prompts)", "R-lens (pooled LRP)", "J++") if n in lenses]:
205
+ row += f" | {name.split(' ')[0]}: {dec(model.unembed(justnorm(H[i, j] @ lenses[name][j].T)))}"
206
+ say(row)
207
+ (out / "readouts.txt").write_text("\n".join(lines) + "\n")
208
+
209
+ # ---- 7. record, then save ----
210
+ torch.save({"ranks": {n: r.cpu() for n, r in RK.items()}, "pair_item": pair_item, "pair_task": [items[i]["task"] for i in pair_item],
211
+ "item_n_targets": {i: len(items[i]["targets"]) for i in test}, "node_keys": keys, "seven": seven}, out / "test_ranks.pt")
212
+ finish_run(out, metrics) # the numbers are safe before any large write
213
+ if not args.smoke:
214
+ meta = {"model": "nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16", "snapshot": snap.name, "n_loops": K,
215
+ "target": "final residual state (after output sublayer 71)", "n_prompts": len(fit_ids), "seq_len": T, "skip_first_positions": args.skip,
216
+ "corpus": "WikiText-103 raw, validation split, paragraphs of >= 128 tokens, first 128 tokens", "seed": args.seed,
217
+ "backward": "LRP: norm denominators constant (not the query/key norms), SiLU identity rule, half rule on the gated MLP product, attention output gate constant",
218
+ "dtype": "float32, TF32 matmuls", "readout": "unembed(l2norm(J @ h)), ranking without punctuation-like tokens", "nodes": keys}
219
+ files = {"rlens": JLens({keys[j]: R[j] for j in range(N)}, meta | {"lens": "R-lens: pooled LRP Jacobian"}),
220
+ "jpp": JLens({keys[j]: JPP[j] for j in range(N)}, meta | {"lens": f"J++: {E} expert Jacobians (router at {args.router_node}), signed weights learned on dev readout items",
221
+ "weights": metrics["weights"]})}
222
+ shm = pathlib.Path("/dev/shm") / f"USER_dory_lens_E002_{args.run}"
223
+ for where in ((shm,) if args.to_shm else (artifacts_dir(out), shm)):
224
+ try:
225
+ where.mkdir(parents=True, exist_ok=True)
226
+ for nm, lens in files.items():
227
+ lens.save(where / f"{nm}_k{K}.safetensors")
228
+ torch.save(router.state_dict() | {"weights": W.cpu(), "counts": n_e.cpu()}, where / f"router_and_weights_k{K}.pt")
229
+ (out / "lens_location.txt").write_text(str(where) + "\n"); print("saved to", where, flush=True); break
230
+ except Exception as ex: # a full volume raises SafetensorError, not OSError
231
+ print("could not save to", where, "->", repr(ex)[:200], flush=True)
232
+ print("finished", flush=True)
as_run/readout_example.py ADDED
@@ -0,0 +1,34 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Apply a fitted Dory J-lens to a prompt: top tokens at every source node for one position.
2
+
3
+ python readout_example.py --lens /path/to/jlens_k2.safetensors --prompt "Fact: ... is" [--position -1] [--every 4]
4
+
5
+ The lens file fixes the loop count (meta["n_loops"]); the model is run with that many loops.
6
+ """
7
+ import argparse, sys, pathlib, torch
8
+ sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[2] / "src"))
9
+ from dory.model import Dory, snapshot_path, load_tokenizer
10
+ from dory.jlens import JLens, logit_lens, node_key
11
+
12
+ ap = argparse.ArgumentParser()
13
+ ap.add_argument("--lens", required=True); ap.add_argument("--prompt", required=True)
14
+ ap.add_argument("--position", type=int, default=-1); ap.add_argument("--every", type=int, default=4)
15
+ ap.add_argument("--topk", type=int, default=8); ap.add_argument("--chat", action="store_true")
16
+ args = ap.parse_args()
17
+
18
+ snap = snapshot_path(); tok = load_tokenizer(snap)
19
+ model = Dory.from_snapshot(snap, device="cuda", dtype=torch.float32)
20
+ lens = JLens.load(args.lens, device="cuda")
21
+ text = args.prompt
22
+ if args.chat:
23
+ text = tok.apply_chat_template([{"role": "user", "content": text}], tokenize=False, add_generation_prompt=True, enable_thinking=False)
24
+ ids = tok(text, return_tensors="pt", add_special_tokens=not args.chat).input_ids.cuda()
25
+ states = {}
26
+ with torch.no_grad():
27
+ final = model.forward(ids, n_loops=lens.k, tap=lambda n, h: states.__setitem__(n, h))
28
+ dec = lambda lg: [tok.decode([t]) for t in lg.topk(args.topk).indices.tolist()]
29
+ print(f"loops {lens.k}; reading token {tok.decode([ids[0, args.position].item()])!r}; model's next-token top-{args.topk}: {dec(model.unembed(final)[0, args.position])}")
30
+ for i, node in enumerate(lens.nodes()):
31
+ if i % args.every and node[0] not in ("x", "u", "s"):
32
+ continue
33
+ h = states[node][0, args.position]
34
+ print(f"{node_key(node):10s} J-lens {dec(lens.logits(model, h, node))} | logit lens {dec(logit_lens(model, h))[:4]}")
as_run/test_jlens_estimator.py ADDED
@@ -0,0 +1,64 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Known-answer tests for the J-lens estimator on the real model (needs a GPU and the checkpoint).
2
+
3
+ Run: CUDA_VISIBLE_DEVICES=1 HF_HOME=/workspace/huggingface python tests/test_jlens_estimator.py
4
+
5
+ 1. Finite differences. Adding eps*v to a node's state at every position of a bin must move
6
+ sum_t final[t] by (bin Jacobian sum) @ v. This checks the node identity, the position bins, the
7
+ summation over targets and the row/column orientation in one go.
8
+ 2. The node just before a norm: the Jacobian at node s.t applied to the state itself is NOT forced to zero,
9
+ but the raw pre-norm tensor would be; here we check the generic identity that a shuffled v gives a
10
+ different answer (the test in 1 is not passing trivially).
11
+ 3. TF32 matmuls against exact fp32 on the same rows.
12
+ """
13
+ import sys, time, pathlib, torch
14
+ sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1] / "src"))
15
+ from dory.model import Dory, snapshot_path, load_tokenizer
16
+ from dory.jlens import prompt_jacobians, source_nodes, BINS
17
+
18
+ torch.manual_seed(0)
19
+ snap = snapshot_path(); tok = load_tokenizer(snap)
20
+ m = Dory.from_snapshot(snap, device="cuda", dtype=torch.float32)
21
+ text = ("The Amazon River in South America is the largest river by discharge volume of water in the world, and "
22
+ "the disputed longest river system in the world in comparison to the Nile. The headwaters of the Apurimac "
23
+ "River on Nevado Mismi had been considered for nearly a century as the Amazon basin's most distant source.")
24
+ ids = tok(text, return_tensors="pt").input_ids[:, :48].cuda()
25
+ T, d, K = ids.shape[1], 2560, 2
26
+ nodes = [("emb",), ("in", 5), ("x",), ("u", 1), ("rec", 1, 30), ("s", 1), ("u", 2), ("rec", 2, 41), ("s", 2), ("out", 62)]
27
+ assert set(nodes) <= set(source_nodes(m, K))
28
+ rows = torch.randperm(d, device="cuda")[:64]
29
+
30
+ torch.backends.cuda.matmul.allow_tf32 = False; torch.backends.cudnn.allow_tf32 = False
31
+ t0 = time.time(); jsum, sq, hsum = prompt_jacobians(m, ids, K, nodes, chunk=64, rows=rows); t_exact = time.time() - t0
32
+ print(f"exact fp32: {len(rows)} rows, {len(nodes)} nodes, T={T}: {t_exact:.1f}s; jsum {tuple(jsum.shape)}")
33
+
34
+ def final_sum(node, lo, hi, delta):
35
+ def tap(n, h):
36
+ if n == node:
37
+ h = h.clone(); h[0, lo:hi] += delta; return h
38
+ with torch.no_grad():
39
+ return m.forward(ids, n_loops=K, tap=tap)[0].double().sum(0)[rows]
40
+
41
+ worst = 1.0
42
+ print("finite-difference check (cosine, and norm ratio autograd/FD); shuffled-v cosine is the null")
43
+ for j, node in enumerate(nodes):
44
+ for b, (lo, hi) in enumerate(BINS):
45
+ hi = T if hi is None else hi
46
+ v = torch.randn(d, device="cuda"); v /= v.norm()
47
+ eps = 2e-3
48
+ fd = (final_sum(node, lo, hi, eps * v) - final_sum(node, lo, hi, -eps * v)) / (2 * eps)
49
+ ag = (jsum[j, b].double() @ v.double())
50
+ null = (jsum[j, b].double() @ v[torch.randperm(d, device="cuda")].double())
51
+ cos = torch.nn.functional.cosine_similarity(fd, ag, dim=0).item()
52
+ cnull = torch.nn.functional.cosine_similarity(fd, null, dim=0).item()
53
+ worst = min(worst, cos)
54
+ print(f" {str(node):14s} positions [{lo:2d},{hi:2d}) cos {cos:.4f} ratio {(ag.norm()/fd.norm()).item():.3f} shuffled-v cos {cnull:+.3f}")
55
+ print("worst cosine:", round(worst, 4))
56
+
57
+ torch.backends.cuda.matmul.allow_tf32 = True; torch.backends.cudnn.allow_tf32 = True
58
+ t0 = time.time(); jsum2, _, _ = prompt_jacobians(m, ids, K, nodes, chunk=64, rows=rows); t_tf = time.time() - t0
59
+ cos = torch.nn.functional.cosine_similarity(jsum.flatten(2), jsum2.flatten(2), dim=-1)
60
+ print(f"TF32: {t_tf:.1f}s (exact {t_exact:.1f}s); cosine with exact per (node, bin): min {cos.min().item():.5f} mean {cos.mean().item():.5f}")
61
+ rel = ((jsum - jsum2).flatten(2).norm(dim=-1) / jsum.flatten(2).norm(dim=-1))
62
+ print(f" relative Frobenius error: max {rel.max().item():.4f}")
63
+ assert worst > 0.99, "finite-difference check failed"
64
+ print("OK")
as_run/test_lrp_rules.py ADDED
@@ -0,0 +1,34 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """The LRP rules must not change the forward pass, and must change the gradient.
2
+ Run: CUDA_VISIBLE_DEVICES=1 HF_HOME=/workspace/huggingface python tests/test_lrp_rules.py"""
3
+ import sys, pathlib, torch
4
+ sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1] / "src"))
5
+ from dory.model import Dory, snapshot_path, load_tokenizer, lrp_rules, LRP
6
+ from dory.jlens import prompt_jacobians
7
+ torch.manual_seed(0)
8
+ snap = snapshot_path(); tok = load_tokenizer(snap)
9
+ m = Dory.from_snapshot(snap, device="cuda", dtype=torch.float32)
10
+ ids = tok("The Amazon River in South America is the largest river by discharge volume of water in the world, and the "
11
+ "disputed longest river system in the world in comparison to the Nile.", return_tensors="pt").input_ids.cuda()
12
+ with torch.no_grad():
13
+ a = m.logits(ids, n_loops=2)
14
+ with lrp_rules():
15
+ b = m.logits(ids, n_loops=2)
16
+ assert not LRP.on
17
+ print("forward: max abs logit difference with the rules on:", (a - b).abs().max().item(), "| logit scale", a.abs().max().item())
18
+ assert (a - b).abs().max().item() < 1e-2
19
+ nodes = [("in", 5), ("x",), ("rec", 1, 30), ("s", 1), ("rec", 2, 30), ("s", 2), ("out", 62)]
20
+ rows = torch.randperm(2560, device="cuda")[:96]
21
+ bins = ((0, 16), (16, None))
22
+ plain, sq_p, hs = prompt_jacobians(m, ids, 2, nodes, chunk=96, rows=rows, bins=bins)
23
+ with lrp_rules():
24
+ lrp, sq_l, _ = prompt_jacobians(m, ids, 2, nodes, chunk=96, rows=rows, bins=bins)
25
+ T = ids.shape[1]
26
+ print(f"{'node':14s} {'|J| plain':>10s} {'|J| LRP':>9s} {'cosine':>7s} own-state term |J h|/|J| per position: {'plain':>6s} {'LRP':>6s} {'random dir (LRP)':>17s}")
27
+ for j, n in enumerate(nodes):
28
+ P, L = plain[j, 1], lrp[j, 1]
29
+ cos = torch.nn.functional.cosine_similarity(P.flatten(), L.flatten(), dim=0).item()
30
+ h = hs[j, 1] / (T - 16); h = h / h.norm() # mean state direction of this prompt
31
+ r = torch.randn_like(h); r /= r.norm()
32
+ print(f"{str(n):14s} {P.norm().item():10.2f} {L.norm().item():9.2f} {cos:7.3f} {'':38s} {(P @ h).norm().item() / P.norm().item():6.3f} {(L @ h).norm().item() / L.norm().item():6.3f} {(L @ r).norm().item() / L.norm().item():17.3f}")
33
+ print("gradient growth toward early nodes, per-position rms norm (plain / LRP):", [f"{sq_p[j, 16:].mean().sqrt().item():.2f}/{sq_l[j, 16:].mean().sqrt().item():.2f}" for j in range(len(nodes))])
34
+ print("OK")
dory/__init__.py ADDED
File without changes
dory/chain.py ADDED
@@ -0,0 +1,62 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """A chain-lens analogue for Dory (after "The Chain Lens [interim]", vgel & Douglas 2026).
2
+
3
+ The published chain lens rolls a lens's top tokens out into short phrases with the model's multi-token-prediction
4
+ head. Dory has no such head. Here the model itself does the roll-out: the state under study is placed at one
5
+ "anchor" position at its own node, the rest of the network is run on it, and the chain tokens that follow may
6
+ attend only to the anchor (from that node on) and to each other. Nothing else of the original prompt is present,
7
+ so whatever the chain says about the prompt came through that one state. Decoding is contrastive against the
8
+ same roll-out from the node's mean state (Li et al. 2022: among tokens within a factor of the best, pick the one
9
+ whose log-probability rises most over the prior).
10
+
11
+ Restriction: a node inside loop t is rolled out with exactly t loops, so the anchor never needs the input-block
12
+ output of a prompt it no longer has.
13
+ """
14
+ from __future__ import annotations
15
+
16
+ import torch
17
+
18
+ from .model import ATTN, Dory
19
+
20
+
21
+ @torch.no_grad()
22
+ def rollout_logits(model: Dory, anchors: torch.Tensor, node: tuple, k: int, chain: torch.Tensor) -> torch.Tensor:
23
+ """Next-token logits after each chain. anchors [B, d] (states at `node`), chain [B, m] token ids, m >= 1."""
24
+ assert node[0] not in ("u", "rec", "s") or node[1] == k, "roll a loop-t node out with k = t loops"
25
+ B, m = chain.shape
26
+ ids = torch.cat([chain[:, :1], chain], dim=1) # position 0 is the anchor (its token is a placeholder)
27
+ block = torch.zeros(m + 1, dtype=torch.bool); block[0] = True
28
+ ATTN.block = block
29
+
30
+ def tap(name, h):
31
+ if name == node:
32
+ h = h.clone(); h[:, 0] = anchors.to(h.dtype)
33
+ ATTN.block = None # from here on the chain can see the anchor
34
+ return h
35
+
36
+ try:
37
+ final = model.forward(ids, n_loops=k, tap=tap)
38
+ finally:
39
+ ATTN.block = None
40
+ return model.unembed(final[:, -1])
41
+
42
+
43
+ @torch.no_grad()
44
+ def chains(model: Dory, h: torch.Tensor, node: tuple, k: int, seeds: torch.Tensor, steps: int = 4,
45
+ prior: torch.Tensor | None = None, alpha: float = 0.1, ban: torch.Tensor | None = None) -> torch.Tensor:
46
+ """Extend each seed token by `steps` tokens. h [d]; seeds [K]; prior [d] = the mean state at `node` for
47
+ contrastive decoding (None = plain greedy). Returns token ids [K, 1 + steps]."""
48
+ K = seeds.shape[0]
49
+ chain = seeds[:, None].clone()
50
+ A = h[None].expand(K, -1)
51
+ for _ in range(steps):
52
+ lp = torch.log_softmax(rollout_logits(model, A, node, k, chain), -1)
53
+ if ban is not None:
54
+ lp = lp.masked_fill(ban, -1e9)
55
+ if prior is None:
56
+ nxt = lp.argmax(-1)
57
+ else:
58
+ lq = torch.log_softmax(rollout_logits(model, prior[None].expand(K, -1), node, k, chain), -1)
59
+ ok = lp >= lp.max(-1, keepdim=True).values + torch.log(torch.tensor(alpha))
60
+ nxt = (lp - lq).masked_fill(~ok, -1e9).argmax(-1)
61
+ chain = torch.cat([chain, nxt[:, None]], dim=1)
62
+ return chain
dory/jlens.py ADDED
@@ -0,0 +1,166 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Jacobian lens (J-lens) for Dory.
2
+
3
+ The lens of Gurnee et al. 2026 ("Verbalizable Representations Form a Global Workspace in Language Models"):
4
+ for a source node v, one d x d matrix
5
+
6
+ J_v = mean over prompts, mean over source positions s, sum over target positions t >= s of
7
+ d final[t] / d h_v[s]
8
+
9
+ and the readout logits = unembed( N( J_v h ) ). The estimator is the paper's pseudocode: one backward pass
10
+ per output dimension with a one-hot injected at every target position, the gradient averaged over source
11
+ positions, then over prompts. Here the backward passes are batched (`is_grads_batched`).
12
+
13
+ Dory specifics.
14
+ * The graph is the unrolled one: every firing of a recurrent sublayer is its own source node, for a fixed
15
+ number of loops k. A lens is therefore fitted for one k.
16
+ * The target is the final residual state (after output sublayer 71), which is what the LM head reads.
17
+ Dory has no final norm layer; the state is already unit norm, and `N` above is that same L2 norm.
18
+ * Source nodes, in forward order: emb; in.00..in.14; x (the input block's output, i.e. after sublayer 15);
19
+ for each loop t: u.t (the loop input N(s_{t-1} + x)), rec.t.16..rec.t.54, s.t (the loop output, i.e.
20
+ after sublayer 55); out.56..out.70. Every sublayer output has exactly one node.
21
+ """
22
+ from __future__ import annotations
23
+
24
+ import json
25
+ from pathlib import Path
26
+
27
+ import torch
28
+ from safetensors.torch import load_file, save_file
29
+
30
+ from .model import Dory, justnorm
31
+
32
+ BINS = ((0, 4), (4, 16), (16, None)) # source-position bins kept separately during the fit
33
+
34
+
35
+ def source_nodes(model: Dory, k: int) -> list[tuple]:
36
+ n_in, n_rec, n_out = model.n_in, model.n_rec, model.n_out
37
+ nodes = [("emb",)] + [("in", i) for i in range(n_in - 1)] + [("x",)]
38
+ for t in range(1, k + 1):
39
+ nodes += [("u", t)] + [("rec", t, i) for i in range(n_in, n_in + n_rec - 1)] + [("s", t)]
40
+ nodes += [("out", i) for i in range(n_in + n_rec, n_in + n_rec + n_out - 1)]
41
+ return nodes
42
+
43
+
44
+ def node_key(node: tuple) -> str:
45
+ return ".".join([node[0]] + [f"{v:02d}" if node[0] in ("in", "rec", "out") and j == len(node) - 2 else str(v)
46
+ for j, v in enumerate(node[1:])])
47
+
48
+
49
+ def key_node(key: str) -> tuple:
50
+ p = key.split(".")
51
+ return (p[0], *[int(v) for v in p[1:]])
52
+
53
+
54
+ def run_with_graph(model: Dory, ids: torch.Tensor, k: int, nodes: list[tuple]):
55
+ """Forward pass that keeps the autograd graph from every source node to the final state."""
56
+ want, got = set(nodes), {}
57
+
58
+ def tap(name, h):
59
+ if name == ("emb",):
60
+ h = h.detach().requires_grad_(True)
61
+ got[name] = h
62
+ return h
63
+ if name in want:
64
+ got[name] = h
65
+
66
+ final = model.forward(ids, n_loops=k, tap=tap)
67
+ return final, [got[n] for n in nodes]
68
+
69
+
70
+ def prompt_jacobians(model: Dory, ids: torch.Tensor, k: int, nodes: list[tuple], chunk: int = 256,
71
+ bins=BINS, rows: torch.Tensor | None = None):
72
+ """Per-prompt Jacobian sums for one prompt of shape [1, T].
73
+
74
+ Returns
75
+ jsum [n_nodes, n_bins, R, d] sum over source positions in the bin of d(sum_t final[t, r]) / d h[s]
76
+ sq [n_nodes, T] squared Frobenius norm of the per-source-position Jacobian (over the R rows)
77
+ hsum [n_nodes, n_bins, d] sum of the source states over positions in the bin
78
+ `rows` restricts the output dimensions (default: all d).
79
+ """
80
+ assert ids.shape[0] == 1
81
+ T, d = ids.shape[1], model.cfg["hidden_size"]
82
+ rows = torch.arange(d, device=ids.device) if rows is None else rows
83
+ bins = [(lo, T if hi is None else hi) for lo, hi in bins]
84
+ with torch.enable_grad():
85
+ final, srcs = run_with_graph(model, ids, k, nodes)
86
+ z = final[0].sum(0) # [d]: summed over target positions
87
+ jsum = torch.zeros(len(nodes), len(bins), len(rows), d, device=ids.device)
88
+ sq = torch.zeros(len(nodes), T, device=ids.device)
89
+ for c0 in range(0, len(rows), chunk):
90
+ r = rows[c0 : c0 + chunk]
91
+ cot = torch.zeros(len(r), d, device=ids.device)
92
+ cot[torch.arange(len(r)), r] = 1.0
93
+ grads = torch.autograd.grad(z, srcs, grad_outputs=cot, is_grads_batched=True, retain_graph=True)
94
+ for j, g in enumerate(grads): # g: [C, 1, T, d]
95
+ g = g[:, 0]
96
+ sq[j] += g.pow(2).sum(dim=(0, 2))
97
+ for b, (lo, hi) in enumerate(bins):
98
+ jsum[j, b, c0 : c0 + len(r)] = g[:, lo:hi].sum(1)
99
+ del grads
100
+ hsum = torch.stack([torch.stack([h.detach()[0, lo:hi].sum(0) for lo, hi in bins]) for h in srcs])
101
+ return jsum, sq, hsum
102
+
103
+
104
+ class JLens:
105
+ """A fitted lens: one d x d matrix per source node, for one loop count."""
106
+
107
+ def __init__(self, J: dict[str, torch.Tensor], meta: dict, mean_state: dict[str, torch.Tensor] | None = None):
108
+ self.J, self.meta, self.mean_state = J, meta, mean_state or {}
109
+
110
+ @property
111
+ def k(self) -> int:
112
+ return int(self.meta["n_loops"])
113
+
114
+ def nodes(self) -> list[tuple]:
115
+ return [key_node(k) for k in self.J]
116
+
117
+ def transport(self, h: torch.Tensor, node: tuple) -> torch.Tensor:
118
+ J = self.J[node_key(node)]
119
+ return h.float() @ J.to(h.device).float().T
120
+
121
+ def logits(self, model: Dory, h: torch.Tensor, node: tuple) -> torch.Tensor:
122
+ """Readout: unembed(N(J h)). h: [..., d] state at `node`."""
123
+ return model.unembed(justnorm(self.transport(h, node)).to(model.lm_head.weight.dtype))
124
+
125
+ def save(self, path: str | Path):
126
+ tensors = {f"J/{k}": v.contiguous().cpu() for k, v in self.J.items()}
127
+ tensors.update({f"mean/{k}": v.contiguous().cpu() for k, v in self.mean_state.items()})
128
+ save_file(tensors, str(path), metadata={"meta": json.dumps(self.meta)})
129
+
130
+ @classmethod
131
+ def load(cls, path: str | Path, device="cpu") -> "JLens":
132
+ from safetensors import safe_open
133
+ with safe_open(str(path), framework="pt") as f:
134
+ meta = json.loads(f.metadata()["meta"])
135
+ t = load_file(str(path), device=device)
136
+ return cls({k[2:]: v for k, v in t.items() if k.startswith("J/")}, meta,
137
+ {k[5:]: v for k, v in t.items() if k.startswith("mean/")})
138
+
139
+
140
+ def logit_lens(model: Dory, h: torch.Tensor) -> torch.Tensor:
141
+ """Baseline: the unembedding applied to the state directly (J = identity)."""
142
+ return model.unembed(justnorm(h.float()).to(model.lm_head.weight.dtype))
143
+
144
+
145
+ def accumulate_grouped(model: Dory, ids: torch.Tensor, k: int, nodes: list[tuple], group_ids: torch.Tensor,
146
+ out: torch.Tensor, chunk: int = 256) -> list[torch.Tensor]:
147
+ """Add one prompt's Jacobian sums into `out` [n_nodes, n_groups, d, d], source positions grouped.
148
+
149
+ group_ids: [T] long, the group of each source position (-1 = leave the position out). For group g,
150
+ out[node, g] += sum over positions s in g of d(sum_t final[t]) / d h_node[s] (rows = output dims).
151
+ Returns the source states [T, d] of every node."""
152
+ assert ids.shape[0] == 1
153
+ d, G = model.cfg["hidden_size"], out.shape[1]
154
+ onehot = torch.nn.functional.one_hot(group_ids.clamp(min=0), G).float() * (group_ids >= 0).float()[:, None] # [T, G]
155
+ with torch.enable_grad():
156
+ final, srcs = run_with_graph(model, ids, k, nodes)
157
+ z = final[0].sum(0)
158
+ for c0 in range(0, d, chunk):
159
+ C = min(chunk, d - c0)
160
+ cot = torch.zeros(C, d, device=ids.device)
161
+ cot[torch.arange(C), c0 + torch.arange(C)] = 1.0
162
+ grads = torch.autograd.grad(z, srcs, grad_outputs=cot, is_grads_batched=True, retain_graph=True)
163
+ for j, g in enumerate(grads):
164
+ out[j, :, c0 : c0 + C] += torch.einsum("tg,ctd->gcd", onehot, g[:, 0])
165
+ del grads
166
+ return [h.detach()[0] for h in srcs]
dory/jpp.py ADDED
@@ -0,0 +1,64 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Pieces of the J++ lens (Ayonrinde & Lindsey 2026) beyond the LRP backward pass: the activation router
2
+ (k-means on PCA projections), and the learned signed weights that combine the expert Jacobians."""
3
+ from __future__ import annotations
4
+
5
+ import torch
6
+
7
+ from .model import justnorm
8
+
9
+
10
+ class Router:
11
+ """Assigns a state to one of E clusters: unit-normalise, centre, project on the top PCA directions, nearest centroid."""
12
+
13
+ def __init__(self, mean, comps, centroids):
14
+ self.mean, self.comps, self.centroids = mean, comps, centroids
15
+
16
+ def project(self, h):
17
+ return (justnorm(h.float()) - self.mean) @ self.comps.T
18
+
19
+ def assign(self, h):
20
+ return torch.cdist(self.project(h), self.centroids).argmin(-1)
21
+
22
+ def state_dict(self):
23
+ return {"router/mean": self.mean, "router/comps": self.comps, "router/centroids": self.centroids}
24
+
25
+
26
+ def fit_router(acts: torch.Tensor, n_clusters: int = 8, n_pca: int = 64, seed: int = 0, iters: int = 100) -> Router:
27
+ x = justnorm(acts.float()); mean = x.mean(0); x = x - mean
28
+ evals, evecs = torch.linalg.eigh((x.T @ x) / len(x))
29
+ comps = evecs[:, -n_pca:].T.contiguous()
30
+ z = x @ comps.T
31
+ g = torch.Generator(device="cpu").manual_seed(seed)
32
+ cent = z[torch.randint(len(z), (1,), generator=g)]
33
+ for _ in range(n_clusters - 1): # k-means++ seeding
34
+ d2 = torch.cdist(z, cent).min(1).values.pow(2)
35
+ cent = torch.cat([cent, z[torch.multinomial((d2 / d2.sum()).cpu(), 1, generator=g).to(z.device)]])
36
+ for _ in range(iters):
37
+ a = torch.cdist(z, cent).argmin(1)
38
+ new = torch.stack([z[a == c].mean(0) if (a == c).any() else cent[c] for c in range(n_clusters)])
39
+ if torch.allclose(new, cent, atol=1e-6):
40
+ break
41
+ cent = new
42
+ return Router(mean, comps, cent)
43
+
44
+
45
+ def fit_expert_weights(feats: torch.Tensor, targets: list[list[int]], unembed, vocab_mask: torch.Tensor,
46
+ init: torch.Tensor, steps: int = 200, lr: float = 0.05) -> torch.Tensor:
47
+ """Signed, unit-norm weights over experts for one node.
48
+
49
+ feats [P, E, d]: each expert's transport of the P labelled states. The readout of weights w is
50
+ unembed(N(sum_e w_e feats[:, e])). Loss: cross-entropy of the labelled token over the kept vocabulary
51
+ (a smooth stand-in for recall@10). `targets[p]` lists acceptable token ids."""
52
+ v = init.clone().float().requires_grad_(True)
53
+ opt = torch.optim.Adam([v], lr=lr)
54
+ P = feats.shape[0]
55
+ tmask = torch.zeros(P, vocab_mask.shape[0], dtype=torch.bool, device=feats.device)
56
+ for p, t in enumerate(targets):
57
+ tmask[p, t] = True
58
+ with torch.enable_grad():
59
+ for _ in range(steps):
60
+ w = v / v.norm()
61
+ lg = unembed(justnorm(torch.einsum("e,ped->pd", w, feats))).masked_fill(vocab_mask, -1e9)
62
+ loss = (torch.logsumexp(lg, -1) - torch.logsumexp(lg.masked_fill(~tmask, -1e9), -1)).mean()
63
+ opt.zero_grad(); loss.backward(); opt.step()
64
+ return (v / v.norm()).detach()
dory/lens_eval.py ADDED
@@ -0,0 +1,55 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """The intermediate-variable readout evaluation (Gurnee et al. 2026 prompt sets; scoring as in the J++ post)."""
2
+ from __future__ import annotations
3
+
4
+ import json
5
+ from pathlib import Path
6
+
7
+ import torch
8
+
9
+ TASKS = ("multihop", "multilingual", "typo", "association", "poetry")
10
+
11
+
12
+ def token_variants(tok, word: str) -> list[int]:
13
+ out = []
14
+ for s in {" " + word, word, " " + word.capitalize(), word.capitalize(), " " + word.lower(), word.lower()}:
15
+ t = tok(s, add_special_tokens=False).input_ids
16
+ if len(t) == 1 and t[0] not in out:
17
+ out.append(t[0])
18
+ return out
19
+
20
+
21
+ def load_items(tok, folder: str | Path, tasks=TASKS) -> list[dict]:
22
+ """Items with: task, name, ids [1, T], pos (readout position), targets (one list of token ids per usable
23
+ intermediate), answer (the expected next word, or None)."""
24
+ items = []
25
+ for task in tasks:
26
+ for it in json.loads((Path(folder) / f"lens-eval-{task}.json").read_text())["items"]:
27
+ ids = tok(it["prompt"].rstrip(), return_tensors="pt").input_ids
28
+ pos = ids.shape[1] - 1
29
+ if task == "poetry": # read at the newline that ends line 1
30
+ nl = [i for i in range(ids.shape[1]) if "\n" in tok.decode([ids[0, i].item()])]
31
+ if not nl:
32
+ continue
33
+ pos = nl[-1]
34
+ targets = [v for v in (token_variants(tok, w) for w in it["intermediates"]) if v]
35
+ if targets:
36
+ items.append({"task": task, "name": it["name"], "ids": ids, "pos": pos, "targets": targets,
37
+ "answer": it.get("target")})
38
+ return items
39
+
40
+
41
+ def semantic_mask(tok, vocab_size: int) -> torch.Tensor:
42
+ """True for tokens dropped from the readout: no letter or digit, or a special token."""
43
+ special = set(tok.all_special_ids)
44
+ out = []
45
+ for i in range(vocab_size):
46
+ s = tok.decode([i])
47
+ out.append(i in special or s.startswith("<|") or s in ("<s>", "</s>", "<unk>", "<think>", "</think>") or not any(c.isalnum() for c in s))
48
+ return torch.tensor(out)
49
+
50
+
51
+ def ranks(logits: torch.Tensor, targets: list[list[int]], mask: torch.Tensor | None) -> list[int]:
52
+ """Rank (1 = top) of each intermediate: the best rank over its token variants. logits [V]."""
53
+ if mask is not None:
54
+ logits = logits.masked_fill(mask, -1e9)
55
+ return [min(int((logits > logits[t]).sum()) + 1 for t in variants) for variants in targets]
dory/model.py ADDED
@@ -0,0 +1,283 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Dory1.0-4B forward pass in plain PyTorch, for interpretability work.
2
+
3
+ Ported from the vLLM reference implementation (vllm/model_executor/models/dory.py, PR 60149) and the
4
+ architecture section of arXiv 2611.03998. No KV cache: every call is one full-sequence forward pass, which
5
+ is the per-loop cache semantics (loop i attends to loop-i keys and values of earlier positions).
6
+
7
+ Naming. The checkpoint counts sublayers: 72 "layers", each either attention or MLP.
8
+ input block sublayers 0..15 run once
9
+ recurrent block sublayers 16..55 run n_loops times, weights shared
10
+ output block sublayers 56..71 run once
11
+ Residual nodes, in forward order (each is a unit-norm vector per position, except "emb"):
12
+ ("emb",) raw token embedding
13
+ ("in", i) after input sublayer i i in 0..15
14
+ ("x",) input block output (same tensor as ("in", 15))
15
+ ("u", t) loop t input, N(s_{t-1} + x) t in 1..n_loops
16
+ ("rec", t, i) after recurrent sublayer i in loop t, before the loop-end norm i in 16..55
17
+ ("s", t) loop t output, N(F(u_t))
18
+ ("out", i) after output sublayer i i in 56..71
19
+ A `tap(name, h)` callback sees every node and may return a replacement tensor.
20
+ """
21
+ from __future__ import annotations
22
+
23
+ import json
24
+ import math
25
+ from pathlib import Path
26
+ from typing import Callable, Optional
27
+
28
+ import torch
29
+ import torch.nn.functional as F
30
+ from safetensors import safe_open
31
+ from torch import nn
32
+
33
+ EPS = 1e-6
34
+ Tap = Optional[Callable[[tuple, torch.Tensor], Optional[torch.Tensor]]]
35
+
36
+
37
+ class _LRP:
38
+ """Backward-pass rules of Layer-wise Relevance Propagation, as used by the R-lens and the J++ lens.
39
+ They add stop-gradients only, so the forward values are unchanged.
40
+ ln norm denominators are treated as constants (every L2 norm except the query and key norms)
41
+ act SiLU(g) is treated as g * const, const = sigmoid(g)
42
+ half the gradient through the gated MLP's product is split evenly between its two branches
43
+ gate the attention output gate is treated as a constant (Dory-specific; the posts do not cover it)
44
+ """
45
+ on = False
46
+ ln = act = half = gate = True
47
+
48
+
49
+ LRP = _LRP()
50
+
51
+
52
+ class _Attn:
53
+ """`block`: optional bool tensor [T]; key positions that no other position may attend to (a position always
54
+ sees itself). Used by the chain roll-out to hide the anchor position until its state has been set."""
55
+ block = None
56
+
57
+
58
+ ATTN = _Attn()
59
+
60
+
61
+ class lrp_rules:
62
+ """Context manager: `with lrp_rules(): ...` turns the LRP backward rules on."""
63
+
64
+ def __init__(self, **flags):
65
+ self.flags = flags
66
+
67
+ def __enter__(self):
68
+ self.saved = {k: getattr(LRP, k) for k in ("on", "ln", "act", "half", "gate")}
69
+ LRP.on = True
70
+ for k, v in self.flags.items():
71
+ setattr(LRP, k, v)
72
+
73
+ def __exit__(self, *a):
74
+ for k, v in self.saved.items():
75
+ setattr(LRP, k, v)
76
+
77
+
78
+ def justnorm(x: torch.Tensor, dim: int = -1, lrp: bool = True) -> torch.Tensor:
79
+ """nGPT normalisation: x * rsqrt(sum(x^2) + eps), accumulated in fp32.
80
+ Under the LRP rules the denominator is detached, unless `lrp=False` (query and key norms)."""
81
+ xf = x.float()
82
+ scale = torch.rsqrt(xf.pow(2).sum(dim=dim, keepdim=True) + EPS)
83
+ if LRP.on and LRP.ln and lrp:
84
+ scale = scale.detach()
85
+ return (xf * scale).to(x.dtype)
86
+
87
+
88
+ def slerp_norm(a: torch.Tensor, b: torch.Tensor, lr: torch.Tensor) -> torch.Tensor:
89
+ ya, yb = justnorm(a).float(), justnorm(b).float()
90
+ return justnorm(ya + lr.float() * (yb - ya)).to(a.dtype)
91
+
92
+
93
+ def rope(x: torch.Tensor, positions: torch.Tensor, theta: float) -> torch.Tensor:
94
+ """Rotary embedding, half-split ("neox") layout, on all of the last dim. x: [B, T, H, D]."""
95
+ d = x.shape[-1]
96
+ inv_freq = 1.0 / (theta ** (torch.arange(0, d, 2, device=x.device, dtype=torch.float32) / d))
97
+ ang = positions.float()[:, None] * inv_freq[None, :] # [T, D/2]
98
+ cos, sin = ang.cos()[None, :, None, :], ang.sin()[None, :, None, :]
99
+ xf = x.float()
100
+ x1, x2 = xf[..., : d // 2], xf[..., d // 2 :]
101
+ return torch.cat([x1 * cos - x2 * sin, x2 * cos + x1 * sin], dim=-1).to(x.dtype)
102
+
103
+
104
+ class DoryAttention(nn.Module):
105
+ def __init__(self, cfg: dict, layer_type: str):
106
+ super().__init__()
107
+ d, hd = cfg["hidden_size"], cfg["head_dim"]
108
+ self.nq, self.nkv, self.hd = cfg["num_attention_heads"], cfg["num_key_value_heads"], hd
109
+ std = cfg["initializer_range"]
110
+ self.q_proj = nn.Linear(d, self.nq * hd, bias=False)
111
+ self.k_proj = nn.Linear(d, self.nkv * hd, bias=False)
112
+ self.v_proj = nn.Linear(d, self.nkv * hd, bias=False)
113
+ self.g_proj = nn.Linear(d, self.nq * hd, bias=False)
114
+ self.o_proj = nn.Linear(self.nq * hd, d, bias=False)
115
+ self.sqk = nn.Parameter(torch.empty(self.nq * hd))
116
+ self.s_gate = nn.Parameter(torch.empty(self.nq * hd))
117
+ self.sqk_factor = hd**0.5 / std
118
+ g0 = cfg.get("attention_output_gate_logit_scale_init") or d**0.5
119
+ self.s_gate_factor = g0**0.5 / std
120
+ self.theta = cfg["rope_parameters"][layer_type]["rope_theta"]
121
+ self.window = cfg["sliding_window"] if layer_type == "sliding_attention" else None
122
+
123
+ def forward(self, h: torch.Tensor, positions: torch.Tensor) -> torch.Tensor:
124
+ B, T, _ = h.shape
125
+ q = self.q_proj(h).view(B, T, self.nq, self.hd)
126
+ k = self.k_proj(h).view(B, T, self.nkv, self.hd)
127
+ v = self.v_proj(h).view(B, T, self.nkv, self.hd)
128
+ gate = self.g_proj(h)
129
+ q, k = rope(q, positions, self.theta), rope(k, positions, self.theta)
130
+ sqk = (self.sqk.float() * self.sqk_factor).view(1, 1, self.nq, self.hd)
131
+ q = (sqk.square() * justnorm(q, lrp=False).float()).to(h.dtype) # qk-norm after RoPE
132
+ k = justnorm(k, lrp=False)
133
+ rep = self.nq // self.nkv
134
+ q = q.transpose(1, 2) # [B, nq, T, hd]
135
+ k = k.transpose(1, 2).repeat_interleave(rep, dim=1)
136
+ v = v.transpose(1, 2).repeat_interleave(rep, dim=1)
137
+ i = torch.arange(T, device=h.device)
138
+ mask = i[None, :] <= i[:, None] # causal: key j <= query i
139
+ if self.window is not None and T > self.window:
140
+ mask = mask & (i[:, None] - i[None, :] < self.window)
141
+ if ATTN.block is not None:
142
+ mask = mask & (~ATTN.block.to(h.device)[None, :] | torch.eye(T, dtype=torch.bool, device=h.device))
143
+ out = F.scaled_dot_product_attention(q, k, v, attn_mask=mask, scale=1.0 / math.sqrt(self.hd))
144
+ out = out.transpose(1, 2).reshape(B, T, self.nq * self.hd)
145
+ sg = (self.s_gate.float() * self.s_gate_factor).square()
146
+ og = torch.sigmoid(gate.float() * sg) # output gate
147
+ if LRP.on and LRP.gate:
148
+ og = og.detach()
149
+ out = (out.float() * og).to(h.dtype)
150
+ return self.o_proj(justnorm(out))
151
+
152
+
153
+ class DoryMLP(nn.Module):
154
+ def __init__(self, cfg: dict):
155
+ super().__init__()
156
+ d, m = cfg["hidden_size"], cfg["intermediate_size"]
157
+ self.gate_proj = nn.Linear(d, m, bias=False)
158
+ self.up_proj = nn.Linear(d, m, bias=False)
159
+ self.down_proj = nn.Linear(m, d, bias=False)
160
+ self.suv_gate = nn.Parameter(torch.empty(m))
161
+ self.suv_up = nn.Parameter(torch.empty(m))
162
+ self.suv_factor = d**0.5
163
+
164
+ def forward(self, h: torch.Tensor) -> torch.Tensor:
165
+ g = self.gate_proj(h).float() * (self.suv_gate.float() * self.suv_factor)
166
+ u = self.up_proj(h).float() * (self.suv_up.float() * self.suv_factor)
167
+ if LRP.on and (LRP.act or LRP.half):
168
+ a = g * torch.sigmoid(g).detach() if LRP.act else F.silu(g) # identity rule
169
+ prod = 0.5 * (a * u.detach() + a.detach() * u) if LRP.half else a * u # half rule (same forward value)
170
+ else:
171
+ prod = F.silu(g) * u
172
+ return self.down_proj(justnorm(prod.to(h.dtype)))
173
+
174
+
175
+ class DoryLayer(nn.Module):
176
+ """One sublayer (attention or MLP) and the nGPT residual blend."""
177
+
178
+ def __init__(self, cfg: dict, idx: int):
179
+ super().__init__()
180
+ lt = cfg["layer_types"][idx]
181
+ self.is_attention = lt != "mlp"
182
+ self.alpha_scale = cfg["ngpt_alpha_init"] / cfg["initializer_range"]
183
+ self.mixer = DoryAttention(cfg, lt) if self.is_attention else DoryMLP(cfg)
184
+ self.alpha = nn.Parameter(torch.empty(cfg["hidden_size"])) # attn_alpha or mlp_alpha in the file
185
+
186
+ def rate(self) -> torch.Tensor:
187
+ return (self.alpha.float() * self.alpha_scale).abs()
188
+
189
+ def forward(self, h: torch.Tensor, positions: torch.Tensor) -> torch.Tensor:
190
+ out = self.mixer(h, positions) if self.is_attention else self.mixer(h)
191
+ return slerp_norm(h, out, self.rate())
192
+
193
+
194
+ class Dory(nn.Module):
195
+ def __init__(self, cfg: dict):
196
+ super().__init__()
197
+ self.cfg = cfg
198
+ self.n_in, self.n_rec, self.n_out = cfg["n_input_layers"], cfg["n_recurrent_layers"], cfg["n_output_layers"]
199
+ assert self.n_in + self.n_rec + self.n_out == cfg["num_hidden_layers"] == len(cfg["layer_types"])
200
+ self.embeddings = nn.Embedding(cfg["vocab_size"], cfg["hidden_size"])
201
+ self.layers = nn.ModuleList(DoryLayer(cfg, i) for i in range(cfg["num_hidden_layers"]))
202
+ self.lm_head = nn.Linear(cfg["hidden_size"], cfg["vocab_size"], bias=False)
203
+ self.logit_scale = nn.Parameter(torch.empty(cfg["vocab_size"]))
204
+
205
+ # ---- loading -------------------------------------------------------------------------------
206
+ @classmethod
207
+ def from_snapshot(cls, path: str | Path, device="cuda", dtype=torch.float32) -> "Dory":
208
+ path = Path(path)
209
+ cfg = json.loads((path / "config.json").read_text())
210
+ model = cls(cfg)
211
+ own = dict(model.named_parameters())
212
+ used = set()
213
+ with safe_open(str(path / "model.safetensors"), framework="pt") as f:
214
+ keys = list(f.keys())
215
+ for k in keys:
216
+ name = (k.replace("backbone.", "", 1)
217
+ .replace(".attn_alpha", ".alpha").replace(".mlp_alpha", ".alpha")
218
+ .replace("lm_head.logit_scale", "logit_scale"))
219
+ assert name in own, f"checkpoint tensor {k} has no home (mapped to {name})"
220
+ t = f.get_tensor(k)
221
+ assert tuple(t.shape) == tuple(own[name].shape), (k, t.shape, own[name].shape)
222
+ own[name].data = t.to(dtype)
223
+ used.add(name)
224
+ missing = set(own) - used
225
+ assert not missing, f"model parameters with no checkpoint tensor: {sorted(missing)[:5]}"
226
+ model.n_checkpoint_tensors = len(keys)
227
+ return model.to(device).eval().requires_grad_(False)
228
+
229
+ # ---- forward -------------------------------------------------------------------------------
230
+ def unembed(self, h: torch.Tensor) -> torch.Tensor:
231
+ return self.lm_head(h).float() * (self.logit_scale.float() / self.cfg["initializer_range"])
232
+
233
+ def run_output_block(self, h: torch.Tensor, positions: torch.Tensor, tap: Tap = None) -> torch.Tensor:
234
+ for i in range(self.n_in + self.n_rec, self.n_in + self.n_rec + self.n_out):
235
+ h = _tap(tap, ("out", i), self.layers[i](h, positions))
236
+ return h
237
+
238
+ def forward(self, input_ids: torch.Tensor, n_loops: int = 2, tap: Tap = None,
239
+ inputs_embeds: Optional[torch.Tensor] = None) -> torch.Tensor:
240
+ """Returns the final residual state [B, T, d]; call `unembed` for logits."""
241
+ positions = torch.arange(input_ids.shape[1], device=input_ids.device)
242
+ h = self.embeddings(input_ids) if inputs_embeds is None else inputs_embeds
243
+ h = _tap(tap, ("emb",), h)
244
+ for i in range(self.n_in):
245
+ h = _tap(tap, ("in", i), self.layers[i](h, positions))
246
+ x = _tap(tap, ("x",), h)
247
+ state = x
248
+ for t in range(1, n_loops + 1):
249
+ h = _tap(tap, ("u", t), justnorm(state + x))
250
+ for i in range(self.n_in, self.n_in + self.n_rec):
251
+ h = _tap(tap, ("rec", t, i), self.layers[i](h, positions))
252
+ state = _tap(tap, ("s", t), justnorm(h))
253
+ return self.run_output_block(state, positions, tap)
254
+
255
+ def logits(self, input_ids: torch.Tensor, n_loops: int = 2, tap: Tap = None) -> torch.Tensor:
256
+ return self.unembed(self.forward(input_ids, n_loops=n_loops, tap=tap))
257
+
258
+
259
+ def _tap(tap: Tap, name: tuple, h: torch.Tensor) -> torch.Tensor:
260
+ if tap is None:
261
+ return h
262
+ r = tap(name, h)
263
+ return h if r is None else r
264
+
265
+
266
+ def snapshot_path() -> Path:
267
+ """The local snapshot folder in the shared Hugging Face cache (needs HF_HOME set)."""
268
+ from huggingface_hub import snapshot_download
269
+ return Path(snapshot_download("nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16", local_files_only=True,
270
+ allow_patterns=["*.json", "*.jinja", "*.safetensors"]))
271
+
272
+
273
+ def load_tokenizer(path: str | Path):
274
+ """The tokenizer, built from the snapshot's files directly. AutoTokenizer cannot be used: transformers
275
+ 5.17 has no `dory` config class and refuses the checkpoint's config."""
276
+ from transformers import PreTrainedTokenizerFast
277
+ path = Path(path)
278
+ tc = json.loads((path / "tokenizer_config.json").read_text())
279
+ special = {k: (v["content"] if isinstance(v, dict) else v)
280
+ for k, v in tc.items() if k in ("bos_token", "eos_token", "unk_token", "pad_token") and v}
281
+ tok = PreTrainedTokenizerFast(tokenizer_file=str(path / "tokenizer.json"), **special)
282
+ tok.chat_template = (path / "chat_template.jinja").read_text()
283
+ return tok
fit_records/jpp_k1_n64_e8_s0_r2/bootstrap.json ADDED
@@ -0,0 +1,59 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "k": 1,
3
+ "n_test_items": 181,
4
+ "items_per_task": {
5
+ "multihop": 11,
6
+ "multilingual": 22,
7
+ "typo": 48,
8
+ "association": 51,
9
+ "poetry": 49
10
+ },
11
+ "recall10_seven": {
12
+ "logit lens": {
13
+ "point": 0.24431818181818182,
14
+ "lo": 0.1803030303030303,
15
+ "hi": 0.31174242424242427
16
+ },
17
+ "J-lens": {
18
+ "point": 0.3188543599257885,
19
+ "lo": 0.25778447742733457,
20
+ "hi": 0.3856060606060606
21
+ },
22
+ "R-lens": {
23
+ "point": 0.39604205318491037,
24
+ "lo": 0.33013296227581945,
25
+ "hi": 0.4577844774273346
26
+ },
27
+ "J++": {
28
+ "point": 0.465319764269344,
29
+ "lo": 0.39860444177671067,
30
+ "hi": 0.5307873149259704
31
+ }
32
+ },
33
+ "differences": {
34
+ "J++ minus R-lens": {
35
+ "point": 0.06927771108443365,
36
+ "lo": 0.027833860817054135,
37
+ "hi": 0.12239304812834229,
38
+ "share_of_resamples_above_zero": 1.0
39
+ },
40
+ "R-lens minus J-lens": {
41
+ "point": 0.07718769325912189,
42
+ "lo": 0.03787878787878782,
43
+ "hi": 0.1233998144712431,
44
+ "share_of_resamples_above_zero": 1.0
45
+ },
46
+ "J++ minus J-lens": {
47
+ "point": 0.14646540434355554,
48
+ "lo": 0.09046755065662632,
49
+ "hi": 0.207606224307905,
50
+ "share_of_resamples_above_zero": 1.0
51
+ },
52
+ "J-lens minus logit lens": {
53
+ "point": 0.07453617810760665,
54
+ "lo": 0.041960420531849174,
55
+ "hi": 0.10854205318491034,
56
+ "share_of_resamples_above_zero": 1.0
57
+ }
58
+ }
59
+ }
fit_records/jpp_k1_n64_e8_s0_r2/metrics.json ADDED
@@ -0,0 +1,1773 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "n_prompts": 64,
3
+ "seconds_per_prompt": 20.64584706351161,
4
+ "n_nodes": 73,
5
+ "seven_nodes": [
6
+ "in.09",
7
+ "rec.1.18",
8
+ "rec.1.27",
9
+ "rec.1.36",
10
+ "rec.1.45",
11
+ "rec.1.54",
12
+ "out.63"
13
+ ],
14
+ "positions_per_expert": [
15
+ 509.0,
16
+ 1688.0,
17
+ 131.0,
18
+ 1933.0,
19
+ 1353.0,
20
+ 496.0,
21
+ 235.0,
22
+ 823.0
23
+ ],
24
+ "n_dev_pairs": 247,
25
+ "n_test_items": 181,
26
+ "n_test_pairs": 247,
27
+ "rlens_split_half_cosine": {
28
+ "emb": 0.9834,
29
+ "in.00": 0.984,
30
+ "in.01": 0.9884,
31
+ "in.02": 0.9879,
32
+ "in.03": 0.9876,
33
+ "in.04": 0.9885,
34
+ "in.05": 0.9892,
35
+ "in.06": 0.9902,
36
+ "in.07": 0.9903,
37
+ "in.08": 0.9911,
38
+ "in.09": 0.9914,
39
+ "in.10": 0.9911,
40
+ "in.11": 0.9908,
41
+ "in.12": 0.9906,
42
+ "in.13": 0.9897,
43
+ "in.14": 0.9891,
44
+ "x": 0.9874,
45
+ "u.1": 0.9874,
46
+ "rec.1.16": 0.9912,
47
+ "rec.1.17": 0.9921,
48
+ "rec.1.18": 0.9924,
49
+ "rec.1.19": 0.9927,
50
+ "rec.1.20": 0.9929,
51
+ "rec.1.21": 0.993,
52
+ "rec.1.22": 0.994,
53
+ "rec.1.23": 0.9939,
54
+ "rec.1.24": 0.9939,
55
+ "rec.1.25": 0.9937,
56
+ "rec.1.26": 0.9937,
57
+ "rec.1.27": 0.9936,
58
+ "rec.1.28": 0.9937,
59
+ "rec.1.29": 0.9937,
60
+ "rec.1.30": 0.994,
61
+ "rec.1.31": 0.9939,
62
+ "rec.1.32": 0.994,
63
+ "rec.1.33": 0.9938,
64
+ "rec.1.34": 0.9941,
65
+ "rec.1.35": 0.9939,
66
+ "rec.1.36": 0.9948,
67
+ "rec.1.37": 0.9949,
68
+ "rec.1.38": 0.9951,
69
+ "rec.1.39": 0.9953,
70
+ "rec.1.40": 0.9955,
71
+ "rec.1.41": 0.9959,
72
+ "rec.1.42": 0.9959,
73
+ "rec.1.43": 0.9963,
74
+ "rec.1.44": 0.9963,
75
+ "rec.1.45": 0.9966,
76
+ "rec.1.46": 0.9966,
77
+ "rec.1.47": 0.9969,
78
+ "rec.1.48": 0.9967,
79
+ "rec.1.49": 0.9971,
80
+ "rec.1.50": 0.9972,
81
+ "rec.1.51": 0.9974,
82
+ "rec.1.52": 0.9976,
83
+ "rec.1.53": 0.9978,
84
+ "rec.1.54": 0.998,
85
+ "s.1": 0.9982,
86
+ "out.56": 0.9982,
87
+ "out.57": 0.9984,
88
+ "out.58": 0.9985,
89
+ "out.59": 0.9988,
90
+ "out.60": 0.9989,
91
+ "out.61": 0.9991,
92
+ "out.62": 0.9992,
93
+ "out.63": 0.9994,
94
+ "out.64": 0.9994,
95
+ "out.65": 0.9996,
96
+ "out.66": 0.9996,
97
+ "out.67": 0.9998,
98
+ "out.68": 0.9998,
99
+ "out.69": 0.9999,
100
+ "out.70": 0.9999
101
+ },
102
+ "weights": {
103
+ "emb": [
104
+ 0.0118,
105
+ -0.0275,
106
+ 0.0942,
107
+ 0.1199,
108
+ 0.2122,
109
+ -0.004,
110
+ 0.091,
111
+ -0.9605
112
+ ],
113
+ "in.00": [
114
+ -0.0119,
115
+ -0.1259,
116
+ 0.04,
117
+ 0.1856,
118
+ 0.3156,
119
+ -0.0234,
120
+ 0.2456,
121
+ -0.8874
122
+ ],
123
+ "in.01": [
124
+ -0.3284,
125
+ -0.0822,
126
+ 0.0572,
127
+ 0.1817,
128
+ 0.4926,
129
+ -0.0356,
130
+ 0.1851,
131
+ -0.7556
132
+ ],
133
+ "in.02": [
134
+ -0.0952,
135
+ 0.0074,
136
+ 0.0673,
137
+ 0.1834,
138
+ 0.29,
139
+ -0.0619,
140
+ 0.1855,
141
+ -0.9113
142
+ ],
143
+ "in.03": [
144
+ -0.1388,
145
+ 0.0066,
146
+ 0.0772,
147
+ 0.1993,
148
+ 0.4121,
149
+ -0.0641,
150
+ 0.1288,
151
+ -0.8628
152
+ ],
153
+ "in.04": [
154
+ 0.1336,
155
+ -0.0674,
156
+ 0.0761,
157
+ 0.2362,
158
+ 0.4591,
159
+ -0.071,
160
+ 0.1336,
161
+ -0.826
162
+ ],
163
+ "in.05": [
164
+ 0.6695,
165
+ 0.0561,
166
+ 0.1494,
167
+ 0.4061,
168
+ 0.379,
169
+ -0.0905,
170
+ -0.1097,
171
+ -0.4445
172
+ ],
173
+ "in.06": [
174
+ 0.6872,
175
+ -0.1138,
176
+ 0.0619,
177
+ 0.3767,
178
+ 0.5223,
179
+ 0.0121,
180
+ -0.1154,
181
+ -0.2878
182
+ ],
183
+ "in.07": [
184
+ 0.3178,
185
+ -0.163,
186
+ 0.0706,
187
+ 0.4536,
188
+ 0.7934,
189
+ 0.0203,
190
+ -0.1078,
191
+ -0.1423
192
+ ],
193
+ "in.08": [
194
+ 0.077,
195
+ -0.1401,
196
+ 0.0624,
197
+ 0.4983,
198
+ 0.8196,
199
+ 0.049,
200
+ -0.1135,
201
+ -0.188
202
+ ],
203
+ "in.09": [
204
+ -0.0791,
205
+ -0.0619,
206
+ 0.1051,
207
+ 0.5802,
208
+ 0.7662,
209
+ 0.0205,
210
+ -0.2191,
211
+ 0.0821
212
+ ],
213
+ "in.10": [
214
+ 0.1086,
215
+ -0.0462,
216
+ 0.1045,
217
+ 0.5446,
218
+ 0.626,
219
+ 0.0216,
220
+ -0.298,
221
+ 0.4444
222
+ ],
223
+ "in.11": [
224
+ 0.251,
225
+ -0.0326,
226
+ 0.0943,
227
+ 0.5144,
228
+ 0.5016,
229
+ 0.0293,
230
+ -0.3103,
231
+ 0.5601
232
+ ],
233
+ "in.12": [
234
+ 0.4894,
235
+ -0.1731,
236
+ 0.0863,
237
+ 0.5286,
238
+ 0.519,
239
+ 0.0442,
240
+ -0.2893,
241
+ 0.2977
242
+ ],
243
+ "in.13": [
244
+ 0.7228,
245
+ -0.1956,
246
+ 0.0682,
247
+ 0.5061,
248
+ 0.2912,
249
+ 0.0585,
250
+ -0.2776,
251
+ 0.115
252
+ ],
253
+ "in.14": [
254
+ 0.609,
255
+ -0.3487,
256
+ 0.0091,
257
+ 0.3924,
258
+ 0.4804,
259
+ 0.0796,
260
+ -0.1019,
261
+ -0.3255
262
+ ],
263
+ "x": [
264
+ 0.4651,
265
+ -0.3153,
266
+ 0.0256,
267
+ 0.6965,
268
+ 0.1971,
269
+ 0.117,
270
+ -0.1956,
271
+ -0.3282
272
+ ],
273
+ "u.1": [
274
+ 0.4651,
275
+ -0.3153,
276
+ 0.0256,
277
+ 0.6965,
278
+ 0.1971,
279
+ 0.117,
280
+ -0.1956,
281
+ -0.3282
282
+ ],
283
+ "rec.1.16": [
284
+ 0.8371,
285
+ -0.2285,
286
+ 0.0657,
287
+ 0.2973,
288
+ 0.1826,
289
+ 0.0625,
290
+ -0.1643,
291
+ -0.3002
292
+ ],
293
+ "rec.1.17": [
294
+ 0.804,
295
+ -0.1915,
296
+ 0.0754,
297
+ 0.313,
298
+ 0.1475,
299
+ 0.0728,
300
+ -0.1888,
301
+ -0.3881
302
+ ],
303
+ "rec.1.18": [
304
+ 0.7946,
305
+ -0.1763,
306
+ 0.0787,
307
+ 0.3127,
308
+ 0.1469,
309
+ 0.0711,
310
+ -0.1769,
311
+ -0.4191
312
+ ],
313
+ "rec.1.19": [
314
+ 0.8139,
315
+ -0.1884,
316
+ 0.0786,
317
+ 0.2458,
318
+ 0.1446,
319
+ 0.0796,
320
+ -0.1439,
321
+ -0.4331
322
+ ],
323
+ "rec.1.20": [
324
+ 0.7458,
325
+ -0.2203,
326
+ 0.067,
327
+ 0.2798,
328
+ 0.2381,
329
+ 0.0858,
330
+ -0.1503,
331
+ -0.4752
332
+ ],
333
+ "rec.1.21": [
334
+ 0.7587,
335
+ -0.2616,
336
+ 0.0847,
337
+ 0.2859,
338
+ 0.3008,
339
+ 0.0793,
340
+ -0.1586,
341
+ -0.3809
342
+ ],
343
+ "rec.1.22": [
344
+ 0.6023,
345
+ -0.4176,
346
+ 0.053,
347
+ 0.327,
348
+ 0.4647,
349
+ 0.0799,
350
+ -0.1332,
351
+ -0.3363
352
+ ],
353
+ "rec.1.23": [
354
+ 0.5203,
355
+ -0.4631,
356
+ 0.0641,
357
+ 0.3205,
358
+ 0.5297,
359
+ 0.0835,
360
+ -0.1509,
361
+ -0.3124
362
+ ],
363
+ "rec.1.24": [
364
+ 0.5854,
365
+ -0.4451,
366
+ 0.0643,
367
+ 0.3098,
368
+ 0.4819,
369
+ 0.0805,
370
+ -0.1483,
371
+ -0.3137
372
+ ],
373
+ "rec.1.25": [
374
+ 0.5383,
375
+ -0.4749,
376
+ 0.0806,
377
+ 0.3246,
378
+ 0.5227,
379
+ 0.0751,
380
+ -0.1674,
381
+ -0.2568
382
+ ],
383
+ "rec.1.26": [
384
+ 0.6258,
385
+ -0.4544,
386
+ 0.0893,
387
+ 0.3007,
388
+ 0.4519,
389
+ 0.0622,
390
+ -0.1592,
391
+ -0.2648
392
+ ],
393
+ "rec.1.27": [
394
+ 0.6476,
395
+ -0.4593,
396
+ 0.091,
397
+ 0.2788,
398
+ 0.4718,
399
+ 0.0548,
400
+ -0.1778,
401
+ -0.1628
402
+ ],
403
+ "rec.1.28": [
404
+ 0.5335,
405
+ -0.4826,
406
+ 0.0558,
407
+ 0.2733,
408
+ 0.5513,
409
+ 0.0603,
410
+ -0.1436,
411
+ -0.2766
412
+ ],
413
+ "rec.1.29": [
414
+ 0.5407,
415
+ -0.5243,
416
+ 0.0567,
417
+ 0.2866,
418
+ 0.5047,
419
+ 0.0583,
420
+ -0.1397,
421
+ -0.2642
422
+ ],
423
+ "rec.1.30": [
424
+ 0.4891,
425
+ -0.5035,
426
+ 0.0554,
427
+ 0.2427,
428
+ 0.6434,
429
+ 0.0424,
430
+ -0.139,
431
+ -0.1015
432
+ ],
433
+ "rec.1.31": [
434
+ 0.458,
435
+ -0.5283,
436
+ 0.0531,
437
+ 0.2387,
438
+ 0.6442,
439
+ 0.0576,
440
+ -0.1662,
441
+ -0.0742
442
+ ],
443
+ "rec.1.32": [
444
+ 0.4925,
445
+ -0.4891,
446
+ 0.0465,
447
+ 0.2269,
448
+ 0.6568,
449
+ 0.0589,
450
+ -0.168,
451
+ -0.0397
452
+ ],
453
+ "rec.1.33": [
454
+ 0.4799,
455
+ -0.5228,
456
+ 0.0398,
457
+ 0.2583,
458
+ 0.6312,
459
+ 0.0577,
460
+ -0.1603,
461
+ -0.0259
462
+ ],
463
+ "rec.1.34": [
464
+ 0.5368,
465
+ -0.4674,
466
+ 0.0411,
467
+ 0.2413,
468
+ 0.6371,
469
+ 0.0549,
470
+ -0.157,
471
+ -0.0001
472
+ ],
473
+ "rec.1.35": [
474
+ 0.601,
475
+ -0.4382,
476
+ 0.0342,
477
+ 0.2337,
478
+ 0.5902,
479
+ 0.0852,
480
+ -0.1842,
481
+ 0.038
482
+ ],
483
+ "rec.1.36": [
484
+ 0.5883,
485
+ -0.4241,
486
+ 0.0374,
487
+ 0.208,
488
+ 0.6145,
489
+ 0.0776,
490
+ -0.1851,
491
+ 0.1068
492
+ ],
493
+ "rec.1.37": [
494
+ 0.5688,
495
+ -0.4752,
496
+ 0.0354,
497
+ 0.1774,
498
+ 0.6071,
499
+ 0.0824,
500
+ -0.1672,
501
+ 0.1209
502
+ ],
503
+ "rec.1.38": [
504
+ 0.4808,
505
+ -0.4455,
506
+ 0.0634,
507
+ 0.1957,
508
+ 0.7147,
509
+ 0.0344,
510
+ -0.1192,
511
+ 0.0429
512
+ ],
513
+ "rec.1.39": [
514
+ 0.5001,
515
+ -0.4727,
516
+ 0.0732,
517
+ 0.2655,
518
+ 0.6508,
519
+ 0.0478,
520
+ -0.1327,
521
+ 0.0851
522
+ ],
523
+ "rec.1.40": [
524
+ 0.3616,
525
+ -0.5138,
526
+ 0.0807,
527
+ 0.4835,
528
+ 0.5865,
529
+ 0.0498,
530
+ -0.1169,
531
+ -0.0696
532
+ ],
533
+ "rec.1.41": [
534
+ 0.3943,
535
+ -0.4843,
536
+ 0.1072,
537
+ 0.4328,
538
+ 0.6225,
539
+ 0.0558,
540
+ -0.1247,
541
+ -0.0702
542
+ ],
543
+ "rec.1.42": [
544
+ 0.4523,
545
+ -0.4374,
546
+ 0.0958,
547
+ 0.4374,
548
+ 0.5955,
549
+ 0.0707,
550
+ -0.1599,
551
+ -0.1356
552
+ ],
553
+ "rec.1.43": [
554
+ 0.4328,
555
+ -0.4369,
556
+ 0.1106,
557
+ 0.4406,
558
+ 0.605,
559
+ 0.0718,
560
+ -0.1644,
561
+ -0.1314
562
+ ],
563
+ "rec.1.44": [
564
+ 0.516,
565
+ -0.4267,
566
+ 0.0903,
567
+ 0.4301,
568
+ 0.5523,
569
+ 0.0821,
570
+ -0.1802,
571
+ -0.1199
572
+ ],
573
+ "rec.1.45": [
574
+ 0.5225,
575
+ -0.4415,
576
+ 0.0955,
577
+ 0.4586,
578
+ 0.5035,
579
+ 0.069,
580
+ -0.1664,
581
+ -0.1632
582
+ ],
583
+ "rec.1.46": [
584
+ 0.5596,
585
+ -0.5187,
586
+ 0.0829,
587
+ 0.3179,
588
+ 0.5312,
589
+ 0.0931,
590
+ -0.1374,
591
+ 0.003
592
+ ],
593
+ "rec.1.47": [
594
+ 0.5508,
595
+ -0.4863,
596
+ 0.1207,
597
+ 0.3771,
598
+ 0.5223,
599
+ 0.0915,
600
+ -0.1478,
601
+ -0.0202
602
+ ],
603
+ "rec.1.48": [
604
+ 0.4385,
605
+ -0.6335,
606
+ 0.0993,
607
+ 0.3641,
608
+ 0.478,
609
+ 0.1399,
610
+ -0.0315,
611
+ -0.1219
612
+ ],
613
+ "rec.1.49": [
614
+ 0.3306,
615
+ -0.6566,
616
+ 0.065,
617
+ 0.39,
618
+ 0.4616,
619
+ 0.1207,
620
+ 0.0474,
621
+ -0.2709
622
+ ],
623
+ "rec.1.50": [
624
+ 0.6652,
625
+ -0.604,
626
+ 0.1126,
627
+ 0.2149,
628
+ 0.2388,
629
+ 0.1358,
630
+ 0.0698,
631
+ -0.2311
632
+ ],
633
+ "rec.1.51": [
634
+ 0.4914,
635
+ -0.7538,
636
+ 0.0467,
637
+ 0.0584,
638
+ 0.3994,
639
+ 0.1382,
640
+ -0.0261,
641
+ 0.0731
642
+ ],
643
+ "rec.1.52": [
644
+ 0.4632,
645
+ -0.7535,
646
+ 0.0273,
647
+ 0.0881,
648
+ 0.4378,
649
+ 0.1316,
650
+ -0.007,
651
+ 0.012
652
+ ],
653
+ "rec.1.53": [
654
+ 0.4444,
655
+ -0.7388,
656
+ 0.0276,
657
+ 0.0242,
658
+ 0.4909,
659
+ 0.0815,
660
+ -0.0425,
661
+ 0.0771
662
+ ],
663
+ "rec.1.54": [
664
+ 0.5122,
665
+ -0.7653,
666
+ -0.0189,
667
+ 0.0347,
668
+ 0.3479,
669
+ 0.1674,
670
+ -0.0189,
671
+ 0.031
672
+ ],
673
+ "s.1": [
674
+ 0.473,
675
+ -0.7917,
676
+ -0.0555,
677
+ 0.2088,
678
+ 0.2374,
679
+ 0.1165,
680
+ -0.0524,
681
+ 0.1734
682
+ ],
683
+ "out.56": [
684
+ 0.4466,
685
+ -0.8001,
686
+ 0.008,
687
+ -0.0408,
688
+ 0.3442,
689
+ 0.146,
690
+ -0.0219,
691
+ 0.1357
692
+ ],
693
+ "out.57": [
694
+ 0.5862,
695
+ -0.718,
696
+ 0.0094,
697
+ -0.2274,
698
+ 0.199,
699
+ 0.1518,
700
+ -0.0087,
701
+ 0.1622
702
+ ],
703
+ "out.58": [
704
+ 0.4015,
705
+ -0.8097,
706
+ -0.0134,
707
+ -0.0816,
708
+ 0.3644,
709
+ 0.1407,
710
+ -0.0216,
711
+ 0.1528
712
+ ],
713
+ "out.59": [
714
+ 0.4171,
715
+ -0.8216,
716
+ -0.0132,
717
+ -0.0794,
718
+ 0.352,
719
+ 0.1235,
720
+ 0.0407,
721
+ 0.0603
722
+ ],
723
+ "out.60": [
724
+ 0.4743,
725
+ -0.2017,
726
+ -0.0352,
727
+ -0.3491,
728
+ 0.5492,
729
+ 0.168,
730
+ -0.02,
731
+ -0.5302
732
+ ],
733
+ "out.61": [
734
+ 0.3688,
735
+ -0.1325,
736
+ -0.0056,
737
+ -0.3239,
738
+ 0.6522,
739
+ 0.1089,
740
+ 0.0351,
741
+ -0.5504
742
+ ],
743
+ "out.62": [
744
+ 0.1938,
745
+ -0.1879,
746
+ 0.2843,
747
+ -0.8423,
748
+ 0.3508,
749
+ 0.0765,
750
+ 0.0839,
751
+ -0.028
752
+ ],
753
+ "out.63": [
754
+ -0.0697,
755
+ -0.3262,
756
+ 0.3325,
757
+ -0.7131,
758
+ 0.4577,
759
+ 0.2201,
760
+ 0.057,
761
+ 0.0918
762
+ ],
763
+ "out.64": [
764
+ -0.2015,
765
+ 0.1697,
766
+ 0.1213,
767
+ -0.8303,
768
+ 0.3663,
769
+ 0.1137,
770
+ 0.1099,
771
+ 0.2595
772
+ ],
773
+ "out.65": [
774
+ 0.3952,
775
+ 0.0539,
776
+ 0.0829,
777
+ -0.841,
778
+ 0.3247,
779
+ 0.0589,
780
+ 0.0725,
781
+ -0.1124
782
+ ],
783
+ "out.66": [
784
+ 0.5139,
785
+ 0.3821,
786
+ 0.0831,
787
+ -0.6778,
788
+ 0.0258,
789
+ 0.1521,
790
+ -0.1446,
791
+ -0.281
792
+ ],
793
+ "out.67": [
794
+ 0.7194,
795
+ 0.3443,
796
+ 0.0695,
797
+ -0.3841,
798
+ -0.1379,
799
+ 0.1167,
800
+ -0.31,
801
+ -0.2878
802
+ ],
803
+ "out.68": [
804
+ 0.7616,
805
+ -0.0498,
806
+ -0.0146,
807
+ -0.1498,
808
+ -0.0486,
809
+ 0.2745,
810
+ -0.5515,
811
+ -0.114
812
+ ],
813
+ "out.69": [
814
+ 0.7504,
815
+ 0.189,
816
+ -0.0668,
817
+ -0.3122,
818
+ 0.1857,
819
+ 0.2444,
820
+ -0.3397,
821
+ -0.2993
822
+ ],
823
+ "out.70": [
824
+ 0.0101,
825
+ -0.0578,
826
+ 0.2731,
827
+ -0.6258,
828
+ -0.4067,
829
+ 0.158,
830
+ 0.5093,
831
+ 0.2839
832
+ ]
833
+ },
834
+ "weights_shuffled_labels": {
835
+ "emb": [
836
+ 0.0497,
837
+ -0.2571,
838
+ -0.12,
839
+ 0.1697,
840
+ 0.5601,
841
+ 0.0261,
842
+ 0.1878,
843
+ -0.7339
844
+ ],
845
+ "in.00": [
846
+ 0.496,
847
+ -0.142,
848
+ -0.089,
849
+ 0.1937,
850
+ 0.0537,
851
+ 0.0172,
852
+ 0.2231,
853
+ -0.7971
854
+ ],
855
+ "in.01": [
856
+ 0.0224,
857
+ -0.1956,
858
+ -0.109,
859
+ 0.2651,
860
+ 0.4562,
861
+ -0.0093,
862
+ 0.2669,
863
+ -0.7743
864
+ ],
865
+ "in.02": [
866
+ 0.5329,
867
+ -0.014,
868
+ -0.0821,
869
+ 0.2288,
870
+ 0.0778,
871
+ 0.026,
872
+ 0.1352,
873
+ -0.7948
874
+ ],
875
+ "in.03": [
876
+ 0.2442,
877
+ 0.1266,
878
+ 0.0583,
879
+ 0.1786,
880
+ -0.0565,
881
+ -0.0032,
882
+ 0.1109,
883
+ -0.9346
884
+ ],
885
+ "in.04": [
886
+ 0.8103,
887
+ -0.0061,
888
+ -0.0685,
889
+ 0.2195,
890
+ -0.0736,
891
+ 0.0148,
892
+ 0.0686,
893
+ -0.5293
894
+ ],
895
+ "in.05": [
896
+ 0.8913,
897
+ 0.0989,
898
+ -0.0563,
899
+ 0.084,
900
+ -0.3849,
901
+ 0.0049,
902
+ 0.0114,
903
+ 0.193
904
+ ],
905
+ "in.06": [
906
+ 0.9294,
907
+ 0.0076,
908
+ -0.0573,
909
+ 0.2445,
910
+ -0.0164,
911
+ 0.0608,
912
+ -0.0595,
913
+ -0.256
914
+ ],
915
+ "in.07": [
916
+ 0.909,
917
+ 0.0595,
918
+ -0.0837,
919
+ 0.3059,
920
+ -0.043,
921
+ 0.0802,
922
+ -0.0495,
923
+ -0.2426
924
+ ],
925
+ "in.08": [
926
+ 0.5422,
927
+ 0.076,
928
+ -0.1286,
929
+ 0.5722,
930
+ 0.4898,
931
+ 0.153,
932
+ -0.0925,
933
+ -0.2905
934
+ ],
935
+ "in.09": [
936
+ 0.1496,
937
+ 0.2204,
938
+ -0.0463,
939
+ 0.5344,
940
+ 0.3305,
941
+ 0.1281,
942
+ -0.2389,
943
+ 0.6772
944
+ ],
945
+ "in.10": [
946
+ 0.1191,
947
+ 0.1277,
948
+ -0.1905,
949
+ 0.3472,
950
+ 0.253,
951
+ 0.1155,
952
+ -0.1656,
953
+ 0.8413
954
+ ],
955
+ "in.11": [
956
+ 0.2196,
957
+ 0.1792,
958
+ -0.1691,
959
+ 0.2189,
960
+ 0.0159,
961
+ 0.0984,
962
+ -0.1245,
963
+ 0.9043
964
+ ],
965
+ "in.12": [
966
+ 0.327,
967
+ 0.3988,
968
+ 0.0561,
969
+ 0.0962,
970
+ -0.2858,
971
+ 0.059,
972
+ -0.1935,
973
+ 0.774
974
+ ],
975
+ "in.13": [
976
+ 0.2914,
977
+ 0.0621,
978
+ -0.1489,
979
+ 0.2528,
980
+ -0.0136,
981
+ 0.1057,
982
+ -0.1701,
983
+ 0.8859
984
+ ],
985
+ "in.14": [
986
+ 0.824,
987
+ -0.1505,
988
+ -0.176,
989
+ 0.3371,
990
+ 0.1318,
991
+ 0.1773,
992
+ 0.0006,
993
+ -0.3241
994
+ ],
995
+ "x": [
996
+ 0.6632,
997
+ 0.1033,
998
+ -0.1874,
999
+ 0.4285,
1000
+ -0.338,
1001
+ 0.2332,
1002
+ -0.0212,
1003
+ -0.4021
1004
+ ],
1005
+ "u.1": [
1006
+ 0.6632,
1007
+ 0.1033,
1008
+ -0.1874,
1009
+ 0.4284,
1010
+ -0.3381,
1011
+ 0.2332,
1012
+ -0.0212,
1013
+ -0.4021
1014
+ ],
1015
+ "rec.1.16": [
1016
+ 0.7552,
1017
+ 0.0764,
1018
+ -0.0954,
1019
+ -0.197,
1020
+ -0.1441,
1021
+ 0.0715,
1022
+ -0.0796,
1023
+ 0.5863
1024
+ ],
1025
+ "rec.1.17": [
1026
+ 0.8413,
1027
+ -0.0112,
1028
+ -0.1411,
1029
+ -0.1382,
1030
+ -0.16,
1031
+ 0.0894,
1032
+ -0.0663,
1033
+ 0.4638
1034
+ ],
1035
+ "rec.1.18": [
1036
+ 0.7904,
1037
+ 0.069,
1038
+ -0.1274,
1039
+ -0.1807,
1040
+ -0.2206,
1041
+ 0.08,
1042
+ -0.0584,
1043
+ 0.513
1044
+ ],
1045
+ "rec.1.19": [
1046
+ 0.8455,
1047
+ -0.005,
1048
+ -0.1446,
1049
+ -0.1714,
1050
+ -0.2055,
1051
+ 0.0872,
1052
+ -0.0058,
1053
+ 0.43
1054
+ ],
1055
+ "rec.1.20": [
1056
+ 0.8355,
1057
+ -0.0728,
1058
+ -0.1627,
1059
+ -0.1564,
1060
+ -0.0906,
1061
+ 0.1016,
1062
+ -0.0232,
1063
+ 0.4761
1064
+ ],
1065
+ "rec.1.21": [
1066
+ 0.8272,
1067
+ -0.0963,
1068
+ -0.1601,
1069
+ -0.1374,
1070
+ -0.0656,
1071
+ 0.0873,
1072
+ -0.02,
1073
+ 0.4997
1074
+ ],
1075
+ "rec.1.22": [
1076
+ 0.9241,
1077
+ 0.0399,
1078
+ 0.0649,
1079
+ -0.2712,
1080
+ -0.2018,
1081
+ 0.0694,
1082
+ -0.0593,
1083
+ 0.1326
1084
+ ],
1085
+ "rec.1.23": [
1086
+ 0.7681,
1087
+ -0.4228,
1088
+ -0.218,
1089
+ -0.0208,
1090
+ 0.2454,
1091
+ 0.115,
1092
+ -0.013,
1093
+ 0.3312
1094
+ ],
1095
+ "rec.1.24": [
1096
+ 0.9404,
1097
+ -0.3042,
1098
+ 0.0837,
1099
+ -0.0684,
1100
+ 0.0309,
1101
+ 0.0769,
1102
+ -0.0572,
1103
+ -0.0368
1104
+ ],
1105
+ "rec.1.25": [
1106
+ 0.8777,
1107
+ -0.433,
1108
+ 0.0941,
1109
+ 0.0044,
1110
+ 0.1221,
1111
+ 0.0758,
1112
+ -0.0377,
1113
+ -0.1059
1114
+ ],
1115
+ "rec.1.26": [
1116
+ 0.8899,
1117
+ -0.4195,
1118
+ 0.084,
1119
+ -0.0069,
1120
+ 0.0905,
1121
+ 0.0632,
1122
+ -0.0329,
1123
+ -0.1084
1124
+ ],
1125
+ "rec.1.27": [
1126
+ 0.9159,
1127
+ -0.3713,
1128
+ 0.0883,
1129
+ -0.0328,
1130
+ 0.0632,
1131
+ 0.0625,
1132
+ -0.0614,
1133
+ -0.0514
1134
+ ],
1135
+ "rec.1.28": [
1136
+ 0.8173,
1137
+ -0.4437,
1138
+ 0.0677,
1139
+ -0.0893,
1140
+ 0.2949,
1141
+ 0.0759,
1142
+ -0.0543,
1143
+ -0.164
1144
+ ],
1145
+ "rec.1.29": [
1146
+ 0.9363,
1147
+ -0.1796,
1148
+ 0.0877,
1149
+ -0.2362,
1150
+ 0.0715,
1151
+ 0.0945,
1152
+ -0.1157,
1153
+ -0.0142
1154
+ ],
1155
+ "rec.1.30": [
1156
+ 0.7028,
1157
+ -0.4881,
1158
+ -0.1097,
1159
+ -0.1218,
1160
+ 0.3837,
1161
+ 0.0873,
1162
+ 0.0558,
1163
+ 0.288
1164
+ ],
1165
+ "rec.1.31": [
1166
+ 0.6783,
1167
+ -0.5322,
1168
+ -0.1223,
1169
+ -0.1127,
1170
+ 0.3789,
1171
+ 0.0896,
1172
+ 0.0856,
1173
+ 0.2647
1174
+ ],
1175
+ "rec.1.32": [
1176
+ 0.7203,
1177
+ -0.3709,
1178
+ -0.095,
1179
+ -0.2815,
1180
+ 0.353,
1181
+ 0.0929,
1182
+ 0.0761,
1183
+ 0.3412
1184
+ ],
1185
+ "rec.1.33": [
1186
+ 0.7454,
1187
+ -0.339,
1188
+ -0.1044,
1189
+ -0.3241,
1190
+ 0.2693,
1191
+ 0.0965,
1192
+ 0.0856,
1193
+ 0.3525
1194
+ ],
1195
+ "rec.1.34": [
1196
+ 0.7382,
1197
+ -0.2921,
1198
+ -0.0855,
1199
+ -0.3596,
1200
+ 0.3212,
1201
+ 0.0983,
1202
+ 0.0758,
1203
+ 0.3383
1204
+ ],
1205
+ "rec.1.35": [
1206
+ 0.8117,
1207
+ -0.1975,
1208
+ -0.0866,
1209
+ -0.3546,
1210
+ 0.1452,
1211
+ 0.1335,
1212
+ 0.0111,
1213
+ 0.3603
1214
+ ],
1215
+ "rec.1.36": [
1216
+ 0.7065,
1217
+ 0.0468,
1218
+ -0.0646,
1219
+ -0.5688,
1220
+ 0.0479,
1221
+ 0.1575,
1222
+ -0.0536,
1223
+ 0.3755
1224
+ ],
1225
+ "rec.1.37": [
1226
+ 0.7368,
1227
+ -0.0103,
1228
+ -0.096,
1229
+ -0.5132,
1230
+ -0.0439,
1231
+ 0.1534,
1232
+ -0.0082,
1233
+ 0.3986
1234
+ ],
1235
+ "rec.1.38": [
1236
+ 0.7296,
1237
+ 0.0698,
1238
+ -0.1145,
1239
+ -0.5099,
1240
+ -0.1029,
1241
+ 0.1509,
1242
+ 0.0018,
1243
+ 0.3953
1244
+ ],
1245
+ "rec.1.39": [
1246
+ 0.7124,
1247
+ 0.2035,
1248
+ -0.0826,
1249
+ -0.4527,
1250
+ -0.3259,
1251
+ 0.1502,
1252
+ -0.0265,
1253
+ 0.3315
1254
+ ],
1255
+ "rec.1.40": [
1256
+ 0.4417,
1257
+ 0.5521,
1258
+ 0.041,
1259
+ -0.4374,
1260
+ -0.5262,
1261
+ 0.085,
1262
+ -0.0529,
1263
+ 0.1421
1264
+ ],
1265
+ "rec.1.41": [
1266
+ 0.407,
1267
+ 0.5653,
1268
+ 0.0436,
1269
+ -0.4606,
1270
+ -0.526,
1271
+ 0.0756,
1272
+ -0.0212,
1273
+ 0.1335
1274
+ ],
1275
+ "rec.1.42": [
1276
+ 0.3767,
1277
+ 0.6006,
1278
+ 0.0591,
1279
+ -0.51,
1280
+ -0.4522,
1281
+ 0.0745,
1282
+ -0.0383,
1283
+ 0.1493
1284
+ ],
1285
+ "rec.1.43": [
1286
+ 0.3835,
1287
+ 0.5895,
1288
+ 0.0699,
1289
+ -0.4796,
1290
+ -0.4984,
1291
+ 0.0758,
1292
+ -0.0189,
1293
+ 0.1263
1294
+ ],
1295
+ "rec.1.44": [
1296
+ 0.4112,
1297
+ 0.5803,
1298
+ 0.0675,
1299
+ -0.5343,
1300
+ -0.4107,
1301
+ 0.0859,
1302
+ -0.0536,
1303
+ 0.1589
1304
+ ],
1305
+ "rec.1.45": [
1306
+ 0.3838,
1307
+ 0.5874,
1308
+ 0.0566,
1309
+ -0.4866,
1310
+ -0.4946,
1311
+ 0.0692,
1312
+ -0.0196,
1313
+ 0.1338
1314
+ ],
1315
+ "rec.1.46": [
1316
+ 0.2063,
1317
+ 0.6423,
1318
+ 0.0437,
1319
+ -0.5804,
1320
+ -0.4066,
1321
+ -0.0237,
1322
+ 0.1364,
1323
+ 0.1469
1324
+ ],
1325
+ "rec.1.47": [
1326
+ 0.182,
1327
+ 0.651,
1328
+ 0.0378,
1329
+ -0.5472,
1330
+ -0.444,
1331
+ -0.0427,
1332
+ 0.1504,
1333
+ 0.144
1334
+ ],
1335
+ "rec.1.48": [
1336
+ 0.2067,
1337
+ 0.6315,
1338
+ 0.0454,
1339
+ -0.5658,
1340
+ -0.4498,
1341
+ 0.006,
1342
+ 0.1189,
1343
+ 0.1408
1344
+ ],
1345
+ "rec.1.49": [
1346
+ 0.1734,
1347
+ 0.6373,
1348
+ 0.0363,
1349
+ -0.5702,
1350
+ -0.4561,
1351
+ 0.013,
1352
+ 0.1082,
1353
+ 0.1322
1354
+ ],
1355
+ "rec.1.50": [
1356
+ 0.6352,
1357
+ -0.0612,
1358
+ 0.1155,
1359
+ -0.687,
1360
+ -0.1216,
1361
+ 0.1816,
1362
+ 0.1846,
1363
+ 0.1602
1364
+ ],
1365
+ "rec.1.51": [
1366
+ 0.6459,
1367
+ -0.2203,
1368
+ 0.1355,
1369
+ -0.6087,
1370
+ -0.0746,
1371
+ 0.2478,
1372
+ 0.2727,
1373
+ 0.0629
1374
+ ],
1375
+ "rec.1.52": [
1376
+ 0.6237,
1377
+ -0.1675,
1378
+ -0.0156,
1379
+ -0.3021,
1380
+ 0.0394,
1381
+ 0.2867,
1382
+ 0.3556,
1383
+ -0.5303
1384
+ ],
1385
+ "rec.1.53": [
1386
+ 0.4509,
1387
+ 0.3897,
1388
+ -0.0353,
1389
+ -0.5129,
1390
+ 0.0665,
1391
+ 0.0941,
1392
+ 0.2207,
1393
+ -0.5644
1394
+ ],
1395
+ "rec.1.54": [
1396
+ 0.3577,
1397
+ 0.0127,
1398
+ -0.2097,
1399
+ -0.6816,
1400
+ 0.281,
1401
+ 0.2625,
1402
+ 0.3377,
1403
+ -0.3185
1404
+ ],
1405
+ "s.1": [
1406
+ 0.0157,
1407
+ 0.0208,
1408
+ 0.0626,
1409
+ -0.9013,
1410
+ 0.2632,
1411
+ 0.0835,
1412
+ 0.2133,
1413
+ 0.2475
1414
+ ],
1415
+ "out.56": [
1416
+ -0.3768,
1417
+ -0.1347,
1418
+ 0.157,
1419
+ -0.6898,
1420
+ 0.2342,
1421
+ 0.1381,
1422
+ 0.4609,
1423
+ 0.2304
1424
+ ],
1425
+ "out.57": [
1426
+ -0.5109,
1427
+ -0.3087,
1428
+ 0.1849,
1429
+ -0.411,
1430
+ -0.0477,
1431
+ 0.1975,
1432
+ 0.5208,
1433
+ 0.358
1434
+ ],
1435
+ "out.58": [
1436
+ -0.5757,
1437
+ -0.2338,
1438
+ 0.1841,
1439
+ -0.396,
1440
+ 0.2493,
1441
+ -0.037,
1442
+ 0.509,
1443
+ 0.3172
1444
+ ],
1445
+ "out.59": [
1446
+ -0.613,
1447
+ -0.4831,
1448
+ 0.1631,
1449
+ -0.0219,
1450
+ 0.0467,
1451
+ 0.0809,
1452
+ 0.4635,
1453
+ 0.3744
1454
+ ],
1455
+ "out.60": [
1456
+ -0.3279,
1457
+ -0.408,
1458
+ 0.0696,
1459
+ -0.0106,
1460
+ -0.3734,
1461
+ -0.0142,
1462
+ 0.7029,
1463
+ 0.2954
1464
+ ],
1465
+ "out.61": [
1466
+ -0.3705,
1467
+ -0.4785,
1468
+ 0.0727,
1469
+ -0.0334,
1470
+ -0.3736,
1471
+ 0.0942,
1472
+ 0.6327,
1473
+ 0.2803
1474
+ ],
1475
+ "out.62": [
1476
+ -0.3983,
1477
+ -0.2952,
1478
+ 0.3403,
1479
+ -0.159,
1480
+ -0.5147,
1481
+ 0.2026,
1482
+ 0.4885,
1483
+ 0.2616
1484
+ ],
1485
+ "out.63": [
1486
+ -0.4483,
1487
+ -0.2495,
1488
+ 0.3827,
1489
+ -0.1574,
1490
+ -0.4999,
1491
+ 0.2173,
1492
+ 0.449,
1493
+ 0.2585
1494
+ ],
1495
+ "out.64": [
1496
+ -0.3415,
1497
+ -0.3009,
1498
+ 0.2425,
1499
+ -0.1649,
1500
+ -0.5324,
1501
+ 0.2438,
1502
+ 0.4093,
1503
+ 0.4431
1504
+ ],
1505
+ "out.65": [
1506
+ -0.3477,
1507
+ -0.3198,
1508
+ 0.1964,
1509
+ -0.189,
1510
+ -0.5498,
1511
+ 0.1787,
1512
+ 0.5152,
1513
+ 0.3209
1514
+ ],
1515
+ "out.66": [
1516
+ -0.3482,
1517
+ -0.2668,
1518
+ 0.363,
1519
+ -0.1126,
1520
+ -0.5485,
1521
+ 0.0757,
1522
+ 0.456,
1523
+ 0.3854
1524
+ ],
1525
+ "out.67": [
1526
+ -0.2228,
1527
+ -0.3284,
1528
+ 0.347,
1529
+ -0.2348,
1530
+ -0.5103,
1531
+ -0.0221,
1532
+ 0.5322,
1533
+ 0.3505
1534
+ ],
1535
+ "out.68": [
1536
+ 0.6091,
1537
+ 0.1633,
1538
+ 0.1399,
1539
+ -0.4393,
1540
+ -0.1128,
1541
+ 0.0836,
1542
+ 0.1625,
1543
+ -0.5862
1544
+ ],
1545
+ "out.69": [
1546
+ 0.6688,
1547
+ 0.1546,
1548
+ 0.1027,
1549
+ -0.3459,
1550
+ 0.0276,
1551
+ 0.1944,
1552
+ 0.1199,
1553
+ -0.5879
1554
+ ],
1555
+ "out.70": [
1556
+ 0.1084,
1557
+ -0.2913,
1558
+ 0.2023,
1559
+ -0.5036,
1560
+ -0.4744,
1561
+ 0.2459,
1562
+ 0.466,
1563
+ 0.3258
1564
+ ]
1565
+ },
1566
+ "results": {
1567
+ "logit lens": {
1568
+ "recall10_seven": 0.24431818181818182,
1569
+ "recall1_seven": 0.09848484848484848,
1570
+ "recall10_by_task": {
1571
+ "multihop": 0.45454545454545453,
1572
+ "multilingual": 0.45454545454545453,
1573
+ "typo": 0.3125,
1574
+ "association": 0.0,
1575
+ "poetry": 0.0
1576
+ },
1577
+ "recall10_any_node_in_block": {
1578
+ "input block": 0.0375,
1579
+ "loop 1": 0.17954545454545454,
1580
+ "output block": 0.25984848484848483
1581
+ },
1582
+ "recall10_per_node": {
1583
+ "in.09": 0.008333333333333333,
1584
+ "rec.1.18": 0.02954545454545455,
1585
+ "rec.1.27": 0.0,
1586
+ "rec.1.36": 0.008333333333333333,
1587
+ "rec.1.45": 0.048863636363636366,
1588
+ "rec.1.54": 0.08787878787878788,
1589
+ "out.63": 0.1678030303030303
1590
+ }
1591
+ },
1592
+ "J-lens (E001, 50 prompts)": {
1593
+ "recall10_seven": 0.3188543599257885,
1594
+ "recall1_seven": 0.1534090909090909,
1595
+ "recall10_by_task": {
1596
+ "multihop": 0.45454545454545453,
1597
+ "multilingual": 0.5568181818181818,
1598
+ "typo": 0.5625,
1599
+ "association": 0.0,
1600
+ "poetry": 0.02040816326530612
1601
+ },
1602
+ "recall10_any_node_in_block": {
1603
+ "input block": 0.03333333333333333,
1604
+ "loop 1": 0.32832405689548544,
1605
+ "output block": 0.30786951144094005
1606
+ },
1607
+ "recall10_per_node": {
1608
+ "in.09": 0.025,
1609
+ "rec.1.18": 0.055681818181818186,
1610
+ "rec.1.27": 0.06060606060606061,
1611
+ "rec.1.36": 0.08522727272727273,
1612
+ "rec.1.45": 0.2264301175015461,
1613
+ "rec.1.54": 0.1428030303030303,
1614
+ "out.63": 0.17196969696969694
1615
+ }
1616
+ },
1617
+ "R-lens (pooled LRP)": {
1618
+ "recall10_seven": 0.39604205318491037,
1619
+ "recall1_seven": 0.2063543599257885,
1620
+ "recall10_by_task": {
1621
+ "multihop": 0.5454545454545454,
1622
+ "multilingual": 0.6022727272727273,
1623
+ "typo": 0.7916666666666666,
1624
+ "association": 0.0,
1625
+ "poetry": 0.04081632653061224
1626
+ },
1627
+ "recall10_any_node_in_block": {
1628
+ "input block": 0.13952762923351159,
1629
+ "loop 1": 0.3523964131106988,
1630
+ "output block": 0.32083333333333336
1631
+ },
1632
+ "recall10_per_node": {
1633
+ "in.09": 0.1,
1634
+ "rec.1.18": 0.11704545454545454,
1635
+ "rec.1.27": 0.12575757575757576,
1636
+ "rec.1.36": 0.1414965986394558,
1637
+ "rec.1.45": 0.22377860235003094,
1638
+ "rec.1.54": 0.18977272727272726,
1639
+ "out.63": 0.19053030303030302
1640
+ }
1641
+ },
1642
+ "best single expert": {
1643
+ "recall10_seven": 0.4322801848011932,
1644
+ "recall1_seven": 0.21444259521990613,
1645
+ "recall10_by_task": {
1646
+ "multihop": 0.5454545454545454,
1647
+ "multilingual": 0.625,
1648
+ "typo": 0.7916666666666666,
1649
+ "association": 0.11764705882352941,
1650
+ "poetry": 0.08163265306122448
1651
+ },
1652
+ "recall10_any_node_in_block": {
1653
+ "input block": 0.15026737967914436,
1654
+ "loop 1": 0.4410632434792099,
1655
+ "output block": 0.3647363490850886
1656
+ },
1657
+ "recall10_per_node": {
1658
+ "in.09": 0.07702762923351159,
1659
+ "rec.1.18": 0.1321969696969697,
1660
+ "rec.1.27": 0.11704545454545454,
1661
+ "rec.1.36": 0.1784800283749864,
1662
+ "rec.1.45": 0.2413265306122449,
1663
+ "rec.1.54": 0.2434755720470006,
1664
+ "out.63": 0.20497289825020917
1665
+ }
1666
+ },
1667
+ "J++ shuffled labels": {
1668
+ "recall10_seven": 0.23753865182436612,
1669
+ "recall1_seven": 0.09271799628942487,
1670
+ "recall10_by_task": {
1671
+ "multihop": 0.09090909090909091,
1672
+ "multilingual": 0.3068181818181818,
1673
+ "typo": 0.7083333333333334,
1674
+ "association": 0.0,
1675
+ "poetry": 0.08163265306122448
1676
+ },
1677
+ "recall10_any_node_in_block": {
1678
+ "input block": 0.14777592855323948,
1679
+ "loop 1": 0.3393598348430281,
1680
+ "output block": 0.16516379278984322
1681
+ },
1682
+ "recall10_per_node": {
1683
+ "in.09": 0.10833333333333332,
1684
+ "rec.1.18": 0.035521026592455165,
1685
+ "rec.1.27": 0.08741496598639456,
1686
+ "rec.1.36": 0.08400587507730366,
1687
+ "rec.1.45": 0.07482993197278912,
1688
+ "rec.1.54": 0.07605132962275821,
1689
+ "out.63": 0.0
1690
+ }
1691
+ },
1692
+ "J++": {
1693
+ "recall10_seven": 0.465319764269344,
1694
+ "recall1_seven": 0.21468769325912182,
1695
+ "recall10_by_task": {
1696
+ "multihop": 0.6363636363636364,
1697
+ "multilingual": 0.6363636363636364,
1698
+ "typo": 0.875,
1699
+ "association": 0.11764705882352941,
1700
+ "poetry": 0.061224489795918366
1701
+ },
1702
+ "recall10_any_node_in_block": {
1703
+ "input block": 0.15610926188657281,
1704
+ "loop 1": 0.4292621594092182,
1705
+ "output block": 0.4748767688893739
1706
+ },
1707
+ "recall10_per_node": {
1708
+ "in.09": 0.12916666666666668,
1709
+ "rec.1.18": 0.14445187165775403,
1710
+ "rec.1.27": 0.15229500891265596,
1711
+ "rec.1.36": 0.17888519043981227,
1712
+ "rec.1.45": 0.24604205318491035,
1713
+ "rec.1.54": 0.2792376041325621,
1714
+ "out.63": 0.17727272727272728
1715
+ }
1716
+ },
1717
+ "J++ (full vocabulary)": {
1718
+ "recall10_seven": 0.40758803521408565,
1719
+ "recall1_seven": 0.16241496598639454,
1720
+ "recall10_by_task": {
1721
+ "multihop": 0.6363636363636364,
1722
+ "multilingual": 0.48863636363636365,
1723
+ "typo": 0.8125,
1724
+ "association": 0.0392156862745098,
1725
+ "poetry": 0.061224489795918366
1726
+ },
1727
+ "recall10_any_node_in_block": {
1728
+ "input block": 0.14558823529411766,
1729
+ "loop 1": 0.3869006693586525,
1730
+ "output block": 0.45856660846156655
1731
+ },
1732
+ "recall10_per_node": {
1733
+ "in.09": 0.11666666666666667,
1734
+ "rec.1.18": 0.11869429590017826,
1735
+ "rec.1.27": 0.12058823529411766,
1736
+ "rec.1.36": 0.16146094801556987,
1737
+ "rec.1.45": 0.19263296227581944,
1738
+ "rec.1.54": 0.20227272727272724,
1739
+ "out.63": 0.17272727272727273
1740
+ }
1741
+ },
1742
+ "J-lens (full vocabulary)": {
1743
+ "recall10_seven": 0.2991573902288188,
1744
+ "recall1_seven": 0.13787878787878788,
1745
+ "recall10_by_task": {
1746
+ "multihop": 0.45454545454545453,
1747
+ "multilingual": 0.5,
1748
+ "typo": 0.5208333333333334,
1749
+ "association": 0.0,
1750
+ "poetry": 0.02040816326530612
1751
+ },
1752
+ "recall10_any_node_in_block": {
1753
+ "input block": 0.025,
1754
+ "loop 1": 0.27036951144094,
1755
+ "output block": 0.29536951144094004
1756
+ },
1757
+ "recall10_per_node": {
1758
+ "in.09": 0.016666666666666666,
1759
+ "rec.1.18": 0.0375,
1760
+ "rec.1.27": 0.058333333333333334,
1761
+ "rec.1.36": 0.0625,
1762
+ "rec.1.45": 0.13021799628942485,
1763
+ "rec.1.54": 0.12386363636363637,
1764
+ "out.63": 0.1678030303030303
1765
+ }
1766
+ }
1767
+ },
1768
+ "rlens_split_half_median_by_block": {
1769
+ "input block": 0.9892006516456604,
1770
+ "loop 1": 0.9940623044967651,
1771
+ "output block": 0.999410092830658
1772
+ }
1773
+ }
fit_records/jpp_k1_n64_e8_s0_r2/prompts.json ADDED
@@ -0,0 +1,258 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [
2
+ {
3
+ "i": 0,
4
+ "start": "Given the chaos , the governor of Yunnan ignored the imperial orders of evacuation . The Mongol army commanded by Prince Ye @"
5
+ },
6
+ {
7
+ "i": 1,
8
+ "start": "From the 1920s Australia 's defence thinking was dominated by British Imperial defence policy , which was embodied"
9
+ },
10
+ {
11
+ "i": 2,
12
+ "start": "Tim 's driving days started as a toddler when he was given a go @-@ kart that he often drove"
13
+ },
14
+ {
15
+ "i": 3,
16
+ "start": "As the Marines formed into a horseshoe @-@ shaped line around Hill 123 , Tamura 's"
17
+ },
18
+ {
19
+ "i": 4,
20
+ "start": "USA Today 's Scott Bowles awarded the film three out of four stars , labeling it a \" surprising comedy that rivals"
21
+ },
22
+ {
23
+ "i": 5,
24
+ "start": "Ringo Starr identified his initial inspiration for \" Back Off Boogaloo \" as having come from Marc Bolan ,"
25
+ },
26
+ {
27
+ "i": 6,
28
+ "start": "The ship 's secondary armament consisted of fourteen 45 @-@ calibre 6 @-@ inch ("
29
+ },
30
+ {
31
+ "i": 7,
32
+ "start": "Sources disagree about Radcliffe 's personal wealth ; he was reported to have earned \u00a3 1 million for the first Harry"
33
+ },
34
+ {
35
+ "i": 8,
36
+ "start": "In April 2010 , there were 6 @,@ 260 people employed in the healthcare field"
37
+ },
38
+ {
39
+ "i": 9,
40
+ "start": "\" Exit Through the Kwik @-@ E @-@ Mart \" is the fifteenth episode of the twenty @-"
41
+ },
42
+ {
43
+ "i": 10,
44
+ "start": "On 25 June 1950 , the North Korean Army ( KPA ) crossed the border into South"
45
+ },
46
+ {
47
+ "i": 11,
48
+ "start": "A British parliamentary motion put forward by Labour MP Kate Hoey in February 2013 to retrospectively condemn the"
49
+ },
50
+ {
51
+ "i": 12,
52
+ "start": "At the fort , Moultrie ordered his men to concentrate their fire on the two large man @-@ of @"
53
+ },
54
+ {
55
+ "i": 13,
56
+ "start": "In Part II of the series , Shikamaru is assigned the task of locating two members of the criminal organization Ak"
57
+ },
58
+ {
59
+ "i": 14,
60
+ "start": "The Ming dynasty had seen considerable advancement in the process of colour printing in China . At his studio , Hu Zhengyan experiment"
61
+ },
62
+ {
63
+ "i": 15,
64
+ "start": "\" Exit Through the Kwik @-@ E @-@ Mart \" was written by Marc Wilmore and directed by"
65
+ },
66
+ {
67
+ "i": 16,
68
+ "start": "In 629 Tikal founded Dos Pilas , some 110 kilometres ( 68 mi )"
69
+ },
70
+ {
71
+ "i": 17,
72
+ "start": "The British fleet weighed anchor at Cape Fear on May 31 , and arrived outside Charleston Harbor the next day . Moul"
73
+ },
74
+ {
75
+ "i": 18,
76
+ "start": "The initial phase saw the Australians advance towards the Hongorai River . Following the end of the early fighting , the Australian"
77
+ },
78
+ {
79
+ "i": 19,
80
+ "start": "The reactions of the native Aboriginal inhabitants to the sudden arrival of British settlers in Australia were varied , but were inevitably hostile when"
81
+ },
82
+ {
83
+ "i": 20,
84
+ "start": "The German offensive ground to a halt in mid @-@ July and a brief lull followed , during which the Australians"
85
+ },
86
+ {
87
+ "i": 21,
88
+ "start": "The city 's growth has reflected the push and pull of many social and economic factors . The total population increased in each"
89
+ },
90
+ {
91
+ "i": 22,
92
+ "start": "Following the classification of Tropical Depression Two @-@ E on June 19 , the Government of Mexico issued a tropical"
93
+ },
94
+ {
95
+ "i": 23,
96
+ "start": "The Natives remained in London following the England international . They defeated London Welsh on 18 February , before losing first"
97
+ },
98
+ {
99
+ "i": 24,
100
+ "start": "In February 1997 , a United States immigration court judge granted asylum to a German Scientologist who claimed she"
101
+ },
102
+ {
103
+ "i": 25,
104
+ "start": "Some of the most important trade routes began at Canton and Macau in Qing Dynasty China . Early in each year a large convoy"
105
+ },
106
+ {
107
+ "i": 26,
108
+ "start": "The fourth installment in the Slammiversary name was announced in January 2008 to take place on June"
109
+ },
110
+ {
111
+ "i": 27,
112
+ "start": "Capel Lligwy has national recognition and statutory protection from alteration as it has been designated as a Grade II listed building"
113
+ },
114
+ {
115
+ "i": 28,
116
+ "start": "The Trout Creek Mountains are semi @-@ arid because they are in the eastern rain shadow of mountain ranges to"
117
+ },
118
+ {
119
+ "i": 29,
120
+ "start": "Asahi , meaning \" rising sun \" , a poetic name for Japan from a stanza of waka poetry , was laid"
121
+ },
122
+ {
123
+ "i": 30,
124
+ "start": "Beckham 's term as governor ended December 10 , 1907 . In January 19"
125
+ },
126
+ {
127
+ "i": 31,
128
+ "start": "Donald Sutherland portrays Jack Pellitt , Bobby 's father and Kurt 's boss . On July 27 ,"
129
+ },
130
+ {
131
+ "i": 32,
132
+ "start": "As British offshoots , the Australian colonies participated in Britain 's small wars of the 19th century"
133
+ },
134
+ {
135
+ "i": 33,
136
+ "start": "Retail is another major employer in the county , with 5 @,@ 280 people employed in April "
137
+ },
138
+ {
139
+ "i": 34,
140
+ "start": "The tour returned to Pocono a month later , and Richmond battled for another victory in a fog @-@ shortened"
141
+ },
142
+ {
143
+ "i": 35,
144
+ "start": "\" ... Thirteen Years Later \" received mixed reviews from critics . The A.V. Club 's Todd VanDerWer"
145
+ },
146
+ {
147
+ "i": 36,
148
+ "start": "Bonnie Stiernberg , for Paste , was more positive , writing that she preferred the use of comedy , following the"
149
+ },
150
+ {
151
+ "i": 37,
152
+ "start": "In common with many small Scottish islands , Papa Stour 's population peaked in the 19th century and has"
153
+ },
154
+ {
155
+ "i": 38,
156
+ "start": "The unexpected deployment to East Timor in 1999 led to significant changes in Australian defence policy and to an enhancement"
157
+ },
158
+ {
159
+ "i": 39,
160
+ "start": "Government publications on the dangers of sects increased between 1996 and 1998 , and"
161
+ },
162
+ {
163
+ "i": 40,
164
+ "start": "Richmond started 1982 without a ride before getting a one @-@ race deal to drive for Billie"
165
+ },
166
+ {
167
+ "i": 41,
168
+ "start": "The music for Sonic the Hedgehog was composed and produced by Masato Nakamura , bassist and lead songwriter of the"
169
+ },
170
+ {
171
+ "i": 42,
172
+ "start": "Apple Records issued the single on 17 March 1972 in Britain , as Apple R 59"
173
+ },
174
+ {
175
+ "i": 43,
176
+ "start": "Meridian is home to two post @-@ secondary educational institutions . Meridian Community College , founded in 19"
177
+ },
178
+ {
179
+ "i": 44,
180
+ "start": "Ayola began her career in the theatre , performing in a number of Shakespearean plays including Twelfth Night , A M"
181
+ },
182
+ {
183
+ "i": 45,
184
+ "start": "Cooke 's opportunistic diversion had determined that the Spanish forces in the Philippines posed no immediate threat , although the mission"
185
+ },
186
+ {
187
+ "i": 46,
188
+ "start": "To further justify its stance , the German government has also pointed to the long history of U.S. court cases involving Scient"
189
+ },
190
+ {
191
+ "i": 47,
192
+ "start": "Headed by police chief Lee Shelbourn since 2009 , the Meridian Police Department consists of "
193
+ },
194
+ {
195
+ "i": 48,
196
+ "start": "A port , Sonic the Hedgehog Genesis , was released for the Game Boy Advance ( GBA ) on November"
197
+ },
198
+ {
199
+ "i": 49,
200
+ "start": "As is often the case with huge ancient ruins , knowledge of the site was never completely lost in the region . It seems"
201
+ },
202
+ {
203
+ "i": 50,
204
+ "start": "According to Eliakim Sibanda , a professor and human rights speaker who wrote a history of ZAPU , N"
205
+ },
206
+ {
207
+ "i": 51,
208
+ "start": "The Raid on Manila of January 1798 was a Royal Navy false flag military operation during the French Revolutionary"
209
+ },
210
+ {
211
+ "i": 52,
212
+ "start": "Five infantry divisions of the AIF saw action in France and Belgium , leaving Egypt in March 1916 ."
213
+ },
214
+ {
215
+ "i": 53,
216
+ "start": "Die Another Day had its world premiere on 18 November 2002 at the Royal Albert Hall in London"
217
+ },
218
+ {
219
+ "i": 54,
220
+ "start": "To get revenge on Homer , Bart goes around Springfield spray @-@ painting graffiti of Homer 's face and the"
221
+ },
222
+ {
223
+ "i": 55,
224
+ "start": "The Natives next defeated both Northamptonshire and Kent , before defeats by Moseley and Burton @-"
225
+ },
226
+ {
227
+ "i": 56,
228
+ "start": "Headlam remained in Darwin as Controller of Operations at Headquarters North @-@ Western Area Command until 12 May "
229
+ },
230
+ {
231
+ "i": 57,
232
+ "start": "The advance entered Palestine and an initial , unsuccessful attempt was made to capture Gaza on 26 March 191"
233
+ },
234
+ {
235
+ "i": 58,
236
+ "start": "At the same time , a Japanese company from Watanabe 's battalion infiltrated through a gap between the east side of the"
237
+ },
238
+ {
239
+ "i": 59,
240
+ "start": "Route 29 begins at a modified cloverleaf interchange with Interstate 195 and Interstate 295"
241
+ },
242
+ {
243
+ "i": 60,
244
+ "start": "Under the overall command of Australian Major General Peter Cosgrove , INTERFET began arriving on 12 September 1"
245
+ },
246
+ {
247
+ "i": 61,
248
+ "start": "Australia 's involvement in the Vietnam War was driven largely by the rise of communism in Southeast Asia after the Second World"
249
+ },
250
+ {
251
+ "i": 62,
252
+ "start": "The fact that Scientologists have to pay large fees for auditing and other Scientology services has brought controversy to Scientology"
253
+ },
254
+ {
255
+ "i": 63,
256
+ "start": "Moving forward under a creeping barrage as they moved beyond Tokinotu , the 24th Infantry Battalion was the"
257
+ }
258
+ ]
fit_records/jpp_k1_n64_e8_s0_r2/readouts.txt ADDED
@@ -0,0 +1,80 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ router at in.08: 56000 positions from 500 paragraphs; cluster shares [0.076, 0.241, 0.018, 0.268, 0.195, 0.072, 0.034, 0.095]
2
+ positions per expert: [509, 1688, 131, 1933, 1353, 496, 235, 823]
3
+ items per task, all / kept (multihop and multilingual kept only where the model's top next token starts the answer) / dev / test: multihop 84/22/11/11; multilingual 107/44/22/22; typo 96/96/48/48; association 102/102/51/51; poetry 98/98/49/49
4
+ weights fitted on 247 dev (item, intermediate) pairs in 109s
5
+
6
+ TEST HALF: 181 items, 247 (item, intermediate) pairs. Seven scored nodes: ['in.09', 'rec.1.18', 'rec.1.27', 'rec.1.36', 'rec.1.45', 'rec.1.54', 'out.63']
7
+ recall = macro over the five tasks; an intermediate counts if it is in the top k at ANY of the listed nodes; punctuation-like tokens dropped unless marked
8
+ lens @10 seven @1 seven | multihop multiling typo associati poetry | input block loop 1 output block
9
+ logit lens 0.244 0.098 | 0.455 0.455 0.312 0.000 0.000 | 0.037 0.180 0.260
10
+ J-lens (E001, 50 prompts) 0.319 0.153 | 0.455 0.557 0.562 0.000 0.020 | 0.033 0.328 0.308
11
+ R-lens (pooled LRP) 0.396 0.206 | 0.545 0.602 0.792 0.000 0.041 | 0.140 0.352 0.321
12
+ best single expert 0.432 0.214 | 0.545 0.625 0.792 0.118 0.082 | 0.150 0.441 0.365
13
+ J++ shuffled labels 0.238 0.093 | 0.091 0.307 0.708 0.000 0.082 | 0.148 0.339 0.165
14
+ J++ 0.465 0.215 | 0.636 0.636 0.875 0.118 0.061 | 0.156 0.429 0.475
15
+ J++ (full vocabulary) 0.408 0.162 | 0.636 0.489 0.812 0.039 0.061 | 0.146 0.387 0.459
16
+ J-lens (full vocabulary) 0.299 0.138 | 0.455 0.500 0.521 0.000 0.020 | 0.025 0.270 0.295
17
+
18
+ recall@10 at each of the seven nodes on its own
19
+ lens in.09 rec.1.18 rec.1.27 rec.1.36 rec.1.45 rec.1.54 out.63
20
+ logit lens 0.008 0.030 0.000 0.008 0.049 0.088 0.168
21
+ J-lens (E001, 50 prompts) 0.025 0.056 0.061 0.085 0.226 0.143 0.172
22
+ R-lens (pooled LRP) 0.100 0.117 0.126 0.141 0.224 0.190 0.191
23
+ best single expert 0.077 0.132 0.117 0.178 0.241 0.243 0.205
24
+ J++ shuffled labels 0.108 0.036 0.087 0.084 0.075 0.076 0.000
25
+ J++ 0.129 0.144 0.152 0.179 0.246 0.279 0.177
26
+
27
+ R-lens split-half cosine and J++ weights at the seven nodes (weights in router-cluster order)
28
+ in.09 split-half 0.991 weights [-0.08, -0.06, 0.11, 0.58, 0.77, 0.02, -0.22, 0.08] best expert 3
29
+ rec.1.18 split-half 0.992 weights [0.79, -0.18, 0.08, 0.31, 0.15, 0.07, -0.18, -0.42] best expert 3
30
+ rec.1.27 split-half 0.994 weights [0.65, -0.46, 0.09, 0.28, 0.47, 0.05, -0.18, -0.16] best expert 0
31
+ rec.1.36 split-half 0.995 weights [0.59, -0.42, 0.04, 0.21, 0.61, 0.08, -0.19, 0.11] best expert 0
32
+ rec.1.45 split-half 0.997 weights [0.52, -0.44, 0.1, 0.46, 0.5, 0.07, -0.17, -0.16] best expert 0
33
+ rec.1.54 split-half 0.998 weights [0.51, -0.77, -0.02, 0.03, 0.35, 0.17, -0.02, 0.03] best expert 0
34
+ out.63 split-half 0.999 weights [-0.07, -0.33, 0.33, -0.71, 0.46, 0.22, 0.06, 0.09] best expert 0
35
+ R-lens split-half cosine, median by block: input block 0.99; loop 1 0.99; output block 1.00
36
+
37
+ [multihop] 'Fact: The smallest country in the world is located on the continent of' read at token ' of'; intermediates [' Vatican']
38
+ in.09 | J-lens: ['’S', ' America', '’s', '“The', ' American', ' world'] | R-lens: [' American', ' to', ' and', ' in', ' America', ' the'] | J++: [' American', ' to', ' the', ' a', ' America', ' and']
39
+ rec.1.18 | J-lens: [' America', ' American', 'American', ' United', ' Americans', ' Canada'] | R-lens: [' American', ' United', ' the', ' and', ' America', ' to'] | J++: [' American', ' United', ' the', ' America', ' South', ' world']
40
+ rec.1.27 | J-lens: [' America', ' American', 'American', 'America', ' continents', ' Africa'] | R-lens: [' United', ' American', ' America', ' the', ' and', ' that'] | J++: [' the', ' American', ' United', ' America', ' a', ' The']
41
+ rec.1.36 | J-lens: [' America', ' countries', ' continents', 'American', ' American', 'America'] | R-lens: [' American', ' United', ' America', ' country', ' countries', ' British'] | J++: [' American', ' United', ' country', ' British', ' South', ' the']
42
+ rec.1.45 | J-lens: [' Europe', ' Asia', ' continents', ' continent', ' Africa', ' European'] | R-lens: [' Europe', ' Asia', ' European', ' continents', ' continent', ' Africa'] | J++: [' Europe', ' continent', ' Asia', ' European', ' continents', ' continental']
43
+ rec.1.54 | J-lens: [' Europe', ' Asia', ' continents', ' Africa', ' continent', 'Europe'] | R-lens: [' Europe', ' Asia', ' continents', ' Africa', ' European', ' continent'] | J++: [' continent', ' United', ' European', ' Europe', ' continents', ' North']
44
+ out.63 | J-lens: [' Europe', ' Asia', ' Africa', ' Oce', ' Euras', 'Europe'] | R-lens: [' Europe', ' Asia', ' Africa', ' Oce', ' North', ' South'] | J++: [' Europe', ' Asia', ' Australia', 'Europe', ' Europ', ' North']
45
+
46
+ [multilingual] 'Cuando el agua se congela se convierte en "' read at token ' "'; intermediates [' Spanish', ' freeze', 'Water', ' ice']
47
+ in.09 | J-lens: ['s', '"I', '"The', '"A', '"It', '"What'] | R-lens: ['"I', '"The', '"A', '"What', "'s", '"You'] | J++: ['s', ' words', ' fake', ' automatically', ' AI', ' completely']
48
+ rec.1.18 | J-lens: ['"I', 's', '"The', '"A', '"What', '"It'] | R-lens: [' the', ' a', ' to', ' very', "'s", ' and'] | J++: [' the', ' very', ' water', ' a', ' one', ' The']
49
+ rec.1.27 | J-lens: ['"The', '"A', '"If', '"It', '"I', '"What'] | R-lens: [' the', ' or', ' a', ' water', ' very', "'s"] | J++: [' the', ' a', ' water', ' very', ' The', ' s']
50
+ rec.1.36 | J-lens: ['water', ' water', ' underwater', ' fish', ' fishermen', ' submerged'] | R-lens: [' water', ' the', ' a', ' or', ' fish', ' natural'] | J++: [' water', ' fish', ' Water', ' sea', ' underwater', ' natural']
51
+ rec.1.45 | J-lens: [' frozen', 'water', ' snow', ' freeze', ' ice', ' frost'] | R-lens: [' water', ' cold', ' frozen', ' snow', ' ice', ' a'] | J++: [' ice', ' frozen', ' water', ' snow', ' cold', ' freeze']
52
+ rec.1.54 | J-lens: [' ice', ' snow', ' frozen', ' freeze', ' frost', ' Ice'] | R-lens: [' ice', ' snow', ' frozen', ' freeze', ' cold', ' water'] | J++: [' ice', ' snow', ' Ice', ' Snow', ' frozen', ' water']
53
+ out.63 | J-lens: [' ice', ' Ice', 'ice', 'Ice', ' snow', ' ICE'] | R-lens: [' ice', ' Ice', 'ice', 'Ice', ' snow', ' ICE'] | J++: [' ice', 'ice', 'gl', ' Ice', 'Ice', ' ic']
54
+
55
+ [typo] 'He finished the race in secnod' read at token 'od'; intermediates [' second']
56
+ in.09 | J-lens: ['’s', '“We', '’d', '’S', '’n', ' favoured'] | R-lens: [' merely', ' perhaps', ' mere', '’d', ' rather', '’s'] | J++: [' merely', ' perhaps', ' mere', ' rather', 'less', ' nod']
57
+ rec.1.18 | J-lens: [' tho', ' cong', ';s', 'ly', ' stru', ' bt'] | R-lens: [' time', ' only', ' of', ' s', ' a', ' for'] | J++: [' time', ' the', ' a', ' speed', '’s', ' very']
58
+ rec.1.27 | J-lens: [' tho', ';s', ' ats', '[s', ' tand', ' cong'] | R-lens: [' time', ' s', ' of', ' th', 'ed', ' seconds'] | J++: [' the', ' a', ' time', ' seconds', ' of', ' as']
59
+ rec.1.36 | J-lens: [' tand', '[s', ' bec', ' thi', ' creat', ' ei'] | R-lens: [' time', ' s', ' t', ' seconds', ' d', ' th'] | J++: [' time', ' seconds', ' a', ' the', ' clock', ' s']
60
+ rec.1.45 | J-lens: ['[s', ' tand', ' thi', ' thir', '&q', ' seconds'] | R-lens: [' time', ' s', ' t', ' times', ' seconds', ' a'] | J++: [' time', ' the', ' seconds', ' s', ' a', ' t']
61
+ rec.1.54 | J-lens: [' time', 'time', '[s', 'ss', 'ly', 'stwo'] | R-lens: [' time', ' t', ' s', ' w', 'ly', ' i'] | J++: [' time', ' S', ' New', ' SO', ' Time', ' British']
62
+ out.63 | J-lens: ['[s', 'ly', 'ary', 'stwo', '(s', ' thir'] | R-lens: [' s', 'ly', ' t', ' time', '(s', 'ary'] | J++: ['l', 'i', 'f', 'a', 'the', 'r']
63
+
64
+ [association] 's, tilted her head, then stepped forward and scraped the whole left corner back to canvas.' read at token '.'; intermediates ['Art']
65
+ in.09 | J-lens: [' â', 'â', '’s', '—but', '—the', '`s'] | R-lens: [' a', ' in', ' as', ' to', ' and', ' by'] | J++: [' a', ' at', ' to', ' then', ' in', ' as']
66
+ rec.1.18 | J-lens: ['—but', '—not', '—the', '—and', '—that', '—a'] | R-lens: [' a', ' the', ' one', ' just', ' back', ' then'] | J++: [' back', ' very', ' the', ' then', ' a', ' left']
67
+ rec.1.27 | J-lens: [' photographed', ' grotes', '—not', '—that', '—to', ' scrub'] | R-lens: [' a', ' then', ' back', ' the', ' st', ' one'] | J++: [' the', ' a', ' then', ' back', ' w', ' left']
68
+ rec.1.36 | J-lens: [' photographed', '—No', ' photograph', ' grotes', ' afterward', ' walls'] | R-lens: [' then', ' back', ' the', ' a', ' next', ' again'] | J++: [' back', ' photograph', ' a', ' then', ' the', ' hand']
69
+ rec.1.45 | J-lens: [' afterward', ' next', '—No', ' Next', ' suddenly', ' finally'] | R-lens: [' then', ' after', ' next', ' afterward', ' his', ' the'] | J++: [' his', ' then', ' next', ' the', ' The', ' a']
70
+ rec.1.54 | J-lens: [' she', ' She', ' her', ' hers', ' herself', '她'] | R-lens: [' she', ' her', ' She', ' herself', ' The', ' hers'] | J++: [' She', ' The', ' W', ' U', ' â', ' her']
71
+ out.63 | J-lens: [' she', ' She', ' her', ' herself', ' hers', ' paint'] | R-lens: [' she', ' her', ' She', ' herself', ' The', ' Her'] | J++: ['“The', '2', '1', '4', '9', '7']
72
+
73
+ [poetry] 'uplet:\nThe traveler paused before the ancient gate,\nAnd stepped across to meet his waiting' read at token ',\n'; intermediates [' Fate']
74
+ in.09 | J-lens: ['—a', '—or', '—and', '—but', '—that', '—to'] | R-lens: ['—and', ' perhaps', ' to', '—or', ' a', '—a'] | J++: [' perhaps', ' to', ' then', ' a', ' either', ' at']
75
+ rec.1.18 | J-lens: ['—a', '—to', '—and', '—but', '—or', '—as'] | R-lens: [' a', ' to', ' perhaps', '—a', ' f', ' d'] | J++: [' a', ' story', ' the', ' something', ' poem', ' ancient']
76
+ rec.1.27 | J-lens: ['—a', '—to', '—and', ' weary', '—but', ' poetic'] | R-lens: [' poetic', ' delight', ' poet', ' weary', '—a', '—and'] | J++: [' poem', ' poetic', ' poet', ' poetry', ' a', ' rain']
77
+ rec.1.36 | J-lens: [' poetic', ' dreaming', ' weary', ' delight', ' whispered', ' tonight'] | R-lens: [' a', ' ha', ' water', ' poetic', ' d', ' night'] | J++: [' water', ' a', ' poetic', ' poem', ' story', ' poet']
78
+ rec.1.45 | J-lens: [' dreaming', ' hoping', ' whispered', ' dreams', ' revealing', ' dream'] | R-lens: [' a', ' where', ' with', ' to', ' and', ' fear'] | J++: [' a', ' sight', ' the', ' st', ' light', ' his']
79
+ rec.1.54 | J-lens: [' Where', ' where', ' Then', ' hoping', ' wondering', 'where'] | R-lens: [' where', ' and', ' Where', ' his', ' then', ' And'] | J++: [' Cor', ' His', ' Coron', ' Sur', ' Ch', ' Crow']
80
+ out.63 | J-lens: [' Where', ' dreaming', ' wondering', 'Where', ' whispered', ' And'] | R-lens: [' where', ' Where', ' And', 'Where', ' wondering', ' and'] | J++: ['A', 'The', 'An', 'P', 'Le', 'L']
fit_records/jpp_k2_n64_e8_s0/bootstrap.json ADDED
@@ -0,0 +1,59 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "k": 2,
3
+ "n_test_items": 181,
4
+ "items_per_task": {
5
+ "multihop": 10,
6
+ "multilingual": 23,
7
+ "typo": 48,
8
+ "association": 51,
9
+ "poetry": 49
10
+ },
11
+ "recall10_seven": {
12
+ "logit lens": {
13
+ "point": 0.18927536231884057,
14
+ "lo": 0.13659420289855073,
15
+ "hi": 0.2510869565217392
16
+ },
17
+ "J-lens": {
18
+ "point": 0.2676323572907424,
19
+ "lo": 0.21503623188405802,
20
+ "hi": 0.33100192250813365
21
+ },
22
+ "R-lens": {
23
+ "point": 0.32132800946465545,
24
+ "lo": 0.26124297545105,
25
+ "hi": 0.38531351671103226
26
+ },
27
+ "J++": {
28
+ "point": 0.42311681194216816,
29
+ "lo": 0.35769268576996016,
30
+ "hi": 0.49189749812968664
31
+ }
32
+ },
33
+ "differences": {
34
+ "J++ minus R-lens": {
35
+ "point": 0.10178880247751271,
36
+ "lo": 0.023397489430554863,
37
+ "hi": 0.17837852532317278,
38
+ "share_of_resamples_above_zero": 0.996
39
+ },
40
+ "R-lens minus J-lens": {
41
+ "point": 0.05369565217391303,
42
+ "lo": 0.016666666666666663,
43
+ "hi": 0.10003623188405802,
44
+ "share_of_resamples_above_zero": 0.9995
45
+ },
46
+ "J++ minus J-lens": {
47
+ "point": 0.15548445465142574,
48
+ "lo": 0.09362553717139027,
49
+ "hi": 0.22082546061903022,
50
+ "share_of_resamples_above_zero": 1.0
51
+ },
52
+ "J-lens minus logit lens": {
53
+ "point": 0.07835699497190185,
54
+ "lo": 0.04909420289855071,
55
+ "hi": 0.10934634723454598,
56
+ "share_of_resamples_above_zero": 1.0
57
+ }
58
+ }
59
+ }
fit_records/jpp_k2_n64_e8_s0/metrics.json ADDED
@@ -0,0 +1,2643 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "n_prompts": 64,
3
+ "seconds_per_prompt": 32.213693115860224,
4
+ "n_nodes": 114,
5
+ "seven_nodes": [
6
+ "in.14",
7
+ "rec.1.28",
8
+ "rec.1.42",
9
+ "rec.2.16",
10
+ "rec.2.30",
11
+ "rec.2.44",
12
+ "out.58"
13
+ ],
14
+ "positions_per_expert": [
15
+ 509.0,
16
+ 1688.0,
17
+ 131.0,
18
+ 1933.0,
19
+ 1353.0,
20
+ 496.0,
21
+ 235.0,
22
+ 823.0
23
+ ],
24
+ "n_dev_pairs": 246,
25
+ "n_test_items": 181,
26
+ "n_test_pairs": 250,
27
+ "rlens_split_half_cosine": {
28
+ "emb": 0.9841,
29
+ "in.00": 0.9846,
30
+ "in.01": 0.9889,
31
+ "in.02": 0.9884,
32
+ "in.03": 0.9883,
33
+ "in.04": 0.9891,
34
+ "in.05": 0.9898,
35
+ "in.06": 0.9907,
36
+ "in.07": 0.9909,
37
+ "in.08": 0.9916,
38
+ "in.09": 0.992,
39
+ "in.10": 0.9917,
40
+ "in.11": 0.9916,
41
+ "in.12": 0.9913,
42
+ "in.13": 0.9906,
43
+ "in.14": 0.9901,
44
+ "x": 0.989,
45
+ "u.1": 0.9722,
46
+ "rec.1.16": 0.9822,
47
+ "rec.1.17": 0.9846,
48
+ "rec.1.18": 0.9855,
49
+ "rec.1.19": 0.986,
50
+ "rec.1.20": 0.9867,
51
+ "rec.1.21": 0.9864,
52
+ "rec.1.22": 0.9902,
53
+ "rec.1.23": 0.9895,
54
+ "rec.1.24": 0.9897,
55
+ "rec.1.25": 0.989,
56
+ "rec.1.26": 0.989,
57
+ "rec.1.27": 0.9885,
58
+ "rec.1.28": 0.9887,
59
+ "rec.1.29": 0.9884,
60
+ "rec.1.30": 0.9889,
61
+ "rec.1.31": 0.9883,
62
+ "rec.1.32": 0.9884,
63
+ "rec.1.33": 0.9878,
64
+ "rec.1.34": 0.9881,
65
+ "rec.1.35": 0.9876,
66
+ "rec.1.36": 0.9892,
67
+ "rec.1.37": 0.9891,
68
+ "rec.1.38": 0.9892,
69
+ "rec.1.39": 0.989,
70
+ "rec.1.40": 0.9892,
71
+ "rec.1.41": 0.9892,
72
+ "rec.1.42": 0.9893,
73
+ "rec.1.43": 0.9892,
74
+ "rec.1.44": 0.9891,
75
+ "rec.1.45": 0.9889,
76
+ "rec.1.46": 0.99,
77
+ "rec.1.47": 0.9898,
78
+ "rec.1.48": 0.989,
79
+ "rec.1.49": 0.989,
80
+ "rec.1.50": 0.9892,
81
+ "rec.1.51": 0.9894,
82
+ "rec.1.52": 0.99,
83
+ "rec.1.53": 0.9905,
84
+ "rec.1.54": 0.99,
85
+ "s.1": 0.9903,
86
+ "u.2": 0.9904,
87
+ "rec.2.16": 0.9923,
88
+ "rec.2.17": 0.993,
89
+ "rec.2.18": 0.9931,
90
+ "rec.2.19": 0.9934,
91
+ "rec.2.20": 0.9935,
92
+ "rec.2.21": 0.9936,
93
+ "rec.2.22": 0.9944,
94
+ "rec.2.23": 0.9942,
95
+ "rec.2.24": 0.9943,
96
+ "rec.2.25": 0.9942,
97
+ "rec.2.26": 0.9942,
98
+ "rec.2.27": 0.9941,
99
+ "rec.2.28": 0.9942,
100
+ "rec.2.29": 0.9941,
101
+ "rec.2.30": 0.9944,
102
+ "rec.2.31": 0.9943,
103
+ "rec.2.32": 0.9943,
104
+ "rec.2.33": 0.9942,
105
+ "rec.2.34": 0.9944,
106
+ "rec.2.35": 0.9943,
107
+ "rec.2.36": 0.995,
108
+ "rec.2.37": 0.9951,
109
+ "rec.2.38": 0.9953,
110
+ "rec.2.39": 0.9955,
111
+ "rec.2.40": 0.9957,
112
+ "rec.2.41": 0.9961,
113
+ "rec.2.42": 0.9961,
114
+ "rec.2.43": 0.9965,
115
+ "rec.2.44": 0.9965,
116
+ "rec.2.45": 0.9968,
117
+ "rec.2.46": 0.9967,
118
+ "rec.2.47": 0.997,
119
+ "rec.2.48": 0.9969,
120
+ "rec.2.49": 0.9972,
121
+ "rec.2.50": 0.9973,
122
+ "rec.2.51": 0.9975,
123
+ "rec.2.52": 0.9977,
124
+ "rec.2.53": 0.9979,
125
+ "rec.2.54": 0.998,
126
+ "s.2": 0.9982,
127
+ "out.56": 0.9983,
128
+ "out.57": 0.9985,
129
+ "out.58": 0.9986,
130
+ "out.59": 0.9988,
131
+ "out.60": 0.9989,
132
+ "out.61": 0.9991,
133
+ "out.62": 0.9993,
134
+ "out.63": 0.9994,
135
+ "out.64": 0.9994,
136
+ "out.65": 0.9996,
137
+ "out.66": 0.9997,
138
+ "out.67": 0.9998,
139
+ "out.68": 0.9998,
140
+ "out.69": 0.9999,
141
+ "out.70": 0.9999
142
+ },
143
+ "weights": {
144
+ "emb": [
145
+ 0.0332,
146
+ -0.0123,
147
+ 0.1021,
148
+ 0.1028,
149
+ 0.1415,
150
+ -0.0052,
151
+ 0.0698,
152
+ -0.9761
153
+ ],
154
+ "in.00": [
155
+ 0.1324,
156
+ -0.3416,
157
+ -0.0753,
158
+ 0.1994,
159
+ 0.3698,
160
+ -0.0312,
161
+ 0.3119,
162
+ -0.7651
163
+ ],
164
+ "in.01": [
165
+ -0.2589,
166
+ -0.0658,
167
+ 0.0616,
168
+ 0.1736,
169
+ 0.438,
170
+ -0.0325,
171
+ 0.1606,
172
+ -0.8222
173
+ ],
174
+ "in.02": [
175
+ -0.1155,
176
+ 0.025,
177
+ 0.0598,
178
+ 0.162,
179
+ 0.2465,
180
+ -0.0443,
181
+ 0.1567,
182
+ -0.9322
183
+ ],
184
+ "in.03": [
185
+ -0.0947,
186
+ 0.0171,
187
+ 0.0733,
188
+ 0.1829,
189
+ 0.3693,
190
+ -0.0525,
191
+ 0.1152,
192
+ -0.8941
193
+ ],
194
+ "in.04": [
195
+ 0.1437,
196
+ -0.0421,
197
+ 0.0705,
198
+ 0.2061,
199
+ 0.4142,
200
+ -0.0535,
201
+ 0.1068,
202
+ -0.8627
203
+ ],
204
+ "in.05": [
205
+ 0.5241,
206
+ 0.0588,
207
+ 0.152,
208
+ 0.3703,
209
+ 0.4347,
210
+ -0.0865,
211
+ -0.072,
212
+ -0.6
213
+ ],
214
+ "in.06": [
215
+ 0.8063,
216
+ -0.0752,
217
+ 0.072,
218
+ 0.3158,
219
+ 0.4323,
220
+ 0.007,
221
+ -0.1612,
222
+ -0.1623
223
+ ],
224
+ "in.07": [
225
+ 0.6036,
226
+ -0.1241,
227
+ 0.0859,
228
+ 0.4033,
229
+ 0.6493,
230
+ 0.0069,
231
+ -0.1239,
232
+ -0.1152
233
+ ],
234
+ "in.08": [
235
+ 0.484,
236
+ -0.0699,
237
+ 0.0811,
238
+ 0.46,
239
+ 0.6223,
240
+ 0.043,
241
+ -0.1147,
242
+ -0.3748
243
+ ],
244
+ "in.09": [
245
+ 0.3905,
246
+ -0.0087,
247
+ 0.1334,
248
+ 0.6098,
249
+ 0.6204,
250
+ 0.0153,
251
+ -0.2202,
252
+ -0.1553
253
+ ],
254
+ "in.10": [
255
+ 0.5523,
256
+ 0.0832,
257
+ 0.1505,
258
+ 0.6082,
259
+ 0.4085,
260
+ 0.0018,
261
+ -0.3103,
262
+ 0.1799
263
+ ],
264
+ "in.11": [
265
+ 0.6485,
266
+ 0.1027,
267
+ 0.1263,
268
+ 0.4948,
269
+ 0.2353,
270
+ -0.0095,
271
+ -0.2904,
272
+ 0.4103
273
+ ],
274
+ "in.12": [
275
+ 0.8532,
276
+ -0.0242,
277
+ 0.0949,
278
+ 0.4259,
279
+ 0.1453,
280
+ 0.0215,
281
+ -0.2359,
282
+ 0.0617
283
+ ],
284
+ "in.13": [
285
+ 0.8992,
286
+ -0.0411,
287
+ 0.069,
288
+ 0.3578,
289
+ -0.0167,
290
+ 0.0359,
291
+ -0.2027,
292
+ -0.1199
293
+ ],
294
+ "in.14": [
295
+ 0.8608,
296
+ -0.2006,
297
+ 0.0375,
298
+ 0.2931,
299
+ 0.2319,
300
+ 0.0514,
301
+ -0.1249,
302
+ -0.2438
303
+ ],
304
+ "x": [
305
+ 0.6783,
306
+ -0.1697,
307
+ 0.0693,
308
+ 0.5981,
309
+ 0.0407,
310
+ 0.1,
311
+ -0.2328,
312
+ -0.2878
313
+ ],
314
+ "u.1": [
315
+ 0.3409,
316
+ -0.0757,
317
+ 0.1688,
318
+ 0.8651,
319
+ -0.0202,
320
+ -0.0136,
321
+ -0.2664,
322
+ -0.1718
323
+ ],
324
+ "rec.1.16": [
325
+ 0.675,
326
+ 0.0332,
327
+ 0.1183,
328
+ 0.3685,
329
+ -0.2853,
330
+ 0.0422,
331
+ -0.1827,
332
+ -0.5263
333
+ ],
334
+ "rec.1.17": [
335
+ 0.3972,
336
+ -0.1151,
337
+ 0.1444,
338
+ 0.5424,
339
+ 0.0302,
340
+ -0.0101,
341
+ -0.1071,
342
+ -0.7081
343
+ ],
344
+ "rec.1.18": [
345
+ 0.5138,
346
+ 0.0266,
347
+ 0.1261,
348
+ 0.4258,
349
+ -0.1495,
350
+ 0.0392,
351
+ -0.1819,
352
+ -0.6936
353
+ ],
354
+ "rec.1.19": [
355
+ 0.3861,
356
+ -0.1213,
357
+ 0.1432,
358
+ 0.5276,
359
+ 0.0957,
360
+ 0.0091,
361
+ -0.146,
362
+ -0.7119
363
+ ],
364
+ "rec.1.20": [
365
+ 0.4165,
366
+ -0.0451,
367
+ 0.1422,
368
+ 0.5125,
369
+ -0.0061,
370
+ 0.051,
371
+ -0.227,
372
+ -0.6982
373
+ ],
374
+ "rec.1.21": [
375
+ 0.2268,
376
+ -0.1621,
377
+ 0.1617,
378
+ 0.6121,
379
+ 0.2401,
380
+ 0.0132,
381
+ -0.1988,
382
+ -0.6513
383
+ ],
384
+ "rec.1.22": [
385
+ 0.2683,
386
+ -0.2177,
387
+ 0.1469,
388
+ 0.572,
389
+ 0.2912,
390
+ 0.04,
391
+ -0.2015,
392
+ -0.6362
393
+ ],
394
+ "rec.1.23": [
395
+ 0.2561,
396
+ -0.3289,
397
+ 0.1568,
398
+ 0.5343,
399
+ 0.3751,
400
+ 0.0428,
401
+ -0.1798,
402
+ -0.5842
403
+ ],
404
+ "rec.1.24": [
405
+ 0.3085,
406
+ -0.3117,
407
+ 0.1375,
408
+ 0.5272,
409
+ 0.354,
410
+ 0.0454,
411
+ -0.2016,
412
+ -0.5855
413
+ ],
414
+ "rec.1.25": [
415
+ 0.304,
416
+ -0.3648,
417
+ 0.156,
418
+ 0.5229,
419
+ 0.4032,
420
+ 0.0318,
421
+ -0.2002,
422
+ -0.5225
423
+ ],
424
+ "rec.1.26": [
425
+ 0.3839,
426
+ -0.3639,
427
+ 0.1408,
428
+ 0.5213,
429
+ 0.3407,
430
+ 0.0314,
431
+ -0.1886,
432
+ -0.5254
433
+ ],
434
+ "rec.1.27": [
435
+ 0.3208,
436
+ -0.4377,
437
+ 0.1544,
438
+ 0.5278,
439
+ 0.4354,
440
+ 0.0274,
441
+ -0.208,
442
+ -0.4117
443
+ ],
444
+ "rec.1.28": [
445
+ 0.4013,
446
+ -0.4073,
447
+ 0.1129,
448
+ 0.4838,
449
+ 0.3828,
450
+ 0.0478,
451
+ -0.1805,
452
+ -0.4949
453
+ ],
454
+ "rec.1.29": [
455
+ 0.3934,
456
+ -0.4695,
457
+ 0.1089,
458
+ 0.4831,
459
+ 0.3963,
460
+ 0.0592,
461
+ -0.1987,
462
+ -0.4238
463
+ ],
464
+ "rec.1.30": [
465
+ 0.3538,
466
+ -0.4681,
467
+ 0.1092,
468
+ 0.4861,
469
+ 0.462,
470
+ 0.0376,
471
+ -0.1479,
472
+ -0.4132
473
+ ],
474
+ "rec.1.31": [
475
+ 0.3485,
476
+ -0.5421,
477
+ 0.1083,
478
+ 0.4773,
479
+ 0.4703,
480
+ 0.057,
481
+ -0.1769,
482
+ -0.2991
483
+ ],
484
+ "rec.1.32": [
485
+ 0.3738,
486
+ -0.5387,
487
+ 0.0929,
488
+ 0.4792,
489
+ 0.4757,
490
+ 0.0557,
491
+ -0.1677,
492
+ -0.2726
493
+ ],
494
+ "rec.1.33": [
495
+ 0.3173,
496
+ -0.5668,
497
+ 0.0979,
498
+ 0.4952,
499
+ 0.5011,
500
+ 0.0488,
501
+ -0.1934,
502
+ -0.18
503
+ ],
504
+ "rec.1.34": [
505
+ 0.422,
506
+ -0.5318,
507
+ 0.0975,
508
+ 0.4704,
509
+ 0.4715,
510
+ 0.0394,
511
+ -0.1888,
512
+ -0.221
513
+ ],
514
+ "rec.1.35": [
515
+ 0.2341,
516
+ -0.5465,
517
+ 0.1171,
518
+ 0.465,
519
+ 0.5959,
520
+ 0.0306,
521
+ -0.2365,
522
+ -0.0675
523
+ ],
524
+ "rec.1.36": [
525
+ 0.2434,
526
+ -0.486,
527
+ 0.1286,
528
+ 0.6185,
529
+ 0.5129,
530
+ -0.0405,
531
+ -0.1474,
532
+ -0.1379
533
+ ],
534
+ "rec.1.37": [
535
+ 0.29,
536
+ -0.5141,
537
+ 0.1096,
538
+ 0.5673,
539
+ 0.5151,
540
+ 0.009,
541
+ -0.1978,
542
+ -0.1148
543
+ ],
544
+ "rec.1.38": [
545
+ 0.2135,
546
+ -0.4799,
547
+ 0.1092,
548
+ 0.619,
549
+ 0.5163,
550
+ -0.0602,
551
+ -0.0715,
552
+ -0.2316
553
+ ],
554
+ "rec.1.39": [
555
+ 0.1371,
556
+ -0.4818,
557
+ 0.1147,
558
+ 0.6399,
559
+ 0.5496,
560
+ -0.0677,
561
+ -0.0841,
562
+ -0.1128
563
+ ],
564
+ "rec.1.40": [
565
+ 0.2275,
566
+ -0.4098,
567
+ 0.129,
568
+ 0.6983,
569
+ 0.4242,
570
+ -0.0543,
571
+ -0.1126,
572
+ -0.2837
573
+ ],
574
+ "rec.1.41": [
575
+ 0.2526,
576
+ -0.3732,
577
+ 0.1393,
578
+ 0.7256,
579
+ 0.3616,
580
+ -0.0529,
581
+ -0.1227,
582
+ -0.3201
583
+ ],
584
+ "rec.1.42": [
585
+ 0.3384,
586
+ -0.275,
587
+ 0.1482,
588
+ 0.752,
589
+ 0.222,
590
+ -0.0099,
591
+ -0.2207,
592
+ -0.3525
593
+ ],
594
+ "rec.1.43": [
595
+ 0.251,
596
+ -0.315,
597
+ 0.1487,
598
+ 0.7663,
599
+ 0.2894,
600
+ -0.0375,
601
+ -0.1744,
602
+ -0.336
603
+ ],
604
+ "rec.1.44": [
605
+ 0.3234,
606
+ -0.2774,
607
+ 0.1482,
608
+ 0.7741,
609
+ 0.246,
610
+ -0.0266,
611
+ -0.2096,
612
+ -0.3034
613
+ ],
614
+ "rec.1.45": [
615
+ 0.3811,
616
+ -0.2499,
617
+ 0.1541,
618
+ 0.7654,
619
+ 0.2101,
620
+ -0.0384,
621
+ -0.198,
622
+ -0.3128
623
+ ],
624
+ "rec.1.46": [
625
+ 0.5102,
626
+ -0.3313,
627
+ 0.1553,
628
+ 0.6745,
629
+ 0.2552,
630
+ -0.032,
631
+ -0.2067,
632
+ -0.2046
633
+ ],
634
+ "rec.1.47": [
635
+ 0.4591,
636
+ -0.3385,
637
+ 0.156,
638
+ 0.7117,
639
+ 0.2837,
640
+ -0.0553,
641
+ -0.1778,
642
+ -0.1692
643
+ ],
644
+ "rec.1.48": [
645
+ 0.4999,
646
+ -0.3437,
647
+ 0.1452,
648
+ 0.69,
649
+ 0.2193,
650
+ -0.0344,
651
+ -0.1734,
652
+ -0.2353
653
+ ],
654
+ "rec.1.49": [
655
+ 0.4867,
656
+ -0.3568,
657
+ 0.1451,
658
+ 0.7181,
659
+ 0.2182,
660
+ -0.0518,
661
+ -0.1688,
662
+ -0.1425
663
+ ],
664
+ "rec.1.50": [
665
+ 0.5647,
666
+ -0.3077,
667
+ 0.1455,
668
+ 0.6928,
669
+ 0.1603,
670
+ -0.056,
671
+ -0.1513,
672
+ -0.183
673
+ ],
674
+ "rec.1.51": [
675
+ 0.6163,
676
+ -0.2447,
677
+ 0.1504,
678
+ 0.6845,
679
+ 0.0851,
680
+ -0.0458,
681
+ -0.1773,
682
+ -0.1686
683
+ ],
684
+ "rec.1.52": [
685
+ 0.6003,
686
+ -0.2294,
687
+ 0.1195,
688
+ 0.6882,
689
+ 0.0931,
690
+ -0.0196,
691
+ -0.1627,
692
+ -0.2522
693
+ ],
694
+ "rec.1.53": [
695
+ 0.6358,
696
+ -0.2981,
697
+ 0.1113,
698
+ 0.5909,
699
+ 0.2126,
700
+ -0.0205,
701
+ -0.1619,
702
+ -0.2711
703
+ ],
704
+ "rec.1.54": [
705
+ 0.538,
706
+ -0.3043,
707
+ 0.1335,
708
+ 0.7056,
709
+ 0.1127,
710
+ -0.0497,
711
+ -0.1553,
712
+ -0.251
713
+ ],
714
+ "s.1": [
715
+ 0.5017,
716
+ -0.2541,
717
+ 0.1261,
718
+ 0.6984,
719
+ 0.0904,
720
+ -0.0149,
721
+ -0.1561,
722
+ -0.3839
723
+ ],
724
+ "u.2": [
725
+ 0.7577,
726
+ -0.1384,
727
+ 0.0816,
728
+ 0.5634,
729
+ -0.0058,
730
+ 0.0353,
731
+ -0.1892,
732
+ -0.2135
733
+ ],
734
+ "rec.2.16": [
735
+ 0.7826,
736
+ -0.1179,
737
+ 0.0735,
738
+ 0.3671,
739
+ 0.0482,
740
+ 0.0642,
741
+ -0.1711,
742
+ -0.4447
743
+ ],
744
+ "rec.2.17": [
745
+ 0.7222,
746
+ -0.1314,
747
+ 0.0838,
748
+ 0.4038,
749
+ 0.1178,
750
+ 0.0743,
751
+ -0.1969,
752
+ -0.4826
753
+ ],
754
+ "rec.2.18": [
755
+ 0.7064,
756
+ -0.1885,
757
+ 0.0794,
758
+ 0.3942,
759
+ 0.2125,
760
+ 0.0735,
761
+ -0.1838,
762
+ -0.4684
763
+ ],
764
+ "rec.2.19": [
765
+ 0.7449,
766
+ -0.1972,
767
+ 0.0828,
768
+ 0.3219,
769
+ 0.2249,
770
+ 0.0904,
771
+ -0.1636,
772
+ -0.4586
773
+ ],
774
+ "rec.2.20": [
775
+ 0.6895,
776
+ -0.2369,
777
+ 0.0608,
778
+ 0.3506,
779
+ 0.2877,
780
+ 0.0932,
781
+ -0.1546,
782
+ -0.4759
783
+ ],
784
+ "rec.2.21": [
785
+ 0.6885,
786
+ -0.2877,
787
+ 0.0822,
788
+ 0.3327,
789
+ 0.3751,
790
+ 0.0778,
791
+ -0.1612,
792
+ -0.3913
793
+ ],
794
+ "rec.2.22": [
795
+ 0.5092,
796
+ -0.4419,
797
+ 0.0446,
798
+ 0.3739,
799
+ 0.5244,
800
+ 0.0843,
801
+ -0.1341,
802
+ -0.3219
803
+ ],
804
+ "rec.2.23": [
805
+ 0.4545,
806
+ -0.4683,
807
+ 0.0604,
808
+ 0.3665,
809
+ 0.5689,
810
+ 0.0882,
811
+ -0.1707,
812
+ -0.275
813
+ ],
814
+ "rec.2.24": [
815
+ 0.5075,
816
+ -0.4575,
817
+ 0.0512,
818
+ 0.3621,
819
+ 0.5277,
820
+ 0.0904,
821
+ -0.1575,
822
+ -0.2967
823
+ ],
824
+ "rec.2.25": [
825
+ 0.4711,
826
+ -0.4792,
827
+ 0.0655,
828
+ 0.3639,
829
+ 0.5625,
830
+ 0.0832,
831
+ -0.1772,
832
+ -0.2387
833
+ ],
834
+ "rec.2.26": [
835
+ 0.5548,
836
+ -0.4595,
837
+ 0.076,
838
+ 0.3533,
839
+ 0.5007,
840
+ 0.0735,
841
+ -0.1725,
842
+ -0.2543
843
+ ],
844
+ "rec.2.27": [
845
+ 0.549,
846
+ -0.4736,
847
+ 0.0685,
848
+ 0.3354,
849
+ 0.5448,
850
+ 0.0646,
851
+ -0.184,
852
+ -0.1493
853
+ ],
854
+ "rec.2.28": [
855
+ 0.5085,
856
+ -0.4804,
857
+ 0.052,
858
+ 0.3033,
859
+ 0.5678,
860
+ 0.0658,
861
+ -0.1624,
862
+ -0.2507
863
+ ],
864
+ "rec.2.29": [
865
+ 0.5124,
866
+ -0.5326,
867
+ 0.052,
868
+ 0.3096,
869
+ 0.5195,
870
+ 0.0621,
871
+ -0.1537,
872
+ -0.2404
873
+ ],
874
+ "rec.2.30": [
875
+ 0.4388,
876
+ -0.4909,
877
+ 0.0597,
878
+ 0.268,
879
+ 0.6745,
880
+ 0.0428,
881
+ -0.1565,
882
+ -0.0987
883
+ ],
884
+ "rec.2.31": [
885
+ 0.3978,
886
+ -0.5118,
887
+ 0.0579,
888
+ 0.2637,
889
+ 0.6834,
890
+ 0.0555,
891
+ -0.1801,
892
+ -0.0663
893
+ ],
894
+ "rec.2.32": [
895
+ 0.457,
896
+ -0.4777,
897
+ 0.0491,
898
+ 0.2541,
899
+ 0.6775,
900
+ 0.0586,
901
+ -0.1788,
902
+ -0.0403
903
+ ],
904
+ "rec.2.33": [
905
+ 0.436,
906
+ -0.5123,
907
+ 0.0444,
908
+ 0.282,
909
+ 0.6589,
910
+ 0.0577,
911
+ -0.1675,
912
+ -0.0198
913
+ ],
914
+ "rec.2.34": [
915
+ 0.5078,
916
+ -0.4541,
917
+ 0.0437,
918
+ 0.2574,
919
+ 0.661,
920
+ 0.0539,
921
+ -0.167,
922
+ -0.0049
923
+ ],
924
+ "rec.2.35": [
925
+ 0.5612,
926
+ -0.4306,
927
+ 0.0339,
928
+ 0.239,
929
+ 0.6312,
930
+ 0.0784,
931
+ -0.1874,
932
+ 0.0416
933
+ ],
934
+ "rec.2.36": [
935
+ 0.5083,
936
+ -0.4543,
937
+ 0.0347,
938
+ 0.2234,
939
+ 0.6601,
940
+ 0.0647,
941
+ -0.1754,
942
+ 0.1161
943
+ ],
944
+ "rec.2.37": [
945
+ 0.4933,
946
+ -0.4923,
947
+ 0.0321,
948
+ 0.1918,
949
+ 0.6541,
950
+ 0.0732,
951
+ -0.1652,
952
+ 0.1261
953
+ ],
954
+ "rec.2.38": [
955
+ 0.4699,
956
+ -0.4335,
957
+ 0.0645,
958
+ 0.2277,
959
+ 0.7196,
960
+ 0.0224,
961
+ -0.1244,
962
+ 0.0379
963
+ ],
964
+ "rec.2.39": [
965
+ 0.4864,
966
+ -0.4557,
967
+ 0.0724,
968
+ 0.2837,
969
+ 0.6654,
970
+ 0.0327,
971
+ -0.1433,
972
+ 0.0752
973
+ ],
974
+ "rec.2.40": [
975
+ 0.4145,
976
+ -0.4784,
977
+ 0.0823,
978
+ 0.4818,
979
+ 0.578,
980
+ 0.0432,
981
+ -0.1403,
982
+ -0.0694
983
+ ],
984
+ "rec.2.41": [
985
+ 0.4376,
986
+ -0.4424,
987
+ 0.1044,
988
+ 0.4426,
989
+ 0.6085,
990
+ 0.0551,
991
+ -0.167,
992
+ -0.069
993
+ ],
994
+ "rec.2.42": [
995
+ 0.4679,
996
+ -0.4237,
997
+ 0.0894,
998
+ 0.4627,
999
+ 0.5663,
1000
+ 0.0669,
1001
+ -0.173,
1002
+ -0.1561
1003
+ ],
1004
+ "rec.2.43": [
1005
+ 0.455,
1006
+ -0.4154,
1007
+ 0.1052,
1008
+ 0.4716,
1009
+ 0.5642,
1010
+ 0.0713,
1011
+ -0.1854,
1012
+ -0.1707
1013
+ ],
1014
+ "rec.2.44": [
1015
+ 0.5106,
1016
+ -0.4144,
1017
+ 0.0853,
1018
+ 0.4659,
1019
+ 0.5261,
1020
+ 0.075,
1021
+ -0.1911,
1022
+ -0.156
1023
+ ],
1024
+ "rec.2.45": [
1025
+ 0.5273,
1026
+ -0.4193,
1027
+ 0.0921,
1028
+ 0.4913,
1029
+ 0.4731,
1030
+ 0.0618,
1031
+ -0.1798,
1032
+ -0.1905
1033
+ ],
1034
+ "rec.2.46": [
1035
+ 0.524,
1036
+ -0.5424,
1037
+ 0.0708,
1038
+ 0.3988,
1039
+ 0.4898,
1040
+ 0.0901,
1041
+ -0.1338,
1042
+ -0.0349
1043
+ ],
1044
+ "rec.2.47": [
1045
+ 0.4738,
1046
+ -0.5485,
1047
+ 0.0996,
1048
+ 0.4271,
1049
+ 0.5062,
1050
+ 0.0791,
1051
+ -0.1315,
1052
+ -0.0506
1053
+ ],
1054
+ "rec.2.48": [
1055
+ 0.4301,
1056
+ -0.63,
1057
+ 0.0973,
1058
+ 0.3821,
1059
+ 0.4821,
1060
+ 0.1241,
1061
+ -0.0605,
1062
+ -0.1059
1063
+ ],
1064
+ "rec.2.49": [
1065
+ 0.3567,
1066
+ -0.626,
1067
+ 0.0939,
1068
+ 0.4056,
1069
+ 0.485,
1070
+ 0.1236,
1071
+ -0.0454,
1072
+ -0.2344
1073
+ ],
1074
+ "rec.2.50": [
1075
+ 0.6856,
1076
+ -0.5993,
1077
+ 0.1267,
1078
+ 0.2092,
1079
+ 0.283,
1080
+ 0.1228,
1081
+ -0.0316,
1082
+ -0.1214
1083
+ ],
1084
+ "rec.2.51": [
1085
+ 0.533,
1086
+ -0.7165,
1087
+ 0.0663,
1088
+ 0.0152,
1089
+ 0.415,
1090
+ 0.1315,
1091
+ -0.0452,
1092
+ 0.0803
1093
+ ],
1094
+ "rec.2.52": [
1095
+ 0.5176,
1096
+ -0.724,
1097
+ 0.0513,
1098
+ 0.0456,
1099
+ 0.4314,
1100
+ 0.1261,
1101
+ -0.0303,
1102
+ 0.0124
1103
+ ],
1104
+ "rec.2.53": [
1105
+ 0.4734,
1106
+ -0.7193,
1107
+ 0.0398,
1108
+ -0.0097,
1109
+ 0.4945,
1110
+ 0.072,
1111
+ -0.047,
1112
+ 0.0707
1113
+ ],
1114
+ "rec.2.54": [
1115
+ 0.4988,
1116
+ -0.7617,
1117
+ -0.0137,
1118
+ 0.0508,
1119
+ 0.3795,
1120
+ 0.154,
1121
+ -0.012,
1122
+ -0.0195
1123
+ ],
1124
+ "s.2": [
1125
+ 0.4388,
1126
+ -0.7964,
1127
+ -0.073,
1128
+ 0.2176,
1129
+ 0.2605,
1130
+ 0.1021,
1131
+ -0.0381,
1132
+ 0.202
1133
+ ],
1134
+ "out.56": [
1135
+ 0.4299,
1136
+ -0.8094,
1137
+ 0.0057,
1138
+ -0.0095,
1139
+ 0.3416,
1140
+ 0.1356,
1141
+ -0.0273,
1142
+ 0.1552
1143
+ ],
1144
+ "out.57": [
1145
+ 0.5859,
1146
+ -0.7236,
1147
+ 0.013,
1148
+ -0.2133,
1149
+ 0.1999,
1150
+ 0.1443,
1151
+ -0.0126,
1152
+ 0.1627
1153
+ ],
1154
+ "out.58": [
1155
+ 0.3938,
1156
+ -0.8,
1157
+ -0.02,
1158
+ -0.1059,
1159
+ 0.3746,
1160
+ 0.1401,
1161
+ -0.025,
1162
+ 0.181
1163
+ ],
1164
+ "out.59": [
1165
+ 0.4304,
1166
+ -0.7927,
1167
+ -0.0173,
1168
+ -0.1496,
1169
+ 0.37,
1170
+ 0.1333,
1171
+ 0.0357,
1172
+ 0.0885
1173
+ ],
1174
+ "out.60": [
1175
+ 0.4945,
1176
+ -0.2772,
1177
+ -0.0573,
1178
+ -0.3919,
1179
+ 0.5433,
1180
+ 0.198,
1181
+ -0.0143,
1182
+ -0.4327
1183
+ ],
1184
+ "out.61": [
1185
+ 0.3585,
1186
+ -0.2174,
1187
+ -0.0337,
1188
+ -0.3522,
1189
+ 0.6598,
1190
+ 0.1476,
1191
+ 0.0659,
1192
+ -0.4873
1193
+ ],
1194
+ "out.62": [
1195
+ 0.2117,
1196
+ -0.2984,
1197
+ 0.2719,
1198
+ -0.7629,
1199
+ 0.4186,
1200
+ 0.1833,
1201
+ -0.0345,
1202
+ 0.0158
1203
+ ],
1204
+ "out.63": [
1205
+ -0.1238,
1206
+ -0.396,
1207
+ 0.3333,
1208
+ -0.6169,
1209
+ 0.4289,
1210
+ 0.3635,
1211
+ -0.0389,
1212
+ 0.1363
1213
+ ],
1214
+ "out.64": [
1215
+ -0.3856,
1216
+ -0.0232,
1217
+ 0.1997,
1218
+ -0.68,
1219
+ 0.2947,
1220
+ 0.3339,
1221
+ 0.0374,
1222
+ 0.3858
1223
+ ],
1224
+ "out.65": [
1225
+ 0.1543,
1226
+ -0.037,
1227
+ 0.1237,
1228
+ -0.832,
1229
+ 0.5021,
1230
+ 0.016,
1231
+ 0.1021,
1232
+ 0.0671
1233
+ ],
1234
+ "out.66": [
1235
+ 0.4529,
1236
+ 0.1859,
1237
+ 0.1994,
1238
+ -0.8403,
1239
+ 0.0954,
1240
+ 0.0424,
1241
+ -0.0013,
1242
+ -0.0594
1243
+ ],
1244
+ "out.67": [
1245
+ 0.6267,
1246
+ 0.2412,
1247
+ 0.0727,
1248
+ -0.6854,
1249
+ 0.0654,
1250
+ 0.0975,
1251
+ -0.1351,
1252
+ -0.2048
1253
+ ],
1254
+ "out.68": [
1255
+ 0.7988,
1256
+ 0.0338,
1257
+ -0.0868,
1258
+ -0.1257,
1259
+ 0.1183,
1260
+ 0.1276,
1261
+ -0.4874,
1262
+ -0.264
1263
+ ],
1264
+ "out.69": [
1265
+ 0.8221,
1266
+ 0.0564,
1267
+ -0.1201,
1268
+ -0.3344,
1269
+ 0.0932,
1270
+ 0.2474,
1271
+ -0.2949,
1272
+ -0.1944
1273
+ ],
1274
+ "out.70": [
1275
+ 0.2206,
1276
+ -0.0573,
1277
+ 0.1605,
1278
+ -0.6597,
1279
+ -0.3658,
1280
+ 0.1033,
1281
+ 0.5708,
1282
+ 0.1294
1283
+ ]
1284
+ },
1285
+ "weights_shuffled_labels": {
1286
+ "emb": [
1287
+ 0.0925,
1288
+ -0.0753,
1289
+ 0.0796,
1290
+ 0.119,
1291
+ 0.3115,
1292
+ 0.0092,
1293
+ 0.0607,
1294
+ -0.9298
1295
+ ],
1296
+ "in.00": [
1297
+ 0.1948,
1298
+ -0.2513,
1299
+ -0.0908,
1300
+ 0.1979,
1301
+ 0.3474,
1302
+ -0.0008,
1303
+ 0.2807,
1304
+ -0.8075
1305
+ ],
1306
+ "in.01": [
1307
+ -0.037,
1308
+ -0.1952,
1309
+ -0.1094,
1310
+ 0.2444,
1311
+ 0.496,
1312
+ -0.0046,
1313
+ 0.2559,
1314
+ -0.7598
1315
+ ],
1316
+ "in.02": [
1317
+ 0.2944,
1318
+ 0.1407,
1319
+ 0.0496,
1320
+ 0.1402,
1321
+ -0.1091,
1322
+ 0.0153,
1323
+ 0.0516,
1324
+ -0.9256
1325
+ ],
1326
+ "in.03": [
1327
+ 0.1905,
1328
+ 0.1293,
1329
+ 0.0616,
1330
+ 0.1575,
1331
+ -0.0255,
1332
+ -0.0008,
1333
+ 0.0849,
1334
+ -0.9542
1335
+ ],
1336
+ "in.04": [
1337
+ 0.6878,
1338
+ 0.0882,
1339
+ 0.0434,
1340
+ 0.1511,
1341
+ -0.0979,
1342
+ -0.0054,
1343
+ -0.018,
1344
+ -0.696
1345
+ ],
1346
+ "in.05": [
1347
+ 0.913,
1348
+ 0.2129,
1349
+ 0.0551,
1350
+ 0.0497,
1351
+ -0.3286,
1352
+ -0.0022,
1353
+ -0.0835,
1354
+ 0.0236
1355
+ ],
1356
+ "in.06": [
1357
+ 0.9621,
1358
+ 0.0543,
1359
+ -0.0413,
1360
+ 0.1633,
1361
+ -0.113,
1362
+ 0.0536,
1363
+ -0.1066,
1364
+ -0.1269
1365
+ ],
1366
+ "in.07": [
1367
+ 0.9435,
1368
+ 0.1358,
1369
+ -0.0038,
1370
+ 0.1671,
1371
+ -0.2187,
1372
+ 0.0535,
1373
+ -0.0909,
1374
+ -0.067
1375
+ ],
1376
+ "in.08": [
1377
+ 0.7592,
1378
+ 0.2315,
1379
+ 0.0403,
1380
+ 0.4073,
1381
+ 0.1052,
1382
+ 0.0958,
1383
+ -0.1063,
1384
+ -0.4135
1385
+ ],
1386
+ "in.09": [
1387
+ 0.6826,
1388
+ 0.3954,
1389
+ 0.1118,
1390
+ 0.5315,
1391
+ -0.0119,
1392
+ 0.0997,
1393
+ -0.2679,
1394
+ -0.0287
1395
+ ],
1396
+ "in.10": [
1397
+ 0.5542,
1398
+ 0.5136,
1399
+ 0.1224,
1400
+ 0.3282,
1401
+ -0.3299,
1402
+ 0.0608,
1403
+ -0.2713,
1404
+ 0.3466
1405
+ ],
1406
+ "in.11": [
1407
+ 0.481,
1408
+ 0.4683,
1409
+ 0.0611,
1410
+ 0.0519,
1411
+ -0.3444,
1412
+ 0.0492,
1413
+ -0.1969,
1414
+ 0.619
1415
+ ],
1416
+ "in.12": [
1417
+ 0.6301,
1418
+ 0.4389,
1419
+ 0.0726,
1420
+ 0.0484,
1421
+ -0.4479,
1422
+ 0.0505,
1423
+ -0.171,
1424
+ 0.4126
1425
+ ],
1426
+ "in.13": [
1427
+ 0.6882,
1428
+ 0.4099,
1429
+ 0.0618,
1430
+ 0.0288,
1431
+ -0.4961,
1432
+ 0.0579,
1433
+ -0.1737,
1434
+ 0.2722
1435
+ ],
1436
+ "in.14": [
1437
+ 0.9441,
1438
+ 0.0059,
1439
+ -0.1627,
1440
+ 0.1508,
1441
+ 0.0311,
1442
+ 0.1532,
1443
+ -0.0939,
1444
+ 0.1618
1445
+ ],
1446
+ "x": [
1447
+ 0.7389,
1448
+ 0.3015,
1449
+ -0.1767,
1450
+ 0.2131,
1451
+ -0.4037,
1452
+ 0.2181,
1453
+ -0.1001,
1454
+ -0.2566
1455
+ ],
1456
+ "u.1": [
1457
+ 0.4097,
1458
+ 0.6095,
1459
+ -0.0606,
1460
+ -0.1193,
1461
+ -0.6159,
1462
+ 0.1102,
1463
+ -0.1234,
1464
+ -0.19
1465
+ ],
1466
+ "rec.1.16": [
1467
+ 0.623,
1468
+ 0.399,
1469
+ 0.0006,
1470
+ -0.2565,
1471
+ -0.4893,
1472
+ 0.0792,
1473
+ -0.1938,
1474
+ 0.322
1475
+ ],
1476
+ "rec.1.17": [
1477
+ 0.6435,
1478
+ 0.3598,
1479
+ -0.066,
1480
+ -0.2347,
1481
+ -0.5518,
1482
+ 0.0816,
1483
+ -0.126,
1484
+ 0.2646
1485
+ ],
1486
+ "rec.1.18": [
1487
+ 0.6161,
1488
+ 0.4016,
1489
+ -0.0042,
1490
+ -0.2682,
1491
+ -0.5535,
1492
+ 0.066,
1493
+ -0.1495,
1494
+ 0.2325
1495
+ ],
1496
+ "rec.1.19": [
1497
+ 0.8594,
1498
+ -0.1541,
1499
+ 0.0839,
1500
+ 0.115,
1501
+ -0.3398,
1502
+ 0.0012,
1503
+ 0.1546,
1504
+ -0.2795
1505
+ ],
1506
+ "rec.1.20": [
1507
+ 0.7497,
1508
+ -0.1084,
1509
+ 0.1256,
1510
+ 0.1925,
1511
+ -0.297,
1512
+ -0.0223,
1513
+ 0.2323,
1514
+ -0.4803
1515
+ ],
1516
+ "rec.1.21": [
1517
+ 0.5126,
1518
+ -0.2218,
1519
+ 0.2033,
1520
+ 0.4485,
1521
+ -0.184,
1522
+ -0.0779,
1523
+ 0.3743,
1524
+ -0.5154
1525
+ ],
1526
+ "rec.1.22": [
1527
+ 0.6168,
1528
+ -0.2907,
1529
+ 0.1647,
1530
+ 0.3568,
1531
+ -0.0502,
1532
+ -0.0306,
1533
+ 0.2468,
1534
+ -0.5623
1535
+ ],
1536
+ "rec.1.23": [
1537
+ 0.8413,
1538
+ -0.327,
1539
+ 0.1142,
1540
+ 0.1884,
1541
+ -0.0963,
1542
+ 0.056,
1543
+ 0.0049,
1544
+ -0.3526
1545
+ ],
1546
+ "rec.1.24": [
1547
+ 0.7432,
1548
+ -0.3079,
1549
+ 0.1345,
1550
+ 0.2688,
1551
+ -0.1182,
1552
+ -0.0033,
1553
+ 0.1873,
1554
+ -0.4619
1555
+ ],
1556
+ "rec.1.25": [
1557
+ 0.7242,
1558
+ -0.3694,
1559
+ 0.1675,
1560
+ 0.2994,
1561
+ -0.0877,
1562
+ -0.005,
1563
+ 0.185,
1564
+ -0.4236
1565
+ ],
1566
+ "rec.1.26": [
1567
+ 0.7904,
1568
+ -0.3059,
1569
+ 0.1319,
1570
+ 0.229,
1571
+ -0.1829,
1572
+ -0.0071,
1573
+ 0.1738,
1574
+ -0.3849
1575
+ ],
1576
+ "rec.1.27": [
1577
+ 0.776,
1578
+ -0.3748,
1579
+ 0.1321,
1580
+ 0.2486,
1581
+ -0.1755,
1582
+ -0.0172,
1583
+ 0.223,
1584
+ -0.3118
1585
+ ],
1586
+ "rec.1.28": [
1587
+ 0.7132,
1588
+ -0.4049,
1589
+ 0.1182,
1590
+ 0.2557,
1591
+ -0.0054,
1592
+ -0.0084,
1593
+ 0.2245,
1594
+ -0.4444
1595
+ ],
1596
+ "rec.1.29": [
1597
+ 0.726,
1598
+ -0.3393,
1599
+ 0.1296,
1600
+ 0.2518,
1601
+ -0.0859,
1602
+ -0.0104,
1603
+ 0.2446,
1604
+ -0.4585
1605
+ ],
1606
+ "rec.1.30": [
1607
+ 0.6903,
1608
+ 0.3662,
1609
+ 0.0298,
1610
+ -0.4333,
1611
+ -0.4264,
1612
+ 0.0342,
1613
+ 0.0048,
1614
+ 0.1329
1615
+ ],
1616
+ "rec.1.31": [
1617
+ 0.7101,
1618
+ 0.3367,
1619
+ 0.0414,
1620
+ -0.4414,
1621
+ -0.4104,
1622
+ 0.0383,
1623
+ -0.0075,
1624
+ 0.1261
1625
+ ],
1626
+ "rec.1.32": [
1627
+ 0.7394,
1628
+ 0.3079,
1629
+ 0.0338,
1630
+ -0.4792,
1631
+ -0.3165,
1632
+ 0.0527,
1633
+ -0.0175,
1634
+ 0.1564
1635
+ ],
1636
+ "rec.1.33": [
1637
+ 0.7204,
1638
+ 0.3044,
1639
+ 0.0386,
1640
+ -0.5149,
1641
+ -0.2886,
1642
+ 0.058,
1643
+ -0.046,
1644
+ 0.1813
1645
+ ],
1646
+ "rec.1.34": [
1647
+ 0.77,
1648
+ 0.2603,
1649
+ 0.0356,
1650
+ -0.5018,
1651
+ -0.2341,
1652
+ 0.0698,
1653
+ -0.0533,
1654
+ 0.1542
1655
+ ],
1656
+ "rec.1.35": [
1657
+ 0.685,
1658
+ 0.3549,
1659
+ 0.0112,
1660
+ -0.4906,
1661
+ -0.2926,
1662
+ 0.0748,
1663
+ -0.1027,
1664
+ 0.2497
1665
+ ],
1666
+ "rec.1.36": [
1667
+ 0.5623,
1668
+ 0.4508,
1669
+ 0.0389,
1670
+ -0.5575,
1671
+ -0.2719,
1672
+ 0.0678,
1673
+ -0.1831,
1674
+ 0.2371
1675
+ ],
1676
+ "rec.1.37": [
1677
+ 0.5599,
1678
+ 0.4557,
1679
+ 0.0287,
1680
+ -0.5272,
1681
+ -0.324,
1682
+ 0.0711,
1683
+ -0.1795,
1684
+ 0.2403
1685
+ ],
1686
+ "rec.1.38": [
1687
+ 0.5194,
1688
+ 0.4901,
1689
+ 0.034,
1690
+ -0.5565,
1691
+ -0.285,
1692
+ 0.0761,
1693
+ -0.1978,
1694
+ 0.2305
1695
+ ],
1696
+ "rec.1.39": [
1697
+ 0.5227,
1698
+ 0.4757,
1699
+ 0.0345,
1700
+ -0.5639,
1701
+ -0.2748,
1702
+ 0.0774,
1703
+ -0.1882,
1704
+ 0.2538
1705
+ ],
1706
+ "rec.1.40": [
1707
+ 0.4846,
1708
+ 0.5266,
1709
+ 0.044,
1710
+ -0.5398,
1711
+ -0.2947,
1712
+ 0.0797,
1713
+ -0.2358,
1714
+ 0.2138
1715
+ ],
1716
+ "rec.1.41": [
1717
+ 0.4848,
1718
+ 0.5234,
1719
+ 0.045,
1720
+ -0.551,
1721
+ -0.2946,
1722
+ 0.0815,
1723
+ -0.2275,
1724
+ 0.2006
1725
+ ],
1726
+ "rec.1.42": [
1727
+ 0.473,
1728
+ 0.5243,
1729
+ 0.0507,
1730
+ -0.5613,
1731
+ -0.2422,
1732
+ 0.0882,
1733
+ -0.2607,
1734
+ 0.2222
1735
+ ],
1736
+ "rec.1.43": [
1737
+ 0.4899,
1738
+ 0.497,
1739
+ 0.0525,
1740
+ -0.5654,
1741
+ -0.2326,
1742
+ 0.087,
1743
+ -0.2652,
1744
+ 0.2417
1745
+ ],
1746
+ "rec.1.44": [
1747
+ 0.5007,
1748
+ 0.4917,
1749
+ 0.042,
1750
+ -0.5543,
1751
+ -0.2164,
1752
+ 0.0929,
1753
+ -0.2723,
1754
+ 0.2625
1755
+ ],
1756
+ "rec.1.45": [
1757
+ 0.5084,
1758
+ 0.4938,
1759
+ 0.0312,
1760
+ -0.5451,
1761
+ -0.2546,
1762
+ 0.0836,
1763
+ -0.2397,
1764
+ 0.2652
1765
+ ],
1766
+ "rec.1.46": [
1767
+ 0.5294,
1768
+ 0.4343,
1769
+ 0.0504,
1770
+ -0.5984,
1771
+ -0.1588,
1772
+ 0.0947,
1773
+ -0.2499,
1774
+ 0.2717
1775
+ ],
1776
+ "rec.1.47": [
1777
+ 0.5147,
1778
+ 0.4477,
1779
+ 0.0485,
1780
+ -0.6113,
1781
+ -0.1549,
1782
+ 0.0819,
1783
+ -0.2276,
1784
+ 0.2758
1785
+ ],
1786
+ "rec.1.48": [
1787
+ 0.5063,
1788
+ 0.4381,
1789
+ 0.0482,
1790
+ -0.6371,
1791
+ -0.1394,
1792
+ 0.0837,
1793
+ -0.2114,
1794
+ 0.269
1795
+ ],
1796
+ "rec.1.49": [
1797
+ 0.4607,
1798
+ 0.461,
1799
+ 0.0372,
1800
+ -0.6428,
1801
+ -0.1344,
1802
+ 0.079,
1803
+ -0.21,
1804
+ 0.3039
1805
+ ],
1806
+ "rec.1.50": [
1807
+ 0.6289,
1808
+ 0.2982,
1809
+ 0.0313,
1810
+ -0.6075,
1811
+ -0.0994,
1812
+ 0.0997,
1813
+ -0.2007,
1814
+ 0.2923
1815
+ ],
1816
+ "rec.1.51": [
1817
+ 0.6512,
1818
+ 0.2699,
1819
+ 0.0399,
1820
+ -0.6083,
1821
+ -0.0823,
1822
+ 0.1059,
1823
+ -0.2073,
1824
+ 0.2654
1825
+ ],
1826
+ "rec.1.52": [
1827
+ 0.6316,
1828
+ 0.3374,
1829
+ 0.0141,
1830
+ -0.5902,
1831
+ -0.1256,
1832
+ 0.1382,
1833
+ -0.2101,
1834
+ 0.2444
1835
+ ],
1836
+ "rec.1.53": [
1837
+ 0.7187,
1838
+ 0.2505,
1839
+ 0.011,
1840
+ -0.5569,
1841
+ -0.0806,
1842
+ 0.1487,
1843
+ -0.1624,
1844
+ 0.2354
1845
+ ],
1846
+ "rec.1.54": [
1847
+ 0.6973,
1848
+ -0.0093,
1849
+ -0.019,
1850
+ -0.6054,
1851
+ 0.1728,
1852
+ 0.1231,
1853
+ -0.1778,
1854
+ 0.265
1855
+ ],
1856
+ "s.1": [
1857
+ 0.7171,
1858
+ -0.1229,
1859
+ -0.0095,
1860
+ -0.5744,
1861
+ 0.2286,
1862
+ 0.1063,
1863
+ -0.1853,
1864
+ 0.2069
1865
+ ],
1866
+ "u.2": [
1867
+ 0.8649,
1868
+ 0.2381,
1869
+ -0.146,
1870
+ -0.1791,
1871
+ -0.2055,
1872
+ 0.1942,
1873
+ -0.1184,
1874
+ 0.2187
1875
+ ],
1876
+ "rec.2.16": [
1877
+ 0.7395,
1878
+ 0.3077,
1879
+ 0.031,
1880
+ -0.3799,
1881
+ -0.2567,
1882
+ 0.0504,
1883
+ -0.1432,
1884
+ 0.3525
1885
+ ],
1886
+ "rec.2.17": [
1887
+ 0.7366,
1888
+ 0.1899,
1889
+ -0.0891,
1890
+ -0.3525,
1891
+ -0.2052,
1892
+ 0.0701,
1893
+ -0.1003,
1894
+ 0.4818
1895
+ ],
1896
+ "rec.2.18": [
1897
+ 0.7378,
1898
+ 0.31,
1899
+ 0.0394,
1900
+ -0.3975,
1901
+ -0.2948,
1902
+ 0.0446,
1903
+ -0.1283,
1904
+ 0.3078
1905
+ ],
1906
+ "rec.2.19": [
1907
+ 0.7306,
1908
+ 0.2793,
1909
+ -0.013,
1910
+ -0.4164,
1911
+ -0.3308,
1912
+ 0.0605,
1913
+ -0.0563,
1914
+ 0.3135
1915
+ ],
1916
+ "rec.2.20": [
1917
+ 0.7391,
1918
+ 0.1029,
1919
+ -0.1231,
1920
+ -0.3513,
1921
+ -0.1252,
1922
+ 0.0853,
1923
+ -0.0645,
1924
+ 0.5268
1925
+ ],
1926
+ "rec.2.21": [
1927
+ 0.7641,
1928
+ 0.278,
1929
+ 0.0468,
1930
+ -0.4579,
1931
+ -0.2582,
1932
+ 0.0546,
1933
+ -0.079,
1934
+ 0.226
1935
+ ],
1936
+ "rec.2.22": [
1937
+ 0.7602,
1938
+ -0.1216,
1939
+ -0.1471,
1940
+ -0.3072,
1941
+ 0.1142,
1942
+ 0.0906,
1943
+ -0.0681,
1944
+ 0.5152
1945
+ ],
1946
+ "rec.2.23": [
1947
+ 0.7245,
1948
+ -0.1828,
1949
+ -0.2076,
1950
+ -0.2825,
1951
+ 0.1588,
1952
+ 0.111,
1953
+ -0.0821,
1954
+ 0.5239
1955
+ ],
1956
+ "rec.2.24": [
1957
+ 0.7562,
1958
+ -0.1164,
1959
+ -0.1573,
1960
+ -0.3338,
1961
+ 0.0931,
1962
+ 0.0997,
1963
+ -0.0665,
1964
+ 0.5054
1965
+ ],
1966
+ "rec.2.25": [
1967
+ 0.7246,
1968
+ -0.0884,
1969
+ -0.179,
1970
+ -0.3697,
1971
+ 0.0646,
1972
+ 0.1003,
1973
+ -0.0637,
1974
+ 0.5292
1975
+ ],
1976
+ "rec.2.26": [
1977
+ 0.7771,
1978
+ 0.1597,
1979
+ 0.0484,
1980
+ -0.5282,
1981
+ -0.1654,
1982
+ 0.0572,
1983
+ -0.0754,
1984
+ 0.2302
1985
+ ],
1986
+ "rec.2.27": [
1987
+ 0.7499,
1988
+ -0.064,
1989
+ -0.1334,
1990
+ -0.3929,
1991
+ 0.0719,
1992
+ 0.0809,
1993
+ -0.0713,
1994
+ 0.4946
1995
+ ],
1996
+ "rec.2.28": [
1997
+ 0.7696,
1998
+ 0.0359,
1999
+ 0.0379,
2000
+ -0.5869,
2001
+ 0.0875,
2002
+ 0.065,
2003
+ -0.0837,
2004
+ 0.2042
2005
+ ],
2006
+ "rec.2.29": [
2007
+ 0.7433,
2008
+ 0.143,
2009
+ 0.034,
2010
+ -0.5996,
2011
+ -0.0194,
2012
+ 0.0672,
2013
+ -0.0905,
2014
+ 0.2306
2015
+ ],
2016
+ "rec.2.30": [
2017
+ 0.7337,
2018
+ 0.0219,
2019
+ 0.0147,
2020
+ -0.5786,
2021
+ 0.1847,
2022
+ 0.0815,
2023
+ -0.0425,
2024
+ 0.2891
2025
+ ],
2026
+ "rec.2.31": [
2027
+ 0.731,
2028
+ -0.1352,
2029
+ 0.0281,
2030
+ -0.5341,
2031
+ 0.2996,
2032
+ 0.0939,
2033
+ -0.0478,
2034
+ 0.2459
2035
+ ],
2036
+ "rec.2.32": [
2037
+ 0.6709,
2038
+ -0.0416,
2039
+ 0.0071,
2040
+ -0.5927,
2041
+ 0.3048,
2042
+ 0.0944,
2043
+ -0.0434,
2044
+ 0.3052
2045
+ ],
2046
+ "rec.2.33": [
2047
+ 0.6601,
2048
+ -0.0133,
2049
+ 0.0255,
2050
+ -0.63,
2051
+ 0.2622,
2052
+ 0.0948,
2053
+ -0.0555,
2054
+ 0.2928
2055
+ ],
2056
+ "rec.2.34": [
2057
+ 0.6623,
2058
+ 0.0176,
2059
+ 0.0048,
2060
+ -0.6189,
2061
+ 0.2765,
2062
+ 0.1015,
2063
+ -0.0517,
2064
+ 0.2976
2065
+ ],
2066
+ "rec.2.35": [
2067
+ 0.705,
2068
+ 0.0359,
2069
+ -0.0095,
2070
+ -0.5779,
2071
+ 0.1968,
2072
+ 0.133,
2073
+ -0.0935,
2074
+ 0.32
2075
+ ],
2076
+ "rec.2.36": [
2077
+ 0.5626,
2078
+ 0.277,
2079
+ 0.0357,
2080
+ -0.7037,
2081
+ 0.0554,
2082
+ 0.1234,
2083
+ -0.1178,
2084
+ 0.2796
2085
+ ],
2086
+ "rec.2.37": [
2087
+ 0.5561,
2088
+ 0.3275,
2089
+ 0.0452,
2090
+ -0.6939,
2091
+ -0.049,
2092
+ 0.1162,
2093
+ -0.1141,
2094
+ 0.2666
2095
+ ],
2096
+ "rec.2.38": [
2097
+ 0.5256,
2098
+ 0.4096,
2099
+ 0.0526,
2100
+ -0.673,
2101
+ -0.1511,
2102
+ 0.1165,
2103
+ -0.0987,
2104
+ 0.2327
2105
+ ],
2106
+ "rec.2.39": [
2107
+ 0.5316,
2108
+ 0.4691,
2109
+ 0.0607,
2110
+ -0.5626,
2111
+ -0.336,
2112
+ 0.1209,
2113
+ -0.0977,
2114
+ 0.2004
2115
+ ],
2116
+ "rec.2.40": [
2117
+ 0.4141,
2118
+ 0.592,
2119
+ 0.0662,
2120
+ -0.4776,
2121
+ -0.465,
2122
+ 0.0901,
2123
+ -0.0649,
2124
+ 0.1302
2125
+ ],
2126
+ "rec.2.41": [
2127
+ 0.3841,
2128
+ 0.6009,
2129
+ 0.0682,
2130
+ -0.4952,
2131
+ -0.4692,
2132
+ 0.0832,
2133
+ -0.0343,
2134
+ 0.1154
2135
+ ],
2136
+ "rec.2.42": [
2137
+ 0.3523,
2138
+ 0.6286,
2139
+ 0.0796,
2140
+ -0.545,
2141
+ -0.391,
2142
+ 0.0946,
2143
+ -0.0444,
2144
+ 0.1168
2145
+ ],
2146
+ "rec.2.43": [
2147
+ 0.3569,
2148
+ 0.6252,
2149
+ 0.0931,
2150
+ -0.5124,
2151
+ -0.4384,
2152
+ 0.0958,
2153
+ -0.0284,
2154
+ 0.0909
2155
+ ],
2156
+ "rec.2.44": [
2157
+ 0.3855,
2158
+ 0.62,
2159
+ 0.0883,
2160
+ -0.5378,
2161
+ -0.3824,
2162
+ 0.0995,
2163
+ -0.0494,
2164
+ 0.1068
2165
+ ],
2166
+ "rec.2.45": [
2167
+ 0.3623,
2168
+ 0.6307,
2169
+ 0.0746,
2170
+ -0.4773,
2171
+ -0.4733,
2172
+ 0.0825,
2173
+ -0.0192,
2174
+ 0.0802
2175
+ ],
2176
+ "rec.2.46": [
2177
+ 0.2149,
2178
+ 0.6539,
2179
+ 0.0541,
2180
+ -0.5536,
2181
+ -0.4365,
2182
+ 0.0115,
2183
+ 0.117,
2184
+ 0.1121
2185
+ ],
2186
+ "rec.2.47": [
2187
+ 0.1606,
2188
+ 0.6661,
2189
+ 0.0473,
2190
+ -0.5607,
2191
+ -0.4182,
2192
+ -0.0303,
2193
+ 0.1639,
2194
+ 0.1056
2195
+ ],
2196
+ "rec.2.48": [
2197
+ 0.185,
2198
+ 0.647,
2199
+ 0.0567,
2200
+ -0.5876,
2201
+ -0.4159,
2202
+ 0.0308,
2203
+ 0.1129,
2204
+ 0.1095
2205
+ ],
2206
+ "rec.2.49": [
2207
+ 0.1364,
2208
+ 0.6526,
2209
+ 0.0459,
2210
+ -0.5982,
2211
+ -0.4117,
2212
+ 0.0275,
2213
+ 0.1095,
2214
+ 0.1156
2215
+ ],
2216
+ "rec.2.50": [
2217
+ 0.3022,
2218
+ 0.4719,
2219
+ 0.0785,
2220
+ -0.7445,
2221
+ -0.2732,
2222
+ 0.0934,
2223
+ 0.1455,
2224
+ 0.1452
2225
+ ],
2226
+ "rec.2.51": [
2227
+ 0.4139,
2228
+ 0.3099,
2229
+ 0.1041,
2230
+ -0.7693,
2231
+ -0.2113,
2232
+ 0.1593,
2233
+ 0.2359,
2234
+ 0.0658
2235
+ ],
2236
+ "rec.2.52": [
2237
+ 0.4769,
2238
+ 0.2545,
2239
+ 0.0987,
2240
+ -0.5882,
2241
+ -0.1142,
2242
+ 0.2153,
2243
+ 0.3293,
2244
+ -0.4293
2245
+ ],
2246
+ "rec.2.53": [
2247
+ 0.3808,
2248
+ 0.5166,
2249
+ 0.0126,
2250
+ -0.5263,
2251
+ -0.0185,
2252
+ 0.0523,
2253
+ 0.1824,
2254
+ -0.5241
2255
+ ],
2256
+ "rec.2.54": [
2257
+ 0.2117,
2258
+ 0.152,
2259
+ 0.0356,
2260
+ -0.7721,
2261
+ 0.1778,
2262
+ 0.1757,
2263
+ 0.3644,
2264
+ -0.3734
2265
+ ],
2266
+ "s.2": [
2267
+ -0.0814,
2268
+ 0.1765,
2269
+ 0.0713,
2270
+ -0.8992,
2271
+ 0.1995,
2272
+ 0.0527,
2273
+ 0.2285,
2274
+ 0.2318
2275
+ ],
2276
+ "out.56": [
2277
+ -0.5499,
2278
+ -0.2238,
2279
+ 0.1802,
2280
+ -0.4029,
2281
+ -0.043,
2282
+ 0.112,
2283
+ 0.5957,
2284
+ 0.2889
2285
+ ],
2286
+ "out.57": [
2287
+ -0.5148,
2288
+ -0.3069,
2289
+ 0.1874,
2290
+ -0.2909,
2291
+ -0.2005,
2292
+ 0.144,
2293
+ 0.5755,
2294
+ 0.3589
2295
+ ],
2296
+ "out.58": [
2297
+ -0.6227,
2298
+ -0.3668,
2299
+ 0.1841,
2300
+ 0.0163,
2301
+ -0.0616,
2302
+ -0.0516,
2303
+ 0.576,
2304
+ 0.3245
2305
+ ],
2306
+ "out.59": [
2307
+ -0.5704,
2308
+ -0.4759,
2309
+ 0.1603,
2310
+ 0.1052,
2311
+ -0.142,
2312
+ 0.0518,
2313
+ 0.5097,
2314
+ 0.3588
2315
+ ],
2316
+ "out.60": [
2317
+ -0.2816,
2318
+ -0.2781,
2319
+ 0.0797,
2320
+ 0.1895,
2321
+ -0.4174,
2322
+ -0.2376,
2323
+ 0.7174,
2324
+ 0.236
2325
+ ],
2326
+ "out.61": [
2327
+ -0.3842,
2328
+ -0.3742,
2329
+ 0.095,
2330
+ 0.0752,
2331
+ -0.4314,
2332
+ -0.0357,
2333
+ 0.6664,
2334
+ 0.2572
2335
+ ],
2336
+ "out.62": [
2337
+ -0.3823,
2338
+ -0.2985,
2339
+ 0.2659,
2340
+ -0.0659,
2341
+ -0.5726,
2342
+ 0.1991,
2343
+ 0.4847,
2344
+ 0.2955
2345
+ ],
2346
+ "out.63": [
2347
+ -0.4132,
2348
+ -0.2914,
2349
+ 0.3451,
2350
+ -0.0672,
2351
+ -0.554,
2352
+ 0.2223,
2353
+ 0.4275,
2354
+ 0.2858
2355
+ ],
2356
+ "out.64": [
2357
+ -0.2935,
2358
+ -0.3169,
2359
+ 0.2121,
2360
+ -0.1009,
2361
+ -0.5811,
2362
+ 0.2209,
2363
+ 0.4082,
2364
+ 0.4529
2365
+ ],
2366
+ "out.65": [
2367
+ -0.3181,
2368
+ -0.3938,
2369
+ 0.2051,
2370
+ -0.1113,
2371
+ -0.5641,
2372
+ 0.2116,
2373
+ 0.4735,
2374
+ 0.3194
2375
+ ],
2376
+ "out.66": [
2377
+ -0.3426,
2378
+ -0.296,
2379
+ 0.3131,
2380
+ -0.0251,
2381
+ -0.59,
2382
+ 0.1302,
2383
+ 0.4067,
2384
+ 0.4074
2385
+ ],
2386
+ "out.67": [
2387
+ -0.2388,
2388
+ -0.3856,
2389
+ 0.3382,
2390
+ -0.1089,
2391
+ -0.5669,
2392
+ 0.0634,
2393
+ 0.4801,
2394
+ 0.3349
2395
+ ],
2396
+ "out.68": [
2397
+ 0.5145,
2398
+ 0.1684,
2399
+ 0.544,
2400
+ -0.3927,
2401
+ -0.035,
2402
+ -0.3193,
2403
+ -0.0516,
2404
+ -0.3884
2405
+ ],
2406
+ "out.69": [
2407
+ 0.7339,
2408
+ 0.1894,
2409
+ 0.0255,
2410
+ -0.3348,
2411
+ 0.1234,
2412
+ 0.0844,
2413
+ 0.0495,
2414
+ -0.5367
2415
+ ],
2416
+ "out.70": [
2417
+ 0.0913,
2418
+ -0.2626,
2419
+ 0.2553,
2420
+ -0.5138,
2421
+ -0.4692,
2422
+ 0.1795,
2423
+ 0.5024,
2424
+ 0.2978
2425
+ ]
2426
+ },
2427
+ "results": {
2428
+ "logit lens": {
2429
+ "recall10_seven": 0.18927536231884057,
2430
+ "recall1_seven": 0.07163043478260869,
2431
+ "recall10_by_task": {
2432
+ "multihop": 0.2,
2433
+ "multilingual": 0.41304347826086957,
2434
+ "typo": 0.3333333333333333,
2435
+ "association": 0.0,
2436
+ "poetry": 0.0
2437
+ },
2438
+ "recall10_any_node_in_block": {
2439
+ "input block": 0.0375,
2440
+ "loop 1": 0.18239130434782608,
2441
+ "loop 2": 0.18239130434782608,
2442
+ "output block": 0.24532204186022236
2443
+ },
2444
+ "recall10_per_node": {
2445
+ "in.14": 0.0125,
2446
+ "rec.1.28": 0.0,
2447
+ "rec.1.42": 0.04637681159420289,
2448
+ "rec.2.16": 0.029166666666666667,
2449
+ "rec.2.30": 0.008333333333333333,
2450
+ "rec.2.44": 0.03369565217391304,
2451
+ "out.58": 0.13710144927536233
2452
+ }
2453
+ },
2454
+ "J-lens (E001, 50 prompts)": {
2455
+ "recall10_seven": 0.2676323572907424,
2456
+ "recall1_seven": 0.12217391304347827,
2457
+ "recall10_by_task": {
2458
+ "multihop": 0.2,
2459
+ "multilingual": 0.5760869565217391,
2460
+ "typo": 0.5416666666666666,
2461
+ "association": 0.0,
2462
+ "poetry": 0.02040816326530612
2463
+ },
2464
+ "recall10_any_node_in_block": {
2465
+ "input block": 0.03333333333333333,
2466
+ "loop 1": 0.1869418202063434,
2467
+ "loop 2": 0.2964477965099083,
2468
+ "output block": 0.2887555456965395
2469
+ },
2470
+ "recall10_per_node": {
2471
+ "in.14": 0.008333333333333333,
2472
+ "rec.1.28": 0.004166666666666667,
2473
+ "rec.1.42": 0.01884057971014493,
2474
+ "rec.2.16": 0.029166666666666667,
2475
+ "rec.2.30": 0.07101449275362318,
2476
+ "rec.2.44": 0.1979946761313221,
2477
+ "out.58": 0.16681159420289854
2478
+ }
2479
+ },
2480
+ "R-lens (pooled LRP)": {
2481
+ "recall10_seven": 0.32132800946465545,
2482
+ "recall1_seven": 0.16792221236320617,
2483
+ "recall10_by_task": {
2484
+ "multihop": 0.3,
2485
+ "multilingual": 0.6195652173913043,
2486
+ "typo": 0.6666666666666666,
2487
+ "association": 0.0,
2488
+ "poetry": 0.02040816326530612
2489
+ },
2490
+ "recall10_any_node_in_block": {
2491
+ "input block": 0.13308823529411767,
2492
+ "loop 1": 0.13243862762496303,
2493
+ "loop 2": 0.3607468204673173,
2494
+ "output block": 0.28170289855072467
2495
+ },
2496
+ "recall10_per_node": {
2497
+ "in.14": 0.07083333333333333,
2498
+ "rec.1.28": 0.020833333333333336,
2499
+ "rec.1.42": 0.025277284826974267,
2500
+ "rec.2.16": 0.09583333333333334,
2501
+ "rec.2.30": 0.10452898550724639,
2502
+ "rec.2.44": 0.22118308192842356,
2503
+ "out.58": 0.20601449275362316
2504
+ }
2505
+ },
2506
+ "best single expert": {
2507
+ "recall10_seven": 0.3778587087008717,
2508
+ "recall1_seven": 0.1742838874680307,
2509
+ "recall10_by_task": {
2510
+ "multihop": 0.4,
2511
+ "multilingual": 0.6413043478260869,
2512
+ "typo": 0.7083333333333334,
2513
+ "association": 0.0784313725490196,
2514
+ "poetry": 0.061224489795918366
2515
+ },
2516
+ "recall10_any_node_in_block": {
2517
+ "input block": 0.14769820971867006,
2518
+ "loop 1": 0.24763279224733373,
2519
+ "loop 2": 0.41346399429336944,
2520
+ "output block": 0.2927983802216539
2521
+ },
2522
+ "recall10_per_node": {
2523
+ "in.14": 0.08351449275362319,
2524
+ "rec.1.28": 0.06243606138107416,
2525
+ "rec.1.42": 0.06105072463768115,
2526
+ "rec.2.16": 0.10845055413469737,
2527
+ "rec.2.30": 0.14606777493606138,
2528
+ "rec.2.44": 0.24160124919533027,
2529
+ "out.58": 0.26351449275362315
2530
+ }
2531
+ },
2532
+ "J++ shuffled labels": {
2533
+ "recall10_seven": 0.22480479148181015,
2534
+ "recall1_seven": 0.08369565217391303,
2535
+ "recall10_by_task": {
2536
+ "multihop": 0.2,
2537
+ "multilingual": 0.21739130434782608,
2538
+ "typo": 0.625,
2539
+ "association": 0.0,
2540
+ "poetry": 0.08163265306122448
2541
+ },
2542
+ "recall10_any_node_in_block": {
2543
+ "input block": 0.15184378099065715,
2544
+ "loop 1": 0.16059271534700836,
2545
+ "loop 2": 0.2991234319814882,
2546
+ "output block": 0.14858739147833047
2547
+ },
2548
+ "recall10_per_node": {
2549
+ "in.14": 0.058333333333333334,
2550
+ "rec.1.28": 0.07318840579710144,
2551
+ "rec.1.42": 0.03143670511682934,
2552
+ "rec.2.16": 0.07509612540668441,
2553
+ "rec.2.30": 0.04166666666666667,
2554
+ "rec.2.44": 0.12261978704525289,
2555
+ "out.58": 0.008333333333333333
2556
+ }
2557
+ },
2558
+ "J++": {
2559
+ "recall10_seven": 0.42311681194216816,
2560
+ "recall1_seven": 0.23251109139307896,
2561
+ "recall10_by_task": {
2562
+ "multihop": 0.4,
2563
+ "multilingual": 0.6413043478260869,
2564
+ "typo": 0.875,
2565
+ "association": 0.11764705882352941,
2566
+ "poetry": 0.08163265306122448
2567
+ },
2568
+ "recall10_any_node_in_block": {
2569
+ "input block": 0.16210592932825302,
2570
+ "loop 1": 0.258334203246516,
2571
+ "loop 2": 0.43875237051342275,
2572
+ "output block": 0.4740592758842667
2573
+ },
2574
+ "recall10_per_node": {
2575
+ "in.14": 0.1335144927536232,
2576
+ "rec.1.28": 0.1147911338448423,
2577
+ "rec.1.42": 0.13452750665483584,
2578
+ "rec.2.16": 0.14413895993179882,
2579
+ "rec.2.30": 0.15863171355498723,
2580
+ "rec.2.44": 0.2441052942916297,
2581
+ "out.58": 0.3032099361483723
2582
+ }
2583
+ },
2584
+ "J++ (full vocabulary)": {
2585
+ "recall10_seven": 0.38822763888163964,
2586
+ "recall1_seven": 0.17353815439219167,
2587
+ "recall10_by_task": {
2588
+ "multihop": 0.4,
2589
+ "multilingual": 0.5869565217391305,
2590
+ "typo": 0.8333333333333334,
2591
+ "association": 0.0392156862745098,
2592
+ "poetry": 0.08163265306122448
2593
+ },
2594
+ "recall10_any_node_in_block": {
2595
+ "input block": 0.13942881500426257,
2596
+ "loop 1": 0.1906334272839571,
2597
+ "loop 2": 0.392588470170677,
2598
+ "output block": 0.45010360666005533
2599
+ },
2600
+ "recall10_per_node": {
2601
+ "in.14": 0.11666666666666667,
2602
+ "rec.1.28": 0.06666666666666667,
2603
+ "rec.1.42": 0.06486616385684708,
2604
+ "rec.2.16": 0.1125,
2605
+ "rec.2.30": 0.1437766410912191,
2606
+ "rec.2.44": 0.19638790298728187,
2607
+ "out.58": 0.28709740417906293
2608
+ }
2609
+ },
2610
+ "J-lens (full vocabulary)": {
2611
+ "recall10_seven": 0.25911786453711916,
2612
+ "recall1_seven": 0.10278985507246377,
2613
+ "recall10_by_task": {
2614
+ "multihop": 0.2,
2615
+ "multilingual": 0.5543478260869565,
2616
+ "typo": 0.5208333333333334,
2617
+ "association": 0.0,
2618
+ "poetry": 0.02040816326530612
2619
+ },
2620
+ "recall10_any_node_in_block": {
2621
+ "input block": 0.020833333333333336,
2622
+ "loop 1": 0.0892798423717313,
2623
+ "loop 2": 0.2450961254066844,
2624
+ "output block": 0.28042221236320614
2625
+ },
2626
+ "recall10_per_node": {
2627
+ "in.14": 0.008333333333333333,
2628
+ "rec.1.28": 0.004166666666666667,
2629
+ "rec.1.42": 0.016666666666666666,
2630
+ "rec.2.16": 0.025,
2631
+ "rec.2.30": 0.06666666666666667,
2632
+ "rec.2.44": 0.15940771960958297,
2633
+ "out.58": 0.1626449275362319
2634
+ }
2635
+ }
2636
+ },
2637
+ "rlens_split_half_median_by_block": {
2638
+ "input block": 0.9900870323181152,
2639
+ "loop 1": 0.9890202283859253,
2640
+ "loop 2": 0.9944029450416565,
2641
+ "output block": 0.9994319081306458
2642
+ }
2643
+ }
fit_records/jpp_k2_n64_e8_s0/prompts.json ADDED
@@ -0,0 +1,258 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [
2
+ {
3
+ "i": 0,
4
+ "start": "Given the chaos , the governor of Yunnan ignored the imperial orders of evacuation . The Mongol army commanded by Prince Ye @"
5
+ },
6
+ {
7
+ "i": 1,
8
+ "start": "From the 1920s Australia 's defence thinking was dominated by British Imperial defence policy , which was embodied"
9
+ },
10
+ {
11
+ "i": 2,
12
+ "start": "Tim 's driving days started as a toddler when he was given a go @-@ kart that he often drove"
13
+ },
14
+ {
15
+ "i": 3,
16
+ "start": "As the Marines formed into a horseshoe @-@ shaped line around Hill 123 , Tamura 's"
17
+ },
18
+ {
19
+ "i": 4,
20
+ "start": "USA Today 's Scott Bowles awarded the film three out of four stars , labeling it a \" surprising comedy that rivals"
21
+ },
22
+ {
23
+ "i": 5,
24
+ "start": "Ringo Starr identified his initial inspiration for \" Back Off Boogaloo \" as having come from Marc Bolan ,"
25
+ },
26
+ {
27
+ "i": 6,
28
+ "start": "The ship 's secondary armament consisted of fourteen 45 @-@ calibre 6 @-@ inch ("
29
+ },
30
+ {
31
+ "i": 7,
32
+ "start": "Sources disagree about Radcliffe 's personal wealth ; he was reported to have earned \u00a3 1 million for the first Harry"
33
+ },
34
+ {
35
+ "i": 8,
36
+ "start": "In April 2010 , there were 6 @,@ 260 people employed in the healthcare field"
37
+ },
38
+ {
39
+ "i": 9,
40
+ "start": "\" Exit Through the Kwik @-@ E @-@ Mart \" is the fifteenth episode of the twenty @-"
41
+ },
42
+ {
43
+ "i": 10,
44
+ "start": "On 25 June 1950 , the North Korean Army ( KPA ) crossed the border into South"
45
+ },
46
+ {
47
+ "i": 11,
48
+ "start": "A British parliamentary motion put forward by Labour MP Kate Hoey in February 2013 to retrospectively condemn the"
49
+ },
50
+ {
51
+ "i": 12,
52
+ "start": "At the fort , Moultrie ordered his men to concentrate their fire on the two large man @-@ of @"
53
+ },
54
+ {
55
+ "i": 13,
56
+ "start": "In Part II of the series , Shikamaru is assigned the task of locating two members of the criminal organization Ak"
57
+ },
58
+ {
59
+ "i": 14,
60
+ "start": "The Ming dynasty had seen considerable advancement in the process of colour printing in China . At his studio , Hu Zhengyan experiment"
61
+ },
62
+ {
63
+ "i": 15,
64
+ "start": "\" Exit Through the Kwik @-@ E @-@ Mart \" was written by Marc Wilmore and directed by"
65
+ },
66
+ {
67
+ "i": 16,
68
+ "start": "In 629 Tikal founded Dos Pilas , some 110 kilometres ( 68 mi )"
69
+ },
70
+ {
71
+ "i": 17,
72
+ "start": "The British fleet weighed anchor at Cape Fear on May 31 , and arrived outside Charleston Harbor the next day . Moul"
73
+ },
74
+ {
75
+ "i": 18,
76
+ "start": "The initial phase saw the Australians advance towards the Hongorai River . Following the end of the early fighting , the Australian"
77
+ },
78
+ {
79
+ "i": 19,
80
+ "start": "The reactions of the native Aboriginal inhabitants to the sudden arrival of British settlers in Australia were varied , but were inevitably hostile when"
81
+ },
82
+ {
83
+ "i": 20,
84
+ "start": "The German offensive ground to a halt in mid @-@ July and a brief lull followed , during which the Australians"
85
+ },
86
+ {
87
+ "i": 21,
88
+ "start": "The city 's growth has reflected the push and pull of many social and economic factors . The total population increased in each"
89
+ },
90
+ {
91
+ "i": 22,
92
+ "start": "Following the classification of Tropical Depression Two @-@ E on June 19 , the Government of Mexico issued a tropical"
93
+ },
94
+ {
95
+ "i": 23,
96
+ "start": "The Natives remained in London following the England international . They defeated London Welsh on 18 February , before losing first"
97
+ },
98
+ {
99
+ "i": 24,
100
+ "start": "In February 1997 , a United States immigration court judge granted asylum to a German Scientologist who claimed she"
101
+ },
102
+ {
103
+ "i": 25,
104
+ "start": "Some of the most important trade routes began at Canton and Macau in Qing Dynasty China . Early in each year a large convoy"
105
+ },
106
+ {
107
+ "i": 26,
108
+ "start": "The fourth installment in the Slammiversary name was announced in January 2008 to take place on June"
109
+ },
110
+ {
111
+ "i": 27,
112
+ "start": "Capel Lligwy has national recognition and statutory protection from alteration as it has been designated as a Grade II listed building"
113
+ },
114
+ {
115
+ "i": 28,
116
+ "start": "The Trout Creek Mountains are semi @-@ arid because they are in the eastern rain shadow of mountain ranges to"
117
+ },
118
+ {
119
+ "i": 29,
120
+ "start": "Asahi , meaning \" rising sun \" , a poetic name for Japan from a stanza of waka poetry , was laid"
121
+ },
122
+ {
123
+ "i": 30,
124
+ "start": "Beckham 's term as governor ended December 10 , 1907 . In January 19"
125
+ },
126
+ {
127
+ "i": 31,
128
+ "start": "Donald Sutherland portrays Jack Pellitt , Bobby 's father and Kurt 's boss . On July 27 ,"
129
+ },
130
+ {
131
+ "i": 32,
132
+ "start": "As British offshoots , the Australian colonies participated in Britain 's small wars of the 19th century"
133
+ },
134
+ {
135
+ "i": 33,
136
+ "start": "Retail is another major employer in the county , with 5 @,@ 280 people employed in April "
137
+ },
138
+ {
139
+ "i": 34,
140
+ "start": "The tour returned to Pocono a month later , and Richmond battled for another victory in a fog @-@ shortened"
141
+ },
142
+ {
143
+ "i": 35,
144
+ "start": "\" ... Thirteen Years Later \" received mixed reviews from critics . The A.V. Club 's Todd VanDerWer"
145
+ },
146
+ {
147
+ "i": 36,
148
+ "start": "Bonnie Stiernberg , for Paste , was more positive , writing that she preferred the use of comedy , following the"
149
+ },
150
+ {
151
+ "i": 37,
152
+ "start": "In common with many small Scottish islands , Papa Stour 's population peaked in the 19th century and has"
153
+ },
154
+ {
155
+ "i": 38,
156
+ "start": "The unexpected deployment to East Timor in 1999 led to significant changes in Australian defence policy and to an enhancement"
157
+ },
158
+ {
159
+ "i": 39,
160
+ "start": "Government publications on the dangers of sects increased between 1996 and 1998 , and"
161
+ },
162
+ {
163
+ "i": 40,
164
+ "start": "Richmond started 1982 without a ride before getting a one @-@ race deal to drive for Billie"
165
+ },
166
+ {
167
+ "i": 41,
168
+ "start": "The music for Sonic the Hedgehog was composed and produced by Masato Nakamura , bassist and lead songwriter of the"
169
+ },
170
+ {
171
+ "i": 42,
172
+ "start": "Apple Records issued the single on 17 March 1972 in Britain , as Apple R 59"
173
+ },
174
+ {
175
+ "i": 43,
176
+ "start": "Meridian is home to two post @-@ secondary educational institutions . Meridian Community College , founded in 19"
177
+ },
178
+ {
179
+ "i": 44,
180
+ "start": "Ayola began her career in the theatre , performing in a number of Shakespearean plays including Twelfth Night , A M"
181
+ },
182
+ {
183
+ "i": 45,
184
+ "start": "Cooke 's opportunistic diversion had determined that the Spanish forces in the Philippines posed no immediate threat , although the mission"
185
+ },
186
+ {
187
+ "i": 46,
188
+ "start": "To further justify its stance , the German government has also pointed to the long history of U.S. court cases involving Scient"
189
+ },
190
+ {
191
+ "i": 47,
192
+ "start": "Headed by police chief Lee Shelbourn since 2009 , the Meridian Police Department consists of "
193
+ },
194
+ {
195
+ "i": 48,
196
+ "start": "A port , Sonic the Hedgehog Genesis , was released for the Game Boy Advance ( GBA ) on November"
197
+ },
198
+ {
199
+ "i": 49,
200
+ "start": "As is often the case with huge ancient ruins , knowledge of the site was never completely lost in the region . It seems"
201
+ },
202
+ {
203
+ "i": 50,
204
+ "start": "According to Eliakim Sibanda , a professor and human rights speaker who wrote a history of ZAPU , N"
205
+ },
206
+ {
207
+ "i": 51,
208
+ "start": "The Raid on Manila of January 1798 was a Royal Navy false flag military operation during the French Revolutionary"
209
+ },
210
+ {
211
+ "i": 52,
212
+ "start": "Five infantry divisions of the AIF saw action in France and Belgium , leaving Egypt in March 1916 ."
213
+ },
214
+ {
215
+ "i": 53,
216
+ "start": "Die Another Day had its world premiere on 18 November 2002 at the Royal Albert Hall in London"
217
+ },
218
+ {
219
+ "i": 54,
220
+ "start": "To get revenge on Homer , Bart goes around Springfield spray @-@ painting graffiti of Homer 's face and the"
221
+ },
222
+ {
223
+ "i": 55,
224
+ "start": "The Natives next defeated both Northamptonshire and Kent , before defeats by Moseley and Burton @-"
225
+ },
226
+ {
227
+ "i": 56,
228
+ "start": "Headlam remained in Darwin as Controller of Operations at Headquarters North @-@ Western Area Command until 12 May "
229
+ },
230
+ {
231
+ "i": 57,
232
+ "start": "The advance entered Palestine and an initial , unsuccessful attempt was made to capture Gaza on 26 March 191"
233
+ },
234
+ {
235
+ "i": 58,
236
+ "start": "At the same time , a Japanese company from Watanabe 's battalion infiltrated through a gap between the east side of the"
237
+ },
238
+ {
239
+ "i": 59,
240
+ "start": "Route 29 begins at a modified cloverleaf interchange with Interstate 195 and Interstate 295"
241
+ },
242
+ {
243
+ "i": 60,
244
+ "start": "Under the overall command of Australian Major General Peter Cosgrove , INTERFET began arriving on 12 September 1"
245
+ },
246
+ {
247
+ "i": 61,
248
+ "start": "Australia 's involvement in the Vietnam War was driven largely by the rise of communism in Southeast Asia after the Second World"
249
+ },
250
+ {
251
+ "i": 62,
252
+ "start": "The fact that Scientologists have to pay large fees for auditing and other Scientology services has brought controversy to Scientology"
253
+ },
254
+ {
255
+ "i": 63,
256
+ "start": "Moving forward under a creeping barrage as they moved beyond Tokinotu , the 24th Infantry Battalion was the"
257
+ }
258
+ ]
fit_records/jpp_k2_n64_e8_s0/readouts.txt ADDED
@@ -0,0 +1,80 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ router at in.08: 56000 positions from 500 paragraphs; cluster shares [0.076, 0.241, 0.018, 0.268, 0.195, 0.072, 0.034, 0.095]
2
+ positions per expert: [509, 1688, 131, 1933, 1353, 496, 235, 823]
3
+ items per task, all / kept (multihop and multilingual kept only where the model's top next token starts the answer) / dev / test: multihop 84/20/10/10; multilingual 107/45/22/23; typo 96/96/48/48; association 102/102/51/51; poetry 98/98/49/49
4
+ weights fitted on 246 dev (item, intermediate) pairs in 167s
5
+
6
+ TEST HALF: 181 items, 250 (item, intermediate) pairs. Seven scored nodes: ['in.14', 'rec.1.28', 'rec.1.42', 'rec.2.16', 'rec.2.30', 'rec.2.44', 'out.58']
7
+ recall = macro over the five tasks; an intermediate counts if it is in the top k at ANY of the listed nodes; punctuation-like tokens dropped unless marked
8
+ lens @10 seven @1 seven | multihop multiling typo associati poetry | input block loop 1 loop 2 output block
9
+ logit lens 0.189 0.072 | 0.200 0.413 0.333 0.000 0.000 | 0.037 0.182 0.182 0.245
10
+ J-lens (E001, 50 prompts) 0.268 0.122 | 0.200 0.576 0.542 0.000 0.020 | 0.033 0.187 0.296 0.289
11
+ R-lens (pooled LRP) 0.321 0.168 | 0.300 0.620 0.667 0.000 0.020 | 0.133 0.132 0.361 0.282
12
+ best single expert 0.378 0.174 | 0.400 0.641 0.708 0.078 0.061 | 0.148 0.248 0.413 0.293
13
+ J++ shuffled labels 0.225 0.084 | 0.200 0.217 0.625 0.000 0.082 | 0.152 0.161 0.299 0.149
14
+ J++ 0.423 0.233 | 0.400 0.641 0.875 0.118 0.082 | 0.162 0.258 0.439 0.474
15
+ J++ (full vocabulary) 0.388 0.174 | 0.400 0.587 0.833 0.039 0.082 | 0.139 0.191 0.393 0.450
16
+ J-lens (full vocabulary) 0.259 0.103 | 0.200 0.554 0.521 0.000 0.020 | 0.021 0.089 0.245 0.280
17
+
18
+ recall@10 at each of the seven nodes on its own
19
+ lens in.14 rec.1.28 rec.1.42 rec.2.16 rec.2.30 rec.2.44 out.58
20
+ logit lens 0.013 0.000 0.046 0.029 0.008 0.034 0.137
21
+ J-lens (E001, 50 prompts) 0.008 0.004 0.019 0.029 0.071 0.198 0.167
22
+ R-lens (pooled LRP) 0.071 0.021 0.025 0.096 0.105 0.221 0.206
23
+ best single expert 0.084 0.062 0.061 0.108 0.146 0.242 0.264
24
+ J++ shuffled labels 0.058 0.073 0.031 0.075 0.042 0.123 0.008
25
+ J++ 0.134 0.115 0.135 0.144 0.159 0.244 0.303
26
+
27
+ R-lens split-half cosine and J++ weights at the seven nodes (weights in router-cluster order)
28
+ in.14 split-half 0.990 weights [0.86, -0.2, 0.04, 0.29, 0.23, 0.05, -0.12, -0.24] best expert 3
29
+ rec.1.28 split-half 0.989 weights [0.4, -0.41, 0.11, 0.48, 0.38, 0.05, -0.18, -0.49] best expert 3
30
+ rec.1.42 split-half 0.989 weights [0.34, -0.28, 0.15, 0.75, 0.22, -0.01, -0.22, -0.35] best expert 0
31
+ rec.2.16 split-half 0.992 weights [0.78, -0.12, 0.07, 0.37, 0.05, 0.06, -0.17, -0.44] best expert 0
32
+ rec.2.30 split-half 0.994 weights [0.44, -0.49, 0.06, 0.27, 0.67, 0.04, -0.16, -0.1] best expert 0
33
+ rec.2.44 split-half 0.996 weights [0.51, -0.41, 0.09, 0.47, 0.53, 0.08, -0.19, -0.16] best expert 0
34
+ out.58 split-half 0.999 weights [0.39, -0.8, -0.02, -0.11, 0.37, 0.14, -0.02, 0.18] best expert 0
35
+ R-lens split-half cosine, median by block: input block 0.99; loop 1 0.99; loop 2 0.99; output block 1.00
36
+
37
+ [multihop] 'Fact: The third month of the year is named after the Roman god of' read at token ' of'; intermediates ['March']
38
+ in.14 | J-lens: [' holy', ' biblical', ' religion', ' trump', ' assass', ' priest'] | R-lens: [' his', ' the', ' death', ' and', ' human', ' war'] | J++: [' his', ' war', ' the', ' worship', ' death', ' music']
39
+ rec.1.28 | J-lens: [' 어�', ' chamb', ' canton', 'aniu', 'лова', ' Himal'] | R-lens: [' the', ' and', ' his', ' in', ' a', ' to'] | J++: [' his', ' the', ' ancient', ' worship', ' he', ' religious']
40
+ rec.1.42 | J-lens: ['Lib', 'astr', ' ancient', 'Stand', 'lust', 'raged'] | R-lens: [' ancient', ' the', ' and', ' his', ' centuries', ' famous'] | J++: [' ancient', ' worship', ' Ancient', ' symbol', ' Roman', ' pagan']
41
+ rec.2.16 | J-lens: [' holy', ' terror', ' sacred', ' religion', ' gods', ' death'] | R-lens: [' his', ' ancient', ' the', ' war', ' human', ' death'] | J++: [' war', ' ancient', ' military', ' his', ' religious', ' death']
42
+ rec.2.30 | J-lens: [' ancient', ' terror', ' pagan', ' civilization', ' sacred', ' religion'] | R-lens: [' ancient', ' his', ' war', ' military', ' famous', ' the'] | J++: [' his', ' ancient', ' war', ' famous', ' military', ' the']
43
+ rec.2.44 | J-lens: [' ancient', ' gods', ' pagan', ' mythology', ' antiquity', ' myth'] | R-lens: [' ancient', ' symbol', ' Ancient', ' Roman', ' worship', ' sacred'] | J++: [' ancient', ' Roman', ' Ancient', ' symbol', ' pagan', ' gods']
44
+ out.58 | J-lens: [' agriculture', ' gods', ' fertility', ' thunder', ' commerce', ' Mars'] | R-lens: [' agriculture', ' thunder', ' fertility', ' gods', ' protection', ' luck'] | J++: [' Roman', ' Latin', ' Greek', ' Romans', ' Mars', ' Ga']
45
+
46
+ [multilingual] 'Das Gegenteil von "erste" ist "' read at token ' "'; intermediates ['German', 'Order', 'First', ' Last']
47
+ in.14 | J-lens: ['s', 'a', 'o', 'e', '"I', '"How'] | R-lens: ['a', 's', 'o', ' a', '"I', ' very'] | J++: [' very', ' word', ' thing', ' words', ' something', 'a']
48
+ rec.1.28 | J-lens: [' 어�', ' chamb', ' Stur', ' Fries', ' canton', ' Commiss'] | R-lens: [' or', 'a', ' to', ' in', ' a', 's'] | J++: [' word', ' or', ' name', ' phrase', ' words', ' called']
49
+ rec.1.42 | J-lens: ['s', '[A', 'after', 'as', '.Act', '[a'] | R-lens: [' or', 'a', 's', 'or', ' only', ' not'] | J++: [' or', ' first', ' name', ' only', ' not', ' in']
50
+ rec.2.16 | J-lens: ['s', 'a', 'o', 'e', '"I', '"The'] | R-lens: ['a', ' not', 's', ' only', ' or', ' completely'] | J++: [' word', ' name', ' words', ' one', ' very', ' French']
51
+ rec.2.30 | J-lens: ['or', 'yes', 'something', 'a', 's', 'same'] | R-lens: [' or', 'a', 'or', ' something', 'o', 'e'] | J++: [' something', ' name', ' word', ' words', ' phrase', ' English']
52
+ rec.2.44 | J-lens: ['second', 'first', 'others', ' afterward', 'other', 'または'] | R-lens: [' second', ' first', ' later', ' or', ' not', ' subsequent'] | J++: [' second', ' first', ' later', ' next', ' last', ' subsequent']
53
+ out.58 | J-lens: [' second', 'second', 'last', ' last', ' later', ' latter'] | R-lens: [' second', ' later', ' last', 'second', ' subsequent', ' latter'] | J++: [' Later', ' second', ' Second', ' Last', ' later', ' Ne']
54
+
55
+ [typo] 'He finished the race in secnod' read at token 'od'; intermediates ['Second']
56
+ in.14 | J-lens: [' feb', ' confi', ' fir', ' fai', '’S', ' countr'] | R-lens: [' merely', ' perhaps', 'ly', 'less', ' a', ' of'] | J++: ['’s', '’d', ' Ross', ' Ono', ' signalling', ' Sonny']
57
+ rec.1.28 | J-lens: [' 어�', ' chamb', ' commiss', ' quay', 'rib', ' Commiss'] | R-lens: [' of', ' the', ' only', ' a', ' then', ' to'] | J++: [' Ross', ' station', 'station', ' Sonny', ' Station', ' Ond']
58
+ rec.1.42 | J-lens: [' ina', ' anot', ' ond', ' tho', ' ona', ' ei'] | R-lens: [' time', ' a', '1', ' s', ' of', ' only'] | J++: ['@gmail', ' tweet', ' at', ' Sunday', ' station', ' the']
59
+ rec.2.16 | J-lens: ['ly', ';s', ' tho', 'less', 'nt', ' cong'] | R-lens: [' of', ' only', ' s', ' a', ' time', 'ly'] | J++: ['’s', ' the', ' a', ' time', ' only', ' an']
60
+ rec.2.30 | J-lens: [' tand', ' tho', ' ond', ';s', '[s', ' ave'] | R-lens: [' s', ' time', 's', ' th', 'n', ' t'] | J++: [' the', ' seconds', ' time', ' a', ' one', ' of']
61
+ rec.2.44 | J-lens: ['[s', ' seconds', '&q', ' thir', ' tand', ' iii'] | R-lens: [' time', ' s', ' t', ' seconds', ' of', ' times'] | J++: [' time', ' the', ' seconds', ' a', '’s', ' s']
62
+ out.58 | J-lens: ['ly', '[s', ' forty', ' thirty', 'less', ' sixty'] | R-lens: ['ly', ' time', ' s', ' t', ' twenty', ' thirty'] | J++: [' Second', ' Sec', ' SO', ' sec', ' CD', ' Se']
63
+
64
+ [association] 's, tilted her head, then stepped forward and scraped the whole left corner back to canvas.' read at token '.'; intermediates [' art']
65
+ in.14 | J-lens: ['—but', '—a', '—the', '—not', ' practically', '—as'] | R-lens: [' a', ' the', ' one', ' very', ' then', ' by'] | J++: ['’s', ' then', ' very', ' a', ' the', ' one']
66
+ rec.1.28 | J-lens: [' suddenly', ' enorm', '—not', ' frightened', ' practically', ' nearly'] | R-lens: [' a', ' the', ' at', ' one', ' just', ' on'] | J++: [' the', ' at', ' a', ' then', ' left', ' later']
67
+ rec.1.42 | J-lens: ['gare', 'raid', ' tand', 'bogen', 'graf', 'aston'] | R-lens: [' a', ' the', ' one', ' at', ' just', ' only'] | J++: [' the', ' at', ' back', ' left', ' a', ' behind']
68
+ rec.2.16 | J-lens: [' â', '—the', '—not', '—and', '—a', '—but'] | R-lens: [' a', ' the', ' one', ' just', ' back', ' then'] | J++: [' the', ' a', '’s', ' back', ' left', ' then']
69
+ rec.2.30 | J-lens: [' stripping', ' photographed', ' scrub', ' tearing', ' walls', ' grotes'] | R-lens: [' back', ' then', ' a', ' the', ' again', ' one'] | J++: [' back', ' then', ' a', ' the', ' w', ' hand']
70
+ rec.2.44 | J-lens: ['—No', ' suddenly', ' afterward', ' next', ' finally', ' Next'] | R-lens: [' then', ' the', ' his', ' again', ' next', ' after'] | J++: [' his', ' the', ' then', ' next', ' a', ' hand']
71
+ out.58 | J-lens: [' she', ' She', ' her', ' hers', '她', ' herself'] | R-lens: [' she', ' her', ' She', ' herself', ' hers', '她'] | J++: [' She', ' she', ' her', 'She', ' Art', ' The']
72
+
73
+ [poetry] 'uplet:\nThe traveler paused before the ancient gate,\nAnd stepped across to meet his waiting' read at token ',\n'; intermediates [' fate']
74
+ in.14 | J-lens: ['—a', '—but', '—to', '—that', '—and', '—in'] | R-lens: ['—and', '—a', ' a', ' perhaps', ' to', '—or'] | J++: [' a', ' something', ' the', ' perhaps', ' an', ' to']
75
+ rec.1.28 | J-lens: [' tand', 'inde', ' ina', 'don', 'ionat', ' tob'] | R-lens: [' delight', ' beautifully', ' weary', ' joy', ' perhaps', ' loving'] | J++: [' poem', ' poet', ' poetry', ' poetic', ' poems', ' morning']
76
+ rec.1.42 | J-lens: [' tand', 'gare', 'laden', 'raid', 'lott', 'lda'] | R-lens: [' a', ' to', ' the', ' just', ' s', ' w'] | J++: [' story', ' poem', ' night', ' scene', ' sight', ' the']
77
+ rec.2.16 | J-lens: ['—and', '—to', '—a', '—but', '—as', '—that'] | R-lens: [' to', ' a', ' and', '—and', ' an', ' perhaps'] | J++: [' a', ' the', ' an', ' story', ' to', ' ancient']
78
+ rec.2.30 | J-lens: [' delight', ' poetic', '—a', ' weary', '—to', '—but'] | R-lens: [' poetic', ' delight', ' poet', ' gently', ' poem', ' a'] | J++: [' poetic', ' poem', ' a', ' poet', ' poetry', ' Old']
79
+ rec.2.44 | J-lens: [' dreaming', ' hoping', ' whispered', ' dreams', ' wondering', ' dream'] | R-lens: [' where', ' with', ' a', ' fear', ' and', ' hoping'] | J++: [' a', ' fear', ' sight', ' the', ' dream', ' his']
80
+ out.58 | J-lens: [' Where', ' wondering', ' And', ' where', ' Then', ' hoping'] | R-lens: [' where', ' Where', ' And', ' and', ' wondering', 'Where'] | J++: [' Sur', 'Where', ' Mi', ' Where', ' Cor', 'Co']
fit_records/k1_n50_skip16_s0/metrics.json ADDED
@@ -0,0 +1,2646 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "n_prompts": 50,
3
+ "n_nodes": 73,
4
+ "seconds_per_prompt": 24.88944085597992,
5
+ "split_half_cosine": {
6
+ "emb": 0.6111,
7
+ "in.00": 0.6306,
8
+ "in.01": 0.7214,
9
+ "in.02": 0.7285,
10
+ "in.03": 0.7237,
11
+ "in.04": 0.7589,
12
+ "in.05": 0.7714,
13
+ "in.06": 0.7979,
14
+ "in.07": 0.8061,
15
+ "in.08": 0.8429,
16
+ "in.09": 0.8581,
17
+ "in.10": 0.8566,
18
+ "in.11": 0.8611,
19
+ "in.12": 0.8592,
20
+ "in.13": 0.849,
21
+ "in.14": 0.8529,
22
+ "x": 0.7797,
23
+ "u.1": 0.7797,
24
+ "rec.1.16": 0.8915,
25
+ "rec.1.17": 0.9145,
26
+ "rec.1.18": 0.9229,
27
+ "rec.1.19": 0.9304,
28
+ "rec.1.20": 0.9322,
29
+ "rec.1.21": 0.9367,
30
+ "rec.1.22": 0.9599,
31
+ "rec.1.23": 0.959,
32
+ "rec.1.24": 0.9594,
33
+ "rec.1.25": 0.9583,
34
+ "rec.1.26": 0.9584,
35
+ "rec.1.27": 0.9594,
36
+ "rec.1.28": 0.9591,
37
+ "rec.1.29": 0.9609,
38
+ "rec.1.30": 0.9627,
39
+ "rec.1.31": 0.9628,
40
+ "rec.1.32": 0.9634,
41
+ "rec.1.33": 0.9638,
42
+ "rec.1.34": 0.9654,
43
+ "rec.1.35": 0.9655,
44
+ "rec.1.36": 0.97,
45
+ "rec.1.37": 0.9711,
46
+ "rec.1.38": 0.9724,
47
+ "rec.1.39": 0.9741,
48
+ "rec.1.40": 0.9756,
49
+ "rec.1.41": 0.9777,
50
+ "rec.1.42": 0.9781,
51
+ "rec.1.43": 0.981,
52
+ "rec.1.44": 0.9808,
53
+ "rec.1.45": 0.983,
54
+ "rec.1.46": 0.9836,
55
+ "rec.1.47": 0.9851,
56
+ "rec.1.48": 0.9851,
57
+ "rec.1.49": 0.9867,
58
+ "rec.1.50": 0.9869,
59
+ "rec.1.51": 0.9882,
60
+ "rec.1.52": 0.9895,
61
+ "rec.1.53": 0.9906,
62
+ "rec.1.54": 0.9911,
63
+ "s.1": 0.9923,
64
+ "out.56": 0.9924,
65
+ "out.57": 0.9934,
66
+ "out.58": 0.9938,
67
+ "out.59": 0.9949,
68
+ "out.60": 0.9953,
69
+ "out.61": 0.996,
70
+ "out.62": 0.9966,
71
+ "out.63": 0.9975,
72
+ "out.64": 0.9975,
73
+ "out.65": 0.9984,
74
+ "out.66": 0.9984,
75
+ "out.67": 0.9991,
76
+ "out.68": 0.9992,
77
+ "out.69": 0.9996,
78
+ "out.70": 0.9997
79
+ },
80
+ "cosine_skip0_vs_skip16": {
81
+ "emb": 0.8987,
82
+ "in.00": 0.9065,
83
+ "in.01": 0.928,
84
+ "in.02": 0.9361,
85
+ "in.03": 0.9349,
86
+ "in.04": 0.9507,
87
+ "in.05": 0.9523,
88
+ "in.06": 0.9585,
89
+ "in.07": 0.9594,
90
+ "in.08": 0.9663,
91
+ "in.09": 0.9676,
92
+ "in.10": 0.968,
93
+ "in.11": 0.9677,
94
+ "in.12": 0.969,
95
+ "in.13": 0.9577,
96
+ "in.14": 0.9689,
97
+ "x": 0.9749,
98
+ "u.1": 0.9749,
99
+ "rec.1.16": 0.9769,
100
+ "rec.1.17": 0.9781,
101
+ "rec.1.18": 0.9796,
102
+ "rec.1.19": 0.976,
103
+ "rec.1.20": 0.98,
104
+ "rec.1.21": 0.9797,
105
+ "rec.1.22": 0.9816,
106
+ "rec.1.23": 0.9792,
107
+ "rec.1.24": 0.9823,
108
+ "rec.1.25": 0.984,
109
+ "rec.1.26": 0.9823,
110
+ "rec.1.27": 0.9833,
111
+ "rec.1.28": 0.984,
112
+ "rec.1.29": 0.9834,
113
+ "rec.1.30": 0.985,
114
+ "rec.1.31": 0.9883,
115
+ "rec.1.32": 0.9858,
116
+ "rec.1.33": 0.989,
117
+ "rec.1.34": 0.9888,
118
+ "rec.1.35": 0.9894,
119
+ "rec.1.36": 0.9896,
120
+ "rec.1.37": 0.9883,
121
+ "rec.1.38": 0.9902,
122
+ "rec.1.39": 0.9896,
123
+ "rec.1.40": 0.9914,
124
+ "rec.1.41": 0.9919,
125
+ "rec.1.42": 0.9925,
126
+ "rec.1.43": 0.992,
127
+ "rec.1.44": 0.9925,
128
+ "rec.1.45": 0.9928,
129
+ "rec.1.46": 0.9928,
130
+ "rec.1.47": 0.9939,
131
+ "rec.1.48": 0.9952,
132
+ "rec.1.49": 0.9956,
133
+ "rec.1.50": 0.996,
134
+ "rec.1.51": 0.9961,
135
+ "rec.1.52": 0.996,
136
+ "rec.1.53": 0.9904,
137
+ "rec.1.54": 0.9972,
138
+ "s.1": 0.9974,
139
+ "out.56": 0.9979,
140
+ "out.57": 0.9978,
141
+ "out.58": 0.998,
142
+ "out.59": 0.998,
143
+ "out.60": 0.9986,
144
+ "out.61": 0.9965,
145
+ "out.62": 0.9996,
146
+ "out.63": 0.9996,
147
+ "out.64": 0.9996,
148
+ "out.65": 0.999,
149
+ "out.66": 0.9997,
150
+ "out.67": 0.9994,
151
+ "out.68": 0.9999,
152
+ "out.69": 0.9997,
153
+ "out.70": 1.0
154
+ },
155
+ "cosine_skip4_vs_skip16": {
156
+ "emb": 0.9335,
157
+ "in.00": 0.9382,
158
+ "in.01": 0.9576,
159
+ "in.02": 0.9568,
160
+ "in.03": 0.958,
161
+ "in.04": 0.9661,
162
+ "in.05": 0.9689,
163
+ "in.06": 0.9734,
164
+ "in.07": 0.9758,
165
+ "in.08": 0.9796,
166
+ "in.09": 0.9827,
167
+ "in.10": 0.9819,
168
+ "in.11": 0.9827,
169
+ "in.12": 0.9823,
170
+ "in.13": 0.9818,
171
+ "in.14": 0.9828,
172
+ "x": 0.9833,
173
+ "u.1": 0.9833,
174
+ "rec.1.16": 0.9887,
175
+ "rec.1.17": 0.9908,
176
+ "rec.1.18": 0.9915,
177
+ "rec.1.19": 0.9923,
178
+ "rec.1.20": 0.9927,
179
+ "rec.1.21": 0.993,
180
+ "rec.1.22": 0.9945,
181
+ "rec.1.23": 0.9948,
182
+ "rec.1.24": 0.9946,
183
+ "rec.1.25": 0.995,
184
+ "rec.1.26": 0.9947,
185
+ "rec.1.27": 0.9949,
186
+ "rec.1.28": 0.9949,
187
+ "rec.1.29": 0.9953,
188
+ "rec.1.30": 0.9956,
189
+ "rec.1.31": 0.9958,
190
+ "rec.1.32": 0.9958,
191
+ "rec.1.33": 0.996,
192
+ "rec.1.34": 0.9961,
193
+ "rec.1.35": 0.9961,
194
+ "rec.1.36": 0.9964,
195
+ "rec.1.37": 0.9965,
196
+ "rec.1.38": 0.9968,
197
+ "rec.1.39": 0.997,
198
+ "rec.1.40": 0.9969,
199
+ "rec.1.41": 0.9972,
200
+ "rec.1.42": 0.9972,
201
+ "rec.1.43": 0.9974,
202
+ "rec.1.44": 0.9975,
203
+ "rec.1.45": 0.9976,
204
+ "rec.1.46": 0.9975,
205
+ "rec.1.47": 0.9977,
206
+ "rec.1.48": 0.9981,
207
+ "rec.1.49": 0.9983,
208
+ "rec.1.50": 0.9984,
209
+ "rec.1.51": 0.9985,
210
+ "rec.1.52": 0.9986,
211
+ "rec.1.53": 0.9984,
212
+ "rec.1.54": 0.9991,
213
+ "s.1": 0.9992,
214
+ "out.56": 0.9993,
215
+ "out.57": 0.9993,
216
+ "out.58": 0.9994,
217
+ "out.59": 0.9994,
218
+ "out.60": 0.9996,
219
+ "out.61": 0.9996,
220
+ "out.62": 0.9998,
221
+ "out.63": 0.9998,
222
+ "out.64": 0.9999,
223
+ "out.65": 0.9998,
224
+ "out.66": 0.9999,
225
+ "out.67": 0.9999,
226
+ "out.68": 1.0,
227
+ "out.69": 1.0,
228
+ "out.70": 1.0
229
+ },
230
+ "frobenius_norm_skip16": {
231
+ "emb": 1.8699,
232
+ "in.00": 2.445,
233
+ "in.01": 3.4117,
234
+ "in.02": 3.2139,
235
+ "in.03": 3.5241,
236
+ "in.04": 3.3825,
237
+ "in.05": 3.4804,
238
+ "in.06": 3.1565,
239
+ "in.07": 3.3926,
240
+ "in.08": 3.4897,
241
+ "in.09": 4.6102,
242
+ "in.10": 3.8056,
243
+ "in.11": 4.6314,
244
+ "in.12": 4.055,
245
+ "in.13": 4.5315,
246
+ "in.14": 3.923,
247
+ "x": 8.2757,
248
+ "u.1": 8.2762,
249
+ "rec.1.16": 4.5408,
250
+ "rec.1.17": 6.1005,
251
+ "rec.1.18": 5.2451,
252
+ "rec.1.19": 6.1525,
253
+ "rec.1.20": 5.7525,
254
+ "rec.1.21": 6.3778,
255
+ "rec.1.22": 10.3105,
256
+ "rec.1.23": 15.7066,
257
+ "rec.1.24": 11.2153,
258
+ "rec.1.25": 15.166,
259
+ "rec.1.26": 12.2378,
260
+ "rec.1.27": 14.8734,
261
+ "rec.1.28": 12.0632,
262
+ "rec.1.29": 13.3218,
263
+ "rec.1.30": 10.7848,
264
+ "rec.1.31": 16.409,
265
+ "rec.1.32": 9.2905,
266
+ "rec.1.33": 15.7074,
267
+ "rec.1.34": 9.4805,
268
+ "rec.1.35": 13.8823,
269
+ "rec.1.36": 10.1258,
270
+ "rec.1.37": 12.5738,
271
+ "rec.1.38": 11.2839,
272
+ "rec.1.39": 14.4589,
273
+ "rec.1.40": 11.7483,
274
+ "rec.1.41": 14.5842,
275
+ "rec.1.42": 11.7739,
276
+ "rec.1.43": 16.8372,
277
+ "rec.1.44": 12.8936,
278
+ "rec.1.45": 15.0309,
279
+ "rec.1.46": 15.2036,
280
+ "rec.1.47": 16.8973,
281
+ "rec.1.48": 17.1669,
282
+ "rec.1.49": 24.9805,
283
+ "rec.1.50": 19.2652,
284
+ "rec.1.51": 27.6534,
285
+ "rec.1.52": 21.6746,
286
+ "rec.1.53": 29.3857,
287
+ "rec.1.54": 28.927,
288
+ "s.1": 173.9085,
289
+ "out.56": 30.3121,
290
+ "out.57": 37.7185,
291
+ "out.58": 36.1919,
292
+ "out.59": 44.5618,
293
+ "out.60": 40.2901,
294
+ "out.61": 57.1865,
295
+ "out.62": 42.9757,
296
+ "out.63": 49.8286,
297
+ "out.64": 42.4901,
298
+ "out.65": 73.5946,
299
+ "out.66": 45.7986,
300
+ "out.67": 77.3094,
301
+ "out.68": 46.5387,
302
+ "out.69": 71.286,
303
+ "out.70": 53.9406
304
+ },
305
+ "position_norm_over_late_median": {
306
+ "emb": [
307
+ 6.137,
308
+ 6.327,
309
+ 5.197,
310
+ 4.555,
311
+ 8.183,
312
+ 4.69,
313
+ 3.833,
314
+ 3.522,
315
+ 4.844,
316
+ 4.395,
317
+ 5.306,
318
+ 2.576,
319
+ 1.916,
320
+ 2.839,
321
+ 3.54,
322
+ 2.823,
323
+ 3.588,
324
+ 2.32,
325
+ 1.935,
326
+ 2.013
327
+ ],
328
+ "in.00": [
329
+ 4.81,
330
+ 5.775,
331
+ 4.744,
332
+ 4.235,
333
+ 7.315,
334
+ 4.132,
335
+ 3.748,
336
+ 3.391,
337
+ 4.458,
338
+ 3.863,
339
+ 6.092,
340
+ 2.583,
341
+ 1.81,
342
+ 2.596,
343
+ 3.234,
344
+ 2.867,
345
+ 3.312,
346
+ 2.655,
347
+ 2.04,
348
+ 1.964
349
+ ],
350
+ "in.01": [
351
+ 5.946,
352
+ 5.48,
353
+ 4.889,
354
+ 4.238,
355
+ 6.569,
356
+ 3.96,
357
+ 3.782,
358
+ 3.548,
359
+ 4.354,
360
+ 4.125,
361
+ 5.498,
362
+ 2.56,
363
+ 2.066,
364
+ 2.905,
365
+ 3.054,
366
+ 3.026,
367
+ 3.232,
368
+ 3.256,
369
+ 2.324,
370
+ 2.268
371
+ ],
372
+ "in.02": [
373
+ 3.452,
374
+ 4.996,
375
+ 4.333,
376
+ 3.956,
377
+ 6.598,
378
+ 3.929,
379
+ 3.71,
380
+ 3.461,
381
+ 4.149,
382
+ 3.735,
383
+ 6.439,
384
+ 2.554,
385
+ 2.023,
386
+ 2.76,
387
+ 2.952,
388
+ 2.957,
389
+ 2.912,
390
+ 3.447,
391
+ 2.145,
392
+ 1.991
393
+ ],
394
+ "in.03": [
395
+ 3.611,
396
+ 4.708,
397
+ 4.062,
398
+ 3.729,
399
+ 6.398,
400
+ 3.665,
401
+ 3.817,
402
+ 3.237,
403
+ 3.994,
404
+ 3.825,
405
+ 6.384,
406
+ 2.605,
407
+ 2.033,
408
+ 2.753,
409
+ 2.774,
410
+ 2.877,
411
+ 2.803,
412
+ 3.152,
413
+ 2.12,
414
+ 1.969
415
+ ],
416
+ "in.04": [
417
+ 2.745,
418
+ 4.146,
419
+ 3.661,
420
+ 3.33,
421
+ 6.203,
422
+ 3.497,
423
+ 3.51,
424
+ 3.328,
425
+ 3.564,
426
+ 3.549,
427
+ 5.807,
428
+ 2.378,
429
+ 2.089,
430
+ 2.853,
431
+ 2.725,
432
+ 2.99,
433
+ 2.892,
434
+ 3.398,
435
+ 2.184,
436
+ 2.071
437
+ ],
438
+ "in.05": [
439
+ 3.166,
440
+ 3.884,
441
+ 3.581,
442
+ 3.283,
443
+ 5.797,
444
+ 3.356,
445
+ 3.294,
446
+ 3.249,
447
+ 3.525,
448
+ 3.495,
449
+ 6.128,
450
+ 2.253,
451
+ 2.074,
452
+ 2.853,
453
+ 2.778,
454
+ 2.959,
455
+ 2.939,
456
+ 3.372,
457
+ 2.154,
458
+ 2.027
459
+ ],
460
+ "in.06": [
461
+ 3.412,
462
+ 3.638,
463
+ 3.267,
464
+ 2.888,
465
+ 5.875,
466
+ 3.129,
467
+ 3.038,
468
+ 3.056,
469
+ 3.47,
470
+ 3.267,
471
+ 5.72,
472
+ 2.158,
473
+ 1.971,
474
+ 2.71,
475
+ 2.723,
476
+ 2.704,
477
+ 2.813,
478
+ 2.493,
479
+ 2.009,
480
+ 1.893
481
+ ],
482
+ "in.07": [
483
+ 3.795,
484
+ 3.549,
485
+ 3.395,
486
+ 2.986,
487
+ 5.244,
488
+ 3.05,
489
+ 3.011,
490
+ 3.081,
491
+ 3.294,
492
+ 3.23,
493
+ 5.393,
494
+ 2.096,
495
+ 2.008,
496
+ 2.639,
497
+ 2.602,
498
+ 2.728,
499
+ 2.816,
500
+ 2.392,
501
+ 2.033,
502
+ 1.873
503
+ ],
504
+ "in.08": [
505
+ 3.301,
506
+ 3.586,
507
+ 3.439,
508
+ 2.949,
509
+ 5.361,
510
+ 3.136,
511
+ 3.103,
512
+ 3.132,
513
+ 3.151,
514
+ 3.209,
515
+ 5.549,
516
+ 2.139,
517
+ 2.037,
518
+ 2.707,
519
+ 2.686,
520
+ 2.768,
521
+ 2.898,
522
+ 2.459,
523
+ 2.133,
524
+ 1.939
525
+ ],
526
+ "in.09": [
527
+ 3.751,
528
+ 3.492,
529
+ 3.544,
530
+ 2.98,
531
+ 4.599,
532
+ 2.974,
533
+ 2.989,
534
+ 3.133,
535
+ 2.942,
536
+ 3.057,
537
+ 4.55,
538
+ 2.005,
539
+ 2.113,
540
+ 2.573,
541
+ 2.441,
542
+ 2.811,
543
+ 2.802,
544
+ 2.547,
545
+ 2.269,
546
+ 1.899
547
+ ],
548
+ "in.10": [
549
+ 3.452,
550
+ 3.355,
551
+ 3.482,
552
+ 3.036,
553
+ 4.896,
554
+ 3.034,
555
+ 3.007,
556
+ 3.25,
557
+ 2.967,
558
+ 3.129,
559
+ 4.92,
560
+ 2.046,
561
+ 2.096,
562
+ 2.576,
563
+ 2.495,
564
+ 2.754,
565
+ 2.919,
566
+ 2.58,
567
+ 2.332,
568
+ 1.917
569
+ ],
570
+ "in.11": [
571
+ 3.573,
572
+ 3.233,
573
+ 3.465,
574
+ 3.125,
575
+ 4.232,
576
+ 2.906,
577
+ 3.002,
578
+ 3.359,
579
+ 2.911,
580
+ 3.121,
581
+ 4.231,
582
+ 2.034,
583
+ 2.215,
584
+ 2.571,
585
+ 2.379,
586
+ 2.943,
587
+ 2.865,
588
+ 2.586,
589
+ 2.401,
590
+ 1.911
591
+ ],
592
+ "in.12": [
593
+ 3.074,
594
+ 3.08,
595
+ 3.468,
596
+ 3.232,
597
+ 4.579,
598
+ 3.015,
599
+ 3.058,
600
+ 3.46,
601
+ 3.067,
602
+ 3.284,
603
+ 4.547,
604
+ 2.098,
605
+ 2.257,
606
+ 2.649,
607
+ 2.407,
608
+ 2.964,
609
+ 2.981,
610
+ 2.776,
611
+ 2.54,
612
+ 1.975
613
+ ],
614
+ "in.13": [
615
+ 5.862,
616
+ 3.127,
617
+ 3.542,
618
+ 3.492,
619
+ 4.177,
620
+ 3.036,
621
+ 3.081,
622
+ 3.569,
623
+ 3.078,
624
+ 3.284,
625
+ 4.337,
626
+ 2.091,
627
+ 2.302,
628
+ 2.646,
629
+ 2.331,
630
+ 2.982,
631
+ 3.007,
632
+ 2.979,
633
+ 2.558,
634
+ 1.977
635
+ ],
636
+ "in.14": [
637
+ 2.805,
638
+ 2.704,
639
+ 3.115,
640
+ 3.087,
641
+ 3.828,
642
+ 2.819,
643
+ 2.86,
644
+ 3.229,
645
+ 2.882,
646
+ 3.135,
647
+ 4.132,
648
+ 1.97,
649
+ 2.111,
650
+ 2.514,
651
+ 2.218,
652
+ 2.634,
653
+ 2.762,
654
+ 2.855,
655
+ 2.537,
656
+ 1.862
657
+ ],
658
+ "x": [
659
+ 1.242,
660
+ 1.175,
661
+ 1.516,
662
+ 1.5,
663
+ 1.646,
664
+ 1.602,
665
+ 1.833,
666
+ 2.159,
667
+ 2.126,
668
+ 2.174,
669
+ 2.183,
670
+ 1.644,
671
+ 1.676,
672
+ 1.825,
673
+ 1.962,
674
+ 2.339,
675
+ 1.963,
676
+ 1.943,
677
+ 2.472,
678
+ 1.431
679
+ ],
680
+ "u.1": [
681
+ 1.242,
682
+ 1.175,
683
+ 1.516,
684
+ 1.5,
685
+ 1.646,
686
+ 1.602,
687
+ 1.833,
688
+ 2.159,
689
+ 2.126,
690
+ 2.174,
691
+ 2.183,
692
+ 1.644,
693
+ 1.676,
694
+ 1.825,
695
+ 1.962,
696
+ 2.339,
697
+ 1.963,
698
+ 1.943,
699
+ 2.472,
700
+ 1.431
701
+ ],
702
+ "rec.1.16": [
703
+ 2.528,
704
+ 1.972,
705
+ 2.286,
706
+ 2.471,
707
+ 2.712,
708
+ 2.269,
709
+ 2.379,
710
+ 2.802,
711
+ 2.459,
712
+ 2.653,
713
+ 3.463,
714
+ 1.86,
715
+ 1.961,
716
+ 2.188,
717
+ 1.932,
718
+ 2.402,
719
+ 2.389,
720
+ 2.597,
721
+ 2.049,
722
+ 1.714
723
+ ],
724
+ "rec.1.17": [
725
+ 3.445,
726
+ 1.961,
727
+ 2.2,
728
+ 2.221,
729
+ 2.527,
730
+ 2.175,
731
+ 2.355,
732
+ 2.529,
733
+ 2.425,
734
+ 2.528,
735
+ 3.244,
736
+ 1.804,
737
+ 1.883,
738
+ 2.064,
739
+ 1.885,
740
+ 2.261,
741
+ 2.252,
742
+ 2.444,
743
+ 2.011,
744
+ 1.711
745
+ ],
746
+ "rec.1.18": [
747
+ 3.343,
748
+ 1.922,
749
+ 2.209,
750
+ 2.209,
751
+ 2.61,
752
+ 2.166,
753
+ 2.356,
754
+ 2.548,
755
+ 2.467,
756
+ 2.545,
757
+ 3.38,
758
+ 1.831,
759
+ 1.886,
760
+ 2.086,
761
+ 1.84,
762
+ 2.212,
763
+ 2.344,
764
+ 2.445,
765
+ 1.964,
766
+ 1.73
767
+ ],
768
+ "rec.1.19": [
769
+ 4.978,
770
+ 2.008,
771
+ 2.108,
772
+ 2.124,
773
+ 2.451,
774
+ 2.117,
775
+ 2.35,
776
+ 2.468,
777
+ 2.433,
778
+ 2.48,
779
+ 3.191,
780
+ 1.802,
781
+ 1.89,
782
+ 2.071,
783
+ 1.83,
784
+ 2.173,
785
+ 2.291,
786
+ 2.454,
787
+ 2.001,
788
+ 1.741
789
+ ],
790
+ "rec.1.20": [
791
+ 4.164,
792
+ 2.002,
793
+ 2.047,
794
+ 2.166,
795
+ 2.508,
796
+ 2.035,
797
+ 2.377,
798
+ 2.521,
799
+ 2.461,
800
+ 2.537,
801
+ 3.312,
802
+ 1.821,
803
+ 1.901,
804
+ 2.105,
805
+ 1.862,
806
+ 2.178,
807
+ 2.39,
808
+ 2.509,
809
+ 1.996,
810
+ 1.771
811
+ ],
812
+ "rec.1.21": [
813
+ 4.659,
814
+ 2.098,
815
+ 2.077,
816
+ 2.184,
817
+ 2.454,
818
+ 2.078,
819
+ 2.422,
820
+ 2.581,
821
+ 2.48,
822
+ 2.525,
823
+ 3.276,
824
+ 1.844,
825
+ 1.918,
826
+ 2.153,
827
+ 1.887,
828
+ 2.219,
829
+ 2.393,
830
+ 2.564,
831
+ 2.022,
832
+ 1.81
833
+ ],
834
+ "rec.1.22": [
835
+ 5.515,
836
+ 2.022,
837
+ 1.988,
838
+ 2.005,
839
+ 2.182,
840
+ 2.024,
841
+ 2.201,
842
+ 2.393,
843
+ 2.375,
844
+ 2.249,
845
+ 2.893,
846
+ 1.699,
847
+ 1.867,
848
+ 2.115,
849
+ 1.761,
850
+ 2.152,
851
+ 2.216,
852
+ 2.448,
853
+ 1.914,
854
+ 1.753
855
+ ],
856
+ "rec.1.23": [
857
+ 6.693,
858
+ 1.805,
859
+ 1.833,
860
+ 1.781,
861
+ 2.011,
862
+ 1.867,
863
+ 2.103,
864
+ 2.179,
865
+ 2.293,
866
+ 2.057,
867
+ 2.659,
868
+ 1.651,
869
+ 1.844,
870
+ 1.937,
871
+ 1.684,
872
+ 2.153,
873
+ 1.915,
874
+ 2.263,
875
+ 1.969,
876
+ 1.648
877
+ ],
878
+ "rec.1.24": [
879
+ 5.455,
880
+ 1.97,
881
+ 1.929,
882
+ 1.962,
883
+ 2.077,
884
+ 1.969,
885
+ 2.161,
886
+ 2.36,
887
+ 2.331,
888
+ 2.16,
889
+ 2.74,
890
+ 1.676,
891
+ 1.853,
892
+ 2.088,
893
+ 1.726,
894
+ 2.13,
895
+ 2.123,
896
+ 2.433,
897
+ 1.908,
898
+ 1.728
899
+ ],
900
+ "rec.1.25": [
901
+ 4.958,
902
+ 1.904,
903
+ 1.82,
904
+ 1.797,
905
+ 1.922,
906
+ 1.845,
907
+ 2.079,
908
+ 2.173,
909
+ 2.251,
910
+ 1.983,
911
+ 2.495,
912
+ 1.625,
913
+ 1.787,
914
+ 1.964,
915
+ 1.642,
916
+ 2.065,
917
+ 1.922,
918
+ 2.291,
919
+ 1.88,
920
+ 1.637
921
+ ],
922
+ "rec.1.26": [
923
+ 5.577,
924
+ 2.008,
925
+ 1.942,
926
+ 1.973,
927
+ 2.058,
928
+ 1.972,
929
+ 2.175,
930
+ 2.372,
931
+ 2.31,
932
+ 2.161,
933
+ 2.631,
934
+ 1.674,
935
+ 1.865,
936
+ 2.117,
937
+ 1.733,
938
+ 2.122,
939
+ 2.126,
940
+ 2.447,
941
+ 1.92,
942
+ 1.74
943
+ ],
944
+ "rec.1.27": [
945
+ 5.247,
946
+ 2.214,
947
+ 1.978,
948
+ 1.933,
949
+ 2.031,
950
+ 1.952,
951
+ 2.139,
952
+ 2.269,
953
+ 2.276,
954
+ 2.084,
955
+ 2.536,
956
+ 1.669,
957
+ 1.86,
958
+ 2.085,
959
+ 1.672,
960
+ 2.087,
961
+ 2.029,
962
+ 2.39,
963
+ 1.895,
964
+ 1.722
965
+ ],
966
+ "rec.1.28": [
967
+ 5.29,
968
+ 2.017,
969
+ 1.892,
970
+ 1.929,
971
+ 2.037,
972
+ 1.906,
973
+ 2.118,
974
+ 2.254,
975
+ 2.269,
976
+ 2.08,
977
+ 2.55,
978
+ 1.645,
979
+ 1.814,
980
+ 2.105,
981
+ 1.7,
982
+ 2.101,
983
+ 2.05,
984
+ 2.408,
985
+ 1.886,
986
+ 1.719
987
+ ],
988
+ "rec.1.29": [
989
+ 5.821,
990
+ 2.11,
991
+ 1.949,
992
+ 1.939,
993
+ 2.078,
994
+ 1.936,
995
+ 2.108,
996
+ 2.227,
997
+ 2.253,
998
+ 2.02,
999
+ 2.545,
1000
+ 1.638,
1001
+ 1.795,
1002
+ 2.123,
1003
+ 1.671,
1004
+ 2.09,
1005
+ 2.03,
1006
+ 2.47,
1007
+ 1.885,
1008
+ 1.714
1009
+ ],
1010
+ "rec.1.30": [
1011
+ 5.243,
1012
+ 2.012,
1013
+ 1.879,
1014
+ 1.864,
1015
+ 1.988,
1016
+ 1.85,
1017
+ 1.988,
1018
+ 2.119,
1019
+ 2.132,
1020
+ 1.892,
1021
+ 2.489,
1022
+ 1.546,
1023
+ 1.723,
1024
+ 2.014,
1025
+ 1.604,
1026
+ 2.025,
1027
+ 1.973,
1028
+ 2.42,
1029
+ 1.795,
1030
+ 1.643
1031
+ ],
1032
+ "rec.1.31": [
1033
+ 4.065,
1034
+ 1.861,
1035
+ 1.842,
1036
+ 1.743,
1037
+ 1.856,
1038
+ 1.821,
1039
+ 2.034,
1040
+ 1.979,
1041
+ 2.146,
1042
+ 1.809,
1043
+ 2.399,
1044
+ 1.573,
1045
+ 1.753,
1046
+ 1.963,
1047
+ 1.576,
1048
+ 2.054,
1049
+ 1.796,
1050
+ 2.232,
1051
+ 1.869,
1052
+ 1.57
1053
+ ],
1054
+ "rec.1.32": [
1055
+ 5.081,
1056
+ 2.024,
1057
+ 1.903,
1058
+ 1.846,
1059
+ 2.029,
1060
+ 1.844,
1061
+ 1.992,
1062
+ 2.092,
1063
+ 2.109,
1064
+ 1.855,
1065
+ 2.535,
1066
+ 1.519,
1067
+ 1.696,
1068
+ 1.953,
1069
+ 1.543,
1070
+ 1.981,
1071
+ 1.881,
1072
+ 2.312,
1073
+ 1.77,
1074
+ 1.593
1075
+ ],
1076
+ "rec.1.33": [
1077
+ 3.876,
1078
+ 1.884,
1079
+ 1.866,
1080
+ 1.774,
1081
+ 1.915,
1082
+ 1.844,
1083
+ 2.061,
1084
+ 1.997,
1085
+ 2.09,
1086
+ 1.857,
1087
+ 2.401,
1088
+ 1.541,
1089
+ 1.743,
1090
+ 1.941,
1091
+ 1.576,
1092
+ 1.997,
1093
+ 1.737,
1094
+ 2.283,
1095
+ 1.835,
1096
+ 1.558
1097
+ ],
1098
+ "rec.1.34": [
1099
+ 4.22,
1100
+ 2.054,
1101
+ 1.992,
1102
+ 1.871,
1103
+ 2.124,
1104
+ 1.911,
1105
+ 2.076,
1106
+ 2.098,
1107
+ 2.173,
1108
+ 1.9,
1109
+ 2.643,
1110
+ 1.542,
1111
+ 1.758,
1112
+ 1.975,
1113
+ 1.569,
1114
+ 2.011,
1115
+ 1.874,
1116
+ 2.414,
1117
+ 1.833,
1118
+ 1.621
1119
+ ],
1120
+ "rec.1.35": [
1121
+ 4.184,
1122
+ 2.032,
1123
+ 2.15,
1124
+ 1.807,
1125
+ 2.265,
1126
+ 2.006,
1127
+ 2.236,
1128
+ 2.102,
1129
+ 2.322,
1130
+ 1.871,
1131
+ 3.083,
1132
+ 1.608,
1133
+ 1.81,
1134
+ 1.918,
1135
+ 1.608,
1136
+ 2.047,
1137
+ 1.844,
1138
+ 2.32,
1139
+ 1.845,
1140
+ 1.627
1141
+ ],
1142
+ "rec.1.36": [
1143
+ 4.391,
1144
+ 1.986,
1145
+ 1.986,
1146
+ 1.951,
1147
+ 1.858,
1148
+ 1.91,
1149
+ 1.94,
1150
+ 2.146,
1151
+ 2.158,
1152
+ 1.893,
1153
+ 2.498,
1154
+ 1.51,
1155
+ 1.796,
1156
+ 1.951,
1157
+ 1.577,
1158
+ 2.021,
1159
+ 1.947,
1160
+ 2.476,
1161
+ 1.854,
1162
+ 1.643
1163
+ ],
1164
+ "rec.1.37": [
1165
+ 5.308,
1166
+ 2.07,
1167
+ 2.092,
1168
+ 2.009,
1169
+ 1.91,
1170
+ 1.995,
1171
+ 1.997,
1172
+ 2.144,
1173
+ 2.251,
1174
+ 1.881,
1175
+ 2.555,
1176
+ 1.55,
1177
+ 1.849,
1178
+ 1.952,
1179
+ 1.616,
1180
+ 2.06,
1181
+ 1.897,
1182
+ 2.352,
1183
+ 1.869,
1184
+ 1.654
1185
+ ],
1186
+ "rec.1.38": [
1187
+ 4.34,
1188
+ 1.786,
1189
+ 1.717,
1190
+ 1.798,
1191
+ 1.661,
1192
+ 1.804,
1193
+ 1.781,
1194
+ 2.026,
1195
+ 2.02,
1196
+ 1.806,
1197
+ 2.19,
1198
+ 1.451,
1199
+ 1.709,
1200
+ 1.859,
1201
+ 1.423,
1202
+ 1.912,
1203
+ 1.824,
1204
+ 2.266,
1205
+ 1.783,
1206
+ 1.579
1207
+ ],
1208
+ "rec.1.39": [
1209
+ 5.006,
1210
+ 1.77,
1211
+ 1.717,
1212
+ 1.785,
1213
+ 1.659,
1214
+ 1.813,
1215
+ 1.755,
1216
+ 1.991,
1217
+ 1.997,
1218
+ 1.769,
1219
+ 2.113,
1220
+ 1.451,
1221
+ 1.702,
1222
+ 1.807,
1223
+ 1.402,
1224
+ 1.868,
1225
+ 1.765,
1226
+ 2.199,
1227
+ 1.742,
1228
+ 1.543
1229
+ ],
1230
+ "rec.1.40": [
1231
+ 3.598,
1232
+ 1.696,
1233
+ 1.641,
1234
+ 1.753,
1235
+ 1.554,
1236
+ 1.728,
1237
+ 1.7,
1238
+ 2.004,
1239
+ 1.941,
1240
+ 1.757,
1241
+ 2.089,
1242
+ 1.43,
1243
+ 1.691,
1244
+ 1.749,
1245
+ 1.369,
1246
+ 1.818,
1247
+ 1.764,
1248
+ 2.147,
1249
+ 1.742,
1250
+ 1.537
1251
+ ],
1252
+ "rec.1.41": [
1253
+ 3.698,
1254
+ 1.674,
1255
+ 1.636,
1256
+ 1.757,
1257
+ 1.523,
1258
+ 1.719,
1259
+ 1.703,
1260
+ 1.983,
1261
+ 1.923,
1262
+ 1.751,
1263
+ 2.047,
1264
+ 1.424,
1265
+ 1.686,
1266
+ 1.732,
1267
+ 1.367,
1268
+ 1.826,
1269
+ 1.744,
1270
+ 2.153,
1271
+ 1.74,
1272
+ 1.532
1273
+ ],
1274
+ "rec.1.42": [
1275
+ 3.304,
1276
+ 1.657,
1277
+ 1.599,
1278
+ 1.734,
1279
+ 1.529,
1280
+ 1.708,
1281
+ 1.676,
1282
+ 1.993,
1283
+ 1.924,
1284
+ 1.744,
1285
+ 2.066,
1286
+ 1.403,
1287
+ 1.665,
1288
+ 1.71,
1289
+ 1.336,
1290
+ 1.776,
1291
+ 1.753,
1292
+ 2.102,
1293
+ 1.711,
1294
+ 1.527
1295
+ ],
1296
+ "rec.1.43": [
1297
+ 4.021,
1298
+ 1.807,
1299
+ 1.712,
1300
+ 1.833,
1301
+ 1.594,
1302
+ 1.778,
1303
+ 1.749,
1304
+ 2.007,
1305
+ 1.967,
1306
+ 1.82,
1307
+ 2.055,
1308
+ 1.47,
1309
+ 1.712,
1310
+ 1.74,
1311
+ 1.38,
1312
+ 1.838,
1313
+ 1.691,
1314
+ 2.095,
1315
+ 1.774,
1316
+ 1.548
1317
+ ],
1318
+ "rec.1.44": [
1319
+ 3.669,
1320
+ 1.672,
1321
+ 1.603,
1322
+ 1.748,
1323
+ 1.545,
1324
+ 1.713,
1325
+ 1.673,
1326
+ 2.002,
1327
+ 1.905,
1328
+ 1.768,
1329
+ 2.099,
1330
+ 1.408,
1331
+ 1.678,
1332
+ 1.715,
1333
+ 1.324,
1334
+ 1.79,
1335
+ 1.732,
1336
+ 2.104,
1337
+ 1.726,
1338
+ 1.526
1339
+ ],
1340
+ "rec.1.45": [
1341
+ 3.77,
1342
+ 1.708,
1343
+ 1.637,
1344
+ 1.808,
1345
+ 1.576,
1346
+ 1.75,
1347
+ 1.718,
1348
+ 2.049,
1349
+ 1.929,
1350
+ 1.809,
1351
+ 2.137,
1352
+ 1.424,
1353
+ 1.731,
1354
+ 1.731,
1355
+ 1.326,
1356
+ 1.821,
1357
+ 1.723,
1358
+ 2.104,
1359
+ 1.76,
1360
+ 1.568
1361
+ ],
1362
+ "rec.1.46": [
1363
+ 3.523,
1364
+ 1.655,
1365
+ 1.664,
1366
+ 1.733,
1367
+ 1.546,
1368
+ 1.734,
1369
+ 1.685,
1370
+ 2.013,
1371
+ 1.922,
1372
+ 1.773,
1373
+ 2.113,
1374
+ 1.405,
1375
+ 1.707,
1376
+ 1.683,
1377
+ 1.309,
1378
+ 1.792,
1379
+ 1.629,
1380
+ 2.089,
1381
+ 1.718,
1382
+ 1.528
1383
+ ],
1384
+ "rec.1.47": [
1385
+ 3.033,
1386
+ 1.663,
1387
+ 1.672,
1388
+ 1.757,
1389
+ 1.564,
1390
+ 1.736,
1391
+ 1.714,
1392
+ 2.043,
1393
+ 1.917,
1394
+ 1.805,
1395
+ 2.088,
1396
+ 1.42,
1397
+ 1.704,
1398
+ 1.703,
1399
+ 1.314,
1400
+ 1.798,
1401
+ 1.619,
1402
+ 2.053,
1403
+ 1.722,
1404
+ 1.515
1405
+ ],
1406
+ "rec.1.48": [
1407
+ 2.839,
1408
+ 1.72,
1409
+ 1.687,
1410
+ 1.615,
1411
+ 1.631,
1412
+ 1.749,
1413
+ 1.665,
1414
+ 1.99,
1415
+ 1.952,
1416
+ 1.742,
1417
+ 2.07,
1418
+ 1.455,
1419
+ 1.688,
1420
+ 1.688,
1421
+ 1.348,
1422
+ 1.716,
1423
+ 1.533,
1424
+ 1.919,
1425
+ 1.682,
1426
+ 1.503
1427
+ ],
1428
+ "rec.1.49": [
1429
+ 3.13,
1430
+ 1.718,
1431
+ 1.688,
1432
+ 1.662,
1433
+ 1.575,
1434
+ 1.719,
1435
+ 1.682,
1436
+ 1.975,
1437
+ 1.935,
1438
+ 1.746,
1439
+ 1.951,
1440
+ 1.471,
1441
+ 1.7,
1442
+ 1.721,
1443
+ 1.366,
1444
+ 1.744,
1445
+ 1.531,
1446
+ 1.885,
1447
+ 1.679,
1448
+ 1.506
1449
+ ],
1450
+ "rec.1.50": [
1451
+ 2.91,
1452
+ 1.713,
1453
+ 1.685,
1454
+ 1.622,
1455
+ 1.666,
1456
+ 1.75,
1457
+ 1.651,
1458
+ 1.971,
1459
+ 1.95,
1460
+ 1.738,
1461
+ 2.046,
1462
+ 1.443,
1463
+ 1.644,
1464
+ 1.692,
1465
+ 1.336,
1466
+ 1.704,
1467
+ 1.502,
1468
+ 1.884,
1469
+ 1.663,
1470
+ 1.502
1471
+ ],
1472
+ "rec.1.51": [
1473
+ 2.998,
1474
+ 1.732,
1475
+ 1.786,
1476
+ 1.67,
1477
+ 1.616,
1478
+ 1.727,
1479
+ 1.664,
1480
+ 2.01,
1481
+ 1.979,
1482
+ 1.755,
1483
+ 2.092,
1484
+ 1.46,
1485
+ 1.672,
1486
+ 1.712,
1487
+ 1.355,
1488
+ 1.77,
1489
+ 1.483,
1490
+ 1.855,
1491
+ 1.672,
1492
+ 1.523
1493
+ ],
1494
+ "rec.1.52": [
1495
+ 3.265,
1496
+ 1.693,
1497
+ 1.633,
1498
+ 1.593,
1499
+ 1.607,
1500
+ 1.719,
1501
+ 1.595,
1502
+ 2.017,
1503
+ 1.934,
1504
+ 1.748,
1505
+ 1.952,
1506
+ 1.401,
1507
+ 1.61,
1508
+ 1.686,
1509
+ 1.296,
1510
+ 1.683,
1511
+ 1.463,
1512
+ 1.824,
1513
+ 1.632,
1514
+ 1.53
1515
+ ],
1516
+ "rec.1.53": [
1517
+ 8.101,
1518
+ 1.781,
1519
+ 1.74,
1520
+ 1.719,
1521
+ 1.676,
1522
+ 1.792,
1523
+ 1.606,
1524
+ 2.093,
1525
+ 2.007,
1526
+ 1.779,
1527
+ 1.975,
1528
+ 1.443,
1529
+ 1.602,
1530
+ 1.708,
1531
+ 1.361,
1532
+ 1.725,
1533
+ 1.444,
1534
+ 1.788,
1535
+ 1.633,
1536
+ 1.569
1537
+ ],
1538
+ "rec.1.54": [
1539
+ 3.26,
1540
+ 1.614,
1541
+ 1.501,
1542
+ 1.408,
1543
+ 1.618,
1544
+ 1.59,
1545
+ 1.418,
1546
+ 1.569,
1547
+ 1.709,
1548
+ 1.457,
1549
+ 1.621,
1550
+ 1.366,
1551
+ 1.353,
1552
+ 1.494,
1553
+ 1.238,
1554
+ 1.398,
1555
+ 1.338,
1556
+ 1.517,
1557
+ 1.456,
1558
+ 1.311
1559
+ ],
1560
+ "s.1": [
1561
+ 3.041,
1562
+ 1.552,
1563
+ 1.482,
1564
+ 1.433,
1565
+ 1.467,
1566
+ 1.643,
1567
+ 1.454,
1568
+ 1.6,
1569
+ 1.744,
1570
+ 1.471,
1571
+ 1.554,
1572
+ 1.394,
1573
+ 1.395,
1574
+ 1.474,
1575
+ 1.275,
1576
+ 1.498,
1577
+ 1.341,
1578
+ 1.587,
1579
+ 1.482,
1580
+ 1.297
1581
+ ],
1582
+ "out.56": [
1583
+ 2.396,
1584
+ 1.5,
1585
+ 1.43,
1586
+ 1.375,
1587
+ 1.539,
1588
+ 1.487,
1589
+ 1.329,
1590
+ 1.52,
1591
+ 1.602,
1592
+ 1.454,
1593
+ 1.451,
1594
+ 1.312,
1595
+ 1.31,
1596
+ 1.456,
1597
+ 1.231,
1598
+ 1.364,
1599
+ 1.29,
1600
+ 1.422,
1601
+ 1.415,
1602
+ 1.315
1603
+ ],
1604
+ "out.57": [
1605
+ 2.566,
1606
+ 1.516,
1607
+ 1.46,
1608
+ 1.405,
1609
+ 1.548,
1610
+ 1.513,
1611
+ 1.359,
1612
+ 1.543,
1613
+ 1.617,
1614
+ 1.458,
1615
+ 1.464,
1616
+ 1.333,
1617
+ 1.313,
1618
+ 1.443,
1619
+ 1.266,
1620
+ 1.405,
1621
+ 1.289,
1622
+ 1.436,
1623
+ 1.43,
1624
+ 1.309
1625
+ ],
1626
+ "out.58": [
1627
+ 2.522,
1628
+ 1.494,
1629
+ 1.429,
1630
+ 1.366,
1631
+ 1.551,
1632
+ 1.505,
1633
+ 1.332,
1634
+ 1.521,
1635
+ 1.623,
1636
+ 1.456,
1637
+ 1.461,
1638
+ 1.336,
1639
+ 1.286,
1640
+ 1.413,
1641
+ 1.267,
1642
+ 1.389,
1643
+ 1.264,
1644
+ 1.394,
1645
+ 1.414,
1646
+ 1.299
1647
+ ],
1648
+ "out.59": [
1649
+ 2.874,
1650
+ 1.525,
1651
+ 1.484,
1652
+ 1.426,
1653
+ 1.535,
1654
+ 1.537,
1655
+ 1.354,
1656
+ 1.557,
1657
+ 1.629,
1658
+ 1.47,
1659
+ 1.523,
1660
+ 1.373,
1661
+ 1.298,
1662
+ 1.417,
1663
+ 1.322,
1664
+ 1.434,
1665
+ 1.281,
1666
+ 1.417,
1667
+ 1.423,
1668
+ 1.296
1669
+ ],
1670
+ "out.60": [
1671
+ 2.634,
1672
+ 1.348,
1673
+ 1.302,
1674
+ 1.356,
1675
+ 1.301,
1676
+ 1.381,
1677
+ 1.266,
1678
+ 1.413,
1679
+ 1.492,
1680
+ 1.328,
1681
+ 1.469,
1682
+ 1.275,
1683
+ 1.189,
1684
+ 1.268,
1685
+ 1.253,
1686
+ 1.337,
1687
+ 1.17,
1688
+ 1.255,
1689
+ 1.297,
1690
+ 1.206
1691
+ ],
1692
+ "out.61": [
1693
+ 7.185,
1694
+ 1.559,
1695
+ 1.589,
1696
+ 1.83,
1697
+ 1.223,
1698
+ 1.548,
1699
+ 1.33,
1700
+ 1.659,
1701
+ 1.566,
1702
+ 1.549,
1703
+ 2.061,
1704
+ 1.554,
1705
+ 1.133,
1706
+ 1.219,
1707
+ 1.64,
1708
+ 1.499,
1709
+ 1.151,
1710
+ 1.229,
1711
+ 1.246,
1712
+ 1.133
1713
+ ],
1714
+ "out.62": [
1715
+ 1.29,
1716
+ 0.94,
1717
+ 0.945,
1718
+ 0.977,
1719
+ 1.012,
1720
+ 1.057,
1721
+ 1.061,
1722
+ 1.086,
1723
+ 1.121,
1724
+ 1.111,
1725
+ 1.109,
1726
+ 1.015,
1727
+ 1.076,
1728
+ 1.09,
1729
+ 1.041,
1730
+ 1.106,
1731
+ 1.041,
1732
+ 1.078,
1733
+ 1.128,
1734
+ 1.074
1735
+ ],
1736
+ "out.63": [
1737
+ 1.339,
1738
+ 0.961,
1739
+ 0.983,
1740
+ 1.005,
1741
+ 1.026,
1742
+ 1.073,
1743
+ 1.105,
1744
+ 1.116,
1745
+ 1.139,
1746
+ 1.114,
1747
+ 1.128,
1748
+ 1.047,
1749
+ 1.125,
1750
+ 1.108,
1751
+ 1.091,
1752
+ 1.125,
1753
+ 1.076,
1754
+ 1.106,
1755
+ 1.15,
1756
+ 1.081
1757
+ ],
1758
+ "out.64": [
1759
+ 1.355,
1760
+ 0.928,
1761
+ 0.944,
1762
+ 0.997,
1763
+ 0.995,
1764
+ 1.058,
1765
+ 1.076,
1766
+ 1.085,
1767
+ 1.1,
1768
+ 1.099,
1769
+ 1.126,
1770
+ 1.034,
1771
+ 1.077,
1772
+ 1.086,
1773
+ 1.057,
1774
+ 1.105,
1775
+ 1.038,
1776
+ 1.072,
1777
+ 1.143,
1778
+ 1.058
1779
+ ],
1780
+ "out.65": [
1781
+ 4.247,
1782
+ 1.404,
1783
+ 1.416,
1784
+ 1.821,
1785
+ 0.899,
1786
+ 1.414,
1787
+ 1.557,
1788
+ 1.455,
1789
+ 1.376,
1790
+ 1.38,
1791
+ 1.941,
1792
+ 1.593,
1793
+ 1.315,
1794
+ 1.323,
1795
+ 1.66,
1796
+ 1.389,
1797
+ 1.177,
1798
+ 1.007,
1799
+ 1.327,
1800
+ 0.973
1801
+ ],
1802
+ "out.66": [
1803
+ 1.425,
1804
+ 0.949,
1805
+ 0.962,
1806
+ 1.009,
1807
+ 0.981,
1808
+ 1.045,
1809
+ 1.052,
1810
+ 1.058,
1811
+ 1.09,
1812
+ 1.09,
1813
+ 1.114,
1814
+ 1.038,
1815
+ 1.079,
1816
+ 1.064,
1817
+ 1.048,
1818
+ 1.095,
1819
+ 1.04,
1820
+ 1.042,
1821
+ 1.099,
1822
+ 1.046
1823
+ ],
1824
+ "out.67": [
1825
+ 3.29,
1826
+ 1.298,
1827
+ 1.319,
1828
+ 1.641,
1829
+ 0.954,
1830
+ 1.311,
1831
+ 1.446,
1832
+ 1.354,
1833
+ 1.352,
1834
+ 1.288,
1835
+ 1.705,
1836
+ 1.459,
1837
+ 1.263,
1838
+ 1.246,
1839
+ 1.486,
1840
+ 1.338,
1841
+ 1.169,
1842
+ 1.014,
1843
+ 1.24,
1844
+ 0.994
1845
+ ],
1846
+ "out.68": [
1847
+ 1.518,
1848
+ 0.935,
1849
+ 0.935,
1850
+ 1.028,
1851
+ 0.945,
1852
+ 0.995,
1853
+ 0.998,
1854
+ 0.983,
1855
+ 1.004,
1856
+ 1.016,
1857
+ 1.077,
1858
+ 1.033,
1859
+ 0.971,
1860
+ 0.961,
1861
+ 1.028,
1862
+ 1.022,
1863
+ 0.95,
1864
+ 0.927,
1865
+ 0.991,
1866
+ 0.97
1867
+ ],
1868
+ "out.69": [
1869
+ 3.502,
1870
+ 1.376,
1871
+ 1.459,
1872
+ 1.904,
1873
+ 0.981,
1874
+ 1.407,
1875
+ 1.577,
1876
+ 1.407,
1877
+ 1.482,
1878
+ 1.345,
1879
+ 1.893,
1880
+ 1.75,
1881
+ 1.324,
1882
+ 1.219,
1883
+ 1.617,
1884
+ 1.509,
1885
+ 1.107,
1886
+ 0.937,
1887
+ 1.26,
1888
+ 0.937
1889
+ ],
1890
+ "out.70": [
1891
+ 0.967,
1892
+ 0.935,
1893
+ 0.941,
1894
+ 0.969,
1895
+ 0.97,
1896
+ 1.001,
1897
+ 0.972,
1898
+ 0.984,
1899
+ 0.983,
1900
+ 1.002,
1901
+ 0.996,
1902
+ 0.98,
1903
+ 0.96,
1904
+ 0.987,
1905
+ 0.975,
1906
+ 1.003,
1907
+ 0.978,
1908
+ 0.977,
1909
+ 0.972,
1910
+ 0.979
1911
+ ]
1912
+ },
1913
+ "heldout": {
1914
+ "emb": {
1915
+ "top1_agree_model_skip16": 0.001,
1916
+ "top1_agree_model_skip4": 0.001,
1917
+ "top1_agree_model_skip0": 0.0,
1918
+ "top1_agree_model_logit_lens": 0.002,
1919
+ "nonsemantic_share_top10_skip16": 0.735,
1920
+ "nonsemantic_share_top10_skip0": 0.637,
1921
+ "top10_jaccard_skip0_vs_16": 0.366,
1922
+ "top10_jaccard_skip4_vs_16": 0.44
1923
+ },
1924
+ "in.00": {
1925
+ "top1_agree_model_skip16": 0.002,
1926
+ "top1_agree_model_skip4": 0.0,
1927
+ "top1_agree_model_skip0": 0.0,
1928
+ "top1_agree_model_logit_lens": 0.001,
1929
+ "nonsemantic_share_top10_skip16": 0.806,
1930
+ "nonsemantic_share_top10_skip0": 0.703,
1931
+ "top10_jaccard_skip0_vs_16": 0.345,
1932
+ "top10_jaccard_skip4_vs_16": 0.421
1933
+ },
1934
+ "in.01": {
1935
+ "top1_agree_model_skip16": 0.003,
1936
+ "top1_agree_model_skip4": 0.002,
1937
+ "top1_agree_model_skip0": 0.0,
1938
+ "top1_agree_model_logit_lens": 0.0,
1939
+ "nonsemantic_share_top10_skip16": 0.733,
1940
+ "nonsemantic_share_top10_skip0": 0.736,
1941
+ "top10_jaccard_skip0_vs_16": 0.502,
1942
+ "top10_jaccard_skip4_vs_16": 0.586
1943
+ },
1944
+ "in.02": {
1945
+ "top1_agree_model_skip16": 0.004,
1946
+ "top1_agree_model_skip4": 0.002,
1947
+ "top1_agree_model_skip0": 0.002,
1948
+ "top1_agree_model_logit_lens": 0.0,
1949
+ "nonsemantic_share_top10_skip16": 0.75,
1950
+ "nonsemantic_share_top10_skip0": 0.742,
1951
+ "top10_jaccard_skip0_vs_16": 0.505,
1952
+ "top10_jaccard_skip4_vs_16": 0.557
1953
+ },
1954
+ "in.03": {
1955
+ "top1_agree_model_skip16": 0.002,
1956
+ "top1_agree_model_skip4": 0.001,
1957
+ "top1_agree_model_skip0": 0.0,
1958
+ "top1_agree_model_logit_lens": 0.001,
1959
+ "nonsemantic_share_top10_skip16": 0.714,
1960
+ "nonsemantic_share_top10_skip0": 0.681,
1961
+ "top10_jaccard_skip0_vs_16": 0.486,
1962
+ "top10_jaccard_skip4_vs_16": 0.53
1963
+ },
1964
+ "in.04": {
1965
+ "top1_agree_model_skip16": 0.0,
1966
+ "top1_agree_model_skip4": 0.0,
1967
+ "top1_agree_model_skip0": 0.0,
1968
+ "top1_agree_model_logit_lens": 0.0,
1969
+ "nonsemantic_share_top10_skip16": 0.745,
1970
+ "nonsemantic_share_top10_skip0": 0.675,
1971
+ "top10_jaccard_skip0_vs_16": 0.511,
1972
+ "top10_jaccard_skip4_vs_16": 0.585
1973
+ },
1974
+ "in.05": {
1975
+ "top1_agree_model_skip16": 0.001,
1976
+ "top1_agree_model_skip4": 0.0,
1977
+ "top1_agree_model_skip0": 0.0,
1978
+ "top1_agree_model_logit_lens": 0.0,
1979
+ "nonsemantic_share_top10_skip16": 0.694,
1980
+ "nonsemantic_share_top10_skip0": 0.677,
1981
+ "top10_jaccard_skip0_vs_16": 0.541,
1982
+ "top10_jaccard_skip4_vs_16": 0.612
1983
+ },
1984
+ "in.06": {
1985
+ "top1_agree_model_skip16": 0.002,
1986
+ "top1_agree_model_skip4": 0.001,
1987
+ "top1_agree_model_skip0": 0.0,
1988
+ "top1_agree_model_logit_lens": 0.0,
1989
+ "nonsemantic_share_top10_skip16": 0.729,
1990
+ "nonsemantic_share_top10_skip0": 0.725,
1991
+ "top10_jaccard_skip0_vs_16": 0.56,
1992
+ "top10_jaccard_skip4_vs_16": 0.62
1993
+ },
1994
+ "in.07": {
1995
+ "top1_agree_model_skip16": 0.001,
1996
+ "top1_agree_model_skip4": 0.001,
1997
+ "top1_agree_model_skip0": 0.0,
1998
+ "top1_agree_model_logit_lens": 0.001,
1999
+ "nonsemantic_share_top10_skip16": 0.684,
2000
+ "nonsemantic_share_top10_skip0": 0.716,
2001
+ "top10_jaccard_skip0_vs_16": 0.557,
2002
+ "top10_jaccard_skip4_vs_16": 0.617
2003
+ },
2004
+ "in.08": {
2005
+ "top1_agree_model_skip16": 0.002,
2006
+ "top1_agree_model_skip4": 0.001,
2007
+ "top1_agree_model_skip0": 0.001,
2008
+ "top1_agree_model_logit_lens": 0.001,
2009
+ "nonsemantic_share_top10_skip16": 0.733,
2010
+ "nonsemantic_share_top10_skip0": 0.732,
2011
+ "top10_jaccard_skip0_vs_16": 0.566,
2012
+ "top10_jaccard_skip4_vs_16": 0.639
2013
+ },
2014
+ "in.09": {
2015
+ "top1_agree_model_skip16": 0.003,
2016
+ "top1_agree_model_skip4": 0.002,
2017
+ "top1_agree_model_skip0": 0.001,
2018
+ "top1_agree_model_logit_lens": 0.0,
2019
+ "nonsemantic_share_top10_skip16": 0.692,
2020
+ "nonsemantic_share_top10_skip0": 0.642,
2021
+ "top10_jaccard_skip0_vs_16": 0.514,
2022
+ "top10_jaccard_skip4_vs_16": 0.595
2023
+ },
2024
+ "in.10": {
2025
+ "top1_agree_model_skip16": 0.003,
2026
+ "top1_agree_model_skip4": 0.001,
2027
+ "top1_agree_model_skip0": 0.0,
2028
+ "top1_agree_model_logit_lens": 0.0,
2029
+ "nonsemantic_share_top10_skip16": 0.638,
2030
+ "nonsemantic_share_top10_skip0": 0.596,
2031
+ "top10_jaccard_skip0_vs_16": 0.547,
2032
+ "top10_jaccard_skip4_vs_16": 0.617
2033
+ },
2034
+ "in.11": {
2035
+ "top1_agree_model_skip16": 0.006,
2036
+ "top1_agree_model_skip4": 0.004,
2037
+ "top1_agree_model_skip0": 0.003,
2038
+ "top1_agree_model_logit_lens": 0.002,
2039
+ "nonsemantic_share_top10_skip16": 0.617,
2040
+ "nonsemantic_share_top10_skip0": 0.558,
2041
+ "top10_jaccard_skip0_vs_16": 0.529,
2042
+ "top10_jaccard_skip4_vs_16": 0.595
2043
+ },
2044
+ "in.12": {
2045
+ "top1_agree_model_skip16": 0.003,
2046
+ "top1_agree_model_skip4": 0.001,
2047
+ "top1_agree_model_skip0": 0.0,
2048
+ "top1_agree_model_logit_lens": 0.0,
2049
+ "nonsemantic_share_top10_skip16": 0.667,
2050
+ "nonsemantic_share_top10_skip0": 0.602,
2051
+ "top10_jaccard_skip0_vs_16": 0.563,
2052
+ "top10_jaccard_skip4_vs_16": 0.64
2053
+ },
2054
+ "in.13": {
2055
+ "top1_agree_model_skip16": 0.009,
2056
+ "top1_agree_model_skip4": 0.002,
2057
+ "top1_agree_model_skip0": 0.001,
2058
+ "top1_agree_model_logit_lens": 0.0,
2059
+ "nonsemantic_share_top10_skip16": 0.652,
2060
+ "nonsemantic_share_top10_skip0": 0.551,
2061
+ "top10_jaccard_skip0_vs_16": 0.458,
2062
+ "top10_jaccard_skip4_vs_16": 0.604
2063
+ },
2064
+ "in.14": {
2065
+ "top1_agree_model_skip16": 0.018,
2066
+ "top1_agree_model_skip4": 0.009,
2067
+ "top1_agree_model_skip0": 0.006,
2068
+ "top1_agree_model_logit_lens": 0.001,
2069
+ "nonsemantic_share_top10_skip16": 0.441,
2070
+ "nonsemantic_share_top10_skip0": 0.457,
2071
+ "top10_jaccard_skip0_vs_16": 0.552,
2072
+ "top10_jaccard_skip4_vs_16": 0.654
2073
+ },
2074
+ "x": {
2075
+ "top1_agree_model_skip16": 0.023,
2076
+ "top1_agree_model_skip4": 0.015,
2077
+ "top1_agree_model_skip0": 0.015,
2078
+ "top1_agree_model_logit_lens": 0.001,
2079
+ "nonsemantic_share_top10_skip16": 0.403,
2080
+ "nonsemantic_share_top10_skip0": 0.427,
2081
+ "top10_jaccard_skip0_vs_16": 0.626,
2082
+ "top10_jaccard_skip4_vs_16": 0.671
2083
+ },
2084
+ "u.1": {
2085
+ "top1_agree_model_skip16": 0.021,
2086
+ "top1_agree_model_skip4": 0.015,
2087
+ "top1_agree_model_skip0": 0.015,
2088
+ "top1_agree_model_logit_lens": 0.001,
2089
+ "nonsemantic_share_top10_skip16": 0.45,
2090
+ "nonsemantic_share_top10_skip0": 0.466,
2091
+ "top10_jaccard_skip0_vs_16": 0.649,
2092
+ "top10_jaccard_skip4_vs_16": 0.696
2093
+ },
2094
+ "rec.1.16": {
2095
+ "top1_agree_model_skip16": 0.028,
2096
+ "top1_agree_model_skip4": 0.017,
2097
+ "top1_agree_model_skip0": 0.013,
2098
+ "top1_agree_model_logit_lens": 0.003,
2099
+ "nonsemantic_share_top10_skip16": 0.416,
2100
+ "nonsemantic_share_top10_skip0": 0.413,
2101
+ "top10_jaccard_skip0_vs_16": 0.59,
2102
+ "top10_jaccard_skip4_vs_16": 0.687
2103
+ },
2104
+ "rec.1.17": {
2105
+ "top1_agree_model_skip16": 0.026,
2106
+ "top1_agree_model_skip4": 0.016,
2107
+ "top1_agree_model_skip0": 0.011,
2108
+ "top1_agree_model_logit_lens": 0.0,
2109
+ "nonsemantic_share_top10_skip16": 0.367,
2110
+ "nonsemantic_share_top10_skip0": 0.403,
2111
+ "top10_jaccard_skip0_vs_16": 0.605,
2112
+ "top10_jaccard_skip4_vs_16": 0.696
2113
+ },
2114
+ "rec.1.18": {
2115
+ "top1_agree_model_skip16": 0.026,
2116
+ "top1_agree_model_skip4": 0.018,
2117
+ "top1_agree_model_skip0": 0.013,
2118
+ "top1_agree_model_logit_lens": 0.002,
2119
+ "nonsemantic_share_top10_skip16": 0.394,
2120
+ "nonsemantic_share_top10_skip0": 0.425,
2121
+ "top10_jaccard_skip0_vs_16": 0.621,
2122
+ "top10_jaccard_skip4_vs_16": 0.722
2123
+ },
2124
+ "rec.1.19": {
2125
+ "top1_agree_model_skip16": 0.025,
2126
+ "top1_agree_model_skip4": 0.022,
2127
+ "top1_agree_model_skip0": 0.017,
2128
+ "top1_agree_model_logit_lens": 0.001,
2129
+ "nonsemantic_share_top10_skip16": 0.216,
2130
+ "nonsemantic_share_top10_skip0": 0.261,
2131
+ "top10_jaccard_skip0_vs_16": 0.613,
2132
+ "top10_jaccard_skip4_vs_16": 0.722
2133
+ },
2134
+ "rec.1.20": {
2135
+ "top1_agree_model_skip16": 0.029,
2136
+ "top1_agree_model_skip4": 0.023,
2137
+ "top1_agree_model_skip0": 0.016,
2138
+ "top1_agree_model_logit_lens": 0.002,
2139
+ "nonsemantic_share_top10_skip16": 0.243,
2140
+ "nonsemantic_share_top10_skip0": 0.271,
2141
+ "top10_jaccard_skip0_vs_16": 0.628,
2142
+ "top10_jaccard_skip4_vs_16": 0.737
2143
+ },
2144
+ "rec.1.21": {
2145
+ "top1_agree_model_skip16": 0.025,
2146
+ "top1_agree_model_skip4": 0.019,
2147
+ "top1_agree_model_skip0": 0.016,
2148
+ "top1_agree_model_logit_lens": 0.003,
2149
+ "nonsemantic_share_top10_skip16": 0.233,
2150
+ "nonsemantic_share_top10_skip0": 0.258,
2151
+ "top10_jaccard_skip0_vs_16": 0.635,
2152
+ "top10_jaccard_skip4_vs_16": 0.762
2153
+ },
2154
+ "rec.1.22": {
2155
+ "top1_agree_model_skip16": 0.026,
2156
+ "top1_agree_model_skip4": 0.02,
2157
+ "top1_agree_model_skip0": 0.013,
2158
+ "top1_agree_model_logit_lens": 0.001,
2159
+ "nonsemantic_share_top10_skip16": 0.309,
2160
+ "nonsemantic_share_top10_skip0": 0.312,
2161
+ "top10_jaccard_skip0_vs_16": 0.606,
2162
+ "top10_jaccard_skip4_vs_16": 0.75
2163
+ },
2164
+ "rec.1.23": {
2165
+ "top1_agree_model_skip16": 0.025,
2166
+ "top1_agree_model_skip4": 0.022,
2167
+ "top1_agree_model_skip0": 0.018,
2168
+ "top1_agree_model_logit_lens": 0.0,
2169
+ "nonsemantic_share_top10_skip16": 0.23,
2170
+ "nonsemantic_share_top10_skip0": 0.262,
2171
+ "top10_jaccard_skip0_vs_16": 0.59,
2172
+ "top10_jaccard_skip4_vs_16": 0.769
2173
+ },
2174
+ "rec.1.24": {
2175
+ "top1_agree_model_skip16": 0.027,
2176
+ "top1_agree_model_skip4": 0.02,
2177
+ "top1_agree_model_skip0": 0.017,
2178
+ "top1_agree_model_logit_lens": 0.0,
2179
+ "nonsemantic_share_top10_skip16": 0.269,
2180
+ "nonsemantic_share_top10_skip0": 0.288,
2181
+ "top10_jaccard_skip0_vs_16": 0.615,
2182
+ "top10_jaccard_skip4_vs_16": 0.752
2183
+ },
2184
+ "rec.1.25": {
2185
+ "top1_agree_model_skip16": 0.028,
2186
+ "top1_agree_model_skip4": 0.026,
2187
+ "top1_agree_model_skip0": 0.02,
2188
+ "top1_agree_model_logit_lens": 0.0,
2189
+ "nonsemantic_share_top10_skip16": 0.237,
2190
+ "nonsemantic_share_top10_skip0": 0.275,
2191
+ "top10_jaccard_skip0_vs_16": 0.639,
2192
+ "top10_jaccard_skip4_vs_16": 0.769
2193
+ },
2194
+ "rec.1.26": {
2195
+ "top1_agree_model_skip16": 0.029,
2196
+ "top1_agree_model_skip4": 0.018,
2197
+ "top1_agree_model_skip0": 0.013,
2198
+ "top1_agree_model_logit_lens": 0.0,
2199
+ "nonsemantic_share_top10_skip16": 0.276,
2200
+ "nonsemantic_share_top10_skip0": 0.328,
2201
+ "top10_jaccard_skip0_vs_16": 0.605,
2202
+ "top10_jaccard_skip4_vs_16": 0.762
2203
+ },
2204
+ "rec.1.27": {
2205
+ "top1_agree_model_skip16": 0.025,
2206
+ "top1_agree_model_skip4": 0.018,
2207
+ "top1_agree_model_skip0": 0.015,
2208
+ "top1_agree_model_logit_lens": 0.0,
2209
+ "nonsemantic_share_top10_skip16": 0.29,
2210
+ "nonsemantic_share_top10_skip0": 0.312,
2211
+ "top10_jaccard_skip0_vs_16": 0.607,
2212
+ "top10_jaccard_skip4_vs_16": 0.774
2213
+ },
2214
+ "rec.1.28": {
2215
+ "top1_agree_model_skip16": 0.027,
2216
+ "top1_agree_model_skip4": 0.02,
2217
+ "top1_agree_model_skip0": 0.017,
2218
+ "top1_agree_model_logit_lens": 0.0,
2219
+ "nonsemantic_share_top10_skip16": 0.262,
2220
+ "nonsemantic_share_top10_skip0": 0.318,
2221
+ "top10_jaccard_skip0_vs_16": 0.604,
2222
+ "top10_jaccard_skip4_vs_16": 0.772
2223
+ },
2224
+ "rec.1.29": {
2225
+ "top1_agree_model_skip16": 0.028,
2226
+ "top1_agree_model_skip4": 0.02,
2227
+ "top1_agree_model_skip0": 0.018,
2228
+ "top1_agree_model_logit_lens": 0.0,
2229
+ "nonsemantic_share_top10_skip16": 0.228,
2230
+ "nonsemantic_share_top10_skip0": 0.287,
2231
+ "top10_jaccard_skip0_vs_16": 0.595,
2232
+ "top10_jaccard_skip4_vs_16": 0.778
2233
+ },
2234
+ "rec.1.30": {
2235
+ "top1_agree_model_skip16": 0.028,
2236
+ "top1_agree_model_skip4": 0.022,
2237
+ "top1_agree_model_skip0": 0.02,
2238
+ "top1_agree_model_logit_lens": 0.0,
2239
+ "nonsemantic_share_top10_skip16": 0.266,
2240
+ "nonsemantic_share_top10_skip0": 0.322,
2241
+ "top10_jaccard_skip0_vs_16": 0.583,
2242
+ "top10_jaccard_skip4_vs_16": 0.782
2243
+ },
2244
+ "rec.1.31": {
2245
+ "top1_agree_model_skip16": 0.029,
2246
+ "top1_agree_model_skip4": 0.026,
2247
+ "top1_agree_model_skip0": 0.021,
2248
+ "top1_agree_model_logit_lens": 0.0,
2249
+ "nonsemantic_share_top10_skip16": 0.213,
2250
+ "nonsemantic_share_top10_skip0": 0.275,
2251
+ "top10_jaccard_skip0_vs_16": 0.661,
2252
+ "top10_jaccard_skip4_vs_16": 0.79
2253
+ },
2254
+ "rec.1.32": {
2255
+ "top1_agree_model_skip16": 0.028,
2256
+ "top1_agree_model_skip4": 0.026,
2257
+ "top1_agree_model_skip0": 0.018,
2258
+ "top1_agree_model_logit_lens": 0.001,
2259
+ "nonsemantic_share_top10_skip16": 0.254,
2260
+ "nonsemantic_share_top10_skip0": 0.316,
2261
+ "top10_jaccard_skip0_vs_16": 0.624,
2262
+ "top10_jaccard_skip4_vs_16": 0.782
2263
+ },
2264
+ "rec.1.33": {
2265
+ "top1_agree_model_skip16": 0.033,
2266
+ "top1_agree_model_skip4": 0.027,
2267
+ "top1_agree_model_skip0": 0.026,
2268
+ "top1_agree_model_logit_lens": 0.0,
2269
+ "nonsemantic_share_top10_skip16": 0.243,
2270
+ "nonsemantic_share_top10_skip0": 0.297,
2271
+ "top10_jaccard_skip0_vs_16": 0.673,
2272
+ "top10_jaccard_skip4_vs_16": 0.792
2273
+ },
2274
+ "rec.1.34": {
2275
+ "top1_agree_model_skip16": 0.033,
2276
+ "top1_agree_model_skip4": 0.029,
2277
+ "top1_agree_model_skip0": 0.026,
2278
+ "top1_agree_model_logit_lens": 0.002,
2279
+ "nonsemantic_share_top10_skip16": 0.245,
2280
+ "nonsemantic_share_top10_skip0": 0.288,
2281
+ "top10_jaccard_skip0_vs_16": 0.676,
2282
+ "top10_jaccard_skip4_vs_16": 0.807
2283
+ },
2284
+ "rec.1.35": {
2285
+ "top1_agree_model_skip16": 0.032,
2286
+ "top1_agree_model_skip4": 0.027,
2287
+ "top1_agree_model_skip0": 0.026,
2288
+ "top1_agree_model_logit_lens": 0.002,
2289
+ "nonsemantic_share_top10_skip16": 0.247,
2290
+ "nonsemantic_share_top10_skip0": 0.298,
2291
+ "top10_jaccard_skip0_vs_16": 0.689,
2292
+ "top10_jaccard_skip4_vs_16": 0.8
2293
+ },
2294
+ "rec.1.36": {
2295
+ "top1_agree_model_skip16": 0.033,
2296
+ "top1_agree_model_skip4": 0.03,
2297
+ "top1_agree_model_skip0": 0.026,
2298
+ "top1_agree_model_logit_lens": 0.006,
2299
+ "nonsemantic_share_top10_skip16": 0.239,
2300
+ "nonsemantic_share_top10_skip0": 0.271,
2301
+ "top10_jaccard_skip0_vs_16": 0.688,
2302
+ "top10_jaccard_skip4_vs_16": 0.8
2303
+ },
2304
+ "rec.1.37": {
2305
+ "top1_agree_model_skip16": 0.038,
2306
+ "top1_agree_model_skip4": 0.031,
2307
+ "top1_agree_model_skip0": 0.026,
2308
+ "top1_agree_model_logit_lens": 0.002,
2309
+ "nonsemantic_share_top10_skip16": 0.236,
2310
+ "nonsemantic_share_top10_skip0": 0.275,
2311
+ "top10_jaccard_skip0_vs_16": 0.681,
2312
+ "top10_jaccard_skip4_vs_16": 0.805
2313
+ },
2314
+ "rec.1.38": {
2315
+ "top1_agree_model_skip16": 0.041,
2316
+ "top1_agree_model_skip4": 0.039,
2317
+ "top1_agree_model_skip0": 0.032,
2318
+ "top1_agree_model_logit_lens": 0.015,
2319
+ "nonsemantic_share_top10_skip16": 0.234,
2320
+ "nonsemantic_share_top10_skip0": 0.257,
2321
+ "top10_jaccard_skip0_vs_16": 0.717,
2322
+ "top10_jaccard_skip4_vs_16": 0.817
2323
+ },
2324
+ "rec.1.39": {
2325
+ "top1_agree_model_skip16": 0.042,
2326
+ "top1_agree_model_skip4": 0.041,
2327
+ "top1_agree_model_skip0": 0.039,
2328
+ "top1_agree_model_logit_lens": 0.018,
2329
+ "nonsemantic_share_top10_skip16": 0.21,
2330
+ "nonsemantic_share_top10_skip0": 0.229,
2331
+ "top10_jaccard_skip0_vs_16": 0.723,
2332
+ "top10_jaccard_skip4_vs_16": 0.834
2333
+ },
2334
+ "rec.1.40": {
2335
+ "top1_agree_model_skip16": 0.052,
2336
+ "top1_agree_model_skip4": 0.046,
2337
+ "top1_agree_model_skip0": 0.046,
2338
+ "top1_agree_model_logit_lens": 0.017,
2339
+ "nonsemantic_share_top10_skip16": 0.177,
2340
+ "nonsemantic_share_top10_skip0": 0.204,
2341
+ "top10_jaccard_skip0_vs_16": 0.746,
2342
+ "top10_jaccard_skip4_vs_16": 0.826
2343
+ },
2344
+ "rec.1.41": {
2345
+ "top1_agree_model_skip16": 0.04,
2346
+ "top1_agree_model_skip4": 0.039,
2347
+ "top1_agree_model_skip0": 0.039,
2348
+ "top1_agree_model_logit_lens": 0.016,
2349
+ "nonsemantic_share_top10_skip16": 0.23,
2350
+ "nonsemantic_share_top10_skip0": 0.256,
2351
+ "top10_jaccard_skip0_vs_16": 0.765,
2352
+ "top10_jaccard_skip4_vs_16": 0.842
2353
+ },
2354
+ "rec.1.42": {
2355
+ "top1_agree_model_skip16": 0.067,
2356
+ "top1_agree_model_skip4": 0.062,
2357
+ "top1_agree_model_skip0": 0.063,
2358
+ "top1_agree_model_logit_lens": 0.025,
2359
+ "nonsemantic_share_top10_skip16": 0.163,
2360
+ "nonsemantic_share_top10_skip0": 0.174,
2361
+ "top10_jaccard_skip0_vs_16": 0.778,
2362
+ "top10_jaccard_skip4_vs_16": 0.858
2363
+ },
2364
+ "rec.1.43": {
2365
+ "top1_agree_model_skip16": 0.069,
2366
+ "top1_agree_model_skip4": 0.068,
2367
+ "top1_agree_model_skip0": 0.062,
2368
+ "top1_agree_model_logit_lens": 0.001,
2369
+ "nonsemantic_share_top10_skip16": 0.167,
2370
+ "nonsemantic_share_top10_skip0": 0.185,
2371
+ "top10_jaccard_skip0_vs_16": 0.775,
2372
+ "top10_jaccard_skip4_vs_16": 0.859
2373
+ },
2374
+ "rec.1.44": {
2375
+ "top1_agree_model_skip16": 0.075,
2376
+ "top1_agree_model_skip4": 0.07,
2377
+ "top1_agree_model_skip0": 0.069,
2378
+ "top1_agree_model_logit_lens": 0.039,
2379
+ "nonsemantic_share_top10_skip16": 0.167,
2380
+ "nonsemantic_share_top10_skip0": 0.183,
2381
+ "top10_jaccard_skip0_vs_16": 0.783,
2382
+ "top10_jaccard_skip4_vs_16": 0.853
2383
+ },
2384
+ "rec.1.45": {
2385
+ "top1_agree_model_skip16": 0.086,
2386
+ "top1_agree_model_skip4": 0.079,
2387
+ "top1_agree_model_skip0": 0.08,
2388
+ "top1_agree_model_logit_lens": 0.041,
2389
+ "nonsemantic_share_top10_skip16": 0.176,
2390
+ "nonsemantic_share_top10_skip0": 0.178,
2391
+ "top10_jaccard_skip0_vs_16": 0.788,
2392
+ "top10_jaccard_skip4_vs_16": 0.864
2393
+ },
2394
+ "rec.1.46": {
2395
+ "top1_agree_model_skip16": 0.09,
2396
+ "top1_agree_model_skip4": 0.093,
2397
+ "top1_agree_model_skip0": 0.09,
2398
+ "top1_agree_model_logit_lens": 0.045,
2399
+ "nonsemantic_share_top10_skip16": 0.14,
2400
+ "nonsemantic_share_top10_skip0": 0.161,
2401
+ "top10_jaccard_skip0_vs_16": 0.797,
2402
+ "top10_jaccard_skip4_vs_16": 0.86
2403
+ },
2404
+ "rec.1.47": {
2405
+ "top1_agree_model_skip16": 0.098,
2406
+ "top1_agree_model_skip4": 0.09,
2407
+ "top1_agree_model_skip0": 0.084,
2408
+ "top1_agree_model_logit_lens": 0.058,
2409
+ "nonsemantic_share_top10_skip16": 0.168,
2410
+ "nonsemantic_share_top10_skip0": 0.185,
2411
+ "top10_jaccard_skip0_vs_16": 0.814,
2412
+ "top10_jaccard_skip4_vs_16": 0.865
2413
+ },
2414
+ "rec.1.48": {
2415
+ "top1_agree_model_skip16": 0.109,
2416
+ "top1_agree_model_skip4": 0.102,
2417
+ "top1_agree_model_skip0": 0.1,
2418
+ "top1_agree_model_logit_lens": 0.061,
2419
+ "nonsemantic_share_top10_skip16": 0.161,
2420
+ "nonsemantic_share_top10_skip0": 0.196,
2421
+ "top10_jaccard_skip0_vs_16": 0.816,
2422
+ "top10_jaccard_skip4_vs_16": 0.867
2423
+ },
2424
+ "rec.1.49": {
2425
+ "top1_agree_model_skip16": 0.126,
2426
+ "top1_agree_model_skip4": 0.123,
2427
+ "top1_agree_model_skip0": 0.124,
2428
+ "top1_agree_model_logit_lens": 0.058,
2429
+ "nonsemantic_share_top10_skip16": 0.138,
2430
+ "nonsemantic_share_top10_skip0": 0.168,
2431
+ "top10_jaccard_skip0_vs_16": 0.818,
2432
+ "top10_jaccard_skip4_vs_16": 0.872
2433
+ },
2434
+ "rec.1.50": {
2435
+ "top1_agree_model_skip16": 0.176,
2436
+ "top1_agree_model_skip4": 0.17,
2437
+ "top1_agree_model_skip0": 0.165,
2438
+ "top1_agree_model_logit_lens": 0.088,
2439
+ "nonsemantic_share_top10_skip16": 0.151,
2440
+ "nonsemantic_share_top10_skip0": 0.18,
2441
+ "top10_jaccard_skip0_vs_16": 0.823,
2442
+ "top10_jaccard_skip4_vs_16": 0.874
2443
+ },
2444
+ "rec.1.51": {
2445
+ "top1_agree_model_skip16": 0.214,
2446
+ "top1_agree_model_skip4": 0.206,
2447
+ "top1_agree_model_skip0": 0.198,
2448
+ "top1_agree_model_logit_lens": 0.035,
2449
+ "nonsemantic_share_top10_skip16": 0.18,
2450
+ "nonsemantic_share_top10_skip0": 0.205,
2451
+ "top10_jaccard_skip0_vs_16": 0.82,
2452
+ "top10_jaccard_skip4_vs_16": 0.876
2453
+ },
2454
+ "rec.1.52": {
2455
+ "top1_agree_model_skip16": 0.25,
2456
+ "top1_agree_model_skip4": 0.246,
2457
+ "top1_agree_model_skip0": 0.223,
2458
+ "top1_agree_model_logit_lens": 0.106,
2459
+ "nonsemantic_share_top10_skip16": 0.181,
2460
+ "nonsemantic_share_top10_skip0": 0.209,
2461
+ "top10_jaccard_skip0_vs_16": 0.825,
2462
+ "top10_jaccard_skip4_vs_16": 0.885
2463
+ },
2464
+ "rec.1.53": {
2465
+ "top1_agree_model_skip16": 0.291,
2466
+ "top1_agree_model_skip4": 0.269,
2467
+ "top1_agree_model_skip0": 0.248,
2468
+ "top1_agree_model_logit_lens": 0.173,
2469
+ "nonsemantic_share_top10_skip16": 0.142,
2470
+ "nonsemantic_share_top10_skip0": 0.17,
2471
+ "top10_jaccard_skip0_vs_16": 0.757,
2472
+ "top10_jaccard_skip4_vs_16": 0.885
2473
+ },
2474
+ "rec.1.54": {
2475
+ "top1_agree_model_skip16": 0.32,
2476
+ "top1_agree_model_skip4": 0.31,
2477
+ "top1_agree_model_skip0": 0.304,
2478
+ "top1_agree_model_logit_lens": 0.154,
2479
+ "nonsemantic_share_top10_skip16": 0.165,
2480
+ "nonsemantic_share_top10_skip0": 0.184,
2481
+ "top10_jaccard_skip0_vs_16": 0.845,
2482
+ "top10_jaccard_skip4_vs_16": 0.909
2483
+ },
2484
+ "s.1": {
2485
+ "top1_agree_model_skip16": 0.337,
2486
+ "top1_agree_model_skip4": 0.334,
2487
+ "top1_agree_model_skip0": 0.329,
2488
+ "top1_agree_model_logit_lens": 0.0,
2489
+ "nonsemantic_share_top10_skip16": 0.139,
2490
+ "nonsemantic_share_top10_skip0": 0.153,
2491
+ "top10_jaccard_skip0_vs_16": 0.864,
2492
+ "top10_jaccard_skip4_vs_16": 0.922
2493
+ },
2494
+ "out.56": {
2495
+ "top1_agree_model_skip16": 0.325,
2496
+ "top1_agree_model_skip4": 0.319,
2497
+ "top1_agree_model_skip0": 0.309,
2498
+ "top1_agree_model_logit_lens": 0.211,
2499
+ "nonsemantic_share_top10_skip16": 0.126,
2500
+ "nonsemantic_share_top10_skip0": 0.14,
2501
+ "top10_jaccard_skip0_vs_16": 0.875,
2502
+ "top10_jaccard_skip4_vs_16": 0.919
2503
+ },
2504
+ "out.57": {
2505
+ "top1_agree_model_skip16": 0.357,
2506
+ "top1_agree_model_skip4": 0.352,
2507
+ "top1_agree_model_skip0": 0.34,
2508
+ "top1_agree_model_logit_lens": 0.246,
2509
+ "nonsemantic_share_top10_skip16": 0.108,
2510
+ "nonsemantic_share_top10_skip0": 0.12,
2511
+ "top10_jaccard_skip0_vs_16": 0.872,
2512
+ "top10_jaccard_skip4_vs_16": 0.918
2513
+ },
2514
+ "out.58": {
2515
+ "top1_agree_model_skip16": 0.364,
2516
+ "top1_agree_model_skip4": 0.35,
2517
+ "top1_agree_model_skip0": 0.338,
2518
+ "top1_agree_model_logit_lens": 0.242,
2519
+ "nonsemantic_share_top10_skip16": 0.113,
2520
+ "nonsemantic_share_top10_skip0": 0.125,
2521
+ "top10_jaccard_skip0_vs_16": 0.886,
2522
+ "top10_jaccard_skip4_vs_16": 0.929
2523
+ },
2524
+ "out.59": {
2525
+ "top1_agree_model_skip16": 0.375,
2526
+ "top1_agree_model_skip4": 0.365,
2527
+ "top1_agree_model_skip0": 0.347,
2528
+ "top1_agree_model_logit_lens": 0.269,
2529
+ "nonsemantic_share_top10_skip16": 0.103,
2530
+ "nonsemantic_share_top10_skip0": 0.113,
2531
+ "top10_jaccard_skip0_vs_16": 0.886,
2532
+ "top10_jaccard_skip4_vs_16": 0.932
2533
+ },
2534
+ "out.60": {
2535
+ "top1_agree_model_skip16": 0.383,
2536
+ "top1_agree_model_skip4": 0.384,
2537
+ "top1_agree_model_skip0": 0.374,
2538
+ "top1_agree_model_logit_lens": 0.273,
2539
+ "nonsemantic_share_top10_skip16": 0.106,
2540
+ "nonsemantic_share_top10_skip0": 0.117,
2541
+ "top10_jaccard_skip0_vs_16": 0.908,
2542
+ "top10_jaccard_skip4_vs_16": 0.949
2543
+ },
2544
+ "out.61": {
2545
+ "top1_agree_model_skip16": 0.386,
2546
+ "top1_agree_model_skip4": 0.383,
2547
+ "top1_agree_model_skip0": 0.374,
2548
+ "top1_agree_model_logit_lens": 0.206,
2549
+ "nonsemantic_share_top10_skip16": 0.104,
2550
+ "nonsemantic_share_top10_skip0": 0.117,
2551
+ "top10_jaccard_skip0_vs_16": 0.868,
2552
+ "top10_jaccard_skip4_vs_16": 0.949
2553
+ },
2554
+ "out.62": {
2555
+ "top1_agree_model_skip16": 0.391,
2556
+ "top1_agree_model_skip4": 0.391,
2557
+ "top1_agree_model_skip0": 0.393,
2558
+ "top1_agree_model_logit_lens": 0.311,
2559
+ "nonsemantic_share_top10_skip16": 0.104,
2560
+ "nonsemantic_share_top10_skip0": 0.112,
2561
+ "top10_jaccard_skip0_vs_16": 0.944,
2562
+ "top10_jaccard_skip4_vs_16": 0.967
2563
+ },
2564
+ "out.63": {
2565
+ "top1_agree_model_skip16": 0.424,
2566
+ "top1_agree_model_skip4": 0.42,
2567
+ "top1_agree_model_skip0": 0.42,
2568
+ "top1_agree_model_logit_lens": 0.353,
2569
+ "nonsemantic_share_top10_skip16": 0.093,
2570
+ "nonsemantic_share_top10_skip0": 0.101,
2571
+ "top10_jaccard_skip0_vs_16": 0.948,
2572
+ "top10_jaccard_skip4_vs_16": 0.964
2573
+ },
2574
+ "out.64": {
2575
+ "top1_agree_model_skip16": 0.439,
2576
+ "top1_agree_model_skip4": 0.436,
2577
+ "top1_agree_model_skip0": 0.436,
2578
+ "top1_agree_model_logit_lens": 0.364,
2579
+ "nonsemantic_share_top10_skip16": 0.098,
2580
+ "nonsemantic_share_top10_skip0": 0.104,
2581
+ "top10_jaccard_skip0_vs_16": 0.946,
2582
+ "top10_jaccard_skip4_vs_16": 0.967
2583
+ },
2584
+ "out.65": {
2585
+ "top1_agree_model_skip16": 0.459,
2586
+ "top1_agree_model_skip4": 0.453,
2587
+ "top1_agree_model_skip0": 0.448,
2588
+ "top1_agree_model_logit_lens": 0.373,
2589
+ "nonsemantic_share_top10_skip16": 0.095,
2590
+ "nonsemantic_share_top10_skip0": 0.102,
2591
+ "top10_jaccard_skip0_vs_16": 0.934,
2592
+ "top10_jaccard_skip4_vs_16": 0.969
2593
+ },
2594
+ "out.66": {
2595
+ "top1_agree_model_skip16": 0.465,
2596
+ "top1_agree_model_skip4": 0.465,
2597
+ "top1_agree_model_skip0": 0.459,
2598
+ "top1_agree_model_logit_lens": 0.43,
2599
+ "nonsemantic_share_top10_skip16": 0.089,
2600
+ "nonsemantic_share_top10_skip0": 0.095,
2601
+ "top10_jaccard_skip0_vs_16": 0.949,
2602
+ "top10_jaccard_skip4_vs_16": 0.967
2603
+ },
2604
+ "out.67": {
2605
+ "top1_agree_model_skip16": 0.464,
2606
+ "top1_agree_model_skip4": 0.461,
2607
+ "top1_agree_model_skip0": 0.459,
2608
+ "top1_agree_model_logit_lens": 0.142,
2609
+ "nonsemantic_share_top10_skip16": 0.081,
2610
+ "nonsemantic_share_top10_skip0": 0.09,
2611
+ "top10_jaccard_skip0_vs_16": 0.936,
2612
+ "top10_jaccard_skip4_vs_16": 0.966
2613
+ },
2614
+ "out.68": {
2615
+ "top1_agree_model_skip16": 0.5,
2616
+ "top1_agree_model_skip4": 0.499,
2617
+ "top1_agree_model_skip0": 0.498,
2618
+ "top1_agree_model_logit_lens": 0.45,
2619
+ "nonsemantic_share_top10_skip16": 0.084,
2620
+ "nonsemantic_share_top10_skip0": 0.088,
2621
+ "top10_jaccard_skip0_vs_16": 0.964,
2622
+ "top10_jaccard_skip4_vs_16": 0.98
2623
+ },
2624
+ "out.69": {
2625
+ "top1_agree_model_skip16": 0.551,
2626
+ "top1_agree_model_skip4": 0.552,
2627
+ "top1_agree_model_skip0": 0.548,
2628
+ "top1_agree_model_logit_lens": 0.176,
2629
+ "nonsemantic_share_top10_skip16": 0.079,
2630
+ "nonsemantic_share_top10_skip0": 0.083,
2631
+ "top10_jaccard_skip0_vs_16": 0.946,
2632
+ "top10_jaccard_skip4_vs_16": 0.976
2633
+ },
2634
+ "out.70": {
2635
+ "top1_agree_model_skip16": 0.464,
2636
+ "top1_agree_model_skip4": 0.462,
2637
+ "top1_agree_model_skip0": 0.461,
2638
+ "top1_agree_model_logit_lens": 0.594,
2639
+ "nonsemantic_share_top10_skip16": 0.076,
2640
+ "nonsemantic_share_top10_skip0": 0.078,
2641
+ "top10_jaccard_skip0_vs_16": 0.976,
2642
+ "top10_jaccard_skip4_vs_16": 0.985
2643
+ }
2644
+ },
2645
+ "lens_bytes": 1914412664
2646
+ }
fit_records/k1_n50_skip16_s0/prompts.json ADDED
@@ -0,0 +1,202 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [
2
+ {
3
+ "i": 0,
4
+ "start": "Given the chaos , the governor of Yunnan ignored the imperial orders of evacuation . The Mongol army commanded by Prince Ye @"
5
+ },
6
+ {
7
+ "i": 1,
8
+ "start": "From the 1920s Australia 's defence thinking was dominated by British Imperial defence policy , which was embodied"
9
+ },
10
+ {
11
+ "i": 2,
12
+ "start": "Tim 's driving days started as a toddler when he was given a go @-@ kart that he often drove"
13
+ },
14
+ {
15
+ "i": 3,
16
+ "start": "As the Marines formed into a horseshoe @-@ shaped line around Hill 123 , Tamura 's"
17
+ },
18
+ {
19
+ "i": 4,
20
+ "start": "USA Today 's Scott Bowles awarded the film three out of four stars , labeling it a \" surprising comedy that rivals"
21
+ },
22
+ {
23
+ "i": 5,
24
+ "start": "Ringo Starr identified his initial inspiration for \" Back Off Boogaloo \" as having come from Marc Bolan ,"
25
+ },
26
+ {
27
+ "i": 6,
28
+ "start": "The ship 's secondary armament consisted of fourteen 45 @-@ calibre 6 @-@ inch ("
29
+ },
30
+ {
31
+ "i": 7,
32
+ "start": "Sources disagree about Radcliffe 's personal wealth ; he was reported to have earned \u00a3 1 million for the first Harry"
33
+ },
34
+ {
35
+ "i": 8,
36
+ "start": "In April 2010 , there were 6 @,@ 260 people employed in the healthcare field"
37
+ },
38
+ {
39
+ "i": 9,
40
+ "start": "\" Exit Through the Kwik @-@ E @-@ Mart \" is the fifteenth episode of the twenty @-"
41
+ },
42
+ {
43
+ "i": 10,
44
+ "start": "On 25 June 1950 , the North Korean Army ( KPA ) crossed the border into South"
45
+ },
46
+ {
47
+ "i": 11,
48
+ "start": "A British parliamentary motion put forward by Labour MP Kate Hoey in February 2013 to retrospectively condemn the"
49
+ },
50
+ {
51
+ "i": 12,
52
+ "start": "At the fort , Moultrie ordered his men to concentrate their fire on the two large man @-@ of @"
53
+ },
54
+ {
55
+ "i": 13,
56
+ "start": "In Part II of the series , Shikamaru is assigned the task of locating two members of the criminal organization Ak"
57
+ },
58
+ {
59
+ "i": 14,
60
+ "start": "The Ming dynasty had seen considerable advancement in the process of colour printing in China . At his studio , Hu Zhengyan experiment"
61
+ },
62
+ {
63
+ "i": 15,
64
+ "start": "\" Exit Through the Kwik @-@ E @-@ Mart \" was written by Marc Wilmore and directed by"
65
+ },
66
+ {
67
+ "i": 16,
68
+ "start": "In 629 Tikal founded Dos Pilas , some 110 kilometres ( 68 mi )"
69
+ },
70
+ {
71
+ "i": 17,
72
+ "start": "The British fleet weighed anchor at Cape Fear on May 31 , and arrived outside Charleston Harbor the next day . Moul"
73
+ },
74
+ {
75
+ "i": 18,
76
+ "start": "The initial phase saw the Australians advance towards the Hongorai River . Following the end of the early fighting , the Australian"
77
+ },
78
+ {
79
+ "i": 19,
80
+ "start": "The reactions of the native Aboriginal inhabitants to the sudden arrival of British settlers in Australia were varied , but were inevitably hostile when"
81
+ },
82
+ {
83
+ "i": 20,
84
+ "start": "The German offensive ground to a halt in mid @-@ July and a brief lull followed , during which the Australians"
85
+ },
86
+ {
87
+ "i": 21,
88
+ "start": "The city 's growth has reflected the push and pull of many social and economic factors . The total population increased in each"
89
+ },
90
+ {
91
+ "i": 22,
92
+ "start": "Following the classification of Tropical Depression Two @-@ E on June 19 , the Government of Mexico issued a tropical"
93
+ },
94
+ {
95
+ "i": 23,
96
+ "start": "The Natives remained in London following the England international . They defeated London Welsh on 18 February , before losing first"
97
+ },
98
+ {
99
+ "i": 24,
100
+ "start": "In February 1997 , a United States immigration court judge granted asylum to a German Scientologist who claimed she"
101
+ },
102
+ {
103
+ "i": 25,
104
+ "start": "Some of the most important trade routes began at Canton and Macau in Qing Dynasty China . Early in each year a large convoy"
105
+ },
106
+ {
107
+ "i": 26,
108
+ "start": "The fourth installment in the Slammiversary name was announced in January 2008 to take place on June"
109
+ },
110
+ {
111
+ "i": 27,
112
+ "start": "Capel Lligwy has national recognition and statutory protection from alteration as it has been designated as a Grade II listed building"
113
+ },
114
+ {
115
+ "i": 28,
116
+ "start": "The Trout Creek Mountains are semi @-@ arid because they are in the eastern rain shadow of mountain ranges to"
117
+ },
118
+ {
119
+ "i": 29,
120
+ "start": "Asahi , meaning \" rising sun \" , a poetic name for Japan from a stanza of waka poetry , was laid"
121
+ },
122
+ {
123
+ "i": 30,
124
+ "start": "Beckham 's term as governor ended December 10 , 1907 . In January 19"
125
+ },
126
+ {
127
+ "i": 31,
128
+ "start": "Donald Sutherland portrays Jack Pellitt , Bobby 's father and Kurt 's boss . On July 27 ,"
129
+ },
130
+ {
131
+ "i": 32,
132
+ "start": "As British offshoots , the Australian colonies participated in Britain 's small wars of the 19th century"
133
+ },
134
+ {
135
+ "i": 33,
136
+ "start": "Retail is another major employer in the county , with 5 @,@ 280 people employed in April "
137
+ },
138
+ {
139
+ "i": 34,
140
+ "start": "The tour returned to Pocono a month later , and Richmond battled for another victory in a fog @-@ shortened"
141
+ },
142
+ {
143
+ "i": 35,
144
+ "start": "\" ... Thirteen Years Later \" received mixed reviews from critics . The A.V. Club 's Todd VanDerWer"
145
+ },
146
+ {
147
+ "i": 36,
148
+ "start": "Bonnie Stiernberg , for Paste , was more positive , writing that she preferred the use of comedy , following the"
149
+ },
150
+ {
151
+ "i": 37,
152
+ "start": "In common with many small Scottish islands , Papa Stour 's population peaked in the 19th century and has"
153
+ },
154
+ {
155
+ "i": 38,
156
+ "start": "The unexpected deployment to East Timor in 1999 led to significant changes in Australian defence policy and to an enhancement"
157
+ },
158
+ {
159
+ "i": 39,
160
+ "start": "Government publications on the dangers of sects increased between 1996 and 1998 , and"
161
+ },
162
+ {
163
+ "i": 40,
164
+ "start": "Richmond started 1982 without a ride before getting a one @-@ race deal to drive for Billie"
165
+ },
166
+ {
167
+ "i": 41,
168
+ "start": "The music for Sonic the Hedgehog was composed and produced by Masato Nakamura , bassist and lead songwriter of the"
169
+ },
170
+ {
171
+ "i": 42,
172
+ "start": "Apple Records issued the single on 17 March 1972 in Britain , as Apple R 59"
173
+ },
174
+ {
175
+ "i": 43,
176
+ "start": "Meridian is home to two post @-@ secondary educational institutions . Meridian Community College , founded in 19"
177
+ },
178
+ {
179
+ "i": 44,
180
+ "start": "Ayola began her career in the theatre , performing in a number of Shakespearean plays including Twelfth Night , A M"
181
+ },
182
+ {
183
+ "i": 45,
184
+ "start": "Cooke 's opportunistic diversion had determined that the Spanish forces in the Philippines posed no immediate threat , although the mission"
185
+ },
186
+ {
187
+ "i": 46,
188
+ "start": "To further justify its stance , the German government has also pointed to the long history of U.S. court cases involving Scient"
189
+ },
190
+ {
191
+ "i": 47,
192
+ "start": "Headed by police chief Lee Shelbourn since 2009 , the Meridian Police Department consists of "
193
+ },
194
+ {
195
+ "i": 48,
196
+ "start": "A port , Sonic the Hedgehog Genesis , was released for the Game Boy Advance ( GBA ) on November"
197
+ },
198
+ {
199
+ "i": 49,
200
+ "start": "As is often the case with huge ancient ruins , knowledge of the site was never completely lost in the region . It seems"
201
+ }
202
+ ]
fit_records/k1_n50_skip16_s0/readouts.txt ADDED
@@ -0,0 +1,98 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ non-semantic tokens (no letter or digit): 5208 of 131072
2
+
3
+ EARLY POSITIONS. Jacobian size by source position, relative to the median over positions 16+ (k=1, 50 prompts)
4
+ node p0 p1 p2 p3 p4 p5 p6 p7 p8-15 mean
5
+ emb 6.14 6.33 5.20 4.56 8.18 4.69 3.83 3.52 3.53
6
+ in.04 2.74 4.15 3.66 3.33 6.20 3.50 3.51 3.33 3.24
7
+ in.12 3.07 3.08 3.47 3.23 4.58 3.02 3.06 3.46 2.91
8
+ x 1.24 1.17 1.52 1.50 1.65 1.60 1.83 2.16 1.99
9
+ u.1 1.24 1.17 1.52 1.50 1.65 1.60 1.83 2.16 1.99
10
+ rec.1.20 4.16 2.00 2.05 2.17 2.51 2.03 2.38 2.52 2.27
11
+ rec.1.28 5.29 2.02 1.89 1.93 2.04 1.91 2.12 2.25 2.03
12
+ rec.1.36 4.39 1.99 1.99 1.95 1.86 1.91 1.94 2.15 1.93
13
+ rec.1.44 3.67 1.67 1.60 1.75 1.55 1.71 1.67 2.00 1.71
14
+ rec.1.52 3.26 1.69 1.63 1.59 1.61 1.72 1.60 2.02 1.66
15
+ s.1 3.04 1.55 1.48 1.43 1.47 1.64 1.45 1.60 1.48
16
+ out.60 2.63 1.35 1.30 1.36 1.30 1.38 1.27 1.41 1.33
17
+ out.68 1.52 0.93 0.94 1.03 0.94 1.00 1.00 0.98 1.01
18
+
19
+ LENS VARIANTS (skip the first 0 / 4 / 16 source positions) and held-out readouts, positions 16+ of 8 held-out paragraphs
20
+ node |J| split-half cos 0v16 cos 4v16 top10 Jacc 0v16 4v16 agree model: skip0 skip4 skip16 logit lens junk top10 skip0 skip16
21
+ emb 1.9 0.611 0.899 0.933 0.37 0.44 0.00 0.00 0.00 0.00 0.64 0.73
22
+ in.04 3.4 0.759 0.951 0.966 0.51 0.58 0.00 0.00 0.00 0.00 0.68 0.74
23
+ in.12 4.1 0.859 0.969 0.982 0.56 0.64 0.00 0.00 0.00 0.00 0.60 0.67
24
+ x 8.3 0.780 0.975 0.983 0.63 0.67 0.01 0.01 0.02 0.00 0.43 0.40
25
+ u.1 8.3 0.780 0.975 0.983 0.65 0.70 0.01 0.01 0.02 0.00 0.47 0.45
26
+ rec.1.20 5.8 0.932 0.980 0.993 0.63 0.74 0.02 0.02 0.03 0.00 0.27 0.24
27
+ rec.1.28 12.1 0.959 0.984 0.995 0.60 0.77 0.02 0.02 0.03 0.00 0.32 0.26
28
+ rec.1.36 10.1 0.970 0.990 0.996 0.69 0.80 0.03 0.03 0.03 0.01 0.27 0.24
29
+ rec.1.44 12.9 0.981 0.993 0.998 0.78 0.85 0.07 0.07 0.07 0.04 0.18 0.17
30
+ rec.1.52 21.7 0.989 0.996 0.999 0.82 0.89 0.22 0.25 0.25 0.11 0.21 0.18
31
+ s.1 173.9 0.992 0.997 0.999 0.86 0.92 0.33 0.33 0.34 0.00 0.15 0.14
32
+ out.60 40.3 0.995 0.999 1.000 0.91 0.95 0.37 0.38 0.38 0.27 0.12 0.11
33
+ out.68 46.5 0.999 1.000 1.000 0.96 0.98 0.50 0.50 0.50 0.45 0.09 0.08
34
+
35
+ PROMPT 'Fact: The currency used in the country shaped like a boot is' intermediate: Italy model top-3: [' the', ' called', ' not']
36
+ in.06 | skip16: rank(Italy)=5831 [' “', '.', '.[', ' „', '“', '...'] | logit lens: ['nt', ' “', ' par', ' st']
37
+ in.14 | skip16: rank(Italy)=3012 ['—a', '—that', '—the', '.', '—but', '—'] | logit lens: [' both', ' tongue', 'olar', ':']
38
+ x | skip16: rank(Italy)=3723 ['.', ' "[', '?', '.[', ':', '...'] | logit lens: ['raux', 'reros', 'acca', 'acco']
39
+ rec.1.22 | skip16: rank(Italy)=368 ['...', '...\n\n', '?', '..."', '?\n\n', '..."\n\n'] | logit lens: ['ähler', 'untz', ' الإع', 'oux']
40
+ rec.1.30 | skip16: rank(Italy)=313 ['?', '?"', '?"\n\n', '?\n\n', '"?', '?:'] | logit lens: [' currently', 'start', 'elier', 'èque']
41
+ rec.1.38 | skip16: rank(Italy)=194 ['?\n', '?\n\n', '?', '?"\n', '...\n', '?"\n\n'] | logit lens: ['ouer', 'ourdes', 'nam', 'inato']
42
+ rec.1.46 | skip16: rank(Italy)=137 ['?\n', ' currency', '?\n\n', ' Currency', '?', ' coins'] | logit lens: ['erus', ' called', 'called', ' actually']
43
+ rec.1.54 | skip16: rank(Italy)=281 [' called', ' known', ' named', ' actually', ' currency', ' the'] | logit lens: [' called', 'called', ' actually', 'currency']
44
+ s.1 | skip16: rank(Italy)=535 [' called', ' currency', ' known', ' named', ' actually', ' likely'] | logit lens: ['ふっ', 'くっ', 'りば', 'ふん']
45
+ out.62 | skip16: rank(Italy)=2003 [' called', ' currency', ' USD', ' coins', ' Bitcoin', ' currencies'] | logit lens: [' called', 'الين', ' actually', ' typically']
46
+ out.70 | skip16: rank(Italy)=1754 [' the', ' called', ' actually', ' peg', ' USD', ' Sey'] | logit lens: [' the', ' called', ' not', ' actually']
47
+
48
+ PROMPT 'Fact: The number of legs on the animal that spins webs is' intermediate: spider model top-3: [' ', ' not', ' eight']
49
+ in.06 | skip16: rank(spider)=81928 ['.', ' “', ';', ' „', '--', ':'] | logit lens: [' “', 'de', 'nt', ' Morse']
50
+ in.14 | skip16: rank(spider)=93791 ['—.', '—but', '—that', '.–', '—a', '—which'] | logit lens: ['pon', ' tongue', '商务', 'iste']
51
+ x | skip16: rank(spider)=12595 ['.', ':', ' "[', '.[', '…\n\n', ' […'] | logit lens: ['raux', 'acca', 'reros', ' herself']
52
+ rec.1.22 | skip16: rank(spider)=115 ['...\n\n', '...', '..."\n\n', '..."', '?\n\n', '…\n\n'] | logit lens: ['UUID', 'ارضة', 'elier', 'ouer']
53
+ rec.1.30 | skip16: rank(spider)=156 ['?\n\n', '?"\n\n', ' weird', '?**\n\n', '?', ' animals'] | logit lens: ['ous', 'ождения', 'atang', ' fewer']
54
+ rec.1.38 | skip16: rank(spider)=525 ['?\n\n', '?\n', '...\n\n', '...\n', '?', '?”\n\n'] | logit lens: ['ouer', 'orat', 'ieu', 'orp']
55
+ rec.1.46 | skip16: rank(spider)=616 ['?\n', '?\n\n', '?', ':\n', '?"\n', '?”'] | logit lens: ['ouden', 'óns', '一动', 'usqu']
56
+ rec.1.54 | skip16: rank(spider)=966 [' unknown', '?\n', ' zero', ' irrelevant', ' none', '?\n\n'] | logit lens: ['無断', '/ajax', '一动', 'ouer']
57
+ s.1 | skip16: rank(spider)=920 ['?\n', ' unknown', ' irrelevant', ' zero', ' likely', ' none'] | logit lens: ['ふっ', 'くっ', 'りば', 'ふん']
58
+ out.62 | skip16: rank(spider)=30 ['?\n', ' incorrect', ' irrelevant', ' unknown', ' zero', ' equal'] | logit lens: [' always', ' typically', ' actually', ' incorrect']
59
+ out.70 | skip16: rank(spider)=55 [' eight', ' always', ' irrelevant', ' zero', ' actually', ' unknown'] | logit lens: [' ', ' **', ' eight', ' not']
60
+
61
+ PROMPT 'The capital of the country where sushi originated is' intermediate: Japan model top-3: [' Tokyo', ' _', ' ___']
62
+ in.06 | skip16: rank(Japan)=63757 [' ...', '.', ' ...\n\n', ' **', '...\n\n', ' ...\n'] | logit lens: [' Moran', 'elha', ' quot', 'dur']
63
+ in.14 | skip16: rank(Japan)=71880 ['...\n', '**.', '—.', '.—', '....\n', '</think>'] | logit lens: ['nil', ' nil', ' par', 'feld']
64
+ x | skip16: rank(Japan)=24402 ['.', '...', ' ...', ':', '..."', '".'] | logit lens: ['raux', 'acca', 'acco', 'avar']
65
+ rec.1.22 | skip16: rank(Japan)=5489 [' city', ' cities', ' [...]', ' City', 'city', ' **['] | logit lens: ['ähler', 'oje', ' Tort', '�']
66
+ rec.1.30 | skip16: rank(Japan)=3318 [' city', ' cities', ' **', 'city', ' City', '?'] | logit lens: ['uny', 'oum', 'inan', 'ikian']
67
+ rec.1.38 | skip16: rank(Japan)=1078 [':\n', ' **', '...\n', ':**', '?\n', '?**'] | logit lens: ['uny', 'ouer', 'ambo', 'auro']
68
+ rec.1.46 | skip16: rank(Japan)=4907 ['?\n', ':\n', ' ?\n', '?"\n', '?**', '?'] | logit lens: [' answer', 'answer', ' Answer', 'Answer']
69
+ rec.1.54 | skip16: rank(Japan)=487 [':\n', '?\n', ' ___', ' _', '...\n', ' City'] | logit lens: [' _', ' retra', ' what', ' called']
70
+ s.1 | skip16: rank(Japan)=670 [':\n', ' ___', ' _', '?\n', '____', ' (_'] | logit lens: ['くっ', 'ふっ', 'りば', 'ふん']
71
+ out.62 | skip16: rank(Japan)=3 [' Tokyo', 'Tokyo', ' Japan', ' Osaka', ' Kyoto', ' Jakarta'] | logit lens: [' Tokyo', ' Japan', ' ___', 'urgo']
72
+ out.70 | skip16: rank(Japan)=9 [' Tokyo', ' ___', ':\n', ' _', ' Edo', ' Kyoto'] | logit lens: [' Tokyo', ' _', ' ___', ':\n']
73
+
74
+ PROMPT 'Fact: The color of the planet fourth from the Sun is' intermediate: Mars model top-3: [' orange', ' not', ' blue']
75
+ in.06 | skip16: rank(Mars)=368 ['.', ' “', '...', ';', ':', ' **'] | logit lens: [' dur', ' st', 'idus', ' Wend']
76
+ in.14 | skip16: rank(Mars)=34 ['.', '?', ' "...', '?.', '.".', '?:'] | logit lens: [' again', ' pall', ' po', ' captured']
77
+ x | skip16: rank(Mars)=341 ['.', ' "...', ';', ':', '...', '?'] | logit lens: ['raux', 'acca', 'reros', ' herself']
78
+ rec.1.22 | skip16: rank(Mars)=48 ['...', '..."', '?', ' Earth', ' colors', ' colored'] | logit lens: ['मान', 'chev', ' now', ' structure']
79
+ rec.1.30 | skip16: rank(Mars)=121 ['?', '?"', '?\n\n', '?"\n\n', '"?', '?**\n\n'] | logit lens: ['نج', 'atang', 'uny', 'start']
80
+ rec.1.38 | skip16: rank(Mars)=76 ['?\n', '?\n\n', '..."', '?', '?"', '...\n'] | logit lens: ['ouer', 'ikian', 'ourde', 'oué']
81
+ rec.1.46 | skip16: rank(Mars)=49 ['?\n', '?\n\n', ' named', ' name', '?', ' colors'] | logit lens: [' named', 'ieras', 'avas', ' name']
82
+ rec.1.54 | skip16: rank(Mars)=72 [' named', ' unknown', ' green', '?\n\n', ' unnamed', '?\n'] | logit lens: [' not', '아버', ' actually', ' color']
83
+ s.1 | skip16: rank(Mars)=23 [' unnamed', ' named', ' unknown', ' ___', '?\n', ' green'] | logit lens: ['ふっ', 'くっ', 'りば', 'ふん']
84
+ out.62 | skip16: rank(Mars)=24 [' yellow', ' orange', ' brown', ' green', ' redd', ' gold'] | logit lens: [' orange', ' brown', ' yellow', ' redd']
85
+ out.70 | skip16: rank(Mars)=10 [' redd', ' orange', ' yellow', ' brown', ' Mercury', ' Venus'] | logit lens: [' orange', ' blue', ' redd', ' yellow']
86
+
87
+ PROMPT 'es 23? Answer briefly.<|im_end|>\n<|im_start|>assistant\n<think></think>' intermediate: None model top-3: ['4', '5', '1']
88
+ in.06 | skip16: [' *“', ' **', ' “', ' **[', ' ``', ' **"'] | logit lens: [' Rok', 'enia', 'چه', ' DEN']
89
+ in.14 | skip16: [' “', ' *“', ' **[', ' **', ' **"', ' [**'] | logit lens: ['stadt', 'tn', 'umont', 'oty']
90
+ x | skip16: [' “', ' *“', ' **[', ' ``', ' (“', ' *„'] | logit lens: ['inos', '्त', 'onis', ' gola']
91
+ rec.1.22 | skip16: [' **', ' **[', ' **"', ' *“', ' (**', ' **('] | logit lens: ['tha', '�', 'onis', 'kis']
92
+ rec.1.30 | skip16: [' **', ' **"', ' **[', ' **(', ' (**', ' *“'] | logit lens: ['atang', 'ocke', 'one', 'antan']
93
+ rec.1.38 | skip16: [' **', ' **[', ' **"', ' **(', '\u202f', ' “'] | logit lens: ['uad', '/checkout', 'ikian', 'ourdes']
94
+ rec.1.46 | skip16: [' ', ' answer', ' Answer', ' approximately', ' “', ' Approximately'] | logit lens: ['uad', ' contex', 'endit', ' Córd']
95
+ rec.1.54 | skip16: [' ', '4', '5', '3', '6', '7'] | logit lens: ['disf', 'anse', '無断', ' 結果']
96
+ s.1 | skip16: [' ', '4', '5', '3', '1', '6'] | logit lens: ['ふっ', 'くっ', 'りば', 'ふん']
97
+ out.62 | skip16: ['4', ' ', '3', '5', '1', ' forty'] | logit lens: ['onin', '4', 'akume', 'elis']
98
+ out.70 | skip16: ['4', '5', '1', '3', 'The', '2'] | logit lens: ['4', '5', '3', '1']
fit_records/k1_n50_skip16_s0/run.json ADDED
@@ -0,0 +1,25 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "started": "2026-11-15T23:56:59+0000",
3
+ "finished": "2026-11-16T00:18:29+0000",
4
+ "packages": {
5
+ "torch": "2.14.0+cu126",
6
+ "transformers": "5.17.0",
7
+ "peft": "0.21.0",
8
+ "numpy": "2.4.6",
9
+ "safetensors": "0.8.0"
10
+ },
11
+ "cuda_visible_devices": "1",
12
+ "config": {
13
+ "k": 1,
14
+ "n_prompts": 50,
15
+ "seq_len": 128,
16
+ "chunk": 256,
17
+ "skip": 16,
18
+ "seed": 0,
19
+ "run": "k1_n50_skip16_s0",
20
+ "pilot": false,
21
+ "tf32": true,
22
+ "dtype": "float32",
23
+ "corpus": "WikiText-103 raw, validation split (Salesforce/wikitext)"
24
+ }
25
+ }
fit_records/k2_n50_skip16_s0/metrics.json ADDED
@@ -0,0 +1,4122 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "n_prompts": 50,
3
+ "n_nodes": 114,
4
+ "seconds_per_prompt": 38.76371139526367,
5
+ "split_half_cosine": {
6
+ "emb": 0.3835,
7
+ "in.00": 0.4085,
8
+ "in.01": 0.5089,
9
+ "in.02": 0.5011,
10
+ "in.03": 0.4883,
11
+ "in.04": 0.5076,
12
+ "in.05": 0.5167,
13
+ "in.06": 0.5228,
14
+ "in.07": 0.5372,
15
+ "in.08": 0.5943,
16
+ "in.09": 0.6049,
17
+ "in.10": 0.5999,
18
+ "in.11": 0.5959,
19
+ "in.12": 0.5861,
20
+ "in.13": 0.5641,
21
+ "in.14": 0.5571,
22
+ "x": 0.3266,
23
+ "u.1": 0.0226,
24
+ "rec.1.16": 0.0632,
25
+ "rec.1.17": 0.0763,
26
+ "rec.1.18": 0.0779,
27
+ "rec.1.19": 0.0853,
28
+ "rec.1.20": 0.0846,
29
+ "rec.1.21": 0.0911,
30
+ "rec.1.22": 0.1659,
31
+ "rec.1.23": 0.1512,
32
+ "rec.1.24": 0.1674,
33
+ "rec.1.25": 0.1538,
34
+ "rec.1.26": 0.1693,
35
+ "rec.1.27": 0.1747,
36
+ "rec.1.28": 0.1881,
37
+ "rec.1.29": 0.1862,
38
+ "rec.1.30": 0.1997,
39
+ "rec.1.31": 0.1856,
40
+ "rec.1.32": 0.2364,
41
+ "rec.1.33": 0.2401,
42
+ "rec.1.34": 0.2882,
43
+ "rec.1.35": 0.2881,
44
+ "rec.1.36": 0.3116,
45
+ "rec.1.37": 0.3094,
46
+ "rec.1.38": 0.3467,
47
+ "rec.1.39": 0.3854,
48
+ "rec.1.40": 0.3712,
49
+ "rec.1.41": 0.3868,
50
+ "rec.1.42": 0.3935,
51
+ "rec.1.43": 0.4113,
52
+ "rec.1.44": 0.419,
53
+ "rec.1.45": 0.4327,
54
+ "rec.1.46": 0.5218,
55
+ "rec.1.47": 0.5908,
56
+ "rec.1.48": 0.6889,
57
+ "rec.1.49": 0.7611,
58
+ "rec.1.50": 0.7794,
59
+ "rec.1.51": 0.8101,
60
+ "rec.1.52": 0.8328,
61
+ "rec.1.53": 0.869,
62
+ "rec.1.54": 0.8691,
63
+ "s.1": 0.8763,
64
+ "u.2": 0.8811,
65
+ "rec.2.16": 0.9296,
66
+ "rec.2.17": 0.9424,
67
+ "rec.2.18": 0.9446,
68
+ "rec.2.19": 0.9488,
69
+ "rec.2.20": 0.9485,
70
+ "rec.2.21": 0.9519,
71
+ "rec.2.22": 0.967,
72
+ "rec.2.23": 0.9665,
73
+ "rec.2.24": 0.9662,
74
+ "rec.2.25": 0.9656,
75
+ "rec.2.26": 0.965,
76
+ "rec.2.27": 0.9654,
77
+ "rec.2.28": 0.9649,
78
+ "rec.2.29": 0.9658,
79
+ "rec.2.30": 0.9671,
80
+ "rec.2.31": 0.9674,
81
+ "rec.2.32": 0.9673,
82
+ "rec.2.33": 0.9678,
83
+ "rec.2.34": 0.9686,
84
+ "rec.2.35": 0.9687,
85
+ "rec.2.36": 0.9725,
86
+ "rec.2.37": 0.9734,
87
+ "rec.2.38": 0.9747,
88
+ "rec.2.39": 0.976,
89
+ "rec.2.40": 0.9774,
90
+ "rec.2.41": 0.9793,
91
+ "rec.2.42": 0.9796,
92
+ "rec.2.43": 0.9824,
93
+ "rec.2.44": 0.982,
94
+ "rec.2.45": 0.984,
95
+ "rec.2.46": 0.9845,
96
+ "rec.2.47": 0.9859,
97
+ "rec.2.48": 0.9858,
98
+ "rec.2.49": 0.9874,
99
+ "rec.2.50": 0.9875,
100
+ "rec.2.51": 0.9887,
101
+ "rec.2.52": 0.9899,
102
+ "rec.2.53": 0.9909,
103
+ "rec.2.54": 0.9914,
104
+ "s.2": 0.9925,
105
+ "out.56": 0.9926,
106
+ "out.57": 0.9936,
107
+ "out.58": 0.9939,
108
+ "out.59": 0.995,
109
+ "out.60": 0.9953,
110
+ "out.61": 0.9959,
111
+ "out.62": 0.9967,
112
+ "out.63": 0.9975,
113
+ "out.64": 0.9976,
114
+ "out.65": 0.9983,
115
+ "out.66": 0.9985,
116
+ "out.67": 0.9991,
117
+ "out.68": 0.9993,
118
+ "out.69": 0.9996,
119
+ "out.70": 0.9997
120
+ },
121
+ "cosine_skip0_vs_skip16": {
122
+ "emb": 0.834,
123
+ "in.00": 0.8524,
124
+ "in.01": 0.8928,
125
+ "in.02": 0.8964,
126
+ "in.03": 0.9002,
127
+ "in.04": 0.9057,
128
+ "in.05": 0.9045,
129
+ "in.06": 0.9072,
130
+ "in.07": 0.9163,
131
+ "in.08": 0.9258,
132
+ "in.09": 0.9369,
133
+ "in.10": 0.9332,
134
+ "in.11": 0.9383,
135
+ "in.12": 0.9348,
136
+ "in.13": 0.9287,
137
+ "in.14": 0.933,
138
+ "x": 0.967,
139
+ "u.1": 0.9652,
140
+ "rec.1.16": 0.8649,
141
+ "rec.1.17": 0.8714,
142
+ "rec.1.18": 0.8678,
143
+ "rec.1.19": 0.8601,
144
+ "rec.1.20": 0.859,
145
+ "rec.1.21": 0.8593,
146
+ "rec.1.22": 0.8876,
147
+ "rec.1.23": 0.9068,
148
+ "rec.1.24": 0.9149,
149
+ "rec.1.25": 0.9206,
150
+ "rec.1.26": 0.9195,
151
+ "rec.1.27": 0.9228,
152
+ "rec.1.28": 0.9238,
153
+ "rec.1.29": 0.9352,
154
+ "rec.1.30": 0.9475,
155
+ "rec.1.31": 0.9493,
156
+ "rec.1.32": 0.9545,
157
+ "rec.1.33": 0.9566,
158
+ "rec.1.34": 0.9636,
159
+ "rec.1.35": 0.9614,
160
+ "rec.1.36": 0.9627,
161
+ "rec.1.37": 0.9555,
162
+ "rec.1.38": 0.9641,
163
+ "rec.1.39": 0.9563,
164
+ "rec.1.40": 0.967,
165
+ "rec.1.41": 0.9638,
166
+ "rec.1.42": 0.9662,
167
+ "rec.1.43": 0.9608,
168
+ "rec.1.44": 0.9654,
169
+ "rec.1.45": 0.9626,
170
+ "rec.1.46": 0.9721,
171
+ "rec.1.47": 0.9734,
172
+ "rec.1.48": 0.975,
173
+ "rec.1.49": 0.9745,
174
+ "rec.1.50": 0.978,
175
+ "rec.1.51": 0.9784,
176
+ "rec.1.52": 0.9791,
177
+ "rec.1.53": 0.9696,
178
+ "rec.1.54": 0.9802,
179
+ "s.1": 0.9822,
180
+ "u.2": 0.9818,
181
+ "rec.2.16": 0.9816,
182
+ "rec.2.17": 0.9818,
183
+ "rec.2.18": 0.9827,
184
+ "rec.2.19": 0.9791,
185
+ "rec.2.20": 0.9823,
186
+ "rec.2.21": 0.9821,
187
+ "rec.2.22": 0.9832,
188
+ "rec.2.23": 0.9811,
189
+ "rec.2.24": 0.9837,
190
+ "rec.2.25": 0.9852,
191
+ "rec.2.26": 0.9837,
192
+ "rec.2.27": 0.9846,
193
+ "rec.2.28": 0.9851,
194
+ "rec.2.29": 0.9846,
195
+ "rec.2.30": 0.986,
196
+ "rec.2.31": 0.9891,
197
+ "rec.2.32": 0.9867,
198
+ "rec.2.33": 0.9896,
199
+ "rec.2.34": 0.9894,
200
+ "rec.2.35": 0.99,
201
+ "rec.2.36": 0.9902,
202
+ "rec.2.37": 0.989,
203
+ "rec.2.38": 0.9908,
204
+ "rec.2.39": 0.9903,
205
+ "rec.2.40": 0.9919,
206
+ "rec.2.41": 0.9924,
207
+ "rec.2.42": 0.9929,
208
+ "rec.2.43": 0.9925,
209
+ "rec.2.44": 0.9929,
210
+ "rec.2.45": 0.9931,
211
+ "rec.2.46": 0.9932,
212
+ "rec.2.47": 0.9942,
213
+ "rec.2.48": 0.9955,
214
+ "rec.2.49": 0.9958,
215
+ "rec.2.50": 0.9962,
216
+ "rec.2.51": 0.9962,
217
+ "rec.2.52": 0.9962,
218
+ "rec.2.53": 0.9912,
219
+ "rec.2.54": 0.9973,
220
+ "s.2": 0.9975,
221
+ "out.56": 0.9979,
222
+ "out.57": 0.9979,
223
+ "out.58": 0.9981,
224
+ "out.59": 0.998,
225
+ "out.60": 0.9986,
226
+ "out.61": 0.9966,
227
+ "out.62": 0.9996,
228
+ "out.63": 0.9996,
229
+ "out.64": 0.9996,
230
+ "out.65": 0.999,
231
+ "out.66": 0.9997,
232
+ "out.67": 0.9994,
233
+ "out.68": 0.9998,
234
+ "out.69": 0.9997,
235
+ "out.70": 1.0
236
+ },
237
+ "cosine_skip4_vs_skip16": {
238
+ "emb": 0.8535,
239
+ "in.00": 0.8709,
240
+ "in.01": 0.9152,
241
+ "in.02": 0.9097,
242
+ "in.03": 0.9168,
243
+ "in.04": 0.9141,
244
+ "in.05": 0.9144,
245
+ "in.06": 0.9138,
246
+ "in.07": 0.9261,
247
+ "in.08": 0.9333,
248
+ "in.09": 0.9479,
249
+ "in.10": 0.9424,
250
+ "in.11": 0.9495,
251
+ "in.12": 0.9436,
252
+ "in.13": 0.9471,
253
+ "in.14": 0.9424,
254
+ "x": 0.9742,
255
+ "u.1": 0.9688,
256
+ "rec.1.16": 0.8697,
257
+ "rec.1.17": 0.8783,
258
+ "rec.1.18": 0.8715,
259
+ "rec.1.19": 0.8686,
260
+ "rec.1.20": 0.8643,
261
+ "rec.1.21": 0.8661,
262
+ "rec.1.22": 0.8928,
263
+ "rec.1.23": 0.9164,
264
+ "rec.1.24": 0.9241,
265
+ "rec.1.25": 0.9304,
266
+ "rec.1.26": 0.9281,
267
+ "rec.1.27": 0.9325,
268
+ "rec.1.28": 0.93,
269
+ "rec.1.29": 0.9414,
270
+ "rec.1.30": 0.9574,
271
+ "rec.1.31": 0.9567,
272
+ "rec.1.32": 0.965,
273
+ "rec.1.33": 0.9648,
274
+ "rec.1.34": 0.9743,
275
+ "rec.1.35": 0.9713,
276
+ "rec.1.36": 0.9761,
277
+ "rec.1.37": 0.9724,
278
+ "rec.1.38": 0.9797,
279
+ "rec.1.39": 0.9762,
280
+ "rec.1.40": 0.9816,
281
+ "rec.1.41": 0.9801,
282
+ "rec.1.42": 0.9807,
283
+ "rec.1.43": 0.9787,
284
+ "rec.1.44": 0.9819,
285
+ "rec.1.45": 0.9809,
286
+ "rec.1.46": 0.9848,
287
+ "rec.1.47": 0.9846,
288
+ "rec.1.48": 0.9852,
289
+ "rec.1.49": 0.9856,
290
+ "rec.1.50": 0.9865,
291
+ "rec.1.51": 0.9871,
292
+ "rec.1.52": 0.9879,
293
+ "rec.1.53": 0.9887,
294
+ "rec.1.54": 0.9885,
295
+ "s.1": 0.9893,
296
+ "u.2": 0.9894,
297
+ "rec.2.16": 0.9919,
298
+ "rec.2.17": 0.9931,
299
+ "rec.2.18": 0.9933,
300
+ "rec.2.19": 0.9937,
301
+ "rec.2.20": 0.9939,
302
+ "rec.2.21": 0.9941,
303
+ "rec.2.22": 0.995,
304
+ "rec.2.23": 0.9954,
305
+ "rec.2.24": 0.9951,
306
+ "rec.2.25": 0.9954,
307
+ "rec.2.26": 0.9952,
308
+ "rec.2.27": 0.9953,
309
+ "rec.2.28": 0.9953,
310
+ "rec.2.29": 0.9956,
311
+ "rec.2.30": 0.9959,
312
+ "rec.2.31": 0.9961,
313
+ "rec.2.32": 0.996,
314
+ "rec.2.33": 0.9962,
315
+ "rec.2.34": 0.9963,
316
+ "rec.2.35": 0.9964,
317
+ "rec.2.36": 0.9966,
318
+ "rec.2.37": 0.9966,
319
+ "rec.2.38": 0.9969,
320
+ "rec.2.39": 0.9971,
321
+ "rec.2.40": 0.9971,
322
+ "rec.2.41": 0.9973,
323
+ "rec.2.42": 0.9973,
324
+ "rec.2.43": 0.9975,
325
+ "rec.2.44": 0.9976,
326
+ "rec.2.45": 0.9977,
327
+ "rec.2.46": 0.9975,
328
+ "rec.2.47": 0.9977,
329
+ "rec.2.48": 0.9981,
330
+ "rec.2.49": 0.9983,
331
+ "rec.2.50": 0.9985,
332
+ "rec.2.51": 0.9986,
333
+ "rec.2.52": 0.9987,
334
+ "rec.2.53": 0.9984,
335
+ "rec.2.54": 0.9992,
336
+ "s.2": 0.9992,
337
+ "out.56": 0.9993,
338
+ "out.57": 0.9993,
339
+ "out.58": 0.9994,
340
+ "out.59": 0.9994,
341
+ "out.60": 0.9996,
342
+ "out.61": 0.9996,
343
+ "out.62": 0.9998,
344
+ "out.63": 0.9998,
345
+ "out.64": 0.9999,
346
+ "out.65": 0.9998,
347
+ "out.66": 0.9999,
348
+ "out.67": 0.9999,
349
+ "out.68": 1.0,
350
+ "out.69": 1.0,
351
+ "out.70": 1.0
352
+ },
353
+ "frobenius_norm_skip16": {
354
+ "emb": 2.2146,
355
+ "in.00": 2.8234,
356
+ "in.01": 3.7672,
357
+ "in.02": 3.5732,
358
+ "in.03": 3.9527,
359
+ "in.04": 3.7878,
360
+ "in.05": 3.8775,
361
+ "in.06": 3.5445,
362
+ "in.07": 3.7666,
363
+ "in.08": 3.7813,
364
+ "in.09": 4.968,
365
+ "in.10": 4.1077,
366
+ "in.11": 4.9831,
367
+ "in.12": 4.3913,
368
+ "in.13": 4.9565,
369
+ "in.14": 4.3409,
370
+ "x": 12.6086,
371
+ "u.1": 9.8178,
372
+ "rec.1.16": 2.6905,
373
+ "rec.1.17": 3.0037,
374
+ "rec.1.18": 2.4852,
375
+ "rec.1.19": 2.5136,
376
+ "rec.1.20": 2.3265,
377
+ "rec.1.21": 2.4131,
378
+ "rec.1.22": 2.7469,
379
+ "rec.1.23": 4.3637,
380
+ "rec.1.24": 2.8677,
381
+ "rec.1.25": 4.0631,
382
+ "rec.1.26": 3.0122,
383
+ "rec.1.27": 3.5127,
384
+ "rec.1.28": 2.628,
385
+ "rec.1.29": 2.8786,
386
+ "rec.1.30": 2.2008,
387
+ "rec.1.31": 3.4212,
388
+ "rec.1.32": 1.566,
389
+ "rec.1.33": 2.4946,
390
+ "rec.1.34": 1.3129,
391
+ "rec.1.35": 1.851,
392
+ "rec.1.36": 1.2502,
393
+ "rec.1.37": 1.4612,
394
+ "rec.1.38": 1.2233,
395
+ "rec.1.39": 1.3458,
396
+ "rec.1.40": 1.1301,
397
+ "rec.1.41": 1.2334,
398
+ "rec.1.42": 1.0051,
399
+ "rec.1.43": 1.2664,
400
+ "rec.1.44": 0.9525,
401
+ "rec.1.45": 0.9805,
402
+ "rec.1.46": 0.9992,
403
+ "rec.1.47": 0.947,
404
+ "rec.1.48": 0.9459,
405
+ "rec.1.49": 1.2021,
406
+ "rec.1.50": 0.9008,
407
+ "rec.1.51": 1.2039,
408
+ "rec.1.52": 0.9174,
409
+ "rec.1.53": 1.1964,
410
+ "rec.1.54": 1.1898,
411
+ "s.1": 7.1281,
412
+ "u.2": 10.1868,
413
+ "rec.2.16": 4.5114,
414
+ "rec.2.17": 5.9077,
415
+ "rec.2.18": 5.1911,
416
+ "rec.2.19": 6.0113,
417
+ "rec.2.20": 5.6144,
418
+ "rec.2.21": 6.2263,
419
+ "rec.2.22": 9.9843,
420
+ "rec.2.23": 15.179,
421
+ "rec.2.24": 10.8628,
422
+ "rec.2.25": 14.6502,
423
+ "rec.2.26": 11.8505,
424
+ "rec.2.27": 14.3837,
425
+ "rec.2.28": 11.6669,
426
+ "rec.2.29": 12.9162,
427
+ "rec.2.30": 10.4724,
428
+ "rec.2.31": 16.046,
429
+ "rec.2.32": 9.0777,
430
+ "rec.2.33": 15.4935,
431
+ "rec.2.34": 9.3138,
432
+ "rec.2.35": 13.7334,
433
+ "rec.2.36": 10.0192,
434
+ "rec.2.37": 12.5121,
435
+ "rec.2.38": 11.2781,
436
+ "rec.2.39": 14.5057,
437
+ "rec.2.40": 11.7871,
438
+ "rec.2.41": 14.6978,
439
+ "rec.2.42": 11.8053,
440
+ "rec.2.43": 17.0247,
441
+ "rec.2.44": 12.9506,
442
+ "rec.2.45": 15.1354,
443
+ "rec.2.46": 15.2942,
444
+ "rec.2.47": 17.0247,
445
+ "rec.2.48": 17.2956,
446
+ "rec.2.49": 25.1086,
447
+ "rec.2.50": 19.3662,
448
+ "rec.2.51": 27.7479,
449
+ "rec.2.52": 21.7482,
450
+ "rec.2.53": 29.3495,
451
+ "rec.2.54": 29.0059,
452
+ "s.2": 174.9567,
453
+ "out.56": 30.3758,
454
+ "out.57": 37.8704,
455
+ "out.58": 36.2843,
456
+ "out.59": 44.738,
457
+ "out.60": 40.3314,
458
+ "out.61": 57.1707,
459
+ "out.62": 43.1408,
460
+ "out.63": 50.0746,
461
+ "out.64": 42.5106,
462
+ "out.65": 73.4981,
463
+ "out.66": 45.8319,
464
+ "out.67": 77.2833,
465
+ "out.68": 46.5442,
466
+ "out.69": 71.2196,
467
+ "out.70": 54.0014
468
+ },
469
+ "position_norm_over_late_median": {
470
+ "emb": [
471
+ 6.134,
472
+ 4.976,
473
+ 4.179,
474
+ 3.822,
475
+ 6.508,
476
+ 4.17,
477
+ 3.346,
478
+ 3.278,
479
+ 3.988,
480
+ 19.194,
481
+ 3.905,
482
+ 6.855,
483
+ 1.634,
484
+ 3.68,
485
+ 3.99,
486
+ 2.741,
487
+ 4.022,
488
+ 1.845,
489
+ 1.471,
490
+ 1.599
491
+ ],
492
+ "in.00": [
493
+ 4.856,
494
+ 4.642,
495
+ 3.976,
496
+ 3.601,
497
+ 6.016,
498
+ 3.888,
499
+ 3.923,
500
+ 3.09,
501
+ 3.541,
502
+ 17.654,
503
+ 4.305,
504
+ 6.359,
505
+ 1.602,
506
+ 3.262,
507
+ 3.552,
508
+ 2.711,
509
+ 4.17,
510
+ 2.292,
511
+ 1.565,
512
+ 1.562
513
+ ],
514
+ "in.01": [
515
+ 4.971,
516
+ 3.673,
517
+ 3.513,
518
+ 3.016,
519
+ 4.616,
520
+ 3.589,
521
+ 3.689,
522
+ 2.885,
523
+ 2.991,
524
+ 13.134,
525
+ 3.46,
526
+ 4.757,
527
+ 1.61,
528
+ 2.707,
529
+ 2.694,
530
+ 2.643,
531
+ 4.057,
532
+ 2.453,
533
+ 1.472,
534
+ 1.524
535
+ ],
536
+ "in.02": [
537
+ 3.171,
538
+ 3.516,
539
+ 3.201,
540
+ 2.878,
541
+ 4.815,
542
+ 3.513,
543
+ 4.066,
544
+ 2.819,
545
+ 2.909,
546
+ 13.597,
547
+ 3.946,
548
+ 5.021,
549
+ 1.544,
550
+ 2.659,
551
+ 2.702,
552
+ 2.692,
553
+ 3.725,
554
+ 2.68,
555
+ 1.391,
556
+ 1.412
557
+ ],
558
+ "in.03": [
559
+ 3.32,
560
+ 3.314,
561
+ 3.089,
562
+ 2.697,
563
+ 4.596,
564
+ 3.53,
565
+ 4.001,
566
+ 2.819,
567
+ 2.804,
568
+ 13.294,
569
+ 4.021,
570
+ 4.434,
571
+ 1.578,
572
+ 2.609,
573
+ 2.43,
574
+ 2.526,
575
+ 3.93,
576
+ 2.444,
577
+ 1.348,
578
+ 1.399
579
+ ],
580
+ "in.04": [
581
+ 2.549,
582
+ 2.927,
583
+ 2.938,
584
+ 2.38,
585
+ 4.371,
586
+ 3.302,
587
+ 4.106,
588
+ 2.814,
589
+ 2.568,
590
+ 15.373,
591
+ 3.627,
592
+ 4.717,
593
+ 1.589,
594
+ 2.76,
595
+ 2.414,
596
+ 2.723,
597
+ 4.173,
598
+ 2.628,
599
+ 1.344,
600
+ 1.444
601
+ ],
602
+ "in.05": [
603
+ 2.823,
604
+ 2.619,
605
+ 2.887,
606
+ 2.24,
607
+ 3.85,
608
+ 3.158,
609
+ 3.653,
610
+ 2.63,
611
+ 2.458,
612
+ 15.428,
613
+ 3.698,
614
+ 4.736,
615
+ 1.513,
616
+ 2.825,
617
+ 2.278,
618
+ 2.694,
619
+ 4.063,
620
+ 2.413,
621
+ 1.305,
622
+ 1.354
623
+ ],
624
+ "in.06": [
625
+ 2.984,
626
+ 2.52,
627
+ 2.681,
628
+ 2.043,
629
+ 4.004,
630
+ 2.941,
631
+ 3.268,
632
+ 2.508,
633
+ 2.509,
634
+ 16.865,
635
+ 3.534,
636
+ 5.31,
637
+ 1.474,
638
+ 2.894,
639
+ 2.299,
640
+ 2.826,
641
+ 3.669,
642
+ 1.752,
643
+ 1.286,
644
+ 1.259
645
+ ],
646
+ "in.07": [
647
+ 2.937,
648
+ 2.166,
649
+ 2.535,
650
+ 1.834,
651
+ 3.129,
652
+ 2.558,
653
+ 3.021,
654
+ 2.309,
655
+ 2.187,
656
+ 13.784,
657
+ 3.029,
658
+ 4.312,
659
+ 1.383,
660
+ 2.369,
661
+ 1.884,
662
+ 2.388,
663
+ 3.237,
664
+ 1.459,
665
+ 1.119,
666
+ 1.089
667
+ ],
668
+ "in.08": [
669
+ 2.651,
670
+ 2.219,
671
+ 2.689,
672
+ 1.891,
673
+ 3.366,
674
+ 2.712,
675
+ 3.286,
676
+ 2.522,
677
+ 2.239,
678
+ 15.054,
679
+ 3.243,
680
+ 4.722,
681
+ 1.457,
682
+ 2.543,
683
+ 2.057,
684
+ 2.682,
685
+ 3.014,
686
+ 1.579,
687
+ 1.203,
688
+ 1.176
689
+ ],
690
+ "in.09": [
691
+ 2.859,
692
+ 1.973,
693
+ 2.542,
694
+ 1.748,
695
+ 2.74,
696
+ 2.479,
697
+ 3.135,
698
+ 2.775,
699
+ 2.182,
700
+ 12.351,
701
+ 2.865,
702
+ 3.818,
703
+ 1.69,
704
+ 2.383,
705
+ 1.763,
706
+ 2.712,
707
+ 2.489,
708
+ 1.511,
709
+ 1.136,
710
+ 1.053
711
+ ],
712
+ "in.10": [
713
+ 2.797,
714
+ 1.934,
715
+ 2.548,
716
+ 1.812,
717
+ 2.966,
718
+ 2.621,
719
+ 3.186,
720
+ 2.847,
721
+ 2.161,
722
+ 13.306,
723
+ 2.964,
724
+ 4.293,
725
+ 1.605,
726
+ 2.521,
727
+ 1.909,
728
+ 2.81,
729
+ 2.655,
730
+ 1.582,
731
+ 1.167,
732
+ 1.071
733
+ ],
734
+ "in.11": [
735
+ 3.014,
736
+ 1.904,
737
+ 2.594,
738
+ 1.926,
739
+ 2.732,
740
+ 2.6,
741
+ 3.405,
742
+ 3.166,
743
+ 2.311,
744
+ 12.447,
745
+ 3.071,
746
+ 4.093,
747
+ 2.002,
748
+ 2.49,
749
+ 1.9,
750
+ 2.986,
751
+ 2.702,
752
+ 1.632,
753
+ 1.171,
754
+ 1.103
755
+ ],
756
+ "in.12": [
757
+ 2.668,
758
+ 1.708,
759
+ 2.598,
760
+ 1.906,
761
+ 2.699,
762
+ 2.717,
763
+ 3.552,
764
+ 3.053,
765
+ 2.158,
766
+ 13.359,
767
+ 3.014,
768
+ 4.499,
769
+ 1.804,
770
+ 2.639,
771
+ 1.864,
772
+ 3.194,
773
+ 2.817,
774
+ 1.692,
775
+ 1.154,
776
+ 1.096
777
+ ],
778
+ "in.13": [
779
+ 5.607,
780
+ 1.81,
781
+ 2.75,
782
+ 2.175,
783
+ 2.604,
784
+ 3.02,
785
+ 3.881,
786
+ 3.296,
787
+ 2.353,
788
+ 13.421,
789
+ 3.12,
790
+ 4.762,
791
+ 1.972,
792
+ 2.772,
793
+ 2.004,
794
+ 3.357,
795
+ 2.991,
796
+ 1.917,
797
+ 1.229,
798
+ 1.165
799
+ ],
800
+ "in.14": [
801
+ 2.544,
802
+ 1.644,
803
+ 2.48,
804
+ 1.982,
805
+ 2.467,
806
+ 2.905,
807
+ 3.51,
808
+ 3.1,
809
+ 2.308,
810
+ 14.467,
811
+ 3.086,
812
+ 5.621,
813
+ 1.989,
814
+ 2.896,
815
+ 1.981,
816
+ 3.53,
817
+ 2.881,
818
+ 1.903,
819
+ 1.198,
820
+ 1.152
821
+ ],
822
+ "x": [
823
+ 1.129,
824
+ 0.681,
825
+ 1.284,
826
+ 0.894,
827
+ 1.022,
828
+ 1.7,
829
+ 2.146,
830
+ 2.485,
831
+ 2.114,
832
+ 5.354,
833
+ 2.938,
834
+ 2.581,
835
+ 2.123,
836
+ 1.703,
837
+ 1.2,
838
+ 3.087,
839
+ 2.023,
840
+ 1.326,
841
+ 0.983,
842
+ 0.928
843
+ ],
844
+ "u.1": [
845
+ 1.098,
846
+ 0.473,
847
+ 0.798,
848
+ 0.595,
849
+ 0.834,
850
+ 1.785,
851
+ 2.329,
852
+ 2.663,
853
+ 2.25,
854
+ 6.056,
855
+ 3.189,
856
+ 2.885,
857
+ 2.294,
858
+ 1.699,
859
+ 1.049,
860
+ 3.379,
861
+ 2.14,
862
+ 1.017,
863
+ 0.732,
864
+ 0.713
865
+ ],
866
+ "rec.1.16": [
867
+ 2.217,
868
+ 1.063,
869
+ 1.651,
870
+ 1.23,
871
+ 1.525,
872
+ 3.083,
873
+ 5.448,
874
+ 4.528,
875
+ 3.372,
876
+ 18.91,
877
+ 4.255,
878
+ 8.951,
879
+ 3.284,
880
+ 4.018,
881
+ 1.726,
882
+ 5.801,
883
+ 3.88,
884
+ 1.743,
885
+ 0.854,
886
+ 0.987
887
+ ],
888
+ "rec.1.17": [
889
+ 3.31,
890
+ 1.046,
891
+ 1.424,
892
+ 1.136,
893
+ 1.525,
894
+ 3.18,
895
+ 6.674,
896
+ 4.471,
897
+ 3.534,
898
+ 18.056,
899
+ 4.031,
900
+ 7.825,
901
+ 3.448,
902
+ 3.999,
903
+ 1.671,
904
+ 5.841,
905
+ 3.627,
906
+ 1.749,
907
+ 0.837,
908
+ 0.974
909
+ ],
910
+ "rec.1.18": [
911
+ 2.847,
912
+ 1.039,
913
+ 1.585,
914
+ 1.194,
915
+ 1.436,
916
+ 2.952,
917
+ 7.221,
918
+ 4.658,
919
+ 4.021,
920
+ 22.013,
921
+ 4.274,
922
+ 9.152,
923
+ 3.748,
924
+ 4.439,
925
+ 1.812,
926
+ 7.048,
927
+ 3.749,
928
+ 1.942,
929
+ 0.872,
930
+ 1.075
931
+ ],
932
+ "rec.1.19": [
933
+ 4.501,
934
+ 1.105,
935
+ 1.247,
936
+ 1.175,
937
+ 1.439,
938
+ 3.139,
939
+ 7.11,
940
+ 4.367,
941
+ 3.77,
942
+ 19.869,
943
+ 4.483,
944
+ 8.282,
945
+ 3.943,
946
+ 4.219,
947
+ 1.781,
948
+ 6.86,
949
+ 3.771,
950
+ 2.049,
951
+ 0.857,
952
+ 1.056
953
+ ],
954
+ "rec.1.20": [
955
+ 3.935,
956
+ 1.254,
957
+ 1.175,
958
+ 1.325,
959
+ 1.61,
960
+ 3.297,
961
+ 7.764,
962
+ 4.913,
963
+ 4.049,
964
+ 23.637,
965
+ 5.162,
966
+ 9.746,
967
+ 4.627,
968
+ 4.912,
969
+ 2.07,
970
+ 8.419,
971
+ 4.163,
972
+ 2.4,
973
+ 0.968,
974
+ 1.207
975
+ ],
976
+ "rec.1.21": [
977
+ 5.398,
978
+ 1.47,
979
+ 1.349,
980
+ 1.477,
981
+ 1.899,
982
+ 3.698,
983
+ 8.506,
984
+ 5.603,
985
+ 4.479,
986
+ 23.665,
987
+ 5.899,
988
+ 9.897,
989
+ 5.575,
990
+ 5.344,
991
+ 2.338,
992
+ 9.034,
993
+ 5.09,
994
+ 2.66,
995
+ 1.065,
996
+ 1.355
997
+ ],
998
+ "rec.1.22": [
999
+ 4.732,
1000
+ 1.224,
1001
+ 1.265,
1002
+ 1.266,
1003
+ 1.544,
1004
+ 3.631,
1005
+ 7.268,
1006
+ 5.085,
1007
+ 4.428,
1008
+ 22.456,
1009
+ 5.709,
1010
+ 9.236,
1011
+ 4.478,
1012
+ 5.095,
1013
+ 2.186,
1014
+ 9.232,
1015
+ 4.945,
1016
+ 2.469,
1017
+ 1.059,
1018
+ 1.168
1019
+ ],
1020
+ "rec.1.23": [
1021
+ 6.262,
1022
+ 1.139,
1023
+ 1.231,
1024
+ 1.068,
1025
+ 1.349,
1026
+ 3.395,
1027
+ 6.808,
1028
+ 5.2,
1029
+ 3.834,
1030
+ 17.403,
1031
+ 5.933,
1032
+ 8.104,
1033
+ 4.492,
1034
+ 4.453,
1035
+ 2.152,
1036
+ 9.51,
1037
+ 4.916,
1038
+ 2.489,
1039
+ 1.055,
1040
+ 1.088
1041
+ ],
1042
+ "rec.1.24": [
1043
+ 5.299,
1044
+ 1.209,
1045
+ 1.2,
1046
+ 1.267,
1047
+ 1.448,
1048
+ 3.443,
1049
+ 7.024,
1050
+ 5.315,
1051
+ 4.112,
1052
+ 18.095,
1053
+ 6.145,
1054
+ 8.961,
1055
+ 5.322,
1056
+ 5.539,
1057
+ 2.484,
1058
+ 7.03,
1059
+ 5.917,
1060
+ 2.652,
1061
+ 1.169,
1062
+ 1.238
1063
+ ],
1064
+ "rec.1.25": [
1065
+ 5.581,
1066
+ 1.184,
1067
+ 1.165,
1068
+ 1.17,
1069
+ 1.332,
1070
+ 3.492,
1071
+ 6.88,
1072
+ 5.228,
1073
+ 4.34,
1074
+ 16.28,
1075
+ 6.533,
1076
+ 7.969,
1077
+ 5.487,
1078
+ 5.432,
1079
+ 2.445,
1080
+ 7.772,
1081
+ 5.819,
1082
+ 2.874,
1083
+ 1.202,
1084
+ 1.146
1085
+ ],
1086
+ "rec.1.26": [
1087
+ 5.674,
1088
+ 1.312,
1089
+ 1.106,
1090
+ 1.232,
1091
+ 1.374,
1092
+ 3.243,
1093
+ 6.045,
1094
+ 5.56,
1095
+ 4.321,
1096
+ 18.109,
1097
+ 6.888,
1098
+ 8.497,
1099
+ 6.175,
1100
+ 5.702,
1101
+ 2.349,
1102
+ 5.602,
1103
+ 6.388,
1104
+ 2.591,
1105
+ 1.205,
1106
+ 1.13
1107
+ ],
1108
+ "rec.1.27": [
1109
+ 5.778,
1110
+ 1.429,
1111
+ 1.2,
1112
+ 1.163,
1113
+ 1.338,
1114
+ 3.366,
1115
+ 5.762,
1116
+ 4.882,
1117
+ 3.816,
1118
+ 16.929,
1119
+ 6.883,
1120
+ 8.963,
1121
+ 6.025,
1122
+ 5.439,
1123
+ 2.232,
1124
+ 5.676,
1125
+ 6.58,
1126
+ 2.512,
1127
+ 1.161,
1128
+ 1.075
1129
+ ],
1130
+ "rec.1.28": [
1131
+ 4.926,
1132
+ 1.175,
1133
+ 1.138,
1134
+ 1.028,
1135
+ 1.135,
1136
+ 3.392,
1137
+ 5.422,
1138
+ 2.561,
1139
+ 2.257,
1140
+ 17.513,
1141
+ 7.325,
1142
+ 8.96,
1143
+ 6.449,
1144
+ 5.879,
1145
+ 2.152,
1146
+ 5.641,
1147
+ 6.672,
1148
+ 1.791,
1149
+ 1.139,
1150
+ 0.996
1151
+ ],
1152
+ "rec.1.29": [
1153
+ 5.887,
1154
+ 1.245,
1155
+ 1.193,
1156
+ 1.035,
1157
+ 1.172,
1158
+ 3.662,
1159
+ 5.776,
1160
+ 2.715,
1161
+ 2.224,
1162
+ 16.602,
1163
+ 7.01,
1164
+ 8.837,
1165
+ 6.549,
1166
+ 6.201,
1167
+ 2.127,
1168
+ 5.704,
1169
+ 6.89,
1170
+ 1.882,
1171
+ 1.179,
1172
+ 1.0
1173
+ ],
1174
+ "rec.1.30": [
1175
+ 4.457,
1176
+ 1.175,
1177
+ 1.295,
1178
+ 0.974,
1179
+ 1.164,
1180
+ 2.322,
1181
+ 4.582,
1182
+ 1.677,
1183
+ 1.973,
1184
+ 10.54,
1185
+ 5.085,
1186
+ 6.179,
1187
+ 6.075,
1188
+ 5.866,
1189
+ 1.773,
1190
+ 5.21,
1191
+ 4.551,
1192
+ 1.628,
1193
+ 1.056,
1194
+ 0.943
1195
+ ],
1196
+ "rec.1.31": [
1197
+ 3.655,
1198
+ 1.106,
1199
+ 1.166,
1200
+ 0.877,
1201
+ 1.083,
1202
+ 2.526,
1203
+ 5.141,
1204
+ 1.634,
1205
+ 1.964,
1206
+ 10.279,
1207
+ 5.682,
1208
+ 6.159,
1209
+ 6.586,
1210
+ 6.553,
1211
+ 1.921,
1212
+ 5.951,
1213
+ 4.765,
1214
+ 1.537,
1215
+ 1.087,
1216
+ 0.889
1217
+ ],
1218
+ "rec.1.32": [
1219
+ 4.704,
1220
+ 1.359,
1221
+ 1.548,
1222
+ 1.072,
1223
+ 1.321,
1224
+ 1.365,
1225
+ 4.459,
1226
+ 1.642,
1227
+ 1.824,
1228
+ 9.378,
1229
+ 4.842,
1230
+ 4.406,
1231
+ 7.409,
1232
+ 5.198,
1233
+ 1.919,
1234
+ 3.124,
1235
+ 4.154,
1236
+ 1.714,
1237
+ 1.198,
1238
+ 1.0
1239
+ ],
1240
+ "rec.1.33": [
1241
+ 3.877,
1242
+ 1.321,
1243
+ 1.342,
1244
+ 1.02,
1245
+ 1.259,
1246
+ 1.405,
1247
+ 5.284,
1248
+ 1.675,
1249
+ 1.993,
1250
+ 9.503,
1251
+ 5.114,
1252
+ 4.678,
1253
+ 8.296,
1254
+ 5.462,
1255
+ 2.011,
1256
+ 3.169,
1257
+ 4.438,
1258
+ 1.72,
1259
+ 1.23,
1260
+ 0.985
1261
+ ],
1262
+ "rec.1.34": [
1263
+ 4.319,
1264
+ 1.577,
1265
+ 1.677,
1266
+ 1.15,
1267
+ 1.546,
1268
+ 1.648,
1269
+ 3.983,
1270
+ 1.722,
1271
+ 2.098,
1272
+ 7.687,
1273
+ 2.845,
1274
+ 3.327,
1275
+ 1.436,
1276
+ 5.375,
1277
+ 1.596,
1278
+ 3.476,
1279
+ 1.893,
1280
+ 1.97,
1281
+ 1.21,
1282
+ 1.155
1283
+ ],
1284
+ "rec.1.35": [
1285
+ 4.387,
1286
+ 1.502,
1287
+ 1.721,
1288
+ 1.08,
1289
+ 1.643,
1290
+ 1.817,
1291
+ 4.811,
1292
+ 1.672,
1293
+ 2.189,
1294
+ 9.174,
1295
+ 3.073,
1296
+ 3.918,
1297
+ 1.469,
1298
+ 6.003,
1299
+ 1.656,
1300
+ 3.727,
1301
+ 1.9,
1302
+ 1.953,
1303
+ 1.244,
1304
+ 1.167
1305
+ ],
1306
+ "rec.1.36": [
1307
+ 5.148,
1308
+ 1.786,
1309
+ 1.95,
1310
+ 1.317,
1311
+ 1.624,
1312
+ 1.926,
1313
+ 5.099,
1314
+ 1.808,
1315
+ 1.586,
1316
+ 9.099,
1317
+ 2.543,
1318
+ 2.478,
1319
+ 1.383,
1320
+ 6.612,
1321
+ 1.858,
1322
+ 2.532,
1323
+ 1.971,
1324
+ 2.139,
1325
+ 1.372,
1326
+ 1.322
1327
+ ],
1328
+ "rec.1.37": [
1329
+ 6.532,
1330
+ 1.864,
1331
+ 2.13,
1332
+ 1.34,
1333
+ 1.649,
1334
+ 2.082,
1335
+ 5.968,
1336
+ 1.8,
1337
+ 1.636,
1338
+ 10.338,
1339
+ 2.636,
1340
+ 2.68,
1341
+ 1.414,
1342
+ 7.523,
1343
+ 1.979,
1344
+ 2.737,
1345
+ 1.981,
1346
+ 2.045,
1347
+ 1.372,
1348
+ 1.322
1349
+ ],
1350
+ "rec.1.38": [
1351
+ 5.023,
1352
+ 1.847,
1353
+ 1.947,
1354
+ 1.288,
1355
+ 1.606,
1356
+ 1.862,
1357
+ 5.051,
1358
+ 1.915,
1359
+ 1.613,
1360
+ 4.104,
1361
+ 2.374,
1362
+ 1.813,
1363
+ 1.47,
1364
+ 5.677,
1365
+ 1.636,
1366
+ 2.012,
1367
+ 1.931,
1368
+ 2.151,
1369
+ 1.453,
1370
+ 1.37
1371
+ ],
1372
+ "rec.1.39": [
1373
+ 6.01,
1374
+ 1.891,
1375
+ 2.56,
1376
+ 1.305,
1377
+ 1.646,
1378
+ 2.063,
1379
+ 5.4,
1380
+ 1.932,
1381
+ 1.624,
1382
+ 4.128,
1383
+ 2.404,
1384
+ 1.804,
1385
+ 1.494,
1386
+ 6.399,
1387
+ 1.77,
1388
+ 2.037,
1389
+ 1.929,
1390
+ 2.148,
1391
+ 1.448,
1392
+ 1.373
1393
+ ],
1394
+ "rec.1.40": [
1395
+ 4.469,
1396
+ 1.923,
1397
+ 1.906,
1398
+ 1.298,
1399
+ 1.572,
1400
+ 1.853,
1401
+ 4.982,
1402
+ 1.938,
1403
+ 1.618,
1404
+ 3.432,
1405
+ 2.355,
1406
+ 1.798,
1407
+ 1.531,
1408
+ 2.171,
1409
+ 1.624,
1410
+ 1.809,
1411
+ 1.981,
1412
+ 2.131,
1413
+ 1.489,
1414
+ 1.416
1415
+ ],
1416
+ "rec.1.41": [
1417
+ 4.696,
1418
+ 1.991,
1419
+ 1.939,
1420
+ 1.301,
1421
+ 1.551,
1422
+ 1.881,
1423
+ 5.339,
1424
+ 1.932,
1425
+ 1.613,
1426
+ 3.454,
1427
+ 2.344,
1428
+ 1.796,
1429
+ 1.54,
1430
+ 2.224,
1431
+ 1.622,
1432
+ 1.822,
1433
+ 1.995,
1434
+ 2.14,
1435
+ 1.483,
1436
+ 1.422
1437
+ ],
1438
+ "rec.1.42": [
1439
+ 4.687,
1440
+ 2.225,
1441
+ 2.047,
1442
+ 1.396,
1443
+ 1.735,
1444
+ 2.044,
1445
+ 5.781,
1446
+ 2.187,
1447
+ 1.79,
1448
+ 3.616,
1449
+ 2.664,
1450
+ 1.978,
1451
+ 1.687,
1452
+ 2.246,
1453
+ 1.767,
1454
+ 1.976,
1455
+ 2.205,
1456
+ 2.346,
1457
+ 1.645,
1458
+ 1.571
1459
+ ],
1460
+ "rec.1.43": [
1461
+ 5.667,
1462
+ 2.407,
1463
+ 2.409,
1464
+ 1.454,
1465
+ 1.783,
1466
+ 2.186,
1467
+ 6.551,
1468
+ 2.203,
1469
+ 1.798,
1470
+ 3.749,
1471
+ 2.692,
1472
+ 2.078,
1473
+ 1.725,
1474
+ 2.205,
1475
+ 1.791,
1476
+ 2.009,
1477
+ 2.164,
1478
+ 2.259,
1479
+ 1.664,
1480
+ 1.626
1481
+ ],
1482
+ "rec.1.44": [
1483
+ 5.207,
1484
+ 2.284,
1485
+ 2.177,
1486
+ 1.464,
1487
+ 1.841,
1488
+ 1.707,
1489
+ 1.842,
1490
+ 2.198,
1491
+ 1.831,
1492
+ 4.02,
1493
+ 2.675,
1494
+ 2.135,
1495
+ 1.757,
1496
+ 2.316,
1497
+ 1.82,
1498
+ 2.061,
1499
+ 2.303,
1500
+ 2.432,
1501
+ 1.711,
1502
+ 1.635
1503
+ ],
1504
+ "rec.1.45": [
1505
+ 5.401,
1506
+ 2.372,
1507
+ 2.186,
1508
+ 1.49,
1509
+ 1.863,
1510
+ 1.723,
1511
+ 1.854,
1512
+ 2.222,
1513
+ 1.834,
1514
+ 4.105,
1515
+ 2.675,
1516
+ 2.182,
1517
+ 1.772,
1518
+ 2.258,
1519
+ 1.81,
1520
+ 2.055,
1521
+ 2.297,
1522
+ 2.326,
1523
+ 1.692,
1524
+ 1.654
1525
+ ],
1526
+ "rec.1.46": [
1527
+ 3.553,
1528
+ 1.715,
1529
+ 2.235,
1530
+ 1.494,
1531
+ 1.671,
1532
+ 1.682,
1533
+ 1.775,
1534
+ 2.066,
1535
+ 1.871,
1536
+ 2.341,
1537
+ 2.557,
1538
+ 1.801,
1539
+ 1.823,
1540
+ 2.091,
1541
+ 1.824,
1542
+ 2.068,
1543
+ 2.151,
1544
+ 2.341,
1545
+ 1.747,
1546
+ 1.679
1547
+ ],
1548
+ "rec.1.47": [
1549
+ 3.234,
1550
+ 1.697,
1551
+ 2.116,
1552
+ 1.544,
1553
+ 1.719,
1554
+ 1.722,
1555
+ 1.836,
1556
+ 2.122,
1557
+ 1.921,
1558
+ 2.393,
1559
+ 2.571,
1560
+ 1.853,
1561
+ 1.87,
1562
+ 2.127,
1563
+ 1.84,
1564
+ 2.112,
1565
+ 2.193,
1566
+ 2.333,
1567
+ 1.778,
1568
+ 1.722
1569
+ ],
1570
+ "rec.1.48": [
1571
+ 3.124,
1572
+ 1.661,
1573
+ 2.138,
1574
+ 1.587,
1575
+ 1.802,
1576
+ 1.756,
1577
+ 1.899,
1578
+ 2.199,
1579
+ 2.018,
1580
+ 2.482,
1581
+ 2.614,
1582
+ 1.944,
1583
+ 1.943,
1584
+ 2.181,
1585
+ 1.95,
1586
+ 2.234,
1587
+ 2.173,
1588
+ 2.432,
1589
+ 1.884,
1590
+ 1.82
1591
+ ],
1592
+ "rec.1.49": [
1593
+ 3.564,
1594
+ 1.744,
1595
+ 2.417,
1596
+ 1.645,
1597
+ 1.74,
1598
+ 1.74,
1599
+ 1.934,
1600
+ 2.187,
1601
+ 2.016,
1602
+ 2.415,
1603
+ 2.5,
1604
+ 1.967,
1605
+ 1.973,
1606
+ 2.212,
1607
+ 2.0,
1608
+ 2.301,
1609
+ 2.161,
1610
+ 2.379,
1611
+ 1.868,
1612
+ 1.843
1613
+ ],
1614
+ "rec.1.50": [
1615
+ 2.054,
1616
+ 1.568,
1617
+ 2.158,
1618
+ 1.658,
1619
+ 1.902,
1620
+ 1.783,
1621
+ 1.911,
1622
+ 2.262,
1623
+ 2.049,
1624
+ 2.434,
1625
+ 2.64,
1626
+ 1.915,
1627
+ 1.943,
1628
+ 2.198,
1629
+ 1.99,
1630
+ 2.301,
1631
+ 2.137,
1632
+ 2.462,
1633
+ 1.939,
1634
+ 1.865
1635
+ ],
1636
+ "rec.1.51": [
1637
+ 2.077,
1638
+ 1.617,
1639
+ 2.444,
1640
+ 1.747,
1641
+ 1.937,
1642
+ 1.805,
1643
+ 1.974,
1644
+ 2.295,
1645
+ 2.115,
1646
+ 2.465,
1647
+ 2.721,
1648
+ 1.957,
1649
+ 2.015,
1650
+ 2.237,
1651
+ 2.019,
1652
+ 2.422,
1653
+ 2.127,
1654
+ 2.538,
1655
+ 1.996,
1656
+ 1.927
1657
+ ],
1658
+ "rec.1.52": [
1659
+ 2.195,
1660
+ 1.482,
1661
+ 2.318,
1662
+ 1.701,
1663
+ 1.953,
1664
+ 1.822,
1665
+ 1.925,
1666
+ 2.354,
1667
+ 2.07,
1668
+ 2.43,
1669
+ 2.704,
1670
+ 1.9,
1671
+ 1.95,
1672
+ 2.236,
1673
+ 1.996,
1674
+ 2.385,
1675
+ 2.179,
1676
+ 2.556,
1677
+ 1.97,
1678
+ 1.752
1679
+ ],
1680
+ "rec.1.53": [
1681
+ 5.219,
1682
+ 1.544,
1683
+ 2.123,
1684
+ 1.705,
1685
+ 2.07,
1686
+ 1.895,
1687
+ 1.925,
1688
+ 2.426,
1689
+ 2.054,
1690
+ 2.397,
1691
+ 2.788,
1692
+ 1.856,
1693
+ 1.916,
1694
+ 2.252,
1695
+ 2.007,
1696
+ 2.304,
1697
+ 2.148,
1698
+ 2.555,
1699
+ 1.932,
1700
+ 1.786
1701
+ ],
1702
+ "rec.1.54": [
1703
+ 1.828,
1704
+ 1.412,
1705
+ 2.32,
1706
+ 1.703,
1707
+ 1.941,
1708
+ 1.861,
1709
+ 1.905,
1710
+ 2.379,
1711
+ 2.079,
1712
+ 2.468,
1713
+ 2.649,
1714
+ 1.881,
1715
+ 1.945,
1716
+ 2.212,
1717
+ 1.993,
1718
+ 2.403,
1719
+ 2.186,
1720
+ 2.576,
1721
+ 1.957,
1722
+ 1.787
1723
+ ],
1724
+ "s.1": [
1725
+ 1.616,
1726
+ 1.38,
1727
+ 2.497,
1728
+ 1.82,
1729
+ 1.831,
1730
+ 1.932,
1731
+ 1.986,
1732
+ 2.467,
1733
+ 2.154,
1734
+ 2.56,
1735
+ 2.631,
1736
+ 1.979,
1737
+ 2.056,
1738
+ 2.209,
1739
+ 2.055,
1740
+ 2.626,
1741
+ 2.176,
1742
+ 2.605,
1743
+ 2.018,
1744
+ 1.83
1745
+ ],
1746
+ "u.2": [
1747
+ 1.666,
1748
+ 1.452,
1749
+ 2.542,
1750
+ 1.876,
1751
+ 1.893,
1752
+ 1.976,
1753
+ 2.002,
1754
+ 2.482,
1755
+ 2.172,
1756
+ 2.564,
1757
+ 2.6,
1758
+ 1.956,
1759
+ 2.055,
1760
+ 2.201,
1761
+ 2.002,
1762
+ 2.569,
1763
+ 2.173,
1764
+ 2.644,
1765
+ 1.999,
1766
+ 1.822
1767
+ ],
1768
+ "rec.2.16": [
1769
+ 2.817,
1770
+ 2.082,
1771
+ 2.695,
1772
+ 2.499,
1773
+ 2.752,
1774
+ 2.238,
1775
+ 2.299,
1776
+ 2.663,
1777
+ 2.496,
1778
+ 2.519,
1779
+ 3.442,
1780
+ 1.829,
1781
+ 2.097,
1782
+ 2.296,
1783
+ 1.83,
1784
+ 2.436,
1785
+ 2.257,
1786
+ 2.98,
1787
+ 2.055,
1788
+ 1.808
1789
+ ],
1790
+ "rec.2.17": [
1791
+ 3.803,
1792
+ 2.109,
1793
+ 2.552,
1794
+ 2.35,
1795
+ 2.625,
1796
+ 2.232,
1797
+ 2.331,
1798
+ 2.561,
1799
+ 2.544,
1800
+ 2.466,
1801
+ 3.224,
1802
+ 1.839,
1803
+ 2.02,
1804
+ 2.188,
1805
+ 1.813,
1806
+ 2.299,
1807
+ 2.236,
1808
+ 2.887,
1809
+ 2.044,
1810
+ 1.81
1811
+ ],
1812
+ "rec.2.18": [
1813
+ 3.724,
1814
+ 2.073,
1815
+ 2.574,
1816
+ 2.348,
1817
+ 2.737,
1818
+ 2.232,
1819
+ 2.359,
1820
+ 2.609,
1821
+ 2.607,
1822
+ 2.468,
1823
+ 3.403,
1824
+ 1.873,
1825
+ 2.043,
1826
+ 2.239,
1827
+ 1.843,
1828
+ 2.308,
1829
+ 2.321,
1830
+ 2.954,
1831
+ 2.061,
1832
+ 1.864
1833
+ ],
1834
+ "rec.2.19": [
1835
+ 5.353,
1836
+ 2.144,
1837
+ 2.294,
1838
+ 2.272,
1839
+ 2.554,
1840
+ 2.196,
1841
+ 2.342,
1842
+ 2.514,
1843
+ 2.542,
1844
+ 2.399,
1845
+ 3.204,
1846
+ 1.851,
1847
+ 2.014,
1848
+ 2.204,
1849
+ 1.824,
1850
+ 2.226,
1851
+ 2.226,
1852
+ 2.881,
1853
+ 2.01,
1854
+ 1.852
1855
+ ],
1856
+ "rec.2.20": [
1857
+ 4.386,
1858
+ 2.081,
1859
+ 2.134,
1860
+ 2.266,
1861
+ 2.565,
1862
+ 2.16,
1863
+ 2.319,
1864
+ 2.517,
1865
+ 2.509,
1866
+ 2.38,
1867
+ 3.263,
1868
+ 1.825,
1869
+ 1.98,
1870
+ 2.186,
1871
+ 1.805,
1872
+ 2.175,
1873
+ 2.272,
1874
+ 2.875,
1875
+ 1.973,
1876
+ 1.848
1877
+ ],
1878
+ "rec.2.21": [
1879
+ 4.784,
1880
+ 2.15,
1881
+ 2.132,
1882
+ 2.272,
1883
+ 2.497,
1884
+ 2.178,
1885
+ 2.329,
1886
+ 2.538,
1887
+ 2.498,
1888
+ 2.374,
1889
+ 3.188,
1890
+ 1.843,
1891
+ 1.967,
1892
+ 2.213,
1893
+ 1.823,
1894
+ 2.166,
1895
+ 2.271,
1896
+ 2.86,
1897
+ 1.956,
1898
+ 1.865
1899
+ ],
1900
+ "rec.2.22": [
1901
+ 5.651,
1902
+ 2.081,
1903
+ 2.062,
1904
+ 2.108,
1905
+ 2.186,
1906
+ 2.097,
1907
+ 2.155,
1908
+ 2.353,
1909
+ 2.421,
1910
+ 2.211,
1911
+ 2.853,
1912
+ 1.757,
1913
+ 1.902,
1914
+ 2.169,
1915
+ 1.783,
1916
+ 2.113,
1917
+ 2.086,
1918
+ 2.691,
1919
+ 2.004,
1920
+ 1.804
1921
+ ],
1922
+ "rec.2.23": [
1923
+ 6.749,
1924
+ 1.829,
1925
+ 1.903,
1926
+ 1.862,
1927
+ 2.013,
1928
+ 1.91,
1929
+ 2.054,
1930
+ 2.097,
1931
+ 2.302,
1932
+ 2.033,
1933
+ 2.593,
1934
+ 1.692,
1935
+ 1.833,
1936
+ 1.948,
1937
+ 1.66,
1938
+ 2.057,
1939
+ 1.802,
1940
+ 2.448,
1941
+ 2.016,
1942
+ 1.661
1943
+ ],
1944
+ "rec.2.24": [
1945
+ 5.504,
1946
+ 2.01,
1947
+ 1.981,
1948
+ 2.042,
1949
+ 2.086,
1950
+ 2.018,
1951
+ 2.103,
1952
+ 2.299,
1953
+ 2.355,
1954
+ 2.161,
1955
+ 2.702,
1956
+ 1.742,
1957
+ 1.867,
1958
+ 2.111,
1959
+ 1.735,
1960
+ 2.08,
1961
+ 1.99,
1962
+ 2.626,
1963
+ 1.976,
1964
+ 1.754
1965
+ ],
1966
+ "rec.2.25": [
1967
+ 5.001,
1968
+ 1.953,
1969
+ 1.89,
1970
+ 1.904,
1971
+ 1.968,
1972
+ 1.912,
1973
+ 2.068,
1974
+ 2.114,
1975
+ 2.286,
1976
+ 2.009,
1977
+ 2.482,
1978
+ 1.713,
1979
+ 1.8,
1980
+ 1.981,
1981
+ 1.649,
1982
+ 2.009,
1983
+ 1.83,
1984
+ 2.488,
1985
+ 1.963,
1986
+ 1.683
1987
+ ],
1988
+ "rec.2.26": [
1989
+ 5.507,
1990
+ 2.008,
1991
+ 1.957,
1992
+ 2.026,
1993
+ 2.05,
1994
+ 1.988,
1995
+ 2.079,
1996
+ 2.267,
1997
+ 2.29,
1998
+ 2.131,
1999
+ 2.588,
2000
+ 1.718,
2001
+ 1.85,
2002
+ 2.092,
2003
+ 1.703,
2004
+ 2.039,
2005
+ 1.96,
2006
+ 2.615,
2007
+ 1.941,
2008
+ 1.747
2009
+ ],
2010
+ "rec.2.27": [
2011
+ 5.117,
2012
+ 2.201,
2013
+ 1.978,
2014
+ 1.986,
2015
+ 2.017,
2016
+ 1.964,
2017
+ 2.039,
2018
+ 2.181,
2019
+ 2.259,
2020
+ 2.06,
2021
+ 2.477,
2022
+ 1.703,
2023
+ 1.834,
2024
+ 2.048,
2025
+ 1.65,
2026
+ 2.007,
2027
+ 1.869,
2028
+ 2.552,
2029
+ 1.922,
2030
+ 1.724
2031
+ ],
2032
+ "rec.2.28": [
2033
+ 5.117,
2034
+ 2.004,
2035
+ 1.884,
2036
+ 1.973,
2037
+ 2.018,
2038
+ 1.916,
2039
+ 2.006,
2040
+ 2.155,
2041
+ 2.226,
2042
+ 2.059,
2043
+ 2.487,
2044
+ 1.674,
2045
+ 1.779,
2046
+ 2.046,
2047
+ 1.654,
2048
+ 1.998,
2049
+ 1.875,
2050
+ 2.555,
2051
+ 1.885,
2052
+ 1.71
2053
+ ],
2054
+ "rec.2.29": [
2055
+ 5.638,
2056
+ 2.111,
2057
+ 1.958,
2058
+ 2.004,
2059
+ 2.083,
2060
+ 1.959,
2061
+ 2.028,
2062
+ 2.161,
2063
+ 2.242,
2064
+ 2.037,
2065
+ 2.514,
2066
+ 1.684,
2067
+ 1.777,
2068
+ 2.077,
2069
+ 1.637,
2070
+ 2.006,
2071
+ 1.874,
2072
+ 2.656,
2073
+ 1.906,
2074
+ 1.721
2075
+ ],
2076
+ "rec.2.30": [
2077
+ 5.126,
2078
+ 2.038,
2079
+ 1.893,
2080
+ 1.943,
2081
+ 2.017,
2082
+ 1.897,
2083
+ 1.942,
2084
+ 2.085,
2085
+ 2.144,
2086
+ 1.962,
2087
+ 2.48,
2088
+ 1.603,
2089
+ 1.72,
2090
+ 1.979,
2091
+ 1.588,
2092
+ 1.952,
2093
+ 1.835,
2094
+ 2.622,
2095
+ 1.865,
2096
+ 1.672
2097
+ ],
2098
+ "rec.2.31": [
2099
+ 3.883,
2100
+ 1.872,
2101
+ 1.848,
2102
+ 1.794,
2103
+ 1.898,
2104
+ 1.864,
2105
+ 1.999,
2106
+ 1.927,
2107
+ 2.138,
2108
+ 1.846,
2109
+ 2.353,
2110
+ 1.628,
2111
+ 1.722,
2112
+ 1.906,
2113
+ 1.541,
2114
+ 1.96,
2115
+ 1.705,
2116
+ 2.366,
2117
+ 1.932,
2118
+ 1.596
2119
+ ],
2120
+ "rec.2.32": [
2121
+ 5.039,
2122
+ 2.098,
2123
+ 1.951,
2124
+ 1.965,
2125
+ 2.111,
2126
+ 1.937,
2127
+ 2.023,
2128
+ 2.11,
2129
+ 2.171,
2130
+ 1.969,
2131
+ 2.582,
2132
+ 1.606,
2133
+ 1.735,
2134
+ 1.97,
2135
+ 1.562,
2136
+ 1.974,
2137
+ 1.812,
2138
+ 2.575,
2139
+ 1.895,
2140
+ 1.671
2141
+ ],
2142
+ "rec.2.33": [
2143
+ 3.672,
2144
+ 1.882,
2145
+ 1.861,
2146
+ 1.818,
2147
+ 1.939,
2148
+ 1.878,
2149
+ 2.029,
2150
+ 1.946,
2151
+ 2.076,
2152
+ 1.888,
2153
+ 2.366,
2154
+ 1.582,
2155
+ 1.719,
2156
+ 1.886,
2157
+ 1.527,
2158
+ 1.92,
2159
+ 1.647,
2160
+ 2.417,
2161
+ 1.882,
2162
+ 1.581
2163
+ ],
2164
+ "rec.2.34": [
2165
+ 4.002,
2166
+ 2.058,
2167
+ 1.957,
2168
+ 1.92,
2169
+ 2.135,
2170
+ 1.931,
2171
+ 2.05,
2172
+ 2.038,
2173
+ 2.159,
2174
+ 1.945,
2175
+ 2.597,
2176
+ 1.583,
2177
+ 1.737,
2178
+ 1.929,
2179
+ 1.516,
2180
+ 1.935,
2181
+ 1.737,
2182
+ 2.6,
2183
+ 1.89,
2184
+ 1.631
2185
+ ],
2186
+ "rec.2.35": [
2187
+ 4.01,
2188
+ 2.054,
2189
+ 2.158,
2190
+ 1.875,
2191
+ 2.304,
2192
+ 2.052,
2193
+ 2.255,
2194
+ 2.08,
2195
+ 2.317,
2196
+ 1.932,
2197
+ 3.049,
2198
+ 1.664,
2199
+ 1.809,
2200
+ 1.899,
2201
+ 1.579,
2202
+ 1.997,
2203
+ 1.74,
2204
+ 2.548,
2205
+ 1.963,
2206
+ 1.663
2207
+ ],
2208
+ "rec.2.36": [
2209
+ 4.031,
2210
+ 1.913,
2211
+ 1.884,
2212
+ 1.95,
2213
+ 1.814,
2214
+ 1.888,
2215
+ 1.899,
2216
+ 2.05,
2217
+ 2.108,
2218
+ 1.899,
2219
+ 2.405,
2220
+ 1.54,
2221
+ 1.737,
2222
+ 1.887,
2223
+ 1.464,
2224
+ 1.928,
2225
+ 1.733,
2226
+ 2.674,
2227
+ 1.861,
2228
+ 1.612
2229
+ ],
2230
+ "rec.2.37": [
2231
+ 4.898,
2232
+ 2.001,
2233
+ 1.995,
2234
+ 2.002,
2235
+ 1.877,
2236
+ 1.979,
2237
+ 1.964,
2238
+ 2.058,
2239
+ 2.198,
2240
+ 1.893,
2241
+ 2.473,
2242
+ 1.586,
2243
+ 1.793,
2244
+ 1.893,
2245
+ 1.492,
2246
+ 1.963,
2247
+ 1.717,
2248
+ 2.488,
2249
+ 1.888,
2250
+ 1.627
2251
+ ],
2252
+ "rec.2.38": [
2253
+ 4.049,
2254
+ 1.747,
2255
+ 1.758,
2256
+ 1.811,
2257
+ 1.644,
2258
+ 1.813,
2259
+ 1.787,
2260
+ 1.974,
2261
+ 2.015,
2262
+ 1.848,
2263
+ 2.176,
2264
+ 1.504,
2265
+ 1.689,
2266
+ 1.84,
2267
+ 1.408,
2268
+ 1.868,
2269
+ 1.678,
2270
+ 2.425,
2271
+ 1.801,
2272
+ 1.583
2273
+ ],
2274
+ "rec.2.39": [
2275
+ 4.729,
2276
+ 1.764,
2277
+ 1.8,
2278
+ 1.837,
2279
+ 1.684,
2280
+ 1.856,
2281
+ 1.798,
2282
+ 1.983,
2283
+ 2.035,
2284
+ 1.854,
2285
+ 2.162,
2286
+ 1.536,
2287
+ 1.73,
2288
+ 1.833,
2289
+ 1.426,
2290
+ 1.871,
2291
+ 1.676,
2292
+ 2.415,
2293
+ 1.803,
2294
+ 1.588
2295
+ ],
2296
+ "rec.2.40": [
2297
+ 3.443,
2298
+ 1.698,
2299
+ 1.726,
2300
+ 1.809,
2301
+ 1.574,
2302
+ 1.78,
2303
+ 1.746,
2304
+ 2.007,
2305
+ 1.987,
2306
+ 1.84,
2307
+ 2.134,
2308
+ 1.508,
2309
+ 1.716,
2310
+ 1.778,
2311
+ 1.385,
2312
+ 1.829,
2313
+ 1.677,
2314
+ 2.357,
2315
+ 1.806,
2316
+ 1.586
2317
+ ],
2318
+ "rec.2.41": [
2319
+ 3.491,
2320
+ 1.659,
2321
+ 1.703,
2322
+ 1.796,
2323
+ 1.531,
2324
+ 1.758,
2325
+ 1.735,
2326
+ 1.969,
2327
+ 1.95,
2328
+ 1.816,
2329
+ 2.081,
2330
+ 1.492,
2331
+ 1.697,
2332
+ 1.748,
2333
+ 1.371,
2334
+ 1.823,
2335
+ 1.646,
2336
+ 2.336,
2337
+ 1.785,
2338
+ 1.567
2339
+ ],
2340
+ "rec.2.42": [
2341
+ 3.189,
2342
+ 1.674,
2343
+ 1.701,
2344
+ 1.807,
2345
+ 1.563,
2346
+ 1.774,
2347
+ 1.736,
2348
+ 2.015,
2349
+ 1.986,
2350
+ 1.838,
2351
+ 2.132,
2352
+ 1.492,
2353
+ 1.7,
2354
+ 1.753,
2355
+ 1.366,
2356
+ 1.806,
2357
+ 1.672,
2358
+ 2.325,
2359
+ 1.784,
2360
+ 1.591
2361
+ ],
2362
+ "rec.2.43": [
2363
+ 3.792,
2364
+ 1.759,
2365
+ 1.774,
2366
+ 1.862,
2367
+ 1.586,
2368
+ 1.802,
2369
+ 1.772,
2370
+ 1.984,
2371
+ 1.992,
2372
+ 1.881,
2373
+ 2.087,
2374
+ 1.528,
2375
+ 1.715,
2376
+ 1.752,
2377
+ 1.38,
2378
+ 1.838,
2379
+ 1.617,
2380
+ 2.241,
2381
+ 1.812,
2382
+ 1.575
2383
+ ],
2384
+ "rec.2.44": [
2385
+ 3.575,
2386
+ 1.681,
2387
+ 1.702,
2388
+ 1.817,
2389
+ 1.576,
2390
+ 1.777,
2391
+ 1.739,
2392
+ 2.021,
2393
+ 1.963,
2394
+ 1.862,
2395
+ 2.161,
2396
+ 1.496,
2397
+ 1.709,
2398
+ 1.758,
2399
+ 1.356,
2400
+ 1.82,
2401
+ 1.66,
2402
+ 2.329,
2403
+ 1.795,
2404
+ 1.583
2405
+ ],
2406
+ "rec.2.45": [
2407
+ 3.642,
2408
+ 1.697,
2409
+ 1.719,
2410
+ 1.861,
2411
+ 1.588,
2412
+ 1.801,
2413
+ 1.769,
2414
+ 2.051,
2415
+ 1.969,
2416
+ 1.888,
2417
+ 2.183,
2418
+ 1.501,
2419
+ 1.753,
2420
+ 1.766,
2421
+ 1.353,
2422
+ 1.84,
2423
+ 1.647,
2424
+ 2.278,
2425
+ 1.819,
2426
+ 1.605
2427
+ ],
2428
+ "rec.2.46": [
2429
+ 3.343,
2430
+ 1.68,
2431
+ 1.736,
2432
+ 1.768,
2433
+ 1.562,
2434
+ 1.761,
2435
+ 1.72,
2436
+ 1.997,
2437
+ 1.955,
2438
+ 1.836,
2439
+ 2.125,
2440
+ 1.481,
2441
+ 1.711,
2442
+ 1.702,
2443
+ 1.34,
2444
+ 1.808,
2445
+ 1.569,
2446
+ 2.243,
2447
+ 1.771,
2448
+ 1.565
2449
+ ],
2450
+ "rec.2.47": [
2451
+ 2.869,
2452
+ 1.678,
2453
+ 1.731,
2454
+ 1.779,
2455
+ 1.566,
2456
+ 1.752,
2457
+ 1.736,
2458
+ 2.01,
2459
+ 1.934,
2460
+ 1.857,
2461
+ 2.088,
2462
+ 1.487,
2463
+ 1.693,
2464
+ 1.712,
2465
+ 1.336,
2466
+ 1.803,
2467
+ 1.557,
2468
+ 2.177,
2469
+ 1.76,
2470
+ 1.54
2471
+ ],
2472
+ "rec.2.48": [
2473
+ 2.64,
2474
+ 1.711,
2475
+ 1.723,
2476
+ 1.608,
2477
+ 1.61,
2478
+ 1.744,
2479
+ 1.675,
2480
+ 1.926,
2481
+ 1.943,
2482
+ 1.764,
2483
+ 2.044,
2484
+ 1.499,
2485
+ 1.647,
2486
+ 1.665,
2487
+ 1.35,
2488
+ 1.697,
2489
+ 1.48,
2490
+ 1.978,
2491
+ 1.702,
2492
+ 1.517
2493
+ ],
2494
+ "rec.2.49": [
2495
+ 2.87,
2496
+ 1.69,
2497
+ 1.704,
2498
+ 1.635,
2499
+ 1.538,
2500
+ 1.693,
2501
+ 1.673,
2502
+ 1.887,
2503
+ 1.905,
2504
+ 1.741,
2505
+ 1.915,
2506
+ 1.497,
2507
+ 1.648,
2508
+ 1.676,
2509
+ 1.352,
2510
+ 1.7,
2511
+ 1.478,
2512
+ 1.911,
2513
+ 1.674,
2514
+ 1.498
2515
+ ],
2516
+ "rec.2.50": [
2517
+ 2.675,
2518
+ 1.682,
2519
+ 1.696,
2520
+ 1.592,
2521
+ 1.62,
2522
+ 1.722,
2523
+ 1.64,
2524
+ 1.885,
2525
+ 1.908,
2526
+ 1.737,
2527
+ 1.998,
2528
+ 1.47,
2529
+ 1.581,
2530
+ 1.646,
2531
+ 1.32,
2532
+ 1.662,
2533
+ 1.44,
2534
+ 1.909,
2535
+ 1.657,
2536
+ 1.492
2537
+ ],
2538
+ "rec.2.51": [
2539
+ 2.859,
2540
+ 1.725,
2541
+ 1.813,
2542
+ 1.661,
2543
+ 1.598,
2544
+ 1.727,
2545
+ 1.679,
2546
+ 1.952,
2547
+ 1.96,
2548
+ 1.78,
2549
+ 2.084,
2550
+ 1.51,
2551
+ 1.636,
2552
+ 1.693,
2553
+ 1.364,
2554
+ 1.753,
2555
+ 1.457,
2556
+ 1.893,
2557
+ 1.671,
2558
+ 1.533
2559
+ ],
2560
+ "rec.2.52": [
2561
+ 3.138,
2562
+ 1.679,
2563
+ 1.656,
2564
+ 1.579,
2565
+ 1.579,
2566
+ 1.702,
2567
+ 1.601,
2568
+ 1.943,
2569
+ 1.909,
2570
+ 1.765,
2571
+ 1.922,
2572
+ 1.447,
2573
+ 1.565,
2574
+ 1.65,
2575
+ 1.302,
2576
+ 1.656,
2577
+ 1.425,
2578
+ 1.858,
2579
+ 1.62,
2580
+ 1.534
2581
+ ],
2582
+ "rec.2.53": [
2583
+ 7.666,
2584
+ 1.783,
2585
+ 1.766,
2586
+ 1.709,
2587
+ 1.649,
2588
+ 1.774,
2589
+ 1.616,
2590
+ 2.032,
2591
+ 1.981,
2592
+ 1.875,
2593
+ 1.969,
2594
+ 1.501,
2595
+ 1.561,
2596
+ 1.673,
2597
+ 1.401,
2598
+ 1.702,
2599
+ 1.41,
2600
+ 1.817,
2601
+ 1.619,
2602
+ 1.578
2603
+ ],
2604
+ "rec.2.54": [
2605
+ 3.104,
2606
+ 1.579,
2607
+ 1.506,
2608
+ 1.368,
2609
+ 1.561,
2610
+ 1.553,
2611
+ 1.403,
2612
+ 1.511,
2613
+ 1.668,
2614
+ 1.451,
2615
+ 1.585,
2616
+ 1.386,
2617
+ 1.317,
2618
+ 1.448,
2619
+ 1.222,
2620
+ 1.358,
2621
+ 1.305,
2622
+ 1.524,
2623
+ 1.422,
2624
+ 1.288
2625
+ ],
2626
+ "s.2": [
2627
+ 2.931,
2628
+ 1.525,
2629
+ 1.489,
2630
+ 1.396,
2631
+ 1.426,
2632
+ 1.616,
2633
+ 1.445,
2634
+ 1.547,
2635
+ 1.715,
2636
+ 1.469,
2637
+ 1.537,
2638
+ 1.42,
2639
+ 1.362,
2640
+ 1.444,
2641
+ 1.273,
2642
+ 1.463,
2643
+ 1.318,
2644
+ 1.591,
2645
+ 1.451,
2646
+ 1.274
2647
+ ],
2648
+ "out.56": [
2649
+ 2.363,
2650
+ 1.486,
2651
+ 1.46,
2652
+ 1.357,
2653
+ 1.51,
2654
+ 1.463,
2655
+ 1.339,
2656
+ 1.499,
2657
+ 1.592,
2658
+ 1.478,
2659
+ 1.434,
2660
+ 1.361,
2661
+ 1.299,
2662
+ 1.436,
2663
+ 1.264,
2664
+ 1.349,
2665
+ 1.284,
2666
+ 1.456,
2667
+ 1.401,
2668
+ 1.313
2669
+ ],
2670
+ "out.57": [
2671
+ 2.539,
2672
+ 1.51,
2673
+ 1.498,
2674
+ 1.399,
2675
+ 1.526,
2676
+ 1.5,
2677
+ 1.376,
2678
+ 1.532,
2679
+ 1.618,
2680
+ 1.493,
2681
+ 1.455,
2682
+ 1.389,
2683
+ 1.311,
2684
+ 1.438,
2685
+ 1.315,
2686
+ 1.397,
2687
+ 1.293,
2688
+ 1.475,
2689
+ 1.426,
2690
+ 1.313
2691
+ ],
2692
+ "out.58": [
2693
+ 2.475,
2694
+ 1.473,
2695
+ 1.452,
2696
+ 1.356,
2697
+ 1.513,
2698
+ 1.476,
2699
+ 1.337,
2700
+ 1.501,
2701
+ 1.61,
2702
+ 1.492,
2703
+ 1.445,
2704
+ 1.377,
2705
+ 1.268,
2706
+ 1.396,
2707
+ 1.341,
2708
+ 1.373,
2709
+ 1.256,
2710
+ 1.421,
2711
+ 1.397,
2712
+ 1.291
2713
+ ],
2714
+ "out.59": [
2715
+ 2.789,
2716
+ 1.495,
2717
+ 1.498,
2718
+ 1.41,
2719
+ 1.49,
2720
+ 1.501,
2721
+ 1.351,
2722
+ 1.533,
2723
+ 1.609,
2724
+ 1.509,
2725
+ 1.502,
2726
+ 1.407,
2727
+ 1.279,
2728
+ 1.399,
2729
+ 1.42,
2730
+ 1.409,
2731
+ 1.268,
2732
+ 1.435,
2733
+ 1.402,
2734
+ 1.282
2735
+ ],
2736
+ "out.60": [
2737
+ 2.567,
2738
+ 1.323,
2739
+ 1.311,
2740
+ 1.345,
2741
+ 1.278,
2742
+ 1.347,
2743
+ 1.265,
2744
+ 1.403,
2745
+ 1.474,
2746
+ 1.382,
2747
+ 1.449,
2748
+ 1.305,
2749
+ 1.175,
2750
+ 1.258,
2751
+ 1.387,
2752
+ 1.326,
2753
+ 1.161,
2754
+ 1.267,
2755
+ 1.281,
2756
+ 1.191
2757
+ ],
2758
+ "out.61": [
2759
+ 7.036,
2760
+ 1.585,
2761
+ 1.637,
2762
+ 1.893,
2763
+ 1.231,
2764
+ 1.565,
2765
+ 1.357,
2766
+ 1.748,
2767
+ 1.595,
2768
+ 1.774,
2769
+ 2.118,
2770
+ 1.617,
2771
+ 1.157,
2772
+ 1.24,
2773
+ 2.122,
2774
+ 1.531,
2775
+ 1.181,
2776
+ 1.275,
2777
+ 1.261,
2778
+ 1.145
2779
+ ],
2780
+ "out.62": [
2781
+ 1.267,
2782
+ 0.945,
2783
+ 0.951,
2784
+ 0.978,
2785
+ 1.017,
2786
+ 1.052,
2787
+ 1.071,
2788
+ 1.086,
2789
+ 1.116,
2790
+ 1.121,
2791
+ 1.104,
2792
+ 1.031,
2793
+ 1.082,
2794
+ 1.098,
2795
+ 1.046,
2796
+ 1.109,
2797
+ 1.045,
2798
+ 1.093,
2799
+ 1.135,
2800
+ 1.079
2801
+ ],
2802
+ "out.63": [
2803
+ 1.311,
2804
+ 0.964,
2805
+ 0.988,
2806
+ 1.004,
2807
+ 1.029,
2808
+ 1.07,
2809
+ 1.112,
2810
+ 1.114,
2811
+ 1.133,
2812
+ 1.119,
2813
+ 1.126,
2814
+ 1.064,
2815
+ 1.134,
2816
+ 1.115,
2817
+ 1.091,
2818
+ 1.124,
2819
+ 1.078,
2820
+ 1.119,
2821
+ 1.157,
2822
+ 1.085
2823
+ ],
2824
+ "out.64": [
2825
+ 1.338,
2826
+ 0.936,
2827
+ 0.952,
2828
+ 0.999,
2829
+ 1.001,
2830
+ 1.057,
2831
+ 1.084,
2832
+ 1.089,
2833
+ 1.104,
2834
+ 1.112,
2835
+ 1.124,
2836
+ 1.052,
2837
+ 1.086,
2838
+ 1.096,
2839
+ 1.082,
2840
+ 1.108,
2841
+ 1.04,
2842
+ 1.088,
2843
+ 1.152,
2844
+ 1.064
2845
+ ],
2846
+ "out.65": [
2847
+ 4.348,
2848
+ 1.494,
2849
+ 1.505,
2850
+ 1.928,
2851
+ 0.954,
2852
+ 1.488,
2853
+ 1.619,
2854
+ 1.598,
2855
+ 1.495,
2856
+ 1.54,
2857
+ 2.064,
2858
+ 1.709,
2859
+ 1.404,
2860
+ 1.411,
2861
+ 2.104,
2862
+ 1.488,
2863
+ 1.236,
2864
+ 1.074,
2865
+ 1.417,
2866
+ 1.036
2867
+ ],
2868
+ "out.66": [
2869
+ 1.395,
2870
+ 0.952,
2871
+ 0.966,
2872
+ 1.01,
2873
+ 0.983,
2874
+ 1.04,
2875
+ 1.054,
2876
+ 1.062,
2877
+ 1.088,
2878
+ 1.096,
2879
+ 1.113,
2880
+ 1.053,
2881
+ 1.089,
2882
+ 1.065,
2883
+ 1.087,
2884
+ 1.094,
2885
+ 1.036,
2886
+ 1.053,
2887
+ 1.105,
2888
+ 1.048
2889
+ ],
2890
+ "out.67": [
2891
+ 3.275,
2892
+ 1.347,
2893
+ 1.365,
2894
+ 1.693,
2895
+ 0.983,
2896
+ 1.346,
2897
+ 1.473,
2898
+ 1.446,
2899
+ 1.43,
2900
+ 1.381,
2901
+ 1.773,
2902
+ 1.53,
2903
+ 1.317,
2904
+ 1.301,
2905
+ 1.819,
2906
+ 1.396,
2907
+ 1.194,
2908
+ 1.049,
2909
+ 1.293,
2910
+ 1.029
2911
+ ],
2912
+ "out.68": [
2913
+ 1.488,
2914
+ 0.941,
2915
+ 0.942,
2916
+ 1.03,
2917
+ 0.95,
2918
+ 1.001,
2919
+ 0.998,
2920
+ 0.999,
2921
+ 1.012,
2922
+ 1.027,
2923
+ 1.086,
2924
+ 1.041,
2925
+ 0.974,
2926
+ 0.967,
2927
+ 1.081,
2928
+ 1.029,
2929
+ 0.953,
2930
+ 0.933,
2931
+ 0.993,
2932
+ 0.973
2933
+ ],
2934
+ "out.69": [
2935
+ 3.467,
2936
+ 1.422,
2937
+ 1.498,
2938
+ 1.936,
2939
+ 1.005,
2940
+ 1.442,
2941
+ 1.601,
2942
+ 1.514,
2943
+ 1.572,
2944
+ 1.423,
2945
+ 1.957,
2946
+ 1.818,
2947
+ 1.364,
2948
+ 1.273,
2949
+ 2.009,
2950
+ 1.574,
2951
+ 1.131,
2952
+ 0.961,
2953
+ 1.282,
2954
+ 0.963
2955
+ ],
2956
+ "out.70": [
2957
+ 0.962,
2958
+ 0.935,
2959
+ 0.941,
2960
+ 0.97,
2961
+ 0.971,
2962
+ 1.001,
2963
+ 0.971,
2964
+ 0.987,
2965
+ 0.983,
2966
+ 1.002,
2967
+ 0.999,
2968
+ 0.981,
2969
+ 0.96,
2970
+ 0.986,
2971
+ 0.979,
2972
+ 1.003,
2973
+ 0.976,
2974
+ 0.98,
2975
+ 0.973,
2976
+ 0.979
2977
+ ]
2978
+ },
2979
+ "heldout": {
2980
+ "emb": {
2981
+ "top1_agree_model_skip16": 0.001,
2982
+ "top1_agree_model_skip4": 0.001,
2983
+ "top1_agree_model_skip0": 0.0,
2984
+ "top1_agree_model_logit_lens": 0.002,
2985
+ "nonsemantic_share_top10_skip16": 0.595,
2986
+ "nonsemantic_share_top10_skip0": 0.549,
2987
+ "top10_jaccard_skip0_vs_16": 0.304,
2988
+ "top10_jaccard_skip4_vs_16": 0.361
2989
+ },
2990
+ "in.00": {
2991
+ "top1_agree_model_skip16": 0.0,
2992
+ "top1_agree_model_skip4": 0.0,
2993
+ "top1_agree_model_skip0": 0.0,
2994
+ "top1_agree_model_logit_lens": 0.001,
2995
+ "nonsemantic_share_top10_skip16": 0.671,
2996
+ "nonsemantic_share_top10_skip0": 0.667,
2997
+ "top10_jaccard_skip0_vs_16": 0.309,
2998
+ "top10_jaccard_skip4_vs_16": 0.346
2999
+ },
3000
+ "in.01": {
3001
+ "top1_agree_model_skip16": 0.0,
3002
+ "top1_agree_model_skip4": 0.0,
3003
+ "top1_agree_model_skip0": 0.0,
3004
+ "top1_agree_model_logit_lens": 0.0,
3005
+ "nonsemantic_share_top10_skip16": 0.607,
3006
+ "nonsemantic_share_top10_skip0": 0.578,
3007
+ "top10_jaccard_skip0_vs_16": 0.377,
3008
+ "top10_jaccard_skip4_vs_16": 0.403
3009
+ },
3010
+ "in.02": {
3011
+ "top1_agree_model_skip16": 0.0,
3012
+ "top1_agree_model_skip4": 0.0,
3013
+ "top1_agree_model_skip0": 0.0,
3014
+ "top1_agree_model_logit_lens": 0.0,
3015
+ "nonsemantic_share_top10_skip16": 0.669,
3016
+ "nonsemantic_share_top10_skip0": 0.672,
3017
+ "top10_jaccard_skip0_vs_16": 0.442,
3018
+ "top10_jaccard_skip4_vs_16": 0.485
3019
+ },
3020
+ "in.03": {
3021
+ "top1_agree_model_skip16": 0.001,
3022
+ "top1_agree_model_skip4": 0.0,
3023
+ "top1_agree_model_skip0": 0.0,
3024
+ "top1_agree_model_logit_lens": 0.001,
3025
+ "nonsemantic_share_top10_skip16": 0.612,
3026
+ "nonsemantic_share_top10_skip0": 0.6,
3027
+ "top10_jaccard_skip0_vs_16": 0.434,
3028
+ "top10_jaccard_skip4_vs_16": 0.477
3029
+ },
3030
+ "in.04": {
3031
+ "top1_agree_model_skip16": 0.0,
3032
+ "top1_agree_model_skip4": 0.0,
3033
+ "top1_agree_model_skip0": 0.0,
3034
+ "top1_agree_model_logit_lens": 0.0,
3035
+ "nonsemantic_share_top10_skip16": 0.683,
3036
+ "nonsemantic_share_top10_skip0": 0.62,
3037
+ "top10_jaccard_skip0_vs_16": 0.48,
3038
+ "top10_jaccard_skip4_vs_16": 0.538
3039
+ },
3040
+ "in.05": {
3041
+ "top1_agree_model_skip16": 0.001,
3042
+ "top1_agree_model_skip4": 0.0,
3043
+ "top1_agree_model_skip0": 0.0,
3044
+ "top1_agree_model_logit_lens": 0.0,
3045
+ "nonsemantic_share_top10_skip16": 0.664,
3046
+ "nonsemantic_share_top10_skip0": 0.668,
3047
+ "top10_jaccard_skip0_vs_16": 0.52,
3048
+ "top10_jaccard_skip4_vs_16": 0.591
3049
+ },
3050
+ "in.06": {
3051
+ "top1_agree_model_skip16": 0.001,
3052
+ "top1_agree_model_skip4": 0.0,
3053
+ "top1_agree_model_skip0": 0.0,
3054
+ "top1_agree_model_logit_lens": 0.0,
3055
+ "nonsemantic_share_top10_skip16": 0.684,
3056
+ "nonsemantic_share_top10_skip0": 0.68,
3057
+ "top10_jaccard_skip0_vs_16": 0.553,
3058
+ "top10_jaccard_skip4_vs_16": 0.609
3059
+ },
3060
+ "in.07": {
3061
+ "top1_agree_model_skip16": 0.003,
3062
+ "top1_agree_model_skip4": 0.001,
3063
+ "top1_agree_model_skip0": 0.001,
3064
+ "top1_agree_model_logit_lens": 0.001,
3065
+ "nonsemantic_share_top10_skip16": 0.668,
3066
+ "nonsemantic_share_top10_skip0": 0.709,
3067
+ "top10_jaccard_skip0_vs_16": 0.554,
3068
+ "top10_jaccard_skip4_vs_16": 0.622
3069
+ },
3070
+ "in.08": {
3071
+ "top1_agree_model_skip16": 0.001,
3072
+ "top1_agree_model_skip4": 0.0,
3073
+ "top1_agree_model_skip0": 0.001,
3074
+ "top1_agree_model_logit_lens": 0.001,
3075
+ "nonsemantic_share_top10_skip16": 0.728,
3076
+ "nonsemantic_share_top10_skip0": 0.752,
3077
+ "top10_jaccard_skip0_vs_16": 0.557,
3078
+ "top10_jaccard_skip4_vs_16": 0.622
3079
+ },
3080
+ "in.09": {
3081
+ "top1_agree_model_skip16": 0.001,
3082
+ "top1_agree_model_skip4": 0.001,
3083
+ "top1_agree_model_skip0": 0.001,
3084
+ "top1_agree_model_logit_lens": 0.0,
3085
+ "nonsemantic_share_top10_skip16": 0.698,
3086
+ "nonsemantic_share_top10_skip0": 0.672,
3087
+ "top10_jaccard_skip0_vs_16": 0.562,
3088
+ "top10_jaccard_skip4_vs_16": 0.633
3089
+ },
3090
+ "in.10": {
3091
+ "top1_agree_model_skip16": 0.001,
3092
+ "top1_agree_model_skip4": 0.0,
3093
+ "top1_agree_model_skip0": 0.001,
3094
+ "top1_agree_model_logit_lens": 0.0,
3095
+ "nonsemantic_share_top10_skip16": 0.626,
3096
+ "nonsemantic_share_top10_skip0": 0.608,
3097
+ "top10_jaccard_skip0_vs_16": 0.557,
3098
+ "top10_jaccard_skip4_vs_16": 0.608
3099
+ },
3100
+ "in.11": {
3101
+ "top1_agree_model_skip16": 0.004,
3102
+ "top1_agree_model_skip4": 0.001,
3103
+ "top1_agree_model_skip0": 0.0,
3104
+ "top1_agree_model_logit_lens": 0.002,
3105
+ "nonsemantic_share_top10_skip16": 0.592,
3106
+ "nonsemantic_share_top10_skip0": 0.55,
3107
+ "top10_jaccard_skip0_vs_16": 0.519,
3108
+ "top10_jaccard_skip4_vs_16": 0.595
3109
+ },
3110
+ "in.12": {
3111
+ "top1_agree_model_skip16": 0.006,
3112
+ "top1_agree_model_skip4": 0.0,
3113
+ "top1_agree_model_skip0": 0.0,
3114
+ "top1_agree_model_logit_lens": 0.0,
3115
+ "nonsemantic_share_top10_skip16": 0.656,
3116
+ "nonsemantic_share_top10_skip0": 0.596,
3117
+ "top10_jaccard_skip0_vs_16": 0.542,
3118
+ "top10_jaccard_skip4_vs_16": 0.619
3119
+ },
3120
+ "in.13": {
3121
+ "top1_agree_model_skip16": 0.016,
3122
+ "top1_agree_model_skip4": 0.006,
3123
+ "top1_agree_model_skip0": 0.004,
3124
+ "top1_agree_model_logit_lens": 0.0,
3125
+ "nonsemantic_share_top10_skip16": 0.667,
3126
+ "nonsemantic_share_top10_skip0": 0.595,
3127
+ "top10_jaccard_skip0_vs_16": 0.481,
3128
+ "top10_jaccard_skip4_vs_16": 0.609
3129
+ },
3130
+ "in.14": {
3131
+ "top1_agree_model_skip16": 0.018,
3132
+ "top1_agree_model_skip4": 0.009,
3133
+ "top1_agree_model_skip0": 0.003,
3134
+ "top1_agree_model_logit_lens": 0.001,
3135
+ "nonsemantic_share_top10_skip16": 0.432,
3136
+ "nonsemantic_share_top10_skip0": 0.417,
3137
+ "top10_jaccard_skip0_vs_16": 0.52,
3138
+ "top10_jaccard_skip4_vs_16": 0.58
3139
+ },
3140
+ "x": {
3141
+ "top1_agree_model_skip16": 0.019,
3142
+ "top1_agree_model_skip4": 0.011,
3143
+ "top1_agree_model_skip0": 0.009,
3144
+ "top1_agree_model_logit_lens": 0.001,
3145
+ "nonsemantic_share_top10_skip16": 0.297,
3146
+ "nonsemantic_share_top10_skip0": 0.355,
3147
+ "top10_jaccard_skip0_vs_16": 0.605,
3148
+ "top10_jaccard_skip4_vs_16": 0.669
3149
+ },
3150
+ "u.1": {
3151
+ "top1_agree_model_skip16": 0.0,
3152
+ "top1_agree_model_skip4": 0.0,
3153
+ "top1_agree_model_skip0": 0.0,
3154
+ "top1_agree_model_logit_lens": 0.001,
3155
+ "nonsemantic_share_top10_skip16": 0.054,
3156
+ "nonsemantic_share_top10_skip0": 0.108,
3157
+ "top10_jaccard_skip0_vs_16": 0.625,
3158
+ "top10_jaccard_skip4_vs_16": 0.655
3159
+ },
3160
+ "rec.1.16": {
3161
+ "top1_agree_model_skip16": 0.011,
3162
+ "top1_agree_model_skip4": 0.006,
3163
+ "top1_agree_model_skip0": 0.008,
3164
+ "top1_agree_model_logit_lens": 0.003,
3165
+ "nonsemantic_share_top10_skip16": 0.139,
3166
+ "nonsemantic_share_top10_skip0": 0.122,
3167
+ "top10_jaccard_skip0_vs_16": 0.318,
3168
+ "top10_jaccard_skip4_vs_16": 0.317
3169
+ },
3170
+ "rec.1.17": {
3171
+ "top1_agree_model_skip16": 0.011,
3172
+ "top1_agree_model_skip4": 0.013,
3173
+ "top1_agree_model_skip0": 0.009,
3174
+ "top1_agree_model_logit_lens": 0.0,
3175
+ "nonsemantic_share_top10_skip16": 0.185,
3176
+ "nonsemantic_share_top10_skip0": 0.264,
3177
+ "top10_jaccard_skip0_vs_16": 0.366,
3178
+ "top10_jaccard_skip4_vs_16": 0.425
3179
+ },
3180
+ "rec.1.18": {
3181
+ "top1_agree_model_skip16": 0.013,
3182
+ "top1_agree_model_skip4": 0.011,
3183
+ "top1_agree_model_skip0": 0.009,
3184
+ "top1_agree_model_logit_lens": 0.002,
3185
+ "nonsemantic_share_top10_skip16": 0.278,
3186
+ "nonsemantic_share_top10_skip0": 0.299,
3187
+ "top10_jaccard_skip0_vs_16": 0.385,
3188
+ "top10_jaccard_skip4_vs_16": 0.416
3189
+ },
3190
+ "rec.1.19": {
3191
+ "top1_agree_model_skip16": 0.013,
3192
+ "top1_agree_model_skip4": 0.01,
3193
+ "top1_agree_model_skip0": 0.003,
3194
+ "top1_agree_model_logit_lens": 0.001,
3195
+ "nonsemantic_share_top10_skip16": 0.107,
3196
+ "nonsemantic_share_top10_skip0": 0.269,
3197
+ "top10_jaccard_skip0_vs_16": 0.226,
3198
+ "top10_jaccard_skip4_vs_16": 0.328
3199
+ },
3200
+ "rec.1.20": {
3201
+ "top1_agree_model_skip16": 0.01,
3202
+ "top1_agree_model_skip4": 0.009,
3203
+ "top1_agree_model_skip0": 0.006,
3204
+ "top1_agree_model_logit_lens": 0.002,
3205
+ "nonsemantic_share_top10_skip16": 0.263,
3206
+ "nonsemantic_share_top10_skip0": 0.344,
3207
+ "top10_jaccard_skip0_vs_16": 0.348,
3208
+ "top10_jaccard_skip4_vs_16": 0.403
3209
+ },
3210
+ "rec.1.21": {
3211
+ "top1_agree_model_skip16": 0.009,
3212
+ "top1_agree_model_skip4": 0.006,
3213
+ "top1_agree_model_skip0": 0.002,
3214
+ "top1_agree_model_logit_lens": 0.003,
3215
+ "nonsemantic_share_top10_skip16": 0.337,
3216
+ "nonsemantic_share_top10_skip0": 0.379,
3217
+ "top10_jaccard_skip0_vs_16": 0.386,
3218
+ "top10_jaccard_skip4_vs_16": 0.392
3219
+ },
3220
+ "rec.1.22": {
3221
+ "top1_agree_model_skip16": 0.01,
3222
+ "top1_agree_model_skip4": 0.003,
3223
+ "top1_agree_model_skip0": 0.003,
3224
+ "top1_agree_model_logit_lens": 0.001,
3225
+ "nonsemantic_share_top10_skip16": 0.352,
3226
+ "nonsemantic_share_top10_skip0": 0.436,
3227
+ "top10_jaccard_skip0_vs_16": 0.369,
3228
+ "top10_jaccard_skip4_vs_16": 0.423
3229
+ },
3230
+ "rec.1.23": {
3231
+ "top1_agree_model_skip16": 0.008,
3232
+ "top1_agree_model_skip4": 0.006,
3233
+ "top1_agree_model_skip0": 0.002,
3234
+ "top1_agree_model_logit_lens": 0.0,
3235
+ "nonsemantic_share_top10_skip16": 0.282,
3236
+ "nonsemantic_share_top10_skip0": 0.317,
3237
+ "top10_jaccard_skip0_vs_16": 0.402,
3238
+ "top10_jaccard_skip4_vs_16": 0.456
3239
+ },
3240
+ "rec.1.24": {
3241
+ "top1_agree_model_skip16": 0.015,
3242
+ "top1_agree_model_skip4": 0.007,
3243
+ "top1_agree_model_skip0": 0.006,
3244
+ "top1_agree_model_logit_lens": 0.0,
3245
+ "nonsemantic_share_top10_skip16": 0.332,
3246
+ "nonsemantic_share_top10_skip0": 0.386,
3247
+ "top10_jaccard_skip0_vs_16": 0.433,
3248
+ "top10_jaccard_skip4_vs_16": 0.485
3249
+ },
3250
+ "rec.1.25": {
3251
+ "top1_agree_model_skip16": 0.012,
3252
+ "top1_agree_model_skip4": 0.007,
3253
+ "top1_agree_model_skip0": 0.003,
3254
+ "top1_agree_model_logit_lens": 0.0,
3255
+ "nonsemantic_share_top10_skip16": 0.277,
3256
+ "nonsemantic_share_top10_skip0": 0.313,
3257
+ "top10_jaccard_skip0_vs_16": 0.452,
3258
+ "top10_jaccard_skip4_vs_16": 0.501
3259
+ },
3260
+ "rec.1.26": {
3261
+ "top1_agree_model_skip16": 0.015,
3262
+ "top1_agree_model_skip4": 0.007,
3263
+ "top1_agree_model_skip0": 0.006,
3264
+ "top1_agree_model_logit_lens": 0.0,
3265
+ "nonsemantic_share_top10_skip16": 0.271,
3266
+ "nonsemantic_share_top10_skip0": 0.326,
3267
+ "top10_jaccard_skip0_vs_16": 0.476,
3268
+ "top10_jaccard_skip4_vs_16": 0.541
3269
+ },
3270
+ "rec.1.27": {
3271
+ "top1_agree_model_skip16": 0.01,
3272
+ "top1_agree_model_skip4": 0.003,
3273
+ "top1_agree_model_skip0": 0.003,
3274
+ "top1_agree_model_logit_lens": 0.0,
3275
+ "nonsemantic_share_top10_skip16": 0.261,
3276
+ "nonsemantic_share_top10_skip0": 0.286,
3277
+ "top10_jaccard_skip0_vs_16": 0.496,
3278
+ "top10_jaccard_skip4_vs_16": 0.536
3279
+ },
3280
+ "rec.1.28": {
3281
+ "top1_agree_model_skip16": 0.007,
3282
+ "top1_agree_model_skip4": 0.003,
3283
+ "top1_agree_model_skip0": 0.003,
3284
+ "top1_agree_model_logit_lens": 0.0,
3285
+ "nonsemantic_share_top10_skip16": 0.291,
3286
+ "nonsemantic_share_top10_skip0": 0.345,
3287
+ "top10_jaccard_skip0_vs_16": 0.491,
3288
+ "top10_jaccard_skip4_vs_16": 0.543
3289
+ },
3290
+ "rec.1.29": {
3291
+ "top1_agree_model_skip16": 0.01,
3292
+ "top1_agree_model_skip4": 0.007,
3293
+ "top1_agree_model_skip0": 0.004,
3294
+ "top1_agree_model_logit_lens": 0.0,
3295
+ "nonsemantic_share_top10_skip16": 0.271,
3296
+ "nonsemantic_share_top10_skip0": 0.383,
3297
+ "top10_jaccard_skip0_vs_16": 0.433,
3298
+ "top10_jaccard_skip4_vs_16": 0.511
3299
+ },
3300
+ "rec.1.30": {
3301
+ "top1_agree_model_skip16": 0.01,
3302
+ "top1_agree_model_skip4": 0.007,
3303
+ "top1_agree_model_skip0": 0.004,
3304
+ "top1_agree_model_logit_lens": 0.0,
3305
+ "nonsemantic_share_top10_skip16": 0.361,
3306
+ "nonsemantic_share_top10_skip0": 0.467,
3307
+ "top10_jaccard_skip0_vs_16": 0.482,
3308
+ "top10_jaccard_skip4_vs_16": 0.55
3309
+ },
3310
+ "rec.1.31": {
3311
+ "top1_agree_model_skip16": 0.011,
3312
+ "top1_agree_model_skip4": 0.008,
3313
+ "top1_agree_model_skip0": 0.007,
3314
+ "top1_agree_model_logit_lens": 0.0,
3315
+ "nonsemantic_share_top10_skip16": 0.295,
3316
+ "nonsemantic_share_top10_skip0": 0.393,
3317
+ "top10_jaccard_skip0_vs_16": 0.5,
3318
+ "top10_jaccard_skip4_vs_16": 0.56
3319
+ },
3320
+ "rec.1.32": {
3321
+ "top1_agree_model_skip16": 0.01,
3322
+ "top1_agree_model_skip4": 0.006,
3323
+ "top1_agree_model_skip0": 0.003,
3324
+ "top1_agree_model_logit_lens": 0.001,
3325
+ "nonsemantic_share_top10_skip16": 0.317,
3326
+ "nonsemantic_share_top10_skip0": 0.459,
3327
+ "top10_jaccard_skip0_vs_16": 0.5,
3328
+ "top10_jaccard_skip4_vs_16": 0.558
3329
+ },
3330
+ "rec.1.33": {
3331
+ "top1_agree_model_skip16": 0.008,
3332
+ "top1_agree_model_skip4": 0.004,
3333
+ "top1_agree_model_skip0": 0.004,
3334
+ "top1_agree_model_logit_lens": 0.0,
3335
+ "nonsemantic_share_top10_skip16": 0.318,
3336
+ "nonsemantic_share_top10_skip0": 0.432,
3337
+ "top10_jaccard_skip0_vs_16": 0.561,
3338
+ "top10_jaccard_skip4_vs_16": 0.625
3339
+ },
3340
+ "rec.1.34": {
3341
+ "top1_agree_model_skip16": 0.008,
3342
+ "top1_agree_model_skip4": 0.006,
3343
+ "top1_agree_model_skip0": 0.004,
3344
+ "top1_agree_model_logit_lens": 0.002,
3345
+ "nonsemantic_share_top10_skip16": 0.351,
3346
+ "nonsemantic_share_top10_skip0": 0.482,
3347
+ "top10_jaccard_skip0_vs_16": 0.532,
3348
+ "top10_jaccard_skip4_vs_16": 0.619
3349
+ },
3350
+ "rec.1.35": {
3351
+ "top1_agree_model_skip16": 0.006,
3352
+ "top1_agree_model_skip4": 0.004,
3353
+ "top1_agree_model_skip0": 0.004,
3354
+ "top1_agree_model_logit_lens": 0.002,
3355
+ "nonsemantic_share_top10_skip16": 0.337,
3356
+ "nonsemantic_share_top10_skip0": 0.463,
3357
+ "top10_jaccard_skip0_vs_16": 0.58,
3358
+ "top10_jaccard_skip4_vs_16": 0.66
3359
+ },
3360
+ "rec.1.36": {
3361
+ "top1_agree_model_skip16": 0.004,
3362
+ "top1_agree_model_skip4": 0.004,
3363
+ "top1_agree_model_skip0": 0.003,
3364
+ "top1_agree_model_logit_lens": 0.006,
3365
+ "nonsemantic_share_top10_skip16": 0.329,
3366
+ "nonsemantic_share_top10_skip0": 0.447,
3367
+ "top10_jaccard_skip0_vs_16": 0.584,
3368
+ "top10_jaccard_skip4_vs_16": 0.67
3369
+ },
3370
+ "rec.1.37": {
3371
+ "top1_agree_model_skip16": 0.007,
3372
+ "top1_agree_model_skip4": 0.003,
3373
+ "top1_agree_model_skip0": 0.004,
3374
+ "top1_agree_model_logit_lens": 0.002,
3375
+ "nonsemantic_share_top10_skip16": 0.275,
3376
+ "nonsemantic_share_top10_skip0": 0.406,
3377
+ "top10_jaccard_skip0_vs_16": 0.543,
3378
+ "top10_jaccard_skip4_vs_16": 0.633
3379
+ },
3380
+ "rec.1.38": {
3381
+ "top1_agree_model_skip16": 0.009,
3382
+ "top1_agree_model_skip4": 0.009,
3383
+ "top1_agree_model_skip0": 0.007,
3384
+ "top1_agree_model_logit_lens": 0.015,
3385
+ "nonsemantic_share_top10_skip16": 0.284,
3386
+ "nonsemantic_share_top10_skip0": 0.44,
3387
+ "top10_jaccard_skip0_vs_16": 0.489,
3388
+ "top10_jaccard_skip4_vs_16": 0.573
3389
+ },
3390
+ "rec.1.39": {
3391
+ "top1_agree_model_skip16": 0.004,
3392
+ "top1_agree_model_skip4": 0.004,
3393
+ "top1_agree_model_skip0": 0.003,
3394
+ "top1_agree_model_logit_lens": 0.018,
3395
+ "nonsemantic_share_top10_skip16": 0.211,
3396
+ "nonsemantic_share_top10_skip0": 0.381,
3397
+ "top10_jaccard_skip0_vs_16": 0.437,
3398
+ "top10_jaccard_skip4_vs_16": 0.554
3399
+ },
3400
+ "rec.1.40": {
3401
+ "top1_agree_model_skip16": 0.003,
3402
+ "top1_agree_model_skip4": 0.004,
3403
+ "top1_agree_model_skip0": 0.006,
3404
+ "top1_agree_model_logit_lens": 0.017,
3405
+ "nonsemantic_share_top10_skip16": 0.25,
3406
+ "nonsemantic_share_top10_skip0": 0.416,
3407
+ "top10_jaccard_skip0_vs_16": 0.47,
3408
+ "top10_jaccard_skip4_vs_16": 0.557
3409
+ },
3410
+ "rec.1.41": {
3411
+ "top1_agree_model_skip16": 0.006,
3412
+ "top1_agree_model_skip4": 0.006,
3413
+ "top1_agree_model_skip0": 0.004,
3414
+ "top1_agree_model_logit_lens": 0.016,
3415
+ "nonsemantic_share_top10_skip16": 0.289,
3416
+ "nonsemantic_share_top10_skip0": 0.462,
3417
+ "top10_jaccard_skip0_vs_16": 0.493,
3418
+ "top10_jaccard_skip4_vs_16": 0.584
3419
+ },
3420
+ "rec.1.42": {
3421
+ "top1_agree_model_skip16": 0.011,
3422
+ "top1_agree_model_skip4": 0.011,
3423
+ "top1_agree_model_skip0": 0.008,
3424
+ "top1_agree_model_logit_lens": 0.027,
3425
+ "nonsemantic_share_top10_skip16": 0.304,
3426
+ "nonsemantic_share_top10_skip0": 0.426,
3427
+ "top10_jaccard_skip0_vs_16": 0.508,
3428
+ "top10_jaccard_skip4_vs_16": 0.596
3429
+ },
3430
+ "rec.1.43": {
3431
+ "top1_agree_model_skip16": 0.011,
3432
+ "top1_agree_model_skip4": 0.01,
3433
+ "top1_agree_model_skip0": 0.009,
3434
+ "top1_agree_model_logit_lens": 0.001,
3435
+ "nonsemantic_share_top10_skip16": 0.296,
3436
+ "nonsemantic_share_top10_skip0": 0.438,
3437
+ "top10_jaccard_skip0_vs_16": 0.48,
3438
+ "top10_jaccard_skip4_vs_16": 0.575
3439
+ },
3440
+ "rec.1.44": {
3441
+ "top1_agree_model_skip16": 0.011,
3442
+ "top1_agree_model_skip4": 0.009,
3443
+ "top1_agree_model_skip0": 0.009,
3444
+ "top1_agree_model_logit_lens": 0.039,
3445
+ "nonsemantic_share_top10_skip16": 0.306,
3446
+ "nonsemantic_share_top10_skip0": 0.444,
3447
+ "top10_jaccard_skip0_vs_16": 0.48,
3448
+ "top10_jaccard_skip4_vs_16": 0.584
3449
+ },
3450
+ "rec.1.45": {
3451
+ "top1_agree_model_skip16": 0.012,
3452
+ "top1_agree_model_skip4": 0.007,
3453
+ "top1_agree_model_skip0": 0.004,
3454
+ "top1_agree_model_logit_lens": 0.044,
3455
+ "nonsemantic_share_top10_skip16": 0.307,
3456
+ "nonsemantic_share_top10_skip0": 0.443,
3457
+ "top10_jaccard_skip0_vs_16": 0.519,
3458
+ "top10_jaccard_skip4_vs_16": 0.598
3459
+ },
3460
+ "rec.1.46": {
3461
+ "top1_agree_model_skip16": 0.035,
3462
+ "top1_agree_model_skip4": 0.025,
3463
+ "top1_agree_model_skip0": 0.021,
3464
+ "top1_agree_model_logit_lens": 0.046,
3465
+ "nonsemantic_share_top10_skip16": 0.312,
3466
+ "nonsemantic_share_top10_skip0": 0.389,
3467
+ "top10_jaccard_skip0_vs_16": 0.576,
3468
+ "top10_jaccard_skip4_vs_16": 0.645
3469
+ },
3470
+ "rec.1.47": {
3471
+ "top1_agree_model_skip16": 0.035,
3472
+ "top1_agree_model_skip4": 0.029,
3473
+ "top1_agree_model_skip0": 0.023,
3474
+ "top1_agree_model_logit_lens": 0.058,
3475
+ "nonsemantic_share_top10_skip16": 0.387,
3476
+ "nonsemantic_share_top10_skip0": 0.475,
3477
+ "top10_jaccard_skip0_vs_16": 0.583,
3478
+ "top10_jaccard_skip4_vs_16": 0.652
3479
+ },
3480
+ "rec.1.48": {
3481
+ "top1_agree_model_skip16": 0.032,
3482
+ "top1_agree_model_skip4": 0.027,
3483
+ "top1_agree_model_skip0": 0.025,
3484
+ "top1_agree_model_logit_lens": 0.061,
3485
+ "nonsemantic_share_top10_skip16": 0.318,
3486
+ "nonsemantic_share_top10_skip0": 0.398,
3487
+ "top10_jaccard_skip0_vs_16": 0.594,
3488
+ "top10_jaccard_skip4_vs_16": 0.666
3489
+ },
3490
+ "rec.1.49": {
3491
+ "top1_agree_model_skip16": 0.035,
3492
+ "top1_agree_model_skip4": 0.021,
3493
+ "top1_agree_model_skip0": 0.019,
3494
+ "top1_agree_model_logit_lens": 0.06,
3495
+ "nonsemantic_share_top10_skip16": 0.349,
3496
+ "nonsemantic_share_top10_skip0": 0.441,
3497
+ "top10_jaccard_skip0_vs_16": 0.605,
3498
+ "top10_jaccard_skip4_vs_16": 0.669
3499
+ },
3500
+ "rec.1.50": {
3501
+ "top1_agree_model_skip16": 0.049,
3502
+ "top1_agree_model_skip4": 0.033,
3503
+ "top1_agree_model_skip0": 0.032,
3504
+ "top1_agree_model_logit_lens": 0.086,
3505
+ "nonsemantic_share_top10_skip16": 0.301,
3506
+ "nonsemantic_share_top10_skip0": 0.396,
3507
+ "top10_jaccard_skip0_vs_16": 0.588,
3508
+ "top10_jaccard_skip4_vs_16": 0.661
3509
+ },
3510
+ "rec.1.51": {
3511
+ "top1_agree_model_skip16": 0.057,
3512
+ "top1_agree_model_skip4": 0.04,
3513
+ "top1_agree_model_skip0": 0.037,
3514
+ "top1_agree_model_logit_lens": 0.035,
3515
+ "nonsemantic_share_top10_skip16": 0.325,
3516
+ "nonsemantic_share_top10_skip0": 0.449,
3517
+ "top10_jaccard_skip0_vs_16": 0.59,
3518
+ "top10_jaccard_skip4_vs_16": 0.673
3519
+ },
3520
+ "rec.1.52": {
3521
+ "top1_agree_model_skip16": 0.104,
3522
+ "top1_agree_model_skip4": 0.07,
3523
+ "top1_agree_model_skip0": 0.052,
3524
+ "top1_agree_model_logit_lens": 0.098,
3525
+ "nonsemantic_share_top10_skip16": 0.361,
3526
+ "nonsemantic_share_top10_skip0": 0.485,
3527
+ "top10_jaccard_skip0_vs_16": 0.582,
3528
+ "top10_jaccard_skip4_vs_16": 0.664
3529
+ },
3530
+ "rec.1.53": {
3531
+ "top1_agree_model_skip16": 0.165,
3532
+ "top1_agree_model_skip4": 0.127,
3533
+ "top1_agree_model_skip0": 0.092,
3534
+ "top1_agree_model_logit_lens": 0.166,
3535
+ "nonsemantic_share_top10_skip16": 0.313,
3536
+ "nonsemantic_share_top10_skip0": 0.452,
3537
+ "top10_jaccard_skip0_vs_16": 0.529,
3538
+ "top10_jaccard_skip4_vs_16": 0.68
3539
+ },
3540
+ "rec.1.54": {
3541
+ "top1_agree_model_skip16": 0.186,
3542
+ "top1_agree_model_skip4": 0.135,
3543
+ "top1_agree_model_skip0": 0.119,
3544
+ "top1_agree_model_logit_lens": 0.152,
3545
+ "nonsemantic_share_top10_skip16": 0.299,
3546
+ "nonsemantic_share_top10_skip0": 0.417,
3547
+ "top10_jaccard_skip0_vs_16": 0.578,
3548
+ "top10_jaccard_skip4_vs_16": 0.666
3549
+ },
3550
+ "s.1": {
3551
+ "top1_agree_model_skip16": 0.164,
3552
+ "top1_agree_model_skip4": 0.121,
3553
+ "top1_agree_model_skip0": 0.11,
3554
+ "top1_agree_model_logit_lens": 0.0,
3555
+ "nonsemantic_share_top10_skip16": 0.3,
3556
+ "nonsemantic_share_top10_skip0": 0.414,
3557
+ "top10_jaccard_skip0_vs_16": 0.593,
3558
+ "top10_jaccard_skip4_vs_16": 0.673
3559
+ },
3560
+ "u.2": {
3561
+ "top1_agree_model_skip16": 0.035,
3562
+ "top1_agree_model_skip4": 0.029,
3563
+ "top1_agree_model_skip0": 0.026,
3564
+ "top1_agree_model_logit_lens": 0.001,
3565
+ "nonsemantic_share_top10_skip16": 0.489,
3566
+ "nonsemantic_share_top10_skip0": 0.531,
3567
+ "top10_jaccard_skip0_vs_16": 0.685,
3568
+ "top10_jaccard_skip4_vs_16": 0.75
3569
+ },
3570
+ "rec.2.16": {
3571
+ "top1_agree_model_skip16": 0.049,
3572
+ "top1_agree_model_skip4": 0.037,
3573
+ "top1_agree_model_skip0": 0.031,
3574
+ "top1_agree_model_logit_lens": 0.008,
3575
+ "nonsemantic_share_top10_skip16": 0.36,
3576
+ "nonsemantic_share_top10_skip0": 0.402,
3577
+ "top10_jaccard_skip0_vs_16": 0.638,
3578
+ "top10_jaccard_skip4_vs_16": 0.744
3579
+ },
3580
+ "rec.2.17": {
3581
+ "top1_agree_model_skip16": 0.039,
3582
+ "top1_agree_model_skip4": 0.028,
3583
+ "top1_agree_model_skip0": 0.022,
3584
+ "top1_agree_model_logit_lens": 0.002,
3585
+ "nonsemantic_share_top10_skip16": 0.311,
3586
+ "nonsemantic_share_top10_skip0": 0.36,
3587
+ "top10_jaccard_skip0_vs_16": 0.66,
3588
+ "top10_jaccard_skip4_vs_16": 0.766
3589
+ },
3590
+ "rec.2.18": {
3591
+ "top1_agree_model_skip16": 0.04,
3592
+ "top1_agree_model_skip4": 0.03,
3593
+ "top1_agree_model_skip0": 0.025,
3594
+ "top1_agree_model_logit_lens": 0.001,
3595
+ "nonsemantic_share_top10_skip16": 0.333,
3596
+ "nonsemantic_share_top10_skip0": 0.376,
3597
+ "top10_jaccard_skip0_vs_16": 0.652,
3598
+ "top10_jaccard_skip4_vs_16": 0.762
3599
+ },
3600
+ "rec.2.19": {
3601
+ "top1_agree_model_skip16": 0.037,
3602
+ "top1_agree_model_skip4": 0.027,
3603
+ "top1_agree_model_skip0": 0.022,
3604
+ "top1_agree_model_logit_lens": 0.002,
3605
+ "nonsemantic_share_top10_skip16": 0.2,
3606
+ "nonsemantic_share_top10_skip0": 0.239,
3607
+ "top10_jaccard_skip0_vs_16": 0.65,
3608
+ "top10_jaccard_skip4_vs_16": 0.775
3609
+ },
3610
+ "rec.2.20": {
3611
+ "top1_agree_model_skip16": 0.036,
3612
+ "top1_agree_model_skip4": 0.029,
3613
+ "top1_agree_model_skip0": 0.019,
3614
+ "top1_agree_model_logit_lens": 0.004,
3615
+ "nonsemantic_share_top10_skip16": 0.228,
3616
+ "nonsemantic_share_top10_skip0": 0.251,
3617
+ "top10_jaccard_skip0_vs_16": 0.668,
3618
+ "top10_jaccard_skip4_vs_16": 0.781
3619
+ },
3620
+ "rec.2.21": {
3621
+ "top1_agree_model_skip16": 0.031,
3622
+ "top1_agree_model_skip4": 0.023,
3623
+ "top1_agree_model_skip0": 0.02,
3624
+ "top1_agree_model_logit_lens": 0.006,
3625
+ "nonsemantic_share_top10_skip16": 0.204,
3626
+ "nonsemantic_share_top10_skip0": 0.229,
3627
+ "top10_jaccard_skip0_vs_16": 0.672,
3628
+ "top10_jaccard_skip4_vs_16": 0.79
3629
+ },
3630
+ "rec.2.22": {
3631
+ "top1_agree_model_skip16": 0.036,
3632
+ "top1_agree_model_skip4": 0.026,
3633
+ "top1_agree_model_skip0": 0.02,
3634
+ "top1_agree_model_logit_lens": 0.003,
3635
+ "nonsemantic_share_top10_skip16": 0.248,
3636
+ "nonsemantic_share_top10_skip0": 0.265,
3637
+ "top10_jaccard_skip0_vs_16": 0.653,
3638
+ "top10_jaccard_skip4_vs_16": 0.784
3639
+ },
3640
+ "rec.2.23": {
3641
+ "top1_agree_model_skip16": 0.035,
3642
+ "top1_agree_model_skip4": 0.029,
3643
+ "top1_agree_model_skip0": 0.02,
3644
+ "top1_agree_model_logit_lens": 0.0,
3645
+ "nonsemantic_share_top10_skip16": 0.197,
3646
+ "nonsemantic_share_top10_skip0": 0.236,
3647
+ "top10_jaccard_skip0_vs_16": 0.626,
3648
+ "top10_jaccard_skip4_vs_16": 0.788
3649
+ },
3650
+ "rec.2.24": {
3651
+ "top1_agree_model_skip16": 0.037,
3652
+ "top1_agree_model_skip4": 0.023,
3653
+ "top1_agree_model_skip0": 0.017,
3654
+ "top1_agree_model_logit_lens": 0.0,
3655
+ "nonsemantic_share_top10_skip16": 0.229,
3656
+ "nonsemantic_share_top10_skip0": 0.257,
3657
+ "top10_jaccard_skip0_vs_16": 0.654,
3658
+ "top10_jaccard_skip4_vs_16": 0.793
3659
+ },
3660
+ "rec.2.25": {
3661
+ "top1_agree_model_skip16": 0.032,
3662
+ "top1_agree_model_skip4": 0.03,
3663
+ "top1_agree_model_skip0": 0.02,
3664
+ "top1_agree_model_logit_lens": 0.0,
3665
+ "nonsemantic_share_top10_skip16": 0.209,
3666
+ "nonsemantic_share_top10_skip0": 0.246,
3667
+ "top10_jaccard_skip0_vs_16": 0.678,
3668
+ "top10_jaccard_skip4_vs_16": 0.805
3669
+ },
3670
+ "rec.2.26": {
3671
+ "top1_agree_model_skip16": 0.031,
3672
+ "top1_agree_model_skip4": 0.027,
3673
+ "top1_agree_model_skip0": 0.019,
3674
+ "top1_agree_model_logit_lens": 0.0,
3675
+ "nonsemantic_share_top10_skip16": 0.243,
3676
+ "nonsemantic_share_top10_skip0": 0.286,
3677
+ "top10_jaccard_skip0_vs_16": 0.645,
3678
+ "top10_jaccard_skip4_vs_16": 0.801
3679
+ },
3680
+ "rec.2.27": {
3681
+ "top1_agree_model_skip16": 0.032,
3682
+ "top1_agree_model_skip4": 0.027,
3683
+ "top1_agree_model_skip0": 0.02,
3684
+ "top1_agree_model_logit_lens": 0.0,
3685
+ "nonsemantic_share_top10_skip16": 0.265,
3686
+ "nonsemantic_share_top10_skip0": 0.278,
3687
+ "top10_jaccard_skip0_vs_16": 0.644,
3688
+ "top10_jaccard_skip4_vs_16": 0.798
3689
+ },
3690
+ "rec.2.28": {
3691
+ "top1_agree_model_skip16": 0.035,
3692
+ "top1_agree_model_skip4": 0.03,
3693
+ "top1_agree_model_skip0": 0.023,
3694
+ "top1_agree_model_logit_lens": 0.0,
3695
+ "nonsemantic_share_top10_skip16": 0.246,
3696
+ "nonsemantic_share_top10_skip0": 0.285,
3697
+ "top10_jaccard_skip0_vs_16": 0.642,
3698
+ "top10_jaccard_skip4_vs_16": 0.797
3699
+ },
3700
+ "rec.2.29": {
3701
+ "top1_agree_model_skip16": 0.033,
3702
+ "top1_agree_model_skip4": 0.032,
3703
+ "top1_agree_model_skip0": 0.023,
3704
+ "top1_agree_model_logit_lens": 0.0,
3705
+ "nonsemantic_share_top10_skip16": 0.228,
3706
+ "nonsemantic_share_top10_skip0": 0.275,
3707
+ "top10_jaccard_skip0_vs_16": 0.634,
3708
+ "top10_jaccard_skip4_vs_16": 0.808
3709
+ },
3710
+ "rec.2.30": {
3711
+ "top1_agree_model_skip16": 0.037,
3712
+ "top1_agree_model_skip4": 0.035,
3713
+ "top1_agree_model_skip0": 0.033,
3714
+ "top1_agree_model_logit_lens": 0.001,
3715
+ "nonsemantic_share_top10_skip16": 0.247,
3716
+ "nonsemantic_share_top10_skip0": 0.29,
3717
+ "top10_jaccard_skip0_vs_16": 0.624,
3718
+ "top10_jaccard_skip4_vs_16": 0.8
3719
+ },
3720
+ "rec.2.31": {
3721
+ "top1_agree_model_skip16": 0.036,
3722
+ "top1_agree_model_skip4": 0.033,
3723
+ "top1_agree_model_skip0": 0.03,
3724
+ "top1_agree_model_logit_lens": 0.0,
3725
+ "nonsemantic_share_top10_skip16": 0.204,
3726
+ "nonsemantic_share_top10_skip0": 0.263,
3727
+ "top10_jaccard_skip0_vs_16": 0.687,
3728
+ "top10_jaccard_skip4_vs_16": 0.799
3729
+ },
3730
+ "rec.2.32": {
3731
+ "top1_agree_model_skip16": 0.038,
3732
+ "top1_agree_model_skip4": 0.033,
3733
+ "top1_agree_model_skip0": 0.03,
3734
+ "top1_agree_model_logit_lens": 0.001,
3735
+ "nonsemantic_share_top10_skip16": 0.236,
3736
+ "nonsemantic_share_top10_skip0": 0.293,
3737
+ "top10_jaccard_skip0_vs_16": 0.648,
3738
+ "top10_jaccard_skip4_vs_16": 0.799
3739
+ },
3740
+ "rec.2.33": {
3741
+ "top1_agree_model_skip16": 0.04,
3742
+ "top1_agree_model_skip4": 0.036,
3743
+ "top1_agree_model_skip0": 0.035,
3744
+ "top1_agree_model_logit_lens": 0.0,
3745
+ "nonsemantic_share_top10_skip16": 0.226,
3746
+ "nonsemantic_share_top10_skip0": 0.275,
3747
+ "top10_jaccard_skip0_vs_16": 0.695,
3748
+ "top10_jaccard_skip4_vs_16": 0.81
3749
+ },
3750
+ "rec.2.34": {
3751
+ "top1_agree_model_skip16": 0.039,
3752
+ "top1_agree_model_skip4": 0.037,
3753
+ "top1_agree_model_skip0": 0.032,
3754
+ "top1_agree_model_logit_lens": 0.001,
3755
+ "nonsemantic_share_top10_skip16": 0.227,
3756
+ "nonsemantic_share_top10_skip0": 0.27,
3757
+ "top10_jaccard_skip0_vs_16": 0.696,
3758
+ "top10_jaccard_skip4_vs_16": 0.813
3759
+ },
3760
+ "rec.2.35": {
3761
+ "top1_agree_model_skip16": 0.037,
3762
+ "top1_agree_model_skip4": 0.031,
3763
+ "top1_agree_model_skip0": 0.035,
3764
+ "top1_agree_model_logit_lens": 0.002,
3765
+ "nonsemantic_share_top10_skip16": 0.236,
3766
+ "nonsemantic_share_top10_skip0": 0.28,
3767
+ "top10_jaccard_skip0_vs_16": 0.712,
3768
+ "top10_jaccard_skip4_vs_16": 0.818
3769
+ },
3770
+ "rec.2.36": {
3771
+ "top1_agree_model_skip16": 0.04,
3772
+ "top1_agree_model_skip4": 0.037,
3773
+ "top1_agree_model_skip0": 0.036,
3774
+ "top1_agree_model_logit_lens": 0.012,
3775
+ "nonsemantic_share_top10_skip16": 0.23,
3776
+ "nonsemantic_share_top10_skip0": 0.256,
3777
+ "top10_jaccard_skip0_vs_16": 0.707,
3778
+ "top10_jaccard_skip4_vs_16": 0.813
3779
+ },
3780
+ "rec.2.37": {
3781
+ "top1_agree_model_skip16": 0.042,
3782
+ "top1_agree_model_skip4": 0.036,
3783
+ "top1_agree_model_skip0": 0.033,
3784
+ "top1_agree_model_logit_lens": 0.002,
3785
+ "nonsemantic_share_top10_skip16": 0.225,
3786
+ "nonsemantic_share_top10_skip0": 0.263,
3787
+ "top10_jaccard_skip0_vs_16": 0.701,
3788
+ "top10_jaccard_skip4_vs_16": 0.815
3789
+ },
3790
+ "rec.2.38": {
3791
+ "top1_agree_model_skip16": 0.049,
3792
+ "top1_agree_model_skip4": 0.044,
3793
+ "top1_agree_model_skip0": 0.04,
3794
+ "top1_agree_model_logit_lens": 0.02,
3795
+ "nonsemantic_share_top10_skip16": 0.226,
3796
+ "nonsemantic_share_top10_skip0": 0.256,
3797
+ "top10_jaccard_skip0_vs_16": 0.724,
3798
+ "top10_jaccard_skip4_vs_16": 0.814
3799
+ },
3800
+ "rec.2.39": {
3801
+ "top1_agree_model_skip16": 0.054,
3802
+ "top1_agree_model_skip4": 0.047,
3803
+ "top1_agree_model_skip0": 0.045,
3804
+ "top1_agree_model_logit_lens": 0.023,
3805
+ "nonsemantic_share_top10_skip16": 0.211,
3806
+ "nonsemantic_share_top10_skip0": 0.231,
3807
+ "top10_jaccard_skip0_vs_16": 0.732,
3808
+ "top10_jaccard_skip4_vs_16": 0.834
3809
+ },
3810
+ "rec.2.40": {
3811
+ "top1_agree_model_skip16": 0.058,
3812
+ "top1_agree_model_skip4": 0.05,
3813
+ "top1_agree_model_skip0": 0.046,
3814
+ "top1_agree_model_logit_lens": 0.025,
3815
+ "nonsemantic_share_top10_skip16": 0.181,
3816
+ "nonsemantic_share_top10_skip0": 0.206,
3817
+ "top10_jaccard_skip0_vs_16": 0.751,
3818
+ "top10_jaccard_skip4_vs_16": 0.842
3819
+ },
3820
+ "rec.2.41": {
3821
+ "top1_agree_model_skip16": 0.057,
3822
+ "top1_agree_model_skip4": 0.05,
3823
+ "top1_agree_model_skip0": 0.045,
3824
+ "top1_agree_model_logit_lens": 0.026,
3825
+ "nonsemantic_share_top10_skip16": 0.229,
3826
+ "nonsemantic_share_top10_skip0": 0.258,
3827
+ "top10_jaccard_skip0_vs_16": 0.773,
3828
+ "top10_jaccard_skip4_vs_16": 0.848
3829
+ },
3830
+ "rec.2.42": {
3831
+ "top1_agree_model_skip16": 0.071,
3832
+ "top1_agree_model_skip4": 0.067,
3833
+ "top1_agree_model_skip0": 0.065,
3834
+ "top1_agree_model_logit_lens": 0.031,
3835
+ "nonsemantic_share_top10_skip16": 0.165,
3836
+ "nonsemantic_share_top10_skip0": 0.177,
3837
+ "top10_jaccard_skip0_vs_16": 0.782,
3838
+ "top10_jaccard_skip4_vs_16": 0.861
3839
+ },
3840
+ "rec.2.43": {
3841
+ "top1_agree_model_skip16": 0.078,
3842
+ "top1_agree_model_skip4": 0.075,
3843
+ "top1_agree_model_skip0": 0.069,
3844
+ "top1_agree_model_logit_lens": 0.001,
3845
+ "nonsemantic_share_top10_skip16": 0.173,
3846
+ "nonsemantic_share_top10_skip0": 0.194,
3847
+ "top10_jaccard_skip0_vs_16": 0.781,
3848
+ "top10_jaccard_skip4_vs_16": 0.856
3849
+ },
3850
+ "rec.2.44": {
3851
+ "top1_agree_model_skip16": 0.08,
3852
+ "top1_agree_model_skip4": 0.08,
3853
+ "top1_agree_model_skip0": 0.075,
3854
+ "top1_agree_model_logit_lens": 0.042,
3855
+ "nonsemantic_share_top10_skip16": 0.174,
3856
+ "nonsemantic_share_top10_skip0": 0.186,
3857
+ "top10_jaccard_skip0_vs_16": 0.788,
3858
+ "top10_jaccard_skip4_vs_16": 0.865
3859
+ },
3860
+ "rec.2.45": {
3861
+ "top1_agree_model_skip16": 0.088,
3862
+ "top1_agree_model_skip4": 0.085,
3863
+ "top1_agree_model_skip0": 0.08,
3864
+ "top1_agree_model_logit_lens": 0.05,
3865
+ "nonsemantic_share_top10_skip16": 0.178,
3866
+ "nonsemantic_share_top10_skip0": 0.183,
3867
+ "top10_jaccard_skip0_vs_16": 0.786,
3868
+ "top10_jaccard_skip4_vs_16": 0.867
3869
+ },
3870
+ "rec.2.46": {
3871
+ "top1_agree_model_skip16": 0.097,
3872
+ "top1_agree_model_skip4": 0.092,
3873
+ "top1_agree_model_skip0": 0.089,
3874
+ "top1_agree_model_logit_lens": 0.048,
3875
+ "nonsemantic_share_top10_skip16": 0.143,
3876
+ "nonsemantic_share_top10_skip0": 0.163,
3877
+ "top10_jaccard_skip0_vs_16": 0.802,
3878
+ "top10_jaccard_skip4_vs_16": 0.864
3879
+ },
3880
+ "rec.2.47": {
3881
+ "top1_agree_model_skip16": 0.107,
3882
+ "top1_agree_model_skip4": 0.1,
3883
+ "top1_agree_model_skip0": 0.096,
3884
+ "top1_agree_model_logit_lens": 0.065,
3885
+ "nonsemantic_share_top10_skip16": 0.172,
3886
+ "nonsemantic_share_top10_skip0": 0.188,
3887
+ "top10_jaccard_skip0_vs_16": 0.808,
3888
+ "top10_jaccard_skip4_vs_16": 0.864
3889
+ },
3890
+ "rec.2.48": {
3891
+ "top1_agree_model_skip16": 0.119,
3892
+ "top1_agree_model_skip4": 0.113,
3893
+ "top1_agree_model_skip0": 0.108,
3894
+ "top1_agree_model_logit_lens": 0.071,
3895
+ "nonsemantic_share_top10_skip16": 0.16,
3896
+ "nonsemantic_share_top10_skip0": 0.192,
3897
+ "top10_jaccard_skip0_vs_16": 0.816,
3898
+ "top10_jaccard_skip4_vs_16": 0.866
3899
+ },
3900
+ "rec.2.49": {
3901
+ "top1_agree_model_skip16": 0.143,
3902
+ "top1_agree_model_skip4": 0.14,
3903
+ "top1_agree_model_skip0": 0.138,
3904
+ "top1_agree_model_logit_lens": 0.063,
3905
+ "nonsemantic_share_top10_skip16": 0.14,
3906
+ "nonsemantic_share_top10_skip0": 0.165,
3907
+ "top10_jaccard_skip0_vs_16": 0.828,
3908
+ "top10_jaccard_skip4_vs_16": 0.875
3909
+ },
3910
+ "rec.2.50": {
3911
+ "top1_agree_model_skip16": 0.183,
3912
+ "top1_agree_model_skip4": 0.172,
3913
+ "top1_agree_model_skip0": 0.169,
3914
+ "top1_agree_model_logit_lens": 0.102,
3915
+ "nonsemantic_share_top10_skip16": 0.15,
3916
+ "nonsemantic_share_top10_skip0": 0.178,
3917
+ "top10_jaccard_skip0_vs_16": 0.827,
3918
+ "top10_jaccard_skip4_vs_16": 0.884
3919
+ },
3920
+ "rec.2.51": {
3921
+ "top1_agree_model_skip16": 0.227,
3922
+ "top1_agree_model_skip4": 0.214,
3923
+ "top1_agree_model_skip0": 0.203,
3924
+ "top1_agree_model_logit_lens": 0.035,
3925
+ "nonsemantic_share_top10_skip16": 0.179,
3926
+ "nonsemantic_share_top10_skip0": 0.205,
3927
+ "top10_jaccard_skip0_vs_16": 0.822,
3928
+ "top10_jaccard_skip4_vs_16": 0.881
3929
+ },
3930
+ "rec.2.52": {
3931
+ "top1_agree_model_skip16": 0.251,
3932
+ "top1_agree_model_skip4": 0.24,
3933
+ "top1_agree_model_skip0": 0.225,
3934
+ "top1_agree_model_logit_lens": 0.109,
3935
+ "nonsemantic_share_top10_skip16": 0.179,
3936
+ "nonsemantic_share_top10_skip0": 0.204,
3937
+ "top10_jaccard_skip0_vs_16": 0.822,
3938
+ "top10_jaccard_skip4_vs_16": 0.886
3939
+ },
3940
+ "rec.2.53": {
3941
+ "top1_agree_model_skip16": 0.287,
3942
+ "top1_agree_model_skip4": 0.28,
3943
+ "top1_agree_model_skip0": 0.26,
3944
+ "top1_agree_model_logit_lens": 0.167,
3945
+ "nonsemantic_share_top10_skip16": 0.141,
3946
+ "nonsemantic_share_top10_skip0": 0.167,
3947
+ "top10_jaccard_skip0_vs_16": 0.766,
3948
+ "top10_jaccard_skip4_vs_16": 0.887
3949
+ },
3950
+ "rec.2.54": {
3951
+ "top1_agree_model_skip16": 0.323,
3952
+ "top1_agree_model_skip4": 0.316,
3953
+ "top1_agree_model_skip0": 0.304,
3954
+ "top1_agree_model_logit_lens": 0.154,
3955
+ "nonsemantic_share_top10_skip16": 0.158,
3956
+ "nonsemantic_share_top10_skip0": 0.178,
3957
+ "top10_jaccard_skip0_vs_16": 0.85,
3958
+ "top10_jaccard_skip4_vs_16": 0.914
3959
+ },
3960
+ "s.2": {
3961
+ "top1_agree_model_skip16": 0.345,
3962
+ "top1_agree_model_skip4": 0.34,
3963
+ "top1_agree_model_skip0": 0.328,
3964
+ "top1_agree_model_logit_lens": 0.0,
3965
+ "nonsemantic_share_top10_skip16": 0.132,
3966
+ "nonsemantic_share_top10_skip0": 0.146,
3967
+ "top10_jaccard_skip0_vs_16": 0.863,
3968
+ "top10_jaccard_skip4_vs_16": 0.915
3969
+ },
3970
+ "out.56": {
3971
+ "top1_agree_model_skip16": 0.333,
3972
+ "top1_agree_model_skip4": 0.323,
3973
+ "top1_agree_model_skip0": 0.316,
3974
+ "top1_agree_model_logit_lens": 0.218,
3975
+ "nonsemantic_share_top10_skip16": 0.124,
3976
+ "nonsemantic_share_top10_skip0": 0.137,
3977
+ "top10_jaccard_skip0_vs_16": 0.872,
3978
+ "top10_jaccard_skip4_vs_16": 0.919
3979
+ },
3980
+ "out.57": {
3981
+ "top1_agree_model_skip16": 0.355,
3982
+ "top1_agree_model_skip4": 0.344,
3983
+ "top1_agree_model_skip0": 0.335,
3984
+ "top1_agree_model_logit_lens": 0.259,
3985
+ "nonsemantic_share_top10_skip16": 0.109,
3986
+ "nonsemantic_share_top10_skip0": 0.119,
3987
+ "top10_jaccard_skip0_vs_16": 0.874,
3988
+ "top10_jaccard_skip4_vs_16": 0.923
3989
+ },
3990
+ "out.58": {
3991
+ "top1_agree_model_skip16": 0.355,
3992
+ "top1_agree_model_skip4": 0.348,
3993
+ "top1_agree_model_skip0": 0.343,
3994
+ "top1_agree_model_logit_lens": 0.266,
3995
+ "nonsemantic_share_top10_skip16": 0.11,
3996
+ "nonsemantic_share_top10_skip0": 0.123,
3997
+ "top10_jaccard_skip0_vs_16": 0.887,
3998
+ "top10_jaccard_skip4_vs_16": 0.93
3999
+ },
4000
+ "out.59": {
4001
+ "top1_agree_model_skip16": 0.362,
4002
+ "top1_agree_model_skip4": 0.352,
4003
+ "top1_agree_model_skip0": 0.345,
4004
+ "top1_agree_model_logit_lens": 0.275,
4005
+ "nonsemantic_share_top10_skip16": 0.101,
4006
+ "nonsemantic_share_top10_skip0": 0.111,
4007
+ "top10_jaccard_skip0_vs_16": 0.89,
4008
+ "top10_jaccard_skip4_vs_16": 0.935
4009
+ },
4010
+ "out.60": {
4011
+ "top1_agree_model_skip16": 0.384,
4012
+ "top1_agree_model_skip4": 0.373,
4013
+ "top1_agree_model_skip0": 0.369,
4014
+ "top1_agree_model_logit_lens": 0.287,
4015
+ "nonsemantic_share_top10_skip16": 0.104,
4016
+ "nonsemantic_share_top10_skip0": 0.115,
4017
+ "top10_jaccard_skip0_vs_16": 0.906,
4018
+ "top10_jaccard_skip4_vs_16": 0.948
4019
+ },
4020
+ "out.61": {
4021
+ "top1_agree_model_skip16": 0.385,
4022
+ "top1_agree_model_skip4": 0.377,
4023
+ "top1_agree_model_skip0": 0.364,
4024
+ "top1_agree_model_logit_lens": 0.203,
4025
+ "nonsemantic_share_top10_skip16": 0.099,
4026
+ "nonsemantic_share_top10_skip0": 0.112,
4027
+ "top10_jaccard_skip0_vs_16": 0.875,
4028
+ "top10_jaccard_skip4_vs_16": 0.945
4029
+ },
4030
+ "out.62": {
4031
+ "top1_agree_model_skip16": 0.397,
4032
+ "top1_agree_model_skip4": 0.392,
4033
+ "top1_agree_model_skip0": 0.391,
4034
+ "top1_agree_model_logit_lens": 0.321,
4035
+ "nonsemantic_share_top10_skip16": 0.101,
4036
+ "nonsemantic_share_top10_skip0": 0.108,
4037
+ "top10_jaccard_skip0_vs_16": 0.943,
4038
+ "top10_jaccard_skip4_vs_16": 0.964
4039
+ },
4040
+ "out.63": {
4041
+ "top1_agree_model_skip16": 0.414,
4042
+ "top1_agree_model_skip4": 0.417,
4043
+ "top1_agree_model_skip0": 0.416,
4044
+ "top1_agree_model_logit_lens": 0.36,
4045
+ "nonsemantic_share_top10_skip16": 0.091,
4046
+ "nonsemantic_share_top10_skip0": 0.099,
4047
+ "top10_jaccard_skip0_vs_16": 0.949,
4048
+ "top10_jaccard_skip4_vs_16": 0.969
4049
+ },
4050
+ "out.64": {
4051
+ "top1_agree_model_skip16": 0.445,
4052
+ "top1_agree_model_skip4": 0.442,
4053
+ "top1_agree_model_skip0": 0.438,
4054
+ "top1_agree_model_logit_lens": 0.371,
4055
+ "nonsemantic_share_top10_skip16": 0.095,
4056
+ "nonsemantic_share_top10_skip0": 0.1,
4057
+ "top10_jaccard_skip0_vs_16": 0.947,
4058
+ "top10_jaccard_skip4_vs_16": 0.967
4059
+ },
4060
+ "out.65": {
4061
+ "top1_agree_model_skip16": 0.469,
4062
+ "top1_agree_model_skip4": 0.464,
4063
+ "top1_agree_model_skip0": 0.459,
4064
+ "top1_agree_model_logit_lens": 0.382,
4065
+ "nonsemantic_share_top10_skip16": 0.092,
4066
+ "nonsemantic_share_top10_skip0": 0.099,
4067
+ "top10_jaccard_skip0_vs_16": 0.923,
4068
+ "top10_jaccard_skip4_vs_16": 0.965
4069
+ },
4070
+ "out.66": {
4071
+ "top1_agree_model_skip16": 0.478,
4072
+ "top1_agree_model_skip4": 0.477,
4073
+ "top1_agree_model_skip0": 0.472,
4074
+ "top1_agree_model_logit_lens": 0.439,
4075
+ "nonsemantic_share_top10_skip16": 0.089,
4076
+ "nonsemantic_share_top10_skip0": 0.094,
4077
+ "top10_jaccard_skip0_vs_16": 0.953,
4078
+ "top10_jaccard_skip4_vs_16": 0.973
4079
+ },
4080
+ "out.67": {
4081
+ "top1_agree_model_skip16": 0.481,
4082
+ "top1_agree_model_skip4": 0.478,
4083
+ "top1_agree_model_skip0": 0.468,
4084
+ "top1_agree_model_logit_lens": 0.146,
4085
+ "nonsemantic_share_top10_skip16": 0.079,
4086
+ "nonsemantic_share_top10_skip0": 0.087,
4087
+ "top10_jaccard_skip0_vs_16": 0.93,
4088
+ "top10_jaccard_skip4_vs_16": 0.959
4089
+ },
4090
+ "out.68": {
4091
+ "top1_agree_model_skip16": 0.51,
4092
+ "top1_agree_model_skip4": 0.511,
4093
+ "top1_agree_model_skip0": 0.509,
4094
+ "top1_agree_model_logit_lens": 0.467,
4095
+ "nonsemantic_share_top10_skip16": 0.083,
4096
+ "nonsemantic_share_top10_skip0": 0.086,
4097
+ "top10_jaccard_skip0_vs_16": 0.963,
4098
+ "top10_jaccard_skip4_vs_16": 0.979
4099
+ },
4100
+ "out.69": {
4101
+ "top1_agree_model_skip16": 0.558,
4102
+ "top1_agree_model_skip4": 0.56,
4103
+ "top1_agree_model_skip0": 0.552,
4104
+ "top1_agree_model_logit_lens": 0.186,
4105
+ "nonsemantic_share_top10_skip16": 0.076,
4106
+ "nonsemantic_share_top10_skip0": 0.081,
4107
+ "top10_jaccard_skip0_vs_16": 0.942,
4108
+ "top10_jaccard_skip4_vs_16": 0.973
4109
+ },
4110
+ "out.70": {
4111
+ "top1_agree_model_skip16": 0.469,
4112
+ "top1_agree_model_skip4": 0.467,
4113
+ "top1_agree_model_skip0": 0.469,
4114
+ "top1_agree_model_logit_lens": 0.609,
4115
+ "nonsemantic_share_top10_skip16": 0.073,
4116
+ "nonsemantic_share_top10_skip0": 0.076,
4117
+ "top10_jaccard_skip0_vs_16": 0.978,
4118
+ "top10_jaccard_skip4_vs_16": 0.985
4119
+ }
4120
+ },
4121
+ "lens_bytes": 2989630584
4122
+ }
fit_records/k2_n50_skip16_s0/prompts.json ADDED
@@ -0,0 +1,202 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [
2
+ {
3
+ "i": 0,
4
+ "start": "Given the chaos , the governor of Yunnan ignored the imperial orders of evacuation . The Mongol army commanded by Prince Ye @"
5
+ },
6
+ {
7
+ "i": 1,
8
+ "start": "From the 1920s Australia 's defence thinking was dominated by British Imperial defence policy , which was embodied"
9
+ },
10
+ {
11
+ "i": 2,
12
+ "start": "Tim 's driving days started as a toddler when he was given a go @-@ kart that he often drove"
13
+ },
14
+ {
15
+ "i": 3,
16
+ "start": "As the Marines formed into a horseshoe @-@ shaped line around Hill 123 , Tamura 's"
17
+ },
18
+ {
19
+ "i": 4,
20
+ "start": "USA Today 's Scott Bowles awarded the film three out of four stars , labeling it a \" surprising comedy that rivals"
21
+ },
22
+ {
23
+ "i": 5,
24
+ "start": "Ringo Starr identified his initial inspiration for \" Back Off Boogaloo \" as having come from Marc Bolan ,"
25
+ },
26
+ {
27
+ "i": 6,
28
+ "start": "The ship 's secondary armament consisted of fourteen 45 @-@ calibre 6 @-@ inch ("
29
+ },
30
+ {
31
+ "i": 7,
32
+ "start": "Sources disagree about Radcliffe 's personal wealth ; he was reported to have earned \u00a3 1 million for the first Harry"
33
+ },
34
+ {
35
+ "i": 8,
36
+ "start": "In April 2010 , there were 6 @,@ 260 people employed in the healthcare field"
37
+ },
38
+ {
39
+ "i": 9,
40
+ "start": "\" Exit Through the Kwik @-@ E @-@ Mart \" is the fifteenth episode of the twenty @-"
41
+ },
42
+ {
43
+ "i": 10,
44
+ "start": "On 25 June 1950 , the North Korean Army ( KPA ) crossed the border into South"
45
+ },
46
+ {
47
+ "i": 11,
48
+ "start": "A British parliamentary motion put forward by Labour MP Kate Hoey in February 2013 to retrospectively condemn the"
49
+ },
50
+ {
51
+ "i": 12,
52
+ "start": "At the fort , Moultrie ordered his men to concentrate their fire on the two large man @-@ of @"
53
+ },
54
+ {
55
+ "i": 13,
56
+ "start": "In Part II of the series , Shikamaru is assigned the task of locating two members of the criminal organization Ak"
57
+ },
58
+ {
59
+ "i": 14,
60
+ "start": "The Ming dynasty had seen considerable advancement in the process of colour printing in China . At his studio , Hu Zhengyan experiment"
61
+ },
62
+ {
63
+ "i": 15,
64
+ "start": "\" Exit Through the Kwik @-@ E @-@ Mart \" was written by Marc Wilmore and directed by"
65
+ },
66
+ {
67
+ "i": 16,
68
+ "start": "In 629 Tikal founded Dos Pilas , some 110 kilometres ( 68 mi )"
69
+ },
70
+ {
71
+ "i": 17,
72
+ "start": "The British fleet weighed anchor at Cape Fear on May 31 , and arrived outside Charleston Harbor the next day . Moul"
73
+ },
74
+ {
75
+ "i": 18,
76
+ "start": "The initial phase saw the Australians advance towards the Hongorai River . Following the end of the early fighting , the Australian"
77
+ },
78
+ {
79
+ "i": 19,
80
+ "start": "The reactions of the native Aboriginal inhabitants to the sudden arrival of British settlers in Australia were varied , but were inevitably hostile when"
81
+ },
82
+ {
83
+ "i": 20,
84
+ "start": "The German offensive ground to a halt in mid @-@ July and a brief lull followed , during which the Australians"
85
+ },
86
+ {
87
+ "i": 21,
88
+ "start": "The city 's growth has reflected the push and pull of many social and economic factors . The total population increased in each"
89
+ },
90
+ {
91
+ "i": 22,
92
+ "start": "Following the classification of Tropical Depression Two @-@ E on June 19 , the Government of Mexico issued a tropical"
93
+ },
94
+ {
95
+ "i": 23,
96
+ "start": "The Natives remained in London following the England international . They defeated London Welsh on 18 February , before losing first"
97
+ },
98
+ {
99
+ "i": 24,
100
+ "start": "In February 1997 , a United States immigration court judge granted asylum to a German Scientologist who claimed she"
101
+ },
102
+ {
103
+ "i": 25,
104
+ "start": "Some of the most important trade routes began at Canton and Macau in Qing Dynasty China . Early in each year a large convoy"
105
+ },
106
+ {
107
+ "i": 26,
108
+ "start": "The fourth installment in the Slammiversary name was announced in January 2008 to take place on June"
109
+ },
110
+ {
111
+ "i": 27,
112
+ "start": "Capel Lligwy has national recognition and statutory protection from alteration as it has been designated as a Grade II listed building"
113
+ },
114
+ {
115
+ "i": 28,
116
+ "start": "The Trout Creek Mountains are semi @-@ arid because they are in the eastern rain shadow of mountain ranges to"
117
+ },
118
+ {
119
+ "i": 29,
120
+ "start": "Asahi , meaning \" rising sun \" , a poetic name for Japan from a stanza of waka poetry , was laid"
121
+ },
122
+ {
123
+ "i": 30,
124
+ "start": "Beckham 's term as governor ended December 10 , 1907 . In January 19"
125
+ },
126
+ {
127
+ "i": 31,
128
+ "start": "Donald Sutherland portrays Jack Pellitt , Bobby 's father and Kurt 's boss . On July 27 ,"
129
+ },
130
+ {
131
+ "i": 32,
132
+ "start": "As British offshoots , the Australian colonies participated in Britain 's small wars of the 19th century"
133
+ },
134
+ {
135
+ "i": 33,
136
+ "start": "Retail is another major employer in the county , with 5 @,@ 280 people employed in April "
137
+ },
138
+ {
139
+ "i": 34,
140
+ "start": "The tour returned to Pocono a month later , and Richmond battled for another victory in a fog @-@ shortened"
141
+ },
142
+ {
143
+ "i": 35,
144
+ "start": "\" ... Thirteen Years Later \" received mixed reviews from critics . The A.V. Club 's Todd VanDerWer"
145
+ },
146
+ {
147
+ "i": 36,
148
+ "start": "Bonnie Stiernberg , for Paste , was more positive , writing that she preferred the use of comedy , following the"
149
+ },
150
+ {
151
+ "i": 37,
152
+ "start": "In common with many small Scottish islands , Papa Stour 's population peaked in the 19th century and has"
153
+ },
154
+ {
155
+ "i": 38,
156
+ "start": "The unexpected deployment to East Timor in 1999 led to significant changes in Australian defence policy and to an enhancement"
157
+ },
158
+ {
159
+ "i": 39,
160
+ "start": "Government publications on the dangers of sects increased between 1996 and 1998 , and"
161
+ },
162
+ {
163
+ "i": 40,
164
+ "start": "Richmond started 1982 without a ride before getting a one @-@ race deal to drive for Billie"
165
+ },
166
+ {
167
+ "i": 41,
168
+ "start": "The music for Sonic the Hedgehog was composed and produced by Masato Nakamura , bassist and lead songwriter of the"
169
+ },
170
+ {
171
+ "i": 42,
172
+ "start": "Apple Records issued the single on 17 March 1972 in Britain , as Apple R 59"
173
+ },
174
+ {
175
+ "i": 43,
176
+ "start": "Meridian is home to two post @-@ secondary educational institutions . Meridian Community College , founded in 19"
177
+ },
178
+ {
179
+ "i": 44,
180
+ "start": "Ayola began her career in the theatre , performing in a number of Shakespearean plays including Twelfth Night , A M"
181
+ },
182
+ {
183
+ "i": 45,
184
+ "start": "Cooke 's opportunistic diversion had determined that the Spanish forces in the Philippines posed no immediate threat , although the mission"
185
+ },
186
+ {
187
+ "i": 46,
188
+ "start": "To further justify its stance , the German government has also pointed to the long history of U.S. court cases involving Scient"
189
+ },
190
+ {
191
+ "i": 47,
192
+ "start": "Headed by police chief Lee Shelbourn since 2009 , the Meridian Police Department consists of "
193
+ },
194
+ {
195
+ "i": 48,
196
+ "start": "A port , Sonic the Hedgehog Genesis , was released for the Game Boy Advance ( GBA ) on November"
197
+ },
198
+ {
199
+ "i": 49,
200
+ "start": "As is often the case with huge ancient ruins , knowledge of the site was never completely lost in the region . It seems"
201
+ }
202
+ ]
fit_records/k2_n50_skip16_s0/readouts.txt ADDED
@@ -0,0 +1,142 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ non-semantic tokens (no letter or digit): 5208 of 131072
2
+
3
+ EARLY POSITIONS. Jacobian size by source position, relative to the median over positions 16+ (k=2, 50 prompts)
4
+ node p0 p1 p2 p3 p4 p5 p6 p7 p8-15 mean
5
+ emb 6.13 4.98 4.18 3.82 6.51 4.17 3.35 3.28 5.75
6
+ in.04 2.55 2.93 2.94 2.38 4.37 3.30 4.11 2.81 4.47
7
+ in.12 2.67 1.71 2.60 1.91 2.70 2.72 3.55 3.05 4.07
8
+ x 1.13 0.68 1.28 0.89 1.02 1.70 2.15 2.49 2.64
9
+ u.1 1.10 0.47 0.80 0.59 0.83 1.79 2.33 2.66 2.85
10
+ rec.1.20 3.93 1.25 1.18 1.33 1.61 3.30 7.76 4.91 7.83
11
+ rec.1.28 4.93 1.18 1.14 1.03 1.13 3.39 5.42 2.56 7.02
12
+ rec.1.36 5.15 1.79 1.95 1.32 1.62 1.93 5.10 1.81 3.51
13
+ rec.1.44 5.21 2.28 2.18 1.46 1.84 1.71 1.84 2.20 2.33
14
+ rec.1.52 2.20 1.48 2.32 1.70 1.95 1.82 1.93 2.35 2.21
15
+ s.1 1.62 1.38 2.50 1.82 1.83 1.93 1.99 2.47 2.28
16
+ u.2 1.67 1.45 2.54 1.88 1.89 1.98 2.00 2.48 2.26
17
+ rec.2.20 4.39 2.08 2.13 2.27 2.57 2.16 2.32 2.52 2.27
18
+ rec.2.28 5.12 2.00 1.88 1.97 2.02 1.92 2.01 2.15 1.99
19
+ rec.2.36 4.03 1.91 1.88 1.95 1.81 1.89 1.90 2.05 1.87
20
+ rec.2.44 3.58 1.68 1.70 1.82 1.58 1.78 1.74 2.02 1.77
21
+ rec.2.52 3.14 1.68 1.66 1.58 1.58 1.70 1.60 1.94 1.65
22
+ s.2 2.93 1.53 1.49 1.40 1.43 1.62 1.45 1.55 1.46
23
+ out.60 2.57 1.32 1.31 1.35 1.28 1.35 1.26 1.40 1.34
24
+ out.68 1.49 0.94 0.94 1.03 0.95 1.00 1.00 1.00 1.03
25
+
26
+ LENS VARIANTS (skip the first 0 / 4 / 16 source positions) and held-out readouts, positions 16+ of 8 held-out paragraphs
27
+ node |J| split-half cos 0v16 cos 4v16 top10 Jacc 0v16 4v16 agree model: skip0 skip4 skip16 logit lens junk top10 skip0 skip16
28
+ emb 2.2 0.384 0.834 0.854 0.30 0.36 0.00 0.00 0.00 0.00 0.55 0.59
29
+ in.04 3.8 0.508 0.906 0.914 0.48 0.54 0.00 0.00 0.00 0.00 0.62 0.68
30
+ in.12 4.4 0.586 0.935 0.944 0.54 0.62 0.00 0.00 0.01 0.00 0.60 0.66
31
+ x 12.6 0.327 0.967 0.974 0.60 0.67 0.01 0.01 0.02 0.00 0.35 0.30
32
+ u.1 9.8 0.023 0.965 0.969 0.62 0.66 0.00 0.00 0.00 0.00 0.11 0.05
33
+ rec.1.20 2.3 0.085 0.859 0.864 0.35 0.40 0.01 0.01 0.01 0.00 0.34 0.26
34
+ rec.1.28 2.6 0.188 0.924 0.930 0.49 0.54 0.00 0.00 0.01 0.00 0.34 0.29
35
+ rec.1.36 1.3 0.312 0.963 0.976 0.58 0.67 0.00 0.00 0.00 0.01 0.45 0.33
36
+ rec.1.44 1.0 0.419 0.965 0.982 0.48 0.58 0.01 0.01 0.01 0.04 0.44 0.31
37
+ rec.1.52 0.9 0.833 0.979 0.988 0.58 0.66 0.05 0.07 0.10 0.10 0.48 0.36
38
+ s.1 7.1 0.876 0.982 0.989 0.59 0.67 0.11 0.12 0.16 0.00 0.41 0.30
39
+ u.2 10.2 0.881 0.982 0.989 0.69 0.75 0.03 0.03 0.04 0.00 0.53 0.49
40
+ rec.2.20 5.6 0.949 0.982 0.994 0.67 0.78 0.02 0.03 0.04 0.00 0.25 0.23
41
+ rec.2.28 11.7 0.965 0.985 0.995 0.64 0.80 0.02 0.03 0.04 0.00 0.28 0.25
42
+ rec.2.36 10.0 0.973 0.990 0.997 0.71 0.81 0.04 0.04 0.04 0.01 0.26 0.23
43
+ rec.2.44 13.0 0.982 0.993 0.998 0.79 0.86 0.07 0.08 0.08 0.04 0.19 0.17
44
+ rec.2.52 21.7 0.990 0.996 0.999 0.82 0.89 0.23 0.24 0.25 0.11 0.20 0.18
45
+ s.2 175.0 0.993 0.998 0.999 0.86 0.92 0.33 0.34 0.34 0.00 0.15 0.13
46
+ out.60 40.3 0.995 0.999 1.000 0.91 0.95 0.37 0.37 0.38 0.29 0.12 0.10
47
+ out.68 46.5 0.999 1.000 1.000 0.96 0.98 0.51 0.51 0.51 0.47 0.09 0.08
48
+
49
+ PROMPT 'Fact: The currency used in the country shaped like a boot is' intermediate: Italy model top-3: [' the', ' called', ' not']
50
+ in.06 | skip16: rank(Italy)=29917 [' “', '“', ' ‘', '.“', ' „', ' (“'] | logit lens: ['nt', ' “', ' par', ' st']
51
+ in.14 | skip16: rank(Italy)=3758 ['—a', '.', '—', ';', '—the', '—that'] | logit lens: [' both', ' tongue', 'olar', ':']
52
+ x | skip16: rank(Italy)=34006 [' faux', '�', ' entirety', ' chamb', ' known', ' "['] | logit lens: ['raux', 'reros', 'acca', 'acco']
53
+ rec.1.22 | skip16: rank(Italy)=9176 ['[@', 'maßen', 'gare', 'bogen', 'raid', 'lott'] | logit lens: ['ähler', 'untz', ' الإع', 'oux']
54
+ rec.1.30 | skip16: rank(Italy)=1896 ['...\n\n', ' ...\n\n', ' Wikipedia', '..."\n\n', ' [...]', ' [['] | logit lens: [' currently', 'start', 'elier', 'èque']
55
+ rec.1.38 | skip16: rank(Italy)=1115 ['.»\n\n', '.Act', '...\n\n', '..."\n\n', ' ...\n\n', '</think>'] | logit lens: ['ouer', 'ourdes', 'nam', 'inato']
56
+ rec.1.46 | skip16: rank(Italy)=379 [' Wikipedia', '.»\n\n', '</think>', '...\n\n', '.Act', ' Wikiped'] | logit lens: ['erus', ' called', 'called', ' actually']
57
+ rec.1.54 | skip16: rank(Italy)=1284 ['...\n\n', '..."\n\n', ' ...\n\n', ' [...]', ' known', ' [...'] | logit lens: [' called', 'called', ' actually', 'currency']
58
+ s.1 | skip16: rank(Italy)=536 ['...\n\n', ' known', ' likely', ' ...\n\n', '...', '\n\n'] | logit lens: ['ふっ', 'くっ', 'りば', 'ふん']
59
+ rec.2.22 | skip16: rank(Italy)=373 ['...\n\n', '...', '..."', '?', '..."\n\n', '?\n\n'] | logit lens: ['ähler', 'UUID', 'oux', 'kowski']
60
+ rec.2.30 | skip16: rank(Italy)=421 ['?', '?"', '?\n\n', '?"\n\n', '?”', '"?'] | logit lens: ['start', ' currently', 'نج', '</s>']
61
+ rec.2.38 | skip16: rank(Italy)=249 ['?\n', '?\n\n', '?', '?"\n', '...\n', '?”\n\n'] | logit lens: ['ouer', 'nam', 'ourdes', 'inato']
62
+ rec.2.46 | skip16: rank(Italy)=144 ['?\n', '?\n\n', ' currency', '?"\n', '?', '?”\n'] | logit lens: ['erus', ' called', 'called', 'ieras']
63
+ rec.2.54 | skip16: rank(Italy)=270 [' called', ' named', ' currency', ' known', ' actually', ' unknown'] | logit lens: [' called', 'called', ' actually', 'currency']
64
+ s.2 | skip16: rank(Italy)=569 [' called', ' currency', ' known', ' named', ' likely', ' currencies'] | logit lens: ['くっ', 'ふっ', 'りば', 'ふん']
65
+ out.62 | skip16: rank(Italy)=2623 [' called', ' currency', ' USD', ' coins', ' Bitcoin', ' named'] | logit lens: [' called', 'الين', 'called', 'ourdes']
66
+ out.70 | skip16: rank(Italy)=1777 [' called', ' the', ' actually', ' USD', ' peg', ' CFA'] | logit lens: [' the', ' called', ' not', ' actually']
67
+
68
+ PROMPT 'Fact: The number of legs on the animal that spins webs is' intermediate: spider model top-3: [' ', ' eight', ' not']
69
+ in.06 | skip16: rank(spider)=114241 [' “', ' ‘‘', ' „', '.”', '.“', '.'] | logit lens: [' “', 'de', 'nt', ' Morse']
70
+ in.14 | skip16: rank(spider)=99758 ['.', '–', ';', '—', '—a', ' (“'] | logit lens: ['pon', ' tongue', '商务', 'iste']
71
+ x | skip16: rank(spider)=44104 ['.', ':.', '—.', ';', ' Known', ' "['] | logit lens: ['raux', 'acca', 'reros', ' herself']
72
+ rec.1.22 | skip16: rank(spider)=48611 ['[@', 'gare', 'bogen', 'raid', 'maßen', 'oment'] | logit lens: ['UUID', 'ارضة', 'elier', 'ouer']
73
+ rec.1.30 | skip16: rank(spider)=3995 ['...\n\n', ' ...\n\n', '...', '..."\n\n', '…\n\n', '?\n\n'] | logit lens: ['ous', 'ождения', 'atang', ' fewer']
74
+ rec.1.38 | skip16: rank(spider)=19564 ['[@', '...\n\n', '</think>', ' ...\n\n', '.»\n\n', 'stwo'] | logit lens: ['ouer', 'orat', 'ieu', 'orp']
75
+ rec.1.46 | skip16: rank(spider)=8151 ['</think>', '.»\n\n', 'stwo', '.Act', ' …\n\n', '..\n\n'] | logit lens: ['ouden', 'óns', '一动', 'usqu']
76
+ rec.1.54 | skip16: rank(spider)=1441 ['...\n\n', '..."\n\n', '</think>', ' ...\n\n', '\n\n', '"\n\n'] | logit lens: ['無断', '/ajax', '一动', 'ouer']
77
+ s.1 | skip16: rank(spider)=1621 ['...\n\n', '\n\n', ' ...\n\n', '..."\n\n', ' only', '?\n\n'] | logit lens: ['ふっ', 'くっ', 'りば', 'ふん']
78
+ rec.2.22 | skip16: rank(spider)=144 ['...\n\n', '..."\n\n', '..."', '?\n\n', '...', '?"\n\n'] | logit lens: ['UUID', 'ارضة', '-Sah', 'ouer']
79
+ rec.2.30 | skip16: rank(spider)=158 ['?\n\n', '?"\n\n', '?**\n\n', ' weird', '?"', '?**'] | logit lens: ['elier', 'Réal', 'ous', 'ulam']
80
+ rec.2.38 | skip16: rank(spider)=468 ['?\n\n', '?\n', '?”\n\n', '?"\n\n', '?**\n\n', '</think>'] | logit lens: ['ouer', 'ieu', 'inato', 'ourdes']
81
+ rec.2.46 | skip16: rank(spider)=573 ['?\n', '?\n\n', '?', '?"\n', '?”', ':\n'] | logit lens: ['ouden', 'óns', '一动', 'ieras']
82
+ rec.2.54 | skip16: rank(spider)=800 [' unknown', '?\n', ' typically', ' usually', ' exactly', ' likely'] | logit lens: ['/ajax', '無断', '一动', 'ouer']
83
+ s.2 | skip16: rank(spider)=720 [' unknown', '?\n', ' likely', ' irrelevant', ' typically', ' exactly'] | logit lens: ['ふっ', 'くっ', 'りば', 'ふん']
84
+ out.62 | skip16: rank(spider)=20 [' always', ' eight', ' typically', ' ten', '?\n', ' twelve'] | logit lens: [' always', ' typically', ' actually', ' eight']
85
+ out.70 | skip16: rank(spider)=34 [' eight', ' always', ' ', ' eighty', ' actually', ' unknown'] | logit lens: [' ', ' eight', ' **', ' always']
86
+
87
+ PROMPT 'The capital of the country where sushi originated is' intermediate: Japan model top-3: [' Tokyo', ' _', ' what']
88
+ in.06 | skip16: rank(Japan)=68610 [' ...\n\n', ' ...', ' [...', '....\n\n', ' ...\n', ' [...]'] | logit lens: [' Moran', 'elha', ' quot', 'dur']
89
+ in.14 | skip16: rank(Japan)=86699 ['**.', '...\n', '.\n', ':\n', '</think>', '.'] | logit lens: ['nil', ' nil', ' par', 'feld']
90
+ x | skip16: rank(Japan)=49576 ['</think>', '**.', ' (...)', '’.', '”.', '.**'] | logit lens: ['raux', 'acca', 'acco', 'avar']
91
+ rec.1.22 | skip16: rank(Japan)=27713 ['[@', 'gare', 'bogen', ').[', 'maßen', "'.["] | logit lens: ['ähler', 'oje', ' Tort', '�']
92
+ rec.1.30 | skip16: rank(Japan)=27002 ['[@', ' ...\n\n', 'gare', 'lott', 'bogen', 'maßen'] | logit lens: ['uny', 'oum', 'inan', 'ikian']
93
+ rec.1.38 | skip16: rank(Japan)=7101 [' ...\n\n', ' …\n\n', ' [...]', '.»\n\n', ' Wikipedia', '..."\n\n'] | logit lens: ['uny', 'ouer', 'ambo', 'auro']
94
+ rec.1.46 | skip16: rank(Japan)=12818 [' …\n', ' …\n\n', ' …', '.”\n\n', ' Wikipedia', '.»\n\n'] | logit lens: [' answer', 'answer', ' Answer', 'Answer']
95
+ rec.1.54 | skip16: rank(Japan)=850 [' ...\n', ' ...\n\n', ':[', ' Wikipedia', ' [...]', '..."\n\n'] | logit lens: [' _', ' retra', ' what', ' called']
96
+ s.1 | skip16: rank(Japan)=397 ['..."\n\n', ' ...\n\n', ' city', '...\n\n', ' ..', ' Wikipedia'] | logit lens: ['くっ', 'ふっ', 'りば', 'ふん']
97
+ rec.2.22 | skip16: rank(Japan)=5262 [' city', ' City', 'city', ' cities', ' **"', 'City'] | logit lens: ['ähler', 'oje', ' شاه', ' Tort']
98
+ rec.2.30 | skip16: rank(Japan)=2976 [' city', ' cities', '?**', 'city', ' **"', ' City'] | logit lens: ['oum', 'uny', 'نج', 'inan']
99
+ rec.2.38 | skip16: rank(Japan)=534 [':\n', '?\n', '...\n', '?**', ':**', '..."'] | logit lens: ['ambo', 'ouer', 'uny', 'auro']
100
+ rec.2.46 | skip16: rank(Japan)=1714 ['?\n', ':\n', '?",\n', '?"\n', ' ?\n', '?**'] | logit lens: [' answer', 'answer', ' Answer', 'arras']
101
+ rec.2.54 | skip16: rank(Japan)=207 [':\n', ' ___', '?\n', ' City', ' Tokyo', ' city'] | logit lens: [' retra', ' what', ' _', ' called']
102
+ s.2 | skip16: rank(Japan)=369 [' ___', ':\n', ' _', '?\n', ' Tokyo', '____'] | logit lens: ['くっ', 'ふっ', 'りば', 'ふん']
103
+ out.62 | skip16: rank(Japan)=3 [' Tokyo', 'Tokyo', ' Japan', ' Kyoto', ' Osaka', ' Jakarta'] | logit lens: [' Tokyo', ' Japan', 'urgo', ' Kyoto']
104
+ out.70 | skip16: rank(Japan)=8 [' Tokyo', ' ___', ' Edo', ' Kyoto', ' _', ':\n'] | logit lens: [' Tokyo', ' _', ' ___', ' what']
105
+
106
+ PROMPT 'Fact: The color of the planet fourth from the Sun is' intermediate: Mars model top-3: [' orange', ' green', ' not']
107
+ in.06 | skip16: rank(Mars)=2494 [' “', ' (“', '“', '.', '”', '.”'] | logit lens: [' dur', ' st', 'idus', ' Wend']
108
+ in.14 | skip16: rank(Mars)=90 ['.', ';', '?', ':', '.\n', ' Himal'] | logit lens: [' again', ' pall', ' po', ' captured']
109
+ x | skip16: rank(Mars)=26765 [' Himal', 'arcie', ' Known', ' chamb', ' Sep', ' Hue'] | logit lens: ['raux', 'acca', 'reros', ' herself']
110
+ rec.1.22 | skip16: rank(Mars)=5901 ['gare', '[@', 'bogen', 'inite', 'maßen', 'raid'] | logit lens: ['मान', 'chev', ' now', ' structure']
111
+ rec.1.30 | skip16: rank(Mars)=3103 [' ...\n\n', '[@', 'gare', '?v', 'lott', 'eron'] | logit lens: ['نج', 'atang', 'uny', 'start']
112
+ rec.1.38 | skip16: rank(Mars)=1174 [' ...\n\n', '</think>', '.»\n\n', ' [...]', '...\n\n', '..."\n\n'] | logit lens: ['ouer', 'ikian', 'ourde', 'oué']
113
+ rec.1.46 | skip16: rank(Mars)=941 ['</think>', '.""', ' Wikipedia', '.»\n\n', '.[', ',['] | logit lens: [' named', 'ieras', 'avas', ' name']
114
+ rec.1.54 | skip16: rank(Mars)=2771 ['.[', ':[', '..."\n\n', '."\n\n', ' [...', ' known'] | logit lens: [' not', '아버', ' actually', ' color']
115
+ s.1 | skip16: rank(Mars)=452 ['...\n\n', ' named', ' unknown', ' known', ' [[', ' ...\n\n'] | logit lens: ['ふっ', 'くっ', 'りば', 'ふん']
116
+ rec.2.22 | skip16: rank(Mars)=34 ['..."', ' Earth', ' colors', ' colored', '...', ' meteor'] | logit lens: [' now', 'मान', ' الإع', 'chev']
117
+ rec.2.30 | skip16: rank(Mars)=115 ['?"\n\n', '?"', '?\n\n', '?', '?**\n\n', '?**'] | logit lens: ['نج', 'atang', 'uny', 'start']
118
+ rec.2.38 | skip16: rank(Mars)=85 ['?\n', '?\n\n', '..."', '?"', '?"\n\n', '"?'] | logit lens: ['ikian', 'ouer', 'ourde', 'oué']
119
+ rec.2.46 | skip16: rank(Mars)=46 ['?\n\n', '?\n', ' named', '?', ' colors', ' name'] | logit lens: ['ieras', ' likely', 'avas', ' not']
120
+ rec.2.54 | skip16: rank(Mars)=82 [' unknown', '?\n\n', ' green', ' named', '?\n', ' unnamed'] | logit lens: [' not', 'ikian', '아버', ' actually']
121
+ s.2 | skip16: rank(Mars)=30 [' unknown', ' ___', ' unnamed', ' green', '?\n', '?\n\n'] | logit lens: ['ふっ', 'くっ', 'りば', 'ふん']
122
+ out.62 | skip16: rank(Mars)=27 [' yellow', ' orange', ' green', ' brown', ' redd', ' blue'] | logit lens: [' orange', ' yellow', ' brown', ' redd']
123
+ out.70 | skip16: rank(Mars)=13 [' redd', ' yellow', ' orange', ' brown', ' Mercury', ' Venus'] | logit lens: [' orange', ' blue', ' yellow', ' redd']
124
+
125
+ PROMPT 'es 23? Answer briefly.<|im_end|>\n<|im_start|>assistant\n<think></think>' intermediate: None model top-3: ['4', '1', '3']
126
+ in.06 | skip16: [' *“', ' **', ' **[', ' **"', ' “', ' *„'] | logit lens: [' Rok', 'enia', 'چه', ' DEN']
127
+ in.14 | skip16: [' “', ' **', ' *“', ' **[', ' ``', ' **"'] | logit lens: ['stadt', 'tn', 'umont', 'oty']
128
+ x | skip16: [' “', '“', ' (“', ' *“', ' **', ' ``'] | logit lens: ['inos', '्त', 'onis', ' gola']
129
+ rec.1.22 | skip16: [' **[', '[@', ').[', '[[', ' [[', '.['] | logit lens: ['tha', '�', 'onis', 'kis']
130
+ rec.1.30 | skip16: [' **[', '**[', ' *“', ':[', ' **«', '[['] | logit lens: ['atang', 'ocke', 'one', 'antan']
131
+ rec.1.38 | skip16: ['[@', ' **[', '.[', ').[', '[[', '**['] | logit lens: ['uad', '/checkout', 'ikian', 'ourdes']
132
+ rec.1.46 | skip16: [':[', ' **[', '</think>', ' \\$', ' [_', ' "**'] | logit lens: ['uad', ' contex', 'endit', ' Córd']
133
+ rec.1.54 | skip16: [' ', ' \\$', '5', '</think>', '3', '4'] | logit lens: ['disf', 'anse', '無断', ' 結果']
134
+ s.1 | skip16: [' ', ' $[', ' \\$', ' millions', ' $', ' billions'] | logit lens: ['ふっ', 'くっ', 'りば', 'ふん']
135
+ rec.2.22 | skip16: [' **', ' **[', ' **"', ' *“', ' **(', ' (**'] | logit lens: ['tha', '�', 'amar', 'onis']
136
+ rec.2.30 | skip16: [' **', ' **"', ' **[', ' **(', ' *“', ' **«'] | logit lens: ['atang', 'ocke', 'antan', 'one']
137
+ rec.2.38 | skip16: [' **[', ' **', ' **"', ' **(', ' “', '\u202f'] | logit lens: ['ourdes', 'uad', '/checkout', 'elio']
138
+ rec.2.46 | skip16: [' Answer', ' answer', ' ', ' Approximately', ' approximately', ' seventy'] | logit lens: ['uad', 'eggi', 'endit', ' contex']
139
+ rec.2.54 | skip16: [' ', '4', '5', '3', '6', '7'] | logit lens: ['disf', 'reform', 'anse', '��']
140
+ s.2 | skip16: [' ', '4', '5', '3', '6', '1'] | logit lens: ['ふっ', 'くっ', 'りば', 'ふん']
141
+ out.62 | skip16: ['4', ' ', '3', '1', '5', ' forty'] | logit lens: ['onin', '4', 'akume', ' ']
142
+ out.70 | skip16: ['4', '1', '3', '5', 'The', '6'] | logit lens: ['4', '1', '3', '5']
fit_records/k2_n50_skip16_s0/run.json ADDED
@@ -0,0 +1,25 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "started": "2026-11-15T22:57:00+0000",
3
+ "finished": "2026-11-15T23:30:05+0000",
4
+ "packages": {
5
+ "torch": "2.14.0+cu126",
6
+ "transformers": "5.17.0",
7
+ "peft": "0.21.0",
8
+ "numpy": "2.4.6",
9
+ "safetensors": "0.8.0"
10
+ },
11
+ "cuda_visible_devices": "1",
12
+ "config": {
13
+ "k": 2,
14
+ "n_prompts": 50,
15
+ "seq_len": 128,
16
+ "chunk": 256,
17
+ "skip": 16,
18
+ "seed": 0,
19
+ "run": "k2_n50_skip16_s0",
20
+ "pilot": false,
21
+ "tf32": true,
22
+ "dtype": "float32",
23
+ "corpus": "WikiText-103 raw, validation split (Salesforce/wikitext)"
24
+ }
25
+ }
fit_records/k3_n50_skip16_s0/metrics.json ADDED
The diff for this file is too large to render. See raw diff
 
fit_records/k3_n50_skip16_s0/prompts.json ADDED
@@ -0,0 +1,202 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [
2
+ {
3
+ "i": 0,
4
+ "start": "Given the chaos , the governor of Yunnan ignored the imperial orders of evacuation . The Mongol army commanded by Prince Ye @"
5
+ },
6
+ {
7
+ "i": 1,
8
+ "start": "From the 1920s Australia 's defence thinking was dominated by British Imperial defence policy , which was embodied"
9
+ },
10
+ {
11
+ "i": 2,
12
+ "start": "Tim 's driving days started as a toddler when he was given a go @-@ kart that he often drove"
13
+ },
14
+ {
15
+ "i": 3,
16
+ "start": "As the Marines formed into a horseshoe @-@ shaped line around Hill 123 , Tamura 's"
17
+ },
18
+ {
19
+ "i": 4,
20
+ "start": "USA Today 's Scott Bowles awarded the film three out of four stars , labeling it a \" surprising comedy that rivals"
21
+ },
22
+ {
23
+ "i": 5,
24
+ "start": "Ringo Starr identified his initial inspiration for \" Back Off Boogaloo \" as having come from Marc Bolan ,"
25
+ },
26
+ {
27
+ "i": 6,
28
+ "start": "The ship 's secondary armament consisted of fourteen 45 @-@ calibre 6 @-@ inch ("
29
+ },
30
+ {
31
+ "i": 7,
32
+ "start": "Sources disagree about Radcliffe 's personal wealth ; he was reported to have earned \u00a3 1 million for the first Harry"
33
+ },
34
+ {
35
+ "i": 8,
36
+ "start": "In April 2010 , there were 6 @,@ 260 people employed in the healthcare field"
37
+ },
38
+ {
39
+ "i": 9,
40
+ "start": "\" Exit Through the Kwik @-@ E @-@ Mart \" is the fifteenth episode of the twenty @-"
41
+ },
42
+ {
43
+ "i": 10,
44
+ "start": "On 25 June 1950 , the North Korean Army ( KPA ) crossed the border into South"
45
+ },
46
+ {
47
+ "i": 11,
48
+ "start": "A British parliamentary motion put forward by Labour MP Kate Hoey in February 2013 to retrospectively condemn the"
49
+ },
50
+ {
51
+ "i": 12,
52
+ "start": "At the fort , Moultrie ordered his men to concentrate their fire on the two large man @-@ of @"
53
+ },
54
+ {
55
+ "i": 13,
56
+ "start": "In Part II of the series , Shikamaru is assigned the task of locating two members of the criminal organization Ak"
57
+ },
58
+ {
59
+ "i": 14,
60
+ "start": "The Ming dynasty had seen considerable advancement in the process of colour printing in China . At his studio , Hu Zhengyan experiment"
61
+ },
62
+ {
63
+ "i": 15,
64
+ "start": "\" Exit Through the Kwik @-@ E @-@ Mart \" was written by Marc Wilmore and directed by"
65
+ },
66
+ {
67
+ "i": 16,
68
+ "start": "In 629 Tikal founded Dos Pilas , some 110 kilometres ( 68 mi )"
69
+ },
70
+ {
71
+ "i": 17,
72
+ "start": "The British fleet weighed anchor at Cape Fear on May 31 , and arrived outside Charleston Harbor the next day . Moul"
73
+ },
74
+ {
75
+ "i": 18,
76
+ "start": "The initial phase saw the Australians advance towards the Hongorai River . Following the end of the early fighting , the Australian"
77
+ },
78
+ {
79
+ "i": 19,
80
+ "start": "The reactions of the native Aboriginal inhabitants to the sudden arrival of British settlers in Australia were varied , but were inevitably hostile when"
81
+ },
82
+ {
83
+ "i": 20,
84
+ "start": "The German offensive ground to a halt in mid @-@ July and a brief lull followed , during which the Australians"
85
+ },
86
+ {
87
+ "i": 21,
88
+ "start": "The city 's growth has reflected the push and pull of many social and economic factors . The total population increased in each"
89
+ },
90
+ {
91
+ "i": 22,
92
+ "start": "Following the classification of Tropical Depression Two @-@ E on June 19 , the Government of Mexico issued a tropical"
93
+ },
94
+ {
95
+ "i": 23,
96
+ "start": "The Natives remained in London following the England international . They defeated London Welsh on 18 February , before losing first"
97
+ },
98
+ {
99
+ "i": 24,
100
+ "start": "In February 1997 , a United States immigration court judge granted asylum to a German Scientologist who claimed she"
101
+ },
102
+ {
103
+ "i": 25,
104
+ "start": "Some of the most important trade routes began at Canton and Macau in Qing Dynasty China . Early in each year a large convoy"
105
+ },
106
+ {
107
+ "i": 26,
108
+ "start": "The fourth installment in the Slammiversary name was announced in January 2008 to take place on June"
109
+ },
110
+ {
111
+ "i": 27,
112
+ "start": "Capel Lligwy has national recognition and statutory protection from alteration as it has been designated as a Grade II listed building"
113
+ },
114
+ {
115
+ "i": 28,
116
+ "start": "The Trout Creek Mountains are semi @-@ arid because they are in the eastern rain shadow of mountain ranges to"
117
+ },
118
+ {
119
+ "i": 29,
120
+ "start": "Asahi , meaning \" rising sun \" , a poetic name for Japan from a stanza of waka poetry , was laid"
121
+ },
122
+ {
123
+ "i": 30,
124
+ "start": "Beckham 's term as governor ended December 10 , 1907 . In January 19"
125
+ },
126
+ {
127
+ "i": 31,
128
+ "start": "Donald Sutherland portrays Jack Pellitt , Bobby 's father and Kurt 's boss . On July 27 ,"
129
+ },
130
+ {
131
+ "i": 32,
132
+ "start": "As British offshoots , the Australian colonies participated in Britain 's small wars of the 19th century"
133
+ },
134
+ {
135
+ "i": 33,
136
+ "start": "Retail is another major employer in the county , with 5 @,@ 280 people employed in April "
137
+ },
138
+ {
139
+ "i": 34,
140
+ "start": "The tour returned to Pocono a month later , and Richmond battled for another victory in a fog @-@ shortened"
141
+ },
142
+ {
143
+ "i": 35,
144
+ "start": "\" ... Thirteen Years Later \" received mixed reviews from critics . The A.V. Club 's Todd VanDerWer"
145
+ },
146
+ {
147
+ "i": 36,
148
+ "start": "Bonnie Stiernberg , for Paste , was more positive , writing that she preferred the use of comedy , following the"
149
+ },
150
+ {
151
+ "i": 37,
152
+ "start": "In common with many small Scottish islands , Papa Stour 's population peaked in the 19th century and has"
153
+ },
154
+ {
155
+ "i": 38,
156
+ "start": "The unexpected deployment to East Timor in 1999 led to significant changes in Australian defence policy and to an enhancement"
157
+ },
158
+ {
159
+ "i": 39,
160
+ "start": "Government publications on the dangers of sects increased between 1996 and 1998 , and"
161
+ },
162
+ {
163
+ "i": 40,
164
+ "start": "Richmond started 1982 without a ride before getting a one @-@ race deal to drive for Billie"
165
+ },
166
+ {
167
+ "i": 41,
168
+ "start": "The music for Sonic the Hedgehog was composed and produced by Masato Nakamura , bassist and lead songwriter of the"
169
+ },
170
+ {
171
+ "i": 42,
172
+ "start": "Apple Records issued the single on 17 March 1972 in Britain , as Apple R 59"
173
+ },
174
+ {
175
+ "i": 43,
176
+ "start": "Meridian is home to two post @-@ secondary educational institutions . Meridian Community College , founded in 19"
177
+ },
178
+ {
179
+ "i": 44,
180
+ "start": "Ayola began her career in the theatre , performing in a number of Shakespearean plays including Twelfth Night , A M"
181
+ },
182
+ {
183
+ "i": 45,
184
+ "start": "Cooke 's opportunistic diversion had determined that the Spanish forces in the Philippines posed no immediate threat , although the mission"
185
+ },
186
+ {
187
+ "i": 46,
188
+ "start": "To further justify its stance , the German government has also pointed to the long history of U.S. court cases involving Scient"
189
+ },
190
+ {
191
+ "i": 47,
192
+ "start": "Headed by police chief Lee Shelbourn since 2009 , the Meridian Police Department consists of "
193
+ },
194
+ {
195
+ "i": 48,
196
+ "start": "A port , Sonic the Hedgehog Genesis , was released for the Game Boy Advance ( GBA ) on November"
197
+ },
198
+ {
199
+ "i": 49,
200
+ "start": "As is often the case with huge ancient ruins , knowledge of the site was never completely lost in the region . It seems"
201
+ }
202
+ ]
fit_records/k3_n50_skip16_s0/readouts.txt ADDED
@@ -0,0 +1,186 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ non-semantic tokens (no letter or digit): 5208 of 131072
2
+
3
+ EARLY POSITIONS. Jacobian size by source position, relative to the median over positions 16+ (k=3, 50 prompts)
4
+ node p0 p1 p2 p3 p4 p5 p6 p7 p8-15 mean
5
+ emb 8.96 3.14 4.05 2.40 3.56 4.47 3.94 4.79 10.14
6
+ in.04 3.92 2.15 3.96 1.57 2.07 5.24 7.34 4.14 7.60
7
+ in.12 4.92 1.27 4.46 1.74 2.33 5.29 7.53 6.01 8.49
8
+ x 2.73 0.93 3.02 1.06 1.38 4.40 5.77 6.61 7.05
9
+ u.1 2.66 0.74 1.38 1.05 1.37 4.38 5.77 6.58 7.00
10
+ rec.1.20 5.19 1.41 1.13 1.42 1.74 4.48 10.77 6.74 10.83
11
+ rec.1.28 7.58 1.21 1.13 0.84 0.75 5.47 8.90 3.87 11.57
12
+ rec.1.36 18.37 4.53 5.06 1.51 1.49 5.34 19.67 3.06 11.37
13
+ rec.1.44 30.48 11.48 10.70 0.99 1.20 4.30 4.53 5.55 6.42
14
+ rec.1.52 23.49 9.81 40.90 0.79 1.14 2.99 3.27 2.47 4.88
15
+ s.1 11.14 12.51 53.45 0.89 1.14 3.33 2.43 2.63 5.73
16
+ u.2 11.43 12.68 53.26 0.91 1.17 3.38 2.56 2.54 5.80
17
+ rec.2.20 7.14 1.58 25.05 1.93 3.12 3.41 4.81 4.12 7.43
18
+ rec.2.28 4.66 1.73 1.68 1.74 2.79 5.10 3.47 2.75 6.04
19
+ rec.2.36 3.89 1.92 1.97 1.96 3.90 6.44 2.23 2.20 3.29
20
+ rec.2.44 3.47 1.83 1.99 2.02 4.10 5.80 1.89 2.30 2.29
21
+ rec.2.52 2.30 1.56 1.90 2.31 3.25 7.48 2.04 2.56 2.31
22
+ s.2 1.79 1.37 1.82 2.49 2.71 8.52 2.00 2.56 2.27
23
+ u.3 1.85 1.44 1.90 2.55 2.79 8.47 2.02 2.58 2.26
24
+ rec.3.20 4.35 2.07 2.11 2.26 2.59 3.06 2.29 2.52 2.24
25
+ rec.3.28 5.09 2.02 1.90 1.99 2.05 1.95 2.02 2.18 2.01
26
+ rec.3.36 4.11 1.97 1.95 2.01 1.88 1.98 1.96 2.12 1.94
27
+ rec.3.44 3.59 1.70 1.73 1.84 1.60 1.82 1.77 2.05 1.80
28
+ rec.3.52 3.13 1.69 1.67 1.59 1.59 1.72 1.62 1.94 1.67
29
+ s.3 2.91 1.53 1.49 1.40 1.43 1.62 1.45 1.54 1.46
30
+ out.60 2.56 1.33 1.31 1.35 1.29 1.34 1.27 1.39 1.35
31
+ out.68 1.48 0.94 0.94 1.03 0.95 1.00 0.99 1.00 1.03
32
+
33
+ LENS VARIANTS (skip the first 0 / 4 / 16 source positions) and held-out readouts, positions 16+ of 8 held-out paragraphs
34
+ node |J| split-half cos 0v16 cos 4v16 top10 Jacc 0v16 4v16 agree model: skip0 skip4 skip16 logit lens junk top10 skip0 skip16
35
+ emb 35.3 0.016 0.706 0.720 0.51 0.52 0.00 0.00 0.00 0.00 0.11 0.14
36
+ in.04 53.6 0.027 0.776 0.782 0.55 0.55 0.00 0.00 0.00 0.00 0.12 0.15
37
+ in.12 57.3 0.028 0.815 0.820 0.55 0.54 0.00 0.00 0.00 0.00 0.20 0.22
38
+ x 259.3 0.010 0.967 0.970 0.82 0.83 0.00 0.00 0.00 0.00 0.16 0.15
39
+ u.1 258.9 -0.002 0.967 0.970 0.91 0.91 0.00 0.00 0.00 0.00 0.10 0.10
40
+ rec.1.20 60.6 0.005 0.852 0.856 0.68 0.72 0.00 0.00 0.00 0.00 0.10 0.10
41
+ rec.1.28 63.9 0.010 0.914 0.917 0.81 0.82 0.00 0.00 0.00 0.00 0.11 0.11
42
+ rec.1.36 26.7 0.012 0.970 0.977 0.97 0.97 0.00 0.00 0.00 0.01 0.10 0.10
43
+ rec.1.44 17.2 0.016 0.973 0.991 0.95 0.96 0.00 0.00 0.00 0.04 0.10 0.10
44
+ rec.1.52 4.3 0.009 0.980 0.995 0.72 0.78 0.00 0.00 0.00 0.10 0.19 0.15
45
+ s.1 23.2 0.006 0.972 0.991 0.69 0.74 0.00 0.00 0.00 0.00 0.16 0.14
46
+ u.2 30.6 0.007 0.967 0.989 0.60 0.67 0.00 0.00 0.00 0.00 0.24 0.20
47
+ rec.2.20 0.9 0.295 0.784 0.889 0.10 0.54 0.01 0.01 0.01 0.00 0.31 0.40
48
+ rec.2.28 1.2 0.542 0.939 0.951 0.45 0.56 0.01 0.01 0.02 0.00 0.56 0.50
49
+ rec.2.36 0.8 0.660 0.962 0.975 0.51 0.60 0.02 0.02 0.02 0.01 0.35 0.26
50
+ rec.2.44 0.7 0.731 0.967 0.980 0.49 0.59 0.03 0.04 0.04 0.04 0.37 0.24
51
+ rec.2.52 0.9 0.879 0.977 0.985 0.57 0.64 0.05 0.06 0.09 0.11 0.49 0.34
52
+ s.2 7.0 0.901 0.980 0.987 0.53 0.61 0.07 0.08 0.12 0.00 0.43 0.28
53
+ u.3 10.1 0.903 0.979 0.987 0.67 0.72 0.03 0.03 0.04 0.00 0.54 0.47
54
+ rec.3.20 5.6 0.950 0.982 0.994 0.67 0.78 0.02 0.03 0.04 0.00 0.25 0.22
55
+ rec.3.28 11.6 0.966 0.985 0.995 0.65 0.79 0.02 0.03 0.04 0.00 0.28 0.24
56
+ rec.3.36 10.0 0.973 0.990 0.997 0.70 0.81 0.04 0.04 0.04 0.01 0.26 0.23
57
+ rec.3.44 13.0 0.982 0.993 0.998 0.78 0.86 0.08 0.08 0.09 0.04 0.19 0.17
58
+ rec.3.52 21.8 0.990 0.996 0.999 0.82 0.89 0.23 0.25 0.26 0.11 0.20 0.18
59
+ s.3 174.9 0.993 0.998 0.999 0.86 0.91 0.33 0.34 0.35 0.00 0.15 0.13
60
+ out.60 40.3 0.995 0.999 1.000 0.91 0.95 0.36 0.38 0.38 0.29 0.12 0.10
61
+ out.68 46.6 0.999 1.000 1.000 0.96 0.98 0.51 0.51 0.51 0.47 0.09 0.08
62
+
63
+ PROMPT 'Fact: The currency used in the country shaped like a boot is' intermediate: Italy model top-3: [' the', ' called', ' not']
64
+ in.06 | skip16: rank(Italy)=112145 ['nem', 'adet', "'.[", ' Opening', ' Closed', 'дей'] | logit lens: ['nt', ' “', ' par', ' st']
65
+ in.14 | skip16: rank(Italy)=108721 ['nem', "'.[", 'adet', ' Opening', 'volta', 'дей'] | logit lens: [' both', ' tongue', 'olar', ':']
66
+ x | skip16: rank(Italy)=112946 ['adet', 'nem', "'.[", ' Opening', ' Closed', 'дей'] | logit lens: ['raux', 'reros', 'acca', 'acco']
67
+ rec.1.22 | skip16: rank(Italy)=3450 [' bre', ' Iwan', ' unit', ' Wilde', ' gain', ' conquer'] | logit lens: ['ähler', 'untz', ' الإع', 'oux']
68
+ rec.1.30 | skip16: rank(Italy)=3398 [' bre', ' unit', ' Wilde', ' Iwan', ' gain', ' conquer'] | logit lens: [' currently', 'start', 'elier', 'èque']
69
+ rec.1.38 | skip16: rank(Italy)=3425 [' bre', ' unit', ' Wilde', ' Iwan', ' gain', ' conquer'] | logit lens: ['ouer', 'ourdes', 'nam', 'inato']
70
+ rec.1.46 | skip16: rank(Italy)=3303 [' bre', ' Wilde', ' unit', ' conquer', ' Iwan', ' gain'] | logit lens: ['erus', ' called', 'called', ' actually']
71
+ rec.1.54 | skip16: rank(Italy)=113685 ['nem', 'adet', "'.[", ' Opening', ' Closed', 'дей'] | logit lens: [' called', 'called', ' actually', 'currency']
72
+ s.1 | skip16: rank(Italy)=3934 [' bre', ' Wilde', ' unit', ' Iwan', ' conquer', 'ieck'] | logit lens: ['ふっ', 'くっ', 'りば', 'ふん']
73
+ rec.2.22 | skip16: rank(Italy)=573 [' claim', '...', ' bre', ' Sard', ' gold', ' Wilde'] | logit lens: ['ähler', 'UUID', 'oux', 'kowski']
74
+ rec.2.30 | skip16: rank(Italy)=158 ['...\n\n', '...', '..."\n\n', '.”\n\n', ' ...\n\n', '?\n\n'] | logit lens: ['start', ' currently', 'نج', '</s>']
75
+ rec.2.38 | skip16: rank(Italy)=297 ['...\n\n', '..."\n\n', '...', '?\n\n', '.\n\n', '.”\n\n'] | logit lens: ['ouer', 'nam', 'ourdes', 'inato']
76
+ rec.2.46 | skip16: rank(Italy)=214 ['...\n\n', '..."\n\n', '.»\n\n', '."\n\n', ' Wikipedia', '?\n\n'] | logit lens: ['erus', ' called', 'called', 'ieras']
77
+ rec.2.54 | skip16: rank(Italy)=1426 ['..."\n\n', '...\n\n', ' [...]', ' [...', ' ...\n\n', '</think>'] | logit lens: [' called', 'called', ' actually', 'currency']
78
+ s.2 | skip16: rank(Italy)=702 ['...\n\n', ' likely', '..."\n\n', '“\n\n', ' known', ' unknown'] | logit lens: ['くっ', 'ふっ', 'りば', 'ふん']
79
+ rec.3.22 | skip16: rank(Italy)=403 ['..."', '...\n\n', '...', '..."\n\n', '?', '?\n\n'] | logit lens: ['ähler', 'kowski', 'UUID', 'enaar']
80
+ rec.3.30 | skip16: rank(Italy)=434 ['?', '?"', '?\n\n', '?"\n\n', '?”', '"?'] | logit lens: ['start', 'نج', ' currently', '</s>']
81
+ rec.3.38 | skip16: rank(Italy)=240 ['?\n', '?\n\n', '?', '?"\n', '...\n', '?”\n\n'] | logit lens: ['ouer', 'nam', 'ierro', 'ourdes']
82
+ rec.3.46 | skip16: rank(Italy)=140 ['?\n', '?\n\n', ' currency', '?', '?"\n', '?”\n'] | logit lens: ['erus', ' called', 'called', 'ieras']
83
+ rec.3.54 | skip16: rank(Italy)=266 [' called', ' named', ' currency', ' known', ' actually', 'called'] | logit lens: [' called', 'called', ' actually', 'currency']
84
+ s.3 | skip16: rank(Italy)=581 [' called', ' currency', ' known', ' named', ' currencies', ' likely'] | logit lens: ['くっ', 'ふっ', 'りば', 'ふん']
85
+ out.62 | skip16: rank(Italy)=2932 [' called', ' currency', ' USD', ' named', ' coins', ' Bitcoin'] | logit lens: [' called', 'الين', 'called', 'ourdes']
86
+ out.70 | skip16: rank(Italy)=2012 [' called', ' the', ' USD', ' actually', ' peg', ' CFA'] | logit lens: [' the', ' called', ' not', ' unique']
87
+
88
+ PROMPT 'Fact: The number of legs on the animal that spins webs is' intermediate: spider model top-3: [' ', ' eight', ' not']
89
+ in.06 | skip16: rank(spider)=126785 ['nem', "'.[", 'adet', ' Opening', 'poca', ' Closed'] | logit lens: [' “', 'de', 'nt', ' Morse']
90
+ in.14 | skip16: rank(spider)=124256 ['nem', 'adet', "'.[", ' Opening', ' Closed', 'дей'] | logit lens: ['pon', ' tongue', '商务', 'iste']
91
+ x | skip16: rank(spider)=122784 ['adet', 'nem', "'.[", ' Opening', ' Closed', 'дей'] | logit lens: ['raux', 'acca', 'reros', ' herself']
92
+ rec.1.22 | skip16: rank(spider)=7428 [' bre', ' Iwan', ' unit', ' Wilde', ' gain', ' conquer'] | logit lens: ['UUID', 'ارضة', 'elier', 'ouer']
93
+ rec.1.30 | skip16: rank(spider)=7503 [' bre', ' Wilde', ' unit', ' Iwan', ' conquer', ' gain'] | logit lens: ['ous', 'ождения', 'atang', ' fewer']
94
+ rec.1.38 | skip16: rank(spider)=7419 [' bre', ' unit', ' Wilde', ' Iwan', ' gain', ' conquer'] | logit lens: ['ouer', 'orat', 'ieu', 'orp']
95
+ rec.1.46 | skip16: rank(spider)=7341 [' bre', ' unit', ' Wilde', ' Iwan', ' gain', ' conquer'] | logit lens: ['ouden', 'óns', '一动', 'usqu']
96
+ rec.1.54 | skip16: rank(spider)=119879 ['nem', 'adet', "'.[", ' Opening', ' Closed', 'дей'] | logit lens: ['無断', '/ajax', '一动', 'ouer']
97
+ s.1 | skip16: rank(spider)=6618 [' bre', ' Wilde', ' unit', ' Iwan', ' conquer', ' gain'] | logit lens: ['ふっ', 'くっ', 'りば', 'ふん']
98
+ rec.2.22 | skip16: rank(spider)=416 ['...\n\n', '...', 'mx', ' bre', 'RD', ' Sard'] | logit lens: ['UUID', 'ارضة', '-Sah', 'ouer']
99
+ rec.2.30 | skip16: rank(spider)=268 ['...\n\n', '..."\n\n', ' ...\n\n', '.”\n\n', '....\n\n', '…\n\n'] | logit lens: ['elier', 'Réal', 'ous', 'ulam']
100
+ rec.2.38 | skip16: rank(spider)=399 ['...\n\n', '.\n\n', '.”\n\n', '?\n\n', '…\n\n', '..."\n\n'] | logit lens: ['ouer', 'ieu', 'inato', 'ourdes']
101
+ rec.2.46 | skip16: rank(spider)=1154 ['</think>', '...\n\n', '.”\n\n', '.\n\n', '."\n\n', '…\n\n'] | logit lens: ['ouden', 'óns', '一动', 'ieras']
102
+ rec.2.54 | skip16: rank(spider)=1580 ['...\n\n', '</think>', '..."\n\n', ' ...\n\n', '"\n\n', '."\n\n'] | logit lens: ['/ajax', '無断', '一动', 'ouer']
103
+ s.2 | skip16: rank(spider)=1031 ['...\n\n', ' only', '..."\n\n', '</think>', ' unknown', '\n\n'] | logit lens: ['ふっ', 'くっ', 'りば', 'ふん']
104
+ rec.3.22 | skip16: rank(spider)=147 ['...\n\n', '..."\n\n', '..."', '?\n\n', '...', '?"\n\n'] | logit lens: ['UUID', 'ارضة', '-Sah', 'ouer']
105
+ rec.3.30 | skip16: rank(spider)=164 ['?\n\n', '?"\n\n', '?**\n\n', ' weird', '?"', '?**'] | logit lens: ['elier', 'ous', 'Réal', 'ulam']
106
+ rec.3.38 | skip16: rank(spider)=481 ['?\n\n', '?\n', '?”\n\n', '?"\n\n', '?**\n\n', '?'] | logit lens: ['ouer', 'ieu', 'inato', 'ierro']
107
+ rec.3.46 | skip16: rank(spider)=578 ['?\n', '?\n\n', '?', ':\n', '?"\n', '?”'] | logit lens: ['ouden', 'óns', 'ieras', '一动']
108
+ rec.3.54 | skip16: rank(spider)=880 [' unknown', '?\n', ' typically', ' exactly', ' usually', ' likely'] | logit lens: ['/ajax', '無断', '一动', 'ouer']
109
+ s.3 | skip16: rank(spider)=836 ['?\n', ' unknown', ' likely', ' irrelevant', ' exactly', ' usually'] | logit lens: ['ふっ', 'くっ', 'りば', 'ふん']
110
+ out.62 | skip16: rank(spider)=28 [' always', ' eight', '?\n', ' equal', ' twelve', ' typically'] | logit lens: [' always', ' typically', ' eight', ' equal']
111
+ out.70 | skip16: rank(spider)=42 [' eight', ' always', ' ', ' actually', ' unknown', ' eighty'] | logit lens: [' ', ' eight', ' always', ' **']
112
+
113
+ PROMPT 'The capital of the country where sushi originated is' intermediate: Japan model top-3: [' Tokyo', ' _', ' what']
114
+ in.06 | skip16: rank(Japan)=5843 [' bre', ' Iwan', ' conquer', 'ieck', ' Wilde', ' gain'] | logit lens: [' Moran', 'elha', ' quot', 'dur']
115
+ in.14 | skip16: rank(Japan)=5037 [' bre', ' conquer', ' Iwan', ' gain', '_m', 'isp'] | logit lens: ['nil', ' nil', ' par', 'feld']
116
+ x | skip16: rank(Japan)=5632 [':', '...', '?', '!', ' attribute', '.'] | logit lens: ['raux', 'acca', 'acco', 'avar']
117
+ rec.1.22 | skip16: rank(Japan)=4886 [' bre', ' Iwan', ' unit', ' Wilde', ' gain', ' conquer'] | logit lens: ['ähler', 'oje', ' Tort', '�']
118
+ rec.1.30 | skip16: rank(Japan)=4985 [' bre', ' Iwan', ' unit', ' Wilde', ' gain', ' conquer'] | logit lens: ['uny', 'oum', 'inan', 'ikian']
119
+ rec.1.38 | skip16: rank(Japan)=4923 [' bre', ' Wilde', ' unit', ' Iwan', ' gain', ' conquer'] | logit lens: ['uny', 'ouer', 'ambo', 'auro']
120
+ rec.1.46 | skip16: rank(Japan)=102837 ['nem', 'adet', ' Opening', "'.[", ' Closed', 'дей'] | logit lens: [' answer', 'answer', ' Answer', 'Answer']
121
+ rec.1.54 | skip16: rank(Japan)=102083 ['nem', 'adet', "'.[", ' Opening', ' Closed', 'дей'] | logit lens: [' _', ' retra', ' what', ' called']
122
+ s.1 | skip16: rank(Japan)=4692 [' bre', ' conquer', ' unit', ' Wilde', ' engineer', ' claim'] | logit lens: ['くっ', 'ふっ', 'りば', 'ふん']
123
+ rec.2.22 | skip16: rank(Japan)=785 ['...\n\n', '...', ' bre', ' Sard', ' name', ' Hubert'] | logit lens: ['ähler', 'oje', ' شاه', ' Tort']
124
+ rec.2.30 | skip16: rank(Japan)=589 [' ...\n\n', '...\n\n', ' …\n\n', '..."\n\n', '…\n\n', ' ...'] | logit lens: ['oum', 'uny', 'نج', 'inan']
125
+ rec.2.38 | skip16: rank(Japan)=520 ['.”\n\n', '...\n\n', '..."\n\n', ' ...\n\n', '”.\n\n', '.\n\n'] | logit lens: ['ambo', 'ouer', 'uny', 'auro']
126
+ rec.2.46 | skip16: rank(Japan)=7212 [' …\n', '.»\n\n', '..."\n\n', '.”\n\n', ' [...]', ' ...\n'] | logit lens: [' answer', 'answer', ' Answer', 'arras']
127
+ rec.2.54 | skip16: rank(Japan)=809 [' ...\n', ' [...]', '..."\n', '.".', ':[', ' ..\n'] | logit lens: [' retra', ' what', ' _', ' called']
128
+ s.2 | skip16: rank(Japan)=428 ['..."\n\n', ' city', ' Britann', ' Wikipedia', ' cities', '.""'] | logit lens: ['くっ', 'ふっ', 'りば', 'ふん']
129
+ rec.3.22 | skip16: rank(Japan)=4216 [' city', ' City', 'city', ' cities', 'City', ' **"'] | logit lens: ['ähler', 'oje', ' شاه', ' Tort']
130
+ rec.3.30 | skip16: rank(Japan)=2724 [' city', ' cities', 'city', '?**', ' City', ' **"'] | logit lens: ['oum', 'uny', 'نج', 'inan']
131
+ rec.3.38 | skip16: rank(Japan)=506 [':\n', '?\n', '...\n', '?**', '..."', ' city'] | logit lens: ['ambo', 'uny', 'ouer', 'auro']
132
+ rec.3.46 | skip16: rank(Japan)=1460 ['?\n', ':\n', '?",\n', '?"\n', ' ?\n', '?**'] | logit lens: [' answer', 'answer', ' Answer', 'arras']
133
+ rec.3.54 | skip16: rank(Japan)=204 [':\n', ' ___', ' City', '?\n', ' Tokyo', ' city'] | logit lens: [' retra', ' what', ' _', ' called']
134
+ s.3 | skip16: rank(Japan)=367 [' ___', ':\n', ' _', ' Tokyo', '?\n', '____'] | logit lens: ['くっ', 'ふっ', 'りば', 'ふん']
135
+ out.62 | skip16: rank(Japan)=4 [' Tokyo', 'Tokyo', ' Kyoto', ' Japan', ' Osaka', ' Jakarta'] | logit lens: [' Tokyo', ' Japan', 'urgo', ' Kyoto']
136
+ out.70 | skip16: rank(Japan)=8 [' Tokyo', ' ___', ' Edo', ' Kyoto', ' _', 'Tokyo'] | logit lens: [' Tokyo', ' _', ' ___', ' what']
137
+
138
+ PROMPT 'Fact: The color of the planet fourth from the Sun is' intermediate: Mars model top-3: [' orange', ' green', ' not']
139
+ in.06 | skip16: rank(Mars)=25820 ['nem', 'adet', "'.[", ' Opening', ' Closed', 'poca'] | logit lens: [' dur', ' st', 'idus', ' Wend']
140
+ in.14 | skip16: rank(Mars)=20483 ['nem', 'adet', "'.[", ' Opening', ' Closed', 'дей'] | logit lens: [' again', ' pall', ' po', ' captured']
141
+ x | skip16: rank(Mars)=30623 ['adet', 'nem', "'.[", ' Opening', ' Closed', 'дей'] | logit lens: ['raux', 'acca', 'reros', ' herself']
142
+ rec.1.22 | skip16: rank(Mars)=89067 [' bre', ' Iwan', ' Wilde', ' unit', ' gain', ' conquer'] | logit lens: ['मान', 'chev', ' now', ' structure']
143
+ rec.1.30 | skip16: rank(Mars)=88981 [' bre', ' unit', ' Wilde', ' Iwan', ' gain', ' conquer'] | logit lens: ['نج', 'atang', 'uny', 'start']
144
+ rec.1.38 | skip16: rank(Mars)=88996 [' bre', ' Wilde', ' unit', ' Iwan', ' gain', ' conquer'] | logit lens: ['ouer', 'ikian', 'ourde', 'oué']
145
+ rec.1.46 | skip16: rank(Mars)=90150 [' bre', ' Wilde', ' conquer', ' Iwan', ' unit', ' gain'] | logit lens: [' named', 'ieras', 'avas', ' name']
146
+ rec.1.54 | skip16: rank(Mars)=33409 ['nem', 'adet', "'.[", ' Opening', ' Closed', 'дей'] | logit lens: [' not', '아버', ' actually', ' color']
147
+ s.1 | skip16: rank(Mars)=89081 [' bre', ' Wilde', ' Iwan', ' unit', ' conquer', 'ieck'] | logit lens: ['ふっ', 'くっ', 'りば', 'ふん']
148
+ rec.2.22 | skip16: rank(Mars)=21283 ['...', '...\n\n', ' Sard', ' colored', ' Hindi', 'RD'] | logit lens: [' now', 'मान', ' الإع', 'chev']
149
+ rec.2.30 | skip16: rank(Mars)=14479 ['...\n\n', ' ...\n\n', '..."\n\n', '...', '....\n\n', '.\n\n'] | logit lens: ['نج', 'atang', 'uny', 'start']
150
+ rec.2.38 | skip16: rank(Mars)=4258 ['...\n\n', '..."\n\n', ' ...\n\n', '...', '....\n\n', '.\n\n'] | logit lens: ['ikian', 'ouer', 'ourde', 'oué']
151
+ rec.2.46 | skip16: rank(Mars)=962 ['</think>', '.""', '."\n\n', '..."\n\n', '.»\n\n', ".'\n\n"] | logit lens: ['ieras', ' likely', 'avas', ' not']
152
+ rec.2.54 | skip16: rank(Mars)=2002 ['..."\n\n', ' [...', '.""', ' [...]', '.[', ':['] | logit lens: [' not', 'ikian', '아버', ' actually']
153
+ s.2 | skip16: rank(Mars)=849 [' unknown', ' known', ' named', ':[', '[[', '.""'] | logit lens: ['ふっ', 'くっ', 'りば', 'ふん']
154
+ rec.3.22 | skip16: rank(Mars)=45 ['..."', ' Earth', ' colors', ' colored', ' "...', ' meteor'] | logit lens: [' now', 'मान', ' الإع', 'headers']
155
+ rec.3.30 | skip16: rank(Mars)=138 ['?"\n\n', '?"', '?\n\n', '?', '?**\n\n', '"?'] | logit lens: ['نج', 'atang', 'uny', ' Leh']
156
+ rec.3.38 | skip16: rank(Mars)=89 ['?\n\n', '..."', '?\n', '?"', '?"\n\n', '"?'] | logit lens: ['ikian', 'ouer', 'ourde', 'oué']
157
+ rec.3.46 | skip16: rank(Mars)=47 ['?\n\n', '?\n', '?', ' named', ' colors', '?"'] | logit lens: ['ieras', ' likely', 'avas', ' not']
158
+ rec.3.54 | skip16: rank(Mars)=78 [' unknown', '?\n\n', ' green', '?\n', '...\n', ' named'] | logit lens: [' not', 'ikian', '아버', ' actually']
159
+ s.3 | skip16: rank(Mars)=27 [' unknown', ' ___', '?\n', '?\n\n', ' unnamed', ' green'] | logit lens: ['ふっ', 'くっ', 'りば', 'ふん']
160
+ out.62 | skip16: rank(Mars)=25 [' yellow', ' orange', ' green', ' brown', ' blue', ' redd'] | logit lens: [' orange', ' yellow', ' brown', ' redd']
161
+ out.70 | skip16: rank(Mars)=13 [' redd', ' yellow', ' brown', ' orange', ' Mercury', ' Venus'] | logit lens: [' orange', ' blue', ' yellow', ' redd']
162
+
163
+ PROMPT 'es 23? Answer briefly.<|im_end|>\n<|im_start|>assistant\n<think></think>' intermediate: None model top-3: ['4', '3', '1']
164
+ in.06 | skip16: [' bre', ' Wilde', ' Iwan', ' unit', ' conquer', ' gain'] | logit lens: [' Rok', 'enia', 'چه', ' DEN']
165
+ in.14 | skip16: ['nem', 'adet', 'poca', 'дей', ' Opening', 'volta'] | logit lens: ['stadt', 'tn', 'umont', 'oty']
166
+ x | skip16: ['nem', 'adet', ' Opening', "'.[", ' Closed', 'дей'] | logit lens: ['inos', '्त', 'onis', ' gola']
167
+ rec.1.22 | skip16: [' bre', ' Wilde', ' Iwan', ' unit', ' gain', ' conquer'] | logit lens: ['tha', '�', 'onis', 'kis']
168
+ rec.1.30 | skip16: [' bre', ' Wilde', ' Iwan', ' unit', ' conquer', ' gain'] | logit lens: ['atang', 'ocke', 'one', 'antan']
169
+ rec.1.38 | skip16: [' bre', ' Iwan', ' Wilde', ' unit', ' gain', ' conquer'] | logit lens: ['uad', '/checkout', 'ikian', 'ourdes']
170
+ rec.1.46 | skip16: ['nem', 'adet', ' Opening', "'.[", ' Closed', 'дей'] | logit lens: ['uad', ' contex', 'endit', ' Córd']
171
+ rec.1.54 | skip16: ['nem', ' Opening', "'.[", 'adet', ' Closed', ' .\n'] | logit lens: ['disf', 'anse', '無断', ' 結果']
172
+ s.1 | skip16: [' bre', ' unit', ' Wilde', ' gain', ' conquer', ' Iwan'] | logit lens: ['ふっ', 'くっ', 'りば', 'ふん']
173
+ rec.2.22 | skip16: [' **[', ' **', ' “', ' *“', '[[', ' **"'] | logit lens: ['tha', '�', 'amar', 'onis']
174
+ rec.2.30 | skip16: [' **', ' **[', ' “', '.[', '[[', ' **"'] | logit lens: ['atang', 'ocke', 'antan', 'one']
175
+ rec.2.38 | skip16: [' **[', ' **', '.[', ' “', '...\n\n', ' **('] | logit lens: ['ourdes', 'uad', '/checkout', 'elio']
176
+ rec.2.46 | skip16: [' **', '\xa0', '\u200b', ' ', ' **[', ':['] | logit lens: ['uad', 'eggi', 'endit', ' contex']
177
+ rec.2.54 | skip16: [' ', '5', '3', '4', '2', '6'] | logit lens: ['disf', 'reform', 'anse', '��']
178
+ s.2 | skip16: [' ', '3', '5', ' $', '4', '1'] | logit lens: ['ふっ', 'くっ', 'りば', 'ふん']
179
+ rec.3.22 | skip16: [' **', ' **[', ' **"', ' *“', ' **(', ' “'] | logit lens: ['tha', '�', 'amar', 'onis']
180
+ rec.3.30 | skip16: [' **', ' **"', ' **[', ' **(', ' *“', ' “'] | logit lens: ['atang', 'ocke', 'antan', 'one']
181
+ rec.3.38 | skip16: [' **[', ' **', ' **"', ' **(', ' “', '\u202f'] | logit lens: ['ourdes', 'uad', '/checkout', 'elio']
182
+ rec.3.46 | skip16: [' Answer', ' answer', ' Approximately', ' ', ' “', ' seventy'] | logit lens: ['uad', 'eggi', 'endit', ' contex']
183
+ rec.3.54 | skip16: [' ', '4', '3', '5', '6', '7'] | logit lens: ['disf', 'reform', 'anse', 'onin']
184
+ s.3 | skip16: [' ', '4', '3', '5', '6', '1'] | logit lens: ['ふっ', 'くっ', 'りば', 'ふん']
185
+ out.62 | skip16: [' ', '3', '4', '1', '5', ' forty'] | logit lens: ['onin', '4', '3', 'akume']
186
+ out.70 | skip16: ['4', '3', '1', '5', 'The', '2'] | logit lens: ['4', '3', '1', '5']
fit_records/k3_n50_skip16_s0/run.json ADDED
@@ -0,0 +1,25 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "started": "2026-11-15T23:56:59+0000",
3
+ "finished": "2026-11-16T00:42:09+0000",
4
+ "packages": {
5
+ "torch": "2.14.0+cu126",
6
+ "transformers": "5.17.0",
7
+ "peft": "0.21.0",
8
+ "numpy": "2.4.6",
9
+ "safetensors": "0.8.0"
10
+ },
11
+ "cuda_visible_devices": "0",
12
+ "config": {
13
+ "k": 3,
14
+ "n_prompts": 50,
15
+ "seq_len": 128,
16
+ "chunk": 128,
17
+ "skip": 16,
18
+ "seed": 0,
19
+ "run": "k3_n50_skip16_s0",
20
+ "pilot": false,
21
+ "tf32": true,
22
+ "dtype": "float32",
23
+ "corpus": "WikiText-103 raw, validation split (Salesforce/wikitext)"
24
+ }
25
+ }
fit_records/k4_n50_skip16_s0/metrics.json ADDED
The diff for this file is too large to render. See raw diff
 
fit_records/k4_n50_skip16_s0/prompts.json ADDED
@@ -0,0 +1,202 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [
2
+ {
3
+ "i": 0,
4
+ "start": "Given the chaos , the governor of Yunnan ignored the imperial orders of evacuation . The Mongol army commanded by Prince Ye @"
5
+ },
6
+ {
7
+ "i": 1,
8
+ "start": "From the 1920s Australia 's defence thinking was dominated by British Imperial defence policy , which was embodied"
9
+ },
10
+ {
11
+ "i": 2,
12
+ "start": "Tim 's driving days started as a toddler when he was given a go @-@ kart that he often drove"
13
+ },
14
+ {
15
+ "i": 3,
16
+ "start": "As the Marines formed into a horseshoe @-@ shaped line around Hill 123 , Tamura 's"
17
+ },
18
+ {
19
+ "i": 4,
20
+ "start": "USA Today 's Scott Bowles awarded the film three out of four stars , labeling it a \" surprising comedy that rivals"
21
+ },
22
+ {
23
+ "i": 5,
24
+ "start": "Ringo Starr identified his initial inspiration for \" Back Off Boogaloo \" as having come from Marc Bolan ,"
25
+ },
26
+ {
27
+ "i": 6,
28
+ "start": "The ship 's secondary armament consisted of fourteen 45 @-@ calibre 6 @-@ inch ("
29
+ },
30
+ {
31
+ "i": 7,
32
+ "start": "Sources disagree about Radcliffe 's personal wealth ; he was reported to have earned \u00a3 1 million for the first Harry"
33
+ },
34
+ {
35
+ "i": 8,
36
+ "start": "In April 2010 , there were 6 @,@ 260 people employed in the healthcare field"
37
+ },
38
+ {
39
+ "i": 9,
40
+ "start": "\" Exit Through the Kwik @-@ E @-@ Mart \" is the fifteenth episode of the twenty @-"
41
+ },
42
+ {
43
+ "i": 10,
44
+ "start": "On 25 June 1950 , the North Korean Army ( KPA ) crossed the border into South"
45
+ },
46
+ {
47
+ "i": 11,
48
+ "start": "A British parliamentary motion put forward by Labour MP Kate Hoey in February 2013 to retrospectively condemn the"
49
+ },
50
+ {
51
+ "i": 12,
52
+ "start": "At the fort , Moultrie ordered his men to concentrate their fire on the two large man @-@ of @"
53
+ },
54
+ {
55
+ "i": 13,
56
+ "start": "In Part II of the series , Shikamaru is assigned the task of locating two members of the criminal organization Ak"
57
+ },
58
+ {
59
+ "i": 14,
60
+ "start": "The Ming dynasty had seen considerable advancement in the process of colour printing in China . At his studio , Hu Zhengyan experiment"
61
+ },
62
+ {
63
+ "i": 15,
64
+ "start": "\" Exit Through the Kwik @-@ E @-@ Mart \" was written by Marc Wilmore and directed by"
65
+ },
66
+ {
67
+ "i": 16,
68
+ "start": "In 629 Tikal founded Dos Pilas , some 110 kilometres ( 68 mi )"
69
+ },
70
+ {
71
+ "i": 17,
72
+ "start": "The British fleet weighed anchor at Cape Fear on May 31 , and arrived outside Charleston Harbor the next day . Moul"
73
+ },
74
+ {
75
+ "i": 18,
76
+ "start": "The initial phase saw the Australians advance towards the Hongorai River . Following the end of the early fighting , the Australian"
77
+ },
78
+ {
79
+ "i": 19,
80
+ "start": "The reactions of the native Aboriginal inhabitants to the sudden arrival of British settlers in Australia were varied , but were inevitably hostile when"
81
+ },
82
+ {
83
+ "i": 20,
84
+ "start": "The German offensive ground to a halt in mid @-@ July and a brief lull followed , during which the Australians"
85
+ },
86
+ {
87
+ "i": 21,
88
+ "start": "The city 's growth has reflected the push and pull of many social and economic factors . The total population increased in each"
89
+ },
90
+ {
91
+ "i": 22,
92
+ "start": "Following the classification of Tropical Depression Two @-@ E on June 19 , the Government of Mexico issued a tropical"
93
+ },
94
+ {
95
+ "i": 23,
96
+ "start": "The Natives remained in London following the England international . They defeated London Welsh on 18 February , before losing first"
97
+ },
98
+ {
99
+ "i": 24,
100
+ "start": "In February 1997 , a United States immigration court judge granted asylum to a German Scientologist who claimed she"
101
+ },
102
+ {
103
+ "i": 25,
104
+ "start": "Some of the most important trade routes began at Canton and Macau in Qing Dynasty China . Early in each year a large convoy"
105
+ },
106
+ {
107
+ "i": 26,
108
+ "start": "The fourth installment in the Slammiversary name was announced in January 2008 to take place on June"
109
+ },
110
+ {
111
+ "i": 27,
112
+ "start": "Capel Lligwy has national recognition and statutory protection from alteration as it has been designated as a Grade II listed building"
113
+ },
114
+ {
115
+ "i": 28,
116
+ "start": "The Trout Creek Mountains are semi @-@ arid because they are in the eastern rain shadow of mountain ranges to"
117
+ },
118
+ {
119
+ "i": 29,
120
+ "start": "Asahi , meaning \" rising sun \" , a poetic name for Japan from a stanza of waka poetry , was laid"
121
+ },
122
+ {
123
+ "i": 30,
124
+ "start": "Beckham 's term as governor ended December 10 , 1907 . In January 19"
125
+ },
126
+ {
127
+ "i": 31,
128
+ "start": "Donald Sutherland portrays Jack Pellitt , Bobby 's father and Kurt 's boss . On July 27 ,"
129
+ },
130
+ {
131
+ "i": 32,
132
+ "start": "As British offshoots , the Australian colonies participated in Britain 's small wars of the 19th century"
133
+ },
134
+ {
135
+ "i": 33,
136
+ "start": "Retail is another major employer in the county , with 5 @,@ 280 people employed in April "
137
+ },
138
+ {
139
+ "i": 34,
140
+ "start": "The tour returned to Pocono a month later , and Richmond battled for another victory in a fog @-@ shortened"
141
+ },
142
+ {
143
+ "i": 35,
144
+ "start": "\" ... Thirteen Years Later \" received mixed reviews from critics . The A.V. Club 's Todd VanDerWer"
145
+ },
146
+ {
147
+ "i": 36,
148
+ "start": "Bonnie Stiernberg , for Paste , was more positive , writing that she preferred the use of comedy , following the"
149
+ },
150
+ {
151
+ "i": 37,
152
+ "start": "In common with many small Scottish islands , Papa Stour 's population peaked in the 19th century and has"
153
+ },
154
+ {
155
+ "i": 38,
156
+ "start": "The unexpected deployment to East Timor in 1999 led to significant changes in Australian defence policy and to an enhancement"
157
+ },
158
+ {
159
+ "i": 39,
160
+ "start": "Government publications on the dangers of sects increased between 1996 and 1998 , and"
161
+ },
162
+ {
163
+ "i": 40,
164
+ "start": "Richmond started 1982 without a ride before getting a one @-@ race deal to drive for Billie"
165
+ },
166
+ {
167
+ "i": 41,
168
+ "start": "The music for Sonic the Hedgehog was composed and produced by Masato Nakamura , bassist and lead songwriter of the"
169
+ },
170
+ {
171
+ "i": 42,
172
+ "start": "Apple Records issued the single on 17 March 1972 in Britain , as Apple R 59"
173
+ },
174
+ {
175
+ "i": 43,
176
+ "start": "Meridian is home to two post @-@ secondary educational institutions . Meridian Community College , founded in 19"
177
+ },
178
+ {
179
+ "i": 44,
180
+ "start": "Ayola began her career in the theatre , performing in a number of Shakespearean plays including Twelfth Night , A M"
181
+ },
182
+ {
183
+ "i": 45,
184
+ "start": "Cooke 's opportunistic diversion had determined that the Spanish forces in the Philippines posed no immediate threat , although the mission"
185
+ },
186
+ {
187
+ "i": 46,
188
+ "start": "To further justify its stance , the German government has also pointed to the long history of U.S. court cases involving Scient"
189
+ },
190
+ {
191
+ "i": 47,
192
+ "start": "Headed by police chief Lee Shelbourn since 2009 , the Meridian Police Department consists of "
193
+ },
194
+ {
195
+ "i": 48,
196
+ "start": "A port , Sonic the Hedgehog Genesis , was released for the Game Boy Advance ( GBA ) on November"
197
+ },
198
+ {
199
+ "i": 49,
200
+ "start": "As is often the case with huge ancient ruins , knowledge of the site was never completely lost in the region . It seems"
201
+ }
202
+ ]
fit_records/k4_n50_skip16_s0/readouts.txt ADDED
@@ -0,0 +1,230 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ non-semantic tokens (no letter or digit): 5208 of 131072
2
+
3
+ EARLY POSITIONS. Jacobian size by source position, relative to the median over positions 16+ (k=4, 50 prompts)
4
+ node p0 p1 p2 p3 p4 p5 p6 p7 p8-15 mean
5
+ emb 8.15 3.06 3.82 2.33 3.48 4.11 3.62 4.37 9.18
6
+ in.04 3.73 2.16 3.83 1.60 2.15 4.97 6.93 3.95 7.21
7
+ in.12 4.69 1.30 4.33 1.74 2.30 5.05 7.16 5.73 8.07
8
+ x 2.27 0.80 2.55 0.93 1.18 3.68 4.79 5.49 5.86
9
+ u.1 2.37 0.66 1.27 0.94 1.22 3.91 5.14 5.86 6.25
10
+ rec.1.20 5.18 1.41 1.13 1.41 1.73 4.47 10.74 6.72 10.80
11
+ rec.1.28 7.57 1.21 1.14 0.84 0.75 5.47 8.88 3.87 11.56
12
+ rec.1.36 14.97 3.70 4.15 1.24 1.30 4.37 16.03 2.50 9.28
13
+ rec.1.44 25.91 9.77 9.11 0.89 1.62 3.68 3.87 4.73 5.53
14
+ rec.1.52 10.71 4.48 18.61 0.66 0.78 1.62 1.81 1.36 2.49
15
+ s.1 4.64 5.21 22.21 0.71 0.74 1.70 1.48 1.32 2.61
16
+ u.2 5.06 5.62 23.53 0.78 0.80 1.82 1.64 1.37 2.80
17
+ rec.2.20 3.86 1.25 10.74 1.55 1.97 2.05 3.16 2.40 3.74
18
+ rec.2.28 4.23 1.37 1.41 1.53 1.91 3.02 1.95 2.01 3.49
19
+ rec.2.36 4.05 1.55 1.72 1.56 2.62 3.87 1.52 2.03 2.78
20
+ rec.2.44 3.99 1.45 1.67 1.60 2.99 3.65 1.38 2.00 2.74
21
+ rec.2.52 4.22 1.20 1.59 1.81 2.20 4.71 1.49 2.27 2.91
22
+ s.2 1.25 0.98 1.50 1.91 1.86 5.32 1.45 2.16 2.21
23
+ u.3 1.34 1.08 1.60 2.03 1.97 5.48 1.51 2.28 2.24
24
+ rec.3.20 4.14 2.06 1.95 2.42 2.78 3.21 2.11 3.09 2.59
25
+ rec.3.28 4.36 1.79 1.65 1.90 2.27 2.80 1.96 2.33 2.21
26
+ rec.3.36 3.53 1.88 1.75 1.91 2.66 3.44 2.06 2.41 2.31
27
+ rec.3.44 2.90 1.64 1.64 1.89 2.60 3.15 1.97 2.43 2.29
28
+ rec.3.52 2.26 1.52 1.65 1.95 2.38 3.73 2.03 2.56 2.31
29
+ s.3 1.72 1.40 1.63 2.13 2.17 4.38 2.10 2.69 2.40
30
+ u.4 1.81 1.50 1.74 2.23 2.28 4.44 2.16 2.76 2.42
31
+ rec.4.20 4.33 2.07 2.10 2.25 2.60 2.41 2.28 2.52 2.24
32
+ rec.4.28 5.17 2.04 1.93 2.02 2.10 2.01 2.04 2.21 2.05
33
+ rec.4.36 4.05 1.94 1.92 1.97 1.87 1.97 1.93 2.09 1.92
34
+ rec.4.44 3.58 1.69 1.72 1.84 1.61 1.85 1.76 2.04 1.80
35
+ rec.4.52 3.12 1.69 1.66 1.59 1.60 1.74 1.62 1.94 1.67
36
+ s.4 2.91 1.53 1.49 1.40 1.44 1.62 1.45 1.53 1.46
37
+ out.60 2.54 1.32 1.30 1.34 1.29 1.34 1.26 1.38 1.34
38
+ out.68 1.48 0.94 0.94 1.03 0.95 1.00 1.00 1.00 1.03
39
+
40
+ LENS VARIANTS (skip the first 0 / 4 / 16 source positions) and held-out readouts, positions 16+ of 8 held-out paragraphs
41
+ node |J| split-half cos 0v16 cos 4v16 top10 Jacc 0v16 4v16 agree model: skip0 skip4 skip16 logit lens junk top10 skip0 skip16
42
+ emb 10.1 0.063 0.712 0.726 0.38 0.40 0.00 0.00 0.00 0.00 0.35 0.44
43
+ in.04 15.4 0.085 0.783 0.789 0.45 0.47 0.00 0.00 0.00 0.00 0.41 0.48
44
+ in.12 16.5 0.103 0.823 0.828 0.46 0.47 0.01 0.00 0.00 0.00 0.52 0.52
45
+ x 73.1 0.047 0.967 0.970 0.74 0.77 0.00 0.00 0.00 0.00 0.51 0.49
46
+ u.1 72.6 -0.003 0.967 0.970 0.86 0.87 0.00 0.00 0.00 0.00 0.53 0.52
47
+ rec.1.20 17.0 -0.006 0.852 0.856 0.64 0.67 0.00 0.00 0.00 0.00 0.44 0.47
48
+ rec.1.28 17.9 -0.002 0.914 0.917 0.79 0.80 0.00 0.00 0.00 0.00 0.49 0.48
49
+ rec.1.36 7.5 -0.001 0.970 0.977 0.94 0.94 0.00 0.00 0.00 0.01 0.40 0.41
50
+ rec.1.44 4.8 -0.006 0.973 0.991 0.90 0.93 0.00 0.00 0.00 0.04 0.40 0.40
51
+ rec.1.52 1.2 0.001 0.980 0.995 0.68 0.74 0.00 0.00 0.00 0.10 0.49 0.43
52
+ s.1 6.6 -0.008 0.972 0.991 0.59 0.68 0.00 0.00 0.00 0.00 0.39 0.36
53
+ u.2 8.7 -0.006 0.967 0.989 0.53 0.62 0.00 0.00 0.00 0.00 0.51 0.46
54
+ rec.2.20 0.3 0.062 0.803 0.885 0.15 0.48 0.00 0.00 0.00 0.00 0.73 0.73
55
+ rec.2.28 0.4 0.116 0.912 0.931 0.42 0.49 0.00 0.00 0.00 0.00 0.78 0.76
56
+ rec.2.36 0.2 0.135 0.927 0.946 0.41 0.49 0.00 0.00 0.00 0.01 0.71 0.63
57
+ rec.2.44 0.2 0.126 0.928 0.948 0.34 0.46 0.00 0.00 0.00 0.04 0.72 0.60
58
+ rec.2.52 0.3 0.129 0.949 0.962 0.75 0.78 0.00 0.00 0.00 0.11 0.96 0.94
59
+ s.2 2.2 0.138 0.959 0.970 0.58 0.64 0.00 0.00 0.00 0.00 0.93 0.93
60
+ u.3 3.0 0.145 0.956 0.968 0.58 0.62 0.00 0.00 0.01 0.00 0.87 0.86
61
+ rec.3.20 0.7 0.445 0.949 0.969 0.49 0.59 0.01 0.01 0.01 0.00 0.56 0.55
62
+ rec.3.28 1.1 0.679 0.964 0.982 0.49 0.61 0.00 0.01 0.01 0.00 0.68 0.70
63
+ rec.3.36 0.7 0.773 0.971 0.985 0.54 0.62 0.01 0.01 0.02 0.01 0.46 0.41
64
+ rec.3.44 0.7 0.803 0.972 0.984 0.56 0.65 0.02 0.03 0.03 0.04 0.41 0.34
65
+ rec.3.52 0.9 0.871 0.980 0.988 0.59 0.68 0.05 0.05 0.08 0.11 0.47 0.37
66
+ s.3 7.1 0.885 0.982 0.989 0.60 0.68 0.09 0.09 0.11 0.00 0.38 0.29
67
+ u.4 10.1 0.889 0.982 0.989 0.69 0.75 0.03 0.03 0.04 0.00 0.52 0.48
68
+ rec.4.20 5.6 0.949 0.983 0.994 0.67 0.78 0.02 0.03 0.04 0.01 0.25 0.22
69
+ rec.4.28 11.6 0.965 0.985 0.995 0.65 0.80 0.02 0.03 0.04 0.00 0.28 0.24
70
+ rec.4.36 10.0 0.973 0.990 0.997 0.70 0.81 0.04 0.04 0.04 0.01 0.26 0.23
71
+ rec.4.44 12.9 0.982 0.993 0.998 0.78 0.85 0.08 0.09 0.09 0.04 0.19 0.17
72
+ rec.4.52 21.7 0.990 0.996 0.999 0.82 0.89 0.24 0.25 0.26 0.11 0.20 0.18
73
+ s.4 174.9 0.993 0.998 0.999 0.86 0.91 0.33 0.34 0.35 0.00 0.15 0.13
74
+ out.60 40.3 0.995 0.999 1.000 0.91 0.95 0.36 0.37 0.38 0.29 0.12 0.10
75
+ out.68 46.6 0.999 1.000 1.000 0.96 0.97 0.50 0.50 0.50 0.46 0.09 0.08
76
+
77
+ PROMPT 'Fact: The currency used in the country shaped like a boot is' intermediate: Italy model top-3: [' the', ' called', ' not']
78
+ in.06 | skip16: rank(Italy)=39469 ['\xa0', ' **', ')\n\n', ' “', '\n\n', ' Mon'] | logit lens: ['nt', ' “', ' par', ' st']
79
+ in.14 | skip16: rank(Italy)=15330 ['.\n', ';', '\xa0', ').', ').\n', '1'] | logit lens: [' both', ' tongue', 'olar', ':']
80
+ x | skip16: rank(Italy)=35724 ['\xa0', '\n\n', ')\n\n', ' Mon', '.\n', ' mon'] | logit lens: ['raux', 'reros', 'acca', 'acco']
81
+ rec.1.22 | skip16: rank(Italy)=51963 ['eback', '/Create', '′\n', '#+', ' #+', '/Project'] | logit lens: ['ähler', 'untz', ' الإع', 'oux']
82
+ rec.1.30 | skip16: rank(Italy)=49848 ['eback', '/Create', '′\n', ' #+', '#+', '/Project'] | logit lens: [' currently', 'start', 'elier', 'èque']
83
+ rec.1.38 | skip16: rank(Italy)=52706 ['eback', '/Create', '′\n', ' #+', '#+', '-ig'] | logit lens: ['ouer', 'ourdes', 'nam', 'inato']
84
+ rec.1.46 | skip16: rank(Italy)=52123 ['eback', '/Create', '′\n', '-ig', ' #+', '**–'] | logit lens: ['erus', ' called', 'called', ' actually']
85
+ rec.1.54 | skip16: rank(Italy)=46530 ['\xa0', '\n\n', ')\n\n', ' Mon', 'Mon', ' mon'] | logit lens: [' called', 'called', ' actually', 'currency']
86
+ s.1 | skip16: rank(Italy)=60920 ['eback', '/Create', 'onim', '#+', 'blos', '日報'] | logit lens: ['ふっ', 'くっ', 'りば', 'ふん']
87
+ rec.2.22 | skip16: rank(Italy)=51866 ['^−', '′\n', '�', 'onim', ' Napoléon', ' quen'] | logit lens: ['ähler', 'UUID', 'oux', 'kowski']
88
+ rec.2.30 | skip16: rank(Italy)=14858 [' [...]', ' [...', ' (...', ' Knockout', ' (...)', '“\n\n'] | logit lens: ['start', ' currently', 'نج', '</s>']
89
+ rec.2.38 | skip16: rank(Italy)=4949 ['^−', ' �', '</think>', ' [...]', '\u2060', ' [...'] | logit lens: ['ouer', 'nam', 'ourdes', 'inato']
90
+ rec.2.46 | skip16: rank(Italy)=19695 ['…\n\n', ' […', ' ‚', '…\n', ' �', ' …\n\n'] | logit lens: ['erus', ' called', 'called', 'ieras']
91
+ rec.2.54 | skip16: rank(Italy)=35278 ['…\n\n', ' […', '…\n', ' ‚', ' �', '\n\n\n'] | logit lens: [' called', 'called', ' actually', 'currency']
92
+ s.2 | skip16: rank(Italy)=11363 ['\n\n\n\n', '\n\n\n', '[[', ' topon', ' Timor', ' […'] | logit lens: ['くっ', 'ふっ', 'りば', 'ふん']
93
+ rec.3.22 | skip16: rank(Italy)=2867 ['…\n\n', ' […', '…\n', '…', ' …\n\n', ' ‚'] | logit lens: ['ähler', 'kowski', 'UUID', 'enaar']
94
+ rec.3.30 | skip16: rank(Italy)=946 ['…\n\n', ' …\n\n', ' […', ' ‚', '\n\n\n\n', '...\n\n'] | logit lens: ['start', 'نج', ' currently', '</s>']
95
+ rec.3.38 | skip16: rank(Italy)=870 ['...\n\n', '..."\n\n', '</think>', '?\n\n', '.»\n\n', '..\n\n'] | logit lens: ['ouer', 'nam', 'ierro', 'ourdes']
96
+ rec.3.46 | skip16: rank(Italy)=553 ['.""', '</think>', '.»\n\n', ' Wikiped', ' Wikipedia', '...\n\n'] | logit lens: ['erus', ' called', 'called', 'ieras']
97
+ rec.3.54 | skip16: rank(Italy)=914 ['..."\n\n', '...\n\n', ' [...]', '</think>', ' [...', '\n\n\n\n'] | logit lens: [' called', 'called', ' actually', 'currency']
98
+ s.3 | skip16: rank(Italy)=646 ['“\n\n', ' likely', '\n\n\n\n', '...\n\n', '..."\n\n', ' known'] | logit lens: ['くっ', 'ふっ', 'りば', 'ふん']
99
+ rec.4.22 | skip16: rank(Italy)=408 ['...\n\n', '..."', '..."\n\n', '...', '?\n\n', '?'] | logit lens: ['ähler', 'kowski', 'UUID', 'enaar']
100
+ rec.4.30 | skip16: rank(Italy)=439 ['?', '?"', '?\n\n', '?"\n\n', '?”', '"?'] | logit lens: ['start', 'نج', ' currently', 'iana']
101
+ rec.4.38 | skip16: rank(Italy)=228 ['?\n', '?\n\n', '?', '?"\n', '...\n', '?"\n\n'] | logit lens: ['ouer', 'nam', 'ierro', 'ourdes']
102
+ rec.4.46 | skip16: rank(Italy)=139 ['?\n', '?\n\n', ' currency', '?', '?"\n', ':\n'] | logit lens: ['erus', ' called', 'called', 'ieras']
103
+ rec.4.54 | skip16: rank(Italy)=275 [' called', ' named', ' currency', ' known', ' actually', 'called'] | logit lens: [' called', 'called', ' actually', 'currency']
104
+ s.4 | skip16: rank(Italy)=597 [' called', ' currency', ' known', ' named', ' currencies', ' likely'] | logit lens: ['くっ', 'ふっ', 'りば', 'ふん']
105
+ out.62 | skip16: rank(Italy)=3039 [' called', ' currency', ' USD', ' named', ' coins', ' Bitcoin'] | logit lens: [' called', 'الين', 'called', 'ourdes']
106
+ out.70 | skip16: rank(Italy)=2162 [' called', ' the', ' USD', ' actually', ' peg', ' CFA'] | logit lens: [' the', ' called', ' not', ' unique']
107
+
108
+ PROMPT 'Fact: The number of legs on the animal that spins webs is' intermediate: spider model top-3: [' ', ' eight', ' always']
109
+ in.06 | skip16: rank(spider)=49957 [' “', ' **', '\xa0', '\n\n', ' controlled', '.”'] | logit lens: [' “', 'de', 'nt', ' Morse']
110
+ in.14 | skip16: rank(spider)=13794 ['\xa0', ')\n\n', '\n\n', '1', '.\n', '2'] | logit lens: ['pon', ' tongue', '商务', 'iste']
111
+ x | skip16: rank(spider)=6028 ['\n\n', ')\n\n', '\xa0', '.\n', ' Mon', '.\n\n'] | logit lens: ['raux', 'acca', 'reros', ' herself']
112
+ rec.1.22 | skip16: rank(spider)=122267 ['eback', '/Create', '′\n', '#+', ' #+', '/Project'] | logit lens: ['UUID', 'ارضة', 'elier', 'ouer']
113
+ rec.1.30 | skip16: rank(spider)=123815 ['eback', '/Create', '′\n', '#+', '日報', '-ig'] | logit lens: ['ous', 'ождения', 'atang', ' fewer']
114
+ rec.1.38 | skip16: rank(spider)=122476 ['eback', '/Create', '′\n', '#+', ' #+', '-ig'] | logit lens: ['ouer', 'orat', 'ieu', 'orp']
115
+ rec.1.46 | skip16: rank(spider)=122206 ['eback', '/Create', '′\n', '#+', ' #+', '-ig'] | logit lens: ['ouden', 'óns', '一动', 'usqu']
116
+ rec.1.54 | skip16: rank(spider)=6552 ['\n\n', ')\n\n', '...\n\n', 'Mon', '\xa0', 'Int'] | logit lens: ['無断', '/ajax', '一动', 'ouer']
117
+ s.1 | skip16: rank(spider)=123438 ['eback', '/Create', '#+', 'onim', '日報', 'blos'] | logit lens: ['ふっ', 'くっ', 'りば', 'ふん']
118
+ rec.2.22 | skip16: rank(spider)=101125 ['#+', '^−', 'isert', '′\n', 'onim', ' quen'] | logit lens: ['UUID', 'ارضة', '-Sah', 'ouer']
119
+ rec.2.30 | skip16: rank(spider)=20289 ['…\n\n', ' ‚', ' …\n\n', ' […', 'isert', '\n\n\n\n\n'] | logit lens: ['elier', 'Réal', 'ous', 'ulam']
120
+ rec.2.38 | skip16: rank(spider)=19244 ['…\n\n', ' …\n\n', ' […', ' ‚', '…\n', '…'] | logit lens: ['ouer', 'ieu', 'inato', 'ourdes']
121
+ rec.2.46 | skip16: rank(spider)=10319 ['…\n\n', ' […', ' ‚', '…\n', ' …\n\n', '…'] | logit lens: ['ouden', 'óns', '一动', 'ieras']
122
+ rec.2.54 | skip16: rank(spider)=5525 ['…\n\n', ' […', '…\n', ' �', '…', ' …\n\n'] | logit lens: ['/ajax', '無断', '一动', 'ouer']
123
+ s.2 | skip16: rank(spider)=2763 ['…\n\n', ' …\n\n', '\n\n', '\n\n\n\n', ' […', '\n\n\n'] | logit lens: ['ふっ', 'くっ', 'りば', 'ふん']
124
+ rec.3.22 | skip16: rank(spider)=620 ['…\n\n', ' […', ' …\n\n', '…\n', '\n\n\n\n', '\n\n\n'] | logit lens: ['UUID', 'ارضة', '-Sah', 'ouer']
125
+ rec.3.30 | skip16: rank(spider)=798 ['…\n\n', ' …\n\n', ' […', ' ‚', '\n\n\n\n\n', '\n\n\n\n'] | logit lens: ['elier', 'ous', 'Réal', 'ulam']
126
+ rec.3.38 | skip16: rank(spider)=1345 ['</think>', '…\n\n', ' …\n\n', '...\n\n', '..\n\n', '..."\n\n'] | logit lens: ['ouer', 'ieu', 'inato', 'ierro']
127
+ rec.3.46 | skip16: rank(spider)=2653 ['</think>', ' …\n\n', '…\n\n', '.»\n\n', '.""', '..\n\n'] | logit lens: ['ouden', 'óns', 'ieras', '一动']
128
+ rec.3.54 | skip16: rank(spider)=1258 ['</think>', ' …\n\n', '…\n\n', '..."\n\n', '...\n\n', '\n\n\n\n'] | logit lens: ['/ajax', '無断', '一动', 'ouer']
129
+ s.3 | skip16: rank(spider)=1070 ['</think>', '(\n\n', '...\n\n', "'\n\n", '..."\n\n', ' -\n\n'] | logit lens: ['ふっ', 'くっ', 'りば', 'ふん']
130
+ rec.4.22 | skip16: rank(spider)=142 ['...\n\n', '..."\n\n', '?\n\n', '..."', '…\n\n', '?"\n\n'] | logit lens: ['UUID', '-Sah', 'ارضة', 'ouer']
131
+ rec.4.30 | skip16: rank(spider)=156 ['?\n\n', '?"\n\n', '?**\n\n', ' weird', '?"', '?**'] | logit lens: ['elier', 'ous', 'Réal', 'ulam']
132
+ rec.4.38 | skip16: rank(spider)=457 ['?\n\n', '?\n', '?"\n\n', '?”\n\n', '?**\n\n', '?'] | logit lens: ['ouer', 'ieu', 'inato', 'ourdes']
133
+ rec.4.46 | skip16: rank(spider)=587 ['?\n', '?\n\n', '?', ':\n', '?"\n', '?”'] | logit lens: ['ouden', 'óns', 'ieras', '一动']
134
+ rec.4.54 | skip16: rank(spider)=905 [' unknown', '?\n', ' exactly', ' typically', ' usually', ' likely'] | logit lens: ['/ajax', '無断', '一动', 'ouer']
135
+ s.4 | skip16: rank(spider)=864 ['?\n', ' unknown', ' likely', ' irrelevant', ' exactly', ' usually'] | logit lens: ['ふっ', 'くっ', 'りば', 'ふん']
136
+ out.62 | skip16: rank(spider)=28 [' always', ' eight', '?\n', ' equal', ' twelve', ' unknown'] | logit lens: [' always', ' typically', ' equal', ' eight']
137
+ out.70 | skip16: rank(spider)=47 [' eight', ' always', ' ', ' actually', ' unknown', ' equal'] | logit lens: [' ', ' eight', ' always', ' **']
138
+
139
+ PROMPT 'The capital of the country where sushi originated is' intermediate: Japan model top-3: [' Tokyo', ' _', ' what']
140
+ in.06 | skip16: rank(Japan)=112051 ['eback', '/Create', '**–', '-ig', ' #+', ' "}\n'] | logit lens: [' Moran', 'elha', ' quot', 'dur']
141
+ in.14 | skip16: rank(Japan)=111478 ['/Create', '**–', 'eback', 'china', '/Project', 'blos'] | logit lens: ['nil', ' nil', ' par', 'feld']
142
+ x | skip16: rank(Japan)=81073 [' ‚', '...,', '`?', '.–', '’.', '—.'] | logit lens: ['raux', 'acca', 'acco', 'avar']
143
+ rec.1.22 | skip16: rank(Japan)=111034 ['eback', '/Create', '′\n', '#+', ' #+', '/Project'] | logit lens: ['ähler', 'oje', ' Tort', '�']
144
+ rec.1.30 | skip16: rank(Japan)=112020 ['eback', '/Create', '′\n', '#+', ' #+', '/Project'] | logit lens: ['uny', 'oum', 'inan', 'ikian']
145
+ rec.1.38 | skip16: rank(Japan)=113188 ['eback', '/Create', '′\n', ' #+', '#+', '-ig'] | logit lens: ['uny', 'ouer', 'ambo', 'auro']
146
+ rec.1.46 | skip16: rank(Japan)=4106 ['\xa0', ')\n\n', '\n\n', ' Mon', 'Mon', '1'] | logit lens: [' answer', 'answer', ' Answer', 'Answer']
147
+ rec.1.54 | skip16: rank(Japan)=3843 ['\xa0', ')\n\n', '\n\n', ' Mon', 'Mon', ').\n'] | logit lens: [' _', ' retra', ' what', ' called']
148
+ s.1 | skip16: rank(Japan)=99190 ['/Create', 'eback', '-\\(', 'impin', '一文', ' \\(['] | logit lens: ['くっ', 'ふっ', 'りば', 'ふん']
149
+ rec.2.22 | skip16: rank(Japan)=83249 ['#+', '^−', '′\n', 'onim', ' ↗', ' #+'] | logit lens: ['ähler', 'oje', ' شاه', ' Tort']
150
+ rec.2.30 | skip16: rank(Japan)=62567 ['^−', ' [...]', ' [...', '�', '“\n\n', ' ↗'] | logit lens: ['oum', 'uny', 'نج', 'inan']
151
+ rec.2.38 | skip16: rank(Japan)=50335 [' …\n\n', ' …\n', '^−', ' […]', '…,', '…\n\n'] | logit lens: ['ambo', 'ouer', 'uny', 'auro']
152
+ rec.2.46 | skip16: rank(Japan)=50064 ['…\n\n', ' […', '…\n', ' …\n\n', ' …\n', ' ‚'] | logit lens: [' answer', 'answer', ' Answer', 'arras']
153
+ rec.2.54 | skip16: rank(Japan)=56462 [' […', '…\n\n', '…\n', ' ‚', ' �', '…'] | logit lens: [' retra', ' what', ' _', ' called']
154
+ s.2 | skip16: rank(Japan)=39215 [' […', '…\n\n', '…\n', '…', '\n\n\n', '\n\n\n\n'] | logit lens: ['くっ', 'ふっ', 'りば', 'ふん']
155
+ rec.3.22 | skip16: rank(Japan)=22902 ['…\n\n', ' […', '…\n', ' …\n\n', '…', ' ‚'] | logit lens: ['ähler', 'oje', ' شاه', ' Tort']
156
+ rec.3.30 | skip16: rank(Japan)=10039 ['…\n\n', ' …\n\n', ' […', '…\n', ' …\n', ' …'] | logit lens: ['oum', 'uny', 'نج', 'inan']
157
+ rec.3.38 | skip16: rank(Japan)=3361 ['…\n\n', ' …\n\n', ' …\n', '…\n', ' …', '…,'] | logit lens: ['ambo', 'uny', 'ouer', 'auro']
158
+ rec.3.46 | skip16: rank(Japan)=8181 [' …\n', ' …\n\n', '…\n', '.""', '.»\n\n', ' ‚'] | logit lens: [' answer', 'answer', ' Answer', 'arras']
159
+ rec.3.54 | skip16: rank(Japan)=522 [' …\n', '.".', ' ..\n', ' Wikipedia', ' city', '".\n'] | logit lens: [' retra', ' what', ' _', ' called']
160
+ s.3 | skip16: rank(Japan)=464 ['..."\n\n', '.""', ' city', ' Britann', '.".', ' Wikipedia'] | logit lens: ['くっ', 'ふっ', 'りば', 'ふん']
161
+ rec.4.22 | skip16: rank(Japan)=4824 [' city', ' City', 'city', ' cities', 'City', ' downtown'] | logit lens: ['ähler', 'oje', ' شاه', ' Tort']
162
+ rec.4.30 | skip16: rank(Japan)=2551 [' city', ' cities', 'city', '?**', ' City', '?:'] | logit lens: ['oum', 'uny', 'نج', 'inan']
163
+ rec.4.38 | skip16: rank(Japan)=480 [':\n', '?\n', '...\n', '?**', '..."', ' city'] | logit lens: ['ambo', 'uny', 'ouer', 'auro']
164
+ rec.4.46 | skip16: rank(Japan)=1372 ['?\n', ':\n', '?",\n', '?"\n', ' ?\n', '?**'] | logit lens: [' answer', 'answer', ' Answer', 'arras']
165
+ rec.4.54 | skip16: rank(Japan)=201 [':\n', ' ___', ' City', '?\n', ' Tokyo', ' city'] | logit lens: [' retra', ' what', ' _', ' called']
166
+ s.4 | skip16: rank(Japan)=365 [' ___', ':\n', ' _', ' Tokyo', '?\n', '____'] | logit lens: ['くっ', 'ふっ', 'りば', 'ふん']
167
+ out.62 | skip16: rank(Japan)=3 [' Tokyo', 'Tokyo', ' Japan', ' Kyoto', ' Osaka', ' Jakarta'] | logit lens: [' Tokyo', ' Japan', 'urgo', ' Kyoto']
168
+ out.70 | skip16: rank(Japan)=8 [' Tokyo', ' ___', ' Edo', ' Kyoto', ' _', 'Tokyo'] | logit lens: [' Tokyo', ' _', ' ___', ' what']
169
+
170
+ PROMPT 'Fact: The color of the planet fourth from the Sun is' intermediate: Mars model top-3: [' orange', ' green', ' not']
171
+ in.06 | skip16: rank(Mars)=1220 ['\xa0', ' **', ')\n\n', '\n\n', '.\n', ').\n'] | logit lens: [' dur', ' st', 'idus', ' Wend']
172
+ in.14 | skip16: rank(Mars)=504 ['.\n', '1', '2', ').', ' Mon', ').\n'] | logit lens: [' again', ' pall', ' po', ' captured']
173
+ x | skip16: rank(Mars)=2814 ['\n\n', ')\n\n', '\xa0', ' Mon', '.\n', ' mon'] | logit lens: ['raux', 'acca', 'reros', ' herself']
174
+ rec.1.22 | skip16: rank(Mars)=105889 ['eback', '/Create', '′\n', '#+', ' #+', '/Project'] | logit lens: ['मान', 'chev', ' now', ' structure']
175
+ rec.1.30 | skip16: rank(Mars)=104997 ['eback', '/Create', '′\n', '#+', ' #+', '-ig'] | logit lens: ['نج', 'atang', 'uny', 'start']
176
+ rec.1.38 | skip16: rank(Mars)=105660 ['eback', '/Create', '′\n', ' #+', '#+', '-ig'] | logit lens: ['ouer', 'ikian', 'ourde', 'oué']
177
+ rec.1.46 | skip16: rank(Mars)=100344 ['eback', '/Create', '′\n', '-ig', '**–', ' #+'] | logit lens: [' named', 'ieras', 'avas', ' name']
178
+ rec.1.54 | skip16: rank(Mars)=5033 ['\n\n', '\xa0', ')\n\n', ' Mon', 'Mon', ').\n'] | logit lens: [' not', '아버', ' actually', ' color']
179
+ s.1 | skip16: rank(Mars)=101268 ['eback', '/Create', '#+', 'blos', '′\n', 'onim'] | logit lens: ['ふっ', 'くっ', 'りば', 'ふん']
180
+ rec.2.22 | skip16: rank(Mars)=88026 ['^−', ' ‚', ' «', '-valu', ' Napoléon', 'ục'] | logit lens: [' now', 'मान', ' الإع', 'chev']
181
+ rec.2.30 | skip16: rank(Mars)=80851 ['…\n\n', ' …\n\n', ' ‚', 'isert', ' #+', ' […'] | logit lens: ['نج', 'atang', 'uny', 'start']
182
+ rec.2.38 | skip16: rank(Mars)=89664 ['…\n\n', ' …\n\n', ' […', '…\n', ' […]', 'isert'] | logit lens: ['ikian', 'ouer', 'ourde', 'oué']
183
+ rec.2.46 | skip16: rank(Mars)=118351 ['…\n\n', ' […', '…\n', ' …\n\n', '…', ' �'] | logit lens: ['ieras', ' likely', 'avas', ' not']
184
+ rec.2.54 | skip16: rank(Mars)=117557 ['…\n\n', ' […', '…\n', ' ‚', ' �', '…'] | logit lens: [' not', 'ikian', '아버', ' actually']
185
+ s.2 | skip16: rank(Mars)=116267 [' […', '…\n\n', '\n\n\n\n', '\n\n\n', " '", '\n\n\n\n\n'] | logit lens: ['ふっ', 'くっ', 'りば', 'ふん']
186
+ rec.3.22 | skip16: rank(Mars)=36894 ['…\n\n', ' […', '…\n', '\n\n\n\n', '\n\n\n', ' …\n\n'] | logit lens: [' now', 'मान', ' الإع', 'headers']
187
+ rec.3.30 | skip16: rank(Mars)=31384 ['…\n\n', ' …\n\n', ' […', '...\n\n', '\n\n\n\n', '\n\n\n\n\n'] | logit lens: ['نج', 'atang', 'uny', ' Leh']
188
+ rec.3.38 | skip16: rank(Mars)=3649 ['...\n\n', '…\n\n', '</think>', ' …\n\n', '..."\n\n', ' ...\n\n'] | logit lens: ['ikian', 'ouer', 'ourde', 'oué']
189
+ rec.3.46 | skip16: rank(Mars)=1496 ['</think>', '.""', '.»\n\n', 'weather', '..."\n\n', '."\n\n'] | logit lens: ['ieras', ' likely', 'avas', ' not']
190
+ rec.3.54 | skip16: rank(Mars)=3444 ['.""', '..."\n\n', " '[", '.".', ' [...', ':['] | logit lens: [' not', 'ikian', '아버', ' actually']
191
+ s.3 | skip16: rank(Mars)=1055 ['.""', ' unknown', " '[", '(\n\n', ' Wikiped', ' known'] | logit lens: ['ふっ', 'くっ', 'りば', 'ふん']
192
+ rec.4.22 | skip16: rank(Mars)=52 ['..."', ' Earth', ' colors', ' colored', ' meteor', ' "...'] | logit lens: [' now', 'मान', ' الإع', 'oum']
193
+ rec.4.30 | skip16: rank(Mars)=145 ['?"\n\n', '?"', '?\n\n', '?', '?**\n\n', '"?'] | logit lens: ['نج', 'atang', 'uny', 'start']
194
+ rec.4.38 | skip16: rank(Mars)=87 ['?\n\n', '?\n', '..."', '?"', '?"\n\n', '?'] | logit lens: ['ikian', 'ouer', 'ourde', 'oué']
195
+ rec.4.46 | skip16: rank(Mars)=47 ['?\n\n', '?\n', '?', ' colors', ' named', '?"'] | logit lens: ['ieras', ' likely', 'avas', ' not']
196
+ rec.4.54 | skip16: rank(Mars)=75 [' unknown', '?\n\n', ' green', '?\n', '...\n', ' named'] | logit lens: [' not', 'ikian', '아버', ' actually']
197
+ s.4 | skip16: rank(Mars)=24 [' unknown', ' ___', '?\n', '?\n\n', ' unnamed', ' green'] | logit lens: ['ふっ', 'くっ', 'りば', 'ふん']
198
+ out.62 | skip16: rank(Mars)=24 [' yellow', ' orange', ' green', ' brown', ' blue', ' redd'] | logit lens: [' orange', ' yellow', ' brown', ' redd']
199
+ out.70 | skip16: rank(Mars)=12 [' redd', ' yellow', ' brown', ' orange', ' Mercury', ' Venus'] | logit lens: [' orange', ' blue', ' yellow', ' redd']
200
+
201
+ PROMPT 'es 23? Answer briefly.<|im_end|>\n<|im_start|>assistant\n<think></think>' intermediate: None model top-3: ['3', '4', '1']
202
+ in.06 | skip16: ['eback', '/Create', '**–', ' #+', 'низ', '#+'] | logit lens: [' Rok', 'enia', 'چه', ' DEN']
203
+ in.14 | skip16: [' **', ' “', '\xa0', ' **[', ' **"', ' *'] | logit lens: ['stadt', 'tn', 'umont', 'oty']
204
+ x | skip16: ['\xa0', ' **', '\n\n', ')\n\n', ' Mon', ' mon'] | logit lens: ['inos', '्त', 'onis', ' gola']
205
+ rec.1.22 | skip16: ['eback', '/Create', '′\n', ' #+', '#+', '/Project'] | logit lens: ['tha', '�', 'onis', 'kis']
206
+ rec.1.30 | skip16: ['eback', '/Create', '′\n', ' #+', '#+', '**–'] | logit lens: ['atang', 'ocke', 'one', 'antan']
207
+ rec.1.38 | skip16: ['eback', '/Create', '′\n', ' #+', '#+', '/Project'] | logit lens: ['uad', '/checkout', 'ikian', 'ourdes']
208
+ rec.1.46 | skip16: [')\n\n', '\xa0', '\n\n', '1', '2', ' Mon'] | logit lens: ['uad', ' contex', 'endit', ' Córd']
209
+ rec.1.54 | skip16: ['\n\n', ')\n\n', 'In', '...\n\n', '1', '2'] | logit lens: ['disf', 'anse', '無断', ' 結果']
210
+ s.1 | skip16: ['eback', '/Create', '#+', 'onim', ' #+', '′\n'] | logit lens: ['ふっ', 'くっ', 'りば', 'ふん']
211
+ rec.2.22 | skip16: [' […', ' ‚', ' \u200b\u200b', '^−', ' #[', ' «'] | logit lens: ['tha', '�', 'amar', 'onis']
212
+ rec.2.30 | skip16: ['[[', ' [[', '.[', '^[', ',[', ':['] | logit lens: ['atang', 'ocke', 'antan', 'one']
213
+ rec.2.38 | skip16: [' \\([', ' [...]', ' [...', ' **[', ' ...', ',['] | logit lens: ['ourdes', 'uad', '/checkout', 'elio']
214
+ rec.2.46 | skip16: ['…\n\n', ' […', ' …\n\n', '…\n', ' …\n', '…'] | logit lens: ['uad', 'eggi', 'endit', ' contex']
215
+ rec.2.54 | skip16: ['…\n\n', ' […', '…\n', '…', ' �', '\n\n\n'] | logit lens: ['disf', 'reform', 'anse', '��']
216
+ s.2 | skip16: ['—', '–', ' ', '[', 'is', 'A'] | logit lens: ['ふっ', 'くっ', 'りば', 'ふん']
217
+ rec.3.22 | skip16: [' **[', ' **', ' “', ' *“', '[[', ' **"'] | logit lens: ['tha', '�', 'amar', 'onis']
218
+ rec.3.30 | skip16: [' **', ' **«', ' **[', ' “', ' *“', '[['] | logit lens: ['atang', 'ocke', 'antan', 'one']
219
+ rec.3.38 | skip16: [' **[', ' **', ' **"', ':[', '**[', '.['] | logit lens: ['ourdes', 'uad', '/checkout', 'elio']
220
+ rec.3.46 | skip16: ['</think>', ':[', ' \\$', ' **[', '<|im_end|>', '\xa0'] | logit lens: ['uad', 'eggi', 'endit', ' contex']
221
+ rec.3.54 | skip16: [' ', '</think>', '</s>', '5', ' $', '3'] | logit lens: ['disf', 'reform', 'anse', 'onin']
222
+ s.3 | skip16: [' ', '</think>', 'estimated', ' $', 'estimate', ' \\$'] | logit lens: ['ふっ', 'くっ', 'りば', 'ふん']
223
+ rec.4.22 | skip16: [' **', ' **[', ' **"', ' *“', ' **(', ' (**'] | logit lens: ['tha', '�', 'amar', 'onis']
224
+ rec.4.30 | skip16: [' **', ' **"', ' **[', ' **(', ' *“', ' “'] | logit lens: ['atang', 'ocke', 'antan', 'one']
225
+ rec.4.38 | skip16: [' **[', ' **', ' **"', ' **(', '\u202f', ' “'] | logit lens: ['ourdes', 'uad', '/checkout', 'elio']
226
+ rec.4.46 | skip16: [' Answer', ' answer', ' ', ' “', ' Approximately', ' seventy'] | logit lens: ['uad', 'eggi', 'endit', ' contex']
227
+ rec.4.54 | skip16: [' ', '3', '4', '5', '6', '7'] | logit lens: ['disf', 'reform', 'onin', '��']
228
+ s.4 | skip16: [' ', '3', '4', '5', '6', '1'] | logit lens: ['ふっ', 'くっ', 'りば', 'ふん']
229
+ out.62 | skip16: ['3', ' ', '4', '1', ' Thirty', '5'] | logit lens: ['onin', '3', 'akume', 'inchi']
230
+ out.70 | skip16: ['3', '4', '1', 'The', '5', '2'] | logit lens: ['3', '4', '1', '5']
fit_records/k4_n50_skip16_s0/run.json ADDED
@@ -0,0 +1,25 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "started": "2026-11-15T22:59:46+0000",
3
+ "finished": "2026-11-15T23:56:22+0000",
4
+ "packages": {
5
+ "torch": "2.14.0+cu126",
6
+ "transformers": "5.17.0",
7
+ "peft": "0.21.0",
8
+ "numpy": "2.4.6",
9
+ "safetensors": "0.8.0"
10
+ },
11
+ "cuda_visible_devices": "0",
12
+ "config": {
13
+ "k": 4,
14
+ "n_prompts": 50,
15
+ "seq_len": 128,
16
+ "chunk": 128,
17
+ "skip": 16,
18
+ "seed": 0,
19
+ "run": "k4_n50_skip16_s0",
20
+ "pilot": false,
21
+ "tf32": true,
22
+ "dtype": "float32",
23
+ "corpus": "WikiText-103 raw, validation split (Salesforce/wikitext)"
24
+ }
25
+ }