Nabidnur commited on
Commit
c09a3f4
·
verified ·
1 Parent(s): 3ee33d3

PISCES (ARIES #1): code, paper, notebooks, results and figures

Browse files
This view is limited to 50 files because it contains too many changes.   See raw diff
Files changed (50) hide show
  1. .gitattributes +4 -0
  2. LICENSE +21 -0
  3. README.md +242 -0
  4. aries/__init__.py +3 -0
  5. aries/_kernels.py +151 -0
  6. aries/pemeans.py +654 -0
  7. aries/pisces.py +63 -0
  8. benchmarks/THIRD_PARTY.md +33 -0
  9. benchmarks/dev_loop.py +79 -0
  10. benchmarks/fb_ssnpp_holdout_rows.npy +3 -0
  11. benchmarks/get_deep1m.py +60 -0
  12. benchmarks/get_paper1m.py +90 -0
  13. benchmarks/ivf_downstream.py +76 -0
  14. benchmarks/mechanism_analysis.py +161 -0
  15. benchmarks/paper_protocol.py +143 -0
  16. benchmarks/plot.py +37 -0
  17. benchmarks/proxy.py +36 -0
  18. benchmarks/run_analysis.sh +26 -0
  19. benchmarks/run_final.sh +20 -0
  20. benchmarks/run_final_pisces.sh +26 -0
  21. benchmarks/run_final_v2.sh +20 -0
  22. benchmarks/run_final_v3.sh +20 -0
  23. benchmarks/spacev_holdout_rows.npy +3 -0
  24. benchmarks/summarize.py +80 -0
  25. benchmarks/time_breakdown.py +72 -0
  26. docs/ABLATIONS.md +506 -0
  27. docs/DERIVATIONS.md +113 -0
  28. figures/architecture.png +3 -0
  29. figures/ivf_and_sizes.png +0 -0
  30. figures/kscaling.png +0 -0
  31. figures/loo_bars.png +0 -0
  32. figures/mechanism.png +0 -0
  33. figures/quality_vs_time.png +0 -0
  34. figures/split_geometry.png +0 -0
  35. figures/split_real.png +0 -0
  36. figures/tensor_flow.png +3 -0
  37. figures/time_breakdown.png +0 -0
  38. figures/ttq_and_seeds.png +0 -0
  39. notebooks/01_quickstart.ipynb +190 -0
  40. notebooks/02_reproduce_kaggle_cpu.ipynb +109 -0
  41. notebooks/03_results_analysis.ipynb +0 -0
  42. notebooks/make_notebooks.py +229 -0
  43. paper/Ball.sty +41 -0
  44. paper/Box.sty +111 -0
  45. paper/PLOTNEURALNET_LICENSE +21 -0
  46. paper/RightBandedBox.sty +130 -0
  47. paper/data/ablation.dat +11 -0
  48. paper/data/ablation_seeds.dat +28 -0
  49. paper/data/heldout_SS_emeans_approx.dat +6 -0
  50. paper/data/heldout_SS_pisces.dat +6 -0
.gitattributes CHANGED
@@ -33,3 +33,7 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ figures/architecture.png filter=lfs diff=lfs merge=lfs -text
37
+ figures/tensor_flow.png filter=lfs diff=lfs merge=lfs -text
38
+ paper/pisces.pdf filter=lfs diff=lfs merge=lfs -text
39
+ results/deep1m_cpu.png filter=lfs diff=lfs merge=lfs -text
LICENSE ADDED
@@ -0,0 +1,21 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Aryan
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
README.md ADDED
@@ -0,0 +1,242 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: mit
3
+ library_name: faiss
4
+ pipeline_tag: feature-extraction
5
+ tags:
6
+ - clustering
7
+ - k-means
8
+ - mini-batch-k-means
9
+ - e-means
10
+ - faiss
11
+ - vector-search
12
+ - vector-quantization
13
+ - approximate-nearest-neighbor
14
+ - unsupervised-learning
15
+ - cpu
16
+ ---
17
+
18
+ # PISCES: principal-axis splitting makes mini-batch k-means faster and better than e-means
19
+
20
+ **ARIES #1** (*Aryan's Iterative Efficiency Series*) · technical report: [`paper/pisces.pdf`](paper/pisces.pdf) ·
21
+ notebooks: [`notebooks/`](notebooks/)
22
+
23
+ **PISCES** stands for *Principal-axis Informed Splitting with Consistent closing, Early revival and Sparse
24
+ assignment*. It is a drop-in improvement on **e-means** (Bessa et al., Meta FAIR, 2026,
25
+ [arXiv:2611.09501](https://arxiv.org/abs/2611.09501)), the state-of-the-art mini-batch k-means that ships in
26
+ `faiss.contrib.e_means`. We test it on e-means' own benchmark and compare against e-means' own code. Every
27
+ comparison below is paired by seed and timed in a single session on one 4-core CPU, and every number is generated
28
+ from the raw per-run files in [`results/final/`](results/final/).
29
+
30
+ | DEEP\* 1M, k = 16,384, the paper's test rows, 10 seeds | change in test MSE vs e-means | 95% CI | seeds won | wall-clock |
31
+ |---|---|---|---|---|
32
+ | 1 epoch | **−0.88%** | [−0.94, −0.82] | 10/10 | **0.42×** |
33
+ | 3 epochs | **−0.27%** | [−0.35, −0.19] | 10/10 | **0.30×** |
34
+ | 5 epochs | **−0.28%** | [−0.32, −0.23] | 10/10 | **0.27×** |
35
+ | PISCES at 3 epochs vs e-means at 5 epochs | −0.10% | | 9/10 | **0.17×** |
36
+ | held-out SPACEV 1M (3 epochs, never used in development) | −0.453% | | 5/5 | 0.37× |
37
+ | held-out SimSearchNet++ 1M (3 epochs) | −0.040% | | 5/5 | 0.34× |
38
+ | e-means given PISCES' sparse assignment (fairness check) | +0.07% | | 3/10 | 0.59× |
39
+
40
+ At e-means' own benchmark setting (k = 16,384), PISCES has lower error than e-means on **all 40 paired
41
+ runs** (3 datasets, 1-5 epochs), at 0.27-0.42× its wall-clock. When e-means gets the same fast assignment, its error
42
+ goes *up* (worse on 17 of 20 runs), so PISCES' advantage is not borrowed engineering. The gain
43
+ depends on k: it peaks around k = 16,384; at k = 65,536 the frozen settings gain almost nothing and the released
44
+ defaults recover −0.19% with a rule chosen on validation runs (see *Scaling in k*).
45
+
46
+ ![Quality vs wall-clock](figures/quality_vs_time.png)
47
+
48
+ ## The two fish
49
+
50
+ When a cluster dies, e-means revives it by copying a donor centroid with a 10⁻⁴ relative nudge. The two
51
+ copies then start on top of each other. PISCES instead places the two children where an optimal
52
+ 2-means split of the donor would put them. They sit at **c ± √(2λ₁/π) · u₁**, symmetric along the
53
+ donor's principal axis, like two fish swimming in opposite directions.
54
+
55
+ ![The PISCES split](figures/split_geometry.png)
56
+
57
+ - **Lemma 1 (half-Gaussian children).** Cutting a Gaussian cluster by the hyperplane through c orthogonal to an
58
+ eigenvector u gives halves with means c ± √(2λ/π)·u.
59
+ - **Lemma 2 (distortion removed by a split).** That split removes (2/π)·N·uᵀΣ²u / uᵀΣu ≤ (2/π)·N·λ₁ of squared
60
+ error, with equality on the top eigenvector. So the best donor is the cluster with the largest N·λ₁, not the
61
+ most populated one.
62
+ - Both quantities are tracked online by an EMA power iteration, at O(Bd) cost per batch.
63
+
64
+ On 400 real DEEP clusters, the prediction of Lemma 2 tracks the gain actually achieved by the split: median
65
+ realized/predicted 1.17, log-log correlation 0.947. The principal-axis split removes a median
66
+ 1.05× as much distortion as an unrestricted 2-means with five restarts. As a donor score, the Gaussian
67
+ closed form also beat two distribution-free estimates that we built to replace it (iteration 21).
68
+
69
+ ![The two fish on real DEEP clusters](figures/split_real.png)
70
+ ![Lemma 2 on real clusters](figures/mechanism.png)
71
+
72
+ ## What PISCES changes
73
+
74
+ ![PISCES inside the e-means engine](figures/architecture.png)
75
+
76
+ One batch as tensors (PlotNeuralNet styles): the grey block is the full B × k distance tensor exact e-means
77
+ computes; PISCES computes only the teal slab, about k/8 wide.
78
+
79
+ ![One PISCES batch as tensors](figures/tensor_flow.png)
80
+
81
+ | component | kind | what it does |
82
+ |---|---|---|
83
+ | principal-axis split | mathematical | children at c ± √(2λ₁/π)·u₁ (Lemma 1) |
84
+ | gain-ranked donor | mathematical | donor = argmax N·λ₁ (Lemma 2) |
85
+ | streaming power iteration | algorithmic | each cluster's axis is tracked from EMA statistics, one power-iteration step per batch |
86
+ | consistent closing statistics | algorithmic | a split in the last epoch also splits the donor's closing sums |
87
+ | early revival + re-fitted step | algorithmic | revival every batch in the first 30% of steps, threshold 0.2, step law × 1.4 |
88
+ | sparse assignment | algorithmic + engineering | IVF over the centroids: √k cells, √k/8 probes (√k/4 in the last epoch) |
89
+ | fused statistics kernel | engineering | one parallel Numba pass computes e-means' sums and PISCES' axis statistics |
90
+ | 8-bit scans with exact re-rank | engineering | 8-bit IVF scans, with the best 4 candidates re-ranked in float32. This returns the float32 answer for 100.000% of rows on a full-scale batch |
91
+
92
+ The last two engineering changes leave the algorithm itself unchanged. On the 10 report seeds, they take
93
+ PISCES to 0.74× the time of PISCES v1.0 (the version without them), with a paired quality difference of
94
+ +0.012%.
95
+
96
+ ![Where the time goes](figures/time_breakdown.png)
97
+
98
+ Profiled in one session (DEEP\* 1M, k = 16,384, 3 epochs): e-means takes 75 s, 96% of it in the
99
+ exact assignment search. PISCES takes 27 s. Its sparse search is still 89% of its time, so
100
+ assignment remains the cost to attack next.
101
+
102
+ ## Analysis
103
+
104
+ **Every component, removed one at a time, at full scale** (validation rows, 3 epochs, 3 seeds; cost is relative
105
+ to PISCES, positive = worse):
106
+
107
+ | removed | change vs e-means | removal costs | time |
108
+ |---|---|---|---|
109
+ | nothing (PISCES) | −0.258% | | 0.31× |
110
+ | principal-axis split | −0.019% | +0.240% | 0.30× |
111
+ | gain-ranked donor | +0.114% | +0.373% | 0.30× |
112
+ | consistent closing | −0.248% | +0.010% | 0.30× |
113
+ | threshold 0.2 → 0.1 | −0.105% | +0.154% | 0.31× |
114
+ | step × 1.4 → × 1.0 | −0.163% | +0.096% | 0.31× |
115
+ | early revival | −0.117% | +0.141% | 0.31× |
116
+ | sparse last epoch (exact instead) | −0.253% | +0.005% | 0.49× |
117
+ | sparse assignment (exact everywhere) | −0.023% | +0.236% | 1.00× |
118
+
119
+ ![Leave-one-out at full scale](figures/loo_bars.png)
120
+
121
+ **Scaling in k** (n = 1M, validation rows, 3 epochs). There are two regimes, and we report both:
122
+
123
+ | k | rows per cluster | PISCES as frozen | time | released defaults | time |
124
+ |---|---|---|---|---|---|
125
+ | 1,024 | ~1,000 | −0.06% (2/3) | 1.80× | **−0.08%** (2/3) | **0.83×** |
126
+ | 4,096 | ~250 | −0.22% (3/3) | 0.42× | same | same |
127
+ | 16,384 | ~60 | −0.31% (5/5) | 0.30× | same | same |
128
+ | 65,536 | ~15 | −0.04% (1/3) | 0.27× | **−0.19%** (3/3) | 0.26× |
129
+
130
+ The speed-up grows with k, as the cost model predicts. PISCES as frozen for the main benchmark had two weak
131
+ ends, and the released defaults add one rule for each:
132
+
133
+ - **k = 1,024:** the exact search is cheap, so building a sparse index every batch does not pay. Below k = 4,096
134
+ PISCES now assigns exactly (iteration 23b).
135
+ - **k = 65,536:** a cluster sees about one row per batch, so its EMA count is too noisy for every-batch revival.
136
+ Fewer revivals alone does not help (threshold 0.1: +0.01%); dropping the every-batch phase does.
137
+ It is now on only when a batch has at least 2 rows per cluster (iteration 24).
138
+
139
+ Both thresholds were chosen on these validation runs, so those two cells are not held-out results. Neither rule
140
+ changes any other number on this page.
141
+
142
+ ![Scaling in k](figures/kscaling.png)
143
+
144
+ **Downstream: the centroids as an IVF index** (10K DEEP1B queries, exact ground truth; this is the job e-means is
145
+ built for). The two indexes are equally good: at equal search cost, the PISCES index' recall@1 differs by
146
+ −0.02 points on average (range −1.29 to +0.78), and reaching recall@1 = 0.9 takes
147
+ 0.99× the distance computations. The list-imbalance factor is 1.194 for PISCES and 1.187
148
+ for e-means. So downstream, PISCES gives the same index in a fraction of the training time, not a better index.
149
+ The codebooks differ in the starved tail: e-means leaves 0.19% of its clusters with fewer than 10 rows,
150
+ PISCES 0.00%.
151
+
152
+ **Mechanism.** PISCES re-seeds 14,426 clusters in a 3-epoch run, against 6,982 for e-means,
153
+ almost all in the first 30% of training. Afterwards we audited both codebooks for global split-merge moves
154
+ that would still pay off. The e-means codebook admits 282 of them, the PISCES codebook
155
+ 45. PISCES does during training what a post-hoc split-merge search would do afterwards.
156
+
157
+ ![Downstream IVF at equal cost and the starved-cluster tail](figures/ivf_and_sizes.png)
158
+
159
+ ## Honest rating
160
+
161
+ Scored by the author's own standard: what would a careful reviewer at a top venue say?
162
+
163
+ | criterion | score | why |
164
+ |---|---|---|
165
+ | beats the newest state of the art on its own benchmark | 8/10 | Better than official e-means on all 40 paired runs at k = 16,384 (two held-out datasets included) and 2.4-3.7× faster. Outside that regime the frozen method needed two post-hoc rules, which are not held-out results |
166
+ | size of the quality gain | 6/10 | −0.88% at 1 epoch, shrinking to −0.28% at 5 epochs. These are real and consistent, but small in absolute terms, as is typical once a clusterer is near convergence |
167
+ | speed | 9/10 | 0.30× the time at 3 epochs; e-means' 5-epoch quality in 0.17× its time; 0.27× at k = 65,536; 0.83× at k = 1,024 with the small-k rule |
168
+ | rigor | 9/10 | Official baseline code; fingerprint-checked data; development on validation rows only; held-out datasets; a fairness baseline; full-scale leave-one-out; 24 logged iterations with every negative result, including the regimes where PISCES does not help |
169
+ | novelty | 6/10 | Principal-direction splits are classical (LBG, PDDP, G-means). What is new is the streaming estimator, gain-ranked revival inside e-means, the consistent closing and the finding that sparse assignment helps only together with them |
170
+ | theory | 6/10 | Two exact lemmas, checked on real clusters. Why approximate assignment *helps* is still a hypothesis |
171
+ | reproducibility | 10/10 | CPU only, one script per table, raw results and data fingerprints included |
172
+ | **overall** | **7.7/10** | A solid, honest improvement on a 2026 state-of-the-art method in the regime it was built for (tens to hundreds of rows per cluster). Not a new paradigm |
173
+
174
+ ## Notebooks
175
+
176
+ | notebook | what it does | time |
177
+ |---|---|---|
178
+ | [`01_quickstart.ipynb`](notebooks/01_quickstart.ipynb) | PISCES vs e-means on a 200k-row DEEP slice | minutes, any CPU |
179
+ | [`02_reproduce_kaggle_cpu.ipynb`](notebooks/02_reproduce_kaggle_cpu.ipynb) | the paper's protocol on a free Kaggle CPU notebook | about 30 min for 5 seeds |
180
+ | [`03_results_analysis.ipynb`](notebooks/03_results_analysis.ipynb) | every table and figure from the raw results, without training | seconds |
181
+
182
+ ## Use it
183
+
184
+ ```bash
185
+ pip install -r requirements.txt
186
+ python -m pytest -q tests # includes a bit-for-bit identity test against faiss e-means
187
+ ```
188
+
189
+ ```python
190
+ import numpy as np
191
+ from aries import pisces
192
+
193
+ x = np.load("train.npy").astype("float32") # (n, d)
194
+ centroids, model, _ = pisces(x, k=16384, nepochs=3, seed=0)
195
+ ```
196
+
197
+ `aries.PEmeans` subclasses faiss's `NumpyEmeans`. With every PISCES option switched off, it reproduces
198
+ official e-means bit for bit.
199
+
200
+ ## Reproduce everything
201
+
202
+ ```bash
203
+ python benchmarks/get_deep1m.py DATA/deep1b # DEEP* 1M with SHA-256 fingerprint checks
204
+ python benchmarks/get_paper1m.py DATA # SPACEV 1M and SimSearchNet++ 1M with the paper's held-out rows
205
+ bash benchmarks/run_final.sh DATA 4 # every number in the report, about six hours on 4 cores
206
+ python paper/make_data.py && (cd paper && pdflatex pisces.tex && pdflatex pisces.tex)
207
+ ```
208
+
209
+ The development history (24 iterations, every run) is in [`docs/ABLATIONS.md`](docs/ABLATIONS.md) and
210
+ [`results/dev/`](results/dev/).
211
+
212
+ **Differences from the e-means paper's setup**, stated up front:
213
+ - assignment runs in CPU float32 (the paper used an H100 in bfloat16);
214
+ - the shared starting rows are drawn with NumPy instead of CUDA.
215
+
216
+ ## Limitations
217
+
218
+ - The gains are largest at small budgets, and the quality gap shrinks as both methods converge.
219
+ - The quality gain depends on rows per cluster: it is largest around 60 (k = 16,384 at n = 1M). At about 15 rows
220
+ per cluster (k = 65,536) the frozen PISCES gains nothing. The released default recovers −0.19%
221
+ there, but that rule was chosen on the same validation runs.
222
+ - Two regime rules (exact assignment below k = 4,096; every-batch revival only when B ≥ 2k) were set on validation
223
+ runs after the main evaluation.
224
+ - The gain on SimSearchNet++ 1M is small (−0.040%).
225
+ - The optimality arguments assume roughly Gaussian clusters. They hold well on DEEP, but other data may differ.
226
+ - We tested only CPU; we did not benchmark a GPU implementation.
227
+
228
+ ## Citation
229
+
230
+ ```bibtex
231
+ @techreport{aryan2026pisces,
232
+ title = {{PISCES}: Principal-Axis Splitting Makes Mini-Batch k-Means Faster and Better than e-means},
233
+ author = {Aryan},
234
+ institution = {ARIES: Aryan's Iterative Efficiency Series},
235
+ number = {1},
236
+ year = {2026}
237
+ }
238
+ ```
239
+
240
+ PISCES builds on e-means and its open-source implementation in faiss (MIT). Please also cite
241
+ [arXiv:2611.09501](https://arxiv.org/abs/2611.09501). Third-party data notices are in
242
+ [`benchmarks/THIRD_PARTY.md`](benchmarks/THIRD_PARTY.md).
aries/__init__.py ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ """ARIES series. #1: PISCES (mini-batch k-means, a drop-in improvement over faiss e-means)."""
2
+ from aries.pisces import pisces # noqa: F401
3
+ from aries.pemeans import PEmeans, PISCES # noqa: F401
aries/_kernels.py ADDED
@@ -0,0 +1,151 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Optional compiled kernels (Numba). Falls back to NumPy when Numba is unavailable."""
2
+ import numpy as np
3
+
4
+ try:
5
+ from numba import njit, prange
6
+
7
+ @njit(parallel=True, fastmath=True, cache=True)
8
+ def _cand_assign_nb(X, C, cn, G, prev, out_l, out_d):
9
+ n, d = X.shape
10
+ m = G.shape[1]
11
+ for i in prange(n):
12
+ p = prev[i]
13
+ best = np.inf
14
+ bl = -1
15
+ xx = 0.0
16
+ for t in range(d):
17
+ xx += X[i, t] * X[i, t]
18
+ for jj in range(m):
19
+ c = G[p, jj]
20
+ s = cn[c]
21
+ for t in range(d):
22
+ s -= 2.0 * X[i, t] * C[c, t]
23
+ if s < best:
24
+ best = s
25
+ bl = c
26
+ out_l[i] = bl
27
+ out_d[i] = max(best + xx, 0.0)
28
+
29
+ @njit(parallel=True, fastmath=False, cache=True)
30
+ def _fused_stats_nb(X, order, starts, labs, C, U, sums, counts, b_axis, b_proj):
31
+ """Per-cluster sums, counts, sum (e.u) e and sum (e.u)^2 over rows sorted by label."""
32
+ nseg = len(starts) - 1
33
+ d = X.shape[1]
34
+ for g in prange(nseg):
35
+ j = labs[g]
36
+ acc = np.zeros(d, np.float64); ax = np.zeros(d, np.float64); e = np.empty(d, np.float64)
37
+ pp = 0.0
38
+ for t in range(starts[g], starts[g + 1]):
39
+ i = order[t]
40
+ p = 0.0
41
+ for q in range(d):
42
+ v = X[i, q]
43
+ acc[q] += v
44
+ e[q] = v - C[j, q]
45
+ p += e[q] * U[j, q]
46
+ for q in range(d):
47
+ ax[q] += p * e[q]
48
+ pp += p * p
49
+ for q in range(d):
50
+ sums[j, q] += acc[q]
51
+ b_axis[j, q] += ax[q]
52
+ counts[j] += starts[g + 1] - starts[g]
53
+ b_proj[j] += pp
54
+
55
+ HAVE_NUMBA = True
56
+ except Exception: # pragma: no cover
57
+ HAVE_NUMBA = False
58
+
59
+
60
+ def cand_assign(X, C, G, prev):
61
+ """Nearest centroid among G[prev[i]] for every row i. Returns (dist, label)."""
62
+ X = np.ascontiguousarray(X, dtype=np.float32)
63
+ C = np.ascontiguousarray(C, dtype=np.float32)
64
+ cn = (C.astype(np.float64) ** 2).sum(1).astype(np.float32)
65
+ L = np.empty(len(X), np.int64); D = np.empty(len(X), np.float32)
66
+ if HAVE_NUMBA:
67
+ _cand_assign_nb(X, C, cn, np.ascontiguousarray(G, dtype=np.int64), prev.astype(np.int64), L, D)
68
+ return D, L
69
+ for s0 in range(0, len(X), 4096):
70
+ r = slice(s0, s0 + 4096); cand = G[prev[r]]; xr = X[r]
71
+ dist = (xr * xr).sum(1)[:, None] - 2 * np.einsum("bd,bmd->bm", xr, C[cand]) + cn[cand]
72
+ j = dist.argmin(1); L[r] = cand[np.arange(len(j)), j]; D[r] = np.maximum(dist[np.arange(len(j)), j], 0)
73
+ return D, L
74
+
75
+
76
+ def cell_batched_assign(X, C, coarse, nprobe, cell_of=None):
77
+ """Approximate nearest-centroid search with exactly the IVF candidate sets
78
+ (each row compares with the centroids of its `nprobe` nearest coarse cells), computed as one
79
+ dense BLAS matrix product per cell instead of per-row scans.
80
+
81
+ X: (n, d) rows; C: (k, d) centroids; coarse: (L, d) coarse cell centres;
82
+ cell_of: optional (k,) coarse cell of every centroid. Returns (squared dist, label)."""
83
+ import faiss
84
+ X = np.ascontiguousarray(X, dtype=np.float32); C = np.ascontiguousarray(C, dtype=np.float32)
85
+ coarse = np.ascontiguousarray(coarse, dtype=np.float32)
86
+ L = len(coarse)
87
+ if cell_of is None:
88
+ cell_of = faiss.knn(C, coarse, 1)[1][:, 0]
89
+ _, qcell = faiss.knn(X, coarse, nprobe) # (n, nprobe) probed cells
90
+ cn = (C * C).sum(1)
91
+ xn = (X * X).sum(1)
92
+ best = np.full(len(X), np.inf, np.float32); lab = np.full(len(X), -1, np.int64)
93
+ flat = qcell.ravel(); rows = np.repeat(np.arange(len(X)), nprobe)
94
+ order = np.argsort(flat, kind="stable"); flat, rows = flat[order], rows[order]
95
+ qstart = np.searchsorted(flat, np.arange(L + 1))
96
+ corder = np.argsort(cell_of, kind="stable"); cstart = np.searchsorted(cell_of[corder], np.arange(L + 1))
97
+ for j in range(L):
98
+ q = rows[qstart[j]:qstart[j + 1]]; cj = corder[cstart[j]:cstart[j + 1]]
99
+ if len(q) == 0 or len(cj) == 0:
100
+ continue
101
+ d = cn[cj][None, :] - 2.0 * (X[q] @ C[cj].T) # (|q|, |cj|) via BLAS
102
+ a = d.argmin(1); v = d[np.arange(len(q)), a]
103
+ better = v < best[q]
104
+ best[q[better]] = v[better]; lab[q[better]] = cj[a[better]]
105
+ miss = lab < 0
106
+ if miss.any(): # rows whose cells were empty
107
+ d0, l0 = faiss.knn(X[miss], C, 1); best[miss] = d0[:, 0] - xn[miss]; lab[miss] = l0[:, 0]
108
+ return np.maximum(best + xn, 0.0), lab
109
+
110
+
111
+ def cell_centric_assign(X, C, coarse, nprobe, cell_of=None):
112
+ """Approximate nearest-centroid search where all rows of a coarse cell share one candidate set:
113
+ the centroids in the `nprobe` coarse cells nearest to that cell's centre. One large BLAS product
114
+ per cell. Returns (squared dist, label)."""
115
+ import faiss
116
+ X = np.ascontiguousarray(X, dtype=np.float32); C = np.ascontiguousarray(C, dtype=np.float32)
117
+ coarse = np.ascontiguousarray(coarse, dtype=np.float32)
118
+ L = len(coarse)
119
+ if cell_of is None:
120
+ cell_of = faiss.knn(C, coarse, 1)[1][:, 0]
121
+ _, near = faiss.knn(coarse, coarse, min(nprobe, L)) # cell -> nearby cells
122
+ _, qc = faiss.knn(X, coarse, 1); qc = qc[:, 0]
123
+ cn = (C * C).sum(1); xn = (X * X).sum(1)
124
+ corder = np.argsort(cell_of, kind="stable"); cstart = np.searchsorted(cell_of[corder], np.arange(L + 1))
125
+ qorder = np.argsort(qc, kind="stable"); qstart = np.searchsorted(qc[qorder], np.arange(L + 1))
126
+ best = np.empty(len(X), np.float32); lab = np.empty(len(X), np.int64)
127
+ for j in range(L):
128
+ q = qorder[qstart[j]:qstart[j + 1]]
129
+ if len(q) == 0:
130
+ continue
131
+ cand = np.concatenate([corder[cstart[c]:cstart[c + 1]] for c in near[j]])
132
+ d = (X[q] @ C[cand].T); d *= -2.0; d += cn[cand][None, :]
133
+ a = d.argmin(1)
134
+ best[q] = d[np.arange(len(q)), a]; lab[q] = cand[a]
135
+ return np.maximum(best + xn, 0.0), lab
136
+
137
+
138
+ def fused_stats(X, labels, C, U, sums, counts, b_axis, b_proj):
139
+ """One pass over a batch: e-means' cluster sums and counts plus PISCES' axis statistics."""
140
+ order = np.argsort(labels, kind="stable")
141
+ lab = labels[order]
142
+ labs, starts = np.unique(lab, return_index=True)
143
+ starts = np.append(starts, len(lab)).astype(np.int64)
144
+ if HAVE_NUMBA:
145
+ _fused_stats_nb(np.ascontiguousarray(X, dtype=np.float32), order.astype(np.int64), starts,
146
+ labs.astype(np.int64), C, U, sums, counts, b_axis, b_proj)
147
+ return
148
+ e = X - C[labels]; p = np.einsum("ij,ij->i", e, U[labels])
149
+ np.add.at(sums, labels, X); counts += np.bincount(labels, minlength=len(C))
150
+ b_axis[labs] += np.add.reduceat((e * p[:, None])[order], starts[:-1], axis=0)
151
+ b_proj += np.bincount(labels, weights=p.astype(np.float64) ** 2, minlength=len(C))
aries/pemeans.py ADDED
@@ -0,0 +1,654 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """PE-means: e-means with principal-axis, gain-ranked revival.
2
+
3
+ This subclasses the official ``faiss.contrib.e_means.NumpyEmeans``. Every line of
4
+ the e-means recipe (batch law, step law, cosine schedule, EMA update, closing
5
+ means) is inherited unchanged. Only dead-cluster revival is replaced:
6
+
7
+ * faiss revival: donors are sampled in proportion to their EMA mass N_j, and
8
+ the donor is copied with a +/-1e-4 relative nudge, so the two copies are
9
+ only pulled apart by later assignments.
10
+ * PE-means revival:
11
+ 1. Each cluster tracks its principal axis u_j by streaming power
12
+ iteration on EMA statistics: V_j <- EMA(sum_i (e_i . u_j) e_i),
13
+ u_j = V_j / |V_j|, with e_i = x_i - c_j. The variance along the axis is
14
+ lambda_j = EMA(sum_i (e_i . u_j)^2) / N_j. Cost: O(B d) per batch,
15
+ negligible next to the O(B k d) assignment.
16
+ 2. Donors are ranked by the distortion a split removes. For a Gaussian
17
+ cluster, the best 2-means split along u_j cuts distortion by
18
+ (2/pi) * N_j * lambda_j (each half keeps variance (1 - 2/pi) * lambda_j
19
+ along u_j), so the score is the EMA sum of squared
20
+ projections (``gain="greedy"``), or that score used as sampling
21
+ weight (``gain="sample"``).
22
+ 3. The donor is split along +/- u_j by delta = sqrt(2/pi) * sqrt(lambda_j),
23
+ the centroids of the two half-Gaussians, so both children start at
24
+ the optimal 2-means split position instead of on top of each other.
25
+
26
+ Each component is a switch, so the ablation in benchmarks/ isolates it.
27
+ """
28
+ import math
29
+
30
+ import numpy as np
31
+ from faiss.contrib.e_means import NumpyEmeans
32
+
33
+ _HALF_GAUSS = math.sqrt(2.0 / math.pi) # E|z| for z ~ N(0,1): child offset in sigma units
34
+
35
+
36
+ class PEmeans(NumpyEmeans):
37
+ def __init__(self, d, k, *args, split="principal", gain="greedy", offset_scale=1.0,
38
+ score="axis", tau=None, revive_every=5, swap=0.0, swap_window=(0.1, 0.9),
39
+ split_pass=False, revive_until=1.0, closing_frac=1.0, child_mass=0.5,
40
+ tau_end=None, every_until=0.0, approx=None, graph=None, approx_last=None,
41
+ momentum=0.0, momentum_until=1.0, probe_sched=None, polish=0, polish_probes=None, polish_second=None,
42
+ hnsw=None, codec="flat", anneal=0.0, anneal_until=0.7, batch_sched=None, fused=False, **kwargs):
43
+ """split: "principal" (PE-means) or "faiss" (official nudge).
44
+ gain: "greedy" (top expected distortion cut), "sample" (Gumbel top-k on it),
45
+ or "usage" (official: Gumbel top-k on N_j).
46
+ offset_scale: multiplies the sqrt(2/pi) sigma child offset (1 = Gaussian optimum).
47
+ score: "axis" ranks donors by N*lambda_1, "total" by N*trace (total distortion).
48
+ tau: revival threshold as a fraction of mean mass (None = faiss default, 0.1 here).
49
+ revive_every: batches between revival checks (faiss: 5).
50
+ swap: at each revival check, also move up to swap*k live clusters whose deletion
51
+ cost (EMA of sum of d2 - d1 over their rows) is below the best donor's split
52
+ gain (2/pi)*N*lambda_1. 0 disables (needs 2-NN assignment).
53
+ swap_window: fraction of training steps during which swaps are allowed.
54
+ split_pass: when a split happens in the last epoch, split the donor's closing
55
+ statistics consistently (half the count each, means at c +/- delta).
56
+ revive_until: no revival or swaps after this fraction of training steps.
57
+ closing_frac: build the closing means from only the last closing_frac of the final
58
+ epoch's batches (fresher assignments; 1.0 = faiss).
59
+ child_mass: EMA mass each child inherits, as a fraction of the donor's (0.5 = faiss).
60
+ tau_end: if set, the revival threshold follows a cosine from tau to tau_end.
61
+ every_until: revive every batch during this first fraction of training, then every
62
+ revive_every batches.
63
+ approx: (coarse_lists, nprobe) to assign with an IVF over the centroids in every epoch
64
+ but the last (exact). None = exact everywhere.
65
+ graph: (m, refresh) graph-pruned assignment with label memory: in every epoch but the
66
+ last, a row is compared only with the m nearest centroids of the centroid it was
67
+ assigned to last time (centroid k-NN graph rebuilt every `refresh` batches).
68
+ Rows without a previous label, and the whole last epoch, are assigned exactly.
69
+ momentum: heavy-ball term on centroid displacements, c += beta * (previous step's move),
70
+ applied during the first momentum_until fraction of steps.
71
+ probe_sched: (p_first, p_last) probes, linearly interpolated over the non-last epochs'
72
+ steps (overrides the sqrt(k)/8 rule).
73
+ polish: number of Lloyd steps after training (assign every row to the final centroids,
74
+ recompute means); polish_probes = IVF probes (None = exact); polish_second = the
75
+ mode of every polish step after the first (default: same as the first)."""
76
+ super().__init__(d, k, *args, **kwargs)
77
+ if split not in ("principal", "faiss", "half", "abs") or gain not in ("greedy", "sample", "usage") \
78
+ or score not in ("axis", "total", "half", "abs"):
79
+ raise ValueError("unknown split, gain or score rule")
80
+ self.split, self.gain, self.score = split, gain, score
81
+ self.offset_scale, self.revive_every = float(offset_scale), int(revive_every)
82
+ if tau is not None:
83
+ self._unused_threshold = float(tau)
84
+ self.swap, self.swap_window = float(swap), tuple(swap_window)
85
+ self.split_pass, self.revive_until = bool(split_pass), float(revive_until)
86
+ self.closing_frac = float(closing_frac)
87
+ self.child_mass, self.tau_end, self.every_until = float(child_mass), tau_end, float(every_until)
88
+ self.approx = approx
89
+ self.approx_last = approx_last # probes for an approximate LAST epoch (None = exact)
90
+ if approx == "auto" and k < APPROX_MIN_K:
91
+ # Small k: the exact GEMM is cheap and the per-batch index build does not pay
92
+ # (k = 1,024: sparse 1.80x e-means' time, exact 0.82x; iteration 23).
93
+ self.approx = None; self.approx_last = None
94
+ self.graph = graph
95
+ self.momentum, self.momentum_until = float(momentum), float(momentum_until)
96
+ self.probe_sched, self.polish, self.polish_probes = probe_sched, int(polish), polish_probes
97
+ self.polish_second = polish_second
98
+ self.hnsw = hnsw # (M, efSearch early, efSearch last epoch, rebuild every b batches)
99
+ self._hnsw = None
100
+ self.anneal, self.anneal_until = float(anneal), float(anneal_until)
101
+ self.batch_sched = batch_sched
102
+ self.fused = bool(fused) # one Numba pass for sums, counts and axis statistics
103
+ self._in_fused = False # (first, last) multipliers of the batch size, linear in rows
104
+ self._half = self._b_half = self._absp = self._b_abs = None
105
+ self.codec = codec # IVF list storage: "flat" (float32) or "fp16" (half-precision scans)
106
+ self._vel = None
107
+ self._mem = self._cgraph = self._cur_index = None
108
+ self._cur_pos = 0
109
+ self.n_graph_rows = 0
110
+ self.n_graph_fallback = 0
111
+ self.graph_checks = []
112
+ self._tau0 = self._unused_threshold
113
+ self._coarse = None
114
+ self.n_swaps = 0
115
+ self._axis = self._axis_sum = self._proj_sq = self._dist = self._delcost = None
116
+
117
+ # ---------------------------------------------------------------- statistics
118
+ def begin(self, global_n, init_centroids):
119
+ out = super().begin(global_n, init_centroids)
120
+ rng = np.random.RandomState(self.seed + 7919)
121
+ axis = rng.standard_normal((self.k, self.d)).astype("float32")
122
+ axis /= np.linalg.norm(axis, axis=1, keepdims=True)
123
+ self._axis = axis # u_j, unit rows
124
+ self._axis_sum = np.zeros((self.k, self.d), "float32") # EMA of sum (e.u) e
125
+ self._proj_sq = np.zeros(self.k, "float64") # EMA of sum (e.u)^2
126
+ self._b_axis = np.zeros((self.k, self.d), "float32")
127
+ self._b_proj = np.zeros(self.k, "float64")
128
+ self._dist = np.zeros(self.k, "float64") # EMA of sum |e|^2
129
+ self._b_dist = np.zeros(self.k, "float64")
130
+ self._delcost = np.zeros(self.k, "float64") # EMA of sum (d2 - d1)
131
+ self._b_del = np.zeros(self.k, "float64")
132
+ if self.split in ("half", "abs") or self.score in ("half", "abs"):
133
+ self._half = np.zeros((self.k, self.d), "float32") # EMA of sum sign(p) e
134
+ self._b_half = np.zeros((self.k, self.d), "float32")
135
+ self._absp = np.zeros(self.k, "float64") # EMA of sum |p|
136
+ self._b_abs = np.zeros(self.k, "float64")
137
+ if self.graph is not None:
138
+ self._mem = np.full(int(global_n), -1, "int64") # last label of every row
139
+ return out
140
+
141
+ def _accumulate(self, x, index=None, labels_out=None):
142
+ self._cur_index, self._cur_pos = index, 0
143
+ return super()._accumulate(x, index, labels_out)
144
+
145
+ def _accumulate_core(self, x, index=None, labels_out=None):
146
+ """e-means' accumulation with the per-row statistics fused into one Numba pass (same
147
+ quantities; only the floating-point summation order differs)."""
148
+ if not (self.fused and self._half is None and self.score == "axis" and self.swap <= 0
149
+ and not self.spherical and self._usage is not None and index is not None):
150
+ return super()._accumulate_core(x, index, labels_out)
151
+ from aries._kernels import fused_stats
152
+ n = len(index)
153
+ pending_n = 0 if self._pending_stats is None else self._pending_stats["n"]
154
+ if pending_n == 0:
155
+ self._batch_stats.fill(0)
156
+ self._batch_objective = np.float64(0)
157
+ self._batch_centroids = self._readout()
158
+ self._batch_accumulation = self._batch_stats
159
+ acc = self._batch_accumulation
160
+ sums, counts = acc[:, :-1], acc[:, -1]
161
+ centroids = self._batch_centroids
162
+ last = self._epoch == len(self.recipe["epoch_rows"]) - 1
163
+ captured = None if labels_out is None else labels_out[:n]
164
+ if captured is None and self.capture_assignments and last:
165
+ captured = np.empty(n, dtype="int64")
166
+ objective = self._batch_objective
167
+ chunk = self.recipe["assignment_chunk_size"]
168
+ for b0 in range(0, n, chunk):
169
+ rows = x[index[b0:min(b0 + chunk, n)]]
170
+ self._in_fused = True
171
+ try:
172
+ distance, labels = self._assign_rows(rows, centroids)
173
+ finally:
174
+ self._in_fused = False
175
+ if captured is not None:
176
+ captured[b0:b0 + len(rows)] = labels
177
+ objective += distance.sum(dtype="float64")
178
+ cnt = np.zeros(self.k, np.float64)
179
+ fused_stats(rows, labels, centroids, self._axis, sums, cnt, self._b_axis, self._b_proj)
180
+ counts += cnt.astype(counts.dtype)
181
+ self._batch_objective = objective
182
+ total_n = pending_n + n
183
+ complete = total_n == self._local_batch_size() or total_n == 0
184
+ stats = {"sums": sums if complete else None, "counts": counts if complete else None,
185
+ "objective": objective, "n": total_n}
186
+ if captured is not None:
187
+ stats["labels"] = captured
188
+ return stats
189
+
190
+ def _graph_assign(self, x, centroids, ids):
191
+ import faiss
192
+ m, refresh = self.graph[:2]
193
+ verify, tol = (self.graph[2], self.graph[3]) if len(self.graph) > 3 else (0.0, 1.0)
194
+ if self._cgraph is None or self._step % refresh == 0 and self._cgraph[0] != self._step:
195
+ _, G = faiss.knn(np.ascontiguousarray(centroids, dtype="float32"),
196
+ np.ascontiguousarray(centroids, dtype="float32"), m)
197
+ self._cgraph = (self._step, G)
198
+ G = self._cgraph[1]
199
+ prev = self._mem[ids]
200
+ known = prev >= 0
201
+ D = np.empty(len(x), "float32"); L = np.empty(len(x), "int64")
202
+ if (~known).any():
203
+ d0, l0 = faiss.knn(x[~known], centroids, 1)
204
+ D[~known], L[~known] = d0[:, 0], l0[:, 0]
205
+ if known.any():
206
+ from aries._kernels import cand_assign
207
+ D[known], L[known] = cand_assign(x[known], centroids, G, prev[known])
208
+ if verify > 0: # self-verification on a random subset
209
+ kidx = np.flatnonzero(known)
210
+ rs = np.random.RandomState(self._step)
211
+ chk = kidx[rs.choice(len(kidx), max(1, int(verify * len(kidx))), replace=False)]
212
+ _, le = faiss.knn(x[chk], centroids, 1)
213
+ miss = float((le[:, 0] != L[chk]).mean())
214
+ self.graph_checks.append(miss)
215
+ if miss > tol: # unsafe: exact batch, refresh the graph next time
216
+ d0, l0 = faiss.knn(x[known], centroids, 1)
217
+ D[known], L[known] = d0[:, 0], l0[:, 0]
218
+ self._cgraph = None
219
+ self.n_graph_fallback += 1
220
+ return D, L
221
+ self.n_graph_rows += int(known.sum())
222
+ return D, L
223
+
224
+ def _hnsw_assign(self, x, centroids, last):
225
+ """Nearest centroid through an HNSW graph over the centroids. The graph is rebuilt every
226
+ `rebuild` batches and after any split; in between, the stored vectors are overwritten in place
227
+ with the current centroids (positions change, links are kept)."""
228
+ import faiss
229
+ M, ef0, ef1, rebuild = self.hnsw
230
+ C32 = np.ascontiguousarray(centroids, dtype="float32")
231
+ stale = self._hnsw is None or self._step - self._hnsw[1] >= rebuild or self._hnsw[2] != self.n_splits
232
+ if stale:
233
+ h = faiss.IndexHNSWFlat(self.d, int(M)); h.hnsw.efConstruction = 40; h.add(C32)
234
+ self._hnsw = [h, self._step, self.n_splits]
235
+ elif self._hnsw[3] != self._step:
236
+ xb = faiss.rev_swig_ptr(faiss.downcast_index(self._hnsw[0].storage).get_xb(), self.k * self.d)
237
+ xb[:] = C32.ravel()
238
+ self._hnsw = self._hnsw[:3] + [self._step]
239
+ h = self._hnsw[0]; h.hnsw.efSearch = int(ef1 if last else ef0)
240
+ D, I = h.search(np.ascontiguousarray(x, dtype="float32"), 1)
241
+ return D[:, 0].copy(), I[:, 0].copy()
242
+
243
+ def _approx_assign(self, x, centroids, nprobe_override=None):
244
+ import faiss
245
+ if self.approx == "auto": # rule: sqrt(k) k-means cells, sqrt(k)/8 probes
246
+ nl = max(8, int(round(math.sqrt(self.k)))); nprobe, mode = max(4, nl // 8), "km"
247
+ else:
248
+ nl, nprobe = self.approx[:2]
249
+ mode = self.approx[2] if len(self.approx) > 2 else "sample"
250
+ if self.probe_sched is not None and nprobe_override is None:
251
+ frac = self._step / max(1, self.recipe["total_steps"])
252
+ p0, p1 = self.probe_sched
253
+ nprobe = int(round(p0 + (p1 - p0) * min(1.0, frac)))
254
+ if nprobe_override is not None:
255
+ nprobe = int(nprobe_override)
256
+ if self._coarse is None or self._coarse[0] != self._epoch:
257
+ C32 = np.ascontiguousarray(centroids, dtype="float32")
258
+ if mode == "km": # coarse cells grouping nearby centroids
259
+ km = faiss.Kmeans(self.d, nl, niter=20, seed=self.seed + self._epoch, max_points_per_centroid=10**9)
260
+ km.train(C32); sub = km.centroids
261
+ else:
262
+ sub = C32[np.random.RandomState(self.seed + self._epoch).choice(len(C32), nl, replace=False)]
263
+ q = faiss.IndexFlatL2(self.d); q.add(np.ascontiguousarray(sub, dtype="float32"))
264
+ self._coarse = (self._epoch, q)
265
+ q = self._coarse[1]
266
+ C32 = np.ascontiguousarray(centroids, dtype="float32"); xq = x
267
+ if self.codec == "fp16":
268
+ ivf = faiss.IndexIVFScalarQuantizer(q, self.d, nl, faiss.ScalarQuantizer.QT_fp16, faiss.METRIC_L2)
269
+ ivf.by_residual = False; ivf.is_trained = True
270
+ elif self.codec == "sq8r": # 8-bit scan, exact float32 re-rank of the best 4
271
+ # faiss' SIMD 8-bit scan needs d % 8 == 0 (else a scalar loop, ~9x slower at d = 100):
272
+ # zero-pad centroids and rows. Padded coordinates add exactly 0; only the summation order (and so
273
+ # float rounding, which can flip a rare near-tie) differs from the scalar loop.
274
+ dp = -(-self.d // 8) * 8
275
+ if dp != self.d:
276
+ pad = lambda a: np.ascontiguousarray(np.pad(np.asarray(a, dtype="float32"), ((0, 0), (0, dp - self.d))))
277
+ C32 = pad(C32); xq = pad(x)
278
+ q = faiss.IndexFlatL2(dp); q.add(pad(faiss.rev_swig_ptr(self._coarse[1].get_xb(), nl * self.d).reshape(nl, self.d)))
279
+ sq = faiss.IndexIVFScalarQuantizer(q, dp, nl, faiss.ScalarQuantizer.QT_8bit, faiss.METRIC_L2)
280
+ sq.by_residual = False; sq.train(C32); sq.add(C32); sq.nprobe = nprobe
281
+ ivf = faiss.IndexRefineFlat(sq, faiss.swig_ptr(C32)); ivf.k_factor = 4
282
+ self._ivf_keep = (sq, C32, q) # keep the arrays alive during the search
283
+ else:
284
+ ivf = faiss.IndexIVFFlat(q, self.d, nl); ivf.is_trained = True
285
+ if self.codec != "sq8r":
286
+ ivf.add(C32); ivf.nprobe = nprobe
287
+ frac = self._step / max(1, self.recipe["total_steps"])
288
+ eps = self.anneal * max(0.0, 1.0 - frac / self.anneal_until) if self.anneal > 0 else 0.0
289
+ D, I = ivf.search(np.ascontiguousarray(xq, dtype="float32"), 2 if eps > 0 else 1)
290
+ if eps > 0: # annealed assignment: a decaying fraction of rows goes to the runner-up
291
+ flip = (np.random.RandomState(self._step + 17).random_sample(len(x)) < eps) & (I[:, 1] >= 0)
292
+ D[flip, 0], I[flip, 0] = D[flip, 1], I[flip, 1]
293
+ D, I = D[:, :1], I[:, :1]
294
+ miss = I[:, 0] < 0
295
+ if miss.any():
296
+ D2, I2 = faiss.knn(x[miss], centroids, 1); D[miss], I[miss] = D2, I2
297
+ return D[:, 0].copy(), I[:, 0].copy()
298
+
299
+ def _assign_rows(self, x, centroids):
300
+ last = self.recipe is not None and self._epoch == len(self.recipe["epoch_rows"]) - 1 \
301
+ if getattr(self, "recipe", None) is not None else True
302
+ ids = None
303
+ if self._mem is not None and self._cur_index is not None:
304
+ ids = np.asarray(self._cur_index[self._cur_pos:self._cur_pos + len(x)])
305
+ self._cur_pos += len(x)
306
+ if self.hnsw is not None and self._usage is not None and len(x) and not self.spherical:
307
+ distance, labels = self._hnsw_assign(x, centroids, last)
308
+ elif self.graph and ids is not None and self._usage is not None and len(x) and not last \
309
+ and not self.spherical:
310
+ distance, labels = self._graph_assign(x, centroids, ids)
311
+ elif self.approx and self._usage is not None and len(x) and not last and not self.spherical:
312
+ distance, labels = self._approx_assign(x, centroids)
313
+ elif self.approx and self.approx_last and self._usage is not None and len(x) and last \
314
+ and not self.spherical:
315
+ npl = self.approx_last
316
+ if npl == "auto": # rule: sqrt(k)/4 probes in the last epoch
317
+ npl = max(8, int(round(math.sqrt(self.k))) // 4)
318
+ distance, labels = self._approx_assign(x, centroids, nprobe_override=npl)
319
+ elif self.swap > 0 and self._usage is not None and len(x) and not self.spherical:
320
+ import faiss
321
+ D, I = faiss.knn(x, centroids, 2)
322
+ distance, labels = D[:, 0].copy(), I[:, 0].copy()
323
+ self._b_del += np.bincount(labels, weights=(D[:, 1] - D[:, 0]).astype("float64"),
324
+ minlength=self.k)
325
+ else:
326
+ distance, labels = super()._assign_rows(x, centroids)
327
+ if self._axis is not None and self._usage is not None and len(x) and not self._in_fused:
328
+ e = x - centroids[labels] # residuals (m, d)
329
+ p = np.einsum("ij,ij->i", e, self._axis[labels]) # projections on u_j
330
+ # Segment sum by label: sort once, then reduceat (np.add.at on 2-D is slow).
331
+ order = np.argsort(labels, kind="stable")
332
+ lab = labels[order]
333
+ uniq, start = np.unique(lab, return_index=True)
334
+ self._b_axis[uniq] += np.add.reduceat(e[order] * p[order, None], start, axis=0)
335
+ if self._half is not None:
336
+ sg = np.sign(p[order])
337
+ self._b_half[uniq] += np.add.reduceat(e[order] * sg[:, None], start, axis=0)
338
+ self._b_abs += np.bincount(labels, weights=np.abs(p).astype("float64"), minlength=self.k)
339
+ self._b_proj += np.bincount(labels, weights=p.astype("float64") ** 2,
340
+ minlength=self.k)
341
+ if self.score == "total":
342
+ self._b_dist += np.bincount(labels, weights=distance.astype("float64"),
343
+ minlength=self.k)
344
+ if ids is not None and self._mem is not None:
345
+ self._mem[ids] = labels
346
+ return distance, labels
347
+
348
+ def _apply_statistics(self, ids, sums, counts, objective, alpha, decay,
349
+ replace_usage=False):
350
+ if self.closing_frac < 1.0 and self._pass_sums is not None:
351
+ rows = self.recipe["epoch_rows"][self._epoch]
352
+ steps = -(-rows // self.recipe["batch_size"])
353
+ if self._epoch_steps == int(round((1.0 - self.closing_frac) * steps)):
354
+ self._pass_counts.fill(0) # restart closing statistics
355
+ self._pass_sums.fill(0)
356
+ if self.momentum > 0 and self._centroids is not None:
357
+ old = self._centroids.copy()
358
+ super()._apply_statistics(ids, sums, counts, objective, alpha, decay, replace_usage)
359
+ if self.momentum > 0 and self._centroids is not None:
360
+ frac = self._step / max(1, self.recipe["total_steps"])
361
+ if self._vel is None:
362
+ self._vel = np.zeros_like(self._centroids)
363
+ t = counts > 0
364
+ move = self._centroids[t] - old[t]
365
+ if frac <= self.momentum_until:
366
+ move += self.momentum * self._vel[t]
367
+ self._centroids[t] = old[t] + move
368
+ self._vel[t] = move
369
+ if self._axis is None:
370
+ return
371
+ touched = self._b_proj > 0
372
+ self._axis_sum *= decay
373
+ self._proj_sq *= decay
374
+ self._axis_sum[touched] += alpha * self._b_axis[touched]
375
+ self._proj_sq[touched] += alpha * self._b_proj[touched]
376
+ if self._half is not None:
377
+ self._half *= decay; self._absp *= decay
378
+ self._half[touched] += alpha * self._b_half[touched]
379
+ self._absp[touched] += alpha * self._b_abs[touched]
380
+ self._b_half.fill(0); self._b_abs.fill(0)
381
+ if self.score == "total":
382
+ self._dist *= decay
383
+ self._dist[touched] += alpha * self._b_dist[touched]
384
+ self._b_dist.fill(0)
385
+ if self.swap > 0:
386
+ self._delcost *= decay
387
+ self._delcost[touched] += alpha * self._b_del[touched]
388
+ self._b_del.fill(0)
389
+ norm = np.linalg.norm(self._axis_sum[touched], axis=1)
390
+ ok = norm > 0
391
+ idx = np.flatnonzero(touched)[ok]
392
+ self._axis[idx] = self._axis_sum[idx] / norm[ok, None] # one power-iteration step
393
+ self._b_axis.fill(0)
394
+ self._b_proj.fill(0)
395
+
396
+ # ---------------------------------------------------------------- revival
397
+ def _choose_splits(self, rng):
398
+ if self.gain == "usage":
399
+ return super()._choose_splits(rng)
400
+ usage = self._usage.astype("float64")
401
+ threshold = self._unused_threshold * usage.sum() / self.k # same tau as faiss
402
+ unused = usage < threshold
403
+ donors = (~unused) & (usage > 0)
404
+ nsplit = min(int(unused.sum()), int(donors.sum()))
405
+ if nsplit == 0 and not self._swap_open():
406
+ return None, None
407
+ if self.score == "half": # N * |E[sign(p) e]|^2: exact gain of the half-mean split
408
+ base = (self._half.astype("float64") ** 2).sum(1) / np.maximum(usage, 1e-12) * (2 / math.pi) * (math.pi / 2)
409
+ elif self.score == "abs": # N * (E|p|)^2: gain of the split along u at the half-means
410
+ base = self._absp ** 2 / np.maximum(usage, 1e-12)
411
+ else:
412
+ base = self._proj_sq if self.score == "axis" else self._dist
413
+ score = np.where(donors, base, -np.inf) # ~ N_j * lambda_j (or trace)
414
+ if self.gain == "sample":
415
+ u = np.maximum(rng.random_sample(self.k), np.finfo("float64").tiny)
416
+ with np.errstate(divide="ignore"):
417
+ score = np.where(donors, np.log(np.maximum(base, 1e-30)) - np.log(-np.log(u)), -np.inf)
418
+ order = np.argsort(score)[::-1]
419
+ source = order[:nsplit]
420
+ target = np.flatnonzero(unused)[:nsplit]
421
+ if self._swap_open():
422
+ source, target = self._add_swaps(source, target, order, unused, donors)
423
+ if len(source) == 0:
424
+ return None, None
425
+ return source, target
426
+
427
+ def _swap_open(self):
428
+ if self.swap <= 0 or self._delcost is None:
429
+ return False
430
+ frac = self._step / max(1, self.recipe["total_steps"])
431
+ return self.swap_window[0] <= frac <= self.swap_window[1]
432
+
433
+ def _add_swaps(self, source, target, order, unused, donors):
434
+ """Pair the next-best donors with the cheapest live clusters while gain > cost."""
435
+ gain = (2.0 / math.pi) * self._proj_sq
436
+ taken = np.zeros(self.k, bool); taken[source] = True; taken[target] = True
437
+ live = donors & ~taken
438
+ cheap = np.flatnonzero(live)[np.argsort(self._delcost[live])]
439
+ rich = [j for j in order[len(source):] if live[j]]
440
+ extra_s, extra_t, used = [], [], set()
441
+ budget = int(self.swap * self.k)
442
+ ci = 0
443
+ for j in rich:
444
+ if len(extra_s) >= budget or ci >= len(cheap):
445
+ break
446
+ if j in used:
447
+ continue
448
+ while ci < len(cheap) and (cheap[ci] in used or cheap[ci] == j):
449
+ ci += 1
450
+ if ci >= len(cheap):
451
+ break
452
+ t = cheap[ci]
453
+ if gain[j] <= self._delcost[t]:
454
+ break
455
+ extra_s.append(j); extra_t.append(t); used.update((j, t)); ci += 1
456
+ if extra_s:
457
+ self.n_swaps += len(extra_s)
458
+ source = np.concatenate([source, np.array(extra_s, dtype=source.dtype if len(source) else np.int64)])
459
+ target = np.concatenate([target, np.array(extra_t, dtype=target.dtype if len(target) else np.int64)])
460
+ return source.astype(np.int64), target.astype(np.int64)
461
+
462
+ def _apply_splits(self, source, target):
463
+ if self.split == "faiss":
464
+ n = super()._apply_splits(source, target)
465
+ self._copy_axis(source, target)
466
+ return n
467
+ c = self._centroids[source].astype("float64")
468
+ usage = self._usage[source].astype("float64")
469
+ lam = self._proj_sq[source] / np.maximum(usage, 1e-12) # variance along u_j
470
+ if self.split == "half": # distribution-free: children at c +/- E[sign(p) e]
471
+ delta = self.offset_scale * self._half[source].astype("float64") / np.maximum(usage, 1e-12)[:, None]
472
+ elif self.split == "abs": # along u at the empirical half-mean E|p|
473
+ delta = (self.offset_scale * self._absp[source] / np.maximum(usage, 1e-12))[:, None] * self._axis[source]
474
+ else:
475
+ delta = (self.offset_scale * _HALF_GAUSS * np.sqrt(np.maximum(lam, 0.0)))[:, None] * self._axis[source]
476
+ gain = (2.0 / math.pi) * self._proj_sq[source]
477
+ self._centroids[source] = (c + delta).astype("float32")
478
+ self._centroids[target] = (c - delta).astype("float32")
479
+ if self.split_pass and self._pass_sums is not None:
480
+ pc = self._pass_counts[source].astype("float64")
481
+ mean = np.where(pc[:, None] > 0, self._pass_sums[source] / np.maximum(pc, 1)[:, None], c)
482
+ self._pass_counts[source] = pc / 2; self._pass_counts[target] = pc / 2
483
+ self._pass_sums[source] = ((mean + delta) * (pc / 2)[:, None]).astype(self._pass_sums.dtype)
484
+ self._pass_sums[target] = ((mean - delta) * (pc / 2)[:, None]).astype(self._pass_sums.dtype)
485
+ if self._delcost is not None:
486
+ self._delcost[source] = gain / 2 # cool-down: not re-swapped at once
487
+ self._delcost[target] = gain / 2
488
+ if self._vel is not None:
489
+ self._vel[source] = 0; self._vel[target] = 0
490
+ half = (self._usage[source] * self.child_mass).astype("float32")
491
+ self._usage[source] = half
492
+ self._usage[target] = half
493
+ self._copy_axis(source, target)
494
+ self.n_splits += len(source)
495
+ self._invalidate_readout()
496
+ return len(source)
497
+
498
+ def _finish_step(self, expected_n):
499
+ frac = (self._step + 1) / max(1, self.recipe["total_steps"])
500
+ if self.tau_end is not None:
501
+ cos = 0.5 * (1 + math.cos(math.pi * min(1.0, frac)))
502
+ self._unused_threshold = float(self.tau_end + (self._tau0 - self.tau_end) * cos)
503
+ # Every-batch revival needs per-batch counts that carry signal: with fewer than EARLY_MIN_ROWS
504
+ # expected rows per cluster per batch (k = 65,536 at n = 1M: one) it is off (iteration 24).
505
+ eu = self.every_until if self.recipe["batch_size"] >= EARLY_MIN_ROWS * self.k else 0.0
506
+ if self.revive_every == 5 and self.revive_until >= 1.0 and eu <= 0:
507
+ return super()._finish_step(expected_n)
508
+ every = 1 if frac <= eu else self.revive_every
509
+ if (self._step + 1) % every == 0 and frac <= self.revive_until:
510
+ self._revive_unused(self._split_rng)
511
+ self._invalidate_readout()
512
+ self._step += 1
513
+ self._epoch_steps += 1
514
+ self._epoch_rows += expected_n
515
+ self._pending_stats = None
516
+ self._batch_accumulation = self._batch_objective = None
517
+ self._batch_centroids = None
518
+
519
+ def finalize(self):
520
+ """Keep each cluster's principal axis and variance along it (faiss clears its state)."""
521
+ if self._axis is not None and self._usage is not None:
522
+ self.final_axis = self._axis.copy()
523
+ self.final_lam = self._proj_sq / np.maximum(self._usage.astype("float64"), 1e-12)
524
+ return super().finalize()
525
+
526
+ def _train_rows(self, x, init_centroids=None):
527
+ if self.batch_sched is None:
528
+ return super()._train_rows(x, init_centroids)
529
+ from faiss.contrib.e_means.base import _epoch_seed
530
+ x = np.ascontiguousarray(x, dtype="float32"); n = len(x)
531
+ if init_centroids is None:
532
+ init_centroids = x[np.random.RandomState(self.seed).permutation(n)[: self.k]]
533
+ self.begin(n, init_centroids)
534
+ B0, a0 = self.recipe["batch_size"], self.recipe["alpha0"]
535
+ R = float(sum(self.recipe["epoch_rows"])); b_first, b_last = self.batch_sched
536
+ plan, done = [], 0
537
+ for rows in self.recipe["epoch_rows"]: # batch sizes, linear in rows seen
538
+ ep, r = [], 0
539
+ while r < rows:
540
+ b = int(round(B0 * (b_first + (b_last - b_first) * (done + r) / R) / 256)) * 256
541
+ b = max(256, min(b, rows - r)); ep.append(b); r += b
542
+ plan.append(ep); done += rows
543
+ self.recipe["total_steps"] = sum(len(ep) for ep in plan)
544
+ self.recipe["epoch_steps"] = [len(ep) for ep in plan]
545
+ for epoch, rows in enumerate(self.recipe["epoch_rows"]):
546
+ order = np.random.RandomState(_epoch_seed(self.seed, epoch)).permutation(n)
547
+ pos = 0
548
+ for b in plan[epoch]:
549
+ self.recipe["batch_size"] = b
550
+ self.recipe["alpha0"] = min(1.0, a0 * (b / B0) ** 0.9) # e-means step law: alpha0 ~ B^0.90
551
+ self.update(self._accumulate(x, order[pos:pos + b])); pos += b
552
+ self.end_epoch()
553
+ self.finalize()
554
+ self.assignments = None
555
+ return self.centroids, None
556
+
557
+ def train(self, x, init_centroids=None):
558
+ c, a = super().train(x, init_centroids)
559
+ xs = np.ascontiguousarray(x, dtype="float32")
560
+ for i in range(self.polish):
561
+ mode = self.polish_probes if i == 0 or self.polish_second is None else self.polish_second
562
+ c, a = self._lloyd_step(xs, c, a, mode)
563
+ if self.polish:
564
+ self.centroids = c
565
+ return c, a
566
+
567
+ def _lloyd_step(self, x, c, prev=None, pp="default"):
568
+ """One Lloyd step from the current centroids. polish_probes: None = exact, int = IVF probes,
569
+ "graph:m" = each row compared only with the m centroids nearest its previous centroid."""
570
+ import faiss
571
+ if pp == "default":
572
+ pp = self.polish_probes
573
+ if isinstance(pp, str) and pp.startswith("graph"):
574
+ m = int(pp.split(":")[1])
575
+ from aries._kernels import cand_assign
576
+ _, G = faiss.knn(c, c, m)
577
+ _, lab = cand_assign(x, c, G, prev)
578
+ I = lab[:, None]
579
+ elif pp is None:
580
+ _, I = faiss.knn(x, c, 1)
581
+ else:
582
+ nl = max(8, int(round(math.sqrt(self.k))))
583
+ km = faiss.Kmeans(self.d, nl, niter=20, seed=self.seed + 99, max_points_per_centroid=10**9)
584
+ km.train(c)
585
+ q = faiss.IndexFlatL2(self.d); q.add(km.centroids)
586
+ ivf = faiss.IndexIVFFlat(q, self.d, nl); ivf.is_trained = True
587
+ ivf.add(c); ivf.nprobe = max(4, nl // 8) if pp == "auto" else int(pp)
588
+ _, I = ivf.search(x, 1)
589
+ miss = I[:, 0] < 0
590
+ if miss.any():
591
+ I[miss] = faiss.knn(x[miss], c, 1)[1]
592
+ lab = I[:, 0]
593
+ ok = lab >= 0
594
+ cnt = np.bincount(lab[ok], minlength=self.k).astype("float64")
595
+ from scipy.sparse import csr_matrix
596
+ M = csr_matrix((np.ones(ok.sum(), "float32"), (lab[ok], np.flatnonzero(ok))), shape=(self.k, len(x)))
597
+ S = np.asarray(M @ x, dtype="float64")
598
+ out = c.copy()
599
+ nz = cnt > 0
600
+ out[nz] = (S[nz] / cnt[nz, None]).astype("float32")
601
+ return out, lab
602
+
603
+ def _remap_memory(self, source, target):
604
+ """Rows remembering a re-used index now belong to the donor's region."""
605
+ if self._mem is None or len(target) == 0:
606
+ return
607
+ remap = np.arange(self.k + 1); remap[np.asarray(target)] = np.asarray(source)
608
+ remap[self.k] = -1
609
+ self._mem = remap[np.where(self._mem < 0, self.k, self._mem)]
610
+
611
+ def _copy_axis(self, source, target):
612
+ self._remap_memory(source, target)
613
+ self._axis_sum[source] *= 0.5
614
+ self._proj_sq[source] *= 0.5
615
+ self._axis[target] = self._axis[source]
616
+ self._axis_sum[target] = self._axis_sum[source]
617
+ self._proj_sq[target] = self._proj_sq[source]
618
+ if self._half is not None:
619
+ self._half[source] *= 0.5; self._absp[source] *= 0.5
620
+ self._half[target] = self._half[source]; self._absp[target] = self._absp[source]
621
+ if self._dist is not None:
622
+ self._dist[source] *= 0.5
623
+ self._dist[target] = self._dist[source]
624
+
625
+
626
+ # Frozen configuration after iterations 1-5 (docs/EXPERIMENTS.md): principal-axis gain-ranked
627
+ # revival, consistent closing statistics, revival threshold 0.2, EMA step law x 1.4.
628
+ APPROX_MIN_K = 4096 # the 'auto' sparse-assignment rule applies from this k on
629
+ EARLY_MIN_ROWS = 2 # early every-batch revival needs >= this many expected rows per cluster per batch
630
+
631
+ PE_V2 = dict(split="principal", gain="greedy", split_pass=True, tau=0.2, alpha_scale=1.4)
632
+
633
+ # Frozen after iterations 6-11 (full-scale validation, 5 seeds): v2 + revival every batch during the
634
+ # first 30% of steps + approximate assignment (sqrt(k) k-means coarse cells, sqrt(k)/8 probes) in
635
+ # every epoch but the last. Better than v2 and ~35-40% faster than official e-means per epoch.
636
+ PE_V3 = dict(PE_V2, every_until=0.3, approx="auto")
637
+
638
+ # PISCES (ARIES series #1), frozen after iterations 12-14: v3 + approximate last epoch with
639
+ # sqrt(k)/4 probes. The approximation helps only together with principal-axis revival: official
640
+ # e-means with the same approximate assignment is *worse* than official e-means (iteration 12).
641
+ PISCES_V10 = dict(PE_V3, approx_last="auto")
642
+
643
+ # PISCES (final, iterations 21-22): the same algorithm as PISCES_V10 with two engineering changes that
644
+ # leave the assignment (practically) unchanged: one fused Numba pass for the per-batch statistics, and
645
+ # 8-bit IVF scans whose best 4 candidates are re-ranked with exact float32 distances.
646
+ PISCES = dict(PISCES_V10, fused=True, codec="sq8r")
647
+
648
+ # Candidate after iterations 15-19: PISCES + one closing Lloyd step (sqrt(k)/8 IVF probes) that
649
+ # re-assigns every row to the closing means and recomputes them.
650
+ PISCES_POLISH = dict(PISCES_V10, polish=1, polish_probes="auto")
651
+
652
+ # PISCES with the fused statistics kernel (engineering only; same algorithm)
653
+ PISCES_FUSED = dict(PISCES_V10, fused=True)
654
+ PISCES_FAST = PISCES
aries/pisces.py ADDED
@@ -0,0 +1,63 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """PISCES — Principal-axis Informed Splitting with Consistent closing, Early revival and
2
+ Sparse assignment. First algorithm of the ARIES series.
3
+
4
+ Two fish swimming in opposite directions: every split places two children at
5
+ c ± sqrt(2/pi)·sigma_1·u_1, symmetric along the cluster's principal axis.
6
+
7
+ `pisces()` is the public entry point. It wraps `PEmeans` (a subclass of faiss's official
8
+ `NumpyEmeans`) with the frozen configuration, applies the re-fitted step law, and optionally grows
9
+ the codebook coarse-to-fine: start with k / 2^g centroids and double them g times by splitting
10
+ every cluster along its principal axis, then train at k.
11
+ """
12
+ import math
13
+ import numpy as np
14
+
15
+ from aries.pemeans import PEmeans, PISCES as PISCES_CONFIG
16
+
17
+ _HALF_GAUSS = math.sqrt(2.0 / math.pi)
18
+
19
+
20
+ def _recipe(n, k, T, alpha_scale):
21
+ from faiss.contrib.e_means.base import _resolve_recipe, _step_law
22
+ B, _ = _resolve_recipe(n, k, max(T, 1e-3), "throughput", None, None)
23
+ return dict(batch_size=B, alpha0=min(1.0, _step_law(n, k, max(T, 1e-3), B) * alpha_scale))
24
+
25
+
26
+ def _train(x, k, T, seed, init, cfg):
27
+ cfg = dict(cfg)
28
+ a = cfg.pop("alpha_scale", 1.0)
29
+ km = PEmeans(x.shape[1], k, nepochs=T, seed=seed, batch_law="throughput", **_recipe(len(x), k, T, a), **cfg)
30
+ c, _ = km.train(x, init_centroids=np.ascontiguousarray(init, dtype="float32"))
31
+ return c, km
32
+
33
+
34
+ def split_all(c, axis, lam):
35
+ """Double the codebook: every centroid becomes two children at c ± sqrt(2/pi)·sqrt(lambda)·u."""
36
+ delta = (_HALF_GAUSS * np.sqrt(np.maximum(lam, 0.0)))[:, None] * axis
37
+ return np.concatenate([c + delta, c - delta]).astype("float32")
38
+
39
+
40
+ def pisces(x, k, nepochs=3, seed=0, init=None, grow=0, grow_epochs=0.5, config=None):
41
+ """Train PISCES on the rows of x (n, d) and return (centroids (k, d), model, info).
42
+
43
+ init: optional (k, d) starting centroids (default: k random rows, as faiss e-means does).
44
+ grow: number of coarse-to-fine doublings before reaching k (0 = none; iteration 13 found it worse)."""
45
+ cfg = dict(PISCES_CONFIG if config is None else config)
46
+ x = np.ascontiguousarray(x, dtype="float32")
47
+ rng = np.random.RandomState(seed)
48
+ info = {"stages": []}
49
+ if grow <= 0:
50
+ if init is None:
51
+ init = x[rng.permutation(len(x))[:k]]
52
+ c, km = _train(x, k, nepochs, seed, init, cfg)
53
+ return c, km, info
54
+ k_s = k >> grow
55
+ c = x[rng.permutation(len(x))[:k_s]]
56
+ per = grow_epochs / grow
57
+ for g in range(grow):
58
+ c, km = _train(x, k_s, per, seed + 101 * (g + 1), c, cfg)
59
+ c = split_all(c, km.final_axis, km.final_lam)
60
+ info["stages"].append(dict(k=k_s, epochs=per))
61
+ k_s *= 2
62
+ c, km = _train(x, k, nepochs - grow_epochs, seed, c, cfg)
63
+ return c, km, info
benchmarks/THIRD_PARTY.md ADDED
@@ -0,0 +1,33 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Third-party material
2
+
3
+ `fb_ssnpp_holdout_rows.npy`, `spacev_holdout_rows.npy` and the SHA-256 fingerprints embedded in
4
+ `get_deep1m.py` and `get_paper1m.py` are copied from
5
+ [faiss `benchs/e_means/paper_splits`](https://github.com/facebookresearch/faiss/tree/main/benchs/e_means/paper_splits),
6
+ so that our evaluation rows are exactly the e-means paper's.
7
+
8
+ ```
9
+ MIT License
10
+
11
+ Copyright (c) Meta Platforms, Inc. and affiliates.
12
+
13
+ Permission is hereby granted, free of charge, to any person obtaining a copy
14
+ of this software and associated documentation files (the "Software"), to deal
15
+ in the Software without restriction, including without limitation the rights
16
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
17
+ copies of the Software, and to permit persons to whom the Software is
18
+ furnished to do so, subject to the following conditions:
19
+
20
+ The above copyright notice and this permission notice shall be included in all
21
+ copies or substantial portions of the Software.
22
+
23
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
24
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
25
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
26
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
27
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
28
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
29
+ SOFTWARE.
30
+ ```
31
+
32
+ The datasets themselves (Deep1B, SimSearchNet++, SPACEV1B) are downloaded at run time from their
33
+ public hosts and are not redistributed here.
benchmarks/dev_loop.py ADDED
@@ -0,0 +1,79 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Development loop on DEEP* (validation rows only; test rows are reserved for final reports).
2
+
3
+ Usage: python benchmarks/dev_loop.py DATA_DIR TAG --n 250000 --k 4096 --epochs 1 3 --seeds 1 2 3 4 5
4
+ --variant name '{"split": "principal", ...}' [--variant ...]
5
+ Each variant is PEmeans keyword arguments as JSON; "official" (faiss NumpyEmeans) is always run.
6
+ Results go to results/dev/TAG.jsonl and a paired summary is printed.
7
+ """
8
+ import argparse, json, os, sys, time
9
+ import numpy as np
10
+ import faiss
11
+
12
+ ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
13
+ sys.path.insert(0, ROOT)
14
+ from faiss.contrib.e_means import NumpyEmeans # noqa: E402
15
+ from aries.pemeans import PEmeans # noqa: E402
16
+
17
+
18
+ def mse(c, rows):
19
+ _, idx = faiss.knn(rows, np.ascontiguousarray(c, dtype="float32"), 1)
20
+ diff = rows.astype(np.float64) - c[idx[:, 0]].astype(np.float64)
21
+ return float((diff * diff).sum() / len(rows))
22
+
23
+
24
+ def main():
25
+ ap = argparse.ArgumentParser()
26
+ ap.add_argument("data"); ap.add_argument("tag")
27
+ ap.add_argument("--n", type=int, default=250000); ap.add_argument("--k", type=int, default=4096)
28
+ ap.add_argument("--epochs", type=float, nargs="+", default=[1, 3])
29
+ ap.add_argument("--seeds", type=int, nargs="+", default=[1, 2, 3, 4, 5])
30
+ ap.add_argument("--variant", nargs=2, action="append", default=[])
31
+ ap.add_argument("--threads", type=int, default=4)
32
+ a = ap.parse_args()
33
+ faiss.omp_set_num_threads(a.threads)
34
+ x = np.ascontiguousarray(np.load(os.path.join(a.data, "train_1M.npy"), mmap_mode="r")[:a.n])
35
+ val = np.load(os.path.join(a.data, "val.npy"))
36
+ variants = [("official", None)] + [(n, json.loads(j)) for n, j in a.variant]
37
+ os.makedirs(os.path.join(ROOT, "results", "dev"), exist_ok=True)
38
+ final = os.path.join(ROOT, "results", "dev", a.tag + ".jsonl")
39
+ out = os.path.join(os.environ.get("DEV_TMP", "/tmp"), a.tag + ".jsonl.partial")
40
+ res = {}
41
+ for T in a.epochs:
42
+ for s in a.seeds:
43
+ init = x[np.random.RandomState(s).permutation(len(x))[:a.k]].copy()
44
+ for name, kw in variants:
45
+ t0 = time.time()
46
+ extra = {}
47
+ kw2 = dict(kw or {})
48
+ bs, asc = kw2.pop("batch_scale", None), kw2.pop("alpha_scale", None)
49
+ if bs is not None or asc is not None: # LawFit: rescale the fitted batch / step
50
+ from faiss.contrib.e_means.base import _resolve_recipe, _step_law
51
+ B, a0 = _resolve_recipe(len(x), a.k, T, "throughput", None, None)
52
+ B = int(B * (bs or 1.0))
53
+ a0 = min(1.0, _step_law(len(x), a.k, T, B) * (asc or 1.0))
54
+ extra = dict(batch_size=B, alpha0=a0)
55
+ base = kw2.pop("base", None)
56
+ km = (NumpyEmeans(x.shape[1], a.k, nepochs=T, seed=s, batch_law="throughput", **extra)
57
+ if kw is None or base == "official"
58
+ else PEmeans(x.shape[1], a.k, nepochs=T, seed=s, batch_law="throughput", **kw2, **extra))
59
+ c, _ = km.train(x, init_centroids=init.copy())
60
+ r = dict(tag=a.tag, n=a.n, k=a.k, epochs=T, seed=s, variant=name, kwargs=kw,
61
+ seconds=time.time() - t0, val_mse=mse(c, val), splits=int(km.n_splits))
62
+ res[(T, name, s)] = r
63
+ with open(out, "a") as f:
64
+ f.write(json.dumps(r) + "\n")
65
+ import shutil
66
+ shutil.copyfile(out, final)
67
+ for T in a.epochs:
68
+ print(f"--- n={a.n} k={a.k} epochs={T} seeds={a.seeds}")
69
+ for name, _ in variants:
70
+ v = np.array([res[(T, name, s)]["val_mse"] for s in a.seeds])
71
+ b = np.array([res[(T, "official", s)]["val_mse"] for s in a.seeds])
72
+ d = 100 * (v - b) / b
73
+ sec = np.mean([res[(T, name, s)]["seconds"] for s in a.seeds])
74
+ print(f"{name:28s} val {v.mean():.5f} vs official {d.mean():+.3f}% "
75
+ f"(sd {d.std():.3f}, wins {int((d < 0).sum())}/{len(d)}) {sec:.1f}s", flush=True)
76
+
77
+
78
+ if __name__ == "__main__":
79
+ main()
benchmarks/fb_ssnpp_holdout_rows.npy ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:03c896c58e9a7d40c4cf3634203ad51635fd28c23861558b84455b14bbaa9738
3
+ size 160128
benchmarks/get_deep1m.py ADDED
@@ -0,0 +1,60 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Download DEEP* 1M exactly as the e-means paper uses it, and verify it.
2
+
3
+ Fetches bytes [0, 8 + 1,010,000 * 96 * 4) of Deep1B's base.1B.fbin from the Hugging Face mirror the
4
+ paper's download script uses (suchun/deep1b-mirror), takes the first 1M rows as train and the next
5
+ 10,000 as the evaluation set, splits that set with permutation seed 1234 (first half validation,
6
+ second half test), and checks all three against the SHA-256 fingerprints in faiss's
7
+ benchs/e_means/paper_splits/fingerprints.json (copied below). Writes train_1M.npy and test.npy.
8
+
9
+ Usage: python benchmarks/get_deep1m.py DATA_DIR
10
+ """
11
+ import hashlib, os, sys, urllib.request
12
+ import numpy as np
13
+
14
+ URL = "https://huggingface.co/datasets/suchun/deep1b-mirror/resolve/main/base.1B.fbin"
15
+ N, EVAL, D = 1_000_000, 10_000, 96
16
+ FINGERPRINTS = { # faiss benchs/e_means/paper_splits/fingerprints.json, entry "deep1b-1M"
17
+ "val": "ca94deceddb13e16d6cfb81365e2e18c319bf704b1d0ea9a28e5067bba03c458", "test": "05d685ccf5e413a0559348b3586ec82a72da637d50b7d2b9a521e70c6cd29b6f", "train_rows": "7b74aeede93a64c6a2e598f021fbaa42061a7e09e9f3f75f1c1254fbb4419368"}
18
+
19
+
20
+ def sha(a):
21
+ return hashlib.sha256(np.ascontiguousarray(a, dtype=np.float32).tobytes()).hexdigest()
22
+
23
+
24
+ def main(out):
25
+ os.makedirs(out, exist_ok=True)
26
+ files = [os.path.join(out, f) for f in ("train_1M.npy", "val.npy", "test.npy")]
27
+ if all(os.path.exists(f) for f in files): # already prepared: verify instead of downloading
28
+ train, val, test = (np.load(f, mmap_mode="r") for f in files)
29
+ sample = np.linspace(0, N - 1, 1000).astype(np.int64)
30
+ got = {"val": sha(val), "test": sha(test), "train_rows": sha(np.array(train[sample]))}
31
+ if all(got[key] == want for key, want in FINGERPRINTS.items()):
32
+ print("DEEP* 1M already in %s; fingerprints match the paper's" % out)
33
+ return
34
+ raw = os.path.join(out, "deep1b_prefix.fbin")
35
+ nbytes = 8 + (N + EVAL) * D * 4
36
+ if not os.path.exists(raw) or os.path.getsize(raw) != nbytes:
37
+ req = urllib.request.Request(URL, headers={"Range": "bytes=0-%d" % (nbytes - 1)})
38
+ with urllib.request.urlopen(req, timeout=600) as r, open(raw, "wb") as f:
39
+ while True:
40
+ b = r.read(1 << 24)
41
+ if not b:
42
+ break
43
+ f.write(b)
44
+ m = np.memmap(raw, dtype=np.float32, mode="r", offset=8, shape=(N + EVAL, D))
45
+ train, ev = m[:N], np.array(m[N:], dtype=np.float32)
46
+ perm = np.random.RandomState(1234).permutation(EVAL)
47
+ val, test = ev[perm[:EVAL // 2]], ev[perm[EVAL // 2:]]
48
+ sample = np.linspace(0, N - 1, 1000).astype(np.int64)
49
+ got = {"val": sha(val), "test": sha(test), "train_rows": sha(np.array(train[sample]))}
50
+ for key, want in FINGERPRINTS.items():
51
+ assert got[key] == want, "%s rows differ from the paper's" % key
52
+ np.save(os.path.join(out, "train_1M.npy"), np.array(train, dtype=np.float32))
53
+ np.save(os.path.join(out, "val.npy"), val)
54
+ np.save(os.path.join(out, "test.npy"), test)
55
+ os.remove(raw)
56
+ print("DEEP* 1M ready in %s; fingerprints match the paper's" % out)
57
+
58
+
59
+ if __name__ == "__main__":
60
+ main(sys.argv[1])
benchmarks/get_paper1m.py ADDED
@@ -0,0 +1,90 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Download SimSearchNet++ 1M or SPACEV 1M exactly as the e-means paper uses them, and verify.
2
+
3
+ Train: the first 1,000,000 rows of the corpus. Evaluation: the 20,000 held-out rows listed in
4
+ faiss benchs/e_means/paper_splits/<name>_holdout_rows.npy (copied here; see THIRD_PARTY.md),
5
+ read in that order, split by permutation seed 1234 into validation (first half) and test. Each
6
+ part is checked against the paper's SHA-256 fingerprints. Only the byte ranges needed are fetched
7
+ (about 256 MB for SimSearchNet++, 100 MB for SPACEV).
8
+
9
+ Usage: python benchmarks/get_paper1m.py {fb-ssnpp-1M|spacev-1M} DATA_DIR
10
+ """
11
+ import hashlib, os, sys, urllib.request
12
+ import numpy as np
13
+
14
+ HERE = os.path.dirname(os.path.abspath(__file__))
15
+ N = 1_000_000
16
+ SPACEV_PART = 4294967295
17
+ SPECS = {
18
+ "fb-ssnpp-1M": dict(
19
+ url="https://dl.fbaipublicfiles.com/billion-scale-ann-benchmarks/FB_ssnpp_database.u8bin",
20
+ d=256, dtype=np.uint8, held="fb_ssnpp_holdout_rows.npy",
21
+ sha=dict(val="d51b5e50430b61a8a4a314a5df903195005da36e4de56e0a11ad3908b0a1dd08",
22
+ test="fae0b470fc0a3c74bcc04245b821bf34045a199185b3ecae2c21f1c2a7b10e40",
23
+ train_rows="cb4988c357268c46c8a33a55cebb5d88199badeb32a72fb6187e93aa2674274b")),
24
+ "spacev-1M": dict(
25
+ parts="https://huggingface.co/datasets/jkhe/spacev1b/resolve/main/vectors.bin/vectors_%d.bin",
26
+ d=100, dtype=np.int8, held="spacev_holdout_rows.npy",
27
+ sha=dict(val="1c59da848fc792157f107733da4a2f288568c51ea4fe44cdbbc1b9cd1352acec",
28
+ test="ecb6c6d304eb772d0891685bb2cf0b417efe15924894879af7b7fcfd431a0549",
29
+ train_rows="07c827691fb18f648271d20258f7e65df5bd09ffbfb5956523e7c543ab2cbd8a")),
30
+ }
31
+
32
+
33
+ def get(spec, a, b):
34
+ """Bytes [a, b) of the (possibly multi-part) corpus file."""
35
+ out = bytearray()
36
+ while a < b:
37
+ if "parts" in spec:
38
+ i, off = divmod(a, SPACEV_PART)
39
+ stop = min(b, (i + 1) * SPACEV_PART)
40
+ url, start, end = spec["parts"] % (i + 1), off, off + stop - a
41
+ else:
42
+ url, start, end, stop = spec["url"], a, b, b
43
+ req = urllib.request.Request(url, headers={"Range": "bytes=%d-%d" % (start, end - 1)})
44
+ for attempt in range(6):
45
+ try:
46
+ with urllib.request.urlopen(req, timeout=600) as r:
47
+ chunk = r.read()
48
+ break
49
+ except Exception:
50
+ if attempt == 5:
51
+ raise
52
+ assert len(chunk) == end - start, "short read"
53
+ out += chunk
54
+ a = stop
55
+ return bytes(out)
56
+
57
+
58
+ def sha(a):
59
+ return hashlib.sha256(np.ascontiguousarray(a, dtype=np.float32).tobytes()).hexdigest()
60
+
61
+
62
+ def main(name, out):
63
+ spec = SPECS[name]
64
+ d, dt = spec["d"], np.dtype(spec["dtype"])
65
+ row = d * dt.itemsize
66
+ os.makedirs(out, exist_ok=True)
67
+ train = np.frombuffer(get(spec, 8, 8 + N * row), dtype=dt).reshape(N, d).astype(np.float32)
68
+ index = np.load(os.path.join(HERE, spec["held"]))
69
+ srt = np.sort(index)
70
+ runs = np.split(srt, np.flatnonzero(np.diff(srt) != 1) + 1)
71
+ rows = {}
72
+ for r in runs:
73
+ blob = np.frombuffer(get(spec, 8 + int(r[0]) * row, 8 + (int(r[-1]) + 1) * row), dtype=dt)
74
+ for j, v in zip(r, blob.reshape(len(r), d)):
75
+ rows[int(j)] = v
76
+ ev = np.stack([rows[int(j)] for j in index]).astype(np.float32)
77
+ perm = np.random.RandomState(1234).permutation(len(ev))
78
+ half = int(round(0.5 * len(ev)))
79
+ val, test = ev[perm[:half]], ev[perm[half:]]
80
+ sample = np.linspace(0, N - 1, 1000).astype(np.int64)
81
+ got = dict(val=sha(val), test=sha(test), train_rows=sha(train[sample]))
82
+ for key, want in spec["sha"].items():
83
+ assert got[key] == want, "%s: %s rows differ from the paper's" % (name, key)
84
+ np.save(os.path.join(out, "train_1M.npy"), train)
85
+ np.save(os.path.join(out, "test.npy"), test)
86
+ print("%s ready in %s; fingerprints match the paper's" % (name, out))
87
+
88
+
89
+ if __name__ == "__main__":
90
+ main(sys.argv[1], sys.argv[2])
benchmarks/ivf_downstream.py ADDED
@@ -0,0 +1,76 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Downstream use: the centroids as the coarse quantizer of an IVF index (the use case e-means targets).
2
+
3
+ For each method and seed, train k centroids on DEEP* 1M, build IndexIVFFlat over the same 1M rows
4
+ with those centroids, and measure on the 10K public DEEP1B queries (exact ground truth on the 1M rows):
5
+ - recall@1 and recall@10 (10-NN found among the 10 returned) at nprobe in NPROBES,
6
+ - distance computations per query (sum of probed list sizes), the real cost of an IVF search,
7
+ - list imbalance factor k * sum(n_i^2) / n^2 (1 = perfectly balanced).
8
+
9
+ Usage: python benchmarks/ivf_downstream.py DATA/deep1b OUT.jsonl --epochs 3 --seeds 10 11 12 --methods official pisces
10
+ """
11
+ import argparse, json, os, sys, time
12
+ import numpy as np
13
+ import faiss
14
+
15
+ ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
16
+ sys.path.insert(0, ROOT)
17
+ sys.path.insert(0, os.path.join(ROOT, "benchmarks"))
18
+ from paper_protocol import METHODS, make_variant # noqa: E402
19
+
20
+ NPROBES = [1, 2, 4, 8, 16, 32, 64, 128]
21
+
22
+
23
+ def read_fbin(path):
24
+ n, d = np.fromfile(path, dtype="int32", count=2)
25
+ return np.fromfile(path, dtype="float32", offset=8).reshape(n, d)
26
+
27
+
28
+ def main():
29
+ ap = argparse.ArgumentParser()
30
+ ap.add_argument("data"); ap.add_argument("out")
31
+ ap.add_argument("--k", type=int, default=16384)
32
+ ap.add_argument("--epochs", type=float, nargs="+", default=[3])
33
+ ap.add_argument("--seeds", type=int, nargs="+", default=[10, 11, 12])
34
+ ap.add_argument("--methods", nargs="+", default=["official", "pisces"])
35
+ ap.add_argument("--threads", type=int, default=4)
36
+ a = ap.parse_args()
37
+ faiss.omp_set_num_threads(a.threads)
38
+ x = np.ascontiguousarray(np.load(os.path.join(a.data, "train_1M.npy")))
39
+ q = np.ascontiguousarray(read_fbin(os.path.join(a.data, "query.public.10K.fbin")))
40
+ _, gt = faiss.knn(q, x, 10)
41
+ from aries.pemeans import PISCES
42
+ METHODS["pisces"] = make_variant(PISCES, len(x))
43
+ done = set()
44
+ if os.path.exists(a.out):
45
+ done = {(r["method"], r["epochs"], r["seed"]) for r in map(json.loads, open(a.out))}
46
+ for T in a.epochs:
47
+ for s in a.seeds:
48
+ init = x[np.random.RandomState(s).permutation(len(x))[:a.k]].copy()
49
+ for m in a.methods:
50
+ if (m, T, s) in done:
51
+ continue
52
+ t0 = time.time()
53
+ c, _ = METHODS[m](x.shape[1], T, s).train(x, init_centroids=init.copy())
54
+ sec = time.time() - t0
55
+ c = np.ascontiguousarray(c, dtype="float32")
56
+ quant = faiss.IndexFlatL2(x.shape[1]); quant.add(c)
57
+ ivf = faiss.IndexIVFFlat(quant, x.shape[1], a.k); ivf.is_trained = True; ivf.add(x)
58
+ sizes = np.array([ivf.invlists.list_size(i) for i in range(a.k)], dtype="float64")
59
+ imb = float(a.k * (sizes ** 2).sum() / sizes.sum() ** 2)
60
+ r = dict(method=m, epochs=T, seed=s, k=a.k, train_seconds=sec, imbalance=imb,
61
+ empty_lists=int((sizes == 0).sum()), curve=[])
62
+ for npb in NPROBES:
63
+ ivf.nprobe = npb
64
+ _, Dq = quant.search(q, npb)
65
+ ndis = float(sizes[Dq].sum(1).mean())
66
+ t1 = time.time(); _, I = ivf.search(q, 10); qs = time.time() - t1
67
+ r1 = float((I[:, 0] == gt[:, 0]).mean())
68
+ r10 = float(np.mean([len(set(I[i]) & set(gt[i])) / 10 for i in range(len(q))]))
69
+ r["curve"].append(dict(nprobe=npb, recall1=r1, recall10=r10, ndis=ndis, seconds=qs))
70
+ with open(a.out, "a") as f:
71
+ f.write(json.dumps(r) + "\n")
72
+ print(m, T, s, f"imb {imb:.3f}", " ".join(f"{c_['nprobe']}:{c_['recall1']:.3f}/{c_['ndis']:.0f}" for c_ in r["curve"]), flush=True)
73
+
74
+
75
+ if __name__ == "__main__":
76
+ main()
benchmarks/mechanism_analysis.py ADDED
@@ -0,0 +1,161 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Why PISCES works: revival timeline, cluster-size balance, and an empirical check of Lemma 2.
2
+
3
+ For one seed on DEEP* 1M (k = 16,384, 3 epochs), train official e-means and PISCES from the same
4
+ starting rows and record:
5
+ * every revival (training step, number of clusters re-seeded),
6
+ * the final cluster sizes on the 1M training rows and per-cluster distortion,
7
+ * Lemma 2 on real clusters: for 400 random clusters with >= 20 rows, the distortion removed by the
8
+ best split along the top principal axis (2-means restricted to the axis, 10 Lloyd iterations
9
+ started at c +/- sqrt(2 lambda_1 / pi) u_1) against the prediction (2/pi) N lambda_1, and against
10
+ an unrestricted 2-means of the cluster (faiss, 5 restarts).
11
+ Writes one JSON file used by paper/make_data.py.
12
+
13
+ Usage: python benchmarks/mechanism_analysis.py DATA/deep1b OUT.json [--seed 10]
14
+ """
15
+ import argparse, json, math, os, sys, time
16
+ import numpy as np
17
+ import faiss
18
+
19
+ ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
20
+ sys.path.insert(0, ROOT)
21
+ from faiss.contrib.e_means import NumpyEmeans # noqa: E402
22
+ from aries.pemeans import PEmeans, PISCES # noqa: E402
23
+
24
+ K = 16384
25
+
26
+
27
+ def recorder(cls):
28
+ class Rec(cls):
29
+ def _apply_splits(self, source, target):
30
+ n = super()._apply_splits(source, target)
31
+ self.split_log.append((int(self._step), int(len(source))))
32
+ return n
33
+ return Rec
34
+
35
+
36
+ def sizes_and_distortion(x, c):
37
+ D, I = faiss.knn(x, c, 1)
38
+ cnt = np.bincount(I[:, 0], minlength=len(c))
39
+ dist = np.bincount(I[:, 0], weights=D[:, 0].astype("float64"), minlength=len(c))
40
+ return cnt, dist, I[:, 0]
41
+
42
+
43
+ def lemma_check(x, c, lab, rng, m=400):
44
+ out = []
45
+ cnt = np.bincount(lab, minlength=len(c))
46
+ pool = np.flatnonzero(cnt >= 20)
47
+ for j in rng.choice(pool, min(m, len(pool)), replace=False):
48
+ X = x[lab == j].astype("float64")
49
+ N = len(X); mu = X.mean(0); E = X - mu
50
+ before = float((E ** 2).sum())
51
+ w, V = np.linalg.eigh(E.T @ E / N)
52
+ lam, u = float(w[-1]), V[:, -1]
53
+ pred = 2 / math.pi * N * lam
54
+ # split restricted to the axis: Lloyd on the projections, started at the Lemma 1 positions
55
+ p = E @ u; a, b = math.sqrt(2 * lam / math.pi), -math.sqrt(2 * lam / math.pi)
56
+ for _ in range(10):
57
+ s = p > (a + b) / 2
58
+ if s.all() or (~s).all():
59
+ break
60
+ a, b = p[s].mean(), p[~s].mean()
61
+ s = p > (a + b) / 2
62
+ A, B = X[s], X[~s]
63
+ axis_after = float(((A - A.mean(0)) ** 2).sum() + ((B - B.mean(0)) ** 2).sum()) if len(A) and len(B) else before
64
+ km = faiss.Kmeans(X.shape[1], 2, niter=20, nredo=5, seed=int(j), max_points_per_centroid=10**9)
65
+ km.train(X.astype("float32"))
66
+ Dd, _ = km.index.search(X.astype("float32"), 1)
67
+ free_after = float(Dd.sum())
68
+ out.append(dict(N=N, lam1=lam, trace=float(np.trace(E.T @ E / N)), predicted=pred,
69
+ axis_split=before - axis_after, free_2means=before - free_after))
70
+ return out
71
+
72
+
73
+ def swap_audit(x, c, lab, D1):
74
+ """How many global split-merge moves would still pay off? A move deletes cluster t (its rows go to
75
+ their runner-up centroid, cost = sum of d2 - d1) and splits cluster j along its top axis (gain
76
+ (2/pi) N_j lambda_1,j, Lemma 2). Greedy pairing of the richest donors with the cheapest clusters,
77
+ never deleting a cluster that absorbs the rows of another deleted one."""
78
+ D2, I2 = faiss.knn(x, c, 2)
79
+ l2 = I2[:, 1]
80
+ cost = np.bincount(lab, weights=(D2[:, 1] - D2[:, 0]).astype("float64"), minlength=len(c))
81
+ e = x - c[lab]
82
+ u = np.random.RandomState(0).standard_normal(c.shape).astype("float32")
83
+ order = np.argsort(lab, kind="stable"); uniq, start = np.unique(lab[order], return_index=True)
84
+ for _ in range(5):
85
+ u /= np.maximum(np.linalg.norm(u, axis=1, keepdims=True), 1e-12)
86
+ p = np.einsum("ij,ij->i", e, u[lab])
87
+ V = np.zeros_like(c); V[uniq] = np.add.reduceat((e * p[:, None])[order], start, axis=0); u = V
88
+ u /= np.maximum(np.linalg.norm(u, axis=1, keepdims=True), 1e-12)
89
+ p = np.einsum("ij,ij->i", e, u[lab])
90
+ gain = 2 / math.pi * np.bincount(lab, weights=p.astype("float64") ** 2, minlength=len(c))
91
+ rich, cheap = np.argsort(-gain), np.argsort(cost)
92
+ blocked = np.zeros(len(c), bool); moves, total, ci = 0, 0.0, 0
93
+ for j in rich:
94
+ if blocked[j]:
95
+ continue
96
+ while ci < len(cheap) and (blocked[cheap[ci]] or cheap[ci] == j):
97
+ ci += 1
98
+ if ci >= len(cheap) or gain[j] <= cost[cheap[ci]]:
99
+ break
100
+ t = cheap[ci]; ci += 1; moves += 1; total += gain[j] - cost[t]
101
+ blocked[j] = blocked[t] = True; blocked[np.unique(l2[lab == t])] = True
102
+ return dict(profitable_moves=moves, predicted_gain_per_row=total / len(x),
103
+ gain_quantiles=np.quantile(gain, [0.5, 0.9, 0.99, 1.0]).tolist(),
104
+ cost_quantiles=np.quantile(cost, [0.0, 0.01, 0.1, 0.5]).tolist())
105
+
106
+
107
+ def split_examples(x, c, lab, rng, m=3, cap=1500):
108
+ """Real clusters for the split-geometry figure: rows projected onto the top two principal axes,
109
+ with the Lemma 1 children, the e-means children (on the centroid) and an exact 2-means."""
110
+ cnt = np.bincount(lab, minlength=len(c))
111
+ out = []
112
+ for j in np.argsort(-cnt)[:m]:
113
+ X = x[lab == j].astype("float64"); mu = X.mean(0); E = X - mu
114
+ w, V = np.linalg.eigh(E.T @ E / len(X)); u1, u2 = V[:, -1], V[:, -2]
115
+ P = np.stack([E @ u1, E @ u2], 1)
116
+ km = faiss.Kmeans(X.shape[1], 2, niter=25, nredo=5, seed=int(j), max_points_per_centroid=10**9)
117
+ km.train(X.astype("float32"))
118
+ K2 = (km.centroids.astype("float64") - mu) @ np.stack([u1, u2], 1)
119
+ sub = rng.choice(len(X), min(cap, len(X)), replace=False)
120
+ out.append(dict(N=int(len(X)), lam1=float(w[-1]), lam2=float(w[-2]), points=P[sub].round(4).tolist(),
121
+ pisces=[[math.sqrt(2 * w[-1] / math.pi), 0.0], [-math.sqrt(2 * w[-1] / math.pi), 0.0]],
122
+ kmeans2=K2.round(4).tolist()))
123
+ return out
124
+
125
+
126
+ def main():
127
+ ap = argparse.ArgumentParser()
128
+ ap.add_argument("data"); ap.add_argument("out")
129
+ ap.add_argument("--seed", type=int, default=10); ap.add_argument("--epochs", type=float, default=3)
130
+ ap.add_argument("--threads", type=int, default=4)
131
+ a = ap.parse_args()
132
+ faiss.omp_set_num_threads(a.threads)
133
+ sys.path.insert(0, os.path.join(ROOT, "benchmarks"))
134
+ from paper_protocol import make_variant # noqa: E402
135
+ x = np.ascontiguousarray(np.load(os.path.join(a.data, "train_1M.npy")))
136
+ init = x[np.random.RandomState(a.seed).permutation(len(x))[:K]].copy()
137
+ res = {"seed": a.seed, "epochs": a.epochs, "k": K}
138
+ for name in ["official", "pisces"]:
139
+ if name == "official":
140
+ km = recorder(NumpyEmeans)(x.shape[1], K, nepochs=a.epochs, seed=a.seed, batch_law="throughput")
141
+ else:
142
+ build = make_variant(PISCES, len(x))
143
+ km = build(x.shape[1], a.epochs, a.seed) # PEmeans with the PISCES configuration
144
+ km.__class__ = recorder(PEmeans)
145
+ km.split_log = []
146
+ t0 = time.time(); c, _ = km.train(x, init_centroids=init.copy()); sec = time.time() - t0
147
+ cnt, dist, lab = sizes_and_distortion(x, c)
148
+ res[name] = dict(seconds=sec, total_steps=int(getattr(km, "recipe", None)["total_steps"]) if getattr(km, "recipe", None) else None,
149
+ split_log=km.split_log, sizes=cnt.tolist(), distortion=dist.tolist(),
150
+ train_mse=float(dist.sum() / len(x)),
151
+ lemma=lemma_check(x, c, lab, np.random.RandomState(a.seed)),
152
+ swaps=swap_audit(x, c, lab, None),
153
+ split_examples=split_examples(x, c, lab, np.random.RandomState(a.seed)))
154
+ print(name, f"{sec:.0f}s", "splits", sum(n for _, n in km.split_log),
155
+ "train mse", round(res[name]["train_mse"], 5), "empty", int((cnt == 0).sum()),
156
+ "profitable swaps", res[name]["swaps"]["profitable_moves"], flush=True)
157
+ json.dump(res, open(a.out, "w"))
158
+
159
+
160
+ if __name__ == "__main__":
161
+ main()
benchmarks/paper_protocol.py ADDED
@@ -0,0 +1,143 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """e-means paper protocol on CPU: DEEP* 1M (k = 16,384), test MSE on the paper's test rows.
2
+
3
+ Data: run ``python benchmarks/get_deep1m.py DATA_DIR`` first. It downloads the first 1,010,000
4
+ rows of Deep1B and checks them against the fingerprints shipped with faiss/benchs/e_means
5
+ (paper_splits/fingerprints.json), so train and test rows are byte-identical to the paper's.
6
+
7
+ Protocol (Bessa et al. 2026, arXiv:2611.09501, Sec. 4 and benchs/e_means/paper_xb.py):
8
+ * k = 16,384 at n = 1M, batch law "throughput", no per-dataset tuning
9
+ * starting centroids: the first k rows of a seeded permutation, identical for every method
10
+ * test error: nearest centroid ranked in float32, distance recomputed in float64
11
+ Differences from the paper, stated up front: CPU float32 assignment (paper: H100 bfloat16), and
12
+ NumPy's seeded permutation for the starting rows (paper: a CUDA generator), shared by all methods.
13
+
14
+ Usage: python benchmarks/paper_protocol.py DATA_DIR OUT.jsonl --epochs 3 --seeds 10 11 12 13 14
15
+ [--methods official pemeans ...] [--threads 4]
16
+ Each finished run is appended to OUT.jsonl; runs already there are skipped (resumable).
17
+ """
18
+ import argparse, json, os, sys, time
19
+ import numpy as np
20
+ import faiss
21
+
22
+ sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
23
+ from faiss.contrib.e_means import NumpyEmeans # noqa: E402
24
+ from aries.pemeans import PEmeans # noqa: E402
25
+
26
+ K = 16384
27
+
28
+ METHODS = {
29
+ "official": lambda d, T, s: NumpyEmeans(d, K, nepochs=T, seed=s, batch_law="throughput"),
30
+ "pemeans": lambda d, T, s: PEmeans(d, K, nepochs=T, seed=s, batch_law="throughput",
31
+ split="principal", gain="greedy"),
32
+ "pe_split_only": lambda d, T, s: PEmeans(d, K, nepochs=T, seed=s, batch_law="throughput",
33
+ split="principal", gain="usage"),
34
+ "pe_gain_only": lambda d, T, s: PEmeans(d, K, nepochs=T, seed=s, batch_law="throughput",
35
+ split="faiss", gain="greedy"),
36
+ "pe_sample": lambda d, T, s: PEmeans(d, K, nepochs=T, seed=s, batch_law="throughput",
37
+ split="principal", gain="sample"),
38
+ }
39
+
40
+
41
+ def test_mse(c, test):
42
+ _, idx = faiss.knn(np.ascontiguousarray(test), np.ascontiguousarray(c, dtype="float32"), 1)
43
+ diff = test.astype(np.float64) - c[idx[:, 0]].astype(np.float64)
44
+ return float((diff * diff).sum() / len(test))
45
+
46
+
47
+ def train_mse(c, x, chunk=1 << 17):
48
+ tot = 0.0
49
+ for s in range(0, len(x), chunk):
50
+ D, _ = faiss.knn(x[s:s + chunk], c, 1)
51
+ tot += float(D.sum(dtype=np.float64))
52
+ return tot / len(x)
53
+
54
+
55
+ def make_variant(kw, n):
56
+ """A PEmeans configuration; alpha_scale / batch_scale rescale the e-means fitted laws."""
57
+ kw = dict(kw)
58
+ bs, asc = kw.pop("batch_scale", None), kw.pop("alpha_scale", None)
59
+
60
+ def build(d, T, s):
61
+ extra = {}
62
+ if bs is not None or asc is not None:
63
+ from faiss.contrib.e_means.base import _resolve_recipe, _step_law
64
+ B, _ = _resolve_recipe(n, K, T, "throughput", None, None)
65
+ B = int(B * (bs or 1.0))
66
+ extra = dict(batch_size=B, alpha0=min(1.0, _step_law(n, K, T, B) * (asc or 1.0)))
67
+ return PEmeans(d, K, nepochs=T, seed=s, batch_law="throughput", **kw, **extra)
68
+ build.kwargs = dict(kw, batch_scale=bs, alpha_scale=asc)
69
+ return build
70
+
71
+
72
+ def code_version():
73
+ import subprocess
74
+ try:
75
+ root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
76
+ rev = subprocess.run(["git", "-C", root, "rev-parse", "--short", "HEAD"], capture_output=True, text=True).stdout.strip()
77
+ dirty = subprocess.run(["git", "-C", root, "status", "--porcelain", "aries"], capture_output=True, text=True).stdout.strip()
78
+ return rev + ("+dirty" if dirty else "")
79
+ except Exception:
80
+ return "unknown"
81
+
82
+
83
+ def main():
84
+ ap = argparse.ArgumentParser()
85
+ ap.add_argument("data"); ap.add_argument("out")
86
+ ap.add_argument("--epochs", type=float, nargs="+", default=[3])
87
+ ap.add_argument("--seeds", type=int, nargs="+", default=[10, 11, 12, 13, 14])
88
+ ap.add_argument("--methods", nargs="+", default=["official", "pemeans"])
89
+ ap.add_argument("--threads", type=int, default=4)
90
+ ap.add_argument("--train-mse", action="store_true")
91
+ ap.add_argument("--dataset", default="deep1b-1M")
92
+ ap.add_argument("--k", type=int, default=16384, help="number of centroids (paper: 16,384)")
93
+ ap.add_argument("--variant", nargs=2, action="append", default=[],
94
+ help="name and PEmeans kwargs as JSON; may include alpha_scale / batch_scale")
95
+ ap.add_argument("--eval", choices=["test", "val"], default="test",
96
+ help="val for design decisions, test for reported results")
97
+ a = ap.parse_args()
98
+ global K
99
+ K = a.k
100
+ faiss.omp_set_num_threads(a.threads)
101
+ x = np.ascontiguousarray(np.load(os.path.join(a.data, "train_1M.npy")))
102
+ test = np.load(os.path.join(a.data, a.eval + ".npy"))
103
+ from aries.pemeans import PE_V2, PE_V3, PISCES, PISCES_V10, PISCES_POLISH, PISCES_FUSED, PISCES_FAST
104
+ METHODS["pisces_v10"] = make_variant(PISCES_V10, len(x))
105
+ METHODS["pisces_fused"] = make_variant(PISCES_FUSED, len(x))
106
+ METHODS["pisces_fast"] = make_variant(PISCES_FAST, len(x))
107
+ METHODS["pisces_polish"] = make_variant(PISCES_POLISH, len(x))
108
+ METHODS["pemeans_v2"] = make_variant(PE_V2, len(x))
109
+ METHODS["pemeans_v3"] = make_variant(PE_V3, len(x))
110
+ METHODS["pisces"] = make_variant(PISCES, len(x))
111
+ METHODS["emeans_approx"] = make_variant(dict(split="faiss", gain="usage", approx="auto"), len(x))
112
+ for name, js in a.variant:
113
+ METHODS[name] = make_variant(json.loads(js), len(x))
114
+ a.methods.append(name) if name not in a.methods else None
115
+ done = set()
116
+ if os.path.exists(a.out):
117
+ for line in open(a.out):
118
+ r = json.loads(line)
119
+ done.add((r.get("dataset"), r["method"], r["epochs"], r["seed"], r.get("eval", "test")))
120
+ for T in a.epochs:
121
+ for seed in a.seeds:
122
+ init = x[np.random.RandomState(seed).permutation(len(x))[:K]].copy()
123
+ for m in a.methods:
124
+ if (a.dataset, m, T, seed, a.eval) in done:
125
+ continue
126
+ t0 = time.time()
127
+ km = METHODS[m](x.shape[1], T, seed)
128
+ c, _ = km.train(x, init_centroids=init.copy())
129
+ sec = time.time() - t0
130
+ r = dict(dataset=a.dataset, k=K, method=m, epochs=T, seed=seed, seconds=sec,
131
+ eval=a.eval, test_mse=test_mse(c, test), splits=int(km.n_splits),
132
+ batch=int(km.recipe["batch_size"]), alpha0=float(km.recipe["alpha0"]),
133
+ threads=a.threads, faiss=faiss.__version__, code=code_version(),
134
+ kwargs=getattr(METHODS[m], "kwargs", None))
135
+ if a.train_mse:
136
+ r["train_mse"] = train_mse(c, x)
137
+ with open(a.out, "a") as f:
138
+ f.write(json.dumps(r) + "\n")
139
+ print(json.dumps(r), flush=True)
140
+
141
+
142
+ if __name__ == "__main__":
143
+ main()
benchmarks/plot.py ADDED
@@ -0,0 +1,37 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Test MSE vs wall-clock for each method (mean over seeds, bars = sd).
2
+
3
+ Usage: python benchmarks/plot.py results/deep1m_cpu.jsonl results/deep1m_cpu.png
4
+ """
5
+ import collections, json, sys
6
+ import numpy as np
7
+ import matplotlib
8
+ matplotlib.use("Agg")
9
+ import matplotlib.pyplot as plt
10
+
11
+ rows = [json.loads(l) for l in open(sys.argv[1])]
12
+ by = collections.defaultdict(list)
13
+ for r in rows:
14
+ by[(r["method"], r["epochs"])].append(r)
15
+ style = {"official": ("#16233F", "o", "e-means (official faiss 1.16)"),
16
+ "pemeans": ("#9FD3D1", "D", "PE-means v1"),
17
+ "pemeans_v2": ("#3FA7A5", "s", "PE-means v2"),
18
+ "pemeans_v3": ("#0E8C8C", "^", "PE-means v3 (ours, final)")}
19
+ fig, ax = plt.subplots(figsize=(6.4, 4.0), dpi=160)
20
+ for m, (c, mk, lab) in style.items():
21
+ pts = sorted((T, np.mean([r["seconds"] for r in v]), np.mean([r["test_mse"] for r in v]), np.std([r["test_mse"] for r in v]))
22
+ for (mm, T), v in by.items() if mm == m)
23
+ if not pts:
24
+ continue
25
+ T, s, mu, sd = map(np.array, zip(*pts))
26
+ ax.errorbar(s, mu, yerr=sd, color=c, marker=mk, lw=2, ms=6, capsize=3, label=lab)
27
+ for t, x, y in zip(T, s, mu):
28
+ ax.annotate(f"{t:g} ep.", (x, y), xytext=(6, 4), textcoords="offset points", fontsize=8, color=c)
29
+ ref = {"deep1b-1M": 0.38921320633218964}.get(rows[0].get("dataset"))
30
+ if ref:
31
+ ax.axhline(ref, color="#C4413A", ls="--", lw=1)
32
+ ax.text(ax.get_xlim()[1], ref, "Lloyd 25 it. (paper, CPU median) ", color="#C4413A", fontsize=8, ha="right", va="bottom")
33
+ ax.set_xlabel("wall-clock seconds (4 CPU threads)"); ax.set_ylabel("test MSE (lower is better)")
34
+ ax.set_title(f"{rows[0].get('dataset', '')}, k = 16,384, mean ± sd over seeds", fontsize=10)
35
+ ax.grid(alpha=0.3); ax.legend(frameon=False)
36
+ fig.tight_layout(); fig.savefig(sys.argv[2])
37
+ print("wrote", sys.argv[2])
benchmarks/proxy.py ADDED
@@ -0,0 +1,36 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Quick proxy: DEEP prefix, small k. Usage: python proxy.py DATA_DIR n k T seeds..."""
2
+ import sys, time, json
3
+ import numpy as np, faiss
4
+ sys.path.insert(0, __file__.rsplit("/benchmarks", 1)[0])
5
+ from faiss.contrib.e_means import NumpyEmeans
6
+ from aries.pemeans import PEmeans
7
+ faiss.omp_set_num_threads(4)
8
+ data, n, k, T = sys.argv[1], int(sys.argv[2]), int(sys.argv[3]), float(sys.argv[4])
9
+ seeds = [int(s) for s in sys.argv[5:]]
10
+ x = np.ascontiguousarray(np.load(f"{data}/train_1M.npy", mmap_mode="r")[:n])
11
+ test = np.load(f"{data}/test.npy")
12
+
13
+ def mse(c):
14
+ D, I = faiss.knn(test, c, 1)
15
+ diff = test.astype(np.float64) - c[I[:, 0]].astype(np.float64)
16
+ return (diff * diff).sum() / len(test)
17
+
18
+ V = {"official": lambda s: NumpyEmeans(x.shape[1], k, nepochs=T, seed=s, batch_law="throughput"),
19
+ "PE(faiss,usage)=identity": lambda s: PEmeans(x.shape[1], k, nepochs=T, seed=s, batch_law="throughput", split="faiss", gain="usage"),
20
+ "PE(principal,usage)": lambda s: PEmeans(x.shape[1], k, nepochs=T, seed=s, batch_law="throughput", split="principal", gain="usage"),
21
+ "PE(faiss,greedy)": lambda s: PEmeans(x.shape[1], k, nepochs=T, seed=s, batch_law="throughput", split="faiss", gain="greedy"),
22
+ "PE(principal,greedy)": lambda s: PEmeans(x.shape[1], k, nepochs=T, seed=s, batch_law="throughput", split="principal", gain="greedy"),
23
+ "PE(principal,sample)": lambda s: PEmeans(x.shape[1], k, nepochs=T, seed=s, batch_law="throughput", split="principal", gain="sample")}
24
+ res = {}
25
+ for name, mk in V.items():
26
+ vals, secs = [], []
27
+ for s in seeds:
28
+ init = x[np.random.RandomState(s).permutation(n)[:k]].copy()
29
+ t = time.time(); km = mk(s); c, _ = km.train(x, init_centroids=init); secs.append(time.time() - t)
30
+ vals.append(mse(c))
31
+ res[name] = vals
32
+ print(f"{name:28s} test MSE {np.mean(vals):.5f} ± {np.std(vals):.5f} ({np.mean(secs):.1f}s) per-seed {[round(v,5) for v in vals]}", flush=True)
33
+ base = np.array(res["official"])
34
+ for name, v in res.items():
35
+ d = 100 * (np.array(v) - base) / base
36
+ print(f"{name:28s} paired change vs official: {d.mean():+.3f}% (wins {int((d<0).sum())}/{len(d)})")
benchmarks/run_analysis.sh ADDED
@@ -0,0 +1,26 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env bash
2
+ # Deep analysis of the frozen PISCES (after run_final_pisces.sh). Usage: bash benchmarks/run_analysis.sh DATA_ROOT [THREADS]
3
+ # 1. leave-one-out ablation at full scale (DEEP* 1M, k = 16,384, validation rows, seeds 1-3, 3 epochs)
4
+ # 2. scaling in k (1,024 / 4,096 / 65,536 at n = 1M; validation rows, seeds 1-3, 3 epochs)
5
+ # 3. downstream IVF index quality (10K DEEP1B queries, exact ground truth on the 1M rows)
6
+ # 4. mechanism: revival timeline, cluster sizes, Lemma 2 on real clusters
7
+ set -euo pipefail
8
+ DATA=$1; T=${2:-4}; cd "$(dirname "$0")/.."
9
+ P="python -I benchmarks/paper_protocol.py"
10
+ R=${RESULTS:-results}; mkdir -p "$R"
11
+ BASE='"split":"principal","gain":"greedy","split_pass":true,"tau":0.2,"every_until":0.3,"approx":"auto","approx_last":"auto","fused":true,"codec":"sq8r"'
12
+ $P "$DATA/deep1b" "$R"/ablation_full_val.jsonl --eval val --epochs 3 --seeds 1 2 3 --threads $T --methods official pisces \
13
+ --variant no_principal_split "{${BASE/\"split\":\"principal\"/\"split\":\"faiss\"},\"alpha_scale\":1.4}" \
14
+ --variant no_gain_ranking "{${BASE/\"gain\":\"greedy\"/\"gain\":\"usage\"},\"alpha_scale\":1.4}" \
15
+ --variant no_consistent_closing "{${BASE/\"split_pass\":true/\"split_pass\":false},\"alpha_scale\":1.4}" \
16
+ --variant tau_0.1 "{${BASE/\"tau\":0.2/\"tau\":0.1},\"alpha_scale\":1.4}" \
17
+ --variant step_x1.0 "{$BASE,\"alpha_scale\":1.0}" \
18
+ --variant no_early_revival "{${BASE/\"every_until\":0.3/\"every_until\":0.0},\"alpha_scale\":1.4}" \
19
+ --variant exact_last_epoch "{${BASE/,\"approx_last\":\"auto\"/},\"alpha_scale\":1.4}" \
20
+ --variant exact_assignment "{\"split\":\"principal\",\"gain\":\"greedy\",\"split_pass\":true,\"tau\":0.2,\"every_until\":0.3,\"fused\":true,\"alpha_scale\":1.4}" \
21
+ --variant no_engineering "{${BASE/,\"fused\":true,\"codec\":\"sq8r\"/},\"alpha_scale\":1.4}"
22
+ for k in 1024 4096 65536; do
23
+ $P "$DATA/deep1b" "$R"/scaling_k${k}_val.jsonl --k $k --eval val --epochs 3 --seeds 1 2 3 --threads $T --methods official pisces
24
+ done
25
+ python -I benchmarks/ivf_downstream.py "$DATA/deep1b" "$R"/ivf_downstream.jsonl --epochs 3 --seeds 10 11 12 --methods official pisces --threads $T
26
+ python -I benchmarks/mechanism_analysis.py "$DATA/deep1b" "$R"/mechanism_seed10.json --seed 10 --threads $T
benchmarks/run_final.sh ADDED
@@ -0,0 +1,20 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env bash
2
+ # Final confirmation of PISCES (iterations 1-22) in one session, so every time ratio is same-machine.
3
+ # Usage: bash benchmarks/run_final.sh DATA_ROOT [THREADS] (results in results/final/)
4
+ set -euo pipefail
5
+ DATA=$1; T=${2:-4}; cd "$(dirname "$0")/.."
6
+ P="python -I benchmarks/paper_protocol.py"
7
+ R=results/final; mkdir -p $R
8
+ # 1. time to quality, validation rows, seeds 1-5
9
+ $P "$DATA/deep1b" $R/deep1m_ttq_val.jsonl --eval val --seeds 1 2 3 4 5 --epochs 3 5 --methods official --threads $T
10
+ $P "$DATA/deep1b" $R/deep1m_ttq_val.jsonl --eval val --seeds 1 2 3 4 5 --epochs 1 2 3 4 --methods pisces --threads $T
11
+ $P "$DATA/deep1b" $R/deep1m_ttq_val.jsonl --eval val --seeds 1 2 3 4 5 --epochs 3 --methods emeans_approx --threads $T
12
+ # 2. DEEP* 1M test rows, report seeds 10-19
13
+ $P "$DATA/deep1b" $R/deep1m_test.jsonl --epochs 1 3 5 --seeds 10 11 12 13 14 15 16 17 18 19 --methods official pisces --threads $T
14
+ $P "$DATA/deep1b" $R/deep1m_test.jsonl --epochs 3 --seeds 10 11 12 13 14 15 16 17 18 19 --methods emeans_approx pisces_v10 --threads $T
15
+ # 3. held-out datasets
16
+ $P "$DATA/spacev" $R/spacev1m_test.jsonl --dataset spacev-1M --epochs 3 --seeds 10 11 12 13 14 --methods official pisces emeans_approx --threads $T
17
+ $P "$DATA/ssnpp" $R/ssnpp1m_test.jsonl --dataset fb-ssnpp-1M --epochs 3 --seeds 10 11 12 13 14 --methods official pisces emeans_approx --threads $T
18
+ # 4. analysis: leave-one-out, scaling in k, downstream IVF, mechanism
19
+ RESULTS=$R bash benchmarks/run_analysis.sh "$DATA" $T
20
+ echo FINAL-DONE
benchmarks/run_final_pisces.sh ADDED
@@ -0,0 +1,26 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env bash
2
+ # Confirmation of the frozen PISCES (ARIES #1). Usage: RESULTS=dir bash benchmarks/run_final_pisces.sh DATA_ROOT [THREADS]
3
+ # Compares official e-means, PISCES, and e-means given the same approximate assignment (fairness).
4
+ set -euo pipefail
5
+ DATA=$1; T=${2:-4}; cd "$(dirname "$0")/.."
6
+ P="python -I benchmarks/paper_protocol.py"
7
+ R=${RESULTS:-results}
8
+ mkdir -p "$R"; for f in deep1m_cpu ssnpp1m_cpu spacev1m_cpu; do
9
+ [ "$R" != results ] && [ -f results/$f.jsonl ] && [ ! -f "$R/$f.jsonl" ] && cp results/$f.jsonl "$R/"; done
10
+ sync_back() { [ "$R" != results ] && cp "$R"/*.jsonl results/ || true; }
11
+ TTQ="$R"/deep1m_pisces_time_to_quality_val.jsonl
12
+ [ -f "$TTQ" ] || { [ -f "$R"/deep1m_v3_time_to_quality_val.jsonl ] && grep '"method": "official"' "$R"/deep1m_v3_time_to_quality_val.jsonl > "$TTQ" || true; }
13
+ # 1. time-to-quality, validation rows
14
+ $P "$DATA/deep1b" "$TTQ" --eval val --seeds 1 2 3 4 5 --epochs 3 5 --methods official --threads $T
15
+ $P "$DATA/deep1b" "$TTQ" --eval val --seeds 1 2 3 4 5 --epochs 1 2 3 4 --methods pisces --threads $T
16
+ $P "$DATA/deep1b" "$TTQ" --eval val --seeds 1 2 3 4 5 --epochs 3 --methods emeans_approx --threads $T
17
+ sync_back
18
+ # 2. DEEP* 1M test rows, report seeds 10-19
19
+ $P "$DATA/deep1b" "$R"/deep1m_cpu.jsonl --epochs 1 3 5 --seeds 10 11 12 13 14 15 16 17 18 19 --methods official pisces --threads $T
20
+ $P "$DATA/deep1b" "$R"/deep1m_cpu.jsonl --epochs 3 --seeds 10 11 12 13 14 15 16 17 18 19 --methods emeans_approx --threads $T
21
+ sync_back
22
+ # 3. held-out datasets
23
+ for D in "ssnpp fb-ssnpp-1M ssnpp1m_cpu" "spacev spacev-1M spacev1m_cpu"; do set -- $D
24
+ $P "$DATA/$1" "$R"/$3.jsonl --dataset $2 --epochs 3 --seeds 10 11 12 13 14 --methods official pisces emeans_approx --threads $T
25
+ done
26
+ sync_back
benchmarks/run_final_v2.sh ADDED
@@ -0,0 +1,20 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env bash
2
+ # Confirmation of the frozen PE-means v2. Usage: bash benchmarks/run_final_v2.sh DATA_ROOT [THREADS]
3
+ set -euo pipefail
4
+ DATA=$1; T=${2:-4}; cd "$(dirname "$0")/.."
5
+ P="python -I benchmarks/paper_protocol.py"
6
+ R=${RESULTS:-results} # set RESULTS to a scratch dir to keep the tree clean
7
+ mkdir -p "$R"; for f in deep1m_cpu ssnpp1m_cpu spacev1m_cpu; do
8
+ [ "$R" != results ] && [ -f results/$f.jsonl ] && [ ! -f "$R/$f.jsonl" ] && cp results/$f.jsonl "$R/"; done
9
+ sync_back() { [ "$R" != results ] && cp "$R"/*.jsonl results/ || true; }
10
+ # 1. time-to-quality on validation rows (dev seeds)
11
+ $P "$DATA/deep1b" "$R"/deep1m_v2_time_to_quality_val.jsonl --eval val --seeds 1 2 3 4 5 --epochs 3 5 --methods official --threads $T
12
+ $P "$DATA/deep1b" "$R"/deep1m_v2_time_to_quality_val.jsonl --eval val --seeds 1 2 3 4 5 --epochs 2 2.5 3 --methods pemeans_v2 --threads $T
13
+ sync_back
14
+ # 2. final DEEP* 1M test rows, report seeds 10-19
15
+ $P "$DATA/deep1b" "$R"/deep1m_cpu.jsonl --epochs 1 3 5 --seeds 10 11 12 13 14 15 16 17 18 19 --methods official pemeans_v2 --threads $T
16
+ sync_back
17
+ # 3. held-out datasets, never used in development
18
+ $P "$DATA/ssnpp" "$R"/ssnpp1m_cpu.jsonl --dataset fb-ssnpp-1M --epochs 3 --seeds 10 11 12 13 14 --methods official pemeans_v2 --threads $T
19
+ $P "$DATA/spacev" "$R"/spacev1m_cpu.jsonl --dataset spacev-1M --epochs 3 --seeds 10 11 12 13 14 --methods official pemeans_v2 --threads $T
20
+ sync_back
benchmarks/run_final_v3.sh ADDED
@@ -0,0 +1,20 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env bash
2
+ # Confirmation of the frozen PE-means v3. Usage: bash benchmarks/run_final_v2.sh DATA_ROOT [THREADS]
3
+ set -euo pipefail
4
+ DATA=$1; T=${2:-4}; cd "$(dirname "$0")/.."
5
+ P="python -I benchmarks/paper_protocol.py"
6
+ R=${RESULTS:-results} # set RESULTS to a scratch dir to keep the tree clean
7
+ mkdir -p "$R"; for f in deep1m_cpu ssnpp1m_cpu spacev1m_cpu; do
8
+ [ "$R" != results ] && [ -f results/$f.jsonl ] && [ ! -f "$R/$f.jsonl" ] && cp results/$f.jsonl "$R/"; done
9
+ sync_back() { [ "$R" != results ] && cp "$R"/*.jsonl results/ || true; }
10
+ # 1. time-to-quality on validation rows (dev seeds)
11
+ $P "$DATA/deep1b" "$R"/deep1m_v3_time_to_quality_val.jsonl --eval val --seeds 1 2 3 4 5 --epochs 3 5 --methods official --threads $T
12
+ $P "$DATA/deep1b" "$R"/deep1m_v3_time_to_quality_val.jsonl --eval val --seeds 1 2 3 4 5 --epochs 2 3 4 --methods pemeans_v3 --threads $T
13
+ sync_back
14
+ # 2. final DEEP* 1M test rows, report seeds 10-19
15
+ $P "$DATA/deep1b" "$R"/deep1m_cpu.jsonl --epochs 1 3 5 --seeds 10 11 12 13 14 15 16 17 18 19 --methods official pemeans_v3 --threads $T
16
+ sync_back
17
+ # 3. held-out datasets, never used in development
18
+ $P "$DATA/ssnpp" "$R"/ssnpp1m_cpu.jsonl --dataset fb-ssnpp-1M --epochs 3 --seeds 10 11 12 13 14 --methods official pemeans_v3 --threads $T
19
+ $P "$DATA/spacev" "$R"/spacev1m_cpu.jsonl --dataset spacev-1M --epochs 3 --seeds 10 11 12 13 14 --methods official pemeans_v3 --threads $T
20
+ sync_back
benchmarks/spacev_holdout_rows.npy ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9979f6f5feba5a8ffdde8d64df86c2cd2887ea5d38fc7909d74f85d02f5d52c8
3
+ size 160128
benchmarks/summarize.py ADDED
@@ -0,0 +1,80 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Summarise paper_protocol.py results as paired statistics against official e-means.
2
+
3
+ Usage: python benchmarks/summarize.py results/deep1m_cpu.jsonl [--md OUT.md]
4
+ """
5
+ import argparse, collections, json
6
+ import numpy as np
7
+ from scipy.stats import wilcoxon
8
+
9
+ # Published reference points for DEEP* 1M, k = 16,384 (arXiv:2611.09501 and faiss benchs/e_means):
10
+ PAPER = {
11
+ "deep1b-1M": {
12
+ "Lloyd 25 it., faiss CPU (paper_targets.json, median of 10 seeds)": 0.38921320633218964,
13
+ "Flash-KMeans 25 it., H100 (paper_targets.json, median)": 0.38925347141324373,
14
+ "e-means 3 ep., H100 bf16 (paper Table 1, median of 30 seeds)": 0.3880},
15
+ "fb-ssnpp-1M": {"Flash-KMeans 25 it., H100 (paper_targets.json, median)": 113555.28402890626},
16
+ "spacev-1M": {"Flash-KMeans 25 it., H100 (paper_targets.json, median)": 7228.99001443477},
17
+ }
18
+
19
+
20
+ def boot_ci(d, reps=10000, seed=0):
21
+ r = np.random.default_rng(seed)
22
+ m = r.choice(d, (reps, len(d))).mean(1)
23
+ return np.percentile(m, [2.5, 97.5])
24
+
25
+
26
+ def main():
27
+ ap = argparse.ArgumentParser(); ap.add_argument("jsonl"); ap.add_argument("--md")
28
+ a = ap.parse_args()
29
+ rows = [json.loads(l) for l in open(a.jsonl)]
30
+ by = collections.defaultdict(dict)
31
+ for r in rows:
32
+ by[(r["epochs"], r["method"])][r["seed"]] = r
33
+ out = ["| epochs | method | seeds | test MSE (mean ± sd) | paired change vs official | 95% CI | wins | Wilcoxon p | seconds |",
34
+ "|---|---|---|---|---|---|---|---|---|"]
35
+ for T in sorted({k[0] for k in by}):
36
+ base = by.get((T, "official"), {})
37
+ for m in sorted({k[1] for k in by if k[0] == T}, key=lambda s: (s != "official", s)):
38
+ cur = by[(T, m)]
39
+ seeds = sorted(set(cur) & set(base))
40
+ v = np.array([cur[s]["test_mse"] for s in sorted(cur)])
41
+ sec = np.mean([cur[s]["seconds"] for s in cur])
42
+ if m == "official" or len(seeds) < 2:
43
+ out.append(f"| {T:g} | {m} | {len(cur)} | {v.mean():.5f} ± {v.std():.5f} | — | — | — | — | {sec:.0f} |")
44
+ continue
45
+ d = np.array([100 * (cur[s]["test_mse"] - base[s]["test_mse"]) / base[s]["test_mse"] for s in seeds])
46
+ lo, hi = boot_ci(d)
47
+ p = wilcoxon(d).pvalue if len(d) >= 5 and np.any(d) else float("nan")
48
+ out.append(f"| {T:g} | {m} | {len(seeds)} | {v.mean():.5f} ± {v.std():.5f} | {d.mean():+.3f}% | "
49
+ f"[{lo:+.3f}, {hi:+.3f}] | {int((d < 0).sum())}/{len(d)} | {p:.3g} | {sec:.0f} |")
50
+ # cross-budget: PE-means at a smaller budget vs official at a larger one (equal or less time)
51
+ cross = []
52
+ for (Tp, m) in sorted(by):
53
+ if m == "official":
54
+ continue
55
+ for (To, mo) in sorted(by):
56
+ if mo != "official" or To <= Tp:
57
+ continue
58
+ seeds = sorted(set(by[(Tp, m)]) & set(by[(To, mo)]))
59
+ if len(seeds) < 2:
60
+ continue
61
+ d = np.array([100 * (by[(Tp, m)][s]["test_mse"] - by[(To, mo)][s]["test_mse"]) / by[(To, mo)][s]["test_mse"] for s in seeds])
62
+ t = np.mean([by[(Tp, m)][s]["seconds"] / by[(To, mo)][s]["seconds"] for s in seeds])
63
+ lo, hi = boot_ci(d)
64
+ cross.append(f"| {m} @ {Tp:g} ep. | official @ {To:g} ep. | {len(seeds)} | {d.mean():+.3f}% | [{lo:+.3f}, {hi:+.3f}] | {int((d < 0).sum())}/{len(d)} | {t:.2f}x |")
65
+ if cross:
66
+ out += ["", "Cross-budget comparison (fewer epochs for PE-means):", "",
67
+ "| method | vs | seeds | paired change | 95% CI | wins | time ratio |", "|---|---|---|---|---|---|---|"] + cross
68
+ ds = rows[0].get("dataset", "deep1b-1M")
69
+ if ds in PAPER:
70
+ out += ["", "Published reference points (different hardware and starting rows; for orientation only):", ""]
71
+ for k, v in PAPER[ds].items():
72
+ out.append(f"- {k}: {v:.5f}")
73
+ text = "\n".join(out)
74
+ print(text)
75
+ if a.md:
76
+ open(a.md, "w").write(text + "\n")
77
+
78
+
79
+ if __name__ == "__main__":
80
+ main()
benchmarks/time_breakdown.py ADDED
@@ -0,0 +1,72 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Where the time goes: cProfile of one 3-epoch run of official e-means, PISCES v1.0 and PISCES on
2
+ DEEP* 1M (k = 16,384), grouped into assignment search, statistics, index building and the rest.
3
+
4
+ Usage: python benchmarks/time_breakdown.py DATA/deep1b OUT.json [--seed 1]
5
+ """
6
+ import argparse, cProfile, json, os, pstats, re, sys, time
7
+ import numpy as np
8
+ import faiss
9
+
10
+ ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
11
+ sys.path.insert(0, ROOT); sys.path.insert(0, os.path.join(ROOT, "benchmarks"))
12
+ from faiss.contrib.e_means import NumpyEmeans # noqa: E402
13
+ from paper_protocol import make_variant # noqa: E402
14
+ from aries.pemeans import PISCES, PISCES_V10 # noqa: E402
15
+
16
+ GROUPS = [ # (group, substrings of "file:function" in the profile; first match wins)
17
+ ("assignment search", ["IndexIVF_search", "IndexRefine_search", "IndexRefineFlat_search", "knn", "IndexFlat_search", "IndexFlatCodes_search",
18
+ "bruteForceKnn", "Index_search", "search_preassigned"]),
19
+ ("index build", ["IndexIVF_add", "Clustering_train", "IndexIVFScalarQuantizer_train", "Index_add", "IndexFlatCodes_add",
20
+ "IndexScalarQuantizer", "Index_train"]),
21
+ ("statistics", ["ufunc' objects}", "_fused_stats_nb", "fused_stats", "argsort", "bincount", "einsum", "unique",
22
+ "_apply_statistics", "reduceat", "fill"]),
23
+ ]
24
+
25
+
26
+ def short(key):
27
+ """Drop machine-specific path prefixes from a profile key."""
28
+ key = re.sub(r"^.*/site-packages/", "", key)
29
+ key = re.sub(r"^.*/(aries|benchmarks)/", r"\1/", key)
30
+ return re.sub(r"^/usr/.*?/lib/python3\.\d+/", "stdlib/", key)
31
+
32
+
33
+ def profile(build, x, init):
34
+ pr = cProfile.Profile(); t0 = time.time(); pr.enable()
35
+ build().train(x, init_centroids=init.copy())
36
+ pr.disable(); wall = time.time() - t0
37
+ st = pstats.Stats(pr).stats
38
+ out = {g: 0.0 for g, _ in GROUPS}
39
+ for (fn, line, name), (cc, nc, tt, ct, callers) in st.items():
40
+ key = f"{fn}:{name}"
41
+ for g, pats in GROUPS:
42
+ if any(p in key for p in pats):
43
+ out[g] += tt
44
+ break
45
+ out["other"] = max(0.0, wall - sum(out.values()))
46
+ out["wall"] = wall
47
+ top = sorted(st.items(), key=lambda t: -t[1][2])[:25] # for auditing the grouping
48
+ out["top"] = [[short(f"{fn}:{name}"), round(tt, 3)] for (fn, line, name), (cc, nc, tt, ct, callers) in top]
49
+ return out
50
+
51
+
52
+ def main():
53
+ ap = argparse.ArgumentParser()
54
+ ap.add_argument("data"); ap.add_argument("out"); ap.add_argument("--seed", type=int, default=1)
55
+ ap.add_argument("--epochs", type=float, default=3); ap.add_argument("--threads", type=int, default=4)
56
+ a = ap.parse_args()
57
+ faiss.omp_set_num_threads(a.threads)
58
+ x = np.ascontiguousarray(np.load(os.path.join(a.data, "train_1M.npy")))
59
+ k = 16384
60
+ init = x[np.random.RandomState(a.seed).permutation(len(x))[:k]].copy()
61
+ make_variant(PISCES, len(x))(x.shape[1], 0.02, 0).train(x[:40000], init_centroids=init[:k].copy()) # JIT warm-up
62
+ res = {}
63
+ for name, build in [("official", lambda: NumpyEmeans(x.shape[1], k, nepochs=a.epochs, seed=a.seed, batch_law="throughput")),
64
+ ("pisces_v10", lambda: make_variant(PISCES_V10, len(x))(x.shape[1], a.epochs, a.seed)),
65
+ ("pisces", lambda: make_variant(PISCES, len(x))(x.shape[1], a.epochs, a.seed))]:
66
+ res[name] = profile(build, x, init)
67
+ print(name, {g: round(v, 1) for g, v in res[name].items() if g != "top"}, flush=True)
68
+ json.dump(res, open(a.out, "w"), indent=1)
69
+
70
+
71
+ if __name__ == "__main__":
72
+ main()
docs/ABLATIONS.md ADDED
@@ -0,0 +1,506 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Experiment log
2
+
3
+ Development rule: every design decision is made on **DEEP\* validation rows** (the paper's
4
+ validation half). Test rows are used only for reported results. SimSearchNet++ 1M and SPACEV 1M
5
+ are **held out**: no PE-means variant is run on them until the design is frozen.
6
+
7
+ Proxy setting unless stated: DEEP\* first 250k rows, k = 4,096 (same n/k ≈ 61 as the paper's
8
+ 1M / 16,384), batch law "throughput", seeds 1-5, matched starting rows, 4 threads.
9
+ Numbers are paired % change in validation MSE vs official faiss e-means (negative = better).
10
+ Raw rows: `results/dev/<tag>.jsonl`.
11
+
12
+ ## Iteration 0: PE-means v1 (full scale, test rows)
13
+
14
+ DEEP\* 1M, k = 16,384, seeds 10-14: −0.31% / −0.27% / −0.20% at 1 / 3 / 5 epochs, 15/15 paired
15
+ wins; PE at 3 epochs ≈ official at 5 epochs (−0.01%) in 0.60× the time. See README.
16
+
17
+ ## Iteration 1: which parts of the revival rule matter (`iter1_knobs`)
18
+
19
+ | variant | 1 epoch | 3 epochs | reading |
20
+ |---|---|---|---|
21
+ | PE-means v1 (offset 1, axis score, greedy, tau 0.1, every 5) | −0.294% (5/5) | −0.257% (5/5) | reference |
22
+ | child offset × 0.5 | −0.242% | −0.231% | the Gaussian optimum (× 1) is best, as Lemma 1 predicts |
23
+ | child offset × 1.5 | −0.292% | −0.218% | |
24
+ | donor score N·trace instead of N·λ1 | −0.173% | −0.092% | Lemma 2 confirmed: the top-axis gain is the right score |
25
+ | donor sampled ∝ gain instead of greedy | −0.126% | −0.146% | greedy wins |
26
+ | tau 0.2 | −0.230% | **−0.414%** | revive more aggressively helps at 3 epochs |
27
+ | tau 0.3 | −0.033% | **−0.413%** | ... but hurts at 1 epoch |
28
+ | revival every batch | **−0.537%** | −0.298% | frequent revival helps early |
29
+ | revival every 2 batches | −0.430% | −0.225% | |
30
+
31
+ Decision: keep offset 1, axis score, greedy. Revival should be more aggressive than e-means'
32
+ default once splits are good, but a fixed tau trades early vs late. Hypothesis for iteration 2:
33
+ a gain-vs-cost **swap** rule decides adaptively when to move a centroid.
34
+
35
+ ## Iteration 2: swap moves, consistent closing statistics (`iter2_swap`)
36
+
37
+ Hypotheses. (a) A streaming swap rule (move a live centroid when the best donor's split gain
38
+ (2/π)·N·λ1 exceeds the cluster's deletion cost Σ(d2 − d1), estimated from 2-NN assignment) decides
39
+ adaptively how aggressive revival should be. (b) faiss's closing means are built from last-epoch
40
+ assignment sums; when a donor is split during that epoch, both children inherit the parent's
41
+ pre-split sums. Splitting those statistics consistently (half count each, means at c ± δ) should
42
+ help most when the last epoch is the whole run.
43
+
44
+ | variant | 1 epoch | 3 epochs | time (3 ep.) |
45
+ |---|---|---|---|
46
+ | PE-means v1 | −0.294% | −0.257% | 9.7 s |
47
+ | **split_pass** (consistent closing stats) | **−0.635%** (5/5) | −0.259% | 9.9 s |
48
+ | swap 1% of k per check | −0.223% | −0.311% | 11.4 s |
49
+ | swap 3% | −0.135% | −0.175% | 11.7 s |
50
+ | every batch + swap 1% | −0.092% | +0.036% (1/5) | 11.6 s |
51
+ | every batch + swap + split_pass | −0.772% | −0.253% | 11.7 s |
52
+ | every batch + tau 0.2 | −0.505% | −0.348% | 9.5 s |
53
+ | no revival in the last 5% of steps | −0.295% | −0.257% | 9.7 s |
54
+
55
+ Decision. **Keep split_pass** (it fixes a real inconsistency; doubles the gain at 1 epoch, never
56
+ hurts, no cost). **Drop swaps**: inconsistent, +18% time, and harmful combined with frequent
57
+ revival (negative result kept). Iteration 3 combines split_pass with faster revival and re-tunes
58
+ the e-means scaling laws (LawFit) for the new revival rule.
59
+
60
+ ## Iteration 3: combinations and LawFit (`iter3_combo_lawfit`)
61
+
62
+ `sp` = split_pass. `alpha × s` / `batch × s` rescale the e-means fitted step / batch laws.
63
+
64
+ | variant | 1 epoch | 3 epochs |
65
+ |---|---|---|
66
+ | PE-means v1 | −0.294% | −0.257% |
67
+ | sp | −0.635% | −0.259% |
68
+ | sp + every batch | −0.825% | −0.298% |
69
+ | sp + tau 0.2 | −0.742% | **−0.419%** |
70
+ | sp + every batch + tau 0.2 | **−0.960%** | −0.348% |
71
+ | sp + every 2 + tau 0.15 | −0.801% | −0.269% |
72
+ | sp + alpha × 0.7 | −0.540% | −0.155% |
73
+ | sp + alpha × 1.4 | −0.729% | −0.342% |
74
+ | sp + batch × 0.5 | −0.805% | −0.219% |
75
+ | sp + batch × 2 | −0.582% | −0.166% |
76
+ | official + batch × 0.5 | −0.255% | −0.095% |
77
+
78
+ Reading. With better revival, a larger EMA step helps at both budgets (the paper's step law
79
+ was fitted with faiss revival), and revival should be more aggressive (tau 0.2). The fitted
80
+ batch law is near-optimal at 3 epochs for both methods. Every-batch revival helps at 1 epoch
81
+ only. Next: combine tau 0.2 with a larger step, and test a fresher closing window.
82
+
83
+ ## Iteration 4: combinations and closing window (`iter4_combine`)
84
+
85
+ | variant | 1 epoch | 3 epochs |
86
+ |---|---|---|
87
+ | sp + tau 0.2 | −0.742% | −0.419% |
88
+ | sp + tau 0.25 | −0.724% | −0.361% |
89
+ | **sp + tau 0.2 + alpha × 1.4** (PE-means v2) | −0.810% (5/5) | −0.414% (5/5) |
90
+ | sp + tau 0.2 + alpha × 2 | −0.704% | −0.184% |
91
+ | sp + every batch + tau 0.2 + alpha × 1.4 | **−1.014%** | −0.308% |
92
+ | sp + closing window 0.5 | +0.042% | +0.696% |
93
+ | sp + closing window 0.75 | −0.438% | +0.035% |
94
+ | official + alpha × 1.4 | +0.068% | +0.038% |
95
+
96
+ Reading. A fresher closing window loses (more rows beat fresher rows): dropped. The larger step
97
+ helps only PE-means, not official e-means, so the change is tied to the better revival rather
98
+ than a flaw in the paper's step law. Finalists for full scale (DEEP\\* 1M, k = 16,384, validation
99
+ rows, fresh seeds 20-22): v2 = sp + tau 0.2 + alpha × 1.4; sp + tau 0.2; and the 1-epoch
100
+ specialist with every-batch revival.
101
+
102
+ ## Iteration 5: full-scale validation (DEEP\* 1M, k = 16,384, validation rows)
103
+
104
+ Variants: `pemeans` = v1; `pe_sp_tau0.2` = sp + tau 0.2; `pe_v2` = sp + tau 0.2 + alpha × 1.4;
105
+ `pe_e1_v2` = v2 + revival every batch. Two seed sets: 20-22 (`results/deep1m_fullscale_val.jsonl`)
106
+ and 1-5 (running; the same variants are named F1/F2/F3 there). Report seeds 10-19 and the test
107
+ rows stay unused until the design is frozen.
108
+
109
+ Seeds 20-22 (paired change vs official e-means):
110
+
111
+ | variant | 1 epoch | 3 epochs | seconds (3 ep.) |
112
+ |---|---|---|---|
113
+ | v1 | −0.248% (3/3) | −0.151% (3/3) | 140 |
114
+ | sp + tau 0.2 | −0.700% (3/3) | −0.196% (3/3) | 152 |
115
+ | **v2** | −0.701% (3/3) | **−0.264%** (3/3) | 144 |
116
+ | v2 + every batch | **−0.933%** (3/3) | −0.256% (3/3) | 168 |
117
+
118
+ Reading. v2 beats v1 at both budgets at no extra cost; the every-batch variant only pays off at one
119
+ epoch and costs 17% more time at three. Full-scale gains at 3 epochs are smaller than on the proxy
120
+ (−0.26% vs −0.41%).
121
+
122
+ Seeds 1-5 (same variants, named F1 = sp + tau 0.2, F2 = v2, F3 = v2 + every batch):
123
+
124
+ | variant | 1 epoch | 3 epochs | seconds (3 ep.) |
125
+ |---|---|---|---|
126
+ | v1 | −0.162% (3/5) | −0.218% (5/5) | 146 |
127
+ | F1 | −0.573% (5/5) | −0.166% (5/5) | 143 |
128
+ | **F2 = v2** | −0.618% (5/5) | −0.214% (4/5) | 145 |
129
+ | F3 | **−0.833%** (5/5) | −0.162% (4/5) | 147 |
130
+
131
+ Pooled over 8 seeds: v2 −0.65% / −0.23% at 1 / 3 epochs, v1 −0.19% / −0.19%.
132
+
133
+ **Decision: freeze PE-means v2** (`aries.pemeans.PE_V2`: principal-axis gain-ranked revival,
134
+ split_pass, tau 0.2, step law × 1.4). It dominates v1 at equal cost. Confirmation
135
+ (`benchmarks/run_final_v2.sh`): time-to-quality on validation rows, final DEEP\* 1M test rows with
136
+ report seeds 10-19, and the held-out SimSearchNet++ / SPACEV 1M datasets.
137
+
138
+ ## Iteration 6: new engineering and schedule ideas (`iter6_v3_candidates`, proxy)
139
+
140
+ | variant (on top of v2) | 1 epoch | 3 epochs | time (3 ep.) |
141
+ |---|---|---|---|
142
+ | v2 | −0.810% | −0.414% | 10.4 s |
143
+ | child mass 0.35 / 0.25 of donor (vs 0.5) | −0.807% / −0.813% | −0.279% / −0.344% | |
144
+ | tau cosine 0.3 → 0.1 | −0.703% | −0.311% | |
145
+ | **revival every batch in the first 30% of steps** (v3) | **−1.005%** | −0.405% | 10.0 s |
146
+ | **approximate assignment**: IVF over centroids (64 cells, 8 probes), exact last epoch | −0.810% | −0.317% | **7.6 s** (official 8.6 s) |
147
+
148
+ Child mass and tau schedules: negative, dropped. Early every-batch revival (v3) helps at 1 epoch.
149
+ Approximate assignment is the first engineering win: faster than official e-means while better.
150
+
151
+ ## Iteration 7: approximate assignment across budgets (`iter7_approx_budget`, proxy)
152
+
153
+ | run | val MSE | time |
154
+ |---|---|---|
155
+ | official 5 epochs | 0.42937 | 13.3 s |
156
+ | v2 3 epochs | 0.42867 | 9.5 s |
157
+ | v3 + approx 4 epochs | **0.42831** | 11.0 s |
158
+ | v3 + approx 5 epochs | 0.42837 | 12.6 s |
159
+
160
+ At this proxy scale (k = 4,096) the assignment is a smaller share of cost, so approximate
161
+ assignment helps but does not dominate; at k = 16,384 it should matter more (to be timed).
162
+
163
+ ## Iteration 8: graph-pruned assignment with label memory (`iter8_graph`, proxy) — negative
164
+
165
+ | variant | 3 epochs | 5 epochs | time (5 ep.) |
166
+ |---|---|---|---|
167
+ | v2 | −0.414% | −0.344% | 15.6 s |
168
+ | v2 + graph (32 neighbours) | +0.015% | **+0.324%** | 13.0 s |
169
+ | v2 + graph (64 neighbours) | −0.035% | −0.018% | 17.0 s |
170
+
171
+ Diagnosis: revival re-uses cluster indices, so a row's remembered label can point to a centroid
172
+ that has been moved elsewhere; its candidate set is then the wrong neighbourhood. Fix for
173
+ iteration 9: when an index is re-used, remap rows remembering it to the donor; and replace the
174
+ NumPy gather/einsum candidate kernel (slower than BLAS) with a compiled (Numba) kernel.
175
+
176
+ ## Iteration 9: fixing graph-pruned assignment (`iter9_graph_fixed`, `iter9b_graph_verified`) — negative
177
+
178
+ Fixes: re-used cluster indices are remapped to the donor in every row's label memory; candidate
179
+ distances run in a compiled Numba kernel (72x faster than faiss's full search for one full-scale
180
+ batch, 0.020 s vs 1.44 s). Still worse: 64 neighbours −0.10% / +0.01% at 3 / 5 epochs (v2:
181
+ −0.41% / −0.34%), though 35% faster. Diagnostic: one epoch after a row was last seen, **21% of rows
182
+ have a nearer centroid outside the candidate set** (assigned distance 8.6% too large): centroids
183
+ move too much between visits under e-means' early large steps and revivals.
184
+
185
+ Self-verifying variant: each batch checks a random 1% of rows exactly and falls back to exact
186
+ assignment when the mismatch exceeds 1% or 3%. Quality returns to v2's exactly, but the check
187
+ fails on almost every batch, so it is only overhead (20 s vs 16 s at 5 epochs). Conclusion: label
188
+ memory does not pay in this regime. Kernel and verification code are kept (`aries/_kernels.py`).
189
+
190
+ ## Iteration 10: approximate assignment at full scale (`iter10_full_scale_approx`)
191
+
192
+ DEEP\* 1M, k = 16,384, validation rows, seeds 1-3. v3 = v2 + revival every batch in the first
193
+ 30% of steps. `apx L×P` = assignment through an IVF over the centroids with L coarse cells and P
194
+ probes in every epoch but the last (exact); `km` = coarse cells from a 20-iteration k-means on the
195
+ centroids, otherwise a random subset of centroids.
196
+
197
+ | variant | 3 epochs | 4 epochs | time 3 / 4 ep. |
198
+ |---|---|---|---|
199
+ | official e-means | 0.38600 | 0.38611 | 115 / 147 s |
200
+ | v2 (exact) | −0.112% (2/3) | −0.216% (3/3) | 130 / 170 s |
201
+ | v3 + apx 128×16 | −0.232% (3/3) | **−0.353%** (3/3) | **76 / 93 s** |
202
+ | v3 + apx 128×16 km | **−0.305%** (3/3) | −0.277% (3/3) | **75 / 93 s** |
203
+ | v3 + apx 128×32 km | −0.119% (2/3) | −0.201% (3/3) | 96 / 121 s |
204
+
205
+ Reading. At full scale the assignment dominates the cost, so approximate assignment makes PE-means
206
+ **35-37% faster than official e-means at equal epochs while winning every seed**. More probes
207
+ (more exact) is consistently *worse*: the approximation appears to help, like annealing noise.
208
+ Iteration 11 isolates this (v3 exact vs 8 / 16 probes, 5 seeds).
209
+
210
+ ## Iteration 11: is the approximation itself helping? (`iter10_11_full_scale_approx`)
211
+
212
+ DEEP\* 1M, k = 16,384, validation rows, 5 seeds (32 probes: 3 seeds), 3 epochs.
213
+
214
+ | variant | vs official | wins | time |
215
+ |---|---|---|---|
216
+ | official e-means | 0.38634 | — | 113 s |
217
+ | v3, exact assignment | −0.173% | 4/5 | 129 s |
218
+ | v3 + apx 128×8 km | −0.209% | 4/5 | 68 s |
219
+ | **v3 + apx 128×16 km** | **−0.340%** | **5/5** | **75 s** |
220
+ | v3 + apx 128×16 (random cells) | −0.319% | 5/5 | 76 s |
221
+ | v3 + apx 128×32 km | −0.119% | 2/3 | 96 s |
222
+
223
+ Reading. There is a clear sweet spot: exact assignment is *worse* than moderately approximate
224
+ assignment (16 probes), and 8 probes is too noisy. Approximate early-epoch assignment acts like
225
+ annealing noise and also removes 42% of the time. Rule that transfers across k: √k coarse cells
226
+ from a short k-means on the centroids, √k / 8 probes, exact final epoch.
227
+
228
+ **Decision: freeze PE-means v3** (`aries.pemeans.PE_V3`). Confirmation (`run_final_v3.sh`):
229
+ time-to-quality (val), DEEP\* 1M test rows seeds 10-19, held-out SimSearchNet++ / SPACEV 1M.
230
+
231
+ ## Confirmation of the frozen PE-means v3 (`run_final_v3.sh`)
232
+
233
+ **Time to quality** (validation rows, seeds 1-5; `results/deep1m_v3_time_to_quality_val_summary.md`):
234
+
235
+ | run | val MSE | vs official at 5 epochs | time ratio |
236
+ |---|---|---|---|
237
+ | official 3 epochs | 0.38634 | +0.198% | 0.59× |
238
+ | official 5 epochs | 0.38558 | — | 1.00× (222 s) |
239
+ | v3 2 epochs | 0.38625 | +0.174% | 0.27× |
240
+ | **v3 3 epochs** | **0.38503** | **−0.143%** (4/5) | **0.33×** |
241
+ | v3 4 epochs | 0.38494 | −0.164% (5/5) | 0.42× |
242
+
243
+ **DEEP\* 1M test rows**, report seeds 10-19 (`results/deep1m_cpu_summary.md`):
244
+
245
+ | epochs | v3 vs official | 95% CI | wins | Wilcoxon p | time |
246
+ |---|---|---|---|---|---|
247
+ | 1 | −0.963% | [−1.042, −0.870] | 10/10 | 0.002 | 43 vs 42 s |
248
+ | 3 | −0.279% | [−0.355, −0.205] | 10/10 | 0.002 | 66 vs 114 s |
249
+ | 5 | −0.215% | [−0.270, −0.152] | 9/10 | 0.004 | 99 vs 199 s |
250
+ | v3 3 ep. vs official 5 ep. | −0.111% | [−0.184, −0.032] | 8/10 | | 0.34× |
251
+
252
+ **Held-out datasets** (never used in development; 3 epochs, seeds 10-14):
253
+
254
+ | dataset | v3 vs official | 95% CI | wins | time |
255
+ |---|---|---|---|---|
256
+ | SPACEV 1M | −0.444% | [−0.512, −0.380] | 5/5 | 68 vs 103 s |
257
+ | SimSearchNet++ 1M | −0.032% | [−0.038, −0.025] | 5/5 | 178 vs 252 s |
258
+
259
+ Verdict for algorithm 1: PE-means v3 is better than official e-means at every budget on every
260
+ dataset tested, by up to ~1% at one epoch, and reaches e-means' 5-epoch quality in about a third of
261
+ the time. The quality margin is small on SimSearchNet++; the speed margin holds everywhere.
262
+
263
+ ## Iteration 12: fairness ablation and approximate last epoch (full scale, val, seeds 1-3, 3 epochs)
264
+
265
+ | variant | vs official | wins | time |
266
+ |---|---|---|---|
267
+ | v3 (exact last epoch) | −0.305% | 3/3 | 75 s |
268
+ | **official e-means + the same approximate assignment** | **+0.128%** | 1/3 | 66 s |
269
+ | official e-means + approximate assignment incl. last epoch (64 probes) | +0.128% | 1/3 | 73 s |
270
+ | **v3 + approximate last epoch, 32 probes** | **−0.304%** | 3/3 | **53 s** |
271
+ | v3 + approximate last epoch, 64 probes | −0.305% | 3/3 | 73 s |
272
+
273
+ Reading. Approximate assignment does **not** improve e-means by itself (it makes it 0.13% worse);
274
+ its benefit appears only together with principal-axis revival. Against e-means given the same
275
+ speed-up, PISCES is ~0.43% better. An approximate last epoch with √k/4 probes is a free 30% speed-up.
276
+
277
+ ## Iteration 13: coarse-to-fine growth (`iter13_growth`, proxy) — negative
278
+
279
+ Start with k/2^g centroids, double g times by splitting every cluster along its principal axis
280
+ (0.5 epoch in total), then train at k.
281
+
282
+ | variant | 1 epoch | 3 epochs |
283
+ |---|---|---|
284
+ | PISCES, no growth | −1.005% | −0.425% |
285
+ | growth g = 2 / 3 / 4 | +1.011% / +1.043% / +1.022% | −0.133% / −0.166% / −0.148% |
286
+
287
+ Splitting every cluster regardless of its gain, from under-trained coarse codebooks, loses to
288
+ random-row starts plus gain-ranked revival. Dropped.
289
+
290
+ ## Iteration 14: cell-batched GEMM assignment (kernel benchmark, one full-scale batch)
291
+
292
+ Profile of a 3-epoch PISCES run: 78% of time in faiss's IVF search (scalar per-query scans).
293
+ Two BLAS reformulations: (a) query-centric candidate sets, one product per cell: identical results,
294
+ same speed (0.71 vs 0.70 s); (b) cell-centric candidate sets shared by all rows of a cell: 1.6x
295
+ faster at 16-32 probes but recall drops (0.874-0.933 vs 0.983-0.996), and iteration 11 showed lower
296
+ recall costs quality. Kept faiss IVF; kernels kept in `aries/_kernels.py`.
297
+
298
+ **Decision: freeze PISCES** = v3 + approximate last epoch (√k/4 probes): `aries.pemeans.PISCES`.
299
+
300
+ ## Final confirmation: PISCES (`run_final_pisces.sh`)
301
+
302
+ PISCES = v3 + approximate last epoch (√k/4 probes). All numbers in `paper/macros.tex`, generated by
303
+ `paper/make_data.py` from the raw rows.
304
+
305
+ | evaluation | PISCES vs official e-means | wins | time ratio |
306
+ |---|---|---|---|
307
+ | DEEP\* 1M test, 1 epoch | −0.88% | 10/10 | 0.68× |
308
+ | DEEP\* 1M test, 3 epochs | −0.28% | 10/10 | 0.49× |
309
+ | DEEP\* 1M test, 5 epochs | −0.22% | 9/10 | 0.43× |
310
+ | PISCES 3 ep. vs official 5 ep. (test) | −0.11% | 8/10 | 0.28× |
311
+ | PISCES 3 / 4 ep. vs official 5 ep. (val) | −0.14% / −0.16% | 4/5, 5/5 | 0.24× / 0.33× |
312
+ | SPACEV 1M held-out, 3 epochs | −0.44% | 5/5 | 0.60× |
313
+ | SimSearchNet++ 1M held-out, 3 epochs | −0.045% | 5/5 | 0.70× |
314
+ | e-means + same sparse assignment, DEEP test / SPACEV / SimSearchNet++ | +0.07% / +0.11% / +0.03% | 3/10, 0/5, 0/5 | 0.64× / 0.72× / 0.80× |
315
+ | PISCES vs e-means + same sparse assignment (DEEP test, 3 ep.) | −0.35% | 10/10 | |
316
+
317
+ Per-seed: PISCES is better than official e-means on 39 of 40 runs.
318
+
319
+ ## Iterations 15-19: after freezing PISCES, six new ideas (proxy: DEEP 250k, k = 4,096, val rows, seeds 1-5)
320
+
321
+ Profile first: 76% of a 3-epoch PISCES run is faiss's IVF search, 5% the axis statistics
322
+ (reduceat), 5% the e-means `np.add.at` sums. Every candidate is a switch in `PEmeans`.
323
+
324
+ **Iteration 15** (`iter15_candidates`): heavy-ball momentum on centroid moves, fewer probes early,
325
+ closing Lloyd polish.
326
+
327
+ | variant | 1 epoch | 3 epochs | time (3 ep.) |
328
+ |---|---|---|---|
329
+ | PISCES | −0.943% | −0.425% | 6.5 s |
330
+ | momentum 0.3 / 0.6 (first 70% of steps) | −0.837% / −0.024% | −0.141% / +0.220% | 6.0 s |
331
+ | probes 4→16 / 8→16 over the early epochs | (same: 1 epoch is all last epoch) | −0.350% / −0.329% | 6.2 s |
332
+ | **+ closing Lloyd polish** (exact or 16 probes) | **−1.362%** | **−0.471%** | 8.1 s |
333
+
334
+ Momentum and probe schedules: negative. Polish: one Lloyd step after training (re-assign every row to
335
+ the closing means, recompute the means). It helps most at 1 epoch, where the closing means include
336
+ rows assigned against the starting rows.
337
+
338
+ **Iteration 16** (`iter16_polish_short`, `iter16_pisces_long`): is the polish better than spending
339
+ the same time on more epochs? Absolute val MSE / seconds:
340
+
341
+ | budget | PISCES | PISCES + polish (8 probes) |
342
+ |---|---|---|
343
+ | 1 epoch | 0.43256 / 2.3 s | **0.43066 / 3.4 s** |
344
+ | 1.5 epochs | 0.43107 / 3.8 s | |
345
+ | 2 epochs | 0.42940 / 4.7 s | 0.42899 / 5.7 s |
346
+ | 2.5 epochs | 0.42907 / 5.3 s | |
347
+ | 3 epochs | 0.42862 / 6.5 s | 0.42838 / 7.6 s |
348
+
349
+ At 1 epoch the polish beats 50% more training (better and faster); at 2-3 epochs it is on the
350
+ time-quality front of plain PISCES (no worse, not clearly better). A second, label-memory polish step
351
+ (each row checked only against the 32 centroids nearest its previous centroid, Numba kernel) adds
352
+ −0.1% at 1 epoch; the label memory is too stale for the *first* step (73% of labels survive the
353
+ closing means at 1 epoch, 9% of true nearest centroids fall outside the candidate set).
354
+
355
+ **Iteration 17** (`iter17_hnsw`): HNSW over the centroids instead of IVF. Microbenchmark on one
356
+ full-scale batch (65,536 rows, k = 16,384): IVF 16 probes 1.0 s at recall 0.996; HNSW (M 16, ef 32)
357
+ 0.18 s build + 0.34 s search at recall 0.995; vectors can be overwritten in place between rebuilds.
358
+ But on the proxy it is slower (k = 4,096) and **worse**: −0.31% to −0.36% at 3 epochs vs −0.43%, and
359
+ higher recall is worse still. Negative.
360
+
361
+ **Iteration 18** (`iter18_fp16_polish`): fp16 IVF lists (same recall, ~30% faster scans in the
362
+ microbenchmark): −0.29% vs −0.43% at 3 epochs. Negative, like HNSW: the particular errors of float32 IVF
363
+ with √k/8 probes are part of why PISCES works (iteration 11), and other approximations do not
364
+ reproduce them. A cheaper last epoch (√k/8 probes) to pay for the polish: −0.32%, negative.
365
+
366
+ **Iteration 19** (`iter19_anneal`): explicit annealed assignment, a fraction ε(t) of rows (decaying
367
+ linearly to 0 at 70% or 30% of training) sent to the runner-up centroid, on top of the polish.
368
+ ε0 = 0.01 / 0.03 / 0.10: −0.461% / −0.397% / −0.361% at 3 epochs vs −0.481% without. Negative.
369
+
370
+ **Decision:** carry PISCES + polish (√k/8 probes) to full-scale validation; drop the rest.
371
+
372
+ ## Iteration 20: polish at full scale; cheaper polish; global split-merge audit
373
+
374
+ **Full-scale polish** (DEEP\* 1M, k = 16,384, val rows, seeds 1-5;
375
+ `results/deep1m_polish_time_to_quality_val.jsonl`). PISCES + one closing Lloyd step (√k/8 probes)
376
+ vs PISCES at the same number of epochs: −0.37% / −0.11% / −0.05% / −0.02% at 1 / 2 / 3 / 4 epochs,
377
+ 5/5 seeds each. A timing check in the same session (`results/dev/iter20_timing_check.jsonl`) shows
378
+ the machine ran ~20% slower than when the PISCES / official time-to-quality rows were recorded
379
+ (official 3 ep. 155 s vs 131 s, PISCES 65 s vs 54 s; the PISCES / official ratio is stable,
380
+ 0.42 vs 0.41), so absolute seconds are only compared within a session. Corrected for that, the
381
+ polished runs sit on PISCES' own time-quality curve: the polish buys about what the same time spent
382
+ on more epochs buys.
383
+
384
+ **Cheaper polish** (label memory, full scale, seed 1): after 3 epochs 96.5% of the training labels
385
+ are still the exact nearest centroid, and the nearest centroid of the polished codebook is among the
386
+ 32 centroids nearest the row's previous centroid for 98.9% of rows. Three label-memory Lloyd steps
387
+ (Numba candidate kernel, 32 candidates) take 5.3 s and give −0.047%; at 1 epoch they give −0.33% in
388
+ 5.7 s (labels are staler there: 72% survive). Again on the time-quality curve, not beyond it.
389
+
390
+ **Global split-merge audit** (proxy, 3 seeds, `swap_proto`): after PISCES, with exact 2-NN deletion
391
+ costs and power-iteration split gains, only 8-10 of 4,096 clusters admit a split-merge move whose
392
+ predicted gain exceeds its cost; applying them changes val MSE by −0.015% beyond the same number
393
+ of Lloyd steps. PISCES' revival already leaves a codebook that is close to swap-optimal (the
394
+ mechanism analysis compares this count with official e-means at full scale).
395
+
396
+ **Decision:** PISCES is unchanged. The closing polish is kept as an option (`polish=1`) for users who
397
+ compare at a fixed number of passes; it is not part of the method.
398
+
399
+ ## Iteration 21: distribution-free splits, batch-size schedule, child mass (`iter21_halfmean_batch`, proxy)
400
+
401
+ Lemma 1 assumes Gaussian clusters. Two distribution-free alternatives, tracked online at the same O(Bd)
402
+ cost: (a) the exact half-mean split, children at c ± E[sign(p) e] and donor score N‖E[sign(p) e]‖²
403
+ (one exact 2-means step of the hyperplane split, for any symmetric distribution); (b) the empirical
404
+ axis half-mean, children at c ± E|p|·u and score N(E|p|)².
405
+
406
+ | variant | 1 epoch | 3 epochs |
407
+ |---|---|---|
408
+ | PISCES (Gaussian closed form √(2λ₁/π), score (2/π)Nλ₁) | −0.943% | −0.425% |
409
+ | half-mean split + half-mean score | −0.288% | −0.120% |
410
+ | E\|p\| split + E\|p\| score | −0.877% | −0.417% |
411
+ | half-mean split, Gaussian score | −0.688% | −0.360% |
412
+ | Gaussian split, E\|p\| score | −0.874% | −0.334% |
413
+ | batch size 0.5B → 1.5B (linear in rows, step ∝ B^0.9) | −0.863% | −0.434% |
414
+ | batch size 0.25B → 1.75B | −0.877% | −0.324% |
415
+ | children keep 70% of the donor's mass | −1.017% | −0.417% |
416
+
417
+ The Gaussian closed form wins: it uses only the second moment along the axis, which the EMA power
418
+ iteration estimates with low variance, while the first-moment estimates mix sign flips of a still-moving
419
+ axis. Batch schedules and child mass: no reliable gain. PISCES' algorithm stays as it is.
420
+
421
+ ## Iteration 22: engineering — fused statistics kernel and 8-bit scans with exact re-rank
422
+
423
+ Profile: 76% of a run is faiss's IVF search, ~22% is NumPy bookkeeping (`np.add.at` sums,
424
+ `reduceat` axis statistics, residual gathers, sorts).
425
+
426
+ * **Fused kernel** (`fused=True`): one parallel Numba pass over the batch, rows sorted by label,
427
+ computes e-means' sums and counts and PISCES' Σ(e·u)e and Σ(e·u)² together. Same quantities (unit
428
+ test against NumPy); only the summation order differs. Full scale: identical splits and val MSE,
429
+ 66.0 → 49.6 s and 61.9 → 52.4 s (seeds 1-2).
430
+ * **Rejected engineering** (same session, one 65,536-row batch, k = 16,384): sorting queries by coarse
431
+ cell (identical results, no speed-up: the scan is compute-bound, not cache-bound); GEMM per IVF list
432
+ with a thread pool (identical results, 0.94-1.16 s vs faiss 0.82-0.97 s). The machine runs dense
433
+ distance computations at 25-30 GFLOPS and faiss's scan already reaches that.
434
+ * **8-bit scans with exact re-rank** (`codec="sq8r"`): IVF lists stored as 8-bit scalar-quantized
435
+ codes, the best 4 candidates re-ranked with exact float32 distances. Agreement with the float32 IVF
436
+ result: 100.000% on the benchmark batch (99.988% with 2 candidates); search 0.68 → 0.53 s at 16 probes.
437
+
438
+ Full scale, val rows, 3 epochs, 5 seeds, same session (`results/dev/iter22_fused_sq8r_full_val.jsonl`):
439
+
440
+ | configuration | val MSE | vs official 5 ep. | time |
441
+ |---|---|---|---|
442
+ | PISCES v1.0 | 0.38502 | −0.145% | ~64 s (same-session timing check) |
443
+ | + fused kernel | 0.38502 | −0.144% | 46.8 s |
444
+ | + fused kernel + 8-bit scan / exact re-rank | 0.38514 | −0.114% | 40.1 s |
445
+
446
+ With 8-bit scans, 3 of 5 seeds are identical to float32 (same splits, same MSE); on 2 a rare
447
+ tie-break difference sends training down a different trajectory (+0.137%, +0.012%), which is
448
+ seed-level noise. **Decision:** final PISCES = PISCES v1.0 + fused kernel + 8-bit scans with exact
449
+ re-rank (`aries.pemeans.PISCES`; the previous configuration is `PISCES_V10`). Same algorithm, about
450
+ 1.6× faster. Final confirmation: `benchmarks/run_final.sh` → `results/final/`.
451
+
452
+ ## Iteration 23: the 8-bit scan needs d % 8 == 0 (found on held-out SPACEV)
453
+
454
+ The first held-out SPACEV run (`results/final/superseded/spacev1m_test_unpadded_scan.jsonl`) had the
455
+ best quality gain of all three datasets (−0.46%, 5/5 seeds), but PISCES took ~350 s against ~130 s for
456
+ e-means, while e-means with the same IVF (float32 lists) took ~70 s. Cause: faiss' SIMD 8-bit
457
+ scalar-quantizer scan is only used when d is a multiple of 8. SPACEV has d = 100 (DEEP 96, SSNPP 256),
458
+ so the scan fell back to a scalar loop. Micro-benchmark, 1 thread, 16,384 centroids, 8,192 rows,
459
+ 16 of 128 probes: d = 100 → 3.10 s, d = 104 → 0.36 s, d = 96 → 0.24 s.
460
+
461
+ **Fix (engineering, no algorithmic change):** zero-pad centroids and rows to the next multiple of 8 for
462
+ the scan and the re-rank. Padded coordinates contribute exactly 0 and the per-dimension 8-bit codes
463
+ of the real coordinates are unchanged; the SIMD kernel sums in a different order, so float rounding can
464
+ flip a rare near-tie (unit test `test_sq8r_assignment_any_dimension`: >99.9% of labels equal to float32
465
+ IVF at d = 96 and d = 100). As in iteration 22, one flipped tie sends a run down a different trajectory,
466
+ so SPACEV MSEs are not bit-identical to the first run (seed 10: 7164.42 → 7162.05) but within seed
467
+ noise. SPACEV was re-run with the fix in the same session as the other follow-ups
468
+ (`results/final/spacev1m_test.jsonl`); seed 10 time: 457 s → 45 s (e-means 116 s).
469
+
470
+ ### Iteration 23b: sparse assignment does not pay at small k (`iter23_small_k_val`)
471
+
472
+ The scaling study (`results/final/scaling_k1024_val.jsonl`) showed PISCES 1.80× slower than e-means at
473
+ k = 1,024. Same session, DEEP* 1M, k = 1,024, val rows, 3 epochs, seeds 1-3, paired vs official:
474
+
475
+ | configuration | change vs e-means | seeds won | time |
476
+ |---|---|---|---|
477
+ | PISCES (sparse assignment, √k = 32 cells, 4 probes) | −0.059% | 2/3 | 1.93× |
478
+ | PISCES v1.0 | −0.060% | 2/3 | 2.68× |
479
+ | PISCES with exact assignment (fused kernel kept) | −0.078% | 2/3 | 0.83× |
480
+
481
+ At k = 1,024 the exact GEMM is cheap (official takes ~9 s) and building the IVF index every batch costs
482
+ more than it saves. **Decision:** the `"auto"` sparse-assignment rule now applies only from
483
+ `APPROX_MIN_K = 4,096` on; below it PISCES uses exact assignment. This changes nothing at k ≥ 4,096 (every
484
+ test-row, held-out and analysis result). The threshold was chosen from these validation runs, so the
485
+ k = 1,024 result with the rule is not a held-out number.
486
+
487
+ ## Iteration 24: early revival needs informative counts (`iter24_k65536_regime_val`)
488
+
489
+ At k = 65,536 and n = 1M the quality gain nearly vanished (−0.04%, 1/3 seeds; `results/final/scaling_k65536_val.jsonl`)
490
+ while PISCES re-seeded 114k clusters against 48k for e-means. With B = 65,536 a cluster sees about one row per
491
+ batch, a Poisson count with relative standard deviation 1. Same data, k = 65,536, val rows, 3 epochs,
492
+ seeds 1-3, paired vs the official runs of the scaling study:
493
+
494
+ | variant | per seed | mean | revivals |
495
+ |---|---|---|---|
496
+ | PISCES | −0.302 / +0.099 / +0.074 | −0.043% | 114,496 |
497
+ | τ = 0.1 | −0.118 / +0.026 / +0.132 | +0.013% | 90,002 |
498
+ | no early revival | −0.369 / −0.056 / −0.143 | **−0.190%** (3/3) | 79,950 |
499
+ | τ = 0.1, no early revival | −0.305 / −0.021 / +0.213 | −0.038% | 65,210 |
500
+
501
+ Fewer revivals alone is not the answer (τ = 0.1 with no early revival revives least and is not better);
502
+ what hurts is the *every-batch* phase acting on one-row counts. **Decision:** early every-batch revival
503
+ applies only when the batch holds at least `EARLY_MIN_ROWS = 2` expected rows per cluster (B ≥ 2k); otherwise
504
+ PISCES revives every 5 batches like e-means. Every reported run at k ≤ 16,384 has B/k ≥ 2 (the 1-epoch runs
505
+ sit exactly at 2), so the rule changes only k = 65,536. It was chosen on these validation runs, so the
506
+ k = 65,536 number with the rule is not a held-out result.
docs/DERIVATIONS.md ADDED
@@ -0,0 +1,113 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # PE-means: principal-axis, gain-ranked revival for e-means
2
+
3
+ Technical note. Code: [`aries/pemeans.py`](../aries/pemeans.py). Benchmark:
4
+ [`benchmarks/paper_protocol.py`](../benchmarks/paper_protocol.py).
5
+
6
+ ## 1. Where the gain can come from
7
+
8
+ e-means (Bessa et al., arXiv:2611.09501) streams mini-batches, keeps an exponential moving average
9
+ of each cluster's mass `N_j` and centroid `c_j`, sets batch size and step from fitted scaling laws,
10
+ and every 5 batches revives clusters whose mass falls below `0.1 · mean(N)`. Revival is the
11
+ component with the largest effect in the paper's ablation (Table 1: +2.97% train MSE, +2.24% test
12
+ MSE without it). It is also busy: on DEEP\* 1M with k = 16,384 we count about 9,000 revivals in the
13
+ first epoch and 6,900 over three epochs.
14
+
15
+ Each revival in faiss picks a donor with probability proportional to `N_j` and places donor and
16
+ child at `c_j · (1 ∓ 1e-4 · s)`, with `s` an alternating ±1 vector. The two copies start on top of
17
+ each other; only later assignments separate them, and the donor may be a heavy but tight cluster
18
+ where a split gains little.
19
+
20
+ ## 2. Two closed-form answers for a Gaussian cluster
21
+
22
+ Let a cluster be `x ~ N(c, Σ)` with mass `N`, and split it by the hyperplane through `c` orthogonal
23
+ to a unit vector `u`, with `σ_u² = uᵀ Σ u`.
24
+
25
+ **Lemma 1 (child positions).** The two halves have means `c ± sqrt(2/π) · σ_u · u` (the mean of a
26
+ half-normal is `σ sqrt(2/π)`).
27
+
28
+ **Lemma 2 (distortion removed).** Along `u` each half keeps variance `σ_u² (1 − 2/π)`; orthogonal
29
+ directions are unchanged. The split therefore lowers total distortion from `N tr(Σ)` to
30
+ `N (tr(Σ) − (2/π) σ_u²)`, a reduction of `(2/π) · N · σ_u²`. This is maximal at `u = u_1`, the top
31
+ eigenvector, where it equals `(2/π) · N · λ_1`.
32
+
33
+ Consequences for revival: put the children at `c ± sqrt(2/π) sqrt(λ_1) u_1`, and give the free
34
+ centroid to the cluster with the largest `N · λ_1`, not the largest `N`.
35
+
36
+ The child placement is the one G-means uses (Hamerly and Elkan, NeurIPS 2003), and principal
37
+ direction splitting goes back to PDDP (Boley, 1998) and LBG codebook splitting (Linde, Buzo and
38
+ Gray, 1980). What is new here is using it inside e-means' streaming revival, with the axis
39
+ estimated online and donors ranked by expected gain.
40
+
41
+ ## 3. Estimating `u_1` and `λ_1` online, at negligible cost
42
+
43
+ For every cluster PE-means keeps `V_j ∈ R^d` and a scalar `P_j`. For each assigned row with
44
+ residual `e = x − c_j` and projection `p = eᵀ u_j`, the batch accumulates `Σ p e` and `Σ p²`; then,
45
+ with the same EMA step `α` e-means uses for the centroids,
46
+
47
+ ```
48
+ V_j ← (1 − α) V_j + α Σ p e u_j ← V_j / |V_j|
49
+ P_j ← (1 − α) P_j + α Σ p² λ_j ≈ P_j / N_j
50
+ ```
51
+
52
+ `Σ p e = (Σ e eᵀ) u_j`, so each update is one step of power iteration on the cluster's EMA scatter
53
+ matrix; `u_j` converges to the top eigenvector at the usual rate `λ_2 / λ_1`. Memory is one extra
54
+ `k × d` array (the size of the centroids); compute is O(B·d) per batch, against O(B·k·d) for
55
+ assignment. On DEEP\* 1M the measured overhead is 7-10% of wall-clock.
56
+
57
+ ## 4. Implementation contract
58
+
59
+ `PEmeans` subclasses the official `faiss.contrib.e_means.NumpyEmeans` and overrides only
60
+ `begin`, `_assign_rows` (to accumulate `Σ p e`, `Σ p²`), `_apply_statistics` (EMA of those),
61
+ `_choose_splits` and `_apply_splits`. Batch law, step law, cosine schedule, EMA update, revival
62
+ threshold and closing means are inherited. With `split="faiss", gain="usage"` it reproduces the
63
+ official centroids bit for bit (`tests/test_pemeans.py`).
64
+
65
+ ## 5. Results so far
66
+
67
+ See the [README](../README.md) for the table. On DEEP\* 1M (the paper's data and test rows,
68
+ k = 16,384, 5 seeds, matched starting centroids, 4 CPU threads), PE-means lowers test MSE by 0.31%,
69
+ 0.27% and 0.20% at 1, 3 and 5 epochs (15 of 15 paired wins), and at 3 epochs matches official
70
+ e-means at 5 epochs in 0.60× the time. On a smaller proxy (DEEP 200k, k = 2,048) the ablation gives:
71
+ principal-axis split alone −0.07%, gain-ranked donor alone −0.12%, both −0.25% (one epoch); the
72
+ two parts add up.
73
+
74
+ ## 6. Limitations
75
+
76
+ * The optimality argument assumes roughly Gaussian clusters; heavy-tailed clusters may need a
77
+ robust axis estimate (planned with o-means).
78
+ * Results are on CPU float32; the paper's numbers are on H100 bfloat16. Starting rows are drawn
79
+ with NumPy rather than a CUDA generator (shared by both methods).
80
+ * Five seeds per budget so far; held-out confirmation on SimSearchNet++ 1M and SPACEV 1M is
81
+ running. No claim beyond these datasets is made yet.
82
+ * The 7-10% overhead is real; the cross-budget comparison (3 vs 5 epochs, 2.7 vs 3 epochs) is the
83
+ fair equal-time view.
84
+
85
+ ## References
86
+
87
+ - Bessa, Mazaré, Szilvasy, Touvron, Picard, Douze, Jégou (2026). e-means: outperforming k-means
88
+ (in one-tenth the epochs). arXiv:2611.09501.
89
+ - Hamerly and Elkan (2003). Learning the k in k-means. NeurIPS.
90
+ - Boley (1998). Principal direction divisive partitioning. Data Mining and Knowledge Discovery.
91
+ - Linde, Buzo and Gray (1980). An algorithm for vector quantizer design. IEEE Trans. Communications.
92
+ - Oja (1982). A simplified neuron model as a principal component analyzer. J. Math. Biology.
93
+
94
+ ## 7. v2 and v3 additions (iterations 2-11, `docs/EXPERIMENTS.md`)
95
+
96
+ * **Consistent closing statistics** (`split_pass`). faiss's final centroids are the mean of each
97
+ cluster's last-epoch assignments. When a donor is split during that epoch, both children
98
+ inherited the parent's pre-split sums; PE-means splits those sums consistently (half the count
99
+ each, means at `c ± delta`). Largest effect at one epoch (−0.29% → −0.64% on the proxy).
100
+ * **Revival threshold 0.2 and step law × 1.4.** With better splits, more aggressive revival and a
101
+ larger EMA step pay off; the larger step alone does not help official e-means (an interaction).
102
+ * **Revival every batch during the first 30% of steps** (v3).
103
+ * **Approximate early-epoch assignment** (v3). Every epoch but the last assigns rows through an IVF
104
+ over the current centroids: √k coarse cells from a 20-iteration k-means on the centroids, √k/8
105
+ probes; the index is rebuilt per batch for O(k·√k·d) work. The last epoch, which builds the closing
106
+ means, stays exact. On DEEP\* 1M this makes an epoch 35-50% cheaper and, surprisingly, *improves*
107
+ quality relative to exact assignment (−0.34% vs −0.17% vs official at 3 epochs); more probes (more
108
+ exact) is worse and fewer probes (8) too noisy. A plausible reading is annealing-like noise early in
109
+ training; this is a hypothesis, not a proven mechanism.
110
+ * **Negative results kept:** streaming swap moves (deletion cost vs split gain), fresher closing
111
+ window, child-mass and threshold schedules, graph-pruned assignment with label memory (21% of rows
112
+ have a nearer centroid outside the candidate set one epoch later; a self-verifying variant falls
113
+ back to exact on almost every batch).
figures/architecture.png ADDED

Git LFS Details

  • SHA256: 0988987bfcecdc049faa35c752a93944731066311cabfe17b96799ac50159b68
  • Pointer size: 131 Bytes
  • Size of remote file: 312 kB
figures/ivf_and_sizes.png ADDED
figures/kscaling.png ADDED
figures/loo_bars.png ADDED
figures/mechanism.png ADDED
figures/quality_vs_time.png ADDED
figures/split_geometry.png ADDED
figures/split_real.png ADDED
figures/tensor_flow.png ADDED

Git LFS Details

  • SHA256: 885375813ad1e73aac33f45099be03470fc0db63f884bacec27d1bd7e02ce5bb
  • Pointer size: 131 Bytes
  • Size of remote file: 169 kB
figures/time_breakdown.png ADDED
figures/ttq_and_seeds.png ADDED
notebooks/01_quickstart.ipynb ADDED
@@ -0,0 +1,190 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "cells": [
3
+ {
4
+ "cell_type": "markdown",
5
+ "id": "85c327d2",
6
+ "metadata": {},
7
+ "source": [
8
+ "# PISCES quickstart\n",
9
+ "\n",
10
+ "PISCES (ARIES #1) against the official e-means of `faiss.contrib.e_means`. Both start from the same\n",
11
+ "k rows; we compare mean squared error on held-out rows and wall-clock time. This uses a 200,000-row\n",
12
+ "slice of DEEP\\* (downloaded with fingerprint checks) and k = 2,048, so it takes a few minutes on a laptop CPU."
13
+ ]
14
+ },
15
+ {
16
+ "cell_type": "code",
17
+ "execution_count": 1,
18
+ "id": "7f427f79",
19
+ "metadata": {
20
+ "execution": {
21
+ "iopub.execute_input": "2026-11-18T00:38:04.054586Z",
22
+ "iopub.status.busy": "2026-11-18T00:38:04.054465Z",
23
+ "iopub.status.idle": "2026-11-18T00:38:04.435329Z",
24
+ "shell.execute_reply": "2026-11-18T00:38:04.433700Z"
25
+ }
26
+ },
27
+ "outputs": [
28
+ {
29
+ "name": "stdout",
30
+ "output_type": "stream",
31
+ "text": [
32
+ "ready\n"
33
+ ]
34
+ }
35
+ ],
36
+ "source": [
37
+ "# Get the code (skip when running inside a clone of this repository)\n",
38
+ "import os, sys, subprocess\n",
39
+ "if not os.path.exists('aries') and os.path.exists('../aries'):\n",
40
+ " os.chdir('..') # opened from notebooks/ inside a clone\n",
41
+ "if not os.path.exists('aries'):\n",
42
+ " subprocess.run([sys.executable, '-m', 'pip', 'install', '-q', 'huggingface_hub'], check=True)\n",
43
+ " from huggingface_hub import snapshot_download\n",
44
+ " root = snapshot_download('Nabidnur/PISCES', local_dir='pisces_repo')\n",
45
+ " os.chdir(root)\n",
46
+ "subprocess.run([sys.executable, '-m', 'pip', 'install', '-q', '--disable-pip-version-check', '-r', 'requirements.txt'], check=True)\n",
47
+ "sys.path.insert(0, os.getcwd()); print('ready')"
48
+ ]
49
+ },
50
+ {
51
+ "cell_type": "code",
52
+ "execution_count": 2,
53
+ "id": "09f24a74",
54
+ "metadata": {
55
+ "execution": {
56
+ "iopub.execute_input": "2026-11-18T00:38:04.436803Z",
57
+ "iopub.status.busy": "2026-11-18T00:38:04.436660Z",
58
+ "iopub.status.idle": "2026-11-18T00:38:04.595549Z",
59
+ "shell.execute_reply": "2026-11-18T00:38:04.594889Z"
60
+ }
61
+ },
62
+ "outputs": [
63
+ {
64
+ "name": "stdout",
65
+ "output_type": "stream",
66
+ "text": [
67
+ "DEEP* 1M already in data/deep1b; fingerprints match the paper's\n",
68
+ "(200000, 96) (5000, 96)\n"
69
+ ]
70
+ }
71
+ ],
72
+ "source": [
73
+ "import os, subprocess, sys\n",
74
+ "subprocess.run([sys.executable, 'benchmarks/get_deep1m.py', 'data/deep1b'], check=True)\n",
75
+ "import numpy as np\n",
76
+ "x = np.ascontiguousarray(np.load('data/deep1b/train_1M.npy', mmap_mode='r')[:200_000])\n",
77
+ "val = np.load('data/deep1b/val.npy')\n",
78
+ "print(x.shape, val.shape)"
79
+ ]
80
+ },
81
+ {
82
+ "cell_type": "code",
83
+ "execution_count": 3,
84
+ "id": "577a520a",
85
+ "metadata": {
86
+ "execution": {
87
+ "iopub.execute_input": "2026-11-18T00:38:04.597106Z",
88
+ "iopub.status.busy": "2026-11-18T00:38:04.596661Z",
89
+ "iopub.status.idle": "2026-11-18T00:38:28.950319Z",
90
+ "shell.execute_reply": "2026-11-18T00:38:28.949117Z"
91
+ }
92
+ },
93
+ "outputs": [
94
+ {
95
+ "name": "stdout",
96
+ "output_type": "stream",
97
+ "text": [
98
+ "seed 1 epochs 1: e-means 0.45583 (0.9s) PISCES 0.45208 (1.5s) change -0.82%\n"
99
+ ]
100
+ },
101
+ {
102
+ "name": "stdout",
103
+ "output_type": "stream",
104
+ "text": [
105
+ "seed 1 epochs 3: e-means 0.45079 (3.1s) PISCES 0.44921 (3.0s) change -0.35%\n"
106
+ ]
107
+ },
108
+ {
109
+ "name": "stdout",
110
+ "output_type": "stream",
111
+ "text": [
112
+ "seed 2 epochs 1: e-means 0.45764 (1.0s) PISCES 0.45328 (1.3s) change -0.95%\n"
113
+ ]
114
+ },
115
+ {
116
+ "name": "stdout",
117
+ "output_type": "stream",
118
+ "text": [
119
+ "seed 2 epochs 3: e-means 0.45022 (2.8s) PISCES 0.44922 (2.4s) change -0.22%\n"
120
+ ]
121
+ },
122
+ {
123
+ "name": "stdout",
124
+ "output_type": "stream",
125
+ "text": [
126
+ "seed 3 epochs 1: e-means 0.45633 (1.0s) PISCES 0.45430 (1.1s) change -0.44%\n"
127
+ ]
128
+ },
129
+ {
130
+ "name": "stdout",
131
+ "output_type": "stream",
132
+ "text": [
133
+ "seed 3 epochs 3: e-means 0.45083 (3.1s) PISCES 0.44903 (2.9s) change -0.40%\n"
134
+ ]
135
+ }
136
+ ],
137
+ "source": [
138
+ "import time, faiss\n",
139
+ "from faiss.contrib.e_means import NumpyEmeans\n",
140
+ "from aries import pisces\n",
141
+ "\n",
142
+ "def mse(c):\n",
143
+ " D, _ = faiss.knn(val, np.ascontiguousarray(c, dtype='float32'), 1)\n",
144
+ " return float(D.astype('float64').mean())\n",
145
+ "\n",
146
+ "k = 2048\n",
147
+ "rows = []\n",
148
+ "for seed in [1, 2, 3]:\n",
149
+ " init = x[np.random.RandomState(seed).permutation(len(x))[:k]].copy()\n",
150
+ " for epochs in [1, 3]:\n",
151
+ " t = time.time(); c, _ = NumpyEmeans(x.shape[1], k, nepochs=epochs, seed=seed, batch_law='throughput').train(x, init_centroids=init.copy()); te = time.time() - t\n",
152
+ " t = time.time(); cp, _, _ = pisces(x, k, nepochs=epochs, seed=seed, init=init.copy()); tp = time.time() - t\n",
153
+ " rows.append((seed, epochs, mse(c), te, mse(cp), tp))\n",
154
+ " print(f'seed {seed} epochs {epochs}: e-means {rows[-1][2]:.5f} ({te:.1f}s) PISCES {rows[-1][4]:.5f} ({tp:.1f}s) change {100*(rows[-1][4]/rows[-1][2]-1):+.2f}%')"
155
+ ]
156
+ },
157
+ {
158
+ "cell_type": "markdown",
159
+ "id": "dc7a3e01",
160
+ "metadata": {},
161
+ "source": [
162
+ "Negative change = PISCES has lower error. Expect roughly −0.4% to −1% at 1 epoch and −0.2% to −0.4% at 3 epochs.\n",
163
+ "At this small k (2,048) PISCES assigns rows exactly (the sparse index only pays from k = 4,096 on), so it takes\n",
164
+ "about the same time as e-means here. The speed-up appears at larger k: on the paper's benchmark (k = 16,384,\n",
165
+ "1M rows) PISCES takes 0.27-0.42× e-means' time."
166
+ ]
167
+ }
168
+ ],
169
+ "metadata": {
170
+ "kernelspec": {
171
+ "display_name": "Python 3",
172
+ "language": "python",
173
+ "name": "python3"
174
+ },
175
+ "language_info": {
176
+ "codemirror_mode": {
177
+ "name": "ipython",
178
+ "version": 3
179
+ },
180
+ "file_extension": ".py",
181
+ "mimetype": "text/x-python",
182
+ "name": "python",
183
+ "nbconvert_exporter": "python",
184
+ "pygments_lexer": "ipython3",
185
+ "version": "3.13.18"
186
+ }
187
+ },
188
+ "nbformat": 4,
189
+ "nbformat_minor": 5
190
+ }
notebooks/02_reproduce_kaggle_cpu.ipynb ADDED
@@ -0,0 +1,109 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "cells": [
3
+ {
4
+ "cell_type": "markdown",
5
+ "metadata": {},
6
+ "source": [
7
+ "# Reproduce the PISCES benchmark on a free Kaggle CPU\n",
8
+ "\n",
9
+ "Kaggle settings: **Accelerator = None (CPU)**, **Internet = On**. This runs the e-means paper's protocol\n",
10
+ "(DEEP\\* 1M, k = 16,384, fingerprint-checked train / test rows, shared starting centroids) for official\n",
11
+ "e-means, PISCES, and e-means given PISCES' sparse assignment (the fairness baseline).\n",
12
+ "Every run is appended to a JSONL file, so a stopped session can be resumed. 3 epochs \u00d7 5 seeds \u00d7 3\n",
13
+ "methods takes about 30 minutes on 4 CPU threads."
14
+ ]
15
+ },
16
+ {
17
+ "cell_type": "code",
18
+ "metadata": {},
19
+ "source": [
20
+ "# Get the code (skip when running inside a clone of this repository)\n",
21
+ "import os, sys, subprocess\n",
22
+ "if not os.path.exists('aries') and os.path.exists('../aries'):\n",
23
+ " os.chdir('..') # opened from notebooks/ inside a clone\n",
24
+ "if not os.path.exists('aries'):\n",
25
+ " subprocess.run([sys.executable, '-m', 'pip', 'install', '-q', 'huggingface_hub'], check=True)\n",
26
+ " from huggingface_hub import snapshot_download\n",
27
+ " root = snapshot_download('Nabidnur/PISCES', local_dir='pisces_repo')\n",
28
+ " os.chdir(root)\n",
29
+ "subprocess.run([sys.executable, '-m', 'pip', 'install', '-q', '--disable-pip-version-check', '-r', 'requirements.txt'], check=True)\n",
30
+ "sys.path.insert(0, os.getcwd()); print('ready')"
31
+ ],
32
+ "outputs": [],
33
+ "execution_count": null
34
+ },
35
+ {
36
+ "cell_type": "code",
37
+ "metadata": {},
38
+ "source": [
39
+ "import subprocess, sys\n",
40
+ "subprocess.run([sys.executable, '-m', 'pytest', '-q', 'tests'], check=True)"
41
+ ],
42
+ "outputs": [],
43
+ "execution_count": null
44
+ },
45
+ {
46
+ "cell_type": "code",
47
+ "metadata": {},
48
+ "source": [
49
+ "subprocess.run([sys.executable, 'benchmarks/get_deep1m.py', '/kaggle/working/data/deep1b'], check=True)"
50
+ ],
51
+ "outputs": [],
52
+ "execution_count": null
53
+ },
54
+ {
55
+ "cell_type": "code",
56
+ "metadata": {},
57
+ "source": [
58
+ "import os\n",
59
+ "T = str(os.cpu_count())\n",
60
+ "OUT = '/kaggle/working/deep1m_kaggle.jsonl'\n",
61
+ "subprocess.run([sys.executable, 'benchmarks/paper_protocol.py', '/kaggle/working/data/deep1b', OUT,\n",
62
+ " '--epochs', '3', '--seeds', '10', '11', '12', '13', '14',\n",
63
+ " '--methods', 'official', 'pisces', 'emeans_approx', '--threads', T], check=True)"
64
+ ],
65
+ "outputs": [],
66
+ "execution_count": null
67
+ },
68
+ {
69
+ "cell_type": "code",
70
+ "metadata": {},
71
+ "source": [
72
+ "subprocess.run([sys.executable, 'benchmarks/summarize.py', OUT], check=True)"
73
+ ],
74
+ "outputs": [],
75
+ "execution_count": null
76
+ },
77
+ {
78
+ "cell_type": "markdown",
79
+ "metadata": {},
80
+ "source": [
81
+ "Optional: held-out datasets (SPACEV 1M, SimSearchNet++ 1M) and the downstream IVF-index benchmark."
82
+ ]
83
+ },
84
+ {
85
+ "cell_type": "code",
86
+ "metadata": {},
87
+ "source": [
88
+ "# subprocess.run([sys.executable, 'benchmarks/get_paper1m.py', '/kaggle/working/data'], check=True)\n",
89
+ "# subprocess.run([sys.executable, 'benchmarks/paper_protocol.py', '/kaggle/working/data/spacev', '/kaggle/working/spacev.jsonl',\n",
90
+ "# '--dataset', 'spacev-1M', '--epochs', '3', '--seeds', '10', '11', '12', '--methods', 'official', 'pisces', '--threads', T], check=True)\n",
91
+ "# subprocess.run([sys.executable, 'benchmarks/ivf_downstream.py', '/kaggle/working/data/deep1b', '/kaggle/working/ivf.jsonl'], check=True)"
92
+ ],
93
+ "outputs": [],
94
+ "execution_count": null
95
+ }
96
+ ],
97
+ "metadata": {
98
+ "kernelspec": {
99
+ "display_name": "Python 3",
100
+ "language": "python",
101
+ "name": "python3"
102
+ },
103
+ "language_info": {
104
+ "name": "python"
105
+ }
106
+ },
107
+ "nbformat": 4,
108
+ "nbformat_minor": 5
109
+ }
notebooks/03_results_analysis.ipynb ADDED
The diff for this file is too large to render. See raw diff
 
notebooks/make_notebooks.py ADDED
@@ -0,0 +1,229 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Writes the three notebooks of this repository (run from the repository root or anywhere).
2
+
3
+ 01_quickstart.ipynb PISCES vs official e-means on a 200k-row DEEP slice, a few minutes on any CPU
4
+ 02_reproduce_kaggle_cpu.ipynb the paper's full protocol on a free Kaggle CPU notebook (Internet on)
5
+ 03_results_analysis.ipynb every table and figure of the report from the raw JSONL results, no training
6
+ """
7
+ import json, os
8
+
9
+ HF_REPO = "Nabidnur/PISCES"
10
+ HERE = os.path.dirname(os.path.abspath(__file__))
11
+
12
+ SETUP = (f"# Get the code (skip when running inside a clone of this repository)\n"
13
+ "import os, sys, subprocess\n"
14
+ "if not os.path.exists('aries') and os.path.exists('../aries'):\n"
15
+ " os.chdir('..') # opened from notebooks/ inside a clone\n"
16
+ "if not os.path.exists('aries'):\n"
17
+ " subprocess.run([sys.executable, '-m', 'pip', 'install', '-q', 'huggingface_hub'], check=True)\n"
18
+ " from huggingface_hub import snapshot_download\n"
19
+ f" root = snapshot_download('{HF_REPO}', local_dir='pisces_repo')\n"
20
+ " os.chdir(root)\n"
21
+ "subprocess.run([sys.executable, '-m', 'pip', 'install', '-q', '--disable-pip-version-check', '-r', 'requirements.txt'], check=True)\n"
22
+ "sys.path.insert(0, os.getcwd()); print('ready')")
23
+
24
+
25
+ def nb(cells, path):
26
+ out = {"cells": [{"cell_type": t, "metadata": {}, "source": s.strip("\n").splitlines(keepends=True),
27
+ **({"outputs": [], "execution_count": None} if t == "code" else {})} for t, s in cells],
28
+ "metadata": {"kernelspec": {"display_name": "Python 3", "language": "python", "name": "python3"},
29
+ "language_info": {"name": "python"}},
30
+ "nbformat": 4, "nbformat_minor": 5}
31
+ json.dump(out, open(os.path.join(HERE, path), "w"), indent=1)
32
+ print("wrote", path)
33
+
34
+
35
+ nb([
36
+ ("markdown", """
37
+ # PISCES quickstart
38
+
39
+ PISCES (ARIES #1) against the official e-means of `faiss.contrib.e_means`. Both start from the same
40
+ k rows; we compare mean squared error on held-out rows and wall-clock time. This uses a 200,000-row
41
+ slice of DEEP\\* (downloaded with fingerprint checks) and k = 2,048, so it takes a few minutes on a laptop CPU.
42
+ """),
43
+ ("code", SETUP),
44
+ ("code", """
45
+ import os, subprocess, sys
46
+ subprocess.run([sys.executable, 'benchmarks/get_deep1m.py', 'data/deep1b'], check=True)
47
+ import numpy as np
48
+ x = np.ascontiguousarray(np.load('data/deep1b/train_1M.npy', mmap_mode='r')[:200_000])
49
+ val = np.load('data/deep1b/val.npy')
50
+ print(x.shape, val.shape)
51
+ """),
52
+ ("code", """
53
+ import time, faiss
54
+ from faiss.contrib.e_means import NumpyEmeans
55
+ from aries import pisces
56
+
57
+ def mse(c):
58
+ D, _ = faiss.knn(val, np.ascontiguousarray(c, dtype='float32'), 1)
59
+ return float(D.astype('float64').mean())
60
+
61
+ k = 2048
62
+ rows = []
63
+ for seed in [1, 2, 3]:
64
+ init = x[np.random.RandomState(seed).permutation(len(x))[:k]].copy()
65
+ for epochs in [1, 3]:
66
+ t = time.time(); c, _ = NumpyEmeans(x.shape[1], k, nepochs=epochs, seed=seed, batch_law='throughput').train(x, init_centroids=init.copy()); te = time.time() - t
67
+ t = time.time(); cp, _, _ = pisces(x, k, nepochs=epochs, seed=seed, init=init.copy()); tp = time.time() - t
68
+ rows.append((seed, epochs, mse(c), te, mse(cp), tp))
69
+ print(f'seed {seed} epochs {epochs}: e-means {rows[-1][2]:.5f} ({te:.1f}s) PISCES {rows[-1][4]:.5f} ({tp:.1f}s) change {100*(rows[-1][4]/rows[-1][2]-1):+.2f}%')
70
+ """),
71
+ ("markdown", """
72
+ Negative change = PISCES has lower error. Expect roughly −0.4% to −1% at 1 epoch and −0.2% to −0.4% at 3 epochs.
73
+ At this small k (2,048) PISCES assigns rows exactly (the sparse index only pays from k = 4,096 on), so it takes
74
+ about the same time as e-means here. The speed-up appears at larger k: on the paper's benchmark (k = 16,384,
75
+ 1M rows) PISCES takes 0.27-0.42× e-means' time.
76
+ """),
77
+ ], "01_quickstart.ipynb")
78
+
79
+ nb([
80
+ ("markdown", """
81
+ # Reproduce the PISCES benchmark on a free Kaggle CPU
82
+
83
+ Kaggle settings: **Accelerator = None (CPU)**, **Internet = On**. This runs the e-means paper's protocol
84
+ (DEEP\\* 1M, k = 16,384, fingerprint-checked train / test rows, shared starting centroids) for official
85
+ e-means, PISCES, and e-means given PISCES' sparse assignment (the fairness baseline).
86
+ Every run is appended to a JSONL file, so a stopped session can be resumed. 3 epochs × 5 seeds × 3
87
+ methods takes about 30 minutes on 4 CPU threads.
88
+ """),
89
+ ("code", SETUP),
90
+ ("code", "import subprocess, sys\nsubprocess.run([sys.executable, '-m', 'pytest', '-q', 'tests'], check=True)"),
91
+ ("code", "subprocess.run([sys.executable, 'benchmarks/get_deep1m.py', '/kaggle/working/data/deep1b'], check=True)"),
92
+ ("code", """
93
+ import os
94
+ T = str(os.cpu_count())
95
+ OUT = '/kaggle/working/deep1m_kaggle.jsonl'
96
+ subprocess.run([sys.executable, 'benchmarks/paper_protocol.py', '/kaggle/working/data/deep1b', OUT,
97
+ '--epochs', '3', '--seeds', '10', '11', '12', '13', '14',
98
+ '--methods', 'official', 'pisces', 'emeans_approx', '--threads', T], check=True)
99
+ """),
100
+ ("code", "subprocess.run([sys.executable, 'benchmarks/summarize.py', OUT], check=True)"),
101
+ ("markdown", "Optional: held-out datasets (SPACEV 1M, SimSearchNet++ 1M) and the downstream IVF-index benchmark."),
102
+ ("code", """
103
+ # subprocess.run([sys.executable, 'benchmarks/get_paper1m.py', '/kaggle/working/data'], check=True)
104
+ # subprocess.run([sys.executable, 'benchmarks/paper_protocol.py', '/kaggle/working/data/spacev', '/kaggle/working/spacev.jsonl',
105
+ # '--dataset', 'spacev-1M', '--epochs', '3', '--seeds', '10', '11', '12', '--methods', 'official', 'pisces', '--threads', T], check=True)
106
+ # subprocess.run([sys.executable, 'benchmarks/ivf_downstream.py', '/kaggle/working/data/deep1b', '/kaggle/working/ivf.jsonl'], check=True)
107
+ """),
108
+ ], "02_reproduce_kaggle_cpu.ipynb")
109
+
110
+ nb([
111
+ ("markdown", """
112
+ # Results analysis: every number in the report, from the raw runs
113
+
114
+ No training here: this reads `results/*.jsonl` (one line per run) and recomputes the paired
115
+ statistics, confidence intervals and figures of the technical report.
116
+ """),
117
+ ("code", SETUP),
118
+ ("code", """
119
+ import json, collections, numpy as np
120
+ import matplotlib.pyplot as plt
121
+
122
+ def load(f):
123
+ g = collections.defaultdict(dict)
124
+ for line in open(f):
125
+ r = json.loads(line); g[(r['method'], float(r['epochs']))][r['seed']] = r
126
+ return g
127
+
128
+ def paired(g, m, T, base='official', Tb=None):
129
+ o, p = g[(base, Tb or T)], g[(m, T)]; s = sorted(set(o) & set(p))
130
+ d = np.array([100 * (p[i]['test_mse'] / o[i]['test_mse'] - 1) for i in s])
131
+ t = np.array([p[i]['seconds'] / o[i]['seconds'] for i in s])
132
+ boot = np.random.default_rng(0).choice(d, (10000, len(d))).mean(1)
133
+ return dict(n=len(s), mean=d.mean(), lo=np.percentile(boot, 2.5), hi=np.percentile(boot, 97.5), wins=int((d < 0).sum()), time=t.mean())
134
+
135
+ deep = load('results/final/deep1m_test.jsonl')
136
+ for T in [1.0, 3.0, 5.0]:
137
+ print(f'{T:.0f} epochs', {k: round(float(v), 3) if not isinstance(v, int) else v for k, v in paired(deep, 'pisces', T).items()})
138
+ """),
139
+ ("code", """
140
+ fig, ax = plt.subplots(figsize=(7, 4))
141
+ for m, style in [('official', 'o-'), ('pisces', '^-'), ('emeans_approx', 's')]:
142
+ pts = sorted((T, np.mean([r['seconds'] for r in v.values()]), np.mean([r['test_mse'] for r in v.values()]))
143
+ for (mm, T), v in deep.items() if mm == m)
144
+ if pts:
145
+ ax.plot([p[1] for p in pts], [p[2] for p in pts], style, label=m)
146
+ ax.set_xlabel('wall-clock seconds (4 threads)'); ax.set_ylabel('test MSE'); ax.legend(); ax.grid(alpha=.3)
147
+ plt.show()
148
+ """),
149
+ ("code", """
150
+ for f, name in [('results/final/spacev1m_test.jsonl', 'SPACEV 1M'), ('results/final/ssnpp1m_test.jsonl', 'SimSearchNet++ 1M')]:
151
+ g = load(f)
152
+ print(name, {k: round(float(v), 3) if not isinstance(v, int) else v for k, v in paired(g, 'pisces', 3.0).items()})
153
+ """),
154
+ ("markdown", """
155
+ ## Leave-one-out ablation at full scale
156
+
157
+ Each component removed from PISCES in turn (DEEP\\* 1M, k = 16,384, validation rows, 3 epochs, seeds 1-3).
158
+ Cost = paired change in MSE relative to PISCES itself (positive = the component was helping).
159
+ """),
160
+ ("code", """
161
+ abl = load('results/final/ablation_full_val.jsonl')
162
+ names = sorted({m for m, T in abl if m not in ('official', 'pisces')})
163
+ cost = {m: paired(abl, m, 3.0, base='pisces')['mean'] for m in names}
164
+ for m in sorted(cost, key=cost.get, reverse=True):
165
+ print(f'{m:24s} removal costs {cost[m]:+.3f}% vs e-means {paired(abl, m, 3.0)["mean"]:+.3f}%')
166
+ fig, ax = plt.subplots(figsize=(7, 3.5))
167
+ order = sorted(cost, key=cost.get)
168
+ ax.barh(order, [cost[m] for m in order], color='#0E8C8C'); ax.axvline(0, color='#C4413A')
169
+ ax.set_xlabel('cost of removing the component (% MSE vs PISCES)'); ax.grid(alpha=.3, axis='x'); plt.tight_layout(); plt.show()
170
+ """),
171
+ ("markdown", """
172
+ ## Scaling in k, and the two regime rules
173
+
174
+ `scaling_k*.jsonl` hold PISCES exactly as frozen. `results/dev/iter23_small_k_val.jsonl` (exact assignment at
175
+ k = 1,024) and `iter24_k65536_regime_val.jsonl` (no every-batch revival at k = 65,536) are the runs behind the two
176
+ rules of the released defaults; both were chosen on these validation runs.
177
+ """),
178
+ ("code", """
179
+ for k in [1024, 4096, 65536]:
180
+ g = load(f'results/final/scaling_k{k}_val.jsonl'); r = paired(g, 'pisces', 3.0)
181
+ print(f'k = {k:6d}: {r["mean"]:+.3f}% ({r["wins"]}/{r["n"]} seeds), {r["time"]:.2f}x the time')
182
+ g = load('results/dev/iter23_small_k_val.jsonl'); r = paired(g, 'exact_k1024', 3.0)
183
+ print(f'k = 1024 with the small-k rule: {r["mean"]:+.3f}% ({r["wins"]}/{r["n"]}), {r["time"]:.2f}x the time')
184
+ g = load('results/final/scaling_k65536_val.jsonl')
185
+ for k_, v in load('results/dev/iter24_k65536_regime_val.jsonl').items():
186
+ g[('diag_' + k_[0], k_[1])] = v
187
+ for m in ['diag_pisces', 'diag_tau_0.1', 'diag_no_early_revival', 'diag_tau_0.1_no_early']:
188
+ r = paired(g, m, 3.0); print(f'k = 65536, {m[5:]:18s}: {r["mean"]:+.3f}% ({r["wins"]}/{r["n"]})')
189
+ """),
190
+ ("markdown", """
191
+ ## Downstream: the trained centroids as an IVF index
192
+
193
+ Recall@1 on 10K DEEP1B queries against the number of distance computations (probes 1-128).
194
+ """),
195
+ ("code", """
196
+ ivf = [json.loads(l) for l in open('results/final/ivf_downstream.jsonl')]
197
+ fig, ax = plt.subplots(figsize=(6, 4))
198
+ for m, c in [('official', '#16233F'), ('pisces', '#0E8C8C')]:
199
+ rs = [r for r in ivf if r['method'] == m]
200
+ nd = np.mean([[p['ndis'] for p in r['curve']] for r in rs], 0); rc = np.mean([[p['recall1'] for p in r['curve']] for r in rs], 0)
201
+ ax.semilogx(nd, rc, 'o-', color=c, label=f"{m} (imbalance {np.mean([r['imbalance'] for r in rs]):.3f})")
202
+ ax.set_xlabel('distance computations per query'); ax.set_ylabel('recall@1'); ax.legend(); ax.grid(alpha=.3); plt.show()
203
+ """),
204
+ ("markdown", """
205
+ ## Mechanism: revivals, Lemma 2 on real clusters, split-merge audit
206
+ """),
207
+ ("code", """
208
+ mech = json.load(open('results/final/mechanism_seed10.json'))
209
+ for m in ['official', 'pisces']:
210
+ L = mech[m]['lemma']; pred = np.array([l['predicted'] for l in L]); ax_ = np.array([l['axis_split'] for l in L])
211
+ print(f"{m:9s} revivals {sum(n for step, n in mech[m]['split_log']):,}"
212
+ f" Lemma 2 realized/predicted (median) {np.median(ax_ / pred):.2f} log-log corr {np.corrcoef(np.log(pred), np.log(ax_))[0, 1]:.3f}"
213
+ f" profitable split-merge moves {mech[m]['swaps']['profitable_moves']}")
214
+ L = mech['pisces']['lemma']
215
+ fig, ax = plt.subplots(figsize=(4.5, 4.5))
216
+ ax.scatter([l['predicted'] for l in L], [l['axis_split'] for l in L], s=8, color='#0E8C8C', label='split along u1')
217
+ ax.scatter([l['predicted'] for l in L], [l['free_2means'] for l in L], s=8, marker='x', color='#DD7A1E', label='unrestricted 2-means')
218
+ lim = [0.4, 2.8]; ax.plot(lim, lim, '--', color='#C4413A', label='y = x (Lemma 2)')
219
+ ax.set_xlabel('predicted 2/pi N lambda1'); ax.set_ylabel('realized gain'); ax.legend(); ax.grid(alpha=.3); plt.show()
220
+ """),
221
+ ("markdown", """
222
+ ## Where the time goes
223
+ """),
224
+ ("code", """
225
+ tb = json.load(open('results/final/time_breakdown.json'))
226
+ for m, v in tb.items():
227
+ print(f"{m:11s} wall {v['wall']:6.1f} s " + " ".join(f"{g} {v[g]:5.1f}" for g in ['assignment search', 'statistics', 'index build', 'other']))
228
+ """),
229
+ ], "03_results_analysis.ipynb")
paper/Ball.sty ADDED
@@ -0,0 +1,41 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ % Modified from PlotNeuralNet (MIT, H. Iqbal): internal macros renamed (\caption -> \pnncaption, \fill -> \pnnfill) so the styles do not clash with LaTeX figure captions.
2
+ \ProvidesPackage{Ball}
3
+ %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
4
+ %This Block can draw small Ball
5
+ %Elementwise or reduction operations can be drawn with this
6
+ %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
7
+
8
+ \tikzset{Ball/.pic={\tikzset{/sphere/.cd,#1}
9
+
10
+ \pgfmathsetmacro{\r}{\radius*\scale}
11
+
12
+ \shade[ball color=\pnnfill,opacity=\opacity] (0,0,0) circle (\r);
13
+ \draw (0,0,0) circle [radius=\r] node[scale=4*\r] {\logo};
14
+
15
+ \coordinate (\name-anchor) at ( 0 , 0 , 0) ;
16
+ \coordinate (\name-east) at ( \r, 0 , 0) ;
17
+ \coordinate (\name-west) at (-\r, 0 , 0) ;
18
+ \coordinate (\name-north) at ( 0 , \r , 0) ;
19
+ \coordinate (\name-south) at ( 0 , -\r, 0) ;
20
+
21
+ \path (\name-south) + (0,-20pt) coordinate (caption-node)
22
+ edge ["\textcolor{black}{\bf \pnncaption}"'] (caption-node); %Ball caption
23
+
24
+ },
25
+ /sphere/.search also={/tikz},
26
+ /sphere/.cd,
27
+ radius/.store in=\radius,
28
+ scale/.store in=\scale,
29
+ caption/.store in=\pnncaption,
30
+ name/.store in=\name,
31
+ fill/.store in=\pnnfill,
32
+ logo/.store in=\logo,
33
+ opacity/.store in=\opacity,
34
+ logo=$\Sigma$,
35
+ fill=green,
36
+ opacity=0.10,
37
+ scale=0.2,
38
+ radius=0.5,
39
+ caption=,
40
+ name=,
41
+ }
paper/Box.sty ADDED
@@ -0,0 +1,111 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ % Modified from PlotNeuralNet (MIT, H. Iqbal): internal macros renamed (\caption -> \pnncaption, \fill -> \pnnfill) so the styles do not clash with LaTeX figure captions.
2
+ \ProvidesPackage{Box}
3
+
4
+ %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
5
+ % This Block can draw simple block of boxes with custom colors.
6
+ % Can be used for conv, deconv etc
7
+ %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
8
+ \tikzset{Box/.pic={\tikzset{/boxblock/.cd,#1}
9
+ \tikzstyle{box}=[every edge/.append style={pic actions, densely dashed, opacity=.7},fill opacity=\opacity, pic actions,fill=\pnnfill]
10
+
11
+ \pgfmathsetmacro{\y}{\cubey*\scale}
12
+ \pgfmathsetmacro{\z}{\cubez*\scale}
13
+
14
+ %Multiple concatenated boxes
15
+ \foreach[count=\i,%
16
+ evaluate=\i as \xlabel using {array({\boxlabels},\i-1)},%
17
+ evaluate=\unscaledx as \k using {\unscaledx*\scale+\prev}, remember=\k as \prev (initially 0)]
18
+ \unscaledx in \cubex
19
+ {
20
+ \pgfmathsetmacro{\x}{\unscaledx*\scale}
21
+ \coordinate (a) at (\k-\x , \y/2 , \z/2);
22
+ \coordinate (b) at (\k-\x ,-\y/2 , \z/2);
23
+ \coordinate (c) at (\k ,-\y/2 , \z/2);
24
+ \coordinate (d) at (\k , \y/2 , \z/2);
25
+ \coordinate (e) at (\k , \y/2 ,-\z/2);
26
+ \coordinate (f) at (\k ,-\y/2 ,-\z/2);
27
+ \coordinate (g) at (\k-\x ,-\y/2 ,-\z/2);
28
+ \coordinate (h) at (\k-\x , \y/2 ,-\z/2);
29
+
30
+ \draw [box]
31
+ (d) -- (a) -- (b) -- (c) -- cycle
32
+ (d) -- (a) -- (h) -- (e) -- cycle
33
+ %dotted edges
34
+ (f) edge (g)
35
+ (b) edge (g)
36
+ (h) edge (g)
37
+ ;
38
+ \path (b) edge ["\xlabel"',midway] (c);
39
+
40
+ \xdef\LastEastx{\k} %\k persists as \LastEastx after loop
41
+ }%Loop ends
42
+ \draw [box] (d) -- (e) -- (f) -- (c) -- cycle; %East face of last box
43
+
44
+ \coordinate (a1) at (0 , \y/2 , \z/2);
45
+ \coordinate (b1) at (0 ,-\y/2 , \z/2);
46
+ \tikzstyle{depthlabel}=[pos=0,text width=14*\z,text centered,sloped]
47
+
48
+ \path (c) edge ["\small\zlabel"',depthlabel](f); %depth label
49
+ \path (b1) edge ["\ylabel",midway] (a1); %height label
50
+
51
+
52
+ \tikzstyle{captionlabel}=[text width=15*\LastEastx/\scale,text centered]
53
+ \path (\LastEastx/2,-\y/2,+\z/2) + (0,-25pt) coordinate (cap)
54
+ edge ["\textcolor{black}{ \bf \pnncaption}"',captionlabel](cap) ; %Block caption/pic object label
55
+
56
+ %Define nodes to be used outside on the pic object
57
+ \coordinate (\name-west) at (0,0,0) ;
58
+ \coordinate (\name-east) at (\LastEastx, 0,0) ;
59
+ \coordinate (\name-north) at (\LastEastx/2,\y/2,0);
60
+ \coordinate (\name-south) at (\LastEastx/2,-\y/2,0);
61
+ \coordinate (\name-anchor) at (\LastEastx/2, 0,0) ;
62
+
63
+ \coordinate (\name-near) at (\LastEastx/2,0,\z/2);
64
+ \coordinate (\name-far) at (\LastEastx/2,0,-\z/2);
65
+
66
+ \coordinate (\name-nearwest) at (0,0,\z/2);
67
+ \coordinate (\name-neareast) at (\LastEastx,0,\z/2);
68
+ \coordinate (\name-farwest) at (0,0,-\z/2);
69
+ \coordinate (\name-fareast) at (\LastEastx,0,-\z/2);
70
+
71
+ \coordinate (\name-northeast) at (\name-north-|\name-east);
72
+ \coordinate (\name-northwest) at (\name-north-|\name-west);
73
+ \coordinate (\name-southeast) at (\name-south-|\name-east);
74
+ \coordinate (\name-southwest) at (\name-south-|\name-west);
75
+
76
+ \coordinate (\name-nearnortheast) at (\LastEastx, \y/2, \z/2);
77
+ \coordinate (\name-farnortheast) at (\LastEastx, \y/2,-\z/2);
78
+ \coordinate (\name-nearsoutheast) at (\LastEastx,-\y/2, \z/2);
79
+ \coordinate (\name-farsoutheast) at (\LastEastx,-\y/2,-\z/2);
80
+
81
+ \coordinate (\name-nearnorthwest) at (0, \y/2, \z/2);
82
+ \coordinate (\name-farnorthwest) at (0, \y/2,-\z/2);
83
+ \coordinate (\name-nearsouthwest) at (0,-\y/2, \z/2);
84
+ \coordinate (\name-farsouthwest) at (0,-\y/2,-\z/2);
85
+
86
+ },
87
+ /boxblock/.search also={/tikz},
88
+ /boxblock/.cd,
89
+ width/.store in=\cubex,
90
+ height/.store in=\cubey,
91
+ depth/.store in=\cubez,
92
+ scale/.store in=\scale,
93
+ xlabel/.store in=\boxlabels,
94
+ ylabel/.store in=\ylabel,
95
+ zlabel/.store in=\zlabel,
96
+ caption/.store in=\pnncaption,
97
+ name/.store in=\name,
98
+ fill/.store in=\pnnfill,
99
+ opacity/.store in=\opacity,
100
+ fill={rgb:red,5;green,5;blue,5;white,15},
101
+ opacity=0.4,
102
+ width=2,
103
+ height=13,
104
+ depth=15,
105
+ scale=.2,
106
+ xlabel={{"","","","","","","","","",""}},
107
+ ylabel=,
108
+ zlabel=,
109
+ caption=,
110
+ name=,
111
+ }
paper/PLOTNEURALNET_LICENSE ADDED
@@ -0,0 +1,21 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ MIT License
2
+
3
+ Copyright (c) 2018 HarisIqbal88
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
paper/RightBandedBox.sty ADDED
@@ -0,0 +1,130 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ \ProvidesPackage{RightBandedBox}
2
+
3
+ %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
4
+ % This Block can draw simple block of boxes with custom colors.
5
+ % Can be used for conv, deconv etc
6
+ %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
7
+ \tikzset{RightBandedBox/.pic={\tikzset{/block/.cd,#1}
8
+
9
+ \tikzstyle{box}=[every edge/.append style={pic actions, densely dashed, opacity=.7},fill opacity=\opacity, pic actions,fill=\fill]
10
+
11
+ \tikzstyle{band}=[every edge/.append style={pic actions, densely dashed, opacity=.7},fill opacity=\bandopacity, pic actions,fill=\bandfill,draw=\bandfill]
12
+
13
+ \pgfmathsetmacro{\y}{\cubey*\scale}
14
+ \pgfmathsetmacro{\z}{\cubez*\scale}
15
+
16
+ %Multiple concatenated boxes
17
+ \foreach[count=\i,%
18
+ evaluate=\i as \xlabel using {array({\boxlabels},\i-1)},%
19
+ evaluate=\unscaledx as \k using {\unscaledx*\scale+\prev}, remember=\k as \prev (initially 0)]
20
+ \unscaledx in \cubex
21
+ {
22
+ \pgfmathsetmacro{\x}{\unscaledx*\scale}
23
+ \coordinate (a) at (\k-\x , \y/2 , \z/2);
24
+ \coordinate (art) at (\k-\x/3 , \y/2 , \z/2); %a_right_third
25
+ \coordinate (b) at (\k-\x ,-\y/2 , \z/2);
26
+ \coordinate (brt) at (\k-\x/3 ,-\y/2 , \z/2); %b_right_third
27
+ \coordinate (c) at (\k ,-\y/2 , \z/2);
28
+ \coordinate (d) at (\k , \y/2 , \z/2);
29
+ \coordinate (e) at (\k , \y/2 ,-\z/2);
30
+ \coordinate (f) at (\k ,-\y/2 ,-\z/2);
31
+ \coordinate (g) at (\k-\x ,-\y/2 ,-\z/2);
32
+ \coordinate (h) at (\k-\x , \y/2 ,-\z/2);
33
+ \coordinate (hrt) at (\k-\x/3 , \y/2 ,-\z/2); %h_right_third
34
+
35
+ %fill box color
36
+ \draw [box]
37
+ (d) -- (a) -- (b) -- (c) -- cycle
38
+ (d) -- (a) -- (h) -- (e) -- cycle;
39
+ %dotted edges
40
+ \draw [box]
41
+ (f) edge (g)
42
+ (b) edge (g)
43
+ (h) edge (g);
44
+ %fill band color
45
+ \draw [band]
46
+ (d) -- (art) -- (brt) -- (c) -- cycle
47
+ (d) -- (art) -- (hrt) -- (e) -- cycle;
48
+ %draw edges again which were covered by band
49
+ \draw [box,fill opacity=0]
50
+ (d) -- (a) -- (b) -- (c) -- cycle
51
+ (d) -- (a) -- (h) -- (e) -- cycle;
52
+
53
+ \path (b) edge ["\xlabel"',midway] (c);
54
+
55
+ \xdef\LastEastx{\k} %\k persists as \LastEastx after loop
56
+ }%Loop ends
57
+ \draw [box] (d) -- (e) -- (f) -- (c) -- cycle; %East face of last box
58
+ \draw [band] (d) -- (e) -- (f) -- (c) -- cycle; %East face of last box
59
+ \draw [pic actions] (d) -- (e) -- (f) -- (c) -- cycle; %East face edges of last box
60
+
61
+ \coordinate (a1) at (0 , \y/2 , \z/2);
62
+ \coordinate (b1) at (0 ,-\y/2 , \z/2);
63
+ \tikzstyle{depthlabel}=[pos=0,text width=14*\z,text centered,sloped]
64
+
65
+ \path (c) edge ["\small\zlabels"',depthlabel](f); %depth label
66
+ \path (b1) edge ["\ylabel",midway] (a1); %height label
67
+
68
+ \tikzstyle{captionlabel}=[text width=15*\LastEastx/\scale,text centered]
69
+ \path (\LastEastx/2,-\y/2,+\z/2) + (0,-25pt) coordinate (cap)
70
+ edge ["\textcolor{black}{ \bf \caption}"',captionlabel] (cap); %Block caption/pic object label
71
+
72
+ %Define nodes to be used outside on the pic object
73
+ \coordinate (\name-west) at (0,0,0) ;
74
+ \coordinate (\name-east) at (\LastEastx, 0,0) ;
75
+ \coordinate (\name-north) at (\LastEastx/2,\y/2,0);
76
+ \coordinate (\name-south) at (\LastEastx/2,-\y/2,0);
77
+ \coordinate (\name-anchor) at (\LastEastx/2, 0,0) ;
78
+
79
+ \coordinate (\name-near) at (\LastEastx/2,0,\z/2);
80
+ \coordinate (\name-far) at (\LastEastx/2,0,-\z/2);
81
+
82
+ \coordinate (\name-nearwest) at (0,0,\z/2);
83
+ \coordinate (\name-neareast) at (\LastEastx,0,\z/2);
84
+ \coordinate (\name-farwest) at (0,0,-\z/2);
85
+ \coordinate (\name-fareast) at (\LastEastx,0,-\z/2);
86
+
87
+ \coordinate (\name-northeast) at (\name-north-|\name-east);
88
+ \coordinate (\name-northwest) at (\name-north-|\name-west);
89
+ \coordinate (\name-southeast) at (\name-south-|\name-east);
90
+ \coordinate (\name-southwest) at (\name-south-|\name-west);
91
+
92
+ \coordinate (\name-nearnortheast) at (\LastEastx, \y/2, \z/2);
93
+ \coordinate (\name-farnortheast) at (\LastEastx, \y/2,-\z/2);
94
+ \coordinate (\name-nearsoutheast) at (\LastEastx,-\y/2, \z/2);
95
+ \coordinate (\name-farsoutheast) at (\LastEastx,-\y/2,-\z/2);
96
+
97
+ \coordinate (\name-nearnorthwest) at (0, \y/2, \z/2);
98
+ \coordinate (\name-farnorthwest) at (0, \y/2,-\z/2);
99
+ \coordinate (\name-nearsouthwest) at (0,-\y/2, \z/2);
100
+ \coordinate (\name-farsouthwest) at (0,-\y/2,-\z/2);
101
+ },
102
+ /block/.search also={/tikz},
103
+ /block/.cd,
104
+ width/.store in=\cubex,
105
+ height/.store in=\cubey,
106
+ depth/.store in=\cubez,
107
+ scale/.store in=\scale,
108
+ xlabel/.store in=\boxlabels,
109
+ ylabel/.store in=\ylabel,
110
+ zlabel/.store in=\zlabels,
111
+ caption/.store in=\caption,
112
+ name/.store in=\name,
113
+ fill/.store in=\fill,
114
+ bandfill/.store in=\bandfill,
115
+ opacity/.store in=\opacity,
116
+ bandopacity/.store in=\bandopacity,
117
+ fill={rgb:red,5;green,5;blue,5;white,15},
118
+ bandfill={rgb:red,5;green,5;blue,5;white,5},
119
+ opacity=0.4,
120
+ bandopacity=0.6,
121
+ width=2,
122
+ height=13,
123
+ depth=15,
124
+ scale=.2,
125
+ xlabel={{"","","","","","","","","",""}},
126
+ ylabel=,
127
+ zlabel=,
128
+ caption=,
129
+ name=,
130
+ }
paper/data/ablation.dat ADDED
@@ -0,0 +1,11 @@
 
 
 
 
 
 
 
 
 
 
 
 
1
+ idx name delta lo hi cost ratio
2
+ 0 Full -0.2582 -0.4253 -0.0344 0.0000 0.3063
3
+ 1 NoSplit -0.0186 -0.2354 0.1023 0.2403 0.3032
4
+ 2 NoGain 0.1141 0.0510 0.1940 0.3735 0.2997
5
+ 3 NoClose -0.2480 -0.4222 -0.0278 0.0102 0.3002
6
+ 4 TauLow -0.1046 -0.1388 -0.0853 0.1543 0.3053
7
+ 5 StepOne -0.1629 -0.2696 -0.0282 0.0958 0.3082
8
+ 6 NoEarly -0.1174 -0.2407 -0.0199 0.1413 0.3060
9
+ 7 ExactLast -0.2529 -0.4177 -0.0272 0.0053 0.4893
10
+ 8 Exact -0.0226 -0.1222 0.0703 0.2363 1.0029
11
+ 9 NoEng -0.3045 -0.4253 -0.1733 -0.0463 0.4248
paper/data/ablation_seeds.dat ADDED
@@ -0,0 +1,28 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ idx cost
2
+ 1 0.4184
3
+ 1 0.1117
4
+ 1 0.1908
5
+ 2 0.5103
6
+ 2 0.1318
7
+ 2 0.4784
8
+ 3 0.0208
9
+ 3 0.0066
10
+ 3 0.0031
11
+ 4 0.1765
12
+ 4 -0.0552
13
+ 4 0.3415
14
+ 5 0.2875
15
+ 5 -0.1565
16
+ 5 0.1564
17
+ 6 0.2958
18
+ 6 -0.0573
19
+ 6 0.1854
20
+ 7 0.0009
21
+ 7 0.0073
22
+ 7 0.0077
23
+ 8 0.2997
24
+ 8 0.1048
25
+ 8 0.3044
26
+ 9 -0.0000
27
+ 9 -0.1389
28
+ 9 -0.0000
paper/data/heldout_SS_emeans_approx.dat ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ seed delta
2
+ 10 0.00857
3
+ 11 0.01713
4
+ 12 0.03986
5
+ 13 0.04352
6
+ 14 0.04691
paper/data/heldout_SS_pisces.dat ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ seed delta
2
+ 10 -0.05606
3
+ 11 -0.08451
4
+ 12 -0.04465
5
+ 13 -0.00290
6
+ 14 -0.01142