| <html lang="en"> | |
| <head> | |
| <meta charset="utf-8"> | |
| <meta name="viewport" content="width=device-width, initial-scale=1"> | |
| <title>Stemma — model provenance from weights alone</title> | |
| <style> | |
| :root{ | |
| --bg:#fbfbfa; --fg:#1a1a18; --muted:#6b6b64; --line:#e2e2dc; | |
| --card:#ffffff; --accent:#2f6f4e; --warn:#8a5a00; --code:#f4f4f0; | |
| --mono:ui-monospace,SFMono-Regular,"SF Mono",Menlo,Consolas,monospace; | |
| } | |
| @media (prefers-color-scheme:dark){ | |
| :root{ --bg:#161714; --fg:#eceae4; --muted:#a3a199; --line:#2e302b; | |
| --card:#1d1f1b; --accent:#7fc39a; --warn:#e0b25e; --code:#22241f; } | |
| } | |
| *{box-sizing:border-box} | |
| body{margin:0;background:var(--bg);color:var(--fg); | |
| font:16px/1.65 -apple-system,BlinkMacSystemFont,"Segoe UI",Inter,Roboto,"Helvetica Neue",sans-serif; | |
| -webkit-font-smoothing:antialiased} | |
| .wrap{max-width:900px;margin:0 auto;padding:56px 22px 96px} | |
| h1{font-size:clamp(30px,5vw,44px);line-height:1.12;letter-spacing:-.022em;margin:0 0 6px;font-weight:650} | |
| h2{font-size:23px;letter-spacing:-.012em;margin:56px 0 14px;font-weight:620; | |
| padding-bottom:8px;border-bottom:1px solid var(--line)} | |
| h3{font-size:17px;margin:30px 0 8px;font-weight:620} | |
| p{margin:0 0 15px} | |
| a{color:var(--accent);text-underline-offset:2px} | |
| .lede{font-size:19px;line-height:1.6;color:var(--fg);margin-bottom:22px} | |
| .muted{color:var(--muted)} | |
| code{font-family:var(--mono);font-size:.9em;background:var(--code); | |
| padding:1.5px 5px;border-radius:4px} | |
| pre{background:var(--code);border:1px solid var(--line);border-radius:10px; | |
| padding:14px 16px;overflow-x:auto;margin:0 0 16px} | |
| pre code{background:none;padding:0;font-size:13.5px;line-height:1.55} | |
| .links{display:flex;flex-wrap:wrap;gap:9px;margin:22px 0 8px} | |
| .links a{display:inline-block;padding:7px 14px;border:1px solid var(--line); | |
| border-radius:999px;background:var(--card);text-decoration:none;font-size:14px;font-weight:520} | |
| .links a:hover{border-color:var(--accent)} | |
| .tw{overflow-x:auto;margin:0 0 18px;border:1px solid var(--line);border-radius:10px;background:var(--card)} | |
| table{border-collapse:collapse;width:100%;font-size:14.5px} | |
| th,td{padding:9px 13px;text-align:left;border-bottom:1px solid var(--line);white-space:nowrap} | |
| th{font-weight:620;font-size:13px;letter-spacing:.02em;text-transform:uppercase;color:var(--muted)} | |
| tr:last-child td{border-bottom:none} | |
| td.n,th.n{text-align:right;font-variant-numeric:tabular-nums} | |
| .win{color:var(--accent);font-weight:640} | |
| .bad{color:var(--warn);font-weight:600} | |
| figure{margin:0 0 26px} | |
| figure img{width:100%;height:auto;display:block;border:1px solid var(--line); | |
| border-radius:10px;background:#fff} | |
| figcaption{font-size:13.5px;color:var(--muted);margin-top:9px;line-height:1.55} | |
| .note{background:var(--card);border:1px solid var(--line);border-left:3px solid var(--accent); | |
| border-radius:8px;padding:14px 17px;margin:0 0 18px;font-size:15px} | |
| .note.warn{border-left-color:var(--warn)} | |
| ol,ul{margin:0 0 16px;padding-left:22px} | |
| li{margin-bottom:9px} | |
| footer{margin-top:60px;padding-top:22px;border-top:1px solid var(--line); | |
| font-size:13.5px;color:var(--muted)} | |
| .kicker{font-size:12.5px;letter-spacing:.09em;text-transform:uppercase; | |
| color:var(--muted);font-weight:640;margin-bottom:10px} | |
| </style> | |
| </head> | |
| <body> | |
| <div class="wrap"> | |
| <div class="kicker">Research prototype · results page</div> | |
| <h1>Stemma</h1> | |
| <p class="lede">Recover derivation <strong>direction</strong>, <strong>multi-parent merges</strong> | |
| and <strong>mixing ratios</strong> from weights alone — reading a few megabytes over HTTP Range | |
| requests, never a full checkpoint.</p> | |
| <div class="links"> | |
| <a href="https://github.com/NagaYu/stemma">Code on GitHub</a> | |
| <a href="https://huggingface.co/NagaYu/stemma-direction">Model</a> | |
| <a href="https://huggingface.co/datasets/NagaYu/stemma-bench">Dataset</a> | |
| <a href="https://github.com/NagaYu/stemma/blob/main/docs/FINDINGS.md">Measured findings</a> | |
| </div> | |
| <div class="note"> | |
| <strong>This page is static.</strong> The interactive analysis runs locally, not here — a Gradio | |
| Space needs a Python backend, which Hugging Face gates behind a PRO subscription. Everything | |
| below is a real measurement, reproducible with the two commands in | |
| <a href="#run">Run it yourself</a>. | |
| </div> | |
| <h2>Why this is different</h2> | |
| <p>Existing weight-level fingerprints are <strong>symmetric by construction</strong>: | |
| <code>sim(A,B) == sim(B,A)</code>. That is a fine design for the question they ask, and a hard | |
| ceiling on the question Stemma asks.</p> | |
| <div class="tw"><table> | |
| <tr><th></th><th>Symmetric fingerprints<br>(AWM / REEF / HuRef-style)</th><th>Stemma</th></tr> | |
| <tr><td>Question</td><td>“Are these two related?”</td><td>“Which came first, from which parents, in what proportion?”</td></tr> | |
| <tr><td>Direction</td><td class="bad">not expressible — 50% by construction</td><td class="win">signed log-likelihood ratio, with abstention</td></tr> | |
| <tr><td>Multi-parent merges</td><td class="bad">not expressible</td><td class="win">sparse non-negative decomposition</td></tr> | |
| <tr><td>Mixing ratios</td><td class="bad">not expressible</td><td class="win">recovered coefficients</td></tr> | |
| <tr><td>Failure mode</td><td class="bad">false “related” on architecture twins</td><td class="win">abstains rather than guessing</td></tr> | |
| </table></div> | |
| <h2>Measured results</h2> | |
| <p class="muted">20 real checkpoints built with actual fine-tuning, quantisation, pruning and | |
| merging — 190 labelled pairs, of which <strong>25 are hard same-architecture / different-seed | |
| controls</strong>. <code>seed=0</code>.</p> | |
| <div class="tw"><table> | |
| <tr><th>Method</th><th class="n">AUC</th><th class="n">FPR@95TPR</th><th class="n">FPR hard controls</th><th>Direction</th><th class="n">Merge F1</th><th class="n">Mixing MAE</th></tr> | |
| <tr><td><strong>Stemma</strong></td><td class="n win">0.994</td><td class="n">0.000</td><td class="n win">0.000</td><td class="win">100% on answered</td><td class="n win">0.867</td><td class="n win">0.070</td></tr> | |
| <tr><td>cosine</td><td class="n">0.987</td><td class="n">0.000</td><td class="n">0.000</td><td class="bad">50% — structural</td><td class="n">n/a</td><td class="n">n/a</td></tr> | |
| <tr><td>CKA / REEF-style</td><td class="n">0.987</td><td class="n">0.000</td><td class="n">0.000</td><td class="bad">50% — structural</td><td class="n">n/a</td><td class="n">n/a</td></tr> | |
| <tr><td>HuRef-style</td><td class="n">0.981</td><td class="n">0.000</td><td class="n">0.000</td><td class="bad">50% — structural</td><td class="n">n/a</td><td class="n">n/a</td></tr> | |
| </table></div> | |
| <p class="muted"><code>n/a</code> is not zero: a symmetric fingerprint produces <em>no mixing | |
| coefficients at all</em>, so there is nothing to score. <code>50%</code> is a structural ceiling, | |
| not a tuning failure.</p> | |
| <h3>Direction, split by relation — the honest view</h3> | |
| <p>An aggregate would let the easy scar-bearing edges hide the hard scar-free ones, so the harness | |
| refuses to report one.</p> | |
| <div class="tw"><table> | |
| <tr><th>Relation</th><th>Group</th><th class="n">n</th><th class="n">Accuracy</th><th class="n">Abstained</th><th class="n">mean |llr|</th></tr> | |
| <tr><td>quantisation</td><td>scar-bearing</td><td class="n">3</td><td class="n win">100.0%</td><td class="n">0.0%</td><td class="n">2.80</td></tr> | |
| <tr><td>pruning</td><td>scar-bearing</td><td class="n">2</td><td class="n win">100.0%</td><td class="n">0.0%</td><td class="n">2.65</td></tr> | |
| <tr><td>vocab extension</td><td>scar-bearing</td><td class="n">2</td><td class="n win">100.0%</td><td class="n">0.0%</td><td class="n">4.98</td></tr> | |
| <tr><td>SFT</td><td>scar-free</td><td class="n">1</td><td class="n">0.0%</td><td class="n">100%</td><td class="n">0.02</td></tr> | |
| <tr><td>LoRA</td><td>scar-free</td><td class="n">1</td><td class="n">0.0%</td><td class="n">100%</td><td class="n">0.01</td></tr> | |
| <tr><td>continued pretrain</td><td>scar-free</td><td class="n">1</td><td class="n">0.0%</td><td class="n">100%</td><td class="n">0.02</td></tr> | |
| </table></div> | |
| <p>Direction is near-deterministic exactly where an operation is <strong>lossy and | |
| irreversible</strong> — you cannot un-quantise, un-prune, or un-extend a vocabulary, so the scar | |
| can only ever appear downstream. Where nothing lossy happened, Stemma <strong>abstains rather | |
| than guessing</strong>.</p> | |
| <h3>Merge recovery</h3> | |
| <div class="tw"><table> | |
| <tr><th>Slice</th><th class="n">n</th><th class="n">Precision</th><th class="n">Recall</th><th class="n">F1</th><th class="n">Mixing MAE</th></tr> | |
| <tr><td>all</td><td class="n">4</td><td class="n win">1.000</td><td class="n">0.792</td><td class="n">0.867</td><td class="n">0.070</td></tr> | |
| <tr><td>DARE</td><td class="n">1</td><td class="n">1.000</td><td class="n">1.000</td><td class="n win">1.000</td><td class="n win">0.0004</td></tr> | |
| <tr><td>SLERP</td><td class="n">1</td><td class="n">1.000</td><td class="n">1.000</td><td class="n win">1.000</td><td class="n">0.027</td></tr> | |
| <tr><td>TIES</td><td class="n">2</td><td class="n">1.000</td><td class="n">0.583</td><td class="n">0.733</td><td class="n">0.126</td></tr> | |
| </table></div> | |
| <p><strong>Precision is 1.000 — no false parent at all.</strong> That is deliberately bought with | |
| recall: for a provenance tool a false parent asserts something about a model that had nothing to | |
| do with the child, which is worse than a miss.</p> | |
| <h3>Transfer cost — the reduction grows with model size</h3> | |
| <div class="tw"><table> | |
| <tr><th>Model</th><th class="n">Checkpoint</th><th class="n">Header only</th><th class="n">Full sketch</th><th class="n">Reduction</th></tr> | |
| <tr><td>SmolLM2-135M-Instruct</td><td class="n">269 MB</td><td class="n">31,397 B (0.012%)</td><td class="n">17.1 MB (6.34%)</td><td class="n">16×</td></tr> | |
| <tr><td><strong>Qwen2.5-7B-Instruct</strong></td><td class="n">15.2 GB</td><td class="n win">27,752 B (0.0002%)</td><td class="n win">98.1 MB (0.644%)</td><td class="n win">155×</td></tr> | |
| </table></div> | |
| <p>Sampling cost is fixed while checkpoints grow, so the ratio improves with scale. Both figures | |
| are live HTTP Range reads against the public Hub; nothing was downloaded.</p> | |
| <h2>Figures</h2> | |
| <figure> | |
| <img src="figures/fig6_direction_by_relation.png" alt="Direction accuracy split by ground-truth relation, scar-bearing versus scar-free"> | |
| <figcaption><strong>The figure to read.</strong> Direction accuracy per relation, with the | |
| scar-bearing group separated from the scar-free one. The aggregate bar in fig. 1 is the average | |
| of these.</figcaption> | |
| </figure> | |
| <figure> | |
| <img src="figures/fig5_summary_matrix.png" alt="Capability matrix: methods against capabilities"> | |
| <figcaption>Capability matrix. The crosses in the direction, multi-parent and mixing-ratio | |
| columns are <em>structural</em>: those statistics are symmetric functions of an unordered pair, | |
| so the questions are not merely hard for them — they are unanswerable.</figcaption> | |
| </figure> | |
| <figure> | |
| <img src="figures/fig1_direction_accuracy.png" alt="Direction accuracy per method against the 50% chance line"> | |
| <figcaption>Every baseline sits exactly on the 0.50 chance line, because cosine/CKA/HuRef | |
| statistics are symmetric in their two arguments. No amount of tuning moves them.</figcaption> | |
| </figure> | |
| <figure> | |
| <img src="figures/fig3_merge_recovery.png" alt="Recovered versus true mixing coefficients"> | |
| <figcaption>Recovered vs. true mixing coefficient for every candidate of every ground-truth | |
| merge. Points on the <em>y = x</em> line are correctly weighted parents; points on the x-axis | |
| are decoys. No symmetric baseline can produce a single point on this plot.</figcaption> | |
| </figure> | |
| <figure> | |
| <img src="figures/fig2_roc.png" alt="ROC curves for relatedness detection"> | |
| <figcaption>Relatedness ROC. This axis is where the symmetric baselines are genuinely | |
| competitive — the separation appears only in the questions above.</figcaption> | |
| </figure> | |
| <figure> | |
| <img src="figures/fig4_transfer.png" alt="Bytes per decision, log scale"> | |
| <figcaption>Bytes per decision, log scale. The full-download bar is read from the safetensors | |
| header, never downloaded.</figcaption> | |
| </figure> | |
| <h2>What weight geometry cannot do</h2> | |
| <p>These limits were measured, not assumed, and they bound how the results above should be read.</p> | |
| <ol> | |
| <li><strong>Direction is near-deterministic only for lossy operations.</strong> Scar-free | |
| SFT/LoRA edges are weakly identifiable from two models alone, and the estimator abstains.</li> | |
| <li><strong>Norm growth is recipe-dependent and cannot be a prior.</strong> Measured over 8 | |
| shared tensors: <code>log‖B‖−log‖A‖</code> was <strong>−0.0171 (0/8 positive)</strong> for | |
| Qwen2.5-0.5B → Instruct but <strong>+0.0113 (8/8 positive)</strong> for SmolLM2-135M → Instruct. | |
| Both are unambiguously base → instruct-tuned. A hand-set sign would have been right on one | |
| family and wrong on the other.</li> | |
| <li><strong>Outgroup rooting is invalid for merge children.</strong> Rooting assumes descendants | |
| drift monotonically away from the root, but merging is a <em>contraction toward the centroid</em>: | |
| <code>0.6·sft + 0.4·cpt</code> partly cancels two perturbations and lands <strong>closer to the | |
| root than either parent</strong> (root→sft 0.000820, root→cpt 0.001610, root→merge | |
| <strong>0.000678</strong>). Every correctly chosen sibling outgroup then pushes the answer the | |
| wrong way. Direction for a merged model must come from the decomposition, not distance | |
| geometry.</li> | |
| <li><strong>Fitting the combiner lost to hand-set priors.</strong> On the same held-out split the | |
| priors scored <strong>1.000</strong> accuracy on decided pairs against the fit's | |
| <strong>0.500</strong> — chance. With 13 features and 21 training pairs the problem is | |
| underdetermined, and the fit gave the quantisation-lattice feature a <em>negative</em> weight, | |
| asserting that the quantised model is the parent. That is physically impossible.</li> | |
| <li><strong>End-to-end DAG reconstruction is weaker than the pairwise numbers.</strong> The | |
| benchmark scores pairwise decisions; whole-graph accuracy is not yet scored, and | |
| <code>trace</code> output should be read as ranked hypotheses for a human.</li> | |
| </ol> | |
| <h2 id="run">Run it yourself</h2> | |
| <pre><code>pip install "git+https://github.com/NagaYu/stemma" | |
| # Which of these two came first? | |
| stemma direction Qwen/Qwen2.5-0.5B Qwen/Qwen2.5-0.5B-Instruct | |
| # Recover merge parents and mixing ratios | |
| stemma decompose org/merged --candidates org/a org/b org/c --base org/base | |
| # Full lineage + licence propagation + AI-BOM | |
| stemma trace org/model --universe universe.txt --out bom.json | |
| # The interactive UI, locally | |
| pip install "git+https://github.com/NagaYu/stemma#egg=stemma[app]" && python app.py</code></pre> | |
| <p class="muted">Nothing here requires a Hugging Face account or token — the direction priors ship | |
| inside the package, and it runs with <code>HF_HUB_OFFLINE=1</code>.</p> | |
| <div class="note warn"> | |
| <strong>Scope and ethics.</strong> Stemma reports <strong>statistical evidence with a | |
| confidence</strong>, and never a determination of infringement or licence non-compliance. | |
| Weight-level similarity and derivation direction are inferences from a small sample of tensors | |
| and can be wrong. Nothing here establishes provenance as fact. | |
| <strong>A human must review every finding before any action is taken.</strong> | |
| </div> | |
| <footer> | |
| Apache-2.0 · Numbers regenerate with <code>python scripts/build_bench.py</code> then | |
| <code>python benchmarks/run.py</code> · | |
| <a href="https://github.com/NagaYu/stemma">github.com/NagaYu/stemma</a> | |
| </footer> | |
| </div> | |
| </body> | |
| </html> | |