Spaces:

NagaYu
/

stemma

Running

File size: 16,114 Bytes

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Stemma — model provenance from weights alone</title>
<style>
  :root{
    --bg:#fbfbfa; --fg:#1a1a18; --muted:#6b6b64; --line:#e2e2dc;
    --card:#ffffff; --accent:#2f6f4e; --warn:#8a5a00; --code:#f4f4f0;
    --mono:ui-monospace,SFMono-Regular,"SF Mono",Menlo,Consolas,monospace;
  }
  @media (prefers-color-scheme:dark){
    :root{ --bg:#161714; --fg:#eceae4; --muted:#a3a199; --line:#2e302b;
           --card:#1d1f1b; --accent:#7fc39a; --warn:#e0b25e; --code:#22241f; }
  }
  *{box-sizing:border-box}
  body{margin:0;background:var(--bg);color:var(--fg);
       font:16px/1.65 -apple-system,BlinkMacSystemFont,"Segoe UI",Inter,Roboto,"Helvetica Neue",sans-serif;
       -webkit-font-smoothing:antialiased}
  .wrap{max-width:900px;margin:0 auto;padding:56px 22px 96px}
  h1{font-size:clamp(30px,5vw,44px);line-height:1.12;letter-spacing:-.022em;margin:0 0 6px;font-weight:650}
  h2{font-size:23px;letter-spacing:-.012em;margin:56px 0 14px;font-weight:620;
     padding-bottom:8px;border-bottom:1px solid var(--line)}
  h3{font-size:17px;margin:30px 0 8px;font-weight:620}
  p{margin:0 0 15px}
  a{color:var(--accent);text-underline-offset:2px}
  .lede{font-size:19px;line-height:1.6;color:var(--fg);margin-bottom:22px}
  .muted{color:var(--muted)}
  code{font-family:var(--mono);font-size:.9em;background:var(--code);
       padding:1.5px 5px;border-radius:4px}
  pre{background:var(--code);border:1px solid var(--line);border-radius:10px;
      padding:14px 16px;overflow-x:auto;margin:0 0 16px}
  pre code{background:none;padding:0;font-size:13.5px;line-height:1.55}
  .links{display:flex;flex-wrap:wrap;gap:9px;margin:22px 0 8px}
  .links a{display:inline-block;padding:7px 14px;border:1px solid var(--line);
           border-radius:999px;background:var(--card);text-decoration:none;font-size:14px;font-weight:520}
  .links a:hover{border-color:var(--accent)}
  .tw{overflow-x:auto;margin:0 0 18px;border:1px solid var(--line);border-radius:10px;background:var(--card)}
  table{border-collapse:collapse;width:100%;font-size:14.5px}
  th,td{padding:9px 13px;text-align:left;border-bottom:1px solid var(--line);white-space:nowrap}
  th{font-weight:620;font-size:13px;letter-spacing:.02em;text-transform:uppercase;color:var(--muted)}
  tr:last-child td{border-bottom:none}
  td.n,th.n{text-align:right;font-variant-numeric:tabular-nums}
  .win{color:var(--accent);font-weight:640}
  .bad{color:var(--warn);font-weight:600}
  figure{margin:0 0 26px}
  figure img{width:100%;height:auto;display:block;border:1px solid var(--line);
             border-radius:10px;background:#fff}
  figcaption{font-size:13.5px;color:var(--muted);margin-top:9px;line-height:1.55}
  .note{background:var(--card);border:1px solid var(--line);border-left:3px solid var(--accent);
        border-radius:8px;padding:14px 17px;margin:0 0 18px;font-size:15px}
  .note.warn{border-left-color:var(--warn)}
  ol,ul{margin:0 0 16px;padding-left:22px}
  li{margin-bottom:9px}
  footer{margin-top:60px;padding-top:22px;border-top:1px solid var(--line);
         font-size:13.5px;color:var(--muted)}
  .kicker{font-size:12.5px;letter-spacing:.09em;text-transform:uppercase;
          color:var(--muted);font-weight:640;margin-bottom:10px}
</style>
</head>
<body>
<div class="wrap">

  <div class="kicker">Research prototype · results page</div>
  <h1>Stemma</h1>
  <p class="lede">Recover derivation <strong>direction</strong>, <strong>multi-parent merges</strong>
  and <strong>mixing ratios</strong> from weights alone — reading a few megabytes over HTTP Range
  requests, never a full checkpoint.</p>

  <div class="links">
    <a href="https://github.com/NagaYu/stemma">Code on GitHub</a>
    <a href="https://huggingface.co/NagaYu/stemma-direction">Model</a>
    <a href="https://huggingface.co/datasets/NagaYu/stemma-bench">Dataset</a>
    <a href="https://github.com/NagaYu/stemma/blob/main/docs/FINDINGS.md">Measured findings</a>
  </div>

  <div class="note">
    <strong>This page is static.</strong> The interactive analysis runs locally, not here — a Gradio
    Space needs a Python backend, which Hugging Face gates behind a PRO subscription. Everything
    below is a real measurement, reproducible with the two commands in
    <a href="#run">Run it yourself</a>.
  </div>

  <h2>Why this is different</h2>
  <p>Existing weight-level fingerprints are <strong>symmetric by construction</strong>:
  <code>sim(A,B) == sim(B,A)</code>. That is a fine design for the question they ask, and a hard
  ceiling on the question Stemma asks.</p>

  <div class="tw"><table>
    <tr><th></th><th>Symmetric fingerprints<br>(AWM / REEF / HuRef-style)</th><th>Stemma</th></tr>
    <tr><td>Question</td><td>“Are these two related?”</td><td>“Which came first, from which parents, in what proportion?”</td></tr>
    <tr><td>Direction</td><td class="bad">not expressible — 50% by construction</td><td class="win">signed log-likelihood ratio, with abstention</td></tr>
    <tr><td>Multi-parent merges</td><td class="bad">not expressible</td><td class="win">sparse non-negative decomposition</td></tr>
    <tr><td>Mixing ratios</td><td class="bad">not expressible</td><td class="win">recovered coefficients</td></tr>
    <tr><td>Failure mode</td><td class="bad">false “related” on architecture twins</td><td class="win">abstains rather than guessing</td></tr>
  </table></div>

  <h2>Measured results</h2>
  <p class="muted">20 real checkpoints built with actual fine-tuning, quantisation, pruning and
  merging — 190 labelled pairs, of which <strong>25 are hard same-architecture / different-seed
  controls</strong>. <code>seed=0</code>.</p>

  <div class="tw"><table>
    <tr><th>Method</th><th class="n">AUC</th><th class="n">FPR@95TPR</th><th class="n">FPR hard controls</th><th>Direction</th><th class="n">Merge F1</th><th class="n">Mixing MAE</th></tr>
    <tr><td><strong>Stemma</strong></td><td class="n win">0.994</td><td class="n">0.000</td><td class="n win">0.000</td><td class="win">100% on answered</td><td class="n win">0.867</td><td class="n win">0.070</td></tr>
    <tr><td>cosine</td><td class="n">0.987</td><td class="n">0.000</td><td class="n">0.000</td><td class="bad">50% — structural</td><td class="n">n/a</td><td class="n">n/a</td></tr>
    <tr><td>CKA / REEF-style</td><td class="n">0.987</td><td class="n">0.000</td><td class="n">0.000</td><td class="bad">50% — structural</td><td class="n">n/a</td><td class="n">n/a</td></tr>
    <tr><td>HuRef-style</td><td class="n">0.981</td><td class="n">0.000</td><td class="n">0.000</td><td class="bad">50% — structural</td><td class="n">n/a</td><td class="n">n/a</td></tr>
  </table></div>
  <p class="muted"><code>n/a</code> is not zero: a symmetric fingerprint produces <em>no mixing
  coefficients at all</em>, so there is nothing to score. <code>50%</code> is a structural ceiling,
  not a tuning failure.</p>

  <h3>Direction, split by relation — the honest view</h3>
  <p>An aggregate would let the easy scar-bearing edges hide the hard scar-free ones, so the harness
  refuses to report one.</p>

  <div class="tw"><table>
    <tr><th>Relation</th><th>Group</th><th class="n">n</th><th class="n">Accuracy</th><th class="n">Abstained</th><th class="n">mean |llr|</th></tr>
    <tr><td>quantisation</td><td>scar-bearing</td><td class="n">3</td><td class="n win">100.0%</td><td class="n">0.0%</td><td class="n">2.80</td></tr>
    <tr><td>pruning</td><td>scar-bearing</td><td class="n">2</td><td class="n win">100.0%</td><td class="n">0.0%</td><td class="n">2.65</td></tr>
    <tr><td>vocab extension</td><td>scar-bearing</td><td class="n">2</td><td class="n win">100.0%</td><td class="n">0.0%</td><td class="n">4.98</td></tr>
    <tr><td>SFT</td><td>scar-free</td><td class="n">1</td><td class="n">0.0%</td><td class="n">100%</td><td class="n">0.02</td></tr>
    <tr><td>LoRA</td><td>scar-free</td><td class="n">1</td><td class="n">0.0%</td><td class="n">100%</td><td class="n">0.01</td></tr>
    <tr><td>continued pretrain</td><td>scar-free</td><td class="n">1</td><td class="n">0.0%</td><td class="n">100%</td><td class="n">0.02</td></tr>
  </table></div>
  <p>Direction is near-deterministic exactly where an operation is <strong>lossy and
  irreversible</strong> — you cannot un-quantise, un-prune, or un-extend a vocabulary, so the scar
  can only ever appear downstream. Where nothing lossy happened, Stemma <strong>abstains rather
  than guessing</strong>.</p>

  <h3>Merge recovery</h3>
  <div class="tw"><table>
    <tr><th>Slice</th><th class="n">n</th><th class="n">Precision</th><th class="n">Recall</th><th class="n">F1</th><th class="n">Mixing MAE</th></tr>
    <tr><td>all</td><td class="n">4</td><td class="n win">1.000</td><td class="n">0.792</td><td class="n">0.867</td><td class="n">0.070</td></tr>
    <tr><td>DARE</td><td class="n">1</td><td class="n">1.000</td><td class="n">1.000</td><td class="n win">1.000</td><td class="n win">0.0004</td></tr>
    <tr><td>SLERP</td><td class="n">1</td><td class="n">1.000</td><td class="n">1.000</td><td class="n win">1.000</td><td class="n">0.027</td></tr>
    <tr><td>TIES</td><td class="n">2</td><td class="n">1.000</td><td class="n">0.583</td><td class="n">0.733</td><td class="n">0.126</td></tr>
  </table></div>
  <p><strong>Precision is 1.000 — no false parent at all.</strong> That is deliberately bought with
  recall: for a provenance tool a false parent asserts something about a model that had nothing to
  do with the child, which is worse than a miss.</p>

  <h3>Transfer cost — the reduction grows with model size</h3>
  <div class="tw"><table>
    <tr><th>Model</th><th class="n">Checkpoint</th><th class="n">Header only</th><th class="n">Full sketch</th><th class="n">Reduction</th></tr>
    <tr><td>SmolLM2-135M-Instruct</td><td class="n">269 MB</td><td class="n">31,397 B (0.012%)</td><td class="n">17.1 MB (6.34%)</td><td class="n">16×</td></tr>
    <tr><td><strong>Qwen2.5-7B-Instruct</strong></td><td class="n">15.2 GB</td><td class="n win">27,752 B (0.0002%)</td><td class="n win">98.1 MB (0.644%)</td><td class="n win">155×</td></tr>
  </table></div>
  <p>Sampling cost is fixed while checkpoints grow, so the ratio improves with scale. Both figures
  are live HTTP Range reads against the public Hub; nothing was downloaded.</p>

  <h2>Figures</h2>

  <figure>
    <img src="figures/fig6_direction_by_relation.png" alt="Direction accuracy split by ground-truth relation, scar-bearing versus scar-free">
    <figcaption><strong>The figure to read.</strong> Direction accuracy per relation, with the
    scar-bearing group separated from the scar-free one. The aggregate bar in fig. 1 is the average
    of these.</figcaption>
  </figure>

  <figure>
    <img src="figures/fig5_summary_matrix.png" alt="Capability matrix: methods against capabilities">
    <figcaption>Capability matrix. The crosses in the direction, multi-parent and mixing-ratio
    columns are <em>structural</em>: those statistics are symmetric functions of an unordered pair,
    so the questions are not merely hard for them — they are unanswerable.</figcaption>
  </figure>

  <figure>
    <img src="figures/fig1_direction_accuracy.png" alt="Direction accuracy per method against the 50% chance line">
    <figcaption>Every baseline sits exactly on the 0.50 chance line, because cosine/CKA/HuRef
    statistics are symmetric in their two arguments. No amount of tuning moves them.</figcaption>
  </figure>

  <figure>
    <img src="figures/fig3_merge_recovery.png" alt="Recovered versus true mixing coefficients">
    <figcaption>Recovered vs. true mixing coefficient for every candidate of every ground-truth
    merge. Points on the <em>y = x</em> line are correctly weighted parents; points on the x-axis
    are decoys. No symmetric baseline can produce a single point on this plot.</figcaption>
  </figure>

  <figure>
    <img src="figures/fig2_roc.png" alt="ROC curves for relatedness detection">
    <figcaption>Relatedness ROC. This axis is where the symmetric baselines are genuinely
    competitive — the separation appears only in the questions above.</figcaption>
  </figure>

  <figure>
    <img src="figures/fig4_transfer.png" alt="Bytes per decision, log scale">
    <figcaption>Bytes per decision, log scale. The full-download bar is read from the safetensors
    header, never downloaded.</figcaption>
  </figure>

  <h2>What weight geometry cannot do</h2>
  <p>These limits were measured, not assumed, and they bound how the results above should be read.</p>

  <ol>
    <li><strong>Direction is near-deterministic only for lossy operations.</strong> Scar-free
    SFT/LoRA edges are weakly identifiable from two models alone, and the estimator abstains.</li>

    <li><strong>Norm growth is recipe-dependent and cannot be a prior.</strong> Measured over 8
    shared tensors: <code>log‖B‖−log‖A‖</code> was <strong>−0.0171 (0/8 positive)</strong> for
    Qwen2.5-0.5B → Instruct but <strong>+0.0113 (8/8 positive)</strong> for SmolLM2-135M → Instruct.
    Both are unambiguously base → instruct-tuned. A hand-set sign would have been right on one
    family and wrong on the other.</li>

    <li><strong>Outgroup rooting is invalid for merge children.</strong> Rooting assumes descendants
    drift monotonically away from the root, but merging is a <em>contraction toward the centroid</em>:
    <code>0.6·sft + 0.4·cpt</code> partly cancels two perturbations and lands <strong>closer to the
    root than either parent</strong> (root→sft 0.000820, root→cpt 0.001610, root→merge
    <strong>0.000678</strong>). Every correctly chosen sibling outgroup then pushes the answer the
    wrong way. Direction for a merged model must come from the decomposition, not distance
    geometry.</li>

    <li><strong>Fitting the combiner lost to hand-set priors.</strong> On the same held-out split the
    priors scored <strong>1.000</strong> accuracy on decided pairs against the fit's
    <strong>0.500</strong> — chance. With 13 features and 21 training pairs the problem is
    underdetermined, and the fit gave the quantisation-lattice feature a <em>negative</em> weight,
    asserting that the quantised model is the parent. That is physically impossible.</li>

    <li><strong>End-to-end DAG reconstruction is weaker than the pairwise numbers.</strong> The
    benchmark scores pairwise decisions; whole-graph accuracy is not yet scored, and
    <code>trace</code> output should be read as ranked hypotheses for a human.</li>
  </ol>

  <h2 id="run">Run it yourself</h2>
  <pre><code>pip install "git+https://github.com/NagaYu/stemma"

# Which of these two came first?
stemma direction Qwen/Qwen2.5-0.5B Qwen/Qwen2.5-0.5B-Instruct

# Recover merge parents and mixing ratios
stemma decompose org/merged --candidates org/a org/b org/c --base org/base

# Full lineage + licence propagation + AI-BOM
stemma trace org/model --universe universe.txt --out bom.json

# The interactive UI, locally
pip install "git+https://github.com/NagaYu/stemma#egg=stemma[app]" &amp;&amp; python app.py</code></pre>
  <p class="muted">Nothing here requires a Hugging Face account or token — the direction priors ship
  inside the package, and it runs with <code>HF_HUB_OFFLINE=1</code>.</p>

  <div class="note warn">
    <strong>Scope and ethics.</strong> Stemma reports <strong>statistical evidence with a
    confidence</strong>, and never a determination of infringement or licence non-compliance.
    Weight-level similarity and derivation direction are inferences from a small sample of tensors
    and can be wrong. Nothing here establishes provenance as fact.
    <strong>A human must review every finding before any action is taken.</strong>
  </div>

  <footer>
    Apache-2.0 · Numbers regenerate with <code>python scripts/build_bench.py</code> then
    <code>python benchmarks/run.py</code> ·
    <a href="https://github.com/NagaYu/stemma">github.com/NagaYu/stemma</a>
  </footer>

</div>
</body>
</html>