README.md

---
license: apache-2.0
library_name: stemma
pretty_name: Stemma direction model
tags:
  - model-provenance
  - lineage
  - safetensors
  - ai-bom
  - model-merging
  - supply-chain
  - not-a-language-model
---

# Stemma direction model

This repository does **not** contain a language model. It contains the small fitted artifacts
that the [Stemma](https://github.com/<user>/stemma) provenance tool loads at runtime:

| File | What it is |
|---|---|
| `direction_model.json` | The fitted `DirectionModel`: `weights` over `stemma.types.DIRECTION_FEATURES`, `bias`, `feature_names`, `scaler_mean`, `scaler_scale`. A regularised logistic combiner — a few dozen floats. |
| `sketch_config.json` | The frozen sketch coordinate system: `version` (`SKETCH_VERSION`), `ROLES`, `DEPTH_BUCKETS`, `FEATURES_PER_SLOT`, `SKETCH_DIM`. A sketch computed under a different config is not comparable. |
| `sketch_index.npz` + `sketch_index.json` | A prebuilt `stemma.phylogeny.SketchIndex` over the benchmark universe: L2-normalisable float32 sketch vectors, presence masks, model ids and per-model metadata. |
| `fit_report.json` | Train/test sizes, held-out accuracy, per-relation accuracy and abstention rate from the fit that produced `direction_model.json`. |

Everything here is generated by `python scripts/push_model.py --repo-id <org>/<name> --bench-dir bench_models --fit`.

## Intended use

**In scope.** Loading into Stemma to (1) orient a candidate parent/child pair, (2) retrieve
candidate parents for a model cheaply from a prebuilt index, (3) produce an AI Bill of Materials
that a human then reviews.

```python
from stemma.direction import DirectionModel, estimate_direction
from stemma.phylogeny import SketchIndex

model  = DirectionModel.load("direction_model.json")
index  = SketchIndex.load("sketch_index")
verdict = estimate_direction("org/a", "org/b", model=model)
```

**Out of scope.** Any automated enforcement, takedown, publication-blocking or procurement
decision. Any use as a legal determination of license compliance or infringement. Any claim that a
model "is" a derivative of another — Stemma reports how consistent the weights are with a
direction of derivation, at a stated confidence, from a small sample of tensors.

## Generation procedure

1. **Benchmark construction.** `scripts/build_bench.py` builds real safetensors checkpoints with
   known lineages — fine-tunes, quantisations, prunings, vocabulary extensions, linear/TIES/DARE
   merges with known mixing ratios, plus unrelated negatives — and writes `ground_truth.json` with
   `models`, `edges` and labelled ordered `pairs`.
2. **Feature extraction.** For each labelled ordered pair `(a, b)`,
   `direction.collect_pair_evidence` Range-reads a handful of shared tensors and
   `direction.direction_features` reduces them to the antisymmetric feature vector named by
   `stemma.types.DIRECTION_FEATURES`. The features are antisymmetric by construction:
   `f(b, a) == -f(a, b)` to within 1e-6, so the fitted combiner cannot learn a positional bias.
3. **Fitting.** `DirectionModel.fit(X, y, l2=...)` on a seeded, deterministic train/test split
   (default seed 0, 25% held out). `y = +1` when `a` is the parent. Because the features are
   antisymmetric, each pair is also usable in its mirrored form; the split is done over *pairs*,
   not over rows, so a pair and its mirror never straddle the split.
4. **Index build.** Every model in the benchmark universe is sketched once
   (`sketch.sketch_model`) and the resulting vectors are stored in a `SketchIndex`.
5. **Packaging.** `scripts/push_model.py` writes the four files above plus this card and uploads
   with `HfApi.create_repo(exist_ok=True)` + `upload_folder`. The script is **dry-run by default**:
   without `--push` it prints exactly what would be uploaded and uploads nothing.

Determinism: every randomised step takes an explicit `seed` (default `0`).

## Evaluation

Held-out accuracy, per-relation accuracy and abstention rate are written to `fit_report.json` at
fit time and mirrored into the repo's README frontmatter-free body by `scripts/push_model.py`. The
full benchmark — relatedness AUC and FPR@95TPR, direction accuracy against the symmetric
baselines, merge parent-set F1 and mixing MAE, bytes- and seconds-per-decision — is produced by
`python benchmarks/run.py` and lives in `benchmarks/results.json`.

No evaluation numbers are quoted in this card. Numbers belong in the generated `fit_report.json`
and `benchmarks/results.json`, so that nothing here can drift away from what was actually measured.

Reporting rules the harness enforces (from `docs/FINDINGS.md`): accuracy is reported **per relation
type**, abstention is reported alongside accuracy-on-non-abstained, cross-architecture distillation
is scored rather than excluded, and the symmetric baselines' 50% direction score is labelled a
structural ceiling rather than a tuning failure.

## Limitations

These are measured, not hypothetical; see `docs/FINDINGS.md`.

- **Direction is near-deterministic only for lossy operations.** Quantisation and pruning scars and
  vocabulary extension are strong because they are irreversible. Scar-free SFT / LoRA / continued
  pretraining is **weakly identifiable from two models alone** and relies on outgroup rooting,
  which needs a usable third relative in the candidate universe.
- **Norm growth is recipe-dependent and sign-flips across families.** Measured:
  `log‖B‖_F − log‖A‖_F` = −0.0171 (**0/8** tensors positive) for `Qwen2.5-0.5B → -Instruct`, and
  +0.0113 (**8/8** positive) for `SmolLM2-135M → -Instruct`, though both pairs are unambiguously
  base → instruct-tuned. `norm_growth_asym` is therefore a *fitted* feature with a small weight and
  never a hand-set sign.
- **The Δ-subspace signal is small (~10⁻³–10⁻²) and signed opposite to the naive intuition.** It is
  a tiebreaker, not a decisive feature.
- **Cross-architecture distillation leaves little weight-level signal.**
- **TIES / DARE are non-linear**, so mixing ratios recovered by the linear decomposer are
  approximate; the residual is reported so the mismatch is visible.
- **Fitted on a benchmark, not on the wild.** The coefficients reflect the operations and model
  families `build_bench.py` produced. Expect degradation on architectures, quantisation formats or
  merge recipes outside that distribution, and refit rather than assuming transfer.
- **Sampled evidence.** Sketches read at most two tensors per (role, depth) slot with sub-sampled
  rows. An adversary who knows the sampling scheme can evade it. This is an auditing aid, not a
  tamper-proof watermark.
- **Version-locked.** A `direction_model.json` is only valid against the `DIRECTION_FEATURES` order
  it was fitted with, and a `SketchIndex` only against its `SKETCH_VERSION`. Both are recorded in
  the artifacts and checked on load.

## Ethics and framing

Stemma reports **statistical evidence** about weight-level similarity and derivation direction,
with a confidence attached to every edge. It does **not** establish provenance as fact and does
**not** constitute a legal determination of license compliance or infringement. A human must
review every finding before any action is taken. Abstention (`direction="unknown"`) is a correct
and expected outcome; a confident wrong answer is worse than no answer here.

## License

Apache-2.0. The artifacts are derived from statistics computed over third-party checkpoints;
Stemma does not redistribute any third-party weights, and those checkpoints' own licenses continue
to govern them.

---

## This build

- Repository: `NagaYu/stemma-direction`
- Generated (UTC): `2026-08-08T13:24:27Z`
- Sketch format: `stemma-sketch-v1` (dim 1456)
- Direction weights: **hand-set priors** (`DirectionModel.default()`), **not fitted**,
  and that is a deliberate, measured choice rather than a missing step.

### Why the priors and not a fit

Fitting an L2 logistic combiner on the benchmark's labelled ordered pairs was tried
and **lost**. On the same held-out split the priors scored **1.000** accuracy on
decided pairs against the fit's **0.500** — chance. Read that with its sample size:
the priors abstained on 5 of 7 and decided only 2, so the accuracy gap rests on 2
decisions against 4 and is suggestive, not conclusive.

The decisive evidence is *what the fit learned*. With 13 features and 21 training
pairs the problem is underdetermined, and the fit assigned `lattice_asym` a
**negative** weight — asserting that the quantised model is the parent. That is
physically impossible: dequantisation cannot restore what rounding destroyed, so the
scar can only ever appear downstream. It also put its largest weight on the statistic
already measured as the weakest. A prior encoding a physical impossibility beats a
coefficient fitted on 21 examples.

`--fit` remains available for anyone with a substantially larger labelled corpus:
  `python scripts/push_model.py --repo-id <id> --bench-dir bench_models --fit`

- Prebuilt index: 20 sketches (backend `faiss`, 0 model(s) unreadable)

Full numbers, including the exact split and per-relation breakdown, are in `fit_report.json` in this repository. The wider benchmark (relatedness AUC / FPR@95TPR, merge F1 and mixing MAE, bytes per decision) is regenerated with `python benchmarks/run.py`.

## What weight geometry cannot do

Two structural limits were measured after this repository was first published, and they
bound how the artifacts here should be used:

1. **Direction is near-deterministic only for *lossy* operations.** Quantisation,
   pruning and vocabulary extension score 100%; scar-free SFT/LoRA/CPT edges abstain
   (mean |llr| ~0.02). The estimator declines rather than guessing, which is the correct
   failure mode for provenance.
2. **Outgroup rooting is invalid for merge children.** Rooting assumes descendants drift
   monotonically away from the root, but merging is a *contraction toward the centroid*:
   `0.6*sft + 0.4*cpt` partly cancels two perturbations and lands **closer to the root
   than either parent** (root→sft 0.000820, root→cpt 0.001610, root→merge 0.000678).
   Every correctly chosen sibling outgroup then pushes the answer the *wrong* way.
   Direction for a merged model must come from the **decomposition**, not from distance
   geometry — merge precision **1.000**, DARE mixing MAE **0.0004**.

Full derivations, with the measurements that produced them, are in
[`docs/FINDINGS.md`](https://github.com/NagaYu/stemma/blob/main/docs/FINDINGS.md).

## Scope and ethics

Stemma reports **statistical evidence with a confidence**, never a determination of
infringement or licence non-compliance. Weight-level similarity and derivation direction
are inferences from a small sample of tensors and can be wrong. A human must review every
finding before any action is taken.