--- license: apache-2.0 library_name: stemma pretty_name: Stemma direction model tags: - model-provenance - lineage - safetensors - ai-bom - model-merging - supply-chain - not-a-language-model --- # Stemma direction model This repository does **not** contain a language model. It contains the small fitted artifacts that the [Stemma](https://github.com//stemma) provenance tool loads at runtime: | File | What it is | |---|---| | `direction_model.json` | The fitted `DirectionModel`: `weights` over `stemma.types.DIRECTION_FEATURES`, `bias`, `feature_names`, `scaler_mean`, `scaler_scale`. A regularised logistic combiner — a few dozen floats. | | `sketch_config.json` | The frozen sketch coordinate system: `version` (`SKETCH_VERSION`), `ROLES`, `DEPTH_BUCKETS`, `FEATURES_PER_SLOT`, `SKETCH_DIM`. A sketch computed under a different config is not comparable. | | `sketch_index.npz` + `sketch_index.json` | A prebuilt `stemma.phylogeny.SketchIndex` over the benchmark universe: L2-normalisable float32 sketch vectors, presence masks, model ids and per-model metadata. | | `fit_report.json` | Train/test sizes, held-out accuracy, per-relation accuracy and abstention rate from the fit that produced `direction_model.json`. | Everything here is generated by `python scripts/push_model.py --repo-id / --bench-dir bench_models --fit`. ## Intended use **In scope.** Loading into Stemma to (1) orient a candidate parent/child pair, (2) retrieve candidate parents for a model cheaply from a prebuilt index, (3) produce an AI Bill of Materials that a human then reviews. ```python from stemma.direction import DirectionModel, estimate_direction from stemma.phylogeny import SketchIndex model = DirectionModel.load("direction_model.json") index = SketchIndex.load("sketch_index") verdict = estimate_direction("org/a", "org/b", model=model) ``` **Out of scope.** Any automated enforcement, takedown, publication-blocking or procurement decision. Any use as a legal determination of license compliance or infringement. Any claim that a model "is" a derivative of another — Stemma reports how consistent the weights are with a direction of derivation, at a stated confidence, from a small sample of tensors. ## Generation procedure 1. **Benchmark construction.** `scripts/build_bench.py` builds real safetensors checkpoints with known lineages — fine-tunes, quantisations, prunings, vocabulary extensions, linear/TIES/DARE merges with known mixing ratios, plus unrelated negatives — and writes `ground_truth.json` with `models`, `edges` and labelled ordered `pairs`. 2. **Feature extraction.** For each labelled ordered pair `(a, b)`, `direction.collect_pair_evidence` Range-reads a handful of shared tensors and `direction.direction_features` reduces them to the antisymmetric feature vector named by `stemma.types.DIRECTION_FEATURES`. The features are antisymmetric by construction: `f(b, a) == -f(a, b)` to within 1e-6, so the fitted combiner cannot learn a positional bias. 3. **Fitting.** `DirectionModel.fit(X, y, l2=...)` on a seeded, deterministic train/test split (default seed 0, 25% held out). `y = +1` when `a` is the parent. Because the features are antisymmetric, each pair is also usable in its mirrored form; the split is done over *pairs*, not over rows, so a pair and its mirror never straddle the split. 4. **Index build.** Every model in the benchmark universe is sketched once (`sketch.sketch_model`) and the resulting vectors are stored in a `SketchIndex`. 5. **Packaging.** `scripts/push_model.py` writes the four files above plus this card and uploads with `HfApi.create_repo(exist_ok=True)` + `upload_folder`. The script is **dry-run by default**: without `--push` it prints exactly what would be uploaded and uploads nothing. Determinism: every randomised step takes an explicit `seed` (default `0`). ## Evaluation Held-out accuracy, per-relation accuracy and abstention rate are written to `fit_report.json` at fit time and mirrored into the repo's README frontmatter-free body by `scripts/push_model.py`. The full benchmark — relatedness AUC and FPR@95TPR, direction accuracy against the symmetric baselines, merge parent-set F1 and mixing MAE, bytes- and seconds-per-decision — is produced by `python benchmarks/run.py` and lives in `benchmarks/results.json`. No evaluation numbers are quoted in this card. Numbers belong in the generated `fit_report.json` and `benchmarks/results.json`, so that nothing here can drift away from what was actually measured. Reporting rules the harness enforces (from `docs/FINDINGS.md`): accuracy is reported **per relation type**, abstention is reported alongside accuracy-on-non-abstained, cross-architecture distillation is scored rather than excluded, and the symmetric baselines' 50% direction score is labelled a structural ceiling rather than a tuning failure. ## Limitations These are measured, not hypothetical; see `docs/FINDINGS.md`. - **Direction is near-deterministic only for lossy operations.** Quantisation and pruning scars and vocabulary extension are strong because they are irreversible. Scar-free SFT / LoRA / continued pretraining is **weakly identifiable from two models alone** and relies on outgroup rooting, which needs a usable third relative in the candidate universe. - **Norm growth is recipe-dependent and sign-flips across families.** Measured: `log‖B‖_F − log‖A‖_F` = −0.0171 (**0/8** tensors positive) for `Qwen2.5-0.5B → -Instruct`, and +0.0113 (**8/8** positive) for `SmolLM2-135M → -Instruct`, though both pairs are unambiguously base → instruct-tuned. `norm_growth_asym` is therefore a *fitted* feature with a small weight and never a hand-set sign. - **The Δ-subspace signal is small (~10⁻³–10⁻²) and signed opposite to the naive intuition.** It is a tiebreaker, not a decisive feature. - **Cross-architecture distillation leaves little weight-level signal.** - **TIES / DARE are non-linear**, so mixing ratios recovered by the linear decomposer are approximate; the residual is reported so the mismatch is visible. - **Fitted on a benchmark, not on the wild.** The coefficients reflect the operations and model families `build_bench.py` produced. Expect degradation on architectures, quantisation formats or merge recipes outside that distribution, and refit rather than assuming transfer. - **Sampled evidence.** Sketches read at most two tensors per (role, depth) slot with sub-sampled rows. An adversary who knows the sampling scheme can evade it. This is an auditing aid, not a tamper-proof watermark. - **Version-locked.** A `direction_model.json` is only valid against the `DIRECTION_FEATURES` order it was fitted with, and a `SketchIndex` only against its `SKETCH_VERSION`. Both are recorded in the artifacts and checked on load. ## Ethics and framing Stemma reports **statistical evidence** about weight-level similarity and derivation direction, with a confidence attached to every edge. It does **not** establish provenance as fact and does **not** constitute a legal determination of license compliance or infringement. A human must review every finding before any action is taken. Abstention (`direction="unknown"`) is a correct and expected outcome; a confident wrong answer is worse than no answer here. ## License Apache-2.0. The artifacts are derived from statistics computed over third-party checkpoints; Stemma does not redistribute any third-party weights, and those checkpoints' own licenses continue to govern them. --- ## This build - Repository: `NagaYu/stemma-direction` - Generated (UTC): `2026-08-08T13:24:27Z` - Sketch format: `stemma-sketch-v1` (dim 1456) - Direction weights: **hand-set priors** (`DirectionModel.default()`), **not fitted**, and that is a deliberate, measured choice rather than a missing step. ### Why the priors and not a fit Fitting an L2 logistic combiner on the benchmark's labelled ordered pairs was tried and **lost**. On the same held-out split the priors scored **1.000** accuracy on decided pairs against the fit's **0.500** — chance. Read that with its sample size: the priors abstained on 5 of 7 and decided only 2, so the accuracy gap rests on 2 decisions against 4 and is suggestive, not conclusive. The decisive evidence is *what the fit learned*. With 13 features and 21 training pairs the problem is underdetermined, and the fit assigned `lattice_asym` a **negative** weight — asserting that the quantised model is the parent. That is physically impossible: dequantisation cannot restore what rounding destroyed, so the scar can only ever appear downstream. It also put its largest weight on the statistic already measured as the weakest. A prior encoding a physical impossibility beats a coefficient fitted on 21 examples. `--fit` remains available for anyone with a substantially larger labelled corpus: `python scripts/push_model.py --repo-id --bench-dir bench_models --fit` - Prebuilt index: 20 sketches (backend `faiss`, 0 model(s) unreadable) Full numbers, including the exact split and per-relation breakdown, are in `fit_report.json` in this repository. The wider benchmark (relatedness AUC / FPR@95TPR, merge F1 and mixing MAE, bytes per decision) is regenerated with `python benchmarks/run.py`. ## What weight geometry cannot do Two structural limits were measured after this repository was first published, and they bound how the artifacts here should be used: 1. **Direction is near-deterministic only for *lossy* operations.** Quantisation, pruning and vocabulary extension score 100%; scar-free SFT/LoRA/CPT edges abstain (mean |llr| ~0.02). The estimator declines rather than guessing, which is the correct failure mode for provenance. 2. **Outgroup rooting is invalid for merge children.** Rooting assumes descendants drift monotonically away from the root, but merging is a *contraction toward the centroid*: `0.6*sft + 0.4*cpt` partly cancels two perturbations and lands **closer to the root than either parent** (root→sft 0.000820, root→cpt 0.001610, root→merge 0.000678). Every correctly chosen sibling outgroup then pushes the answer the *wrong* way. Direction for a merged model must come from the **decomposition**, not from distance geometry — merge precision **1.000**, DARE mixing MAE **0.0004**. Full derivations, with the measurements that produced them, are in [`docs/FINDINGS.md`](https://github.com/NagaYu/stemma/blob/main/docs/FINDINGS.md). ## Scope and ethics Stemma reports **statistical evidence with a confidence**, never a determination of infringement or licence non-compliance. Weight-level similarity and derivation direction are inferences from a small sample of tensors and can be wrong. A human must review every finding before any action is taken.