# GRACE foundation-model deployment changelog

Each entry = one HF release of the foundation models. Versioning follows
`pgs/uq/deployment_protocol.md` §4:

- **`vX`** — single **incremental release version** (1, 2, 3, …), shared by the
  `model-vX` / `checkpoint-vX` / `kk-vX` tags of a release. Monotonic; never reused.
- **`-uq-vN`** — UQ feature version of the artifacts in that release (uqv6 → `-uq-v6`).

This file is the **source of truth**, tracked in-repo. On every deploy a copy is
uploaded to the HF repo root as `CHANGELOG.md` (same `create_commit` as the payloads).
Newest release on top.

---

## v3 — uq-v6 — model tag RE-ISSUED 2026-07-06 (add `compute_dense` to 2L)

- **Deliberate, one-time reset of the immutable `model-v3-uq-v6` tag** (protocol §4 tags are
  normally never moved; done here on explicit request). Only the **SavedModel** payload
  changed — `checkpoint-v3-uq-v6` and `kk-v3-uq-v6` are **untouched** (checkpoints and kokkos
  are byte-identical; the re-export was made *from* those same checkpoints).
- **What changed:** the **10 2L** models (OMAT/SMAX/SMAX-OMAT, medium+large), both fp32 and
  `-fp64` = **20 SavedModels**, re-exported so the SavedModel now also carries the
  **`compute_dense`** signature (fast dense-reshape neighbor engine) beside `compute`
  (segment_sum). Enables `TPCalculator(..., mode="uniform")` / the dense engine on these models.
- **1L unchanged (8 models):** 1L is a local, scalar-SPBF architecture with **no** equivariant
  dense engine, so `compute_dense` does not apply — the 1L SavedModels under the tag are the
  same v3 artifacts.
- Re-export reused the existing uqv6 GMM artifacts (no retrain, no GMM rebuild). Weights
  identical: re-export vs the previous SavedModel agrees to machine precision (fp64 dE=0,
  dF~1e-15; fp32 pure summation-reorder round-off ≤5e-6). `compute` vs `compute_dense` E/F
  agree (fp64 dE=0). fp64 dirs re-exported with `GRACE_UQ_LENIENT_LOAD=1` (old-format
  checkpoints carry one extra non-weight bool flag; validated safe against the deployed fp64).
- **Client URLs unchanged** — `foundation_models.py` already pins `model-v3-uq-v6`; the shifted
  tag serves the compute_dense 2L SavedModels transparently.

---

## v4 — uq-v6 — *DEPLOYED 2026-07-06*

- First **3-layer (3L)** GRACE foundation models, shipped with UQ:
  **`GRACE-3L-OMAT-large`** (base, fitted on OMat24) and
  **`GRACE-3L-OMAT-large-ft-AM`** (fine-tuned on sAlex+MPTraj).
- **uqv6** UQ feature (schema 6, basis-RP **D=132**, normalize + log-norm density);
  per-atom gamma flags both compression and stretch.
- **fp32-only.** These models are natively fp32 (no fp64 build exists), so the bare
  name *is* the fp32 payload — there is **no `-fp64` variant** (unlike the 18 v3
  2L/1L models). kokkos.npz is float-promoted (fp64 storage), served separately.
- **LAMMPS kk = `pair_style grace/3l/kk[/fp32]`** — MIXED/FP32 precision only (there
  is no fp64 3L kk kernel). Requires the schema-6-aware build with `grace/3l/kk`
  compiled in (its cuBLAS-sgemm selftest threshold is sized for fp32 GEMM). KK-vs-TF
  agreement is therefore at fp32 tolerance (E ≤5e-7 eV/atom, F ≤1.5e-6 eV/Å,
  gamma rel ≤3e-5), not the ~1e-8 of the fp64 2L/1L kk.
- Validated (2026-07-06): KK-vs-TF (E/F/stress + gamma, both models) + quick E-nn
  (gamma bounded, no compression blow-up).
- Tags: `model-v4-uq-v6`, `checkpoint-v4-uq-v6`, `kk-v4-uq-v6`.
- Supersedes the earlier upload-only, non-UQ `model-v3` ft-AM SavedModel (left in place;
  immutable).

## v3 — uq-v6 — *DEPLOYED 2026-06-26*

- All 18 foundation models rebuilt with the **uqv6** UQ feature (schema 6:
  L2-normalized basis direction `(B/‖B‖)@R` + log-norm density channels).
  De-collapses GMM covariance and flags **both** compression and stretch.
- **Default precision flipped to fp32.** The bare model name now resolves to the
  **fp32** SavedModel + checkpoint (2× faster, ½ memory; fp32↔fp64 gap negligible —
  gamma max-rel ≤5e-5). The full-precision build is published non-default as
  **`<name>-fp64`**. ⚠ Users who pull the bare name get fp32 on the next cache-miss;
  use `<name>-fp64` for tight finite-difference / bit-reproducible work. kokkos is
  precision-agnostic (one fp64 npz serves both).
- kokkos exports re-baked from the uqv6 artifacts (`uq_schema_version = 6`,
  `uq_feature_dim` 130 (1L) / 131 (2L), `uq_rp_normalize` + `uq_rp_add_density_channel`).
- **kokkos.npz distributed separately** as a raw file (not bundled in the model
  tarball) — see protocol §6.
- Tags: `model-v3-uq-v6`, `checkpoint-v3-uq-v6`, `kk-v3-uq-v6`.
- Done (2026-06-26): all 18 kokkos exported; UFF-GARDEN restaged fp32-default + `-fp64`;
  `foundation_models.py` wired (fp32 default, `-fp64` registered, kokkos separate-dist);
  `bin/upload_v3_{uqv6,kokkos}_hf.sh` written + dry-run-validated; schema-6-aware kk
  `pair_style grace` C++ build **shipped** (kk-vs-TF validated 18/18 OK for E/F/stress
  fp64 GPU + gamma fp64 GPU / fp32 CPU).
- Deployed (2026-06-26): pushed to HF `AMS-ICAMS-RUB/grace-foundation-models` —
  models+checkpoints (72 archives, 10.2 GB) at commit `4aff666` (tags `model-v3-uq-v6`,
  `checkpoint-v3-uq-v6`); kokkos (18 npz, 3.28 GB) at commit `75456fa` (tag `kk-v3-uq-v6`);
  CHANGELOG.md uploaded. Client (`foundation_models.py`/`grace_models.py`) committed `ef000a2`
  already resolves these tags. No blockers left.

## v2 — *superseded by v3 (2026-06-26)*

- Previous generation, live on the legacy HF tags `kk` (model archives, kokkos
  bundled) and `uq-v1` (original 8 OMAT-UQ checkpoints). UQ = uqv2/uqv3-era.
- This is the baseline the v3 rollout supersedes.