Nexa_Mat2 / eval /FINAL_EVIDENCE_SUMMARY.md
Allanatrix's picture
Archive NexaMat with final qualified model card and evidence (#1)
d40d291
|
Raw
History Blame Contribute Delete
4.75 kB

NexaMat Final Public Evidence Summary

This report is the public, sanitized evidence record for the archived NexaMat release. It contains aggregate scientific results and artifact-integrity information only. Private storage locations, infrastructure identifiers, experiment-control commands, account URLs, and internal dataset fingerprints are intentionally omitted.

Scope And Claims

NexaMat is released as a compact native inorganic crystal candidate generator. It demonstrated useful static validity and novelty across a six-profile non-alloy panel. It also produced isolated low-hull candidates in one bounded historical test, but did not demonstrate repeatable high-yield metastable or M.S.U.N. generation.

The full documented stack contains 4,061,859 parameters. The separately published V78 state is the recovered diffusion-decoder artifact, not the full encoder-decoder stack; its manifest reports 594,340 serialized parameter elements across 41 tensors.

Static Native Evidence

All rows below are native/no-repair diagnostic outputs. Profile gates differ, so accepted counts should be interpreted within each profile rather than as a single shared leaderboard.

Profile Raw candidates Accepted LeMat-radius valid Strict geometry Novelty
SSB regression 1,440 513 35.7% 60.4% 100.0%
Battery ionics 1,248 411 33.1% 60.3% 100.0%
Oxide catalysts 1,584 424 26.8% 59.7% 100.0%
Halides/chalcogenides 2,688 436 16.3% 55.1% 100.0%
Semiconductors 672 256 38.1% 65.0% 100.0%
Broad inorganic 2,688 242 9.0% 55.1% 100.0%

The plots in figures/ visualize these measured counts and rates without introducing a composite readiness score.

Stability Evidence

These cohorts were produced at different stages and are not interchangeable. The individual low-hull values are examples, not cohort averages or a reproducible production-rate claim.

Evidence cohort Pool Strict convergence Mean evaluated hull M.S.U.N. evidence Interpretation
Broad stability screen 512 58/512 1.2330 eV/atom 0 Broad-screen failure
Historical bounded screen 64 3/64 0.8750 eV/atom 2/64 Partial support; low reliability
Follow-up force-basin screen 64 4/64 0.8165 eV/atom 2/64 reported Directional improvement; recurrence not established

The historical bounded screen included individual candidates at 0.1786 and 0.1837 eV/atom. These values show that the model reached the low-hull region on isolated samples. They do not establish that typical output lies near 0.18 eV/atom. The intended metastable-yield target was 25-50%; the historical 2/64 result was materially below it.

Mechanism Evidence

The final research phase separated representation quality from controller quality:

Metric Result Reading
True-label stability ROC-AUC 0.70845 Representation gate passed
Minimum balanced accuracy 0.60048 Representation gate passed
Held-out force MAE 0.35973 eV/A Controller gate failed
Held-out force cosine 0.23114 Controller gate failed
Coordinate/lattice/pair contraction Negative Controller gate failed
Frozen physical-field force cosine 0.75450 Strong frozen baseline
Frozen physical-field force MAE 0.05728 eV/A Strong frozen baseline

The evidence supports a specific conclusion: the model learned useful stability-related representations and local physical signals, but the trained actuator did not reliably convert those signals into globally improving native rollouts across domains. Later residual experiments did not improve the frozen physical-field baseline.

Limitations

  • No shared-protocol SOTA comparison is claimed.
  • The stability cohorts differ in selection and stage, so trends are directional rather than a single calibrated production estimate.
  • The 2/64 historical M.S.U.N. result was not reproduced enough times to claim a stable rate.
  • Alloys/intermetallics, organics, MOFs, polymers, and molecular crystals are outside the qualified panel.
  • Generated candidates require independent relaxation, hull evaluation, and expert review before scientific or synthesis decisions.

Release Decision

The project is archived and on ice indefinitely because the metastability campaign consumed substantial time and compute without establishing the desired reliability. This is not a claim that the model is useless: it remains a compact, novel, static-valid candidate generator with partial low-hull support. The public release preserves that useful state while keeping the scientific limitations explicit.