MolMistral β€” SMILES foundation model (job 794 checkpoint, REPAIRED)

Foundation (pretrained, disease-agnostic) weights for a Mistral-style SMILES language model: 12 layers, 12 query heads, 1 KV head, hidden dim 512, intermediate dim 768, 21,831,680 parameters, DeepChem 2.4.0 SMILES vocabulary (621 tokens). Pretrained on a ~3.83M-molecule mixture (MOSES + ChEMBL + ZINC + PubChem, decontaminated against the DPP-4 fine-tune set).

File: best_foundation.keras β€” this is the file to fine-tune from for a new target/disease. Do NOT fine-tune from any DPP-4 fine-tuned checkpoint; those are DPP-4-specific.

⚠️ Defect found and repaired β€” read before use

This checkpoint comes from SPARCq job 794, which FAILED with NaN losses after ~1 clean pretraining epoch of 10. A pre-submission tensor scan of the saved best-val checkpoint (pretrain val_loss 0.2959) found exactly 2 NaN values out of 21,831,680 model weights, both in the token-embedding row for [C@H] (row 33, columns 357 and 415) β€” plus 12 further NaN/Β±Inf values at the same two coordinates in the saved AdamW optimizer-state buffers. [C@H] occurs in 41% of typical fine-tune sequences, so the unmodified file poisons fine-tuning β€” it is the likely reason job 794's own fine-tune stage went NaN from its first epoch.

All 14 entries were repaired in a copy by copying the values at the same columns from the [C@@H] row (row 35, the stereoisomeric counterpart token) within each tensor:

  • weights [33, 357]: NaN β†’ 0.003635966684669256
  • weights [33, 415]: NaN β†’ -0.008388345129787922
  • optimizer buffers: 12 values patched identically (see REPAIR_NOTES.md)

The repaired file was then re-scanned over the whole file (152,821,762 values): 0 non-finite values, and it reproduces the original checkpoint's held-out validation loss (0.272177 vs 0.272176 on 256 rows). Full details: REPAIR_NOTES.md, gate_scan.json (original scan), repaired_gate_scan.json (post-repair scan).

Hashes

File Size (bytes) SHA-256
best_foundation.keras (this repo, REPAIRED) 611,953,638 a27c247f9f064017f04b1d5c480a9f7800ce1745cbce77e8e5a8a28d72ce4f68
Job 794 original (NOT uploaded, defective) 611,953,638 8fa4e0f349e910158552f82d3c559859ddb39c9758cdd4600c74d53090dd4494

The unmodified original was deliberately not uploaded.

Caveat

The foundation saw only ~1 clean epoch of its planned 10-epoch pretraining before job 794 failed, so its quality is unverified pending a fine-tune test. Treat downstream results accordingly.

Downloads last month
36
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support