Upload 6 files

README.md

@@ -1,3 +1,61 @@
----
-license: bsd-3-clause
----
+---
+tags:
+- chemistry
+- molecular-design
+- transformer
+- generative-model
+- predictive-model
+license: bsd-3-clause
+datasets:
+- GuacaMol
+- ZINC
+- MoleculeNet
+gated: true
+extra_gated_fields:
+  Organization: text
+  Intended use: text
+  Contact person: text
+  E-mail: text
+  Country: country
+  Date: date_picker
+  I agree to use this model only for purposes that are non-malicious and ethically responsible: checkbox
+  I have read and accept the BSD 3-Clause license: checkbox
+---
+
+# Hyformer
+
+Hyformer is a joint transformer-based model that unifies a generative decoder with a predictive encoder. Depending on the task, Hyformer uses either a causal or a bidirectional mask, outputting token probabilities or predicted property values.
+
+## Model Details
+- **Paper:** [Synergistic Benefits of Joint Molecule Generation and Property Prediction](https://arxiv.org/abs/2504.16559)
+- **Authors:** Adam Izdebski, Jan Olszewski, Pankhil Gawade, Krzysztof Koras, Serra Korkmaz, Valentin Rauscher, Jakub M. Tomczak, Ewa Szczurek
+- **License:** BSD 3-Clause
+- **Repository:** [https://github.com/szczurek-lab/hyformer](https://github.com/szczurek-lab/hyformer)
+
+## Model checkpoints
+- **[Hyformer_molecules_8M](https://huggingface.co/SzczurekLab/hyformer_molecules_8M):** Trained on GuacaMol dataset ([Brown et al., 2019](https://jcheminf.biomedcentral.com/articles/10.1186/s13321-019-0351-9))
+- **[Hyformer_molecules_50M](https://huggingface.co/SzczurekLab/hyformer_molecules_50M):** Trained on 19M molecules from ZINC, ChEMBL, and other purchasable molecular datasets ([Zhou et al., 2023](https://openreview.net/forum?id=1pPpKc9wR0Y))
+- **[Hyformer_peptides_34M](https://huggingface.co/SzczurekLab/hyformer_peptides_34M):** Trained on 3.5M general-purpose and antimicrobial peptides
+- **[Hyformer_peptides_34M_MIC](https://huggingface.co/SzczurekLab/hyformer_peptides_34M_MIC):** `Hyformer_peptides_34M` jointly fine-tuned on minimal inhibitory concentration values (MIC) against E. coli bacteria
+
+## Gated Access
+This model is available with **gated access**. To request access, please use the Hugging Face gated request form.
+
+## Citation
+If you use this model, please cite:
+
+```
+@misc{izdebski2025synergisticbenefitsjointmolecule,
+      title={Synergistic Benefits of Joint Molecule Generation and Property Prediction}, 
+      author={Adam Izdebski and Jan Olszewski and Pankhil Gawade and Krzysztof Koras and Serra Korkmaz and Valentin Rauscher and Jakub M. Tomczak and Ewa Szczurek},
+      year={2025},
+      eprint={2504.16559},
+      archivePrefix={arXiv},
+      primaryClass={cs.LG},
+      url={https://arxiv.org/abs/2504.16559}, 
+}
+```
+
+## References
+- Brown, Nathan, et al. "GuacaMol: benchmarking models for de novo molecular design." Journal of chemical information and modeling, 2019.
+- Zhou, Gengmo, et al. "Uni-mol: A universal 3d molecular representation learning framework." ICLR, 2023.

ckpt.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:18371a0c1ab3f9808b05ee1ed633991166eab4f86bea3a96e257aad1bf01053c
+size 610097270

model_config.json

@@ -0,0 +1,30 @@
+{
+    "model_name": "Hyformer",
+    "embedding_dim": 512,
+    "embedding_hidden_dim": 2048,
+    "num_heads": 8,
+    "num_local_heads": 8,
+    "head_dim": 64,
+    "num_layers": 12,
+    "bias": false,
+    "attention_dropout": 0.0,
+    "feed_forward_dropout": 0.0,
+    "prediction_dropout": null,
+    "layer_norm_eps": 1e-05,
+    "vocab_size": 596,
+    "max_seq_len": 128,
+    "prediction_task_type": null,
+    "num_prediction_tasks": null,
+    "num_physchem_tasks": 200,
+    "pretrained_filepath": null,
+    "predictor_hidden_size": null,
+    "predictor_dropout": null,
+    "predictor_num_heads": null,
+    "prediction_hidden_dim": 512,
+    "set_separate_task_tokens": null,
+    "flash_attention": true,
+    "dropout": null,
+    "lambda_hparam": null,
+    "pooler_dropout": null,
+    "pooler_hidden_dim": null
+}

tokenizer_config.json

@@ -0,0 +1,6 @@
+{
+    "tokenizer": "SmilesTokenizerSeparateTaskToken",
+    "path_to_vocabulary": "data/vocabularies/deepchem.txt",
+    "max_molecule_length": 128
+  }
+  

trainer_config.json

@@ -0,0 +1,33 @@
+{
+    "compile": true,
+    "enable_ddp": false,
+    "dtype": "bfloat16",
+    "eval_only": false,
+    "max_iters": 250000,
+    "max_epochs": null,
+    "batch_size": 1024,
+    "learning_rate": 0.0006,
+    "weight_decay": 0.1,
+    "beta1": 0.9,
+    "beta2": 0.95,
+    "gradient_accumulation_steps": 1,
+    "grad_clip": 1.0,
+    "decay_lr": true,
+    "warmup_iters": 5000,
+    "lr_decay_iters": 250000,
+    "min_lr": 6e-05,
+    "eval_iters": 48,
+    "eval_interval": 25000,
+    "log_interval": 5000,
+    "always_save_checkpoint": false,
+    "save_checkpoint_every": 50000,
+    "save_checkpoint": true,
+    "save_snapshot": false,
+    "eval_generation": true,
+    "block_size": 128,
+    "tasks": {
+        "generation": 0.8,
+        "reconstruction": 0.1,
+        "physchem": 0.1
+    }
+}

vocab.txt

@@ -0,0 +1,591 @@
+[PAD]
+[unused1]
+[unused2]
+[unused3]
+[unused4]
+[unused5]
+[unused6]
+[unused7]
+[unused8]
+[unused9]
+[unused10]
+[UNK]
+[CLS]
+[SEP]
+[MASK]
+c
+C
+(
+)
+O
+1
+2
+=
+N
+.
+n
+3
+F
+Cl
+>>
+~
+-
+4
+[C@H]
+S
+[C@@H]
+[O-]
+Br
+#
+/
+[nH]
+[N+]
+s
+5
+o
+P
+[Na+]
+[Si]
+I
+[Na]
+[Pd]
+[K+]
+[K]
+[P]
+B
+[C@]
+[C@@]
+[Cl-]
+6
+[OH-]
+\
+[N-]
+[Li]
+[H]
+[2H]
+[NH4+]
+[c-]
+[P-]
+[Cs+]
+[Li+]
+[Cs]
+[NaH]
+[H-]
+[O+]
+[BH4-]
+[Cu]
+7
+[Mg]
+[Fe+2]
+[n+]
+[Sn]
+[BH-]
+[Pd+2]
+[CH]
+[I-]
+[Br-]
+[C-]
+[Zn]
+[B-]
+[F-]
+[Al]
+[P+]
+[BH3-]
+[Fe]
+[C]
+[AlH4]
+[Ni]
+[SiH]
+8
+[Cu+2]
+[Mn]
+[AlH]
+[nH+]
+[AlH4-]
+[O-2]
+[Cr]
+[Mg+2]
+[NH3+]
+[S@]
+[Pt]
+[Al+3]
+[S@@]
+[S-]
+[Ti]
+[Zn+2]
+[PH]
+[NH2+]
+[Ru]
+[Ag+]
+[S+]
+[I+3]
+[NH+]
+[Ca+2]
+[Ag]
+9
+[Os]
+[Se]
+[SiH2]
+[Ca]
+[Ti+4]
+[Ac]
+[Cu+]
+[S]
+[Rh]
+[Cl+3]
+[cH-]
+[Zn+]
+[O]
+[Cl+]
+[SH]
+[H+]
+[Pd+]
+[se]
+[PH+]
+[I]
+[Pt+2]
+[C+]
+[Mg+]
+[Hg]
+[W]
+[SnH]
+[SiH3]
+[Fe+3]
+[NH]
+[Mo]
+[CH2+]
+%10
+[CH2-]
+[CH2]
+[n-]
+[Ce+4]
+[NH-]
+[Co]
+[I+]
+[PH2]
+[Pt+4]
+[Ce]
+[B]
+[Sn+2]
+[Ba+2]
+%11
+[Fe-3]
+[18F]
+[SH-]
+[Pb+2]
+[Os-2]
+[Zr+4]
+[N]
+[Ir]
+[Bi]
+[Ni+2]
+[P@]
+[Co+2]
+[s+]
+[As]
+[P+3]
+[Hg+2]
+[Yb+3]
+[CH-]
+[Zr+2]
+[Mn+2]
+[CH+]
+[In]
+[KH]
+[Ce+3]
+[Zr]
+[AlH2-]
+[OH2+]
+[Ti+3]
+[Rh+2]
+[Sb]
+[S-2]
+%12
+[P@@]
+[Si@H]
+[Mn+4]
+p
+[Ba]
+[NH2-]
+[Ge]
+[Pb+4]
+[Cr+3]
+[Au]
+[LiH]
+[Sc+3]
+[o+]
+[Rh-3]
+%13
+[Br]
+[Sb-]
+[S@+]
+[I+2]
+[Ar]
+[V]
+[Cu-]
+[Al-]
+[Te]
+[13c]
+[13C]
+[Cl]
+[PH4+]
+[SiH4]
+[te]
+[CH3-]
+[S@@+]
+[Rh+3]
+[SH+]
+[Bi+3]
+[Br+2]
+[La]
+[La+3]
+[Pt-2]
+[N@@]
+[PH3+]
+[N@]
+[Si+4]
+[Sr+2]
+[Al+]
+[Pb]
+[SeH]
+[Si-]
+[V+5]
+[Y+3]
+[Re]
+[Ru+]
+[Sm]
+*
+[3H]
+[NH2]
+[Ag-]
+[13CH3]
+[OH+]
+[Ru+3]
+[OH]
+[Gd+3]
+[13CH2]
+[In+3]
+[Si@@]
+[Si@]
+[Ti+2]
+[Sn+]
+[Cl+2]
+[AlH-]
+[Pd-2]
+[SnH3]
+[B+3]
+[Cu-2]
+[Nd+3]
+[Pb+3]
+[13cH]
+[Fe-4]
+[Ga]
+[Sn+4]
+[Hg+]
+[11CH3]
+[Hf]
+[Pr]
+[Y]
+[S+2]
+[Cd]
+[Cr+6]
+[Zr+3]
+[Rh+]
+[CH3]
+[N-3]
+[Hf+2]
+[Th]
+[Sb+3]
+%14
+[Cr+2]
+[Ru+2]
+[Hf+4]
+[14C]
+[Ta]
+[Tl+]
+[B+]
+[Os+4]
+[PdH2]
+[Pd-]
+[Cd+2]
+[Co+3]
+[S+4]
+[Nb+5]
+[123I]
+[c+]
+[Rb+]
+[V+2]
+[CH3+]
+[Ag+2]
+[cH+]
+[Mn+3]
+[Se-]
+[As-]
+[Eu+3]
+[SH2]
+[Sm+3]
+[IH+]
+%15
+[OH3+]
+[PH3]
+[IH2+]
+[SH2+]
+[Ir+3]
+[AlH3]
+[Sc]
+[Yb]
+[15NH2]
+[Lu]
+[sH+]
+[Gd]
+[18F-]
+[SH3+]
+[SnH4]
+[TeH]
+[Si@@H]
+[Ga+3]
+[CaH2]
+[Tl]
+[Ta+5]
+[GeH]
+[Br+]
+[Sr]
+[Tl+3]
+[Sm+2]
+[PH5]
+%16
+[N@@+]
+[Au+3]
+[C-4]
+[Nd]
+[Ti+]
+[IH]
+[N@+]
+[125I]
+[Eu]
+[Sn+3]
+[Nb]
+[Er+3]
+[123I-]
+[14c]
+%17
+[SnH2]
+[YH]
+[Sb+5]
+[Pr+3]
+[Ir+]
+[N+3]
+[AlH2]
+[19F]
+%18
+[Tb]
+[14CH]
+[Mo+4]
+[Si+]
+[BH]
+[Be]
+[Rb]
+[pH]
+%19
+%20
+[Xe]
+[Ir-]
+[Be+2]
+[C+4]
+[RuH2]
+[15NH]
+[U+2]
+[Au-]
+%21
+%22
+[Au+]
+[15n]
+[Al+2]
+[Tb+3]
+[15N]
+[V+3]
+[W+6]
+[14CH3]
+[Cr+4]
+[ClH+]
+b
+[Ti+6]
+[Nd+]
+[Zr+]
+[PH2+]
+[Fm]
+[N@H+]
+[RuH]
+[Dy+3]
+%23
+[Hf+3]
+[W+4]
+[11C]
+[13CH]
+[Er]
+[124I]
+[LaH]
+[F]
+[siH]
+[Ga+]
+[Cm]
+[GeH3]
+[IH-]
+[U+6]
+[SeH+]
+[32P]
+[SeH-]
+[Pt-]
+[Ir+2]
+[se+]
+[U]
+[F+]
+[BH2]
+[As+]
+[Cf]
+[ClH2+]
+[Ni+]
+[TeH3]
+[SbH2]
+[Ag+3]
+%24
+[18O]
+[PH4]
+[Os+2]
+[Na-]
+[Sb+2]
+[V+4]
+[Ho+3]
+[68Ga]
+[PH-]
+[Bi+2]
+[Ce+2]
+[Pd+3]
+[99Tc]
+[13C@@H]
+[Fe+6]
+[c]
+[GeH2]
+[10B]
+[Cu+3]
+[Mo+2]
+[Cr+]
+[Pd+4]
+[Dy]
+[AsH]
+[Ba+]
+[SeH2]
+[In+]
+[TeH2]
+[BrH+]
+[14cH]
+[W+]
+[13C@H]
+[AsH2]
+[In+2]
+[N+2]
+[N@@H+]
+[SbH]
+[60Co]
+[AsH4+]
+[AsH3]
+[18OH]
+[Ru-2]
+[Na-2]
+[CuH2]
+[31P]
+[Ti+5]
+[35S]
+[P@@H]
+[ArH]
+[Co+]
+[Zr-2]
+[BH2-]
+[131I]
+[SH5]
+[VH]
+[B+2]
+[Yb+2]
+[14C@H]
+[211At]
+[NH3+2]
+[IrH]
+[IrH2]
+[Rh-]
+[Cr-]
+[Sb+]
+[Ni+3]
+[TaH3]
+[Tl+2]
+[64Cu]
+[Tc]
+[Cd+]
+[1H]
+[15nH]
+[AlH2+]
+[FH+2]
+[BiH3]
+[Ru-]
+[Mo+6]
+[AsH+]
+[BaH2]
+[BaH]
+[Fe+4]
+[229Th]
+[Th+4]
+[As+3]
+[NH+3]
+[P@H]
+[Li-]
+[7NaH]
+[Bi+]
+[PtH+2]
+[p-]
+[Re+5]
+[NiH]
+[Ni-]
+[Xe+]
+[Ca+]
+[11c]
+[Rh+4]
+[AcH]
+[HeH]
+[Sc+2]
+[Mn+]
+[UH]
+[14CH2]
+[SiH4+]
+[18OH2]
+[Ac-]
+[Re+4]
+[118Sn]
+[153Sm]
+[P+2]
+[9CH]
+[9CH3]
+[Y-]
+[NiH2]
+[Si+2]
+[Mn+6]
+[ZrH2]
+[C-2]
+[Bi+5]
+[24NaH]
+[Fr]
+[15CH]
+[Se+]
+[At]
+[P-3]
+[124I-]
+[CuH2-]
+[Nb+4]
+[Nb+3]
+[MgH]
+[Ir+4]
+[67Ga+3]
+[67Ga]
+[13N]
+[15OH2]
+[2NH]
+[Ho]
+[Cn]