Synthyra
/

Profluent-E1-600M

Fill-Mask

Transformers

Safetensors

Model card Files Files and versions

xet

Community

lhallee commited on Nov 14, 2025

Commit

8fcb589

verified ·

1 Parent(s): 0d6b8d3

Upload README.md with huggingface_hub

Browse files

Files changed (1) hide show

README.md +164 -199

README.md CHANGED Viewed

@@ -1,199 +1,164 @@
----
-library_name: transformers
-tags: []
----
-# Model Card for Model ID
-<!-- Provide a quick summary of what the model is/does. -->
-## Model Details
-### Model Description
-<!-- Provide a longer summary of what this model is. -->
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-### Model Sources [optional]
-<!-- Provide the basic links for the model. -->
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
-## Uses
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-### Direct Use
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-[More Information Needed]
-### Downstream Use [optional]
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-[More Information Needed]
-### Out-of-Scope Use
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-[More Information Needed]
-## Bias, Risks, and Limitations
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-[More Information Needed]
-### Recommendations
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-## How to Get Started with the Model
-Use the code below to get started with the model.
-[More Information Needed]
-## Training Details
-### Training Data
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-[More Information Needed]
-### Training Procedure
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-#### Preprocessing [optional]
-[More Information Needed]
-#### Training Hyperparameters
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-#### Speeds, Sizes, Times [optional]
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-[More Information Needed]
-## Evaluation
-<!-- This section describes the evaluation protocols and provides the results. -->
-### Testing Data, Factors & Metrics
-#### Testing Data
-<!-- This should link to a Dataset Card if possible. -->
-[More Information Needed]
-#### Factors
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-[More Information Needed]
-#### Metrics
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-[More Information Needed]
-### Results
-[More Information Needed]
-#### Summary
-## Model Examination [optional]
-<!-- Relevant interpretability work for the model goes here -->
-[More Information Needed]
-## Environmental Impact
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-## Technical Specifications [optional]
-### Model Architecture and Objective
-[More Information Needed]
-### Compute Infrastructure
-[More Information Needed]
-#### Hardware
-[More Information Needed]
-#### Software
-[More Information Needed]
-## Citation [optional]
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-**BibTeX:**
-[More Information Needed]
-**APA:**
-[More Information Needed]
-## Glossary [optional]
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-[More Information Needed]
-## More Information [optional]
-[More Information Needed]
-## Model Card Authors [optional]
-[More Information Needed]
-## Model Card Contact
-[More Information Needed]

+---
+library_name: transformers
+tags: []
+---
+# NOTE
+The GitHub with the implementation and requirements.txt can be found [here](https://github.com/Synthyra/FastPLMs.git)
+# Profluent-E1
+[Profluent-E1](https://github.com/Synthyra/ESMplusplus) is a faithful implementation of Profluent's models [E1](https://www.profluent.bio/showcase/e1) ([license](https://github.com/Profluent-AI/E1/tree/main?tab=License-1-ov-file)) that integrates Huggingface AutoModel compatability and nice embedding functionality built in.
+## Use with 🤗 transformers
+### Supported models
+```python
+model_dict = {
+    # Synthyra/Profluent-E1_150M
+    'Profluent-E1-150M': 'Profluent-Bio/E1-150m',
+    # Synthyra/Profluent-E1_150M
+    'Profluent-E1-300M': 'Profluent-Bio/E1-300m',
+    # Synthyra/Profluent-E1_150M
+    'Profluent-E1-600M': 'Profluent-Bio/E1-600m',
+}
+```
+```python
+import torch
+from transformers import AutoModelForMaskedLM
+device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+model = AutoModelForMaskedLM.from_pretrained('Synthyra/Profluent-E1-150M', trust_remote_code=True, dtype=torch.bfloat16).eval().to(device)
+sequences = ['MPRTEIN', 'MSEQWENCE']
+batch = model.prep_tokens.get_batch_kwargs(sequences, device=device)
+output = model(**batch) # get all hidden states with output_hidden_states=True
+print(output.logits.shape) # language modeling logits, (batch_size, seq_len, vocab_size), (2, 11, 34)
+print(output.last_hidden_state.shape) # last hidden state of the model, (batch_size, seq_len, hidden_size), (2, 11, 768)
+print(output.loss) # language modeling loss if you passed labels
+#print(output.hidden_states) # all hidden states if you passed output_hidden_states=True (in tuple)
+#print(outout.attentions) # all attention matrices if you passed output_attentions=True (in tuple)
+```
+Our E1 implementation also supports sequence and token level classification tasks like ESM2. Simply pass the number of labels during initialization.
+```python
+from transformers import AutoModelForSequenceClassification, AutoModelForTokenClassification
+model = AutoModelForSequenceClassification.from_pretrained('Synthyra/Profluent-E1-150M', num_labels=2, trust_remote_code=True)
+logits = model(**batch, labels=labels).logits
+print(logits.shape) # (batch_size, num_labels), (2, 2)
+```
+E1 weights were trained in bf16 and are in bf16 by default. You can load them in the precision of your choosing by leveraging the dtype parameter:
+```python
+import torch
+model = AutoModelForMaskedLM.from_pretrained('Synthyra/ESMplusplus_small', trust_remote_code=True, dtype=torch.float) # fp32
+```
+## Embed entire datasets with no new code
+To embed a list of protein sequences **fast**, just call embed_dataset. Sequences are sorted to reduce padding tokens, so the initial progress bar estimation is usually much longer than the actual time it will take.
+Example:
+```python
+embedding_dict = model.embed_dataset(
+    sequences=[
+        'MALWMRLLPLLALLALWGPDPAAA', ... # list of protein sequences
+    ],
+    batch_size=2, # adjust for your GPU memory
+    max_len=512, # adjust for your needs
+    full_embeddings=False, # if True, no pooling is performed
+    embed_dtype=torch.float32, # cast to what dtype you want
+    pooling_types=['mean', 'cls'], # more than one pooling type will be concatenated together
+    sql=False, # if True, embeddings will be stored in SQLite database
+    sql_db_path='embeddings.db',
+    save=True, # if True, embeddings will be saved as a .pth file
+    save_path='embeddings.pth',
+)
+# embedding_dict is a dictionary mapping sequences to their embeddings as tensors for .pth or numpy arrays for sql
+```
+```
+model.embed_dataset()
+Args:
+    sequences: List of protein sequences
+    batch_size: Batch size for processing
+    max_len: Maximum sequence length
+    full_embeddings: Whether to return full residue-wise (True) embeddings or pooled (False)
+    pooling_type: Type of pooling ('mean' or 'cls')
+    sql: Whether to store embeddings in SQLite database - will be stored in float32
+    sql_db_path: Path to SQLite database
+Returns:
+    Dictionary mapping sequences to embeddings, or None if sql=True
+Note:
+    - If sql=True, embeddings can only be stored in float32
+    - sql is ideal if you need to stream a very large dataset for training in real-time
+    - save=True is ideal if you can store the entire embedding dictionary in RAM
+    - sql will be used if it is True and save is True or False
+    - If your sql database or .pth file is already present, they will be scanned first for already embedded sequences
+    - Sequences will be truncated to max_len and sorted by length in descending order for faster processing
+```
+## Fine-tuning with 🤗 peft
+```python
+model = AutoModelForSequenceClassification.from_pretrained('Synthyra/Profluent-E1-150M', num_labels=2, trust_remote_code=True)
+# these modules handle E1 attention layers
+target_modules = ["q_proj", "k_proj", "v_proj", "o_proj"]
+lora_config = LoraConfig(
+    r=8, # choose lora parameters to your liking
+    lora_alpha=16,
+    lora_dropout=0.01,
+    bias="none",
+    target_modules=target_modules,
+)
+# Apply LoRA to the model
+model = get_peft_model(model, lora_config)
+# Unfreeze the classifier head
+for param in model.classifier.parameters():
+    param.requires_grad = True
+```
+For a more thourough example of fine-tuning, check out our example script [here](https://github.com/Synthyra/FastPLMs/blob/main/fine_tuning_example.py).
+## Returning attention maps
+Usually F.scaled_dot_product_attention is used for the attention calculations, which is much faster than native PyTorch. However, it cannot return attention maps.
+ESM++ has the option to ```output_attentions```, which will calculate attention manually. This is much slower, so do not use unless you need the attention maps.
+```python
+output = model(**tokenized, output_attentions=True)
+att = output.attentions
+len(att) # 30, one for each layer, size (batch_size, num_heads, seq_len, seq_len) each
+```
+### Citation
+If you use any of this implementation or work please cite the following DOI and Profluent's paper.
+```
+@misc {FastPLMs,
+    author       = { Hallee, Logan and Bichara, David and Gleghorn, Jason P.},
+    title        = { FastPLMs: Fast, efficient, protien language model inference from Huggingface AutoModel.},
+    year         = {2024},
+    url          = { https://huggingface.co/Synthyra/ESMplusplus_small },
+    DOI          = { 10.57967/hf/3726 },
+    publisher    = { Hugging Face }
+}
+```
+```
+ @article{Jain_Beazer_Ruffolo_Bhatnagar_Madani_2025, title={E1: Retrieval-Augmented Protein Encoder Models},
+    url={https://www.biorxiv.org/content/early/2025/11/13/2025.11.12.688125},
+    DOI={10.1101/2025.11.12.688125},
+    journal={bioRxiv},
+    publisher={Cold Spring Harbor Laboratory},
+    author={Jain, Sarthak and Beazer, Joel and Ruffolo, Jeffrey A and Bhatnagar, Aadyot and Madani, Ali},
+    year={2025}
+}
+```