Upload README.md with huggingface_hub

README.md

@@ -1,190 +1,190 @@
----
-library_name: transformers
-tags: []
----
-
-# NOTE
-The GitHub with the implementation and requirements.txt can be found [here](https://github.com/Synthyra/FastPLMs.git)
-
-# Profluent-E1
-[Synthyra's version of Profluent-E1](https://github.com/Synthyra/Profluent-E1-300M) is a faithful implementation of Profluent's [E1](https://www.profluent.bio/showcase/e1) models ([license](https://github.com/Profluent-AI/E1/tree/main?tab=License-1-ov-file)) that integrates Huggingface AutoModel compatability and nice embedding functionality.
-
-## Attention backends
-
-`sdpa` (PyTorch Scaled Dot Product Attention) is the default. The backend is set via `config.attn_backend` before loading.
-
-| Backend | Key | Notes |
-| :--- | :--- | :--- |
-| PyTorch SDPA | `"sdpa"` | Default. Exact numerics, stable on all hardware. |
-| Flash Attention | `"kernels_flash"` | Fastest on Ampere/Hopper GPUs. Requires `pip install kernels` (pre-built — no hours-long compilation). Outputs are not bitwise identical to SDPA due to online softmax reordering; differences are often small but not guaranteed to be inconsequential — use `"sdpa"` if exact numerics matter. |
-| Flex Attention | `"flex"` | Uses a block-causal mask that skips padding tokens. Near-exact numerics. First use compiles a Triton kernel (30–120 s). Best combined with `torch.compile`. |
-| Auto | `"auto"` | Picks the best available: `kernels_flash` → `flex` → `sdpa`. |
-
-```python
-from transformers import AutoConfig, AutoModelForMaskedLM
-
-config = AutoConfig.from_pretrained("Synthyra/Profluent-E1-150M", trust_remote_code=True)
-config.attn_backend = "flex"  # or "kernels_flash", "sdpa", "auto"
-model = AutoModelForMaskedLM.from_pretrained("Synthyra/Profluent-E1-150M", config=config, trust_remote_code=True)
-```
-
-`torch.compile(model)` is heavily recommended for sustained throughput, especially with Flex Attention.
-
-
-## Use with 🤗 transformers
-### Supported models
-```python
-model_dict = {
-    # Synthyra/Profluent-E1-150M
-    'Profluent-E1-150M': 'Profluent-Bio/E1-150m',
-    # Synthyra/Profluent-E1-150M
-    'Profluent-E1-300M': 'Profluent-Bio/E1-300m',
-    # Synthyra/Profluent-E1-150M
-    'Profluent-E1-600M': 'Profluent-Bio/E1-600m',
-}
-```
-
-```python
-import torch
-from transformers import AutoModelForMaskedLM
-
-device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
-model = AutoModelForMaskedLM.from_pretrained('Synthyra/Profluent-E1-150M', trust_remote_code=True, dtype=torch.bfloat16).eval().to(device)
-
-sequences = ['MPRTEIN', 'MSEQWENCE']
-batch = model.prep_tokens.get_batch_kwargs(sequences, device=device)
-
-output = model(**batch) # get all hidden states with output_hidden_states=True
-print(output.logits.shape) # language modeling logits, (batch_size, seq_len, vocab_size), (2, 11, 34)
-print(output.last_hidden_state.shape) # last hidden state of the model, (batch_size, seq_len, hidden_size), (2, 11, 768)
-print(output.loss) # language modeling loss if you passed labels
-#print(output.hidden_states) # all hidden states if you passed output_hidden_states=True (in tuple)
-#print(outout.attentions) # all attention matrices if you passed output_attentions=True (in tuple)
-```
-
-Our E1 implementation also supports sequence and token level classification tasks like ESM2. Simply pass the number of labels during initialization.
-
-```python
-from transformers import AutoModelForSequenceClassification, AutoModelForTokenClassification
-
-model = AutoModelForSequenceClassification.from_pretrained('Synthyra/Profluent-E1-150M', num_labels=2, trust_remote_code=True)
-logits = model(**batch, labels=labels).logits
-print(logits.shape) # (batch_size, num_labels), (2, 2)
-```
-
-E1 weights were trained in bf16 and are in bf16 by default. You can load them in the precision of your choosing by leveraging the dtype parameter:
-```python
-import torch
-model = AutoModelForMaskedLM.from_pretrained('Synthyra/Profluent-E1-150M', trust_remote_code=True, dtype=torch.float) # fp32
-```
-
-## Embed entire datasets with no new code
-To embed a list of protein sequences **fast**, just call embed_dataset. Sequences are sorted to reduce padding tokens, so the initial progress bar estimation is usually much longer than the actual time it will take.
-
-Example:
-```python
-embedding_dict = model.embed_dataset(
-    sequences=[
-        'MALWMRLLPLLALLALWGPDPAAA', ... # list of protein sequences
-    ],
-    batch_size=2, # adjust for your GPU memory
-    max_len=512, # adjust for your needs
-    full_embeddings=False, # if True, no pooling is performed
-    embed_dtype=torch.float32, # cast to what dtype you want
-    pooling_types=['mean', 'cls'], # more than one pooling type will be concatenated together
-    sql=False, # if True, embeddings will be stored in SQLite database
-    sql_db_path='embeddings.db',
-    save=True, # if True, embeddings will be saved as a .pth file
-    save_path='embeddings.pth',
-)
-# embedding_dict is a dictionary mapping sequences to their embeddings as tensors for .pth or numpy arrays for sql
-```
-
-```
-model.embed_dataset()
-Args:
-    sequences: List of protein sequences
-    batch_size: Batch size for processing
-    max_len: Maximum sequence length
-    full_embeddings: Whether to return full residue-wise (True) embeddings or pooled (False)
-    pooling_type: Type of pooling ('mean' or 'cls')
-    sql: Whether to store embeddings in SQLite database - will be stored in float32
-    sql_db_path: Path to SQLite database
-    
-Returns:
-    Dictionary mapping sequences to embeddings, or None if sql=True
-
-Note:
-    - If sql=True, embeddings can only be stored in float32
-    - sql is ideal if you need to stream a very large dataset for training in real-time
-    - save=True is ideal if you can store the entire embedding dictionary in RAM
-    - sql will be used if it is True and save is True or False
-    - If your sql database or .pth file is already present, they will be scanned first for already embedded sequences
-    - Sequences will be truncated to max_len and sorted by length in descending order for faster processing
-```
-
-## Fine-tuning with 🤗 peft
-```python
-model = AutoModelForSequenceClassification.from_pretrained('Synthyra/Profluent-E1-150M', num_labels=2, trust_remote_code=True)
-# these modules handle E1 attention layers
-target_modules = ["q_proj", "k_proj", "v_proj", "o_proj"]
-
-lora_config = LoraConfig(
-    r=8, # choose lora parameters to your liking
-    lora_alpha=16,
-    lora_dropout=0.01,
-    bias="none",
-    target_modules=target_modules,
-)
-
-# Apply LoRA to the model
-model = get_peft_model(model, lora_config)
-
-# Unfreeze the classifier head
-for param in model.classifier.parameters():
-    param.requires_grad = True
-```
-
-For a more thourough example of fine-tuning, check out our example script [here](https://github.com/Synthyra/FastPLMs/blob/main/fine_tuning_example.py).
-
-
-### Citations
-
-```bibtex
-@misc{FastPLMs,
-  author={Hallee, Logan and Bichara, David and Gleghorn, Jason P.},
-  title={FastPLMs: Fast, efficient, protein language model inference from Huggingface AutoModel.},
-  year={2024},
-  url={https://huggingface.co/Synthyra/ESMplusplus_small},
-  DOI={10.57967/hf/3726},
-  publisher={Hugging Face}
-}
-```
-
-```bibtex
-@article{jain2025e1,
-  title={E1: Retrieval-Augmented Protein Encoder Models},
-  author={Jain, Sarthak and Beazer, Joel and Ruffolo, Jeffrey A and Bhatnagar, Aadyot and Madani, Ali},
-  journal={bioRxiv},
-  DOI={10.1101/2025.11.12.688125},
-  year={2025}
-}
-```
-
-```bibtex
-@article{dong2024flexattention,
-  title={Flex Attention: A Programming Model for Generating Optimized Attention Kernels},
-  author={Dong, Juechu and Feng, Boyuan and Guessous, Driss and Liang, Yanbo and He, Horace},
-  journal={arXiv preprint arXiv:2412.05496},
-  year={2024}
-}
-```
-
-```bibtex
-@inproceedings{paszke2019pytorch,
-  title={PyTorch: An Imperative Style, High-Performance Deep Learning Library},
-  author={Paszke, Adam and Gross, Sam and Massa, Francisco and Lerer, Adam and Bradbury, James and Chanan, Gregory and Killeen, Trevor and Lin, Zeming and Gimelshein, Natalia and Antiga, Luca and Desmaison, Alban and K{\"o}pf, Andreas and Yang, Edward and DeVito, Zach and Raison, Martin and Tejani, Alykhan and Chilamkurthy, Sasank and Steiner, Benoit and Fang, Lu and Bai, Junjie and Chintala, Soumith},
-  booktitle={Advances in Neural Information Processing Systems 32},
-  year={2019}
-}
+---
+library_name: transformers
+tags: []
+---
+
+# NOTE
+The GitHub with the implementation and requirements.txt can be found [here](https://github.com/Synthyra/FastPLMs.git)
+
+# Profluent-E1
+[Synthyra's version of Profluent-E1](https://github.com/Synthyra/Profluent-E1-300M) is a faithful implementation of Profluent's [E1](https://www.profluent.bio/showcase/e1) models ([license](https://github.com/Profluent-AI/E1/tree/main?tab=License-1-ov-file)) that integrates Huggingface AutoModel compatability and nice embedding functionality.
+
+## Attention backends
+
+`sdpa` (PyTorch Scaled Dot Product Attention) is the default. The backend is set via `config.attn_backend` before loading.
+
+| Backend | Key | Notes |
+| :--- | :--- | :--- |
+| PyTorch SDPA | `"sdpa"` | Default. Exact numerics, stable on all hardware. |
+| Flash Attention | `"kernels_flash"` | Fastest on Ampere/Hopper GPUs. Requires `pip install kernels` (pre-built — no hours-long compilation). Outputs are not bitwise identical to SDPA due to online softmax reordering; differences are often small but not guaranteed to be inconsequential — use `"sdpa"` if exact numerics matter. |
+| Flex Attention | `"flex"` | Uses a block-causal mask that skips padding tokens. Near-exact numerics. First use compiles a Triton kernel (30–120 s). Best combined with `torch.compile`. |
+| Auto | `"auto"` | Picks the best available: `kernels_flash` → `flex` → `sdpa`. |
+
+```python
+from transformers import AutoConfig, AutoModelForMaskedLM
+
+config = AutoConfig.from_pretrained("Synthyra/Profluent-E1-150M", trust_remote_code=True)
+config.attn_backend = "flex"  # or "kernels_flash", "sdpa", "auto"
+model = AutoModelForMaskedLM.from_pretrained("Synthyra/Profluent-E1-150M", config=config, trust_remote_code=True)
+```
+
+`torch.compile(model)` is heavily recommended for sustained throughput, especially with Flex Attention.
+
+
+## Use with 🤗 transformers
+### Supported models
+```python
+model_dict = {
+    # Synthyra/Profluent-E1-150M
+    'Profluent-E1-150M': 'Profluent-Bio/E1-150m',
+    # Synthyra/Profluent-E1-150M
+    'Profluent-E1-300M': 'Profluent-Bio/E1-300m',
+    # Synthyra/Profluent-E1-150M
+    'Profluent-E1-600M': 'Profluent-Bio/E1-600m',
+}
+```
+
+```python
+import torch
+from transformers import AutoModelForMaskedLM
+
+device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+model = AutoModelForMaskedLM.from_pretrained('Synthyra/Profluent-E1-150M', trust_remote_code=True, dtype=torch.bfloat16).eval().to(device)
+
+sequences = ['MPRTEIN', 'MSEQWENCE']
+batch = model.prep_tokens.get_batch_kwargs(sequences, device=device)
+
+output = model(**batch) # get all hidden states with output_hidden_states=True
+print(output.logits.shape) # language modeling logits, (batch_size, seq_len, vocab_size), (2, 11, 34)
+print(output.last_hidden_state.shape) # last hidden state of the model, (batch_size, seq_len, hidden_size), (2, 11, 768)
+print(output.loss) # language modeling loss if you passed labels
+#print(output.hidden_states) # all hidden states if you passed output_hidden_states=True (in tuple)
+#print(outout.attentions) # all attention matrices if you passed output_attentions=True (in tuple)
+```
+
+Our E1 implementation also supports sequence and token level classification tasks like ESM2. Simply pass the number of labels during initialization.
+
+```python
+from transformers import AutoModelForSequenceClassification, AutoModelForTokenClassification
+
+model = AutoModelForSequenceClassification.from_pretrained('Synthyra/Profluent-E1-150M', num_labels=2, trust_remote_code=True)
+logits = model(**batch, labels=labels).logits
+print(logits.shape) # (batch_size, num_labels), (2, 2)
+```
+
+E1 weights were trained in bf16 and are in bf16 by default. You can load them in the precision of your choosing by leveraging the dtype parameter:
+```python
+import torch
+model = AutoModelForMaskedLM.from_pretrained('Synthyra/Profluent-E1-150M', trust_remote_code=True, dtype=torch.float) # fp32
+```
+
+## Embed entire datasets with no new code
+To embed a list of protein sequences **fast**, just call embed_dataset. Sequences are sorted to reduce padding tokens, so the initial progress bar estimation is usually much longer than the actual time it will take.
+
+Example:
+```python
+embedding_dict = model.embed_dataset(
+    sequences=[
+        'MALWMRLLPLLALLALWGPDPAAA', ... # list of protein sequences
+    ],
+    batch_size=2, # adjust for your GPU memory
+    max_len=512, # adjust for your needs
+    full_embeddings=False, # if True, no pooling is performed
+    embed_dtype=torch.float32, # cast to what dtype you want
+    pooling_types=['mean', 'cls'], # more than one pooling type will be concatenated together
+    sql=False, # if True, embeddings will be stored in SQLite database
+    sql_db_path='embeddings.db',
+    save=True, # if True, embeddings will be saved as a .pth file
+    save_path='embeddings.pth',
+)
+# embedding_dict is a dictionary mapping sequences to their embeddings as tensors for .pth or numpy arrays for sql
+```
+
+```
+model.embed_dataset()
+Args:
+    sequences: List of protein sequences
+    batch_size: Batch size for processing
+    max_len: Maximum sequence length
+    full_embeddings: Whether to return full residue-wise (True) embeddings or pooled (False)
+    pooling_type: Type of pooling ('mean' or 'cls')
+    sql: Whether to store embeddings in SQLite database - will be stored in float32
+    sql_db_path: Path to SQLite database
+    
+Returns:
+    Dictionary mapping sequences to embeddings, or None if sql=True
+
+Note:
+    - If sql=True, embeddings can only be stored in float32
+    - sql is ideal if you need to stream a very large dataset for training in real-time
+    - save=True is ideal if you can store the entire embedding dictionary in RAM
+    - sql will be used if it is True and save is True or False
+    - If your sql database or .pth file is already present, they will be scanned first for already embedded sequences
+    - Sequences will be truncated to max_len and sorted by length in descending order for faster processing
+```
+
+## Fine-tuning with 🤗 peft
+```python
+model = AutoModelForSequenceClassification.from_pretrained('Synthyra/Profluent-E1-150M', num_labels=2, trust_remote_code=True)
+# these modules handle E1 attention layers
+target_modules = ["q_proj", "k_proj", "v_proj", "o_proj"]
+
+lora_config = LoraConfig(
+    r=8, # choose lora parameters to your liking
+    lora_alpha=16,
+    lora_dropout=0.01,
+    bias="none",
+    target_modules=target_modules,
+)
+
+# Apply LoRA to the model
+model = get_peft_model(model, lora_config)
+
+# Unfreeze the classifier head
+for param in model.classifier.parameters():
+    param.requires_grad = True
+```
+
+For a more thourough example of fine-tuning, check out our example script [here](https://github.com/Synthyra/FastPLMs/blob/main/fine_tuning_example.py).
+
+
+### Citations
+
+```bibtex
+@misc{FastPLMs,
+  author={Hallee, Logan and Bichara, David and Gleghorn, Jason P.},
+  title={FastPLMs: Fast, efficient, protein language model inference from Huggingface AutoModel.},
+  year={2024},
+  url={https://huggingface.co/Synthyra/ESMplusplus_small},
+  DOI={10.57967/hf/3726},
+  publisher={Hugging Face}
+}
+```
+
+```bibtex
+@article{jain2025e1,
+  title={E1: Retrieval-Augmented Protein Encoder Models},
+  author={Jain, Sarthak and Beazer, Joel and Ruffolo, Jeffrey A and Bhatnagar, Aadyot and Madani, Ali},
+  journal={bioRxiv},
+  DOI={10.1101/2025.11.12.688125},
+  year={2025}
+}
+```
+
+```bibtex
+@article{dong2024flexattention,
+  title={Flex Attention: A Programming Model for Generating Optimized Attention Kernels},
+  author={Dong, Juechu and Feng, Boyuan and Guessous, Driss and Liang, Yanbo and He, Horace},
+  journal={arXiv preprint arXiv:2412.05496},
+  year={2024}
+}
+```
+
+```bibtex
+@inproceedings{paszke2019pytorch,
+  title={PyTorch: An Imperative Style, High-Performance Deep Learning Library},
+  author={Paszke, Adam and Gross, Sam and Massa, Francisco and Lerer, Adam and Bradbury, James and Chanan, Gregory and Killeen, Trevor and Lin, Zeming and Gimelshein, Natalia and Antiga, Luca and Desmaison, Alban and K{\"o}pf, Andreas and Yang, Edward and DeVito, Zach and Raison, Martin and Tejani, Alykhan and Chilamkurthy, Sasank and Steiner, Benoit and Fang, Lu and Bai, Junjie and Chintala, Soumith},
+  booktitle={Advances in Neural Information Processing Systems 32},
+  year={2019}
+}
 ```