lhallee commited on
Commit
c8617e1
·
verified ·
1 Parent(s): f5c753b

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +202 -8
README.md CHANGED
@@ -1,11 +1,11 @@
1
  ---
2
  library_name: transformers
3
  tags:
4
- - biology
5
- - esm
6
- - protein
7
- - protein-language-model
8
- - masked-language-modeling
9
  ---
10
 
11
  # ESM++ 6B
@@ -13,26 +13,207 @@ tags:
13
  [ESM++](https://github.com/Synthyra/FastPLMs) is a Hugging Face compatible implementation of [Biohub ESMC](https://biohub.ai/esm/protein) ([license](https://github.com/Biohub/esm/blob/main/LICENSE.md)).
14
  This checkpoint corresponds to the 6 billion parameter ESMC model released as [`biohub/ESMC-6B`](https://huggingface.co/biohub/ESMC-6B).
15
 
 
 
 
 
 
 
 
 
 
 
 
 
 
16
  ```python
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
17
  from transformers import AutoModelForMaskedLM
18
 
19
  model = AutoModelForMaskedLM.from_pretrained(
20
  "Synthyra/ESMplusplus_6B",
21
  trust_remote_code=True,
 
22
  device_map="auto",
23
  )
24
  tokenizer = model.tokenizer
25
 
26
  sequences = ["MPRTEIN", "MSEQWENCE"]
27
  inputs = tokenizer(sequences, padding=True, return_tensors="pt")
28
- inputs = {key: value.to(model.device) for key, value in inputs.items()}
 
 
 
29
 
30
- output = model(**inputs)
31
  print(output.logits.shape)
32
  print(output.last_hidden_state.shape)
33
  ```
34
 
35
- To load the Biohub source weights directly through FastPLMs:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
36
 
37
  ```python
38
  from fastplms.esm_plusplus.modeling_esm_plusplus import ESMplusplusForMaskedLM
@@ -40,8 +221,21 @@ from fastplms.esm_plusplus.modeling_esm_plusplus import ESMplusplusForMaskedLM
40
  model = ESMplusplusForMaskedLM.from_pretrained_esm("esmc-6b")
41
  ```
42
 
 
 
43
  ## Citation
44
 
 
 
 
 
 
 
 
 
 
 
 
45
  ```bibtex
46
  @misc{candido2026language,
47
  title = {Language Modeling Materializes a World Model of Protein Biology},
 
1
  ---
2
  library_name: transformers
3
  tags:
4
+ - biology
5
+ - esm
6
+ - protein
7
+ - protein-language-model
8
+ - masked-language-modeling
9
  ---
10
 
11
  # ESM++ 6B
 
13
  [ESM++](https://github.com/Synthyra/FastPLMs) is a Hugging Face compatible implementation of [Biohub ESMC](https://biohub.ai/esm/protein) ([license](https://github.com/Biohub/esm/blob/main/LICENSE.md)).
14
  This checkpoint corresponds to the 6 billion parameter ESMC model released as [`biohub/ESMC-6B`](https://huggingface.co/biohub/ESMC-6B).
15
 
16
+ The 6B model has 80 transformer layers, hidden size 2560, and 40 attention heads. It is large enough that `dtype=torch.bfloat16` or `torch.float16` plus `device_map="auto"` is usually the practical loading path.
17
+
18
+ ## Attention Backends
19
+
20
+ `sdpa` is the default backend. Set `config.attn_backend` before loading if you want a different attention implementation.
21
+
22
+ | Backend | Key | Notes |
23
+ | :--- | :--- | :--- |
24
+ | PyTorch SDPA | `"sdpa"` | Default. Exact numerics and stable on all hardware. |
25
+ | Flash Attention | `"kernels_flash"` | Fastest on Ampere/Hopper GPUs when `kernels` is installed. Outputs are not bitwise identical to SDPA. |
26
+ | Flex Attention | `"flex"` | Skips padding tokens via block masks. First use compiles a Triton kernel. |
27
+ | Auto | `"auto"` | Picks the best available backend: `kernels_flash`, then `flex`, then `sdpa`. |
28
+
29
  ```python
30
+ import torch
31
+ from transformers import AutoConfig, AutoModelForMaskedLM
32
+
33
+ config = AutoConfig.from_pretrained(
34
+ "Synthyra/ESMplusplus_6B",
35
+ trust_remote_code=True,
36
+ )
37
+ config.attn_backend = "auto"
38
+
39
+ model = AutoModelForMaskedLM.from_pretrained(
40
+ "Synthyra/ESMplusplus_6B",
41
+ config=config,
42
+ trust_remote_code=True,
43
+ dtype=torch.bfloat16,
44
+ device_map="auto",
45
+ )
46
+ ```
47
+
48
+ ## Masked Language Modeling
49
+
50
+ ```python
51
+ import torch
52
  from transformers import AutoModelForMaskedLM
53
 
54
  model = AutoModelForMaskedLM.from_pretrained(
55
  "Synthyra/ESMplusplus_6B",
56
  trust_remote_code=True,
57
+ dtype=torch.bfloat16,
58
  device_map="auto",
59
  )
60
  tokenizer = model.tokenizer
61
 
62
  sequences = ["MPRTEIN", "MSEQWENCE"]
63
  inputs = tokenizer(sequences, padding=True, return_tensors="pt")
64
+ inputs = inputs.to(model.device)
65
+
66
+ with torch.no_grad():
67
+ output = model(**inputs)
68
 
 
69
  print(output.logits.shape)
70
  print(output.last_hidden_state.shape)
71
  ```
72
 
73
+ Pass `output_hidden_states=True` if you need all intermediate hidden states.
74
+
75
+ ## Embed Datasets
76
+
77
+ All FastPLMs sequence models include `embed_dataset`, which handles batching, length sorting, pooling, FASTA parsing, optional resume from existing outputs, and `.pth` or SQLite storage.
78
+
79
+ ```python
80
+ import torch
81
+ from transformers import AutoModelForMaskedLM
82
+
83
+ model = AutoModelForMaskedLM.from_pretrained(
84
+ "Synthyra/ESMplusplus_6B",
85
+ trust_remote_code=True,
86
+ dtype=torch.bfloat16,
87
+ device_map="auto",
88
+ )
89
+
90
+ embedding_dict = model.embed_dataset(
91
+ sequences=[
92
+ "MALWMRLLPLLALLALWGPDPAAA",
93
+ "MSEQWENCE",
94
+ "MPRTEIN",
95
+ ],
96
+ tokenizer=model.tokenizer,
97
+ batch_size=1,
98
+ max_len=1024,
99
+ full_embeddings=False,
100
+ embed_dtype=torch.float32,
101
+ pooling_types=["mean", "cls"],
102
+ num_workers=0,
103
+ save=True,
104
+ save_path="esmplusplus_6b_embeddings.pth",
105
+ )
106
+
107
+ print(embedding_dict["MPRTEIN"].shape)
108
+ ```
109
+
110
+ For residue-level embeddings, set `full_embeddings=True`:
111
+
112
+ ```python
113
+ residue_embeddings = model.embed_dataset(
114
+ sequences=["MALWMRLLPLLALLALWGPDPAAA"],
115
+ tokenizer=model.tokenizer,
116
+ batch_size=1,
117
+ max_len=1024,
118
+ full_embeddings=True,
119
+ embed_dtype=torch.float32,
120
+ save=False,
121
+ )
122
+ ```
123
+
124
+ For very large datasets, write embeddings directly to SQLite:
125
+
126
+ ```python
127
+ model.embed_dataset(
128
+ fasta_path="proteins.fasta",
129
+ tokenizer=model.tokenizer,
130
+ batch_size=1,
131
+ max_len=1024,
132
+ pooling_types=["mean"],
133
+ sql=True,
134
+ sql_db_path="esmplusplus_6b_embeddings.db",
135
+ save=False,
136
+ )
137
+ ```
138
+
139
+ `embed_dataset` returns a dictionary when `sql=False`. With `sql=True`, embeddings are written to the database and loaded as needed.
140
+
141
+ ## Classification Heads
142
+
143
+ ESM++ supports sequence-level and token-level classification through the standard Transformers auto classes.
144
+
145
+ ```python
146
+ import torch
147
+ from transformers import AutoModelForSequenceClassification
148
+
149
+ model = AutoModelForSequenceClassification.from_pretrained(
150
+ "Synthyra/ESMplusplus_6B",
151
+ num_labels=2,
152
+ trust_remote_code=True,
153
+ dtype=torch.bfloat16,
154
+ device_map="auto",
155
+ )
156
+
157
+ tokenized = model.tokenizer(
158
+ ["MPRTEIN", "MSEQWENCE"],
159
+ padding=True,
160
+ return_tensors="pt",
161
+ ).to(model.device)
162
+
163
+ with torch.no_grad():
164
+ logits = model(**tokenized).logits
165
+
166
+ print(logits.shape)
167
+ ```
168
+
169
+ ## LoRA Fine-Tuning
170
+
171
+ ```python
172
+ from peft import LoraConfig, get_peft_model
173
+ from transformers import AutoModelForSequenceClassification
174
+
175
+ model = AutoModelForSequenceClassification.from_pretrained(
176
+ "Synthyra/ESMplusplus_6B",
177
+ num_labels=2,
178
+ trust_remote_code=True,
179
+ dtype=torch.bfloat16,
180
+ device_map="auto",
181
+ )
182
+
183
+ lora_config = LoraConfig(
184
+ r=8,
185
+ lora_alpha=16,
186
+ lora_dropout=0.01,
187
+ bias="none",
188
+ target_modules=[
189
+ "layernorm_qkv.1",
190
+ "out_proj",
191
+ "query",
192
+ "key",
193
+ "value",
194
+ "dense",
195
+ ],
196
+ )
197
+
198
+ model = get_peft_model(model, lora_config)
199
+ ```
200
+
201
+ ## Attention Maps
202
+
203
+ Optimized attention backends do not return attention maps directly. ESM++ can compute them manually with `output_attentions=True`, but this is much slower and memory-heavy for the 6B model.
204
+
205
+ ```python
206
+ with torch.no_grad():
207
+ output = model(**inputs, output_attentions=True)
208
+
209
+ attentions = output.attentions
210
+ print(len(attentions))
211
+ print(attentions[0].shape)
212
+ ```
213
+
214
+ ## Load Biohub Source Weights
215
+
216
+ You can also load the Biohub source weights directly through FastPLMs:
217
 
218
  ```python
219
  from fastplms.esm_plusplus.modeling_esm_plusplus import ESMplusplusForMaskedLM
 
221
  model = ESMplusplusForMaskedLM.from_pretrained_esm("esmc-6b")
222
  ```
223
 
224
+ The source repository is [`biohub/ESMC-6B`](https://huggingface.co/biohub/ESMC-6B).
225
+
226
  ## Citation
227
 
228
+ ```bibtex
229
+ @misc{FastPLMs,
230
+ author={Hallee, Logan and Bichara, David and Gleghorn, Jason P.},
231
+ title={FastPLMs: Fast, efficient, protein language model inference from Huggingface AutoModel.},
232
+ year={2024},
233
+ url={https://huggingface.co/Synthyra/ESMplusplus_6B},
234
+ DOI={10.57967/hf/3726},
235
+ publisher={Hugging Face}
236
+ }
237
+ ```
238
+
239
  ```bibtex
240
  @misc{candido2026language,
241
  title = {Language Modeling Materializes a World Model of Protein Biology},