ebetica commited on
Commit
89c554c
·
verified ·
1 Parent(s): 4eefcf0

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +0 -10
README.md CHANGED
@@ -43,7 +43,6 @@ pip install esm@git+https://github.com/Biohub/esm.git@c94ed8d
43
  ```
44
 
45
  ```py
46
- # pip install esm@git+https://github.com/Biohub/esm.git@c94ed8d
47
  import torch
48
  from transformers import AutoModelForMaskedLM, AutoTokenizer
49
 
@@ -65,7 +64,6 @@ print(f"logits shape: {tuple(output.logits.shape)}")
65
  By default, the model returns only the final layer representations. To return hidden states from **all transformer layers**, set:
66
 
67
  ```py
68
- # pip install esm@git+https://github.com/Biohub/esm.git@c94ed8d
69
  output = model(**inputs, output_hidden_states=True)
70
  ```
71
 
@@ -147,7 +145,6 @@ Performance metrics are detailed in our [ESMC & ESMFold2 paper](https://biohub.a
147
  Instead of scaled dot product attention (sdpa) you can use a flash attention backend. This requires running the model in bfloat16.
148
 
149
  ```py
150
- # pip install esm@git+https://github.com/Biohub/esm.git@c94ed8d
151
  model = (
152
  AutoModelForMaskedLM.from_pretrained(
153
  "biohub/ESMC-6B",
@@ -169,7 +166,6 @@ To get interpretable features from ESMC 6B hidden states and per-layer MLP outpu
169
  * [ESMC SAEs for MLP outputs (all layers)](https://huggingface.co/collections/biohub/esmc-saes-for-mlp-outputs-all-layers)
170
 
171
  ```py
172
- # pip install esm@git+https://github.com/Biohub/esm.git@c94ed8d
173
  import torch
174
  from transformers import AutoModel, AutoTokenizer
175
 
@@ -202,7 +198,6 @@ for i, sae_out in enumerate(output.sae_outputs):
202
  ESMC can predict masked amino acids and compute the corresponding loss:
203
 
204
  ```py
205
- # pip install esm@git+https://github.com/Biohub/esm.git@c94ed8d
206
  import torch
207
  from transformers import AutoModelForMaskedLM, AutoTokenizer
208
 
@@ -229,7 +224,6 @@ print(f"Loss: {output.loss.item():.6f}")
229
  ### Fine-tuning with peft
230
 
231
  ```py
232
- # pip install esm@git+https://github.com/Biohub/esm.git@c94ed8d
233
  from peft import LoraConfig, get_peft_model
234
  from transformers import AutoModelForMaskedLM
235
 
@@ -251,7 +245,6 @@ model.print_trainable_parameters()
251
  To extract attention maps, pass `output_attentions=True`. Note: this is incompatible with `attn_implementation="flash_attention_2"`.
252
 
253
  ```py
254
- # pip install esm@git+https://github.com/Biohub/esm.git@c94ed8d
255
  output = model(**inputs, output_attentions=True)
256
  # output.attentions: tuple of (batch, n_heads, seq_len, seq_len) tensors, one per layer
257
  ```
@@ -263,7 +256,6 @@ output = model(**inputs, output_attentions=True)
263
  You can access the base model without the pretrained LM head:
264
 
265
  ```py
266
- # pip install esm@git+https://github.com/Biohub/esm.git@c94ed8d
267
  import torch
268
  from transformers import AutoModel, AutoTokenizer
269
 
@@ -284,7 +276,6 @@ print(f"last_hidden_state shape: {tuple(output.last_hidden_state.shape)}")
284
  Or use ESMC for Token Classification:
285
 
286
  ```py
287
- # pip install esm@git+https://github.com/Biohub/esm.git@c94ed8d
288
  import torch
289
  from transformers import AutoModelForTokenClassification, AutoTokenizer
290
 
@@ -312,7 +303,6 @@ print(f"first 8 predicted classes: {predicted_tokens_classes[:8]}")
312
  or Sequence Classification:
313
 
314
  ```py
315
- # pip install esm@git+https://github.com/Biohub/esm.git@c94ed8d
316
  import torch
317
  from transformers import AutoModelForSequenceClassification, AutoTokenizer
318
 
 
43
  ```
44
 
45
  ```py
 
46
  import torch
47
  from transformers import AutoModelForMaskedLM, AutoTokenizer
48
 
 
64
  By default, the model returns only the final layer representations. To return hidden states from **all transformer layers**, set:
65
 
66
  ```py
 
67
  output = model(**inputs, output_hidden_states=True)
68
  ```
69
 
 
145
  Instead of scaled dot product attention (sdpa) you can use a flash attention backend. This requires running the model in bfloat16.
146
 
147
  ```py
 
148
  model = (
149
  AutoModelForMaskedLM.from_pretrained(
150
  "biohub/ESMC-6B",
 
166
  * [ESMC SAEs for MLP outputs (all layers)](https://huggingface.co/collections/biohub/esmc-saes-for-mlp-outputs-all-layers)
167
 
168
  ```py
 
169
  import torch
170
  from transformers import AutoModel, AutoTokenizer
171
 
 
198
  ESMC can predict masked amino acids and compute the corresponding loss:
199
 
200
  ```py
 
201
  import torch
202
  from transformers import AutoModelForMaskedLM, AutoTokenizer
203
 
 
224
  ### Fine-tuning with peft
225
 
226
  ```py
 
227
  from peft import LoraConfig, get_peft_model
228
  from transformers import AutoModelForMaskedLM
229
 
 
245
  To extract attention maps, pass `output_attentions=True`. Note: this is incompatible with `attn_implementation="flash_attention_2"`.
246
 
247
  ```py
 
248
  output = model(**inputs, output_attentions=True)
249
  # output.attentions: tuple of (batch, n_heads, seq_len, seq_len) tensors, one per layer
250
  ```
 
256
  You can access the base model without the pretrained LM head:
257
 
258
  ```py
 
259
  import torch
260
  from transformers import AutoModel, AutoTokenizer
261
 
 
276
  Or use ESMC for Token Classification:
277
 
278
  ```py
 
279
  import torch
280
  from transformers import AutoModelForTokenClassification, AutoTokenizer
281
 
 
303
  or Sequence Classification:
304
 
305
  ```py
 
306
  import torch
307
  from transformers import AutoModelForSequenceClassification, AutoTokenizer
308