File size: 1,326 Bytes
bd4f849 03e4fff bd4f849 6e94b21 bd4f849 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 | ---
license: mit
tags:
- protein
- biology
- language-model
- causal-lm
- hyena
- pytorch
pipeline_tag: feature-extraction
---
# Fela
PyTorch written protein language model on the hyena operator (1.6M params)
- Architecture: long conv + MLP blocks, pre-norm, LM head
- Tokenizer: char level over `ACDEFGHIKLMNPQRSTVWYX`, `<pad>`=0, `<eos>`=22, `<unk>`=23
- Data: Pfam-A (filtered to 20–512 residues, standard alphabet only), ~9.5B tokens
- Training: 40k steps, batch 256, bf16, AdamW (wd 0.1), cosine LR 6e-4 → 6e-5
based on the paper: https://www.biorxiv.org/content/10.1101/2024.01.18.576206v1
base model (not finetuned)
## Config
| Parameter | Value |
|---|---|
| d_model | 256 |
| n_layer | 2 |
| d_inner | 1024 |
| vocab_size | 32 |
| l_max | 514 |
| order | 2 |
| filter_order | 64 |
| short_filter_order | 3 |
| emb_dim | 5 |
| w | 10 |
| num_inner_mlps | 2 |
| residual_in_fp32 | true |
## Usage
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("pandeyps/fela", trust_remote_code=True)
tok = AutoTokenizer.from_pretrained("pandeyps/fela", trust_remote_code=True)
ids = tok.encode("MSDKIIEYDETARRAIEAGVNTLADAV", return_tensors="pt")
gen = model.generate(ids, max_new_tokens=64, do_sample=True, temperature=0.7)
print(tok.decode(gen[0]))
|