| --- |
| license: mit |
| tags: |
| - protein |
| - biology |
| - language-model |
| - causal-lm |
| - hyena |
| - pytorch |
| pipeline_tag: feature-extraction |
| --- |
| |
| # Fela |
|
|
| PyTorch written protein language model on the hyena operator (1.6M params) |
|
|
| - Architecture: long conv + MLP blocks, pre-norm, LM head |
| - Tokenizer: char level over `ACDEFGHIKLMNPQRSTVWYX`, `<pad>`=0, `<eos>`=22, `<unk>`=23 |
| - Data: Pfam-A (filtered to 20–512 residues, standard alphabet only), ~9.5B tokens |
| - Training: 40k steps, batch 256, bf16, AdamW (wd 0.1), cosine LR 6e-4 → 6e-5 |
|
|
| based on the paper: https://www.biorxiv.org/content/10.1101/2024.01.18.576206v1 |
|
|
| base model (not finetuned) |
|
|
| ## Config |
|
|
| | Parameter | Value | |
| |---|---| |
| | d_model | 256 | |
| | n_layer | 2 | |
| | d_inner | 1024 | |
| | vocab_size | 32 | |
| | l_max | 514 | |
| | order | 2 | |
| | filter_order | 64 | |
| | short_filter_order | 3 | |
| | emb_dim | 5 | |
| | w | 10 | |
| | num_inner_mlps | 2 | |
| | residual_in_fp32 | true | |
| |
| ## Usage |
| |
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| model = AutoModelForCausalLM.from_pretrained("pandeyps/fela", trust_remote_code=True) |
| tok = AutoTokenizer.from_pretrained("pandeyps/fela", trust_remote_code=True) |
| |
| ids = tok.encode("MSDKIIEYDETARRAIEAGVNTLADAV", return_tensors="pt") |
| gen = model.generate(ids, max_new_tokens=64, do_sample=True, temperature=0.7) |
| print(tok.decode(gen[0])) |
| |
