File size: 1,808 Bytes
62a60db 7507f09 62a60db | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 | ---
tags:
- biology
- immunology
- tcr
- t-cell-receptor
- tokenizer
---
# tcr-vtoken tokenizer
A residue-level TCR tokenizer where **each V gene is one atomic token**. Vocab = 26 base tokens (20 amino
acids + specials) **+ 101 tidytcells-standardised V-gene tokens** (`[V:TRBV20-1]`, `[V:TRAV1-2]`, …) + `[V_UNK]`
= **128** total. It is built for the input grammar `[V:gene] | CDR3`, where the atomic V token stands in for the
germline CDR1/CDR2 (which the V gene determines).
## Files
Standard HF tokenizer (`tokenizer.json`, `vocab.txt`, `added_tokens.json`, …) **+ `vgene_map.json`** — a map
from a raw V-gene symbol to its atomic token (`"TRBV20-1" -> "[V:TRBV20-1]"`; unknown → `[V_UNK]`).
## Use it
```python
import json
from huggingface_hub import hf_hub_download
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("argentel/tcr-vtoken") # base AA + V-gene tokens (128)
vmap = json.load(open(hf_hub_download("argentel/tcr-vtoken", "vgene_map.json"))) # raw V-gene -> token
v_gene, cdr3 = "TRBV20-1", "CASSYSTDTQYF"
vtok = vmap.get(v_gene, "[V_UNK]") # atomic V token
text = f"{vtok} {tok.sep_token} {' '.join(cdr3)}" # "[V:TRBV20-1] | C A S S ..."
enc = tok(text)
print(tok.convert_ids_to_tokens(enc["input_ids"]))
# ['*', '[V:TRBV20-1]', '|', 'C', 'A', 'S', 'S', 'Y', 'S', 'T', 'D', 'T', 'Q', 'Y', 'F', '|']
# *=[CLS] and |=[SEP] are added automatically; [V:TRBV20-1] is ONE atomic token.
print(tok.convert_tokens_to_ids("[V:TRBV20-1]")) # 87 (V tokens have ids >= 26)
```
The V-gene tokens live in `added_tokens.json` (ids 26–127), not `vocab.txt` (which is only the 26 base tokens);
`AutoTokenizer` merges both, so `len(tok) == 128`.
|