README.md

---
tags:
  - biology
  - immunology
  - tcr
  - t-cell-receptor
  - tokenizer
---

# tcr-vtoken tokenizer

A residue-level TCR tokenizer where **each V gene is one atomic token**. Vocab = 26 base tokens (20 amino
acids + specials) **+ 101 tidytcells-standardised V-gene tokens** (`[V:TRBV20-1]`, `[V:TRAV1-2]`, …) + `[V_UNK]`
= **128** total. It is built for the input grammar `[V:gene] | CDR3`, where the atomic V token stands in for the
germline CDR1/CDR2 (which the V gene determines).

## Files
Standard HF tokenizer (`tokenizer.json`, `vocab.txt`, `added_tokens.json`, …) **+ `vgene_map.json`** — a map
from a raw V-gene symbol to its atomic token (`"TRBV20-1" -> "[V:TRBV20-1]"`; unknown → `[V_UNK]`).

## Use it

```python
import json
from huggingface_hub import hf_hub_download
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("argentel/tcr-vtoken")                 # base AA + V-gene tokens (128)
vmap = json.load(open(hf_hub_download("argentel/tcr-vtoken", "vgene_map.json")))   # raw V-gene -> token

v_gene, cdr3 = "TRBV20-1", "CASSYSTDTQYF"
vtok = vmap.get(v_gene, "[V_UNK]")                                         # atomic V token
text = f"{vtok} {tok.sep_token} {' '.join(cdr3)}"                          # "[V:TRBV20-1] | C A S S ..."
enc = tok(text)
print(tok.convert_ids_to_tokens(enc["input_ids"]))
# ['*', '[V:TRBV20-1]', '|', 'C', 'A', 'S', 'S', 'Y', 'S', 'T', 'D', 'T', 'Q', 'Y', 'F', '|']
#   *=[CLS] and |=[SEP] are added automatically; [V:TRBV20-1] is ONE atomic token.
print(tok.convert_tokens_to_ids("[V:TRBV20-1]"))                          # 87  (V tokens have ids >= 26)
```

The V-gene tokens live in `added_tokens.json` (ids 26–127), not `vocab.txt` (which is only the 26 base tokens);
`AutoTokenizer` merges both, so `len(tok) == 128`.