| library_name: pytorch | |
| tags: | |
| - multimodal | |
| - histology | |
| - codex | |
| - retrieval | |
| license: other | |
| # Haiku — Trimodal (CODEX + H&E + Text) Retrieval Model | |
| This repo bundles a fine-tuned Haiku checkpoint together with the tokenizer | |
| and marker assets needed to run inference without any additional downloads | |
| from `xiangjx/musk` or `microsoft/BiomedNLP-BiomedBERT-*`. | |
| ## Contents | |
| - `haiku_state_dict.pt` — model weights (CODEX + H&E + Text encoders + projections) | |
| - `config.json` — architecture config + marker lists | |
| - `tokenizer/` — BiomedBERT tokenizer files (+ bert config) | |
| - `esm_embeddings/` — per-biomarker ESM embeddings (also embedded in state_dict; kept here for downstream use) | |
| - `vocab.pkl` — marker vocabulary | |
| ## Quick start | |
| ```python | |
| from models import Haiku | |
| model, tokenizer, marker_embedding = Haiku.from_pretrained( | |
| "zhihuanglab/Haiku", | |
| device="cuda", | |
| token="hf_...", # omit if HF_TOKEN / hf auth login is set | |
| ) | |
| model.eval() | |
| ``` | |