Fix shared-space dimension: 2048, not 512
Browse filesThe checkpoint's projection_dim is 2048 and all three projection heads output 2048, as stated in the paper (Sec. 3, z in R^2048); commit 23107503 had changed the card to 512. Also note that image patches and expression profiles need their own input preparation, with a pointer to the GitHub README examples.
README.md
CHANGED
|
@@ -14,7 +14,7 @@ library_name: pytorch
|
|
| 14 |
|
| 15 |
# SpatialWhisperer
|
| 16 |
|
| 17 |
-
A trimodal embedding model that aligns hematoxylin & eosin (H&E) image patches, gene-expression profiles, and free-text descriptions into a shared
|
| 18 |
|
| 19 |
This checkpoint (seed 0) is from the ICML 2026 paper *[Transitive Representation Learning Enhances Histopathology Annotation](https://openreview.net/forum?id=Ze7U293Zw4)* (Schaefer et al., PMLR vol. 306). The paper refers to this configuration as the **Trimodal** model: three encoders trained on two paired-modality datasets (transcriptome↔text and image↔transcriptome) that together span three modalities.
|
| 20 |
|
|
@@ -29,7 +29,7 @@ This checkpoint (seed 0) is from the ICML 2026 paper *[Transitive Representation
|
|
| 29 |
| Transcriptome | Geneformer-12L-30M | locked |
|
| 30 |
| Text | BioBERT v1.1 | trained |
|
| 31 |
|
| 32 |
-
Three projection heads map each encoder's pooled features into a shared
|
| 33 |
|
| 34 |
## Training data
|
| 35 |
|
|
@@ -60,7 +60,7 @@ model, tokenizer, transcriptome_processor, image_processor = load_spatialwhisper
|
|
| 60 |
|
| 61 |
First call downloads the SpatialWhisperer checkpoint plus UNI2 and Geneformer weights; subsequent calls load from cache.
|
| 62 |
|
| 63 |
-
Each `get_<modality>_features` call returns `(pooled_features, projected_embed_in_shared_space)`. The second element is the
|
| 64 |
|
| 65 |
```python
|
| 66 |
import torch
|
|
@@ -70,10 +70,10 @@ text_inputs = tokenizer(prompts, return_tensors="pt", padding=True).to(model.dev
|
|
| 70 |
|
| 71 |
with torch.no_grad():
|
| 72 |
_, text_emb = model.model.get_text_features(normalize_embeds=True, **text_inputs)
|
| 73 |
-
print(text_emb.shape) # (2,
|
| 74 |
```
|
| 75 |
|
| 76 |
-
|
| 77 |
|
| 78 |
## Foundation-model setup
|
| 79 |
|
|
|
|
| 14 |
|
| 15 |
# SpatialWhisperer
|
| 16 |
|
| 17 |
+
A trimodal embedding model that aligns hematoxylin & eosin (H&E) image patches, gene-expression profiles, and free-text descriptions into a shared 2048-dimensional space. Enables zero-shot cell-type annotation of H&E patches and natural-language querying over histopathology and spatial transcriptomics data.
|
| 18 |
|
| 19 |
This checkpoint (seed 0) is from the ICML 2026 paper *[Transitive Representation Learning Enhances Histopathology Annotation](https://openreview.net/forum?id=Ze7U293Zw4)* (Schaefer et al., PMLR vol. 306). The paper refers to this configuration as the **Trimodal** model: three encoders trained on two paired-modality datasets (transcriptome↔text and image↔transcriptome) that together span three modalities.
|
| 20 |
|
|
|
|
| 29 |
| Transcriptome | Geneformer-12L-30M | locked |
|
| 30 |
| Text | BioBERT v1.1 | trained |
|
| 31 |
|
| 32 |
+
Three projection heads map each encoder's pooled features into a shared 2048-dimensional space (`projection_dim=2048`). Only the text tower and the three projection heads are trained.
|
| 33 |
|
| 34 |
## Training data
|
| 35 |
|
|
|
|
| 60 |
|
| 61 |
First call downloads the SpatialWhisperer checkpoint plus UNI2 and Geneformer weights; subsequent calls load from cache.
|
| 62 |
|
| 63 |
+
Each `get_<modality>_features` call returns `(pooled_features, projected_embed_in_shared_space)`. The second element is the 2048-D shared-space embedding to compare across modalities.
|
| 64 |
|
| 65 |
```python
|
| 66 |
import torch
|
|
|
|
| 70 |
|
| 71 |
with torch.no_grad():
|
| 72 |
_, text_emb = model.model.get_text_features(normalize_embeds=True, **text_inputs)
|
| 73 |
+
print(text_emb.shape) # torch.Size([2, 2048])
|
| 74 |
```
|
| 75 |
|
| 76 |
+
H&E image patches and gene-expression profiles need their own input preparation (224 × 224 px ImageNet-normalized patches; an AnnData of raw counts); see the [GitHub README](https://github.com/zinagoodlab/spatialwhisperer#use-the-model) for examples covering all three modalities.
|
| 77 |
|
| 78 |
## Foundation-model setup
|
| 79 |
|