Good-Lab commited on
Commit
de57fd2
·
verified ·
1 Parent(s): 2310750

Fix shared-space dimension: 2048, not 512

Browse files

The checkpoint's projection_dim is 2048 and all three projection heads output 2048, as stated in the paper (Sec. 3, z in R^2048); commit 23107503 had changed the card to 512. Also note that image patches and expression profiles need their own input preparation, with a pointer to the GitHub README examples.

Files changed (1) hide show
  1. README.md +5 -5
README.md CHANGED
@@ -14,7 +14,7 @@ library_name: pytorch
14
 
15
  # SpatialWhisperer
16
 
17
- A trimodal embedding model that aligns hematoxylin & eosin (H&E) image patches, gene-expression profiles, and free-text descriptions into a shared 512-dimensional space. Enables zero-shot cell-type annotation of H&E patches and natural-language querying over histopathology and spatial transcriptomics data.
18
 
19
  This checkpoint (seed 0) is from the ICML 2026 paper *[Transitive Representation Learning Enhances Histopathology Annotation](https://openreview.net/forum?id=Ze7U293Zw4)* (Schaefer et al., PMLR vol. 306). The paper refers to this configuration as the **Trimodal** model: three encoders trained on two paired-modality datasets (transcriptome↔text and image↔transcriptome) that together span three modalities.
20
 
@@ -29,7 +29,7 @@ This checkpoint (seed 0) is from the ICML 2026 paper *[Transitive Representation
29
  | Transcriptome | Geneformer-12L-30M | locked |
30
  | Text | BioBERT v1.1 | trained |
31
 
32
- Three projection heads map each encoder's pooled features into a shared 512-dimensional space. Only the text tower and the three projection heads are trained.
33
 
34
  ## Training data
35
 
@@ -60,7 +60,7 @@ model, tokenizer, transcriptome_processor, image_processor = load_spatialwhisper
60
 
61
  First call downloads the SpatialWhisperer checkpoint plus UNI2 and Geneformer weights; subsequent calls load from cache.
62
 
63
- Each `get_<modality>_features` call returns `(pooled_features, projected_embed_in_shared_space)`. The second element is the 512-D shared-space embedding to compare across modalities.
64
 
65
  ```python
66
  import torch
@@ -70,10 +70,10 @@ text_inputs = tokenizer(prompts, return_tensors="pt", padding=True).to(model.dev
70
 
71
  with torch.no_grad():
72
  _, text_emb = model.model.get_text_features(normalize_embeds=True, **text_inputs)
73
- print(text_emb.shape) # (2, 512)
74
  ```
75
 
76
- Image and transcriptome embeddings follow the same pattern; see the [GitHub README](https://github.com/zinagoodlab/spatialwhisperer#use-the-model) for complete examples covering all three modalities.
77
 
78
  ## Foundation-model setup
79
 
 
14
 
15
  # SpatialWhisperer
16
 
17
+ A trimodal embedding model that aligns hematoxylin & eosin (H&E) image patches, gene-expression profiles, and free-text descriptions into a shared 2048-dimensional space. Enables zero-shot cell-type annotation of H&E patches and natural-language querying over histopathology and spatial transcriptomics data.
18
 
19
  This checkpoint (seed 0) is from the ICML 2026 paper *[Transitive Representation Learning Enhances Histopathology Annotation](https://openreview.net/forum?id=Ze7U293Zw4)* (Schaefer et al., PMLR vol. 306). The paper refers to this configuration as the **Trimodal** model: three encoders trained on two paired-modality datasets (transcriptome↔text and image↔transcriptome) that together span three modalities.
20
 
 
29
  | Transcriptome | Geneformer-12L-30M | locked |
30
  | Text | BioBERT v1.1 | trained |
31
 
32
+ Three projection heads map each encoder's pooled features into a shared 2048-dimensional space (`projection_dim=2048`). Only the text tower and the three projection heads are trained.
33
 
34
  ## Training data
35
 
 
60
 
61
  First call downloads the SpatialWhisperer checkpoint plus UNI2 and Geneformer weights; subsequent calls load from cache.
62
 
63
+ Each `get_<modality>_features` call returns `(pooled_features, projected_embed_in_shared_space)`. The second element is the 2048-D shared-space embedding to compare across modalities.
64
 
65
  ```python
66
  import torch
 
70
 
71
  with torch.no_grad():
72
  _, text_emb = model.model.get_text_features(normalize_embeds=True, **text_inputs)
73
+ print(text_emb.shape) # torch.Size([2, 2048])
74
  ```
75
 
76
+ H&E image patches and gene-expression profiles need their own input preparation (224 × 224 px ImageNet-normalized patches; an AnnData of raw counts); see the [GitHub README](https://github.com/zinagoodlab/spatialwhisperer#use-the-model) for examples covering all three modalities.
77
 
78
  ## Foundation-model setup
79