A trimodal embedding model that aligns hematoxylin & eosin (H&E) image patches, gene-expression profiles, and free-text descriptions into a shared 512-dimensional space. Enables zero-shot cell-type annotation of H&E patches and natural-language querying over histopathology and spatial transcriptomics data.
This checkpoint (seed 0) is from the ICML 2026 paper Transitive Representation Learning Enhances Histopathology Annotation (Schaefer et al., PMLR vol. 306). The paper refers to this configuration as the Trimodal model: three encoders trained on two paired-modality datasets (transcriptome↔text and image↔transcriptome) that together span three modalities.
Three projection heads map each encoder's pooled features into a shared 512-dimensional space. Only the text tower and the three projection heads are trained.
Training: 4 epochs, AdamW (lr 1e-5), cosine schedule (3% warmup), batch size 512, single H100. This checkpoint is from epoch 3, global step 14624.
What this checkpoint contains
spatialwhisperer.ckpt — Lightning state-dict (~530 MB, 236 tensors: trained BioBERT text tower + three projection heads) plus the hyper_parameters block. Optimizer/scheduler state is stripped.
The locked foundation-model weights are NOT included. UNI2 and Geneformer are re-instantiated at load time from their original providers. The load_spatialwhisperer_model() convenience wrapper fetches both on first call.
1from spatialwhisperer import load_spatialwhisperer_model
23model, tokenizer, transcriptome_processor, image_processor = load_spatialwhisperer_model()4# model: TranscriptomeTextDualEncoderLightning (frozen, eval mode, on CUDA if available)
First call downloads the SpatialWhisperer checkpoint plus UNI2 and Geneformer weights; subsequent calls load from cache.
Each get_<modality>_features call returns (pooled_features, projected_embed_in_shared_space). The second element is the 512-D shared-space embedding to compare across modalities.
Make a read token visible to your environment — the loader checks HF_TOKEN / HUGGINGFACE_TOKEN, otherwise falls back to huggingface_hub's on-disk cache. The simplest setup is:
huggingface-cli login # paste your read token
Geneformer downloads without gating.
Evaluation
The accompanying code repository reproduces every paper benchmark with one command: pixi run snakemake -j N paper_all. Verified end-to-end on a fresh ILC ampere4 clone (2026-05-31):
These match the paper's reported Trimodal seed-0 numbers exactly. Comparisons to CONCH, PLIP, OmiCLIP, and other baselines are in the paper.
Intended use & limitations
Intended: research on multimodal histopathology, zero-shot cell-type annotation, spatial transcriptomics analysis, and natural-language querying over H&E and gene-expression data.
Not intended: clinical diagnosis or treatment decisions. The model was trained on academic datasets and is not validated for clinical use.
Limitations:
Trained on Visium-scale spots (~55 μm); finer-grained image–expression alignment is not guaranteed.
BioBERT vocabulary constrains the text tower; rare technical terms may be out-of-distribution.
The image tower (UNI2) is locked; performance on tissue types poorly represented in UNI2's pretraining will be lower.
License
CC BY-NC 4.0 (research use). Foundation-model weights (UNI2, Geneformer, BioBERT) carry their own licenses; consult the upstream repositories.
Citation
bibtex
1@inproceedings{schaefer2026spatialwhisperer,
2 title = {Transitive Representation Learning Enhances Histopathology Annotation},
3 author = {Schaefer, Moritz and Piran, Zoe and Walter, Nils Philipp and Awasthi, Animesh and Bock, Christoph and Leskovec, Jure and Good, Zinaida},
4 booktitle = {Proceedings of the 43rd International Conference on Machine Learning},
5 series = {Proceedings of Machine Learning Research},
6 volume = {306},
7 publisher = {PMLR},
8 address = {Seoul, South Korea},
9 month = jul,
10 year = {2026},
11 url = {https://openreview.net/forum?id=Ze7U293Zw4}
12}
