Spatialwhisperer

--- license: cc-by-nc-4.0 tags: - histopathology - spatial-transcriptomics - multimodal - vision-language - clip - cell-type-annotation library_name: pytorch pipeline_tag: zero-shot-image-classification --- # SpatialWhisperer SpatialWhisperer is a trimodal embedding model that aligns hematoxylin & eosin (H&E) image patches, gene-expression profiles, and free-text descriptions into a shared 2048-dimensional space. It enables zero-shot cell-type annotation of H&E patches and natural-language querying over histopathology and spatial transcriptomics data. This repository hosts the main checkpoint (seed=0) from the ICML 2026 paper *Trimodal Learning Enhances Zero-Shot Histopathology Annotation* (anonymized name `\ourmethod`). ## Model architecture Three encoders project into a shared embedding space: | Modality | Encoder | Freezing | |----------------|------------------|----------| | Image (H&E) | UNI2 | locked | | Transcriptome | Geneformer (12L) | locked | | Text | BioBERT v1.1 | unfrozen | Following LiT convention, the freezing pattern is **LUL** (image locked, text unlocked, transcriptome locked). Only the text tower and the three projection heads are trained. Projection dimension is 2048. ## Training data Three paired datasets cover the three modality pairs: - **HEST-1K** — H&E ↔ spatial gene expression (Visium-style spots) - **cellxgene_census** — gene expression ↔ free-text cell/sample metadata - **ARCHS4/GEO** — gene expression ↔ free-text sample descriptions Training was 4 epochs with AdamW at learning rate 1e-5 and cosine schedule (warmup 3%), batch size 512, on a single H100 GPU. This checkpoint reflects epoch 3, global step 14624. ## Evaluation Reported AUROC on cell-type benchmarks (mean across cell types): | Benchmark | SpatialWhisperer | Best published baseline | Δ rel. | |-----------|------------------|-------------------------|---------| | PathoCell | **0.630** | 0.554 | +13.7% | | Lizard | (see paper) | — | +15.9% | | PanNuke | (see paper) | — | +13.7% | Modality-pair benchmarks (Tabula Sapiens, HEST-1K, Skin Conditions) confirm the trimodal model retains per-pair performance under low-n subsampling. See the paper for full numbers. ## How to use The checkpoint is a stripped Lightning state-dict (~505 MB, 236 tensors covering the trained BioBERT text tower and the three 2048-d projection heads) plus its `hyper_parameters...