H&E Evals

Current Standards:

pluto training

GEX Prediction

Morphology-to-gene-expression prediction. Do our H&E embeddings contain molecular signal beyond morphology shortcuts?

Level 1: predict broad cell type

• should be easily solved from morphology which gives us: nucleus size, cytoplasm, epithelial structure, lymphocyte appearance, stromal shape

Level 2: predict highly variable gene expression at single-cell or spot level

• Gene expression is only partially visible from morphology. Some may track obvious cell identity or tissue compartment, but many reflect activation state, immune signaling, stress, hypoxia, stromal remodeling, or local microenvironment
• as a result raw Pearson is dangerous as there could be major autocorrelation with spatial baseline (kNN). We should compare its performance relative to a kNN baseline.

Level 3: differentiate same cell types across different tissues or states

• Same cells can have different states that should be distinguishable.

  eg:

  • T cells in inflamed tumor stroma vs normal tissue
  • epithelial cells in different tumor states
  • fibroblasts with distinct CAF programs

Segmentation

Standard pathology segmentation benchmarks usually focus on nuclear segmentation:

• find each nucleus
• separate touching nuclei
• optionally classify nuclei type

CoNSeP is an example benchmark for nuclear segmentation.

Cell segmentation asks for the full cell boundary, including cytoplasm and membrane-adjacent region. That is harder because H&E often shows nuclei clearly, but cytoplasm and true cell borders are ambiguous or invisible.

Pluto does multi-scale segmentation (across different magnification):

Patch Classification

Patch classification asks whether frozen H&E embeddings separate broad tissue, disease, or compartment labels from small crops. This is usually easier than GEX or segmentation because many labels are visually obvious.

Level 1: supervised patch classification benchmarks

• MHIST: classify colorectal polyp patches as SSA (precancerous) vs hyperplastic (benign)
• PatchCamelyon: predict whether a lymph-node patch contains metastatic tissue.
• BACH: classify breast histology images as normal, benign, in situ, or invasive.
• Usually the label can be solved from coarse morphology or dataset-specific shortcuts.

Level 2: pathology-derived clustering

• Cluster frozen H&E embeddings and compare clusters to pathologist-derived labels or regions.
• Example labels: tumor, immune, TLS, normal, benign, invasive, stromal regions.
• Metrics: ARI, NMI, cluster purity, or retrieval accuracy.
• This is more representation-like than training a classifier because it asks whether the embedding geometry naturally separates pathology compartments.

Level 3: transcriptomic niche alignment

• Compare morphology-derived embedding clusters against RNA-derived spatial niches.
• Example: use Visium kidney/lung TLS (Tertiary Lymphoid Structures) data or existing BRCA spatial transcriptomics data.
• Compute RNA niches from gene expression, cluster H&E embeddings independently, then compare the two partitions using ARI, NMI, or matched-cluster enrichment.
• This is the most biologically meaningful tier because it asks whether morphology embeddings recover molecular tissue organization, not just visual class labels.

Slide-Level Classification

WSI classification asks whether patch embeddings can be aggregated into a slide-level or patient-level prediction. This is harder than patch classification because the label may depend on a small subset of regions, long-range tissue organization, or tumor microenvironment context.

Dataset: SurGen

• colorectal cancer WSIs with mutation labels for KRAS, NRAS, BRAF, mismatch repair status, and 5-year survival.
• useful for biomarker prediction and prognosis from H&E.

Level 1: slide-level diagnosis or subtype classification

• Predict broad labels such as cancer type, tumor vs normal, or histologic subtype.
• This mostly tests whether patch embeddings can support simple MIL (Multiple Instance Learning) aggregation.

Level 2: molecular biomarker prediction

• Predict KRAS, NRAS, BRAF, or MMR status from the WSI.
• Harder because molecular labels are only indirectly visible through morphology and microenvironment patterns.

Level 3: survival / prognosis prediction

• Predict 5-year overall survival or time-to-event risk from the WSI.
• Hardest and noisiest because survival depends on morphology, molecular state, treatment, stage, and clinical confounders.

Robustness + Multimodal Alignment + Attention Biology

Multimodal Evals:

STORM fuses both H&E and ST modalities and outputs spatial domains (regions of tissue that are biological distinct).

The biological analyses are:

• cluster spatial regions
• run differential expression (identify which genes or proteins are quantitatively more or less active in different conditions) between clusters
• identify enriched pathways (biological processes or functional networks that appear more frequently in a user's target list of genes) like immune, metabolic, hypoxia, stroma, etc

Clinical outcome predictions:

1. Start with a patient H&E slide
2. Use STORM to generate virtual ST
3. Combine H&E + virtual ST for outcome prediction
4. Predict prognosis (expected outcome of disease) or immunotherapy response (enhance body's immune system)
5. Get biological interpretation from virtual molecular maps

Metric uses C-index which measures whether the model correctly ranks patients by risk. (above 0.8 is generally really good)

where $i,j$ are two samples being compared. $T_i, T_j$ is observed time fro each patient (survival time or time to relapse). $\delta_i, \delta_j$ is even indicator, and $s_i, s_j$ is model's predicted risk score for patient.

So essentially just: