Metrics
Gene expression and segmentation both run on a fixed external eval slide (Xenium breast S1_Top) over a deterministic $10\times10$ window of $150\text{k}$-pixel tiles. Reconstruction is scored separately, on the deterministic first crop of the held-out HISTAI-mixed validation split, so it measures in-distribution pixel fidelity rather than transfer to the Xenium slide.
Reconstruction (MAE)
Let $\mathcal{M}$ be the set of masked tokens, $x_i \in [0,1]^3$ the target RGB of token $i$, and $\hat{x}_i$ the model prediction. All recon metrics restrict to $\mathcal{M}$, since the visible tokens are trivially reconstructed.
Masked L1 / MSE
\[ \text{L1} = \frac{1}{|\mathcal{M}|}\sum_{i \in \mathcal{M}} \lVert \hat{x}_i - x_i \rVert_1, \qquad \text{MSE} = \frac{1}{|\mathcal{M}|}\sum_{i \in \mathcal{M}} \lVert \hat{x}_i - x_i \rVert_2^2 \]raw per-pixel fidelity on the inpainted region. L1 is robust to outliers, MSE penalizes large misses harder.
Masked SSIM
Let $a$ be the original crop and $b$ the filled reconstruction (visible tokens plus predicted masked tokens). For a local window the structural similarity is
\[ \text{SSIM}(a,b) = \frac{(2\mu_a\mu_b + c_1)(2\sigma_{ab} + c_2)}{(\mu_a^2 + \mu_b^2 + c_1)(\sigma_a^2 + \sigma_b^2 + c_2)} \]and we average the per-pixel SSIM map over masked pixels only:
\[ \text{SSIM}_{\mathcal{M}} = \frac{1}{|\mathcal{M}_\text{px}|}\sum_{p \in \mathcal{M}_\text{px}} \text{SSIM-map}(a,b)_p \]this rewards matching local structure (edges, texture) rather than pixel averages. A blurry mean prediction can score low L1 but poor SSIM, which is what catches lost nuclei detail.
Masked L1 split by tissue type
Otsu's threshold $\tau$ on token brightness $b_i = \tfrac{1}{3}\sum_c x_{i,c}$ splits dark hematoxylin-rich tokens (nuclei) from lighter tissue (background):
\[ \text{L1}_\text{nuc} = \operatorname*{mean}_{i \in \mathcal{M},\, b_i < \tau} \lVert \hat{x}_i - x_i \rVert_1, \qquad \text{L1}_\text{bg} = \operatorname*{mean}_{i \in \mathcal{M},\, b_i \ge \tau} \lVert \hat{x}_i - x_i \rVert_1 \]separating these tells us whether the model is just nailing easy white background or actually reconstructing nuclei, which is the part that matters biologically.
Segmentation (attention probe)
Probes whether encoder attention heads are cell-aware. Sample one query token per unique cell, then for every token pair $(p,q)$ within a radius $R$ form the label
\[ y_{pq} = \mathbb{1}\big[\,\text{cell}(p) = \text{cell}(q)\,\big] \]and score it with head $(L,h)$'s attention weight $a^{(L,h)}_{pq}$. We sweep $R \in \{40, 80, 150\}$ px, roughly one cell, two cells, and a small cluster.
Per-head AUC and best head
ROC AUC has the probabilistic reading "chance a same-cell pair outranks a different-cell pair":
\[ \text{AUC}_{L,h}(R) = \Pr\!\big(a^{(L,h)}_{pq} > a^{(L,h)}_{p'q'} \,\big|\, y_{pq}=1,\, y_{p'q'}=0\big) \] \[ \text{best\_auc}(R) = \max_{L,h} \text{AUC}_{L,h}(R) \]the best head over the whole layer $\times$ head grid is the most cell-selective attention pattern the encoder learned.
Learned attention-stack probe
The per-head AUC scores each head independently. We also test whether same-cell structure is decodable from the full attention stack. For each pair ((p,q)), concatenate the attention weights from all layers and heads:
\[ z_{pq}= \big[ a^{(1,1)}*{pq}, a^{(1,2)}*{pq}, \dots, a^{(L,H)}_{pq} \big]. \]A logistic probe is fit on query-cell train pairs to predict the same-cell label,
\[ \hat{y}_{pq} = \sigma(w^\top z_{pq}+b), \qquad y_{pq}=\mathbb{1}\big[,\text{cell}(p)=\text{cell}(q),\big]. \]The probe is evaluated on held-out query cells:
\[ \text{probe\_auc}(R) = \text{AUC}\big(y_{pq},,\hat{y}_{pq}\big), \qquad \lVert x_p-x_q\rVert \le R. \]We report two comparisons:
\[ \text{probe\_margin}(R) = \text{probe\_auc}(R)-\text{baseline}(R), \] \[ \text{probe\_gain}(R) = \text{probe\_auc}(R)-\text{best\_auc}(R). \]The margin measures learned cell-awareness beyond pure distance. The gain measures whether combining heads beats the best single raw head. This is a diagnostic readout: it asks whether cell structure is distributed across the attention stack, not whether any one head directly segments cells.
Spatial baseline and margin
A pure-locality predictor scores each pair by negative distance, and the margin is the lift over it:
\[ \text{baseline}(R) = \text{AUC}\big(y,\, -d_{pq}\big), \qquad \text{margin}(R) = \text{best\_auc}(R) - \text{baseline}(R) \]Predicted-cell Dice and IoU
For a query token $q$, restrict to a local window
\[ W(q)={p:\lVert x_p-x_q\rVert \le r}, \qquad r=80\text{ px}. \]Each token is scored by attention from a fixed source, either the best AUC head $(L^*,h^*)$ or the mean over heads:
\[ s_p = a^{(L^*,h^*)}_{qp}, \qquad p \in W(q). \]Let $G$ be the ground-truth cell containing $q$, and set $k=|G\cap W(q)|$. Starting from $q$, grow a connected mask $P_{\text{attn}}$ by repeatedly adding the boundary token with largest $s_p$ until $|P_{\text{attn}}|=k$. This gives an area-matched diagnostic: the mask size is matched to the true cell, but the shape is determined by attention.
As a locality baseline, repeat the same growth using only negative distance,
\[ s^{\text{dist}}*p = -d*{pq}, \]giving $P_{\text{dist}}$. The reported overlap scores are
\[ \text{IoU}*{\text{attn}} = \frac{|P*{\text{attn}}\cap G|}{|P_{\text{attn}}\cup G|}, \qquad \text{Dice}*{\text{attn}} = \frac{2|P*{\text{attn}}\cap G|}{|P_{\text{attn}}|+|G|}, \]and the margin over pure locality is
\[ \text{Dice}_{\text{margin}} = \text{Dice}_{\text{attn}} - \text{Dice}_{\text{dist}}. \]Here, \(\text{Dice}_{\text{attn}}\) measures how well the attention-grown region matches the ground-truth cell region, while \(\text{Dice}_{\text{dist}}\) is the pure-locality baseline obtained by growing from distance alone. Thus, \(\text{Dice}_{\text{margin}}\) measures the gain from attention beyond what spatial locality already explains.
Dice is the pixel-level F1 score, while IoU is the stricter Jaccard overlap. They satisfy
\[ \text{Dice} = \frac{2\,\text{IoU}}{1+\text{IoU}}. \]Gene expression
Spatial readout on the cell embeddings. Covered cells (polygons touching tissue) split into train and test via a $5\times5$ block lattice with $4$ inner-ring blocks held out (about $16\%$). The top $K=15$ genes are chosen once by train-set expression variance, gated by a non-zero fraction floor, and held fixed across epochs and models so every checkpoint scores against the same gene set.
Target Normalization (CP10K + log1p)
Let $X \in \mathbb{N}^{N \times G_{\text{full}}}$ be the raw transcript counts for $N$ cells over the full panel of genes. To prevent highly-expressed genes and sequencing depth variations from dominating the metrics, the raw counts $x_{nc}$ for cell $n$ and gene $c$ are normalized by the cell's total library size $L_n = \sum_g x_{ng}$ prior to subsetting. The counts are then scaled to $10,000$ (CP10K) and log-transformed. The final target expression $y_{nc}$ is:
\[ y_{nc} = \ln\left( \frac{x_{nc}}{\max(L_n, 1.0)} \times 10^4 + 1 \right) \]This de-skewed matrix $y$ is then subset to the $K=15$ evaluation genes for all downstream scoring.
Pearson core
Per-gene Pearson between prediction $\hat{y}$ and target $y$ over the $N$ test cells, then averaged over genes:
\[ r_g = \frac{\sum_n (y_{ng} - \bar{y}_g)(\hat{y}_{ng} - \bar{\hat{y}}_g)} {\sqrt{\sum_n (y_{ng} - \bar{y}_g)^2 \;\sum_n (\hat{y}_{ng} - \bar{\hat{y}}_g)^2}}, \qquad \bar{r} = \frac{1}{K}\sum_{g=1}^{K} r_g \]Spatial-KNN baseline
What location alone predicts. Inverse-distance weighting over the $k=8$ nearest training cells:
\[ \hat{y}^{\text{knn}}_i = \frac{\sum_{j \in N_k(i)} w_{ij}\, y_j}{\sum_{j \in N_k(i)} w_{ij}}, \qquad w_{ij} = \frac{1}{d_{ij}} \]Ridge readout, raw and residual
Two ridge heads on the embeddings $\Phi$, one on raw expression and one on the KNN residual $y^{\text{res}} = y - \hat{y}^{\text{knn}}$:
\[ W^\star = \arg\min_W \; \lVert y_{\text{tr}} - \Phi_{\text{tr}} W \rVert_2^2 + \alpha \lVert W \rVert_2^2, \qquad \alpha = 10 \]The residual head is forced to predict only the morphology signal that spatial smoothing cannot already explain. The headline metrics are
\[ \text{residual\_pearson} = \bar{r}\big(\hat{y}^{\text{res}}, y^{\text{res}}\big), \qquad \text{raw\_pearson} = \bar{r}\big(\hat{y}^{\text{raw}}, y\big) \] \[ \text{knn\_pearson} = \bar{r}\big(\hat{y}^{\text{knn}}, y\big), \qquad \Delta_{\text{knn}} = \text{raw\_pearson} - \text{knn\_pearson} \]with $\Delta_{\text{knn}}$ the gain over pure spatial interpolation, and a count of how many of the $K$ genes the model beats KNN on:
\[ \text{genes\_beating\_knn} = \sum_{g=1}^{K} \mathbb{1}\big[\, r_g^{\text{raw}} > r_g^{\text{knn}} \,\big] \]Calibration
Pearson is invariant to scale and shift, so these check the prediction is actually well-scaled, not just correlated. With $R^2 = 1 - \sum(y-\hat{y})^2 / \sum(y-\bar{y})^2$ averaged over genes:
\[ \text{raw\_R2} = \overline{R^2}, \qquad \text{calibrated\_R2} = \overline{r_g^2} \]calibrated\_R2 is the $R^2$ achievable after an optimal affine rescale, so the gap to raw\_R2 is pure miscalibration. Alongside it,
\[ \frac{\sigma_{\hat{y}}}{\sigma_y}, \qquad \text{calib\_slope} = \text{OLS slope of } y \text{ on } \hat{y}, \qquad \text{mean\_bias} = \mathbb{E}[\hat{y}] - \mathbb{E}[y] \]a variance ratio below $1$ or a slope far above $1$ means the model is compressing predictions toward the mean. We also track $\Delta\text{MAE}_{\text{knn}}$, the mean absolute error improvement over the KNN baseline.
Bootstrap intervals
Cell-resampling CIs on the Pearson numbers, run every $20$ epochs since each is $B=1000$ refits. Draw $N$ test cells with replacement, recompute the mean Pearson, and take percentiles:
\[ \bar{r}^{(b)} = \bar{r}\big(\hat{y}_{I_b}, y_{I_b}\big), \quad I_b \sim \text{Unif}(\{1,\dots,N\})^N, \qquad \text{CI}_{95} = \big[\, P_{2.5}(\bar{r}^{(b)}),\; P_{97.5}(\bar{r}^{(b)}) \,\big] \]The delta CI resamples the same cells for both terms so their correlation is preserved:
\[ \delta^{(b)} = \bar{r}\big(\hat{y}^{\text{raw}}_{I_b}, y_{I_b}\big) - \bar{r}\big(\hat{y}^{\text{knn}}_{I_b}, y_{I_b}\big) \]Block std and lineage panels
Spread of residual Pearson across the $4$ held-out blocks,
\[ \text{block\_sd} = \operatorname{std}_{b=1}^{4} \; \text{residual\_pearson}^{(b)} \]flags whether a score rides on one lucky block. Finally, residual Pearson is reported on curated lineage panels (immune, epithelial, vascular, proliferation) for interpretability, each the same $\bar{r}$ restricted to that gene subset.
[TODO] k nearest at least x distance away
use otsu only before
use attention scores
concat pixel i and j logits (stack $\alpha_{i,j}$ across all per head per layer) and fit AUC
- use logits but we learn to transform it (instead of just using logits for auc)
Stability: Folded Inline and Offline Evaluation
The current RNA/GEX eval stack separates graded offline boards from inline training monitors. This prevents live curves, old-convention curves, and leaderboard numbers from being silently mixed. All GEX metrics use CP10K+\(\log1p\) targets, ridge-GCV probes, and an IDW spatial-KNN baseline with \(k=8\). The baseline-subtracted score is
\[ \Delta_{\mathrm{knn}} = r_{\mathrm{ridge}} - r_{\mathrm{IDW\text{-}KNN}}. \]Deltas are comparable only within the same rung, never across rungs, because the spatial unit and KNN floor change.
Offline: Graded Board Rungs
Offline evals are the authoritative leaderboard numbers. Embeddings are extracted and cached once per model; the board scorer then runs model-blind ridge or retrieval probes with no model forward pass. Confidence intervals are bootstrapped over the 12 slides, not over cells.
tile\_gex: 112 \(\mu\mathrm{m}\) tile-level GEX. Ridge-GCV predicts CP10K+\(\log1p\) pseudobulk from tile embeddings. Random 90/10 splits, \(p=1.0\), \(K=10\). Top-20 genes, nonzero floor \(0.05\). This is the main tile board against UNI2, Virchow2, and Prov-GigaPath.tile\_niche: 112 \(\mu\mathrm{m}\) RNA-niche retrieval. Niches are KMeans labels with \(k=10\) from top-50 variable genes. Embeddings retrieve same-niche tiles by cosine similarity beyond a 2240 px exclusion radius. The metric is \[ \texttt{niche\_gap} = \mathrm{mAP}_{\mathrm{HE}} - \mathrm{mAP}_{\mathrm{spatial,shared}}. \]This tests RNA-defined niche structure, not per-gene regression.
window: 7 \(\mu\mathrm{m}\) near-nucleus GEX. Same ridge/delta machinery astile\_gex, but evaluated on fine windows with 25-fold \(5\times5\) spatial block-CV to reduce spatial leakage. The shared KNN bar is still a random 90/10, well-fed, model-independent baseline, so this is a strict fixed-bar delta rather than a split-matched residual. Fine-rung nonzero floor is \(0.02\).cell: single-cell GEX. Same block-CV ridge setup aswindow, but the unit is a native per-cell embedding. This rung is Scout-only because tile foundation encoders do not expose native per-cell vectors.
The tile boards stay all-pixel for frontier comparability: Scout trains tissue-only, but eval feeds the full crop with \(valid=all\mbox{-}ones\), so glass and background tokens are attended and mean-pooled just as they are for frontier tile encoders. The tissue-valid path is kept as a Scout-only diagnostic, not the standing board.
Inline: Per-Epoch Training Monitors
Inline evals are cheap training-time monitors. They encode the current checkpoint during training, shard tiles across DDP ranks, all-reduce the tile embeddings, and log to wandb/metrics.csv. Inline uses the same all-pixel eval convention as the tile board, but a cheaper budget:
inline\_delta\_vs\_knn: live GEX trajectory. It uses the same target normalization, all-pixel pooling, ridge-GCV family, and shared crc32-seeded KNN baseline family astile\_gex. It is on the same biological ruler, but it is not the graded board number because \(p\), \(K\), and tile sampling differ. Read it as the live training slope; use offlinetile\_gexas the graded truth.inline\_niche\_gap: live niche trend. It is not board-comparable because inline subtracts a per-fold spatial baseline, \[ \mathrm{mAP}_{\mathrm{HE}} - \mathrm{mAP}_{\mathrm{spatial,fold}}, \]while offline
tile\_nichesubtracts a shared spatial baseline. Use inline niche for direction only; use offlinetile\_nichefor the real number.
Rule of thumb: tile\_gex, tile\_niche, window, and cell are graded board numbers; inline\_delta\_vs\_knn is the live GEX slope; inline\_niche\_gap is trend-only.