que paso
A new arXiv preprint audits whether frozen foundation-model embeddings remain reliable when white-blood-cell images differ across scanners, sites, stains and preparation pipelines. The authors evaluated 15 encoders across four public single-cell acquisition domains, measuring both classification accuracy and calibration. They report that models with near-saturated in-domain results can perform substantially worse on shifted data.
The authors evaluate 15 frozen encoders drawn from hematology, pathology and general-vision model groups. Rather than retraining the encoders, they attach downstream probes to their embeddings and test performance across four public single-cell acquisition domains involving white-blood-cell classification. The paper frames the central problem as acquisition shift: the image data can change because of scanners, sites, stains or preparation pipelines, even when the underlying task remains the same. The work was submitted to arXiv on Aug. 25, 2026, and the abstract says it was accepted for an oral presentation at the HemaRAI 2026 workshop, a MICCAI 2026 satellite event.
The reported in-domain results are tightly clustered and very high. Linear-probe macro-F1 ranges from 0.98 to 0.997, which the authors describe as saturated performance on the source setting. That ordering does not hold when the models are tested on data from another acquisition domain. Cross-dataset macro-F1 falls by 34% to 72%, according to the abstract. DinoBloom-L, the best model in-domain, drops to 10th among the 15 encoders on the most shifted target, identified as MLL23, at the benchmark's shared 224-pixel input size. RedDino and several general-vision and pathology encoders rank ahead of it in that setting.
The paper also reports that the apparent robustness depends on how the frozen representation is used. One-nearest-neighbor retrieval is more stable on average than a source-fitted linear head: the median rank correlation between source and target performance is 0.65 for 1-NN retrieval versus 0.45 for the linear probe. Neither method, however, universally predicts which encoder will be strongest on the target domain. This means a favorable result from one downstream evaluation method should not automatically be treated as evidence that the underlying representation will transfer reliably.
Confidence estimates deteriorate even more sharply. Source-trained probes are nearly calibrated in-domain, with an expected calibration error of 0.004, but the reported off-domain ECE rises to 0.35, indicating much poorer alignment between confidence and correctness under the tested shift. Source-fitted temperature scaling transfers poorly. The authors further identify MLL23 as DinoBloom's internal cohort. Because the only held-out dataset for DinoBloom is also the benchmark's source domain, the audit cannot isolate possible pretraining exposure from scanner-associated shift. That limitation is central to interpreting the rankings and is not resolved by the reported accuracy results.
Lea la fuente principal: arxiv.org ↗
Por qué es importante
The findings challenge the use of in-domain accuracy as a sufficient test for medical AI models. A system can appear highly accurate and well calibrated on its source data while becoming confidently wrong on images collected under different conditions. That creates a practical evaluation problem for laboratories considering reusable foundation-model representations.
The practical warning is about the gap between recognizing a pattern and knowing when that recognition can be trusted. In this audit, a model can achieve a very high score on data resembling its source setting while losing a large portion of its cross-dataset macro-F1. If a laboratory evaluates only the familiar domain, it may miss the conditions under which the representation becomes less useful. The paper therefore treats robustness as a deployment requirement rather than a secondary research metric.
Calibration matters because a wrong prediction delivered with high confidence can affect how people use an AI output. The abstract does not report a clinical deployment or patient-outcome study, so it does not establish how the measured ECE changes would affect diagnoses, triage or treatment decisions in practice. It does show that confidence behavior can change materially under the tested acquisition shifts. That makes confidence evaluation relevant wherever model outputs are used to prioritize review or influence a human decision.
The study also complicates the idea that a foundation model's reputation or in-domain ranking can stand in for site-specific validation. DinoBloom-L's reversal from first in-domain to 10th on the most shifted target is a result from this benchmark, not a universal ranking of hematology models. The authors' exposure analysis further shows why model provenance and data overlap matter: an apparent advantage may reflect familiarity with a cohort or acquisition process rather than general robustness. Evaluations that omit exposure checks can therefore overstate what a model has learned.
The paper reports that label-free adaptation and marginal-entropy-based model selection look safe under balanced evaluation but fail when the WBC class prior shifts in a more realistic scenario. That result connects technical evaluation choices to operational conditions: the distribution of classes in incoming samples may not match the balanced assumptions used in a benchmark. The authors propose Class-Balanced Re-standardization, a training-free pseudo-label-balanced feature-normalization method, and report that it improves all evaluated target-prior scenario means while partially improving calibration. The abstract also says exceptions and residual miscalibration remain, so the method is a research result rather than a demonstrated solution for clinical use.
Qué ver a continuación
Future evaluations will need to separate scanner-associated distribution shift from possible pretraining exposure, test both accuracy and confidence, and examine changes in WBC class proportions. The paper reports promising but incomplete results for a training-free Class-Balanced Re-standardization method; its performance across encoders, datasets and real clinical workflows remains unresolved.
The next important question is whether the reported shifts correspond to the range of variation encountered in real hematology laboratories. This paper uses four public single-cell acquisition domains, but the abstract does not identify their institutions, scanner models, staining protocols, sample volumes or clinical populations. It also does not report prospective clinical validation, patient outcomes, or a comparison with routinely used diagnostic workflows. Those omissions limit how directly the benchmark results can be translated into deployment decisions.
A second issue is disentangling acquisition shift from pretraining exposure. The authors say MLL23 was DinoBloom's internal cohort and that the benchmark cannot isolate exposure from scanner-associated shift because the only held-out dataset is also the source domain. Further work would need cleaner separation between training or pretraining data and evaluation data, along with documentation of collection sites, devices, stains and preparation pipelines. Without that separation, it remains difficult to know whether a model is robust to acquisition changes or has benefited from prior familiarity with a particular cohort.
The reported failure under WBC class-prior shift deserves closer testing. Balanced evaluation can make adaptation or model-selection methods appear safer than they are when the mix of cell types changes. Future studies should examine whether the same pattern persists across different class distributions, target populations and acquisition settings, and whether confidence remains usable while class frequencies change. The abstract establishes the authors' tested scenario, but it does not establish the size or frequency of such shifts in clinical practice.
Class-Balanced Re-standardization is the paper's most actionable proposed intervention, but its limits are explicit. The abstract reports improvements in all evaluated target-prior scenario means and partial calibration gains, while also noting encoder-level exceptions and remaining miscalibration. What is not yet clear is how the method behaves outside the evaluated encoders and public domains, whether pseudo-label errors can compound during normalization, and how it would fit into a regulated workflow. The broader takeaway to monitor is whether future hematology AI benchmarks jointly report accuracy, calibration, exposure and class-prior robustness instead of treating any one metric as sufficient.
