Diagnostic classifiers and imaging biomarkers are fitted on the embeddings of medical foundation models. These models are replaced as new versions appear. This practice assumes that different models represent images alike. We tested this assumption with more than 750,000 images from 14 datasets in five imaging modalities, 18 public models, and 101 models trained on chest radiographs and histopathology that differ in pretraining objective, label type, size, random seed, initialization, or training patients. Agreement was measured as the mutual k-nearest-neighbor overlap, the share of an image's nearest neighbors common to two models. Two public models shared on average 0.091 of the 10 nearest neighbors of a chest radiograph. A randomly initialized network shared 0.036 with them. In the four other modalities, they shared at most 0.300. Convergence depended on the pretraining objective. Models trained with label supervision converged least in all five controlled settings. On chest radiographs, two label-supervised models that differed only in their random seed shared 0.062 to 0.100 of their nearest neighbors. Two models trained with self-distillation, masked image modeling, or contrastive pretraining shared 0.411 to 0.802. On chest radiographs, agreement depended more on the source dataset and the radiographic view than on the findings. It was not significantly associated with accuracy. A linear mapping between the embeddings of two models, fitted on 4,096 unlabeled radiographs, transferred classifiers for 14 chest findings with 0.987 of their original area under the receiver operating characteristic curve. Convergence can therefore be chosen when a model is trained. A chest radiograph classifier can be transferred to a new model without new labels.

翻译：暂无翻译