Deep learning models for breast cancer detection from mammographic images have significant reliability problems when presented with Out-of-Domain (OOD) inputs such as other imaging modalities (CT, MRI, X-ray) or equipment variations, leading to unreliable detection and misdiagnosis. The current research mitigates the fundamental OOD issue through a comprehensive approach integrating ResNet50-based OOD filtering with YOLO architectures (YOLOv8, YOLOv11, YOLOv12) for accurate detection of breast cancer. Our strategy establishes an in-domain gallery via cosine similarity to rigidly reject non-mammographic inputs prior to processing, ensuring that only domain-associated images supply the detection pipeline. The OOD detection component achieves 99.77\% general accuracy with immaculate 100\% accuracy on OOD test sets, effectively eliminating irrelevant imaging modalities. ResNet50 was selected as the optimum backbone after 12 CNN architecture searches. The joint framework unites OOD robustness with high detection performance ([email protected]: 0.947) and enhanced interpretability through Grad-CAM visualizations. Experimental validation establishes that OOD filtering significantly improves system reliability by preventing false alarms on out-of-distribution inputs while maintaining higher detection accuracy on mammographic data. The present study offers a fundamental foundation for the deployment of reliable AI-based breast cancer detection systems in diverse clinical environments with inherent data heterogeneity.

翻译：基于乳腺钼靶图像的深度学习乳腺癌检测模型在面对域外输入（如CT、MRI、X射线等其他影像模态或设备差异）时存在显著可靠性问题，导致检测不可靠及误诊。本研究通过综合方法缓解这一根本性域外问题，整合基于ResNet50的域外过滤与YOLO架构（YOLOv8、YOLOv11、YOLOv12）以实现乳腺癌的精准检测。该策略通过余弦相似度建立域内图库，在处理前严格排除非乳腺钼靶输入，确保仅与域相关的图像进入检测流程。域外检测组件总体准确率达99.77%，在域外测试集上实现100%完美准确率，有效消除无关影像模态。经过12种CNN架构搜索，ResNet50被选为最优骨干网络。该联合框架将域外鲁棒性与高检测性能（[email protected]: 0.947）相结合，并通过Grad-CAM可视化增强可解释性。实验验证表明，域外过滤通过防止域外输入引发的误报，显著提升系统可靠性，同时维持乳腺钼靶数据的高检测精度。本研究为在数据固有异质性的多样化临床环境中部署可靠的基于AI的乳腺癌检测系统奠定了坚实基础。