Intelligence is key to advancing integrated circuit (IC) fabrication. Recent breakthroughs in Large Multimodal Models (LMMs) have unlocked unparalleled abilities in understanding images and text, fostering intelligent fabrication. Leveraging the power of LMMs, we introduce FabGPT, a customized IC fabrication large multimodal model for wafer defect knowledge query. FabGPT manifests expertise in conducting defect detection in Scanning Electron Microscope (SEM) images, performing root cause analysis, and providing expert question-answering (Q&A) on fabrication processes. FabGPT matches enhanced multimodal features to automatically detect minute defects under complex wafer backgrounds and reduce the subjectivity of manual threshold settings. Besides, the proposed modulation module and interactive corpus training strategy embed wafer defect knowledge into the pre-trained model, effectively balancing Q&A queries related to defect knowledge and original knowledge and mitigating the modality bias issues. Experiments on in-house fab data (SEM-WaD) show that our FabGPT achieves significant performance improvement in wafer defect detection and knowledge querying.

翻译：智能是推动集成电路（IC）制造发展的关键。大型多模态模型（LMMs）近期的突破性进展，在理解图像与文本方面释放了前所未有的能力，促进了智能制造的实现。借助LMMs的强大能力，我们提出了FabGPT，一个为晶圆缺陷知识查询定制的IC制造大型多模态模型。FabGPT展现出在执行扫描电子显微镜（SEM）图像缺陷检测、进行根本原因分析以及提供制造工艺专家问答（Q&A）方面的专业能力。FabGPT通过匹配增强的多模态特征，能够在复杂的晶圆背景下自动检测微小缺陷，并减少手动阈值设置的主观性。此外，所提出的调制模块和交互式语料训练策略将晶圆缺陷知识嵌入到预训练模型中，有效平衡了与缺陷知识相关的问答查询和原始知识，缓解了模态偏差问题。在内部工厂数据（SEM-WaD）上的实验表明，我们的FabGPT在晶圆缺陷检测和知识查询方面取得了显著的性能提升。