Motivation: We explored how explainable artificial intelligence (XAI) can help to shed light into the inner workings of neural networks for protein function prediction, by extending the widely used XAI method of integrated gradients such that latent representations inside of transformer models, which were finetuned to Gene Ontology term and Enzyme Commission number prediction, can be inspected too. Results: The approach enabled us to identify amino acids in the sequences that the transformers pay particular attention to, and to show that these relevant sequence parts reflect expectations from biology and chemistry, both in the embedding layer and inside of the model, where we identified transformer heads with a statistically significant correspondence of attribution maps with ground truth sequence annotations (e.g. transmembrane regions, active sites) across many proteins. Availability and Implementation: Source code can be accessed at https://github.com/markuswenzel/xai-proteins .

翻译：动机：我们探索了可解释人工智能如何帮助揭示神经网络在蛋白质功能预测中的内在机制，通过扩展广泛使用的积分梯度可解释人工智能方法，使得经过微调以预测基因本体术语和酶委员会编号的Transformer模型内部的潜在表示也能被检视。结果：该方法使我们能够识别出Transformer特别关注的序列中的氨基酸，并证明这些相关序列部分在嵌入层和模型内部均反映了生物学和化学的预期特征——我们识别出多个Transformer注意力头，其归因图与许多蛋白质的真实序列注释（例如跨膜区、活性位点）具有统计显著性对应关系。可用性与实现：源代码可在 https://github.com/markuswenzel/xai-proteins 获取。