We present a novel usage of Transformers to make image classification interpretable. Unlike mainstream classifiers that wait until the last fully connected layer to incorporate class information to make predictions, we investigate a proactive approach, asking each class to search for itself in an image. We realize this idea via a Transformer encoder-decoder inspired by DEtection TRansformer (DETR). We learn "class-specific" queries (one for each class) as input to the decoder, enabling each class to localize its patterns in an image via cross-attention. We name our approach INterpretable TRansformer (INTR), which is fairly easy to implement and exhibits several compelling properties. We show that INTR intrinsically encourages each class to attend distinctively; the cross-attention weights thus provide a faithful interpretation of the prediction. Interestingly, via "multi-head" cross-attention, INTR could identify different "attributes" of a class, making it particularly suitable for fine-grained classification and analysis, which we demonstrate on eight datasets. Our code and pre-trained models are publicly accessible at the Imageomics Institute GitHub site: https://github.com/Imageomics/INTR.

翻译：我们提出了一种Transformer的新用法，用于实现可解释的图像分类。与主流分类器在最后一个全连接层才融合类别信息进行预测不同，我们探索了一种主动方法，让每个类别在图像中自行搜索其特征。我们通过受检测Transformer（DETR）启发的Transformer编码器-解码器架构实现了这一思想。我们学习"类别特定"查询（每个类别一个）作为解码器输入，使每个类别能通过交叉注意力在图像中定位其模式。我们将该方法命名为可解释Transformer（INTR），该方法易于实现且展现出若干优异特性。研究表明，INTR本质上促使每个类别进行差异性聚焦；交叉注意力权重因此提供了预测的忠实解释。值得注意的是，通过"多头"交叉注意力，INTR可识别类别的不同"属性"，特别适用于细粒度分类与分析，我们在八个数据集上验证了这一点。我们的代码和预训练模型已在Imageomics研究所GitHub网站公开：https://github.com/Imageomics/INTR。