The promise of multimodal models for real-world applications has inspired research in visualizing and understanding their internal mechanics with the end goal of empowering stakeholders to visualize model behavior, perform model debugging, and promote trust in machine learning models. However, modern multimodal models are typically black-box neural networks, which makes it challenging to understand their internal mechanics. How can we visualize the internal modeling of multimodal interactions in these models? Our paper aims to fill this gap by proposing MultiViz, a method for analyzing the behavior of multimodal models by scaffolding the problem of interpretability into 4 stages: (1) unimodal importance: how each modality contributes towards downstream modeling and prediction, (2) cross-modal interactions: how different modalities relate with each other, (3) multimodal representations: how unimodal and cross-modal interactions are represented in decision-level features, and (4) multimodal prediction: how decision-level features are composed to make a prediction. MultiViz is designed to operate on diverse modalities, models, tasks, and research areas. Through experiments on 8 trained models across 6 real-world tasks, we show that the complementary stages in MultiViz together enable users to (1) simulate model predictions, (2) assign interpretable concepts to features, (3) perform error analysis on model misclassifications, and (4) use insights from error analysis to debug models. MultiViz is publicly available, will be regularly updated with new interpretation tools and metrics, and welcomes inputs from the community.

翻译：多模态模型在现实应用中的前景激发了对其内部机制进行可视化与理解的相关研究，最终目标是帮助利益相关者可视化模型行为、执行模型调试，并增强对机器学习模型的信任。然而，现代多模态模型通常是黑箱神经网络，这使其内部机制难以理解。我们如何可视化这类模型中多模态交互的内部建模过程？本文旨在填补这一空白，提出MultiViz方法，通过将可解释性问题划分为四个阶段来分析多模态模型的行为：(1) 单模态重要性：每种模态如何贡献于下游建模和预测；(2) 跨模态交互：不同模态之间如何相互关联；(3) 多模态表征：单模态和跨模态交互如何在决策级特征中表示；(4) 多模态预测：决策级特征如何组合以生成预测。MultiViz被设计为适用于多种模态、模型、任务和研究领域。通过在6个真实世界任务的8个训练模型上进行实验，我们证明了MultiViz中的互补阶段共同使用户能够：(1) 模拟模型预测；(2) 为特征分配可解释概念；(3) 对模型错误分类进行错误分析；(4) 利用错误分析的见解调试模型。MultiViz已公开提供，并将定期更新新的解释工具和指标，同时欢迎来自社区的输入。