While deep learning methods have achieved impressive success in many vision benchmarks, it remains difficult to understand and explain the representations and decisions of these models. Though vision models are typically trained on 2D inputs, they are often assumed to develop an implicit representation of the underlying 3D scene (for example, showing tolerance to partial occlusion, or the ability to reason about relative depth). Here, we introduce MRD (metamers rendered differentiably), an approach that uses physically based differentiable rendering to probe vision models' implicit understanding of generative 3D scene properties, by finding 3D scene parameters that are physically different but produce the same model activation (i.e. are model metamers). Unlike previous pixel-based methods for evaluating model representations, these reconstruction results are always grounded in physical scene descriptions. This means we can, for example, probe a model's sensitivity to object shape while holding material and lighting constant. As a proof-of-principle, we assess multiple models in their ability to recover scene parameters of geometry (shape) and bidirectional reflectance distribution function (material). The results show high similarity in model activation between target and optimized scenes, with varying visual results. Qualitatively, these reconstructions help investigate the physical scene attributes to which models are sensitive or invariant. MRD holds promise for advancing our understanding of both computer and human vision by enabling analysis of how physical scene parameters drive changes in model responses.

翻译：尽管深度学习方法已在众多视觉基准测试中取得显著成功，但理解和解释这些模型的表征与决策机制仍具挑战。视觉模型通常在二维输入上进行训练，但人们常假设其能形成对底层三维场景的隐式表征（例如表现出对部分遮挡的容忍度，或具备相对深度推理能力）。本文提出MRD（可微分渲染的元模拟刺激），该方法利用基于物理的可微分渲染技术，通过寻找物理属性不同但能产生相同模型激活（即成为模型元模拟刺激）的三维场景参数，来探究视觉模型对生成式三维场景属性的隐式理解。与以往基于像素的模型表征评估方法不同，这种重建结果始终以物理场景描述为基础。这意味着我们能够（例如）在保持材质与光照恒定的条件下，探究模型对物体形状的敏感性。作为原理验证，我们评估了多个模型在恢复几何（形状）与双向反射分布函数（材质）等场景参数方面的能力。结果显示目标场景与优化场景间的模型激活具有高度相似性，但视觉重建结果存在差异。定性分析表明，这些重建有助于探究模型对哪些物理场景属性敏感或不敏感。MRD通过分析物理场景参数如何驱动模型响应变化，为深化计算机视觉与人类视觉的理解提供了新的研究路径。