Recently, encoders like ViT (vision transformer) and ResNet have been trained on vast datasets and utilized as perceptual metrics for comparing sketches and images, as well as multi-domain encoders in a zero-shot setting. However, there has been limited effort to quantify the granularity of these encoders. Our work addresses this gap by focusing on multi-modal 2D projections of individual 3D instances. This task holds crucial implications for retrieval and sketch-based modeling. We show that in a zero-shot setting, the more abstract the sketch, the higher the likelihood of incorrect image matches. Even within the same sketch domain, sketches of the same object drawn in different styles, for example by distinct individuals, might not be accurately matched. One of the key findings of our research is that meticulous fine-tuning on one class of 3D shapes can lead to improved performance on other shape classes, reaching or surpassing the accuracy of supervised methods. We compare and discuss several fine-tuning strategies. Additionally, we delve deeply into how the scale of an object in a sketch influences the similarity of features at different network layers, helping us identify which network layers provide the most accurate matching. Significantly, we discover that ViT and ResNet perform best when dealing with similar object scales. We believe that our work will have a significant impact on research in the sketch domain, providing insights and guidance on how to adopt large pretrained models as perceptual losses.

翻译：近期，诸如ViT（视觉Transformer）和ResNet等编码器在大规模数据集上完成训练，并被用作草图与图像比较的感知度量，以及零样本场景下的多域编码器。然而，量化这些编码器粒度的研究仍显不足。本研究聚焦于单个三维实例的多模态二维投影，填补了这一空白。该任务对检索与基于草图的建模具有关键意义。研究表明，在零样本设置下，草图越抽象，图像匹配错误的概率越高。即便在同一草图域内，由不同个体以不同风格绘制的同一物体的草图，也可能无法准确匹配。本研究的核心发现之一是：对某一类三维形状进行细致微调，能够提升对其他形状类别的性能，甚至达到或超越监督方法的准确率。我们对比并讨论了多种微调策略。此外，我们深入探讨了草图中物体尺度在不同网络层特征相似性中的作用，从而定位实现最精确匹配的网络层。重要的是，我们发现ViT和ResNet在处理相似物体尺度时性能最优。我们相信，本研究将对草图领域的研究产生重要影响，为如何将大型预训练模型作为感知损失函数提供见解与指导。