Perceptual sound matching (PSM) aims to find the input parameters to a synthesizer so as to best imitate an audio target. Deep learning for PSM optimizes a neural network to analyze and reconstruct prerecorded samples. In this context, our article addresses the problem of designing a suitable loss function when the training set is generated by a differentiable synthesizer. Our main contribution is perceptual-neural-physical loss (PNP), which aims at addressing a tradeoff between perceptual relevance and computational efficiency. The key idea behind PNP is to linearize the effect of synthesis parameters upon auditory features in the vicinity of each training sample. The linearization procedure is massively paralellizable, can be precomputed, and offers a 100-fold speedup during gradient descent compared to differentiable digital signal processing (DDSP). We demonstrate PNP on two datasets of nonstationary sounds: an AM/FM arpeggiator and a physical model of rectangular membranes. We show that PNP is able to accelerate DDSP with joint time-frequency scattering transform (JTFS) as auditory feature, while preserving its perceptual fidelity. Additionally, we evaluate the impact of other design choices in PSM: parameter rescaling, pretraining, auditory representation, and gradient clipping. We report state-of-the-art results on both datasets and find that PNP-accelerated JTFS has greater influence on PSM performance than any other design choice.

翻译：感知声音匹配（PSM）旨在寻找合成器的输入参数以最佳地模仿音频目标。基于深度学习的PSM通过优化神经网络来分析和重建预录样本。在此背景下，本文探讨了当训练集由可微合成器生成时如何设计合适的损失函数。我们的主要贡献是感知-神经-物理损失（PNP），旨在解决感知相关性与计算效率之间的权衡。PNP的核心思想是在每个训练样本附近线性化合成参数对听觉特征的影响。该线性化过程具有高度可并行性，可预计算，且相比可微数字信号处理（DDSP）可实现梯度下降过程中100倍的加速。我们在两个非平稳声音数据集上验证了PNP：AM/FM琶音器和矩形膜物理模型。实验表明，PNP能够以联合时频散射变换（JTFS）作为听觉特征加速DDSP，同时保持其感知保真度。此外，我们评估了PSM中其他设计选择的影响：参数缩放、预训练、听觉表示和梯度裁剪。我们在两个数据集上均取得了最优结果，并发现PNP加速的JTFS对PSM性能的影响显著大于其他设计选择。