Nowadays neural-network-based image- and video-quality metrics show better performance compared to traditional methods. However, they also became more vulnerable to adversarial attacks that increase metrics' scores without improving visual quality. The existing benchmarks of quality metrics compare their performance in terms of correlation with subjective quality and calculation time. However, the adversarial robustness of image-quality metrics is also an area worth researching. In this paper, we analyse modern metrics' robustness to different adversarial attacks. We adopted adversarial attacks from computer vision tasks and compared attacks' efficiency against 15 no-reference image/video-quality metrics. Some metrics showed high resistance to adversarial attacks which makes their usage in benchmarks safer than vulnerable metrics. The benchmark accepts new metrics submissions for researchers who want to make their metrics more robust to attacks or to find such metrics for their needs. Try our benchmark using pip install robustness-benchmark.

翻译：当前，基于神经网络的图像与视频质量指标展现出优于传统方法的性能。然而，这些指标也更容易受到对抗攻击的影响——此类攻击能在不提升视觉质量的情况下人为提高指标评分。现有的质量指标基准主要从与主观质量的相关性及计算耗时两方面比较性能，但质量指标对对抗攻击的鲁棒性同样值得深入研究。本文分析了现代指标对不同对抗攻击的鲁棒性。我们借鉴计算机视觉任务中的对抗攻击方法，比较了15种无参考图像/视频质量指标对这些攻击的抵御效果。实验表明，部分指标对对抗攻击具有高度抵抗性，这使得它们在基准测试中的使用比脆弱指标更安全。本基准测试接受研究者提交新指标，旨在帮助研究人员提升指标鲁棒性，或根据自身需求筛选鲁棒指标。欢迎通过 pip install robustness-benchmark 体验我们的基准测试。