Nowadays neural-network-based image- and video-quality metrics show better performance compared to traditional methods. However, they also became more vulnerable to adversarial attacks that increase metrics' scores without improving visual quality. The existing benchmarks of quality metrics compare their performance in terms of correlation with subjective quality and calculation time. However, the adversarial robustness of image-quality metrics is also an area worth researching. In this paper, we analyse modern metrics' robustness to different adversarial attacks. We adopted adversarial attacks from computer vision tasks and compared attacks' efficiency against 15 no-reference image/video-quality metrics. Some metrics showed high resistance to adversarial attacks which makes their usage in benchmarks safer than vulnerable metrics. The benchmark accepts new metrics submissions for researchers who want to make their metrics more robust to attacks or to find such metrics for their needs. Try our benchmark using pip install robustness-benchmark.

翻译：如今，基于神经网络的图像与视频质量指标相比传统方法展现出更优的性能，但同时也更容易受到对抗攻击的影响——此类攻击可在不提升视觉质量的情况下人为提高指标得分。现有质量指标基准测试主要比较其与主观质量的相关性及计算耗时，然而图像质量指标的对抗鲁棒性同样是值得研究的领域。本文分析了现代指标对不同对抗攻击的鲁棒性。我们采用计算机视觉任务中的对抗攻击方法，比较了这些攻击对15种无参考图像/视频质量指标的威胁效率。部分指标对对抗攻击表现出高度抵抗性，这表明它们在基准测试中的使用比脆弱指标更为安全。本基准测试接受研究者提交新型指标，以帮助其提升指标对抗攻击的鲁棒性或发现满足需求的稳健指标。请通过 pip install robustness-benchmark 使用我们的基准测试。