Nowadays neural-network-based image- and video-quality metrics show better performance compared to traditional methods. However, they also became more vulnerable to adversarial attacks that increase metrics' scores without improving visual quality. The existing benchmarks of quality metrics compare their performance in terms of correlation with subjective quality and calculation time. However, the adversarial robustness of image-quality metrics is also an area worth researching. In this paper, we analyse modern metrics' robustness to different adversarial attacks. We adopted adversarial attacks from computer vision tasks and compared attacks' efficiency against 15 no-reference image/video-quality metrics. Some metrics showed high resistance to adversarial attacks which makes their usage in benchmarks safer than vulnerable metrics. The benchmark accepts new metrics submissions for researchers who want to make their metrics more robust to attacks or to find such metrics for their needs. Try our benchmark using pip install robustness-benchmark.

翻译：如今，基于神经网络的图像与视频质量指标相比传统方法展现出更优的性能。然而，它们也更容易受到对抗性攻击的影响——这些攻击能在不提升视觉质量的情况下提高指标评分。现有质量指标基准主要比较其与主观评分的相关性及计算时间，但图像质量指标的对抗鲁棒性同样值得研究。本文分析了现代指标对不同对抗攻击的鲁棒性，采用计算机视觉任务中的对抗攻击方法，比较了15种无参考图像/视频质量指标的抗攻击效率。实验表明，部分指标对对抗攻击具有高抵抗力，这使得它们在基准测试中的使用比脆弱指标更安全。本基准测试欢迎研究人员提交新指标，以提升其抗攻击鲁棒性，或帮助用户找到所需的稳健指标。通过pip install robustness-benchmark即可试用我们的基准测试。