Subjective video quality assessment is crucial for optimizing streaming and compression, yet traditional protocols face limitations in capturing nuanced perceptual differences and ensuring reliable user input. We propose an integrated framework that enhances rater training, enforces attention through real-time scoring, and streamlines pairwise comparisons to recover quality scores with fewer comparisons. Participants first undergo an automated training quiz to learn key video quality indicators (e.g., compression artifacts) and verify their readiness. During the test, a real-time attention scoring mechanism, using "golden" video pairs, monitors and reinforces rater focus by applying penalties for lapses. An efficient chain-based pairwise comparison procedure is then employed, yielding quality scores in Just-Objectionable-Differences (JOD) units. Experiments comparing three groups (no training, training without feedback, and training with feedback) with 80 participants demonstrate that training-quiz significantly improves data quality in terms of golden unit accuracy and reduces tie rate, while real-time feedback further improves data quality and yields the most monotonic quality ratings. The new training, quiz, testing with feedback, 3-phase approach can significantly reduce the non-monotonic cases on the high quality part of the R-Q curve where normal viewer typically prefer the slightly compressed less-grainy content and help train a better objective video quality metric.

翻译：主观视频质量评估对于优化流媒体与压缩技术至关重要，然而传统测试方法在捕捉细微感知差异和确保可靠用户输入方面存在局限。本文提出一种集成框架，通过增强评分者训练、利用实时评分强化注意力机制以及优化成对比较流程，以更少的比较次数恢复质量分数。参与者首先通过自动化训练测验学习关键视频质量指标（如压缩伪影）并验证其准备状态。测试过程中，采用基于"黄金"视频对的实时注意力评分机制，通过对注意力缺失施加惩罚来监控并强化评分者的专注度。随后采用高效的链式成对比较程序，以"恰可察觉差异"单位生成质量分数。通过80名参与者对三组（无训练、无反馈训练、有反馈训练）进行的实验表明：训练测验能显著提升黄金单元准确率并降低平局率，从而改善数据质量；实时反馈则能进一步提高数据质量并产生最单调的质量评分。这种包含训练、测验、反馈测试的三阶段新方法，能显著减少R-Q曲线高质量区域的非单调情况——该区域普通观众通常偏好轻微压缩的低颗粒度内容——并有助于训练更优的客观视频质量评估指标。