In image quality assessment, a collective visual quality score for an image or video is obtained from the individual ratings of many subjects. One commonly used format for these experiments is the two-alternative forced choice method. Two stimuli with the same content but differing visual quality are presented sequentially or side-by-side. Subjects are asked to select the one of better quality, and when uncertain, they are required to guess. The relaxed alternative forced choice format aims to reduce the cognitive load and the noise in the responses due to the guessing by providing a third response option, namely, ``not sure''. This work presents a large and comprehensive crowdsourcing experiment to compare these two response formats: the one with the ``not sure'' option and the one without it. To provide unambiguous ground truth for quality evaluation, subjects were shown pairs of images with differing numbers of dots and asked each time to choose the one with more dots. Our crowdsourcing study involved 254 participants and was conducted using a within-subject design. Each participant was asked to respond to 40 pair comparisons with and without the ``not sure'' response option and completed a questionnaire to evaluate their cognitive load for each testing condition. The experimental results show that the inclusion of the ``not sure'' response option in the forced choice method reduced mental load and led to models with better data fit and correspondence to ground truth. We also tested for the equivalence of the models and found that they were different. The dataset is available at http://database.mmsp-kn.de/cogvqa-database.html.

翻译：在图像质量评估中，通过收集众多受试者的个体评分，可获得图像或视频的集体视觉质量分数。此类实验常用的一种格式是二选一强制选择法：将内容相同但视觉质量不同的两个刺激按顺序或并排呈现，要求受试者选择质量更优者，若不确定则必须猜测。为降低因猜测带来的认知负荷及响应噪声，放宽的强制选择格式引入了第三个响应选项——"不确定"。本研究通过大规模众包实验，对比了包含"不确定"选项与不包含该选项的两种响应格式。为提供明确的质量评估基准，受试者需观看不同点数差异的图像对，并每次选出点数较多者。实验采用被试内设计，共254名参与者，每人需完成40组有无"不确定"选项的成对比较，并在每种测试条件下填写认知负荷问卷。结果表明，在强制选择法中引入"不确定"选项降低了心理负荷，使模型具有更佳的数据拟合度及与基准的一致性。模型等价性检验显示两者存在显著差异。数据集可通过 http://database.mmsp-kn.de/cogvqa-database.html 获取。