We present the structured average intersection-over-union ratio (STRUCT-IOU), a similarity metric between constituency parse trees motivated by the problem of evaluating speech parsers. STRUCT-IOU enables comparison between a constituency parse tree (over automatically recognized spoken word boundaries) with the ground-truth parse (over written words). To compute the metric, we project the ground-truth parse tree to the speech domain by forced alignment, align the projected ground-truth constituents with the predicted ones under certain structured constraints, and calculate the average IOU score across all aligned constituent pairs. STRUCT-IOU takes word boundaries into account and overcomes the challenge that the predicted words and ground truth may not have perfect one-to-one correspondence. Extending to the evaluation of text constituency parsing, we demonstrate that STRUCT-IOU shows higher tolerance to syntactically plausible parses than PARSEVAL (Black et al., 1991).

翻译：我们提出结构化平均交并比（STRUCT-IOU），这是一种基于语音解析器评估问题而设计的成分句法树相似度度量方法。STRUCT-IOU能够比较（基于自动识别的口语词边界生成的）成分句法树与（基于书面词构建的）真实句法树。为计算该度量，我们通过强制对齐将真实句法树投影至语音域，在特定结构化约束条件下将投影后的真实成分与预测成分进行对齐，并计算所有对齐成分对的平均交并比分数。STRUCT-IOU将词边界纳入考量，克服了预测词与真实词之间可能不存在完美一一对应关系的挑战。通过扩展至文本成分句法分析评估，我们证明STRUCT-IOU相比PARSEVAL（Black等人，1991）对句法合理解析结果具有更高的容忍度。