We present the structured average intersection-over-union ratio (STRUCT-IOU), a similarity metric between constituency parse trees motivated by the problem of evaluating speech parsers. STRUCT-IOU enables comparison between a constituency parse tree (over automatically recognized spoken word boundaries) with the ground-truth parse (over written words). To compute the metric, we project the ground-truth parse tree to the speech domain by forced alignment, align the projected ground-truth constituents with the predicted ones under certain structured constraints, and calculate the average IOU score across all aligned constituent pairs. STRUCT-IOU takes word boundaries into account and overcomes the challenge that the predicted words and ground truth may not have perfect one-to-one correspondence. Extending to the evaluation of text constituency parsing, we demonstrate that STRUCT-IOU can address token-mismatch issues, and shows higher tolerance to syntactically plausible parses than PARSEVAL (Black et al., 1991).

翻译：本文提出结构化平均交并比（STRUCT-IOU），这是一种针对成分句法分析树相似度的度量方法，其设计动机源于语音分析器的评估需求。STRUCT-IOU 支持将基于自动识别的口语词边界生成的成分句法分析树，与基于书面文本的真实标注句法树进行比较。为计算该指标，我们通过强制对齐将真实标注句法树投影至语音域，在特定结构化约束下将投影后的真实成分与预测成分进行对齐，并计算所有对齐成分对的平均 IOU 分数。STRUCT-IOU 考虑了词边界因素，克服了预测词汇与真实标注之间可能不存在完美一一对应关系的难题。将其扩展至文本成分句法分析评估时，我们证明 STRUCT-IOU 能够处理词汇不匹配问题，且相较于 PARSEVAL（Black 等人，1991）对句法合理的分析结果表现出更高的容忍度。