Automatic Speech Recognition (ASR) in medical contexts has the potential to save time, cut costs, increase report accuracy, and reduce physician burnout. However, the healthcare industry has been slower to adopt this technology, in part due to the importance of avoiding medically-relevant transcription mistakes. In this work, we present the Clinical BERTScore (CBERTScore), an ASR metric that penalizes clinically-relevant mistakes more than others. We demonstrate that this metric more closely aligns with clinician preferences on medical sentences as compared to other metrics (WER, BLUE, METEOR, etc), sometimes by wide margins. We collect a benchmark of 18 clinician preferences on 149 realistic medical sentences called the Clinician Transcript Preference benchmark (CTP) and make it publicly available for the community to further develop clinically-aware ASR metrics. To our knowledge, this is the first public dataset of its kind. We demonstrate that CBERTScore more closely matches what clinicians prefer.

翻译：在医疗场景中，自动语音识别（ASR）技术有望节省时间、降低成本、提高报告准确性并减少医生职业倦怠。然而，由于避免医学相关转录错误的重要性，医疗行业对该技术的采用速度相对滞后。本研究提出临床BERTScore（CBERTScore），这是一种对临床相关错误进行更严厉惩罚的ASR评价指标。我们证明，与WER、BLEU、METEOR等其他指标相比（某些情况下差距显著），该指标更贴近临床医生对医学语句的偏好。我们收集了149条医学语句上的18项临床医生偏好基准（称为临床转录偏好基准CTP），并将其公开供学界进一步开发具备临床意识的ASR评估指标。据我们所知，这是首个公开的此类数据集。我们的实验表明，CBERTScore更精准地反映了临床医生的实际偏好。