Vision-language models (VLMs) perform well on many document understanding tasks, yet their reliability in specialized, non-English domains remains underexplored. This gap is especially critical in finance, where documents mix dense regulatory text, numerical tables, and visual charts, and where extraction errors can have real-world consequences. We introduce Scribe Finance, the first multimodal benchmark for evaluating French financial document understanding. The dataset contains 1,204 expert-validated questions spanning text extraction, table comprehension, chart interpretation, and multi-turn conversational reasoning, drawn from real investment prospectuses, KIDs, and PRIIPs. We evaluate six open-weight VLMs (8B-124B parameters) using an LLM-as-judge protocol. While models achieve strong performance on text and table tasks (85-90% accuracy), they struggle with chart interpretation (34-62%). Most notably, multi-turn dialogue reveals a sharp failure mode: early mistakes propagate across turns, driving accuracy down to roughly 50% regardless of model size. These results show that current VLMs are effective for well-defined extraction tasks but remain brittle in interactive, multi-step financial analysis. Scribe Finance offers a challenging benchmark to measure and drive progress in this high-stakes setting.

翻译：视觉语言模型（VLM）在许多文档理解任务上表现良好，但其在非英语专业领域的可靠性仍未得到充分探索。这一空白在金融领域尤为关键，因为该领域的文档混合了密集的监管文本、数值表格和可视化图表，且提取错误可能产生实际影响。我们引入了Scribe Finance，这是首个用于评估法文金融文档理解的多模态基准测试。该数据集包含1,204个经专家验证的问题，涵盖文本提取、表格理解、图表解读以及多轮对话推理，所有问题均来自真实的投资说明书、关键信息文件（KID）和合成/标准化关键信息文件（PRIIP）。我们使用LLM-as-judge协议评估了六个开源权重VLM（参数量从8B到124B）。虽然这些模型在文本和表格任务上表现出色（准确率85-90%），但在图表解读方面存在困难（准确率34-62%）。最值得注意的是，多轮对话暴露了一个严重的失败模式：早期错误会跨轮次传播，导致准确率无论模型大小均下降至约50%。这些结果表明，当前VLM在定义明确的提取任务上有效，但在交互式、多步骤的金融分析中仍然脆弱。Scribe Finance为衡量和推动这一高风险场景的进展提供了一个具有挑战性的基准。