We present a case study evaluating large language models (LLMs) with 128K-token context windows on a technical question answering (QA) task. Our benchmark is built on a user manual for an agricultural machine, available in English, French, and German. It simulates a cross-lingual information retrieval scenario where questions are posed in English against all three language versions of the manual. The evaluation focuses on realistic "needle-in-a-haystack" challenges and includes unanswerable questions to test for hallucinations. We compare nine long-context LLMs using direct prompting against three Retrieval-Augmented Generation (RAG) strategies (keyword, semantic, hybrid), with an LLM-as-a-judge for evaluation. Our findings for this specific manual show that Hybrid RAG consistently outperforms direct long-context prompting. Models like Gemini 2.5 Flash and the smaller Qwen 2.5 7B achieve high accuracy (over 85%) across all languages with RAG. This paper contributes a detailed analysis of LLM performance in a specialized industrial domain and an open framework for similar evaluations, highlighting practical trade-offs and challenges.

翻译：我们提出一项案例研究，评估具有128K令牌上下文窗口的大型语言模型在技术问答任务上的表现。我们的基准测试基于一份农业机械的用户手册，该手册提供英语、法语和德语版本。该研究模拟了一个跨语言信息检索场景：用户以英语提问，而检索范围涵盖手册的所有三种语言版本。评估重点关注现实中的"大海捞针"式挑战，并包含不可回答问题以测试模型的幻觉生成倾向。我们比较了九种采用直接提示的长上下文LLM与三种检索增强生成策略（关键词检索、语义检索、混合检索）的表现，并采用LLM作为评判员进行评估。针对该特定手册的研究结果表明：混合RAG策略始终优于直接长上下文提示方法。在使用RAG时，Gemini 2.5 Flash和较小规模的Qwen 2.5 7B等模型在所有语言版本中均实现了高准确率（超过85%）。本文通过详细分析LLM在特定工业领域的表现，并提供一个可复用的开放评估框架，揭示了实际应用中的权衡与挑战。