Modern long-context large language models (LLMs) perform well on synthetic "needle-in-a-haystack" (NIAH) benchmarks, but such tests overlook how noisy contexts arise from biased retrieval and agentic workflows. We argue that haystack engineering is necessary to construct noisy long contexts that faithfully capture key real-world factors -- distraction from heterogeneous biased retrievers and cascading errors in agentic workflows -- to test models' long-context robustness. We instantiate it through HaystackCraft, a new NIAH benchmark built on the full English Wikipedia hyperlink network with multi-hop questions. HaystackCraft evaluates how heterogeneous retrieval strategies (e.g., sparse, dense, hybrid, and graph-based) affect distractor composition, haystack ordering, and downstream LLM performance. HaystackCraft further extends NIAH to dynamic, LLM-dependent settings that simulate agentic operations, where models refine queries, reflect on their past reasonings, and decide when to stop. Experiments with 15 long-context models show that (1) while stronger dense retrievers can introduce more challenging distractors, graph-based reranking simultaneously improves retrieval effectiveness and mitigates more harmful distractors; (2) in agentic tests, even advanced models like Gemini 2.5 Pro and GPT-5 suffer cascading failures from self-generated distractors or struggle to perform early stops. These results highlight persistent challenges in agentic long-context reasoning and establish HaystackCraft as a valuable testbed for future progress.

翻译：现代长上下文大语言模型（LLM）在合成式“大海捞针”基准测试中表现优异，但此类测试忽略了噪声上下文如何源于有偏检索与智能体工作流。我们认为，干草堆工程对于构建能忠实反映现实关键因素——即异构有偏检索器带来的干扰与智能体工作流中的级联错误——的噪声长上下文以检验模型的长上下文鲁棒性至关重要。我们通过HaystackCraft实现了这一理念，这是一个基于完整英文维基百科超链接网络构建的、包含多跳问题的新型“大海捞针”基准。HaystackCraft评估了异构检索策略（如稀疏检索、稠密检索、混合检索及基于图的检索）如何影响干扰项构成、干草堆排序以及下游LLM性能。HaystackCraft进一步将“大海捞针”测试扩展至动态、依赖LLM的智能体操作模拟场景，其中模型需优化查询、反思其先前推理过程并决定何时停止。对15个长上下文模型的实验表明：（1）尽管更强的稠密检索器可能引入更具挑战性的干扰项，但基于图的重新排序能同时提升检索效果并缓解更有害的干扰；（2）在智能体测试中，即使是Gemini 2.5 Pro和GPT-5等先进模型，也会因自生成的干扰项而出现级联失败，或难以实现早期停止。这些结果揭示了智能体长上下文推理中持续存在的挑战，并确立了HaystackCraft作为未来进展的重要测试平台。