Large Language Models (LLMs) have demonstrated remarkable capabilities in handling long texts and have almost perfect performance in traditional retrieval tasks. However, their performance significantly degrades when it comes to numerical calculations in the long-context. Numeric-involved long-context tasks typically cannot be addressed by current LLMs in normal settings due to their inherent limitations in simultaneously handling complex and massive information. Some CoT like prompting methods can improve accuracy but demands massive output tokens, which is costly and slow. To address this issue, we propose a workflow, which decompose a numeric-involved long-context task into 4 low-level subtasks: judging, extracting and processing with code and conclusion. The former 2 subtasks is relatively simple, which allows us to use smaller models for efficiently processing long context. When numerical calculations are required, we use code generated by LLMs to avoid the disadvantage of LLM not being good at calculations. The results in 2 numeric-involved long-context benchmarks demonstrate our workflow can not only improve accuracy, but also significantly reduce the cost of API calls.

翻译：大语言模型在处理长文本方面展现出卓越能力，在传统检索任务中几乎达到完美性能。然而，当涉及长上下文中的数值计算时，其性能显著下降。由于大语言模型固有的局限性，难以同时处理复杂且海量的信息，当前大语言模型在常规设置下通常无法解决涉及数值的长上下文任务。一些类似思维链的提示方法虽能提高准确性，但需要大量输出标记，导致成本高昂且速度缓慢。为解决此问题，我们提出一种工作流程，将涉及数值的长上下文任务分解为四个低层子任务：判断、提取、代码处理和结论生成。前两个子任务相对简单，使我们能够使用较小模型高效处理长上下文。当需要进行数值计算时，我们利用大语言模型生成的代码来规避大语言模型不擅长计算的缺陷。在两个涉及数值的长上下文基准测试中的结果表明，我们的工作流程不仅能提高准确性，还能显著降低API调用成本。