Large language models (LLMs) have quickly emerged as practical and versatile tools that provide new solutions for a wide range of domains. In this paper, we consider the application of LLMs on symmetric tasks where a query is asked on an (unordered) bag of elements. Examples of such tasks include answering aggregate queries on a database table. In general, when the bag contains a large number of elements, LLMs tend to overlook some elements, leading to challenges in generating accurate responses to the query. LLMs receive their inputs as ordered sequences. However, in this problem, we leverage the fact that the symmetric input is not ordered, and reordering should not affect the LLM's response. Observing that LLMs are less likely to miss elements at certain positions of the input, we introduce the problem of LLM input reranking: to find a ranking of the input that maximizes the LLM's accuracy for the given query without making explicit assumptions about the query. Finding the optimal ranking requires identifying (i) the relevance of each input element for answering the query and (ii) the importance of each rank position for the LLM's attention. We develop algorithms for estimating these values efficiently utilizing a helper LLM. We conduct comprehensive experiments on different synthetic and real datasets to validate our proposal and to evaluate the effectiveness of our proposed algorithms. Our experiments confirm that our reranking approach improves the accuracy of the LLMs on symmetric tasks by up to $99\%$ proximity to the optimum upper bound.

翻译：大型语言模型（LLM）已迅速成为实用且多功能的工具，为广泛领域提供了新的解决方案。本文探讨LLM在对称任务中的应用，此类任务针对（无序的）元素集合进行查询。例如，对数据库表进行聚合查询即属此类任务。一般而言，当集合包含大量元素时，LLM容易忽略部分元素，导致难以生成准确的查询响应。LLM以有序序列形式接收输入，但本问题利用对称输入本身无序且重排不应影响LLM响应的特性。通过观察发现LLM对输入序列中特定位置元素的忽略概率较低，我们提出LLM输入重排问题：在不显式假设查询内容的前提下，寻找能使LLM对给定查询达到最高准确度的输入排序方案。寻找最优排序需确定：（1）每个输入元素对回答查询的相关性；（2）每个排序位置对LLM注意力机制的重要性。我们开发了利用辅助LLM高效估计这些值的算法。通过在合成与真实数据集上的综合实验验证方案可行性并评估算法有效性。实验结果表明，我们的重排方法将LLM在对称任务上的准确度提升至最优理论上限的$99\%$接近度。