The extensive scope of large language models (LLMs) across various domains underscores the critical importance of responsibility in their application, beyond natural language processing. In particular, the randomized nature of LLMs, coupled with inherent biases and historical stereotypes in data, raises critical concerns regarding reliability and equity. Addressing these challenges are necessary before using LLMs for applications with societal impact. Towards addressing this gap, we introduce REQUAL-LM, a novel method for finding reliable and equitable LLM outputs through aggregation. Specifically, we develop a Monte Carlo method based on repeated sampling to find a reliable output close to the mean of the underlying distribution of possible outputs. We formally define the terms such as reliability and bias, and design an equity-aware aggregation to minimize harmful bias while finding a highly reliable output. REQUAL-LM does not require specialized hardware, does not impose a significant computing load, and uses LLMs as a blackbox. This design choice enables seamless scalability alongside the rapid advancement of LLM technologies. Our system does not require retraining the LLMs, which makes it deployment ready and easy to adapt. Our comprehensive experiments using various tasks and datasets demonstrate that REQUAL- LM effectively mitigates bias and selects a more equitable response, specifically the outputs that properly represents minority groups.

翻译：大语言模型在各领域的广泛应用，凸显了其在自然语言处理之外的应用中责任性的关键重要性。特别是，大语言模型的随机性，加上数据中固有的偏见和历史刻板印象，引发了关于可靠性和公平性的严重关切。在将大语言模型用于具有社会影响的应用之前，必须解决这些挑战。为填补这一空白，我们提出REQUAL-LM，一种通过聚合寻找可靠且公平的大语言模型输出的新型方法。具体而言，我们开发了一种基于重复采样的蒙特卡洛方法，以找到接近可能输出分布均值的可靠输出。我们正式定义了可靠性和偏见等术语，并设计了一种具有公平性意识的聚合方法，在寻找高可靠性输出的同时最小化有害偏见。REQUAL-LM不需要专用硬件，不增加显著计算负担，并将大语言模型作为黑箱使用。这一设计选择使其能够随着大语言模型技术的快速进步而无缝扩展。我们的系统无需重新训练大语言模型，因此易于部署和适配。通过使用多种任务和数据集进行的全面实验表明，REQUAL-LM有效缓解了偏见，并选择了更公平的响应，特别是能够恰当代表少数群体的输出。