There exist both scalable tasks, like reading comprehension and fact-checking, where model performance improves with model size, and unscalable tasks, like arithmetic reasoning and symbolic reasoning, where model performance does not necessarily improve with model size. Large language models (LLMs) equipped with Chain-of-Thought (CoT) prompting are able to make accurate incremental predictions even on unscalable tasks. Unfortunately, despite their exceptional reasoning abilities, LLMs tend to internalize and reproduce discriminatory societal biases. Whether CoT can provide discriminatory or egalitarian rationalizations for the implicit information in unscalable tasks remains an open question. In this study, we examine the impact of LLMs' step-by-step predictions on gender bias in unscalable tasks. For this purpose, we construct a benchmark for an unscalable task where the LLM is given a list of words comprising feminine, masculine, and gendered occupational words, and is required to count the number of feminine and masculine words. In our CoT prompts, we require the LLM to explicitly indicate whether each word in the word list is a feminine or masculine before making the final predictions. With counting and handling the meaning of words, this benchmark has characteristics of both arithmetic reasoning and symbolic reasoning. Experimental results in English show that without step-by-step prediction, most LLMs make socially biased predictions, despite the task being as simple as counting words. Interestingly, CoT prompting reduces this unconscious social bias in LLMs and encourages fair predictions.

翻译：存在可扩展任务（如阅读理解、事实核查，模型性能随规模提升）与不可扩展任务（如算术推理、符号推理，模型性能未必随规模提升）。配备思维链（Chain-of-Thought, CoT）提示的大语言模型（LLMs）即使在不可扩展任务上也能做出准确的增量预测。然而，尽管具有卓越的推理能力，LLMs仍倾向于内化并复现具有歧视性的社会偏见。CoT能否为不可扩展任务中的隐含信息提供歧视性或平等主义的合理化解释，仍是一个悬而未决的问题。本研究探讨了LLMs逐步预测对不可扩展任务中性别偏见的影响。为此，我们构建了一个不可扩展任务的基准测试：向LLM提供包含女性词、男性词及性别化职业词的词表，要求其统计女性和男性词的数量。在CoT提示中，我们要求LLM在做出最终预测前明确标注每个词属于女性或男性。通过计数与词义处理，该基准兼具算术推理与符号推理的特征。英文实验结果表明：在没有逐步预测的情况下，尽管任务仅为简单的单词计数，多数LLMs仍会做出带有社会偏见的预测。有趣的是，CoT提示减少了LLMs中这种无意识的社会偏见，促进了公平预测。