The prediction has served as a crucial scientific method in modern social studies. With the recent advancement of Large Language Models (LLMs), efforts have been made to leverage LLMs to predict the human features in social life, such as presidential voting. These works suggest that LLMs are capable of generating human-like responses. However, we find that the promising performance achieved by previous studies is because of the existence of input shortcut features to the response. In fact, by removing these shortcuts, the performance is reduced dramatically. To further revisit the ability of LLMs, we introduce a novel social prediction task, Soc-PRF Prediction, which utilizes general features as input and simulates real-world social study settings. With the comprehensive investigations on various LLMs, we reveal that LLMs cannot work as expected on social prediction when given general input features without shortcuts. We further investigate possible reasons for this phenomenon that suggest potential ways to enhance LLMs for social prediction.

翻译：预测已经成为现代社会研究中一项至关重要的科学方法。随着大型语言模型（LLMs）的最新进展，研究者们尝试利用LLMs来预测社会生活中的人类特征，例如总统选举。这些研究表明，LLMs能够生成类似人类的回应。然而，我们发现先前研究所取得的优异表现，实际上是由于输入中存在的快捷特征直接影响到了回应。事实上，去除这些快捷特征后，模型性能会大幅下降。为了进一步审视LLMs的能力，我们提出了一项新的社会预测任务——Soc-PRF预测，该任务使用通用特征作为输入，并模拟真实世界的社会研究场景。通过对多种LLMs的全面调查，我们揭示出：在缺乏快捷特征的一般输入条件下，LLMs无法如预期般有效进行社会预测。我们进一步探讨了导致这一现象的可能原因，并提出了增强LLMs社会预测能力的潜在方向。