Despite their outstanding performance, large language models (LLMs) suffer notorious flaws related to their preference for simple, surface-level textual relations over full semantic complexity of the problem. This proposal investigates a common denominator of this problem in their weak ability to generalise outside of the training domain. We survey diverse research directions providing estimations of model generalisation ability and find that incorporating some of these measures in the training objectives leads to enhanced distributional robustness of neural models. Based on these findings, we present future research directions towards enhancing the robustness of LLMs.

翻译：尽管成绩出色,大型语言模型(LLMs)由于偏爱简单的表面文字关系,而不是完全复杂的语义问题,因而有臭名昭著的缺陷。这项建议调查了这个问题的一个共同点,即它们没有能力在培训领域之外进行概括。我们调查了各种研究方向,对典型的概括能力作出估计,发现将其中一些措施纳入培训目标,可提高神经模型的分布性坚固性。我们根据这些调查结果,提出今后研究方向,以加强LMs的稳健性。