The starting point of our network architecture is the Credibility Transformer which extends the classical Transformer architecture by a credibility mechanism to improve model learning and predictive performance. This Credibility Transformer learns credibilitized CLS tokens that serve as learned representations of the original input features. In this paper we present a new paradigm that augments this architecture by an in-context learning mechanism, i.e., we increase the information set by a context batch consisting of similar instances. This allows the model to enhance the CLS token representations of the instances by additional in-context information and fine-tuning. We empirically verify that this in-context learning enhances predictive accuracy by adapting to similar risk patterns. Moreover, this in-context learning also allows the model to generalize to new instances which, e.g., have feature levels in the categorical covariates that have not been present when the model was trained -- for a relevant example, think of a new vehicle model which has just been developed by a car manufacturer.

翻译：我们网络架构的起点是可信度Transformer，它通过引入可信度机制扩展了经典Transformer架构，以提升模型学习与预测性能。该可信度Transformer学习生成可信化的CLS令牌，作为原始输入特征的学习表征。本文提出一种新范式，通过上下文学习机制对此架构进行增强，即通过由相似实例组成的上下文批次扩展信息集。这使得模型能够利用额外的上下文信息与微调来增强实例的CLS令牌表征。我们通过实证验证，这种上下文学习能通过适应相似风险模式来提升预测准确性。此外，该上下文学习机制还使模型能够泛化到训练时未出现过的实例——例如在分类协变量中具有未见过特征水平的实例；相关案例如汽车制造商刚研发的新型车辆模型。