Distribution shift has been a longstanding challenge for the reliable deployment of deep learning (DL) models due to unexpected accuracy degradation. Although DL has been becoming a driving force for large-scale source code analysis in the big code era, limited progress has been made on distribution shift analysis and benchmarking for source code tasks. To fill this gap, this paper initiates to propose CodeS, a distribution shift benchmark dataset, for source code learning. Specifically, CodeS supports two programming languages (Java and Python) and five shift types (task, programmer, time-stamp, token, and concrete syntax tree). Extensive experiments based on CodeS reveal that 1) out-of-distribution detectors from other domains (e.g., computer vision) do not generalize to source code, 2) all code classification models suffer from distribution shifts, 3) representation-based shifts have a higher impact on the model than others, and 4) pre-trained bimodal models are relatively more resistant to distribution shifts.

翻译：分布偏移一直因意外的精度退化而成为深度学习模型可靠部署的长期挑战。尽管在大代码时代，深度学习已成为大规模源代码分析的核心驱动力，但在源代码任务的分布偏移分析与基准测试方面进展有限。为弥补这一空白，本文首次提出CodeS——一个面向源代码学习的分布偏移基准数据集。具体而言，CodeS支持两种编程语言（Java与Python）及五种偏移类型（任务、程序员、时间戳、令牌与具体语法树）。基于CodeS的广泛实验揭示：1）其他领域（如计算机视觉）的分布外检测器无法泛化至源代码；2）所有代码分类模型均受分布偏移影响；3）基于表示的偏移对模型影响高于其他类型；4）预训练双模态模型对分布偏移具有相对更强的抵抗能力。