The landscape of deep learning has vastly expanded the frontiers of source code analysis, particularly through the utilization of structural representations such as Abstract Syntax Trees (ASTs). While these methodologies have demonstrated effectiveness in classification tasks, their efficacy in regression applications, such as execution time prediction from source code, remains underexplored. This paper endeavours to decode the behaviour of tree-based neural network models in the context of such regression challenges. We extend the application of established models--tree-based Convolutional Neural Networks (CNNs), Code2Vec, and Transformer-based methods--to predict the execution time of source code by parsing it to an AST. Our comparative analysis reveals that while these models are benchmarks in code representation, they exhibit limitations when tasked with regression. To address these deficiencies, we propose a novel dual-transformer approach that operates on both source code tokens and AST representations, employing cross-attention mechanisms to enhance interpretability between the two domains. Furthermore, we explore the adaptation of Graph Neural Networks (GNNs) to this tree-based problem, theorizing the inherent compatibility due to the graphical nature of ASTs. Empirical evaluations on real-world datasets showcase that our dual-transformer model outperforms all other tree-based neural networks and the GNN-based models. Moreover, our proposed dual transformer demonstrates remarkable adaptability and robust performance across diverse datasets.

翻译：深度学习的发展极大地拓展了源代码分析的前沿，特别是通过利用抽象语法树（AST）等结构表示方法。尽管这些方法在分类任务中已展现出有效性，但它们在回归应用（例如从源代码预测执行时间）中的效能仍未得到充分探索。本文致力于解析基于树结构的神经网络模型在此类回归挑战中的行为。我们扩展了已有模型的应用——包括基于树的卷积神经网络（CNN）、Code2Vec 以及基于 Transformer 的方法——通过将源代码解析为 AST 来预测其执行时间。我们的对比分析表明，尽管这些模型是代码表示领域的基准模型，但在执行回归任务时仍存在局限性。为弥补这些不足，我们提出了一种新颖的双 Transformer 方法，该方法同时处理源代码标记和 AST 表示，并采用交叉注意力机制以增强两个领域之间的可解释性。此外，我们探索了将图神经网络（GNN）应用于这一基于树结构的问题，基于 AST 的图结构特性，从理论上论证了其内在兼容性。在真实数据集上的实证评估表明，我们的双 Transformer 模型优于所有其他基于树结构的神经网络及基于 GNN 的模型。此外，我们提出的双 Transformer 模型在不同数据集上展现出显著的适应性和鲁棒性能。