Automation of test oracles is one of the most challenging facets of software testing, but remains comparatively less addressed compared to automated test input generation. Test oracles rely on a ground-truth that can distinguish between the correct and buggy behavior to determine whether a test fails (detects a bug) or passes. What makes the oracle problem challenging and undecidable is the assumption that the ground-truth should know the exact expected, correct, or buggy behavior. However, we argue that one can still build an accurate oracle without knowing the exact correct or buggy behavior, but how these two might differ. This paper presents SEER, a learning-based approach that in the absence of test assertions or other types of oracle, can determine whether a unit test passes or fails on a given method under test (MUT). To build the ground-truth, SEER jointly embeds unit tests and the implementation of MUTs into a unified vector space, in such a way that the neural representation of tests are similar to that of MUTs they pass on them, but dissimilar to MUTs they fail on them. The classifier built on top of this vector representation serves as the oracle to generate "fail" labels, when test inputs detect a bug in MUT or "pass" labels, otherwise. Our extensive experiments on applying SEER to more than 5K unit tests from a diverse set of open-source Java projects show that the produced oracle is (1) effective in predicting the fail or pass labels, achieving an overall accuracy, precision, recall, and F1 measure of 93%, 86%, 94%, and 90%, (2) generalizable, predicting the labels for the unit test of projects that were not in training or validation set with negligible performance drop, and (3) efficient, detecting the existence of bugs in only 6.5 milliseconds on average.

翻译：测试预言机的自动化是软件测试中最具挑战性的方面之一，但与自动测试输入生成相比，其研究仍相对不足。测试预言机依赖于能够区分正确行为与错误行为的“真实基准”，以判定测试是通过（未检测到缺陷）还是失败（检测到缺陷）。预言机问题之所以具有挑战性且不可判定，是因为其假设“真实基准”需知悉精确的预期正确或错误行为。然而，我们认为，即使未知精确的正确或错误行为，仍可构建精确的预言机——只需了解二者可能存在的差异。本文提出SEER，一种基于学习的方法，在缺乏测试断言或其他类型预言机的情况下，能够判定给定被测方法（MUT）上的单元测试是通过还是失败。为构建“真实基准”，SEER将单元测试与MUT的实现联合嵌入到统一向量空间中，使得测试的神经表示在通过的MUT上相似，而在失败的MUT上相异。基于该向量表示构建的分类器作为预言机：当测试输入在MUT中检测到缺陷时生成“失败”标签，否则生成“通过”标签。我们在来自多样化开源Java项目的5000余个单元测试上对SEER进行了广泛实验，结果表明：（1）所生成的预言机在预测失败或通过标签方面有效，整体准确率、精确率、召回率和F1值分别达到93%、86%、94%和90%；（2）具有良好的泛化能力，对训练集或验证集之外项目的单元测试标签预测性能下降可忽略；（3）高效，平均仅需6.5毫秒即可检测缺陷是否存在。