The proliferation of open-source Large Language Models (LLMs) underscores the pressing need for evaluation methods. Existing works primarily rely on external evaluators, focusing on training and prompting strategies. However, a crucial aspect - model-aware glass-box features - is overlooked. In this study, we explore the utility of glass-box features under the scenario of self-evaluation, namely applying an LLM to evaluate its own output. We investigate various glass-box feature groups and discovered that the softmax distribution serves as a reliable indicator for quality evaluation. Furthermore, we propose two strategies to enhance the evaluation by incorporating features derived from references. Experimental results on public benchmarks validate the feasibility of self-evaluation of LLMs using glass-box features.

翻译：开源大语言模型的广泛普及凸显了评估方法的迫切需求。现有研究主要依赖外部评估器，聚焦于训练和提示策略。然而，一个关键方面——模型感知的玻璃箱特征——却被忽视了。本研究探索了玻璃箱特征在自评估场景下的实用性，即利用大语言模型评估自身输出。我们研究了多种玻璃箱特征组，发现softmax分布可作为质量评估的可靠指标。此外，我们提出两种策略，通过融入源自参考的特征来增强评估效果。在公开基准上的实验结果验证了基于玻璃箱特征实现大语言模型自评估的可行性。