Understanding and quantifying uncertainty in large language model (LLM) outputs is critical for reliable deployment. However, traditional evaluation approaches provide limited insight into model confidence at individual token positions during generation. To address this issue, we introduce LogitScope, a lightweight framework for analyzing LLM uncertainty through token-level information metrics computed from probability distributions. By measuring metrics such as entropy and varentropy at each generation step, LogitScope reveals patterns in model confidence, identifies potential hallucinations, and exposes decision points where models exhibit high uncertainty, all without requiring labeled data or semantic interpretation. We demonstrate LogitScope's utility across diverse applications including uncertainty quantification, model behavior analysis, and production monitoring. The framework is model-agnostic, computationally efficient through lazy evaluation, and compatible with any HuggingFace model, enabling both researchers and practitioners to inspect LLM behavior during inference.

翻译：理解和量化大语言模型（LLM）输出的不确定性对于可靠部署至关重要。然而，传统评估方法在生成过程中对单个词元位置的模型置信度仅能提供有限的洞察。为解决这一问题，我们提出LogitScope——一种轻量级框架，通过从概率分布中计算词元级信息指标来分析LLM不确定性。通过在每个生成步骤中测量熵和方差熵等指标，LogitScope能够揭示模型置信度的模式、识别潜在幻觉，并暴露模型表现出高不确定性的决策点，全程无需标注数据或语义解释。我们展示了LogitScope在不确定性量化、模型行为分析和生产监控等多种应用中的实用性。该框架具有模型无关性、通过惰性评估实现计算高效性，并兼容任何HuggingFace模型，使研究人员和从业者都能在推理过程中检查LLM行为。