Large Language Models (LLMs) are increasingly used for knowledge-based reasoning tasks, yet understanding when they rely on genuine knowledge versus superficial heuristics remains challenging. We investigate this question through entity comparison tasks by asking models to compare entities along numerical attributes (e.g., ``Which river is longer, the Danube or the Nile?''), which offer clear ground truth for systematic analysis. Despite having sufficient numerical knowledge to answer correctly, LLMs frequently make predictions that contradict this knowledge. We identify three heuristic biases that strongly influence model predictions: entity popularity, mention order, and semantic co-occurrence. For smaller models, a simple logistic regression using only these surface cues predicts model choices more accurately than the model's own numerical predictions, suggesting heuristics largely override principled reasoning. Crucially, we find that larger models (32B parameters) selectively rely on numerical knowledge when it is more reliable, while smaller models (7--8B parameters) show no such discrimination, which explains why larger models outperform smaller ones even when the smaller models possess more accurate knowledge. Chain-of-thought prompting steers all models towards using the numerical features across all model sizes.

翻译：大型语言模型（LLMs）越来越多地用于基于知识的推理任务，但理解它们何时依赖真实知识而非表面启发式方法仍然具有挑战性。我们通过实体比较任务来研究这个问题，要求模型沿着数值属性比较实体（例如“多瑙河与尼罗河，哪条河流更长？”），这为系统分析提供了清晰的基本事实。尽管拥有足够的数值知识来正确回答，LLMs 经常做出与这些知识相矛盾的预测。我们识别出三种强烈影响模型预测的启发式偏差：实体流行度、提及顺序和语义共现。对于较小的模型，仅使用这些表面线索的简单逻辑回归比模型自身的数值预测更能准确预测模型的选择，这表明启发式方法在很大程度上覆盖了原则性推理。关键的是，我们发现较大的模型（320亿参数）在数值知识更可靠时有选择地依赖它，而较小的模型（70-80亿参数）则没有表现出这种区分能力，这解释了为什么即使较小的模型拥有更准确的知识，较大的模型仍然表现更优。思维链提示在所有模型规模上都能引导模型使用数值特征。