Large language models (LLM) often hallucinate, and while adding citations is a common solution, it is frequently insufficient for accountability as users struggle to verify how a cited source supports a generated claim. Existing methods are typically coarse-grained and fail to distinguish between direct quotes and complex reasoning. In this paper, we introduce Generation-time Fine-grained Provenance, a task where models must generate fluent answers while simultaneously producing structured, sentence-level provenance triples. To enable this, we present ReFInE (Relation-aware Fine-grained Interpretability & Evidence), a dataset featuring expert verified annotations that distinguish between Quotation, Compression, and Inference. Building on ReFInE, we propose GenProve, a framework that combines Supervised Fine-Tuning (SFT) with Group Relative Policy Optimization (GRPO). By optimizing a composite reward for answer fidelity and provenance correctness, GenProve significantly outperforms 14 strong LLMs in joint evaluation. Crucially, our analysis uncovers a reasoning gap where models excel at surface-level quotation but struggle significantly with inference-based provenance, suggesting that verifiable reasoning remains a frontier challenge distinct from surface-level citation.

翻译：大型语言模型（LLM）常产生幻觉，尽管添加引用是常见的解决方案，但对于确保可问责性往往不足，因为用户难以验证所引用的来源如何支撑生成的主张。现有方法通常是粗粒度的，无法区分直接引用与复杂推理。本文提出生成时细粒度来源追溯任务，要求模型在生成流畅答案的同时，产生结构化的句子级来源三元组。为此，我们构建了ReFInE（关系感知的细粒度可解释性与证据）数据集，该数据集包含专家验证的标注，能够区分直接引用、压缩概括与推理推断。基于ReFInE，我们提出GenProve框架，该框架结合了监督微调（SFT）与组相对策略优化（GRPO）。通过优化答案忠实度与来源正确性的复合奖励函数，GenProve在联合评估中显著优于14个强大的LLM。关键的是，我们的分析揭示了一个推理鸿沟：模型在表层引用方面表现优异，但在基于推理的来源追溯上存在显著困难，这表明可验证的推理仍然是区别于表层引用的前沿挑战。