We investigate the use of in-context learning and prompt engineering to estimate the contributions of training data in the outputs of instruction-tuned large language models (LLMs). We propose two novel approaches: (1) a similarity-based approach that measures the difference between LLM outputs with and without provided context, and (2) a mixture distribution model approach that frames the problem of identifying contribution scores as a matrix factorization task. Our empirical comparison demonstrates that the mixture model approach is more robust to retrieval noise in in-context learning, providing a more reliable estimation of data contributions.

翻译：本研究探讨了利用上下文学习与提示工程来估计指令微调大型语言模型（LLM）输出中训练数据的贡献。我们提出了两种新方法：（1）一种基于相似性的方法，通过比较LLM在有提供上下文与无提供上下文时的输出差异来衡量贡献；（2）一种混合分布模型方法，将识别贡献分数的问题构建为矩阵分解任务。我们的实证比较表明，混合模型方法对上下文学习中的检索噪声具有更强的鲁棒性，能够提供更可靠的数据贡献估计。