Large Language Models (LLMs) are increasingly applied in recommendation scenarios due to their strong natural language understanding and generation capabilities. However, they are trained on vast corpora whose contents are not publicly disclosed, raising concerns about data leakage. Recent work has shown that the MovieLens-1M dataset is memorized by both the LLaMA and OpenAI model families, but the extraction of such memorized data has so far relied exclusively on manual prompt engineering. In this paper, we pose three main questions: Is it possible to enhance manual prompting? Can LLM memorization be detected through methods beyond manual prompting? And can the detection of data leakage be automated? To address these questions, we evaluate three approaches: (i) jailbreak prompt engineering; (ii) unsupervised latent knowledge discovery, probing internal activations via Contrast-Consistent Search (CCS) and Cluster-Norm; and (iii) Automatic Prompt Engineering (APE), which frames prompt discovery as a meta-learning process that iteratively refines candidate instructions. Experiments on MovieLens-1M using LLaMA models show that jailbreak prompting does not improve the retrieval of memorized items and remains inconsistent; CCS reliably distinguishes genuine from fabricated movie titles but fails on numerical user and rating data; and APE retrieves item-level information with moderate success yet struggles to recover numerical interactions. These findings suggest that automatically optimizing prompts is the most promising strategy for extracting memorized samples.

翻译：大型语言模型（LLM）凭借其强大的自然语言理解与生成能力，正日益广泛应用于推荐场景。然而，这些模型基于内容未公开的海量语料库进行训练，引发了数据泄露的担忧。近期研究表明，MovieLens-1M 数据集已被 LLaMA 和 OpenAI 系列模型记忆，但此类记忆数据的提取目前完全依赖于人工提示工程。本文提出三个核心问题：能否增强人工提示的效果？能否通过人工提示以外的方法检测 LLM 的记忆化？数据泄露的检测能否实现自动化？为回答这些问题，我们评估了三种方法：（i）越狱提示工程；（ii）无监督潜在知识发现，通过对比一致搜索（CCS）和聚类范数（Cluster-Norm）探测内部激活；（iii）自动提示工程（APE），将提示发现构建为迭代优化候选指令的元学习过程。在 LLaMA 模型上使用 MovieLens-1M 数据集的实验表明：越狱提示未能提升记忆条目的检索效果且结果不稳定；CCS 能可靠区分真实与虚构的电影名称，但无法处理数值型用户与评分数据；APE 在条目级信息检索中取得一定成功，但难以恢复数值型交互记录。这些发现表明，自动优化提示是提取记忆样本最具潜力的策略。