Robotic assistance in household environments requires not only predicting where objects should be placed, but also reasoning about when objects should not be placed at all. Existing approaches to personalized object rearrangement primarily focus on placement decisions under the assumption of clean observations and complete actionability, limiting their applicability in realistic, cluttered, and partially erroneous settings. In this paper, we introduce APOLLO, a hybrid framework for abstention-aware personalized object rearrangement that combines a lightweight, personalized embedding model (PEM) with selective large language model (LLM) assistance. PEM is trained for each user-environment pair using a small number of demonstrations, operates entirely on CPU, and produces uncertainty estimates, which are used to selectively invoke LLM-based reasoning only for ambiguous decisions, balancing efficiency, privacy, and reasoning capability. To evaluate this formulation beyond existing benchmarks, we introduce APOR, a synthetic, LLM-generated dataset that captures room-level, multi-furniture environments, diverse organizational profiles, explicit abstention behavior, and noisy partial scene context. Extensive experiments on both PARSEC and APOR provide initial evidence that APOLLO improves over prior LLM-based baselines in controlled benchmark settings while substantially reducing LLM usage. Code is available at https://github.com/PaInt-Lab/APOLLO.

翻译：家庭环境中的机器人辅助不仅需要预测物品应放置的位置，还需要推理何时不应放置物品。现有的个性化物品重排方法主要假设观察清晰且动作完全可行，这限制了它们在真实、杂乱且部分错误的场景中的适用性。在本文中，我们提出了APOLLO——一个用于回避感知个性化物品重排的混合框架，它结合了轻量级个性化嵌入模型（PEM）与选择性大语言模型（LLM）辅助。PEM针对每个用户-环境对使用少量演示进行训练，完全在CPU上运行，并产生不确定性估计，仅对模糊决策选择性调用基于LLM的推理，以平衡效率、隐私和推理能力。为了在现有基准之外评估该方案，我们引入了APOR——一个合成的、由LLM生成的数据集，它捕获了房间级多家具环境、多样化组织配置文件、明确的回避行为以及带噪声的部分场景上下文。在PARSEC和APOR上的大量实验初步证明，APOLLO在受控基准设置中优于先前基于LLM的基线方法，同时显著减少了LLM的使用。代码可在https://github.com/PaInt-Lab/APOLLO获取。