Many important behavior changes are frictionful; they require individuals to expend effort over a long period with little immediate gratification. Here, an artificial intelligence (AI) agent can provide personalized interventions to help individuals stick to their goals. In these settings, the AI agent must personalize rapidly (before the individual disengages) and interpretably, to help us understand the behavioral interventions. In this paper, we introduce Behavior Model Reinforcement Learning (BMRL), a framework in which an AI agent intervenes on the parameters of a Markov Decision Process (MDP) belonging to a boundedly rational human agent. Our formulation of the human decision-maker as a planning agent allows us to attribute undesirable human policies (ones that do not lead to the goal) to their maladapted MDP parameters, such as an extremely low discount factor. Furthermore, we propose a class of tractable human models that captures fundamental behaviors in frictionful tasks. Introducing a notion of MDP equivalence specific to BMRL, we theoretically and empirically show that AI planning with our human models can lead to helpful policies on a wide range of more complex, ground-truth humans.

翻译：许多重要的行为改变具有摩擦性；它们要求个体在长时间内付出努力，且难以获得即时满足。在此类情境下，人工智能（AI）主体能够提供个性化干预，帮助个体坚持其目标。在这些设定中，AI主体必须快速（在个体放弃之前）且可解释地进行个性化干预，以帮助我们理解行为干预机制。本文提出行为模型强化学习（BMRL）框架，在该框架中，AI主体对属于有限理性人类主体的马尔可夫决策过程（MDP）参数实施干预。我们将人类决策者建模为规划主体，这一表述使我们能够将不良人类策略（即那些无法达成目标的策略）归因于其适应不良的MDP参数，例如极低的折扣因子。此外，我们提出了一类可处理的人类模型，能够捕捉摩擦性任务中的基本行为。通过引入BMRL特有的MDP等价性概念，我们从理论和实证角度证明，基于我们人类模型的AI规划能够在更广泛且更复杂的真实人类主体上产生有益策略。