The value-loading problem is a significant challenge for researchers aiming to create artificial intelligence (AI) systems that align with human values and preferences. This problem requires a method to define and regulate safe and optimal limits of AI behaviors. In this work, we propose HALO (Hormetic ALignment via Opponent processes), a regulatory paradigm that uses hormetic analysis to regulate the behavioral patterns of AI. Behavioral hormesis is a phenomenon where low frequencies of a behavior have beneficial effects, while high frequencies are harmful. By modeling behaviors as allostatic opponent processes, we can use either Behavioral Frequency Response Analysis (BFRA) or Behavioral Count Response Analysis (BCRA) to quantify the hormetic limits of repeatable behaviors. We demonstrate how HALO can solve the 'paperclip maximizer' scenario, a thought experiment where an unregulated AI tasked with making paperclips could end up converting all matter in the universe into paperclips. Our approach may be used to help create an evolving database of 'values' based on the hedonic calculus of repeatable behaviors with decreasing marginal utility. This positions HALO as a promising solution for the value-loading problem, which involves embedding human-aligned values into an AI system, and the weak-to-strong generalization problem, which explores whether weak models can supervise stronger models as they become more intelligent. Hence, HALO opens several research avenues that may lead to the development of a computational value system that allows an AI algorithm to learn whether the decisions it makes are right or wrong.

翻译：价值对齐问题是研究者构建与人类价值观及偏好一致的人工智能系统时所面临的重大挑战。该问题需要一种方法来定义和规范AI行为的 безопас 且最优界限。本文提出HALO（基于拮抗过程的荷尔美对齐）范式，该调控范式运用荷尔美分析来规范AI的行为模式。行为荷尔美现象是指：行为的低频发生具有有益效果，而高频发生则有害。通过将行为建模为稳态拮抗过程，我们可利用行为频率响应分析（BFRA）或行为计数响应分析（BCRA）量化可重复行为的荷尔美界限。我们展示了HALO如何解决"回形针最大化"场景——这是一个思想实验：未受监管的AI若被赋予制造回形针的任务，可能最终将宇宙中所有物质转化为回形针。我们的方法可协助构建基于可重复行为享乐计算的"价值"演化数据库（此类行为的边际效用递减）。这使HALO成为解决价值对齐问题（将人类对齐价值嵌入AI系统）与弱到强泛化问题（探究弱模型在强模型变得更智能时能否监督后者）的可行方案。因此，HALO开辟了多条研究路径，有望发展出允许AI算法自主判断其决策正误的计算价值系统。