The value-loading problem is a significant challenge for researchers aiming to create artificial intelligence (AI) systems that align with human values and preferences. This problem requires a method to define and regulate safe and optimal limits of AI behaviors. In this work, we propose HALO (Hormetic ALignment via Opponent processes), a regulatory paradigm that uses hormetic analysis to regulate the behavioral patterns of AI. Behavioral hormesis is a phenomenon where low frequencies of a behavior have beneficial effects, while high frequencies are harmful. By modeling behaviors as allostatic opponent processes, we can use either Behavioral Frequency Response Analysis (BFRA) or Behavioral Count Response Analysis (BCRA) to quantify the hormetic limits of repeatable behaviors. We demonstrate how HALO can solve the 'paperclip maximizer' scenario, a thought experiment where an unregulated AI tasked with making paperclips could end up converting all matter in the universe into paperclips. Our approach may be used to help create an evolving database of 'values' based on the hedonic calculus of repeatable behaviors with decreasing marginal utility. This positions HALO as a promising solution for the value-loading problem, which involves embedding human-aligned values into an AI system, and the weak-to-strong generalization problem, which explores whether weak models can supervise stronger models as they become more intelligent. Hence, HALO opens several research avenues that may lead to the development of a computational value system that allows an AI algorithm to learn whether the decisions it makes are right or wrong.

翻译：摘要：价值对齐问题对旨在构建符合人类价值观与偏好的人工智能系统的研究者而言是一项重大挑战。该问题需要一种方法来界定和调控AI行为的安全与最优边界。本文提出HALO（基于拮抗过程的毒理学对齐）框架，这是一种利用毒理学分析调控AI行为模式的监督范式。行为毒理学现象表明，低频率行为具有有益效应，而高频率行为则产生危害。通过将行为建模为稳态拮抗过程，我们可使用行为频率响应分析或行为计数响应分析量化可重复行为的毒理学阈值。我们演示了HALO如何解决"回形针最大化"思想实验——在该假设场景中，一个被赋予制造回形针任务且缺乏监管的AI可能将宇宙中所有物质转化为回形针。我们的方法可基于可重复行为的递减边际效用享乐计算，协助构建动态演化的"价值"数据库。这使得HALO有望成为价值对齐问题（将人类价值观嵌入AI系统的核心难题）与弱到强泛化问题（探索弱模型能否在智能提升过程中监督更强模型）的有效解决方案。因此，HALO开辟了多条研究路径，可能推动构建允许AI算法自主判断决策正确性的计算价值系统。