Safe control methods are often intended to behave safely even in worst-case human uncertainties. However, humans may exploit such safety-first systems, which results in greater risk for everyone. Despite their significance, no prior work has investigated and accounted for such factors in safe control. In this paper, we leverage an interaction-based payoff structure from game theory to model humans' short-sighted, self-seeking behaviors and how humans change their strategies toward machines based on prior experience. We integrate such strategic human behaviors into a safe control architecture. As a result, our approach achieves better safety and performance trade-offs when compared to both deterministic worst-case safe control techniques and equilibrium-based stochastic methods. Our findings suggest an urgent need to fundamentally rethink the safe control framework used in human-technology interaction in pursuit of greater safety for all.

翻译：安全控制方法通常旨在即使在最坏情况下的人类不确定性中也能安全运行。然而，人类可能会利用此类安全优先系统，从而导致每个人面临更大的风险。尽管这些问题具有重要意义，但先前没有研究调查并考虑到安全控制中的此类因素。在本文中，我们利用博弈论中基于互动的收益结构来建模人类短视、自利的行为，以及人类如何基于先前经验改变对机器的策略。我们将这种战略性的人类行为整合到安全控制架构中。结果表明，与确定性最坏情况安全控制技术以及基于均衡的随机方法相比，我们的方法实现了更好的安全性与性能权衡。我们的研究结果表明，迫切需要从根本上重新思考人机交互中使用的安全控制框架，以追求所有人的更高安全性。