Generative models are often deployed to make decisions on behalf of users, such as vision-language models (VLMs) identifying which person in a room is a doctor to help visually impaired individuals. Yet, VLM decisions are influenced by the perceived demographic attributes of people in the input, which can lead to biased outcomes like failing to identify women as doctors. Moreover, when reducing bias leads to performance loss, users may have varying needs for balancing bias mitigation with overall model capabilities, highlighting the demand for methods that enable controllable bias reduction during inference. Activation steering is a popular approach for inference-time controllability that has shown potential in inducing safer behavior in large language models (LLMs). However, we observe that current steering methods struggle to correct biases, where equiprobable outcomes across demographic groups are required. To address this, we propose Direct Steering Optimization (DSO) which uses reinforcement learning to find linear transformations for steering activations, tailored to mitigate bias while maintaining control over model performance. We demonstrate that DSO achieves state-of-the-art trade-off between fairness and capabilities on both VLMs and LLMs, while offering practitioners inference-time control over the trade-off. Overall, our work highlights the benefit of designing steering strategies that are directly optimized to control model behavior, providing more effective bias intervention than methods that rely on pre-defined heuristics for controllability.

翻译：生成模型常被部署以代表用户做出决策，例如视觉语言模型（VLM）识别房间中哪个人是医生以帮助视障人士。然而，VLM的决策受到输入中人物感知人口统计属性的影响，这可能导致有偏见的结果，例如未能将女性识别为医生。此外，当减少偏见导致性能损失时，用户可能在平衡偏见缓解与模型整体能力方面有不同需求，这凸显了对在推理过程中实现可控偏见减少方法的需求。激活引导是一种流行的推理时间可控性方法，已显示出在诱导大型语言模型（LLM）更安全行为方面的潜力。然而，我们观察到当前的引导方法难以纠正偏见，因为需要跨人口统计群体的等概率结果。为解决此问题，我们提出了直接引导优化（DSO），该方法使用强化学习来寻找用于引导激活的线性变换，专门用于缓解偏见，同时保持对模型性能的控制。我们证明，DSO在VLM和LLM上实现了公平性与能力之间的最先进权衡，同时为实践者提供了对权衡的推理时间控制。总体而言，我们的工作强调了设计直接优化以控制模型行为的引导策略的好处，相比依赖预定义启发式方法实现可控性的方法，提供了更有效的偏见干预。