Aligning machine learning systems with human expectations is mostly attempted by training with manually vetted human behavioral samples, typically explicit feedback. This is done on a population level since the context that is capturing the subjective Point-Of-View (POV) of a concrete person in a specific situational context is not retained in the data. However, we argue that alignment on an individual level can boost the subjective predictive performance for the individual user interacting with the system considerably. Since perception differs for each person, the same situation is observed differently. Consequently, the basis for decision making and the subsequent reasoning processes and observable reactions differ. We hypothesize that individual perception patterns can be used for improving the alignment on an individual level. We test this, by integrating perception information into machine learning systems and measuring their predictive performance wrt.~individual subjective assessments. For our empirical study, we collect a novel data set of multimodal stimuli and corresponding eye tracking sequences for the novel task of Perception-Guided Crossmodal Entailment and tackle it with our Perception-Guided Multimodal Transformer. Our findings suggest that exploiting individual perception signals for the machine learning of subjective human assessments provides a valuable cue for individual alignment. It does not only improve the overall predictive performance from the point-of-view of the individual user but might also contribute to steering AI systems towards every person's individual expectations and values.

翻译：将机器学习系统与人类期望对齐，目前主要通过经人工验证的人类行为样本（通常为显式反馈）进行训练来实现。这种对齐在群体层面进行，因为捕捉特定情境中具体个人主观视角（POV）的上下文信息并未保留在数据中。然而，我们主张个体层面的对齐能够显著提升与系统交互的个体用户的主观预测性能。由于每个人的感知存在差异，相同情境下的观察结果也不同，进而导致决策基础、后续推理过程及可观察反应均存在差异。我们假设个体感知模式可用于改善个体层面的对齐。为验证这一假设，我们将感知信息整合到机器学习系统中，并测量其相对于个体主观评估的预测性能。在实证研究中，我们针对新型任务——感知引导跨模态蕴含，收集了包含多模态刺激及其对应眼动追踪序列的新数据集，并采用感知引导多模态Transformer进行处理。研究结果表明，利用个体感知信号进行主观人类评估的机器学习，能为个体对齐提供有价值的线索。这不仅从个体用户视角提升了整体预测性能，还有助于引导AI系统朝向符合每个人个体期望与价值观的方向发展。