Mitigating algorithmic bias is a critical task in the development and deployment of machine learning models. While several toolkits exist to aid machine learning practitioners in addressing fairness issues, little is known about the strategies practitioners employ to evaluate model fairness and what factors influence their assessment, particularly in the context of text classification. Two common approaches of evaluating the fairness of a model are group fairness and individual fairness. We run a study with Machine Learning practitioners (n=24) to understand the strategies used to evaluate models. Metrics presented to practitioners (group vs. individual fairness) impact which models they consider fair. Participants focused on risks associated with underpredicting/overpredicting and model sensitivity relative to identity token manipulations. We discover fairness assessment strategies involving personal experiences or how users form groups of identity tokens to test model fairness. We provide recommendations for interactive tools for evaluating fairness in text classification.

翻译：缓解算法偏差是机器学习模型开发与部署中的关键任务。尽管已有多种工具包可协助机器学习从业者解决公平性问题，但关于从业者评估模型公平性所采用的策略及其影响因素——尤其是在文本分类场景中——仍知之甚少。评估模型公平性的两种常见方法是群体公平与个体公平。我们开展了一项包含24名机器学习从业者的研究，以理解其评估模型的策略。呈现给从业者的公平性度量指标（群体公平对比个体公平）会影响他们对哪些模型视为公平的判断。参与者重点关注与预测不足/预测过度相关的风险，以及模型对身份标记操纵的敏感性。我们发现了涉及个人经验或用户如何构建身份标记群组以测试模型公平性的公平性评估策略。最后，我们为文本分类公平性评估的交互式工具提出了建议。