Large Language Models (LLMs) often provide chain-of-thought (CoT) reasoning traces that appear plausible, but may hide internal biases. We call these *unverbalized biases*. Monitoring models via their stated reasoning is therefore unreliable, and existing bias evaluations typically require predefined categories and hand-crafted datasets. In this work, we introduce a fully automated, black-box pipeline for detecting task-specific unverbalized biases. Given a task dataset, the pipeline uses LLM autoraters to generate candidate bias concepts. It then tests each concept on progressively larger input samples by generating positive and negative variations, and applies statistical techniques for multiple testing and early stopping. A concept is flagged as an unverbalized bias if it yields statistically significant performance differences while not being cited as justification in the model's CoTs. We evaluate our pipeline across seven LLMs on three decision tasks (hiring, loan approval, and university admissions). Our technique automatically discovers previously unknown biases in these models (e.g., Spanish fluency, English proficiency, writing formality). In the same run, the pipeline also validates biases that were manually identified by prior work (gender, race, religion, ethnicity). More broadly, our proposed approach provides a practical, scalable path to automatic task-specific bias discovery.

翻译：大型语言模型（LLMs）常提供看似合理的思维链推理轨迹，但可能隐藏内部偏见。我们称此类为*未明言的偏见*。因此，仅通过模型陈述的推理进行监控并不可靠，且现有偏见评估通常需要预定义类别和人工构建的数据集。本研究提出一种全自动、黑盒化的流程，用于检测任务特定的未明言偏见。给定任务数据集，该流程利用LLM自动评估器生成候选偏见概念，随后通过在渐进增大的输入样本上生成正负变体来测试每个概念，并应用多重检验与早期停止的统计技术。若某一概念导致统计显著的性能差异，且未在模型的思维链中被引为判断依据，则被标记为未明言偏见。我们在三个决策任务（招聘、贷款审批和大学录取）上对七种LLM评估了该流程。我们的技术自动发现了这些模型中先前未知的偏见（例如西班牙语流利度、英语熟练度、写作正式性）。在同一运行中，该流程也验证了已有研究通过人工识别的偏见（性别、种族、宗教、民族）。更广泛而言，我们提出的方法为任务特定的自动偏见发现提供了一条实用、可扩展的路径。