Given the surge of harmful AI-generated imagery online, reliably distinguishing authentic images from generated ones has become an urgent research topic. While many proposed detection methods demonstrate high accuracy in controlled settings, their performance often degrades substantially when tested on real-world data. A potential root cause is subtle bias in the detectors' training data. As a result, detectors may rely on spurious correlations instead of learning true forensic artifacts. While recent studies have identified this problem and developed countermeasures, there is not yet an established protocol to evaluate how biased a detector actually is. In this work, we therefore take a step back: First, we discuss what it means for a detector to be biased, and how this differs from a lack of robustness. Second, we propose BIAS-ID, a transparent framework for analyzing and quantifying the presence of transformation biases in AI-generated image detectors. We validate our framework by evaluating six detectors across two datasets, revealing that several state-of-the-art detection methods are strongly affected by common, processing-related biases. Our results highlight the importance of bias-aware evaluation for developing reliable AI-generated image detectors. We provide an easy-to-use library and code to reproduce our results at https://github.com/jonasricker/biasid.

翻译：暂无翻译