The widespread adoption of Large Language Models (LLMs) has made the detection of AI-Generated text a pressing and complex challenge. Although many detection systems report high benchmark accuracy, their reliability in real-world settings remains uncertain, and their interpretability is often unexplored. In this work, we investigate whether contemporary detectors genuinely identify machine authorship or merely exploit dataset-specific artefacts. We propose an interpretable detection framework that integrates linguistic feature engineering, machine learning, and explainable AI techniques. When evaluated on two prominent benchmark corpora, namely PAN CLEF 2025 and COLING 2025, our model trained on 30 linguistic features achieves leaderboard-competitive performance, attaining an F1 score of 0.9734. However, systematic cross-domain and cross-generator evaluation reveals substantial generalisation failure: classifiers that excel in-domain degrade significantly under distribution shift. Using SHAP- based explanations, we show that the most influential features differ markedly between datasets, indicating that detectors often rely on dataset-specific stylistic cues rather than stable signals of machine authorship. Further investigation with in-depth error analysis exposes a fundamental tension in linguistic-feature-based AI text detection: the features that are most discriminative on in-domain data are also the features most susceptible to domain shift, formatting variation, and text-length effects. We believe that this knowledge helps build AI detectors that are robust across different settings. To support replication and practical use, we release an open-source Python package that returns both predictions and instance-level explanations for individual texts.

翻译：大型语言模型（LLMs）的广泛采用使得AI生成文本的检测成为一项紧迫而复杂的挑战。尽管许多检测系统报告了高基准准确率，但其在现实场景中的可靠性仍不确定，且其可解释性常被忽视。本研究探讨了当代检测器是否真正识别机器作者身份，抑或仅是利用数据集特定的人工痕迹。我们提出了一种结合语言特征工程、机器学习与可解释AI技术的可解释检测框架。在PAN CLEF 2025和COLING 2025这两个代表性基准语料库上的评估显示，基于30个语言特征训练的模型达到了排行榜竞争性水平，F1分数为0.9734。然而，系统性跨领域与跨生成器评估揭示了显著的泛化失败：在领域内表现优异的分类器在分布偏移下性能大幅下降。利用基于SHAP的解释，我们发现最具影响力的特征在不同数据集间存在显著差异，这表明检测器往往依赖数据集特定的文体线索而非机器作者身份的稳定信号。通过深入错误分析的进一步调查，揭示了基于语言特征的AI文本检测中的根本性矛盾：在领域内数据上最具区分度的特征，恰恰也是最易受领域偏移、格式变化和文本长度效应影响的特征。我们相信这一认知有助于构建在不同设定下均具鲁棒性的AI检测器。为支持可复现性与实际应用，我们开源了一个Python包，该包可返回对单个文本的预测结果与实例级解释。