In the Indian subcontinent, Telugu, one of India's six classical languages, is the most widely spoken Dravidian Language. Despite its 96 million speaker base worldwide, Telugu remains underrepresented in the global NLP and Machine Learning landscape, mainly due to lack of high-quality annotated resources. This work introduces TeSent, a comprehensive benchmark dataset for sentiment classification, a key text classification problem, in Telugu. TeSent not only provides ground truth labels for the sentences, but also supplements with provisions for evaluating explainability and fairness, two critical requirements in modern-day machine learning tasks. We scraped Telugu texts covering multiple domains from various social media platforms, news websites and web-blogs to preprocess and generate 21,119 sentences, and developed a custom-built annotation platform and a carefully crafted annotation protocol for collecting the ground truth labels along with their human-annotated rationales. We then fine-tuned several SOTA pre-trained models in two ways: with rationales, and without rationales. Further, we provide a detailed plausibility and faithfulness evaluation suite, which exploits the rationales, for six widely used post-hoc explainers applied on the trained models. Lastly, we curate TeEEC, Equity Evaluation Corpus in Telugu, a corpus to evaluate fairness of Telugu sentiment and emotion related NLP tasks, and provide a fairness evaluation suite for the trained classifier models. Our experimental results suggest that training with human rationales improves model accuracy and models' alignment with human reasoning, but does not necessarily reduce bias.

翻译：在印度次大陆，泰卢固语作为印度六大古典语言之一，是最广泛使用的达罗毗荼语。尽管全球有9600万使用者，泰卢固语在全球自然语言处理与机器学习领域仍处于代表性不足的状态，主要归因于高质量标注资源的缺乏。本研究推出了TeSent——一个针对泰卢固语情感分类（文本分类的核心问题）的综合性基准数据集。TeSent不仅为句子提供真实标签，还补充了可解释性与公平性评估机制，这两者是当代机器学习任务中的关键需求。我们从多个社交媒体平台、新闻网站及网络博客中爬取涵盖多领域的泰卢固语文本，经预处理后生成21,119条句子，并开发了定制化标注平台与精心设计的标注流程，以收集真实标签及其人工标注的归因依据。随后，我们以两种方式对多种最先进的预训练模型进行微调：使用归因依据与不使用归因依据。此外，我们基于归因依据构建了详细的合理性与忠实度评估套件，用于对训练模型应用六种广泛使用的后验解释器进行评测。最后，我们构建了泰卢固语公平性评测语料库TeEEC，用于评估泰卢固语情感与情绪相关自然语言处理任务的公平性，并为训练好的分类器模型提供公平性评估套件。实验结果表明，结合人工归因依据进行训练能提升模型准确性及模型与人类推理的对齐度，但未必能减少偏差。