Large Language Models (LLMs) exhibit impressive zero/few-shot inference and generation quality for high-resource languages (HRLs). A few of them have been trained on low-resource languages (LRLs) and give decent performance. Owing to the prohibitive costs of training LLMs, they are usually used as a network service, with the client charged by the count of input and output tokens. The number of tokens strongly depends on the script and language, as well as the LLM's subword vocabulary. We show that LRLs are at a pricing disadvantage, because the well-known LLMs produce more tokens for LRLs than HRLs. This is because most currently popular LLMs are optimized for HRL vocabularies. Our objective is to level the playing field: reduce the cost of processing LRLs in contemporary LLMs while ensuring that predictive and generative qualities are not compromised. As means to reduce the number of tokens processed by the LLM, we consider code-mixing, translation, and transliteration of LRLs to HRLs. We perform an extensive study using the IndicXTREME classification and six generative tasks dataset, covering 15 Indic and 3 other languages, while using GPT-4 (one of the costliest LLM services released so far) as a commercial LLM. We observe and analyze interesting patterns involving token count, cost, and quality across a multitude of languages and tasks. We show that choosing the best policy to interact with the LLM can reduce cost by 90% while giving better or comparable performance compared to communicating with the LLM in the original LRL.

翻译：大语言模型（LLMs）在针对高资源语言（HRLs）的零样本/少样本推理和生成任务中展现出卓越质量。部分模型经过低资源语言（LRLs）训练后亦具备可观性能。由于训练LLMs的巨额成本，其通常以网络服务形式使用，客户端按输入输出令牌数量计费。令牌数高度依赖语言脚本特征、语言类型及LLMs的子词词汇表。研究发现，知名LLMs对LRLs产生的令牌数多于HRLs，导致LRLs面临定价劣势。这源于当前主流LLMs普遍针对HRL词汇表优化。本研究旨在平衡竞争环境：在确保预测与生成质量不受损的前提下，降低当代LLMs处理LRLs的成本。我们采用代码混合、翻译及音译三种方式将LRLs转换为HRLs，以减少LLM处理的令牌数量。通过使用GPT-4（迄今已发布的最昂贵商业LLM服务之一）作为评估对象，基于IndicXTREME分类数据集及六项生成任务数据集（涵盖15种印度语言及3种其他语言）开展广泛研究。我们观察并分析了涉及令牌数、成本与质量的多语言任务间的有趣规律。研究表明，选择最优交互策略可在保持优于或等同于原始LRLs的通信性能时，将成本降低90%。