Cheap-to-Build Very Large-Language Models (CtB-LLMs) with affordable training are emerging as the next big revolution in natural language processing and understanding. These CtB-LLMs are democratizing access to trainable Very Large-Language Models (VLLMs) and, thus, may represent the building blocks of many NLP systems solving downstream tasks. Hence, a little or a large bias in CtB-LLMs may cause huge harm. In this paper, we performed a large investigation of the bias of three families of CtB-LLMs, and we showed that debiasing techniques are effective and usable. Indeed, according to current tests, the LLaMA and the OPT families have an important bias in gender, race, religion, and profession. In contrast to the analysis for other LLMs, we discovered that bias depends not on the number of parameters but on the perplexity. Finally, the debiasing of OPT using LoRA reduces bias up to 4.12 points in the normalized stereotype score.

翻译：廉价可建的超大规模语言模型（CtB-LLMs）因其可负担的训练成本，正成为自然语言处理与理解领域的下一次重大革命。这些CtB-LLMs使得可训练的超大规模语言模型（VLLMs）的获取变得民主化，因而可能成为众多解决下游任务的自然语言处理系统的构建基础。因此，CtB-LLMs中哪怕微小或显著的偏见都可能造成巨大危害。本文对三个CtB-LLM家族的偏见进行了大规模研究，并证明去偏技术是有效且可用的。事实上，根据当前测试，LLaMA和OPT家族在性别、种族、宗教和职业方面存在显著偏见。与其他大型语言模型的分析不同，我们发现偏见不取决于参数量，而取决于困惑度。最后，使用LoRA对OPT进行去偏处理，可将归一化刻板印象评分中的偏见降低最多4.12分。