We view the landscape of large language models (LLMs) through the lens of the recently released BLOOM model to understand the performance of BLOOM and other decoder-only LLMs compared to BERT-style encoder-only models. We achieve this by evaluating the smaller BLOOM model variants (\textit{350m/560m} and \textit{1b3/1b7}) on several NLP benchmark datasets and popular leaderboards. We make the following observations: (1) BLOOM performance does not scale with parameter size, unlike other LLMs like GPT and BERT. Experiments fine-tuning BLOOM models show that the 560m variant performs similarly to or better than the 1b7 variant, (2) Zero-shot cross-lingual and multi-lingual fine-tuning experiments show that BLOOM is at par or worse than monolingual GPT-2 models, and (3) Toxicity analysis of prompt-based text generation using the RealToxicityPrompts dataset shows that the text generated by BLOOM is at least 17\% less toxic than GPT-2 and GPT-3 models.

翻译：我们以近期发布的BLOOM模型为切入点，通过大型语言模型（LLMs）的视角，探究BLOOM及其他仅解码器架构的LLMs与BERT式仅编码器模型相比的性能表现。为此，我们评估了BLOOM小型变体模型（\textit{350m/560m}和\textit{1b3/1b7}）在多个NLP基准数据集及主流排行榜上的表现，并获得以下发现：（1）与GPT、BERT等其他LLMs不同，BLOOM的性能并不随参数规模扩大而提升。对BLOOM模型进行微调实验表明，560m变体的表现与1b7变体相当甚至更优；（2）零样本跨语言及多语言微调实验显示，BLOOM的性能与单语GPT-2模型持平或更差；（3）基于RealToxicityPrompts数据集的提示文本生成毒性分析表明，BLOOM生成文本的毒性比GPT-2和GPT-3模型低至少17%。