Large language models (LLMs) have achieved remarkable success in NLP and multimodal tasks. Despite these successes, their development faces two main challenges: (i) high computational cost; and (ii) difficulty in conducting fair and objective evaluations. LLMs are prohibitively expensive, making it feasible for only a few major players to undertake their training, thereby constraining both research and application opportunities. This underscores the importance of cost-effective LLM training. In this paper, we utilize a growth strategy to significantly reduce LLM training cost. We demonstrate that an LLM with 101B parameters and 0.31TB tokens can be trained on a $100K budget. We also adopt a systematic evaluation paradigm for the IQ evaluation of LLMs, in complement to existing evaluations that focus more on knowledge-oriented abilities. We introduce our benchmark including evaluations on important aspects of intelligence including symbolic mapping, itrule understanding, pattern mining, and anti-interference. Such evaluations minimize the potential impact of memorization. Experimental results show that our model FLM-101B, trained with a budget of $100K, achieves comparable performance to powerful and well-known models, eg GPT-3 and GLM-130B, especially in the IQ benchmark evaluations with contexts unseen in training data. The checkpoint of FLM-101B will be open-sourced at https://huggingface.co/CofeAI/FLM-101B.

翻译：大语言模型（LLMs）在自然语言处理和多模态任务中取得了显著成功。尽管取得了这些成就，其发展仍面临两大挑战：（i）高昂的计算成本；（ii）难以进行公平客观的评估。LLMs成本过高，导致只有少数主要参与者能够承担其训练工作，从而限制了研究和应用机会。这凸显了经济高效LLM训练的重要性。本文采用一种增长策略来显著降低LLM的训练成本。我们证明，一个拥有1010亿参数和0.31TB令牌的LLM可以在10万美元预算下完成训练。我们还采用了一套系统化的评估范式来评估LLM的智商（IQ），以补充现有更侧重于知识导向能力的评估。我们引入了包含智力重要方面评估的基准，包括符号映射、规则理解、模式挖掘和抗干扰能力。此类评估最大程度地降低了记忆化可能造成的影响。实验结果表明，我们的模型FLM-101B在10万美元预算下训练后，达到了与强大且知名模型（如GPT-3和GLM-130B）相当的性能，尤其是在训练数据中未出现上下文的IQ基准评估中表现突出。FLM-101B的检查点将在https://huggingface.co/CofeAI/FLM-101B 进行开源。