The emergence of Large Language Models (LLMs) such as ChatGPT and LLaMA encounter limitations in domain-specific tasks, with these models often lacking depth and accuracy in specialized areas, and exhibiting a decrease in general capabilities when fine-tuned, particularly analysis ability in small sized models. To address these gaps, we introduce ICE-GRT, utilizing Reinforcement Learning from Human Feedback (RLHF) grounded in Proximal Policy Optimization (PPO), demonstrating remarkable ability in in-domain scenarios without compromising general task performance. Our exploration of ICE-GRT highlights its understanding and reasoning ability to not only generate robust answers but also to provide detailed analyses of the reasons behind the answer. This capability marks a significant progression beyond the scope of Supervised Fine-Tuning models. The success of ICE-GRT is dependent on several crucial factors, including Appropriate Data, Reward Size Scaling, KL-Control, Advantage Normalization, etc. The ICE-GRT model exhibits state-of-the-art performance in domain-specific tasks and across 12 general Language tasks against equivalent size and even larger size LLMs, highlighting the effectiveness of our approach. We provide a comprehensive analysis of the ICE-GRT, underscoring the significant advancements it brings to the field of LLM.

翻译：随着ChatGPT和LLaMA等大型语言模型的出现，在领域特定任务中面临局限，这些模型在专业领域常缺乏深度和准确性，且微调后通用能力下降，特别是小型模型的分析能力。为解决这些问题，我们提出ICE-GRT，利用基于近端策略优化的从人类反馈中强化学习，在领域内场景中展现出卓越能力，同时不牺牲通用任务性能。我们对ICE-GRT的探索凸显其理解与推理能力，不仅能生成稳健答案，还能提供答案背后原因的详细分析。这一能力标志着超越监督微调模型的重大进展。ICE-GRT的成功依赖于多个关键因素，包括适中数据、奖励规模缩放、KL控制、优势归一化等。ICE-GRT模型在领域特定任务及12项通用语言任务中，与同等规模甚至更大规模的大型语言模型相比展现出最先进性能，突显了我们方法的有效性。我们对ICE-GRT进行了全面分析，强调了其为大型语言模型领域带来的显著进步。