Effective research ideation is a critical step for scientific research. However, the exponential increase in scientific literature makes it challenging for researchers to stay current with recent advances and identify meaningful research directions. Recent developments in large language models~(LLMs) suggest a promising avenue for automating the generation of novel research ideas. However, existing methods for idea generation either trivially prompt LLMs or directly expose LLMs to extensive literature without indicating useful information. Inspired by the research process of human researchers, we propose a Chain-of-Ideas~(CoI) agent, an LLM-based agent that organizes relevant literature in a chain structure to effectively mirror the progressive development in a research domain. This organization facilitates LLMs to capture the current advancements in research, thereby enhancing their ideation capabilities. Furthermore, we propose Idea Arena, an evaluation protocol that can comprehensively evaluate idea generation methods from different perspectives, aligning closely with the preferences of human researchers. Experimental results indicate that the CoI agent consistently outperforms other methods and shows comparable quality as humans in research idea generation. Moreover, our CoI agent is budget-friendly, with a minimum cost of \$0.50 to generate a candidate idea and its corresponding experimental design.

翻译：有效的研究构思是科学研究的关键步骤。然而，科学文献的指数级增长使得研究人员难以跟进最新进展并识别有意义的研究方向。大型语言模型（LLMs）的最新发展为自动化生成新颖研究思路提供了一条前景广阔的途径。然而，现有的思路生成方法要么简单地提示LLMs，要么直接将LLMs暴露于大量文献而不指明有用信息。受人类研究者研究过程的启发，我们提出了一种思维链（CoI）智能体，这是一种基于LLM的智能体，它以链式结构组织相关文献，从而有效反映研究领域的渐进式发展。这种组织方式有助于LLMs捕捉当前的研究进展，进而增强其构思能力。此外，我们提出了思路竞技场，这是一种评估协议，能够从不同角度全面评估思路生成方法，并与人类研究者的偏好紧密契合。实验结果表明，CoI智能体始终优于其他方法，并在研究思路生成方面展现出与人类相当的质量。此外，我们的CoI智能体具有成本效益，生成一个候选思路及其对应实验设计的最低成本仅为0.50美元。