In this paper, we explore the application of Large Language Models (LLMs) to the pre-training of music. While the prevalent use of MIDI in music modeling is well-established, our findings suggest that LLMs are inherently more compatible with ABC Notation, which aligns more closely with their design and strengths, thereby enhancing the model's performance in musical composition. To address the challenges associated with misaligned measures from different tracks during generation, we propose the development of a Synchronized Multi-Track ABC Notation (SMT-ABC Notation), which aims to preserve coherence across multiple musical tracks. Our contributions include a series of models capable of handling up to 8192 tokens, covering 90% of the symbolic music data in our training set. Furthermore, we explore the implications of the Symbolic Music Scaling Law (SMS Law) on model performance. The results indicate a promising direction for future research in music generation, offering extensive resources for community-led research through our open-source contributions.

翻译：本文探索了大型语言模型（LLMs）在音乐预训练中的应用。尽管MIDI在音乐建模中的广泛使用已得到充分验证，但我们的研究发现，大型语言模型与ABC记谱法具有更天然的兼容性——后者更契合其设计与优势，从而能提升模型在音乐创作中的表现。针对生成过程中不同音轨间小节错位的问题，我们提出了同步多轨ABC记谱法（SMT-ABC记谱法），旨在保持多音乐轨道间的连贯性。我们的贡献包括一系列可处理多达8192个标记的模型，覆盖了训练集中90%的符号音乐数据。此外，我们还探讨了符号音乐缩放定律（SMS Law）对模型性能的影响。研究结果为音乐生成领域的未来研究指明了有前景的方向，并通过我们的开源贡献为社区主导的研究提供了丰富资源。