Curriculum learning (CL) posits that machine learning models -- similar to humans -- may learn more efficiently from data that match their current learning progress. However, CL methods are still poorly understood and, in particular for natural language processing (NLP), have achieved only limited success. In this paper, we explore why. Starting from an attempt to replicate and extend a number of recent curriculum methods, we find that their results are surprisingly brittle when applied to NLP. A deep dive into the (in)effectiveness of the curricula in some scenarios shows us why: when curricula are employed in combination with the popular Adam optimisation algorithm, they oftentimes learn to adapt to suboptimally chosen optimisation parameters for this algorithm. We present a number of different case studies with different common hand-crafted and automated CL approaches to illustrate this phenomenon, and we find that none of them outperforms optimisation with only Adam with well-chosen hyperparameters. As such, our results contribute to understanding why CL methods work, but at the same time urge caution when claiming positive results.

翻译：课程学习（CL）认为，机器学习模型——类似于人类——可能从与其当前学习进度相匹配的数据中更高效地学习。然而，课程学习方法仍未被充分理解，尤其在自然语言处理（NLP）领域仅取得有限成功。本文旨在探究其原因。从尝试复现并扩展近期若干课程学习方法入手，我们发现这些方法在NLP任务中的结果出人意料地脆弱。通过对课程学习在特定场景下（不）有效性的深入分析，我们揭示了原因：当课程学习与流行的Adam优化算法结合使用时，它们常常会学习适应该算法中次优选择的优化参数。我们通过若干常见手工设计及自动化课程学习方法的案例研究来阐释这一现象，并发现没有任何方法能优于仅使用超参数调整得当的Adam优化器。因此，我们的研究成果有助于理解课程学习方法为何有效，同时也提醒在宣称积极结果时需保持谨慎。