Generally capable agents must learn from experience in ways that generalize across tasks and environments. The fundamental problems of learning, including credit assignment, overfitting, forgetting, local optima, and high-variance learning signals, persist whether the learned object lies in parameter space or context space. While these challenges are well understood in classical machine learning optimization, they remain underexplored in context space, leading current methods to be fragmented and ad hoc. We present Reflective Context Learning (RCL), a unified framework for agents that learn through repeated interaction, reflection on behavior and failure modes, and iterative updates to context. In RCL, reflection converts trajectories and current context into a directional update signal analogous to gradients, while mutation applies that signal to improve future behavior in context space. We recast recent context-optimization approaches as instances of this shared learning problem and systematically extend them with classical optimization primitives, including batching, improved credit-assignment signal, auxiliary losses, failure replay, and grouped rollouts for variance reduction. On AppWorld, BrowseComp+, and RewardBench2, these primitives improve over strong baselines, with their relative importance shifting across task regimes. We further analyze robustness to initialization, the effects of batch size, sampling and curriculum strategy, optimizer-state variants, and the impact of allocating stronger or weaker models to different optimization components. Our results suggest that learning through context updates should be treated not as a set of isolated algorithms, but as an optimization problem whose mechanisms can be studied systematically and improved through transferable principles.

翻译：具有通用能力的智能体必须通过经验学习，以在任务和环境中实现泛化。无论学习对象位于参数空间还是上下文空间，学习的基本问题（包括信用分配、过拟合、遗忘、局部最优以及高方差学习信号）始终存在。这些挑战在经典机器学习优化中已得到充分理解，但在上下文空间中仍未得到充分探索，导致当前方法零散且具有临时性。我们提出反思性上下文学习（Reflective Context Learning, RCL），这是一个统一的框架，适用于通过重复交互、对行为和失败模式的反思以及上下文的迭代更新进行学习的智能体。在RCL中，反思将轨迹和当前上下文转化为类似梯度的方向性更新信号，而变异则在上下文空间中应用该信号以改进未来行为。我们将近期上下文优化方法重新表述为这一共享学习问题的实例，并系统地用经典优化原语对其进行扩展，包括批处理、改进的信用分配信号、辅助损失、失败回放以及用于方差缩减的分组展开。在AppWorld、BrowseComp+和RewardBench2上，这些原语相比强基线有所改进，其相对重要性随任务场景而变化。我们进一步分析了对初始化的鲁棒性、批量大小的影响、采样与课程策略、优化器状态变体，以及将更强或更弱模型分配给不同优化组件的影响。我们的结果表明，通过上下文更新进行学习不应被视为一组孤立的算法，而应被视为一个优化问题，其机制可以系统地研究并通过可迁移的原则加以改进。