We present Memory Augmented Policy Optimization (MAPO), a simple and novel way to leverage a memory buffer of promising trajectories to reduce the variance of policy gradient estimate. MAPO is applicable to deterministic environments with discrete actions, such as structured prediction and combinatorial optimization tasks. We express the expected return objective as a weighted sum of two terms: an expectation over the high-reward trajectories inside the memory buffer, and a separate expectation over trajectories outside the buffer. To make an efficient algorithm of MAPO, we propose: (1) memory weight clipping to accelerate and stabilize training; (2) systematic exploration to discover high-reward trajectories; (3) distributed sampling from inside and outside of the memory buffer to scale up training. MAPO improves the sample efficiency and robustness of policy gradient, especially on tasks with sparse rewards. We evaluate MAPO on weakly supervised program synthesis from natural language (semantic parsing). On the WikiTableQuestions benchmark, we improve the state-of-the-art by 2.6%, achieving an accuracy of 46.3%. On the WikiSQL benchmark, MAPO achieves an accuracy of 74.9% with only weak supervision, outperforming several strong baselines with full supervision. Our source code is available at https://github.com/crazydonkey200/neural-symbolic-machines

翻译：我们提出记忆增强政策优化(MAPO),这是利用有希望的轨迹的记忆缓冲来减少政策梯度估计差异的一种简单和新颖的方法,用记忆增强政策优化(MAPO)来利用有希望的轨迹的记忆缓冲,以降低政策梯度估计的差异。MAPO适用于具有分立行动的确定性环境,例如结构化预测和组合优化任务。我们表示预期的回归目标为两个条件的加权和加权之和:对记忆缓冲内高回报轨迹的预期,以及对缓冲外轨迹的单独期望。为了对MAPO的高效算法,我们提议:(1) 加快和稳定培训;(2) 系统探索以发现高回报轨迹;(3) 从记忆缓冲内外分散取样,以扩大培训。MAPO提高政策梯度的抽样效率和稳健性,特别是在报酬微薄的任务上。我们评估MAPO关于自然语言受监管不力的节目合成(Smantical pary)。关于维基表质基准,我们改进了状态-艺术的算法,达到46.3%的精确度。