Expressive generative models have advanced robotic manipulation by capturing complex, multi-modal action distributions over temporally extended trajectories. However, fine-tuning these policies via RL remains challenging due to instability and sample inefficiency. We introduce Posterior Optimization with Clipped Objective (POCO), a principled RL framework that formulates policy improvement as a posterior inference problem tailored for temporal action chunks. Through an Expectation-Maximization procedure, POCO distills a reward-weighted implicit posterior into the policy without likelihood estimation. Furthermore, POCO adopts an offline-to-online paradigm that anchors online exploration to pre-trained priors, and its model-agnostic design scales to fine-tune large VLA models without architectural modifications. Evaluations across 7 simulation benchmarks and 4 contact-rich real-world tasks demonstrate that POCO prevents catastrophic policy collapse, outperforms SOTA baselines, and achieves a 96.7% success rate on real-world tasks. Videos are available at our project website https://cccedric.github.io/poco/.

翻译：富有表现力的生成模型通过捕获时间扩展轨迹上的复杂多模态动作分布，推动了机器人操作技术的进步。然而，由于不稳定性和样本效率低下，通过强化学习对这些策略进行微调仍然具有挑战性。我们提出了一种基于裁剪目标的后验优化（POCO）框架，这是一个原则性的强化学习框架，将策略改进表述为针对时间动作块的后验推断问题。通过期望最大化过程，POCO将奖励加权的隐式后验知识蒸馏到策略中，而无需进行似然估计。此外，POCO采用离线到在线的范式，将在线探索锚定到预训练先验上，其模型无关的设计使其无需架构修改即可扩展用于微调大型视觉-语言-动作模型。在7个模拟基准测试和4个高接触度的真实世界任务上的评估表明，POCO能够防止灾难性的策略崩溃，性能优于现有最先进基线，并在真实世界任务上实现了96.7%的成功率。相关视频可在项目网站https://cccedric.github.io/poco/上获取。