We propose $\textit{iterative inversion}$ -- an algorithm for learning an inverse function without input-output pairs, but only with samples from the desired output distribution and access to the forward function. The key challenge is a $\textit{distribution shift}$ between the desired outputs and the outputs of an initial random guess, and we prove that iterative inversion can steer the learning correctly, under rather strict conditions on the function. We apply iterative inversion to learn control. Our input is a set of demonstrations of desired behavior, given as video embeddings of trajectories (without actions), and our method iteratively learns to imitate trajectories generated by the current policy, perturbed by random exploration noise. Our approach does not require rewards, and only employs supervised learning, which can be easily scaled to use state-of-the-art trajectory embedding techniques and policy representations. Indeed, with a VQ-VAE embedding, and a transformer-based policy, we demonstrate non-trivial continuous control on several tasks. Further, we report an improved performance on imitating diverse behaviors compared to reward based methods.

翻译：我们提出了一种基于迭代反演的算法——无需输入-输出对，仅通过从期望输出分布中采样并利用前向函数即可学习逆函数。该方法面临的核心挑战是期望输出与初始随机猜测输出之间存在分布偏移。我们证明，在对函数施加相当严格条件的前提下，迭代反演能够正确引导学习过程。我们将迭代反演应用于控制学习任务：输入为演示期望行为的轨迹视频嵌入（不含动作信息），该方法通过迭代学习模仿当前策略生成且叠加随机探索噪声的轨迹。本方法无需奖励机制，仅采用监督学习，便于扩展至最先进的轨迹嵌入技术与策略表征。实验证明，借助VQ-VAE嵌入与基于Transformer的策略，我们成功实现了多个任务上非平凡的连续控制。此外，与基于奖励的方法相比，该方法在模仿多样化行为方面展现出更优性能。