Can latent actions and environment dynamics be recovered from offline trajectories when actions are never observed? We study this question in a setting where trajectories are action-free but tagged with demonstrator identity. We assume that each demonstrator follows a distinct policy, while the environment dynamics are shared across demonstrators and identity affects the next observation only through the chosen action. Under these assumptions, the conditional next-observation distribution $p(o_{t+1}\mid o_t,e)$ is a mixture of latent action-conditioned transition kernels with demonstrator-specific mixing weights. We show that this induces, for each state, a column-stochastic nonnegative matrix factorization of the observable conditional distribution. Using sufficiently scattered policy diversity and rank conditions, we prove that the latent transitions and demonstrator policies are identifiable up to permutation of the latent action labels. We extend the result to continuous observation spaces via a Gram-determinant minimum-volume criterion, and show that continuity of the transition map over a connected state space upgrades local permutation ambiguities to a single global permutation. A small amount of labeled action data then suffices to fix this final ambiguity. These results establish demonstrator diversity as a principled source of identifiability for learning latent actions and dynamics from offline RL data.

翻译：能否从从未观测到行为的离线轨迹中恢复潜在行为和环境动态？我们研究轨迹无行为标签但带有演示者身份标识的场景。假设每位演示者遵循不同的策略，而环境动态在所有演示者间共享，且身份仅通过所选行为影响下一观测。基于这些假设，条件下一观测分布$p(o_{t+1}\mid o_t,e)$是潜在行为条件转移核的混合分布，其混合权重因演示者而异。我们证明，在每个状态下，该分布可诱导出可观测条件分布的列随机非负矩阵分解。利用充分分散的策略多样性和秩条件，我们证明了潜在转移和演示者策略在潜在行为标签置换意义下是可辨识的。通过格拉姆行列式最小体积准则，我们将结果推广至连续观测空间，并证明在连通状态空间上转移映射的连续性可将局部置换歧义提升为单一全局置换。少量标注行为数据即可消除最终歧义。这些结果表明，演示者多样性可作为从离线强化学习数据中学习潜在行为与动态的可辨识性理论来源。