Disentangling complex data to its latent factors of variation is a fundamental task in representation learning. Existing work on sequential disentanglement mostly provides two factor representations, i.e., it separates the data to time-varying and time-invariant factors. In contrast, we consider multifactor disentanglement in which multiple (more than two) semantic disentangled components are generated. Key to our approach is a strong inductive bias where we assume that the underlying dynamics can be represented linearly in the latent space. Under this assumption, it becomes natural to exploit the recently introduced Koopman autoencoder models. However, disentangled representations are not guaranteed in Koopman approaches, and thus we propose a novel spectral loss term which leads to structured Koopman matrices and disentanglement. Overall, we propose a simple and easy to code new deep model that is fully unsupervised and it supports multifactor disentanglement. We showcase new disentangling abilities such as swapping of individual static factors between characters, and an incremental swap of disentangled factors from the source to the target. Moreover, we evaluate our method extensively on two factor standard benchmark tasks where we significantly improve over competing unsupervised approaches, and we perform competitively in comparison to weakly- and self-supervised state-of-the-art approaches. The code is available at https://github.com/azencot-group/SKD.

翻译：将复杂数据解耦为其潜在变化因子是表示学习中的基础任务。现有的序列解耦研究主要提供双因子表示，即将数据分离为时变因子和时不变因子。与此不同，我们研究多因子解耦问题，生成多个（超过两个）语义解耦分量。我们方法的关键在于引入强归纳偏置，假设潜在空间中的底层动态可线性表示。在此假设下，自然可以利用近期提出的库普曼自编码器模型。然而，库普曼方法无法保证解耦表示，因此我们提出一种新颖的谱损失项，可生成结构化库普曼矩阵并实现解耦。总体而言，我们提出一个简单易编码的全新深度模型，该模型完全无监督且支持多因子解耦。我们展示了新的解耦能力，例如角色间单个静态因子的交换，以及从源到目标逐步交换解耦因子。此外，我们在双因子标准基准任务上进行了广泛评估，显著优于竞争性无监督方法，并与弱监督和自监督最先进方法相比具有竞争力。代码已开源：https://github.com/azencot-group/SKD。