Binary concepts are empirically used by humans to generalize efficiently. And they are based on Bernoulli distribution which is the building block of information. These concepts span both low-level and high-level features such as "large vs small" and "a neuron is active or inactive". Binary concepts are ubiquitous features and can be used to transfer knowledge to improve model generalization. We propose a novel binarized regularization to facilitate learning of binary concepts to improve the quality of data generation in autoencoders. We introduce a binarizing hyperparameter $r$ in data generation process to disentangle the latent space symmetrically. We demonstrate that this method can be applied easily to existing variational autoencoder (VAE) variants to encourage symmetric disentanglement, improve reconstruction quality, and prevent posterior collapse without computation overhead. We also demonstrate that this method can boost existing models to learn more transferable representations and generate more representative samples for the input distribution which can alleviate catastrophic forgetting using generative replay under continual learning settings.

翻译：二元概念在经验上被人类用于高效泛化，它们基于构成信息基本单元的伯努利分布。这些概念涵盖从低级到高级的特征，例如“大与小”以及“神经元活跃或不活跃”。二元概念是普适性特征，可用于迁移知识以提升模型泛化能力。我们提出一种新颖的二值化正则化方法，以促进自动编码器中二元概念的学习，从而提升数据生成质量。我们在数据生成过程中引入二值化超参数$r$，对称地解耦潜空间。实验表明，该方法可轻松应用于现有变分自编码器（VAE）变体，促进对称解耦、提升重建质量，并在不增加计算开销的前提下防止后验崩溃。我们还证明，该方法能够增强现有模型学习更具迁移性的表征，并为输入分布生成更具代表性的样本，从而在持续学习场景下通过生成重放缓解灾难性遗忘。