Existing approaches for multimodal variational autoencoders (VAEs) face a trade-off between generative quality and coherence-i.e., they struggle to generate realistic and diverse samples that, at the same time, are semantically consistent across modalities. A recent work shows that using a simple approximation to Hölder pooling as an aggregation method improves coherence over the SOTA MMVAE+, despite assuming a single shared representation across all modalities. Yet, it slightly compromises sample diversity. Inspired by this insight, we propose Hölder++, a novel multimodal VAE that improves the generative quality-coherence trade-off through: (i) the first implementation of Hölder pooling without any approximation for multimodal VAEs; (ii) an extended architecture that models distinct shared and private (i.e., modality-specific) representations (Hölder+); and (iii) hierarchical inference that further enhances the disentanglement between the shared and private representations (Hölder++). Our experiments corroborate that Hölder++ consistently improves the generative quality-coherence trade-off, yields more structured latent spaces, and learns shared representations that are informative for downstream tasks.

翻译：现有多模态变分自编码器方法面临生成质量与一致性之间的权衡——即难以生成既真实多样、又在不同模态间语义一致的样本。近期研究表明，尽管假设所有模态共享单一表征，但采用Hölder池化的简单近似作为聚合方法，其一致性优于现有最优MMVAE+方法，然而会略微牺牲样本多样性。受此启发，我们提出Hölder++——一种新型多模态VAE，通过以下方式改进生成质量-一致性权衡：(i) 在多模态VAE中首次实现无近似的Hölder池化；(ii) 扩展架构以建模独立的共享表征与私有（即模态专用）表征（Hölder+）；(iii) 采用层级推断进一步增强共享表征与私有表征的解耦（Hölder++）。实验证明，Hölder++持续改进生成质量-一致性权衡，产生更具结构化的潜空间，并学习到对下游任务具有信息量的共享表征。