Learning a common representation space between vision and language allows deep networks to relate objects in the image to the corresponding semantic meaning. We present a model that learns a shared Gaussian mixture representation imposing the compositionality of the text onto the visual domain without having explicit location supervision. By combining the spatial transformer with a representation learning approach we learn to split images into separately encoded patches to associate visual and textual representations in an interpretable manner. On variations of MNIST and CIFAR10, our model is able to perform weakly supervised object detection and demonstrates its ability to extrapolate to unseen combination of objects.

翻译：在视觉和语言之间学习一个共同的表达空间可使深层网络将图像中的物体与相应的语义含义联系起来。 我们展示了一种模型,这种模型可以学习一个共同的高斯混合表达法,将文字的构成性强加到视觉域,而没有明确的定位监督。 通过将空间变压器与一种代表学习方法相结合,我们学会将图像分成单独的编码补丁,以可解释的方式将视觉和文字表达法联系起来。关于MNIST和CIFAR10的变异,我们的模型能够进行薄弱的监控对象探测,并显示其外推至看不见的物体组合的能力。