We present a method for exploring regions around individual points in a contextualized vector space (particularly, BERT space), as a way to investigate how these regions correspond to word senses. By inducing a contextualized "pseudoword" as a stand-in for a static embedding in the input layer, and then performing masked prediction of a word in the sentence, we are able to investigate the geometry of the BERT-space in a controlled manner around individual instances. Using our method on a set of carefully constructed sentences targeting ambiguous English words, we find substantial regularity in the contextualized space, with regions that correspond to distinct word senses; but between these regions there are occasionally "sense voids" -- regions that do not correspond to any intelligible sense.

翻译：我们提出了一个在环境化矢量空间(特别是BERT空间)中围绕各个点探索区域的方法,以此调查这些区域与词感的对应性。我们通过引入一个背景化的“假冒词”作为输入层静态嵌入的备用词,然后对句子中的单词进行蒙面预测,我们可以围绕单个实例以控制的方式调查BERT-空间的几何。我们使用一套针对模糊的英文词句精心构建的句子的方法,发现在环境化空间中存在大量规律性,而区域与不同的字感相对应;但在这些区域之间,有时存在“严重空虚 -- -- 与任何可理解的感知性不相符的区域。