As the probability (and thus perplexity) of a text is calculated based on the product of the probabilities of individual tokens, it may happen that one unlikely token significantly reduces the probability (i.e., increase the perplexity) of some otherwise highly probable input, while potentially representing a simple typographical error. Also, given that perplexity is a scalar value that refers to the entire input, information about the probability distribution within it is lost in the calculation (a relatively good text that has one unlikely token and another text in which each token is equally likely they can have the same perplexity value), especially for longer texts. As an alternative to scalar perplexity this research proposes a simple algorithm used to calculate vector values based on n-gram perplexities within the input. Such representations consider the previously mentioned aspects, and instead of a unique value, the relative perplexity of each text token is calculated, and these values are combined into a single vector representing the input.

翻译：由于文本的概率（进而其困惑度）是基于各词元概率的乘积计算的，可能出现一个低概率词元（如简单笔误）显著降低原本高概率输入的全局概率（即增加困惑度）的情况。此外，鉴于困惑度作为标量值仅反映整个输入的整体特性，其计算过程会丢失输入内部概率分布的信息（例如：一个包含单个低概率词元的较优文本与另一个各词元概率均等的文本可能具有相同的困惑度值），这种信息损失对于长文本尤为显著。作为标量困惑度的替代方案，本研究提出了一种基于输入内n-gram困惑度计算向量值的简易算法。该表示方法充分考虑了上述因素，通过计算每个文本词元的相对困惑度，并将这些值组合为表征整个输入的统一向量，从而取代传统的单一标量值。