Large language models have become ubiquitous in modern life, finding applications in various domains such as natural language processing, language translation, and speech recognition. Recently, a breakthrough work [Zhao, Panigrahi, Ge, and Arora Arxiv 2023] explains the attention model from probabilistic context-free grammar (PCFG). One of the central computation task for computing probability in PCFG is formulating a particular tensor low rank approximation problem, we can call it tensor cycle rank. Given an $n \times n \times n$ third order tensor $A$, we say that $A$ has cycle rank-$k$ if there exists three $n \times k^2$ size matrices $U , V$, and $W$ such that for each entry in each \begin{align*} A_{a,b,c} = \sum_{i=1}^k \sum_{j=1}^k \sum_{l=1}^k U_{a,i+k(j-1)} \otimes V_{b, j + k(l-1)} \otimes W_{c, l + k(i-1) } \end{align*} for all $a \in [n], b \in [n], c \in [n]$. For the tensor classical rank, tucker rank and train rank, it has been well studied in [Song, Woodruff, Zhong SODA 2019]. In this paper, we generalize the previous ``rotation and sketch'' technique in page 186 of [Song, Woodruff, Zhong SODA 2019] and show an input sparsity time algorithm for cycle rank.

翻译：摘要：大型语言模型已广泛应用于现代生活，在自然语言处理、语言翻译和语音识别等各个领域均有重要应用。近期，一项突破性研究[Zhao, Panigrahi, Ge, and Arora Arxiv 2023]从概率上下文无关文法（PCFG）角度解释了注意力模型。在PCFG中计算概率的核心计算任务之一是制定特定的张量低秩近似问题，我们称之为张量循环秩。给定一个$n \times n \times n$的三阶张量$A$，若存在三个$n \times k^2$大小的矩阵$U, V, W$，使得对所有$a \in [n], b \in [n], c \in [n]$，张量的每个元素均满足： \begin{align*} A_{a,b,c} = \sum_{i=1}^k \sum_{j=1}^k \sum_{l=1}^k U_{a,i+k(j-1)} \otimes V_{b, j + k(l-1)} \otimes W_{c, l + k(i-1)} \end{align*} 则称$A$具有循环秩-$k$。对于张量的经典秩、Tucker秩和列车秩（train rank），文献[Song, Woodruff, Zhong SODA 2019]已进行了深入研究。本文推广了该文献第186页的"旋转与草图"技术，提出了一种适用于循环秩的输入稀疏时间算法。