We present Banyan, a model that efficiently learns semantic representations by leveraging explicit hierarchical structure. While transformers excel at scale, they struggle in low-resource settings. Conversely recent structured models have shown promise as efficient learners, but lack performance. Banyan bridges this gap with two key innovations: an entangled hierarchical tree structure and diagonalized message passing, enabling it to outperform larger transformer models with just 14 non-embedding parameters. It excels in low-resource settings, offering a viable alternative for under-represented languages and highlighting its potential for efficient, interpretable NLP in resource-constrained environments.

翻译：本文提出Banyan模型，该模型通过利用显式层次结构高效学习语义表示。尽管Transformer模型在大规模场景中表现卓越，但在低资源环境下效果受限。反之，近期提出的结构化模型虽展现出高效学习潜力，但性能表现不足。Banyan通过两项关键创新填补了这一空白：纠缠层次树结构与对角化消息传递机制，使其仅用14个非嵌入参数即可超越更大规模的Transformer模型。该模型在低资源场景中表现优异，为资源匮乏语言提供了可行替代方案，并凸显了其在资源受限环境下实现高效可解释自然语言处理的潜力。