While large language models (LLMs) excel at understanding and generating plain text, they are not specifically tailored to handle hierarchical text structures. Extracting the task-desired property from their natural language responses typically necessitates additional processing steps. In fact, selectively comprehending the hierarchical structure of large-scale text is pivotal to understanding its substance. Aligning LLMs more closely with the classification or regression values of specific task through prompting also remains challenging. To this end, we propose a novel framework called Tree-Based Hard Attention with Self-Motivation for Large Language Models (TEAROOM). TEAROOM incorporates a tree-based hard attention mechanism for LLMs to process hierarchically structured text inputs. By leveraging prompting, it enables a frozen LLM to selectively focus on relevant leaves in relation to the root, generating a tailored symbolic representation of their relationship. Moreover, TEAROOM comprises a self-motivation strategy for another LLM equipped with a trainable adapter and a linear layer. The selected symbolic outcomes are integrated into another prompt, along with the predictive value of the task. We iteratively feed output values back into the prompt, enabling the trainable LLM to progressively approximate the golden truth. TEAROOM outperforms existing state-of-the-art methods in experimental evaluations across three benchmark datasets, showing its effectiveness in estimating task-specific properties. Through comprehensive experiments and analysis, we have validated the ability of TEAROOM to gradually approach the underlying golden truth through multiple inferences.

翻译：尽管大语言模型在理解和生成纯文本方面表现出色，但它们并未针对处理层级化文本结构进行专门设计。从自然语言响应中提取任务所需属性通常需要额外的处理步骤。实际上，选择性理解大规模文本的层级结构对于把握其内涵至关重要。通过提示工程使大语言模型更紧密地匹配特定任务的分类或回归值仍具挑战性。为此，我们提出一种名为"基于树结构硬注意力与自我激励机制的大语言模型"（TEAROOM）的新型框架。TEAROOM为大语言模型引入了树结构硬注意力机制以处理层级化文本输入。通过利用提示工程，该机制使冻结的大语言模型能够选择性关注与根节点相关的叶节点，生成反映其关系的定制化符号表示。此外，TEAROOM包含针对另一个配备可训练适配器和线性层的大语言模型的自我激励策略。选定的符号化结果与任务预测值共同被整合到另一提示中。我们通过迭代将输出值反馈至提示，使可训练大语言模型逐步逼近真实值。在三个基准数据集上的实验评估表明，TEAROOM超越现有最先进方法，展现了其在估算任务特定属性方面的有效性。通过综合实验与分析，我们验证了TEAROOM通过多次推理逐步逼近潜在真实值的能力。