Finding preferences expressed in natural language is an important but challenging task. State-of-the-art(SotA) methods leverage transformer-based models such as BERT, RoBERTa, etc. and graph neural architectures such as graph attention networks. Since Large Language Models (LLMs) are equipped to deal with larger context lengths and have much larger model sizes than the transformer-based model, we investigate their ability to classify comparative text directly. This work aims to serve as a first step towards using LLMs for the CPC task. We design and conduct a set of experiments that format the classification task into an input prompt for the LLM and a methodology to get a fixed-format response that can be automatically evaluated. Comparing performances with existing methods, we see that pre-trained LLMs are able to outperform the previous SotA models with no fine-tuning involved. Our results show that the LLMs can consistently outperform the SotA when the target text is large -- i.e. composed of multiple sentences --, and are still comparable to the SotA performance in shorter text. We also find that few-shot learning yields better performance than zero-shot learning.

翻译：从自然语言中识别偏好是一项重要但具有挑战性的任务。最先进方法利用基于Transformer的模型（如BERT、RoBERTa等）和图神经网络架构（如图注意力网络）。由于大语言模型能够处理更长的上下文长度且模型规模远超基于Transformer的模型，我们研究了它们直接对比较性文本进行分类的能力。本文旨在作为将LLM应用于比较性偏好分类任务的第一步。我们设计并开展了一系列实验，将分类任务格式化为LLM的输入提示，以及一种获取可自动评估的固定格式响应的方法。与现有方法的性能比较表明，预训练LLM无需微调即可超越先前最先进模型。我们的结果显示，当目标文本较长（即由多个句子组成）时，LLM能够持续优于最先进方法，而在较短文本中其性能仍与最先进方法相当。我们还发现，少样本学习的表现优于零样本学习。