In this work, we investigate the controllability of large language models (LLMs) on scientific summarization tasks. We identify key stylistic and content coverage factors that characterize different types of summaries such as paper reviews, abstracts, and lay summaries. By controlling stylistic features, we find that non-fine-tuned LLMs outperform humans in the MuP review generation task, both in terms of similarity to reference summaries and human preferences. Also, we show that we can improve the controllability of LLMs with keyword-based classifier-free guidance (CFG) while achieving lexical overlap comparable to strong fine-tuned baselines on arXiv and PubMed. However, our results also indicate that LLMs cannot consistently generate long summaries with more than 8 sentences. Furthermore, these models exhibit limited capacity to produce highly abstractive lay summaries. Although LLMs demonstrate strong generic summarization competency, sophisticated content control without costly fine-tuning remains an open problem for domain-specific applications.

翻译：在这项工作中，我们研究了大型语言模型（LLMs）在科学总结任务中的可控性。我们识别出决定不同类型摘要（如论文评论、摘要和通俗总结）的关键风格和内容覆盖因素。通过控制风格特征，我们发现未经微调的LLMs在MuP评论生成任务中优于人类，无论是在与参考摘要的相似度还是人类偏好方面。此外，我们展示通过基于关键词的无分类器指导（CFG）可以提升LLMs的可控性，同时在与强微调基线的词汇重叠度上相当。然而，我们的结果也表明LLMs无法一致地生成超过8句的长摘要。此外，这些模型在生成高度抽象的通俗总结方面能力有限。尽管LLMs展现出强大的通用总结能力，但在无需昂贵微调的情况下实现精细的内容控制仍是领域特定应用中的未解问题。