The rise of LLMs has deflected a growing portion of human-computer interactions towards LLM-based chatbots. The remarkable abilities of these models allow users to interact using long, diverse natural language text covering a wide range of topics and styles. Phrasing these messages is a time and effort consuming task, calling for an autocomplete solution to assist users. We introduce the task of chatbot interaction autocomplete. We present ChaI-TeA: CHat InTEraction Autocomplete; An autcomplete evaluation framework for LLM-based chatbot interactions. The framework includes a formal definition of the task, coupled with suitable datasets and metrics. We use the framework to evaluate After formally defining the task along with suitable datasets and metrics, we test 9 models on the defined auto completion task, finding that while current off-the-shelf models perform fairly, there is still much room for improvement, mainly in ranking of the generated suggestions. We provide insights for practitioners working on this task and open new research directions for researchers in the field. We release our framework to serve as a foundation for future research.

翻译：大型语言模型（LLM）的兴起使得人机交互的很大一部分转向了基于LLM的聊天机器人。这些模型的卓越能力允许用户使用涵盖广泛主题和风格的长篇、多样化自然语言文本来进行交互。构思这些消息是一项耗时费力的任务，因此需要一种自动补全解决方案来辅助用户。我们提出了聊天机器人交互自动补全这一任务。我们介绍了ChaI-TeA：聊天交互自动补全；这是一个用于评估基于LLM的聊天机器人交互的自动补全框架。该框架包括任务的形式化定义，以及配套的数据集和评估指标。在正式定义任务并确定合适的数据集和指标后，我们使用该框架评估了9个模型在定义的自动补全任务上的表现，发现虽然当前现成的模型表现尚可，但仍有很大的改进空间，主要在于生成建议的排序方面。我们为从事此任务的实践者提供了见解，并为该领域的研究人员开辟了新的研究方向。我们发布了我们的框架，以作为未来研究的基础。