Analysing and modelling interactive behaviour is an important topic in human-computer interaction (HCI) and a key requirement for the development of intelligent interactive systems. Interactive behaviour has a sequential (actions happen one after another) and hierarchical (a sequence of actions forms an activity driven by interaction goals) structure, which may be similar to the structure of natural language. Designed based on such a structure, natural language processing (NLP) methods have achieved groundbreaking success in various downstream tasks. However, few works linked interactive behaviour with natural language. In this paper, we explore the similarity between interactive behaviour and natural language by applying an NLP method, byte pair encoding (BPE), to encode mouse and keyboard behaviour. We then analyse the vocabulary, i.e., the set of action sequences, learnt by BPE, as well as use the vocabulary to encode the input behaviour for interactive task recognition. An existing dataset collected in constrained lab settings and our novel out-of-the-lab dataset were used for evaluation. Results show that this natural language-inspired approach not only learns action sequences that reflect specific interaction goals, but also achieves higher F1 scores on task recognition than other methods. Our work reveals the similarity between interactive behaviour and natural language, and presents the potential of applying the new pack of methods that leverage insights from NLP to model interactive behaviour in HCI.

翻译：分析与建模交互行为是人机交互（HCI）领域的重要课题，也是开发智能交互系统的关键需求。交互行为具有序列性（动作按顺序发生）和层次性（由交互目标驱动的动作序列构成活动）结构，这种结构可能类似于自然语言。基于此类结构设计的自然语言处理（NLP）方法已在各类下游任务中取得突破性成果。然而，目前鲜有研究将交互行为与自然语言联系起来。本文通过应用字节对编码（BPE）这一NLP方法对鼠标和键盘行为进行编码，探索交互行为与自然语言之间的相似性。我们分析了BPE学习到的词汇表（即动作序列集合），并利用该词汇表对输入行为进行编码以完成交互任务识别。采用约束实验室环境下收集的现有数据集及我们新提出的非实验室环境数据集进行评估。结果表明，这种受自然语言启发的方法不仅能学习到反映特定交互目标的动作序列，而且在任务识别中获得了比其他方法更高的F1分数。我们的工作揭示了交互行为与自然语言的相似性，并展示了将NLP领域洞察应用于HCI交互行为建模的新方法体系的潜力。