In this paper we explore the task of modeling (semi) structured object sequences; in particular we focus our attention on the problem of developing a structure-aware input representation for such sequences. In such sequences, we assume that each structured object is represented by a set of key-value pairs which encode the attributes of the structured object. Given a universe of keys, a sequence of structured objects can then be viewed as an evolution of the values for each key, over time. We encode and construct a sequential representation using the values for a particular key (Temporal Value Modeling - TVM) and then self-attend over the set of key-conditioned value sequences to a create a representation of the structured object sequence (Key Aggregation - KA). We pre-train and fine-tune the two components independently and present an innovative training schedule that interleaves the training of both modules with shared attention heads. We find that this iterative two part-training results in better performance than a unified network with hierarchical encoding as well as over, other methods that use a {\em record-view} representation of the sequence \cite{de2021transformers4rec} or a simple {\em flattened} representation of the sequence. We conduct experiments using real-world data to demonstrate the advantage of interleaving TVM-KA on multiple tasks and detailed ablation studies motivating our modeling choices. We find that our approach performs better than flattening sequence objects and also allows us to operate on significantly larger sequences than existing methods.

翻译：本文探讨了（半）结构化对象序列建模的任务，重点研究了为这类序列开发一种结构感知的输入表示方法。在此类序列中，每个结构化对象由一组编码其属性的键值对表示。给定一个键的集合，结构化对象序列可视为每个键的值随时间演化的过程。我们通过特定键的值编码并构建序列表示（时间值建模——TVM），然后对一组键条件化的值序列进行自注意力操作，生成结构化对象序列的表示（键聚合——KA）。我们对这两个组件分别进行预训练和微调，并提出了一种创新的训练计划，该计划通过共享注意力头交替训练两个模块。我们发现，这种迭代式两部分训练在性能上优于采用分层编码的统一网络，以及使用序列的“记录视图”表示或简单“扁平化”表示的其他方法。我们使用真实数据开展实验，证明了TVM-KA交替训练在多个任务上的优势，并通过详细的消融研究验证了建模选择的合理性。结果表明，我们的方法不仅优于扁平化序列对象的方法，还能处理比现有方法规模大得多的序列。