User embeddings play a crucial role in user engagement forecasting and personalized services. Recent advances in sequence modeling have sparked interest in learning user embeddings from behavioral data. Yet behavior-based user embedding learning faces the unique challenge of dynamic user modeling. As users continuously interact with the apps, user embeddings should be periodically updated to account for users' recent and long-term behavior patterns. Existing methods highly rely on stateless sequence models that lack memory of historical behavior. They have to either discard historical data and use only the most recent data or reprocess the old and new data jointly. Both cases incur substantial computational overhead. To address this limitation, we introduce User Stateful Embedding (USE). USE generates user embeddings and reflects users' evolving behaviors without the need for exhaustive reprocessing by storing previous model states and revisiting them in the future. Furthermore, we introduce a novel training objective named future W-behavior prediction to transcend the limitations of next-token prediction by forecasting a broader horizon of upcoming user behaviors. By combining it with the Same User Prediction, a contrastive learning-based objective that predicts whether different segments of behavior sequences belong to the same user, we further improve the embeddings' distinctiveness and representativeness. We conducted experiments on 8 downstream tasks using Snapchat users' behavioral logs in both static (i.e., fixed user behavior sequences) and dynamic (i.e., periodically updated user behavior sequences) settings. We demonstrate USE's superior performance over established baselines. The results underscore USE's effectiveness and efficiency in integrating historical and recent user behavior sequences into user embeddings in dynamic user modeling.

翻译：用户嵌入在用户参与度预测和个性化服务中扮演着关键角色。序列建模的最新进展引发了从行为数据中学习用户嵌入的研究兴趣。然而，基于行为的用户嵌入学习面临动态用户建模这一独特挑战。由于用户持续与应用交互，用户嵌入需定期更新以反映其近期和长期行为模式。现有方法高度依赖缺乏历史行为记忆的无状态序列模型，要么丢弃历史数据仅使用最新数据，要么联合重新处理新旧数据——这两种方式都会产生显著的计算开销。为解决这一局限，我们提出用户状态化嵌入（USE）。USE通过存储先前模型状态并在未来重新调用这些状态，无需全面重新处理即可生成用户嵌入并反映用户演化行为。此外，我们引入了名为“未来W行为预测”的新型训练目标，通过预测更广泛范围的用户行为来突破下一令牌预测的局限。通过将其与“同一用户预测”（一种基于对比学习的目标，用于预测行为序列的不同片段是否属于同一用户）相结合，我们进一步提升了嵌入的区分度和表征能力。我们利用Snapchat用户行为日志在静态（固定用户行为序列）和动态（周期性更新用户行为序列）两种设置下，在8个下游任务上进行了实验。结果表明USE在动态用户建模中整合历史与近期用户行为序列到用户嵌入方面具有卓越性能，其有效性和效率均显著优于现有基线方法。