I examine a conceptual model of a recommendation system (RS) with user inflow and churn dynamics. When inflow and churn balance out, the user distribution reaches a steady state. Changing the recommendation algorithm alters the steady state and creates a transition period. During this period, the RS behaves differently from its new steady state. In particular, A/B experiment metrics obtained in transition periods are biased indicators of the RS's long term performance. Scholars and practitioners, however, often conduct A/B tests shortly after introducing new algorithms to validate their effectiveness. This A/B experiment paradigm, widely regarded as the gold standard for assessing RS improvements, may consequently yield false conclusions. I also briefly discuss the data bias caused by the user retention dynamics.

翻译：本文研究了具有用户流入与流失动态机制的推荐系统（RS）概念模型。当用户流入与流失达到平衡时，用户分布将进入稳态。推荐算法的改变会打破原有稳态并引发过渡期。在过渡期内，推荐系统的行为特征与其新稳态存在显著差异。特别值得注意的是，过渡期内的A/B实验指标作为系统长期性能的评估存在偏差。然而，学术界与工业界通常在新算法上线后立即开展A/B测试以验证其有效性。这种被广泛视为评估推荐系统改进黄金标准的实验范式，可能会产生错误结论。本文还简要探讨了用户留存动态导致的数据偏差问题。