How to get insights from relational data streams in a timely manner is a hot research topic. This type of data stream can present unique challenges, such as distribution drifts, outliers, emerging classes, and changing features, which have recently been described as open environment challenges for machine learning. While existing studies have been done on incremental learning for data streams, their evaluations are mostly conducted with manually partitioned datasets. Thus, a natural question is how those open environment challenges look like in real-world relational data streams and how existing incremental learning algorithms perform on real datasets. To fill this gap, we develop an Open Environment Benchmark named OEBench to evaluate open environment challenges in relational data streams. Specifically, we investigate 55 real-world relational data streams and establish that open environment scenarios are indeed widespread in real-world datasets, which presents significant challenges for stream learning algorithms. Through benchmarks with existing incremental learning algorithms, we find that increased data quantity may not consistently enhance the model accuracy when applied in open environment scenarios, where machine learning models can be significantly compromised by missing values, distribution shifts, or anomalies in real-world data streams. The current techniques are insufficient in effectively mitigating these challenges posed by open environments. More researches are needed to address real-world open environment challenges. All datasets and code are open-sourced in https://github.com/sjtudyq/OEBench.

翻译：如何及时从关系数据流中获取洞察是一个热门研究课题。此类数据流可能呈现独特挑战，例如分布漂移、异常值、新兴类别和特征变化，这些挑战近期被描述为机器学习中的开放环境问题。尽管现有研究已涉及数据流的增量学习，但其评估大多基于人工划分的数据集。因此，一个自然的问题是：这些开放环境挑战在现实世界的关系数据流中如何呈现？现有增量学习算法在真实数据集上的表现如何？为填补这一空白，我们开发了一个名为OEBench的开放环境基准，用于评估关系数据流中的开放环境挑战。具体而言，我们研究了55个真实世界的关系数据流，证实开放环境场景确实广泛存在于真实数据集中，这对流学习算法构成了重大挑战。通过对现有增量学习算法的基准测试，我们发现，在开放环境场景下，增加数据量可能无法持续提升模型精度——机器学习模型可能因真实数据流中的缺失值、分布偏移或异常而严重受损。当前技术尚不足以有效缓解开放环境带来的这些挑战，需要更多研究来应对真实世界的开放环境问题。所有数据集和代码已开源，地址为https://github.com/sjtudyq/OEBench。