Automated Machine Learning (AutoML) has greatly advanced applications of Machine Learning (ML) including model compression, machine translation, and computer vision. Recommender Systems (RecSys) can be seen as an application of ML. Yet, AutoML has found little attention in the RecSys community; nor has RecSys found notable attention in the AutoML community. Only few and relatively simple Automated Recommender Systems (AutoRecSys) libraries exist that adopt AutoML techniques. However, these libraries are based on student projects and do not offer the features and thorough development of AutoML libraries. We set out to determine how AutoML libraries perform in the scenario of an inexperienced user who wants to implement a recommender system. We compared the predictive performance of 60 AutoML, AutoRecSys, ML, and RecSys algorithms from 15 libraries, including a mean predictor baseline, on 14 explicit feedback RecSys datasets. To simulate the perspective of an inexperienced user, the algorithms were evaluated with default hyperparameters. We found that AutoML and AutoRecSys libraries performed best. AutoML libraries performed best for six of the 14 datasets (43%), but it was not always the same AutoML library performing best. The single-best library was the AutoRecSys library Auto-Surprise, which performed best on five datasets (36%). On three datasets (21%), AutoML libraries performed poorly, and RecSys libraries with default parameters performed best. Although, while obtaining 50% of all placements in the top five per dataset, RecSys algorithms fall behind AutoML on average. ML algorithms generally performed the worst.
翻译:自动机器学习(AutoML)已极大推动了机器学习(ML)在模型压缩、机器翻译和计算机视觉等领域的应用。推荐系统(RecSys)可视为ML的一项具体应用。然而,AutoML在RecSys领域未获足够关注,RecSys在AutoML社区中也未得到显著重视。目前仅存在少数功能相对简单的自动化推荐系统(AutoRecSys)库采用了AutoML技术,但这些库多基于学生项目开发,不具备AutoML库的功能完整性和深度开发特性。本研究旨在评估AutoML库在缺乏经验的用户实现推荐系统场景中的表现。我们比较了来自15个库的60种AutoML、AutoRecSys、ML和RecSys算法(含均值预测基线)在14个显式反馈RecSys数据集上的预测性能。为模拟缺乏经验用户的视角,所有算法均采用默认超参数进行评估。实验发现,AutoML和AutoRecSys库表现最佳。在14个数据集中,AutoML库在6个(43%)上排名第一,但最优结果并非始终来自同一AutoML库。单库表现最优的是AutoRecSys库Auto-Surprise,在5个数据集(36%)上取得最佳效果。在3个数据集(21%)上AutoML表现不佳,采用默认参数的RecSys库反而取得最优结果。尽管RecSys算法在全部数据集中占据前五名50%的席位,但其平均性能仍落后于AutoML。ML算法整体表现最差。