We target open-world feature extrapolation problem where the feature space of input data goes through expansion and a model trained on partially observed features needs to handle new features in test data without further retraining. The problem is of much significance for dealing with features incrementally collected from different fields. To this end, we propose a new learning paradigm with graph representation and learning. Our framework contains two modules: 1) a backbone network (e.g., feedforward neural nets) as a lower model takes features as input and outputs predicted labels; 2) a graph neural network as an upper model learns to extrapolate embeddings for new features via message passing over a feature-data graph built from observed data. Based on our framework, we design two training strategies, a self-supervised approach and an inductive learning approach, to endow the model with extrapolation ability and alleviate feature-level over-fitting. We also provide theoretical analysis on the generalization error on test data with new features, which dissects the impact of training features and algorithms on generalization performance. Our experiments over several classification datasets and large-scale advertisement click prediction datasets demonstrate that our model can produce effective embeddings for unseen features and significantly outperforms baseline methods that adopt KNN and local aggregation.

翻译：我们针对开放世界特征外推问题，即输入数据的特征空间在测试阶段发生扩展，且模型需在部分观测特征上训练后，无需重新训练即可处理新特征。该问题对于处理来自不同领域、逐步收集的特征具有重要意义。为此，我们提出了一种基于图表示与学习的新学习范式。我们的框架包含两个模块：1）作为底层模型的主干网络（如前馈神经网络）以特征为输入并输出预测标签；2）作为上层模型的图神经网络通过从观测数据构建的特征-数据图上的消息传递，学习为新特征外推嵌入。基于该框架，我们设计了两种训练策略——自监督方法与归纳学习方法——以赋予模型外推能力并缓解特征级过拟合。我们还提供了关于包含新特征的测试数据上泛化误差的理论分析，揭示了训练特征与算法对泛化性能的影响。在多个分类数据集和大型广告点击预测数据集上的实验表明，我们的模型能为未见特征生成有效嵌入，且显著优于采用KNN和局部聚合的基线方法。