The ability to identify useful features or representations of the input data based on training data that achieves low prediction error on test data across multiple prediction tasks is considered the key to multitask learning success. In practice, however, one faces the issue of the choice of prediction tasks and the availability of test data from the chosen tasks while comparing the relative performance of different features. In this work, we develop a class of pseudometrics called Uniform Kernel Prober (UKP) for comparing features or representations learned by different statistical models such as neural networks when the downstream prediction tasks involve kernel ridge regression. The proposed pseudometric, UKP, between any two representations, provides a uniform measure of prediction error on test data corresponding to a general class of kernel ridge regression tasks for a given choice of a kernel without access to test data. Additionally, desired invariances in representations can be successfully captured by UKP only through the choice of the kernel function and the pseudometric can be efficiently estimated from $n$ input data samples with $O(\frac{1}{\sqrt{n}})$ estimation error. We also experimentally demonstrate the ability of UKP to discriminate between different types of features or representations based on their generalization performance on downstream kernel ridge regression tasks.

翻译：基于训练数据识别输入数据的有用特征或表示的能力，若能在多个预测任务上实现测试数据的低预测误差，则被视为多任务学习成功的关键。然而在实践中，在比较不同特征的相对性能时，人们面临着预测任务的选择问题以及从所选任务中获取测试数据的可用性问题。在本工作中，我们开发了一类称为均匀核探针（UKP）的伪度量，用于比较不同统计模型（如神经网络）学习到的特征或表示，当下游预测任务涉及核岭回归时。所提出的任意两种表示之间的伪度量UKP，在无需访问测试数据的情况下，针对给定核函数的选择，提供了对应一般核岭回归任务类别的测试数据预测误差的均匀度量。此外，表示中期望的不变性仅通过核函数的选择即可被UKP成功捕捉，且该伪度量可从$n$个输入数据样本中以$O(\frac{1}{\sqrt{n}})$的估计误差高效估计。我们还通过实验证明了UKP基于不同特征或表示在下游核岭回归任务上的泛化性能来区分它们的能力。