The demands for higher performance and accuracy in neural networks (NNs) never end. Existing tensor compilation and Neural Architecture Search (NAS) techniques orthogonally optimize the two goals but actually share many similarities in their concrete strategies. We exploit such opportunities by combining the two into one and make a case for Kernel Architecture Search (KAS). KAS reviews NAS from a system perspective and zooms into a more fine-grained level to generate neural kernels with both high performance and good accuracy. To demonstrate the potential of KAS, we build an end-to-end framework, Canvas, to find high-quality kernels as convolution replacements. Canvas samples from a rich set of fine-grained primitives to stochastically and iteratively construct new kernels and evaluate them according to user-specified constraints. Canvas supports freely adjustable tensor dimension sizes inside the kernel and uses two levels of solvers to satisfy structural legality and fully utilize model budgets. The evaluation shows that by replacing standard convolutions with generated new kernels in common NNs, Canvas achieves average 1.5x speedups compared to the previous state-of-the-art with acceptable accuracy loss and search efficiency. Canvas verifies the practicability of KAS by rediscovering many manually designed kernels in the past and producing new structures that may inspire future machine learning innovations.

翻译：神经网络对更高性能和精度的追求永无止境。现有的张量编译与神经架构搜索（NAS）技术正交优化这两个目标，但在具体策略上实则存在诸多共性。我们通过将二者融合来发掘此类机遇，并提出内核架构搜索（KAS）。KAS从系统视角审视NAS，聚焦更细粒度层级，生成兼具高性能与高精度的神经内核。为展现KAS的潜力，我们构建了端到端框架Canvas，用于寻找高质量内核以替代卷积运算。Canvas从丰富的细粒度原语集中采样，通过随机迭代方式构建新内核，并根据用户指定约束进行评估。该框架支持内核内部张量维度的自由调节，并采用两级求解器确保结构合法性及充分利用模型预算。评估表明，通过替换常见神经网络中的标准卷积为所生成的新内核，Canvas在可接受精度损失与搜索效率下，相较先前最优方案实现平均1.5倍加速。Canvas通过重新发现历史上众多手动设计的内核，并生成可能启发未来机器学习创新的新型结构，验证了KAS的实用性。