Vision Graph Neural Networks (ViGs) represent an image as a graph of patch tokens, enabling adaptive, feature-driven neighborhoods. Unlike CNNs with fixed grid biases or Vision Transformers with global token interactions, ViGs rely on dynamic graph convolution: at each layer, a feature-dependent graph is built via k-nearest-neighbor (kNN) search on current patch features, followed by message passing. This per-layer graph construction is the main bottleneck, consuming 50--95\% of graph convolution time on CPUs and GPUs, scaling as $O(N^2)$ with the number of patches $N$, and creating a sequential dependency between graph construction and feature updates. We introduce GraphLeap, a simple reformulation that removes this dependency by decoupling graph construction from feature update across layers. GraphLeap performs the feature update at layer $\ell$ using a graph built from the previous layer's features, while simultaneously using the current layer's features to construct the graph for layer $\ell+1$. This one-layer-lookahead graph construction enables concurrent graph construction and message passing. Although using prior-layer features can introduce minor accuracy degradation, lightweight fine-tuning for a few epochs is sufficient to recover the original accuracy. Building on GraphLeap, we present the first end-to-end FPGA accelerator for Vision GNNs. Our streaming, layer-pipelined design overlaps a kNN graph construction engine with a feature update engine, exploits node- and channel-level parallelism, and enables efficient on-chip dataflow without explicit edge-feature materialization. Evaluated on isotropic and pyramidal ViG models on an Alveo U280 FPGA, GraphLeap achieves up to $95.7\times$ speedup over CPU and $8.5\times$ speedup over GPU baselines, demonstrating the feasibility of real-time Vision GNN inference.

翻译：[translated abstract in Chinese] 视觉图神经网络将图像表示为图块标记的图，从而实现自适应、特征驱动的邻域。与具有固定网格偏置的CNN或具有全局标记相互作用的视觉Transformer不同，视觉图神经网络依赖于动态图卷积：在每一层，通过在当前图块特征上执行k近邻搜索来构建一个依赖于特征的图，随后进行消息传递。这种逐层图构建是主要瓶颈，在CPU和GPU上消耗了图卷积时间的50%至95%，其计算复杂度随图块数量$N$呈$O(N^2)$阶增长，并在图构建与特征更新之间产生顺序依赖。我们提出GraphLeap，一种通过跨层解耦图构建与特征更新来消除这种依赖性的简洁重构方法。GraphLeap在第$\ell$层，使用基于前一层特征构建的图执行特征更新，同时利用当前层的特征为第$\ell+1$层构建图。这种前向一层预构建图的方法实现了图构建与消息传递的并发执行。尽管使用前一层特征可能引入轻微精度下降，但通过仅数轮轻量级微调即可恢复原始精度。基于GraphLeap，我们首次提出面向视觉图神经网络的端到端FPGA加速器。我们的流式、层流水线化设计将kNN图构建引擎与特征更新引擎重叠，利用节点级和通道级并行性，并在无需显式边特征实例化的情况下实现高效片内数据流。在Alveo U280 FPGA上对同构和金字塔型ViG模型进行评估，GraphLeap相比CPU基线实现了高达$95.7\times$的加速，相比GPU基线实现了$8.5\times$的加速，证明了实时视觉图神经网络推理的可行性。