Natural Language Processing (NLP) inference is seeing increasing adoption by mobile applications, where on-device inference is desirable for crucially preserving user data privacy and avoiding network roundtrips. Yet, the unprecedented size of an NLP model stresses both latency and memory, creating a tension between the two key resources of a mobile device. To meet a target latency, holding the whole model in memory launches execution as soon as possible but increases one app's memory footprints by several times, limiting its benefits to only a few inferences before being recycled by mobile memory management. On the other hand, loading the model from storage on demand incurs IO as long as a few seconds, far exceeding the delay range satisfying to a user; pipelining layerwise model loading and execution does not hide IO either, due to the high skewness between IO and computation delays. To this end, we propose Speedy Transformer Inference (STI). Built on the key idea of maximizing IO/compute resource utilization on the most important parts of a model, STI reconciles the latency v.s. memory tension via two novel techniques. First, model sharding. STI manages model parameters as independently tunable shards, and profiles their importance to accuracy. Second, elastic pipeline planning with a preload buffer. STI instantiates an IO/compute pipeline and uses a small buffer for preload shards to bootstrap execution without stalling at early stages; it judiciously selects, tunes, and assembles shards per their importance for resource-elastic execution, maximizing inference accuracy. Atop two commodity SoCs, we build STI and evaluate it against a wide range of NLP tasks, under a practical range of target latencies, and on both CPU and GPU. We demonstrate that STI delivers high accuracies with 1-2 orders of magnitude lower memory, outperforming competitive baselines.

翻译：自然语言处理（NLP）推理正越来越多地被移动应用采用，其中设备端推理对于关键性地保护用户数据隐私和避免网络往返延迟至关重要。然而，NLP模型前所未有的规模同时对延迟和内存造成压力，在移动设备的两大核心资源之间形成矛盾。为达到目标延迟，将整个模型保存在内存中可尽快启动执行，但这会使单个应用的内存占用增加数倍，导致其在被移动内存管理回收前仅能进行少量推理。另一方面，按需从存储加载模型会产生长达数秒的输入输出延迟，远超用户可接受的延迟范围；逐层流水线化模型加载与执行同样无法隐藏输入输出延迟，原因在于输入输出与计算延迟之间存在高度失衡。为此，我们提出快速Transformer推理（STI）。基于最大化模型关键部分输入输出/计算资源利用的核心思想，STI通过两项新技术调和延迟与内存的矛盾。第一，模型分片。STI将模型参数管理为可独立调优的分片，并分析各分片对精度的重要性。第二，结合预加载缓冲区的弹性流水线规划。STI构建输入输出/计算流水线，并使用小型缓冲区预加载分片，从而在早期阶段无需停滞即可启动执行；它根据分片重要性审慎选择、调优并组装分片，实现资源弹性执行，最大化推理精度。基于两款商用片上系统，我们构建STI，并在多种目标延迟范围、CPU与GPU环境下，针对广泛NLP任务进行评估。结果表明，STI在将内存需求降低1-2个数量级的同时实现高精度，显著优于竞争基线方案。