Hybrid cloud-edge infrastructures now support latency-critical workloads ranging from autonomous vehicles and surgical robotics to immersive AR/VR. However, they continue to experience crippling long-tail latency spikes whenever bursty request streams exceed the capacity of heterogeneous edge and cloud tiers. To address these long-tail latency issues, we present Latency-Aware, Predictive In-Memory Routing and Proactive Autoscaling (LA-IMR). This control layer integrates a closed-form, utilization-driven latency model with event-driven scheduling, replica autoscaling, and edge-to-cloud offloading to mitigate 99th-percentile (P99) delays. Our analytic model decomposes end-to-end latency into processing, network, and queuing components, expressing inference latency as an affine power-law function of instance utilization. Once calibrated, it produces two complementary functions that drive: (i) millisecond-scale routing decisions for traffic offloading, and (ii) capacity planning that jointly determines replica pool sizes. LA-IMR enacts these decisions through a quality-differentiated, multi-queue scheduler and a custom-metric Kubernetes autoscaler that scales replicas proactively -- before queues build up -- rather than reactively based on lagging CPU metrics. Across representative vision workloads (YOLOv5m and EfficientDet) and bursty arrival traces, LA-IMR reduces P99 latency by up to 20.7 percent compared to traditional latency-only autoscaling, laying a principled foundation for next-generation, tail-tolerant cloud-edge inference services.

翻译：混合云边基础设施现在支持从自动驾驶车辆、手术机器人到沉浸式AR/VR等延迟关键型工作负载。然而，当突发请求流超出异构边缘和云层的容量时，这些系统仍会遭受严重的尾延迟长尾尖峰问题。为了解决这些长尾延迟问题，我们提出了延迟感知预测性内存路由与主动弹性伸缩（LA-IMR）。该控制层集成了一个基于利用率的闭合式延迟模型、事件驱动调度、副本弹性伸缩以及边到云卸载，以缓解第99百分位（P99）延迟。我们的分析模型将端到端延迟分解为处理、网络和排队组件，将推理延迟表示为实例利用率的仿射幂律函数。校准后，该模型产生两个互补函数，驱动：(i) 毫秒级卸载路由决策，以及 (ii) 联合确定副本池规模的容量规划。LA-IMR 通过一个质量区分多队列调度器和一个基于自定义指标的 Kubernetes 自动伸缩器来执行这些决策——该伸缩器在队列累积之前主动扩展副本，而非基于滞后的CPU指标进行被动响应。在代表性视觉工作负载（YOLOv5m 和 EfficientDet）和突发到达轨迹上，与传统的仅延迟弹性伸缩相比，LA-IMR 将 P99 延迟降低了高达 20.7%，为下一代尾延迟容忍的云边推理服务奠定了原则性基础。