Context retrieval systems for LLM inference face a critical challenge: high retrieval latency creates a fundamental tension between waiting for complete context (poor time-to-first-token) and proceeding without it (reduced quality). Streaming context incrementally--overlapping retrieval with inference--can mitigate this latency, but doing so with concurrent requests introduces new challenges: requests contend for GPU compute and memory, and scheduling must adapt to dynamic context arrivals. We present Stream2LLM, a streaming-aware LLM serving system for concurrent prefill-decode disaggregated deployments. Stream2LLM introduces adaptive scheduling and preemption for two distinct retrieval patterns: append-mode (progressive context accumulation) and update-mode (iterative refinement with cache invalidation). It decouples scheduling decisions from resource acquisition, enabling flexible preemption strategies guided by hardware-specific cost models, and uses longest common prefix matching to minimize redundant computation when input changes dynamically. To evaluate Stream2LLM, we collect two large-scale, real-world streaming workloads based on web crawling and approximate nearest neighbor search. Our evaluation demonstrates that streaming architecture delivers up to 11x TTFT improvements, with cost-aware scheduling providing critical benefits under memory pressure, all while maintaining throughput parity with non-streaming baselines. Code: https://github.com/rajveerb/stream2llm/tree/mlsys_artifact

翻译：面向大语言模型推理的上下文检索系统面临一个关键挑战：高检索延迟在等待完整上下文（导致较差的TTFT）与未等待完整上下文直接处理（导致质量下降）之间存在根本性矛盾。增量式流式处理上下文——将检索过程与推理过程重叠——可缓解此延迟，但在并发请求场景下引入新挑战：请求争夺GPU算力与内存资源，且调度策略需适应动态到达的上下文。本文提出Stream2LLM——一种面向并发预填充-解码分离部署的流式感知LLM服务系统。Stream2LLM针对两种不同检索模式引入自适应调度与抢占策略：追加模式（渐进式上下文累积）与更新模式（带缓存失效的迭代式精炼）。该系统将调度决策与资源获取解耦，实现基于硬件感知成本模型的灵活抢占策略，并通过最长公共前缀匹配技术最小化动态输入变化时的冗余计算。为评估Stream2LLM，我们基于网络爬虫与近似最近邻搜索收集两个大规模真实流式工作负载。实验表明，所提流式架构在保持与非流式基线相同吞吐量的前提下，可实现高达11倍的TTFT提升，其中成本感知调度在内存压力下提供关键性能收益。代码：https://github.com/rajveerb/stream2llm/tree/mlsys_artifact