Large Language Models (LLMs) exhibit strong capabilities in text processing, and recent research has augmented SQL and DataFrame with LLM-powered semantic operators for data analysis. However, LLM-based data processing is hindered by slower token generation speeds compared to relational queries. To enhance real-time responsiveness, we propose OLLA, an LLM-driven online aggregation framework that accelerates semantic processing within relational queries. In contrast to batch-processing systems that yield results only after the entire dataset is processed, our approach incrementally transforms text into a structured data stream and applies online aggregation to provide progressive output. To enhance our online aggregation process, we introduce a semantic stratified sampling approach that improves data selection and expedites convergence to the ground truth. Evaluations show that OLLA reaches 1% accuracy error bound compared with labeled ground truth using less than 4% of the full-data time. It achieves speedups ranging from 1.6$\times$ to 38$\times$ across diverse domains, measured by comparing the time to reach a 5% error bound with that of full-data time. We release our code at https://github.com/olla-project/llm-online-agg.git.

翻译：大型语言模型（LLM）在文本处理方面展现出强大能力，近期研究通过引入LLM驱动的语义算子增强了SQL和DataFrame的数据分析功能。然而，与关系型查询相比，基于LLM的数据处理受限于较慢的令牌生成速度。为提升实时响应能力，我们提出OLLA——一个LLM驱动的在线聚合框架，可加速关系查询中的语义处理。与需要完整处理数据集后才输出结果的批处理系统不同，我们的方法将文本逐步转换为结构化数据流，并应用在线聚合技术以提供渐进式输出。为优化在线聚合过程，我们提出语义分层抽样方法，以改进数据选择并加速向真实值的收敛。评估结果表明，OLLA仅需不到全数据处理时间4%的时长，即可达到与标注真实值相比1%的准确度误差界限。通过比较达到5%误差界限所需时间与全数据处理时间，本框架在多个领域实现了1.6$\times$至38$\times$的加速效果。代码已发布于https://github.com/olla-project/llm-online-agg.git。