Distributed computation is always a tricky topic to deal with, especially in context of various requirements in various scenarios. A popular solution is to use Apache Spark with a setup of multiple systems forming a cluster. However, the prerequisite setup for a Spark cluster often induces an additional overhead, often limiting usage in constrained scenarios, especially in scenarios requiring context propagation. In this paper, we explore a relatively lightweight computational graph execution framework requiring little setup and fast speeds, coupled with context awareness.
翻译:分布式计算始终是一个棘手的课题,尤其是在不同场景下具有不同需求的背景下。一种常见的解决方案是使用Apache Spark,通过配置多个系统形成集群。然而,Spark集群的前置设置往往带来额外的开销,这通常会限制其在资源受限场景中的使用,尤其是在需要上下文传播的场景中。本文探索了一种相对轻量级的计算图执行框架,该框架配置要求低、执行速度快,并具备上下文感知能力。