Scientific research increasingly relies on distributed computational resources, storage systems, networks, and instruments, ranging from HPC and cloud systems to edge devices. Event-driven architecture (EDA) benefits applications targeting distributed research infrastructures by enabling the organization, communication, processing, reliability, and security of events generated from many sources. To support the development of scientific EDA, we introduce Octopus, a hybrid, cloud-to-edge event fabric designed to link many local event producers and consumers with cloud-hosted brokers. Octopus can be scaled to meet demand, permits the deployment of highly available Triggers for automatic event processing, and enforces fine-grained access control. We identify requirements in self-driving laboratories, scientific data automation, online task scheduling, epidemic modeling, and dynamic workflow management use cases, and present results demonstrating Octopus' ability to meet those requirements. Octopus supports producing and consuming events at a rate of over 4.2 M and 9.6 M events per second, respectively, from distributed clients.
翻译:科学研究日益依赖于分布式计算资源、存储系统、网络及仪器设备,涵盖从高性能计算与云系统到边缘设备的广泛范畴。事件驱动架构通过实现对多源事件的编排、通信、处理、可靠性与安全保障,使面向分布式研究基础设施的应用程序受益。为支持科学事件驱动架构的开发,我们提出了Octopus——一种连接云端与边缘的混合事件架构,旨在将众多本地事件生产者与消费者同云端托管的代理服务器相连接。Octopus可根据需求灵活扩展,支持部署高可用触发器以实现自动事件处理,并实施细粒度访问控制。我们通过自主实验室、科学数据自动化、在线任务调度、流行病建模及动态工作流管理等应用场景明确了系统需求,并展示了Octopus满足这些需求的能力验证结果。该系统支持分布式客户端以每秒超过420万事件的生产速率和960万事件的消费速率进行事件处理。