Scientific research increasingly relies on distributed computational resources, storage systems, networks, and instruments, ranging from HPC and cloud systems to edge devices. Event-driven architecture (EDA) benefits applications targeting distributed research infrastructures by enabling the organization, communication, processing, reliability, and security of events generated from many sources. To support the development of scientific EDA, we introduce Octopus, a hybrid, cloud-to-edge event fabric designed to link many local event producers and consumers with cloud-hosted brokers. Octopus can be scaled to meet demand, permits the deployment of highly available Triggers for automatic event processing, and enforces fine-grained access control. We identify requirements in self-driving laboratories, scientific data automation, online task scheduling, epidemic modeling, and dynamic workflow management use cases, and present results demonstrating Octopus' ability to meet those requirements. Octopus supports producing and consuming events at a rate of over 4.2 M and 9.6 M events per second, respectively, from distributed clients.
翻译:科学研究日益依赖于分布式计算资源、存储系统、网络及仪器设备,涵盖从高性能计算与云系统到边缘设备的广泛范畴。事件驱动架构通过实现对多源事件的编排、通信、处理、可靠性与安全保障,为面向分布式研究基础设施的应用带来显著优势。为支持科学事件驱动架构的开发,我们提出章鱼——一种连接云端与边缘的混合事件架构,旨在将众多本地事件生产者与消费者同云托管代理相连接。该架构可按需扩展,支持部署高可用触发器以实现自动事件处理,并实施细粒度访问控制。我们通过自主实验室、科学数据自动化、在线任务调度、流行病建模及动态工作流管理等用例明确了具体需求,并展示章鱼满足这些需求的能力验证结果。该架构支持分布式客户端以每秒超过420万事件的生产速率和960万事件的消费速率进行事件处理。