With the agile development process of most academic and corporate entities, designing a robust computational back-end system that can support their ever-changing data needs is a constantly evolving challenge. We propose the implementation of a data and language-agnostic system design that handles different data schemes and sources while subsequently providing researchers and developers a way to connect to it that is supported by a vast majority of programming languages. To validate the efficacy of a system with this proposed architecture, we integrate various data sources throughout the decentralized finance (DeFi) space, specifically from DeFi lending protocols, retrieving tens of millions of data points to perform analytics through this system. We then access and process the retrieved data through several different programming languages (R-Lang, Python, and Java). Finally, we analyze the performance of the proposed architecture in relation to other high-performance systems and explore how this system performs under a high computational load.
翻译:随着大多数学术和企业实体的敏捷开发进程,设计一个能够支持其不断变化的数据需求的稳健计算后端系统成为一项持续演变的挑战。我们提出了一种数据与语言无关的系统设计实现,它能够处理不同的数据模式和来源,同时为研究人员和开发者提供一种由绝大多数编程语言支持的连接方式。为验证采用该架构的系统的有效性,我们整合了去中心化金融(DeFi)领域的多种数据源,特别是来自DeFi借贷协议的数据,获取了数千万个数据点以便通过该系统进行分析。随后,我们通过多种不同的编程语言(R语言、Python和Java)访问并处理所获取的数据。最后,我们分析了该架构相对于其他高性能系统的性能表现,并探讨了该系统在高计算负载下的运行情况。