Striking a balance between integration and modularity is crucial for a machine learning library to be versatile and user-friendly, especially in handling decision and control tasks that involve large development teams and complex, real-world data, and environments. To address this issue, we propose TorchRL, a generalistic control library for PyTorch that provides well-integrated, yet standalone components. With a versatile and robust primitive design, TorchRL facilitates streamlined algorithm development across the many branches of Reinforcement Learning (RL) and control. We introduce a new PyTorch primitive, TensorDict, as a flexible data carrier that empowers the integration of the library's components while preserving their modularity. Hence replay buffers, datasets, distributed data collectors, environments, transforms and objectives can be effortlessly used in isolation or combined. We provide a detailed description of the building blocks, supporting code examples and an extensive overview of the library across domains and tasks. Finally, we show comparative benchmarks to demonstrate its computational efficiency. TorchRL fosters long-term support and is publicly available on GitHub for greater reproducibility and collaboration within the research community. The code is opensourced on https://github.com/pytorch/rl.

翻译：在集成性与模块化之间取得平衡，对于机器学习库实现通用性及用户友好性至关重要，尤其当涉及由大型开发团队协作处理、包含复杂真实数据与环境的决策控制任务时。针对该问题，我们提出TorchRL——一个面向PyTorch的通用控制库，其提供了高度集成且可独立运行的组件。通过设计通用且鲁棒的基元，TorchRL能够简化强化学习（RL）与控制领域多个分支的算法开发流程。我们引入了一种新型PyTorch基元——TensorDict，作为灵活的数据载体，在保持各组件模块化的同时实现其深度集成。因此，回放缓冲区、数据集、分布式数据采集器、环境、数据变换器以及目标函数均可独立使用或自由组合。本文详细描述了各构建模块，提供了配套代码示例，并从跨领域与任务维度对库进行全面概述。最后，通过对比基准测试验证了其计算效率。TorchRL致力于长期维护，并已在GitHub上开源，以促进研究社区的可复现性与协作。相关代码已公开于https://github.com/pytorch/rl。