A novel method, the Pareto Envelope Augmented with Reinforcement Learning (PEARL), has been developed to address the challenges posed by multi-objective problems, particularly in the field of engineering where the evaluation of candidate solutions can be time-consuming. PEARL distinguishes itself from traditional policy-based multi-objective Reinforcement Learning methods by learning a single policy, eliminating the need for multiple neural networks to independently solve simpler sub-problems. Several versions inspired from deep learning and evolutionary techniques have been crafted, catering to both unconstrained and constrained problem domains. Curriculum Learning is harnessed to effectively manage constraints in these versions. PEARL's performance is first evaluated on classical multi-objective benchmarks. Additionally, it is tested on two practical PWR core Loading Pattern optimization problems to showcase its real-world applicability. The first problem involves optimizing the Cycle length and the rod-integrated peaking factor as the primary objectives, while the second problem incorporates the mean average enrichment as an additional objective. Furthermore, PEARL addresses three types of constraints related to boron concentration, peak pin burnup, and peak pin power. The results are systematically compared against a conventional approach, the Non-dominated Sorting Genetic Algorithm. Notably, PEARL, specifically the PEARL-NdS variant, efficiently uncovers a Pareto front without necessitating additional efforts from the algorithm designer, as opposed to a single optimization with scaled objectives. It also outperforms the classical approach across multiple performance metrics, including the Hyper-volume.

翻译：本文提出了一种新方法——帕累托边界增强强化学习（PEARL），以解决多目标问题带来的挑战，尤其是在工程领域中候选解评估耗时的问题。PEARL与传统的基于策略的多目标强化学习方法不同，它通过学习单一策略，无需使用多个神经网络独立求解较简单的子问题。受深度学习与进化技术启发，我们设计了多种版本，可同时适用于无约束和有约束问题域。在这些版本中，利用课程学习有效管理约束条件。首先在经典多目标基准问题上评估PEARL的性能，此外还在两个实际的压水堆堆芯装料方案优化问题中验证其现实适用性。第一个问题以优化循环长度和棒集成峰因子为主要目标，第二个问题则将平均富集度作为附加目标。此外，PEARL处理了与硼浓度、峰值棒燃耗及峰值棒功率相关的三类约束。结果系统性地与传统方法——非支配排序遗传算法进行了对比。值得注意的是，PEARL（特别是PEARL-NdS变体）无需算法设计者额外努力即可高效找出帕累托前沿，而无需像使用缩放目标进行单次优化那样。同时，它在包含超体积在内的多个性能指标上均优于传统方法。