Vertical Federated Learning (VFL) has gained increasing attention as a novel training paradigm that integrates sample alignment and feature union. However, existing VFL methods face challenges when dealing with heterogeneous local models among participants, which affects optimization convergence and generalization. To address this issue, this paper proposes a novel approach called Vertical Federated learning for training Multi-parties Heterogeneous models (VFedMH). VFedMH focuses on aggregating the embeddings of each participant's knowledge instead of intermediate results during forward propagation. The active party, who possesses labels and features of the sample, in VFedMH securely aggregates local embeddings to obtain global knowledge embeddings, and sends them to passive parties. The passive parties, who own only features of the sample, then utilize the global embeddings to propagate forward on their local heterogeneous networks. However, the passive party does not own the labels, so the local model gradient cannot be calculated locally. To overcome this limitation, the active party assists the passive party in computing its local heterogeneous model gradients. Then, each participant trains their local model using the heterogeneous model gradients. The objective is to minimize the loss value of their respective local heterogeneous models. Additionally, the paper provides a theoretical analysis of VFedMH's convergence performance. Extensive experiments are conducted to demonstrate that VFedMH can simultaneously train multiple heterogeneous models with heterogeneous optimization and outperform some recent methods in model performance.

翻译：纵向联邦学习（Vertical Federated Learning, VFL）作为一种融合样本对齐与特征联合的新型训练范式，日益受到关注。然而，现有VFL方法在处理参与者间异构局部模型时面临挑战，这影响了优化收敛性与泛化能力。针对该问题，本文提出了一种名为VFedMH（面向多方异构模型训练的纵向联邦学习）的创新方法。VFedMH在前向传播过程中，专注于聚合各参与者知识对应的嵌入向量，而非中间结果。持有样本标签与特征的主动方在VFedMH中安全聚合局部嵌入向量以获得全局知识嵌入，并将其发送至被动方。仅拥有样本特征的被动方随后利用全局嵌入在其本地异构网络上进行前向传播。由于被动方不持有标签，无法在本地计算局部模型梯度。为克服该局限，主动方协助被动方计算其本地异构模型梯度。随后，各参与者利用异构模型梯度训练其局部模型，目标是最小化各自局部异构模型的损失值。此外，本文提供了VFedMH收敛性能的理论分析。大量实验表明，VFedMH能够通过异构优化同时训练多个异构模型，并在模型性能方面优于近期部分方法。