Federated reinforcement learning enables decentralized agents to collaboratively improve policies or value estimates without exchanging raw trajectories. However, FedAvg-style parameter averaging is not function-space consistent: when clients use heterogeneous encoders or even identical nonlinear networks, averaged parameters need not correspond to the weighted average of client value functions in any common function space. We propose FedQHD, a federated Q-learning method using hyperdimensional (random-feature) state encoders with a linear readout, so that Q-functions are nonlinear in state yet linear in trainable parameters. This linear structure enables closed-form aggregation. With a shared encoder, the function-space consensus update coincides exactly with weighted averaging of local readout matrices. With heterogeneous encoders, the server constructs a global teacher by averaging client Q-values on a shared anchor-state set, and each client compiles this teacher into its local representation via a single ridge projection. We formalize the federation gap -- the error incurred when compiling a federated teacher into a heterogeneous client representation -- relative to a client-specific oracle projection. We show that this gap decomposes into subspace misalignment, anchor-set conditioning, and regularization bias. We further identify the anchor-to-dimension ratio $m \geq D_i$ as the well-conditioned regime in which the gap reduces to a multiple of the encoder heterogeneity floor. On four continuous-state, discrete-action control benchmarks, FedQHD matches or outperforms FedAvg-style baselines and distillation-based alternatives while requiring substantially less computation, and the empirical dependence of the federation gap on encoder dimension matches our theoretical analysis.

翻译：联邦强化学习使分散的智能体能够在无需交换原始轨迹的情况下协作改进策略或价值估计。然而，FedAvg风格的参数平均在函数空间中并不一致：当客户端使用异构编码器甚至相同的非线性网络时，平均参数不一定对应于任何公共函数空间中客户端价值函数的加权平均。我们提出FedQHD，一种使用超维（随机特征）状态编码器与线性读出层的联邦Q学习方法，使Q函数在状态上非线性但在可训练参数上线性。这种线性结构实现了闭式聚合。当编码器共享时，函数空间共识更新恰好等同于局部读出矩阵的加权平均。当编码器异构时，服务器通过在一组共享锚点状态上平均客户端Q值来构建全局教师，每个客户端通过单次岭投影将教师编译为其局部表示。我们形式化了联邦间隙——将联邦教师编译为异构客户端表示时产生的误差——相对于客户端特定的最优投影。我们证明该间隙可分解为子空间错位、锚点集条件和正则化偏差。进一步确定锚点与维度比 $m \geq D_i$ 为良态区域，在此区间间隙简化为编码器异构基底的倍数。在四个连续状态、离散动作控制基准上，FedQHD在显著降低计算量的同时匹配或超越FedAvg风格基线及蒸馏替代方法，且联邦间隙对编码器维度的经验依赖性与理论分析一致。