Heterogeneous Differential Privacy (HDP) in Federated Learning (FL) allows clients to select individual privacy budgets ($\varepsilon_i$) according to institutional policies and data sensitivity. In practice, many HDP-FL systems employ $\varepsilon$-aware server aggregation to improve model utility by re-weighting client updates according to their declared privacy budgets. However, gradient updates in FL retain structural patterns induced by non-independent and identically-distributed (non-IID) data, and these additional signals exposed by $\varepsilon$-aware aggregation create new opportunities for inference by an honest-but-curious server. In this work, we first show that a server equipped with gradient denoising and surrogate modeling can mount a \emph{Privacy Inference Attack} that infers distributional attributes of clients and links updates from the same client across training rounds, measured via surrogate inference accuracy and linkage success, under realistic knowledge constraints. The Shuffle-Model has been widely studied as a defense against such inference risks by anonymizing update sources, but it is fundamentally incompatible with HDP-FL $\varepsilon$-aware aggregation. To address this challenge, we propose \textbf{IntraShuffler}, a middleware defense framework designed for HDP-FL systems. IntraShuffler introduces a privacy-aware shuffling mechanism that groups clients into privacy-compatible buckets and performs parameter-level shuffling within each bucket to disrupt persistent gradient structure while preserving $\varepsilon$-aware aggregation. Experiments across four different datasets show that IntraShuffler reduces gradient recoverability by over 60% and decreases surrogate inference accuracy from 0.78 to 0.33 while maintaining comparable model utility across multiple FL aggregation rules.

翻译：异构差分隐私联邦学习允许客户端根据机构策略和数据敏感度选择各自的隐私预算（$\varepsilon_i$）。实际应用中，许多HDP-FL系统采用$\varepsilon$感知的服务器聚合机制，通过根据客户端声明的隐私预算重新加权其更新来提升模型效用。然而，联邦学习中的梯度更新保留了非独立同分布数据引发的结构特征，这些由$\varepsilon$感知聚合暴露的附加信号为诚实但好奇的服务器创造了新的推理机会。本文首先证明，配备梯度降噪与代理建模的服务器可在现实知识约束条件下发起**隐私推理攻击**，通过代理推理准确率和链接成功率衡量，推断客户端的分布属性并关联同一客户端在不同训练轮次中的更新。Shuffle-Model作为匿名化更新源以防御此类推理风险的方法已被广泛研究，但其与HDP-FL的$\varepsilon$感知聚合存在根本性不兼容。为应对这一挑战，我们提出**IntraShuffler**——一种面向HDP-FL系统的中间件防御框架。IntraShuffler引入隐私感知混洗机制，将客户端划分为隐私兼容存储桶，并在每个桶内执行参数级混洗以破坏持久梯度结构，同时保留$\varepsilon$感知聚合能力。在四个不同数据集上的实验表明，IntraShuffler将梯度可恢复性降低超过60%，代理推理准确率从0.78降至0.33，同时在多种联邦学习聚合规则下维持了可比的模型效用。