Compact models often lose the structure of their embedding space. The issue shows up when the capacity is tight or the data spans several languages. Such collapse makes it difficult for downstream tasks to build on the resulting representation. Existing compression methods focus on aligning model outputs at a superficial level but fail to preserve the underlying manifold structure. This mismatch often leads to semantic drift in the compact model, causing both task behavior and linguistic properties to deviate from the reference model. To address those issues, we provide a new framework called Embedding Consistency Regulation (ECR). This framework first derives a set of semantic anchors from teacher embeddings (computed once offline). Then, the compact model learns to maintain consistent geometry around these anchors, without relying on matching logits or internal features. ECR adds only a small projection step at inference, without altering the decoding architecture or its runtime behavior. In experiments on a 100K multilingual corpus, ECR consistently stabilizes training and preserves semantic structure across tasks and languages. It also produces a more compact and task-aligned representation space, enabling low-capacity models to learn cleaner manifolds than conventional baselines. ECR works without teacher outputs and is compatible with, but independent of, distillation. Taken together, our results show that ECR helps compact models better follow task requirements and makes them easier to deploy under strict efficiency or privacy limits.

翻译：紧凑模型常会丢失其嵌入空间的结构。该问题在模型容量受限或数据跨越多语言时尤为明显。此类坍缩使得下游任务难以基于所得表示进行构建。现有压缩方法侧重于在表层对齐模型输出，但未能保留底层的流形结构。这种失配常导致紧凑模型出现语义漂移，使其任务行为与语言特性均偏离参考模型。为解决这些问题，我们提出了名为嵌入一致性调节（ECR）的新框架。该框架首先从教师嵌入（离线单次计算）中推导出一组语义锚点，随后紧凑模型学习围绕这些锚点保持几何一致性，而无需依赖对数或内部特征的匹配。ECR仅在推理时增加小型投影步骤，不改变解码架构或其运行时行为。在十万量级多语言语料实验中，ECR能持续稳定训练过程，并在跨任务与跨语言场景中保持语义结构。同时它还能产生更紧凑且任务对齐的表示空间，使低容量模型能学习到比传统基线更清晰的流形。ECR无需教师输出即可工作，且与蒸馏方法兼容但相互独立。综合而言，我们的结果表明ECR能帮助紧凑模型更好地遵循任务要求，并使其在严格效率或隐私限制下更易于部署。