Vision-Language Models (VLMs) have shown promise in generating plotting code from chart images, yet achieving structural fidelity remains challenging. Existing approaches largely rely on supervised fine-tuning, encouraging surface-level token imitation rather than faithful modeling of underlying chart structure, which often leads to hallucinated or semantically inconsistent outputs. We propose Chart Specification, a structured intermediate representation that shifts training from text imitation to semantically grounded supervision. Chart Specification filters syntactic noise to construct a structurally balanced training set and supports a Spec-Align Reward that provides fine-grained, verifiable feedback on structural correctness, enabling reinforcement learning to enforce consistent plotting logic. Experiments on three public benchmarks show that our method consistently outperforms prior approaches. With only 3K training samples, we achieve strong data efficiency, surpassing leading baselines by up to 61.7% on complex benchmarks, and scaling to 4K samples establishes new state-of-the-art results across all evaluated metrics. Overall, our results demonstrate that precise structural supervision offers an efficient pathway to high-fidelity chart-to-code generation. Code and dataset are available at: https://github.com/Mighten/chart-specification-paper

翻译：视觉语言模型（VLMs）在从图表图像生成绘图代码方面展现出潜力，但实现结构保真度仍具挑战性。现有方法主要依赖于监督微调，鼓励表面层级的标记模仿而非对底层图表结构的忠实建模，这常导致产生幻觉或语义不一致的输出。我们提出图表规范，一种结构化的中间表示，将训练从文本模仿转向基于语义的监督。图表规范通过过滤句法噪声构建结构平衡的训练集，并支持一种规范对齐奖励机制，该机制提供关于结构正确性的细粒度、可验证反馈，使强化学习能够强制执行一致的绘图逻辑。在三个公开基准测试上的实验表明，我们的方法持续优于先前方法。仅使用3K训练样本，我们实现了强大的数据效率，在复杂基准测试上超越领先基线高达61.7%，并且扩展到4K样本时在所有评估指标上均建立了新的最先进结果。总体而言，我们的结果表明，精确的结构监督为高保真度的图表到代码生成提供了一条高效路径。代码和数据集可在以下网址获取：https://github.com/Mighten/chart-specification-paper