Padding is often used in tuning LLM models by adding special tokens to shorter training examples to match the length of the longest sequence in each batch. While this ensures uniformity for batch processing, it introduces inefficiencies by including irrelevant padding tokens in the computation and wastes GPU resources. On the other hand, the Hugging Face SFT trainer offers the option to use packing to combine multiple training examples up to the maximum sequence length. This allows for maximal utilization of GPU resources. However, without proper masking of each packed training example, attention will not be computed correctly when using SFT trainer. We enable and then analyse packing and Flash Attention with proper attention masking of each example and show the benefits of this training paradigm.

翻译：在大型语言模型（LLM）的微调过程中，通常采用填充（padding）方法，即通过向较短的训练样本添加特殊标记，使其与批次中最长序列的长度一致。虽然这确保了批量处理的统一性，但将无关的填充标记纳入计算会引入低效性，并浪费GPU资源。另一方面，Hugging Face的SFT训练器提供了使用打包（packing）的选项，可将多个训练样本组合至最大序列长度，从而实现GPU资源的最大化利用。然而，若未对每个打包的训练样本进行适当的掩码处理，在使用SFT训练器时注意力机制将无法正确计算。本文通过为每个样本配置正确的注意力掩码，启用并分析了打包与Flash Attention的结合使用，展示了这一训练范式的优势。