Large language models (LLMs) and large visual language models (LVLMs) have been at the forefront of the artificial intelligence field, particularly for tasks like text generation, video captioning, and question-answering. Typically, it is more applicable to train these models on broader knowledge bases or datasets to increase generalizability, learn relationships between topics, and recognize patterns. Instead, we propose to provide instructional datasets specific to the task of each modality within a distinct domain and then fine-tune the parameters of the model using LORA. With our approach, we can eliminate all noise irrelevant to the given task while also ensuring that the model generates with enhanced precision. For this work, we use Video-LLaVA to generate recipes given cooking videos without transcripts. Video-LLaVA's multimodal architecture allows us to provide cooking images to its image encoder, cooking videos to its video encoder, and general cooking questions to its text encoder. Thus, we aim to remove all noise unrelated to cooking while improving our model's capabilities to generate specific ingredient lists and detailed instructions. As a result, our approach to fine-tuning Video-LLaVA leads to gains over the baseline Video-LLaVA by 2% on the YouCook2 dataset. While this may seem like a marginal increase, our model trains on an image instruction dataset 2.5% the size of Video-LLaVA's and a video instruction dataset 23.76% of Video-LLaVA's.

翻译：大型语言模型（LLMs）与大型视觉语言模型（LVLMs）一直处于人工智能领域的前沿，尤其在文本生成、视频描述和问答等任务中表现突出。通常，在更广泛的知识库或数据集上训练这些模型更具适用性，以增强泛化能力、学习主题间关联并识别模式。相反，我们提出为特定领域内各模态的任务提供专门的指令数据集，然后使用LORA对模型参数进行微调。通过我们的方法，可以消除与给定任务无关的所有噪声，同时确保模型以更高的精度生成内容。在本研究中，我们采用Video-LLaVA为无字幕烹饪视频生成食谱。Video-LLaVA的多模态架构允许我们向其图像编码器提供烹饪图像，向视频编码器提供烹饪视频，并向文本编码器提供通用烹饪问题。因此，我们的目标是消除所有与烹饪无关的噪声，同时提升模型生成具体配料清单和详细操作说明的能力。实验结果表明，在YouCook2数据集上，我们对Video-LLaVA的微调方法相比基线Video-LLaVA取得了2%的性能提升。尽管这一提升看似有限，但我们的模型训练所使用的图像指令数据集规模仅为Video-LLaVA的2.5%，视频指令数据集规模为Video-LLaVA的23.76%。