Videos serve as a powerful medium to convey ideas, tell stories, and provide detailed instructions, especially through long-format tutorials. Such tutorials are valuable for learning new skills at one's own pace, yet they can be overwhelming due to their length and dense content. Viewers often seek specific information, like precise measurements or step-by-step execution details, making it essential to extract and summarize key segments efficiently. An intelligent, time-sensitive video assistant capable of summarizing and detecting highlights in long videos is highly sought after. Recent advancements in Multimodal Large Language Models offer promising solutions to develop such an assistant. Our research explores the use of multimodal models to enhance video summarization and step-by-step instruction generation within specific domains. These models need to understand temporal events and relationships among actions across video frames. Our approach focuses on fine-tuning TimeChat to improve its performance in specific domains: cooking and medical procedures. By training the model on domain-specific datasets like Tasty for cooking and MedVidQA for medical procedures, we aim to enhance its ability to generate concise, accurate summaries of instructional videos. We curate and restructure these datasets to create high-quality video-centric instruction data. Our findings indicate that when finetuned on domain-specific procedural data, TimeChat can significantly improve the extraction and summarization of key instructional steps in long-format videos. This research demonstrates the potential of specialized multimodal models to assist with practical tasks by providing personalized, step-by-step guidance tailored to the unique aspects of each domain.

翻译：视频作为一种强大的媒介，能够传达思想、讲述故事并提供详细指导，尤其是通过长格式教程。此类教程对于按个人节奏学习新技能具有重要价值，但由于其时长和内容密度，也可能令人望而生畏。观众通常寻求特定信息，如精确测量或分步执行细节，因此高效提取和总结关键片段至关重要。能够对长视频进行摘要和亮点检测的智能、时效性视频助手备受期待。多模态大语言模型的最新进展为开发此类助手提供了有前景的解决方案。我们的研究探索了利用多模态模型来增强特定领域内的视频摘要和分步指令生成。这些模型需要理解时间事件以及视频帧间动作的关联关系。我们的方法侧重于对TimeChat进行微调，以提升其在特定领域（烹饪和医疗程序）的性能。通过在领域特定数据集（如烹饪领域的Tasty和医疗程序领域的MedVidQA）上训练模型，我们旨在增强其生成教学视频简洁准确摘要的能力。我们对这些数据集进行筛选和重构，以创建高质量的视频中心化指令数据。研究结果表明，当在领域特定的流程数据上进行微调后，TimeChat能够显著提升对长格式视频中关键教学步骤的提取和摘要能力。本研究证明了专用多模态模型通过提供针对各领域独特方面量身定制的个性化分步指导，来协助实际任务的潜力。