Audio-video joint generation has progressed rapidly, yet substantial challenges still remain. Non-commercial approaches still suffer audio-visual asynchrony, poor lip-speech alignment, and unimodal degradation, which can be stemmed from weak audio-visual correspondence modeling, limited generalization, and scarce high-quality dense-caption data. To address these issues, we introduce Klear and delve into three axes--model architecture, training strategy, and data curation. Architecturally, we adopt a single-tower design with unified DiT blocks and an Omni-Full Attention mechanism, achieving tight audio-visual alignment and strong scalability. Training-wise, we adopt a progressive multitask regime--random modality masking to joint optimization across tasks, and a multistage curriculum, yielding robust representations, strengthening A-V aligned world knowledge, and preventing unimodal collapse. For datasets, we present the first large-scale audio-video dataset with dense captions, and introduce a novel automated data-construction pipeline which annotates and filters millions of diverse, high-quality, strictly aligned audio-video-caption triplets. Building on this, Klear scales to large datasets, delivering high-fidelity, semantically and temporally aligned, instruction-following generation in both joint and unimodal settings while generalizing robustly to out-of-distribution scenarios. Across tasks, it substantially outperforms prior methods by a large margin and achieves performance comparable to Veo 3, offering a unified, scalable path toward next-generation audio-video synthesis.
翻译:音视频联合生成技术发展迅速,但仍面临重大挑战。非商业化方法仍存在音视频不同步、唇语-语音对齐不佳以及单模态退化等问题,其根源可归结于音视频对应关系建模薄弱、泛化能力有限以及高质量密集标注数据稀缺。为解决这些问题,我们提出Klear系统,并从模型架构、训练策略和数据构建三个维度展开研究。架构方面,我们采用单塔式设计,配备统一的DiT模块和全向全注意力机制,实现了紧密的音视频对齐和强大的可扩展性。训练策略上,我们采用渐进式多任务方案——通过随机模态掩码实现跨任务联合优化,并结合多阶段课程学习,从而获得鲁棒的表征、强化音视频对齐的世界知识并防止单模态崩溃。数据构建方面,我们提出了首个大规模带密集标注的音视频数据集,并引入新颖的自动化数据构建流程,该流程可标注并筛选数百万个多样化、高质量、严格对齐的音频-视频-文本三元组。在此基础上,Klear能够扩展到大型数据集,在联合与单模态场景下均能实现高保真度、语义与时序对齐的指令跟随生成,同时在分布外场景中展现出强大的泛化能力。在各项任务中,Klear大幅超越现有方法,其性能与Veo 3相当,为下一代音视频合成提供了统一且可扩展的技术路径。