Stage lighting is a vital component in live music performances, shaping an engaging experience for both musicians and audiences. In recent years, Automatic Stage Lighting Control (ASLC) has attracted growing interest due to the high costs of hiring or training professional lighting engineers. However, most existing ASLC solutions only classify music into limited categories and map them to predefined light patterns, resulting in formulaic and monotonous outcomes that lack rationality. To address this gap, this paper presents Skip-BART, an end-to-end model that directly learns from experienced lighting engineers and predict vivid, human-like stage lighting. To the best of our knowledge, this is the first work to conceptualize ASLC as a generative task rather than merely a classification problem. Our method adapts the BART model to take audio music as input and produce light hue and value (intensity) as output, incorporating a novel skip connection mechanism to enhance the relationship between music and light within the frame grid. To address the lack of available datasets, we create the first stage lighting dataset, along with several pre-training and transfer learning techniques to improve model training with limited data. We validate our method through both quantitative analysis and an human evaluation, demonstrating that Skip-BART outperforms conventional rule-based methods across all evaluation metrics and shows only a limited gap compared to real lighting engineers. To support further research, we have made our self-collected dataset, code, and trained model parameters available at https://github.com/RS2002/Skip-BART .
翻译:舞台灯光是现场音乐表演的关键组成部分,为音乐家和观众塑造沉浸式体验。近年来,由于雇佣或培训专业灯光师成本高昂,自动舞台灯光控制(ASLC)受到日益关注。然而,现有大多数ASLC解决方案仅将音乐分类至有限类别,并将其映射至预定义灯光模式,导致结果程式化、单调且缺乏合理性。为弥补这一不足,本文提出Skip-BART——一种端到端模型,可直接从经验丰富的灯光师处学习并预测生动拟人的舞台灯光。据我们所知,这是首次将ASLC概念化为生成式任务而非单纯分类问题的研究。本方法适配BART模型,以音频音乐作为输入,输出灯光色调与明度(强度),并通过新颖的跳跃连接机制增强帧网格内音乐与灯光的关联性。针对可用数据集匮乏的问题,我们创建了首个舞台灯光数据集,并结合多种预训练与迁移学习技术以提升有限数据下的模型训练效果。通过定量分析与人工评估验证,本方法在所有评估指标上均优于传统基于规则的方法,且与真实灯光师的差距有限。为促进后续研究,我们已将自采集数据集、代码及训练模型参数发布于https://github.com/RS2002/Skip-BART。