We consider the conversion of musical recordings into human-readable sheet music annotated with timestamps. Such output lets a listener clearly visualize rubato (temporally expressive playing), a learner diagnose ensemble precision and timing choices against the written music, and a musicology scholar compare performance styles across recordings of the same work. We introduce (1) a prompt-conditioned encoder-decoder model, named Rubato, trained to output (2) a new textual representation for polyphonic music, named InterMo, which we designed for compatibility with sequence-to-sequence training. Our experiments demonstrate that Rubato produces timestamped piano sheet music from audio with higher notational accuracy than the best existing approaches, which are based on cascades. We find that even if the cascade is given ground-truth MIDI instead of audio, Rubato performs better, suggesting that the ceiling of existing approaches is primarily representational, not acoustic. Further, because Rubato is trained on several related tasks (with prompts), it competes with or outperforms the best single-task systems on related but simpler tasks like MIDI note grounding and beat/downbeat detection. A demo is available at https://nctamer.github.io/rubato-transcription .

翻译：我们研究将音乐录音转化为带时间戳的人类可读乐谱。此类输出能让听者清晰可视化节奏自由（时间表现性演奏），帮助学习者与书面乐谱对照诊断合奏精度与时值选择，并支持音乐学者比较同一作品不同录音的演奏风格。我们提出：(1) 名为Rubato的提示条件编码器-解码器模型，其训练输出为(2) 复调音乐的新型文本表示InterMo——专为与序列到序列训练兼容设计。实验表明，Rubato从音频生成带时间戳钢琴乐谱的记谱准确性优于基于级联的现有最佳方法。即便级联方法以真实MIDI而非音频为输入，Rubato仍表现更优，表明现有方法的瓶颈主要来自表征层面而非声学层面。此外，由于Rubato通过提示在多任务上进行训练，在MIDI音符定位、拍子/强拍检测等相关但更简单任务上，它能与最佳单任务系统竞争甚至超越其表现。演示地址：https://nctamer.github.io/rubato-transcription