Synthesizers are powerful tools that allow musicians to create dynamic and original sounds. Existing commercial interfaces for synthesizers typically require musicians to interact with complex low-level parameters or to manage large libraries of premade sounds. To address these challenges, we implement SynthScribe -- a fullstack system that uses multimodal deep learning to let users express their intentions at a much higher level. We implement features which address a number of difficulties, namely 1) searching through existing sounds, 2) creating completely new sounds, 3) making meaningful modifications to a given sound. This is achieved with three main features: a multimodal search engine for a large library of synthesizer sounds; a user centered genetic algorithm by which completely new sounds can be created and selected given the users preferences; a sound editing support feature which highlights and gives examples for key control parameters with respect to a text or audio based query. The results of our user studies show SynthScribe is capable of reliably retrieving and modifying sounds while also affording the ability to create completely new sounds that expand a musicians creative horizon.

翻译：合成器是能让音乐家创造动态且原创声音的强大工具。现有的合成器商用界面通常要求音乐家操控复杂的低层级参数，或管理大型预制声音库。为应对这些挑战，我们实现了SynthScribe——一个利用多模态深度学习让用户以更高层级表达意图的全栈系统。我们实现了解决多项难题的功能，具体包括：1）搜索已有声音，2）创造全新声音，3）对给定声音进行有意义的修改。这通过三大核心功能实现：面向大型合成器声音库的多模态搜索引擎；基于用户偏好创建并选择全新声音的以用户为中心的遗传算法；以及声音编辑支持功能——该功能根据文本或音频查询，突出显示关键控制参数并提供示例。用户研究结果表明，SynthScribe能够可靠地检索和修改声音，同时具备创造全新声音的能力，从而拓展音乐家的创作视野。