PyCon 2019 in Cleveland, Ohio

Sunday 10 a.m.–1 p.m. in Expo Hall

Real-time voice-to-musical-instrument translation using Python

John Carelli


Is it possible to sing a melody line and to have a recognizable musical instrument, such as a trumpet or a saxophone, reproduce that melody in real time? Further, could it be done in a sufficiently expressive manner that it could be used in a live musical performance? Those are the goals of the project that will be demonstrated in this poster session. The software detects the sung notes and drives a software based player containing a user selected musical instrument to play those pitches, in real time, as they are sung. The software contains specially developed heuristics for pitch stabilization and onset detection to overcome eccentricities in the singing voice such as vibrato, inaccurate attack, inaccurate pitch (sharp or flat), and pitch drift. Unlike “pitch-to-midi” software, the distinguishing feature of this project is the focus on live performance as opposed to the specific generation of MIDI pitch information - which affects how detected pitches are used to drive instrumentation. Python is uniquely suited to this effort with its extensive capabilities and wide variety of supporting libraries. Internally, this project takes advantage of supplemental libraries for streaming audio (pyaudio), basic pitch recognition (aubio), MIDI (mido), threading (to manage latency in pitch detection and playback), as well as a number of other basic libraries. Features and capabilities: <ul> <li>A GUI interface that controls overall functionality. It is implemented using the Kivy application development library for Python.</li> <li>A built-in instrument player with several pre-programmed instruments.</li> <li>An interface to an external instrument player. Pitch information is sent using the MIDI protocol. </li> <li>An Arduino-based, programmable, hardware controller designed specifically for the singer to use in performance. It uses the Python serial library to interface with an Arduino device and provides the singer control over a variety of aspects of musical expression including volume, octave shifts, sustain, and pitch interpretation. Additionally, the programmability allows the singer to adjust controls to taste, or to control other aspects of musical expression, such as sending MIDI control messages to an external player, if one is used.</li> </ul> The app will be available both for demonstration and for visitors to try for themselves.