The shape of musical sound: Real‐time visualizations of expressiveness in music performance.

Despite the complex multi‐dimensional nature of musical expression in the final analysis, musical expression is conveyed by sound. Therefore the expressiveness of music must be present in the sound and therefore should be observable as fundamental and emergent features of the sonic signal. To gain insight into this feature space, a real‐time visualization tool has been developed. The fundamental physical features—pitch, dynamic level, and timbre (as represented by spectral energy distribution)—are extracted from the audio signal and displayed versus time in a real‐time animation. Emergent properties of the sound, such as musical attacks and releases, the dynamic shaping of musical lines, timing of note placements, and the subtle modulation of the tone, loudness, and timbre can be inferred from the fundamental feature set and presented to the user visually. This visualization tool provides a stimulating music performance‐learning environment to help musicians achieve their artistic goals more effectively. Such visualizations and interactions with musical sound also will promote the semantic understanding of an expressive musical language.