Audience-participatory music performance system featuring hand gesture interpretation Available to Purchase

Visual media artists utilize synchronized video effects with music for production of modern live events. This type of artist is referred to as a video jockey, or VJ, which creative expression is used to enhance musical experience. VJ visual effects complement and enhance musical expression. In recent years, with the increase of computing power, dynamic digital art has become capable of immediately reflecting information from the real world, and the demand for interactive art that allows an audience’s action to affect the shared art has been increasing. However, VJing is one-way expression from the artist, and there are not many VJ experiences that allow the audience to contribute to the music and video display. The purpose of this study is to develop a VJ system that allows multiple audience members to simultaneously participate in visual effects generation using hand gestures and contribute to the music being performed. For the production system, we used ball bouncing graphics triggered by hand gesture control using the image recognition framework MediaPipe provided by Google. The tempo of the music played by the artist was shared from Ableton Live, a digital audio workstation, to Unity and we developed a system that synchronizes with the video effects.